Gold data for agents: an engineering field guide

0. Executive Summary

Thesis. Agent quality is bounded by context quality. Most production agent failures trace to retrieval and data problems rather than model capability: the context was missing, stale, contradictory, unattributable, or drowned in noise. Anthropic’s guidance frames this as context engineering: curating the smallest set of high-signal tokens that maximizes the probability of the desired outcome [A3].

Core claims of this guide:

1. Medallion (bronze/silver/gold) is the right ingestion discipline but the wrong definition of “done.” Warehouse gold is optimized for aggregates and dashboards. Agent gold must be optimized for retrieval precision, attribution, non-contradiction, and actionability.
2. Gold for agents is multi-representational. The same truth is served as canonical documents, atomic claims, entity graphs, a semantic layer over structured data, procedures/skills, and tools. Different query types need different representations.
3. Build the evaluation set before the pipeline. A gold layer without a measured retrieval and task-success baseline is unfalsifiable.
4. MCP is the interface layer, not the storage layer. Model Context Protocol standardizes how agents discover and call your gold layer. It does not make bad data good.
5. Security is a data property. Permissions, provenance, and injection resistance must be captured at ingest and propagated to every index. Bolting them on at serving time leaks data.

1. Definitions: What “Gold” Means for Agents

1.1 Medallion architecture (origin)

Popularized by Databricks [D1], the medallion architecture descends from classic staging → ODS → mart patterns.

1.2 The agent reinterpretation

1.3 Properties of agent-grade gold

A gold artifact must satisfy all of the following:

2. Reference Architecture

┌──────────────────────────────────────────────────────────────────────────┐
│ SOURCES: SaaS (Confluence, Jira, Salesforce, Zendesk, Slack), databases, │
│          object stores (PDF/Office), code repos, APIs, event streams     │
└───────────────┬──────────────────────────────────────────────────────────┘
                │  connectors / CDC / webhooks  (capture ACLs + hashes)
                ▼
┌──────────────────────────────────────────────────────────────────────────┐
│ BRONZE  — open table format (Iceberg/Delta), immutable, partitioned by   │
│           source & ingest time; raw blobs in object storage              │
└───────────────┬──────────────────────────────────────────────────────────┘
                │  parse: layout-aware extraction, OCR/VLM, table structure,
                │         code AST, thread reconstruction
                ▼
┌──────────────────────────────────────────────────────────────────────────┐
│ SILVER  — normalized docs (structured MD/JSON), entities w/ canonical IDs│
│           dedup clusters, temporal validity, PII tags, quality scores    │
└───────────────┬──────────────────────────────────────────────────────────┘
                │  distill: claim extraction, conflict resolution, summaries,
                │           procedure synthesis, contextualized chunking
                ▼
┌──────────────────────────────────────────────────────────────────────────┐
│ GOLD (agent knowledge products)                                          │
│  ├─ Canonical document store (versioned, contextualized chunks)          │
│  ├─ Claim / fact store (atomic, attributed, temporal)                    │
│  ├─ Knowledge graph (entities, relations, ownership)                     │
│  ├─ Semantic layer over structured data (metrics, dimensions, joins)     │
│  ├─ Procedures / Skills (runbooks with pre/postconditions)               │
│  ├─ Glossary & ontology (synonyms, acronyms, ID crosswalks)              │
│  └─ Hierarchical summaries (corpus → collection → doc → section)         │
└───────────────┬──────────────────────────────────────────────────────────┘
                │  index
                ▼
┌──────────────────────────────────────────────────────────────────────────┐
│ INDEXES: lexical (BM25) · dense · late-interaction/multivector · graph · │
│          SQL/semantic layer · metadata filters · ACL filters             │
└───────────────┬──────────────────────────────────────────────────────────┘
                │  retrieval services (routing, fusion, rerank, citation)
                ▼
┌──────────────────────────────────────────────────────────────────────────┐
│ SERVING: MCP servers (tools/resources/prompts) · Skills · OpenAPI tools ·│
│          A2A for agent-to-agent delegation                               │
└───────────────┬──────────────────────────────────────────────────────────┘
                ▼
           AGENT RUNTIME  ──► traces (OTel GenAI) ──► eval + flywheel ──┐
                ▲                                                       │
                └────────────────── gold updates ◄──────────────────────┘

Cross-cutting planes: lineage (OpenLineage), data contracts (ODCS), catalog and governance (Unity Catalog / Polaris / DataHub), security (ACL propagation, injection scanning), and evaluation (CI-gated).

3. Phase 1: Requirements and the Evaluation-First Contract

Work backward from the agent’s job. OpenAI’s agent-building guide and Anthropic’s “Building Effective Agents” both stress starting from concrete use cases and adding complexity only when measured need justifies it [O1][A1].

3.1 Task inventory

For each agent capability, specify:

• Intent: “Diagnose a failed payment for a customer.”
• Required knowledge: policy docs, payment state machine, error code catalog, customer records.
• Required actions: read ledger, open ticket, issue refund (with approval).
• Answer form: a cited diagnosis plus a recommended action.
• Risk tier: read-only, reversible write, or irreversible write.

3.2 Query taxonomy

Classify expected queries. Each class implies a different gold representation and retrieval path:

3.3 Build the golden evaluation set first

Before writing pipeline code, assemble 100–500 representative questions. Each case should include:

• the question, sourced from real tickets, search logs, and SME interviews (not invented by engineers);
• the expected answer;
• expected source spans (document ID + section);
• the query class and difficulty;
• negative cases: unanswerable questions where the correct behavior is “I don’t know,” and permission-denied cases.

This set becomes your retrieval benchmark, regression suite, and acceptance contract (Section 11).

3.4 Data contract for the gold layer

Define the gold layer’s SLA explicitly, ideally as an Open Data Contract Standard (ODCS) document [S3]:

dataProduct: payments-knowledge-gold
version: 2.3.0
owner: payments-platform@corp
sla:
  freshness: { policy_docs: "1h", error_catalog: "15m", tickets: "24h" }
  retrieval_recall_at_10: ">= 0.90"      # on golden set
  citation_faithfulness:  ">= 0.95"
  unresolved_conflict_rate: "<= 1%"
schema: ./schemas/gold_chunk.json
access: inherit_source_acl

4. Phase 2: Source Inventory and Ingestion (Bronze)

4.1 Source inventory

For each source, record:

• system of record vs. derivative (e.g., a Confluence copy of a policy vs. the policy repo);
• authority rank (used later for conflict resolution);
• change mechanism (CDC, webhooks, polling, full dump);
• permission model (how ACLs are expressed and how to extract them);
• volatility (expected update frequency);
• content types (prose, tables, scans, code, threads, images);
• legal constraints (retention, residency, PII class).

4.2 Ingestion patterns

4.3 Bronze contract

• Immutable and append-only. Store raw bytes in object storage and metadata in an open table format (Apache Iceberg or Delta Lake) for time travel and replay.
• Capture with every record: source_system, source_id, source_uri, source_version, ingested_at, content_sha256, acl_snapshot, source_modified_at, and a deleted tombstone.
• Capture ACLs at ingest. Reconstructing permissions later is often impossible.
• Lineage: emit OpenLineage events per job [S4].

Rule: bronze is replayable. Every downstream layer must be rebuildable from bronze plus code. This matters every time you change a parser, chunker, or embedding model, which will be often.

5. Phase 3: Parsing and Normalization

Parsing is the most underestimated stage. Garbage extraction caps every downstream metric, and no reranker recovers a table that was flattened into word salad.

5.1 Document parsing

Output format: structured markdown or a document JSON with a hierarchy (title → section → subsection → block) and stable block IDs. The heading path is critical metadata for chunk contextualization later.

Visual-native alternative: ColPali embeds page images directly with a VLM and late-interaction scoring [R10]. It skips OCR for visually rich documents and performs strongly on the ViDoRe benchmark. Use it as a parallel index for scan-heavy or diagram-heavy corpora, not as a replacement for text extraction where text extraction works.

5.2 Non-document sources

• Conversations (Slack, Teams, email): reconstruct threads; resolve users to canonical people IDs; strip signatures and quoted replies; mark status (decision vs. speculation). Conversations are low-authority by default.
• Tickets: separate problem, diagnosis, and resolution fields. Resolved tickets are raw material for procedures and FAQ distillation.
• Code: parse with ASTs (tree-sitter); chunk by symbol (function/class); attach file path, repo, commit, and call-graph edges.
• Structured data: do not embed rows. Model it in a semantic layer (Section 7.5) and expose it via text-to-SQL or metric tools.

5.3 Normalization

• Unicode NFKC, whitespace and boilerplate removal (nav bars, footers, cookie banners).
• Language detection and, if needed, translation with the original retained.
• Date normalization to ISO 8601 with timezone.
• Unit normalization where domain-relevant.

6. Phase 4: Conformance (Silver)

6.1 Deduplication

Duplicates poison retrieval. Top-k fills with near-copies, diversity collapses, and old versions outrank new ones. Deduplication measurably improves LM training [R14], and the same logic applies to retrieval corpora.

Do not just delete duplicates. Cluster them, select a canonical representative (by authority rank, then recency), and keep the others as provenance links. “Five sources agree” is a useful confidence signal.

6.2 Entity resolution

Agents fail at joins humans do implicitly (“Acme,” “ACME Corp,” and cust_00481 are the same customer).

• Build a canonical entity registry with stable IDs for customers, products, services, people, and locations.
• Use probabilistic record linkage (Fellegi–Sunter model [R16]; Splink is a production-grade open implementation).
• Use LLM-assisted matching for ambiguous pairs, with human review queues for low-confidence merges.
• Maintain ID crosswalks: CRM ID ↔ billing ID ↔ support ID.
• Run entity linking (mention → canonical ID) over all silver text so chunks carry entities: [...] metadata.

6.3 Temporal modeling

Use a bitemporal model for facts:

• valid_time: when the fact is true in the world (policy effective dates);
• transaction_time: when your system learned it.

This answers both “What is the policy?” and “What did we believe the policy was on March 3?” The second question matters in audits and incident reviews.

6.4 Quality scoring and filtering

Assign per-document quality signals:

• extraction confidence (OCR/parse quality);
• boilerplate ratio, length, language;
• staleness (age relative to source volatility);
• authority (source rank);
• LLM-based quality classifiers (NeMo Curator provides classifier-based filtering patterns [N1]).

Quarantine low-quality content rather than silently dropping it.

6.5 PII and sensitivity classification

• Detect PII and secrets (Microsoft Presidio, cloud DLP, secret scanners).
• Tag at the block level. Redact, tokenize, or restrict according to policy.
• Scrub secrets before embedding. Embeddings are partially invertible; research has demonstrated substantial text reconstruction from embeddings [R17]. Treat vector stores as containing the source data.

7. Phase 5: Gold Artifact Design

This is where agent gold diverges completely from warehouse gold.

7.1 Principle: multiple representations of one truth

                      ┌─► contextualized chunks  ─► hybrid search
                      ├─► atomic claims          ─► fact lookup, conflict detection
 silver document ─────├─► entities/relations     ─► graph traversal
                      ├─► summaries (hierarchy)  ─► global questions, routing
                      ├─► procedures/skills      ─► task execution
                      └─► Q→A pairs (distilled)  ─► FAQ fast path, eval seeds

All representations share provenance back to the same silver block IDs. This makes them consistent, jointly updatable, and jointly deletable.

7.2 Canonical document store and chunking

Chunking is a retrieval-unit design decision, not a string-splitting utility.

Contextual retrieval (Anthropic, 2024) [A2]: prepend an LLM-generated, document-aware context sentence to each chunk before embedding and before BM25 indexing. In Anthropic’s experiments, the top-20 retrieval failure rate fell by:

• 35% with contextual embeddings alone,
• 49% with contextual embeddings + contextual BM25,
• 67% when reranking was added.

Prompt caching makes this economically feasible at corpus scale.

7.3 Gold chunk schema (example)

{
  "chunk_id": "pol-refunds-eu@v7#s3.2#c1",
  "doc_id": "pol-refunds-eu",
  "doc_version": 7,
  "text": "Refunds for EU consumers must be issued within 14 days of ...",
  "context_header": "Refund Policy (EU) › 3.2 Consumer withdrawals. Effective 2026-01-01.",
  "generated_context": "Statutory refund deadlines for EU consumer withdrawals.",
  "heading_path": ["Refund Policy (EU)", "Timelines", "Consumer withdrawals"],
  "content_type": "policy",
  "entities": ["region:EU", "process:refund", "regulation:consumer-rights-directive"],
  "authority": 0.95,
  "valid_from": "2026-01-01",
  "valid_to": null,
  "observed_at": "2026-09-30T12:04:11Z",
  "scope": { "region": ["EU"], "product": ["*"], "tenant": "*" },
  "acl": { "allow_groups": ["support", "finance"], "deny_groups": [] },
  "source": {
    "system": "confluence",
    "uri": "https://wiki.corp/x/AbC123",
    "version": "412",
    "span": { "block_ids": ["b_118", "b_119"] }
  },
  "supersedes": ["pol-refunds-eu@v6#s3.2#c1"],
  "conflicts": [],
  "quality": { "parse_confidence": 0.99, "staleness_days": 3 },
  "embedding": { "model": "embed-model-x@2026-06", "dim": 1024 }
}

Note the embedding model version field. You will re-embed, and mixing vector spaces silently destroys recall.

7.4 Claim store and conflict resolution

Extract atomic claims (subject–predicate–object + qualifiers) with span-level provenance:

(claim) refund_window(region=EU, customer=consumer) = 14 days
  source: pol-refunds-eu@v7 §3.2 | authority 0.95 | valid_from 2026-01-01
(claim) refund_window(region=EU) = 30 days
  source: slack#support 2025-11-02 | authority 0.2

Conflict resolution policy. Decide this deliberately rather than leaving it to the LLM:

1. Same scope, different values → conflict detected.
2. Resolve by authority rank → validity period → recency → corroboration count.
3. If unresolvable: mark conflicts: [...], surface both with sources, and route to the owner’s review queue.
4. Expose conflict status to the agent so it can hedge or escalate instead of guessing.

The claim store also supports coverage analysis: which entities or topics lack authoritative claims.

7.5 Structured data: semantic layer, not embeddings

For quantitative questions, the gold artifact is a semantic layer: governed metrics, dimensions, join paths, and business definitions. Examples include dbt Semantic Layer/MetricFlow, Cube, LookML, and Snowflake/Databricks metric views.

Why: text-to-SQL over raw schemas remains unreliable on realistic enterprise databases. Benchmarks such as BIRD [R19] and Spider 2.0 [R20] show large gaps between model and human performance on messy, real-world schemas. A semantic layer converts “write correct SQL over 400 tables” into “pick the metric and filters,” which is a much easier problem.

Expose it via tools like query_metric(metric, dimensions, filters, time_range) rather than raw execute_sql, with SQL fallback for power use under read-only, row-level-secured credentials.

7.6 Knowledge graph

Use a graph when queries are multi-hop or relational (ownership, dependencies, org structure, lineage).

• Property graph (Neo4j, Neptune, etc.; ISO GQL standardized in 2024 as ISO/IEC 39075) or RDF with SHACL validation for ontology-heavy domains.
• Populate from structured sources first (CMDB, HRIS, service catalogs), since these are high precision. Then add LLM-extracted relations from text, with confidence scores.
• GraphRAG (Microsoft Research) [R7]: extract an entity graph, detect communities (Leiden), and pre-generate community summaries. It answers global “sensemaking” questions that chunk retrieval cannot. It is expensive to build, so justify it with eval data.
• Ontology discipline: a small, explicit schema of entity and relation types beats an unbounded LLM-invented graph that becomes a hairball.

7.7 Hierarchical summaries

Use RAPTOR-style recursive summarization [R8]: cluster chunks, summarize clusters, and repeat to build a tree. Index all levels. This serves:

• broad questions (retrieve high-level nodes);
• routing (decide which collection to search);
• “table of contents” resources the agent can browse via progressive disclosure.

7.8 Procedures and Skills

Procedural knowledge is the most valuable and most neglected gold artifact. Synthesize it from runbooks, resolved tickets, and SME interviews into a structured form:

procedure: rotate-payment-signing-key
version: 4
owner: payments-sre
preconditions:
  - role: payments-sre
  - no active incident tagged payments-sev1
inputs: [environment]
steps:
  - id: 1
    action: tool:kms.create_key_version
    args: { key: "payments-signing", env: "{environment}" }
    verify: "new version status == ENABLED"
  - id: 2
    action: tool:deploy.rollout
    requires_approval: true
failure_modes:
  - symptom: "signature mismatch errors spike"
    remedy: "roll back via step 2 with previous version"
sources: [runbook-114@v9, INC-2291 postmortem]

Anthropic Agent Skills (Oct 2025) [A5] package procedures as folders (SKILL.md + scripts + resources) loaded via progressive disclosure: only a short description sits in context until the skill is relevant. This is an excellent delivery format for procedural gold.

7.9 Glossary and ontology

Include acronyms, synonyms, internal codenames, deprecated names, and ID formats. Use them for:

• query expansion (lexical recall on jargon);
• entity linking;
• injection into the system prompt or a resource for domain grounding.

7.10 Distilled Q→A pairs

Generate candidate Q→A pairs from gold documents (Promptagator/InPars-style synthetic query generation [R21]), then have SMEs validate the high-traffic ones. Uses:

• FAQ fast path (high-precision exact matches);
• eval set expansion;
• training data for fine-tuning embedders and rerankers on your domain.

8. Phase 6: Indexing and Retrieval

8.1 Hybrid retrieval is the baseline

Dense retrieval misses exact identifiers, error codes, and rare jargon. Lexical retrieval misses paraphrase. BEIR showed BM25 remains a strong zero-shot baseline that many dense models failed to beat out of domain [R11]. Production systems combine both.

Reciprocal Rank Fusion (RRF) [R12]:

RRF is simple, robust, and requires no score calibration across retrievers.

8.2 Retrieval stack

query
  → rewrite / decompose (multi-hop) / expand (glossary, HyDE [R22])
  → route (query class → retriever(s))            ← Section 3.2 taxonomy
  → candidate generation (BM25 ∪ dense ∪ graph ∪ metadata filter), k≈50–200
  → ACL + scope + validity filter  (pre-filter, enforced in the index)
  → fusion (RRF)
  → rerank (cross-encoder or LLM reranker), keep k≈5–20
  → dedup / diversify (MMR)
  → expand (parent section, linked entities)
  → package with citations + metadata

8.3 Components

8.4 Adaptive and agentic retrieval

• Self-RAG [R5]: the model decides when to retrieve and critiques relevance and support.
• Corrective RAG (CRAG) [R6]: evaluates retrieval quality and triggers fallback or re-query when confidence is low.
• Agentic search: the agent issues multiple targeted queries via tools, inspects results, and refines. Anthropic notes the shift toward “just-in-time” retrieval, where agents hold lightweight references (paths, IDs, queries) and load data on demand [A3].

Implication for gold design: expose retrieval as composable tools (search, get-document, get-entity, list-related, query-metric) rather than one opaque “ask the knowledge base” endpoint. Agents increasingly drive retrieval themselves.

8.5 Long context does not replace retrieval

• Liu et al., “Lost in the Middle” [R3]: accuracy is highest when relevant information is at the start or end of the context and degrades for information in the middle.
• NVIDIA’s RULER [R4]: many models fail to sustain performance up to their claimed context lengths on tasks harder than needle retrieval.
• Chroma’s “Context Rot” study (2025) [R27]: performance degrades as input length grows, even on simple tasks.

Long context raises the ceiling on how much relevant material you can include. It does not remove the need for precision. Retrieval quality still dominates, and token cost and latency scale with stuffing.

9. Phase 7: Serving Interfaces: MCP, Tools, Skills, A2A

9.1 Where MCP fits

Model Context Protocol (MCP) is an open protocol, introduced by Anthropic in November 2024, that standardizes how AI applications connect to external data and tools [M1]. It has been adopted by OpenAI (Agents SDK and Responses API, 2025), Google, Microsoft, and most major agent frameworks. In December 2025 it was contributed to the Linux Foundation’s Agentic AI Foundation.

Position in the stack:

   Agent runtime (host)  ── MCP client ──►  MCP server  ──►  gold layer services
                                             (interface)     (retrieval, graph,
                                                              semantic layer,
                                                              procedures)

MCP is the interface contract between agents and your gold layer. It solves the N×M integration problem: write one server, and any MCP-capable agent can use it. It is not a database, retrieval algorithm, or permission system by itself.

9.2 MCP primitives mapped to gold artifacts

Key spec evolution (verify the current revision at modelcontextprotocol.io):

• 2025-03-26: Streamable HTTP transport replaced HTTP+SSE; OAuth 2.1 authorization framework.
• 2025-06-18: structured tool output (outputSchema), elicitation, and MCP servers classified as OAuth Resource Servers with RFC 8707 resource indicators to prevent token misuse.
• Later revisions: continued work on async operations, auth, and registry/discovery. Pin your SDK and track the changelog.

9.3 Designing an MCP server for gold data

Anthropic’s “Writing effective tools for agents” [A4] and OpenAI’s agent guide [O1] converge on the following principles:

1. Design for tasks, not endpoints. One find_customer_context(customer) that returns a consolidated, cited view beats five CRUD wrappers the agent must orchestrate.
2. Return high-signal, token-efficient output. Support response_format: "concise" | "detailed", pagination, and truncation with guidance (“37 more results; refine by date”).
3. Return semantic identifiers and citations. Use human-readable names alongside IDs, plus source_uri and version on every item.
4. Write tool descriptions as if onboarding a new engineer. Include when to use the tool, when not to, parameter semantics, and examples. Tool descriptions are prompts.
5. Return actionable errors. “No results for E4O21; did you mean E4021?” beats 404.
6. Enforce permissions server-side using the end user’s identity. Pass user identity via OAuth and filter in the index. Never rely on the model to respect access rules.
7. Make it evaluable. Every tool should be exercised by the eval suite.

Example (Python, official MCP SDK’s FastMCP interface):

from mcp.server.fastmcp import FastMCP, Context
from pydantic import BaseModel

mcp = FastMCP("payments-knowledge")

class Hit(BaseModel):
    chunk_id: str
    title: str
    snippet: str
    source_uri: str
    doc_version: int
    valid_from: str | None
    authority: float
    conflicts: list[str]

class SearchResult(BaseModel):
    hits: list[Hit]
    total: int
    note: str | None = None

@mcp.tool()
async def search_knowledge(
    query: str,
    region: str | None = None,
    as_of: str | None = None,
    limit: int = 8,
    ctx: Context = None,
) -> SearchResult:
    """Search curated payments knowledge (policies, runbooks, error catalog).

    Use for: policy questions, error meanings, how-to procedures.
    Do NOT use for: live system status (use `payment_api_status`) or metrics
    (use `query_metric`). Results include citations; always cite `source_uri`.
    `as_of` (ISO date) returns knowledge valid at that date.
    """
    user = ctx.request_context.user          # identity from OAuth
    hits, total = await retrieval.hybrid_search(
        query=query, user=user, region=region, as_of=as_of, k=limit
    )                                          # ACL + validity filtered in-index
    note = None if total <= limit else f"{total - limit} more; narrow with region/as_of."
    return SearchResult(hits=hits, total=total, note=note)

@mcp.resource("gold://docs/{doc_id}")
async def get_doc(doc_id: str) -> str:
    """Full canonical document (latest version) as markdown with section anchors."""
    return await store.render_markdown(doc_id)

9.4 Adjacent standards and patterns

9.5 Single agent vs. domain-expert agent

• Single agent + MCP gold server: simplest, and the right default [A1].
• Domain agent behind A2A or as a sub-agent: use when the domain needs its own reasoning, memory, policies, or a different model. Anthropic reports that multi-agent research systems use substantially more tokens than single agents [A6], so the gain must justify the cost.

10. Context Engineering at Inference Time

The gold layer determines what can reach the model. Context engineering determines what does [A3].

10.1 Context budget allocation

┌──────────────────────────────────────────────────────┐
│ System prompt: role, domain rules, glossary excerpt  │  stable → prompt-cacheable
│ Tool definitions (MCP tool list)                     │  stable → cacheable; keep lean
│ Skills: descriptions only (progressive disclosure)   │
├──────────────────────────────────────────────────────┤
│ Task state / plan / memory notes                     │
│ Retrieved evidence (ranked, cited, deduped)          │  most relevant at start/end
│ Conversation (compacted)                             │
└──────────────────────────────────────────────────────┘

10.2 Practices

• Fewer, better tokens. Rerank aggressively; send 5–15 strong chunks, not 50 mediocre ones.
• Position matters. Put the strongest evidence first or last, not buried in the middle [R3].
• Format evidence consistently. Use XML/JSON blocks with source, version, valid_from, authority, and conflicts. Instruct the model to cite by ID and to say “not found” when evidence is absent.
• Prune the tool surface. Large tool lists degrade selection accuracy and consume context. Load tool subsets dynamically by task, or use tool-search patterns.
• Compaction and external memory. Summarize long trajectories; persist structured notes outside the context window [A3]. MemGPT [R28] formalized tiered memory for LLMs.
• Prompt caching. Stable prefixes (system prompt, tools, glossary) should be cache-friendly. Order content from most stable to least stable.

11. Evaluation

Evaluate each layer separately, then end to end. If you only measure final answers, you cannot tell whether retrieval or generation failed.

11.1 Metric stack

11.2 Frameworks and methods

• RAGAS [R31]: reference-free RAG metrics (faithfulness, answer relevance, context precision/recall).
• ARES [R32]: trained LLM judges with prediction-powered inference for confidence intervals using a small human-labeled set.
• ALCE [R29]: citation quality evaluation.
• OpenAI Evals [O3], plus platforms such as Braintrust, LangSmith, Arize Phoenix, and Langfuse.
• LLM-as-judge [R33]: scalable, but with known biases (position, verbosity, self-preference). Calibrate the judge against human labels and report agreement. An uncalibrated judge is a random number generator with good manners.

11.3 Evaluation as CI

• Every change to parsers, chunkers, embedding models, rerankers, prompts, tool descriptions, or the model version runs the retrieval and end-to-end suites.
• Gate merges on non-regression of key metrics, with confidence intervals. Small eval sets produce noisy deltas.
• Version the eval set. Add every production failure as a new case.

11.4 τ-bench and consistency

τ-bench [R30] introduced pass^k: the probability an agent succeeds on all k trials of the same task. Agents that look acceptable on pass@1 often show sharply lower pass^k. For production, consistency matters more than peak performance, so measure it.

12. Operations: Freshness, Versioning, Observability

12.1 Incremental processing

• Drive everything from content hashes: an unchanged hash means skip parse, chunk, and embed.
• Propagate changes through the DAG (source block → chunks → claims → graph edges → summaries).
• Deletes and permission changes are first-class events. They must propagate to every index within the SLA. Measure deletion lag.

12.2 Versioning

12.3 Re-embedding and index migrations

Use blue/green indexes. Build the new index (new model or chunking) in parallel, run the eval suite against both, shift traffic, and keep rollback available. Open table formats (Iceberg/Delta time travel) make this tractable for the underlying data.

12.4 Orchestration and processing

• Orchestration: Dagster (asset-oriented, a good fit for lineage of gold artifacts), Airflow, Prefect.
• Processing: Spark/Databricks, Ray (GPU-heavy parsing/embedding), NVIDIA NeMo Curator (GPU-accelerated dedup/classification at scale [N1]).
• Transformations: dbt for the structured/semantic-layer side.

12.5 Observability

• Trace every agent run with OpenTelemetry GenAI semantic conventions: query, rewrites, retrieved IDs, scores, reranked set, tool calls, tokens, latency, and final answer.
• Dashboards: retrieval empty-rate, low-confidence rate, conflict-surfaced rate, freshness lag per source, cost per task, and user feedback.
• Store retrieved chunk IDs with every answer. This is required for debugging, audit, and the flywheel.

13. Security and Governance

13.1 Threat model

The OWASP Top 10 for LLM Applications (2025) [S1] lists prompt injection as LLM01 and adds vector/embedding weaknesses (LLM08). Greshake et al. [R34] demonstrated indirect prompt injection: malicious instructions embedded in retrieved content hijack the model. Your gold layer is an attack surface. Anything an attacker can write into a source (a ticket, a wiki page, an email) can reach the model.

13.2 Controls

13.3 Governance frameworks

• NIST AI RMF 1.0 [S2] and the Generative AI Profile (NIST AI 600-1) for risk-management structure.
• ISO/IEC 42001 for AI management systems, if certification matters.
• EU AI Act obligations, depending on use case and jurisdiction.
• Catalog and policy: Unity Catalog (open-sourced 2024), Apache Polaris, DataHub, OpenMetadata.

14. The Data Flywheel: Closing the Loop

NVIDIA’s Data Flywheel blueprint [N3] formalizes using production interaction data to continuously improve models and retrieval. The same loop applies to gold data:

production traces ──► mine failures ──► classify root cause ──► fix the right layer
      ▲                                                               │
      └──────────── re-evaluate (CI) ◄── add eval cases ◄─────────────┘

Root-cause taxonomy (fix at the correct layer):

14.1 Serving data vs. training data

Keep these separate:

• Serving gold: facts, procedures, and tools retrieved at inference. It changes daily and must be attributable.
• Training data: trajectories, preference pairs, and hard negatives for fine-tuning embedders, rerankers, or the agent model.

15. Technology Map

This field moves quickly. Treat specific products as examples and select them by running your eval set, not by reading leaderboards or vendor benchmarks.

16. Anti-Patterns

1. “Dump everything into a vector DB.” Bronze served as gold. Noise, duplicates, secrets, and contradictions go straight to the model.
2. Dense-only retrieval. Fails on IDs, error codes, and jargon. Hybrid is the floor.
3. Embedding tables and metrics. RAG over CSV rows for “what was revenue in Q2” produces fabricated arithmetic. Use a semantic layer.
4. Post-filtering ACLs. Retrieving top-k and then dropping unauthorized results loses recall and risks leakage in logs and caches.
5. No validity time. The agent cheerfully quotes the 2023 policy.
6. Letting the LLM resolve conflicts silently. Conflict resolution is a data policy, not a sampling outcome.
7. One giant ask_kb tool. It is opaque, unevaluable, and prevents agentic retrieval.
8. Endpoint-mirroring MCP servers. Fifty CRUD tools that wrap a REST API one-to-one bloat context and confuse tool selection.
9. Eval by vibes. “It looked good in the demo” is not a metric.
10. Mixed embedding versions in one index. This causes silent recall collapse.
11. Fine-tuning facts. Facts belong in retrieval; fine-tuning is for behavior and retrieval components.
12. Treating long context as a substitute for curation. Context rot is measurable [R27].
13. Trusting retrieved text as instructions. This is the core of indirect prompt injection.

17. Maturity Model and Launch Checklist

17.1 Maturity levels

17.2 Launch checklist

Data

• Every gold record has source URI, version, span, valid_from/valid_to, authority, scope, and ACL
• Bronze is immutable and the full pipeline replays from it
• Deletes and permission changes propagate end to end within the SLA (measured)
• Dedup clusters with canonical selection; entity IDs resolved
• Conflict policy implemented; unresolved conflicts surfaced and routed to owners
• PII and secrets handled before embedding

Retrieval

• Hybrid (lexical + dense) + rerank, with ACL/validity pre-filtering
• Query classes routed to suitable representations (SQL, graph, chunks, procedures)
• Single embedding model version per index; blue/green migrations

Serving

• MCP server with task-level tools, clear descriptions, structured outputs, pagination, citations, and actionable errors
• User identity flows via OAuth; authorization enforced server-side
• Irreversible actions require approval

Evaluation and operations

• Golden set with expected source spans, negative cases, and permission cases
• Retrieval, faithfulness, citation, abstention, and task-success metrics in CI
• LLM judge calibrated against human labels
• OTel traces with retrieved IDs stored per answer
• Red-team suite for injection and leakage; leak rate = 0

18. References

Verify current versions of specifications and documentation; this field revises quickly.

Industry: Anthropic

Industry: OpenAI

Industry: NVIDIA

Industry: Other

Standards and governance

Research