AI Products · Deep dive
Architecture of an AI product in production: from data to guardrails
The six layers that turn a prototype into a dependable AI product: governed ingestion, measurable retrieval, deterministic orchestration, evals in CI, end-to-end guardrails, and control over cost and latency.
AI PRODUCT · PRODUCTION ARCHITECTURE
In · Raw request + sources
Out · Answer you can ship, trace and trust
Data & ingestion (lineage, PII redaction)
Every document enters through structured parsing, sensitivity classification and deterministic PII redaction before it reaches the model, and each chunk keeps its link to the source file, the policy version and the run that produced it: without this layer no answer survives an audit.
Retrieval (hybrid search + reranking)
Retrieval fuses semantic vector search and lexical BM25 search via Reciprocal Rank Fusion, then a cross-encoder reranker reorders the candidates to surface the genuinely relevant passages: this is where answer faithfulness is won or lost.
Orchestration (bounded agent, tool calls)
A deterministic workflow decides where you are in the process while the model chooses what to do within a constrained set of actions, with tool calls over standard protocols, durable execution and human-in-the-loop on high-impact actions: the agent stays predictable even under load.
Evaluation & observability (evals in CI, tracing)
A golden dataset built from real failures, graders calibrated against human reviewers and a CI gate block regressions before release, with traces following open semantic conventions (OpenTelemetry GenAI) to stay interoperable and investigable.
Guardrails & security (input/output, injection)
Guardrails validate input, retrieved context and output end-to-end against prompt injection — the most exploited vulnerability — classifying untrusted intermediate context and evaluating every action against the user's original intent.
Cost & latency (routing, caching, TTFT)
Complexity-based routing, semantic caching and token budgets keep cost and time-to-first-token in check, with streaming for perceived latency, so the bill does not explode and quality does not degrade as traffic grows.
The demo convinces, the product collapses
An AI prototype works because it runs on known inputs, single-tenant, with no adversarial traffic. Production rewrites every rule.
- Confident but wrong answers. The model cites the wrong documents with an authoritative tone: the bottleneck is retrieval, not generation.
- Non-reproducible behaviour. The same input yields different outcomes and nobody knows why, because there are no traces and no regression dataset.
- Cost and latency out of control. Verbose prompts, no caching, no routing by complexity: the bill grows and time-to-first-token degrades under load.
- An open attack surface. Hostile instructions hidden in a document or a tool output hijack the agent: prompt injection is the most exploited vulnerability.
- No traceability. When a user reports an error, you cannot trace back to the source document, the chunk and the run that produced it.
For teams taking an AI from proof-of-concept to a production system with audits, SLAs and cost ceilings.
What makes a prototype a real product
An AI product in production is a distributed system in which the model is only the planner. Six layers hold it up.
Data and ingestion
Documents enter through structured parsing, sensitivity classification and deterministic PII redaction before they reach the model. Every output traces back to the source file, the policy version and the run that produced it.
Retrieval
Recovery combines semantic search (vectors) and lexical search (BM25), fused via Reciprocal Rank Fusion, then a cross-encoder reranker reorders the candidates. This is where answer faithfulness is won or lost.
Orchestration and agents
A deterministic workflow decides where you are in the process; the model decides what to do within a constrained set of actions. Tool calls run over standard protocols, with durable execution and human-in-the-loop on high-impact actions.
Evaluation and observability
A golden dataset built from real failures, graders calibrated against human reviewers, and a CI gate that blocks regressions before release. Traces follow open semantic conventions (OpenTelemetry GenAI) to stay interoperable.
Guardrails and cost
Guardrails validate input, retrieved context and output end-to-end against prompt injection; complexity-based routing, semantic caching and token budgets keep cost and latency in check without degrading quality.
The journey of a single request
What a request crosses when the architecture is complete, from the data layer to the returned answer.
The document is parsed, classified by sensitivity, stripped of PII and indexed with purpose tags; every chunk keeps its link to the source.
Hybrid vector + keyword search, rank fusion, then cross-encoder reranking to surface the passages that are genuinely relevant.
The orchestrator routes the request, calls the tools it needs and pauses the agent at critical decision points; every invocation is logged.
Output validated against policy and injection signals, evaluated on sampled traces, then returned with caching and streaming for perceived latency.
Six layers crossed per request, each measurable and traceable independently.
Where everything changes
Same model, different world. The gap is not in the prompt: it is in the layers around it.
| Dimension | Prototype | Production product |
|---|---|---|
| Data | Hand-uploaded files | Governed ingestion, PII redacted, traceable lineage |
| Retrieval | Vector search only | Hybrid + reranking, faithfulness measured |
| Control | Free-roaming agent | Deterministic workflow, constrained tools, human-in-the-loop |
| Quality | Tested by hand | Evals in CI on a golden dataset, regression gate |
| Security and cost | No guardrails | Defense in depth, routing, caching, token budgets |
The value of the six layers
This is not architecture theory: it is what lets you put your name on an answer in front of a client or an audit.
Answers that survive an audit
Every output traces to the document, the chunk and the policy version: explainable, reproducible, defensible.
Reliability under real traffic
Deterministic orchestration and end-to-end guardrails keep behaviour predictable even with adversarial, multi-tenant inputs.
Quality that does not regress
Evals in CI block degradations before release, not after a user complaint.
Governed cost and latency
Complexity-based routing, semantic caching and token budgets keep the bill and time-to-first-token under control at scale.
In production an agent is not a prompt: it is a distributed system in which the model is the planner.
What clients ask us
Why is the answer confident but wrong?
Almost always the problem is upstream: retrieval surfaced the wrong documents and the model produced a well-formed answer grounded in the wrong source. You fix it with hybrid retrieval, reranking and faithfulness measurement, not with the prompt.
How do we defend against prompt injection?
With defense in depth: classify input and retrieved context before the model, validate output against policy, and evaluate every agent action against the user's original intent, isolating the untrusted intermediate context.
What is needed beyond a model that passes evals?
A prototype that passes evals is one ingredient. You also need golden datasets from real failures, traces, CI gates, canary, rollback and runbooks: these are engineered, not assumed.
Cases
From problem to result — anonymised.
The confident but wrong citation
Problem A document assistant answered with an authoritative tone while citing the wrong contract: it worked in the demo but collapsed trust in front of clients because no one could trace the source.
Method Replaced vector-only search with hybrid retrieval (vectors + BM25) fused via Reciprocal Rank Fusion and cross-encoder reranking, added chunk-to-source lineage and a faithfulness measure on sampled answers.
Result Every output now traces back to the document, chunk and policy version that ground it; irrelevant answers are caught before the user instead of surfacing as a complaint.
The agent hijacked by a document
Problem A tool-using agent executed hostile instructions hidden inside uploaded documents and tool outputs: a prompt injection that opened an attack surface in a multi-tenant environment.
Method Introduced defense in depth — classifying input and retrieved context before the model, validating output against policy, isolating untrusted intermediate context and adding human-in-the-loop on high-impact actions.
Result Agent behaviour stayed predictable even with adversarial inputs; every action is evaluated against the user's original intent and tool invocations are logged and traceable.
Quality that regressed in silence
Problem The same input yielded different outcomes and degradations surfaced only after a user complaint, because there were no traces and no regression dataset to rely on.
Method Built a golden dataset from real failures, calibrated graders against human reviewers, added a CI eval gate with canary and rollback, and instrumented traces conforming to OpenTelemetry GenAI.
Result Regressions are blocked before release instead of after a complaint, and the system's behaviour is now reproducible and investigable run by run.
Go deeper
RAG knowledge base that does not invent→
The retrieval layer in depth: source grounding, traceable citations, hallucination control.
Governed AI adoption→
How to take AI to production with audits, roles and policy — not just a good prototype.
Custom or platform: build-vs-buy→
When to build each layer custom and when to compose via platform or API.