AI applications, an interactive architecture walkthrough
RAG chatbot over your docs: how the architecture works
How a retrieval-augmented chatbot ingests company documents, retrieves permitted passages with hybrid search and reranking, and answers with citations.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate.
Interactive diagram
Overview
A retrieval-augmented generation (RAG) chatbot answers questions from documents the language model was never trained on. An ingestion pipeline parses, chunks and embeds documents into a vector index and a keyword index while keeping their access permissions. At question time the system rewrites the query into a standalone question, retrieves only the passages the user may see, reranks them, and asks the model to answer with citations.
Trade-offs and what changes at scale
Smaller chunks retrieve precisely but lose context; larger ones carry context but dilute the match and cost more tokens. Hybrid search recovers exact terms that embeddings miss, at the price of running a second index. A reranker adds latency and cost for better precision, and the semantic cache saves model calls only when similar questions repeat within the same permission scope, so it needs a short lifetime and invalidation when documents change. At scale, vector indexes are memory-hungry and are sharded by tenant or collection, permissions and deletions must be re-synced as sources change, query and chunk vectors must come from the same embedding model version (changing models means re-embedding and re-indexing everything, and invalidates the semantic cache), and answer quality needs continuous evaluation, not a one-off test.
How it works, step by step
The walkthrough above, as text. Each step lights up the components named under it.
The sync worker polls Google Drive, Confluence and the PDF bucket incrementally, so only changed documents are reprocessed.
Each document's source permissions (ACLs) are saved in PostgreSQL at sync time and copied onto every chunk as the groups allowed to read it. An identity mapping connects SSO groups to each source's own principals, and later permission changes are re-synced, so retrieval can enforce them. Re-syncing keeps them fresh: permission changes update the tags in both indexes, and source deletions remove the document's chunks.
Documents are queued, so spikes don't overload parsing. The worker extracts text, splits it into overlapping chunks and records chunk metadata.
Chunks are embedded through the embedding API and written to the vector database, and their text goes to the keyword index, both with document IDs and permission tags. After that they are searchable.
A user asks a question in the web client. The API validates the token with auth, which also supplies the user's groups for permission filtering.
First the rewriter uses recent history to turn follow-ups like 'what about its pricing?' into a standalone question. This small model call comes before the cache, because a follow-up means nothing without its history.
The API checks the semantic cache with the standalone question, keyed together with the user's permission scope, so answers are not shared across permission scopes. Entries expire quickly and are invalidated when documents or permissions change. A hit skips retrieval and the large model call; a miss continues.
Retrieval embeds the standalone question, then runs vector search and keyword search over the chunks the user is allowed to see: the user's groups from SSO are applied as a filter inside both searches, so the top results are already permitted. Hybrid search catches exact terms that embeddings miss.
The top candidates are re-checked against live permissions in the metadata store, since access can change after indexing, and their titles and links are fetched for citations. Restricted content is removed before the LLM sees anything, so it can't leak into answers.
A cross-encoder reranker rescores the permitted candidates and keeps only the best few chunks, which improves precision and cuts token cost.
The prompt assembler numbers the chunks for citations and delimits document text and tells the model to treat it as data. That reduces, but does not remove, prompt-injection risk; the bot has no tools with side effects. If nothing relevant was retrieved, the bot says it doesn't know instead of guessing. The LLM streams its answer over SSE to the client, and the API saves it to history and cache.
Thumbs up/down feedback (sent through the authenticated API) and traces of prompts, retrieval results and ingestion runs flow into evaluation, which shows where retrieval or generation is weak.
Sequence diagram
The same walkthrough as a UML sequence diagram, one lane per layer or group (21 components in all), left to right in the order a request reaches them, numbered by step. A loop on a lane is a step that stays inside that group. Solid arrows are synchronous calls, dashed are asynchronous.
Key components
Ingestion pipelinelayer
- Confluence: Wiki pages source (confluence)
- Connector / Sync Worker: Incremental sync, ACL capture and refresh
- Embedding Model API: Text to vectors
- Google Drive: Document source (googledrive)
- Parse Queue: Documents awaiting parsing
- Parsing & Chunking Worker: Extract text, split into chunks
- PDF Object Storage: Customer bucket the connector reads
Data layerlayer
- Conversation History: Chats and messages (postgresql)
- Keyword Index: BM25 in OpenSearch or Elasticsearch
- Metadata Store: Documents, chunks, ACLs, titles (postgresql)
- Semantic Cache: Answers keyed by standalone question + scope (redis)
- Vector Database: Chunk embeddings (pgvector) (pgvector)
Query flowlayer
- API Backend: Chat orchestration, SSE streaming
- Auth / SSO: User identity and groups
- LLM API: Streaming generation
- Prompt Assembler: Context + citation numbering
- Query Rewriter: Standalone question from history
- Reranker Model: Cross-encoder rescoring
- Retrieval Service: Hybrid search with ACL pre-filter
- Web Chat Client: Chat UI, citations, thumbs
Evaluation & observabilitylayer
- Tracing & Evaluation: Prompts, retrieval quality, feedback
Sources
Public writing this explainer draws on. Read these for what the real systems actually do.
Make it yours
Copy this diagram and its walkthrough into your own workspace, then change anything: swap Postgres for DynamoDB, add a cache, rewrite the narration. Free, and it uses no AI credits.
Related walkthroughs
- Semantic search with embeddingsHow semantic search embeds passages into a vector index, runs keyword search alongside it, fuses both rankings, and reranks the top results.
- Recommendation systemHow a recommendation system retrieves thousands of candidates from millions of items, ranks them with a model, and filters for freshness and diversity.
- AI agent platform with toolsHow an AI agent platform runs a model-and-tools loop on workers, checkpoints each step, scopes tools to the user, and pauses risky actions for approval.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate. Real systems differ in the details. Browse the library.