AI applications, an interactive architecture walkthrough
Semantic search with embeddings: how the architecture works
How semantic search embeds passages into a vector index, runs keyword search alongside it, fuses both rankings, and reranks the top results.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate.
Interactive diagram
Overview
Semantic search finds content by meaning rather than exact words: documents are split into passages, each passage is turned into an embedding vector, and a query is answered by finding the passages whose vectors are closest to the query's vector. An approximate nearest-neighbour index, such as HNSW, makes that search fast over millions of passages.
Embeddings alone miss exact terms such as product codes, so this design runs keyword (BM25) search on the same passages in parallel and merges the two ranked lists with reciprocal rank fusion. A cross-encoder reranker then rescores a short list of candidates, permissions are applied inside both index queries, and labelled test queries measure relevance before changes ship. This is a simplified learning design, not any company's search system.
Trade-offs and what changes at scale
Approximate indexes trade some recall for speed and memory, and filtered queries can return too few results unless the index is asked for more candidates. Reranking improves the top results but adds tens of milliseconds and compute per query. Changing the embedding model requires re-embedding everything, so model choice is effectively a migration. At larger scale vectors are sharded and compressed with quantization to fit in memory, and indexes are partitioned by tenant or permission group so filters stay cheap.
How it works, step by step
The walkthrough above, as text. Each step lights up the components named under it.
When a document changes in the Content DB, a change event reaches the Ingest Worker, which extracts clean text and records who may see it. Indexing from change events keeps search within seconds of the source, including deletes.
The Chunker splits each document into passages of a few hundred words with a small overlap and stores them in the Passage Store. One vector per passage matches specific answers better than one vector per long document.
The Embedding Service turns each passage into a dense vector in batches and upserts it into the Vector Index. Batching keeps GPU or API cost per passage low.
The same passages are indexed in the Keyword Index for BM25 search. Exact matches on names, codes and rare words still matter, and keyword search handles them reliably.
A user's question reaches the Search API, which checks the Query Cache and otherwise embeds the query with the same model used for the passages. Query and passage vectors are only comparable if they come from the same model.
The Vector Index returns the nearest passages using an HNSW index, an approximate search that trades a little recall for much lower latency. The user's groups from the Permissions Service are applied as a filter in the same query; filtering after a small top-k can leave too few results, so the index is asked for more.
In parallel, the Keyword Index returns its own top passages with the same permission filter. Running both covers questions phrased in different words and searches for exact terms.
Rank Fusion merges the two lists with reciprocal rank fusion, scoring each passage by its rank in each list. Using ranks avoids comparing BM25 scores with cosine similarities, which are on different scales.
The Reranker reads the top 50 or so passages together with the query and rescores each pair, then the best few are returned. A pair model is too slow for the whole corpus but more accurate on a short list.
Queries, clicks and queries with no useful result go to Search Events and on to Search Analytics. No-result queries are the clearest list of content or relevance gaps.
Relevance Evaluation runs a set of labelled queries against the Search API and measures recall and nDCG before any change ships. Embedding models, chunk sizes and fusion weights are compared on the same set.
Switching to a new embedding model means the Re-embedding Job rebuilds every vector into a new index, and traffic moves once the new index is complete. Old and new vectors cannot be mixed in one index.
Sequence diagram
The same walkthrough as a UML sequence diagram, one lane per layer or group (18 components in all), left to right in the order a request reaches them, numbered by step. A loop on a lane is a step that stays inside that group. Solid arrows are synchronous calls, dashed are asynchronous.
Key components
Content Sourceslayer
- Change Events: created, updated, deleted (kafka)
- Content DB: articles, products, tickets (postgresql)
Indexing Pipelinedomain
- Chunker: split into overlapping passages
- Embedding Service: text -> dense vector, batched
- Ingest Worker: fetch text, clean, attach ACLs
- Re-embedding Job: rebuilds vectors on model change
Indexesdomain
- Keyword Index: BM25 over the same passages (elasticsearch)
- Passage Store: passage text and source links (s3)
- Vector Index: HNSW over passage embeddings (pgvector)
Query Pathdomain
- Permissions Service: groups the user belongs to
- Query Cache: popular queries and their vectors (redis)
- Rank Fusion: merges keyword and vector results
- Reranker: scores query-passage pairs
- Search API: auth, ACL filter, orchestration
- Search UI: natural-language queries
Evaluationdomain
- Relevance Evaluation: recall and nDCG on labelled queries
- Search Analytics: query and click history (clickhouse)
- Search Events: queries, clicks, no-result queries (kafka)
Sources
Public writing this explainer draws on. Read these for what the real systems actually do.
Make it yours
Copy this diagram and its walkthrough into your own workspace, then change anything: swap Postgres for DynamoDB, add a cache, rewrite the narration. Free, and it uses no AI credits.
Related walkthroughs
- RAG chatbot over your docsHow a retrieval-augmented chatbot ingests company documents, retrieves permitted passages with hybrid search and reranking, and answers with citations.
- Recommendation systemHow a recommendation system retrieves thousands of candidates from millions of items, ranks them with a model, and filters for freshness and diversity.
- AI agent platform with toolsHow an AI agent platform runs a model-and-tools loop on workers, checkpoints each step, scopes tools to the user, and pauses risky actions for approval.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate. Real systems differ in the details. Browse the library.