AI applications, an interactive architecture walkthrough
AI agent platform with tools: how the architecture works
How an AI agent platform runs a model-and-tools loop on workers, checkpoints each step, scopes tools to the user, and pauses risky actions for approval.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate.
Interactive diagram
Overview
An AI agent platform runs a language model in a loop: the model reads the task and context, decides to call a tool, sees the result, and repeats until it can answer. The platform supplies everything around that loop, including a queue of runs, workers that execute steps, a registry of tools each user may call, sandboxes for running code, memory retrieval, and a checkpoint after every step so long runs survive worker failures.
Tools are exposed through a common protocol such as MCP and called with the user's own scoped credentials, so an agent can never do more than its user could. Content returned by tools is treated as untrusted data, and risky actions pause for human approval. Every model and tool call is traced and metered, and step and token budgets end runaway loops. This is a simplified learning design, not any company's production platform.
Trade-offs and what changes at scale
More tools and more autonomy make agents more useful but widen what a prompt injection can do, so permissions, sandboxing and approvals matter more than model quality for safety. Checkpointing every step costs storage and some latency but makes long runs resumable and auditable. Model calls dominate cost and latency, so caching context, choosing smaller models for simple steps and limiting steps are the main levers. At larger scale runs are spread across worker pools per tenant, and sandboxes are pooled and recycled to keep start-up time low.
How it works, step by step
The walkthrough above, as text. Each step lights up the components named under it.
A user asks for a multi-step task in the Chat UI, and the gateway puts a run on the Run Queue instead of handling it inside the request. Agent runs can take minutes, longer than an HTTP request should stay open.
An Agent Runtime worker claims the run and loads its conversation from the Run State Store. Each step of the loop is checkpointed there, so a crashed worker's run resumes on another worker from the last step.
Before the first model call, the runtime asks the Memory Service for relevant notes and documents from Vector Memory. Only the closest matches go into the prompt, which keeps the context small.
The Tool Registry returns the tool schemas this tenant and user may use. The model only ever sees tools it is allowed to call, which is simpler than rejecting calls afterwards.
The runtime sends the messages and tool schemas through the LLM Gateway to the Model Provider, which answers with text or a structured tool call. The gateway handles retries and records token usage per run.
A tool call goes to one of the MCP Tool Servers over the Model Context Protocol, using the user's own scoped token from the Credential Vault. The agent can then do only what that user could do.
When the model writes code, it runs in the Code Sandbox with time and memory limits and no network unless the tool allows it. Results and files go to the Artifact Store, not the worker's disk.
Tool output passes through Guardrails before the model sees it, which flags text that tries to give the agent new instructions. Prompt injection through fetched content is the main risk once an agent can act.
Actions marked risky, such as sending email, deleting data or spending money, pause the run and ask the user through the Approval Service. Because the run is checkpointed, it can wait hours for the answer without holding a worker.
Each step's progress is streamed back through the API Gateway, so the user sees which tool is running. Long runs feel slow mostly when nothing visible happens.
Every model call, tool call and result is written to the Trace Store with the run id. When an agent does something unexpected, the trace shows exactly what it saw and why it acted.
Usage Metering totals tokens and tool time per tenant from the traces, and the runtime stops runs that exceed their step or token budget. Budgets stop a looping agent from running up cost.
Sequence diagram
The same walkthrough as a UML sequence diagram, one lane per layer or group (18 components in all), left to right in the order a request reaches them, numbered by step. A loop on a lane is a step that stays inside that group. Solid arrows are synchronous calls, dashed are asynchronous.
Key components
API Gateway
- API Gateway: auth, tenant quotas, streaming
Clientslayer
- API Client: run agents from other systems
- Chat UI: start tasks, watch progress, approve
Orchestrationdomain
- Agent Runtime: loop: think, call tool, observe
- LLM Gateway: model routing, retries, usage
- Model Provider: hosted language models
- Run Queue: agent runs waiting for a worker (redis)
- Run State Store: messages and checkpoint per step (postgresql)
Memory & Filesdomain
- Artifact Store: files the agent reads and writes (s3)
- Memory Service: recall facts and past results
- Vector Memory: embedded notes and documents (pgvector)
Toolsdomain
- Approval Service: human sign-off on risky actions
- Code Sandbox: isolated runtime, no default network
- Credential Vault: per-user OAuth tokens for tools
- MCP Tool Servers: search, tickets, docs, internal APIs
- Tool Registry: tool schemas, per-tenant allowlists
Safety & Operationsdomain
- Guardrails: checks inputs, tool output, final answer
- Trace Store: every model and tool call (clickhouse)
- Usage Metering: tokens and tool time per tenant
Sources
Public writing this explainer draws on. Read these for what the real systems actually do.
- Model Context Protocol specification (2025-06-18): hosts, clients, servers, tool safety
- langchain-ai/langgraph: stateful agents with durable execution and human-in-the-loop
- google/gvisor: application kernel for sandboxing untrusted containers
- OWASP GenAI LLM Top 10 2026 (prompt injection, excessive agency)
Make it yours
Copy this diagram and its walkthrough into your own workspace, then change anything: swap Postgres for DynamoDB, add a cache, rewrite the narration. Free, and it uses no AI credits.
Related walkthroughs
- LLM gateway and routerHow an LLM gateway gives every team one API, enforces token limits and budgets, routes to the right model, and falls back when a provider fails.
- AI code review botHow an AI code review bot queues pull request webhooks, gathers diff context in a sandbox, verifies model findings, and posts inline comments.
- RAG chatbot over your docsHow a retrieval-augmented chatbot ingests company documents, retrieves permitted passages with hybrid search and reranking, and answers with citations.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate. Real systems differ in the details. Browse the library.