Web fundamentals, an interactive architecture walkthrough
Distributed job scheduler: how the architecture works
How a distributed job scheduler turns cron schedules into runs, shares work with row locks, and retries jobs safely when workers fail.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate.
Interactive diagram
Overview
A distributed job scheduler runs work at a given time or on a recurring schedule, on a pool of workers, and keeps running it correctly when machines fail. This design keeps all schedules and runs in a relational database: a single elected scheduler turns cron schedules into run rows, dispatchers claim due rows and push them onto a ready queue, and workers execute them while sending heartbeats.
Correctness comes from a few database rules rather than from perfect timing. A unique key on schedule and fire time stops a schedule from firing twice during a leader handover, row locks with SKIP LOCKED let several dispatchers share work, and an attempt counter with backoff bounds retries. Because a worker can die after doing the work but before reporting it, jobs run at least once and must be idempotent. This is a simplified learning design, not any specific product's architecture.
Trade-offs and what changes at scale
A database as the job store gives transactions and simple queries, but polling it for due work limits throughput, so very high job volumes move due runs into a queue or partition the jobs table by time. A single scheduler leader is simple and avoids double firing but adds a short gap during failover. Missed runs after an outage need an explicit policy, catch up or skip, since either can be wrong for a given job. Fairness between tenants and separate worker pools for long jobs matter more as usage grows than raw scheduling speed.
How it works, step by step
The walkthrough above, as text. Each step lights up the components named under it.
An operator defines a nightly report in the Admin Console with a cron expression, a timezone and a concurrency policy, and the Scheduler API stores it in the Job DB. Keeping schedules in the database means any scheduler instance can pick them up.
The Application Service enqueues one-off jobs, such as "send this email in 10 minutes", with an idempotency key. A retried request with the same key returns the existing job instead of creating a second one.
Only the instance holding the Leader Lease turns schedules into runs, and the Scheduler Standby takes over when the lease is not renewed. Two active schedulers would otherwise fire every schedule twice.
The Scheduler Leader computes each schedule's next fire times and inserts run rows with a unique key on schedule and fire time. If a lease handover overlaps, the second insert fails instead of creating a duplicate run.
For multi-step workflows, the Dependency Resolver marks a task ready only after all its upstream tasks have succeeded. The workflow is a graph stored in the Job DB, so a restart resumes where it stopped.
Each Dispatcher instance selects due runs with a FOR UPDATE SKIP LOCKED query and marks them queued. Rows locked by one dispatcher are skipped by the others, so several dispatchers share the work without blocking each other.
Before pushing a run onto the Ready Queue, the dispatcher checks per-queue and per-tenant concurrency limits. A tenant that schedules thousands of jobs at once then cannot starve everyone else.
General Workers take short jobs and Batch Workers take long ones, each with a timeout. A job can run more than once after a failure, so job code must be idempotent.
Running workers write heartbeats to the Heartbeat Store, and the Reaper requeues any attempt whose heartbeat has stopped. That is how a job survives a worker that crashed or lost its network.
A failed attempt is rescheduled with exponential backoff and an attempt counter in the Job DB. After the last allowed attempt the run moves to the Dead-letter Queue for a person to look at.
Workers stream output to Job Logs in object storage, and the Webhook Notifier calls the job owner when a run finishes. Logs stay out of the database, which only holds state.
Finished runs are copied to Run History for reporting, and Observability tracks the delay between a run's scheduled time and its actual start. Growing delay is the first sign that workers or dispatchers need more capacity.
Sequence diagram
The same walkthrough as a UML sequence diagram, one lane per layer or group (19 components in all), left to right in the order a request reaches them, numbered by step. A loop on a lane is a step that stays inside that group. Solid arrows are synchronous calls, dashed are asynchronous.
Key components
Job Statedomain
- Job DB: schedules, runs, attempts, state (postgresql)
- Run History: durations and outcomes for reporting (clickhouse)
Scheduler API
- Scheduler API: validate jobs, idempotency keys
Clientslayer
- Admin Console: define schedules, inspect runs
- Application Service: enqueues one-off and delayed jobs
Schedulingdomain
- Dependency Resolver: releases DAG tasks when parents finish
- Leader Lease: one active scheduler at a time (zookeeper)
- Scheduler Leader: turns cron schedules into due runs
- Scheduler Standby: takes over if the lease lapses
Dispatchdomain
- Dispatcher: claims due runs, applies concurrency limits
- Ready Queue: runs ready to execute, per queue name (redis)
Executionlayer
- Batch Workers: long jobs, large machines
- Dead-letter Queue: runs that used every retry
- General Workers: short jobs, many in parallel
- Heartbeat Store: last heartbeat per running attempt (redis)
- Job Logs: stdout, stderr and outputs (s3)
- Reaper: requeues attempts whose worker went silent
- Webhook Notifier: calls back on success or failure
Observability
- Observability: schedule delay, queue depth, failures
Sources
Public writing this explainer draws on. Read these for what the real systems actually do.
- Kubernetes docs: CronJob (concurrency policy, missed schedules, idempotent jobs)
- Apache Airflow docs: Scheduler (multiple schedulers, row-level locking)
- riverqueue/river: Postgres-backed job queue (transactional enqueue, retries, unique jobs)
- Redis docs: Streams (pending entries, claiming work from failed consumers)
Make it yours
Copy this diagram and its walkthrough into your own workspace, then change anything: swap Postgres for DynamoDB, add a cache, rewrite the narration. Free, and it uses no AI credits.
Related walkthroughs
- Distributed lock serviceHow a distributed lock service grants locks through a consensus log, frees them with expiring leases, and uses fencing tokens to stop stale holders.
- URL shortenerHow a link shortener creates unique short keys, serves redirects from a cache in a few milliseconds, and counts clicks off the redirect path.
- Distributed rate limiterHow an API rate limiter shares token buckets across gateways, answers 429 with a retry hint, fails safely and sheds load when the system is stressed.
This is a learning explainer based on public engineering writing and general design practice. It is simplified, it is not the actual architecture of any company, and it may be inaccurate. Real systems differ in the details. Browse the library.