An event-driven order processing system in Go: HTTP ingress → Kafka → consumer-group workers → PostgreSQL, with a separate gRPC read path.
Built to be measured, broken and recovered, not just to run. Every number below is from a recorded experiment, and the raw reports are in the repository.
The console drives the live system, so it answers only while the stack is running on the author's machine — during a demo. The measurements below stand on their own.
Writes are acknowledged as soon as Kafka has them (202 Accepted) and persisted
asynchronously. Reads go through a separate service over gRPC. That asymmetry is the
architecture: the two paths saturate, fail and get tuned independently.
Each configuration is deliberately overloaded and the drain measured, because workers only run flat out while a backlog exists.
| Workers | 3 partitions | 12 partitions |
|---|---|---|
| 1 | 1828 ev/s | 1922 ev/s |
| 2 | 2333 ev/s | — |
| 4 | 3348 ev/s | 1932 ev/s |
| 8 | 3807 ev/s | 2410 ev/s |
Quadrupling the partition count made the system slower, not faster — the opposite of the textbook expectation. Partition count caps how many consumers can participate, but only binds if nothing else saturates first, and here something else did.
The fix the scaling table implied, measured. One worker, draining a 48,001-record backlog, batch size the only variable.
The ceiling was the worker's per-record round trip — not partition count, and not Postgres. The same database absorbed 21× the write rate once the round trips were amortised. It ships defaulted to 1 so every measurement here stays reproducible; the trade batching buys is per-record latency, since a record waits for its whole batch.
4 workers, 3 partitions, 15 seconds at each offered rate.
| Offered | 503s | Lost | Accept p99 | End-to-end p99 | Peak lag |
|---|---|---|---|---|---|
| 1,000 | 0 | 0 | 1 ms | 1 ms | 2 |
| 5,000 | 0 | 0 | 3 ms | 5.4 s | 26,908 |
| 20,000 | 0 | 0 | 15 ms | 71 s | 261,292 |
At 20,000/s — over 5× the sustainable write rate — all 300,001 orders were accepted and persisted. The entire overload became latency, not errors. Overload is not a failure mode here.
Every mode induced deliberately and measured, not asserted.
| Component lost | Result | Correctness | Availability |
|---|---|---|---|
| Worker (graceful stop) | rebalance in 1.35 s | intact | intact |
| Worker (killed) | rebalance in 43.2 s, lag spike | intact | intact |
| Database, outage < retry budget | latency only | intact | intact |
| Database, outage > retry budget | 6 dead letters per 150,001, tagged | intact | intact |
| Broker | 384 refusals per 60,001 | intact | degraded |
Correctness survived every experiment. Nothing was lost, duplicated in the database, or silently dropped. The broker is the only hard dependency, because it is the one component with no queue in front of it.
Going from 3 partitions to 12 cut throughput at every worker count — 3807 ev/s down to 2410 ev/s at 8 workers. Partition count caps how many consumers can participate; it only binds when nothing else saturates first. Here something else already had.
A single worker's 1828 ev/s matched one serial round trip per record almost exactly. Batching those round trips took the same PostgreSQL to 41,082 ev/s, which settled what the bottleneck had actually been.
Offered 5× the sustainable rate, the system dropped nothing. The backlog converted to 71 s of end-to-end latency and 261,292 records of consumer lag, and then drained. The only genuine loss of availability came from removing the broker.
Excluded to keep the system small enough to understand completely: Kubernetes, cloud deployment, multi-broker clusters, database replicas, CI/CD, Kafka transactions, auth, and a schema registry. Each is a project in its own right, and none would have made the measurements above more informative.