We crashed our event pipeline 3,827 times

427controlled runs

1,485,269orders committed

3,827crashes injected at named points

223real process kills

01 · Publication

Publishing from the request path loses orders

The obvious design commits the order and then publishes the event. When the process dies between those two lines of code, the order exists and nobody hears about it. Flip the order of the steps and the failure flips too: the event goes out, the transaction rolls back, and billing charges for an order that was never created, a phantom event.

We crashed the order service at exactly those points with a fixed probability per request. At a 1 % crash rate the commit-first design lost 1.57 % of its committed orders. Each crash cost about 1.6 events, because a dying process takes every request in flight with it. A broker outage was worse: with five quick retries per request, a 15 second outage lost 41 % of all orders in the run, nearly every order committed while the broker was away.

Lost and phantom events per 100 committed orders

Lost eventsPhantom events

Five runs of 20 s at 100 orders per second per bar. All three outbox variants delivered every one of their 40,580 committed orders, with no lost or phantom event.

The transactional outbox writes the event into the same database transaction as the order and lets a separate relay publish it later. In our runs it delivered every committed order across 1.3 million orders, through crashes, broker partitions and process kills. It also made the order service faster: with the broker out of the request path, the service accepted 50 % more orders per second at saturation.

02 · Duplicates

Duplicates still arrive

The outbox guarantees that an event is sent at least once. It says nothing about at most once. When the relay crashes after the broker has confirmed a batch but before it marks the rows as sent, it publishes the whole batch again after the restart. Our batches held 11.5 rows on average, and each relay crash produced almost exactly that many duplicates.

The consumer side surprised us. It runs eight handlers on one connection, and a crash cuts the connection for all of them. Every handler that had committed but not yet acknowledged gets its message back. So even a crash before the commit creates duplicates, through the neighbours.

Duplicate effects in our database, per 1000 orders

No deduplicationInbox tableIdempotent handler

Crash probability 2 % at the marked point, five runs per bar. "With key" means the consumer passes the order id to the payment provider as an idempotency key.

An inbox table or an idempotent handler removed every duplicate from our own database. The payment provider still charged 31 to 51 orders per thousand twice. Only the idempotency key fixed that.

The reason is simple once you see it. A handler that called the provider and then crashed before its commit leaves no trace in our database, so the redelivered message passes the deduplication check and calls the provider again. Deduplication has to live where the effect happens.

03 · The relay

Choosing a relay is choosing a point on a curve

Something has to move events from the outbox table to the broker. A polling relay asks the database every interval, change data capture (CDC) streams new rows from the write-ahead log, and our adaptive relay polls with a backoff and wakes up on a database notification. Each one trades latency for database work in a predictable way: polling every 100 ms delivered at a median of 69 ms and cost the database 30 queries per second even with no traffic at all, almost exactly what a back-of-the-envelope model gives.

Latency against idle database cost

Median end-to-end latency at the selected load against the statements per second the relay sends to an idle database. Both axes are logarithmic. CDC sends none and sits on the left edge.

CDC is fast and silent when idle. The adaptive relay with notifications was the fastest of all at low and medium load, at 13.6 ms, and as cheap as a once-a-second poll when idle. Under heavy load it turns expensive: every notification triggers a poll, and at 150 orders per second it sent more queries than polling every 10 ms.

04 · Broker outage

Backoff can sleep through the recovery

We cut the relay off from the broker for 5, 15 and 30 seconds while orders kept coming. No design lost an event; the outbox simply grew. What differed was how fast each retry policy noticed that the broker was back, and how hard it worked while waiting.

After the broker returns

Medians of five runs with four relay workers at 100 orders per second. Waiting cost counts broker connection attempts and database polls per second during the outage.

Unsent events during one 30-second outage

One run per policy. The backlog grows by 100 events per second for everyone; exponential backoff leaves it sitting for another 21 seconds after the broker is back.

Exponential backoff with a 30 second cap was asleep in the middle of a long wait when the broker came back and resumed 21 seconds late. Fixed retries every 100 ms resumed instantly but hammered the database with 38 polls per second for the whole outage. The probe-gated design we propose resumed in under a second while polling the database at most once per second. It stops polling the database as soon as the broker looks dead and probes the broker with a cheap connection attempt instead.

05 · Downstream outage

Retries turn an outage into dead letters

When the payment provider failed for 20 seconds, consumers with bounded retries gave up on messages and parked them in the dead-letter queue: 288 messages with fixed retries, 48 with exponential backoff. They also kept calling the failing provider up to 88 times per second. A small model of the retry windows predicted five of these six dead-letter counts to the message.

Dead-lettered messages per run

Medians of five runs. Beside each bar: the calls per second that reached the failing provider during the outage.

The adaptive consumer opens a circuit breaker, stops taking messages and lets a single prober check the provider's health. Messages wait in the broker instead of failing. It dead-lettered nothing and called the broken provider less than twice per second. The price is latency for the messages that arrived during the outage.

06 · Everything at once

Four architectures, one bad minute

Finally we assembled four complete systems and gave each the same minute: 100 orders per second, the order service, relay and consumer killed with SIGKILL every 15 to 20 seconds, a 5-second broker partition and a 5-second payment outage.

Only the adaptive design came through with nothing lost, nothing dead-lettered and nothing charged twice. It paid for that with a slower tail than the dual write, which is fast precisely because it never waits for anything, including correctness.

07 · What we do now

A checklist for event-driven services

Never publish from inside the request. Write the event to an outbox table in the same transaction as the business change.
Deduplicate in every consumer. An inbox table keyed by message id or an idempotent write keyed by the business id; about a fifth of consumer throughput well spent.
Send an idempotency key to every external system. Database deduplication cannot protect a call that left no trace in the database.
Size relay batches with duplicates in mind. One relay crash re-sends one whole batch.
Cap retry delays at about a second and add jitter. A 30-second cap sleeps through recoveries.
Hold messages during a dependency outage. Pause behind a circuit breaker and probe; held messages flow again on their own once the dependency is back.

How we measured

The bench is a .NET 10 system with two PostgreSQL 17 databases, RabbitMQ 4.1 and Toxiproxy for network faults, all in containers with fixed CPU limits. Crashes are injected at six named points between the commit, the publication and the processing, both as in-process crashes and as real process exits. Every run counts lost, duplicate and phantom events against one ledger and takes all latencies from database clocks. We ran 427 runs in about four hours of measurement, five repetitions per configuration in random order, and checked the protocol family with the TLA+ model checker before measuring it.

Want the full technical report or the details of the bench? Write to us.

Glossary

PostgreSQL: A popular open-source database. In our bench it is the system's long-term memory: an order written there survives crashes and restarts.
RabbitMQ: A widely used message broker. Think of it as a post office between services: it accepts messages, keeps them in queues and delivers them to whoever has subscribed.
Transaction: A group of database changes that succeed or fail together. If anything goes wrong halfway, the database undoes the whole group, so nobody ever sees half an order.
Commit: The moment a transaction becomes permanent. Until the commit its changes can still disappear; after it they survive even a power cut.
Event: A short message saying that something happened, such as “order 42 was created”. Other services listen for events and react to them, for example by billing the customer.
Phantom event: An announcement of something that never happened. The event was sent, but the database change it describes was cancelled, so other services act on an order that does not exist.
Message broker: The middleman that carries messages between services, RabbitMQ in our case. While it is unreachable, services can neither send messages through it nor receive them.
Transactional outbox: The service writes its outgoing message into a table of its own database, inside the same transaction as the order, and a separate worker sends it later. The order and its message are saved together or not at all.
Relay: The background worker that picks up unsent messages from the outbox table, hands them to the broker and marks them as sent.
Network partition: A network failure that cuts the system into pieces that cannot reach each other, while every piece keeps running.
Process kill: Stopping a running program abruptly from the outside, like pulling its plug. It gets no chance to finish what it was doing or to clean up.
At least once: A delivery promise: every message arrives, though some may arrive twice. Whoever receives the messages has to cope with repeats.
At most once: The mirror-image promise: a message never arrives twice, but it may not arrive at all.
Consumer: The service on the receiving end. It takes messages from the broker and acts on them; here it records each order in the billing ledger and charges the customer.
Acknowledgement: The consumer's receipt to the broker: “done, you can delete this message”. Until the broker gets it, the broker keeps the message and hands it out again if the consumer disappears.
Inbox table: A table in which the consumer notes the id of every message it has processed, in the same transaction as the work itself. A repeat finds its id already there and is skipped.
Idempotent: Giving the same result however many times it is repeated. Pressing a lift button five times still calls one lift.
Idempotency key: A unique tag sent with a request, such as the order number. The receiving system remembers the tags it has seen and ignores a repeat, so a retried payment is charged only once.
Deduplication: Recognising copies of a message that has already been handled and throwing them away.
Polling: Asking “anything new?” again and again at a fixed interval, like checking the mailbox every few minutes. Short intervals react quickly but cost work even when nothing has arrived.
Change data capture: Following the database's own record of changes as they happen, so new rows are noticed without asking for them. Quick to react, and free while nothing changes.
Write-ahead log: The database's diary. Every change is written there first, before it reaches the tables, which lets the database recover after a crash and lets change data capture see new rows.
Database notification: A built-in PostgreSQL signal (LISTEN/NOTIFY). The database pings everyone listening as soon as new data is committed, so the relay can wake up at once.
Latency: Delay. Here it is the time from the moment an order is committed until the billing service has recorded it.
Median: The middle value: half of the measurements were faster and half slower. A few extreme values cannot drag it around the way they drag an average.
Retry policy: The rule for what to do after a failed attempt: how many times to try again and how long to wait in between.
Exponential backoff: Waiting twice as long after each failed attempt: 1 s, 2 s, 4 s, 8 s and so on. It spares a struggling system, but a long wait can sleep through the moment it recovers.
Probe-gated recovery: While a dependency is down, real work pauses and a single cheap test call, the probe, checks from time to time whether it is back. Work resumes as soon as a probe succeeds.
Bounded retries: Retrying a fixed number of times and then giving up. A message whose attempts run out is moved to the dead-letter queue.
Dead-letter queue: A side queue for messages the system has given up on. They are kept, and each one needs a person to inspect and resend it.
Circuit breaker: Works like the fuse box at home: after repeated failures it trips and blocks further calls to the broken dependency, which gives the dependency room to recover.
SIGKILL: The operating system's “stop now” signal. The process ends on the spot without saving or tidying up, much like a power cut for a single program.
Tail latency: The delay of the slowest requests, for example the slowest 5 %. Averages hide it, and it is exactly what the unluckiest customers wait.
Dual write: Writing the same fact to two separate systems one after the other, here the database and the broker, with no transaction covering both. If the program stops between the two writes, they disagree.
Jitter: A little randomness added to each retry delay, so that thousands of clients do not all retry at the same instant and knock the recovering service over again.
Toxiproxy: A tool that sits on the network between two services and misbehaves on command: it can cut, slow down or break the connection, so failures can be staged on purpose.
TLA+ model checker: A tool that takes a precise description of a protocol, tries every possible order of its steps and reports any sequence that breaks a stated rule, before the real code ever runs.