Event-Driven Architecture: The Failure Modes Teams Hit in Production

Event-driven architecture is easy to demo and brutal to run. The failure modes teams hit in production are consistent: the same event gets processed twice, events arrive out of order, the database and the event stream stop agreeing, one bad message blocks a queue, and no one can trace a flow across six services. None of these show up in a proof of concept. All of them show up under load.

We get called in when an event-driven system has become impossible to reason about. The pattern is almost always the same. The team adopted events to decouple services, then discovered that decoupling moved the hard problems from call stacks into the gaps between services, where they’re harder to see. Below are the five failure modes we see most, and the specific mitigation for each.

Why does an event-driven system process the same event twice?

Because most brokers guarantee at-least-once delivery, and that means duplicates are a designed-in trade-off, not a defect. A consumer crashes after doing the work but before acknowledging the message, a visibility timeout expires, a network retry fires, or a partition rebalances. The broker redelivers. Your handler runs again.

The dangerous version is partial failure: a handler charges a card, then times out before acking. The broker redelivers, and the customer is charged twice. Confluent’s own delivery-semantics documentation is blunt that at-least-once is the practical default, and that true exactly-once across a broker and your database is not something the broker can hand you for free.

The fix is idempotency, not a better broker

Exactly-once delivery is impossible in a distributed system, a consequence of the Two Generals problem. What production systems actually build is effectively-once processing: at-least-once delivery plus a consumer that produces the same result whether it sees a message once or five times.

• Persist an idempotency key. Store every processed event id (or a natural business key) and check it before acting: seen it, skip; new, process and record.
• Push keys downstream. When you call a payment provider, pass an idempotency key so the retry is a no-op on their side too.
• Prefer state-based updates. Advance state only if a condition holds (‘set status to paid where status is pending’), so a replay can’t overwrite a later truth.

This is the same discipline that keeps a write-heavy system honest under retries, and it pairs closely with the caching and consistency thinking we cover in our notes on application-level caching.

What happens when events arrive out of order?

State gets corrupted, quietly. Kafka guarantees ordering only within a single partition, never across partitions. Spread related events across partitions for throughput and you’ve thrown away the ordering you assumed you had. Aiven’s breakdown of Kafka ordering walks through how even a single producer can see events land in an unexpected sequence.

The classic corruption looks like this: a PaymentCompleted event updates an order correctly, then a delayed OrderInitiated arrives and overwrites it, so a paid customer sees an unpaid order. The mitigation is to partition by a stable key (order id, customer id) so every event for one entity lands on one partition in order, and to make consumers reject stale updates using an event timestamp or version number. This is the distributed-systems tax that teams underestimate when they move off a monolith, a theme we dig into in microservices versus monolith under a real performance test.

Why do your database and your event stream stop agreeing?

The dual-write problem. A service commits to its database, then publishes an event in a second, separate operation. If the process dies between the two, one happened and the other didn’t. Now the database says the order shipped and no downstream service ever heard about it, or the reverse. There’s no transaction spanning both, so they drift.

The transactional outbox pattern

Instead of publishing directly, write the event into an outbox table in the same database transaction as your business change. A separate relay reads the outbox and publishes to the broker. Either both the state change and the intent to publish commit, or neither does. Chris Richardson’s outbox pattern is explicit that this avoids two-phase commit and preserves publish order. The relay can still publish a message twice if it crashes after sending, which is exactly why the idempotent consumers from the first section are non-negotiable. Each layer gets one job and a guarantee it can actually keep. Getting this right depends on transaction boundaries you can defend, which is why we treat schema and write-path design as load-bearing in database design for ten million daily transactions.

Where do poison messages go, and why do dead-letter queues fill silently?

A poison message is one a consumer can never process: malformed payload, a referenced record that no longer exists, a schema it doesn’t understand. Without a dead-letter queue it loops forever, blocking valid messages behind it and burning compute. With one, after a set number of failed receives (the maxReceiveCount threshold), the broker routes it aside. AWS documents this redrive policy for SQS directly.

The trap is that a dead-letter queue is a graveyard nobody visits. Messages land there and go unnoticed until a customer reports missing data. Two rules keep it honest: alert on the age of the oldest message, not just the count, and redrive deliberately after you’ve read the failures, never as a blind bulk replay that re-poisons the main queue.

Why is an event-driven system so hard to debug?

Because the logic that used to live in one call stack now lives in the gaps between services, and eventual consistency makes ‘correct but not yet’ look identical to ‘broken’. A user updates a setting, the read model hasn’t caught up, and support gets a bug report for a system working as designed. You can’t step through it with a debugger; the flow is a sequence of async hops with no single thread to follow.

The mitigations are unglamorous and mandatory. Propagate a correlation id on every event so one business action is traceable end to end. Run distributed tracing so a flow across six consumers renders as one timeline. Design the UI to communicate pending state honestly instead of pretending writes are instant. The same reasoning-at-a-distance problem shows up in micro-frontend decisions that bite eighteen months in, and it’s one of the first things that breaks when a startup scales past two million ARR.

Should you adopt event-driven architecture at all?

Often the honest answer is: not yet, or not everywhere. Events buy you decoupling and independent scaling, and they cost you ordering guarantees, transactional simplicity, and easy debugging. If a single service with in-process calls meets your load, keep it. We frequently recommend a modular monolith as the backend default and reserve events for the seams that genuinely need to decouple, not the whole system on principle.

If you already run events and the system has become hard to reason about, the fix is rarely a rewrite. It’s introducing idempotency, an outbox, correlation ids, and dead-letter monitoring one seam at a time, so you stabilise the parts that hurt without freezing delivery. If you’re in that situation, start a conversation with us. No pitch, just a look at where your events are drifting.

Frequently Asked Questions

Is exactly-once delivery possible in event-driven architecture?

Not as delivery. True exactly-once delivery is impossible in a distributed system because of the Two Generals problem. What teams build instead is effectively-once processing: at-least-once delivery from the broker combined with idempotent consumers that record each event id and refuse to act on a duplicate. The result behaves as exactly-once from the outside, without the impossible guarantee underneath.

What is the dual-write problem and how does the outbox pattern solve it?

The dual-write problem is committing to your database and then publishing an event in two separate operations. If the process dies between them, the two systems disagree. The transactional outbox writes the event into an outbox table in the same database transaction as the business change, then a relay publishes it. Both commit together or neither does, avoiding two-phase commit while preserving order.

How do you handle out-of-order events in Kafka?

Kafka only guarantees order within a single partition. Partition by a stable key so every event for one entity (an order id, a customer id) lands on the same partition in sequence. Then make consumers reject stale updates using an event timestamp or version number, so a delayed event can’t overwrite a newer state. Do not assume global ordering across partitions; you don’t get it.

What should you do with messages in a dead-letter queue?

Treat the dead-letter queue as a workflow, not a bin. Alert on the age of the oldest message rather than queue size, so failures surface immediately. Read the actual payloads and logs to find the root cause, fix the consumer or the data, then redrive deliberately and in scope. Never bulk-replay blindly, or you re-inject the same poison messages that failed the first time.

When should a team avoid event-driven architecture?

When a simpler design meets the load. Events cost you ordering guarantees, transactional simplicity, and debuggability in exchange for decoupling and independent scaling. If in-process calls within one service or a modular monolith handle your traffic, that’s usually the right default. Reserve events for the specific seams that genuinely need to decouple, and add the reliability machinery (idempotency, outbox, tracing) before you rely on them.