Event-Driven Architecture with Apache Kafka

Event-driven architecture (EDA) has become the backbone of modern distributed systems, but it is also one of the most misapplied patterns in the industry. Teams reach for Kafka because "it's what everyone uses" and then build a system that is harder to understand than a monolith, with all the operational overhead of a distributed system and none of the benefits.

This article explains how EDA works, when it is the right choice, and how to apply it correctly using Apache Kafka — covering topics, partitions, consumer groups, event sourcing, and the Outbox pattern for reliable event publishing.

Why Events, Not Requests?

In a request-driven architecture, Service A calls Service B synchronously. The communication is direct, but it creates tight coupling:

Temporal coupling: If B is down, A fails. A must know B's address and availability.
Behavioral coupling: A must know what B does with the request. A change in B's behavior requires coordination with A.
Scaling coupling: B must scale to handle A's request rate at peak.

In an event-driven architecture, A publishes an event ("something happened") to a broker. B — and any other interested service — subscribes and reacts asynchronously. The producer does not know about the consumer. They are decoupled in time, behavior, and scale.

Order Service
producer

→

Kafka Topic

order.placed

→

Billing Service
consumer

Notification Service
consumer

Analytics Service
consumer

One event, multiple independent consumers — no code change needed to add a new subscriber

Kafka Fundamentals

Topics and Partitions

A Topic is a named, durable log of events. Topics are split into Partitions — the unit of parallelism. Events within a partition are ordered. Events across partitions are not.

Partition assignment is controlled by the message key. All events with the same key go to the same partition, guaranteeing order for that key. For an order service, the key is typically the order ID — ensuring all events for a given order are processed in sequence.

# Producer: publishing with a partition key (Java/Spring Boot)
@Service
public class OrderEventPublisher {

    private final KafkaTemplate<String, OrderEvent> kafka;

    public void publishOrderPlaced(Order order) {
        OrderEvent event = OrderEvent.builder()
            .eventType("order.placed")
            .orderId(order.getId())
            .customerId(order.getCustomerId())
            .totalAmount(order.getTotal())
            .occurredAt(Instant.now())
            .build();

        // Key = orderId → all events for this order go to the same partition
        kafka.send("orders", order.getId(), event);
    }
}

Consumer Groups

A Consumer Group is a set of consumers that cooperatively process a topic. Kafka assigns each partition to exactly one consumer in the group — enabling horizontal scaling. If you have 12 partitions and 4 consumers, each consumer processes 3 partitions.

Different services consuming the same topic use different consumer groups. Each group maintains its own independent offset — Billing and Notifications both read every event, independently.

Sizing tip: The number of partitions is the ceiling on consumer group parallelism. You cannot have more active consumers in a group than partitions. Plan partition counts conservatively — they can be increased but never decreased without downtime.

The Outbox Pattern: Reliable Event Publishing

One of the most common and damaging bugs in event-driven systems: the application saves to the database, then publishes the event, and crashes between the two. The database has the data but the event was never published. Downstream services never know the order was placed.

The Outbox Pattern solves this with transactional guarantees:

In the same database transaction that updates application state, write the event to an outbox table.
A separate relay process reads unprocessed rows from the outbox and publishes them to Kafka.
On successful publish, mark the row as processed.

-- Outbox table schema
CREATE TABLE outbox_events (
    id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    aggregate_type VARCHAR(100) NOT NULL,  -- 'Order'
    aggregate_id   VARCHAR(100) NOT NULL,  -- order ID
    event_type     VARCHAR(100) NOT NULL,  -- 'order.placed'
    payload        JSONB        NOT NULL,
    occurred_at    TIMESTAMPTZ  NOT NULL DEFAULT NOW(),
    published_at   TIMESTAMPTZ  -- NULL = not yet published
);

-- Application code: single transaction
BEGIN;
  INSERT INTO orders (...) VALUES (...);
  INSERT INTO outbox_events (aggregate_type, aggregate_id, event_type, payload)
    VALUES ('Order', $orderId, 'order.placed', $eventJson);
COMMIT;

For high-volume systems, use Debezium — a CDC (Change Data Capture) tool that tails the database WAL log and publishes events to Kafka automatically, without polling overhead. The outbox table becomes the source of truth for all events.

Event Sourcing

Event Sourcing takes the event log one step further: instead of storing the current state of an entity, you store the sequence of events that led to that state. Current state is a derived projection computed by replaying the event log.

# Event store: the source of truth
events = [
    {"type": "OrderCreated",   "orderId": "123", "items": [...], "at": "10:00"},
    {"type": "ItemAdded",      "orderId": "123", "item": {...},   "at": "10:05"},
    {"type": "PaymentApplied", "orderId": "123", "amount": 99.9, "at": "10:06"},
    {"type": "OrderShipped",   "orderId": "123", "trackingId": "T456", "at": "14:30"},
]

# Current state is derived by replaying
def rebuild_order(events):
    order = {}
    for event in events:
        match event["type"]:
            case "OrderCreated":   order = {"id": event["orderId"], "status": "created", ...}
            case "PaymentApplied": order["status"] = "paid"
            case "OrderShipped":   order["status"] = "shipped"; order["trackingId"] = event["trackingId"]
    return order

When to Use Event Sourcing

Event Sourcing adds significant complexity. Apply it selectively:

Use it when: you need a full audit trail, temporal queries ("what was the order status at 2pm yesterday?"), or the ability to replay events to build new projections.
Avoid it when: you just need async communication between services. For that, the Outbox pattern is sufficient.

Anti-pattern: Using Event Sourcing for every entity in the system. The additional infrastructure, snapshot strategies, and projection management costs are only justified for aggregates where history truly matters.

Schema Evolution: Kafka Schema Registry

Events are contracts. Once published, consumers depend on the event structure. Changing an event schema without coordination breaks consumers. The Confluent Schema Registry enforces compatibility rules:

BACKWARD: New consumers can read events written by old producers (add optional fields, remove fields).
FORWARD: Old consumers can read events written by new producers.
FULL: Both. The safest but most restrictive.

# Avro schema for order.placed (v1)
{
  "type": "record",
  "name": "OrderPlaced",
  "namespace": "com.example.orders",
  "fields": [
    {"name": "orderId",    "type": "string"},
    {"name": "customerId", "type": "string"},
    {"name": "totalAmount","type": "double"},
    {"name": "occurredAt", "type": "string"}
  ]
}

# v2: adding an optional field (BACKWARD compatible)
{"name": "currency", "type": ["null", "string"], "default": null}

Dead Letter Topics

What happens when a consumer fails to process an event? Retrying indefinitely blocks the partition and prevents processing of newer events. The solution is a Dead Letter Topic (DLT): after N retries, the event is moved to a .DLT topic for inspection and reprocessing.

Spring Kafka's @RetryableTopic handles this automatically, with configurable retry topics and exponential backoff between attempts before sending to the DLT.

Key Takeaways

Use EDA to eliminate temporal coupling between services — not just because it is fashionable.
Choose partition keys that guarantee order for business-critical event sequences (order ID, user ID).
Always use the Outbox Pattern or CDC (Debezium) to ensure atomicity between database writes and event publishing.
Use the Schema Registry with BACKWARD compatibility to evolve events without breaking consumers.
Reserve Event Sourcing for aggregates where history and auditability are genuine requirements.
Always configure Dead Letter Topics — poison messages should never halt a consumer indefinitely.

Why Events, Not Requests?Por Que Eventos e Não Requisições?

Kafka FundamentalsFundamentos do Kafka

Topics and PartitionsTópicos e Partições

Consumer GroupsGrupos de Consumidores

The Outbox Pattern: Reliable Event PublishingO Padrão Outbox: Publicação Confiável de Eventos

Event SourcingEvent Sourcing

When to Use Event SourcingQuando Usar Event Sourcing

Schema Evolution: Kafka Schema RegistryEvolução de Schema: Kafka Schema Registry

Dead Letter TopicsDead Letter Topics

Key TakeawaysPontos Principais

Why Events, Not Requests?

Kafka Fundamentals

Topics and Partitions

Consumer Groups

The Outbox Pattern: Reliable Event Publishing

Event Sourcing

When to Use Event Sourcing

Schema Evolution: Kafka Schema Registry

Dead Letter Topics

Key Takeaways