Scaling event-driven systems with Kafka

title: "Scaling event-driven systems with Kafka" date: "2026-01-01" excerpt: "Tips for designing durable event schemas and preventing topic bloat."

tags: "kafka,event-driven,architecture" source: "original"

Kafka enables high-throughput event distribution, but a scalable platform depends on contracts, ownership, and operational discipline. Without those, topic growth and schema drift create long-term fragility.

Treat Event Schemas as Public APIs

Schema evolution should follow compatibility policies just like REST versioning.

Prefer:

• Additive field changes
• Explicit deprecation windows
• Consumer migration plans before removal

Avoid changing field semantics in place. A field that means one thing in Q1 and another in Q3 is a production incident waiting to happen.

{
  "eventType": "PaymentAuthorized",
  "version": "2",
  "paymentId": "pay_1029",
  "authorizedAt": "2026-01-11T12:43:18Z",
  "currency": "USD",
  "amountMinor": 1499
}

Design Topics Around Domains

A topic-per-team strategy usually creates overlap and weak discoverability. A better approach is domain-oriented topic taxonomy with clear ownership metadata.

Example naming style:

• payments.authorization.v1
• exams.submission.v1
• users.profile-updated.v1

Define retention, compaction, and ACL defaults at the platform level so every new topic starts with safe settings.

Partition with Business Invariants

Partition keys should preserve required ordering while enabling parallelism.

• If order must be preserved per account, key by accountId.
• If order is not strict, key by a stable sharding identifier.

ProducerRecord<String, byte[]> record =
		new ProducerRecord<>("payments.authorization.v1", accountId, payloadBytes);

Changing partitioning after scale is expensive, so make this decision deliberately.

Build Idempotent Consumers

Kafka delivery is typically at least once. Consumers must handle duplicates safely.

Practical patterns:

• Idempotency table keyed by eventId
• Upsert semantics for projection updates
• Transactional outbox for producer consistency
• Explicit dead-letter handling for poison messages

create table processed_events (
	event_id varchar(128) primary key,
	processed_at timestamp not null
);

Scale with Backpressure Awareness

Consumer lag is necessary but insufficient. Combine lag with processing latency and downstream saturation to decide scaling actions.

Track:

1. Consumer lag growth rate
2. Processing time per message
3. Retry queue depth
4. Database pool utilization

Autoscaling by lag alone can hide bottlenecks if the true limit is downstream throughput.

Governance That Teams Will Use

Governance should be lightweight and automated:

• Topic creation templates
• Schema compatibility checks in CI
• Ownership labels and on-call metadata
• Default dashboards generated from topic descriptors

This keeps standards consistent without creating manual approval bottlenecks.

Final Takeaway

The objective is sustainable evolution, not just high message volume. A disciplined Kafka platform lets teams onboard new consumers quickly, evolve business logic safely, and recover from incidents without destabilizing the ecosystem.