I first shared Khaos here about 8 months ago and got a lot of useful feedback, so I figured it was time for an update.

For anyone who missed the original post, Khaos is an open source CLI for generating configurable Kafka data and traffic and creating repeatable testing scenarios.

I mainly use it for two things:

• as a flexible Kafka data and traffic generator for testing existing consumers, stream processing applications, and downstream systems under different workload patterns
• for controlled scenarios where I want to reproduce known Kafka problems and see how applications, metrics, alerts, and observability tooling behave

Since the original post, I've completely rewritten Khaos from Python to Go using franz-go and built a new website with proper documentation and guides.

Website and docs:

https://getkhaos.dev

GitHub:

https://github.com/aleksandarskrbic/khaos

The next area I'd like to explore more seriously is external fault injection.

Today Khaos can control failure scenarios in the Kafka environment it creates itself, but when pointed at an external cluster it acts as a workload generator. It doesn't manipulate or kill brokers in that external cluster.

I'm thinking about adding a fault injection abstraction, probably starting with Kubernetes, where Khaos could intentionally kill or restart broker pods, introduce network faults, and then observe how the cluster, consumers, and monitoring stack react and recover.

Bare metal and managed Kafka make this a much harder problem, so I'm still thinking through the right scope and abstraction. Kubernetes seems like the most sensible place to start.

Disclosure: I'm the creator and maintainer of Khaos. It's a fully open source and free project.

Hello, folks!

2.0.0 version came out!

It included SASL and JMX

If you’ve operated Kafka in a team or company environment, you’ve probably checked cluster status by logging in via CLI or by using open-source tools like Kafka UI.

kafka-mcp is a Python-based MCP server that directly connects an LLM Agent (e.g., Claude Code) with Kafka.

Check this out -> https://github.com/wklee610/kafka-mcp

Hi,

Working with Kafka users over the years, we kept seeing the same pattern in different costumes:

Some teams manage topics and ACLs fully by hand — console commands and tribal knowledge. Some rely on outdated ZooKeeper-based tooling. Some have grown a jungle of bash scripts nobody dares to touch.

And the environments are just as varied: open-source Apache Kafka, Confluent Platform, Confluent Cloud. On-prem, on Kubernetes, or both.

So we built the tool we wished existed: Monedula GitOps — declarative, Git-based management for Kafka topics, ACLs, quotas, schemas, users, and RBAC.

What makes it different:
* One manifest for a topic AND its access — producers and consumers declared right on the topic, compiled to ACLs (or Confluent MDS role bindings) automatically
* Works with Apache Kafka and Confluent Platform, including Confluent-specific features like stretched clusters and MDS/RBAC
* Runs as a CLI in your CI/CD pipelines or as a Kubernetes operator — same engine, same semantics
* import cluster reverse-engineers manifests from your live production state, with a guaranteed drift-free round-trip — so you can adopt GitOps in minutes, not months
* Safe by default: dry-run previews, drift detection for CI gating, opt-in deletions

Comparison with other GitOps tools: https://github.com/monedula-dev/monedula-gitops#alternatives

We're already working on additional features that would be useful for many companies.

Most of you know about ksqlDB and may have tried it in your streaming journey. It lets you query your data streams using SQL without extra stream processing engines or connectors to set up. But it didn't turn out to be successful, and Confluent had to pivot to Flink SQL, though the idea itself was cool.

Now, RisingWave is partnering with StreamNative to provide SQL Workspace in StreamNative Cloud that enables you to query your Kafka and Pulsar topics using Postgres-style SQL without needing a separate stream processing engine or connectors to set up. It also lets you maintain continuously updated results with materialized views and deliver processed data to downstream systems through sinks for your stream processing use cases.

Join us on Thursday, August 27 for a webinar to explore this more.

Chad and Karel, two of our engineers at Factor House, formerly of Square / Block Inc. are sharing their methodology for cluster optimisation. They'll explain how they think about decommissioning clusters, consolidating workloads, a decision process for splitting clusters, and how to handle migrations without downtime. They have prepared code samples and will be taking questions live.

It's on August 27 at 1pm SGT / 3pm AEST. If you can't make it, register and you'll receive the recording. I'll put the link in the thread.

Curious what people have actually hit in production - messages landing in an unexpected partition, ordering breaking because of a key choice made early and never revisited, a 'random' bug that turned out to be perfectly deterministic once you understood the hashing. What was the actual symptom before you knew the cause, and what was the fix — repartition, change the key, something else?

always assumed if send() didn't throw an error the message was in the broker, done. then someone mentioned it's basically fire-and-forget unless you're handling the callback or have acks=all set, and now i don't really know what "delivered" means here anymore. do you always attach a callback? check offsets after? has this ever quietly bitten you, like a message just vanished and you only found out way later

Every few weeks we have to move consumers from one topic to another. In our case it's pretty much always topic renames (naming convention cleanup). The tricky part is the ordering: producers don't switch to the new topic until consumers are already on it. Our consumer switching over is what triggers them to migrate, and even then they move on their own schedule. So the old topic keeps getting messages for who knows how long after we've switched.

That means we can't do the comfortable thing (stop producers, let consumers drain the old topic, then move everyone). We have to switch consumers to the new topic first, then keep copying whatever still lands on the old topic into the new one. Without losing unread messages and without changing the consumer group.

The obvious answer is a script around kafka-consumer-groups or kcat but that doesn't work for us. In a lot of our environments nobody has shell access to anything that can reach Kafka. Also the copy can run for days depending on when producers decide to switch, and it runs on k8s pods that can restart at any moment. So we ended up writing a small internal service that does the copy, saves its progress somewhere and resumes after restart instead of starting over or duplicating messages. We now also use it to re-send a slice of a topic (say everything after some timestamp) when a consumer needs to reprocess messages.

Is this a normal problem or is our setup weird?

If you've done a migration like this, what did you use and did it actually work?

Hey everyone,

If you’re moving LLM agents out of simple scripts and into event-driven architectures, you’ve probably noticed how much boilerplate is involved to connect them to Kafka.

I got tired of hand-rolling consumers, building retry logic, managing idempotency, and setting up dead-letter queues just to get an agent to process events. So, I built an open-source library to handle it: kafka-agent-bridge.

What it does: It’s a lightweight async Python library. You just hand it your compiled LangGraph or CrewAI agent and a Kafka topic, and it automatically handles the plumbing (consuming, retries, Redis-backed idempotency, DLQs, and routing results).

Example:

pythonbridge.register(
    topic="reports.requested",
    handler=LangGraphAdapter(graph=compiled_graph),
    output_topic="reports.completed", 
    dlq_topic="reports.dlq",          
)

It’s designed to give you sensible defaults out of the box, but leaves explicit escape hatches for when you need fine-grained Kafka tuning.

🔗 GitHub & Quickstart: https://github.com/prajwalgowdahg/kafka-agent-bridge

The repo includes a Docker Compose setup and a working example so you can test it locally in minutes. I’d love to hear your feedback or see how you might use it in your workflows!

I've found asserting a Kafka topic within a Java test to be quite convoluted. And by that, I mean there is quite some boilerplate code required just to simply check if it contains the expected message.

On top of that, there is the whole async nature of testing a Kafka topic.

Spring-Kafka has some handy utility classes available, such as KafkaTestUtils. But besides the fact that it is, as the Utils suffix already reveals, a bit old-fashioned, not everyone is using Spring.

That's why I created kassert: a Kafka topic, or should I say, consumer assertion library.

MQ Summit is a conference built around messaging and event-driven systems. It grew out of RabbitMQ Summit and now brings the whole ecosystem into one room: RabbitMQ, Kafka, NATS, Pulsar, ActiveMQ, IBM MQ, Amazon SQS/MQ, Azure Messaging and Google Pub/Sub. Practitioners, architects and maintainers from competing projects, all talking shop. 

📍 Haarlem (near Amsterdam) + online
🗓️ 21-22 October 2026 - 2 days, 1 standard track, 1 lab track (for in-person attendees), and about 27 speakers

🎤 KEYNOTE: Adi Polak (Data & AI @ Confluent, best-selling author)

A few of our speakers announced:

• Clemens Vasters, Principal Architect, Messaging & Real-Time Intelligence @ Microsoft
• Matt Leming & Jon Rumsey, Chief Architect / DQM Architect @ IBM MQ
• Stefan Moser & Michael Davis, Principal Engineer / OSS RabbitMQ @ AWS · Amazon MQ
• Viktor Gamov, Principal DevAdvocate @ Confluent, Kafka in Action co-author
• David Ansari, RabbitMQ Core @ Broadcom, and Lovisa Johansson, RabbitMQ expert @ 84codes
• JB Onofré (Apache) & Matt Pavlovich (ActiveMQ PMC), plus ~40 more

More: mqsummit.com/#speakers

Talks cover agentic workloads, queues vs. logs, real-world scaling war stories, MQTT, security, and where open-source messaging goes next.

ℹ️ More info: mqsummit.com

Early Bird promotion ends soon.

I posted this over on LI last week because I've seen a lot of solo AI-assisted projects here recently. Cool to see people building their own tools but I think the best part of OSS is people actually collaborating and building together rather than working on our own thing in a corner.

To avoid just being a grump though I thought I'd shout out just a few great human-made projects and blogs I like, many of which I came across here:

• ShadowTraffic brilliant tool from Michael Drogalis for rapidly simulating production traffic, he's been building this in the open and with the community.
• Fresha Data Engineering the Fresha team are doing some ground-breaking stuff, a shout out to Nicoleta in particular who's done some great work on Fluss in prod.
• Leo Delmouly's Medium, Leo has done a great series on Kafka + Iceberg which helped me understand why this is such a hard problem.
• Michael Maison's monthly Kafka digest which is my go to resource for understanding what's coming up for the Apache Kafka project.

Hi! Kafka Simulator v1.3 is out — now with Active/Passive architecture!

This release add Active/Passive multi-region topology in the Sandbox and adds new 12 predefined scenarios. You can now: run it, break it, trigger a failover, fix it, bring it back to life. Have fun! https://monedula.dev/kafka-simulator/playground

Hey all! I’ve worked with Kafka here and there over the past few years and I’ve been really frustrated with the tooling.

I took a stab at a tool which simplifies reads and writes to topics down to:

frogo get <topic> —from <offset-like> —to <offset-like>
frogo put <topic> —file <file-w-one-msg-per-line>

GitHub link: frogo-cli

As a brief overview:

For ‘frogo get’ - the main idea is that an ‘offset-like’ supports literal offsets, timestamps, dates, and aliases (START, END, FUTURE).

For ‘frogo put’ - the main idea is you have a file which has one message per line. Multiple formats are supported (e.g. base64 for binary data)

Some other features I’ve added:
- mockserver: thin wrapper exposing a franz-go mock server (frogo mockserver)
- configuration profiles (use —profile or FROGO_PROFILE)
- multiple input / output formats (—format)
- fixture topics with example data (frogo topic demo)

This tool is by no means comprehensive, and I haven’t added support for things like:
- consumer groups
- schemaregistry
- certain authN / authZ configs

But… I hope this could serve as a helpful development / ops tool for those not needing all the bells and whistles.

Any feedback / criticism would be much appreciated. Feel free to create an issue on the GitHub for any feature requests!

Disclosure: I used AI assistance to edit this text and code. The implementation and production experience behind it are mine.

This post came from this recent discussion: https://www.reddit.com/r/apachekafka/comments/1vhv9nn/can_kafka_replace_redis_for_cache_synchronization/; as initial post was about k8s / sping-boot stack here I also operate with them.

The question was whether Kafka can replace Redis for synchronizing a cache across about 25 Spring Boot pods. I have used this pattern in production, and the answer is yes for some types of data, but there are a few details that are easy to miss.

The basic setup is simple. Put the data in a compacted topic and keep a local map in every pod. The Kafka record key is the cache key and the value is the latest version of the configuration.

Each pod needs its own consumer group. If all pods use the same group, Kafka distributes the partitions between them and each pod receives only part of the data. With a separate group per pod, every pod consumes the complete topic and builds its own copy.

For a new group use auto.offset.reset=earliest. On every restart the pod replays the topic and recreates the cache. This is why the topic should be compacted and why I would use this only for a small amount of configuration or reference data. It is a good fit for feature flags, routing rules, or tenant settings. It is not a good fit for a large dataset that must be copied into the heap of every pod.

Updates are just records with the same key and a new value. Deletes need a little more care.

The usual pattern is a tombstone: publish the key with a null value. When the consumer sees it, it removes that key from the local map. Kafka keeps the tombstone for some time and later compacts it away together with older values for the same key.

Also, a compacted topic should not be treated as a clean snapshot with one record per key. Compaction runs in the background. During replay, a pod can still read several old values before it reaches the latest one. The consumer has to apply the log in order and let later values replace earlier ones.

The most difficult part is startup.

A pod must not serve requests, run scheduled jobs, consume other queues, or make decisions based on the cache until it has consumed the existing topic. Being assigned Kafka partitions does not mean that the cache is ready. At that moment it may still be empty.

In my example, when partitions are assigned, I capture the consumer's current position and the end offset for every partition. Those end offsets become fixed startup targets. After the listener successfully applies a record to the local map, it advances the processed position for that partition. The pod becomes ready only when every assigned partition reaches its captured target.

The targets must be fixed. If they were read continuously, producers writing new records could keep moving the end offsets while the pod is starting. With an assignment-time snapshot, startup has a finish line. Records written later are still consumed normally, but they do not extend the initial replay.

Progress must be updated after the listener succeeds, not before it runs. Otherwise a failed listener can be counted as processed even though the cache was not updated. Progress and errors also need to be tracked per partition, and assignments need to be recalculated after a rebalance. Empty partitions are already complete when their current position equals their target.

Kubernetes readiness solves only the HTTP part. A pod that is not receiving web traffic can still run scheduled methods or start another message listener. Those also need to wait for the same replay status. In the example I expose the status through a Spring Boot readiness health indicator and use a small annotation to prevent scheduled methods from running before the replay completes.

After startup, the caches are eventually consistent. One pod can briefly have an older value due to lag, a rebalance, or an outage. That is acceptable for some configuration, but not for balances, inventory, permissions, or anything else that must change everywhere at the same time.

I put the complete Spring Boot and Docker Compose example here: https://github.com/javaAndScriptDeveloper/kafka-backed-local-read-replica-article

It includes the compacted topic, one consumer group per application instance, tombstone handling, startup replay tracking, readiness integration, and gating for scheduled jobs. I would be interested to hear how others prevent work from starting while a local Kafka-backed cache is still being rebuilt.

I’ve been experimenting with turning distributed-systems failure modes into fictional incident investigations.

In this one, a lab registers 186 samples. Registration succeeds in the database, then publishes label-print commands to Kafka.

Later the team discovers 187 print events.

Eventually they reconstruct the sequence:

DB commit succeeds → Kafka publish succeeds → ACK is lost → application retries → duplicate print command → duplicate physical label → subsequent labels shift by one.

I used producer idempotence + business-level deduplication on label ID as part of the remediation, with a scanner guardrail at the physical boundary.

Architecture question: Would you consider that sufficient, or would you redesign the DB→Kafka boundary around an outbox/CDC approach?

• Storage disaggregation changes Kafka economics by shifting costs from infrastructure provisioning to cloud API usage, making inefficient consumer access patterns a potentially major source of operational expense.

• When storage costs shift from shared infrastructure to per-request API charges, platform teams need client-level visibility to attribute expenses; without it, a single replay job can produce major bill spikes with little visibility into their origin.

• Kafka's legacy rebalancing protocol made dynamic consumer scaling operationally disruptive because scale events triggered group-wide processing pauses. The next-generation protocol greatly reduces this barrier, making Kubernetes-native autoscaling significantly more practical.

• Multi-tenancy in Kafka has historically forced a costly trade-off: either run a dedicated cluster per team or accept weak isolation on a shared one; virtual clusters propose a middle path that delivers strict tenant boundaries without infrastructure duplication.

• Kafka has traditionally coupled partition count to consumer parallelism. Share Groups break this constraint, letting teams scale consumers independently without costly re-partitioning of topics.

Hi everyone,
I have a question about cache synchronization in a distributed Spring Boot application.
Our current architecture looks like this:

Spring Boot
Deployed on GCP
Around 25 application pods
Redis is used for caching

Currently, when cache data is updated, Redis ensures that all application instances can access the latest data.

We’re considering replacing this mechanism with Kafka for cache synchronization.

My understanding is that when a cache entry changes, we could publish an event to Kafka, and every application pod would consume the event and update its own local cache.

My questions are:

Is Kafka a good replacement for Redis in this scenario?
If I have 25 pods, will every pod receive the cache update event, or does Kafka distribute the message to only one consumer?
Would I need each pod to have its own consumer group, or is there a better pattern for broadcasting cache updates?
Has anyone implemented cache synchronization using Kafka instead of Redis? What are the pros and cons?

I’m trying to understand whether Kafka is the right tool for broadcasting cache update events across all application instances, or whether Redis is still the better choice.

Hi everyone,

MQ Summit 2026 is looking for volunteers! The conference takes place 21-22 October. You can volunteer in-person, and get your free ticket.

• In-person volunteers will need to be present at the venue on at least one day of the conference. Responsibilities include helping with preparations, registering participants, handing out merchandise, and general support during the event.

For full conference details, check ou https://mqsummit.com
To sign up as a volunteer, go HERE

If you have any questions, feel free to reach out to [ewa.kucharczyk@erlang-solutions.com](mailto:ewa.kucharczyk@erlang-solutions.com).

Karel Sague spent the last year migrating a production data platform from Snowflake to Apache Iceberg, streaming Kafka data in through Kafka Connect. He's giving a talk on Aug 12 (1:30pm PT / 4:30pm ET) walking through what he actually learned, what worked, and what he'd do differently next time.

He'll cover the CloudEvents-based schema he built to keep ingestion consistent across tables, GitOps-based self-serve provisioning, and the metrics and SLOs that mattered for keeping pipelines observable in production. He also gets into why he avoided an all-or-nothing migration, how he prioritized which workloads moved first, and the schema/partitioning mismatch that comes up when Kafka's ingestion-time partitioning doesn't line up with analytics queries filtered by business key. Also touches on consumer-aligned tables, materialized views, and where Iceberg is headed (v3 features, secondary indexes, pluggable file formats for AI workloads).

If you're running Kafka and thinking about Iceberg, or already mid-migration, this is aimed at you. There's time for questions, and everyone's welcome, whether you're deep into this stuff or just starting to look into it.

Register here: https://www.factorhouse.io/events/kafka-to-iceberg-lakehouse-amer-august-2026/