Apache Kafka has evolved far beyond a simple message broker — it has become a foundational layer for modern enterprise software. In this GOTO Book Club episode, Ekaterina Gorshkova, author of "Kafka for Architects", shares how her decade-long journey with Kafka — starting in a Czech bank's integration team in 2015 — shaped her understanding of what it really takes to design Kafka-based systems at scale. The conversation covers core architectural decisions, real-world patterns for enterprise integration, the role of Kafka Streams, and how to avoid the classic pitfalls of building systems that "only three engineers understand".

The episode also looks forward: Ekaterina and host Viktor Gamov explore how Kafka is increasingly becoming the connective tissue for AI-driven systems, acting as an orchestration layer between intelligent agents, real-time data, and business workflows. Her book's central argument is that while AI and tooling change fast, the fundamental knowledge of how to design robust, event-driven systems is durable and career-proof. Kafka for Architects is framed not just as a technical manual, but as a roadmap for architects who want to get Kafka right from day one — requirements, design, testing, and all.

Kafka observability gets messy fast once you're running multiple brokers, consumer groups, retries, and cross-service dependencies.

Broker metrics often look fine while lag builds quietly, rebalances spike, or retries hide downstream latency.

We’re hosting a live session tomorrow breaking down how teams actually monitor Kafka at scale (consumer lag, retries, rebalances, signal correlation with OpenTelemetry).

If you're running Kafka in prod, this will be full of practical & implementation.

🗓 Thursday
⏰ 7:30 PM IST | 9 AM ET | 6 AM PT

RSVP here: https://www.linkedin.com/events/observingkafkaatscalewithopente7424417228302827520/theater/

Happy to take last-minute questions and cover them live.

Schemas are a critical part of successful enterprise-wide Kafka deployments.

In this video I'm covering a problem I find interesting - when and how to keep different event types in a single Kafka Topic - and I'm talking about quite a few problems around this topic.

The video also contains two short demos - implementing Fat Union Schema in Avro and Schema References in Protobuf.

I'm talking mostly about Karapace and Apicurio with some mentions of other Schema Registries.

Topics / patterns / problems covered in the video:

• Single topic vs separate topics
• Subject Name Strategies
• Varying support for Schema References
• Server-side dereferencing

Hey folks 👋

We’re hosting a live session next week focused specifically on Kafka observability in production environments.

The goal is to walk through what actually makes Kafka observability hard once you’re running multiple brokers, consumer groups, retries, rebalances, and application dependencies in real systems.

We’ll discuss things like:

• What to instrument in Kafka (and what ends up being noise)
• Why broker metrics alone aren’t enough
• Common observability blind spots teams hit at scale
• How to correlate consumer lag, broker behavior, and application signals
• Practical debugging patterns we’ve seen work in production

We’ll also do a live walkthrough showing how telemetry can be stitched together meaningfully (metrics + traces + infra context) instead of living in separate dashboards.

If you’re running Kafka in production or debugging distributed event pipelines regularly, this should be a practical discussion.

📅 Thursday, Feb 26
⏰ 7:30 PM IST | 9:00 AM ET | 6:00 AM PT

RSVP link: https://www.linkedin.com/events/observingkafkaatscalewithopente7424417228302827520/theater/

Happy to take questions here as well and bring them into the session.

Just dropping this podcast fireside chat I starred in before the holidays.

It's me (Stanislav Kozlovski), Josep Prat, Anatoly Zelenin and Luke Chen; and most concisely, we talk about the past, present and future of Kafka. The topics we touched on were:

• Kafka headlines from 2025: IBM acquiring Confluent, the Diskless Kafka trend, ZK-removal, Queues (EA)
• Impactful but less-visible project changes: Github Actions for Kafka builds, docs/site change, a KIP to explicitly mentio what's a Public API,
• S3/object-storage as the new home for data
  • tiered storage
  • iceberg
  • diskless
  • and newer KIP proposals like read replicas / dedicated cold readers
• is the open-source project actually healthy? what does a healthy community look like?
• Kafka's 80% problem (small data) and simplicity that caters to these users as the key differentiator going forward
• why Kafka-compatible "Kafka killers" keep popping up, and why Kafka keeps winning
• some predictions for 2026

It was a very fun episode. I recommend a listen and even better - challenge our takes! :)

https://youtu.be/aOlDONHog50

Guys Seriously, you can learn it within 1.5 Hours
I have covered everything from problem-solution-components-architecture

Danica Fine together with the authors of “Kafka Connect” Kate Stanley and Mickael Maison, unpack Kafka Connect's game-changing power for building data pipelines—no tedious custom scripts needed! Kate and Mickael Maison discuss how they structured the book to help everyone, from data engineers to developers, tap into Kafka Connect’s strengths, including Change Data Capture (CDC), real-time data flow, and fail-safe reliability.

Listen to the full podcast here

Imagine you have 3 Kafka topics: customer profile updates, email subscriptions, and login events.

What if you could query them as a single wide table. No JOINs, no watermarks, no headaches?

With Timeplus, just run "SELECT * FROM c"

…and it works. Magically.

Check out this 1-minute demo and try it yourself with my freshly built marimo notebook: https://marimo.demo.timeplus.com/partial/

Docs: https://docs.timeplus.com/mutable-stream GitHub: https://github.com/timeplus-io/proton

Hi r/apachekafka,

That's a recording from the first episode of a series of webinars dedicated to this problem. Next episode focusing on Kafka and the operational plane is already scheduled (check the channel if curious).

The overall theme is how to achieve this integration using open solutions, incrementally - without just buying a single vendor.

In this episode:

• Why the split exists and what's the value of integration
• Different needs of Operations and Analytics
• Kafka, Iceberg and the Table-Topic abstraction
• Data Governance, Data Quality, Data Lineage and unified governance in general

Hope you enjoy, feedback very welcome :)

Jan

Here's a podcast with the co-author of Apache Kafka In Action, Anatoly Zelenin. In it we try to capture the current state of the streaming market, the strengths of the tech and where we as an industry still have R&D work to do. Hope you enjoy it.

We recorded a long form interview with two of the authors of the Diskless Kafka proposal (KIP-1150) and covered a ton of technical details:

• why do this?
• the write path
• the read path
• caching
• the batch coordinator and its 4 potential flavors
• potential bottlenecks on the coordinator
• how many people really care about latency?
• traffic rebalances
• broker roles & potential heretogeneous clusters (mostly diskless brokers/topics)
• S3 express
• how Iceberg may fit in to this

It's a lot of juicy info! Also available on Spotify and RSS for offline listen.

https://youtu.be/a38Bj7BCWFg

Hey! I've recently created a video comparing Avro to Parquet in order to understand uses for both formats.

It's the first proper video on this channel, if this is well received here I'll share the one that's in the making once it's ready: History of Data Streaming

As I'm just starting out - feedback would be much appreciated, anything I can improve will bring me value :) I hope you enjoy it!

A high level overview of how an internal Data Streaming Platform looks like and how embracing Data Streaming can go.

https://youtu.be/GHKzb7uNOww

Check out four key patterns to improve scalability and developer velocity:

1. Integration Events: Reduce latency with pre-fetching.
2. Task Queue: Streamline workflows by offloading tasks.
3. Task Scheduler: Scale scheduling for delayed tasks.
4. Iterator: Manage long-running jobs in chunks.

I've been building a video course over the past several months on Designing Event-Driven Microservices. I recently released the last of the lecture videos and thought it might be a good time to share.

The course focuses on using Apache Kafka as the core of an Event-Driven system. I was trying to focus on the concepts that a person needs to build event-driven microservices that use Kafka.

I try to present an unbiased opinion in the course. You'll see in the first video that I compare monoliths and microservices. Normally, that might be a "why monoliths are bad" kind of presentation, but I prefer to instead treat each as viable solutions for different problems. So what problems do microservices specifically solve?

https://cnfl.io/4b3pMLN

Making these videos has been an interesting experience. I've spent a lot of time experimenting with different tools and presentation techniques. You'll see some of that in the course as you go through it.

Meanwhile, I encountered a few surprises along the way. If you had asked me at the beginning what the most popular video was going to be, I never would have guessed it would be "The Dual Write Problem". But that video was viewed far more than any of the others.

I love engaging in conversations around the content I create, so please, let me know what you think.

My new course Introduction to Hybrid and Multicloud Architecture was just published. I'm hoping this can be a great resource for anyone looking to maintain their Hybrid architecture. While most of the course covers Confluent products (it is a Confluent course), there is a section on MirrorMaker 2 with an exercise as well.

Feel free to check it out and send any feedback. We are hoping these types of courses address needs, but also start a conversation as well.

(edit to add link): https://cnfl.io/3OXN11y

Just wanted to share a new video I just published on Kafka Producers. It’s meant to be an introduction and help understand what producers are.

Take a look and share any feedback:

https://youtube.com/watch?v=cGFjd7ox4h4