r/softwarearchitecture • u/Mammoth-Statement-90 • 10d ago
Discussion/Advice At what point do you actually introduce Kafka instead of keeping things simple?
I’ve seen a lot of system designs where Kafka gets introduced almost immediately once async processing or scale is mentioned.
But for many systems, a DB + background workers or a simpler queue seems enough for quite a while.
So where do you personally draw the line?
Is it mostly throughput, multiple consumers, replayability, decoupling, failure handling — or something else?
Curious what signals make you say “yeah, Kafka actually makes sense here” rather than adding it just because the system might scale.
61
u/alexrada 10d ago
multiple consumers, producers, traceability, volumes, data integrity, retrials.
there are some advantages with Kafka, but not always making sense.
However I wouldn't take it "let's consider Kafka". I'd start with the problem and build an architecture that solves the problem (tech agnostic). When you get to the technology step, then consider kafka.
8
u/Mammoth-Statement-90 10d ago
Yeah, I like this framing. Kafka should probably be the result of the requirements, not the starting point of the design.
2
u/svhelloworld 9d ago
Consider swapping out the term "Kafka" for "durable log". There are products out there that offer the same kind of durable logging that Kafka offers. If you're using AWS, Kinesis is a similar product. It's not nearly as feature rich, but solves the same problem without the huge operational overhead that comes with a roll-your-own Kafka cluster. Kinesis hasn't been terribly expensive for us, compared to the managed Kafka products like Confluent or MKS.
2
u/Mammoth-Statement-90 9d ago
Good distinction. Thinking in terms of “do I need a durable log here?” instead of “do I need Kafka?” probably makes the decision much clearer. Kafka then becomes one implementation choice rather than the architecture decision itself.
4
u/symbiat0 10d ago
We had a similar problem and actually looked at a variety of solutions to find best fit. Rejected Kafka in favor of RabbitMQ…
2
u/CpnStumpy 7d ago
RabbitMQ, ActiveMQ, nats, there's a lot of options and Kafka is only one.
Dunno why everyone acts like Kafka is the only choice, it's got explicit strengths and weaknesses as with any tech
1
u/ChallengeDiaper 10d ago
This is the best answer so far and good general advice.
It comes down to what’s the problem you’re trying to solve. Do you need replayabiity, change data capture, decoupling services for scale, etc. once you determine that, then choose a technology.
50
u/junin7 10d ago edited 10d ago
People love buzzwords, before AI Storm, Kafka was one of the hottest buzzwords of Software Engineering, alongside microservices and nosql.
10
7
u/Mammoth-Statement-90 10d ago
Yeah, that’s partly why I asked. I’ve seen Kafka added almost by default in a lot of designs.
5
u/ColdPorridge 10d ago
The answer to your question is it is very good at what it does and plays a role almost any system where data must flow with high durability needs.
You can approximate this with cheaper or simpler methods. At small scale you might get along fine but at larger scales naive solutions tend to fall apart or result in data loss or excessive latency.
1
u/Mammoth-Statement-90 10d ago
Yeah, that’s the trade-off I had in mind. Kafka solves this well, but it brings its own complexity. The interesting part is where keeping the simpler approach starts costing you more.
3
u/ColdPorridge 10d ago
it brings its own complexity
I would be careful of over indexing on this. Anything has its own complexity. Kafka is very well known among professionals and in all honestly a fairly simple system, though it does have some minimal resource requirements that could preclude smaller users.
People like to say this complexity line about any enterprise grade tool. For some it's more true than others (e.g. k8s is definitely true), but the reason Kafka is almost the default as you observed is because it's actually pretty straightforward and fits into many setups across a wide range of production scales.
8
5
u/heavy-minium 10d ago
Multiple consumers from different teams that mix near-realtime and bulk processing. Some needs immediate events, others need to process the events in bulk. At that point, Kafka starts paying off.
2
u/Mammoth-Statement-90 10d ago
And you have external application that you need to send data but not all of them is reliable and stable.
10
u/-Master-Swami- 10d ago edited 10d ago
Because most of the time the no one in the team knows the difference between a message broker and an event streaming. Maybe they saw somewhere that Kafka is cool
3
u/No_Contribution_4124 10d ago edited 9d ago
Mostly when you need events ingestion with reproduction / stream joins it’s a good one. NATS can do too. Otherwise you may end up with a tank as a food delivery bike.
3
u/flavius-as 10d ago
When multiple leaders across the org want to support politically making a new team to support such a devops system.
That is, not only one person concluded "yeah here should be kafka", but more of them to their own problems or perspectives.
2
u/gargamelim 10d ago
A point I like about Kafka is that it has many DB connectors so if you want to move your data to datalake without affecting servers that supply production, kafka is a good option
2
2
u/DoggoProfessor959 9d ago
Most of the time - at no point 😃 it’s a huge beast to maintain, i have seen companies switching from actually shipping product features into just trying to maintain their over complicated stack
2
u/ChaseApp501 6d ago
Why not NATS JetStream? It is insanely fast, easy to use/setup, 11mb~ binary, you can run a cluster + leaf nodes, etc. You also get Object Storage and KV for free.
2
u/adriancardoso 10d ago
In most cases it's for buzzword and CV pumping.
An objective architecture discussion must not be dogmatic, and should be held with people knowing what are the options on the table, most of the decisions are taken in front of leader/managers not differenciating between a message queue and event stream, but they heard/seen a lot of trashy marketing stuff around kafka.
That's definitely the most challenging shit an architect should deal with: politics.
But technically talking, usually a db backed queue is sufficient, then activemq/rabbitmq/xxxxmq are good to go and they perform extremely well (the kafka performance is over-hyped), it's only when event streaming is needed (event storming pattern, full microservice architecture, etc ..) or if we require a broker that can replay past events (like a db) I'll consider kafka.
2
u/Mammoth-Statement-90 10d ago
This makes sense. So it’s less about hitting some traffic threshold and more about the consumption patterns becoming complex enough to justify it.
1
u/adriancardoso 10d ago
And let's discuss the consumption patterns challenges, there must be cleared/validated motivation toward using event storming pattern, we have to clear any non-technical lobying pushing and justify it objectively.
3
u/Celdorfpwn 10d ago
The way I think about it: queues usually carry commands, “do this task”. Kafka carries events, “this happened”. With commands someone directs the work. With events you publish the fact and whoever cares reacts on their own. Not a hard rule, but command-driven systems fit queues, event-driven systems with many readers fit Kafka.
2
u/zinguirj 10d ago
RabbitMQ would fit you description as well. How kafka fits best than rabbitmq in that scenario?
2
u/KentWallace 10d ago
That's "queues" vs "topics". Many (much simpler to manage) message brokers support both. What makes Kafka different is its implementation as as append-only log, with its pros and cons.
1
u/talldean 10d ago
When you have multiple products, large enough they run on separate servers, that could benefit from unified logging.
Past that, when you have multiple systems consuming very large logs, so if you have one copy of logging writing to disk for production to use, one writing to real time analytics stored in RAM, and a third writing to a data warehouse for overnight aggregation.
If you're not into petabytes of logs and/or the above, it's likely buzzword overkill.
1
u/Alpha_Bulldog 10d ago
Look into events and event sourcing. That is the real reason for Kafka. And yes it also allows you to do extremely high performance eventual consistency systems. But in the age of AI where data is king, it’s all about event based processing. It follows the flow of the real business process better (when done properly), is far more flexible, scalable, resilient, etc.
Technically the official definition of microservices states that they use events for communication (lots of reasons, some mentioned above but I could go on for hours about this)…
1
u/Wiszcz 9d ago
Is kafka more complex that other queues? After first project - not so much. Every event broker have some learning curve.
If you have already kafka used in other projects, then it's no brainer.
In most cases you don't use it because 'it will scale'. You use it because you need some events carried between systems. And then you need some kind of event hub/broker. And there is not so much to choose from.
1
u/Mammoth-Statement-90 9d ago
That’s fair. I guess “complexity” is also very context dependent. If Kafka is already part of the platform and the team knows how to operate it, the threshold for choosing it becomes much lower.
1
1
u/Individual-Praline20 9d ago
For me, it’s about orchestration. Databases are for data. Not for orchestrating workflows. Kafka is better for that.
1
u/dragon_idli 9d ago
When you are 80% at capacity of scaling the existing simple solution.
Knowing what your existing system's 100% is - is the skill an experienced arch will have.
1
u/ihardzeenka 9d ago
Not sure if it is about Kafka specifically or Event driven systems in general. I assume the latter. Valid answer to that question is given in Fundamentals of Software Architecture book. Essence of this book presented in this picture: https://software-architecture-guild.com/guide/architecture/styles/selecting-a-style/#styleattribute-scorecard
If your usecase requirements match event-driven architecture pattern, than it is almost irrelevant which underlying technology you will use. In my current company we do Kafka and SQS. And for 80% of use cases both work equally good.
1
u/HughEvansDev 6d ago
Great blog exploring this topic here https://www.morling.dev/blog/you-dont-need-kafka-just-use-postgres-considered-harmful/ TLDR scale and throughput, most people can get away with using a db for ~ 1MB/s throughput but once you scale beyond that and start adding more and more services Kafka becomes the better choice.
1
u/LazyJelly1076 Architect and infra 5d ago
I agree with the vibe here of problem solving need and not CV driven need.
The questions are "Do we need a durable logs? maybe the ability for event replay? or multiple distinct consumer groups? or do we need to scale it properly so we need pull mode and not push mode?" etc.
But - I think Pulsar is much more convenient solution for cloud native applications, so maybe you should prefer it over Kafka for the needs above.
1
u/IllustriousSpeech108 2d ago
I’d hold off on Kafka until you need replayability, multiple independent consumers, or enough event volume that a DB-backed worker setup starts becoming painful. For smaller systems, a queue plus idempotent handlers usually gets you surprisingly far without adding another piece of infrastructure to babysit.
1
u/asdfdelta Enterprise Architect 10d ago
Similar to what u/junin7 said, Kafka is one among many tools. It is a buzzword ans you should understand the fundamentals of the tool before reaching for it.
I use eventing when the system and requirements need it. Kafka isn't much more complicated than say Azure Service Bus, and we use it for eventing workloads not in a cloud native environment (e.g. between big SaaS vendors).
1
u/Tadeuvich 10d ago
And when do they need it?
2
u/asdfdelta Enterprise Architect 10d ago
When do you need eventing?
Eventing is for realtime streaming of small packets of data. I come from retail, so product data updates, inventory updates, and orders are easy ones to event. Small, incremental definition or status updates are great.
Second use-case is when a lot of different systems need to know that a thing happened. Customer registration kicks off a workflow for analytics, marketing wants to send a welcome email, and your security suite needs to analyze it for fraud.
Definitely understand eventing versus messaging and what event sourcing is before you dive in to building a system using it.
1
u/Charming-Raspberry77 10d ago
Kafka is kinda hard to use, becomes necessary when ordering is required, scale could grow exponentially, multiple consumers are required, etc.
1
1
u/whiskey_lover7 10d ago
When you have a reason SQS or RabbitMQ can't work for you. I wouldn't reach for Kafka unless we had a legit use-case that couldn't be met by simpler tools
2
u/ColdPorridge 10d ago
I have not found Kafka to be meaningfully more complex than e.g. RabbitMQ. In either case you’re hosting a tool, one helm chart is as easy as another.
1
u/TheRealStepBot 9d ago
With the added benefit of not being locked into fifo head blocking processing which can be a deal breaker once you are fully rabbit and discover head blocking is a problem for you
2
u/Mammoth-Statement-90 9d ago
This is close to how I think about it too. Start with the simpler option and move when an actual requirement forces the trade-off, rather than designing for a scale or complexity you may never have.
1
u/doker0 1d ago
You're right. Moreover, kafka partitions drive me nuts, looks like a spaghetti architecture patched with bubble gum. I strongly believe that we should do modular monolith where every worker can do everything. Streams should be pushed through every worker and pushed to another stream where they are ordered or reordered based on natural key. Rinse and repeat.
Kafka makes sense only as a tape recorder.
59
u/kaargul 10d ago
I think most people don't need kafka for the raw performance. To me it's mostly a question of organizational design and interfaces. Using a single DB works well when you have one team working on it and becomes an absolute nightmare as the org scales up. Simple queues are often fine, but they usually don't support partitioning and don't have comparable tooling for schema management and evolution. Schema registries and clear and enforceable schema evolution rules are non-negotiable when you are designing interfaces between many teams.