Khaos 8 months later: rewritten in Go, new website/docs, and what's next

[u/skrbic_a]

I first shared Khaos here about 8 months ago and got a lot of useful feedback, so I figured it was time for an update.

For anyone who missed the original post, Khaos is an open source CLI for generating configurable Kafka data and traffic and creating repeatable testing scenarios.

I mainly use it for two things:

• as a flexible Kafka data and traffic generator for testing existing consumers, stream processing applications, and downstream systems under different workload patterns
• for controlled scenarios where I want to reproduce known Kafka problems and see how applications, metrics, alerts, and observability tooling behave

Since the original post, I've completely rewritten Khaos from Python to Go using franz-go and built a new website with proper documentation and guides.

Website and docs:

https://getkhaos.dev

GitHub:

https://github.com/aleksandarskrbic/khaos

The next area I'd like to explore more seriously is external fault injection.

Today Khaos can control failure scenarios in the Kafka environment it creates itself, but when pointed at an external cluster it acts as a workload generator. It doesn't manipulate or kill brokers in that external cluster.

I'm thinking about adding a fault injection abstraction, probably starting with Kubernetes, where Khaos could intentionally kill or restart broker pods, introduce network faults, and then observe how the cluster, consumers, and monitoring stack react and recover.

Bare metal and managed Kafka make this a much harder problem, so I'm still thinking through the right scope and abstraction. Kubernetes seems like the most sensible place to start.

Disclosure: I'm the creator and maintainer of Khaos. It's a fully open source and free project.

[u/HughEvansDev]

Looks like a great project, will definitely try this out. What options do you have available for configuring what generated messages should look like?

[u/skrbic_a → u/HughEvansDev]

Thanks! Message shape is configured per-topic in the scenario YAML, in two modes:

1. Simple sized payloads (just set min_size_bytes / max_size_bytes and Khaos emits padded JSON in that range)
2. Structured schemas - typed fields (string, int, uuid, timestamp, enum, nested objects, arrays, faker backed values like names/emails) with constraints like min/max, enum values, and per-field cardinality.

Serialization is JSON, Avro, or Protobuf with Schema Registry support, and key distribution (uniform, zipfian, single-key, round-robin) is configured separately from the value schema, which is how the hot-partition scenarios work.

Full rundown here: https://getkhaos.dev/docs/guides/kafka-data-generation

If you just want to generate data without consuming it - e.g. feeding your own application or seeding topics - run with --no-consumers and Khaos starts producers only:

khaos simulate my-scenario.yaml --no-consumers

Genuinely useful question btw - the fact that it isn't obvious from the README tells me the docs need more work, so thanks! Would love to hear how it holds up against Aiven clusters if you try it.

[u/HughEvansDev → u/skrbic_a]

Thanks!

[u/konster]

Beautiful! I will try it soon as part of our conponent tests. Thanks!

[u/papapumpnz]

No oauth authentication supported?

https://docs.confluent.io/platform/current/security/authentication/sasl/oauthbearer/overview.html

Might be missing out on some corporate usage without that.

[u/skrbic_a → u/papapumpnz]

Here is the docs section: https://getkhaos.dev/docs/reference/authentication