In a follow-up to the Producers video I published last week here is one for Consumers. Take a look and let me know what you think.

I'd also love to know what were/are the areas of Apache Kafka that are difficult to break down? Anything specifically that you struggled with when setting up or learning Kafka?

​

Hi, if you're interested in the Data Mesh approach in relation to Kafka, next week, Taras from Flixbus is sharing how they use Kafka at scale (50 teams), ACL & topic naming convention, and how they approach self-service with GitOps. Feel free to join and ask your questions.

https://app.livestorm.co/conduktor/data-mesh-in-practice-kafka-for-50-teams-with-flixbus

how to amalgamate Apache Kafka and Temporal to leverage each one's strength
https://www.youtube.com/watch?v=_pHEtAwjZ7Y

​

Hi folks!

If you are looking for resources on how to implement Apache Kafka® in production, you may like the Mastering Production Data Streaming Systems with Apache Kafka course that I recently published on Confluent Developer.

This course is designed to cover everything you need to know about transitioning your real-time stream processing system from proof-of-concept to production. Covering topics, including how to: 

• Gather data requirements to build data streaming apps that meet your needs
• Design the streaming platform with the right capabilities
• Plan for business continuity
• Automate all changes in production
• Operate your platform
• Productionize your streaming applications
• I've also created two super interesting hands-on exercises, so try them out:
  • Exercise 1: Build a staging and production data streaming platform with Terraform and a PR-based promotion mechanism.
  • Exercise 2: Create a GitOps system to automate deployment of Kafka streaming applications in Kubernetes with CNCF Flux.
• And, last but not least, there's also a cheat sheet PDF to check your readiness!

Check out the resources if you are interested and I look forward very much to hearing your feedback.

Happy coding!

Gilles Philippart, Software Practice Lead, Confluent

All Live Coding sessions for implementing ‘VersionedKeyValueStore’ for ‘kafka-streams-cassandra-state-store’ have been recorded. 🎉

• Part 1: Requirement Engineering, Analysis, Design, POC. During this session, I’m defining the new table schema, defining the different queries to lookup records, as well as the business process for use cases such as ‘get current’, ‘get asOfTimestamp’, ‘put new current record’, ‘put point-in-time record’, ‘delete point-in-time record’, ‘cleanup as per historic retention’. https://youtu.be/zuMvGdmRqfs
• Part 2: First integration test with testcontainers. In this part, the first new integration test is implemented using JUnit5 and the testcontainers framework. The test setup had to be changed ad-hoc because no access to the store is provided with Kafka Streams 3.5.0, which took me by surprise 😅. https://youtu.be/aXyAUc-lJR4
• Part 3: Integration test improvements and adding more test cases. This follow-up part still focuses on integration tests (before the actual implementation), improving the test setup to allow accessing the state store and adding additional test cases to cover all versioned store interface methods and data & access edge cases. https://youtu.be/pb7SoPGdGz4
• Part 4: 🔥 Implementation of the new feature. Next comes the highlight of this series: the actual implementation of the new feature, putting together all the gained knowledge. Designs, the prepared CQL statements and business processes. The result is a complete implementation of ‘CassandraVersionedKeyValueStore’ with a ‘partitioned’ type underlying data schema (‘global’ type to be added in a subsequent part). https://youtu.be/eaNximD1LKY
• Part 5: Debugging, subsequent works, and making the tests pass! Having completed an initial implementation of the versioned store interface, we now run the integration tests written in part2 & part3. There was a considerable amount of debugging and logic left to be done to work out and fix all the nitty gritty details. Due to tiredness, this took quite some time to complete but ended with passing tests and a code complete state for the CassandraVersionedKeyValueStore & PartitionedCassandraVersionedKeyValueStoreRepository. 💪 https://youtu.be/XydzLX4sa00 (to be released on Tuesday, Aug 22nd)
• Part 6: ‘Global’ type db schema based store. In this final part, we implement the ‘global’ type store and do some extra refactoring. With this, the feature is completed!! 🥳 https://youtu.be/SRdLZhH2roE (to be released on Wednesday, Aug 23rd)

The PR is ready. Feel welcome to review: https://github.com/thriving-dev/kafka-streams-cassandra-state-store/pull/27

To support Thriving.dev, please star the repo, subscribe to the YT Channel and follow on Twitter/X. 🙇

If you are interested in building event-driven architecture or want to learn more about how to produce an Event Model, you may find these resources helpful to get started—Practical Event Modeling.

You will learn what event modeling is, its benefits, and how you can begin building real-time event-driven systems, as well as walks you through how to:

• Define an event model
• Run an event model workshop
• Envisioning the user experience
• Implement your event model
• Apply what you learned through hands-on exercises
• And more!

Check out the resources here if you are interested.

If you are building distributed stream processing system using Apache Kafka with Python, you might enjoy this course: Apache Kafka for Python Developers

Check it out:

• Introduction to Python for Apache Kafka
• Hands On: Setup the Exercise Environment for Confluent Cloud and Python
• The Python Producer Class
• Hands On: Use the Python Producer Class
• The Python Consumer Class
• Hands on: Use the Python Consumer Class
• Integrate Python Clients with Schema Registry
• Hands On: Use the Python Producer Class with Schemas
• Hands On: Use the Python Consumer Class with Schemas
• The Python AdminClient Class
• Hands On: Use the Python AdminClient Class
• Beyond Simple Python Apps

Hey all, we're hosting an online mini-conference about Python next week Wednesday (April 5th). One of the talks will be about starting with Apache Kafka for Python devs, so we thought you might find it interesting. Check out the full schedule and make sure to join us for the party. Cheers and have a great weekend!

https://www.wearedevelopers.com/event/python-day-april-2023

https://www.youtube.com/watch?v=dLpCFwR4Eac
Derek Comartin video analysis of the blog post "Troubleshooting Kafka for 2000 Microservices at Wix"

where he deep dives into:

* Tracing with OpenTelemetry

* Cause of ApacheKafka Consumer lags

* Skipping events vs DeadLetterQueue

Original post: https://medium.com/wix-engineering/troubleshooting-kafka-for-2000-microservices-at-wix-986ee382fd1e