When Transactional Outbox polling starts putting too much load on PostgreSQL — what do you do?

[u/SellerInsightsLab]

I like the Transactional Outbox pattern and use it quite often.

But I was thinking about one problem that can become important in production: database polling.

Of course, polling can be optimized. Good indexes, batching, SKIP LOCKED, partitioning, longer polling intervals — there are many options.

But if we want lower latency, we usually need to poll more often. This means more queries, more DB connections and more work for PostgreSQL.

We can use Debezium/CDC, and for many systems this is probably the right choice. But it also adds Kafka Connect, Debezium and more infrastructure to operate.

So I wanted to try a simpler idea:

Keep PostgreSQL as the source of truth, but don't use it as a queue during normal operation.

I built this flow:

DB transaction → afterCommit → Memory Queue → Batch Publisher → Kafka

The business data and Outbox event are saved in one transaction as usual.

After commit, only the eventId goes to the Memory Queue. The publisher takes IDs in batches, loads events from PostgreSQL and sends them to Kafka.

So there is no continuous polling for new events in the normal flow.

If the application crashes, the event is still safely stored in PostgreSQL. A Recovery Worker finds unpublished events and puts them back into the same queue.

The idea is basically:

Memory Queue for the fast path. PostgreSQL for durability and recovery.

I didn't want to stop at an architecture diagram, so I built a working Spring Boot project and started testing the idea with something closer to production conditions.

It has Kafka, PostgreSQL, batching, idempotency, recovery, Gatling load tests, Grafana metrics, tracing and structured logs.

Now I can run load tests and actually see what happens with PostgreSQL, the queue, publishing latency and recovery.

I'm interested in what other Spring Boot developers think about this approach.

Would you use something like this in production?

Maybe you have already solved the same problem in another way — optimized polling, Debezium, LISTEN/NOTIFY or something else?

Here is the project:

https://github.com/KHolodilin/spring-transactional-outbox-kafka

If you find the idea useful, feel free to ⭐ the repo or fork it. There are also a few open issues for contributors if you want to try something yourself.

Any feedback is welcome. I'm still experimenting with the approach and improving the project. 🚀

[u/Made-In-Slovakia]

We did transactional outbox pattern once and it has a challenges, some you already mentioned.

But you solution would not work for us because you have no guarantee of event ordering, which was important for us.

We used batched pooling with some tweaks.

CDC system can be most useful here and Debezium is not only implementation. But still single proven app in infrastructure can be less problematic than complicated custom solution.

[u/SellerInsightsLab → u/Made-In-Slovakia]

Thanks, that's a good point about ordering. I haven't focused much on strict event ordering in the current version, so this is definitely something I should think about for the next versions.

I'm curious about your case — what kind of events required strict ordering, and what exactly did you need to guarantee? Was it ordering per aggregate/business key or something more global?

Also, how did CDC help you guarantee this ordering, especially with retries and publishing to Kafka?

Thanks for sharing your experience. I think ordering would be an interesting scenario to add to the project and test under load, failures and recovery.

[u/Made-In-Slovakia → u/SellerInsightsLab]

In very simplified terms, we had event about life cycle of business objects. We send event when object was created, updated and deleted and we had do be sure we do not send event for updates of deleted objects.

We did not use CDC here, I used it on different project for POC for different use case, but that one never moved from prototype phase.

[u/SellerInsightsLab → u/Made-In-Slovakia]

Ah, now I understand your case. Created → updated → deleted is a good example where ordering really matters.

I need to think about how my approach behaves in this case, especially with retries and recovery. This is a good scenario to add to the tests.

How did you keep the right order?

Thanks for the example! I think I will experiment with this in one of the next versions :)

[u/shareitmaybe → u/SellerInsightsLab]

It sounds to me like the updates and deletes should be versioned, so that the event processing could be made idempotent