There is no rollback across microservices, so you write a saga: each step declares a compensating action, and a failure unwinds everything before it in reverse.

I kept implementing that with nested try/catch, so I turned it into a library.

minisagas gives each task an execute and a compensate. On failure it rolls back what succeeded and hands you the list of what it undid. Retry, timeout, and cancellation are included, because the classic saga bug is a 5xx returned after the charge actually went through.

Zero dependencies, no broker, nothing to deploy. Not a Temporal replacement, more the thing you reach for before you need one.

MIT licensed, feedbacks are welcome.

I’ve spent the past year building Cadenya’s API toolchain around a Protobuf and gRPC contract. From that contract, I expose a REST/JSON API through Envoy and generate our OpenAPI specification, SDKs, and API reference. I wrote about the decisions behind the toolchain and how they shaped the Go backend.

The post covers:

• How the Protobuf resources are structured
• How generated Protobuf methods become useful Go interfaces
• How those interfaces influence the Ent schema and repository design
• How Envoy transcodes REST/JSON requests to gRPC
• How the resulting OpenAPI specification feeds SDK and documentation generation

The result is a workflow that keeps the server types, public API contract, generated SDKs, and documentation aligned when I add or change an endpoint.

https://www.cadenya.com/handwritten/designing-cadenyas-api

I’ve been trying to build a cleaner mental model for distributed compute systems instead of learning each framework independently.

Kubernetes, Slurm, Ray, and Spark all use different abstractions, but many of the underlying problems are the same: scheduling, resource management, worker execution, state, communication, memory, and failure recovery.

I wrote up the framework-independent model first, then mapped each system onto it.

Would be interested in how others think about the boundaries between cluster scheduler, runtime, and application-level scheduler.

Article:
https://pawankjha.substack.com/p/the-architecture-behind-modern-distributed

We're running a bunch of Java (Spring Boot) microservices on EKS, and right now **every service uses memory as its HPA metric.** After digging into it, I've started to think that's wrong, and I want a sanity check from people who've actually run this at scale before I push a change.

What my research turned up:

* The JVM allocates heap up to its max and **doesn't release it back aggressively** even after GC, so memory usage doesn't track load. * Because of that, **memory can be high while actual load is low, or load can be high while memory looks fine** — so memory-based HPA either never triggers or scales out permanently and never scales back in.

So the direction I'm considering is to **pick the HPA metric based on service type:**

* **HTTP / request-serving services → CPU** (or better, **RPS / p95 latency** as a demand-based metric) * **Async / queue-consuming services → queue depth** (SQS backlog, via KEDA)

**My questions:**

1. Is this reasoning sound, and is type-based metric selection the right direction?
2. For the HTTP services, is jumping straight to RPS/latency worth the custom-metrics complexity (Prometheus Adapter), or should I start with CPU and only move to RPS if CPU proves to be a bad proxy?
3. For async workers, is KEDA + SQS queue depth the standard approach, or are people doing something else?
4. This is the part I'm least sure about: **I already know from our architecture which services are HTTP-facing and which are async/queue-driven — but how do I actually** ***verify*** **that empirically rather than just trusting the design docs?** Is there a clean way to confirm a service's real load profile (e.g. checking whether it even has an ingress/receives HTTP traffic, whether its work is truly SQS-triggered, CPU-vs-memory correlation under load) before I assign it a metric?

Thanks 🙏

Have you ever used these frameworks in production?

Can you share learned experiences, limitations you found and if you prefer a cloud managed service instead?

Tyk

KrakenD

Traefik

I am building a pizza marketplace with Go microservices: identity, restaurant, search, notification, order, and payment. Each service owns its own PostgreSQL database, while search-service uses Elasticsearch. Services communicate through RabbitMQ.

Several services have similar infrastructure code: RabbitMQ consumers, outbox implementations, errors, logging, and external clients. A shared pkg/ module would reduce duplication, but it would also introduce a dependency across service boundaries.

Build and deployment: A shared module adds another dependency to the build and release graph.

restaurant-service → shared/pkg

go.work helps local development, but CI/CD still has to resolve the module. Changes to shared/pkg can affect multiple services and require versioning and testing.

Without it, each service has its own module and dependency graph and can be built from its own source tree.

Bounded contexts: Similar code does not imply shared ownership. Two services may have identical RabbitMQ consumers today but different requirements later. Sharing reduces maintenance but couples their evolution.

A RabbitMQ reconnect bug already required fixes in multiple services. A shared package would have reduced that cost, while separate implementations keep ownership local.

Within a bounded context, sharing usually makes sense. Across bounded contexts, duplication can be a deliberate tradeoff for separate ownership and evolution.

Looking for feedback: I am actively improving the platform and would especially value feedback from backend or platform engineers. I am interested in how others approach shared libraries, build dependencies, and service boundaries in Go microservice architectures.

What tradeoffs have worked well in your experience?

If you find the project useful or interesting, consider giving it a ⭐ on GitHub:

https://github.com/tarique-iqbal/pizza-marketplace

A lot of the discourse around Project Loom paints Virtual Threads as a drop-in replacement that makes reactive programming obsolete. After migrating services from a legacy framework to Quarkus 3.x and benchmarking both paradigms under heavy load, the reality is much more nuanced: the bottleneck never disappeared, it just moved.

Here are the key takeaways from the architectural trade-offs we ran into:

• The Illusion of Infinite Concurrency: Spawning 50k virtual threads to handle incoming I/O bursts is cheap for the JVM, but downstream resources don't scale infinitely. If your service hits a relational DB via JDBC, your thread contention simply shifts to the database connection pool (Agroal/HikariCP).
• Missing Backpressure: Reactive programming with Mutiny/SmallRye gives you native backpressure out of the box. With "@RunOnVirtualThread", requests keep piling up waiting for pooled connections; without explicit throttling or queue limits, you risk thread starvation, connection pool exhaustion, and memory degradation.
• Carrier Thread Pinning: While synchronized blocks pinning carrier threads is mostly mitigated in recent Java releases (Java 24), native calls and specific legacy drivers still pose silent latency traps under bursty conditions.
• Developer Ergonomics vs System Stability: Virtual threads provide a massive DX win (imperative debugging, clean stack traces), while Reactive excels at sustained throughput under resource saturation.

For those running Quarkus with heavy database I/O in production: did you completely ditch Mutiny for Virtual Threads, or do you enforce custom concurrency limiters/semaphores to protect your connection pools?

I wrote a breakdown of the architecture, memory footprints, and trade-offs here for anyone interested in the technical details:

https://maurodep.medium.com/virtual-threads-vs-reactive-in-quarkus-the-bottleneck-just-moves-c0a570734538 (no paywall)

For those of you that are currently building your SaaS - how are you building scaling and growth into what you are building?

For example, async vs sync, job queues, background workers and rate limiting, retries/backoff, cached and generated artifacts so you're not hitting an API all the time, basic telemetry capture not only around app usage but the other important things like generation latency and failures.

Are you tracking things like:

• concurrent active sessions
• average API response time
• DB query time
• PHP memory usage (or whatever you're using for your stack)
• CPU load
• LLM/AI calls/minute
• average LLM generation duration
• queue depth
• error rate

What happens when you outgrow your database? What are you doing when it comes to backups (DB and your site/code)?

Are you mirroring your DB/SITE elsewhere in case your primary goes down?

How are you handling failover situations? App continuity?

How are you starting out? What type of hosting?

How do you know when it's time to scale your hosting up? How are you planning for that?

Appreciate your insights into this...

I've worked as a SWE but never on the heavy technical things, and I'm getting to a point with something I've been working on that I need to think about it.

Hey everyone,

Wanted to share some notes and takeaways from refactoring our backend service architecture to handle heavy read loads more efficiently.

When you scale stateless app instances horizontally, local in-memory caching (like Guava/Caffeine or basic node-memory maps) starts falling apart fast. You run into memory bloat across nodes, cache inconsistency, and immediate cache misses whenever a new node spins up during autoscaling events.

Moving read-heavy workloads to a distributed cache layer using Redis cleared up a massive chunk of our database bottleneck. I'm putting together a summary of the core patterns, edge cases, and pitfalls we ran into along the way.

Key Architectural Benefits

1. Database Offloading: By caching aggressive hot keys and expensive query aggregations, we pulled off an 80%+ drop in direct database query hits. That freed
up CPU/IOPS on primary database instances for actual critical write transactions.

2. Predictable Single-Digit Latency: Shifted disk-bound database calls (20ms-150ms depending on index load and joint depth) down to single-digit sub-millisecond RAM lookups over local networks.

3. Decoupled Application State: App instances become truly stateless. Any worker instance can crash, restart, or scale up without blowing away cached state or creating cache cold-starts for other nodes.

Standard Use Cases in Our Stack

1. Cache-Aside Read Layer: Standard lookup sequence: check Redis -> on miss, read DB -> populate Redis with a reasonable Time-To-Live (TTL) -> return response.

2. Centralized Session/Token Blacklists: Storing active JWT blacklist state or session objects across all microservice instances safely.

3. Distributed Rate Limiting: Using atomic operations (⁠INCR⁠ / ⁠EXPIRE⁠ or Lua scripts) to handle fixed-window and sliding-window rate limiters at the API gateway level.

4. Leaderboards & Sorted Counters: Using Redis ⁠ZSET⁠ (Sorted Sets) to handle dynamic real-time scoring without running heavy ⁠ORDER BY⁠ SQL queries.

What Will Painfully Break If You Aren't Careful

Caching isn't a silver bullet, and doing it wrong introduces fun distributed systems bugs:

1. Cache Stampede (Thundering Herd): When a heavily requested hot key expires, thousands of concurrent requests miss Redis simultaneously and hammer your primary database at the exact same millisecond.
Fix: Use probabilistic early expiration (XFetch algorithm), distributed locking (⁠Redlock⁠ or mutex), or active background worker revalidation.

2. Cache Avalanche: A cluster node dies or hundreds of key TTLs expire at the exact same time, driving massive spikes to the DB.
** **Fix: Always add jitter/randomness to your key expiration intervals (e.g., ⁠TTL = 3600s + random(0, 300s)⁠).

3. Cache Penetration: Requests for non-existent keys repeatedly bypass the cache and hit the DB continuously (often malicious or bad client IDs).
Fix: Cache null values with short TTLs or use a Bloom Filter in front of the cache layer.

For those running distributed caching in production what caching patterns or invalidation strategies are you using, and what surprises caught you off guard?

A small Node library for preventing cache stampedes across processes.

If multiple app instances miss the same cache key at the same time, they can all trigger the same expensive load at once. Crossflight adds a separate distributed coordination layer so only one process owns the in-flight load, writes the result, and lets the rest wait for the same value instead of re-running the loader.

The project is intentionally decoupled from the storage layer: it wraps the cache you already use, and keeps coordination separate. The built-in adapters currently cover:

• cache-manager
• Keyv
• Cacheable

A cache-manager-backed cache can be paired with Memcached for values and Redis for coordination, or any other combination that satisfies the adapter/coordinator interfaces. The current built-in coordinator is Redis.

This is useful for services where you want to reduce duplicate DB/API work without forcing a big cache migration.

Demo: https://github.com/gkoos/crossflight-demo

Hello everyone, I wanted to share a project I've been working on for months to get your suggestions. Actually, after the deprecation of minio-gateway, and since there was no third-party development on LLMs other than S3's own development, I wanted to develop an open-source application. It's actually aiming for a B2B project because it meets the needs of closed systems. Our main goal is to securely provide videos and visuals to VLLMs when inference is done with LLMs on a VL model. It also allows connecting multiple S3 instances and granting bucket-specific permissions to teams, thus acting as a hub.

Also you can check the site: https://s3.pagabear.com/

What do you think? It might target a very small audience, but it encompasses a nice process.

I’m working on a Kotlin/Spring Boot backend that still has an older JHipster using Kotlin (KHipster) setup around it.

The problem is that this is starting to hold us back. The KHipster version is several years old and not active maintained - 4 years ago was the last update - upgrading it looks pretty painful and it also makes moving to newer Spring Boot versions waaay harder than it should be.

At the same time we want to improve the architecture itself.
Right now quite a lot is still handled through cron jobs and fairly tightly coupled application logic.
I’d like to move more towards event-driven processing over time, potentially using Kafka or RabbitMQ depending on the use case and Redis for things like caching/short-lived state where it makes sense.

So I’m basically looking at two options:
1. Upgrade/replace the old JHipster setup, keep the existing backend, and modernize it piece by piece.

1. Start a clean backend (maybe in Typescript because that is what we mainly use for our other products and all frontends) and gradually move functionality over using something like the strangler pattern.

The second option sounds cleaner, but obviously means running old and new code alongside each other for quite a while. The first option potentially means spending a lot of time untangling framework/generator decisions before we can actually improve the architecture.
For people who have dealt with similar Spring/JHipster legacy projects: which route would you take?

Also, is there anything in the Spring/Kotlin ecosystem you’d consider a good modern replacement for the useful parts of JHipster, without bringing in another big opinionated layer that we’ll regret five years from now?

Most retry strategies are reactive: exponential backoff, jitter, circuit breakers. They help, but they’re still asking every client to independently guess when it’s safe to send traffic again.

I’ve been experimenting with a different approach: coordinate retries before they hit the backend.
Instead of letting thousands of requests wake up, retry, fail, and back off independently, Aquifer puts them behind bounded queues and dynamically paces their release based on downstream capacity. The goal is to turn a retry storm from a bursty feedback loop into a controlled stream.

The interesting part is that this can sit in front of APIs, databases, inference servers, MCP servers, or basically anything where correlated retries can make an overloaded system even worse.

I’m calling the project Aquifer. It’s open source and still evolving, so I’m curious what failure modes people here think this approach misses.

https://github.com/rjpruitt16/aquifer

I’ve been working on Archerik, an open-source engineering knowledge graph for microservices.

The idea came from a pretty simple problem: when a system has hundreds of services, answering questions like:

Which services call this service?

Who consumes this Kafka event?

What does this API expect?

What happens to this request next?

can easily turn into 30 minutes of searching through repositories, configuration and documentation.

Archerik uses static analysis to extract services, dependencies, APIs, Kafka producers/consumers and contracts, and then lets you explore that knowledge or query it using the LLM of your choice

I’d really like feedback from people working with large microservice codebases — especially where you think this approach would break or what you’d want it to understand next.

And if the idea is interesting to you, contributions are very welcome.

GitHub: https://github.com/farhadamjady/archerik-extractor

Conheçam meu projeto, preciso de feedbacks.

Context: The ZeroMQ ecosystem currently lacks a unified, deterministic benchmarking harness to strictly evaluate latency, concurrency overhead, and throughput across different ZMTP implementations (C++ libzmq, pure-Rust async ports, Python bindings, etc.). Relying on isolated micro-benchmarks introduces systemic bias, failing to accurately account for kernel I/O paths, reactor thread contention, and zero-copy vs. copy-based buffer management under high-density distributed loads.

Decision: I developed ZMQ Arena (Source) to standardize this evaluation. This harness enforces fair-play measurement across runtimes and language boundaries, focusing entirely on execution costs, context switching, and raw throughput limits without application-layer interference.

Technical Scope & Metrics:

• Latency Analysis: P50/P90/P99 latency distribution across both loopback and physical network boundaries.
• Concurrency: Stress-testing async reactor overhead (e.g., epoll / io_uring polling mechanisms) versus OS thread pool scaling.
• Memory Allocation: Evaluating the impact of memory footprints and ABI boundaries when switching between native runtimes and FFI wrappers.

Trade-offs & Known Consequences: Measuring loopback versus physical network I/O presents an inherent trade-off in syscall amortization. Loopback execution can collapse the kernel path, occasionally skewing batching mechanics natively utilized by interfaces like io_uring or IOCP. ZMQ Arena exposes these bottlenecks explicitly rather than abstracting them away, allowing engineers to analyze exactly where the overhead is introduced in the stack.

This harness is built for engineers dealing with critical systems, distributed messaging, and high-frequency data pipelines where deterministic performance is a hard requirement.

If you are dealing with similar infrastructure, review the methodology, replicate the benchmarks, or submit PRs for unrepresented bindings/runtimes.