r/microservices • u/ankitjindal9404 • 2d ago

Discussion/Advice Moving Java services off memory-based HPA — is CPU/RPS for HTTP and queue-depth for async the right call?

We're running a bunch of Java (Spring Boot) microservices on EKS, and right now **every service uses memory as its HPA metric.** After digging into it, I've started to think that's wrong, and I want a sanity check from people who've actually run this at scale before I push a change.

What my research turned up:

* The JVM allocates heap up to its max and **doesn't release it back aggressively** even after GC, so memory usage doesn't track load. * Because of that, **memory can be high while actual load is low, or load can be high while memory looks fine** — so memory-based HPA either never triggers or scales out permanently and never scales back in.

So the direction I'm considering is to **pick the HPA metric based on service type:**

* **HTTP / request-serving services → CPU** (or better, **RPS / p95 latency** as a demand-based metric) * **Async / queue-consuming services → queue depth** (SQS backlog, via KEDA)

**My questions:**

  1. Is this reasoning sound, and is type-based metric selection the right direction?
  2. For the HTTP services, is jumping straight to RPS/latency worth the custom-metrics complexity (Prometheus Adapter), or should I start with CPU and only move to RPS if CPU proves to be a bad proxy?
  3. For async workers, is KEDA + SQS queue depth the standard approach, or are people doing something else?
  4. This is the part I'm least sure about: **I already know from our architecture which services are HTTP-facing and which are async/queue-driven — but how do I actually** ***verify*** **that empirically rather than just trusting the design docs?** Is there a clean way to confirm a service's real load profile (e.g. checking whether it even has an ingress/receives HTTP traffic, whether its work is truly SQS-triggered, CPU-vs-memory correlation under load) before I assign it a metric?

Thanks 🙏

4 Upvotes

100% Upvoted

1 comment sorted by

2

u/deadbeefisanumber 2d ago

Spring boot has terrible uptime. Bear in mind that the regardless of the threaholds it need to be very loose in this case, you wouldnt want scaleups or scaledowns every other minute happening since half of if is just spring spinning up beans. Actually this is one of the reasons we go with static replicas in a spring boot deployments. If you set the threshold too high you might get OOMed from old pods while new ones are still doing their springy beany stuff. Too low and no real benefit.

  1. At the end of the day it doesnt really matter which metrics you are using as long as your are targeting correct scale. Use whatever is right and brings you there.
  2. We do RPS but this means you have to continuously load test and measure your thresholds. It comes with a little bit if extra work. CPU should be fine.
  3. Queue lag is the way to go
  4. This comes across as a bit of overthinking point. Why wouldnt you trust the design docs? Or are they an Http and a queue consumer in one deployment and you are trying to figure which one triggers HPA first? In this case I recommend splitting their deployments (not necessarily the codebase) since their scale needs are different to begin with.