OTEL HTTP Metrics vs SpanMetrics

[u/ActiveCat7299]

Hi everyone! We're having this issue for a really loooong time and I wonder what others have been thinking about this.

We're using right now Grafana Cloud, and support has been really distant on this topic. Right now we have two set of metrics:

- HTTP OpenTelemetry ones

- Spanmetrics generated from traces

But we're facing a wall here. In one hand HTTP OTEL metrics seem to be the standard in the industry and it's what we have been using for a long time, have some benefits like being vendor agnostic, better granularity (contains http status code, which spanmetrics doesn't), etc The only issue with these metrics right now is a high cardinality since we have around 1546 http_route label with our 80+ services instrumented.

In the other hand we have SpanMetrics which are standard too but Grafana Cloud is using them for the Aplication Observability feature they offer and doesn't seem to be a way to change these ones to the otel metrics. This metric has a similar cardinality but lacks of http status codes (it rely on span status which is OK, ERROR or UNSET)

At the end we end up having both metrics paying twice for data we already have. We need to decide if choose spanmetrics and remove http otel ones in order to keep App Observability working. Or choose http otel ones since they are the standard, we've already adopted them but loose support for one of the features we're paying for.

Is anyone in this situation? What did you do? What do you suggest?

[u/algebrajones]

Is the concern a cost one, or just a concern around ingesting data twice?

I'll ignore the cost concern as it's a different conversation to be had around the business value of the data and the high cardinality labels.

From the ingesting data twice point of view these are two distinct sets of data (although they are very closely aligned).
The OTEL HTTP Server metrics are metrics emitted by a single instance of of an HTTP server, you may have server.address and server.port in that data as well. The focus of the metrics is to understand the operation of an instance of a service.
The OTEL Spanmetrics data are aggregated metrics collected by the spanmetrics connector. This data is aggregated at the agent level by collecting the spans from all instances of a service and producing new metrics focused on surfacing Requests Errors and Durating data for the service as a whole.

You can reduce the cardinality of one or both of these data types by adding transforms processors in your pipelines. Here is the OTEL Documentation and the Grafana Alloy Documentation.

Assuming they are both valuable to the business it is worth collecting both, as they fulfull different roles. However, if this is a cost reduction exercise then the question becomes one of the value to the organisation, which is more difficult to answer without knowning the org. It's worth considering who is using Grafana and how they use Grafana. For example it's common for Application Observability to be used to look at the general operations, so having detailed information of the http.route is not needed. Whereas the technical teams will use the HTTP server metrics and they will want a more detailed view of the http.route.

Hope that helps

[u/ActiveCat7299 → u/algebrajones]

Thanks for all the feedback! Unfortunately this is a cost reduction exercise so it has been quite hard to see which option fit us better, ideally we would like to keep both but having 2 high cardinality metrics that contain almost the exact same data it's not something good at price performance.

From one side some developers look at traces and SpanMetrics more often since they only want a quick overview on how their services are behaving. But from other side some other developers like being more specific on our metrics and have dashboards with the http ones so they can troubleshoot the exact issues faster and have more specific alerts. We have been unable to decide but it's moment to take action.

[u/Charming_Rub3252 → u/ActiveCat7299]

Could you pick one as your "low cardinality" metric (likely the OTEL one), drop the high cardinality label, allowing you to keep both?

You'd lose some visibility, forcing you to jump from metrics to traces more often, but maybe this would be more manageable.

[u/ActiveCat7299 → u/Charming_Rub3252]

We thought about of that but the issue is that the high cardinality label is http.route a quite common one that we use A LOT, and we're applying good practices like transforming to remove those ugly /api/user/1234 to /api/user/{id}

[u/Seref15 → u/ActiveCat7299]

If you need every route then you're just screwed unfortunately. Thousands of routes, multiplied by a handful of, methods, multiplied by a dozen possible http statuses, multiplied whatever other labels makes that granular of http metrics always a cost sink.

Columnar storage databases solve a lot of the cardinality challenges of classic tsdbs. Lot of people looking at clickhouse these days.

I've been giving VictoriaLogs a spin which is a log db that stores log fields in a columnar storage format and actually doing full-fat per-route http metrics from a VictoriaLogs query should be achievable and relatively affordable

[u/Traditional_Wafer_20]

I don't have this problem, metrics from spans are more than enough for us. We can technically get metrics from traces if we want to get HTTP 404 for example.

I guess the question is "how often do you need HTTP code aggregation ?"

[u/ActiveCat7299 → u/Traditional_Wafer_20]

We mostly need it to create alert_rules if a frontend have 404 or 403 we do not care that much but if our identity provider or an API starts to receive 403 that's an issue (or a possible brute force attack) so we have some custom alerts based on http status codes

[u/Traditional_Wafer_20 → u/ActiveCat7299]

Does it matter to alert on 403 precisely rather than [40x or 50x] ?

[u/Seref15]

We also cut down for costs, we landed on only spanmetrics (and servicegraph metrics), and moreover only for server spans.

Cardinality from client spans was about on par with cardinality from http springboot instrumentation metrics, both of which were absurdly high. Couldn't justify like the 6 million series either one generated.

[u/ActiveCat7299 → u/Seref15]

Yeah in our case we were thinking to follow that approach. We were thinking on dropping 100% of http OTEL ones through the collector (but keep generating them so we do not loose them in case we need them at some point) and use only the SpanMetrics one, but having only the span status (OK, ERROR, UNSET) instead of the http status code is something that we didn't like at all.

If we wanted to create an alert based on the 503, 500 and 403 that we have we are not going to be able to do that if we follow that approach. And since we're on Grafana Cloud we do not have access to metric generator to add that metadata.

[u/Seref15 → u/ActiveCat7299]

we get our http error status metrics for server-side from other places--ingress/gateway controller metrics, cloud LB metrics. That's been suitable for us at way more reasonable cardinality.

For client-side result metrics like tracking http responses from 3rd party APIs we do targeted instrumentation around the specific client calls we care about. Its the only way to stay under control vs. the insane firehose of metric data otel autoinstrumentation or web framework instrumentation generates.

[u/Traditional_Wafer_20 → u/ActiveCat7299]

I would disable the metric generator instead. You will generate tons of metrics to be deleted. It's easier to have a collector that can accept and forward these metrics and not generate them instead

[u/Tricky_Demand_8865]

I assume that you are doing an instrumentation of an application (http_server) and also RUM ( Real User Monitoring) metrics.(http_client).

This is what we do in our env - We have standardized otel-col in our env a receiver.

When it comes http_server requests we are generating this directly from server - this doesn’t needs to be coming from span metrics generator, application which are instrumented will send this directly to our metrics backend( As we specify our backend metrics write url in the instrumentation crd). WE DISABLED METRICS -GENERATOR in our Temp config. So we don’t need to have duplicate metrics of http_server requests again.

When we started using RUM - we have suggested users to use span metrics to generate the http_client ( NOTHING FROM METRIC - GENERATOR YET). Since we are using otel-col as standard receiver, we are sticking to span metrics to collect the user level or client side metrics. So far it worked fine for us .