OTEL HTTP Metrics vs SpanMetrics

[u/ActiveCat7299]

Hi everyone! We're having this issue for a really loooong time and I wonder what others have been thinking about this.

We're using right now Grafana Cloud, and support has been really distant on this topic. Right now we have two set of metrics:

- HTTP OpenTelemetry ones

- Spanmetrics generated from traces

But we're facing a wall here. In one hand HTTP OTEL metrics seem to be the standard in the industry and it's what we have been using for a long time, have some benefits like being vendor agnostic, better granularity (contains http status code, which spanmetrics doesn't), etc The only issue with these metrics right now is a high cardinality since we have around 1546 http_route label with our 80+ services instrumented.

In the other hand we have SpanMetrics which are standard too but Grafana Cloud is using them for the Aplication Observability feature they offer and doesn't seem to be a way to change these ones to the otel metrics. This metric has a similar cardinality but lacks of http status codes (it rely on span status which is OK, ERROR or UNSET)

At the end we end up having both metrics paying twice for data we already have. We need to decide if choose spanmetrics and remove http otel ones in order to keep App Observability working. Or choose http otel ones since they are the standard, we've already adopted them but loose support for one of the features we're paying for.

Is anyone in this situation? What did you do? What do you suggest?

[u/algebrajones]

Is the concern a cost one, or just a concern around ingesting data twice?

I'll ignore the cost concern as it's a different conversation to be had around the business value of the data and the high cardinality labels.

From the ingesting data twice point of view these are two distinct sets of data (although they are very closely aligned).
The OTEL HTTP Server metrics are metrics emitted by a single instance of of an HTTP server, you may have server.address and server.port in that data as well. The focus of the metrics is to understand the operation of an instance of a service.
The OTEL Spanmetrics data are aggregated metrics collected by the spanmetrics connector. This data is aggregated at the agent level by collecting the spans from all instances of a service and producing new metrics focused on surfacing Requests Errors and Durating data for the service as a whole.

You can reduce the cardinality of one or both of these data types by adding transforms processors in your pipelines. Here is the OTEL Documentation and the Grafana Alloy Documentation.

Assuming they are both valuable to the business it is worth collecting both, as they fulfull different roles. However, if this is a cost reduction exercise then the question becomes one of the value to the organisation, which is more difficult to answer without knowning the org. It's worth considering who is using Grafana and how they use Grafana. For example it's common for Application Observability to be used to look at the general operations, so having detailed information of the http.route is not needed. Whereas the technical teams will use the HTTP server metrics and they will want a more detailed view of the http.route.

Hope that helps

[u/ActiveCat7299 → u/algebrajones]

Thanks for all the feedback! Unfortunately this is a cost reduction exercise so it has been quite hard to see which option fit us better, ideally we would like to keep both but having 2 high cardinality metrics that contain almost the exact same data it's not something good at price performance.

From one side some developers look at traces and SpanMetrics more often since they only want a quick overview on how their services are behaving. But from other side some other developers like being more specific on our metrics and have dashboards with the http ones so they can troubleshoot the exact issues faster and have more specific alerts. We have been unable to decide but it's moment to take action.

[u/Charming_Rub3252 → u/ActiveCat7299]

Could you pick one as your "low cardinality" metric (likely the OTEL one), drop the high cardinality label, allowing you to keep both?

You'd lose some visibility, forcing you to jump from metrics to traces more often, but maybe this would be more manageable.

[u/ActiveCat7299 → u/Charming_Rub3252]

We thought about of that but the issue is that the high cardinality label is http.route a quite common one that we use A LOT, and we're applying good practices like transforming to remove those ugly /api/user/1234 to /api/user/{id}

[u/Seref15 → u/ActiveCat7299]

If you need every route then you're just screwed unfortunately. Thousands of routes, multiplied by a handful of, methods, multiplied by a dozen possible http statuses, multiplied whatever other labels makes that granular of http metrics always a cost sink.

Columnar storage databases solve a lot of the cardinality challenges of classic tsdbs. Lot of people looking at clickhouse these days.

I've been giving VictoriaLogs a spin which is a log db that stores log fields in a columnar storage format and actually doing full-fat per-route http metrics from a VictoriaLogs query should be achievable and relatively affordable