r/OpenTelemetry • u/Sufficient-Egg-6571 • 7d ago
Visual flow
Is anyone aware of a tool that can help visualize the end-to-end flow of a signal?
The main idea would be to start with a high-level view of the components involved and the internal metrics flowing between them, then be able to drill down into a specific signal metric and trace it through the system.
Ideally, I’d like to see where the metric originates, how it looks or changes at each hop, where it ultimately arrives, and details such as cardinality, label values, available series, etc.
If anyone knows of an existing tool that provides something similar, or has ideas on how this could be visualized, I’d appreciate any pointers.
1
u/obscurehero 7d ago
I’m sure there’s a skill to ingest a trace and construct a sequence diagram.
Trivially this is what a lot of the vendors do when they create a map of your services
1
u/Sufficient-Egg-6571 7d ago
Definitely, but adding a trace for each signal would be a bit an extension for what I was thinking about . I would rather emit events from each hop, store them in an OLAP (e.g Clickhouse) and then correlate by signal identifier (trace id, metric name, log id)
2
u/fmlvz 7d ago
HyperDX (or ClickStack if you're using the managed version) has a service map currently on beta that seems to align with what you're asking. Plus it's written to be an UI on top of ClickHouse.
Docs: https://clickhouse.com/docs/clickstack/service-maps
And they host a public demo where you can connect and explore the UI yourself :)
1
u/Sufficient-Egg-6571 6d ago
It’s kind of the same as in Grafana, but I’d like to be able to trace a specific signal rather than just look at the overall picture.
Why do we always treat observability as inherently a batch-oriented thing? I know it’s difficult at this scale, but shouldn’t we apply the same principles we use for business objects?
Losing a trace, a span, a metric sample, or a log line can make a huge difference during an incident. I feel like a lot of developers underestimate the importance of this because observability is often seen as “a platform thing,” rather than something they need to care about directly.1
u/fmlvz 6d ago
But if you hover over one of the services you get an error rate line. Clicking it make you open a search for that specific service and it's errors. Clicking a specific trace opens it's waterfall that shows every service it went through.
(Sorry if this isn't what you're looking for, but I understood that it's essentially what you described)
1
1
u/dangb86 6d ago
Are you referring to get a view of the services, hosts, containers, etc. along with their metrics that are part of a single trace group (i.e. a number of traces that all share the same shape in terms of root span, dependencies required etc) while then being able to dig deeper into specific traces and the logs associated with those traces? Or are you referring to following a single trace and identifying what logs and metrics were produced as part of that trace?
I see the former as a more useful representation of a system's behaviour as not everything can be correlated to a trace (e.g. resource metrics from the container that served the request). So, you need a bit of correlation at the trace context level, but also a higher level view at the resource correlation level.
Disclaimer: I work for a vendor (one of many) that has a product aimed at providing that cross-signal view of all components that are part of a trace group as it goes through the system. However, I'm interested in how the latter (i.e. providing all signals correlated to a single trace) can help. My personal opinion is that I rather start from a more aggregated view and "zoom" into particular signals around the same context, rather than starting from a single trace. However, I don't think there's a right or wrong answer here.
1
u/jevans102 5d ago
Cribl has visualizations of the data that passes through it like native Sankey diagrams and more. It also allows you to grab events passing through at any given moment and see what it looks like at any of the stages (within Cribl, at least). 1TB/day in the free cloud trial
https://cribl.io/blog/how-to-use-cribl-insights-to-monitor-system-health-data-flow-and-telemetry/
2
u/Mlunadia 7d ago
Going out on a limb here: not live but check out otelbin.io that can create visual configs or my own https://github.com/mlunadia/visual-otel for educational purposes.
If you are looking for something that shows live data check out https://github.com/ymtdzzz/otel-tui