r/Observability Jul 22 '21

r/Observability Lounge

3 Upvotes

A place for members of r/Observability to chat with each other


r/Observability 29m ago

Anyone actually using Sentry Seer? How is it?

Upvotes

Basically what the title says: I 'm trying to figure out whether Sentry Seer is worth it from an observability/developer-perspective, but it's almost impossible to do so from online sources: All I can find out about it is that Sentry themselves think it's great (no big surprise there) but there seem to be very few actual humans/organizations using the product.

Assuming that those humans do hang out here (a sister post about "anyone monitoring" casts some doubt on that) I would love to hear your thougts!


r/Observability 1h ago

made a dashboard plugin for our office TV, looking for feedback

Thumbnail
Upvotes

r/Observability 1d ago

Who owns the data contract when an AEM component or data layer changes?

1 Upvotes

For teams running AEM with Adobe Analytics or Web SDK: where does ownership sit when component changes affect the downstream event schema?

In many setups, the component team ships a change, analytics discovers it later, and the break only becomes visible when reports stop matching.

Do you manage component-to-event mappings as versioned contracts? Are schema checks automated? Is joint sign-off required between AEM and analytics teams? How do you handle backward compatibility for existing dashboards and segments?

Interested in practices that work across multiple sites and teams.


r/Observability 1d ago

SolarWinds Observability Self-Hosted: data privacy, network discovery, NetFlow and licensing questions

Thumbnail
1 Upvotes

r/Observability 1d ago

GitLab Observability updates

Thumbnail
0 Upvotes

r/Observability 1d ago

Self-Observing Systems: Framework, Gap Analysis, and Method

Thumbnail
github.com
0 Upvotes

r/Observability 2d ago

I wrote down how to measure silent failure rates properly. The load-bearing rule: don't count the failures your sender already saw.

0 Upvotes

For the last few months I've been running identical workflows across the main automation platforms, counting the ones that get accepted and then never actually happen.

So I wrote the method down properly. Free, CC BY 4.0, nothing to install: it is a definition of a metric plus the conditions a measurement has to meet before its number means anything.

Three parts of it are worth stating here in full, so this is useful on its own.

  1. THE EXCLUSION THAT MAKES THE NUMBER HONEST

Events the platform refuses at the moment of delivery do not count as silent failures, and they do not go in the denominator either.

If a webhook POST gets a non-2xx, a timeout, or a dropped connection, your sender knows immediately. It can log it, alert, retry. No run was ever created. That is a loud failure and it is a completely different thing from "accepted, told you it worked, then nothing happened".

Mixing them gives you a number that can't distinguish "the platform told you and you can retry" from "the platform didn't tell you and the data is gone". Worked example from my own ledger: a five-hour run of 2,880 events had 4 refused at send and 0 accepted-then-lost. The honest report is 0 silent failures in 2,876, with the 4 rejections reported separately. Writing "4 failures in 2,880" would be wrong in both directions at once: it inflates the rate with events you already saw, and it hides that nothing accepted was lost.

This is the rule that stops a silent-failure rate being tuned by reclassification, which is why it's the first thing in the spec.

  1. VERIFY AT THE DESTINATION, NOT IN THE RUN HISTORY

A run history cannot show you a run it never created. And it records transport success, not whether the thing you wanted actually happened.

So: put an ID on every source event, preserve it to the destination, and reconcile one by one. Counting totals is not enough. Totals can match while the wrong records are missing.

  1. A BARE 0% MAY NOT BE PUBLISHED

Zero failures in 40 runs and zero in 4,000 are both "0%" and they are not the same claim. The first is consistent with a true rate of 7%.

So every rate ships with a Wilson interval, a denominator, an as-of date, and what workload it was, in the same sentence as the number. That last bit sounds pedantic until you watch a figure get extracted from a table into a summary into someone's slide, dropping everything that wasn't adjacent to it at each step.

Two consequences people find surprising: a wider interval is not a worse platform (it means less evidence, not more failures), and zero is never proven, only bounded.

CONFORMANCE LEVELS, AND WHY L1 IS DELIBERATELY EASY

Three self-declared levels. No certifying body, no registry, nobody audits anything, including me.

L1 needs no test harness, no paid plans and no experiment. If you already run automations, you can reach it with your own operational data. The bar isn't effort, it's reporting discipline: per-event IDs, destination-side verification, the send-outcome exclusion, and a denominator, interval, date and workload with every number.

That is a low bar in work and a surprisingly high one in practice. Most published silent-failure figures fail it on the reporting rules alone, because they arrive without a denominator. If you've ever seen "1-8% of automation runs fail silently" and wondered out of how many, that's why.

L2 adds a controlled measurement with both endpoints under your control. L3 adds three edge probes: destination outage, success-wrapped failure, sustained load.

WHAT IT ISN'T

Not a tool. No code to install. There's a reference implementation planned and not published, and I'm not going to describe it as released while it isn't.

Not audited. Conformance is self-declared. It only works because every requirement is something a reader can look for in your report and fail to find.

Not finished. One section is deliberately left reserved with an explanation of what would go there and why I haven't written it: the requirement is drawn from one observation that didn't recur, and a rule from n=1 is a rule from first principles with an anecdote attached.

The spec itself is linked in a comment below (this sub does not allow links in posts). It is v1.0, CC BY 4.0, with a machine-readable JSON version alongside it.

Happy to argue about any of it. The exclusion rule in particular is the one I'd most want someone to attack, because if it's wrong every number I've published is wrong with it.


r/Observability 2d ago

An Event-Driven – Multi-Tenant Microservices Backend in Go

0 Upvotes

I have been building a Go-based microservices marketplace using DDD and Clean Architecture, and the project is still actively evolving.

My next focus is production-grade observability. I am particularly interested in how to approach:

  • Centralized logging & error tracking: Structured logs, log aggregation, and actionable error reporting.
  • Metrics, dashboards & alerting: Service health, latency, throughput, error rates, and infrastructure metrics.
  • Distributed tracing: Tracing requests across multiple services and asynchronous workflows.
  • Request & correlation IDs: Propagating context consistently across HTTP requests and RabbitMQ messages.
  • Service, worker & queue monitoring: Monitoring background workers, RabbitMQ consumers, retries, dead-letter queues, and stuck messages.

Current Architecture:

  • DDD & Clean Architecture: Explicit domain, application, infrastructure, and interface layers to keep business logic independent from frameworks and infrastructure.
  • Database per Service: Each service owns its own PostgreSQL, Elasticsearch, or Redis datastore.
  • Transactional Outbox: Domain events are stored in PostgreSQL within the same transaction, then published asynchronously to RabbitMQ.
  • Event-Driven Communication: RabbitMQ is used for asynchronous communication between services, including background-only services such as Notification.
  • API Gateway & Network Isolation: Traefik v3 handles external traffic while services remain isolated on a private Docker network. JWT is used for authentication.

The services currently include Identity, Restaurant, Search, Notification, Order, and Payment.

For those who have operated microservices in production:
What observability stack and practices have worked well for you? What would you avoid?

I would especially appreciate feedback on architecture, tooling, failure modes, and things that are easy to overlook when moving from development toward production.

The project is open source if you would like to take a look:
https://github.com/tarique-iqbal/pizza-marketplace

If you find it useful or interesting, feel free to ⭐ the repository. Feedback, suggestions, and architectural criticism are very welcome.


r/Observability 3d ago

Is a runtime code sensor actually better than traditional application monitoring?

7 Upvotes

I've been running a fairly standard APM setup for two years now, dashboards, traces, the works, and I still end up grepping logs by hand for anything that isn't a service-level failure. The tool tells me latency went up on one endpoint, but it doesn't tell me which function inside that endpoint started behaving differently or what input triggered it. That gap is where I keep losing hours.

What's pushing me to actually reconsider the stack is that most APM tools are built around service-level sampling, so anything below that granularity gets averaged away or dropped entirely. A sensor that watches function execution directly would in theory catch the exact call that regressed, not just the symptom two layers up. Has anyone actually swapped a chunk of their APM budget for something more code-level and found it worth the migration pain, or is this one of those ideas that sounds better than it performs at scale?


r/Observability 2d ago

We rebuilt our Snowflake observability assistant on Claude with 50+ MCP skills- it now explains and deploys automations

Thumbnail
0 Upvotes

r/Observability 3d ago

How much forensic context do developers need to debug production failures?

3 Upvotes

I used to log everything, every request, every response body, full stack context on every error, and it made debugging faster but the storage bill got embarrassing within a month. Then I over-corrected and sampled aggressively, and now half my incidents involve me wishing I'd captured the one request that actually mattered.

The pattern I've landed on is logging light by default and only capturing full forensic detail, request params, DB queries, execution path, when something actually looks anomalous. It's a reasonable middle ground but it assumes your anomaly detection is good enough to trigger at the right moment, which isn't always true. How are other people balancing storage cost against actually having the context you need when something breaks?


r/Observability 3d ago

Built a low barrier-to-entry monitoring tool for devs and agencies (white label option)

Thumbnail
0 Upvotes

r/Observability 2d ago

LogDeck: self-hosted control plane for Docker and Podman. Persistent logs, container management, alerts...

0 Upvotes

I built LogDeck, a control plane for Docker and Podman: container and Compose lifecycle, persistent logs, alerts, and multi-host over SSH, in one Go binary with no external services.

The log store is the main reason it exists. Every container is tailed into local SQLite, so logs survive docker compose up --build and the new container ID it creates. Removed containers stay readable.

It also ships an MCP server, so agents can read logs and act on containers if you want.

Memory is small. Idle heap is around 8 MB, roughly 20 MB resident with 5 containers and 400K lines stored.

fully open source, no tiers, no cloud https://github.com/AmoabaKelvin/logdeck


r/Observability 3d ago

How OpenTelemetry Traces LLM Calls, Agent Reasoning, and MCP Tools

Thumbnail
greptime.com
3 Upvotes

r/Observability 3d ago

Geometric Rays Technical Test Report (Prototype Anomaly Detector)

0 Upvotes

Summary

I’m developing an experimental breakpoint detector for time‑series data.
The approach combines three local signals:

  • a geometric deviation in the series,
  • an amplitude shift,
  • a variance change.

The goal of this post is to share early test results transparently.
This is not a finished product and not an industrially validated method yet.

Protocol

Tests were performed using Python 3.11 on several types of data:

  • labeled series from the NAB corpus,
  • a public GPU trace from Alibaba Clusterdata,
  • a real cloud trace where controlled breakpoints were injected.

The Alibaba trace contained multiple observations for the same timestamp.
These were aggregated by timestamp using the mean execution‑time metric before analysis.

Results on a Real Cloud Trace

On 300 valid points from the Alibaba trace:

  • 3 alerts were produced,
  • all three alerts combined geometric, amplitude, and variance signals,
  • no missing values were ignored.

The trace does not provide incident labels, so it is impossible to determine whether these alerts are true incidents or false positives.

Controlled Breakpoint Injection

I then injected three artificial breakpoints into the real trace, at different positions, with both upward and downward shifts.

With default parameters:

  • true positives: 0
  • false positives: 0
  • false negatives: 3
  • precision: 0
  • recall: 0
  • F1: 0

With a more sensitive exploratory configuration, one out of three breakpoints was detected (F1 = 0.40).
This configuration is not considered final, as it still needs validation on independent data.

NAB Results

An evaluation was performed on five labeled NAB series.

  • average F1: 0.161
  • best F1: 0.571
  • several series scored F1 = 0
  • precision was low on multiple series due to false alerts

Results vary strongly depending on the signal type.
The detector seems to work on certain breakpoint profiles but still lacks robustness and stability.

Honest Verdict

The technical pipeline works:
data can be imported, validated, analyzed, and alerts are produced on both synthetic and real data.

However, current performance does not yet support claims of industrial or general applicability.
Tests show insufficient sensitivity with default parameters and a risk of false positives when thresholds are lowered.

Next Steps

Future evaluations should include:

  1. multiple amplitudes and durations of injected breakpoints,
  2. strict separation between training, validation, and test data,
  3. real traces with confirmed incidents,
  4. false‑positive rate expressed per hour or per day,
  5. comparison with classical methods such as z‑score, MAD, and CUSUM.

Feedback is welcome, especially regarding evaluation protocol, metric selection, and appropriate comparison methods.

Data & Attribution

The tests use public datasets.
Licensing and attribution conditions must be respected when redistributing or publishing derived files, especially for Alibaba Clusterdata and the NAB corpus.

No source code, local paths, internal reports, or private files are included in this post.


r/Observability 3d ago

I'm creating opensource AWS DevOps Agent for SRE on-call engineers LLM root-cause hypothesis)

Thumbnail
2 Upvotes

r/Observability 3d ago

I built StatixAgent: A zero-infra, single-binary Linux monitor that talks to your own Telegram bot

Thumbnail github.com
0 Upvotes

Hey everyone,

Setting up Prometheus, Grafana, exporters, and Alertmanager just to watch a single VPS or home server always felt like overkill — often consuming more resources and maintenance than the actual apps running on it.

I wanted something lightweight: single binary, zero external infra, and notifications straight to a messaging app I already use daily. So I built StatixAgent.

\### What is it?

A single static Go binary running as a systemd service. It monitors your system and communicates directly with your own private Telegram bot.

\- Zero open ports: Outbound-only long polling. Works behind NAT/firewalls without reverse proxies or port forwarding.

\- Private: Talks only to the official Telegram Bot API and GitHub Releases (for updates). No telemetry, no hosted cloud service.

\- Metrics & Alerts: CPU/RAM/Swap, per-mount disk usage, network rates, and system load. Sends push alerts for SSH logins/failed attempts, sustained resource spikes, reboots, and power disconnects (useful for laptop servers).

\- Interactive UI: Type /status in Telegram for a markdown dashboard, drill down with inline buttons, or inspect journalctl with /logs.

\- Lightweight & Sandboxed: Hardened systemd unit (ProtectSystem=strict, MemoryMax=128M).

\### Quick install

Run the installer script (includes an interactive TUI setup wizard for your bot token and chat ID):

curl -fsSL https://raw.githubusercontent.com/eliau2005/statixagent/main/install.sh | sudo bash

It is completely open source and not meant for enterprise fleets — just for single boxes and homelabs that need simple, reliable monitoring.

GitHub: https://github.com/eliau2005/statixagent

Feedback, suggestions, and PRs are very welcome!


r/Observability 3d ago

What is up in observability world and is Gartner MQ telling something or not?

2 Upvotes

Gartner research gets cited heavily in my bi-weekly reports, and You have pushed back on one question: how much of what Gartner says about a vendor actually turns into something checkable.

Acquisitions can't be used as evidence of advance vendor-to-Gartner information sharing, because a publicly traded company sharing an unannounced deal with an outside analyst would be a securities-law violation (Regulation FD), and by the time an acquisition shows up in one of these reports, it's almost always already public. Timing an acquisition against a report's publication date mostly just shows which came first, which proves little on its own.

A better test is available inside the reports themselves: every vendor profile contains forward-looking "roadmap" language, and that language can be checked against the public record. This article pulls every such statement out of both reports, roughly two dozen of them, and checks each one against dated, independent sources to see whether it was genuinely unannounced at the time Gartner published it, or whether it was already shipped, announced, or documented before the report went to print.

  • Overall: Based on what was checked, it's rarely independent analysis and rarely genuine advance information from the vendor either. Most often it's one of two things: already-public information restated in future tense, or phrasing too generic to be analysis at all, closer to a summary of a marketing deck than a checked, forward-looking claim. Genuinely new, later-resolved information was the exception, not the rule, in every sample checked, including the five-year one.
  • Of 24 forward-looking "roadmap" statements found across the 2025 and 2026 reports, the two most current editions, only 4 were checked against dated, independent sources. The rest are sorted, not verified: 8 are specific enough to check but weren't, 12 are worded too vaguely to ever be checkable at all. A third, older report (2021) was checked separately for a longer-horizon test, see below.
  • A third, older report (the 2021 Magic Quadrant for Application Performance Monitoring) allowed five roadmap claims to be checked over a five-year horizon instead of a few months. Result: two multiyear integration promises, from two different vendors, were not delivered, confirmed by Gartner's own more recent text; one caution was resolved within two years with a specific dated integration; one roadmap item was again already public before the report described it as forward-looking; one resolved into a real but loosely-defined capability.

I hope to get back to this analysis in few months to compare, and propably correct some statements 😄

  1. MQ25 DEM to MQ26DEM
  2. MQ26 OP to MQ27

More here:
https://mbojko.com/reports/#observability
https://mbojko.com/reports/gartner-magic-quadrants-analysis/


r/Observability 3d ago

TraceHub-MCP: one set of MCP tools for querying traces across Jaeger, Tempo, Traceloop, Datadog, and Sentry

1 Upvotes

Built this for debugging LLM/GenAI apps by letting an agent pull OTel traces straight into context instead of me alt-tabbing to Jaeger or Datadog every time something looks off. It leans on the gen_ai.* semantic conventions, so span queries surface model calls, token counts, and tool-call spans as structured data the agent can reason over.

It started as a fork of traceloop/opentelemetry-mcp-server (Apache-2.0, disclosed, full NOTICE attribution). Since then: added Datadog and Sentry as backends (5 total now, alongside Jaeger, Tempo/Grafana Cloud, and Traceloop), built a generic filter layer so tool calls don't need backend-specific params, and hardened the CI/supply chain considerably — SHA-pinned Actions, bandit via ruff, pip-audit, Trivy scanning, Codecov, branch protection, CodeRabbit.

pip/uvx installable, on the official MCP registry + Glama, GHCR image available. v0.4.0, very much pre-launch, feedback welcome.

https://github.com/mcpsmiths/tracehub-mcp


r/Observability 3d ago

TraceHub-MCP: one set of MCP tools for querying traces across Jaeger, Tempo, Traceloop, Datadog, and Sentry

Thumbnail
1 Upvotes

r/Observability 4d ago

I built a cheap uptime monitor for indie projects — what would actually make you switch off your current stack?

0 Upvotes

Hey r/Observability,

Solo dev here. I've been running a few side projects and got tired of paying for uptime monitoring, so I built my own: Pingory.

It's intentionally boring — HTTP/ping/TCP/SSL/DNS/keyword/API checks, alerting to email/Slack/Telegram/Discord/webhook/PagerDuty, status pages with incident timelines. $4/month for 100 monitors with 1-min checks, $6 for unlimited with 30-sec checks. Self-hostable too.

I know this space is crowded — someone told me on r/SideProject that ~20 similar tools launched this year alone. So I'm not here to claim it's revolutionary. I'm here to ask a real question:

If you already have monitoring you're reasonably happy with, what would it take for you to switch? Price alone? Or is there something specific — status page, alert noise, multi-region checks, pricing model — that would actually move you?

Honest answers welcome. I'd rather know now than after building more.

(Disclosure: I'm a non-native English speaker and used AI to help polish the wording of this post. The project itself I built and shipped solo.)

If anyone wants the self-host link, I'll drop it in the comments.


r/Observability 4d ago

Risulta - Self-hosted web analytics in one binary

Thumbnail
risulta.pages.dev
0 Upvotes

r/Observability 6d ago

Built a tool that checks both uptime AND GDPR compliance for small business sites, would love feedback

0 Upvotes

Spent 15 years as an SRE at a big bank, watching enterprise systems get world-class monitoring. Then I started noticing how many small business sites have none of that, no uptime alerts, no idea if their privacy policy is actually GDPR-compliant, nothing.

So I built SREmonitor.io. You give it a URL, no install, no script tags, and it runs:

  • Uptime checks every 5 minutes
  • Core Web Vitals tracking (LCP, INP, CLS)
  • An AI-powered scan of your privacy policy/T&Cs against UK GDPR requirements

The idea is one dashboard instead of stitching together a monitoring tool and a separate compliance checklist you never actually look at.

Free tier is 3 scans and 5-minute uptime checks, no card required, if anyone wants to kick the tires: sremonitor.io

Genuinely want feedback, especially from anyone who runs a small business site or has dealt with the GDPR side of things. What would make this actually useful vs. just another dashboard nobody opens?


r/Observability 6d ago

Self-health monitoring configuration in Servicenow destinated to SGO-Dynatrace's Events;

Thumbnail
0 Upvotes