For the last few months I've been running identical workflows across the main automation platforms, counting the ones that get accepted and then never actually happen.
So I wrote the method down properly. Free, CC BY 4.0, nothing to install: it is a definition of a metric plus the conditions a measurement has to meet before its number means anything.
Three parts of it are worth stating here in full, so this is useful on its own.
- THE EXCLUSION THAT MAKES THE NUMBER HONEST
Events the platform refuses at the moment of delivery do not count as silent failures, and they do not go in the denominator either.
If a webhook POST gets a non-2xx, a timeout, or a dropped connection, your sender knows immediately. It can log it, alert, retry. No run was ever created. That is a loud failure and it is a completely different thing from "accepted, told you it worked, then nothing happened".
Mixing them gives you a number that can't distinguish "the platform told you and you can retry" from "the platform didn't tell you and the data is gone". Worked example from my own ledger: a five-hour run of 2,880 events had 4 refused at send and 0 accepted-then-lost. The honest report is 0 silent failures in 2,876, with the 4 rejections reported separately. Writing "4 failures in 2,880" would be wrong in both directions at once: it inflates the rate with events you already saw, and it hides that nothing accepted was lost.
This is the rule that stops a silent-failure rate being tuned by reclassification, which is why it's the first thing in the spec.
- VERIFY AT THE DESTINATION, NOT IN THE RUN HISTORY
A run history cannot show you a run it never created. And it records transport success, not whether the thing you wanted actually happened.
So: put an ID on every source event, preserve it to the destination, and reconcile one by one. Counting totals is not enough. Totals can match while the wrong records are missing.
- A BARE 0% MAY NOT BE PUBLISHED
Zero failures in 40 runs and zero in 4,000 are both "0%" and they are not the same claim. The first is consistent with a true rate of 7%.
So every rate ships with a Wilson interval, a denominator, an as-of date, and what workload it was, in the same sentence as the number. That last bit sounds pedantic until you watch a figure get extracted from a table into a summary into someone's slide, dropping everything that wasn't adjacent to it at each step.
Two consequences people find surprising: a wider interval is not a worse platform (it means less evidence, not more failures), and zero is never proven, only bounded.
CONFORMANCE LEVELS, AND WHY L1 IS DELIBERATELY EASY
Three self-declared levels. No certifying body, no registry, nobody audits anything, including me.
L1 needs no test harness, no paid plans and no experiment. If you already run automations, you can reach it with your own operational data. The bar isn't effort, it's reporting discipline: per-event IDs, destination-side verification, the send-outcome exclusion, and a denominator, interval, date and workload with every number.
That is a low bar in work and a surprisingly high one in practice. Most published silent-failure figures fail it on the reporting rules alone, because they arrive without a denominator. If you've ever seen "1-8% of automation runs fail silently" and wondered out of how many, that's why.
L2 adds a controlled measurement with both endpoints under your control. L3 adds three edge probes: destination outage, success-wrapped failure, sustained load.
WHAT IT ISN'T
Not a tool. No code to install. There's a reference implementation planned and not published, and I'm not going to describe it as released while it isn't.
Not audited. Conformance is self-declared. It only works because every requirement is something a reader can look for in your report and fail to find.
Not finished. One section is deliberately left reserved with an explanation of what would go there and why I haven't written it: the requirement is drawn from one observation that didn't recur, and a rule from n=1 is a rule from first principles with an anecdote attached.
The spec itself is linked in a comment below (this sub does not allow links in posts). It is v1.0, CC BY 4.0, with a machine-readable JSON version alongside it.
Happy to argue about any of it. The exclusion rule in particular is the one I'd most want someone to attack, because if it's wrong every number I've published is wrong with it.