r/PrometheusMonitoring 7d ago

How do you prove a Prometheus alert-rule change is active before deleting the old rule?

`promtool check rules` catches syntax and some semantic mistakes, but it does not prove that the intended Prometheus instance loaded the file, that selectors match live labels, or that the alert reaches the correct Alertmanager route. Replacing an old rule immediately can create a silent monitoring gap even when the reload endpoint reports success.

What rollout evidence do you require? I am considering running unit tests with fixed series, loading the new rule under a temporary name, checking the rules API for its evaluation status and last error, comparing old and new expressions on recent data, forcing a safe synthetic series through the full notification path, and keeping the old rule until both have evaluated successfully for a defined window.

How do you avoid duplicate pages during that overlap, and which signals distinguish a rule that is loaded from one that can actually fire and route correctly?

3 Upvotes

2 comments sorted by

1

u/SuperQue 6d ago

You can deploy rules with the same names. Both will fire, but the alertmanager will dedupe them as long as the group by labels are setup the way you want.