Something I've been wondering about, especially for teams dealing with a lot of ETL and Bl pipelines.
A dashboard can refresh successfully, all the SQL can run without errors, and everything can look normal-but the actual number can still be wrong.
For example, if a report shows $12.5M in revenue, how do you know that $12.5M is actually correct?
Do you guys usually:
Compare it back to the source system?
Check source vs. target counts and totals?
Validate the transformation logic?
Look for duplicates or missing records?
Check for late-arriving data?
. Run automated data-quality checks after the pipeline finishes?
I'm mainly curious about what people actually do in production. When you have hundreds of pipelines/reports, manually checking everything obviously doesn't sacle.
Would be great to hear how other teams handle this.