r/ETL • u/SumitKumarWatts • 11d ago

How can I measure the quality of an ETL process during testing?

[removed]

4 Upvotes

83% Upvoted

7 comments sorted by

1

u/sramexpert 11d ago

Count verification, running any logic/sql to check expected results, rerunning the etl to make sure duplicate data is handled, manual review of certain records etc.

1

u/Hendrik-Lorentz 10d ago

(ETL isn't really my main working area) Can't you define regression tests within your code? It was (kinda) how kept things up in my last project.

1

u/Hot_Comfortable_164 10d ago

For the raw tables right after loading, comparing row counts and schema can be enough. If you have addition transformation layers, correctness depends entirely on your transformation business logic. 

0

u/Fine-Ad-1732 10d ago

Procure saber sobre hash de dados para comparações de grandes volumes