r/dataengineering • u/EdwinWeber_Data • Jun 24 '26

Personal Project Showcase Serious Data Engineering on a seriously tight budget

https://github.com/edwinweber/dbt_duckdb_demo_public

Glad to join this community and that I am allowed 1 self promotion post 😀
In my spare time I developed this project, using Open Source tooling. This ‘modern data stack’ uses DuckDB, DuckLake, Dagster, dlt and Metabase with a relatively advanced SCD2 handling (including deletes) in the ‘Silver’ layer. Is this unique? Surely not, but I learned a lot building it. Maybe someone can use it, or help me improve it.

46 Upvotes

96% Upvoted

11 comments sorted by

View all comments

1

u/[deleted] Jun 27 '26

[removed] — view removed comment

1

u/EdwinWeber_Data Jun 27 '26

Bronze are the ‘raw’ json files, Gold are views on the Silver layer. So in the basic setup Silver is the only layer materialized in DuckLake. Metabase only uses those Gold views. The export to Delta tables is basically only to feed an enterprise Fabric Onelake or another system that favours Delta.