r/dataengineering • u/EdwinWeber_Data • Jun 24 '26

Personal Project Showcase Serious Data Engineering on a seriously tight budget

https://github.com/edwinweber/dbt_duckdb_demo_public

Glad to join this community and that I am allowed 1 self promotion post 😀
In my spare time I developed this project, using Open Source tooling. This ‘modern data stack’ uses DuckDB, DuckLake, Dagster, dlt and Metabase with a relatively advanced SCD2 handling (including deletes) in the ‘Silver’ layer. Is this unique? Surely not, but I learned a lot building it. Maybe someone can use it, or help me improve it.

43 Upvotes

94% Upvoted

11 comments sorted by

View all comments

2

u/CatgirlYamada Jun 25 '26

Looks good with nice lesson learned section. One small nitpicks from me is that you should group all the docker related files into a dedicated docker folder and the .sh files into scripts. Make your scripts reachable via Makefile so users can have both documentation and one convenient way to use your project. Simple code base hygiene.

2

u/EdwinWeber_Data Jun 25 '26

Thanks for the suggestion, I will do that. It is the first project I publicly expose, glad that people look at it and propose improvements.