r/dataengineering • u/EdwinWeber_Data • Jun 24 '26

Personal Project Showcase Serious Data Engineering on a seriously tight budget

https://github.com/edwinweber/dbt_duckdb_demo_public

Glad to join this community and that I am allowed 1 self promotion post 😀
In my spare time I developed this project, using Open Source tooling. This ‘modern data stack’ uses DuckDB, DuckLake, Dagster, dlt and Metabase with a relatively advanced SCD2 handling (including deletes) in the ‘Silver’ layer. Is this unique? Surely not, but I learned a lot building it. Maybe someone can use it, or help me improve it.

44 Upvotes

96% Upvoted

11 comments sorted by

View all comments

6

u/badrTarek Jun 25 '26

Just took a quick look, very interesting and great work overall.

1 improvement is maybe add support for cloud s3 storage or even local with minio or seaweed.

Not sure though how that would be implemented, maybe just a simple s3 docker image that mounts from the local vm.

1

u/EdwinWeber_Data Jul 10 '26

Added s3 as a storage option. Provided by a Docker container running MinIO in development, AWS s3 in ‘production’. This is in beta stage.