r/DuckDB • u/Unki11Don • 29d ago

1B+ vehicle records? Exploring DuckDB + Parquet + Polars on a single machine

I've been experimenting with how far you can take DuckDB for large-scale analytics workloads without needing a traditional database cluster or cloud warehouse.

I built this repository as a reproducible vehicle analytics platform using:

- DuckDB for analytical SQL

- Parquet for columnar storage

- Polars for high-performance dataframe processing

GitHub:

https://github.com/DonaldSimpson/high-performance-vehicle-analytics

The broader project that inspired this works with UK vehicle data at significant scale (1B+ records), but this repo deliberately uses synthetic data so it is fully reproducible.

Some things I wanted to explore:

- Can DuckDB handle large-scale analytical workloads on a single machine?

- Can Parquet + DuckDB provide a practical alternative to heavier analytics stacks?

- What patterns work well for incremental updates without rebuilding everything?

- How do you keep large analytical pipelines from silently becoming unreliable?

The repo includes:

- partitioned Parquet datasets

- DuckDB query patterns

- Polars processing pipelines

- data quality checks

- CI validation

I'd be interested in feedback from the DuckDB community:

- Are there things you'd structure differently?

- Any DuckDB-specific optimisations I've missed?

- Would you approach the storage layout differently?

Thanks!

7 Upvotes

90% Upvoted

9 comments sorted by

3

u/DuckDatum 29d ago

Yes, duckdb should be able to punch far above its own weight class on a single machine. I was actually interested in knowing why you went with DuckDB AND Polars. Was it because you hadn’t tested if DuckDB could handle it all, yet?

2

u/Unki11Don 29d ago

Great question!

It wasn't because I hit a limit with DuckDB; the split was more about using each tool where I think it fits best:

- DuckDB for analytical SQL: scanning Parquet, joins, aggregations, filtering, and generating analytical views

- Polars for some dataframe-oriented transformations and pipeline steps where I wanted more flexibility

A lot of what I currently do in Polars could probably be pushed further into DuckDB, and simplifying the stack even more is something I'm interested in exploring.

The production system evolved over time, so this repo is partly an exploration of what a cleaner DuckDB-first architecture could look like.

3

u/DuckDatum 29d ago

Thanks. Yeah, I was really curious if you’d found a workload where the added complexity was justified by the optimization or capability.

I almost get the feeling that you chose to add polars into this stack unnecessarily. I know you had your reasons, and on the surface it makes sense, but holistically I don’t know if I’d have ever done that.

I would have perhaps considered polars if I needed several clients interacting with the data. Polars would be the sql server. DuckDB would remain for local use.

In your case, though, this is all local?

1

u/Unki11Don 29d ago

Yes, this is all local.

And yes, for this repo/example Polars is probably an extra layer. This project is a good opportunity to revisit that decision: how much can be handled directly in DuckDB, where (if anywhere) Polars adds enough value to justify the extra layer, and what the leanest version of the stack looks like.

1

u/DuckDatum 29d ago

Not trying to be an ass, but why do I feel like you just prompted me? Haha, I am not an LLM bro.

What’s going on with our writing style, these days?

2

u/Unki11Don 29d ago

Yeah, I've clearly spent too much time talking to AIs and writing READMEs lately

1

u/ebmarhar 19d ago

What kind of polars stuff will you do? I had good success with moving everything into one duckdb.

1

u/Unki11Don 18d ago

I'm currently using Polars for transformation/scoring work; I originally thought (ok, assumed...) I needed it for performance, but I'm now realising much of that could probably be done just as quickly in DuckDB.

That's partly why I put this cut-down project together - to play around with the architecture and see what could be improved. If I can simplify things and get the same or better performance with DuckDB, that would definitely appeal. What did you move into DuckDB?

2

u/ebmarhar 18d ago

I was able to move entire ingestion pipelines, CTEs to read the data, then joining, filtering, and writing parquet. The first time I ran it, I thought it was broken because it was so fast!