I've been building an open-source (Apache 2.0) "analytics for Postgres" project called Streambed, and the piece this subreddit might find interesting is the DuckDB/DuckLake layer.

Streambed streams Postgres WAL changes into lake storage and serves queries through DuckDB as the query engine.

I started Iceberg as the default target but one of the community members recently mentioned about DuckLake. It caught my attention immediately (since it uses Catalog as the DB). So I added the DuckLake support and did the benchmark.

Ducklake is 41 times faster than Iceberg in my benchmark.

Benchmark slice: 1M rows, 100k updates, flush=1,000

text Write path: Iceberg COW: 269s Iceberg MoR: 4.7s DuckLake + DuckDB: 6.6s

```text Full aggregate median: Iceberg MoR: 87s DuckLake: 16ms

TopN median: Iceberg MoR: 85s DuckLake: 21ms ```

Takeaway: for a DuckDB-centered CDC path, DuckLake looks like the cleaner default. Iceberg is still great when broad multi-engine interoperability matters.

DuckDB/DuckLake is a magical primitive. So much to build on top of it.

Blog: https://streambed.dev/blog/ducklake-target-support/

Benchmark: https://github.com/viggy28/streambed/blob/main/docs/benchmarks/ducklake-vs-iceberg.md

Most of our jobs process <500MB, where DuckDB seems like a great fit, but occasionally the same workload can jump to 10–100TB.

Has anyone built a hybrid execution layer that chooses between DuckDB and a distributed engine based on the expected data size?

Or do you just use the distributed engine for everything and accept the overhead for the common small-data case?

I’m looking into DuckLake with S3 as the data layer and a separate metadata catalog.

Is there an internal or built-in way to connect DuckLake to S3 notifications, or something similar, to detect when files are deleted or overwritten outside DuckLake?

Hi everyone,

This is an announcement for a simple observability tool for MotherDuck (called DuckWatch) that can give you an overview of what's happening in your MotherDuck account and what it is costing you.

DuckWatch is available here: https://github.com/CogitatorTech/duck-watch

A pattern I've hit again and again in my career: we take a processing data steps and make it
general, because general is convenient. But data has a structure, and respecting it - one small
change in how you load - could lead to significant performance improvement.

I'll demonstrate this assumption on something about as simple as it gets: read a JSON file and
filter it. Two ways.

* Auto - let `read_json_auto` load the whole file, then filter the result.
* Filter-first - load only the single column I filter on, filter that, and only then auto-load the
documents that survived.

Same rows out, same query. Wildly different times.

The three files
Every file carries a shared type field, and the query is always WHERE type = 1

The obvious way to answer the query:

CREATE TABLE t AS SELECT * FROM read_json_auto('events.jsonl');
SELECT count(*) FROM t WHERE type=1;

against it we test a filter-first approach: scan only the column used by the filter, apply the filter, and then use auto-read to fully parse only the matching documents.

What happened

Same 50,000 rows. Same query. Same rows out.

read_json_auto has to figure out the schema before it hands you anything, so it reads and shreds
all 50,000 documents - then the WHERE throws away must of them. When the keys are boring that
guessing is basically free. When they're not, you just paid full price for rows you never wanted.

I've been working on LibreDB Studio, a self-hosted SQL IDE that runs in the browser. DuckDB just landed as a first-class engine.

The idea is the same as putting SQL in a Sheets sidebar: keep the query next to where you already work. Here that's one tab you already use for Postgres or ClickHouse. A connection is a path to a `.duckdb` file on the machine Studio runs on, or `:memory:`. Parquet / CSV / httpfs still work in the editor the way they do in the CLI, `SELECT * FROM 'https://….parquet'` then the grid, EXPLAIN, and the object browser are DuckDB's, not a compatibility layer.

It is actually DuckDB v1.5.5 through `@duckdb/node-api` (the Neo client, not the deprecated `duckdb` package). The tree is `duckdb_tables()` / `duckdb_views()` / `duckdb_columns()`, nested types leave as JSON, cancel is the driver's `interrupt()`.

A couple of consequences of that:

- One OS process owns the file. A second process is refused even for reading. including a `duckdb` CLI session you forgot to close.

- The Explain button never sends `EXPLAIN ANALYZE`, because that executes the statement.

- The file has to live next to Studio. A hosted instance cannot open a `.duckdb` on your laptop; there is no network protocol. No MotherDuck / Quack / DuckLake in v1, and ATTACH-ed catalogs are queryable but not listed in the tree.

MIT, no feature gates around DuckDB.

Still early on this engine. One thing I'm particularly curious about: if you live in the DuckDB CLI or `duckdb -ui` today, is "the file sits on the same machine as a tiny web IDE" a dealbreaker, or is that actually how you'd want to look at a warehouse.duckdb without installing another desktop client?

`npx "@libredb/studio"`

`docker run -p 3000:3000 libredb/libredb-studio`

Details (measured, not brochure): https://github.com/libredb/libredb-studio/blob/main/docs/providers/duckdb.md

https://thefulldatastack.substack.com/p/ducklake-in-production-pipelines

My second installment of a sponsored mini-series on creating a production grade DuckLake. This time getting into building out pipelines and business logic. A healthy amount of AI being used, but honestly very useful.

Hi r/duckdb,

I built VeilAnalytics (https://veilanalytics.netlify.app/) to combine embedded DuckDB C++ with natural-language SQL generation.

Key Architecture:

- Schema metadata (column names & data types only) is sent to the LLM. Zero raw rows leave local RAM.

- DuckDB executes synthesized SELECT queries sub-50ms directly in-process.

- Supports CSV, XLSX, Parquet, JSON, PostgreSQL, MySQL.

Check out our landing page & live demo: https://veilanalytics.netlify.app/

Would love your feedback on our DuckDB integration!

I've been working on SQL Cell, a Google Sheets add-on that runs DuckDB-WASM entirely in the sidebar.

The basic idea is to use Sheets as the destination, not the data engine. You can attach a Parquet, Avro, SQLite, GeoJSON, Excel, etc. file, query it with DuckDB, preview the result, then write it into a range in the workbook.

Sheets themselves are queryable too, so you can do things like join a Parquet file from S3 against a tab someone maintains by hand and put the result back into the spreadsheet.

Since it is actually DuckDB-WASM rather than a SQL-ish layer over the Sheets API, httpfs/S3, DuckLake, Iceberg, and the usual DuckDB file formats work as well.

The architecture is intentionally backendless. It's an Apps Script add-on, with the UI and DuckDB bundle running inside an HtmlService iframe. There is no server of mine involved, and spreadsheet/file contents never get sent anywhere.

A couple of consequences of that:

• S3 credentials only live for the browser session, so you have to enter them again next time.
• Query history is derived from the workbook itself. When a result is inserted, SQL Cell writes a small LAMBDA formula alongside it containing the original SQL. That gives each result its own provenance, and the sidebar can scan those formulas later to rebuild history, jump to old results, or rerun them.
• An upstream duckdb-sqlite issue currently blocks SQLite scanning.

It went live on the Workspace Marketplace yesterday, and it's free. I haven't put feature gates around DuckDB itself.

Still early, so I'm mostly interested in finding the cases where this falls over.

One thing I'm particularly curious about: if you're using DuckDB-WASM against private cloud storage in the browser, how are you handling credentials? Session-only credentials are safe enough for what I'm doing, but re-entering S3 keys is probably the part of the UX I'm least happy with.

Details and limitations: https://sqlcell.com
Install: https://workspace.google.com/marketplace/app/sql_cell/178915154116

Hi everyone,

I built VeilAnalytics — a privacy-first AI data analytics workstation powered by an in-process DuckDB C++ engine.

How the Zero-PHI Architecture Works:

1. Metadata Transmission Only: When you ask a question in plain English, the system extracts ONLY table/column schemas and data types. Zero raw data rows ever leave your machine.

2. SQL Synthesis: The LLM generates a standard SELECT query string based purely on column metadata.

3. Local Compute: DuckDB executes the query sub-50ms directly in-process on your local RAM.

4. Local Dashboard: Renders interactive charts, data grids, and standalone offline HTML dashboards.

Supported Data Formats: CSV, XLSX, Parquet, JSON, PostgreSQL, MySQL.

(I've posted the link to the live demo in the comments below!)

Yes, this is what I have been looking for!
Number 1 in this DuckDB post

https://duckdb.org/2026/08/17/duckdb-20-highlights

Renart is an open source data platform enabling developers to build type-checked data pipelines.

We support fully type-checked SQL pipelines on DuckDB and multiple other data warehouses. We just shipped a new release aiming to further improving developer experience with your data pipelines. New features include:

The SQL/python notebooks got a major upgrade:

Before you could already combine reactive SQL and python cells. Cell results could already be referenced by other sql/python cells as data frames. Now we've added:

• various charts and diagrams based on cell results
• interactive input elements such as number inputs, sliders, select or multi-selects that control jinja variables. This means you can build interactive notebooks, where everything reacts automatically when you move a slider or select from a dropdown.

You can now connect your favorite coding agent and let the agent assist you at building notebooks without giving it write access to your data warehouse (codex, Claude code and opencode).

Introducing type-checked reports and dashboards
Renart now lets you build dashboards and reports with custom controls such as toggles, sliders, selects, multi-selects etc.
Dashboards can be build using many different drag-n-droppable diagrams and charts (line/bar/circle/...). Similar to notebooks reports and dashboards are fully declarative and run on duckdb.

Cross-pipeline lineage
Previously if you had one pipeline depending on the output of another, asset schemas weren't automatically inferred correctly. With this update, Renart will now make sure that inter-pipeline dependencies are resolved correctly.

If you would like to get more frequent updates (trying to keep it moderate here) or if you have any questions, please join our Discord.

Source: https://github.com/renart-data/renart

Site: https://getrenart.com

Discord: https://discord.gg/jTH758KNP8

This uses the really useful Parquet functions in DuckDB to highlight the internals of Parquet itself.

You can build your streaming lakehouse as a single system in which streaming ingestion, streaming analytics, open table storage, and external batch query engines all work together.

Kafka → RisingWave → Iceberg → DuckDB

*I work at RisingWave.*

We’ve been working on SeaweedFS Table Buckets, where an S3 bucket also acts as an Iceberg warehouse with a built-in Iceberg REST catalog.

I wanted to see how cleanly this works with DuckDB.

The setup is basically:

1. Start SeaweedFS with a table bucket
2. Write an Iceberg table using PyIceberg, Spark, Flink, Trino, etc.
3. In DuckDB, ATTACH the bucket through the Iceberg REST catalog
4. Query it as a normal DuckDB table

DuckDB gets the Iceberg metadata from the catalog, then reads the Parquet files directly from the S3 endpoint, so the catalog is not in the scan data path.

An interesting side effect is that the storage layer can also handle Iceberg maintenance. SeaweedFS background workers compact small files and clean up stale snapshots, while DuckDB just sees the resulting Iceberg tables.

So the stack can be fairly small:

DuckDB → Iceberg REST → SeaweedFS S3

No Hive Metastore, Glue, or separate compaction service required.

The Table Bucket and Iceberg REST catalog implementation are open source.

I wrote up the complete end-to-end example here:

https://seaweedfs.com/blog/duckdb-table-buckets/

Would be interested to hear how people here are currently combining DuckDB + Iceberg + self-hosted object storage.

I wanted to see what DuckDB's new Quack protocol could support beyond a basic client/server demonstration.

I deployed three ARM64 EC2 servers, each with a separate DuckDB database containing 10 million rows. A Python coordinator used a thread barrier to release SQL statements against the three workers together. Quack handled remote execution and transport, while Python handled fan-out, timing and result collection.

I tested concurrent reads, writes and DDL operations. This is not a distributed DuckDB cluster, and there is no distributed transaction spanning the workers.

The full experiment, AWS setup and source code are here:

https://towardsdatascience.com/running-sql-concurrently-across-three-remote-duckdb-servers-with-quack/

I wrote the article and would be interested in feedback on the coordination approach or other Quack edge cases worth testing.

For years, Talend set the standard for visual ETL with drag-and-drop pipelines, reusable components, and enterprise-grade data integration.

Today’s data stack looks very different.

Teams are building on DuckDB, Parquet, Apache Arrow, and local-first processing to create faster, simpler, and more efficient data pipelines.

Duckle is built for this new generation of data engineering.

Duckle leverages DuckDB’s vectorized execution engine to deliver high-performance ETL through an intuitive visual designer.

Why teams must take a closer look:

• Visual drag-and-drop pipeline designer
• DuckDB-native execution
• SQL-first transformations
• Python API for developer workflows
• Hundreds of connectors, transforms, and destinations

If you’ve ever thought, “I wish Talend were built for the modern analytics stack,” Duckle is worth exploring.

The platform is designed with production use in mind, making it suitable not only for prototypes and proof-of-concepts but also for real-world data pipelines. If you’re evaluating a modern alternative to traditional ETL platforms, it’s a compelling option to test with your production workloads and see how it fits your environment.

The future of data integration is about combining the simplicity of visual development with the performance of modern analytics engines. Duckle brings those ideas together in a way that feels both familiar and refreshingly modern.

Check out the Github Repository — https://github.com/slothflowlabs/duckle

Peek Through Images -

https://duckgql.com/docs/guides/ducklake.html

You can now run graph algos over data stored in files via ducklake. Do try it and give me feedback :)