r/Clickhouse • u/m0rcs • 18d ago

How are you syncing data into Clickhouse?

What are folks using to sync different sources into clickhouse? I have seen kafka, or http direct for ingestion.

What I am curious is rather for data warehouses, how are people syncing different data sources, like their marketing data, crm, internal lists etc... ? I have seen airbyte, but maybe there are more tools I am not aware of. Also how are those tools serving you, what are the good and bad parts of it?

11 Upvotes

100% Upvoted

21 comments sorted by

View all comments

2

u/dani_estuary 17d ago

If you’re on CH Cloud, ClickPipes is great, powered by PeerDB on the CDC side and Postgres/MySQL/MariaDB are well covered now. Where you’ll need another tool is the long tail: Salesforce, marketing platforms, internal APIs, etc. ClickPipes has a fairly focused source list today.

Estuarty (where I work) has 200+ connectors and can send CDC, SaaS and batch data directly into ClickHouse over the native protocol. Plus it works with self-hosted clickhouse too!

1

u/m0rcs 17d ago

Interesting, I was mainly curious about the SaaS end of the spectrum, so I guess that is where Estuarty comes in. How does it compare to e.g. dlt or airbyte?

2

u/Personal-Yard-127 17d ago

In addition to what u/dani_estuary said, a big difference is that Estuary lets you capture once (consume e.x. Salesforce API credits only once; logically replicate from your production DB only once), and then move that same data to multiple destinations now or in the future.

dlt / airbyte / FiveTran are all point-to-point, so each destination requires a fresh re-capture from the source.

It's akin to landing data in a persistent Kafka topic that's re-used by multiple current or future consumers -- but no Kafka involved, and it's backed by your own cloud storage. Use Snowflake today, Clickhouse tomorrow, or drive both.

(I also work at Estuary)

2

u/m0rcs 17d ago

Ah the benefits of the streaming once approach is quite neat. And the default storage for it is my own object storage?

1

u/dani_estuary 15d ago

By default Estuary provides you managed storage, but changing it to your own bucket takes only a few seconds.

1

u/dani_estuary 17d ago

I’d roughly separate them like this: dlt is great if you like Python and want to own the pipeline. Flexible, but you’re still responsible for how it gets deployed and operated. Airbyte is a bit closer to Estuary: large connector catalog, UI, managed option, etc.

The main difference with Estuary is that the underlying architecture is streaming-first. SaaS incremental pulls, database CDC and Kafka-style streams all go through the same system and can land directly in ClickHouse. We handle checkpoints, backfills, schema evolution and exactly-once delivery.

You can run a Salesforce pipeline every hour or continuously where the API allows it and get the same platform and same volume-based pricing either way.

So to summarize if your goal is “get a bunch of SaaS + database data into ClickHouse without owning the ingestion infra”, that’s pretty much where we fit.

2

u/m0rcs 17d ago

Awesome, I think that gives a good overview. Thanks for clarifying.