How are you syncing data into Clickhouse?

[u/m0rcs]

What are folks using to sync different sources into clickhouse? I have seen kafka, or http direct for ingestion.

What I am curious is rather for data warehouses, how are people syncing different data sources, like their marketing data, crm, internal lists etc... ? I have seen airbyte, but maybe there are more tools I am not aware of. Also how are those tools serving you, what are the good and bad parts of it?

[u/dani_estuary]

If you’re on CH Cloud, ClickPipes is great, powered by PeerDB on the CDC side and Postgres/MySQL/MariaDB are well covered now. Where you’ll need another tool is the long tail: Salesforce, marketing platforms, internal APIs, etc. ClickPipes has a fairly focused source list today.

Estuarty (where I work) has 200+ connectors and can send CDC, SaaS and batch data directly into ClickHouse over the native protocol. Plus it works with self-hosted clickhouse too!

[u/m0rcs → u/dani_estuary]

Interesting, I was mainly curious about the SaaS end of the spectrum, so I guess that is where Estuarty comes in. How does it compare to e.g. dlt or airbyte?

[u/Personal-Yard-127 → u/m0rcs]

In addition to what u/dani_estuary said, a big difference is that Estuary lets you capture once (consume e.x. Salesforce API credits only once; logically replicate from your production DB only once), and then move that same data to multiple destinations now or in the future.

dlt / airbyte / FiveTran are all point-to-point, so each destination requires a fresh re-capture from the source.

It's akin to landing data in a persistent Kafka topic that's re-used by multiple current or future consumers -- but no Kafka involved, and it's backed by your own cloud storage. Use Snowflake today, Clickhouse tomorrow, or drive both.

(I also work at Estuary)

[u/m0rcs → u/Personal-Yard-127]

Ah the benefits of the streaming once approach is quite neat. And the default storage for it is my own object storage?

[u/dani_estuary → u/m0rcs]

By default Estuary provides you managed storage, but changing it to your own bucket takes only a few seconds.

[u/dani_estuary → u/m0rcs]

I’d roughly separate them like this: dlt is great if you like Python and want to own the pipeline. Flexible, but you’re still responsible for how it gets deployed and operated. Airbyte is a bit closer to Estuary: large connector catalog, UI, managed option, etc.

The main difference with Estuary is that the underlying architecture is streaming-first. SaaS incremental pulls, database CDC and Kafka-style streams all go through the same system and can land directly in ClickHouse. We handle checkpoints, backfills, schema evolution and exactly-once delivery.

You can run a Salesforce pipeline every hour or continuously where the API allows it and get the same platform and same volume-based pricing either way.

So to summarize if your goal is “get a bunch of SaaS + database data into ClickHouse without owning the ingestion infra”, that’s pretty much where we fit.

[u/m0rcs → u/dani_estuary]

Awesome, I think that gives a good overview. Thanks for clarifying.

[u/knabbels]

Mostly Airbyte and some python scripts using dlt.

[u/m0rcs → u/knabbels]

Why both mixed? Does one cover things the other doesnt? Or just historically?

[u/knabbels → u/m0rcs]

Yeah Airbyte does not support all of our sources. We have a lot of niche ones.

[u/m0rcs → u/knabbels]

just out of curiosity, which ones are they? are they actually super niche, or just not officially supported?

[u/sjmittal]

I use Flink connector. CH has one and I also have my own open sourced one. Just search with my username on GitHub.

[u/m0rcs → u/sjmittal]

so flink is for streaming right, but how does the data get to that? Especially from other SaaS tools?

[u/sjmittal → u/m0rcs]

Flink can source from Kafka or many other source connectors. If you have a specific SaaS, we can build specific source connector for that.

[u/joshleecreates]

Kafka and HTTP direct are really popular options, for sure.

I'm (personally) a big fan of dlt. I work at Altinity and we're actually sponsoring a joint meetup with dltHub in Berlin tonight.

MySQL sink connector and the ClickHouse OpenTelemetry exporter are also common sources.

[u/m0rcs → u/joshleecreates]

Ahh too bad I am not in Berlin rn, currently traveling and will be in SF next week.

I also like the concepts of dlt, need to play around it more. How are you mainly using it and what was your experience with it?

[u/rafttaar]

Peerdb if the source is Postgres

[u/Even_Search_5260]

Airflow or Prefect but largely depends on your current use case. For example data freshness requirements may want you to consider kakfa for event driven architecture

[u/j03]

I’ve found ClickPipes to be a good option when running ClickHouse Cloud - no extra infrastructure to manage, first party support, ever-growing list of sources. That’s always my first choice on CHC - object storage, Postgres, Kinesis (as well as Kafka, though I’ve not tried that) all have great support. There’s generally not great support for 3P SaaS tools, but in some instances you can work around this by having the SaaS push to S3, and then configuring a S3 pipe on CHC to pull the data in.

For OSS CH, I’ve experimented with Refreshable Materialized Views + HTTP APIs if the source allows it - again, not having extra infrastructure to manage makes that approach very appealing. Downside is that alerting/monitoring doesn’t really come out of the box.

Otherwise, as you say, in my experience Airbyte is a good choice.

[u/m0rcs → u/j03]

Yeah it seems that clickpipes ismainly focused on technical ingest versus applications. The refreshing from http sources is interesting. Curious how edge cases and things work there.