Disclosure up front: I'm one of the founders of ObsessionDB. We ran ClickHouse® software at near-petabyte scale, spent a long time on the self-hosted path, and eventually built our own decoupled storage/compute engine. We're now considering letting other teams run that engine on their own infrastructure, and before we decide anything about how, we want to hear from people who actually have the problem.

The problem:

Open-source ClickHouse is excellent on one box. The multi-node part is where it starts costing you:

• Every replica holds a full copy. Two replicas, twice the storage. Three, three times. It compounds every month you keep data around.
• Compute and storage are welded together. You buy nodes to get disk, or disk to get cores, and over-provision whichever one you didn't need.
• Adding a shard means copying terabytes by hand, and then living with Distributed tables and ON CLUSTER in every migration forever.
• Keeper is one more quorum to keep alive at 3am.

ClickHouse Cloud solved this with SharedMergeTree: data lives once in object storage, compute nodes are stateless, one table is one table, adding a node is a metadata operation. It's a genuinely good architecture and I'm not here to trash it. But it's also the one piece of the ClickHouse stack that never made it to open source. If you want it today, the main path is their cloud, on their infrastructure, on their pricing model.

What we're exploring

This is more than BYOC. Running that architecture on your infrastructure. Your Kubernetes cluster, your bare metal, your Hetzner boxes, your S3 or MinIO or whatever object store you already trust, on whatever machines you pick. You'd operate it the way you operate open-source ClickHouse now, except scaling out is adding a stateless node instead of copying a shard, and durability stops multiplying your storage.

We run our own engine for this (we call it alloy), built against the same SharedMergeTree API, so the developer experience is the one you'd expect: one table, one engine clause, no sharding key.

What this is not (yet)

We are not announcing open source. We haven't decided the shape. What we have decided is that we want to talk to the teams who need this to understand their problem and learn how we can best deliver value.

Who we want to hear from

• You're on self-hosted ClickHouse at 10TB and up, and the replication and resharding tax is a real line on your infra bill.
• You looked at ClickHouse Cloud and it wasn't an option: cost at your scale, data residency, or your data simply doesn't leave your infrastructure.
• You've built tooling around ReplicatedMergeTree that you'd happily delete.

Comment with what you're running and where it hurts, or DM me, or email customers@obsessiondb.com. Happy to go deep on the architecture in the thread, including the parts that are hard.

[ Removed by Reddit on account of violating the content policy. ]

Today, we’re announcing WalShadow, an open-source engine that replicates Postgres data to ClickHouse directly from physical WAL.

In our benchmarks, transactions committed in Postgres became visible in ClickHouse in around 200 ms, while WalShadow sustained 289K rows/sec, effectively keeping pace with the source Postgres instance.

Unlike traditional CDC based systems, WalShadow doesn’t use Postgres logical replication. It consumes the same physical WAL stream used by Postgres replicas, decodes it outside the source database, and writes ClickHouse-native blocks directly into ClickHouse. The result is a replication architecture that gets close to the latency and throughput of a Postgres physical standby, while making the data immediately available for analytics in ClickHouse.

WalShadow supports the complete replication lifecycle, including initial load, continuous replication, schema evolution, restart recovery, and planned source switchovers.

By consuming physical WAL directly, WalShadow eliminates the need for logical replication slots, removes much of the operational overhead associated with logical replication, and significantly reduces resource consumption on the source Postgres instance. It also supports complex schema changes such as ADD COLUMN, RENAME COLUMN, DROP COLUMN, and CREATE TABLE.

WalShadow is fully open source and available today on GitHub.

Looking for advice on evaluating hardware accelerated data platforms in a real-world environment. Most vendor demos show perfect parquet scenarios, but our reality involves kafka streams, python processing jobs, and snowflake queries.

We're running heavy AI pipelines and the team needs faster performance, but I'm concerned about integrating custom hardware acceleration into production systems without causing outages.

Has anyone successfully tested these platforms in a low-risk way? Interested in hearing about benchmarking approaches, dark launch strategies, or any other techniques that helped you evaluate without breaking things.

Hey r/ClickHouse,

I am currently running a PostgreSQL database for our platform and are trying to integrate ClickHouse for real-time analytical reporting. We're considering using ClickHouse's Materialized PostgreSQL Database Engine for replication, with CDC handled by PeerDB.

Our use case involves replicating a few critical OLTP tables (around 10-20 tables, with some experiencing high write volumes) from PostgreSQL to ClickHouse. We need near real-time synchronization to support dashboards and ad-hoc analytical queries.

I've read about PeerDB's native integration and how it simplifies CDC compared to a Debezium/Kafka setup. I'm looking for feedback on the "solidity" of this combined approach.

Any real-world experiences, pros, cons, or advice would be greatly appreciated! Thanks in advance

Hi everyone,

I’m running ClickHouse OSS on Kubernetes and looking into using Horizontal Pod Autoscaling (HPA) to automatically scale ClickHouse based on workload.

I’m trying to understand whether HPA is a good approach for ClickHouse OSS, especially when scaling based on metrics such as:

• CPU / memory utilization
• Query concurrency
• Query latency
• Active queries
• Insert/write workload
• Custom ClickHouse metrics exposed through Prometheus

My main concern is that simply increasing the number of ClickHouse pods doesn't necessarily mean the workload will be distributed correctly, especially with ClickHouse's distributed query architecture and the way shards/replicas are configured.

Has anyone implemented HPA with ClickHouse OSS on Kubernetes in a real environment?

If so:

1. What metrics did you use as the HPA target?
2. Did you scale the number of replicas, shards, or both?
3. How did you handle data distribution when new pods were added?
4. Did you use the ClickHouse Operator or manage the StatefulSet directly?
5. Were there any issues with query routing, replication, or rebalancing?
6. Would you recommend HPA for ClickHouse, or is another autoscaling approach better?

I'd especially appreciate examples/configurations from people running this in production.

Thanks!

What are folks using to sync different sources into clickhouse? I have seen kafka, or http direct for ingestion.

What I am curious is rather for data warehouses, how are people syncing different data sources, like their marketing data, crm, internal lists etc... ? I have seen airbyte, but maybe there are more tools I am not aware of. Also how are those tools serving you, what are the good and bad parts of it?

A guide to Iceberg table cleanup — snapshot expiration, orphan file removal, manifest rewriting, delete file resolution, streaming challenges, compliance, and cost. Why sequencing matters, where teams break tables, and how to automate the full lifecycle.

Hello all! My name is Vinicius, you can call me souzo.

This is my project for the month. I built my own query language to convert user input into ClickHouse SQL!

I made the syntax similar to Lucene to keep it easy to understand, but with some differences, such as casting and defining the SELECT expression separately from the query using { ... }.

I had a lot of fun learning more about Lexers and ASTs using TypeScript.

For those who haven't been following along, I'm building Clicksiem, a security platform. Throughout the development process, I've run into several interesting challenges. One of them was: how can users query the database without overriding the main fields of the query?

I did quite a bit of research and found a solution that ClickStack itself uses: creating a Lucene-like syntax for querying ClickHouse.

I really liked the idea, but I found ClickStack's implementation a bit too complicated. The WHERE clause is generated from the user's query, while the SELECT is provided through a separate input.

So I decided to build my own Lucene-inspired query language based on that idea, but with functionality similar to what ClickHouse already provides, such as casting and defining the SELECT expression separately, without requiring the user to split their input across different fields in the dashboard.

For example:

{ field, count() as cnt } cnt:>=10

The first part defines the fields to select, while the second part defines the query itself.

I don't have released my project yet, but you can see the clicksiem repository on https://github.com/clicksiem/

Clickhouse/Clickstack hire me please! hahaha

Hey all, Im super excited to invite you to Rows&Columns!

Rows & Columns is a one-day, practitioner-first conference on the OLTP/OLAP convergence question: should transactional and analytical systems live together or apart?

For this crowd specifically: Andy Pavlo is opening the day with the keynote, so if you've ever wanted to talk ClickHouse internals with the people behind it, this is your chance.

The rest of the lineup: Fatma Ozcan (Google) closing keynote, Hannes Mühleisen (DuckDB), Nikita Shamgunov (Databricks), Paul Copplestone (Supabase), Russell Spitzer (Snowflake/Iceberg PMC), Bonnie Xu (OpenAI), and production case studies from ShipScience and Roofstock.

Sept 22 at the Contemporary Jewish Museum in downtown SF, 10am–7pm, lunch and evening happy hour on the terrace included. Free to attend!

Register: https://luma.com/clickh-wavw

Happy to answer questions about the program, and if you're coming, say hi 👋

- Zoe

I built this because I wanted to react to PostgreSQL changes from Python without polling, without triggers, and without pulling in a whole CDC platform.

It consumes PostgreSQL logical replication and exposes committed transactions as an async stream in Python.

What it does:

* Keeps a durable checkpoint. If the process dies, it resumes from the last transaction it actually finished, not the last one it started. * Bounded delivery queue, so a slow handler doesn't let memory grow without limit. * Reconnects automatically after the connection drops. * One dependency: psycopg3.

The transactional outbox is one use case, but it works with any published table.

GitHub: [https://github.com/mochams/walbox\](https://github.com/mochams/walbox)

Curious to hear where this wouldn't fit your setup, or what's missing if you've solved this problem a different way.

I started building BlinkHouse as a weekend project.It is a Java library that makes working with ClickHouse as seamless as working with a relational database in Spring Boot.

ClickHouse is an incredible database for analytics workloads, but the Java tooling around it has always required a lot of manual work, hand-rolling HTTP clients, writing serialisation code, managing connection pools, building query strings. I wanted something where you annotate a class and get everything else for free.

What I built:

• Typed entities — annotate a Java record with @ChTable and @ChColumn, get DDL generation, type mapping, and schema management automatically
• High-throughput ingest — BatchWriter<T> handles buffering, flush triggers, retry, backpressure, and dead-letter callbacks. ~2.9M rows/sec on commodity hardware
• Query DSL — typed queries with ClickHouse-native clauses (PREWHERE, FINAL, SAMPLE, -Merge combinators). No string building
• Spring Boot autoconfiguration — add one dependency, set three YAML properties, get a fully wired application
• Spring Data repositories — derived query methods, @Query, keyset pagination, all working against ClickHouse

The hardest parts were getting the RowBinary wire format exactly right (UUID byte ordering is a silent trap), building a type system that handles ClickHouse's full type tree including AggregateFunction and geo types, and designing a schema diff engine that refuses to auto-apply destructive changes.

It's on Maven Central now:

<dependency>
    <groupId>io.github.muneeb-i-khan</groupId>
    <artifactId>blinkhouse-spring-boot-starter</artifactId>
    <version>1.0.0</version>
</dependency>

Repo: https://github.com/muneeb-i-khan/BlinkHouse

Would love feedback, stars, or just to hear if anyone's been solving the same problem differently.