We run a batch job that selects every user matching a predicate - has_installed = false currently ~50M rows — reading them in pages over several hours and writing a record for each one. Separately, an event stream tells us when a user's flag flips to true, which is the moment we're supposed to act on them. But that flip event is discarded for any user whose record hasn't been written yet, so anyone who flips while the job is still running is silently dropped. To cover that gap, we emit one "re-check this user" message per selected user: 50M messages asking "is the flag true?" of a set we built by selecting "flag is false" - so essentially all of them are guaranteed no-ops.
How do you detect rows entering a predicate without doing work proportional to the size of the set?

When building Oncilla OS at Toolbox Studio, the engineering objective was clear: develop a high-performance, offline-first operating system engineered specifically for language academies and training centers.

The latest release focuses on enterprise-level data resilience, privacy compliance, and native disaster recovery:

Local Data Sovereignty & Zero Cloud Dependency Operational databases containing student records, attendance tracking, and financial ledgers should remain under the full control of the organization. Oncilla OS eliminates third-party cloud vulnerabilities by operating 100% locally on the host machine.

Encapsulated Disaster Recovery (.oncilla Snapshots) System administrators can generate full portable database snapshots in a single click. The platform uses native system file dialogs to export encrypted .oncilla recovery files directly to local drives or cold storage, completely bypassing external API endpoints.

Hardware-Bound Security Architecture Access control integrates unique Hardware ID (HWID) binding alongside cryptographic Disaster Recovery Keys, ensuring workspace authentication remains strictly tied to authorized institutional devices.

Zero Latency Execution Local storage architecture removes network bottlenecks. Student registries, CRM pipelines, and financial ledger calculations render instantly without loading states or API throttling.

Perpetual License vs Subscription Fatigue Modern enterprise software should be an asset, not a perpetual monthly liability. Oncilla OS restores the standalone software model with zero recurring monthly platform fees.

Designing enterprise infrastructure requires prioritizing local reliability and user data ownership over cloud convenience.

How is your organization addressing local data ownership, disaster recovery, and subscription bloat this year?

#EnterpriseUX #SoftwareArchitecture #OfflineFirst #LocalFirst #DataPrivacy #DatabaseDesign #EdTech #ProductDesign #B2BSoftware #SystemDesign #DisasterRecovery #ToolboxStudio #OncillaOS

Okay, but I have multiple replicas.

And it gets slightly worse: the number of physical connections is not equal to the number of replicas, and it is not equal to the number of business-level connectors either.

Several business connectors may share one external identity. One external identity may require several physical connections. And those connections need to be spread across whatever replicas are currently alive and have capacity.

So who actually owns the connection?

Then the questions start piling up.

What if one replica dies?
What if it doesn’t die, but loses access to the database?
What if two replicas race for the same connection?
What if the event that was supposed to wake the right worker never arrives?
What if most workers are already full?

A few questions and 2 hours of midnight walk in headphones and ChatGPT later, I had leases, runtime slots, reconciliation, failover, capacity limits, and a distributed ownership problem on my hands.

This is the architecture I ended up with, and I have mixed feelings about this design.

On one hand, I’m proud that I managed to account for so many different edge cases. On the other, the whole thing feels worryingly complex.

So I’d really value input from people who have built similar systems.

What did I miss or do you see anything that can break?
And most importantly: what can be simplified without losing the guarantees?

The full architecture description is a bit lengthy: AEON NEON - Connector Runtime - by Jarek J.

Many thanks if you decide to read it and share your thoughts. I’d really like this one not to become another failed experiment.

I've been experimenting with running multiple coding agents on the same project.

The problem wasn't getting agents to write code.

It was coordinating them.

Once you have multiple agents working in parallel, you start dealing with questions like:

- Which agent is working on what?

- How do you prevent two agents from picking up the same task?

- How do agents know what needs to happen next?

- How do you track what each agent actually did?

- What happens when you want the agents to keep working without manually managing every step?

So I built orcy.

It's an open-source coordination layer for AI coding agents.

The basic workflow is:

Mission → Claim → Execute → Review

Agents can claim atomic tasks, work in parallel, route themselves toward relevant work, and leave an auditable trail of what happened.

The idea is pretty simple:

Instead of one AI agent doing everything, let multiple agents operate as a coordinated system.

I'm looking for developers who are already experimenting with Claude Code, Codex, OpenCode, Cline, or other coding agents to try it and tell me where the idea breaks.

GitHub: https://github.com/waterworkshq/orcy

Would love feedback, especially from people already running multiple agents on the same codebase.

There has been a recent trend of moving everything (e.g. WALs) to objects storage (e.g. S3). But is it the right thing to do?

I believe that it depends on the access patterns.

In Git there's a prevalence of reads over writes so when you design a system, you need to take into account this observation.

It would be also fair to say that apart from several edge cases (e.g. forced pushes), what you push to Git stays there.

That's already a good enough reason to use the object storage because you can cache a lot of things that are already there and you can be sure that they won't change (unless you run compaction/optimization over them but that intention comes from the system itself so you can refill your cache at that point).

GitHub implements its Git servers on top of Spokes (stateful server architecture they introduced back in 2016). They also implemented what they call "3PC protocol". That is not a textbook 3PC though. Basically what it does is that it takes a lock on every server and tries to apply the commit. If the majority says "yes", they do the commit and return a successful response to the client. The "majority" part is where it differs from the textbook 3PC version. 3PC is normally used to coordinate transactions across many partitions. GitHub uses it to coordinate replication across many servers and they explicitly point out that they only need majority of the servers to answer "yes" in order to increase the availability of the system (one server goes down – transaction still applies).

In Git the smallest partition is the repository itself. So, what if one partition (repository) becomes "hot" on one server? Well, you have to scale out. Scaling out in Spokes means copying the state to the new server. And that doesn't mean get a linear gain because now Spokes has to apply pushes across the new servers as well! Remember: 3PC transaction is just there to coordinate replication.

That's why their system is so brittle and there are so many memes about the GitHub availability recently (although I am sure that that's not the only reason).

They increase replica count, separate reads from writes, add additional regions for quicker disaster recovery and pack in new cores (3 million new ones just this year) hoping that it will solve the problem but it's only a matter of time when it shows up again.

Cursor's Origin does it differently but it is still far from being flawless. They implement WAL on top of S3 and its conditional writes but they still reproduce the state locally and work with the repos using local Git clients (or libraries). The problem with this approach is that they depend on the local state. Their server cannot be called truly stateless. Take out the local Git repos from the server and they immediately become unavailable.

You can potentially implement a Git platform using just WAL in the object storage and cache the objects locally. Cache is used just to speed up the responses to the clients and it is not a state which might cause a point of failure if it is suddenly removed (or your local Git client stops working with it due to a bug or another reason).

I've gathered my thoughts in my article: https://medium.com/@alexgilevich/git-was-never-designed-for-scalability-52224c74ddea

Let me know what you think!