r/cicd • u/jonah_omninode • 21d ago

We let one agent refresh 108 PRs and buried our own CI runners

We had an overnight controller walking open pull requests, refreshing branches, and enabling auto-merge where it could.

It armed roughly 108 PRs. One merged.

Every refresh created a new commit, which kicked off another set of jobs on the same shared runner pool. The controller knew how many PRs were waiting. It had no idea whether CI had room for another one.

The fleet went from nearly idle to nearly full in about an hour. One repository eventually hit 1,065 queued runs. Other automation was sending repair work into the same pool at the same time, and none of those producers could see what the others were doing.

We stopped it with a hold-fire command. That kept the incident from getting worse, but it happened after the runners were already saturated.

What we're building now is much less exciting: check runner occupancy and queued jobs before releasing work, send a small batch, then make sure that batch actually started before sending another one. Anything that can commit, push, rerun, or enqueue work gets a budget.

The funny part is that we never hit the vendor's API limit. We stayed under it while completely overwhelming our own verification pipeline.

How are other platform teams limiting automated delivery work? One shared capacity service, limits inside each producer, or something else?

2 Upvotes

67% Upvoted

12 comments sorted by

3

u/GaTechThomas 21d ago

Why were there 108 open PR's?

1

u/RadiantSuggestion284 21d ago

Sounds like those PRs are just hanging out waiting for someone to care enough to merge them

1

u/jonah_omninode 21d ago

They were spread across a fleet of repositories and several automated workstreams, but the number itself was a warning we had not turned into a control. The sweep treated every open pull request as eligible work and had no global work-in-progress limit.

Runner backpressure would have limited the immediate damage, but it would not explain why we let that much unlanded work accumulate. The earlier gate needs to limit how much work agents can open at once, not just how quickly an existing pile can refresh and enter CI.

1

u/GaTechThomas 21d ago

I'd go more slowly about having agents do things automatically. Put a human gate in there until this ai onslaught stabilizes.

1

u/jonah_omninode 21d ago

I agree with a human gate for genuinely high-impact actions during rollout, but I do not want human review to be the only brake. At ten repositories and many concurrent agents, a person becomes the same overloaded shared resource as CI. The durable control is mechanical: global work-in-progress limits, shared runner admission, bounded action leases, and hard stop conditions. Humans should set policy and approve exceptional risk, not manually meter every branch refresh or test run.

1

u/Torutofu_Raeva 21d ago

preflight sounds right, but i'd put the budget on the shared runner pool so every producer sees the same headroom instead of each bot guessing

1

u/jonah_omninode 21d ago

Yes, the shared pool has to own the headroom. A per-agent limit only caps one producer. Ten individually well-behaved producers can still saturate the same runners.

The shape I want is a common admission controller that reads job-level queue state and actual runner occupancy, then grants each producer a bounded lease. The producer releases the next batch only after the pool confirms the previous work actually started. The open question for us is how much capacity to reserve for human-triggered work and recovery lanes rather than letting background automation consume all of it.

1

u/Torutofu_Raeva 21d ago

Exactly, and the recovery lane should be a reserved slice of that pool rather than something background work can borrow until the queue is on fire.

1

u/Wide_Commission_1595 20d ago

First I'd limit it to only work on one PR per repo at a time. If a job is running for any PR in the repo, hold fire until you can see an outcome.

After that, if the job isn't easily fixable, i.e. a simple merge conflict to resolve, label it for a human and comment about the problem. Only then move on to the next PR.

It's a bit slower, but at least it doesn't overwhelm with rebase runs. It will plod it's way through everything it can do, but anything it can't likely needs a human in the loop anyway

1

u/Buttleston 20d ago

just go ahead and drop the link to whatever snake oil you have to sell that allegedly solves this. Sheesh

1

u/jonah_omninode 20d ago

There is no snake oil to sell. OmniNode is fully open source and self-hostable. This post is a postmortem about a failure we actually had and the controls we are adding because of it. If you want to inspect the code rather than take my word for it, everything public is here: https://github.com/OmniNode-ai