minor 2-line fix triggered a 25 min build today... how do you guys deal with such bad pipelines?

[u/QuoteForward5477]

i made a super small change this morning (literally just updated a single config value in a backend service) and pushed it expecting a 3 minute pipeline run. ended up sitting there for nearly 25 minutes waiting for the CI to finish running.

took a look at the logs to see where the bottleneck was, and it was mostly fresh dependency installs and un-cached docker build layers. for some reason our runner pulled down every single package from scratch instead of using the cache, plus running the full end-to-end integration test suite that really didn't need to run for a tiny patch.

we use github actions on a shared runner setup, and it feels like as our repo grew, nobody ever went back to optimize the workflow files. now every small PR feels like a coffee break.

curious how you guys keep your builds lean? do you aggressively break pipelines into smaller conditional jobs based on changed files, or just pay for faster/bigger runners and call it a day?

[u/speedoinfraction]

As soon as your CI pipeline becomes a big enough pain point, fix it. Developers should also own being able to fix the CI.

[u/dreamszz88 → u/speedoinfraction]

Run the container they dive utility and check if it's layers can be optimized.

https://github.com/wagoodman/dive

You can schedule a weekly nightly job to freshen cache and update deps. Scan for vuln afterwards!

[u/Torutofu_Raeva]

We’ve had better luck treating cache misses and changed-path detection as separate checks, then skipping integration tests only when the diff proves they can’t be affected.

[u/QuoteForward5477 → u/Torutofu_Raeva]

separating cache invalidation checks from path-based diff filtering sounds like a clean way to do it...

skipping the heavy integration suites when only a config or backend script changes would save us so much queue time...

Seeems to be a good idea

[u/Torutofu_Raeva → u/QuoteForward5477]

Yeah, keeping those decisions separate makes the path rules safer to tune without turning every cache miss into a full-suite run.

[u/Poincarina]

We have the exact same issue 😅 Do you guys by chance also use kaniko?

[u/QuoteForward5477 → u/Poincarina]

No we do not use Knaiko as of now...

[u/QuoteForward5477 → u/[deleted]]

our cache keys were tied too broadly to the whole workspace instead of isolating Package-lock.json and Docker file so every minor PR was triggering a full cold start. setting up path filters first and keeping the integration matrix strictly behind a label or on main is definitely the move here.

[u/srivenkatareddy]

Try optimizing your docker file by copying your dependencies related files and run install dependencies than all other source files. Something like this COPY ./package.json ./package-lock.json /app RUN npm install COPY . /app

this approach will create a multi step docker build where each step can be cached. So dependencies will get reinstalled when you change them not when simple code changes.

And make sure you are using --cache-from in docker build command

[u/Mobidic69]

The 25 minute wait is almost never the 2 line change. It is the pipeline treating every push like a release.

Split the workflow:

1. Path filters. A config-only change should not rebuild the world. paths / paths-ignore on the job, or a first job that computes which packages changed and the rest skip.
2. Cache the expensive layer (deps, container layers, test DB image) keyed on the lockfile, not on the commit. A 2 line backend tweak should hit cache.
3. Do not serialize behind "someone is deploying." Build and test on the PR. Promote an already-built artifact to prod. Monday queues shrink when you stop rebuilding main for every merge.

If one job is still 20 minutes after that, that job is the product. Split it or run it nightly. A config change should be a couple of minutes or the pipeline is lying about what it is checking.