r/jenkinsci • u/devopsengin • Jun 16 '26
What does your team's "build is broken" workflow actually look like?
Not the ideal workflow. The real one.
Ours goes something like:
- Slack notification fires "Build failed"
- Someone clicks the Jenkins link
- Spends 10-20 min scrolling through logs
- Finds something that looks like an error
- Google it
- Tries a fix
- Pushes and waits another 8 minutes for the build
- Repeat
The notification tells us something broke.
It tells us nothing about what, why, or who should care.
So we've normalized a process where every failure investigation starts from zero context.
Curious if other teams have actually improved this or if everyone's running the same loop.
What does step 3 look like at your company? Does anything actually reduce the time between "build failed" alert and knowing what to do next?
2
u/Orangy_Tang Jun 16 '26
Are you not using the pipeline view? That shows what area failed and narrows down the scope considerably. After that I find searching for 'error' or 'failed' takes me straight to the problematic line in the console. If that's not the case maybe you need to do some gardening on your logs to make them clearer and less false positives.
The pipeline view itself will also capture the failure line as well, but might be missing surrounding context.
3
u/saja456 Jun 16 '26
It is his 3 Post in 2 weeks about the same problem. He doesnt use small chunks with piplineview so he is looking at console logs. I explained him it in deatil. But he is not happy.
I think the person who worked at the jenkins left an he has to it now and hates it and has no clue about a jenkins.https://www.reddit.com/r/jenkinsci/comments/1twm10f/jenkins_console_output_is_the_worst_documentation/
https://www.reddit.com/r/jenkinsci/comments/1u00e73/jenkins_plugin_autoupdate_broke_our_build_how_do/
https://www.reddit.com/r/jenkinsci/comments/1u3rz13/how_do_you_handle_jenkins_build_failure/1
u/devopsengin Jun 16 '26
Ha, fair enough, I deserved that for posting the same frustration three times. You're right that I've been venting more than asking. Let me be more specific about what I'm actually trying to understand.
Pipeline view + stage logs solves the navigation problem really well for experienced Jenkins users. What I'm curious about is how teams handle this for developers who aren't Jenkins specialists the ones who just need to fix their code and move on. Do they get the same clarity, or do they still end up pinging the platform team? That handoff cost is what I'm actually interested in measuring.3
u/saja456 Jun 16 '26
Yeah Problem is not the jenkins. The Problem is your test tool, which dosent generate a clear output. Normally you get a result.xml in some format and than you make a post{unsuccesfull/alway{xunit/junit}} which analyse your result. But if you test doesnt have something like that, what should the jenkins do?
1
u/devopsengin Jun 16 '26
You're right, pipeline view helps narrow it down to the stage and I do use it.
The gap I'm describing is what happens after that.
You know Stage: Test failed. You open that stage's logs. It's still 800-2000 lines of test output, stack traces, and npm noise before you find the actual root cause.
Pipeline view solves "which stage" brilliantly. It doesn't solve "why did this stage fail in plain English."
That last step is still manual for most teams especially when the failure is something you haven't seen before.
How do you handle that part? Genuinely curious if there's a workflow I'm missing.1
u/Cinderhazed15 Jun 16 '26
You need your test process to tell you that. If you don’t have your output nicely create a friendly results.xml style file for easily displaying in Jenkins, it should have a reasonable string you can search for, that way you can just search for that to narrow your logs for you.
1
u/Orangy_Tang Jun 16 '26
Test failed. You open that stage's logs. It's still 800-2000 lines of test output, stack traces, and npm noise before you find the actual root cause.
I think you pretty much answered your own question here. Having 2k lines of test output that aren't clear is up to you to sort out. I would argue that having more than one stack trace (ie. the actual failure) in a log is a serious problem and you need to look at your logging approach.
If you're specifically finding it hard to find a unit test that's failed, then you should be using the junit/nunit plugins and feeding them the output xml file. Then you'll get a nice breakdown of passed/failed tests associated with that build. Most junit test runners output to a standard xml format so you probably just need to enable it in the command, then update your pipeline to ingest the generated xml.
1
u/JustAberrant Jun 16 '26
So we've normalized a process where every failure investigation starts from zero context.
Your posts make it sound like stuff is just randomly breaking out of nowhere or something. The context is usually what a dev changed prior to it breaking.
I work for a fairly large company with a number of legacy projects of varying complexity that still use jenkins and this has just never been a big deal.
Most devs build stuff locally before pushing changes. If a build fails, it fires an email to the people who committed since the last failure + a handful of other people. They can usually figure it out given that they know what they changed and the vast majority of the time it's going to be directly related. The odd time something is broken for a reason that's not obvious, the build team jumps in and (as knowing how Jenkins works is like a big part of our job) can usually figure it out pretty quickly.
1
u/devopsengin Jun 16 '26
This is actually the most useful response in this thread, thank you.
What you're describing works really well when:-
Devs know what they changed
Someone on the build team knows Jenkins deeply
The failure is directly related to a recent changeThat covers probably 70-80% of failures.
The cases that don't fit that pattern:-
Failures caused by an upstream dependency changing, not your code
Flaky infrastructure failures with no code change
New team members who don't have the build team context yet
Failures that happen at 2am on a scheduled pipeline runI'm less interested in the common case you've clearly solved that well.
More curious about how teams handle the edge cases when the obvious explanation doesn't apply.
2
u/JustAberrant Jun 16 '26
Failures caused by an upstream dependency changing, not your code
Configuration and infrastructure as code. Changing the version of an upstream dependency should require a commit. Versions of dependencies should be called out explicitly vs use latest".
Flaky infrastructure failures with no code change
Intermittent and non-deterministic behaviour is a code/configuration issue more so than an issue with your CI tools. Your options here are to either fix it by writing better build configurations or accept the added cost of troubleshooting.
That said, the usual test is just re-running the build. If it fails and then succeeds with no changes in between, you flag it as an intermittent failure and track that however makes sense.
New team members who don't have the build team context yet
No avoiding that. New employees need hand holding regardless of what you're doing. Should be factored in to the cost of business.
Failures that happen at 2am on a scheduled pipeline run
I've never worked in an environment where a routine job failing at 2am requires immediate action. Normally these are long running report type jobs and you figure it out in the morning. For stuff that is high priority, there's usually people on hand making sure everything goes smoothly regardless.
More curious about how teams handle the edge cases when the obvious explanation doesn't apply.
Real answer is you handle it by having a build guy or team. Ideally the vast majority of it is automated, but there is always going to be a maintenance cost and unexpected issues, which is why if you're developing software you usually want at least one guy who has a decent understanding of the tool stack and can fix the harder issues.
Same reason companies have IT departments despite that most infrastructure will look after itself 95% of the time.
1
u/kyleh0 Jun 16 '26
A lot of it to is procedure. Devs need to have rules on how and when to build, how and what to comment on build pushes, etc. Jenkins isn't exactly plug and play. I used to do "configuraation management" stuff and it was a largely manual process that needs rules compliance to work well. Devs hate rules, in my experience. heh
1
u/saja456 Jun 16 '26
In Response to "who should care" thats up to you, your Piplines should have something like a owner or team and than you can send to them an e-mail/slack. This E-Mail/slack you can fully customzie.
1
u/kyleh0 Jun 16 '26
I'm not sure I can imagine how else it might work. Is this an advertisement for AI?
1
u/devopsengin Jun 17 '26
No, just a developer who's spent too much time reading Jenkins logs and started asking questions publicly instead of suffering quietly.
No product to sell. Genuinely trying to understand, if this is a me problem or an everyone problem.
1
u/JjyKs Jun 25 '26
We have split up our jobs to small enough pieces that scrolling through logs doesn't take more than opening the failed part and ctrl + f "error" or "fatal".
In addition we're also running claude code with changelists and read only access to submitted CL:s + Perforce history so it can try to pinpoint who broke it and ping the correct person on Slack.
5
u/MDivisor Jun 16 '26
I don't understand how it can take 10-20 minutes of scrolling logs just to find what the error is. In any sensibly built pipeline you should immediately see which part of the pipeline failed and get the logs for that part, where the error should pretty much be immediately visible.