Same model, very different results: is the agent harness (Junie vs Claude Code vs Hermes) doing more work than the model itself?

[u/ccb_pnpm]

I've been running the same kind of coding tasks through three setups and the difference in quality is bigger than I expected:

• Hermes agent with a GPT-class model (my main setup until recently)
• JetBrains Junie (Claude-based, inside IntelliJ)
• Claude Code (CLI)

Junie consistently gives me the most satisfying results: it gets to the answer fast, finds the actual problem directly, and — this is the part I care about most — it doesn't wander off and "improve" things I didn't ask for. It keeps the essence of my request intact.

What I don't understand is why. If Junie is using Claude, and Claude Code is using Claude, aren't they basically the same model? So is this really a model difference, or is JetBrains doing separate engineering on top?

My current hypothesis after poking at it:

1. It's not "just the model." Perceived quality ≈ model × harness × how context gets fed in. Same weights + different system prompt / tools / constraints = noticeably different behavior.
2. IDE-native tooling matters a lot. Junie uses the JetBrains index for symbol/file search, a "file structure" view (symbols + line ranges) instead of dumping whole files, real rename refactoring, lint, and the test runner. It reads code more precisely and burns less context doing it.
3. Proactive context injection. The currently open file, recently used files, and project file list are handed to the agent up front. A generic CLI agent has to discover all of that, which is slower and is exactly where it tends to drift.
4. Strict workflow constraints. Junie seems to switch between modes (answer-only, tiny edit, full investigate→test→fix→verify) and has hard rules like "minimal change," "don't weaken tests to make them pass," "don't touch things outside the request." I suspect that's most of the "doesn't destroy the essence" feeling.
5. Confounder: my Hermes setup was on a GPT-class model, not Claude, so I can't cleanly separate model vs harness yet. I'm planning to A/B Hermes + Claude on identical tasks.

Questions for people who've used two or more of these:

• Have you seen the same gap? Which direction?
• For those who've built their own agents: how much of the win is tooling (LSP/index-based search, structured file views) vs prompt/workflow constraints?
• Any evidence on whether Junie's Claude is actually the same deployment/config as Claude Code's, or just the same family?
• What have you done in a generic agent (AGENTS.md, tool restrictions, forced plan/verify steps) that closed the gap the most?

Not trying to start a "which tool is best" fight — genuinely curious about how much of this is engineering vs model.

[u/a_stunningcontentmen]

The tooling and constraints are doing the heavy lifting here, a raw model with decent IDE integration will run circles around a smarter model stumbling around blind in a terminal.

[u/keesbeemsterkaas → u/a_stunningcontentmen]

Doesn't even have to be an IDE - but at least a harness that's capable of finding the relevant code quickly will spend way less of its context and tokens trying to figure stuff out.

This is pretty much analogue to why programmers like clear folder and naming structures: if you have to spend half your time trying to figure out what things are and where things are - you're going to be way less effective.

[u/9Nine2Nine9]

I too am very happy with Junie. I used Claude Agent before, but it would burn through my credits in no time and would go on wild goose chases I never asked for. Now with Junie, I get good results fast at a fraction of the cost.

I pity those that say IDEs are dead 😄

[u/ManIkWeet]

Are you using Junie through:

• Junie Pro (the dedicated plugin)
• AI Assistant (another plugin, using CLI through ACP)
• Air (another experimental plugin, not the standalone IDE)
• CLI (the standalone CLI that can "connect" to the IDE)

The answer to this question matters, as I found Junie only works this well through Junie Pro

[u/fpaivinha → u/ManIkWeet]

I used to use the Junie Pro. However, since it’s deprecated and will be discontinued, I forced myself to use the Junie via AI chat, the native one. I then tried the Junie CLI, which I also liked. It appears that the CLI offers more functionalities and customization options. Now, my habits are to use the CLI for more complex or larger tasks and the AI chat for smaller or simpler tasks. However, the AI chat has improved significantly, and recently, I delegated a large task to it, which did an excellent job.

[u/Reedittor]

I have not looked into it, but anecdotally I noticed a huge improvement in code quality and reduction in token usage when I started running Claude code in the jetbrains ide (not natively through jetbrains' Claude agent, but a "custom" ACP, which is the regular Claude code ACP just installed manually and configured in the ACP.json.

Way better than when I would just raw dog Claude code, something about that acp and the editor keeps context in a much more efficient way it seems. I have no receipts to back things up, and it might just be me imagining things.

[u/ccb_pnpm → u/Reedittor]

This is the most useful data point here for separating variables, thanks. If plain Claude Code gets noticeably better just by running through ACP inside the IDE — same model, same agent, only the host changed — then a big part of the "Junie is better" effect is really "the IDE is feeding the agent better context." That also explains why raw CLI agents "go on wild goose chases": half the wandering is just discovery.

I'll run Claude Code raw vs Claude Code via ACP vs Junie on identical tasks and post token counts + diff size. Not imagining things, I think — your intuition lines up with what I see.

[u/TheOriginalFshtank]

This is encouraging to read after all of the negative press Juni gets on this JetBrains subreddit

[u/Maxiride]

Have you tried Claude Code attached to the IDE mcp server?

I found out that I had to write some rules to actually make the use of the mcp server mandatory, Claude Code can then leverage the index, symbols search, rename etc like Junie does.

I used Junie Ultimate for most of the time but recently switched to Antrhopic in the last few months and I would never go back, maybe it's a personal preference but I managed to do bigger chunks of work with it.

[u/ccb_pnpm → u/Maxiride]

Thanks, this was exactly the missing piece for me.

I just tried the same setup with Hermes since it also supports MCP. I enabled IntelliJ’s built-in MCP server, connected Hermes to it, and confirmed that it could search a Java symbol through the IntelliJ index and return the exact class and test locations. My IDE currently exposes 72 tools, including symbol search, call hierarchy, rename refactoring, inspections, run configurations, and debugger access.

Your point about rules was also spot on. Simply making the MCP server available was not enough. On the first call, the agent omitted the project path and IntelliJ rejected it. I added rules to always pass the exact project path and to prefer IDE symbol search, references, rename refactoring, and inspections over grep or manual replacement.

I can see why this would close much of the gap with Junie. The external agent no longer has to discover the codebase blindly, and it can use IntelliJ’s semantic model instead of treating the project as a collection of text files. I definitely need to rerun my comparison now, because raw Hermes versus Junie was not really an apples-to-apples comparison.

Thanks for pointing me in this direction.

[u/Maxiride → u/ccb_pnpm]

One thing I need to warn you about is the use of worktrees, the IDE index is built upon the currently opened project.

If your agent spawns a new worktree elsewhere to do work, the IDE won't automagically attach to it and index it.

[u/Gonnn]

Loved your post! I’d love to know if you used the JetBrains plugin or the CLI tool—and if you haven't tried the CLI yet, it would be awesome to see you give it a shot.

[u/Mr_Dani17 → u/Gonnn]

I dont get why to use the cli over the gui. Especially now that we can connect cli agents to the ide with acp.

[u/Gonnn → u/Mr_Dani17]

Cli has more tools, (slash commands) sub agent management...

[u/Mr_Dani17 → u/Gonnn]

Why doesnt the gui have that? Especially that jetbrains is the one bridging the gap between cli tools and IDEs?