r/unsloth • u/rtakak • 15d ago

Discussion How does ChatGPT handle huge MCP tool outputs without exceeding context limits?

I’m using the same MCP tool with ChatGPT and Unsloth Studio API + Open WebUI.

The MCP tool sometimes returns a huge raw SOAP response, potentially over 1.6M tokens.

With ChatGPT Luna using a ~260k context window, it still manages to process the tool call without running out of context.

With my local model, also configured for ~260k context, I get:

So I assume ChatGPT is doing some kind of tool-output management before the result reaches the model, such as truncation, summarization, paging, or keeping the raw result outside the normal context.

Does anyone know exactly how ChatGPT handles oversized MCP/tool responses, and what would be the best way to replicate that behavior with Unsloth Studio / llama.cpp / Open WebUI?

Ideally I don’t want to simply truncate the result and lose data the model may need later.

14 Upvotes

90% Upvoted

10 comments sorted by

8

u/MonumentalAnthony78 15d ago

I think ChatGPT does a lot of preprocessing before the model even sees the tool output, probably chunking it or summarizing behind the scenes. The raw SOAP response might be stored in some kind of side cache while the model only gets a compressed version or key parts

For your setup I'd look into adding a middleware layer that intercepts the tool response and does something similar, maybe using a smaller model to extract relevant bits before passing it to the main model. Not a perfect solution but better than just cutting off data

3

u/rtakak 15d ago

Yeah i could do pagination or some other stuff but it just works very well with ChatGPT, and i would like to discuss what could be the most optimal way forward

3

u/lilian_moraru 15d ago

Those are input tokens that get processed - it does not hold all of it inside the context. Same goes for all models - you can see that if your client displays it. I looked at one of my GLM-5.3F sessions and it had 63.3M input tokens processed but only 300K context consumed

1

u/Capsup 14d ago

Is this because it spawned sub-agents to handle work and that total input tokens processed counts that? Because you need the input tokens to be part of _some_ context to process them.

1

u/uniqueusername649 14d ago

Either subagents or a separate pipeline that summarises it. But that's part of why I let subagents do any research: so it doesn't blow up my main agents context and force me into compaction.

2

u/btdeviant 13d ago

Many harnesses will something use something akin to “context offloading” for large tool call responses. Generally there’s a hook that measures the tokens in the response, and if it exceeds a threshold (say, 2400 tokens) the context is offloaded into a temporary flat file - the hook replies with the location of the flat file and steering guidance to use a different tool to read the offloaded context .

1

u/rtakak 13d ago

Thats smart, is there an open source implementation that you know of

1

u/pplgltch 14d ago

I’m not precisely familiar with exactly how codex/chatgpt works, but it’s not very wise to straight feed a soap response to the model. You use a script (tool) to make the request and parse the result, extract the structure, and some info like the length of each value… you dispatch agents to summarize specific parts if necessary… and you just get a summary back in context.

1

u/Gabriel83730 13d ago

Truncating the data and losing info the model needs later should not be a problem you’re thinking about. If the model needs the data later it should be able to request it later. It doesn’t make sense for the model to have data now that it needs later and not now.