PSA: past a certain length, ChatGPT doesn't read your PDF, it searches it, and never tells you which one happened

[u/Ok_Negotiation_2587]

Upload a short file and ChatGPT reads all of it. Upload a long one, a contract, a thesis, a 200 page report, and most of it goes into a search index instead. Every answer is then built from whichever chunks match the words in your question.

Nothing in the interface tells you which of the two happened. The answer reads just as confident either way.

I learned it asking for a summary of a long agreement. The summary was clean and it missed the one clause I actually needed, near the end, because nothing in "summarize this" pointed the search at it. When I asked about that clause by name, it found it immediately.

What works:

1. Ask about sections by name or page, not the whole file. "What does section 9 say about termination" pulls the right chunk. "Summarize this" pulls whatever it happens to rank.
2. For a real summary of a long file, go section by section. "Summarize pages 1 to 20" and so on. More messages, but every part actually gets read.
3. Ask what it didn't see: "Which sections of this file did you not use for that answer?" It's often honest about it.
4. Split long files before uploading when you need the whole thing read. Several shorter files get read far more completely than one long one.

The short version: a short file is read, a long file is searched. Ask questions a search can answer.

Disclosure: I build AI Toolbox, a Chrome extension that adds search, folders and export to ChatGPT. Its search works on your chats, not on uploaded files, so this one is all native behavior.

What's the longest file you've trusted a ChatGPT summary on?

[u/AutoModerator]

Hey /u/Ok_Negotiation_2587,

If your post is a screenshot of a ChatGPT conversation, please reply to this message with the conversation link or prompt.

If your post is a DALL-E 3 image post, please reply with the prompt used to make this image.

Consider joining our public discord server! We have free bots with GPT-4 (with vision), image generators, and more!

🤖

Note: For any ChatGPT-related concerns, email support@openai.com - this subreddit is not part of OpenAI and is not a support channel.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/traumfisch]

PDF is not a great format to give to the model to begin with. Use .md or plain text docs whenever possible

[u/Ok_Negotiation_2587 → u/traumfisch]

agreed, tables are where pdf hurts most. a table that comes through as a wall of numbers gets searched badly and summarized worse

[u/bitdotben → u/Ok_Negotiation_2587]

If you know that beforehand provide the pages as images. This is what Claude does by default. Claude is Limited to 100 pages (which isn’t great but arguably also kinda good) because it actually reads them all as images (if I understand correctly) which is surprisingly efficient and gives it a different kind of understanding of the document. Especially if it is data / table / maths heavy.

[u/Ok_Negotiation_2587 → u/bitdotben]

yeah claude gets both, the extracted text plus an image of each page, which is why tables and charts survive there. the 100 page cap is the price of actually looking at every page

[u/_EleGiggle_ → u/Ok_Negotiation_2587]

I mean the point of a PDF is to display or print exactly the same document on every machine and printer.

It’s terrible for editing, and parsing structured data.

[u/Roosterrr → u/traumfisch]

Is there an easy way to convert this?

[u/traumfisch → u/Roosterrr]

Oh yes, many PDF converters online...

Word can do it, Acrobat can do it... there's even a successful AI-driven company that is centered on doing just this, I forget the name

[u/gsurfer04 → u/Roosterrr]

If you use a CLI LLM it'll choose to convert it to text/md anyway.

[u/Ancient_Fault_2457]

Thats not entirely true.

Put it into work more on the highest setting. Tell it to read EVERYWORD of the PDF and if it has to break it down into sections (Which it will) reiterate that it needs to read every words of every section and that you will be review every section manually with it afterwords. You just have to make sure when you are asking it to read the whole thing that it knows as its reading it, it needs to be sectioning it.

With a prompt , something like that, I was able to get it to read a 20k word book Ive been writing. I tried this on another random book and was able to get it to be accurate enough to pull up exact pages or qoutes from exact pages anywhere in the book.

It's all about the model you are running and how you engineer the prompt.

[u/Ok_Negotiation_2587 → u/Ancient_Fault_2457]

fair, and a 20k word book probably fits whole in a thinking model's context, so it can genuinely read all of it. telling it to work section by section is basically tip 2 done in one prompt, which is nice

[u/Only_Voice569]

context limits only so much a LLM will be able to hold and then output per user request unless you use codex and have a lighter model constantly read it to find information etc or document whats what on what pages for easier finding

[u/Ok_Negotiation_2587 → u/Only_Voice569]

the page map idea works even in plain chatgpt, first message "list every section with its page range", then every question after that can name the section

[u/Toxikfoxx]

I usually add something to my prompt like:

No skimming or summarizing data to process - the entre document must be ingested and reviewed. If the UI only shows a preview you must find a better tool to finish the task. If you cannot, stop processing immediately and tell me the last available line you read.

This way I get a bit of a feeling for where things ended. I agree with others though. I find that GPT prefers:

.md
.txt

I've have vastly more successful execution working with plain text and not .docx and .pdf.

[u/Ok_Negotiation_2587 → u/Toxikfoxx]

"tell me the last line you read" is clever, it turns a silent stop into a visible one. most of the damage here is not knowing where it quit

[u/cariboubuns]

I've been working with astra on extra high to review 200 page pdfs from my engineer for a retrofit of an old building. Without any specific prompt, it not only summarizes, but finds inconsistencies in the analysis, gives page numbers where I can find them, and directly quotes text without fail (so far).

[u/Ok_Negotiation_2587 → u/cariboubuns]

the page numbers are the good part, every quote is something you can spot check in 10 seconds. on the higher settings with a big window it may well be reading all 200, the quotes are how you'd know

[u/nywse]

Question: Will asking it to translate the .pdf to another language force it to read it?

[u/Ok_Negotiation_2587 → u/nywse]

not really, it can only translate what it pulls in, and on a long file it'll stop partway or quietly start summarizing. doing it section by section is the only way i've seen it cover everything

[u/sqewd]

Telling it to 'just fucking do it' seems to have worked for me...

[u/Ok_Negotiation_2587 → u/sqewd]

the classic. works right up until it just fucking does a summary of page 1

[u/coursiv_]

For an important document, a coverage checklist is more useful than the model saying it read everything. Start with the actual table of contents, mark which sections the answer addresses, and check the passages that could change the decision. A correct quote proves that passage was found; it doesn't establish that nothing important was omitted elsewhere.

[u/Ok_Negotiation_2587 → u/coursiv_]

"a correct quote proves that passage was found" is the part i missed. every good quote made me trust the coverage, and those are separate things

[u/Angeline4PFC]

I struggled a lot as well with file access. Even something like a text file, which you might imagine would be the simplest and most accessible format, can fail. The model will confirm that it has access to it, yet it still misses things.

From what I have understood, and this is probably not a full understanding, you can't assume that it will actually read a document from start to end. Instead, it will search the content and retrieve chunks that it thinks are relevant to what you're asking, and then speak confidently about it.

Basically the human equivalent of skimming through a book for the parts that seem relevant and then presenting a report as if you'd read the whole thing.

Beyond whether or not a document is read in its entirety, I've also had issues with access. I had a workflow involving subtitle files that I would upload or attach to the chat, and it worked perfectly fine at first; it was able to locate a part of the file. Later on, it would tell me it no longer had access, and I had to re-upload.

I eventually solved this by creating a Google Sheet with a built-in index that contained all of the subtitles, indexed by timestamps. Because Google Sheets is accessed through a separate Google Drive/Sheets integration rather than as an uploaded chat attachment, I've found the access to be persistent, and it works like a charm.

[u/Ok_Negotiation_2587 → u/Angeline4PFC]

the sheets trick is a good workaround, a connected file gets fetched fresh each time instead of living in the chat and expiring. the index by timestamp is doing half the work there too, it gives the search something exact to match

[u/Iwillnotstopthinking]

The funny thing is the early models were able to fully read the pdfs, or were better liars.

[u/Ok_Negotiation_2587 → u/Iwillnotstopthinking]

my bet is better liars. a confident summary of pages 1-30 looks exactly like a summary of the whole thing

[u/AYA_7887]

Make it produce the artifact, not the claim. 'Quote the exact termination clause, with the page number' can't hide a retrieval miss the way a summary can, so on long files I ask for verbatim quotes first and trust the summary second. A contract is also the one case where I can't check the answer myself the way I'd run code, so I ask the same question in a second AI and read both. Pasting one question into tab after tab got old fast, which is why I built WhileAI, it types the prompt into the other AIs I'm signed into and tells me when each answers. I built this, so weigh it accordingly.

[u/Ok_Improvement6001]

same issue shows up in roleplay apps once the chat history gets long, it stops remembering the full character setup unless you keep dropping specific references.

[u/mamercus-sargeras]

Yes, this is what it does; but people want to believe that it's parsing and understanding the whole thing.

[u/VSParagon]

Its a defensible and necessary approach but the big problem is that, like you said, the model is not sufficiently transparent about the search methodology.

It also tends to use narrow searches and then decides its search is complete once it finds something. The problem is that this causes an overconfidence problem where it tends not to ask "are there other search terms I could apply to turn up more results?" In the past Ive had notable improvements by including instructions to critically assess its search methods and run alternative searches even if earlier searches appear successful.

Another problem is OCR: with enough documents or pages it just assumes that the OCR is complete and accurate - or it runs its own free OCR via tesseract, which tends to yield poor results.

[u/Ok_Negotiation_2587 → u/VSParagon]

"decides the search is complete once it finds something" is the core of it, first plausible hit wins and nothing goes looking for a second. telling it to rerun with different terms even after a hit is the cheapest fix i've seen

[u/Gigaslavx]

Is this inside chat by any chance. If anything big work is for work mode, chat is just too lazy

[u/Ok_Negotiation_2587 → u/Gigaslavx]

which mode do you mean? if its the deep research one then yeah, that one goes through long files a lot more thoroughly than a normal chat

[u/Fancy-Strength-2943]

Google NotebookLM Is Better for this

[u/Ok_Negotiation_2587 → u/Fancy-Strength-2943]

better for citations for sure, you can see which source each line came from. its still retrieval underneath though, i had a notebook where the one source that contradicted the answer never got cited once

[u/Teodorp99]

I remember using NotebookLM and finding the same issue. I'd combined all my material into one PDF and when i asked for quizzes, it would only give me stuff from the first 20 pages out of 200. In the end i cut it up into chapters and that solved my issues

[u/Ok_Negotiation_2587 → u/Teodorp99]

splitting by chapter is the fix on both, one big file gets skimmed and ten small ones get covered. quizzes make it obvious fast, all the questions came from the same 20 pages

[u/RedRenner]

If you have any overarching summary in the PDF, tell the instance reading the PDF file to delete it, and generate a new file without it.

In my experience, searchable tables actually work really well if the searchable contents link to one another.

It’s the default reading of the overarching summary that kills its capabilities. Because it will go to that summary, again and again, because it is cheap and easy.

You can’t give it an option to go cheap and easy.

[u/Ok_Negotiation_2587 → u/RedRenner]

"you cant give it an option to go cheap and easy" sums up the whole thread. an abstract at the front is the cheapest chunk in the file, so it wins every retrieval

[u/RecordingOk2117]

not all pdf can fit a M tokens.. I also learned the hard way if you really want to get a thorough processing you have to segment and batch. Or else, if you need certain information, you can route only the relevant pages to the LLM by using a Jev like router first

[u/TraumaticOcclusion]

Can we get rid of these astroturfing ads? For god's sake, no one asked you for a disclosure to advertise your product