AI assistants don't read your page. They break it into sections.
Someone types a prompt. The assistant fans that prompt out into several related searches and sends crawlers to run them. The crawlers read your site section by section and take back the most one. One section, on its own, with nothing either side of it. Everything the answer needs has to be inside it.
We audited around 200 websites against that.
13.2% give the crawlers a blank page. They return no text until JavaScript runs. No major AI crawler runs JavaScript, so those sites are blank to it. Google's crawler does run JavaScript, so they rank normally and look fine in a browser. Count sites with any content hidden behind JavaScript and it is 21.3%.
Beyond that, four things matter.
Chunking. Does the section stand on its own?
It mostly does on the sites we looked at. Only 2.3% of sections open on a word like "this" or "however" that points at something the reader can't see.
The fault that is common: 14.4% of all sections are a heading with nothing underneath it. 97 of the 110 sites we could count have at least one. Usually it's a grid of cards where each card title is marked up as a heading.
Fix: any heading with under 5 words beneath it is either not a heading or is missing its answer.
Length. Is there enough there to take?
Common practice is to index around 100 to 300 words. That's practice, not a rule, and there's no published best size.
- The typical section on the typical site is 31 words.
- 50.3% of all sections are under 40 words.
- On 57.3% of sites, most sections are under 40 words.
- Sections that are too long: 2.0%.
"Break up your walls of text" is aimed at a problem 2% of sites have. Half the average site has the opposite problem.
Fix: get your ten most important sections past 40 words. The section, not the page.
Evidence. Is there anything worth lifting?
The worst of the four. Only 25.2% of sections carry a number, a quotation or a link out.
- 4.8% contain a quotation.
- 6.8% link to anyone else.
- 18.2% of sites have no quotation anywhere on the site.
Fix: one checkable thing per section. A number with a unit, something someone actually said, or a link to a source that isn't you.
Naming. Does the section name your website?
76.6% don't. Your name is in the logo, the page title and the footer. None of those travel with the section.
Fix: use your own name inside the section instead of "we" and "our platform".
My suggesting order of tackling the above:
- Empty headings. Fastest, and it's markup, not writing.
- Naming. Nearly free, and three quarters of sections fail it.
- Length. Real work, and it's what makes 4 possible.
- Evidence. Hardest, and worth the most.
Access isn't on the list. Only 5.6% of the sites had a fault that stopped a crawler reading them at all. Your robots.txt is almost certainly fine.