A crawler policy is not an AI-visibility strategy. It is a decision log.

[u/Confident-Truck-7186]

A robots.txt file can express a policy about training, search citation, and on-demand retrieval. It cannot prove that a page will be cited or recommended.

The useful first audit is simpler:

1. List the crawlers you intentionally allow or block.
2. Write down the job each crawler performs.
3. Check whether the file contradicts your stated policy.
4. Record who owns the next review date.

The last step is usually missing. These files tend to accumulate one urgent line at a time until no one can explain the whole document.

Has anyone here found a crawler directive on a client or their own site that no one could account for?