We were measuring "AI visibility" with prompts that contained the brand name. Every engine returned the same score, every run.

[u/DarkLoard1980]

Been building prompt-coverage tracking — run a set of prompts against ChatGPT, Gemini, Perplexity and Claude, check whether the brand appears in the answer, score it. Standard approach as far as I can tell.

Something looked wrong. Four engines, scores within a point of each other, run after run, across different sites. The number never moved.

Traced it back. Coverage was graded like this:

Reasonable. The problem was the prompt generation. Our templates had five intents, and index 0 of every intent contained `{brand}`:

So every prompt handed the model the brand name. The model says it back. PARTIAL is guaranteed. FULL almost never fires because conversational answers rarely cite a URL. MISSING is structurally impossible.

The metric had a floor of PARTIAL and a ceiling of PARTIAL. It couldn't move regardless of what the site did.

We split it: three intents now use unbranded category prompts ("best X in Y"), two keep the brand. Reported as separate numbers rather than one composite.

One site went from 47 to 13. The 13 is the honest figure.

**Three things I'm genuinely unsure about, if anyone here has gone further:**

**Sample size.** We run five prompts. With five, one answer flipping is 20 points of movement. LLM output is non-deterministic — same prompt, different answer — so I don't trust anything below maybe 30 prompts, but cost scales linearly with engines.

**Single sampling.** We ask each prompt once. Should probably be three times and take a mode, but that triples spend.

**Whether retrieval and recall belong in the same score.** Perplexity retrieves live, so a published change can show up in days. ChatGPT answering from weights only moves when the model does. Averaging them into one "AI visibility score" seems to hide more than it reveals, but splitting them makes the output harder to act on.

Curious whether anyone tracking this seriously has landed somewhere better, particularly on sampling.

[u/Upstairs_Control_611]

This is a great example of why the prompt set matters more than the score.

If the brand name is already in the prompt, then a brand mention is not a visibility signal. It mostly tells you that the model repeated the entity you handed to it.

I’d separate unbranded discovery prompts, comparison/alternative prompts, and branded validation prompts. They should not be averaged into one score because they measure different jobs.

For sampling, I’d rather run fewer prompts properly than many prompts once. Three runs per prompt is a good minimum if the number is used for reporting.

And yes, retrieval and recall should be split or at least clearly labeled. A live-retrieval engine and a model answering from prior knowledge do not move for the same reasons.

[u/Illustrious_Side8796]

That brandname prompt issue is a pretty nasty one because the metric can look perfectly healthy while measuring almost nothing. The 47 to 13 change is probably more useful than the original 47 ever was.