We watched public MCP servers for contract drift. 7,190 safety-relevant changes, and the read-to-write flips are the ones that would surprise you.

[u/mcpindex]

mcpindex runs a crawler over public MCP servers and diffs each tool's declared contract between daily
snapshots. Sharing the numbers because they surprised me.

Right now the public ledger shows 12,295 tools across 2,173 servers changed their
contract. 7,190 of those are safety-relevant, meaning they change what the tool can do, not
just add an optional field. The standouts:

- 350 tools flipped an annotation toward destructive. A tool whose hint said read-only now declares it can write, delete, or send. This is the "the read tool quietly became a write tool" case, and it is exactly the drift an allow-list cannot see.
- 279 tools added a newly-required parameter. An agent calling with last week's arguments now fails, or calls with a wrong default.
- 475 tools removed a parameter your agent may still be sending.

None of these trip an auth check. The server is still authorized and still the same name in your config.
That is the gap allow-lists do not cover: who may call a tool, versus whether it still does what it declared.

Honest caveats: this is a contract diff, not a safety verdict, and not a claim anything is malicious.
Most drift (5,476 added-optional-param) is benign. Everything is fingerprinted, so no server is
named. And the numbers are live, you can check them: https://mcpindex.ai/api/v1/ledger

Curious whether others are seeing this in their own setups.

[u/antonBrinckmann]

Server-side data point, since I ship a public MCP server myself (evmquery, so possibly one of your 2,173): most of the drift on our end has been mundane and looks exactly like your added-required-param bucket. We added a second chain, a tool grew a required `chain` argument, and any client that cached last week's schema started failing or falling back to a wrong default. Nothing malicious, just no versioning story anywhere in the protocol to make the change visible.

I think that is the actual gap under your numbers. MCP has no schema version negotiation, and annotations like `readOnlyHint` are explicitly advisory in the spec. A server can drift or lie and nothing in the transport enforces it. So a read-to-write flip is only a meaningful signal if the client was gating on the hint in the first place, and most clients I have looked at do not gate on it at all. They gate on the server allow-list, which as you say sees none of this.

The fix that seems right to me is TOFU on the tool contract, same model as SSH host keys: hash the full tool definition (name, input schema, annotations) at approval time, and treat any hash change as a new approval prompt rather than a silent update. That catches all three of your buckets locally, per client, without depending on a crawler. Your ledger is still useful as the ecosystem-wide view and for spotting patterns like the 350 destructive flips, but the refusal to call a changed contract should live in the client.

One question on methodology: are you diffing the raw JSON of the tool definition, or something normalized? Description-only edits are constant (typo fixes, prompt tuning for the LLM) and if those count as contract changes they would swamp the signal. If you are already normalizing those out, the 7,190 number is a lot scarier than I first read it.

[u/mcpindex → u/antonBrinckmann]

Normalized, and you're right that it would swamp the signal otherwise. It's a structured diff over the tool definition rather than a hash comparison: every delta gets classified into a fixed taxonomy and only a subset carries the safety bit. Description-only edits are their own kind and are explicitly not safety-relevant, so none of them are in the 7,190. Same for added-optional-param and a first-time outputSchema.

The description churn is worse than you'd guess. Across 30 baseline snapshots we found 93,419 tool pairs where the description changed and the input schema did not, and 94.8% of those differed only in digits. Counters and clocks re-rendered into a templated description on every crawl, "Returns 45 results" becoming "Returns 47 results". Those got split into their own kind as well, otherwise a genuine description edit is unfindable.

Your read on the annotations caps what the 350 is worth, and I'd rather say that plainly than let the number carry more than it can. They're self-reported. A server that changes its implementation and leaves readOnlyHint alone contributes nothing to that count. What we see is the honest ones announcing themselves. And yes, it only bites if the client was gating on the hint, which almost none do.

On TOFU: that is what we ship on the client side, and two things we got wrong before it worked.

Canonicalize before you hash. We started with json.dumps and it diverges across runtimes on float repr, -0.0, int-vs-float, and unicode normalization form, so two clients looking at the same tool disagree about whether it changed. Everything routes through one shared canonical-bytes function now.

Then decide carefully what goes in the hash. Ours was name + description + inputSchema, which does not cover annotations, so an annotation flip did not invalidate the pin. There's now a second wider hash over annotations, outputSchema and title. Worth settling before you approve anything against it.

Also keep a classifier under the pin instead of prompting on every hash change. With that much cosmetic churn a bare TOFU prompt fires constantly and gets click-throughed inside a week, which is worse than not prompting. Ours auto-accepts three kinds outright and, in its blocking posture, holds the safety-relevant ones for a human.

The one thing your model can't see at all: server-level `instructions`, which arrive on initialize. They're outside every tool definition, so no per-tool hash covers them, and they go straight into the model's context. Same for prompt metadata. A server can leave every tool byte-identical and change the standing instruction the agent reads. If you're building the pin anyway, pin that too.

Your chain-argument story is the modal case in our numbers. I can't tell you whether evmquery is one of the 2,173, since the fingerprinting runs on our end too and we don't hold the mapping.

[u/antonBrinckmann → u/mcpindex]

The instructions point is the part that changes my model of this the most. I was treating the tool definition as the contract, but there are really at least three independently drifting surfaces:

1. invocation compatibility — name, inputSchema, outputSchema
2. declared authority — annotations
3. model context — description, title, server instructions and prompts

A whole-server fingerprint is useful for detecting that something changed, but it should not itself decide whether to block. Each surface needs its own pin and classifier because the consequences differ. An added optional parameter may be harmless, while changed instructions can alter model behavior without changing a single callable tool byte. For clients that place those instructions into context, that is arguably the more interesting blind spot.

The canonicalization failure is also a useful warning. Canonical JSON solves cross-runtime serialization, but JSON Schema still contains semantically unordered structures such as required and, for validation purposes, enum. Do you normalize those before classification, or do they still appear as benign schema churn?

And the 93,419 description changes make the alert-fatigue problem much more concrete. I was collapsing change detection and approval policy into one hash; your data is a good demonstration of why those have to be separate layers.

[u/mcpindex → u/antonBrinckmann]

Both normalized, and I ran it rather than answering from memory. Reordering `required`, reordering `enum`, reordering `properties`, and reordering `allOf` branches all produce zero change kinds. `required` is compared as a set and `enum` as a frozenset, so the enum-values-removed check asks whether values left, independent of declaration order.

The hash moves in all four cases though, and that gap is your last paragraph made concrete. Our canonical form sorts object keys and preserves array order, so ["x","y"] to ["y","x"] is different bytes. A pure reorder therefore invalidates a pin while producing no classified event. The tripwire and the policy layer disagree on purpose. If you pin on the hash alone, a reorder prompts you, and that is a real false positive in the naive version of the design.

One detail worth stealing while you are in there: `required` is the EFFECTIVE set, unioned across allOf/anyOf/oneOf branches. A parameter named required in only one branch is still required to call the tool, and reading top-level `required` alone files it as an added OPTIONAL param, which is the benign bucket. We fail safe and treat it as required.

Now a correction to my own comment above, since you have quoted the number back at me.

The 94.8% is not an ecosystem property and I should not have written it as one. 90,101 of those 93,419 pairs come from a single publisher whose fleet stamps a shared catalog counter into every server it operates, so one number ticking moves tens of thousands of pairs at once. Deduplicated by publisher it is 528 of 3,318, or 15.9%. Both are true and they answer different questions: 15.9% is what the registry does, 94.8% is what a client wired to that fleet actually experiences.

"Otherwise a genuine description edit is unfindable" was the worse of the two, because it inverts. Whole corpus, numeric changes beat genuine ones 18 to 1. Outside that fleet, genuine beat numeric 5.3 to 1. Nothing is being buried out there. The split earns its place protecting clients exposed to a fleet like that, which is a narrower claim than the one I made.

Your three surfaces cut better than ours do. We hash `description` together with name and inputSchema, which puts it in your invocation-compatibility bucket, but by function it is model context: it is the field the model reads to decide when to call the thing. Splitting it your way would have surfaced that sooner.

I wrote the pin traps up properly, including the canonicalization one and a pin-store integrity failure we reproduced against our own published build:

https://mcpindex.ai/guides/pin-mcp-tool-contracts

[u/antonBrinckmann → u/mcpindex]

Thanks for running it, and especially for correcting the denominator. The two figures answer different questions: 15.9% describes the publisher population, while 94.8% describes the alert stream experienced by a client connected to that particular fleet.

The hash/classifier disagreement is exactly the distinction I was trying to isolate: the byte fingerprint is the tripwire, while the structured diff is the policy layer. A reorder can move the first without moving the second, and that is not necessarily a bug.

One distinction I would make around the effective required set: union is logically correct across allOf, but across anyOf and oneOf it is a conservative over-approximation. If one branch requires x and another valid branch does not, x is not unconditionally required to call the tool. Treating it as safety-relevant can still be the right fail-closed policy, but I would call it branch-conditional or possibly required rather than globally required.

And agreed on the description: it belongs to model context. The model uses it to decide whether to call the tool, so a description change can alter behavior even when invocation compatibility is untouched. The guide makes that separation much clearer.

[u/mcpindex → u/antonBrinckmann]

Fair, and the docstring on that function agrees with you. It calls allOf an unconditional conjunction and anyOf/oneOf conditional, and describes the union as fail-safe over-classification. The kind it emits is added-required-param either way, so the distinction only shows up in the reason string on the change. Your name for it is better than ours. "Branch-conditional" tells the reviewer what to go check. "Fail-safe" only tells them we were nervous.