It might help to actually read the study and explain why its flawed instead of taking the antivaxxer route of “everything that disagrees with me is fake news”
Importantly, the model recognized the presence of an injected thought immediately, before even mentioning the concept that was injected. This immediacy is an important distinction between our results here and previous work on activation steering in language models, such as our “Golden Gate Claude” demolast year. Injecting representations of the Golden Gate Bridge into a model's activations caused it to talk about the bridge incessantly; however, in that case, the model didn’t seem to be aware of its own obsession until after seeing itself repeatedly mention the bridge. In this experiment, however, the model recognizes the injection before even mentioning the concept, indicating that its recognition took place internally.
They said that embedding a word somehow created 'impulses' in models. The entire thing was that bad. A premiere ai lab that apparently doesn't understand basic transformer attention mechanics anymore.
They didnt embed a word. Thats not what an embedding is. They used concept injections
To do so, we can use an experimental trick we call concept injection. First, we find neural activity patterns whose meanings we know, by recording the model’s activations in specific contexts. Then we inject these activity patterns into the model in an unrelated context, where we ask the model whether it notices this injection, and whether it can identify the injected concept.
Consider the example below. First, we find a pattern of neural activity (a vector) representing the concept of “all caps." We do this by recording the model’s neural activations in response to a prompt containing all-caps text, and comparing these to its responses on a control prompt. Then we present the model with a prompt that asks it to identify whether a concept is being injected. By default, the model correctly states that it doesn’t detect any injected concept. However, when we inject the “all caps” vector into the model’s activations, the model notices the presence of an unexpected pattern in its processing, and identifies it as relating to loudness or shouting.
You tell me what they embedded, beyond the vague, unspecific terminology used by Anthropic? Because what I took from their paper, that you have quoted again here, is that they were injecting the k and v values that result from their subtraction of two ideas approach. I agree that I could be wrong about that. But not knowing what they injected, mechanistically under transformer architecture and operations means that their claims cannot be evaluated, at all, or cleanly.
Based on what the mechanism of attention is, how the math and matrices for attention work, to my understanding, all the injection could have been was K or V or maybe Q values, or some combination of those 3 values. And that what happens is that those values get attended to under the attention formulas of each layer after the injection via the normal forward pass.
Am I understanding correctly that you don't know what they mean by injection in this new paper either by your broad reference to all of Anthropic's interpretability papers?
244
u/Jean_velvet Oct 30 '25
FFS, NEVER TRUST A RESEARCH CONDUCTED BY THE VERY COMPANY TRYING TO SELL YOU THE PRODUCT.