In Machina N400: Pinpointing Where a Causal Language Model Detects Semantic Violations

A quick note before the piece proper: the “researchers” described here are my collaborator, Revekka Kyriakoglou (Google Scholar), and me. What follows is our own account of the study.

A “dog” floating in a coffee cup, or a farmer harvesting “sunlight”: a single wrong word like that registers almost instantly. Neuroscientists have traced the neural signature of this mental collision for over four decades. It’s called the N400, an electrical signature in the brain that fires roughly 400 milliseconds after reality contradicts linguistic expectations. It’s the brain’s “wait, what?” moment, a burst of prediction error.

What about LLMs, though? How do they deal with a collision of expectations? That’s the question we set out to answer, using the Phi-2 language model as a stand-in. We treated the transformer as a transparent, time-unfolding laboratory (an “in-machina” laboratory, in the paper’s terms) and tried to pinpoint exactly where and how an N400-style moment happens inside an AI’s hidden layers. What we found: silicon and synapses appear to converge on a similar underlying principle. We say that carefully: at least within this scope, we do not claim a complete isomorphism.

In Phi-2, nonsense registers only after a hard-won computational realization, not instantly.

Noticing Nonsense Takes Computational Depth

One of the clearest findings: semantic violations are invisible to the model’s early layers. We ran a linear decoder over Phi-2’s hidden states, and the decodability showed a clear delay before it caught on.

In the bottom third of the model’s layers, the decoder was effectively blind: it could not tell a plausible sentence from a nonsensical one. Anything happening before layer 18 was statistically indistinguishable from noise. Only in a cluster spanning layers 18 to 30 (out of 32) did the model start reliably exceeding chance, peaking at layer 22 (mean AUC = 0.723). One likely explanation is that the model has to build the sentence’s syntactic skeleton first, before it has enough context to judge its meaning.

The AI “Panic” Phase (Dimensionality Expansion)

When the model “sees” a “dog” in its “coffee,” its internal geometry goes through a substantial two-part shift. We tracked this with the Participation Ratio (PR), which measures how many “dimensions” the model uses to represent information at any given point.

In layers 1 through 6, the model enters a high-dimensional exploratory phase. When the violation occurs, the PR spikes, exceeding the control condition by up to 1.9 dimensions at layer 5. The model spreads activation across a much wider subspace while searching for any possible world in which a dog belongs in a cup. At layer 12, that trend reverses. The representation hits a dimensionality crossover and begins to contract.

That’s the computational version of a double-take. First, the system widens its search for meaning, then it narrows in on the detected error.

The “Syntax First” Rule Still Applies to Machines

This result aligns with classical psycholinguistics. In humans, brain responses like the Early Left Anterior Negativity (ELAN) handle syntax, the “how” of a sentence, between 100 and 300 ms, well before the N400 handles the “what” of meaning (although we are fully aware of contradictory claims). Here, we refer to the seminal work of Angela Friederici as an anchoring point.

Phi-2 reproduces that same ordering, just spatially instead of temporally. Semantic evaluation is postponed until the mid-to-late layers, which fits the idea that the model needs “sufficient syntactic constraints” in place before it can flag a meaning as impossible. Of course, we are not claiming that this is the only account; we are simply pointing to the striking similarity.

This is the clearest argument for why the in-machina approach is worth the effort: a human brain cannot be easily sliced into 32 discrete, reproducible layers to watch a single thought unfold, but Phi-2 can. It lets us test theories about cortical algorithms with a level of mechanistic detail that remains unavailable in living subjects. Keep in mind that, currently, LLMs are the only accessible non-human system capable of generating and understanding natural language.

A Tighter Bottleneck for Semantic Truth

At the layer 12 bottleneck, normal and nonsense sentences take sharply different paths, revealing a curious irony in how the model represents truth. Normal sentences follow a steady representational path, with dimensionality (PR) continuing to climb through the final layers; for the model, an ordinary sentence’s “truth” remains complex and multidimensional. Violation sentences do the opposite. They undergo what we call aggressive anisotropy, a sudden collapse in variance, and after the bottleneck their representation concentrates along fewer principal axes as the model fixates on the error.

We believe that this contraction reflects concentrated “attention” and not information loss: the model narrows so intensely on the semantic violation that the rest of the sentence’s features flatten out.

Where This Leaves Us

This work pinpoints where an AI model registers something like a semantic “intuition,” but it rests on a single 2.7-billion-parameter model, Phi-2. The next step is to test whether the pattern holds across other architectures: comparing autoregressive models like Phi-2 against bidirectional encoders like BERT, which “see” the whole sentence at once rather than one token at a time, would show whether the delay is specific to this model or to transformers more broadly.

Scale is the other open question. Whether nonsense detection speeds up as models grow toward trillions of parameters, or whether the syntax-before-semantics delay holds regardless of substrate, wetware or silicon, is not yet known. What we do know, for now, is specific: encountering a dog in a coffee cup triggers, in Phi-2, a measurable, delayed reorganization of its internal geometry, rather than an instantaneous flag for an unlikely word.

You can find the full work here.