NYAA

Current experiments · running

J-Lens

Can you read a model's next words from its middle layers?

Why it matters

A 2026 paper, 'Verbalizable Representations Form a Global Workspace in Language Models' (Gurnee et al., arXiv 2607.15495), introduced the Jacobian lens: a way to read a model's middle layers as words. A finding like that is only as strong as its replications on other models. We tried it on a different model, Swift 1.5 27B, a fine-tune of Qwen 3.8 27B, and wrote the pass marks down before each run, so that either outcome would count.

How we test it

We build the lens, a map from each middle layer of the model to its final answer, from a set of documents.

We score it on six documents it has never seen: how often its top guess is among the model's own next tokens, compared with a plain word-frequency guess and with the logit lens, and how far it lands from the model's final answer.

Every pass mark is written down before the run, and a different agent, one that did not run the test, grades it. When a number turns out wrong, the correction stays in the repo beside the claim.

How we test the J-lens The model passes a text through its layers to its own answer for the next word. Three readers each turn the middle layers, 20 to 44, into a guess at that word: the logit lens, our J-lens map built from documents, and a blend of the two. On six documents the map has never seen, each guess is compared with the model's own answer. THE MODEL layer 1 and the early layers The middle layers layers 20 to 44 the last layers Its own answer for the next word THREE READERS OF THE MIDDLE Logit lens the model's own last step, used as it is J-lens (ours) a map from each middle layer to the answer, built from documents The blend mostly the logit lens, a small dose of the map THE TEST Against its own answer On six documents the map has never seen, each reader's guess is compared with the model's own answer: how far it lands from it (lower is better), and how often its top guess is among the model's next tokens. A plain word-frequency guess is scored the same way. the model's own answer is the yardstick Every pass mark is written before the run; a different agent grades it.

How we test the J-lens: three readers of the model's middle layers, each scored against the model's own answer. Drawn from the method above.

Where we are

  1. 2026-11-03

    Built the measuring instrument. Its first two tests were later withdrawn as proof, when we found a bug in our own checker.

    CLAIMS.md rows 1.1 to 1.7, 4.2

  2. 2026-11-04

    First map, from 6 documents. The first pre-registered test failed its bar: in layers 20 to 44 the lens found well under 1 of the next 10 tokens.

    CLAIMS.md rows 2.1, 2.2, 2.7

  3. 2026-11-05

    A random-probe shortcut for building the map failed its bar: its top guess matched the exact map 70%, 60% and 37% of the time at three layers, against a pass mark of 90%.

    CLAIMS.md rows 3.1

  4. 2026-11-06

    Found a bug in our own checker: its answer-key copy of the model had no causal mask, so it could read ahead. The two tests that ran on it were withdrawn as proof.

    CLAIMS.md rows 4.1, 4.2

  5. 2026-11-07

    Map rebuilt from 22 documents. It cleared its pre-written bar against the 6-document map (the score it used was later judged too easy), but in the middle layers it is still worse than the logit lens.

    CLAIMS.md rows 5.2 to 5.4

  6. 2026-11-10

    A blend that is mostly the logit lens with a small dose of the map lands about 8% closer to the model's own final answer on fresh text (reported only; no pass mark was set for it in advance). Reading past the very next token worked for no method.

    CLAIMS.md rows 6.1, 6.4, 6.6

  7. Running

    A bigger map: 54 documents instead of 22, planned over seven nights, with its pass mark signed before the run. Nobody looks at the result until it is finished. It tests whether more data improves the Jacobian lens against the logit lens; it does not re-test reading past the next token.

    CLAIMS.md row 6.10

What's next

A bigger map: 54 documents instead of 22, planned over seven nights, with its pass mark signed before the run. Nobody looks at the result until it is finished. It tests whether more data improves the Jacobian lens against the logit lens; it does not re-test reading past the next token.

What we found so far

Word frequency0.075

Logit lens0.070

Jacobian lens0.054

Shrunk Jacobian lens0.097

Trained lens0.219

Next token, top-1, middle layers, 306 held-out positions: word frequency 0.075; logit lens 0.070; Jacobian lens 0.054; shrunk Jacobian lens 0.097 (not distinguishable from frequency); trained lens 0.219. One token further ahead: word frequency 0.073; every lens 0.019 or lower. Bars are to scale.

J-Lens · results so far

Results so far, run by run.

At mid-depth, neither the Jacobian nor the logit lens predicts upcoming tokens better than word frequency; only the trained lens beats frequency, and only for the next token.

The sentence and the row were checked against the result file by a different agent, one that didn't run it. The code, the result files and one row per public claim are in CLAIMS.md.

Does our map get better with more documents? Distance from the model's own answer in its middle layers (KL, nats; lower is better) against the number of documents the map is built from, 6 to 22 measured. Our original map falls from 10.4 to 8.3 but stays above the simplest method (7.12, flat). The blend sits below the simplest method at every point. Two fitted curves past 22 are not data. no measurement past 22 documents yet 6 7 8 9 10 11 6 10 14 18 22 30 40 54 documents the map is built from distance from the model's own answer (KL, nats) simplest method: flat, it uses no documents (7.12) registered curve (not data) curve fitted after the data (not data) our original map (circles) the blend: mostly the simplest method, a small dose of our map (squares)

Does our map get better with more documents? Distance from the model's own answer in its middle layers, mean KL over layers 20 to 44 on six check passages, by the number of documents the map is built from. Lower is better. Bars: 1 SD across document subsets (none at 6 and 22). Blend strength was chosen on these same six passages; its never-seen result is on fresh text (not shown). The dotted curves are fits, not data. Source: dose_kl.json 6be964a0; every point is in the data file. jlens-dose-points.csv

Evidence