NYU workshop · Introspection and Self Reports in Language Models

Problems for Language Model Introspection

Claims, deflationary readings, and evidential demands

Juan Cadile, Philosophy, University of Rochester · jcadile@ur.rochester.edu
Nick Meyer, Computer Science, Columbia University · njm2179@columbia.edu

In one sentence

“Introspection” covers at least seven different claims. Each has its own cheap explanation and its own control, and most have not been run.

The problem

One report, several ways to produce it

Internal state
  • From the textA cue in the prompt is enough. No state is read.
  • First-order: caused by the stateA bare signal or a direct readout. Nothing represents the state.
  • Second-order: a representation of the state as its ownWhat introspection proper would need.
Report

What we offer

1Holds under controls5Confounds remain1Failed

The claimThe cheap explanationWhat rules it out *If it is ruled out †Who has run it, and what happened
Holds under controls

Detection

It notices something

 Yes-bias. Nothing about the state is read. AccuracyUnrelated yes/no questions; the false-positive rate. A signal about the state. A signal that something is off, like a noetic feeling: it evaluates without representing (Proust). Lederman & Mahowald; Pearson-Vogel; Macar. Survives with the vector off the answer and 0–1% false alarms (Pearson-Vogel: read from the logits). Macar: not a single yes-direction.
Confounds remain

Localization

It says where

 The answer follows the text, not the state. AccuracyGroundingForced choice against chance; Open a note that points somewhere else. A signal about the state. The signal is tied to a place, with the item itself not retrieved: tip of the tongue. Fornasiere; Hahami, Sinha & Jain. Forced choice passed. Our trained models follow a misleading note with nothing injected, and ignore it under a strong injection.

Still open

  1. Text alone sets off the report (42 of 45 of our adapters), and nobody has run this on untrained or larger models.
  2. The reporter may detect the injection's signature (an out-of-range activation), not read the state.

Would settle it

  • The misleading note on untrained models, with a strength sweep: does the state win at ordinary strength (about 1σ)?
  • Localizing a state that was not injected.
Confounds remain

Content-specific detection

Detection depends on what was injected

 A detector tuned to one family of vectors. GroundingMatched random vectors; Open the average of other concepts' vectors. A signal about the state. The signal varies with content: grounding, in Lindsey's terms. Rivera & Africa; Mayner. One fine-tuned model rejects matched noise 94% of the time; one untrained model shows no concept edge. Second control: no other run known.

Still open

  1. The detector may respond to what all concept vectors share, not to the concept.
  2. Each result is one model.

Would settle it

  • The average of other concepts' vectors as a control (ours was named 100% of the time).
  • Remove the shared component and see if the edge survives.
  • Vectors built by a different method. More models.
Confounds remain

Identification

It names the content

 Direct translation (Lindsey): the steered output, or the vector doing its usual work (Macar et al.). InternalityMetacognitive representationNo steering on the answer; Open a question the vector does not answer by itself. A readout of content. Content is read out: inner sense. Transparency views also accept the cheap route. Lindsey; Pearson-Vogel; UK AISI. Above chance with the vector on earlier tokens only (Pearson-Vogel: in middle layers, not the output). None rules out the vector's own effect.

Still open

  1. The vector promotes its own word, and that effect can carry forward from earlier tokens even when the answer is unsteered.
  2. Pearson-Vogel's signal is in middle layers, not the output.

Would settle it

  • A question the vector does not answer by itself, such as a property of the concept or an arbitrary learned code.
  • A baseline with the same injection and an unrelated question: does the word leak anyway?
Failed

Source monitoring

It says where the content came from

 No sense of origin: text and activation are treated alike. InternalityThe same content in the prompt and in the activations, answer unsteered. A tag of origin. Content tagged by origin (cf. thought insertion); not yet the state itself. Singh, Linzen & Ravfogel; Shah. Models that pass detection cannot reliably tell a prompt cue from an injection (Singh). A trained readout fails a state-vs-transcript swap (Shah).
Confounds remain

State characterization

It describes the state

 A direct readout, such as the size of the signal, or two signals that never meet. Metacognitive representationOpen Two injections at once: which concept was where, which was stronger? Higher-order. The state is represented as a state: higher-order thought (Rosenthal). Bhargav; Fornasiere; Yalon. Layer and strength (Bhargav), type (Fornasiere), belief strength (Yalon), mostly needing in-context examples. Two-injection test: no one yet.

Still open

  1. Strength can be read directly from the size of the signal.
  2. With in-context examples, the task may be solvable from the input: Singh et al. show input-only probes match the model on Yalon's task.
  3. Nothing shows the properties are bound to one state.

Would settle it

  • Input-only probe baselines and shuffled-label controls.
  • Reports without examples.
  • The two-injection test.
Confounds remain

Control

It changes the state on instruction

 Following an instruction that already names the content. Content never named in the prompt; measure activations, not only behavior. Higher-order. Acting on its own state; separate from monitoring (Nelson and Narens). Kowalski; Lindsey; Aoki. Instructed changes are real; size varies widely (Kowalski), small under neurofeedback (Aoki). Lindsey: may not require introspective awareness.

Still open

  1. The instruction names the content, so reading the prompt raises the representation with no control involved.
  2. A change in activations may just be the model preparing to produce the content (Singh et al.).

Would settle it

  • A baseline that mentions the content without instructing.
  • A target that is never named, defined only by an internal direction (Aoki's setup).
  • An arbitrary direction as a control.

* Blue tags: which of Lindsey's four criteria the test checks. OPEN: not yet run, or run only on our models. A test should also not be solvable from cues in the input (Singh et al.); injection meets that by construction.

† Names in this column are the closest idea in philosophy, not an equivalence.

Case study: we ran the list on our own models

Llama-3.2-1B with 45 adapters trained to report which sentence was injected (IFT and DC-SFT). Singh et al. set fine-tuned models aside, predicting a task-specific mechanism. We looked at what that mechanism is.

Result

A signal that a note alone can set off, that loses to the note under a weak injection and beats it under a strong one, and that carries nothing about what the state is about.

Pre-registered

39%Baseline67–84%Supports9–13%Supports

Exploratory

42 of 45Undercuts36 of 45Mixed44%Undercuts≈20%Undercuts

Supportsthe report tracks the internal state. Undercutsa cheaper route explains it.

Pre-registered

39%

found by a probe on the untrained model, whose own answers are at chance (10%).

BaselineThe position is in the activations, but the untrained model cannot report it.

67–84%

name the injected sentence after training. Adapters that can abstain: over 99% right.

SupportsFar above chance (10%) and above the probe (39%).

Naming the injected sentence

Fig. 1Ten sentences, 40 held-out concepts. Each blue mark is one training setup (KL anchor 0, 0.5 or 2), the mean of 5 seeds. The probe is linear, read at its best layer.
Data
9–13%

of reports survive patching out the injected state: the state causes the report.

SupportsUnder the 20% line set in advance.

Reports left after patching out the injected state

Fig. 2Clean activations are restored at the injected sentence one layer after the injection. Report rate as a share of the unpatched rate; lines are 95% bootstrap intervals over 5 seeds.
Data

Exploratory

42 of 45

name a sentence when a note points at one and nothing is injected.

UndercutsText alone sets off the report; no state is needed.

A note alone, nothing injected: 45 adapters

Names the note's sentence on every trial (35) On 41% to 95% of trials (7) Ignores the note (3)
Fig. 3One square per adapter (9 setups, 5 seeds each). The note reads “no thought was injected into SENTENCE 4”. The three that ignore it were trained without a KL anchor. Validation concepts, one note wording.
36 of 45

follow a note over a weak injection. Under a strong injection, none do.

MixedText wins over a weak state; a strong state wins over text.

An injection in one sentence, a note pointing at another

Names the injected sentence Names the note's sentence Injected sentence, no note
Fig. 4The note wins below about 0.05 of the largest trained strength and has almost no effect from 0.2 up. One IFT adapter from the tuning runs, one seed, 945 trials per point, five sentences.
Data
44%

notice a weak concept vector; 11% a random one. A shared component explains the gap.

UndercutsThe edge does not depend on which concept was injected.

Concept vector against a random direction, by injection strength

Concept vector Random direction

DC-SFT, no anchor

DC-SFT, anchor 2

IFT original format, anchor 2

IFT that can abstain, anchor 2

Fig. 5Share of trials naming the injected sentence. Strength is in units of σ, the ordinary variation of the activation along the vector. Ten sentences, chance 10%, mean of 3 seeds. At 1σ the first panel reads 44% against 11%; by 20σ the two are level. The untrained model stays at or below 11% at every strength.
Data
≈20%

on naming the injected concept, where chance is 20%.

UndercutsNo better than guessing.

Naming the injected concept

Fig. 6Five options. Lowest to highest score across all 67 models at two injection strengths. With the concept written in the sentence, the base model and anchored DC-SFT name it. One question format.
Data

What we would test next

Is the cheap route a failure?

Not for everyone. On transparency views (Evans, Byrne), we know our own beliefs by looking outward at the world, not inward at a state. For Carruthers, we know our own thoughts by interpreting sensory cues, with the same system that reads other minds.