Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

The Confidence Manifold: Geometric Structure of Correctness Representations in Language Models

T0 review · 5 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a language model's knowledge of whether a claim is correct lives in a 3 to 8 dimensional subspace of its internal activations, where separating true from false reduces to a simple mean shift between two class centroid

desk verdict A careful in-domain measurement paper whose headline geometric claim is plausible but not yet established; the abstract promises experiments the body does not contain, and the negation/assertiveness confound in TruthfulQA is not ruled out. read the letter →

arxiv 2602.08159 v2 pith:26JLRVLA submitted 2026-02-08 cs.LG cs.AIcs.CL

Seonglae Cho , Zekun Wu , Kleyton Da Costa , Adriano Koshiyama This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords confidencemanifoldcorrectnessdetectionactivationprobingmeanshiftlow-ranksubspacecausalsteeringsemanticentropyhallucination
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models often state falsehoods with the same fluency as facts, so output-level uncertainty cannot tell you when they are wrong. This paper tries to establish that the model's own internal representation of correctness is geometrically simple: the signal that separates correct from incorrect answers occupies only 3 to 8 dimensions of the residual stream, and adding more dimensions actually hurts detection. It further claims that classification within this subspace is dominated by a mean shift between the two classes, so a centroid-based detector built from two mean vectors matches a fully trained probe, and only 25 labeled examples recover most of the full-data accuracy on the smallest model. Causal steering along the learned direction shifts error rates by about ten percentage points while random directions do nothing, implying the direction is not just a correlate but an active lever. The paper also reports a consistent gap between internal probes and output-based uncertainty methods: internal probes score 0.80 to 0.97 AUC while semantic entropy and similar methods hover near chance on confidently asserted misconceptions.

What carries the argument

The central object is the 'confidence manifold': the low-dimensional subspace of residual-stream activations that separates correct from incorrect claims. It is located with partial least squares regression, which projects activations onto directions maximizing covariance with the correctness labels; sweeping the number of components reveals the 3 to 8 dimensional discriminative rank. The central identity is that the distance between class centroids in this subspace matches the performance of a trained logistic probe, which the paper interprets as class separation being a mean shift. Activation steering (adding a scaled probe direction during generation) serves as the causal lever, and the c

What would settle it

Train a probe on the same benchmark but with incorrect answers rewritten to match the style of correct answers (same terseness, same hedging, same polarity framing); if detection AUC collapses to chance, the so-called confidence manifold is encoding style rather than factual correctness. A second test: measure the centroid distance between correct answers and confidently asserted false claims generated from outside the benchmark; if the distance vanishes, the manifold does not encode correctness per se.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the discriminative signal for factual correctness is concentrated in a low-dimensional subspace—3 to 8 dimensions depending on the model—within the transformer residual stream. Using partial least squares to sweep the number of components, the paper finds that performance peaks in this range and degrades when more dimensions are added. Across nine models from five architecture families, no nonlinear classifier (convex hull, Mahalanobis, kernel SVM) beats a linear probe, and the distance between correct and incorrect class centroids matches probe AUC, indicating the classes are separated by a mean shift rather than by covariance structur

Load-bearing premise

The load-bearing assumption is that the paired correct and incorrect answers in the benchmark isolate factual correctness from answer style—specifically, that the negation or refutation register of correct answers is not what the probe is actually detecting.

Editorial extensions

If this is right

  • Correctness detection can be reduced to two mean vectors in a learned low-rank subspace, eliminating the need for discriminative training once the subspace is found.
  • On a small base model, 25 labeled examples recover about 89% of full-data detection accuracy, suggesting few-shot detectors are practical.
  • Internal probes outperform output-based uncertainty methods on adversarial misconceptions (0.80–0.97 AUC vs 0.44–0.64), so detectors that ignore internal activations are leaving nearly all the signal on the table.
  • Steering along the learned direction produces a monotonic, roughly 11 percentage point swing in error rate while random and orthogonal controls show no effect, implying the direction is causally relevant for behavior.
  • Projecting to 5 PLS dimensions improves cross-dataset transfer by 10–14 absolute AUC points on one large model, indicating that removing dataset-specific noise exposes a more universal correctness signal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the geometry is truly a mean shift, then the same two-vector detector should transfer to other languages and modalities; a cheap test would be to fit centroids on English QA and apply them to multilingual or multimodal answers.
  • The 3–8 dimensional discriminative rank suggests hallucination detection could be inserted into decoding as a low-cost linear filter, but that would require the steering direction to transfer reliably across prompts and domains, which the paper only partially tests.
  • Because the paraphrase control only varies surface templates and not the negation axis—correct refutations versus confident assertions—a stronger control that swaps answer polarity while holding the sentence frame would directly test whether the probe is capturing style rather than factuality.
  • The mean-shift interpretation predicts that a Gaussian-optimal classifier (linear discriminant analysis) should exactly match logistic regression on all models; a direct comparison would confirm or refute the 'identical covariance' assumption implicit in the centroid result.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper sets out to characterize the geometry of correctness representations in transformer residual streams. It trains logistic probes and PLS projections on TruthfulQA paired correct/incorrect answers for 9 models (GPT-2 family and instruction-tuned 1B–7B), and claims: (i) the discriminative signal is 3–8 dimensional; (ii) the classes are linearly separable and separated by a mean shift, so centroid distance matches probe AUC; (iii) 25 labels recover ~89% of full-data performance; (iv) internal probes (0.80–0.97 AUC) far outperform output-based methods (0.44–0.64 AUC); (v) steering along the learned direction changes error rates by 10.9pp with random/orthogonal controls null; (vi) the structure is consistent across architectures. The evaluation protocol is unusually careful: GroupKFold by question, nested CV for hyperparameters, length-residualized and length-balanced controls, paraphrase variance decomposition, and random/orthogonal steering controls.

Significance. If the geometric claims survive, they would give a simple, practical detector and a strong constraint on mechanistic accounts of factuality in LLMs: correctness would be a low-rank linear feature in the residual stream, detectable from two centroids. The paper's strengths are its careful controls (Appendix G nested CV; Appendix H paraphrase variance; Appendix F steering holdout) and the breadth of models. Its principal weakness is that the binary label from TruthfulQA's paired answers may be entangled with a negation/assertiveness style axis; the current controls do not exclude this, and the below-chance HaluEval transfer is a red flag. The advertised abstract also contains claims (erasure, DAS, 11 models) that do not appear in the body.

major comments (5)
  1. [§5.5, Table 8, Appendix H] The central claim that the detected signal is factual correctness is not yet separated from a polarity/assertiveness confound. TruthfulQA's paired answers are not matched on negation: correct answers to misconception questions typically refute ("No...", "not true"), while incorrect answers are fluent assertions. The Appendix H paraphrase control varies only four surface templates ("The answer is...", "To be precise...") and leaves the polarity axis untouched; the F-ratio 17.40 therefore does not rule out style. Table 8 is consistent with the confound: TruthfulQA-trained probes transfer below chance to HaluEval (0.18–0.47 AUC), meaning hallucinated text is systematically flagged as correct. The remark in §5.5 that HaluEval is excluded because it tests "summarization faithfulness" is a post-hoc exclusion; the below-chance transfer is evidence that the direction is not a universal correctne
  2. [§5.1, Table 1, Appendix G] The "3–8D" headline is the argmax of a PLS component sweep, not a demonstrated intrinsic rank. For most models the differences between 3D, 4D, 5D, and 8D are within a few thousandths of AUC (e.g., Llama-3B: 0.917/0.917/0.919/0.910 with reported std of 0.02–0.03), and the nested CV in Appendix G selects 5–8D rather than a sharp 3D peak. The claim "performance degrades with additional dimensions" is supported only by a noisy trend. I ask for paired significance tests of the selected peak against adjacent dimensions and a statement of how flat the curve is. Without this, "occupies 3–8 dimensions" is not an established geometric bound; at most one can say the signal is captured by a small single-digit number of PLS components. The term "dimensions" also conflates PLS components with intrinsic signal dimensions.
  3. [§5.7, Figure 2] The steering experiment inherits the same confound. The steering vector is the probe weight, so if the probe separates on assertiveness/negation, adding ±α will make outputs more or less assertive and thereby change the error rate on TruthfulQA without any change in factual knowledge. The random/orthogonal controls test direction specificity but not semantic content. Please add controls that measure the stylistic axis (e.g., negation rate, hedging markers) under steering, or repeat steering with a direction trained on negation-matched pairs. Without this, the causal claim "confidence direction is causally relevant" is not established over the style alternative.
  4. [Abstract vs. body] The submission-level abstract claims "11 models (124M–14B)", a "2–8 dimensional subspace", "9.1 points on six models", "erasure drops detection to chance", and "distributed alignment search ... at most five causal dimensions". The body reports 9 models up to 7B, a 3–8D peak, a single steering experiment on GPT-2 (Section 5.7), and no erasure or DAS experiments. This is not a minor wording issue; it makes the advertised scope irreproducible. The abstract must either be rewritten to match the body or the missing experiments must be included.
  5. [§5.3, §6] The "mean shift" conclusion is supported by comparing a linear probe with a centroid classifier in the same supervised PLS subspace. Because PLS components are constructed using labels, both classifiers are evaluating the same projection, and under approximately whitened within-class covariance a linear probe is equivalent to a centroid rule. This does not invalidate the practical claim, but it is not a test of whether the original high-dimensional class structure is a mean shift. A test on a label-agnostic subspace (e.g., PCA) or explicit comparison of full-covariance vs diagonal Gaussian models would be needed to support "making detection geometric rather than learned".
minor comments (5)
  1. [§3.2] Procrustes alignment is defined but no cross-model Procrustes results are reported; if the analysis is not used, remove the definition or add the results.
  2. [§4.1/§4.4] The paper mentions an "80/20 stratified split" in §4.1 but the results use 5-fold GroupKFold. Clarify the relationship between these protocols.
  3. [Table 8] In-domain HaluEval AUC is above 0.98 for all models while cross-domain transfer from TruthfulQA is below chance. This stark discrepancy deserves a sentence in the main text, not only the appendix discussion.
  4. [Figure 2] The baseline error rate of 0.56 should be defined in the caption (is it GPT-2 on the 617 held-out questions?), and the absolute error counts for α=±5 should be reported.
  5. [Title/terminology] The term "confidence manifold" is used for a supervised discriminative subspace. Consider using a more neutral term such as "correctness subspace" to avoid confusion with confidence-calibration literature.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the main geometric claims rest on clean held-out evaluation; the only minor issue is a non-load-bearing self-citation and a slight by-construction flavor in the PLS-based mean-shift interpretation, neither of which forces the result.

full rationale

The derivation chain is largely self-contained and properly separated. Probes, centroids, few-shot PLS, and preprocessing are fit on training folds (GroupKFold grouped by question ID) and evaluated on held-out folds; nested CV checks hyperparameter selection, and steering uses a 200-question train / 617-question test holdout (Appendix G and F). The 3–8D claim comes from a CV-selected PLS sweep, and the centroid-vs-probe parity is measured out-of-sample in a fixed PLS subspace; it could have failed if class covariances differed, so it is not forced by construction. The PLS objective does define the subspace using the labels, which gives the 'mean shift' interpretation a mild self-definitional flavor, but the paper's independent checks (nonlinear classifiers, few-shot transfer, controls) give it empirical content. The only overlapping-author citation (Su et al. 2024 includes Zekun Wu) appears in background and is not load-bearing; the truth-direction premise is independently supported by Azaria & Mitchell, Burns et al., and Marks & Tegmark. The HaluEval below-chance transfer and its exclusion from the transfer average are construct-validity/confound concerns, not a circular derivation. The abstract's distributed-alignment-search claim is not detailed in the body, but that is a completeness issue rather than circularity. Overall, no step reduces to its inputs by construction.

Assumptions & free parameters 6 free parameters · 5 assumptions · 1 invented entities

The central claim rests on: (a) TruthfulQA's contrastive supervision being a clean correctness signal (contested by the HaluEval inversion); (b) a PLS CV-peak being read as intrinsic rank; and (c) last-token probing plus single-layer steering being the right lens. Per-model layer/dimension choices, C=0.1, the steering scale, N=25, and SE hyperparameters are hand-fitted or sweep-selected. No new physical entities; the 'confidence manifold' is a reframing of the truth-direction construct with falsifiable handles.

free parameters (6)
  • PLS component count (per-model peak) = 3D (GPT-2 family) to 4-8D (instruction-tuned)
    The central '3-8D signal' claim is the CV-AUC peak of a PLS sweep per model (Table 1, Fig 6); interpreting this fitting peak as the intrinsic rank of the signal is the paper's key interpretive step.
  • Optimal layer (per model) = e.g., GPT-2 L11/12, Mistral L23/32, Llama-3B L12/28
    Headline AUCs use each model's best CV-selected layer (Table 4); this per-model selection is checked by nested CV only for Qwen2-7B and GPT-2-Large (Appendix G).
  • Logistic regression L2 strength C = 0.1
    Hand-chosen (Appendix A.2); not swept in the main text, so reported AUCs are conditional on it.
  • Steering scale and range = alpha in [-5,+5]; direction scaled to 5% of mean activation norm
    Hand-picked protocol (Section 3.3, Appendix F); the 10.9pp effect size and the null random-direction results depend on this scale, with no scale-sensitivity analysis reported.
  • Few-shot label budget N = 25 (per class)
    The headline '89-90% of full-data' uses N=25; at N=5 AUC is 0.60 and at N=100 it is 0.76 (Table 9, Appendix E.4), so the headline operating point is chosen from the sweep.
  • Semantic entropy hyperparameters = K=5, p=0.9, T=0.7, DeBERTa-v3-large NLI
    Baseline protocol (Appendix D.1); the reported output-method AUC band (0.44-0.64) depends on these choices.
assumptions (5)
  • domain assumption TruthfulQA paired correct/incorrect answers isolate factual correctness from answer style.
    Section 4.1/5.4: all supervision and the steering direction come from this pairing; correct answers often negate misconceptions while incorrect answers are fluent assertions. The below-chance HaluEval transfer (Table 8) is consistent with a partial style/assertiveness signal.
  • domain assumption The peak of a CV-AUC sweep over PLS components measures the intrinsic discriminative dimensionality of the representation.
    Section 3.2 and Table 1: the central 3-8D claim interprets a PLS rank sweep as a property of the model's geometry; it may instead reflect estimator overfitting at higher component counts.
  • domain assumption Last-token residual-stream activations carry the full correctness signal.
    Sections 3.1 and 4.4 (following Gurnee et al. 2023); all probes, centroids, and steering use this single token position.
  • domain assumption Adding a single normalized probe-weight vector at every token of the optimal layer is a valid causal intervention on the correctness mechanism.
    Section 3.3, Eq. 2, Appendix F: single-layer additive steering may shift surface generation without actually manipulating the probed representation.
  • standard math Levina-Bickel MLE yields a valid intrinsic-dimension estimate on high-dimensional activation manifolds.
    Section 3.2, Eq. 1: standard estimator; its local-Poisson assumptions are unverified on residual-stream data, and the reported 8-12D manifold numbers inherit this.
invented entities (1)
  • The 'confidence manifold' (low-dimensional correctness subspace with discriminative rank and phase structure) independent evidence
    purpose: Names the object of the paper's geometric claims: a 3-8D discriminative subspace with mean-shift class structure and a three-phase layer organization.
    It has operational handles (probes, PLS sweeps, steering, few-shot labels), so it is falsifiable; but it is a reification under a new name of the already-established linear truth direction (Marks & Tegmark 2024), and its phase structure (Appendix B.1) is a descriptive clustering of similarity matrices.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Confidence Manifold: Geometric Structure of Correctness Representations in Language Models." pith.science (2026). https://pith.science/paper/26JLRVLA

@misc{pith2026260208159,
  author       = {Pith},
  title        = {Pith review of: The Confidence Manifold: Geometric Structure of Correctness Representations in Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/26JLRVLA}},
  note         = {Machine review of arXiv:2602.08159}
}
read the original abstract

When a language model asserts that "the capital of Australia is Sydney," does it know this is wrong? Models assert misconceptions with the same fluency as facts, so the question cannot be answered from output uncertainty. Truth-related signals are known to exist in the residual stream, but not their geometry: how many dimensions carry the signal, how simple a detector can be, and whether it transfers. We characterize this geometry across 11 models (124M-14B) and test it causally with activation steering, concept erasure, and distributed alignment search. The structure is simple: two class centroids in a 2-8 dimensional subspace match a trained linear probe, and 25 labeled examples recover 90% of full-data AUC on GPT-2. Steering shifts hallucination rates by 9.1 points on six models, erasure drops detection to chance, and distributed alignment search, the only method that bounds rank, localizes at most five causal dimensions. The internal advantage is regime-specific: probes far outperform P(True) and semantic entropy on adversarial misconceptions but tie on standard QA. Single-dataset probes transfer near-randomly until joint multi-dataset training restores 0.73-0.91 AUC. That centroid distance matches probe performance indicates class separation is a mean shift, making detection geometric rather than learned.

Figures

Figures reproduced from arXiv: 2602.08159 by the authors.

Figure 1
Figure 1. Layer-wise evolution across 9 models. (a) Detection performance peaks at different depths: GPT-2 family at final layers (100%), instruction-tuned models at mid-layers (43–75%). (b) Intrinsic dimension decreases through layers, converging to 8–12D at optimal layers. & Tegmark, 2024). For concept c, direction wc enables both extraction (h ⊤wc correlates with c) and intervention (adding αwc steers behavior). This exten… view at source ↗
Figure 2
Figure 2. Steering intervention analysis. Error rate on held-out TruthfulQA questions vs. steering coefficient α ∈ [−5, 5]. Interventions modify the forward pass at the optimal layer: h ′ = h + α · wˆ . The learned confidence direction (green) produces a monotonic 10.9 percentage point swing: α = −5 increases error rate to 0.63 (steering toward uncertainty), α = +5 decreases it to 0.52 (steering toward confidence). Random dir… view at source ↗
Figure 3
Figure 3. 3D PLS visualization of the confidence manifold. Row 1: instruction-tuned models (Qwen2-7B, Mistral-7B, Llama-3B). Row 2: GPT-2 family (base models). Convex hulls show class regions; stars mark centroids. GPT-2 family shows clearer visual separation despite lower AUC (0.80–0.84), while instruction-tuned models achieve higher AUC (0.91–0.97) with more overlap in 3D projection. See Appendix E.1 for smaller instruction… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Universal geometric patterns across architectures. (a) Normalized intrinsic dimension (MLE) by layer depth. All models compress from early to late layers (mean curve in black), with peak dimension at 10–20% depth. (b) Dimension-performance correlation: lower intrinsic …
Figure 5
Figure 5. Figure 5: Intrinsic dimension evolution by architecture. (a) Raw MLE estimates show all models compress from 20–55D to 8–12D, except Mistral-7B which exhibits late-layer expansion (80–100D at 90%+ depth). (b) Normalized dimension enables cross-model comparison: models follow a c…
Figure 6
Figure 6. Figure 6: AUC surface over layer and PLS dimension for Mistral-7B. The surface shows probe performance as a function of layer depth (x-axis, 0–32) and PLS dimension (y-axis, 1–120). Color indicates AUC; red line traces maximum per layer. Peak performance (0.90 AUC) occurs at lay…
Figure 7
Figure 7. Figure 7: Geometric confidence estimation methods on GPT-2 (8D PLS space). Five approaches: linear probe (0.773 AUC), centroid distance (0.771), local density (0.701), KNN-10 (0.748), and ensemble (0.764). Scatter plots show 2D projections colored by P(Factual); stars indicate c…
Figure 8
Figure 8. Figure 8: 3D PLS visualization of the confidence manifold for smaller instruction-tuned models (Qwen2-1.5B, Llama-1B, Gemma-2B). These models achieve AUC 0.91–0.93, comparable to larger instruction-tuned models. However, the visual separation appears less distinct due to higher …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. On the Robustness of LLMs' Internal Representation of Code Correctness

    cs.SE 2026-08 conditional novelty 6.0 of 10

    A systematic study shows that an LLM's internal code-correctness signal depends strongly on extraction configuration and fitting data, and that mutation-isolated bugs do not transfer to ranking real candidates.

Reference graph

Works this paper leans on

4 extracted references · 3 linked inside Pith · cited by 1 Pith paper

  1. [4]

    confident misconception

    URL https://transformer-circuits. pub/2024/scaling-monosemanticity/ind ex.html. Thorne, J., Vlachos, A., Christodoulopoulos, C., and Mit- tal, A. FEVER: a large-scale dataset for fact extrac- tion and VERification. In Walker, M., Ji, H., and Stent, A. (eds.),Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putat...

    arXiv 2024

  2. [1141]

    acl-long.1141/

    URL https://aclanthology.org/2025. acl-long.1141/. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.- A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023. URL https: //arxiv.org/abs/2310.0682...

    arXiv 2025

  3. [2024]

    doi: 10.18653/v1/2024.blackboxnlp-1.19

    Association for Computational Linguistics. doi: 10.18653/v1/2024.blackboxnlp-1.19. URL https: //aclanthology.org/2024.blackboxnlp-1 .19/. Lin, S., Hilton, J., and Evans, O. TruthfulQA: Measuring how models mimic human falsehoods. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.),Proceedings of the 60th Annual Meeting of the Association for Compu- ta...

    arXiv 2024

  4. [2025]

    ISBN 979-8-89176-251-0

    Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long

    doi:10.18653/v1/2025.acl-long 2025

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.