{"id":"763e22a8-cec1-4394-ae41-9c45369b4e17","arxiv_id":"2504.20271","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Prompted linear probing is the most data-efficient activation monitoring method, while SAE-based probing beats raw activation probing when inference-time compute is limited.","lead":"Researchers at OpenAI compared several ways to monitor a language model's outputs by reading its internal activations, including adding task-specific prompts and using sparse autoencoders. They recommend prompted probing when inference-time compute is available, and SAE-based probing when it is not, based on experiments on moderation, hallucination, and sentiment tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The zero-shot baseline is likely pessimistic by the paper's own admission (Sec. 5), so the claimed advantage of prompted probing over zero-shot may not hold; the comparison also uses different templates for the two methods.","rationale":"The reader's weakest assumption focuses on representativeness of tasks and model, which is an external-validity concern. The more immediately load-bearing concern is internal validity: the comparator that motivates the central recommendation is explicitly acknowledged by the authors to be pessimistic. Section 5 states that zero-shot estimates are likely pessimistic because the prompts were not carefully engineered and no reasoning model or test-time compute was used; the paper also reports that few-shot prompting did not help, leaving open the possibility that better zero-shot prompting would. If a stronger zero-shot baseline closes the gap with prompted probing, then the recommended method provides no clear benefit over simply asking the model, making the compute-budget recommendation moot. Furthermore, the comparison is not fully controlled because zero-shot uses the prefix+suffix template while the main prompted probing results use suffix-only, so template effects are mixed with method effects. This does not necessarily overturn the paper, but it makes the recommended ranking contingent on the zero-shot baseline being representative. Since the reader already assigned CONDITIONAL, the verdict should remain unchanged; however, the condition should explicitly require demonstrating robustness to a stronger zero-shot baseline.","tokens_in":12297,"tokens_out":7338,"duration_ms":74971,"concrete_test":"Re-run the zero-shot baseline on the same four test sets using an improved prompt (e.g., include the official rubric/definition of the target concept, or allow the model to produce a chain-of-thought before the final Yes/No answer), and also re-run prompted probing with the same template as the zero-shot baseline (prefix+suffix). Compare AUROC across training set sizes. If the improved zero-shot AUROC meets or exceeds prompted probing AUROC in the regimes where prompted probing is recommended, the central claim fails. Also report confidence intervals across multiple seeds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central recommendation (Sec. 5) is to use prompted probing when inference-time compute is available, justified by superior data efficiency and generalization relative to zero-shot prompting. However, the paper itself concedes in Sec. 5 that 'our estimates of zero-shot monitoring performance are likely pessimistic' because (1) no effort was put into zero-shot prompt engineering and (2) no reasoning model or test-time compute was used. Since the zero-shot baseline is the direct comparator in the key results (Fig. 2) and the abstract's claim that 'asking the model zero-shot is a reasonable baseline,' a stronger baseline could materially change the ranking. For example, a rubric-based prompt or chain-of-thought could push zero-shot above prompted probing in the low-data regime, eliminating the 'superior data efficiency' rationale. Additionally, the comparison is not fully controlled: zero-shot uses the prefix+suffix template (Sec. 3.5) while the main prompted probing results use the suffix-only template (Sec. 4.3), so template effects are confounded with method effects. If the zero-shot baseline were artificially low, the paper's top-line recommendation would be unsupported even on the four tasks studied.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper systematically compares monitoring methods for LLM outputs: direct zero-shot prompting, raw last-token activation probing, prompted probing (probing activations after a task-specific prompt), SAE-based probing (with max-pooling or pre-activations), the LAT Scan method of Zou et al., and few-shot prompting. Experiments are run on a single ChatGPT-4o snapshot across four binary tasks: harassment and violence moderation, SimpleQA hallucination detection, and RottenTomatoes sentiment classification. The authors report AUROC as a function of the number of positive training examples, and also examine out-of-distribution generalization on the moderation datasets. Their central claims are: (1) zero-shot prompting is a strong baseline when inference-time compute is cheap; (2) activation probing methods substantially outperform it when training data are sufficient; (3) prompted probing is recommended when inference-time compute is available because of data efficiency and generalization; (4) when inference-time compute is limited, SAE-based probing outperforms raw activation probing. They also report that prompted probing and SAE probing do not stack substantially, and that max-pooling helps SAE probing but hurts raw activation probing.","tokens_in":12473,"tokens_out":2687,"duration_ms":29216,"significance":"If the stated rankings are correct, the paper provides concrete, compute-budget-dependent guidance for practitioners choosing among activation-monitoring methods, a question on which recent literature reports mixed results. The work's strengths include systematic comparison of several methods on multiple tasks, inclusion of the LAT Scan baseline, replication of known findings (max-pooling helps SAEs, hurts raw probes), and an explicit acknowledgment of limitations in Section 5. Because the main recommendations depend on empirical rankings, the absence of a separate validation procedure and the acknowledged pessimism of the zero-shot baseline are consequential; the practical guidance cannot be fully supported until those issues are addressed. The paper is nonetheless a useful contribution to the empirical literature on activation monitoring, and its claims are crisply falsifiable.","major_comments":[{"comment":"Hyperparameters such as the model layer depth (75%), Q for SAE latents, and logistic regression regularization C are selected by sweeping AUROC on the same fixed test sets used for the final reported results. The text does not describe any held-out validation set or nested cross-validation. This selection on the evaluation set introduces optimistic bias into the reported AUROCs and can inflate apparent differences between methods, so the quantitative rankings in Figures 2-6 may be distorted. Please rerun the final evaluations on a separate validation split (or use nested cross-validation) and report the corresponding results.","section":"§3.4, Figures 7 and 10"},{"comment":"The authors themselves state that their estimates of zero-shot monitoring performance are likely pessimistic because they did not engineer zero-shot prompts and did not use a reasoning model or test-time compute. Since zero-shot prompting is the direct comparator in Figure 2 and the basis for the abstract's claim that it is a reasonable baseline, a stronger zero-shot baseline (e.g., rubric-based prompting, chain-of-thought, or best-of-N sampling) could plausibly change the ranking, particularly in the low-data regime where prompted probing is said to have superior data efficiency. The paper's central recommendation for prompted probing therefore needs either a stronger zero-shot baseline or substantially qualified conclusions.","section":"§5, second limitation paragraph"},{"comment":"The zero-shot baseline uses the prefix+suffix template, while the main prompted-probing results use the suffix-only template. The authors note in §4.3 that 'for all experiments in this work we only show results from suffix-only prompting' and that 'adding a prefix in prompted probing would slightly improve on the key results.' This means the comparison between zero-shot prompting and prompted probing confounds method differences with template differences. To support the claim that prompting is important and that prompted probing is data-efficient, the zero-shot baseline should be evaluated with the same suffix-only template (or the main prompted-probing results should be shown for the matching prefix+suffix template).","section":"§3.5, §4.3"},{"comment":"No error bars, confidence intervals, or repeated-seed runs are reported anywhere in Section 4. Many claims are based on differences between methods that appear modest (e.g., prompted probing vs. SAE max-pooled probing in high-data regimes, and the mixed OOD results in Figure 3). Without uncertainty quantification or at least multiple random training-set subsamples, it is difficult to know whether the stated rankings are reliable, particularly in the small-sample regime (e.g., 5-20 positive examples). Please add repeated runs or confidence intervals, or explicitly report the variability.","section":"§4, Figures 2-6"}],"minor_comments":[{"comment":"The prefix-only prompting variant mentioned in §4.3 and Figure 8 is not defined in Section 3.2; please describe it in the methods section for completeness.","section":"§3.2"},{"comment":"The text says LAT Scan is unsupervised because 'we do not use labels at any step prior to computing the first principal component,' but the sign of the direction is chosen by maximizing AUROC on the train set. This use of labels is minor but should be acknowledged explicitly to avoid confusion about the method's supervision requirements.","section":"§3.4, LAT Scan description"},{"comment":"The description of balancing says 'We always balance the positive and negative examples in the training set,' but test sets are not balanced (e.g., 149 positive vs. 373 negative for harassment). Please clarify why this imbalance is acceptable for AUROC reporting and whether any class-weighting was used in the logistic regression.","section":"§3.1"},{"comment":"The caption and text in §4.2 say a '2-level classifier' combines a probe logit with the model's output logit, but Section 3.7 describes a second-level classifier on the two logits; the figure description is vague about how the two signals are combined (concatenation, weighted sum, etc.). Please be explicit.","section":"Figure 6"},{"comment":"The related-work section mentions several prior comparisons of SAE probing vs. linear probing, including negative results from Kantamneni et al. and Smith et al., but the discussion in Section 5 describing the paper's position relative to those works is brief. A short summary in the related-work section of why results might differ (e.g., model, SAE training data, tasks) would improve readability.","section":"§2, related work"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical study with practical recommendations, but the evaluation protocol has load-bearing weaknesses: hyperparameter selection on the test sets and a zero-shot baseline that the authors themselves call pessimistic. The template mismatch between zero-shot and prompted-probing evaluations adds a confound. I believe these issues are fixable within the scope of the manuscript, but they require additional experiments or substantial reanalysis, hence major revision rather than rejection. I would also encourage the editor to consider whether the journal values empirical benchmarking papers of this type; the manuscript's contribution is more practical guidance than methodological novelty."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely useful empirical comparison of monitoring methods, and the within-probing rankings are plausible. The zero-shot comparisons, though, are shakier than the paper lets on.\n\nWhat is new: the suffix-only prompted probing template, probing SAE pre-activations, and the null result that prompted probing and SAE benefits do not stack. The paper also does a clean job replicating that max-pooling helps SAE probes but not raw probes. Methods are described at reimplementation level, and the limitations section is candid—the authors admit their zero-shot estimate is likely pessimistic and that a fine-tuning baseline would beat them.\n\nThe data-efficiency and out-of-distribution experiments are the strongest part. The conditional recommendation—prompted probing when inference-time compute is cheap, SAE probing when it is not, raw probing when both are tight—follows from the plots.\n\nSoft spots, in rough order of size. First, the stress-test concern is correct: zero-shot uses the prefix+suffix template, prompted probing uses suffix-only, and the authors concede they did not engineer zero-shot prompts or use reasoning. So the headline claim that prompted probing beats zero-shot at low data is not a controlled comparison. It may still be true, but the paper does not show it. Second, hyperparameters (layer, Q, C) are selected by sweeping on the same fixed test sets used for final numbers, with no separate validation set and no error bars or seed variation. The differences between SAE and raw probing are often small; without intervals, I can't tell which gaps are real. Third, the hallucination labels are LLM-generated and not validated. Fourth, everything is one ChatGPT-4o snapshot, so model-transfer is unknown.\n\nAll of this is conditionable rather than fatal. The core comparison among the three probing families is less exposed to the zero-shot baseline problem, and the authors are appropriately humble about scope. This is a paper for safety practitioners who need a decision rule today and for probing researchers who want a broad benchmark. It deserves serious peer review; I would referee it and I'd ask for a controlled zero-shot baseline, validation-based hyperparameter selection, error bars, and ideally a second model.","headline":"Useful empirical map of probing methods for LLM monitoring, with a shaky zero-shot baseline and no error bars; the within-probing rankings are plausible.","tokens_in":13064,"tokens_out":2443,"would_cite":true,"duration_ms":24883,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Prompted probing beats raw activation probing for LLM monitoring when inference-time compute is available, and sparse-autoencoder probing wins when it is not.","keywords":["activation monitoring","linear probing","sparse autoencoders","prompted probing","zero-shot prompting","hallucination detection","sentiment classification","LLM safety"],"falsifier":"Run the same four task types on a broader set of chat models and a held-out set of monitoring categories, recording AUROC at 50 and 1000 training positives. The central recommendation is refuted if prompted probing is not among the top methods in the low-data regime on some new task, or if SAE-based probing does not beat raw activation probing when prompting is unavailable.","tokens_in":12020,"feed_emoji":"🔍","tokens_out":9568,"duration_ms":86374,"temperature":0.7,"pith_summary":"Monitoring a chat-trained language model's outputs is a resource-allocation problem: every method trades labeled data, inference-time compute, and training-time compute. This paper develops and compares prompted probing, raw activation probing, sparse-autoencoder probing, and zero-shot prompting on four tasks, and it argues that no single method dominates. The central ranking is that zero-shot prompting is a reasonable baseline when extra forward passes are cheap; with enough labeled data, activation probes clearly beat it; prompted probing is the most data-efficient probe and generalizes well, so it should be used when inference-time compute is available; and SAE-based probing beats raw activation probing when it is not. The practical takeaway is a compute-budget-dependent decision rule for building output monitors without fine-tuning the model.","feed_headline":"Prompted probes beat raw LLM monitoring probes when compute allows","feed_subtitle":"Zero-shot prompting is a strong baseline, but sparse-autoencoder probes win when inference compute is limited.","key_machinery":"The machinery that carries the argument is prompted probing, defined as a linear probe trained on the last-token residual-stream activation from a forward pass whose prompt explicitly asks the model to judge the target category. The prompt slot is varied with a suffix-only template that shares a single forward pass across categories, which makes the inference-time cost small. The supporting machinery is SAE max-pooled probing: a sparse autoencoder with 512k latents and TopK activation is trained offline, the TopK nonlinearity is swapped for a JumpReLU at probe time so latents can be computed independently, the latent activations are max-pooled across tokens, and a logistic regression probe is trained on the top $Q=1000$ latents picked by a class-mean-difference score. Max-pooling is the crucial step for SAE probes and hurts raw-activation probes, so the privileged SAE basis is what carries the no-prompting regime.","core_discovery":"The paper reports that monitoring quality is not a matter of picking one universally best method; it depends on which compute budget is available. On four tasks (harassment, violence, factual hallucination, and sentiment) with a single chat-trained model, it finds that zero-shot prompted output is a strong baseline, but all trained probing techniques can beat it once enough labeled examples are available. Prompted probing—prompting the model with the monitoring question and training a logistic-regression probe on the last-token activation—is the most data-efficient technique, and it generalizes best under English-to-non-English and single-message-to-conversation shifts. When prompting is too expensive, max-pooled probing on sparse-autoencoder latents outperforms probing raw activations, although its out-of-distribution generalization is sometimes worse. Combinations of these methods do not stack: adding SAE latents to prompted probing, or prompted model output to prompted probing, adds little beyond the stronger single method.","pith_inferences":["Beyond the paper, the strong low-data performance of prompted probing suggests it could be used to build monitors for new categories with only tens of labeled examples, which the paper's data-efficiency curves hint at but do not formally test.","Beyond the paper, if a monitoring model and the policy share weights, prompting-based methods may be vulnerable to collusion triggered by the prompt; the paper raises this concern, and an extension would compare un-prompted probes against prompted probes under adversarial optimization pressure.","Beyond the paper, the finding that prompted probing concentrates task-relevant signal in the final-token activation could be tested directly with activation-patching experiments to identify which attention heads move that signal.","Beyond the paper, the paper's pessimistic zero-shot baseline leaves open that a reasoning model with chain-of-thought might beat all probing methods; a natural extension is to rerun these benchmarks with reasoning tokens before the answer."],"forward_implications":["With enough labeled data, any of the trained activation-probing methods beats zero-shot prompted output on the tasks studied.","When inference-time compute is available, prompted probing is the best default because it needs fewer labels and generalizes well under distribution shift.","When inference-time compute is not available, SAE max-pooled probing is preferable to raw activation probing, subject to its weaker out-of-distribution generalization.","Combining prompted probing with SAE latents or with the model's own output does not meaningfully improve on the better single method, so practitioners can skip the stacking complexity.","Practical defaults that follow from the experiments: save activations near 75% of model depth, use about 1,000 SAE latents, and max-pool only when probing SAE latents."],"supporting_citations":[{"why":"Proposed SAE latent probing with max-pooling; the central baseline method this work replicates and extends.","marker":"Bricken et al. (2024)"},{"why":"Introduced Linear Artificial Tomography and prompted-probing-style methods that motivate the prompted probing comparison.","marker":"Zou et al. (2023)"},{"why":"Prior comparison finding SAE-based probes rarely beat non-SAE baselines; contextualizes this paper's more favorable SAE results.","marker":"Kantamneni et al. (2025)"},{"why":"Prior negative results on SAE downstream tasks and out-of-distribution generalization that this paper engages.","marker":"Smith et al. (2025)"},{"why":"Supplies the SimpleQA dataset used to build the hallucination-detection task.","marker":"Wei et al., 2024"},{"why":"Supplies the RottenTomatoes dataset used for sentiment classification.","marker":"Pang & Lee, 2005"},{"why":"Defines the TopK sparse autoencoder training setup whose hyperparameters this paper adopts.","marker":"Gao et al., 2024"},{"why":"Defines the JumpReLU activation used to replace TopK at probe time.","marker":"Rajamanoharan et al., 2024"}],"fun_headline_variants":["Compute budget picks your LLM monitoring probe: prompted vs SAE","Prompted probing wins with compute; SAE probing wins without","Monitoring? Let your compute budget choose the probe method","Zero-shot is a baseline; trained probes win with data","Best LLM monitor probe: prompted if compute, SAE if not"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking of methods holds only if the four evaluation tasks and the single chat model tested are representative of monitoring settings generally; if the ranking is task-specific or model-specific, the compute-budget recommendation does not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Compute budget picks your LLM monitoring probe: prompted vs SAE","Prompted probing wins with compute; SAE probing wins without","Monitoring? Let your compute budget choose the probe method","Zero-shot is a baseline; trained probes win with data","Best LLM monitor probe: prompted if compute, SAE if not"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00073,"raw_usage":{"total_tokens":3304,"prompt_tokens":1014,"completion_tokens":2290,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":2204}},"tokens_in":630,"tokens_out":2290,"duration_ms":16880,"temperature":1.0,"reasoning_tokens":2204,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:33:43.491111+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same four task types on a broader set of chat models and a held-out set of monitoring categories, recording AUROC at 50 and 1000 training positives. The central recommendation is refuted if prompted probing is not among the top methods in the low-data regime on some new task, or if SAE-based probing does not beat raw activation probing when prompting is unavailable.","supporting_citations":[{"cited_title":"Using Dictionary Learning Features as Classifiers , October 2024","cited_arxiv_id":null,"evidence_quote":"Proposed SAE latent probing with max-pooling; the central baseline method this work replicates and extends."},{"cited_title":"Negative Results for Sparse Autoencoders On Downstream Tasks and Deprioritising SAE Research ( Mechanistic Interpretability Team Progress Update ), March 2025","cited_arxiv_id":null,"evidence_quote":"Prior negative results on SAE downstream tasks and out-of-distribution generalization that this paper engages."}],"review_version":1}