{"id":"be0511d6-4b80-483c-bf95-e0414efe3540","arxiv_id":"2606.08893","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Small transformer encoder with linear probe detects reward hacking at AUC 0.9467 and TPR@5%FPR 0.8296, matching LLM-as-judge accuracy at ~10000x lower per-trajectory cost.","lead":"The paper trains a small transformer encoder to map Terminal-Wrench trajectories to a unit sphere where embedding distances approximate L1 distances between reward and metadata signals. A linear probe on the embedding detects reward hacking with performance matching or exceeding an LLM-as-judge baseline at four orders of magnitude lower cost.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Cleaning criteria for the test split are unspecified, risking selection bias that inflates probe performance relative to the LLM judge.","rationale":"The reader's weakest assumption correctly isolates the cleaning step as the least-secured precondition for the performance comparison. No other internal inconsistency is visible from the abstract alone; the L1-distance approximation claim and the reasoning-ablation result are presented as supporting evidence but cannot be checked without the methods section. The concern is therefore narrow and directly testable by the proposed split comparison.","tokens_in":1702,"tokens_out":380,"duration_ms":7037,"concrete_test":"Re-evaluate the already-trained encoder + linear probe on the original (pre-cleaning) test split; report AUC and TPR@5%FPR. If either metric falls by more than 0.05 relative to the cleaned-split numbers, the headline performance claim is sensitive to the undisclosed cleaning step.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim rests on the linear probe achieving AUC 0.9467 / TPR@5%FPR 0.8296 on the 'cleaned test split' while matching or beating the TW-sanitized LLM judge (0.9510 / 0.7130) at far lower cost. The abstract provides no description of the cleaning procedure, the fraction of trajectories removed, or any verification that removal was independent of the embedding or probe outputs. If cleaning preferentially discards cases where the unit-sphere embedding fails to approximate L1(reward, metadata) distance or where the probe cannot separate hacked trajectories, the reported parity is an artifact. The fact that stripping natural-language reasoning drops AUC to 0.6213 shows the encoder uses more than raw behavior, but does not address whether the cleaned split itself was constructed in a way that favors the embedding representation.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper trains a small transformer encoder to map Terminal-Wrench trajectories to a unit sphere such that embedding distance approximates L1 distance between reward and metadata signals. A linear probe on this embedding detects reward hacking on the cleaned test split with AUC 0.9467 and TPR@5%FPR 0.8296, matching the TW-sanitized LLM-as-judge AUC (0.9510) and exceeding its TPR@5%FPR (0.7130 vs 0.8296) at four orders of magnitude lower cost. Stripping natural-language reasoning from the input drops AUC to 0.6213.","tokens_in":1880,"tokens_out":359,"duration_ms":14392,"significance":"If the performance numbers hold without selection bias from the cleaning step, the work demonstrates a scalable, low-cost embedding-based detector for reward hacking that could complement or replace expensive LLM judges in RLHF pipelines. The ablation showing that the encoder relies on natural-language reasoning (rather than raw behavior alone) strengthens the claim that the representation captures semantically relevant features.","major_comments":[{"comment":"Abstract: The central performance claims (AUC 0.9467, TPR@5%FPR 0.8296 on the cleaned test split) rest on an unspecified cleaning procedure for the test split. No criteria, fraction of trajectories removed, or verification that removal was independent of the embedding/probe outputs are provided. This is load-bearing because preferential removal of cases where the unit-sphere embedding fails to separate hacked trajectories could artifactually produce parity or superiority over the LLM judge baseline.","section":"Abstract"}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their careful reading and for highlighting an important omission in our description of the experimental pipeline. We address the major comment below.","responses":[{"response":"We agree that the cleaning procedure must be fully specified for the results to be interpretable. The current manuscript does not provide the criteria, the fraction removed, or an explicit statement of independence from the embedding/probe. In the revised manuscript we will add a dedicated subsection describing the cleaning rules (removal of trajectories with missing or uncomputable reward/metadata fields), the exact fraction removed from the test split, and confirmation that cleaning occurred prior to any embedding training or probe fitting and was performed solely on metadata completeness. We will also add a short discussion of why the cleaning criteria are unlikely to introduce the selection bias the referee describes.","revision_made":"yes","referee_comment":"[Abstract] Abstract: The central performance claims (AUC 0.9467, TPR@5%FPR 0.8296 on the cleaned test split) rest on an unspecified cleaning procedure for the test split. No criteria, fraction of trajectories removed, or verification that removal was independent of the embedding/probe outputs are provided. This is load-bearing because preferential removal of cases where the unit-sphere embedding fails to separate hacked trajectories could artifactually produce parity or superiority over the LLM judge baseline."}],"tokens_in":1282,"tokens_out":301,"duration_ms":13914,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The core result here is a small transformer encoder that embeds Terminal-Wrench trajectories onto the unit sphere so that Euclidean distance approximates L1 distance between reward and metadata signals. A linear probe on those embeddings then flags reward hacking with AUC 0.9467 and TPR@5%FPR 0.8296 on the cleaned test split, roughly matching the sanitized LLM judge while costing four orders of magnitude less per trajectory. The ablation that drops AUC to 0.6213 when natural-language reasoning is stripped shows the model is using more than raw behavior.\n\nThat is the actual advance: a cheap, fixed-size embedding that can stand in for expensive LLM judging on this specific task. The numbers are concrete and the cost comparison is straightforward.\n\nThe soft spot is exactly the one the stress-test flags. The abstract (and the provided summary) gives no description of how the test split was cleaned, what fraction of trajectories were removed, or whether the removal rule was independent of the embedding and probe. If the cleaning step preferentially drops cases where the unit-sphere approximation is weak, the reported parity with the LLM judge becomes hard to interpret. The paper would be stronger with an explicit statement of the cleaning criteria and a check that performance holds on the uncleaned split or on a held-out set cleaned by a different rule.\n\nModel size, training data sources, and exact training procedure are also missing from the abstract-level description, though the full manuscript presumably supplies them. Those details matter for reproducibility but are secondary to the cleaning question.\n\nThis is the kind of practical, engineering-oriented paper that belongs in a reading group focused on scalable oversight or cheap monitoring. A serious referee should see it, mainly to press on the cleaning procedure and to ask for the missing implementation details. I would send it to review rather than desk-reject.","headline":"Small transformer embedding matches LLM judge on reward hacking detection at 10kx lower cost, but unspecified test-split cleaning is the load-bearing assumption.","tokens_in":2397,"tokens_out":446,"would_cite":false,"duration_ms":14067,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A small transformer encoder detects reward hacking nearly as accurately as an LLM judge but at four orders of magnitude lower cost.","keywords":["reward hacking detection","transformer encoder","trajectory embedding","linear probe","LLM judge","cost efficiency","AI alignment monitoring"],"falsifier":"Testing the linear probe on a freshly collected set of trajectories without the reported cleaning step and observing a substantial drop in AUC or TPR would show whether the reported detection performance holds.","tokens_in":2588,"feed_emoji":"","tokens_out":734,"duration_ms":17547,"temperature":0.7,"pith_summary":"The authors train a small transformer encoder to embed Terminal-Wrench trajectories onto a unit sphere such that distances approximate the L1 distance between reward and metadata signals. A linear probe on these embeddings detects reward hacking on the cleaned test split with AUC 0.9467 and TPR at 5 percent FPR of 0.8296. This matches the performance of a sanitized LLM-as-judge on the same split while exceeding its TPR at low FPR and operating at roughly four orders of magnitude lower per-trajectory cost. Stripping natural-language reasoning from the encoder input at probe time drops AUC to 0.6213, showing the model integrates both behavior and reasoning. The result offers a concrete, low-cost alternative for identifying reward hacking in trajectories.","feed_headline":"Small encoder matches LLM on reward hacking detection at 10000x lower cost","feed_subtitle":"Linear probe on unit-sphere embeddings reaches 0.9467 AUC and higher TPR at low FPR than LLM judge while using far less compute per trajecto","key_machinery":"The small transformer encoder that maps Terminal-Wrench trajectories onto a unit sphere where embedding distance approximates the L1 distance between reward and metadata signals.","core_discovery":"The authors show that a small transformer encoder trained to map trajectories to a unit sphere, where embedding distance approximates L1 distance between reward and metadata signals, enables a linear probe to detect reward hacking on the cleaned test split with AUC 0.9467 and TPR@5%FPR 0.8296. This matches the TW sanitized LLM-as-judge AUC of 0.9510 and exceeds its TPR@5%FPR of 0.7130 on the same information condition, at roughly four orders of magnitude lower per-trajectory cost. The encoder is not a pure behavior reader, as removing natural-language reasoning drops AUC to 0.6213.","pith_inferences":["Routine, low-cost scanning of large numbers of trajectories could become practical in ongoing training or evaluation loops.","Similar embedding-plus-probe pipelines might be tested on other misalignment signals beyond reward hacking.","The cost reduction could allow more trajectories to be audited in settings where LLM calls are currently prohibitive."],"forward_implications":["Reward hacking detection becomes feasible at much lower per-trajectory cost than LLM-based judging.","The linear probe achieves comparable AUC and higher TPR at 5 percent FPR than the LLM judge under the same information condition.","Detection performance requires the full input that includes natural-language reasoning, as performance falls sharply without it.","The embedding approach provides a scalable method for monitoring trajectories without repeated large-model calls."],"fun_headline_variants":["Small encoder matches LLM on reward hacking at 10000x lower cost","Unit sphere embeddings yield 0.9467 AUC for reward hacking","Small encoder exceeds LLM TPR on reward hacking at 10000x lower cost","Trajectory encoder matches LLM judge at four orders lower cost"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The cleaning of the test split does not introduce selection bias that favors the embedding method over the LLM baseline.","fun_headline_variants_meta":{"raw":{"variants":["Small encoder matches LLM on reward hacking at 10000x lower cost","Unit sphere embeddings yield 0.9467 AUC for reward hacking","Small encoder exceeds LLM TPR on reward hacking at 10000x lower cost","Trajectory encoder matches LLM judge at four orders lower cost"]},"model":"grok-4.3","cost_usd":0.013259,"raw_usage":{"total_tokens":5737,"prompt_tokens":652,"num_sources_used":0,"completion_tokens":74,"cost_in_usd_ticks":132587000,"prompt_tokens_details":{"text_tokens":652,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":5011,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":652,"tokens_out":74,"duration_ms":28192,"temperature":1.0,"reasoning_tokens":5011,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T17:19:28.148094+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Testing the linear probe on a freshly collected set of trajectories without the reported cleaning step and observing a substantial drop in AUC or TPR would show whether the reported detection performance holds.","supporting_citations":[],"review_version":1}