{"id":"abe1d370-6969-4af0-9671-1d004e53bb92","arxiv_id":"2607.22766","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"An inference-only, k-nearest-neighbor graph method approximates data value for LLM alignment data by measuring zero-shot versus one-shot log-likelihood shifts, and flags suspected label contradictions in HelpSteer2 and HH-RLHF.","lead":"This paper introduces a data-auditing pipeline that scores how much one training example helps or hurts a language model by measuring changes in the model's probability when the example is added to the prompt. It applies the method to two public alignment datasets and claims to have found thousands of label errors and flawed evaluation labels.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"δ(n_i) is asserted, not derived: an unweighted average of one-shot ΔLL values is not Shapley's coalition expectation unless all higher-order interactions vanish, and no retraining experiment validates the ICL surrogate.","rationale":"The reader is right that the Shapley claim is not supported: Appendix A asserts the approximations rather than deriving them, and no experiment validates ΔLL against actual retraining. My principal concern sharpens this: even if the ICL surrogate and the neighborhood truncation were granted, the first-order averaging step is not the Shapley expectation. Replacing a weighted sum over coalition subsets with one unweighted pairwise average is valid only under an additivity assumption that the paper never states or tests. This is an internal mathematical gap, not just a question of empirical calibration. The operational achievements—99.1% search-space reduction and the discovery of plausible label errors—are real and should be credited, but they do not establish the advertised mathematical foundation. The paper's own Section 5 limitation, where ΔLL is described as 'a mathematical anomaly filter rather than a standalone detector,' is consistent with treating the pipeline as a heuristic. A focused retraining-correlation study, or even a small counterexample to the averaging step, would settle whether the Shapley claim can be rescued.","tokens_in":15170,"tokens_out":6093,"duration_ms":57549,"concrete_test":"Run a small-scale validation: on a 1,000–2,000 record random subset of HelpSteer2 (or a smaller surrogate dataset), compute δ(n_i) with the published pipeline, then estimate ground-truth Data Shapley/leave-one-out values by fine-tuning the same base model (possibly a smaller 0.5B–1B model) on random coalitions and measuring validation loss or helpfulness-score prediction utility. Report the Spearman rank correlation between δ and the retraining-based values, and the precision of the top-100 flagged records against actual retraining effects. Also symbolically test the Appendix A claim: exhibit a simple 3-record additive utility game and a non-additive game, and check whether Eq. 3.4's average equals Shapley; if the rankings differ in the non-additive case, the approximation requires an unstated additivity condition.","verdict_should_be":"REJECT","load_bearing_attack":"Appendix A's 'theoretical derivation' is a list of three asserted analogies, not a theorem. The decisive unsupported step is item 3: exact Shapley φ_i is a weighted expectation over all coalitions S⊆D\\{i} of marginal utility v(S∪{i})−v(S), whereas the paper's δ(n_i) is an unweighted average of pairwise ΔLL_{k,i} over N_out(n_i). These quantities coincide only if the utility game is first-order/additive, i.e., if all higher-order interactions among training records vanish. That condition is nowhere stated, and it is implausible for LLM training data, where multiple similar records can jointly reinforce or conflict in nonlinear ways. Item 1's 'in-context learning as a zero-gradient surrogate for parameter updates' is likewise asserted without an error bound or any experiment comparing ΔLL to actual fine-tuning effects. Section 4 and Appendix D tune thresholds and measure LLM-judge agreement, but never measure whether δ ranks records the way retraining-based Shapley or leave-one-out values would. Thus the advertised 'mathematically grounded O(K) Shapley approximation' lacks both a valid derivation and empirical calibration. The audits may be useful heuristics for surfacing candidate label errors, but the central claim—that δ captures Shapley marginal contribution—is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an inference-only pipeline for auditing LLM alignment datasets. It embeds prompts, builds a directed k-NN graph, computes a directed edge score ΔLL_{i,j} as the conditional log-likelihood shift of target record i when neighbor j is inserted as a one-shot demonstration, and aggregates these scores into per-record metrics: exemplified score, exemplifying score δ(n_i), and one-of-a-kind score. The exemplifying score is claimed to approximate the Data Shapley value in O(K) time, with a theoretical derivation deferred to Appendix A. The pipeline then flags low-advantage records subject to a mutual-neighbor constraint and verifies candidates with an LLM judge. Case studies on HelpSteer2 and HH-RLHF report large reductions in the manual audit search space and thousands of claimed label errors or preference inversions, including a subset of the HH-RLHF evaluation split on which several models disagree with human labels at elevated rates.","tokens_in":15630,"tokens_out":3038,"duration_ms":30114,"significance":"If the central Shapley-approximation claim were valid, the pipeline would be a notable contribution: an O(K), inference-only method for record-level data valuation in generative LLMs, with concrete applications to dataset sanitation and benchmark auditing. The paper's strengths are its clear empirical workflow, the large-scale application to HelpSteer2 and HH-RLHF, the inclusion of ablations across embedding and scoring models, and the explicit attempt to reduce the search space for human auditors. However, the theoretical connection to Shapley is asserted rather than proven, the in-context-learning surrogate for parameter updates is not validated against retraining, and the final confirmation of label errors relies on another LLM rather than human ground truth. The empirical findings may still be useful as heuristics for surfacing candidate annotation errors, but the advertised 'mathematically grounded Shapley approximation' is not established by the manuscript.","major_comments":[{"comment":"The paper claims that δ(n_i) 'captures the Shapley marginal contribution in O(K)', but the derivation does not connect exact Shapley to the proposed estimator. Exact Shapley, as stated in Appendix A, is a weighted expectation over all coalitions S⊆D\\{i} with binomial weights 1 / C(n-1, |S|). In contrast, δ(n_i) is an unweighted average of pairwise ΔLL_{k,i} over N_out(n_i). These two quantities coincide only if the underlying utility game is first-order or additive, i.e., if all higher-order interactions among training records vanish. This condition is nowhere stated, and it is implausible for LLM training data, where multiple similar records can jointly reinforce or conflict in nonlinear ways. The unweighted average over a fixed neighborhood is therefore not a Shapley value, and the claim that the method is 'mathematically grounded' in Shapley is unsupported.","section":"Section 3.4 and Appendix A, item 3"},{"comment":"The first approximation in Appendix A asserts that one-shot in-context learning acts as a 'zero-gradient surrogate for parameter updates', but no error bound or empirical calibration is provided. The paper never measures whether ΔLL or δ(n_i) correlates with actual fine-tuning effects, leave-one-out retraining, or a retraining-based Shapley estimate. Appendix D tunes hyperparameters and evaluates LLM-judge agreement, but it does not validate the surrogate. Without such validation, δ(n_i) is a heuristic influence score, not a demonstrable approximation of the marginal contribution of a record under gradient-based training. This is a load-bearing issue because the entire O(K) Shapley claim rests on this surrogate.","section":"Section 3.4, Appendix A item 1, and Appendix D"},{"comment":"The evaluation-split result is partly circular. The 156 suspicious records are selected using the Qwen3.5-9B model's own log-likelihood shifts, and then the same model family (including the fine-tuned Qwen3.5-9B) is shown to disagree with human labels on those records. This is to some extent a constructed property of the selection procedure. The claim that these are 'objectively flawed' ground-truth labels is further weakened by the fact that the final confirmation is performed by an LLM judge (Gemini 3.1 Pro), not by human annotators; the manuscript acknowledges this in Section 5. To support the benchmark-vulnerability conclusion, the authors would need a control set matched on semantic difficulty or cluster structure, and human validation of at least a sample of the flagged records.","section":"Section 4.2.2 and Table 3"}],"minor_comments":[{"comment":"The phrase 'processing low human helpfulness score' should be 'possessing low human helpfulness score' or similar.","section":"Section 4.1"},{"comment":"The 'LLM Direct Validation' row reports disagreement rates on 3,944 records, but the preceding text describes a direct LLM evaluation of all 128,000 pairs. The selection of these 3,944 records is not clearly specified and should be stated explicitly.","section":"Table 3 and Section 4.2.2"},{"comment":"Reference [18] is titled 'Qwen3.5-omni technical report', while the text refers to Qwen3.5-9B as the scoring model; the relationship between the 9B model and the omni report should be clarified.","section":"References and Section 3.1"},{"comment":"The row for K=5 reports '0 confirmed contradictions' with a disagreement rate of 0%, but it is unclear whether the disagreement rate is computed on an empty set or on a set of records that were not confirmed; this should be stated.","section":"Appendix D, Table 5"}],"recommendation":"reject","confidential_remarks":"The empirical pipeline may have value as a heuristic auditing tool, but the central Shapley-approximation claim is not supported by the derivation in Appendix A, and the evaluation-split results are partially circular due to selection by the same model whose disagreement is then measured. The lack of human validation for the 'confirmed' label errors is a further concern for a paper whose main contribution is data-quality auditing. These issues are load-bearing and would require either removing the Shapley framing or substantially reworking the theory and validation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: if you want a cheap way to find candidate label errors in preference or instruction data, this pipeline is worth reading. But the headline claim that δ(n_i) is a Shapley approximation doesn't survive contact with the math. Appendix A lists three analogies; it never shows how an unweighted average of one-shot ΔLL values equals the Shapley expectation. That equality would require all higher-order interactions among training records to vanish, which is implausible for LLM data and is never stated, let alone tested.\n\nWhat's genuinely new: combining k-NN semantic graphs with zero-shot vs one-shot conditional log-likelihood shifts to get record-level influence scores. I haven't seen that exact combination in the data valuation literature. The empirical work is also coherent. They run the pipeline on HelpSteer2 and HH-RLHF, report ablations over embedding/scorer models and hyperparameters, and include concrete examples that look like real annotation failures. Cutting the manual audit surface from 8,434 records to 77 candidate pairs is a real operational win.\n\nWhere it gets shaky: the Shapley bridge. The marginal contribution of a record in a coalition game is v(S∪{i})−v(S); the paper replaces v with one-shot log-likelihood on a single neighbor. Even if I accept that one-shot prompting is a proxy for parameter updates—and that's an asserted analogy—averaging pairwise shifts over out-degree neighbors is not the Shapley expectation. It's a different statistic. No experiment compares δ to retraining-based Shapley or leave-one-out retraining, so the central calibration is missing. Also, the 'confirmed' contradictions are verified by Gemini, not by human annotators. The paper admits this limitation and calls them potential contradictions, but then Table 3 treats the LLM verdicts as ground truth for the disagreement-rate comparison. That's a weaker form of evidence than it appears.\n\nNone of this makes the pipeline useless. As a heuristic for triaging suspicious records, it's plausible and cheap. But the intro and abstract oversell it as mathematically grounded. If they revised to present δ as an influence heuristic and added a calibration study against retraining-based values, the paper would be solid.\n\nVerdict for you: send it to review if you're on that program committee, but expect a reject in current form. It deserves referees' time because the empirical work is substantial and the problem is important; the theory needs a major fix.","headline":"Useful heuristic for triaging label errors in LLM alignment data, but the Shapley framing is asserted, not derived, and the LLM-judge validation is not external ground truth.","tokens_in":15959,"tokens_out":2764,"would_cite":false,"duration_ms":22972,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that per-record data value for LLMs can be measured from one-shot in-context log-likelihood shifts, replacing retraining with a few forward passes.","keywords":["data valuation","Shapley approximation","in-context learning","log-likelihood shift","dataset auditing","RLHF preferences","benchmark integrity","semantic k-NN graph"],"falsifier":"Train a small model multiple times on a small corpus, each run holding out a different record flagged by the pipeline — the most negative $\\delta$, the most positive $\\delta$, and random controls — and compare validation loss or preference accuracy; if the held-out records do not move performance in the direction the surrogate predicts, the one-shot approximation is not measuring marginal contribution.","tokens_in":14971,"feed_emoji":"🔍","tokens_out":13923,"duration_ms":106434,"temperature":0.7,"pith_summary":"The paper claims that the per-record value of a training example in an LLM alignment corpus can be approximated without retraining: measure how much adding that record as a one-shot in-context example shifts the conditional log-likelihood of its semantic neighbors, and average those shifts over the record's out-degree neighborhood. This turns the exponential Shapley computation into $O(K)$ forward passes per record. If the approximation holds, dataset auditing that once required thousands of fine-tuning runs or millions of judge calls can be done with embeddings and lightweight scoring, and it can pinpoint which human labels are wrong. Applying the pipeline to HelpSteer2 reduced the manual audit surface by 99.1% and uncovered factually hallucinated, format-violating, and superficially scored records; applying it to the HH-RLHF training and evaluation splits surfaced thousands of safety and factual preference inversions, including benchmarks where capable models are penalized for choosing the safer or more helpful response.","feed_headline":"No-retraining audit flags thousands of bad human preference labels","feed_subtitle":"One-shot log-likelihood probes replace retraining, cutting the audit surface by 99.1 percent and exposing benchmark label errors.","key_machinery":"The load-bearing object is the directed influence graph $G=(V,E)$ built from semantic $k$-NN retrieval. For each ordered neighbor pair, a reference LLM computes the conditional log-likelihood shift $\\Delta\\mathrm{LL}_{i,j}=\\mathrm{LL}_{\\mathrm{probe}}(i,j)-\\mathrm{LL}_{\\mathrm{base}}(i)$: how much the one-shot demonstration $(x_j,y_j)$ changes the log-likelihood of target response $y_i$. The edge $j\\to i$ carries this weight. The exemplifying score $\\delta(n_i)$ averages the weights on outgoing edges and is the Shapley proxy; the exemplified score $e(n_i)$ averages incoming influence. Both are converted to local Z-scores ($\\mathrm{Adv}_\\delta$, $\\mathrm{Adv}_e$) using the in-degree neighborhood as baseline, and a mutual-nearest-neighbor constraint keeps only records consistently penalized by multiple mutual neighbors, which the paper calls structural contradictions.","core_discovery":"The paper's central claim is that the exemplifying score $\\delta(n_i)$ — the average one-shot predictive influence that record $i$ exerts on the records which retrieve it as a nearest neighbor — is a faithful approximation to that record's Shapley marginal contribution under the target LLM's learning dynamics. The reasoning is that a one-shot in-context demonstration acts as a zero-gradient surrogate for a parameter update; that contributions to semantically unrelated records are zero, so the coalition space can be truncated to the out-degree $k$-NN neighborhood; and that averaging first-order 1-shot interactions captures the marginal signal. From these scores the pipeline derives localized advantage metrics, normalized against the in-degree neighborhood, to mark records that receive perfect human labels yet degrade their neighbors' predictive probability. On HelpSteer2 and HH-RLHF, the pipeline reports 18 confirmed contradiction pairs and 10 unique perfect-score records in HelpSteer2, thousands of preference-label inversions in HH-RLHF, and model disagreement rates of 62–75% on the verified evaluation anomalies.","pith_inferences":["A testable extension is to feed the continuous advantage scores back into fine-tuning as soft loss weights, down-weighting gradient-conflicting records instead of deleting them; the paper lists this as future work.","Because the pipeline never touches gradients, it could be run continuously on streaming preference data, flagging label drift as semantic neighborhoods shift.","The same $\\Delta\\mathrm{LL}$ machinery could be repurposed to synthesize hard contrastive negatives for preference training, using conflicting demonstrations as adversarial examples.","If the surrogate survives a retraining check, the method should transfer beyond alignment data to any generative task with a well-defined conditional likelihood, such as summarization or code-generation audit."],"forward_implications":["If $\\delta(n_i)$ is a valid Shapley proxy, per-record data valuation for generative LLMs becomes an inference-time operation: one embedding index plus a few forward passes per neighborhood, with no gradient updates.","Dataset sanitization can concentrate expert arbitration on a tiny candidate set — the HelpSteer2 audit shrinks from 8,434 records to 77 pairs — making human re-annotation feasible at corpus scale.","Evaluation benchmarks can be audited with the same machinery as training data, so flawed ground-truth labels can be relabeled before the benchmark is used to compare models.","The mathematical filter and the LLM arbitrator do complementary work: the influence metrics cut the search space by over 99%, while the LLM raises precision; neither alone concentrates disagreement as tightly as the full pipeline."],"supporting_citations":[{"why":"Defines the Shapley value whose marginal-contribution structure the pipeline approximates.","marker":"[14]"},{"why":"Introduces Data Shapley for machine learning and establishes the exact exponential retraining cost the pipeline avoids.","marker":"[5]"},{"why":"Shows exact Shapley values for KNN models can be computed efficiently, the point-wise attribution result this work extends to LLM generation.","marker":"[7]"},{"why":"Cluster Shapley is the coarse-grained baseline that sacrifices record-level granularity, which this pipeline claims to restore.","marker":"[22]"},{"why":"Language Model Arithmetic requires one fine-tune per data source, the retraining burden this inference-only method bypasses.","marker":"[15]"},{"why":"Supplies the HH-RLHF training and evaluation splits used to find preference inversions and benchmark vulnerabilities.","marker":"[2]"},{"why":"Supplies the HelpSteer2 scored records used to isolate perfect-score annotations that contradict their neighborhoods.","marker":"[20]"},{"why":"Embedding model used to build the semantic k-NN neighborhoods that define the coalition space.","marker":"[24]"},{"why":"Scoring LLM whose zero-shot and one-shot conditional log-likelihoods produce the influence weights.","marker":"[18]"},{"why":"Approximate nearest-neighbor search used to scale k-NN retrieval to dataset size.","marker":"[10]"}],"fun_headline_variants":["No-retrain audit uncovers thousands of flipped preference labels","Influence scores expose benchmark label failures in RLHF","One-shot probes cut audit search by 99.1% and find errors","Shapley-free influence pipeline reveals hidden data conflicts","Auditing LLM data without retraining finds label inversions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a one-shot in-context log-likelihood shift faithfully measures the marginal contribution that record would have under real gradient-based training, and that records outside the semantic neighborhood contribute zero; the paper validates this indirectly through disagreement-rate spikes rather than by comparing the surrogate to actual retraining outcomes.","fun_headline_variants_meta":{"raw":{"variants":["No-retrain audit uncovers thousands of flipped preference labels","Influence scores expose benchmark label failures in RLHF","One-shot probes cut audit search by 99.1% and find errors","Shapley-free influence pipeline reveals hidden data conflicts","Auditing LLM data without retraining finds label inversions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000273,"raw_usage":{"total_tokens":1696,"prompt_tokens":1065,"completion_tokens":631,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":681,"completion_tokens_details":{"reasoning_tokens":547}},"tokens_in":681,"tokens_out":631,"duration_ms":5946,"temperature":1.0,"reasoning_tokens":547,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:28:57.262111+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a small model multiple times on a small corpus, each run holding out a different record flagged by the pipeline — the most negative $\\delta$, the most positive $\\delta$, and random controls — and compare validation loss or preference accuracy; if the held-out records do not move performance in the direction the surrogate predicts, the one-shot approximation is not measuring marginal contribution.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Shapley value whose marginal-contribution structure the pipeline approximates."},{"cited_title":"Shapley-based Data Valuation for LLM Alignment via Sequential Preference Optimization","cited_arxiv_id":"2512.15765","evidence_quote":"Language Model Arithmetic requires one fine-tune per data source, the retraining burden this inference-only method bypasses."}],"review_version":2}