Pith. sign in

REVIEW 3 major objections 4 minor 36 references

Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that per-record data value for LLMs can be measured from one-shot in-context log-likelihood shifts, replacing retraining with a few forward passes.

desk verdict Useful heuristic for triaging label errors in LLM alignment data, but the Shapley framing is asserted, not derived, and the LLM-judge validation is not external ground truth. read the letter →

arxiv 2607.22766 v1 pith:QVCTTPUV submitted 2026-07-24 cs.LG cs.AI

Yunting Song , Matthew Watson , Peter Grabowski , Jun Qin This is my paper · ORCID
classification cs.LGcs.AI
keywords datavaluationShapleyapproximationin-contextlearninglog-likelihoodshiftdatasetauditingRLHFpreferencesbenchmarkintegritysemantick-NNgraph
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the per-record value of a training example in an LLM alignment corpus can be approximated without retraining: measure how much adding that record as a one-shot in-context example shifts the conditional log-likelihood of its semantic neighbors, and average those shifts over the record's out-degree neighborhood. This turns the exponential Shapley computation into $O(K)$ forward passes per record. If the approximation holds, dataset auditing that once required thousands of fine-tuning runs or millions of judge calls can be done with embeddings and lightweight scoring, and it can pinpoint which human labels are wrong. Applying the pipeline to HelpSteer2 reduced the manual audit surface by 99.1% and uncovered factually hallucinated, format-violating, and superficially scored records; applying it to the HH-RLHF training and evaluation splits surfaced thousands of safety and factual preference inversions, including benchmarks where capable models are penalized for choosing the safer or more helpful response.

What carries the argument

The load-bearing object is the directed influence graph $G=(V,E)$ built from semantic $k$-NN retrieval. For each ordered neighbor pair, a reference LLM computes the conditional log-likelihood shift $\Delta\mathrm{LL}_{i,j}=\mathrm{LL}_{\mathrm{probe}}(i,j)-\mathrm{LL}_{\mathrm{base}}(i)$: how much the one-shot demonstration $(x_j,y_j)$ changes the log-likelihood of target response $y_i$. The edge $j\to i$ carries this weight. The exemplifying score $\delta(n_i)$ averages the weights on outgoing edges and is the Shapley proxy; the exemplified score $e(n_i)$ averages incoming influence. Both are converted to local Z-scores ($\mathrm{Adv}_\delta$, $\mathrm{Adv}_e$) using the in-degree neighborhood as baseline, and a mutual-nearest-neighbor constraint keeps only records consistently penalized by multiple mutual neighbors, which the paper calls structural contradictions.

What would settle it

Train a small model multiple times on a small corpus, each run holding out a different record flagged by the pipeline — the most negative $\delta$, the most positive $\delta$, and random controls — and compare validation loss or preference accuracy; if the held-out records do not move performance in the direction the surrogate predicts, the one-shot approximation is not measuring marginal contribution.

Watch

Extended reading notes

Core claim

The paper's central claim is that the exemplifying score $\delta(n_i)$ — the average one-shot predictive influence that record $i$ exerts on the records which retrieve it as a nearest neighbor — is a faithful approximation to that record's Shapley marginal contribution under the target LLM's learning dynamics. The reasoning is that a one-shot in-context demonstration acts as a zero-gradient surrogate for a parameter update; that contributions to semantically unrelated records are zero, so the coalition space can be truncated to the out-degree $k$-NN neighborhood; and that averaging first-order 1-shot interactions captures the marginal signal. From these scores the pipeline derives localized advantage metrics, normalized against the in-degree neighborhood, to mark records that receive perfect human labels yet degrade their neighbors' predictive probability. On HelpSteer2 and HH-RLHF, the pipeline reports 18 confirmed contradiction pairs and 10 unique perfect-score records in HelpSteer2, thousands of preference-label inversions in HH-RLHF, and model disagreement rates of 62–75% on the verified evaluation anomalies.

Load-bearing premise

The load-bearing premise is that a one-shot in-context log-likelihood shift faithfully measures the marginal contribution that record would have under real gradient-based training, and that records outside the semantic neighborhood contribute zero; the paper validates this indirectly through disagreement-rate spikes rather than by comparing the surrogate to actual retraining outcomes.

Editorial extensions

If this is right

  • If $\delta(n_i)$ is a valid Shapley proxy, per-record data valuation for generative LLMs becomes an inference-time operation: one embedding index plus a few forward passes per neighborhood, with no gradient updates.
  • Dataset sanitization can concentrate expert arbitration on a tiny candidate set — the HelpSteer2 audit shrinks from 8,434 records to 77 pairs — making human re-annotation feasible at corpus scale.
  • Evaluation benchmarks can be audited with the same machinery as training data, so flawed ground-truth labels can be relabeled before the benchmark is used to compare models.
  • The mathematical filter and the LLM arbitrator do complementary work: the influence metrics cut the search space by over 99%, while the LLM raises precision; neither alone concentrates disagreement as tightly as the full pipeline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to feed the continuous advantage scores back into fine-tuning as soft loss weights, down-weighting gradient-conflicting records instead of deleting them; the paper lists this as future work.
  • Because the pipeline never touches gradients, it could be run continuously on streaming preference data, flagging label drift as semantic neighborhoods shift.
  • The same $\Delta\mathrm{LL}$ machinery could be repurposed to synthesize hard contrastive negatives for preference training, using conflicting demonstrations as adversarial examples.
  • If the surrogate survives a retraining check, the method should transfer beyond alignment data to any generative task with a well-defined conditional likelihood, such as summarization or code-generation audit.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes an inference-only pipeline for auditing LLM alignment datasets. It embeds prompts, builds a directed k-NN graph, computes a directed edge score ΔLL_{i,j} as the conditional log-likelihood shift of target record i when neighbor j is inserted as a one-shot demonstration, and aggregates these scores into per-record metrics: exemplified score, exemplifying score δ(n_i), and one-of-a-kind score. The exemplifying score is claimed to approximate the Data Shapley value in O(K) time, with a theoretical derivation deferred to Appendix A. The pipeline then flags low-advantage records subject to a mutual-neighbor constraint and verifies candidates with an LLM judge. Case studies on HelpSteer2 and HH-RLHF report large reductions in the manual audit search space and thousands of claimed label errors or preference inversions, including a subset of the HH-RLHF evaluation split on which several models disagree with human labels at elevated rates.

Significance. If the central Shapley-approximation claim were valid, the pipeline would be a notable contribution: an O(K), inference-only method for record-level data valuation in generative LLMs, with concrete applications to dataset sanitation and benchmark auditing. The paper's strengths are its clear empirical workflow, the large-scale application to HelpSteer2 and HH-RLHF, the inclusion of ablations across embedding and scoring models, and the explicit attempt to reduce the search space for human auditors. However, the theoretical connection to Shapley is asserted rather than proven, the in-context-learning surrogate for parameter updates is not validated against retraining, and the final confirmation of label errors relies on another LLM rather than human ground truth. The empirical findings may still be useful as heuristics for surfacing candidate annotation errors, but the advertised 'mathematically grounded Shapley approximation' is not established by the manuscript.

major comments (3)
  1. [Section 3.4 and Appendix A, item 3] The paper claims that δ(n_i) 'captures the Shapley marginal contribution in O(K)', but the derivation does not connect exact Shapley to the proposed estimator. Exact Shapley, as stated in Appendix A, is a weighted expectation over all coalitions S⊆D\{i} with binomial weights 1 / C(n-1, |S|). In contrast, δ(n_i) is an unweighted average of pairwise ΔLL_{k,i} over N_out(n_i). These two quantities coincide only if the underlying utility game is first-order or additive, i.e., if all higher-order interactions among training records vanish. This condition is nowhere stated, and it is implausible for LLM training data, where multiple similar records can jointly reinforce or conflict in nonlinear ways. The unweighted average over a fixed neighborhood is therefore not a Shapley value, and the claim that the method is 'mathematically grounded' in Shapley is unsupported.
  2. [Section 3.4, Appendix A item 1, and Appendix D] The first approximation in Appendix A asserts that one-shot in-context learning acts as a 'zero-gradient surrogate for parameter updates', but no error bound or empirical calibration is provided. The paper never measures whether ΔLL or δ(n_i) correlates with actual fine-tuning effects, leave-one-out retraining, or a retraining-based Shapley estimate. Appendix D tunes hyperparameters and evaluates LLM-judge agreement, but it does not validate the surrogate. Without such validation, δ(n_i) is a heuristic influence score, not a demonstrable approximation of the marginal contribution of a record under gradient-based training. This is a load-bearing issue because the entire O(K) Shapley claim rests on this surrogate.
  3. [Section 4.2.2 and Table 3] The evaluation-split result is partly circular. The 156 suspicious records are selected using the Qwen3.5-9B model's own log-likelihood shifts, and then the same model family (including the fine-tuned Qwen3.5-9B) is shown to disagree with human labels on those records. This is to some extent a constructed property of the selection procedure. The claim that these are 'objectively flawed' ground-truth labels is further weakened by the fact that the final confirmation is performed by an LLM judge (Gemini 3.1 Pro), not by human annotators; the manuscript acknowledges this in Section 5. To support the benchmark-vulnerability conclusion, the authors would need a control set matched on semantic difficulty or cluster structure, and human validation of at least a sample of the flagged records.
minor comments (4)
  1. [Section 4.1] The phrase 'processing low human helpfulness score' should be 'possessing low human helpfulness score' or similar.
  2. [Table 3 and Section 4.2.2] The 'LLM Direct Validation' row reports disagreement rates on 3,944 records, but the preceding text describes a direct LLM evaluation of all 128,000 pairs. The selection of these 3,944 records is not clearly specified and should be stated explicitly.
  3. [References and Section 3.1] Reference [18] is titled 'Qwen3.5-omni technical report', while the text refers to Qwen3.5-9B as the scoring model; the relationship between the 9B model and the omni report should be clarified.
  4. [Appendix D, Table 5] The row for K=5 reports '0 confirmed contradictions' with a disagreement rate of 0%, but it is unclear whether the disagreement rate is computed on an empty set or on a set of records that were not confirmed; this should be stated.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the Shapley link is an explicitly stated approximation and the benchmark audit, while not externally verified, does not reduce its output to its input by construction.

full rationale

The paper's central Shapley claim is introduced as an approximation, not as an identity: Section 3.4 states that δ(n_i) 'approximates the Data Shapley mathematically' by treating in-context learning as a zero-gradient surrogate, truncating coalitions to N_out(n_i), and using first-order averaging. Appendix A lists these as 'three foundational approximations' and asserts that averaging pairwise shifts 'transforms the combinatorial Shapley equation into a linear inference computation'; no equation reduces the exact φ_i to δ, and δ is not fitted to φ. An unproved approximation is a correctness risk, but not circularity. The evaluation-split experiment selects records via the reference model's ΔLL and then reports elevated disagreement rates on those records; this is a selection and external-validity limitation, and the paper explicitly concedes in Section 5 that final verification relies on an LLM evaluator rather than human ground truth. Elevated disagreement is correlated with, but not definitionally equal to, the selection score, and the paper checks two additional model families. There is no load-bearing self-citation chain, no fitted parameter renamed as a prediction, and no equation that is equivalent to its input by construction. The derivation chain therefore contains no significant circular step.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claim rests on several unproven domain assumptions: in-context log-likelihood shifts stand in for gradient updates, k-NN neighborhoods localize Shapley coalitions, semantic similarity tracks functional equivalence, and LLM judges provide ground truth. The free parameters K, tau_adv, MNN count, and cosine threshold are hand-chosen and tuned on the target datasets. No external retraining-based benchmark is used to justify the Shapley bridge.

free parameters (4)
  • Neighborhood size K = 15 (default)
    Controls the size of the localized coalition space; sensitivity analysis shows K=5 yields zero confirmed contradictions, and K in [10,25] is stable. It is a user-chosen hyperparameter, not fitted to an external ground truth.
  • Advantage threshold tau_adv = -2.0 (local Z-score)
    Determines which nodes are flagged as anomalies; stricter thresholds raise disagreement rates but shrink the candidate pool. The default is selected as a balance between yield and precision.
  • Minimum degrading mutual neighbors = 5 (default)
    MNN constraint used to require that multiple mutual neighbors degrade the target; sensitivity is small on HH-RLHF eval but the value is still hand-chosen.
  • Response cosine similarity threshold for twin verification = 0.5
    Used in HelpSteer2 Stage 2 to find behaviorally similar low-score neighbors; the cut-off is arbitrary and no sensitivity analysis is reported for it.
assumptions (6)
  • ad hoc to paper In-context learning is a surrogate for parameter updates when measuring marginal data contribution.
    Appendix A item 1 states prepending a record as a one-shot demonstration acts as a transient inference-time parameter update. This is a domain assumption that is never validated against actual retraining.
  • ad hoc to paper Marginal contributions of a record outside its k-NN out-degree neighborhood are zero, so coalition truncation loses nothing.
    Appendix A item 2 asserts topically unrelated records have zero contribution and the coalition can be restricted to Nout(ni). No empirical evidence is given for this in generative LLM training.
  • ad hoc to paper First-order one-shot sub-sampling approximates the full Shapley expectation over subsets.
    Appendix A item 3 replaces the combinatorial average over subsets with a single 0-shot versus 1-shot comparison. This is the load-bearing step that turns a heuristic into a claimed Shapley approximation.
  • domain assumption Semantic k-NN neighborhoods correspond to functionally equivalent contexts, so conflicting predictions in a cluster indicate annotation contradictions.
    Section 3.1 and Section 3.5 assume embedding similarity captures functional proximity; the paper itself concedes embeddings can cluster topically related but functionally distinct queries, which is why a final verification stage is needed.
  • domain assumption An LLM evaluator is a reliable arbiter of whether a flagged pair is a true annotator contradiction.
    Section 3.5 and Appendix E use Gemini as the expert evaluator. The limitations section states LLM judges are subjective and non-deterministic, so the 'confirmed' counts are not human ground truth.
  • domain assumption The embedding model and scoring model choices are adequate for the target datasets.
    Section 4 relies on Qwen3-Embedding-4B and Qwen3.5-9B. The ablation in Appendix C shows results shift when the scorer is changed, indicating sensitivity to this choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation." pith.science (2026). https://pith.science/paper/QVCTTPUV

@misc{pith2026260722766,
  author       = {Pith},
  title        = {Pith review of: Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QVCTTPUV}},
  note         = {Machine review of arXiv:2607.22766}
}
read the original abstract

The alignment of Large Language Models (LLMs) is increasingly bottlenecked by data quality. As datasets scale, massive preference and instruction-tuning corpora inevitably accumulate hidden structural contradictions, safety risks, and systemic human annotation errors. Standard dataset auditing methods, such as semantic deduplication or LLM-as-a-judge, struggle to capture the actual predictive impact of individual records and often miss deep functional rule clashes. To address this, we introduce a scalable, inference-only data valuation pipeline that approximates the Shapley value without iterative model retraining. By mapping semantic k-NN neighborhoods into a directed graph, our framework evaluates data utility directly through a reference LLM's probability distribution using zero-shot and one-shot conditional log-likelihood shifts. Our pipeline then translates these predictive influence scores into localized advantage metrics to isolate gradient-conflicting records. We demonstrate the pipeline's efficacy in sanitizing two heavily vetted alignment datasets. First, applying our pipeline to the HelpSteer2 dataset reduced the manual audit search space by 99.1%, successfully uncovering falsely-labeled records across diverse failure modes. Second, applying our automated audit strategy to Anthropic's HH-RLHF training and evaluation splits identified thousands of hidden safety and factual preference inversions. Crucially, by extending this audit to the evaluation split, we expose severe vulnerabilities in current benchmark integrity: highly capable models frequently predict the safer or more helpful response, only to be penalized by objectively flawed human ground-truth labels. Overall, our work provides a mathematically grounded, highly efficient diagnostic tool to uncover human label failures, sanitize evaluation benchmarks, and ensure the integrity of LLM alignment data.

Figures

Figures reproduced from arXiv: 2607.22766 by the authors.

Figure 1
Figure 1. Workflow of the proposed influence-based data valuation pipeline. Phase 1 constructs a [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Template for the zero-shot and one-shot prompt given a base prompt and one of its [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 16 canonical work pages

  1. [1]

    Semd- edup: Data-efficient learning at web-scale through semantic deduplication.arXiv preprint arXiv:2303.09540, 2023

    Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos. Semd- edup: Data-efficient learning at web-scale through semantic deduplication.arXiv preprint arXiv:2303.09540, 2023. URLhttps://arxiv.org/abs/2303.09540

    arXiv 2023

  2. [3]

    BGE M3- Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation.arXiv preprint arXiv:2402.03216, 2024

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. BGE M3- Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation.arXiv preprint arXiv:2402.03216, 2024. URL https://arxiv. org/abs/2402.03216

    arXiv 2024

  3. [4]

    Alpagasus: Training a better alpaca with fewer data.arXiv preprint arXiv:2307.08701, 2023

    Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training a better alpaca with fewer data.arXiv preprint arXiv:2307.08701, 2023. URL https://arxiv.org/abs/2307.08701

    arXiv 2023

  4. [5]

    Data shapley: Equitable valuation of data for machine learning

    Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. InInternational conference on machine learning, pages 2242–2251. PMLR, 2019

    2019

  5. [6]

    Textbooks are all you need.arXiv preprint arXiv:2306.11644, 2023

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need.arXiv preprint arXiv:2306.11644, 2023. URL http://arxiv. org/abs/2306.11644

    arXiv 2023

  6. [7]

    Efficient task-specific data valuation for nearest neighbor algorithms.arXiv preprint arXiv:1908.08619, 2019

    Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nezihe Merve Gurel, Bo Li, Ce Zhang, Costas J Spanos, and Dawn Song. Efficient task-specific data valuation for nearest neighbor algorithms.arXiv preprint arXiv:1908.08619, 2019. URL https://arxiv.org/abs/1908. 08619

    arXiv 1908

  7. [8]

    Vineppo: Accurate credit assignment in RL for LLM mathematical reasoning

    Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Accurate credit assignment in RL for LLM mathematical reasoning. InThe 4th Workshop on Mathematical Reasoning and AI at NeurIPS’24, 2024

    work page 2024

  8. [9]

    Deduplicating training data makes language mod- els better

    Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language mod- els better. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8424–8445, 2022

    2022

Show all 36 references
  1. [10]

    Yu A Malkov and Dmitry A Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs.IEEE transactions on pattern analysis and machine intelligence, 42(4):824–836, 2018

    2018

  2. [11]

    WaKA: Data Attribution using K-Nearest Neighbors and Membership Privacy Principles.arXiv preprint arXiv:2411.01357, 2024

    Patrick Mesana, Clément Bénesse, Hadrien Lautraite, Gilles Caporossi, and Sébastien Gambs. WaKA: Data Attribution using K-Nearest Neighbors and Membership Privacy Principles.arXiv preprint arXiv:2411.01357, 2024. URLhttps://arxiv.org/abs/2411.01357

    2024 arXiv

  3. [12]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730...

    2022

  4. [13]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741, 2023

    2023

  5. [14]

    Lloyd S. Shapley. A value for n-person games. In Harold W. Kuhn and Albert W. Tucker, editors,Contributions to the Theory of Games II, volume 28 ofAnnals of Mathematics Studies, pages 307–317. Princeton University Press, Princeton, NJ, 1953

    1953

  6. [15]

    Data Valuation for LLM Fine- Tuning: Efficient Shapley Value Approximation via Language Model Arithmetic.arXiv preprint arXiv:2512.15765, 2025

    Mélissa Tamine, Otmane Sakhi, and Benjamin Heymann. Data Valuation for LLM Fine- Tuning: Efficient Shapley Value Approximation via Language Model Arithmetic.arXiv preprint arXiv:2512.15765, 2025. URLhttps://arxiv.org/abs/2512.15765

    2025 arXiv

  7. [16]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023. URL https: //arxiv.org/abs...

    2023 arXiv

  8. [17]

    Gemma 3 technical report.arXiv preprint arXiv:2503.19786, March 2025

    Gemma Team. Gemma 3 technical report.arXiv preprint arXiv:2503.19786, March 2025. URL https://arxiv.org/abs/2503.19786

    2025 arXiv

  9. [18]

    Qwen3.5-omni technical report.arXiv preprint arXiv:2604.15804, 2026

    Qwen Team. Qwen3.5-omni technical report.arXiv preprint arXiv:2604.15804, 2026. URL https://arxiv.org/abs/2604.15804

    2026 arXiv

  10. [19]

    Large language models are not fair evaluators

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, et al. Large language models are not fair evaluators. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap...

    2024

  11. [20]

    HelpSteer 2: Open-source dataset for training top-performing reward models.Advances in Neural Information Processing Systems, 37:1474–1501, 2024

    Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh N Sreedhar, and Oleksii Kuchaiev. HelpSteer 2: Open-source dataset for training top-performing reward models.Advances in Neural Information Processing Systems, 37:1474–1501, 2024

    2024

  12. [21]

    Tokenshapley: Token level context attribution with shapley value

    Yingtai Xiao, Yuqing Zhu, Sirat Samyoun, Wanrong Zhang, Jiachen T Wang, and Jian Du. Tokenshapley: Token level context attribution with shapley value. InFindings of the Association for Computational Linguistics: ACL 2025, pages 3882–3894, 2025

    2025

  13. [22]

    Fair Document Valuation in LLM Summaries via Shapley Values.arXiv preprint arXiv:2505.23842, 2025

    Zikun Ye and Hema Yoganarasimhan. Fair Document Valuation in LLM Summaries via Shapley Values.arXiv preprint arXiv:2505.23842, 2025. URL https://arxiv.org/abs/ 2505.23842

    2025 arXiv

  14. [23]

    Localized Data Shapley: Accelerating Valuation for Nearest Neighbor Algorithms

    Guangyi Zhang, Yanhao Wang, Chengliang Chai, Qiyu Liu, and Wei Wang. Localized Data Shapley: Accelerating Valuation for Nearest Neighbor Algorithms. InAdvances in Neural Information Processing Systems, 2025

    2025

  15. [24]

    Qwen3 embedding: Advancing text embedding and reranking through foundation models.arXiv preprint arXiv:2506.05176, 2025

    Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 embedding: Advancing text embedding and reranking through foundation models.arXiv preprint arXiv:2506.05176, 2025. URL https://arxiv.org/ab...

    2025 arXiv

  16. [25]

    Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information processing systems, 36:46595–46623, 2023

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information processing systems, 36:46595–46623, 2023

    2023

  17. [26]

    Lima: Less is more for alignment.Advances in Neural Information Processing Systems, 36:55006–55021, 2023

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment.Advances in Neural Information Processing Systems, 36:55006–55021, 2023. 11 A Theoretical Justification: Bridging Exact...

    2023

  18. [28]

    For generative LLMs, prepending recordi as a one-shot demonstration acts as a transient, inference-time parameter update

    In-Context Learning as a Surrogate for Parameter Updates:Exact Shapley defines marginal utility as v(S∪{i})−v(S) , where v requires retraining a model from scratch. For generative LLMs, prepending recordi as a one-shot demonstration acts as a transient, inference-time paramete...

  19. [29]

    Truncation of the Coalition Space:The standard Shapley equation iterates overallsubsets in the datasetD. However, in massive natural language corpora, the marginal contribution of recordi to a topically unrelated recordj approaches zero (e.g., introducing a math problem does n...

  20. [30]

    By measuring the 1-shot versus 0-shot interaction, we isolate the pure, unconditioned marginal signal of the record

    First-Order (1-Shot) Sub-sampling:Rather than computing the exponential combinations of subset sizes|S|within the local neighborhood, we apply a first-order approximation. By measuring the 1-shot versus 0-shot interaction, we isolate the pure, unconditioned marginal signal of ...

  21. [31]

    Did both models fail in the exact same way? (e.g., both missed a requested format, both provided questions without answers)

    Identify the Shared Error:Read both inputs and responses. Did both models fail in the exact same way? (e.g., both missed a requested format, both provided questions without answers)

  22. [32]

    Evaluate Annotator A (The High Score):Based on the true quality of Record A, did Annotator A lazily miss the shared error and wrongly give Record A a high score?

  23. [33]

    Evaluate Annotator B (The Low Score):Based on the true quality of Record B, did Annotator B correctly identify the error, making their low score justified, or did they unfairly penalize it?

  24. [34]

    reasoning

    Render a Verdict:Is this a true annotator contradiction where one human missed a critical flaw that the other caught? You must output exclusively in the following JSON format without Markdown wrapping. Do not include quotes around the boolean value: { "reasoning": "First, stat...

  25. [35]

    Use the [Guidelines] (e.g., Harmlessness > Helpfulness)

    Independent Label Audit:For each record, independently assess if the [Target Response] (the chosen option) is truly better than the alternative. Use the [Guidelines] (e.g., Harmlessness > Helpfulness)

  26. [36]

    If Record 1 favors behavior X over Y , but Record 2 favors Y over X in a functionally similar context, a contradiction exists IF one of those choices is objectively wrong

    Detect Judgment Conflict:Compare the decisions. If Record 1 favors behavior X over Y , but Record 2 favors Y over X in a functionally similar context, a contradiction exists IF one of those choices is objectively wrong

  27. [37]

    Road Condition Reporting: A Literature Review

    Threshold for Certainty:If both options are poor, or if the choice is a ’toss-up’ based on subjective preference, do NOT mark it as a contradiction. Only flag cases where a label should strictly be flipped to maintain dataset integrity. CRITICAL OUTPUT: •is_contradiction: Set ...

  28. [2022]

    URLhttps://arxiv.org/abs/2204.05862

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.