Prompting Beats Fine-Tuning: Generative Expected Value Scoring for Statutory Term Retrieval

Alvin Wang; Jaromir Savelka

REVIEW 3 major objections 6 minor 30 references

Prompting Beats Fine-Tuning: Generative Expected Value Scoring for Statutory Term Retrieval

T0 review · 3 major / 6 minor · reviewed 2026-07-11 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Zero-shot prompting with expected-value scoring ranks statutory explanatory sentences better than fine-tuned encoders, beating prior state of the art.

desk verdict Solid empirical legal-IR update: EV prompting of frontier decoders beats ModernBERT and prior SOTA on the established statutory-term sentence ranking task, with useful context ablations. read the letter →

arxiv 2607.05582 v1 pith:KDR2UCNC submitted 2026-07-06 cs.IR

Alvin Wang , Jaromir Savelka This is my paper · ORCID

classification cs.IR

keywords informationretrievalstatutoryinterpretationcase-lawanalysisrelevantsentencesexpected-valuescoringzero-shotpromptingencoder-onlyfine-tuningNDCGranking

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Lawyers often need case-law sentences that explain vague statutory terms, but most retrieved sentences add little interpretive value. This paper shows that zero-shot prompting of large decoder-only models, when they emit class probabilities that are turned into an expected-value score over four explanatory-value labels, ranks those sentences more effectively than supervised fine-tuning of modern encoder models. On a fixed dataset of 26,959 sentences covering 42 U.S. Code concepts, the best prompted system surpasses all previously reported results at standard NDCG cutoffs. ModernBERT variants largely only match older BERT-family baselines, and adding paragraph-level context often hurts rather than helps, especially on large sparse queries. The practical claim is that instruction-tuned generative models can deliver stronger ranking of explanatory legal sentences without any task-specific fine-tuning, while expanded context is not a free performance boost.

What carries the argument

Generative expected-value scoring: the model outputs a probability distribution over the four annotation classes (high, certain, potential, no value), and the ranking score is the expected value of that distribution under temperature 0.0.

What would settle it

Re-score the same 42-term collection with an independent, non-probability ranking method or with explicitly calibrated probabilities; if the prompted EV systems no longer lead at NDCG@10 and NDCG@100, especially on large sparse queries, the central ranking claim fails.

Watch

Extended reading notes

Core claim

Across all 42 concepts and standard NDCG cutoffs, zero-shot decoder-only prompting with generative expected-value scoring achieves the strongest overall effectiveness on ranking case-law sentences by explanatory usefulness for statutory terms, with the best system surpassing all previously reported state-of-the-art results, while ModernBERT fine-tuning largely matches earlier BERT-family baselines and context expansion does not reliably help.

Load-bearing premise

The model-emitted class probabilities at temperature zero are well enough calibrated, and survive format repair, that their expected value is a fair, stable ranking signal of explanatory usefulness across models.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is a clean extension of the statutory-term sentence ranking task from Savelka & Ashley and the recent DeBERTa/Qwen SOTA. The new pieces are a ModernBERT re-implementation of the old encoder setups, a broader open-weight plus GPT-5.x sweep, expected-value scoring from four-class probability outputs, and a clear context-ablation story. The headline numbers land: GPT-5.4 EV gets overall NDCG@10/@100 of 0.82/0.87 and beats the ~0.79/0.85 prior mark; ModernBERT sp2snt sits near the old BERT baselines (~0.66/0.77).

What they do well is the experimental hygiene. Six-fold stratified CV, fixed recipe, NDCG@10/100, and the small/large × sparse/dense splits make the ordering readable. The context result is the most useful analytical bit: adding paragraph context or special query markers hurts, especially on large sparse queries, for both encoders and prompted models. That is a real, practical takeaway for this task—more text is not free signal. They also report open-weight numbers (GPT-OSS-120B, Qwen-3-32B, LLaMA variants) so the pattern is not only proprietary.

Soft spots are real but secondary. The SOTA peak sits on GPT-5.x APIs; temperature-0 EV assumes usable class probabilities, and they use aggressive JSON/regex repair without a calibration or repair-ablation check. That does not invent the ranking—open models and ModernBERT still show the same family ordering—but it means the exact gap to [13] could move if you re-ran DeBERTa/Qwen under the same EV protocol or if repair shifts mass. No code or prompts are released, which is the main reproducibility ding. Free parameters (LR, batch size, token limits, class indexing) are ordinary for this kind of paper.

Math and data look fine: established 26k-sentence / 42-concept set, standard NDCG, disclosed self-extension of their demo. Citation pattern is appropriate. This is for people building legal IR / statutory-interpretation tools who need a current bake-off and a warning about context expansion. I would send it to peer review; it is accept-shaped with a request for prompts, open-weight reproduction notes, and a short calibration/repair caveat. Worth engaging if you work in this niche.

Referee Report

3 major / 6 minor

Summary. The paper studies ranking of case-law sentences by explanatory usefulness for statutory terms on an established dataset (26,959 sentences, 42 U.S. Code concepts, four graded labels). It compares (i) supervised fine-tuning of ModernBERT under six input encodings (sentence-only through provision+context) with six-fold stratified CV, and (ii) zero-shot prompting of proprietary and open-weight decoder-only models that emit a four-class probability distribution whose expected value is used as the ranking score. Main empirical claims: ModernBERT largely matches earlier BERT-family baselines; expanded context often hurts, especially on large-sparse queries; and generative EV prompting, led by GPT-5.4 (overall NDCG@10/100 of 0.82/0.87), surpasses previously reported SOTA on this task.

Significance. If the results hold, the work is a clear, practically relevant contribution to legal IR and statutory interpretation support: it shows that carefully prompted decoder-only models can outperform task-specific fine-tuned encoders on graded explanatory-sentence ranking without labeled training, and it documents that simply adding more statutory or paragraph context is not a free win. Strengths include use of a public labeled benchmark, stratified multi-fold evaluation with small/large and sparse/dense query slices, explicit comparison of open-weight and proprietary models, and a transparent expected-value scoring recipe. The context-degradation analysis and the engineering trade-off discussion (training cost vs. API latency) are useful for practitioners. The work is an honest extension of the authors’ prior demo and of prior BERT/DeBERTa/Qwen baselines rather than a closed-loop self-evaluation.

major comments (3)

[Section 4.2; Table 2; Section 5.2] Section 4.2 and Table 2: The headline SOTA ordering rests on generative expected-value scores ∑ i·P(class=i) from temperature-0.0 model outputs. The manuscript does not report calibration diagnostics (e.g., reliability diagrams, ECE, or rank correlation of EV vs. hard argmax labels), nor does it quantify how the Regex/JSON repair pipeline in §5.2 shifts probability mass across the four classes. Without that, it is hard to know whether GPT-5.4’s lead over GPT-OSS-120B, Qwen-3-32B, and ModernBERT sp2snt is driven by better semantic ranking or by better-calibrated / repair-friendly probability formats. A short ablation (EV vs. discrete label; repaired vs. raw valid JSON only; optional temperature/logit-bias check) would make the central claim much more robust.
[Table 2; Section 6; Related Work] Table 2 and Related Work / Results: Prior SOTA numbers from [13] (DeBERTa-large overall .79/.79; Qwen2.5-72B .78/.85) are reported only as overall aggregates with no subset breakdowns and were obtained under different prompting/fine-tuning protocols. The paper’s claim that GPT-5.4 “surpasses all previously reported state-of-the-art results” is therefore only partially controlled. Either re-evaluate the [13] systems under the same EV protocol and fold structure, or clearly frame the comparison as cross-paper rather than same-protocol SOTA and report confidence intervals / paired tests on the 42-term means.
[Section 7; Table 2; Table 1] Section 7 (Context Expansion) and Table 2: The finding that sp2ctx and sp2snt+ctx degrade large-sparse performance (e.g., ModernBERT sp2snt LgSp NDCG@10 0.53 → 0.28 for sp2snt+ctx) is important but under-analyzed. It is unclear whether the drop is due to noise dilution, tokenizer truncation of the marked sentence, or the non-pretrained special markers used in sp+qry2snt. A brief error analysis or attention/saliency check on a few large-sparse terms would turn this from an observation into a transferable design lesson.

minor comments (6)

[Table 2] Table 2: Prior-work rows leave subset cells blank (“–”). Either fill them from the original papers or state explicitly that subset metrics were not available, so readers do not misread blanks as zeros.
[Section 4.2] Section 4.2: The class index mapping for EV (i = 0..3) should be stated once with the label order (No / Potential / Certain / High) so the formula is unambiguous without consulting the annotation guidelines.
[Section 5.1; Section 7] Section 5.1: Batch size is given as 16 in the training recipe and as 8 in the Colab timing note in §7; reconcile or explain the difference.
[Figure 1] Figure 1 is reproduced from [6]; ensure the caption and permission note remain clear in the camera-ready version and that axis labels remain legible at print size.
[References; Section 4.2] Several model cards and blog posts (GPT-5.2, GPT-5.4, Llama 4) are cited via URLs with future-dated years; pin model versions/API dates used for the runs to aid reproducibility as proprietary endpoints change.
[Title; Abstract; Section 8] Minor wording: abstract and conclusion say “prompting beats fine-tuning” as a general slogan; the body correctly scopes this to the present task and dataset—keep that scope in the title/abstract claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical IR ranking on an external labeled dataset with a fixed EV transform of model outputs.

full rationale

The paper’s central claim is an empirical comparison of fine-tuned ModernBERT variants versus zero-shot decoder-only prompting with generative expected-value scoring on the established 26,959-sentence statutory-term dataset of [6], evaluated by NDCG@10/100 against prior reported numbers (including [13]). The EV score is defined as a fixed, non-fitted transform ∑ i·P(class=i) of the model’s four-class distribution at temperature 0.0; it does not re-use evaluation labels as method parameters. Fine-tuning is ordinary supervised learning on the same external labels used for evaluation (standard stratified folds), not a prediction forced by construction. Self-extension of the authors’ demo [7] and co-author prior work [6] is disclosed and supplies the task/dataset, not a uniqueness theorem or ansatz that forces the SOTA ordering. No equation or claim reduces by definition to its inputs; the headline ranking is an experimental outcome on held-out folds and published baselines. Score 0 is therefore appropriate.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central claim is an empirical ranking comparison. It rests on standard IR metrics, the pre-existing four-class annotation scheme, fixed training/prompt hyperparameters, and the modeling choice that EV of class probabilities is the ranking score. No new physical entities; free parameters are ordinary ML/prompt settings that affect absolute scores but are shared across fair comparisons when held fixed.

free parameters (4)

ModernBERT learning rate
Set to 2e-5 with AdamW, weight decay 0.01, linear schedule, 10 epochs, batch 16; checkpoint chosen by val NDCG@100.
Prompt batch size and temperature
25 sentences per user message, temperature 0.0; affects probability estimates used for EV.
Class index mapping for EV
EV = sum i * P(class=i) for i in 0..3; the numeric coding of High/Certain/Potential/No value is a design choice that defines the ranking score.
Context token limits
512 / 840 / 1280 / 1440 token caps for different ModernBERT input formats; chosen by hand relative to provision and paragraph lengths.

assumptions (4)

domain assumption The four explanatory-value categories and student annotations (Krippendorff alpha 0.79) are a valid ground truth for ranking usefulness.
Entire evaluation (NDCG) is defined against these labels from [6]; Section 3.
domain assumption NDCG@10 and NDCG@100 are appropriate primary metrics for this ranking task.
Section 5.3; standard IR choice but still an assumption about what practitioners need.
ad hoc to paper Compacted annotation guidelines in the prompt are a faithful operationalization of the original labeling rules.
Section 4.2; zero-shot performance depends on this prompt compression.
standard math Stratified six-fold splits by concept yield unbiased estimates of generalization across statutory terms.
Section 5.1 experimental design inherited from [6].

invented entities (1)

Generative Expected Value Scoring (EV of four-class probability distribution)
purpose: Turn decoder-only class probabilities into a continuous ranking score without fine-tuning.
Named contribution in the title/method; it is a standard expectation transform, not a new physical object, and has no independent evidence requirement beyond ranking performance on this dataset.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Prompting Beats Fine-Tuning: Generative Expected Value Scoring for Statutory Term Retrieval." pith.science (2026). https://pith.science/paper/KDR2UCNC

@misc{pith2026260705582,
  author       = {Pith},
  title        = {Pith review of: Prompting Beats Fine-Tuning: Generative Expected Value Scoring for Statutory Term Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KDR2UCNC}},
  note         = {Machine review of arXiv:2607.05582}
}

read the original abstract

Legal concepts in statutes are often expressed using vague terms, and practitioners frequently turn to case law to interpret them. We study the task of ranking case-law sentences by their usefulness for explaining a concept or target statutory term, using an established dataset of 26,959 sentences covering 42 U.S. Code concepts labeled into four explanatory-value categories. We compare two families of methods: (i) supervised fine-tuning of encoder-only models (ModernBERT) and (ii) zero-shot prompting of decoder-only models. We show that across all concepts and standard NDCG cutoffs, ModernBERT largely matches earlier BERT-family baselines. In contrast, prompting decoder-only models achieves the strongest overall effectiveness, with our best system surpassing all previously reported state-of-the-art results on this task.

Figures

Figures reproduced from arXiv: 2607.05582 by the authors.

**Figure 1.** The graph on the left shows the distribution of the labels. The graph on the right presents the distribution of the number of sentences retrieved for each term. The graph is reproduced from [6] with the permission of the authors. 2. Related Work In prior work, researchers employed a variety of traditional information retrieval measures and their combinations, e.g., BM25, novelty, topic modeling [8, 9, 10]. These tur… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 8 linked inside Pith

[13]

Savelka, K

J. Savelka, K. D. Ashley, Learning to rank sentences for explaining statutory terms., in: ASAIL@ JURIX, 2020

2020
[1]

H. L. Hart, The Concept of Law, 2nd ed., Clarendon Press, 1994

1994
[2]

Endicott, Vagueness in Law, Oxford University Press, 2000

T. Endicott, Vagueness in Law, Oxford University Press, 2000

2000
[3]

Endicott, Law and Language the stanford encyclopedia of philosophy, http://plato.stanford.edu/,

T. Endicott, Law and Language the stanford encyclopedia of philosophy, http://plato.stanford.edu/,
[4]

Accessed: 2016-02-03

2016
[5]

Daci, Legal principles, legal values and legal norms: are they the same or different?, Academicus International Scientific Journal 02 (2010) 109–115

J. Daci, Legal principles, legal values and legal norms: are they the same or different?, Academicus International Scientific Journal 02 (2010) 109–115

2010
[6]

D. N. MacCormick, R. S. Summers, Interpreting Statutes, Darmouth, 1991

1991
[7]

Šavelka, K

J. Šavelka, K. D. Ashley, Discovering explanatory sentences in legal case decisions using pre-trained language models, in: Findings of the association for computational linguistics: EMNLP 2021, 2021, pp. 4273–4283

2021

Show all 30 references

[8]

A. Wang, J. Savelka, Explanatory case-law retrieval with encoder and decoder language models, in: Proceedings of the 21st International Conference on Artificial Intelligence and Law, 2026

2026
[9]

Savelka, H

J. Savelka, H. Xu, K. D. Ashley, Improving sentence retrieval from case law for statutory interpre- tation, in: Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law, 2019, pp. 113–122

2019
[10]

Savelka, Discovering sentences for argumentation about the meaning of statutory terms, Ph.D

J. Savelka, Discovering sentences for argumentation about the meaning of statutory terms, Ph.D. thesis, University of Pittsburgh, 2020

2020
[11]

Šavelka, K

J. Šavelka, K. D. Ashley, Legal information retrieval for understanding statutory terms, Artificial Intelligence and Law (2021) 1–45

2021
[12]

Šavelka, K

J. Šavelka, K. D. Ashley, Extracting case law sentences for argumentation about the meaning of statutory terms, in: Proceedings of the Third Workshop on Argument Mining (ArgMining2016), 2016, pp. 50–59

2016
[14]

Libal, A

T. Libal, A. Smywiński-Pohl, A. Kaczmarczyk, M. Król, Are manual annotations necessary for statutory interpretations retrieval?, in: Proceedings of the Twentieth International Conference on Artificial Intelligence and Law, 2025, pp. 414–419

2025
[15]

Krippendorff, Computing krippendorff’s alpha-reliability, Computing 1 (2011) 25

K. Krippendorff, Computing krippendorff’s alpha-reliability, Computing 1 (2011) 25

2011
[16]

Warner, A

B. Warner, A. Chaffin, B. Clavié, O. Weller, O. Hallström, S. Taghadouini, A. Gallagher, R. Biswas, F. Ladhak, T. Aarsen, et al., Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference, in: Proceedi...

2025
[17]

Devlin, M.-W

J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, V...

2019
[18]

Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized bert pretraining approach, arXiv preprint arXiv:1907.11692 (2019)

1907 arXiv
[19]

J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, Y. Liu, Roformer: Enhanced transformer with rotary position embedding, Neurocomputing 568 (2024) 127063

2024
[20]

Hurst, A

A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al., Gpt-4o system card, arXiv preprint arXiv:2410.21276 (2024)

2024 arXiv
[21]

URL: https://arxiv.org/abs/2508.10925

OpenAI, gpt-oss-120b & gpt-oss-20b model card, 2025. URL: https://arxiv.org/abs/2508.10925. arXiv:2508.10925

2025 arXiv
[22]

Addendum to the GPT-5 System Card

OpenAI, Introducing GPT-5.2, https://openai.com/index/introducing-gpt-5-2/, 2025. Addendum to the GPT-5 System Card

2025
[23]

OpenAI, GPT-5.4 Thinking system card, https://openai.com/index/gpt-5-4-thinking-system-card/, 2026

2026
[24]

Grattafiori, A

A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, et al., The Llama 3 herd of models, 2024. URL: https://arxiv.org/abs/2407.21783.arXiv:2407.21783

2024 arXiv
[25]

Meta AI, The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation, https://ai.meta.com/blog/llama-4-multimodal-intelligence/, 2025

2025
[26]

URL: https://arxiv.org/abs/2505.09388

Qwen Team, Qwen3 technical report, 2025. URL: https://arxiv.org/abs/2505.09388. arXiv:2505.09388

2025 arXiv
[27]

Loshchilov, F

I. Loshchilov, F. Hutter, Decoupled weight decay regularization, arXiv preprint arXiv:1711.05101 (2017)

2017 arXiv
[28]

J. Lin, R. Nogueira, A. Yates, Pretrained transformers for text ranking: Bert and beyond, arXiv preprint arXiv:2010.06467 (2020)

2010 arXiv
[29]

Y. Wang, L. Wang, Y. Li, D. He, T.-Y. Liu, A theoretical analysis of ndcg type ranking measures, in: Conference on learning theory, PMLR, 2013, pp. 25–54

2013
[30]

P. J. Sager, A. Kamaraj, B. F. Grewe, T. Stadelmann, Deep retrieval at checkthat! 2025: Identifying scientific papers from implicit social media mentions via hybrid retrieval and re-ranking, 2025. URL: https://arxiv.org/abs/2505.23250.arXiv:2505.23250

2025 arXiv

Pith tools

Reviewed July 11, 2026 · model on record in the stance chip above.

[1] [13]

Savelka, K

J. Savelka, K. D. Ashley, Learning to rank sentences for explaining statutory terms., in: ASAIL@ JURIX, 2020

2020

[2] [1]

H. L. Hart, The Concept of Law, 2nd ed., Clarendon Press, 1994

1994

[3] [2]

Endicott, Vagueness in Law, Oxford University Press, 2000

T. Endicott, Vagueness in Law, Oxford University Press, 2000

2000

[4] [3]

Endicott, Law and Language the stanford encyclopedia of philosophy, http://plato.stanford.edu/,

T. Endicott, Law and Language the stanford encyclopedia of philosophy, http://plato.stanford.edu/,

[5] [4]

Accessed: 2016-02-03

2016

[6] [5]

Daci, Legal principles, legal values and legal norms: are they the same or different?, Academicus International Scientific Journal 02 (2010) 109–115

J. Daci, Legal principles, legal values and legal norms: are they the same or different?, Academicus International Scientific Journal 02 (2010) 109–115

2010

[7] [6]

D. N. MacCormick, R. S. Summers, Interpreting Statutes, Darmouth, 1991

1991

[8] [7]

Šavelka, K

J. Šavelka, K. D. Ashley, Discovering explanatory sentences in legal case decisions using pre-trained language models, in: Findings of the association for computational linguistics: EMNLP 2021, 2021, pp. 4273–4283

2021