Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Time To Impeach LLM-as-a-Judge: Programs are the Future of Evaluation

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that LLM judges can be replaced by synthesized executable judging programs, aggregated with weak supervision, yielding cheaper, more consistent, less biased evaluation and a distilled reward model that beats…

desk verdict A clever program-based judge idea that is currently overclaimed because the headline comparison uses a stronger synthesizer than the judge baselines. read the letter →

arxiv 2506.10403 v1 pith:ULYM7QQU submitted 2025-06-12 cs.LG cs.AI

Tzu-Heng Huang , Harit Vishwakarma , Frederic Sala This is my paper
classification cs.LGcs.AI
keywords PAJAMALLM-as-a-judgeprogramsynthesisweaksupervisionrewardmodeldistillationbiasmitigationautomatedevaluationpreferenceaggregation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that evaluating LLM outputs does not have to mean paying a large model to score every answer. PAJAMA asks an LLM once to write small Python functions that encode judging criteria, then runs those functions locally and combines their noisy verdicts with weak supervision, a statistical technique for combining noisy labels. If the paper is right, evaluation becomes orders of magnitude cheaper, inspectable line by line, and less vulnerable to position, gender, formatting, and citation biases. The strongest evidence is on hard preference pairs: a reward model distilled from PAJAMA labels beats reward models distilled from LLM-as-a-judge on the CHAT-HARD subset of RewardBench, with +2.19% on Prometheus and +8.67% on JudgeLM, at roughly a thousandfold lower data-labeling cost. The authors also report that the approach loses ground on safety-related subsets, which they attribute to synthesized rules that struggle with emotional or sentimental content.

What carries the argument

The load-bearing object is the judging program: a compact Python function, synthesized by an LLM, that returns a quality score for a response and can be run locally for any new prompt. The paper writes the program's preference as $\bar{\lambda}_i = +1$ if program $i$ scores $y_1$ above $y_2$ and $-1$ otherwise, then models the joint distribution of the $m$ program outputs given the latent true preference $Y$ as $P(\bar{\lambda}_1,\dots,\bar{\lambda}_m \mid x,y_1,y_2) = \frac{1}{Z_\theta}\exp(-\theta_i \sum_i \bar{\lambda}_i Y)$, where $\theta_i$ is the learned reliability weight for program $i$. That identity turns noisy, complementary program outputs into a consensus preference label, and the same labels can be distilled into a reward model that generalizes beyond the programs.

What would settle it

Take preference pairs with known human ground truth, synthesize many programs under a prompt that biases all of them toward longer, better-formatted answers, and run PAJAMA's weak-supervision aggregation; if the learned reliability weights do not detect the shared bias and the preferred-response accuracy collapses toward the biased signal, the assumed independence of program errors given the true label is violated. A simpler test is to compare PAJAMA's aggregated labels with majority voting on a deliberately correlated pool of programs: the paper predicts weak supervision should beat voting by 5.2%, and if the gap inverts or disappears, the joint-distribution model is the weak link.

Watch

Extended reading notes

Core claim

PAJAMA's central claim is that a judge LLM's evaluation logic can be extracted as executable code before responses are seen, converting an expensive black-box scorer into a collection of cheap, inspectable programs. Each program scores a response on one of six criteria (structure, relevance, readability, bias, factuality, safety); the per-program preferences are discretized to $+1$ or $-1$ and modeled as noisy observations of the true preference under a weak supervision model that learns a reliability weight per program. The aggregated consensus labels are used directly or distilled into a reward model. On the CHAT-HARD split of RewardBench, distilled PAJAMA reward models outperform LLM-as-a-judge-distilled models by +2.19% on Prometheus and +8.67% on JudgeLM, and on four injected-bias tests PAJAMA improves consistency by 15.83% and lowers the biased-response win rate by 23.7% relative to Qwen2.5-14B. The paper also reports a caveat: on safety subsets PAJAMA underperforms the base model by 2.7%, suggesting the synthesized rules do not handle emotional or sentimental policies well.

Load-bearing premise

The load-bearing assumption is that, once the true preferred answer is known, the judging programs make their errors independently; if the programs share systematic blind spots, such as all rewarding length or formatting, the learned accuracy weights and the final consensus labels can be systematically wrong.

Editorial extensions

If this is right

  • Distilled PAJAMA reward models outperform LLM-as-a-judge-distilled models on RewardBench's CHAT-HARD split, by +2.19% on Prometheus and +8.67% on JudgeLM, while preference-label collection cost drops by roughly 2500 to 3500 times.
  • Program-based judges are more stable under injected biases: consistency rises 15.83% and biased-answer win rate falls 23.7% compared with Qwen2.5-14B.
  • Accuracy scales with the number of programs: PAJAMA reaches 82.2% with 52 programs on Prometheus, 5.2% above majority voting, with no observed plateau.
  • Because programs are stored and run locally, API cost scales with the number of programs rather than the dataset size, so re-running evaluations on new queries is nearly free.
  • Aggregated program labels can be distilled into a reward model that generalizes beyond the constituent programs, so the synthesized judges act as an intermediate representation rather than the final evaluator.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The absence of a plateau in the scaling curve suggests the binding constraint is not the number of programs but the language model's ability to synthesize diverse, accurate criteria; better code-generation capability should translate directly into better evaluation, a consequence the paper does not test.
  • Because each program's logic is fully exposed, evaluation pipelines become auditable and versionable like software; one could diff judging programs across benchmark versions or inspect what features are actually scored to detect contamination.
  • If judge programs were pooled and shared across tasks, the once-per-task synthesis cost would amortize even more strongly than the paper reports; this follows from the paper's cost model but is not proposed by the authors.
  • The reported safety-subset drop suggests criteria-based rules capture surface features but miss the nuance of refusal and harm policies; whether better program synthesis can close that gap remains an open question.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces PAJAMA (Program-As-a-Judge for Automated Model Assessment), a framework that uses an LLM (GPT-4o) to synthesize executable Python judging programs, aggregates their outputs via weak supervision (Snorkel), and optionally distills the resulting preference labels into a compact reward model (Gemma-2B-it). The authors claim that this approach drastically reduces evaluation cost (by three orders of magnitude), mitigates LLM-as-a-judge biases (improving consistency by 15.83% and reducing biased-response win rate by 23.7% relative to Qwen2.5-14B), and yields better distilled reward models on the CHAT-HARD subset of RewardBench for two of three datasets (Prometheus and JudgeLM).

Significance. If the central claims were fully supported, the program-as-a-judge idea would be a valuable contribution to LLM evaluation, offering auditable and reusable judging logic and a potentially cheaper alternative to direct LLM judging. The paper has genuine strengths: it evaluates on external, human-labeled benchmarks (RewardBench and held-out splits), includes a program-count scaling analysis (Figure 2), and its central comparison is not circular because the distilled model is measured against independent labels. However, the empirical support is substantially weakened by a confounded baseline comparison, incomplete cost accounting, and selective reporting that omits several large negative results.

major comments (5)
  1. [Section 3.1 (Setup and Results)] The headline comparison is confounded by the teacher model: PAJAMA's programs are synthesized with GPT-4o, whereas the LLM-as-a-judge training labels for Prometheus and JudgeLM come from GPT-4 and for PandaLM from GPT-3.5-Turbo. Any advantage of the stronger GPT-4o teacher is inherited by the PAJAMA-distilled reward model, so the reported +2.19% (Prometheus) and +8.67% (JudgeLM) CHAT-HARD gains cannot be attributed to the program-based representation alone. A control using the same underlying model (e.g., GPT-4o) for both direct labeling and program synthesis is necessary to support the central claim that programs are the cause of the improvement.
  2. [Section 2.3 (Judgment Ensemble)] The weak supervision model is mis-specified as written. The displayed probability is Pr(λ1,...,λm | x, y1, y2) = (1/Zθ) exp(− θi mX i=1 ¯λiY) where Y is the true preference label. This expression is not a valid joint distribution: θi appears outside the sum, the negative exponent would down-weight agreement, the symbol 'mX' is undefined, and the right-hand side does not condition on x, y1, y2 at all. The intended model is presumably a conditional-independence label model of the form P(λ|Y) ∝ exp(∑_i θ_i λ_i Y), but the assumption and the correct form need to be stated explicitly because the entire aggregation pipeline and all downstream results rely on this model.
  3. [Table 1 (Cost estimates)] The cost comparison is incomplete and inconsistent. The reported PAJAMA cost of $0.053 per dataset appears to account only for executing the synthesized programs, excluding the cost of generating the 52 programs with GPT-4o and the local compute required to run embedding models, BERT-based classifiers, and other external dependencies. At the same time, the 'LLM-as-a-Judge' cost column is labeled 'using GPT4o' even though the actual training labels for Prometheus and JudgeLM were produced with GPT-4 and for PandaLM with GPT-3.5-Turbo. Since 'orders of magnitude lower cost' is a core claim, the cost model needs to be fully specified, including one-time synthesis costs, amortization across datasets, and a consistent pricing basis for the baseline.
  4. [Abstract and Section 3.1 (Results)] The abstract reports only the two CHAT-HARD gains and omits that PAJAMA loses on PandaLM CHAT-HARD by 10.52 points (42.43 vs. 31.91) and is substantially worse than LLM-as-a-judge on the Chat, Safety, and Reasoning categories of RewardBench for all three datasets (e.g., Chat: 75.00 vs. 91.90 on Prometheus, 66.76 vs. 95.25 on JudgeLM, 83.24 vs. 94.41 on PandaLM). This selective reporting overstates the central claim; the abstract and conclusion should present the full pattern of results, including the negative outcomes, or the claims should be narrowed accordingly.
  5. [Table 2 and Section 3.2 (Bias Mitigation)] The bias-mitigation claim is not supported by the full table. PAJAMA achieves the highest biased-response win rate on position bias (55.63%, vs. 50.59% for Llama3-8B and 43.43% for Qwen2.5-14B), and its consistency is far below every baseline on gender (15.49%), rich-content (27.46%), and reference (2.82%) biases. The aggregate 15.83% consistency improvement is driven entirely by the position-bias result, and the 'biased response win rate' average is computed over three unspecified biases (the table header says '3 biases' without saying which), which excludes the high position-bias rate. The paper should report per-bias results transparently, state which biases are excluded and why, and provide variance or error bars across the three prompting trials.
minor comments (5)
  1. [Section 2.3] The symbol 'mX' in the exponential expression is undefined; this appears to be a typesetting error for a summation index.
  2. [Section 3.1 (Setup)] Reference [27] is cited for GPT-4o, but [27] is the GPT-4 Technical Report; the correct citation for GPT-4o is [19] (GPT-4o System Card).
  3. [Table 2] The table header is garbled and the column structure is ambiguous; a multi-row header with clear 'Consistency' and 'Biased Response Win Rate' subheadings for each of the four bias types is needed.
  4. [Figures 3-5] The radar plots are difficult to read at the printed size and the category-level numerical values are better conveyed in a table; consider replacing or supplementing these figures.
  5. [Abstract] The phrase 'improving judgment consistency by 15.83% and reducing biased responses by 23.7% on average' should specify that these numbers are relative to Qwen2.5-14B and are averages across a subset of bias types, not universal improvements.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: PAJAMA's headline CHAT-HARD gains are measured on external RewardBench labels and human-judged bias data, not on PAJAMA's own program outputs.

full rationale

PAJAMA's derivation chain is input-to-output: GPT-4o synthesizes executable judging programs; the programs are executed to emit pairwise preferences; Snorkel/weak supervision aggregates these noisy preferences into labels; a Gemma-2B-it reward model is distilled from those labels; and the distilled model is scored on RewardBench, an external, human-labeled benchmark, plus the Chen et al. bias dataset. None of the steps fits a parameter to the target metric and then reports that metric as a prediction. The weak-supervision joint distribution (Pr(λ̅_1,...,λ̅_m | x,y_1,y_2) = (1/Z_θ) exp(-θ_i Σ λ̅_i Y)) is an explicitly stated modeling assumption, not a theorem derived from the target result, and the accuracy weights θ_i are estimated on unlabeled program outputs, not from RewardBench. The paper's self-citations (e.g., [15,16,24] for weak supervision) support an established framework whose independent core is due to external prior work (Ratner, Ré, et al.); they are not invoked to forbid alternatives or to supply an unverified uniqueness result. A reviewer could object that PAJAMA's programs are synthesized with GPT-4o while the LLM-judge baselines use GPT-4 or GPT-3.5-Turbo labels, and that the abstract reports only the CHAT-HARD wins while losses on CHAT, Safety, and Reasoning are omitted. That is a fairness/selective-reporting concern, not circularity: the benchmark labels are external and the comparison is not an identity by construction. Under the stated circularity criteria, no load-bearing step reduces to its own input; the central empirical claim is self-contained against external data.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

PAJAMA introduces no new physical or conceptual entities; it is a system composition of known techniques. The main ledger entries are the hand-chosen program count and the assumptions about weak supervision and program execution cost.

free parameters (1)
  • Number of synthesized programs = 52
    Chosen by the authors; performance scales with this number (Figure 2), so it is a hand-selected hyperparameter rather than an optimized constant.
assumptions (3)
  • domain assumption Conditional independence of program outputs given the true preference label
    The weak supervision model in Section 2.3 assumes a factorized form over the noisy outputs; if programs are strongly correlated this assumption may fail and degrade label quality.
  • domain assumption Synthesized programs contain sufficient signal about response quality
    The method's success depends on GPT-4o's ability to write judging code that actually reflects quality; this is validated empirically but not guaranteed in general.
  • domain assumption Local execution of programs is effectively free
    The cost comparison in Table 1 counts only API fees for generating programs, not the compute time to run embedding and BERT models on 60K queries; this favors the cost comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Time To Impeach LLM-as-a-Judge: Programs are the Future of Evaluation." pith.science (2026). https://pith.science/paper/ULYM7QQU

@misc{pith2026250610403,
  author       = {Pith},
  title        = {Pith review of: Time To Impeach LLM-as-a-Judge: Programs are the Future of Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ULYM7QQU}},
  note         = {Machine review of arXiv:2506.10403}
}
read the original abstract

Large language models (LLMs) are widely used to evaluate the quality of LLM generations and responses, but this leads to significant challenges: high API costs, uncertain reliability, inflexible pipelines, and inherent biases. To address these, we introduce PAJAMA (Program-As-a-Judge for Automated Model Assessment), a new alternative that uses LLMs to synthesize executable judging programs instead of directly scoring responses. These synthesized programs can be stored and run locally, costing orders of magnitude less while providing interpretable, and auditable judging logic that can be easily adapted. Program-based judges mitigate biases, improving judgment consistency by 15.83% and reducing biased responses by 23.7% on average compared to a Qwen2.5-14B-based LLM-as-a-judge. When program judgments are distilled into a model, PAJAMA outperforms LLM-as-a-judge on the challenging CHAT-HARD subset of RewardBench, outperforming metrics by 2.19% on Prometheus and 8.67% on the JudgeLM dataset, all at three orders of magnitude lower cost.

Figures

Figures reproduced from arXiv: 2506.10403 by the authors.

Figure 1
Figure 1. PAJAMA’s General Workflow. we introduce six distinct criteria—each expressible as code—to guide LLMs in generating useful programs. Additionally, individual program outputs can be noisy, and different programs may capture complementary signals. We address this by combining our approach with the weak supervision framework [13, 14, 15, 16]. By modeling program outputs, we aggregate program judges into a collective sig… view at source ↗
Figure 2
Figure 2. PAJAMA’s performance can scale with the number of synthesized programs. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Prometheus Dataset. alpaca-easy alpaca-len mt-easy alpaca-hard mt-med 0.2 0.4 0.6 0.8 1.0 LLM-as-Judge: 0.953 Program-as-Judge: 0.668 Base Model: 0.715 CHAT mt-hard bar-natural bar-neighbor bar-GPTInst bar-GPTOut bar-manual 0.2 0.4 0.6 0.8 1.0 LLM-as-Judge: 0.487 Program-as-Judge: 0.573 Base Model: 0.383 CHAT-HARD ref-danger ref-offensv xs-respond xs-refuse donotans 0.2 0.4 0.6 0.8 1.0 LLM-as-Judge: 0.778 Program-as… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: JudgeLM Dataset. alpaca-easy alpaca-len mt-easy alpaca-hard mt-med 0.2 0.4 0.6 0.8 1.0 LLM-as-Judge: 0.944 Program-as-Judge: 0.832 Base Model: 0.715 CHAT mt-hard bar-natural bar-neighbor bar-GPTInst bar-GPTOut bar-manual 0.2 0.4 0.6 0.8 1.0 LLM-as-Judge: 0.424 Program-…
Figure 5
Figure 5. Figure 5: PandaLM Dataset. 9 [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TREK: A Travel Reasoning and Evaluation Kit for LLM Agents in Complex Trip Planning

    cs.CL 2026-07 conditional novelty 6.5 of 10

    On 800 jointly constrained trip tasks with a deterministic scorer and achievable gold, the best of 15 LLM agents fully solves only 46.2% of feasible plans, with unstated persona needs as the universal bottleneck.

Reference graph

Works this paper leans on

35 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    Systematic evaluation of llm-as-a-judge in llm alignment tasks: Explainable metrics and diverse prompt templates

    Wei, H.; He, S.; Xia, T.; Liu, F.; Wong, A.; Lin, J.; Han, M. Systematic evaluation of llm-as-a-judge in llm alignment tasks: Explainable metrics and diverse prompt templates. arXiv preprint arXiv:2408.13006 2024,

    arXiv 2024

  2. [2]

    N.; Li, T.; Li, D.; Zhu, B.; Zhang, H.; Jordan, M.; 5 Gonzalez, J

    Chiang, W.-L.; Zheng, L.; Sheng, Y .; Angelopoulos, A. N.; Li, T.; Li, D.; Zhu, B.; Zhang, H.; Jordan, M.; 5 Gonzalez, J. E.; others Chatbot arena: An open platform for evaluating llms by human preference. Forty-first International Conference on Machine Learning. 2024

    work page 2024

  3. [3]

    Advances in Neural Information Processing Systems 2023, 36, 46595–46623

    Zheng, L.; Chiang, W.-L.; Sheng, Y .; Zhuang, S.; Wu, Z.; Zhuang, Y .; Lin, Z.; Li, Z.; Li, D.; Xing, E.; others Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems 2023, 36, 46595–46623

    work page 2023

  4. [4]

    Zhuge, M.; Zhao, C.; Ashley, D.; Wang, W.; Khizbullin, D.; Xiong, Y .; Liu, Z.; Chang, E.; Krishnamoorthi, R.; Tian, Y .; others Agent-as-a-judge: Evaluate agents with agents.arXiv preprint arXiv:2410.10934 2024,

    arXiv 2024

  5. [5]

    Qurating: Selecting high-quality data for training language models

    Wettig, A.; Gupta, A.; Malik, S.; Chen, D. Qurating: Selecting high-quality data for training language models. arXiv preprint arXiv:2402.09739 2024,

    arXiv 2024

  6. [6]

    F.; Leike, J.; Brown, T.; Martic, M.; Legg, S.; Amodei, D

    Christiano, P. F.; Leike, J.; Brown, T.; Martic, M.; Legg, S.; Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems 2017, 30

    work page 2017

  7. [7]

    Anthropic The Claude 3 Model Family: Opus, Sonnet, Haiku. 2024

    work page 2024

  8. [8]

    Judgelm: Fine-tuned large language models are scalable judges.arXiv preprint arXiv:2310.17631 2023,

    Zhu, L.; Wang, X.; Wang, X. Judgelm: Fine-tuned large language models are scalable judges.arXiv preprint arXiv:2310.17631 2023,

    arXiv 2023

Show all 35 references
  1. [9]

    xfinder: Robust and pinpoint answer extraction for large language models

    Yu, Q.; Zheng, Z.; Song, S.; Li, Z.; Xiong, F.; Tang, B.; Chen, D. xfinder: Robust and pinpoint answer extraction for large language models. arXiv preprint arXiv:2405.11874 2024,

    2024 arXiv

  2. [10]

    Discovering bias in latent space: An unsupervised debiasing approach

    Adila, D.; Zhang, S.; Han, B.; Wang, Y . Discovering bias in latent space: An unsupervised debiasing approach. arXiv preprint arXiv:2406.03631 2024,

    2024 arXiv

  3. [11]

    H.; Chen, S.; Liu, Z.; Jiang, F.; Wang, B

    Chen, G. H.; Chen, S.; Liu, Z.; Jiang, F.; Wang, B. Humans or LLMs as the Judge? A Study on Judgement Bias. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Miami, Florida, USA, 2024; pp 8301–8327

    2024

  4. [12]

    arXiv preprint arXiv:2410.02736 2024,

    Ye, J.; Wang, Y .; Huang, Y .; Chen, D.; Zhang, Q.; Moniz, N.; Gao, T.; Geyer, W.; Huang, C.; Chen, P.-Y .; others Justice or prejudice? quantifying biases in llm-as-a-judge. arXiv preprint arXiv:2410.02736 2024,

    2024 arXiv

  5. [13]

    J.; De Sa, C

    Ratner, A. J.; De Sa, C. M.; Wu, S.; Selsam, D.; Ré, C. Data programming: Creating large training sets, quickly. Advances in neural information processing systems 2016, 29

    2016

  6. [14]

    H.; Ehrenberg, H.; Fries, J.; Wu, S.; Ré, C

    Ratner, A.; Bach, S. H.; Ehrenberg, H.; Fries, J.; Wu, S.; Ré, C. Snorkel: Rapid training data creation with weak supervision. Proceedings of the VLDB Endowment. International Conference on Very Large Data Bases. 2017; p 269

    2017

  7. [15]

    Training complex models with multi-task weak supervision

    Ratner, A.; Hancock, B.; Dunnmon, J.; Sala, F.; Pandey, S.; Ré, C. Training complex models with multi-task weak supervision. Proceedings of the AAAI Conference on Artificial Intelligence. 2019; pp 4763–4771

    2019

  8. [16]

    Fast and three-rious: Speeding up weak supervision with triplet methods

    Fu, D.; Chen, M.; Sala, F.; Hooper, S.; Fatahalian, K.; Ré, C. Fast and three-rious: Speeding up weak supervision with triplet methods. International Conference on Machine Learning. 2020; pp 3280–3291

    2020

  9. [17]

    Y .; Chandu, K.; Dziri, N.; Kumar, S.; Zick, T.; Choi, Y .; others Rewardbench: Evaluating reward models for language modeling.arXiv preprint arXiv:2403.13787 2024,

    Lambert, N.; Pyatkin, V .; Morrison, J.; Miranda, L.; Lin, B. Y .; Chandu, K.; Dziri, N.; Kumar, S.; Zick, T.; Choi, Y .; others Rewardbench: Evaluating reward models for language modeling.arXiv preprint arXiv:2403.13787 2024,

    2024 arXiv

  10. [18]

    The Twelfth International Conference on Learning Representations

    Kim, S.; Shin, J.; Cho, Y .; Jang, J.; Longpre, S.; Lee, H.; Yun, S.; Shin, S.; Kim, S.; Thorne, J.; others Prometheus: Inducing fine-grained evaluation capability in language models. The Twelfth International Conference on Learning Representations. 2023

    2023

  11. [19]

    P.; Perelman, A.; Ramesh, A.; Clark, A.; Ostrow, A.; Welihinda, A.; Hayes, A.; Radford, A.; others Gpt-4o system card

    Hurst, A.; Lerer, A.; Goucher, A. P.; Perelman, A.; Ramesh, A.; Clark, A.; Ostrow, A.; Welihinda, A.; Hayes, A.; Radford, A.; others Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024,

    2024 arXiv

  12. [20]

    Multi-Granularity, M.-L. M.-F. M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. 2024,

    2024

  13. [21]

    P.; Fishburne Jr, R

    Kincaid, J. P.; Fishburne Jr, R. P.; Rogers, R. L.; Chissom, B. S. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. 1975,

    1975

  14. [22]

    FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledge

    Feng, S.; Balachandran, V .; Bai, Y .; Tsvetkov, Y . FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledge. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore, 2023; pp 933–952. 6

    2023

  15. [23]

    Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection

    Vidgen, B.; Thrush, T.; Waseem, Z.; Kiela, D. Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection. ACL. 2021

    2021

  16. [24]

    Universalizing weak supervision

    Shin, C.; Li, W.; Vishwakarma, H.; Roberts, N.; Sala, F. Universalizing weak supervision. 2022

    2022

  17. [25]

    Weak-to-Strong Generalization Through the Data-Centric Lens

    Shin, C.; Cooper, J.; Sala, F. Weak-to-Strong Generalization Through the Data-Centric Lens. 2025

    2025

  18. [26]

    PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization

    Wang, Y .; Yu, Z.; Zeng, Z.; Yang, L.; Wang, C.; Chen, H.; Jiang, C.; Xie, R.; Wang, J.; Xie, X.; Ye, W.; Zhang, S.; Zhang, Y . PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization. International Conference on Learning Representations (ICLR) 2024,

    2024

  19. [27]

    L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; others Gpt-4 technical report

    Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; others Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023,

    2023 arXiv

  20. [28]

    G.; Hardin, C.; Bhupatiraju, S.; Hussenot, L.; Mesnard, T.; Shahriari, B.; Ramé, A.; others Gemma 2: Improving open language models at a practical size

    Team, G.; Riviere, M.; Pathak, S.; Sessa, P. G.; Hardin, C.; Bhupatiraju, S.; Hussenot, L.; Mesnard, T.; Shahriari, B.; Ramé, A.; others Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118 2024,

    2024 arXiv

  21. [29]

    Llms-as-judges: a comprehensive survey on llm-based evaluation methods

    Li, H.; Dong, Q.; Chen, J.; Su, H.; Zhou, Y .; Ai, Q.; Ye, Z.; Liu, Y . Llms-as-judges: a comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579 2024,

    2024 arXiv

  22. [30]

    Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form Text

    Badshah, S.; Sajjad, H. Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form Text. arXiv preprint arXiv:2408.09235 2024,

    2024

  23. [31]

    The ALCHEmist: Automated Labeling 500x CHEaper than LLM Data Annotators

    Huang, T.-H.; Cao, C.; Bhargava, V .; Sala, F. The ALCHEmist: Automated Labeling 500x CHEaper than LLM Data Annotators. Neural Information Processing Systems (NeurIPS). 2024

    2024

  24. [32]

    Scriptoriumws: A code generation assistant for weak supervision

    Huang, T.-H.; Cao, C.; Schoenberg, S.; Vishwakarma, H.; Roberts, N.; Sala, F. Scriptoriumws: A code generation assistant for weak supervision. arXiv preprint arXiv:2502.12366 2025,

    2025 arXiv

  25. [33]

    Autows-bench-101: Benchmarking automated weak supervision with 100 labels

    Roberts, N.; Li, X.; Huang, T.-H.; Adila, D.; Schoenberg, S.; Liu, C.-Y .; Pick, L.; Ma, H.; Albarghouthi, A.; Sala, F. Autows-bench-101: Benchmarking automated weak supervision with 100 labels. Advances in Neural Information Processing Systems 2022, 35, 8912–8925

    2022

  26. [34]

    Evaluating Sample Utility for Data Selection by Mimicking Model Weights

    Huang, T.-H.; Bilkhu, M.; Sala, F.; Movellan, J. Evaluating Sample Utility for Data Selection by Mimicking Model Weights. arXiv preprint arXiv:2501.06708 2025,

    2025 arXiv

  27. [35]

    ""Calculate readability metrics for response

    Vishwakarma, H.; Sala, F. Lifting weak supervision to structured prediction. Advances in Neural Information Processing Systems 2022, 35, 37563–37574. 7 A Related Work Our work studies modeling synthesized judging programs to evaluate LLM generations, intersecting two key areas...

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.