pith. sign in

arxiv: 2606.11196 · v1 · pith:2W5SNTKHnew · submitted 2026-04-20 · 💻 cs.CL · cs.AI· cs.CR· cs.LG

PoQ-Judge: A Multi-Architecture Evaluation Framework for Cost-Aware Proof-of-Quality in Decentralized LLM Inference

Arther Tian , Alex Ding , Frank Chen , Simon Wu , Aaron Chan This is my paper

Pith reviewed 2026-07-05 16:12 UTC · model glm-5.2

classification 💻 cs.CL cs.AIcs.CRcs.LG
keywords reference-free evaluationdecentralized inferenceproof of qualityLLM-as-judgequality-cost tradeoffcomposite scoringonline calibrationcascade evaluation
0
0 comments X

The pith

Trained 184M judge matches reference-based evaluators without ground truth

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper addresses a deployment gap in decentralized LLM inference networks: how to score output quality when you have no reference answer to compare against. The authors train three compact encoder models (10M–184M parameters) via a two-stage pipeline—pre-training on UltraFeedback, then fine-tuning on GPT-labeled domain data—to produce quality scores from query–output pairs alone. The strongest model, a DeBERTa-based judge, achieves Pearson correlation 0.747 with a ground-truth quality proxy on a held-out test set, exceeding all reference-based evaluators from the authors' prior framework. When integrated into a composite scoring system alongside structural priors, the reference-free mode achieves 0.645 Pearson—statistically indistinguishable from the best single reference-based evaluator (0.647)—while requiring no reference answers at all. The paper also shows that gradient-based online calibration automatically identifies semantic quality as the dominant scoring dimension (assigning it 4.7× the initial weight) and suppresses negatively-correlated dimensions, recovering insights that previously required manual ablation. A cascade evaluation protocol routes confident samples through cheap structural checks, achieving 72.7% cost savings. The results are dramatically task-dependent: QA correlation reaches 0.830 but summarization drops to 0.199, which the authors attribute to the token-level F1 ground-truth proxy being a poor measure of summarization quality.

Core claim

The central mechanism is a two-stage training pipeline that transfers broad evaluation knowledge from a large AI-feedback corpus (UltraFeedback, 45k samples) to a narrow domain-specific distribution (1,400 GPT-labeled samples), producing compact regression models that score query–output pairs on a continuous quality scale without reference answers. The DeBERTa judge (184M parameters) trained this way achieves Pearson 0.747 with the ground-truth proxy, and its reference-free composite mode (0.645) matches the best reference-based single evaluator (0.647). The paper demonstrates that the evaluation task in Proof-of-Quality systems—scoring a (query, output) pair—can be cast as regression ands.

What carries the argument

Two-stage training pipeline (UltraFeedback pre-training + GPT-labeled domain fine-tuning); three judge architectures spanning a quality–cost Pareto frontier (TextCNN ~10M, MiniLM 22M, DeBERTa 184M); composite scoring with reference-free, full, and auto modes; gradient-based online dimension calibration; cascade evaluation protocol with confidence-based early stopping

Load-bearing premise

The ground-truth quality proxy—token-level F1 between model output and reference answer—is assumed to be a valid quality measure for training and evaluating judges. This proxy yields Pearson 0.830 on QA but only 0.199 on summarization, and the paper itself acknowledges it poorly captures summarization quality. Since judges are trained on GPT labels that partially inherit this proxy's limitations, the headline 0.747 correlation is driven almost entirely by QA performance.

What would settle it

If the DeBERTa judge were evaluated on a broader task distribution where token-level F1 is a poor quality proxy (e.g., creative writing, code generation, instruction-following), its correlation with true human quality judgments would drop substantially, revealing that the learned evaluation ability is narrow rather than general.

Figures

Figures reproduced from arXiv: 2606.11196 by Aaron Chan, Alex Ding, Arther Tian, Frank Chen, Simon Wu.

Figure 1
Figure 1. Figure 1: Overview of the PoQ-Judge framework. Three judge architectures are trained via a [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Two-stage training pipeline for PoQ-Judge models. Stage 1 pre-trains on the large-scale [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Left: GT Pearson correlation for all judge models and the best reference-based baseline [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Judge performance by task type. QA Pearson correlations are strong across all architectures; [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Left: Composite scoring modes compared by GT Pearson. Reference-free mode with the [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Left: Average inference reward across calibration scenarios. Right: Final dimension [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Left: GT Pearson vs. cost savings across budget levels, showing the low-budget regime [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Fine-tuning dynamics across judge architectures. Left: validation loss. Right: validation [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗
read the original abstract

Decentralized LLM inference networks need lightweight, reference-free quality evaluation for Proof of Quality (PoQ). We present PoQ-Judge, a framework that trains dedicated judge models to score query-output pairs without ground-truth references. We study three architectures across the quality-cost tradeoff: a TextCNN judge, a MiniLM cross-encoder, and a DeBERTa judge. Using two-stage training on UltraFeedback plus GPT-labeled in-domain data, the best model reaches 0.747 Pearson correlation with the ground-truth proxy on a held-out test set, outperforming reference-based evaluators from prior work. As a reference-free component in composite scoring, it achieves 0.645 Pearson correlation, matching the best single reference-based evaluator while removing the need for reference answers. We also show that online calibration identifies semantic quality as the dominant dimension and that cascade evaluation reduces cost by 72.7 percent with only modest quality loss. Results are much stronger on QA than summarization, pointing to proxy quality as the main remaining limitation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 7 minor

Summary. The paper introduces PoQ-Judge, a framework for training lightweight, reference-free judge models to score query-output pairs in decentralized LLM inference networks under Proof of Quality (PoQ). Three architectures are studied: TextCNN (~10M), MiniLM (22M), and DeBERTa (184M), trained via a two-stage pipeline (UltraFeedback pre-training, GPT-labeled domain fine-tuning). The DeBERTa judge achieves Pearson 0.747 with a token-level F1 ground-truth proxy on a held-out test set (n=300), reportedly exceeding all reference-based evaluators from the authors' prior framework. A reference-free composite scoring mode achieves 0.645 Pearson, matching the best single reference-based evaluator. The paper also studies online dimension calibration (gradient-based weight learning recovers interpretable dimension rankings) and a cascade evaluation protocol (72.7% cost savings in low-budget regime). Results show strong task dependence: QA Pearson 0.830 vs. summarization 0.199, attributed to the F1 proxy's limitations for summarization.

Significance. The paper addresses a genuine deployment gap in decentralized LLM inference: the strongest quality dimension (reference-based semantic similarity) is unavailable at inference time. Training compact encoder models as reference-free judges is a reasonable and practical approach. The framework provides falsifiable predictions (quality-cost Pareto frontier, per-task breakdowns), honest reporting of failures (summarization Pearson 0.199, calibration not improving reward, cascade Layer 2 bypassed), and bootstrap confidence intervals. The two-stage training pipeline and the integration into composite scoring are clearly described. However, the central claim of superiority over reference-based evaluators rests on an aggregate comparison that is not fully supported by the reported data, as detailed below. The paper is the fourth in a self-cited series, with the testbed, ground-truth proxy, and baselines all inherited from prior work.

major comments (3)
  1. §5.1, Table 5 vs. Table 6: The headline claim that the DeBERTa judge (0.747 aggregate Pearson) 'exceeds all reference-based evaluators' (best: STS-stsb at 0.647) is reported only in aggregate over n=300. Table 6 provides per-task breakdowns for the judges (QA=0.830, summarization=0.199 for DeBERTa), but no corresponding per-task breakdown is provided for the reference-based baselines. Since the aggregate is dominated by QA performance, and STS-stsb computes embedding similarity between output and reference—which should correlate well with token-level F1 on extractive QA—it is plausible that STS-stsb also achieves high QA Pearson. If so, the 0.747 vs. 0.647 aggregate difference may reflect how each method's QA and summarization scores are averaged rather than genuine superiority of the trained judge. The claim 'exceeds all reference-based evaluators' requires per-task baseline numbers to,
  2. §5.5, Table 10: The cascade evaluation protocol—presented as a contribution of this paper—does not function as designed. Layer 2 (lightweight judge) is bypassed in all configurations, with evaluations jumping directly from Layer 1 to Layer 3. The paper attributes this to the confidence estimator not accumulating sufficient confidence from Layer 2 dimensions, and notes the judge dimension 'returns a default score when not properly initialized with a loaded model.' This means the cascade's 72.7% cost savings come entirely from Layer 1 structural checks (Pearson 0.509), not from the trained judge models that are the paper's central contribution. The cascade contribution should be either fixed (so that Layer 2 actually uses a trained judge) or substantially reframed as a structural-prior-only filtering mechanism rather than a judge-based cascade.
  3. §4.1, §5.2: The ground-truth proxy (token-level F1) yields Pearson 0.830 on QA but only 0.199 on summarization, and the paper acknowledges it 'poorly captures summarization quality' (§6). Since judges are trained on GPT labels that 'partially inherit' this proxy's limitations, the 0.747 headline number is driven almost entirely by QA performance. This is a load-bearing issue for the central claim because the paper frames the result as general quality assessment ability. The abstract and conclusion should explicitly qualify the headline claim as primarily reflecting QA performance, and the paper should discuss whether the claim of 'exceeding reference-based evaluators' holds when restricted to tasks where the proxy is valid. This is related to but distinct from the per-task breakdown requested above.
minor comments (7)
  1. §1, paragraph 7: The abstract states the DeBERTa judge exceeds 'all reference-based evaluators' (best: STS-stsb at 0.647), but the introduction (§1, paragraph 7) cites sts_paraphrase at 0.629 as the best baseline. Table 5 shows STS-stsb at 0.647 as the strongest. Please reconcile these references so the reader knows which baseline is the strongest.
  2. Table 9: The 'LLM judge' dimension receives a weight of 0.601, but the paper's trained judge is called 'PoQ-Judge' or 'DeBERTa judge' elsewhere. Please use consistent naming. Also, clarify whether the 'LLM judge' dimension in the calibration experiment uses the DeBERTa judge or a different model.
  3. §5.4, Table 8: None of the calibration strategies improve average inference reward over the uncalibrated baseline. The paper provides a reasonable explanation (calibration-GT alignment vs. consensus-reward alignment divergence), but this result somewhat undercuts the calibration contribution. Consider stating more prominently in the abstract that calibration is presented as a diagnostic tool rather than a reward-improving mechanism.
  4. Figure 1: The diagram is dense and hard to parse at normal zoom. Consider splitting into separate sub-figures for training pipeline, composite scoring, and deployment mechanisms.
  5. §3.2: The domain fine-tuning data (1,400 train, 300 val, 300 test) is quite small. The paper notes this is sufficient for adaptation but does not discuss data augmentation or the risk of overfitting to the specific inference model pool (Table 3). A brief discussion of generalization would strengthen this section.
  6. §7: The related work section is thorough but does not discuss recent work on learning-to-evaluate approaches (e.g., G-Eval, BLEURT-style learned metrics) in the context of the PoQ-Judge approach. A sentence or two comparing the reference-free regression approach to these learned metric approaches would strengthen positioning.
  7. Table 3: The parameter count for Gemma-2-2B is listed as 2.6B, which does not match the model name. Please verify.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee raises three major points, all of which are substantive. We agree that (1) per-task breakdowns for reference-based baselines are needed to support the headline comparison claim, (2) the cascade contribution requires reframing given that Layer 2 is bypassed, and (3) the abstract and conclusion must explicitly qualify the headline claim as primarily reflecting QA performance. We will implement all three changes. On the first point, we can and will provide the per-task baseline numbers; on the second, we will reframe rather than claim a fix we have not yet implemented; on the third, we agree the qualification is necessary and will revise accordingly.

read point-by-point responses

  1. Referee: §5.1, Table 5 vs. Table 6: The headline claim that the DeBERTa judge (0.747 aggregate Pearson) 'exceeds all reference-based evaluators' (best: STS-stsb at 0.647) is reported only in aggregate over n=300. Table 6 provides per-task breakdowns for the judges (QA=0.830, summarization=0.199 for DeBERTa), but no corresponding per-task breakdown is provided for the reference-based baselines. Since the aggregate is dominated by QA performance, and STS-stsb computes embedding similarity between output and reference—which should correlate well with token-level F1 on extractive QA—it is plausible that STS-stsb also achieves high QA Pearson. If so, the 0.747 vs. 0.647 aggregate difference may reflect how each method's QA and summarization scores are averaged rather than genuine superiority of the trained judge. The claim 'exceeds all reference-based evaluators' requires per-task baseline numbers.

    Authors: The referee is correct that the aggregate comparison is insufficient without per-task breakdowns for the reference-based baselines. We will add a per-task decomposition for all five reference-based evaluators (STS-paraphrase, STS-stsb, STS-MiniLM, CE-MiniLM, CE-DeBERTa) in a revised Table 6, computed on the same n=150 per-task splits used for the judges. We can confirm that the per-task numbers are available from our existing evaluation pipeline and will be included in the revision. We also agree that the claim 'exceeds all reference-based evaluators' should be qualified in light of the per-task analysis. Specifically, if STS-stsb achieves comparable QA Pearson to the DeBERTa judge, the aggregate difference may indeed reflect the averaging structure rather than genuine reference-free superiority. We will revise the headline claim to state the comparison precisely: the DeBERTa judge achieves the highest aggregate Pearson among all evaluators, with the per-task analysis revealing that this advantage is concentrated on QA. We will not claim per-task superiority over reference-based baselines unless the per-task numbers support it. The abstract, introduction, and conclusion will be updated accordingly. revision: yes

  2. Referee: §5.5, Table 10: The cascade evaluation protocol—presented as a contribution of this paper—does not function as designed. Layer 2 (lightweight judge) is bypassed in all configurations, with evaluations jumping directly from Layer 1 to Layer 3. The paper attributes this to the confidence estimator not accumulating sufficient confidence from Layer 2 dimensions, and notes the judge dimension 'returns a default score when not properly initialized with a loaded model.' This means the cascade's 72.7% cost savings come entirely from Layer 1 structural checks (Pearson 0.509), not from the trained judge models that are the paper's central contribution. The cascade contribution should be either fixed (so that Layer 2 actually uses a trained judge) or substantially reframed as a structural-prior-only filtering mechanism rather than a judge-based cascade.

    Authors: The referee's characterization is accurate. Layer 2 is bypassed in all configurations, and the 72.7% cost savings come entirely from Layer 1 structural checks. We acknowledge that this means the cascade results do not demonstrate the intended judge-based cascade behavior and that presenting the cascade as a functioning three-layer protocol overstates what the experiments show. We will not claim to have fixed the Layer 2 initialization issue, as we have not yet done so. Instead, we will substantially reframe the cascade contribution in the revision. Specifically: (1) We will retitle and restructure Section 5.5 to present the results as a structural-prior filtering mechanism rather than a functioning three-layer judge cascade. (2) We will remove language implying that Layer 2 operates as designed and will explicitly state that the current implementation does not utilize trained judges in the cascade path. (3) We will downgrade the cascade from a listed contribution to a preliminary analysis of structural-prior filtering, with the full judge-based cascade described as future work. (4) The abstract and conclusion will be revised to describe the cascade result as 'structural-prior filtering achieving 72.7% cost savings' rather than 'cascade evaluation.' We believe the structural filtering result itself is still worth reporting—it demonstrates a practical low-cost operating point—but it should not be presented as evidence that the trained judges function in a cascade. revision: yes

  3. Referee: §4.1, §5.2: The ground-truth proxy (token-level F1) yields Pearson 0.830 on QA but only 0.199 on summarization, and the paper acknowledges it 'poorly captures summarization quality' (§6). Since judges are trained on GPT labels that 'partially inherit' this proxy's limitations, the 0.747 headline number is driven almost entirely by QA performance. This is a load-bearing issue for the central claim because the paper frames the result as general quality assessment ability. The abstract and conclusion should explicitly qualify the headline claim as primarily reflecting QA performance, and the paper should discuss whether the claim of 'exceeding reference-based evaluators' holds when restricted to tasks where the proxy is valid.

    Authors: We agree. The 0.747 aggregate Pearson is driven primarily by QA performance (0.830), and the summarization result (0.199) reflects ground-truth proxy limitations rather than judge capability. The abstract and conclusion should state this explicitly. We will revise the abstract to read something like: 'The DeBERTa judge achieves Pearson 0.747 on the held-out test set, though this aggregate is driven primarily by QA performance (0.830); summarization performance (0.199) is limited by the token-level F1 ground-truth proxy.' The conclusion will be similarly qualified. Regarding whether the 'exceeding reference-based evaluators' claim holds when restricted to tasks where the proxy is valid: this is exactly the question that the per-task baseline breakdown (requested in the first major comment) is designed to answer. If STS-stsb achieves comparable QA Pearson, then the claim of superiority does not hold even on QA, and we will say so. If the DeBERTa judge's QA Pearson (0.830) exceeds STS-stsb's QA Pearson, we will state the comparison as QA-specific. In either case, we will not frame the result as general quality assessment ability across tasks. We will also add a discussion of whether the comparison is meaningful on summarization given that the proxy itself is invalid for that task—on summarization, neither the judges nor the reference-based evaluators can be meaningfully compared because the ground-truth signal is too weak to serve as an evaluation target. revision: yes

Circularity Check

0 steps flagged

No significant circularity. Self-citations are contextual, not load-bearing. The central claim (DeBERTa judge achieves 0.747 Pearson) is independently measured on a held-out test set against an externally defined proxy.

full rationale

The paper trains three judge models via a two-stage pipeline (UltraFeedback pre-training + GPT-labeled fine-tuning) and evaluates them on a held-out test set (n=300) against a ground-truth proxy (token-level F1). The headline result — DeBERTa judge achieves 0.747 Pearson — is measured on this disjoint test set and is not forced by construction. The baselines it exceeds (STS-stsb at 0.647, etc.) are inherited from prior work [41] by the same authors, but these are standard pre-trained sentence-embedding models (Sentence-BERT, SimCSE) applied as evaluators; they are not defined by the authors' prior framework and could be independently reproduced. The composite scoring framework (Eq. 3, 7) and the PoQ simulation testbed are also inherited from [41], but the paper's contribution — training reference-free judges — is a genuinely new component evaluated against these inherited baselines. The ground-truth proxy (token-level F1) is a standard NLP metric, not defined in terms of the paper's outputs. The gradient calibration result (Table 9: semantic quality gets 4.7x weight) is an empirical finding from simulation, not a definitional consequence. While the paper is the fourth in a self-cited series [39, 40, 41] and reuses the testbed, baselines, and proxy from prior work, these self-citations provide context and comparability rather than serving as load-bearing logical premises. The central experimental claims are falsifiable on the held-out test set without reference to the prior papers' conclusions. The absence of per-task baseline breakdowns (noted by the skeptic) is a correctness/completeness concern, not a circularity issue — the aggregate comparison is still an independent measurement, just potentially uninterpretable. No step in the derivation chain reduces to its inputs by definition or construction.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 0 invented entities

The paper introduces no new theoretical entities, particles, forces, or mathematical objects. The 'Proof of Quality' concept and the composite scoring framework are inherited from prior work [44, 41]. The judge architectures (TextCNN, MiniLM, DeBERTa) are existing models applied to a new task. The cascade protocol and online calibration are deployment mechanisms built from standard techniques (gradient descent, UCB bandit, EMA).

free parameters (6)
  • TextCNN learning rates = 1e-3 (pre-train), 5e-4 (fine-tune)
    Architecture-specific learning rates chosen by the authors for the TextCNN judge (§3.2, Table 11).
  • Encoder learning rates = 2e-5 (pre-train), 5e-6 (fine-tune)
    Standard fine-tuning rates chosen for MiniLM and DeBERTa to avoid catastrophic forgetting (§3.2, Table 11).
  • Early stopping patience = Not specified numerically
    Patience-based early stopping on validation Pearson; patience value not stated (§3.2).
  • Cascade confidence thresholds τ_l = Not specified
    Layer-specific confidence thresholds for cascade early stopping are described but not numerically specified (§3.5).
  • Calibration anchor rate α = 0.05 (5%)
    Chosen as the fraction of rounds with reference-based anchor signals for calibration (§3.4, Table 8).
  • Weight bounds [w_min, w_max] = [0, 5.0] (implied by Table 9)
    Weight clipping bounds for gradient calibration; EMA results show weights saturating at 5.0 (§5.4).
axioms (4)
  • domain assumption Token-level F1 between model output and reference answer is a valid ground-truth quality proxy for training and evaluating judges.
    Used throughout as the primary evaluation metric (§4.1). The paper itself acknowledges this is weak for summarization (§5.2, §6).
  • domain assumption GPT-4o-mini labels provide a sufficient training signal for quality regression.
    Stage 2 fine-tuning data is labeled by GPT-4o-mini (§3.2). The paper acknowledges teacher model bias as a limitation (§6).
  • domain assumption The PoQ simulation parameters (cost normalization, reward functions, trust weighting) from prior work [39, 40] are valid for evaluating calibration and cascade mechanisms.
    Simulation reuses prior configurations without re-validation (§4.4).
  • domain assumption Pearson correlation with the ground-truth proxy is the appropriate metric for judge quality.
    All judge comparisons use Pearson correlation against GT proxy (§4.4). No human evaluation or alternative quality metric is used.

pith-pipeline@v1.1.0-glm · 22372 in / 3332 out tokens · 186327 ms · 2026-07-05T16:12:03.146213+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 46 canonical work pages · 3 internal anchors

  1. [1]

    Finite-time analysis of the multiarmed bandit problem.Machine Learning, 47(2–3):235–256, 2002

    Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem.Machine Learning, 47(2–3):235–256, 2002

    work page 2002

  2. [2]

    How to backdoor federated learning

    Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. InProceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108, pages 2938–2948. PMLR, 2020

    work page 2020

  3. [3]

    Suc- cinct non-interactive arguments for a von Neumann architecture

    Eli Ben-Sasson, Alessandro Chiesa, Daniel Genkin, Eran Tromer, and Madars Virza. Suc- cinct non-interactive arguments for a von Neumann architecture. In23rd USENIX Security Symposium, pages 781–796, 2014

    work page 2014

  4. [4]

    Machine learning with adversaries: Byzantine tolerant gradient descent

    Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. Machine learning with adversaries: Byzantine tolerant gradient descent. InAdvances in Neural Information Processing Systems, volume 30, 2017

    work page 2017

  5. [5]

    Petals: Collaborative inference and fine-tuning of large models

    Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Maksim Riabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin, and Colin Raffel. Petals: Collaborative inference and fine-tuning of large models. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 558–568. Association for...

    work page 2023

  6. [6]

    Practical Byzantine fault tolerance

    Miguel Castro and Barbara Liskov. Practical Byzantine fault tolerance. InProceedings of the Third Symposium on Operating Systems Design and Implementation (OSDI ’99), pages 173–186, 1999

    work page 1999

  7. [7]

    Humans or LLMs as the judge? a study on judgement bias

    Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans or LLMs as the judge? a study on judgement bias. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8301–8327, 2024

    work page 2024

  8. [8]

    Chatbot arena: An open platform for evaluating LLMs by human preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, et al. Chatbot arena: An open platform for evaluating LLMs by human preference. InProceedings of the 41st International Conference on Machine Learning (ICML), 2024

    work page 2024

  9. [9]

    UltraFeedback: Boosting Language Models with Scaled AI Feedback

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. UltraFeedback: Boosting language models with scaled AI feedback.arXiv preprint arXiv:2310.01377, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  10. [10]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. InAdvances in Neural Information Processing Systems, 2022

    work page 2022

  11. [11]

    Philip Dawid and Allan M

    A. Philip Dawid and Allan M. Skene. Maximum likelihood estimation of observer error-rates using the EM algorithm.Journal of the Royal Statistical Society. Series C (Applied Statistics), 28(1):20–28, 1979

    work page 1979

  12. [12]

    The hidden vulnerability of distributed learning in Byzantium

    El Mahdi El Mhamdi, Rachid Guerraoui, and Sébastien Rouault. The hidden vulnerability of distributed learning in Byzantium. InProceedings of the 35th International Conference on Machine Learning. PMLR, 2018. 24

    work page 2018

  13. [13]

    SimCSE: Simple contrastive learning of sentence embeddings

    Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, 2021

    work page 2021

  14. [14]

    DeBERTa: Decoding-enhanced BERT with disentangled attention

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding-enhanced BERT with disentangled attention. InInternational Conference on Learning Representations, 2021

    work page 2021

  15. [15]

    DeBERTaV3: Improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. DeBERTaV3: Improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing. InInternational Conference on Learning Representations, 2023

    work page 2023

  16. [16]

    TrueSkill™: A Bayesian skill rating system

    Ralf Herbrich, Tom Minka, and Thore Graepel. TrueSkill™: A Bayesian skill rating system. In Advances in Neural Information Processing Systems, 2007

    work page 2007

  17. [17]

    Teaching machines to read and comprehend

    Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. InAdvances in Neural Information Processing Systems, volume 28, 2015

    work page 2015

  18. [18]

    Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, et al

    Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Ar- jun Nitin Bhagoji, et al. Advances and open problems in federated learning.Foundations and Trends in Machine Learning, 14(1–2):1–210, 2021

    work page 2021

  19. [19]

    Kamvar, Mario T

    Sepandar D. Kamvar, Mario T. Schlosser, and Hector Garcia-Molina. The EigenTrust algorithm for reputation management in P2P networks. InProceedings of the 12th International Conference on World Wide Web, pages 640–651, 2003

    work page 2003

  20. [20]

    Prometheus: Inducing fine-grained evalua- tion capability in language models

    Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. Prometheus: Inducing fine-grained evalua- tion capability in language models. InInternational Conference on Learning Representations, 2024

    work page 2024

  21. [21]

    Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models

    Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Gra- ham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source lan- guage model specialized in evaluating other language models.arXiv preprint arXiv:2405.01535, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  22. [22]

    Convolutional neural networks for sentence classification

    Yoon Kim. Convolutional neural networks for sentence classification. InProceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1746–1751, 2014

    work page 2014

  23. [23]

    Evaluating the factual consistency of abstractive text summarization

    Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. Evaluating the factual consistency of abstractive text summarization. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 9332–9346, 2020

    work page 2020

  24. [24]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principles (SOSP ’23), 2023

    work page 2023

  25. [25]

    Bennett, and Marti A

    Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. SummaC: Re-visiting NLI-based models for inconsistency detection in summarization.Transactions of the Association for Computational Linguistics, 10:163–177, 2022. 25

    work page 2022

  26. [26]

    Holistic evaluation of language models.Transactions on Machine Learning Research, 2023

    Percy Liang, Rishi Bommasani, et al. Holistic evaluation of language models.Transactions on Machine Learning Research, 2023

    work page 2023

  27. [27]

    ROUGE: A package for automatic evaluation of summaries

    Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. InText Summa- rization Branches Out, pages 74–81, 2004

    work page 2004

  28. [28]

    G- Eval: NLG evaluation using GPT-4 with better human alignment

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G- Eval: NLG evaluation using GPT-4 with better human alignment. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, 2023

    work page 2023

  29. [29]

    Multidimensional quality metrics: A flexible system for assessing translation quality

    Arle Richard Lommel, Aljoscha Burchardt, and Hans Uszkoreit. Multidimensional quality metrics: A flexible system for assessing translation quality. InProceedings of Translating and the Computer 35, 2013

    work page 2013

  30. [30]

    Eliciting informative feedback: The peer-prediction method.Management Science, 51(9):1359–1373, 2005

    Nolan Miller, Paul Resnick, and Richard Zeckhauser. Eliciting informative feedback: The peer-prediction method.Management Science, 51(9):1359–1373, 2005

    work page 2005

  31. [31]

    BLEU: A method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: A method for automatic evaluation of machine translation. InProceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, 2002

    work page 2002

  32. [32]

    Pinocchio: Nearly practical verifiable computation

    Bryan Parno, Jon Howell, Craig Gentry, and Mariana Raykova. Pinocchio: Nearly practical verifiable computation. In2013 IEEE Symposium on Security and Privacy, 2013

    work page 2013

  33. [33]

    SQuAD: 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, 2016

    work page 2016

  34. [34]

    Raykar, Shipeng Yu, Linda H

    Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. Learning from crowds.Journal of Machine Learning Research, 11:1297–1322, 2010

    work page 2010

  35. [35]

    Farinha, and Alon Lavie

    Ricardo Rei, Craig Stewart, Ana C. Farinha, and Alon Lavie. COMET: A neural framework for MT evaluation. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2685–2702, 2020

    work page 2020

  36. [36]

    Sentence-BERT: Sentence embeddings using siamese BERT- networks

    Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using siamese BERT- networks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, 2019

    work page 2019

  37. [37]

    BLEURT: Learning robust metrics for text generation

    Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: Learning robust metrics for text generation. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, 2020

    work page 2020

  38. [38]

    Online learning and online convex optimization.Foundations and Trends in Machine Learning, 4(2):107–194, 2012

    Shai Shalev-Shwartz. Online learning and online convex optimization.Foundations and Trends in Machine Learning, 4(2):107–194, 2012

    work page 2012

  39. [40]

    URLhttps://arxiv.org/abs/2512.16317

    work page arXiv

  40. [41]

    Adaptive and ro- bust cost-aware proof of quality for decentralized LLM inference networks.arXiv preprint arXiv:2601.21189, 2026

    Arther Tian, Alex Ding, Frank Chen, Simon Wu, and Aaron Chan. Adaptive and ro- bust cost-aware proof of quality for decentralized LLM inference networks.arXiv preprint arXiv:2601.21189, 2026. URLhttps://arxiv.org/abs/2601.21189. 26

    work page arXiv 2026

  41. [42]

    A multi-dimensional quality scoring framework for decentralized LLM inference with proof of quality.arXiv preprint arXiv:2603.04028, 2026

    Arther Tian, Alex Ding, Frank Chen, Simon Wu, and Aaron Chan. A multi-dimensional quality scoring framework for decentralized LLM inference with proof of quality.arXiv preprint arXiv:2603.04028, 2026. URLhttps://arxiv.org/abs/2603.04028

    work page arXiv 2026

  42. [43]

    Byzantine-robust dis- tributed learning: Towards optimal statistical rates

    Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett. Byzantine-robust dis- tributed learning: Towards optimal statistical rates. InProceedings of the 35th International Conference on Machine Learning, volume 80, pages 5650–5659. PMLR, 2018

    work page 2018

  43. [44]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. BERTscore: Evaluating text generation with BERT. InInternational Conference on Learning Representa- tions, 2020

    work page 2020

  44. [45]

    Proof of Quality: A Costless Paradigm for Trustless Generative AI Model Inference on Blockchains

    Zhenjie Zhang, Yuyang Rao, Hao Xiao, Xiaokui Xiao, and Yin Yang. Proof of quality: A costless paradigm for trustless generative AI model inference on blockchains.arXiv preprint arXiv:2405.17934, 2024. URLhttps://arxiv.org/abs/2405.17934

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  45. [46]

    Meyer, and Steffen Eger

    Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP-IJCNLP), pages 563–578, 2019

    work page 2019

  46. [47]

    Judging LLM-as-a-judge with MT-Bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM-as-a-judge with MT-Bench and chatbot arena. InAdvances in Neural Information Processing Systems, volume 36, pages 46595–46623, 2023. 27

    work page 2023