When Can You Debias an LLM Judge? Identifiability Limits, a Test, and Designs for Top-k Ranking

Delu Zeng; Jian Xu; John Paisley; Qibin Zhao

REVIEW 3 major objections 5 minor 32 references

When Can You Debias an LLM Judge? Identifiability Limits, a Test, and Designs for Top-k Ranking

T0 review · 3 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Pairwise comparisons of LLM judge outputs cannot identify the quality/bias split; any 'debiased' score is a prior's assumption, and only paired renderings of the same content restore identifiability.

desk verdict Non-identifiability of the quality/bias split in LLM judging is real, honestly credited, and usefully framed; the paired-design escape works in controlled pools but its real-world transfer rests on an unvalidated shared-θ assumption. read the letter →

arxiv 2607.02104 v2 pith:B7CRNILA submitted 2026-07-02 cs.LG

Jian Xu , Delu Zeng , John Paisley , Qibin Zhao This is my paper · ORCID

classification cs.LG MSC 62F1562J1268T50

keywords LLM-as-a-judgeBradley-Terrymodelidentifiabilitydebiasverbositybiastop-krankingBayesianactivelearningpairedcomparison

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the standard fix for biased LLM judges—adding a bias covariate to a Bradley–Terry model and estimating the bias away—cannot work from pairwise comparisons alone. The reason is exact: reparametrizing every item's quality by δ times its covariate value and subtracting δ from the bias coefficient leaves every comparison probability unchanged, so no set of comparisons carries any information about the quality/bias split; the profile likelihood is flat to 0.0000 nats on all 48 real judge-pools and stays flat when comparisons are multiplied 26 times. If this is right, every 'debiased' score reported by such models is selected by the prior (quality a priori uncorrelated with the covariate), not recovered from data. The paper then characterizes when that prior assumption pays—benefit falls monotonically with corr(θ,x) and reverses at a configuration-dependent crossing near 0.22–0.60—and supplies two escapes: a trusted-anchor gate that tests the assumption per judge and task, and a paired rendering design that restores identifiability by having the same content appear at two covariate values. Across fifteen real judges, correction improves top-k recall by +0.20 to +0.32 on five biased-but-competent cheap judges and is a no-op on frontier ones.

What carries the argument

The load-bearing object is the invariance (Proposition 2) arising from the design matrix: the covariate difference x_a − x_b equals Σ_i x_i (item indicator column i), so the covariate column lies in the span of the quality columns and the design is rank-deficient along the direction (δx_1,…,δx_N,−δ,0). This 'flat direction' makes the likelihood exactly silent about the quality/bias split; the prior breaks the tie via Eq. (3), selecting the decomposition dCov(φ,x)/Var(x) shrunk by τ. The paper's two constructive fixes both remove the flat direction: the trusted-anchor gate uses a few gold labels to test the operational consequence of the prior assumption per (judge, covariate, task), and the

What would settle it

Compute the profile likelihood range max_{θ,κ} ℓ(c,θ,κ) over c ∈ [ĉ−1, ĉ+1] on a single-rendering item pool (one covariate value per item) and increase the number of comparisons 10×, 100×, 1000×; Proposition 2 predicts the range remains exactly 0 (up to numerical precision) at every size. A range that grows with sample size would falsify the exact non-identifiability claim. Conversely, on a paired-rendering pool the same range should grow with pool size (10.8→172.7 nats); a flat range there would falsify the design's ability to restore identifiability.

Watch

Extended reading notes

Core claim

The central claim is Proposition 2: with a free quality parameter per item and a fixed item-level covariate, the bias-aware Bradley–Terry likelihood is invariant under θ'_i = θ_i + δ x_i, c' = c − δ for any δ. Consequently no set of comparisons—within a covariate level, across levels, or in any number—identifies the bias coefficient c or the qualities θ separately; only apparent quality φ_i = θ_i + c x_i up to an additive constant and the position term κ are identified. Numerically the profile likelihood over c is flat to 0.0000 nats on all 48 real judge-pools and unchanged as comparisons scale 26×. The 'debiased' estimate is therefore not an estimate: the prior attributes to bias precisely

Load-bearing premise

The correction only works if the presentation feature modeled as bias carries no true quality signal—that is, if a terse and a verbose rendering of the same content have identical underlying quality; if changing the rendering changes what the answer actually says, the tied-θ model is misspecified and the restored identifiability is an artifact of the constructed pool.

Editorial extensions

If this is right

Any 'debiased' score from the single-rendering covariate model is a prior-induced decomposition, not a measurement; reporting it as an estimate is a category error.
Prior-based correction is justified only while corr(θ,x) stays below a configuration-dependent crossing (0.22–0.60); this explains why the same model helps on LLMBar and hurts on SummEval and Nectar.
The trusted-anchor gate decides per (judge, covariate, task) whether to enable correction, with no false enables in 6,000 decisions at K≥10 anchors and a pool-level false-enable bound of ≤6%.
Paired rendering restores likelihood identification: profile evidence over the bias coefficient rises from 0.0000 to 172.7 nats as pool size grows, and a prior-free MLE exists with only ordinary shrinkage bias remaining.
Correction improves top-k recall by +0.20 to +0.32 on five biased-but-competent cheap judges and is a no-op on frontier ones (Spearman ρ = −0.84 with competence), concentrating benefit where at-scale evaluation happens.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The non-identifiability should generalize to any item-level covariate that takes a single value per item—formatting, style, self-preference—because the same column-spanning argument applies; only features that vary within an item or within a pair (like presentation order) are estimable from comparisons alone.
A practical recipe follows: run the trusted-anchor gate with a handful of strong-judge or human labels on each new (judge, covariate, task); if the gate enables, use the bias-aware posterior, and if the task allows multiple renderings, use the paired design to obtain a true estimate instead of a prior artifact.
Because the crossing point is configuration-dependent, one could pre-screen a new benchmark by estimating corr(θ,x) on a small labeled sample and predicting whether correction will help, making the paper's characterization a cheap pre-test rather than a fixed threshold.
Benchmark builders who want debiasing to be meaningful should deliberately include multiple renderings of the same content in their pools; otherwise any measured bias coefficient is a prior artifact and cannot be compared across judges.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the central negative result is right and worth knowing. You cannot separate per-item quality θ from an item-level covariate effect c in a Bradley–Terry model using comparisons alone, because the covariate column is spanned by the item columns. The paper doesn't pretend this is new—it credits the fixed-effects/conditional-likelihood literature—but the transfer to LLM-judge debiasing is genuine, the flat-profile measurements are exact, and Eq. (3) makes clear that any “debiased” coefficient is a prior-selected decomposition, not an estimate. That part is solid.

The practical contributions are two escapes: a trusted-anchor gate and a paired rendering design. The gate is sensible, tested honestly (Clopper–Pearson bounds, pool-level units), and I have no serious objection. The paired design works in the constructed pools, where elaboration is content-free by design. The soft spot is exactly what the stress-test says: once you move to real tasks, “elaboration” often changes specificity or content, so the shared-θ assumption fails and the identifiability restoration is an artifact of the pool construction. The paper is transparent about the general issue in its Limitations section—it explicitly says the method helps only when the covariate is spurious, and shows SummEval/Nectar where length tracks quality and correction hurts. But it never validates the paired design on a real task where content-free elaboration is plausible. That is a gap, not a fatal flaw.

Other soft spots are minor and mostly acknowledged: frontier-judge claims rest on n=1–2 pools; API outputs are snapshots and not independently reproducible; no code or data are released; and the acquisition heuristic is honestly scoped as secondary, with its edge over Thompson/LUCB vanishing under a stochastic judge. The statistical care is genuinely unusual—pool-level exact permutation tests, explicit statements when power is unobtainable, and negative results reported (Claude-Haiku, Coder-6.7B). That counts for a lot.

For a reader: this is for anyone building LLM-judge pipelines or doing preference learning. The non-identifiability result should change how you read “bias-corrected” leaderboard numbers. It deserves a serious referee; the main revisions I would want are artifact release and an external test of the paired design on a task with genuinely content-free elaboration.

Referee Report

3 major / 5 minor

Summary. The paper studies the identifiability of bias coefficients in a Bradley–Terry model for LLM pairwise judgments. Its central negative result (Prop. 2, §4) is an exact design-column identity: with a free item-quality parameter θ_i per item and a fixed item-level covariate x_i, the covariate column lies in the span of the item columns, so the likelihood is invariant under θ'_i = θ_i + δ x_i, c' = c - δ. Consequently, pairwise comparisons alone cannot separate item quality from an item-level presentation bias, and any reported "debiased" coefficient is selected by the prior, not identified by data. The paper demonstrates this empirically with flat profile likelihoods (0.0000 nats) across 48 real judge-pools, and it proposes two escapes: a trusted-anchor gate that tests the operational validity of correction per (judge, covariate, task), and a paired-rendering design in which two renderings of the same base content share one θ, restoring likelihood identifiability. Secondary contributions include a top-k-aware acquisition heuristic and a large empirical study of fifteen LLM judges, showing that correction helps biased-but-competent cheap judges and is a no-op on frontier judges. The manuscript is unusually careful about statistical inference (exact permutation tests, pool as the unit of analysis), honest about small samples and protocol-dependence, and explicit about the scope of its claims.

Significance. If the results hold, the paper makes a valuable and actionable point for the LLM-judging community: the common practice of adding bias covariates to a Bradley–Terry model and reporting the coefficient as "estimated debias" is formally unjustified — the coefficient is a prior-selected decomposition, not a likelihood-identified quantity. The proof of Prop. 2 is concise and rigorous, and the empirical flatness measurement is a clear, reproducible demonstration of a non-obvious failure mode. The trusted-anchor gate is a sensible, honestly evaluated safeguard, and the paired-rendering design is a theoretically clean remedy under an explicit tied-θ assumption. The paper also performs careful statistical analysis, using exact paired permutation tests and acknowledging when significance is unobtainable due to small pool counts. The main weakness is that the paired-rendering escape is validated only on pools constructed to satisfy the tied-θ assumption, and the paper does not establish that real rendering manipulations can meet this condition or provide a way to verify it in practice.

major comments (3)

[§5, paired rendering design; Limitations] The paired-rendering design restores identifiability only under the assumption that terse and verbose renderings of a base content have exactly the same true quality θ. In the controlled pools this holds by construction (content-free elaboration clauses, fixed statement counts), but on real tasks elaboration often changes specificity, clarity, or factual content. When θ_v ≠ θ_t, the within-base comparison logit is θ_v − θ_t + c(x_v − x_t) + κ, and the design again has the covariate column lying in the span of the item columns, so the null direction of Prop. 2 returns. The paper's own external-validity results (SummEval, Nectar) show that length often legitimately tracks quality, which is precisely the regime where the tied-θ assumption is violated. This is load-bearing for the positive claim that the paired design is a practical escape; the manuscript should either provide a method to te
[Abstract and §5, paired rendering design] The abstract and §5 describe the paired design as "the only design-based intervention we study that restores likelihood identification," and the conclusion recommends it as an actionable escape. However, the only empirical validation is on the constructed pools; no real-task paired-rendering evaluation is reported. The LLMBar example in §5 uses supervision from gold labels, and the SummEval/Nectar results show harmful correction when length is legitimate. Thus the practical scope of the paired-design claim is narrower than the presentation suggests. The authors should either add a real-task demonstration with a quality-neutrality check or temper the wording to make clear that the design's applicability depends on an assumption that must be verified per rendering protocol.
[§5, Table 3; gate evaluation] The trusted-anchor gate is evaluated with 100 resamples per pool and Clopper–Pearson bounds, which is commendable. However, the claim "no false enables in 6,000 decisions at K≥10 anchors" is presented prominently; the actual risk bound is 0/60 pools, with an upper bound of 6%, and the paper correctly notes this. The concern is that the gate's behavior on "legitimate pools" is defined by the same synthetic construction that ensures corr(θ, x) is high; real legitimate covariates may have different patterns of correlation with θ, and the gate's false-enable rate could differ. This is not a fatal flaw, but the manuscript should more clearly separate the synthetic validation from any implied real-world safety guarantee.

minor comments (5)

[§4, Eq. (3)] The derivation of the MAP estimate would benefit from an explicit statement that Eq. (3) is obtained by profiling out the additive constant α; the text explains this, but a short algebraic step would help readers see why the centered inner products appear.
[§5, Table 2] The profile-likelihood range is reported as 0.0000 nats in the unpaired design. Since the proof of Prop. 2 establishes exact flatness, the numerical zeros should be described as ``numerically zero to the displayed precision,'' to avoid over-interpreting floating-point results.
[§5, Figure 3] The crossing point is stated to be configuration-dependent (0.22–0.60), and Table 7 in the appendix supports this. Consider moving Table 7 into the main text or at least citing it when the crossing is first discussed, since the reader may otherwise take the 0.45 figure as a universal threshold.
[Throughout] Minor typos and formatting artifacts: e.g., "top-k identification" appears as "topk" in a few places, and the bold-faced "0.0000 nats" in the abstract is visually odd. A careful proofread would improve readability.
[References] Several cited works appear in the related-work section but not in the reference list (e.g., the discussion of fixed-effects/incidental parameters cites Neyman and Scott 1948 and Lancaster 2000, which are listed, but some other citations such as "Zeng et al. 2024" appear in text and references; please ensure all in-text citations are in the bibliography).

Circularity Check

0 steps flagged · score 0.0 of 10

No meaningful circularity: the non-identifiability theorem is derived from the model, and the paper explicitly discloses that its correction is prior-selected rather than data-recovered.

full rationale

The derivation chain is self-contained. Proposition 2 is an algebraic consequence of Eq. (1): substituting θ'_i = θ_i + δx_i, c' = c − δ leaves every comparison logit unchanged, and the design identity x_a − x_b = Σ_i x_i(e_{a,i} − e_{b,i}) shows the c column lies in the span of the item columns. No fitted quantity is passed off as a prediction. Eq. (3) is derived from the stated Gaussian priors and is explicitly labeled as a prior-induced decomposition: 'our estimator is therefore not "separating quality from presentation using the data"; it is applying the assumption that true quality is a priori uncorrelated with the presentation covariate.' This is disclosure of prior dependence, not concealed circularity. The paired-rendering design is also derived: a within-base comparison has logit c(x_verbose − x_terse) + κ under the tied-θ model; the paper states the assumption ('a shared θ_i then appears at two covariate values') and its own limitation that on SummEval 'length legitimately signals quality' and debiasing hurts. That is an external-validity caveat, not a reduction of the claim to its inputs. The LLMBar validation is explicitly supervised/cross-validated, and the paper says it 'validates the mechanism... rather than the unsupervised top-k pipeline.' There are no load-bearing self-citations; the closest prior results (Neyman–Scott, Chamberlain, Andersen, Rasch) are external, and the paper says it 'claims no theorem that would surprise a psychometrician.' The 'debiasing is prior-selected' claim is the paper's own negative result, not a hidden assumption of its derivation.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

Central claims rest on the BT likelihood model, Gaussian priors, and the controlled-benchmark construction. The non-identifiability theorem is conditional on a fixed covariate per item; the paired design adds the shared-theta assumption. The trusted-anchor gate assumes anchor labels are correct. No new physical or latent entities are introduced; 'apparent quality' is a derived reparameterization, not an independent postulate.

free parameters (2)

quality prior precision lambda = 1 (chosen, not fitted)
Sets the shrinkage of theta in Eq. (4) and the Laplace posterior; sensitivity swept across 0.2, 1, 5 without changing qualitative conclusions.
bias/position prior precision lambda_b = tau^-2 = 0.1 (chosen, not fitted)
Controls how strongly the prior selects the theta/c split; Eq. (3) depends explicitly on tau^-2. Fixed moderate value used throughout; sensitivity swept.

assumptions (6)

domain assumption Judge verdicts follow a logistic Bradley-Terry model with additive quality difference, covariate difference, and position term (Eq. 1).
The entire identifiability analysis is inside this model; if the true judge is not logistic-additive, the negative result does not directly apply.
domain assumption Each item has one fixed quality theta_i and one fixed covariate x_i in the unpaired design.
Needed for the design-column identity (6); if covariates vary within an item across queries, the invariance breaks, as the paired design exploits.
domain assumption Gaussian priors theta ~ N(0, lambda^-1 I) and c, kappa ~ N(0, tau^2).
The MAP estimate and Eq. (3) depend on these priors; they are modeling choices, not derived facts.
domain assumption In the controlled benchmark, ground-truth quality is theta_i = (#true - #false) with fixed statement count, and corr(theta, verbosity) is approximately zero by construction.
Makes top-k ground truth known; the bias-aware gain is measured under this constructed zero-correlation regime.
ad hoc to paper In the paired design, the terse and verbose renderings of a base content share exactly the same theta.
This shared-theta tying is what makes within-base comparisons read c(x_a - x_a') + kappa; it is introduced specifically to break the invariance and is only guaranteed by the authors' pool construction.
domain assumption Trusted anchors are correct ground-truth labels (from a strong judge or humans).
The gate's safety bound is conditional on anchor correctness; biased or non-representative anchors would invalidate the false-enable bounds.

how reviews work

0 comments

Cite this review

Pith. "Pith review of When Can You Debias an LLM Judge? Identifiability Limits, a Test, and Designs for Top-k Ranking." pith.science (2026). https://pith.science/paper/B7CRNILA

@misc{pith2026260702104,
  author       = {Pith},
  title        = {Pith review of: When Can You Debias an LLM Judge? Identifiability Limits, a Test, and Designs for Top-k Ranking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B7CRNILA}},
  note         = {Machine review of arXiv:2607.02104}
}

abstract

Large language models (LLMs) are increasingly used as cheap, scalable judges that compare candidate outputs pairwise. Because such judges prefer verbose or well-formatted answers, the natural fix is to add bias covariates to a Bradley--Terry model and estimate the bias away. We show this cannot work as advertised: the quality/bias split is \emph{not identified} by pairwise comparisons, and the failure is exact -- across $48$ real judge-pools the profile likelihood over the coefficient is flat to $\mathbf{0.0000}$ \textbf{nats}, and scaling the comparisons $26\times$ buys none. A ``debiased'' score is selected by the prior, not recovered from data. Our contribution is accordingly not a better estimator but a characterization of \emph{when prior-based correction is justified}, plus designs that supply the missing information when it is not. The assumption the prior encodes -- quality is a priori uncorrelated with the covariate -- pays only while $\mathrm{corr}(\theta,x)$ stays below a crossing point (configuration-dependent, $0.22$--$0.60$), which is what makes the same model help on LLMBar and hurt on SummEval and Nectar. We give two escapes: a \textbf{trusted-anchor gate} that decides per (judge, covariate, task) (no false enables in $6{,}000$ decisions at $K\ge10$ anchors, a rate our sample bounds at $\le6\%$), and a \textbf{paired rendering design}. Across fifteen real LLM judges bias is heterogeneous and capability-dependent: correction improves \topk{} recall by $0.20$--$0.32$ on five biased-but-competent cheap judges and is a no-op on frontier ones (Spearman $\rho{=}{-}0.84$ between competence and gain over the $14$ competent judges, $p{<}10^{-3}$), concentrating the benefit where at-scale evaluation happens.

Figures

Figures reproduced from arXiv: 2607.02104 by the authors.

**Figure 3.** top-k recall vs. comparison budget (Llama-3.1- 8B judge, 5 oracles × 8 seeds, ±s.e.). The top-k-aware rule reaches the bias-aware ceiling fastest; global-uncertainty is worst at small budgets; the naive model (no bias term) plateaus at a wrong top-k regardless of budget. External validity on LLMBar. Our controlled benchmark isolates spurious verbosity by construction; LLMBar (Zeng et al. 2024) tests the same phenome… view at source ↗

**Figure 2.** External validity on LLMBar (judge: GPT-4o [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Figures from the paper (9 more)

**Figure 4.** Figure 4: How the budget is spent (Llama judge, B=200). top-k-aware acquisition concentrates comparisons on items near the top-k boundary (∼3× more than round-robin at distance 0) and starves items far from it, whereas round-robin spreads effort uniformly. This is the mechanism …

**Figure 3.** Figure 3: The prior assumption governs the outcome. Sweep [PITH_FULL_IMAGE:figures/full_fig_p007_3.png]

**Figure 6.** Figure 6: External validity on LLMBar (judge: GPT-4o [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 4.** Figure 4: Why naive aggregation fails (Llama judge, full [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

**Figure 7.** Figure 7: Cost–quality frontier. A cheap biased judge (GPT [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

**Figure 5.** Figure 5: Where the method helps (our tested judges). Each [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

**Figure 7.** Figure 7: How the budget is spent (Llama judge, B=200). top-k-aware acquisition concentrates comparisons on items near the top-k boundary (∼3× more than round-robin at distance 0) and starves items far from it, whereas round-robin spreads effort uniformly. This is the mechanism …

**Figure 8.** Figure 8: Cost–quality frontier for this workload and this pricing snapshot. A cheap biased judge (GPT-4o-mini) with our correction reaches 0.89 recall (at its best budget B ≈200) for ≈20× less than a frontier judge (GPT-5.1). The ratio is an instance, not a constant: it is comp…

**Figure 9.** Figure 9: Acquisition-score ablation (Llama, B=120, ±s.e.). Each factor of the score (5) helps; dropping the top-kboundary entropy term costs the most (it collapses to global D-optimality). All variants beat random. ping the boundary-entropy term—the top-k-aware part— costs the…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 4 linked inside Pith

[1]

Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education

Clancey, William J. Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education. Proceedings of the Eighth International Joint Conference on Artificial Intelligence (IJCAI-83)
[2]

Classification Problem Solving

Clancey, William J. Classification Problem Solving. Proceedings of the Fourth National Conference on Artificial Intelligence
[3]

, title =

Robinson, Arthur L. , title =. 1980 , doi =. https://science.sciencemag.org/content/208/4447/1019.full.pdf , journal =

1980
[4]

New Ways to Make Microcircuits Smaller---Duplicate Entry

Robinson, Arthur L. New Ways to Make Microcircuits Smaller---Duplicate Entry. Science
[5]

Clancey and Glenn Rennels , abstract =

Diane Warner Hasling and William J. Clancey and Glenn Rennels , abstract =. Strategic explanations for a diagnostic consultation system , journal =. 1984 , issn =. doi:https://doi.org/10.1016/S0020-7373(84)80003-6 , url =

doi:10.1016/s0020-7373(84)80003-6 1984
[6]

and Rennels, Glenn R

Hasling, Diane Warner and Clancey, William J. and Rennels, Glenn R. and Test, Thomas. Strategic Explanations in Consultation---Duplicate. The International Journal of Man-Machine Studies
[7]

Poligon: A System for Parallel Problem Solving

Rice, James. Poligon: A System for Parallel Problem Solving
[8]

Transfer of Rule-Based Expertise through a Tutorial Dialogue

Clancey, William J. Transfer of Rule-Based Expertise through a Tutorial Dialogue

Show all 32 references

[9]

The Engineering of Qualitative Models

Clancey, William J. The Engineering of Qualitative Models
[10]

2023 , eprint=

Attention Is All You Need , author=. 2023 , eprint=

2023
[11]

Pluto: The 'Other' Red Planet

NASA. Pluto: The 'Other' Red Planet
[12]

The Method of Paired Comparisons , author=

Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons , author=. Biometrika , volume=
[13]

Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and others , booktitle=. Judging
[14]

Proceedings of the Association for Computational Linguistics (ACL) , year=

Large Language Models are not Fair Evaluators , author=. Proceedings of the Association for Computational Linguistics (ACL) , year=
[15]

International Conference on Learning Representations (ICLR) , year=

Evaluating Large Language Models at Evaluating Instruction Following , author=. International Conference on Learning Representations (ICLR) , year=
[16]

Length-Controlled

Dubois, Yann and Galambosi, Bal. Length-Controlled. arXiv preprint arXiv:2404.04475 , year=

arXiv
[17]

arXiv preprint arXiv:2310.10076 , year=

Verbosity Bias in Preference Labeling by Large Language Models , author=. arXiv preprint arXiv:2310.10076 , year=

arXiv
[18]

From Generation to Judgment: Opportunities and Challenges of

Li, Dawei and Jiang, Bohan and Huang, Liangjie and others , journal=. From Generation to Judgment: Opportunities and Challenges of
[19]

Findings of the Association for Computational Linguistics (ACL) , year=

Bayesian Prompt Ensembles: Model Uncertainty Estimation for Black-Box Large Language Models , author=. Findings of the Association for Computational Linguistics (ACL) , year=
[20]

Ross, Brendan Leigh and Vouitsis, No. Textual. arXiv preprint arXiv:2506.10060 , year=

arXiv
[21]

Huang, Hang and others , journal=
[22]

Advances in Neural Information Processing Systems (NeurIPS) , year=

Deep Bayesian Active Learning for Preference Modeling in Large Language Models , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=
[23]

arXiv preprint arXiv:1112.5745 , year=

Bayesian Active Learning for Classification and Preference Learning , author=. arXiv preprint arXiv:1112.5745 , year=

arXiv
[24]

Online Rank Elicitation for

Sz. Online Rank Elicitation for. Advances in Neural Information Processing Systems (NeurIPS) , year=
[25]

The Annals of Statistics , volume=

Active Ranking from Pairwise Comparisons and When Parametric Assumptions Do Not Help , author=. The Annals of Statistics , volume=
[26]

Econometrica: journal of the Econometric Society , pages=

Consistent estimates based on partially consistent observations , author=. Econometrica: journal of the Econometric Society , pages=. 1948 , publisher=

1948
[27]

The review of economic studies , volume=

Analysis of covariance with qualitative data , author=. The review of economic studies , volume=. 1980 , publisher=

1980
[28]

Journal of econometrics , volume=

The incidental parameter problem since 1948 , author=. Journal of econometrics , volume=. 2000 , publisher=

1948
[29]

Journal of the Royal Statistical Society Series B: Statistical Methodology , volume=

Asymptotic properties of conditional maximum-likelihood estimators , author=. Journal of the Royal Statistical Society Series B: Statistical Methodology , volume=. 1970 , publisher=

1970
[30]

Psychometrika , volume=

Measurement invariance, factor analysis and factorial invariance , author=. Psychometrika , volume=. 1993 , publisher=

1993
[31]

2012 , publisher=

Differential item functioning , author=. 2012 , publisher=

2012
[32]

1989 , school=

Many-faceted Rasch measurement , author=. 1989 , school=

1989

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.

[1] [1]

Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education

Clancey, William J. Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education. Proceedings of the Eighth International Joint Conference on Artificial Intelligence (IJCAI-83)

[2] [2]

Classification Problem Solving

Clancey, William J. Classification Problem Solving. Proceedings of the Fourth National Conference on Artificial Intelligence

[3] [3]

, title =

Robinson, Arthur L. , title =. 1980 , doi =. https://science.sciencemag.org/content/208/4447/1019.full.pdf , journal =

1980

[4] [4]

New Ways to Make Microcircuits Smaller---Duplicate Entry

Robinson, Arthur L. New Ways to Make Microcircuits Smaller---Duplicate Entry. Science

[5] [5]

Clancey and Glenn Rennels , abstract =

Diane Warner Hasling and William J. Clancey and Glenn Rennels , abstract =. Strategic explanations for a diagnostic consultation system , journal =. 1984 , issn =. doi:https://doi.org/10.1016/S0020-7373(84)80003-6 , url =

doi:10.1016/s0020-7373(84)80003-6 1984

[6] [6]

and Rennels, Glenn R

Hasling, Diane Warner and Clancey, William J. and Rennels, Glenn R. and Test, Thomas. Strategic Explanations in Consultation---Duplicate. The International Journal of Man-Machine Studies

[7] [7]

Poligon: A System for Parallel Problem Solving

Rice, James. Poligon: A System for Parallel Problem Solving

[8] [8]

Transfer of Rule-Based Expertise through a Tutorial Dialogue

Clancey, William J. Transfer of Rule-Based Expertise through a Tutorial Dialogue