Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

The paper claims that IRT-estimated task difficulty is log-linearly related to human completion time, so model responses alone can be used to predict human task times and to forecast AI capability growth.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

BRIDGE shows that item-response-theory difficulty estimated from model performance tracks log human completion time, enabling human time prediction and a ~6-month doubling forecast for frontier task horizons.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection BRIDGE's core calibration is real and useful, but the universal transfer claim is tested on a handful of tasks and the forecasting doubles down on that weak link. the 5 major comments →

arxiv 2602.07267 v2 pith:EJSKY2OZ submitted 2026-02-06 cs.AI cs.CLcs.LG

BRIDGE: Predicting Human Task Completion Time From Model Performance

Fengyuan Liu , Jay Gala , Nilaksh , Dzmitry Bahdanau , Siva Reddy , Hugo Larochelle This is my paper
classification cs.AI cs.CLcs.LG
keywords Item Response Theorytask difficultyhuman completion timecapability forecastingbenchmark transferlatent difficultytask-length horizonexponential growth
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to prove that a task's difficulty for a machine—estimated from binary success/failure of many models—is the same quantity, up to a log-linear rescaling, as the time a human expert needs to finish it. If that holds, expensive human timing studies are needed only once: after calibrating on a set of 170 timed tasks, any new benchmark can be assigned predicted human completion times from model logs alone. The paper validates the transfer on two out-of-distribution benchmarks, then uses the mapping to forecast capability, finding that the task length at which frontier models hit 50% success doubles about every six months. A sympathetic reader would care because this turns benchmark scores into a human-interpretable, forecastable unit without new human annotation.

Core claim

The paper claims that the latent difficulty of a task, estimated by a two-parameter logistic Item Response Theory model from binary model success/failure, is log-linearly related to the time a knowledgeable human needs to complete it. After calibrating on a set of 170 tasks with expert time annotations, the paper predicts human completion times for out-of-distribution benchmarks, and uses the same mapping to track frontier model capability over time, finding that the task length at which models succeed 50% of the time doubles roughly every six months. The central relationship is log(h) = slope × b + intercept, with a reported fit of R² = 0.81 on the calibration set.

What carries the argument

A two-parameter logistic Item Response Theory (2PL IRT) model, which turns binary model–task outcomes into a latent difficulty per task, a discrimination per task, and an ability per model, together with a linear calibration log(h) = slope·b + intercept connecting latent difficulty b to the logarithm of human completion time h. The calibration resolves IRT's scale ambiguity and is what allows difficulty values from any future benchmark to be read as human minutes.

Load-bearing premise

The single global log-linear mapping, learned from 170 timed tasks in one family of benchmarks, transfers unchanged to every other task family, so a one-unit increase in latent difficulty always means the same multiplicative increase in human time.

What would settle it

Take a new domain with expert time annotations, fit the same two-parameter IRT model on model responses, and compare predicted times from the calibration line against actual recorded times; if a class of multi-hour creative tasks is systematically underpredicted, the transfer assumption is false and domain-specific recalibration is required.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • New benchmarks can receive human-time labels without running human studies, by fitting the same psychometric model to model responses and applying the calibrated line.
  • Capability can be reported as 'task length solvable at 50% success' instead of raw benchmark scores, making progress easier to compare across domains.
  • The estimated doubling time of roughly six months, obtained without human annotation, corroborates prior human-timed forecasts, suggesting model-only evaluation can track long-horizon progress.
  • Predicted task-length distributions can reveal within-benchmark structure, e.g., tasks that only require a valid submission being shorter than those that earn a medal.
  • Raising the reliability threshold from 50% to 80% success shrinks the solvable horizon to roughly one-third, a contraction that may matter for safe deployment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the log-linear mapping is domain-invariant, one could build a 'predicted person-hours' leaderboard for any benchmark with sufficient model coverage, turning accuracy scores into labour-market units.
  • The transfer assumption can be tested cheaply: collect expert times for a few dozen tasks in a new domain and check whether the same slope and intercept hold; systematic bias would argue for per-domain calibration curves.
  • The 6-month doubling time rests on the assumption that model ability grows exponentially; if growth is instead a power law, the horizon forecast would need to be refit, though the ordering of models would likely survive.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces BRIDGE, a framework that fits a two-parameter logistic Item Response Theory (2PL IRT) model to binary success/failure outcomes of models across several benchmarks (METR, SWE-bench, MLE-bench, GDPval, Cybench), yielding per-task latent difficulty b_i and per-model ability θ_j. On the subset of METR tasks with human completion-time annotations, the authors fit a log-linear calibration log(h) = slope × b + intercept (Eq. 3) and report R²=0.81. They then use this calibration to predict human completion times for other benchmarks without human-time labels, validating against SWE-bench coarse time buckets and Cybench human first-solve times. Finally, they forecast the frontier model's 'task-length horizon' over release dates from the IRT difficulty that corresponds to 50% (and 80%) success, finding an approximate doubling time of 6 months, consistent with METR's earlier estimate.

Significance. If the central claim holds — that a single global affine map from IRT difficulty to log human time transfers across benchmarks — BRIDGE would provide a scalable, human-interpretable evaluation tool and an independent capability-forecasting method, which is a potentially valuable contribution. The paper's strengths include the use of a principled psychometric model (2PL IRT), a publicly available code repository, comparison against several baselines (logit success rate, LLM estimators), and explicit acknowledgment of some limitations (e.g., the LLM-as-a-judge proxy for GDPval, the exclusion of all-unsolved Cybench tasks). The empirical calibration on METR is compelling, but the paper's central transfer assumption is tested on only two out-of-distribution benchmarks, one of which is a coarse four-bucket classification with modest accuracy and the other is a 13-task subset after selection. The 6-month forecast inherits the calibration slope and does not propagate uncertainty from the IRT and calibration stages. The result is plausible but not yet established at the level claimed by the abstract.

major comments (5)
  1. [§3, Eq. (3)] The central claim is that IRT difficulty b and log human time are linearly related by a single global slope/intercept fit on METR. The motivating argument in §3 ('If capability grows exponentially... implies a linear relationship') is not a derivation: θ_j and b_i are estimated from model performance, and the assertion that this latent scale coincides with log human time is an empirical assumption, not a consequence of exponential capability growth. This assumption is load-bearing for every downstream prediction, including the 6-month horizon. I recommend testing it directly: fit Eq. (3) separately within SWAA/HCAST/RE-Bench and test slope homogeneity; also perform leave-one-benchmark-out calibration where the mapping is fit on a subset of METR suites and evaluated on held-out METR suites and on the OOD benchmarks. Without such evidence, the transfer claim is not established.
  2. [§4.3, Figure 3] The SWE-bench validation is a coarse four-bucket classification, and the reported numbers are modest: overall accuracy 41.6%, weighted κ=0.231, and per-bucket accuracy of 18% for the 15–60 min bucket (n=261) and 21% for 1–4 hr (n=42), while the largest bucket (<15 min, n=194) gets 77%. The phrase 'substantially better alignment' is relative to baselines that are near-zero or systematically biased; absolute calibration in the middle range is weak. Please report calibration on continuous predictions (e.g., predicted median interpolated against bucket boundaries), bucket-wise error distributions, and an ordinal evaluation metric. A confusion matrix alone does not support the conclusion that predicted times are 'quantitatively accurate' for SWE-bench.
  3. [§4.1.2, Figure 4] Cybench validation uses only 13 tasks after excluding tasks unsolved by all models. This exclusion is not neutral: tasks that no model solves are likely to be the most difficult and longest, so the calibration is validated on a truncated, easier subset. With n=13, R²=0.45 and '92% within 2×' = 12/13 give wide confidence intervals; the reported point estimates are fragile. Please report results on the full Cybench task set, with an explicit treatment for unsolved tasks (e.g., censored at a lower bound or with imputed difficulty), and show leave-one-out predictions. Also state how the exclusion criterion affects the estimated calibration slope and intercept, since the paper's own logic says unsolved tasks have unreliable b estimates, which itself casts doubt on using IRT difficulty for the long tail.
  4. [§4.4, Figure 6] The 6-month doubling time is obtained by a linear fit to log task length vs. release date, where task length is a deterministic function of the estimated b for the frontier model. The bootstrap uncertainty shown (2000 resamples over frontier models) does not propagate uncertainty in the IRT parameters (a_i, b_i, θ_j) or in the calibration slope/intercept of Eq. (3). Since the horizon values all inherit the estimated calibration slope, the reported 95% CIs are likely far too narrow. Please propagate the posterior distribution of the IRT and calibration parameters (e.g., by resampling from the MCMC posterior) and report the sensitivity of the doubling time to the calibration slope. This is essential for any claim about 6 vs. 7 months.
  5. [§5.3] The authors acknowledge that different benchmark compositions change model rankings, but this also affects the task-difficulty scale. Because b_i is jointly estimated from all benchmarks, adding or removing a benchmark (or model) can shift all b_i estimates and thus all predicted human times. The paper does not report stability of b_i or of the Eq. (3) calibration under benchmark/model subsampling. This is directly relevant to the assumption that there is a single 'shared' latent scale that transfers across domains. Please add a sensitivity analysis (e.g., bootstrap over tasks/models with refitting, or a jackknife over benchmarks) and show how much the predicted times and the doubling-time estimate change.
minor comments (6)
  1. [Figure 3] The label 'BIRDGE (Ours)' is a typo for 'BRIDGE'.
  2. [Figure 2] The x-axis is labeled 'Task difficulty (b)' and the y-axis is labeled 'Task Length (human time)' on a log scale. Please state explicitly in the caption that both axes are on a log scale, and add units (minutes) to the regression equation.
  3. [§4.3] The calibration R²=0.81 is reported without confidence intervals or residual diagnostics. Please report the standard errors of slope and intercept and a residual plot, especially because the data include SWAA tasks with very short times.
  4. [§4.1.2, Cybench] The exclusion of tasks unsolved by all models is described in a parenthetical. Given its importance to the validation, this should be stated prominently and treated as a limitation in the main body, not only in a footnote-like remark.
  5. [Appendix C.1] The GDPval LLM-as-a-judge threshold (score ≥ 4) is acknowledged as a proxy. This is fine as a limitation, but the main text currently states that GDPval results are predicted without caveat; please add a caveat in §4.3.
  6. [§2.2] The sentence 'both b_i and θ_j are scale invariant' is imprecise; the model is invariant under affine transformations of the latent scale, as stated later in the same paragraph. Consider rephrasing to avoid confusion.

Circularity Check

0 steps flagged

No significant circularity: the METR calibration is a transparent fitted mapping, and the transfer claims are tested against held-out human annotations.

full rationale

BRIDGE's central empirical claim is that IRT difficulty b, estimated from model success/failure, is log-linearly related to human completion time. The paper is explicit that Equation (3) is fitted on METR human times ('we regress log h_i on b_i to obtain the log-linear mapping') and is then applied to new benchmarks. This is a standard calibration procedure, not a self-definitional reduction: b is estimated from model responses, while h comes from external human annotation; neither is defined in terms of the other. The transfer of the mapping is tested on held-out annotations (SWE-bench Verified buckets and Cybench first-solve times) that were not used to fit Equation (3), so the reported R²=0.45 (Cybench) and κ=0.231 (SWE-bench) are genuine out-of-sample checks. The 6-month doubling forecast is obtained by mapping IRT ability θ to human time via the METR-calibrated slope and then fitting a trend over release dates; although the y-axis units come from the calibration, the growth rate is determined by model-ability growth, so the doubling time is not forced by construction. The paper also acknowledges the fragility of its forecasting in Section 5.3 ('different benchmark compositions... inherent uncertainty'), which is a limitation on validity, not circularity. No load-bearing self-citations, imported uniqueness theorems, or ansatz-by-citation are present: the only close prior anchor is METR (Kwa et al., 2025), which is external and used as calibration/validation data. The empirical assumption of a single global b-to-log-time mapping transferring across domains is the main scientific risk, but that is an assumption subject to external falsification, not a circular derivation.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 1 invented entities

The central claim rests on fitted IRT parameters, a calibration line fitted to METR human times, and transferability assumptions. No physical entities are invented; the only invented construct is the shared latent difficulty scale inherited from IRT, which has a falsifiable handle through human-time predictions.

free parameters (6)
  • Per-task IRT difficulty b_i = MCMC estimates; not listed individually
    Fitted from the 51%-observed success/failure matrix; mapped to log human time in Eq. 3.
  • Per-task IRT discrimination a_i = MCMC estimates; not listed individually
    Controls the slope of item response curves; used for the 80% threshold analysis.
  • Per-model IRT ability θ_j = MCMC estimates; not listed individually
    Frontier model θ at 50% success is converted to human time horizon.
  • Calibration slope and intercept (Eq. 3) = slope ≈ 0.815 per unit b (2.26× per +1 b); intercept not reported
    OLS fit on 170 METR tasks; this is the learned bridge from latent difficulty to human time.
  • Trial-binarization threshold = 0.5 (success if ≥50% of METR trials succeed)
    Hand-chosen in Section 4.1.1; affects the binary responses used to estimate b and θ.
  • Release-window and smoothing choices = 2-month windows; 15-task smoothing
    Hand-chosen in Section 4.4; affect frontier-model selection and the probability-time curves.
axioms (5)
  • domain assumption 2PL IRT: P(y=1)=σ(a_i(θ_j−b_i)) with unidimensional latent traits and local independence.
    Adopted in Section 2.2; the whole bridge assumes one scalar difficulty/ability explains model responses across heterogeneous benchmarks.
  • domain assumption Human completion time is a global log-linear function of latent difficulty: log h = slope·b + intercept.
    Equation 3 in Section 3; fitted on METR and assumed to transfer to SWE-bench, Cybench, MLE-bench, and GDPval.
  • domain assumption Exponential model capability growth implies linear growth in θ.
    Section 3 uses this to motivate the log-linear time relation and the exponential forecast; it is not proven.
  • domain assumption METR human time annotations are trustworthy and comparable across SWAA, HCAST, and RE-Bench.
    Section 4.1.1; the calibration line depends entirely on these annotations.
  • ad hoc to paper LLM-as-a-judge with Gemini 3 Pro, score ≥4, approximates expert success on GDPval.
    Appendix C.1; the authors acknowledge the judge had no expert human responses for comparison.
invented entities (1)
  • Shared model-human latent difficulty scale (IRT b interpreted as log human time) independent evidence
    purpose: To transfer difficulty estimated from model pass/fail data to human completion time across benchmarks.
    Not a physical entity; the scale is a statistical construct. Its falsifiable handle is the OOD human-time predictions on SWE-bench and Cybench.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of BRIDGE: Predicting Human Task Completion Time From Model Performance." pith.science (2026). https://pith.science/paper/EJSKY2OZ

@misc{pith2026260207267,
  author       = {Pith},
  title        = {Pith review of: BRIDGE: Predicting Human Task Completion Time From Model Performance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EJSKY2OZ}},
  note         = {Machine review of arXiv:2602.07267}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Evaluating the real-world capabilities of AI systems requires grounding benchmark performance in human-interpretable measures of task difficulty. Existing approaches that rely on direct human task completion time annotations are costly, noisy, and difficult to scale across benchmarks. In this work, we propose BRIDGE, a unified psychometric framework that learns a latent difficulty scale from model responses and anchors it to human task completion time. Using a two-parameter logistic Item Response Theory model, we jointly estimate latent task difficulty and model capability from model performance data across multiple benchmarks. We demonstrate that latent task difficulty varies linearly with the logarithm of human completion time, allowing human task completion time to be inferred for new benchmarks from model performance alone. Leveraging this alignment, we forecast frontier model capabilities in terms of human task length and independently reproduce METR's exponential scaling results, with the 50% solvable task horizon doubling approximately every 6 months.

Figures

Figures reproduced from arXiv: 2602.07267 by Dzmitry Bahdanau, Fengyuan Liu, Hugo Larochelle, Jay Gala, Nilaksh, Siva Reddy.

Figure 1
Figure 1. Figure 1: Overview of BRIDGE. Model responses across different benchmarks (clustered by colors) are [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Task length (human completion time) vs. latent task difficulty ( [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Alignment between annotated human completion time buckets and estimated human completion [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Alignment between actual human completion time (first-solve time) and estimated completion times [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Success probability versus estimated human task completion time for different models, smoothed [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Forecasting trends of task length horizon over model release date without human task time [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Distribution of predicted human task completion times using BRIDGE across benchmarks. His [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Success probability versus estimated human task completion time for different models, smoothed [PITH_FULL_IMAGE:figures/full_fig_p021_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Forecasting trends of task length horizon over model release date. Task length where model can [PITH_FULL_IMAGE:figures/full_fig_p021_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Prompt template for LLM-as-a-Judge evaluation on GDPval. Placeholders { [PITH_FULL_IMAGE:figures/full_fig_p023_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Prompt template for LLM-based time estimation on SWE-bench. Placeholders { [PITH_FULL_IMAGE:figures/full_fig_p024_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Prompt template for LLM-based time estimation on Cybench. Placeholders in { [PITH_FULL_IMAGE:figures/full_fig_p026_12.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation

    cs.AI 2026-07 conditional novelty 6.0

    A unified corpus of 957k trial outcomes shows frontier progress is uneven and strict all-pass aggregation obscures capability and can reorder agents.

Reference graph

Works this paper leans on

14 extracted references · 1 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Has moderate familiarity with the repository (not a first-time contributor, but not the original author)

  2. [2]

    Has access to standard development tools and documentation

  3. [3]

    Will write proper tests and follow repository conventions

  4. [4]

    Needs to understand the existing code before making changes

  5. [5]

    Will need to verify the fix doesn’t introduce regressions

  6. [6]

    estimated_minutes

    Is working without AI assistance (traditional development workflow) ## The GitHub Issue <problem_statement> {problem_statement} </problem_statement> ## Your Task Analyze the problem statement and estimate the total time in minutes. Your response MUST be in the following JSON format: { "estimated_minutes": <number - your point estimate in minutes>, "justif...

  7. [9]

    Has solid CTF experience (has competed in several CTFs before)

  8. [10]

    Is familiar with the category (e.g., knows common crypto attacks, web exploits, reverse engineering tools)

  9. [11]

    Has access to standard security tools (Ghidra, Burp Suite, pwntools, etc.)

  10. [12]

    Is working alone without AI assistance

  11. [13]

    May need to research specific vulnerabilities or techniques

  12. [14]

    estimated_minutes

    Needs to understand the challenge before exploiting it ## Challenge Description <challenge_prompt> {challenge_prompt} </challenge_prompt> ## Your Task Analyze the challenge and estimate the total time in minutes. Your response MUST be in the following JSON format: { "estimated_minutes": <number - your point estimate in minutes>, "justification": "<A detai...

  13. [1250]

    URLhttp://dx.doi.org/10.1287/ijoc.2022.1250. METR. How does time horizon vary across domains?, 07 2025. URL https://metr.org/blog/ 2025-07-14-how-does-time-horizon-vary-across-domains/. Prathiba Natesan, Ratna Nandakumar, Tom Minka, and Jonathan D Rubright. Bayesian prior choice in irt estimation using mcmc and variational bayes.Frontiers in psychology, 7...

    arXiv 2022

  14. [2025]

    almost no work done,

    URLhttps://arxiv.org/abs/2503.17354. Pedro Rodriguez, Joe Barrow, Alexander Miserlis Hoyle, John P. Lalor, Robin Jia, and Jordan Lee Boyd-Graber. Evaluation examples are not equally informative: How should that change nlp leader- boards? InAnnual Meeting of the Association for Computational Linguistics, 2021. URL https: //api.semanticscholar.org/CorpusID:...

    Pith/arXiv arXiv 2021

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.