Pith. sign in

REVIEW 3 major objections 5 minor 64 references

Ask-E: An Environment for Calibrated Question Generation

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A model trained only to write calibrated math questions improves at solving them, with no stronger teacher in the loop.

desk verdict Ask-E's benchmark is a solid new evaluation setup, but the headline transfer claim is within sampling noise and needs a paired significance test before it can be believed. read the letter →

arxiv 2608.06933 v1 pith:FFSGVX6H submitted 2026-08-07 cs.CL cs.AI

Sarah Pratt , Jae Sung Park , Scott Geng , Ali Farhadi This is my paper · ORCID
classification cs.CLcs.AI
keywords calibratedquestiongenerationdifficultycalibrationlanguagemodelbenchmarkingreinforcementlearningmathematicalreasoninganswer-distributionsupervisionquestion-answertransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that generating math questions at a precisely targeted difficulty is a measurable capability, harder than answering, and trainable on its own. The Ask-E environment sets a target as a pair of solver models: a question is "calibrated" when exactly one of the two solvers can answer it correctly. Even the strongest tested asker reaches only 44.9% calibration, far from the ceiling, and calibration rates correlate with general model strength. After reinforcement learning on calibration alone, with no new math data, no contact with stronger models, and no reward for the correctness of its own answers, a small asker model improves on several contest-math benchmarks. The payoff, if right, is a supervision loop that does not require a more capable teacher as models advance.

What carries the argument

The load-bearing object is the calibrated-question criterion: a generated math question is scored as "calibrated" when exactly one of two boundary-solver models answers it correctly, placing its difficulty inside the capability gap between the pair. Around this criterion the environment adds a probing dialogue in which the asker tests both solvers before committing a final question, and a cross-check solver whose repeated independent answers filter out questions without a stable answer. The same criterion plays both roles in the paper: it is the benchmark's success label and the reward that drives RL training, so the asymmetry of supervision is built into the metric itself.

What would settle it

Re-score every benchmark session using multiple independent samples per boundary solver and count how often the session label flips between calibrated, too easy, and too hard; if label flips are common, the measured calibration rates are not a stable property of the generated questions.

Watch

Extended reading notes

Core claim

The central claim is that calibrated question generation requires and reinforces the same reasoning needed to solve the questions, and that a model can be improved purely by being rewarded for asking questions that separate two weaker or equal solvers. The paper operationalizes this with a concrete criterion: a session succeeds when exactly one of two boundary solvers matches the cross-check answer, and this same binary outcome serves as both benchmark score and RL reward. The authors train a 4-billion-parameter open-weight asker in this environment using only boundary solvers at or below its own ability, masking all solver tokens and using its own current weights as the cross-check, so that no stronger model's signal can leak into the policy. They find that this training lifts pass@8 accuracy on AIME, HMMT, and IMO AnswerBench, and improves avg@8 on three of four benchmarks, which they present as evidence that asking well-calibrated questions transfers to answering them.

Load-bearing premise

The calibration signal assumes that one sampled answer per solver at temperature 0.7 measures a stable capability gap, but a question that both solvers could solve with probability near 0.5 would be labeled "calibrated" about half the time by chance, and the reported rates have no chance baseline.

Editorial extensions

If this is right

  • Benchmarking calibration rate ranks models in the same order as their general ability, and even the best model scores below 50%, leaving room to track progress.
  • RL training on calibrated question generation improves downstream math answering with no new math data, no interaction with stronger models, and no correctness reward for the model's own answers.
  • The environment scales with the field: as frontier models improve, the boundary set can be replaced by stronger solvers to raise the difficulty ceiling without changing the protocol.
  • Probing is a major component of success: removing probing rounds drops calibration from 44.9% to 16.3%.
  • A session is gradable only when three independent cross-check samples agree, so the scoring filters out questions without a single stable answer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because each boundary solver is sampled once per session at temperature 0.7, a question near both solvers' ability might produce the calibrated label by chance about half the time; subtracting this chance baseline from the reported rates would give a clearer measure of genuine calibration skill.
  • The transfer gains are modest (about +1.1 percentage points pooled avg@8), so the supporting case would be stronger with a sweep over asker sizes or more seeds; the same environment could be tested as a curriculum with progressively harder boundary pairs.
  • The same "exactly one solver succeeds" criterion could be ported to other verifiable domains, such as code, formal proofs, or knowledge-grounded QA, making Ask-E a template for frontier-targeting benchmarks that do not require a stronger teacher.
  • The reward signal is an answer-distribution property rather than a correctness check, so future versions should guard against stochastic-solver exploits, for example by repeated sampling or by measuring per-question variance in solver responses.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. Ask-E is an environment for calibrated question generation. A question-asker model receives a target difficulty range defined by two boundary-solver LMs, probes the pair over several dialogue rounds, and emits a final math question; a session is 'calibrated' when exactly one boundary solver matches the cross-check solver's answer. The paper benchmarks five asker models on 190 boundary pairs (10 sessions each), reporting calibration rates from 25.2% to 44.9% correlated with model ability; control analyses include a no-probing baseline (16.3%), regrading with a different cross-check solver (<=0.6 pp shifts), a difficulty probe, and a direction analysis. The paper also trains Qwen3.5-4B with RL in the environment, using only weak boundary solvers, a self-judge cross-check, and no correctness reward, and reports small downstream gains on AIME, HMMT, and IMO-AnswerBench, interpreted as transfer of question-generation skill to question answering.

Significance. Ask-E is a genuinely new evaluation paradigm: the asker is graded by the differential performance of solvers at or below its own level, inverting the usual supervision direction. The benchmark is carefully constructed (20 boundary solvers, 190 pairs, consensus cross-checking, released code and rollouts), and the control analyses — no-probing baseline, cross-check swap, difficulty probe, and the one-sidedness of calibrated wins — are exactly the checks a model-behavior benchmark needs. The transfer experiment is tested on external public benchmarks (AIME, HMMT, IMO-AnswerBench) rather than on Ask-E's own scoring, so the central result is not circular by construction. If the transfer effect held up under proper paired significance testing, it would be an important finding; as it stands, the environment and benchmark are a solid standalone contribution, while the transfer claim is not yet supported by the reported evidence.

major comments (3)
  1. [§5.2, Table 1, Figure 11, Appendix I] The central claim that Ask-E training 'improves at question answering' on math benchmarks is not yet established statistically. On the pooled 550-problem suite pass@8 rises by 1.4 pp and avg@8 by 1.1 pp, but the per-benchmark changes are mixed: AIME 2022–24 pass@8 rises 2.2 pp (2 of 90 problems) while its avg@8 falls 0.7 pp; AIME 2025 pass@8 is a tie; HMMT pass@8 rises 2.2 pp on 30 problems; IMO-AnswerBench pass@8 rises 1.0 pp (4 of 400 problems). No paired significance test is reported. For a paired binary outcome with n = 550 and a baseline rate near 0.74, the standard error of the difference is approximately 2.5–2.6 pp, so the +1.4 pp pooled gain is within one standard error and would need a McNemar test (or a confidence interval excluding zero) to be distinguished from chance. The per-seed consistency argument in Appendix I does not answer this point: all three seeds are evaluated on the same 550 problems, so agreement across seeds only rules out training-run stochasticity, not benchmark sampling noise (and at the per-benchmark level even the direction is not uniform, e.g., AIME 2022–24 pass@8 seed 1 is 86.67% versus the 87.78% baseline). The abstract, Section 5.2, and Section 6 claim a transfer effect; this requires either paired significance tests that support it or a suitably weakened claim.
  2. [§3, §4.1, Appendix G.3] The benchmark's absolute calibration rates have no chance-level baseline. Each boundary solver is sampled once per session at temperature 0.7, so any question on which both solvers succeed with probability near 0.5 is labeled 'calibrated' with probability near 0.5 regardless of asker skill; the 'below 50% / substantial headroom' framing in the abstract and Section 4.2 uses 50% as an implicit reference without justifying it. The no-probing baseline (16.3%, Appendix G.3) and the direction analysis of Figure 10 (the empirically weaker solver wins only 8.8% of calibrated sessions) give partial evidence that calibrated outcomes track genuine capability gaps rather than coin-flip noise, and I do not believe the comparative ranking of askers is threatened by this issue. Nevertheless, a proper control — for example, the calibration rate of a set of existing math problems scored on the same 190 pairs, or multiple boundary-solver samples per session to estimate P(exactly one correct) per question — is needed before the absolute rates can be interpreted as measuring asker skill, and the same stochasticity directly enters the training reward of Section 5.1.
  3. [§5.1, Appendix A, Appendix E] The training-time reward is less protected against the degenerate strategies that Appendix A itself acknowledges. Appendix A states that the benchmark's three-sample cross-check consensus filters out ambiguous questions, but during training the cross-check is a single sample at temperature 0.0 from the asker's own current weights, with no consensus and no equivalence check. Under this reward, a question for which each boundary solver is correct with probability 1/2 (for example, an ambiguous or unstable question) yields an expected reward of about +0.5 (P(calibrated) = 0.5, P(too easy) = 0.25, P(too hard) = 0.25 against +1.0/+0.2/−0.2) without the question lying in any genuine capability gap, and the 'bad cross-check' filter that would flag such questions is never applied to training rollouts. The manuscript reports that degenerate behavior is unobserved (under 1% bad-cross-check) at benchmark evaluation, but that is not evidence about the training distribution. Please measure the three-sample-consensus bad-cross-check rate on held-out training rollouts, or otherwise show that the calibration improvement during training survives the benchmark's stricter scoring.
minor comments (5)
  1. [References] Reference [7] still contains the placeholders 'Commit abcdefg • accessed DD Mon YYYY'; please fill in the actual commit hash and access date.
  2. [Appendix F] The decision to score responses that hit the token cap as incorrect is justified by a plausibility argument ('models that are uncertain tend to repeat flawed reasoning'); since this choice affects the dependent variable, a brief sensitivity check (for example, excluding truncated responses and re-ranking askers) would strengthen the measurement section.
  3. [Figure 11, Appendix I] The label 'Pooled (macro, n=550)' is ambiguous; please state explicitly whether the pooled number is a macro-average over the four benchmarks or a problem-level pool, and report the weighting used.
  4. [§4.1] Boundary solvers are run with thinking effort lowered or disabled 'to encourage brevity'; because solver behavior defines the benchmark outcome, a sentence reporting whether results change under default thinking settings would increase confidence in the measurement.
  5. [§6] The conclusion restates the transfer claim as established; its wording should be aligned with whatever statistical support is reported after revision (see Major Comment 1).

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central transfer claim is tested on external benchmarks and does not reduce to the Ask-E reward; the only minor issue is a non-load-bearing self-citation in Related Work.

full rationale

Ask-E's central transfer claim (Section 5.2, Table 1) is tested on external public math benchmarks (AIME, HMMT, IMO AnswerBench) against an untrained Qwen3.5-4B baseline. The Ask-E training reward (Section 5.1: calibrated iff exactly one boundary solver matches the cross-check answer) is not used to compute any downstream accuracy, so the transfer result does not reduce to the training signal by construction. The definition of 'calibrated' in the abstract and Section 3 is a task definition rather than a derived prediction; the benchmark's 44.9% rate is a measurement under that definition, and the 'measuring question difficulty' check (Figure 6) uses an independent solver's fail rate as a proxy, not the Ask-E reward itself. The only self-citation is [18], the Delta Learning Hypothesis by co-author Scott Geng, invoked in Related Work as motivation ('We view our work as an extension of this direction'); since Section 5.2 provides direct experimental evidence for transfer, this citation is not load-bearing. The absence of a chance baseline for calibration rates and the lack of paired significance tests for the modest 1-3 pp benchmark gains are statistical-robustness concerns, not circularity. No derivation step collapses into its own inputs; the score of 2 reflects the minor self-citation rather than any structural circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper's claims rest on hand-chosen RL weights and sampling temperatures, and on the unproven premise that the two-solver calibration definition measures solution-path understanding. No new physical or mathematical entities are introduced.

free parameters (4)
  • reward weights = calibrated +1.0; too easy +0.2; too hard -0.2; format -0.05; diversity alpha 1.0
    Hand-chosen in Section 5.1 and Appendix E to shape the RL signal; the relative weights determine which failure modes are penalized and therefore affect the downstream transfer result.
  • boundary solver temperature = 0.7
    Chosen in Section 4.1 for boundary solvers; it sets the stochasticity of each solver's correctness label, and a single sample per session determines whether a session counts as calibrated.
  • probing rounds = 4 (benchmark); 3 (training)
    Hand-chosen dialogue length in Section 4.1 and Appendix E; changes how much ability information the asker receives before its final question.
  • cross-check consensus samples = 3 (benchmark); 1 (training)
    Hand-chosen in Section 3 and Appendix F; more samples reduce reward noise but the cutoff is arbitrary and does not control boundary-solver stochasticity.
assumptions (4)
  • domain assumption Generating a question calibrated to the gap between two solvers requires understanding the path to its solution, i.e., capability beyond either solver.
    This is the key insight in Section 1 that motivates both the benchmark and the transfer claim; it is asserted, not proven, and is exactly the skill the paper claims training instills.
  • domain assumption A question is well-posed if three independent cross-check samples agree on its answer.
    Used in Section 3 to define gradable sessions and filter ambiguous questions; it does not check that boundary-solver answers are stable.
  • domain assumption A single temperature-0.7 sample from a boundary solver is a faithful observation of its ability on a question.
    The calibration label in Section 4.1 and Appendix F is computed from one sample per solver; if sample correctness is noisy, labels are noisy and the benchmark's null rate is unknown.
  • domain assumption The boundary-solver pairs, ordered by parameter count and family, span a meaningful range of mathematical skill levels.
    Assumed in Section 4.1 when constructing the 190 target ranges; the paper never independently validates the ordering beyond using size as a proxy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Ask-E: An Environment for Calibrated Question Generation." pith.science (2026). https://pith.science/paper/FFSGVX6H

@misc{pith2026260806933,
  author       = {Pith},
  title        = {Pith review of: Ask-E: An Environment for Calibrated Question Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FFSGVX6H}},
  note         = {Machine review of arXiv:2608.06933}
}
read the original abstract

Today, we improve models by training and evaluating them on problems at the frontier of their abilities. Creating such problems is itself a demanding task, requiring the ability to probe model limits and generalize beyond existing question distributions. It also means placing problems at a precise difficulty level, which requires understanding what it takes to solve them. In short, generating problems calibrated to a model's current frontier demands capability beyond it, an increasingly burdensome constraint as models improve. Our key insight is that we can leverage this constraint to our advantage: a model that can generate problems consistently calibrated to a given frontier must possess capability beyond it. Accordingly, we present Ask-E, an environment that benchmarks and trains models on their ability to write questions at a given skill level, rather than answer them. Concretely, we define target skill levels as ranges bounded by the capabilities of two existing language models. A generated question is successfully calibrated if exactly one of the two models can solve it, placing it precisely within the target range and differentiating the capabilities of these models. Ask-E serves both as a benchmark and a training environment, where models generate problems calibrated to a variety of skill levels. We find that even frontier models achieve below 50% calibration on the benchmark, leaving significant headroom to measure future progress. We also show that training on this environment leads to improvements across a number of downstream math benchmarks even with no new math data, no interaction with stronger models, and no correctness-based reward.

Figures

Figures reproduced from arXiv: 2608.06933 by the authors.

Figure 1
Figure 1. Evaluation as mutual supervision. Answering hard questions clearly demands skill, but so does writing them at the right difficulty. Solving an IMO problem requires deep mathematical ability, and so does writing a problem that differentiates between top IMO contestants. This mutual demand presents a problem for standard evaluation, where improving models require increasingly capable question askers to produce questio… view at source ↗
Figure 2
Figure 2. Overview of the Ask-E environment. The question asker must generate a question calibrated to a target difficulty range, whose lower and upper bounds correspond to the ability levels of two solver models. A question is too easy if both boundary solvers answer correctly, too hard if both answer incorrectly, and calibrated if exactly one succeeds, placing its difficulty inside the target range. (Right) The asker gauges… view at source ↗
Figure 3
Figure 3. A detailed walkthrough of the Ask-E environment for a single session. (1) A boundary pair is selected from the boundary solver set, which mixes API-based and locally served models. (2) The asker probes the pair with questions and observes their responses to gauge the gap in their abilities. (3) The asker generates a final question designed to fall within this gap. (4) Both boundary solvers and the cross-check solver… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Benchmark performance of five models on Ask-E. (Left) Calibration rate across all sessions. Error bars correspond to 95% confidence intervals computed across 10 independent runs of all 190 boundary pairs. (Right) Breakdown of generated question outcomes into too easy, …
Figure 5
Figure 5. Figure 5: Pairwise calibration outcomes for Gemini 3.1 Pro across all 190 boundary solver pairs. Bound￾ary solvers are ordered by approximate model size, increasing upward and leftward (arrows). The left heatmap shows calibration rate, which is highest when the pair consists of …
Figure 6
Figure 6. Figure 6: Difficulty of questions generated for easy versus hard target ranges. Each asker model generates questions targeting an easier range (Qwen3-1.7B to Gemma-3-4b-it) and a harder range (Gemini-3-Flash to Gemini-2.5-Flash). An independent solver model (Qwen3.5-2B) attempts…
Figure 7
Figure 7. Figure 7: Training curves across three seeds. Calibration rate, too easy rate, and too hard rate over 200 training steps. All show improvement in calibration rate. 5 RL Training on Ask-E In addition to a benchmark, Ask-E can be used as a training environment. The model is traine…
Figure 8
Figure 8. Figure 8: Benchmark outcomes under two cross-check solvers. (Top) The standard grading, a consensus of three independent Gemini 3.1 Pro cross-check samples. (Bottom) A single Claude Opus 5 cross-check, with each asker’s calibrated rate annotated with its difference from the stan…
Figure 9
Figure 9. Figure 9: Pairwise outcomes for the no-probing baseline. Gemini 3.1 Pro generates 50 questions with the same task description but no probing rounds, and each question is graded against all 190 boundary pairs under the standard grading. The calibration rate falls from 44.9% (with…
Figure 10
Figure 10. Figure 10: Direction of calibrated outcomes. (Left) Calibrated wins by the pair’s weaker solver for the Gemini 3.1 Pro asker: each cell counts the calibrated sessions in which the empirically weaker solver of the pair (fewer total correct answers across the pair’s ten questions)…
Figure 11
Figure 11. Figure 11: Downstream math transfer per benchmark and pooled over the 550-problem suite. Bars show the untrained Qwen3.5-4B baseline and the mean of the three trained seeds; whiskers show ± one standard deviation across seeds, and dots mark individual seeds. Annotations give the…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 30 canonical work pages

  1. [1]

    Phi-4-mini techni- cal report: Compact yet powerful multimodal language models via mixture-of-loras

    Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. Phi-4-mini techni- cal report: Compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:2503.01743, 2025

    arXiv 2025

  2. [2]

    Aimo validation aime

    AI-MO. Aimo validation aime. https://huggingface.co/datasets/AI-MO/ aimo-validation-aime, 2024. Licensed under Apache-2.0

    work page 2024

  3. [3]

    Analysis of llms for educational question classification and generation

    Said Al Faraby, Ade Romadhony, et al. Analysis of llms for educational question classification and generation. Computers and Education: Artificial Intelligence , 7:100298, 2024

    work page 2024

  4. [4]

    System card: Claude opus 4.7

    Anthropic. System card: Claude opus 4.7. Technical report, April 2026. URL https://cdn.sanity. io/files/4zrzovbb/website/037f06850df7fbe871e206dad004c3db5fd50340.pdf

    work page 2026

  5. [5]

    System card: Claude opus 5

    Anthropic. System card: Claude opus 5. Technical report, 2026. URLhttps://www-cdn.anthropic. com/b514064af1408018e64b1ad24e7d5e75850b4ffd/Claude%20Opus%205%20System%20Card. pdf

    work page 2026

  6. [6]

    Constitutional ai: Harmlessness from ai feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022

    arXiv 2022

  7. [7]

    Verifiers: Environments for llm reinforcement learning

    William Brown. Verifiers: Environments for llm reinforcement learning. https://github.com/ PrimeIntellect-ai/verifiers, 2025. Commit abcdefg • accessed DD Mon YYYY

    work page 2025

  8. [8]

    Minimax-m1: Scaling test-time compute efficiently with lightning attention

    Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. Minimax-m1: Scaling test-time compute efficiently with lightning attention. arXiv preprint arXiv:2506.13585, 2025

    arXiv 2025

Show all 64 references
  1. [9]

    Self-questioning language models

    Lili Chen, Mihir Prabhudesai, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. Self-questioning language models. arXiv preprint arXiv:2508.03682, 2025

    2025 arXiv

  2. [10]

    Self-play fine-tuning converts weak language models to strong language models.arXiv preprint arXiv:2401.01335, 2024

    Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models.arXiv preprint arXiv:2401.01335, 2024

    2024 arXiv

  3. [11]

    U-math: A university-level benchmark for evaluating mathematical skills in llms

    Konstantin Chernyshev, Vitaliy Polshkov, Ekaterina Artemova, Alex Myasnikov, Vlad Stepanov, Alexei Miasnikov, and Sergei Tilga. U-math: A university-level benchmark for evaluating mathematical skills in llms. arXiv preprint arXiv:2412.03205, 2024

    2024

  4. [12]

    Gemini 2.5: Pushing the frontier with ad- vanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with ad- vanced reasoning, multimodality, long context, and next generation agentic capab...

    2025 arXiv

  5. [13]

    Deepseek-v4: Towards highly efficient million-token context intelligence, 2026

    DeepSeek-AI. Deepseek-v4: Towards highly efficient million-token context intelligence, 2026

    2026

  6. [14]

    Beyond benchmarks: Matharena as an evaluation platform for mathematics with llms

    Jasper Dekoninck, Nikola Jovanović, Tim Gehrunger, Kári Rögnvalddson, Ivo Petrov, Chenhao Sun, and Martin Vechev. Beyond benchmarks: Matharena as an evaluation platform for mathematics with llms

  7. [15]

    How useful are educational questions generated by large language models? InInternational Conference on Artificial Intelligence in Education, pages 536–542

    Sabina Elkins, Ekaterina Kochmar, Iulian Serban, and Jackie CK Cheung. How useful are educational questions generated by large language models? InInternational Conference on Artificial Intelligence in Education, pages 536–542. Springer, 2023

    2023

  8. [16]

    When judgment becomes noise: How design failures in llm judge benchmarks silently undermine validity

    Benjamin Feuer, Chiung-Yi Tseng, Astitwa Sarthak Lathe, Oussama Elachqar, and John P Dickerson. When judgment becomes noise: How design failures in llm judge benchmarks silently undermine validity. arXiv preprint arXiv:2509.20293, 2025

    2025

  9. [17]

    Riemann-bench: A benchmark for moonshot mathematics

    Suhaas Garre, Erik Knutsen, Sushant Mehta, and Edwin Chen. Riemann-bench: A benchmark for moonshot mathematics. arXiv preprint arXiv:2604.06802, 2026. 11

    2026 arXiv

  10. [18]

    The delta learning hypothesis: Preference tuning on weak data can yield strong gains

    Scott Geng, Hamish Ivison, Chun-Liang Li, Maarten Sap, Jerry Li, Ranjay Krishna, and Pang Wei Koh. The delta learning hypothesis: Preference tuning on weak data can yield strong gains. arXiv preprint arXiv:2507.06187, 2025

    2025 arXiv

  11. [19]

    Gemini 3 flash model card

    Google DeepMind. Gemini 3 flash model card. Technical report, December 2025. URL https: //storage.googleapis.com/deepmind-media/Model-Cards/Gemini-3-Flash-Model-Card. pdf

    2025

  12. [20]

    Gemini 3.1 flash-lite model card

    Google DeepMind. Gemini 3.1 flash-lite model card. Technical report, 2026. URLhttps://storage. googleapis.com/deepmind-media/Model-Cards/Gemini-3-1-Flash-Lite-Model-Card. pdf

    2026

  13. [21]

    Gemini 3.1 pro model card

    Google DeepMind. Gemini 3.1 pro model card. Technical report, February 2026. URL https: //storage.googleapis.com/deepmind-media/Model-Cards/Gemini-3-1-Pro-Model-Card. pdf

    2026

  14. [22]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  15. [23]

    Deepseek-r1: Incentivizing reasoning capability in llms via rein- forcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via rein- forcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  16. [24]

    Curiosity-driven red-teaming for large language models

    Zhang-Wei Hong, Idan Shenfeld, Tsun-Hsuan Wang, Yung-Sung Chuang, Aldo Pareja, James Glass, Akash Srivastava, and Pulkit Agrawal. Curiosity-driven red-teaming for large language models. arXiv preprint arXiv:2402.19464, 2024

    2024 arXiv

  17. [25]

    R-zero: Self-evolving reasoning llm from zero data

    Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu. R-zero: Self-evolving reasoning llm from zero data. arXiv preprint arXiv:2508.05004, 2025

    2025 arXiv

  18. [26]

    Prime-rl, 2025

    Prime Intellect. Prime-rl, 2025. URL https://github.com/PrimeIntellect-ai/prime-rl

    2025

  19. [27]

    Dynabench: Rethinking benchmarking in nlp

    Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vid- gen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. Dynabench: Rethinking benchmarking in nlp. In Proceedings of the 2021 conference of the North American chapter of the A...

    2021

  20. [28]

    Language self-play for data-free training

    Jakub Grudzien Kuba, Mengting Gu, Qi Ma, Yuandong Tian, Vijai Mohan, and Jason Chen. Language self-play for data-free training. arXiv preprint arXiv:2509.07414, 2025

    2025

  21. [29]

    Gon- zalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gon- zalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating System...

    2023

  22. [30]

    Math-verify: Math verification library, 2025

    Hynek Kydlíček. Math-verify: Math verification library, 2025. URL https://github.com/ huggingface/math-verify

    2025

  23. [31]

    Rewardbench: Evaluating reward models for language modeling

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. In Findings of the Association for Computation...

    2025

  24. [32]

    Questbench: Can llms ask the right question to acquire informa- tion in reasoning tasks? arXiv preprint arXiv:2503.22674, 2025

    Belinda Z Li, Been Kim, and Zi Wang. Questbench: Can llms ask the right question to acquire informa- tion in reasoning tasks? arXiv preprint arXiv:2503.22674, 2025

    2025

  25. [33]

    Autobencher: Towards declarative benchmark construction.arXiv preprint arXiv:2407.08351, 2024

    Xiang Lisa Li, Farzaan Kaiyom, Evan Zheran Liu, Yifan Mai, Percy Liang, and Tatsunori Hashimoto. Autobencher: Towards declarative benchmark construction.arXiv preprint arXiv:2407.08351, 2024. 12

    2024 arXiv

  26. [34]

    Ministral 3

    Alexander H Liu, Kartik Khandelwal, Sandeep Subramanian, Victor Jouault, Abhinav Rastogi, Adrien Sadé, Alan Jeffares, Albert Jiang, Alexandre Cahill, Alexandre Gavaudan, et al. Ministral 3. arXiv preprint arXiv:2601.08584, 2026

    2026 arXiv

  27. [35]

    Spiral: Self-play on zero-sum games incentivizes reasoning via multi-agent multi-turn reinforcement learning

    Bo Liu, Leon Guertler, Simon Yu, Zichen Liu, Penghui Qi, Daniel Balcells, Mickel Liu, Cheston Tan, Weiyan Shi, Min Lin, et al. Spiral: Self-play on zero-sum games incentivizes reasoning via multi-agent multi-turn reinforcement learning. arXiv preprint arXiv:2506.24119, 2025

    2025

  28. [36]

    Spice: Self-play in corpus environments improves reasoning

    Bo Liu, Chuanyang Jin, Seungone Kim, Weizhe Yuan, Wenting Zhao, Ilia Kulikov, Xian Li, Sainbayar Sukhbaatar, Jack Lanchantin, and Jason Weston. Spice: Self-play in corpus environments improves reasoning. arXiv preprint arXiv:2510.24684, 2025

  29. [37]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  30. [38]

    Towards robust mathematical rea- soning

    Minh-Thang Luong, Dawsen Hwang, Hoang H Nguyen, Golnaz Ghiasi, Yuri Chervonyi, Insuk Seo, Junsu Kim, Garrett Bingham, Jonathan Lee, Swaroop Mishra, et al. Towards robust mathematical rea- soning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Pr...

    2025

  31. [39]

    Learning to ask informative questions: En- hancing llms with preference optimization and expected information gain

    Davide Mazzaccara, Alberto Testoni, and Raffaella Bernardi. Learning to ask informative questions: En- hancing llms with preference optimization and expected information gain. InFindings of the Association for Computational Linguistics: EMNLP 2024 , pages 5064–5074, 2024

    2024

  32. [40]

    Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language mod- els

    Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Fara- jtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language mod- els. arXiv preprint arXiv:2410.05229, 2024

    arXiv

  33. [41]

    gpt-oss-120b & gpt-oss-20b model card, 2025

    OpenAI. gpt-oss-120b & gpt-oss-20b model card, 2025. URL https://arxiv.org/abs/2508. 10925

    2025

  34. [42]

    Aime 2025

    OpenCompass. Aime 2025. https://huggingface.co/datasets/opencompass/AIME2025, 2025

    2025

  35. [43]

    How to get your llm to generate challenging problems for evaluation

    Arkil Patel, Siva Reddy, and Dzmitry Bahdanau. How to get your llm to generate challenging problems for evaluation. arXiv preprint arXiv:2502.14678, 2025

    2025 arXiv

  36. [44]

    Do reasoning models ask better questions? a formal information-theoretic analysis on multi-turn llm games

    Daniel M Pedrozo, Telma W de L Soares, and Bryan LM de Oliveira. Do reasoning models ask better questions? a formal information-theoretic analysis on multi-turn llm games. arXiv preprint arXiv:2601.17716, 2026

    2026

  37. [45]

    Qwen3.5: Towards native multimodal agents, February 2026

    Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https://qwen.ai/ blog?id=qwen3.5

    2026

  38. [46]

    Ai-assisted generation of difficult math questions

    Vedant Shah, Dingli Yu, Kaifeng Lyu, Simon Park, Jiatong Yu, Yinghui He, Nan Rosemary Ke, Michael Mozer, Yoshua Bengio, Sanjeev Arora, et al. Ai-assisted generation of difficult math questions. arXiv preprint arXiv:2407.21009, 2024

    2024 arXiv

  39. [47]

    OpenAI GPT-5 system card

    Aaditya Singh et al. OpenAI GPT-5 system card. 2025. URLhttps://arxiv.org/abs/2601.03267

    2025 arXiv

  40. [48]

    Beyondbench: Contamination-resistant evaluation of reasoning in language models

    Gaurav Srivastava, Aafiya Shamshad Hussain, Zhenyu Bi, Swastik Roy, Priya Pitre, Meng Lu, Morteza Ziyadi, and Xuan Wang. Beyondbench: Contamination-resistant evaluation of reasoning in language models. In The Fourteenth International Conference on Learning Representations

  41. [49]

    Debate, train, evolve: Self-evolution of language model reasoning

    Gaurav Srivastava, Zhenyu Bi, Meng Lu, and Xuan Wang. Debate, train, evolve: Self-evolution of language model reasoning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 32752–32798, 2025

    2025

  42. [50]

    Question generation for adaptive education

    Megha Srivastava and Noah Goodman. Question generation for adaptive education. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) , pages...

    2021

  43. [51]

    Omega: Can llms reason outside the box in math? evaluating exploratory, compositional, and transfor- mative generalization

    Yiyou Sun, Shawn Hu, Georgia Zhou, Ken Zheng, Hannaneh Hajishirzi, Nouha Dziri, and Dawn Song. Omega: Can llms reason outside the box in math? evaluating exploratory, compositional, and transfor- mative generalization. arXiv preprint arXiv:2506.18880, 2025

    2025 arXiv

  44. [52]

    Gemma Team. Gemma 3. 2025. URL https://goo.gle/Gemma3Report

    2025

  45. [53]

    Learning to ask: When llm agents meet unclear instruction

    Wenxuan Wang, Shi Juluan, Zixuan Ling, Yuk-Kit Chan, Chaozheng Wang, Cheryl Lee, Youliang Yuan, Jen-tse Huang, Wenxiang Jiao, and Michael R Lyu. Learning to ask: When llm agents meet unclear instruction. In Proceedings of the 2025 Conference on Empirical Methods in Natural Lan...

    2025

  46. [54]

    Qg-net: a data-driven question generation model for educational content

    Zichao Wang, Andrew S Lan, Weili Nie, Andrew E Waters, Phillip J Grimaldi, and Richard G Baraniuk. Qg-net: a data-driven question generation model for educational content. In Proceedings of the fifth annual ACM conference on learning at scale , pages 1–10, 2018

    2018

  47. [55]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025

    2025 arXiv

  48. [56]

    Codeclash: Benchmarking goal-oriented software engineering

    John Yang, Kilian Lieret, Joyce Yang, Carlos E Jimenez, Ofir Press, Ludwig Schmidt, and Diyi Yang. Codeclash: Benchmarking goal-oriented software engineering. arXiv preprint arXiv:2511.00839, 2025

    2025 arXiv

  49. [57]

    Spell: Self-play reinforcement learning for evolving long-context language models

    Ziyi Yang, Weizhou Shen, Chenliang Li, Ruijun Chen, Fanqi Wan, Ming Yan, Xiaojun Quan, and Fei Huang. Spell: Self-play reinforcement learning for evolving long-context language models. arXiv preprint arXiv:2509.23863, 2025

    2025

  50. [58]

    Self-rewarding language models

    Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. Self-rewarding language models. In Forty-first International Conference on Machine Learning, 2024

    2024

  51. [59]

    Absolute zero: Reinforced self-play reasoning with zero data

    Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data. arXiv preprint arXiv:2505.03335, 2025

    2025 arXiv

  52. [60]

    Gsm-infinite: How do your llms behave over infinitely increasing context length and reasoning complexity? arXiv preprint arXiv:2502.05252, 2025

    Yang Zhou, Hongyi Liu, Zhuoming Chen, Yuandong Tian, and Beidi Chen. Gsm-infinite: How do your llms behave over infinitely increasing context length and reasoning complexity? arXiv preprint arXiv:2502.05252, 2025

    2025 arXiv

  53. [61]

    Dyval: Dy- namic evaluation of large language models for reasoning tasks.arXiv preprint arXiv:2309.17167, 2023

    Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. Dyval: Dy- namic evaluation of large language models for reasoning tasks.arXiv preprint arXiv:2309.17167, 2023

    2023 arXiv

  54. [62]

    Twinstar: A novel design for enhanced test question generation using dual-llm engine

    Qingfeng Zhuge, Han Wang, and Xuyang Chen. Twinstar: A novel design for enhanced test question generation using dual-llm engine. Applied Sciences, 15(6):3055, 2025. 14 A Limitations and Broader Impact Limitations. We identify two primary limitations of this work. First, there ...

    2025

  55. [64]

    no solution

    Then $E_0 = 1 + (5/6)E_0 + (1/6)E_1$ and $E_1 = 1 + (5/6)E_0 + (1/6)\cdot 0$. Solving gives $E_0 = 42$. \boxed{42} #Question# A fair standard 6-sided die is rolled repeatedly. What is the expected number of rolls needed to obtain two consecutive 6s? Question forwarded to both ...

    2022

  56. [2026]

    URL https://arxiv.org/abs/2605.00674

    arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.