Pith. sign in

REVIEW 3 cited by

Multiple-Choice Questions are Efficient and Robust LLM Evaluators

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.11966 v4 pith:D5QLEIIW submitted 2024-05-20 cs.CL

Ziyin Zhang , Zhaokun Jiang , Lizhen Xu , Hongkun Hao , Rui Wang This is my paper · ORCID
classification cs.CL
keywords constructedperformancedatasetllmsmultiple-choicerobustversionanswers
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present GSM-MC, a multiple-choice (MC) dataset constructed by collecting answers and incorrect predictions on GSM8K from 60 open-source models. Through extensive experiments, we show that LLMs' performance on the MC version of this popular benchmark is strongly correlated with their performance on the original version and is quite robust to distractor choices and option orders, while the evaluation time is reduced by a factor of up to 30. Following similar procedures, we introduce MATH-MC, constructed from MATH, and PythonIO, a new program reasoning MC dataset constructed from HumanEval and MBPP. Experimental results indicate that LLMs' performance on these MC benchmarks leaves much room for improvement. Our data and code are available at https://github.com/Geralt-Targaryen/MC-Evaluation.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models

    cs.SE 2026-07 conditional novelty 7.0 of 10

    RepoReasoner is a repository-level code-reasoning benchmark with output-prediction and call-chain tasks; the best LLM reaches only 69.1% Pass@1 even with oracle context, with low recall in dependency tracing.

  2. How Overconfidence in Initial Choices and Underconfidence Under Criticism Modulate Change of Mind in Large Language Models

    cs.LG 2025-07 conditional novelty 7.0 of 10

    LLMs show a choice-supportive bias when their initial answer is visible and overweight opposing advice relative to a Bayesian ideal observer, explaining their combined stubbornness and sensitivity to criticism.

  3. Final Checkpoints Are Not Enough: Analyzing Latent Reasoning Faithfulness Along Training Trajectories

    cs.LG 2026-07 conditional novelty 6.5 of 10

    Latent reasoning faithfulness is a property of training stage and answer format, not of architecture or the final checkpoint alone.

Pith tools