{"id":"cdac27ca-02c6-4bea-bfba-ea1c17517ca4","arxiv_id":"2608.09538","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"TCS-Bench presents 300 research-level theorem-proving tasks from top theory papers; the best tested model solves 68 percent, and the authors report an automated verifier with over 90 percent human-expert agreement.","lead":"The authors built TCS-Bench, a set of 300 theorem-proving tasks drawn from recent STOC, FOCS, and SODA papers, and tested frontier AI models on them. They also built an automated proof verifier that they report agrees with human experts over 90 percent of the time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reference verifier may be scoring similarity to the ground-truth proof rather than mathematical correctness; its prompt explicitly requires matching ground-truth case structure, so the 90% accuracy claim does not yet support the benchmark's reported scores.","rationale":"The reader's weakest assumption is that the verifier's accuracy measured on the calibration set may not generalize to the full benchmark, especially for correct proofs that differ structurally from the ground truth. My review reinforces that assumption with concrete textual evidence: the verifier prompt in Appendix A.1 explicitly requires the student proof to match the ground-truth proof's case boundaries and logical structure, so the verifier is not purely checking validity. The calibration procedure in Section 4.2 also uses solver-generated proofs and tunes the prompt with GEPA on the same labeled set, with no reported held-out evaluation. Because every model score in Tables 1 and 2 is produced by this verifier, the benchmark's central reliability claim is not yet established. This does not undermine the benchmark construction itself, which is a useful contribution, but it does mean the reported model accuracies should be treated as provisional. The reader's CONDITIONAL verdict already captures this situation, so I would keep the verdict unchanged rather than moving it. The concrete held-out test proposed above would settle whether the concern is real: if the verifier accepts structurally divergent correct proofs at high accuracy with the frozen prompt, the central claim is supported; if not, the benchmark scores need recalibration or a redesigned verifier.","tokens_in":20600,"tokens_out":3961,"duration_ms":42335,"concrete_test":"Construct a held-out set of at least 100 proof/statement pairs in which each correct proof is written by an independent human expert instructed to avoid copying the ground-truth proof's structure and, where possible, to use a different proof strategy, paired with plausible incorrect proofs. Run the reference verifier on this set both with and without the ground-truth proof as input. If accuracy on structurally divergent correct proofs falls below roughly 85%, or differs substantially from the reported >90%, the verifier is measuring ground-truth similarity rather than mathematical correctness. Report the accuracy on this held-out set after freezing the GEPA-tuned prompt.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the verifier reliably judges proof correctness, which would make the 300 task scores meaningful. The weakest point is not just the absence of a held-out verifier test: the verifier's design appears to score similarity to a particular proof. Section 4.1 states that the verifier receives the ground-truth proof, and the prompt in Appendix A.1 instructs it to compare the student's logic against the ground_truth_proof and to mark 0 if the student skips a boundary case that the ground-truth proof relies on. A correct proof that establishes the target through a genuinely different route can therefore be rejected even though it is valid. The 50 correct calibration proofs were generated by the solver and reviewed by humans (Section 4.2), so they likely share a limited structural family; they do not demonstrate acceptance of diverse correct proofs. Moreover, the >90% figure is reported on the same expert-labeled set used by GEPA to tune the prompt, with no held-out accuracy or confidence interval. Unless the verifier is tested on correct proofs that differ structurally from the ground truth, the reported accuracy and all model rankings remain uncalibrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TCS-Bench, a benchmark of 300 theorem-proving tasks extracted from papers published at STOC, FOCS, and SODA between 2020 and 2026. For each task, the authors construct a self-contained context from the source paper's dependency graph, hide a target statement's proof, and optionally mask intermediate lemmas to create a curriculum of difficulty levels. The paper then evaluates several frontier LLMs (and an agentic harness called Colosseum) on the benchmark, scoring generated proofs with an automated verifier agent built from Gemini 3.1 Flash. The central methodological claim is that this verifier achieves over 90% accuracy against 100 human-expert proof judgments, which the paper uses to justify the trustworthiness of the reported model accuracies in Tables 1 and 2. The manuscript also reports that a cross-model selection rule within Colosseum improves accuracy from 54.0% to 67.7% on the benchmark.","tokens_in":20949,"tokens_out":5133,"duration_ms":49325,"significance":"If the verifier-accuracy claim were properly supported, TCS-Bench would be a genuinely valuable contribution: it addresses a real gap in research-level mathematical reasoning benchmarks, offers a transparent and reproducible construction pipeline, and provides a scalable way to grade natural-language proofs. The paper's strengths include the public release of tasks, the dependency-DAG-based difficulty scaling, the explicit quality-filtering checks, and the detailed prompts in Appendix A.1. However, the entire empirical contribution — including all model rankings and the Colosseum selection results — rests on the verifier's reliability. Because the current evidence for that reliability is an in-sample calibration after prompt tuning on the same small set of human-labeled proofs, the central claim is not yet established.","major_comments":[{"comment":"The >90% accuracy figure is an in-sample estimate, not a held-out evaluation. Section 4.2 states that GEPA was run on the 100-example expert-labeled set to produce the prompt, and the same set is then used to report accuracy; no separate test split, no confidence interval, and no cross-validation are described. The contribution bullet in Section 1.1 calls this a \"held-out set of 100 human-labeled proofs,\" which is contradicted by Section 4.2. Since every score in Tables 1 and 2 is produced by this verifier, the paper's central quantitative claim — that the reported model accuracies are trustworthy measurements of proof-generation ability — is not supported by the evidence as written. The authors should report accuracy on a held-out human-labeled set that was not used for GEPA tuning, with a confidence interval, and should clarify the relationship between the development and evaluation sets.","section":"Section 4.2; Section 1.1"},{"comment":"The verifier receives the ground-truth proof as an input and is instructed to compare the student's logic, variable tracking, and case-splitting against it. In particular, the Case Integrity criterion says that if the ground_truth_proof relies on a specific case split, the student must address those exact boundaries, and the prompt explicitly requires matching the logical depth and cases of the ground truth. This design systematically penalizes correct proofs that establish the target through a different but valid route, which is a core failure mode for a proof verifier. The 50 correct calibration proofs were generated by the same solver and human-reviewed, so they likely share a limited structural family; the paper provides no evidence that the verifier accepts correct proofs outside that family. The authors should test the verifier on a set of correct proofs that are intentionally structurally diverse — e.g., human-written proofs from different authors, proofs from different models, and proofs that use alternative case splits — and report the acceptance rate on that set.","section":"Section 4.1; Appendix A.1"},{"comment":"The cross-model selection results in Table 2 and the associated AUC values (0.854 and 0.637) are computed on TCS-Bench itself, and the accuracy gains are graded by the same unvalidated verifier. The text states that the stronger critique direction was selected using \"cross-model critique accuracy measured on TCS-Bench.\" If the verifier is biased toward ground-truth-similar proofs of the kind described in the previous comment, then the apparent improvement from 54.0% to 67.7% may reflect selection for verifier-approved style rather than mathematical correctness. These results should be re-reported after the verifier is validated on held-out, structurally diverse proofs, and the selection rule should be tuned on a set disjoint from the final evaluation set.","section":"Section 5.2"}],"minor_comments":[{"comment":"The bullet claiming a \"held-out set of 100 human-labeled proofs\" is inconsistent with Section 4.2, which describes using that same set for GEPA prompt tuning; please revise the wording to describe the actual data split.","section":"Section 1.1"},{"comment":"Table 1 has a stray formatting artifact in the header (\"Accuracy()\"), and the heading \"Results & Analysis\" appears twice in Section 5.1; these should be cleaned up.","section":"Section 5.1"},{"comment":"There are several typographical issues in Section 5.2, including \"AUC of 0 .854\" (spurious space), \"0 .637\", and \"47problems\" missing a space; please proofread the numerical expressions.","section":"Section 5.2"},{"comment":"The paper states that the context is compressed to fit a standard context window of 10,000 tokens, but Section 5.1 describes models with 128K token budgets; please clarify whether the 10,000-token limit is a fixed benchmark design choice or an artifact of the evaluation setup.","section":"Section 3.3"},{"comment":"The GEPA tuning procedure is described only as \"the GEPA improvement pipeline\"; for reproducibility, please report the number of GEPA iterations, the prompt search space, the number of candidate prompts evaluated, and the final prompt selection criterion.","section":"Section 4.2"},{"comment":"The example tasks in Figures 2 and 3 are internally inconsistent: the target statement about Algorithm 2 does not match the lemma about bounded sequences (u_n). Please replace these with a coherent running example or remove the illustrative statements.","section":"Figures 2 and 3"}],"recommendation":"major_revision","confidential_remarks":"The core issue is the verifier calibration: the >90% figure is in-sample after GEPA tuning, and the verifier prompt is structurally biased toward ground-truth-similar proofs. I do not view this as a reject because the benchmark construction is thoughtful and the problem is addressable with an additional held-out human-labeled evaluation and a diversity-focused verifier test. I would encourage the editor to require the authors to release the calibration annotations and the exact data split, and to report verifier accuracy with a confidence interval before the paper is accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read of TCS-Bench.\n\nThe benchmark construction is the real contribution. Turning the dependency graph of published TCS papers into proof-completion tasks with rank-based difficulty masking is a genuinely new format, and the quality filtering pipeline is careful. The 300 tasks are public, which is the right call. The cross-model analysis in Section 5.2 is also interesting: finding that self-rejection is more informative than self-acceptance, and that Gemini 3.6 Flash critiques of Gemini 3.1 Pro proofs reach AUC 0.854, is a useful empirical observation even if it is preliminary.\n\nBut the central quantitative claim—verifier accuracy above 90%—does not yet support the model scores. The calibration is in-sample: GEPA tuned the verifier prompt on the same 100 human-labeled examples that are then used to report accuracy, with no held-out test and no confidence interval. That alone would be a fixable but real gap.\n\nThe bigger problem is that the verifier receives the ground-truth proof as input, and the prompt in Appendix A.1 instructs it to compare the student's logic and cases against that ground truth. Case Integrity explicitly says that if the ground-truth proof relies on a specific case split, the student must address those boundaries, and the output format requires the student proof to 'match the logical depth/cases of the ground truth.' A correct proof that takes a genuinely different route, or uses a different case structure, can be marked 0. The 50 correct calibration proofs were generated by the solver and human-reviewed, so they likely represent a narrow structural family. Nothing in the paper shows the verifier accepts diverse correct proofs. Until that is demonstrated, the >90% figure is a similarity score, not a correctness score, and Tables 1 and 2 are uncalibrated. The cross-model selection rule is also tuned on the benchmark itself, though the reported threshold insensitivity softens that concern.\n\nThat said, the paper is not a mess. The benchmark design is sound and the verifier issue is precisely the kind of thing peer review should catch. I would send it to review with the requirement that the authors add a held-out verifier evaluation, ideally including correct proofs deliberately written in different structural styles, and report agreement with human judgments as a function of structural similarity. If the verifier holds up, this becomes a standard reference. If it doesn't, the task set is still useful but the scores are not.\n\nWorth bringing to reading group, but not something I'd cite for its numbers until the verifier question is resolved.","headline":"The benchmark design is genuinely useful, but the reported scores rest on a verifier that is calibrated in-sample and anchored to the ground-truth proof; fix that and this becomes a standard resource.","tokens_in":21402,"tokens_out":3838,"would_cite":true,"duration_ms":35831,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TCS-Bench measures whether LLMs can prove 300 research-level theorems from top TCS venues, and the paper reports a verifier that matches human experts on over 90 percent of a labeled proof set.","keywords":["TCS-Bench","theorem proving","large language models","proof verification","theoretical computer science","benchmark design","dependency graph","LLM evaluation"],"falsifier":"Take 100 model-generated proofs that were never used in the verifier-prompt optimization, have human experts label each proof correct or incorrect, and run the reference verifier on the same proofs; if its agreement falls materially below the reported 90 percent, the benchmark's model scores are not calibrated measurements.","tokens_in":20427,"feed_emoji":"📊","tokens_out":10544,"duration_ms":90728,"temperature":0.7,"pith_summary":"TCS-Bench is a benchmark built from 300 theorem-proving tasks extracted from papers at STOC, FOCS, and SODA, where each task gives a model a self-contained context of definitions and prior lemmas and asks it to prove a target result whose original proof is withheld. The paper's central claim is that an automated verification agent can grade the resulting proofs reliably: it reports over 90 percent agreement with human expert judgments on a 100-proof labeled set, which is what makes the benchmark's model scores meaningful. With that verifier, reported accuracy ranges from about 30 percent to 68 percent across public frontier models, with an agentic harness that combines two models via cross-model selection reaching 67.7 percent. If the verifier accuracy holds, TCS-Bench offers a repeatable, contamination-resistant way to track progress in research-level mathematical reasoning.","feed_headline":"Top model proves 68% of research TCS theorems in new test","feed_subtitle":"An automated verifier matches human experts on over 90 percent of labeled proofs, so the 300-task scores can be trusted.","key_machinery":"The central machinery is the dependency-graph pipeline. A deterministic parser identifies all theorem-like environments in a paper's LaTeX source, an LLM pass maps proofs to statements and draws dependency edges, acyclicity is checked programmatically, and each statement receives a rank equal to the longest directed path ending at it; task contexts hide all statements of rank at least the target's rank, and optional masking of intermediate lemmas produces a difficulty curriculum. The evaluation machinery is the verifier: four independent calls to a cheap language model, each given the context, target statement, candidate proof, and ground-truth proof, with a majority of three or more 'correct' verdicts producing a pass. Its prompt was tuned on a 100-proof human-labeled set through a prompt-evolution pipeline, and the reported result is over 90 percent agreement with human experts.","core_discovery":"The paper introduces TCS-Bench and claims that the difficulty of real theoretical-computer-science research can be measured. Each task is generated from the LaTeX source of a published paper by extracting the dependency graph of its statements and hiding all statements of rank at least the target's rank, so the solver must reconstruct the proof from provided definitions and intermediate lemmas, with more intermediate results masked to create harder variants. The load-bearing discovery is the automated proof verifier: a cheap language model is called four times on the context, target statement, candidate proof, and ground-truth proof, and the candidate passes if at least three calls mark it correct. The paper reports that this verifier matches human expert labels on over 90 percent of the 100-proof calibration set, and on that basis presents the benchmark's model accuracy scores as trustworthy measurements.","pith_inferences":["If the verifier's accuracy generalizes beyond the 100-proof calibration set, the same dependency-graph recipe could produce proof-generation benchmarks in any mathematical field with available LaTeX sources, since nothing in the construction is specific to theoretical computer science.","Because the verifier is given the ground-truth proof as an input, its judgments may be biased toward proofs that resemble that reference; a held-out evaluation with correct proofs that are structurally different would settle whether the 90 percent figure transfers.","The cross-model selection finding suggests a general recipe for improving proof reliability without retraining: pair a generator with an independent critic and trust the critic when the generator's self-verification accepts too easily; whether the reported asymmetry holds for other model pairs is an open question.","The rank-based difficulty stratification offers a ready test: mask progressively more intermediate lemmas and measure whether model accuracy degrades smoothly, which would quantify how much proof scaffolding a model can reconstruct from context."],"forward_implications":["If the verifier accuracy holds, the reported model scores become comparable, reproducible measurements of research-level proof-generation ability rather than informal judgments.","The difficulty curriculum from dependency masking ensures the benchmark is informative across the full capability spectrum instead of saturating at easy or impossible tasks.","Because tasks are drawn from papers that can postdate training cutoffs and can be added continuously, the benchmark can be refreshed in a way that resists contamination from memorized training data.","The cross-model selection results show that routing between two independent proof-producing runs, using a second model's critiques, lifts accuracy from 54.0 percent to 67.7 percent, a gain the paper attributes to self-rejection being informative while self-acceptance is not.","The 32-point gap between the strongest model (68 percent) and perfect performance (100 percent) is presented as the current frontier of context-dependent automated mathematical reasoning."],"supporting_citations":[{"why":"Supplies the prompt-evolution pipeline used to tune the verifier prompt to the reported over-90-percent accuracy.","marker":"[3]"},{"why":"Defines the research-replication benchmark paradigm that TCS-Bench adapts from empirical replication to theoretical proof generation.","marker":"[28]"},{"why":"Provides the comparison point of a generator-verifier gap with automatic verification for research-level mathematics.","marker":"[32]"},{"why":"Establishes the frontier capability level in formal theorem proving that motivates the need for harder research-level benchmarks.","marker":"[8]"},{"why":"Documents AI systems contributing to open research problems, the capability that TCS-Bench aims to measure.","marker":"[11]"}],"fun_headline_variants":["New benchmark tests AI proofs of research theorems","Verifier matches experts on 90% of AI proofs","AI theorem provers scored on 300 TCS problems","Proof verifier hits 90% agreement with humans","New benchmark scores AI on 300 TCS proof tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the verifier's over-90-percent agreement with human experts, measured on a 100-proof calibration set whose prompt was tuned by an automated prompt optimizer, transfers to all 300 tasks and to every proof a model produces—including correct proofs that are structured very differently from the ground-truth proof the verifier receives as a reference.","fun_headline_variants_meta":{"raw":{"variants":["New benchmark tests AI proofs of research theorems","Verifier matches experts on 90% of AI proofs","AI theorem provers scored on 300 TCS problems","Proof verifier hits 90% agreement with humans","New benchmark scores AI on 300 TCS proof tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000547,"raw_usage":{"total_tokens":2559,"prompt_tokens":835,"completion_tokens":1724,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":451,"completion_tokens_details":{"reasoning_tokens":1647}},"tokens_in":451,"tokens_out":1724,"duration_ms":12871,"temperature":1.0,"reasoning_tokens":1647,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:13:57.714125+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take 100 model-generated proofs that were never used in the verifier-prompt optimization, have human experts label each proof correct or incorrect, and run the reference verifier on the same proofs; if its agreement falls materially below the reported 90 percent, the benchmark's model scores are not calibrated measurements.","supporting_citations":[],"review_version":2}