Pith. sign in

REVIEW 4 major objections 5 minor 32 references

FormulaOne: Measuring the Depth of Algorithmic Reasoning Beyond Competitive Programming

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FormulaOne, a benchmark built from Monadic Second-Order logic on graphs, shows frontier reasoning models solve under 1 percent of its hard algorithmic problems, indicating a reasoning-depth gap that competitive-programming scores do not…

desk verdict A genuinely new, verifiable benchmark with a credible <1% model failure rate; the missing human baseline tempers the 'expert-level reasoning' interpretation but doesn't sink the main result. read the letter →

arxiv 2507.13337 v1 pith:YRAGE2BY submitted 2025-07-17 cs.AI cs.CCmath.LO

Gal Beniamini , Yuval Dor , Alon Vinnikov , Shir Granot Peled , Or Weinstein , Or Sharir , Noam Wies , Tomer Nussbaum
show 5 more authors
Ido Ben Shaul Tomer Zekharya Yoav Levine Shai Shalev-Shwartz Amnon Shashua
This is my paper · ORCID
classification cs.AIcs.CCmath.LO
keywords MSOlogicongraphstreedecompositionsdynamicprogrammingalgorithmicreasoningbenchmarkweightedmodelcountingfrontierAIevaluationtreewidthRLVRenvironment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces FormulaOne, a benchmark of 120 graph-algorithm problems designed to measure the depth of algorithmic reasoning rather than breadth of knowledge. The problems are all instances of weighted model counting for properties expressible in Monadic Second-Order (MSO) logic on graphs with small treewidth, and each is guaranteed by an algorithmic meta-theorem to have an efficient dynamic-programming solution. The authors report that frontier reasoning models, given substantial scaffolding and multiple sampled completions, solve fewer than one percent of these problems, while a curated easier companion set, FormulaOne-Warmup, is handled substantially better. The intended conclusion is that current models can imitate competitive programming but cannot yet perform the long, interdependent chains of reasoning that real research problems require.

What carries the argument

The carrying object is the pairing of MSO logic with tree-decomposition dynamic programming. MSO logic is a formal language for graph properties in which one may quantify over vertices, edges, and sets of them; an algorithmic meta-theorem guarantees that any MSO-definable problem can be solved in time linear in the graph order once a bounded-width tree decomposition is supplied. The paper's domain-specific language converts MSO formulas into human-readable problem statements, verifying programs, test suites, and state-complexity estimates, and the evaluation interface asks models to supply only the five dynamic-programming transition callbacks. This machinery makes the benchmark simultaneously hard, automatically verifiable, scalable, and in-distribution, which is why the near-total model failure is taken as evidence about reasoning depth rather than about unfamiliar formats.

What would settle it

Run the same 120 problems with a debugging loop in which the model can execute its code on sample tests and revise its five callback implementations before final scoring; if success rises substantially, the under-one-percent result is an artifact of one-shot code generation rather than a reasoning limit. Alternatively, administer the identical benchmark to a panel of top human competitive programmers; if they also solve near 1 percent, the human-expert comparison collapses.

Watch

Extended reading notes

Core claim

The central claim is that a benchmark generated from MSO logic on graphs can separate frontier models' competitive-programming competence from expert-level algorithmic reasoning, and that under this separation current models fall far short. Each FormulaOne problem presents a graph, a tree decomposition, and vertex weights, and asks for the sum of weights of all subsets satisfying an MSO-definable constraint, modulo $10^{9}$+7. Because constraints are MSO-definable, every problem is solvable by a fixed-parameter linear dynamic program running over the supplied tree decomposition; the evaluation harness reduces the model's task to implementing five callbacks for leaf, introduce, forget, join, and root extraction. The paper reports that frontier reasoning models solve at most one of the 120 hard problems even with multiple attempts, and argues this failure is fundamental rather than a prompt-engineering artifact. The dataset also connects several of its hardest problems to the Strong Exponential Time Hypothesis, so genuinely better algorithms discovered on it would have theoretical consequences.

Load-bearing premise

The load-bearing premise is that the callback-based interface is a fair test of reasoning depth and that expert human programmers would score far above 1 percent on it; if either premise fails, the low success rate would not establish a reasoning-depth gap.

Editorial extensions

If this is right

  • If the under-one-percent result holds, current competitive-programming benchmark scores overstate how close frontier models are to expert algorithmic reasoning, because FormulaOne is deliberately in-distribution.
  • The MSO-based generation framework can produce a virtually unlimited stream of new problems with guaranteed correct answers, making FormulaOne a candidate training environment for reinforcement learning with verifiable rewards.
  • A model that solves a substantial share of FormulaOne would demonstrate a capability current systems lack, and one that improves on the known algorithm for a SETH-linked problem would carry consequences for complexity theory.
  • FormulaOne-Warmup provides a graded ladder of same-distribution problems, allowing incremental measurement of progress before models can attack the hard set.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An obvious test the paper does not run is to let models debug against failing test cases: the reported result is for multiple independent completions with no feedback loop, so part of the gap may reflect inability to find and fix implementation bugs rather than inability to design the dynamic program.
  • The human-expert baseline is asserted, not measured; a controlled study giving top human competitive programmers the same five-callback interface would sharpen the claim that the benchmark tracks reasoning depth.
  • The connection between reasoning depth and model failure could be tested directly by generating formulas with graded numbers of interdependent state features and checking whether model success decays smoothly with that measure.
  • Extending the framework to optimisation objectives, different graph parameters, or requiring models to produce the tree decomposition themselves would likely lower success further, but would also change what the benchmark isolates.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces FormulaOne, a benchmark of 120 problems composed of dynamic programming over graphs of treewidth at most 3, generated from Monadic Second-Order (MSO) logic formulas. Each problem asks the model to compute a weighted model count modulo 1e9+7, given the graph, a tree decomposition, and vertex weights. To focus the task, the authors provide an evaluation environment with five callback functions (leaf, introduce, forget, join, extract), which hides the full graph during DP updates. They evaluate o3, o3-Pro, Gemini 2.5 Pro, and Grok 4 Heavy and report success rates below 1% (0 or 1 of 120), plus a 100-problem warmup set with substantially higher success rates. The paper argues that this demonstrates that frontier models lack the deep algorithmic reasoning needed for real-world research problems and claims connections to SETH-based lower bounds. The authors state they release the full corpus and evaluation framework.

Significance. If the results are properly calibrated, FormulaOne would be a valuable benchmark: it is generated from MSO logic with automated verifiers, it is in-distribution for language models in the sense of being an algorithmic coding task, and it offers a potentially unbounded stream of verifiable hard problems for reinforcement learning with verifiable rewards. The paper's strengths include a carefully described evaluation environment, a thoughtful analysis of common DP failure modes, a detailed worked solution for one problem in Appendix A, and a warmup dataset that spans a difficulty gradient. However, the central interpretation rests on two unverified premises: that expert humans would score far above 1% on the same interface, and that the self-generated DSL verifiers are correct. Both are load-bearing for the claim that the benchmark measures the depth of algorithmic reasoning rather than interface unfamiliarity. Until these are addressed, the benchmark's significance as a measure of reasoning depth remains conditional.

major comments (4)
  1. [Section 2.1 and Section 5.2] The headline claim that frontier models are 'far from expert-level understanding' depends on the premise, stated in Section 2.1, that a human expert of Codeforces-calibre should 'by rights' score highly on these problems. No human baseline is reported anywhere in the manuscript. The one fully worked solution in Appendix A shows that at least one problem is humanly solvable, but it does not calibrate the other 119 problems. Since the evaluation interface requires implementing five callbacks that operate only on the current bag of the tree decomposition, hides the full graph during DP updates, and does not allow iterative debugging, the <1% success rate could reflect interface unfamiliarity rather than reasoning depth. The authors should add a human baseline using the same interface and time limits, or substantially weaken the expert-level interpretation.
  2. [Section 4.2 and Section 2.2] The ground truth for the benchmark is generated by the authors' own DSL and verifiers, with no described independent verification. The paper states that the verifier maps any input to the correct answer, but the correctness of those verifiers is itself asserted by the same pipeline. I found no independent brute-force cross-checks, no differential testing against known algorithms, and no manual verification of a random sample. The abstract and introduction say the corpus and evaluation framework are released, but no URL, repository, or artifact is provided in the manuscript. Without code and data, the reported success rates cannot be audited or reproduced. Please release the artifacts and add independent validation (e.g., brute-force checks on small graphs with an independently written verifier, or a manual verification protocol).
  3. [Section 2.1 and Section 3] The paper claims that 'the best-known time complexity of a large portion of our dataset is, in fact, optimal under SETH' and that algorithmic progress on these problems could refute a central conjecture in theoretical computer science. I could not find a concrete identification of which problems in the dataset have this property, nor citations of the specific lower bounds. This is a strong theoretical claim that goes beyond the benchmark's empirical findings and, as written, overstates the significance. The authors should either specify the dataset problems and cite the relevant SETH lower-bound results, or replace the claim with a more modest statement about conceptual proximity to fine-grained complexity results.
  4. [Section 5.2 and Table/Figure 8] The model comparison is not apples-to-apples: o3 and Gemini 2.5 Pro are scored with 10 attempts (@10), while o3-Pro and Grok 4 Heavy are scored with one attempt (@1), and Grok was additionally given internet access. With 120 problems, the difference between 0 and 1 solved problems is not statistically meaningful, yet the text describes Grok as 'solving none' in a way that implies a ranking. Please report per-attempt success rates, confidence intervals, and use matched evaluation budgets when comparing models.
minor comments (5)
  1. [Section 5.1 and Section 3.2] The prompt is said to 'be found in full here' and the dominating-set solution is said to be 'written in Python, here', but no links or appendix content are present in the manuscript. Please include the actual artifacts or remove the pointers.
  2. [Appendix A] The appendix is titled 'Algorithm for Maximal Cluster Graph' and is described as a full solution, but it contains no runnable code and no formal proof of correctness for the DP transitions, such as the obligation-bit merge rule at join nodes. Consider adding a complete implementation and a correctness argument.
  3. [Figure 10] The category-level success rates are computed over the combined FormulaOne and FormulaOne-Warmup datasets, but some categories have very small sample sizes (e.g., EPSILON with 4 problems and MODULAR with 10); the displayed rates are therefore noisy and should be interpreted with caution.
  4. [Section 4.3] The claim that the sampling process ensures that 'with high probability' all possible gadgets are present in the tested graphs is not quantified. Please state what 'all possible gadgets' means and give the probability or a formal argument.
  5. [Abstract and Introduction] The abstract says 'We release the full corpus along with a comprehensive evaluation framework', while the introduction says 'we will release the full dataset'; please make the release status consistent and, if released, provide a permanent link.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the <1% model results are an independent empirical measurement; the missing human baseline is a validity concern, not a circular reduction.

full rationale

The paper's central result is an empirical evaluation, not a derivation that reduces to its inputs. Models are asked to implement five DP callback functions; their completions are run in a purpose-built evaluation harness against independently generated verifiers and test suites, and the reported <1% success rate is a measured outcome. The generation of problems from MSO logic and the guarantee of tractability rest on Courcelle's theorem, an external mathematical result, and the verifiers are constructed from the authors' DSL rather than from the models' outputs. No parameter is fitted to the measured success rates, no prediction is a renamed fit, and no load-bearing self-citation appears in the argument: the cited complexity-theory literature is external and the SETH connection is contextual framing rather than an input to the evaluation. The main interpretive risk, noted in Section 2.1, is that the claim that expert humans 'should, by rights, be able to score highly' is asserted without a human baseline; however, this is an unsupported empirical premise about interpretability, not a circular step in the derivation chain. The Appendix's full worked solution is illustrative, not used to calibrate or fit the benchmark outcomes. Therefore the core measurement is self-contained and no circularity is present.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The benchmark construction introduces no fitted constants; the listed free parameters are hand-chosen design and evaluation choices. The central claim rests on standard theorems (Courcelle, clique-in-bag) and on the unproven correctness of the DSL-generated verifiers, which is the most consequential premise. No new physical or postulated entities appear.

free parameters (5)
  • Core dataset size = 120 problems
    The size is hand-chosen and sets the statistical resolution of the core result, but it is not fitted to any target.
  • Warmup dataset size = 100 problems
    The size is hand-chosen to provide an easier auxiliary set from the same distribution.
  • Treewidth bound = 3 (with JOIN bags of width at most 2)
    This design choice controls tractability and difficulty, but it is not derived from data.
  • Time limit per test = 100 seconds
    The limit is set at roughly 100 times the expected runtime, which is a hand-chosen trade-off between correctness and efficiency.
  • Sampling attempts = 10 for o3 and Gemini, 1 for o3-Pro and Grok 4
    The number of attempts per problem is a protocol choice that changes the probability of at least one success.
assumptions (5)
  • standard math Courcelle's theorem: every MSO-definable graph property is solvable by linear-time DP on bounded-treewidth graphs.
    This theorem underwrites the claim that every FormulaOne problem has an efficient solution; it is cited as [Cou90] in Section 1.
  • domain assumption The provided tree decomposition is valid and can be treated as part of the input.
    The evaluation framework supplies the tree decomposition and restricts the model to local bag views, which simplifies the task substantially; Section 4.1.
  • domain assumption The DSL-generated verifiers correctly compute the weighted model count for each MSO formula.
    The test suite is generated from verifiers built by the DSL; their correctness is asserted but not formally proven in the preprint; Section 4.2.
  • domain assumption The stochastic graph sampling covers all relevant small gadgets in all bag-relative positions.
    The test coverage argument depends on this high-probability property; Section 4.3.
  • standard math Every clique of the graph appears inside some bag of the tree decomposition.
    This standard fact is used in Section 3.5 to reduce bipartiteness of a cograph to per-bag triangle checks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FormulaOne: Measuring the Depth of Algorithmic Reasoning Beyond Competitive Programming." pith.science (2026). https://pith.science/paper/YRAGE2BY

@misc{pith2026250713337,
  author       = {Pith},
  title        = {Pith review of: FormulaOne: Measuring the Depth of Algorithmic Reasoning Beyond Competitive Programming},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YRAGE2BY}},
  note         = {Machine review of arXiv:2507.13337}
}
read the original abstract

Frontier AI models demonstrate formidable breadth of knowledge. But how close are they to true human -- or superhuman -- expertise? Genuine experts can tackle the hardest problems and push the boundaries of scientific understanding. To illuminate the limits of frontier model capabilities, we turn away from contrived competitive programming puzzles, and instead focus on real-life research problems. We construct FormulaOne, a benchmark that lies at the intersection of graph theory, logic, and algorithms, all well within the training distribution of frontier models. Our problems are incredibly demanding, requiring an array of reasoning steps. The dataset has three key properties. First, it is of commercial interest and relates to practical large-scale optimisation problems, such as those arising in routing, scheduling, and network design. Second, it is generated from the highly expressive framework of Monadic Second-Order (MSO) logic on graphs, paving the way toward automatic problem generation at scale; ideal for building RL environments. Third, many of our problems are intimately related to the frontier of theoretical computer science, and to central conjectures therein, such as the Strong Exponential Time Hypothesis (SETH). As such, any significant algorithmic progress on our dataset, beyond known results, could carry profound theoretical implications. Remarkably, state-of-the-art models like OpenAI's o3 fail entirely on FormulaOne, solving less than 1% of the questions, even when given 10 attempts and explanatory fewshot examples -- highlighting how far they remain from expert-level understanding in some domains. To support further research, we additionally curate FormulaOne-Warmup, offering a set of simpler tasks, from the same distribution. We release the full corpus along with a comprehensive evaluation framework.

Figures

Figures reproduced from arXiv: 2507.13337 by the authors.

Figure 1
Figure 1. Performance of frontier reasoning models on the FormulaOne dataset. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Three problems, expressible by MSO logic, solvable by dynamic programming on tree-like graphs. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Local modifications to bags (dashed boxes). On the left, a vertex (green) is added (“introduced”), [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Snapshot of a dynamic programming algorithm, traversing over a tree decomposition. The current [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Partial solutions as viewed through a bag, and the corresponding DP state. Vertices in the [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: A bag and its state. Vertices in the set [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: A few different bags of cardinality 2, encircled in blue, that are possible when traversing over [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Performance of top frontier models on the FormulaOne dataset. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Performance of top frontier models on the FormulaOne-Warmup dataset. There are 100 problems [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Model Performance by problem label on the [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 25 canonical work pages

  1. [1]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 , 2021

    arXiv 2021

  2. [2]

    On non-serial dynamic programming

    Umberto Bertele and Francesco Brioschi. On non-serial dynamic programming. J. Comb. Theory, Ser. A , 14(2):137--148, 1973

    work page 1973

  3. [3]

    Rankings of graphs

    Hans L Bodlaender, Jitender S Deogun, Klaus Jansen, Ton Kloks, Dieter Kratsch, Haiko M \"u ller, and Zsolt Tuza. Rankings of graphs. SIAM Journal on Discrete Mathematics , 11(1):168--181, 1998

    work page 1998

  4. [4]

    Dynamic programming on graphs with bounded treewidth

    Hans L Bodlaender. Dynamic programming on graphs with bounded treewidth. In Automata, Languages and Programming: 15th International Colloquium Tampere, Finland, July 11--15, 1988 Proceedings 15 , pages 105--118. Springer, 1988

    1988

  5. [5]

    A linear time algorithm for finding tree-decompositions of small treewidth

    Hans L Bodlaender. A linear time algorithm for finding tree-decompositions of small treewidth. In Proceedings of the twenty-fifth annual ACM symposium on Theory of computing , pages 226--234, 1993

    1993

  6. [6]

    Handle-rewriting hypergraph grammars

    Bruno Courcelle, Joost Engelfriet, and Grzegorz Rozenberg. Handle-rewriting hypergraph grammars. Journal of computer and system sciences , 46(2):218--270, 1993

    work page 1993

  7. [7]

    Lower bounds based on the exponential-time hypothesis

    Marek Cygan, Fedor V Fomin, ukasz Kowalik, Daniel Lokshtanov, D \'a niel Marx, Marcin Pilipczuk, Micha Pilipczuk, Saket Saurabh, Marek Cygan, Fedor V Fomin, et al. Lower bounds based on the exponential-time hypothesis. Parameterized Algorithms , pages 467--521, 2015

    work page 2015

  8. [8]

    On the measure of intelligence

    Fran c ois Chollet. On the measure of intelligence. 2019

    work page 2019

Show all 32 references
  1. [9]

    The complexity of satisfiability of small depth circuits

    Chris Calabro, Russell Impagliazzo, and Ramamohan Paturi. The complexity of satisfiability of small depth circuits. In International Workshop on Parameterized and Exact Computation , pages 75--85. Springer, 2009

    2009

  2. [10]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. 2021

    2021

  3. [11]

    Arc-agi-2: A new challenge for frontier ai reasoning systems

    Francois Chollet, Mike Knoop, Gregory Kamradt, Bryan Landers, and Henry Pinkard. Arc-agi-2: A new challenge for frontier ai reasoning systems. arXiv preprint arXiv:2505.11831 , 2025

    2025 arXiv

  4. [12]

    Arc prize 2024: Technical report

    Francois Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers. Arc prize 2024: Technical report. arXiv preprint arXiv:2412.04604 , 2024

    2024 arXiv

  5. [13]

    The monadic second-order logic of graphs

    Bruno Courcelle. The monadic second-order logic of graphs. i. recognizable sets of finite graphs. Information and computation , 85(1):12--75, 1990

    1990

  6. [14]

    Competitive programming with large reasoning models

    Ahmed ElKishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, et al. Competitive programming with large reasoning models. 2025

    2025

  7. [15]

    Ruiz, Julian Schrittwieser, Grzegorz Swirszcz, et al

    Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Francisco J.R. Ruiz, Julian Schrittwieser, Grzegorz Swirszcz, et al. Discovering faster matrix multiplication algorithms with reinforcement learning. ...

    2022

  8. [16]

    Dynasp2.5: Dynamic programming on tree decompositions in action

    Johannes K Fichte, Markus Hecher, Michael Morak, and Stefan Woltran. Dynasp2.5: Dynamic programming on tree decompositions in action. Algorithms , 14(3):81, 2021

    2021

  9. [17]

    Faster algorithms for finding and counting subgraphs

    Fedor V Fomin, Daniel Lokshtanov, Venkatesh Raman, Saket Saurabh, and BV Raghavendra Rao. Faster algorithms for finding and counting subgraphs. Journal of Computer and System Sciences , 78(3):698--706, 2012

    2012

  10. [18]

    Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai

    Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, et al. Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai. arXiv preprint arXiv:...

    2024 arXiv

  11. [19]

    S-functions for graphs

    Rudolf Halin. S-functions for graphs. Journal of geometry , 8:171--186, 1976

    1976

  12. [20]

    F. Harary. Graph Theory . Addison-Wesley, Reading, MA, 1969

    1969

  13. [21]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. 2021

    2021

  14. [22]

    Ale-bench: A benchmark for long-horizon objective-driven algorithm engineering

    Yuki Imajuku, Kohki Horie, Yoichi Iwata, Kensho Aoki, Naohiro Takahashi, and Takuya Akiba. Ale-bench: A benchmark for long-horizon objective-driven algorithm engineering. 2025

    2025

  15. [23]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. 2024

    2024

  16. [24]

    Enumeration of minimal hitting sets parameterized by treewidth

    Batya Kenig and Dan Shlomo Mizrahi. Enumeration of minimal hitting sets parameterized by treewidth. 2024

    2024

  17. [25]

    Known algorithms on graphs of bounded treewidth are probably optimal

    Daniel Lokshtanov, D \'a niel Marx, and Saket Saurabh. Known algorithms on graphs of bounded treewidth are probably optimal. In Proceedings of the twenty-second annual ACM-SIAM symposium on Discrete Algorithms , pages 777--789. SIAM, 2011

    2011

  18. [26]

    Mankowitz, Andrea Michi, Anton Zhernov, Marco Gelmi, Marco Selvi, Cosmin Paduraru, Edouard Leurent, Shariq Iqbal, Jean-Baptiste Lespiau, Alex Ahern, et al

    Daniel J. Mankowitz, Andrea Michi, Anton Zhernov, Marco Gelmi, Marco Selvi, Cosmin Paduraru, Edouard Leurent, Shariq Iqbal, Jean-Baptiste Lespiau, Alex Ahern, et al. Faster sorting algorithms discovered using deep reinforcement learning. Nature , 618:257--263, 2023

    2023

  19. [27]

    Ruiz, Abbas Mehrabian, M

    Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco J.R. Ruiz, Abbas Mehrabian, M. Pawan Kumar, Abigail See, Swarat Chaudhuri, George Holland, Alex Davies, Sebastian Nowozin, Pushmee...

    2025

  20. [28]

    Humanity's last exam

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Hugh Zhang, Dan Hendrycks, Alexandr Wang, et al. Humanity's last exam. 2025

    2025

  21. [29]

    Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings

    Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, et al. Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings. 2025

    2025

  22. [30]

    Graph minors

    Neil Robertson and Paul D Seymour. Graph minors. i. excluding a forest. Journal of Combinatorial Theory, Series B , 35(1):39--61, 1983

    1983

  23. [31]

    Graph minors

    Neil Robertson and Paul D Seymour. Graph minors. iii. planar tree-width. Journal of Combinatorial Theory, Series B , 36(1):49--64, 1984

    1984

  24. [32]

    k-nlc graphs and polynomial algorithms

    Egon Wanke. k-nlc graphs and polynomial algorithms. Discrete Applied Mathematics , 54(2-3):251--266, 1994

    1994

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.