Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Proceedings of the 2024 XCSP3 Competition

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The 2024 XCSP3 competition ranks Picat first in the CSP and Fast COP tracks, CPMpy ortools first in the COP and parallel COP tracks, Exchequer first in Mini COP, and CPMpy chuffed first in Mini CSP.

desk verdict A solid annual benchmark record with credible rankings, but the printed models have errors and the instance counts don't add up, so the archive is the only ground truth. read the letter →

arxiv 2412.00117 v1 pith:J7RCLBHZ submitted 2024-11-28 cs.AI

Gilles Audemard , Christophe Lecoutre , Emmanuel Lonca This is my paper · ORCID
classification cs.AI
keywords XCSP3competitionconstraintprogrammingsatisfactionoptimizationXCSP3-corePyCSP3solverbenchmarkingranking
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

These proceedings record the official outcome of the 2024 XCSP3 competition, an annual benchmark in which constraint solvers run on a newly selected set of problems. The organizers chose 34 problems, restricted the input language to the XCSP3-core kernel, and generated 200 CSP plus 250 COP instances for the standard tracks and 150 each for the mini tracks. The claimed result is the ranking: Picat first in the CSP and Fast COP tracks, CPMpy ortools first in the COP and parallel COP tracks, Exchequer first in the Mini COP track, and CPMpy chuffed first in the Mini CSP track. If the ranking is correct, it identifies which solver approaches currently perform best on this benchmark suite under fixed time limits and a published scoring rule.

What carries the argument

The load-bearing objects are the 2024 instance suite and the scoring protocol. XCSP3-core is the restricted input format used throughout: integer variables with a kernel of 24 popular global constraints such as allDifferent, sum, table, cumulative, noOverlap, and binPacking. The 34 selected problems, almost all new, are modelled in the Python library PyCSP3 and generated as series of increasing difficulty, giving 200 CSP and 250 COP instances for the standard tracks and 150 each for the mini tracks. Solvers run on identical cluster nodes with time limits enforced by runsolver, and the ranking algorithm discards off-competition entries, removes less efficient variants of the same solver, and applies the published point rules. This combination of fresh problem selection and fixed protocol gives the rankings their content.

What would settle it

Re-run the ranked solver configurations on the archived instance files under the stated CPU time limits and apply the published scoring rule; if the top finishers in any track change, the reported rankings do not reproduce.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a measured performance ranking rather than a theoretical theorem. The official 2024 XCSP3 competition results put Picat at the top of the CSP and Fast COP tracks, CPMpy ortools at the top of the COP and parallel COP tracks, Exchequer at the top of the Mini COP track, and CPMpy chuffed at the top of the Mini CSP track, with Fun-sCOP, CoSoCo, Choco, toulbar2, miniBTD, Nacre, miniRBO, and CPMpy gurobi also in the top places. The ranking was produced by counting solved instances for satisfaction problems and by awarding points for best-known bounds, with full credit only when optimality is proved, under time limits of 30 CPU-minutes per instance, 3 minutes in the Fast COP track, and 4 CPUs in the parallel track. Because the instance selector was also the author of one solver, that solver was run but kept off the official ranking. The paper asserts these lists accurately reflect solver performance on the 2024 instance suite.

Load-bearing premise

The rankings are valid only if the archived instance files match the models printed in these proceedings and every solver output was checked against the correct instance; the proceedings themselves warn that the printed models and the archive may differ.

Editorial extensions

If this is right

  • If the rankings hold, Picat's first places show that a SAT-encoding route to XCSP3-core is the strongest submitted approach for satisfaction and for fast optimization on this suite.
  • If the rankings hold, CPMpy ortools' wins in the COP and parallel COP tracks show that a lazy-clause-generation solver with parallel search leads on the 250 optimization instances.
  • If the rankings hold, Exchequer's Mini COP win shows that translating XCSP3 into a C program and using a bounded model checker can be competitive in a mini track.
  • Because the COP scoring gives full credit only when optimality is proved, the optimization podium measures proof capability, not merely the ability to find good solutions.
  • The reported results are tied to the exact 2024 instance archive; the proceedings' own warning means the printed models should not be assumed to reproduce the runs byte for byte.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Had the selector's solver been allowed to compete, the standings might shift, because the paper reports that its mixed-heuristic variant scores better than its default configuration in several tracks while being excluded from the ranking.
  • A testable extension of this work would be to reselect the instance suite with a different selector and re-run the same solver configurations, which would reveal how much of the ranking reflects selection bias.
  • If the warned mismatch between the printed models and the archive affects any problem, then the models in this document should be treated as documentation and only the archive used for exact reproduction.
  • The half-point rule for shared, unproven best bounds means that a solver which finds good solutions quickly but rarely proves optimality can rank below a slower solver that closes the proof; computing rankings under alternative scoring would show how much this design choice matters.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript documents the 2024 XCSP3 constraint-solver competition: it describes the 34 selected problems, gives PyCSP3 models for the instance generators, presents the participating solvers and their configurations, states the hardware and timeout settings and the scoring rules, and lists the official ranking for each of the six tracks. The central claimed result is that the listed rankings, e.g., Picat first in CSP and Fast COP, CPMpy ortools first in COP and parallel COP, and Exchequer first in Mini COP, reflect the solvers' actual performance on the 2024 instance suite under the stated conditions. The paper points to an external archive and website for the exact instances and detailed solver traces.

Significance. If the rankings are accurate, the document is a useful record of the relative performance of a broad set of CP, SAT-based, MIP, and SMT technologies on a fresh, publicly available benchmark suite. The manuscript has real strengths: it states the scoring rule, reports hardware and timeout information, discloses that the instance-selection organizer's solver ACE was run off-competition, and defers to an archive and a website with per-instance traces. These features make the competition more auditable than many solver competitions. The value of the proceedings, however, depends on whether the benchmark described inside the document is the benchmark that was actually run; several internal inconsistencies currently prevent the reader from checking that identity from the manuscript alone.

major comments (4)
  1. [Section 4.1 (Tables 4.1–4.2)] The timeout information is inconsistent. Table 4.1 lists 40 minutes for CSP, COP, and parallel COP and 4 minutes for Fast COP, and Table 4.2 lists 40 minutes for both mini tracks, but the bullets immediately below state that fast-COP sequential solvers were allocated 3 minutes of CPU time and 4.5 minutes of wall-clock time, and that all other sequential solvers were allocated 30 minutes of CPU time and 45 minutes of wall-clock time. The paper must state which limits were actually enforced, because the rankings are only meaningful relative to the true per-track timeouts.
  2. [Section 4.1 vs Sections 2.1–2.2] The stated totals of 200 CSP instances and 250 COP instances do not match the per-series counts given in the problem sections. Summing the stated series sizes in Sections 2.1.1–2.1.16 gives 195 CSP instances, and summing the COP series in Sections 2.2.1–2.2.18 gives 254 COP instances (counting the BinPacking series as 6 + 3 + 6). The totals or the per-problem series counts must be corrected, and they must be made consistent with the archived instance set, since the rankings are defined over those instances.
  3. [Section 4.1, hardware paragraph] The hardware description is internally contradictory: the cluster nodes are said to have 32 GiB of RAM (24 GiB for jobs), but each solver is said to have been allocated 64 GiB of RAM independently of the track. A per-solver allocation larger than the node memory is not possible as written. Please clarify the node memory and the actual per-job memory allocation; this is part of the run conditions needed to reproduce the results.
  4. [Sections 2.2.9, 2.2.13, 2.2.15, 2.2.18] Several printed PyCSP3 models are not valid code as shown. MaximumDensityOscillatingLife contains the expression `(k1 k2)` with a missing comma; StillLife has an unmatched parenthesis in the `Sum` call; TravelingTournament assigns `istances = data` but then references `distances`; and Wordpress uses the undefined identifier `lbWP`. The front matter already warns that the printed models may differ from the archived instances, but the scale of these discrepancies means the benchmark cannot be reconstructed from the document alone. Please either replace the faulty listings with versions that compile, or explicitly label them as illustrative and give a precise pointer to the authoritative archive versions, ideally with version or checksum information.
minor comments (5)
  1. [Section 2.1.5] The informal description says that every two vectors must have Hamming distance at most `k`, but the model and the accompanying comment enforce `Hamming(row1, row2) >= k`; the description should be aligned with the intended semantics.
  2. [Sections 2.1.16 and 2.2.2] The file names `WordQuare.py` and `AztecDimaondSym.py` appear to be misspelled for `WordSquare.py` and `AztecDiamondSym.py`; please correct the typos.
  3. [Section 2.2.17] The data paragraph writes `Values of (n, h, s)` but the model description defines the word size as `m`; the notation should be consistent.
  4. [Section 4.1] The validation procedure for solver outputs is only mentioned as "validating solvers and rankings" without describing how solutions or unsat answers were checked; adding one sentence about the checker used (e.g., XCSP3 Tools) would improve reproducibility.
  5. [Section 4.2] The rankings list only the top three solvers per track without the corresponding scores; since the paper relies on external traces for verification, a brief per-track score table or an explicit pointer to where the full score tables are stored would make the proceedings more self-contained.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the rankings are computed from solver executions on the benchmark suite, not derived from the rankings themselves or from fitted parameters.

full rationale

The paper's central claim is an empirical ranking of solvers, obtained by executing the submitted solvers on a fixed set of XCSP3 instances and applying the stated scoring rules. There is no derivation chain in which a predicted quantity is defined in terms of the claimed outcome, no fitted parameter that is later renamed as a prediction, and no uniqueness theorem imported from the authors' prior work to force a conclusion. The acknowledged self-involvement (ACE used to assess instance difficulty and therefore declared off-competition; one organizer's solver CoSoCo appears in a ranking) is a conflict-of-interest and governance matter, not a circular inference: the ranking is still determined by the measured runs. References to earlier XCSP3 format papers and to ACE define the benchmark format and solver capabilities, but they do not substitute for the executed results. The documented reproducibility caveat and the instance-count and model-code discrepancies are real correctness and documentation concerns, but they do not make the ranking its own input. The paper is therefore self-contained with respect to circularity, and the score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new scientific entities or fitted model parameters. The listed free parameters are the procedural choices that define the competition results: the COP scoring rule and the time limits. The axioms are the background assumptions needed for the rankings to be accepted as accurate, chiefly the identity of the benchmark instances, the correctness of validation, and the fairness of the ACE-based selection.

free parameters (2)
  • COP scoring rule = 1 point for a proven optimal solution, 0.5 for a shared best bound without optimality proof, 0 otherwise
    The rankings depend directly on this hand-chosen scoring rule from Section 4.1; changing the partial-credit assignment would change the relative ordering in the COP tracks.
  • Per-track timeout settings = 30 minutes CPU for standard tracks, 3 minutes CPU for fast COP, 40 minutes wall for parallel COP
    These choices define what counts as success in each track and therefore shape the rankings; they are procedural settings rather than fitted scientific parameters.
assumptions (3)
  • domain assumption The archived instance set used in the runs matches the models and data described in the proceedings.
    The front matter states that reproduction requires the very same set of XCSP3 instances from the archive and that the printed models may differ slightly from the archived instances.
  • domain assumption Solver outputs and final rankings were correctly validated by the organizers.
    The front matter credits 'validating solvers and rankings' to C. Lecoutre and E. Lonca; no certificates or machine-checked proofs are provided.
  • domain assumption The ACE-based difficulty assessment used for instance selection does not compromise the fairness of the benchmark for the ranked solvers.
    Section 1 acknowledges that ACE was used to assess instance difficulty and was therefore declared off-competition; the residual bias on the other solvers is not quantified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Proceedings of the 2024 XCSP3 Competition." pith.science (2026). https://pith.science/paper/J7RCLBHZ

@misc{pith2026241200117,
  author       = {Pith},
  title        = {Pith review of: Proceedings of the 2024 XCSP3 Competition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J7RCLBHZ}},
  note         = {Machine review of arXiv:2412.00117}
}
read the original abstract

This document represents the proceedings of the 2024 XCSP3 Competition. The results of this competition of constraint solvers were presented at CP'24 (30th International Conference on Principles and Practice of Constraint Programming).

Figures

Figures reproduced from arXiv: 2412.00117 by the authors.

Figure 1
Figure 1. Illustration of pivotal moments for collecting information about conflicts: this correspond [PITH_FULL_IMAGE:figures/full_fig_p066_1.png] view at source ↗
Figure 1
Figure 1. Overview of CPMpy’s transformation pipeline [PITH_FULL_IMAGE:figures/full_fig_p079_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CP-Model-Zoo: A Natural Language Query System for Constraint Programming Models

    cs.AI 2025-09 conditional novelty 5.0 of 10

    CP-Model-Zoo retrieves the most relevant MiniZinc model from a pool of 67 problems using text embeddings, with mean reciprocal rank between 0.87 and 1.00 in the paper's experiments.

Reference graph

Works this paper leans on

21 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    Proceedings of the 2022 XCSP3 Competition

    G. Audemard, C. Lecoutre, and E. Lonca. Proceedings of the 2022 XCSP3 competition. Technical Report arXiv:2209.00917, CoRR, 2022. https://arxiv.org/abs/2209.00917

    work page Pith review arXiv 2022

  2. [2]

    Audemard, C

    G. Audemard, C. Lecoutre, and E. Lonca. Proceedings of the 2023 XCSP3 competition. Technical Report arXiv:2312.05877, CoRR, 2023. https://arxiv.org/abs/2312.05877

    arXiv 2023

  3. [3]

    Belov, N

    G. Belov, N. Boland, M. Savelsbergh, and P. Stuckey. Local search for a cargo assembly planning problem. In Proceedings of CPAIOR’14, pages 159–175, 2014

    work page 2014

  4. [4]

    Berlekamp, J

    E. Berlekamp, J. Conway, and R. Guy. Winning ways for your mathematical plays, vol. 2: games in particular . Academic Press, 1982

    work page 1982

  5. [5]

    Boussemart, C

    F. Boussemart, C. Lecoutre, G. Audemard, and C. Piette. XCSP3: an integrated format for benchmarking combinatorial constrained problems. Technical Report arXiv:1611.03398, CoRR, 2016. https://arxiv.org/abs/1611.03398

    arXiv 2016

  6. [6]

    Boussemart, C

    F. Boussemart, C. Lecoutre, G. Audemard, and C. Piette. XCSP3-core: A format for repre- senting constraint satisfaction/optimization problems. Technical Report arXiv:2009.00514, CoRR, 2020. https://arxiv.org/abs/2009.00514

    arXiv 2009

  7. [7]

    A. Deza, C. Liu, P. Vaezipoor, and E. Khalil. Fast matrix multiplication without tears: A constraint programming approach. In Proceedings of CP’23, pages 14:1–14:15, 2023

    work page 2023

  8. [8]

    Duque, J.-F

    R. Duque, J.-F. D ´ ıaz, and A. Arbelaez. SABIO: An Implementation of MIP and CP for Interactive Soccer Queries. In Proceedings of CP’16, pages 575–583, 2016

    work page 2016

Show all 21 references
  1. [9]

    Easton, G

    K. Easton, G. Nemhauser, and M. Trick. Solving the travelling tournament problem: A combined integer programming and constraint programming approach. In Proceedings of PATAT’02, pages 100–112, 2002

    2002

  2. [10]

    Erascu, F

    M. Erascu, F. Micota, and D. Zaharie. Scalable optimal deployment in the cloud of component-based applications using optimization modulo theory, mathematical program- ming and symmetry breaking. Journal of Logical and Algebraic Methods in Programming , 121:100664, 2021

    2021

  3. [11]

    I. Gent, C. Jefferson, I. Miguel, and P. Nightingale. Generating special-purpose stateless propagators for arbitrary constraints. In Proceedings of CP’10, pages 206–220, 2010

    2010

  4. [12]

    Jefferson, A

    C. Jefferson, A. Miguel, I. Miguel, and A. Tarim. Modelling and solving english peg solitaire. Computers & Operations Research, 33(10):2935–2959, 2006

    2006

  5. [13]

    Lam and P

    E. Lam and P. Van Hentenryck. A branch-and-price-and-check model for the vehicle routing problem with location congestion. Constraints, 21(3):394–412, 2016

    2016

  6. [15]

    Lecoutre and N

    C. Lecoutre and N. Szczepanski. PyCSP 3: Modeling combinatorial con- strained problems in Python. Technical Report arXiv:2009.00326, CoRR, 2020. https://arxiv.org/abs/2009.00326

    2009 arXiv

  7. [16]

    Marriott and P

    K. Marriott and P. Stuckey. Programming with Constraints. MIT Press, 1998

    1998

  8. [17]

    Musliu, A

    N. Musliu, A. Schutt, and P. Stuckey. Solver independent rotating workforce scheduling. In Proceedings of CPAIOR’18, pages 429–445, 2018

    2018

  9. [18]

    Roussel, T

    S. Roussel, T. Polacsek, and A. Chan. Assembly line preliminary design optimization for an aircraft. In Proceedings of CP’23, pages 32:1–32–19, 2023

    2023

  10. [19]

    Savelsbergh and O

    M. Savelsbergh and O. Smith. Cargo assembly planning. EURO Journal on Transportation and Logistics, 4(3):321–354, 2015

    2015

  11. [20]

    K. Xu, F. Boussemart, F. Hemery, and C. Lecoutre. Random constraint satisfaction: easy generation of hard (satisfiable) instances. Artificial Intelligence, 171(8-9):514–534, 2007

    2007

  12. [21]

    The Picat-SAT compiler

    Neng-Fa Zhou and H ˚ akan Kjellerstrand. The Picat-SAT compiler. In Proceedings of PADL’16, pages 48–62. Springer, 2016

    2016

  13. [2023]

    103 104 BIBLIOGRAPHY

    https://arxiv.org/abs/2302.05405. 103 104 BIBLIOGRAPHY

    arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.