Pith. sign in

REVIEW 4 major objections 5 minor 24 references

Per-Domain Generalizing Policies: On Validation Instances and Scaling Behavior

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that dynamically generated validation instances, growing in size while coverage stays informative, make per-domain GNN planning policies scale further, and it reports gains in all 9 domains tested.

desk verdict Solid, fair empirical method paper with a genuinely useful evaluation protocol; the missing plan-length sensitivity analysis is a real gap but not a fatal one. read the letter →

arxiv 2505.00439 v1 pith:ZE4FPSS6 submitted 2025-05-01 cs.LG cs.AI

Timo P. Gros , Nicola J. Müller , Daniel Fiser , Isabel Valera , Verena Wolf , Jörg Hoffmann This is my paper · ORCID
classification cs.LGcs.AI
keywords per-domaingeneralizationpolicyvalidationscalingbehaviorgraphneuralnetworksplanningdomainscoverageestimationinstancegenerationsequentialconfidenceintervals
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Per-domain planning policies are usually selected by validating them on a fixed set of instances larger than the training instances. This paper claims that the validation instances should instead be generated on the fly, with increasing instance size, so that the validation signal stays informative as the policy improves. The paper also introduces a statistical evaluation protocol that produces coverage curves with guaranteed confidence per instance size, replacing coarse fixed test sets. Across 9 IPC'23 domains, GNN policies selected by the new dynamic validation generalize to larger instances and accumulate more total coverage than policies selected by fixed loss or fixed coverage validation.

What carries the argument

The load-bearing mechanism is Algorithm 2, a loop that starts validation at size $n_0+1$ and increments $n$ while coverage stays above threshold $\tau$, generating $m$ instances per size from a CSP encoding of the domain's instance generator. The CSP maps a desired object count $n$ to valid generator parameters (e.g., in Childsnack, $n = 3v_1 + v_2 + v_3 + 3$ with $v_1 \le v_3$), so each size has a well-defined distribution of instances. Feasibility is controlled by a plan length bound $L$; the evaluation version (Algorithm 3) adds $n$ to $L$, uses all CSP solutions, and applies Chow-Robbins sequential Student's t-intervals to collect enough instances per size to guarantee the coverage estimate is within $\epsilon$ with confidence $1-\kappa$. These algorithms turn "does the policy scale?" into a measurable curve with error bars.

What would settle it

Compute optimal plan lengths for instances at the largest sizes that the dynamic-validation policies solve in, say, Blocksworld; if a large fraction of solved instances have optimal length above $3N + n$, the coverage curves are censored. A cheaper test is to re-run dynamic validation with $L$ doubled and with $L$ halved and check whether the Scale and SumCov rankings across the three validation methods change.

Watch

Extended reading notes

Core claim

Selecting a policy by its coverage on dynamically grown validation instances improves scaling behavior over selecting by loss or coverage on a fixed validation set, in all 9 domains tested and in both reported measures: the largest instance size at which coverage stays above threshold (Scale) and the summed statistical coverage across sizes (SumCov). The paper argues this happens because dynamic validation keeps confronting the policy with instances just beyond its current competence, so the selected policy is the one that degrades most gracefully with instance size. The evaluation protocol additionally shows that fixed IPC test sets hide the degradation curve: average coverage per size is usually 0% or 100% on those sets, while systematic size-scaled sampling reveals a smooth decline.

Load-bearing premise

The evaluation assumes that the plan length bound $L = 3N + n$ grows at least as fast as the true optimal plan length of the instances being solved; if optimal plans grow faster with instance size, coverage is artificially capped and the reported scaling numbers understate what the policies can do.

Editorial extensions

If this is right

  • Validation is no longer a one-time fixed dataset decision: any training loop that re-runs validation can use dynamic sizes, and the CSP generator encodings make the validation distribution explicit.
  • Scaling comparisons between policies become statistically grounded: each point on the coverage curve carries a guaranteed confidence interval, so differences can be attributed to the policy rather than to a handful of hand-picked instances.
  • Because policy selection is the only difference in the experiments, the observed gains imply that fixed-set validation leaves substantial scaling performance on the table even when the underlying GNN training is unchanged.
  • The same evaluation machinery applies to any trajectory-based property, such as plan length, not just to coverage, as the paper demonstrates.
  • The method is policy-representation agnostic and can be attached to supervised or reinforcement-learning training loops.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step that the paper leaves implicit: the validation score could be used as a training signal rather than only a selection signal, for instance by weighting training instances whose sizes are near the current validation frontier.
  • The CSP-based generator decomposition is reusable beyond validation: it gives a principled way to build curricula over instance sizes, and the same per-size instance distribution could be used to measure data efficiency as training set size varies.
  • Since the plan length bound is the main hidden censor, domains whose optimal plan length grows faster than linearly in the number of objects would punish all three validation methods equally but might change their ranking; testing sensitivity to $L$ is the cheapest robustness check.
  • The several-fold validation overhead is confined to training time, so dynamic validation is cheap compared to the cost of selecting a badly scaling policy; in settings where policies are retrained often, the overhead should be weighed against the measured Scale and SumCov gains.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies validation-set choice for learning per-domain generalizing GNN planning policies. It proposes a dynamic validation procedure (Algorithm 2) that, after each training epoch, generates validation instances of increasing size on the fly, continuing until the policy's coverage on a size falls below a threshold tau, with a plan-length bound L=3N. It also introduces an evaluation methodology (Algorithm 3) that generates test instances per size using a sequential Student-t confidence-interval stopping rule and reports statistical coverage as a function of instance size, with Scale and SumCov summary metrics. Experiments on 9 IPC'23 domains compare policies selected by fixed-set loss validation, fixed-set coverage validation, and the proposed dynamic coverage validation, using the same training runs and teacher planner for all methods. The paper reports that dynamic validation yields the best Scale and SumCov scores in all 9 domains and that dynamically selected policies also tend to produce shorter plans.

Significance. If the central claim holds, the paper makes a useful practical contribution: a method that selects better-scaling policies with only a modest overhead during training, plus a more systematic evaluation methodology for scaling behavior in per-domain planning. The experimental design has notable strengths: all three validation methods are applied to the same training runs, so differences are attributable to the validation procedure; validation and evaluation use the same instance generators and size-scaling schemes; the CSP-based instance generation is explicit and reproducible; and the public code/data release supports verification. The sequential confidence-interval evaluation is a genuine improvement over ad hoc IPC test sets, as illustrated by the Blocksworld example in Figure 1. However, the empirical claim rests on a few assumptions that need explicit support, notably the plan-length bound and the aggregation of the three training seeds.

major comments (4)
  1. [Section 3, Algorithm 3] The loop condition in Algorithm 3 reads 'while P(|Ĉ_n − C_n| > ϵ) < κ', but the text states that instances are generated until, with confidence at least 1−κ, the error between the estimated and true coverage is at most ϵ. As written, the loop would stop immediately whenever the already-estimated error probability is below κ, which is the opposite of the intended continuation criterion. The condition should be '> κ' (or '≥ κ'). Because the sequential confidence-interval procedure is a central contribution, this pseudocode error must be fixed and the released implementation checked against the intended stopping rule.
  2. [Section 4, Table 1 and Appendix A] The experiments use three random seeds (Appendix A), but Table 1 reports only a single Scale and SumCov value per domain and method, with no indication of whether these are averages, best seeds, or one representative seed, and no spread or significance information. The paper's central claim is that dynamic validation is best in all 9 domains and 'consistently' improves scaling behavior; without per-seed results or standard errors, the consistency across seeds is not supported. Please report per-seed values or averaged values with error bars, and state how the curves in Figure 2 aggregate seeds.
  3. [Section 3, Table 2] The plan-length bound L=3N+n is load-bearing for both the proposed validation and the comparative evaluation, since coverage is defined by whether a plan is found within L. The manuscript does not justify the constant 3, does not report how often runs fail because of the bound rather than because the instance is truly unsolvable, and provides no sensitivity analysis over L. This matters especially because dynamic validation uses the tighter fixed bound L=3N while probing larger instance sizes, so it may preferentially select policies that finish quickly under a time cap; the reported advantage could then be partly an artifact of the bound. I request a sensitivity analysis (e.g., L=2N+n and L=6N+n, or a time-based bound) and a breakdown of failure causes into 'exceeded L' versus 'no plan exists'.
  4. [Section 4, Scale and SumCov definitions] The Scale measure is not independence of the proposed validation procedure: it uses the same coverage threshold τ and consecutive-failure count ζ as the dynamic validation stopping rule in Algorithm 2, and the evaluation plan-length bound is an affine extension of the validation bound. This alignment means the primary metric is tuned to the dynamic validation objective by construction, which could favor that method independently of its effect on generalization. Please show that the ranking is stable under reasonable variations of τ, ζ, and the evaluation L, or report an additional metric that does not share parameters with the validation procedure.
minor comments (5)
  1. [Throughout] Please fix typos: 'seclection' in the Conclusion, 'covarage' in the Figure 2 caption, and 't-inverval' and 'Chow-Robbin's method' in Section 3.
  2. [Section 4, Figure 3] The comparison of average plan length in Figure 3 discards runs that time out, so the curves are computed over different sets of solved instances for each policy. The text acknowledges a common-instance comparison as future work, but the sentence 'dynamic coverage policies ... find plans of equal or even shorter length' should explicitly state that the comparison is only on the instances each policy solves.
  3. [Section 4, Table 2] Table 2 gives the numeric plan-length bounds for validation and evaluation, but the underlying values of N (average teacher plan length on the largest training instances) are not reported. Please include N per domain so the L values can be reproduced.
  4. [Section 2, Algorithm 2] Algorithm 2 draws validation inputs from only 100 CSP solutions, while the evaluation uses all solutions (CSP(n,∞)). This sampling approximation could bias the validation distribution toward common generator inputs; please discuss or quantify the potential mismatch.
  5. [Section 4, Benchmarks] The exclusions of Floortile, Spanner, Miconic, and Sokoban are clearly motivated, but the abstract's 'all 9 domains used' should be understood as 'all 9 domains in our benchmark set' to avoid overgeneralizing to all IPC'23 domains.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the empirical comparison is self-contained and the validation/evaluation objectives, while aligned, are not identical by construction.

full rationale

The paper's central claim is an empirical comparison: policies selected by three validation methods are compared with an independent evaluation protocol (Algorithm 3) on newly generated test instances. I checked the claimed derivation chain for identity steps. Dynamic validation (Algorithm 2) computes v_pi as a sum of per-size coverages under a fixed plan-length bound L=3N, while the evaluation (Algorithm 3) computes statistical coverage under L=3N+n and reports Scale and SumCov. The validation objective is similar to SumCov but not identical: dynamic validation uses m=10 instances per size, stops after a single size below threshold tau, and uses a fixed L, whereas evaluation uses Chow-Robbins/sequential Student's t-intervals, zeta=2 consecutive failures, and an instance-size-dependent L. Therefore the claim that dynamic validation improves scaling behavior is not forced by construction; it depends on empirical policy behavior. The paper also states that fixed-set coverage validation imposed the same plan-length limit as dynamic coverage, controlling the most obvious confound. The plan-length schedule L=3N and L=3N+n is a heuristic and lacks sensitivity analysis, but that is an experimental validity concern, not circularity, because optimal plan length is not defined in terms of the reported Scale/SumCov scores. No load-bearing self-citation, imported uniqueness theorem, or ansatz-smuggling citation appears; the comparison is self-contained against the fixed-set baselines.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim relies on several manually chosen hyperparameters (m, tau, zeta, epsilon, kappa, plan length multiplier) and on the representativeness of the CSP-based instance generation and the teacher planner's optimality. No novel physical or formal entities are introduced.

free parameters (5)
  • m (per-size validation instances) = 10
    Number of instances generated per size during dynamic validation; set manually and fixed across domains (Section 2, Algorithm 2).
  • tau (coverage threshold) = 0.30
    Dynamic validation stops growing instance size when estimated coverage drops below 30%; also used as the threshold in scaling evaluation (Sections 2 and 3).
  • zeta (consecutive failures) = 2
    Scaling evaluation stops after two consecutive sizes below threshold, defining the Scale metric (Algorithm 3).
  • epsilon, kappa (confidence parameters) = epsilon=0.05, kappa=0.1
    Target half-width and confidence for the sequential Student's t interval in scaling evaluation (Section 3).
  • Plan length multiplier = 3
    Validation uses L=3N and evaluation uses L=3N+n, where N is the average teacher plan length on largest training instances; the factor 3 is chosen manually (Sections 2 and 3, Table 2).
assumptions (4)
  • domain assumption Instance generators and CSP encodings correctly capture all valid domain instances of a given size.
    Uniform sampling over CSP solutions is assumed to yield a representative distribution over size-n instances (Section 2, Systematic instance size scaling).
  • domain assumption The teacher planner (Fast Downward, seq-opt-merge-and-shrink) provides optimal plans and the plan failures during dataset construction do not bias training toward unrepresentatively easy instances.
    Training and validation instances are only those solvable within 20 min and 64 GB (Appendix B).
  • standard math The Chow-Robbins sequential Student's t interval method correctly estimates mean coverage under sequential sampling.
    Used in Algorithm 3 for statistical coverage (Section 3, Chow and Robbins 1965).
  • ad hoc to paper The plan length bound L=3N+n is sufficient for all instance sizes evaluated.
    No sensitivity analysis is provided; coverage failures due to the bound are counted as policy failures (Section 3, Algorithm 3, Table 2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Per-Domain Generalizing Policies: On Validation Instances and Scaling Behavior." pith.science (2026). https://pith.science/paper/ZE4FPSS6

@misc{pith2026250500439,
  author       = {Pith},
  title        = {Pith review of: Per-Domain Generalizing Policies: On Validation Instances and Scaling Behavior},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZE4FPSS6}},
  note         = {Machine review of arXiv:2505.00439}
}
read the original abstract

Recent work has shown that successful per-domain generalizing action policies can be learned. Scaling behavior, from small training instances to large test instances, is the key objective; and the use of validation instances larger than training instances is one key to achieve it. Prior work has used fixed validation sets. Here, we introduce a method generating the validation set dynamically, on the fly, increasing instance size so long as informative and feasible.We also introduce refined methodology for evaluating scaling behavior, generating test instances systematically to guarantee a given confidence in coverage performance for each instance size. In experiments, dynamic validation improves scaling behavior of GNN policies in all 9 domains used.

Figures

Figures reproduced from arXiv: 2505.00439 by the authors.

Figure 1
Figure 1. Scaling behavior (average coverage over instance [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Statistical coverage Cˆ n over n of policies selected using fixed-set loss (blue), fixed-set coverage (orange), and dynamic coverage (purple) validation on 9 domains. The instance sizes used for training are within the vertical red lines, the instance sizes used in fixed validation sets are within the two green lines. Dynamic covarage validation starts at the lower green line. We observed that the largest validation… view at source ↗
Figure 3
Figure 3. Evaluation of plan length using the same policies and instance sizes as in Figure 2. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 16 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Z.; Thi \'e baux, S.; and Trevizan, F

    Chen, D. Z.; Thi \'e baux, S.; and Trevizan, F. 2024. Learning Domain-Independent Heuristics for Grounded and Lifted Planning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 20078--20086

    2024

  4. [4]

    S.; and Robbins, H

    Chow, Y. S.; and Robbins, H. 1965. On the Asymptotic Theory of Fixed-Width Sequential Confidence Intervals for the Mean . The Annals of Mathematical Statistics, 36(2): 457--462

    work page 1965

  5. [5]

    Garg, S.; Bajpai, A.; and Mausam . 2019. Size Independent Neural Transfer for RDDL Planning. In Proceedings of the 29th International Conference on Automated Planning and Scheduling ( ICAPS'19 ) , 631--636. AAAI Press

    work page 2019

  6. [6]

    o ller, D.; Hoffmann, J.; Klauck, M.; Meerkamp, H.; M \

    Gros, T. P.; Gro , J.; H \"o ller, D.; Hoffmann, J.; Klauck, M.; Meerkamp, H.; M \"u ller, N. J.; Schaller, L.; and Wolf, V. 2023. DSMC Evaluation Stages: Fostering Robust and Safe Behavior in Deep Reinforcement Learning--Extended Version. ACM Transactions on Modeling and Computer Simulation, 33(4): 1--28

    work page 2023

  7. [7]

    u ller, N. J.; H \

    Gros, T. P.; M \"u ller, N. J.; H \"o ller, D.; and Wolf, V. 2024. Safe Reinforcement Learning Through Regret and State Restorations in Evaluation Stages. In Principles of Verification: Cycling the Probabilistic Landscape: Essays Dedicated to Joost-Pieter Katoen on the Occasion of His 60th Birthday, Part III, 18--38. Springer

    work page 2024

  8. [8]

    Groshev, E.; Goldstein, M.; Tamar, A.; Srivastava, S.; and Abbeel, P. 2018. Learning Generalized Reactive Policies Using Deep Neural Networks. In Proceedings of the 28th International Conference on Automated Planning and Scheduling (ICAPS'18), 408--416. AAAI Press

    work page 2018

Show all 24 references
  1. [9]

    Helmert, M. 2006. The Fast Downward Planning System. Journal of Artificial Intelligence Research, 26: 191--246

    2006

  2. [10]

    Hoffmann, J.; Edelkamp, S.; Th \'i ebaux, S.; Englert, R.; Liporace, F.; and Tr \"u g, S. 2006. Engineering Benchmarks for Planning: the Domains Used in the Deterministic Part of IPC-4 . Journal of Artificial Intelligence Research, 26: 453--541

    2006

  3. [11]

    Kingma, D. P. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

    2014 arXiv

  4. [12]

    J.; S \'a nchez, P.; Hoffmann, J.; Wolf, V.; and Gros, T

    M \"u ller, N. J.; S \'a nchez, P.; Hoffmann, J.; Wolf, V.; and Gros, T. P. 2024. Comparing State-of-the-art Graph Neural Networks and Transformers for General Policy Learning. In ICAPS Workshop on Planning and Reinforcement Learning (PRL)

    2024

  5. [13]

    Rivlin, O.; Hazan, T.; and Karpas, E. 2020. Generalized planning with deep reinforcement learning. arXiv preprint arXiv:2005.02305

    2020 arXiv

  6. [14]

    E.; Putelli, L.; Serina, I.; Chiari, M.; and Olivato, M

    Rossetti, N.; Tummolo, M.; Gerevini, A. E.; Putelli, L.; Serina, I.; Chiari, M.; and Olivato, M. 2024. Learning General Policies for Planning through GPT Models. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 34, 500--508

    2024

  7. [15]

    Sharma, V.; Arora, D.; Singla, P.; et al. 2023. SymNet 3.0: exploiting long-range influences in learning generalized neural policies for relational MDPs. In Uncertainty in Artificial Intelligence, 1921--1931. PMLR

    2023

  8. [16]

    St hlberg, S.; Bonet, B.; and Geffner, H. 2022 a . Learning general optimal policies with graph neural networks: Expressive power, transparency, and limits. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 32, 629--637

    2022

  9. [17]

    St hlberg, S.; Bonet, B.; and Geffner, H. 2022 b . Learning generalized policies without supervision using gnns. arXiv preprint arXiv:2205.06002

    2022 arXiv

  10. [18]

    St hlberg, S.; Bonet, B.; and Geffner, H. 2023. Learning general policies with policy gradient methods. In Proceedings of the International Conference on Principles of Knowledge Representation and Reasoning, 647--657

    2023

  11. [19]

    St hlberg, S.; Bonet, B.; and Geffner, H. 2024. Learning General Policies for Classical Planning Domains: Getting Beyond C \_2 . arXiv preprint arXiv:2403.11734

    2024 arXiv

  12. [20]

    Taitler, A.; Alford, R.; Espasa, J.; Behnke, G.; Fišer, D.; Gimelfarb, M.; Pommerening, F.; Sanner, S.; Scala, E.; Schreiber, D.; Segovia-Angus, J.; and Seipp, J. 2024. The 2023 International Planning Competition. The AI Magazine, 45(2)

    2024

  13. [21]

    Torralba, \' A .; Seipp, J.; and Sievers, S. 2021. Automatic Instance Generation for Classical Planning. In Proceedings of the 31st International Conference on Automated Planning and Scheduling (ICAPS'21), 376--384

    2021

  14. [22]

    W.; and Xie, L

    Toyer, S.; Thi \' e baux, S.; Trevizan, F. W.; and Xie, L. 2020. ASNets: Deep Learning for Generalised Planning. Journal of Artificial Intelligence Research, 68: 1--68

    2020

  15. [23]

    Toyer, S.; Trevizan, F.; Thi \'e baux, S.; and Xie, L. 2018. Action schema networks: Generalised policies with deep learning. In Proceedings of the AAAI Conference on Artificial Intelligence

    2018

  16. [24]

    X.; and Thi \' e baux, S

    Wang, R. X.; and Thi \' e baux, S. 2024. Learning Generalised Policies for Numeric Planning. In Proceedings of the 34th International Conference on Automated Planning and Scheduling (ICAPS'24), 633--642

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.