{"id":"706216f9-c6d6-470b-bc9e-18562ed9b803","arxiv_id":"2505.00439","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Dynamically generating larger validation instances during training selects planning policies that generalize to larger instances better than fixed validation sets, across all 9 domains tested.","lead":"The paper introduces a dynamic validation-set generator that grows instance sizes on the fly, improving scaling behavior of learned planning policies in all 9 test domains. It also proposes a statistical evaluation method that measures coverage per instance size with guaranteed confidence intervals.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Plan-length bound L=3N+n is a potential confound: dynamic validation selects for solving within a short fixed bound, so its coverage advantage may be an artifact of the bound; no sensitivity analysis over L is provided.","rationale":"Reader's verdict: CONDITIONAL. I agree with the conditional verdict and with the reader's identification of the plan-length bound as the weakest load-bearing assumption. The paper's strongest claim is a universal empirical statement over nine domains. The comparison is otherwise fair: same training runs, same evaluation pipeline, fixed-set baselines included. The main threat to that universality is not missing error bars (important but direction-agnostic) nor the disclosed domain exclusions (which are explicit and qualify the claim), but the plan-length cap that defines coverage. The cap enters both selection and evaluation; absent a sensitivity analysis, the observed advantage of dynamic validation could be an artifact of selection pressure toward short plans. The proposed test would settle this by relaxing L and by measuring how often the cap is the binding constraint. If the advantage persists under much larger bounds, the claim is robust; if not, the paper should be revised to state the result as 'improvement under bounded plan length.' Note also that a literal reading of Algorithm 3's while condition appears inverted (it continues while the confidence requirement is already met); this is a secondary reproducibility issue that should be corrected, but the experimental implementation and released code may well be correct. Overall, the conditional accept with a request for sensitivity analysis is the right call; no verdict change is needed.","tokens_in":8376,"tokens_out":9128,"duration_ms":104153,"concrete_test":"Instrument the runPolicy call in Algorithm 3 to record failure reason (plan-length bound exceeded vs. no plan found) and rerun the full evaluation with relaxed bounds, e.g., L'=6N+2n and L''=12N+4n, for all 9 domains and all three validation policies. Recompute Scale and SumCov. Also count, at each reported Scale size, how many instances are solved only because L is as large as it is. If dynamic coverage remains best in every domain under L'' and the failure-reason counts show few bound-limited runs, the concern is resolved; if any domain flips or the gap narrows materially, the bound is a confound and the central claim must be qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is an empirical comparison of validation methods, and the evaluation's definition of 'coverage' depends on a plan-length cap. Dynamic validation (Algorithm 2) uses a fixed bound L=3N, where N is the average teacher plan length on the largest training instances (Table 2). The evaluation (Algorithm 3) uses L=3N+n. These bounds are not validated against true optimal plan lengths, and no sensitivity analysis is reported. This matters in two ways. First, if optimal plan length grows faster than L, coverage is understated for all policies. Second, and more importantly, the bias may favor dynamic validation: because dynamic validation explores larger instances while keeping L fixed, it exerts stronger selection pressure for policies that finish within L steps. The selected dynamic policies may be 'good under a time cap' rather than 'good at scaling'; Figure 3 indeed shows they find shorter plans. Under a larger bound, fixed-set policies that currently fail due to the cap could solve additional instances, potentially reducing or reversing the reported advantage. The paper never reports how often failures are due to exceeding L rather than true unsolvability. Thus the unconditional statement that dynamic validation improves scaling behavior is not yet settled.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies validation-set choice for learning per-domain generalizing GNN planning policies. It proposes a dynamic validation procedure (Algorithm 2) that, after each training epoch, generates validation instances of increasing size on the fly, continuing until the policy's coverage on a size falls below a threshold tau, with a plan-length bound L=3N. It also introduces an evaluation methodology (Algorithm 3) that generates test instances per size using a sequential Student-t confidence-interval stopping rule and reports statistical coverage as a function of instance size, with Scale and SumCov summary metrics. Experiments on 9 IPC'23 domains compare policies selected by fixed-set loss validation, fixed-set coverage validation, and the proposed dynamic coverage validation, using the same training runs and teacher planner for all methods. The paper reports that dynamic validation yields the best Scale and SumCov scores in all 9 domains and that dynamically selected policies also tend to produce shorter plans.","tokens_in":8663,"tokens_out":7706,"duration_ms":82756,"significance":"If the central claim holds, the paper makes a useful practical contribution: a method that selects better-scaling policies with only a modest overhead during training, plus a more systematic evaluation methodology for scaling behavior in per-domain planning. The experimental design has notable strengths: all three validation methods are applied to the same training runs, so differences are attributable to the validation procedure; validation and evaluation use the same instance generators and size-scaling schemes; the CSP-based instance generation is explicit and reproducible; and the public code/data release supports verification. The sequential confidence-interval evaluation is a genuine improvement over ad hoc IPC test sets, as illustrated by the Blocksworld example in Figure 1. However, the empirical claim rests on a few assumptions that need explicit support, notably the plan-length bound and the aggregation of the three training seeds.","major_comments":[{"comment":"The loop condition in Algorithm 3 reads 'while P(|Ĉ_n − C_n| > ϵ) < κ', but the text states that instances are generated until, with confidence at least 1−κ, the error between the estimated and true coverage is at most ϵ. As written, the loop would stop immediately whenever the already-estimated error probability is below κ, which is the opposite of the intended continuation criterion. The condition should be '> κ' (or '≥ κ'). Because the sequential confidence-interval procedure is a central contribution, this pseudocode error must be fixed and the released implementation checked against the intended stopping rule.","section":"Section 3, Algorithm 3"},{"comment":"The experiments use three random seeds (Appendix A), but Table 1 reports only a single Scale and SumCov value per domain and method, with no indication of whether these are averages, best seeds, or one representative seed, and no spread or significance information. The paper's central claim is that dynamic validation is best in all 9 domains and 'consistently' improves scaling behavior; without per-seed results or standard errors, the consistency across seeds is not supported. Please report per-seed values or averaged values with error bars, and state how the curves in Figure 2 aggregate seeds.","section":"Section 4, Table 1 and Appendix A"},{"comment":"The plan-length bound L=3N+n is load-bearing for both the proposed validation and the comparative evaluation, since coverage is defined by whether a plan is found within L. The manuscript does not justify the constant 3, does not report how often runs fail because of the bound rather than because the instance is truly unsolvable, and provides no sensitivity analysis over L. This matters especially because dynamic validation uses the tighter fixed bound L=3N while probing larger instance sizes, so it may preferentially select policies that finish quickly under a time cap; the reported advantage could then be partly an artifact of the bound. I request a sensitivity analysis (e.g., L=2N+n and L=6N+n, or a time-based bound) and a breakdown of failure causes into 'exceeded L' versus 'no plan exists'.","section":"Section 3, Table 2"},{"comment":"The Scale measure is not independence of the proposed validation procedure: it uses the same coverage threshold τ and consecutive-failure count ζ as the dynamic validation stopping rule in Algorithm 2, and the evaluation plan-length bound is an affine extension of the validation bound. This alignment means the primary metric is tuned to the dynamic validation objective by construction, which could favor that method independently of its effect on generalization. Please show that the ranking is stable under reasonable variations of τ, ζ, and the evaluation L, or report an additional metric that does not share parameters with the validation procedure.","section":"Section 4, Scale and SumCov definitions"}],"minor_comments":[{"comment":"Please fix typos: 'seclection' in the Conclusion, 'covarage' in the Figure 2 caption, and 't-inverval' and 'Chow-Robbin's method' in Section 3.","section":"Throughout"},{"comment":"The comparison of average plan length in Figure 3 discards runs that time out, so the curves are computed over different sets of solved instances for each policy. The text acknowledges a common-instance comparison as future work, but the sentence 'dynamic coverage policies ... find plans of equal or even shorter length' should explicitly state that the comparison is only on the instances each policy solves.","section":"Section 4, Figure 3"},{"comment":"Table 2 gives the numeric plan-length bounds for validation and evaluation, but the underlying values of N (average teacher plan length on the largest training instances) are not reported. Please include N per domain so the L values can be reproduced.","section":"Section 4, Table 2"},{"comment":"Algorithm 2 draws validation inputs from only 100 CSP solutions, while the evaluation uses all solutions (CSP(n,∞)). This sampling approximation could bias the validation distribution toward common generator inputs; please discuss or quantify the potential mismatch.","section":"Section 2, Algorithm 2"},{"comment":"The exclusions of Floortile, Spanner, Miconic, and Sokoban are clearly motivated, but the abstract's 'all 9 domains used' should be understood as 'all 9 domains in our benchmark set' to avoid overgeneralizing to all IPC'23 domains.","section":"Section 4, Benchmarks"}],"recommendation":"major_revision","confidential_remarks":"The main technical risk is the plan-length confound: the evaluation metric and the dynamic validation procedure share the same L, τ, and ζ, so the reported advantage may be partly a metric-alignment effect rather than a genuine scaling-behavior improvement. If the authors can provide the requested sensitivity analysis and per-seed statistics, I expect the central claim to be defensible. The pseudocode bug in Algorithm 3 also needs correction before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know about this one: it’s a solid method paper, not a breakthrough, but it delivers a genuinely useful idea and runs a fair comparison. The central claim—that dynamically generating validation instances at increasing sizes selects GNN policies that scale better than fixed-set loss or coverage validation—holds up in all 9 domains tested. I’d send it to review.\n\nWhat’s actually new: prior per-domain policy learning uses a fixed validation set, which limits how far beyond training size you can assess scaling. Here they generate validation instances on the fly, increasing size while coverage stays above a threshold. They also introduce a more systematic evaluation protocol based on Chow-Robbins sequential confidence intervals, so coverage per instance size comes with a statistical guarantee. The CSP encoding of instance generators to control object count is a nice, practical contribution. The comparison is fair: same training runs, same evaluation method, only validation differs. Code and data are on Zenodo. They also disclose the four IPC'23 domains they dropped and why, which is the right kind of transparency.\n\nThe soft spots are real but manageable. First, no variance is reported for the Scale/SumCov numbers in Table 1 despite three seeds. You need per-seed spread or confidence intervals. Second, the plan-length bound L is a load-bearing choice and there is no sensitivity analysis. Dynamic validation uses L=3N fixed; evaluation uses L=3N+n. If optimal plan length grows faster, coverage is understated for everyone, and dynamic validation might be selecting policies that are good under a time cap rather than genuinely better at scaling. I’d push back a little: the fixed-set coverage validation uses the same fixed L, so between those two the selection pressure is identical, and the dynamic policies also produce shorter plans on solved instances (Figure 3), which weakens the pure artifact story. Still, the authors should report how often failures are due to exceeding L and run at least one alternate L to show the ranking is stable.\n\nWho’s this for? People working on per-domain generalized planning, and anyone evaluating scaling behavior of learned policies. The evaluation method alone is worth borrowing. I agree with the conditional accept: with the sensitivity analysis and error bars, it’s a clean contribution. Yes, send it to peer review.","headline":"Solid, fair empirical method paper with a genuinely useful evaluation protocol; the missing plan-length sensitivity analysis is a real gap but not a fatal one.","tokens_in":9147,"tokens_out":3173,"would_cite":true,"duration_ms":31493,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that dynamically generated validation instances, growing in size while coverage stays informative, make per-domain GNN planning policies scale further, and it reports gains in all 9 domains tested.","keywords":["per-domain generalization","policy validation","scaling behavior","graph neural networks","planning domains","coverage estimation","instance generation","sequential confidence intervals"],"falsifier":"Compute optimal plan lengths for instances at the largest sizes that the dynamic-validation policies solve in, say, Blocksworld; if a large fraction of solved instances have optimal length above $3N + n$, the coverage curves are censored. A cheaper test is to re-run dynamic validation with $L$ doubled and with $L$ halved and check whether the Scale and SumCov rankings across the three validation methods change.","tokens_in":8210,"feed_emoji":"📈","tokens_out":6086,"duration_ms":59273,"temperature":0.7,"pith_summary":"Per-domain planning policies are usually selected by validating them on a fixed set of instances larger than the training instances. This paper claims that the validation instances should instead be generated on the fly, with increasing instance size, so that the validation signal stays informative as the policy improves. The paper also introduces a statistical evaluation protocol that produces coverage curves with guaranteed confidence per instance size, replacing coarse fixed test sets. Across 9 IPC'23 domains, GNN policies selected by the new dynamic validation generalize to larger instances and accumulate more total coverage than policies selected by fixed loss or fixed coverage validation.","feed_headline":"Growing validation sets improve scaling in all 9 domains","feed_subtitle":"Policies chosen by on-the-fly larger instances beat fixed loss and coverage sets on every domain tested.","key_machinery":"The load-bearing mechanism is Algorithm 2, a loop that starts validation at size $n_0+1$ and increments $n$ while coverage stays above threshold $\\tau$, generating $m$ instances per size from a CSP encoding of the domain's instance generator. The CSP maps a desired object count $n$ to valid generator parameters (e.g., in Childsnack, $n = 3v_1 + v_2 + v_3 + 3$ with $v_1 \\le v_3$), so each size has a well-defined distribution of instances. Feasibility is controlled by a plan length bound $L$; the evaluation version (Algorithm 3) adds $n$ to $L$, uses all CSP solutions, and applies Chow-Robbins sequential Student's t-intervals to collect enough instances per size to guarantee the coverage estimate is within $\\epsilon$ with confidence $1-\\kappa$. These algorithms turn \"does the policy scale?\" into a measurable curve with error bars.","core_discovery":"Selecting a policy by its coverage on dynamically grown validation instances improves scaling behavior over selecting by loss or coverage on a fixed validation set, in all 9 domains tested and in both reported measures: the largest instance size at which coverage stays above threshold (Scale) and the summed statistical coverage across sizes (SumCov). The paper argues this happens because dynamic validation keeps confronting the policy with instances just beyond its current competence, so the selected policy is the one that degrades most gracefully with instance size. The evaluation protocol additionally shows that fixed IPC test sets hide the degradation curve: average coverage per size is usually 0% or 100% on those sets, while systematic size-scaled sampling reveals a smooth decline.","pith_inferences":["A natural next step that the paper leaves implicit: the validation score could be used as a training signal rather than only a selection signal, for instance by weighting training instances whose sizes are near the current validation frontier.","The CSP-based generator decomposition is reusable beyond validation: it gives a principled way to build curricula over instance sizes, and the same per-size instance distribution could be used to measure data efficiency as training set size varies.","Since the plan length bound is the main hidden censor, domains whose optimal plan length grows faster than linearly in the number of objects would punish all three validation methods equally but might change their ranking; testing sensitivity to $L$ is the cheapest robustness check.","The several-fold validation overhead is confined to training time, so dynamic validation is cheap compared to the cost of selecting a badly scaling policy; in settings where policies are retrained often, the overhead should be weighed against the measured Scale and SumCov gains."],"forward_implications":["Validation is no longer a one-time fixed dataset decision: any training loop that re-runs validation can use dynamic sizes, and the CSP generator encodings make the validation distribution explicit.","Scaling comparisons between policies become statistically grounded: each point on the coverage curve carries a guaranteed confidence interval, so differences can be attributed to the policy rather than to a handful of hand-picked instances.","Because policy selection is the only difference in the experiments, the observed gains imply that fixed-set validation leaves substantial scaling performance on the table even when the underlying GNN training is unchanged.","The same evaluation machinery applies to any trajectory-based property, such as plan length, not just to coverage, as the paper demonstrates.","The method is policy-representation agnostic and can be attached to supervised or reinforcement-learning training loops."],"supporting_citations":[{"why":"Supplies the GNN policy architecture and supervised training procedure that the experiments build on.","marker":"Ståhlberg, Bonet, and Geffner 2022a"},{"why":"Supplies the no-revisit mechanism that prevents the greedy policy from cycling during evaluation.","marker":"Ståhlberg, Bonet, and Geffner 2022b"},{"why":"Introduced coverage-based validation on a fixed set, the prior baseline that dynamic coverage is compared against.","marker":"Rossetti et al. 2024"},{"why":"Provides the IPC'23 domains and their instance generators used as benchmarks.","marker":"Taitler et al. 2024"},{"why":"Provides the sequential Student's t-interval method used to guarantee the precision of the statistical coverage estimates.","marker":"Chow and Robbins 1965"},{"why":"The Fast Downward planner is used as the teacher that computes optimal plans for training and validation instances.","marker":"Helmert 2006"},{"why":"Supplies community conventions and existing generators for systematic instance size scaling.","marker":"Hoffmann et al. 2006"}],"fun_headline_variants":["Dynamic validation beats fixed sets in all 9 domains","On-the-fly validation boosts scaling in 9/9 domains","Dynamic validation wins on scaling across all 9 domains","Larger validation instances on the fly improve scaling","Dynamic validation sets: better scaling in every domain"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the plan length bound $L = 3N + n$ grows at least as fast as the true optimal plan length of the instances being solved; if optimal plans grow faster with instance size, coverage is artificially capped and the reported scaling numbers understate what the policies can do.","fun_headline_variants_meta":{"raw":{"variants":["Dynamic validation beats fixed sets in all 9 domains","On-the-fly validation boosts scaling in 9/9 domains","Dynamic validation wins on scaling across all 9 domains","Larger validation instances on the fly improve scaling","Dynamic validation sets: better scaling in every domain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001,"raw_usage":{"total_tokens":4152,"prompt_tokens":782,"completion_tokens":3370,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":398,"completion_tokens_details":{"reasoning_tokens":3293}},"tokens_in":398,"tokens_out":3370,"duration_ms":21598,"temperature":1.0,"reasoning_tokens":3293,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:41:52.757816+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute optimal plan lengths for instances at the largest sizes that the dynamic-validation policies solve in, say, Blocksworld; if a large fraction of solved instances have optimal length above $3N + n$, the coverage curves are censored. A cheaper test is to re-run dynamic validation with $L$ doubled and with $L$ halved and check whether the Scale and SumCov rankings across the three validation methods change.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the IPC'23 domains and their instance generators used as benchmarks."},{"cited_title":"S.; and Robbins, H","cited_arxiv_id":null,"evidence_quote":"Provides the sequential Student's t-interval method used to guarantee the precision of the statistical coverage estimates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies community conventions and existing generators for systematic instance size scaling."}],"review_version":1}