{"id":"1b52e0f2-71e8-46ec-a4de-408b3e9a65da","arxiv_id":"2608.12426","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Across 15 language models, the probability of satisfying all k constraints falls below 50% at 7 constraints for the best model and at 3 or fewer for most, and the paper attributes this to near-independent per-constraint failures, a claim its own numbers undercut.","lead":"This paper introduces a benchmark that varies the number of simultaneous instructions given to 15 large language models, and reports that the chance of satisfying all instructions collapses as the count grows, even though each individual instruction is usually followed. A general reader might care because it quantifies when complex AI systems start failing under multi-part requests and tests whether common fixes like planning or retries actually help.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'near-independence' mechanism is contradicted by the paper's own k=8 numbers: with mCSR=40.7%, independent failures predict sCSR≈0.075%, not 5.7%; the ~76x excess requires strong positive dependence or probe-difficulty clustering that the reported φ=+0.067 does not establish.","rationale":"The paper's empirical contribution—a deterministic, per-k benchmark of compositional constraint satisfaction across 15 models—is substantial, and the qualitative collapse of sCSR is credible. The load-bearing issue is the proposed mechanism. The reader identified the same broad weakness, though their specific arithmetic (using aggregate mCSR^k as 'the' independence baseline) is only one possible baseline; with probe heterogeneity, an independence model can in principle produce sCSR larger than mCSR^k. However, the paper does not report the per-probe product baseline, and the figures it does report (Section O synergy up to +26pp; φ=+0.067; mCSR decaying to 40.7% at k=8 while sCSR=5.7%) cannot all be reconciled with 'failures are nearly independent, which is what makes the accumulation multiplicative.' The 76x ratio between observed sCSR and the k-conditional independence prediction is a dependence effect too large to wave away as weak coupling. Either constraints within a probe are strongly positively dependent (shared output features make probes all-pass or all-fail together), or probes are strongly heterogeneous in difficulty; both possibilities contradict the paper's conclusion that there is no structure to exploit and that only per-constraint reliability matters. The correct fix would be to model probe-level difficulty explicitly and report the conditional-independence baseline; until then, the central claim is unsupported. The constraint-count and release contradictions are secondary but reinforce the need for revision. I therefore agree with the reader's REJECT verdict, with the caveat that the rejection should be framed as 'mechanism unsupported as stated' rather than 'the benchmark is worthless.'","tokens_in":43309,"tokens_out":20947,"duration_ms":207974,"concrete_test":"Compute the correct independence baseline for k=8: for each probe, multiply the pass rates of its constituent constraints (estimated within the same k stratum, and separately using k=1 rates), then average over probes; compare to observed sCSR=5.7%. Independently fit a multilevel model with a per-probe latent difficulty factor to the binary constraint-success matrix at k=8 and evaluate E[P^8] under the fitted distribution, alongside the estimated pairwise φ. If E[P^8]≈5.7% with φ≈0.067, near-independence survives as conditional independence given probe difficulty; if the predicted sCSR remains near 0.1-1%, the multiplicative accumulation mechanism is falsified and the paper must be reframed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Abstract, Section 3.1, Section 4, Section 6) is that constraint failures are nearly independent, making sCSR the product of k marginal rates. The paper's own headline numbers refute this mechanism as stated. At k=8, reported mCSR is 40.7% while sCSR is 5.7%. Under independent failures at the k-conditional marginal rate, expected all-pass is 0.407^8 ≈ 0.075%; observed is ~76x larger. Equivalently, the implied per-constraint all-pass rate is 0.057^(1/8) ≈ 0.70, far above the measured 40.7%. The sCSR fit in Figure 1 (P(k)=e^{−0.376k}, i.e., factor 0.686 per constraint) is not the product of the reported decaying mCSR; it is a separate exponential fit. The paper's own synergy analysis (Section O) reports actual sCSR exceeding independence predictions by up to +26pp at low k, which is inconsistent with near independence. The mean pairwise φ=+0.067 is presented as evidence of near independence, but a 76x elevation of the all-pass tail is a strong dependence signature, reflecting either tightly coupled output-feature clusters or substantial probe-difficulty heterogeneity; either interpretation undermines the 'no pairing to exploit' and 'only per-constraint reliability helps' conclusions. The correct baseline should average per-probe products of marginal pass rates, not raise aggregate mCSR to the kth power; this baseline is never reported, and no version consistent with mCSR decay and φ=0.067 is likely to match 5.7%. Secondary inconsistencies (36 vs 30 constraint types; Appendix A says code 'will be released' while the abstract says it is released) further weaken the manuscript. The benchmark measurements may be salvageable, but the multiplicative-independence explanation is not supported by the reported numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Constraint Saturation Evaluation (CSE), a procedurally generated benchmark for multi-constraint instruction following with 36 deterministic constraint types, k=1–12, 15 models, and 369,753 constraint checks. It reports three main findings: per-constraint pass rate decays multiplicatively (mCSR ≈ 72.0% × 0.922^{k−1}); structural constraints degrade faster than lexical ones, with a comprehension-maintenance gap predicting degradation; and constraint failures are 'nearly independent,' so joint sCSR is approximately the product of k marginal rates. The paper also reports three interventions (pre-generation planning, post-hoc self-correction, and best-of-5 retries) and concludes that only per-constraint reliability, not constraint pairing or prompt structuring, can move the compositional threshold.","tokens_in":43653,"tokens_out":8849,"duration_ms":91883,"significance":"If the empirical trajectory and ranking results were reliable, CSE would be a useful diagnostic resource: deterministic verification avoids the LLM-as-judge circularity, the k-trajectory across 15 models is well matched to deployment concerns, and the promised release of probes, verifiers, outputs, and code would support reproducibility. The mCSR-versus-sCSR divergence and the per-constraint degradation hierarchy are genuinely interesting observations. However, the central mechanistic claim that failures are nearly independent and that sCSR is the product of marginal rates is not established, and the paper's own synergy analysis appears to contradict it. Because the headline conclusions about the 'only lever' and the impossibility of constraint-selection mitigation depend on this mechanism, the contribution as currently framed cannot be accepted.","major_comments":[{"comment":"The paper asserts that sCSR 'collapses because it is the product of k marginal rates,' but this is a theorem under independence, not an empirical consequence of the reported measurements. The correct independence baseline is the average over probes of the product of per-probe constraint pass rates; the aggregate mCSR is not a valid substitute when per-constraint rates are heterogeneous. The paper never reports this baseline. At k=8, mCSR is 40.7% and sCSR is 5.7%; if marginal rates were homogeneous, independence would predict about 0.075%, and while heterogeneous probes can move the product upward, no calculation in the paper shows how 5.7% arises. The near-multiplicative claim is therefore unsupported as stated.","section":"Sections 3.1 and 4, Eqs. (1)–(2)"},{"comment":"The synergy analysis directly contradicts the near-independence claim. Section O reports that actual sCSR exceeds the independence prediction by +26.4pp at k=1 and by positive amounts at every k from 1 to 11. But by the definitions in Eqs. (1)–(2), sCSR equals mCSR at k=1, so any baseline computed from the same per-constraint rates must give zero synergy at k=1. The reported +26pp at k=1 implies either that the 'independence prediction' is not the product-of-marginal-rates rule stated in Section 3.1 or that the synergy computation is internally inconsistent. Moreover, positive synergy is positive correlation in success outcomes, which is not the same as 'failures are nearly independent.'","section":"Section O and Figure 2"},{"comment":"The two fitted decay models are not shown to be consistent with each other. The paper fits mCSR(k)=72.0%×0.922^{k−1} and separately fits sCSR(k)=1.000·e^{−0.376k}+0.003, corresponding to a per-constraint factor of about e^{−0.376}≈0.686. If sCSR really were the product of the k marginal rates, the sCSR fit would have to be derived from the mCSR fit; no such derivation is presented, and the per-constraint factors differ by a factor of roughly 0.686/0.922≈0.74. The paper either needs to derive the probe-level curve from the marginal curve or explain why near-independence permits a separate exponential fit with a different rate.","section":"Section 4, 'Two-regime decay model'"},{"comment":"The distinction between 'independence across constraints within a response' and 'dependence across resamples of the same probe' is asserted but not tested. The best-of-5 result—about two-fifths of the independent-resampling lift—is direct evidence of strong probe-level correlation across draws. A probe-level random effect (topic difficulty, parameter difficulty, or a deterministic failure mode) would produce correlated draws while also inflating the sCSR tail relative to a product-of-marginals baseline. A mixed model or per-probe conditional analysis is needed before one can conclude that 'only per-constraint reliability helps'; the pairwise φ≈+0.067 analysis does not rule out such an effect.","section":"Section 5, 'Two senses of independence'"}],"minor_comments":[{"comment":"The text refers to '30 constraint types,' 'all 30 constraint verifiers,' and 'Ten of 30 constraints,' while the benchmark is defined as 36 constraint types (Section 3.1, Table 2). These counts should be reconciled.","section":"Limitations and Appendix Q"},{"comment":"Section 3.1 mentions 'Composability profiles for all 38 candidates,' but Section K describes 36 final constraints plus at least two rejected candidates. The relation between the 38 candidates and the 36-constraint pool should be stated explicitly.","section":"Section 3.1"},{"comment":"The numbers for easy- and hard-tercile synergy are not aligned: Section 4 text reports '+33pp' and '+18pp,' while Section O reports '+32.6pp' and '+17.7pp.' Such inconsistencies make the synergy analysis difficult to audit.","section":"Figure 2 and Section O"},{"comment":"The term 'phase transition' is used for what appears to be a smooth exponential decay with no discontinuity. Either an operational definition of 'phase transition' should be given, or a less loaded term should be used.","section":"Title and Section 4"},{"comment":"The intervention results in Table 3 are reported without confidence intervals or significance tests, despite k* changes of one constraint and small model counts; the planning result in particular is consistent with no effect under reasonable noise.","section":"Table 3"}],"recommendation":"reject","confidential_remarks":"I would not send this manuscript back for simple revision. The core mechanism claim is contradicted by the paper's own synergy analysis, and the missing per-probe product baseline means the central derivation is not merely incomplete but currently incoherent. A reanalysis that computes proper independence baselines and drops or substantially revises the near-independence conclusions could salvage the descriptive benchmark contribution, but that is a major rewrite rather than a local fix. I also note the unusually large appendix contains several internal count inconsistencies (36 vs 30 constraints) that an editor may want checked against the released data before further review."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look for the benchmark, but the central claim collapses. The real takeaway: CSE is a well-constructed, large-scale, deterministic multi-constraint benchmark, and the per-k degradation trajectories across 15 models are practically informative. The 36-constraint taxonomy, compatibility checking, and the three intervention ablations are done carefully. The empirical fact that probe-level success collapses while per-constraint rates stay moderate is real and deployment-relevant.\n\nThe trouble is the paper's main explanatory mechanism. The authors claim failures are 'nearly independent' and that joint success is the product of k per-constraint rates. Their own headline numbers refute that. At k=8 they report mCSR=40.7% and sCSR=5.7%. If per-constraint failures were independent at that average rate, sCSR would be about 0.075% — a factor of roughly 75 lower. So the tail is strongly positively correlated. The mean pairwise φ=+0.067 is too weak to account for that; the correlation is concentrated in the all-pass tail, likely through shared output features (sentence count errors fail every sentence-dependent constraint). The correct independence baseline is the average over probes of the product of each probe's constituent marginal rates, not the aggregate mCSR raised to the kth power. The paper never reports that baseline, and Section O's own synergy analysis shows actual sCSR exceeding the independence prediction by up to +26pp at low k. So the 'multiplicative accumulation' story, which is the paper's central contribution, is not supported.\n\nSecondary issues: the 'phase transition' framing is misleading for a smooth exponential decay; the text inconsistently says 36 and 30 constraint types; the abstract says code/data are released while Appendix A says 'will be released upon publication'. These are minor compared to the independence problem, but they don't help.\n\nWhat survives: the benchmark itself, the per-k measurements, the degradation hierarchy, and the finding that planning doesn't help while retries/self-correction buy only one or two constraints. Those are worth having, if the data are real and the code is actually released.\n\nBottom line: this deserves a serious referee because the empirical apparatus is substantial and the flaws are identifiable. But as written, the central claim is wrong. I'd reject with an invitation to resubmit after reframing the independence analysis, reporting the proper per-probe baseline, reconciling the synergy result, and releasing the code.","headline":"Useful benchmark and per-k measurements, but the central 'near-independence' mechanism is contradicted by the paper's own numbers; the main explanation doesn't hold.","tokens_in":44303,"tokens_out":2697,"would_cite":false,"duration_ms":27577,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that LLM instruction-following collapses because constraint failures multiply: each added constraint trims per-constraint pass rate by a fixed factor, so the chance of satisfying all k constraints falls steeply even when…","keywords":["instruction following","constraint satisfaction","compositional evaluation","phase transition","LLM evaluation","multiplicative decay","constraint saturation"],"falsifier":"Recompute sCSR at k=8 from the reported per-constraint rates: if per-constraint pass rates average 40.7% and successes were independent, the all-eight success rate would be roughly (0.407)^8 = 0.00117, or 0.12%, not the reported 5.7%. A direct check of whether the marginals at k=8 are approximately equal across constraints, and whether their geometric mean is near 40.7% or much higher, would settle whether the independence assumption holds or whether a small set of high-pass-rate constraints is carrying the joint success rate.","tokens_in":43003,"feed_emoji":"📉","tokens_out":7734,"duration_ms":65377,"temperature":0.7,"pith_summary":"The paper asks how reliably large language models can obey many instructions at once, and answers with a partial negative: performance on any single constraint decays gently as the number of simultaneous constraints grows, but the probability of satisfying all of them collapses. It introduces Constraint Saturation Evaluation (CSE), a procedurally generated benchmark with 36 deterministically verifiable constraint types, and tests 15 models with more than 369,000 checks. The central quantitative claim is that each added constraint cuts the per-constraint success rate by a fixed fraction, roughly 7.8 percent, while the all-constraint success rate at eight constraints is only 5.7 percent even though individual constraints still pass about 41 percent of the time. The paper argues that this collapse is mostly the arithmetic product of many small, nearly independent failures, and that structural constraints that require sustained tracking degrade about twice as fast as simple lexical ones.","feed_headline":"LLMs pass constraints separately but collapse on all at once","feed_subtitle":"One benchmark shows per-instruction skill around 41% but only 5.7% success on eight simultaneous requirements.","key_machinery":"The central object is Constraint Saturation Evaluation (CSE), a procedurally generated benchmark that varies the number of simultaneous constraints (k) from 1 to 12 across 36 deterministically verifiable constraint types. Two metrics carry the argument: sCSR (strict constraint satisfaction rate), the fraction of probes where all k constraints pass, and mCSR (marginal CSR), the average per-constraint pass rate. The divergence between these metrics, with mCSR decaying gently while sCSR collapses, is what the paper calls the phase transition. The load-bearing identity is the multiplicative model P(k) = a·r^(k−1), fitted on k=1–8 and validated on k=9–12. The key quantitative measure for the independence claim is the pairwise phi coefficient among constraint failures, which the paper reports as small on average.","core_discovery":"The central discovery is that the steep decline in compositional instruction following is mostly a consequence of near-multiplicative accumulation of per-constraint failures, not pairwise constraint interference. The paper measures that at k=8 the average per-constraint pass rate is 40.7%, but the rate of simultaneously satisfying all eight constraints is only 5.7%. It fits the per-constraint decay as mCSR(k)=72.0% x 0.922^(k−1) with 0.2pp held-out MAE, and the probe-level success collapses below 2% by k=11. The paper reports that pairwise co-failure correlations are weak (mean phi = +0.067; 54% of pairs within |phi| <= 0.05), with only one negative pair, implying that selecting or rearranging constraints cannot mitigate the collapse. It also finds a constraint hierarchy: structural constraints lose 2.0x more baseline capability per added constraint than lexical ones, and this is predicted by the comprehension-maintenance gap, the difference between a model's continuous compliance score and its strict binary pass rate. Constraints that require sustained tracking, like word counting, degrade fastest, while binary constraints like including a mandatory word are compositionally immune.","pith_inferences":["If the near-multiplicative failure model holds, it implies a strong testable prediction for any new constraint type: its contribution to the joint failure rate is determined by its standalone pass rate and its coupling to shared output features, so the collapse curve for a model can be assembled from single-constraint measurements plus a small number of pairwise output-feature couplings.","The paper's own synergy results at low k suggest that its near-independence claim may be best understood as a high-k phenomenon only: at k=1–4, actual sCSR exceeds the product-of-marginals prediction by up to 26pp, which cannot be an arithmetic artifact in the same way as the high-k floor.","The finding that best-of-N retries deliver only two-fifths of the lift predicted by independent resampling suggests a second, within-probe correlation layer: failures are near-independent across constraints but largely deterministic across resamples, which could be probed directly by measuring resample correlation separately from across-constraint correlation.","The structural-vs-lexical degradation gap, if it reflects sustained tracking demand rather than chance, predicts that scaffolding that externalises tracking (for example, structured generation that checks counts during decoding) should flatten the hierarchy rather than uniformly improve all constraints."],"forward_implications":["Production systems that assume gradual degradation will overshoot: a system that is reliable with 3 constraints can fail most probes with 5–6 constraints, even though each individual constraint is still satisfied at moderate rates.","The compositional half-life k* gives a one-number estimate of how many simultaneous constraints a model can handle, ranging from 1 to 7 across the 15 models tested, and the paper's decay factors allow estimating the maximum k for any target compliance rate.","Inference-time mitigations have bounded effect: planning scaffolds do not shift the threshold, while post-hoc self-correction and best-of-5 retries recover only one to two constraints, because a given probe's failure is largely deterministic across resamples.","Improving per-constraint reliability is the only identified lever that moves the collapse threshold, since constraint pairing and rearrangement cannot mitigate a near-multiplicative failure accumulation.","Continuous compliance scores can overestimate true satisfaction by up to 50pp near the transition (e.g., W1 at 78.6% score vs 28.4% strict), so strict binary verification is necessary for measuring real instruction-following quality."],"supporting_citations":[{"why":"Introduces IFEval, the deterministic verification paradigm for instruction-following constraints that CSE extends to k=1–12.","marker":"Zhou et al. 2023"},{"why":"Introduces COLLIE, the closest prior work using a formal constraint grammar to show composition increases difficulty; CSE extends this with systematic per-k scaling.","marker":"Yao et al. 2024"},{"why":"FollowBench, the prior closest per-k analysis up to k=5, whose sCSR/mCSR metrics and incremental-level design CSE adopts and extends.","marker":"Jiang et al. 2024b"},{"why":"RECAST, which pushes constraint counts to 13+ and motivates the need for per-k trajectories that CSE provides.","marker":"Guo et al. 2025"},{"why":"ConceptMix, the cross-modal text-to-image composition benchmark, provides the external comparison showing the same multiplicative decay pattern.","marker":"Wu et al. 2024"},{"why":"Provides the k-SAT phase transition framework that the paper contrasts with its own interaction-free multiplicative transition.","marker":"Mertens et al. 2006"}],"fun_headline_variants":["Multi-task LLMs: ok one at a time, fail together","Constraint stacking: LLMs pass 41% singly, 5.7% at 8","Five instructions is LLM breaking point","Independent failures sink LLM instruction stacking","LLM constraint collapse: 8 requirements, 5.7% success"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole prediction rests on the assumption that when a model fails one constraint it does not make failing the others much more likely; if that assumption is off, the steep predictable collapse could actually be driven by a few constraints that fail together, not by many independent small failures.","fun_headline_variants_meta":{"raw":{"variants":["Multi-task LLMs: ok one at a time, fail together","Constraint stacking: LLMs pass 41% singly, 5.7% at 8","Five instructions is LLM breaking point","Independent failures sink LLM instruction stacking","LLM constraint collapse: 8 requirements, 5.7% success"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000614,"raw_usage":{"total_tokens":2930,"prompt_tokens":1098,"completion_tokens":1832,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":714,"completion_tokens_details":{"reasoning_tokens":1745}},"tokens_in":714,"tokens_out":1832,"duration_ms":14579,"temperature":1.0,"reasoning_tokens":1745,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:23:23.575106+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute sCSR at k=8 from the reported per-constraint rates: if per-constraint pass rates average 40.7% and successes were independent, the all-eight success rate would be roughly (0.407)^8 = 0.00117, or 0.12%, not the reported 5.7%. A direct check of whether the marginals at k=8 are approximately equal across constraints, and whether their geometric mean is near 40.7% or much higher, would settle whether the independence assumption holds or whether a small set of high-pass-rate constraints is carrying the joint success rate.","supporting_citations":[],"review_version":1}