{"id":"1fbbec9a-8e74-4af1-85b1-ac1388285477","arxiv_id":"2608.12895","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Same-model agent pairs co-fail on 90% of missions where either fails, and a moment-based linear program certifies a reliability floor without any independence assumption.","lead":"This paper shows that two copies of the same AI model often fail together far more often than independence would predict, so multiplying component reliabilities overstates the safety of redundant systems. It then offers a new way to certify a reliability floor without assuming independence.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline co-failure estimate rests entirely on the authors' own deterministic scorer; its construct validity is the least secure link, and the paper's internal controls cannot validate it. Independent scorer or human re-scoring is needed before the 90% co-failure claim is relied upon.","rationale":"The reader's weakest assumption identifies exactly the load-bearing link: the deterministic scoring code must correctly measure contract compliance for the 90% co-failure and the arm contrasts to be valid. I agree, and I would sharpen one point: the paper's argument that scorer bias affects only absolute levels and not differential contrasts is weakened by the design's non-shared missions across arms. Since each arm draws its own missions via identifiers that embed the condition name, a scorer artifact triggered by particular mission features could differentially affect arms, so the SM-DV contrast is not automatically protected. The internal negative control and E5 use the same scorer, so they cannot validate it. However, the concern is testable rather than fatal: the theoretical contributions (Tier-1 LP, anytime-valid e-process, coverage-collapse theorem) are independent of the scorer's validity, and the empirical result is accompanied by released artifacts that permit re-scoring. The reader's CONDITIONAL verdict already accounts for this risk, so I do not recommend moving the verdict; an independent re-scoring check would settle whether the concern actually lands.","tokens_in":42058,"tokens_out":12093,"duration_ms":131499,"concrete_test":"Obtain the per-mission logs (offered on request, Section C.1) for the three series2 primary arms. Have an independent team, provided only with the contract definitions in Section 3 and the mission generators, implement a new deterministic scorer without seeing the released scorer. Compare hard verdicts mission-by-mission (Cohen's kappa) and recompute n11/n10/n01, J, and log OR for each arm, including the same_model and SM-DV contrast. If kappa < 0.9, or if the same_model co-failure rate or the SM-DV log OR contrast moves by more than the reported bootstrap CI width, the headline is scorer-dependent; if the verdicts agree and the numbers reproduce, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—two instances of mistral-small-24b co-fail on 90.0% of missions on which either fails (log OR 6.66, CI [6.38, 7.00])—is produced entirely by the authors' deterministic gold scoring code (Section 10.1). Contract compliance is the operational construct, and the scorer is its only measurement instrument. Section 11.3 concedes that the contracts and scoring code 'share an author.' The paper's defense—that scorer bias would move absolute compliance levels but not differential contrasts—is not fully secure: because mission identifiers embed the condition name and each arm draws its own missions (Sections 10.1, 11.3), a scorer rule that misfires on certain mission types can inflate co-failure counts in one arm more than another, contaminating the SM-DV log OR contrast as well as the 90% figure. The internal negative control (Table 4) and cross-backend replication (E5) both use the same scorer, so neither can validate the instrument. The theoretical certificate (Theorems 5.2, 6.1) is unaffected, but the headline empirical finding, and hence the paper's demonstration that C5 fails in practice, rests on this unvalidated measurement chain.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper tests the conditional-independence condition C5 that licenses multiplicative compositional reliability bounds for multi-agent systems. In a preregistered campaign of 18,000 two-agent handoff missions scored by deterministic code, two instances of mistral-small-24b co-fail on 90.0% of missions on which either fails (log OR = 6.66, 95% CI [6.38, 7.00]); substituting a different model significantly reduces the association in six of six contrasts, while substituting a different vendor, with the model already different, does not. The paper then develops a finite-sample reliability certificate based on a linear program over measured co-execution moments with a Bonferroni-Clopper-Pearson box, proves it sound and sharp, adds an anytime-valid e-process certificate, and proves that bootstrap bounds on a fitted dependence model lose coverage of the true reliability as the sample grows. The evaluation also reports a negative control, cross-backend replication, and an ablation of the i.i.d. assumption. The paper is unusually transparent about its own limitations, including the post-hoc choice of the log odds ratio for H2 and the repository-based rather than external preregistration timestamp.","tokens_in":42253,"tokens_out":10924,"duration_ms":110989,"significance":"If the empirical finding holds, it is significant: it provides controlled evidence that the conditional-independence assumption fails for same-model composition, with a signed, practically important consequence for redundant agent designs. The theoretical machinery is a genuine contribution: the moment-set LP certificate of Theorem 5.2 is a sound finite-sample, copula-agnostic bound, sharp for the supplied moments, and the coverage-collapse result of Theorem 4.2 is a useful warning about fitted-dependence certificates. The paper ships complete proofs, released code, deterministic scoring, and regenerable statistics, which are strengths. The anytime-valid certificate is standard machinery but is applied cleanly and with careful empirical checking. The main risks are empirical rather than theoretical: the headline co-failure numbers depend entirely on the authors' own deterministic scorer, and the substitution manipulation confounds model identity with model capability; both are acknowledged but not resolved.","major_comments":[{"comment":"The headline co-failure estimate (J = 0.9003, log OR = 6.66) and the arm contrasts of Table 3 are produced entirely by the authors' deterministic gold scorer. Because mission identifiers embed the condition name and each arm draws its own missions, a scorer rule that misfires on particular mission or output types can inflate co-failure counts in one arm more than another, so the Section 11.3 defense that the same contracts and scorer are used in all arms does not fully secure the differential claim. The internal negative control of Table 4 holds the compared pair fixed and therefore cannot validate the scorer's behavior across different model identities. An independent scorer or human rescoring of a stratified subsample is needed before the 90% co-failure finding is relied on.","section":"Sections 10.1, 10.2.1, 11.3"},{"comment":"The manipulation confounds model identity with model capability: the same_vendor substitution uses ministral-8b, a weaker model, and the different_vendor arm uses gemma-3-12b-it. The six significant same_model-versus-substitution contrasts, and the Section 10.9 conclusion that the operative variable is the model not the vendor, could in part be driven by capability-correlated failure modes rather than by shared model weights. The marginal-free statistics remove the effect of different marginal failure rates but not the effect of capability-correlated failure modes, and the negative control does not address this because the control pair is never substituted. A design with models matched on failure rate, or a direct manipulation of shared weights, is required to support the causal attribution in the abstract.","section":"Sections 10.1, 10.2.2, 11.3"}],"minor_comments":[{"comment":"The quantity 2(p11p00 − p10p01) is labelled Kendall's τa, but the conventional sample tau-a for binary variables is a factor of 2 larger (approximately 4(p11p00 − p10p01) with a finite-population correction). Please state the scaling convention explicitly, since the registered H1 threshold is stated on τa.","section":"Equation (25), Definition 3.17"},{"comment":"The phrase 'a registered hypothesis reported as a null' for the same_vendor versus different_vendor comparison should be qualified: the registered H2 was a three-level ordering on τa, and the vendor-level comparison is a post-hoc decomposition of that ordering. The paper's Section 11.3 disclosure is accurate, but the abstract's wording invites the reading that the vendor null was itself preregistered.","section":"Abstract and Section 10.2.2"},{"comment":"The preregistration timestamp rests on repository history rather than on an external registry. The paper discloses this, but given the confirmatory role of the registration, an independent timestamp or third-party registration would materially strengthen the claim.","section":"Section C.4"},{"comment":"The verdict label 'conflict' in the SV−DV rows is not defined in the caption; please state that it means the marginal-sensitive and marginal-free statistics disagree in sign or significance, as explained in the text.","section":"Table 3 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper is transparent about its v1 lineage and the withdrawn patent claim, and it re-states the v1 formulas rather than merely citing them; I do not see a novelty-disclosure problem. The theoretical results appear sound. The main obstacle to acceptance is the unvalidated deterministic scorer: since both the 90% co-failure headline and the arm contrasts depend on that instrument, I would want an independent scoring or human rescoring check before the empirical claim is treated as established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this paper is worth a serious referee and probably publication after revision. The headline finding—two instances of mistral-small-24b co-fail on 90% of missions where either fails, with a preregistered 18,000-mission experiment and deterministic scoring—is about as well-supported as a single-paper result gets, with one genuine soft spot. The theoretical work around it is original and, as far as I checked, correct.\n\nWhat is actually new: the preregistered manipulation of model sharing at three levels (same model, same vendor, different vendor) is the first controlled test of the conditional-independence assumption that compositional reliability bounds silently use. The moment-set LP certificate (Theorem 5.2) is a clever, sound way to get a finite-sample lower bound without assuming a copula, and the coverage-collapse theorem (Theorem 4.2) makes a serious point: fitting a dependence model and bootstrapping it can give you a certificate that is confidently wrong, and more data makes it worse. The anytime-valid e-process application is also clean. All of this is backed by released artifacts, machine-checkable scripts, and unusually candid disclosure of nulls, post-hoc choices, and limitations.\n\nThe soft spots are real but not fatal, and the paper mostly names them itself. The biggest is the scorer. The 90% figure rests entirely on the authors' own deterministic gold code, and since the contracts and scorer 'share an author' (Section 11.3), construct validity is the weakest link. The internal negative control is good, but it uses the same scorer, so it cannot validate the instrument. That said, the tasks are narrow and objective—order arithmetic, refund policies, watchlist checks—so the risk of systematic differential bias across arms is lower than the stress-test note implies, but an independent re-scoring on a sample would settle it. The H2 statistic was chosen after seeing the data (disclosed), and the preregistration timestamp depends on repo history rather than an external registry. Both are minor-to-moderate, and the paper's transparency about them earns credit.\n\nWho is this for? Anyone building multi-agent reliability evaluations, runtime guardrails, or compositional guarantees. It would be a strong reading-group paper. My own verdict: the central claim holds up, the theory is sound, and the artifacts are reproducible. I would send it to peer review, and in the review I'd ask for an external scorer audit and a clearer statement of what the certificate does not cover.","headline":"A serious, unusually transparent empirical and theoretical package on correlated failures in multi-agent systems; the 90% co-failure finding is solid modulo one legitimate scorer-validity concern, and the moment-set certificate is a real contribution.","tokens_in":42833,"tokens_out":1619,"would_cite":true,"duration_ms":18742,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62N05","90C05","62F40","60G42"],"pacs":[],"model":"deepseek-v4-flash","headline":"Two instances of one model co-fail on 90.0% of missions where either fails, so the independence assumption behind compositional reliability bounds gives way, and the paper responds with a certificate that assumes no dependence structure.","keywords":["multi-agent reliability","correlated failure","conditional independence","compositional bounds","redundancy over-credit","moment-set certificate","linear programming","anytime-valid inference"],"falsifier":"Rescore a random sample of the confirmatory missions with a contract set and scoring code written independently of the paper's authors, and compare the same-model co-failure rate of 90.0% and the log-odds contrasts; if the co-failure pattern moves materially the headline is a scoring artifact, while reproduction would confirm genuine model dependence. A sharper variant runs the substitution with two models matched on marginal failure rate but from different lineages, separating shared competence from shared inductive bias.","tokens_in":41813,"feed_emoji":"⚠️","tokens_out":17120,"duration_ms":129323,"temperature":0.7,"pith_summary":"Compositional reliability bounds for multi-agent systems multiply component reliabilities, a step licensed by a conditional-independence assumption that is routinely stated and rarely tested. This paper tests it and finds it false where it matters most: two instances of one model, composed in a two-agent handoff, co-fail on 90.0% of the missions on which either fails, in a preregistered evaluation of 18,000 missions scored by deterministic code with no LLM judge. The error is signed and runs against the operator: positive dependence inflates joint failure above the independence product, so redundant paths are over-credited exactly when their components share a model. Dropping the assumption leaves a vacuous bound, and fitting a dependence model is provably worse, so the paper constructs a finite-sample certificate that assumes no dependence structure: a linear program over the joint distribution constrained by measured co-execution moments, sound and sharp for the information supplied, and monotone in the moment family. A careful reader should care because standard compositional certification is quietly wrong in the redundancy case, and this paper replaces it with a guarantee that degrades honestly.","feed_headline":"Agent pairs from one model co-fail on 90% of missions","feed_subtitle":"Tests on 18,000 missions show independence-based reliability bounds over-credit same-model redundancy.","key_machinery":"Two objects carry the argument. Diagnostically, the signed compositional gap: for two components, the true joint failure probability exceeds the independence product by exactly $Cov(h_1, h_2)$, the covariance of the two hard verdicts, so positive dependence always over-credits redundancy and the direction of the error is fixed without further estimation. Constructively, the moment-set certificate: a linear program over the $2^m$ cells of the joint law on $\\{0,1\\}^m$, minimizing the all-success probability subject to the constraint that measured co-execution moments — single-stage successes, pairwise co-successes, and optionally triple co-successes — lie inside a Bonferroni–Clopper–Pearson box around their empirical values; the box covers by the union bound, the true law is feasible whenever it covers, and the LP minimum is the certified floor. A companion betting e-process, $E_R = \\prod_{r \\le R}\\bigl(1 + \\lambda_r (y_r - p_0)\\bigr)$, gives an anytime-valid certificate whose null constrains only a conditional mean, so it needs no independence assumption at all.","core_discovery":"The paper's claim is that the conditional-independence condition C5 of compositional contract theory fails for same-model composition, that the failure is large and signed, and that it can be certified around without any dependence assumption. In the preregistered confirmatory arm, two instances of mistral-small-24b in a two-agent handoff co-fail on 2177 of the 2418 missions on which either fails — 90.0%, against the 14.6% the independence product predicts — with log odds ratio 6.66 (95% CI [6.38, 7.00]) and $\\phi = 0.916$. Substituting a different model into the second agent reduces the association significantly in six of six contrasts across three topologies; substituting a different vendor, with the model already different, does not, a registered hypothesis reported as a null. The paper further shows that the assumption-free Fréchet–Hoeffding sandwich, which brackets a joint probability from marginals alone, is vacuous — its certified floor is zero whenever mean component reliability falls below $1 - 1/m$ — and that a certificate built on a fitted dependence model loses coverage of the truth as the sample grows, because the identification gap is $O(1)$ while the bootstrap haircut is $O(n^{-1/2})$. Its remedy is a linear program over the joint law on $\\{0,1\\}^m$, minimized over all distributions whose co-execution moments lie in a Bonferroni–Clopper–Pearson box: valid with no dependence assumption, sharp for the moments supplied, and monotone in the moment family, and on four-stage data enriching ten moment functionals to fourteen narrows the identified interval by 85.7% and lifts the certified floor from 0.2455 to 0.4116.","pith_inferences":["If the co-failure rate transfers beyond the retail and financial task domains tested, any safety case built on same-model redundant review is effectively relying on a single point of failure: a reviewer and a writer drawn from one model supply far less independent evidence than the design assumes, and the honest redundancy count may be one rather than two.","The coverage-collapse theorem is not specific to agents: any bootstrap interval built on a misspecified parametric model with a fixed identification gap will, past some sample size, sit entirely off the truth, which cautions against model-based certificates in copula-based risk and reliability practice generally.","Because the anytime-valid e-process needs no independence assumption, it is the one certificate in the paper that survives the failure the paper documents; a natural deployment pattern is a continuously re-earned certificate that a team may watch and stop on at bounded type-I cost.","A testable extension suggested by the paper's own limitation section: run the substitution design with two models matched on marginal failure rate but from different lineages, which would separate the contribution of shared competence from shared inductive bias to the co-failure signal."],"forward_implications":["At the measured dependence the independence product over-credits redundancy by a wide margin: the two agents would co-fail on 14.6% of missions if independent and actually co-fail on 36.3%, so a dashboard that multiplies reliabilities reports a reassuring number whose governing assumption the data reject.","Model identity, not vendor identity, is the axis on which to choose redundancy: substituting a different model reduced the association significantly in six of six contrasts across three topologies, while substituting a different vendor, with the model already different, produced no consistent reduction.","A certificate built on a fitted dependence model is worse than no certificate: its bootstrap lower bound loses coverage of the true reliability as the sample grows, so more data narrows an interval around the wrong target with no visible symptom.","Operators can tighten the certified floor without any dependence assumption by measuring more co-execution moments: on four-stage data, enriching ten moment functionals to fourteen narrowed the identified interval by 85.7% and lifted the floor from 0.2455 to 0.4116, and under pre-allocated Bonferroni spending the tightening is monotone by construction.","Marginal-bounded dependence statistics — Jaccard, $\\phi$, Kendall's $\\tau_a$ — can reverse the apparent ordering of conditions when the compared agents fail at different rates, so a marginal-free statistic such as the log odds ratio should be reported alongside them."],"supporting_citations":[{"why":"Supplies the v1 Agent Behavioral Contract framework, including the conditional-independence condition C5 under test, and the single-agent evidence carried forward.","marker":"Bhardwaj, 2026"},{"why":"Quantifies correlated failure in a multi-agent triage ensemble; the phenomenon this paper turns into a manipulated-variable contrast.","marker":"McDonnell et al., 2026"},{"why":"Gives the sharp sandwich bounds that make the assumption-free certificate vacuous in the regime of interest.","marker":"Fréchet, 1951; Hoeffding, 1940"},{"why":"Supplies the linear-programming view of extremal joint probabilities that the moment-set certificate instantiates.","marker":"Boole, 1854; Hailperin, 1965"},{"why":"Provides the exact binomial intervals that form the Bonferroni moment box on which Theorem 5.2 rests.","marker":"Clopper and Pearson, 1934"},{"why":"Contributes the moment-problem bounding machinery the LP certificate applies to agent pipelines.","marker":"Bertsimas and Popescu, 2005"},{"why":"Is the inequality behind the anytime-valid certificate, bounding the crossing probability of the e-process.","marker":"Ville, 1939"},{"why":"Grounds the preregistration practice that gives the confirmatory co-failure measurement its evidential standing.","marker":"Nosek et al., 2018"}],"fun_headline_variants":["Same-model agents co-fail 90% of the time, breaking independence bounds","Independence assumption fails: same-model agents co-fail 90%","90% co-failure for same-model agents: independence-based reliability over-credited","Test of 18,000 missions: same-model agents co-fail 90%","Same-model reliability certificates over-credit: 90% co-failure observed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline result stands on the deterministic gold scoring code measuring contract compliance correctly, even though the same authors wrote both the contracts and the scorer; a systematic scoring bias would make the co-failure counts and arm contrasts artifacts rather than true model dependence.","fun_headline_variants_meta":{"raw":{"variants":["Same-model agents co-fail 90% of the time, breaking independence bounds","Independence assumption fails: same-model agents co-fail 90%","90% co-failure for same-model agents: independence-based reliability over-credited","Test of 18,000 missions: same-model agents co-fail 90%","Same-model reliability certificates over-credit: 90% co-failure observed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000545,"raw_usage":{"total_tokens":2782,"prompt_tokens":1296,"completion_tokens":1486,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":912,"completion_tokens_details":{"reasoning_tokens":1382}},"tokens_in":912,"tokens_out":1486,"duration_ms":10753,"temperature":1.0,"reasoning_tokens":1382,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:03:59.353760+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rescore a random sample of the confirmatory missions with a contract set and scoring code written independently of the paper's authors, and compare the same-model co-failure rate of 90.0% and the log-odds contrasts; if the co-failure pattern moves materially the headline is a scoring artifact, while reproduction would confirm genuine model dependence. A sharper variant runs the substitution with two models matched on marginal failure rate but from different lineages, separating shared competence from shared inductive bias.","supporting_citations":[{"cited_title":"Harnessing Disagreement: Detecting Correlated Agreement Blindness in Multi-Agent Triage","cited_arxiv_id":"2607.19899","evidence_quote":"Quantifies correlated failure in a multi-agent triage ensemble; the phenomenon this paper turns into a manipulated-variable contrast."},{"cited_title":"Optimal inequalities in probability theory: A convex optimization approach","cited_arxiv_id":null,"evidence_quote":"Contributes the moment-problem bounding machinery the LP certificate applies to agent pipelines."},{"cited_title":"\\'Etude critique de la notion de collectif","cited_arxiv_id":null,"evidence_quote":"Is the inequality behind the anytime-valid certificate, bounding the crossing probability of the e-process."}],"review_version":1}