{"id":"7c553f18-dbdd-46d8-bf66-182183709362","arxiv_id":"2607.04667","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Software assurance should shift from formal correctness to graded confidence in fitness for purpose, recorded as credentials with credibility and provenance.","lead":"A software-engineering keynote argues that \"correct\" is the wrong target; the real goal should be confidence that software is fit for its purpose, with evidence graded by credibility and provenance. It proposes a credential-based framework to help engineers choose cost-effective assurance methods, especially for AI-generated code.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central premise conflates statistical foundations with unsoundness; the asserted inherent limitation of AI to probabilistic, low-rigor assurance is undemonstrated and contradicted by rigorous probabilistic frameworks such as PAC learning and conformal prediction.","rationale":"The paper is a position/keynote, so I evaluate whether its central diagnosis is supported. The strongest claim, per the reader, is that AI's statistical foundation does not permit rigorous reasoning. That assertion does heavy lifting: it separates 'traditional' assurance from AI, motivates the V-model break, and supports moving to confidence/credentials. I looked for the point where this condition is least secure. The tacit-context premise (reader's choice) is also asserted anecdotally, but it is at least empirically plausible and could be refined. The statistical-unsoundness claim is more clearly problematic because the paper itself cites PAC learning, which is a rigorous probabilistic framework. The text calls ML 'probably approximately correct' as if that precludes rigor, but PAC bounds are formal; the field of probabilistic verification exists. No counterargument or definition of 'rigorous reasoning' is provided. This is not merely a disagreement with consensus; it is an internal inconsistency: the paper's own reference undercuts its dichotomy. My proposed check—whether conformal prediction can give finite-sample guarantees for LLM outputs—would empirically and formally test whether statistical foundations can support rigorous assurance. If it succeeds, the central premise needs revision; the reframing may still stand, but not for the reason claimed. Thus I keep the reader's CONDITIONAL verdict (unchanged), with the specific caveat that §4's 'unsound, statistical basis' must be either justified or dropped.","tokens_in":14958,"tokens_out":5455,"duration_ms":62338,"concrete_test":"Apply conformal prediction to an LLM code-generation task. Given a set of natural-language programming problems, generate candidate programs with an LLM and use a held-out calibration set to construct a prediction set with a finite-sample coverage guarantee (e.g., 95% probability that a correct program is among the candidates). If such a guarantee can be established and empirically verified, the statistical basis of LLM outputs can support rigorous probabilistic assertions, directly contradicting the claim that 'the statistical foundation of machine learning is unsound; it does not permit rigorous reasoning.' This tests whether the paper's central premise is a fact or an assertion.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that generative AI's statistical basis 'does not permit rigorous reasoning' and 'inherently limits assurances about the results to probabilistic assertions' (Abstract, §4)—is load-bearing: it grounds the proposed separation of AI from classical assurance and motivates the reframing from correctness to confidence. The claim is asserted, not demonstrated, and as stated is untenable. In §4 ('Unsound, statistical basis'), the paper equates 'statistical' with 'unsound' and asserts that ML 'cannot support the hard assurances needed by the safety engineering process.' But statistical reasoning can be rigorous: Valiant's PAC framework (cited as [36]) is a formal mathematical theory providing quantifiable, high-confidence error bounds; randomized algorithms and probabilistic model checking are routine in SE; and modern ML offers conformal prediction and PAC-Bayes bounds with finite-sample guarantees. Unless the paper defines a narrower, unconventional sense of 'rigorous reasoning' that excludes probabilistic guarantees—and explains why such guarantees do not count—the premise fails. The contrast between 'rigorous reasoning' and 'probably approximately correct' sets up a false dichotomy. This matters because much of the paper's 'AI age' urgency (e.g., restricting AI to low-rigor assurance, the V-model break) depends on the claimed inherent statistical limitation, not merely on cost or tacit knowledge. If probabilistic assurances can be rigorous, AI-generated components could be placed on the same formal footing as other probabilistic systems, undercutting the conclusion that a fundamental reframing is required.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"Shaw argues that software engineering's traditional ideal of formal correctness is not only practically unattainable but conceptually insufficient, because fitness for purpose depends on tacit and implicit context that escapes formalization and digital records. She extends this to generative AI: LLMs are trained on 'digital shadows' and their statistical, 'probably approximately correct' basis cannot support rigorous assurance. The paper proposes to replace the correctness mindset with confidence in fitness for purpose, using 'credentials'—tuples of attribute, value, credibility, provenance—to make evidence graded and comparable, and a 'field guide' to assurance techniques that would support cost-effective selection. The manuscript is a keynote narrative, explicitly leaving the field guide and optimization as open challenges.","tokens_in":15272,"tokens_out":8753,"duration_ms":104106,"significance":"The paper's value is its clear articulation of a shift already occurring in practice: from verification-oriented correctness to risk- and evidence-based fitness-for-purpose. The examples are effective; the credentials idea offers a compact representation for heterogeneous assurance; and the call for a comparative 'field guide' is a useful research agenda. If the strong claims about AI's inherent limitations are appropriately qualified, the paper could serve as a generative position statement. In its current form, however, the central technological premise is asserted rather than demonstrated, and that premise carries much of the argument's urgency.","major_comments":[{"comment":"The load-bearing premise is stated in the Abstract and in §4 ('Unsound, statistical basis'): generative AI's statistical foundation 'does not permit rigorous reasoning' and 'inherently limits assurances... to probabilistic assertions.' This is not demonstrated, and as stated it is difficult to defend. PAC learning—the exact source of the phrase 'probably approximately correct' [36]—is a rigorous mathematical framework with finite-sample error bounds. Probabilistic model checking, randomized algorithms, and conformal prediction are accepted forms of rigorous probabilistic assurance. If the intended claim is narrower (e.g., that current LLM decoding does not provide such guarantees), the paper must say so explicitly and explain why PAC-style guarantees cannot be extended to LLMs. As written, the dichotomy between 'rigorous reasoning' and 'probabilistic assertions' carries the conclusion th","section":"Abstract and §4, Figure 10"},{"comment":"The 'digital shadows' argument treats inaccessibility of tacit/implicit context as an inherent limitation of LLMs. But the paper concedes that implicit context is in principle inferable and cites Unblocked [35] as an attempt to reconstruct exactly this context. The examples (sunflower associations, cargo cults, chemical plant operators) are evocative but do not establish that assurance-relevant tacit knowledge is absent from LLM training data or cannot be recovered. The paper should recast this as an empirical/architectural limitation—e.g., measured performance on tasks requiring implicit design rationale—or give a formal account of why context cannot be represented. Without this, Figure 7's red 'lost information' is an assertion, not a result.","section":"§3, Figure 7"},{"comment":"The constructive framework in §5–§6 is a promise, not yet a specification. The credential tuple is described with one example, but no semantics are given for credibility values, provenance types, or the combination operation ('propagate new values' is mentioned without rules). The field guide is a list of attributes, and the optimization is left as an open question. If these are intended as the paper's contribution, at least one instantiation—even synthetic—should show how the tuple fields and the selection mechanism would be used together. As it stands, the central reframing is supported by arguments from practice, but the allegedly enabling machinery remains a research agenda.","section":"§5–§6"}],"minor_comments":[{"comment":"The text citing Figure 2 uses '[39]', but [39] is an image credit, not the StackOverflow survey; the intended source appears to be [40]/[28].","section":"Figure 2 / references"},{"comment":"Image credit [43] says Figure 5/6b are 'from [4]', but [4] is the CrowdStrike RCA; the likely intended source is [6] (Evans 1917).","section":"Image credits"},{"comment":"'peoples' heads' should be 'people's heads'.","section":"§3 heading"},{"comment":"The term 'unsound' has a precise meaning in logic; the statistical basis of an ML model is not itself an argument. Consider 'non-deductive' or 'probabilistic' to avoid loading the term.","section":"§4 heading"},{"comment":"Credentials would benefit from positioning relative to existing assurance case notations (e.g., GSN) and evidence-based software engineering; currently the link to systematic reviews [17] is the only bridge.","section":"§5"},{"comment":"The axes 'consequences of failure' and 'degree of automation' are intuitive but not defined; a reader cannot operationalize where the 'good enough vs correctness' boundary lies.","section":"§2, Figure 4"},{"comment":"The paper uses 'LLM' and 'machine learning' interchangeably. The statistical-basis argument applies to ML broadly, but the context/digital-shadows argument is specific to LLMs; clarify the scope of each claim.","section":"Throughout"},{"comment":"Important support is given by unpublished or non-archival sources ([24], [31], [34]). For a peer-reviewed venue, accessible versions or independent corroboration would strengthen the argument.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a keynote position paper, so a full empirical or formal validation may not be required. The statistical-limitation claim, however, is a correctness risk: it is load-bearing for the AI-specific conclusions and is currently an overstatement. The paper can be repaired by reframing the claim as an empirical/architectural limitation of current LLMs rather than an in-principle impossibility, and by explicitly allowing rigorous probabilistic assurance (PAC, conformal, probabilistic model checking) as a target. That revision would make the reframing substantially stronger. The heavy reliance on the author's own and unpublished work is acceptable for a keynote but should be noted if the paper is submitted as a research contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read on Shaw's FSE keynote. It's a classic Mary Shaw position piece: readable, full of good examples, and it pushes the community to think systematically about assurance. The genuinely valuable move is to reframe 'correctness' as confidence in fitness for purpose, and to suggest credentials—attribute-value-credibility-provenance tuples—as a way to record graded evidence. That is a practical idea, and the call for a field guide to assurance techniques is a constructive challenge. The paper also does well to emphasize tacit context and digital shadows; the sunflower and cargo cult examples are memorable and make the point that much relevant knowledge never appears in the training corpus.\n\nThe soft spot is the central claim in Section 4 that the 'statistical basis of machine learning is unsound' and that it 'does not permit rigorous reasoning.' That is asserted, not demonstrated. Statistical reasoning can be rigorous—PAC learning, conformal prediction, and probabilistic model checking give formal guarantees. The paper sets up a false contrast between symbolic rigor and probability. If the author intended a narrower claim—that LLM outputs today do not come with usable formal guarantees—that is more defensible, but it is not what the abstract says. That matters because the paper uses this 'inherent limitation' to justify separating AI from classical assurance. However, the reframing does not actually depend on that claim. The tacit-knowledge argument and the cost-aware view of assurance are enough to motivate confidence-based thinking, even if AI's statistical basis were perfectly rigorous.\n\nThe credentials idea is not new—it comes from the author's own prior work—but it is clearly re-presented here, with a concrete latency example. It is a sketch, not a validated framework. There is no instantiation or evaluation, and the paper acknowledges the field guide is an open problem. For a keynote, that is fine.\n\nOverall: a paper for the community to argue with, not a technical result. It deserves peer review because it will shape discussion, but a referee should push back hard on the 'unsound' language. I'd bring it to a reading group and would cite it for the credentials framing.","headline":"A senior voice worth hearing, but its key claim about AI's 'unsound' statistical basis is overstated; the reframing to confidence and credentials stands on its own.","tokens_in":15778,"tokens_out":2895,"would_cite":true,"duration_ms":32673,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Generative AI's statistical foundation forecloses formal correctness, so software assurance should be reframed as graded confidence in fitness for purpose, with each property's value, credibility, and provenance recorded.","keywords":["software correctness","software assurance","fitness for purpose","sufficient correctness","tacit context","digital shadows","software credentials","limitations of AI"],"falsifier":"A controlled study in which engineers write down the unrecorded rationale behind critical design decisions, then an LLM is given only the code and documentation and asked to reconstruct that rationale; if the LLM faithfully recovers a substantial fraction of the decisions, the 'scant shadows' premise collapses. A second test: a requirement that mentions a symbol with culture-specific meaning (e.g., 'sunflower') with no explicit context, and an LLM prompt that includes the client's background; if the LLM reliably selects the culturally relevant association, then some tacit knowledge is recovera","tokens_in":14808,"feed_emoji":"🧾","tokens_out":7304,"duration_ms":74426,"temperature":0.7,"pith_summary":"The paper argues that software engineering's long-standing goal of formal correctness is unattainable for modern software and especially for AI-influenced systems, because the context that actually determines fitness for purpose is mostly tacit and undocumented. Generative AI is built on a statistical basis that yields 'probably approximately correct' results, so it can support only probabilistic assurances, not rigorous proof. The paper proposes reframing assurance as confidence in fitness for purpose, and introduces 'credentials': collections of attribute-value pairs augmented with credibility and provenance. It calls for an assurance plan per project, a field guide to assurance techniques, and a cost-benefit choice among them. If the argument holds, the community should stop equating good software with proven software and instead engineer assurance deliberately.","feed_headline":"AI forces software assurance to trade correctness for confidence","feed_subtitle":"Grade each property's credibility and provenance, not chase a single proof of correctness.","key_machinery":"The credential: a set of four-tuples { <attribute, value, credibility, provenance> } that replaces monolithic specifications with a graded, provenance-tagged record of each property of interest. The associated 'fitness for purpose landscape' maps the world-machine distinction, and the 'digital shadow' concept explains why AI only sees incomplete traces. The credential is what carries the argument: it makes 'confidence' a concrete, inspectable artifact and enables the proposed optimization of assurance resources.","core_discovery":"On the paper's own terms, the central claim is that the 'correctness' mindset that still shapes software engineering—the idea that enough information and rigor would let us reason our way to correct software—is no longer viable, and generative AI makes that impossibility explicit. Because LLMs are trained only on 'digital shadows' of reality, the tacit and implicit context that encodes design intent, domain expertise, and cultural norms is mostly lost to them; because their predictions are statistical, they cannot support the hard, rigorous assurances that formal verification promises. The paper therefore redefines the assurance task as establishing confidence that a system is fit for its pu","pith_inferences":["The credential model is a natural fit for 'nutrition-label' style reporting on AI systems, where users see not just a score but the source and certainty behind it; this could be developed into a standard exchange format.","If LLMs continue to improve, some knowledge the paper classifies as tacit may become inferable from larger corpora; the framework's boundary would then shift, suggesting the paper's inherent-limitation claim is a technological transition rather than a permanent epistemic ceiling.","The paper's framing implies a research agenda for cost models: measuring the marginal confidence gained per dollar spent on each assurance technique, which no existing SE literature systematically provides.","One could test the 'digital shadow' premise empirically by constructing a corpus of projects with known but undocumented design rationale and measuring how much of it an LLM can reconstruct from code alone."],"forward_implications":["Each project would begin with an assurance plan that specifies the important properties and the required level of confidence for each, rather than aiming at blanket correctness.","Credentials make it possible to combine evidence of different rigor—formal proof, empirical tests, expert judgment—with explicit provenance and credibility, so a portfolio of assurance techniques can be compared on the same footing.","Assurance choices become a deliberative resource-allocation problem: either maximize confidence for available resources or compute the resources needed to reach required confidence.","A community-maintained 'field guide' of assurance techniques, with uniform descriptions of properties, cost, coverage, and rigor, becomes a prerequisite for principled tradeoffs.","AI-generated code and AI-supported tools would be subject to the same confidence/provenance regime as any other software component."],"fun_headline_variants":["AI reveals why software correctness was always a fiction","From proof to probability: software assurance in the AI era","Generative AI makes software confidence a statistical bet","Tacit knowledge's digital shadow limits AI-based assurance","Software engineers must accept confidence over certainty"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The argument depends on the premise that the tacit context, design intent, and cultural knowledge that shape software leave only scant traces in the digital record, so AI models trained on that record cannot recover them.","fun_headline_variants_meta":{"raw":{"variants":["AI reveals why software correctness was always a fiction","From proof to probability: software assurance in the AI era","Generative AI makes software confidence a statistical bet","Tacit knowledge's digital shadow limits AI-based assurance","Software engineers must accept confidence over certainty"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1286,"prompt_tokens":803,"completion_tokens":483,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":410}},"tokens_in":547,"tokens_out":483,"duration_ms":5964,"temperature":1.0,"reasoning_tokens":410,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T08:33:16.584751+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled study in which engineers write down the unrecorded rationale behind critical design decisions, then an LLM is given only the code and documentation and asked to reconstruct that rationale; if the LLM faithfully recovers a substantial fraction of the decisions, the 'scant shadows' premise collapses. A second test: a requirement that mentions a symbol with culture-specific meaning (e.g., 'sunflower') with no explicit context, and an LLM prompt that includes the client's background; if the LLM reliably selects the culturally relevant association, then some tacit knowledge is recovera","supporting_citations":[],"review_version":2}