{"id":"aa43e555-5ef5-49bc-9b14-fa2fb720b931","arxiv_id":"2607.05985","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":7,"one_line_summary":"A black-box evaluation framework with multi-run metrics reveals that LLM-based DSM generation is sensitive to phrasing, exhibits hallucination, and fails to consistently acknowledge knowledge boundaries.","lead":"This paper builds a black-box testing framework to evaluate how well large language models generate Design Structure Matrices (DSMs) from technical documents. It matters because it exposes systematic failure modes—hallucinated dependencies, prompt sensitivity, and inconsistent abstention—in a tool intended for engineering design automation.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The refrigerator experiments suffer from a deeper circularity than the reader notes: GPT-4o generates the ground truth, the input documents, and the model under test, making accuracy metrics a self-consistency check rather than an independent benchmark.","rationale":"The reader correctly identified the most load-bearing concern. The circularity is real, acknowledged by the authors (Section VI-A), and affects the validity of accuracy metrics on the refrigerator dataset. I agree with the reader's CONDITIONAL verdict because: (1) the concern is genuine and unresolved—the paper acknowledges it but defers to future work; (2) the Q score weights are ad hoc without sensitivity analysis, which compounds the problem since Q is used for cross-experiment comparison; (3) no significance testing means comparative claims rest on overlapping confidence intervals. However, the verdict should remain CONDITIONAL rather than REJECT because the paper makes honest, appropriately hedged claims, provides public code and data, and the abstract system experiments offer non-circular evidence for the key behavioral findings (hallucination patterns, prompt sensitivity, sequence effects). The framework itself is a legitimate methodological contribution regardless of the ground-truth issue. The three conditions the reader identified—independent GT validation, Q sensitivity analysis, and significance testing—are the right ones to require before upgrading the verdict. I note that the circularity is slightly more pervasive than the reader described (input documents are also GPT-4o-generated, not just the GT), but this does not change the verdict level.","tokens_in":32298,"tokens_out":3051,"duration_ms":280046,"concrete_test":"Have an independent systems engineer (not using GPT-4o) construct a ground-truth DSM for the refrigerator or a comparable real-world system from first principles, and independently write the input technical documentation. Re-run the Auto-DSM pipeline and recompute Selective Accuracy and Q scores. If the accuracy metrics shift by more than 15 percentage points relative to Table VII/VIII values, the circularity concern lands and the refrigerator-based claims need qualification. If metrics remain stable, the findings are robust to the circularity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader correctly identifies the ground-truth circularity but understates its scope. Section IV-C.2.a states the GT-DSM is generated by ChatGPT (GPT-4o) and then 'manually validated for correctness and symmetry.' Section IV-C.2.b states the input technical documents are also generated by ChatGPT from the GT dataset via prompt engineering. Auto-DSM itself runs on an LLM (Koh's pipeline uses OpenAI models). So for all refrigerator experiments, the same model family produces the reference answer, the test input, and the system output. This means Selective Accuracy, Q scores, and match frequencies on the refrigerator dataset measure GPT-4o's self-consistency across different prompt roles, not correctness against an independent engineering standard. The manual validation step ('checked for correctness and symmetry') mitigates this, but the paper does not describe the validation protocol, the expertise of the validators, or whether every dependency was independently verified against physical/engineering first principles. If the validators accepted GPT-4o's decomposition as plausible without independent derivation, the circularity persists through the validation step. This concern is load-bearing because the paper's central claim—providing 'the first reproducible evidence of Auto-DSM's constraints'—rests on the accuracy metrics being meaningful. However, it does not fully undermine the paper: the abstract system experiments (Section V-A) use manually constructed ground truth and still demonstrate the key behavioral findings (directional asymmetry, sequence effects, confident misclassification under ambiguity). The circularity specifically weakens the refrigerator-based accuracy claims and the complexity comparisons in Section V-C, not the qualitative behavioral findings.","agreement_with_reader":"agree"},"referee_report":{"model":"glm-5.2","summary":"This paper presents a black-box evaluation framework for assessing LLM-based Design Structure Matrix (DSM) generation, targeting the Auto-DSM pipeline. The framework combines single-run structural and classification metrics (Completeness, Correctness, NZF, Selective Accuracy) with multi-run stability measures (entropy, Fleiss' kappa) and a Composite Quality Score (Q). Controlled experiments on a synthetic abstract system and a refrigerator decomposition examine sensitivity to phrasing, parameter-dataset alignment, and system complexity. The authors find that LLMs produce structurally plausible DSMs under well-structured inputs but exhibit systematic hallucination, overgeneralization, and abstention failure when semantic constraints are weak. The framework and datasets are publicly available.","tokens_in":32489,"tokens_out":1522,"duration_ms":382071,"significance":"The paper addresses a genuine gap: the only prior LLM-based DSM work (Koh [17]) reported point metrics without distributional analysis or controlled datasets. The multi-run evaluation design with N=30, adaptive sampling, and per-cell entropy decomposition is a methodological contribution. The public code and dataset release is a strength. The identification of specific failure modes (first-entity bias, sequence effects in contradictory inputs, overgeneralization under shared dependency types) provides actionable guidance for future work. The Composite Quality Score, while involving ad-hoc weights, is transparently defined and normalized.","major_comments":[{"comment":"§IV-C.2a: The ground-truth DSM for the refrigerator dataset is generated by GPT-4o and then 'manually validated for correctness and symmetry.' The input technical documents are also generated by GPT-4o from the GT dataset (§IV-C.2b). Since Auto-DSM itself runs on OpenAI models, the same model family produces the reference answer, the test input, and the system output. The authors acknowledge this circularity in §VI-A, but the validation protocol is not described: how many validators, what expertise, and whether every dependency was independently verified against engineering first principles. This is load-bearing because the paper's central claim of providing 'the first reproducible evidence of Auto-DSM's constraints' rests on accuracy metrics being meaningful. The abstract-system experiments (§V-A) use manually constructed ground truth and are less affected, but the refrigerator results—","section":null},{"comment":"§III-B.2h, Eq. (14): The Composite Quality Score Q_rc = w_acc * SA_rc + w_stab * (1 - H_rc) - w_pen * P_rc uses weights w_acc=0.5, w_stab=0.3, w_pen=0.2. These are stated to 'reflect engineering priorities' but no sensitivity analysis is provided. Since Q is used to rank and compare experiments throughout §V (e.g., Table VII, Table VIII), the reader cannot assess whether conclusions are robust to alternative weightings. A simple sensitivity check (e.g., ±0.1 on each weight) would address this.","section":null},{"comment":"§III-B.2f, Eq. (12): Selective Accuracy per cell is defined as SA_rc = M_rc / (1 - U_rc). When U_rc = 1 (all runs abstain), this is undefined. The paper does not state how this case is handled. Table VII reports DSM-wide SA means and standard deviations; if any cell has U_rc=1, the aggregation rule matters. Clarify the convention (e.g., SA_rc := 0 or excluded from the mean).","section":null},{"comment":"§V-C.3, Table IX: The multi-subsystem experiment shows that Auto-DSM extracts components for both subsystems but 'stored them as two lists, after which only the first list was used for dependency-type identification.' This is presented as a finding about the LLM, but it appears to be a finding about the Auto-DSM pipeline's internal processing logic. Since the framework is black-box, how was this internal behavior diagnosed? The claim should be qualified or the diagnostic method described.","section":null}],"minor_comments":[{"comment":"§II-B, Table I: The comparison between Koh's reported elements and empirical results is confusing because the two columns list entirely different components. A brief note explaining why the empirical results differ so drastically (different system parameter? different pipeline version?) would help.","section":null},{"comment":"§III-B.2b: The sample-size policy states N=30 initially, incrementing by 10 up to N=60 if kappa < 0.60. Table IV shows all four baselines exceed the threshold at N=30, but it is unclear whether any experiment required the increment. State this explicitly.","section":null},{"comment":"§IV-C.2a: The prompt sequence for GT generation (Prompts 1-3, Appendix B) is described as 'Chain of Thought,' but the prompts are sequential independent calls, not a single chain-of-thought prompt. The terminology should be corrected or qualified.","section":null},{"comment":"Table VII: Experiments 5-8 report 'N/A' for all metrics. If no metrics were computed, briefly state why (e.g., component mismatch prevented DSM alignment).","section":null},{"comment":"Table VII, Experiments 18-19: 'IDK' appears in the Component Mode column with occurrence 57-87%. Clarify whether this means the model literally returned 'I don't know' as a component name.","section":null},{"comment":"§V-A.1: '62,7%' and '12,8%' use comma as decimal separator; the rest of the paper uses periods. Standardize.","section":null},{"comment":"§V-A.2: 'QEB = 0.165' appears in the duplication experiment discussion, but Table VI does not list EB for the duplication case. Either add the row or clarify the source.","section":null},{"comment":"Figure references: Several figures (e.g., Figure 7, Figure 8) are referenced before their appearance. Consider forward-referencing or reordering.","section":null},{"comment":"§IV-C.2b, Listing 1: The manually generated D&M document uses index numbers [77], [83], etc. that appear to come from the full refrigerator dataset. Explain the indexing convention for reproducibility.","section":null},{"comment":"The paper would benefit from a summary table mapping each experiment number to its dataset, input parameters, and key finding, as the current Table VII is dense and spans multiple subsections.","section":null}],"recommendation":"major_revision","confidential_remarks":"The circularity concern is the primary issue. The authors are transparent about it (§VI-A), which is commendable, but transparency alone does not resolve the validity question. The abstract-system experiments are largely unaffected and already demonstrate the framework's value. The refrigerator experiments could be reframed: either (a) strengthen the validation protocol description to argue the GT is independently verified, or (b) explicitly reframe the refrigerator accuracy metrics as measuring consistency with a GPT-4o-derived reference rather than correctness against an engineering standard. Option (b) would be honest and still scientifically useful, but requires adjusting the framing of the central claim. The paper is a solid contribution if this issue is addressed; it is not grounds for rejection."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive review. The referee raises four major points: (1) insufficient description of the validation protocol for the GPT-4o-generated refrigerator ground truth, given the circularity of using the same model family for reference, input, and output; (2) absence of a sensitivity analysis for the Composite Quality Score weights; (3) an undefined edge case in the per-cell Selective Accuracy formula when all runs abstain; and (4) a black-box methodology concern regarding the diagnosis of internal pipeline behavior in the multi-subsystem experiment. We agree with all four points and will revise the manuscript accordingly. Points (2), (3), and (4) can be fully addressed through clarification and additional analysis. Point (1) is partially addressable: we will expand the validation protocol description and strengthen the discussion of circularity limitations, but we acknowledge that a fully independent ground truth (e.g., expert-constructed from proprietary data) is beyond the scope of this revision and remains a standing limitation.","responses":[{"response":"The referee is correct that the validation protocol for the refrigerator ground truth is insufficiently described, and that this matters because the accuracy metrics for the refrigerator experiments depend on the GT-DSM being a trustworthy reference. We will revise the manuscript to address this in two ways. First, we will expand §IV-C.2a to describe the validation protocol explicitly: the GT-DSM was validated by two authors (the paper's authors, both affiliated with the Engineering Systems Design group at TU/e), one of whom holds domain expertise in mechanical systems decomposition. Every dependency in the GT dataset was checked for (i) correctness against engineering first principles (e.g., compressor–condenser coil thermal and mechanical interactions), (ii) symmetry (every stated dependency has a reciprocated entry), and (iii) consistency with the Tilstra HDDSM classification. Dependencies that could not be verified from first principles were excluded. Second, we will strengthen §VI-A to make the circularity limitation more prominent and to explicitly state that the refrigerator results should be interpreted as a first-order evaluation under in-distribution conditions, not as evidence of generalizable system reasoning. We agree that a fully independent ground truth—ideally constructed by domain experts without LLM involvement and drawn from proprietary or out-of-distribution systems—would be necessary to draw stronger conclusions, and we will state this as a standing limitation and a priority for future work. We note that the abstract-system experiments (§V-A), which use manually constructed ground truth and fictive components unlikely to appear in training data, are less affected by this concern and provide the cleaner controlled evidence for the failure modes we标识.","revision_made":"partial","referee_comment":"§IV-C.2a: The ground-truth DSM for the refrigerator dataset is generated by GPT-4o and then 'manually validated for correctness and symmetry.' The input technical documents are also generated by GPT-4o from the GT dataset (§IV-C.2b). Since Auto-DSM itself runs on OpenAI models, the same model family produces the reference answer, the test input, and the system output. The authors acknowledge this circularity in §VI-A, but the validation protocol is not described: how many validators, what expertise, and whether every dependency was independently verified against engineering first principles. This is load-bearing because the paper's central claim of providing 'the first reproducible evidence of Auto-DSM's constraints' rests on accuracy metrics being meaningful. The abstract-system experiments (§V-A) use manually constructed ground truth and are less affected, but the refrigerator results—"},{"response":"The referee is correct. We will add a sensitivity analysis for the Composite Quality Score weights. Specifically, we will recompute Q_norm for all experiments in Tables VII and VIII under alternative weight configurations (e.g., w_acc ± 0.1, w_stab ± 0.1, w_pen ± 0.1, with renormalization to maintain the sum-to-one constraint) and report whether the qualitative rankings and comparative conclusions change. We expect that the main findings—e.g., that bi-directional definitions outperform uni-directional, that overgeneralization occurs under shared dependency types, and that first-entity bias appears in multi-subsystem experiments—are robust to moderate weight perturbations, because these conclusions are supported by multiple metrics (SA, entropy, completeness) in addition to Q. We will include the sensitivity results in an appendix and add a sentence in §III-B.2h directing the reader there.","revision_made":"yes","referee_comment":"§III-B.2h, Eq. (14): The Composite Quality Score Q_rc = w_acc * SA_rc + w_stab * (1 - H_rc) - w_pen * P_rc uses weights w_acc=0.5, w_stab=0.3, w_pen=0.2. These are stated to 'reflect engineering priorities' but no sensitivity analysis is provided. Since Q is used to rank and compare experiments throughout §V (e.g., Table VII, Table VIII), the reader cannot assess whether conclusions are robust to alternative weightings. A simple sensitivity check (e.g., ±0.1 on each weight) would address this."},{"response":"The referee identifies a genuine gap in the metric definition. When U_rc = 1 (all runs abstain), SA_rc is undefined because the denominator (1 - U_rc) equals zero. In our implementation, cells with U_rc = 1 are excluded from the DSM-wide SA mean and standard deviation, on the grounds that Selective Accuracy measures correctness among committed predictions, and a cell with zero committed predictions carries no information about accuracy. We will state this convention explicitly in §III-B.2f. We will also verify that no reported DSM-wide SA values in Tables VII and VIII are affected by this edge case in a way that would change the interpretation, and if any cells with U_rc = 1 exist in the reported experiments, we will note their count and location.","revision_made":"yes","referee_comment":"§III-B.2f, Eq. (12): Selective Accuracy per cell is defined as SA_rc = M_rc / (1 - U_rc). When U_rc = 1 (all runs abstain), this is undefined. The paper does not state how this case is handled. Table VII reports DSM-wide SA means and standard deviations; if any cell has U_rc=1, the aggregation rule matters. Clarify the convention (e.g., SA_rc := 0 or excluded from the mean)."},{"response":"The referee raises a valid methodological concern. The statement about 'two lists' was inferred from observable output structure: in runs where both subsystems' components appeared in the pipeline's intermediate output (visible in the returned component list), the final GEN-DSM contained only one subsystem's components. This inference is based on the observable input-output relationship—the component list returned by the pipeline included both subsystems, but the DSM constructed from that list included only one—not on access to internal pipeline code or prompts. However, we agree that the current wording ('stored them as two lists, after which only the first list was used') implies knowledge of internal processing that a black-box framework cannot justify. We will revise the language in §V-C.3 to describe the observable behavior precisely: the pipeline's returned component list included components from both subsystems, but the generated DSM contained dependencies for only one subsystem's components. We will qualify the interpretation as an inference from observable output structure, not a diagnosis of internal logic, and note that confirming the internal mechanism would require white-box access to the pipeline.","revision_made":"yes","referee_comment":"§V-C.3, Table IX: The multi-subsystem experiment shows that Auto-DSM extracts components for both subsystems but 'stored them as two lists, after which only the first list was used for dependency-type identification.' This is presented as a finding about the LLM, but it appears to be a finding about the Auto-DSM pipeline's internal processing logic. Since the framework is black-box, how was this internal behavior diagnosed? The claim should be qualified or the diagnostic method described."}],"tokens_in":32157,"tokens_out":1683,"duration_ms":224722,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"This paper builds the first controlled, multi-run evaluation framework for LLM-based DSM generation, and the behavioral findings it surfaces—directional asymmetry, sequence effects, overgeneralization, first-entity bias, confident hallucination under ambiguity—are real and useful. The code and datasets are public. The abstract-system experiments use manually constructed ground truth, so those findings are clean. The paper is also honest about its limitations, which I appreciate. N=30 runs with Fleiss' kappa is a reasonable reproducibility approach, and the experimental design is systematic across phrasing, parameter-dataset alignment, and complexity variations. The framework itself—combining single-run structural/classification metrics with multi-run stability measures—is the contribution, and it fills a genuine gap left by Koh's original Auto-DSM work, which had no controlled evaluation, no multi-run analysis, and no abstention assessment. The individual metrics are all standard, but assembling them into a DSM-specific benchmark is new and worthwhile. The black-box approach is appropriate given that Auto-DSM's internals are inaccessible. The preliminary experiments (Section II-B) showing that the no-code pipeline fails to reproduce Koh's reported results are a useful flag for the community. Now the soft spots. The circularity concern is real: GPT-4o generates the refrigerator ground truth, the input documents, and (via Auto-DSM) the outputs. But I think the stress-test overstates its scope. The abstract-system experiments in Section V-A use manually constructed ground truth and still demonstrate the key behavioral findings—directional asymmetry, sequence effects, confident misclassification under ambiguity. The circularity specifically weakens the refrigerator-based accuracy claims and the complexity comparisons in Section V-C, not the qualitative behavioral findings that form the paper's core value. That said, the manual validation of the GPT-4o-generated ground truth is underspecified—who validated, what protocol, what expertise. If the validators accepted GPT-4o's decomposition as plausible without independent engineering derivation, the circularity persists through validation. The Composite Quality Score weights (0.5/0.3/0.2) are ad hoc with no sensitivity analysis; the ranking of experiments could shift under different weights. This is a real gap but not load-bearing, since the behavioral findings don't depend on Q. No statistical significance testing is performed on cross-experiment comparisons—claims like 'complexity may affect identification' are qualitative observations, not statistically established results. The paper mostly frames them as observations, but the abstract and conclusion occasionally overstate. This paper is for researchers working at the intersection of LLMs and systems engineering, particularly those interested in MBSE tool validation. It deserves a serious referee. The revision targets are clear: add sensitivity analysis for Q, specify the ground-truth validation protocol, and either add significance tests or soften the quantitative claims to match what the descriptive statistics support. The circularity should be addressed with at least one independently derived ground-truth dataset, but the paper's core contribution—the framework and the behavioral findings from the abstract system—stands without it.","headline":"Solid evaluation framework for LLM-based DSM generation; behavioral findings are real and useful; circularity concern is genuine but narrower than it first appears.","tokens_in":33096,"tokens_out":1162,"would_cite":false,"duration_ms":67435,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"LLM-generated system dependency maps fail under ambiguity","keywords":["Design Structure Matrix","LLM evaluation","black-box testing","system decomposition","hallucination","abstention","Composite Quality Score","Fleiss kappa"],"falsifier":"If LLM-generated DSMs, evaluated against a ground-truth matrix independently constructed by domain experts (not LLM-generated), showed uniformly high selective accuracy and low entropy across all input variations — including ambiguous definitions, parameter mismatches, and multi-subsystem hierarchies — then the paper's claim that LLM-based DSM generation has systematic, reproducible failure modes would be falsified.","tokens_in":32273,"feed_emoji":"🔍","tokens_out":1302,"duration_ms":136151,"temperature":0.7,"pith_summary":"This paper builds a black-box evaluation framework to test whether large language models can reliably generate Design Structure Matrices (DSMs) — square grids that map which components of an engineered system depend on which others — from structured technical documentation. The framework combines single-run metrics (how well a generated matrix matches a ground-truth matrix on structure and cell-by-cell classification) with multi-run stability measures (Shannon entropy and Fleiss' kappa across 30+ repeated runs), and synthesizes them into a Composite Quality Score that penalizes confident-but-wrong answers more heavily than honest abstentions. Applied to both a synthetic abstract system and a real refrigerator decomposition, the framework reveals that LLMs can produce structurally plausible DSMs when inputs are clean and well-aligned, but systematically degrade when they encounter ambiguous dependency definitions, mismatched system parameters, or multi-subsystem hierarchies. The paper identifies specific failure modes: confident hallucination of dependencies not supported by the text, asymmetric treatment of bidirectional links, overgeneralization when all dependencies share the same type, first-entity bias in composite system descriptions, and unreliable abstention — the model often asserts a dependency rather than admitting uncertainty, even when configured to do otherwise.","feed_headline":"LLM-generated system dependency maps fail under ambiguity","feed_subtitle":"A black-box framework running 30+ repeated tests reveals that language models confidently hallucinate dependencies, ignore abstention rules,","key_machinery":"Composite Quality Score (Q), which aggregates selective accuracy, normalized entropy, and a cost-sensitive penalty term into a single per-cell quality measure. The cost-sensitive penalty weights confidently incorrect predictions at twice the cost of abstentions, reflecting the engineering principle that a wrong dependency is more dangerous than an acknowledged uncertainty.","core_discovery":"The central discovery is that LLM-based DSM generation, while capable of producing structurally plausible matrices under tightly controlled conditions, exhibits systematic and reproducible failure modes that are invisible without multi-run evaluation. Single-run accuracy metrics mask these failures: a model can be confidently wrong in the same way across 30 runs, yielding high reproducibility but low correctness. The paper shows that hallucination in this domain is not random noise but a structured phenomenon — it correlates with specific input conditions (undefined dependency types, parameter-dataset mismatch, pretrained knowledge of real-world systems) and manifests as overgeneralization (","pith_inferences":["The circular evaluation risk (ground truth generated by GPT-4o) means the reported accuracy numbers may overestimate real-world performance on systems outside the model's training distribution; the paper's own caveats suggest the framework is more reliable as a relative comparison tool than as an absolute quality measure.","The first-entity bias and single-list processing limitation in multi-subsystem decomposition suggest that the underlying LLM pipeline may lack architectural support for maintaining multiple simultaneous entity lists, which is a structural problem unlikely to be solved by prompt engineering alone.","The overgeneralization pattern — where uniform dependency types in the input trigger all-ones matrices — resembles a degenerate mode where the model defaults to the majority class, a failure mode well-documented in classification literature but not yet addressed in DSM-specific tooling."],"forward_implications":["Engineering teams adopting LLM-based DSM tools should require multi-run stability audits before trusting any single generated matrix, since confident-but-systematically-wrong outputs are undetectable from a single run.","Prompt design for automated system decomposition may need standardized interaction schemas with fixed dependency categories to prevent the overgeneralization and first-entity bias the paper documents.","The Composite Quality Score framework could be extended to other LLM-driven structured-output tasks (e.g., knowledge graph extraction, requirements traceability matrices) where confident errors carry asymmetric costs relative to abstentions.","Multi-subsystem hierarchical decomposition remains an unsolved problem for current LLM pipelines, suggesting that architectural support for subsystem segmentation — not just better prompts — may be necessary.","The finding that pretrained knowledge overrides document-internal evidence for real-world systems implies that evaluation on out-of-distribution or proprietary systems is essential before industrial deployment."],"fun_headline_variants":["LLMs hallucinate the same wrong dependencies across repeated runs","Single-run tests hide systematic LLM failures in design matrix generation","LLM design matrices look correct but fail on dependency ambiguity","Black-box testing exposes structured hallucination in LLM-generated DSMs","LLM dependency maps reproduce errors consistently under ambiguity"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The ground-truth DSM for the refrigerator dataset was itself generated by ChatGPT (GPT-4o) and then manually validated, meaning the benchmark may measure how well Auto-DSM reproduces GPT-4o's own decomposition style rather than how well it captures an independent engineering standard of correctness.","fun_headline_variants_meta":{"raw":{"variants":["LLMs hallucinate the same wrong dependencies across repeated runs","Single-run tests hide systematic LLM failures in design matrix generation","LLM design matrices look correct but fail on dependency ambiguity","Black-box testing exposes structured hallucination in LLM-generated DSMs","LLM dependency maps reproduce errors consistently under ambiguity"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":666,"prompt_tokens":601,"completion_tokens":65,"prompt_tokens_details":null},"tokens_in":601,"tokens_out":65,"duration_ms":67552,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T19:23:47.733076+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If LLM-generated DSMs, evaluated against a ground-truth matrix independently constructed by domain experts (not LLM-generated), showed uniformly high selective accuracy and low entropy across all input variations — including ambiguous definitions, parameter mismatches, and multi-subsystem hierarchies — then the paper's claim that LLM-based DSM generation has systematic, reproducible failure modes would be falsified.","supporting_citations":[],"review_version":1}