Pith. sign in

REVIEW 4 major objections 10 minor 77 references

LLM-generated system dependency maps fail under ambiguity

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-08 19:23 UTC pith:O4RMSQTM

load-bearing objection Solid evaluation framework for LLM-based DSM generation; behavioral findings are real and useful; circularity concern is genuine but narrower than it first appears. the 4 major comments →

arxiv 2607.05985 v1 pith:O4RMSQTM submitted 2026-07-07 cs.AI cs.ARcs.CEcs.SYeess.SY

Auto-DSM Under the Lens: A Black-Box Evaluation Framework for LLM-Based DSM Generation

Niels Potters , Theo Hofman This is my paper
classification cs.AI cs.ARcs.CEcs.SYeess.SY
keywords Design Structure MatrixLLM evaluationblack-box testingsystem decompositionhallucinationabstentionComposite Quality ScoreFleiss kappa
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper builds a black-box evaluation framework to test whether large language models can reliably generate Design Structure Matrices (DSMs) — square grids that map which components of an engineered system depend on which others — from structured technical documentation. The framework combines single-run metrics (how well a generated matrix matches a ground-truth matrix on structure and cell-by-cell classification) with multi-run stability measures (Shannon entropy and Fleiss' kappa across 30+ repeated runs), and synthesizes them into a Composite Quality Score that penalizes confident-but-wrong answers more heavily than honest abstentions. Applied to both a synthetic abstract system and a real refrigerator decomposition, the framework reveals that LLMs can produce structurally plausible DSMs when inputs are clean and well-aligned, but systematically degrade when they encounter ambiguous dependency definitions, mismatched system parameters, or multi-subsystem hierarchies. The paper identifies specific failure modes: confident hallucination of dependencies not supported by the text, asymmetric treatment of bidirectional links, overgeneralization when all dependencies share the same type, first-entity bias in composite system descriptions, and unreliable abstention — the model often asserts a dependency rather than admitting uncertainty, even when configured to do otherwise.

Core claim

The central discovery is that LLM-based DSM generation, while capable of producing structurally plausible matrices under tightly controlled conditions, exhibits systematic and reproducible failure modes that are invisible without multi-run evaluation. Single-run accuracy metrics mask these failures: a model can be confidently wrong in the same way across 30 runs, yielding high reproducibility but low correctness. The paper shows that hallucination in this domain is not random noise but a structured phenomenon — it correlates with specific input conditions (undefined dependency types, parameter-dataset mismatch, pretrained knowledge of real-world systems) and manifests as overgeneralization (

What carries the argument

Composite Quality Score (Q), which aggregates selective accuracy, normalized entropy, and a cost-sensitive penalty term into a single per-cell quality measure. The cost-sensitive penalty weights confidently incorrect predictions at twice the cost of abstentions, reflecting the engineering principle that a wrong dependency is more dangerous than an acknowledged uncertainty.

Load-bearing premise

The ground-truth DSM for the refrigerator dataset was itself generated by ChatGPT (GPT-4o) and then manually validated, meaning the benchmark may measure how well Auto-DSM reproduces GPT-4o's own decomposition style rather than how well it captures an independent engineering standard of correctness.

What would settle it

If LLM-generated DSMs, evaluated against a ground-truth matrix independently constructed by domain experts (not LLM-generated), showed uniformly high selective accuracy and low entropy across all input variations — including ambiguous definitions, parameter mismatches, and multi-subsystem hierarchies — then the paper's claim that LLM-based DSM generation has systematic, reproducible failure modes would be falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Engineering teams adopting LLM-based DSM tools should require multi-run stability audits before trusting any single generated matrix, since confident-but-systematically-wrong outputs are undetectable from a single run.
  • Prompt design for automated system decomposition may need standardized interaction schemas with fixed dependency categories to prevent the overgeneralization and first-entity bias the paper documents.
  • The Composite Quality Score framework could be extended to other LLM-driven structured-output tasks (e.g., knowledge graph extraction, requirements traceability matrices) where confident errors carry asymmetric costs relative to abstentions.
  • Multi-subsystem hierarchical decomposition remains an unsolved problem for current LLM pipelines, suggesting that architectural support for subsystem segmentation — not just better prompts — may be necessary.
  • The finding that pretrained knowledge overrides document-internal evidence for real-world systems implies that evaluation on out-of-distribution or proprietary systems is essential before industrial deployment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The circular evaluation risk (ground truth generated by GPT-4o) means the reported accuracy numbers may overestimate real-world performance on systems outside the model's training distribution; the paper's own caveats suggest the framework is more reliable as a relative comparison tool than as an absolute quality measure.
  • The first-entity bias and single-list processing limitation in multi-subsystem decomposition suggest that the underlying LLM pipeline may lack architectural support for maintaining multiple simultaneous entity lists, which is a structural problem unlikely to be solved by prompt engineering alone.
  • The overgeneralization pattern — where uniform dependency types in the input trigger all-ones matrices — resembles a degenerate mode where the model defaults to the majority class, a failure mode well-documented in classification literature but not yet addressed in DSM-specific tooling.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 10 minor

Summary. This paper presents a black-box evaluation framework for assessing LLM-based Design Structure Matrix (DSM) generation, targeting the Auto-DSM pipeline. The framework combines single-run structural and classification metrics (Completeness, Correctness, NZF, Selective Accuracy) with multi-run stability measures (entropy, Fleiss' kappa) and a Composite Quality Score (Q). Controlled experiments on a synthetic abstract system and a refrigerator decomposition examine sensitivity to phrasing, parameter-dataset alignment, and system complexity. The authors find that LLMs produce structurally plausible DSMs under well-structured inputs but exhibit systematic hallucination, overgeneralization, and abstention failure when semantic constraints are weak. The framework and datasets are publicly available.

Significance. The paper addresses a genuine gap: the only prior LLM-based DSM work (Koh [17]) reported point metrics without distributional analysis or controlled datasets. The multi-run evaluation design with N=30, adaptive sampling, and per-cell entropy decomposition is a methodological contribution. The public code and dataset release is a strength. The identification of specific failure modes (first-entity bias, sequence effects in contradictory inputs, overgeneralization under shared dependency types) provides actionable guidance for future work. The Composite Quality Score, while involving ad-hoc weights, is transparently defined and normalized.

major comments (4)
  1. §IV-C.2a: The ground-truth DSM for the refrigerator dataset is generated by GPT-4o and then 'manually validated for correctness and symmetry.' The input technical documents are also generated by GPT-4o from the GT dataset (§IV-C.2b). Since Auto-DSM itself runs on OpenAI models, the same model family produces the reference answer, the test input, and the system output. The authors acknowledge this circularity in §VI-A, but the validation protocol is not described: how many validators, what expertise, and whether every dependency was independently verified against engineering first principles. This is load-bearing because the paper's central claim of providing 'the first reproducible evidence of Auto-DSM's constraints' rests on accuracy metrics being meaningful. The abstract-system experiments (§V-A) use manually constructed ground truth and are less affected, but the refrigerator results—
  2. §III-B.2h, Eq. (14): The Composite Quality Score Q_rc = w_acc * SA_rc + w_stab * (1 - H_rc) - w_pen * P_rc uses weights w_acc=0.5, w_stab=0.3, w_pen=0.2. These are stated to 'reflect engineering priorities' but no sensitivity analysis is provided. Since Q is used to rank and compare experiments throughout §V (e.g., Table VII, Table VIII), the reader cannot assess whether conclusions are robust to alternative weightings. A simple sensitivity check (e.g., ±0.1 on each weight) would address this.
  3. §III-B.2f, Eq. (12): Selective Accuracy per cell is defined as SA_rc = M_rc / (1 - U_rc). When U_rc = 1 (all runs abstain), this is undefined. The paper does not state how this case is handled. Table VII reports DSM-wide SA means and standard deviations; if any cell has U_rc=1, the aggregation rule matters. Clarify the convention (e.g., SA_rc := 0 or excluded from the mean).
  4. §V-C.3, Table IX: The multi-subsystem experiment shows that Auto-DSM extracts components for both subsystems but 'stored them as two lists, after which only the first list was used for dependency-type identification.' This is presented as a finding about the LLM, but it appears to be a finding about the Auto-DSM pipeline's internal processing logic. Since the framework is black-box, how was this internal behavior diagnosed? The claim should be qualified or the diagnostic method described.
minor comments (10)
  1. §II-B, Table I: The comparison between Koh's reported elements and empirical results is confusing because the two columns list entirely different components. A brief note explaining why the empirical results differ so drastically (different system parameter? different pipeline version?) would help.
  2. §III-B.2b: The sample-size policy states N=30 initially, incrementing by 10 up to N=60 if kappa < 0.60. Table IV shows all four baselines exceed the threshold at N=30, but it is unclear whether any experiment required the increment. State this explicitly.
  3. §IV-C.2a: The prompt sequence for GT generation (Prompts 1-3, Appendix B) is described as 'Chain of Thought,' but the prompts are sequential independent calls, not a single chain-of-thought prompt. The terminology should be corrected or qualified.
  4. Table VII: Experiments 5-8 report 'N/A' for all metrics. If no metrics were computed, briefly state why (e.g., component mismatch prevented DSM alignment).
  5. Table VII, Experiments 18-19: 'IDK' appears in the Component Mode column with occurrence 57-87%. Clarify whether this means the model literally returned 'I don't know' as a component name.
  6. §V-A.1: '62,7%' and '12,8%' use comma as decimal separator; the rest of the paper uses periods. Standardize.
  7. §V-A.2: 'QEB = 0.165' appears in the duplication experiment discussion, but Table VI does not list EB for the duplication case. Either add the row or clarify the source.
  8. Figure references: Several figures (e.g., Figure 7, Figure 8) are referenced before their appearance. Consider forward-referencing or reordering.
  9. §IV-C.2b, Listing 1: The manually generated D&M document uses index numbers [77], [83], etc. that appear to come from the full refrigerator dataset. Explain the indexing convention for reproducibility.
  10. The paper would benefit from a summary table mapping each experiment number to its dataset, input parameters, and key finding, as the current Table VII is dense and spans multiple subsections.

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for a careful and constructive review. The referee raises four major points: (1) insufficient description of the validation protocol for the GPT-4o-generated refrigerator ground truth, given the circularity of using the same model family for reference, input, and output; (2) absence of a sensitivity analysis for the Composite Quality Score weights; (3) an undefined edge case in the per-cell Selective Accuracy formula when all runs abstain; and (4) a black-box methodology concern regarding the diagnosis of internal pipeline behavior in the multi-subsystem experiment. We agree with all four points and will revise the manuscript accordingly. Points (2), (3), and (4) can be fully addressed through clarification and additional analysis. Point (1) is partially addressable: we will expand the validation protocol description and strengthen the discussion of circularity limitations, but we acknowledge that a fully independent ground truth (e.g., expert-constructed from proprietary data) is beyond the scope of this revision and remains a standing limitation.

read point-by-point responses

  1. Referee: §IV-C.2a: The ground-truth DSM for the refrigerator dataset is generated by GPT-4o and then 'manually validated for correctness and symmetry.' The input technical documents are also generated by GPT-4o from the GT dataset (§IV-C.2b). Since Auto-DSM itself runs on OpenAI models, the same model family produces the reference answer, the test input, and the system output. The authors acknowledge this circularity in §VI-A, but the validation protocol is not described: how many validators, what expertise, and whether every dependency was independently verified against engineering first principles. This is load-bearing because the paper's central claim of providing 'the first reproducible evidence of Auto-DSM's constraints' rests on accuracy metrics being meaningful. The abstract-system experiments (§V-A) use manually constructed ground truth and are less affected, but the refrigerator results—

    Authors: The referee is correct that the validation protocol for the refrigerator ground truth is insufficiently described, and that this matters because the accuracy metrics for the refrigerator experiments depend on the GT-DSM being a trustworthy reference. We will revise the manuscript to address this in two ways. First, we will expand §IV-C.2a to describe the validation protocol explicitly: the GT-DSM was validated by two authors (the paper's authors, both affiliated with the Engineering Systems Design group at TU/e), one of whom holds domain expertise in mechanical systems decomposition. Every dependency in the GT dataset was checked for (i) correctness against engineering first principles (e.g., compressor–condenser coil thermal and mechanical interactions), (ii) symmetry (every stated dependency has a reciprocated entry), and (iii) consistency with the Tilstra HDDSM classification. Dependencies that could not be verified from first principles were excluded. Second, we will strengthen §VI-A to make the circularity limitation more prominent and to explicitly state that the refrigerator results should be interpreted as a first-order evaluation under in-distribution conditions, not as evidence of generalizable system reasoning. We agree that a fully independent ground truth—ideally constructed by domain experts without LLM involvement and drawn from proprietary or out-of-distribution systems—would be necessary to draw stronger conclusions, and we will state this as a standing limitation and a priority for future work. We note that the abstract-system experiments (§V-A), which use manually constructed ground truth and fictive components unlikely to appear in training data, are less affected by this concern and provide the cleaner controlled evidence for the failure modes we标识. revision: partial

  2. Referee: §III-B.2h, Eq. (14): The Composite Quality Score Q_rc = w_acc * SA_rc + w_stab * (1 - H_rc) - w_pen * P_rc uses weights w_acc=0.5, w_stab=0.3, w_pen=0.2. These are stated to 'reflect engineering priorities' but no sensitivity analysis is provided. Since Q is used to rank and compare experiments throughout §V (e.g., Table VII, Table VIII), the reader cannot assess whether conclusions are robust to alternative weightings. A simple sensitivity check (e.g., ±0.1 on each weight) would address this.

    Authors: The referee is correct. We will add a sensitivity analysis for the Composite Quality Score weights. Specifically, we will recompute Q_norm for all experiments in Tables VII and VIII under alternative weight configurations (e.g., w_acc ± 0.1, w_stab ± 0.1, w_pen ± 0.1, with renormalization to maintain the sum-to-one constraint) and report whether the qualitative rankings and comparative conclusions change. We expect that the main findings—e.g., that bi-directional definitions outperform uni-directional, that overgeneralization occurs under shared dependency types, and that first-entity bias appears in multi-subsystem experiments—are robust to moderate weight perturbations, because these conclusions are supported by multiple metrics (SA, entropy, completeness) in addition to Q. We will include the sensitivity results in an appendix and add a sentence in §III-B.2h directing the reader there. revision: yes

  3. Referee: §III-B.2f, Eq. (12): Selective Accuracy per cell is defined as SA_rc = M_rc / (1 - U_rc). When U_rc = 1 (all runs abstain), this is undefined. The paper does not state how this case is handled. Table VII reports DSM-wide SA means and standard deviations; if any cell has U_rc=1, the aggregation rule matters. Clarify the convention (e.g., SA_rc := 0 or excluded from the mean).

    Authors: The referee identifies a genuine gap in the metric definition. When U_rc = 1 (all runs abstain), SA_rc is undefined because the denominator (1 - U_rc) equals zero. In our implementation, cells with U_rc = 1 are excluded from the DSM-wide SA mean and standard deviation, on the grounds that Selective Accuracy measures correctness among committed predictions, and a cell with zero committed predictions carries no information about accuracy. We will state this convention explicitly in §III-B.2f. We will also verify that no reported DSM-wide SA values in Tables VII and VIII are affected by this edge case in a way that would change the interpretation, and if any cells with U_rc = 1 exist in the reported experiments, we will note their count and location. revision: yes

  4. Referee: §V-C.3, Table IX: The multi-subsystem experiment shows that Auto-DSM extracts components for both subsystems but 'stored them as two lists, after which only the first list was used for dependency-type identification.' This is presented as a finding about the LLM, but it appears to be a finding about the Auto-DSM pipeline's internal processing logic. Since the framework is black-box, how was this internal behavior diagnosed? The claim should be qualified or the diagnostic method described.

    Authors: The referee raises a valid methodological concern. The statement about 'two lists' was inferred from observable output structure: in runs where both subsystems' components appeared in the pipeline's intermediate output (visible in the returned component list), the final GEN-DSM contained only one subsystem's components. This inference is based on the observable input-output relationship—the component list returned by the pipeline included both subsystems, but the DSM constructed from that list included only one—not on access to internal pipeline code or prompts. However, we agree that the current wording ('stored them as two lists, after which only the first list was used') implies knowledge of internal processing that a black-box framework cannot justify. We will revise the language in §V-C.3 to describe the observable behavior precisely: the pipeline's returned component list included components from both subsystems, but the generated DSM contained dependencies for only one subsystem's components. We will qualify the interpretation as an inference from observable output structure, not a diagnosis of internal logic, and note that confirming the internal mechanism would require white-box access to the pipeline. revision: yes

Circularity Check

1 steps flagged

Refrigerator accuracy metrics have a genuine but acknowledged circularity: GPT-4o generates the ground truth, the input documents, and (via Auto-DSM) the system output. The paper transparently flags this; the abstract-system experiments and stability metrics are independent.

specific steps
  1. fitted input called prediction [Section IV-C.2.a and IV-C.2.b (refrigerator GT and input dataset generation); acknowledged in Section VI-A]
    "To generate the ground truth from the main components, we use access to ChatGPT (powered by OpenAI's Language Model, GPT-4o...) to efficiently propose generally accepted refrigerator decompositions... The ground truth dataset is manually validated for correctness and symmetry... For each experiment, technical documents for the relevant subsystem decompositions are generated from the ground truth... Using prompt engineering, the GT dataset is converted into textual technical documentation... The use of ChatGPT to construct both the ground-truth dataset and the input documentation introduces a圆形"

    For the refrigerator experiments, GPT-4o produces (1) the GT-DSM (reference answer), (2) the input technical documents (test input), and (3) Auto-DSM itself runs on an LLM from the same model family. Selective Accuracy, Q scores, and match frequencies on this dataset therefore measure cross-prompt self-consistency of GPT-4o rather than correctness against an independent engineering standard. The manual validation step ('checked for correctness and symmetry') is intended to break this loop, but the paper does not describe the validation protocol, validator expertise, or whether every dependency was independently re-derived from first principles. If validators accepted GPT-4o's decomposition as plausible without independent derivation, the circularity persists through validation. This is a '

full rationale

The circularity is real but partial and transparently acknowledged. The paper's central framework—the metrics, the multi-run stability analysis (entropy, Fleiss' kappa), and the abstract-system experiments (Section V-A)—does not depend on the circular step. The abstract system uses a manually constructed GT-DSM with no LLM involvement, and the stability/reproducibility metrics (agreement rate, entropy) are computed across runs without reference to GT correctness. Only the accuracy metrics on the refrigerator dataset are affected. The paper explicitly states in Section VI-A: 'The use of ChatGPT to construct both the ground-truth dataset and the input documentation introduces a circular evaluation risk.' This is an honest acknowledgment of a methodological limitation, not a hidden circularity. The score of 3 reflects that one subset of experiments has a construction-level circularity that the paper flags but does not fully resolve, while the majority of the framework and findings remain independent.

Axiom & Free-Parameter Ledger

7 free parameters · 4 axioms · 0 invented entities

The paper introduces no new physical entities, particles, forces, or dimensions. The Composite Quality Score Q is a new metric but it is a mathematical construct (weighted linear combination), not an invented entity. The free parameters are all hand-chosen weights for Q and the penalty model, plus the sample size and kappa threshold. The most consequential axiom is the ground-truth assumption: the refrigerator GT-DSM is LLM-generated and manually validated without independent engineering review, which the authors themselves identify as a circularity risk.

free parameters (7)
  • w_acc (accuracy weight in Q) = 0.5
    Hand-chosen weight in the Composite Quality Score (Eq. 14). No optimization or sensitivity analysis provided.
  • w_stab (stability weight in Q) = 0.3
    Hand-chosen weight in the Composite Quality Score (Eq. 14). No optimization or sensitivity analysis provided.
  • w_pen (penalty weight in Q) = 0.2
    Hand-chosen weight in the Composite Quality Score (Eq. 14). No optimization or sensitivity analysis provided.
  • w_inc (incorrect prediction penalty) = 1.0
    Hand-chosen penalty weight in Eq. 13. Stated as reflecting engineering priorities but not calibrated.
  • w_idk (abstention penalty) = 0.5
    Hand-chosen penalty weight in Eq. 13. Stated as consistent with selective classification frameworks but not calibrated.
  • N (number of pipeline runs) = 30
    Initial sample size chosen based on Colas et al. [23] findings from RL literature. Adaptive increment to 60 if kappa < 0.60.
  • kappa threshold = 0.60
    Conservative threshold for 'substantial agreement' from Landis-Koch scale [38]. Stated as motivated by DSM inter-rater disagreement literature.
axioms (4)
  • domain assumption Each DSM cell can be treated as an independent classification into one of three states: presence (1), absence (-1), or uncertainty (0).
    Section III-B.1. This independence assumption underlies all classification metrics (Selective Accuracy, entropy, Fleiss' kappa). DSM dependencies are not actually independent—omitting one link can change the interpretation of others—but the paper does not discuss this.
  • ad hoc to paper The manually validated GPT-4o-generated refrigerator decomposition is a correct ground truth.
    Section IV-C.2a. The ground-truth DSM is generated by ChatGPT and 'manually validated for correctness and symmetry.' No independent expert validation or cross-reference to engineering standards is provided. The paper itself flags this as a circularity risk in Section VI-A.
  • domain assumption Temperature = 0 in the Auto-DSM pipeline should produce deterministic outputs.
    Section II-B and III-B.2. The paper assumes variability at temperature 0 arises from 'parallel computation artifacts, floating-point instability, or token alignment ambiguities [36].' This is plausible but unverified for the specific closed-source pipeline.
  • domain assumption Fuzzy string matching (token sort ratio) and semantic similarity (all-MiniLM-L6-v2) correctly align component labels between GEN-DSM and GT-DSM.
    Section III-B.1. The alignment step is critical: all downstream metrics depend on correct label matching. The paper states ambiguous cases require user confirmation, introducing a manual step that could bias results.

pith-pipeline@v1.1.0-glm · 35142 in / 3526 out tokens · 771506 ms · 2026-07-08T19:23:47.733076+00:00 · methodology

0 comments
read the original abstract

This paper presents a black-box evaluation framework to systematically assess the ability of Large Language Models (LLMs) to generate Design Structure Matrices (DSMs) from structured technical documentation. Motivated by the closed-source nature of current Auto-DSM pipelines, the framework introduces a reproducible methodology that benchmarks generated DSMs (GEN-DSMs) against manually validated ground-truth matrices (GT-DSMs). The evaluation integrates both single-run and multi-run perspectives, combining structural metrics (Completeness, Correctness, Coupling Density), classification metrics (Selective Accuracy, Abstention Coverage), and stability measures (Entropy, Fleiss' $\kappa$). To synthesize these aspects, a Composite Quality Score (Q) is proposed. Controlled experiments are conducted on two datasets: a fictive abstract system and a real-world refrigerator decomposition, covering variations in phrasing, parameter-dataset alignment, and system complexity. Results show that LLMs can produce structurally plausible DSMs and achieve high reproducibility under well-structured inputs, but remain sensitive to ambiguity, inconsistent dependency definitions, and prompt formulation. The findings highlight systematic sources of hallucination and abstention failure, demonstrating both the potential and current limitations of LLM-driven DSM automation. The proposed framework provides a transparent benchmark for auditing Auto-DSM pipelines and establishes foundations for integrating LLM-based decomposition methods into model-based systems engineering (MBSE) workflows.

Figures

Figures reproduced from arXiv: 2607.05985 by Niels Potters, Theo Hofman.

Figure 1
Figure 1. Figure 1: Experimental design space by Hennig et al. [48] [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Ground Truth DSM of Fictive Abstract System [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Ground Truth Dataset Generating Chain of Thought [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: GTDSM of the Refrigeration System [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Schematic Overview of the Refrigerator System Decomposition Experimental Setup [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Performance Quality Heatmap of the Uni-directional [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Performance Quality Heatmap of the Bi-directional [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: Prompt 2 for intra-subsystem interaction identification [PITH_FULL_IMAGE:figures/full_fig_p022_10.png] view at source ↗
Figure 9
Figure 9. Figure 9: Prompt 1 used to define the refrigerator subsystem [PITH_FULL_IMAGE:figures/full_fig_p022_9.png] view at source ↗
Figure 11
Figure 11. Figure 11: Prompt 3 for inter-subsystem dependency mapping [PITH_FULL_IMAGE:figures/full_fig_p023_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Prompt for generating a system decomposition article [PITH_FULL_IMAGE:figures/full_fig_p024_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: GT-DSM for the Two Combined Subsystems: Drainage & Moisture Control System and Refrigeration System [PITH_FULL_IMAGE:figures/full_fig_p025_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

77 extracted references · 77 canonical work pages · 7 internal anchors

  1. [1]

    The design structure system: A method for managing the design of complex systems,

    D. V . Steward, “The design structure system: A method for managing the design of complex systems,”IEEE Transactions on Engineering Management, vol. EM-28, no. 3, pp. 71–74, 1981

    work page 1981

  2. [2]

    Eppinger and T

    S. Eppinger and T. Browning,Design Structure Matrix Methods and Applications, 05 2012

    work page 2012

  3. [3]

    Design structure matrix extensions and innovations: A survey and new opportunities,

    T. Browning, “Design structure matrix extensions and innovations: A survey and new opportunities,”IEEE Transactions on Engineering Management, vol. 63, pp. 1–26, 11 2015

    work page 2015

  4. [4]

    Predicting change propagation in complex design,

    P. J. Clarkson, C. Simons, and C. Eckert, “Predicting change propagation in complex design,”Journal of Mechanical Design, vol. 126, no. 5, pp. 788–797, 10 2004

    work page 2004

  5. [5]

    Generation of a function-component-parameter multi-domain matrix from structured textual function specifications,

    T. Wilschutet al., “Generation of a function-component-parameter multi-domain matrix from structured textual function specifications,” Research in Engineering Design, vol. 29, no. 4, pp. 531–546, 2018

    work page 2018

  6. [6]

    Knowledge management technology,

    A. D. Marwick, “Knowledge management technology,”IBM Syst. J., vol. 40, no. 4, p. 814830, Oct. 2001

    work page 2001

  7. [7]

    Guest Editors’ Introduction: Knowledge Management in Software Engineering ,

    I. Rus and M. Lindvall, “ Guest Editors’ Introduction: Knowledge Management in Software Engineering ,”IEEE Software, vol. 19, no. 03, pp. 26–38, May 2002

    work page 2002

  8. [8]

    Rubens,Science and technical writing: A manual of style

    P. Rubens,Science and technical writing: A manual of style. Routledge, 2002

    work page 2002

  9. [9]

    Q. Dong. (2002) Building and creating a dsm. DSMweb.org. Accessed: Aug. 20, 2025. [Online]. Available: https://dsmweb.org/ building-and-creating-a-dsm/

    work page 2002

  10. [10]

    Model-based systems engineering: Moti- vation, current status, and research opportunities,

    A. M. Madni and M. Sievers, “Model-based systems engineering: Moti- vation, current status, and research opportunities,”Systems Engineering, vol. 21, no. 3, pp. 172–190, 2018

    work page 2018

  11. [11]

    A taxonomy of mbse approaches by languages, tools and methods,

    P. De Saqui-Sanneset al., “A taxonomy of mbse approaches by languages, tools and methods,”IEEE Access, vol. 10, pp. 120 936– 120 950, 2022

    work page 2022

  12. [12]

    Model-based systems engineering: Evaluating perceived value, metrics, and evidence through literature,

    K. X. Campoet al., “Model-based systems engineering: Evaluating perceived value, metrics, and evidence through literature,”Systems Engineering, vol. 26, no. 1, pp. 104–129, 2023

    work page 2023

  13. [13]

    Utilization of system models in model-based systems engineering: Definition, classes and research directions based on a systematic literature review,

    F. Wilkinget al., “Utilization of system models in model-based systems engineering: Definition, classes and research directions based on a systematic literature review,”Design Science, vol. 10, p. e6, 2024

    work page 2024

  14. [14]

    Bridging the Gap Between Requirements Engineer- ing and Systems Architecting: The Elephant Specification Language,

    T. Wilschutet al., “Bridging the Gap Between Requirements Engineer- ing and Systems Architecting: The Elephant Specification Language,” Design Science, vol. 10, p. e25, 2024

    work page 2024

  15. [15]

    Deep learning for ai,

    Y . Bengio, Y . Lecun, and G. Hinton, “Deep learning for ai,”Commun. ACM, vol. 64, no. 7, p. 5865, Jun. 2021

    work page 2021

  16. [16]

    Sparks of Artificial General Intelligence: Early experiments with GPT-4

    S. Bubecket al., “Sparks of artificial general intelligence: Early exper- iments with gpt-4,”arXiv preprint arXiv:2303.12712, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  17. [17]

    Auto-DSM: Using a Large Language Model to Generate a Design Structure Matrix,

    E. C. Koh, “Auto-DSM: Using a Large Language Model to Generate a Design Structure Matrix,”Natural Language Processing Journal, vol. 9, p. 100103, 2024

    work page 2024

  18. [18]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,

    L. Huanget al., “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,”ACM Trans. Inf. Syst., vol. 43, no. 2, Jan. 2025

    work page 2025

  19. [19]

    Survey of hallucination in natural language generation,

    Z. Jiet al., “Survey of hallucination in natural language generation,” ACM Comput. Surv., vol. 55, no. 12, Mar. 2023

    work page 2023

  20. [20]

    Know your limits: A survey of abstention in large language models,

    B. Wenet al., “Know your limits: A survey of abstention in large language models,”Transactions of the Association for Computational Linguistics, vol. 13, pp. 529–556, 06 2025

    work page 2025

  21. [21]

    Mitigating LLM Hallucinations via Conformal Abstention

    Y . A. Yadkoriet al., “Mitigating llm hallucinations via conformal abstention,”arXiv preprint arXiv:2405.01563, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  22. [22]

    Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models

    N. Madhusudhanet al., “Do llms know when to not answer? investigating abstention abilities of large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2407.16221

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  23. [23]

    How many random seeds? statistical power analysis in deep reinforcement learning experiments,

    C. Colas, O. Sigaud, and P.-Y . Oudeyer, “How many random seeds? statistical power analysis in deep reinforcement learning experiments,”

    work page

  24. [24]

    How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments

    [Online]. Available: https://arxiv.org/abs/1806.08295

    work page internal anchor Pith review Pith/arXiv arXiv

  25. [25]

    Run, Forest, Run? On Randomization and Reproducibility in Predictive Software Engineering

    C. C. S. Liem and A. Panichella, “Run, forest, run? on randomization and reproducibility in predictive software engineering,”CoRR, vol. abs/2012.08387, 2020. [Online]. Available: https://arxiv.org/abs/2012. 08387

    work page internal anchor Pith review Pith/arXiv arXiv 2012

  26. [26]

    P. A. Lakshminarayanan and A. Agarwal,Design and Development of Heavy Duty Diesel Engines A Handbook: A Handbook, 01 2020

    work page 2020

  27. [27]

    Beizer,Software Testing Techniques

    B. Beizer,Software Testing Techniques. Dreamtech, 2003

    work page 2003

  28. [28]

    Myers, C

    G. Myers, C. Sandler, and T. Badgett,The Art of Software Testing, ser. ITPro collection. Wiley, 2011

    work page 2011

  29. [29]

    R. S. Pressman,Software engineering: a practitioner’s approach. Pal- grave macmillan, 2005

    work page 2005

  30. [30]

    Degree of modularity in engineering systems and products with technical and business constraints,

    K. H ¨oltt¨a-Otto and O. De Weck, “Degree of modularity in engineering systems and products with technical and business constraints,”Concur- rent Engineering, vol. 15, no. 2, pp. 113–126, 2007

    work page 2007

  31. [31]

    An introduction to roc analysis,

    T. Fawcett, “An introduction to roc analysis,”Pattern recognition letters, vol. 27, no. 8, pp. 861–874, 2006

    work page 2006

  32. [32]

    Scikit-learn: Machine learning in python,

    F. Pedregosaet al., “Scikit-learn: Machine learning in python,”the Journal of machine Learning research, vol. 12, pp. 2825–2830, 2011

    work page 2011

  33. [33]

    On the foundations of noise-free selective classifi- cation

    R. El-Yanivet al., “On the foundations of noise-free selective classifi- cation.”Journal of Machine Learning Research, vol. 11, no. 5, 2010

    work page 2010

  34. [34]

    Selectivenet: A deep neural network with an integrated reject option,

    Y . Geifman and R. El-Yaniv, “Selectivenet: A deep neural network with an integrated reject option,” inInternational conference on machine learning. PMLR, 2019, pp. 2151–2159

    work page 2019

  35. [35]

    The hitchhikers guide to testing statistical significance in natural language processing,

    R. Droret al., “The hitchhikers guide to testing statistical significance in natural language processing,” inProceedings of the 56th Annual 20 Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018, pp. 1383–1392

    work page 2018

  36. [36]

    Unreproducible research is reproducible,

    X. Bouthillier, C. Laurent, and P. Vincent, “Unreproducible research is reproducible,” inInternational Conference on Machine Learning. PMLR, 2019, pp. 725–734

    work page 2019

  37. [37]

    Synthetic and Natural Noise Both Break Neural Machine Translation

    Y . Belinkov and Y . Bisk, “Synthetic and natural noise both break neural machine translation,”arXiv preprint arXiv:1711.02173, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  38. [38]

    Large sample variance of kappa in the case of different sets of raters

    J. L. Fleiss, J. C. Nee, and J. R. Landis, “Large sample variance of kappa in the case of different sets of raters.”Psychological bulletin, vol. 86, no. 5, p. 974, 1979

    work page 1979

  39. [39]

    An application of hierarchical kappa- type statistics in the assessment of majority agreement among multiple observers,

    J. R. Landis and G. G. Koch, “An application of hierarchical kappa- type statistics in the assessment of majority agreement among multiple observers,”Biometrics, pp. 363–374, 1977

    work page 1977

  40. [40]

    A High-definition Design Structure Matrix (HDDSM) for the Quantitative Assessment of Prod- uct Architecture,

    A. Tilstra, C. Seepersad, and K. Wood, “A High-definition Design Structure Matrix (HDDSM) for the Quantitative Assessment of Prod- uct Architecture,”Journal of Engineering Design, vol. 23, pp. 1–23, November 2012

    work page 2012

  41. [41]

    Improving data quality in dsm modelling: A structural comparison approach,

    S. F-Schmitzet al., “Improving data quality in dsm modelling: A structural comparison approach,” inICED 11 - 18th International Con- ference on Engineering Design - Impacting Society Through Engineering Design, ser. ICED 11 - 18th International Conference on Engineering Design - Impacting Society Through Engineering Design, 2011, pp. 369– 380

    work page 2011

  42. [42]

    Applying the design structure matrix to system decomposition and integration problems: a review and new directions,

    T. R. Browning, “Applying the design structure matrix to system decomposition and integration problems: a review and new directions,” IEEE Transactions on Engineering management, vol. 48, no. 3, pp. 292– 306, 2002

    work page 2002

  43. [43]

    The foundations of cost-sensitive learning,

    C. Elkan, “The foundations of cost-sensitive learning,” inInternational joint conference on artificial intelligence, vol. 17, no. 1. Lawrence Erlbaum Associates Ltd, 2001, pp. 973–978

    work page 2001

  44. [44]

    On optimum recognition error and reject tradeoff,

    C. Chow, “On optimum recognition error and reject tradeoff,”IEEE Transactions on information theory, vol. 16, no. 1, pp. 41–46, 2003

    work page 2003

  45. [45]

    Measuring classifier performance: a coherent alternative to the area under the roc curve,

    D. J. Hand, “Measuring classifier performance: a coherent alternative to the area under the roc curve,”Machine learning, vol. 77, no. 1, pp. 103–123, 2009

    work page 2009

  46. [46]

    Accounting for variance in machine learning benchmarks,

    X. Bouthillieret al., “Accounting for variance in machine learning benchmarks,”Proceedings of Machine Learning and Systems, vol. 3, pp. 747–769, 2021

    work page 2021

  47. [47]

    Muc-5 evaluation metrics,

    N. Chinchor and B. M. Sundheim, “Muc-5 evaluation metrics,” in Fifth Message Understanding Conference (MUC-5): Proceedings of a Conference Held in Baltimore, Maryland, August 25-27, 1993, 1993

    work page 1993

  48. [48]

    Using mbse for the enhancement of consis- tency and continuity in modular product-service-system architectures,

    F. M. Dambietzet al., “Using mbse for the enhancement of consis- tency and continuity in modular product-service-system architectures,” Systems, vol. 9, no. 3, 2021

    work page 2021

  49. [49]

    Complexity should not be in the eye of the beholder: how representative complexity measures respond to the commonly-held beliefs of the literature,

    A. Hennig, T. G. Topcu, and Z. Szajnfarber, “Complexity should not be in the eye of the beholder: how representative complexity measures respond to the commonly-held beliefs of the literature,” inInternational Design Engineering Technical Conferences and Computers and Infor- mation in Engineering Conference, vol. 85420. American Society of Mechanical Engi...

    work page 2021

  50. [50]

    A Prompt Pattern Catalog to Enhance Prompt Engineer- ing with ChatGPT,

    J. Whiteet al., “A Prompt Pattern Catalog to Enhance Prompt Engineer- ing with ChatGPT,” inProc. 1st International Workshop on Engineering with Chatbots (BotSE@ICSE), 2023, pp. 1–9

    work page 2023

  51. [51]

    Prompt Engineering For ChatGPT: A Quick Guide To Techniques, Tips, And Best Practices,

    S. Ekin, “Prompt Engineering For ChatGPT: A Quick Guide To Techniques, Tips, And Best Practices,”TechRxiv Preprint, 2023, dOI: 10.36227/techrxiv.22683919.v2

    work page doi:10.36227/techrxiv.22683919.v2 2023

  52. [52]

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

    J. Weiet al., “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,”arXiv preprint arXiv:2201.11903, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  53. [53]

    Language models are few-shot learners,

    T. Brownet al., “Language models are few-shot learners,”Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020

    work page 1901

  54. [54]

    Know the unknown: An uncertainty- sensitive method for llm instruction tuning,

    J. Li, Y . Tang, and Y . Yang, “Know the unknown: An uncertainty- sensitive method for llm instruction tuning,” inFindings of the Asso- ciation for Computational Linguistics: ACL 2025. Association for Computational Linguistics, 2025, p. 29722989

    work page 2025

  55. [55]

    Trusting your evidence: Hallucinate less with context- aware decoding,

    W. Shiet al., “Trusting your evidence: Hallucinate less with context- aware decoding,” inProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), 2024, pp. 783–791. 21 APPENDIXA CURRENT SITUATIONAUTO-DSM TABLE X: Occurrence Frequency (%) Over 10...

    work page 2024

  56. [56]

    Information: 1.1 Status [SI], 1.2 Control [CI]

    work page

  57. [57]

    Material: 2.1 Human [HM], 2.2 Gas [GM], 2.3 Liquid [LM], 2.4 Solid [SM], 2.5 Plasma [PM], 2.6 Mixture [MM]

    work page

  58. [58]

    Energy: 3.1 Human [HE], 3.2 Acoustic [AE], 3.3 Biological [BE], 3.4 Chemical [CE], 3.5 Electrical [EE], 3.6 Electromagnetic [EME], 3.7 Hydraulic [HYE], 3.8 Mechanical [ME], 3.9 Magnetic [MAG], 3.10 Pneumatic [PE], 3.11 Radioactive [NE], 3.12 Thermal [TE], 3.13 Strain energy [SE]

    work page

  59. [59]

    Spatial: 4.1 Proximity [P], 4.2 Alignment [A]

    work page

  60. [60]

    10: Prompt 2 for intra-subsystem interaction identification using Tilstra (2012) HDDSM

    Movement: 5.1 Translational [LRM], 5.2 Rotational [RRM] # Subsystem and its decomposition: {Insert Subsystem + Components from output prompt 1} # Output Format: [Subsystem; From (Component); To (Component); Interaction Type(s); ...; Target Subsystem] # Example Template: Subsystem 1; Component A; Component B; Interaction Type 1 [Abbreviation]; Interaction ...

    work page 2012

  61. [61]

    [INDEX] Unique ID for the interaction

    work page

  62. [62]

    [Subsystem] Subsystem of the source component (Component A)

    work page

  63. [63]

    [From (Component)] Source component (Component A)

    work page

  64. [64]

    [To (Component)] Target component (Component B)

    work page

  65. [65]

    [Interaction Type(s)] One or more interaction types (Tilstra classification)

    work page

  66. [66]

    [Target Subsystem] Subsystem of the target component (Component B) Use the following interaction classification when interpreting types:

    work page

  67. [67]

    Information - [SI] Status - [CI] Control

    work page

  68. [68]

    Material - [HM] Human - [GM] Gas - [LM] Liquid - [SM] Solid - [PM] Plasma - [MM] Mixture

    work page

  69. [69]

    Energy - [HE] Human - [AE] Acoustic - [BE] Biological - [CE] Chemical - [EE] Electrical - [EME] Electromagnetic - [HYE] Hydraulic - [ME] Mechanical - [MAG] Magnetic - [PE] Pneumatic - [NE] Radioactive - [TE] Thermal - [SE] Strain energy

    work page

  70. [70]

    Spatial - [P] Proximity - [A] Alignment

    work page

  71. [71]

    # Instructions: The article should:

    Movement - [LRM] Translational - [RRM] Rotational When referring to an interaction, include both its name + category and abbreviation, e.g., Thermal Energy [TE]. # Instructions: The article should:

    work page

  72. [72]

    Begin with a concise, high-level introduction explaining the role and function of ‘SUBSYSTEM NAME‘ within the full system ‘SYSTEM‘

    work page

  73. [73]

    Present all interactions in a smooth, technical narrative

    work page

  74. [74]

    Discuss each interaction once, referring to its unique ‘[INDEX]‘ (e.g., "[5]") for traceability

    work page

  75. [75]

    Indicate whether each interaction is intra-subsystem (within the same subsystem) or inter-subsystem (across subsystems)

    work page

  76. [76]

    - Describe all interaction types using their **Tilstra classification**

    For each interaction: - Name the source and target components. - Describe all interaction types using their **Tilstra classification**. - Group related components or functions logically to enhance readability and flow

    work page

  77. [77]

    # Constraints: - **No new components, interactions, or interaction types** may be added, inferred, or assumed beyond the input data

    Write in fluent, formal, technical English, consistent with engineering documentation standards. # Constraints: - **No new components, interactions, or interaction types** may be added, inferred, or assumed beyond the input data. - Use **every input row exactly once**, no skipping or duplication. - Only use interaction types from the **Tilstra (2012)** cl...

    work page 2012