Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

An Empirical Exploration of ChatGPT's Ability to Support Problem Formulation Tasks for Mission Engineering and a Documentation of its Performance Variability

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ChatGPT-3.5, given a NASA mission design problem, correctly identifies only about half of the ground-truth stakeholders and consistently misses external systems and environmental factors, while its outputs vary substantially across…

desk verdict A small, honest empirical study whose qualitative finding about LLMs missing non-human stakeholders is robust; the headline variance statistic needs disaggregation before it can be trusted. read the letter →

arxiv 2502.03511 v1 pith:XQQOTETC submitted 2025-02-05 cs.SE cs.AIcs.CL

Max Ofsa , Taylan G. Topcu This is my paper · ORCID
classification cs.SEcs.AIcs.CL
keywords systemsengineeringmissionAI4SEhuman-AIcollaborationproblemformulationstakeholderidentificationChatGPTLLMvariability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a general-purpose large language model can help with problem formulation in mission engineering, specifically stakeholder identification, and whether its answers are stable enough to trust. Using a NASA Ceres mission design challenge as a reference problem, the authors ran 25 independent ChatGPT-3.5 threads and scored the outputs against a 12-stakeholder ground-truth list. They find that the model captures a useful but partial set of stakeholders—about 6.76 correct on average—while almost always omitting external systems and environmental conditions even when the prompt explicitly mentions them. They also document large variance across threads, with a standard deviation of 3.19 in total stakeholders identified, leading them to conclude that LLM assistance can cut some expert workload but should be treated as a stochastic tool, not a reliable standalone analysis.

What carries the argument

The mechanism that carries the argument is an empirical protocol combining (1) a fixed reference problem—the NASA RASC-AL 2021 Ceres mission need statement, deliberately chosen because it predates ChatGPT-3.5's knowledge cutoff—and (2) repeated independent sampling: each prompt is run in a fresh thread, ten parallel threads per prompt iteration, yielding 25 threads total. The outputs are then qualitatively coded against a manually constructed ground-truth list of 12 stakeholders using three rubrics: correctness (direct match to ground truth), traceability (whether the stakeholder is real and appropriate), and specificity (whether the abstraction level fits early problem formulation). The parallel-thread design is what turns a single-response evaluation into a measurement of output variability, and the ground-truth list provides the yardstick for computing miss rates and the model's human-stakeholder bias.

What would settle it

Have an independent expert, blind to the authors' scores, code the same 25 outputs against the same 12-stakeholder ground-truth list; if inter-rater reliability on 'correct versus missed' is low (e.g., Cohen's kappa below 0.6), the documented miss rates and the claim that external systems/environmental factors are always omitted are artifacts of the scoring rubric rather than stable properties of ChatGPT-3.5's behavior.

Watch

Extended reading notes

Core claim

The central discovery is that a freely available LLM (ChatGPT-3.5) can act as a rough first-pass stakeholder generator for an open-ended mission engineering problem, but its performance has a sharp blind spot: it reliably identifies human and organizational stakeholders (e.g., NASA administration, mission operations, manufacturers, academia) while structurally ignoring non-human stakeholders such as the operational environment, launch/transportation infrastructure, communications networks, and collaborating mission assets, despite explicit prompting to treat environmental factors and engineered systems as stakeholders. The model also drifts toward solution-specific over-specification (e.g., naming specific facilities like JPL or programs like the Commercial Crew Program) and never reproduces the same answer twice, which the authors interpret as evidence that every LLM response is a roll of the dice. Because correct, missed, and over-specified categories are all sizable and variable, the paper argues that practitioners should adopt a stochastic view of LLM capabilities.

Load-bearing premise

The conclusions hinge on the assumption that the authors' list of 12 ground-truth stakeholders is complete and correct for the Ceres mission, and that their judgments of 'correct,' 'over-specified,' and 'wrong' would be shared by another expert rater.

Editorial extensions

If this is right

  • Practitioners should treat any single LLM response to a problem-formulation question as one sample from a noisy distribution; decisions should be based on multiple independent runs or an ensemble.
  • LLM-generated stakeholder lists can serve as a starting point for domain experts, shifting work from generation to correction and refinement, but non-experts risk anchoring on over-specified or solution-specific outputs.
  • Explicitly prompting for non-human stakeholders (engineered systems, environments) is not sufficient to overcome the model's bias; downstream verification against a structured stakeholder taxonomy is still required.
  • Evaluation of LLM-assisted mission engineering should include variance metrics (e.g., standard deviation across threads), not just average quality, because variability itself is a key cost of using stochastic tools.
  • The systematic omission of external systems and environmental factors suggests that LLM assistance alone cannot be trusted for complete system boundary definition in mission engineering.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the bias toward human stakeholders is rooted in training corpora, then newer or fine-tuned LLMs may still exhibit the same structural omission of non-human elements, and the paper's finding may generalize beyond ChatGPT-3.5.
  • A practical follow-up would be to test whether a multi-thread ensemble—e.g., taking the union of stakeholders across 10 parallel runs—raises recall toward the ground truth; the data suggest it might, since each thread finds a different half.
  • The high variance in over-specified stakeholders (SD 2.72) hints that solution-seeking behavior is not uniform; it may be triggered by particular prompt phrasings, which could potentially be suppressed by explicitly forbidding named facilities or programs in the prompt.
  • The ground-truth list itself collapses stakeholder hierarchy (primary, secondary, sponsor), and the paper's method could be extended to see whether LLMs perform differently when asked to classify stakeholders by type rather than simply list them.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reports an empirical study in which the authors prompted ChatGPT-3.5 to identify stakeholders for a NASA RASC-AL Ceres mission needs statement, collected 25 parallel output threads, and qualitatively coded each identified stakeholder as correct, over-specified, wrong, or missed against a 12-item author-constructed ground truth list. The headline findings are that the model identifies on average 6.76 of the 12 ground-truth stakeholders (SD 1.56), consistently misses external systems and environmental factors, produces a substantial number of over-specified stakeholders (mean 4.80, SD 2.72), and exhibits large variability in total stakeholder counts across threads (SD 3.19). The authors interpret these results as indicating that a general-purpose LLM can provide a partial initial stakeholder list but is not reliable enough for standalone problem formulation, and that practitioners should treat LLM outputs stochastically.

Significance. The study addresses a timely and practically important question about whether general-purpose LLMs can support early systems-engineering and mission-engineering activities. Its strengths include a concrete reference problem with a published needs statement, a clearly described iterative prompting methodology, and a quantitative documentation of variability across repeated inquiries—a dimension that many LLM evaluation papers neglect. If the reported effects hold, the paper provides useful evidence for the systems-engineering community about the limits of current open-access LLMs, particularly their systematic omission of non-human stakeholders and their tendency toward solution-specific over-specification. The contribution is exploratory and small-scale rather than definitive, but the paper is transparent about its scope and limitations, and the central descriptive claims are checkable from the reported counts. For these reasons, the study is within the scope of the venue and of potential value to practitioners and researchers, provided the methodological issues identified below are addressed.

major comments (3)
  1. [Section 3.2 and Table 2] The variance claim in Table 2 is not supported as stated because the 25 threads may not be independent draws from a single fixed prompt. The text says that 'for each of our prompts we created threads of ten parallel independent inquiries,' yet the results section reports 25 total threads. Step 2c and 2d describe iterative prompt refinement, so it is unclear how many prompt versions were used, how many threads each version received, and which prompts advanced to Step 3. If multiple prompt wordings are pooled, the standard deviation of 3.19 for total stakeholders conflates differences due to prompt wording with stochastic sampling from a fixed prompt. Please report the exact prompt(s), the number of threads per prompt version, and per-prompt means and standard deviations; if only one prompt version was used in the analyzed 25 threads, state this explicitly and explain the discrepancy with the ten-threads-per-prompt phrasing.
  2. [Section 3.3 and Table 1] The correctness and missed counts are grounded entirely in an author-derived 12-stakeholder ground truth list, with no external validation and no inter-rater reliability check. The paper states that the list 'was developed in response to the needs statement' but does not describe the derivation process, the expertise of the coders, or any independent audit. As a result, the finding that approximately half of the ground-truth stakeholders are missed (mean 5.24) is partly a property of the completeness and appropriateness of that list. Please provide a justification for the completeness of the list (e.g., derivation from a standard reference, an expert panel, or a documented iterative process) and, ideally, a second coder agreement measure (such as Cohen's kappa) on a subset of threads. The observation that Operational Environment was never identified in any thread is more robust because it does not depend on ground-truth completeness, but the quantitative missed counts do depend on that list.
  3. [Section 4.2 and Figure 3] The paper reports aggregate counts and standard deviations but does not provide the per-thread raw data that support the stacked bar chart in Figure 3. Without a table or appendix listing the stakeholders identified in each of the 25 threads and their assigned classifications, the reader cannot verify the coding decisions, the claim that NASA Administration and Mission Operations were always correctly identified, or the claim that the Commercial Crew Program was a notable error. This is particularly important because the boundary between 'correct' and 'over-specified' is qualitative and may be applied inconsistently across threads. Please include the per-thread classification data, either as a supplementary table or in an appendix, so the empirical claims are reproducible.
minor comments (5)
  1. [Section 1] The phrase 'preform trade space analysis' should be 'perform trade space analysis'.
  2. [Section 4.1, Figure 3] The sentence 'A review of the threads in Fig 3 suggest that ChatGPT never provided a completely correct or completely wrong output' contains a subject-verb agreement error ('suggest' should be 'suggests'), and the figure caption text 'ChatGP T,s' contains a typographical error.
  3. [Section 5] The phrase 'this concern is not just intransient to mission engineering tasks' is unclear; 'intransient' does not communicate the intended meaning, and the sentence should be revised (e.g., 'not unique to' or 'not transient in').
  4. [References] Reference list numbering skips from [9] to [11], and several references are cited only in lists without corresponding in-text citations; please reconcile the reference list and ensure all citations are complete.
  5. [Section 3.2] The paper states that prompts were evaluated for alignment with the ground truth list but does not define criteria for what made a prompt 'relevant' versus 'less aligned'; please operationalize this step, because it affects which threads were analyzed.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper is an empirical evaluation against an author-constructed rubric; the quantitative findings are direct observations, not fitted predictions, and the one self-citation is interpretive only.

full rationale

The paper makes no first-principles derivation claim; it reports an empirical measurement. The evaluation is anchored to a ground truth list that the authors constructed themselves (Table 1, Sec. 3.3), so the correctness and miss rates are relative to that author-defined benchmark; this is a real threat to external validity and was acknowledged in the paper ('these stakeholders were derived without distinction within the hierarchy of stakeholder types'), but it is not a circular reduction because the LLM outputs were not used to define the list and no fitted parameter is renamed as a prediction. The finding that environmental/external-system stakeholders were always absent is a direct observation over the threads, not an equation forced by the rubric. The variance statistics in Table 2 are arithmetic summaries of coded outputs, and the 'missed ground truth stakeholders were identified by simply subtracting' statement in Sec. 3.3 is an explicit scoring convention, not a hidden equivalence. The only self-citation touching an interpretive claim is [30] (Husain, Wach, Topcu) for 'solution seeking behavior'; it is illustrative, not load-bearing. The pooled-thread variance concern raised about iterative prompt refinement (Sec. 3.2) is a methodological validity limitation, not a circularity pattern, and the paper itself flags the small single-model sample ('we only experimented with a rather small data set of 25 independent threads, and a single LLM'). Therefore no circular step is exhibited.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim does not rest on numeric fitted parameters or invented entities. It does rest on the author-constructed ground truth list, assumptions about task representativeness, comparable thread conditions despite prompt iteration, and the validity of the qualitative coding. These are the main unvalidated inputs.

assumptions (4)
  • ad hoc to paper The 12-stakeholder ground truth list (Table 1) is complete and correct for the Ceres mission needs statement.
    The authors derived the list themselves without external validation, inter-rater reliability, or citation to a published stakeholder register; the correctness and missed counts are computed against this list.
  • domain assumption Stakeholder identification is a representative core problem formulation task in mission engineering.
    The paper motivates this in Sections 1 and 2.2 but does not empirically establish that performance on this single task represents broader ME problem formulation.
  • ad hoc to paper The 25 threads were produced under comparable conditions, despite iterative prompt refinement in Steps 2c-2d.
    Prompts that produced less aligned outputs were refined and rerun; the final pooled threads may mix multiple prompt versions, so observed variance may conflate prompt differences with model stochasticity.
  • domain assumption Qualitative coding of correctness, traceability, and specificity is reliable without a formal coding protocol.
    Coding was performed by the authors with no inter-rater reliability check; the specificity criterion excludes over-specified stakeholders based on subjective judgment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Empirical Exploration of ChatGPT's Ability to Support Problem Formulation Tasks for Mission Engineering and a Documentation of its Performance Variability." pith.science (2026). https://pith.science/paper/XQQOTETC

@misc{pith2026250203511,
  author       = {Pith},
  title        = {Pith review of: An Empirical Exploration of ChatGPT's Ability to Support Problem Formulation Tasks for Mission Engineering and a Documentation of its Performance Variability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XQQOTETC}},
  note         = {Machine review of arXiv:2502.03511}
}
read the original abstract

Systems engineering (SE) is evolving with the availability of generative artificial intelligence (AI) and the demand for a systems-of-systems perspective, formalized under the purview of mission engineering (ME) in the US Department of Defense. Formulating ME problems is challenging because they are open-ended exercises that involve translation of ill-defined problems into well-defined ones that are amenable for engineering development. It remains to be seen to which extent AI could assist problem formulation objectives. To that end, this paper explores the quality and consistency of multi-purpose Large Language Models (LLM) in supporting ME problem formulation tasks, specifically focusing on stakeholder identification. We identify a relevant reference problem, a NASA space mission design challenge, and document ChatGPT-3.5's ability to perform stakeholder identification tasks. We execute multiple parallel attempts and qualitatively evaluate LLM outputs, focusing on both their quality and variability. Our findings portray a nuanced picture. We find that the LLM performs well in identifying human-focused stakeholders but poorly in recognizing external systems and environmental factors, despite explicit efforts to account for these. Additionally, LLMs struggle with preserving the desired level of abstraction and exhibit a tendency to produce solution specific outputs that are inappropriate for problem formulation. More importantly, we document great variability among parallel threads, highlighting that LLM outputs should be used with caution, ideally by adopting a stochastic view of their abilities. Overall, our findings suggest that, while ChatGPT could reduce some expert workload, its lack of consistency and domain understanding may limit its reliability for problem formulation tasks.

Figures

Figures reproduced from arXiv: 2502.03511 by the authors.

Figure 1
Figure 1. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Trust at Your Own Peril: A Mixed Methods Exploration of the Ability of Large Language Models to Generate Expert-Like Systems Engineering Artifacts and a Characterization of Failure Modes

    cs.CL 2025-02 conditional novelty 5.0 of 10

    Closed-source LLMs can produce systems engineering artifact text that scores nearly identically to a human expert benchmark on MAUVE text similarity, but expert review of the best-scoring outputs reveals three failure...

Reference graph

Works this paper leans on

64 extracted references · 50 canonical work pages · cited by 1 Pith paper

  1. [1]

    NASA Systems Engineering Handbook

    S. R. Hirshorn, L. D. Voss, and L. K. Bromley, “NASA Systems Engineering Handbook.” Feb. 17, 2017. Accessed: Nov. 02, 2023. [Online]. Available: https://ntrs.nasa.gov/citations/20170001761

    arXiv 2017

  2. [2]

    Larson, D

    W. Larson, D. Kirkpatrick, J. Sellers, L. Thomas, and D. Verma, Applied Space Systems Engineering. McGraw-Hill Education, 2009

    work page 2009

  3. [3]

    Systems Engineering and Analysis (5th Edition),

    Blanchard and Fabrycky, “Systems Engineering and Analysis (5th Edition),” Up. Saddle River NJ Pearson Educ., 2011

    work page 2011

  4. [4]

    Architecting principles for systems-of-systems,

    M. W. Maier, “Architecting principles for systems-of-systems,” Syst. Eng., vol. 1, no. 4, pp. 267–284, 1998, doi: 10.1002/(SICI)1520- 6858(1998)1:4<267::AID-SYS3>3.0.CO;2-D

    doi:10.1002/(sici)1520- 1998

  5. [5]

    Mission Engineering Guide,

    Office of the Deputy Director for Engineering, “Mission Engineering Guide,” Office of the Under Secretary of Defense for Research and Engineering, Washington, D.C., 2020. [Online]. Available: https://ac.cto.mil/wp-content/uploads/2020/12/MEG-v40_20201130_shm.pdf

    work page 2020

  6. [6]

    A System-of-Systems perspective for information fusion system design and evaluation,

    A. K. Raz, C. R. Kenley, and D. A. DeLaurentis, “A System-of-Systems perspective for information fusion system design and evaluation,” Inf. Fusion, vol. 35, pp. 148–165, May 2017, doi: 10.1016/j.inffus.2016.10.002

    work page doi:10.1016/j.inffus.2016.10.002 2017

  7. [7]

    Foundational issues in engineering systems: A framing paper,

    J. Moses, “Foundational issues in engineering systems: A framing paper,” Eng. Syst. Monogr., p. 2, 2004

    work page 2004

  8. [8]

    So You Think Your System Is Complex?: Why and How Existing Complexity Measures Rarely Agree,

    A. Hennig, T. G. Topcu, and Z. Szajnfarber, “So You Think Your System Is Complex?: Why and How Existing Complexity Measures Rarely Agree,” J. Mech. Des., vol. 144, no. 4, Art. no. 4, Nov. 2021, doi: 10.1115/1.4052701

    work page doi:10.1115/1.4052701 2021

Show all 64 references
  1. [9]

    Defense acquisitions: Assessments of selected weapon programs,

    U. S. Government Accountability Office, “Defense acquisitions: Assessments of selected weapon programs,” GAO, Washington, DC, GAO-11-233SP, 2011

    2011

  2. [11]

    NASA: Assessments of Major Projects,

    U. S. Government Accountability Office, “NASA: Assessments of Major Projects,” GAO-19-262SP, May 2019. Accessed: Feb. 25,

    2019

  3. [12]

    Mission Engineering and Design Using Real-Time Strategy Games: An Explainable AI Approach,

    A. Dachowicz et al., “Mission Engineering and Design Using Real-Time Strategy Games: An Explainable AI Approach,” J. Mech. Des., vol. 144, no. 021710, Nov. 2021, doi: 10.1115/1.4052841

    2021 doi

  4. [13]

    Conceptual, Mathematical, and Analytical Foundations for Mission Engineering and System of Systems Analysis,

    A. K. Raz, M. Bhuyian, J. L. Bricio-Neto, C. Santos, and D. Maxwell, “Conceptual, Mathematical, and Analytical Foundations for Mission Engineering and System of Systems Analysis,” IEEE Syst. J., pp. 1–11, 2024, doi: 10.1109/JSYST.2024.3409231

    2024

  5. [14]

    Synthesizing Designs With Interpart Dependencies Using Hierarchical Generative Adversarial Networks,

    W. Chen and M. Fuge, “Synthesizing Designs With Interpart Dependencies Using Hierarchical Generative Adversarial Networks,” J. Mech. Des., vol. 141, no. 11, Art. no. 11, Sep. 2019, doi: 10.1115/1.4044076

    2019 doi

  6. [15]

    Deep Generative Models in Engineering Design: A Review,

    L. Regenwetter, A. H. Nobari, and F. Ahmed, “Deep Generative Models in Engineering Design: A Review,” J. Mech. Des., vol. 144, no. 071704, Mar. 2022, doi: 10.1115/1.4053859

    2022 doi

  7. [16]

    Daphne: A Virtual Assistant for Designing Earth Observation Distributed Spacecraft Missions,

    A. V. I. Martin and D. Selva, “Daphne: A Virtual Assistant for Designing Earth Observation Distributed Spacecraft Missions,” IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., vol. 13, pp. 30–48, 2020, doi: 10.1109/JSTARS.2019.2948921

    2020

  8. [17]

    Idea generation with Technology Semantic Network,

    S. Sarica, B. Song, J. Luo, and K. L. Wood, “Idea generation with Technology Semantic Network,” Artif. Intell. Eng. Des. Anal. Manuf. AI EDAM, vol. 35, no. 3, Art. no. 3, Aug. 2021, doi: 10.1017/S0890060421000020

    2021 doi

  9. [18]

    Semantic Networks for Engineering Design: State of the Art and Future Directions,

    J. Han, S. Sarica, F. Shi, and J. Luo, “Semantic Networks for Engineering Design: State of the Art and Future Directions,” J. Mech. Des., vol. 144, no. 2, Art. no. 2, Sep. 2021, doi: 10.1115/1.4052148

    2021 doi

  10. [19]

    A commentary of GPT-3 in MIT Technology Review 2021,

    M. Zhang and J. Li, “A commentary of GPT-3 in MIT Technology Review 2021,” Fundam. Res., vol. 1, no. 6, pp. 831–833, Nov. 2021, doi: 10.1016/j.fmre.2021.11.011

    2021 doi

  11. [20]

    GPT-4 Passes the Bar Exam,

    D. M. Katz, M. J. Bommarito, S. Gao, and P. Arredondo, “GPT-4 Passes the Bar Exam,” Mar. 15, 2023, Rochester, NY: 4389233. doi: 10.2139/ssrn.4389233

    2023 doi

  12. [21]

    ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope,

    P. P. Ray, “ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope,” Internet Things Cyber-Phys. Syst., vol. 3, pp. 121–154, Jan. 2023, doi: 10.1016/j.iotcps.2023.04.003

    2023 doi

  13. [22]

    Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models,

    J. Rasmussen, “Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models,” IEEE Trans. Syst. Man Cybern., vol. SMC-13, no. 3, pp. 257–266, May 1983, doi: 10.1109/TSMC.1983.6313160

    1983

  14. [23]

    Newell and H

    A. Newell and H. A. Simon, Human problem solving, vol. 104. Prentice-Hall Englewood Cliffs, NJ, 1972

    1972

  15. [24]

    The structure of ill structured problems,

    H. A. Simon, “The structure of ill structured problems,” Artif. Intell., vol. 4, no. 3, pp. 181–201, Dec. 1973, doi: 10.1016/0004- 3702(73)90011-8

    1973 doi

  16. [25]

    The Architecture of Complexity,

    H. A. Simon, “The Architecture of Complexity,” Proc. Am. Philos. Soc., vol. 106, pp. 468–482, 1962

    1962

  17. [26]

    Bar-yam, Dynamics Of Complex Systems

    Y. Bar-yam, Dynamics Of Complex Systems. CRC Press, 2019

    2019

  18. [27]

    Newell and H

    A. Newell and H. A. Simon, Human problem solving. in Human problem solving. Oxford, England: Prentice-Hall, 1972, pp. xiv, 920

    1972

  19. [28]

    GPT-3: Its Nature, Scope, Limits, and Consequences,

    L. Floridi and M. Chiriatti, “GPT-3: Its Nature, Scope, Limits, and Consequences,” Minds Mach., vol. 30, no. 4, pp. 681–694, Dec. 2020, doi: 10.1007/s11023-020-09548-1

    2020 doi

  20. [29]

    Impact of Chat GPT on Scientific Research: Opportunities, Risks, Limitations, and Ethical 10 Issues,

    O. I. Obaid, A. H. Ali, and M. G. Yaseen, “Impact of Chat GPT on Scientific Research: Opportunities, Risks, Limitations, and Ethical 10 Issues,” Iraqi J. Comput. Sci. Math., pp. 13–17, Sep. 2023, doi: 10.52866/ijcsm.2023.04.04.002

    2023 doi

  21. [30]

    Can Large Language Models Accelerate Digital Transformation by Generating Expert-Like Systems Engineering Artifacts? Insights from an Empirical Exploration,

    M. Husain, P. Wach, and T. G. Topcu, “Can Large Language Models Accelerate Digital Transformation by Generating Expert-Like Systems Engineering Artifacts? Insights from an Empirical Exploration,” in Conference on Systems Engineering Research, Springer, 2024, pp. 371–385. Acces...

    2024 doi

  22. [31]

    Microfoundations of strategic problem formulation,

    M. Baer, K. T. Dirks, and J. A. Nickerson, “Microfoundations of strategic problem formulation,” Strateg. Manag. J., vol. 34, no. 2, Art. no. 2, 2013, doi: 10.1002/smj.2004

    2013 doi

  23. [32]

    A facilitated expert-based approach to architecting ‘openable’ complex systems,

    Z. Szajnfarber and A. Vrolijk, “A facilitated expert-based approach to architecting ‘openable’ complex systems,” Syst. Eng., vol. 21, no. 1, Art. no. 1, 2018, doi: https://doi.org/10.1002/sys.21419

    2018 doi

  24. [33]

    Space mission analysis and design,

    W. J. Larson and J. R. Wertz, “Space mission analysis and design,” Microcosm, Inc., Torrance, CA (US), 1992. [Online]. Available: https://www.osti.gov/scitech/biblio/10146889

    1992

  25. [34]

    Towards a solver-aware systems architecting framework: leveraging experts, specialists and the crowd to design innovative complex systems,

    Z. Szajnfarber, T. G. Topcu, and H. Lifshitz-Assaf, “Towards a solver-aware systems architecting framework: leveraging experts, specialists and the crowd to design innovative complex systems,” Des. Sci., vol. 8, ed 2022, doi: 10.1017/dsj.2022.7

    2022 doi

  26. [35]

    Understanding the differences between how novice and experienced designers approach design tasks,

    S. Ahmed, K. M. Wallace, and L. T. Blessing, “Understanding the differences between how novice and experienced designers approach design tasks,” Res. Eng. Des., vol. 14, 32no. 1, pp. 1–11, Feb. 2003, doi: 10.1007/s00163-002-0023-z

    2003 doi

  27. [36]

    Expertise in design: an overview,

    N. Cross, “Expertise in design: an overview,” Des. Stud., vol. 25, no. 5, pp. 427–441, 2004

    2004

  28. [37]

    W. G. Vincenti, What engineers know and how they know it: analytical studies from aeronautical history. Baltimore: Johns Hopkins University Press, 1990

    1990

  29. [38]

    M. W. Maier, The art of systems architecting. CRC press, 2009

    2009

  30. [39]

    Skill Gaps, Skill Shortages and Skill Mismatches: Evidence for the US,

    P. Cappelli, “Skill Gaps, Skill Shortages and Skill Mismatches: Evidence for the US,” National Bureau of Economic Research, Cambridge, MA, w20382, Aug. 2014. doi: 10.3386/w20382

    2014 doi

  31. [40]

    Estimation of the workload boundary in socio-technical infrastructure management systems: The case of Belgian railroads,

    T. G. Topcu, K. Triantis, and B. Roets, “Estimation of the workload boundary in socio-technical infrastructure management systems: The case of Belgian railroads,” Eur. J. Oper. Res., vol. 278, no. 1, Art. no. 1, Oct. 2019, doi: 10.1016/j.ejor.2019.04.009

    2019 doi

  32. [41]

    How does agency impact human-AI collaborative design space exploration? A case study on ship design with deep generative models,

    S. Khan, P. Kaklis, and K. Goucher-Lambert, “How does agency impact human-AI collaborative design space exploration? A case study on ship design with deep generative models,” ArXiv Prepr. ArXiv230510451, 2023

    2023

  33. [42]

    Past Themes – RASC-AL

    “Past Themes – RASC-AL.” Accessed: Nov. 06, 2023. [Online]. Available: https://rascal.nianet.org/rascal-archives/past-themes/

    2023

  34. [43]

    MEG-2020.pdf

    “MEG-2020.pdf.” Accessed: Nov. 06, 2023. [Online]. Available: https://www.cto.mil/wp-content/uploads/2023/06/MEG-2020.pdf

    2020

  35. [44]

    Mission Engineering Integration and Interoperability (I&I),

    J. Moreland, “Mission Engineering Integration and Interoperability (I&I),” Lead. Edge, 2015

    2015

  36. [45]

    National Aeronautics and Space Act of 1958 (Unamended)

    “National Aeronautics and Space Act of 1958 (Unamended).” Accessed: Nov. 29, 2023. [Online]. Available: https://history.nasa.gov/spaceact.html

    1958

  37. [46]

    Mission Areas and Core Capabilities | FEMA.gov

    “Mission Areas and Core Capabilities | FEMA.gov.” Accessed: Nov. 14, 2024. [Online]. Available: https://www.fema.gov/emergency- managers/national-preparedness/mission-core-capabilities

    2024

  38. [47]

    Mission Engineering | CISA

    “Mission Engineering | CISA.” Accessed: Nov. 14, 2024. [Online]. Available: https://www.cisa.gov/about/divisions- offices/cybersecurity-division/mission-engineering

    2024

  39. [48]

    Kossiakoff, S

    A. Kossiakoff, S. M. Biemer, S. J. Seymour, and D. A. Flanigan, Systems Engineering Principles and Practice. John Wiley & Sons, 2020

    2020

  40. [49]

    A systems-theoretic articulation of stakeholder needs and system requirements,

    A. Salado, “A systems-theoretic articulation of stakeholder needs and system requirements,” Syst. Eng., vol. 24, no. 2, pp. 83–99, 2021

    2021

  41. [50]

    The concept of order of conflict in requirements engineering,

    A. Salado and R. Nilchiani, “The concept of order of conflict in requirements engineering,” IEEE Syst. J., vol. 10, no. 1, Art. no. 1, 2016

    2016

  42. [51]

    G. M. Weinberg, An Introduction to General Systems Thinking. Wiley, 1975

    1975

  43. [52]

    On the criteria to be used in decomposing systems into modules,

    D. L. Parnas, “On the criteria to be used in decomposing systems into modules,” Pioneers Their Contrib. Softw. Eng., vol. 15, no. 12, Art. no. 12, 1972

    1972

  44. [53]

    The Dark Side of Modularity: How Decomposing Problems can Increase System Complexity,

    T. G. Topcu, S. Mukherjee, A. I. Hennig, and Z. Szajnfarber, “The Dark Side of Modularity: How Decomposing Problems can Increase System Complexity,” J. Mech. Des., vol. 144, no. 3, Art. no. 3, 2021, doi: 10.1115/1.4052391

    2021 doi

  45. [54]

    Change propagation analysis in complex technical systems,

    M. Giffin, O. de Weck, G. Bounova, R. Keller, C. Eckert, and P. J. Clarkson, “Change propagation analysis in complex technical systems,” J. Mech. Des., vol. 131, no. 8, Art. no. 8, 2009

    2009

  46. [55]

    A systematic review of requirements change management,

    S. Jayatilleke and R. Lai, “A systematic review of requirements change management,” Inf. Softw. Technol., vol. 93, pp. 163–185, 2018

    2018

  47. [56]

    PaDGAN: Learning to Generate High-Quality Novel Designs,

    W. Chen and F. Ahmed, “PaDGAN: Learning to Generate High-Quality Novel Designs,” J. Mech. Des., vol. 143, no. 3, Art. no. 3, Mar. 2021, doi: 10.1115/1.4048626

    2021 doi

  48. [57]

    Human confidence in artificial intelligence and in themselves: The evolution and impact of confidence on adoption of AI advice,

    L. Chong, G. Zhang, K. Goucher-Lambert, K. Kotovsky, and J. Cagan, “Human confidence in artificial intelligence and in themselves: The evolution and impact of confidence on adoption of AI advice,” Comput. Hum. Behav., vol. 127, p. 107018, Feb. 2022, doi: 10.1016/j.chb.2021.107018

    2022

  49. [58]

    DesignQA: A Multimodal Benchmark for Evaluating Large Language Models’ Understanding of Engineering Documentation,

    A. C. Doris et al., “DesignQA: A Multimodal Benchmark for Evaluating Large Language Models’ Understanding of Engineering Documentation,” Aug. 23, 2024, arXiv: arXiv:2404.07917. Accessed: Nov. 10, 2024. [Online]. Available: http://arxiv.org/abs/2404.07917

    2024 arXiv

  50. [59]

    Knowledge cutoff date of September 2021 - ChatGPT,

    “Knowledge cutoff date of September 2021 - ChatGPT,” OpenAI Developer Forum. Accessed: Nov. 06, 2023. [Online]. Available: https://community.openai.com/t/knowledge-cutoff-date-of-september-2021/66215

    2021

  51. [60]

    Large Language Models: A Comprehensive Survey of its Applications, Challenges, Limitations, and Future Prospects,

    M. U. Hadi et al., “Large Language Models: A Comprehensive Survey of its Applications, Challenges, Limitations, and Future Prospects,” Nov. 16, 2023, TechRxiv. doi: 10.36227/techrxiv.23589741.v4

    2023 doi

  52. [61]

    LLMMaps -- A Visual Metaphor for Stratified Evaluation of Large Language Models,

    P. Puchert, P. Poonam, C. van Onzenoodt, and T. Ropinski, “LLMMaps -- A Visual Metaphor for Stratified Evaluation of Large Language Models,” Oct. 12, 2023, arXiv: arXiv:2304.00457. doi: 10.48550/arXiv.2304.00457

    2023 doi arXiv

  53. [62]

    GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models,

    I. Mirzadeh, K. Alizadeh, H. Shahrokhi, O. Tuzel, S. Bengio, and M. Farajtabar, “GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models,” Oct. 07, 2024, arXiv: arXiv:2410.05229. Accessed: Oct. 13, 2024. [Online]. Available: http://arxiv....

    2024 arXiv

  54. [63]

    Assessing Large Language Models Used for Extracting Table Information from Annual Financial Reports,

    D. Balsiger, H.-R. Dimmler, S. Egger-Horstmann, and T. Hanne, “Assessing Large Language Models Used for Extracting Table Information from Annual Financial Reports,” Computers, vol. 13, no. 10, Art. no. 10, Oct. 2024, doi: 10.3390/computers13100257

    2024 doi

  55. [64]

    Do ChatGPT 4o, 4, and 3.5 Generate ‘Similar’ Ratings? Findings and Implications,

    D. E. O’Leary, “Do ChatGPT 4o, 4, and 3.5 Generate ‘Similar’ Ratings? Findings and Implications,” IEEE Intell. Syst., vol. 39, no. 5, pp. 78–81, Sep. 2024, doi: 10.1109/MIS.2024.3441136

    2024

  56. [2021]

    Available: https://www.gao.gov/products/GAO-19-262SP

    [Online]. Available: https://www.gao.gov/products/GAO-19-262SP

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.