Pith. sign in

REVIEW 4 major objections 5 minor 129 references

Trust at Your Own Peril: A Mixed Methods Exploration of the Ability of Large Language Models to Generate Expert-Like Systems Engineering Artifacts and a Characterization of Failure Modes

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Off-the-shelf large language models can produce systems engineering text that automated similarity metrics cannot distinguish from a human expert's, yet the same text carries serious, hard-to-detect failure modes: premature requirements…

desk verdict A transparent single-case mixed-methods study whose real contribution is the three failure modes, worth peer review despite an overclaimed MAUVE result and a ground-truth assumption that deserves scrutiny. read the letter →

arxiv 2502.09690 v1 pith:FCHUILD2 submitted 2025-02-13 cs.CL cs.AI

Taylan G. Topcu , Mohammed Husain , Max Ofsa , Paul Wach This is my paper · ORCID
classification cs.CLcs.AI
keywords systemsengineeringlargelanguagemodelsgenerativeAIhuman-AIcollaborationproblemformulationpromptfailuremodestextsimilarity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether off-the-shelf large language models, used without fine-tuning, can generate expert-grade systems engineering artifacts, and whether those artifacts deserve trust. It reports that with carefully engineered prompts, the most capable models produce text that a state-of-the-art similarity algorithm cannot reliably distinguish from a human-expert benchmark, with scores rising from near zero to above 0.9 once the prompt specifies exact response length. Yet a close qualitative reading of the highest-scoring AI output shows three serious failure modes: premature requirements definition, where the model writes binding 'shall' statements at a stage where the document should only bound the problem; unsubstantiated numerical estimates, whose asserted figures are internally inconsistent (such as a unit cost far exceeding total ownership cost); and a propensity to overspecify with unrequested or unverifiable bounds. The authors contend this is a cautionary tale: semantic similarity to expert prose is not evidence of engineering soundness, and blindly adopting AI-generated feedback in systems engineering can propagate misleading constraints into design decisions.

What carries the argument

The argument is carried by a two-stage comparison rig. Stage one is quantitative: the human-expert benchmark is chunked into 52 prompt-response pairs, three closed-source commercial LLMs are prompted under three configurations that vary in specificity, and a divergence-frontier text-similarity measure (a 0-1 score comparing machine and human text distributions through Kullback-Leibler divergence frontiers) selects the single most expert-like AI output set. The decisive prompt change was not domain content but a fixed response-length bound, which lifted similarity from near zero to above 0.9. Stage two is qualitative: two independent coders and a third synthesizer code the closest-matching AI artifacts against the benchmark, with an explicit counterexample screen, producing the three failure modes. That two-stage rig is what lets the paper claim simultaneously that the text is indistinguishable by machine and defective by expert judgment.

What would settle it

Have a panel of systems engineering experts independently produce artifacts for the same problem statement used here, then run the paper's full pipeline; if the LLM outputs fall inside the range of variation across the human experts' artifacts, or if blinded expert reviewers cannot reliably pick out the AI-generated artifacts as lower quality, the claimed contrast between 'expert-like similarity' and 'serious, detectable failure modes' would weaken.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is an asymmetry that automated evaluation misses: multi-purpose LLMs can imitate the surface of expert systems engineering work while failing the substance. Using a human-expert artifact set for a notional unmanned ground vehicle program as the benchmark, the paper chunks the artifacts into 52 prompt-response pairs and shows that, under the most specific prompt configuration, the resulting LLM text achieves similarity scores (0.91-0.99 on a divergence-frontier measure where 1 means indistinguishable from human text) while a generic prompt configuration scores near zero. The qualitative pass then shows the same text is not expert-quality: the model converts needs into binding requirements at the wrong stage of the development lifecycle, produces numerical thresholds and objectives without analytical basis and with internal contradictions (e.g., a $383M unit cost alongside a $25M total ownership cost), and layers on additional constraints that were neither requested nor traceable to the prompt. The authors characterize these as novice-like mistakes presented in expert-sounding language, and conclude that the systems engineering community should treat AI-suggested artifacts with caution until verification and validation methods catch up.

Load-bearing premise

The load-bearing premise is that the single human-expert artifact set used as the benchmark is the correct ground truth for what a systems engineering artifact should say, so that every deviation the AI makes is coded as a failure rather than as an alternative acceptable formulation.

Editorial extensions

If this is right

  • Automated text-similarity metrics are not sufficient certification for AI-generated systems engineering artifacts, since near-perfect similarity can coexist with serious content errors.
  • Prompt specificity, especially explicit length and scope constraints, is a major lever on output quality, so the same model can look incompetent or expert-like depending on who is driving the prompt.
  • If these failure modes persist, accepting AI-generated capability-document segments without expert review can inject over-constrained requirements, fabricated cost bounds, and unverifiable constraints into early design.
  • The useful near-term role for off-the-shelf LLMs in systems engineering is limited to formatting, summarization, and reframing of text, not the open-ended problem-formulation tasks tested here.
  • Newer LLMs may score even higher on similarity, which would make the failure modes harder to spot, not less relevant.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not attempt: rerun the same pipeline with prompts that explicitly forbid 'shall' statements, require a traceability note for every number, and instruct the model to flag estimates as unverified; if the three failure modes mostly disappear, they are partly prompt-controllable rather than intrinsic.
  • The same argument likely transfers to other high-stakes domains where documents are certified by style and surface completeness, such as policy, compliance, or medical documentation, a generalization the authors gesture at but do not develop.
  • An automated red-flag detector is within reach: since the paper documents a unit-cost estimate larger than the total-ownership-cost estimate in the same artifact set, a consistency check over numerical claims could catch the worst unsubstantiated estimates without needing an expert.
  • The single-benchmark design means the paper's qualitative conclusions are best read as existence proofs of failure modes, not as measurement of their frequency across the space of acceptable expert formulations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents an empirical study in which one human-expert systems engineering artifact set (the Bulldog UGV case study) is chunked into prompt-response pairs and fed to three closed-source LLMs (GPT-3.5 Turbo, GPT-4, Claude) through three increasingly specific prompting configurations. The generated artifact chunks are compared to the human benchmark with the MAUVE similarity metric, and the highest-scoring set (Claude, configuration 3) is then analyzed qualitatively. The paper reports that careful prompting yields high MAUVE similarity, but that the qualitative analysis reveals three failure modes: premature requirements definition, unsubstantiated numerical estimates, and propensity to overspecify. The authors frame the study as a cautionary tale about the risks of adopting multi-purpose LLM outputs in systems engineering without expert verification.

Significance. If the qualitative findings hold, the paper makes a useful contribution to the emerging literature on LLM use in systems engineering: it provides concrete prompt-response pairs, a transparent disclosure of non-blinding and stochasticity, and a plausible taxonomy of failure modes that could inform future verification and validation research. The study is refreshingly conservative in its framing and does not overstate the usefulness of LLMs for problem formulation. However, the central quantitative claim is stronger than the evidence, and the qualitative taxonomy is anchored to a single human-expert artifact. The paper's value is therefore as a case study with transferable insights, not as a general proof of indistinguishability or a validated taxonomy of LLM failure modes.

major comments (4)
  1. [Abstract and §4.1, Table 1] The abstract's claim that 'the state-of-the-art algorithms cannot differentiate AI-generated artifacts from the human-expert benchmark' is not supported by the evidence. MAUVE is a distributional similarity score, not a classification test; the paper reports one MAUVE value per model-configuration pair, with no confidence intervals, no repeated sampling, and no decision threshold. The correct statement is that MAUVE assigned high similarity in this single run. In addition, only MAUVE is used, so the plural 'algorithms' in the abstract overstates the scope. This is load-bearing because the abstract's framing is precisely the indistinguishability claim.
  2. [§3.1.2 and §4.2] The three failure modes are operationalized as deviations from a single human-expert artifact, the Bulldog case study. Section 3.1.2 explicitly concedes that 'there could possibly be other acceptable answers to a SE problem formulation question,' yet the qualitative coding in §4.2 treats every difference from Bulldog as a failure. For example, Table 9 codes the LLM's enumeration of sea/air/road/rail/helicopter transport as overconstraining, but an independent expert panel could reasonably accept that as an alternative bounding of the transportation OSA. Because the failure-mode taxonomy is the load-bearing contribution, the paper needs either an external expert panel to adjudicate the deviations or a systematic acknowledgment that these are differences from one artifact, not errors in any absolute sense.
  3. [§3.1.3, §3.2, Fig. 5] The MAUVE comparison is partially circular. The system prompt in Fig. 5 includes mission details from Bulldog, the user prompts are constructed from Bulldog text chunks, and the reference distribution for MAUVE is the same Bulldog text. High similarity therefore partly measures the model's ability to echo context that was supplied in the prompt. Additionally, Prompt Configuration 3's length bounding (Fig. 10) is a form of calibration, which sits uneasily with the abstract's claim that the procedure was applied 'without any fine-tuning or calibration.' The paper should report a control condition in which the model is prompted without the benchmark-derived context, or at minimum temper the wording from 'cannot differentiate' to 'received high similarity scores under this prompting protocol.'
  4. [§4.2] The qualitative analysis is conducted only on the single highest-MAUVE artifact set (Claude, configuration 3), and no inter-coder reliability statistics are reported. The paper states that two independent coders were used, but it does not report agreement metrics such as Cohen's kappa, and the coders knew they were analyzing LLM outputs. Since the failure-mode taxonomy is central to the paper's conclusions, the authors should report coding reliability and, ideally, apply the same coding to at least one additional model or prompting condition to support the generalization claim made in §4.2 and §5. This is a load-bearing support for the paper's main qualitative contribution.
minor comments (5)
  1. [§2.1.1] The historical timeline contains factual inaccuracies: GPT-3 was released in 2020, and ChatGPT was released in November 2022, not 2022/2023 as stated. These dates should be corrected.
  2. [Abstract] The phrase 'while the two-material appear very similar' contains a grammar error; it should read 'while the two materials appear very similar.'
  3. [Throughout] The name 'Bull Dog' is used with inconsistent capitalization ('Bulldog' vs. 'Bull Dog'); the authors should standardize the spelling.
  4. [§3.1.3] The text says the dataset is chunked into '50 instances' and then later states '52 prompt-response pairs'; the inconsistency should be resolved with a precise count.
  5. [Fig. 7] Figure 7 is credited as adopted from Pillutla et al. 49, but the caption does not include a permission or license note; if the figure is reproduced from a copyrighted source, a permissions statement should be added.

Circularity Check

1 steps flagged · score 4.0 of 10

Quantitative indistinguishability claim is partially circular because the Bulldog benchmark is both the prompt source and the MAUVE reference; qualitative failure modes remain independent.

  1. self definitional [Abstract; Section 3 Methodology; Section 3.1.3 Data Curation; Section 3.3.1 Quantitative Perspective]
    "Abstract: 'a human expert-generated SE artifact was taken as a benchmark, parsed, and fed into various LLMs through prompt engineering to generate segments of typical SE artifacts.' Section 3: 'We then provide various LLMs with chunks of curated data taken from this benchmark and evaluate their ability to generate SE artifacts.' Section 3.3.1: 'we will use the MAUVE algorithm to measure the similarities of human vs. AI generated text.'"

    The reference distribution P in MAUVE is the Bulldog text, and the same Bulldog text is the source of the 52 prompt-response pairs used to generate Q. The paper explicitly says it 'distil[s] Bulldog artifacts into a series of roughly 52 prompt-response pairs' and gives the LLM 'chunks of curated data taken from this benchmark.' The user prompts also instruct the model to write 'based on the information provided.' Therefore the MAUVE comparison measures similarity between the model's output and the corpus that was used to condition that output. High similarity is partly a prompt-following/echo effect, so the abstract's claim that the algorithms 'cannot differentiate' AI artifacts from the human benchmark is not an independent test of expert-like generation.

full rationale

The main derivation chain is empirical rather than analytic, so most potential circularity concerns are external-validity issues rather than definitional reductions. The one genuine structural circularity is in the quantitative claim: the Bulldog benchmark is simultaneously the source text for the system/user prompts and the reference distribution P for MAUVE. Section 3.1.3 says the authors 'distil Bulldog artifacts into a series of roughly 52 prompt-response pairs,' and Section 3 says the LLMs are given 'chunks of curated data taken from this benchmark'; Section 3.3.1 then compares Q against the human artifact with MAUVE. Consequently the high MAUVE scores in Table 1 partly measure the model's ability to follow and echo the very content it was given, so the abstract's 'cannot differentiate' overstates the independence of the test. This is not a fitted-parameter or self-citation reduction, and it does not infect the qualitative analysis: the three failure modes are coded with reference to external standards (CDD's problem-bounding role, plausibility of cost figures, traceability to the prompt) and are presented as deviations from a single expert artifact. The paper itself flags the single-ground-truth assumption in Section 3.1.2 ('there could possibly be other acceptable answers') and the non-blinded coding in Section 5.2, which is the right place for those concerns. No load-bearing self-citation is present; reference [48] only marks the conference predecessor. Overall, there is partial circularity in one supporting quantitative claim, while the central qualitative contribution retains independent content.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The paper fits no numerical model parameters, but the evaluation contains one hand-set control, response length per prompt, and several load-bearing assumptions. The most consequential is treating the single Bulldog artifact as ground truth for quality; the second is treating single-shot MAUVE at the minimum instance count as a stable measure of expert-likeness.

free parameters (1)
  • Response length target per prompt instance = Not reported; derived from the reference/label chunks
    Prompt Configuration 3 appended fixed response-length constraints 'based on the associated label/response' (Section 3.2, Figure 10). This hand-set evaluation control was the main driver of the MAUVE increase and is therefore a free parameter of the evaluation procedure.
assumptions (5)
  • domain assumption The Bulldog human-expert artifact is an appropriate ground truth for SE artifact quality.
    Section 3.1.2 deems the team-generated, DoD-approved Bulldog artifact suitable as ground truth; the qualitative analysis treats all deviations from it as deficiencies.
  • domain assumption LLM outputs can be treated as deterministic for a single-shot evaluation.
    Section 5.2 explicitly calls this a strong assumption; the study did not measure response variance or repeat runs.
  • standard math MAUVE with 52 instances is a valid measure of similarity between machine and human text.
    Section 3.3.1 adopts MAUVE and acknowledges the framework requires a minimum of 50 instances; the study uses only 52.
  • ad hoc to paper Length-bounding prompts to match the reference is legitimate prompt engineering rather than calibration.
    Prompt Configuration 3 set fixed response lengths 'based on the associated label/response' (Section 3.2), which materially inflates MAUVE scores; interpreting this as capability requires assuming this is not fitting.
  • domain assumption Closed-source models GPT-3.5 Turbo, GPT-4, and Claude are representative of multi-purpose LLMs.
    Section 3.1.1 selects three closed-source models as representative; generalization beyond them and to newer models is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Trust at Your Own Peril: A Mixed Methods Exploration of the Ability of Large Language Models to Generate Expert-Like Systems Engineering Artifacts and a Characterization of Failure Modes." pith.science (2026). https://pith.science/paper/FCHUILD2

@misc{pith2026250209690,
  author       = {Pith},
  title        = {Pith review of: Trust at Your Own Peril: A Mixed Methods Exploration of the Ability of Large Language Models to Generate Expert-Like Systems Engineering Artifacts and a Characterization of Failure Modes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FCHUILD2}},
  note         = {Machine review of arXiv:2502.09690}
}
read the original abstract

Multi-purpose Large Language Models (LLMs), a subset of generative Artificial Intelligence (AI), have recently made significant progress. While expectations for LLMs to assist systems engineering (SE) tasks are paramount; the interdisciplinary and complex nature of systems, along with the need to synthesize deep-domain knowledge and operational context, raise questions regarding the efficacy of LLMs to generate SE artifacts, particularly given that they are trained using data that is broadly available on the internet. To that end, we present results from an empirical exploration, where a human expert-generated SE artifact was taken as a benchmark, parsed, and fed into various LLMs through prompt engineering to generate segments of typical SE artifacts. This procedure was applied without any fine-tuning or calibration to document baseline LLM performance. We then adopted a two-fold mixed-methods approach to compare AI generated artifacts against the benchmark. First, we quantitatively compare the artifacts using natural language processing algorithms and find that when prompted carefully, the state-of-the-art algorithms cannot differentiate AI-generated artifacts from the human-expert benchmark. Second, we conduct a qualitative deep dive to investigate how they differ in terms of quality. We document that while the two-material appear very similar, AI generated artifacts exhibit serious failure modes that could be difficult to detect. We characterize these as: premature requirements definition, unsubstantiated numerical estimates, and propensity to overspecify. We contend that this study tells a cautionary tale about why the SE community must be more cautious adopting AI suggested feedback, at least when generated by multi-purpose LLMs.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

129 extracted references · 62 canonical work pages

  1. [1]

    Defense Acquisitions: Assessments of Selected Weapon Programs

    Government Accountability Office US. Defense Acquisitions: Assessments of Selected Weapon Programs. GAO; 2011

    2011

  2. [2]

    Navy Shipbuilding: Past Performance Provides Valuable Lessons for Future Investments.; 2018

    Government Accountability Office US. Navy Shipbuilding: Past Performance Provides Valuable Lessons for Future Investments.; 2018. Accessed February 25, 2021. https://www.gao.gov/products/GAO-18-238SP

    2018

  3. [3]

    F-35 Joint Strike Fighter: Development Is Nearly Complete, but Deficiencies Found in Testing Need to Be Resolved [Reissued with Revisions Jun

    Government Accountability Office US. F-35 Joint Strike Fighter: Development Is Nearly Complete, but Deficiencies Found in Testing Need to Be Resolved [Reissued with Revisions Jun. 13, 2018].; 2018. Accessed February 25, 2021. https://www.gao.gov/products/GAO-18- 321

    2018

  4. [4]

    NASA: Assessments of Major Projects.; 2019

    Government Accountability Office US. NASA: Assessments of Major Projects.; 2019. Accessed February 25, 2021. https://www.gao.gov/products/GAO-19-262SP

    2019

  5. [5]

    U. S. Government Accountability Office. James Webb Space Telescope: Technical Challenges Have Caused Schedule Strain and May Increase Costs. 2020;(GAO-20-224). Accessed February 25, 2021. https://www.gao.gov/products/GAO-20-224

    2020

  6. [6]

    F-35 Joint Strike Fighter: More Actions Needed to Explain Cost Growth and Support Engine Modernization Decision.; 2023

    Government Accountability Office US. F-35 Joint Strike Fighter: More Actions Needed to Explain Cost Growth and Support Engine Modernization Decision.; 2023. Accessed June 27,

    2023

  7. [7]

    Why are megaprojects, including nuclear power plants, delivered overbudget and late? reasons and remedies

    Locatelli G. Why are megaprojects, including nuclear power plants, delivered overbudget and late? reasons and remedies. arXiv preprint arXiv:180207312. Published online 2018

    2018

  8. [8]

    Complexity Is in the Eye of the Beholder: How Representative Complexity Measures Respond to the Commonly-Held Beliefs of the Literature

    Hennig A, Topcu TG, Szajnfarber Z. Complexity Is in the Eye of the Beholder: How Representative Complexity Measures Respond to the Commonly-Held Beliefs of the Literature. In: Proceeding of the ASME 2020 International Design Engineering Technical Conferences and Computers and Information Engineering Conference. ; 2021

    2020

Show all 129 references
  1. [9]

    How to deal with the complexity of future cyber-physical systems? Designs

    Törngren M, Grogan PT. How to deal with the complexity of future cyber-physical systems? Designs. 2018;2(4):40

    2018

  2. [10]

    Joint All Domain Command and Control: Background and Issues for Congress

    Hoehn JR. Joint All Domain Command and Control: Background and Issues for Congress. Published online 2021

    2021

  3. [11]

    AI4SE and SE4AI: A Research Roadmap

    McDermott T, DeLaurentis D, Beling P, Blackburn M, Bone M. AI4SE and SE4AI: A Research Roadmap. INSIGHT. 2020;23(1):8-14. doi:10.1002/inst.12278

    2020 doi

  4. [12]

    The magical number seven, plus or minus two: some limits on our capacity for processing information

    Miller GA. The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychological review. 1956;63(2):81

    1956

  5. [13]

    The Architecture of Complexity

    Simon HA. The Architecture of Complexity. Proceedings of the American Philosophical Society. 1962;106:468-482. 33

    1962

  6. [14]

    What Engineers Know and How They Know It: Analytical Studies from Aeronautical History

    Vincenti WG. What Engineers Know and How They Know It: Analytical Studies from Aeronautical History. Johns Hopkins University Press; 1990

    1990

  7. [15]

    Does Open Innovation Open Doors for Underrepresented Groups to Contribute to Technology Innovation?: Evidence from a Space Robotics Challenge

    Topcu TG, Zhang L “Lydia”, Szajnfarber Z. Does Open Innovation Open Doors for Underrepresented Groups to Contribute to Technology Innovation?: Evidence from a Space Robotics Challenge. Space Policy. 2023;64:101550. doi:10.1016/j.spacepol.2023.101550

    2023

  8. [16]

    A call for consensus on the use of representative model worlds in systems engineering and design

    Szajnfarber Z, Grogan PT, Panchal JH, Gralla EL. A call for consensus on the use of representative model worlds in systems engineering and design. Systems Engineering. 2020;23(4):436-442. doi:https://doi.org/10.1002/sys.21536

    2020 doi

  9. [17]

    Death of the Renaissance Man

    Jones BF. The Burden of Knowledge and the “Death of the Renaissance Man”: Is Innovation Getting Harder? The Review of Economic Studies. 2009;76(1):283-317. doi:10.1111/j.1467- 937X.2008.00531.x

    2009 arXiv

  10. [18]

    TWELVE SYSTEMS ENGINEERING ROLES

    Sheard SA. TWELVE SYSTEMS ENGINEERING ROLES. INCOSE International Symp. 1996;6(1):478-485. doi:10.1002/j.2334-5837.1996.tb02042.x

    1996

  11. [19]

    The structure of ill structured problems

    Simon HA. The structure of ill structured problems. Artificial Intelligence. 1973;4(3):181-

    1973

  12. [20]

    Problem forming, problem finding and problem solving in design

    Simon HA. Problem forming, problem finding and problem solving in design. Design & systems. Published online 1995:245-257

    1995

  13. [21]

    Elemental patterns of verification strategies

    Salado A, Kannan H. Elemental patterns of verification strategies. Systems Engineering. 2019;22(5):370-388. doi:10.1002/sys.21481

    2019 doi

  14. [22]

    Systems integration and architecting: An overview of principles, practices, and perspectives

    Sage AP, Lynch CL. Systems integration and architecting: An overview of principles, practices, and perspectives. Syst Engin. 1998;1(3):176-227. doi:10.1002/(SICI)1520- 6858(1998)1:3<176::AID-SYS3>3.0.CO;2-L

    1998 doi

  15. [23]

    Requirements development, verification, and validation exhibited in famous failures

    Bahill AT, Henderson SJ. Requirements development, verification, and validation exhibited in famous failures. Systems Engineering. 2005;8(1):1-14. doi:10.1002/sys.20017

    2005 doi

  16. [24]

    A Dynamic Theory of Organizational Knowledge Creation

    Nonaka I. A Dynamic Theory of Organizational Knowledge Creation. Organization Science. 1994;5(1):14-37

    1994

  17. [25]

    Organizational Learning: From Experience to Knowledge

    Argote L, Miron-Spektor E. Organizational Learning: From Experience to Knowledge. Organization Science. 2011;22(5):1123-1137. doi:10.1287/orsc.1100.0621

    2011

  18. [26]

    Transferring, translating, and transforming: An integrative framework for managing knowledge across boundaries

    Carlile PR. Transferring, translating, and transforming: An integrative framework for managing knowledge across boundaries. Organization science. 2004;15(5):555-568

    2004

  19. [27]

    Coping with changes: Causes, findings, and strategies

    Fricke E, Gebhard B, Negele H, Igenbergs E. Coping with changes: Causes, findings, and strategies. Syst Engin. 2000;3(4):169-179. doi:10.1002/1520-6858(2000)3:4<169::AID- SYS1>3.0.CO;2-W 34

    2000 doi

  20. [28]

    Workload-based change propagation analysis in engineering design

    Tang DB, Yin LL, Wang Q, Ullah I, Zhu HH, Leng S. Workload-based change propagation analysis in engineering design. Concurrent Engineering. 2016;24(1):17-34. doi:10.1177/1063293X15608510

    2016 doi

  21. [29]

    Information hiding in product development: the design churn effect

    Yassine A, Joglekar N, Braha D, Eppinger S, Whitney D. Information hiding in product development: the design churn effect. Res Eng Design. 2003;14(3):145-161. doi:10.1007/s00163-003-0036-2

    2003 doi

  22. [30]

    Towards Developing Metrics to Evaluate Digital Engineering

    Henderson K, McDermott T, Van Aken E, Salado A. Towards Developing Metrics to Evaluate Digital Engineering. Systems Engineering. 2023;26(1):3-31. doi:10.1002/sys.21640

    2023 doi

  23. [31]

    DoD Digital Engineering Strategy.; 2018:36

    Office of the Deputy Assistant Secretary of Defense for Systems Engineering. DoD Digital Engineering Strategy.; 2018:36

    2018

  24. [32]

    Digital engineering transformation across the Department of Defense

    Zimmerman P, Gilbert T, Salvatore F. Digital engineering transformation across the Department of Defense. The Journal of Defense Modeling and Simulation. 2019;16(4):325- 338

    2019

  25. [33]

    Why Digital Transformations Fail: The Surprising Disciplines of How to Take Off and Stay Ahead

    Saldanha T. Why Digital Transformations Fail: The Surprising Disciplines of How to Take Off and Stay Ahead. Berrett-Koehler Publishers; 2019

    2019

  26. [34]

    Digital Transformation: How to Beat the 90% Failure Rate? IEEE Engineering Management Review

    Ramesh N, Delen D. Digital Transformation: How to Beat the 90% Failure Rate? IEEE Engineering Management Review. 2021;49(3):22-25. doi:10.1109/EMR.2021.3070139

    2021

  27. [35]

    Artificial Intelligence and Future of Systems Engineering

    McDermott TA, Blackburn MR, Beling PA. Artificial Intelligence and Future of Systems Engineering. In: Lawless WF, Mittu R, Sofge DA, Shortell T, McDermott TA, eds. Systems Engineering and Artificial Intelligence. Springer International Publishing; 2021:47-59. doi:10.1007/978-3...

    2021 doi

  28. [36]

    Special Issue: Artificial Intelligence and Engineering Design

    Allison JT, Cardin MA, McComb C, et al., eds. Special Issue: Artificial Intelligence and Engineering Design. Journal of Mechanical Design. 2022;144(2). doi:10.1115/1.4053111

    2022 doi

  29. [37]

    A functional basis for engineering design: reconciling and evolving previous efforts

    Hirtz J, Stone RB, McAdams DA, Szykman S, Wood KL. A functional basis for engineering design: reconciling and evolving previous efforts. Research in engineering Design. 2002;13:65-82

    2002

  30. [38]

    Deployment of an ontological framework of functional design knowledge

    Kitamura Y, Kashiwase M, Fuse M, Mizoguchi R. Deployment of an ontological framework of functional design knowledge. Advanced Engineering Informatics. 2004;18(2):115-127

    2004

  31. [39]

    Daphne: An intelligent assistant for architecting earth observing satellite systems

    Bang H, Virós Martin A, Prat A, Selva D. Daphne: An intelligent assistant for architecting earth observing satellite systems. In: 2018 AIAA Information Systems-AIAA Infotech@ Aerospace. ; 2018:1366

    2018

  32. [40]

    Daphne: A virtual assistant for designing earth observation distributed spacecraft missions

    i Martin AV, Selva D. Daphne: A virtual assistant for designing earth observation distributed spacecraft missions. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing. 2019;13:30-48. 35

    2019

  33. [41]

    A systematic review of intelligent assistants

    Islas-Cota E, Gutierrez-Garcia JO, Acosta CO, Rodríguez LF. A systematic review of intelligent assistants. Future Generation Computer Systems. 2022;128:45-62

    2022

  34. [42]

    Interactive critiquing as a form of decision support: An empirical evaluation

    Guerlain SA, Smith PJ, Obradovich JH, et al. Interactive critiquing as a form of decision support: An empirical evaluation. Human factors. 1999;41(1):72-89

    1999

  35. [43]

    SEDAR: expert critiquing system for flat and low-slope roof design and review

    Fu MC, Hayes CC, East EW. SEDAR: expert critiquing system for flat and low-slope roof design and review. Journal of Computing in Civil Engineering. 1997;11(1):60-68

    1997

  36. [44]

    When faced with increasing complexity: the effectiveness of artificial intelligence assistance for drone design

    Song B, Soria Zurita NF, Nolte H, Singh H, Cagan J, McComb C. When faced with increasing complexity: the effectiveness of artificial intelligence assistance for drone design. Journal of Mechanical Design. 2022;144(2)

    2022

  37. [45]

    Decoding the agility of artificial intelligence-assisted human design teams

    Song B, Gyory JT, Zhang G, et al. Decoding the agility of artificial intelligence-assisted human design teams. Design Studies. 2022;79:101094

    2022

  38. [46]

    Evaluating Designer Learning and Performance in Interactive Deep Generative Design

    Chaudhari AM, Selva D. Evaluating Designer Learning and Performance in Interactive Deep Generative Design. Journal of Mechanical Design. 2023;145(5):051403

    2023

  39. [47]

    Complex Problem Solving through Human-AI Collaboration: Literature Review on Research Contexts

    Memmert L, Bittner EA. Complex Problem Solving through Human-AI Collaboration: Literature Review on Research Contexts. In: HICSS. ; 2022:1-10

    2022

  40. [48]

    Can Large Language Models Accelerate Digital Transformation by Generating Expert-Like Systems Engineering Artifacts? Insights from an Empirical Exploration

    Husain M, Wach P, Topcu TG. Can Large Language Models Accelerate Digital Transformation by Generating Expert-Like Systems Engineering Artifacts? Insights from an Empirical Exploration. In: Conference on Systems Engineering Research. Springer; 2024:371-385. Accessed August 21, ...

    2024 doi

  41. [49]

    Mauve: Measuring the gap between neural text and human text using divergence frontiers

    Pillutla K, Swayamdipta S, Zellers R, et al. Mauve: Measuring the gap between neural text and human text using divergence frontiers. Advances in Neural Information Processing Systems. 2021;34:4816-4828

    2021

  42. [50]

    Leveraging Large Language Models for Tradespace Exploration

    Apaza G, Selva D. Leveraging Large Language Models for Tradespace Exploration. Journal of Spacecraft and Rockets. Published online May 8, 2024:1-19. doi:10.2514/1.A35834

    2024 doi

  43. [51]

    Spacetransformers: Language modeling for space systems

    Berquand A, Darm P, Riccardi A. Spacetransformers: Language modeling for space systems. IEEE Access. 2021;9:133111-133122

    2021

  44. [52]

    SafeAeroBERT: Towards a Safety-Informed Aerospace-Specific Language Model

    Andrade SR, Walsh HS. SafeAeroBERT: Towards a Safety-Informed Aerospace-Specific Language Model. In: AIAA AVIATION 2023 Forum. American Institute of Aeronautics and Astronautics; 2023. doi:10.2514/6.2023-3437

    2023 doi

  45. [53]

    Daniel Jurafsky, James H. Martin. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Prentice Hall; 2008. Accessed May 14, 2024. https://scholar.google.com/citations?view_op=view_citation&hl=en&use...

    2008

  46. [54]

    Rule-based systems

    Hayes-Roth F. Rule-based systems. Commun ACM. 1985;28(9):921-932. doi:10.1145/4284.4286

    1985

  47. [55]

    ELIZA—a computer program for the study of natural language communication between man and machine

    Weizenbaum J. ELIZA—a computer program for the study of natural language communication between man and machine. Commun ACM. 1966;9(1):36-45. doi:10.1145/365153.365168

    1966

  48. [56]

    Foundations of Statistical Natural Language Processing

    Manning C, Schutze H. Foundations of Statistical Natural Language Processing. MIT press

  49. [57]

    Prediction and Entropy of Printed English

    Shannon CE. Prediction and Entropy of Printed English. Bell System Technical Journal. 1951;30(1):50-64. doi:10.1002/j.1538-7305.1951.tb01366.x

    1951

  50. [58]

    The DRAGON system–An overview

    Baker J. The DRAGON system–An overview. IEEE Transactions on Acoustics, speech, and signal Processing. 1975;23(1):24-29

    1975

  51. [59]

    Perplexity—a measure of the difficulty of speech recognition tasks

    Jelinek F, Mercer RL, Bahl LR, Baker JK. Perplexity—a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America. 1977;62(S1):S63-S63

    1977

  52. [60]

    Learning representations by back-propagating errors

    Rumelhart DE, Hinton GE, Williams RJ. Learning representations by back-propagating errors. nature. 1986;323(6088):533-536

    1986

  53. [61]

    Finding Structure in Time

    Elman JL. Finding Structure in Time. Cognitive Science. 1990;14(2):179-211. doi:10.1207/s15516709cog1402_1

    1990 doi

  54. [62]

    Long short-term memory

    Hochreiter S, Schmidhuber J. Long short-term memory. Neural computation. 1997;9(8):1735-1780

    1997

  55. [63]

    Sequence to sequence learning with neural networks

    Sutskever I, Vinyals O, Le QV. Sequence to sequence learning with neural networks. Advances in neural information processing systems. 2014;27. Accessed May 14, 2024. https://proceedings.neurips.cc/paper/2014/hash/a14ac55a4f27472c5d894ec1c3c743d2- Abstract.html

    2014

  56. [64]

    Attention is All you Need

    Ashish V. Attention is All you Need. Advances in Neural Information Processing Systems. 2017;30:I

    2017

  57. [65]

    Emergent Abilities of Large Language Models

    Wei J, Tay Y, Bommasani R, et al. Emergent Abilities of Large Language Models. Published online October 26, 2022. Accessed November 15, 2023. http://arxiv.org/abs/2206.07682

    2022 arXiv

  58. [66]

    Policy shaping: Integrating human feedback with reinforcement learning

    Griffith S, Subramanian K, Scholz J, Isbell CL, Thomaz AL. Policy shaping: Integrating human feedback with reinforcement learning. Advances in neural information processing systems. 2013;26. Accessed November 15, 2023. https://proceedings.neurips.cc/paper_files/paper/2013/hash...

    2013

  59. [67]

    AI Prompt Engineering Isn’t the Future

    Acar OA. AI Prompt Engineering Isn’t the Future. HARVARD BUSINESS REVIEW. Published online 2023

    2023

  60. [68]

    The role of natural language in requirements engineering

    Ryan K. The role of natural language in requirements engineering. In: [1993] Proceedings of the IEEE International Symposium on Requirements Engineering. IEEE; 1993:240-242. Accessed August 8, 2024. https://ieeexplore.ieee.org/abstract/document/324852/

    1993

  61. [69]

    State of practice in requirements engineering: contemporary data

    Kassab M, Neill C, Laplante P. State of practice in requirements engineering: contemporary data. Innovations Syst Softw Eng. 2014;10(4):235-241. doi:10.1007/s11334-014-0232-4

    2014 doi

  62. [70]

    Natural language processing for requirements engineering: The best is yet to come

    Dalpiaz F, Ferrari A, Franch X, Palomares C. Natural language processing for requirements engineering: The best is yet to come. IEEE software. 2018;35(5):115-119

    2018

  63. [71]

    Norbert: Transfer learning for requirements classification

    Hey T, Keim J, Koziolek A, Tichy WF. Norbert: Transfer learning for requirements classification. In: 2020 IEEE 28th International Requirements Engineering Conference (RE). IEEE; 2020:169-179. Accessed August 8, 2024. https://ieeexplore.ieee.org/abstract/document/9218141/

    2020

  64. [72]

    Extracting and classifying requirements from software engineering contracts

    Sainani A, Anish PR, Joshi V, Ghaisas S. Extracting and classifying requirements from software engineering contracts. In: 2020 IEEE 28th International Requirements Engineering Conference (RE). IEEE; 2020:147-157. Accessed August 8, 2024. https://ieeexplore.ieee.org/abstract/do...

    2020

  65. [73]

    Semantically enhanced software traceability using deep learning techniques

    Guo J, Cheng J, Cleland-Huang J. Semantically enhanced software traceability using deep learning techniques. In: 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). IEEE; 2017:3-14. Accessed August 8, 2024. https://ieeexplore.ieee.org/abstract/document/7985645/

    2017

  66. [74]

    Traceability transformed: Generating more accurate links with pre-trained bert models

    Lin J, Liu Y, Zeng Q, Jiang M, Cleland-Huang J. Traceability transformed: Generating more accurate links with pre-trained bert models. In: 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE; 2021:324-335. Accessed August 8,

    2021

  67. [75]

    Eliciting attribute-level user needs from online reviews with deep language models and information extraction

    Han Y, Moghaddam M. Eliciting attribute-level user needs from online reviews with deep language models and information extraction. Journal of Mechanical Design. 2021;143(6):061403

    2021

  68. [76]

    DesignQA: A Multimodal Benchmark for Evaluating Large Language Models’ Understanding of Engineering Documentation

    Doris AC, Grandi D, Tomich R, et al. DesignQA: A Multimodal Benchmark for Evaluating Large Language Models’ Understanding of Engineering Documentation. Published online August 23, 2024. Accessed September 2, 2024. http://arxiv.org/abs/2404.07917

    2024 arXiv

  69. [77]

    Deep Generative Models in Engineering Design: A Review

    Regenwetter L, Nobari AH, Ahmed F. Deep Generative Models in Engineering Design: A Review. Journal of Mechanical Design. 2022;144(071704). doi:10.1115/1.4053859

    2022 doi

  70. [78]

    Idea generation with Technology Semantic Network

    Sarica S, Song B, Luo J, Wood KL. Idea generation with Technology Semantic Network. Artificial Intelligence for Engineering Design, Analysis and Manufacturing : AI EDAM. 2021;35(3):265-283. doi:10.1017/S0890060421000020 38

    2021 doi

  71. [79]

    PaDGAN: Learning to Generate High-Quality Novel Designs

    Chen W, Ahmed F. PaDGAN: Learning to Generate High-Quality Novel Designs. Journal of Mechanical Design. 2021;143(3):031703. doi:10.1115/1.4048626

    2021 doi

  72. [80]

    Conceptual design generation using large language models

    Ma K, Grandi D, McComb C, Goucher-Lambert K. Conceptual design generation using large language models. In: International Design Engineering Technical Conferences and Computers and Information in Engineering Conference. Vol 87349. American Society of Mechanical Engineers; 2023:...

    2023

  73. [81]

    LLM4CAD: Multi-Modal Large Language Models For 3D Computer- Aided Design Generation

    Li X, Sun Y, Sha Z. LLM4CAD: Multi-Modal Large Language Models For 3D Computer- Aided Design Generation. In: ASME IDETC-DTM. ; 2024. Accessed September 2, 2024. https://sidilab.net/wp-content/uploads/2024/07/idetc2024_llm4cad_final.pdf

    2024

  74. [82]

    AutoTRIZ: Artificial Ideation with TRIZ and Large Language Models

    Jiang S, Luo J. AutoTRIZ: Artificial Ideation with TRIZ and Large Language Models. Published online May 22, 2024. Accessed September 2, 2024. http://arxiv.org/abs/2403.13002

    2024

  75. [83]

    Daphne: A Virtual Assistant for Designing Earth Observation Distributed Spacecraft Missions

    Martin AVI, Selva D. Daphne: A Virtual Assistant for Designing Earth Observation Distributed Spacecraft Missions. IEEE J Sel Top Appl Earth Observations Remote Sensing. 2020;13:30-48. doi:10.1109/JSTARS.2019.2948921

    2020

  76. [84]

    Human Versus Artificial Intelligence: A Data- Driven Approach to Real-Time Process Management During Complex Engineering Design

    Gyory JT, Soria Zurita NF, Martin J, et al. Human Versus Artificial Intelligence: A Data- Driven Approach to Real-Time Process Management During Complex Engineering Design. Journal of Mechanical Design. 2021;144(2). doi:10.1115/1.4052488

    2021 doi

  77. [85]

    Learning Comes from Experience: The Effects on Human Learning and Performance of a Virtual Assistant for Design Space Exploration

    Viros I Martin A, Selva D. Learning Comes from Experience: The Effects on Human Learning and Performance of a Virtual Assistant for Design Space Exploration. In: Gero JS, ed. Design Computing and Cognition’20. Springer International Publishing; 2022:655-665. doi:10.1007/978-3-...

    2022 doi

  78. [86]

    Human confidence in artificial intelligence and in themselves: The evolution and impact of confidence on adoption of AI advice

    Chong L, Zhang G, Goucher-Lambert K, Kotovsky K, Cagan J. Human confidence in artificial intelligence and in themselves: The evolution and impact of confidence on adoption of AI advice. Computers in Human Behavior. 2022;127:107018. doi:10.1016/j.chb.2021.107018

    2022

  79. [87]

    Trust, Workload, and Performance in Human–Artificial Intelligence Partnering: The Role of Artificial Intelligence Attributes in Solving Classification Problems

    Lotfalian Saremi M, Ziv I, Asan O, Bayrak AE. Trust, Workload, and Performance in Human–Artificial Intelligence Partnering: The Role of Artificial Intelligence Attributes in Solving Classification Problems. Journal of Mechanical Design. 2025;147(1). Accessed August 9, 2024. ht...

    2025

  80. [88]

    A cautionary tale about the impact of AI on human design teams

    Zhang G, Raina A, Cagan J, McComb C. A cautionary tale about the impact of AI on human design teams. Design Studies. 2021;72:100990

    2021

  81. [89]

    A strategic decision-making architecture toward hybrid teams for dynamic competitive problems

    Bayrak AE, McComb C, Cagan J, Kotovsky K. A strategic decision-making architecture toward hybrid teams for dynamic competitive problems. Decision Support Systems. 2021;144:113490. doi:10.1016/j.dss.2020.113490 39

    2021

  82. [90]

    Unifying Large Language Models and Knowledge Graphs: A Roadmap

    Pan S, Luo L, Wang Y, Chen C, Wang J, Wu X. Unifying Large Language Models and Knowledge Graphs: A Roadmap. Published online June 20, 2023. Accessed November 15,

    2023

  83. [91]

    Data-Driven Innovation: What is it? IEEE Transactions on Engineering Management

    Luo J. Data-Driven Innovation: What is it? IEEE Transactions on Engineering Management. Published online 2022:1-7. doi:10.1109/TEM.2022.3145231

    2022

  84. [92]

    Advancing Education on Digital Artifacts

    Wach P, Clark D, Geoff K, et al. Advancing Education on Digital Artifacts. In: Verma D, Madni AM, Hoffenson S, Xiao L, eds. The Proceedings of the 2023 Conference on Systems Engineering Research. Conference on Systems Engineering Research Series. Springer Nature Switzerland; 2...

    2023 doi

  85. [93]

    Databricks

    Best Practices for LLM Evaluation of RAG Applications. Databricks. September 12, 2023. Accessed August 14, 2024. https://www.databricks.com/blog/LLM-auto-eval-best-practices- RAG

    2023

  86. [94]

    When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method

    Zhang B, Liu Z, Cherry C, Firat O. When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method. Published online February 27, 2024. Accessed November 20, 2024. http://arxiv.org/abs/2402.17193

    2024 arXiv

  87. [95]

    http://arxiv.org/abs/2306.08302

    arXiv

  88. [96]

    BigTranslate: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languages

    Yang W, Li C, Zhang J, Zong C. BigTranslate: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languages. Published online November 21,

  89. [97]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy A. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:201011929. Published online 2020

    2020

  90. [98]

    The Power of Scale for Parameter-Efficient Prompt Tuning

    Lester B, Al-Rfou R, Constant N. The Power of Scale for Parameter-Efficient Prompt Tuning. Published online September 2, 2021. Accessed November 20, 2024. http://arxiv.org/abs/2104.08691

    2021 arXiv

  91. [99]

    Scaling Laws for Transfer

    Hernandez D, Kaplan J, Henighan T, McCandlish S. Scaling Laws for Transfer. Published online February 2, 2021. Accessed November 20, 2024. http://arxiv.org/abs/2102.01293

    2021 arXiv

  92. [100]

    Training language models to follow instructions with human feedback

    Ouyang L, Wu J, Jiang X, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems. 2022;35:27730-27744

    2022

  93. [101]

    Bleu: a method for automatic evaluation of machine translation

    Papineni K, Roukos S, Ward T, Zhu WJ. Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. ; 2002:311-318. Accessed November 15, 2023. https://aclanthology.org/P02-1040.pdf 40

    2002

  94. [102]

    http://arxiv.org/abs/2305.18098

    Accessed November 20, 2024. http://arxiv.org/abs/2305.18098

    2024 arXiv

  95. [103]

    Evaluating quantitative measures for assessing functional similarity in engineering design

    Nandy A, Dong A, Goucher-Lambert K. Evaluating quantitative measures for assessing functional similarity in engineering design. Journal of Mechanical Design. 2022;144(3):031401

    2022

  96. [104]

    Evaluating Compositionality in Sentence Embeddings

    Dasgupta I, Guo D, Stuhlmüller A, Gershman SJ, Goodman ND. Evaluating Compositionality in Sentence Embeddings. Published online May 17, 2018. Accessed October 31, 2024. http://arxiv.org/abs/1802.04302

    2018 arXiv

  97. [105]

    The Curious Case of Neural Text Degeneration

    Holtzman A, Buys J, Du L, Forbes M, Choi Y. The Curious Case of Neural Text Degeneration. Published online February 14, 2020. Accessed August 11, 2024. http://arxiv.org/abs/1904.09751

    2020 arXiv

  98. [106]

    Understanding the Performance and Estimating the Cost of LLM Fine-Tuning

    Xia Y, Kim J, Chen Y, et al. Understanding the Performance and Estimating the Cost of LLM Fine-Tuning. Published online August 8, 2024. Accessed October 31, 2024. http://arxiv.org/abs/2408.04693

    2024 arXiv

  99. [107]

    Building Theories from Case Study Research

    Eisenhardt KM. Building Theories from Case Study Research. The Academy of Management Review. 1989;14(4):532-550. doi:10.2307/258557

    1989 doi

  100. [108]

    Rouge: A package for automatic evaluation of summaries

    Lin CY. Rouge: A package for automatic evaluation of summaries. In: Text Summarization Branches Out. ; 2004:74-81. Accessed November 15, 2023. https://aclanthology.org/W04-1013.pdf

    2004

  101. [109]

    Qualitative Data Analysis: An Expanded Sourcebook

    Miles MB, Huberman AM. Qualitative Data Analysis: An Expanded Sourcebook. SAGE; 1994

    1994

  102. [110]

    Applied Space Systems Engineering

    Larson W, Kirkpatrick D, Sellers J, Thomas L, Verma D. Applied Space Systems Engineering. McGraw-Hill Education; 2009

    2009

  103. [111]

    DoDAF Architecture Framework Version 2.02.; 2010

    US DoD. DoDAF Architecture Framework Version 2.02.; 2010. https://dodcio.defense.gov/Library/DoD-Architecture-Framework/

    2010

  104. [112]

    Case Study Research: Design and Methods

    Yin RK. Case Study Research: Design and Methods. SAGE; 2003

    2003

  105. [113]

    U. S. Government Accountability Office (GAO). F-35 Sustainment: Costs Continue to Rise While Planned Use and Availability Have Decreased | U.S. GAO.; 2024. Accessed August 12, 2024. https://www.gao.gov/products/gao-24-106703

    2024

  106. [114]

    Qualitative methods for engineering systems: Why we need them and how to use them

    Szajnfarber Z, Gralla E. Qualitative methods for engineering systems: Why we need them and how to use them. Systems Engineering. 2017;20(6):497-511. doi:https://doi.org/10.1002/sys.21412

    2017 doi

  107. [115]

    Mission Engineering Guide

    OUSD(R&E). Mission Engineering Guide. Office of the Under Secretary of Defense for Research and Engineering; 2020. https://ac.cto.mil/wp-content/uploads/2020/12/MEG- v40_20201130_shm.pdf 41

    2020

  108. [116]

    MIL-STD 810G Department of Defense Test Method Standard: Environmental Engineering Coniderations and Laboratory Tests

    US DoD. MIL-STD 810G Department of Defense Test Method Standard: Environmental Engineering Coniderations and Laboratory Tests. Published online 2008

    2008

  109. [117]

    GPT-4 passes the bar exam

    Katz DM, Bommarito MJ, Gao S, Arredondo P. GPT-4 passes the bar exam. Phil Trans R Soc A. 2024;382(2270):20230254. doi:10.1098/rsta.2023.0254

    2024

  110. [118]

    Projected Acquisition Costs for the Army’s Ground Combat Vehicles.; 2021:17

    Congressional Budget Office (CBO). Projected Acquisition Costs for the Army’s Ground Combat Vehicles.; 2021:17. www.cbo.gov/publication/57085

    2021

  111. [119]

    ChatGPT passing USMLE shines a spotlight on the flaws of medical education

    Mbakwe AB, Lourentzou I, Celi LA, Mechanic OJ, Dagan A. ChatGPT passing USMLE shines a spotlight on the flaws of medical education. PLOS Digit Health. 2023;2(2):e0000205. doi:10.1371/journal.pdig.0000205

    2023 doi

  112. [120]

    of” vs. “in

    Sapol S, Szajnfarber Z. Flexibility “of” vs. “in” Systems: A Complementary Strategy for Designing Fleet-Based Systems for Uncertainty. Journal of Mechanical Design. Published online December 3, 2021:1-19. doi:10.1115/1.4053157

    2021 doi

  113. [121]

    LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples

    Yao JY, Ning KP, Liu ZH, Ning MN, Liu YY, Yuan L. LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples. Published online August 4, 2024. Accessed August 24, 2024. http://arxiv.org/abs/2310.01469

    2024 arXiv

  114. [122]

    Truth-o-meter: Collaborating with llm in fighting its hallucinations

    Galitsky BA. Truth-o-meter: Collaborating with llm in fighting its hallucinations. Published online 2023. Accessed August 24, 2024. https://www.preprints.org/manuscript/202307.1723

    2023

  115. [123]

    An Empirical Exploration of ChatGPT’s Ability to Support Problem Formulation Tasks for Mission Engineering and a Documentation of its Performance Variability

    Ofsa M, Topcu TG. An Empirical Exploration of ChatGPT’s Ability to Support Problem Formulation Tasks for Mission Engineering and a Documentation of its Performance Variability. Published online February 5, 2025. doi:10.48550/arXiv.2502.03511

    2025 doi arXiv

  116. [124]

    Performance of GPT-3.5 and GPT-4 on the Japanese medical licensing examination: comparison study

    Takagi S, Watari T, Erabi A, Sakaguchi K. Performance of GPT-3.5 and GPT-4 on the Japanese medical licensing examination: comparison study. JMIR Medical Education. 2023;9(1):e48002

    2023

  117. [126]

    Human Problem Solving

    Newell A, Simon HA. Human Problem Solving. Vol 104. Prentice-Hall Englewood Cliffs, NJ; 1972

    1972

  118. [201]

    doi:10.1016/0004-3702(73)90011-8

    doi

  119. [1999]

    Accessed May 14, 2024. https://books.google.com/books?hl=en&lr=&id=YiFDxbEX3SUC&oi=fnd&pg=PR16&dq= %5B4%5D+Manning,+Christopher,+and+Hinrich+Schutze.+Foundations+of+statistical+nat ural+language+processing.+MIT+press,+1999.&ots=v0rkttiLRJ&sig=OJ9H2N6- RfoN2Jy0ErMiILrEgls

    2024

  120. [2023]

    https://www.gao.gov/products/gao-23-106047

  121. [2024]

    https://ieeexplore.ieee.org/abstract/document/9402118/

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.