{"id":"4061dc06-b838-4902-89ad-3a008be5d592","arxiv_id":"2505.10640","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper catalogs the lifecycle stages and production-readiness challenges of software built around foundation models (FMware) and proposes an action plan of engineering practices and research directions.","lead":"This paper is a tutorial and survey that maps the challenges of building production software around large language models, from model selection to deployment. It offers a structured lifecycle and a roadmap of practices, aimed at researchers and engineers who want foundation-model systems to be reliable and trustworthy.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'systematic' catalogue omits prompt injection and related adversarial-input security risks, so the roadmap may misdirect production efforts.","rationale":"Reader's verdict identifies the right risk area: the catalogue's completeness and generality are the load-bearing assumptions. I agree partially, and sharpen it with a concrete, checkable omission. The paper presents itself as a comprehensive, systematically derived map of FMware challenges, but an established high-severity risk class — adversarial input security, especially prompt injection — is absent from lifecycle, challenges, and action plan. This is not a matter of differing from consensus; it is a correctness/completeness gap in the paper's own terms, since it claims systematic identification. It is also independent of the authors' enterprise context: prompt injection is reported across LLM applications, including enterprise and open-source settings, so it is not merely an omission from a niche ecosystem. The preferred fix is modest: either add a security-focused challenge/directive (or a cross-cutting security section), or qualify the 'systematic' claim to 'based on our industrial experience and selected literature'. Our concern does not overturn the paper's organizational value; a revision can resolve it, so the verdict remains conditional. Independent support: the paper is a tutorial/survey with 107 references and a coherent lifecycle; the gap is in coverage, not in internal logic. No formal verification or empirical validation is claimed, so none is missing. We therefore do not escalate to reject, and we keep the reader's CONDITIONAL verdict.","tokens_in":17687,"tokens_out":4808,"duration_ms":45313,"concrete_test":"Automated keyword audit: scan the manuscript (Sections 2-4) for a predefined set of established LLM-app risk terms: 'prompt injection', 'indirect prompt injection', 'jailbreak', 'data poisoning', 'supply chain', 'model stealing', 'sensitive information disclosure'. Also code the guarding subsection (§3.5.1) and cross-cutting concerns (§2.7) for any security category. If the keyword set yields zero occurrences and no security category is present, the omission is confirmed. As a stronger check, compare the catalogue's challenge list against OWASP Top 10 for LLM Applications 2025 and list which entries are absent. This directly settles whether the 'systematically identifies' claim is supportable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central value is a 'curated catalogue' (Abstract) and the claim that the authors 'systematically identify[] these challenges' (Section 1). For a production-readiness roadmap, this requires covering the known security risk surface of LLM-based systems. Yet the full text contains no occurrence of 'prompt injection', 'injection', 'jailbreak', 'data poisoning', or 'supply chain'. The guarding discussion (§3.5.1) addresses output filtering, semantic filtering, false positives/negatives, and confidence estimation, but never adversarial input manipulation such as direct or indirect prompt injection, which is the top OWASP LLM risk and a common real-world failure mode. Cross-cutting concerns (§2.7) list only Memory Management and Compliance; security has no dedicated lifecycle stage or cross-cutting section. The action plan (§4) has directives for observability, compliance, performance, and deployment, but no security directive. Consequently, the catalogue is not as comprehensive or systematic as claimed, and teams following the roadmap could miss a critical production risk class. This is a load-bearing gap for the roadmap's adequacy, not merely a stylistic omission.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper, written for a KDD 2025 tutorial, presents a survey and roadmap for building trustworthy, production-ready software systems that integrate foundation models as core components, which it terms FMware. It organizes the field into a lifecycle spanning FM selection, alignment, prompting, grounding, agents and orchestration, testing and optimization, observability, guarding, deployment, and cross-cutting concerns such as memory management and compliance. Drawing on industrial anecdotes and prior research, it argues that FMware engineering differs fundamentally from traditional software engineering, then proposes an action plan of directives organized by lifecycle stage. The central claims are that the paper systematically identifies the challenges of FMware and provides a technology roadmap for addressing them.","tokens_in":17829,"tokens_out":2008,"duration_ms":20828,"significance":"If the catalogue of challenges and the roadmap were comprehensive and well-validated, the paper would be a useful shared reference for both researchers and practitioners, synthesizing a fragmented literature and connecting it to concrete production concerns such as LinkedIn's extra four months of deployment effort, CodeCompose's model-sizing trade-offs, and ChatGPT's operating costs. The lifecycle framing and the emphasis on semantic observability, memory management, and compliance are timely and reflect real industrial experience. The paper is also honest about the immaturity of current practice, citing concrete evidence such as the moderate correlation of AI-as-judge evaluation with human judgment. However, the paper's contribution is qualitative and curated rather than systematically derived: none of the twenty-plus directives is empirically evaluated, and the catalogue is not benchmarked against an independently constructed taxonomy, so the roadmap's adequacy depends on the completeness of the authors' own challenge list.","major_comments":[{"comment":"The catalogue omits the entire class of adversarial-input security risks, most notably prompt injection, jailbreaks, indirect injection through retrieved or tool-returned content, data poisoning, and model supply-chain risks. The guarding discussion (§3.5.1) addresses output filtering, semantic filtering, false positives/negatives, and confidence estimation, but never considers that the inputs to an FMware system can be adversarial. Security is also absent from the cross-cutting concerns (§2.7), which list only Memory Management and Compliance, and the action plan (§4) contains no security directive. Because the paper's central claim in §1 is that it 'systematically identifies these challenges' and provides a production-readiness roadmap, this omission is load-bearing: teams following the roadmap would receive no guidance on what is widely regarded as one of the most critical production risk classes for LLM-based systems, corresponding to the top OWASP LLM risk. This should be remedied by adding a dedicated security subsection or cross-cutting concern, with corresponding directives, and by discussing the relationship between guarding and adversarial input robustness.","section":"§3.5.1 Guarding; §2.7 Cross-cutting concerns; §4 Action Plan"},{"comment":"The paper claims to 'systematically identify' the challenges of FMware, but it provides no survey methodology: no search strategy, inclusion/exclusion criteria, source selection process, or procedure for validating the representativeness of the challenge set. The catalogue appears to be heavily drawn from the authors' prior works [43], [76], [44], [57], and [30], and the paper does not compare its taxonomy with an independently derived one. This makes it impossible to assess whether the challenge list is comprehensive beyond the authors' large-enterprise industrial context, or whether the priority ordering of the roadmap would transfer to startups, open-source projects, low-resource deployments, or non-enterprise domains. The completeness of the catalogue is load-bearing for the roadmap claim, so the paper should either add a methodology subsection describing how the catalogue was assembled and validated, or soften the 'systematically identifies' claim to reflect the curated and experience-based nature of the survey.","section":"§1 Introduction; §2 FMware Lifecycle; §3 Challenges"},{"comment":"The action plan presents more than twenty directives (e.g., 'Multi-Prompt Architectures and Human-in-the-Loop Workflows', 'Automated FMwareBOM Generation and Formal Verification', 'Resource-Aware QA Framework and Smart Caching') without any evidence that these directives are effective, feasible, or prioritized relative to one another. The paper states that 'some directives can be somewhat readily applied to mitigate challenges, whereas others are research topics in themselves', but it does not say which are which, nor does it provide any case study, pilot, or empirical assessment. As a tutorial roadmap the proposals may be reasonable, but the manuscript's contribution framing in §1 — a roadmap for addressing the challenges — would be strengthened by explicitly marking each directive as validated practice, emerging practice, or open research, and by discussing any known limitations or trade-offs.","section":"§4 Action Plan"}],"minor_comments":[{"comment":"There is a grammatical slip in 'We then discuss key the challenges and present a critical overview' — 'key the' should be 'the key'.","section":"§1 Introduction"},{"comment":"Several sentences use informal constructions, such as 'Below we discuss about the different stages' and 'we henceforth present an action plan'; the paper would benefit from a light language pass for consistency with a formal proceedings style.","section":"§2.1, §2.2, §2.3"},{"comment":"The author list contains a formatting error: 'Gustavo A. Oliva1,2,,' has a doubled comma, and affiliation markers are inconsistently applied across the author list.","section":"Author block"},{"comment":"A few references are missing author or venue details, for example [79] lists 'RobBagby' as the author and [88] lists 'Unknown'; these should be completed for a camera-ready version.","section":"References"},{"comment":"The lifecycle figure uses the labels 'Agent(s)' and 'Agent orchestration' as separate boxes while the text in §2.3 discusses both under a single heading; the visual relationship between these boxes and the 'Guarding' and 'Memory management' boxes would be clearer if the figure legend explained the cross-cutting vs. sequential nature of the stages.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is essentially an extended tutorial abstract/survey positioned as a curated catalogue, and its main risk is that the 'systematic' framing overclaims relative to what is delivered. The missing security/adversarial-risk coverage is a concrete, fixable gap rather than a fundamental flaw, and the paper's industrial experience gives it genuine value. I would encourage the editor to ask for a revision that adds a security subsection or cross-cutting concern, clarifies the survey methodology or softens the systematic claim, and annotates the action plan directives by maturity level. These changes are within the manuscript's scope and would make the roadmap claim defensible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nQuick take: this is a competent tutorial survey, not a research paper. It repackages the authors' earlier FMware work (FSE 2024 catalogue [43], roadmap [76]) into a lifecycle plus action plan for a KDD audience. If you need a readable map of FMware engineering challenges, it works. If you're after new results, there are none.\n\nWhat's new and good: the lifecycle (selection, alignment, prompting, grounding, agents, testing, observability, guarding, compliance) is a clean organizing device, and the action plan contains a few concrete ideas—FMwareBOM, declarative intent-aware optimization, curriculum engineering—though most trace to the authors' prior roadmap. The paper is honest about building on [43] and [76], and the industry anecdotes (LinkedIn's four extra months, CodeCompose's model choice, ChatGPT's $700k/day) add practical grounding. The writing is clear and the references are extensive.\n\nSoft spots, in proportion. First, the 'systematically identifies these challenges' claim is overstated. There is no methodology for catalogue derivation, and the catalogue omits a whole risk class: adversarial input security. No mention of prompt injection, jailbreaks, data poisoning, or supply-chain attacks anywhere. The guarding section covers output filtering and confidence estimation but never adversarial inputs. For a roadmap aimed at trustworthy production-ready FMware, that is a real hole, not a nitpick. The action plan also lacks a security directive. Second, none of the directives are validated; they are reasonable hypotheses, but the paper doesn't frame them that way. Third, it says it 'improves' on [43] and [76] but never specifies what is new. That matters for a reader tracking the literature.\n\nWho it's for: practitioners and newcomers wanting a structured overview, and KDD tutorial attendees. Researchers will find little new. It deserves peer review as a tutorial paper, conditional on revision—the security gap needs addressing and the 'systematic' claim needs softening. I wouldn't cite it in my own work over the original FSE catalogue, but a serious editor should send it to review rather than desk reject.\n\nRecommendation: engage, ask for a revision that adds security (or at least a cross-cutting concern) and clearly delimits what this paper adds beyond prior work.","headline":"A well-organized tutorial survey that repackages the authors' own prior FMware work; useful for practitioners, but the 'systematic' catalogue misses adversarial security and the new-content claim needs clarifying.","tokens_in":18491,"tokens_out":3641,"would_cite":false,"duration_ms":33218,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FMware's scattered engineering pain points organize into one lifecycle, and a stage-by-stage action plan can move these systems from demos to trustworthy production.","keywords":["FMware","foundation models","large language models","AI4SE","software engineering","production readiness","LLM lifecycle","trustworthy AI"],"falsifier":"Survey developers of FMware outside large enterprises—startups, open-source maintainers, public-sector teams—and check whether the engineering problems they report map onto the lifecycle stages and directives. A substantial cluster of reported challenges that fits no stage or has no directive would falsify the roadmap's coverage claim.","tokens_in":17459,"feed_emoji":"🧭","tokens_out":8433,"duration_ms":71931,"temperature":0.7,"pith_summary":"FMware—software that integrates foundation models as core components—behaves unlike traditional software: outputs are stochastic, dependencies shift with model updates, and compute is expensive, so standard engineering practices break down. The paper argues that the resulting difficulties are not a random list but fall into a lifecycle spanning model selection, data and model alignment, prompting, grounding, agents and orchestration, testing and optimization, deployment and maintenance, with guarding, memory, and compliance as cross-cutting concerns. It pairs each stage with an action plan of concrete directives, some ready to apply and others open research problems, drawn from the authors' industrial experience and recent work. If the roadmap is right, teams gain a shared map of the field and a practical route from impressive demos to production-ready, trustworthy FMware.","feed_headline":"One lifecycle maps the road from FMware demo to production","feed_subtitle":"A curated catalogue of LLM engineering pains plus stage-by-stage directives aims to close the demo-to-production gap.","key_machinery":"The central object is the FMware lifecycle (Figure 1): a structured view that runs from FM selection through data and model alignment, prompting, grounding, agents and agent orchestration, system testing and optimization, and deployment and maintenance, with guarding, memory management, and compliance as cross-cutting layers. The lifecycle does the argument's work by converting a scattered catalogue of pain points into a shared vocabulary and a stage-by-stage map. The action plan mirrors the lifecycle stage-for-stage, so each directive names the stage it repairs; this one-to-one mapping between challenge and remedy is what gives the roadmap its practical force.","core_discovery":"On the paper's own terms, the contribution is the coherent organization itself. It claims that production-ready FMware has a distinct engineering identity: because model behavior changes with prompt wording, inference parameters, model updates, and input distributions, deterministic testing, observability, and compliance assumptions no longer hold. The survey assembles evidence for each lifecycle stage—ad hoc model selection among more than 1.4 million open-source models, brittle and non-portable prompts, hallucination-prone agent orchestration, AI judges correlating only moderately with human judgment (a reported 0.51), static guardrails that paraphrasing bypasses, and compliance tooling that does not span integrated FM components. It then claims these challenges are addressable through the action plan's directives, from semantic caching, prompt versioning, and structured retries to longer-range programs such as multi-level observability, FMwareBOM generation, and formal verification of compliance. A sympathetic reader takes away a structured map and a technology roadmap, not a proof that any single directive works.","pith_inferences":["If the lifecycle is a genuine map, it should pass a classification test: independent practitioners tagging the same set of FMware incidents should agree on which stage each incident belongs to; the paper does not run that validation.","The directives mix ready-to-apply tactics with research-level programs, so a natural extension—left implicit—is a maturity ranking that lets teams sequence adoption from low-effort wins such as semantic caching and prompt version control toward harder items such as formal verification and multi-level observability.","The reported 0.51 correlation for AI judges suggests an operational rule the paper does not state: treat LLM-as-judge as a triage filter that flags outputs for human review, not as a final oracle.","The stage-by-stage structure invites a benchmark suite: each open problem could become a measurable task, such as prompt portability across model families or cache correctness after model updates."],"forward_implications":["Teams can use the lifecycle as a pre-production checklist, making sure selection, alignment, prompting, grounding, orchestration, testing, observability, guarding, and compliance are each dealt with before release.","Testing must shift from asserting exact outputs to evaluating behavior across distributions, using continuous evaluation pipelines, dependency-aware regression testing, and metamorphic relations derived from user feedback.","Compliance needs FMware-wide artifacts: an FMwareBOM that extends SPDX 3.0 AI and dataset profiles to cover code, data, models, and agents, with automated license and regulatory checks.","Observability has to become semantic—tracing reasoning paths, agent coordination, and drift—not just collecting latency and token metrics.","Agent architecture should follow software-engineering discipline: single-responsibility components, permission models, structured human oversight, and controlled execution snapshots for reproducibility."],"supporting_citations":[{"why":"Supplies the curated catalogue of FMware challenges that this tutorial consolidates and extends.","marker":"[43]"},{"why":"The companion roadmap paper this work improves on; its action plan evolves from it.","marker":"[76]"},{"why":"The prior KDD tutorial survey on LLM-assisted development that this work contrasts with by studying FMware itself.","marker":"[29]"},{"why":"Industry evidence that validation and testing grow prohibitively expensive as FMware complexity rises.","marker":"[72]"},{"why":"Reports the 0.51 GPT-4-to-human correlation used to question AI-as-judge reliability.","marker":"[60]"},{"why":"The retrieval-augmented generation foundation for the paper's grounding discussion.","marker":"[56]"},{"why":"Provides the real-time text-generation safeguarding approach cited for guarding mechanisms.","marker":"[30]"},{"why":"An industry deployment account of four extra months spent refining the final 20% of a system, used as evidence that productionization is the hard part.","marker":"[2]"}],"fun_headline_variants":["FMware's path from demo to production, mapped in one lifecycle","A guide to making LLM-powered software trustworthy in production","From prompt jitters to compliance: a FMware roadmap","Tackling the 1.4M-model maze: production FMware guide","Closing the demo-to-production gap for FMware"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The roadmap depends on the curated catalogue of challenges being complete and representative enough to ground its directives; the paper does not show that a list built mainly from large-enterprise industrial experience covers startups, open-source projects, or low-resource deployments.","fun_headline_variants_meta":{"raw":{"variants":["FMware's path from demo to production, mapped in one lifecycle","A guide to making LLM-powered software trustworthy in production","From prompt jitters to compliance: a FMware roadmap","Tackling the 1.4M-model maze: production FMware guide","Closing the demo-to-production gap for FMware"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000529,"raw_usage":{"total_tokens":2529,"prompt_tokens":905,"completion_tokens":1624,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":1538}},"tokens_in":521,"tokens_out":1624,"duration_ms":9200,"temperature":1.0,"reasoning_tokens":1538,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:06:18.890717+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Survey developers of FMware outside large enterprises—startups, open-source maintainers, public-sector teams—and check whether the engineering problems they report map onto the lifecycle stages and directives. A substantial cluster of reported challenges that fits no stage or has no directive would falsify the roadmap's coverage claim.","supporting_citations":[{"cited_title":"Oliva, Dayi Lin, and Ahmed E","cited_arxiv_id":null,"evidence_quote":"The companion roadmap paper this work improves on; its action plan evolves from it."},{"cited_title":"Lewis, E","cited_arxiv_id":null,"evidence_quote":"The retrieval-augmented generation foundation for the paper's grounding discussion."}],"review_version":1}