Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

A Functional Software Reference Architecture for LLM-Integrated Systems

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes a preliminary functional reference architecture with four layers—Presentation, Application logic, LLM integration, and Data management—plus Monitoring and Guardrail sidecars, and validates it by mapping three…

desk verdict A readable, honest preliminary reference architecture for LLM-integrated systems; the validation is descriptive rather than testing, but the paper is worth refereeing as an early-stage framework proposal. read the letter →

arxiv 2501.12904 v1 pith:ICOQ3CMP submitted 2025-01-22 cs.SE

Alessio Bucaioni , Martin Weyssow , Junda He , Yunbo Lyu , David Lo This is my paper · ORCID
classification cs.SE
keywords softwarereferencearchitecturefunctionallargelanguagemodelsLLM-integratedsystemsarchitecturalconcernsvalidationmonitoringguardrails
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that LLM-integrated software systems lack a commonly accepted reference architecture, leaving designers without systematic guidance on security, privacy, modularity, and interoperability. It proposes a preliminary functional reference architecture organized into four layers—Presentation, Application logic, LLM integration, and Data management—with Monitoring and Guardrail sidecars that cut across them. The authors claim this structure captures the major architectural concerns of systems that serve pre-trained or fine-tuned models, and that it applies to real systems by mapping three open-source projects in computer vision, text processing, and coding onto the layers. A sympathetic reader would care because, if the architecture holds, teams could use one shared vocabulary to design, evaluate, and evolve LLM-integrated systems instead of reinventing structures for each deployment.

What carries the argument

The carrying object is the four-layer functional reference architecture itself, shown as a diagram with components and connectors. A functional reference architecture here means a black-box view focused on entities and their interactions rather than implementation technologies. The layers provide separation of concerns, the Orchestrator controls task flow, and the two sidecars, Monitoring and Guardrail, are the mechanism for addressing cross-cutting quality attributes. The validation works by instantiating each of the three open-source systems into the same table of components, which is what turns the diagram into a general framework.

What would settle it

Find one production-grade LLM-integrated system whose components cannot be assigned to the four layers without either leaving major functionality outside the layers or splitting a single component across multiple layers; the paper's claim of a general reference architecture would then fail. A concrete candidate would be a fully agentic system where the model itself selects its own tools and memory, or a single-model monolithic chatbot with no vector store, no adapters, and no orchestrator.

Watch

Extended reading notes

Core claim

The central claim is that a functional reference architecture for LLM-integrated systems consists of four layers: Presentation handles user interaction and external connectivity; Application logic orchestrates workflows; LLM integration encapsulates pre-processing, inference with pre-trained models and task-specific adapters, and post-processing; Data management stores checkpoints, vector embeddings, interaction memory, and integrations. Monitoring and Guardrail sidecars span the layers to collect metrics and enforce security, privacy, and fairness. The paper maps MaxKB, Continue, and InternVL onto these components and argues that the mapping shows the architecture can represent systems across domains and can expose missing pieces such as task-specific adapters or guardrails.

Load-bearing premise

The framework's generality rests on the assumption that MaxKB, Continue, and InternVL are representative enough that mapping them onto the four layers demonstrates the architecture applies to LLM-integrated systems as a whole.

Editorial extensions

If this is right

  • Teams can use the four layers as a design checklist when building a new LLM-integrated system, choosing where pre-processing, orchestration, memory, and model access should live.
  • Existing systems can be audited by mapping them onto the architecture, turning absent components like Guardrail or task-specific adapters into visible gaps.
  • The architecture provides a shared vocabulary for reasoning about trade-offs such as latency, privacy, and fairness across different LLM-integrated systems.
  • Because MaxKB, Continue, and InternVL all fit the structure, the architecture is presented as domain-independent rather than tied to knowledge bases, coding assistants, or vision models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper treats alignment of the three chosen systems as validation; a stronger future test would be to apply the mapping to systems that were not designed with the RA in mind, including agentic multi-model systems and local-only deployments.
  • The Guardrail and Monitoring sidecars point toward a possible reusable service: the same compliance and observability components could be packaged once and shared across many LLM-integrated systems, which the paper sketches but does not develop.
  • A natural quantitative extension would be to measure whether teams that adopt the layers produce systems with lower integration cost or fewer missing safety components, compared with teams that do not.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a preliminary functional reference architecture (RA) for LLM-integrated systems. It derives eleven architectural concerns from a limited literature search (38 retrieved, 3 primary studies after snowballing) supplemented by the authors' experience, and then presents a four-layer architecture (Presentation, Application logic, LLM integration, Data management) with Monitoring and Guardrail sidecars. The paper evaluates the RA against the identified concerns in Section IV-A and claims to validate its applicability by mapping three open-source systems (MaxKB, Continue, InternVL) onto the architecture in Section IV-B. The manuscript explicitly frames the contribution as emerging and lists several limitations, including the need for further evaluation.

Significance. If the proposed RA were shown to provide a reliable common vocabulary and a sound structuring principle, it could support design, comparison, and evolution of LLM-integrated systems. The paper has genuine strengths: it is transparently positioned as a preliminary proposal, the concern derivation procedure is described (albeit on a small corpus), and the three selected systems cover different modalities and application types. However, the validation is descriptive rather than evaluative: the concern-satisfaction table reuses the same components that were designed for those concerns, and the system mapping lacks a decision rule, selection criteria, or independent assessment. The general applicability claim is therefore not currently supported, although the underlying architecture may still be a useful starting point.

major comments (3)
  1. [Section IV-A, Table II] The evaluation of how the RA addresses the identified concerns is circular. Each row of Table II asserts that a layer, component, or sidecar that was explicitly designed to address the concern in Section III satisfies that concern, but no independent evidence, quality metric, or baseline is provided. For example, the 'Performance and scalability' row cites distributed processing, asynchronous workflows, auto-scaling, and caching as mechanisms, which are design properties of the proposed RA rather than demonstrated outcomes. To make the central claim load-bearing, the paper needs either an external evaluation (e.g., measurable properties of existing systems built with the RA) or a clear downgrade of this section from 'evaluation' to 'tracing concerns to architectural mechanisms'.
  2. [Section IV-B, Table III] The applicability validation is not a test. Section IV-B and Table III map MaxKB, Continue, and InternVL to the RA, but no documented selection criteria are given for choosing these three systems, no decision rule states what counts as an Orchestrator, Middleware, Pre-processing component, or Post-processing component, and no independent annotators or baseline are used. The empty cells for Task-specific adapter and Guardrail in two of the three systems mean that nearly any system with a user interface, an API, and a model call can be described with the RA's vocabulary. As stated, Table III demonstrates only that three systems can be described in the RA's terms, not that the RA guides design, evaluation, and evolution. The paper should either provide a coding rubric, negative-case analysis, or inter-rater reliability, or explicitly reframe the section as an illustrative instantiation rather than validation.
  3. [Introduction, first paragraph] The claim that 'no documented RAs exist' for LLM-integrated systems is contradicted by the paper's own cited primary studies. Reference [19] is titled 'Towards responsible generative AI: A reference architecture for designing foundation model based agents', and reference [14] provides a taxonomy of foundation-model-based systems, both of which are relevant reference models. The novelty claim should be carefully qualified (e.g., 'no documented functional RA focused on LLM inference-serving systems, to our knowledge') so that the motivating premise is accurate.
minor comments (6)
  1. [Section IV] The text refers to 'Table IV-A' in two places, but the captions are 'TABLE II' and 'TABLE III'; the cross-references should be fixed.
  2. [Table II] In the 'Security, privacy, compliance and fairness' row, the text says authentication is managed at 'Integrator', but no component named 'Integrator' appears in Figure 1; the intended component should be named consistently.
  3. [References] Reference [15] is cited in the text as ISO/IEC/IEEE 42010:2022 but the reference entry gives 'ISO/IEC 20233:2019 - Information technology – Cloud computing – Interoperability and portability', which is a different standard; the mismatch should be corrected.
  4. [Section II] There is a typo in 'OW ASP highlighting risks like prompt injection'; this should read 'OWASP'.
  5. [Reference [8]] The supplementary spreadsheet link contains spaces and appears to be an unstable Google Docs URL; a stable DOI or permanent URL should be provided.
  6. [Figure 1] Figure 1 is visually dense, with many overlapping boxes and arrows; a larger diagram with clearer separation between the four layers and the two sidecars would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reference architecture is proposed and illustrated, not derived from its own validation.

full rationale

The paper's chain is: identify concerns from a literature search, propose a functional RA, then map the RA to the concerns (Table II) and to three open-source systems (Table III). No equation, fitted parameter, or uniqueness theorem is involved; the mapping is an illustrative application, not a falsifiable prediction. The RA's content is grounded in external sources (Behere and Törngren [6], Garcés et al. [12]) and the authors' design reasoning; self-citations ([8], [11], [20], [27]) support the search process and prior experience but are not load-bearing for the central claim. Section IV-A's statement that 'LLM integration is handled by the LLM integration layer' is a design traceability restatement, not a derivation of a result from its own input. Section V explicitly concedes that 'further evaluation across real-world systems spanning diverse domains is essential,' which is a limitation, not a forced validation. The reader's concern about the weakness of Section IV-B's validation is a threat to external validity, but it is not circularity in the sense of a prediction reducing to its inputs. Therefore no circular step can be exhibited, and the score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 1 invented entities

No numeric free parameters are involved. The conceptual content rests on three domain assumptions: the concern set is representative, mapping examples is a valid validation, and the reference architecture methodology is appropriate. The reference architecture itself is an invented conceptual entity with no independent falsifiable evidence.

assumptions (3)
  • domain assumption A common set of architectural concerns can be identified for LLM-integrated systems, and these concerns generalize beyond the three primary studies.
    Section II derives the concern set from three selected studies plus the authors' experience, and explicitly disclaims exhaustiveness, so the generality of the concerns is assumed.
  • domain assumption Mapping the components of an open-source system onto the reference architecture layers is sufficient evidence of the architecture's applicability.
    Section IV-B validates by placing MaxKB, Continue, and InternVL features into the layer table; no independent criteria are given for what would falsify the reference architecture.
  • domain assumption The functional reference architecture method from Garces et al. and Behere and Torngren is an appropriate foundation for this domain.
    Section III adopts the preliminary and functional definitions from [12] and [6] without questioning their suitability for LLM-integrated systems.
invented entities (1)
  • Four-layer functional reference architecture with Monitoring and Guardrail sidecars
    purpose: Provide a standardized functional decomposition to guide design, evaluation, and evolution of LLM-integrated systems.
    The architecture is the paper's proposal; it is supported by illustrative mappings and the authors' own experience, but no external adoption or falsifiable prediction exists.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Functional Software Reference Architecture for LLM-Integrated Systems." pith.science (2026). https://pith.science/paper/ICOQ3CMP

@misc{pith2026250112904,
  author       = {Pith},
  title        = {Pith review of: A Functional Software Reference Architecture for LLM-Integrated Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ICOQ3CMP}},
  note         = {Machine review of arXiv:2501.12904}
}
read the original abstract

The integration of large language models into software systems is transforming capabilities such as natural language understanding, decision-making, and autonomous task execution. However, the absence of a commonly accepted software reference architecture hinders systematic reasoning about their design and quality attributes. This gap makes it challenging to address critical concerns like privacy, security, modularity, and interoperability, which are increasingly important as these systems grow in complexity and societal impact. In this paper, we describe our \textit{emerging} results for a preliminary functional reference architecture as a conceptual framework to address these challenges and guide the design, evaluation, and evolution of large language model-integrated systems. We identify key architectural concerns for these systems, informed by current research and practice. We then evaluate how the architecture addresses these concerns and validate its applicability using three open-source large language model-integrated systems in computer vision, text processing, and coding.

Figures

Figures reproduced from arXiv: 2501.12904 by the authors.

Figure 1
Figure 1. Preliminary functional software RA for LLMs-integrated Systems [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Specification and Detection of LLM Code Smells

    cs.SE 2025-12 conditional novelty 6.0 of 10

    A catalog of five LLM code smells and a static detection tool find that 60.5% of 200 open-source LLM-using Python projects exhibit at least one smell.

Reference graph

Works this paper leans on

27 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [19]

    Towards responsible generative ai: A reference architecture for designing foun- dation model based agents,

    Q. Lu, L. Zhu, X. Xu, Z. Xing, S. Harrer, and J. Whittle, “Towards responsible generative ai: A reference architecture for designing foun- dation model based agents,” in 2024 IEEE 21st International Conference on Software Architecture Companion (ICSA-C) . IEEE, 2024, pp. 119– 126

    work page 2024

  2. [14]

    A taxonomy for autonomous llm-powered multi-agent architectures

    T. H ¨andler, “A taxonomy for autonomous llm-powered multi-agent architectures.” in KMIS, 2023, pp. 85–98

    work page 2023

  3. [1]

    Continue, open-source ai code assistant

    “Continue, open-source ai code assistant.” https://github.com/ continuedev/continue, accessed: 2024-12-09

    work page 2024

  4. [2]

    Maxkb: An open-source knowledge base q&a system,

    1Panel-dev, “Maxkb: An open-source knowledge base q&a system,” https://github.com/1Panel-dev/MaxKB, 2024, accessed: 2024-12-12

    work page 2024

  5. [3]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    arXiv 2023

  6. [4]

    Evaluating strategies for study selection in systematic literature studies,

    N. B. Ali and K. Petersen, “Evaluating strategies for study selection in systematic literature studies,” in Procs of ESEM , 2014

    work page 2014

  7. [5]

    Bass, Software architecture in practice

    L. Bass, Software architecture in practice . Pearson Education India, 2012

    work page 2012

  8. [6]

    A functional reference architecture for autonomous driving,

    S. Behere and M. T ¨orngren, “A functional reference architecture for autonomous driving,” Information and Software Technology , vol. 73, pp. 136–150, 2016

    work page 2016

Show all 27 references
  1. [7]

    On the opportunities and risks of foundation models,

    R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill et al. , “On the opportunities and risks of foundation models,” arXiv preprint arXiv:2108.07258, 2021

    2021 arXiv

  2. [8]

    Automatic search, and selection,

    Bucaioni et al., “Automatic search, and selection,” https://docs.google. com/spreadsheets/d/1Tm3m4iMFQsZMHGfhCH6NjhbIq UKyk8T/ edit?usp=share link, accessed: 2024-12-02

    2024

  3. [9]

    LangChain,

    H. Chase, “LangChain,” Oct. 2022. [Online]. Available: https: //github.com/langchain-ai/langchain

    2022

  4. [10]

    The artificial intelligence act,

    European Union, “The artificial intelligence act,” https: //artificialintelligenceact.eu, accessed: 2024-12-02

    2024

  5. [11]

    Standardisation in digital twin architectures in manufacturing,

    E. Ferko, A. Bucaioni, P. Pelliccione, and M. Behnam, “Standardisation in digital twin architectures in manufacturing,” in 2023 IEEE 20th International Conference on Software Architecture (ICSA). IEEE, 2023, pp. 70–81

    2023

  6. [12]

    Three decades of software reference architectures: A systematic mapping study,

    L. Garc ´es, S. Mart ´ınez-Fern´andez, L. Oliveira, P. Valle, C. Ayala, X. Franch, and E. Y . Nakagawa, “Three decades of software reference architectures: A systematic mapping study,” Journal of Systems and Software, vol. 179, p. 111004, 2021

    2021

  7. [13]

    Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources,

    T. Greenhalgh and R. Peacock, “Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources,” BMJ, vol. 331, no. 7524, pp. 1064–1065, 2005

    2005

  8. [15]

    2019, accessed: 2024-12-08

    ISO/IEC 20233:2019 - Information technology – Cloud computing – Interoperability and portability , International Organization for Standardization Std., Oct. 2019, accessed: 2024-12-08. [Online]. Available: https://www.iso.org/standard/74393.html

    2019

  9. [16]

    A systematic review of systematic review process research in software engineering,

    B. Kitchenham and P. Brereton, “A systematic review of systematic review process research in software engineering,” Information and software technology, 2013

    2013

  10. [17]

    Klarna AI Assistant Handles Two-Thirds of Customer Service Chats in Its First Month,

    Klarna, “Klarna AI Assistant Handles Two-Thirds of Customer Service Chats in Its First Month,” https://www.klarna.com/international/press/ klarna-ai-assistant-handles-two-thirds-of-customer-service-chats-in-its-first-month/, accessed: 2024-12-02

    2024

  11. [18]

    A taxonomy of foundation model based systems through the lens of software archi- tecture,

    Q. Lu, L. Zhu, X. Xu, Y . Liu, Z. Xing, and J. Whittle, “A taxonomy of foundation model based systems through the lens of software archi- tecture,” in Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering-Software Engineering for AI , 2024, pp. 1–6

    2024

  12. [20]

    Architecting ml-enabled systems: Challenges, best practices, and design decisions,

    R. Nazir, A. Bucaioni, and P. Pelliccione, “Architecting ml-enabled systems: Challenges, best practices, and design decisions,” Journal of Systems and Software , vol. 207, p. 111860, 2024

    2024

  13. [21]

    Internvl,

    OpenGVLab, “Internvl,” 2024, accessed: 2024-12-12. [Online]. Available: https://github.com/OpenGVLab/InternVL

    2024

  14. [22]

    Automotive architecture framework: The experience of volvo cars,

    P. Pelliccione, E. Knauss, R. Heldal, S. M. ˚Agren, P. Mallozzi, A. Alminger, and D. Borgentun, “Automotive architecture framework: The experience of volvo cars,” Journal of systems architecture , vol. 77, pp. 83–100, 2017

    2017

  15. [23]

    Democratizing llms: An exploration of cost-performance trade-offs in self-refined open- source models,

    S. Shashidhar, A. Chinta, V . Sahai, Z. Wang, and H. Ji, “Democratizing llms: An exploration of cost-performance trade-offs in self-refined open- source models,” arXiv preprint arXiv:2310.07611 , 2023

    2023 arXiv

  16. [24]

    Guidelines for snowballing in systematic literature studies and a replication in software engineering,

    C. Wohlin, “Guidelines for snowballing in systematic literature studies and a replication in software engineering,” in Procs of EASE . ACM, 2014, pp. 38:1–38:10

    2014

  17. [25]

    Transformers: State- of-the-art natural language processing,

    T. Wolf, L. Debut, V . Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz et al., “Transformers: State- of-the-art natural language processing,” in Proceedings of the 2020 conference on empirical methods in natural language processing: system de...

    2020

  18. [26]

    Configurable foundation models: Building llms from a modular perspective,

    C. Xiao, Z. Zhang, C. Song, D. Jiang, F. Yao, X. Han, X. Wang, S. Wang, Y . Huang, G. Linet al., “Configurable foundation models: Building llms from a modular perspective,” arXiv preprint arXiv:2409.02877 , 2024

    2024 arXiv

  19. [27]

    Robustness, security, privacy, explainability, efficiency, and usability of large language models for code,

    Z. Yang, Z. Sun, T. Z. Yue, P. Devanbu, and D. Lo, “Robustness, security, privacy, explainability, efficiency, and usability of large language models for code,” arXiv preprint arXiv:2403.07506 , 2024

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.