Pith. sign in

REVIEW 4 major objections 5 minor 91 references

A Roadmap for Tamed Interactions with Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that prompt programming should be a first-class software engineering concern and proposes LSL, a domain-specific language that scripts and constrains LLM interactions.

desk verdict A coherent, well-situated SE4AI roadmap for a DSL (LSL) that constrains and scripts LLM interactions, but with no implementation or evaluation — and an abstract that overclaims a demonstrated benefit. read the letter →

arxiv 2510.24819 v2 pith:UGLPYRMY submitted 2025-10-28 cs.SE cs.PL

Vincenzo Scotti , Jan Keim , Tobias Hey , Andreas Metzger , Anne Koziolek , Raffaela Mirandola This is my paper · ORCID
classification cs.SEcs.PL
keywords LLMscriptinglanguagedomain-specificpromptprogrammingconstraineddecodinggenerativegrammarsAIsoftwareengineeringreliabilityinteractiontransparency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that LLM-powered software is unreliable in part because prompt logic is buried in general-purpose code, where it stays implicit and hard to analyze. The authors propose LSL, a domain-specific scripting language for LLM interactions that makes prompts, context scopes, output constraints, and control flow explicit program artifacts. The central move is to separate deterministic orchestration from probabilistic generation and to constrain the model's output with generative grammars, so generated text is at least syntactically compliant. A sympathetic reader would care because this offers a path to bring LLM applications under standard software engineering discipline—verification, testing, debugging, reuse—while stopping short of claiming the model's content is factually correct.

What carries the argument

The central object is LSL, a DSL whose interpreter manages LLM calls through interaction blocks, context scopes, output constraints, and control flow. The load-bearing mechanism is constrained decoding with generative grammars: while the model generates, the interpreter restricts the next-token distribution so that only strings admitted by the grammar can appear, which is what turns an unreliable text generator into a component whose output is guaranteed well-formed. This mechanism is what makes the interaction explicit enough for verification tools, since termination, dead branches, and output-schema compliance become properties of a script rather than properties of a black-box model.

What would settle it

If a model-serving backend exposes no token-level likelihoods, and the interpreter then cannot enforce a specified JSON schema except by post-hoc retry, the paper's promised syntactic guarantee fails for that backend.

Watch

Extended reading notes

Core claim

The paper's central claim is that prompt programming should be treated as a first-class software engineering concern, and that a DSL called LSL can deliver this by making interaction logic explicit, analyzable, and amenable to verification and validation. LSL structures LLM calls as interaction blocks with context scopes and output constraints, separating deterministic control flow from probabilistic model behavior. The authors illustrate the approach on structured generation scenarios and report improved failure localization and interaction transparency. The guarantee LSL offers is syntactic conformance of generated output to a grammar or schema, not semantic correctness or factual accuracy.

Load-bearing premise

The whole approach depends on LLM-serving backends exposing token-level likelihoods during generation, since grammar-constrained decoding cannot be implemented without access to those scores.

Editorial extensions

If this is right

  • Failure localization improves because an error can be traced to the specific scripted step or output constraint that produced it.
  • Interaction transparency increases because context management, sub-task boundaries, and data flows become visible parts of the program instead of hidden prompt assembly.
  • Scripts become checkable: model checking can verify termination, absence of unreachable branches, and conformance of generated JSON to API constraints.
  • Robustness techniques such as few-shot learning, retrieval-augmented generation, and chain-of-thought can be automated and tuned by the interpreter rather than hand-crafted for each prompt.
  • Prompts and interaction patterns become reusable, versionable artifacts that evolve under the same practices as code.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If LSL matured, the reusable unit of LLM application development would likely shift from raw prompts to scripted interaction patterns, making prompt and data versioning a natural part of continuous integration pipelines.
  • A testable extension would be a benchmark that runs the same scripted task with and without grammar-constrained decoding, isolating whether LSL's practical benefit comes from the syntactic guarantee or simply from the discipline of explicit scripting.
  • The cognitive-scaffolding role suggests LSL could change who owns prompt behavior: domain experts without machine-learning training could build and maintain reliable automation, making prompt design a product artifact rather than a model detail.
  • Because the approach hinges on token-level scoring, its viability tracks whether model providers standardize a scoring interface; a thin adapter that enforces grammar constraints on any backend exposing logprobs would be a concrete way to test that dependency.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that prompt programming should be treated as a first-class software-engineering concern and proposes LSL, a domain-specific language for scripting LLM interactions. LSL is intended to make interaction logic explicit by providing abstractions for interaction blocks, context scopes, output constraints, and control flow, separating deterministic orchestration from probabilistic model behavior. The paper motivates the design from LLM unreliability, surveys related tools, illustrates the idea with a knowledge-grounded chat and a database-creation workflow (Figures 3 and 4), discusses opportunities in reliability, robustness, and trustworthiness, and lays out a roadmap of foundational work, benchmarks, interpreters, standardization, and ecosystem building. The manuscript is explicitly a roadmap and contains no implementation or empirical evaluation.

Significance. If realized, LSL would offer a principled foundation for analyzable, maintainable, and partially verifiable LLM-based systems, with the specific contribution of syntactic guarantees on generated outputs through grammar-constrained decoding. The paper's strengths are its clear problem framing, the explicit separation of deterministic and probabilistic concerns, the broad and relevant positioning against existing frameworks, and the honest acknowledgment of the limits of the approach (e.g., no semantic correctness guarantees, Sections 6.3 and 6.4). The illustrative examples are useful for communicating the design. The main limitation is that the claimed benefits are not demonstrated: there is no interpreter, no formal semantics, and no empirical evidence, and the central syntactic-compliance guarantee is conditional on an interface for token-level scoring that is not currently standardized. The contribution is therefore best read as a research agenda rather than an established result.

major comments (4)
  1. [Section 6.2 (and Section 4.2)] The paper's headline guarantee of syntactic compliance is load-bearing and is conditional on an unstandardized interface. Section 4.2 states that generative grammars can constrain decoding, and Section 5.1 claims improved reliability because outputs are syntactically correct; however, Section 6.2 concedes that "not all APIs or inference servers support the necessary functionalities" and that access to a scoring method is crucial for enforcing decoding strategies independently of the LLM implementation. Because closed commercial APIs generally do not expose per-token logits for arbitrary developer-supplied grammars, the central benefit reduces to provider-specific behavior or post-hoc validation in exactly the settings the paper names in Section 2.1. Please scope the guarantee, e.g., by stating that syntactic compliance holds only for backends exposing token-level scoring, and specify what LSL guarantees or does not guarantee on other backends. Listing standardization as future work in Section 7 does not resolve this, because the abstract states the guarantee unconditionally.
  2. [Abstract and Section 4.1] The abstract's phrase "showing improved failure localization and interaction transparency" overstates what the paper provides. Figures 3 and 4 are illustrative pseudocode, and Section 7 lists benchmarks and evaluation as future work; no implementation or measurement supports the word "showing." Please rephrase to "illustrating" or "arguing for," and place the evaluation of these properties explicitly in the future-work agenda.
  3. [Section 5.3.1] The claim that one can "verify the generated JSON to call a remote API always matches the API constraints by looking at the grammar constraining the LLM output" conflates grammar-level syntactic conformance with API-contract conformance. API constraints typically include semantic conditions (required fields, types, value ranges, cross-field relations) that go beyond a context-free grammar of JSON. Moreover, "always" is too strong because conformance depends on the decoder actually enforcing the grammar, which is exactly the capability in question in Section 6.2. Please restate this as a conditional syntactic guarantee and distinguish it from semantic contract verification.
  4. [Section 4.1 and Figures 3–4] The manuscript never specifies the syntax or semantics of LSL; Figures 3 and 4 use ad-hoc pseudocode with no grammar, scoping, or evaluation rules. Since the key claimed benefit is that interaction logic becomes "explicit, analyzable, and amenable to verification and validation," the absence of a formal or at least precise definition of LSL makes those claims difficult to assess. Please either provide a minimal formal core (e.g., the grammar of interaction blocks and output constraints) or explicitly relegate formal semantics to foundational work in Section 7.
minor comments (5)
  1. [Section 3.2] There is an incomplete sentence: "Llama CPP allows specifying a generative grammar to constrain the generated The closest tool..." Please complete the sentence.
  2. [Figure 3] The pseudocode contains the typo "refrences," which should be "references."
  3. [Section 4.3] The text begins "Figure 4 shows how both a semi-structured task... and a structured task..."; the semi-structured knowledge-grounded chat is shown in Figure 3, so the citation should be "Figures 3 and 4."
  4. [Figure 4] In the `extract_relations` function, the variable `relations` is initialized but the function returns `relationships`; please align the variable names.
  5. [Section 4.4] The phrases "continuous integration" and "continuous delivery" are missing spaces in the manuscript text and should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: LSL's proposal is a design vision whose claims rest on known constrained-decoding mechanisms, not on fitted inputs or self-referential definitions.

full rationale

This is a vision/roadmap paper, not an empirical derivation, so the standard circularity patterns do not apply. The central claim is that prompt programming should be a first-class Software Engineering concern and that the proposed DSL, LSL, makes interaction logic explicit, analyzable, and amenable to verification and validation. No equation or predictive quantity is derived from fitted data, and no output is claimed to follow from an input by construction. The syntactic-compliance guarantee is explicitly grounded in generative grammars and constrained decoding, which the paper credits to existing frameworks such as llama.cpp and Guidance rather than to the authors' prior work. The paper's self-citations ([24], [28], [39], [77], [78]) appear in background, related work, and motivating examples; none carries the load of the central proposal. The acknowledged limitation in Section 6.2, that enforcing decoding strategies requires access to token likelihoods that not all APIs expose, is a feasibility condition on an envisioned interpreter, not a circular reduction: the proposed benefit is conditional on an interface, but the claim does not define LSL's value in terms of itself. The abstract's statement about 'improved failure localization and interaction transparency' is plausibly unsupported by an empirical evaluation, but that is a correctness or evidence concern, not a circularity concern. The paper's claims are honest about scope, explicitly disclaiming semantic correctness and factual accuracy, and the derivation chain, such as it is, is self-contained.

Assumptions & free parameters 0 free parameters · 4 assumptions · 1 invented entities

The central claims rest on design assumptions rather than measurements. There are no fitted parameters. LSL is a proposed artifact without an implementation or independent evidence, so the ledger is dominated by domain assumptions and one invented entity with no external handle.

assumptions (4)
  • domain assumption Generative grammars can be applied freely to LLM decoding to constrain generated text to well-formed strings (Section 4.2).
    The paper's reliability claims rest on grammar-constrained decoding. It cites llama.cpp and Guidance as supporting evidence, but no independent benchmark or measurement is provided in this manuscript.
  • domain assumption A standardized interface to LLMs and external resources can be defined with sufficient access to model scores (Section 6.2).
    The interpreter needs token likelihoods for constrained decoding. The paper acknowledges this as an open standardization challenge, so the feasibility of such an interface is assumed rather than demonstrated.
  • ad hoc to paper Separating deterministic logic from probabilistic model behavior makes LLM interactions analyzable and verifiable without losing expressivity (Sections 4 and 5.3).
    This is the central design premise of LSL. It is argued from examples, but not demonstrated with a prototype or a controlled study.
  • domain assumption LLMs can generate any Unicode output sequence, so LSL scripts can be model-agnostic (Section 4.1).
    Underlies the promise of seamless interaction with any underlying LLM. The assumption ignores provider-specific constraints beyond Unicode character sets.
invented entities (1)
  • LSL (LLM Scripting Language)
    purpose: A proposed DSL to script, constrain, and verify LLM interactions
    Only illustrative syntax is given in Figures 3 and 4. No grammar, interpreter, implementation, or falsifiable prediction is provided, so there is no independent evidence for the entity outside the paper's own examples.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Roadmap for Tamed Interactions with Large Language Models." pith.science (2026). https://pith.science/paper/UGLPYRMY

@misc{pith2026251024819,
  author       = {Pith},
  title        = {Pith review of: A Roadmap for Tamed Interactions with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UGLPYRMY}},
  note         = {Machine review of arXiv:2510.24819}
}
read the original abstract

Large Language Models (LLMs) are increasingly embedded in software systems ( GenAIware), enabling new forms of automation and interaction However, their probabilistic nature and reliance on prompt programming challenge reliability, robustness, and maintainability In current practice, prompt-related concerns (e.g., context management, interaction logic, output validation) are embedded in general-purpose code, leading to implicit, hard-to-analyze systems We argue that prompt programming should be treated as a first-class Software Engineering (SE ) concern and propose LLM Scripting Language (LSL ), a Domain Specific Language ( DSL) for structuring LLM interactions as analyzable programs LSL introduces abstractions for interaction blocks, context scopes, output constraints, and control flow, separating deterministic logic from probabilistic model behavior while ensuring syntactic compliance From an SE perspective, LSL supports disciplined development by making interaction logic explicit, analyzable, and amenable to verification and validation It also acts as cognitive scaffolding, externalizing prompt design into programmable artifacts that reduce implicit reasoning and support systematic debugging, evolution, and reuse We illustrate these properties in a structured generation scenario, showing improved failure localization and interaction transparency While LSL does not guarantee semantic correctness or factual accuracy, it provides a principled foundation for more analyzable and maintainable prompt-based systems.

Figures

Figures reproduced from arXiv: 2510.24819 by the authors.

Figure 1
Figure 1. LLM prompting approaches. Auto-regressive (or causal) LLMs, which represent the vast majority of current models, are designed to generate text autoregressively. They yield the probability distribution from which to sample the next token one at a time, conditioning the distribution of this next token on all the preceding ones (the context). From a practical perspective, LLMs are used by prompting with some prefix tex… view at source ↗
Figure 2
Figure 2. Idea behind LSL: the domain expert uses the DSL to write scripts for different tasks, end users accessing their AI-powered applications or servers running their AI-powered applications interact with the LLM selecting the script corresponding to the required functionality. We propose LSL: a DSL to format and constrain the input and output of an LLM. The idea is to develop a scripting language, its interpreter and the… view at source ↗
Figure 3
Figure 3. Example semi-structured use case (system messages, instructions, data, and subroutine outputs): knowledge-grounded chat. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Example structured use case (system messages, instructions, data and subroutine outputs): database creation [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Roadmap to the development of LSL [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

91 extracted references · 32 canonical work pages

  1. [1]

    OpenAI API Reference: Documentation for Accessing and Interacting with OpenAI Models

    2020. OpenAI API Reference: Documentation for Accessing and Interacting with OpenAI Models. https://platform.openai.com/docs/api-reference/

    2020

  2. [2]

    LangChain Documentation: Introduction to Building Applications with Language Models

    2022. LangChain Documentation: Introduction to Building Applications with Language Models. https://python.langchain.com/docs/introduction/

    2022

  3. [3]

    CrewAI: Framework for Building Autonomous Multi-Agent LLM Workflows

    2023. CrewAI: Framework for Building Autonomous Multi-Agent LLM Workflows. https://github.com/crewAIInc/crewAI

    2023

  4. [4]

    FlowiseAI: Low-Code Platform for Building LLM-Powered Applications Visually

    2023. FlowiseAI: Low-Code Platform for Building LLM-Powered Applications Visually. https://github.com/FlowiseAI/Flowise

    2023

  5. [5]

    Google AI Studio: Access Gemini Models and API Tools for AI Development

    2023. Google AI Studio: Access Gemini Models and API Tools for AI Development. https://ai.google.dev/

    2023

  6. [6]

    Guardrails AI: Framework for Validating and Controlling LLM Outputs

    2023. Guardrails AI: Framework for Validating and Controlling LLM Outputs. https://github.com/guardrails-ai/guardrails

    2023

  7. [7]

    Guidance: Programmable Control for Language Models Using Constraints and Templates

    2023. Guidance: Programmable Control for Language Models Using Constraints and Templates. https://github.com/guidance-ai/guidance

    2023

  8. [8]

    Haystack by deepset: Framework for Developing End-to-End NLP Applications with LLMs and RAG

    2023. Haystack by deepset: Framework for Developing End-to-End NLP Applications with LLMs and RAG. https://github.com/deepset-ai/haystack

    2023

Show all 91 references
  1. [9]

    IBM watsonx.ai: Enterprise Studio for Training, Validating, and Deploying AI Models

    2023. IBM watsonx.ai: Enterprise Studio for Training, Validating, and Deploying AI Models. https://www.ibm.com/products/watsonx-ai

    2023

  2. [10]

    llama-cpp-python: Python Bindings for llama.cpp for Easy Integration

    2023. llama-cpp-python: Python Bindings for llama.cpp for Easy Integration. https://github.com/abetlen/llama-cpp-python

    2023

  3. [11]

    Llama.cpp: Efficient LLM Inference in C/C++ for LLaMA Models on CPUs

    2023. Llama.cpp: Efficient LLM Inference in C/C++ for LLaMA Models on CPUs. https://github.com/ggerganov/llama.cpp

    2023

  4. [12]

    Llama.cpp Grammars: Enforcing Syntax Constraints on Language Model Outputs

    2023. Llama.cpp Grammars: Enforcing Syntax Constraints on Language Model Outputs. https://github.com/ggerganov/llama.cpp/blob/master/ grammars

    2023

  5. [13]

    LlamaIndex: Data Framework for Connecting LLMs with External Data Sources

    2023. LlamaIndex: Data Framework for Connecting LLMs with External Data Sources. https://github.com/run-llama/llama_index

    2023

  6. [14]

    Semantic Kernel: Microsoft’s Framework for Integrating LLMs with Traditional Programming Models

    2023. Semantic Kernel: Microsoft’s Framework for Integrating LLMs with Traditional Programming Models. https://github.com/microsoft/semantic- kernel

    2023

  7. [15]

    Google Gemini API Documentation: Generating Structured Output with Language Models

    2024. Google Gemini API Documentation: Generating Structured Output with Language Models. https://ai.google.dev/gemini-api/docs/structured- output

    2024

  8. [16]

    OpenAI: Introducing Structured Outputs in the API for Reliable Response Formatting

    2024. OpenAI: Introducing Structured Outputs in the API for Reliable Response Formatting. https://openai.com/index/introducing-structured- outputs-in-the-api

    2024

  9. [17]

    NIST AI. 2023. Artificial Intelligence Risk Management Framework (AI RMF 1.0). https://doi.org/10.6028/NIST.AI.100-1

    2023 doi

  10. [18]

    Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, et al. 2023. Gemini: A Family of Highly Capable Multimodal Models. CoRRabs/2312.11805 (2023). https://doi.org/10.48550/ARXIV.2312.11805 arXiv:2312.11805

    2023 doi arXiv

  11. [19]

    Scott Barnett, Stefanus Kurniawan, Srikanth Thudumu, Zach Brannelly, and Mohamed Abdelrazek. 2024. Seven Failure Points When Engineering a Retrieval Augmented Generation System. InProceedings of the IEEE/ACM 3rd International Conference on AI Engineering - Software Engineering...

    2024

  12. [20]

    Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, et al. 2023. Eliciting Latent Predictions from Transformers with the Tuned Lens.CoRRabs/2303.08112 (2023). https://doi.org/10.48550/ARXIV.2303.08112 arXiv:2303.08112

    2023 doi arXiv

  13. [21]

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, et al . 2024. DeepSeek LLM: Scaling Open-Source Language Models with Longtermism.CoRRabs/2401.02954 (2024). https://doi.org/10.48550/ARXIV.2401.02954 arXiv:2401.02954

    2024 doi arXiv

  14. [22]

    Antonio Brogi, Jose Carrasco, Javier Cubo, Francesco D’Andria, Elisabetta Di Nitto, et al . 2016. SeaClouds: An Open Reference Architecture for Multi-cloud Governance. InSoftware Architecture - 10th European Conference, ECSA 2016, Copenhagen, Denmark, November 28 - December 2,...

    2016 doi

  15. [23]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, et al. 2020. Language Models are Few-Shot Learners. InAdvances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 202...

    2020

  16. [24]

    Nicolò Brunello, Davide Rigamonti, Andrea Sassella, Vincenzo Scotti, and Mark James Carman. 2025. InTraVisTo: Inside Transformer Visualisation Tool.CoRRabs/2507.13858 (2025). https://doi.org/10.48550/ARXIV.2507.13858 arXiv:2507.13858

    2025 doi arXiv

  17. [25]

    Zhenpeng Chen, Chong Wang, Weisong Sun, Guang Yang, Xuanzhe Liu, et al. 2025. Promptware Engineering: Software Engineering for LLM Prompt Development.CoRRabs/2503.02400 (2025). https://doi.org/10.48550/ARXIV.2503.02400 arXiv:2503.02400

    2025 doi

  18. [26]

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, et al. 2022. Scaling Instruction-Finetuned Language Models.CoRRabs/2210.11416 (2022). https://doi.org/10.48550/ARXIV.2210.11416 arXiv:2210.11416

    2022 doi arXiv

  19. [27]

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, et al. 2024. Scaling Instruction-Finetuned Language Models.J. Mach. Learn. Res. 25 (2024), 70:1–70:53. https://jmlr.org/papers/v25/23-0870.html

    2024

  20. [28]

    Simone Corbo, Luca Bancale, Valeria De Gennaro, Livia Lestingi, Vincenzo Scotti, et al . 2025. How Toxic Can You Get? Search-based Toxicity Testing for Large Language Models.IEEE Trans. Software Eng.(2025). https://doi.org/10.1109/TSE.2025.3607625

    2025

  21. [29]

    Tri Dao. 2024. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https://openreview.net/forum?id=mZn2Xyh9Ec

    2024

  22. [30]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. InAdvances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, Neu...

    2022

  23. [31]

    DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, et al. 2024. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model.CoRRabs/2405.04434 (2024). https://doi.org/10.48550/ARXIV.2405.04434 arXiv:2405.04434

    2024 doi arXiv

  24. [32]

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, et al . 2024. DeepSeek-V3 Technical Report.CoRRabs/2412.19437 (2024). https: //doi.org/10.48550/ARXIV.2412.19437 arXiv:2412.19437

    2024 doi arXiv

  25. [33]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human ...

    2019

  26. [34]

    Felix Dobslaw, Robert Feldt, Juyeon Yoon, and Shin Yoo. 2025. Challenges in Testing Large Language Model Based Software: A Faceted Taxonomy. CoRRabs/2503.00481 (2025). https://doi.org/10.48550/ARXIV.2503.00481 arXiv:2503.00481

    2025 doi

  27. [35]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, et al . 2024. The Llama 3 Herd of Models.CoRR abs/2407.21783 (2024). https://doi.org/10.48550/ARXIV.2407.21783 arXiv:2407.21783

    2024 doi arXiv

  28. [36]

    Beyza Eken, Samodha Pallewatta, Nguyen Tran, Ayse Tosun, and Muhammad Ali Babar. 2025. A Multivocal Review of MLOps Practices, Challenges and Open Issues.ACM Comput. Surv.58, 2, Article 39 (Sept. 2025), 35 pages. https://doi.org/10.1145/3747346

    2025 doi

  29. [37]

    Timothy Elvira, Tyler Thomas Procko, and Omar Ochoa. 2024. Requirements Elicitation for Machine Learning Applications: A Research Preview. In 2024 Conference on AI, Science, Engineering, and Technology (AIxSET). 218–221. https://doi.org/10.1109/AIxSET62544.2024.00042

    2024

  30. [38]

    2023.Information Technology - Artificial Intelligence - AI System Life Cycle Processes

    International Organization for Standardization and International Electrotechnical Commission. 2023.Information Technology - Artificial Intelligence - AI System Life Cycle Processes. Standard. International Organization for Standardization

    2023

  31. [39]

    Raffaello Fornasiere, Nicolò Brunello, Vincenzo Scotti, and Mark Carman. 2024. Medical Information Extraction with Large Language Models. In Proceedings of the 7th International Conference on Natural Language and Speech Processing (ICNLSP 2024), Mourad Abbas and Abed Alhakim F...

    2024

  32. [40]

    Gallegos, Ryan A

    Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, et al . 2024. Bias and Fairness in Large Language Models: A Survey.Computational Linguistics50, 3 (09 2024), 1097–1179. https://doi.org/10.1162/coli_a_00524 arXiv:https://direct.mit.edu/coli/article...

    2024 doi

  33. [41]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, et al. 2023. Retrieval-Augmented Generation for Large Language Models: A Survey. CoRRabs/2312.10997 (2023). https://doi.org/10.48550/ARXIV.2312.10997 arXiv:2312.10997

    2023 doi arXiv

  34. [42]

    GitHub. 2021. Introducing GitHub Copilot: your AI pair programmer. https://github.blog/news-insights/product-news/introducing-github-copilot- ai-pair-programmer/ Manuscript submitted to ACM A Roadmap for Tamed Interactions with Large Language Models 21

    2021

  35. [43]

    Goodfellow, Yoshua Bengio, and Aaron C

    Ian J. Goodfellow, Yoshua Bengio, and Aaron C. Courville. 2016.Deep Learning. MIT Press. http://www.deeplearningbook.org/

    2016

  36. [44]

    Shane Greenstein. 2023. The AI Gold Rush.IEEE Micro43, 6 (2023), 126–128. https://doi.org/10.1109/MM.2023.3322049

    2023

  37. [45]

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, et al. 2022. Training Compute-Optimal Large Language Models.CoRRabs/2203.15556 (2022). https://doi.org/10.48550/ARXIV.2203.15556 arXiv:2203.15556

    2022 doi arXiv

  38. [46]

    Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, et al. 2024. Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon...

    2024 doi

  39. [47]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, et al. 2023. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions.CoRRabs/2311.05232 (2023). https://doi.org/10.48550/ARXIV.2311.05232 arXiv:2311.05232

    2023 doi arXiv

  40. [48]

    Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, et al. 2024. Position: TrustLLM: Trustworthiness in Large Language Models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net. https://openreview....

    2024

  41. [49]

    Goucher, Adam Perelman, Aditya Ramesh, et al

    Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, et al . 2024. GPT-4o System Card.CoRRabs/2410.21276 (2024). https://doi.org/10.48550/ARXIV.2410.21276 arXiv:2410.21276

    2024 doi arXiv

  42. [50]

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, et al . 2023. Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations.CoRRabs/2312.06674 (2023). https://doi.org/10.48550/ARXIV.2312.06674 arXiv:2312.06674

    2023 doi arXiv

  43. [51]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, et al. 2023. Survey of Hallucination in Natural Language Generation.ACM Comput. Surv. 55, 12 (2023), 248:1–248:38. https://doi.org/10.1145/3571730

    2023 doi

  44. [52]

    Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, et al

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, et al. 2023. Mistral 7B.CoRRabs/2310.06825 (2023). https://doi.org/10.48550/ARXIV.2310.06825 arXiv:2310.06825

    2023 doi arXiv

  45. [53]

    Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, et al

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, et al. 2024. Mixtral of Experts.CoRRabs/2401.04088 (2024). https://doi.org/10.48550/ARXIV.2401.04088 arXiv:2401.04088

    2024 doi arXiv

  46. [54]

    Dan Jurafsky and James H. Martin. 2009.Speech and language processing: an introduction to natural language processing, computational linguistics, and speech recognition, 2nd Edition. Prentice Hall, Pearson Education International. https://www.worldcat.org/oclc/315913020

    2009

  47. [55]

    Brown, Benjamin Chess, et al

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, et al. 2020. Scaling Laws for Neural Language Models.CoRR abs/2001.08361 (2020). arXiv:2001.08361 https://arxiv.org/abs/2001.08361

    2020 arXiv

  48. [56]

    Donald E. Knuth. 1984. Literate Programming.Comput. J.27, 2 (1984), 97–111. https://doi.org/10.1093/COMJNL/27.2.97

    1984 doi

  49. [57]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large Language Models are Zero-Shot Reasoners. InAdvances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Or...

    2022

  50. [58]

    Jonathan Laurent and André Platzer. 2025. Oracular Programming: A Modular Foundation for Building LLM-Enabled Software.CoRRabs/2502.05310 (2025). https://doi.org/10.48550/ARXIV.2502.05310 arXiv:2502.05310

    2025 doi

  51. [59]

    Dingcheng Li, Zheng Chen, Eunah Cho, Jie Hao, Xiaohu Liu, et al . 2022. Overcoming Catastrophic Forgetting During Domain Adaptation of Seq2seq Language Generation. InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisti...

    2022

  52. [60]

    Liang, Melissa Lin, Nikitha Rao, and Brad A

    Jenny T. Liang, Melissa Lin, Nikitha Rao, and Brad A. Myers. 2025. Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts.Proc. ACM Softw. Eng.2, FSE (2025), 1591–1614. https://doi.org/10.1145/3729342

    2025 doi

  53. [61]

    LMSYS. 2023. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. https://lmsys.org/blog/2023-03-30-vicuna/

    2023

  54. [62]

    Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, et al. 2024. Gemma: Open Models Based on Gemini Research and Technology.CoRRabs/2403.08295 (2024). https://doi.org/10.48550/ARXIV.2403.08295 arXiv:2403.08295

    2024 doi arXiv

  55. [63]

    Shervin Minaee, Tomás Mikolov, Narjes Nikzad, Meysam Chenaghlu, Richard Socher, et al . 2024. Large Language Models: A Survey.CoRR abs/2402.06196 (2024). https://doi.org/10.48550/ARXIV.2402.06196 arXiv:2402.06196

    2024 doi arXiv

  56. [64]

    Butler, Chris Parnin, Thomas Zimmermann, et al

    Nadia Nahar, Christian Kästner, Jenna L. Butler, Chris Parnin, Thomas Zimmermann, et al. 2025. Beyond the Comfort Zone: Emerging Solutions to Overcome Challenges in Integrating LLMs into Software Products. In47th IEEE/ACM International Conference on Software Engineering: Softw...

    2025

  57. [65]

    Arshaan Nazir, Thadaka Kalyan Chakravarthy, David Amore Cecchini, Rakshit Khajuria, Prikshit Sharma, et al. 2024. LangTest: A comprehensive evaluation library for custom LLM and NLP models.Softw. Impacts19 (2024), 100619. https://doi.org/10.1016/J.SIMPA.2024.100619

    2024

  58. [66]

    2021.Crafting Interpreters

    Robert Nystrom. 2021.Crafting Interpreters. Genever Benning. https://craftinginterpreters.com/

    2021

  59. [67]

    OpenAI. 2023. GPT-4 Technical Report.CoRRabs/2303.08774 (2023). https://doi.org/10.48550/ARXIV.2303.08774 arXiv:2303.08774

    2023 doi arXiv

  60. [68]

    Mauro Pezzè, Matteo Ciniselli, Luca Di Grazia, Niccolò Puccinelli, and Ketai Qiu. 2024. The Trailer of the ACM 2030 Roadmap for Software Engineering.ACM SIGSOFT Softw. Eng. Notes49, 4 (2024), 31–40. https://doi.org/10.1145/3696117.3696126

    2024

  61. [69]

    Emanuele Pucci, Ludovica Piro, Salvatore Andolina, and Maristella Matera. 2024. From Conversational Web to Inclusive Conversations with LLMs. InProceedings of the 2024 International Conference on Advanced Visual Interfaces, A VI 2024, Arenzano, Genoa, Italy, June 3-7, 2024, Cr...

    2024

  62. [70]

    Rabe and Charles Staats

    Markus N. Rabe and Charles Staats. 2021. Self-attention Does Not Need O(n 2) Memory.CoRRabs/2112.05682 (2021). arXiv:2112.05682 https: //arxiv.org/abs/2112.05682

    2021 arXiv

  63. [71]

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving Language Understanding by Generative Pre-Training. https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf

    2018

  64. [72]

    Traian Rebedea, Razvan Dinu, Makesh Narsimhan Sreedhar, Christopher Parisien, and Jonathan Cohen. 2023. NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails. InProceedings of the 2023 Conference on Empirical Methods in Natural Language ...

    2023 doi

  65. [73]

    Lillicrap, et al

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.CoRRabs/2403.05530 (2024). https://doi.org/10.48550/ARXIV.2403.05530 arXiv:2403.05530

    2024 doi arXiv

  66. [74]

    Morgane Rivière, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, et al. 2024. Gemma 2: Improving Open Language Models at a Practical Size.CoRRabs/2408.00118 (2024). https://doi.org/10.48550/ARXIV.2408.00118 arXiv:2408.00118

    2024 doi arXiv

  67. [75]

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, et al . 2023. Code Llama: Open Foundation Models for Code.CoRR abs/2308.12950 (2023). https://doi.org/10.48550/ARXIV.2308.12950 arXiv:2308.12950

    2023 doi arXiv

  68. [76]

    Bach, Lintang Sutawika, et al

    Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, et al. 2022. Multitask Prompted Training Enables Zero-Shot Task Generalization. InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenRevie...

    2022

  69. [77]

    Vincenzo Scotti and Mark James Carman. 2024. LLM Support for Real-Time Technical Assistance. InMachine Learning and Knowledge Discovery in Databases. Research Track and Demo Track - European Conference, ECML PKDD 2024, Vilnius, Lithuania, September 9-13, 2024, Proceedings, Par...

    2024 doi

  70. [78]

    Vincenzo Scotti, Licia Sbattella, and Roberto Tedesco. 2024. A Primer on Seq2Seq Models for Generative Chatbots.ACM Comput. Surv.56, 3 (2024), 75:1–75:58. https://doi.org/10.1145/3604281

    2024 doi

  71. [79]

    Yuchen Shao, Yuheng Huang, Jiawei Shen, Lei Ma, Ting Su, et al. 2025. Are LLMs Correctly Integrated into Software Systems?. In47th IEEE/ACM International Conference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 - May 6, 2025. IEEE, 1178–1190. https://doi.org...

    2025

  72. [80]

    Kensen Shi, Deniz Altinbüken, Saswat Anand, Mihai Christodorescu, Katja Grünwedel, et al. 2025. Natural Language Outlines for Code: Literate Programming in the LLM Era. InProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, FSE Compa...

    2025

  73. [81]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. InAdvances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, Neur...

    2023

  74. [82]

    Weapon of Mass Deception

    Alejo José G. Sison, Marco Tulio Daza, Roberto Gozalo-Brizuela, and Eduardo C. Garrido-Merchán. 2024. ChatGPT: More Than a "Weapon of Mass Deception" Ethical Challenges and Responses from the Human-Centered Artificial Intelligence (HCAI) Perspective.Int. J. Hum. Comput. Intera...

    2024

  75. [83]

    Lintang Sutawika, Hailey Schoelkopf, Leo Gao, Baber Abbasi, Stella Biderman, et al . 2024. EleutherAI/lm-evaluation-harness: v0.4.7. https: //doi.org/10.5281/zenodo.14506035

    2024 doi

  76. [84]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, et al. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. CoRRabs/2307.09288 (2023). https://doi.org/10.48550/ARXIV.2307.09288 arXiv:2307.09288

    2023 doi arXiv

  77. [85]

    Igor Tufanov, Karen Hambardzumyan, Javier Ferrando, and Elena Voita. 2024. LM Transparency Tool: Interactive Tool for Analyzing Transformer Language Models.CoRRabs/2404.07004 (2024). https://doi.org/10.48550/ARXIV.2404.07004 arXiv:2404.07004

    2024 doi arXiv

  78. [86]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, et al. 2017. Attention is All you Need. InAdvances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, Is...

    2017

  79. [87]

    Irene Weber. 2024. Large Language Models as Software Components: A Taxonomy for LLM-Integrated Applications.CoRRabs/2406.10300 (2024). https://doi.org/10.48550/ARXIV.2406.10300 arXiv:2406.10300

    2024 doi arXiv

  80. [88]

    Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, et al. 2021. Ethical and social risks of harm from Language Models. CoRRabs/2112.04359 (2021). arXiv:2112.04359 https://arxiv.org/abs/2112.04359

    2021 arXiv

  81. [89]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, et al. 2020. Transformers: State-of-the-Art Natural Language Processing. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, EMNLP 2020 - D...

    2020 doi

  82. [90]

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, et al. 2023. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework.CoRRabs/2308.08155 (2023). https://doi.org/10.48550/ARXIV.2308.08155 arXiv:2308.08155

    2023 doi arXiv

  83. [91]

    Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, et al. 2024. Explainability for Large Language Models: A Survey.ACM Trans. Intell. Syst. Technol.15, 2 (2024), 20:1–20:38. https://doi.org/10.1145/3639372 Manuscript submitted to ACM

    2024 doi

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.