AI-Assisted Modeling: DSL-Driven AI Interactions

Bernhard Steffen; Daniel Busch; Edward A. Lee; Moez Ben Haj Hmida; Steven Smyth

REVIEW 4 major objections 4 minor 14 references

AI-Assisted Modeling: DSL-Driven AI Interactions

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes that AI-assisted modeling for domain-specific languages becomes more reliable and controllable when generation is split into observable, interactive stages rather than one-shot prompts, with syntax delegated to Tool API f

desk verdict A plausible interactive DSL modeling workflow with a working prototype, but the reliability claim is unmeasured and the evaluation is thin. read the letter →

arxiv 2509.05160 v1 pith:IWRJOURF submitted 2025-09-05 cs.PL cs.SE

Steven Smyth , Daniel Busch , Moez Ben Haj Hmida , Edward A. Lee , Bernhard Steffen This is my paper · ORCID

classification cs.PLcs.SE

keywords domain-specificlanguagesAI-assistedmodelingLinguaFrancaobservationpointsinteractionToolAPIautomaticdiagramsynthesisincrementalmodelrefinement

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that AI-assisted model generation for domain-specific languages (DSLs) becomes more reliable and controllable when the generation is split into observable stages rather than a single one-shot prompt. The authors argue that for DSLs with unusual or polyglot syntax—Lingua Franca being their test case—general-purpose LLMs often fail silently, and that two mechanisms fix this: observation and interaction points at each transformation stage, and delegating syntax contextualization to Tool API functions that return concrete, grammar-valid fragments. If the claim holds, developers could build formal models through natural language and voice, see the diagram and source update instantly after every step, and correct mistakes early. The paper demonstrates the idea in a working Visual Studio Code extension and leaves quantitative validation—success rates, failure cases—to future work.

What carries the argument

The load-bearing mechanism is the pairing of observation points (OPs) and interaction points (IPs) with Tool API functions. An OP is a stage where an intermediate output—transcription, source, diagram, or diagnostics—is visible to the modeler; an IP is a stage where the modeler can intervene by voice, text, or direct editing. The Tool API functions, such as createTimer, give the LLM a way to obtain precise concrete syntax on demand, so the model does not have to memorize uncommon DSL syntax from prompt context alone. Automatic diagram synthesis via transient views and automatic layout turns the textual working model into immediate semantic feedback.

What would settle it

Give the same model-building task to the prototype's stage-wise Tool API workflow and to a plain one-shot prompt on a fixed set of Lingua Franca tasks, scoring syntax validity and semantic match blindly; if the one-shot variant matches or beats the staged variant, the claim that observation/interaction points improve reliability is not supported.

Watch

Extended reading notes

Core claim

The central claim is that placing observation and interaction points across the transformation chain—speech-to-text, prompt, generated source model, synthesized diagram, and later compiler diagnostics—converts DSL model generation from a black-box one-shot operation into a stage-wise, steerable feedback loop. Syntax contextualization is shifted from prompt text to Tool API descriptions: instead of spelling out Lingua Franca grammar in the prompt, the LLM is given callable functions such as createTimer that return standardized concrete syntax for single language elements, generated from the language grammar. The prototype shows this through a spoken and typed sequence of refinements that asse

Load-bearing premise

Everything rests on the LLM actually producing valid and semantically intended Lingua Franca fragments at each incremental step when guided by the hand-written Tool API descriptions, because the prototype reports no success rates or failure cases to verify this.

Editorial extensions

If this is right

Modelers can develop complete DSL models incrementally through natural-language or voice commands, with instant diagram feedback after each step, instead of relying on entire-model one-shot generation.
The observation/interaction-point structure should transfer directly to other graphical DSLs such as SCCharts, which already shares the transient-view diagram technology.
Tool API descriptions can be at least partially generated from grammar files with associated comments, lowering the cost of adding a new DSL to the workflow.
The approach is orthogonal to LLM improvements: fine-tuning and retrieval-based techniques can be combined with the stage-wise workflow to further improve output quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural next experiment is closing the loop on compiler diagnostics: the paper notes this observation point is not yet used, and feeding compile errors back into the LLM is the most direct test of whether stage-wise refinement actually improves correctness.
The grammar-to-Tool-API generation sketch implies a broader testing strategy: generate descriptors automatically for several DSLs and measure whether incremental tool-guided generation outperforms one-shot generation on each, separating the methodology from the specifics of Lingua Franca.
The same staged interaction design could be applied to general-purpose multilingual code, since Tool API calls that return concrete syntax localize correctness checks and give the user an early interception point.
Voice as an input modality may make modeling accessible to non-experts and non-native speakers, but this remains untested; a usability study comparing voice-driven and text-driven refinement would be needed.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Josh—

Quick take on arXiv:2509.05160. The genuinely new piece is the staged observation/interaction-point model for DSL generation, plus the use of Tool API calls as grammar-level syntax providers. That combination is not in the cited literature, and the Lingua Franca demonstrator in VS Code looks real. The demo video, if it matches the transcript in Fig. 4, shows a smooth voice-driven iterative workflow with instant diagram synthesis.

The authors are also honest about some limits. They explicitly say the compiler diagnostics observation point is not yet utilized, and that the optimal granularity of Tool API functions remains an open question. They don't pretend the SCCharts one-shot example in Fig. 1 is complete. The related work is fair, and the self-citations are all background.

Where the paper falls short is the central claim: that the approach improves reliability and controllability of model generation. Reliability is not measured. No success rates, no baseline comparison against one-shot or iterative prompting, no failure analysis. The only worked example in the text and video shows the LLM missing Lingua Franca's mandatory target language. That's a controllability success—the modeler corrects it and moves on—but it does not support a reliability improvement. The abstract also mentions formal verification such as model checking, but that's motivation, not integrated behavior.

The stress-test note is mostly right. I'd push back on one point: the createTimer tool description omitting the attributes parameter is a concrete instance of the handwritten-description problem, but it's not fatal. The authors acknowledge the approach is early and that the API could be generated from grammar comments. The real issue is you could ship this as a recipe for people to build on, but the paper itself gives no evidence that the recipe works more reliably than existing iterative IDE agents.

Bottom line: this is a worthwhile tool paper for the model-driven engineering and AI-assist communities. It deserves a serious referee, but the referee should push for empirical grounding—even a small user study with task completion rates and failure cases would transform it. I wouldn't cite it yet in my own work, but I'd bring it to a reading group.

Recommendation: accept for peer review, with major revision expectations.

Referee Report

4 major / 4 minor

Summary. The paper proposes an AI-assisted modeling workflow for domain-specific languages (DSLs) in which modelers interact with an LLM through speech or text, receive instantaneous graphical diagram feedback, and refine models in a stage-wise manner. The central idea is to increase the number of observation and interaction points in the transformation pipeline and to delegate syntactic contextualization to Tool API functions rather than relying on one-shot prompting. The approach is demonstrated by a Visual Studio Code extension for the Lingua Franca language, which combines whisper-1 transcription, an o4-mini LLM, Tool API definitions for LF constructs, and automatic diagram synthesis via the existing LF/KIELER toolchain. The paper contributes the general workflow concept, a running prototype, a short transcript of a successful modeling session, and a discussion of how Tool API definitions might be generated from the language grammar.

Significance. If the workflow's claims were supported, the paper would make a useful contribution to DSL tooling: separating LLM interactions into typed, grammar-aligned tool calls and providing multi-stage visual feedback is a plausible way to make AI-assisted model generation more controllable than one-shot prompting. The prototype is a concrete, reproducible artifact, with a demonstration video, and the authors are transparent about several open questions, including Tool API granularity and the absence of compiler diagnostics in the current loop. However, the paper's headline claims of improved reliability and of support for formal verification such as model checking are not substantiated by the presented evidence. The single transcript actually contains a correctness failure (the omitted mandatory target language), and no quantitative evaluation or comparison against baselines is provided. The contribution is therefore better framed as a workflow architecture with a feasibility demonstration than as a validated improvement in reliability.

major comments (4)

[§1 Contribution / §3 Prototype Demonstrator] The central claim that the workflow 'improves the reliability and controllability of model generation' is not supported by the evidence. The only demonstration is a single successful transcript (Fig. 4 and the video), and footnote 5 records that the assistant failed to set LF's mandatory target language, requiring modeler correction. No success rate, failure analysis, or comparison against one-shot prompting or a tool-less iterative loop is reported. Since the Tool API is the main mechanism intended to ensure syntactic validity, its reliability is load-bearing; yet the manual createTimer description (Lst. 1) omits the attributes parameter that the grammar (Lst. 2) allows and the grammar-generated description (Lst. 3) includes. To support the reliability clause, the authors should add a controlled evaluation (e.g., generate a set of LF models with and without Tool-API decomposition, measu
[Abstract / §1 Introduction] The abstract and introduction state that the approach 'facilitates visual inspection and formal verification, such as model checking.' However, no model checking is performed in the paper; the prototype only provides automatic diagram synthesis. The paper itself notes that the compiler diagnostics observation point is 'not yet utilized in the current prototype' (§2). The phrase 'formal verification' therefore overstates what is demonstrated. Please either remove this claim or clearly mark it as future work, and describe the current validation as 'visual inspection of automatically synthesized diagrams.'
[§2 Observation and Interaction] The notion of 'immediate feedback after each transformation step' is weaker than it appears because compiler diagnostics are not wired into the loop. The current feedback consists of the LLM-produced text and the diagram; diagram synthesis may reject malformed fragments, but no semantic or type-level validation is performed. Thus the 'validation support' mentioned in the abstract is not realized. If the paper's contribution is the workflow itself, this is acceptable, but the text should not imply automated validation of the generated model beyond what the diagram synthesis provides.
[§3 Future Meta Tooling] The claim that Tool API descriptions can be generated automatically from grammar comments is plausible but not demonstrated at scale. Listing 3 shows a ChatGPT-generated description for one construct (a timer), but no evidence is given that such descriptions are complete or correct across LF's syntax. Because the correctness of the generated tool descriptions is directly tied to the reliability of the overall approach, the paper should include either a systematic comparison of generated vs. manually written descriptions over multiple constructs or an explicit statement that this is a hypothesis that remains to be validated.

minor comments (4)

[Listing 2 caption] Typo: 'definintion' should be 'definition'.
[Figure 1 prompt] Typo: 'controler' should be 'controller'.
[§4 Related Work] Ungrammatical sentence: 'These trend represents an orthogonal direction' should be 'These trends represent an orthogonal direction'.
[Figure 3] The observation/interaction point annotations (IP/OP markers) are hard to read in the PDF. Consider enlarging the figure or adding a legend table that maps each marker to its description.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are architectural and prototype-demonstrated, not derived from fitted inputs or from a self-citation chain.

full rationale

This paper contains no formal derivation chain, fitted parameters, or first-principles prediction whose output is equivalent to its input. The core "derivation" is an engineering proposal: introduce observation/interaction points and delegate syntax contextualization to Tool API functions. The Tool API mechanism is described transparently as providing exact source code for language elements, so the correctness of generated syntax is by construction of the tool call, not a hidden empirical prediction smuggled back as a result. The paper explicitly notes that other techniques (RAG, fine-tuning) are alternatives, so the choice of Tool API is not forced by a self-citation or uniqueness theorem. Self-citations [2,3] are background references for earlier LLM-and-DSL work and are not the load-bearing justification for the current prototype's claims; the prototype is demonstrated with external components (whisper-1, o4-mini, the Lingua Franca extension, ELK/KIELER). The paper's own limitations, such as Section 2's statement that compiler diagnostics are "not yet utilized in the current prototype," Section 3's admission that Tool API calls "have been created manually," and the footnote reporting the missing mandatory target language, weaken the empirical support for the "improves reliability" claim, but they are evidence of unsupported strength of claim, not circularity. No step reduces to its own input, and no fitted value is renamed as a prediction. Therefore the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No fitted parameters or invented physical/formal entities appear. The paper's contribution is a workflow and prototype; the assumptions above are the load-bearing premises about LLM reliability, speech accuracy, and semantic fidelity of diagram synthesis.

assumptions (4)

domain assumption An LLM guided by Tool API functions can produce syntactically valid and semantically intended DSL fragments for each language construct.
Section 2 states that Tool API calls are used to retrieve correct syntax; Section 3 relies on manually written descriptors. No systematic success-rate data is provided.
domain assumption Speech transcription via whisper-1 is accurate and timely enough to support rapid modeling.
Section 3 reports 'preliminary tests' suggesting transcription is almost instant, but no evaluation or transcription accuracy data is given.
domain assumption Automatic synthesis of diagrams from textual Lingua Franca models provides a semantically accurate basis for visual inspection.
Section 1 claims visualizations 'accurately represent the semantics' of AI-generated code; this depends on the Lingua Franca extension and KIELER layout engine, and is not formally verified.
domain assumption Incremental LLM-produced edits to a textual model preserve the validity and intended semantics of the overall model.
The workflow in Section 2 assumes each transformation can be merged into the current model; the prototype does not yet use compilation diagnostics to check this, as admitted in Section 3.

how reviews work

0 comments

Cite this review

Pith. "Pith review of AI-Assisted Modeling: DSL-Driven AI Interactions." pith.science (2026). https://pith.science/paper/IWRJOURF

@misc{pith2026250905160,
  author       = {Pith},
  title        = {Pith review of: AI-Assisted Modeling: DSL-Driven AI Interactions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IWRJOURF}},
  note         = {Machine review of arXiv:2509.05160}
}

read the original abstract

AI-assisted programming greatly increases software development performance. We enhance this potential by integrating transparency through domain-specific modeling techniques and providing instantaneous, graphical visualizations that accurately represent the semantics of AI-generated code. This approach facilitates visual inspection and formal verification, such as model checking. Formal models can be developed using programming, natural language prompts, voice commands, and stage-wise refinement, with immediate feedback after each transformation step. This support can be tailored to specific domains or intended purposes, improving both code generation and subsequent validation processes. To demonstrate the effectiveness of this approach, we have developed a prototype as a Visual Studio Code extension for the Lingua Franca language. This prototype showcases the potential for novel domain-specific modeling practices, offering an advancement in how models are created, visualized, and verified.

Figures

Figures reproduced from arXiv: 2509.05160 by the authors.

**Figure 1.** SCCharts one-shot AI Steamboiler with prompt (Smyth, SYNCHRON’23) referring to the specification problem from Abrial [1] Solution. We combine (i) state-of-the-art DSL modeling techniques with instantaneous automatic visualizations, and (ii) AIassisted programming to enable iterative and interactive support for DSL development. (i) Textual domain-specific (natural) languages are well suited to guide AI assistants—m… view at source ↗

**Figure 2.** The modeling concept of DSL-driven AI interaction: A modeler interacts with the system through speech or text inputs and receives feedback from multiple transformation stages, including instantaneous diagram synthesis and compilation results. This workflow forms a feedback loop, enabling both the modeler and the system to incrementally refine the model. in which case the speech basically becomes the sole input for t… view at source ↗

**Figure 3.** Demonstrator of rapid, natural modeling with Lingua Franca in Visual Studio Code. The observation points [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Demonstration of the prototype in action, following the transcript from the accompanying video [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 12 canonical work pages

[1]

Jean-Raymond Abrial. 2005. Steam-boiler control specification problem.Formal Methods for Industrial Applications: Specifying and Programming the Steam Boiler Control(2005), 500–509

work page 2005
[2]

Daniel Busch, Alexander Bainczyk, Steven Smyth, and Bernhard Steffen. 2025. LLM-Based Code Generation and System Migration in Language-Driven Engi- neering. InSTTT. Springer, 375–390

work page 2025
[3]

Daniel Busch, Gerrit Nolte, Alexander Bainczyk, and Bernhard Steffen. 2023. ChatGPT in the loop: a natural language extension for domain-specific modeling languages. InInternational Conference on Bridging the Gap between AI and Reality. Springer, 375–390

work page 2023
[4]

Marten Lohstroh, Christian Menard, Soroush Bateni, and Edward A Lee. 2021. Toward a lingua franca for deterministic concurrent systems.ACM Transactions on Embedded Computing Systems (TECS)20, 4 (2021), 1–27

work page 2021
[5]

Christian Schneider, Miro Spönemann, and Reinhard von Hanxleden. 2013. Just Model! – Putting Automatic Synthesis of Node-Link-Diagrams into Practice. InProceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC ’13). IEEE, San Jose, CA, USA, 75–82. doi:10.1109/VLHCC. 2013.6645246

work page arXiv 2013
[7]

Steven Smyth, Alexander Schulz-Rosengarten, and Reinhard von Hanxleden

work page
[8]

Dominik Sobania, Martin Briesch, Carol Hanna, and Justyna Petke. 2023. An analysis of the automatic bug fixing performance of chatgpt. In2023 IEEE/ACM International Workshop on Automated Program Repair (APR). IEEE, 23–30

work page 2023
[9]

Bernhard Steffen, Falk Howar, and Maik Merten. 2011. Introduction to active automata learning from a practical perspective.Formal Methods for Eternal Networked Software Systems: 11th International School on Formal Methods for the Design of Computer, Communication and Software Systems, SFM 2011, Bertinoro, Italy, June 13-18, 2011. Advanced Lectures 11(2011...

work page 2011

Show all 14 references

[10]

Haoye Tian, Weiqi Lu, Tsz On Li, Xunzhu Tang, Shing-Chi Cheung, Jacques Klein, and Tegawendé F Bissyandé. 2023. Is ChatGPT the ultimate programming assistant–how far is it?arXiv preprint arXiv:2304.11938(2023)

2023 arXiv
[11]

Norbert Tihanyi, Ridhi Jain, Yiannis Charalambous, Mohamed Amine Ferrag, Youcheng Sun, and Lucas C Cordeiro. 2023. A new era in software security: Towards self-healing software via large language models and formal verification. arXiv preprint arXiv:2305.14752(2023)

2023 arXiv
[12]

Reinhard von Hanxleden, Björn Duderstadt, Christian Motika, Steven Smyth, Michael Mendler, Joaquín Aguado, Stephen Mercer, and Owen O’Brien. 2014. SCCharts: Sequentially Constructive Statecharts for Safety-Critical Applications. InProc. ACM SIGPLAN Conference on Programming La...

2014
[13]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reason- ing in large language models.Advances in neural information processing systems 35 (2022), 24824–24837

2022
[2018]

InProceedings of the 8th Interna- tional Symposium on Leveraging Applications of Formal Methods, Verification and Validation (ISoLA 2018) (LNCS, Vol

Towards Interactive Compilation Models. InProceedings of the 8th Interna- tional Symposium on Leveraging Applications of Formal Methods, Verification and Validation (ISoLA 2018) (LNCS, Vol. 11244). Springer, Limassol, Cyprus, 246–260. doi:10.14279/tuj.eceasst.78.1098

2018 doi
[2019]

Guidance in model-based compilations.Electronic Communications of the EASST78 (2019)

2019

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Jean-Raymond Abrial. 2005. Steam-boiler control specification problem.Formal Methods for Industrial Applications: Specifying and Programming the Steam Boiler Control(2005), 500–509

work page 2005

[2] [2]

Daniel Busch, Alexander Bainczyk, Steven Smyth, and Bernhard Steffen. 2025. LLM-Based Code Generation and System Migration in Language-Driven Engi- neering. InSTTT. Springer, 375–390

work page 2025

[3] [3]

Daniel Busch, Gerrit Nolte, Alexander Bainczyk, and Bernhard Steffen. 2023. ChatGPT in the loop: a natural language extension for domain-specific modeling languages. InInternational Conference on Bridging the Gap between AI and Reality. Springer, 375–390

work page 2023

[4] [4]

Marten Lohstroh, Christian Menard, Soroush Bateni, and Edward A Lee. 2021. Toward a lingua franca for deterministic concurrent systems.ACM Transactions on Embedded Computing Systems (TECS)20, 4 (2021), 1–27

work page 2021

[5] [5]

Christian Schneider, Miro Spönemann, and Reinhard von Hanxleden. 2013. Just Model! – Putting Automatic Synthesis of Node-Link-Diagrams into Practice. InProceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC ’13). IEEE, San Jose, CA, USA, 75–82. doi:10.1109/VLHCC. 2013.6645246

work page arXiv 2013

[6] [7]

Steven Smyth, Alexander Schulz-Rosengarten, and Reinhard von Hanxleden

work page

[7] [8]

Dominik Sobania, Martin Briesch, Carol Hanna, and Justyna Petke. 2023. An analysis of the automatic bug fixing performance of chatgpt. In2023 IEEE/ACM International Workshop on Automated Program Repair (APR). IEEE, 23–30

work page 2023

[8] [9]

Bernhard Steffen, Falk Howar, and Maik Merten. 2011. Introduction to active automata learning from a practical perspective.Formal Methods for Eternal Networked Software Systems: 11th International School on Formal Methods for the Design of Computer, Communication and Software Systems, SFM 2011, Bertinoro, Italy, June 13-18, 2011. Advanced Lectures 11(2011...

work page 2011