Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

Querying Large Automotive Software Models: Agentic vs. Direct LLM Approaches

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A tool-using LLM agent can answer structural questions about a large automotive software model with accuracy comparable to full-context prompting while consuming a tiny fraction of the tokens, the paper reports.

desk verdict A real, reproducible token-efficiency result is oversold by an accuracy claim the paper's own table contradicts; worth refereeing with major revisions. read the letter →

arxiv 2506.13171 v1 pith:LQBV775N submitted 2025-06-16 cs.SE cs.AI

Lukasz Mazur , Nenad Petrovic , James Pontes Miranda , Ansgar Radermacher , Robert Rasche , Alois Knoll This is my paper
classification cs.SEcs.AI
keywords softwaremodelqueryingagenticLLMEcoremetamodelautomotivetokenefficiencyLLM-as-a-judgemodel-drivenengineeringReAct
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper experimentally compares two ways of getting a large language model to answer questions about a software model: pasting the entire model into the prompt, versus letting a tool-using agent open, search, and scroll through the model file on demand. On a 13,572-line automotive metamodel in Ecore format that tokenizes to roughly 118,000 prompt tokens, the agentic setup cuts prompt tokens to roughly 640-780 while preserving 90% accuracy for the two models that handle tool use well. For two other models, accuracy falls sharply in agent mode, so the efficiency gain is not uniform across LLMs. The paper argues that because automotive models are large, privacy-sensitive, and often exceed practical context limits, agentic querying is not merely a cheaper alternative but the only viable path for local deployment. If the results hold, engineers can query large models with small, locally run models and avoid sending proprietary designs to external APIs.

What carries the argument

The load-bearing mechanism is the ReAct loop: the LLM alternates reasoning with tool calls, deciding when to open, search, scroll, or jump within a file, and the agent sees only a 50-line window (2-line overlap) of the model at a time. The prompt initially contains only the path to the model directory, not the model content, so context grows incrementally with whatever the agent chooses to retrieve. This stands in contrast to the reference setup, which places the entire 13,572-line Ecore file in the prompt. The measured quantity that makes the comparison work is token usage: the agent's prompt tokens are roughly 180 times smaller than the direct prompt's, and for capable models the answer quality does not pay for that saving.

What would settle it

Re-score the 20 answers with human experts or with a judge model that is not one of the systems being compared; if the agent's judged accuracy falls materially below direct prompting, the comparability claim fails. Run the same setup on a multi-file or behavioral model; if the agent's advantage disappears, the 'only viable solution' conclusion does not generalize.

Watch

Extended reading notes

Core claim

The central claim is that a ReAct-style agent equipped with general file-access tools can answer structural questions about a software model with accuracy comparable to direct full-context prompting, provided the underlying LLM is capable enough to orchestrate tool calls, while using orders of magnitude fewer tokens. The authors report 90% accuracy for GPT-4.1 mini and o4-mini in both the direct and agent configurations, versus roughly 118,000-137,000 prompt tokens for direct prompting and 640-780 for the agent. For Gemini 2.5 Flash and GPT-4o mini, the agent setup degrades accuracy (from 80% to 40% and from 45% to 10%, respectively), which the authors attribute to weaker state coherence and tool orchestration. The dominant failure mode in agent mode is incomplete inheritance traversal and, for some models, looping until the 100-iteration limit; these failures were counted as incorrect. The paper concludes that for large automotive models, where full-model prompting is infeasible or undesirable for privacy reasons, the agentic approach is not just practical but the only viable solution.

Load-bearing premise

The comparison stands on the assumption that accuracy judged by GPT-4.1 mini on 20 hand-written structural questions about one Ecore model is a valid proxy for human-judged correctness across automotive model-querying tasks.

Editorial extensions

If this is right

  • For models that can orchestrate tool calls well, agentic querying preserves accuracy (90% for GPT-4.1 mini and o4-mini) while reducing prompt tokens from roughly 118,000-137,000 to 640-780.
  • For models that struggle with tool use, accuracy falls, with Gemini 2.5 Flash dropping from 80% to 40% and GPT-4o mini from 45% to 10%, so the token saving does not come for free for every LLM.
  • Direct full-context prompting remains at least as accurate as the agent whenever the whole model fits in the context window; the agent's real advantage is efficiency, not better answers.
  • The token reduction makes locally deployable, privacy-compliant smaller models practical for querying large software artifacts.
  • The dominant failure in agent mode is incomplete inheritance traversal and looping until the iteration limit, so further gains are likely to come from better retrieval rather than more prompt engineering.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A consequence the authors leave implicit is that even a small accuracy penalty would often be acceptable in industry, because local execution and IP protection can outweigh a few percentage points of question-answering accuracy.
  • Because the agent's tools are generic file operations, adding model-aware tools such as direct class lookup and automatic inheritance expansion would target the dominant failure mode the paper identified rather than relying on the LLM to browse XML by hand.
  • The token gap is so large that the paper's 'only viable solution' claim can be tested by scaling past the 118,000-token test model: on a multi-file or larger model, direct prompting would simply not fit, making the comparison one-sided.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper reports a small empirical study comparing two ways of querying a large Ecore metamodel (about 118,000 tokens) with LLMs: direct full-context prompting and a ReAct-style agent equipped with file-access tools. Using 20 hand-written structural questions and four LLMs (GPT-4o mini, GPT-4.1 mini, o4-mini, Gemini 2.5 Flash), it measures answer accuracy (with LLM-as-a-judge), precision/recall/F1, and token usage. The authors find that the agentic approach uses far fewer prompt tokens and claim that it achieves accuracy comparable to direct prompting, concluding that agents are 'the only viable solution' for large automotive models. The paper also discusses failure modes and threats to validity.

Significance. If the accuracy equivalence held across models, the result would be practically valuable: it would show that locally deployable LLMs with tool use can query large structured models without paying the full-context token cost. The study is transparent about its setup (open-source model, defined tools, deterministic temperature settings) and the token-count tables are clear. However, the headline claim is model-dependent and the efficiency advantage is structural rather than empirically discovered. The paper's contribution is therefore a modest, useful data point on agentic vs. direct prompting for MDE, not a definitive demonstration that agents are the only viable approach.

major comments (4)
  1. [Abstract, Section IV.A, Section V, Section VII] The abstract and conclusion state that 'the agentic approach achieves accuracy comparable to direct prompting,' but Table III contradicts this for two of the four models: GPT-4o mini drops from 45% to 10% (2/20 correct) and Gemini 2.5 Flash from 80% to 40% (8/20 correct) in agent mode. Section V itself acknowledges these drops, creating an internal inconsistency. The claim 'only viable solution' is further unsupported because no alternative retrieval-based baselines (e.g., RAG) were compared, and the 118k-token model fits within every assessed context window. Please qualify the accuracy claim to models with strong tool-use capabilities, report per-model statistical tests (e.g., Fisher's exact test on the 20-question counts), and soften the 'only viable' conclusion.
  2. [Section IV.B, Table IV] The token-usage reduction is a structural property of the comparison, not an empirical finding: direct prompting necessarily ingests the whole model (~118k–137k prompt tokens), while the agent reads only small windows (~640–780 prompt tokens). This should be framed as an expected design consequence, not a 'striking' discovery. More importantly, the efficiency advantage does not by itself establish 'only viable solution,' since any retrieval-based approach (RAG, indexing, etc.) would also avoid full-context ingestion; the paper does not compare against such alternatives.
  3. [Section III.E, Section VI.c] The evaluation uses GPT-4.1 mini as the LLM-as-a-judge, and GPT-4.1 mini is also one of the models being evaluated. This creates a risk of self-preference bias in the primary accuracy metric. The paper mentions only 'manual spot-checking for a randomly selected subset' without reporting the spot-check results or quantifying agreement. Please add a human-evaluated subset (or use a judge model outside the evaluated set) and report inter-rater reliability or a bias analysis.
  4. [Section III.B, Section VI.d] The agent uses a window size of 50 lines and an overlap of 2 lines, selected 'empirically' without sensitivity analysis. Section V reports that models often fail to scroll or search effectively, so the windowing parameters likely have a material impact on agent accuracy. Since this is a free parameter of the design, the paper should either provide a sensitivity test or explicitly discuss how the result might change with different window sizes. This is load-bearing for the accuracy comparison.
minor comments (5)
  1. [Section IV.A vs. Abstract] The sentence in Section IV.A, 'Overall, the direct full-context prompting setup performed better or comparably for several models,' already contradicts the abstract's unconditional 'comparable accuracy' claim; please align the abstract and discussion.
  2. [Section VII] The conclusion states 'The agentic approach proved effective with smaller LLMs,' but GPT-4o mini's 10% agent accuracy demonstrates that this is not true for all small models; please specify which models were effective.
  3. [Table IV caption] The prompt-token count for direct prompting varies across models (e.g., 137,461 tokens for Gemini 2.5 Flash vs. ~118,776 for others) presumably due to tokenizer differences; please state this explicitly in the text.
  4. [Section III.D] Please state whether the 20-question dataset and the reference answers are publicly available (e.g., in a repository) to support reproducibility.
  5. [Section VI] The threats-to-validity list is useful but does not rank the severity of each threat; consider identifying which threats most affect the central claim (e.g., LLM-as-judge bias and single-question-set design).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the accuracy and token-usage comparisons are empirical measurements, and the token-efficiency gap, while structurally expected, is reported as a measured outcome rather than derived from a definition.

full rationale

The paper's central comparison is between two differently defined architectures: direct full-context prompting always ingests the entire model, while the agent reads windowed excerpts on demand. The token-efficiency difference follows from these definitions, but the paper does not present it as a derivation or a prediction; it reports measured token counts (e.g., 118,776 vs. 639 prompt tokens for GPT-4.1 mini) and explicitly calls the difference 'striking, though not surprising.' The accuracy claim is empirical and model-dependent: Table III shows that Gemini 2.5 Flash drops from 80% to 40% and GPT-4o mini from 45% to 10% in agent mode, so the abstract's 'comparable accuracy' generalization is an overstatement, but that is an internal-evidence inconsistency, not a circular reduction. The use of GPT-4.1 mini as both judge and one of the evaluated models is a methodological validity threat acknowledged in Section VI; it does not make correctness equal to model output by construction. There are no load-bearing self-citations, no fitted parameters renamed as predictions, and no uniqueness or ansatz imported from prior work by the same authors. The paper's limitations (single model, 20 questions, LLM-as-a-judge) are explicitly listed and weigh on generalizability, not on circularity. No step reduces to its own input by definition or by self-citation, so the appropriate finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No new entities are introduced. The modeling choices amount to two free parameters (window size, iteration limit) and three domain assumptions about evaluation validity and representativeness. The efficiency result is largely structural rather than empirical, while the accuracy result depends on the assumptions.

free parameters (2)
  • file window size and overlap = 50 lines, 2-line overlap
    Section III-B.2 states these were 'selected empirically to balance efficiency and context completeness' for the specific model, affecting both token usage and accuracy.
  • agent iteration limit = 100
    Section III-B.2 sets a recursion limit of 100 to avoid infinite loops; the failure analysis shows 15 recursive errors, so this threshold influences the measured accuracy.
assumptions (3)
  • domain assumption The software model is the complete scope of truth for correctness judgments.
    Section III-E: 'the software model defines the complete scope of truth' for factual correctness. This treats the Ecore file as authoritative and ignores any external semantics.
  • domain assumption LLM-as-a-Judge with GPT-4.1 mini yields valid correctness scores, with only random spot-checking by humans.
    Section III-E acknowledges LLM judges may misjudge and uses one of the evaluated models as judge, but only performs manual spot checks on a random subset, so judge validity is assumed.
  • domain assumption The INCHRON Ecore metamodel and the 20 hand-written questions are representative of automotive software-model querying tasks.
    Section III-D uses a single model and 20 questions in four structural categories; the threats to validity section admits this limits generalizability, yet broad conclusions are drawn from it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Querying Large Automotive Software Models: Agentic vs. Direct LLM Approaches." pith.science (2026). https://pith.science/paper/LQBV775N

@misc{pith2026250613171,
  author       = {Pith},
  title        = {Pith review of: Querying Large Automotive Software Models: Agentic vs. Direct LLM Approaches},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LQBV775N}},
  note         = {Machine review of arXiv:2506.13171}
}
read the original abstract

Large language models (LLMs) offer new opportunities for interacting with complex software artifacts, such as software models, through natural language. They present especially promising benefits for large software models that are difficult to grasp in their entirety, making traditional interaction and analysis approaches challenging. This paper investigates two approaches for leveraging LLMs to answer questions over software models: direct prompting, where the whole software model is provided in the context, and an agentic approach combining LLM-based agents with general-purpose file access tools. We evaluate these approaches using an Ecore metamodel designed for timing analysis and software optimization in automotive and embedded domains. Our findings show that while the agentic approach achieves accuracy comparable to direct prompting, it is significantly more efficient in terms of token usage. This efficiency makes the agentic approach particularly suitable for the automotive industry, where the large size of software models makes direct prompting infeasible, establishing LLM agents as not just a practical alternative but the only viable solution. Notably, the evaluation was conducted using small LLMs, which are more feasible to be executed locally - an essential advantage for meeting strict requirements around privacy, intellectual property protection, and regulatory compliance. Future work will investigate software models in diverse formats, explore more complex agent architectures, and extend agentic workflows to support not only querying but also modification of software models.

Figures

Figures reproduced from arXiv: 2506.13171 by the authors.

Figure 1
Figure 1. Evaluation setups for software model question answering. (a) Direct LLM call with full model in prompt. (b) Agent using file-access tools to retrieve [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Example responses to a model-related question using (a) Direct Full [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Large Language Models for Fault Localization: An Empirical Study

    cs.SE 2025-10 conditional novelty 4.0 of 10

    An empirical study of four LLMs for statement-level Java fault localization finds Gemini-2.5-Flash most accurate, bug-report context strongly helpful, and few-shot/CoT gains inconsistent.

  2. Survey of GenAI for Automotive Software Development: From Requirements to Executable Code

    cs.SE 2025-07 conditional novelty 3.0 of 10

    A review of roughly 60 papers and 9 industry respondents finds GPT-family models dominate automotive code generation while requirements handling lags due to confidentiality constraints.

Reference graph

Works this paper leans on

35 extracted references · 29 canonical work pages · cited by 2 Pith papers

  1. [1]

    Engineering automotive software,

    M. Broy, I. H. Kruger, A. Pretschner, and C. Salzmann, “Engineering automotive software,”Proc. IEEE, vol. 95, no. 2, pp. 356–373, Feb

    work page

  2. [2]

    Automotive software engineering: A systematic mapping study,

    A. Haghighatkhah, A. Banijamali, O.-P. Pakanen, M. Oivo, and P. Ku- vaja, “Automotive software engineering: A systematic mapping study,” J. Syst. Softw., vol. 128, pp. 25–55, Jun. 2017. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0164121217300560

    work page 2017

  3. [3]

    Large language models for software engineering: Survey and open problems,

    A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for software engineering: Survey and open problems,” inProc. IEEE/ACM Int. Conf. on Softw. Eng. (ICSE), May 2023, pp. 31–53. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/10449667

    work page arXiv 2023

  4. [4]

    PlantUML

    “PlantUML.” [Online]. Available: https://plantuml.com/

    work page

  5. [5]

    Limitations of ChatGPT in conceptual modeling: insights from experiments in metamodeling,

    F. Muff and H.-G. Fill, “Limitations of ChatGPT in conceptual modeling: insights from experiments in metamodeling,” inModellierung 2024 - Workshopband. Gesellschaft f ¨ur Informatik e.V ., Mar. 2024. [Online]. Available: https://dl.gi.de/handle/20.500.12116/43782

    work page 2024

  6. [6]

    On the use of large language models in model-driven engineering,

    J. Di Rocco, D. Di Ruscio, C. Di Sipio, P. T. Nguyen, and R. Rubei, “On the use of large language models in model-driven engineering,” Softw. Syst. Model., vol. 24, no. 3, pp. 923–948, Jun. 2025. [Online]. Available: https://doi.org/10.1007/s10270-025-01263-8

    doi:10.1007/s10270-025-01263-8 2025

  7. [7]

    Unified Modeling Language (UML) Specification Version 2.5.1

    “Unified Modeling Language (UML) Specification Version 2.5.1.” [Online]. Available: https://www.omg.org/spec/UML/

    work page

  8. [8]

    Eclipse Modeling Framework (EMF)

    “Eclipse Modeling Framework (EMF).” [Online]. Available: https: //eclipse.dev/emf/

    work page

Show all 35 references
  1. [9]

    AUTOSAR (Automotive Open System Architecture)

    “AUTOSAR (Automotive Open System Architecture).” [Online]. Available: https://www.autosar.org/

  2. [10]

    Model-based automotive software development,

    K. Vinoth Kannan, “Model-based automotive software development,” inAutomotive Embedded Systems: Key Technologies, Innovations, and Applications, M. Kathiresh and R. Neelaveni, Eds. Cham: Springer International Publishing, Apr. 2021, pp. 71–87. [Online]. Available: https://doi....

    2021 doi

  3. [11]

    Understanding the landscape of software modelling assistants for MDSE tools: A systematic mapping,

    D. Mosquera, M. Ruiz, O. Pastor, and J. Spielberger, “Understanding the landscape of software modelling assistants for MDSE tools: A systematic mapping,”Inf. Softw. Technol., vol. 173, p. 107492, Sep. 2024. [Online]. Available: https://www.sciencedirect.com/science/ article/pi...

    2024

  4. [12]

    Table meets LLM: Can large language models understand structured table data? A benchmark and empirical study,

    Y . Sui, M. Zhou, M. Zhou, S. Han, and D. Zhang, “Table meets LLM: Can large language models understand structured table data? A benchmark and empirical study,” inProc. 17th ACM Int. Conf. Web Search Data Mining, ser. WSDM ’24. New York, NY , USA: Association for Computing Mac...

    2024

  5. [13]

    Retrieval-augmented generation for large language models: A survey,

    Y . Gao, Y . Xiong, X. Gao, K. Jia, J. Pan, Y . Bi, Y . Dai, J. Sun, M. Wang, and H. Wang, “Retrieval-augmented generation for large language models: A survey,” Mar. 2024, arXiv:2312.10997. [Online]. Available: http://arxiv.org/abs/2312.10997

    2024 arXiv

  6. [14]

    Cognitive architectures for language agents,

    T. Sumers, S. Yao, K. Narasimhan, and T. Griffiths, “Cognitive architectures for language agents,”Trans. Mach. Learn. Res., Oct

  7. [15]

    ART: Automatic multi-step reasoning and tool-use for large language models,

    B. Paranjape, S. Lundberg, S. Singh, H. Hajishirzi, L. Zettlemoyer, and M. T. Ribeiro, “ART: Automatic multi-step reasoning and tool-use for large language models,” Mar. 2023, arXiv:2303.09014. [Online]. Available: http://arxiv.org/abs/2303.09014

    2023 arXiv

  8. [16]

    Model Context Protocol

    “Model Context Protocol.” [Online]. Available: https:// modelcontextprotocol.io/

  9. [17]

    Agent2Agent Protocol (A2A)

    “Agent2Agent Protocol (A2A).” [Online]. Available: https://google-a2a. github.io/A2A/

  10. [18]

    Chain-of-Thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. V . Le, and D. Zhou, “Chain-of-Thought prompting elicits reasoning in large language models,” inAdvances Neural Inf. Process. Syst., S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds.,...

    2022

  11. [19]

    Self-consistency improves chain of thought reasoning in language models,

    X. Wang, J. Wei, D. Schuurmans, Q. V . Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,” inProc. 11th Int. Conf. Learn. Represent. (ICLR), Sep. 2022. [Online]. Available: https://openreview.net/forum...

    2022

  12. [20]

    Evaluating open-domain question answering in the era of large language models,

    E. Kamalloo, N. Dziri, C. Clarke, and D. Rafiei, “Evaluating open-domain question answering in the era of large language models,” inProc. 61st Annu. Meeting Assoc. Comput. Linguistics, A. Rogers, J. Boyd-Graber, and N. Okazaki, Eds. Toronto, Canada: Association for Computation...

    2023

  13. [21]

    Evaluation of semantic answer similarity metrics,

    F. Mustafazade and P. F. Ebbinghaus, “Evaluation of semantic answer similarity metrics,”Int. J. Natural Lang. Comput., vol. 11, no. 3, pp. 43–57, Jun. 2022. [Online]. Available: http://arxiv.org/abs/2206.12664

    2022 arXiv

  14. [22]

    Can LLMs replace human evaluators? An empirical study of LLM-as-a-Judge in software engineering,

    R. Wang, J. Guo, C. Gao, G. Fan, C. Y . Chong, and X. Xia, “Can LLMs replace human evaluators? An empirical study of LLM-as-a-Judge in software engineering,” Feb. 2025, arXiv:2502.06193. [Online]. Available: http://arxiv.org/abs/2502.06193

    2025 arXiv

  15. [23]

    Reference-guided verdict: LLMs-as- Judges in automatic evaluation of free-form text,

    S. Badshah and H. Sajjad, “Reference-guided verdict: LLMs-as- Judges in automatic evaluation of free-form text,” Aug. 2024, arXiv:2408.09235. [Online]. Available: http://arxiv.org/abs/2408.09235

    2024

  16. [24]

    Mermaid

    “Mermaid.” [Online]. Available: https://mermaid.js.org

  17. [25]

    ReAct: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y . Cao, “ReAct: Synergizing reasoning and acting in language models,” inProc. 11th Int. Conf. Learn. Represent. (ICLR), Sep. 2023. [Online]. Available: https://openreview.net/forum?id=WE vluYUL-X

    2023

  18. [26]

    SWE-agent: Agent-computer interfaces enable automated software engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “SWE-agent: Agent-computer interfaces enable automated software engineering,”Advances Neural Inf. Process. Syst., vol. 37, pp. 50 528–50 652, Dec. 2024. [Online]. Available: https://proceedings....

    2024

  19. [27]

    INCHRON’s am2inc Ecore model

    “INCHRON’s am2inc Ecore model.” [Online]. Available: https: //github.com/inchron/am2inc/blob/main/EcoreModels/root.ecore

  20. [28]

    Eclipse APP4MC

    “Eclipse APP4MC.” [Online]. Available: https://eclipse.dev/app4mc/

  21. [29]

    chronSUITE from INCHRON

    “chronSUITE from INCHRON.” [Online]. Available: https://www. inchron.com/chronsuite/

  22. [30]

    INCHRON’s APP4MC / Amalthea Importer

    “INCHRON’s APP4MC / Amalthea Importer.” [Online]. Available: https://www.inchron.com/amalthea/

  23. [31]

    Factuality of large language models: A survey,

    Y . Wang, M. Wang, M. A. Manzoor, F. Liu, G. N. Georgiev, R. J. Das, and P. Nakov, “Factuality of large language models: A survey,” in Proc. Conf. Empirical Methods Natural Lang. Process., Y . Al-Onaizan, M. Bansal, and Y .-N. Chen, Eds. Miami, Florida, USA: Association for Co...

    2024

  24. [32]

    LangSmith Hub Prompt: Evaluation for RAG answer accuracy vs a reference

    “LangSmith Hub Prompt: Evaluation for RAG answer accuracy vs a reference.” [Online]. Available: https://smith.langchain.com/hub/ langchain-ai/rag-answer-vs-reference

  25. [33]

    Available: https://docs.ragas.io/en/v0.2.15/

    “Ragas.” [Online]. Available: https://docs.ragas.io/en/v0.2.15/

  26. [2007]

    Available: https://ieeexplore.ieee.org/document/4142919

    [Online]. Available: https://ieeexplore.ieee.org/document/4142919

  27. [2023]

    Available: https://openreview.net/forum?id=1i6ZCvflQJ

    [Online]. Available: https://openreview.net/forum?id=1i6ZCvflQJ

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.