Pith. sign in

REVIEW 2 major objections 5 minor 73 references

Enterprise LLM agents become auditable when product rules leave prompts and live in a code-owned harness of manifests, claims, contracts, and validators.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 13:25 UTC pith:QMQ7Y7TX

load-bearing objection Solid systems paper with real measurements: code-owned contracts hold under model swap and beat prompt-only and bolt-on guardrails on a pinned public slice—useful, not oversold. the 2 major comments →

arxiv 2607.08028 v1 pith:QMQ7Y7TX submitted 2026-07-09 cs.AI cs.CLcs.SE

From Prompts to Contracts: Harness Engineering for Auditable Enterprise LLM Agents

Joongho Ahn , Moonsoo Kim This is my paper
classification cs.AI cs.CLcs.SE
keywords harness engineeringenterprise LLM agentssource-backed claimsanswer contractscomposition boundaryaudit tracesenforcement ablationoutput hygiene
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Enterprise LLM apps often start as prompt-and-retrieval demos, then stall when productization demands source boundaries, entity routing, answer contracts, and reproducible traces. This paper argues that those guarantees do not come from better prompting: they require a harness that moves deterministic product behavior into code, manifests, schemas, and validators around a replaceable model composition boundary, while only source-backed claims may authorize runtime facts. Instantiated on a public slice of five Korean corporate groups and twenty-five listed companies, the harness kept its contracts on fixed scenarios, flagged deliberate breaks under fault injection, and held its code-owned checks across three hosted models on 270 composition-boundary runs. An enforcement ablation is the load-bearing result: with the model fixed, prompt instructions alone let recommendation language and internal-trace leaks reach the reader; a bolt-on guardrail blocked them but over-refused and cut utility; only code-owned validation-plus-fallback kept both safety and full utility. A sympathetic reader cares because this turns an exploratory prototype into an application whose source, control, and validation artifacts can be versioned, tested, and audited.

Core claim

The paper's central claim is that code-owned harness enforcement is load-bearing for enterprise answer contracts and is not reproducible by prompt instruction alone. Across fixed scenarios and fault injection the harness preserves source-grounding, entity-routing, trace, output-hygiene, and recommendation-language contracts; under model substitution the code-owned checks passed on all 270 composition-boundary runs with failures confined to the model-composed side; and in an enforcement-layer ablation holding the model fixed, prompt-only admitted every adversarial recommendation and leak violation while the harness blocked all of them and preserved 120/120 utility where a bolt-on external gua

What carries the argument

The harness: a code-owned control layer (source gates, entity routing, claim eligibility, answer contracts, trace generation, and validation) wrapped around a replaceable composition boundary, with source-backed claims—not the model or the wiki—as the authority for what may be asserted at runtime, and deterministic fallback when live composition fails the contract.

Load-bearing premise

That a fixed set of hand-authored scenarios, expected-claim pins, and a small suite of adversarial bait prompts is a good enough stand-in for the product failures that matter in real enterprise deployment, while correctness of the promoted claims themselves is treated as out of scope.

What would settle it

Rerun the enforcement-layer ablation with the same model and paired scenarios: if prompt-only (or another non-code-owned layer) blocks every recommendation-language and internal-trace-leakage violation on the adversarial set while still answering all benign scenarios at full utility—matching the harness on both safety and 120/120 utility—then the claim that code-owned enforcement is load-bearing fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Productization of prompt-dominant LLM demos should relocate source policy, routing, answer structure, and leakage rules into versioned code and schemas rather than longer prompts.
  • Swapping the composition model need not move reader-facing safety or audit contracts if those checks live in the harness rather than in model behavior.
  • Bolt-on refuse/redact guardrails can stop the same violations but may sacrifice utility; a validation-plus-deterministic-fallback path can keep both.
  • Each produced answer becomes an auditable artifact with routing, claim selection, source states, and validation results recorded outside the visible text.
  • The same manifest-and-claim pattern can register non-document operational signals once they are treated as governed sources rather than unbounded context.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Teams that treat RAG faithfulness scores as the main reliability metric may still ship unauthorized recommendations or leaked internal fields unless a per-answer contract gate is enforced in code.
  • Orchestration frameworks that compose agents but leave claim admission and output contracts to the application will inherit the same prompt-vs-harness gap shown in the ablation.
  • A practical next measurement would stress the promotion gates themselves—incorrect or incomplete claim admission—since the paper intentionally scopes those out of the reported pass rates.
  • In regulated briefing domains, the adoption ladder implies that moving from retrieval-supported assistants to claim-governed agents is the step where audit and compliance become feasible rather than aspirational.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes harness engineering: relocating enterprise LLM-agent behavior from prompt-dominant prototypes into code-owned contracts (source manifests, source-backed claims, entity routing, answer structure, traces, and validators) around a replaceable composition boundary. Instantiated on a public slice of five Korean corporate groups (25 listed companies, 113 runtime claims), it evaluates three RQs: (1) fixed-scenario contract preservation plus one-to-one fault-injection negative control; (2) model substitution at the composition boundary (3 hosted models × 30 scenarios × 3 repeats = 270 runs), separating code-owned from model-composed failures; (3) an enforcement-layer ablation (harness vs prompt-only vs bolt-on external guardrail) holding the model fixed, showing prompt-only admits recommendation-language and internal-trace-leakage violations while the harness blocks them and preserves full utility (120/120) versus the guardrail’s 88/120. Claims are scoped to contract preservation, not investment-answer quality.

Significance. If the results hold under the paper’s stated scope, this is a useful systems contribution for enterprise LLM productization: it supplies a concrete architecture (source-to-claim pipeline, replaceable composition boundary, deterministic fallback), a measurement design that separates harness guarantees from model phrasing, and evidence that code-owned enforcement can preserve both safety and utility where prompt instructions alone and a bolt-on guardrail do not. Strengths that should count in the assessment include the public repository and Zenodo-pinned baseline, committed evaluation artifacts, fault-injection negative control, Wilson intervals and χ² on model pass rates, McNemar tests on the paired ablation, and explicit separation of code-owned vs model-composed checks. The work is engineering evidence, not a domain-value claim about investment briefings; within that frame it is a reusable pattern with inspectable artifacts.

major comments (2)
  1. [Section 5.5 / Appendix A5] Section 5.5 and Appendix A5: The RQ3 load-bearing claim rests on ten author-written adversarial prompts (two templates × five groups) that explicitly solicit buy/sell/target-price language or internal claim IDs, JSON traces, and raw extractions—the same patterns the language and leakage validators already check. Internally, the contrast (prompt-only 30/30 violations admitted vs harness 0/0, utility 120/120 vs external-guardrail 88/120) is well measured and statistically reported. Externally, the baits are co-designed with the contract suite, so the ablation shows the gate blocks the violations it was written to detect rather than that code-owned enforcement is necessary against broader product-level failures (subtle recommendation framing, mixed-entity provenance, partial leakage, stale sources). Either add at least one further adversarial family not isomorphic to the existing validators
  2. [Section 5.2 / Table 6] Section 5.2, Table 6, and Section 7: “Source grounding” is operationalized as resolution of author-pinned expected claim references (109/109) and related trace fields, while correctness of the 113 promoted claims is explicitly out of scope. That is a coherent engineering check, but the contract-area label and abstract phrasing (“source-grounding … contracts”) can be read as factual provenance correctness. Rename or gloss the metric as expected-claim reference integrity / contract-envelope preservation, and state in the RQ1 discussion that pass rates do not certify promotion quality or mixed-entity source errors. Without that clarification, the strongest reader-facing claim overreaches the measured quantity.
minor comments (5)
  1. [Figure 1 / Figure 5] Figure 1 and Figure 5 are clear architecturally; ensure the camera-ready versions keep the ablation-only branches of Figure 5 visually distinct from the production path (caption already helps).
  2. [Table 7] Table 7 final pass rates (e.g., Claude 62/90) sit below the 270/270 code-owned pass; the text explains this well, but a one-line callout in the table caption that final failure is model-composed only would reduce misreading.
  3. [Section 2] Related Work is thorough; a short explicit contrast table (schema/rail tools vs source-to-claim authority layer vs this paper’s product-level measurement) would help readers place the contribution faster.
  4. [Table 10 / Section 6.4] Table 10’s adoption ladder is labeled a planning heuristic; keep that disclaimer adjacent to any abstract/conclusion mention so it is not taken as an empirical maturity model.
  5. [References] Minor consistency: arXiv id and some reference years in the provided text look future-dated relative to conventional timelines; verify all access dates, DOIs, and citation years before camera-ready.

Circularity Check

1 steps flagged

Deterministic fixed-scenario passes are largely by construction; ablation and model-substitution results are empirical, not circular derivations.

specific steps
  1. self definitional [§3.5 Reference Implementation; §5.2 Scenario Contract Outcomes / Table 6]
    "The baseline reported in the paper uses a deterministic, schema-checked composer for fixed validation scenarios and runtime-interface tests. The composer is a rule-based template engine that fills validated answer sections from selected source-backed claims using no generative model call. This baseline fixes the system contract... Within the bounded public-data slice, all configured source, trace, answer, and output-hygiene contracts passed. ... Total 30/30 109/109 30/30 30/30 60/60 0"

    For the deterministic fixed-scenario path, the system under test is a rule-based template engine written to emit the contracted sections from pinned expected claims. Reporting that those scenarios pass source-claim, trace, answer, and hygiene checks is therefore largely equivalent to verifying that the code implements the contracts it was written to implement—not an independent prediction. Fault injection and the live-LLM/ablation arms are not subject to this reduction.

full rationale

This is a systems-engineering evaluation paper, not a first-principles derivation. It does not fit parameters and rename them as predictions, import uniqueness theorems from the authors, or smuggle an ansatz via self-citation. The only mild self-consistency is that RQ1's all-pass rates on the fixed scenarios use a rule-based deterministic composer built to satisfy the same contracts it is scored against—so those positive passes are largely by design. The paper is transparent about that baseline choice and supplies independent empirical content: fault injection (validators fire on deliberately broken dimensions), model substitution (code-owned checks held 270/270 while model-composed checks varied), and the enforcement-layer ablation (prompt-only admitted 30 adversarial rec/leak violations that the harness blocked; external guardrail over-refused). The skeptic concern that adversarial baits match the validators is a construct-validity / external-validity issue about whether the scenario suite is a sufficient proxy for enterprise failure modes—not a reduction of a claimed prediction to its fitted inputs. Claim correctness and investment usefulness are explicitly out of scope. Score 2 for the minor by-construction element in the deterministic baseline; central load-bearing and substitution claims remain independently measured.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

This is an engineering systems paper, not a fitted physical theory. Load-bearing content is mostly domain design choices and evaluation protocol assumptions rather than free parameters fit to nature. The central claims rest on: (1) author-defined contracts and scenarios as the reliability unit; (2) a bounded public Korean corporate-group slice and promotion gates; (3) a deterministic fallback as the utility-preserving recovery path; (4) temperature and latency engineering knobs for the live checks. No new physical entities; invented items are software constructs.

free parameters (4)
  • live-LLM temperature
    Set to 0.2 for the 270-run composition-boundary check; affects model-composed variance though not the code-owned pass counts.
  • latency budget (1500 ms)
    Engineering threshold used as a contract dimension in fault injection and latency optimization; chosen, not derived.
  • claims-per-answer bound and expected-claim pins
    Runtime claim selection is bounded and fixed scenarios pin expected claim sets; these design choices define what ‘pass’ means for RQ1.
  • 25 priority claims reporting layer / 113-claim runtime layer
    Promotion and reporting selection reflect author judgment of validation needs and affiliate-mix coverage, not an external gold standard.
axioms (5)
  • domain assumption Enterprise reliability for this product class is adequately measured by per-answer contracts (source grounding, entity routing, trace completeness, output hygiene, recommendation-language absence) rather than by answer quality or investment correctness.
    Stated throughout Sections 1, 5, and 7; evaluation design follows this unit of analysis.
  • domain assumption Source manifests and promoted source-backed claims—not retrieved raw text or the compiled wiki—are the runtime authority for factual statements.
    Core of the source-to-claim pipeline (Sections 3.2–3.3, 4); wiki is explicitly non-authoritative.
  • ad hoc to paper A deterministic schema-checked composer is an acceptable utility-preserving fallback when live model output fails the output contract.
    Used as production recovery path and as the mechanism that keeps harness utility at 120/120 in the ablation (Sections 3.5, 5.5).
  • ad hoc to paper The 30 fixed scenarios plus 10 adversarial bait prompts sufficiently stress the contracts under test for the reference slice.
    Validation protocol and Appendix A1–A5; no external customer-log or expert-labeled failure corpus.
  • standard math Standard software-engineering notions of test harnesses, schemas, and validation gates apply to LLM agent control layers.
    ISO/IEC/IEEE harness vocabulary and SE-for-AI / MLOps citations in Related Work; used as background, not proved.
invented entities (2)
  • Enterprise LLM agent harness (code-owned control layer with replaceable composition boundary) independent evidence
    purpose: Relocate source eligibility, routing, claim admission, answer contracts, traces, and validation from prompts into versioned artifacts.
    Central architectural construct; related to prior harness/guardrail literature but specified here with source-to-claim authority and measured enforcement.
  • Source-backed runtime claim (promoted atomic statement with provenance, entity scope, and use policy) no independent evidence
    purpose: Make factual admission a code-owned gate independent of model phrasing and unbounded retrieval.
    Key intermediate representation of the pipeline; independent of any single model run but author-promoted.

pith-pipeline@v1.1.0-grok45 · 30032 in / 3728 out tokens · 41052 ms · 2026-07-10T13:25:57.477033+00:00 · methodology

0 comments
read the original abstract

Enterprise large language model (LLM) applications often begin as prototypes whose behavior is carried by prompts and retrieval context. Productization adds requirements for source boundaries, entity routing, answer contracts, and reproducible traces. We present a harness-engineering approach that reconstructs this pattern into a traceable, auditable LLM-agent architecture: deterministic behavior moves into code, manifests, schemas, and validation artifacts around a replaceable composition boundary, while source-backed claims remain the authority for runtime answers. We instantiate it on a public-data slice of five Korean corporate groups (25 listed companies) and evaluate three research questions. (1) The harness preserves its source-grounding, entity-routing, trace, output-hygiene, and recommendation-language contracts across the fixed validation scenarios; a fault-injection control confirms the validators flag deliberately broken contracts. (2) The checks the harness enforces held under model substitution: across three hosted models, they passed on all 270 composition-boundary runs; failures were confined to the model-composed side and were caught and recorded. (3) The code-owned guarantees are load-bearing, not reproducible by prompting alone: holding the model fixed and varying only the enforcement layer, prompt instructions alone let recommendation-language and internal-trace-leakage violations reach the reader, which the harness blocks entirely. A bolt-on external guardrail prevents such violations too but over-refuses, dropping utility to 88/120 where the harness preserves full utility (120/120); in this ablation, only code-owned enforcement preserves both safety and utility. The result is a reusable engineering pattern for turning exploratory prototypes into auditable applications with versioned source, control, and validation artifacts.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

73 extracted references · 73 canonical work pages · 27 internal anchors

  1. [1]

    [Vibe coding], 2025

    Andrej Karpathy. [Vibe coding], 2025. URL https : / / x . com / karpathy / status / 1886192184808149383. X Post, 2025-02-02; accessed 2026-07-09

    work page 2025

  2. [2]

    Vibe coding as a reconfiguration of intent mediation in software development: Definition, implications, and research agenda

    Christian Meske, Tobias Hermanns, Esther von der Weiden, Kai-Uwe Loser, and Thorsten Berger. Vibe coding as a reconfiguration of intent mediation in software development: Definition, implications, and research agenda. IEEE Access, 13:213242–213259, 2025. doi: 10.1109/ACCESS.2025.3645466

    work page doi:10.1109/access.2025.3645466 2025

  3. [3]

    Survey of hallucination in natural language generation

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), 2023. doi: 10.1145/3571730

    work page doi:10.1145/3571730 2023

  4. [4]

    Software engineering for AI-based systems: A survey

    Silverio Martínez-Fernández, Justus Bogner, Xavier Franch, Marc Oriol, Julien Siebert, Adam Trendowicz, Anna Maria Vollmer, and Stefan Wagner. Software engineering for AI-based systems: A survey. ACM Transactions on Software Engineering and Methodology, 31(2):1–59, 2022. doi: 10.1145/3487043

    work page doi:10.1145/3487043 2022

  5. [5]

    Machine learning operations (MLOps): Overview, definition, and architecture

    Dominik Kreuzberger, Niklas Kühl, and Sebastian Hirschl. Machine learning operations (MLOps): Overview, definition, and architecture. IEEE Access, 11:31866–31879, 2023. doi: 10.1109/ACCESS.2023.3262138

    work page doi:10.1109/access.2023.3262138 2023

  6. [6]

    AI Leadership Program, 2026

    CEO AI. AI Leadership Program, 2026. URL https://ceoai.kr/ai-leadership. Accessed 2026-07-09

    work page 2026

  7. [7]

    Replit, 2026

    Replit, Inc. Replit, 2026. URLhttps://replit.com. Accessed 2026-07-09

    work page 2026

  8. [8]

    NeMo guardrails: A toolkit for controllable and safe LLM applications with programmable rails

    Traian Rebedea, Razvan Dinu, Makesh Narsimhan Sreedhar, Christopher Parisien, and Jonathan Cohen. NeMo guardrails: A toolkit for controllable and safe LLM applications with programmable rails. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 431–445, Singapore, 2023. Association for Co...

    work page doi:10.18653/v1/2023.emnlp-demo.40 2023

  9. [9]

    Building Guardrails for Large Language Models

    Yi Dong, Ronghui Mu, Gaojie Jin, Yi Qi, Jinwei Hu, Xingyu Zhao, Jie Meng, Wenjie Ruan, and Xiaowei Huang. Position: Building guardrails for large language models requires systematic design. In Proceedings of the 41st International Conference on Machine Learning, pages 11375–11394, 2024. doi: 10.48550/arXiv.2402.01822

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2402.01822 2024

  10. [10]

    Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering

    Chenyu Zhou, Huacan Chai, Wenteng Chen, Zihan Guo, Rong Shan, Yuanyi Song, Tianyi Xu, Yingxuan Yang, Aofan Yu, Weiming Zhang, Congming Zheng, Jiachen Zhu, Zeyu Zheng, Zhuosheng Zhang, Xingyu Lou, Changwang Zhang, Zhihui Fu, Jun Wang, Weiwen Liu, Jianghao Lin, and Weinan Zhang. Externalization in LLM agents: A unified review of memory, skills, protocols an...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2604.08224 2026

  11. [11]

    DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. DSPy: Compiling declarative language model calls into self-improving pipelines. In The Twelfth International Conference on Learning Representati...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2310.03714 2024

  12. [12]

    Prompting is programming: A query language for large language models

    Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. Prompting is programming: A query language for large language models. Proceedings of the ACM on Programming Languages, 7 (PLDI):1946–1969, 2023. doi: 10.1145/3591300

    work page doi:10.1145/3591300 1946

  13. [13]

    Systems and software engineering – vocabulary

    ISO/IEC/IEEE. Systems and software engineering – vocabulary. Technical Report ISO/IEC/IEEE 24765:2017, International Organization for Standardization, International Electrotechnical Commission, and Institute of Electrical and Electronics Engineers, 2017. URLhttps://www.iso.org/standard/71952.html. Accessed 2026-07-09

    work page 2017

  14. [14]

    Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, 27 and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459–9474, 2020. doi: 10.48...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2005.11401 2020

  15. [15]

    Retrieval-Augmented Generation for Large Language Models: A Survey

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997, 2023. doi: 10.48550/arXiv.2312.10997

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2312.10997 2023

  16. [16]

    Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learn- ing to retrieve, generate, and critique through self-reflection. In International Conference on Learning Representations, 2024. doi: 10.48550/arXiv.2310.11511

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2310.11511 2024

  17. [17]

    FActScore: Fine-grained atomic evalua- tion of factual precision in long form text generation

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evalua- tion of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12076–12100, Singapore, 2023. A...

    work page doi:10.18653/v1/2023.emnlp-main.741 2023

  18. [18]

    Enabling large language models to generate text with citations

    Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6465–6488, Singapore, 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.398

    work page doi:10.18653/v1/2023.emnlp-main.398 2023

  19. [19]

    Liu, Tianyi Zhang, and Percy Liang

    Nelson F. Liu, Tianyi Zhang, and Percy Liang. Evaluating verifiability in generative search engines. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7001–7025, Singapore, 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.467

    work page doi:10.18653/v1/2023.findings-emnlp.467 2023

  20. [20]

    PersonaRAG: Enhancing retrieval-augmented generation systems with user-centric agents

    Saber Zerhoudi and Michael Granitzer. PersonaRAG: Enhancing retrieval-augmented generation systems with user-centric agents. arXiv preprint arXiv:2407.09394, 2024. doi: 10.48550/arXiv.2407.09394

    work page doi:10.48550/arxiv.2407.09394 2024

  21. [21]

    Measuring attribution in natural language generation models

    Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. Measuring attribution in natural language generation models. Computational Linguistics, 49(4):777–840, 2023. doi: 10.1162/coli_a_00486

    work page doi:10.1162/coli_a_00486 2023

  22. [22]

    Jennings

    Michael Wooldridge and Nicholas R. Jennings. Intelligent agents: Theory and practice. The Knowledge Engineering Review, 10(2):115–152, 1995. doi: 10.1017/S0269888900008122

    work page doi:10.1017/s0269888900008122 1995

  23. [23]

    Jennings, Katia Sycara, and Michael Wooldridge

    Nicholas R. Jennings, Katia Sycara, and Michael Wooldridge. A roadmap of agent research and development. Autonomous Agents and Multi-Agent Systems, 1(1):7–38, 1998. doi: 10.1023/A:1010090405266

    work page doi:10.1023/a:1010090405266 1998

  24. [24]

    ReAct: Synergizing Reasoning and Acting in Language Models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023. doi: 10.48550/arXiv.2210.03629

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2210.03629 2023

  25. [25]

    Toolformer: Language Models Can Teach Themselves to Use Tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Advances in Neural Information Processing Systems, volume 36, 2023. doi: 10.48550/arXiv.2302.04761

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2302.04761 2023

  26. [26]

    A survey on large language model based autonomous agents

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6): 186345, 2024. doi: 10.1007/s11704-024-40231-1

    work page doi:10.1007/s11704-024-40231-1 2024

  27. [27]

    Deepak Bhaskar Acharya, Karthigeyan Kuppan, and B. Divya. Agentic AI: Autonomous intelligence for complex goals—a comprehensive survey. IEEE Access, 13:18912–18936, 2025. doi: 10.1109/ACCESS.2025.3532853

    work page doi:10.1109/access.2025.3532853 2025

  28. [28]

    ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie 28 Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. In International Conference on Learning Represe...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2307.16789 2024

  29. [29]

    Autonomous AI agents for multi-platform social media marketing: A simultaneous deployment study

    Joongho Ahn and Moonsoo Kim. Autonomous AI agents for multi-platform social media marketing: A simultaneous deployment study. Electronics, 14(21):4161, 2025. doi: 10.3390/ electronics14214161

    work page 2025

  30. [30]

    Eliza: A Web3 friendly AI Agent Operating System

    Shaw Walters, Sam Gao, Shakker Nerd, Feng Da, Warren Williams, Ting-Chien Meng, Amie Chow, Hunter Han, Frank He, Allen Zhang, Ming Wu, Timothy Shen, Maxwell Hu, and Jerry Yan. Eliza: A Web3 friendly AI agent operating system. arXiv preprint arXiv:2501.06781,

    work page internal anchor Pith review Pith/arXiv arXiv

  31. [31]

    doi: 10.48550/arXiv.2501.06781

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2501.06781

  32. [32]

    Why Do Multi-Agent LLM Systems Fail?

    Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Why do multi-agent LLM systems fail? In Advances in Neural Information Processing Systems 38 (NeurIPS 2025), Datasets and Benchmarks Track, 2025. doi...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2503.13657 2025

  33. [33]

    White, Doug Burger, and Chi Wang

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W. White, Doug Burger, and Chi Wang. AutoGen: Enabling next-gen LLM applications via multi-agent conversations. In First Conference on Language Modeling, 2024. doi: 10.48550/arXiv.2308. 08155

    work page doi:10.48550/arxiv.2308 2024

  34. [34]

    Agents, 2026

    LangChain. Agents, 2026. URL https://docs.langchain.com/oss/python/langchain/ agents. Accessed 2026-07-09

    work page 2026

  35. [35]

    LangGraph, 2026

    LangChain. LangGraph, 2026. URL https://github.com/langchain-ai/langgraph . Accessed 2026-07-09

    work page 2026

  36. [36]

    CrewAI, 2026

    CrewAI Inc. CrewAI, 2026. URL https://github.com/crewAIInc/crewAI . Accessed 2026-07-09

    work page 2026

  37. [37]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2005.14165 1901

  38. [38]

    Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35, 2023. doi: 10.1145/3560815

    work page doi:10.1145/3560815 2023

  39. [39]

    A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications

    Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927, 2024. doi: 10.48550/arXiv.2402.07927

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2402.07927 2024

  40. [40]

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837, 2022. doi: 10.48550/arXiv.2201.11903

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2201.11903 2022

  41. [41]

    Symeonidis

    Nikolaos Malamas, Emmanouil Tsardoulias, Konstantinos Panayiotou, and Andreas L. Symeonidis. Toward efficient vibe coding: An LLM-based agent for low-code software development. Journal of Computer Languages, 85:101367, 2025. doi: 10.1016/j.cola.2025. 101367

    work page doi:10.1016/j.cola.2025 2025

  42. [42]

    Instructor, 2026

    567 Labs. Instructor, 2026. URL https://github.com/567-labs/instructor. Accessed 2026-07-09. 29

    work page 2026

  43. [43]

    Guidance, 2026

    Guidance AI. Guidance, 2026. URL https : / / github . com / guidance-ai / guidance. Accessed 2026-07-09

    work page 2026

  44. [44]

    Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP

    Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive NLP. arXiv preprint arXiv:2212.14024, 2022. doi: 10.48550/ arXiv.2212.14024

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  45. [45]

    DSPy, 2026

    Stanford NLP. DSPy, 2026. URL https://github.com/stanfordnlp/dspy . Accessed 2026-07-09

    work page 2026

  46. [46]

    Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama Guard: LLM-based input-output safeguard for human-AI conversations. arXiv preprint arXiv:2312.06674, 2023. doi: 10.48550/arXiv.2312.06674

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2312.06674 2023

  47. [47]

    Constitutional AI: Harmlessness from AI Feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, K...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2212.08073 2022

  48. [48]

    Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses

    Jiahang Lin, Shichun Liu, Chengjun Pan, Lizhi Lin, Shihan Dou, Zhiheng Xi, Xuanjing Huang, Hang Yan, Zhenhua Han, Tao Gui, and Yu-Gang Jiang. Agentic harness engineer- ing: Observability-driven automatic evolution of coding-agent harnesses. arXiv preprint arXiv:2604.25850, 2026. doi: 10.48550/arXiv.2604.25850

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2604.25850 2026

  49. [49]

    Meta-Harness: End-to-End Optimization of Model Harnesses

    Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-Harness: End-to-end optimization of model harnesses. arXiv preprint arXiv:2603.28052,

    work page internal anchor Pith review Pith/arXiv arXiv

  50. [50]

    doi: 10.48550/arXiv.2603.28052

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2603.28052

  51. [51]

    AI Harness Engineering: A Runtime Substrate for Foundation-Model Software Agents

    Hailin Zhong and Shengxin Zhu. AI harness engineering: A runtime substrate for foundation- model software agents. arXiv preprint arXiv:2605.13357, 2026. doi: 10.48550/arXiv.2605. 13357

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2605 2026

  52. [52]

    Code as Agent Harness

    Xuying Ning, Katherine Tieu, Dongqi Fu, Tianxin Wei, Zihao Li, Yuanchen Bei, Jiaru Zou, Mengting Ai, Zhining Liu, Ting-Wei Li, Lingjie Chen, Yanjun Zhao, Ke Yang, Bingxuan Li, Cheng Qian, Gaotang Li, Xiao Lin, Zhichen Zeng, Ruizhong Qiu, Sirui Chen, Yifan Sun, Xiyuan Yang, Ruida Wang, Rui Pan, Chenyuan Yang, Dylan Zhang, Liri Fang, Zikun Cui, Yang Cao, Pa...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2605.18747 2026

  53. [53]

    Harness engineering for language agents: The harness layer as control, agency, and runtime

    Chaoyue He, Xin Zhou, Di Wang, Hong Xu, Wei Liu, and Chunyan Miao. Harness engineering for language agents: The harness layer as control, agency, and runtime. Preprints.org preprint,

    work page

  54. [54]

    Version 2; accessed 2026-07-09

    doi: 10.20944/preprints202603.1756.v2. Version 2; accessed 2026-07-09

    work page doi:10.20944/preprints202603.1756.v2 2026

  55. [55]

    LLM Wiki, 2026

    Andrej Karpathy. LLM Wiki, 2026. URL https : / / gist . github . com / karpathy / 442a6bf555914893e9891c11519de94f. Accessed 2026-07-09

    work page 2026

  56. [56]

    Retrieval as Reasoning: Self-Evolving Agent-Native Retrieval via LLM-Wiki

    Haoliang Ming, Feifei Li, Xiaoqing Wu, and Wenhui Que. Retrieval as reasoning: Self- evolving agent-native retrieval via LLM-wiki. arXiv preprint arXiv:2605.25480, 2026. doi: 10.48550/arXiv.2605.25480. 30

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2605.25480 2026

  57. [57]

    Juan M. Huerta. WiCER: Wiki-memory compile, evaluate, refine: Iterative knowledge compilation for LLM wiki systems. arXiv preprint arXiv:2605.07068, 2026. doi: 10.48550/ arXiv.2605.07068

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  58. [58]

    Assuring the machine learning lifecycle: Desiderata, methods, and challenges

    Rob Ashmore, Radu Calinescu, and Colin Paterson. Assuring the machine learning lifecycle: Desiderata, methods, and challenges. ACM Computing Surveys, 54(5):1–39, 2021. doi: 10.1145/3453444

    work page doi:10.1145/3453444 2021

  59. [59]

    Auditing large language models: A three-layered approach

    Jakob Mökander, Jonas Schuett, Hannah Rose Kirk, and Luciano Floridi. Auditing large language models: A three-layered approach. AI and Ethics, 4(4):1085–1115, 2024. doi: 10.1007/s43681-023-00289-2

    work page doi:10.1007/s43681-023-00289-2 2024

  60. [60]

    RAGAs: Automated evaluation of retrieval augmented generation

    Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. RAGAs: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150–158, St. Julians, Malta, 2024. Association for Computational Linguistics. doi: 10...

    work page doi:10.18653/v1/2024.eacl-demo.16 2024

  61. [61]

    ARES: An automated evaluation framework for retrieval-augmented generation systems

    Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. ARES: An automated evaluation framework for retrieval-augmented generation systems. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 338–354, Mexico City, Mexico, 2024. Association for...

    work page doi:10.18653/v1/2024.naacl-long.20 2024

  62. [62]

    Holistic Evaluation of Language Models

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2211.09110 2023

  63. [63]

    Evaluation and benchmarking of LLM agents: A survey

    Mahmoud Mohammadi, Yipeng Li, Jane Lo, and Wendy Yip. Evaluation and benchmarking of LLM agents: A survey. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’25), pages 6129–6139, 2025. doi: 10.1145/3711896. 3736570

    work page doi:10.1145/3711896 2025

  64. [64]

    promptfoo, 2026

    promptfoo. promptfoo, 2026. URL https://github.com/promptfoo/promptfoo. Accessed 2026-07-09

    work page 2026

  65. [65]

    enterprise-llm-agent-harness, 2026

    Joongho Ahn. enterprise-llm-agent-harness, 2026. URL https://github.com/hammerbaki/ enterprise-llm-agent-harness. Release tag public-baseline-v0.5.16.4, commit e8e60fb; accessed 2026-07-09

    work page 2026

  66. [66]

    2026 Korean corporate-group designation results, 2026

    Korea Fair Trade Commission. 2026 Korean corporate-group designation results, 2026. URL https://www.ftc.go.kr/www/selectBbsNttView.do?bordCd=3&key=12&nttSn=47410& searchCtgry=01%2C02. Press release with attached PDF; accessed 2026-07-09

    work page 2026

  67. [67]

    OpenDART: Introduction to open API, 2026

    Financial Supervisory Service. OpenDART: Introduction to open API, 2026. URL https: //opendart.fss.or.kr/intro/main.do. Accessed 2026-07-09

    work page 2026

  68. [68]

    KRX OPEN API: Service usage guide, 2026

    Korea Exchange. KRX OPEN API: Service usage guide, 2026. URL https://openapi. krx.co.kr/contents/OPP/INFO/OPPINFO003.jsp. Accessed 2026-07-09

    work page 2026

  69. [69]

    Search API: News search, 2026

    NAVER Developers. Search API: News search, 2026. URL https://developers.naver. com/docs/serviceapi/search/news/news.md. Accessed 2026-07-09. 31

    work page 2026

  70. [70]

    Edwin B. Wilson. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158):209–212, 1927. doi: 10.1080/01621459.1927. 10502953

    work page doi:10.1080/01621459.1927 1927

  71. [71]

    Karl Pearson. On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 50(302):157–175, 1900. doi: 10.1080/14786440009463897

    work page doi:10.1080/14786440009463897 1900

  72. [72]

    Note on the sampling error of the difference between correlated proportions or percentages

    Quinn McNemar. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157, 1947. doi: 10.1007/BF02295996

    work page doi:10.1007/bf02295996 1947

  73. [73]

    Paulk, Bill Curtis, Mary Beth Chrissis, and Charles V

    Mark C. Paulk, Bill Curtis, Mary Beth Chrissis, and Charles V. Weber. Capability maturity model, version 1.1. IEEE Software, 10(4):18–27, 1993. doi: 10.1109/52.219617. 32

    work page internal anchor Pith review doi:10.1109/52.219617 1993