Pith. sign in

REVIEW 5 cited by

Specifications: The missing link to making the development of LLM systems an engineering discipline

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2412.05299 v2 pith:A3SVB452 submitted 2024-11-25 cs.SE cs.AIcs.CL

Ion Stoica , Matei Zaharia , Joseph Gonzalez , Ken Goldberg , Koushik Sen , Hao Zhang , Anastasios Angelopoulos , Shishir G. Patil
show 3 more authors
Lingjiao Chen Wei-Lin Chiang Jared Q. Davis
This is my paper · ORCID
classification cs.SEcs.AIcs.CL
keywords systemsmodularreliablespecificationsbuildingcomponentsdevelopmentfuture
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Despite the significant strides made by generative AI in just a few short years, its future progress is constrained by the challenge of building modular and robust systems. This capability has been a cornerstone of past technological revolutions, which relied on combining components to create increasingly sophisticated and reliable systems. Cars, airplanes, computers, and software consist of components-such as engines, wheels, CPUs, and libraries-that can be assembled, debugged, and replaced. A key tool for building such reliable and modular systems is specification: the precise description of the expected behavior, inputs, and outputs of each component. However, the generality of LLMs and the inherent ambiguity of natural language make defining specifications for LLM-based components (e.g., agents) both a challenging and urgent problem. In this paper, we discuss the progress the field has made so far-through advances like structured outputs, process supervision, and test-time compute-and outline several future directions for research to enable the development of modular and reliable LLM-based systems through improved specifications.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AssumptionMiner: Extracting, Tracing, and Revising Implicit Assumptions in LLM Code Generation

    cs.SE 2026-07 conditional novelty 7.0 of 10

    AssumptionMiner extracts implicit assumptions from LLM-generated code with a cross-model ensemble F1 of 0.816 on a new 180-task benchmark, and enables assumption-guided targeted regeneration.

  2. Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions

    cs.SE 2026-02 conditional novelty 6.0 of 10

    Most MCP tool descriptions (97.1%) contain quality smells, and augmenting them improves agent success by a median of 5.85 percentage points at a 67.46% increase in execution steps.

  3. Standardizing Intelligence: Aligning Generative AI for Regulatory and Operational Compliance

    cs.CY 2025-02 conditional novelty 5.0 of 10

    The paper proposes C3F, a two-axis framework rating GenAI compliance capability and standard criticality, and applies it to 15 models and 34 standards.

  4. Position: Scaling LLM Agents Requires Asymptotic Analysis with LLM Primitives

    cs.CL 2025-02 conditional novelty 4.0 of 10

    The paper proposes that asymptotic analysis with LLM primitives, treating one forward pass as the cost unit, is the right framework for scaling multi-agent LLM systems.

  5. Querying Databases with Function Calling

    cs.DB 2025-01 conditional novelty 4.0 of 10

    A new tool definition and synthetic benchmark show top LLMs can format database query calls via function calling, with the best models scoring around 74% exact-match accuracy.

Pith tools