REVIEW 15 cited by
Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Language Model Programs, i.e. sophisticated pipelines of modular language model (LM) calls, are increasingly advancing NLP tasks, but they require crafting prompts that are jointly effective for all modules. We study prompt optimization for LM programs, i.e. how to update these prompts to maximize a downstream metric without access to module-level labels or gradients. To make this tractable, we factorize our problem into optimizing the free-form instructions and few-shot demonstrations of every module and introduce several strategies to craft task-grounded instructions and navigate credit assignment across modules. Our strategies include (i) program- and data-aware techniques for proposing effective instructions, (ii) a stochastic mini-batch evaluation function for learning a surrogate model of our objective, and (iii) a meta-optimization procedure in which we refine how LMs construct proposals over time. Using these insights we develop MIPRO, a novel algorithm for optimizing LM programs. MIPRO outperforms baseline optimizers on five of seven diverse multi-stage LM programs using a best-in-class open-source model (Llama-3-8B), by as high as 13% accuracy. We have released our new optimizers and benchmark in DSPy at http://dspy.ai
Forward citations
Cited by 15 Pith papers
-
Harness-R1: Learning to Edit Executable Runtime Harnesses from Agent Failure Trajectories
A dedicated harness-editor policy trained with RL on the realized outcomes of executable patches raises frozen-agent success by 9.3 points across WebShop, ALFWorld, and DBBench.
-
FlowBot: Inducing LLM Workflows with Bilevel Optimization and Textual Gradients
FlowBot automatically induces LLM workflows through bilevel optimization with textual gradients, achieving competitive performance against human-crafted baselines.
-
MemoHarness: Agent Harnesses That Learn from Experience
MemoHarness learns from past executions to adapt an LLM agent's control layer per test case, improving task success over fixed harnesses on shell, code, and reasoning benchmarks.
-
IUU+DB: Tracking Illegal, Unreported, and Unregulated Fishing, Seafood Fraud, and Labor Abuse through LLM-driven Information Extraction
IUU+DB is an LLM-driven pipeline that classifies documents, extracts structured incident data on IUU+ activities, and supports deduplication and trend analysis.
-
DetPO: In-Context Learning with Multi-Modal LLMs for Few-Shot Object Detection
Detection Prompt Optimization (DetPO) improves few-shot object detection with black-box MLLMs by iteratively refining text prompts from TP/FP/FN errors on few-shot examples, gaining up to 9.7 mAP over prior black-box methods.
-
MASPOB: Bandit-Based Prompt Optimization for Multi-Agent Systems with Graph Neural Networks
MASPOB combines a GNN surrogate, LinUCB-style uncertainty, and coordinate ascent to optimize prompts in fixed-topology multi-agent LLM systems, beating AFlow and MIPRO on average across six benchmarks.
-
Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions
Most MCP tool descriptions (97.1%) contain quality smells, and augmenting them improves agent success by a median of 5.85 percentage points at a 67.46% increase in execution steps.
-
FEM-Bench: A Structured Scientific Reasoning Benchmark for Evaluating Code-Generating LLMs
The paper introduces FEM-Bench, a 33-task computational mechanics benchmark, and shows that state-of-the-art LLMs complete at most 30/33 tasks with multiple attempts and fail entirely on geometric-stiffness-related tasks.
-
Improved Representation Steering for Language Models
RePS, a reference-free bidirectional preference optimization objective, improves representation steering and suppression for Gemma models, outperforming language-modeling objectives and approaching prompting performance.
-
Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0
In a two-phase Terminal-Bench evaluation, only regression-aware RELAI-VCL compounded optimization gains, reaching the highest pass rate at every stage.
-
Maestro: Joint Graph & Config Optimization for Reliable AI Agents
Maestro jointly searches agent graph topology and node configurations, reporting higher test scores than prompt-only optimizers on HotpotQA, IFBench, and two internal agent tasks.
-
Is It Time To Treat Prompts As Code? A Multi-Use Case Study For Prompt Optimization Using DSPy
A five-task case study shows DSPy prompt optimization can improve LLM accuracy on some tasks, notably contradiction detection (46.2% to 64.0%), but results vary and no code or data are released.
-
Evaluating Hybrid Retrieval Augmented Generation using Dynamic Test Sets: LiveRAG Challenge
A hybrid BM25-plus-dense RAG system finished 4th in faithfulness and 11th in correctness at LiveRAG 2025, and its analysis found vocabulary alignment between questions and documents to be the most important performanc...
-
Transforming Expert Knowledge into Scalable Ontology via Large Language Models
An LLM-based taxonomy alignment framework reaches 0.97 F1 using many-shot prompting and expert calibration, but the claimed superiority over the 0.68 human benchmark is based on a non-comparable baseline.
-
Toward Knowledge-Guided AI for Inverse Design in Manufacturing: A Perspective on Domain, Physics, and Human-AI Synergy
A perspective arguing that inverse design in manufacturing improves when expert-guided problem definition, physics-informed ML, and LLM interfaces are combined.
Discussion (0). Sign in to comment.