REVIEW 7 cited by
Self-Supervised Prompt Optimization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Well-designed prompts are crucial for enhancing Large language models' (LLMs) reasoning capabilities while aligning their outputs with task requirements across diverse domains. However, manually designed prompts require expertise and iterative experimentation. While existing prompt optimization methods aim to automate this process, they rely heavily on external references such as ground truth or by humans, limiting their applicability in real-world scenarios where such data is unavailable or costly to obtain. To address this, we propose Self-Supervised Prompt Optimization (SPO), a cost-efficient framework that discovers effective prompts for both closed and open-ended tasks without requiring external reference. Motivated by the observations that prompt quality manifests directly in LLM outputs and LLMs can effectively assess adherence to task requirements, we derive evaluation and optimization signals purely from output comparisons. Specifically, SPO selects superior prompts through pairwise output comparisons evaluated by an LLM evaluator, followed by an LLM optimizer that aligns outputs with task requirements. Extensive experiments demonstrate that SPO outperforms state-of-the-art prompt optimization methods, achieving comparable or superior results with significantly lower costs (e.g., 1.1% to 5.6% of existing methods) and fewer samples (e.g., three samples). The code is available at https://github.com/FoundationAgents/SPO.
Forward citations
Cited by 7 Pith papers
-
HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses
A constrained, feedback-augmented, component-wise harness evolution loop improves coding agent Pass@1 on SWE-bench Verified and transfers to held-out tasks and another base model.
-
Self-Supervised Skill Optimization
A comparative loop that extracts evidence for and against observed behaviors from judge preferences over unlabeled executions can optimize agent skill documents without any ground-truth labels or rewards.
-
Synthetic Interaction Data for Scalable Personalization in Large Language Models
PersonaGym simulates noisy multi-turn user–assistant interactions to build PersonaAtlas, and PPOpt learns to rewrite user prompts from interaction history, improving judged personalization on synthetic benchmarks.
-
ORPP: Self-Optimizing Role-playing Prompts to Enhance Language Model Capabilities
ORPP generates task-specific role-playing prompts through iterative reward-model-guided optimization on a small sample, then uses few-shot transfer to create prompts for new questions.
-
Concise Reasoning, Big Gains: Pruning Long Reasoning Trace with Difficulty-Aware Prompting
Fine-tuning small models on difficulty-adapted, shortened reasoning traces (LiteCoT) yields equal or better benchmark accuracy than training on much longer traces, with far fewer tokens.
-
Silence is Not Consensus: Disrupting Agreement Bias in Multi-Agent LLMs via Catfish Agent for Clinical Decision Making
A "catfish" agent that injects structured dissent into multi-agent LLM teams improves clinical question-answering accuracy by reducing premature consensus.
-
Tournament of Prompts: Evolving LLM Instructions Through Structured Debates and Elo Ratings
DEEVO evolves better LLM prompts by debating outputs and selecting survivors with Elo ratings, without requiring labeled data or a hand-written fitness function.
Discussion (0). Sign in to comment.