REVIEW 6 cited by
A Declarative System for Optimizing AI Workloads
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
A long-standing goal of data management systems has been to build systems which can compute quantitative insights over large corpora of unstructured data in a cost-effective manner. Until recently, it was difficult and expensive to extract facts from company documents, data from scientific papers, or metrics from image and video corpora. Today's models can accomplish these tasks with high accuracy. However, a programmer who wants to answer a substantive AI-powered query must orchestrate large numbers of models, prompts, and data operations. For even a single query, the programmer has to make a vast number of decisions such as the choice of model, the right inference method, the most cost-effective inference hardware, the ideal prompt design, and so on. The optimal set of decisions can change as the query changes and as the rapidly-evolving technical landscape shifts. In this paper we present Palimpzest, a system that enables anyone to process AI-powered analytical queries simply by defining them in a declarative language. The system uses its cost optimization framework to implement the query plan with the best trade-offs between runtime, financial cost, and output data quality. We describe the workload of AI-powered analytics tasks, the optimization methods that Palimpzest uses, and the prototype system itself. We evaluate Palimpzest on tasks in Legal Discovery, Real Estate Search, and Medical Schema Matching. We show that even our simple prototype offers a range of appealing plans, including one that is 3.3x faster and 2.9x cheaper than the baseline method, while also offering better data quality. With parallelism enabled, Palimpzest can produce plans with up to a 90.3x speedup at 9.1x lower cost relative to a single-threaded GPT-4 baseline, while obtaining an F1-score within 83.5% of the baseline. These require no additional work by the user.
Forward citations
Cited by 6 Pith papers
-
Cut Costs, Not Accuracy: LLM-Powered Data Processing with Guarantees
BARGAIN uses betting-based anytime-valid tests and adaptive, target-aware sampling to set model-cascade thresholds, delivering non-asymptotic quality guarantees and up to 86% greater cost savings than SUPG.
-
ASMR: Agentic Schema Generation for Ship Maintenance Report Writing
ASMR extracts concepts with an LLM, clusters them into candidate fields, then uses RL to select compact non-redundant schemas for each ship-report form type.
-
Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platforms
Murakkab uses declarative workflow specs and a profile-guided MILP optimizer to reduce GPU, energy, and cost for agentic workflow serving while meeting percentile-defined SLOs.
-
Research Challenges in Relational Database Management Systems for LLM Queries
Open-source SQL-plus-LLM database integrations fail on most representative LLM queries and run orders of magnitude slower than enterprise APIs, due to missing structured outputs, poor batching, and plan-unaware LLM calls.
-
Metadata Management for AI-Augmented Data Workflows
TableVault is a metadata governance framework that records ingestion events, operation status, execution parameters, and lineage for human-AI data workflows, demonstrated on a document classification case study.
-
TableVault: Managing Dynamic Data Collections for LLM-Augmented Workflows
TableVault describes a system design for managing versioned, reproducible dataframe collections in LLM-augmented workflows, but it ships no implementation or evaluation.
Discussion (0). Sign in to comment.