REVIEW 7 cited by
AutoRAG-HP: Automatic Online Hyper-Parameter Tuning for Retrieval-Augmented Generation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
Recent advancements in Large Language Models have transformed ML/AI development, necessitating a reevaluation of AutoML principles for the Retrieval-Augmented Generation (RAG) systems. To address the challenges of hyper-parameter optimization and online adaptation in RAG, we propose the AutoRAG-HP framework, which formulates the hyper-parameter tuning as an online multi-armed bandit (MAB) problem and introduces a novel two-level Hierarchical MAB (Hier-MAB) method for efficient exploration of large search spaces. We conduct extensive experiments on tuning hyper-parameters, such as top-k retrieved documents, prompt compression ratio, and embedding methods, using the ALCE-ASQA and Natural Questions datasets. Our evaluation from jointly optimization all three hyper-parameters demonstrate that MAB-based online learning methods can achieve Recall@5 $\approx 0.8$ for scenarios with prominent gradients in search space, using only $\sim20\%$ of the LLM API calls required by the Grid Search approach. Additionally, the proposed Hier-MAB approach outperforms other baselines in more challenging optimization scenarios. The code will be made available at https://aka.ms/autorag.
Forward citations
Cited by 7 Pith papers
-
RAG-Stack: Co-Optimizing RAG Serving Performance and Quality
RAG-Stack jointly optimizes RAG algorithm choices and serving-system settings via sub-metric-aware multi-objective Bayesian optimization plus an analytical performance model, reporting Pareto frontiers covering 52.5% ...
-
Orchestration for Domain-specific Edge-Cloud Language Models
ECO-LLM jointly selects query processing, retrieval, and model components per query, cutting cost by 60% and latency up to 6x versus model routing in edge-cloud tests.
-
Demystifying and Enhancing the Efficiency of Large Language Model Based Search Agents
SearchAgent-X speeds up LLM-based search agents by prioritizing requests that benefit most from cached computations and by stopping approximate retrieval early when the model is ready, with no measured loss in answer quality.
-
Multi-Armed Bandits-Based Optimization of Decision Trees
A multi-armed bandit-based dynamic pruning method for decision trees is proposed, reporting improved generalization over greedy cost-complexity and reduced-error pruning on benchmark datasets.
-
FlexRAG: A Flexible and Comprehensive Framework for Retrieval-Augmented Generation
FlexRAG is a modular, open-source RAG framework with text, multimodal, and web retrieval, plus evaluation tools and efficient memory-mapped indexing.
-
Optimizing the Interface Between Knowledge Graphs and LLMs for Complex Reasoning
Hyperparameter tuning of Cognee's knowledge graph pipeline yields consistent but uneven gains across three multi-hop QA benchmarks, with best configurations varying by dataset and metric.
-
Research on the Online Update Method for Retrieval-Augmented Generation (RAG) Model with Incremental Learning
An online-update RAG method built from dynamic memory, distillation, hierarchical indexing, and cross-attention is presented, but without any working equations or experimental data.
Discussion (0). Continue with ORCID to comment.