REVIEW 3 cited by
PaRaDe: Passage Ranking using Demonstrations with Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recent studies show that large language models (LLMs) can be instructed to effectively perform zero-shot passage re-ranking, in which the results of a first stage retrieval method, such as BM25, are rated and reordered to improve relevance. In this work, we improve LLM-based re-ranking by algorithmically selecting few-shot demonstrations to include in the prompt. Our analysis investigates the conditions where demonstrations are most helpful, and shows that adding even one demonstration is significantly beneficial. We propose a novel demonstration selection strategy based on difficulty rather than the commonly used semantic similarity. Furthermore, we find that demonstrations helpful for ranking are also effective at question generation. We hope our work will spur more principled research into question generation and passage ranking.
Forward citations
Cited by 3 Pith papers
-
Selecting Demonstrations for Many-Shot In-Context Learning via Gradient Matching
CLG selects many-shot demonstrations by matching fine-tuning gradients of a small language model to the full training set, improving accuracy over random selection by 2-4%.
-
Evaluating and Improving Graph to Text Generation with Large Language Models
Introducing PlanGTG, a 29k-pair instruction dataset with reordering and attribution subtasks, and fine-tuning 7B LLMs on it improves graph-to-text generation on WebNLG and DART over untuned and dataset-tuned baselines.
-
PaSa: An LLM Agent for Comprehensive Academic Paper Search
PaSa, a two-agent LLM system trained with session-level RL, reports substantially higher recall than existing academic search baselines on complex paper-finding queries.
Discussion (0). Continue with ORCID to comment.