REVIEW 3 cited by
Retrieve, Annotate, Evaluate, Repeat: Leveraging Multimodal LLMs for Large-Scale Product Retrieval Evaluation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Evaluating production-level retrieval systems at scale is a crucial yet challenging task due to the limited availability of a large pool of well-trained human annotators. Large Language Models (LLMs) have the potential to address this scaling issue and offer a viable alternative to humans for the bulk of annotation tasks. In this paper, we propose a framework for assessing the product search engines in a large-scale e-commerce setting, leveraging Multimodal LLMs for (i) generating tailored annotation guidelines for individual queries, and (ii) conducting the subsequent annotation task. Our method, validated through deployment on a large e-commerce platform, demonstrates comparable quality to human annotations, significantly reduces time and cost, facilitates rapid problem discovery, and provides an effective solution for production-level quality control at scale.
Forward citations
Cited by 3 Pith papers
-
CliniQ: A Multi-faceted Benchmark for Electronic Health Record Retrieval with Semantic Match Assessment
CliniQ is a public EHR retrieval benchmark with 77,206 LLM-annotated relevance judgments, showing that BM25 is a strong baseline and that semantic matches drive dense-retriever gains.
-
Scaling Search Relevance: Augmenting App Store Ranking with LLM-Generated Judgments
Using a fine-tuned 3B LLM to generate millions of textual relevance labels for App Store search improves the ranker's behavioral/textual Pareto frontier and lifts conversion by 0.24%.
-
Automatic Dataset Generation for Knowledge Intensive Question Answering Tasks
Fine-tuning Llama-3-8b and Mistral-7b-v0.3 on LLM-generated QA pairs from IBM Technotes can improve no-context QA scores over training on human-annotated TechQA data, but the evaluation may be inflated by test-documen...
Discussion (0). Continue with ORCID to comment.