REVIEW 4 cited by
Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Language Models (LLMs) are advancing at an amazing speed and have become indispensable across academia, industry, and daily applications. To keep pace with the status quo, this survey probes the core challenges that the rise of LLMs poses for evaluation. We identify and analyze two pivotal transitions: (i) from task-specific to capability-based evaluation, which reorganizes benchmarks around core competencies such as knowledge, reasoning, instruction following, multi-modal understanding, and safety; and (ii) from manual to automated evaluation, encompassing dynamic dataset curation and "LLM-as-a-judge" scoring. Yet, even with these transitions, a crucial obstacle persists: the evaluation generalization issue. Bounded test sets cannot scale alongside models whose abilities grow seemingly without limit. We will dissect this issue, along with the core challenges of the above two transitions, from the perspectives of methods, datasets, evaluators, and metrics. Due to the fast evolving of this field, we will maintain a living GitHub repository (links are in each section) to crowd-source updates and corrections, and warmly invite contributors and collaborators.
Forward citations
Cited by 4 Pith papers
-
Disentangling Language and Culture for Evaluating Multilingual Large Language Models
A new dual-axis evaluation framework shows multilingual LLMs answer culture-specific questions best when the question language matches the cultural context, with partial neuron-level evidence for the effect.
-
Data Doping or True Intelligence? Evaluating the Transferability of Injected Knowledge in LLMs
LLMs fine-tuned on comprehension tasks like question answering retain injected facts at more than double the rate of models fine-tuned on translation or JSON mapping, but all models struggle to apply the facts in new ...
-
Continuous Monitoring of Large-Scale Generative AI via Deterministic Knowledge Graph Structures
A continuous monitor compares an LLM-built knowledge graph with a rule-built knowledge graph from the same news stream and flags structural drift as possible hallucination.
-
Toward Edge General Intelligence with Multiple-Large Language Model (Multi-LLM): Architecture, Trust, and Orchestration
A survey of multi-LLM systems in edge computing, covering architectures, enabling technologies, trust mechanisms, applications, and open datasets for edge general intelligence.
Discussion (0). Continue with ORCID to comment.