REVIEW 9 cited by
OLMES: A Standard for Language Model Evaluations
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Progress in AI is often demonstrated by new models claiming improved performance on tasks measuring model capabilities. Evaluating language models can be particularly challenging, as choices of how a model is evaluated on a task can lead to large changes in measured performance. There is no common standard setup, so different models are evaluated on the same tasks in different ways, leading to claims about which models perform best not being reproducible. We propose OLMES, a completely documented, practical, open standard for reproducible LLM evaluations. In developing this standard, we identify and review the varying factors in evaluation practices adopted by the community - such as details of prompt formatting, choice of in-context examples, probability normalizations, and task formulation. In particular, OLMES supports meaningful comparisons between smaller base models that require the unnatural "cloze" formulation of multiple-choice questions against larger models that can utilize the original formulation. OLMES includes well-considered, documented recommendations guided by results from existing literature as well as new experiments resolving open questions.
Forward citations
Cited by 9 Pith papers
-
Fluid Language Model Benchmarking
Fluid Benchmarking, combining IRT-based ability estimation with Fisher-information-based adaptive item selection, improves LM evaluation across efficiency, validity, variance, and saturation in pretraining settings.
-
Safety Under Scaffolding: How Evaluation Conditions Shape Measured Safety
Map-reduce scaffolding degrades measured safety mainly by stripping multiple-choice options (40–89% of the loss is format conversion); scaffold architecture explains only 0.4% of variance and composite safety scores h...
-
Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models
Online self-play between attacker and defender roles of a single LLM improves safety robustness and attack diversity across Llama and Qwen models.
-
PLATE: Plasticity-Tunable Efficient Adapters for Geometry-Aware Continual Learning
PLATE constructs frozen weight-derived bases B and Q and trains only a small core A, reducing catastrophic forgetting during data-free continual fine-tuning.
-
Pre-Training LLMs on a budget: A comparison of three optimizers
In budget-constrained 2.7B-parameter LLM pre-training, Lion is fastest, Sophia reaches the lowest loss, but AdamW wins on downstream benchmarks.
-
Low-rank Momentum Factorization for Memory Efficient Training
MoFaSGD keeps a low-rank factored momentum and uses its singular vectors as the update direction, achieving LoRA-level memory with competitive fine-tuning performance, but its convergence proof is flawed.
-
FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
An adaptive, per-language data filtering and deduplication pipeline produces multilingual LLM pre-training corpora that beat prior public datasets on 11 of 14 evaluated languages, and a 20TB, 1,868 language-script dat...
-
On Recipe Memorization and Creativity in Large Language Models: Is Your Model a Creative Cook, a Bad Cook, or Merely a Plagiator?
Mixtral's recipe ingredients are mostly traceable to online recipes, and an LLM-as-judge pipeline can reproduce human memorization annotations with up to 78 percent accuracy.
-
Llama-3.1-FoundationAI-SecurityLLM-8B-Instruct Technical Report
Foundation-Sec-8B-Instruct, an instruction-tuned 8B cybersecurity LLM, is released and claimed to beat Llama 3.1-8B-Instruct on CTIBench-RCM and CTIBench-MCQA while remaining competitive on general instruction-following.
Discussion (0). Sign in to comment.