REVIEW 6 cited by
Atla Selene Mini: A General Purpose Evaluation Model
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We introduce Atla Selene Mini, a state-of-the-art small language model-as-a-judge (SLMJ). Selene Mini is a general-purpose evaluator that outperforms the best SLMJs and GPT-4o-mini on overall performance across 11 out-of-distribution benchmarks, spanning absolute scoring, classification, and pairwise preference tasks. It is the highest-scoring 8B generative model on RewardBench, surpassing strong baselines like GPT-4o and specialized judges. To achieve this, we develop a principled data curation strategy that augments public datasets with synthetically generated critiques and ensures high quality through filtering and dataset ablations. We train our model on a combined direct preference optimization (DPO) and supervised fine-tuning (SFT) loss, and produce a highly promptable evaluator that excels in real-world scenarios. Selene Mini shows dramatically improved zero-shot agreement with human expert evaluations on financial and medical industry datasets. It is also robust to variations in prompt format. Preliminary results indicate that Selene Mini is the top-ranking evaluator in a live, community-driven Judge Arena. We release the model weights on HuggingFace (https://hf.co/AtlaAI/Selene-1-Mini-Llama-3.1-8B) and Ollama to encourage widespread community adoption.
Forward citations
Cited by 6 Pith papers
-
Reflection or Re-Generation? Why LLM Revision Fails Where Human Revision Succeeds
Matched two-pass experiments show human revisers improve on objective and subjective tasks, while LLM self-revision yields near-zero information gain on objective tasks and negative information gain on subjective task...
-
Persistent Sparse Autoencoders: Learning Feature Timescales in Language Models
Persistent SAEs learn per-feature persistence coefficients from reconstruction, splitting features into fast local detectors and slow topic-tracking states that retain prompt-injection signals over long contexts.
-
Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization
JudgeBiasBench measures 12 LLM-judge bias types under a 4-dimension taxonomy, and bias-aware GRPO/InfoNCE training cuts BSR substantially while preserving general evaluation performance.
-
ReliableEval: A Recipe for Stochastic LLM Evaluation via Method of Moments
ReliableEval estimates the minimum number of meaning-preserving prompt resamplings needed to make an LLM evaluation reliable, and applies it to show frontier LLMs are notably prompt-sensitive.
-
Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation
A taxonomy-driven survey arguing that reward design is the central mechanism shaping reliable LLM reasoning, with maps of reward paradigms, reward-hacking failure modes, and benchmark pitfalls.
-
Reward Reasoning Model
Reward models that think before judging, trained via reinforcement learning without human-written reasoning traces, outperform standard reward models and improve with more test-time compute.
Discussion (0). Continue with ORCID to comment.