arxiv: 2604.01375 · v2 · submitted 2026-04-01 · 💻 cs.AI

Recognition: no theorem link

RIFT: A RubrIc Failure Mode Taxonomy and Automated Diagnostics

Zhengyang Qi , Charles Dickens , Derek Pham , Amanda Dsouza , Armin Parchami , Frederic Sala , Paroma Varma

Authors on Pith no claims yet

Pith reviewed 2026-05-13 22:22 UTC · model grok-4.3

classification 💻 cs.AI

keywords rubric failure modesLLM evaluationtaxonomyautomated diagnosticsgrounded theoryvalidity failuresrubric quality metrics

0 comments

The pith

RIFT introduces a taxonomy of eight failure modes in three categories to diagnose issues in LLM rubric design.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Rubric-based evaluation helps assess open-ended tasks in LLMs but lacks ways to diagnose rubric failures from aggregate signals. The paper develops RIFT through grounded theory on rubrics from five sources covering instruction following, code generation, creative writing, and research tasks. It defines eight failure modes split into Reliability Failures, Content Validity Failures, and Consequential Validity Failures. Independent annotators achieve 87% pairwise agreement with 0.64 Cohen's kappa, and proposed automated metrics reach up to 0.925 F1 alignment with human labels. This provides a systematic approach to improve rubric quality for better LLM evaluations.

Core claim

RIFT is a taxonomy consisting of eight failure modes organized into three high-level categories: Reliability Failures, Content Validity Failures, and Consequential Validity Failures, developed by iteratively annotating rubrics from diverse data sources until saturation and validated with human agreement and automated metrics.

What carries the argument

The RIFT taxonomy of eight rubric failure modes grouped into reliability, content validity, and consequential validity categories.

Load-bearing premise

The eight failure modes identified from the five sampled data sources form a sufficiently complete taxonomy for all possible rubric use cases.

What would settle it

Discovery of a previously unseen failure mode in rubrics from a new domain that cannot be classified into any of the eight existing modes.

read the original abstract

Rubric-based evaluation is widely used in LLM benchmarks and training pipelines for open-ended, less verifiable tasks. While prior work has demonstrated the effectiveness of rubrics using downstream signals such as reinforcement learning outcomes, there remains no principled way to diagnose how a rubric itself fails from such aggregated or downstream signals alone. To address this gap, we introduce RIFT: RubrIc Failure mode Taxonomy, a taxonomy for systematically characterizing failure modes in rubric composition and design. RIFT consists of eight failure modes organized into three high-level categories: Reliability Failures, Content Validity Failures, and Consequential Validity Failures. RIFT is developed using grounded theory by iteratively annotating rubrics drawn from five diverse data sources spanning general instruction following, code generation, creative writing, and expert-level deep research, until no new failure modes are identified. We evaluate the consistency of the taxonomy by measuring agreement among independent human annotators, observing fair agreement overall (87% pairwise agreement and 0.64 average Cohen's kappa). Finally, to support scalable diagnosis, we propose automated rubric quality metrics and show that they align with human failure-mode annotations, achieving up to 0.925 F1.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

RIFT gives a practical eight-mode taxonomy for diagnosing rubric failures in LLM evals plus automated metrics that hit 0.925 F1, but completeness rests on saturation from only five sources.

read the letter

RIFT introduces a taxonomy of eight failure modes for rubrics in LLM evaluation, split into reliability failures, content validity failures, and consequential validity failures. The authors built it through grounded-theory annotation on rubrics from five sources covering instruction following, code generation, creative writing, and expert research, stopping when no new modes appeared. They also supply automated metrics that align with human labels at up to 0.925 F1. This is the concrete new element: a direct way to inspect rubric quality instead of relying only on downstream signals like RL outcomes. The work is grounded in actual rubrics rather than abstract theory, and the reported alignment between the automated checks and the human annotations is a clear positive. The main limitation is scope. The taxonomy depends on saturation being reached in those five sources, yet inter-annotator agreement is only fair at 0.64 kappa. Without an external validation set or tests on additional domains, it is possible that other failure modes exist outside the sampled areas, which would narrow the taxonomy's claimed generality and the metrics calibrated to it. This paper is aimed at researchers who design or audit rubrics for open-ended LLM tasks in benchmarks and training pipelines. Anyone who needs a structured checklist for spotting rubric problems would find the categories and the automated diagnostics immediately usable. I would send it for peer review. The contribution is practical, the evidence is presented plainly, and the experiments are reproducible enough that referees can assess the generality question directly.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces RIFT, a taxonomy of eight rubric failure modes organized into three high-level categories (Reliability Failures, Content Validity Failures, and Consequential Validity Failures). The taxonomy is constructed via grounded-theory annotation of rubrics from five data sources (general instruction following, code generation, creative writing, expert-level deep research) until saturation, with reported inter-annotator agreement of 87% pairwise and 0.64 Cohen's kappa. Automated rubric quality metrics are proposed and shown to align with the human annotations, reaching up to 0.925 F1.

Significance. If the taxonomy proves sufficiently general, the work supplies a needed diagnostic lens for rubric failures in LLM evaluation that cannot be recovered from downstream signals alone. The automated metrics add a practical, scalable component. The grounded-theory process and explicit agreement metrics are strengths that support internal consistency.

major comments (2)

[Taxonomy construction] The central claim that the eight modes constitute a sufficiently complete taxonomy for rubric use in LLM evaluation rests on saturation from only five sampled sources (abstract and taxonomy-development section). No external validation set or statistical test of exhaustiveness is reported, which directly limits the scope of the automated metrics calibrated to these modes.
[Evaluation of taxonomy consistency] The moderate inter-annotator agreement (0.64 kappa) is load-bearing for the reliability of the mode definitions used to train/validate the automated metrics; this level of subjectivity should be addressed with additional adjudication or clearer decision rules before the metrics can be treated as robust.

minor comments (2)

[Automated metrics] The exact formulas, features, and any tunable thresholds for the automated metrics should be stated explicitly (including how the 0.925 F1 is computed) so that the alignment result can be reproduced.
[Data sources] Provide the precise counts and selection criteria for the rubrics drawn from each of the five sources to strengthen replicability.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript introducing the RIFT taxonomy. The comments highlight important considerations around generalizability and annotation reliability. We address each point below and indicate planned revisions.

read point-by-point responses

Referee: [Taxonomy construction] The central claim that the eight modes constitute a sufficiently complete taxonomy for rubric use in LLM evaluation rests on saturation from only five sampled sources (abstract and taxonomy-development section). No external validation set or statistical test of exhaustiveness is reported, which directly limits the scope of the automated metrics calibrated to these modes.

Authors: We agree that the taxonomy was developed via theoretical saturation within the five sampled domains (general instruction following, code generation, creative writing, and expert-level deep research) rather than through an external validation set or formal statistical test of exhaustiveness. Grounded theory methodology prioritizes saturation as the stopping criterion for identifying core categories. In the revised manuscript we will add an explicit Limitations subsection that discusses the scope of the current sources, acknowledges the possibility of additional modes in unrepresented domains (e.g., multimodal or real-time interaction settings), and qualifies the expected generalizability of the automated metrics. We cannot introduce a new external validation set at this stage, but the limitation will be stated clearly and positioned as an avenue for future work. revision: partial
Referee: [Evaluation of taxonomy consistency] The moderate inter-annotator agreement (0.64 kappa) is load-bearing for the reliability of the mode definitions used to train/validate the automated metrics; this level of subjectivity should be addressed with additional adjudication or clearer decision rules before the metrics can be treated as robust.

Authors: We acknowledge that a Cohen's kappa of 0.64 indicates moderate agreement and that this level of subjectivity merits greater transparency. In the revision we will expand the annotation protocol section to include the explicit decision rules applied for each failure mode, provide concrete examples of edge cases that were discussed during adjudication, and describe the process by which final labels were reached after independent coding. These additions will improve reproducibility of the taxonomy definitions and thereby strengthen the foundation for the automated metrics without changing the reported agreement figures. revision: yes

Circularity Check

0 steps flagged

No circularity: taxonomy built from external sources with independent validation of metrics

full rationale

The paper constructs the RIFT taxonomy via grounded theory by iteratively annotating rubrics from five independent external data sources until saturation, with no new modes emerging. Automated rubric quality metrics are then proposed and evaluated for alignment with the resulting human failure-mode annotations, reporting up to 0.925 F1. This is direct empirical validation on the annotated data rather than any derivation that reduces to its inputs by construction. No self-definitional steps, fitted parameters renamed as predictions, load-bearing self-citations, or uniqueness claims imported from prior author work appear in the provided text. The chain remains self-contained against the sampled sources and human labels.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The taxonomy is built bottom-up from annotation rather than derived from prior equations; the main untested premise is that saturation on five sources yields a general set of modes.

axioms (1)

domain assumption Grounded theory annotation until saturation produces an exhaustive taxonomy of rubric failures
Iteratively annotated rubrics from five sources until no new failure modes appeared.

invented entities (1)

RIFT taxonomy of eight rubric failure modes no independent evidence
purpose: Systematic characterization of failures in rubric composition and design
Constructed via grounded theory with no independent falsifiable test outside the annotation agreement.

pith-pipeline@v0.9.0 · 5522 in / 1338 out tokens · 65803 ms · 2026-05-13T22:22:09.022276+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

7 extracted references · 7 canonical work pages

[1]

Can we trust ai benchmarks? an interdisciplinary re- view of current issues in ai evaluation.arXiv preprint arXiv:2502.06559,

Grounded theory research: A design frame- work for novice researchers.Open Medicine, 7:1–8. Maria Eriksson, Erasmo Purificato, Arman Noroozian, Joao Vinagre, Guillaume Chaslot, Emilia Gomez, and David Fernandez-Llorca. 2025. Can we trust ai benchmarks? an interdisciplinary review of current issues in ai evaluation.Preprint, arXiv:2502.06559. Barney G. Gla...

work page arXiv 2025
[2]

suggested_labels

Betterbench: Assessing ai benchmarks, un- covering issues, and establishing best practices. In Advances in Neural Information Processing Systems, volume 37, pages 21763–21813. Curran Associates, Inc. MohammadHossein Rezaei, Robert Vacareanu, Zihao Wang, Clinton Wang, Bing Liu, Yunzhong He, and Afra Feyza Akyürek. 2025. Online rubrics elicitation from pair...

work page arXiv 2025
[3]

failure_modes: The complete list of failure modes after applying changes. Each failure mode should have: •label: concise identifier (e.g., contradictory_criteria, missing_edge_cases) •description: clear, specific, and actionable description that explains HOW to determine if a rubric has this failure mode (what to look for, what conditions must be met) •ra...

work page
[4]

Added ‘contradictory_criteria’ based on rubric critiques

changes_summary: A list of strings describing what changes you made (e.g., "Added ‘contradictory_criteria’ based on rubric critiques", "Clarified description of ‘ambiguous_criterion’", "Merged ‘x’ and ‘y’ into ‘z’") If no changes are needed based on these critiques, return the current running refinement unchanged with an empty changes_summary. D RIFT: Rub...

work page
[5]

two sentences

List the prompt’s core requirements: • required deliverables/components, • must/must-not constraints, • required format/ordering/sections, • and genre-critical qualities the prompt clearly expects (e.g., functional correct- ness for code; “two sentences”; “valid JSON”; “include 10 items”; “chronologi- cal order”)

work page
[6]

For each requirement, check whether ANY rubric criterion covers it

work page
[7]

quiet,” “fast Wi-Fi,

If one or more requirements have no corre- sponding criterion, apply. Do NOT apply if: • The rubric mentions the requirement but is vague or subjective (useSubjective). • The rubric mentions the requirement but it cannot be graded consistently due to missing keys/tolerances/bounded audit steps (useUn- grounded). • The rubric grades a different task or add...

work page