Validity in agentic AI evaluation degrades multiplicatively across task generation, simulation, and judging, so most reported benchmark scores retain far less information than they appear to.
Cohen, A coefficient of agreement for nominal scales, Educa- tional and Psychological Measurement 20 (1960) 37–46
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
citation-role summary
background 1
citation-polarity summary
fields
cs.AI 1years
2026 1verdicts
CONDITIONAL 1roles
background 1polarities
unclear 1representative citing papers
citing papers explorer
-
Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation
Validity in agentic AI evaluation degrades multiplicatively across task generation, simulation, and judging, so most reported benchmark scores retain far less information than they appear to.