Recognition: unknown
Measuring Choice Difficulty
Pith reviewed 2026-05-07 10:59 UTC · model grok-4.3
The pith
Three common measures of choice difficulty are unrelated in a standard decision model.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
In a binary-option Bayesian expected-utility framework, the three measures of difficulty given by ex-ante value of information, the degree of choice randomness, and ex-post in the correctness of the chosen option are in general unrelated; the same unrelatedness holds for measures such as attenuation. Sufficient conditions exist under which their orders coincide, including restrictions to Blackwell experiments that capture the logit model and restrictions on payoffs. When the task pays only for correctness, and when willingness-to-accept to switch is measured in utility units, the relevant measures become equivalent.
What carries the argument
Binary-option Bayesian expected-utility framework that defines understanding as the ex-ante value of information, choice randomness as a separate statistic, and as the probability the chosen option is ex-post correct.
If this is right
- Observing more random choices does not imply that the decision maker places lower value on learning the true state.
- In tasks that pay solely for being correct, measured can serve as a direct proxy for the value of understanding.
- Willingness to pay to switch options, when scaled in utility terms, equals the value of information.
- Findings on choice difficulty from economics experiments cannot be carried over to psychophysics tasks without verifying that the payoff structure satisfies the alignment conditions.
- Other candidate measures of difficulty such as attenuation are also unrelated to the three main measures in the same framework.
Where Pith is reading between the lines
- Experiments that aim to study difficulty may need to measure or control for multiple indicators separately rather than treating any single one as sufficient.
- The result raises the possibility that meta-analyses pooling studies that use different difficulty proxies are comparing unlike quantities.
- Extensions of the framework to choices with more than two options or to repeated decisions could show when the measures begin to correlate.
- Direct elicitation of willingness-to-accept to switch in the same units as the other measures offers a practical way to test the equivalence result in the lab.
Load-bearing premise
The analysis assumes a binary choice setting under Bayesian expected utility together with the specific mathematical definitions given for each difficulty measure.
What would settle it
An experiment that elicits all three measures from the same subjects in binary choices and finds they are always positively correlated even when the information structure and payoffs lie outside the paper's identified sufficient conditions would contradict the general unrelatedness result.
Figures
read the original abstract
We provide a theoretical framework to understand how widely used measures of choice difficulty relate. In a binary-option Bayesian expected-utility framework, we show that three measures of difficulty, (i) understanding (ex-ante value), (ii) choice randomness, and (iii) confidence that the chosen option is ex post correct, are, in general, unrelated, and that this result extends to other potential measures like attenuation. We provide intuitive sufficient conditions which align the orders, using both restrictions on Blackwell experiments that capture well known classes (such as logit) and restrictions on payoffs and demonstrate that in psychophysical tasks that pay only for correctness, confidence coincides with understanding. We show willingness-to-accept to switch, when measured in utils, is equivalent to understanding. Our results suggest caution in interpreting measures of choice difficulty as well as the degree of portability between economics and psychophysics experiments
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript develops a theoretical framework in a binary-option Bayesian expected-utility setting to relate common measures of choice difficulty. It defines three measures—(i) understanding via ex-ante value, (ii) choice randomness, and (iii) ex-post confidence that the chosen option is correct—and shows they are unrelated in general. The result is extended to other measures such as attenuation. Sufficient conditions for alignment are derived using restrictions on Blackwell experiments (including logit and similar classes) and on payoffs. Special cases are analyzed: in psychophysical tasks paying only for correctness, confidence coincides with understanding; willingness-to-accept to switch, measured in utils, is equivalent to understanding. The paper cautions against direct interpretation of difficulty measures and against assuming portability between economics and psychophysics experiments.
Significance. If the derivations hold, the paper offers a useful clarification of why widely used difficulty proxies often fail to align, supplying both a general negative result and intuitive positive conditions under which orders coincide. The explicit treatment of Blackwell informativeness restrictions and the psychophysics equivalence result are strengths that could guide experimental design. The framework is model-relative by construction, which limits over-claiming while still delivering falsifiable predictions for when measures will or will not correlate. This contributes to the literature on information, stochastic choice, and measurement in behavioral economics.
minor comments (4)
- §2: The extension to 'attenuation' as an additional difficulty measure is mentioned in the abstract but should receive an explicit formal definition and equation parallel to the three primary measures to ensure the 'extends to other potential measures' claim is fully operational.
- §3: The statement of general unrelatedness would benefit from a short table or enumerated list of the specific counter-examples or parameter regions used to establish the result, making the scope of 'in general' immediately visible to readers.
- §4.2: The payoff-restriction conditions for alignment are intuitive but could be accompanied by a brief numerical illustration (e.g., two payoff matrices) to show how the orders become monotonic, improving accessibility without lengthening the text.
- Notation: The symbol for ex-ante value is introduced without an immediate comparison to the standard value-of-information expression; a one-sentence remark on the relationship would prevent possible confusion with existing literature.
Simulated Author's Rebuttal
Thank you for the detailed summary and positive assessment of our framework. We appreciate the recognition of the general negative result on measure misalignment, the value of the Blackwell and payoff restrictions, and the psychophysics equivalence finding. No major comments were specified in the report, so we have no targeted revisions at this time.
Circularity Check
No significant circularity in the derivation chain
full rationale
The paper constructs a binary-option Bayesian expected-utility framework and derives within that model that the three defined difficulty measures (ex-ante value, choice randomness, ex-post confidence) are unrelated in general, with sufficient conditions (Blackwell restrictions, payoff restrictions) under which they align. All results follow directly from the stated definitions and assumptions; no equation reduces to its inputs by construction, no parameters are fitted and then relabeled as predictions, and no load-bearing claims rest on self-citations. Special cases such as psychophysical tasks paying only for correctness are analyzed consistently inside the same framework. The derivation is therefore self-contained and model-relative by design.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Binary choice under Bayesian updating with expected-utility maximization
Reference graph
Works this paper leans on
-
[1]
Quantifying lottery choice complexity,
v2. Enke, Benjamin and Cassidy Shubatt, “Quantifying lottery choice complexity,” Tech- nical Report, National Bureau of Economic Research 2023. 46 and Thomas Graeber, “Cognitive uncertainty,”The Quarterly Journal of Eco- nomics, 2023,138(4), 2021–2067. , , Ryan Oprea, and Jeffrey Yang, “Behavioral attenuation,” Technical Report, National Bureau of Economi...
-
[2]
more random
These generalized Fechnerian models have no direct role for “beliefs”, although He and Naten- zon (2023) demonstrate that they can be rewritten in a way consistent with a Bayesian expected utility story under specific assumptions. However, they immediately imply that the value of a choice problem is directly related to the degree of randomness. Propositio...
2023
-
[3]
choice payoff dominates
This means that increases (or decreases) inλdo not uniformly make choice more or less attenuated (in contrast, recall that even in the standardλ-Luce model, we know that in each state, randomness must decrease). The same intuition applies more broadly to the generalized Fechnerian framework where choosingx P(x) =f( u(x)−u(y) λ ). We assumefis strictly inc...
2023
-
[4]
Case 2:ρσ(x|ω∗)< 1
Thenxis the (weakly) most-chosen option in thex-optimal stateω∗, soσis indicative inω∗. Case 2:ρσ(x|ω∗)< 1
-
[5]
By (5),ρσ(x|ω∗′)≤ρσ(x|ω∗)< 1
-
[6]
57 This leads to the following corollary24 Corollary 1.Ifσis less random thanσ′state-by-state, then there exists a stateω∗ withW(σ|ω∗)≥W(σ′|ω∗)
Henceρσ(y|ω∗′)> 1 2, and sinceω∗′∈Ω(y), the correct optionyis chosen more than half the time inω∗′—soσis indicative inω∗′.□ 23As the proof will demonstrate this requires key features of our environment: symmetric priors and binary options. 57 This leads to the following corollary24 Corollary 1.Ifσis less random thanσ′state-by-state, then there exists a st...
-
[7]
neutral mass
In the tied state, they differ: σ′(s1|ω3) = 1 2, σ(s 1|ω3) = 1 2 +ε, for someε∈(0,1 2). Under both experiments, the DM choosesxafters1 andyafters 2. To verify: the expected utility differenceE[u(x)−u(y)|s1]is proportional toπ(ω1)σ(s1|ω1)(uH− uL)−π(ω2)σ(s1|ω2)(uH−uL) =p(u H−uL)[θ−(1−θ)]>0(the tied state contributes nothing sinceu(x|ω3) =u(y|ω3)), and symme...
1988
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.