{"id":"3cf44b90-49a1-4106-b8f5-75df94998b50","arxiv_id":"2607.20864","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Position bias in LLM multiple-choice benchmarks is measurable only in a roughly 60-95% baseline-accuracy band; above that, ceiling effects hide it, so a null result is not evidence of no bias.","lead":"An evaluation toolkit runs all 24 answer-order permutations per multiple-choice question and shows that LLM position bias is statistically detectable only when model accuracy sits in a mid-range band. The paper argues that clean \"no bias\" results on easy benchmarks are a measurement ceiling, not proof of unbiasedness.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Chi-squared tests treat 24 permutations of the same question as independent; clustered reanalysis may shift active/inactive labels and the 60–95% band.","rationale":"The reader's weakest assumption correctly identifies the pseudo-replication of 24 permutations per question as the most load-bearing statistical flaw. My independent reading of the paper finds the same issue: every inferential quantity in Section 4 (p-values, bootstrap CIs, active/inactive labels) treats runs as independent, while the design is a cluster sample by question. The Goldilocks zone—the paper's central contribution—is defined by which cells are detectable; if the labels change under clustering, the zone boundaries and the interpretation of null results are not established. The proposed cluster bootstrap is a direct, feasible check. The floor-side extrapolation is also a limitation, but it is explicitly disclosed by the author and does not threaten the upper-bound claim as severely as the independence violation. The paper's reproducible artifacts, preregistration, and threshold sensitivity analysis are genuine strengths, so a flat rejection would be unfair; the verdict remains CONDITIONAL pending the clustered reanalysis, consistent with the reader's assessment.","tokens_in":17291,"tokens_out":2626,"duration_ms":27692,"concrete_test":"Reanalyze all 20 cells using a question-level cluster bootstrap: resample 50 questions with replacement from each cell, retain all 24 permutations per sampled question, aggregate counts into a 2×4 table, and compute chi-squared/p and V for each resample. If any currently active gpt-4o-mini or claude-haiku cell fails to reach p<0.05 in the clustered distribution, the band boundaries and the 'absence of signal is not evidence of unbiasedness' claim lose their current empirical support.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim of a 60–95% Goldilocks detection zone depends on the active/inactive classification of 20 vendor×subject cells. That classification rests on chi-squared p-values and bootstrap CIs computed over N=1,200 runs per cell (Sections 2.3, 4.1). But the 1,200 runs are not independent: they are 50 questions each repeated under 24 exhaustive permutations of the same content. With temperature-0 generation, responses within a question are deterministic functions of the prompt and therefore highly correlated; across questions, difficulty varies. This violates the independence assumption of the chi-squared test, inflating the effective sample size and making p-values and CIs overconfident. The paper explicitly acknowledges content-level noise as a confound (Section 2.1) yet does not account for it in the significance tests. If within-question correlation is substantial, cells currently labeled active (notably the gpt-4o-mini cells with V≈0.10–0.18) could lose significance under a question-level clustered analysis, changing which cells sit inside the claimed band and thus undermining the upper-edge claim and the interpretation of null results as 'not measurable' rather than 'unbiased.' A secondary issue, the unobserved floor below 0.60, is explicitly flagged by the author as extrapolation; but the independence flaw affects the core empirical anchor.","agreement_with_reader":"agree"},"referee_report":null,"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing: the tool is real. inspect_permute is an open-source extension that runs exhaustive answer-order permutations, and the paper ships the package, logs, and a reproducible pipeline. That is a concrete contribution to how position-bias measurements are done. The empirical matrix across four vendors and five MMLU subjects is also new, and the descriptive patterns—monotone A-to-D decrease for gpt-4o-mini, D-drop for haiku—are worth looking at. The paper is also unusually honest: it discloses the classifier flaw, reports the preregistration misses unfiltered, and explicitly labels the three-band model post hoc.\n\nThe soft spot is the statistical inference underneath the Goldilocks zone. The chi-squared tests and bootstrap CIs use N=1,200 runs per cell, but those are 50 questions × 24 permutations of the same content. Under temperature 0, responses within a question are deterministic and highly correlated; they are not independent observations. A question-level clustered analysis is the right unit, and with effective N near 50, some cells currently called 'active' (gpt-4o-mini with V around 0.10–0.18) could easily lose significance. That would change which cells sit inside the claimed 60–95% band, and the 'absence of signal means not measurable' reading depends on the band boundaries being right. The paper acknowledges content-level noise as a conceptual confound but does not carry it into the significance tests. This is fixable—bootstrap by question, or fit a mixed model—but it needs to be done before the central claim can be trusted.\n\nTwo smaller points. The floor edge below 60% accuracy is explicitly extrapolated—no cell in the sweep is below 0.61—so that half of the band is speculative, though the paper says so. And the preregistration anchor was posted after ten cells had already been run; it's a partial preregistration, not a clean one, and the predictions for the unobserved cells were informed by the first half. The author doesn't hide this, but it's weaker than the abstract's 'before half the data was observed' suggests.\n\nBottom line: this is a paper for people who care about position-bias methodology and benchmark calibration. The tool is worth having, the Goldilocks concept is worth testing, and the author is thinking seriously. But the central empirical claim is not established as stated. A serious referee should ask for a clustered reanalysis and a more careful lower-bound claim. I would not reject it; I would send it back for revision.","headline":"Useful diagnostic tool and a plausible 'Goldilocks zone' framing, but the significance tests that anchor the zone treat 24 permutations of the same question as 24 independent observations, which overstates confidence and likely shifts which cells are called biased.","tokens_in":18107,"tokens_out":4979,"would_cite":true,"duration_ms":51269,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM position bias is only measurable between 60% and 95% accuracy.","keywords":["position bias","LLM evaluation","multiple-choice benchmarks","ceiling effects","permutation testing","Cramér's V","benchmark calibration","pre-registration"],"falsifier":"Take a frontier model with base accuracy above 0.95 on a standard multitask subject, append a prompt-level instruction to prefer option D when uncertain, and run the 24-permutation diagnostic on the same 50 questions. If the test still reports Cramér's V below 0.05 with p above 0.05, ceiling-effect masking is confirmed; if it detects the injected bias, the claimed upper edge of the Goldilocks zone is not universal.","tokens_in":17103,"feed_emoji":"🎯","tokens_out":7881,"duration_ms":69578,"temperature":0.7,"pith_summary":"The paper argues that position bias (a multiple-choice language model's tendency to answer according to the screen position of options rather than their content) can be statistically detected only when the model's base accuracy on a benchmark falls in a roughly 60 to 95 percent band. Below that band, processing noise swamps the bias signal; above it, ceiling effects compress response variance below the resolution of the chi-squared test. Because current frontier-tier models score above 95 percent on standard multitask benchmark subjects, the paper concludes that their flat permutation results should be read as 'not measurable on this instrument,' not as evidence of impartiality. This matters because it implies that many published no-bias claims on saturated benchmarks are unverifiable, and it reframes the field's central question: benchmark calibration must come before bias measurement. The claim is supported by a 24,000-call sweep across four vendor models and five subjects using exhaustive answer-order permutations, with pre-registered falsifier predictions.","feed_headline":"LLM position bias is only measurable between 60% and 95% accuracy","feed_subtitle":"A 24,000-call sweep shows flat results on saturated benchmarks are 'not measurable,' not evidence of impartiality.","key_machinery":"The central instrument is inspect_permute, a package that runs every answer-order permutation of each multiple-choice question (24 calls for a 4-option question) and records the presented position the model chooses on each run. It reports each vendor-by-subject cell as a chi-squared test, Cramér's V effect size, bootstrap confidence intervals, and Spearman's rank correlation between position and accuracy. The V-by-rho coordinate plane is the device that separates inactive cells from two mechanism classes: processing_load (monotone A-to-D decline, rho near -1) and content_ambiguity (final-position drop, rho near -0.4). The Goldilocks zone, the 60 to 95 percent base-accuracy window within whic","core_discovery":"The discovery is a capability-band structure for position-bias detection. In the sweep, every cell with statistically significant bias lies inside a base-accuracy band of roughly 0.60 to 0.95, and no cell above 0.95 shows a detectable signal. The detectable cells separate into two mechanism types with distinct shape signatures: a monotone accuracy decline from option A to D, attributed to processing-load pressure in weaker models, and a non-monotone drop at the final position only, attributed to content ambiguity in a narrow capability band. The paper's central interpretive claim is that an absent chi-squared signal at high accuracy is not evidence that a model is unbiased; it is evidence th","pith_inferences":["If the Goldilocks zone generalizes, the same ceiling masking should apply to other behavioral probes measured on saturated benchmarks (sycophancy, safety-rule adherence, instruction-following), so 'no signal' on an easy evaluation should never be read as model-level absence.","A direct test of the upper band: inject a known position preference into a frontier model scoring above 95 percent (e.g., by instruction to prefer option D) and run the 24-permutation diagnostic; if Cramér's V stays flat, ceiling compression is confirmed, and if it jumps, the upper edge is model- or benchmark-dependent.","The paper's lightweight git-hash pre-registration protocol is portable to any evaluation study and could become a standard way to mark the boundary between confirmatory and exploratory claims without a registered-report venue.","The two-mechanism taxonomy suggests a testable predictor: models trained with longer reasoning horizons should shift from processing_load toward content_ambiguity as their accuracy enters the mid-band, a prediction the current sweep can only hint at."],"forward_implications":["A no-bias claim on a benchmark where the model scores above roughly 95 percent is unverifiable: the correct reading is that the instrument cannot resolve the model, not that the model is impartial.","Evaluators should run a cheap calibration scout before a full sweep and skip saturated cells, saving the order-of-magnitude wall-clock cost that reasoning-trace models impose.","Single-shuffle protocols cannot separate position bias from content noise and sampling error; exhaustive permutation (or a calibrated sampling of it) is the minimum instrument for a verifiable no-bias claim.","The two mechanism signatures imply different interventions: processing-load bias points to reasoning-budget or format changes, while content-ambiguity bias points to item and option-design changes.","The cost asymmetry between vendors (7.5x average, 18.5x peak wall-clock in this sweep) is a structural barrier to independent replication, shaping who can contest vendor evaluations."],"fun_headline_variants":["60-95%: the only window to detect LLM position bias","Ceiling effects hide position bias in top models","Bias measurable only between 60 and 95% accuracy"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing assumption is that the 1,200 runs in a cell are independent, even though they are 50 questions repeated under 24 permutations of identical content; if responses within a question are correlated, the chi-squared p-values and bootstrap intervals are overconfident, and the floor edge itself is extrapolated rather than observed.","fun_headline_variants_meta":{"raw":{"variants":["60-95%: the only window to detect LLM position bias","Ceiling effects hide position bias in top models","Bias measurable only between 60 and 95% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000633,"raw_usage":{"total_tokens":2812,"prompt_tokens":853,"completion_tokens":1959,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":1904}},"tokens_in":597,"tokens_out":1959,"duration_ms":16382,"temperature":1.0,"reasoning_tokens":1904,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T09:07:30.404079+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a frontier model with base accuracy above 0.95 on a standard multitask subject, append a prompt-level instruction to prefer option D when uncertain, and run the 24-permutation diagnostic on the same 50 questions. If the test still reports Cramér's V below 0.05 with p above 0.05, ceiling-effect masking is confirmed; if it detects the injected bias, the claimed upper edge of the Goldilocks zone is not universal.","supporting_citations":[],"review_version":1}