{"id":"f0a25f0e-e8b4-404f-bff9-8c1ac99922e6","arxiv_id":"2411.11437","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Reviewer slates that are topically diverse, seniority-diverse, or from distinct publication networks produce broader or less redundant reviews, while geographic and organizational diversity show little or no effect.","lead":"This paper tests whether assigning reviewers from different organizations, countries, seniority levels, topics, or publication networks changes how much of a paper the reviews cover and how much they repeat each other. Using thousands of ICML 2020 reviews, it finds that some forms of diversity help coverage or reduce redundancy, while geographic diversity does neither.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The within-paper differencing in Eq. 3 does not actually cancel the anchor reviewer because the coverage and redundancy measures in Appendix A are non-additive pair functions; the core causal estimates may absorb anchor-specific text style.","rationale":"The reader's weakest-assumption statement identified conditional ignorability and zero-imputation as the main risks, and those are genuine. My stress-test points to a more specific and more internal threat: the identification argument in Eq. 3 assumes the anchor reviewer cancels from the differenced outcome, but the outcome measures defined in Appendix A are non-additive functions of the two review texts. The anchor does not cancel because coverage uses union/max operations and redundancy uses intersection/all-pair similarities. This is not an external confounder that sensitivity analysis would have to guess at; it is a structural mismatch between the estimating equation and the outcome definitions. If the anchor's review style or abstract overlap affects how much the paired reviewer's diversity can move the outcome, the coefficients in Table 1 are not clean estimates of the causal effect, even under full conditional ignorability. The proposed anchor-permutation and interaction tests can settle this empirically. I also note the factor-of-two error in Eq. 3, which the reader flagged: since δ is coded as +1/-1, subtracting Eq. 2b from Eq. 2a yields 2γd*, not γd*. That error scales all reported effect sizes but does not change signs or p-values, so it is secondary. The qualitative pattern in Table 1 might survive, but the causal magnitudes and policy recommendations should not be taken at face value until the additivity assumption is checked. Because the reader already returned a CONDITIONAL verdict, my read does not change that verdict, so I mark it UNCHANGED.","tokens_in":20116,"tokens_out":11407,"duration_ms":136123,"concrete_test":"Use the same ICML 2020 data to test the additivity assumption behind Eq. 3. For papers with at least four reviewers, recompute the main diversity coefficients separately for each possible anchor reviewer in the same paper (the reviewer common to the diverse and non-diverse pair). Under exact anchor cancellation, estimates should agree up to sampling noise. Then stratify these anchor-specific estimates by the anchor review's own lexical/semantic overlap with the abstract and by review length; if the estimates vary with anchor overlap or length, the outcome measures are non-additive and the within-paper difference does not remove the anchor's contribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing step is the anchor-cancellation argument in Section 5.5. Equation 3 is obtained by subtracting pair regressions that share reviewer r1, and the text claims that r1's profile, expertise, and review 'do not contribute to the difference,' leaving only r2 and r3 terms. This is true only if the outcome y(r1,r2;S) is additively separable in the two reviewers' contributions. The outcome definitions in Appendix A are not additive. Lexical coverage (Eq. 4) is the fraction of abstract n-grams covered by the union of the two reviews; semantic coverage (Eq. 5) is a max over sentences of the concatenated reviews; lexical redundancy (Eq. 6) is the intersection of n-grams between R1 and R2; semantic redundancy (Eq. 7) and weighted semantic redundancy (Eq. 8) use max or all-pair similarities between the review texts. In all of these, R1 enters through union, intersection, max, or pairwise products, so it does not cancel by subtraction. For example, if the anchor review already covers every abstract n-gram, the lexical-coverage difference between a diverse and a non-diverse pair is zero regardless of the second reviewer; if the anchor has no overlap, the same difference is driven entirely by r2 vs r3. Because anchor reviewers are not randomly chosen and their writing style is not included as a control, the estimated γd* in Table 1 can reflect an interaction between anchor style and pair composition rather than a causal effect of diversity. The separate factor-of-two mislabeling in Eq. 3 (δ=+1 versus δ=-1 gives 2γd*, not γd*) is a scaling issue; even after correcting it, the anchor-cancellation problem remains and is not addressed by the paper's checks.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether reviewer-slate diversity along five axes (organization, geography, seniority, topic, and co-authorship) causally affects two group-level review outcomes: coverage of the paper and review criteria, and redundancy across reviews. Using ICML 2020 data with nearly 5,000 submissions, the authors propose lexical, semantic, and type-based outcome measures, then estimate effects with a within-paper differencing regression and, as robustness, a propensity-score-matched permutation analysis. They report that co-authorship and seniority diversity increase type coverage, topical diversity increases paper coverage, and organizational, seniority, topical, and co-authorship diversity reduce redundancy, while geographical diversity has no significant effect. The paper concludes with policy recommendations for reviewer assignment systems. The central claim is causal: that the estimated diversity coefficients in Table 1 can guide assignment policies to improve review utility.","tokens_in":20454,"tokens_out":9715,"duration_ms":128633,"significance":"If the estimates were valid, the work would be a valuable and actionable contribution. It operationalizes two understudied group-level review properties, examines five diversity axes on a large real conference dataset, includes human validation for some semantic measures, applies multiple-testing correction, and provides a non-parametric robustness check. The paper is also unusually clear in translating its findings into concrete assignment-policy recommendations. However, the main estimand is compromised by a non-additivity problem in the outcome measures, and the reported numerical results appear to contain a factor-of-two error; these issues affect the central causal claims and require a substantive reanalysis before the conclusions can be relied upon.","major_comments":[{"comment":"The anchor-cancellation argument is invalid for the outcome definitions used in the paper. In Eq. (3) the paper subtracts y(r1,r2;S) - y(r1,r3;S) and claims that the common reviewer r1 does not contribute to the difference. That is true for the linear profile and expertise covariates, but the outcome y itself is not additively separable in R1 and R2: lexical coverage (Eq. 4) uses the union of n-grams over the concatenated reviews, semantic coverage (Eq. 5) takes a max over the concatenated review set, lexical redundancy (Eq. 6) is the intersection of R1 and R2, and semantic redundancy (Eqs. 7-8) involves max or all-pair similarities across the two texts. Thus R1 still enters the difference through union, intersection, max, and pairwise product operations. For example, if the anchor review already covers every abstract n-gram, the lexical-coverage difference between a diverse and a non-diverse pair is zero regardless of the second reviewer; if the anchor has no overlap, the same difference is driven entirely by r2 versus r3. Because anchors are not randomly chosen and anchor writing style is not controlled, the gamma estimates in Table 1 can reflect an interaction between anchor style and pair composition rather than a causal effect of diversity. The non-parametric difference in Appendix D inherits the same problem.","section":"Section 5.5, Eq. (3); Appendix A, Eqs. (4)-(8)"},{"comment":"The algebra of the within-paper difference drops a factor of two. With delta_d*(r1,r2)=1 and delta_d*(r1,r3)=-1, subtracting Eq. (2b) from Eq. (2a) gives y(r1,r2;S) - y(r1,r3;S) approximately equal to 2*gamma_d* plus the covariate-difference terms, not gamma_d* as written in Eq. (3). Unless the model is reparameterized so that the coefficient in Eq. (3) is gamma_d*/2, the estimates reported in Table 1 are twice the gamma_d* defined in Eq. (1). The authors should state which convention was used and correct either the equations or the reported effect sizes.","section":"Section 5.5, Eqs. (2a), (2b), and (3)"},{"comment":"The human validation undermines one of the outcome measures used as evidence. Appendix C reports that lexical redundancy agreed with human annotators only at chance level (p=0.623), yet Table 1 and Section 6.2 treat significant lexical-redundancy effects for organizational, seniority, topical, and co-authorship diversity as support for Hypothesis 2. In addition, the coverage measures used for Hypothesis 1 are not included in the human annotation study at all, despite Section 5.2 saying that the automated outcome measures were validated. The paper should either restrict its evidentiary claims to the human-validated semantic measures or provide separate validation for lexical redundancy and the coverage measures before using them as the basis for policy recommendations.","section":"Appendix C; Table 1; Section 6.2"},{"comment":"The causal reading of gamma_d* requires conditional ignorability after controlling for submission, reviewer profiles, and TPMS expertise, but this assumption is not probed. Review length, review effort, and the optimization constraints of the actual assignment algorithm are plausible unmeasured confounders that could correlate with both pair diversity and coverage/redundancy, and no sensitivity analysis addresses them. In addition, the treatment of missing profile data (51.0% missing Semantic Scholar profiles and 17.5% missing Google Scholar profiles) as zeros is asserted in Section 5.4 without a missingness analysis or complete-case robustness check. These gaps should be addressed before the results can support the paper's causal language and policy conclusions.","section":"Section 5.5; Figure 1; Section 5.3"}],"minor_comments":[{"comment":"Table 1 reports only point estimates and significance stars; standard errors or confidence intervals should be included, especially because many significant effect sizes are small in magnitude.","section":"Table 1"},{"comment":"The phrase 'without loss of generality' is not accurate for selecting an anchor reviewer and a diverse/non-diverse pair among papers with three or four reviews; the paper should specify how anchors and pairs are selected and whether estimates are averaged over all valid pairings.","section":"Section 5.5"},{"comment":"The co-authorship diversity measure should define 'co-authorship distance' explicitly, for example as the shortest path in the co-author graph, and the treatment of missing profile vectors as delta=0 should be stated more precisely than 'ensures a consistent estimation.'","section":"Section 5.4"},{"comment":"The normalization to [0,1] described in Appendix B is not shown as a formula; provide the exact transformation for each outcome, especially for the non-symmetric weighted semantic redundancy in Eq. (8).","section":"Appendix A/B"},{"comment":"For type coverage, the paper should state whether the DistilBERT aspect and argument classifiers were applied to individual review sentences or to the concatenated review text, and how the reported 90.11% and 82.96% accuracies were measured.","section":"Section 5.2.1"}],"recommendation":"major_revision","confidential_remarks":"The main issue is technical rather than editorial: the anchor-cancellation argument fails for the non-additive outcome definitions, so the central estimates in Table 1 require re-estimation, and the factor-of-two discrepancy means the numerical results need careful checking after reanalysis. I see no circularity concern, since treatments and outcomes are measured independently, but the causal interpretation needs stronger support through sensitivity analysis. This is not an accept-level manuscript in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper asks the right question—whether diversity of assigned reviewer slates changes the coverage and redundancy of the review text—and it brings real data to it: roughly 5k ICML 2020 submissions, pre-rebuttal reviews, reviewer profiles, and TPMS expertise scores. The null result for geographical diversity and the consistent co-authorship effect are genuinely new and useful for assignment algorithms. The non-parametric propensity-matched analysis is a good complement to the regressions, and the human validation of the semantic measures is more than this literature usually does.\n\nThe problem is the parametric identification. Section 5.5's differencing is supposed to remove the anchor reviewer r1 by subtracting the two pair regressions. That only works if the outcome is additively separable in the two reviewers' contributions. It isn't. Every outcome in Appendix A is a union, intersection, max, or all-pairs product involving both review texts. Lexical coverage is the fraction of abstract n-grams covered by the concatenated reviews; semantic coverage is a max over the concatenation; lexical redundancy is the intersection; semantic redundancy is a symmetric max; weighted redundancy is an all-pairs sum. In all of those, r1 enters nonlinearly, so y(r1,r2) - y(r1,r3) does not cancel r1. The estimate can then absorb an interaction between the anchor's writing style and pair composition, which is exactly the confounder the design was meant to remove. On top of that, Eq. 3 drops a factor of two: subtracting Eq. 2a from 2b gives 2γd*, not γd*. The signs might be right, but the effect sizes, and the causal interpretation, are on shaky ground as written.\n\nTwo smaller things: Table 1 reports no standard errors or confidence intervals, so we can't judge precision. And lexical redundancy failed human validation (random agreement, p=0.623) yet is still reported as evidence; the paper's own appendix says only the semantic measures are valid.\n\nI wouldn't dismiss the paper. The question matters, the empirical design is mostly thoughtful, and the non-parametric results give some independent support for the qualitative findings. But the central causal claims need a corrected estimator that deals with the non-additive outcomes—perhaps by modeling the pair outcome directly with the anchor's text as a control rather than relying on differencing—plus uncertainty quantification and a decision on lexical redundancy. That's real referee work, but it's worth doing.","headline":"A real new empirical question with a useful null result, but the within-paper differencing doesn't cancel the anchor reviewer and the effect sizes are not credible as written.","tokens_in":21010,"tokens_out":3711,"would_cite":false,"duration_ms":36154,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Assigning reviewers who differ in co-authorship, seniority, or topic changes what the collective reviews cover and how much they repeat, while geographical diversity does neither.","keywords":["peer review","reviewer assignment","reviewer diversity","review coverage","review redundancy","causal inference","observational study","group decision-making"],"falsifier":"A randomized assignment experiment would settle the claim: for the same set of papers, randomly assign reviewer pairs that are matched on TPMS expertise and profile but differ only in co-authorship or topical diversity, then compare coverage and redundancy; if the differences vanish, the observational effects are confounded. Short of that, a sensitivity analysis introducing an unmeasured confounder (for example, review length or time spent) into Equation 3 would reveal whether the estimated $\\gamma$ values survive plausible confounding.","tokens_in":19914,"feed_emoji":"📝","tokens_out":5820,"duration_ms":53730,"temperature":0.7,"pith_summary":"This paper asks whether the mix of reviewers assigned to a paper changes what the reviews, taken together, cover and whether they repeat one another. Using roughly 5,000 submissions from a machine-learning conference, it treats five axes of reviewer diversity—organization, geography, seniority, topic, and co-authorship—as treatments and measures two group-level review outcomes: coverage (of the paper's content and of review criteria) and redundancy (overlap between reviews). The central finding is that the axes act differently: co-authorship and seniority diversity raise coverage of review types, topical diversity raises coverage of the paper's content, and all axes except geography reduce redundancy. If the causal reading holds, program chairs can nudge review utility by adding specific diversity constraints to assignment algorithms without sacrificing reviewer expertise.","feed_headline":"Reviewer diversity changes what peer reviews cover—and repeat","feed_subtitle":"Causal estimates from 5,000 submissions show which diversity axes broaden reviews—and which only cut redundancy.","key_machinery":"The load-bearing device is within-paper pair differencing. Each submission is reviewed by three or four reviewers, so the analysis builds triples $(r_1, r_2, r_3)$ in which $(r_1, r_2)$ is diverse and $(r_1, r_3)$ is non-diverse along a target dimension $d^*$. Subtracting the two linear outcome equations removes the submission term $\\omega_S$, so paper content and quality drop out of the comparison; the remaining regression estimates $\\gamma_{d^*}$ with controls for the other diversity dimensions, each reviewer's profile vector, and the TPMS expertise scores $E(r, S)$. The same logic is applied non-parametrically by propensity-score matching diverse and non-diverse pairs within a paper.","core_discovery":"The paper's central claim is that reviewer-slate diversity has causal, dimension-specific effects on review coverage and redundancy. In the parametric analysis, co-authorship diversity increases argument- and aspect-type coverage (effects 0.0064 and 0.0059) and seniority diversity increases the same (0.0058 and 0.0074), while topical diversity increases lexical and semantic paper coverage (0.1098 and 0.0929). Lexical and semantic redundancy decrease under organizational (−0.0218, −0.0078), co-authorship (−0.0258, −0.0097), topical (−0.0524, −0.0290), and seniority (−0.0061, −0.0015) diversity; geography shows no significant effect on any outcome. Co-authorship diversity is the only axis that also lowers redundancy within a single review criterion (weighted semantic redundancy). The paper reports that non-parametric propensity-score matching corroborates the significant parametric results.","pith_inferences":["The within-paper differencing design transfers to any conference dataset with three or more reviewers per paper, reviewer profiles, and assignment-expertise scores; the causal contrast does not require outcome data from re-reviewing the same paper.","If geographical diversity truly has no effect on coverage or redundancy, venues that add geography constraints to assignments for other reasons (for example, reducing collusion) should treat those constraints as orthogonal to review breadth—the paper's evidence does not speak to collusion outcomes.","A useful next experiment would separate 'co-authors' from 'shared co-author' pairs: the paper's co-authorship diversity collapses both into the non-diverse category, so the active mechanism could be direct collaboration rather than network proximity."],"forward_implications":["Reviewer assignment systems can target co-authorship diversity (no shared co-authors) to increase type coverage and lower redundancy, including within individual review criteria.","Topical diversity is the lever for covering more of the paper's content itself, while it does not raise coverage of review criteria.","Organizational and seniority diversity reduce redundancy but do not by themselves raise paper coverage, so the choice of diversity axis depends on the desired outcome.","Geographical diversity shows no effect on coverage or redundancy, so geography-based assignment constraints cannot be justified by review-utility gains.","The effects are not explained by lower review quality: diversity shows no significant correlation with meta-reviewers' ratings of review quality."],"supporting_citations":[{"why":"Supplies the TPMS expertise scores used to control for reviewer expertise in the causal model.","marker":"Charlin and Zemel, 2013"},{"why":"Provides the aspect typology (Summary, Soundness, Clarity, etc.) used to measure type coverage.","marker":"Yuan et al., 2022"},{"why":"Provides the argument typology (Evaluation, Fact, Request, Reference, Quote) used to measure type coverage.","marker":"Hua et al., 2019"},{"why":"Provides Sentence-BERT embeddings used to compute semantic paper coverage and semantic redundancy.","marker":"Reimers and Gurevych, 2019"},{"why":"Supplies the false-discovery-rate correction applied to all significance tests.","marker":"Benjamini and Hochberg, 1995"},{"why":"Supplies propensity-score matching used in the non-parametric robustness analysis.","marker":"Rosenbaum and Rubin, 1983"},{"why":"Precedent for the linear observational modeling approach in peer-review settings.","marker":"Tomkins et al., 2017"},{"why":"Supplies LDA topic modeling used to build reviewer topic vectors, the basis of topical diversity.","marker":"Blei et al., 2003"}],"fun_headline_variants":["Group diversity shapes peer review: coverage up, redundancy down","Not all reviewer diversity boosts coverage; some only trim redundancy","Causal study: which reviewer mixes improve coverage and cut redundancy","5,000 submissions reveal diversity's split effect on review quality","Diverse reviewers: wider net, less repetition—but not for all types"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"After differencing within a paper and controlling for reviewer profiles and TPMS expertise, the paper assumes the only systematic reason diverse and non-diverse pairs differ in coverage or redundancy is the diversity itself—an untested assumption that review length, review effort, or hidden assignment constraints are not also driving the differences.","fun_headline_variants_meta":{"raw":{"variants":["Group diversity shapes peer review: coverage up, redundancy down","Not all reviewer diversity boosts coverage; some only trim redundancy","Causal study: which reviewer mixes improve coverage and cut redundancy","5,000 submissions reveal diversity's split effect on review quality","Diverse reviewers: wider net, less repetition—but not for all types"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000272,"raw_usage":{"total_tokens":1680,"prompt_tokens":1039,"completion_tokens":641,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":655,"completion_tokens_details":{"reasoning_tokens":554}},"tokens_in":655,"tokens_out":641,"duration_ms":7165,"temperature":1.0,"reasoning_tokens":554,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:31:11.955364+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A randomized assignment experiment would settle the claim: for the same set of papers, randomly assign reviewer pairs that are matched on TPMS expertise and profile but differ only in co-authorship or topical diversity, then compare coverage and redundancy; if the differences vanish, the observational effects are confounded. Short of that, a sensitivity analysis introducing an unmeasured confounder (for example, review length or time spent) into Equation 3 would reveal whether the estimated $\\gamma$ values survive plausible confounding.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the TPMS expertise scores used to control for reviewer expertise in the causal model."},{"cited_title":"https://aclanthology.org/N19-1219","cited_arxiv_id":null,"evidence_quote":"Provides the argument typology (Evaluation, Fact, Request, Reference, Quote) used to measure type coverage."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides Sentence-BERT embeddings used to compute semantic paper coverage and semantic redundancy."}],"review_version":1}