{"id":"7e9f7219-c637-4bec-a4db-ca4aa77d4cab","arxiv_id":"2506.21362","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Counterfactual Voting Adjustment fits an answer-level logistic voting model and averages over rank and prior-vote distributions to estimate quality, yielding rankings that align better with GPT-4o helpfulness judgments than vote counts or CVP on most StackExchange communities.","lead":"This paper proposes Counterfactual Voting Adjustment (CVA), a causal model that separates answer quality from position and herding biases in online helpfulness votes. The authors test it on StackExchange data and report that CVA-based rankings align better with GPT-4o quality scores than vote-count or prior CVP rankings.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CVA's counterfactual integration is rank-invariant under Eq. (5): it cannot change the ranking produced by the fitted logistic regression, so the causal adjustment's claimed benefit is unsupported.","rationale":"The abstract's central claim is that CVA removes position and herding biases and produces quality-based rankings that better match true quality. For that claim to hold, the estimator must actually do something beyond the underlying logistic fit. The paper's own Eq. (5) shows that, under the stated population-distribution interpretation, Qhat_ij is a strictly increasing transform of q_ij + nu_i * L_ij; hence the counterfactual averaging cannot reorder answers within a question. This is an internal, checkable property, not merely a disagreement with the field's consensus. The current comparisons against voteDiff and CVP do not isolate the causal contribution, because no fixed-effects logistic regression baseline without the counterfactual step is reported. The reader identified the ignorability assumption in Eq. (1) as the weakest assumption; that is a genuine limitation and the paper concedes it, but the rank-invariance problem is more damaging because it attacks the claim that the CVA step itself is doing the work. The semi-synthetic experiment compounds the problem: it generates data from the exact model being fit, so 'recovering predefined content quality' is guaranteed by construction when optimization succeeds. The real-data evaluation using GPT-4o as ground truth for both sentiment and helpfulness is not externally validated and cannot repair this. I agree with the reject verdict, though I would weight the redundancy of the counterfactual adjustment as the more load-bearing concern. The cross-community descriptive analysis and the underlying logistic model may have value, but the causal contribution of CVA is not established.","tokens_in":20565,"tokens_out":13690,"duration_ms":184214,"concrete_test":"On any real community, fit Eq. (2) and compute both the CVA estimate Qhat_ij and the raw linear predictor r_ij = q_ij + nu_i * L_ij. Within each question, compute Kendall's tau between the CVA ranking and the r_ij ranking. If tau = 1 in every question, the counterfactual integration is rank-invariant and contributes no ranking signal; then add the r_ij ranking as a baseline in Table 2 and show whether its residuals and KT differ from CVA's. If the two rankings differ because P is answer-specific, the paper must derive the exact distribution used and demonstrate that it represents the stated top-ranked/even-vote counterfactual, which the current text does not do.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Under Eq. (5) and the model in Eq. (2), the CVA estimate is Qhat_ij = sum_t E_{tilde m,tilde d}[sigmoid(q_ij + lambda * tilde R + nu_i * L_ij + beta/(1+tilde D))], where tilde R is a function of tilde m. If P(tilde m,tilde d) is the population distribution over answers, as Appendix A states, the inner expectation is a strictly increasing function of (q_ij + nu_i * L_ij), because sigmoid is strictly increasing and the integration distribution is fixed. Therefore, within any question (nu_i constant), the CVA ranking is identical to the ranking of the fitted linear predictor from the fixed-effects logistic regression; the integration over rank and prior votes changes only the scale, not the order. This makes the central claim that CVA reranking outperforms non-causal model-based rerankings untestable as reported: no baseline consisting of the same logistic model without the counterfactual averaging is included, and such a baseline would tie CVA on every rank-correlation metric. The 'counterfactual adjustment' is not the active ingredient in the reported improvements over voteDiff or CVP; the active ingredient, if any, is adding position and herding regressors to a vote model. This also undercuts the semi-synthetic validation: the data are generated from exactly this logistic model (Appendix D), so recovering the predefined q is a consistency check, not evidence that the adjustment removes real-world biases.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Counterfactual Voting Adjustment (CVA), a causal framework for estimating response quality from binary helpfulness votes in online Q&A platforms. The framework models individual votes with a logistic regression that includes answer quality, herding (positive vote ratio), relative length, and display rank, then produces a quality estimate by integrating the fitted vote probability over a population distribution of vote histories and ranks. The authors evaluate CVA on toy examples, a semi-synthetic dataset generated from the same model, and real StackExchange data using comment sentiment and GPT-4o helpfulness scores as quality proxies, and report that CVA rankings align better with these proxies than vote-difference rankings and CVP. They also provide a cross-community analysis of position-bias and herding-bias sensitivities across 120 communities.","tokens_in":20795,"tokens_out":13571,"duration_ms":149446,"significance":"If the central claim were established, CVA would be a practical tool for debiasing helpfulness votes in large Q&A platforms, and the cross-community analysis of 120 StackExchange communities is a useful descriptive contribution. The paper's strengths include the scale of the real-world evaluation, the interpretable parametric model with explicit bias terms, and the attempt to address both position and herding biases simultaneously. However, these strengths do not compensate for the issues below; the empirical evaluation does not isolate the purported causal adjustment, the semi-synthetic validation is circular, and the key identification assumption is implausible and untested.","major_comments":[{"comment":"The CVA estimator integrates the fitted vote probability over the marginal population distribution P(m̃,d̃), not over the conditional distribution P(m̃,d̃|B). This is not the backdoor adjustment formula, and the claimed identification of counterfactual votes is not established. Furthermore, if P is the same population distribution for every answer (as suggested by 'across all questions'), then for each question the CVA estimate is a strictly increasing function of q_ij + ν_i L_ij, so the CVA ranking is identical to the ranking of the fitted linear predictor; the counterfactual integration cannot change the ranking. If P is instead answer-position-dependent, the integration can change rankings, but then the target described in Section 3 (fixed top rank, even vote split) is not what is computed. In neither case does the paper demonstrate that the causal adjustment is the active ingredient behind the improvements over voteDiff and CVP; a baseline consisting of the same logistic model without counterfactual averaging is missing.","section":"Section 3 / Appendix A, Eq. (5)"},{"comment":"The semi-synthetic experiment generates votes from exactly the model in Eq. (2) using coefficients learned from real data and then fits the same model. Under this protocol, recovering the predefined q_ij is a self-consistency check rather than independent evidence that CVA removes position or herding biases. A meaningful validation would generate data from a different process—e.g., with unobserved confounders, misspecified functional forms, or different mechanisms for rank and prior-vote assignment—to test the model's robustness.","section":"Section 5.1 / Appendix D"},{"comment":"The ignorability condition V_ij^t(g) ⊥ M_ij^t, D_ij^t | B_ij^t is load-bearing for the causal interpretation, yet B_ij^t contains only the relative length of the response, as the paper itself concedes. In StackExchange, factors such as answer author reputation, question topic, and user expertise plausibly affect both the answer's displayed rank/vote history and the individual vote, so the backdoor adjustment is likely biased. The manuscript provides no sensitivity analysis or evidence to support this assumption.","section":"Section 3, Eq. (1)"},{"comment":"The claim that the estimator in Eq. (5) is optimal (satisfying counterfactual invariance while minimizing KL divergence to the original votes) is attributed to Theorem 1 of Wang et al. (2023), but the theorem is neither stated nor proved, and the relation of that theorem to Eq. (5) is not demonstrated. As written, the optimality claim is unsupported.","section":"Appendix A"},{"comment":"The two real-data quality proxies are not validated for this purpose. Comment sentiment may inherit the same position/herding biases as votes, and GPT-4o helpfulness scores are used as ground truth without any comparison to human judgments on this task; the cited evidence (Kamalloo & Rafiei, 2023) concerns open-domain question answering, not helpfulness evaluation. These proxies are too weak to support the conclusion that CVA rankings better match true quality.","section":"Section 5.2 / Appendix F"}],"minor_comments":[{"comment":"The displayed equation after Eq. (5) uses R_ij^t and D_ij^t inside the sigmoid rather than the counterfactual R̃ and D̃; this is confusing and should be corrected.","section":"Appendix A"},{"comment":"The sentence 'We trained 4 times using the data before T each time' is unclear and does not explain how the quality estimates in Figure 3 are obtained.","section":"Section 4.1"},{"comment":"The criteria for the reported community statistics (#Questions, #Answers, #Votes, #Comments) and the significance test used for the p-values are not described; please specify the test and the unit of analysis.","section":"Table 2"},{"comment":"There is a typo 'StackExchagne' that should read 'StackExchange'.","section":"Section 5"},{"comment":"The caption contains subjective, unsupported interpretations (e.g., 'bitcoin emerges as a kind of new language') that are not corroborated by any quantitative analysis.","section":"Figure 1"},{"comment":"The code and implementation details for fitting the model and computing the integral in Eq. (5) are not provided, making the experiments difficult to reproduce.","section":"Section 3 / Reproducibility"}],"recommendation":"reject","confidential_remarks":"The manuscript's causal claims are not supported by the evidence; in particular, the semi-synthetic validation and the lack of an ablation leave the central contribution unverified. The citation of Wang et al. (2023) for the optimality claim appears inappropriate without stating the theorem. I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the CVA counterfactual adjustment is rank-invariant by the paper's own math: the estimate in Eq. (5) is a strictly increasing function of the fitted answer intercept (plus a length term), so within any question it produces exactly the same ordering as a plain fixed-effects logistic regression. The stress-test note is correct. Second, the reported improvements over CVP and voteDiff therefore come from adding the position and herding terms to the vote model, not from the causal integration step. The abstract's claim of \"accurately recovering content quality\" is not supported by the evidence as presented.\n\nCredit where it's due. The model is a clean extension of CVP, adding an explicit position-bias term beta/(1+D), and the authors fit it to a large, real dataset across 120 StackExchange communities. The descriptive map of herding versus position sensitivities in Figure 1 is genuinely useful and could inform future work on platform design. The authors also state their ignorability assumption and GPT-4o proxy openly, which is more transparent than many causal papers in this area.\n\nThe soft spots are substantial, and the reader's rejection is fair. The semi-synthetic experiment generates data from the exact logistic model being fit, so recovering the predefined quality is a self-consistency check of the optimizer, not validation of the model. The real-data evaluation compares CVA against CVP and voteDiff but never against the same logistic regression without the counterfactual averaging; on every rank-correlation metric, that baseline would tie CVA exactly, which makes the causal contribution untestable as reported. The ignorability condition in Eq. (1) conditions only on relative length of the response, and the authors concede in the limitations paragraph that unobserved confounders may exist; that concession is appropriate but undercuts any causal reading. Table 2 also shows several communities where CVA is not significantly better than CVP (philosophy, cstheory on sentiment KT, codegolf.meta on helpfulness). No code or data is released.\n\nBottom line: this deserves serious referee time because the model and the empirical study have real content, but it needs major revision. An editor should require a no-counterfactual baseline, a rewritten abstract that separates predictive value from causal claims, and ideally code or data. As it stands, I would not cite it for the causal finding, but the cross-community analysis could be worth citing if the paper is revised to be more careful.","headline":"The paper's counterfactual adjustment is rank-invariant under its own equations, so the reported ranking gains come from the added position and herding regressors, not from the causal averaging step; the large-scale descriptive study is the salvageable part.","tokens_in":21404,"tokens_out":4476,"would_cite":false,"duration_ms":53002,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Helpfulness votes carry hidden rank and crowd biases; a counterfactual adjustment recovers the answer quality underneath.","keywords":["causal inference","helpfulness voting","position bias","herding bias","counterfactual adjustment","backdoor adjustment","StackExchange","content ranking"],"falsifier":"One decisive test is a randomized field experiment on a Q&A platform: assign display ranks randomly and show or hide prior vote counts, then compare true vote rates with CVA's counterfactual predictions; if CVA systematically mispredicts votes under randomization, its adjustment is misspecified. A cheaper semi-synthetic falsifier is to generate voting data where an unobserved confounder such as author reputation drives both rank and votes; if CVA no longer recovers the predefined quality rankings, the ignorability assumption is load-bearing in exactly the way the paper concedes.","tokens_in":20200,"feed_emoji":"🗳️","tokens_out":6445,"duration_ms":62606,"temperature":0.7,"pith_summary":"Online helpfulness votes are not a clean signal of content quality: an answer displayed higher attracts more votes, and visible prior votes nudge later voters to conform. The paper proposes the Counterfactual Voting Adjustment (CVA), a causal procedure that asks what a vote would have been if the answer had appeared at a different display rank and with a balanced prior vote count, then averages over those counterfactual settings to estimate quality. On semi-synthetic data where true quality is known, CVA recovers quality rankings more accurately than raw vote-difference rankings and than the earlier Chinese Voting Process model. On 120 real StackExchange communities, reranking answers by CVA quality aligns better with comment sentiment and with GPT-4o helpfulness judgments than the platform's vote-based ranking or non-causal model rerankings. If this holds, platforms could rank content by something closer to underlying quality rather than by early popularity.","feed_headline":"Votes hide answer quality; a counterfactual fix recovers it","feed_subtitle":"Reranking StackExchange answers by adjusted votes beats raw counts on sentiment and GPT-4o quality checks.","key_machinery":"The load-bearing object is the counterfactual vote estimate $\\hat{Q}^T_{ij} = \\sum_t \\int E[V^t_{ij}(\\tilde{g}^t_{ij}) \\mid B^t_{ij}]\\, P(\\tilde{m}^t_{ij}, \\tilde{d}^t_{ij})\\, d\\tilde{m}^t_{ij}\\, d\\tilde{d}^t_{ij}$, computed from a logistic voting model with quality, herding, length, and rank terms. It does the work of a backdoor adjustment: conditioning on pre-reading features $B$ (here only relative length) and then integrating over the treatment variables (existing votes $M$ and display rank $D$) removes their influence from the quality estimate while leaving the response text fixed. The same fitted model also yields a position-bias coefficient $\\beta$ and a herding-bias odds ratio, which is how the paper compares voting dynamics across communities.","core_discovery":"The paper's central claim is that position bias and herding bias in helpfulness votes can be jointly undone without randomized experiments, so long as the voting trajectory is observed. CVA models the probability of a positive vote as a Bernoulli draw with $\\mathrm{sigmoid}(q^t_{ij} + \\lambda R^t_{ij} + \\nu_i L^t_{ij} + \\beta/(1 + D^t_{ij}))$, where $q$ is the answer's quality, $R$ the perceived positive-vote ratio, $L$ the response's relative length, and $D$ the displayed rank. It then forms the quality estimate by integrating out the existing-vote distribution and display rank, an instance of backdoor adjustment, so the estimate is invariant to presentation context while staying close to the observed voting behavior. The fitted coefficients $\\lambda$ and $\\beta$ quantify each community's herding and position sensitivity, and the quality estimates rerank answers closer to independent quality proxies than vote counts or the CVP baseline.","pith_inferences":["The authors leave implicit a direct policy test: re-estimate CVA under two different platform ranking rules; quality estimates that stay stable across rules would confirm the adjustment is removing bias rather than fitting each rule's quirks.","A natural stress test for the ignorability assumption is to inject an unobserved confounder such as author reputation or question topic into the semi-synthetic generator and check whether CVA still recovers predefined quality; this would quantify the cost of the paper's stated limitation.","Because GPT-4o serves as the real-data ground truth, a subsample validation against human expert judgments would check whether the LLM proxy itself carries biases that CVA then appears to correct.","If platforms act on CVA rankings, they change the very display positions that the model treats as exogenous, so CVA would need to be re-estimated iteratively under the new ranking policy before deployment."],"forward_implications":["Platforms that log vote trajectories, not just final vote tallies, can estimate answer quality and re-rank content without needing randomized position experiments.","Answers that received votes while ranked low, or votes against a prevailing majority, count more in CVA's quality estimate, so late and less-visible high-quality answers can surface.","The community-level coefficients $\\beta$ and herding odds provide a quantitative map of how different StackExchange communities respond to rank and social conformity.","CVA outperforms CVP especially in communities with strong position bias because it models rank and prior votes jointly rather than separately.","The framework transfers to other platforms with binary helpfulness voting and vote history, such as product or service review sites."],"supporting_citations":[{"why":"Supplies the baseline Chinese Voting Process model that CVA is compared against and the logistic vote formulation that CVA extends.","marker":"Lee et al. (2016)"},{"why":"Provides the ignorability, positivity, and stable unit treatment value assumptions that justify the counterfactual vote estimates.","marker":"Imbens & Rubin (2015)"},{"why":"Provides the backdoor adjustment criterion used to construct the CVA quality estimator.","marker":"Pearl (2010)"},{"why":"Supplies the optimality and counterfactual-invariance result that motivates the adjusted quality estimate as closest to observed votes while invariant to presentation context.","marker":"Wang et al. (2023)"},{"why":"Justifies using an LLM as a judge for quality evaluation in the real-data validation.","marker":"Chiang & Lee (2023)"},{"why":"Supports treating GPT-4o as a reliable open-domain question-answering evaluator and ground truth proxy.","marker":"Kamalloo & Rafiei (2023)"}],"fun_headline_variants":["Counterfactual vote adjustment beats raw counts on quality","Undo herding and position bias without experiments","Rerank answers by adjusted votes, not raw totals","Causal model recovers answer quality from biased votes","Vote context modeling improves fairness in online platforms"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that, after conditioning on the answer's relative length, no unmeasured factor such as author reputation or question topic affects both where an answer is displayed and how users vote it; if that premise fails, CVA's counterfactual quality estimates inherit confounding bias.","fun_headline_variants_meta":{"raw":{"variants":["Counterfactual vote adjustment beats raw counts on quality","Undo herding and position bias without experiments","Rerank answers by adjusted votes, not raw totals","Causal model recovers answer quality from biased votes","Vote context modeling improves fairness in online platforms"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000197,"raw_usage":{"total_tokens":1362,"prompt_tokens":944,"completion_tokens":418,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":343}},"tokens_in":560,"tokens_out":418,"duration_ms":4781,"temperature":1.0,"reasoning_tokens":343,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:28:18.061345+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One decisive test is a randomized field experiment on a Q&A platform: assign display ranks randomly and show or hide prior vote counts, then compare true vote rates with CVA's counterfactual predictions; if CVA systematically mispredicts votes under randomization, its adjustment is misspecified. A cheaper semi-synthetic falsifier is to generate voting data where an unobserved confounder such as author reputation drives both rank and votes; if CVA no longer recovers the predefined quality rankings, the ignorability assumption is load-bearing in exactly the way the paper concedes.","supporting_citations":[{"cited_title":"H., and Mimno, D","cited_arxiv_id":null,"evidence_quote":"Supplies the baseline Chinese Voting Process model that CVA is compared against and the logistic vote formulation that CVA extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ignorability, positivity, and stable unit treatment value assumptions that justify the counterfactual vote estimates."},{"cited_title":"Causal inference","cited_arxiv_id":null,"evidence_quote":"Provides the backdoor adjustment criterion used to construct the CVA quality estimator."},{"cited_title":"Adjusting machine learning decisions for equal opportunity and counterfactual fairness","cited_arxiv_id":null,"evidence_quote":"Supplies the optimality and counterfactual-invariance result that motivates the adjusted quality estimate as closest to observed votes while invariant to presentation context."},{"cited_title":"and Rafiei","cited_arxiv_id":null,"evidence_quote":"Supports treating GPT-4o as a reliable open-domain question-answering evaluator and ground truth proxy."}],"review_version":1}