{"id":"7bae4d8f-d63a-4a5c-abcc-49618fc3e3e4","arxiv_id":"2508.21334","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Group fairness scores in recommender systems can be much better than individual fairness scores, and group-level measures are not reliable proxies for individual-level fairness.","lead":"This paper compares group-level and individual-level fairness in recommender systems using four large language model recommenders on three datasets. It finds that a system can be fair across user groups while individual users receive very unequal recommendation quality, so group fairness scores alone can mislead.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The group-vs-individual fairness gap is largely a mathematical consequence of aggregating user scores to group means; the empirical tables may not support the 'first evidence' claim.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: comparing fairness on group-mean effectiveness to fairness on individual effectiveness is not a like-for-like comparison, and the observed gap is likely a mathematical consequence of inequality decomposition. My reading confirms this and adds that the paper's own decomposability statement for the Atkinson index already guarantees individual >= between-group, making the 'first empirical evidence' claim overstated. The paper still has value as a descriptive demonstration and a comparative study of measures, but the novelty claim needs reframing and the evaluation needs a null baseline and threshold sensitivity analysis. The reader's conditional verdict already captures these requirements, so no change in verdict is needed.","tokens_in":15706,"tokens_out":7003,"duration_ms":71738,"concrete_test":"Keep the user-level NDCG scores from GLM-4-9B (NS) on ML-1M fixed, and randomly permute users into groups of the same sizes as the intersectional groups used in Fig. 3 (1,000 draws); recompute the individual-vs-group Gini/Atk gaps for each draw. If the observed gap is within the null distribution, the headline is an aggregation artifact of the measures rather than evidence about these recommender systems; if it is outside, the recommender's pattern is empirically meaningful. Also report the same comparison at fuzzy-matching thresholds 0.5 and 0.9 to rule out threshold-driven zero-inflation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is that the paper's central finding is guaranteed by the way the fairness scores are constructed, rather than being an empirical property of recommender systems. In Sec. 3 (RQ2 and Fig. 3), group fairness is computed on group-mean NDCG while individual fairness is computed on per-user NDCG. For the measures highlighted (Gini, Atkinson, CV), averaging scores within a group is a mean-preserving contraction: it cannot increase inequality. Consequently, for any data, any recommender, and any non-trivial grouping, individual-level Gini/Atk will be greater than or equal to the between-group value whenever within-group variance is positive. The paper even states (Sec. 2, Individual fairness) that Atkind decomposes into between- and within-group fairness with no residuals, which directly implies individual >= between-group. Thus Table 2 and Fig. 3 do not provide 'first empirical evidence' that group fairness can mask individual unfairness; they instantiate a known inequality-decomposition property. The magnitude of the gap is further amplified because per-user NDCG is heavily zero-inflated from the 0.75 fuzzy-matching threshold, while group means smooth those zeros. Without a null baseline (e.g., random grouping) or a demonstration that the gap exceeds what any grouping would produce, the headline claim is not falsifiable as an empirical discovery.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies the relationship between group and individual fairness evaluation in recommender systems. Using four LLM-based recommenders, each with a sensitive and a non-sensitive prompt variant, across three datasets, the authors compute per-user and per-group effectiveness (NDCG and Precision) and then apply nine group-fairness measures and three individual-fairness measures to those base scores. They report that group fairness scores are systematically better (lower inequality) than individual fairness scores, that no individual fairness measure reliably agrees with group fairness measures in ranking models (Kendall's tau), and that within-group unfairness is close to individual unfairness. The paper concludes that group fairness can mask individual unfairness and presents this as the first empirical evidence of the disjointness of the two fairness concepts.","tokens_in":16152,"tokens_out":4038,"duration_ms":38075,"significance":"If the empirical claims were properly grounded, the paper would be a useful practical warning for recommender-system practitioners and a systematic comparison of fairness measures under a common base score. The manuscript's strengths include public code, a wide coverage of the group-fairness measure space, and the explicit computation of between- versus within-group decompositions. However, the headline conclusion is largely predetermined by the measurement construction for the decomposable inequality measures, and the ranking-agreement analysis is statistically underpowered, so the significance of the results as stated is currently not established.","major_comments":[{"comment":"The result that individual Gini, Atkinson, and CV scores are always worse than the corresponding between-group scores is a mathematical consequence of computing the measures on per-user scores rather than group means. For any partition of users, averaging within groups is a mean-preserving contraction, so the between-group component of a decomposable inequality measure cannot exceed the total inequality; the paper itself notes in Section 2 that Atkind decomposes without residuals into between- and within-group components. The manuscript therefore cannot claim, as it does in the Discussion, that the observed gap is 'empirical evidence' of disjointness. The claim should be reframed as an illustration of a known decomposition property, and the magnitude of the gap should be compared against a null baseline (e.g., random grouping, or a baseline derived from the within-group variance) to establish that the observed values are informative.","section":"Section 3 (RQ2, Fig. 3) and Section 2 (Individual fairness)"},{"comment":"Kendall's tau is computed from only 8 runs (4 LLMs × 2 prompt types) without repeated data splits or confidence intervals. For n=8, the 95% confidence interval of Kendall's tau is approximately ±0.4, so the conclusion that 'no individual Fair measure consistently has equivalent rankings' is not supported by the reported precision. The authors should report bootstrap or permutation intervals, or increase the number of runs, before drawing the 'no reliable proxy' conclusion.","section":"Section 3 (RQ1, Fig. 2)"},{"comment":"The only relevance signal is fuzzy string matching at a fixed TF-IDF n-gram threshold of 0.75, and per-user NDCG is heavily zero-inflated at this threshold, which mechanically inflates individual inequality relative to group means. A sensitivity analysis across thresholds (e.g., 0.6, 0.7, 0.8) is required to verify that the qualitative gap between group and individual fairness is not an artifact of the matching threshold.","section":"Section 2 (Evaluation, LLM matching threshold)"}],"minor_comments":[{"comment":"The caption states that 'the figures should not be used to quantify the gap between two families of measures,' which appears to contradict the text in Section 3 stating that individual Fair scores are 'much less fair' than group scores; please clarify what conclusions the reader may legitimately draw from the figure.","section":"Fig. 3 caption"},{"comment":"The phrase '8 runs across 3 datasets' is ambiguous; the runs come from four LLMs each used with a sensitive and a non-sensitive prompt, so please phrase it as '4 LLMs with sensitive and non-sensitive prompts across 3 datasets.'","section":"Abstract"},{"comment":"The main text does not specify the inequality-aversion parameter ε used for the Atkinson index; please state it in the main text or point to the exact appendix equation so the decomposition can be reproduced.","section":"Section 2 (Atkinson index)"},{"comment":"The statement that similar results for collaborative-filtering recommenders are omitted for brevity should be supported by placing the full results in the repository or an appendix; otherwise the abstract's general claim about recommender systems is too broad for the evidence presented.","section":"Footnote 10"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for RecSys and the authors have compiled a broad set of fairness measures, but the central novelty claim is at risk because the direction of the group-versus-individual gap is mathematically guaranteed for the decomposable inequality measures highlighted in the paper. I recommend the editor require a null baseline, a sensitivity analysis of the matching threshold, and uncertainty quantification for the rank correlations before considering publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper does a genuine service: it applies nine group fairness measures and three individual measures to LLM recommenders across three datasets, with code and a clean measurement pipeline. The ranking-agreement analysis (RQ1) is new and useful—showing that no individual measure is a reliable proxy for group measures in these runs. The within-group/between-group decomposition (RQ3/RQ4) is also worth reading.\n\nThe soft spot is the headline. The paper claims \"first empirical evidence\" that group fairness can mask individual unfairness, and points to Table 2 and Figure 3. But for the measures involved—Gini, Atkinson, CV, SD—this gap is largely a mathematical consequence of the aggregation step. When you compute a spread measure on group means versus on individual scores, the group-mean version cannot be more unequal than the individual version, because averaging removes within-group variance. The paper itself cites the Atkinson decomposition (no residuals between individual and between+within), which already implies individual >= between-group. So the evidence tables are instantiations of known inequality-decomposition properties, not empirical discoveries about recommender systems. The descriptive claim is still fair: yes, at k=10 for these runs, group scores look better than individual scores. But the novelty is overstated.\n\nThe evaluation also has some softness: 8 LLM runs, no error bars or repeated splits, Kendall's tau over only 8 points, and no sensitivity analysis for the 0.75 item-matching threshold. A null baseline (e.g., random grouping, or all users in one group) would help show whether the observed gap exceeds what any grouping would produce. These are fixable with moderate effort.\n\nThe paper is honest in that it states the Atkinson decomposition explicitly, so the flaw is not concealed. But the interpretation needs to change: this is a measurement study showing how measures behave under aggregation, not a novel empirical law.\n\nI'd send it to peer review—the comparison and the code are worth refereeing, and the authors can address the interpretation with revisions. It's a conditional accept at best.","headline":"An honest, systematic comparison of group vs individual fairness measures, but the headline result is largely a built-in property of the measures, not an empirical discovery.","tokens_in":16482,"tokens_out":2774,"would_cite":false,"duration_ms":27238,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Group-fair recommendations can be very unfair to individuals.","keywords":["group fairness","individual fairness","fairness evaluation","recommender systems","LLM-based recommenders","intersectionality","inequality measures","Kendall's tau"],"falsifier":"Run a broader sweep of recommender systems, say twenty or more covering both collaborative filtering and LLM-based models, on the same three datasets, and compute the paper's group and individual Gini scores on NDCG@10 together with Kendall's tau at the 0.9 equivalence threshold; if any individual fairness measure consistently agrees with a group fairness measure, or if individual fairness scores are frequently no worse than group fairness scores, the paper's central claim would be contradicted.","tokens_in":15517,"feed_emoji":"⚖️","tokens_out":12910,"duration_ms":106209,"temperature":0.7,"pith_summary":"This paper tries to establish that group fairness and individual fairness in recommender systems are not interchangeable, and that between-group fairness scores can mask large per-user unfairness. It does so by applying the same inequality measures — SD, Gini, and Atkinson — to group-mean effectiveness and to individual-user effectiveness, which makes the two fairness notions directly comparable on a common scale. Across eight runs of four LLM-based recommenders and three datasets, the authors find that fairness scores computed for individuals are always worse than group fairness scores, and no individual fairness measure reliably reproduces the ranking of systems given by a group fairness measure. The finding matters because standard practice evaluates only between-group fairness, so a system can be certified as fair while serving some users far worse than others.","feed_headline":"Group-fair recommendations can be very unfair to individuals","feed_subtitle":"Across eight models and three datasets, group-fairness scores hide large per-user quality gaps.","key_machinery":"The central machinery is a two-step evaluation pipeline: first compute an effectiveness base score per user (NDCG at cutoff 10), then apply the same inequality measures — standard deviation, Gini, and Atkinson — either to the group-mean base scores for group fairness or to the per-user base scores for individual fairness. The Atkinson index is load-bearing because it is additively decomposable into between-group and within-group components without residual, which lets the paper trace how individual unfairness splits into a between-group part and a within-group part. This decomposition produces the staircase-like pattern reported in the results — between-group unfairness stays low while within-group and individual unfairness are high — which is the mechanism by which group fairness hides individual unfairness. The paper also uses Kendall's tau with an equivalence threshold of 0.9 to test whether any individual measure could stand in as a proxy for a group measure.","core_discovery":"The paper's central claim is that, measured with the same families of fairness measures on the same effectiveness scores, group fairness and individual fairness are empirically disjoint: a recommender can be highly fair between groups and very unfair to individual users, because group-level scores average away within-group variation. The authors demonstrate this by computing NDCG at cutoff 10 for eight runs of four open-weight LLM recommenders on three datasets, then applying ten between-group fairness measures and three individual fairness measures to those effectiveness scores. They find that individual fairness scores are consistently worse than between-group fairness scores; that within-group unfairness is consistently worse than between-group unfairness and tracks individual unfairness; and that no individual fairness measure ranks systems equivalently to any group fairness measure under Kendall's tau at the 0.9 equivalence threshold. The only group measure showing moderate agreement with individual Gini and Atkinson is the coefficient of variation. The conclusion is that between-group fairness evaluation alone can certify systems that serve some users far worse than others, so evaluation should report between-group, within-group, and individual fairness together.","pith_inferences":["A practical audit rule follows: for any recommender, decompose per-user NDCG inequality with the Atkinson index into between- and within-group parts; if the within-group component dominates, the group fairness score is nearly uninformative about user-level fairness. This is an extension the paper does not itself propose as a tool.","The same aggregation mechanism plausibly extends to item or provider fairness, where effectiveness or exposure is averaged per provider group; the paper anticipates this but provides no empirical test, so measuring provider-side individual unfairness under the same decomposition is a natural next experiment.","A synthetic control study could separate the mathematical necessity from the empirical discovery: generate per-user effectiveness values with controlled within-group variance, and confirm that the gap between group and individual fairness scores is exactly the within-group variance captured by the decomposition, which would show whether the LLM recommenders add anything beyond the general inequali","Future mitigation work could add a per-user inequality penalty, such as a Gini term on per-user NDCG, to the loss function; the paper's decomposition suggests such a penalty would primarily reduce within-group unfairness, and whether that also improves or harms between-group fairness is a testable question."],"forward_implications":["Practitioners who evaluate only between-group fairness may certify systems that are markedly less fair to individual users; the paper shows this occurring for LLM-based recommenders on all three datasets.","No existing individual fairness measure can serve as a reliable proxy for a group fairness measure, so evaluation should include both group and individual scores rather than assuming one implies the other.","Fairness scores worsen as more sensitive attributes form intersectional groups, so the number of groups itself affects how fair a system appears.","Within-group unfairness is consistently larger than between-group unfairness and comparable to individual unfairness, meaning that optimizing between-group fairness alone leaves the largest source of unfairness unaddressed.","Using the same measure families for group and individual fairness makes the two scores directly comparable, which previous work could not do because different measure families have different sensitivities and ranges."],"supporting_citations":[{"why":"Surveys the fairness landscape and shows prior work evaluates group or individual fairness separately, establishing the gap the paper fills.","marker":"[11]"},{"why":"Provides the definition of group fairness as similar effectiveness between user groups.","marker":"[14]"},{"why":"Provides the definition of individual fairness as similar effectiveness for all users and an example of differing evaluation objectives.","marker":"[63]"},{"why":"Supplies the Gini index, the central inequality measure applied to both group-mean and individual effectiveness.","marker":"[19]"},{"why":"Supplies the Atkinson index, which enables the between- and within-group decomposition.","marker":"[2]"},{"why":"Establishes that the Atkinson index is additively decomposable into between- and within-group components without residual.","marker":"[4]"},{"why":"Supplies the Kendall's tau equivalence threshold of 0.9 used to test whether measures can serve as proxies.","marker":"[37]"},{"why":"Supplies the individual-level standard deviation measure used for individual fairness.","marker":"[43]"},{"why":"Argues fairness measures from different families differ in sensitivity and ranges, motivating the same-family comparison.","marker":"[46]"}],"fun_headline_variants":["Group fairness masks severe individual unfairness","Fair for groups, unfair for individuals in recommenders","Group-fair ranking can hide per-user unfairness","Individual and group fairness rarely align in ranking"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that comparing fairness scores computed on group-mean effectiveness with fairness scores computed on individual-user effectiveness is meaningful, rather than an artifact of aggregating different numbers of units.","fun_headline_variants_meta":{"raw":{"variants":["Group fairness masks severe individual unfairness","Fair for groups, unfair for individuals in recommenders","Group-fair ranking can hide per-user unfairness","Individual and group fairness rarely align in ranking"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000231,"raw_usage":{"total_tokens":1467,"prompt_tokens":909,"completion_tokens":558,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":500}},"tokens_in":525,"tokens_out":558,"duration_ms":5148,"temperature":1.0,"reasoning_tokens":500,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:41:15.440310+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a broader sweep of recommender systems, say twenty or more covering both collaborative filtering and LLM-based models, on the same three datasets, and compute the paper's group and individual Gini scores on NDCG@10 together with Kendall's tau at the 0.9 equivalence threshold; if any individual fairness measure consistently agrees with a group fairness measure, or if individual fairness scores are frequently no worse than group fairness scores, the paper's central claim would be contradicted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the definition of group fairness as similar effectiveness between user groups."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Gini index, the central inequality measure applied to both group-mean and individual effectiveness."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Atkinson index, which enables the between- and within-group decomposition."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes that the Atkinson index is additively decomposable into between- and within-group components without residual."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Kendall's tau equivalence threshold of 0.9 used to test whether measures can serve as proxies."}],"review_version":1}