{"id":"2b4de107-9fd1-4633-9e0e-837a2c89528c","arxiv_id":"2501.04316","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"LLM-based hiring pipelines show notable demographic sensitivity in resume ranking and summarization, with sensitivity to name changes comparable to non-demographic changes.","lead":"The paper tests whether large language models treat resumes differently based on the name and background signals of the candidate, in both summarizing and ranking tasks. It finds that LLM hiring tools, especially retrieval systems, often change which candidates are selected after small demographic changes, suggesting fair hiring with these tools is not yet reliable.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Retrieval-stage 'bias' may be generic brittleness: exclusion asymmetries lack significance tests, so the central claim's primary evidence is not yet established.","rationale":"The reader's weakest-assumption critique of the summarization proxies is reasonable: Appendix A.9 validates that the automated measures detect differences along the three manipulated characteristics, but does not show that demographic-induced differences in reading ease, polarity, or regard would change an HR decision. I do not see this as the most load-bearing issue for the paper's central claim, because the headline conclusion is that bias is 'primarily stemming from the retrieval stage,' and the retrieval analysis does not depend on those proxies. The more decisive soft spot is in the retrieval evidence itself. The exclusion metric defines a fairness violation as any top-n rank change after a demographic perturbation, yet the paper's own control experiments show that non-demographic perturbations, such as within-group names, typos, and spacing, cause comparable exclusion rates. The authors are transparent about this and frame the result as possible brittleness, but the central claim of 'considerable bias' then rests on the demographic-vs-baseline comparisons and on directional asymmetries (e.g., W→B more sensitive than B→W). Those asymmetries are presented without uncertainty quantification. Missing confidence intervals and significance tests are not merely a presentational issue: with 154 job postings and many resumes per posting, aggregate point estimates could be driven by a few outliers. A bootstrap or permutation check would settle whether the asymmetry is real. Without it, the strongest claim that retrieval 'leads to discriminatory outcomes' is conditional. I would keep the verdict CONDITIONAL, with the added condition that the retrieval asymmetries be shown statistically significant and larger than within-group name baselines.","tokens_in":18201,"tokens_out":6448,"duration_ms":70505,"concrete_test":"For each retrieval model and n in {5, 10, 100}, bootstrap the exclusion metric and the W→B vs B→W and M→F vs F→M deltas over job postings or resumes (10,000 resamples) and compute 95% percentile confidence intervals. Then run a paired permutation test comparing each demographic perturbation's exclusion to the within-group name baseline on the same resumes. If any model's directional delta confidence interval includes zero, or if demographic exclusion is not significantly larger than within-group exclusion, the claim that retrieval exhibits demographic bias rather than generic brittleness is unsupported for that model.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that an LLM hiring pipeline 'exhibits considerable bias, primarily stemming from the retrieval stage.' The load-bearing evidence is the exclusion metric and its directional asymmetries (Section 4.2.2). However, exclusion is defined as any top-n rank change after demographic perturbation, and the paper's own non-demographic baselines (Section 4.2.2, Figure 7) show that within-group name changes, typos, and spacing changes produce exclusion rates of similar magnitude. What would distinguish demographic bias from general brittleness is the directional asymmetry (e.g., W→B > B→W for most models, and gender asymmetries in the opposite direction). These asymmetries are reported as point estimates aggregated across job postings and resumes, with no confidence intervals, standard errors, or significance tests. If the asymmetries are within noise, the retrieval stage demonstrates sensitivity to small input changes but not 'considerable bias... leading to discriminatory outcomes.' A second related gap: demographic names from Yin et al. (2024) differ in tokenization length and lexical surface form, and only name frequency is binned (Section 3.3); without controlling these per-name properties or showing that the directional asymmetries survive such controls, exclusion cannot be cleanly attributed to group membership. This is the most load-bearing gap because the paper's headline conclusion depends on retrieval, not on the summarization proxies.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies allocational fairness of LLM-based hiring pipelines using two tasks: resume summarization and candidate retrieval/ranking. The authors construct a synthetic resume dataset from real resumes, adding four demographic name groups (Black/White male/female) and, in some conditions, demographically tailored extracurricular content. They define invariance violations for summarization (the percentage of paired t-tests rejecting equality across a battery of text-proxy measures) and exclusion for retrieval (the percentage of top-n candidates dropping out after demographic perturbation), together with the non-uniformity metric of Wilson and Caliskan (2024). They evaluate six summarization models and four embedding models. Results: race-perturbed summaries show more invariance violations than gender; retrieval models show high exclusion for both gender and race perturbations, with directional asymmetries; non-demographic perturbations (within-group names, typos, spacing) cause comparable sensitivity. The paper concludes that LLM hiring pipelines are brittle and can exhibit considerable bias, primarily in the retrieval stage.","tokens_in":18423,"tokens_out":5876,"duration_ms":53330,"significance":"If the findings hold, this is a valuable, ecologically grounded contribution: it moves fairness evaluation from classification tasks to generative and retrieval settings, introduces reusable metrics (invariance violations, exclusion) and a public benchmark, and includes a human preference validation for the summarization proxies. The paper also honestly reports that non-demographic perturbations produce similar exclusion magnitudes, which is an interesting robustness finding. The benchmark and code release will be useful to the community. The metrics are defined without fitted free parameters (aside from the standard alpha significance level), and the central empirical results are reproducible in principle from the released artifacts. Main weaknesses: the directional asymmetry claims lack uncertainty quantification, and the name perturbation design does not control for tokenization length or lexical form, both of which could affect the central retrieval conclusion. The summarization proxy measures are validated only on artificially manipulated dimensions, not on demographic perturbations, so the allocational interpretation of the summarization results is provisional.","major_comments":[{"comment":"The paper's central claim that retrieval models exhibit \"considerable bias, primarily stemming from the retrieval stage\" rests on the directional asymmetry of exclusion (e.g., mistral-embed M→F 63.28% vs. F→M 27.93%; W→B vs. B→W trends). These values are presented as point estimates aggregated over job postings without confidence intervals or significance tests. Because Figure 7 shows that non-demographic perturbations (within-group name changes, spacing, typos) produce exclusion rates of comparable magnitude, the demographic asymmetry is the only evidence distinguishing demographic bias from generic brittleness. Without a statistical test of the directional differences (e.g., bootstrapping over job postings or a paired permutation test), the observed asymmetries may be within sampling noise, and the \"considerable bias\" conclusion is not yet established.","section":"§4.2.2, Figure 5"},{"comment":"Name perturbations are matched on frequency (via Pile bins) but not on tokenization length or lexical surface form. The authors cite An and Rudinger (2023) in the related work, which shows that tokenization length alone can produce first-name bias in model outputs. Since the Yin et al. (2024) name sets differ systematically in length and spelling patterns (e.g., \"Latanya\" vs. \"Emily\"), the reported directional effects W→B > B→W (Figure 5) and the sensitivity to within-group name changes (Figure 7) could be confounded by tokenization artifacts. The manuscript should either match names across groups on tokenization length (in addition to frequency) or include token length as a covariate in a per-resume logistic regression predicting exclusion from perturbation direction. This is necessary to attribute the retrieval asymmetry to demographic group membership.","section":"§3.3 and §3.2"},{"comment":"The validation of the summarization proxy measures does not bridge the gap from text-property differences to allocational harm. The human preference study shows that HR professionals have preferences when summaries differ along deliberately manipulated dimensions (quantification, focus, individual impact) and that the five automated measures detect those manipulated differences. However, the invariance violations reported in Figure 2 are computed on demographic perturbations, not on the manipulated dimensions; there is no evidence that the demographic-induced changes in reading ease, polarity, regard, etc. correspond to changes that would affect an HR decision. Consequently, the Discussion's claim that \"20% of remaining candidates being depicted in less preferable ways\" (Section 5) overstates what the data show. The summarization results should be reported as differences in proxy measures, with the downstream allocational interpretation clearly flagged as an assumption awaiting validation.","section":"§2.1 and §A.9"}],"minor_comments":[{"comment":"The displayed formula for exclusion is malformed; it should read exclusion_n(p) = |{d ∈ D_n(p) : rank_p(d′) > n}| / |D_n(p)|.","section":"§2.2, Eq. (2)"},{"comment":"Figure 2 is visually cluttered and difficult to compare across models; consider supplementing it with a table of numeric invariance-violation values per model and perturbation type.","section":"Figure 2"},{"comment":"The limitations section should explicitly discuss the fact that Command-R generated the seed resumes and is also evaluated as a summarization model; this coupling could introduce artifacts for that model and should be acknowledged as a potential confound in the summarization results.","section":"§3.1 and §3.4"},{"comment":"The paper does not justify why 11 professions/fields were selected for job posts from the 22/24 available; please state the selection criterion or note that the choice was based on job-posting availability.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The paper is a solid empirical study of fairness in LLM hiring pipelines, with a novel benchmark and metrics. The main risk is that the headline conclusion about retrieval-stage bias relies on directional asymmetries that are not statistically tested, and the name sets are confounded with tokenization length. Both are fixable with additional analyses. The summarization proxy validation is indirect but still useful. I recommend major revision rather than rejection, because the central question is important and the issues are addressable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should read this if you work on fairness in LLM pipelines. The new synthetic resume benchmark and the exclusion metric are real contributions, and the authors deserve credit for putting non-demographic perturbations on the same footing as demographic ones. But the headline claim — that retrieval shows \"considerable bias\" leading to \"discriminatory outcomes\" — is stronger than the evidence supports. The directional asymmetries in exclusion (Section 4.2.2, Figure 5) are the only thing separating demographic bias from general brittleness, and they are presented as point estimates with no confidence intervals or significance tests. Given that within-group name changes produce comparable exclusion rates (Figure 7a), those asymmetries could be noise. This is the main soft spot, and it is load-bearing for the abstract's conclusion.\n\nWhat the paper does well: it constructs a realistic two-stage hiring pipeline (summary generation + retrieval), uses 10 models, and releases a dataset that others can build on. The exclusion metric is a good idea — it directly measures whether a perturbed resume drops out of the top-n, which is the allocational question. The finding that models are about as sensitive to typos and spacing as to name changes is worth publishing on its own; it reframes the conversation from \"bias\" to \"brittleness,\" and the authors make that point explicitly in the discussion.\n\nThe other soft spots are smaller. The summarization proxy measures (reading ease, polarity, etc.) are validated by a small human study showing that HR staff have preferences for summaries differing in certain dimensions, but the study doesn't show that demographic-induced changes in those measures would change a real hiring decision. That gap is acknowledged in the appendix, but it should temper the summarization claims. The name sets from Yin et al. differ in tokenization length and other properties; only frequency is binned, so a reviewer should ask whether confounding remains. And Command-R generated the synthetic resumes and is also one of the summarization models, which is a mild but unmentioned coupling.\n\nThis paper belongs in a peer-reviewed venue. Send it to review, but ask the authors to add error bars or significance tests on the directional exclusions and to soften the \"considerable bias\" language unless the asymmetries survive that test. The benchmark and metric are worth citing now.","headline":"Solid benchmark and new metric, but the retrieval-bias claim rests on untested asymmetries; deserves review with revisions.","tokens_in":18951,"tokens_out":3796,"would_cite":true,"duration_ms":34796,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM hiring pipeline that retrieves and summarizes resumes is allocationally unfair, with the retrieval stage as the main source of bias.","keywords":["allocational fairness","LLM hiring bias","resume summarization","retrieval fairness","demographic perturbation","invariance violations","exclusion metric","synthetic resume benchmark"],"falsifier":"Have HR professionals shortlist candidates from paired summaries of identical resumes that differ only by demographic perturbation; if summaries with statistically different reading ease, polarity, or regard do not produce different shortlisting rates, the summarization-stage bias does not translate into allocational harm. For the retrieval stage, compare exclusion rates under demographic name swaps against exclusion under frequency-matched but demographically uninformative name swaps; if the control shows equally high exclusion, the observed 'bias' is general brittleness rather than demographic discrimination.","tokens_in":18004,"feed_emoji":"⚖️","tokens_out":6014,"duration_ms":55562,"temperature":0.7,"pith_summary":"This paper asks whether an LLM-based hiring pipeline—resume retrieval followed by resume summarization—treats otherwise identical candidates differently when their demographic signals change. Using a new synthetic resume benchmark with controlled name and extracurricular perturbations, it finds that race changes alter generated summaries up to 20% of the time (gender only 3%), and that retrieval models exclude demographically perturbed resumes from top-5 lists nearly half the time on average, up to 74% in specific settings. The authors argue the harm is real: candidates who differ only by name can be filtered out or described less favorably, and the retrieval stage is the larger source of bias. A striking additional finding is that non-demographic changes—typos and spacing—produce comparable ranking shifts, suggesting much of the measured unfairness is general brittleness rather than targeted demographic bias.","feed_headline":"LLM hiring screens drop up to 74% of resumes after a name change","feed_subtitle":"Race and gender swaps, and even spacing edits, knock identical resumes out of top rankings—mostly at the retrieval stage.","key_machinery":"The central object is the paired perturbation design: each resume has demographically perturbed twins that keep content fixed, so any output difference is attributable to demographic signal. For summarization, the paper counts invariance violations—the fraction of paired t-tests (with multiple-comparison correction) that reject equality across five automated text measures (reading ease, reading time, polarity, subjectivity, regard). For retrieval, it proposes the exclusion metric—the fraction of top-n resumes that fall out of the top-n set after perturbation—alongside the non-uniformity chi-square test of demographic composition in top-x% results. These metrics turn the abstract question 'is the system fair?' into concrete per-stage measurements that can be compared across models and perturbation types.","core_discovery":"The paper claims that an LLM hiring pipeline with automated resume retrieval and summarization is not allocationally fair, and that the retrieval stage is the primary driver. On identical resumes differing only by gender- or race-typed names (sometimes plus demographically tailored extracurriculars), generated summaries show statistically meaningful differences far more often for race than for gender—up to 20% versus about 3%—and retrieval models select resumes non-uniformly across demographic groups and drop perturbed resumes out of the top-n pool at high rates (averaging 45.75% for top-5, up to 74% in specific settings). The paper further claims that non-demographic perturbations (typos, spacing, within-group name changes) cause similar or even higher exclusion, so much of the apparent demographic bias is better described as general model brittleness. In the worst composite case the authors report roughly 70% of candidates filtered at screening and about 20% of remaining candidates depicted less favorably after racial indicators change.","pith_inferences":["Because the paper finds comparable exclusion for within-group name changes and formatting edits, the exclusion metric may be measuring embedding-model instability as much as demographic bias; a fair-system audit should always pair exclusion with a non-demographic control before attributing disparity to protected attributes.","The 20%-versus-3% race/gender gap in summarization suggests race-linked content cues are more strongly encoded in resume text than gender cues; testing intersectional race-gender perturbations would likely show effects that are not additive.","Directional asymmetries (e.g., larger White-to-Black than Black-to-White exclusion) imply that fairness interventions need to be direction-aware; a symmetric swap design, as used here, is the right auditing default but should be extended to more groups and to non-name demographic signals such as age and nationality.","The dependence of exclusion magnitude on resume formatting (structured generated resumes were nearly twice as sensitive to spacing changes as web-scraped ones) suggests that standardizing resume formats before retrieval could reduce both apparent and real disparities."],"forward_implications":["In a standard retrieval-then-summarize hiring pipeline, the retrieval stage is the main source of allocational harm; fixing only the summarizer would leave most of the bias in place.","Because swapping only a candidate's name can drop a resume from the top-5 set nearly half the time on average, screening systems that filter to small candidate pools can silently exclude qualified applicants from underrepresented groups.","Race perturbations change generated summaries more often than gender perturbations do, so summary-based screening can portray otherwise identical candidates differently along racial lines.","The high sensitivity to typos and spacing means some apparent demographic bias is actually general model brittleness; audits should control for non-demographic perturbations before attributing exclusion to demographics.","Adding demographically tailored extracurricular information increases exclusion asymmetrically, especially for White-to-Black name changes, so real resumes with multiple demographic signals may produce worse disparities than name-only tests suggest."],"supporting_citations":[{"why":"Supplies the non-uniformity metric, cosine-similarity retrieval setup, and the name-perturbation methodology this paper extends.","marker":"(Wilson and Caliskan, 2024)"},{"why":"Provides the curated White/Black male/female name lists used to create the demographic perturbations.","marker":"(Yin et al., 2024)"},{"why":"Inspires the addition of demographically tailored extracurricular information as a second demographic signal.","marker":"(Glazko et al., 2024)"},{"why":"Frames the invariance-testing approach that motivates the summarization fairness metric.","marker":"(Ribeiro et al., 2020)"},{"why":"Provides the multiple-comparison correction used when counting statistically significant summary differences.","marker":"(Benjamini and Hochberg, 1995)"},{"why":"Supplies the regard measure and classifier used as one of the five automated summary proxies.","marker":"(Sheng et al., 2019)"},{"why":"Provides the public Kaggle resume dataset used as a second, less-structured resume corpus.","marker":"(Bhawal, 2021)"},{"why":"Used to generate the synthetic resumes and to add extracurricular information in the perturbation pipeline.","marker":"(Cohere, 2024)"}],"fun_headline_variants":["Name change drops up to 74% of resumes from LLM hiring top-5","Race swaps alter LLM hiring summaries 7x more than gender swaps","Retrieval stage, not summarization, drives LLM hiring unfairness","Spacing edits drop LLM resumes as often as race swaps in hiring","LLM hiring bias is model brittleness, not just demographic prejudice"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The summarization-stage finding rests on the assumption that the five automated text measures (reading ease, reading time, polarity, subjectivity, regard) capture differences that would actually change a recruiter's assessment; the paper validates that HR professionals have preferences when summaries differ on other characteristics, but never shows that demographic-induced differences in these measures alter real hiring decisions.","fun_headline_variants_meta":{"raw":{"variants":["Name change drops up to 74% of resumes from LLM hiring top-5","Race swaps alter LLM hiring summaries 7x more than gender swaps","Retrieval stage, not summarization, drives LLM hiring unfairness","Spacing edits drop LLM resumes as often as race swaps in hiring","LLM hiring bias is model brittleness, not just demographic prejudice"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000399,"raw_usage":{"total_tokens":2076,"prompt_tokens":923,"completion_tokens":1153,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":1054}},"tokens_in":539,"tokens_out":1153,"duration_ms":10634,"temperature":1.0,"reasoning_tokens":1054,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:35:38.764526+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have HR professionals shortlist candidates from paired summaries of identical resumes that differ only by demographic perturbation; if summaries with statistically different reading ease, polarity, or regard do not produce different shortlisting rates, the summarization-stage bias does not translate into allocational harm. For the retrieval stage, compare exclusion rates under demographic name swaps against exclusion under frequency-matched but demographically uninformative name swaps; if the control shows equally high exclusion, the observed 'bias' is general brittleness rather than demographic discrimination.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the curated White/Black male/female name lists used to create the demographic perturbations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the multiple-comparison correction used when counting statistically significant summary differences."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the public Kaggle resume dataset used as a second, less-structured resume corpus."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Used to generate the synthetic resumes and to add extracurricular information in the perturbation pipeline."}],"review_version":1}