{"id":"5168fef9-779b-4c53-9cce-e91b6067b137","arxiv_id":"2506.00256","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLMs selecting among identical job candidates consistently favored those who disclosed no disability, penalizing both disability disclosure and refusal to answer.","lead":"The paper tested whether large language models favor job applicants who explicitly say they have no disability over equally qualified applicants who disclose a disability or decline to answer. It found consistent preference for the no-disability option across five LLMs and 16 occupations, which matters because the disclosure question is standard on many job applications.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Experiment 1's candidate entries contain no qualifications, education, or experience, so the claimed preference may be an artifact of forced choice under artificial parity rather than a property of real LLM-driven selection.","rationale":"I did not find evidence of an execution error or intentional manipulation: the large samples, consistent chi-square rejections, and publicly promised artifacts make it plausible that the tested models do show a real preference for the 'no disability' option in this exact setup. The load-bearing weakness is external validity, and the reader's 'ecological validity' concern points in that direction. I sharpen it: the candidate entries in Table 1 contain no actual qualification information, so the experiment only demonstrates a tie-breaking preference under artificially perfect parity. The central claim's phrase 'not on qualifications' requires qualifications to be present as input; otherwise the model has nothing to weigh them against. Additionally, the nondisclosure comparison is operationalized solely as 'I do not want to answer,' and Experiment 1 lacks a condition with no disability field at all, which is a common real-world representation. These are concrete, testable gaps, and addressing them determines whether the headline conclusion generalizes beyond the stripped-down prompt format. Because the reader already issued a CONDITIONAL verdict for addressable methodological concerns, my critique does not move the verdict; it strengthens the reason for the condition.","tokens_in":8930,"tokens_out":8883,"duration_ms":97743,"concrete_test":"Replicate Experiment 1 with realistic full resume blocks for each occupation: identical education, years of experience, skills, and job history, differing only in the disability-disclosure line, and add a fourth condition where the disability line is absent rather than verbalized as 'I do not want to answer.' Pre-register a minimum meaningful effect (e.g., Cramér's V >= 0.1) and report point estimates with confidence intervals. If the no-disability preference is not present or is below threshold when qualifications are visible, the central claim as generalized to real LLM-driven hiring is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim—that LLMs consistently favored candidates who disclosed no disability and that this is a systematic preference in LLM-based hiring tools—rests almost entirely on Experiment 1. In the candidate profile example (Table 1, Exp.1), each entry contains only name, gender, race, and the disability-response line: there is no education, experience, or qualification field. The prompt asserts that 'The candidates possess similar qualifications, education, and experience,' but the model receives no such content to process. The experiment therefore measures how an LLM breaks a tie when the only varying attribute is the disclosure statement. It does not test whether disclosure would override actual qualifications in a realistic resume, and it does not include a condition in which the disability line is entirely absent (only the verbalized option 'I do not want to answer'). The broad conclusion about LLM-driven candidate selection is thus load-bearing on the ecological assumption that this stripped-down forced-choice format reproduces real selection behavior. If actual pipelines score or rank resumes with heterogeneous qualifications, the observed preference could shrink, disappear, or even reverse.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether LLM-based candidate selection is affected by disability disclosure. In Experiment 1, three candidate profiles are identical except for the disability-disclosure response (\"Yes, I have a disability\", \"No, I do not have a disability\", and \"I do not want to answer\"), and five LLMs are asked to select one candidate for 16 occupations with small employment-rate gaps between persons with and without disabilities. The authors report large chi-square statistics and conclude that the models consistently favor the \"no disability\" candidate over both the \"disability\" and the \"do not want to answer\" candidates. Experiments 2 and 3 vary gender and race under a fixed \"Yes\" disability disclosure and compare with a condition where disability information is absent, reporting additional gender and race effects. Qualitative examples of model justifications are provided, and the code and data are publicly released.","tokens_in":9090,"tokens_out":4334,"duration_ms":42701,"significance":"If the central claim holds, the finding is practically important: identical candidates would be selected differently solely on the basis of the wording of a voluntary disclosure item, which has direct implications for AI-assisted hiring and for applicant guidance. Strengths include the large sample sizes, the use of five different models, the externally sourced name-race-gender dataset, the explicit control of candidate attributes, and the public release of artifacts. The paper also states its normative fairness baseline openly. However, the experimental format is highly stylized, and the statistical reporting does not currently support the strength of the claims. The main contribution is a controlled measurement of tie-breaking behavior rather than a demonstration of how real LLM-driven pipelines process disclosure in the presence of varied qualifications.","major_comments":[{"comment":"The candidate profiles in Table 1 contain only name, gender, race, and the disability-response line; the prompt's assertion that \"candidates possess similar qualifications, education, and experience\" gives the model no actual qualification content to process. The experiment therefore measures how an LLM breaks a tie when the only varying attribute is the disclosure statement, not whether disclosure would override qualifications in a realistic resume. This gap is load-bearing for the abstract's broad conclusion about \"LLM-driven candidate selection.\" I request either (a) a condition with realistic resume content, including varied qualifications, or (b) a narrowed claim stating that the result concerns forced-choice tie-breaking under artificial parity. Additionally, there is no condition in which the disability field is entirely absent; only \"I do not want to answer\" is used, so the comparison to non-disclosure is incomplete.","section":"Experimental Setup, Experiment 1; Table 1"},{"comment":"All Experiment 1 significance claims are based on chi-square tests with sample sizes around 102k, so p < 0.05 is practically guaranteed and does not indicate the magnitude of the preference. The manuscript should report effect sizes (e.g., Cramér's V) or selection proportions with confidence intervals. The claim that the trend is \"consistent across all occupations\" is not checkable because no per-occupation counts or tests are reported for Experiment 1. If per-occupation tests were performed, a multiple-comparison correction is required. Also, p-values should be reported as \"< 0.0001\" rather than \"0.0000\".","section":"Results and Discussion, Table 2"},{"comment":"The design compares a \"Yes, I have a disability\" profile with a condition in which disability information is entirely absent, but it does not cross the three disclosure responses with gender or race. As a result, RQ2 and RQ3, as stated in the Introduction, are not answered: the experiments show effects of adding versus omitting a disability statement, not the intersection of disclosure choice with gender/race. The text should either be reframed accordingly or expanded to include the full crossing.","section":"Experiment 2 and Experiment 3; RQ2/RQ3"}],"minor_comments":[{"comment":"Table 1 is difficult to parse as formatted; a table with separate rows for each profile and aligned fields would improve readability.","section":"Table 1"},{"comment":"Figures 1-3 omit axis labels and error bars; they should report the underlying proportions or counts, and the y-axis should be labeled.","section":"Figures 1-3"},{"comment":"The \"Desired Behavior of LLMs\" section presents neutrality as the expected behavior; this is a normative choice and should be labeled as such rather than as a property of the models.","section":"Desired Behavior of LLMs"},{"comment":"The paper says \"We use all the default parameters\" but does not list the specific sampling settings, temperature, or decoding parameters; reporting these would improve reproducibility.","section":"Experimental Setup"},{"comment":"There are several typos and spacing issues (e.g., \"disability disclosure, influences\" in the Introduction, and \"University of South Florida\" in the acknowledgments); the manuscript needs a careful proofreading pass.","section":"Throughout"},{"comment":"The reference list includes inconsistent author-name formatting (e.g., \"Daum ´e Hii\" in Nghiem et al.); please align with the venue style.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a CS/CL fairness venue. The main risk is overgeneralization from a stylized tie-breaking task; a revision that adds realistic resume conditions or carefully tempers the claims would make it publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a clean, well-scoped measurement showing that when candidates are otherwise identical, LLMs systematically prefer explicit 'no disability' over 'prefer not to answer' and over 'yes disability', across five models and sixteen occupations. The new bit is the 'prefer not to answer' penalty; prior work mostly looked at disability-related resume text. The study does what it claims, and the data and code are public.\n\nWhat it does well: large samples (102k for Experiment 1), consistent chi-square results, an occupation set chosen to avoid a confounding employment gap, and a sensible baseline prompt with two rephrasings. The qualitative examples are few but they do show the mechanism: models reasoning that a candidate without disability is more adaptable, more confident, or less of an accommodation burden. That is real evidence of stereotype propagation, not just a statistical artifact.\n\nThe biggest soft spot is ecological validity. In Experiment 1, the candidate profiles contain no actual qualifications, education, or experience. The prompt tells the model the candidates are similar, but the model never sees any such content. So the result is a forced-choice tie-break under artificial parity, not a realistic resume screen. That is exactly what the paper sets out to measure, so I would not call it fatal, but the abstract's language about 'LLM-driven candidate selection' overreaches. A more precise claim would be 'when a model is asked to choose among candidates who differ only in disclosure phrasing.' Also, the paper gives chi-square and p-values but no effect sizes or confidence intervals; with 100k samples, almost anything is significant. A Cramer's V or similar would help. Minor issues: no sampling configuration or commit hash for exact replication, and the qualitative analysis is a handful of examples, fine as illustration.\n\nWho this is for: fairness researchers, HR-tech auditors, and anyone building LLM evaluation suites. It is a useful, reproducible measurement protocol. I would not cite it as proof that real hiring pipelines discriminate this way, but I would cite it as evidence that models can behave this way under controlled conditions.\n\nRecommendation: send it to peer review. Ask for effect sizes, a sentence or two on the forced-choice limitation, and a small robustness check with profiles that include at least one qualification field to see whether the preference survives when the model has actual job-relevant content to weigh.","headline":"A clean, well-scoped measurement of disclosure-phrasing bias in LLM hiring recommendations; treat the abstract's generality with caution but send it to review.","tokens_in":9597,"tokens_out":1396,"would_cite":true,"duration_ms":15687,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper reports that, with qualifications held identical, LLMs preferred candidates who disclosed 'no disability,' and even candidates who declined to answer were selected less often than those who said 'no.'","keywords":["disability disclosure","LLM hiring bias","candidate selection","AI fairness","resume screening","intersectional bias","prefer not to answer","chi-squared test"],"falsifier":"A falsifying observation would be a real or simulated resume-screening run, using full resumes plus a separate voluntary-disclosure form or candidates with realistic differences in experience, in which the 'no disability' preference shrinks to chance or reverses across the same five models; if that occurs, the list-format triplet result does not generalize to operational hiring pipelines.","tokens_in":8721,"feed_emoji":"⚖️","tokens_out":7440,"duration_ms":65129,"temperature":0.7,"pith_summary":"This paper tests whether the wording of voluntary disability disclosure changes which candidates large language models select for a job. In Experiment 1, the authors built triplets of candidates who shared the same name, gender, race, qualifications, education, and experience, and varied only the disclosure answer: 'Yes, I have a disability or have had one in the past,' 'No, I do not have a disability and have not had one in the past,' and 'I do not want to answer.' Across five LLMs, sixteen occupations, and over one hundred thousand prompts, models consistently chose the candidate who said 'no disability' more often than either other profile, with all differences significant by chi-square tests. The paper's point is that LLM hiring tools appear to penalize disability disclosure, and even non-disclosure, on the basis of phrasing alone rather than job-relevant merit.","feed_headline":"LLMs favor 'no disability' applicants with identical skills","feed_subtitle":"Five models preferred applicants who said 'no disability'; declining to answer also lowered selection odds.","key_machinery":"The machinery is a matched-triplet comparison: three candidate profiles that are word-for-word identical in name, gender, race, and stated qualifications, differing only in the disability-disclosure answer. The same selection prompt is repeated over 16 occupations with near-parity employment rates for disabled and non-disabled workers, randomized candidate order, three phrasing variants, and 102,000 samples, with chi-square tests quantifying whether selection counts deviate from an equal split. This design isolates the causal effect of disclosure wording from every other candidate attribute.","core_discovery":"The central discovery is that LLM-driven candidate selection is systematically skewed by disability disclosure phrasing. When profiles were otherwise identical, all five tested models—GPT4o-mini, Gemma2-9B, Mistral-7B, Qwen2.5-7B, and Llama3.2-3B—selected the explicitly 'no disability' candidate at the highest rate, in every occupation studied, and the effect was statistically significant. Candidates who chose 'I do not want to answer' were also less likely to be selected than those who explicitly said 'no,' though the ordering relative to disclosed disability varied by model. Follow-up experiments varying gender or race, with disability status held constant, show that disclosure also shifts which gender and racial groups are favored, with models often justifying choices by appealing to diversity while actually favoring one group disproportionately.","pith_inferences":["A consequence that follows but is not tested here: applicants face a disclosure dilemma, because both 'yes' and 'I do not want to answer' carry a measured selection penalty relative to 'no.'","The chosen model justifications suggest the effect may stem from pretraining associations between disability and accommodation needs or lower adaptability; directly measuring those associations would be a natural follow-up.","The same matched-triplet protocol could be applied to other voluntarily disclosed protected attributes, such as veteran status; comparable results would show the bias is not disability-specific.","It remains open whether the bias can be removed by instruction prompts that tell the model to ignore disclosure information; that is a concrete, testable extension of the paper's setup."],"forward_implications":["If these results hold in operational settings, LLM-based screening tools penalize candidates who disclose a disability even when qualifications are identical to other candidates.","The 'prefer not to answer' option does not protect candidates; in the tested setup it also lowered selection rates relative to stating 'no disability.'","The bias appears across all five tested models and across all 16 occupations, not as a quirk of one model or one job type.","Gender and race effects shift when disability information is present: for example, Qwen showed no significant race difference in some occupations with disability information but significant differences without it.","To make LLM hiring fair, disability disclosure status should not be part of the candidate representation the model scores on."],"supporting_citations":[{"why":"Supplies the first-name and race/gender categorization used to build the candidate profiles.","marker":"Nghiem et al. (2024)"},{"why":"Previous demonstration of disability bias in GPT-based resume screening that this study extends to disclosure phrasing.","marker":"Glazko et al. (2024)"},{"why":"Provides disability-centered perspectives on LLMs that motivate measuring disclosure-related bias.","marker":"Gadiraju et al. (2023)"},{"why":"Establishes implicit bias against people with disabilities in pretrained language models, the prior result being tested in a hiring setting.","marker":"Venkit, Srinath, and Wilson (2022)"},{"why":"Supplies the chi-squared test used to determine whether selection differences are statistically significant.","marker":"Greenwood and Nikulin (1996)"},{"why":"Documents gender and race bias in language-model resume screening, the comparator for the intersectional experiments.","marker":"Wilson and Caliskan (2024)"}],"fun_headline_variants":["All 5 LLMs favor 'no disability' in identical candidate profiles","Disability disclosure shifts LLM hiring: 'no' beats 'decline'","Saying 'no disability' boosts LLM selection; decline hurts too","LLM hiring bias: disclosure of disability lowers selection odds","Identical skills, different odds: LLMs prefer 'no disability'"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that asking a model to 'select one candidate' from a short list of otherwise identical profiles captures how LLM hiring tools actually encounter disability disclosure, rather than being an artificial format that overstates the effect.","fun_headline_variants_meta":{"raw":{"variants":["All 5 LLMs favor 'no disability' in identical candidate profiles","Disability disclosure shifts LLM hiring: 'no' beats 'decline'","Saying 'no disability' boosts LLM selection; decline hurts too","LLM hiring bias: disclosure of disability lowers selection odds","Identical skills, different odds: LLMs prefer 'no disability'"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000214,"raw_usage":{"total_tokens":1395,"prompt_tokens":886,"completion_tokens":509,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":414}},"tokens_in":502,"tokens_out":509,"duration_ms":5078,"temperature":1.0,"reasoning_tokens":414,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:08:45.668488+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A falsifying observation would be a real or simulated resume-screening run, using full resumes plus a separate voluntary-disclosure form or candidates with realistic differences in experience, in which the 'no disability' preference shrinks to chance or reverses across the same five models; if that occurs, the list-format triplet result does not generalize to operational hiring pipelines.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the first-name and race/gender categorization used to build the candidate profiles."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Previous demonstration of disability bias in GPT-based resume screening that this study extends to disclosure phrasing."},{"cited_title":"i wouldn’t say offensive but","cited_arxiv_id":null,"evidence_quote":"Provides disability-centered perspectives on LLMs that motivate measuring disclosure-related bias."},{"cited_title":"N.; Srinath, M.; and Wilson, S","cited_arxiv_id":null,"evidence_quote":"Establishes implicit bias against people with disabilities in pretrained language models, the prior result being tested in a hiring setting."},{"cited_title":"E., and Nikulin, M","cited_arxiv_id":null,"evidence_quote":"Supplies the chi-squared test used to determine whether selection differences are statistically significant."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents gender and race bias in language-model resume screening, the comparator for the intersectional experiments."}],"review_version":1}