arxiv: 2604.25932 · v1 · submitted 2026-04-03 · 💻 cs.CY · cs.AI

Recognition: no theorem link

Sociodemographic Biases in Educational Counselling by Large Language Models

Tomasz Adamczyk , Wiktoria Mieleszczenko-Kowszewicz , Beata Bajcar , Grzegorz Chodak , Aleksander Szcz\k{e}sny , Maciej Markiewicz , Karolina Ostrowska , Aleksandra Sawczuk

show 1 more author

Przemys{\l}aw Kazienko

Authors on Pith no claims yet

Pith reviewed 2026-05-13 18:41 UTC · model grok-4.3

classification 💻 cs.CY cs.AI

keywords LLM biasessociodemographic biaseducational counselingAI fairnessstudent vignettesbias reductionlarge language models

0 comments

The pith

Large language models show sociodemographic biases in educational counseling that decrease with more detailed student information.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tests how six large language models respond to questions about students described in 900 different vignettes. Each vignette is varied across 14 sociodemographic factors like race, gender, socioeconomic status, and immigrant background. All models produced biased responses, with patterns that partly match known human biases but also differ. The key finding is that vague or minimal student descriptions nearly triple the size of these biases, while providing concrete details about the individual student cuts them down substantially. This matters because it points to a practical way to make AI counseling tools fairer by using richer descriptions rather than relying on stereotypes.

Core claim

All six LLMs exhibit measurable sociodemographic biases in their educational counseling responses across race, gender, socioeconomic status, and immigrant background. These biases partially align with human biases but diverge in important ways, and their magnitude varies substantially by model. Critically, the precision of student descriptions controls bias strength: vague information amplifies disparities by nearly three times, while concrete, individualized metrics substantially reduce them.

What carries the argument

A set of 900 systematically varied student vignettes tested against 14 sociodemographic identifiers plus control, with bias quantified through differences in model-generated counseling responses.

Load-bearing premise

That the biases detected in model responses to constructed student vignettes would match those arising in live interactions with actual students.

What would settle it

An experiment that tracks real student outcomes or has human counselors rate the same vignettes and compares alignment with LLM outputs.

Figures

Figures reproduced from arXiv: 2604.25932 by Aleksander Szcz\k{e}sny, Aleksandra Sawczuk, Beata Bajcar, Grzegorz Chodak, Karolina Ostrowska, Maciej Markiewicz, Przemys{\l}aw Kazienko, Tomasz Adamczyk, Wiktoria Mieleszczenko-Kowszewicz.

**Figure 2.** Figure 2: Bias pattern across all contexts. Each cell shows the mean difference (∆) from the control condition (“a student”) for each demographic group and information level, and the significance level. Black boxes highlight the key findings discussed in the text [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** Figure 3: Model comparison. Left: mean bias and z-scores by demographic group. Right: mean bias and z-scores by information density level. Positive z-scores indicate strongerthan-average positive bias [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

read the original abstract

As Large Language Models (LLMs) are increasingly integrated into educational settings, understanding their potential biases is critical. This study examines sociodemographic biases in LLM-based educational counselling. We evaluate responses from six LLMs answering questions about 900 vignettes describing students in diverse circumstances. Each vignette is systematically tested across 14 sociodemographic identifiers - spanning race and gender, socioeconomic status, and immigrant background - along with a control condition, yielding 243,000 model responses. Our findings indicate that (1) all models exhibit measurable biases, (2) bias patterns partially align with documented human biases but diverge in notable ways, (3) the magnitude of these biases is strongly influenced by the precision of the student descriptions, where vague or minimal information amplifies disparities nearly threefold, while concrete, individualised metrics substantially reduce them, and (4) bias profiles vary substantially across models. These results demonstrate the importance of context-rich and personalised educational representations, suggesting that AI-driven educational decisions benefit from detailed student-specific information to promote fairness and equity.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

This paper shows LLMs exhibit sociodemographic biases in educational counseling that shrink with richer student details, but the vignette design leaves real-world generalization open.

read the letter

Hi, the main point is that the six tested LLMs all display measurable biases tied to race, gender, socioeconomic status and immigrant background when responding to student vignettes, and that bias levels drop substantially when the descriptions shift from vague to concrete and individualized. They ran 243,000 responses across 900 vignettes and 14 identifiers, which gives the work real scale for an audit study. The precision effect is the clearest new angle: vague prompts amplify disparities while specific metrics reduce them, and the models show different bias profiles from each other and only partial overlap with documented human patterns. That controlled variation in context richness is the part that actually adds something usable. The experiment is systematic enough on its own terms and the volume of data is a plus for spotting patterns. The soft spot is the static vignette setup itself. Real counseling is multi-turn and students supply or correct details on the fly, so it is not obvious the reported threefold difference survives when the model can ask follow-ups or receive unscripted information. The abstract also skips the exact bias formula and statistical controls, which makes it hard to judge how robust the quantification is until the methods are checked. This is for people working on fairness in educational AI or anyone deploying LLMs in schools. It is timely and the scale justifies a referee even with the usual limits of vignette work. I would send it for peer review.

Referee Report

2 major / 2 minor

Summary. The manuscript reports results from a controlled experiment generating 243,000 responses from six LLMs to 900 student vignettes, each crossed with 14 sociodemographic identifiers plus a control. It claims that all models exhibit measurable biases, that these biases partially align with but also diverge from documented human biases, that bias magnitude is strongly modulated by description precision (nearly threefold amplification under vague/minimal information versus substantial reduction under concrete individualized metrics), and that bias profiles differ substantially across models.

Significance. If the central precision-modulation claim holds after methodological clarification, the work would be significant for AI deployment in education, underscoring the value of context-rich student representations for fairness. The experiment's scale across models and identifiers is a clear strength; however, the absence of reported statistical procedures and generalization tests currently limits verifiability and impact.

major comments (2)

[Methods] Methods section: no statistical methods, bias quantification formula, inter-rater reliability checks, or prompt-sensitivity controls are described, making it impossible to verify the 'nearly threefold amplification' claim for vague versus concrete descriptions.
[Experimental Design] Experimental Design and Results: the design uses only single-turn responses to fixed vignettes; this does not test whether the reported threefold precision effect survives in multi-turn interactions where models can request or receive additional unscripted context, undermining the generalizability of the central modulation finding.

minor comments (2)

[Abstract] Abstract: the phrase 'measurable biases' is used without even a brief parenthetical definition of the metric; adding one sentence would improve immediate clarity.
[Results] Results: tables or figures reporting the bias magnitudes should include confidence intervals or standard errors to allow readers to assess the precision of the threefold claim.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. The comments highlight important areas for improving methodological transparency and clarifying the scope of our findings. We have revised the manuscript to address these points where feasible while maintaining the integrity of the original experimental design.

read point-by-point responses

Referee: [Methods] Methods section: no statistical methods, bias quantification formula, inter-rater reliability checks, or prompt-sensitivity controls are described, making it impossible to verify the 'nearly threefold amplification' claim for vague versus concrete descriptions.

Authors: We have substantially expanded the Methods section to include the missing details. Bias is quantified as the normalized difference in mean educational outcome scores (e.g., recommended support levels, GPA projections) across sociodemographic groups relative to the control condition. Statistical analysis employs one-way ANOVA for overall group effects followed by post-hoc pairwise comparisons with Bonferroni correction; the amplification factor is computed as the ratio of bias magnitudes between the vague/minimal and concrete/individualized description conditions. Because all evaluations use automated rubric-based parsing of model outputs rather than human raters, inter-rater reliability checks do not apply and this has been explicitly stated. Prompt sensitivity was assessed via a supplementary analysis on 100 vignettes using three paraphrased prompt variants, confirming stable bias patterns. These additions allow direct verification of the reported amplification effect. revision: yes
Referee: [Experimental Design] Experimental Design and Results: the design uses only single-turn responses to fixed vignettes; this does not test whether the reported threefold precision effect survives in multi-turn interactions where models can request or receive additional unscripted context, undermining the generalizability of the central modulation finding.

Authors: The single-turn, fixed-vignette design was chosen to enable precise isolation of sociodemographic and precision effects under fully controlled conditions. We acknowledge that this limits direct claims about multi-turn dynamics and have added an explicit limitations paragraph in the Discussion section noting that future work should examine adaptive, multi-turn counseling where models can solicit additional context. Within the single-turn paradigm, however, the precision-modulation result remains robust and is relevant to many real-world initial-query scenarios. No new experiments were conducted for this revision. revision: partial

Circularity Check

0 steps flagged

No circularity: purely empirical measurement of LLM vignette responses

full rationale

The paper conducts a direct empirical evaluation by generating and analyzing 243,000 LLM responses to 900 fixed vignettes varied across 14 sociodemographic identifiers. No equations, fitted parameters, derivations, or self-citation chains are present that could reduce any claim to its own inputs by construction. The reported threefold amplification of bias under vague descriptions is a measured outcome from the experimental conditions, not a prediction derived from prior fits or self-referential premises. Methodological assumptions about vignette validity are external to any derivation loop and do not trigger the enumerated circularity patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the assumption that vignette responses are a valid proxy for real counseling bias and that the chosen 14 identifiers and 900 vignettes adequately sample the space of educational scenarios.

axioms (1)

domain assumption Vignette responses from LLMs can be used to measure real-world sociodemographic biases in educational counseling
The entire measurement pipeline depends on this untested mapping from simulated to actual counseling contexts.

pith-pipeline@v0.9.0 · 5526 in / 1184 out tokens · 54240 ms · 2026-05-13T18:41:20.345076+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

25 extracted references · 25 canonical work pages · 2 internal anchors

[1]

arXiv preprint arXiv:2403.15281 (2024)

An, J., Huang, D., Lin, C., Tai, M.: Measuring gender and racial biases in large language models. arXiv preprint arXiv:2403.15281 (2024)

work page arXiv 2024
[2]

In: Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society

Arzaghi, M., Carichon, F., Farnadi, G.: Understanding intrinsic socioeconomic bi- ases in large language models. In: Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. vol. 7, pp. 49–60 (2024)

work page 2024
[3]

Teaching and Teacher Education123, 103985 (2023) 14 T

Batruch, A., Geven, S., Kessenich, E., van de Werfhorst, H.G.: Are tracking rec- ommendations biased? a review of teachers’ role in the creation of inequalities in tracking decisions. Teaching and Teacher Education123, 103985 (2023) 14 T. Adamczyk et al

work page 2023
[4]

Bender, E.M., Gebru, T., McMillan-Major, A., Shmitchell, S.: On the dangers of stochastic parrots: Can language models be too big? In: Proceedings of the 2021 ACM conference on fairness, accountability, and transparency. pp. 610–623 (2021)

work page 2021
[5]

On the Opportunities and Risks of Foundation Models

Bommasani, R., Hudson, D., Adeli, E.e.a.: On the opportunities and risks of foun- dation models. arXiv preprint arXiv:2108.07258 (2021)

work page internal anchor Pith review Pith/arXiv arXiv 2021
[6]

Frontiers in psychology10, 2832 (2019)

Pit-ten Cate, I.M., Glock, S.: Teachers’ implicit attitudes toward students from different social groups: A meta-analysis. Frontiers in psychology10, 2832 (2019)

work page 2019
[7]

Social Science Research66(2017), 170–186 (2017)

Cherng, H.Y.S.: If they think i can: Teacher bias and youth of color expectations and achievement. Social Science Research66(2017), 170–186 (2017)

work page 2017
[8]

Frontiers in Psychology12, 712356 (2021)

Costa, S., Langher, V., Pirchio, S.: Teachers’ implicit attitudes toward ethnic mi- nority students: A systematic review. Frontiers in Psychology12, 712356 (2021)

work page 2021
[9]

arXiv preprint arXiv:2510.18902 (2025)

Eze, P., Lunn, S., Berhane, B.: Evaluating LLMs for career guidance: Comparative analysis of computing competency recommendations across ten African countries. arXiv preprint arXiv:2510.18902 (2025)

work page arXiv 2025
[10]

Computational Linguistics50(3), 1097–1179 (2024)

Gallegos, I.O., Rossi, R.A., Barrow, J., Tanjim, M.M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., Ahmed, N.K.: Bias and fairness in large language models: A survey. Computational Linguistics50(3), 1097–1179 (2024)

work page 2024
[11]

Zeitschrift für Entwicklungspsychologie und Pädagogische Psychologie (2024)

Gentrup, S., Olczyk, M., Lorenz, G.: Teacher stereotypes and teacher expecta- tions at the intersection of student gender and socioeconomic status. Zeitschrift für Entwicklungspsychologie und Pädagogische Psychologie (2024)

work page 2024
[12]

In: Proceedings of the International AAAI Conference on Web and Social Media

Giorgi, T., Cima, L., Fagni, T., Avvenuti, M., Cresci, S.: Human and llm biases in hate speech annotations: A socio-demographic analysis of annotators and targets. In: Proceedings of the International AAAI Conference on Web and Social Media. vol. 19, pp. 653–670 (2025)

work page 2025
[13]

Routledge New York (2018)

Gullo, G.L., Staats, C., Capatosto, K.: Implicit bias in schools. Routledge New York (2018)

work page 2018
[14]

In: Proceedings of the 5th Workshop on Gender Bias in Natural Language Processing (GeBNLP)

Gupta, V., Venkit, P.N., Wilson, S., Passonneau, R.J.: Sociodemographic bias in language models: A survey and forward path. In: Proceedings of the 5th Workshop on Gender Bias in Natural Language Processing (GeBNLP). pp. 295–322 (2024)

work page 2024
[15]

In: IIAI-AAI-Winter 2024

Hasegawa, O., Tsurube, T., Ueno, H., Komatsugawa, H.: Research on learning advising using open source LLMs. In: IIAI-AAI-Winter 2024. pp. 253–256. IEEE (2024)

work page 2024
[16]

In: Proc

Kocoń, J., Gruza, M., Bielaniewicz, J., Grimling, D., Kanclerz, K., Miłkowski, P., Kazienko, P.: Learning personal human biases and representations for subjective tasks in natural language processing. In: Proc. IEEE ICDM (2021)

work page 2021
[17]

Lorenz, G.: Subtle discrimination: Do stereotypes among teachers trigger bias in their expectations and widen ethnic achievement gaps? Social Psychology of Edu- cation24(2), 537–571 (2021)

work page 2021
[18]

In: Proc

Miłkowski, P., Gruza, M., Kanclerz, K., Kazienko, P., Grimling, D., Kocoń, J.: Personal bias in prediction of emotions elicited by textual opinions. In: Proc. ACL- IJCNLP Student Research Workshop (2021)

work page 2021
[19]

Sociology of Education95(4), 302–319 (2022)

Okura, K.: Stereotype promise: Racialized teacher appraisals of asian american academic achievement. Sociology of Education95(4), 302–319 (2022)

work page 2022
[20]

In: Findings of IJCNLP 2025

Shailya, K., Mishra, A.K., Krishnan, G.S., Ravindran, B.: Where should I study? Biased language models decide! evaluating fairness in LMs for academic recom- mendations. In: Findings of IJCNLP 2025. pp. 2291–2317. ACL (2025)

work page 2025
[21]

British Educational Research Journal44(5), 847–874 (2018) Sociodemographic Biases in Educational Counselling by LLMs 15

Timmermans, A.C., de Boer, H., Amsing, H.T., van der Werf, M.P.: Track recom- mendation bias: Gender, migration background and ses bias over a 20-year period in the dutch context. British Educational Research Journal44(5), 847–874 (2018) Sociodemographic Biases in Educational Counselling by LLMs 15

work page 2018
[22]

Ethical and social risks of harm from Language Models

Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al.: Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359 (2021)

work page internal anchor Pith review Pith/arXiv arXiv 2021
[23]

Plos one14(5), e0216803 (2019)

Weinberg, D., Stevens, G.W., Finkenauer, C., Brunekreef, B., Smit, H.A., Wijga, A.H.: The pathways from parental and neighbourhood socioeconomic status to adolescent educational attainment: An examination of the role of cognitive ability, teacher assessment, and educational expectations. Plos one14(5), e0216803 (2019)

work page 2019
[24]

In: Findings of NAACL 2025

Weissburg, I., Anand, S., Levy, S., Jeong, H.: LLMs are biased teachers: Evaluating LLM bias in personalized education. In: Findings of NAACL 2025. pp. 5650–5698. ACL (2025)

work page 2025
[25]

PLoS One 20(10), e0335485 (2025)

Yigiter, M.S.: The effect of socioeconomic status on academic achievement: A big data study across countries and time with integrative data analysis. PLoS One 20(10), e0335485 (2025)

work page 2025