REVIEW 4 major objections 4 minor 1 cited by
How Deep Is Representational Bias in LLMs? The Cases of Caste and Religion
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read GPT-4 Turbo generates stories about India that overrepresent dominant religious and caste groups, and diversity prompts barely change that.
desk verdict Abstract-only snapshot of a plausible bias audit whose headline turns on a census baseline that life-event narratives may not satisfy. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a comparative audit design: the authors generate a large corpus of stories using prompts that vary in how strongly they encourage diversity, then code the religion and caste of story characters and compare the resulting distributions against census population baselines. The comparison against the census baseline is what converts raw story counts into a measure of over- or underrepresentation, and the variation in prompt strength is what tests whether bias can be dislodged by instruction.
What would settle it
A direct test would be to collect demographic data on actual participants in the life events being narrated (e.g., attendees at weddings by religion and caste in India) and compare the model's story character distributions to those event-specific baselines; if the model's distributions match those baselines, the claimed overrepresentation would not be bias but a reflection of the event's real demographics.
Extended reading notes
Core claim
The paper's central claim is that GPT-4 Turbo's representations of caste and religion in narrative text are systematically skewed: when asked to tell stories about weddings, births, and other life events in India, the model assigns characters to dominant religious and caste groups at rates far above their census population shares. The overrepresentation persists across a spectrum of prompts, from neutral ones to those explicitly asking for diversity, and the effect of such prompts is limited and inconsistent. This leads the authors to argue that representational bias in LLMs is not a shallow prompt-level artifact but is encoded deeply in the model's behavior, with a winner-take-all quality that may be more extreme than the distributional bias in the training data itself. The paper thus proposes that correcting bias requires more than resampling training data; it calls for fundamental changes in how models are developed and evaluated.
Load-bearing premise
The study assumes that the right benchmark for a story about a life event in India is the national census distribution of religion and caste, so that any deviation counts as bias; if the life-event context itself skews toward particular groups, the overrepresentation measure would be an artifact of that baseline.
Editorial extensions
If this is right
- If GPT-4 Turbo overrepresents dominant groups in India, then applications built on it may reproduce that skew in any generated narrative content, from marketing to educational materials.
- The limited and inconsistent effect of diversity prompts suggests that simple prompt engineering is unlikely to be a sufficient mitigation strategy.
- The winner-take-all pattern suggests the model's bias may be more extreme than the statistical skew of its training data, implying that data rebalancing alone may not fix representation.
- The audit method extends existing bias measurements from single-turn, Global North-centric identities to multi-turn narrative generation and caste/religion, offering a template for auditing other understudied identities.
Reading between the lines
- A direct extension of this audit would be to test event-specific baselines: if wedding stories, for instance, are compared against the demographics of actual wedding participants rather than the national census, the magnitude of the claimed overrepresentation could change substantially.
- The same audit design could be applied to other countries with official census categories, such as ethnicity in the UK or race in the US, to see whether winner-take-all bias is a general property of LLMs or specific to the Indian context.
- A comparison of model outputs with human-authored stories about the same life events would help separate model bias from genre conventions that may naturally concentrate narratives on certain groups.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript reports an audit of GPT-4 Turbo in which the model generated over 7,200 stories about significant life events in India. The authors compare the distribution of caste and religion in these stories against census population shares and report a consistent overrepresentation of culturally dominant groups that is not dislodged by diversity-encouraging prompts. They conclude that representational bias is 'deep' and that data diversification alone is unlikely to correct it.
Significance. If the findings are robust, the paper makes a valuable contribution by extending bias audits beyond Global North identities to caste and religion, which are underexplored. The dataset and codebook link is a strength, and the empirical scope (over 7,200 generated stories) is non-trivial. However, the abstract alone does not provide enough statistical or methodological detail to verify the central claims, so the significance is conditional on the full methodology.
major comments (4)
- [Abstract] The abstract states that GPT-4 Turbo responses 'consistently overrepresent culturally dominant groups far beyond their statistical representation,' but it reports no effect sizes, confidence intervals, or statistical tests. This is a load-bearing omission because the entire conclusion rests on the magnitude and consistency of the gap. Please provide the quantitative measures (e.g., overrepresentation ratios, chi-square or log-linear analyses) either in the abstract or by pointing to a table.
- [Abstract] The comparison baseline is the 'actual population distribution in India as recorded in census data.' The abstract does not justify why the census composition is the appropriate expected distribution for stories about weddings, deaths, and other life events. If narrative contexts systematically condition on region, religion, or caste, the overrepresentation finding may reflect prompt-task framing rather than model bias. The methodology must either use conditional baselines (e.g., state/region-specific shares, event-specific demographics) or explicitly argue why the national unconditional distribution is the correct null.
- [Abstract] The claim that the model is 'more biased than the likely distribution bias in their training data' is unfalsifiable as stated: no training-data distribution is measured, and the qualifier 'likely' marks it as speculation. Please either operationalize a comparison to a measurable proxy for training-data bias (e.g., a corpus-based estimate) or remove this claim from the abstract.
- [Abstract] The abstract does not describe the prompt protocol: how many prompt variants, what 'encourage diversity to varying extents' means, whether prompts were repeated, or how the 'stickiness' was operationalized. Without this, the claim of 'limited and inconsistent efficacy' of nudges cannot be evaluated.
minor comments (4)
- [Abstract] The abstract says 'over 7,200 stories'; please give the exact number and the number per condition.
- [Abstract] The census year should be cited (e.g., 2011 Census of India) and the caste category (SC/ST/OBC vs. general) should be defined.
- [Abstract] The model version should be specified (e.g., GPT-4 Turbo via API, with date) and sampling parameters (temperature, top-p) reported.
- [Abstract] The abstract could benefit from a sentence describing the codebook and dataset release, which is already linked.
Circularity Check
No circularity: the census baseline is an external benchmark and no fitted parameter is renamed as a prediction.
full rationale
The abstract-only manuscript makes an external comparison: it prompts GPT-4 Turbo to generate stories and compares the religion and caste distribution of the characters to Indian census population data. This is a benchmark comparison against an independent, externally defined distribution, not a quantity derived from the model outputs or from the authors' own assumptions. There is no fitted parameter that is later called a prediction, no self-citation used as a load-bearing justification, and no equation or definition that equates the measured outcome with the input by construction. The weakest point in the paper is the unstated assumption that census population shares are the correct expected baseline for life-event narratives such as weddings; that is a validity or correctness concern, not a circularity concern. The abstract also asserts that the model is 'more biased than the likely distribution bias in their training data,' but no training-data distribution is measured. This claim is unfalsifiable from the provided information and is an evidentiary gap, but it is not circular: it is an unsupported estimate, not a result that reduces to its own inputs. Overall, the derivation chain is not circular, and the appropriate score is 0.
Assumptions & free parameters
assumptions (2)
- domain assumption Census population proportions are the correct baseline for representational fairness in life-event stories.
- domain assumption Character religion and caste in generated stories are identifiable and countable from text.
Cite this review
Pith. "Pith review of How Deep Is Representational Bias in LLMs? The Cases of Caste and Religion." pith.science (2026). https://pith.science/paper/VLM26PUJ
@misc{pith2026250803712,
author = {Pith},
title = {Pith review of: How Deep Is Representational Bias in LLMs? The Cases of Caste and Religion},
year = {2026},
howpublished = {\url{https://pith.science/paper/VLM26PUJ}},
note = {Machine review of arXiv:2508.03712}
}
read the original abstract
Representational bias in large language models (LLMs) has predominantly been measured through single-response interactions and has focused on Global North-centric identities like race and gender. We expand on that research by conducting a systematic audit of GPT-4 Turbo to reveal how deeply encoded representational biases are and how they extend to less-explored dimensions of identity. We prompt GPT-4 Turbo to generate over 7,200 stories about significant life events (such as weddings) in India, using prompts designed to encourage diversity to varying extents. Comparing the diversity of religious and caste representation in the outputs against the actual population distribution in India as recorded in census data, we quantify the presence and "stickiness" of representational bias in the LLM for religion and caste. We find that GPT-4 responses consistently overrepresent culturally dominant groups far beyond their statistical representation, despite prompts intended to encourage representational diversity. Our findings also suggest that representational bias in LLMs has a winner-take-all quality that is more biased than the likely distribution bias in their training data, and repeated prompt-based nudges have limited and inconsistent efficacy in dislodging these biases. These results suggest that diversifying training data alone may not be sufficient to correct LLM bias, highlighting the need for more fundamental changes in model development. Dataset and Codebook: https://github.com/agrimaseth/How-Deep-Is-Representational-Bias-in-LLMs
Forward citations
Cited by 1 Pith paper
-
Validating LLMs in social science: Epistemic threats and emerging norms
In 50 LLM measurement tasks from 27 top-journal papers, LLM outputs are often central to claims yet validation is limited, mostly convergent, and frequently incomplete.
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.