Pith. sign in

REVIEW 3 major objections 5 minor 15 references

Evaluating Personality Traits in Large Language Models: Insights from Psychological Questionnaires

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Large language models show distinct, measurable personality profiles in psychological questionnaires.

desk verdict The central 'distinct profiles within the same family' claim rests on a scale-mixing error that the paper itself flags elsewhere; the qualitative pattern is plausible but the headline result needs reanalysis on a common scale. read the letter →

arxiv 2502.05248 v1 pith:SCRI4GRI submitted 2025-02-07 cs.CL cs.AIcs.MA

Pranav Bhandari , Usman Naseem , Amitava Datta , Nicolas Fay , Mehwish Nasim This is my paper · ORCID
classification cs.CLcs.AIcs.MA
keywords largelanguagemodelspersonalitytraitsBigFivepsychometricstrainingdatacontaminationcoefficientofvariationquestionnaireparaphrasingrole-playingagents
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether large language models have measurable personality traits, and answers yes. Using five established Big Five questionnaires (BFI, HEXACO, TIPI, MINI-IPIP, and IPIP-NEO-60), the authors reword every item to avoid the models having memorized the originals, then run the tests many times at near-zero temperature. They find that all tested models score high on Agreeableness and low on Neuroticism, while Openness and Conscientiousness are generally high, and that models from the same family can nonetheless show different dominant traits. The paper's central claim is that these profiles are stable enough to compare across models, and that psychometric tools built for humans transfer to LLMs when contamination is addressed.

What carries the argument

The argument rests on a small battery of trait inventories: the Big Five Inventory (44 items), HEXACO (100), TIPI (10), MINI-IPIP (20), and IPIP-NEO-60 (60), each scored on Likert scales. To guard against memorization of test items, each question was reworded by GPT-4o and accepted only if a sentence-similarity model gave a cosine similarity of at least 0.7, with human supervision for failures. Questionnaires were then administered in randomized order in batches of ten, repeated 100 times per model at minimum temperature. The coefficient of variation across those runs is the metric that separates stable dimensions (low CV) from unstable ones (high CV).

What would settle it

Run the original and the paraphrased versions of the same questionnaires on the same models; if per-dimension mean scores differ by more than the run-to-run variability observed at minimum temperature, the paraphrase step is not preserving the psychometric construct. A second check: have human raters classify the paraphrased items by Big Five dimension; if agreement with the original item assignments is poor, the 0.7 threshold is too weak.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that LLMs exhibit unique dominant traits, varying characteristics, and distinct personality profiles even within the same model family. Mean scores across five inventories place Agreeableness highest and Neuroticism lowest for all models tested; GPT-4 and GPT-4o-mini are most Agreeable, Llama-3-8B-Instruct is most Conscientious, Llama-3.1-8B-Instruct is most Open, and Llama-3.2-3B-Instruct is most Agreeable. Across repeated runs, Neuroticism had the highest coefficient of variation for most models, while Extraversion and Agreeableness were the most consistent dimensions. The authors interpret this as evidence that LLM alignment and training choices leave a detectable, comparable personality signature.

Load-bearing premise

The entire analysis depends on the assumption that the reworded questions still ask the same thing as the originals; if the rewording changes what is being measured, none of the personality scores can be trusted.

Editorial extensions

If this is right

  • If the central claim holds, personality testing of LLMs can be done with standard human questionnaires, as long as items are paraphrased and validated first.
  • Same-family models with different training or sizes can be compared by their dominant trait; for example, Llama-3, Llama-3.1, and Llama-3.2 each show a different dominant dimension.
  • Dimensions with low coefficient of variation (Extraversion, Agreeableness) can be reported with confidence, whereas Neuroticism scores from a single inventory should be treated as unreliable.
  • The method yields a contamination-resistant baseline for future model releases: run the five inventories, report mean scores, CV, and dominant dimension.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 0.7 cosine-similarity threshold is a computational proxy for meaning preservation; a small human-rating study of the paraphrased items against the original trait definitions would tell whether the proxy is adequate.
  • Because Neuroticism is the most variable dimension, emotional-stability scores may be partly an artifact of questionnaire format; a test that swaps ordering, adds filler items, or uses forced-choice formats could reveal this.
  • The high Agreeableness and low Neuroticism pattern across all models may reflect safety-oriented fine-tuning rather than an inherent disposition; comparing base and instruction-tuned checkpoints of the same model would separate those explanations.
  • One practical extension: the same contamination-resistant protocol could be used to track whether personality profiles drift across model updates, treating 'personality drift' as a regression-testable property.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper administers five established personality questionnaires (BFI, HEXACO, TIPI, MINI-IPIP, and IPIP-NEO-60) to five instruction-tuned LLMs (GPT-4, GPT-4o-mini, Llama-3-8B-Instruct, Llama-3.1-8B-Instruct, and Llama-3.2-3B-Instruct). To mitigate training-data contamination, the authors paraphrase each item with GPT-4o and accept rewrites whose cosine similarity to the original exceeds 0.7. Items are administered in random order over 100 runs at near-zero temperature, dimension scores are averaged per questionnaire, and the paper reports mean Likert scores in Table 2, coefficients of variation across questionnaires (excluding TIPI) in Section 3.2, and dimensional dominance based on the mean across all five questionnaires in Section 3.3. The authors conclude that LLMs show high Agreeableness, Openness, and Conscientiousness, low Neuroticism, and distinct dominance profiles within the Llama family.

Significance. The paper's strength is its systematic multi-instrument design: using five psychometric scales, randomizing item order, running many iterations at low temperature, and reporting variability across instruments is a more thorough protocol than earlier single-questionnaire studies. If the measurement assumptions hold, the approach could offer a cheap, reproducible way to profile alignment-relevant behavioural tendencies of new LLMs, and the within-family comparisons (Llama 3 vs. 3.1 vs. 3.2) are a useful design for isolating the effect of fine-tuning on elicited traits. However, the significance is conditional on (i) the cross-instrument comparability of raw Likert scores and (ii) the construct validity of the paraphrased items; the current manuscript does not establish either.

major comments (3)
  1. [Section 3.3, Table 3] The dimensional-dominance means in Table 3 average raw Likert scores from the 1–7 TIPI with 1–5 scores from BFI, HEXACO, MINI-IPIP, and IPIP-NEO without any normalization, even though Section 3.2 excludes TIPI from the CV analysis for precisely this scale mismatch. Recomputing the Table 3 means after excluding TIPI changes Llama-3.2's dominant dimension from Agreeableness (reported 4.22) to Openness (3.95) and reduces Llama-3's Conscientiousness lead to a tie with Agreeableness, so the headline claim of distinct personality profiles within the same Llama family is not supported on the current scale. Please recompute dominance on a common scale (for example, per-inventory z-scores) or exclude TIPI consistently, and update the text and the conclusion accordingly.
  2. [Section 2.1] The validation of paraphrased items using a cosine-similarity threshold of 0.7 with a generic sentence-transformer is not sufficient to establish that reworded items retain the original psychometric construct, and the manuscript does not report how many items fell below the threshold or what 'human supervision' involved. Because every downstream score, CV, and dominance ranking depends on this equivalence, please report the paraphrase-validation statistics (for example, number of items rewritten, examples of accepted and rejected items, inter-rater agreement on meaning preservation), and address the self-referential risk that GPT-4o, the paraphraser, comes from the same vendor as two of the evaluated models.
  3. [Sections 3.1 and 3.3] The paper reports no measures of uncertainty for the mean dimension scores or for the dominance rankings, despite collecting n=100 responses per model. Without standard errors, confidence intervals, or significance tests, differences such as Llama-3.2's 0.16-point advantage of Agreeableness over Openness in Table 3 cannot be distinguished from sampling variability; please report per-model and per-dimension standard deviations and standard errors, or explicitly re-frame the dominance results as descriptive.
minor comments (5)
  1. [Section 3.2] The phrase 'As mentioned in the methodology Section 3.3' should refer to Section 2.3, where the coefficient of variation is actually defined.
  2. [Table 1] The inventory listed as 'NEO-PI-R[8]' is actually the IPIP-NEO-60 (Maples-Keller et al., 2019); align the table label with the text and reference.
  3. [Conclusion] The statement 'Llama models highlight Conscientiousness or Openness' is inconsistent with Table 3, which assigns Llama-3.2 to Agreeableness; the summary should match the final dominance table, especially after the scale recomputation.
  4. [Section 3.2] The CV in Table 3 is computed over only four questionnaires per dimension after excluding TIPI, and these are different instruments with different item formats; the interpretation of Neuroticism's 'highest variability' should acknowledge that between-instrument differences contribute to this metric.
  5. [Throughout] There are minor typographical issues, including 'Macquaire University' for 'Macquarie University', and the acronyms Ex, Ag, Co, Ne, Op are defined only in the Table 2 caption; consider defining them at first use in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported personality profiles are direct summaries of LLM responses to externally published questionnaires.

full rationale

The paper's central claim is that LLMs exhibit measurable personality profiles. The derivation chain is: take five established, externally published personality inventories (BFI, HEXACO, TIPI, MINI-IPIP, IPIP-NEO), administer them to LLMs via prompts, score the Likert responses according to each inventory's published scoring rules, and summarize the resulting means, coefficients of variation, and dominant dimensions. The 'personality profile' is therefore an arithmetic summary of measured responses, not the output of a fitted model, so no fitted parameter is renamed as a prediction and no result is forced by construction. The only preprocessing step is paraphrasing questionnaire items with GPT-4o and accepting paraphrases with cosine similarity at least 0.7; this threshold is the authors' own choice and is not derived from the target personality scores, so it is a construct-validity concern rather than a circular step. The paper does not rely on a load-bearing self-citation: the cited questionnaires and prior LLM-personality studies are independent external work, and none of the load-bearing claims reduce to a citation by the present authors. The scale-mixing issue in Section 3.3, where TIPI's 1-7 Likert scores are averaged together with 1-5 scales despite the paper's own exclusion of TIPI from CV calculations in Section 3.2, is a real methodological flaw that may change the reported dominance ranking, but it is an aggregation error, not circularity: the result is not equivalent to its input by definition. For these reasons, the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The ledger is short because the paper is empirical rather than theoretical. The main burden sits on two hand-made choices: the 0.7 paraphrase threshold and the unnormalized cross-inventory averaging. The axioms concern transferring human psychometric constructs to LLMs and treating self-reports as trait measurements.

free parameters (2)
  • Cosine similarity threshold for paraphrase acceptance = 0.7
    Hand-chosen cutoff in Section 2.1 to decide whether reworded items preserve original meaning; no sensitivity analysis, and it controls which items are used in all subsequent scores.
  • Aggregation of TIPI with 1-5 scale inventories = included unnormalized
    Section 3.3 computes dimensional dominance as the mean of raw scores across questionnaires, including TIPI's 1-7 Likert scale, which inflates TIPI-driven scores and affects the dominance conclusion.
assumptions (4)
  • domain assumption Personality questionnaires validated for humans preserve their psychometric properties when administered to LLMs.
    The whole methodology applies human instruments to LLMs without recalibration (Section 2).
  • ad hoc to paper Paraphrasing an item with GPT-4o and accepting it when cosine similarity >= 0.7 preserves construct equivalence.
    Stated in Section 2.1; no external validation of the paraphrased items.
  • ad hoc to paper Raw Likert scores from different inventories can be averaged to compare dimensions.
    Used in Section 3.3 to define dimensional dominance.
  • domain assumption LLM self-reports reflect stable latent traits rather than prompt framing or randomness.
    Required to interpret the outputs as personality profiles (Section 3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Personality Traits in Large Language Models: Insights from Psychological Questionnaires." pith.science (2026). https://pith.science/paper/SCRI4GRI

@misc{pith2026250205248,
  author       = {Pith},
  title        = {Pith review of: Evaluating Personality Traits in Large Language Models: Insights from Psychological Questionnaires},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SCRI4GRI}},
  note         = {Machine review of arXiv:2502.05248}
}
read the original abstract

Psychological assessment tools have long helped humans understand behavioural patterns. While Large Language Models (LLMs) can generate content comparable to that of humans, we explore whether they exhibit personality traits. To this end, this work applies psychological tools to LLMs in diverse scenarios to generate personality profiles. Using established trait-based questionnaires such as the Big Five Inventory and by addressing the possibility of training data contamination, we examine the dimensional variability and dominance of LLMs across five core personality dimensions: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism. Our findings reveal that LLMs exhibit unique dominant traits, varying characteristics, and distinct personality profiles even within the same family of models.

Figures

Figures reproduced from arXiv: 2502.05248 by the authors.

Figure 1
Figure 1. Methodology involves combining system and user [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Radar plots to visualise all five personality questionnaires across their dimensions, comparing results from all LLMs. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 5 canonical work pages

  1. [1]

    M Brent Donnellan, Frederick L Oswald, Brendan M Baird, and Richard E Lucas

    work page

  2. [2]

    Samuel D Gosling, Peter J Rentfrow, and William B Swann. 2003. A very brief measure of the Big-Five personality domains. Journal of Research in Personality 37, 6 (2003), 504–528. https://doi.org/10.1016/S0092-6566(03)00046-1

    doi:10.1016/s0092-6566(03)00046-1 2003

  3. [3]

    Thilo Hagendorff, Ishita Dasgupta, Marcel Binz, Stephanie C. Y. Chan, Andrew Lampinen, Jane X. Wang, Zeynep Akata, and Eric Schulz. 2024. Machine Psy- chology. arXiv:2303.13988 [cs.CL] https://arxiv.org/abs/2303.13988

    arXiv 2024

  4. [4]

    Guangyuan Jiang, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. 2023. Evaluating and Inducing Personality in Pre-trained Language Models. In Thirty-seventh Conference on Neural Information Processing Systems . https://openreview.net/forum?id=I9xE1Jsjfx Evaluating Personality Traits in Large Language Models: Insights from Psychological Qu...

    work page 2023

  5. [5]

    Hang Jiang, Xiajie Zhang, Xubo Cao, Cynthia Breazeal, Deb Roy, and Jad Kabbara

  6. [6]

    Oliver P John, Eileen M Donahue, and Robert L Kentle. 1991. Big five inventory. Journal of personality and social psychology (1991)

    1991

  7. [7]

    Yuan Li, Yue Huang, Hongyi Wang, Xiangliang Zhang, James Zou, and Lichao Sun. 2024. Quantifying ai psychology: A psychometrics benchmark for large language models. arXiv preprint arXiv:2406.17675 (2024)

    arXiv 2024

  8. [8]

    Jessica L Maples-Keller, Rachel L Williamson, Chelsea E Sleep, Nathan T Carter, W Keith Campbell, and Joshua D Miller. 2019. Using item response theory to develop a 60-item representation of the NEO PI–R using the International Personality Item Pool: Development of the IPIP–NEO–60. Journal of personality assessment 101, 1 (2019), 4–15

    work page 2019

Show all 15 references
  1. [9]

    Marilù Miotto, Nicola Rossberg, and Bennett Kleinberg. 2022. Who is GPT-3? An exploration of personality, values and demographics. In Proceedings of the Fifth Workshop on Natural Language Processing and Computational Social Science (NLP+CSS). Association for Computational Ling...

    2022 doi

  2. [10]

    Morten Moshagen, Isabel Thielmann, Benjamin E Hilbig, and Ingo Zettler. 2019. Meta-analytic investigations of the HEXACO Personality Inventory (-Revised). Zeitschrift für Psychologie (2019)

    2019

  3. [11]

    Greg Serapio-García, Mustafa Safdari, Clément Crepy, Luning Sun, Stephen Fitz, Peter Romero, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. 2023. Personality Traits in Large Language Models. arXiv:2307.00184 [cs.CL] https: //arxiv.org/abs/2307.00184

    2023 arXiv

  4. [12]

    Richard Shiffrin and Melanie Mitchell. 2023. Probing the psychol- ogy of AI models. Proceedings of the National Academy of Sciences 120, 10 (2023), e2300963120. https://doi.org/10.1073/pnas.2300963120 arXiv:https://www.pnas.org/doi/pdf/10.1073/pnas.2300963120

    2023 doi

  5. [13]

    Yamshchikov

    Aleksandra Sorokovikova, Natalia Fedorova, Sharwin Rezagholi, and Ivan P. Yamshchikov. 2024. LLMs Simulate Big Five Personality Traits: Further Evidence. arXiv:2402.01765 [cs.CL] https://arxiv.org/abs/2402.01765

    2024 arXiv

  6. [2006]

    Psychological assessment 18, 2 (2006), 192

    The mini-IPIP scales: tiny-yet-effective measures of the Big Five factors of personality. Psychological assessment 18, 2 (2006), 192

    2006

  7. [2024]

    In Findings of the Association for Computational Linguistics: NAACL 2024

    PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traits. In Findings of the Association for Computational Linguistics: NAACL 2024. Association for Computational Linguistics, Mexico City, Mexico, 3605–3627. https://doi.org/10.18653/v1/2024.f...

    2024 doi

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.