Documents Are People and Words Are Items: A Psychometric Approach to Textual Data with Contextual Embeddings

Jinsong Chen

REVIEW 4 major objections 6 minor 1 cited by

This paper proposes that a document-by-word matrix of dot-product scores between contextual embeddings and document embeddings can be treated as psychometric response data, and that factor analysis of this matrix on a STEM Wikipedia corpus

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

By scoring every document-word pair with contextual BERT embeddings and factor-analyzing the resulting matrix, the paper extracts interpretable latent knowledge dimensions from Wikipedia STEM articles.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection A promising psychometric framing for text embeddings, but the core assumption that contextual scores for absent words are document-sensitive is unvalidated and likely testable; worth refereeing, not worth citing yet. the 4 major comments →

arxiv 2509.08920 v1 pith:36UQJBQI submitted 2025-09-10 cs.CL stat.APstat.ME

Documents Are People and Words Are Items: A Psychometric Approach to Textual Data with Contextual Embeddings

Jinsong Chen This is my paper

classification cs.CL stat.APstat.ME MSC 62H2562P1568T50

keywords contextual embeddingspsychometric modelingfactor analysislatent factorscontextual scoredocument-word response matrixbifactor analysisclassical item analysis

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish a bridge between large language models and psychometrics. It treats each document as an individual and each common keyword as an item, then constructs a numeric 'contextual score' for every word–document pair by taking the dot product of the word's conditional contextual embedding with the document's embedding. These scores become the response matrix for factor analysis. Applied to roughly 20,000 STEM Wikipedia articles and 1,423 keywords, the analysis yields 60 narrow first-order factors and 13 broader second-order factors, with a bifactor version producing interpretable general factors labeled by top-loading words such as chemical elements, astronomy, programming, and physiology. If the construction holds, any text corpus can be turned into a measurement instrument for latent knowledge constructs.

Core claim

The central claim is that contextual scores—defined as the dot product between a word's conditional contextual embedding and a document's embedding—behave like item responses: they are roughly normal, linearly related for most word pairs, stable across random halves of the corpus, and correlated in ways that reveal latent structure. Factor analysis of the document-by-word matrix then yields factors that the author interprets as knowledge dimensions in STEM. The paper further shows that a second-order factor analysis and a Schmid-Leiman bifactor transformation separate a small number of reliable general factors from a large number of narrow minor factors, and that top-loaded keywords define e

What carries the argument

The central object is the conditional contextual embedding (CCE): for a word w_j and a document d_i, the language model is given a rephrased prompt such as 'find the contextual meaning of w_j given the following context: d_i' and produces a vector W_ij. This is dotted with the document embedding H_i (mean-pooled token embeddings, or the [CLS] token) to give the contextual score Y_ij = W_ij · H_i. This single scalar per word–document pair is the response datum; the full matrix Y is then analyzed by factor analysis on its covariance/correlation structure, with higher-order and bifactor extensions to handle a large number of narrow factors.

Load-bearing premise

The method works only if a word absent from a document can be assigned a meaningful contextual embedding by rephrasing the document as a prompt, so that the resulting dot-product score carries real information about how that word relates to the document.

What would settle it

Randomly shuffle the order of sentences within each document (destroying local context) and recompute scores; if the factor structure remains largely unchanged, the scores are not genuinely capturing contextual meaning. Alternatively, ask human raters whether an absent word like 'quantum' is relevant to a given cooking article and check whether its contextual score agrees with those ratings better than chance.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

Documents can be assigned factor scores, turning a corpus into a quantitative measurement of latent knowledge dimensions.
Because word items can be analyzed with classical item statistics, keyword selection and scale construction can be done with psychometric criteria (item-total correlations, convergent/discriminant validity) rather than raw frequencies.
The same framework extends to any text-rich field, allowing educators, psychologists, and lawyers to measure latent constructs in their own corpora.
Second-order factors remain stable in number across full and filtered data, suggesting the method can recover a robust high-level structure from noisy local word correlations.
The bifactor solution, separating general from minor factors, provides a way to interpret the large number of narrow factors that text data tends to produce.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the conditional contextual embedding for an absent word is never empirically validated against human judgments; a simple test would be to compare scores for known synonymous and antonymous word-document pairs.
Editorial inference: because dot-product scores scale with document embedding norm, longer documents may systematically inflate scores; cosine similarity, which the paper mentions but does not adopt, would remove this artifact and is a natural alternative to test.
Editorial inference: the method could be inverted to build a diagnostic instrument—selecting words that discriminate between documents, then using a person's own text to estimate their standing on the same factors—which the paper only gestures toward in its discussion of education.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A promising psychometric framing for text embeddings, but the core assumption that contextual scores for absent words are document-sensitive is unvalidated and likely testable; worth refereeing, not worth citing yet.

read the letter

Colleague,

The paper's real contribution is a concrete pipeline: define a contextual score Y_ij = W_ij · H_i (Eq. 1) where W_ij is a word's embedding generated from a prompt that names the word and provides the entire document, and H_i is a document embedding; then treat the resulting word-by-document matrix as item-response data and run factor analysis. The empirical demonstration on Wiki STEM is clearly described, and the authors deserve credit for checking rephrasing invariance, full-vs-filtered stability, and split-half distribution stability, and for flagging the unreliability of first-order factors and the overextraction issue. The OSF supplements are a plus.

The soft spot is load-bearing. The paper never tests whether W_ij actually varies with the document in a way that matters. The rephrasing correlations in Table 1 only show that the prompt form doesn't change the score; they don't show that the score is sensitive to the document content. The null hypothesis is that W_ij ≈ e_j (the word's static embedding), so Y_ij ≈ e_j · H_i. In that case the entire factor structure is just the geometry of document embeddings H_i, and the "contextual" part of the CCE is decorative. The authors report that [CLS]-based scores are highly collinear (average r ≈ .63), which is consistent with a document-dominated score. They discard those, but the mean-pooling scores could still be dominated by H_i times a fixed word direction plus noise.

This is not a fatal flaw in the writing; the paper is honestly exploratory. But it means the central claim—that factor analysis on these scores uncovers latent knowledge dimensions—is not yet supported. The remedy is simple and cheap: replace W_ij with a static embedding (GloVe or a context-free BERT word vector) and see whether the factor solution changes. If it doesn't, the method is just factor analysis on document embeddings, which is not new. Also, the post-hoc removal of word pairs changes the first-order factors (60 to 53), and the ChatGPT labels need expert validation; the authors don't hide these issues.

Who is this for? Anyone working at the intersection of psychometrics and NLP, especially on automated scoring, text mining for education, or measurement invariance with text. The paper deserves a serious referee—the method is novel and the flaw is testable. I would send it to review with a request for a static baseline, external validation, and sensitivity analyses. As it stands, I would not cite it in my own work yet.

Referee Report

4 major / 6 minor

Summary. The paper proposes a two-stage pipeline for psychometric analysis of textual data. Documents are treated as persons, words as items, and a 'contextual score' is defined in Eq. (1) as Y_ij = W_ij · H_i, the dot product of a word-conditional contextual embedding (CCE) and a document embedding. For words not present in a document, the CCE is generated from a rephrased prompt of the form 'find the contextual meaning of \'word\' given the following context: {document}'. The resulting document-by-word response matrix is then submitted to exploratory factor analysis, second-order factor analysis, and Schmid-Leiman bifactor analysis on the Wiki STEM corpus (20,000 documents, 1,423 words). The paper reports 60/53 first-order factors, 13 second-order factors, and 13 general factors labeled by ChatGPT, and it illustrates classical item analysis. The central claim is that this method can uncover latent knowledge dimensions and assign factor scores to documents.

Significance. If the construction were valid, the paper would offer a useful bridge between LLM contextual embeddings and psychometric latent-variable modeling, with potential applications in education, psychology, and law. The pipeline is clearly described, the reporting of factor counts and loadings is transparent, and the inclusion of full/filtered comparisons, split-half density checks, and classical item analysis shows methodological care. The key novelty—treating absent words as items via conditional contextual embeddings—is, however, not yet supported by evidence of context sensitivity. A static-embedding baseline could reproduce the entire factor solution, in which case the reported 'latent knowledge dimensions' would be artifacts of document-embedding geometry. The lack of external validation also limits the strength of the central claim. These issues are addressable with additional experiments; the paper is a reasonable exploratory proposal but does not yet establish its main assertion.

major comments (4)

[Eq. (1), 'Obtaining contextual scores as response data'] The construction assumes that W_ij, the CCE of word j conditioned on document i, carries document-specific signal for words absent from document i. The only evidence offered (Table 1) is that six rephrasing prompts give highly correlated scores; this is prompt invariance, not context sensitivity. The null model is W_ij ≈ e_j + noise, where e_j is a word-identity vector independent of the document. Then Y_ij ≈ e_j · H_i, and the item covariance matrix is determined by the covariance of document embeddings H_i. Under this null, the 60/53 first-order factors and 13 general factors are linear combinations of H_i, not 'latent knowledge dimensions'. This is load-bearing because the entire factor solution is computed from Y. To rule out this null, report a static-embedding baseline (replace W_ij with a word embedding not conditioned on the document) and compare factor structures; also report a
[Section 'Bifactor Analysis', Tables 5–7] Factor labels are assigned by ChatGPT from the top-loading words in the same dataset that produced the factors. No external criterion validates the factors as 'latent knowledge dimensions' rather than topic/word-co-occurrence clusters. The split-half stability check (Figure 4) only concerns marginal score distributions, not the factor structure. To support the central claim, compare factor scores with external document categories (e.g., Wikipedia portals or article ratings), contrast the factor solution with a standard topic model (LDA/NMF) and show added value, and report label reliability across multiple raters and prompt variants. Without such evidence, the interpretation remains post hoc.
[Section 'Stage Two: Conducting Factor Analysis', Table 2 and Figure 7] The statement that 13 second-order factors are 'reliably extracted' is based only on the same count appearing for full and filtered data. This does not demonstrate that the factors correspond. The first-order solutions differ in number (60 vs. 53) and in item composition, and the second-order factors account for only about 42% of first-order variance. Report factor congruence coefficients (e.g., Tucker's phi) after Procrustes alignment between full and filtered solutions, and ideally across other preprocessing choices (n=10, min occurrence 10, r>.8 threshold). Without this, the reliability claim for the higher-order structure is unsupported.
[Data preprocessing, 'Data Preprocessing before contextual score' and 'Experiment'] Several free parameters (n_keywords_per_document=10, min_occurrences=10, correlation-removal threshold r>.8, document length 50–500, first 20k documents) are set without sensitivity analysis. Since these choices alter the item pool and the factor count (60 vs. 53 in Table 2), the robustness of the reported 13 general factors is unclear. The exploratory status of the paper reduces, but does not eliminate, the need for a sensitivity analysis when the central claim is a 'demonstration' of the method's potential.

minor comments (6)

[Eq. (1)] The symbol ⊙ typically denotes the Hadamard (elementwise) product; the paper appears to mean the dot product. Please use ⋅ or the word 'dot product' to avoid ambiguity.
[Appendix A, Figure 8; Appendix B, Figure 9] The captions say 'Full Data' but the figures correspond to the filtered data. Please correct the captions.
[Section 'Experiment Stage One'] 'high performing computing' should read 'high-performance computing'.
[Table 7 note] There is a typo 'in in Table 7'. Also, 'Labels are suggested by ChatGPT' should include the exact model version and prompt for reproducibility.
[Section 'Classical Item Analysis'] The term 'multi-trait multi-method' is a strong claim; the reported within/between item-total correlations are not a full MTMM analysis. Please soften the wording or provide the actual multitrait-multimethod matrix.
[Section 'Data Preprocessing before contextual score'] The choice of TF-IDF over TextRank and RAKE is justified only by 'can extract more keywords'; provide the comparison counts or a reference to support the claim.

Circularity Check

0 steps flagged

No significant circularity: the factor-analytic pipeline is exploratory and does not reduce to its inputs; self-citations are future-work suggestions only.

full rationale

The paper's derivation chain is operational rather than circular. It defines contextual scores in Eq. (1) as Y_ij = W_ij · H_i, then applies exploratory factor analysis to the covariance/correlation matrix of those scores. The factors are, by the standard EFA model in Eqs. (2)-(4), extracted from the score matrix itself; this is a descriptive statistical decomposition, not a prediction of an external criterion from fitted inputs. Interpreting factors via their top-loading words is ordinary EFA practice, not a circularity: the factor labels are post hoc summaries of the loadings, not inputs to the estimation. The paper explicitly labels the approach exploratory and notes the absence of confirmatory theory ('the existing method is primarily exploratory due to the absence of hypothesis or theoretical support'), so no claim is made that the factors were independently predicted and then verified. The split-half density comparison (Fig. 4) supports distributional stability but is not used to complete a circular argument. The self-citations (Chen, 2022; Chen et al., 2021; Chen & Zhang, 2024) appear only in the Discussion as optional future extensions of the method, not as load-bearing evidence for the current results. The skeptical concern that absent-word CCEs may be dominated by document-level information, making factor structure an artifact of H_i, is a substantive validity threat, but it is not a circularity: even under that null hypothesis, the derivation would be wrong because the scores encode the wrong quantity, not because the conclusion was assumed in the inputs. No equation reduces by construction to its own output, no fitted parameter is renamed as a prediction, and no load-bearing claim rests on a self-citation chain.

Axiom & Free-Parameter Ledger

7 free parameters · 6 axioms · 3 invented entities

The paper introduces a new measurement construct (contextual score) and a new embedding object (CCE), both defined entirely within the paper. The free parameters are all hand-set thresholds and data-dependent factor counts. The axioms are mostly standard statistical assumptions combined with the domain assumption that BERT embeddings capture meaning; the weakest is the ad hoc choice of dot product as the similarity measure.

free parameters (7)

n_keywords_per_document = 10
TF-IDF top-n keywords per document set to 10, described as 'often used' in exploratory analyses, but no sensitivity analysis is provided.
min_occurrences = 10
Keywords must occur in at least 10 documents to enter the analysis, chosen to balance computational cost and usefulness.
correlation_removal_threshold = 0.8
Word pairs with correlation above 0.8 had one member removed to reduce multicollinearity; the threshold is post hoc and changes the factor solution from 60 to 53 factors.
document_length_range = 50-500 tokens
Documents outside this range are dropped due to BERT limitations or insufficient information; arbitrary filter.
document_subset_size = 20000
Only the first 20,000 documents are used, with the claim that most keywords are obtained; this is a computational convenience.
number_of_first_order_factors = 60 (full) / 53 (filtered)
Determined by parallel analysis; the paper itself notes the >85% variance explained suggests overextraction.
number_of_second_order_factors = 13
Determined by parallel analysis on first-order factor correlations; consistent across full and filtered data.

axioms (6)

domain assumption BERT contextual embeddings capture contextual word meaning sufficiently for measuring semantic alignment.
The first stage relies on BERT-large to produce conditional contextual embeddings; no independent check against human meaning judgments is performed.
ad hoc to paper The dot product of a word-level CCE and a document embedding is a valid scalar measure of word-document alignment.
Introduced in Eq. (1); chosen because it yields a wider range and approximate normality, but no theoretical or empirical justification is provided.
domain assumption Contextual scores satisfy item-individual independence: a word's score distribution is stable across documents.
Checked only via split-half density overlap (Figure 4), not via formal measurement invariance or DIF tests.
domain assumption Normality and linearity assumptions for factor analysis are approximately satisfied with N=20,000.
Supported only by scatter and density plots for a few example words; no formal normality tests are reported.
domain assumption Parallel analysis correctly identifies the number of factors in high-dimensional, correlated textual data.
The paper notes that other methods fail or give unreasonable numbers, and the resulting 60 factors explain >85% variance, which the authors themselves flag as likely overextraction.
standard math The Schmid-Leiman proportionality constraint holds approximately for the exploratory bifactor re-estimation.
Explicitly acknowledged as approximate in Eqs. (10)-(11); the transformation is standard in psychometrics but its application here is not validated.

invented entities (3)

Conditional Contextual Embedding (CCE) no independent evidence
purpose: A word embedding conditioned on a specific document, used to compute contextual scores for word-document pairs.
Introduced in this paper; no external validation that CCEs for words absent from a document are meaningful.
Contextual score no independent evidence
purpose: Scalar response value Y_ij = W_ij · H_i measuring alignment between word j and document i in Eq. (1).
The core new measurement; its validity is the central claim but is only demonstrated internally.
Latent knowledge dimensions no independent evidence
purpose: General factors in the bifactor model, labeled via ChatGPT top words, intended to represent knowledge domains in the corpus.
No external criterion (e.g., document metadata, expert ratings) is used to confirm that the factors correspond to real knowledge dimensions.

reviewed 2026-08-04 · how reviews work

0 comments

Cite this review

Pith. "Pith review of Documents Are People and Words Are Items: A Psychometric Approach to Textual Data with Contextual Embeddings." pith.science (2026). https://pith.science/paper/36UQJBQI

@misc{pith2026250908920,
  author       = {Pith},
  title        = {Pith review of: Documents Are People and Words Are Items: A Psychometric Approach to Textual Data with Contextual Embeddings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/36UQJBQI}},
  note         = {Machine review of arXiv:2509.08920}
}

Share X Bluesky LinkedIn Reddit HN

read the original abstract

This research introduces a novel psychometric method for analyzing textual data using large language models. By leveraging contextual embeddings to create contextual scores, we transform textual data into response data suitable for psychometric analysis. Treating documents as individuals and words as items, this approach provides a natural psychometric interpretation under the assumption that certain keywords, whose contextual meanings vary significantly across documents, can effectively differentiate documents within a corpus. The modeling process comprises two stages: obtaining contextual scores and performing psychometric analysis. In the first stage, we utilize natural language processing techniques and encoder based transformer models to identify common keywords and generate contextual scores. In the second stage, we employ various types of factor analysis, including exploratory and bifactor models, to extract and define latent factors, determine factor correlations, and identify the most significant words associated with each factor. Applied to the Wiki STEM corpus, our experimental results demonstrate the method's potential to uncover latent knowledge dimensions and patterns within textual data. This approach not only enhances the psychometric analysis of textual data but also holds promise for applications in fields rich in textual information, such as education, psychology, and law.

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

From Text to Parameters: Predicting Item Parameters from Embedding Regularization with Reliability and Design Ceilings
cs.CL 2026-07 accept novelty 6.0

Text embeddings recover 57-63% of the reliable variance in exam-item difficulty, and apparent differences in predictability across IRT parameters are mostly artifacts of calibration noise rather than text signal.

Reference graph

Works this paper leans on

5 extracted references · 2 linked inside Pith · cited by 1 Pith paper

[1]

Adhikari, A., Ram, A., Tang, R., & Lin, J. (2019). Rethinking complex neural network architectures for document classification.Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4046–4051. Akbik, A., Blythe, D., & Vollgraf, R. (20...

2019
[210]

32 Chen, J., & Zhang, Y. (2024). Research design and model estimation under the partially confirmatory latent variable modeling framework with multi-univariate bayesian lassos.Structural Equation Modeling: A Multidisciplinary Journal, 1–15. Chowdhary, K., & Chowdhary, K. (2020). Natural language processing.Fundamentals of artificial intelligence, 603–649....

Pith/arXiv arXiv 2024
[272]

(2007).The text mining handbook: Advanced approaches in analyzing unstructured data

Feldman, R., & Sanger, J. (2007).The text mining handbook: Advanced approaches in analyzing unstructured data. Cambridge university press. Gorsuch, R. L. (2014).Factor analysis: Classic edition. Routledge. Jennrich, R. I., & Bentler, P. M. (2011). Exploratory bi-factor analysis.Psychometrika,76, 537–549. Jurafsky, D., & Martin, J. H. (2008). Speech and la...

Pith/arXiv arXiv 2007
[450]

D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners.Advances in neural information processing systems,33, 1877–1901. Browne, M. W. (2001). An overview of analytic rotation in exploratory factor analysis. Multivariate behavioral re...

2020
[468]

O’Reilly Media, Inc

Bird, S., Klein, E., & Loper, E. (2009).Natural language processing with python: Analyzing text with the natural language toolkit. " O’Reilly Media, Inc." Braeken, J., & Van Assen, M. A. (2017). An empirical kaiser criterion.Psychological methods,22(3),

2009

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.

[1] [1]

Adhikari, A., Ram, A., Tang, R., & Lin, J. (2019). Rethinking complex neural network architectures for document classification.Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4046–4051. Akbik, A., Blythe, D., & Vollgraf, R. (20...

2019

[2] [210]

32 Chen, J., & Zhang, Y. (2024). Research design and model estimation under the partially confirmatory latent variable modeling framework with multi-univariate bayesian lassos.Structural Equation Modeling: A Multidisciplinary Journal, 1–15. Chowdhary, K., & Chowdhary, K. (2020). Natural language processing.Fundamentals of artificial intelligence, 603–649....

Pith/arXiv arXiv 2024

[3] [272]

(2007).The text mining handbook: Advanced approaches in analyzing unstructured data

Feldman, R., & Sanger, J. (2007).The text mining handbook: Advanced approaches in analyzing unstructured data. Cambridge university press. Gorsuch, R. L. (2014).Factor analysis: Classic edition. Routledge. Jennrich, R. I., & Bentler, P. M. (2011). Exploratory bi-factor analysis.Psychometrika,76, 537–549. Jurafsky, D., & Martin, J. H. (2008). Speech and la...

Pith/arXiv arXiv 2007

[4] [450]

D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners.Advances in neural information processing systems,33, 1877–1901. Browne, M. W. (2001). An overview of analytic rotation in exploratory factor analysis. Multivariate behavioral re...

2020

[5] [468]

O’Reilly Media, Inc

Bird, S., Klein, E., & Loper, E. (2009).Natural language processing with python: Analyzing text with the natural language toolkit. " O’Reilly Media, Inc." Braeken, J., & Van Assen, M. A. (2017). An empirical kaiser criterion.Psychological methods,22(3),

2009

Documents Are People and Words Are Items: A Psychometric Approach to Textual Data with Contextual Embeddings

The reading

Core claim §

What carries the argument

Load-bearing premise §

What would settle it §

If this is right

Where Pith is reading between the lines

discussion (0)

Core claim

Load-bearing premise

What would settle it