REVIEW 4 major objections 4 minor 1 cited by
Modelling Intertextuality with N-gram Embeddings
T0 review · 4 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read The paper claims that intertextuality between two texts can be quantified by averaging the cosine similarities of their n-gram embeddings above a threshold, and that this score can map influence across a large corpus.
desk verdict A simple, reproducible n-gram embedding similarity method for intertextuality, but the validation is too weak to support the central claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the intertextuality score defined by Equations (1)–(2): the mean of all pairwise cosine similarities, taken over n-gram embedding pairs whose absolute cosine exceeds a threshold tau. The n-grams are embedded as averaged word vectors from a CBOW model trained on a diachronic English corpus after rule-based spelling normalisation and lemmatisation. Tau is the adjustable filter that stops irrelevant near-orthogonal n-gram pairs from diluting the score, and the paper finds qualitative rankings are stable across thresholds from 0.01 to 0.20. A vectorised matrix formulation (Algorithm 1) computes all pairwise similarities as one dense matrix product, making the method fast en
What would settle it
Run the method on a deliberately allusive text pair and on a topically overlapping but non-allusive pair, such as two encyclopedia entries on the same subject. If the non-allusive pair scores as high or higher, or if known allusions across a curated benchmark generate no more high-threshold cosine pairs than randomly matched texts, the central claim is falsified.
Extended reading notes
Core claim
The central claim is that intertextuality can be modelled by Equation (1): for two texts A and B, Intertextuality(A,B) = (1/|S|) sum over S of s, where S is the set of pairwise cosine similarities between the vector representations of n-grams from each text that exceed a threshold tau in absolute value. The author grounds this in the semiotic account of intertextuality as references between signifieds, and argues semantic relatedness, computed distributionally, is the computable core. The n-gram level is chosen because single words are too general while longer phrases are more evocative; thresholding removes near-orthogonal, meaningless pairs. The model is shown to order four test texts as l
Load-bearing premise
The score assumes that the web of allusion and reference that literary critics call intertextuality shows up as high average semantic similarity between n-gram embeddings; if allusive relationships are not generally reflected in distributional similarity, the method measures topical proximity rather than intertextuality.
Editorial extensions
If this is right
- Any two texts, however far apart in time, genre, or language, can receive one intertextuality number, so literary influence can be mapped as a weighted network over hundreds or thousands of works.
- The score decomposes into concrete n-gram pairs, so a researcher can inspect exactly which local verbal echoes drive a high score rather than treating it as an opaque similarity value.
- Because the score averages many pairwise comparisons, it can be computed with vectorised matrix operations, allowing corpus-scale studies to run on a single workstation.
- The threshold offers a tunable trade-off between capturing distant thematic resonance at low tau and isolating near-quotations at high tau, letting researchers adapt the model to different questions.
- Community detection on the resulting graph surfaces groupings, such as all works of a single author falling together, which can serve either as validation or as hypotheses for close reading.
Reading between the lines
- A likely limit the paper does not fully confront is that the score measures distributional semantic overlap, not intentional allusion; two texts that independently treat the same topic could score high, so the metric may operationalise readerly intertextuality more than source-tracing.
- The method could be sharpened by weighting n-grams by evocativeness or salience rather than averaging all surviving pairs equally; the paper mentions this direction but leaves it open.
- The same pipeline could be tested cross-lingually with multilingual embeddings, since the score only requires a shared vector space; if it works, intertextuality between translated and source texts becomes measurable.
- A direct comparison against existing reuse detectors on a corpus with known allusions would clarify whether the semantic-average score adds signal beyond lexical matching.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a quantitative model of intertextuality: for two texts A and B, extract n-grams, embed them, compute pairwise cosine similarities, discard those below a threshold τ, and average the rest (Eqs. 1–2). The author trains a diachronic CBOW embedding model on the Oxford Text Archive after a rule-based normalisation/lemmatisation pipeline, evaluates it via a thesaurus graph hit rate, then validates the intertextuality measure on four texts with known intertextual relationships. A scalability experiment on 267 texts produces a directed-influence network and an undirected community structure, which are presented as further validation. The paper claims that the model is simple, interpretable, efficient, and captures intertextual degrees consistent with literary scholarship.
Significance. If the proposed measure genuinely tracks intertextuality rather than generic topical similarity, the paper would provide a scalable, interpretable tool for computational literary studies, with the added strength of releasing code and a trained model. The matrix-based implementation (Algorithm 1) and the normalisation pipeline are useful engineering contributions. However, the current evidence is insufficient to establish the central claim: the validation is small, lacks quantitative reporting and baselines, and is vulnerable to confounds (author, genre, period, training overlap). The potential significance is real, but the paper underdelivers on demonstration.
major comments (4)
- [§4, Eqs. (1)–(2)] The validation study does not provide numerical results. The text states that 'the results align with established literary interpretation' and refers to 'heat maps in Table 1', but no scores, error bars, or statistical tests are reported. The ranking among four texts—same-author Brontë novels highest, metaphysical poets next, cross-genre pairs lowest—is exactly what a generic topical or stylistic similarity score would produce. Without a baseline (e.g., TF-IDF cosine, word-overlap Jaccard, or a random embedding control) and without negative controls that are known to have low intertextuality yet high thematic overlap, the evidence that Eq. (1) measures intertextuality rather than semantic relatedness is not established. The manuscript itself concedes in §2 that 'semantic relatedness is the easiest to encapsulate in terms of computability', so this is a load-bearing concern, not a minor o
- [§3 and §4] The embedding model is trained on the Oxford Text Archive, which covers texts from the 16th to early 20th centuries. The four validation texts (Herbert, Donne, Brontë) fall squarely in that period, and the paper gives no indication that they were excluded from the training corpus. If they were included, the validation measures the model's ability to memorise or to encode in-domain similarity, not its generalisation to unseen texts. The author should either confirm that the validation texts were held out, or re-run the validation on texts not present in the training corpus. This is essential for the credibility of the central claim.
- [§6, directed-network centrality] The centrality analysis is presented as validating the model, but the graph construction builds in the result: 'the graph can be directed with later works pointing at earlier works'. Under that construction, earlier texts trivially have more incoming edges and thus higher in-degree-based centrality (PageRank, eigenvector, etc.). The finding that 'centrality is mostly related to a text's year of publication' is a direct consequence of the edge direction, not an empirical discovery. The single example of Goethe's Werther being ranked higher than its contemporaries is anecdotal and not compared against a null model that randomises edge weights or a baseline that removes the temporal direction. This section does not provide independent validation of Eqs. (1)–(2).
- [§4 and §5, comparisons] The paper claims that the embedding-based approach 'overcomes the limitations of exact lexical matching', but it offers no comparison against the existing methods it cites (TextPAIR, Passim, Tesserae, or the word-embedding approaches of Barbu and Trausan-Matu, Burns et al., etc.). A direct comparison on the same four-text validation, or on a small labelled set, would be necessary to support the contribution's novelty and effectiveness. Without such a comparison, the reader cannot tell whether the proposed score adds value over a simple bag-of-words TF-IDF cosine, which would also produce the same ordering in the four-text example.
minor comments (4)
- [§2, footnote 2] The footnote correctly notes that averaging word vectors discards word order, making the method 'more akin to bag-of-words than n-gram'. This should be stated more prominently in the main text, because it materially affects the interpretation of the 'n-gram' model and the claim of capturing localised semantic links.
- [§4, Table 1] Table 1 appears to contain only column headers in the manuscript; the actual heat map values are missing. Please include the numerical intertextuality scores, ideally with the hyperparameter grid (n = 3–6, τ = 0.01–0.50) so the claim that results are 'qualitatively' stable can be checked.
- [§5, Algorithm 1] The notation in Algorithm 1 is confusing: A and B are initially defined as text sets, then redefined as matrices; line 8 uses A⊘B for element-wise division of two matrices, but the symbol ⊘ is not defined. Clarify the indexing and variable types.
- [§3 and Appendix A] The normalisation pipeline uses 19 rules plus 7 Americanism conversions, but no validation of the pipeline is reported (e.g., precision/recall on a historical spelling gold standard). Given that the model's quality depends on this preprocessing, a small evaluation would help.
Circularity Check
No significant circularity: Eq. (1)-(2) is an explicit operational definition evaluated against external literary judgment; no load-bearing self-citation or fitted-input-as-prediction step is present.
full rationale
The proposed model is defined by Eq. (1)-(2): intertextuality is the average of above-threshold cosine similarities between n-gram embeddings. This is an operational definition, not a derived result. The Section 4 validation takes four texts whose relative intertextuality is asserted from literary criticism and checks whether the defined score reproduces that ordering; no parameter is fitted to the ordering, and the paper reports stability across thresholds and n. The Section 6 network analysis is a downstream application of the same definition and does not feed back into Eq. (1). The only self-reference is the GitHub availability link (Xing 2025), which is not load-bearing. The paper's own limitations (footnote 2: averaging discards word order; Section 7: evocative phrases are not weighted) weaken construct validity but do not make Eq. (1) equivalent to its inputs. The validation's susceptibility to author/genre/period confounding and the absence of baselines or negative controls are substantive empirical concerns, but under the stated circularity criteria they do not constitute a demonstrated reduction of a prediction to a fit or to a self-citation chain. Therefore no circular step meeting the required evidence standard can be identified.
Assumptions & free parameters
free parameters (5)
- Similarity threshold tau =
Tested values: 0.01, 0.05, 0.10, 0.20, 0.50; 0.20 used in scalability test
- n-gram size n =
Tested values: 3, 4, 5, 6; 3 used in scalability test
- Embedding dimension =
350
- Stop-list frequency cutoff =
500 most frequent words
- CBOW window size =
3
assumptions (4)
- domain assumption Distributional semantics: co-occurrence statistics in a large corpus capture semantic relatedness relevant to intertextuality.
- domain assumption Global text properties emerge from aggregating local pairwise similarities (bottom-up).
- domain assumption The Oxford Text Archive and rule-based normalization adequately cover modern English for the intended comparisons.
- domain assumption The four chosen texts have known intertextual degrees that serve as ground truth.
Cite this review
Pith. "Pith review of Modelling Intertextuality with N-gram Embeddings." pith.science (2026). https://pith.science/paper/DZ6EKI6V
@misc{pith2026250906637,
author = {Pith},
title = {Pith review of: Modelling Intertextuality with N-gram Embeddings},
year = {2026},
howpublished = {\url{https://pith.science/paper/DZ6EKI6V}},
note = {Machine review of arXiv:2509.06637}
}
read the original abstract
Intertextuality is a central tenet in literary studies. It refers to the intricate links between literary texts that are created by various types of references. This paper proposes a new quantitative model of intertextuality to enable scalable analysis and network-based insights: perform pairwise comparisons of the embeddings of n-grams from two texts and average their results as the overall intertextuality. Validation on four texts with known degrees of intertextuality, alongside a scalability test on 267 diverse texts, demonstrates the method's effectiveness and efficiency. Network analysis further reveals centrality and community structures, affirming the approach's success in capturing and quantifying intertextual relationships.
Figures
Forward citations
Cited by 1 Pith paper
-
Creative Transformation in Literary Texts: Modelling Change Across Representational Levels
Historically linked literary pairs show selective multi-level retention-and-divergence profiles that separate from controls more strongly than any single similarity channel.
Reference graph
Works this paper leans on
-
[1]
G. Allen. Intertextuality. Routledge, 2022. ISBN 978-1-032-12256-4
work page 2022
-
[2]
T. Allen and C. Cooney. Plundering Philosophers : Identifying Sources of the Encyclopédie . Journal of the Association for History and Computing, 2010. URL http://hdl.handle.net/2027/spo.3310410.0013.107
-
[3]
M.-S. Barbu and S. Trausan-Matu. Intertextuality detection in literary texts using Word2Vec models. In 2017 21st International Conference on System Theory , Control and Computing ( ICSTCC ) , pages 262--265, Oct. 2017. doi:10.1109/ICSTCC.2017.8107044. URL https://ieeexplore.ieee.org/abstract/document/8107044
-
[4]
J. Barré. Latent structures in french fiction. In W. Haverals, M. Koolen, and L. Thompson, editors, Proceedings of the Computational Humanities Research Conference 2024 , volume 3834 of CEUR Workshop Proceedings , pages 21--36, Aarhus, Denmark, Dec. 2024. CEUR. URL https://ceur-ws.org/Vol-3834/#paper97
work page 2024
- [5]
-
[6]
S. Bird, E. Klein, and E. Loper. Natural Language Processing with Python : Analyzing Text with the Natural Language Toolkit . O'Reilly Media, Inc., June 2009. ISBN 978-0-596-55571-9
work page 2009
-
[7]
D. M. Blei. Probabilistic topic models. Commun. ACM, 55 0 (4): 0 77--84, Apr. 2012. ISSN 0001-0782. doi:10.1145/2133806.2133826. URL https://dl.acm.org/doi/10.1145/2133806.2133826
arXiv 2012
-
[8]
P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov. Enriching Word Vectors with Subword Information . Transactions of the Association for Computational Linguistics, 5: 0 135--146, June 2017. ISSN 2307-387X. doi:10.1162/tacl_a_00051
Show all 43 references
-
[9]
Bollmann
M. Bollmann. A Large - Scale Comparison of Historical Text Normalization Systems . In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics : Human Language Technologies...
2019 doi
-
[10]
A. Broder. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 ( Cat . No . 97TB100171 ) , pages 21--29, June 1997. doi:10.1109/SEQUEN.1997.666900. URL https://ieeexplore.ieee.org/abstract/document/666900
1997
-
[11]
P. J. Burns, J. A. Brofos, K. Li, P. Chaudhuri, and J. P. Dexter. Profiling of Intertextuality in Latin Literature Using Word Embeddings . In K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani-Tur, I. Beltagy, S. Bethard, R. Cotterell, T. Chakraborty, and Y. Zhou, editors,...
2021
-
[12]
J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, and Z. Liu. M3- Embedding : Multi - Linguality , Multi - Functionality , Multi - Granularity Text Embeddings Through Self - Knowledge Distillation . In L.-W. Ku, A. Martins, and V. Srikumar, editors, Findings of the Association for ...
2024 doi
-
[13]
Coffee, J.-P
N. Coffee, J.-P. Koenig, S. Poornima, R. Ossewaarde, C. Forstall, and S. Jacobson. Intertextuality in the Digital Age . Transactions of the American Philological Association, 142 0 (2): 0 383--422, 2012. ISSN 2575-7199. URL https://muse.jhu.edu/pub/1/article/490829
2012
-
[14]
Coffee, J.-P
N. Coffee, J.-P. Koenig, S. Poornima, C. W. Forstall, R. Ossewaarde, and S. L. Jacobson. The Tesserae Project : intertextual analysis of Latin poetry. Literary and Linguistic Computing, 28 0 (2): 0 221--228, June 2013. ISSN 0268-1145. doi:10.1093/llc/fqs033. URL https://doi.or...
2013 doi
-
[15]
Denis and S
I. Denis and S. Boodts. The pseudo- Augustinian S . App . 121 and its medieval textual connections: A testcase for the development of network visualizations in the PASSIM Research Tool . Journal of Historical Network Research, 9 0 (1), Feb. 2024. ISSN 2535-8863. doi:10.25517/j...
2024 doi
-
[16]
Düring, M
M. Düring, M. Romanello, M. Ehrmann, K. Beelen, D. Guido, B. Deseure, E. Bunout, J. Keck, and P. Apostolopoulos. impresso Text Reuse at Scale . An interface for the exploration of text reuse data in semantically enriched historical newspapers. Frontiers in Big Data, 6, Nov. 20...
2023
-
[17]
Fedchenko, D
V. Fedchenko, D. M. Nicolosi, and G. Roe. À la recherche des réseaux intertextuels : défis de la recherche littéraire à grande échelle. Humanités numériques, 0 (9), June 2024. ISSN 2736-2337. doi:10.4000/11wmw. URL https://journals.openedition.org/revuehn/3940
2024 doi
-
[18]
J. Firth. A synopsis of linguistic theory, 1930-1955. Studies in linguistic analysis, pages 10--32, 1957
1930
-
[19]
Forstall, N
C. Forstall, N. Coffee, T. Buck, K. Roache, and S. Jacobson. Modeling the scholars: Detecting intertextuality through enhanced word-level n-gram matching. Digital Scholarship in the Humanities, 30 0 (4): 0 503--515, Dec. 2015. ISSN 2055-7671. doi:10.1093/llc/fqu014. URL https:...
2015 doi
-
[20]
C. W. Forstall and W. J. Scheirer. Quantitative Intertextuality : Analyzing the Markers of Information Reuse , volume 1. Springer, May 2019
2019
-
[21]
Freitag, M
D. Freitag, M. Blume, J. Byrnes, E. Chow, S. Kapadia, R. Rohwer, and Z. Wang. New Experiments in Distributional Representations of Synonymy . In I. Dagan and D. Gildea, editors, Proceedings of the Ninth Conference on Computational Natural Language Learning ( CoNLL -2005) , pag...
2005
-
[22]
Gladstone, S
C. Gladstone, S. J. Denis, and V. Hanoka. TextPAIR , 2009. URL https://github.com/ARTFL-Project/text-pair
2009
-
[23]
A. A. Hagberg, D. A. Schult, and P. J. Swart. Exploring Network Structure , Dynamics , and Function using NetworkX . In G. Varoquaux, T. Vaught, and J. Millman, editors, Proceedings of the 7th Python in Science Conference , pages 11 -- 15, Pasadena, CA USA, 2008
2008
-
[24]
W. L. Hamilton, J. Leskovec, and D. Jurafsky. Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change . In K. Erk and N. A. Smith, editors, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( Volume 1: Long Papers ) , pages 1...
2016 doi
-
[25]
W. Iser. Grasping a text. In The Act of Reading : A Theory of Aesthetic Response , pages 107--134. Johns Hopkins University Press, Baltimore, Maryland, 1980. ISBN 0-8018-2371-4
1980
-
[26]
Jana and P
A. Jana and P. Goyal. Can Network Embedding of Distributional Thesaurus Be Combined with Word Vectors for Better Representation ? In M. Walker, H. Ji, and A. Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational L...
2018 doi
-
[27]
Kristeva
J. Kristeva. Poésie et négativité. Homme, 8 0 (2): 0 36--63, 1968. doi:10.3406/hom.1968.366977. URL https://www.persee.fr/doc/hom_0439-4216_1968_num_8_2_366977
1968
-
[28]
Kristeva
J. Kristeva. Word, Dialogue , and Novel . In Desire in Language : A Semiotic Approach to Literature and Art . Columbia University Press, 2024. ISBN 978-0-231-21455-1
2024
-
[29]
Li and L
Y. Li and L. Mullen. textreuse: Detect Text Reuse and Document Similarity . 2024. URL https://docs.ropensci.org/textreuse
2024
-
[30]
Shakespeare in the Vectorian Age
B. Liebl and M. Burghardt. “ Shakespeare in the Vectorian Age ” – An evaluation of different word embeddings and NLP parameters for the detection of Shakespeare quotes. In S. DeGaetano, A. Kazantseva, N. Reiter, and S. Szpakowicz, editors, Proceedings of the 4th Joint SIGHUM W...
2020
-
[31]
Manjavacas, F
E. Manjavacas, F. Karsdorp, and M. Kestemont. A Statistical Foray into Contextual Aspects of Intertextuality . In Proceedings of the Workshop on Computational Humanities Research ( CHR 2020) , volume 2723, pages 77--96. CEUR Workshop Proceedings, Oct. 2020
2020
-
[32]
McGovern, H
H. McGovern, H. Sirin, and T. Lippincott. Characterizing the Effects of Translation on Intertextuality using Multilingual Embedding Spaces , Jan. 2025. URL http://arxiv.org/abs/2501.10731
2025 arXiv
-
[33]
Merriam-webster unabridged thesaurus, 2025
Merriam-Webster. Merriam-webster unabridged thesaurus, 2025. URL https://premium.britannica.com/mw-unabridged/
2025
- [34]
-
[35]
L. Németh. Hunspell, 2014. URL https://github.com/hunspell/hunspell
2014
-
[36]
Pennington, R
J. Pennington, R. Socher, and C. Manning. GloVe : Global Vectors for Word Representation . In A. Moschitti, B. Pang, and W. Daelemans, editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing ( EMNLP ) , pages 1532--1543, Doha, Qatar, Oct...
2014 doi
-
[37]
D. A. Smith, R. Cordell, and E. M. Dillon. Infectious texts: Modeling text reuse in nineteenth-century newspapers. In 2013 IEEE International Conference on Big Data , pages 86--94, Oct. 2013. doi:10.1109/BigData.2013.6691675. URL https://ieeexplore.ieee.org/abstract/document/6691675
2013
-
[38]
D. A. Smith, R. Cordell, and A. Mullen. Computational Methods for Uncovering Reprinted Texts in Antebellum Newspapers . American Literary History, 27 0 (3): 0 E1--E15, Sept. 2015. ISSN 0896-7148. doi:10.1093/alh/ajv029. URL https://doi.org/10.1093/alh/ajv029
2015 doi
-
[39]
The University of Oxford Text Archive , 1976
University of Oxford . The University of Oxford Text Archive , 1976
1976
-
[40]
Wang and M
Q. Wang and M. J. Zaki. HG2Vec : Improved Word Embeddings from Dictionary and Thesaurus Based Heterogeneous Graph . In N. Calzolari, C.-R. Huang, H. Kim, J. Pustejovsky, L. Wanner, K.-S. Choi, P.-M. Ryu, H.-H. Chen, L. Donatelli, H. Ji, S. Kurohashi, P. Paggio, N. Xue, S. Kim,...
2022
-
[41]
Y. Xing. Intertextuality, 2025. URL https://github.com/Crissium/intertextuality
2025
-
[42]
Zhang, J
Q. Zhang, J. Kang, J. Qian, and X. Huang. Continuous word embeddings for detecting local text reuses at the semantic level. In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval , SIGIR '14, pages 797--806, New York, N...
2014
-
[43]
Řehůřek and P
R. Řehůřek and P. Sojka. Software Framework for Topic Modelling with Large Corpora . In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks , pages 45--50, Valletta, Malta, May 2010. ELRA
2010
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.