Pith. sign in

REVIEW 3 major objections 6 minor 19 references

A New Pair of GloVes

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper shows that GloVe embeddings retrained on 2024 Wikipedia, Gigaword, and Dolma match the 2014 vectors on analogy and similarity while improving NER on recent, non-Western newswire and noisy social text.

desk verdict Useful GloVe resource report with a real statistical reporting gap; the temporal NER gains are plausible but not yet demonstrated. read the letter →

arxiv 2507.18103 v1 pith:HXIEIADF submitted 2025-07-24 cs.CL cs.LG

Riley Carlson , John Bauer , Christopher D. Manning This is my paper
classification cs.CLcs.LG
keywords GloVeembeddingswordnamedentityrecognitiontemporaladaptationvocabularycoveragestaticcorpusupdateDolma
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the widely used 2014 GloVe word embeddings are ready for a refresh: retrained on 2024 Wikipedia, Gigaword, and a Dolma subset, the new vectors add hundreds of thousands of post-2014 words to the vocabulary while keeping analogy and similarity performance roughly level. Its central claim is that the updated embeddings improve named-entity recognition on temporally dependent text, with consistent F1 gains on the 2023 Worldwide non-Western newswire set, the modernized CoNLL-PP set, and the noisy-emerging-entity WNUT17 set—for example, 84.64 versus 82.1 per-entity F1 on Worldwide at 50 dimensions. The authors also supply the exact data versions, preprocessing, and hyperparameters that the 2014 release lacked. If the claim holds, refreshing static embeddings is a low-cost way to make efficient, low-resource NLP systems track contemporary language.

What carries the argument

The load-bearing mechanism is the GloVe training procedure itself—factorizing a global word-word cooccurrence matrix with a symmetric context window of 10 and AdaGrad optimization—plus two update choices: a Minimum Frequency Threshold of 20, adopted from recent GloVe work on statistical uncertainty, sets the Wiki/Giga vocabulary at 1,291,146 words; and a Dolma subset of about 220 billion tokens from Common Crawl, C4, Reddit, and Project Gutenberg supplies a 1.2-million-word vocabulary rich in informal and social-media language. These choices determine which new words enter the embeddings and therefore drive the NER improvements the paper reports.

What would settle it

Re-run the four NER evaluations with the same code and at least ten random seeds per embedding set and compute confidence intervals; if the 2024-versus-2014 F1 gaps on Worldwide, CoNLL-PP, and WNUT17 include zero at most dimensions, the central temporal-dependency claim would not be supported.

Watch

Extended reading notes

Core claim

The paper presents two new sets of 2024 English GloVe vectors—Wiki/Giga in 50, 100, 200, and 300 dimensions, and Dolma in 300 dimensions—and shows they contain over 700,000 words absent from the 2014 Wiki/Giga vocabulary and over 500,000 words absent from the 2014 840B Common Crawl vectors, including covid, chatgpt, blockchain, and current slang. On the Google and MSR analogy benchmarks and the WordSim353, SimLex999, and MEN similarity benchmarks, the 2024 vectors score comparably to the 2014 vectors, with a consistent tendency to encode tight synonym and hypernym-hyponym relations better while underestimating looser thematic associations and occasionally overrating antonym similarity. On NER, the 2024 vectors match the 2014 vectors on the original CoNLL-03 set but beat them on CoNLL-PP, the 2023 non-Western Worldwide newswire set, and WNUT17; confusion-matrix comparisons attribute the gains to correct tagging of newly prominent entities such as COVID-19 and non-Western person names that the 2014 embeddings mislabel. The paper's conclusion is that temporally dependent tasks should use the 2024 embeddings, while older-domain benchmarks show no loss.

Load-bearing premise

The claimed NER advantage assumes that a single run of one NER model with default settings and no embedding finetuning gives stable score differences, so the reported one-to-two-point gains are signal rather than run-to-run variation.

Editorial extensions

If this is right

  • For temporally dependent tasks such as modern named-entity recognition, chatbot interaction, and social-media analysis, the 2024 embeddings are the better default than the 2014 release.
  • The 2024 Wiki/Giga vectors add over 700,000 words absent from the 2014 Wiki/Giga set, reducing out-of-vocabulary failures on contemporary text.
  • On the original 2003 CoNLL data the two embedding generations score nearly identically, so refreshing the corpus does not sacrifice established structural quality.
  • The largest relative NER gains appear at 50 dimensions, so even small and efficient models reap the benefit; the 300-dimensional Dolma vectors give the highest absolute F1 on Worldwide and WNUT17.
  • Newly prominent vocabulary that the 2014 vectors could not represent, such as covid and the slang term finna, is handled correctly by the 2024 vectors, which is the mechanism behind the F1 gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural test the paper leaves implicit is whether the roughly one-to-two-point F1 gaps survive multiple random seeds and significance testing; the reported single runs cannot rule out noise.
  • The design does not isolate whether the gains come from newer Wikipedia text, the doubled Gigaword, or the social-media-rich Dolma subset; a controlled experiment holding vocabulary size and training script fixed while ablating each corpus would separate these contributions.
  • The same lexicon-renewal logic should transfer to other temporally sensitive tasks where static embeddings still appear, such as event extraction, temporal reasoning, and stance detection on contemporary text.
  • For the community, the paper's value may be as much its documentation discipline as its embeddings: exact dump dates, tokenizer versions, seeds, and hyperparameters make the refresh reproducible and set a template for future static-embedding updates.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper trains two new sets of 2024 English GloVe embeddings, one on 2024 Wikipedia plus doubled Gigaword and one on a Dolma subset, and compares them with the original 2014 embeddings. Evaluation covers vocabulary coverage, word analogy and similarity benchmarks, and NER on CoNLL-03, CoNLL-PP, Worldwide newswire, and WNUT17. The authors report that the 2024 vectors add contemporary vocabulary, perform comparably on analogy and similarity, and improve NER on temporally dependent datasets, especially non-Western newswire.

Significance. If the NER improvements are robust, this is a useful contribution: a documented, reproducible update of a widely used static embedding resource, with transparent data provenance and hyperparameter reporting. The external benchmark evaluation and the qualitative vocabulary analysis are strengths. However, the central NER claim currently rests on single-run F1 differences of 0.2 to 2.5 points with no confidence intervals or significance tests, so the magnitude and consistency of the improvement are not yet established.

major comments (3)
  1. [Section 4.3, Tables 8 and 9] The headline claim that the 2024 embeddings “consistently outperform” on Worldwide and WNUT17 is not supported by the reported numbers as they stand. All entries are single-run F1 scores with no confidence intervals, significance tests, or multiple-seed experiments, and the paper does not state whether “Average Test F1” is averaged over seeds, folds, or entity types. The differences are frequently small, and some rows run counter to the claim (e.g., Table 8, 100d Worldwide per-token: 84.25 vs 84.58; Table 9, 300d WNUT17 per-token: 33.33 vs 33.73). The phrase “significantly outperforming” for the 50d Worldwide result (84.64 vs 82.1) is therefore not justified. Please provide paired bootstrap confidence intervals or repeated training runs with different random seeds, and restrict “consistent” language to configurations where the evidence supports it.
  2. [Table 4 and Section 5] The abstract's claim that the 2024 embeddings “perform comparably” on analogy and similarity is not supported by a statistical comparison, and the point estimates trend downward for 2024 on nearly every metric and dimension (e.g., MSR 100d: 0.486 vs 0.550; SimLex999 300d: 0.338 vs 0.371; MEN 300d: 0.690 vs 0.737). If the intended claim is “not materially worse,” an equivalence test or a clear statement of the tolerance margin is needed; otherwise the text should acknowledge a small but consistent degradation on these benchmarks.
  3. [Section 5, Tables 10–13] The confusion-matrix differences and the worked examples (e.g., “Bolsonaro” tagged PER vs LOC) are based on single runs and selected examples. They are useful as qualitative illustrations, but the text presents them as evidence of a systematic pattern (“these patterns are evident”). Without counts aggregated over the full test set with uncertainty, the claim that the 2024 embeddings specifically reduce person/location confusions for non-Western names is not established. Please either add aggregate error-type statistics with variance or explicitly label these as anecdotal.
minor comments (6)
  1. [Tables 4 and 7] Several table entries lack spacing between adjacent numbers (e.g., “0.6140.544” in Table 4 and “90.7284.0690.5082.74” in Table 7), and Table 7's “Per Entity (PP)” and “Per Token (PP)” should define PP as CoNLL-PP in the caption.
  2. [Section 3.1] The selection of MFT=20 by maximizing cosine similarity with WLS vectors should state whether this was done on the final corpus or a held-out sample; as written, it is an internal criterion and should not be described as a generally optimal threshold.
  3. [Section 4.3] The phrase “significantly outperforming” is used without a significance test; replace it with “reported a higher F1” or add proper statistical inference.
  4. [References] The Jastrzebski et al. reference contains LaTeX artifacts (“Stanis law Jastrzebski” and “Le´ sniak”), and the Vallebueno et al. entry has inconsistent spacing in “Christopher D Manning.”
  5. [Table 1 and Section 2] The token count “11.9 billion” for the 2024 Wiki/Giga corpus should clarify whether it includes the doubled Gigaword, since Section 2 says Gigaword was included twice.
  6. [Section 5] Statements such as the 2024 embeddings “do know” the right answer in analogy errors are not directly supported by the reported evaluation; consider reporting nearest-neighbor ranks for the error cases instead.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central evaluation is external and self-contained, and the training-configuration choices are not repackaged as predictions.

full rationale

The paper's central claim is that embeddings trained on updated corpora perform comparably on analogy and similarity tasks and better on recent, temporally dependent NER data. The supporting evidence comes from external benchmarks (Google/MSR analogies, WordSim353, SimLex999, MEN, CoNLL-03, WNUT17) and an independent NER framework (Stanza) with no embedding finetuning, so the reported comparisons are not derived from the training pipeline by construction. The MFT=20 vocabulary threshold is selected by maximizing cosine similarity to GloVe-V WLS vectors from prior work co-authored by one of the present authors, but this is a disclosed hyperparameter choice for training, not a fitted parameter renamed as an evaluation result, and it does not determine the downstream NER scores. The Worldwide dataset (Shan et al., 2023) is also from the same group, but it is an external annotated corpus used only for evaluation. The statistical critique that single-run F1 differences without variance or significance testing are fragile is a robustness and experimental-reporting concern, not circularity. No equation or result in the paper reduces to its own input by definition.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

No new entities or forces are postulated. The paper's contribution is resource-based, so the main ledger items are manual training choices: MFT, vocabulary caps, corpus reweighting, and the implicit assumption that single-run benchmark numbers are stable.

free parameters (4)
  • Minimum Frequency Threshold (MFT) = 20 for Wiki/Giga vocabulary
    Selected because MFT of 20 maximized average cosine similarity between trained vectors and the authors' GloVe-V WLS vectors, a self-referential tuning criterion; the threshold directly controls vocabulary size (1,291,146 words) and thus the content of the embeddings.
  • Dolma maximum vocabulary size = 1.2 million
    Chosen by hand as a cap for the Dolma-derived vocabulary; no external justification or ablation is given.
  • Dolma subset percentages = Common Crawl 5%, C4 40%, plus all of Reddit and Gutenberg
    Chosen by hand to reach a 220-billion-token subset; no justification or ablation is provided.
  • Gigaword duplication factor = two copies of Gigaword
    Introduced to rebalance the doubled Wikipedia size; the factor of two is a manual choice without demonstrated optimality.
assumptions (3)
  • domain assumption The standard GloVe training algorithm and AdaGrad optimizer, as implemented in the GloVe repository demo.sh, produce embeddings whose quality is faithfully summarized by the chosen benchmarks.
    The paper trusts the original GloVe pipeline and the Jastrzebski evaluation package without independent verification of the implementation.
  • ad hoc to paper Single-run F1 scores on NER datasets are stable enough to support the reported conclusions without significance testing or multiple seeds.
    All NER comparisons in Tables 7-9 are single estimates; the paper implicitly assumes differences of a fraction of an F1 point are meaningful.
  • domain assumption The Dolma subset (with 5% Common Crawl sampling) constitutes a representative sample of contemporary English usage for training embeddings.
    The 5% and 40% sampling choices are presented without justification of representativeness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A New Pair of GloVes." pith.science (2026). https://pith.science/paper/HXIEIADF

@misc{pith2026250718103,
  author       = {Pith},
  title        = {Pith review of: A New Pair of GloVes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HXIEIADF}},
  note         = {Machine review of arXiv:2507.18103}
}
read the original abstract

This report documents, describes, and evaluates new 2024 English GloVe (Global Vectors for Word Representation) models. While the original GloVe models built in 2014 have been widely used and found useful, languages and the world continue to evolve and we thought that current usage could benefit from updated models. Moreover, the 2014 models were not carefully documented as to the exact data versions and preprocessing that were used, and we rectify this by documenting these new models. We trained two sets of word embeddings using Wikipedia, Gigaword, and a subset of Dolma. Evaluation through vocabulary comparison, direct testing, and NER tasks shows that the 2024 vectors incorporate new culturally and linguistically relevant words, perform comparably on structural tasks like analogy and similarity, and demonstrate improved performance on recent, temporally dependent NER datasets such as non-Western newswire data.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 5 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Giusepppe Attardi. 2015. WikiExtractor . https://github.com/attardi/wikiextractor

    work page 2015

  4. [4]

    Elia Bruni, Nam Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. J. Artif. Int. Res., 49(1):1–47

    2014

  5. [5]

    Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. https://doi.org/10.18653/v1/W17-4418 Results of the WNUT 2017 shared task on novel and emerging entity recognition . In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 140--147, Copenhagen, Denmark. Association for Computational Linguistics

    doi:10.18653/v1/w17-4418 2017

  6. [6]

    Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. https://doi.org/10.1145/503104.503110 Placing search in context: The concept revisited . ACM Transactions on Information Systems - TOIS, 20:406--414

    arXiv 2001

  7. [7]

    Felix Hill, Roi Reichart, and Anna Korhonen. 2015. https://doi.org/10.1162/COLI_a_00237 S im L ex-999: Evaluating semantic models with (genuine) similarity estimation . Computational Linguistics, 41(4):665--695

    doi:10.1162/coli_a_00237 2015

  8. [8]

    Stanisław Jastrzebski, Damian Leśniak, and Wojciech Marian Czarnecki. 2017. https://doi.org/10.48550/ARXIV.1702.02170 How to evaluate word embeddings? O n importance of data efficiency and simple supervised tasks . ArXiv preprint arXiv:1702.02170

    work page Pith review arXiv doi:10.48550/arxiv.1702.02170 2017

Show all 19 references
  1. [9]

    Shuheng Liu and Alan Ritter. 2023. https://doi.org/10.18653/v1/2023.acl-long.459 Do C o NLL -2003 named entity taggers still work well in 2023? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8254--8271,...

    2023 doi

  2. [10]

    Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013 a . http://arxiv.org/abs/1301.3781 Efficient estimation of word representations in vector space . ArXiv preprint arXiv:1301.3781

    2013 arXiv

  3. [11]

    Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013 b . https://aclanthology.org/N13-1090 Linguistic regularities in continuous space word representations . In Proceedings of the 2013 Conference of the North A merican Chapter of the Association for Computational Linguistics: ...

    2013

  4. [12]

    Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2011. https://doi.org/https://doi.org/10.35111/wk4f-qt80 English G igaword F ifth E dition . Linguistic Data Consortium, Philadelphia, PA. LDC2011T07

    2011 doi

  5. [13]

    Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. https://doi.org/10.3115/v1/D14-1162 G lo V e: Global vectors for word representation . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing ( EMNLP ) , pages 1532--1543, Doh...

    2014 doi

  6. [14]

    Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. https://doi.org/10.18653/v1/2020.acl-demos.14 S tanza: A python natural language processing toolkit for many human languages . In Proceedings of the 58th Annual Meeting of the Association for Co...

    2020 doi

  7. [15]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. http://jmlr.org/papers/v21/20-074.html Exploring the limits of transfer learning with a unified text-to-text transformer . Journal of Machine Lea...

    2020

  8. [16]

    Alexander Shan, John Bauer, Riley Carlson, and Christopher Manning. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.788 Do `` E nglish'' named entity recognizers work well on global E nglishes? In Findings of the Association for Computational Linguistics: EMNLP 2023, pag...

    2023 doi

  9. [17]

    Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A

    Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Mu...

    2024 arXiv

  10. [18]

    Tjong Kim Sang and Fien De Meulder

    Erik F. Tjong Kim Sang and Fien De Meulder. 2003. https://www.aclweb.org/anthology/W03-0419 Introduction to the C o NLL -2003 shared task: Language-independent named entity recognition . In Proceedings of the Seventh Conference on Natural Language Learning at HLT - NAACL 2003 ...

    2003

  11. [19]

    Andrea Vallebueno, Cassandra Handan-Nader, Christopher D Manning, and Daniel E. Ho. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.510 Statistical uncertainty in word embeddings: G lo V e- V . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Pr...

    2024 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.