EntEval: A Holistic Evaluation Benchmark for Entity Representations

Karl Stratos; Kevin Gimpel; Mingda Chen; Yang Chen; Zewei Chu

REVIEW 3 major objections 5 minor 76 references

EntEval: A Holistic Evaluation Benchmark for Entity Representations

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read EntEval, a seven-task benchmark, claims entity understanding can be measured holistically with frozen linear probes, and Wikipedia hyperlink pretraining improves ELMo on several tasks.

desk verdict EntEval is a genuinely useful benchmark for entity representations, but CERP's auto-generated negative labels are unvalidated and could be wrong, so the rankings should be treated with caution until that is addressed. read the letter →

arxiv 1909.00137 v2 pith:BLVYJUSS submitted 2019-08-31 cs.CL

Mingda Chen , Zewei Chu , Yang Chen , Karl Stratos , Kevin Gimpel This is my paper · ORCID

classification cs.CL

keywords entityrepresentationsevaluationbenchmarkcontextualizedwordtypingdisambiguationsimilarityWikipediahyperlinksELMo

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EntEval is a test suite of seven tasks—entity typing, coreference arc prediction, entity factuality prediction, contextualized entity relationship prediction, entity similarity and relatedness, entity relationship typing, and named entity disambiguation—that together probe how well a representation captures knowledge about entities. The paper's proposal is that freezing the representation and training only a linear classifier on these tasks yields a holistic, standardized comparison that has been missing. Its results show that pretrained contextual word representations such as ELMo and BERT outperform static entity embeddings, with ELMo stronger in zero-shot settings and BERT improving more after supervised training. The paper also introduces a Wikipedia-hyperlink training objective that improves ELMo-based entity representations on several EntEval tasks.

What carries the argument

The load-bearing object is EntEval itself: seven tasks run under a frozen-representation protocol in which entity representations are not fine-tuned and only a linear classifier is trained, using the concatenation $[x_1, x_2, x_1 \odot x_2, |x_1 - x_2|]$ for pairwise tasks. Contextualized entity representations (CER) encode the mention span from its sentence, while descriptive entity representations (DER) encode the entity's Wikipedia description; model-specific pooling differs between ELMo and BERT. The training-side mechanism is EntELMo, which adds two bag-of-words reconstruction losses to ELMo's bidirectional language-model loss: $\ell_{\text{ctx}}$ reconstructs the mention's context from the description embedding, and $\ell_{\text{desc}}$ reconstructs the described entity's words from the mention-context embedding. These losses turn Wikipedia hyperlink structure into a pretraining signal, and the paper's layer-wise heatmaps identify which ELMo and BERT layers carry entity knowledge.

What would settle it

One concrete check: run a surface-feature baseline on CERP and CAP that looks only at token-level cues (presence of 'not', string overlap of entity names, word-frequency similarity). If that baseline matches or beats ELMo on those tasks, or if removing sentences containing 'not' collapses CERP accuracy, then those tasks are measuring artifacts rather than entity knowledge, and the benchmark's ranking would not support its claim.

Watch

Extended reading notes

Core claim

The paper's central claim is that entity representations can and should be evaluated as a whole, not task by task, and that contextualized word representations from pretrained language models are the current best carriers of entity knowledge. On the paper's evidence, dynamically encoded entity representations are markedly better than static per-entity embeddings for disambiguation, higher layers of ELMo and BERT transfer better to entity tasks, and BERT's [CLS]-based representations need supervised adaptation while ELMo works zero-shot. A second claim is that Wikipedia's natural hyperlink annotations are usable training signal: decoding a linked entity description from its mention context and decoding the mention context from the description, as bag-of-words objectives added to ELMo's language-modeling loss, improves performance on entity typing, factuality prediction, relationship prediction, and disambiguation, though not on description-only similarity.

Load-bearing premise

The benchmark's validity is the load-bearing premise: each task's automatically built positives and negatives must force the model to use entity knowledge rather than surface cues such as mention-name overlap, negation words, or description length.

Editorial extensions

If this is right

Any new encoder can be scored in the same frozen-linear-probe setup, so entity-representation progress becomes comparable across models.
The layer analysis implies that selecting the right layer matters: higher ELMo and BERT layers transfer better to entity tasks, while BERT's lower layers are better on entity similarity.
The hyperlink objective gives a data-driven recipe for injecting entity knowledge into existing language models without changing the model architecture.
Because BERT needs supervised training while ELMo performs zero-shot, the benchmark suggests different use cases: ELMo for low-resource entity tasks and BERT when labels are available.
Static entity embeddings from prior entity-linking systems underperform dynamically encoded representations on CoNLL-YAGO disambiguation, so new systems should build on contextual encoders.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves implicit that the benchmark's difficulty hinges on the generated negatives: if CERP negatives are identifiable by the word 'not' or CAP bins still leak name similarity, the relative rankings could change without any change in entity understanding.
A direct extension would be to run EntEval on newer pretrained models and on entity representations trained with knowledge-graph objectives, since the benchmark's layer-wise setup makes such comparisons straightforward.
One could also test the hyperlink objective in reverse, decoding descriptions from contexts while explicitly preventing the model from copying entity names, to isolate whether the gains come from entity knowledge or surface memorization.
If validated, EntEval-style probes could serve as a screening test before deploying entity representations in retrieval or question answering, since the tasks predict which failure modes a model will show.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Let me give you the short version: EntEval is a real contribution, a multi-task benchmark for entity representations with code released and a linear-probe protocol modeled on SentEval. If you work on entity embeddings, you'll probably end up citing it. But there's one spot you should check before relying on it: the CERP task's auto-generated labels. The paper creates negatives by swapping in GloVe-similar entities, then creates positives by negating those swaps. For relations like "is a," a similar entity is often also in the same class, so "Vermont is a state" gets labeled false and "Vermont is not a state" gets labeled true. The authors never validate the generated labels. That makes the CERP numbers, and the EntELMo improvement there, hard to interpret. This is a genuine flaw in one of the seven tasks, and since CERP is one of the four advertised task families, it undercuts the "holistic" claim until fixed.

What the paper does well: the benchmark design is sensible. It reuses existing datasets (ultra-fine typing, FEVER, PreCo, KORE, CoNLL-YAGO, Rare) under one frozen-encoder protocol, which is exactly what the entity-representation community lacks. The per-layer analysis is useful, especially the observation that BERT's [CLS] does poorly on zero-shot similarity. The hyperlink bag-of-words reconstruction objective for ELMo is a reasonable recipe, and the authors are appropriately cautious in the body about where it helps and hurts.

Soft spots beyond CERP: there are no error bars or significance tests anywhere, and several reported differences are small, e.g., EntELMo at 59.9 vs. baseline at 59.6 on CERP is within noise. Also, EntELMo is trained with smaller dimensions and one epoch, so the comparison to off-the-shelf ELMo is not apples-to-apples; the authors do compare to their own baseline, which is the right ablation, but the abstract's "improve strong baselines on multiple EntEval tasks" is true only in a selective sense, since EntELMo is worse than ELMo on average.

Verdict: send it to a serious referee. The benchmark fills a gap and the code is out. A good reviewer will ask for CERP label validation, error bars, and a slightly more careful abstract. If those are fixed, this becomes a standard reference for entity representation evaluation.

Referee Report

3 major / 5 minor

Summary. The paper introduces EntEval, a benchmark for evaluating entity representations across seven task families: entity typing (ET), coreference arc prediction (CAP), entity factuality prediction (EFP), contextualized entity relationship prediction (CERP), entity similarity and relatedness (ESR), entity relationship typing (ERT), and named entity disambiguation (NED). The benchmark is designed for contextualized and descriptive entity representations, with linear probes and frozen encoders. The paper also proposes EntELMo, an ELMo variant trained on a large Wikipedia hyperlink dataset with bag-of-words reconstruction losses that tie mention contexts to entity descriptions. Experiments compare GloVe, BERT, ELMo, and EntELMo variants, and include per-layer analyses and comparisons to static entity-embedding baselines on CoNLL-YAGO.

Significance. EntEval addresses a real gap: there is currently no standardized multi-task evaluation of entity representations. The paper's strengths are its diverse task coverage, its use of existing manually annotated data where feasible, its released evaluation and data-processing code, and its per-layer analysis, which provides useful observations about ELMo and BERT. If the automatically constructed portions of the benchmark are validated, EntEval could become a useful community resource. However, the paper's central claim that the proposed hyperlink-based training improves strong baselines is not supported by the reported experiments: EntELMo does not outperform the original ELMo on any EntEval task, and most highlighted improvements over the authors' smaller baseline are under one point with no significance testing.

major comments (3)

[§5.2, Table 3; Abstract] The abstract's claim that the training techniques 'improve strong baselines on multiple EntEval tasks' is not supported by Table 3. EntELMo is numerically worse than the original ELMo on every shared task except ESR-related comparisons, and its average is 56.5 versus ELMo's 58.8. The only favorable comparison is against the 'EntELMo baseline' (average 56.7), which is a smaller model trained by the authors and is not a strong baseline. Furthermore, the differences highlighted in Section 5.2 (CERP 59.9 vs 59.6, EFP 72.4 vs 71.5, ET 32.2 vs 31.3, NED 49.0 vs 48.5) are all less than one point and are reported without variance or significance tests. The central claim should be re-scoped to an ablation, or the model should be compared with the original ELMo under matched training conditions.
[§3.4, CERP negative sampling rules] The CERP construction never verifies that rule-1 substitutions produce false statements. Given a ConceptNet assertion 'A is B' (for example, 'Connecticut is a state'), replacing A with a GloVe-similar entity C (for example, 'Vermont') can yield a true statement that is labeled negative; rule 3 then converts it into a false statement labeled positive, which is also incorrect. The paper excludes only a few relation types, and important relations such as 'is a' are retained. No manual or automatic validation of the generated labels is reported. Consequently, CERP accuracy—and the claim that EntELMo improves on CERP—is uninterpretable if the false-label rate is substantial. The authors should measure and report the label error rate, or restrict negative sampling to relations where substitution is truth-changing.
[§3, Table 1; §5.1] Several EntEval datasets are newly constructed through automatic filtering and sampling, and the benchmark is used to rank models (e.g., ELMo above BERT on ESR and ET, BERT above ELMo on CAP) without any repeated runs or significance testing. The CAP task is generated by binning and balancing on GloVe mention similarity, EFP is a random subsample of FEVER with a randomly chosen mention, ERT uses only 5/10/10 tuples per relation, and the KORE test set has only 20 by 20 pairs. Given the small test sizes and the automatic pipelines, the reported differences, many of which are a few points, cannot be distinguished from noise. The authors should run multiple seeds or provide bootstrap confidence intervals for the small test sets, particularly for the claims about relative model rankings.

minor comments (5)

[§3.1] The sentence 'In this task, we will contextualized entity representations' is missing a verb; it should read 'we will use contextualized entity representations.'
[§3.4 and Table 3] The task name is misspelled as 'Contexualized Entity Relationship Prediction' in the section title and as 'CERP: contexualized...' in Table 3; it should be 'Contextualized.'
[§4.3] The notation 'letn' is introduced without a formal definition; it should be defined as the loss that decodes only the entity mention span from the description, and the relation of each loss term in Eq. (1) to the three EntELMo variants should be stated explicitly.
[§4.3] The phrase 'we use the same bidirectional language modeling loss llang(x1:Tx) + llang(y1:Ty) in ELMo' is ambiguous; it should clarify that EntELMo adds ELMo's language-model loss on both the context and the description sequences, rather than reusing a single loss value.
[Figure 6] The heatmap would be easier to read if each row were labeled with the corresponding model (ELMo, EntELMo baseline, etc.) and if the color scale were annotated with numeric values, since the current 'Low High' legend is not quantitative.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation: EntEval is an external benchmark suite and EntELMo is an ablation-controlled training study.

full rationale

The paper's central claims are (i) that EntEval provides a diverse evaluation of entity representations and (ii) that hyperlink-based training objectives improve ELMo-based representations. Neither claim reduces to its own inputs by construction. The EntEval tasks are built from externally published datasets (ultra-fine entity typing, PreCo, FEVER, ConceptNet, KORE, WikiSRS, Freebase, CoNLL-YAGO, Rare) and evaluated with frozen representations plus a linear classifier or cosine similarity, so task scores are not defined in terms of the representations being ranked. The EntELMo comparison is an ablation: EntELMo baseline is trained on the same Wikipedia data without the hyperlink losses, so the comparison isolates the proposed objective rather than reusing a fitted constant as a prediction. The only self-citation is Chen et al. (2019a) listed among related work on evaluating pretrained representations; it is not load-bearing for any claim in this paper. The reviewer-identified concern about CERP's automatically generated negatives potentially being mislabeled (e.g., a GloVe-similar replacement may also satisfy the relation) is a benchmark-validity threat, not a circularity: the labels are not constructed from the evaluated models and no equation in the paper equates the prediction with the training signal. For these reasons, no specific circular step is established, and the paper is largely self-contained as an empirical evaluation.

Assumptions & free parameters 8 free parameters · 4 assumptions · 0 invented entities

The paper's empirical claims rest on benchmark validity and data assumptions rather than on derived constants. The listed free parameters are training and dataset-construction choices, none fitted to EntEval outcomes. No invented entities are introduced.

free parameters (8)

EntELMo LSTM dimensions = hidden 600, projection 300
Chosen for computational tractability, Section 5.1; affects EntELMo results but not the benchmark definition.
EntELMo training epochs = 1
Stopped after one epoch due to compute, Section 5.1; baseline and EntELMo share this.
Negative sampling size = 1024
Used in ELMo-style log loss approximation, Section 5.1.
BOW reconstruction positive samples = at most 50
Random sample of target tokens for reconstruction losses, Section 5.1.
EFP dataset sizes = 10000/2000/2000
Randomly sampled train/validation/test from FEVER, Section 3.3; affects EFP numbers.
NED missing-candidate prior = 1e-6
Assigned to gold candidates absent from CrossWikis, Section 3.7; affects NED scores.
Description truncation length = 100 tokens
Wikipedia descriptions capped at 100 words, Section 4.3; affects all descriptive representations.
ERT instances per relation type = 5/10/10 train/valid/test
Randomly sampled per type, Section 3.6; could make ERT performance estimates noisy.

assumptions (4)

domain assumption Wikipedia hyperlinks align mention context with the linked entity's description
Invoked in Section 4.3 to define l_ctx and l_desc; noisy links would weaken the training signal.
domain assumption Linear probing with frozen representations measures entity knowledge
Section 3 freezes encoders and uses linear classifiers or cosine similarity; the benchmark's validity depends on this assumption.
domain assumption Reused annotation datasets and automatic filtering preserve task validity
EntEval builds on FEVER, PreCo, ConceptNet, Freebase, KORE, WikiSRS, and CoNLL-YAGO with automatic preprocessing, so any annotation noise transfers to the benchmark.
domain assumption Negative sampling prevents surface-level shortcut solving
CAP bins by GloVe cosine similarity and CERP replaces entities with similar ones, Sections 3.2 and 3.4; the paper does not verify that shortcuts are fully removed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of EntEval: A Holistic Evaluation Benchmark for Entity Representations." pith.science (2026). https://pith.science/paper/BLVYJUSS

@misc{pith2026190900137,
  author       = {Pith},
  title        = {Pith review of: EntEval: A Holistic Evaluation Benchmark for Entity Representations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BLVYJUSS}},
  note         = {Machine review of arXiv:1909.00137}
}

read the original abstract

Rich entity representations are useful for a wide class of problems involving entities. Despite their importance, there is no standardized benchmark that evaluates the overall quality of entity representations. In this work, we propose EntEval: a test suite of diverse tasks that require nontrivial understanding of entities including entity typing, entity similarity, entity relation prediction, and entity disambiguation. In addition, we develop training techniques for learning better entity representations by using natural hyperlink annotations in Wikipedia. We identify effective objectives for incorporating the contextual information in hyperlinks into state-of-the-art pretrained language models and show that they improve strong baselines on multiple EntEval tasks.

Figures

Figures reproduced from arXiv: 1909.00137 by the authors.

**Figure 2.** Two examples from the EFP. TRUE: Gin and vermouth can make a martini FALSE: Connecticut is not a state [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 4.** An example from CoNLL-YAGO. Only four candidates are shown due to space constraints. The target mention is underlined. Sentences in gray are Wikipedia descriptions. The gold standard is boldfaced. Hoffart et al., 2011) and Rare Entity Prediction (Rare; Long et al., 2017). For CoNLL-YAGO, following Hoffart et al. (2011) and Yamada et al. (2016), we used the 27,816 mentions with valid entries in the knowledge base.… view at source ↗

Figures from the paper (2 more)

**Figure 5.** Figure 5: An example of hyperlinks in Wikipedia. “France” is linked to the Wikipedia page of “France national football team” instead of the country France. ful description of the mentioned entity. The same Wikipedia page may correspond to many different entity mentions. Likewise…

**Figure 6.** Figure 6: Heatmap showing per-layer performances for ELMo, EntELMo baseline, EntELMo, BERT Base, and BERT Large. LSTM layers capture most of the entity knowledge from context and descriptions. The BERT layers show more diversity. Lower layers perform better on ESR (entity simila…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 37 canonical work pages

[1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In ICLR

work page 2017
[4]

Gabor Angeli and Christopher D. Manning. 2014. https://doi.org/10.3115/v1/D14-1059 N atural LI : Natural logic inference for common sense reasoning . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing ( EMNLP ) , pages 534--545, Doha, Qatar. Association for Computational Linguistics

work page doi:10.3115/v1/d14-1059 2014
[5]

Yonatan Belinkov, Llu \' s M \`a rquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2017. https://www.aclweb.org/anthology/I17-1001 Evaluating layers of representation in neural machine translation on part-of-speech and semantic tagging tasks . In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volu...

work page 2017
[6]

Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1247--1250. AcM

2008
[7]

Bowman, Gabor Angeli, Christopher Potts, and Christopher D

Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. https://doi.org/10.18653/v1/D15-1075 A large annotated corpus for learning natural language inference . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632--642, Lisbon, Portugal. Association for Computational Linguistics

doi:10.18653/v1/d15-1075 2015
[8]

Hong Chen, Zhenhua Fan, Hao Lu, Alan Yuille, and Shu Rong. 2018. https://www.aclweb.org/anthology/D18-1016 P re C o: A large-scale dataset in preschool vocabulary for coreference resolution . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 172--181, Brussels, Belgium. Association for Computational Linguistics

work page 2018

Show all 76 references

[9]

Mingda Chen, Zewei Chu, and Kevin Gimpel. 2019 a . Evaluation benchmarks and learning criteria for discourse-aware sentence representations. In Proc. of EMNLP

2019
[10]

Sihao Chen, Daniel Khashabi, Wenpeng Yin, Chris Callison-Burch, and Dan Roth. 2019 b . https://doi.org/10.18653/v1/N19-1053 Seeing things from a different angle:discovering diverse perspectives about claims . In Proceedings of the 2019 Conference of the North A merican Chapter...

2019 doi
[11]

Eunsol Choi, Omer Levy, Yejin Choi, and Luke Zettlemoyer. 2018. https://www.aclweb.org/anthology/P18-1009 Ultra-fine entity typing . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 87--96, Melbourne, Aus...

2018
[12]

Elizabeth Clark, Yangfeng Ji, and Noah A. Smith. 2018. https://doi.org/10.18653/v1/N18-1204 Neural text generation in stories using entity representations as context . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Lin...

2018 doi
[13]

Alexis Conneau and Douwe Kiela. 2018. Senteval: An evaluation toolkit for universal sentence representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018)

2018
[14]

Alexis Conneau, Germ \'a n Kruszewski, Guillaume Lample, Lo \" c Barrault, and Marco Baroni. 2018. https://www.aclweb.org/anthology/P18-1198 What you can cram into a single \ & ! \# * vector: Probing sentence embeddings for linguistic properties . In Proceedings of the 56th An...

2018
[15]

Luciano Del Corro, Abdalghani Abujabal, Rainer Gemulla, and Gerhard Weikum. 2015. https://doi.org/10.18653/v1/D15-1103 FINET : Context-aware fine-grained named entity typing . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 868-...

2015 doi
[16]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

2018 arXiv
[17]

Greg Durrett and Dan Klein. 2013. https://www.aclweb.org/anthology/D13-1203 Easy victories and uphill battles in coreference resolution . In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1971--1982, Seattle, Washington, USA. Asso...

2013
[18]

Greg Durrett and Dan Klein. 2014. https://doi.org/10.1162/tacl_a_00197 A joint model for entity analysis: Coreference, typing, and linking . Transactions of the Association for Computational Linguistics, 2:477--490

2014 doi
[19]

Matthew Francis-Landau, Greg Durrett, and Dan Klein. 2016. https://doi.org/10.18653/v1/N16-1150 Capturing semantic similarity for entity linking with convolutional neural networks . In Proceedings of the 2016 Conference of the North A merican Chapter of the Association for Com...

2016 doi
[20]

Octavian-Eugen Ganea and Thomas Hofmann. 2017. https://doi.org/10.18653/v1/D17-1277 Deep joint entity disambiguation with local neural attention . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2619--2629, Copenhagen, Denmark. ...

2017 doi
[21]

Zhaochen Guo and Denilson Barbosa. 2014. https://doi.org/10.1145/2661829.2661887 Robust entity linking via random walks . In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, CIKM '14, pages 499--508, New York, NY, USA. ACM

2014
[22]

Nitish Gupta, Sameer Singh, and Dan Roth. 2017. https://doi.org/10.18653/v1/D17-1284 Entity linking via joint encoding of types, descriptions, and context . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2681--2690, Copenhagen,...

2017 doi
[23]

He He, Anusha Balakrishnan, Mihail Eric, and Percy Liang. 2017. https://doi.org/10.18653/v1/P17-1162 Learning symmetric collaborative dialogue agents with dynamic knowledge graph embeddings . In Proceedings of the 55th Annual Meeting of the Association for Computational Lingui...

2017 doi
[24]

Zhengyan He, Shujie Liu, Mu Li, Ming Zhou, Longkai Zhang, and Houfeng Wang. 2013. https://www.aclweb.org/anthology/P13-2006 Learning entity representation for entity disambiguation . In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Vo...

2013
[25]

Sepp Hochreiter and J \"u rgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735--1780

1997
[26]

Johannes Hoffart, Stephan Seufert, Dat Ba Nguyen, Martin Theobald, and Gerhard Weikum. 2012. Kore: keyphrase overlap relatedness for entity disambiguation. In Proceedings of the 21st ACM international conference on Information and knowledge management, pages 545--554. ACM

2012
[27]

Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen F \"u rstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust disambiguation of named entities in text. In Proceedings of the Conference on Empirical Methods in Natural L...

2011
[28]

Matthew Honnibal and Ines Montani. 2017. spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing. To appear

2017
[29]

Hongzhao Huang, Larry Heck, and Heng Ji. 2015. Leveraging deep neural networks and knowledge graphs for entity disambiguation. arXiv preprint arXiv:1504.07678

2015 arXiv
[30]

S \'e bastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. https://doi.org/10.3115/v1/P15-1001 On using very large target vocabulary for neural machine translation . In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics a...

2015 doi
[31]

Yangfeng Ji, Chenhao Tan, Sebastian Martschat, Yejin Choi, and Noah A. Smith. 2017. https://doi.org/10.18653/v1/D17-1195 Dynamic entity representations in neural language models . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages ...

2017 doi
[32]

Ben Kantor and Amir Globerson. 2019. https://www.aclweb.org/anthology/P19-1066 Coreference resolution with entity equalization . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 673--677, Florence, Italy. Association for Computa...

2019
[33]

Phong Le and Ivan Titov. 2018. https://doi.org/10.18653/v1/P18-1148 Improving entity linking by modeling latent relations between mentions . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1595--1604, Me...

2018 doi
[34]

Phong Le and Ivan Titov. 2019. https://www.aclweb.org/anthology/P19-1187 Boosting entity linking performance by leveraging unlabeled documents . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1935--1945, Florence, Italy. Assoc...

2019
[35]

Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. https://doi.org/10.18653/v1/D17-1018 End-to-end neural coreference resolution . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188--197, Copenhagen, Denmark. Associ...

2017 doi
[36]

Xiang Li, Aynaz Taheri, Lifu Tu, and Kevin Gimpel. 2016. https://doi.org/10.18653/v1/P16-1137 Commonsense knowledge base completion . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1445--1455, Berlin, G...

2016 doi
[37]

Xiao Ling, Sameer Singh, and Daniel S. Weld. 2015. https://doi.org/10.1162/tacl_a_00141 Design challenges for entity linking . Transactions of the Association for Computational Linguistics, 3:315--328

2015 doi
[38]

Liu, Matt Gardner, Yonatan Belinkov, Matthew E

Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019 a . Linguistic knowledge and transferability of contextual representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics...

2019
[39]

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019 b . Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692

2019 arXiv
[40]

Liu, Matthew E

Robert Logan, Nelson F. Liu, Matthew E. Peters, Matt Gardner, and Sameer Singh. 2019. https://www.aclweb.org/anthology/P19-1598 B arack ' s wife hillary: Using knowledge graphs for fact-aware language modeling . In Proceedings of the 57th Annual Meeting of the Association for ...

2019
[41]

Lajanugen Logeswaran, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, and Honglak Lee. 2019. https://www.aclweb.org/anthology/P19-1335 Zero-shot entity linking by reading entity descriptions . In Proceedings of the 57th Annual Meeting of the Association for Compu...

2019
[42]

Teng Long, Emmanuel Bengio, Ryan Lowe, Jackie Chi Kit Cheung, and Doina Precup. 2017. World knowledge for reading comprehension: Rare entity prediction with hierarchical lstms using external descriptions. In Proceedings of the 2017 Conference on Empirical Methods in Natural La...

2017
[43]

Pedro Henrique Martins, Zita Marinho, and Andr \'e F. T. Martins. 2019. https://www.aclweb.org/anthology/P19-2026 Joint learning of named entity recognition and entity linking . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student...

2019
[44]

Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. http://papers.nips.cc/paper/7209-learned-in-translation-contextualized-word-vectors.pdf Learned in translation: Contextualized word vectors . In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S....

2017
[45]

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. https://doi.org/10.18653/v1/D18-1260 Can a suit of armor conduct electricity? a new dataset for open book question answering . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language P...

2018 doi
[46]

Shikhar Murty, Patrick Verga, Luke Vilnis, and Andrew McCallum. 2017. Finer grained entity typing with typenet. arXiv preprint arXiv:1711.05795

2017 arXiv
[47]

Lai, and Eric Fosler-Lussier

Denis Newman-Griffis, Albert M. Lai, and Eric Fosler-Lussier. 2018. https://www.aclweb.org/anthology/W18-3026 Jointly embedding entities and text with distant supervision . In Proceedings of The Third Workshop on Representation Learning for NLP , pages 195--206, Melbourne, Aus...

2018
[48]

Rasha Obeidat, Xiaoli Fern, Hamed Shahbazi, and Prasad Tadepalli. 2019. https://doi.org/10.18653/v1/N19-1087 Description-based zero-shot fine-grained entity typing . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Lingu...

2019 doi
[49]

Yasumasa Onoe and Greg Durrett. 2019. Learning to denoise distantly-labeled data for entity typing. In NAACL-HLT

2019
[50]

Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. http://www.aclweb.org/anthology/D14-1162 Glove: Global vectors for word representation . In Empirical Methods in Natural Language Processing (EMNLP), pages 1532--1543

2014
[51]

Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018 a . https://doi.org/10.18653/v1/N18-1202 Deep contextualized word representations . In Proceedings of the 2018 Conference of the North A merican Chapter of the As...

2018 doi
[52]

Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018 b . https://doi.org/10.18653/v1/D18-1179 Dissecting contextual word embeddings: Architecture and representation . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pa...

2018 doi
[53]

Maxim Rabinovich and Dan Klein. 2017. Fine-grained entity typing with high-multiplicity assignments. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 330--334

2017
[54]

Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain yourself! leveraging language models for commonsense reasoning. arXiv preprint arXiv:1906.02361

2019 arXiv
[55]

Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728

2019 arXiv
[56]

Xing Shi, Inkit Padhi, and Kevin Knight. 2016. https://doi.org/10.18653/v1/D16-1159 Does string-based neural MT learn source syntax? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1526--1534, Austin, Texas. Association for Comp...

2016 doi
[57]

Sameer Singh, Amarnag Subramanya, Fernando Pereira, and Andrew McCallum. 2012. Wikilinks: A large-scale cross-document coreference corpus labeled via links to Wikipedia . Technical Report UM-CS-2012-015

2012
[58]

Robert Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Thirty-First AAAI Conference on Artificial Intelligence

2017
[59]

Spitkovsky and Angel X

Valentin I. Spitkovsky and Angel X. Chang. 2012. http://www.lrec-conf.org/proceedings/lrec2012/pdf/266_Paper.pdf A cross-lingual dictionary for E nglish W ikipedia concepts . In Proceedings of the Eighth International Conference on Language Resources and Evaluation ( LREC -201...

2012
[60]

Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. Ernie: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223

2019 arXiv
[61]

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. https://doi.org/10.18653/v1/N19-1421 C ommonsense QA : A question answering challenge targeting commonsense knowledge . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associa...

2019 doi
[62]

James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human La...

2018
[63]

Trieu H Trinh and Quoc V Le. 2018. A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847

2018 arXiv
[64]

Andreas Vlachos and Sebastian Riedel. 2014. https://doi.org/10.3115/v1/W14-2508 Fact checking: Task definition and dataset construction . In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science , pages 18--22, Baltimore, MD, USA. Assoc...

2014 doi
[65]

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. https://doi.org/10.18653/v1/W18-5446 GLUE : A multi-task benchmark and analysis platform for natural language understanding . In Proceedings of the 2018 EMNLP Workshop B lackbox NLP : A...

2018 doi
[66]

William Yang Wang. 2017. https://doi.org/10.18653/v1/P17-2067 `` liar, liar pants on fire '' : A new benchmark dataset for fake news detection . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 422--426,...

2017 doi
[67]

Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. https://doi.org/10.1162/tacl_a_00240 Mind the GAP : A balanced corpus of gendered ambiguous pronouns . Transactions of the Association for Computational Linguistics, 6:605--617

2018 doi
[68]

Rush, and Stuart M

Sam Wiseman, Alexander M. Rush, and Stuart M. Shieber. 2016. https://doi.org/10.18653/v1/N16-1114 Learning global features for coreference resolution . In Proceedings of the 2016 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human ...

2016 doi
[69]

Yadollah Yaghoobzadeh and Hinrich Sch \"u tze. 2015. https://doi.org/10.18653/v1/D15-1083 Corpus-level fine-grained entity typing using contextual information . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 715--725, Lisbon, P...

2015 doi
[70]

Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2016. https://doi.org/10.18653/v1/K16-1025 Joint learning of the embedding of words and entities for named entity disambiguation . In Proceedings of The 20th SIGNLL Conference on Computational Natural Langu...

2016 doi
[71]

Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2017. Learning distributed representations of texts and entities from knowledge base. Transactions of the Association for Computational Linguistics, 5(1):397--411

2017
[72]

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237

2019 arXiv
[73]

Wenpeng Yin and Dan Roth. 2018. https://doi.org/10.18653/v1/D18-1010 T wo W ing OS : A two-wing optimization strategy for evidential claim verification . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 105--114, Brussels, Belgiu...

2018 doi
[74]

Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A large-scale adversarial dataset for grounded commonsense inference. arXiv preprint arXiv:1808.05326

2018 arXiv
[75]

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. https://www.aclweb.org/anthology/P19-1472 H ella S wag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 47...

2019
[76]

Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. https://www.aclweb.org/anthology/P19-1139 ERNIE : Enhanced language representation with informative entities . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguis...

2019

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In ICLR

work page 2017

[4] [4]

Gabor Angeli and Christopher D. Manning. 2014. https://doi.org/10.3115/v1/D14-1059 N atural LI : Natural logic inference for common sense reasoning . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing ( EMNLP ) , pages 534--545, Doha, Qatar. Association for Computational Linguistics

work page doi:10.3115/v1/d14-1059 2014

[5] [5]

Yonatan Belinkov, Llu \' s M \`a rquez, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. 2017. https://www.aclweb.org/anthology/I17-1001 Evaluating layers of representation in neural machine translation on part-of-speech and semantic tagging tasks . In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volu...

work page 2017

[6] [6]

Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1247--1250. AcM

2008

[7] [7]

Bowman, Gabor Angeli, Christopher Potts, and Christopher D

Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. https://doi.org/10.18653/v1/D15-1075 A large annotated corpus for learning natural language inference . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632--642, Lisbon, Portugal. Association for Computational Linguistics

doi:10.18653/v1/d15-1075 2015

[8] [8]

Hong Chen, Zhenhua Fan, Hao Lu, Alan Yuille, and Shu Rong. 2018. https://www.aclweb.org/anthology/D18-1016 P re C o: A large-scale dataset in preschool vocabulary for coreference resolution . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 172--181, Brussels, Belgium. Association for Computational Linguistics

work page 2018