Pith. sign in

REVIEW 2 cited by

Crawling the Internal Knowledge-Base of Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2301.12810 v1 pith:OGBOBIWW submitted 2023-01-30 cs.CL cs.AI

Roi Cohen , Mor Geva , Jonathan Berant , Amir Globerson This is my paper · ORCID
classification cs.CLcs.AI
keywords factslanguagecrawlingmodelsbodyentitygeneratedgiven
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Language models are trained on large volumes of text, and as a result their parameters might contain a significant body of factual knowledge. Any downstream task performed by these models implicitly builds on these facts, and thus it is highly desirable to have means for representing this body of knowledge in an interpretable way. However, there is currently no mechanism for such a representation. Here, we propose to address this goal by extracting a knowledge-graph of facts from a given language model. We describe a procedure for ``crawling'' the internal knowledge-base of a language model. Specifically, given a seed entity, we expand a knowledge-graph around it. The crawling procedure is decomposed into sub-tasks, realized through specially designed prompts that control for both precision (i.e., that no wrong facts are generated) and recall (i.e., the number of facts generated). We evaluate our approach on graphs crawled starting from dozens of seed entities, and show it yields high precision graphs (82-92%), while emitting a reasonable number of facts per entity.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Simulating Tabular Datasets through LLMs to Rapidly Explore Hypotheses about Real-World Entities

    cs.AI 2024-11 conditional novelty 6.0 of 10

    LLMs can approximate tabular datasets about real-world entities well enough for rapid hypothesis exploration, and fidelity improves with model scale.

  2. When to Speak, When to Abstain: Contrastive Decoding with Abstention

    cs.CL 2024-12 conditional novelty 5.0 of 10

    CDA is a training-free decoding method that weights parametric, contextual, and abstention distributions using null-prompt calibrated entropy, letting LLMs answer when they can and abstain when they cannot.

Pith tools