Hidden Entity Detection from GitHub Leveraging Large Language Models

Brigitte Mathiak; Danilo Dessi; Lu Gan; Martin Blum; Ralf Schenkel; Stefan Dietze

REVIEW 3 major objections 6 minor 25 references

Hidden Entity Detection from GitHub Leveraging Large Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Few-shot LLMs cannot reliably detect and classify dataset and software URLs in GitHub READMEs for knowledge graph construction.

desk verdict A narrow, honestly reported negative result on 7B LLMs for URL extraction from GitHub READMEs, with a useful new dataset; just don't buy the broader 'off-the-shelf models' conclusion. read the letter →

arxiv 2501.04455 v1 pith:JWT4RCDJ submitted 2025-01-08 cs.CL cs.DL

Lu Gan , Martin Blum , Danilo Dessi , Brigitte Mathiak , Ralf Schenkel , Stefan Dietze This is my paper · ORCID

classification cs.CLcs.DL

keywords NamedEntityRecognitionLargeLanguageModelsFew-shotlearningKnowledgegraphconstructionURLclassificationGitHubREADMEsDatasetdetectionSoftware

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether off-the-shelf large language models can find and classify the URLs in GitHub READMEs that point to datasets and software, a task needed to populate knowledge graphs from research repositories. Using few-shot prompting of Llama 2 7B, Mistral 7B, and their 4-bit quantized versions on 811 repositories with 1,439 manually annotated URLs, the authors report that URL detection is often accurate, but fine-grained classification into dataset direct links, dataset landing pages, software, and other is not. The best model reaches strict precision and recall around 0.5 in the joint extraction-and-classification task, and dynamic example selection does not help. The paper concludes that off-the-shelf models are inadequate for this high-precision, high-recall task, and that expectations for LLM-based knowledge graph population should be tempered.

What carries the argument

The study's machinery is a few-shot prompt template that casts the model as a human annotator, instructs it to emit a JSON array of URL/label pairs, and supplies four static or four dynamically selected examples; the four labels are Dataset Direct Link, Dataset Landing Page, Software, and Other. Evaluation follows the MUC-5 precision/recall schema applied under four matching modes—strict, exact, partial, and type—with a longest-common-substring bipartite matching step that aligns model output URLs to gold URLs. The control knob is the comparison between the joint extraction-and-classification task and a classification-only task that feeds the URL and its context to the model.

What would settle it

If any unmodified off-the-shelf 7B-class LLM, using the paper's own prompt templates on the same 1,439-URL gold standard, achieved strict precision and strict recall both above 0.9 in the extraction-and-classification task, the central claim that off-the-shelf models are inadequate for this task would be refuted.

Watch

Extended reading notes

Core claim

On its own terms, the paper's finding is that a straightforward few-shot LLM pipeline—instructions plus four worked examples, with output parsed into JSON—is not yet a viable substitute for manual annotation or regex-based heuristics when the goal is a knowledge graph with reliable entity types. The models can reproduce or partially match a URL most of the time, with exact-match precision and recall often above 0.8, but the four-way type assignment collapses: strict precision for Llama 2 7B in the extraction-and-classification task is at best 0.371 and for Mistral 7B at best 0.519. The bottleneck is not seeing URLs but deciding what they are, particularly separating a dataset direct link from a dataset landing page and both from software. The paper further argues that richer context (classification-only prompts with the URL supplied) does not rescue the models, and that a simpler two-way dataset-versus-not decision is substantially more reliable.

Load-bearing premise

The results depend on the manually assigned gold-standard labels for 1,439 URLs being correct and consistent, and on the longest-common-substring matching between model output and gold URLs faithfully representing what the model actually detected.

Editorial extensions

If this is right

LLM outputs must be post-processed and matched to expected URLs before scoring, and output-format adherence alone costs 3–14% of responses; any production pipeline needs a structured-output wrapper or repair step.
Regex- or link-based extraction remains a stronger baseline for finding URLs in READMEs than generative extraction, since LLMs miss URLs and hallucinate non-existent ones.
A binary dataset-versus-not classification is a more realistic LLM use case than four-way typing, with Mistral reaching about 0.75 type-level recall in the classification-only setting.
Static and dynamic few-shot examples perform comparably in this task, so the extra cost of similarity-based example selection is not justified by the measured results.
Because the strict and type scores are far below exact scores, evaluation for KG population must report entity-type accuracy separately from boundary detection, not just URL overlap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural hybrid that follows from these numbers is to keep regex or link extractors for URL discovery and reserve the LLM for classifying already-extracted URLs, isolating the one subtask where models show partial ability.
The broad 'Dataset Landing Page' class, which includes folders, GitHub files, and software that generates a dataset, may be inflating confusion; a finer or hierarchical label set could change the measured ceiling.
Because quantization cost little on several settings and Mistral 7B beat Llama 2 7B, testing current 7–8B instruction-tuned models on the same 1,439-URL gold standard is a cheap, decisive extension.
The negative result is itself evidence for the field that prompt engineering alone is unlikely to close the gap, and that KG population should invest in task-specific training data or constrained decoding.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The genuinely new things here are the dataset (811 repositories, 1,439 manually labeled URLs) and the first evaluation of few-shot open 7B LLMs on URL-as-entity detection and classification in READMEs. The paper is scoped, the resources are released, and the reporting of parsing failures is honest. That is real value for a niche but active problem in knowledge-graph population.

What the paper does well: it separates extraction from classification, defines four evaluation modes (strict, exact, partial, type), and reports both. It also compares static and dynamic few-shot prompting and finds no clear win, which is a useful counterpoint to earlier claims about dynamic examples. The finding that Llama 2 7B and Mistral 7B, including quantized variants, get strict precision/recall around 0.3–0.5 on this task is credible for those models.

The soft spots are in proportion: the conclusion overreaches from two 7B open models to "off-the-shelf models" generally. No larger or commercial model is tested, so the claim should be narrowed. There is also no measured regex baseline, even though the paper says regex-based methods are better; that is an assertion, not a result. Zero-shot results are dropped because of low parsing rates, and unparsable outputs are scored as invalid, which mixes formatting failure with task failure. No error bars or significance tests are reported, and the numbers in Tables 2 and 3 are close enough in places that some bootstrapping would help.

The stress-test concern about gold-standard reliability is real but not fatal. The DatasetLandingPage class is broad, no inter-annotator agreement is reported, and no adjudication protocol is described. That could add label noise to strict and type scores. But the exact and partial URL extraction scores are high across models, and those do not depend on class labels. So the extraction half of the negative result probably survives label noise; the classification half is shakier. Because the dataset is published, reannotation or an IAA study is actually feasible, and I would ask for that in revision.

Who gets value from this: researchers building KG pipelines from GitHub or other semi-structured sources, and people benchmarking small open LLMs on structured extraction. It is a legitimate empirical contribution, not a tech breakthrough. I would send it to peer review, with revisions: tighten the claim to the tested models, add a regex baseline, report IAA, and give error bars. It deserves a serious referee, and with those changes it would be a solid workshop or short-conference paper.

Referee Report

3 major / 6 minor

Summary. The paper investigates whether open-weight 7B LLMs (Llama 2 7B, Mistral 7B, and their 4-bit quantized versions) can extract and classify dataset and software URLs from GitHub READMEs using few-shot prompts. The authors construct a gold standard of 1,439 URLs from 811 repositories with four classes, apply four prompt variants (static/dynamic examples, extraction+classification vs classification-only), and report precision and recall under strict, exact, partial, and type matching. They find modest classification performance, frequent parsing failures, and conclude that off-the-shelf models are inadequate for knowledge-graph-grade URL identification and classification.

Significance. If taken at face value, the paper provides a useful negative result and a publicly released manually annotated dataset for an underexplored NER variant in which the entities are URLs. Strengths include the manual annotation effort, the comparison of four model variants and four prompt settings, and the explicit analysis of output parsing failures. The evaluation is measured against an independent gold standard, and no model parameters are fitted to the test outcome, so the reported numbers are not circular. The contribution is weakened by the narrow model scope, the absence of reliability evidence for the gold standard, and the lack of statistical analysis; these gaps currently prevent the strong conclusion about 'off-the-shelf models' from being fully supported.

major comments (3)

[Section 2.3 (Gold Standard Data)] The definition of Dataset Landing Page is exceptionally broad: it includes index pages, software repositories that merely generate or download a dataset, and even a URL to a file in a GitHub repository where the dataset is stored in another file. Since 678 of the 1,439 URLs (47%) belong to this class, the boundary between Dataset Landing Page and Software, and between Dataset Landing Page and Dataset Direct Link, becomes a matter of annotator judgment. The paper reports no inter-annotator agreement, no adjudication protocol, and no annotation error analysis. If labels in this largest class are applied inconsistently, the strict and type precision/recall values in Tables 2 and 3, and hence the central claim that LLMs are inadequate, could be artifacts of label noise rather than genuine model limitations. Please report inter-annotator agreement on a subsample, an adjudication procedure, or a sensitivity analysis that shows the main conclusions are robust to reasonable label disagreements.
[Section 2.4-2.5 and Table 1] The evaluation treats unparsable outputs as invalid and, for zero-shot learning, the parsed ratio is so low (e.g., 18/733 for Llama 2 7b) that zero-shot results are omitted entirely. Consequently, the evidence for the claim that 'off-the-shelf models are inadequate' rests only on few-shot results from which 3.1% to 14.2% of outputs were discarded. No confidence intervals, error bars, or significance tests are reported for the differences between models or prompt variants, so it is unclear whether the observed differences in strict and type scores are meaningful. Please report zero-shot parsing-recovered results if they can be salvaged, provide uncertainty estimates for the main metric comparisons, or explicitly restrict the conclusion to the few-shot setting with a caveat about discarded outputs.
[Section 4 (Conclusion and Outlook)] The concluding sentence that 'off-the-shelf models are inadequate for addressing intricate tasks demanding high precision and recall' goes beyond the evidence, which covers only Llama 2 7B, Mistral 7B, and their 4-bit quantized versions. No larger open models or commercial LLMs are tested, and no quantitative comparison with a regular-expression or majority-class baseline is provided, even though the text states that regex-based methods achieve higher precision for URL detection. Please scope the conclusion to the tested model family and size, or add experiments with larger/commercial models and at least one non-LLM baseline to support the broader claim.

minor comments (6)

[Section 2.2 (Large Language Models)] The text mentions 'LLama 3 and Llama 3.1' but the experiments only use Llama 2; please clarify which models are actually evaluated and fix the typo 'LLama'.
[Figure 1] The prompt instructions define class labels in camelCase (e.g., 'DatasetLandingPage') while the example output uses lowercase snake_case (e.g., 'dataset_landing_page'); please ensure the parser maps both formats consistently and state which canonical form is used in evaluation.
[Section 3 (Results and Discussion)] The observation that dynamic examples do not improve performance 'in contrast with recent literature [25]' is made without a statistical test; please add a caveat or a significance test before drawing this comparison.
[Abstract] The abstract emphasizes zero-shot learning as part of the investigated approach, but zero-shot results are excluded from the reported evaluation due to parsing failures; please adjust the abstract or include the zero-shot parsing statistics to avoid overstating the scope.
[Introduction & Background] The sentence 'All the resources of this paper are available here' does not include an actual URL or repository identifier; please provide a link or DOI.
[References] Reference [17] cites the Llama 1 paper 'Llama: Open and efficient foundation language models', but the text refers to Llama 2; please update the reference to the correct Llama 2 publication.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: all metrics are measured against an independent manual gold standard, with no fitted parameters or self-referential predictions.

full rationale

The paper's derivation chain is observational rather than constructive. In Section 2.3, the gold standard of 1,439 URLs is manually annotated by researchers who open each URL in a browser, and these labels are independent of the LLM outputs produced in the experiments. The few-shot examples in Section 2.1 are either fixed or selected by textual similarity to the input context, so no test-label information is encoded as a prompt parameter. Precision and recall are computed in Section 2.5 by comparing parsed LLM outputs against this fixed gold standard using the standard MUC schema, and no model parameter or threshold is fitted to the evaluation set. The conclusion that off-the-shelf LLMs are inadequate for high-precision URL identification and classification is therefore a measured empirical claim, not a consequence of how the inputs were defined. Concerns about the breadth of the DatasetLandingPage category, the absence of inter-annotator agreement measures, and the longest-common-substring matching heuristic are validity threats to the gold standard, not instances of circular reasoning. The paper is self-contained against an external benchmark in the relevant sense: its target quantities are measured, not derived from its assumptions.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the quality and consistency of the manual gold standard, the URL-matching heuristic, and the representativeness of the unarXiv-seeded repository sample; none of these are externally verified, and the paper reports no inter-annotator agreement. No free parameters are fitted: the prompt examples are fixed or similarity-selected but not used to tune the model. No new entities are postulated.

assumptions (3)

domain assumption The manually annotated labels are correct ground truth for URL classes (Dataset Direct Link, Dataset Landing Page, Software, Other).
Section 2.3: all URLs are manually annotated by researchers opening each URL; no inter-annotator agreement is reported, and the Dataset Landing Page class is broad (includes software repositories that generate datasets).
domain assumption Longest-common-substring bipartite matching correctly aligns predicted URLs with gold URLs.
Section 2.4: predicted URL spans are matched to ground-truth URLs by longest common substring ratio; this can assign credit to partially overlapping or hallucinated URLs and affects precision and recall.
domain assumption The sample of 811 GitHub repositories seeded from unarXiv is representative of research software and dataset mentions.
Section 2.3: the gold standard is built from GitHub URLs in unarXiv papers; results may not transfer to general GitHub READMEs.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Hidden Entity Detection from GitHub Leveraging Large Language Models." pith.science (2026). https://pith.science/paper/JWT4RCDJ

@misc{pith2026250104455,
  author       = {Pith},
  title        = {Pith review of: Hidden Entity Detection from GitHub Leveraging Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JWT4RCDJ}},
  note         = {Machine review of arXiv:2501.04455}
}

read the original abstract

Named entity recognition is an important task when constructing knowledge bases from unstructured data sources. Whereas entity detection methods mostly rely on extensive training data, Large Language Models (LLMs) have paved the way towards approaches that rely on zero-shot learning (ZSL) or few-shot learning (FSL) by taking advantage of the capabilities LLMs acquired during pretraining. Specifically, in very specialized scenarios where large-scale training data is not available, ZSL / FSL opens new opportunities. This paper follows this recent trend and investigates the potential of leveraging Large Language Models (LLMs) in such scenarios to automatically detect datasets and software within textual content from GitHub repositories. While existing methods focused solely on named entities, this study aims to broaden the scope by incorporating resources such as repositories and online hubs where entities are also represented by URLs. The study explores different FSL prompt learning approaches to enhance the LLMs' ability to identify dataset and software mentions within repository texts. Through analyses of LLM effectiveness and learning strategies, this paper offers insights into the potential of advanced language models for automated entity detection.

Figures

Figures reproduced from arXiv: 2501.04455 by the authors.

**Figure 1.** Prompt template for extraction and classification (E+CL) task. We provide four examples for each few-shot prompting query. In the prompt template, we showcase an example input and output in Example 1. <s>[INST]<<sys>>You act as a human annotator. First read the instructions and given examples, then only annotate the last given input accordingly without extra words. Your annotation has to use valid JSON syntax.<</sys… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 17 canonical work pages

[1]

Verma, R

S. Verma, R. Bhatia, S. Harit, S. Batish, Scholarly knowledge graphs through structuring scholarly communication: a review, Complex & Intelligent Systems 9 (2023) 1059–1095

work page 2023
[2]

H. Ye, N. Zhang, H. Chen, H. Chen, Generative knowledge graph construction: A review, arXiv preprint arXiv:2210.12714 (2022)

arXiv 2022
[3]

Dessí, F

D. Dessí, F. Osborne, D. R. Recupero, D. Buscaldi, E. Motta, Scicero: A deep learning and nlp approach for generating scientific knowledge graphs in the computer science domain, Knowledge-Based Systems 258 (2022) 109945

work page 2022
[4]

Al-Moslmi, M

T. Al-Moslmi, M. G. Ocaña, A. L. Opdahl, C. Veres, Named entity extraction for knowledge graphs: A literature overview, IEEE Access 8 (2020) 32862–32881

work page 2020
[5]

Milošević, W

N. Milošević, W. Thielemann, Comparison of biomedical relationship extraction methods and models for knowledge graph creation, Journal of Web Semantics 75 (2023) 100756

work page 2023
[6]

S. Deng, N. Zhang, J. Kang, Y. Zhang, W. Zhang, H. Chen, Meta-learning with dynamic- memory-based prototypical network for few-shot event detection, in: Proceedings of the 13th International Conference on Web Search and Data Mining, 2020, pp. 151–159

work page 2020
[7]

Rossi, D

A. Rossi, D. Barbosa, D. Firmani, A. Matinata, P. Merialdo, Knowledge graph embedding for link prediction: A comparative analysis, ACM Transactions on Knowledge Discovery from Data (TKDD) 15 (2021) 1–49

work page 2021
[8]

Carta, A

S. Carta, A. Giuliani, L. Piano, A. S. Podda, L. Pompianu, S. G. Tiddia, Iterative zero-shot llm prompting for knowledge graph construction, arXiv preprint arXiv:2307.01128 (2023)

arXiv 2023

Show all 25 references

[9]

Schindler, B

D. Schindler, B. Zapilko, F. Krüger, Investigating software usage in the social sciences: A knowledge graph approach, in: European Semantic Web Conference, Springer, 2020, pp. 271–286

2020
[10]

Dessí, F

D. Dessí, F. Osborne, D. Reforgiato Recupero, D. Buscaldi, E. Motta, Cs-kg: A large-scale knowledge graph of research entities and claims in computer science, in: International Semantic Web Conference, Springer, 2022, pp. 678–696

2022
[11]

Accessed: 2024-02-28

HuggingfaceURL, Hugging Face – The AI community building the future., https:// huggingface.co/, 2024. Accessed: 2024-02-28

2024
[12]

Accessed: 2024-02-28

PyTorchHubURL, PyTorch Hub, https://pytorch.org/hub/, 2024. Accessed: 2024-02-28

2024
[13]

Accessed: 2024-02- 28

GitHubURL, GitHub: Let’s build from here, https://github.com/, 2024. Accessed: 2024-02- 28

2024
[14]

Accessed: 2024-02-28

BitBucketURL, Bitbucket | Git solution for teams using Jira, https://bitbucket.org/, 2024. Accessed: 2024-02-28

2024
[15]

Accessed: 2024-02-28

ZenodoURL, Zenodo, https://zenodo.org/, 2024. Accessed: 2024-02-28

2024
[16]

Ferrari Dacrema, P

M. Ferrari Dacrema, P. Cremonesi, D. Jannach, Are we really making much progress? a worrying analysis of recent neural recommendation approaches, in: Proceedings of the 13th ACM conference on recommender systems, 2019, pp. 101–109

2019
[17]

Touvron, T

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al., Llama: Open and efficient foundation language models, arXiv preprint arXiv:2302.13971 (2023)

2023 arXiv
[18]

A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al., Mistral 7b, arXiv preprint arXiv:2310.06825 (2023)

2023 arXiv
[19]

Ainslie, J

J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, S. Sanghai, Gqa: Training generalized multi-query transformer models from multi-head checkpoints, arXiv preprint arXiv:2305.13245 (2023)

2023 arXiv
[20]

Child, S

R. Child, S. Gray, A. Radford, I. Sutskever, Generating long sequences with sparse trans- formers, arXiv preprint arXiv:1904.10509 (2019)

2019 arXiv
[21]

Beltagy, M

I. Beltagy, M. E. Peters, A. Cohan, Longformer: The long-document transformer, arXiv preprint arXiv:2004.05150 (2020)

2020 arXiv
[22]

Saier, M

T. Saier, M. Färber, unarXive: A Large Scholarly Data Set with Publications’ Full-Text, Annotated In-Text Citations, and Links to Metadata, Scientometrics 125 (2020) 3085–3108. URL: https://doi.org/10.1007/s11192-020-03382-z

2020 doi
[23]

Saier, M

T. Saier, M. Färber, unarXive: A Large Scholarly Data Set with Publications’ Full-Text, Annotated In-Text Citations, and Links to Metadata, 2020. URL: https://doi.org/10.5281/ zenodo.4313164. doi:10.5281/ZENODO.4313164, version 4

2020 doi
[24]

Chinchor, B

N. Chinchor, B. Sundheim, MUC-5 evaluation metrics, in: Fifth Message Understanding Conference (MUC-5): Proceedings of a Conference Held in Baltimore, Maryland, August 25-27, 1993, 1993. URL: https://aclanthology.org/M93-1007

1993
[25]

B. Ding, C. Qin, L. Liu, Y. K. Chia, B. Li, S. Joty, L. Bing, Is GPT-3 a good data annotator?, in: A. Rogers, J. Boyd-Graber, N. Okazaki (Eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Com...

2023 doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

Verma, R

S. Verma, R. Bhatia, S. Harit, S. Batish, Scholarly knowledge graphs through structuring scholarly communication: a review, Complex & Intelligent Systems 9 (2023) 1059–1095

work page 2023

[2] [2]

H. Ye, N. Zhang, H. Chen, H. Chen, Generative knowledge graph construction: A review, arXiv preprint arXiv:2210.12714 (2022)

arXiv 2022

[3] [3]

Dessí, F

D. Dessí, F. Osborne, D. R. Recupero, D. Buscaldi, E. Motta, Scicero: A deep learning and nlp approach for generating scientific knowledge graphs in the computer science domain, Knowledge-Based Systems 258 (2022) 109945

work page 2022

[4] [4]

Al-Moslmi, M

T. Al-Moslmi, M. G. Ocaña, A. L. Opdahl, C. Veres, Named entity extraction for knowledge graphs: A literature overview, IEEE Access 8 (2020) 32862–32881

work page 2020

[5] [5]

Milošević, W

N. Milošević, W. Thielemann, Comparison of biomedical relationship extraction methods and models for knowledge graph creation, Journal of Web Semantics 75 (2023) 100756

work page 2023

[6] [6]

S. Deng, N. Zhang, J. Kang, Y. Zhang, W. Zhang, H. Chen, Meta-learning with dynamic- memory-based prototypical network for few-shot event detection, in: Proceedings of the 13th International Conference on Web Search and Data Mining, 2020, pp. 151–159

work page 2020

[7] [7]

Rossi, D

A. Rossi, D. Barbosa, D. Firmani, A. Matinata, P. Merialdo, Knowledge graph embedding for link prediction: A comparative analysis, ACM Transactions on Knowledge Discovery from Data (TKDD) 15 (2021) 1–49

work page 2021

[8] [8]

Carta, A

S. Carta, A. Giuliani, L. Piano, A. S. Podda, L. Pompianu, S. G. Tiddia, Iterative zero-shot llm prompting for knowledge graph construction, arXiv preprint arXiv:2307.01128 (2023)

arXiv 2023