{"id":"788f0c54-66d9-422e-aa50-92c9466ab2d3","arxiv_id":"2505.22349","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ChatPD automatically builds a paper-dataset network by using LLMs to extract dataset mentions from papers and a graph-based algorithm to match them to known datasets, outperforming PapersWithCode in coverage.","lead":"ChatPD uses large language models to automatically read research papers and extract which datasets they use, then links each mention to a known dataset entry. The result is a continuously updated, searchable paper-dataset network that can help researchers find relevant datasets and improve reproducibility.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Quantitative superiority claims rest on unvalidated author-created labels; unless independent annotation confirms the ground truth, the reported 0.99 extraction precision and 0.88 ER F1 are not established.","rationale":"The paper has real strengths: the system is deployed, code is open-sourced, and the graph-completion ER method is transparent and reproducible. The LLM extraction pipeline is plausible and likely competitive. My concern is not that the authors are dishonest; it is that every quantitative headline in the paper is evaluated only against labels the authors created, with no protocol, agreement statistic, or error bar. I also see a potential structural circularity in the ER evaluation: because the manual linking was performed on or beside the same name/URL identity attributes that Algorithm 1/2 consumes, the 0.88 F1 may measure exact/alias attribute matching rather than general entity resolution, and the comparison to SiGMa/ZeroER may be unfair given those methods lack a no-match option. These issues do not disprove the central claim but make it conditional: the claim should be accepted only with independent validation. Therefore I keep the reader's CONDITIONAL verdict and propose a concrete re-annotation study to settle it.","tokens_in":17952,"tokens_out":10323,"duration_ms":115069,"concrete_test":"Independently re-annotate a random subset of the 119 papers and 1,000 descriptions: two annotators who are not authors, blind to ChatPD's outputs, follow a written protocol (what counts as 'used', how to link to PwC when name/URL differ). Compute Cohen's kappa; keep only instances where both agree, and recompute Fig. 3 and Table 2. If kappa < 0.8, or if extraction F1 or ER F1 drops by more than 0.05 on the agreed subset, the reported superiority is not robust. To test the circularity concern, run a second pass in which name and URL fields are masked, and check whether the graph algorithm's F1 collapses when the annotators cannot use identity attributes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claims--ChatPD's ~0.99 extraction precision/F1 0.872 (Fig. 3, Table 7) and its 0.9826 precision / 0.8727 recall / 0.8829 F1 in entity resolution (Table 2)--are all scored against ground truth created by the authors (Sections 5.1.1 and 5.2.1). No annotation protocol, inter-annotator agreement, confidence interval, or external validation is reported. This makes every headline number unfalsifiable: if the labels encode the annotators' expectations (e.g., that datasets named in the experiments are 'used'), the LLM's agreement with them is partly self-fulfilling. The ER evaluation has an additional structural issue. The manual linking of 1,000 descriptions to PwC entities (Section 5.2.1) was performed with the same identity cues--dataset name and URL--that the proposed Algorithm 1/2 uses exclusively (Section 4.3.1: 'if a dataset description shares the same name or URL as an existing dataset entity, we can conclude that the description refers to that entity'). If annotators used those fields to decide links, the gold standard is generated by the same rule the algorithm implements; the 0.88 F1 then measures exact/alias attribute matching, not general entity resolution, and the comparison with SiGMa/ZeroER is misleading because those methods lack an explicit 'no match' option and were not adapted to the 52.6% of descriptions that have no PwC entity. Thus both label noise and label-construction circularity threaten the core claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ChatPD, an end-to-end system that collects arXiv cs.AI papers, uses LLMs (primarily GPT-4o-mini) with a constrained JSON prompt to extract dataset usage descriptions from paper sections, and then resolves those descriptions to dataset entities in PapersWithCode using a graph-completion and inference algorithm. The system also discovers new dataset entities via a mention-frequency threshold and is deployed as a public dataset-discovery service. The main empirical claims are that ChatPD reaches approximately 0.99 precision in dataset-usage extraction (F1 0.872 in Table 7), outperforms PapersWithCode and rule-based baselines, and reaches 0.8829 F1 in entity resolution, beating SiGMa and ZeroER.","tokens_in":18256,"tokens_out":5954,"duration_ms":59959,"significance":"If the quantitative claims are established, this is a useful systems contribution: an automated, low-cost pipeline for building a paper-dataset network, with a deployed service and a large released network (60,126 papers, 4,224 entities, 137,004 usage records). The multi-LLM comparison, the explicit graph algorithms, the cost analysis, and the open-sourced deployment are strengths. The central caveat is that the headline numbers rest on author-created labels and an entity-resolution gold standard that may share the same matching cues as the algorithm, so the current evidence cannot fully separate a genuinely better extraction/resolution method from one that is tuned to the annotation rule.","major_comments":[{"comment":"The extraction ground truth is created by manually annotating 119 papers without reporting an annotation protocol, inter-annotator agreement, or any variance estimate. Since the paper's central claim of approximately 0.99 precision and 0.872 F1 is computed against these labels, it is not yet possible to assess whether the labels are consistent or whether the LLM's outputs are accidentally aligned with the annotators' expectations. Please add a second annotator on a subset, report agreement (e.g., Cohen's kappa), and provide confidence intervals (e.g., bootstrap by paper) for all headline extraction metrics.","section":"Section 5.1.1, Fig. 3, Table 7"},{"comment":"The entity-resolution evaluation may be partly circular. The manual linking of 1,000 dataset descriptions to PwC entities appears to have been performed with the same identity evidence, dataset name and URL, that Algorithms 1 and 2 use exclusively, and no annotation protocol is reported. If annotators used those fields, the gold standard is generated by the same rule the algorithm implements, so the 0.8829 F1 measures exact and alias attribute matching rather than general entity resolution. Please describe the annotation instructions, report inter-annotator agreement, and provide examples of links that require semantic inference beyond name/URL; if such examples are rare, the comparison with SiGMa and ZeroER should be reframed.","section":"Section 5.2.1, Table 2"},{"comment":"The comparison with SiGMa and ZeroER is not apples-to-apples because 526 of the 1,000 descriptions (52.6%) have no PwC entity, and SiGMa and ZeroER are not given an explicit 'no match' option in the reported setup. It is unclear how unlinked descriptions are scored and whether the baselines ever abstain. Please specify the evaluation protocol for the 526 unlinked descriptions, adapt the baselines with a threshold or an additional 'no match' class, and report results both on all 1,000 descriptions and on the 474 linkable descriptions separately.","section":"Table 2, Section 5.2.1"},{"comment":"The new-entity discovery threshold lambda is a free parameter set to 3 in deployment with no sensitivity analysis. The reported 444 new entities, and therefore the claimed coverage advantage over PwC, depends directly on this threshold. Please vary lambda (e.g., 2, 3, 5, 10) and validate a random sample of newly discovered entities by external inspection to measure the precision of the discovery rule.","section":"Section 4.3.2, Table 4"}],"minor_comments":[{"comment":"The abstract and introduction state that ChatPD achieves 'about 90% precision and recall in entity resolution tasks,' but Table 2 reports precision 0.9826 and recall 0.8727; the claim should be rephrased to match the actual numbers, e.g., 'about 98% precision and 87% recall.'","section":"Abstract, Section 1"},{"comment":"The 1.5k-token input has markedly lower recall than the full-paper input (0.780 vs. 0.850); the text says the two are 'close,' which is fair, but the claim 'even outperforms it in certain metrics like Precision' should be accompanied by the observation that recall drops by about 7 points.","section":"Section 4.2.3, Table 7"},{"comment":"The loop condition 'while iteration_limit is not reached' should be replaced with a concrete stopping criterion (e.g., iterate until no new edges are added) or an explicit statement of the default iteration limit in the pseudocode.","section":"Algorithm 1"},{"comment":"Section 6.1 reports an offline sample of 35,310 cs.AI papers, while Section 6.2 reports 32,959 arXiv cs.AI papers in 2024; the relationship between these two corpora should be clarified.","section":"Section 6.1, Section 6.2"},{"comment":"The columns 'PwC (2024.11.16)' and 'PwC (2025.01.16)' should be described in the text, including how these dates were obtained and what a checkmark means; a versioned snapshot or access date for the PwC database would make the comparison reproducible.","section":"Table 3"},{"comment":"The concluding statement that 'there may be some errors' due to LLM limitations is not quantified; adding a small error analysis or examples of failure cases would make the limitation concrete.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The systems contribution is real and the deployment is valuable, but the evaluation's dependence on author-created gold standards is the main risk. I would ask for the annotation instructions, a second annotator agreement study, and the release of the annotated data alongside the code before recommending acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ChatPD is a worthwhile systems paper. It combines LLM-based dataset extraction with a graph-completion entity resolution step to build a paper-dataset network, and it actually ships: live service, open-source code, and a network of 60k papers, 4k datasets, and 137k usage records. The cost numbers are striking — about $0.00063 per paper with GPT-4o-mini — and the 444 new datasets it finds beyond PapersWithCode give the system independent value. That part deserves credit.\n\nThe extraction evaluation is reasonably convincing. On 119 manually annotated papers from NeurIPS/KDD, GPT-4o-mini with a 1500-token input reaches 0.987 precision and 0.872 F1, beating PapersWithCode (0.733 F1), a spaCy NER model, and regex. The result that a truncated input performs near full-text is useful practically.\n\nThe soft spots are real but not disqualifying. The ground truth is author-created, with no inter-annotator agreement and no variance estimates. For a task like 'which datasets are used in this paper,' that is less worrying than for fine-grained IE, but the headline 0.99 precision should be read as 'agreement with the annotators' expectations,' not an objective gold standard. The entity-resolution evaluation has a more specific issue: the manual linking of 1,000 descriptions to PwC entities was likely done using the same name/URL cues that the graph algorithm uses, so the 0.88 F1 partly measures rule compliance. That said, the algorithm still has to handle aliases and ambiguous URLs, so the result is not meaningless. The comparison to SiGMa and ZeroER is also a bit rough, since those methods lack an explicit 'no match' option and were not adapted to the ~52% unlinked descriptions. A fairer baseline adaptation would strengthen the claim.\n\nThe new-entity threshold lambda is fixed at 3 with no sensitivity analysis, and the graph-completion iteration limit is similarly ad hoc. These are minor; the system's behavior is reasonably stable.\n\nOverall, the central claim holds: an LLM pipeline plus graph completion can build a broad paper-dataset network cheaply and with decent quality. This is a useful contribution to dataset discovery infrastructure, more than a research advance in ER. It deserves a serious peer review. I would push for independent annotation of at least a sample, error bars, and a more careful baseline setup before publication, but I would not desk-reject it.","headline":"A useful LLM-based paper-dataset network with credible system claims, but the evaluation numbers need independent ground-truth validation before they harden.","tokens_in":18770,"tokens_out":2588,"would_cite":true,"duration_ms":27060,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ChatPD claims that an LLM pipeline can extract dataset usage from papers and resolve variant names to canonical entities, outperforming the manually curated PapersWithCode and reaching about 90% precision and recall in entity resolution.","keywords":["dataset discovery","paper-dataset network","large language models","information extraction","entity resolution","graph completion","PapersWithCode","arXiv"],"falsifier":"Ask an independent team, blind to the authors' labels, to annotate the same 119 papers and 1,000 dataset descriptions, or a fresh sample, and rerun the comparisons; if ChatPD no longer beats PapersWithCode on extraction or SiGMa and ZeroER on entity resolution, the central claim fails. A cheaper check is to hand-verify a random sample of the 444 newly discovered dataset entities, each of which should have a working specific URL, a correct name, and genuine use in the cited papers, not a generic repository link or an LLM hallucination.","tokens_in":17742,"feed_emoji":"🔗","tokens_out":10201,"duration_ms":97410,"temperature":0.7,"pith_summary":"This paper claims that a fully automatic LLM pipeline can build a paper-dataset network that is more complete and cheaper than the manually curated PapersWithCode platform. ChatPD combines three modules: collecting paper text, prompting an LLM to emit structured dataset descriptions, and a graph-completion entity-resolution step that maps variant names and URLs to canonical dataset entities. On the authors' own annotations of 119 papers, extraction with GPT-4o-mini beats PwC across exact-match ratio, precision, recall, and F1; on 1,000 manually linked dataset descriptions, the graph method reaches an F1 of 0.8829, ahead of SiGMa and ZeroER. At deployment scale the system built a network of 60,126 arXiv cs.AI papers, 4,224 dataset entities, and 137,004 usage records, including 444 dataset entities not present in PwC. If the manual labels are trustworthy, the implication is that LLM extraction plus graph inference could replace much of the costly human cataloguing behind dataset-discovery platforms.","feed_headline":"LLM system maps 60k papers to datasets, beating manual curation","feed_subtitle":"Automated extraction plus graph inference hits about 88% F1 and finds 444 datasets the curated platform missed.","key_machinery":"The entity-resolution core is a graph whose nodes are dataset descriptions (D-nodes), candidate canonical entities (E-nodes), and identity attributes (I-nodes) such as 'dataset name' and 'dataset URL'. When a description shares an I-node with an entity, the description is matched to that entity; after a match, every I-node belonging to the description is also attached to the entity, so subsequent descriptions sharing any of those identifiers get matched transitively. Ambiguous I-nodes that would link to multiple entities, for instance a generic Kaggle URL, are removed to preserve integrity. This iterative completion runs up to three iterations and is the mechanism that lifts recall from 0.6477 for graph inference without completion to 0.8727.","core_discovery":"The central discovery is that the two hard parts of building a paper-dataset network, recognizing dataset mentions in free text and deciding which mentions name the same dataset, can both be automated with current LLMs plus a deterministic graph algorithm. The paper reports extraction precision around 0.99 with recall around 0.87 for GPT-4o-mini on 119 KDD and NeurIPS papers, against PwC's precision around 0.83 on the same papers. For entity resolution, its Graph Completion and Inference algorithm yields precision 0.9826 and recall 0.8727, an F1 of 0.8829, on 1,000 dataset descriptions, outperforming SiGMa with F1 0.6778 and ZeroER with F1 0.6300. The authors also argue that the same machinery rediscovers datasets PwC missed, listing 444 new entities, including UltraFeedback, which appears in more than 40 papers and was still absent from PwC at the time of writing.","pith_inferences":["Editorial extension: the D-node, E-node, and I-node graph design does not depend on the dataset domain, so the same completion-and-inference machinery could build paper-code, paper-model, or paper-figure networks from the same paper text.","Editorial extension: the new-entity rule of a URL plus at least three mentions will undercount real datasets that are cited without URLs and may overcount entities whose papers share a generic URL, so a human audit of a random sample of the 444 new entities would quantify both error directions.","Editorial extension: the 1500-token input biases extraction toward experimental sections, and the appendix's agentic framework raises recall but lowers precision by pulling in related-work mentions, suggesting that a section-aware mention-role classifier is a natural next step.","Editorial extension: if the results replicate, the long-term shift is from manual annotation to manual calibration of machine-generated records, turning human effort into verification rather than initial cataloguing."],"forward_implications":["If the reported numbers hold, dataset cataloguing can be refreshed automatically: in 2024 arXiv cs.AI, ChatPD extracted dataset information from 85.5% of papers with accessible text while PwC covered 34.5% of the same set.","The new-entity discovery rule adds 444 dataset entities not in PwC, including UltraFeedback, used in more than 40 papers, so manual platforms can be expected to lag behind automated extraction on emerging resources.","The 1500-token cost optimization makes continuous extraction economically feasible at about $0.00063 per paper, with performance close to full-text input.","Because the pipeline needs only paper text, a user could build a private paper-dataset network from local PDFs without depending on any external platform."],"supporting_citations":[{"why":"Supplies the PapersWithCode platform and its dataset database, the comparison baseline for extraction and the seed entity set for resolution.","marker":"[39]"},{"why":"Provides the studies of what researchers look for in a dataset, which determine the JSON template fields the LLM is asked to extract.","marker":"[25]"},{"why":"SiGMa is the unsupervised entity-resolution baseline that Graph Completion and Inference must beat.","marker":"[28]"},{"why":"ZeroER is the other unsupervised entity-resolution baseline, used to show the graph method's F1 advantage.","marker":"[59]"},{"why":"DeepSeek-V3 is one of the alternative LLM backends whose extraction performance supports the claim that the approach is not tied to a single model.","marker":"[35]"},{"why":"Qwen2.5-7b-instruct is the lightweight, locally deployable LLM whose competitive extraction results support the cost and robustness claims.","marker":"[53]"},{"why":"ar5iv supplies the text-format papers that the deployed pipeline feeds to the extraction module.","marker":"[52]"}],"fun_headline_variants":["AI maps research papers to datasets, beats manual curation","LLM system finds 444 datasets missed by PapersWithCode","Graph inference boosts dataset entity resolution to 88% F1","Automated paper-dataset extraction hits 99% precision","LLM plus graph algorithm automates dataset linking"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every headline performance number rests on labels that the authors wrote themselves for 119 papers and 1,000 dataset descriptions, with no second annotator, no agreement score, and no confidence interval, so if those labels are biased or noisy the claimed superiority over PapersWithCode and the baseline entity-resolution methods is not established.","fun_headline_variants_meta":{"raw":{"variants":["AI maps research papers to datasets, beats manual curation","LLM system finds 444 datasets missed by PapersWithCode","Graph inference boosts dataset entity resolution to 88% F1","Automated paper-dataset extraction hits 99% precision","LLM plus graph algorithm automates dataset linking"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001165,"raw_usage":{"total_tokens":4830,"prompt_tokens":960,"completion_tokens":3870,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":3790}},"tokens_in":576,"tokens_out":3870,"duration_ms":28325,"temperature":1.0,"reasoning_tokens":3790,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:08:57.015669+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask an independent team, blind to the authors' labels, to annotate the same 119 papers and 1,000 dataset descriptions, or a fresh sample, and rerun the comparisons; if ChatPD no longer beats PapersWithCode on extraction or SiGMa and ZeroER on entity resolution, the central claim fails. A cheaper check is to hand-verify a random sample of the 444 newly discovered dataset entities, each of which should have a working specific URL, a correct name, and genuine use in the cited papers, not a generic repository link or an LLM hallucination.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the PapersWithCode platform and its dataset database, the comparison baseline for extraction and the seed entity set for resolution."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the studies of what researchers look for in a dataset, which determine the JSON template fields the LLM is asked to extract."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SiGMa is the unsupervised entity-resolution baseline that Graph Completion and Inference must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ZeroER is the other unsupervised entity-resolution baseline, used to show the graph method's F1 advantage."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ar5iv supplies the text-format papers that the deployed pipeline feeds to the extraction module."}],"review_version":1}