Pith. sign in

REVIEW 4 major objections 5 minor 21 references

AskChem replaces paper-ranked search with provenance-carrying claims so chemists and agents can assemble verifiable cross-paper answers.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 02:00 UTC pith:SBAFYN5R

load-bearing objection Shipped claim-level chemistry IR with real provenance plumbing and a clean DOI-hallucination win; the bench proves resolvability more than semantic fidelity of the extracted claims. the 4 major comments →

arxiv 2607.28618 v1 pith:SBAFYN5R submitted 2026-07-30 cs.CL cs.AIcs.IRcs.LG

AskChem: Claim-Centered Infrastructure for Chemistry Literature Synthesis

Bing Yan , Gregory Wolfe , Stefano Martiniani , Kyunghyun Cho This is my paper
classification cs.CL cs.AIcs.IRcs.LG
keywords claim-centered retrievalchemistry literature synthesisprovenance groundingfaceted taxonomyevidence graphscientific RAGagent toolscitation groundedness
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Chemistry answers are often scattered as specific findings across many papers, but search tools still return ranked document lists that force people and AI agents to hunt, check, and stitch evidence by hand. This paper argues that the right retrieval unit is the atomic, typed scientific claim, each tied to a source DOI and a verbatim quote or explicit evidence locator. Over a shared store of such claims, AskChem adds three complementary structures: a stabilized faceted taxonomy for hierarchical search and browse, an evidence graph that links claims by relations such as supports or contradicts, and an exploratory living taxonomy that places papers under governing principles. The live system indexes millions of claims from a large chemistry corpus and exposes the same objects through a web UI and agent APIs. On a cross-paper benchmark, grounding a language-model reader in AskChem raises resolvable citations to 100% and yields the highest citation density among the systems compared.

Core claim

Making the provenance-carrying claim the central object of chemistry literature infrastructure—and grounding a reader in that claim store—improves cross-paper synthesis: on AskChem-Bench it produces fully resolvable DOIs and the densest verified citation set among five tested settings, while remaining usable as interactive search and as an agent tool.

What carries the argument

The provenance-carrying claim: an atomic, typed assertion extracted from a paper, always grounded by a source DOI plus a verbatim quote or evidence locator, and shared as the identity across faceted taxonomy paths, evidence-graph edges, and agent-facing APIs.

Load-bearing premise

That schema checks and source grounding are enough to treat extracted claims as reliable synthesis units, even though they do not prove every claim correctly interprets its source.

What would settle it

Run the same AskChem-Bench cross-paper questions with human expert audit of whether each cited claim’s meaning matches its source quote, and check whether DOI resolvability and answer quality still beat document-only retrieval when semantic claim errors are counted against the system.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Cross-paper chemistry questions can be answered from claim-level evidence with citations that resolve to real papers rather than fabricated DOIs.
  • The same claim objects can serve both human browsing and agent tools without rebuilding separate retrieval stacks.
  • Supports, extends, and contradicts links let users move from one finding to related or conflicting evidence across papers.
  • Faceted and principle-centered views give operational and exploratory organization over one shared claim store at corpus scale.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Claim-level stores may become a standard substrate for scientific RAG, reducing citation hallucination more than paper-rankers alone.
  • If semantic claim validation lags provenance checks, community flagging and expert audits will matter as much as scale for high-stakes use.
  • The design generalizes beyond chemistry wherever answers must be assembled from many scattered measurements and conditions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. AskChem proposes claim-centered infrastructure for chemistry literature search and synthesis: papers are segmented into atomic, typed claims grounded by DOI plus a verbatim quote or evidence locator, then indexed (2.4M claims / 147K papers) under a stabilized faceted taxonomy, an evidence graph (supports/contradicts/extends/etc.), and an exploratory principle-centered Living Taxonomy. The system is deployed with web, REST, SDK, and MCP access. Evaluation covers provenance checks (100% source-grounded claims), a small expert audit of graph edges (97.9% type precision on 146 decidable edges), corpus-scale operation, and AskChem-Bench (30 cross-paper questions) where a GPT-5.5 reader grounded in AskChem achieves 100% CrossRef-resolvable DOIs (vs 88.3% without retrieval) and the highest citation density among five systems (Table 1; Fig. 6).

Significance. If the result holds under tighter semantic validation, the work is a useful systems contribution: it shifts retrieval from ranked papers to reusable, provenance-carrying claims and ships a live, agent-accessible store at meaningful chemistry scale. Strengths that should be credited include the deployed open index and interfaces, explicit provenance fields on every claim, CrossRef-verified DOI metrics, judge calibration against expert labels (κ=0.914), a domain-expert edge audit, and public code/data/benchmark artifacts. The paper is strongest as infrastructure for grounded retrieval and citation hygiene; its significance for end-to-end chemical synthesis quality is currently less established than the DOI-groundedness result.

major comments (4)
  1. [§7 RQ3; Table 1; Fig. 6] Abstract, §7 RQ3, Table 1, and Fig. 6: the headline claim that claim-centered retrieval “improves cross-paper chemistry synthesis” is mainly supported by DOI existence (100%) and citation density (18.1). Those metrics establish citation plumbing, not that structured claim fields (outcomes, conditions, reactants) faithfully interpret sources. RQ1 and Limitations already state schema checks “do not prove… semantic… correctness.” Fig. 6 only contrasts fabricated vs resolvable DOIs. Please add a semantic fidelity audit on the claims actually used in bench answers (value/condition match to quote or full text), or narrow the claim to citation groundedness/density.
  2. [Table 1; §7 RQ3; Appendix A] Table 1: Paperclip also reaches 100% DOI existence, so the jump from the no-retrieval baseline largely shows grounded RAG vs parametric memory, not a unique benefit of claim segmentation. Edison Scientific substantially outperforms AskChem on grounded specificity (29.2 vs 5.9) and slightly on on-topic rate. Without a paper-level RAG control using the same reader/budget, or an ablation that disables claim structure/taxonomy/graph while keeping the corpus, the causal role of the claim unit remains under-identified. Add such a control or revise the comparative claim.
  3. [§4; §5; §7 RQ2] §4 and RQ2: the stabilized faceted taxonomy is described as a retrieval signal in hybrid search, yet the paper reports neither isolation of taxonomy-recall gain nor expert validation of path placement. Living Taxonomy (§5) is explicitly exploratory and unvalidated. These structures are load-bearing in the architecture diagram and contributions list; either quantify their contribution (ablation on AskChem-Bench or retrieval metrics) or move them to clearly optional/exploratory status in the abstract and contribution bullets.
  4. [§3; Appendix B] §3 evidence-graph audit: 97.9% edge-type precision on 146 decidable edges from a stratified sample of 148 is encouraging but thin for a core navigational layer (171K edges). Report sampling fractions by relation type, confidence calibration, and error modes (especially contradicts), and state whether graph edges were used in the AskChem-Bench grounded reader or only in UI neighborhood views.
minor comments (5)
  1. [Appendix A] Model names such as GPT-5.5 and Gemini 3.1 Pro should be pinned to provider snapshot/version dates for reproducibility; Appendix A protocol is otherwise helpful.
  2. [Figure 3] Figure 3 mixes corpus statistics with quality claims; clarify that claim-type and subfield histograms are automatic labels, not expert-coded distributions.
  3. [Table 1; Appendix A] Table 1 “Grounded specificity” definition (Appendix A) is token-level and favors long quantitative dumps; discuss this bias when interpreting Edison’s large lead.
  4. [Figure 3; Limitations] Limitations correctly notes shallow abstract extraction and partial chemistry coverage; surface the 66%/34% full-paper vs abstract split earlier when discussing claim completeness for conditions/limitations.
  5. [§1; §5] Minor polish: spacing in “AskChemaddresses” / “CO 2” and consistent capitalization of Living Taxonomy vs faceted taxonomy.

Circularity Check

0 steps flagged

No significant circularity: empirical systems paper evaluated against external CrossRef checks and independent baselines.

full rationale

AskChem is an infrastructure/systems paper whose load-bearing claims are empirical (claim extraction at corpus scale; hybrid retrieval; AskChem-Bench metrics), not a first-principles derivation. The headline result—100% CrossRef-resolvable DOIs and highest citation density when grounding GPT-5.5 in the claim store versus a no-retrieval baseline and four other systems (Table 1; Fig. 6)—is checked by an external registry (CrossRef) and a relevance judge calibrated to domain-expert labels (κ=0.914). Nothing in the paper defines a quantity in terms of the target it then “predicts,” fits a parameter and renames the fit as prediction, or imports a uniqueness/ansatz result from overlapping-author prior work as a forced mathematical fact. Judging AskChem against the claims it exposes (Appendix A) is the appropriate protocol for a retrieval tool, not a self-definitional loop. Limitations and RQ1 explicitly disclaim semantic correctness of extractions; that is honesty about scope, not circular reasoning. Score 0.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 5 invented entities

Load-bearing commitments are engineering and evaluation assumptions, not physical axioms: LLMs can segment papers into typed claims; provenance fields proxy trustworthiness for retrieval; hybrid FTS+vector+taxonomy fusion is an adequate retriever; a 30-question bench and CrossRef DOI checks measure synthesis quality; expert edge sample generalizes. No fitted physical constants. Invented entities are software constructs (claim store, facets, living taxonomy), with independent existence as the live service and released index.

free parameters (4)
  • Hybrid search fusion / evidence budget (≤40 claims; 3–4 subqueries) = at most 40 claims; 3–4 keyword subqueries (Appendix A)
    Retrieval protocol choices that directly shape AskChem-Bench answers; not derived from first principles.
  • Extraction model mix and temperatures = provider-default temperature; JSON-object constrained decoding
    Abstract vs full-text pipelines (GPT-5-mini, Gemini 3.1 Pro, legacy GPT-4o slice) and provider-default temperature control what enters the claim store.
  • Taxonomy stabilization thresholds (synonym norm, fuzzy clustering)
    Corpus-induced L1/L2/L3 paths depend on clustering/normalization choices that can merge or split categories (Limitations).
  • Relevance judge (Gemini 3.1 Pro) calibration = 93% agreement; κ=0.914 on 100 labels
    Paper relevance scores rest on a judge calibrated to 100 expert labels; metric depends on that model+label set.
axioms (5)
  • domain assumption A claim with claim_type, source DOI, and verbatim quote/evidence_locator is sufficiently grounded for retrieval and citation use.
    Stated in §2–3 and RQ1; authors note this is traceability not full semantic correctness.
  • domain assumption LLM structured extraction with schema validation yields usable atomic chemistry claims at scale.
    Core pipeline premise (§3, Extractor details); errors acknowledged in Limitations.
  • domain assumption CrossRef-resolvable DOIs and citation density are valid primary proxies for answer groundedness on cross-paper tasks.
    Evaluation design §7 / Appendix A; does not by itself prove factual accuracy of reported numbers.
  • standard math Reciprocal rank fusion of FTS5, paper recall, taxonomy-node recall, and dense vectors is an appropriate hybrid retriever.
    Uses Cormack et al. 2009 RRF as standard IR component (§4).
  • ad hoc to paper A stratified expert sample of 148 edges adequately indicates evidence-graph relation quality (~97.9% type precision).
    RQ2 / §3 audit; small n relative to 171K edges.
invented entities (5)
  • Provenance-carrying Claim as primary retrieval object (typed fields + DOI + quote/locator) independent evidence
    purpose: Replace paper-ranked lists with verifiable atomic findings for search, browse, and agents.
    Central representation §2; analogous to SAM-style segmentation metaphor in intro.
  • Stabilized faceted taxonomy (corpus-induced then canonicalized L1–L3 views) independent evidence
    purpose: Hierarchical retrieval, browsing, grouping, temporal overlays over the claim store.
    §4; operational index, not claimed as gold ontology.
  • Evidence graph over claims (supports/contradicts/extends/derives_from/cites_as_evidence) independent evidence
    purpose: Cross-paper navigation and conflict/extension surfacing.
    §3; 171K edges with limited expert audit.
  • Living Taxonomy (principle/theory/model-centered exploratory tree) no independent evidence
    purpose: Situate papers under governing scientific ideas beyond operational facets.
    §5; explicitly exploratory, abstention/proposed branches, placement not fully validated.
  • AskChem-Bench (30 cross-paper chemistry questions) independent evidence
    purpose: Measure citation groundedness, density, relevance vs other systems.
    §7 / Appendix A; custom bench released via API.

pith-pipeline@v1.2.0-daily-grok45 · 15294 in / 3826 out tokens · 73441 ms · 2026-07-31T02:00:04.980770+00:00 · methodology

0 comments
read the original abstract

Chemistry literature synthesis often requires assembling specific findings scattered across many publications, yet existing literature-search systems primarily return ranked document lists. As a result, scientists and AI agents need to locate relevant information, verify their provenance, and assemble cross-paper answers manually. We present AskChem, a claim-centered infrastructure for cross-paper chemistry search. AskChem changes the unit of retrieval from the paper to the provenance-carrying claim: each paper is converted into atomic, typed claims, each grounded by a source DOI and a verbatim quote or an explicit evidence locator. Over this shared claim store, AskChem exposes complementary structures for search and synthesis: a stabilized faceted taxonomy for hierarchical retrieval and browsing, an evidence graph linking claims through relations, and an exploratory living taxonomy that situates indexed papers under scientific principles. AskChem currently indexes 2.4M claims from 147K papers and provides a web interface, as well as REST, SDK, and MCP access for AI agents. On AskChem-Bench, grounding a GPT-5.5 reader in AskChem yields 100% resolvable DOIs, compared with 88.3% without retrieval, and the highest citation density among five tested systems. AskChem is live at https://askchem.org.

Figures

Figures reproduced from arXiv: 2607.28618 by Bing Yan, Gregory Wolfe, Kyunghyun Cho, Stefano Martiniani.

Figure 1
Figure 1. Figure 1: The AskChem interface returns provenance￾carrying claims. documents, leaving the scientist to open papers, lo￾cate relevant evidence, verify reported values, and assemble the answer by hand. This document-centered interface is increasingly misaligned with how scientists and AI agents use the literature. LLM agents now survey papers and plan experiments (Boiko et al., 2023), but inherit the limitations of t… view at source ↗
Figure 2
Figure 2. Figure 2: AskChem replaces papers with provenance [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Corpus-scale coverage and automatic quality checks for the deployed index. These statistics characterize [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The same “CO2 reduction” claims across 10 views: reaction, substance, application, technique, mechanism, data, claim type, and time panels expose faceted structure; author view has a coauthor relationship graph that exposes collaborations; network view displays the evidence graph [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The principle-centered living taxonomy answers what governs a contribution. Unlike the stabilized faceted [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Same question (AskChem-Bench ca04, “electrocatalysts for CO2 reduction to CO/formate”), answering alone, GPT-5.5 fabricates 6 of 14 DOIs; when grounded in AskChem’s retrieved claims, all 22 citations resolve. RQ4: Does AskChem operate at corpus scale? The deployed service jointly queries 2.4M claims, 307K taxonomy nodes, and 171K evidence edges through the same REST schema used by the web interface, SDK, a… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 4 linked inside Pith

  1. [1]

    Nature , volume=

    Autonomous chemical research with large language models , author=. Nature , volume=. 2023 , doi=

    2023

  2. [2]

    and Swienty-Busch, J

    Lawson, Alexander J. and Swienty-Busch, J. The Making of. The Future of the History of Chemical Information , series=. 2014 , doi=

    2014

  3. [3]

    , edition=

    Ridley, Damon D. , edition=. Information Retrieval:. 2009 , doi=

    2009

  4. [4]

    arXiv preprint arXiv:1805.02262 , year=

    Construction of the literature graph in semantic scholar , author=. arXiv preprint arXiv:1805.02262 , year=

    Pith/arXiv arXiv

  5. [5]

    Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

    Segment anything , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=. 2023 , doi=

    2023

  6. [6]

    2026 , howpublished =

    Paperclip: The Command-Line Interface for Scientific Literature , author =. 2026 , howpublished =

    2026

  7. [7]

    2025 , howpublished =

    NotebookLM Adds Deep Research and Support for More Source Types , author =. 2025 , howpublished =

    2025

  8. [8]

    arXiv preprint arXiv:2409.13740 , year=

    Language Agents Achieve Superhuman Synthesis of Scientific Knowledge , author=. arXiv preprint arXiv:2409.13740 , year=

    Pith/arXiv arXiv

  9. [9]

    2025 , howpublished =

    Announcing Edison Scientific , author =. 2025 , howpublished =

    2025

  10. [10]

    Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

    Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction , author=. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=. 2018 , doi=

    2018

  11. [11]

    Retrieval-Augmented Generation for Knowledge-Intensive

    Lewis, Patrick and Perez, Ethan and Piktus, Aleksandra and Petroni, Fabio and Karpukhin, Vladimir and Goyal, Naman and K. Retrieval-Augmented Generation for Knowledge-Intensive. Advances in Neural Information Processing Systems (NeurIPS) , volume=. 2020 , url=

    2020

  12. [12]

    2019 , doi=

    Reimers, Nils and Gurevych, Iryna , booktitle=. 2019 , doi=

    2019

  13. [13]

    and Clarke, Charles L

    Cormack, Gordon V. and Clarke, Charles L. A. and Buettcher, Stefan , booktitle=. Reciprocal Rank Fusion Outperforms. 2009 , doi=

    2009

  14. [14]

    Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL , pages=

    Semantic Taxonomy Induction from Heterogenous Evidence , author=. Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL , pages=. 2006 , doi=

    2006

  15. [15]

    Proceedings of the 33rd ACM International Conference on Information and Knowledge Management (CIKM) , pages=

    Chain-of-Layer: Iteratively Prompting Large Language Models for Taxonomy Induction from Limited Examples , author=. Proceedings of the 33rd ACM International Conference on Information and Knowledge Management (CIKM) , pages=. 2024 , doi=. 2402.07386 , archivePrefix=

    Pith/arXiv arXiv 2024

  16. [16]

    Findings of the Association for Computational Linguistics: EACL 2024 , pages=

    Do Language Models Know When They're Hallucinating References? , author=. Findings of the Association for Computational Linguistics: EACL 2024 , pages=. 2024 , doi=

    2024

  17. [17]

    2024 , howpublished =

    Model Context Protocol , author =. 2024 , howpublished =

    2024

  18. [18]

    2023 , doi=

    Kim, Sunghwan and Chen, Jie and Cheng, Tiejun and Gindulyte, Asta and He, Jia and He, Siqian and Li, Qingliang and Shoemaker, Benjamin A and Thiessen, Paul A and Yu, Bo and others , journal=. 2023 , doi=

    2023

  19. [19]

    2012 , doi=

    Gaulton, Anna and Bellis, Louisa J and Bento, A Patricia and Chambers, Jon and Davies, Mark and Hersey, Anne and Light, Yvonne and McGlinchey, Shaun and Michalovich, David and Al-Lazikani, Bissan and Overington, John P , journal=. 2012 , doi=

    2012

  20. [20]

    2205.01833 , archivePrefix=

    Priem, Jason and Piwowar, Heather and Orr, Richard , year=. 2205.01833 , archivePrefix=

    Pith/arXiv arXiv

  21. [21]

    Journal of Informetrics , volume=

    Google Scholar, Web of Science, and Scopus: A Systematic Comparison of Citations in 252 Subject Categories , author=. Journal of Informetrics , volume=. 2018 , doi=

    2018