pith. sign in

arxiv: 2606.21658 · v1 · pith:3KFYDBWEnew · submitted 2026-06-19 · ✦ hep-ex · astro-ph.IM· hep-ph

Towards LLM-Powered Automation of a Dark Matter Constraint Repository

Lanqing Yuan , Karthik Ramanathan This is my paper

Pith reviewed 2026-06-26 12:19 UTC · model grok-4.3

classification ✦ hep-ex astro-ph.IMhep-ph
keywords dark matterLLM pipelineconstraint repositorylimit extractionarXiv monitoringcoupling classificationphysics automation
0
0 comments X

The pith

An LLM pipeline extracts dark matter coupling limits from papers with 90.5% classification accuracy and 0.33 dex median residual.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper builds an LLM pipeline that scans arXiv for new dark matter results, pulls out limit curves, converts them into code, and submits proposals for human review in an existing constraint repository. On a benchmark of 346 papers where the repository itself supplies the ground truth, the system correctly identifies coupling types 90.5 percent of the time and produces a median residual of 0.33 dex while covering 76 percent of the mass range on average. The method reconciles noisy individual extractions through consensus voting, applies a layer that standardizes physics conventions, and uses a scoring approach that distinguishes real extraction mistakes from cases where papers follow incompatible conventions. This setup targets the growing workload on volunteer maintainers as the rate of new results rises. The pipeline has already generated proposals, though none have been merged, and the work notes that governance of AI-produced scientific data remains open.

Core claim

The pipeline classifies the coupling type correctly for 90.5% of papers and reaches a median coupling residual of 0.33 dex, with 76% mean mass-range coverage, by treating each extraction as a noisy sample reconciled through consensus voting, a physics convention canonicalization layer, and a scoring methodology that separates genuine extraction error from non-comparability.

What carries the argument

Consensus voting across multiple LLM extractions combined with a canonicalization layer for physics conventions and a scoring method that isolates extraction error from non-comparable cases.

If this is right

  • Repository maintainers shift from manual extraction to reviewing AI-generated limit proposals.
  • The system scales extraction effort with the rising volume of dark matter papers without proportional increase in volunteer time.
  • Performance remains lower on rare coupling types that use idiosyncratic conventions, with macro-averaged residual of 1.1 dex.
  • Governance questions for AI-generated scientific data must be resolved before proposals can be merged.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same consensus-plus-canonicalization pattern could reduce manual curation in other experimental constraint collections such as neutrino or collider limits.
  • Deployment success ultimately depends on whether human reviewers accept the proposed changes at rates high enough to offset the initial setup cost.
  • Extending the scoring method to flag papers with non-standard conventions could further improve macro-averaged accuracy.

Load-bearing premise

The existing volunteer-curated repository supplies reliable ground truth that can be used to measure how accurately the pipeline recovers limits.

What would settle it

Independent expert re-extraction of limits from a fresh set of 50 papers yields a median residual above 0.8 dex or coupling classification accuracy below 80 percent.

Figures

Figures reproduced from arXiv: 2606.21658 by Karthik Ramanathan, Lanqing Yuan.

Figure 1
Figure 1. Figure 1: Pipeline architecture. Papers flow from arXiv through keyword and LLM filtering, two-stage [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Highlighted dark photon constraint projection plot from a production pull request for a [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
read the original abstract

Dark matter constraint repositories are critical community infrastructure, giving experimentalists and theorists a shared landscape of existing bounds. Yet the most widely-used repositories are maintained by individual volunteers, creating a sustainability risk as the pace of new results accelerates. We present a large language model (LLM) pipeline that monitors arXiv, extracts limit curves from papers, integrates them as code, and opens pull requests (PRs) for human review. On a 346-paper benchmark whose ground truth is the upstream-curated repository itself, the pipeline classifies the coupling type correctly for 90.5% of papers and reaches a median coupling residual of 0.33 dex (a factor of two for 48% of curves), with 76% mean mass-range coverage. This is driven by treating each extraction as a noisy sample reconciled through consensus voting, a physics convention canonicalization layer built with the agentic physics assistant Get Physics Done (GPD), and a scoring methodology that separates genuine extraction error from non-comparability. The remaining difficulty is concentrated in rare coupling types with idiosyncratic conventions (macro-averaged residual 1.1 dex). The pipeline is deployed and has generated limit proposals; none have merged. Governance of AI-generated scientific data is itself an unsolved problem.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The paper presents an LLM pipeline for automating updates to a dark matter constraint repository by monitoring arXiv, extracting limit curves from papers, canonicalizing physics conventions via an agentic assistant (GPD), reconciling via consensus voting, and opening PRs for human review. On a 346-paper benchmark whose ground truth is the upstream-curated repository itself, the pipeline classifies coupling type correctly for 90.5% of papers, reaches a median coupling residual of 0.33 dex, and achieves 76% mean mass-range coverage. Remaining errors concentrate in rare couplings (macro-averaged residual 1.1 dex). The pipeline is deployed and has generated proposals, though none have merged; governance of AI-generated data is noted as unsolved.

Significance. If the evaluation methodology is strengthened with independent checks, the work could meaningfully address sustainability risks for volunteer-maintained dark matter repositories amid rising publication rates. Concrete benchmark metrics on 346 papers, the consensus-voting approach, and the deployed status provide practical evidence of LLM utility for scientific data curation. Credit for reproducible deployment and explicit discussion of governance challenges.

major comments (1)
  1. [Benchmark evaluation] Benchmark evaluation (abstract and associated section): The headline metrics (90.5% coupling-type accuracy, 0.33 dex median residual, 76% mass coverage) are computed by treating the target upstream-curated repository as ground truth. Because the pipeline is explicitly designed to reproduce and extend that same repository, these figures risk measuring reproduction of existing conventions rather than independent extraction fidelity. The abstract notes a scoring layer to separate genuine error from non-comparability and that errors concentrate in rare couplings, but without an external validation set (e.g., blinded expert re-annotation on a held-out subset) or cross-check against raw numerical data in the source papers, it remains unclear whether the reported residuals and coverage figures are load-bearing evidence of pipeline accuracy.
minor comments (2)
  1. [Abstract] The abstract states that 'a scoring methodology that separates genuine extraction error from non-comparability' is used, yet provides no explicit criteria, thresholds, or examples of how non-comparable cases (especially idiosyncratic rare couplings) are flagged; adding a short methods paragraph or table would improve interpretability of the macro-averaged 1.1 dex residual.
  2. [Deployment and results] The deployment paragraph notes that proposals have been generated but 'none have merged.' A brief discussion of review outcomes or barriers would help readers assess the practical readiness of the human-in-the-loop step.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their constructive review and for identifying a key methodological consideration. We respond to the single major comment below.

read point-by-point responses

  1. Referee: Benchmark evaluation (abstract and associated section): The headline metrics (90.5% coupling-type accuracy, 0.33 dex median residual, 76% mass coverage) are computed by treating the target upstream-curated repository as ground truth. Because the pipeline is explicitly designed to reproduce and extend that same repository, these figures risk measuring reproduction of existing conventions rather than independent extraction fidelity. The abstract notes a scoring layer to separate genuine error from non-comparability and that errors concentrate in rare couplings, but without an external validation set (e.g., blinded expert re-annotation on a held-out subset) or cross-check against raw numerical data in the source papers, it remains unclear whether the reported residuals and coverage figures are load-bearing evidence of pipeline accuracy.

    Authors: We agree that the benchmark measures the pipeline's fidelity to the conventions and extractions already present in the upstream repository rather than fully independent extraction from raw paper content. This design choice follows directly from the stated goal of automating maintenance and extension of that specific community resource; the metrics therefore quantify how reliably the system can reproduce expert-curated results at the scale of hundreds of papers. The scoring layer and macro-averaged analysis of rare couplings are intended to qualify the headline numbers and flag non-comparable cases. We acknowledge that an external validation set (e.g., blinded expert re-annotation) would provide stronger evidence of independent accuracy. In the revised manuscript we have added an explicit limitations paragraph clarifying this point, reiterating that the current figures demonstrate practical reproduction performance, and noting that future work should include independent checks against raw numerical data or expert re-annotation. No changes were made to the reported metrics themselves. revision: partial

Circularity Check

0 steps flagged

No circularity; benchmark is standard reproduction test

full rationale

The paper reports empirical performance of an LLM extraction pipeline on a 346-paper benchmark that uses the upstream repository as ground truth. No mathematical derivations, equations, fitted parameters, or self-citations appear in the provided text. The evaluation measures reproduction fidelity for an automation task, which is a direct and non-circular test of the stated goal. The methodology (consensus voting, canonicalization) is described as separating error from non-comparability without reducing any result to its own inputs by construction. This matches the default case of a self-contained empirical methods paper.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The abstract does not introduce or rely on any explicit free parameters, mathematical axioms, or newly invented physical entities. The approach depends on existing LLM capabilities and the external GPD assistant tool.

pith-pipeline@v0.9.1-grok · 5751 in / 1377 out tokens · 34833 ms · 2026-06-26T12:19:15.289160+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

13 extracted references · 4 canonical work pages

  1. [1]

    Axion dark matter,

    C. B. Adams et al., “Axion dark matter,” inSnowmass 2021, arXiv:2203.14923, 2022

    work page arXiv 2021

  2. [2]

    New experimental approaches in the search for axion-like particles,

    I. G. Irastorza and J. Redondo, “New experimental approaches in the search for axion-like particles,”Prog. Part. Nucl. Phys.102, 89, 2018

    2018

  3. [3]

    Axions and the strong CP problem,

    J. E. Kim and G. Carosi, “Axions and the strong CP problem,”Rev. Mod. Phys.82, 557, 2010

    2010

  4. [4]

    Dark photon limits: A handbook,

    A. Caputo, A. J. Millar, C. A. J. O’Hare, and E. Vitagliano, “Dark photon limits: a handbook,” Phys. Rev. D104(9), 095029, 2021, arXiv:2105.04565 [hep-ph]

    work page arXiv 2021

  5. [5]

    AxionLimits,

    C. A. J. O’Hare, “AxionLimits,”https://github.com/cajohare/AxionLimits, 2020

    2020

  6. [6]

    Cosmology of axion dark matter,

    C. A. J. O’Hare, “Cosmology of axion dark matter,”PoSCOSMICWISPers, 040, 2024

    2024

  7. [7]

    INSPIRE-HEP: High-Energy Physics Literature Database,

    INSPIRE Collaboration, “INSPIRE-HEP: High-Energy Physics Literature Database,”https: //inspirehep.net, 2024

    2024

  8. [8]

    The Claude model family,

    Anthropic, “The Claude model family,”https://www.anthropic.com, 2024

    2024

  9. [9]

    Get Physics Done (GPD),

    Physical Superintelligence PBC, “Get Physics Done (GPD),” v1.2.2, 2026.https://github. com/psi-oss/get-physics-done

    2026

  10. [10]

    PyMuPDF,

    Artifex Software, “PyMuPDF,”https://github.com/pymupdf/PyMuPDF, 2015–2025

    2015

  11. [11]

    Direct Detection of Dark Photon Dark Matter with the James Webb Space Telescope,

    H. An, S. Ge, J. Liu, and Z. Lu, “Direct Detection of Dark Photon Dark Matter with the James Webb Space Telescope,” arXiv:2402.17140, 2024

    work page arXiv 2024

  12. [12]

    Dark Matter Detection Using Phonon Sensing in Amorphous Materials,

    I. M. Bloch, S. Knapen, X. Li, A. Madden, and G. Marocco, “Dark Matter Detection Using Phonon Sensing in Amorphous Materials,” arXiv:2603.22390, 2026

    work page arXiv 2026

  13. [13]

    Review of particle physics,

    S. Navas et al. (Particle Data Group), “Review of particle physics,”Phys. Rev. D110(3), 030001, 2024. 5

    2024