Pith. sign in

REVIEW 4 major objections 6 minor 13 references

Towards Large Language Models for Lunar Mission Planning and In Situ Resource Utilization

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A general-purpose LLM, given the full text of a lunar sample paper, extracts table compositions with under 5% midpoint error for most ground-truthed samples, and does far better than querying the model from memory alone.

desk verdict A small, honest feasibility study that is worth refereeing, but the abstract's 'generally effective' outruns the ten-sample ground truth it stands on. read the letter →

arxiv 2504.20125 v1 pith:YX63U543 submitted 2025-04-28 cs.DL cs.AI

Michael Pekala , Gregory Canal , Samuel Barham , Milena B. Graziano , Morgan Trexler , Leslie Hamilton , Elizabeth Reilly , Christopher D. Stiles This is my paper · ORCID
classification cs.DLcs.AI
keywords lunarsamplecompendiumlargelanguagemodelsdataextractioncompositiontablesinsituresourceutilizationmissionplanninginterval-valuedscientificdocumentmining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a general-purpose large language model, given the full text of a lunar-science paper, can reliably turn the composition tables in that paper into structured, interval-valued data suitable for mission planning and in situ resource utilization. Its central finding is that the model, when supplied with the paper, produces midpoint estimates within 5% relative error for the majority of the ten ground-truthed samples, and performs clearly better than querying the model from memory alone. The authors aim to show that an automated preprocessing step can build a structured lunar-composition database from the roughly 700-document Lunar Sample Compendium, where values are represented as ranges to reflect real sample heterogeneity. This matters because lunar mission planners need local resource estimates, and most relevant measurements are scattered across heterogeneous publications rather than consolidated in existing datasets.

What carries the argument

The load-bearing mechanism is a two-step preprocessing pipeline: extract raw text from each PDF with a conventional library, then prompt an off-the-shelf LLM to output a CSV table giving, for each element or compound, the sample id and the observed weight range as an interval. The prompt explicitly instructs the model to aggregate multiple measurements into a min-max interval and to report units, and the collated intervals are compared with ground truth using midpoint difference, relative midpoint error, and interval precision and recall. The interval representation is what lets the pipeline absorb the paper's central complication—lunar samples are not homogeneous and multiple studies report different values—without pretending the data are point measurements.

What would settle it

Take a fresh set of LSC documents not used in the reported ground truth—say 50 samples spanning all six Apollo missions—have two domain experts independently annotate composition intervals with a written rule for blank cells and implied units, then run the same prompt and measure relative midpoint error. The central claim would be undercut if fewer than half of the with-document estimates fall under 5% relative error, or if the with-document condition is not systematically better than the standalone baseline on the same items.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that off-the-shelf LLM extraction, rather than general retrieval-augmented question answering, is a workable first-pass mechanism for mining composition tables from lunar sample literature. For the ten samples manually ground-truthed from the Lunar Sample Compendium, the LLM given the document achieves less than 5% relative error on the midpoint of the extracted interval for the majority of non-trace composition points, whereas the same model queried without the document shows systematically larger errors and little sensitivity to sample identity. The extracted data are represented as intervals because lunar samples are polymineralic and analyzed by multiple groups, and the paper reports precision and recall on those intervals alongside midpoint error. The paper also reports qualitative full-corpus results: the most frequent oxides extracted across 728 documents match expectations, though fine-grained mineralogy and some trace-unit entries remain unreliable.

Load-bearing premise

The load-bearing assumption is that the ten manually annotated samples, taken from the same Lunar Sample Compendium documents the model is asked to read, are correctly and representatively annotated, so that the reported error rates on those samples stand in for performance on the full 728-document corpus.

Editorial extensions

If this is right

  • The preprocessing pipeline can be run over all 728 downloaded LSC documents to produce a single structured composition table, since the prompt asks for the same CSV format regardless of document.
  • The large gap between the with-document and standalone conditions shows that the extracted values are being read from the supplied text, not recalled from the model's training data.
  • Representing each value as an interval preserves the spread across research groups and mineral phases, giving downstream mission-planning tools an explicit uncertainty band.
  • The paper's identified weak spots—mineralogy-specific breakdowns, trace elements in ppm and ppb, and blank or implied-unit entries—are concrete targets for prompt refinement and richer ground truth.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the method generalizes beyond the ten ground-truthed samples, the same prompt-based pipeline could be pointed at other heterogeneous sample compendia, such as Martian meteorites or returned asteroidal material, where the interval representation would absorb similar inter-laboratory spread.
  • Beyond the paper: the reported 5% midpoint error could understate or overstate task quality depending on use; mission-relevant tolerances may be tighter or looser than 5%, so a thresholded cost metric tied to regolith simulant or synthesis requirements would be a more decision-relevant evaluation than generic interval precision.
  • Beyond the paper: a natural testable extension is to combine this preprocessing database with retrieval-augmented querying, using the extracted CSV as a tool the model calls at plan time, and to compare that against a pure chunked-retrieval baseline on the same corpus.
  • Beyond the paper: because the ground truth is interval-valued and derived from the same documents the LLM reads, part of the measured error may reflect annotation choices about which rows count, how units are interpreted, and how mineral phases are handled; an independent expert re-annotation with disagreement tracking would separate annotation noise from model failure.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes an LLM-based preprocessing pipeline for extracting chemical composition data from a corpus of Lunar Sample Compendium (LSC) documents, representing compositions as intervals (min-max) rather than point values. The authors evaluate ChatGPT4o on ten manually ground-truthed samples (nine documents), comparing a 'with document' condition (full paper text in the context window) against a 'standalone' baseline (no document provided). They report that the with-document LLM achieves <5% relative midpoint error for the majority of non-trace compositions and outperforms the standalone baseline, and conclude that off-the-shelf LLMs are generally effective for extracting tabular composition data. The paper also provides qualitative interval comparisons, precision/recall metrics, and an appendix with full-corpus analyses.

Significance. If the central claim holds, the paper would demonstrate a practical, low-cost method for converting heterogeneous lunar sample literature into structured, interval-valued composition data, which could support mission planning, simulant development, and downstream modeling. The interval-based evaluation metrics are thoughtfully defined, and the comparison against a standalone baseline is a useful control for assessing whether the LLM is actually using the provided document content. However, the significance is strongly limited by the scale and rigor of the evaluation: the evidence base is only ten samples, the ground-truthing procedure lacks a documented sampling rule and inter-annotator reliability, and the headline quantitative claims omit explicit restrictions and statistical uncertainty. The paper is better framed as a pilot feasibility study than as a validated pipeline.

major comments (4)
  1. [Section 3.1 / Section 4.2] The central claim that the LLM is 'generally effective' across the 728-document corpus rests on ground truth from only ten samples, with no stated sampling rule beyond 'at least one sample from each Apollo mission.' The manuscript provides no inter-annotator reliability measure and no error analysis of the manual transcription. Because the LSC documents are highly heterogeneous (as Figures 2, 3, and 5 illustrate), a non-random or convenience sample of ten well-characterized samples cannot support the unqualified generalization in the abstract. Please specify the exact sample selection procedure, justify its representativeness, or substantially temper the corpus-level claim.
  2. [Section 4.2 / Abstract] The abstract states that the LLM 'achieves less than 5% relative error for the majority of the points we ground truthed,' but the analysis in Section 4.2 is restricted to 'non-trace compositions' and to 'inliers,' with no formal definition of 'inlier.' The abstract also omits the restriction to non-trace compositions. Moreover, the exact proportion of ground-truthed points that meet the <5% threshold is never reported, and the outliers visible in Figure 7 are not quantified. The claim should be restated with the precise denominator, the inclusion/exclusion criteria, and the fraction of points within the threshold (including outliers).
  3. [Section 3.1 / Figure 3] The manuscript explicitly notes for sample 14321 that 'our current ground truthing and LLM prompting strategy does not attempt to disambiguate among the various phases,' but it does not state whether the ground truth interval for that sample includes only whole-rock values or also phase-specific columns. This choice materially changes the ground truth intervals and therefore the computed errors. The ground-truthing protocol should specify how multi-phase tables were handled, and affected samples should either be excluded or analyzed separately.
  4. [Section 4 / Results] The text claims that the with-document LLM performs 'significantly better' than the standalone baseline, but no statistical test, confidence interval, or effect-size measure is provided. With only ten samples and per-sample composition pairs, the comparison could be assessed with a paired test (e.g., Wilcoxon signed-rank) over the matched (sample, composition) errors. Please add an appropriate statistical analysis or remove the word 'significantly.'
minor comments (6)
  1. [Figure 3 caption] Typo: 'disambigute' should be 'disambiguate.'
  2. [Section 4] Typo: 'less that 5%' should be 'less than 5%.'
  3. [Section 3.2] Duplicate word: 'the the' should be 'the.'
  4. [Figure 4 caption] Typo: 'as a an hourglass' should be 'as an hourglass.'
  5. [Appendix A.1] The statement that 'the most frequent oxides appear to be consistent with what is expected' is not supported by a quantitative comparison; consider adding a reference distribution or at least a qualitative description of the expected composition ranges.
  6. [General] No link or repository is provided for the extracted CSV data, the prompt, or the evaluation scripts. Making these publicly available would substantially aid reproducibility and allow other researchers to build on the pipeline.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the extraction claim is benchmarked against independent manual ground truth, not against the LLM's own outputs or fitted parameters.

full rationale

This paper is an empirical evaluation of LLM-based table extraction, not a derivation. The central quantitative claim—'the LLM, when provided with the paper, achieves less than 5% relative error for the majority of the points we ground truthed' (Section 4)—is supported by comparing LLM outputs with manually annotated ground truth for ten samples (Section 3.1). The authors state that they 'downloaded 728 pdf documents from the LSC website [8] and manually annotated chemical composition information for ten samples to use for ground truth.' No LLM output is used to construct the ground truth, no model parameter is fitted to the ground truth, and no equation in the paper defines the evaluation metric in terms of the LLM's output. The interval representation (min–max over reported values) is applied symmetrically to both ground truth and estimates, but this is the task definition rather than a circular reduction. The paper's only flagged confound is that the LSC documents are public and could be in LLM training data (footnote 4); the authors address this with a standalone baseline, and this is a measurement confound, not circular reasoning. There are no load-bearing self-citations, no imported uniqueness theorems, and no fitted parameter renamed as a prediction. The evaluation is self-contained and the claim does not reduce to its inputs.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim does not introduce new physical entities. Free parameters are limited to two hand-chosen processing thresholds that affect secondary corpus-level analyses, and the key assumptions are about ground-truth quality and text extraction coverage.

free parameters (2)
  • document chunk size = ~25,000 characters
    Long documents are split into about 25,000-character pieces respecting page boundaries (Section 3.2). This hand-chosen threshold can influence extraction quality and is not optimized or varied.
  • minimum frequency cutoff = 5 occurrences
    In Appendix A.1, compositions appearing five or fewer times across the corpus are discarded as presumed extraction errors; the cutoff is arbitrary and the discarded entries are not verified.
assumptions (4)
  • domain assumption The authors' manual ground truth for ten samples is accurate and reflects the complete composition tables for those samples.
    All evaluation metrics compare LLM outputs to this ground truth, but the annotation procedure and inter-annotator agreement are not described (Section 3.1).
  • domain assumption Text extracted by PyMuPDF from the PDFs contains all relevant table data; data present only in figures or images is not needed for the evaluation.
    The pipeline does not process images or figures (Section 3), so any composition values only in figures are invisible to both the LLM and the current ground truth.
  • domain assumption ChatGPT-4o with temperature 0 is a representative 'off-the-shelf LLM' for the conclusions.
    Only one model and setting are tested, while the abstract generalizes to off-the-shelf LLMs broadly (Sections 3.2 and 4).
  • domain assumption The ten selected samples are representative of the Apollo corpus used in the study.
    The samples cover each Apollo mission but the selection rule is not specified, so generalization to the full corpus is an assumption (Section 3.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Large Language Models for Lunar Mission Planning and In Situ Resource Utilization." pith.science (2026). https://pith.science/paper/YX63U543

@misc{pith2026250420125,
  author       = {Pith},
  title        = {Pith review of: Towards Large Language Models for Lunar Mission Planning and In Situ Resource Utilization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YX63U543}},
  note         = {Machine review of arXiv:2504.20125}
}
read the original abstract

A key factor for lunar mission planning is the ability to assess the local availability of raw materials. However, many potentially relevant measurements are scattered across a variety of scientific publications. In this paper we consider the viability of obtaining lunar composition data by leveraging LLMs to rapidly process a corpus of scientific publications. While leveraging LLMs to obtain knowledge from scientific documents is not new, this particular application presents interesting challenges due to the heterogeneity of lunar samples and the nuances involved in their characterization. Accuracy and uncertainty quantification are particularly crucial since many materials properties can be sensitive to small variations in composition. Our findings indicate that off-the-shelf LLMs are generally effective at extracting data from tables commonly found in these documents. However, there remains opportunity to further refine the data we extract in this initial approach; in particular, to capture fine-grained mineralogy information and to improve performance on more subtle/complex pieces of information.

Figures

Figures reproduced from arXiv: 2504.20125 by the authors.

Figure 1
Figure 1. Envisioned tool-enabled LLM for mission planning. In this paper, we focus on the preprocessing [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Example of chemical composition data from LSC document [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Excerpt from the LSC document for sample 14321. Here, the columns denoting different studies [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (10 more)
Figure 7
Figure 7. Figure 7: fig. 7. When provided with the paper content, the relative error tends to be below 5%, although there are a [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 4
Figure 4. Figure 4: Weight percentages for four relatively abundant oxides. Blue intervals denote the manually extracted [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Excerpt from LSC document 71595.pdf. Note that SiO2 quantities are not explicitly reported for these samples (and, hence, not included in ground truth; see also fig. 4). largely encouraging, though several potential challenges stem from the complexity of the underlying…
Figure 6
Figure 6. Figure 6: LLM data extraction performance for non-trace compositions. The left panel shows performance [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: LLM data extraction performance measured in percent relative error, i.e., eq. (4). As in fig. 6, [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Precision and recall performance (when provided with LSC papers) for most abundant compositions. [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Estimated compositions vs. ground truth for sample 15415, where the LLM “with document” [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: Compounds / elements (y-axis) and number of times an associated interval was extracted by the [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: Example weight percentage intervals extracted from the LSC corpus considered in this study. [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: Recall for samples that were ground truthed. Cells shaded in blue denote composition/sample id [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 7 canonical work pages

  1. [1]

    From local to global: A graph RAG approach to query-focused summarization.arXiv preprint arXiv:2404.16130, 2024

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. From local to global: A graph RAG approach to query-focused summarization.arXiv preprint arXiv:2404.16130, 2024

    arXiv 2024

  2. [2]

    A survey on RAG meeting LLMs: Towards retrieval-augmented large language models

    Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A survey on RAG meeting LLMs: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 6491–6501, 2024

    work page 2024

  3. [3]

    Retrieval-augmented generation for large language models: A survey.arXiv preprint arXiv:2312.10997, 2023

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-augmented generation for large language models: A survey.arXiv preprint arXiv:2312.10997, 2023

    arXiv 2023

  4. [4]

    Robust planning with LLM-Modulo framework: Case study in travel planning.arXiv preprint arXiv:2405.20625, 2024

    Atharva Gundawar, Mudit Verma, Lin Guan, Karthik Valmeekam, Siddhant Bhambri, and Subbarao Kambhampati. Robust planning with LLM-Modulo framework: Case study in travel planning.arXiv preprint arXiv:2405.20625, 2024

    arXiv 2024

  5. [5]

    DSPy: Compiling declarative language model calls into self-improving pipelines.arXiv preprint arXiv:2310.03714 , 2023

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vard- hamanan, Saiful Haq, Ashutosh Sharma, Thomas T Joshi, Hanna Moazam, et al. DSPy: Compiling declarative language model calls into self-improving pipelines.arXiv preprint arXiv:2310.03714 , 2023

    arXiv

  6. [6]

    Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics, 12:157–173, 2024

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics, 12:157–173, 2024

    2024

  7. [7]

    Large language model for table processing: A survey.Frontiers of Computer Science , 19(2):192350, 2025

    Weizheng Lu, Jing Zhang, Ju Fan, Zihao Fu, Yueguo Chen, and Xiaoyong Du. Large language model for table processing: A survey.Frontiers of Computer Science , 19(2):192350, 2025

    work page 2025

  8. [8]

    The lunar sample compendium.https://curator.jsc.nasa.gov/lunar/lsc/

    Charles Meyer. The lunar sample compendium.https://curator.jsc.nasa.gov/lunar/lsc/. Accessed: 2025-02-20

    work page 2025

Show all 13 references
  1. [9]

    Lunar sample compendium, 2005

    Charles Meyer. Lunar sample compendium, 2005

    2005

  2. [10]

    Flexible, model-agnostic method for materials data extraction from text using general purpose language models.Digital Discovery, 3(6):1221–1235, 2024

    Maciej P Polak, Shrey Modi, Anna Latosinska, Jinming Zhang, Ching-Wen Wang, Shaonan Wang, Ayan Deep Hazra, and Dane Morgan. Flexible, model-agnostic method for materials data extraction from text using general purpose language models.Digital Discovery, 3(6):1221–1235, 2024

    2024

  3. [11]

    Extracting accurate materials data from research papers with conversational language models and prompt engineering.Nature Communications, 15(1):1569, 2024

    Maciej P Polak and Dane Morgan. Extracting accurate materials data from research papers with conversational language models and prompt engineering.Nature Communications, 15(1):1569, 2024

    2024

  4. [12]

    From text to insight: large language models for materials science data extraction.arXiv preprint arXiv:2407.16867 , 2024

    Mara Schilling-Wilhelmi, Martiño Ríos-García, Sherjeel Shabih, María Victoria Gil, Santiago Miret, Christoph T Koch, José A Márquez, and Kevin Maik Jablonka. From text to insight: large language models for materials science data extraction.arXiv preprint arXiv:2407.16867 , 2024

    2024 arXiv

  5. [13]

    composition

    PyMuPDF Team. Pymupdf documentation.https://pymupdf.readthedocs.io/en/latest/. Accessed: 2025-03-16. 12 A Additional results A.1 Full Corpus Analysis In this section we present some qualitative analyses describing what the LLM was able to extract from the 700+ document corpus ...

    2025

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.