Geo-Expert: Towards Expert-Level Geological Reasoning via Parameter-Efficient Fine-Tuning

Chenyou Guo; Yizhou Zhang; Ze Liu; Zhaorui Jiang; Zongqi Liu

arxiv: 2605.24844 · v2 · pith:D3JSPPYO · submitted 2026-05-24 · cs.AI · cs.CL

Geo-Expert: Towards Expert-Level Geological Reasoning via Parameter-Efficient Fine-Tuning

Chenyou Guo , Zongqi Liu , Yizhou Zhang , Zhaorui Jiang , Ze Liu This is my paper

Reviewed by Pith2026-06-30 11:43 UTCgrok-4.3pith:D3JSPPYO open to challenge →

classification cs.AI cs.CL

keywords geological reasoningparameter-efficient fine-tuningLoRAlarge language modelsdomain adaptationGeo-Eval benchmarkearth sciencesinstruction synthesis

0 comments

The pith

Fine-tuning an 8B model on geological instructions lets it outperform 70B generalists and GPT-4o on expert reasoning.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that parameter-efficient fine-tuning of relatively small LLMs on a custom geological instruction dataset can produce models that reason at expert level about subsurface structures and deep-time evolution. This would matter because general-purpose LLMs frequently hallucinate on these topics while existing Earth-science AI focuses mainly on surface sensing. By applying LoRA to Qwen3-8B, Qwen3-32B and Gemma-3-27B bases and testing on their new Geo-Eval benchmark, the authors show the smallest variant beating much larger general models and the mid-size variant approaching frontier performance.

Core claim

Geo-Expert models are created by applying Low-Rank Adaptation to base models on a high-quality, custom-curated geological instruction dataset generated through the authors' synthesis pipeline. On the novel Geo-Eval benchmark the resulting 8B model surpasses open-weight 70B generalist LLMs and proprietary GPT-4o at specialized geological reasoning, while the 32B variant approaches the performance of frontier reasoning models. The work therefore supplies both a concrete performance result and a reproducible recipe for building domain-aligned scientific LLMs.

What carries the argument

Low-Rank Adaptation (LoRA) fine-tuning on a custom-curated geological instruction dataset, measured against the authors' new Geo-Eval benchmark.

If this is right

Domain-specific smaller models can deliver higher accuracy than larger general models on narrow scientific reasoning tasks.
The resulting 8B model supplies a competitive cost-performance option for practical geological applications.
The same fine-tuning recipe can be repeated to create expert LLMs in other scientific disciplines.
Scaling within the domain-aligned family (8B to 32B) yields further gains that approach frontier capability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Data curation and domain alignment appear more decisive than raw parameter count for this class of reasoning problems.
Similar methods could be tested on other Earth-science sub-domains such as paleontology or mineral exploration.
Widespread adoption would lower the barrier for geologists to use reliable AI assistance without relying on the largest proprietary systems.

Load-bearing premise

The custom-curated instruction dataset processed with the authors' synthesis pipeline and the novel Geo-Eval benchmark provide an unbiased and comprehensive measure of expert-level geological reasoning that generalizes beyond the training distribution.

What would settle it

Evaluating the released 8B Geo-Expert model on a fresh collection of geological reasoning questions drawn from recent field reports or textbooks not used in the instruction synthesis pipeline, and finding that it no longer outperforms GPT-4o or 70B generalists.

Figures

Figures reproduced from arXiv: 2605.24844 by Chenyou Guo, Yizhou Zhang, Ze Liu, Zhaorui Jiang, Zongqi Liu.

**Figure 1.** Figure 1: Overview of the Geo-Expert framework. The pipeline consists of three main stages: (1) extracting and sanitizing text from canonical geology textbooks; (2) synthesizing high-quality, CoT-enhanced instruction pairs via a domain-structured generation pipeline; (3) applying parameter-efficient fine-tuning on base LLMs. linear layers (all-linear) for Low-Rank Adaptation (LoRA) (Hu et al., 2021). For the compact… view at source ↗

read the original abstract

While general-purpose Large Language Models (LLMs) applied to Geology often hallucinate when reasoning about subsurface structures and deep-time evolution, current AI in Earth sciences predominantly targets surface remote sensing and GIS. To bridge this gap, we introduce Geo-Expert, a family of parameter-efficient geological LLMs fine-tuned on a custom-curated, high-quality instruction dataset processed using our custom instruction synthesis pipeline. We investigate the impact of model scaling and architecture by fine-tuning three base models: Qwen3-8B, Qwen3-32B, and Gemma-3-27B, with Low-Rank Adaptation (LoRA) method. Our extensive evaluation on a novel domain-specific benchmark, Geo-Eval, reveals that a domain-aligned 8B model can outperform open-weight 70B generalists and proprietary GPT-4o on specialized geological reasoning, while a 32B variant approaches frontier reasoning models. The optimized 8B model further offers a competitive cost-performance ratio for deployment. This work provides a reproducible recipe for democratizing scientific LLMs and establishes a baseline for geological artificial intelligence.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

Standard LoRA fine-tuning on a custom geology dataset produces an 8B model that beats larger generalists on the new Geo-Eval benchmark, but the paper supplies almost no information on how that benchmark was built or validated.

read the letter

The main point is that the authors take Qwen3-8B and 32B, apply ordinary LoRA, train on their own instruction data for geology, and report that the 8B version beats 70B open models plus GPT-4o on Geo-Eval while the 32B gets close to frontier performance. They also note the small model is cheaper to run. That is the result they want people to take away.

They have done the straightforward work of curating a domain dataset and running the fine-tunes on three base models. That produces a usable recipe for anyone who wants to adapt an LLM to subsurface geology questions instead of remote sensing. The cost-performance note for the 8B model is the part that could actually matter for labs that cannot afford big inference.

The soft spot is the evaluation. The abstract gives no description of how Geo-Eval questions were written, filtered, or checked by experts, and no mention of any decontamination against the synthesis pipeline used for training data. If the benchmark overlaps with the training distribution, the reported gains could come from reduced shift rather than better reasoning. There are also no error bars, significance tests, or controls described. Without those details the central claim cannot be assessed.

This paper is for people already working on domain-adapted LLMs in narrow scientific fields. A reader who needs a starting point for geological instruction data or a benchmark to compare against might find the artifacts useful once they are released. The work does not introduce new methods or resolve open questions in parameter-efficient tuning.

I would send it to peer review so referees can examine the benchmark construction and run the necessary checks. The claims are testable once the data and protocol are available.

Referee Report

2 major / 0 minor

Summary. The paper introduces Geo-Expert, a family of LoRA fine-tuned LLMs (Qwen3-8B, Qwen3-32B, Gemma-3-27B) on a custom-curated geological instruction dataset generated via a custom synthesis pipeline. It claims that the resulting 8B model outperforms open-weight 70B generalists and GPT-4o on a novel Geo-Eval benchmark for specialized geological reasoning, while the 32B variant approaches frontier models, and highlights the 8B variant's favorable cost-performance ratio.

Significance. If the Geo-Eval results hold after proper decontamination and expert validation, the work would demonstrate that parameter-efficient domain adaptation can yield smaller, deployable models that exceed much larger general-purpose LLMs on narrow scientific reasoning tasks. This would supply a reproducible recipe for scientific LLM specialization and establish an initial baseline for geological AI.

major comments (2)

[Abstract] Abstract: performance numbers are reported for Geo-Eval with no accompanying information on benchmark construction, question sourcing, expert validation statistics, evaluation protocol, statistical significance, error bars, or decontamination steps relative to the custom synthesis pipeline. This information is required to evaluate whether the headline outperformance reflects genuine generalization or reduced distribution shift.
[Evaluation] Evaluation section: the central claim that an 8B domain-aligned model outperforms 70B generalists rests on Geo-Eval measuring out-of-distribution expert-level reasoning. Without explicit details on how the benchmark was generated, filtered, or validated independently of the training-data synthesis pipeline, it is impossible to rule out the possibility that performance gains arise from in-distribution effects rather than improved reasoning.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for highlighting the need for greater transparency on Geo-Eval construction and validation. We agree these details are essential to substantiate claims of genuine generalization and will incorporate them in the revised manuscript.

read point-by-point responses

Referee: [Abstract] Abstract: performance numbers are reported for Geo-Eval with no accompanying information on benchmark construction, question sourcing, expert validation statistics, evaluation protocol, statistical significance, error bars, or decontamination steps relative to the custom synthesis pipeline. This information is required to evaluate whether the headline outperformance reflects genuine generalization or reduced distribution shift.

Authors: We acknowledge the abstract omits these details due to length constraints. In revision we will expand the Evaluation and Methods sections with a full description of benchmark construction (question sourcing from peer-reviewed geological literature and exam materials), expert validation (three domain experts with inter-rater agreement statistics), evaluation protocol (zero-shot and few-shot settings, multiple temperature samples), statistical significance testing, error bars from repeated runs, and explicit decontamination steps confirming no overlap with the instruction synthesis pipeline. A brief summary sentence will be added to the abstract. revision: yes
Referee: [Evaluation] Evaluation section: the central claim that an 8B domain-aligned model outperforms 70B generalists rests on Geo-Eval measuring out-of-distribution expert-level reasoning. Without explicit details on how the benchmark was generated, filtered, or validated independently of the training-data synthesis pipeline, it is impossible to rule out the possibility that performance gains arise from in-distribution effects rather than improved reasoning.

Authors: We agree that independent validation details are required to support the out-of-distribution claim. The revised manuscript will add a dedicated subsection describing the benchmark generation process (separate curation team and sources), filtering criteria, expert review protocol, and decontamination analysis (n-gram overlap checks and manual inspection against training instructions). This will allow readers to assess whether gains reflect improved reasoning rather than distribution shift. revision: yes

Circularity Check

0 steps flagged

No significant circularity in claimed results

full rationale

The paper reports empirical fine-tuning results on a custom instruction dataset and evaluation on the novel Geo-Eval benchmark. No equations, derivations, or mathematical steps are present in the abstract or described claims. No load-bearing self-citations, fitted parameters renamed as predictions, or self-definitional reductions are exhibited. The central claims rest on experimental performance numbers rather than any chain that reduces to its own inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review supplies no explicit free parameters, axioms, or invented entities; the central claim rests on unstated assumptions about data quality and benchmark validity.

pith-pipeline@v0.9.1-grok · 5732 in / 1241 out tokens · 34538 ms · 2026-06-30T11:43:24.819391+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

4 extracted references

[1]

Focus on SEMANTIC CORRECTNESS and GEOLOGICAL ACCURACY, not verbatim matching
[2]

igneous" vs

Terminology Flexibility: Be lenient with accepted geological synonyms (e.g., "igneous" vs "magmatic", "thrust fault" vs "reverse fault" depending on context)
[3]

Completeness: A 10/10 answer must contain all core geological facts mentioned in the reference
[4]

score": 8,

Penalty: Deduct heavily for hallucinations, factually incorrect statements, or mixing up crucial concepts (e.g., misidentifying rock types or eras). [Output Format] Output ONLY a JSON object with this exact structure: {"score": 8, "reason": "brief professional explanation"} Question: {question} Reference Answer: {reference} Model Answer: {model_ans} B.3. ...

[1] [1]

Focus on SEMANTIC CORRECTNESS and GEOLOGICAL ACCURACY, not verbatim matching

[2] [2]

igneous" vs

Terminology Flexibility: Be lenient with accepted geological synonyms (e.g., "igneous" vs "magmatic", "thrust fault" vs "reverse fault" depending on context)

[3] [3]

Completeness: A 10/10 answer must contain all core geological facts mentioned in the reference

[4] [4]

score": 8,

Penalty: Deduct heavily for hallucinations, factually incorrect statements, or mixing up crucial concepts (e.g., misidentifying rock types or eras). [Output Format] Output ONLY a JSON object with this exact structure: {"score": 8, "reason": "brief professional explanation"} Question: {question} Reference Answer: {reference} Model Answer: {model_ans} B.3. ...