Pith. sign in

REVIEW 3 major objections 5 minor 32 references

ChemAU: Harness the Reasoning of LLMs in Chemical Research with Adaptive Uncertainty Estimation

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ChemAU improves LLM chemistry reasoning by scoring each reasoning step and triggering a fine-tuned chemistry model to correct uncertain steps.

desk verdict The empirical idea is plausible, but the adaptive uncertainty formula as parameterized cannot trigger selectively, which collapses the central claim. read the letter →

arxiv 2506.01116 v1 pith:KWJD64JH submitted 2025-06-01 cs.AI q-bio.QM

Xinyi Liu , Lipeng Ma , Yixuan Li , Weidong Yang , Qingyuan Zhou , Jiayi Song , Shuhao Li , Ben Fei This is my paper
classification cs.AIq-bio.QM
keywords chemistryreasoninguncertaintyestimationchain-of-thoughtadaptivedomain-specificmodelknowledgesupplementationlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ChemAU is a reasoning framework that makes general-purpose large language models more reliable on chemistry questions. It has the model write out its chain of thought, scores each step for uncertainty, and whenever a step looks unreliable it sends that step to a small chemistry-specialized model, which returns the correct domain knowledge before the main model continues. The paper claims this uncertainty-triggered correction raises accuracy on chemistry questions from GPQA, MMLU-Pro, and SuperGPQA across three open-source LLMs, and improves the quality of uncertainty estimates themselves. A sympathetic reader should care because the approach targets the specific knowledge gaps that cause chemistry hallucinations rather than regenerating whole answers or retrieving broad background text.

What carries the argument

The load-bearing mechanism is the adaptive uncertainty estimator. It scores the $i$-th reasoning step of a chain with $L_R$ steps as $U_i = \max_j -\log(p_{ij}) + \alpha (L_R - i)$, using $\alpha = -0.08$ and triggering further processing when $U_i > \theta$ with $\theta = -1.5$. The position term $L_R - i$ is what makes the estimate adaptive: the paper argues that chemistry tokens become artificially more probable as reasoning proceeds, so earlier steps need a different treatment from later ones. When a step triggers, the framework decomposes it into atomic knowledge points, asks a fine-tuned Qwen2.5-1.5B chemistry model to supply corrected knowledge, and reintroduces that knowledge with the confirmed earlier steps into the general LLM to regenerate the rest of the chain.

What would settle it

Run ChemAU on a chemistry benchmark whose questions have been checked not to overlap with the author-constructed training data for the domain model; if the accuracy gain over the base LLM largely disappears, the improvement comes from memorized benchmark content rather than from uncertainty-guided knowledge supplementation.

Watch

Extended reading notes

Core claim

The paper's central claim is that chemistry reasoning fails not because general LLMs cannot reason, but because they lack precise chemistry knowledge at particular steps, and that those steps can be located by a position-aware uncertainty signal. The authors observe that chemistry-specific tokens start with low logits and rise as the reasoning chain grows, so a fixed uncertainty threshold misclassifies steps. Their adaptive estimator adjusts each step's uncertainty by its position in the chain; a step above the trigger threshold is decomposed into atomic chemistry-knowledge points, corrected by a fine-tuned chemistry domain model, and then fed back into the general model to resume reasoning. The reported result is consistent accuracy gains across three backbones, with the largest improvement on MMLU-Pro for LLaMA-3, and ablation results indicating both the domain model and the step-wise trigger are necessary.

Load-bearing premise

The load-bearing premise is that the fine-tuned 1.5B chemistry model gives correct corrections whenever the uncertainty trigger fires, and that a flagged step really is a knowledge gap rather than a benign low-probability token.

Editorial extensions

If this is right

  • ChemAU improves chemistry accuracy across all three tested open-source models, with LLaMA-3 reaching 53.56% on MMLU-Pro, above the unaided 14B DeepSeek-R1 model.
  • Corrections are localized to single reasoning steps, so the framework can repair a chain without regenerating or rescoring the whole response.
  • The method requires token logits, so it applies to any open-source LLM and does not need multiple sampled generations for uncertainty.
  • The ablation results imply that telling the general model a step is uncertain is not enough; the externally supplied chemistry knowledge is what fixes the error.
  • Retrieval-augmented generation with broad knowledge is expected to trail ChemAU, because imprecise retrieved knowledge can mislead the reasoning chain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same position-dependent logit inflation could occur in other terminology-heavy domains such as law or medicine, where a fixed-threshold uncertainty estimator would face the same failure mode.
  • A direct extension would swap the fine-tuned chemistry model for a public knowledge source or a tool-based verifier, while keeping the step-level uncertainty trigger.
  • Because the domain model's training corpus is not described, a natural next test is to rebuild ChemAU with a public chemistry dataset and check whether the accuracy gains persist.
  • For black-box LLMs, approximating token logits by sampling many continuations could test whether the adaptive position term still helps without direct logit access.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ChemAU, a framework that augments a general LLM's chain-of-thought reasoning on chemistry problems with an adaptive, position-weighted uncertainty estimator. The estimator is intended to identify specific reasoning steps that are likely to contain chemistry knowledge errors, after which a fine-tuned 1.5B chemistry domain model is invoked to supply corrected knowledge and the LLM regenerates the remainder of the chain. The method is evaluated on chemistry subsets of GPQA, MMLU-Pro, and SuperGPQA with three base LLMs (Qwen2.5-7B-Instruct, LLaMA-3-8B-Instruct, DeepSeek-R1-Distill-Qwen-14B), reporting accuracy gains over vanilla CoT and RAG baselines, alongside ablations on the domain model and on step-wise versus chain-level uncertainty.

Significance. If the claimed adaptive uncertainty mechanism genuinely identified the steps that need domain knowledge, ChemAU would be a useful contribution to LLM reasoning in specialized scientific domains. The paper includes a multi-model, multi-dataset evaluation and two ablation studies, which are welcome. However, the central uncertainty formula as parameterized is numerically incapable of performing the claimed step-selective detection: with alpha = -0.08 and theta = -1.5, the trigger condition U_i > theta is satisfied for every step in the paper's own example and for essentially any chain of realistic length. The reported accuracy gains therefore cannot be attributed to adaptive, step-selective uncertainty estimation, and the paper's central claim is not supported by its evidence.

major comments (3)
  1. [Section 3.1, Eq. (2), Appendix B] With alpha = -0.08 and theta = -1.5, the decision rule U_i > theta is vacuous for any practical reasoning chain. Since -log(p_ij) >= 0 for probabilities in (0,1], U_i is bounded below by alpha(L_R - i). In the paper's own 3-step example, U_1 >= -0.16, U_2 >= -0.08, and U_3 >= 0, so all three steps are above theta = -1.5 and are flagged as uncertain. For any chain with L_R <= 18 steps, the minimum possible value of U_i is alpha(17) = -1.36, which is still greater than -1.5, so every step is flagged. The threshold can only begin to bind for chains of 20 or more steps, and only when every token in the first step has probability exactly 1. Thus the mechanism does not implement the claimed step-selective triggering; it always consults the domain model, and the comparison against Max(-log p) in Figure 5 does not test selective detection.
  2. [Section 3.3 versus Section 3.1] The paper states in Section 3.3 that 'the certainty for earlier reasoning steps should be stricter' and that the method 'assigns higher uncertainty values to reasoning steps at earlier positions'. Equation (2) with alpha = -0.08 produces the opposite: for i < L_R, the additive term alpha(L_R - i) is negative and is largest in magnitude for the smallest i, so earlier steps receive a larger negative penalty and therefore a lower U_i. The sign of the position term is thus inconsistent with the design rationale stated in the paper.
  3. [Section 3.2 and Section 4.1] The specialized chemistry domain model is created by instruction fine-tuning Qwen2.5-1.5B-Instruct on an 'author-constructed chemistry knowledge dataset', but the dataset's size, provenance, construction process, and quality controls are not described. Without this information, and without any analysis of potential overlap between the domain model's training data and the evaluation benchmarks, the reported accuracy improvements cannot be attributed to the uncertainty-driven correction mechanism rather than to training-data leakage or to the domain model's general knowledge. This is a load-bearing gap for the paper's attribution claim.
minor comments (5)
  1. [Figure 1 and Section 3.1] The paper uses 'logit' and 'probability' interchangeably. The values 0.72, 0.81, and 1.0 in Figure 1 are probabilities, not logits (which can be any real number). Please clarify whether p_ij in Eq. (2) denotes token probabilities or logits, and use consistent terminology throughout.
  2. [Eq. (2)] The paper does not specify whether the step index i starts at 0 or 1, which changes the value of L_R - i. Please state the indexing convention.
  3. [Figure 5] The legend in Figure 5 uses 'Max(p)' while the text and Eq. (1) define 'Max(-log p)'. This inconsistency makes the baseline hard to interpret.
  4. [Section 5] The main text reports that removing the domain model causes accuracy to drop 'up to 16.39%', while the caption of Figure 6 states 'up to 14.87%'. These numbers should be reconciled.
  5. [Section 1] The claim that ChemAU is 'the first framework to introduce a model collaboration strategy for chemistry reasoning tasks' is too strong and is not supported by a systematic literature search; it is likely to be contested.

Circularity Check

1 steps flagged · score 6.0 of 10

With α=-0.08 and θ=-1.5, Eq. (2) always exceeds the threshold for the paper's own 3-step chains, so step-selective uncertainty detection collapses by construction into unconditional domain-model consultation.

  1. other [Section 3.1, Eq. (2); Section 3.3; Appendix B]
    "Ui(R, Pi) = max_j −log(pij) + α(LR − i) ... If the uncertainty exceeds a predefined threshold θ, formally expressed as: if Ui(R, Pi) > θ, indicating that this reasoning step exhibits a high likelihood of containing potential errors ... The basic threshold value is set to -1.5, while the hyperparameter α, which incorporates the relative positions of reasoning steps into the uncertainty estimation model, is set to -0.08."

    By the paper's own definitions, p_ij is a token probability, so max_j −log(p_ij) ≥ 0. With α = −0.08, the position term satisfies α(LR − i) ≥ −0.08(LR − 1). For the paper's 3-step example, and for every chain with LR ≤ 19, U_i ≥ −0.16 > −1.5 = θ, so the condition U_i > θ holds for every reasoning step. The adaptive rule therefore never marks any step as certain; it is equivalent, by construction, to always invoking the domain model. Moreover, α(LR − i) is most negative for the earliest step (i=1), which lowers—not raises—the uncertainty of early steps, contradicting the Section 3.3 claim that earlier steps receive higher uncertainty values. Consequently, the Figure 5 comparison of Eq.

full rationale

The paper's headline accuracy results are evaluated on external benchmarks (GPQA, MMLU-Pro, SuperGPQA), so they are not derived from a self-citation chain, and no load-bearing self-citations are present. The central circularity-adjacent defect is internal to the uncertainty estimator. Eq. (2) with the stated α and θ yields U_i > θ for every step of any chain of length ≤19, including the paper's own 3-step example. The step-selection rule is therefore a constant 'always flag'; the claimed adaptive, step-selective identification of knowledge gaps reduces by construction to unconditional consultation of the domain model. This makes the Figure 5 comparison between Eq. (2) and Max(−log p) uninformative as evidence for selective uncertainty detection, and the reported gains cannot be attributed to the mechanism claimed. The undisclosed author-constructed fine-tuning dataset is a data-leakage and transferability concern, not a demonstrated derivation circularity under the hard rules of this review.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the correctness of the domain model, the validity of the logit-rise phenomenon, and the decomposition of steps into atomic knowledge. None of these are independently verified, and the hyperparameters are not shown to be chosen on a separate validation set.

free parameters (2)
  • alpha (position penalty coefficient) = -0.08
    Chosen without a described tuning procedure; sign contradicts the paper's claim that earlier steps get higher uncertainty.
  • theta (uncertainty threshold) = -1.5
    Given threshold appears to flag almost all steps under the stated formula, so the triggering behavior is unclear; no validation procedure described.
assumptions (3)
  • domain assumption The logit values of chemistry-specific tokens progressively increase over reasoning steps as a general phenomenon.
    Supported only by one illustrative example (Figure 1); no systematic measurement.
  • domain assumption The fine-tuned 1.5B chemistry domain model provides accurate corrections for flagged reasoning steps.
    The model is trained on an undisclosed author-constructed dataset; its knowledge quality is not independently evaluated.
  • domain assumption Reasoning steps can be decomposed into atomic chemistry knowledge points that faithfully capture the error.
    Section 3.2 assumes this decomposition and that the domain model can evaluate each point.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ChemAU: Harness the Reasoning of LLMs in Chemical Research with Adaptive Uncertainty Estimation." pith.science (2026). https://pith.science/paper/KWJD64JH

@misc{pith2026250601116,
  author       = {Pith},
  title        = {Pith review of: ChemAU: Harness the Reasoning of LLMs in Chemical Research with Adaptive Uncertainty Estimation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KWJD64JH}},
  note         = {Machine review of arXiv:2506.01116}
}
read the original abstract

Large Language Models (LLMs) are widely used across various scenarios due to their exceptional reasoning capabilities and natural language understanding. While LLMs demonstrate strong performance in tasks involving mathematics and coding, their effectiveness diminishes significantly when applied to chemistry-related problems. Chemistry problems typically involve long and complex reasoning steps, which contain specific terminology, including specialized symbol systems and complex nomenclature conventions. These characteristics often cause general LLMs to experience hallucinations during the reasoning process due to their lack of specific knowledge. However, existing methods are struggling to effectively leverage chemical expertise and formulas. Moreover, current uncertainty estimation methods, designed to mitigate potential reasoning errors, are unable to precisely identify specific steps or key knowledge. In this work, we propose a novel framework called ChemAU, which incorporates our adaptive uncertainty estimation method that applies different uncertainty values based on the position of reasoning steps within the whole reasoning chain. Leveraging this method, ChemAU identifies gaps in chemistry knowledge and precisely supplements chemical expertise with the specialized domain model, thereby correcting and updating the previously flawed reasoning chain. Our experiments with three popular LLMs across three chemistry datasets demonstrate that ChemAU significantly enhances both reasoning accuracy and uncertainty estimation.

Figures

Figures reproduced from arXiv: 2506.01116 by the authors.

Figure 1
Figure 1. Chemistry-specific tokens exhibit a distinctive pattern during the reasoning process: as reasoning steps [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of our proposed LLM reasoning framework with [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. In chemistry problem-solving, reasoning steps are closely interconnected, with each subsequent step typically [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Performance comparison of different reasoning approaches across three chemistry datasets. Each sub-figure [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Performance comparison of uncertainty estimation methods on chemistry reasoning tasks. The figure [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: (Left) The comparisons of the accuracy (%) of our complete reasoning framework against a variant without [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Different uncertainty estimation methods demonstrate varying performance on the same sentence in everyday [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Models designed to assess semantic similarity perform poorly when evaluating sentences containing specific [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: A prompt template designed to guide the general LLM in generating an initial reasoning chain for chemistry [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 12
Figure 12. Figure 12: Knowledge Augmentation Comparison. Precise and relevant knowledge augmentation facilitates correct [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]
Figure 10
Figure 10. Figure 10: (Left) The comparisons of the accuracy (%) of our complete reasoning framework against a variant without [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: A prompt template designed to guide the general LLM in regenerating the reasoning chain for chemistry [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

32 extracted references · 1 canonical work pages

  1. [2]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    arXiv

  2. [5]

    Can large language models empower molecular property prediction?arXiv preprint arXiv:2307.07443,

    Chen Qian, Huayi Tang, Zhirui Yang, Hong Liang, and Yong Liu. Can large language models empower molecular property prediction?arXiv preprint arXiv:2307.07443,

    arXiv

  3. [6]

    Crispr-gpt: An llm agent for automated design of gene-editing experiments.arXiv preprint arXiv:2404.18021, 2024a

    Kaixuan Huang, Yuanhao Qu, Henry Cousins, William A Johnson, Di Yin, Mihir Shah, Denny Zhou, Russ Altman, Mengdi Wang, and Le Cong. Crispr-gpt: An llm agent for automated design of gene-editing experiments.arXiv preprint arXiv:2404.18021, 2024a. Sheng Wang, Yuzhi Guo, Yuhong Wang, Hongmao Sun, and Junzhou Huang. Smiles-bert: large scale unsupervised pre-t...

    arXiv

  4. [8]

    Molgpt: molecular generation using a transformer- decoder model.Journal of chemical information and modeling, 62(9):2064–2076,

    Viraj Bagal, Rishal Aggarwal, PK Vinod, and U Deva Priyakumar. Molgpt: molecular generation using a transformer- decoder model.Journal of chemical information and modeling, 62(9):2064–2076,

    work page

  5. [10]

    Uncertainty quantification and confidence calibration in large language models: A survey.arXiv preprint arXiv:2503.15850,

    Xiaoou Liu, Tiejin Chen, Longchao Da, Chacha Chen, Zhen Lin, and Hua Wei. Uncertainty quantification and confidence calibration in large language models: A survey.arXiv preprint arXiv:2503.15850,

    arXiv

  6. [11]

    Look before you leap: An exploratory study of uncertainty measurement for large language models.arXiv preprint arXiv:2307.10236,

    15 ChemAUA PREPRINT Yuheng Huang, Jiayang Song, Zhijie Wang, Shengming Zhao, Huaming Chen, Felix Juefei-Xu, and Lei Ma. Look before you leap: An exploratory study of uncertainty measurement for large language models.arXiv preprint arXiv:2307.10236,

    arXiv

  7. [12]

    An evaluation of estimative uncertainty in large language models

    Zhisheng Tang, Ke Shen, and Mayank Kejriwal. An evaluation of estimative uncertainty in large language models. arXiv preprint arXiv:2405.15185,

    arXiv

  8. [13]

    Quantifying uncertainty in answers from any language model via intrinsic and extrinsic confidence assessment.arXiv preprint arXiv:2308.16175, 2,

    Jiuhai Chen and Jonas Mueller. Quantifying uncertainty in answers from any language model via intrinsic and extrinsic confidence assessment.arXiv preprint arXiv:2308.16175, 2,

    arXiv

Show all 32 references
  1. [14]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115,

    arXiv

  2. [15]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Let- man, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    arXiv

  3. [16]

    Supergpqa: Scaling llm evaluation across 285 graduate disciplines.arXiv preprint arXiv:2502.14739,

    Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, et al. Supergpqa: Scaling llm evaluation across 285 graduate disciplines.arXiv preprint arXiv:2502.14739,

    arXiv

  4. [17]

    Uncertainty quantification in fine-tuned llms using lora ensembles.arXiv preprint arXiv:2402.12264,

    Oleksandr Balabanov and Hampus Linander. Uncertainty quantification in fine-tuned llms using lora ensembles.arXiv preprint arXiv:2402.12264,

    arXiv

  5. [18]

    A survey on uncertainty quan- tification of large language models: Taxonomy, open research challenges, and future directions.arXiv preprint arXiv:2412.05563,

    Ola Shorinwa, Zhiting Mei, Justin Lidard, Allen Z Ren, and Anirudha Majumdar. A survey on uncertainty quan- tification of large language models: Taxonomy, open research challenges, and future directions.arXiv preprint arXiv:2412.05563,

    arXiv

  6. [19]

    The internal state of an llm knows when it’s lying.arXiv preprint arXiv:2304.13734,

    Amos Azaria and Tom Mitchell. The internal state of an llm knows when it’s lying.arXiv preprint arXiv:2304.13734,

    arXiv

  7. [20]

    Luq: Long-text uncertainty quantification for llms

    Caiqi Zhang, Fangyu Liu, Marco Basaldella, and Nigel Collier. Luq: Long-text uncertainty quantification for llms. arXiv preprint arXiv:2403.20279, 2024b. Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizav...

    arXiv

  8. [21]

    Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models.arXiv preprint arXiv:2303.08896,

    Potsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models.arXiv preprint arXiv:2303.08896,

    arXiv

  9. [22]

    Fact-checking the output of large language models via token-level uncertainty quantification.arXiv preprint arXiv:2403.04696,

    Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, et al. Fact-checking the output of large language models via token-level uncertainty quantification.arXi...

    arXiv

  10. [23]

    Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692,

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692,

    1907 arXiv

  11. [24]

    Sentence-bert: Sentence embeddings using siamese bert-networks.arXiv preprint arXiv:1908.10084,

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks.arXiv preprint arXiv:1908.10084,

    1908 arXiv

  12. [25]

    Bertscore: Evaluating text generation with bert.arXiv preprint arXiv:1904.09675,

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert.arXiv preprint arXiv:1904.09675,

    1904 arXiv

  13. [26]

    Decomposing uncertainty for large language models through input clarification ensembling.arXiv preprint arXiv:2311.08718,

    Bairu Hou, Yujian Liu, Kaizhi Qian, Jacob Andreas, Shiyu Chang, and Yang Zhang. Decomposing uncertainty for large language models through input clarification ensembling.arXiv preprint arXiv:2311.08718,

    arXiv

  14. [27]

    Ambigqa: Answering ambiguous open- domain questions.arXiv preprint arXiv:2004.10645,

    Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. Ambigqa: Answering ambiguous open- domain questions.arXiv preprint arXiv:2004.10645,

    2004 arXiv

  15. [29]

    Personal llm agents: Insights and survey about the capability, efficiency and security.arXiv preprint arXiv:2401.05459, 2024b

    Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al. Personal llm agents: Insights and survey about the capability, efficiency and security.arXiv preprint arXiv:2401.05459, 2024b. Kevin Maik Jablonka, Phi...

    arXiv

  16. [30]

    Chemdfm: A large language foundation model for chemistry.arXiv preprint arXiv:2401.14818,

    Zihan Zhao, Da Ma, Lu Chen, Liangtai Sun, Zihao Li, Yi Xia, Bo Chen, Hongshen Xu, Zichen Zhu, Su Zhu, et al. Chemdfm: A large language foundation model for chemistry.arXiv preprint arXiv:2401.14818,

    arXiv

  17. [31]

    Chemeval: A comprehensive multi-level chemical evaluation for large language models.arXiv preprint arXiv:2409.13989, 2024b

    Yuqing Huang, Rongyang Zhang, Xuesong He, Xuyang Zhi, Hao Wang, Xin Li, Feiyang Xu, Deguang Liu, Huadong Liang, Yi Li, et al. Chemeval: A comprehensive multi-level chemical evaluation for large language models.arXiv preprint arXiv:2409.13989, 2024b. Andrey Malinin and Mark Gal...

    2002 arXiv

  18. [32]

    Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models

    Jinhao Duan, Hao Cheng, Shiqi Wang, Alex Zavalny, Chenan Wang, Renjing Xu, Bhavya Kailkhura, and Kaidi Xu. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. arXiv preprint arXiv:2307.01379,

    arXiv

  19. [2019]

    Smiles transformer: Pre-trained molecular fingerprint for low data drug discovery.arXiv preprint arXiv:1911.04738,

    Shion Honda, Shoi Shi, and Hiroki R Ueda. Smiles transformer: Pre-trained molecular fingerprint for low data drug discovery.arXiv preprint arXiv:1911.04738,

    1911 arXiv

  20. [2020]

    Clam: Selective clarification for ambiguous questions with generative language models.arXiv preprint arXiv:2212.07769,

    16 ChemAUA PREPRINT Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Clam: Selective clarification for ambiguous questions with generative language models.arXiv preprint arXiv:2212.07769,

    arXiv

  21. [2022]

    Self-reflection in llm agents: Effects on problem-solving performance.arXiv preprint arXiv:2405.06682,

    Matthew Renze and Erhan Guven. Self-reflection in llm agents: Effects on problem-solving performance.arXiv preprint arXiv:2405.06682,

    arXiv

  22. [2023]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    arXiv

  23. [2024]

    Legalagentbench: Evaluating llm agents in legal domain.arXiv preprint arXiv:2412.17259, 2024a

    Haitao Li, Junjie Chen, Jingli Yang, Qingyao Ai, Wei Jia, Youfeng Liu, Kai Lin, Yueyue Wu, Guozhi Yuan, Yiran Hu, et al. Legalagentbench: Evaluating llm agents in legal domain.arXiv preprint arXiv:2412.17259, 2024a. Sagar Goyal, Eti Rastogi, Sree Prasanna Rajagopal, Dong Yuan,...

    arXiv

  24. [2025]

    Chemllm: A chemical large language model.arXiv preprint arXiv:2402.06852, 2024a

    Di Zhang, Wei Liu, Qian Tan, Jingdan Chen, Hang Yan, Yuliang Yan, Jiatong Li, Weiran Huang, Xiangyu Yue, Wanli Ouyang, et al. Chemllm: A chemical large language model.arXiv preprint arXiv:2402.06852, 2024a. Junxian Li, Di Zhang, Xunzhi Wang, Zeying Hao, Jingdi Lei, Qian Tan, C...

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.