REVIEW 3 major objections 5 minor 28 references
Quantification of Biodiversity from Historical Survey Text with LLM-based Best-Worst Scaling
T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read LLM-based Best-Worst Scaling can extract continuous species-frequency estimates from historical survey text without manual training annotation.
desk verdict A transparent, honest feasibility study of LLM-based BWS for historical biodiversity texts; the method is promising but the headline R² measures self-consistency, not ecological accuracy. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is Best-Worst Scaling (BWS) with LLM annotators. In BWS, a set of four texts is shown and the annotator picks the one conveying the highest quantity ('best') and the one conveying the lowest quantity ('worst'); each text receives a score $s(i) = \frac{\#\text{best}(i) - \#\text{worst}(i)}{\#\text{overall}(i)}$, normalized to $[0,1]$. With set size 4 and $N=2$, each text appears in eight comparisons, giving 17 discrete score levels. These scores become the regression target, and the paper shows that a transfer-learning setup—LaBSE fine-tuned on GPT-4 BWS scores of 114 n-gram quantifiers—yields the best predictions, with R²=0.73. The mechanism suppresses rating-scale design choices and manual annotation while retaining fine granularity.
What would settle it
Find a historical source with independently known species abundances, such as a region with documented counts or modern distribution data projected back, and compare the LLM-BWS regression predictions against those ground-truth frequencies; systematic divergence would falsify the assumption that BWS scores encode true frequency.
Extended reading notes
Core claim
The paper's central claim is that LLM-based Best-Worst Scaling (BWS) can replace manual rating-scale annotation for species quantity estimation from historical text. From 1,000 survey texts describing animal occurrence, the authors create 2,000 four-text tuples and ask GPT-4, DeepSeek-V3, and Ministral-8B to choose the text with the highest and the lowest quantity in each tuple. Aggregated BWS scores, normalized to [0,1], serve as regression targets; a LaBSE-based transfer model trained on GPT-4 scores predicts them with R²=0.73 and MAE=0.107. The authors conclude this is more cost-effective and similarly robust compared to a 7-class multi-class approach, since it avoids manual training labels and rating-scale design decisions while producing finer-grained continuous output.
Load-bearing premise
The pipeline assumes that the BWS scores produced by LLMs reflect true ecological frequency, but the paper never validates these scores against external abundance data, so absolute accuracy remains unknown.
Editorial extensions
If this is right
- If the BWS scores are valid, historical survey texts can be converted into continuous per-species frequency estimates without manual annotation, enabling large-scale quantitative biodiversity reconstruction.
- The same LLM-BWS plus regression pipeline transfers to other domains where rating scales are arbitrary, such as sentiment intensity or text difficulty, because it relies only on comparative judgments.
- The best-performing transfer model uses only 114 n-gram quantifiers, and training curves still improve, so adding more data should raise performance further.
- The method inherits LLM biases and the subjectivity of the original observers; the paper notes that biases in historical texts could be reinforced rather than corrected.
- Extinction and very high-frequency cases are structurally hard for the regression, so downstream users should treat the extremes of the scale with caution.
Reading between the lines
- The unusually high agreement between GPT-4 and DeepSeek-V3 could mean the comparative task is objective enough that independent large models converge on the same ordering, but the paper itself notes this could also reflect internal model similarity.
- A testable extension is to have the LLM score each text directly on the same latent scale and use BWS only as a calibration check, which would isolate how much of the signal comes from the comparative format itself.
- The method could be applied to the other historical corpora the paper mentions, such as Bavarian flora records or the Württemberg Oberamtsbeschreibungen, where cross-region comparisons or modern red-list data might serve as indirect validation.
- The cost structure changes dramatically when manual annotation is removed, so even rare classes like 'Extinct' become learnable at marginal cost; the case study suggests such extremes should be interpreted as continuous scores rather than hard classification boundaries.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper addresses the problem of estimating species frequency from historical survey texts, using a corpus of 2,555 transcribed entries from an 1845 Bavarian forestry survey. The authors compare three task formulations: binary presence/absence classification, a 7-class ordinal classification, and a regression task in which continuous scores are generated by Best-Worst Scaling (BWS) performed by LLMs (Ministral-8B, DeepSeek-V3, GPT-4). They report human-LLM agreement on a small BWS annotation sample, train regression models on LLM-generated BWS scores (with LaBSE-based transfer learning giving the best reported R²=0.73, MAE=0.107), and present a case study mapping regression predictions to human multi-class annotations. The paper concludes that the LLM-BWS regression approach is more cost-effective and "similarly robust" compared to fine-grained multi-class classification.
Significance. If the central claim is supported, LLM-based BWS could substantially lower the cost of quantifying species frequency from historical documents, which is valuable for biodiversity and historical ecology research. The paper has concrete strengths: it makes the data and code available, evaluates multiple LLMs, includes a human agreement study on the BWS task, and is transparent about the lack of an external benchmark. The main limitation is that the regression's headline accuracy is measured against the LLM's own BWS labels, so the paper currently demonstrates self-consistency rather than external validity. The human agreement results provide partial support, but they are limited to 100 BWS tuples and do not validate the final quantity estimates against true abundance or against human scores for the regression targets. The comparison to multi-class classification is also cross-metric, which weakens the "similarly robust" conclusion.
major comments (3)
- [§3.3.2 / Table 4 / Limitations] The reported R²=0.73 and MAE=0.107 in Table 4 are computed against held-out BWS scores produced by GPT-4, the same model whose BWS scores were used as training targets for the transfer model. This establishes that the regressor can reproduce GPT-4's BWS judgments, not that those judgments are accurate estimates of ecological frequency. The human-LLM agreement in Table 3 is measured on the BWS tuple task (100 tuples) and is not used to generate the regression targets. The paper's limitations paragraph concedes "we also test on BWS values," but the abstract and conclusion claim the approach is "similarly robust" and "adequate." To support the central claim, the authors should either (a) evaluate the trained regressor against human BWS scores for a held-out set of texts, or (b) evaluate against the human multi-class annotations used in the case study by reporting rank correlation or ordinal mapping. Without such validation, the claim conflates internal consistency with external accuracy.
- [§3.2 vs. §3.3.2] The comparison between the multi-class approach (F1 micro/macro in Table 2) and the regression approach (R² and MAE in Table 4) is cross-metric, so the statement that the regression approach is "similarly robust" is not directly supported. A model with poor ordinal calibration could achieve high R² on BWS scores while misranking species relative to human judgments. The authors should report a common evaluation, for example mapping regression predictions to the 7 ordinal classes and computing macro-F1, or converting the multi-class classifier's outputs to a continuous score and computing R² or rank correlation on the same texts.
- [§3.3.1 / Eq. (1)] With N=2 and set size 4, each text appears in 8 comparison sets, yielding scores that take only 17 discrete values. The paper does not report reliability of these BWS scores (e.g., split-half reliability or bootstrap variance over the sampled tuples). Because these scores are used as regression targets, measurement error in the BWS scores directly caps the achievable R² and affects the interpretation of the regression results. The authors should either report score reliability or discuss how the discrete, relatively coarse score distribution influences the reported error metrics.
minor comments (5)
- [§3.3.1 / Eq. (1)] The denominator "#overall(i)" in Eq. (1) is not defined explicitly; it should state that it is the number of comparison sets in which text i appears (here, 8).
- [Table 3] The p-value (p=.118) for the GPT-4 vs. DeepSeek-V3 comparison is reported without stating the statistical test used; please specify the test and whether any multiple-comparison correction was applied.
- [§3.1] The phrase "78% majority baseline" should be clarified as the majority-class baseline, since the 78% refers to the proportion of PRESENT labels in the unique-text set.
- [Figure 4] The identifiers "SP 0015" and "SP 0005" in the caption are not explained; please either spell out the species names or refer to a table that defines the identifiers.
- [Code availability] The repository link "github.org/maelkolb/biodivquant" appears to be a typo for "github.com"; please correct it so the code is accessible.
Circularity Check
The regression's headline R²=0.73 is measured against GPT-4's own BWS labels, so it validates self-consistency, not ecological accuracy; the paper's own Limitations admit 'we also test on BWS values.'
-
fitted input called prediction
[Section 3.3.2 / Table 4; Limitations]
"Furthermore, without a reliable dataset to benchmark against, it is difficult to assess the absolute accuracy of the BWS-based regression approach, because we also test on BWS values."
Equation (1) defines the regression target s(i) from LLM Best-Worst choices, and Table 4 reports R² and MAE for models trained and evaluated on those same LLM-generated BWS values. The abstract's 'quantity estimation' is therefore prediction of GPT-4's or DeepSeek-V3's own judgments, not of external species frequency. A high R² measures how well LaBSE reproduces the LLM's BWS scoring, which is a self-consistency check; the paper concedes that absolute accuracy cannot be assessed because the evaluation also uses BWS values.
-
fitted input called prediction
[Section 3.3.2, transfer learning setup]
"for which we scale the 114 n-gram quantifiers as extracted from the binary Logistic Regression with another GPT-4 BWS, then match these scores to the texts and tune a LaBSE model on the same train/test split before using it for the final task."
The transfer-learning supervision is again GPT-4 BWS: the n-gram quantifier scores are themselves produced by 'another GPT-4 BWS,' and the final LaBSE model is tuned on the same GPT-4 BWS targets. No human labels or external abundance data enter the training signal, so the reported transfer gain reflects improved alignment with GPT-4's quantity judgments rather than improved estimation of true ecological frequency.
full rationale
Most of the paper's pipeline is self-contained or anchored in standard external references: the BWS design follows Kiritchenko and Mohammad (2017), the data set is published on Zenodo, and the binary and multi-class classifiers are evaluated against manual annotation. No load-bearing self-citation chain or imported uniqueness theorem is present. The circularity is confined to the continuous-quantity evaluation. Equation (1) derives the regression target from LLM BWS choices, and Table 4 reports R²/MAE of models trained and tested on those same LLM-generated scores. The paper explicitly concedes in the Limitations that 'we also test on BWS values.' Consequently, the R²=0.73 headline measures how well a text embedder reproduces GPT-4's BWS judgments, not how well it estimates true species frequency. The human-LLM agreement study and the case-study mapping to human multi-class labels provide partial independent support, but the human BWS set is small (100 tuples) and is not used to rescale the regression targets; the final 'similarly robust' comparison also mixes R² on LLM-BWS labels with F1 on human labels. These issues make the central claim partially circular rather than fully derived from external evidence.
Assumptions & free parameters
assumptions (3)
- domain assumption Frequency-related information in historical texts can be accurately mapped to numerical frequency estimates.
- domain assumption LLM BWS judgments approximate human judgments of quantity sufficiently for downstream regression.
- standard math The BWS score formula produces a valid interval scale of quantity.
Cite this review
Pith. "Pith review of Quantification of Biodiversity from Historical Survey Text with LLM-based Best-Worst Scaling." pith.science (2026). https://pith.science/paper/FPJY3RQ5
@misc{pith2026250204022,
author = {Pith},
title = {Pith review of: Quantification of Biodiversity from Historical Survey Text with LLM-based Best-Worst Scaling},
year = {2026},
howpublished = {\url{https://pith.science/paper/FPJY3RQ5}},
note = {Machine review of arXiv:2502.04022}
}
read the original abstract
In this study, we evaluate methods to determine the frequency of species via quantity estimation from historical survey text. To that end, we formulate classification tasks and finally show that this problem can be adequately framed as a regression task using Best-Worst Scaling (BWS) with Large Language Models (LLMs). We test Ministral-8B, DeepSeek-V3, and GPT-4, finding that the latter two have reasonable agreement with humans and each other. We conclude that this approach is more cost-effective and similarly robust compared to a fine-grained multi-class approach, allowing automated quantity estimation across species.
Figures
Reference graph
Works this paper leans on
-
[1]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774
arXiv 2023
-
[4]
Christopher Bagdon, Prathamesh Karmalkar, Harsha Gurulingappa, and Roman Klinger. 2024. https://doi.org/10.18653/v1/2024.naacl-long.439 you are an expert annotator : Automatic best -- worst-scaling annotations for emotion intensity modeling . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguisti...
-
[5]
Stig Johan Berggren, Taraka Rama, and Lilja vrelid. 2019. https://doi.org/10.18653/v1/W19-4409 Regression or classification? automated essay scoring for N orwegian . In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 92--102, Florence, Italy. Association for Computational Linguistics
-
[6]
Stefan Br \"o nnimann, Christian Pfister, and Sam White. 2018. Archives of nature and archives of societies. In The Palgrave Handbook of Climate History, pages 27--36. Palgrave Macmillan UK, London
work page 2018
-
[7]
Robert M Dorazio, Nicholas J Gotelli, and Aaron M Ellison. 2011. Modern methods of estimating biodiversity from presence-absence surveys. Biodiversity loss in a changing planet, pages 277--302
work page 2011
-
[8]
Maria Dornelas, Anne E. Magurran, Stephen T. Buckland, Anne Chao, Robin L. Chazdon, Robert K. Colwell, Tom Curtis, Kevin J. Gaston, Nicholas J. Gotelli, Matthew A. Kosnik, Brian McGill, Jenny L. McCune, Hélène Morlon, Peter J. Mumby, Lise Øvreås, Angelika Studeny, and Mark Vellend. 2013. Quantifying temporal change in biodiversity: challenges and opportun...
work page 2013
Show all 28 references
-
[9]
Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. https://arxiv.org/abs/2007.01852 Language-agnostic bert sentence embedding
2022 arXiv
-
[10]
Thomas van Goethem and Jan Luiten van Zanden. 2021. Biodiversity trends in a historical perspective. In How Was Life? Volume II: New Perspectives on Well-being and Global Inequality since 1820. Organisation for Economic Co-Operation and Development (OECD)
2021
-
[11]
Sk Rezaul Hoque and Sk Rima Sultana. 2024. Addressing global environmental problems: Challenges, solutions, and opportunities. The Social Science Review: A Multidisciplinary Journal, 2(2):124--130
2024
-
[12]
Kelly J Iknayan, Morgan W Tingley, Brett J Furnas, and Steven R Beissinger. 2014. Detecting diversity: emerging methods to estimate species diversity. Trends in ecology & evolution, 29(2):97--106
2014
-
[13]
Kenneth G Johnson, Stephen J Brooks, Phillip B Fenberg, Adrian G Glover, Karen E James, Adrian M Lister, Ellinor Michel, Mark Spencer, Jonathan A Todd, Eugenia Valsami-Jones, Jeremy R Young, and John R Stewart. 2011. Climate change and biosphere response: Unlocking the collect...
2011
-
[14]
Mike Kestemont, Folgert Karsdorp, Elisabeth de Bruijn, Matthew Driscoll, Katarzyna A Kapitan, P \'a draig \'O Mach \'a in, Daniel Sawyer, Remco Sleiderink, and Anne Chao. 2022. Forgotten books: The application of unseen species models to the survival of culture. Science, 375(6...
2022
-
[15]
Svetlana Kiritchenko and Saif Mohammad. 2017. https://doi.org/10.18653/v1/P17-2074 Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (...
2017 doi
-
[16]
Vamsi Krishna Kommineni, Waqas Ahmed, Birgitta Koenig-Ries, and Sheeba Samuel. 2024. Automating information retrieval from biodiversity literature using large language models: A case study. Biodivers. Inf. Sci. Stand., 8
2024
-
[17]
Christian K \"o nig, Patrick Weigelt, Julian Schrader, Amanda Taylor, Jens Kattge, and Holger Kreft. 2019. Biodiversity data integration—the significance of data resolution and domain. PLoS biology, 17(3):e3000183
2019
-
[18]
Lars Langer, Manuel Burghardt, Roland Borgards, Ronny Richter, and Christian Wirth. 2024. The relation between biodiversity in literature and social and spatial situation of authors: Reflections on the nature--culture entanglement. People and Nature, 6(1):54--74
2024
-
[19]
Rensis Likert. 1932. A technique for the measurement of attitudes. Archives of Psychology
1932
-
[20]
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437
2024 arXiv
-
[21]
Andy L \"u cking, Christine Driller, Manuel Stoeckel, Giuseppe Abrami, Adrian Pachzelt, and Alexander Mehler. 2022. Multiple annotation for biodiversity: developing an annotation framework among biology, linguistics and text technology. Language resources and evaluation, 56(3)...
2022
-
[22]
Chen Qian, Fuli Feng, Lijie Wen, Chunping Ma, and Pengjun Xie. 2021. https://doi.org/10.18653/v1/2021.acl-long.422 Counterfactual inference for text classification debiasing . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11...
2021 doi
-
[23]
Malte Rehbein, Andrea Belen Escobari Vargas, Sarah Fischer, Anton Güntsch, Bettina Haas, Giada Matheisen, Tobias Perschl, Alois Wieshuber, and Thore Engel. 2024. https://doi.org/10.5281/zenodo.14008158 Historical animal observation records by bavarian forestry offices (1845): ...
2024 doi
-
[24]
why should i trust you?
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. " why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135--1144
2016
-
[25]
Howard Schuman and Stanley Presser. 1996. Questions and answers in attitude surveys: Experiments on question form, wording, and context. Sage
1996
-
[26]
Anders Telenius. 2011. Biodiversity information goes public: Gbif at your service. Nordic Journal of Botany, 29(3):378--381
2011
-
[27]
Zhao Wang and Aron Culotta. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.308 Identifying spurious correlations for robust text classification . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3431--3440, Online. Association for Computat...
2020 doi
-
[28]
1824--1886
Württembergische Oberamtsbeschreibungen . 1824--1886. https://de.wikisource.org/\ /Württembergische\_Oberamtsbeschreibungen Wikisource . Accessed: 30 Jan. 2025
2025
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.