REVIEW 4 major objections 5 minor 29 references
Extracting Information in a Low-resource Setting: Case Study on Bioinformatics Workflows
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A small annotated corpus of 52 bioinformatics articles is enough to train a named-entity recognizer that extracts workflow components at an F-measure of 70.4, close to the level of agreement between human annotators.
desk verdict BioToFlow is a genuine resource with solid baselines, but the 'comparable to IAA' claim is unfalsifiable as reported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is BioToFlow, a manually annotated corpus of 52 bioinformatics articles (78,419 tokens, about 8% nested entities) built on a three-category schema of 16 entities: core workflow components such as tools, data, methods, and files; the execution environment; and contextual details such as versions, parameters, and bibliographic references. The extraction pipeline pairs the SciBERT language model with a biLSTM-CRF tagger that supports nested entities, trained and evaluated on five random splits with five random seeds. The other machinery pieces are an entity mapping from the SoftCite software-mention corpus to BioToFlow's schema, and a vocabulary-injection step that adds tool and binary names from bioinformatics tool registries into SciBERT's tokenizer.
What would settle it
Build a held-out set of workflow articles from venues and workflow systems outside the original PubMed query, such as Galaxy or Common Workflow Language, annotate them with the same 16-entity schema, and run the trained model; a large F1 drop would show that the corpus is not representative of bioinformatics workflows in general.
Extended reading notes
Core claim
The central claim is that high-performance named-entity recognition for bioinformatics workflows is achievable in a low-resource setting. Using the new BioToFlow corpus of 52 articles annotated with 16 types of workflow-related entities, a SciBERT encoder combined with a bidirectional LSTM-CRF sequence tagger reaches an average F-measure of 70.4% in relaxed evaluation, within the range of the inter-annotator agreement measured on the same corpus. The paper reports that few-shot prompting with an autoregressive language model reaches only 38.3% F1, that transferring the SoftCite software-mention corpus covers only a subset of entities, and that adding bioinformatics tool vocabulary improves Tool extraction from 74.8% to 77.0% while leaving the overall score essentially unchanged. The conclusion is that domain-specific annotated data and domain-adapted encoders matter more than generic few-shot ability or large external corpora.
Load-bearing premise
The articles in BioToFlow were chosen by a narrow PubMed search for Nextflow or Snakemake plus GitHub, and the test set comes from the same source; if workflow articles elsewhere use different vocabulary or entity distributions, the 70.4 F1 may not carry over.
Editorial extensions
If this is right
- With only 52 annotated articles, an encoder-based SciBERT NER model reaches 70.4 overall F1 in relaxed evaluation, matching the level of human annotator agreement.
- Transfer from the larger SoftCite software-mention corpus supports only the five of sixteen BioToFlow entity types that have schema counterparts, leaving Data, Container, Hardware, and Description without direct coverage.
- Fusing SoftCite in silver-annotated form gives a slight overall gain (70.7 versus 70.4), showing that the large external corpus adds marginal value rather than a breakthrough.
- Adding bioinformatics tool and binary names to the SciBERT vocabulary improves Tool F1 from 74.8 to 77.0 when the model is fine-tuned afterward, while other entities stay flat or decline.
- Few-shot prompting of an autoregressive language model lands below 40 F1, indicating that generative prompting is not competitive for this task at current model sizes.
Reading between the lines
- The recipe of building a small schema-specific annotated corpus and fine-tuning an encoder model likely transfers to other low-resource scientific subdomains where entity vocabularies are specialized and generic few-shot models underperform.
- If extraction truly operates at human-agreement levels, the structured output could feed automatic workflow registration in repositories, reducing manual curation effort; the paper points toward this application but does not build it.
- A natural next test is cross-corpus generalization: annotate workflow articles from other workflow systems or from sections outside Material and Methods and measure whether the 70.4 F1 holds outside the original PubMed query pool.
- Entity linking between extracted workflow components in text and components found in public code repositories is an untested extension that would turn these extractions into directly reusable workflow descriptions.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces BioToFlow, a manually annotated corpus of 52 bioinformatics workflow articles (26 Nextflow, 26 Snakemake) covering 16 entity types, and evaluates four extraction strategies: few-shot NER with Llama-3-8B-Instruct, encoder-based NER with SciBERT/NLStruct trained on SoftCite and on BioToFlow, fusion of the two corpora, and injection of bioinformatics tool vocabulary. The main quantitative result is a 70.4 overall F1 (relaxed) for SciBERT trained on BioToFlow, which the abstract states is 'comparable to inter-annotator agreement,' leading to the conclusion that high-performance information extraction for bioinformatics workflows is achievable.
Significance. The contribution is a valuable new public resource: the BioToFlow corpus, the annotation schema, and the released code provide a concrete benchmark for workflow-oriented NER in bioinformatics. The experimental design is solid in its use of five random splits and five random seeds with reported standard deviations, and the paper compares generative and encoder-based approaches systematically. If the central claim is confirmed, the paper demonstrates that a relatively small in-domain annotated corpus plus an encoder model can yield useful extraction performance for workflow entities, and that knowledge injection can help specific entity types such as Tool. The main qualifications concern the evidentiary support for the 'comparable to inter-annotator agreement' claim and the generality of the conclusion beyond the narrow corpus selection.
major comments (4)
- [Section 2.2] The manuscript never reports exact inter-annotator agreement values, only that 'IAA obtained were all higher than 70% in relaxed mode.' Consequently, the abstract's claim that the 70.4 F1 system is 'comparable to inter-annotator agreement' cannot be checked: 70.4 is consistent with any IAA value above 70. Please report exact relaxed and strict IAA values (overall and per entity), the number of pairwise annotator comparisons, and the range or confidence interval, and state whether the gold standard was adjudicated or taken from a single annotator. Because model F1 against a gold standard and annotator-versus-annotator F1 are not directly commensurable, the 'comparable' wording should be supported by a concrete comparison or replaced with a more modest statement.
- [Section 2.2 and Conclusion] The corpus was selected through a PubMed query requiring 'nextflow' or 'snakemake' in the title or abstract and 'github' anywhere, then restricted to the Material and Methods and Implementation sections, and the 13-article test set is drawn from the same pool. The broad conclusion in the abstract and Conclusion ('high-performance information extraction for bioinformatics workflows is achievable') therefore rests on the untested assumption that this narrow pool represents bioinformatics workflow articles generally. Please either temper the claim to Nextflow/Snakemake workflow articles or add an external test set with other workflow systems, venues, or article sections to assess generalization.
- [Section 4.1 and Table 5] The text states that the SoftCite-trained model is tested on 13 BioToFlow articles, but Table 5 appears to report performance on the SoftCite test set (232 articles), and no BioToFlow test results for the SoftCite-only model are given. Please clarify what evaluation Table 5 actually reports, and provide the cross-corpus results on the 13 BioToFlow test articles if they exist, because those results are the direct evidence for transfer from the large corpus.
- [Section 4.3] In the 'silver' SoftCite configuration, each SoftCite article is annotated using predictions from 'a model trained on BioToFlow,' but the paper does not state whether that model was trained on the same 39-article training splits used in the fusion experiments or on the full BioToFlow corpus. If the full corpus, including the 13 test articles, was used to generate silver labels, then the fusion results on the BioToFlow test set are leakage-prone. Please specify the training procedure for the silver-label model and, if necessary, rerun the experiment using a model trained only on the training splits.
minor comments (5)
- [Section 4.1] The phrase 'significantly better' is used without any significance test; given the overlapping standard deviations in Tables 5 and 6, please add paired significance tests or replace 'significantly' with a descriptive comparison.
- [Table 3] The entity mapping table contains a duplicate row ('url Biblio' appears twice) and maps SoftCite 'environment' to both 'Tool' and 'Environment'; please reconcile the table to avoid ambiguity in the mapping procedure.
- [Tables 5 and 6] The term 'Overall-focused' is used without a definition; please state which subset of entities is included in this focused score.
- [Abstract] The phrase 'informationschema' in the abstract lacks a space; this appears to be a formatting artifact and should be corrected.
- [Throughout] The manuscript alternates between 'Github' and 'GitHub'; please standardize the capitalization.
Circularity Check
No significant circularity: the BioToFlow NER result comes from a standard held-out train/test evaluation, and the self-citations are not load-bearing; the only concern is an imprecise IAA comparison, not a circular one.
full rationale
Walking the derivation chain, the central result (70.4 F1 from SciBERT/NLStruct on BioToFlow) comes from a standard supervised train/test split: 75% of the corpus is used for training and validation, 25% is held out for testing, with five random splits and five random seeds. No equation in the paper re-derives this number from its own input, and the test articles are not used to fit any parameter. The self-citations ([22] for the preliminary corpus and [18] for choosing Llama-3-8B-Instruct) are descriptive or methodological choices, not results that the paper's conclusion is forced to assume. The 'silver SoftCite' experiment annotates an external corpus with a BioToFlow-trained model and then trains on the merged data; this is self-training, not a circular derivation, because the final evaluation is still on held-out BioToFlow test articles. The abstract's 'comparable to inter-annotator agreement' is weakly supported, since Section 2.2 only reports that all IAA values are above 70% in relaxed mode, without exact scores or ranges, but this is an under-specified comparison rather than a claim that reduces by construction to its own input. Overall, no significant circularity was found.
Assumptions & free parameters
free parameters (3)
- learning_rate =
1e-3
- sequence_length =
256
- train_iterations =
4000
assumptions (4)
- domain assumption The 16-entity workflow representation schema is an adequate and valid model of bioinformatics workflows.
- domain assumption The PubMed query ((nextflow OR snakemake) AND github) selects a representative sample of bioinformatics workflow articles.
- domain assumption F-measure is an appropriate equivalence to kappa for inter-annotator agreement on named entities.
- domain assumption SciBERT's tokenizer and embeddings are suitable for scientific text, and the added vocabulary tokens are a valid way to represent bioinformatics tool names.
Cite this review
Pith. "Pith review of Extracting Information in a Low-resource Setting: Case Study on Bioinformatics Workflows." pith.science (2026). https://pith.science/paper/TZNTAOBJ
@misc{pith2026241119295,
author = {Pith},
title = {Pith review of: Extracting Information in a Low-resource Setting: Case Study on Bioinformatics Workflows},
year = {2026},
howpublished = {\url{https://pith.science/paper/TZNTAOBJ}},
note = {Machine review of arXiv:2411.19295}
}
read the original abstract
Bioinformatics workflows are essential for complex biological data analyses and are often described in scientific articles with source code in public repositories. Extracting detailed workflow information from articles can improve accessibility and reusability but is hindered by limited annotated corpora. To address this, we framed the problem as a low-resource extraction task and tested four strategies: 1) creating a tailored annotated corpus, 2) few-shot named-entity recognition (NER) with an autoregressive language model, 3) NER using masked language models with existing and new corpora, and 4) integrating workflow knowledge into NER models. Using BioToFlow, a new corpus of 52 articles annotated with 16 entities, a SciBERT-based NER model achieved a 70.4 F-measure, comparable to inter-annotator agreement. While knowledge integration improved performance for specific entities, it was less effective across the entire information schema. Our results demonstrate that high-performance information extraction for bioinformatics workflows is achievable.
Figures
Reference graph
Works this paper leans on
-
[1]
Com- putational linguistics34(4), 555–596 (2008)
Artstein, R., Poesio, M.: Inter-coder agreement for computational linguistics. Com- putational linguistics34(4), 555–596 (2008)
work page 2008
-
[2]
In: Inui, K., Jiang, J., Ng, V., Wan, X
Beltagy, I., Lo, K., Cohan, A.: SciBERT: A Pretrained Language Model for Scientific Text. In: Inui, K., Jiang, J., Ng, V., Wan, X. (eds.) Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 3615–3620. Association for Computation...
-
[3]
https://doi.org/10.48550/arXiv.2305.16326, http://arxiv.org/abs/2305.16326
Chen, Q., Hu, Y., Peng, X., Xie, Q., Jin, Q., Gilson, A., Singer, M.B., Ai, X., et al.: A systematic evaluation of large language models for biomedical natural language processing: benchmarks, baselines, and recommendations (Sep 2024). https://doi.org/10.48550/arXiv.2305.16326, http://arxiv.org/abs/2305.16326
-
[4]
Future Generation Computer Systems75, 284–298 (2017)
Cohen-Boulakia, S., Belhajjame, K., Collin, O., Chopard, J., Froidevaux, C., et al.: Scientific workflows for computational reproducibility in the life sciences: Status, challenges and opportunities. Future Generation Computer Systems75, 284–298 (2017)
work page 2017
-
[5]
Nature Biotech35, 316–319 (Apr 2017)
Di Tommaso, P., Chatzou, M., Floden, E.W., Barja, P., Palumbo, E., Notredame, C.: Nextflow enables reproducible computational workflows. Nature Biotech35, 316–319 (Apr 2017). https://doi.org/10.1038/nbt.3820
doi:10.1038/nbt.3820 2017
-
[6]
Du, C., Cohoon, J., Lopez, P., Howison, J.: Softcite dataset: A dataset of soft- ware mentions in biomedical and economic research publications. Journal of the Association for Information Science and Technology72(7), 870–884 (2021), https://onlinelibrary.wiley.com/doi/abs/10.1002/asi.24454 Information Extraction in Bioinformatics Workflows 13
-
[7]
Wiley-ISTE (Jul 2016), https://hal.archives-ouvertes
Fort, K.: Collaborative Annotation for Reliable Natural Language Processing: Tech- nical and Sociological Aspects. Wiley-ISTE (Jul 2016), https://hal.archives-ouvertes. fr/hal-01324322
work page 2016
-
[8]
Fort, K., Nazarenko, A., Rosset, S.: Modeling the complexity of manual annotation tasks: a grid of analysis. In: Kay, M., Boitet, C. (eds.) Proceedings of COLING
Show all 29 references
-
[9]
In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track
Gee, L., Zugarini, A., Rigutini, L., Torroni, P.: Fast vocabulary transfer for language model compression. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track. pp. 409–416. Association for Computational Linguistics, Abu Dh...
2022
-
[10]
In: Proc
Grouin, C., Rosset, S., Zweigenbaum, P., Fort, K., Galibert, O., Quintard, L.: Pro- posal for an extension of traditional named entities: From guidelines to evaluation, an overview. In: Proc. of Linguistic Annotation Workshop (LAW-V). pp. 92–100. Portland, OR (23-24 june 2011)...
2011
-
[11]
Nature Methods15(7), 475–476 (Jul 2018)
Grüning, B., Dale, R., Sjödin, A., Chapman, B.A., Rowe, J., Tomkins-Tinch, C.H., Valieris, R., Köster, J.: Bioconda: sustainable and comprehensive software distribu- tion for the life sciences. Nature Methods15(7), 475–476 (Jul 2018). https://doi.org/ 10.1038/s41592-018-0046-7...
2018 doi
-
[12]
In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Hong, J., Kim, T., Lim, H., Choo, J.: AVocaDo: Strategy for Adapting Vocabulary to Downstream Domain. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 4692–4700. Association for Computational Linguistics, Online and Punta Cana, Do...
2021 doi
-
[13]
Nucleic Acids Research44(D1), D38–D47 (Jan 2016)
Ison, J., Rapacki, K., Ménager, H., Kalaš, M., Rydza, E., Chmura, P., Anthon, C., Beard, N., Berka, K., Bolser, D., Booth, T., Bretaudeau, A., Brezovsky, J., Casadio, R., Cesareni, G., Coppens, F., Cornell, M., Cuccuru, G., et al.: Tools and data services registry: a community...
2016 doi
-
[14]
Istrate, A.M., Li, D., Taraborelli, D., Torkar, M., Veytsman, B., Williams, I.: A large dataset of software mentions in the biomedical literature (Sep 2022), http://arxiv.org/abs/2209.00693
2022 arXiv
-
[15]
BMC Bioinformatics21(1), 257 (Jun 2020)
Kieser, S., Brown, J., Zdobnov, E.M., Trajkovski, M., McCue, L.A.: ATLAS: a Snakemake workflow for assembly, annotation, and genomic binning of metagenome sequence data. BMC Bioinformatics21(1), 257 (Jun 2020). https://doi.org/10. 1186/s12859-020-03585-4, https://doi.org/10.11...
2020 doi
-
[16]
Bioinformatics (Oxford, England)28, 2520–2 (Aug 2012)
Köster, J., Rahmann, S.: Snakemake - a scalable bioinformatics workflow engine. Bioinformatics (Oxford, England)28, 2520–2 (Aug 2012). https://doi.org/10.1093/ bioinformatics/bts480
2012
-
[17]
arXiv preprint arXiv:2112.00405 (2021)
Liu, Z., Jiang, F., Hu, Y., Shi, C., Fung, P.: NER-BERT: a pre-trained model for low-resource entity tagging. arXiv preprint arXiv:2112.00405 (2021)
2021 arXiv
-
[18]
In: Al-Onaizan, Y., Bansal, M., Chen, Y.N
Naguib, M., Tannier, X., Névéol, A.: Few-shot clinical entity recognition in English, French and Spanish: masked language models outperform generative model prompting. In: Al-Onaizan, Y., Bansal, M., Chen, Y.N. (eds.) Findings of the Association for Computational Linguistics: ...
2024
-
[19]
Briefings in bioinformatics22, 146–163 (Dec 2019)
Neves, M., Ševa, J.: An extensive review of tools for manual annotation of documents. Briefings in bioinformatics22, 146–163 (Dec 2019). https://doi.org/10.1093/bib/ bbz130
2019 doi
-
[20]
NLM: MEDLINE, PubMed, and PMC (PubMed Central): How are they different? (2021), https://www.nlm.nih.gov/bsd/difference.html
2021
-
[21]
Transactions of the Association for Computational Linguistics11, 1132–1146 (Sep 2023)
Pan, H., Zhang, Q., Dragut, E., Caragea, C., Latecki, L.J.: DMDD: A Large- Scale Dataset for Dataset Mentions Detection. Transactions of the Association for Computational Linguistics11, 1132–1146 (Sep 2023). https://doi.org/10.1162/ tacl_a_00592, https://doi.org/10.1162/tacl_a_00592
2023 doi
-
[22]
In: Balaguer, M., Bendahman, N., Ho-dac, L.M., Mauclair, J., G Moreno, J., Pinquier, J
Sebe, C., Cohen-Boulakia, S., Ferret, O., Névéol, A.: Extraction d‘entités nommées décrivant des chaînes de traitement bioinformatiques dans des articles scientifiques en anglais. In: Balaguer, M., Bendahman, N., Ho-dac, L.M., Mauclair, J., G Moreno, J., Pinquier, J. (eds.) Ac...
2024
-
[23]
In: Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics
Stenetorp, P., Pyysalo, S., Topic, G., Ohta, T., Ananiadou, S., Tsujii, J.: brat: a Web-based Tool for NLP-Assisted Text Annotation. In: Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics. pp. 102–1...
2012
-
[24]
Bioinformatics33(16), 2580–2582 (Aug 2017)
da Veiga Leprevost, F., Grüning, B.A., Alves Aflitos, S., Röst, H.L., Uszkoreit, J., Barsnes, H., Vaudel, M., et al.: BioContainers: an open-source and community- driven framework for software standardization. Bioinformatics33(16), 2580–2582 (Aug 2017). https://doi.org/10.1093...
2017 doi
-
[25]
Database : the journal of biological databases and curation2013, bat019 (Jan 2013)
Verspoor, K., Jimeno-Yepes, A., Cavedon, L., McIntosh, T., Herten-Crabb, A., Thomas, Z., Plazzer, J.P.: Annotating the biomedical literature for the Human Variome. Database : the journal of biological databases and curation2013, bat019 (Jan 2013). https://doi.org/10.1093/datab...
2013 doi
-
[26]
Thèse de doctorat, Sorbonne université (Dec 2021), https://www.theses.fr/2021SORUS541
Wajsbürt, P.: Extraction et normalisation d’entités simples et structurées dans les documents médicaux. Thèse de doctorat, Sorbonne université (Dec 2021), https://www.theses.fr/2021SORUS541
2021
-
[27]
PeerJ9, e10947 (Mar 2021), https://peerj.com/articles/10947
Yates, J.A.F., Lamnidis, T.C., Borry, M., Valtueña, A.A., Fagernäs, Z., Clayton, S., Garcia, M.U., Neukamm, J., Peltzer, A.: Reproducible, portable, and efficient ancient genome reconstruction with nf-core/eager. PeerJ9, e10947 (Mar 2021), https://peerj.com/articles/10947
2021
-
[28]
In: Duh, K., Gomez, H., Bethard, S
Zaratiana, U., Tomeh, N., Holat, P., Charnois, T.: GLiNER: Generalist model for named entity recognition using bidirectional transformer. In: Duh, K., Gomez, H., Bethard, S. (eds.) Proceedings of the 2024 Conference of the North American Chap- ter of the Association for Comput...
2024 doi
-
[2012]
pp. 895–910. The COLING 2012 Organizing Committee, Mumbai, India (Dec 2012), https://aclanthology.org/C12-1055
2012
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.