Pith. sign in

REVIEW 4 major objections 7 minor 47 references

Selecting 5%-10% of scientific texts is enough to match full-corpus LLM entity extraction.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

ALLabel combines diversity, similarity, and uncertainty sampling to pick the most useful texts for human annotation, reaching near-full-data NER accuracy with 5-10% of labels.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection A useful three-stage AL pipeline for ICL-based NER with plausible wins on scientific datasets, but the headline convergence/budget-reduction numbers rest on experiments that are not reported. the 4 major comments →

arxiv 2509.07512 v1 pith:VZKEU67W submitted 2025-09-09 cs.CL cs.AIcs.IR

ALLabel: Three-stage Active Learning for LLM-based Entity Recognition using Demonstration Retrieval

Zihan Chen , Lei Shi , Weize Wu , Qiji Zhou , Yue Zhang This is my paper
classification cs.CL cs.AIcs.IR
keywords active learningin-context learningnamed entity recognitiondemonstration retrievalscientific text miningannotation efficiencylarge language modelsuncertainty sampling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes ALLabel, a method for choosing which texts in a scientific corpus to annotate when the annotations will be used as in-context demonstrations for a large language model doing entity recognition. It claims that by splitting a fixed annotation budget into three stages—diversity, similarity, and uncertainty sampling—the selected demonstrations support entity extraction as accurate as using the entire annotated corpus, while costing only about 5% to 10% of the annotations. The experiments cover MOF synthesis, general materials, and chemical reactions, and the method is reported to beat random selection, core-set, perplexity, and BATCHER at every pool size tested. The practical payoff would be cheap, high-quality entity extraction for data-driven chemistry and materials research.

Core claim

ALLabel's central claim is that a retrieval corpus built from carefully chosen demonstrations can replace full-corpus annotation in LLM in-context entity recognition. The workflow assumes a budget M: it picks M/5 samples with a warm-start core-set rule to maximize diversity, then 3M/5 samples that are most useful as demonstrations for many queries using a sum-rank score over BM25 similarities, then M/5 samples by uncertainty-similarity—queries whose nearest demonstration is least similar are used to find the final batch. On CSD-MOFs, NC 2024 General, and USPTO, ALLabel reports F1 within 2 points of full-corpus extraction using 5.0%, 9.1%, and 8.0% of the data respectively, and it outperforms

What carries the argument

The mechanism is a budget-split pipeline ending in a human-annotated retrieval corpus used for k-shot in-context learning: M/5 samples by diversity, 3M/5 by similarity, M/5 by uncertainty. The diversity stage uses a warm-start core-set algorithm over a text-similarity matrix; the similarity stage scores each candidate by sumrank, a weighted rank sum of how often and how highly the sample would be retrieved as a demonstration for other queries; the uncertainty stage identifies weak test points whose nearest demonstration is least similar and resamples for them.

Load-bearing premise

The central claim depends on the unlabeled pool and the test queries being drawn from the same distribution, since every sample already has ground-truth annotation and is scored as a test query; on genuinely new scientific texts, the 5–10% budget figure may not hold.

What would settle it

Take a scientific NER dataset, split it into non-overlapping parts (for example, by publication or reaction family), run ALLabel's selection only on the training part, and evaluate on the held-out part with no ground-truth reuse. If the annotation fraction needed to come within 2 F1 of full-corpus performance exceeds about 15%, or if random selection of the same size matches ALLabel, the reported budget savings and selection advantage would be falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • At the same annotation budget, ALLabel reports 5.1–5.8 F1 points over random sampling across the three datasets, and about a 9.1% average annotation-budget reduction over the best alternative for matching full-corpus performance.
  • The deterministic selection removes run-to-run variance, unlike random and core-set baselines, making evaluations more reproducible.
  • The method transfers to other ICL tasks: the paper reports ALLabel surpassing baselines on paraphrase identification (MRPC, PAWS) and natural language inference (MNLI), suggesting active demonstration selection is task-general.
  • The selection order matters: diversity-first, then similarity, then uncertainty, with a 1:3:1 budget split, is reported best, indicating that coverage seeds the corpus before exploitation.
  • If the 5–10% result holds, domain scientists can build high-accuracy extraction pipelines from a few dozen to a few hundred expert-labeled examples instead of thousands.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's in-domain evaluation means its 5–10% budget figures may be optimistic for deployment; a held-out split from different journals or reaction families would test how much annotation is really needed.
  • Counting budget in samples rather than tokens may mis-rank cost; the authors acknowledge this, and token-priced budgets could favor short, easy-to-label texts.
  • The uncertainty stage uses retrieval similarity as a proxy for LLM confidence; models that expose token log-probabilities could make the final selection more direct.
  • A natural extension is iterative ALLabel: after the first round, use LLM predictions on unselected points to dynamically add demonstrations where accuracy is lowest.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes ALLabel, a three-stage active learning framework for selecting samples to annotate for LLM-based in-context learning (ICL) named entity recognition (NER) in scientific domains. The three stages are: diversity sampling (a warm-start core-set algorithm), similarity sampling (a sumrank heuristic over the retrieval similarity matrix), and uncertainty sampling (selecting weak test points based on retrieval similarity and applying similarity sampling on them). The selected samples form a ground-truth demonstration pool for ICL retrieval. Experiments on CSD-MOFs, NC 2024 General, and USPTO compare ALLabel with Random, Core-set, Perplexity, and BATCHER under pool sizes 10–60 using GPT-4o and DeepSeek-V3. The paper claims consistent gains over all baselines, convergence to near-full-corpus F1 with only 5–10% of the data annotated, and a 9.1% average annotation-budget reduction over the best alternative. Ablations investigate the contribution of each stage, sampling order, and stage proportion.

Significance. If the central claims hold, ALLabel offers a practical, deterministic recipe for reducing annotation costs in specialized-domain NER with LLMs. The paper addresses a real bottleneck—manual annotation of scientific text—and it evaluates on three domain datasets rather than only general-domain NER. The combination of diversity, similarity, and uncertainty within a single ICL-based active learning pipeline is a reasonable contribution, and the reported margins over baselines at large pool sizes are substantial. The paper also reports experiments with two LLMs and across additional NLP tasks, which strengthens generalizability. However, the empirical support is weakened by a major reporting gap (Table 3), by hyperparameter tuning on a dataset that later appears as a headline result, and by the absence of uncertainty estimates for the deterministic methods. These issues need to be addressed before the paper can be accepted.

major comments (4)
  1. [§5.2 / Table 3] Table 3 lists convergence proportions for all methods, but Table 2 only reports F1 for pool sizes up to 60. For CSD-MOFs, the claimed Random convergence proportion of 25.1% corresponds to ~175 samples; Core-set 18.3% (~127), Perplexity 14.3% (~100), and BATCHER 15.1% (~105). None of these values can be derived from Table 2. At pool size 60, Random reaches 89.6 vs. the full-dataset F1 of 94.4 (gap 4.8), Core-set 90.4 (gap 4.0), Perplexity 91.5 (gap 2.9), and BATCHER 91.0 (gap 3.4); only ALLabel (93.3, gap 1.1) is within the 2% threshold. No experiments or extrapolation procedure are described in the paper or appendices that would justify the baseline entries. Since the Section 5.2 sentence "ALLabel reduces annotation budget by 9.1% compared to the best alternative" is computed from Table 3, this claim is currently unsupported. The authors should report full F1-vs-pool-size curves up to th
  2. [§5.2 / Appendix D.4-D.5] The framework's free design parameters—the 1:3:1 stage split, the D-S-U order, and the x=3M/5 rule for similarity sampling—are selected by ablations on CSD-MOFs (Tables 11 and 12), and CSD-MOFs is then a headline dataset in Table 2. Thus the reported advantage on CSD-MOFs is not an unbiased out-of-sample estimate; it is the result of tuning on the same data used for evaluation. Although the same hyperparameters also yield gains on NC 2024 General and USPTO, which mitigates overfitting, the central claim for CSD-MOFs should be verified with hyperparameters fixed a priori, e.g., by using a development split, or by re-running the main comparison with a default split such as 1:1:1 to demonstrate sensitivity.
  3. [§5.1 / Tables 2, 9] ALLabel, Perplexity, and BATCHER are reported as single deterministic runs; only Random and Core-set have standard deviations across five runs. The claim that ALLabel "consistently outperforms all baselines" is therefore not accompanied by uncertainty estimates for the main comparisons. For example, at pool size 10 on CSD-MOFs, ALLabel scores 87.8 versus Core-set 86.1±0.7; at some pool sizes the differences are comparable to the reported standard deviations. Please provide variance estimates or paired significance tests for all methods, ideally with multiple random seeds for the LLM decoder, so that the consistency claim is statistically grounded.
  4. [§5.1 / Abstract claim] The experimental protocol is an in-domain simulation: all samples have expert annotations, and when a sample is used as a test query it is considered unlabeled. This is a standard way to evaluate active-learning selection, but it does not directly support the Abstract's strong claim that "selectively annotating only 5%-10% of the dataset with ALLabel can achieve performance comparable to the method annotating the entire dataset" in the motivating deployment scenario, where a new corpus or a shifted distribution must be labeled. The selection procedure uses only unlabeled features, so the method itself is not circular; however, the convergence percentages in Table 3 and the practical budget conclusion would be more convincing with a held-out evaluation (e.g., selection on a development corpus and evaluation on a held-out test set) or with an explicit statement that the claim is limited to
minor comments (7)
  1. [Table 3] Typo: "Comparsions" should be "Comparisons".
  2. [Eq. (2)] The notation x is used both as the similarity-stage sampling size and as a threshold. State explicitly that x=3M/5 in the default configuration and how x changes when the stage proportions are ablated in Appendix D.5.
  3. [Appendix B.2] The deduplication step is only described at the end of Appendix B.2, but it affects the dataset sizes reported in Table 1. Clarify in the main text or table caption that the sizes are post-deduplication.
  4. [Table 5] Minor typo: "sectpions" should be "sections"; the role description also appears truncated for MOFs.
  5. [Figure 2] The heat maps would benefit from a colorbar label and a statement of whether colors are scaled per subplot or globally; the current description is qualitative.
  6. [Appendix B.1] BM25 hyperparameters k1 and b are not reported. Providing these values would improve reproducibility.
  7. [Appendix D.3] Table 10 reports average scores across four NLP datasets but does not state pool sizes or number of runs. The main text says the pool size varies from 30 to 100; please match the table to that description.

Circularity Check

1 steps flagged

Hyperparameters tuned on CSD-MOFs are then reported as headline results on CSD-MOFs; the core selection loop itself is label-free.

specific steps
  1. fitted input called prediction [Section 6.3 / Appendices D.4-D.5 (hyperparameter ablations on CSD-MOFs) vs Section 5.2 (main results on CSD-MOFs)]
    "Additionally, we conduct ablation experiments on the sampling order and proportion of the three stages, which confirm that the sequence (D-S-U) and division proportion (1:3:1) employed by ALLabel are both optimal. ... Experimental results show that the 1:3:1 division proportion achieves the best performance across most pool sizes, which is adopted by ALLabel."

    The framework's configuration (D-S-U order and 1:3:1 split) is selected by maximizing F1 on CSD-MOFs in Appendices D.4 and D.5. Section 5.2 then uses CSD-MOFs as a headline dataset when claiming ALLabel 'consistently outperforms all baselines' and computing the 5.0% convergence proportion. The CSD-MOFs result is therefore a fitted outcome, not an independent prediction: the compared configuration was chosen on exactly this dataset and evaluation metric. The claim is not fully forced because the ablations compare only a few discrete configurations and ALLabel also wins on the two untuned datasets, but the headline consistency claim overstates independence for CSD-MOFs.

full rationale

The paper's selection machinery is not circular in the equation-level sense: the diversity, similarity, and uncertainty-similarity stages operate on text similarity (BM25) and never on the ground-truth entity F1 used for evaluation. The uncertainty-similarity proxy is validated empirically on 50 samples per dataset (Appendix C), not assumed by definition. There is no self-citation chain or uniqueness theorem doing load-bearing work. The one genuine circularity concern is selection-on-test-set: the D-S-U order and 1:3:1 split are chosen via ablation on CSD-MOFs (Section 6.3, Appendices D.4-D.5), and CSD-MOFs is then reported as a primary dataset in Table 2 and used for the 5.0% convergence claim. That is a fitted-input-called-prediction pattern for that dataset. The other two datasets (NC 2024 General, USPTO) were not used for hyperparameter selection, so they provide independent evidence that the method has real content. Separately, Table 3's baseline convergence proportions (e.g., Random 25.1% on CSD-MOFs) imply pool sizes far above the reported maximum of 60 and are not supported by Table 2 or any described extrapolation; I treat that as a correctness/reporting gap, not a circularity, and it does not raise the circularity score. Overall score 4 reflects partial circularity in the CSD-MOFs headline result while acknowledging the independent evidence on the other datasets.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 0 invented entities

The paper introduces no new physical or ontological entities. Its load-bearing inputs are a set of modeling assumptions and tuned procedure parameters: similarity as a proxy for demonstration utility, the uncertainty-similarity proxy, the in-domain evaluation protocol, and the 1:3:1 budget split with D-S-U order selected on the evaluation dataset.

free parameters (7)
  • k (number of demonstrations per query) = 3
    Main experiments fix 3-shot ICL; ablations show k=5 performs slightly better at pool sizes >=20, so the choice is not forced by the data.
  • stage budget split (diversity:similarity:uncertainty) = 1:3:1
    Chosen via ablation on CSD-MOFs (Appendix D.5) and then applied to all main results.
  • stage order = D-S-U
    Chosen via ablation on CSD-MOFs (Appendix D.4); reported best on most pool sizes.
  • convergence tolerance = 2% F1 gap
    Post hoc threshold in Table 3 used to define 'comparable to full annotation' and to derive the 5%-10% claim.
  • x in sumrank formula = 3M/5
    Defines the number of top-ranked demonstrations that contribute to sumrank; tied to the 1:3:1 budget split.
  • weak query fraction in uncertainty stage = M/5
    Number of least-covered queries selected as weak test points; part of the 1:3:1 split.
  • BM25 hyperparameters k1 and b = not reported
    Formula in Eq. 3 includes these standard parameters but their values are not given, so exact similarity scores cannot be reproduced.
axioms (5)
  • domain assumption Text similarity is a valid proxy for demonstration usefulness in ICL.
    All three stages rank samples by BM25 or Sentence-BERT similarity. Section 4.1 says distance is inversely correlated with similarity, but the paper does not directly validate that higher similarity maximizes extraction F1.
  • domain assumption Lower similarity between retrieved demonstration and query implies higher LLM uncertainty.
    Used in Section 4.3 and Appendix C to justify uncertainty sampling. The supporting experiment uses only 50 random queries per dataset and averaged similarity bins without error bars.
  • domain assumption In-domain evaluation is a valid measure of real annotation-selection value.
    Section 5.1 notes every sample is manually annotated and each sample is used as both a test query and a potential demonstration, so the reported gains are measured on the same distribution as the selection pool.
  • ad hoc to paper The sumrank heuristic in Eq. 2 ranks demonstration utility correctly.
    The piecewise linear and inverse-rank weights are introduced without derivation or external benchmark; ablations show the heuristic helps, but the specific weights are not grounded.
  • domain assumption GPT-4o with the provided prompt template is a reliable NER annotator.
    Full-corpus F1 scores of 85.8-94.4 in Table 1 support this for the three datasets, but exact prompt templates for NC 2024 General and USPTO are not shown.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of ALLabel: Three-stage Active Learning for LLM-based Entity Recognition using Demonstration Retrieval." pith.science (2026). https://pith.science/paper/VZKEU67W

@misc{pith2026250907512,
  author       = {Pith},
  title        = {Pith review of: ALLabel: Three-stage Active Learning for LLM-based Entity Recognition using Demonstration Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VZKEU67W}},
  note         = {Machine review of arXiv:2509.07512}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Many contemporary data-driven research efforts in the natural sciences, such as chemistry and materials science, require large-scale, high-performance entity recognition from scientific datasets. Large language models (LLMs) have increasingly been adopted to solve the entity recognition task, with the same trend being observed on all-spectrum NLP tasks. The prevailing entity recognition LLMs rely on fine-tuned technology, yet the fine-tuning process often incurs significant cost. To achieve a best performance-cost trade-off, we propose ALLabel, a three-stage framework designed to select the most informative and representative samples in preparing the demonstrations for LLM modeling. The annotated examples are used to construct a ground-truth retrieval corpus for LLM in-context learning. By sequentially employing three distinct active learning strategies, ALLabel consistently outperforms all baselines under the same annotation budget across three specialized domain datasets. Experimental results also demonstrate that selectively annotating only 5\%-10\% of the dataset with ALLabel can achieve performance comparable to the method annotating the entire dataset. Further analyses and ablation studies verify the effectiveness and generalizability of our proposal.

Figures

Figures reproduced from arXiv: 2509.07512 by Lei Shi, Qiji Zhou, Weize Wu, Yue Zhang, Zihan Chen.

Figure 1
Figure 1. Figure 1: ALLabel combines human annotation with LLM annotation in an active learning workflow, which consists [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: F1-scores of (a) ALLabel and (b) Random Sampling varying with demonstration pool size and number of [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: ALLabel’s 5-shot extraction performance on [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 31 canonical work pages

  1. [1]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  2. [2]

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, and 1 others. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1--53

    work page 2024

  3. [3]

    Aron Culotta and Andrew McCallum. 2005. Reducing labeling effort for structured prediction tasks. In AAAI, volume 5, pages 746--751

    work page 2005

  4. [4]

    John Dagdelen, Alexander Dunn, Sanghoon Lee, Nicholas Walker, Andrew S Rosen, Gerbrand Ceder, Kristin A Persson, and Anubhav Jain. 2024. Structured information extraction from scientific text with large language models. Nature Communications, 15(1):1418

    work page 2024

  5. [5]

    DeepSeek-AI. 2024. https://arxiv.org/abs/2412.19437 Deepseek-v3 technical report . Preprint, arXiv:2412.19437

    Pith/arXiv arXiv 2024

  6. [6]

    Shizhe Diao, Pengcheng Wang, Yong Lin, Rui Pan, Xiang Liu, and Tong Zhang. 2024. Active prompting with chain-of-thought for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 1330--1350

    work page 2024

  7. [7]

    Bill Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Third international workshop on paraphrasing (IWP2005)

    work page 2005

  8. [8]

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, and 1 others. 2022. A survey on in-context learning. arXiv preprint arXiv:2301.00234

    Pith/arXiv arXiv 2022

  9. [9]

    Meihao Fan, Xiaoyue Han, Ju Fan, Chengliang Chai, Nan Tang, Guoliang Li, and Xiaoyong Du. 2024. Cost-effective in-context learning for entity resolution: A design space exploration. In 2024 IEEE 40th International Conference on Data Engineering (ICDE), pages 3696--3709. IEEE

    work page 2024

  10. [10]

    Hila Gonen, Srini Iyer, Terra Blevins, Noah A Smith, and Luke Zettlemoyer. 2023. Demystifying prompts in language models via perplexity estimation. In The 2023 Conference on Empirical Methods in Natural Language Processing

    work page 2023

  11. [11]

    Bernal Jim \'e nez Guti \'e rrez, Nikolas McNeal, Clayton Washington, You Chen, Lang Li, Huan Sun, and Yu Su. 2022. Thinking about gpt-3 in-context learning for biomedical ie? think again. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4497--4512

    work page 2022

  12. [12]

    Mehran Kazemi, Sid Mittal, and Deepak Ramachandran. 2023. Understanding finetuning for factual knowledge extraction from language models. arXiv preprint arXiv:2301.11293

    Pith/arXiv arXiv 2023

  13. [13]

    Itay Levy, Ben Bogin, and Jonathan Berant. 2023. Diverse demonstrations improve in-context compositional generalization. In The 61st Annual Meeting Of The Association For Computational Linguistics

    work page 2023

  14. [14]

    Jiachang Liu, Dinghan Shen, Yizhe Zhang, William B Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for gpt-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100--114

    work page 2022

  15. [15]

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1--35

    work page 2023

  16. [16]

    Man Luo, Xin Xu, Zhuyun Dai, Panupong Pasupat, Mehran Kazemi, Chitta Baral, Vaiva Imbrasaite, and Vincent Y Zhao. 2023. Dr. icl: Demonstration-retrieved in-context learning. arXiv preprint arXiv:2305.14128

    Pith/arXiv arXiv 2023

  17. [17]

    Katerina Margatina, Timo Schick, Nikolaos Aletras, and Jane Dwivedi - Yu. 2023. Active learning principles for in-context learning with large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 5011--5034

    work page 2023

  18. [18]

    Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022 , pages 11048--11064

    work page 2022

  19. [19]

    Xuran Ming, Shoubin Li, Mingyang Li, Lvlong He, and Qing Wang. 2024. Autolabel: Automated textual data annotation method based on active learning and large language model. In Knowledge Science, Engineering and Management - 17th International Conference, KSEM 2024 , volume 14887, pages 400--411

    work page 2024

  20. [20]

    OpenAI. 2024. https://arxiv.org/abs/2410.21276 Gpt-4o system card . Preprint, arXiv:2410.21276

    Pith/arXiv arXiv 2024

  21. [21]

    Jane Pan, Tianyu Gao, Howard Chen, and Danqi Chen. 2023. What in-context learning "learns" in-context: Disentangling task recognition and task learning. In Findings of the Association for Computational Linguistics: ACL 2023 , pages 8298--8319

    work page 2023

  22. [22]

    Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics, 11:1316--1331

    2023

  23. [23]

    N Reimers. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084

    Pith/arXiv arXiv 2019

  24. [24]

    Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij B Gupta, Xiaojiang Chen, and Xin Wang. 2021. A survey of deep active learning. ACM computing surveys (CSUR), 54(9):1--40

    work page 2021

  25. [25]

    Stephen Robertson, Hugo Zaragoza, and 1 others. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval , 3(4):333--389

    2009

  26. [26]

    Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655--2671

    work page 2022

  27. [27]

    Erik Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142--147

    work page 2003

  28. [28]

    o der, Lydia M \

    Christopher Schr \"o der, Lydia M \"u ller, Andreas Niekler, and Martin Potthast. 2023. Small-text: Active learning for text classification in python. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 84--95

    work page 2023

  29. [29]

    Ozan Sener and Silvio Savarese. 2018. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations

    work page 2018

  30. [30]

    Burr Settles. 2009. Active learning literature survey

    work page 2009

  31. [31]

    Ard Snijders, Douwe Kiela, and Katerina Margatina. 2023. Investigating multi-source active learning for natural language inference. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2187--2209

    work page 2023

  32. [32]

    Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and 1 others. 2022. Selective annotation makes language models better few-shot learners. arXiv preprint arXiv:2209.01975

    Pith/arXiv arXiv 2022

  33. [33]

    Sarveswara Rao Vangala, Sowmya Ramaswamy Krishnan, Navneet Bung, Dhandapani Nandagopal, Gomathi Ramasamy, Satyam Kumar, Sridharan Sankaran, Rajgopal Srinivasan, and Arijit Roy. 2024. Suitability of large language models for extraction of high-quality chemical reaction dataset from patent literature. Journal of Cheminformatics, 16(1):131

    work page 2024

  34. [34]

    Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021. Want to reduce labeling cost? gpt-3 can help. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4195--4205

    2021

  35. [35]

    Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and 1 others. 2023. Larger language models do in-context learning differently. arXiv preprint arXiv:2303.03846

    Pith/arXiv arXiv 2023

  36. [36]

    Jiarong Wei, Yancong Lin, and Holger Caesar. 2024. Basal: Size-balanced warm start active learning for lidar semantic segmentation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 18258--18264

    work page 2024

  37. [37]

    Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2018, pages 1112--1122. Association for Computational Linguistics (ACL)

    work page 2018

  38. [38]

    Ruixuan Xiao, Yiwen Dong, Junbo Zhao, Runze Wu, Minmin Lin, Gang Chen, and Haobo Wang. 2023. Freeal: Towards human-free active learning in the era of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14520--14535

    work page 2023

  39. [39]

    Zhipeng Xu, Zhenghao Liu, Yibin Liu, Chenyan Xiong, Yukun Yan, Shuo Wang, Shi Yu, Zhiyuan Liu, and Ge Yu. 2024. Activerag: Revealing the treasures of knowledge via active learning. arXiv preprint arXiv:2402.13547

    arXiv 2024

  40. [40]

    Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu, and Lingpeng Kong. 2023. Compositional exemplars for in-context learning. In International Conference on Machine Learning, pages 39818--39833. PMLR

    work page 2023

  41. [41]

    Ruoyu Zhang, Yanzeng Li, Yongliang Ma, Ming Zhou, and Lei Zou. 2023. Llmaaa: Making large language models as active annotators. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13088--13103

    work page 2023

  42. [42]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, and 1 others. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068

    Pith/arXiv arXiv 2022

  43. [43]

    Wei Zhang, Qinggong Wang, Xiangtai Kong, Jiacheng Xiong, Shengkun Ni, Duanhua Cao, Buying Niu, Mingan Chen, Yameng Li, Runze Zhang, and 1 others. 2024. Fine-tuning large language models for chemical text mining. Chemical Science, 15(27):10600--10611

    work page 2024

  44. [44]

    Yuan Zhang, Jason Baldridge, and Luheng He. 2019. Paws: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1298--1308

    work page 2019

  45. [45]

    Zhiling Zheng, Ali H Alawadhi, Saumil Chheda, S Ephraim Neumann, Nakul Rampal, Shengchao Liu, Ha L Nguyen, Yen-hsu Lin, Zichao Rong, J Ilja Siepmann, and 1 others. 2023. Shaping the water-harvesting behavior of metal--organic frameworks aided by fine-tuned gpt models. Journal of the American Chemical Society, 145(51):28284--28295

    work page 2023

  46. [46]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  47. [47]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.