Pith. sign in

REVIEW 3 major objections 8 minor 62 references

ATGen: A Framework for Active Text Generation

T0 review · 3 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Active learning lets text-generation models match random-sampling quality with about a third of the annotated data, across four NLG tasks and with human or LLM annotators.

desk verdict Solid AL-for-NLG systems contribution with real code and believable benchmarks, but the headline effort/cost savings are inferred from instance counts, not measured. read the letter →

arxiv 2506.23342 v1 pith:EYSC544K submitted 2025-06-29 cs.CL cs.AI

Akim Tsvigun , Daniil Vasilev , Ivan Tsvigun , Ivan Lysenko , Talgat Bektleuov , Aleksandr Medvedev , Uliana Vinogradova , Nikita Severin
show 7 more authors
Mikhail Mozikov Andrey Savchenko Rostislav Grigorev Ramil Kuleev Fedor Zhdanov Artem Shelmanov Ilya Makarov
This is my paper · ORCID
classification cs.CLcs.AI
keywords activelearningtextgenerationnaturallanguageannotationcostLLMbenchmarkframeworkexperimentaldesignefficientfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Active learning has proven itself for classification, but its application to text generation has lagged. This paper claims that active learning transfers to text generation and backs the claim with benchmarks on open-domain question answering, math reasoning, reading comprehension, and summarization, where the HUDS, HADAS, and Facility Location strategies reach the same model quality with roughly a third of the annotated data that random sampling needs. The vehicle is ATGen, a framework that makes the full active-learning loop—fine-tune, score the unlabeled pool, select a batch, label it by a human or an LLM, retrain—easy to run and compare. The authors conclude that active learning can cut human annotation effort and, in the automatic regime, reduce LLM API costs by 2–4x. The manual-annotation results are simulated with ground-truth labels rather than measured human effort, so the size of the human-side saving is conditional on that simulation.

What carries the argument

The engine is the active learning loop: the acquisition model, fine-tuned with low-rank adapters and served by a fast inference engine, scores the unlabeled pool, a query strategy picks a batch, the oracle labels it, and the model is retrained. Three strategies carry the empirical result. HUDS combines normalized negative log-likelihood uncertainty with distance to the center of an uncertainty stratum in a BERT sentence-embedding space. HADAS estimates hallucination susceptibility through semantic-frame, discourse, and content-verifiability checks. Facility Location is a selection rule that chooses a diverse, representative subset covering the unlabeled pool. ATGen's contribution is packaging this loop and these strategies into reusable software with human and LLM annotation paths, so that strategy comparisons are run under identical training, evaluation, and annotation conditions.

What would settle it

A decisive check is a human-annotation study: hire annotators to label the batches HUDS selects and the batches random sampling selects on the same task under the ATGen interface, recording wall-clock time and label accuracy, then retrain the acquisition model on each labeled set. If the active-learning curve does not reach equal end-task quality with less human time, the paper's human-effort claim fails; a parallel cost log of LLM annotation API spend plus local retraining cost would test the 2–4x saving directly.

Watch

Extended reading notes

Core claim

The central claim is that active learning strategies built for text generation—HUDS, HADAS, and Facility Location—consistently beat random sampling on the choice of which unlabeled texts to annotate next, in both simulated-manual and LLM-annotator settings. On TriviaQA, random sampling requires more than 12% of the dataset to match what these strategies reach with just 4%, about three times less data; on GSM8K, RACE, and AESLC the same strategies keep their edge, though LLM annotation on GSM8K lowers absolute quality for every method. Because the same end-task performance is reached with fewer labeled examples, the paper attributes to active learning a reduction in annotator effort and, for LLM-based annotation, a 2–4x reduction in API costs. ATGen is the enabling platform: a web GUI for human labeling, pluggable API or local LLM labelers, a common set of implemented strategies and metrics, and benchmark scripts that make the comparison fair and reproducible.

Load-bearing premise

The load-bearing premise is that simulated annotation—assigning ground-truth outputs to selected instances—stands in faithfully for real human annotation; since the paper measures no human time, cost, or label quality, the claim that active learning reduces human effort is supported only if that simulation transfers to actual annotator work.

Editorial extensions

If this is right

  • On the four tested tasks, using HUDS, HADAS, or Facility Location instead of random sampling saves roughly two-thirds of the annotation budget at equal model quality.
  • In a fully automatic pipeline, fewer LLM calls at equal quality translate to a 2–4x reduction in API cost, with local retraining of a small model counted as effectively free.
  • LLM annotation is not a full substitute for humans in specialized domains: on GSM8K, the LLM annotator's labeling errors dragged down every strategy's absolute quality.
  • ATGen gives researchers a single interface to benchmark new active learning strategies, so the field can move beyond a small set of hand-built methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the simulated manual setup resembles real annotators, the savings are in wall-clock effort; but actual annotation speed also depends on interface design, task complexity, and annotator fatigue, none of which the benchmark measures.
  • The ranking of strategies may shift with the acquisition model, since uncertainty estimates come from the model's own predictions; running the same benchmark with larger or domain-adapted acquisition models is a direct way to test the stability of the result.
  • Because LLM annotators make errors that accumulate over active learning iterations, a natural extension is noise-aware selection—choosing instances that are informative and likely to be labeled correctly—which the paper does not explore.
  • The paper recommends experimental-design strategies for human annotation because they avoid iterative latency, but it does not directly compare that human-in-the-loop wall-clock tradeoff; a timing study would decide which regime is actually cheaper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. ATGen is a framework for active learning (AL) in natural language generation (NLG), supporting human and LLM-based annotation, parameter-efficient fine-tuning, efficient inference, a web GUI, and a benchmark of AL strategies. The authors evaluate seven AL/ED strategies against random sampling on TriviaQA, GSM8K, RACE, and AESLC, using Qwen3-1.7B as the acquisition model and both ground-truth oracle and DeepSeek-R1 annotation. They find that HUDS, HADAS, and Facility Location reach target quality with fewer annotated instances (about 3x fewer on TriviaQA) and claim that AL reduces human annotation effort and LLM API costs by 2-4x.

Significance. The framework addresses a real gap: there is no unified AL benchmarking platform for NLG with modern LLM training and inference support. The open-source release under MIT, the integration of PEFT (LoRA, QLoRA, DoRA) and inference engines (vLLM, SGLang, Unsloth), and the inclusion of both classic and recent AL strategies make it a useful infrastructure contribution. The empirical comparison in a unified setting is valuable, and the code availability improves reproducibility. If the quantitative claims are validated with direct cost and effort measurements, the paper would provide strong evidence for AL in the LLM era.

major comments (3)
  1. [Section 4.2 and Abstract] The claim that AL reduces LLM API call costs by 2-4x is inferred from the number of annotated instances without measuring tokens or dollar spend. Because AL-selected instances may differ in length and reasoning depth (especially with DeepSeek-R1), and because ATGen itself supports per-token pricing (Figure 8), the paper should report token counts and price-model-based cost curves, or explicitly restrict the claim to a reduction in the number of annotation calls.
  2. [Section 4.1.1 and Abstract] The claim that ATGen reduces human annotator effort is not directly supported. The simulated 'manual' scenario assigns ground-truth outputs, which is a standard oracle simulation but does not measure annotator time, cost, or quality. To support the human-effort claim, the authors should either run a small human annotation study or limit the claim to 'number of annotations' rather than 'effort.'
  3. [Section 4.1.1 and Figures 3-6] The text states that experiments are averaged over multiple random seeds, but the figures do not show error bars, confidence bands, or statistical significance. Without this information, the observed separation between AL strategies and random sampling cannot be distinguished from noise. Please add variance reporting (e.g., standard deviation shading) or significance tests.
minor comments (8)
  1. [References] The entries for Bhatt et al. 2024a and Bhatt et al. 2024b are identical and should be consolidated into one reference.
  2. [Section 4.2] The phrase '2-4x' is not derived from any specific table or comparison in the presented results; specify the data points that support this range.
  3. [Limitations] The Limitations section does not mention that the headline savings are based on instance counts rather than measured effort or cost; add this caveat.
  4. [Figures 3 and 4] The legend includes the unnecessary variable name 'variable' in all panels; clean up the plot legends.
  5. [Introduction and Related Work] The introduction states that AL reduces annotations by '3-5 times' while Related Work says '3-6 times'; make the range consistent.
  6. [References] The reference list entry for 'OpenAI. Introducing o3 and o4-mini' lacks a complete citation; please provide a public URL or arXiv identifier.
  7. [Section 3.1.3] The phrase 'synchronous analogue' is slightly unclear; consider 'synchronous counterpart'.
  8. [Section 1] The novelty claim of being the first to combine PEFT and vLLM for AL would be stronger with a brief comparison to existing tools that partially support such features.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the benchmark results are self-contained empirical measurements.

full rationale

ATGen is a systems and benchmarking paper. Its central empirical claim is that existing AL strategies (HUDS, HADAS, Facility Location, and IDDS) reach a target quality level with fewer annotated instances than random sampling on four public benchmarks. The derivation chain is empirical: Section 4.1.1 specifies the simulated oracle protocol, Section 4.2 reports learning curves, and the 3x/2-4x reduction figures are read off those curves. No parameter is fitted and then renamed as a prediction; the AL strategies are pre-existing methods cited from prior work (Azeemi et al., Xia et al., Bhatt et al., Tsvigun et al.) rather than derived inside this paper. The self-citations to ALToolbox and IDDS concern software and a previously published strategy, but the benchmark results are computed on public datasets and do not depend on accepting those self-citations as theorems. The ground-truth simulation is the standard AL evaluation protocol and is not defined in terms of the paper's conclusions. The only weakness is that the abstract's wording about 'effort of human annotators' and 'costs associated with API calls' goes beyond what is measured—no annotator time, dollar spend, or token counts are logged—but that is an unsupported inference about the external validity of the results, not a circular derivation. The quality-versus-data-volume finding itself remains independent of that inference, so no circular step is exhibited.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

ATGen is a software framework, not a new theoretical entity. No new particles, forces, or conserved quantities are proposed. The framework's components are existing libraries and published AL strategies, and the central claim rests on the experimental assumptions listed above.

assumptions (4)
  • domain assumption Ground-truth labels simulate human annotation accurately enough to estimate annotation effort.
    Section 4.1.1: experiments 'emulate the AL annotation cycle' by assigning ground-truth outputs; no human time or cost is measured, yet the abstract claims reduced human effort.
  • domain assumption DeepSeek-R1 is a representative LLM annotation oracle.
    Section 4.1.1: LLM-based annotation experiments use DeepSeek-R1 for all LLM-labeled runs; the cost-saving estimates depend on this oracle's quality and pricing.
  • domain assumption Qwen3-1.7B is a representative acquisition model for NLG active learning.
    Section 4.1.2: all benchmark runs use Qwen/Qwen3-1.7B; no evidence is provided that results transfer to other model sizes or families.
  • domain assumption The implemented AL strategies faithfully represent their published versions.
    Section 3.1.1 states ATGen implements the strategies; the paper does not provide per-strategy implementation details or tests against reference implementations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ATGen: A Framework for Active Text Generation." pith.science (2026). https://pith.science/paper/EYSC544K

@misc{pith2026250623342,
  author       = {Pith},
  title        = {Pith review of: ATGen: A Framework for Active Text Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EYSC544K}},
  note         = {Machine review of arXiv:2506.23342}
}
read the original abstract

Active learning (AL) has demonstrated remarkable potential in reducing the annotation effort required for training machine learning models. However, despite the surging popularity of natural language generation (NLG) tasks in recent years, the application of AL to NLG has been limited. In this paper, we introduce Active Text Generation (ATGen) - a comprehensive framework that bridges AL with text generation tasks, enabling the application of state-of-the-art AL strategies to NLG. Our framework simplifies AL-empowered annotation in NLG tasks using both human annotators and automatic annotation agents based on large language models (LLMs). The framework supports LLMs deployed as services, such as ChatGPT and Claude, or operated on-premises. Furthermore, ATGen provides a unified platform for smooth implementation and benchmarking of novel AL strategies tailored to NLG tasks. Finally, we present evaluation results for state-of-the-art AL strategies across diverse settings and multiple text generation tasks. We show that ATGen reduces both the effort of human annotators and costs associated with API calls to LLM-based annotation agents. The code of the framework is available on GitHub under the MIT license. The video presentation is available at http://atgen-video.nlpresearch.group

Figures

Figures reproduced from arXiv: 2506.23342 by the authors.

Figure 1
Figure 1. The ATGen configuration form to launch ⚠️ [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 3
Figure 3. Performance of AL selection strategies on the TriviaQA dataset with different annotation sources. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Performance of AL selection strategies on the GSM8K dataset with different annotation oracles. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Performance of AL and ED strategies with emulation of “manual” labeling on AESLC in terms of the [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Performance of AL and ED strategies with emulation of “manual” labeling on the RACE dataset. [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: A Bash command example to benchmark a AL strategy with the name “STRATEGY_NAME”. [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Advanced Bash code example to benchmark the strategy “huds” on the dataset “TriviaQA”, annotating 1% [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 27 canonical work pages

  1. [1]

    Anthropic . 2025. https://www.anthropic.com/news/claude-4 Introducing Claude 4

    work page 2025

  2. [2]

    Abdul Hameed Azeemi, Ihsan Ayyub Qazi, and Agha Ali Raza. 2025. https://aclanthology.org/2025.coling-main.206/ To label or not to label: Hybrid active learning for neural machine translation . In Proceedings of the 31st International Conference on Computational Linguistics, pages 3071--3082, Abu Dhabi, UAE. Association for Computational Linguistics

    2025

  3. [3]

    Nathan Beck, Suraj Kothawade, Durga Sivasubramanian, Apurva Dani, Rishabh Iyer, and Ganesh Ramakrishnan. 2021. Distil: Deep diversified interactive learning. https://github.com/decile-team/distil

    work page 2021

  4. [5]

    Truong, Stephen Mussmann, Yinglun Zhu, Jeffrey A

    Gantavya Bhatt, Yifang Chen, Arnav Mohanty Das, Jifan Zhang, Sang T. Truong, Stephen Mussmann, Yinglun Zhu, Jeffrey A. Bilmes, Simon S. Du, Kevin G. Jamieson, Jordan T. Ash, and Robert D. Nowak. 2024 b . https://doi.org/10.48550/ARXIV.2401.06692 An experimental design framework for label-efficient supervised finetuning of large language models . CoRR, abs...

    arXiv doi:10.48550/arxiv.2401.06692 2024

  5. [6]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

  6. [7]

    Vila-Suero Daniel and Aranda Francisco. 2023. https://github.com/argilla-io/argilla Argilla - Open-source framework for data-centric NLP

    work page 2023

  7. [8]

    Michael Han Daniel Han and Unsloth team. 2023. http://github.com/unslothai/unsloth Unsloth

    2023

  8. [9]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai D...

    arXiv 2025

Show all 62 references
  1. [10]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/1feb87871436031bdc0f2beaa62a049b-Abstract-Conference.html Qlora: Efficient finetuning of quantized llms . In Advances in Neural Information Processing S...

    2023

  2. [11]

    Jonas Golde, Patrick Haller, Felix Hamborg, Julian Risch, and Alan Akbik. 2023. https://doi.org/10.18653/V1/2023.EMNLP-DEMO.1 Fabricator: An open source toolkit for generating labeled training data with teacher llms . In Proceedings of the 2023 Conference on Empirical Methods ...

    2023 doi

  3. [12]

    Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2023. https://doi.org/10.18653/v1/2023.acl-long.806 Unnatural instructions: Tuning language models with (almost) no human labor . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics...

    2023 doi

  4. [13]

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pages 2790--2799. PMLR

    2019

  5. [14]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lora: Low-rank adaptation of large language models . In The Tenth International Conference on Learning Representat...

    2022

  6. [15]

    Junjie Hu and Graham Neubig. 2021. https://aclanthology.org/2021.wmt-1.117 Phrase-level active learning for neural machine translation . In Proceedings of the Sixth Conference on Machine Translation, pages 1087--1099, Online. Association for Computational Linguistics

    2021

  7. [16]

    Kuan-Hao Huang. 2021. Deepal: Deep active learning in python. arXiv preprint arXiv:2111.15258

    2021 arXiv

  8. [17]

    Human Signal . 2023. Adala: A framework for autonomous data labeling agents. https://github.com/HumanSignal/Adala

    2023

  9. [18]

    Jeffrey Ip and Kritin Vongthongsri. 2025. https://github.com/confident-ai/deepeval deepeval

    2025

  10. [19]

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. https://doi.org/10.18653/v1/P17-1147 T rivia QA : A large scale distantly supervised challenge dataset for reading comprehension . In Proceedings of the 55th Annual Meeting of the Association for Computational...

    2017 doi

  11. [20]

    Jan-Christoph Klie, Michael Bugert, Beto Boullosa, Richard Eckart de Castilho, and Iryna Gurevych. 2018. https://www.aclweb.org/anthology/C18-2002 The INCE p TION platform: Machine-assisted and knowledge-oriented interactive annotation . In Proceedings of the 27th Internationa...

    2018

  12. [21]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. https://doi.org/10.1145/3600006.3613165 Efficient memory management for large language model serving with pagedattention . In Proceedings of the 29...

    2023

  13. [22]

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. https://doi.org/10.18653/v1/D17-1082 RACE : Large-scale R e A ding comprehension dataset from examinations . In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages...

    2017 doi

  14. [23]

    Lewis and William A

    David D. Lewis and William A. Gale. 1994. https://doi.org/10.1007/978-1-4471-2099-5\_1 A sequential algorithm for training text classifiers . In Proceedings of the 17th Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval. Dublin, Irel...

    1994 doi

  15. [24]

    Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario S a s ko, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh,...

    2021 arXiv

  16. [25]

    Dongyuan Li, Ying Zhang, Zhen Wang, Shiyin Tan, Satoshi Kosugi, and Manabu Okumura. 2024. https://aclanthology.org/2024.findings-emnlp.523 Active learning for abstractive text summarization via llm-determined curriculum and certainty gain maximization . In Findings of the Asso...

    2024

  17. [26]

    Xu, Ouyu Lan, and Xiang Ren

    Bill Yuchen Lin, Dongho Lee, Frank F. Xu, Ouyu Lan, and Xiang Ren. 2019. Alpacatag: An active learning-based crowd annotation framework for sequence tagging. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), Demo Track

    2019

  18. [27]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  19. [28]

    Qiang Liu, Yanqiao Zhu, Zhaocheng Liu, Yufeng Zhang, and Shu Wu. 2021. https://doi.org/10.1145/3459637.3482080 Deep active learning for text classification with diverse interpretations . In CIKM '21: The 30th ACM International Conference on Information and Knowledge Management...

    2021

  20. [29]

    Shih - Yang Liu, Chien - Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu - Chiang Frank Wang, Kwang - Ting Cheng, and Min - Hung Chen. 2024. https://doi.org/10.48550/ARXIV.2402.09353 Dora: Weight-decomposed low-rank adaptation . CoRR, abs/2402.09353

    2024 doi arXiv

  21. [30]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.153 G -eval: NLG evaluation using gpt-4 with better human alignment . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language ...

    2023 doi

  22. [31]

    Katerina Margatina, Giorgos Vernikos, Lo \" c Barrault, and Nikolaos Aletras. 2021. https://doi.org/10.18653/V1/2021.EMNLP-MAIN.51 Active learning by acquiring contrastive examples . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMN...

    2021 doi

  23. [32]

    Mirchandani and Richard L

    Pitu B. Mirchandani and Richard L. Francis, editors. 1991. https://www.wiley.com/en-us/Discrete+Location+Theory-p-9780471892335 Discrete Location Theory . Wiley

    1991

  24. [33]

    Birger Mo \" e ll. 2024. https://doi.org/10.48550/ARXIV.2405.09300 Comparing the efficacy of GPT-4 and chat-gpt in mental health care: A blind assessment of large language models for psychological support . CoRR, abs/2405.09300

    2024 doi arXiv

  25. [34]

    Minh Van Nguyen, Nghia Ngo, Bonan Min, and Thien Nguyen. 2022. https://doi.org/10.18653/v1/2022.naacl-demo.14 FAMIE : A fast active learning framework for multilingual information extraction . In Proceedings of the 2022 Conference of the North American Chapter of the Associati...

    2022 doi

  26. [35]

    https://openai.com/index/introducing-o3-and-o4-mini/ Introducing o3 and o4-mini

    OpenAI . https://openai.com/index/introducing-o3-and-o4-mini/ Introducing o3 and o4-mini

  27. [36]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei - Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 Bleu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, ...

    2002

  28. [37]

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. https://arxiv.org/abs/2304.03277 Instruction tuning with gpt-4 . Preprint, arXiv:2304.03277

    2023 arXiv

  29. [38]

    Yotam Perlitz, Ariel Gera, Michal Shmueli - Scheuer, Dafna Sheinwald, Noam Slonim, and Liat Ein - Dor. 2023. https://doi.org/10.18653/V1/2023.EMNLP-MAIN.611 Active learning for natural language generation . In Proceedings of the 2023 Conference on Empirical Methods in Natural ...

    2023 doi

  30. [39]

    Nicholas Roy and Andrew McCallum. 2001. Toward optimal active learning through sampling estimation of error reduction. In Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001), Williams College, Williamstown, MA, USA, June 28 - July 1, 2001 , p...

    2001

  31. [40]

    o der, Lydia M \

    Christopher Schr \"o der, Lydia M \"u ller, Andreas Niekler, and Martin Potthast. 2023. https://doi.org/10.18653/v1/2023.eacl-demo.11 Small-text: Active learning for text classification in python . In Proceedings of the 17th Conference of the European Chapter of the Associatio...

    2023 doi

  32. [41]

    Christopher Schr \"o der, Andreas Niekler, and Martin Potthast. 2022. https://doi.org/10.18653/v1/2022.findings-acl.172 Revisiting uncertainty-based query strategies for active learning with transformers . In Findings of the Association for Computational Linguistics: ACL 2022,...

    2022 doi

  33. [42]

    Ozan Sener and Silvio Savarese. 2018. https://openreview.net/forum?id=H1aIuk-RW Active learning for convolutional neural networks: A core-set approach . In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Confe...

    2018

  34. [43]

    Burr Settles. 2009. Active learning literature survey. Computer Sciences Technical Report 1648, University of Wisconsin--Madison

    2009

  35. [44]

    Burr Settles and Mark Craven. 2008. https://www.aclweb.org/anthology/D08-1112/ An analysis of active learning strategies for sequence labeling tasks . In 2008 Conference on Empirical Methods in Natural Language Processing, EMNLP 2008, Proceedings of the Conference, 25-27 Octob...

    2008

  36. [45]

    Dylov, and Alexander Panchenko

    Artem Shelmanov, Dmitry Puzyrev, Lyubov Kupriyanova, Denis Belyakov, Daniil Larionov, Nikita Khromov, Olga Kozlova, Ekaterina Artemova, Dmitry V. Dylov, and Alexander Panchenko. 2021. https://doi.org/10.18653/V1/2021.EACL-MAIN.145 Active learning for sequence tagging with deep...

    2021 doi

  37. [46]

    Yanyao Shen, Hyokun Yun, Zachary Lipton, Yakov Kronrod, and Animashree Anandkumar. 2017. https://doi.org/10.18653/v1/W17-2630 Deep active learning for named entity recognition . In Proceedings of the 2nd Workshop on Representation Learning for NLP , pages 252--256, Vancouver, ...

    2017 doi

  38. [47]

    Akim Tsvigun, Ivan Lysenko, Danila Sedashov, Ivan Lazichny, Eldar Damirov, Vladimir Karlov, Artemy Belousov, Leonid Sanochkin, Maxim Panov, Alexander Panchenko, Mikhail Burtsev, and Artem Shelmanov. 2022 a . https://doi.org/10.18653/V1/2022.FINDINGS-EMNLP.377 Active learning f...

    2022 doi

  39. [48]

    Dylov, Igor Galitskiy, and Artem Shelmanov

    Akim Tsvigun, Leonid Sanochkin, Daniil Larionov, Gleb Kuzmin, Artem Vazhentsev, Ivan Lazichny, Nikita Khromov, Danil Kireev, Aleksandr Rubashevskii, Olga Shahmatova, Dmitry V. Dylov, Igor Galitskiy, and Artem Shelmanov. 2022 b . https://doi.org/10.18653/v1/2022.emnlp-demos.41 ...

    2022 doi

  40. [49]

    Akim Tsvigun, Artem Shelmanov, Gleb Kuzmin, Leonid Sanochkin, Daniil Larionov, Gleb Gusev, Manvel Avetisian, and Leonid Zhukov. 2022 c . https://doi.org/10.18653/v1/2022.findings-naacl.90 Towards computationally feasible deep active learning . In Findings of the Association fo...

    2022 doi

  41. [50]

    Nicola Ueffing and Hermann Ney. 2007. https://doi.org/10.1162/COLI.2007.33.1.9 Word-level confidence estimation for machine translation . Comput. Linguistics, 33(1):9--40

    2007 doi

  42. [51]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/v1/2023.acl-long.754 Self-instruct: Aligning language models with self-generated instructions . In Proceedings of the 61st Annual Mee...

    2023 doi

  43. [52]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \' e mi Louf, Morgan Funtowicz, and Jamie Brew. 2019. https://arxiv.org/abs/1910.03771 Huggingface's transformers: State-of-the-art natural language processin...

    2019 arXiv

  44. [53]

    Rossi, Anup B

    Yu Xia, Xu Liu, Tong Yu, Sungchul Kim, Ryan A. Rossi, Anup B. Rao, Tung Mai, and Shuai Li. 2024. https://doi.org/10.48550/ARXIV.2404.01588 Hallucination diversity-aware active learning for text summarization . CoRR, abs/2404.01588

    2024 doi arXiv

  45. [54]

    Xiao, Aidan N

    Tim Z. Xiao, Aidan N. Gomez, and Yarin Gal. 2020. https://arxiv.org/abs/2006.08344 Wat zei je? detecting out-of-distribution translations with variational transformers . CoRR, abs/2006.08344

    2020 arXiv

  46. [55]

    Michelle Yuan, Hsuan-Tien Lin, and Jordan Boyd-Graber. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.637 Cold-start active learning through self-supervised language modeling . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)...

    2020 doi

  47. [56]

    Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. https://proceedings.neurips.cc/paper/2021/file/e4d2b6e6fdeca3e60e0f1a62fee3d9dd-Paper.pdf Bartscore: Evaluating generated text as text generation . In Advances in Neural Information Processing Systems, volume 34, pages 27263--...

    2021

  48. [57]

    Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. https://aclanthology.org/2023.acl-long.634 A lign S core: Evaluating factual consistency with a unified alignment function . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vo...

    2023

  49. [58]

    Tetreault

    Rui Zhang and Joel R. Tetreault. 2019. https://doi.org/10.18653/v1/p19-1043 This email could save your life: Introducing the task of email subject line generation . In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Ital...

    2019 doi

  50. [59]

    Weinberger, and Yoav Artzi

    Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with bert . In International Conference on Learning Representations

    2020

  51. [60]

    Yuekai Zhao, Haoran Zhang, Shuchang Zhou, and Zhihua Zhang. 2020. https://doi.org/10.18653/V1/2020.FINDINGS-EMNLP.162 Active learning approaches to enhancing neural machine translation: An empirical study . In Findings of the Association for Computational Linguistics: EMNLP 20...

    2020 doi

  52. [61]

    Gonzalez, Clark W

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark W. Barrett, and Ying Sheng. 2024. http://papers.nips.cc/paper\_files/paper/2024/hash/724be4472168f31ba1c9ac630f15dec8-Abstract...

    2024

  53. [62]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  54. [63]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.