Pith. sign in

REVIEW 4 major objections 5 minor 60 references

Large Language Models in the Task of Automatic Validation of Text Classifier Predictions

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that LLM annotators can replace human annotators for validating text-classifier predictions without loss of quality, with a binary ensemble exceeding human accuracy at 65% coverage.

desk verdict A genuinely useful industrial recipe for LLM validation annotation with probability-based abstention, but the 'better than humans' headline rests on a comparison the paper's own protocol rigs. read the letter →

arxiv 2505.18688 v2 pith:Y3Z6RJON submitted 2025-05-24 cs.CL cs.AIcs.LG

Aleksandr Tsymbalov , Mikhail Khovrichev This is my paper
classification cs.CLcs.AIcs.LG
keywords largelanguagemodelstextannotationdataclassificationvalidationprob-approachabstentionthresholdretrieval-augmentedgenerationLoRAfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that large language models can take over the job of validating a text classifier's predictions in a production support-chat system, replacing or substantially supplementing human annotators without lowering annotation quality. The authors build this around a 250-intent Russian-language intent-classification pipeline, comparing LLM annotations against labels from highly paid expert annotators. Their best binary pipeline—an ensemble of two open-weight LLMs whose first-token probabilities are averaged and thresholded—reaches F1 0.8292 and accuracy 0.8332 at 65% coverage, higher than both full human annotation (F1 0.7977, accuracy 0.8018) and human annotation on the same 65% of instances. Their best multi-class pipeline also exceeds human total accuracy. The reason this matters is practical: annotation for retraining and drift monitoring is a recurring cost, and the paper reports roughly a sixfold speedup and a reduction of a two-day human benchmark to three hours of LLM time.

What carries the argument

The load-bearing mechanism is the prob-approach: the LLM is prompted to answer with a single token (0 or 1 for binary, a digit for multi-class), and the softmax probability of that first token is used to create an abstention zone—if the maximum answer probability falls below a hand-tuned threshold, the item is marked 'unk' and sent to human annotators. The text-approach variant lets the LLM write out reasoning before the answer, which helps interpretation but, the paper shows, makes thresholding ineffective because reasoning pushes probabilities to extremes. The final pipeline wraps the prob-approach in retrieval-augmented generation (RAG, adding relevant support documents to the prompt), LoRA fine-tuning (a low-rank adaptation of model weights) of a 32-billion-parameter model on high-quality reasoning traces, and an ensemble that averages probability predictions of two LLMs before thresholding.

What would settle it

Train two production classifiers on identical data labeled respectively by expert human annotators and by the best LLM ensemble, then evaluate both on a fresh expert-labeled test set; if the LLM-labeled classifier does not equal or outperform the expert-labeled one, the claim that LLMs can replace human annotators without sacrificing annotation quality is not established.

Watch

Extended reading notes

Core claim

The paper's central claim is that in a production Russian-language support-chat pipeline with roughly 250 intents, LLM-based annotation can replace or substantially supplement human annotators for validating text-classifier predictions without loss of quality. On a binary validation benchmark, an ensemble that averages the token probabilities of two open-weight LLMs and abstains below a threshold reaches F1 0.8292 and accuracy 0.8332 while covering 65% of instances, above both full-human performance (F1 0.7977, accuracy 0.8018) and human performance on the same 65% slice (F1 0.8133, accuracy 0.8174). On the multi-class benchmark the ensemble's total accuracy (accuracy times coverage) is 0.3001 versus 0.1489 for humans, with per-intent accuracy 0.7030 versus 0.6977. The intended consequence is that ongoing retraining and drift monitoring can be supported by continuous LLM annotation at roughly one-sixth the human time cost.

Load-bearing premise

The load-bearing premise is that the expert-labeled benchmark plus the chosen comparison protocol is a fair measure of annotation quality; if the protocol tilts toward the LLM (humans' confusing answers counted as wrong, no confusing option for LLMs, thresholds tuned to exceed human scores), the claim of no quality loss is not established.

Editorial extensions

If this is right

  • At the reported thresholds, 65% of binary validation annotations can be handed to the LLM ensemble with measured accuracy above human levels; the remaining 35% are deferred to human annotators.
  • Multi-class annotation—selecting the correct intent from the classifier's top five—can be fully automated with Accuracytotal 0.3001 versus 0.1489 for humans, removing a labor bottleneck in retraining.
  • Annotation latency drops by roughly a factor of six; the multi-class benchmark that took a human annotator about two days takes the pipeline about three hours.
  • LLM annotation can be run continuously on many GPUs, making drift monitoring and incremental learning feasible at production scale without hiring additional specialists.
  • If the higher measured accuracy of LLM labels holds, classifiers retrained on LLM-annotated data should drift more slowly than ones retrained on human-annotated data, since the paper's false-negative and false-positive experiments show the classifier is sensitive to label errors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The headline comparison is protocol-dependent: removing the 'confusing' option from humans and tuning thresholds to beat human scores both favor the LLM, so a neutral protocol would be a sharper test of 'no loss of quality'.
  • The abstention thresholds are hand-tuned per model and benchmark; in a new domain or language they would need recalibration, and the paper itself flags threshold sensitivity as an open issue, suggesting an adaptive thresholding rule as a practical extension.
  • An end-to-end test comparing a deployed classifier retrained exclusively on LLM labels versus exclusively on expert labels would directly quantify the business payoff; the paper lists this as missing even though its central claim implies such a test should pass.
  • Because the reported gains come from averaging probabilities of two models, a testable extension is whether a single model with better calibration plus temperature scaling could achieve the same ensemble result at lower inference cost.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes replacing or supplementing human annotators with open-weight LLMs in a production text-classification validation pipeline. It develops a binary annotation task (does the predicted intent match the text?) and a multi-class annotation task (select the correct intent from the top five predicted classes), using two response modes: a text-approach that generates reasoning and a prob-approach that uses the probability of the first generated token to define an abstention threshold. The method is augmented with RAG, LoRA fine-tuning, and LLM ensembles. On an in-house benchmark of roughly 18,000 binary and 12,000 multi-class examples labeled by expensive expert annotators, the paper reports that the binary ensemble reaches F1 0.8292 and accuracy 0.8332 at 65% coverage, exceeding human performance, and that the multi-class ensemble reaches accuracy 0.7030 at 42.7% coverage versus human accuracy 0.6977 at 52.4% coverage. The conclusion claims that human annotators can be effectively replaced or substantially supplemented without sacrificing annotation quality.

Significance. If the central comparison were valid, the paper would be a practically useful demonstration that open-weight LLMs can absorb a large fraction of production annotation work at substantially lower latency, with careful treatment of RAG and LoRA. The strengths are the scale and realism of the dataset, the use of open-source models, the comparison of several pipeline components (RAG, SFT, ensembles, thresholding), and the explicit speed measurements. The authors also honestly list missing end-to-end measurements and threshold-calibration caveats in the Discussion. However, as detailed below, the headline claim of 'no loss of annotation quality' is currently supported by a comparison protocol that favors the LLM, so the significance of the paper depends on fixing the evaluation rather than on the raw numbers.

major comments (4)
  1. [§6.7.2, Table 6.10] The headline binary comparison is not apples-to-apples. The LLM row reports metrics on the 65% of examples that survive the LLM ensemble's own confidence threshold, while the row 'Human Based on the same 65% data as LLM' evaluates human annotators on exactly the subset selected by the LLM. A model that is allowed to choose the subset on which it is scored will, by construction, look better than a model forced to answer everything, so the comparison conflates annotation quality with the selection rule. To support the Section 8 claim, the authors should report (i) LLM metrics with abstentions counted as errors on 100% coverage, or (ii) a comparison at matched coverage where the covered subset is chosen by a single pre-registered policy or by random sampling, and (iii) selectivity curves such as Figure 6.4 for both humans and the LLM ensemble. Without this, Table 6.10 does not demonstrate 'no loss of annotation quality'.
  2. [§4.3] The threshold search policy is explicitly stated as choosing thresholds so that the approach 'would show results higher than human annotators on annotation metrics'. Since the central quantitative claim is that LLM metrics exceed human metrics, tuning the threshold with the explicit target of exceeding human numbers makes the outcome a tuning artifact rather than an independent finding. The authors should show that the conclusion is robust across a range of thresholds (for example, by plotting the full accuracy-coverage curve and marking the human operating point), or specify a threshold-selection rule that does not use human benchmark performance and then report the resulting metrics.
  3. [§3.4.2, §3.2.2, Table 6.9] The multi-class comparison is asymmetric: human 'conf' responses are counted as wrong in the 'Accuracy (conf=wrong)' column, while the LLM is never offered a 'conf' option. In Table 6.9, the human row has accuracy 0.6977 on 52.4% coverage but only 0.3113 when conf is counted as wrong, whereas the ensemble is scored as 0.7030 without ever having to flag ambiguity. This biases the comparison against humans. Either the LLM should be allowed an analogous ambiguity or refusal option, or the human metrics should be recomputed on the subset of rows where experts did not use 'conf', with both systems evaluated on the same label set.
  4. [§8 vs. §7, Discussion item 4] The Discussion explicitly concedes that no end-to-end experiment was performed comparing classifiers trained on human annotations versus LLM annotations, yet the Conclusion states that human annotators 'can be effectively replaced ... without sacrificing annotation quality.' Because the stated motivation in Sections 1 and 3.5 is downstream classifier quality and incremental learning, the absence of any downstream measurement means the replacement claim is an extrapolation rather than a demonstrated result. The authors should either soften the conclusion or add an experiment that trains the text classifier on LLM-annotated data and evaluates it against the expert benchmark.
minor comments (5)
  1. [Appendix A] The tag '[RETREIVED]' is misspelled in all four prompts; it should be '[RETRIEVED]'.
  2. [Throughout] Several sentences begin with a lowercase 'it' (for example, Sections 3.1.2, 3.5.2, and 6.1.2), and there are other capitalization inconsistencies that should be corrected by copyediting.
  3. [Figure 6.4] The caption says 'increase in accuracy with a decrease in the threshold (decrease in coverage)', but the plot shows accuracy increasing as coverage decreases; the axis labels and caption should be made consistent and explicit.
  4. [Table 6.1 and §3.2.2] The human row reports both Coverage (conf='unk') and Coverage (conf != 'unk'); the meaning of these two coverage definitions and how they relate to the multi-class annotation protocol should be explained directly in Section 3.2.2.
  5. [References] Reference [16] has inconsistent capitalization and formatting in its title; please normalize it to the journal's reference style.

Circularity Check

2 steps flagged · score 4.0 of 10

Partial circularity: the headline 'LLM annotation beats human annotators' is partly a threshold-tuning target (Sec. 4.3) and is evaluated on a 65% subset selected by the LLM's own confidence (Table 6.10); the expert benchmark was held out, so the result is not fully forced.

  1. fitted input called prediction [Section 4.3 (Methodological Notes, Threshold search policy); reported in Section 6.7.2, Table 6.10]
    "The threshold values for the prob-approach were also chosen therefore the approach would show results higher than human annotators on annotation metrics, but not the highest possible, since rejecting classification logically reduces coverage and increases precision (because only examples that the LLM is confident about are annotated)."

    The paper's central claim (Section 8: 'without sacrificing annotation quality' and 'effectively replaced') rests on the metrics in which the LLM thresholds beat human metrics (Table 6.10). Section 4.3 states that the thresholds were chosen so the approach 'would show results higher than human annotators on annotation metrics.' The headline comparison is therefore an optimization target of the fitted threshold, not an independent experimental finding. Because Section 4.1 confirms the benchmark was not used for fitting, this is partial rather than complete circularity, but the superiority conclusion is encoded in the tuning objective.

  2. other [Section 6.7.2, Table 6.10, as compared with Section 3.3.2 (thresholding)]
    "The quality of annotation on the same data (65%) as humans is higher for LLMs. This may mean that LLM annotates simple tasks better than humans, but with more complex tasks (where the probability of answers is below the threshold), human annotators still perform better."

    The 'same 65% data' is not an independently defined human benchmark: it is the subset retained by thresholding the LLM's own token probabilities (Section 3.3.2: 'if this probability is below the threshold, the classification is rejected ... otherwise the confident answer is returned'). Scoring the human annotator on the LLM-selected subset means the LLM defines the test set by its own confidence. The claim that the LLM 'annotates simple tasks better than humans' is thus largely a restatement of the selection rule rather than evidence of superior annotation quality.

full rationale

This is an empirical comparison paper, so the classic derivation-collapse forms of circularity are largely absent: no equation in the paper reduces to a fitted value, and there is no self-citation chain or imported uniqueness theorem. The expert benchmark (Section 3.1.4) is an external gold standard, and Section 4.1 states that all parameters, including thresholds, were selected on separate production samples without test-data leakage. However, Section 4.3 explicitly says the prob-approach thresholds were chosen so the approach 'would show results higher than human annotators on annotation metrics,' making the headline claim of superiority partly a tuning target rather than an independent discovery. In addition, Table 6.10's 'Human Based on the same 65% data as LLM' row computes human performance on the subset created by the LLM's own confidence threshold, so the 'LLM beats humans on the same 65%' comparison is conditioned on the LLM's self-selected easy examples. These issues affect the validity of the central comparison, but because the benchmark itself was not used for fitting and the final numbers are measured rather than derived, the paper is not fully circular: it is a partially compromised empirical claim, not a definitional reduction. Score 4 reflects this partial circularity while acknowledging the independent held-out benchmark.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new particles, forces, or theoretical objects. Its contributions are methodological and empirical. The main dependencies are the proprietary dataset, the choice of thresholds, LoRA hyperparameters, and the RAG configuration, plus the assumption that expert agreement and token probability are valid measures.

free parameters (3)
  • Abstention thresholds for prob-approach = Not given numerically; reported as coverage values (e.g., 65%, 63%, 59%, 54%, 57%, 47%) for each model and ensemble
    Thresholds determine when the LLM abstains (predicts 'unk') and are tuned per model and task. Section 4.3 states they were chosen so the LLM would outperform human annotators.
  • LoRA hyperparameters (rank, alpha, dropout) = r=4, alpha=8, dropout=0.1
    Selected by search for both classification LoRA and reasoning LoRA. Section 5.3 states the main parameters were reviewed.
  • RAG configuration = 5 documents from 25 retrieved, retriever multilingual-e5-large-instruct, reranker bge-m3
    Number of documents and retriever/reranker pair were chosen empirically from experiments in Section 6.4.
assumptions (4)
  • domain assumption Expert annotations are the ground truth for measuring annotation quality.
    Section 3.1.4 states that quality annotations were collected from highly qualified experts and used as the benchmark to compare regular human and LLM annotations.
  • domain assumption Agreement of an annotator with the expert label is the correct and sufficient measure of annotation quality.
    Section 3.4.2 defines annotation accuracy solely as agreement with the fixed human expert labels, without considering annotator disagreement or downstream classifier impact.
  • domain assumption The first-token probability of the LLM is a trustworthy confidence signal for abstention.
    Section 3.3.2 uses the maximum token probability and a threshold to decide between confident annotation and rejection, assuming these probabilities calibrate the likelihood of correctness.
  • domain assumption Thresholds and hyperparameters tuned on the separate production sample generalize to the benchmark without leakage or systematic bias.
    Section 4.1 claims parameters were searched on separate samples without test leakage, but Section 4.3 reveals the selection target was to exceed human metrics, which may bias the comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Large Language Models in the Task of Automatic Validation of Text Classifier Predictions." pith.science (2026). https://pith.science/paper/Y3Z6RJON

@misc{pith2026250518688,
  author       = {Pith},
  title        = {Pith review of: Large Language Models in the Task of Automatic Validation of Text Classifier Predictions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y3Z6RJON}},
  note         = {Machine review of arXiv:2505.18688}
}
read the original abstract

Machine learning models for text classification are trained to predict a class for a given text. To do this, training and validation samples must be prepared: a set of texts is collected, and each text is assigned a class. These classes are usually assigned by human annotators with different expertise levels, depending on the specific classification task. Collecting such samples from scratch is labor-intensive because it requires finding specialists and compensating them for their work; moreover, the number of available specialists is limited, and their productivity is constrained by human factors. While it may not be too resource-intensive to collect samples once, the ongoing need to retrain models (especially in incremental learning pipelines) to address data drift (also called model drift) makes the data collection process crucial and costly over the model's entire lifecycle. This paper proposes several approaches to replace human annotators with Large Language Models (LLMs) to test classifier predictions for correctness, helping ensure model quality and support high-quality incremental learning.

Figures

Figures reproduced from arXiv: 2505.18688 by the authors.

Figure 3.1
Figure 3.1. Text-approach The first approach ( [PITH_FULL_IMAGE:figures/full_fig_p009_3_1.png] view at source ↗
Figure 3.2
Figure 3.2. Prob-approach The second approach ( [PITH_FULL_IMAGE:figures/full_fig_p010_3_2.png] view at source ↗
Figure 3.3
Figure 3.3. RAG example Not all company-specific terms may be known to LLM, as well as some client questions that can actually be answered using methodological materials written for support chat operators. To increase LLMs awareness when solving a task, a RAG system ( [PITH_FULL_IMAGE:figures/full_fig_p016_3_3.png] view at source ↗
Figures from the paper (6 more)
Figure 3.4
Figure 3.4. Figure 3.4: Proposed pipeline The final proposed approach is shown in [PITH_FULL_IMAGE:figures/full_fig_p017_3_4.png]
Figure 3.5
Figure 3.5. Figure 3.5: Simple ensemble A simple example of an ensemble is shown in [PITH_FULL_IMAGE:figures/full_fig_p018_3_5.png]
Figure 6.1
Figure 6.1. Figure 6.1: Impact of False Positive on the classifier (5 experiments) [PITH_FULL_IMAGE:figures/full_fig_p023_6_1.png]
Figure 6.2
Figure 6.2. Figure 6.2: Impact of False Negative on the classifier (5 experiments) [PITH_FULL_IMAGE:figures/full_fig_p024_6_2.png]
Figure 6.3
Figure 6.3. Figure 6.3: Distribution of response probabilities with and without reasoning [PITH_FULL_IMAGE:figures/full_fig_p027_6_3.png]
Figure 6.4
Figure 6.4. Figure 6.4: Example of accuracy dependence on coverage for a binary ensemble [PITH_FULL_IMAGE:figures/full_fig_p034_6_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

60 extracted references · 39 canonical work pages

  1. [1]

    Jiangpeng He, Runyu Mao, Zeman Shao, and Fengqing Zhu.Incremental Learning In Online Scenario. 2021. arXiv:2003.13191 [cs.CV]. url: https://arxiv.org/abs/2003.13191

    work page Pith review arXiv 2021

  2. [2]

    Model Drift: Why It Happens and How to Detect It

    IBM. Model Drift: Why It Happens and How to Detect It. Accessed: 2024-07-16.url: https: //www.ibm.com/think/topics/model-drift

    work page 2024

  3. [3]

    A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications

    Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications. 2025. arXiv:2402.07927 [cs.AI]. url: https://arxiv.org/abs/2402. 07927

    arXiv 2025

  4. [4]

    A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Tasks

    Shubham Vatsal and Harsh Dubey. A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Tasks. 2024. arXiv:2407.12994 [cs.CL]. url: https: //arxiv.org/abs/2407.12994

    arXiv 2024

  5. [6]

    Lefteris Loukas, Ilias Stogiannidis, Prodromos Malakasiotis, and Stavros Vassos.Breaking the Bank with ChatGPT: Few-Shot Text Classification for Finance. 2023. arXiv:2308.14634 [cs.CL]. url: https://arxiv.org/abs/2308.14634

    work page Pith review arXiv 2023

  6. [7]

    I˜ nigo Casanueva, Tadas Temˇ cinas, Daniela Gerz, Matthew Henderson, and Ivan Vuli´ c.Ef- ficient Intent Detection with Dual Sentence Encoders. 2020. arXiv: 2003.04807 [cs.CL] . url: https://arxiv.org/abs/2003.04807

    arXiv 2020

  7. [8]

    GPT-4 Technical Report

    OpenAI. “GPT-4 Technical Report”. In: CoRR abs/2303.08774 (2023). doi: 10 . 48550 / ARXIV.2303.08774. arXiv: 2303.08774. url: https://doi.org/10.48550/arXiv.2303. 08774

    arXiv doi:10.48550/arxiv.2303.08774 2023

  8. [9]

    Open, Closed, or Small Language Models for Text Classification?2023

    Hao Yu, Zachary Yang, Kellin Pelrine, Jean Francois Godbout, and Reihaneh Rabbany. Open, Closed, or Small Language Models for Text Classification?2023. arXiv: 2308.10092 [cs.CL]. url: https://arxiv.org/abs/2308.10092. 38

    arXiv 2023

Show all 60 references
  1. [10]

    Llama 2: Open Foundation and Fine-Tuned Chat Models

    Hugo Touvron, Louis Martin, Kevin Stone, Paul Albert, Abdullah Almahairi, Yacine Babaei, Nikita Bashlykov, Suchin Batra, Priya Bhargava, Shagun Bhosale, ..., and Thomas Scialom. “Llama 2: Open Foundation and Fine-Tuned Chat Models”. In: CoRR abs/2307.09288 (2023). doi: 10 . 48...

    2023 doi arXiv

  2. [11]

    A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models

    Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Yuhan Liu Shichun and Cui, Zeyang Zhou, Chao Gong, Jie Shen Yang and Zhou, Siming Chen, Tao Gui, Qi Zhang, and Xuanjing Huang. “A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models”. In: CoRR abs/2303.1042...

    2023 doi arXiv

  3. [12]

    Schnabel, Johanne R

    Julian A. Schnabel, Johanne R. Trippas, Falk Scholer, and Danula Hettiachchi.Multi-stage Large Language Model Pipelines Can Outperform GPT-4o in Relevance Assessment. 2025. arXiv: 2501.14296 [cs.IR]. url: https://arxiv.org/abs/2501.14296

    2025 arXiv

  4. [13]

    GPT-4o System Card

    OpenAI. GPT-4o System Card. https : / / openai . com / index / gpt - 4o - system - card/. Accessed: 2025-05-16. 2024

    2025

  5. [14]

    ChatGPT outperforms crowd workers for text-annotation tasks

    Fabrizio Gilardi, Meysam Alizadeh, and Ma¨ el Kubli. “ChatGPT outperforms crowd workers for text-annotation tasks”. In:Proceedings of the National Academy of Sciences120.30 (July 2023). issn: 1091-6490. doi: 10.1073/pnas.2305016120 . url: http://dx.doi.org/10. 1073/pnas.2305016120

    2023 doi

  6. [15]

    Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labels

    Nicholas Pangakis and Samuel Wolken. Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labels. 2024. arXiv: 2406 . 17633 [cs.CL]. url: https://arxiv.org/abs/2406.17633

    2024 arXiv

  7. [16]

    Open-source Large Language Models can Generate Labels from Radiology Reports for Training Convolutional Neural Networks

    Fares Al Mohamad, Leonhard Donle, Felix Dorfner, Laura Romanescu, Kristin Drechsler, Mike P. Wattjes, Jawed Nawabi, Marcus R. Makowski, Hartmut H¨ antze, Lisa Adams, Lina Xu, Felix Busch, Aymen Meddeb, and Keno Kyrill Bressem. “Open-source Large Language Models can Generate La...

    2025 doi

  8. [17]

    Hamidreza Rouzegar and Masoud Makrehchi.Enhancing Text Classification through LLM- Driven Active Learning and Human Annotation. 2024. arXiv: 2406.12114 [cs.CL] . url: https://arxiv.org/abs/2406.12114. 39

    2024 arXiv

  9. [18]

    Moxin Li, Wenjie Wang, Fuli Feng, Fengbin Zhu, Qifan Wang, and Tat-Seng Chua.Think Twice Before Trusting: Self-Detection for Large Language Models through Comprehensive Answer Reflection. 2024. arXiv: 2403 . 09972 [cs.CL]. url: https : / / arxiv . org / abs / 2403.09972

    2024 arXiv

  10. [19]

    Large Language Models are Zero-Shot Reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large Language Models are Zero-Shot Reasoners. 2023. arXiv: 2205.11916 [cs.CL] . url: https://arxiv.org/abs/2205.11916

    2023 arXiv

  11. [20]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal,ArielHerbert-Voss,GretchenKrueger,TomHenighan,RewonChild,AdityaRamesh, Daniel M. Ziegler, Jeffrey Wu, Cle...

    2020 arXiv

  12. [21]

    LoRA: Low-Rank Adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. “LoRA: Low-Rank Adaptation of large language models”. In: (2021). eprint: 2106.09685 (cs.CL)

    2021 arXiv

  13. [22]

    The probabilistic relevance framework: BM25 and beyond

    Stephen E. Robertson and Hugo Zaragoza. “The probabilistic relevance framework: BM25 and beyond”. In:Foundations and Trends in Information Retrieval3.4 (2009), pp. 333–389

    2009

  14. [23]

    Efficient Memory Management for Large Language Model Serving with PagedAttention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. “Efficient Memory Management for Large Language Model Serving with PagedAttention”. In:Proceedings of the ACM SIGOPS 29th Symposium on Operating Syste...

    2023

  15. [24]

    TheLlama3HerdofModels

    LaurensVanDerMaatenetal.“TheLlama3HerdofModels”.In: arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  16. [25]

    Qwen2.5: A Party of Foundation Models

    Qwen Team. Qwen2.5: A Party of Foundation Models. Sept. 2024.url: https://qwenlm. github.io/blog/qwen2.5/

    2024

  17. [26]

    Vikhr: Constructing a State-of-the-art Bilingual Open-Source Instruction-Following Large Language Model for Russian

    Aleksandr Nikolich, Konstantin Korolev, Sergei Bratchikov, Igor Kiselev, and Artem Shel- manov. “Vikhr: Constructing a State-of-the-art Bilingual Open-Source Instruction-Following Large Language Model for Russian”. In:Proceedings of the 4rd Workshop on Multilingual Representat...

    2024 arXiv

  18. [27]

    NeMo Mistral Toolkit for Large Language Models

    NVIDIA Corporation and Mistral AI. NeMo Mistral Toolkit for Large Language Models. https://developer.nvidia.com/nemo/mistral. Accessed: 2025-05-16. 2024

    2025

  19. [28]

    Gemma Team. “Gemma”. In: (2024). doi: 10.34740/KAGGLE/M/3301. url: https://www. kaggle.com/m/3301

    2024 doi

  20. [29]

    Introducing Athene-V2: Advancing Beyond the Limits of Scaling with Targeted Post-training

    Nexusflow. Introducing Athene-V2: Advancing Beyond the Limits of Scaling with Targeted Post-training. https://nexusflow.ai/blogs/athene-v2.BlogpostannouncingAthene-V2-Chat-72B. Nov. 2024

    2024

  21. [30]

    MultilingualE5TextEmbeddings:ATechnicalReport

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. “MultilingualE5TextEmbeddings:ATechnicalReport”.In: arXiv preprint arXiv:2402.05672 (2024)

    2024 arXiv

  22. [31]

    Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu.BGE M3- Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. 2024. arXiv:2402.03216 [cs.CL]

    2024 arXiv

  23. [32]

    The Russian-focused embedders’ exploration: ruMTEB benchmark and Russian embedding model design

    Artem Snegirev, Maria Tikhonova, Anna Maksimova, Alena Fenogenova, and Alexander Abramov. The Russian-focused embedders’ exploration: ruMTEB benchmark and Russian embedding model design. 2024. arXiv:2408.12503 [cs.CL]

    2024 arXiv

  24. [33]

    The Faiss library

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazar´ e, Maria Lomeli, Lucas Hosseini, and Herv´ e J´ egou. “The Faiss library”. In: (2024). arXiv: 2401.08281 [cs.LG]

    2024 arXiv

  25. [34]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforce- ment Learning

    DeepSeek-AI et al. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforce- ment Learning. 2025. arXiv: 2501.12948 [cs.CL] . url: https://arxiv.org/abs/2501. 12948. 41 A Appendix Multi-class prompt Hello! Here are the instructions you should follow when communica...

    2025 arXiv

  26. [38]

    Use the exact class name that was provided in the list, as this is necessary for subsequent quality assessment

    If you are able to classify the text clearly, enter the correct class after the [ANSWER] tag. Use the exact class name that was provided in the list, as this is necessary for subsequent quality assessment

  27. [39]

    unk" ("unk

    The text may not belong to any of the provided classes or may not contain any intent at all. In this case, display the answer in the format [ANSWER]"unk" ("unk" number from the task list)

  28. [40]

    unk" ("unk

    If the context of the user’s message is insufficient to provide an unambiguous answer using the provided list of classes, display the answer in the format [ANSWER]"unk" ("unk" number from the task list). Please note that this intent should only be used in extreme cases when yo...

  29. [41]

    If you find this difficult, do it as you see fit and then translate your thoughts into Russian

    Write and reason in Russian. If you find this difficult, do it as you see fit and then translate your thoughts into Russian

  30. [42]

    When solving tasks, be sure to write down all your reasoning in the text and analyze it while writing your answer

  31. [43]

    Stick to the required output format:

  32. [44]

    [REASONING] Reasoning regarding the selection of the appropriate class

    [TEXT] user’s text 2. [REASONING] Reasoning regarding the selection of the appropriate class. Reason consistently, consider all possible options, and use the class description and examples provided for it

  33. [45]

    If you decide that the answer should be one of the intents in the list, make sure you answer **ONLY** with the number of that intent

    [ANSWER] Answer - the number of one of the intents in the list. If you decide that the answer should be one of the intents in the list, make sure you answer **ONLY** with the number of that intent

  34. [46]

    This data was obtained from the internal information system and may contain additional useful information about the user’s text

    You may also encounter additional data enclosed in the [RETREIVED] tag. This data was obtained from the internal information system and may contain additional useful information about the user’s text. 42 Multi-class prob-approach prompt Hello! Here are the instructions you sho...

  35. [47]

    You receive text that a user has written in the app’s support chat, along with several hypothetical intents (classes) that this text may refer to

    You are an intelligent data annotation assistant. You receive text that a user has written in the app’s support chat, along with several hypothetical intents (classes) that this text may refer to. Along with the intent names, you will also receive a human-written description o...

  36. [50]

    Use the exact number that was provided in the list, as this is necessary for subsequent quality assessment

    If you are able to classify the text clearly, enter the correct class after the [ANSWER] tag. Use the exact number that was provided in the list, as this is necessary for subsequent quality assessment

  37. [51]

    unk" ("unk

    The text may not belong to any of the provided classes or may not contain any intent at all. In this case, enter your answer in the format [ANSWER]"unk" ("unk" number from the task list)

  38. [52]

    unk" ("unk

    If the context of the user’s message from [TEXT] is insufficient to provide a clear answer using the provided list of classes, enter your answer in the format [ANSWER]:"unk" ("unk" number from the task list). Please note that this intent should only be used in extreme cases wh...

  39. [53]

    If you decide that the answer should be one of the intents in the list, make sure you respond with **ONLY** the number of that intent

    Stick to the required output format: [ANSWER] Answer - number of one of the intents in the list. If you decide that the answer should be one of the intents in the list, make sure you respond with **ONLY** the number of that intent

  40. [54]

    This data was obtained from the internal information system and may contain additional useful information about the user’s text

    You may also encounter additional data enclosed in the [RETREIVED] tag. This data was obtained from the internal information system and may contain additional useful information about the user’s text. 43 Binary prompt Hello! Here are the instructions you should follow when com...

  41. [58]

    If you find this difficult, do it as you see fit and then translate your thoughts into Russian

    Write and think in Russian. If you find this difficult, do it as you see fit and then translate your thoughts into Russian

  42. [60]

    Reason step by step; for convenience, you can break your reasoning down into separate points

    When solving problems, be sure to write down all your reasoning in the text and analyze it while writing your answer. Reason step by step; for convenience, you can break your reasoning down into separate points

  43. [61]

    Follow the required format for presenting your conclusions:

  44. [62]

    Reason consistently, step by step, consider all possible options, and be sure to use the class description and examples provided for it

    [REASONING] Reasoning used to verify the correctness of the prediction. Reason consistently, step by step, consider all possible options, and be sure to use the class description and examples provided for it. Also, clarify the meaning of ambiguous words that may imply that the...

  45. [63]

    yes" or

    [ANSWER] The answer is a single line "yes" or "no" in lowercase (small letters). If the text belongs to the intended class or is similar to the examples provided, answer "yes." If the text cannot be related to this class and is not similar to any of the examples given, answer ...

  46. [64]

    This data was obtained from the internal information system and may contain additional useful information about the user’s text

    You may also encounter additional data enclosed in the [RETREIVED] tag. This data was obtained from the internal information system and may contain additional useful information about the user’s text. 44 Binary prob-approach prompt Hello! Here are the instructions you should f...

  47. [65]

    You receive text that a user has written in the application’s support chat and the suggested class to which this text may belong

    You are an intelligent data annotation assistant. You receive text that a user has written in the application’s support chat and the suggested class to which this text may belong. You need to determine whether the text actually belongs to the suggested class or not. In additio...

  48. [66]

    it is very important that you follow the instructions carefully and are very careful when making decisions

  49. [67]

    For each correctly completed task, you will receive bonuses that you can spend on self-development and training

  50. [68]

    Carefully compare the text with the examples to avoid falling into a trap

    Please note that the task text contains both suitable and unsuitable examples. Carefully compare the text with the examples to avoid falling into a trap. it is also important to consider product names or additional external context. If a class refers to one product or situatio...

  51. [69]

    0" or "1

    Follow the required format for presenting your conclusions: 1. [TEXT] User’s text, [ANSWER] The answer is a single line "0" or "1" in lowercase (small letters). If the text belongs to the intended class or is similar to the examples provided, answer "1." If the text cannot be ...

  52. [70]

    This data was obtained from the internal information system and may contain additional useful information about the user’s text

    You may also encounter additional data enclosed in the [RETREIVED] tag. This data was obtained from the internal information system and may contain additional useful information about the user’s text. 45

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.