Pith. sign in

REVIEW 4 major objections 6 minor 67 references

Adaptable and Precise: Enterprise-Scenario LLM Function-Calling Capability Training Pipeline

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a 7B model, fine-tuned with LoRA on a few thousand synthetic and human-augmented samples, can surpass GPT-4 and GPT-4o in function-calling accuracy within a specific enterprise scenario.

desk verdict A useful enterprise fine-tuning write-up whose headline 'beats GPT-4' claim rests on a circular test set; worth reading for the pipeline details, not for the comparison. read the letter →

arxiv 2412.15660 v1 pith:COO6NNUJ submitted 2024-12-20 cs.AI cs.CLcs.SE

Guancheng Zeng , Wentao Ding , Beining Xu , Chi Zhang , Wenqiang Han , Gang Li , Jingjing Mo , Pengxu Qiu
show 3 more authors
Xinran Tao Wang Tao Haowen Hu
This is my paper · ORCID
classification cs.AIcs.CLcs.SE
keywords functioncallingLLMfine-tuningLoRAdatasynthesisaugmentationAIagenttoolselectionASTevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents a training pipeline for adapting a small open-source LLM to call enterprise APIs accurately in a specific business scenario, and reports that a 7B model trained with this pipeline outperforms GPT-4 and GPT-4o in function-calling accuracy on its test sets. The pipeline combines AI-generated and human-augmented training data, LoRA fine-tuning on four 24GB GPUs in about five hours, and AST-based evaluation that separates structural, tool-selection, and parameter errors. If the result holds, it would let small and medium enterprises build domain-specific AI agents without large compute budgets or sending data to the cloud. The empirical evidence is a digital HR agent with 14 workflows, where the best fine-tuned model reaches 97.6% overall accuracy on one test set and 95.6% on another.

What carries the argument

The mechanism is a data-augmentation and fine-tuning loop built around structured function-calling instructions. The pipeline first turns tool metadata (name, description, parameter names, types, and necessity) into seed questions using a high-performance LLM, then applies four augmentation strategies—replacement, rewriting, simplification, and error introduction—to expand each seed into multiple variants. All variants of a seed share the same function-calling instruction, which acts as the training label. LoRA adapters keep the fine-tuning parameter-efficient, and an AST parser decomposes model outputs into structural, tool-selection, and parameter errors, allowing each failure type to be measured and attributed. The same evaluation setup also supports confusion-matrix analysis of tool selection and multi-LoRA adapter merging for post-deployment updates.

What would settle it

Have domain experts who never saw the training data write a fresh set of questions across the 14 workflows, run DHR_train_3_ft and GPT-4 on it, and compare overall accuracy; if the 7B model's score falls below GPT-4's, the paper's superiority claim is falsified.

Watch

Extended reading notes

Core claim

The central discovery is that scenario-specific function-calling can be instilled into a 7B instruction-tuned model through a closed loop of data synthesis, augmentation, LoRA fine-tuning, and AST-based evaluation. In the digital HR scenario, starting from the 14 workflow tools and 207 human-annotated seed questions, the pipeline generated 1,260 AI-augmented and 1,035 human-augmented training samples; the model trained on the combined set (DHR_train_3_ft) reached 100% structural completeness, 97.6% tool-selection accuracy, and 100% parameter-filling accuracy on the 207-question DHR_test_A set, with 97.6% overall accuracy on that set and 95.6% on the separate 135-question DHR_test_B set. On the same test sets, GPT-4 scored 79.2% and 88.1% overall, and GPT-4o scored 32.9% and 34.1%, so the fine-tuned 7B model is reported to surpass both.

Load-bearing premise

The claim that the fine-tuned model beats GPT-4 rests on the assumption that the test sets measure generalization rather than memorization: DHR_test_A is exactly the 207 manually annotated questions that seeded the augmented training data, and DHR_test_B's 135 questions are not documented as being independent of the training-generation prompts.

Editorial extensions

If this is right

  • A 7B model specialized to one business domain can outperform much larger general-purpose models on that domain's function-calling task.
  • The whole training runs in about five hours on four 24GB GPUs, so the recipe is reproducible for small and medium enterprises that want on-premise agents.
  • Mixing AI-generated and human-annotated seed data helps more than either alone, so a modest amount of expert annotation can be amplified by synthesis.
  • Shortening tool descriptions during training improves robustness across long, short, and absent descriptions, which is a practical guideline for API documentation.
  • Merging LoRA adapters trained on different data splits can beat either adapter alone, suggesting a training-free path for incorporating new data cycles after deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the reported margin over GPT-4 is likely inflated because DHR_test_A is built from the same 207 seed questions that generated the augmented training data; a genuinely held-out test set could shrink the gap.
  • Editorial inference: the pipeline's 100% parameter-filling accuracy may not survive contact with wider parameter distributions than the synthetic templates produce; real user logs with unseen names, dates, and free-text fields would be the stress test.
  • Editorial inference: the four augmentation strategies are presented as a bundle, so their individual contributions are untested; ablating each one separately would show which strategy carries the gain.
  • Editorial inference: the multi-LoRA merging result points toward a practical continual-learning recipe—train a fresh adapter on each quarterly data cycle and merge adapters to update the deployed agent without full retraining.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes a training pipeline for enterprise-scenario LLM function-calling, comprising scenario-specific data synthesis and augmentation, LoRA fine-tuning, and AST-based evaluation. Using a digital HR intelligent-agent scenario with 14 workflows, the authors generate 1,260 AI-augmented and 1,035 manually augmented training samples, fine-tune Qwen2.5-Coder-7B-Instruct with LoRA on four 24GB GPUs, and report that the resulting model surpasses GPT-4 and GPT-4o in structural completeness, tool-selection accuracy, and parameter-filling accuracy on their test sets. The paper also presents ablations on base models, training data composition, cutoff length, tool description length, and LoRA adapter merging.

Significance. If the empirical claims were sound, the pipeline would be a valuable practical contribution: it offers a low-resource recipe for adapting small open-weight models to specialized enterprise function-calling scenarios, and the confusion-matrix-based tool-selection analysis is a useful diagnostic. The paper also provides a transparent description of data synthesis and augmentation templates, and the LoRA-merging experiments address the practical issue of data life-cycle management. However, the central claim of surpassing GPT-4 rests on test sets that are not demonstrably independent of the training data, and the evaluation methodology lacks the statistical support needed to distinguish genuine generalization from memorization. The pipeline concept is defensible, but the current evidence does not establish the headline result.

major comments (4)
  1. [Section 4.2.1 and 4.2.3] DHR_test_A is exactly the 207 manually annotated seed questions used to generate the 1,035 augmented training instances in DHR_train_2 and DHR_train_3. Since the augmentation strategies (replacement, rewriting, simplification, error introduction) produce training queries that share entities, tool-selection patterns, and parameter values with the seed intents, the reported near-perfect accuracy on DHR_test_A largely reflects the model's exposure to augmented variants of those same intents. The abstract's claim of 'surpassing GPT-4 and GPT-4o in accuracy on the test set' is therefore not supported as evidence of generalization to unseen queries.
  2. [Section 4.2.3] DHR_test_B consists of only 135 questions and its construction is not described. The paper states that it was designed to 'prevent evaluation bias caused by potential over-fitting to seed data,' but without a clear account of how these questions were created, whether they are disjoint from all training and augmentation data, and whether they were produced by the same GPT-4 prompt templates used for training-data synthesis, DHR_test_B cannot be trusted as an independent held-out set. The reader cannot verify that the claimed superiority over GPT-4 on DHR_test_B is not also subject to distributional overlap with the training data.
  3. [Table 3 and Section 4.3] The column headers in Table 3 are ambiguous: it is not clear which columns correspond to DHR_test_A and which to DHR_test_B, nor which metrics are reported for each set. Additionally, all results are from single runs with no error bars, confidence intervals, or multiple seeds. Given the small test-set sizes (207 and 135), differences such as the 95.6% vs 88.1% tool-selection accuracy cannot be assessed for statistical significance. The paper should report variance across repeated fine-tuning runs and clearly separate the metrics for each test set.
  4. [Sections 4.3 and 5] The claim of surpassing GPT-4 is evaluated only on the two custom test sets described above, with no public function-calling benchmark (e.g., BFCL, API-Bench) used as an external anchor. Furthermore, all ablation studies in Section 5 (base-model comparison, data composition, cutoff length, description length, LoRA merging) are conducted on DHR_test_A, which is contaminated by overlap with the training seeds. Consequently, the ablations do not provide reliable evidence about what drives generalization to genuinely new queries, and the pipeline's adaptability beyond the specific HR scenario remains unverified.
minor comments (6)
  1. [Table 3] The table layout should be restructured so that each test set has its own set of clearly labeled columns (e.g., Structure, Tool, Parameter) rather than merging the headings as at present.
  2. [Section 4.2.2] The paper says checkpoints were 'typically selected between the 7th and 10th iterations' but does not specify the selection criterion; this should be stated (e.g., validation-set performance) to ensure the reported results are not cherry-picked.
  3. [Figures 3 and 4] The confusion-matrix heatmaps and F1-score bar charts would benefit from explicit axis labels and a legend indicating which rows/columns correspond to each workflow; currently the reader cannot interpret the confusion pattern without the caption.
  4. [Appendix A and B] The prompt templates are difficult to read because of irregular spacing, inconsistent capitalization, and stray braces; they should be typeset in a clean monospaced format.
  5. [Section 5.4] The 'No Description' condition is surprising as a training configuration because the model has no information about the tool's purpose; a brief explanation of why this condition was included and what the authors expected would improve readability.
  6. [References] Reference [23] appears to use a Wikipedia-style URL; the citation should be completed with proper bibliographic details.

Circularity Check

1 steps flagged · score 6.0 of 10

DHR_test_A is literally the 207 seed questions from which the 1,035 training examples in DHR_train_2/DHR_train_3 were augmented, so the reported 'surpassing GPT-4' result on that set is forced by construction.

  1. self definitional [Section 4.2.1 (Data Synthesis) and Section 4.2.3 (Model Evaluation)]
    "From these 207 manually annotated seeds, we generated a total of 1,035 augmented training data instances, with 5 augmentations for each seed instance. ... A total of 207 manually annotated seed questions were selected as the DHR_test_A test set to compare the effects of different hyper-parameter settings and mixed data on model fine-tuning."

    DHR_test_A is not an independent held-out set: it is exactly the seed set used to generate the augmented training data. Section 3.3.3 states that 'all augmented questions generated from the same seed question use the same function-calling instructions,' so each DHR_test_A query has five training variants with the same target function call, differing only through replacement, rewriting, simplification, or error introduction. The reported 95.6% tool-selection accuracy versus GPT-4's 88.1% on DHR_test_A therefore measures near-duplicate recognition of training intents, not generalization to unseen queries.

full rationale

The training pipeline itself is not circular: data synthesis, LoRA fine-tuning, AST-based parsing, and confusion-matrix analysis are standard components that do not reduce to their own outputs. The circularity is confined to the evaluation design. Section 4.2.1 describes how 207 manually annotated seeds were expanded into 1,035 augmented training instances at five paraphrases per seed, and Section 4.2.3 then selects exactly those same 207 seed questions as DHR_test_A. Because every augmented training variant shares the seed's function-calling instruction, the model has effectively been trained on near-duplicates of the test questions, with the same labels. High scores on DHR_test_A therefore reflect paraphrase robustness and memorization rather than generalization to novel queries. The paper itself acknowledges the risk by creating DHR_test_B 'to prevent evaluation bias caused by potential over-fitting to seed data,' but the abstract and headline comparisons do not separate the contaminated A set from the fresher B set. DHR_test_B is only 135 questions and its generation process is not specified, so it cannot fully rescue the broad superiority claim. A score of 6 reflects that one central evaluation result reduces by construction, while the existence of a second, partially independent test set prevents the entire claim from being purely definitional.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The pipeline introduces no new entities or fitted scientific parameters. The central claim rests on domain assumptions about the representativeness of the workflows and the validity of the AST evaluation.

free parameters (4)
  • LoRA rank r = 8
    Chosen by hand, standard small value; not fitted to data.
  • LoRA alpha = 16
    Chosen as twice the rank, standard practice.
  • peak learning rate = 8.0e-5
    Hand-selected; not swept.
  • training epochs = 10 iterations, checkpoints 7-10
    Selected based on validation performance; not central to the claim.
assumptions (4)
  • domain assumption The 14 workflows and their descriptions accurately reflect real enterprise HR usage.
    No external validation of tool descriptions; model quality depends on these descriptions.
  • domain assumption The 207 manually annotated seed questions are representative of real user queries.
    These seeds are both the source of training augmentation and the DHR_test_A test set.
  • domain assumption AST-based parsing correctly captures function-calling correctness.
    The modified BFCL AST evaluation is treated as ground truth; no manual verification of a sample is reported.
  • domain assumption High accuracy on the private test sets transfers to real-world usage.
    The paper claims 'practical applications' success without quantitative real-world metrics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adaptable and Precise: Enterprise-Scenario LLM Function-Calling Capability Training Pipeline." pith.science (2026). https://pith.science/paper/COO6NNUJ

@misc{pith2026241215660,
  author       = {Pith},
  title        = {Pith review of: Adaptable and Precise: Enterprise-Scenario LLM Function-Calling Capability Training Pipeline},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/COO6NNUJ}},
  note         = {Machine review of arXiv:2412.15660}
}
read the original abstract

Enterprises possess a vast array of API assets scattered across various functions, forming the backbone of existing business processes. By leveraging these APIs as functional tools, enterprises can design diverse, scenario-specific agent applications, driven by on-premise function-calling models as the core engine. However, generic models often fail to meet enterprise requirements in terms of computational efficiency, output accuracy, and stability, necessitating scenario-specific adaptation. In this paper, we propose a training pipeline for function-calling capabilities tailored to real-world business scenarios. This pipeline includes the synthesis and augmentation of scenario-specific function-calling data, model fine-tuning, and performance evaluation and analysis. Using this pipeline, we generated 1,260 fully AI-generated samples and 1,035 augmented manually-labeled samples in digital HR agent scenario. The Qwen2.5-Coder-7B-Instruct model was employed as the base model and fine-tuned using the LoRA method on four GPUs with 24GB VRAM. Our fine-tuned model demonstrated outstanding performance in evaluations and practical applications, surpassing GPT-4 and GPT-4o in accuracy on the test set. These results validate the reliability of the proposed pipeline for training scenario-specific function-calling models.

Figures

Figures reproduced from arXiv: 2412.15660 by the authors.

Figure 1
Figure 1. The Overall Training Pipeline of Enterprise-Scenario Function-Calling LLM [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Data Volume and Distribution of 14 Digital HR Scenario Tools [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Tool Selection Confusion Matrix Heatmap [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: F1 Score Comparison of Different Models across Various Tools. For [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Comparison of Model Performance with Different [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Comparison of Model Performance with Different Tool Description Length [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 27 canonical work pages

  1. [1]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao et al. “React: Synergizing reasoning and acting in language models”. In:arXiv preprint arXiv:2210.03629 (2022)

    arXiv 2022

  2. [2]

    The rise and potential of large language model based agents: A survey

    Zhiheng Xi et al. “The rise and potential of large language model based agents: A survey”. In: arXiv preprint arXiv:2309.07864 (2023). 15

    arXiv 2023

  3. [3]

    Tdag: A multi-agent framework based on dynamic task decomposition and agent generation

    Yaoxiang Wang et al. “Tdag: A multi-agent framework based on dynamic task decomposition and agent generation”. In: arXiv preprint arXiv:2402.10178 (2024)

    arXiv 2024

  4. [4]

    Advancing Agentic Systems: Dynamic Task Decomposition, Tool Integration and Evaluation using Novel Metrics and Dataset

    Shankar Kumar Jeyakumar, Alaa Alameer Ahmad, and Adrian Garret Gabriel. “Advancing Agentic Systems: Dynamic Task Decomposition, Tool Integration and Evaluation using Novel Metrics and Dataset”. In: NeurIPS 2024 Workshop on Open-World Agents

    work page 2024

  5. [5]

    Research of the Enterprise Application Integration Platform Based on Multi-agent

    Guozheng Wang et al. “Research of the Enterprise Application Integration Platform Based on Multi-agent”. In: 2009 Fifth International Joint Conference on INC, IMS and IDC. 2009, pp. 329–331. DOI: 10.1109/NCM.2009.112

    work page doi:10.1109/ncm.2009.112 2009

  6. [6]

    Enterprise Design, Operations and Computing with AI Agents: Accountability using DSL

    Zoran Milosevic and Igor Dejanovic. “Enterprise Design, Operations and Computing with AI Agents: Accountability using DSL”. In: ()

    work page

  7. [7]

    Enhancing trust in LLM-based AI automation agents: New considerations and future challenges

    Sivan Schwartz, Avi Yaeli, and Segev Shlomov. “Enhancing trust in LLM-based AI automation agents: New considerations and future challenges”. In: arXiv preprint arXiv:2308.05391 (2023)

    arXiv 2023

  8. [8]

    Scaling instruction-finetuned language models

    Hyung Won Chung et al. “Scaling instruction-finetuned language models”. In: Journal of Machine Learning Research 25.70 (2024), pp. 1–53

    work page 2024

Show all 67 references
  1. [9]

    Alpaca: A strong, replicable instruction-following model

    Rohan Taori et al. “Alpaca: A strong, replicable instruction-following model”. In: Stanford Center for Research on Foundation Models. https://crfm. stanford. edu/2023/03/13/alpaca. html 3.6 (2023), p. 7

    2023

  2. [10]

    AI and privacy concerns: a smart meter case study

    Jillian Carmody, Samir Shringarpure, and Gerhard Van de Venter. “AI and privacy concerns: a smart meter case study”. In: Journal of Information, Communication and Ethics in Society 19.4 (2021), pp. 492–505

    2021

  3. [11]

    Lora: Low-rank adaptation of large language models

    Edward J Hu et al. “Lora: Low-rank adaptation of large language models”. In: arXiv preprint arXiv:2106.09685 (2021)

    2021 arXiv

  4. [12]

    Qwen technical report

    Jinze Bai et al. “Qwen technical report”. In: arXiv preprint arXiv:2309.16609 (2023)

    2023 arXiv

  5. [13]

    A survey of large language models

    Wayne Xin Zhao et al. “A survey of large language models”. In: arXiv preprint arXiv:2303.18223 (2023)

    2023 arXiv

  6. [14]

    On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey

    Lin Long et al. On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey. 2024. arXiv: 2406.15126 [cs.CL]. URL: https://arxiv.org/abs/2406.15126

    2024 arXiv

  7. [15]

    Self-instruct: Aligning language models with self-generated instructions

    Yizhong Wang et al. “Self-instruct: Aligning language models with self-generated instructions”. In: arXiv preprint arXiv:2212.10560 (2022)

    2022 arXiv

  8. [16]

    Wizardlm: Empowering large language models to follow complex instructions

    Can Xu et al. “Wizardlm: Empowering large language models to follow complex instructions”. In: arXiv preprint arXiv:2304.12244 (2023)

    2023 arXiv

  9. [17]

    Toolformer: Language models can teach themselves to use tools

    Timo Schick et al. “Toolformer: Language models can teach themselves to use tools”. In: Advances in Neural Information Processing Systems 36 (2024)

    2024

  10. [18]

    Toolllm: Facilitating large language models to master 16000+ real-world apis

    Yujia Qin et al. “Toolllm: Facilitating large language models to master 16000+ real-world apis”. In: arXiv preprint arXiv:2307.16789 (2023)

    2023 arXiv

  11. [19]

    Toolace: Winning the points of llm function calling

    Weiwen Liu et al. “Toolace: Winning the points of llm function calling”. In:arXiv preprint arXiv:2409.00920 (2024)

    2024 arXiv

  12. [20]

    Toolalpaca: Generalized tool learning for language models with 3000 simulated cases

    Qiaoyu Tang et al. “Toolalpaca: Generalized tool learning for language models with 3000 simulated cases”. In: arXiv preprint arXiv:2306.05301 (2023)

    2023 arXiv

  13. [21]

    Apigen: Automated pipeline for generating verifiable and diverse function- calling datasets

    Zuxin Liu et al. “Apigen: Automated pipeline for generating verifiable and diverse function- calling datasets”. In: arXiv preprint arXiv:2406.18518 (2024)

    2024 arXiv

  14. [22]

    Measuring massive multitask language understanding

    Dan Hendrycks et al. “Measuring massive multitask language understanding”. In: arXiv preprint arXiv:2009.03300 (2020)

    2020 arXiv

  15. [23]

    Cmmlu: Measuring massive multitask language understanding in chinese

    Haonan Li et al. “Cmmlu: Measuring massive multitask language understanding in chinese”. In: arXiv preprint arXiv:2306.09212 (2023)

    2023 arXiv

  16. [24]

    C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models

    Yuzhen Huang et al. “C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models”. In: Advances in Neural Information Processing Systems 36 (2024)

    2024

  17. [25]

    Training verifiers to solve math word problems

    Karl Cobbe et al. “Training verifiers to solve math word problems”. In: arXiv preprint arXiv:2110.14168 (2021)

    2021 arXiv

  18. [26]

    Mle-bench: Evaluating machine learning agents on machine learning engineering

    Jun Shern Chan et al. “Mle-bench: Evaluating machine learning agents on machine learning engineering”. In: arXiv preprint arXiv:2410.07095 (2024)

    2024 arXiv

  19. [27]

    OpenCompass: A Universal Evaluation Platform for Foundation Models

    OpenCompass Contributors. OpenCompass: A Universal Evaluation Platform for Foundation Models. https://github.com/open-compass/opencompass. 2023. 16

    2023

  20. [28]

    Gorilla: Large language model connected with massive apis

    Shishir G Patil et al. “Gorilla: Large language model connected with massive apis”. In: arXiv preprint arXiv:2305.15334 (2023)

    2023 arXiv

  21. [29]

    On the tool manipulation capability of open-source large language models

    Qiantong Xu et al. “On the tool manipulation capability of open-source large language models”. In: arXiv preprint arXiv:2305.16504 (2023)

    2023 arXiv

  22. [30]

    AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents

    Chang Ma et al. “AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents”. In: arXiv preprint arXiv:2401.13178 (2024)

    2024 arXiv

  23. [31]

    https://gorilla.cs.berkeley

    Fanjia Yan et al.Berkeley Function Calling Leaderboard. https://gorilla.cs.berkeley. edu/blogs/8_berkeley_function_calling_leaderboard.html. 2024

    2024

  24. [32]

    Springer Nature, 2019

    Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren.Automated machine learning: methods, systems, challenges. Springer Nature, 2019

    2019

  25. [33]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei et al. “Chain-of-thought prompting elicits reasoning in large language models”. In: Advances in neural information processing systems 35 (2022), pp. 24824–24837

    2022

  26. [34]

    Language models are few-shot learners

    Tom Brown et al. “Language models are few-shot learners”. In:Advances in neural information processing systems 33 (2020), pp. 1877–1901

    2020

  27. [35]

    Adapterfusion: Non-destructive task composition for transfer learning

    Jonas Pfeiffer et al. “Adapterfusion: Non-destructive task composition for transfer learning”. In: arXiv preprint arXiv:2005.00247 (2020)

    2020 arXiv

  28. [36]

    Learning to represent programs with graphs

    Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. “Learning to represent programs with graphs”. In: arXiv preprint arXiv:1711.00740 (2017)

    2017 arXiv

  29. [37]

    A systematic analysis of performance measures for classification tasks

    Marina Sokolova and Guy Lapalme. “A systematic analysis of performance measures for classification tasks”. In: Information processing & management 45.4 (2009), pp. 427–437

    2009

  30. [38]

    Parameter-efficient transfer learning for NLP

    Neil Houlsby et al. “Parameter-efficient transfer learning for NLP”. In:International conference on machine learning. PMLR. 2019, pp. 2790–2799

    2019

  31. [39]

    Deep compression: Compressing deep neu- ral networks with pruning, trained quantization and huffman coding

    Song Han, Huizi Mao, and William J Dally. “Deep compression: Compressing deep neu- ral networks with pruning, trained quantization and huffman coding”. In: arXiv preprint arXiv:1510.00149 (2015)

    2015 arXiv

  32. [40]

    Transfer learning in natural language processing

    Sebastian Ruder et al. “Transfer learning in natural language processing”. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: Tutorials. 2019, pp. 15–18

    2019

  33. [41]

    Scaling laws for neural language models

    Jared Kaplan et al. “Scaling laws for neural language models”. In: arXiv preprint arXiv:2001.08361 (2020)

    2020 arXiv

  34. [42]

    Making pre-trained language models better few-shot learners

    Tianyu Gao, Adam Fisch, and Danqi Chen. “Making pre-trained language models better few-shot learners”. In: arXiv preprint arXiv:2012.15723 (2020)

    2020 arXiv

  35. [43]

    From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning

    Ming Li et al. “From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning”. In: arXiv preprint arXiv:2308.12032 (2023)

    2023 arXiv

  36. [44]

    Cheap and fast–but is it good? evaluating non-expert annotations for natural language tasks

    Rion Snow et al. “Cheap and fast–but is it good? evaluating non-expert annotations for natural language tasks”. In: Proceedings of the 2008 conference on empirical methods in natural language processing. 2008, pp. 254–263

    2008

  37. [45]

    Eda: Easy data augmentation techniques for boosting performance on text classification tasks

    Jason Wei and Kai Zou. “Eda: Easy data augmentation techniques for boosting performance on text classification tasks”. In: arXiv preprint arXiv:1901.11196 (2019)

    2019 arXiv

  38. [46]

    Methodologies for data quality assessment and improvement

    Carlo Batini et al. “Methodologies for data quality assessment and improvement”. In: ACM computing surveys (CSUR) 41.3 (2009), pp. 1–52

    2009

  39. [47]

    Beyond accuracy: What data quality means to data consumers

    Richard Y Wang and Diane M Strong. “Beyond accuracy: What data quality means to data consumers”. In: Journal of management information systems 12.4 (1996), pp. 5–33

    1996

  40. [48]

    Don’t stop pretraining: Adapt language models to domains and tasks

    Suchin Gururangan et al. “Don’t stop pretraining: Adapt language models to domains and tasks”. In: arXiv preprint arXiv:2004.10964 (2020)

    2020 arXiv

  41. [49]

    The pile: An 800gb dataset of diverse text for language modeling

    Leo Gao et al. “The pile: An 800gb dataset of diverse text for language modeling”. In: arXiv preprint arXiv:2101.00027 (2020)

    2020 arXiv

  42. [50]

    Training language models to follow instructions with human feedback

    Long Ouyang et al. “Training language models to follow instructions with human feedback”. In: Advances in neural information processing systems 35 (2022), pp. 27730–27744

    2022

  43. [51]

    Exploring the limits of transfer learning with a unified text-to-text trans- former

    Colin Raffel et al. “Exploring the limits of transfer learning with a unified text-to-text trans- former”. In: Journal of machine learning research21.140 (2020), pp. 1–67

    2020

  44. [52]

    Smaller, weaker, yet better: Training llm reasoners via compute-optimal sampling

    Hritik Bansal et al. “Smaller, weaker, yet better: Training llm reasoners via compute-optimal sampling”. In: arXiv preprint arXiv:2408.16737 (2024)

    2024 arXiv

  45. [53]

    Climbing towards NLU: On meaning, form, and understanding in the age of data

    Emily M Bender and Alexander Koller. “Climbing towards NLU: On meaning, form, and understanding in the age of data”. In:Proceedings of the 58th annual meeting of the association for computational linguistics. 2020, pp. 5185–5198. 17

    2020

  46. [54]

    Llamafactory: Unified efficient fine-tuning of 100+ language models

    Yaowei Zheng et al. “Llamafactory: Unified efficient fine-tuning of 100+ language models”. In: arXiv preprint arXiv:2403.13372 (2024)

    2024 arXiv

  47. [55]

    Universal language model fine-tuning for text classifi- cation

    Jeremy Howard and Sebastian Ruder. “Universal language model fine-tuning for text classifi- cation”. In: arXiv preprint arXiv:1801.06146 (2018)

    2018 arXiv

  48. [56]

    A study of cross-validation and bootstrap for accuracy estimation and model selection

    Ron Kohavi et al. “A study of cross-validation and bootstrap for accuracy estimation and model selection”. In: Ijcai. V ol. 14. 2. Montreal, Canada. 1995, pp. 1137–1145

    1995

  49. [57]

    How to fine-tune bert for text classification?

    Chi Sun et al. “How to fine-tune bert for text classification?” In: Chinese computational linguistics: 18th China national conference, CCL 2019, Kunming, China, October 18–20, 2019, proceedings 18. Springer. 2019, pp. 194–206

    2019

  50. [58]

    Attention is all you need

    A Vaswani. “Attention is all you need”. In:Advances in Neural Information Processing Systems (2017)

    2017

  51. [59]

    Dropout: a simple way to prevent neural networks from overfitting

    Nitish Srivastava et al. “Dropout: a simple way to prevent neural networks from overfitting”. In: The journal of machine learning research 15.1 (2014), pp. 1929–1958

    2014

  52. [60]

    ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

    Samyam Rajbhandari et al. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. 2020. arXiv: 1910 . 02054 [cs.LG]. URL: https : / / arxiv . org / abs / 1910 . 02054

    2020

  53. [61]

    Model merging with SVD to tie the Knots

    George Stoica et al. “Model merging with SVD to tie the Knots”. In: arXiv preprint arXiv:2410.19735 (2024)

    2024 arXiv

  54. [62]

    PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods

    Sourab Mangrulkar et al. PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods. https://github.com/huggingface/peft. 2022

    2022

  55. [63]

    Compilers: Principles, techniques and tools, 2nd editio

    Alfred Aho et al. Compilers: Principles, techniques and tools, 2nd editio. 2007

    2007

  56. [64]

    Evaluating large language models trained on code

    Mark Chen et al. “Evaluating large language models trained on code”. In: arXiv preprint arXiv:2107.03374 (2021)

    2021 arXiv

  57. [65]

    Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation

    David MW Powers. “Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation”. In: arXiv preprint arXiv:2010.16061 (2020)

    2020 arXiv

  58. [66]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov et al. “Direct preference optimization: Your language model is secretly a reward model”. In: Advances in Neural Information Processing Systems 36 (2024). 18 Appendix A Data Synthesis Prompt Templates Prompt Templates are translated to English. Appendix A.1 Ques...

    2024

  59. [67]

    "" Appendix A.2 Question Generation with real name prompt = f

    Question content """ Appendix A.2 Question Generation with real name prompt = f """ You are a data an no ta to r . Your task is to generate a set of diverse qu es tio ns for the given function . The c o n s t r u c t e d qu es tio ns will be directly used as p a r a m e t e r ...

    2021

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.