Pith. sign in

REVIEW 4 major objections 5 minor 46 references

Efficient Learning Content Retrieval with Knowledge Injection

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a small quantized model fine-tuned with QLoRA and paired with a vector database outperforms larger base models on domain-specific course-guidance questions.

desk verdict The reported performance numbers mostly measure how closely the models imitate GPT-4, not whether the answers are factually right, because GPT-4 generated the training pairs, the RAG content, and the scoring ground truth. read the letter →

arxiv 2412.00125 v1 pith:IMX5IWGB submitted 2024-11-28 cs.CL

Batuhan Sariturk , Rabia Bayraktar , Merve Elmas Erdem This is my paper · ORCID
classification cs.CL
keywords RetrievalAugmentedGenerationQLoRAparameter-efficientfine-tuningsmalllanguagemodelsdomain-specificchatbotquestionansweringcoursenavigationBERTScore
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a small, quantized language model can be turned into a practical course-guidance chatbot on a limited budget by combining two cheap techniques: QLoRA fine-tuning on a few hundred question-answer pairs and retrieval-augmented generation over a small vector database of course documents. The authors build and compare base models, fine-tuned models, RAG-augmented base models, and the combined RAG-plus-fine-tuned system using the Phi-2 and Phi-3 model families, scoring answers with thirteen automatic metrics. Their central finding is that the RAG-supported fine-tuned Phi-2 beats both larger base models and fine-tuning alone on almost every metric, with BERTScore precision around 0.84 and F1 around 0.82. If true, the practical payoff is that organizations with modest GPUs and no large labeled dataset can still build workable domain chatbots for navigating educational content.

What carries the argument

The load-bearing object is a two-stage pipeline: a quantized low-rank adaptation (QLoRA) fine-tune of the base model, followed by retrieval-augmented generation (RAG) that pulls the top five matching chunks of course text from a vector database and prepends them to the prompt. QLoRA makes the fine-tune affordable by freezing the 4-bit quantized weights and training only small adapter matrices; RAG supplies timely, external facts without retraining. The vector database is built by chunking the course catalog text and embedding it with a sentence transformer, and the generation chain is configured with temperature 0.2, a repetition penalty of 1.1, and a 300-token cap. This combination is what the paper credits for the metric gains.

What would settle it

Have ICT instructors rate the chatbot's answers for correctness without seeing the GPT-4-generated ground truth. The paper's own sample provides an immediate test case: its fine-tuned Phi-2 answer says the HCIP course covers computer hardware and operating systems, while the reference answer describes advanced routing, switching, and security—so an expert verdict on even one such answer would show whether the metric scores track factual accuracy.

Watch

Extended reading notes

Core claim

The paper's central claim is that a retrieval-augmented, QLoRA-fine-tuned small language model is enough to answer course-guidance questions about a technical curriculum. The authors fine-tune Phi-2 (2.7 billion parameters) and Phi-3-Mini on 500 GPT-4-generated Q&A pairs drawn from 14 courses, then build a vector database from 420 Q&A pairs extracted from course catalogs in JSON, PDF, and PPT form. Three system families are compared—RAG over base models, fine-tuned models alone, and RAG over fine-tuned models—using thirteen metrics in four categories. According to the paper, the RAG-supported fine-tuned Phi-2 surpasses all other variants on almost every metric, including a reported BERTScore precision of 0.84 and F1 of 0.82, and the authors explain the surplus as successful content extraction from the vector database rather than memorization.

Load-bearing premise

The whole evaluation pipeline assumes that GPT-4's answers, generated from the same course documents used to create the training set and the retrieval database, are correct and useful; if they are not, the reported metrics measure how closely a model imitates GPT-4 rather than how well it answers learners.

Editorial extensions

If this is right

  • A 2.7-billion-parameter model fine-tuned on 500 generated Q&A pairs and backed by a 420-pair vector database scores higher than the untuned RAG systems and the fine-tune-only models on almost all 13 reported metrics.
  • The combined RAG-plus-fine-tuned system raises BERTScore F1 from 0.76 for fine-tuned Phi-2 alone to about 0.83, with precision reported at 0.84 and recall just above 0.84.
  • Because QLoRA quantizes weights to 4-bit and RAG avoids storing all knowledge in the model, the authors argue the pipeline is feasible on limited hardware, such as a single T4 GPU for the retrieval side.
  • The authors argue that RAG keeps answers timely and updatable without retraining while fine-tuning gives domain-specific phrasing, so the two mechanisms are complementary rather than competing.
  • The reported results suggest that small models can be used for closed-domain Q&A systems without the cost of retraining a large model whenever new course content appears.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported superiority may partly measure self-consistency with GPT-4 rather than factual accuracy, because the same GPT-4 model, prompted over the same documents, generated the training pairs, the RAG content, and the reference answers.
  • A human expert study could turn this from a metric-based demonstration into a deployment-ready claim; if manual ratings disagree with the metric ranking, the automatic scores would need reweighting.
  • The same recipe—QLoRA plus a small vector database—is a natural template for other closed-domain assistants, such as policy lookup or product-support bots, where a large model is too costly or private data must stay local.
  • Scaling the retrieval side, for example by increasing chunk overlap, adding more documents, or reranking passages, may be a cheaper path to improvement than scaling the language model, given how much a 420-pair database changed the small model's scores.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a domain-specific chatbot for Huawei ICT course guidance by combining QLoRA fine-tuning of Phi-2 and Phi-3 models with a retrieval-augmented generation (RAG) system built from a 420-pair vector database. The authors evaluate fine-tuned, RAG, and RAG-plus-fine-tuned configurations using BLEU, ROUGE, METEOR, and BERTScore, and claim that the RAG-supported fine-tuned Phi-2 model achieves precision 0.84 and F1 0.82 and 'surpassed other models in almost all metrics.' The experimental pipeline is described with specific hyperparameters, resource usage, and example outputs, but the evaluation is built on GPT-4-generated data on both the system side and the reference side.

Significance. If the central claim were established by a valid evaluation, the paper would be a useful low-resource recipe for domain-specific chatbots: it reports concrete QLoRA hyperparameters, chunking settings, retriever choices, and GPU measurements, and it compares several configurations rather than only one. The paper also makes a falsifiable comparative claim (RAG + fine-tuning beats base and fine-tuned models) that is practically meaningful. However, the current evaluation cannot support that claim because the ground truth, the fine-tuning data, and the RAG content all come from the same GPT-4 prompting process over the same documents, making the automatic metrics a measure of stylistic self-consistency rather than factual correctness. The lack of an independent held-out test set with human-verified answers, and the absence of statistical significance tests, further weaken the conclusions.

major comments (4)
  1. [Section III-A and Supplementary Table IX] The evaluation is circular: GPT-4 is used to generate the 500 fine-tuning Q&A pairs (Section III-A1), the 420-pair RAG content (Section III-A2), and the Ground Truth answers in Supplementary Table IX. Consequently, the high scores in Table VI (e.g., BERTScore-F1 0.827 for RAG+Phi-2) measure how closely the system reproduces GPT-4's style and content choices, not whether the answers are factually correct course guidance. This directly undermines the abstract's claim of precision 0.84 and F1 0.82, as well as the Section IV-C statement that the RAG-supported model 'surpassed other models in almost all metrics.' The authors should provide a human-verified golden standard or, at minimum, a held-out test set whose references are manually checked against the Huawei course materials and not generated by the same model used to create the training data.
  2. [Section IV, Tables IV and VI] The central comparison is not properly controlled. Table VI compares only fine-tuned Phi-2, fine-tuned Phi-3, and RAG+fine-tuned Phi-2, while Table IV reports separate RAG experiments with base models. There is no evidence that the same evaluation questions and the same Ground Truth references were used across these two tables, so the claim in Section IV-C that RAG+Phi-2 'surpassed other models in almost all metrics' does not follow from the reported numbers. A fair test of the proposed method requires a single, identical test set applied to all four configurations: base, fine-tuned, RAG+base, and RAG+fine-tuned, with the same automatic metrics and a single reference set.
  3. [Supplementary Table IX, 'What is the HCIE?' row] The chosen metrics cannot detect factual errors, and the paper contains a concrete example of such an error. The RAG+Phi-2 answer describes HCIE as 'HCIP-Cloud Computing' certification, confusing two different Huawei certification levels, yet the paper reports a high BERTScore-F1 of 0.827 for RAG+Phi-2 (Table VI) with no penalty for this error because the GPT-4-generated Ground Truth is also a generic description. This illustrates that BLEU, ROUGE, METEOR, and BERTScore, while useful for lexical/semantic similarity, are insufficient to support the paper's claim that the chatbot provides accurate course guidance. The authors should add a manual fact-check of the generated answers against the official course catalog, at least on a sample, and report the factual error rate.
  4. [Section IV-B, 'same questions as the Phi-2 model'] The test-set characteristics are missing. The paper does not state the number of evaluation questions, how they were selected, whether they overlap with the 500 fine-tuning pairs or the 420 RAG pairs, or whether any data splitting was performed. Without this information, the scores in Table VI cannot be interpreted, and the risk of contamination between training, retrieval, and evaluation is high. The authors should report the test-set size, the source of test questions, and the overlap statistics with the training and retrieval corpora.
minor comments (5)
  1. [Section III-B] The text says 'Phi-2 is a 2.7 million parameter SLM,' which is incorrect; Phi-2 has 2.7 billion parameters. Please correct the unit.
  2. [Section IV-D] The sentence 'the Phi-3-Mini model was trained with 7 billion parameters' contradicts the earlier statement in Section III-B that Phi-3-Mini uses 3.8B parameters. Clarify which model and configuration is meant.
  3. [Supplementary Material] The main text repeatedly refers to 'Table I of Supplementary Material,' but the supplementary table is labeled Table IX. Update all cross-references.
  4. [Data Availability Statements] The sentence 'The data underlying this article were provided by Huawei under by permission' is grammatically incomplete; it should read 'under permission' or 'under a data-sharing agreement with permission from Huawei.'
  5. [Section IV-A] The description of the RAG evaluation states that Phi-3-Mini-4k with the PDF source 'has the highest BLEU and METEOR metric results among all three RAG systems,' but the differences are very small (BLEU 0.051 vs. 0.043; METEOR 0.274 vs. 0.272). These differences are likely within noise, so the wording should be softened or significance testing should be provided.

Circularity Check

2 steps flagged · score 6.0 of 10

Evaluation is circular: GPT-4 generates the fine-tuning data, the RAG corpus, and the ground-truth references, so the reported metrics measure fidelity to GPT-4 rather than factual correctness.

  1. self definitional [Section III-A (Dataset), Section IV-B, Supplementary Table IX]
    "We use GPT-4 [31] for generating question-answer pairs ... Q&A datasets have been generated using Huawei Course ICT Catalog PDF and PPT documents and GPT-4 model [32] ... Ground Truth (GT) answers to each question must be prepared to calculate the metrics shown in Table VI."

    The proposed system is built from GPT-4 outputs: the 500 fine-tuning Q&A pairs (Section III-A1) and the 420-pair RAG vector database (Section III-A2) are both generated by GPT-4 from the same Huawei Talent course material. The evaluation reference (GT) used by BLEU, ROUGE, METEOR, and BERTScore is drawn from the same GPT-4-generated corpus. All four metric families score lexical or semantic similarity to that GT, so the reported numbers measure how closely the fine-tuned/RAG system reproduces GPT-4's paraphrases of the course catalog, not whether the answers are factually correct for learners. The later claim that RAG+Phi-2 'surpassed other models in almost all metrics' is therefore a comparison in GPT-4 self-similarity, not an external validation.

  2. fitted input called prediction [Abstract and Section IV-C]
    "The precision value of the Phi-2 model supported by RAG was 0.84 and the F1 score was 0.82 ... Based on the evaluation metrics, the Phi-2 model supported by the vector database has surpassed other models in almost all metrics."

    These scores are presented as evidence that the proposed method is suitable for real-life course-guidance applications. But the model was fitted on GPT-4-generated Q&A pairs, the RAG content is GPT-4-generated, and the ground-truth answers are also GPT-4-generated. The automatic metrics cannot detect factual errors that the reference itself contains: Supplementary Table IX shows RAG+Phi-2 describing HCIE as 'HCIP-Cloud Computing', conflating certification levels, yet Table VI reports BERTScore-F1 0.827 for the same system. With no independent, human-verified ground truth, the 'prediction' of real-life superiority is forced by a self-referential evaluation loop rather than by demonstrated factual accuracy.

full rationale

The paper is an empirical comparison rather than a mathematical derivation; there are no first-principles equations whose conclusions equal their assumptions, and no load-bearing self-citations (the only self-citation, [44], is contextual). The circularity is in the evaluation design: GPT-4 is used to generate the fine-tuning Q&A pairs (Section III-A1), the RAG vector-database Q&A pairs (Section III-A2), and the Ground Truth answers used for scoring (Section IV-B, Supplementary Table IX). All four metric families measure lexical or semantic similarity to the GT reference, so the reported numbers are largely a measure of agreement with GPT-4's paraphrase of the same Huawei course documents. The central claim that RAG+Phi-2 'surpassed other models in almost all metrics' therefore holds only relative to a self-referential standard; it does not demonstrate factual correctness for a course-guidance chatbot. The paper's own Supplementary Table IX even shows a factual error (HCIE described as 'HCIP-Cloud Computing') that the metrics cannot penalize. Because the comparison is still a real measurement of similarity to a fixed reference, the circularity is partial rather than total: the systems are ranked relative to each other, but the absolute 'precision 0.84, F1 0.82' and the 'real-life applications' conclusion are not independently supported. Hence score 6.

Assumptions & free parameters 11 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theory or entities; its central claim rests on hand-chosen hyperparameters and on the assumption that GPT-4-generated Q&A pairs are a valid basis for both training and evaluation. The free parameters are mostly standard generation and retrieval settings whose values are reported but not justified by ablations.

free parameters (11)
  • LoRA rank = 32
    Hyperparameter for QLoRA, chosen without reported tuning; affects adaptation capacity.
  • LoRA alpha = 64
    Scaling factor for LoRA updates, chosen by hand.
  • LoRA dropout = 0.05
    Regularization in LoRA layers.
  • learning rate = 2.5e-5
    Used with 8-bit Adam, set for 5k steps.
  • training steps = 5000 (22 epochs)
    Stopped based on loss curves; authors note steps can be cut to 3k-4k to prevent memorization.
  • temperature = 0.2
    Generation parameter, influences answer randomness.
  • repetition penalty = 1.1
    Generation parameter to avoid repeated tokens.
  • max new tokens = 300
    Generation length cap.
  • chunk size = 200
    Text chunking for vector database.
  • chunk overlap = 0
    No overlap between chunks.
  • retrieval top k = 5
    Number of retrieved chunks per query in RAG.
assumptions (3)
  • domain assumption GPT-4-generated Q&A pairs accurately represent real learner questions and correct answers for the Huawei course catalog.
    The fine-tuning set (500 pairs), RAG content (420 pairs), and ground-truth answers are all generated by GPT-4 from the same documents (Section III-A, Supplementary Table IX); if GPT-4 output is biased or wrong, every stage inherits the error.
  • domain assumption The course catalog documents (JSON, PPT, DOC) contain sufficient information to answer all evaluation questions.
    RAG retrieves only from these documents, so any question not covered by the catalog cannot be answered factually (Section III-C).
  • domain assumption Automatic lexical and embedding metrics (BLEU, ROUGE, METEOR, BERTScore) are valid indicators of answer quality for this task.
    No human evaluation is reported; the selection of the best method is based entirely on these metrics (Section IV-F).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Learning Content Retrieval with Knowledge Injection." pith.science (2026). https://pith.science/paper/IMX5IWGB

@misc{pith2026241200125,
  author       = {Pith},
  title        = {Pith review of: Efficient Learning Content Retrieval with Knowledge Injection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IMX5IWGB}},
  note         = {Machine review of arXiv:2412.00125}
}
read the original abstract

With the rise of online education platforms, there is a growing abundance of educational content across various domain. It can be difficult to navigate the numerous available resources to find the most suitable training, especially in domains that include many interconnected areas, such as ICT. In this study, we propose a domain-specific chatbot application that requires limited resources, utilizing versions of the Phi language model to help learners with educational content. In the proposed method, Phi-2 and Phi-3 models were fine-tuned using QLoRA. The data required for fine-tuning was obtained from the Huawei Talent Platform, where courses are available at different levels of expertise in the field of computer science. RAG system was used to support the model, which was fine-tuned by 500 Q&A pairs. Additionally, a total of 420 Q&A pairs of content were extracted from different formats such as JSON, PPT, and DOC to create a vector database to be used in the RAG system. By using the fine-tuned model and RAG approach together, chatbots with different competencies were obtained. The questions and answers asked to the generated chatbots were saved separately and evaluated using ROUGE, BERTScore, METEOR, and BLEU metrics. The precision value of the Phi-2 model supported by RAG was 0.84 and the F1 score was 0.82. In addition to a total of 13 different evaluation metrics in 4 different categories, the answers of each model were compared with the created content and the most appropriate method was selected for real-life applications.

Figures

Figures reproduced from arXiv: 2412.00125 by the authors.

Figure 1
Figure 1. Path for study and exam Using detailed information about a total of 20 courses found in the Huawei Course ICT Catalog PDF and PPT documents, datasets containing a total of 420 Q&A pairs have been generated to establish RAG systems. Example pairs from the generated datasets can be seen in [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Q&A pair examples from generated datasets [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. A Simple Representation of QLoRA QLoRA was developed to reduce memory usage; it can be used with 4-bit Normal Float, Double Quantization, and Paged optimizer techniques. 4-bit Normal Float can achieve 16-bit performance with 4-bit data with an equal number of elements assigned to the input tensor [22], [33]. Phi-2 is a 2.7 million parameter SLM introduced by Mi￾crosoft. Phi-2 can understand text and code, produce cu… view at source ↗
Figures from the paper (8 more)
Figure 5
Figure 5. Figure 5: Proposed Method BLEU [37], ROUGE [38], METEOR [39], and BERT [12] score. Each of these metrics evaluates different aspects of the sys￾tems. For example; BLEU is particularly valuable for assessing the context relevance, faithfulness, and answer relevance of the generat…
Figure 6
Figure 6. Figure 6: Used prompt template After performing these steps, we need a “retriever”. The retriever acts as an intermediary that facilitates communication between the generated vector database and the LLM. As mentioned earlier, we used LangChain library to create the retriever and…
Figure 7
Figure 7. Figure 7: Phi-3 Train and Validation Loss Graphs In the trained model, the training loss decreased to 0.07. When working with limited data, training can be cut into 3k − 4k steps to prevent the model from memorizing, and the number of steps can be completely rearranged according…
Figure 6
Figure 6. Figure 6: While creating the LLM chain, the tokenizer was created us￾ing the Phi-2 base model and the transformers text generation pipeline from the Python libraries was used. To create the text generation pipeline, the temperature was set as 0.2, repetition penality as 1.1 and …
Figure 8
Figure 8. Figure 8: Comparison of different models. same decrease can be observed in validation loss. This shows that the Phi-2 model can learn better than the Phi-3 model with a limited dataset. If necessary updates are made to the hyper-parameters used in training, the desired values in…
Figure 10
Figure 10. Figure 10: Phi-2 GPU Power Usage during data transfer and temperature, repetition penalty, and max new tokens used when creating the LLM chain will directly affect content extraction. If the temperature is selected high in an application, the randomness in the answers will incre…
Figure 9
Figure 9. Figure 9: Phi-2 Train and Validation Loss Graphs Parameters such as chunks size, chunks overlap determined TABLE VIII HYPER-PARAMETERS AND VALUES Parameter Value Temperature 0.2 Repetition penalty 1.1 Max new tokens 300 Chunks size 200 Chunks Overlap 0 [PITH_FULL_IMAGE:figures/…
Figure 11
Figure 11. Figure 11: Phi-3 GPU Power Usage It is known that the GPU requirement of the model will [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 15 canonical work pages

  1. [1]

    Improving language understanding by generative pre-training,

    A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al., “Improving language understanding by generative pre-training,” 2018

    2018

  2. [2]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” 2023. [Online]. Available: https://arxiv.org/abs/1910.10683

    arXiv 2023

  3. [3]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  4. [4]

    Textbooks are all you need ii: phi-1.5 technical report,

    Y . Li, S. Bubeck, R. Eldan, A. D. Giorno, S. Gunasekar, and Y . T. Lee, “Textbooks are all you need ii: phi-1.5 technical report,” 2023. [Online]. Available: https://arxiv.org/abs/2309.05463

    arXiv 2023

  5. [5]

    Mistral 7b,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al., “Mistral 7b,” arXiv preprint arXiv:2310.06825 , 2023

    arXiv 2023

  6. [6]

    A survey on evaluation of large language models,

    Y . Chang, X. Wang, J. Wang, Y . Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y . Wang, W. Ye, Y . Zhang, Y . Chang, P. S. Yu, Q. Yang, and X. Xie, “A survey on evaluation of large language models,” 2023

    2023

  7. [7]

    Augmenting black-box llms with medical textbooks for clinical question answering,

    Y . Wang, X. Ma, and W. Chen, “Augmenting black-box llms with medical textbooks for clinical question answering,” 2024. [Online]. Available: https://arxiv.org/abs/2309.02233

    arXiv 2024

  8. [8]

    A survey on legal question–answering systems,

    J. Martinez-Gil, “A survey on legal question–answering systems,” Computer Science Review, vol. 48, p. 100552, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1574013723000199

    work page 2023

Show all 46 references
  1. [9]

    Retrieval-augmented generation for large language models: A survey,

    Y . Gao, Y . Xiong, X. Gao, K. Jia, J. Pan, Y . Bi, Y . Dai, J. Sun, M. Wang, and H. Wang, “Retrieval-augmented generation for large language models: A survey,” 2024. [Online]. Available: https://arxiv.org/abs/2312.10997

    2024 arXiv

  2. [10]

    Fine- tune the entire rag architecture (including dpr retriever) for question- answering,

    S. Siriwardhana, R. Weerasekera, E. Wen, and S. Nanayakkara, “Fine- tune the entire rag architecture (including dpr retriever) for question- answering,” 2021. [Online]. Available: https://arxiv.org/abs/2106.11517 13

    2021 arXiv

  3. [11]

    Dense passage retrieval for open-domain question answering,

    V . Karpukhin, B. O ˘guz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W. tau Yih, “Dense passage retrieval for open-domain question answering,” 2020. [Online]. Available: https://arxiv.org/abs/2004.04906

    2020 arXiv

  4. [12]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  5. [13]

    Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehen- sion,

    M. Lewis, Y . Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V . Stoyanov, and L. Zettlemoyer, “Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehen- sion,” arXiv preprint arXiv:1910.13461 , 2019

    1910 arXiv

  6. [14]

    Ai-ta: Towards an intelligent question-answer teaching assistant using open-source llms,

    Y . Hicke, A. Agarwal, Q. Ma, and P. Denny, “Ai-ta: Towards an intelligent question-answer teaching assistant using open-source llms,”

  7. [15]

    Buddybot: Ai powered chatbot for enhancing english language learning,

    J. Dhivvya and S. B. Karnati, “Buddybot: Ai powered chatbot for enhancing english language learning,” in 2024 IEEE International Con- ference on Interdisciplinary Approaches in Technology and Management for Social Innovation (IATMSI) , vol. 2. IEEE, 2024, pp. 1–6

    2024

  8. [16]

    Llama-lora neural prompt engineering: A deep tuning framework for automatically generating chinese text logical reasoning thinking chains,

    S. Chen, W. Wang, X. Chen, P. Lu, Z. Yang, and Y . Du, “Llama-lora neural prompt engineering: A deep tuning framework for automatically generating chinese text logical reasoning thinking chains,” Data Intelli- gence, pp. 1–53, 2024

    2024

  9. [17]

    Fine-tuning and utilization methods of domain-specific llms,

    C. Jeong, “Fine-tuning and utilization methods of domain-specific llms,” arXiv preprint arXiv:2401.02981 , 2024

    2024 arXiv

  10. [18]

    Customising general large language models for specialised emotion recognition tasks,

    L. Peng, Z. Zhang, T. Pang, J. Han, H. Zhao, H. Chen, and B. W. Schuller, “Customising general large language models for specialised emotion recognition tasks,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2024, ...

    2024

  11. [19]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,”

  12. [20]

    Evaluation of llm chatbots for osint-based cyber threat awareness,

    S. Shafee, A. Bessani, and P. M. Ferreira, “Evaluation of llm chatbots for osint-based cyber threat awareness,” 2024. [Online]. Available: https://arxiv.org/abs/2401.15127

    2024 arXiv

  13. [21]

    Mtlora: A low-rank adaptation approach for efficient multi-task learning,

    A. Agiza, M. Neseem, and S. Reda, “Mtlora: A low-rank adaptation approach for efficient multi-task learning,” 2024. [Online]. Available: https://arxiv.org/abs/2403.20320

    2024 arXiv

  14. [22]

    Qlora: Efficient finetuning of quantized llms,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  15. [23]

    Available: https://huggingface.co/docs/bitsandbytes/main/en/ index

    [Online]. Available: https://huggingface.co/docs/bitsandbytes/main/en/ index

  16. [24]

    Fine tuning llm for enterprise: Practical guidelines and recommendations,

    M. R. J, K. VM, H. Warrier, and Y . Gupta, “Fine tuning llm for enterprise: Practical guidelines and recommendations,” 2024. [Online]. Available: https://arxiv.org/abs/2404.10779

    2024 arXiv

  17. [25]

    Roberta: A robustly optimized bert pretraining approach,

    Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” 2019. [Online]. Available: https://arxiv.org/abs/1907.11692

    2019 arXiv

  18. [26]

    Atlas: Few-shot learning with retrieval augmented language models,

    G. Izacard, P. Lewis, M. Lomeli, L. Hosseini, F. Petroni, T. Schick, J. Dwivedi-Yu, A. Joulin, S. Riedel, and E. Grave, “Atlas: Few-shot learning with retrieval augmented language models,” 2022. [Online]. Available: https://arxiv.org/abs/2208.03299

    2022 arXiv

  19. [27]

    Retrieval-augmented generation for knowledge-intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W. tau Yih, T. Rockt ¨aschel, S. Riedel, and D. Kiela, “Retrieval-augmented generation for knowledge-intensive nlp tasks,” 2021. [Online]. Available: https://arxiv.org/abs/2005.11401

    2021 arXiv

  20. [28]

    Realm: Retrieval-augmented language model pre-training,

    K. Guu, K. Lee, Z. Tung, P. Pasupat, and M.-W. Chang, “Realm: Retrieval-augmented language model pre-training,” 2020. [Online]. Available: https://arxiv.org/abs/2002.08909

    2020 arXiv

  21. [29]

    In-context retrieval-augmented language models,

    O. Ram, Y . Levine, I. Dalmedigos, D. Muhlgay, A. Shashua, K. Leyton- Brown, and Y . Shoham, “In-context retrieval-augmented language models,” 2023. [Online]. Available: https://arxiv.org/abs/2302.00083

    2023 arXiv

  22. [30]

    Available: https://e.huawei.com/en/talent/ict-academy/#/ ict-courses-list

    [Online]. Available: https://e.huawei.com/en/talent/ict-academy/#/ ict-courses-list

  23. [31]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  24. [32]

    Transformers: State- of-the-art natural language processing,

    T. Wolf, L. Debut, V . Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y . Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush, “Transformers: State- of-the-art natur...

    2020

  25. [33]

    A fine-tuning enhanced rag system with quan- tized influence measure as ai judge,

    K. Rangan and Y . Yin, “A fine-tuning enhanced rag system with quan- tized influence measure as ai judge,” arXiv preprint arXiv:2402.17081 , 2024

    2024 arXiv

  26. [34]

    Exploiting llm quantization,

    K. Egashira, M. Vero, R. Staab, J. He, and M. Vechev, “Exploiting llm quantization,” arXiv preprint arXiv:2405.18137 , 2024

    2024 arXiv

  27. [35]

    Phi- 3 technical report: A highly capable language model locally on your phone,

    M. Abdin, S. A. Jacobs, A. A. Awan, J. Aneja, A. Awadallah, H. Awadalla, N. Bach, A. Bahree, A. Bakhtiari, H. Behl, et al. , “Phi- 3 technical report: A highly capable language model locally on your phone,” arXiv preprint arXiv:2404.14219 , 2024

    2024 arXiv

  28. [36]

    Eval- uation of retrieval-augmented generation: A survey,

    H. Yu, A. Gan, K. Zhang, S. Tong, Q. Liu, and Z. Liu, “Eval- uation of retrieval-augmented generation: A survey,” arXiv preprint arXiv:2405.07437, 2024

    2024 arXiv

  29. [37]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , P. Isabelle, E. Charniak, and D. Lin, Eds. Philadelphia, Pennsylvania,...

    2002

  30. [38]

    ROUGE: A package for automatic evaluation of summaries,

    C.-Y . Lin, “ROUGE: A package for automatic evaluation of summaries,” in Text Summarization Branches Out. Barcelona, Spain: Association for Computational Linguistics, July 2004, pp. 74–81. [Online]. Available: https://aclanthology.org/W04-1013

    2004

  31. [39]

    METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,” in Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , J. Goldstein, A. Lavie, C....

    2005

  32. [40]

    Bertscore: Evaluating text generation with bert,

    T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi, “Bertscore: Evaluating text generation with bert,” arXiv preprint arXiv:1904.09675, 2019

    1904 arXiv

  33. [41]

    LangChain,

    C. Harrison, “LangChain,” Oct. 2022. [Online]. Available: https: //github.com/langchain-ai/langchain

    2022

  34. [42]

    Bitsandbytes,

    T. Dettmers, “Bitsandbytes,” Nov. 2021. [Online]. Available: https: //github.com/TimDettmers/bitsandbytes

    2021

  35. [43]

    The faiss library,

    M. Douze, A. Guzhva, C. Deng, J. Johnson, G. Szilvasy, P.-E. Mazar ´e, M. Lomeli, L. Hosseini, and H. J ´egou, “The faiss library,” 2024

    2024

  36. [44]

    Assessing fine-tuning efficacy in llms: A case study with learning guidance chatbots,

    R. Bayraktar, B. Sarıt ¨urk, and M. E. Erdem, “Assessing fine-tuning efficacy in llms: A case study with learning guidance chatbots,” Inter- national Journal of Innovative Science and Research Technology, vol. 9, no. 5, pp. 2461–2471, May 2024. 14 SUPPLEMENTARY MATERIAL : E FF...

    2024

  37. [2021]

    Available: https://arxiv.org/abs/2106.09685

    [Online]. Available: https://arxiv.org/abs/2106.09685

    arXiv

  38. [2023]

    Available: https://arxiv.org/abs/2311.02775

    [Online]. Available: https://arxiv.org/abs/2311.02775

    arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.