REVIEW 4 major objections 4 minor 1 cited by
Learnware of Language Models: Specialized Small Language Models Can Do Big
T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read A dock of about 100 specialized 8B language models, selected per task by parameter-vector matching, beats much larger general LLMs in finance and medicine.
desk verdict A novel LoRA parameter-vector specification for matching specialized SLMs, with headline results that rest on a transductive evaluation protocol where the user spec is fit to the full labeled test set. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the parameter-vector specification: a small language model is fine-tuned to fit the target conditional distribution—the developer's model output $p(h(x)|x)$ or the user's labels $p(y|x)$—and only the LoRA matrix $B$ is updated, so the specification is the flattened $B$ of a low-rank update of the form $BA$. This compresses model capability to under a million parameters. The same machinery generates the user's task vector, and cosine similarity between user and model vectors is the selection rule. It does the work of the whole paradigm here: it connects a task to a model without exposing raw data, without loading and evaluating every candidate, and without training a router on auxiliary datasets.
What would settle it
Choose a finance task from the paper's benchmark, build the user's task vector from only a few dozen labeled examples (or from unlabeled inputs with predicted labels), select one learnware by the cosine rule, and evaluate on the held-out test split. If accuracy falls to the random-selection level, the claim that the specification identifies useful specialists only holds in the full-data, in-sample setting and not in the privacy-preserving few-shot setting the paradigm targets.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that a collection of specialized 8B SLMs—fine-tuned on separate instruction datasets—can be organized as learnwares and outperform both the base models and much larger general LLMs on specialized benchmarks. For each model, the developer fine-tunes a small auxiliary LM in LoRA space to fit the model's predictions, producing a flattened $B$ matrix as the specification; the user does the same on her labeled task data to obtain a task vector. Selecting the learnware with smallest cosine distance for each task yields an average score of 66.60 in finance versus 58.35 for Qwen2.5-72B and 58.25 for Qwen1.5-110B, and 75.13 in medicine versus 74.70 for Flan-PaLM-540B. In mathematics the same system improves over random selection and best-single but still trails Qwen1.5-110B, which the paper attributes to reasoning abilities that 8B specialists lack. The paper frames these as preliminary evidence that the learnware paradigm can organize specialized SLMs into a scalable, privacy-preserving alternative to a monolithic LLM.
Load-bearing premise
The reported gains rest on the user building her requirement specification from the full labeled task data and then being scored on that same data, so the paper does not show that selection still works when only a small labeled sample or unlabeled data is available.
Editorial extensions
If this is right
- In finance tasks, the dock beats Qwen1.5-110B, Qwen2.5-72B, and Llama3.1-70B-Instruct by at least 14%, so a portfolio of 8B specialists can replace a large general model for these tasks.
- Because the dock also beats its own best single fine-tuned model and random selection, the advantage comes from the specification-based selection mechanism, not from one exceptional specialist.
- In medicine, the dock's 8B models surpass Flan-PaLM-540B on the Open Medical LLM Leaderboard, indicating the same selection mechanism transfers to a second domain.
- In mathematics, selection improves over random and best-single but does not beat the 110B model, so the mechanism is limited when tasks need reasoning that 8B specialists do not possess.
- The workflow keeps raw data private on both sides: developers submit only models plus parameter-vector specs, and users submit only the vector computed from their own data.
Reading between the lines
- The paper leaves untested whether the 14-percent finance advantage survives when the user's requirement vector is built from a small labeled sample; that is the natural next experiment, since the paradigm's privacy story depends on it.
- Because the specification fits $p(y|x)$ rather than the input distribution, the matching may transfer across tasks that share label structure; the paper's identification analysis already shows one FPB-tuned model being selected for several other classification tasks, which suggests a testable clustering of tasks by label-conditional similarity.
- The scheme can be viewed as a router learned from parameter vectors without preference data or auxiliary datasets; if it holds, it would scale to growing model markets without retraining the router, but it inherits the assumption that cosine distance in LoRA space orders usefulness, which the paper does not directly verify.
- A further testable extension is to replace the single-model-per-task selection with assembling multiple learnwares for one task, which the learnware paradigm allows but this paper does not explore.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper applies the learnware paradigm to language models by constructing a simulated dock of roughly one hundred 8B-parameter specialized SLMs fine-tuned on finance, healthcare, and mathematics datasets. Each learnware consists of a model plus a parameter-vector specification obtained by fitting the model's predictions with a LoRA-style low-rank adaptation. Users generate a task specification from their own labeled data using Algorithm 2 and the system selects the learnware with the highest cosine similarity of the specification vectors. The central empirical claim is that under a 'Task-Level' evaluation protocol the system outperforms the base SLMs on all benchmarks, beats Qwen1.5-110B, Qwen2.5-72B, and Llama3.1-70B-Instruct by at least 14% in finance, and surpasses Flan-PaLM-540B in medicine, while acknowledging that it loses to Qwen1.5-110B on the mathematics benchmark.
Significance. If the results hold, the paper provides one of the first systematic demonstrations that a collection of specialized small language models, coordinated by specification matching rather than by training a router on external data, can rival or exceed much larger general-purpose LLMs in specialized domains while keeping raw data local. The paper is honest about the mathematics results, where the large model wins, and it ships code and integrates the method into the Beimingwu platform, which are concrete strengths. The key open question is whether the reported advantages survive when the user's task specification is built from a small labeled sample rather than from the full evaluation dataset, and whether the benchmark construction in finance introduces a favorable bias.
major comments (4)
- [Section 4.1.1 and Algorithm 2] The Task-Level evaluation protocol is transductive: the text states that 'the user can generate her requirement specification from the whole task data,' and Algorithm 2 fits the task vector to labeled examples (x,y). The same data are then used to compute accuracy. This means the selection of the learnware is in-sample with respect to the test labels, and the reported gains over 70B+ LLMs may reflect the selector having effectively seen the test set. The paper should provide experiments that vary the amount of labeled data available for building the user specification (e.g., 5%, 20%, 50% of the task data) with evaluation on a held-out split, or an unlabeled-data variant of the specification, to establish that the matching quality and the performance margins persist in the realistic few-label or label-free setting.
- [Appendix B.1.2 and Table A4] The financial evaluation benchmark is constructed by selecting 17 of the 34 FinBen tasks based on the three 8B base models' scores falling between 20 and 90 on those tasks. Because the learnware candidates are fine-tuned from these same base models, this task-selection rule can bias the benchmark in favor of the learnware system and against the large LLMs, particularly for tasks where the base models score near zero or near ceiling. The paper should either report results on all 34 FinBen tasks or provide a principled justification for excluding the other 17 tasks, along with a sensitivity analysis that checks whether the 'at least 14%' margin in Section 4.1.2 is robust to task selection.
- [Section 4.1.3 and Table 2] The claim that the system 'surpasses Flan-PaLM-540B' in healthcare rests on a 0.43-point average advantage (75.13 vs. 74.70) while the per-task win/tie/loss against Flan-PaLM-540B is 3/0/6. Given the lack of error bars or significance testing, this narrow average margin combined with a negative per-task record is not sufficient to support the abstract's wording of 'surpasses.' The authors should temper the claim, report confidence intervals, or provide a per-task discussion that explains why the average is a meaningful summary when the system loses on the majority of tasks.
- [Section 4.1.2 and Table 1] The 'at least 14%' improvement over large LLMs is a relative improvement computed on the selected 17-task benchmark, and the paper itself notes that the three forecasting tasks (BigData22, ACL18, CIKM18) are problematic because fine-tuned models underperform their base versions. Because the selection procedure and the transductive evaluation are both potentially favorable to the Learnware column, the robustness of the headline margin is not established. A sensitivity analysis that removes the three forecasting tasks or reports results under a small-sample user specification would clarify whether the 14% claim is an artifact of the protocol.
minor comments (4)
- [Appendix B.2.2] Several typos appear in this section, including 'fist' for 'first', 'ourperform' for 'outperform', 'axonomy' for 'taxonomy', and 'strangeness' used in an unclear way; these should be corrected.
- [Algorithm 2 and Section 3] Algorithm 2 explicitly requires labeled user data (x,y), so the privacy-preserving claim should be stated precisely: the user's raw data stay local, but the user must possess labels to build a task specification. The paper should acknowledge this limitation in Section 3, as it currently presents the workflow as if unlabeled data suffice.
- [Tables 1 and 2] The 'win/tie/loss' rows count ties against the same model when scores are equal, but the convention is not defined; for example, Table 1 reports 11 ties with Oracle, while Table 2 reports 6 ties. Adding a footnote that defines a tie and explains why ties with Oracle are possible would improve readability.
- [Figure A4] The node labels in Figure A4 (e.g., 'learnware-sm_cikm', 'user-multifin_en') are not defined in the caption or the text; a brief explanation of the naming convention would make the identification results interpretable.
Circularity Check
Headline gains rest on a transductive protocol: the user's task specification is fitted to the full labeled task data (Task-Level evaluation) and accuracy is then reported on that same task, so the selector has effectively seen the test labels.
-
fitted input called prediction
[Section 4.1.1 (Task-Level evaluation); Algorithm 2 and Eq. (1) in Appendix A]
"Following the pipeline described in Sections 2 and 3, we generate specifications for each model, and select the most suitable learnware for user task requirements. The identified learnware is then evaluated on the corresponding task to assess overall system performance under the Task-Level evaluation setting. Task-Level evaluation means that the user can generate her requirement specification from the whole task data."
Algorithm 2 fits the user task vector tau_u to the labels y of D_u, and Eq. (1) defines tau_u as the minimizer of L(g(f(x, theta_0 + tau)), y) over (x, y) in D_u. Under Task-Level evaluation, D_u is 'the whole task data', and the same corresponding task is later used to compute the reported accuracy. Thus the selector is constructed from the very labels on which the selected model is scored; the reported margins over 70B+ LLMs are not an out-of-sample test of specification matching. No experiment with a small labeled sample or with unlabeled user data is reported, so the headline claim that specialized SLMs can outperform large LLMs is supported only by an in-sample, transductive selection procedure rather than by a genuine prediction.
full rationale
The core specification construction is not circular in the usual mathematical sense: parameter-vector specifications for models and users are generated with fixed hyperparameters by fitting p(y|x) or p(h(x)|x), and no benchmark accuracy is used to tune the specification parameters. Self-citations to prior learnware work are contextual and not load-bearing for the empirical results. However, the headline evaluation is transductive: the user's task vector is fitted to the full labeled task data under Task-Level evaluation, and the same task is then scored. This gives the selector access to the test labels before the 'prediction' is made, so the reported gains over 70B+ LLMs are partly an artifact of the evaluation protocol rather than demonstrated generalization to unseen tasks. The paper itself flags only the limited characterization of advanced LLM capabilities, not this protocol leakage. Because the central claim reduces to an in-sample selection procedure, the circularity score is 6 rather than 0-2.
Assumptions & free parameters
free parameters (5)
- Spec generator training steps =
400
- Spec generator LoRA rank and alpha =
rank=16, alpha=32
- Spec generator learning rate and weight decays =
peak lr 1e-5, L2 decay 0.5, L1 decay 1.0
- Spec generator base model =
Qwen2.5-0.5B
- Finance benchmark task-selection interval =
scores between 20 and 90 on at least one base model
assumptions (6)
- domain assumption LoRA low-rank factorization tau approximately equals BA faithfully approximates the task vector, with A randomly initialized and fixed.
- domain assumption The conditional distribution p(y|x) for developers and p_u(y|x) for users is a sufficient representation of model capability and task requirement for identification.
- domain assumption Cosine similarity between flattened B vectors ranks candidate learnwares by suitability for the user task.
- domain assumption The re-implemented lm-evaluation-harness tasks faithfully reproduce the official FinBen, Open Medical LLM Leaderboard, and MathEval scores.
- domain assumption The user can generate her requirement specification from the entire labeled task dataset.
- ad hoc to paper The relative performance of base models on the full 34-task FinBen benchmark is a valid basis for selecting which tasks enter the evaluation benchmark.
Cite this review
Pith. "Pith review of Learnware of Language Models: Specialized Small Language Models Can Do Big." pith.science (2026). https://pith.science/paper/EO7SPL6O
@misc{pith2026250513425,
author = {Pith},
title = {Pith review of: Learnware of Language Models: Specialized Small Language Models Can Do Big},
year = {2026},
howpublished = {\url{https://pith.science/paper/EO7SPL6O}},
note = {Machine review of arXiv:2505.13425}
}
read the original abstract
The learnware paradigm offers a novel approach to machine learning by enabling users to reuse a set of well-trained models for tasks beyond the models' original purposes. It eliminates the need to build models from scratch, instead relying on specifications (representations of a model's capabilities) to identify and leverage the most suitable models for new tasks. While learnware has proven effective in many scenarios, its application to language models has remained largely unexplored. At the same time, large language models (LLMs) have demonstrated remarkable universal question-answering abilities, yet they face challenges in specialized scenarios due to data scarcity, privacy concerns, and high computational costs, thus more and more specialized small language models (SLMs) are being trained for specific domains. To address these limitations systematically, the learnware paradigm provides a promising solution by enabling maximum utilization of specialized SLMs, and allowing users to identify and reuse them in a collaborative and privacy-preserving manner. This paper presents a preliminary attempt to apply the learnware paradigm to language models. We simulated a learnware system comprising approximately 100 learnwares of specialized SLMs with 8B parameters, fine-tuned across finance, healthcare, and mathematics domains. Each learnware contains an SLM and a specification, which enables users to identify the most relevant models without exposing their own data. Experimental results demonstrate promising performance: by selecting one suitable learnware for each task-specific inference, the system outperforms the base SLMs on all benchmarks. Compared to LLMs, the system outperforms Qwen1.5-110B, Qwen2.5-72B, and Llama3.1-70B-Instruct by at least 14% in finance domain tasks, and surpasses Flan-PaLM-540B (ranked 7th on the Open Medical LLM Leaderboard) in medical domain tasks.
Figures
Forward citations
Cited by 1 Pith paper
-
Delta Activations: A Representation for Finetuned Large Language Models
Delta Activations embed finetuned LLMs as the average difference in hidden states between the finetuned model and its base model on a small set of generic prompts, yielding domain clusters and approximate additive com...
Reference graph
Works this paper leans on
-
[1]
Phi-3 technical report: A highly capable language model locally on your phone
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219,
-
[2]
It means that we are sketching the required capabilitypu(y|x) for the user task (Lu,Du). Algorithm 2 Build the parameter vector specification for users Require: The user task (Du,Lu) required to solve. 1: initialize pre-trained modelf (x, θ0 + τ ) with an all-0 task vector τ 2: construct functiong such thatg◦f (x, θ0 + τ )∈Y if necessary 3: repeat 4: for ...
work page 2022
-
[3]
Training verifiers to solve math word problems
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv:2110.14168,
-
[4]
Evan Frick, Connor Chen, Joseph Tennyson, Tianle Li, Wei-Lin Chiang, Anastasios N Angelopoulos, and Ion Stoica. Prompt-to-leaderboard. arXiv:2502.14855,
-
[5]
URL https://zenodo.org/records/12608602. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and et al. The llama 3 herd of models. arXiv:2407.21783,
-
[6]
Actually we first compare the performance of the three base models on the above datasets and try to fine-tune the base model with relatively better performance on the corresponding dataset. Therefore, finally in our model hub, there are fine-tuned Llama3.1-8B on FiQA-SA dataset, fine-tuned Llama3.1- 8B-Instruct on Australian dataset, and fine-tuned Qwen2....
-
[7]
Measuring massive multitask language understanding
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv:2009.03300,
arXiv 2009
-
[8]
The LoRA rank is set to 16, alpha is set to 32, and the target modules is set to "q_proj", "k_proj" and "v_proj". Training of each parameter vector is conducted on one A100 GPU. Others. Since the specification generation for developers in our experiments is to fit p(y|x) but notp(h(x)|x), we do not distinguish between different models fine-tuned with the ...
work page 2025
Show all 31 references
-
[9]
Yihuai Lan, Lei Wang, Qiyuan Zhang, Yunshi Lan, Bing Tian Dai, Yan Wang, Dongxiang Zhang, and Ee-Peng Lim
URL https://huggingface.co/akjindal53244/Arithmo-Mistral-7B . Yihuai Lan, Lei Wang, Qiyuan Zhang, Yunshi Lan, Bing Tian Dai, Yan Wang, Dongxiang Zhang, and Ee-Peng Lim. Mwptoolkit: An open-source framework for deep learning-based math word problem solvers. arXiv:2109.00799,
-
[10]
Small language models for application interactions: A case study
Beibin Li, Yi Zhang, Sébastien Bubeck, Jeevan Pathuri, and Ishai Menache. Small language models for application interactions: A case study. arXiv preprint arXiv:2405.20347,
-
[11]
Cmmlu: Measuring massive multitask language understanding in chinese
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv:2306.09212, 2023a. Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, and You Zhang. Chatd...
-
[12]
Lila: A unified benchmark for mathematical reasoning
Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. Lila: A unified benchmark for mathematical reasoning. arXiv:2210.17517,
-
[14]
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y .K. Li, Y . Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300,
-
[15]
Language models are mul- tilingual chain-of-thought reasoners
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush V osoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. Language models are mul- tilingual chain-of-thought reasoners. arXiv:2210.03057,
-
[16]
Large language models encode clinical knowledge
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. arXiv:2212.13138,
-
[17]
A comparative study between full- parameter and lora-based fine-tuning on chinese instruction data for instruction following large language model
Xianghui Sun, Yunjie Ji, Baochang Ma, and Xiangang Li. A comparative study between full- parameter and lora-based fine-tuning on chinese instruction data for instruction following large language model. arXiv:2304.08109,
-
[18]
Handling learnwares from heterogeneous feature spaces with explicit label exploitation
Peng Tan, Hai-Tian Liu, Zhi-Hao Tan, and Zhi-Hua Zhou. Handling learnwares from heterogeneous feature spaces with explicit label exploitation. In Advances in Neural Information Processing Systems 37 (NeurIPS), pages 12767–12795, 2024a. Peng Tan, Zhi-Hao Tan, Yuan Jiang, and Zh...
-
[19]
11 Chien Van Nguyen, Xuan Shen, Ryan Aponte, Yu Xia, Samyadeep Basu, Zhengmian Hu, Jian Chen, Mihir Parmar, Sasidhar Kunapuli, Joe Barrow, et al
URL https://qwenlm.github.io/blog/qwen1.5/. 11 Chien Van Nguyen, Xuan Shen, Ryan Aponte, Yu Xia, Samyadeep Basu, Zhengmian Hu, Jian Chen, Mihir Parmar, Sasidhar Kunapuli, Joe Barrow, et al. A survey of small language models. arXiv preprint arXiv:2410.20011,
-
[20]
A comprehensive survey of small language models in the era of large language models: Techniques, enhancements, applications, collaboration with llms, and trustworthiness
Fali Wang, Zhiwei Zhang, Xianren Zhang, Zongyu Wu, Tzuhao Mo, Qiuhao Lu, Wanjing Wang, Rui Li, Junjie Xu, Xianfeng Tang, et al. A comprehensive survey of small language models in the era of large language models: Techniques, enhancements, applications, collaboration with llms,...
-
[21]
Pixiu: A large language model, instruction data and evaluation benchmark for finance
Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. Pixiu: A large language model, instruction data and evaluation benchmark for finance. arXiv:2306.05443, 2023a. Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhan...
-
[22]
Identifying helpful learnwares without examining the whole market
Yi Xie, Zhi-Hao Tan, Yuan Jiang, and Zhi-Hua Zhou. Identifying helpful learnwares without examining the whole market. In Proceedings of the 26th European Conference on Artificial Intelligence (ECAI), pages 2752–2759, 2023b. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo...
-
[23]
Metamath: Bootstrap your own mathematical questions for large language models
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv:2309.12284,
-
[24]
Mammoth: Building math generalist models through hybrid instruction tuning
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv:2309.05653,
-
[25]
Alpacare: Instruction-tuned large language models for medical application
Xinlu Zhang, Chenxin Tian, Xianjun Yang, Lichang Chen, Zekun Li, and Linda Ruth Petzold. Alpacare: Instruction-tuned large language models for medical application. arXiv:2310.14558,
-
[28]
For a given instruction tuning dataset, we try multiple sets of hyperparameters
The LoRA target modules is set to all-linear. For a given instruction tuning dataset, we try multiple sets of hyperparameters. We use validation set loss to select the best checkpoint and hyperparameters. Training is conducted on one A100 node containing 4 GPUs. Evaluation det...
2024
-
[100]
The better score is emphasized in bold. Dataset Metric Single-Combined Learnware Oracle Australian Acc 45.68 56.83 56.83 LendingClub Acc 84.60 92.07 92.07 FiQA-SA Acc 74.90 76.38 76.38 FPB Acc 82.17 84.25 84.25 German Acc 66.00 67.06 67.06 Headlines AvgF1 92.80 95.61 95.61 NER...
2024
-
[2020]
Measuring mathematical problem solving with the math dataset
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv:2103.03874,
-
[2021]
Agieval: A human-centric benchmark for evaluating foundation models
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv:2304.06364,
-
[2022]
Orca-math: Unlocking the potential of slms in grade school math
Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of slms in grade school math. arXiv:2402.14830,
-
[2023]
Medalpaca–an open-source collection of medical conversational ai models and training data
Tianyu Han, Lisa C Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexan- der Löser, Daniel Truhn, and Keno K Bressem. Medalpaca–an open-source collection of medical conversational ai models and training data. arXiv:2304.08247,
-
[2024]
Mathqa: Towards interpretable math word problem solving with operation-based formalisms
Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. arXiv:1905.13319,
1905 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.