Pruning General Large Language Models into Customized Expert Models

Guizhen Chen; Kenji Kawaguchi; Lidong Bing; Wenxuan Zhang; Yirao Zhao

REVIEW 4 major objections 6 minor 1 cited by

Pruning General Large Language Models into Customized Expert Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a large general language model can be pruned into a smaller expert model tuned to a specific language, domain, and task by removing only the neurons that are irrelevant across all three dimensions, with no…

desk verdict A useful pruning recipe undermined by an unverified gap between its formal definition and its reported pruning ratios. read the letter →

arxiv 2506.02561 v1 pith:6FER66X3 submitted 2025-06-03 cs.CL cs.AI

Yirao Zhao , Guizhen Chen , Kenji Kawaguchi , Lidong Bing , Wenxuan Zhang This is my paper

classification cs.CLcs.AI

keywords modelpruningexpertmodelscustomizationneuronimportanceLLMcompressionmultilingualdomainadaptationnopost-training

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Cus-Prun is a method for turning a large general-purpose language model into a smaller expert model without any post-training. The paper positions the target use case along three dimensions—language, domain, and task—and, for each dimension, identifies the neurons whose removal least changes model output on a small corpus. The expert model is built by pruning only the neurons that are irrelevant across every specified dimension, so neurons that any dimension depends on are kept. Across multilingual, multidomain, and multitask benchmarks, the paper reports that this consistently outperforms existing no-retraining pruning methods while preserving most general ability, on models from 8B to 70B parameters.

What carries the argument

The machinery is per-neuron output-perturbation scoring. Each neuron $N_i^{(l)}$ is an individual row or column in a parameter matrix of an attention or feed-forward layer, and its relevance to a document $c$ is measured as the $L^2$ distance between the layer output with the neuron and the layer output without it. Neurons whose impact falls in the lowest $\sigma\%$ are declared irrelevant for that dimension. Corpora for each dimension are constructed by ablating the other dimensions, so the language corpus, for example, spans many domains and tasks in that language. The final expert model removes the intersection of the per-dimension irrelevant sets, and the paper adopts a parallel version of the sequential detection loop for efficiency.

What would settle it

Generate two different 50-document corpora for the same language-domain-task profile using different prompt styles or sources, run Cus-Prun with the same pruning ratio on both, and compare the selected irrelevant-neuron sets and downstream test scores; if the sets or scores disagree substantially, the method's output is corpus-sensitive rather than a stable property of the target expertise.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that customized expertise can be carved out of a dense LLM by intersecting, rather than pooling, the sets of neurons that are irrelevant to each specified dimension. Given a requested expert model described by a language, a domain, and a task, the paper builds corpora that vary the unspecified dimensions, scores each neuron by the $L^2$ change in its layer's output when the neuron is removed, and flags the lowest $\sigma\%$ of neurons as irrelevant for that dimension. The pruned model is the original model with only the intersection of those per-dimension irrelevant sets removed, so neurons that any dimension depends on survive. The paper's experiments report that this preserves 83–94% of dense-model performance in three proof-of-concept scenarios and beats SliceGPT, LLM-Pruner, and ShortGPT by large margins on expert tasks while degrading general benchmarks much less than those baselines.

Load-bearing premise

The load-bearing premise is that a neuron judged unimportant by measuring output changes on a small corpus for each dimension—sometimes only 50 generated documents—will also be unimportant for the real target use case; if those corpora are unrepresentative, pruned models could lose exactly the capabilities the user needs.

Editorial extensions

If this is right

Users could produce a German-medical-QA model, a Chinese-e-commerce-sentiment model, or any similar language-domain-task combination from one general base model using small text corpora and no retraining.
Specialization granularity is adjustable: the same scoring works for one dimension, two dimensions, or all three, so a single pipeline covers language-only, language-domain, and full-profile customization requests.
Because pruning removes only neurons irrelevant to all chosen dimensions, general English capabilities are preserved much better than with layer removal or matrix approximation baselines; for example, Llama3-8B retains a 51.4 average general score versus at most 22.3 for baselines at 25% pruning.
The method extends beyond 25% pruning: combined with layer pruning, it keeps MMLU at 48.4 and expert capability at 50.6 on Llama3-8B when 45% of parameters are removed, where ShortGPT nearly collapses.
The approach transfers across model families and sizes, with reported gains on Llama3-8B, Mistral-12B, Llama2-13B, and Llama3-70B in multilingual, multidomain, and multitask settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The intersection rule implies an inherent tradeoff the paper does not quantify: adding more dimensions shrinks the prunable set, so a three-dimensional expert model will likely be less compressed than a one-dimensional one; measuring that tradeoff would tell users when customization costs too much size reduction.
Because the paper relies on small corpora that can be manually collected or generated, a natural test is whether corpus quality and coverage, rather than the pruning rule itself, drive the gains; comparing neuron sets selected from different small corpora would settle this.
The paper leaves open whether pruned expert models can be post-trained; if they can, Cus-Prun could serve as a cheap initialization for further fine-tuning, and if not, its no-retraining property is the ceiling.
The same per-neuron scoring could be applied to other customization axes the paper names as limitations, such as query format or input structure, giving a direct extension to interaction-style expertise.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Charles, quick read of arXiv:2506.02561. Short version: the idea is sensible and the experiments are broad, but the paper's formal pruning rule and its stated pruning ratios don't obviously line up, and the "consistently outperforms" headline is too strong.

What's actually new: the paper takes the language-specific neuron idea from Tang et al. and Zhao et al., extends it to domain and task dimensions, and prunes the intersection of the per-dimension irrelevant sets. That's a clean, practical recipe, and it produces expert models without post-training. The testbed is solid: four model families/sizes, several multilingual/domain/task benchmarks, and code is public. The method is simple enough to reproduce.

The soft spots are real. The formal definition in Eq. 2 and Algorithm 1 keeps a neuron only if it's in the bottom-sigma impact set for every document in the dimension corpus. For a 50-document corpus, that intersection should be tiny unless per-document rankings are almost perfectly correlated. The paper never reports the actual number of pruned neurons or post-pruning parameter count. If the implementation prunes a true 25%, it's not following the equations; if it follows the equations, Table 1 compares Cus-Prun at a lower pruning ratio than the baselines. Either way, the advertised comparison isn't established. This is easy to fix by reporting actual counts, but it's load-bearing.

The evaluation also lacks error bars or multiple seeds. The margins are often huge, so significance is likely, but we can't confirm. The multilingual setup uses English splits of the evaluation datasets (GSM8K for MGSM, SQuAD for XQuAD, XSum for XLSum) as the corpus, which is tuning on data very close to the test sets. And the abstract's "consistently outperforms" is contradicted by their own Table 1 for Llama3-8B multitask, where LLM-Pruner scores higher; they disclose this in Section 4.4, but the abstract overclaims. The GPT-4o-generated corpora in the preliminary study are a minor uncontrolled variable; real corpora elsewhere help.

The authors do include a limitations section and are transparent about the one exception, which counts in their favor.

Who this is for: practitioners who want fast, task-specific pruning without retraining. The core idea is plausible and worth a serious look. But the pruning-ratio ambiguity needs to be resolved before the headline numbers can be trusted.

My recommendation: send it to review with major revision. Ask for actual pruned parameter counts, error bars, and a revision of the "consistent" claim. The paper deserves referee time.

Referee Report

4 major / 6 minor

Summary. The paper proposes Cus-Prun, a training-free structural pruning method that customizes a general LLM into an expert model along three dimensions (language, domain, task). For each selected dimension, it constructs a corpus, measures the output-perturbation impact of removing each neuron, marks neurons in the lowest-σ impact set as irrelevant per document, and then prunes the intersection of the per-dimension irrelevant-neuron sets. Experiments are reported on Llama3-8B, Mistral-12B, Llama2-13B, and Llama3-70B across multilingual, multidomain, and multitask benchmarks, together with one- and two-dimensional adaptive settings, claiming that Cus-Prun consistently outperforms LLM-Pruner, SliceGPT, and ShortGPT at a nominal 25% pruning ratio and remains effective up to 45% when combined with layer pruning.

Significance. The idea of identifying irrelevant neurons along user-specified dimensions and pruning their intersection is simple, practical, and aligns with a real need for compact, deployment-ready expert models without post-training. The paper's strengths include a clearly described algorithm, public code, experiments across multiple model families and sizes, and an honest limitation section. If the central quantitative claims were fully supported, this would be a useful contribution to the pruning literature. However, as written, the claimed improvements are not yet established: the formal pruning rule may not actually remove the advertised fraction of parameters, and the pruning corpora partly overlap with the evaluation benchmarks, so the headline comparisons in Table 1 and Table 3 are not verified.

major comments (4)

[§2.1, Eq. (2), Algorithm 1, Table 1, Table 3] The formal selection rule in Eq. (2) and Algorithm 1 (lines 13–17) defines a neuron as irrelevant only if it is in the lowest-σ impact set for every document in the corpus, and Algorithm 1 has no budget-enforcement step that makes the final pruned set equal to the requested σ. The paper never reports the actual number of pruned neurons or the post-pruning parameter count for any Cus-Prun model. If the intersection of per-document bottom-σ sets is much smaller than σ, then the equal-ratio comparisons in Table 1 and the 25%/35%/45% claims in Table 3 compare models of different actual sizes. If the implementation instead enforces the nominal ratio by ranking neurons on averaged impact, it does not match the formal definition. Please report actual post-pruning parameter counts, add a budget-adjustment step that guarantees the advertised sparsity, or re-run the comparisons at matched actual sparsity.
[§4.2, Appendix A.3.1] In the multilingual setting, the corpus C(D,T)_Exp is constructed from the English splits of the same datasets used for evaluation: GSM8K for MGSM, the English split of M3Exam for M3Exam, SQuAD for XQuAD, and XSum for XLSum. This is benchmark leakage: the pruning decisions are derived from material drawn from the target benchmarks' own distributions, so the reported expert-capability scores are not an independent evaluation of customization to a new use case. Additionally, hyperparameters are tuned on the validation set of XLSum and then applied to the other multilingual test sets. Please construct pruning corpora that do not overlap with evaluation datasets, or provide a clear argument that this overlap does not inflate the results.
[§4.1, Table 1, Figure 2, Figure 4] All reported results come from single runs with no error bars, no multiple seeds, and no variance estimates. Several claimed improvements are modest (e.g., Llama2-13B multidomain average: Cus-Prun 33.1 vs. SliceGPT 30.3 and ShortGPT 30.3), and in the multitask setting on Llama3-8B, LLM-Pruner achieves 54.7 while Cus-Prun achieves 52.2, an exception the text acknowledges. Without repeated runs or confidence intervals, the claim that Cus-Prun "consistently outperforms" other methods is not statistically supported. Please provide standard deviations or repeated-seed results, especially for the close comparisons.
[§3, §4.5, Table 3] The preliminary evaluation in Section 3 uses only 50 GPT-4o-generated documents per dimension, and the aggressive-pruning experiments in Section 4.5 combine layer pruning with Cus-Prun without separating the contributions of the two components. The paper reports no sensitivity analysis for corpus size or corpus generator, and it does not report how much of the 35% and 45% parameter reduction in Table 3 comes from layer pruning versus neuron pruning. As a result, the claim that Cus-Prun remains effective "up to nearly half the parameters" is hard to interpret. Please report the actual sparsity contributed by each component and include a corpus-size ablation.

minor comments (6)

[§3, Main Results] In the sentence "Cus-Prun largely preserves the performance of the dense model, retraining 92%, 83%, and 94%," the word "retraining" should be "retaining."
[§4.3 heading] The section heading "MultilDomain Setting" contains a typo and should read "Multidomain Setting."
[Abstract, §4.4] The abstract and conclusion state that Cus-Prun "consistently outperforms" other methods, but §4.4 explicitly reports an exception for LLM-Pruner on Llama3-8B in the multitask setting; the summary claims should be qualified accordingly.
[Appendix A.3.3] The appendix contains a typo, "conterfact," and the citation "Amazon Summary (Wang et al., 2022;?)" has a placeholder question mark that should be completed.
[Table 1] Several entries in Table 1 are visually corrupted by adjacent numbers running together (e.g., "32.946.7", "0.061.876.0", "23.2.18.0"), which makes the table difficult to read.
[Tables 5–7] The appendix tables use "Expert Capability" and "Specific Capability" inconsistently for the same column; the terminology should be unified.

Circularity Check

2 steps flagged · score 6.0 of 10

Expert-capability scores are partly in-sample: pruning masks are fitted on English versions of the evaluation benchmarks (and once on the evaluation corpus itself).

fitted input called prediction [Section 4.2 (multilingual setting); Appendix A.3.1; Table 1]
"For C(D,T) Exp, we employ the corresponding datasets in English, including GSM8K (Cobbe et al., 2021) for MGSM, the English split of M3Exam (Zhang et al., 2023) for M3Exam, SQuAD (Rajpurkar, 2016) for XQuAD, and XSum (Narayan et al., 2018) for XLSum."

Table 1's 'Multilingual Expert' column is computed on MGSM, M3Exam, XQuAD, and XLSum. The C(D,T) pruning corpus is built from the English versions of precisely those benchmarks: GSM8K for MGSM, English M3Exam for M3Exam, SQuAD for XQuAD (XQuAD is a translation of SQuAD), and XSum for XLSum. Thus the domain-task mask is fitted to the same task content on which expert performance is later reported; the evaluation is in-sample at the task/content level. The remaining English-to-German/Chinese/Thai transfer is real, so this is not total circularity, but the 'expert capability' number is largely a fit to the evaluation task rather than an independent prediction.
fitted input called prediction [Section 4.4 (multitask setting); Appendix A.3.3]
"Specifically, for CTExp we employ specific task corpus, including XSum corpus (Abacha and Demner-Fushman, 2019) for summarization task ... medical summarization task (MedSum (Abacha and Demner-Fushman, 2019), 3-shots, abbreviated as MSum)."

The same reference (Abacha and Demner-Fushman, 2019) is used for the summarization-task corpus CTExp and for the MedSum (MSum) evaluation in Table 1. As written, the multitask pruning mask is fitted on the evaluation corpus itself, so the reported MSum score is an in-sample fit rather than a prediction on a held-out task. If the 'XSum' name is a typo for Narayan et al., then the paper contains a citation error that must be corrected; under the literal text, the step is circular.

full rationale

The method's core operation—retaining neurons whose removal least perturbs outputs on a corpus—is internally sensible and not circular when the corpus is disjoint from the test data. The non-circular parts are the general-capability evaluations (ARC-c, GSM8K, MMLU), the one- and two-dimensional settings with held-out corpora, and the cross-lingual residual gap in the multilingual experiments. However, the paper's own appendix shows that the domain-task corpora used to select the masks are the English versions of the exact multilingual evaluation benchmarks, and the multitask section's summarization corpus is the same dataset as MedSum. Those expert scores are therefore fitted to the evaluation task content, matching pattern 2 (fitted input called prediction). Self-citations to Zhao et al. (2024b) are not load-bearing: the existence of language-specific neurons is also supported by independent citations and the present empirical results. The pruning-ratio mismatch flagged in Eq. 2/Algorithm 1 (intersection of per-document bottom-sigma sets may not equal sigma) is a correctness/reproducibility concern rather than a circularity: it undermines equal-ratio comparisons but does not make a derivation self-referential. Weighing the in-sample expert evaluation against the independent general evaluations, the circularity is partial, so score 6.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. It relies on a set of assumptions about neuron-level interpretability, the validity of output-perturbation importance, and corpus representativeness. The main free parameters are the pruning ratio and corpus sizes, neither of which is grounded in a principled derivation.

free parameters (3)

Pruning ratio sigma = 25% (main experiments), up to 45% in aggressive experiments
The pruning ratio is a threshold for determining which neurons are 'irrelevant'. The paper uses 25% as the default, following prior work, and also explores 35% and 45%. The choice is not derived from first principles; it is a user-specified hyperparameter.
Corpus sizes for each dimension = Not explicitly stated for main experiments; preliminary study uses 50 documents per dimension
The number of documents used to identify irrelevant neurons is a free parameter. The paper states hyperparameters including corpus sizes are tuned on validation sets of some target datasets, but it does not report the actual sizes used in the main experiments. This is a significant unstated free parameter.
Impact threshold for neuron irrelevance = Lowest sigma% of impact scores
The criterion for considering a neuron irrelevant is its impact being in the lowest sigma% of all neurons. This is a percentile threshold that effectively depends on the pruning ratio, so it is not a separate fitted parameter, but the choice of relative vs. absolute threshold is a modeling decision.

assumptions (4)

domain assumption Neurons are the appropriate granularity for pruning, and individual neurons can be identified as language/domain/task specific.
The whole method rests on the assumption that neuron-level pruning can selectively remove capabilities without catastrophic damage. This is supported by prior interpretability work (e.g., Tang et al., 2024) but is not established for all dimensions and models in this paper.
domain assumption The impact of removing a neuron on the layer output, as measured by |h_N_i(c) - h_i(c)|, is a valid proxy for the neuron's importance to the task.
This is the core of the method (Section 2.1). The paper uses this as the definition of relevance, but does not compare against alternative importance measures (e.g., gradient-based, weight magnitude) or validate that this proxy is well-calibrated.
ad hoc to paper Corpora generated by GPT-4o (in the preliminary study) or drawn from public datasets (in later sections) are representative of the target dimension distributions.
In Section 3, the corpora are generated by a proprietary model. In Section 4, some corpora are the English equivalents of the target datasets. There is no analysis of corpus diversity or representativeness.
ad hoc to paper The intersection of per-dimension irrelevant neuron sets is the correct way to construct a multi-dimensional expert model.
Equation 5 defines the expert model by intersecting the irrelevant sets. The paper does not compare against alternative combination rules (e.g., union, or pruning only along the most important dimension). The intersection is plausible but not validated.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Pruning General Large Language Models into Customized Expert Models." pith.science (2026). https://pith.science/paper/6FER66X3

@misc{pith2026250602561,
  author       = {Pith},
  title        = {Pith review of: Pruning General Large Language Models into Customized Expert Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6FER66X3}},
  note         = {Machine review of arXiv:2506.02561}
}

abstract

Large language models (LLMs) have revolutionized natural language processing, yet their substantial model sizes often require substantial computational resources. To preserve computing resources and accelerate inference speed, it is crucial to prune redundant parameters, especially for experienced users who often need compact expert models tailored to specific downstream scenarios. However, most existing pruning methods focus on preserving the model's general capabilities, often requiring extensive post-training or suffering from degraded performance due to coarse-grained pruning. In this work, we design a $\underline{Cus}$tom $\underline{Prun}$ing method ($\texttt{Cus-Prun}$) to prune a large general model into a smaller lightweight expert model, which is positioned along the "language", "domain" and "task" dimensions. By identifying and pruning irrelevant neurons of each dimension, $\texttt{Cus-Prun}$ creates expert models without any post-training. Our experiments demonstrate that $\texttt{Cus-Prun}$ consistently outperforms other methods, achieving minimal loss in both expert and general capabilities across various models from different model families and sizes.

Figures

Figures reproduced from arXiv: 2506.02561 by the authors.

**Figure 1.** Given a requirement for an expert model across three dimensions (language, domain, and task), Cus-Prun (i) identifies irrelevant neurons for each dimension using corresponding corpora (left), and (ii) prunes overlapping irrelevant neurons across dimensions to obtain the expert model (right). capabilities may not be useful or relevant to our desired application scenario. This misalignment with practical requirement… view at source ↗

**Figure 2.** Concrete examples of applying Cus-Prun to prune 25% of Llama3-8B-Base’s parameters into threedimensional expert models. Numbers above each box indicate performance on the whole test set, with the first evaluated by Rouge-L, and the other two by accuracy. we adopt SliceGPT (Ashkboos et al., 2024) as a baseline, which replaces each weight matrix with a smaller proxy matrix without considering the specific use-case r… view at source ↗

**Figure 3.** Chinese Medical LLM performance. Numbers are quality [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Performance of Cus-Prun in obtaining language-specific models. linguistically diverse languages: German, Chinese, and Thai. We conduct experiments based on the Llama3-8b model. To identify languagespecific (while domain- and task-agnostic) neurons, we employ a divers…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Visualising Information Flow in Word Embeddings with Diffusion Tensor Imaging
cs.CL 2026-01 conditional novelty 6.0 of 10

Applying diffusion tensor imaging to LLM hidden states produces a new visualisation of token-to-token and layer-to-layer 'information flow'.

Reference graph

Works this paper leans on

68 extracted references · 25 canonical work pages · cited by 1 Pith paper

[1]

Asma Ben Abacha and Dina Demner-Fushman. 2019. On the summarization of consumer health questions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2228--2234

work page 2019
[2]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023
[3]

Duarte M Alves, Jos \'e Pombal, Nuno M Guerreiro, Pedro H Martins, Jo \ a o Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, et al. 2024. Tower: An open multilingual large language model for translation-related tasks. arXiv preprint arXiv:2402.17733

arXiv 2024
[4]

Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623--4637

work page 2020
[5]

Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. 2024. Slicegpt: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations

work page 2024
[6]

Mohammed Attia, Younes Samih, Ali Elkahky, and Laura Kallmeyer. 2018. Multilingual multi-class sentiment classification using convolutional neural networks. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)

work page 2018
[7]

Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. 2024. Llemma: An open language model for mathematics. In The Twelfth International Conference on Learning Representations

work page 2024
[8]

Wenhu Chen, Ming-Wei Chang, Eva Schlinger, William Yang Wang, and William W Cohen. 2020. Open question answering over tables and text. In International Conference on Learning Representations

work page 2020

Show all 68 references

[9]

Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R Routledge, et al. 2021. Finqa: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods i...

2021
[10]

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457

2018 arXiv
[11]

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

2021 arXiv
[12]

Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2...

2018
[13]

Yiming Cui, Ziqing Yang, and Xin Yao. 2023. Efficient and effective text encoding for chinese llama and alpaca. arXiv preprint arXiv:2304.08177

2023 arXiv
[14]

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36

2024
[15]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

2024 arXiv
[16]

Iker Garc \' a-Ferrero, Rodrigo Agerri, Aitziber Atutxa, Elena Cabrio, Iker de la Iglesia, Alberto Lavelli, Bernardo Magnini, Benjamin Molinet, Johana Ramirez-Romero, German Rigau, et al. 2024. Medical mt5: An open-source multilingual text-to-text llm for the medical domain. I...

2024
[17]

Josh A Goldstein, Girish Sastry, Micah Musser, Renee DiResta, Matthew Gentzel, and Katerina Sedova. 2023. Generative language models and automated influence operations: Emerging threats and potential mitigations. arXiv preprint arXiv:2301.04246

2023 arXiv
[18]

Tahmid Hasan, Abhik Bhattacharjee, Md Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M Sohel Rahman, and Rifat Shahriyar. 2021. Xl-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics...

2021
[19]

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations

2021
[20]

Weiyu Huang, Guohao Jian, Yuezhou Hu, Jun Zhu, and Jianfei Chen. 2024. Pruning large language models with semi-structural adaptive sparse training. arXiv preprint arXiv:2407.20584

2024 arXiv
[21]

Wonseok Hwang, Dongjun Lee, Kyoungyeon Cho, Hanuhl Lee, and Minjoon Seo. 2022. A multi-task benchmark for korean legal language understanding and judgement prediction. Advances in Neural Information Processing Systems, 35:32537--32551

2022
[22]

Phillip Keung, Yichao Lu, György Szarvas, and Noah A. Smith. 2020. The multilingual amazon reviews corpus. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing

2020
[23]

Vishal A Kharde and SS Sonawane. 2016. Sentiment analysis of twitter data: A survey of techniques. International Journal of Computer Applications, 975:8887

2016
[24]

Simon Kurz, Zhixue Zhao, Jian-Jia Chen, and Lucie Flek. 2024. Language-specific calibration for pruning multilingual language models. arXiv preprint arXiv:2408.14398

2024
[25]

Jianquan Li, Xidong Wang, Xiangbo Wu, Zhiyi Zhang, Xiaolong Xu, Jie Fu, Prayag Tiwari, Xiang Wan, and Benyou Wang. 2023 a . https://arxiv.org/abs/2305.01526 Huatuo-26m, a large-scale chinese medical qa dataset . Preprint, arXiv:2305.01526

2023 arXiv
[26]

Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. 2024. Evaluating quantized large language models. arXiv preprint arXiv:2402.18158

2024 arXiv
[27]

Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. 2023 b . Large language models in finance: A survey. In Proceedings of the fourth ACM international conference on AI in finance, pages 374--382

2023
[28]

Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao. 2023 c . Losparse: Structured compression of large language models based on low-rank and sparse approximation. In International Conference on Machine Learning, pages 20336--20350. PMLR

2023
[29]

Yuchao Li, Fuli Luo, Chuanqi Tan, Mengdi Wang, Songfang Huang, Shen Li, and Junjie Bai. 2022. Parameter-efficient sparsity for large language models fine-tuning. arXiv preprint arXiv:2205.11005

2022 arXiv
[30]

Yunlong Liang, Fandong Meng, Songming Zhang, Yufeng Chen, Jinan Xu, Jie Zhou, et al. 2024. Multilingual knowledge editing with language-agnostic factual neurons. arXiv preprint arXiv:2406.16416

2024 arXiv
[31]

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81

2004
[32]

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. 2024. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87--100

2024
[33]

Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, Lei Zhu, et al. 2023. Benchmarking large language models on cmexam--a comprehensive chinese medical exam dataset. arXiv preprint arXiv:2306.03030

2023 arXiv
[34]

Xinyin Ma, Gongfan Fang, and Xinchao Wang. 2023. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36:21702--21720

2023
[35]

Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. 2024. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853

2024 arXiv
[36]

Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. 2024. Compact language models via pruning and knowledge distillation. arXiv preprint arXiv:2407.14679

2024 arXiv
[37]

Micah Musser. 2023. A cost analysis of generative language models and influence operations. arXiv preprint arXiv:2308.03740

2023 arXiv
[38]

Shashi Narayan, Shay Cohen, and Maria Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In 2018 Conference on Empirical Methods in Natural Language Processing

2018
[39]

James O’Neill, Polina Rozenshtein, Ryuichi Kiryo, Motoko Kubota, and Danushka Bollegala. 2021. I wish i would have loved this one, but i didn’t--a multilingual dataset for counterfactual detection in product review. In Proceedings of the 2021 Conference on Empirical Methods in...

2021
[40]

Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Lear...

2022
[41]

P Rajpurkar. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250

2016 arXiv
[42]

Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arX...

2024 arXiv
[43]

Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950

2023 arXiv
[44]

Pratyusha Sharma, Jordan T Ash, and Dipendra Misra. 2024. The truth is in there: Improving reasoning in language models with layer-selective rank reduction. In The Twelfth International Conference on Learning Representations

2024
[45]

Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. 2023. Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference on Learning Representations

2023
[46]

Jiwon Song, Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, et al. 2024. Sleb: Streamlining llms through redundancy verification and elimination of transformer blocks. In Forty-first International Conference on Machine Learning

2024
[47]

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human ...

2019
[48]

Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. 2024. Language-specific neurons: The key to multilingual capabilities in large language models. arXiv preprint arXiv:2402.16438

2024 arXiv
[49]

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118

2024 arXiv
[50]

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

2023 arXiv
[51]

Pingjie Wang, Ziqing Fan, Shengchao Hu, Zhe Chen, Yanfeng Wang, and Yu Wang. 2024. Reconstruct the pruned model without any retraining. arXiv preprint arXiv:2407.13331

2024 arXiv
[52]

Smith, Hannaneh Hajishirzi, and Daniel Khashabi

Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby ...

2022 arXiv
[53]

Ziheng Wang, Jeremy Wohlwend, and Tao Lei. 2019. Structured pruning of large language models. arXiv preprint arXiv:1910.04732

2019 arXiv
[54]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

2022
[55]

Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. 2024. Sheared llama: Accelerating language model pre-training via structured pruning. In The Twelfth International Conference on Learning Representations

2024
[56]

Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xiaopeng Zhang, and Qi Tian. 2023. Qa-lora: Quantization-aware low-rank adaptation of large language models. arXiv preprint arXiv:2309.14717

2023 arXiv
[57]

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024 a . Qwen2 technical report. arXiv preprint arXiv:2407.10671

2024 arXiv
[58]

Yifei Yang, Zouying Cao, and Hai Zhao. 2024 b . Laco: Large language model pruning via layer collapse. arXiv preprint arXiv:2402.11187

2024 arXiv
[59]

Wenxuan Zhang, Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2023. M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models. Advances in Neural Information Processing Systems, 36:5484--5505

2023
[60]

Wenxuan Zhang, Hou Pong Chan, Yiran Zhao, Mahani Aljunied, Jianyu Wang, Chaoqun Liu, Yue Deng, Zhiqiang Hu, Weiwen Xu, Yew Ken Chia, Xin Li, and Lidong Bing. 2024 a . Seallms 3: Open foundation and chat multilingual large language models for southeast asian languages. CoRR, ab...

2024 arXiv
[61]

Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing. 2024 b . Sentiment analysis in the era of large language models: A reality check. In Findings of the Association for Computational Linguistics: NAACL 2024 , pages 3881--3906

2024
[62]

Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28

2015
[63]

Yang Zhang, Yawei Li, Xinpeng Wang, Qianli Shen, Barbara Plank, Bernd Bischl, Mina Rezaei, and Kenji Kawaguchi. 2024 c . Finercut: Finer-grained interpretable layer pruning for large language models. arXiv preprint arXiv:2405.18218

2024 arXiv
[64]

Pengxiang Zhao, Hanyu Hu, Ping Li, Yi Zheng, Zhefeng Wang, and Xiaoming Yuan. 2024 a . A convex-optimization-based layer-wise post-training pruner for large language models. arXiv preprint arXiv:2408.03728

2024 arXiv
[65]

Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. 2024 b . How do large language models handle multilingualism? arXiv preprint arXiv:2402.18815

2024 arXiv
[66]

Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. 2023. A survey on model compression for large language models. arXiv preprint arXiv:2308.07633

2023 arXiv
[67]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[68]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Asma Ben Abacha and Dina Demner-Fushman. 2019. On the summarization of consumer health questions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2228--2234

work page 2019

[2] [2]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023

[3] [3]

Duarte M Alves, Jos \'e Pombal, Nuno M Guerreiro, Pedro H Martins, Jo \ a o Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, et al. 2024. Tower: An open multilingual large language model for translation-related tasks. arXiv preprint arXiv:2402.17733

arXiv 2024

[4] [4]

Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623--4637

work page 2020

[5] [5]

Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. 2024. Slicegpt: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations

work page 2024

[6] [6]

Mohammed Attia, Younes Samih, Ali Elkahky, and Laura Kallmeyer. 2018. Multilingual multi-class sentiment classification using convolutional neural networks. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)

work page 2018

[7] [7]

Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Q Jiang, Jia Deng, Stella Biderman, and Sean Welleck. 2024. Llemma: An open language model for mathematics. In The Twelfth International Conference on Learning Representations

work page 2024

[8] [8]

Wenhu Chen, Ming-Wei Chang, Eva Schlinger, William Yang Wang, and William W Cohen. 2020. Open question answering over tables and text. In International Conference on Learning Representations

work page 2020