REVIEW 19 cited by
A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLMs) have demonstrated emergent abilities in text generation, question answering, and reasoning, facilitating various tasks and domains. Despite their proficiency in various tasks, LLMs like PaLM 540B and Llama-3.1 405B face limitations due to large parameter sizes and computational demands, often requiring cloud API use which raises privacy concerns, limits real-time applications on edge devices, and increases fine-tuning costs. Additionally, LLMs often underperform in specialized domains such as healthcare and law due to insufficient domain-specific knowledge, necessitating specialized models. Therefore, Small Language Models (SLMs) are increasingly favored for their low inference latency, cost-effectiveness, efficient development, and easy customization and adaptability. These models are particularly well-suited for resource-limited environments and domain knowledge acquisition, addressing LLMs' challenges and proving ideal for applications that require localized data handling for privacy, minimal inference latency for efficiency, and domain knowledge acquisition through lightweight fine-tuning. The rising demand for SLMs has spurred extensive research and development. However, a comprehensive survey investigating issues related to the definition, acquisition, application, enhancement, and reliability of SLM remains lacking, prompting us to conduct a detailed survey on these topics. The definition of SLMs varies widely, thus to standardize, we propose defining SLMs by their capability to perform specialized tasks and suitability for resource-constrained settings, setting boundaries based on the minimal size for emergent abilities and the maximum size sustainable under resource constraints. For other aspects, we provide a taxonomy of relevant models/methods and develop general frameworks for each category to enhance and utilize SLMs effectively.
Forward citations
Cited by 19 Pith papers
-
Does Mixture-of-Experts Actually Help Inference on Consumer and Edge Hardware? An Empirical Study
On bandwidth-bound edge hardware, MoE inference cost tracks total parameters rather than active ones, so sparse activation fails to deliver the expected throughput or energy gains.
-
Punching Above Their Weight: Classification-Head Fine-Tuning of Tiny Language Models (TLMs) for Verifiable Multiple-Choice Tasks
Classification-head LoRA fine-tuning of sub-3B Qwen3 models outperforms label-generation SFT by 2–3% on HellaSwag, WinoGrande and PIQA and yields SOTA numbers competitive with GPT-3/PaLM/GPT-4.
-
PPE-Bench: A Benchmark for Evaluating MLLM Unlearning under Private-Public Entanglement
Existing MLLM unlearning methods reduce private-attribute leakage on entangled images but substantially harm co-occurring public figures and landmarks, with private knowledge often re-emerging after public finetuning.
-
Mitigating Attention Localization in Small Scale: Self-Attention Refinement via One-step Belief Propagation
A one-step belief propagation refinement with a repulsive Potts prior increases attention entropy and improves downstream accuracy of small Transformers, with GTD as a diagnostic for multi-hop attention.
-
TweakLLM: A Routing Architecture for Dynamic Tailoring of Cached Responses
A small language model can rewrite cached large-model responses to fit similar new queries, preserving quality while cutting inference cost.
-
Bradley-Terry and Multi-Objective Reward Modeling Are Complementary
Jointly training a Bradley-Terry preference head and a multi-attribute regression head on a shared embedding improves reward-model robustness to reward hacking and boosts multi-objective scoring performance.
-
MedOrchestra: A Hybrid Cloud-Local LLM Approach for Clinical Data Interpretation
A cloud-local hybrid, where the cloud writes subtask prompts offline and a local model executes them on patient data, reached 70-85% staging accuracy, above local baselines and clinicians.
-
Energy- and Memory-Efficient PEFT Methods for Personalized On-Device SLMs on Consumer GPUs
On consumer GPUs, LoRA+ gives the best energy-focused fine-tuning score in 19 of 24 small-model task configurations, while QLoRA wins the memory-focused score when peak VRAM is the binding constraint.
-
Automatic Failure Attribution and Critical Step Prediction Method for Multi-Agent Systems Based on Causal Inference
A hierarchical causal attribution framework (performance causal inversion, Shapley values, CDC-MAS step discovery) localizes failure agents and steps in LLM multi-agent systems, reporting up to 36.2% step accuracy.
-
AI Propaganda factories with language models
Small language models sustain political personas and become more ideologically extreme when replying to counter-arguments, according to a language-model judge.
-
Detecting Voice Phishing with Precision: Fine-Tuning Small Language Models
Fine-tuning Llama-3-8B with human-authored voice phishing criteria outperforms chain-of-thought prompting and approaches GPT-4-level accuracy on a new adversarial Korean voice phishing benchmark.
-
Large Language Models for Next-Generation Wireless Network Management: A Survey and Tutorial
A survey and tutorial that organizes LLM-enabled wireless network optimization into formulation, solution, and verification stages, with case studies drawn from the authors' own prior papers.
-
Advancing SLM Tool-Use Capability using Reinforcement Learning
GRPO with a strict reward for structured JSON output improves tool-call accuracy on small models, from 0.98%-6.1% to 22%-71% on the xLAM benchmark, though without baselines or variance.
-
Unifying Mixture of Experts and Multi-Head Latent Attention for Efficient Language Models
Combining MoE, MLA, and RoPE in small transformers improves perplexity on TinyStories while cutting KV cache size, but several headline numbers conflict internally.
-
Enhancing Reasoning Capabilities in SLMs with Reward Guided Dataset Distillation
AdvDistill uses group relative advantages computed from rule-based rewards to weight teacher responses during distillation, reportedly improving a 1.5B student on math tasks beyond its 7B teacher.
-
Latent Multi-Head Attention for Small Language Models
MLA with rotary embeddings at half latent rank keeps validation loss nearly unchanged while cutting KV-cache memory by roughly half on small language models.
-
Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques
A survey of LLM routing and hierarchical inference techniques that proposes an unvalidated unified evaluation metric called the Inference Efficiency Score.
-
The State of Large Language Models for African Languages: Progress and Challenges
Across 20 reviewed language models, only about 42 of Africa's 2,000+ languages receive any support, and just 3 of 23 active scripts are commonly used.
-
Causal Distillation: Transferring Structured Explanations from Large to Compact Language Models
Small language models fine-tuned on GPT-4 causal explanations score high on a new teacher-similarity metric, but the paper provides no independent evidence that causal reasoning was transferred.
Discussion (0). Sign in to comment.