REVIEW 7 cited by
Teaching Small Language Models to Reason
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Chain of thought prompting successfully improves the reasoning capabilities of large language models, achieving state of the art results on a range of datasets. However, these reasoning capabilities only appear to emerge in models with a size of over 100 billion parameters. In this paper, we explore the transfer of such reasoning capabilities to models with less than 100 billion parameters via knowledge distillation. Specifically, we finetune a student model on the chain of thought outputs generated by a larger teacher model. Our experiments show that the proposed method improves task performance across arithmetic, commonsense and symbolic reasoning datasets. For example, the accuracy of T5 XXL on GSM8K improves from 8.11% to 21.99% when finetuned on PaLM-540B generated chains of thought.
Forward citations
Cited by 7 Pith papers
-
Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization
Byte-Prefix Marginalization maps a teacher's next-token distribution onto the student's vocabulary through shared byte prefixes plus an explicit residual, giving a mass-preserving target for on-policy distillation acr...
-
Student-Centered Distillation Narrows the Agentic Gap Between Small and Large LLMs
A 7B LLM agent trained with student-led distillation and one-step teacher corrections nearly matches a 72B teacher on reasoning and tool-use benchmarks.
-
Do Cognitively Interpretable Reasoning Traces Improve LLM Performance?
On CoTemp QA, supervised fine-tuning with raw R1 traces gave the best model accuracy while human raters found those traces least interpretable, showing model-useful traces and human-readable traces can diverge.
-
Security Tensors as a Cross-Modal Bridge: Extending Text-Aligned Safety to Vision in LVLM
Security tensors, learned input perturbations, transfer a language model's text-based safety behavior to visual inputs, improving harmful-image rejection in LVLMs while largely preserving benign performance.
-
SynthEHR-Eviction: Enhancing Eviction SDoH Detection with LLM-Augmented Synthetic EHR Data
An LLM-augmented synthetic data pipeline produces the largest public eviction-focused SDoH dataset (14 categories) and fine-tuned open LLMs that outperform prompt-optimized GPT-4o on the authors' test sets.
-
Enhancing Reasoning Capabilities of Small Language Models with Blueprints and Prompt Template Search
Small language models achieve higher accuracy on math, coding, and logic benchmarks when their prompts contain LLM-generated reasoning blueprints and a per-model, per-task searched template.
-
Towards Concise and Adaptive Thinking in Large Reasoning Models: A Survey
A comprehensive review that categorizes methods for shortening and adaptively triggering chain-of-thought reasoning in large language models.
Discussion (0). Sign in to comment.