Pith. sign in

REVIEW 4 cited by

Mixed Distillation Helps Smaller Language Model Better Reasoning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2312.10730 v2 pith:SRTGKVAZ submitted 2023-12-17 cs.CL cs.AI

Chenglin Li , Qianglong Chen , Liangyue Li , Caiyu Wang , Yicheng Li , Zulong Chen , Yin Zhang This is my paper · ORCID
classification cs.CLcs.AI
keywords modelsllmsreasoningsmallertasksdistillationlanguageperformance
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

While large language models (LLMs) have demonstrated exceptional performance in recent natural language processing (NLP) tasks, their deployment poses substantial challenges due to high computational and memory demands in real-world applications. Recent studies have focused on enhancing smaller models through knowledge distillation from LLMs, yielding promising results. However, these models often struggle to match the performance of LLMs, especially in tasks that require reasoning. In this work, we introduce Mixed Distillation (MD) framework, which capitalizes on the strengths of Program of Thought (PoT) and Chain of Thought (CoT) capabilities within LLMs, combining multiple prompting techniques and distilling these capabilities into smaller models. Our experimental results show that MD significantly enhances the single-path and multi-path reasoning ability of smaller models in various tasks. In terms of accuracy and generality of reasoning tasks, the model generated by it exceeds the comprehensive performance of two individually distilled models. Notably, LLaMA2-7B and CodeLlama-7B using MD achieved remarkable improvements of (84.5%) and (85.5%), respectively, outperforming GPT-3.5-Turbo by (2.5%) and (3.5%), on the SVAMP benchmark.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Thinkless: LLM Learns When to Think

    cs.CL 2025-05 conditional novelty 7.0 of 10

    A 1.5B model learns to decide when to use short versus long reasoning via control tokens and a decoupled GRPO objective, reducing thinking-mode usage by 50-90% on math benchmarks with minor accuracy loss.

  2. Don't Wait to Reply: Towards Responsive yet Thoughtful Dialogue through Proactive Thinking

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Proactive Thinking precomputes reasoning for anticipated user replies during dialogue idle time via anticipated rollouts and speculative continual thinking, reducing latency without accuracy loss on three time-aware b...

  3. Curriculum Learning for Efficient Chain-of-Thought Distillation via Structure-Aware Masking and GRPO

    cs.LG 2026-02 conditional novelty 6.0 of 10

    A three-stage curriculum (masked reconstruction, GRPO compression, teacher-guided rewriting) distills long chain-of-thought into a concise 3B student, reporting 76.19% on GSM8K (up from 64.90%) with 167 output tokens ...

  4. VeriThinker: Learning to Verify Makes Reasoning Model Efficient

    cs.LG 2025-05 conditional novelty 6.0 of 10

    VeriThinker shows that fine-tuning a reasoning model only on a solution-verification task reduces chain-of-thought length on MATH500 and AIME by 20-45% while preserving or slightly improving accuracy.

Pith tools