Pith. sign in

REVIEW 4 major objections 7 minor 1 cited by

Evaluating Small Language Models for News Summarization: Implications and Factors Influencing Performance

T0 review · 4 major / 7 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Small language models under 4B parameters can match 70B-parameter LLMs on news summarization quality while writing shorter summaries.

desk verdict A useful SLM summarization benchmark with a new dataset, but the headline claim that sub-4B models match 70B LLMs rests on sub-point BertScore gaps with no error bars. read the letter →

arxiv 2502.00641 v2 pith:YDZDX7AS submitted 2025-02-02 cs.CL cs.AI

Borui Xu , Yao Chen , Zeyi Wen , Weiguo Liu , Bingsheng He This is my paper · ORCID
classification cs.CLcs.AI
keywords smalllanguagemodelsnewssummarizationBertScorereference-basedevaluationLLM-generatedreferencesinstructiontuningpromptdesignfactualconsistency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the best small language models, those under 4 billion parameters, can produce news summaries as good as 70B-parameter LLMs while writing shorter ones. It tests 19 SLMs on 2,000 news articles drawn from four datasets, scoring relevance and coherence with BertScore against LLM-written reference summaries, factual consistency with a hallucination detector, and length. The top performers are Phi3-Mini and Llama3.2-3B-Ins. The paper also reports that simple prompts work better than elaborate ones for these models, and that instruction tuning helps some model families but not others. If the claim holds, resource-constrained applications can get near-LLM summarization quality on edge devices.

What carries the argument

The load-bearing mechanism is the LLM-augmented reference-based evaluation setup. Reference summaries are produced by two instruction-tuned 70B chat models, Qwen1.5-72B-Chat and Llama2-70B-Chat, using a fixed two-sentence prompt; each SLM summary is scored by BertScore F1 against both references and the scores are averaged. BertScore, a semantic similarity metric computed through BERT contextual embeddings, is the proxy for relevance and coherence, HHEM-2.1-Open supplies the factual-consistency rate, and average summary length captures compression. The paper validates the pipeline by showing that BertScore with LLM references correlates with human judgments better than BertScore with original dataset references on three existing human-evaluation datasets.

What would settle it

Run the benchmark against references written by professional news editors rather than by the two 70B chat models, then have human annotators score every model's output, not just five models on twenty articles, on relevance and coherence. If Phi3-Mini and Llama3.2-3B-Ins no longer rank with or above the 70B models, the paper's central claim fails.

Watch

Extended reading notes

Core claim

The central claim is that among models under 4B parameters, summarization quality varies widely, and the best ones close the gap to 70B LLMs. On the four-dataset, 2,000-sample benchmark, Phi3-Mini and Llama3.2-3B-Ins average BertScore above 74, placing them in the same range as Llama3-70B-Ins and Qwen2-72B-Ins on the BBC2024 comparison, with comparable factual consistency and shorter outputs. The paper argues this parity is real because its metric, BertScore against LLM-written references, was validated against human ratings on existing benchmarks and in a small human check. It also reports that simple prompts outperform complex ones for these models and that instruction tuning helps the Llama3.2 family but not Qwen2 or InternLM2, which runs against the assumption that instruction tuning is an automatic summarization upgrade.

Load-bearing premise

The load-bearing premise, flagged in the paper's own Limitations section, is that BertScore scored against LLM-written reference summaries is an accurate proxy for human-judged relevance and coherence for every model, dataset, and prompt in the benchmark; if that correlation is an artifact of reference style, the parity claim collapses.

Editorial extensions

If this is right

  • Applications needing on-device news summarization can deploy Phi3-Mini or Llama3.2-3B-Ins and expect 70B-class relevance and coherence in roughly half the summary length.
  • Simple two-sentence prompts are the recommended operating point; adding instructions can reduce factual consistency and lengthen outputs for small models.
  • Instruction tuning is not a reliable summarization upgrade: only the Llama3.2 family improved substantially, so model-family choice matters more than tuning status.
  • The benchmark's size-based guidance, Brio or Qwen2-0.5B under 1B, Llama3.2-1B-Ins at 1-2B, and Llama3.2-3B-Ins or Phi3-Mini above 2B, gives deployers a concrete starting point.
  • Using LLM-generated references instead of original dataset summaries raises the correlation of reference-based scoring with human judgment, so future summarization benchmarks can be cheaper and more reproducible.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the benchmark's zero-shot, greedy-decoding protocol probably measures a lower bound on what these SLMs can do; sampling or a few in-context examples could shift the ranking, and the paper presents no evidence either way.
  • Editorial inference: the inconsistent instruction-tuning results suggest that pretraining data and within-family architecture, rather than post-training per se, may drive summarization gains; that is a testable hypothesis the paper leaves open.
  • Editorial inference: the Limitations section's own caveat, that LLM references may reward summaries that imitate the reference generators, could be probed by scoring with a third reference LLM or human-written references; if rankings stay stable, the headline is on firmer ground.
  • Editorial inference: because only five models and twenty articles were human-scored, the strongest parity claim rests on BertScore; a full human read of all 19 models would tell whether the metric is the message.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. This paper presents a benchmark of 19 small language models (SLMs) under 4B parameters for news summarization, evaluated on 2,000 news samples from CNN/DM, XSum, Newsroom, and a new BBC2024 dataset. Summaries are scored with BertScore against LLM-generated references from Qwen1.5-72B-Chat and Llama2-70B-Chat, with factual consistency measured by HHEM-2.1-Open and length as a compression proxy. The authors report that Phi3-Mini and Llama3.2-3B-Ins match the quality of 70B LLMs while producing shorter summaries, that simple prompts are preferable for SLMs, and that instruction tuning has inconsistent effects. A small human evaluation and a correlation analysis on existing benchmarks are used to validate the reference-based approach.

Significance. If the reported rankings are statistically robust, the paper provides a useful, reproducible benchmark and practical model-selection guidance for edge-device summarization. The release of code and results, the construction of a fresh 2024 news test set, and the use of multiple LLM-generated references to mitigate reference-quality issues are commendable and strengthen the paper's value. However, the central 'parity with 70B' claim currently rests on BertScore gaps smaller than the observed across-dataset variation, with no uncertainty quantification; the human validation is too small to support the fine-grained resolution required. These issues are addressable with the released data.

major comments (4)
  1. [§5.4, Table 5] The central claim that SLMs are comparable to 70B LLMs is based on BertScore differences of 0.06–1.11 points (e.g., Phi3-Mini 73.72 vs. Qwen2-72B-Ins 73.78; Llama3.2-3B-Ins 74.95 vs. Llama3-70B-Ins 76.06). No confidence intervals, bootstrap estimates, or significance tests are reported anywhere, and within-model variation across datasets in Table 3 is larger than these gaps (e.g., Llama3.2-3B-Ins ranges from 72.81 to 75.33). The Limitations section acknowledges reference bias but not sampling uncertainty, so the parity claim is unsupported at the reported precision. I recommend paired bootstrap or Wilcoxon signed-rank tests on the per-sample scores; the released code and results should make this straightforward.
  2. [§5.4 vs. §4.4] Section 5.4 compares SLMs with LLMs using only Llama2-70B-Chat as the reference and only the BBC2024 dataset, whereas Section 4.4 explicitly motivates averaging over two reference LLMs (Qwen1.5-72B-Chat and Llama2-70B-Chat) to mitigate same-series bias. Because Llama3.2-3B-Ins, the top SLM, belongs to the same model family as the reference generator, the parity result in Table 5 may be inflated by reference-model similarity. The paper should present the LLM comparison with both references and across all four datasets, or clearly justify the single-reference, single-dataset design.
  3. [§5.5, Table 7] The human evaluation uses 20 news articles and 5 SLMs, and the reported Kendall's tau of 1.0 is computed on only 5 system-level points. This cannot validate the metric at the 1-point BertScore resolution needed to distinguish models in Table 5; it only supports a coarse ordering. The paper should either expand the human evaluation substantially or explicitly state that fine-grained differences (sub-1-point BertScore gaps) are not human-validated.
  4. [§4.3, §5.1, Limitations] BertScore is described as measuring both relevance and coherence (e.g., 'maintains logical flow' in §4.3, and the 'Coherence evaluation' column in Table 1), but BertScore is a token-level semantic similarity metric and is not designed for coherence assessment. The Limitations section concedes this, yet the main text still interprets BertScore differences as coherence differences. This is load-bearing for the conclusion that SLMs generate coherent summaries; the authors should either use a coherence-specific metric or restrict the coherence claims to what BertScore can support.
minor comments (7)
  1. [§1] The sentence 'Section 5 presents presents the benchmarking results' contains a duplicated word; please remove the second 'presents'.
  2. [Table 7 caption] The caption reads 'Humance evaluation'; this should be 'Human evaluation'.
  3. [§5.2] The text refers to 'Figre 4'; this should be 'Figure 4'.
  4. [Table 4] The model name 'LiteLama' is misspelled; it should be 'LiteLlama' to match Table 2 and the rest of the text.
  5. [§5.4] The model name 'Llam3-70B-Ins' is missing an 'a'; it should be 'Llama3-70B-Ins'.
  6. [References] The references list 'Pu et al. 2023a' and 'Pu et al. 2023b' as two separate works with the same title and venue; this appears to be a duplicate entry and should be corrected.
  7. [§6.1, Table 6] The claim that complex prompts degrade SLM summarization performance is largely driven by a single model, Qwen2-0.5B-Ins, whose BertScore drops from 68.64 (Prompt 1) to 60.31 (Prompt 2); other models change by less than a point. Please temper the 'overly complex prompts may lead to a decline' conclusion or provide significance testing across models.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the SLM-vs-LLM comparison is a direct external measurement, with acknowledged metric limitations but no definitional or fitted-input circularity.

full rationale

This paper is an empirical benchmark, not a derivation; there is no load-bearing chain of equations or fitted parameters that reduces a claimed prediction to its inputs. The central claim that Phi3-Mini and Llama3.2-3B-Ins approach 70B LLM quality rests on Table 5, where both SLMs and LLMs are scored with the same BertScore metric against the same reference summaries generated by Qwen1.5-72B-Chat and Llama2-70B-Chat (Section 4.4). No parameter is fitted to the target comparison, and the compared LLM scores are independently obtained measurements, not constructed from SLM scores. The use of LLM-generated references is an explicit methodological choice, not a hidden identity: the reference generators are external models rather than the authors' models, and the approach is validated against external human-evaluation datasets (SummEval, Bench-CNN/DM, Bench-XSum) in Table 1. The small human validation (Section 5.5, 20 articles, 5 models) and the absence of significance testing are validity and statistical-inference concerns, not circularity, because they do not make the SLM and LLM scores equal by construction. The Limitations section explicitly acknowledges reference bias and coherence-metric limitations. There is no load-bearing self-citation: supporting works such as Zhang et al. 2024b, Fabbri et al. 2021, and Pu et al. 2023a have no author overlap with the present paper, and the paper's own findings are presented as measurements rather than as consequences of those citations. The 'comparable to 70B LLMs' statement is an empirical observation whose strength depends on metric validity and sampling uncertainty, but it is not circular.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claims are empirical measurements rather than derivations, so there are no free parameters fitted to data. The claims rest on a chain of domain assumptions about evaluation design: the two-sentence summary format, the use of BertScore for coherence, the superiority of LLM-generated references, and the transfer of metric validation to new datasets. The BBC2024 dataset is a new artifact but is a data resource rather than a postulated entity. The instruction-tuning and prompt findings depend mainly on assumptions 1 and 2, because the ranking that motivates the prompt and tuning analysis is produced by the same metric that those assumptions underpin.

assumptions (5)
  • domain assumption Two-sentence summaries are an appropriate and sufficient target for news summarization.
    All prompt templates in Figure 3 instruct models to summarize in two sentences, and LLM reference summaries are generated with the same constraint. The paper provides no justification that two sentences is the right level of compression for news.
  • domain assumption BertScore F1 captures both relevance and coherence.
    Section 4.3 states that BertScore can measure relevance and coherence, but it is a token-level semantic similarity metric and is not designed to evaluate coherence. The paper's Limitations section concedes this, yet the main ranking of models relies on it.
  • domain assumption LLM-generated summaries are higher quality than original dataset references.
    Section 3 cites Zhang et al. 2024b and Pu et al. 2023 to justify replacing original references with LLM summaries. This is load-bearing because the entire evaluation is scored against these references, and an error in this assumption propagates to all model rankings.
  • domain assumption The metric validation on Bench-CNN/DM, Bench-XSum, and SummEval generalizes to CNN/DM, XSum, Newsroom, and BBC2024.
    Table 1 validates LLM-referenced BertScore on three human-evaluation datasets, but the main benchmark uses different splits and a new BBC2024 dataset, so the correlation with human judgment is assumed to transfer.
  • domain assumption A subset of 500 samples per dataset is sufficient to differentiate models.
    Section 4.1 relies on Maynez et al. 2023 for the claim that 500 samples are sufficient. This is a reasonable borrowed heuristic, but it is not re-verified for these models or the HHEM hallucination metric.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Small Language Models for News Summarization: Implications and Factors Influencing Performance." pith.science (2026). https://pith.science/paper/YDZDX7AS

@misc{pith2026250200641,
  author       = {Pith},
  title        = {Pith review of: Evaluating Small Language Models for News Summarization: Implications and Factors Influencing Performance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YDZDX7AS}},
  note         = {Machine review of arXiv:2502.00641}
}
read the original abstract

The increasing demand for efficient summarization tools in resource-constrained environments highlights the need for effective solutions. While large language models (LLMs) deliver superior summarization quality, their high computational resource requirements limit practical use applications. In contrast, small language models (SLMs) present a more accessible alternative, capable of real-time summarization on edge devices. However, their summarization capabilities and comparative performance against LLMs remain underexplored. This paper addresses this gap by presenting a comprehensive evaluation of 19 SLMs for news summarization across 2,000 news samples, focusing on relevance, coherence, factual consistency, and summary length. Our findings reveal significant variations in SLM performance, with top-performing models such as Phi3-Mini and Llama3.2-3B-Ins achieving results comparable to those of 70B LLMs while generating more concise summaries. Notably, SLMs are better suited for simple prompts, as overly complex prompts may lead to a decline in summary quality. Additionally, our analysis indicates that instruction tuning does not consistently enhance the news summarization capabilities of SLMs. This research not only contributes to the understanding of SLMs but also provides practical insights for researchers seeking efficient summarization solutions that balance performance and resource use.

Figures

Figures reproduced from arXiv: 2502.00641 by the authors.

Figure 1
Figure 1. Variation of SLMs in size and BertScore for [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison of text summarization evaluation. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The prompt templates for the language model [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Example summaries from SLMs and LLMs. The bold part is the same as the reference, the under￾line indicates irrelevant content, and the red indicates incorrect content. Summaries with BertScore above 70, such as those from Llama3.2-3B-Ins, demonstrate simi￾lar quality t…
Figure 5
Figure 5. Figure 5: Average summary length comparison. SLMs with high BertScore generate 50-70 word summaries. we report the factual consistency rate in [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Examples summaries generated by Llama3.2- [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Pairwise system-level Kendall’s tau correla [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Large AI Models to Agentic AI: A Tutorial on Future Intelligent Communications

    cs.AI 2025-05 conditional novelty 2.0 of 10

    This paper is a broad tutorial on applying LAMs and agentic AI to 6G, largely restating existing research rather than introducing new results.

Reference graph

Works this paper leans on

62 extracted references · 20 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    arXiv 2024

  4. [4]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  5. [5]

    Stella Biderman, Kieran Bicheno, and Leo Gao. 2022. Datasheet for the pile. arXiv preprint arXiv:2201.07311

    arXiv 2022

  6. [6]

    Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. 2021. https://doi.org/10.5281/zenodo.5297715 GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow . If you use this software, please cite it using these metadata

    doi:10.5281/zenodo.5297715 2021

  7. [7]

    Microsoft Research Blog. 2023. https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/ Phi-2: The surprising power of small language models . Accessed: 2024-06-27

    work page 2023

  8. [8]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

Show all 62 references
  1. [9]

    Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang,...

    2024 arXiv

  2. [10]

    Roshni Chakraborty, Maitry Bhavsar, Sourav Kumar Dandapat, and Joydeep Chandra. 2019. https://doi.org/10.1109/TCSS.2019.2926144 Tweet summarization of news articles: An objective ordering-based perspective . IEEE Transactions on Computational Social Systems, 6(4):761--777

    2019

  3. [11]

    Cheng-Han Chiang and Hung-yi Lee. 2023. https://aclanthology.org/2023.acl-long.870 Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15607-...

    2023

  4. [12]

    Together Computer. 2023. https://github.com/togethercomputer/RedPajama-Data Redpajama: an open dataset for training large language models

    2023

  5. [13]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

    2019 doi

  6. [14]

    Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. 2018. https://doi.org/10.18653/v1/D18-1409 B andit S um: Extractive summarization as a contextual bandit . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing...

    2018 doi

  7. [15]

    Alexander R Fabbri, Wojciech Kry \'s ci \'n ski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391--409

    2021

  8. [16]

    Xue-Yong Fu, Md Tahmid Rahman Laskar, Elena Khasanova, Cheng Chen, and Shashi Tn. 2024. https://doi.org/10.18653/v1/2024.naacl-industry.33 Tiny titans: Can smaller large language models punch above their weight in the real world for meeting summarization? In Proceedings of the...

    2024 doi

  9. [17]

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The P ile: An 800 GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027

    2020 arXiv

  10. [18]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2023 doi

  11. [19]

    Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. 2023 b . Human-like summarization evaluation with chatgpt. arXiv preprint arXiv:2304.02554

    2023 arXiv

  12. [20]

    Pierre-Etienne Genest and Guy Lapalme. 2012. https://aclanthology.org/P12-2069 Fully abstractive approach to guided summarization . In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 354--358, Jeju Island,...

    2012

  13. [21]

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie H...

    2024 arXiv

  14. [22]

    Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022. News summarization and evaluation in the era of gpt-3. arXiv preprint arXiv:2209.12356

    2022 arXiv

  15. [23]

    Max Grusky, Mor Naaman, and Yoav Artzi. 2018. http://aclweb.org/anthology/N18-1065 Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies . In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguis...

    2018

  16. [24]

    Xiaotian Han. 2024. https://huggingface.co/ahxt/LiteLlama-460M-1T Litellama-460m-1t . Accessed: 2024-06-27

    2024

  17. [25]

    Karl Moritz Hermann, Tom\' a s Ko c isk\' y , Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volum...

    2015

  18. [26]

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. 2024. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395

    2024 arXiv

  19. [27]

    Dandan Huang, Leyang Cui, Sen Yang, Guangsheng Bao, Kun Wang, Jun Xie, and Yue Zhang. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.33 What have we achieved on text summarization? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EM...

    2020 doi

  20. [28]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  21. [29]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  22. [30]

    Teven Le Scao, Thomas Wang, Daniel Hesslow, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Elsahar, Niklas Muennighoff, Jason Phang, Ofir Press, Colin Raffel, Victor Sanh, Sheng Shen, Lintang Sutawika, Jaesung Tae, Zheng Xin Yong, Julien Launay, and Iz Beltagy. 2022. https:...

    2022 doi

  23. [31]

    Haozhou Li, Qinke Peng, Xu Mou, Ying Wang, Zeyuan Zeng, and Muhammad Fiaz Bashir. 2023. https://doi.org/10.1109/TASLP.2023.3304473 Abstractive financial news summarization via transformer-bilstm encoder and graph attention-based decoder . IEEE/ACM Transactions on Audio, Speech...

    2023

  24. [32]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  25. [33]

    Yixin Liu, Alexander Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, and Arman Cohan. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.280 Benchmarking generation and evaluation capabilities of large language models f...

    2024 doi

  26. [34]

    Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022. https://doi.org/10.18653/v1/2022.acl-long.207 BRIO : Bringing order to abstractive summarization . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers...

    2022 doi

  27. [35]

    Lane, and Mengwei Xu

    Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, Xiwen Zhang, Nicholas D. Lane, and Mengwei Xu. 2024. https://arxiv.org/abs/2409.15790 Small language models: Survey, measurements, and insights . Preprint, arXiv:2409.15790

    2024 arXiv

  28. [36]

    Hans Peter Luhn. 1958. The automatic creation of literature abstracts. IBM Journal of research and development, 2(2):159--165

    1958

  29. [37]

    Goran Mato s evi \' c . 2019. Text summarization techniques for meta description generation in process of search engine optimization. In Artificial Intelligence and Algorithms in Intelligent Systems, pages 165--173, Cham. Springer International Publishing

    2019

  30. [38]

    Joshua Maynez, Priyanka Agrawal, and Sebastian Gehrmann. 2023. https://doi.org/10.18653/v1/2023.acl-long.511 Benchmarking large language model capabilities for conditional generation . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (...

    2023 doi

  31. [39]

    Meta AI . 2024. https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/ Llama 3.2: Revolutionizing edge ai and vision with open, customizable models . Accessed: 2024-09-28

    2024

  32. [40]

    Microsoft Azure . 2024. https://azure.microsoft.com/en-us/blog/introducing-phi-3-redefining-whats-possible-with-slms/ Introducing phi-3: Redefining what's possible with slms

    2024

  33. [41]

    Cohen, and Mirella Lapata

    Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. https://doi.org/10.18653/v1/D18-1206 Don ' t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization . In Proceedings of the 2018 Conference on Empirical Methods in Natu...

    2018 doi

  34. [42]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 B leu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311--3...

    2002

  35. [44]

    Xiao Pu, Mingqi Gao, and Xiaojun Wan. 2023 b . Summarization is (almost) dead. arXiv preprint arXiv:2309.09558

    2023 arXiv

  36. [45]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1--67

    2020

  37. [46]

    Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. 2024. https://arxiv.org/abs/2402.07927 A systematic survey of prompt engineering in large language models: Techniques and applications . Preprint, arXiv:2402.07927

    2024 arXiv

  38. [47]

    Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. https://doi.org/10.18653/v1/2020.acl-main.704 BLEURT : Learning robust metrics for text generation . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881--7892, Online. Ass...

    2020 doi

  39. [48]

    Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You, and Lidong Bing. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.278 Large language models are not yet human-level evaluators for abstractive summarization . In Findings of the Association for Computational Linguisti...

    2023 doi

  40. [49]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295

    2024 arXiv

  41. [50]

    Priyam Tejaswin, Dhruv Naik, and Pengfei Liu. 2021. https://doi.org/10.18653/v1/2021.findings-acl.303 How well do you know your summarization datasets? In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3436--3449, Online. Association for Comp...

    2021 doi

  42. [51]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  43. [52]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Attention is all you need . In Advances in Ne...

    2017

  44. [53]

    Vectara. 2024. Hallucination evaluation model. https://huggingface.co/vectara/hallucination_evaluation_model. Accessed: 2024-10-02

    2024

  45. [54]

    Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. Is chatgpt a good nlg evaluator? a preliminary study. arXiv preprint arXiv:2303.04048

    2023 arXiv

  46. [55]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng...

    2024 arXiv

  47. [56]

    Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. https://proceedings.neurips.cc/paper/2021/file/e4d2b6e6fdeca3e60e0f1a62fee3d9dd-Paper.pdf Bartscore: Evaluating generated text as text generation . In Advances in Neural Information Processing Systems, volume 34, pages 27263--...

    2021

  48. [57]

    Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020 a . Pegasus: pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org

    2020

  49. [58]

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024 a . https://arxiv.org/abs/2401.02385 Tinyllama: An open-source small language model . Preprint, arXiv:2401.02385

    2024 arXiv

  50. [59]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2...

    2022 arXiv

  51. [60]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020 b . https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with bert . In International Conference on Learning Representations

    2020

  52. [61]

    Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. 2024 b . Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics, 12:39--57

    2024

  53. [62]

    Xingxing Zhang, Mirella Lapata, Furu Wei, and Ming Zhou. 2018. https://doi.org/10.18653/v1/D18-1088 Neural latent extractive document summarization . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 779--784, Brussels, Belgium. A...

    2018 doi

  54. [63]

    Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning, pages 12697--12706. PMLR

    2021

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.