Pith. sign in

REVIEW 2 major objections 6 minor 31 references

JRE-L: Journalist, Reader, and Editor LLMs in the Loop for Science Journalism for the General Audience

T0 review · 2 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a loop of three small open-source LLMs—journalist, reader, and editor—produces science articles with lower readability-formula scores than GPT-4, across three benchmark corpora.

desk verdict The reader-as-probe loop is a genuinely neat idea, but the paper's headline claim that small open models beat GPT-4 on accessibility rests entirely on surface readability formulas, and the paper's own human evaluation does not support it. read the letter →

arxiv 2501.16865 v1 pith:BZKHIHEN submitted 2025-01-28 cs.CL

Gongyao Jiang , Xinran Shi , Qiong Luo This is my paper · ORCID
classification cs.CL
keywords sciencejournalismlargelanguagemodelsmulti-agentcollaborationreadabilitylaysummarizationiterativerevisiongeneralaudienceautomatic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes JRE-L, a writing loop in which three open-source language models take the roles of journalist, general reader, and editor, and argues that this loop makes automatically generated popular science articles more readable than existing methods. The concrete claim is that two 7B-parameter models and one 1.8B-parameter model, cooperating without any training, produce articles whose Coleman-Liau, Flesch-Kincaid, and Dale-Chall scores are lower than those of articles prompted from GPT-4 and other strong baselines, with the gap statistically significant at p<0.01. The mechanism is that a deliberately weak reader model takes notes on the journalist's draft, and its note quality exposes writing problems that the editor then turns into concrete revision advice. If true, the result matters because it suggests that resource-light, parameter-free role-play can match or beat much larger closed models on an accessibility metric for science communication.

What carries the argument

The load-bearing object is the JRE-L loop itself: an iterative, training-free cycle in which a 7B journalist model, a 1.8B reader model, and a 7B editor model pass prompts in a write-read-assess-revise sequence. The reader's note-taking task is the distinctive mechanism; the authors deliberately choose a smaller, weaker model so that unclear writing produces incomplete or confused notes, converting article readability into measurable signal through error propagation. The editor reads the paper, the article, and the notes, then issues specific revision advice; the journalist incorporates the advice and the cycle repeats, with readability gains concentrated in the first iterations and plateauing by the third.

What would settle it

Give non-expert readers a comprehension test on paired articles generated by JRE-L and GPT-4 without revealing which is which; if the JRE-L articles are not understood better despite lower readability-formula scores, the paper's central measure of accessibility fails. The paper's own human evaluation already hints at this, since raters found no statistically significant difference in readability between JRE-L and GPT-4.

Watch

Extended reading notes

Core claim

JRE-L's central claim is that the three-role collaboration loop—journalist writes, a smaller reader model reads and takes notes, an editor evaluates the notes and issues revision suggestions, journalist revises—produces popular science articles with lower automatic readability scores (CLI, FKGL, DCRS) than single-LLM prompting, including GPT-4, across three benchmark corpora (SCITech, eLife, PLOS). The paper further claims that the reader LLM is the load-bearing element: because the 1.8B reader has weaker comprehension, its notes become comprehensive only when the article explains technical terms clearly, so note quality propagates readability problems back to the editor. In human evaluation, JRE-L received numerically higher readability ratings than GPT-4 without a statistically significant difference on any dimension; the paper interprets this as achieving GPT-4-level accessibility with far smaller models.

Load-bearing premise

The paper's quantitative claim rests on treating the Coleman-Liau Index, Flesch-Kincaid Grade Level, and Dale-Chall Readability Score as valid measures of how accessible an article is to the general public.

Editorial extensions

If this is right

  • Science journalism pipelines could achieve GPT-4-level readability using open-source 7B and 1.8B models, cutting cost and avoiding closed APIs.
  • The framework requires no fine-tuning or parameter updates, so it can be applied to new domains by changing the input paper alone.
  • Ablation shows that removing the reader, the editor, or the collaboration degrades readability scores, implying all three roles contribute to the improvement.
  • The loop generalizes across model families (Qwen and LLaMA) and across scientific domains (computer science, biomedical, and life sciences).
  • Readability gains saturate around the third iteration, suggesting a practical stopping rule for the revision cycle.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's headline claim is about readability formulas, not verified comprehension; a fair follow-up would test whether lower CLI, FKGL, and DCRS actually translate into better understanding by lay readers.
  • The reader-as-error-propagation idea could extend beyond journalism to any text simplification task where the target audience's likely misunderstandings can be simulated by a weaker model.
  • If human raters do not perceive a significant readability difference from GPT-4, the practical advantage of JRE-L may lie in cost and openness rather than in raw accessibility.
  • The framework's editor depends on the reader's notes; replacing note-taking with a direct readability judgment failed in pilot experiments, which suggests the value comes from the proxy reader rather than from LLM self-assessment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes JRE-L, a framework in which three open-source LLMs (a 7B journalist, a 1.8B reader, and a 7B editor) collaborate in an iterative write-read-feedback-revise loop to turn scientific paper abstracts into popular-science articles. The journalist drafts an article, the smaller reader LLM takes notes to expose comprehension failures, and the editor evaluates the notes and gives revision advice, which the journalist incorporates over multiple iterations. The authors evaluate on three existing corpora (SCITech, eLife, PLOS) using the Coleman-Liau Index, Flesch-Kincaid Grade Level, and Dale-Chall Readability Score, plus a four-annotator human study covering readability, information conveyance, authenticity, and interestingness. They report lower readability-formula scores for JRE-L than for all baselines, including GPT-4, and report human ratings close to those of GPT-4. They also present ablations, iteration trends, and case studies, and release their code.

Significance. If the central claim holds, the paper is a useful empirical contribution: it shows that a small, fully open-source ensemble of LLMs can match or beat a much larger closed-source model on readability metrics for popular-science writing, and it provides a reproducible recipe (roles, prompts, iteration) plus extensive ablations. The framework is simple and the code release enables follow-up. The paper also documents failed design attempts, which is informative. The main quantitative advantage, however, is only as strong as the validity of the readability formulas, and the human evaluation does not independently confirm an advantage over GPT-4; this limits the significance of the headline 'more accessible' claim to the narrow metric level.

major comments (2)
  1. [Section 4.2 (Table 1) and Section 4.3 (Table 2)] The abstract claims JRE-L generates articles 'more accessible' than GPT-4, but the only statistically significant evidence for this is the automatic readability-formula scores (CLI, FKGL, DCRS) in Table 1. These formulas count characters, syllables, sentence length, and word frequency (Section 4.1) and do not measure comprehension by a general audience. The human evaluation in Table 2 shows no statistically significant difference between JRE-L and GPT-4 on the 'Readability' dimension (3.95 vs 3.80 within field; 3.65 vs 3.40 outside field). The Limitations section itself concedes that 'these statistical measures may miss semantic information.' Since the framework's iterative loop plausibly optimizes exactly the surface features the formulas reward, the headline advantage over GPT-4 may be an artifact of the metrics rather than a genuine gain in accessibility. I ask the authors to either (a) provide a human or model-based comprehension measure that validates the formula difference, or (b) revise the abstract's wording to state that JRE-L improves readability-formula scores.
  2. [Appendix C (Hyperparameters)] The final reported output is selected post hoc: the authors state, 'We iterate five times and empirically select the output from the third iteration as the final result.' This selection is made on the test set, and Table 1 reports the scores of this chosen iteration without accounting for the selection process. The statistical significance marks (†, ††) therefore compare the best-of-five-iterations output of JRE-L against single-shot outputs of baselines, which is an unfair comparison and inflates the apparent advantage. The paper should either fix the iteration count and the chosen index in advance (e.g., by a validation split), apply a multiple-comparison correction for the five iterations, or report the full trajectory for all baselines with a pre-specified stopping rule.
minor comments (6)
  1. [Figure 4 caption] The caption spells 'Iteraction' instead of 'Iteration.'
  2. [Figure 2] The label 'Sugesstions for article revision' contains a typo; it should be 'Suggestions for article revision.'
  3. [Section 5.2] The text says 'As depicted in Table 7' when referring to ablation results, but the ablation results appear in Table 3; the detailed results are in Appendix F, Table 9. Please correct the cross-reference.
  4. [Section 4.3] The human evaluation uses only four participants and reports Krippendorff's alpha of 0.52, which the authors themselves describe as 'slightly lower' than prior work's 0.57. With such low inter-annotator agreement and a small sample, the absence of a significant difference between JRE-L and GPT-4 on readability should be interpreted cautiously; this point should be acknowledged in the main text, not only implicitly through the table.
  5. [Appendix C] The statement that 'all approaches in our comparison study share the same set of hyperparameters setting' is misleading because the JRE-L framework has an iteration count and a selected iteration index, while single-LLM baselines do not. Please clarify which hyperparameters are shared and which are specific to the iterative framework.
  6. [Appendix B] The 'Failed Attempts' appendix is useful, but it would be even more helpful to report the quantitative evidence for why the alternative formulations (reflection, reading comprehension, single-prompt) underperformed, rather than only describing them qualitatively.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: JRE-L's generation loop is not the evaluation metric, and its claims are tested against external baselines and external readability formulas.

full rationale

The derivation chain in the paper is p0 = J(x), ri = R(pi-1), si = E(pi-1, ri), pi = J(x, pi-1, si); none of these equations references, fits, or computes CLI, FKGL, or DCRS. The readability formulas are standard, pre-existing metrics adopted from prior work (Goldsack et al., 2022; Cardenas et al., 2023), and Table 1 compares JRE-L against many external baselines, including GPT-4, fine-tuned models, and other multi-agent frameworks, on held-out test splits of SCITech, eLife, and PLOS. The reader LLM's notes and the editor's suggestions are internal control signals in the generation loop, not the evaluation outcome; Appendix B documents failed alternative designs, showing the framework was not constructed by defining the metric and then reading out the result. The abstract's claim of 'more accessible' articles is statistically supported mainly by the readability formulas, and the human evaluation (Table 2) shows no significant advantage over GPT-4; this is a metric-validity or overclaiming concern, not derivation-level circularity. The authors themselves acknowledge in the Limitations section that 'these statistical measures may miss semantic information,' confirming that the formulas are treated as proxies rather than as the definition of accessibility. The one self-citation in Section 3.1 (Zhang et al., 2024, which includes the first author) is a routine prompt-strategy citation and is not load-bearing for the framework's correctness or for any stated result. Therefore, no specific reduction of a prediction to its input, no fitted parameter renamed as a prediction, and no self-citation chain forcing the conclusion is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The framework introduces no new physical or formal entities. It rests on domain assumptions about readability measurement and the reader-proxy mechanism, plus a few empirically chosen hyperparameters.

free parameters (4)
  • iteration count n = 5
    Number of revision rounds; chosen empirically, not theoretically justified.
  • selected iteration index = 3
    The output of the third iteration is selected as final 'for a balance of performance and cost-effectiveness' (Appendix C). This is a post-hoc choice based on the observed readability trend.
  • reader model size = Qwen-1.5-1.8B
    A smaller model is deliberately chosen to simulate a general reader. Replacement with a 7B model yields similar results, so this is a design choice rather than a fitted parameter.
  • top_p = 0.4
    Sampling hyperparameter tuned for stability, shared across methods.
assumptions (4)
  • domain assumption CLI, FKGL, and DCRS readability formulas are valid proxies for accessibility to the general audience.
    Used to compare all methods and to support the 'more accessible' claim. The limitations section acknowledges these 'statistical indexes may miss semantic information.'
  • domain assumption A smaller, weaker reader LLM's note-taking quality reflects the comprehensibility of the article for non-experts.
    Central mechanism in Section 3.2; no external validation that note quality correlates with human comprehension.
  • domain assumption LLM agents reliably follow role instructions and produce constructive feedback.
    The entire loop depends on prompt adherence. Appendix B documents failures in some earlier attempts.
  • domain assumption The abstract of the original paper contains sufficient content to write a popular science article.
    Authors use abstracts as input, a choice noted in the limitations section.

how reviews work

0 comments
Cite this review

Pith. "Pith review of JRE-L: Journalist, Reader, and Editor LLMs in the Loop for Science Journalism for the General Audience." pith.science (2026). https://pith.science/paper/BZKHIHEN

@misc{pith2026250116865,
  author       = {Pith},
  title        = {Pith review of: JRE-L: Journalist, Reader, and Editor LLMs in the Loop for Science Journalism for the General Audience},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BZKHIHEN}},
  note         = {Machine review of arXiv:2501.16865}
}
read the original abstract

Science journalism reports current scientific discoveries to non-specialists, aiming to enable public comprehension of the state of the art. This task is challenging as the audience often lacks specific knowledge about the presented research. We propose a JRE-L framework that integrates three LLMs mimicking the writing-reading-feedback-revision loop. In JRE-L, one LLM acts as the journalist, another LLM as the general public reader, and the third LLM as an editor. The journalist's writing is iteratively refined by feedback from the reader and suggestions from the editor. Our experiments demonstrate that by leveraging the collaboration of two 7B and one 1.8B open-source LLMs, we can generate articles that are more accessible than those generated by existing methods, including prompting single advanced models such as GPT-4 and other LLM-collaboration strategies. Our code is publicly available at github.com/Zzoay/JRE-L.

Figures

Figures reproduced from arXiv: 2501.16865 by the authors.

Figure 1
Figure 1. An article written by a science journalist [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of our iterative ASJ framework, JRE-L. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Accessible content helps the reader take com [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Performance improvement over iterations, with iteration 0 producing the initial writing. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: The questionnaire for participants to evaluate articles. [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 28 canonical work pages

  1. [1]

    How well do you know about the topic of this content?

    work page

  2. [2]

    I have limited knowledge about this topic

    work page

  3. [3]

    I am somewhat familiar with this topic

    work page

  4. [4]

    markdown

    Llms as writing assistants: Exploring perspec- tives on sense of ownership and reasoning. arXiv preprint arXiv:2404.00027. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface’s transformers: State-of- the-art natural language processing...

    arXiv 2023

  5. [5]

    I have written research papers on this topic

    work page

  6. [7]

    I have never heard about this topic before

    work page

  7. [10]

    I have good knowledge and understanding of this topic

    work page

  8. [12]

    How well does this article convey the information of the original paper correctly?

    work page

Show all 31 references
  1. [13]

    How accurate is this article (without factual or common sense errors)?

  2. [14]

    How easy or difficult was it for you to read the article?

  3. [15]

    Very difficult to read

  4. [16]

    Somewhat difficult to read

  5. [17]

    Neutral/Neither easy nor difficult to read

  6. [18]

    Somewhat easy to read

  7. [19]

    How interesting did you find the article?

  8. [20]

    Not interesting at all

  9. [21]

    Slightly interesting

  10. [22]

    Moderately interesting

  11. [23]

    15 Journalist

    Very interesting Figure 5: The questionnaire for participants to evaluate articles. 15 Journalist. You are a science journalist for general audiences. Given a paper’s summary, you are assigned to rewrite it into a short understandable article for general audiences. Follow the ...

  12. [24]

    First, extract all technical terms with their context from the article

  13. [25]

    1...2...3

    Then, explain the technical terms based on their context. Follow the rules strictly: - Extraction should mention the specific location of each technical term in the article. - Explanation should be first extracted from the article; if not found, it can be some common-sense or ...

  14. [26]

    You are a senior editor

    Editor. You are a senior editor. Here are a scientific paper summary, and a short popular science article. A general reader has read the science article and takes some notes. Please take the following steps:

  15. [27]

    First, evaluate the **reader’s notes** based on these factors: content accuracy, lexical and technical complexity, and information conveyance (from the original content)

  16. [28]

    1...2...3

    Then, based on the above evaluation, list some brief yet informative writing advice that may benefit the popular science article, to make the article easier for general readers without specialized knowledge to read and understand. Specifically, the advice should benefit these ...

  17. [29]

    Revision You are a science journalist for general audiences. Given the paper summary and a short summary of the popular science article, you are assigned to rewrite the popular science summary for general audiences, who have no specialized knowledge. You are given some writing...

  18. [30]

    Choose and refine the most relevant and suitable advice for writing improvement

  19. [31]

    Follow the rules strictly: - Keep your article short yet informative

    Then, based on the refined advice and the paper summary, rewrite the popular science article. Follow the rules strictly: - Keep your article short yet informative. - Don’t include visualization, references or links. - Revision must not go beyond the original paper, but can be ...

  20. [299]

    Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Min Zhang, Qingsong Wen, Wei Ye, Shikun Zhang, and Yue Zhang

    Association for Computational Linguistics. Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Min Zhang, Qingsong Wen, Wei Ye, Shikun Zhang, and Yue Zhang. 2024b. Autosurvey: Large language models can automatically write surveys. CoRR,...

    arXiv

  21. [2021]

    Therefore, human evaluation remains the main method for such assessments

    to assess the faithfulness of ASJ-generated content, yet the results exhibited significant vari- ances. Therefore, human evaluation remains the main method for such assessments. We created a questionnaire for human evaluation using a 1-5 Likert scale, as shown in Figure 5. All...

    2024

  22. [2022]

    Open source software available from https://github.com/heartexlabs/label-studio

    Label Studio: Data labeling soft- ware. Open source software available from https://github.com/heartexlabs/label-studio. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale...

    2023 arXiv

  23. [2024]

    arXiv preprint arXiv:2403.17768

    Scinews: From scholarly complexities to pub- lic narratives–a dataset for scientific news report gen- eration. arXiv preprint arXiv:2403.17768. Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. 2024. Chatde...

    2024 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.