The Narrative Construction of Generative AI Efficacy by the Media: A Case Study of the Role of ChatGPT in Higher Education

Aditya Johri; Ali Unlu; Yinan Sun

REVIEW 4 major objections 4 minor 8 references

The Narrative Construction of Generative AI Efficacy by the Media: A Case Study of the Role of ChatGPT in Higher Education

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read US news media built a predominantly positive narrative about ChatGPT in higher education, framing it as an opportunity for adaptation rather than a threat.

desk verdict A useful descriptive topic map of U.S. media coverage of ChatGPT in higher ed, but the 65.2% positive-sentiment headline rests on a 30-label training set and should be treated as tentative. read the letter →

arxiv 2507.09239 v1 pith:NMSLCZQB submitted 2025-07-12 cs.CY

Yinan Sun , Ali Unlu , Aditya Johri This is my paper

classification cs.CY

keywords ChatGPTgenerativeAIhighereducationnewsmedianarrativesagenda-settingtheoryLDAtopicmodelingsentimentanalysisSetFit

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that U.S. printed news media, in the first two years after ChatGPT's release, constructed a largely positive narrative about the tool's role in higher education. Analyzing 198 articles from November 2022 to October 2024, it finds six recurring topics and reports that 65.2 percent of articles carried positive sentiment. The most favorable coverage concerns policy, curriculum, and teaching practices and human-centered learning, while the most negative concerns college admissions and integrity and entry-level job displacement. If the finding holds, media framing is likely steering public and institutional expectations toward adaptation and opportunity rather than disruption and risk. The study applies agenda-setting theory to generative AI, treating the news media's topic selection and tone as a force shaping how society receives ChatGPT.

What carries the argument

The argument is carried by two computational tools paired with agenda-setting theory. LDA (Latent Dirichlet Allocation) is the generative probabilistic model that groups documents into latent topics through word co-occurrence; here a six-topic solution was chosen by coherence and perplexity scores. SetFit (Sentence Transformer Fine-Tuning) is a few-shot fine-tuning framework that classifies each article's sentiment as positive or negative after training on 30 hand-labeled articles. Agenda-setting theory supplies the interpretive machinery: first-level agenda setting maps to topic frequency (what the media makes salient), and second-level agenda setting maps to sentiment tone (what attributes the media attaches). The central claim depends on these tools converting raw news text into topic and sentiment measurements.

What would settle it

Have two independent human coders label all 198 articles as positive or negative; if the human-coded share of positive articles falls far below 65.2 percent, or near 50 percent, the central positivity claim fails. A cheaper check is to retrain the SetFit model several times on different 70/30 splits of the 30 labeled articles and see whether accuracy and the resulting positive share of the 198 articles remain stable.

Watch

Extended reading notes

Core claim

The paper claims that U.S. news media's discourse about ChatGPT in higher education is constructive and positive, emphasizing adaptation and opportunity over disruption and risk. The authors extracted six topics from 198 articles using LDA topic modeling: jobs/industry/young workers; AI/ML skills and career aspirations; collaboration/decision-making/bias; college admissions/academic integrity; policy/curriculum/teaching practices; and human-centered learning/teaching. Sentiment analysis with SetFit classified 129 of the 198 articles (65.2 percent) as positive. Topic 5 (policy, curriculum, and teaching practices) showed the strongest positive sentiment, while Topic 4 (admissions and integrity) showed the strongest negative sentiment and Topic 1 (jobs and young workers) was also negative. The findings are interpreted through two-level agenda-setting: topic frequency transfers salience, and tone transfers attributes, together producing a sociotechnical imaginary of AI in education as a reform tool.

Load-bearing premise

The 65.2 percent positive figure depends on a sentiment model trained on 30 hand-labeled articles and tested on 9, so the model must generalize to 168 unseen articles without any demonstrated stability.

Editorial extensions

If this is right

If the narrative is as constructive as the paper claims, public and institutional attention is being directed toward how to integrate generative AI through policy, curriculum, and teaching rather than toward whether to ban it.
The two negatively framed topics (college admissions and integrity, and entry-level jobs and young workers) mark the places where resistance to ChatGPT in higher education is most likely to persist.
The positive U.S. higher-education frame contrasts with more alarmist frames reported in other countries and other beats, so studies of AI coverage should be domain-specific.
For universities and AI developers, the topic-level sentiment map identifies where public trust is weakest and where communication efforts would have the most work to do.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not track who is quoted in the articles; if the positive tone comes mostly from university administrators and faculty, the media narrative may be an echo of institutional self-promotion rather than independent journalism. Tracing quoted sources across the 198 articles would test this.
Agenda-setting predicts an audience effect the paper does not measure: if the media really are setting the agenda, public surveys in the same period should show higher perceived benefits than risks for ChatGPT in education specifically, and that gap should widen with news exposure.
The domain contrast the paper finds, positive in education but negative in admissions and jobs, suggests the same technology is framed differently across beats; a paired analysis of the same outlets' business or technology coverage would reveal whether the positivity is a higher-education artifact or an outlet-level stance.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a decent descriptive mapping of how U.S. print media talked about ChatGPT in higher education between 2022 and 2024. The six-topic taxonomy is plausible and the close reading gives it a texture that pure LDA usually lacks. But the headline number — 65.2% of articles positive — comes from a sentiment model trained on 30 hand-labeled articles and tested on 9. I would not cite that percentage as anything but a hypothesis.

What the paper does well: it fills a genuine gap (most prior coverage studies were UK, Malaysia, Japan, Germany), the corpus construction is transparent with clear inclusion criteria, and the topic prevalence results (policy/curriculum 39%, human-centered learning 21%) are interpretable and useful. The authors are also honest in the limitations section about the binary sentiment scale and the small dataset.

Where it falls down: the sentiment analysis is the load-bearing part of the 'constructive and positive narrative' claim, and it is structurally under-powered. Thirty labels is tiny even for SetFit; a 70/30 split leaves 9 test articles. The reported 88.89% accuracy has a Wilson interval from about 57% to 98%, so you cannot distinguish a mediocre classifier from a good one. No cross-validation, no inter-coder reliability, no code or data to audit. Because the model is forced to label every article positive or negative, neutral pieces get pushed to one side, and the 15/15 training balance may create a threshold bias. A shift of even 10 percentage points would change the takeaway from 'predominantly positive' to 'roughly balanced.'

The topic-level sentiment orderings (Topic 5 positive, Topic 4 negative) are more plausible qualitatively, because they align with the content summaries, but they inherit the same classifier noise.

The agenda-setting framing is mostly a label: the paper does not measure salience transfer or public opinion, which would be needed to substantiate that claim. That is a framing issue, not fatal.

Who is this for? Researchers in education technology or media studies who want a snapshot of U.S. media discourse. It is not evidence about ChatGPT's actual efficacy. A serious editor should send it to review, with the expectation that the authors either release the labeled data and code, add uncertainty quantification, or soften the positivity claim into a tentative pattern. I would not desk-reject it.

Referee Report

4 major / 4 minor

Summary. The paper examines how U.S. print news media constructed narratives about ChatGPT's role in higher education between November 2022 and October 2024. After filtering Nexis Uni results to 198 articles, the authors apply LDA topic modeling to identify six themes and a SetFit sentiment classifier to label articles as positive or negative. They report that 65.2% of articles are positive, with the most positive sentiment in policy/curriculum and human-centered learning topics and the most negative in college admissions and jobs/young workers. The findings are interpreted through first- and second-level agenda-setting theory, concluding that U.S. media have formed a constructive, positive narrative emphasizing adaptation and opportunity over disruption and risk.

Significance. If the empirical claims were fully supported, the paper would be a useful contribution to media discourse research on generative AI in education, and its agenda-setting framing is appropriate for the research questions. The study addresses a genuine gap—U.S.-specific print media coverage of ChatGPT in higher education—and the topic taxonomy is potentially reusable. The authors are also candid about several limitations, including the binary sentiment scale and small dataset. However, the central quantitative claim of a predominantly positive narrative rests on a sentiment classifier validated on only nine test instances, which is not sufficient evidence as reported. The contribution is therefore promising but not yet convincing.

major comments (4)

[§4.3.2, §5.2] The sentiment classifier is trained on 30 hand-labeled articles (15 positive, 15 negative) and evaluated on a single 70/30 split of 9 test instances, yielding 8/9 correct (88.89% accuracy, F1 0.886). No cross-validation, confidence interval, or inter-coder reliability for the 30 labels is reported. With n=9, the 95% Wilson interval for accuracy is roughly 57–98%, so the model's true error rate is not pinned down. The corpus-level 65.2% positive figure and the topic-level sentiment ordering in Section 5.2 depend directly on applying this model to 168 unseen articles. I recommend a larger labeled set, k-fold cross-validation with classwise metrics, confidence intervals, inter-coder reliability, and an error analysis; the headline positivity claim is not supported by the currently reported validation.
[§4.3.2, §6.1] The model uses a binary positive/negative scale with no neutral category, and the labeled training set is deliberately balanced at 15/15. Applied to a corpus that likely contains many mixed or neutral articles, this design forces each article to one pole; if the classifier has even a moderate positive bias, the 65.2% estimate could shift by well over 10 percentage points. The limitation is acknowledged in Section 6.1, but no sensitivity analysis is provided. I recommend evaluating with a neutral category or, at a minimum, quantifying robustness through a threshold analysis or manual review of a random sample of model predictions.
[§5.2, Fig. 3] Topic-level average sentiments are compared (e.g., 0.58 for Topic 5, -0.29 for Topic 4) and used to support the conclusion of a largely positive, constructive narrative, but no error bars, confidence intervals, or statistical tests are reported. The topics have highly unequal article counts (77 versus 10), so the averages have very different uncertainties and the ranking may be within noise. Additionally, the text states that a monthly sentiment analysis was conducted, but no monthly figure or table appears in the results; either the corresponding display should be added or the claim should be removed.
[§4.1, §4.3.1] The filtering pipeline from 3,313 to 198 articles involves judgment calls for near-duplicate removal and exclusion criteria, but no inter-coder reliability for these screening decisions is reported. Likewise, the LDA topic selection is described only as choosing the six-topic solution as optimal based on coherence and perplexity, without reporting the quantitative scores, model parameters, random seeds, or validation of the single-topic assignment of each article. For reproducibility and confidence in RQ1, please report the coherence/perplexity values, model settings, and ideally a reliability check on topic labeling or multi-label assignment.

minor comments (4)

[§4.3.2] The subsection heading 'Sentimental Analysis' should be 'Sentiment Analysis' to match standard terminology.
[§5.2, References] There are citation inconsistencies: 'Burk & O'Brien' in Section 5.2 should be 'Burke & O'Brien' as in the reference list, and the 'Inside UNC Charlotte' reference is dated 2024 in the reference list but cited as 2023 in the text.
[Fig. 1] The caption says 'Coherence and Perplexity Scores for Six Topics in LDA Modeling,' but the comparison is conducted for 5 to 10 topics; the caption should reflect the full sweep.
[References] Some entries in the reference list (e.g., Bryan, 2023; Teague, 2015) do not appear to be cited in the text; please ensure all listed references are cited and all in-text citations are in the list.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the sentiment and topic claims are empirical outputs of models applied to unseen articles, not re-statements of the labeled inputs.

full rationale

The paper's central claims—six LDA topics, a 65.2% positive-sentiment rate, and topic-level sentiment ordering—are produced by unsupervised topic modeling and a SetFit classifier fine-tuned on 30 hand-labeled articles. The labeled set is a balanced 15-positive/15-negative training input, while the reported corpus-level positivity rate is a prediction over the remaining articles and is not a parameter fitted to reproduce that rate. No equation in the paper defines the topic proportions or sentiment percentages in terms of the training labels, and no self-citation carries a load-bearing argument; the agenda-setting framework is an interpretive lens rather than an input to the measurements. The small labeled set, single 9-article test split, and absence of confidence intervals are threats to statistical reliability and generalizability, but they do not make the 65.2% estimate equivalent to an input by construction. The paper's own limitations section acknowledges the coarse positive/negative dichotomy and small dataset, which further confirms that these are validity concerns rather than circular reasoning. Therefore, the derivation chain is self-contained with respect to circularity, and the appropriate score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The main findings depend on a hand-selected topic count and a sentiment model trained on a tiny labeled set. The theoretical lens (agenda-setting) is assumed, not tested. No new entities are introduced.

free parameters (4)

Number of LDA topics (k) = 6
Selected among 5-10 using coherence and perplexity (Section 4.3.1). The topic taxonomy depends on this choice.
SetFit training iterations = 100
Chosen by hand in Section 4.3.2, not via a validation sweep.
SetFit epochs = 5
Chosen by hand in Section 4.3.2.
SetFit batch size = 4
Chosen by hand in Section 4.3.2.

assumptions (3)

domain assumption Agenda Setting Theory implies media coverage influences public issue salience.
Invoked in Section 3 to frame the research questions, but not empirically tested.
domain assumption Each article can be assigned to a single dominant topic.
Used in Section 4.3.1 and acknowledged in Section 6.1 as a simplification.
domain assumption The Nexis Uni corpus after manual screening represents U.S. news media discourse.
Assumed in Section 4.1; screening criteria are subjective and not validated externally.

how reviews work

0 comments

Cite this review

Pith. "Pith review of The Narrative Construction of Generative AI Efficacy by the Media: A Case Study of the Role of ChatGPT in Higher Education." pith.science (2026). https://pith.science/paper/NMSLCZQB

@misc{pith2026250709239,
  author       = {Pith},
  title        = {Pith review of: The Narrative Construction of Generative AI Efficacy by the Media: A Case Study of the Role of ChatGPT in Higher Education},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NMSLCZQB}},
  note         = {Machine review of arXiv:2507.09239}
}

read the original abstract

The societal role of technology, including artificial intelligence (AI), is often shaped by sociocultural narratives. This study examines how U.S. news media construct narratives about the efficacy of generative AI (GenAI), using ChatGPT in higher education as a case study. Grounded in Agenda Setting Theory, we analyzed 198 articles published between November 2022 and October 2024, employing LDA topic modeling and sentiment analysis. Our findings identify six key topics in the media discourse, with sentiment analysis revealing generally positive portrayals of ChatGPT's integration into higher education through policy, curriculum, teaching practices, collaborative decision-making, skill development, and human-centered learning. In contrast, media narratives express more negative sentiment regarding their impact on entry-level jobs and college admissions. This research highlights how media coverage can influence public perceptions of GenAI in education and provides actionable insights for policymakers, educators, and AI developers navigating its adoption and representation in public discourse.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

8 extracted references · 7 canonical work pages

[1]

H., & Cheragui, M

Abdedaiem, A., Dahou, A. H., & Cheragui, M. A. (2023). Fake news detection in low resource languages using the setfit framework. Inteligencia Artificial, 26(72), 178-201. Akpan, I. J., Kobara, Y. M., Owolabi, J., Akpan, A. A., & Offodile , O. F. (2025). Conversational and generative artificial intelligence and human–chatbot interaction in education and re...

work page 2023
[10]

Brossard, D. (2013). New media landscapes and the science information consumer. Proceedings of the National Academy of Sciences, 110(supplement_3), 14096-14101. Bruinsma, B., Fredén, A., Hansson, K., Johansson, M., Kisić -Merino, P., & Saynova, D. (2024). Setting the AI Agenda —Evidence from Sweden in the ChatGPT Era. arXiv Preprint arXiv:2409.16946. Brya...

work page Pith review arXiv 2013
[66]

O’keeffe, A. (2013). Media and discourse analysis. In The Routledge handbook of discourse analysis (pp. 441-454). Routledge. Pannerselvam, K., Rajiakodi, S., Thavareesan, S., Thangasamy, S., & Ponnusamy, K. (2024, March). Setfit: A robust approach for offensive content detection in tamil-english code-mixed conversations using sentence transfer fine-tuning...

work page 2013
[110]

A., Sevenans, J., & Van Aelst, P

Harder, R. A., Sevenans, J., & Van Aelst, P. (2017). Intermedia agenda-setting in the social media age: How traditional players dominate the news agenda in election times. The International Journal of Press/Politics, 22(3), 275–293. Hairgrove, T. (2023, May 17). Multiple tamu-C students exonerated after being suspected of using AI to cheat. https://www.he...

work page arXiv 2017
[384]

(2024, May 15)

Levin, J. (2024, May 15). How well can yu faculty detect chatgpt?. https://yucommentator.org/. https://yucommentator.org/2024/05/how-well-can-yu-faculty-detect-chatgpt. Li, L., Ma, Z., Fan, L., Lee, S., Yu, H., & Hemphill, L. (2024). ChatGPT in education: A discourse analysis of worries and concerns on social media. Education and Information Technologies ...

work page 2024
[1127]

Landscape of Generative AI in Global News: Topics, Sentiments, and Spatiotemporal Analysis

Wilson, J. (2023, February 3). Writing without thinking? there’s a place for CHATGPT - if used properly: Guest commentary. Baltimore Sun. https://www.baltimoresun.com/2023/02/03/writing-without-thinking-theres-a-place-for- chatgpt-if-used-properly-guest-commentary. Wiseman, P. (2024, September 2). Ai may not steal many jobs after all. it may just make wor...

work page Pith review arXiv 2009
[1301]

Y., Richard, L

Gan, L., Yang, T., Huang, Y., Yang, B., Luo, Y. Y., Richard, L. W. C., & Guo, D. (2023, October). Experimental comparison of three topic modeling methods with LDA, Top2Vec and BERTopic. In International Symposium on Artificial Intelligence and Robotics (pp. 376-391). Singapore: Springer Nature Singapore. Gattone, C. F. (2012). The social scientist as publ...

work page 2012
[2020]

(2023a, January 26)

Singer, N. (2023a, January 26). https://www.nytimes.com/2023/01/26/technology/ai -masters- degree-texas.html Sok, S., & Heng, K. (2024). Opportunities, challenges, and strategies for using ChatGPT in higher education: A literature review. Journal of Digital Educational Technology, 4(1), ep2401. Singer, N. (2023b, September 1). Ban or embrace? colleges wre...

arXiv 2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

H., & Cheragui, M

Abdedaiem, A., Dahou, A. H., & Cheragui, M. A. (2023). Fake news detection in low resource languages using the setfit framework. Inteligencia Artificial, 26(72), 178-201. Akpan, I. J., Kobara, Y. M., Owolabi, J., Akpan, A. A., & Offodile , O. F. (2025). Conversational and generative artificial intelligence and human–chatbot interaction in education and re...

work page 2023

[2] [10]

Brossard, D. (2013). New media landscapes and the science information consumer. Proceedings of the National Academy of Sciences, 110(supplement_3), 14096-14101. Bruinsma, B., Fredén, A., Hansson, K., Johansson, M., Kisić -Merino, P., & Saynova, D. (2024). Setting the AI Agenda —Evidence from Sweden in the ChatGPT Era. arXiv Preprint arXiv:2409.16946. Brya...

work page Pith review arXiv 2013

[3] [66]

O’keeffe, A. (2013). Media and discourse analysis. In The Routledge handbook of discourse analysis (pp. 441-454). Routledge. Pannerselvam, K., Rajiakodi, S., Thavareesan, S., Thangasamy, S., & Ponnusamy, K. (2024, March). Setfit: A robust approach for offensive content detection in tamil-english code-mixed conversations using sentence transfer fine-tuning...

work page 2013

[4] [110]

A., Sevenans, J., & Van Aelst, P

Harder, R. A., Sevenans, J., & Van Aelst, P. (2017). Intermedia agenda-setting in the social media age: How traditional players dominate the news agenda in election times. The International Journal of Press/Politics, 22(3), 275–293. Hairgrove, T. (2023, May 17). Multiple tamu-C students exonerated after being suspected of using AI to cheat. https://www.he...

work page arXiv 2017

[5] [384]

(2024, May 15)

Levin, J. (2024, May 15). How well can yu faculty detect chatgpt?. https://yucommentator.org/. https://yucommentator.org/2024/05/how-well-can-yu-faculty-detect-chatgpt. Li, L., Ma, Z., Fan, L., Lee, S., Yu, H., & Hemphill, L. (2024). ChatGPT in education: A discourse analysis of worries and concerns on social media. Education and Information Technologies ...

work page 2024

[6] [1127]

Landscape of Generative AI in Global News: Topics, Sentiments, and Spatiotemporal Analysis

Wilson, J. (2023, February 3). Writing without thinking? there’s a place for CHATGPT - if used properly: Guest commentary. Baltimore Sun. https://www.baltimoresun.com/2023/02/03/writing-without-thinking-theres-a-place-for- chatgpt-if-used-properly-guest-commentary. Wiseman, P. (2024, September 2). Ai may not steal many jobs after all. it may just make wor...

work page Pith review arXiv 2009

[7] [1301]

Y., Richard, L

Gan, L., Yang, T., Huang, Y., Yang, B., Luo, Y. Y., Richard, L. W. C., & Guo, D. (2023, October). Experimental comparison of three topic modeling methods with LDA, Top2Vec and BERTopic. In International Symposium on Artificial Intelligence and Robotics (pp. 376-391). Singapore: Springer Nature Singapore. Gattone, C. F. (2012). The social scientist as publ...

work page 2012

[8] [2020]

(2023a, January 26)

Singer, N. (2023a, January 26). https://www.nytimes.com/2023/01/26/technology/ai -masters- degree-texas.html Sok, S., & Heng, K. (2024). Opportunities, challenges, and strategies for using ChatGPT in higher education: A literature review. Journal of Digital Educational Technology, 4(1), ep2401. Singer, N. (2023b, September 1). Ban or embrace? colleges wre...

arXiv 2024