CoachGPT: A Scaffolding-based Academic Writing Assistant

Fumian Chen; Hui Fang; Joshua Wilson; Sotheara Veng; Xiaoming Li

REVIEW 4 major objections 5 minor 20 references

CoachGPT: A Scaffolding-based Academic Writing Assistant

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that an LLM-based writing assistant can teach academic writing by guiding students through eleven fixed stages instead of writing essays for them.

desk verdict A well-designed scaffolding assistant whose evaluation is far too weak to support its central claim of proven effectiveness. read the letter →

arxiv 2506.18149 v1 pith:U5C4ATE7 submitted 2025-06-22 cs.AI

Fumian Chen , Sotheara Veng , Joshua Wilson , Xiaoming Li , Hui Fang This is my paper · ORCID

classification cs.AI

keywords academicwritingassistantlargelanguagemodelsscaffoldingAIagentpersonalizedfeedbacknovicewriterssecond-language

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CoachGPT is a web-based academic writing assistant that tries to fix what the authors see as the central educational flaw of large language models: LLMs produce essays instead of teaching students to write. To do this, it converts an instructor's assignment into a fixed sequence of eleven sub-tasks, from pre-writing and resource selection through thesis, outline, paragraph drafting, revision, word choice, and grammar check, and at each stage an LLM gives real-time, criteria-based feedback while refusing to write for the student. The paper argues this scaffolding structure makes it unique among existing writing assistants and that three cognitive interviews show it is useful, user-friendly, and attentive. The intended significance is that students with limited access to instructors, including second-language writers, can get guided, self-paced writing instruction without the learning harms of asking a chatbot to write the essay for them.

What carries the argument

The load-bearing mechanism is the scaffolding structure: an educational process that meets learners at their current level and gradually builds skill toward independent performance. In CoachGPT this takes concrete form as eleven strictly ordered stages -- pre-writing, identifying resources, thesis statement, outline building, introduction, body paragraph, body paragraph wrap-up, conclusion, general revising, word choice evaluation, and grammar check -- implemented as an AI agent with a Vue.js front end, a FastAPI and LangChain backend for short-term memory, and a MySQL database for long-term memory. The LLM's behavior is controlled through prompt-engineering techniques: persona prompting ('act as a writing coach'), limiters that forbid rewriting and idea generation, criteria-based feedback requests, structured output to reduce cognitive load, and input validation to force meaningful engagement. Together these convert an educator's instructions into machine-managed sub-tasks, which is the mechanism the paper claims makes the assistant educational rather than generative.

What would settle it

Randomly assign forty novice academic writers to draft the same essay with CoachGPT or with a plain LLM chat interface, then have blind raters score both the final essays and a follow-up essay written without any tool; if the two groups score alike, the claim that the scaffolding structure improves writing is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that a writing assistant can teach academic writing rather than merely produce it by adopting a scaffolding strategy. CoachGPT ingests an assignment prompt, breaks the writing process into eleven stages that users must follow in order, and uses a large language model with a fixed 'writing coach' persona to give personalized feedback at each stage. The feedback is deliberately constrained so the model never rewrites the student's text or supplies essay ideas; instead it checks alignment, evaluates thesis statements and outlines against criteria, and flags word-choice and grammar issues. The authors assert that no existing writing assistant combines end-to-end scaffolding, real-time feedback, and personalized interaction in this way, and that their qualitative user study with three writers demonstrates the system's usefulness. On the paper's own terms, CoachGPT's contribution is the design of a responsible LLM writing coach, not a faster essay generator.

Load-bearing premise

The conclusion that CoachGPT improves academic writing depends entirely on three users saying they felt helped, with no objective measure of writing quality, no learning gain, and no comparison against a baseline tool.

Editorial extensions

If this is right

A writer can move through the entire essay process inside one tool, with the assignment goal, chat history, and stage-by-stage drafts saved for later reference.
The model's prompts prevent it from supplying essay content, so the finished text remains the student's own work rather than generated prose.
Users without reliable access to writing instructors or paid tools can receive structured, criteria-based guidance in a browser-based application.
Each stage's feedback is personalized to the user's own input, which the paper ties to an attentive and immersive writing experience.
CoachGPT provides a testbed for studying how large language models can be used responsibly in education, not just for text generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: The paper's own definition of scaffolding implies that support should fade as learners gain competence, but CoachGPT's eleven stages are fixed; testing an adaptive version that removes hints after successful stages would be a direct extension.
Inference: The paper frames the problem as LLMs 'generating essays without teaching,' yet it measures only user satisfaction, not learning; an independent study of writing gains would be needed before accepting that CoachGPT closes that gap.
Inference: The same 'educator instructions to sub-tasks' pipeline could apply to other genres and disciplines, since the eleven stages are generic essay-structure steps rather than domain-specific content.
Inference: The long-term memory and real-time feedback architecture could be reused for peer review or iterative revision studies, not only first-draft scaffolding.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

CoachGPT is worth reading for the design, not the evidence. The 11-stage scaffolding workflow is a genuine artifact: it forces a writing process instead of just generating text, and the no-rewriting constraint is a thoughtful way to keep LLM feedback educational. The prompt engineering section is concrete and useful. But the user study cannot bear the weight of the abstract's claim that it 'proves effectiveness.' Three cognitive interviews, no baseline, no objective writing or learning metric, and no released protocol or transcripts. Section 3.1 says the messages were fine-tuned based on 'our user study'—the same interviews used as evidence—so the evaluation is partly self-referential. That is not a minor flaw; it is the difference between a system paper and a claims-paper. The participants are not shown to represent the target population of under-resourced learners, and the reported outcomes are subjective impressions. At most this is preliminary usability feedback. I'd like to see the scaffolding design taken seriously in a proper study: a baseline condition (e.g., generic ChatGPT), a larger and more representative sample, and measures of revision quality or learning transfer. As it stands, the system is coherent and clearly described, and the authors engage honestly with related work; there is no internal contradiction. But the conclusions overreach the data. If the authors reframe it as a system/demo with preliminary usability findings, it is fine. As is, a serious referee should push for major revision or claim-tempering. I would send it to peer review rather than desk-reject, because the artifact deserves scrutiny and a good revision could make it solid. I wouldn't cite it in my own work yet, but I'd bring it to a reading group for the prompt design discussion.

Referee Report

4 major / 5 minor

Summary. The paper presents CoachGPT, a web-based LLM-powered academic writing assistant organized around an 11-stage linear scaffolding structure, from pre-writing and resource identification through thesis, outline, paragraph writing, revision, word choice, and grammar check. The system takes instructions from educators, decomposes them into sub-tasks, and uses prompt-engineered LLM responses to provide real-time, criteria-based feedback without rewriting the student's text. The authors claim that this scaffolding structure is unique among writing assistants and that their user study, consisting of three cognitive interviews, proves the system's effectiveness and usefulness, especially for students with limited educational resources and those who prefer self-paced learning.

Significance. If the effectiveness claims were adequately supported, CoachGPT would be a useful contribution to LLM-based writing pedagogy: it operationalizes a recognized educational theory (scaffolding) in a concrete, deployable tool and directly addresses an important limitation of current LLM assistants, namely that they generate essays rather than teach. The paper's strengths are its concrete system description, the explicit 11-stage pipeline, the detailed prompt-engineering techniques (persona prompting, constraints, criteria-based feedback, output presentation, input validation), and a working web deployment. These are valuable for builders of similar educational systems. However, the paper's central evidential claim is not supported by the evaluation as reported; the current evidence supports only preliminary usability feedback from a small qualitative study. Because the abstract and conclusion present this study as proof, the paper overclaims, and the significance of the contribution is currently undermined by that mismatch.

major comments (4)

[Abstract; Section 5; Section 6] The user study cannot bear the claim, made in the Abstract and in Section 6, that it "proves the effectiveness" of CoachGPT. The study consists of cognitive interviews with three participants only, analyzed with qualitative content analysis, and the reported outcomes are subjective impressions (e.g., "attentive," "adaptive," "intuitive"). There are no objective measures of writing quality, revision quality, learning gain, or task completion, and there is no baseline or comparison condition. Please either add a controlled study with objective outcome measures and a comparison condition or rephrase the abstract and conclusion to describe preliminary usability feedback rather than proof of effectiveness.
[Section 3.1] The statement that CoachGPT "fine-tuned messages at each stage based on our user study" makes the Section 5 evaluation partly self-referential: the system was adjusted on the basis of the same interviews that are then used to demonstrate its usefulness. This compromises the independence of the evidence. Please clarify precisely what was changed after the user study, report which version of the system the participants saw, and acknowledge this limitation when interpreting the results.
[Section 5] The qualitative study is reported without the methodological details needed to assess its reliability: no interview protocol, participant recruitment or inclusion criteria, codebook, inter-rater reliability calculation, or transcript excerpts with identifiers are provided. Without these, the reader cannot verify how the categories "personalized output," "navigation and interface," and "scaffolding needs" were derived, how many participants endorsed each point, or whether the analysis was consistent. Please provide the protocol, coding scheme, reliability information, and supporting excerpts.
[Section 5; Abstract] The paper motivates the system for "individuals with limited educational resources," but the three participants are not shown to represent that population: no demographic, language-background, or educational-resource information is given, and two participants are described as having "some experience" in academic writing. The absence of this information weakens the claimed generality of the findings. Please report participant background and either collect data from the targeted population or explicitly limit the generalization claims.

minor comments (5)

[Section 3.1] There are two typographical occurrences of "world choice" in the prose describing the final stages; the intended term appears to be "word choice."
[Section 2] The text states that previous studies "can be categorized into three categories" but then lists four: brainstorming, text structuring, text polishing, and on-demand feedback.
[Section 4 / Figure 2] Figure 2 is referenced but not described in the body text; a brief explanation of the tech-stack components would improve readability.
[Section 1; Section 5] The Introduction claims the interface is "user-friendly" and easy to use, but Section 5 reports that Jamie struggled with the term "key questions" and users requested interface changes; the earlier claim should be tempered or reconciled with these findings.
[References] Several reference entries contain apparent typos or inconsistencies, such as "Springer Opener" (likely Springer Open) in [1], "Schlichktrull" (likely Schlichtkrull) in [4], and inconsistent author name spellings in [12]; please correct these before publication.

Circularity Check

1 steps flagged · score 2.0 of 10

No mathematical derivation exists; the only circularity is a mild self-referential evaluation loop, since CoachGPT's messages were fine-tuned on the same user study that is later cited as proof of effectiveness.

fitted input called prediction [Section 3.1 (System Architecture, scaffolding structure) and Section 6 (Conclusion and Future Work)]
"CoachGPT fine-tuned messages at each stage based on our user study to ensure a personal and immersive writing experience. ... Our user study proves the effectiveness of CoachGPT in assisting academic writing with its user-friendly interface and attentive writing experience."

The paper states that CoachGPT's messages were fine-tuned based on 'our user study,' and then the same user study (Section 5, three cognitive interviews) is used in Section 6 as proof that CoachGPT is effective. The positive subjective reactions, such as feeling 'attentive' and 'adaptive,' are therefore not independent confirmation of the scaffolding design; they are partly the echo of the very feedback used to tune the system. This is a mild evaluative loop rather than a mathematical derivation reducing to its input, and the text does not state that the tuning study and the evaluation study were distinct.

full rationale

This paper contains no formal derivation, so the classic circularity failure mode (an equation or fitted parameter that is equivalent to the output by construction) does not apply. The central claim is an empirical usability claim supported by three cognitive interviews. The one genuine circularity concern is textual: Section 3.1 states that CoachGPT's messages were fine-tuned based on 'our user study,' and Section 6 invokes the same user study as proof of the system's effectiveness. Because the evaluation is entirely self-reported and no separate validation cohort or objective writing-quality measure is reported, the positive feedback partially reflects the tuning input. This warrants a score of 2 rather than 0. It is not higher because the claimed contribution is a system description plus preliminary qualitative feedback, not a derived quantitative result, and the paper contains no load-bearing self-citation chain. The insufficiency of three cognitive interviews as proof of broad effectiveness is an evidence-strength concern, not a circularity concern, and is therefore noted here but not scored as severe circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no fitted parameters or postulated entities; its burden is carried by domain assumptions about scaffolding theory, LLM feedback quality, and the validity of qualitative self-report as evidence of effectiveness.

assumptions (3)

domain assumption Scaffolding instruction improves learning outcomes.
Adopted from Vygotsky [13,20] and Lin et al. [9]; the paper's central design premise is that this 11-stage sequential scaffolding guides writers, but no empirical test isolates scaffolding's effect.
domain assumption LLM-generated feedback is accurate and pedagogically appropriate for academic writing.
The system relies on an unnamed LLM to evaluate reliability of resources, thesis statements, outlines, paragraphs, and grammar; no evaluation of feedback accuracy or safety is provided.
domain assumption Self-reported user satisfaction is a valid measure of writing-assistant effectiveness.
The user study measures only subjective experience via cognitive interviews, and the paper generalizes to usefulness and effectiveness without objective writing outcomes.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CoachGPT: A Scaffolding-based Academic Writing Assistant." pith.science (2026). https://pith.science/paper/U5C4ATE7

@misc{pith2026250618149,
  author       = {Pith},
  title        = {Pith review of: CoachGPT: A Scaffolding-based Academic Writing Assistant},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U5C4ATE7}},
  note         = {Machine review of arXiv:2506.18149}
}

read the original abstract

Academic writing skills are crucial for students' success, but can feel overwhelming without proper guidance and practice, particularly when writing in a second language. Traditionally, students ask instructors or search dictionaries, which are not universally accessible. Early writing assistants emerged as rule-based systems that focused on detecting misspellings, subject-verb disagreements, and basic punctuation errors; however, they are inaccurate and lack contextual understanding. Machine learning-based assistants demonstrate a strong ability for language understanding but are expensive to train. Large language models (LLMs) have shown remarkable capabilities in generating responses in natural languages based on given prompts. Still, they have a fundamental limitation in education: they generate essays without teaching, which can have detrimental effects on learning when misused. To address this limitation, we develop CoachGPT, which leverages large language models (LLMs) to assist individuals with limited educational resources and those who prefer self-paced learning in academic writing. CoachGPT is an AI agent-based web application that (1) takes instructions from experienced educators, (2) converts instructions into sub-tasks, and (3) provides real-time feedback and suggestions using large language models. This unique scaffolding structure makes CoachGPT unique among existing writing assistants. Compared to existing writing assistants, CoachGPT provides a more immersive writing experience with personalized feedback and guidance. Our user studies prove the usefulness of CoachGPT and the potential of large language models for academic writing.

Figures

Figures reproduced from arXiv: 2506.18149 by the authors.

**Figure 1.** CoachGPT’s Scaffolding Structure detrimental effects on learning when misused. They are typically not trained with education-focused data, and their implementations are not geared toward education needs. Specifically, LLMs can encourage plagiarism and undermine authentic learning. Many news reports about students using ChatGPT for their essay writing and assignments have raised concerns about ethical issues. The ab… view at source ↗

**Figure 2.** CoachGPT’s Tech Stack criteria (e.g., coherence, cohesion, clarity) and use simple language. One prompt stated: "Provide your response on the criteria in this order: spelling, grammar, and punctuation". Input Validation: Input validation ensures that students engage meaningfully with the tool. A prompt included: "If the user does not type any paragraph or just random text, please direct them to type the paragraph." … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 15 canonical work pages

[1]

Brian R. Belland. 2017.Instructional scaffolding in STEM education: Strategies and efficacy evidence. Springer Opener

work page 2017
[2]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners.Advances in neural information processing systems33 (2020), 1877–1901

2020
[3]

Chuxue Cao, Ziqing Yuan, and Hailiang Chen. 2024. ScholarGPT: Fine-Tuning Large Language Models for Discipline-Specific Academic Paper Writing. (2024)

work page 2024
[4]

Eric Chamoun, Michael Schlichktrull, and Andreas Vlachos. 2024. Automated focused feedback generation for scientific writing assistance.arXiv preprint arXiv:2405.20477(2024)

work page Pith review arXiv 2024
[5]

Tira Nur Fitria. 2021. Grammarly as AI-powered English writing assistant: Students’ alternative for writing English.Metathesis: Journal of English Language, Literature, and Teaching5, 1 (2021), 65–78

work page 2021
[6]

L Flower. 1981. A cognitive process theory of writing.Composition and commu- nication(1981)

work page 1981
[7]

Hsiu-Fang Hsieh and Sarah E. Shannon. 2005. Three approaches to qualitative content analysis.Qualitative Health Research15, 9 (2005), 1277–1288

2005
[8]

Yewon Kim, Mina Lee, Donghwi Kim, and Sung-Ju Lee. 2023. Towards ex- plainable ai writing assistants for non-native english speakers.arXiv preprint arXiv:2304.02625(2023)

arXiv 2023

Show all 20 references

[9]

Tzu-Chiang Lin, Ying-Shao Hsu, Shu-Sheng Lin, Maio-Li Changlai, Kun-Yuan Yang, and Ting-Ling Lai. 2012. A review of empirical evidence on scaffolding for science education.International Journal of Science and Mathematics Education10, 2 (2012), 437–455

2012
[10]

Yuanchao Liu, Xin Wang, Ming Liu, and Xiaolong Wang. 2016. Write-righter: An academic writing assistant system. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 30

2016
[11]

Mahmoudi and C

F. Mahmoudi and C. Bugra. 2020. The Effects of Using Rubrics and Face to Face Feedback in Teaching Writing Skill in Higher Education.International Online Journal of Education and Teaching7, 1 (2020), 150–158

2020
[12]

Justin Mücke, Daria Waldow, Luise Metzger, Philipp Schauz, Marcel Hoffman, Nicolas Lell, and Ansgar Scherp. 2023. Fine-Tuning Language Models for Scientific Writing Support. InInternational Cross-Domain Conference for Machine Learning and Knowledge Extraction. Springer, 301–318

2023
[13]

J. Nordlof. 2014. Vygotsky, scaffolding, and the role of theory in writing center work.The Writing Center Journal34 (2014), 45–64

2014
[14]

C. Olea, H. Tucker, J. Phelan, C. Pattison, S. Zhang, M. Lieb, and J. White. 2024. Evaluating persona prompting for question answering tasks. InProceedings of the 10th International Conference on Artificial Intelligence and Soft Computing. Sydney, Australia

2024
[15]

Olaf Resch and Aglika Yankova. 2019. Open knowledge interface: A digital assistant to support students in writing academic assignments. InProceedings of the 1st ACM SIGSOFT international workshop on education through advanced software engineering and artificial intelligence. 13–16

2019
[16]

Schulhoff, M

S. Schulhoff, M. Ilie, N. Balepur, K. Kahadze, A. Liu, C. Si, and P. Resnik. 2024. The Prompt Report: A Systematic Survey of Prompting Techniques.arXiv preprint arXiv:2406.06608 (2024)

2024 arXiv
[17]

Tony Silva. 1993. Toward an understanding of the distinct nature of L2 writing: The ESL research and its implications.TESOL quarterly27, 4 (1993), 657–677

1993
[18]

Koemhong Sol and Kimkong Heng. 2024. AI-powered chatbots as personalized academic writing assistants for non-native English speakers.Encyclopedia of educational innovation(2024), 1–5

2024
[19]

2011.Cognitive load theory

John Sweller, Paul Ayres, and Slava Kalyuga. 2011.Cognitive load theory. Springer, New York

2011
[20]

1978.Mind in society: The development of higher psychological processes

Lev Vygotsky. 1978.Mind in society: The development of higher psychological processes. Harvard University Press

1978

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Brian R. Belland. 2017.Instructional scaffolding in STEM education: Strategies and efficacy evidence. Springer Opener

work page 2017

[2] [2]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners.Advances in neural information processing systems33 (2020), 1877–1901

2020

[3] [3]

Chuxue Cao, Ziqing Yuan, and Hailiang Chen. 2024. ScholarGPT: Fine-Tuning Large Language Models for Discipline-Specific Academic Paper Writing. (2024)

work page 2024

[4] [4]

Eric Chamoun, Michael Schlichktrull, and Andreas Vlachos. 2024. Automated focused feedback generation for scientific writing assistance.arXiv preprint arXiv:2405.20477(2024)

work page Pith review arXiv 2024

[5] [5]

Tira Nur Fitria. 2021. Grammarly as AI-powered English writing assistant: Students’ alternative for writing English.Metathesis: Journal of English Language, Literature, and Teaching5, 1 (2021), 65–78

work page 2021

[6] [6]

L Flower. 1981. A cognitive process theory of writing.Composition and commu- nication(1981)

work page 1981

[7] [7]

Hsiu-Fang Hsieh and Sarah E. Shannon. 2005. Three approaches to qualitative content analysis.Qualitative Health Research15, 9 (2005), 1277–1288

2005

[8] [8]

Yewon Kim, Mina Lee, Donghwi Kim, and Sung-Ju Lee. 2023. Towards ex- plainable ai writing assistants for non-native english speakers.arXiv preprint arXiv:2304.02625(2023)

arXiv 2023