arxiv: 2605.05598 · v1 · submitted 2026-05-07 · 💻 cs.AI · cs.HC

Recognition: unknown

Prober.ai: Gated Inquiry-Based Feedback via LLM-Constrained Personas for Argumentative Writing Development

Ran Bi , Shiyao Wei , Yuanyiyi Zhou

Authors on Pith no claims yet

Pith reviewed 2026-05-08 11:50 UTC · model grok-4.3

classification 💻 cs.AI cs.HC

keywords LLM-constrained personasinquiry-based feedbackargumentative writinggated reflectionToulmin argumentation theoryAI tutoringcognitive preservationpersona-specific prompts

0 comments

The pith

Prober.ai constrains an LLM to generate only inquiry-based questions about argumentative weaknesses instead of rewriting student text.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents Prober.ai as a writing tool that reverses how AI typically assists students by limiting the model to asking targeted questions rather than producing improved versions of the writing. It uses a two-phase Challenge and Unlock process that requires students to reflect on those questions before any revision ideas become available. The questions are shaped around Toulmin's model of argumentation to focus on identifying weaknesses in claims, evidence, and reasoning. This setup addresses the problem of students outsourcing critical thinking to AI, which the authors link to weaker argumentative skills. If the approach works, it shows a way to incorporate AI into education while keeping the cognitive work of writing with the student.

Core claim

We present Prober.ai, a web-based writing environment that inverts the conventional AI-tutoring paradigm: rather than generating or rewriting student text, the system constrains an LLM through persona-specific system prompts and structured JSON output schemas to produce only targeted, inquiry-based questions about argumentative weaknesses, implemented via a gated Challenge and Unlock architecture that requires mandatory student reflection before revision suggestions are unlocked.

What carries the argument

The central mechanism is the use of persona-specific system prompts combined with structured JSON output schemas that restrict the LLM to outputting only inquiry-based questions aligned with Toulmin's argumentation theory.

If this is right

Revision suggestions remain unavailable until students have reflected on questions about their own arguments.
AI assistance supports writing instruction without replacing the student's effort to spot weaknesses.
Feedback can scale to many users while still prioritizing skill development over instant text fixes.
The focus shifts to building self-correction abilities through repeated inquiry rather than external corrections.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same constrained-question design could be tested in other skill areas such as scientific reasoning or problem solving to encourage active processing.
Educational AI tools might adopt gating mechanisms more broadly to balance immediate help with long-term learner independence.
Developers could explore combining this approach with classroom teacher oversight to create layered feedback systems.
Controlled trials measuring changes in students' ability to construct arguments without AI prompts would directly test the intended benefit.

Load-bearing premise

The assumption that prompt and schema constraints will reliably force the LLM to produce only high-quality, pedagogically effective inquiry questions without drifting into direct revisions or unhelpful output.

What would settle it

A comparison study in which students using Prober.ai show no greater gains in independent argumentative reasoning or essay quality than students using standard AI tools that generate or rewrite text.

Figures

Figures reproduced from arXiv: 2605.05598 by Ran Bi, Shiyao Wei, Yuanyiyi Zhou.

**Figure 1.** Figure 1: Conceptual processing pipeline of Prober.ai. The LLM performs argument parsing, feature detection, epistemic state classification, trigger prioritization, and question module selection as internal reasoning steps. Only the final inquiry-based questions are surfaced to the student. 3.4 Persona System Two complementary personas are implemented, each addressing distinct dimensions of argumentative quality: … view at source ↗

**Figure 2.** Figure 2: The Write–Challenge–Defend–Improve cycle. The loop is designed so that cognitive view at source ↗

read the original abstract

The proliferation of large language models (LLMs) in educational settings has paradoxically undermined the cognitive processes they purport to support. Students increasingly outsource critical thinking to AI assistants that generate polished text on demand, resulting in measurable cognitive debt and diminished argumentative reasoning skills. We present Prober.ai, a web-based writing environment that inverts the conventional AI-tutoring paradigm: rather than generating or rewriting student text, the system constrains an LLM (Gemini 3 Flash Preview) through persona-specific system prompts and structured JSON output schemas to produce only targeted, inquiry-based questions about argumentative weaknesses. A two-phase interaction architecture -- Challenge and Unlock -- implements a pedagogical friction mechanism whereby revision suggestions are gated behind mandatory student reflection. The system's design is grounded in Toulmin's argumentation theory, research on peer feedforward questioning mechanisms, and evidence on AI-supported feedback in writing instruction. A functional prototype was developed in 36 hours during the NY EdTech Hackathon (March 2026), where it was awarded second place. We describe the system architecture, the prompt engineering methodology for constraining LLM output to pedagogically aligned JSON schemas, and discuss implications for scalable, cognition-preserving AI integration in writing education.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

This is a clear description of a hackathon prototype that constrains LLMs to ask Toulmin-style questions and gate revisions behind reflection, but it stops at the design with zero evaluation data.

read the letter

The paper's core contribution is the architecture of Prober.ai: an LLM (Gemini) wrapped in persona prompts and JSON schemas so it only outputs targeted questions about argumentative structure instead of rewriting text. The two-phase Challenge-Unlock flow adds friction by requiring student responses before any suggestions appear. This setup is explained plainly and draws directly from Toulmin's model plus peer feedforward research, which gives the design some grounding without overclaiming originality.

Referee Report

0 major / 2 minor

Summary. The paper presents Prober.ai, a web-based prototype for argumentative writing development that inverts typical AI tutoring by constraining an LLM (Gemini 3 Flash Preview) via persona-specific system prompts and structured JSON output schemas to generate only targeted, inquiry-based questions about argumentative weaknesses, grounded in Toulmin's theory. It implements a two-phase Challenge/Unlock gated interaction that requires student reflection before any revision suggestions are provided. The system was built in 36 hours at the NY EdTech Hackathon and awarded second place; the manuscript describes the architecture, prompt-engineering methodology, and discusses implications for cognition-preserving AI use in education.

Significance. The described architecture offers a concrete, replicable example of using prompt constraints and output schemas to limit LLM behavior to pedagogically aligned question generation rather than text production. If the approach scales and proves reliable, it could contribute to tools that mitigate cognitive debt in AI-assisted writing by enforcing active reflection, drawing appropriately on Toulmin's model and peer feedforward research. The hackathon implementation demonstrates rapid feasibility of the design.

minor comments (2)

Abstract: the model reference 'Gemini 3 Flash Preview' should be clarified or corrected for accuracy, as it is not a standard current version name.
The description of the prompt engineering methodology would be strengthened by including at least one concrete example of a persona prompt and corresponding JSON schema to support replication and evaluation of the constraint mechanism.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for their positive summary of the manuscript, recognition of the system's architecture and pedagogical grounding, and recommendation for minor revision. We appreciate the acknowledgment of the rapid prototype development and its potential to address cognitive debt in AI-assisted writing. No major comments were provided in the report, so we have no specific points to address point-by-point at this stage. We will incorporate any minor editorial or clarification changes suggested during the revision process.

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper is a purely descriptive account of a 36-hour hackathon prototype that presents an LLM-constrained inquiry system grounded in Toulmin's argumentation theory and existing peer-feedforward literature. It contains no equations, no fitted parameters, no quantitative predictions, no uniqueness theorems, and no self-citations that bear load on any central claim. All design decisions are explicitly attributed to external pedagogical sources and standard prompt-engineering techniques rather than to any self-referential construction, making the contribution self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

As a system design and prototype description paper with no mathematical modeling or data fitting, there are no free parameters, axioms, or invented entities to ledger.

pith-pipeline@v0.9.0 · 5517 in / 1272 out tokens · 79723 ms · 2026-05-08T11:50:21.966585+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

7 extracted references · 4 canonical work pages

[1]

K., & Ga s evi\' c , D

Ba, S., Yang, L., Yan, Z., Looi, C. K., & Ga s evi\' c , D. (2025). Unraveling the mechanisms and effectiveness of AI -assisted feedback in education: A systematic literature review. Computers and Education Open, 9, 100284. https://doi.org/10.1016/j.caeo.2025.100284

work page doi:10.1016/j.caeo.2025.100284 2025
[2]

Bi, R., Yan, J. (2026). Pedagogy vs. preference: A nalyzing the alignment gap in student- LLM interactions in the wild. Manuscript in preparation

2026
[3]

Gao, X., Noroozi, O., Gulikers, J., Biemans, H. J. A., & Banihashem, S. K. (2024). Students' online peer feedback uptake in argumentative essay writing. Proceedings of the International Society of the Learning Sciences. https://repository.isls.org/handle/1/10608

2024
[4]

J., Driessen, E

Kinnear, B., Schumacher, D. J., Driessen, E. W., & Varpio, L. (2022). How argumentation theory can inform assessment validity: A critical review. Medical Education, 56(11), 1064--1075

2022
[5]

Your brain on ChatGPT: Accumulation of cognitive debt when using an AI assistant for essay writing task

Kosmyna, N., Hauptmann, E., Yuan, Y. T., Situ, J., Liao, X. H., Beresnitzky, A. V., & Maes, P. (2025). Your brain on ChatGPT : A ccumulation of cognitive debt when using an AI assistant for essay writing task. arXiv preprint arXiv:2506.08872, 4

work page arXiv 2025
[6]

Latifi, S., Noroozi, O., & Talaee, E. (2021). Peer feedback or peer feedforward? E nhancing students' argumentative peer learning processes and outcomes. British Journal of Educational Technology, 52(2), 768--784. https://doi.org/10.1111/bjet.13054

work page doi:10.1111/bjet.13054 2021
[7]

Noroozi, O., Biemans, H., & Mulder, M. (2016). Relations between scripted online peer feedback processes and quality of written argumentative essay. The Internet and Higher Education, 31, 20--31. https://doi.org/10.1016/j.iheduc.2016.05.002

work page doi:10.1016/j.iheduc.2016.05.002 2016