REVIEW 4 major objections 4 minor 21 references
AgentSME for Simulating Diverse Communication Modes in Smart Education
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read AgentSME claims that letting two LLM-backed student agents exchange their chain-of-thought reasoning before answering yields the highest accuracy across three simulated communication modes, with gains largest on the hardest questions.
desk verdict The Echo accuracy gain is consistent but currently confounded with extra sampling and cross-model ensembling; worth a rigorous revision, not acceptance as-is. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the Echo communication loop implemented in Algorithm 3 (AgentEcho): two agents are randomly paired, each first answers the question alone with chain-of-thought reasoning, the two reasoning texts are swapped, and each agent then re-answers with the partner's reasoning appended to its prompt. It is this one bidirectional exchange—not any change in the underlying LLM, prompt, or question format—that the paper credits for the accuracy gains. The other two algorithms act as contrasts: AgentSolo isolates native ability, and AgentMono injects a mentor's reasoning text in a one-way direction. Context is reset after every question so that performance differences are attributed to within-question exploration and peer interaction. The diversity metrics (Inverse Simpson, Honoré's statistic, information entropy) are secondary machinery used to characterize output style rather than correctness.
What would settle it
Run the same difficulty-stratified CMMLU society questions with a two-attempt Solo baseline in which each agent answers twice independently and keeps its better or majority answer, with no peer text. If that baseline matches Echo's HIGH-difficulty accuracy band of 0.284–0.587, the claimed communication advantage would be explained by added inference rather than by exchange.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that the directionality of inter-agent communication is a first-order determinant of LLM agent performance, not a cosmetic design detail. Across all six models and all three difficulty strata, the Echo mode—one round in which two agents reason independently, exchange their chain-of-thought text, and then produce revised answers—outperforms both Solo and Mono, giving the overall ranking AgentEcho > AgentMono ≈ AgentSolo. The advantage grows with task difficulty: for the 20 HIGH-difficulty questions, mean accuracy rises from a 0.035–0.120 range under Solo to a 0.284–0.587 range under Echo (gains of 0.249–0.467, with $p<0.01$), while Mono improves accuracy by at most 0.001 on the same set. The paper also reports that lexical diversity, measured by Inverse Simpson, Honoré's statistic, and information entropy, does not track accuracy linearly, and that DeepSeek-Reasoner shows the most expansive diversity profile once two-way interaction is allowed.
Load-bearing premise
The load-bearing premise is that Echo's accuracy gain comes from the two-way exchange itself, not merely from the extra reasoning pass it gives each agent, since Echo gives twice as many reasoning opportunities as Solo and no control equalizes compute or sampling.
Editorial extensions
If this is right
- If the claim holds, LLM tutoring systems should use peer dialogue for hard questions, since Mono's hard-question gain is at most 0.001 while Echo raises accuracy by 0.249–0.467.
- Echo's benefit grows with difficulty, so collaboration budgets should concentrate on questions where individual models are weakest.
- Echo also increases round-to-round standard deviation (up to 0.187 on hard questions), so deployments needing stable scores should pair it with repeated sampling or voting.
- Accuracy and lexical diversity are decoupled: DeepSeek-Reasoner is accurate and diverse under Echo, while GPT-4o is accurate with stable, less diverse output.
- Base-capacity models gain the most (Qwen-Turbo-Latest rises from 0.050 to 0.521 on HIGH), suggesting bidirectional exchange partly compensates for weaker individual reasoning.
Reading between the lines
- The paper does not include a compute-matched control: Echo gives each agent two reasoning passes whereas Solo gives one, so until a two-attempt Solo or self-consistency baseline is run, part of the reported gain could come from extra inference rather than from communication per se.
- A testable extension is to vary the number of exchange rounds; if one round of Echo is near the ceiling, extra rounds should yield diminishing returns, while a multi-round discussion would be justified if gains continue.
- Transferred to human learning, the pattern suggests encouraging reciprocal explanation rather than one-way instruction on difficult material, although the paper only demonstrates this for LLM agents.
- The difficulty labels are derived from the same six models' error rates, so 'hard' means hard for these models; relabeling with other models or human test-takers could shift where Echo's advantage peaks.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes AgentSME, an LLM-agent framework for smart education that contrasts three communication modes: Solo (independent answering), Mono (one-directional mentoring), and Echo (bidirectional peer exchange). It evaluates six LLMs on the society subset of CMMLU and reports accuracy plus three lexical-diversity metrics. The paper's central claim is that Echo consistently yields the highest accuracy, especially on high-difficulty questions, while DeepSeek-Reasoner shows the greatest lexical diversity in Echo mode. The main positive result is that one round of exchanged chain-of-thought reasoning improves accuracy across all six models relative to Solo and Mono, with reported p-values below 0.01.
Significance. If the causal attribution to communication mode were supported, the result would be a concrete and useful demonstration that a single bidirectional chain-of-thought exchange improves LLM answer accuracy across model tiers, with practical implications for multi-agent smart-education simulations. The paper's strengths are the breadth of models tested (six, split into base- and high-capacity groups), the consistent directional improvement across all six models in Table I, and the complementary diversity metrics. The contribution is weakened by the absence of compute-matched controls, by difficulty labels derived from the same models that are later evaluated, and by pseudocode ambiguities; the current evidence supports a correlation between the Echo protocol and higher accuracy rather than a clean causal claim. No code or prompts are provided, so external reproduction is not currently possible.
major comments (4)
- [II-A, Algorithm 3, Table I] The sentence in Section II-A that context reset ensures 'any performance variation can be fully attributed to the agent's exploration and peer interactions' is not supported by the experimental design. AgentEcho gives each agent two generation opportunities (initial reasoning plus a revision), while AgentSolo gives one; per pair, it also pools two different LLMs. The Mono condition adds a mentor's reasoning but does not double the learner's sampling budget. The HIGH-difficulty gains of 0.249-0.467 in Table I are therefore also compatible with a pure extra-sampling or ensemble-complementarity effect. To support the attribution to bidirectional communication, the authors should add a compute-matched control, for example AgentSolo with two independent CoT samples combined by self-consistency, and a version of Echo in which both agents are the same LLM.
- [III-A, Table I] The difficulty stratification is circular for the main claim. Section III-A computes difficulty thresholds (20%, 50%, 80% average error) from five rounds using the same six LLMs that are later evaluated, and the filtering keeps only 85 of the 264 CMMLU society questions (20 HIGH + 21 MEDIUM + 44 LOW). Because the same models define 'high difficulty,' the statement that Echo helps most on high-difficulty questions is partially built into the data construction rather than being an independent property of the benchmark. I recommend deriving difficulty from a held-out model set or from an external norm, and reporting the 179 excluded questions separately.
- [Algorithm 3, lines 6-9; Algorithm 2, line 6] The pseudocode has prompt-assignment errors that make the reported experiments non-reproducible as written. In Algorithm 3, line 6 generates RC_i from agent a_i using ASK_j and line 7 generates RC_j from agent a_j using ASK_i, so the 'initial independent reasoning' is actually produced with the other agent's prompt; the update in lines 8-9 then appears to exchange reasoning in the opposite direction. Algorithm 2 similarly builds ASK_i for the learner but passes it to the mentor in line 6 instead of ASK_m. The authors should correct these lines and clarify which prompt each agent receives at each stage.
- [III-B, Table I] The statistical support for the headline claim is under-specified. Table I lists only p-values, all equal to 0.002, with no statement of the test used (e.g., paired Wilcoxon signed-rank), no multiple-comparison correction across the 18 model-by-difficulty comparisons, and no effect-size or confidence-interval reporting. With R=10, a discrete accuracy grid, and post-hoc filtering, the identical p-values are suspicious; please report the test statistic, correction method, and per-comparison intervals.
minor comments (4)
- [III-C] The text says 'Echo (Fig. 1(a))' but the Echo panel is Fig. 1(c); please correct the cross-reference.
- [III-B, Table I] The '/' entries in the P-value row for Echo columns are unexplained; clarify which pairwise comparison each p-value refers to.
- [Abstract, III-C] The abstract says 'DeepSeek exhibits the greatest diversity,' but Section III-C specifically attributes the largest Echo-mode diversity to DeepSeek-Reasoner; please clarify whether the abstract refers only to that model or to both DeepSeek variants.
- [Nomenclature and Section II-B] The number of agents M=6 is the number of LLM identities, but in Echo mode each round generates 15 unordered pairs; the effective number of responses per round differs from Solo/Mono and should be stated explicitly.
Circularity Check
Difficulty labels are constructed from the evaluated models' own Solo error rates, making the difficulty-dependence of Echo's gains partly definitional; the central Echo-vs-Solo comparison remains independent.
-
self definitional
[Section III-A (Experiment Configuration) and Table I]
"To obtain the difficulty distribution of the questions and establish a baseline, this paper first conducts five independent reasoning rounds for each of the six models. ... The questions in CMMLU are then categorized into three difficulty levels (HIGH, MEDIUM, LOW) based on the average error rate in the five rounds. The error rate is calculated as one minus the accuracy. ... HIGH-difficulty questions: In five rounds of independent responses, the average error rate of LLM exceeded 80%."
The difficulty labels are defined by the same models' solo error rates (error = 1 - Solo accuracy). Therefore 'HIGH difficulty' is, by construction, the subset where AgentSolo accuracy is low, and Table I's report that AgentSolo accuracy on HIGH is only 0.035-0.120 is a restatement of the selection criterion rather than an independent finding. The paper then uses this same internally constructed split to claim that Echo's accuracy advantage increases with task difficulty. Because the split guarantees low Solo accuracy (and hence large headroom) on HIGH-difficulty items, the difficulty-dependence of Echo's gain is partly built into the dataset construction instead of being an external property of the benchmark.
full rationale
The central result (AgentEcho > AgentSolo and AgentMono on CMMLU accuracy) is an empirical comparison computed from independent response logs; it is not obtained by fitting a parameter to the outcome or by deriving Echo's accuracy from the difficulty definitions. No load-bearing self-citation is present: the three references authored or co-authored by T.-F. Zhao ([16], [19], [20]) are background citations on information cocoons and evolutionary algorithms and are not used to justify AgentSME's claims. There is no imported uniqueness theorem and no ansatz smuggled in via citation. The one partial circularity is the difficulty stratification: HIGH/MEDIUM/LOW are defined from the average error rate of the very six models later evaluated, and AgentSolo's low accuracy on HIGH-difficulty items is therefore a restatement of the selection rule rather than an independent finding. Because this split also determines which questions are called 'high difficulty,' the claim that Echo's advantage grows with difficulty is partly built into the dataset construction (floor/headroom effect). The Echo-vs-Solo accuracy magnitudes and the diversity comparisons remain independent, so the circularity is minor and non-load-bearing. Separately, but not as circularity: AgentEcho uses two responses per agent pair versus one in Solo, so the accuracy gain is confounded with extra sampling/compute, and Algorithm 3's swapped prompt lines (RC_i from ASK_j, RC_j from ASK_i) are a reproducibility ambiguity.
Assumptions & free parameters
free parameters (3)
- Difficulty thresholds =
error rates 20%, 50%, 80%
- Sampling hyperparameters =
temperature = 0.7, top-p = 0.95
- Echo exchange rounds =
1
assumptions (4)
- domain assumption Accuracy on single-choice CMMLU questions is a valid proxy for agent learning capability in smart education.
- domain assumption Lexical diversity of chain-of-thought text, measured by Inverse Simpson, Honoré's statistic, and information entropy, captures cognitive or expressive diversity.
- domain assumption Resetting the context after every question ensures that performance differences are fully attributable to the communication mode.
- domain assumption Random pairing of agents powered by different LLMs does not bias the Echo comparison.
Cite this review
Pith. "Pith review of AgentSME for Simulating Diverse Communication Modes in Smart Education." pith.science (2026). https://pith.science/paper/GM6G4WJ2
@misc{pith2026250803109,
author = {Pith},
title = {Pith review of: AgentSME for Simulating Diverse Communication Modes in Smart Education},
year = {2026},
howpublished = {\url{https://pith.science/paper/GM6G4WJ2}},
note = {Machine review of arXiv:2508.03109}
}
read the original abstract
Generative agent models specifically tailored for smart education are critical, yet remain relatively underdeveloped. A key challenge stems from the inherent complexity of educational contexts: learners are human beings with various cognitive behaviors, and pedagogy is fundamentally centered on personalized human-to-human communication. To address this issue, this paper proposes AgentSME, a unified generative agent framework powered by LLM. Three directional communication modes are considered in the models, namely Solo, Mono, and Echo, reflecting different types of agency autonomy and communicative reciprocity. Accuracy is adopted as the primary evaluation metric, complemented by three diversity indices designed to assess the diversity of reasoning contents. Six widely used LLMs are tested to validate the robustness of communication modes across different model tiers, which are equally divided into base-capacity and high-capacity configurations. The results show that generative agents that employ the Echo communication mode achieve the highest accuracy scores, while DeepSeek exhibits the greatest diversity. This study provides valuable information to improve agent learning capabilities and inspire smart education models.
Figures
Reference graph
Works this paper leans on
-
[1]
Roger Nkambou, Riichiro Mizoguchi, and Jacqueline Bour- deau.Advances in intelligent tutoring systems, volume 308. Springer, 2010
work page 2010
-
[2]
Au- totutor: A simulation of a human tutor.Cognitive Systems Research, 1(1):35–51, 1999
Arthur C Graesser, Katja Wiemer-Hastings, Peter Wiemer- Hastings, Roger Kreuz, Tutoring Research Group, et al. Au- totutor: A simulation of a human tutor.Cognitive Systems Research, 1(1):35–51, 1999
work page 1999
-
[3]
Ghader Kurdi, Jared Leo, Bijan Parsia, Uli Sattler, and Salam Al-Emari. A systematic review of automatic question gen- eration for educational purposes.International Journal of Artificial Intelligence in Education, 30:121–204, 2020
work page 2020
-
[4]
Yaganteeswarudu Akkem, Saroj Kumar Biswas, and Aruna Varanasi. A comprehensive review of synthetic data gener- ation in smart farming by using variational autoencoder and generative adversarial network.Engineering Applications of Artificial Intelligence, 131:107881, 2024
work page 2024
-
[5]
Yuheng Cheng, Ceyao Zhang, Zhengwen Zhang, Xiangrui Meng, Sirui Hong, Wenhao Li, Zihao Wang, Zekai Wang, Feng Yin, Junhua Zhao, et al. Exploring large language model based intelligent agents: Definitions, methods, and prospects.arXiv preprint arXiv:2401.03428, 2024
arXiv 2024
-
[6]
Personalized tutoring through conversational agents
Dejian Liu, Ronghuai Huang, Ying Chen, Michael Agyemang Adarkwah, Xiangling Zhang, Xin Li, Junjie Zhang, and Ting Da. Personalized tutoring through conversational agents. In Using Educational Robots to Enhance Learning: An Analysis of 100 Academic Articles, pages 59–85. Springer, 2024
work page 2024
-
[7]
Benjamin D Nye, Dillon Mee, and Mark G Core. Generative large language models for dialog-based tutoring: An early consideration of opportunities and concerns. InLLM@ AIED, pages 78–88, 2023
work page 2023
-
[8]
Salman Zulfiqar, Rongting Zhou, Fahad Asmi, and Affan Yasin. Using simulation system for collaborative learn- ing to enhance learner’s performance.Cogent Education, 5(1):1424678, 2018
work page 2018
Show all 21 references
-
[9]
Eduagent: Generative student agents in learning.arXiv preprint arXiv:2404.07963, 2024
Songlin Xu, Xinyu Zhang, and Lianhui Qin. Eduagent: Generative student agents in learning.arXiv preprint arXiv:2404.07963, 2024
2024 arXiv
-
[10]
Agent4edu: Generating learner response data by generative agents for intelligent education systems.arXiv preprint arXiv:2501.10332, 2025
Weibo Gao, Qi Liu, Linan Yue, Fangzhou Yao, Rui Lv, Zheng Zhang, Hao Wang, and Zhenya Huang. Agent4edu: Generating learner response data by generative agents for intelligent education systems.arXiv preprint arXiv:2501.10332, 2025
2025 arXiv
-
[11]
Generative agents: Interactive simulacra of human behavior
Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Mered- ith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. InProceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22, 2023
2023
-
[12]
A commentary of gpt-3 in mit technology review 2021.Fundamental Research, 1(6):831– 833, 2021
Min Zhang and Juntao Li. A commentary of gpt-3 in mit technology review 2021.Fundamental Research, 1(6):831– 833, 2021
2021
-
[13]
Chatgpt for good? on opportunities and challenges of large language models for education.Learning and individual differences, 103:102274, 2023
Enkelejda Kasneci, Kathrin Seßler, Stefan K ¨uchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan G ¨unnemann, Eyke H ¨ullermeier, et al. Chatgpt for good? on opportunities and challenges of large language models for education.Learning and...
2023
-
[14]
Classmeta: Designing interactive virtual classmate to promote vr classroom partici- pation
Ziyi Liu, Zhengzhe Zhu, Lijun Zhu, Enze Jiang, Xiyun Hu, Kylie A Peppler, and Karthik Ramani. Classmeta: Designing interactive virtual classmate to promote vr classroom partici- pation. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pages 1–17, 2024
2024
-
[15]
The icap framework: Linking cognitive engagement to active learning outcomes
Michelene TH Chi and Ruth Wylie. The icap framework: Linking cognitive engagement to active learning outcomes. Educational psychologist, 49(4):219–243, 2014
2014
-
[16]
Modeling information cocoons in networked populations: Insights from backgrounds and preferences.IEEE Transactions on Computational Social Systems, 11(3):4497– 4510, 2024
Ming Gu, Tian-Fang Zhao, Liang Yang, Xiao-Kun Wu, and Wei-Neng Chen. Modeling information cocoons in networked populations: Insights from backgrounds and preferences.IEEE Transactions on Computational Social Systems, 11(3):4497– 4510, 2024
2024
-
[17]
Unleashing the po- tential of chatbots in education: A state-of-the-art analysis
Rainer Winkler and Matthias S ¨ollner. Unleashing the po- tential of chatbots in education: A state-of-the-art analysis. InAcademy of management proceedings, volume 2018, page 15903. Academy of Management Briarcliff Manor, NY 10510, 2018
2018
-
[18]
Cognitive archi- tectures: Research issues and challenges.Cognitive Systems Research, 10(2):141–160, 2009
Pat Langley, John E Laird, and Seth Rogers. Cognitive archi- tectures: Research issues and challenges.Cognitive Systems Research, 10(2):141–160, 2009
2009
-
[19]
Evolutionary divide-and-conquer algorithm for virus spreading control over networks.IEEE Transactions on Cybernetics, 51(7):3752– 3766, 2021
Tian-Fang Zhao, Wei-Neng Chen, Sam Kwong, Tian-Long Gu, Hua-Qiang Yuan, Jie Zhang, and Jun Zhang. Evolutionary divide-and-conquer algorithm for virus spreading control over networks.IEEE Transactions on Cybernetics, 51(7):3752– 3766, 2021
2021
-
[20]
Ming Gu, Tian-Fang Zhao, Jinghui Zhong, and Wei-Neng Chen. Progressive community merging cooperative coevolu- tion algorithm for influence blocking maximization in social networks.IEEE Transactions on Network Science and Engi- neering, 12(3):2093–2106, 2025
2025
-
[21]
Cmmlu: Mea- suring massive multitask language understanding in chinese
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Mea- suring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212, 2023
2023 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.