Pith. sign in

REVIEW 89 references

CAFES: A Collaborative Multi-Agent Framework for Multi-Granular Multimodal Essay Scoring

T0 review · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A student-teacher multi-agent pipeline with positive-only feedback improves QWK agreement with human essay scores by 21% on a multimodal benchmark, with gains concentrated in traits where baselines were weakest.

arxiv 2505.13965 v1 pith:XYRB5OM5 submitted 2025-05-20 cs.CL cs.AI

Jiamin Su , Yibo Yan , Zhuoran Gao , Han Zhang , Xiang Liu , Xuming Hu This is my paper · ORCID
classification cs.CLcs.AI
keywords multimodalcafesframeworkcollaborativeessayfeedbackhumanmulti-agent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Automated essay scoring usually means one model reads an essay and outputs a grade. On multimodal essays, where students interpret charts or diagrams, single models tend to be harsh: they focus on surface errors and under-recognize strengths, so their scores disagree with human raters. CAFES splits the job across three roles. A student model first gives scores for ten traits such as grammar, coherence, and vocabulary. A teacher model then reads the same essay and writes positive comments that highlight what the essay does well in each trait. Finally, the teacher model acts as a reflective scorer, revising the student scores in light of that positive feedback. The paper tests this on EssayJudge, a dataset of 1,054 essays with images. Across eleven student models, from small open-source ones to GPT-4o-mini, the pipeline raises Quadratic Weighted Kappa from 0.29 to 0.36 on average. The largest gains appear in grammatical diversity and lexical diversity, traits where single models were most conservative. Ablations show that removing the feedback pool makes reflection hurt, and that using one model for both roles also hurts. The authors argue this mirrors a human cycle of scoring, feedback, and revision.
Extended reading notes

Core claim

CAFES improves average Quadratic Weighted Kappa against human ground truth from 0.29 to 0.36 (21% relative) across eleven MLLMs on the EssayJudge dataset, with the gains driven by positive-only trait feedback and teacher-student role separation. The paper states: 'Our proposed CAFES framework yields consistent and significant improvements of QWK across each student MLLM on most traits.'

Load-bearing premise

The dataset EssayJudge, used for both development and evaluation, is the only multimodal AES benchmark and was introduced by the same research group (Su et al. 2025, including several CAFES co-authors). The paper also assumes that GPT-4o is a reliable 'teacher' oracle whose positive feedback and reflective revisions move scores toward human judgment, with no independent validation of the teacher's own agreement with human raters on this dataset. Section 4.1 states GPT-4o is the default teacher 'given its strong performance in AES', citing the same group's earlier work. If the teacher model is itself biased (e.g., systematically generous), the observed QWK improvements could be an artifact of shifting student scores toward the teacher's bias rather than toward human judgment. Additionally, no inter-rater reliability of the ground-truth human scores is reported; if human labels are noisy, QWK improvements against that reference are hard to interpret.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The framework does not introduce new physical or mathematical entities; its free parameters are design choices of the prompt pipeline. The two largest hidden assumptions are the reliability of EssayJudge labels (from the authors' own benchmark) and the teacher model GPT-4o's alignment with human judgment. The positive-only feedback constraint is a hand-chosen design that the abalation only tests by removal, not by replacement.

free parameters (4)
  • Number of reflection rounds = 1
    The paper uses a single round of feedback and reflection. No search over this hyperparameter is reported, though the framework allows iterations. This choice affects QWK improvements.
  • Score range zero-to-five = 0 to 5
    The rubric maps essays to integer scores 0-5 for each trait. This discrete grid limits possible QWK values and is a design choice inherited from EssayJudge.
  • Teacher model GPT-4o = GPT-4o
    GPT-4o is chosen as the default teacher across all experiments, justified by prior AES performance from the same group. The teacher is not varied in the main experiments, so results may be specific to this model.
  • Positive-only feedback constraint = only strengths
    The Feedback Pool Manager is instructed to extract only positive feedback. This design choice drastically affects the reflective revision, and the paper motivates it with the claim that MLLMs are overly strict, but no comparison with neutral or negative feedback is made (except removing feedback entirely).
assumptions (4)
  • domain assumption QWK is an appropriate metric for human-model alignment on ordinal trait scores.
    The paper selects QWK after reviewing previous AES studies, but does not discuss its limitations, e.g., its sensitivity to marginal distributions and its inability to capture systematic bias, which matters because the paper's mechanism explicitly changes the marginal score distribution.
  • domain assumption The EssayJudge human scores are reliable ground truth.
    No inter-annotator agreement, adjudication process, or human-rater variance is reported for the 1,054 essays. Section 4.1 introduces the dataset but does not provide evidence of label quality.
  • domain assumption LLM responses can be reliably parsed into the required JSON score object.
    The Initial and Reflective scorers are instructed to return strict JSON. If parsing failures or malformed outputs occur, they are not discussed. No handling of refusal or invalid scores is described.
  • domain assumption The ten trait rubrics are mutually exclusive and jointly capture writing quality.
    The paper treats the ten traits as independent dimensions, but traits like coherence and organizational structure overlap conceptually. The reflective scorer may revise one trait based on feedback about another, which could inflate agreement in a way that is not decomposable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CAFES: A Collaborative Multi-Agent Framework for Multi-Granular Multimodal Essay Scoring." pith.science (2026). https://pith.science/paper/XYRB5OM5

@misc{pith2026250513965,
  author       = {Pith},
  title        = {Pith review of: CAFES: A Collaborative Multi-Agent Framework for Multi-Granular Multimodal Essay Scoring},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XYRB5OM5}},
  note         = {Machine review of arXiv:2505.13965}
}
read the original abstract

Automated Essay Scoring (AES) is crucial for modern education, particularly with the increasing prevalence of multimodal assessments. However, traditional AES methods struggle with evaluation generalizability and multimodal perception, while even recent Multimodal Large Language Model (MLLM)-based approaches can produce hallucinated justifications and scores misaligned with human judgment. To address the limitations, we introduce CAFES, the first collaborative multi-agent framework specifically designed for AES. It orchestrates three specialized agents: an Initial Scorer for rapid, trait-specific evaluations; a Feedback Pool Manager to aggregate detailed, evidence-grounded strengths; and a Reflective Scorer that iteratively refines scores based on this feedback to enhance human alignment. Extensive experiments, using state-of-the-art MLLMs, achieve an average relative improvement of 21% in Quadratic Weighted Kappa (QWK) against ground truth, especially for grammatical and lexical diversity. Our proposed CAFES framework paves the way for an intelligent multimodal AES system. The code will be available upon acceptance.

Figures

Figures reproduced from arXiv: 2505.13965 by the authors.

Figure 1
Figure 1. Comparisons among the traditional AES method (a), MLLM-based method (b), and our proposed multi-agent CAFES framework (c) on AES task. et al., 2025; Su et al., 2025; Li et al., 2024). Traditional AES methods suffer from several critical limitations, as shown in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The framework of our proposed CAFES. The system follows a three-stage process: ❶ Initial scoring via the student MLLM; ❷ Feedback generation for each trait via the teacher MLLM; and ❸ Final reflective scoring with justification-based revision via the teacher MLLM. methods (e.g., Logistic Regression, SVMs) rely on handcrafted features (Chen and He, 2013; Yan￾nakoudakis and Briscoe, 2012), including length￾based attri… view at source ↗
Figure 3
Figure 3. Reflective scorer’s JSON output format. This reflective revision mechanism ensures that the final assessment fairly incorporates the strengths recognized in the essay, while avoiding unnecessary or overly aggressive adjustments. 4 Experiments and Analysis 4.1 Experimental Setup Statistic Number Total Multimodal Essays 1,054 Image Type - Single-Image 703 (66.7%) - Multi-Image 351 (33.3%) Multimodal Essay Type - Flow … view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Trait-level score improvements after reflection [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Improvements of QWK score across all traits based on different student MLLMs. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Improvements of average QWK score across [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 9
Figure 9. Figure 9: QWK changes w/ and w/o Feedback Pool for [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 10
Figure 10. Figure 10: QWK scores across ten traits for two MLLMs (GPT-4o and LLaMA-3.2-Vision-90B), w/ and w/o the teacher-student collaboration mechanism. 4.5 Ablation Study We conduct two ablation studies to test key compo￾nents of CAFES. The first removes the Feedback [PITH_FULL_IMAGE:…
Figure 11
Figure 11. Figure 11: A representative case study illustrating [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]
Figure 12
Figure 12. Figure 12: Prompt for Initial Scorer [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Prompt for Feedback Pool Manager. Task Definition: You are evaluating a set of essay scores originally provided by another assistant reviewer. A detailed feedback report—including both positive and negative comments across 10 traits—is available for reference, but sho…
Figure 14
Figure 14. Figure 14: Prompt for Reflective Scorer [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]
Figure 15
Figure 15. Figure 15: Average trait-specific scores assigned by closed-source and open-source MLLMs at both the initial stage [PITH_FULL_IMAGE:figures/full_fig_p021_15.png]
Figure 16
Figure 16. Figure 16: Score distributions of closed-source and open-source MLLMs at both the initial scoring stage and after [PITH_FULL_IMAGE:figures/full_fig_p021_16.png]
Figure 17
Figure 17. Figure 17: Improvements of QWK score across all traits based on different student MLLM. [PITH_FULL_IMAGE:figures/full_fig_p022_17.png]
Figure 18
Figure 18. Figure 18: A case study illustrating CAFES ’s score revision process. And the student MLLM is Claude-3.5-Sonnet, and the teacher MLLM is GPT-4o [PITH_FULL_IMAGE:figures/full_fig_p022_18.png]
Figure 19
Figure 19. Figure 19: A case study illustrating CAFES ’s score revision process. And the student MLLM is GPT-4o-mini, and the teacher MLLM is GPT-4o [PITH_FULL_IMAGE:figures/full_fig_p022_19.png]
Figure 20
Figure 20. Figure 20: A case study illustrating CAFES ’s score revision process. And the student MLLM is Qwen2.5-VL-32B, and the teacher MLLM is GPT-4o [PITH_FULL_IMAGE:figures/full_fig_p023_20.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

89 extracted references · 34 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Anthropic. 2024. https://www.anthropic.com/news/claude-3-5-sonnet Claude 3.5 sonnet

    2024

  4. [4]

    John Atkinson and Diego Palma. 2025. An llm-based hybrid approach for enhanced automated essay scoring. Scientific Reports, 15(1):14551

    2025

  5. [5]

    Yigal Attali and Jill Burstein. 2006. Automated essay scoring with e-rater® v.2. journal of technology, learning, and assessment, 4(3). Journal of Technology, Learning, and Assessment, 4

    2006

  6. [6]

    Stacey Bailey and Detmar Meurers. 2008. Diagnosing meaning errors in short answers to reading comprehension questions. In Proceedings of the Third Workshop on Innovative Use of NLP for Building Educational Applications , pages 107--115

    2008

  7. [7]

    Yida Cai, Kun Liang, Sanwoo Lee, Qinghan Wang, and Yunfang Wu. 2025. Rank-then-score: Enhancing large language models for automated essay scoring. arXiv preprint arXiv:2504.05736

    arXiv 2025

  8. [8]

    Yue Cao, Hanqi Jin, Xiaojun Wan, and Zhiwei Yu. 2020. Domain-adaptive neural automated essay scoring. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '20, page 1011–1020. Association for Computing Machinery

    2020

Show all 89 references
  1. [9]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, and 1 others. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1--45

    2024

  2. [10]

    Hongbo Chen and Ben He. 2013. Automated essay scoring by maximizing human-machine agreement. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1741--1752

    2013

  3. [11]

    Junkai Chen, Zhijie Deng, Kening Zheng, Yibo Yan, Shuliang Liu, PeiJun Wu, Peijie Jiang, Jia Liu, and Xuming Hu. 2025 a . Safeeraser: Enhancing safety in multimodal large language models through multimodal machine unlearning. arXiv preprint arXiv:2502.12520

    2025

  4. [13]

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, and 23 others. 2025 c . https://arxiv.org/abs/2412....

    2025 arXiv

  5. [14]

    Jaeyoon Choi, Tamara Tate, Daniel Ritchie, Nia Nixon, and Mark Warschauer. 2025. Anchor is the key: Toward accessible automated essay scoring with large language models through prompting

    2025

  6. [15]

    Zhendong Chu, Shen Wang, Jian Xie, Tinghui Zhu, Yibo Yan, Jinheng Ye, Aoxiao Zhong, Xuming Hu, Jing Liang, Philip S Yu, and 1 others. 2025. Llm agents for education: Advances and applications. arXiv preprint arXiv:2503.11733

    2025

  7. [16]

    M a d a lina Cozma, Andrei Butnaru, and Radu Tudor Ionescu. 2018 a . Automated essay scoring with string kernels and word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 503--509

    2018

  8. [17]

    M a d a lina Cozma, Andrei Butnaru, and Radu Tudor Ionescu. 2018 b . https://doi.org/10.18653/v1/P18-2080 Automated essay scoring with string kernels and word embeddings . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Sho...

    2018 doi

  9. [18]

    Yunkai Dang, Kaichen Huang, Jiahao Huo, Yibo Yan, Sirui Huang, Dongrui Liu, Mengxi Gao, Jie Zhang, Chen Qian, Kun Wang, and 1 others. 2024. Explainable and interpretable multimodal large language models: A comprehensive survey. arXiv preprint arXiv:2412.02104

    2024 arXiv

  10. [19]

    Google DeepMind. 2025. https://deepmind.google/technologies/gemini/flash/ Gemini 2.5 flash

    2025

  11. [20]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  12. [21]

    Sylviane Granger, Estelle Dagneaux, Fanny Meunier, and Magali Paquot. 2009. International Corpus of Learner English. Version 2. Handbook and CD-ROM

    2009

  13. [22]

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, and 6 others. 2024. https://arxiv.org/abs/...

    2024 arXiv

  14. [23]

    Kaichen Huang, Jiahao Huo, Yibo Yan, Kun Wang, Yutao Yue, and Xuming Hu. 2024. Miner: Mining the underlying pattern of modality-specific neurons in multimodal large language models. arXiv preprint arXiv:2410.04819

    2024 arXiv

  15. [24]

    Jiahao Huo, Yibo Yan, Boren Hu, Yutao Yue, and Xuming Hu. 2024. Mmneuron: Discovering neuron-level domain-specific interpretation in multimodal large language model. arXiv preprint arXiv:2406.11193

    2024 arXiv

  16. [25]

    Jiahao Huo, Yibo Yan, Xu Zheng, Yuanhuiyi Lyu, Xin Zou, Zhihua Wei, and Xuming Hu. 2025. Mmunlearner: Reformulating multimodal machine unlearning in the era of multimodal large language models. arXiv preprint arXiv:2502.11051

    2025 arXiv

  17. [26]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  18. [27]

    Thorben Jansen, Jennifer Meyer, Johanna Fleckenstein, Andrea Horbach, Stefan Keller, and Jens M \"o ller. 2024. Individualizing goal-setting interventions using automated writing evaluation to support secondary school students’ text revisions. Learning and Instruction, 89:101847

    2024

  19. [28]

    Zhiwei Jiang, Tianyi Gao, Yafeng Yin, Meng Liu, Hua Yu, Zifeng Cheng, and Qing Gu. 2023. https://doi.org/10.18653/v1/2023.acl-long.696 Improving domain generalization for prompt-aware essay scoring via disentangled representation learning . In Proceedings of the 61st Annual Me...

    2023 doi

  20. [29]

    Firuz Kamalov, David Santandreu Calonge, Linda Smail, Dilshod Azizov, Dimple R Thadani, Theresa Kwong, and Amara Atif. 2025. Evolution of ai in education: Agentic workflows. arXiv preprint arXiv:2504.20082

    2025

  21. [30]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361

    2020 arXiv

  22. [31]

    Zixuan Ke and Vincent Ng. 2019. Automated essay scoring: A survey of the state of the art. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19 , pages 6300--6308. International Joint Conferences on Artificial Intelligence Org...

    2019

  23. [32]

    Anindita Kundu and Denilson Barbosa. 2024. https://arxiv.org/abs/2409.13120 Are large language models good essay graders?

    2024 arXiv

  24. [33]

    Sanwoo Lee, Yida Cai, Desong Meng, Ziyang Wang, and Yunfang Wu. 2024 a . Unleashing large language models' proficiency in zero-shot essay scoring. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 181--198

    2024

  25. [34]

    Sanwoo Lee, Yida Cai, Desong Meng, Ziyang Wang, and Yunfang Wu. 2024 b . https://doi.org/10.18653/v1/2024.findings-emnlp.10 Unleashing large language models' proficiency in zero-shot essay scoring . In Findings of the Association for Computational Linguistics: EMNLP 2024, page...

    2024 doi

  26. [35]

    Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. https://arxiv.org/abs/2303.17760 Camel: Communicative agents for "mind" exploration of large language model society . Preprint, arXiv:2303.17760

    2023 arXiv

  27. [36]

    Hang Li, Tianlong Xu, Chaoli Zhang, Eason Chen, Jing Liang, Xing Fan, Haoyang Li, Jiliang Tang, and Qingsong Wen. 2024. Bringing generative ai to adaptive learning in education. arXiv preprint arXiv:2402.14601

    2024 arXiv

  28. [37]

    Shengjie Li and Vincent Ng. 2024 a . Automated essay scoring: A reflection on the state of the art. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17876--17888

    2024

  29. [38]

    Shengjie Li and Vincent Ng. 2024 b . Automated essay scoring: Recent successes and future directions. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24 , pages 8114--8122

    2024

  30. [39]

    Shengjie Li and Vincent Ng. 2024 c . Icle++: Modeling fine-grained traits for holistic essay scoring. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), page...

    2024

  31. [40]

    Wenchao Li and Haitao Liu. 2024. Applying large language models for automated essay scoring for non-native japanese. Humanities and Social Sciences Communications, 11(1):1--15

    2024

  32. [41]

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. 2024. Encouraging divergent thinking in large language models through multi-agent debate. In Proceedings of the 2024 Conference on Empirical Methods in Natural Langua...

    2024

  33. [42]

    Shalev Lifshitz, Sheila A McIlraith, and Yilun Du. 2025. Multi-agent verification: Scaling test-time compute with multiple verifiers. arXiv preprint arXiv:2502.20379

    2025 arXiv

  34. [43]

    Chun Then Lim, Chih How Bong, Wee Sian Wong, and Nung Kion Lee. 2021. A comprehensive review of automated essay scoring (aes) research and development. Pertanika Journal of Science & Technology, 29(3):1875--1899

    2021

  35. [44]

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024 a . https://llava-vl.github.io/blog/2024-01-30-llava-next/ Llava-next: Improved reasoning, ocr, and world knowledge

    2024

  36. [45]

    Zeyang Liu, Xinrui Yang, Shiguang Sun, Long Qian, Lipeng Wan, Xingyu Chen, and Xuguang Lan. 2024 b . Grounded answers for multi-agent decision-making problem through generative world model. Advances in Neural Information Processing Systems, 37:46622--46652

    2024

  37. [46]

    Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, and 1 others. 2024. Deepseek-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525

    2024 arXiv

  38. [47]

    Watheq Mansour, Salam Albatarni, Sohaila Eltanbouly, and Tamer Elsayed. 2024. https://arxiv.org/abs/2403.06149 Can large language models automatically score proficiency of written essays? Preprint, arXiv:2403.06149

    2024 arXiv

  39. [48]

    Sandeep Mathias and Pushpak Bhattacharyya. 2018. ASAP ++: Enriching the ASAP automated essay grading dataset with essay attribute scores. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation ( LREC 2018)

    2018

  40. [49]

    Atsushi Mizumoto and Masaki Eguchi. 2023. Exploring the potential of using an ai language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2):100050

    2023

  41. [50]

    OpenAI. 2024. https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/ Gpt-4o mini: advancing cost-efficient intelligence

    2024

  42. [51]

    Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. 2025. Tool learning with large language models: A survey. Frontiers of Computer Science, 19(8):198343

    2025

  43. [52]

    Dadi Ramesh and Suresh Kumar Sanampudi. 2022. An automated essay scoring systems: a systematic literature review. Artificial Intelligence Review, 55(3):2495--2527

    2022

  44. [53]

    SeungWoo Song, Junghun Yuk, ChangSu Choi, HanGyeol Yoo, Hyeonseok Lim, KyungTae Lim, and Jungyeul Park. 2025 a . Unified automated essay scoring and grammatical error correction. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4412--4426

    2025

  45. [54]

    Shezheng Song, Xiaopeng Li, Shasha Li, Shan Zhao, Jie Yu, Jun Ma, Xiaoguang Mao, Weimin Zhang, and Meng Wang. 2025 b . How to bridge the gap between modalities: Survey on multimodal large language model. IEEE Transactions on Knowledge and Data Engineering

    2025

  46. [55]

    Yishen Song, Qianta Zhu, Huaibo Wang, and Qinhua Zheng. 2024. Automated essay scoring and revising based on open-source large language models. IEEE Transactions on Learning Technologies

    2024

  47. [56]

    Christian Stab and Iryna Gurevych. 2014. Annotating argument components and relations in persuasive essays. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers , pages 1501--1510

    2014

  48. [57]

    Jiamin Su, Yibo Yan, Fangteng Fu, Han Zhang, Jingheng Ye, Xiang Liu, Jiahao Huo, Huiyu Zhou, and Xuming Hu. 2025. Essayjudge: A multi-granular benchmark for assessing automated essay scoring capabilities of multimodal large language models

    2025

  49. [58]

    Khanh-Tung Tran, Dung Dao, Minh-Duong Nguyen, Quoc-Viet Pham, Barry O'Sullivan, and Hoang D. Nguyen. 2025. Multi-agent collaboration mechanisms: A survey of llms

    2025

  50. [59]

    Masaki Uto. 2021. A review of deep-neural automated essay scoring models. Behaviormetrika, 48(2):459--484

    2021

  51. [60]

    Masaki Uto, Yikuan Xie, and Maomi Ueno. 2020. Neural automated essay scoring incorporating handcrafted features. In Proceedings of the 28th international conference on computational linguistics, pages 6077--6088

    2020

  52. [61]

    Sowmya Vajjala. 2016. Automated assessment of non-native learner essays: Investigating the role of linguistic features. CoRR

    2016

  53. [62]

    Kun Wang, Guibin Zhang, Zhenhong Zhou, Jiahao Wu, Miao Yu, Shiqian Zhao, Chenlong Yin, Jinhu Fu, Yibo Yan, Hanjun Luo, and 1 others. 2025. A comprehensive survey in llm (-agent) full stack safety: Data, training and deployment. arXiv preprint arXiv:2504.15585

    2025 arXiv

  54. [63]

    Yongjie Wang, Chuang Wang, Ruobing Li, and Hui Lin. 2022. On the use of bert for automated essay scoring: Joint learning of multi-scale essay representation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: H...

    2022

  55. [64]

    Yucheng Wang, Zhongyu Wei, Yaqian Zhou, and Xuanjing Huang. 2018. https://doi.org/10.18653/v1/D18-1090 Automatic essay scoring incorporating rating schema via reinforcement learning . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pa...

    2018 doi

  56. [65]

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. 2023. https://arxiv.org/abs/2308.08155 Autogen: Enabling next-gen llm applications via mu...

    2023 arXiv

  57. [66]

    Xuansheng Wu, Padmaja Pravin Saraf, Gyeong-Geon Lee, Ehsan Latif, Ninghao Liu, and Xiaoming Zhai. 2024. Unveiling scoring processes: Dissecting the differences between llms and human graders in automatic scoring. arXiv preprint arXiv:2407.18328

    2024 arXiv

  58. [67]

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, and 1 others. 2023. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864

    2023 arXiv

  59. [68]

    Wei Xia, Shaoguang Mao, and Chanjing Zheng. 2024. Empirical study of large language models as automated essay scoring tools in english composition\_taking toefl independent writing task for example. arXiv preprint arXiv:2401.03401

    2024 arXiv

  60. [69]

    Changrong Xiao, Wenxing Ma, Qingping Song, Sean Xin Xu, Kunpeng Zhang, Yufang Wang, and Qi Fu. 2024. https://arxiv.org/abs/2401.06431 Human-ai collaborative essay scoring: A dual-process framework with llms . Preprint, arXiv:2401.06431

    2024 arXiv

  61. [70]

    Wenbo Xu, Muhammad Shahreeza, Wai Lam Hoo, and Wudao Yang. 2025. Explainable ai for education: Enhancing essay scoring via rubric-aligned chain-of-thought prompting

    2025

  62. [71]

    Yibo Yan and Joey Lee. 2024. Georeasoner: Reasoning on geospatially grounded context for natural language understanding. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 4163--4167

    2024

  63. [72]

    Yibo Yan, Jiamin Su, Jianxiang He, Fangteng Fu, Xu Zheng, Yuanhuiyi Lyu, Kun Wang, Shen Wang, Qingsong Wen, and Xuming Hu. 2024 a . A survey of mathematical reasoning in the era of multimodal large language model: Benchmark, method & challenges. arXiv preprint arXiv:2412.11936

    2024 arXiv

  64. [73]

    Yibo Yan, Shen Wang, Jiahao Huo, Hang Li, Boyan Li, Jiamin Su, Xiong Gao, Yi-Fan Zhang, Tianlong Xu, Zhendong Chu, and 1 others. 2024 b . Errorradar: Benchmarking complex mathematical reasoning of multimodal large language models via error detection. arXiv preprint arXiv:2410.04509

    2024 arXiv

  65. [74]

    Yibo Yan, Shen Wang, Jiahao Huo, Jingheng Ye, Zhendong Chu, Xuming Hu, Philip S Yu, Carla Gomes, Bart Selman, and Qingsong Wen. 2025 a . Position: Multimodal large language models can significantly advance scientific reasoning. arXiv preprint arXiv:2502.02871

    2025 arXiv

  66. [75]

    Yibo Yan, Shen Wang, Jiahao Huo, Philip S Yu, Xuming Hu, and Qingsong Wen. 2025 b . Mathagent: Leveraging a mixture-of-math-agent framework for real-world multimodal mathematical error detection. arXiv preprint arXiv:2503.18132

    2025 arXiv

  67. [76]

    Yibo Yan, Haomin Wen, Siru Zhong, Wei Chen, Haodong Chen, Qingsong Wen, Roger Zimmermann, and Yuxuan Liang. 2024 c . Urbanclip: Learning text-enhanced urban region profiling with contrastive language-image pretraining from the web. In Proceedings of the ACM on Web Conference 2...

    2024

  68. [77]

    Kaixun Yang, Mladen Rakovi \'c , Yuyang Li, Quanlong Guan, Dragan Ga s evi \'c , and Guangliang Chen. 2024. Unveiling the tapestry of automated essay scoring: A comprehensive investigation of accuracy, fairness, and generalizability. In Proceedings of the AAAI Conference on Ar...

    2024

  69. [78]

    Helen Yannakoudakis and Ted Briscoe. 2012. Modeling coherence in ESOL learner texts. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP , pages 33--43

    2012

  70. [79]

    Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading ESOL texts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 180--189

    2011

  71. [80]

    Jingheng Ye, Shen Wang, Deqing Zou, Yibo Yan, Kun Wang, Hai-Tao Zheng, Zenglin Xu, Irwin King, Philip S Yu, and Qingsong Wen. 2025. Position: Llms can be good tutors in foreign language education. arXiv preprint arXiv:2502.05467

    2025 arXiv

  72. [81]

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, and 1 others. 2024. Yi: Open foundation models by 01. ai. arXiv preprint arXiv:2403.04652

    2024 arXiv

  73. [82]

    Yuan Yuan, Zhaojian Li, and Bin Zhao. 2025. A survey of multimodal learning: Methods, applications, and future. ACM Computing Surveys

    2025

  74. [83]

    Zhiqiang Yuan, Weitong Chen, Hanlin Wang, Kai Yu, Xin Peng, and Yiling Lou. 2024. Transagent: An llm-based multi-agent system for code translation. arXiv preprint arXiv:2409.19894

    2024 arXiv

  75. [84]

    Guibin Zhang, Kaijie Chen, Guancheng Wan, Heng Chang, Hong Cheng, Kun Wang, Shuyue Hu, and Lei Bai. 2025. Evoflow: Evolving diverse agentic workflows on the fly. arXiv preprint arXiv:2502.07373

    2025 arXiv

  76. [85]

    Guibin Zhang, Yanwei Yue, Zhixun Li, Sukwon Yun, Guancheng Wan, Kun Wang, Dawei Cheng, Jeffrey Xu Yu, and Tianlong Chen. 2024 a . Cut the crap: An economical communication pipeline for llm-based multi-agent systems. arXiv preprint arXiv:2410.02506

    2024 arXiv

  77. [86]

    Guibin Zhang, Yanwei Yue, Xiangguo Sun, Guancheng Wan, Miao Yu, Junfeng Fang, Kun Wang, Tianlong Chen, and Dawei Cheng. 2024 b . G-designer: Architecting multi-agent communication topologies via graph neural networks. arXiv preprint arXiv:2410.11782

    2024 arXiv

  78. [87]

    Kening Zheng, Junkai Chen, Yibo Yan, Xin Zou, and Xuming Hu. 2024. Reefknot: A comprehensive benchmark for relation hallucination evaluation, analysis and mitigation in multimodal large language models. arXiv preprint arXiv:2408.09429

    2024 arXiv

  79. [88]

    Guanyu Zhou, Yibo Yan, Xin Zou, Kun Wang, Aiwei Liu, and Xuming Hu. 2024. Mitigating modality prior-induced hallucinations in multimodal large language models via deciphering attention causality. arXiv preprint arXiv:2410.04780

    2024 arXiv

  80. [89]

    Xin Zou, Yizhou Wang, Yibo Yan, Sirui Huang, Kening Zheng, Junkai Chen, Chang Tang, and Xuming Hu. 2024. Look twice before you answer: Memory-space visual retracing for hallucination mitigation in multimodal large language models. arXiv preprint arXiv:2410.03577

    2024 arXiv

  81. [90]

    Xingchen Zou, Yibo Yan, Xixuan Hao, Yuehong Hu, Haomin Wen, Erdong Liu, Junbo Zhang, Yong Li, Tianrui Li, Yu Zheng, and 1 others. 2025. Deep learning for cross-domain data fusion in urban computing: Taxonomy, advances, and outlook. Information Fusion, 113:102606

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.