Pith. sign in

REVIEW 5 major objections 4 minor 58 references

How well can LLMs Grade Essays in Arabic?

T0 review · 5 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that generative large language models do not outperform a smaller fine-tuned Arabic BERT model at scoring Arabic essays, with the best LLM reaching QWK 0.67 against AraBERT's 0.88 on the AR-AES dataset.

desk verdict Useful first benchmark of generative LLMs for Arabic essay scoring, but the headline BERT-vs-LLM gap rests on an imported AraBERT baseline the authors never re-ran, so the comparison needs verification before it can be taken at face value. read the letter →

arxiv 2501.16516 v1 pith:OTYNPOJB submitted 2025-01-27 cs.CL cs.AI

Rayed Ghazawi , Edwin Simpson This is my paper · ORCID
classification cs.CLcs.AI
keywords ArabicNLPautomatedessayscoringlargelanguagemodelsBERTpromptengineeringfew-shotlearningfine-tuningtokenization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether generative large language models can grade Arabic essays as reliably as smaller, task-specialised models. It evaluates ChatGPT-4, Llama 2, Aya, Jais, and ACEGPT on the AR-AES dataset of 2,046 authentic student essays, using zero-shot, few-shot, and fine-tuning setups. Its central finding is that none of the generative LLMs matches the fine-tuned Arabic BERT model, which scores QWK 0.88 against the best LLM result of 0.67 from ACEGPT. The study also shows that mixing English instructions with Arabic essay content substantially improves LLM outputs, and that an Arabic-aware tokenizer reduces computational cost without hurting accuracy. If the finding holds, Arabic assessment systems should prefer fine-tuned encoder models over generative LLMs until the gap is closed.

What carries the argument

The comparison is carried by the Quadratic Weighted Kappa (QWK) metric, which measures agreement between model scores and the course director's marks while accounting for chance and ordinality. The dataset is AR-AES, 2,046 essays across 12 prompts, with the published AraBERT result serving as the baseline to beat. The main adaptation mechanism is label-supervised adaptation (LS-LLaMA), which appends a classification layer to the LLM's final hidden states and fine-tunes with cross-entropy instead of asking the model to generate a score. A custom SentencePiece tokenizer trained on the dataset and merged into Llama's vocabulary reduces average sequence length from 1,532 to 410 tokens, making experiments feasible while slightly improving QWK. For prompted models, the effective lever is bilingual prompting, in which instructions are English while question, rubric, golden answer, and essay stay Arabic.

What would settle it

Re-run AraBERT on the same 70/15/15 split with the same stemmed and normalised preprocessing used for the LLMs; if its full-dataset QWK drops to 0.67 or below, the paper's central claim that BERT-class models outperform generative LLMs would fail.

Watch

Extended reading notes

Core claim

The paper's claim is that generative LLMs do not yet surpass specialised pretrained encoders for Arabic automated essay scoring. Across the whole AR-AES dataset, ACEGPT, an Arabic-focused Llama-based model fine-tuned with label-supervised adaptation, achieved the highest LLM score with QWK 0.67, while the smaller AraBERT model reported in the dataset's original study reached QWK 0.88. The paper attributes the gap to BERT's bidirectional encoder being better suited to classification with small labelled sets, and to LLMs' poor character-level tokenization of Arabic, which inflates sequence length and computational cost. It also finds that fine-tuning through a classification head (LS-LLaMA) works better than instruction tuning or prompting, and that bilingual prompts—English task instructions with Arabic questions, rubrics, and essays—lift ChatGPT-4's zero-shot scores by about 49.49% over monolingual prompts. On individual questions, few-shot ChatGPT-4 can exceed human inter-rater agreement (QWK 0.93 on Question 8 vs 0.78 human), but this does not translate into overall superiority.

Load-bearing premise

The comparison assumes the published AraBERT scores were obtained under the same data split, preprocessing, and ground-truth definition as the new LLM runs, because the BERT baseline is taken from the authors' prior paper rather than re-run in this study.

Editorial extensions

If this is right

  • For Arabic AES with small labelled datasets, fine-tuned BERT-class models remain the empirically safer choice; generative LLMs are not yet competitive despite their larger scale.
  • Fine-tuning LLMs with a label-supervised classification head is the reliable route for Arabic scoring, outperforming instruction tuning and few-shot or zero-shot prompting by large margins.
  • Prompt engineers for Arabic NLP should use English instructions with Arabic content: the paper reports a 49.49% gain over monolingual prompts in ChatGPT-4 zero-shot scoring.
  • Adding Arabic word-level tokens to an LLM's vocabulary cuts sequence length roughly fourfold and slightly raises QWK, a practical step for any Arabic LLM deployment.
  • Few-shot LLMs can match or beat human inter-rater agreement on individual scoring questions, suggesting LLMs are already useful as a second marker even though they trail the best fine-tuned model overall.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the result is likely sensitive to training-set size; with the 1,432 training essays used here, encoders dominate, but a larger Arabic AES corpus could close or reverse the gap given the LLMs' scale.
  • Beyond the paper: the custom-tokenizer result implies a general, low-cost recipe for low-resource languages—extend an LLM's vocabulary with word-level tokens before fine-tuning—that this study only demonstrates for Arabic.
  • Beyond the paper: bilingual prompting may generalise beyond scoring to other Arabic tasks where instruction following matters, such as short-answer grading or rubric-based feedback generation, though the paper does not test those tasks.
  • Beyond the paper: a direct head-to-head rerun of AraBERT under the exact split and preprocessing used for the LLMs would confirm whether the 0.21 QWK gap is real or partly an artifact of protocol differences.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper evaluates five generative LLMs (ChatGPT-4, Llama 2/OpenLLaMA, Aya, Jais, and ACEGPT) on the AR-AES Arabic essay-scoring dataset using zero-shot, few-shot, and fine-tuning settings, with additional experiments on prompt engineering, Arabic-specific tokenization, and preprocessing. The main empirical result is that ACEGPT achieves the best LLM QWK of 0.67 on the full dataset, while the smaller fine-tuned AraBERT model achieves 0.88, leading the authors to conclude that fine-tuned encoder models remain more reliable than generative LLMs for Arabic AES. The paper also reports a 49.49% improvement from mixed English-instruction/Arabic-content prompts and a custom SentencePiece tokenizer that reduces sequence length and compute.

Significance. If the headline comparison is valid, this is a practically useful result for Arabic AES: it suggests that fine-tuned transformer encoders such as AraBERT remain the state of the art, and that generative LLMs offer no accuracy advantage despite much higher computational cost. The study is one of the first to benchmark multiple generative LLMs on authentic Arabic student essays, and the per-course and per-question QWK tables in Table 3 are a useful reference for future work. However, the central comparison rests on an AraBERT baseline imported from the authors' own prior paper [8] rather than re-run under the present protocol, the QWK formula in Eq. (1) is misstated, and the custom tokenizer is trained on the full dataset including the test split. These issues affect the load-bearing claims and must be addressed before the conclusion can be accepted.

major comments (5)
  1. [§7, Table 3] The headline comparison between ACEGPT (QWK 0.67) and AraBERT (QWK 0.88) relies entirely on AraBERT scores imported from the authors' prior paper [8] rather than re-run in this study. Section 7 states only that the experimental protocol was 'delineated by' [8] and 'adapted', and Section 4.3 introduces a preprocessing pipeline for fine-tuned LLMs (punctuation removal, Arabic character normalization, ISRI stemming) without verifying that the [8] AraBERT experiments used the identical 70/15/15 split and the same preprocessing. Unless the AraBERT baseline is re-run under the present protocol, or explicit evidence of identical split and preprocessing is supplied, the central claim that generative LLMs do not outperform BERT on AR-AES is not established.
  2. [§5, Eq. (1)] Equation (1) does not define the standard quadratic weighted kappa. In the conventional formula the denominator is ∑_{i,j} w_{i,j} E_{i,j} with E_{i,j} = n_{i,1} n_{j,2} / M, where M is the total number of essays; the printed denominator ∑_{i,j} w_{i,j} n_{i,1} n_{j,2} is a factor of M too large. Since N in the text is defined as the number of marks, it cannot serve as this divisor. Please correct the equation and state the exact implementation used to produce the Table 3 numbers (e.g., sklearn.metrics.cohen_kappa_score with weights='quadratic').
  3. [§4.2] The custom SentencePiece tokenizer was trained on the AR-AES dataset itself, which includes the test split used for evaluation. Fitting the tokenizer vocabulary on test-set essays is a form of data leakage for the fine-tuned Llama and ACEGPT results: it can only assist the model in representing test-set vocabulary and potentially inflates the reported QWK values. The tokenizer should be trained on the training split only, or the paper should demonstrate that tokenizer vocabulary overlap with the test set does not affect the fine-tuning results.
  4. [§7, Table 4] The statement that LLMs 'surpass human agreement in most questions' compares model agreement with the course director's marks against the inter-rater agreement between two human raters reported in [8]. These are not directly comparable: model-vs-director agreement and human-vs-human agreement measure different rater pairings and different sources of variance, so a higher QWK in one comparison does not mean the model agrees with the human more than humans agree with each other. This claim should be rephrased as a comparison of model-to-director agreement with human-to-human agreement, and any 'surpassing' conclusion requires a matched analysis on the same reference rater.
  5. [§6, Table 2] The 49.49% improvement attributed to mixed-language prompting is derived from only three selected questions (Q1, Q9, Q11), with no confidence intervals, significance tests, or repeated-split evaluation. Table 2 shows substantial variation across those three questions (e.g., Q1 0.24 vs Q9 0.66 in the bilingual zero-shot row), so the percentage should be presented as a preliminary and illustrative observation, not as a general effect. More broadly, all QWK values in Table 3 are point estimates from a single 70/15/15 split without confidence intervals, which limits the interpretability of small differences between models.
minor comments (4)
  1. [§4.2, Figure 1] The Arabic example 'ابحرم' appears to be a rendering or transliteration error for 'مرحبا' (welcome); please correct the Arabic text in both the body and Figure 1 and verify the tokenization counts against the corrected word.
  2. [§7] The phrase 'archeticture differences' should be 'architectural differences'.
  3. [Tables 3 and A1] The table formatting is inconsistent: course names are abbreviated differently ('Intro Info Sci', 'Management Info Sys', 'Environmental Chemistry', 'Biotechnology') and some cells in Table A1 are missing percent signs or use inconsistent spacing. Please unify the style and check every cell for completeness.
  4. [§4.5, §6] Please clarify the few-shot sample counts: Section 4.5 says three examples per class for ChatGPT, which with six classes implies 18 examples in the prompt, while Section 6 says 'three example answers for each class' but also shows 'Examples of Each Score' with class labels 0-5. The total number of examples in each few-shot prompt should be stated explicitly, especially since the paper later notes Jais had to be reduced to one example per class due to context length.

Circularity Check

1 steps flagged · score 4.0 of 10

The central BERT-versus-LLM comparison imports the AraBERT 0.88 QWK from the authors' prior paper [8] without re-running it under this paper's preprocessing and split, making the headline claim load-bearing on a self-citation.

  1. self citation load bearing [Section 4.5 (Model Implementations), Section 7 (Results and Discussion), and Table 3]
    "The study introducing the dataset employed the large AraBERT model, comprising 12 encoder blocks, 1024 hidden dimensions, 16 attention heads, and a maximum sequence length of 512. The model has 370 million parameters. A classification head with a single fully connected layer was added to AraBERT for essay scoring [8]. / Our study used the experimental protocol delineated by Ghazawi and Simpson [8]. We adapted this framework for our exploratory analysis involving four LLMs (Llama, ChatGPT, Aya, Jais, and ACEGPT)"

    The headline claim that generative LLMs do not outperform smaller pretrained models ('ACEGPT ... QWK of 0.67, but was outperformed by a smaller BERT-based model with a QWK of 0.88') rests on the AraBERT 0.88 figure, which is not produced in this paper. Section 4.5 describes the AraBERT system in past tense and cites [8], the authors' own prior paper; Section 7 says the current study 'used the experimental protocol delineated by Ghazawi and Simpson [8]' but does not re-run AraBERT on the same preprocessed, split data. The comparison in Table 3 thus treats a self-cited fitted result as a within-study baseline.

full rationale

The paper is an empirical benchmark rather than a mathematical derivation, so most circularity patterns do not apply. The LLM QWK values (e.g., ACEGPT 0.67, ChatGPT-4 few-shot 0.64) are computed against human course-director marks within this paper, so they are not equivalent by construction to any fitted input. The one load-bearing circularity-adjacent step is the AraBERT baseline: the central claim that BERT-based models outperform generative LLMs is supported by a QWK of 0.88 taken from the authors' own prior paper [8], with the current paper stating only that it 'used the experimental protocol delineated by Ghazawi and Simpson [8]' and adapting it. No re-run of AraBERT under the current preprocessing (punctuation removal, normalization, ISRI stemming) or the exact 70/15/15 split is reported, so the comparison reduces, at its weakest link, to a self-citation. This is not a full circular derivation: the LLM side is independently measured, and the prior baseline is publicly available, so the claim is checkable. Accordingly, score 4. A separate concern that the custom SentencePiece tokenizer was trained on the same AR-AES dataset used for evaluation is a data-leakage or validity risk, but it does not by construction force the predicted QWK values and is not counted as circularity here.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim is empirical, so the ledger contains experimental assumptions rather than derivation axioms. The most serious entry is the tokenizer fitted to the evaluation corpus, which is a genuine free parameter trained on target data. The other entries are common assumptions in AES evaluations, but their violation would weaken the reported magnitudes.

free parameters (2)
  • Custom SentencePiece tokenizer vocabulary trained on AR-AES corpus = unspecified vocabulary size
    Trained on the full AR-AES dataset, including test essays. This fitted component is used in fine-tuning Llama and ACEGPT, and it is a potential source of data leakage.
  • Number of few-shot examples per class = 3 for ChatGPT, Aya, ACEGPT; 1 for Jais
    Chosen to fit Jais's 2048-token context window; it is a hand-set experimental choice that affects few-shot results but not the main fine-tuning comparison.
assumptions (4)
  • domain assumption Course director's marks are the ground truth for scoring quality.
    All QWK scores compare model predictions against the course director's grades; the reliability of those grades is not assessed in this paper.
  • domain assumption AraBERT scores from [8] were obtained under the same data split and preprocessing protocol as the LLM scores reported here.
    The paper 'adapted' the framework of [8] but did not re-run AraBERT; if the splits or preprocessing differ, the comparative claim is compromised.
  • domain assumption A single 70/15/15 random split yields stable QWK estimates.
    No repeated splits or confidence intervals are reported; question-level test sets contain 16 to 42 essays.
  • standard math QWK is an appropriate metric for comparing graders and models on ordinal 0-5 scores.
    QWK is the standard metric in AES and the paper uses the standard formula (Eq. 1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of How well can LLMs Grade Essays in Arabic?." pith.science (2026). https://pith.science/paper/OTYNPOJB

@misc{pith2026250116516,
  author       = {Pith},
  title        = {Pith review of: How well can LLMs Grade Essays in Arabic?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OTYNPOJB}},
  note         = {Machine review of arXiv:2501.16516}
}
read the original abstract

This research assesses the effectiveness of state-of-the-art large language models (LLMs), including ChatGPT, Llama, Aya, Jais, and ACEGPT, in the task of Arabic automated essay scoring (AES) using the AR-AES dataset. It explores various evaluation methodologies, including zero-shot, few-shot in-context learning, and fine-tuning, and examines the influence of instruction-following capabilities through the inclusion of marking guidelines within the prompts. A mixed-language prompting strategy, integrating English prompts with Arabic content, was implemented to improve model comprehension and performance. Among the models tested, ACEGPT demonstrated the strongest performance across the dataset, achieving a Quadratic Weighted Kappa (QWK) of 0.67, but was outperformed by a smaller BERT-based model with a QWK of 0.88. The study identifies challenges faced by LLMs in processing Arabic, including tokenization complexities and higher computational demands. Performance variation across different courses underscores the need for adaptive models capable of handling diverse assessment formats and highlights the positive impact of effective prompt engineering on improving LLM outputs. To the best of our knowledge, this study is the first to empirically evaluate the performance of multiple generative Large Language Models (LLMs) on Arabic essays using authentic student data.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 39 canonical work pages

  1. [8]

    Automated essay scoring in Arabic: a dataset and analysis of a BERT-based system

    Ghazawi, R., Simpson, E.: Automated essay scoring in Arabic: a dataset and analysis of a Bert-based system. arXiv preprint arXiv:2407.11212 (2024)

    work page Pith review arXiv 2024

  2. [1]

    In: Proceedings of the AAAI Conference on Artificial Intelligence, vol

    Yang, K., Raković, M., Li, Y., Guan, Q., Gašević, D., Chen, G.: Unveiling the tapestry of automated essay scoring: A comprehensive investigation of accu- racy, fairness, and generalizability. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. 22466–22474 (2024)

    work page 2024

  3. [2]

    In: Proceedings of the 16th International Conference on Computational Processing of Portuguese (2024)

    Silveira, I.C., Barbosa, A., Mauá, D.D.: A new benchmark for automatic es- say scoring in Portuguese. In: Proceedings of the 16th International Conference on Computational Processing of Portuguese (2024). https://github.com/kamel- usp/aes_enem

    work page 2024

  4. [3]

    https://doi.org/10.13140/RG.2.2.13615.51365/1

    Franci, Y., Franci, Y.A.: Enhancing Automated Essay Evaluation: The Impact of Advanced Generative Pre-trained Transformers on Educational Feed- back (2023). https://doi.org/10.13140/RG.2.2.13615.51365/1 . https://www. researchgate.net/publication/376271077

    work page arXiv 2023

  5. [4]

    Meta-Radiology 1, 100017 (2023) https://doi.org/10

    Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., Wu, Z., Zhao, L., Zhu, D., Li, X., Qiang, N., Shen, D., Liu, T., Ge, B.: Summary of ChatGPT-related research and perspective towards the future of Large Language Models. Meta-Radiology 1, 100017 (2023) https://doi.org/10. 1016/j.metrad.2023.100017

    arXiv 2023

  6. [5]

    In: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, pp

    Ahmed, T., Devanbu, P.: Few-shot training LLMs for project-specific code- summarization. In: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, pp. 1–5 (2022)

    work page 2022

  7. [6]

    arXiv preprint arXiv:2401.06431 (2024)

    Xiao, C., Ma, W., Xu, S.X., Zhang, K., Wang, Y., Fu, Q.: From automation to augmentation: Large Language Models elevating essay scoring landscape. arXiv preprint arXiv:2401.06431 (2024)

    arXiv 2024

  8. [7]

    (eds.) Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp

    Mansour, W.A., Albatarni, S., Eltanbouly, S., Elsayed, T.: Can Large Language Models automatically score proficiency of written essays? In: Calzolari, N., Kan, M.-Y., Hoste, V., Lenci, A., Sakti, S., Xue, N. (eds.) Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp....

    work page 2024

Show all 58 references
  1. [9]

    grading essays by computer

    Page, E.B.: The imminence of... grading essays by computer. The Phi Delta Kappan 47(5), 238–243 (1966)

    1966

  2. [10]

    In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 19 pp

    Yannakoudakis, H., Briscoe, T., Medlock, B.: A new dataset and method for automatically grading ESOL texts. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 19 pp. 180–189 (2011)

    2011

  3. [11]

    Association for Computational Linguistics, 1741–1752 (2013)

    Chen, H., He, B.: Automated essay scoring by maximizing human-machine agreement. Association for Computational Linguistics, 1741–1752 (2013)

    2013

  4. [12]

    In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp

    Phandi, P., Chai, K.M.A., Ng, H.T.: Flexible domain adaptation for automated essay scoring using correlated linear regression. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 431–439 (2015)

    2015

  5. [13]

    In: Levy, R., Specia, L

    Dong, F., Zhang, Y., Yang, J.: Attention-based recurrent convolutional neu- ral network for automatic essay scoring. In: Levy, R., Specia, L. (eds.) Proceedings of the 21st Conference on Computational Natural Language Learn- ing (CoNLL 2017), pp. 153–162. Association for Compu...

    2017 doi

  6. [14]

    Expert Systems 39(10), 13068 (2022)

    Li, X., Yang, H., Hu, S., Geng, J., Lin, K., Li, Y.: Enhanced hybrid neural network for automated essay scoring. Expert Systems 39(10), 13068 (2022)

    2022

  7. [15]

    In: 2018 International Conference on Asian Language Processing (IALP), pp

    Chen, M., Li, X.: Relevance-based automated essay scoring via hierarchical re- current model. In: 2018 International Conference on Asian Language Processing (IALP), pp. 378–383 (2018). IEEE

    2018

  8. [16]

    arXiv preprint arXiv:2205.03835 (2022)

    Wang, Y., Wang, C., Li, R., Lin, H.: On the use of BERT for automated es- say scoring: Joint learning of multi-scale essay representation. arXiv preprint arXiv:2205.03835 (2022)

    2022 arXiv

  9. [17]

    In: Findings of the Association for Computational Linguistics: EMNLP 2020, pp

    Yang, R., Cao, J., Wen, Z., Wu, Y., He, X.: Enhancing automated essay scor- ing performance via fine-tuning pre-trained language models with combination of regression and ranking. In: Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1560–1569 (2020)

    2020

  10. [18]

    In: International Conference on Intelligent Systems Design and Applications, pp

    Machhout, R.A., Zribi, C.B.O., Bouzid, S.M.: Arabic automatic essay scoring systems: An overview study. In: International Conference on Intelligent Systems Design and Applications, pp. 1164–1176 (2021). Springer

    2021

  11. [19]

    AI Communications 27(2), 103–111 (2014) https://doi.org/10.3233/AIC-130586

    Alghamdi, M., Alkanhal, M., Al-Badrashiny, M., Al-Qabbany, A., Areshey, A., Alharbi, A.: A hybrid automatic scoring system for Arabic essays. AI Communications 27(2), 103–111 (2014) https://doi.org/10.3233/AIC-130586

    2014 doi

  12. [20]

    International Journal of Computer Science Issues 13(5), 45–50 (2016)

    Al-Shalabi, E.F.: An automated system for essay scoring of online exams in Ara- bic based on stemming techniques and levenshtein edit operations. International Journal of Computer Science Issues 13(5), 45–50 (2016)

    2016

  13. [21]

    In: Proceedings of the 17th International Conference on Natural Language Processing (ICON), 20 pp

    Alqahtani, A., Alsaif, A.: Automated Arabic essay evaluation. In: Proceedings of the 17th International Conference on Natural Language Processing (ICON), 20 pp. 181–190. NLP Association of India (NLPAI), Indian Institute of Technology Patna, Patna, India (2020). https://aclant...

    2020

  14. [22]

    2019 IEEE international symposium on signal processing and information technology (ISSPIT) (2019)

    Alqahtani, A., Alsaif, A.: Automatic Evaluation for Arabic Essays: A Rule- Based System. 2019 IEEE international symposium on signal processing and information technology (ISSPIT) (2019)

    2019

  15. [23]

    Journal of Theoretical and Applied Information Technology (2021)

    Al-Shargabi, B., Alzyadat, R., Hamad, F.: AEGD: Arabic essay grading dataset for machine learning. Journal of Theoretical and Applied Information Technology (2021)

    2021

  16. [24]

    Education and Information Technologies 26(1), 1165–1181 (2021) https://doi.org/10.1007/s10639-020-10300-6

    Gaheen, M.M., ElEraky, R.M., Ewees, A.A.: Automated students Arabic essay scoring using trained neural network by E-Jaya optimization to support per- sonalized system of instruction. Education and Information Technologies 26(1), 1165–1181 (2021) https://doi.org/10.1007/s10639-...

    2021 doi

  17. [25]

    Artificial Intelligence Review 55(3), 2495–2527 (2022)

    Ramesh, D., Sanampudi, S.K.: An automated essay scoring systems: a system- atic literature review. Artificial Intelligence Review 55(3), 2495–2527 (2022)

    2022

  18. [26]

    Customer Needs and Solutions 11(1), 3 (2024)

    Krugmann, J.O., Hartmann, J.: Sentiment analysis in the age of generative AI. Customer Needs and Solutions 11(1), 3 (2024)

    2024

  19. [27]

    arXiv preprint arXiv:2307.09288 (2023)

    Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bash- lykov, N., Batra, S., Bhargava, P., Bhosale, S., et al.: Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

    2023 arXiv

  20. [28]

    arXiv preprint arXiv:2302.13971 (2023)

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  21. [29]

    arXiv preprint arXiv:2308.16149 (2023)

    Sengupta, N., Sahu, S.K., Jia, B., Katipomu, S., Li, H., Koto, F., Afzal, O.M., Kamboj, S., Pandit, O., Pal, R., et al.: Jais and Jais-Chat: Arabic-centric foun- dation and instruction-tuned open generative Large Language Models. arXiv preprint arXiv:2308.16149 (2023)

    2023 arXiv

  22. [30]

    Electronics 13(4), 764 (2024)

    Ammar, A., Koubaa, A., Benjdira, B., Nacar, O., Sibaee, S.: Prediction of Arabic legal rulings using large language models. Electronics 13(4), 764 (2024)

    2024

  23. [31]

    In: 2023 20th ACS/IEEE International Conference on Computer Systems and Applications (AICCSA), pp

    Tamer, A., Hassan, A.-A., Ali, A., Salah, N., Medhat, W.: Fine tuning of Large Language Models for Arabic Language. In: 2023 20th ACS/IEEE International Conference on Computer Systems and Applications (AICCSA), pp. 1–4 (2023). IEEE

    2023

  24. [32]

    arXiv preprint arXiv:2405.00602 (2024)

    Katuka, G.A., Gain, A., Yu, Y.-Y.: Investigating automatic scoring and feedback using Large Language Models. arXiv preprint arXiv:2405.00602 (2024)

    2024 arXiv

  25. [33]

    Learning and Individual Differences 103, 102274 (2023) https://doi.org/10.1016/j.lindif

    Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, 21 F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., Stadler, M., Weller, J...

    2023

  26. [34]

    Computers and Education: Artificial Intelligence 6, 100210 (2024)

    Latif, E., Zhai, X.: Fine-tuning ChatGPT for automatic scoring. Computers and Education: Artificial Intelligence 6, 100210 (2024)

    2024

  27. [35]

    International Journal of Artificial Intelligence in Education, 1–26 (2024)

    Henkel, O., Hills, L., Roberts, B., McGrane, J.: Can llms grade open response reading comprehension questions? an empirical study using the roars dataset. International Journal of Artificial Intelligence in Education, 1–26 (2024)

    2024

  28. [36]

    In: Proceedings of the Eleventh ACM Conference on Learning@ Scale, pp

    Henkel, O., Hills, L., Boxer, A., Roberts, B., Levonian, Z.: Can large language models make the grade? an empirical study evaluating LLMs’ ability to mark short answer questions in k-12 education. In: Proceedings of the Eleventh ACM Conference on Learning@ Scale, pp. 300–304 (2024)

    2024

  29. [37]

    Journal of Applied Learning and Teaching 6(1) (2023)

    Rasul, T., Nair, S., Kalendra, D., Robin, M., Oliveira Santini, F., Ladeira, W.J., Sun, M., Day, I., Rather, R.A., Heathcote, L.: The role of ChatGPT in higher education: Benefits, challenges, and future research directions. Journal of Applied Learning and Teaching 6(1) (2023)

    2023

  30. [38]

    arXiv preprint arXiv:2402.12840 (2024)

    Koto, F., Li, H., Shatnawi, S., Doughman, J., Sadallah, A.B., Alraeesi, A., Al- mubarak, K., Alyafeai, Z., Sengupta, N., Shehata, S., et al.: ArabicMMLU: Assessing massive multitask language understanding in Arabic. arXiv preprint arXiv:2402.12840 (2024)

    2024 arXiv

  31. [39]

    arXiv preprint arXiv:2303.08774 (2023)

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: GPT-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    2023 arXiv

  32. [40]

    assessing performance and accuracy of explanations

    Ghosn, Y., El Sardouk, O., Jabbour, Y., Jrad, M., Hussein Kamareddine, M., Abbas, N., Saade, C., Abi Ghanem, A.: ChatGPT 4 versus ChatGPT 3.5 on the final FRCR part a sample questions. assessing performance and accuracy of explanations. medRxiv, 2023–09 (2023)

    2023

  33. [41]

    JAAOS-Journal of the American Academy of Orthopaedic Surgeons 31(23), 1173–1179 (2023)

    Massey, P.A., Montgomery, C., Zhang, A.S.: Comparison of ChatGPT–3.5, ChatGPT-4, and orthopaedic resident performance on orthopaedic assessment ex- aminations. JAAOS-Journal of the American Academy of Orthopaedic Surgeons 31(23), 1173–1179 (2023)

    2023

  34. [42]

    AI 4(4), 949–969 (2023) 22

    Plevris, V., Papazafeiropoulos, G., Jiménez Rios, A.: Chatbots put to the test in math and logic problems: a comparison and assessment of ChatGPT-3.5, ChatGPT-4, and Google Bard. AI 4(4), 949–969 (2023) 22

    2023

  35. [43]

    arXiv preprint arXiv:2402.07827 (2024)

    Üstün, A., Aryabumi, V., Yong, Z.-X., Ko, W.-Y., D’souza, D., Onilude, G., Bhandari, N., Singh, S., Ooi, H.-L., Kayid, A., et al.: Aya model: An in- struction finetuned open-access multilingual language model. arXiv preprint arXiv:2402.07827 (2024)

    2024 arXiv

  36. [44]

    In: Calzolari, N., Béchet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Odijk, J., Piperidis, S

    Alyafeai, Z., Masoud, M., Ghaleb, M., Al-shaibani, M.S.: Masader: Meta- data sourcing for Arabic text and speech data resources. In: Calzolari, N., Béchet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Odijk,...

    2022

  37. [45]

    In: Rogers, A., Boyd-Graber, J., Okazaki, N

    Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Biderman, S., Le Scao, T., Bari, M.S., Shen, S., Yong, Z.X., Schoelkopf, H., Tang, X., Radev, D., Aji, A.F., Almubarak, K., Albanie, S., Alyafeai, Z., Webson, A., Raff, E., Raffel, C.: Crosslingual generalization through mu...

    2023

  38. [46]

    Human Language Technologies (Volume 1: Long Papers), 8139–8163 (2024) https://doi

    Huang, H., Yu, F., Zhu, J., Sun, X., Cheng, H., Dingjie, S., Chen, Z., Alharthi, M., An, B., He, J., Liu, Z., Chen, J., Li, J., Wang, B., Zhang, L., Sun, R., Wan, X., Li, H., Xu, J.: AceGPT, localizing large language models in Arabic. Human Language Technologies (Volume 1: Lon...

    2024 doi

  39. [47]

    Demidova, A., Atwany, H., Rabih, N., Sha’ban, S.: Arabic train at NADI 2024 shared task: LLMs’ ability to translate Arabic dialects into Modern Standard Arabic. In: Habash, N., Bouamor, H., Eskander, R., Tomeh, N., Abu Farha, I., Abdelali, A., Touileb, S., Hamed, I., Onaizan, ...

    2024

  40. [48]

    In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) (2018)

    Mathias, S., Bhattacharyya, P.: ASAP++: Enriching the ASAP automated essay grading dataset with essay attribute scores. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) (2018)

    2018

  41. [49]

    arXiv preprint arXiv:2407.15390 (2024)

    Bari, M.S., Alnumay, Y., Alzahrani, N.A., Alotaibi, N.M., Alyahya, H.A., Al- Rashed, S., Mirza, F.A., Alsubaie, S.Z., Alahmed, H.A., Alabduljabbar, G., et 23 al.: ALLaM: Large Language Models for Arabic and English. arXiv preprint arXiv:2407.15390 (2024)

    2024 arXiv

  42. [50]

    Advances in Neural Information Processing Systems 36 (2024)

    Petrov, A., La Malfa, E., Torr, P., Bibi, A.: Language model tokenizers intro- duce unfairness between languages. Advances in Neural Information Processing Systems 36 (2024)

    2024

  43. [51]

    : Beyond English: Evaluating LLMs for Arabic grammatical error correction

    Kwon, S., Bhatia, G., Abdul-Mageed, M., et al. : Beyond English: Evaluating LLMs for Arabic grammatical error correction. In: Proceedings of ArabicNLP 2023, pp. 101–119 (2023)

    2023

  44. [52]

    Applied Sciences 14(5), 2074 (2024)

    Patil, R., Gudivada, V.: A review of current trends, techniques, and challenges in Large Language Models (LLMs). Applied Sciences 14(5), 2074 (2024)

    2024

  45. [53]

    Edwards, A., Camacho-Collados, J.: Language models for text classification: Is in-context learning enough? arXiv preprint arXiv:2403.17661 (2024)

    2024 arXiv

  46. [54]

    In: Kochmar, E., Bexte, M., Burstein, J., Horbach, A., Laarmann-Quante, R., Tack, A., Yaneva, V., Yuan, Z

    Chamieh, I., Zesch, T., Giebermann, K.: LLMs in short answer scoring: Lim- itations and promise of zero-shot and few-shot approaches. In: Kochmar, E., Bexte, M., Burstein, J., Horbach, A., Laarmann-Quante, R., Tack, A., Yaneva, V., Yuan, Z. (eds.) Proceedings of the 19th Works...

    2024

  47. [55]

    arXiv preprint arXiv:2307.03042 (2023)

    Gema, A., Daines, L., Minervini, P., Alex, B.: Parameter-efficient fine-tuning of Llama for the clinical domain. arXiv preprint arXiv:2307.03042 (2023)

    2023 arXiv

  48. [56]

    arXiv preprint arXiv:2310.01208 (2023)

    Li, Z., Li, X., Liu, Y., Xie, H., Li, J., Wang, F.-l., Li, Q., Zhong, X.: Label supervised Llama finetuning. arXiv preprint arXiv:2310.01208 (2023)

    2023 arXiv

  49. [57]

    https://github.com/openlm-research/open_llama

    Geng, X., Liu, H.: OpenLLaMA: An Open Reproduction of LLaMA (2023). https://github.com/openlm-research/open_llama

    2023

  50. [58]

    : ChatGPT: Jack of all trades, master of none

    Kocoń, J., Cichecki, I., Kaszyca, O., Kochanek, M., Szydło, D., Baran, J., Bielaniewicz, J., Gruza, M., Janz, A., Kanclerz, K., et al. : ChatGPT: Jack of all trades, master of none. Information Fusion 99, 101861 (2023) 24 Appendix A Tables T able A1: Performance of Models in P...

    2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.