Pith. sign in

REVIEW 3 major objections 5 minor 48 references

chDzDT: Word-level morphology-aware language model for Algerian social media text

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 4.7M-parameter character-level transformer, trained word-by-word, predicts grammatical features of Arabic, English, and French verbs more accurately than subword models tens of times larger—on frozen embeddings.

desk verdict Solid evidence that isolated-word character-level pretraining beats sentence-level PLMs on morphological tagging, but the Algerian-tailoring claim outruns the evidence—direct dialect results trail DziriBERT. read the letter →

arxiv 2509.01772 v1 pith:YRDE4FMK submitted 2025-09-01 cs.CL cs.AI

Abdelkrime Aries This is my paper · ORCID
classification cs.CLcs.AI
keywords AlgeriandialectArabizicharacter-levellanguagemodelpre-trainedmodelsmorphologylow-resourcelanguagesmaskedmodelingmulti-labelclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a transformer trained on single words, one character at a time, can learn enough morphology to outperform much larger subword- and sentence-level models on morphology-heavy tasks. The model, chDzDT, is pre-trained on isolated words with character-level masked language modeling plus a multi-label head that predicts which of five corpora a word comes from. On frozen-embedding morphological tagging, its largest 4.7M-parameter variant beats AraBERT, BERT, FlauBERT, DziriBERT, and CANINE in Arabic, English, and French. If the central claim holds, character-level word encoding is a viable and parameter-cheap path for morphologically rich, low-resource dialects, and could supplement subword models as an out-of-vocabulary-robust preprocessing layer.

What carries the argument

chDzDT is a BERT-like encoder whose token vocabulary is characters rather than subwords. Each word is fed as a fixed-length character sequence of up to 20 characters, with special single-character tokens serving the roles of [CLS], [PAD], [MASK], and [SEP]. The model jointly optimizes two objectives: character-level masked language modeling over masked character positions, and a sigmoid multi-label classification head on the word-level representation predicting corpus-provenance labels (Arabic, Berber, Algerian, English, French). This dual objective is what lets the model encode both orthographic regularities and coarse linguistic-origin signals without ever seeing a full sentence.

What would settle it

Build a gold-standard morphological tagging benchmark from Algerian YouTube comments in Arabic script and Arabizi, following the same verb-feature annotation as UniMorph, and run chDzDT frozen and fine-tuned against DziriBERT and CANINE. If chDzDT does not match or beat those baselines on this directly Algerian data, the central advertised benefit fails its most direct test.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a character-level transformer trained on isolated words—with no sentence context—captures enough morphological structure that a shallow decoder on frozen embeddings recovers verbal grammatical features more accurately than large contextual language models trained on billions of tokens. Across Arabic, English, and French verb morphological tagging, chDzDT_5x4x128 achieves the highest accuracy on every tested grammatical feature, with overall scores of 0.9453, 0.9545, and 0.9117, respectively, while using roughly 4.7M parameters against 110–127M for the baselines. Fine-tuning improves these scores further for Arabic and French. The paper also reports stron

Load-bearing premise

That morphological regularities learned from standard Arabic, English, and French transfer to the unstandardized Algerian dialect and Arabizi; all the strong morphology results are measured on standard languages, and the paper's only direct Algerian evidence is much weaker.

Editorial extensions

If this is right

  • Morphological tagging can be done accurately with compact character-level word encoders, without sentence context or subword tokenizers.
  • The 4.7M-parameter size is roughly 25 times smaller than standard PLM baselines, suggesting morphology features can be obtained cheaply enough for on-device or resource-limited settings.
  • Character-level embeddings are resilient to obfuscation and spelling variation, a property directly relevant to noisy social-media text and Arabizi.
  • The multi-label corpus-provenance objective may serve as a general pre-training recipe for multilingual, multi-script, code-switched dialects.
  • The paper's own corollary is that chDzDT-like encoders are best used as a morphology-first component combined with sentence-level syntactic or semantic encoders.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not directly test the advertised Algerian-dialect benefit on Arabic-script Algerian morphology; an obvious extension is building a gold Algerian/Arabizi morphological tagging benchmark and running the same frozen and fine-tuned comparison.
  • The corpus-provenance multi-label head is never ablated: a clean experiment removing that head would show whether the reported morphological gains come from the auxiliary supervision or from character-level masked language modeling alone.
  • The smallest 32-dimensional variant is often competitive with larger ones on morphology tasks, suggesting that embedding width, not depth or head count, is the main capacity axis worth scaling further.
  • If the standard-to-dialect transfer assumption holds, the same character-level isolated-word recipe should transfer to other under-resourced and code-switched Maghrebi varieties; testing on Moroccan or Tunisian Arabizi would clarify the boundary of the claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces chDzDT, a character-level BERT-style transformer pre-trained on isolated words from a multilingual corpus (Algerian YouTube comments, Arabic/French/English/Kabyle Wikipedia, and Tatoeba) using two objectives: character-level masked language modeling and multi-label classification of the word's language-provenance labels (AR, BER, DZ, EN, FR). The manuscript contributes a morphological analysis of Algerian YouTube comments, a constructed word-level lexicon dataset, three model variants, and an extensive evaluation covering intrinsic tasks (morphological consistency, orthographic-noise robustness, morphemic probing, compositional vector arithmetic, semantic-similarity alignment) and downstream tasks (morphological tagging, PoS tagging, sentiment analysis), with a further ablation study. The headline result is that the largest variant, chDzDT_5x4x128 (4.7M parameters), achieves the highest frozen-embedding accuracy on Arabic, English, and French morphological tagging (Tables 15–17), outperforming much larger subword and character-level baselines. The paper also reports negative results honestly, including near-zero semantic-similarity correlation. Direct Algerian/Arabizi evidence is limited and weaker: Arabizi PoS reaches 0.5397 vs DziriBERT 0.8050, and Algerian sentiment 0.6071–0.6229 vs 0.7042–0.7167.

Significance. If the results hold, the paper makes a worthwhile empirical contribution: it demonstrates that a compact character-level word encoder can capture enough morphological structure to beat much larger subword and sentence-level character models on morphological tagging, and it provides reusable code and models. The evaluation is well grounded in external, independently authored benchmarks (Arramooz, Qutrub, MorphyNet, UniMorph, Universal Dependencies, MorphoLex, SemEval-2017, Twifil), and the frank reporting of null semantic-similarity correlation is a strength. However, the advertised central contribution—a model 'tailored for Algerian morphology'—is not directly supported by the evidence: the morphological tagging, morphemic probing, and composition results are all measured on standard Arabic, English, and French, while the direct Algerian/Arabizi downstream results are below the dedicated DziriBERT baseline. The paper's stated transfer assumption in §6 ('insights gained from standard languages are expected to generalize to the Algerian dialect') is load-bearing and untested. The contribution as currently evidenced is narrower than the abstract and conclusion claim.

major comments (3)
  1. [§6, Tables 15–20] The abstract and conclusion claim a model 'tailored for Algerian morphology' and a demonstration for 'morphologically rich, low-resource dialects,' but all the strongest morphology evidence is from standard Arabic, English, and French. The direct Algerian/Arabizi results are substantially weaker: Arabizi PoS is 0.5397 vs DziriBERT 0.8050 (Table 19), and Algerian sentiment is 0.6071–0.6229 vs 0.7042–0.7167 (Table 20). The paper explicitly relies on the assumption stated in §6 that 'insights gained from standard languages are expected to generalize to the Algerian dialect due to shared morphological patterns,' yet this transfer is never tested. This is not an internal inconsistency, but it is a scope gap in the central claim. I ask the authors to either provide direct Algerian/Arabizi morphological evidence (e.g., annotated Algerian verb paradigms or an Arabizi morphological tagging benchm
  2. [§4.3 and §5.2, Eqs. (3)–(4)] The multi-label provenance head is presented as a key architectural choice replacing NSP, yet no ablation isolates its contribution. The labels are acknowledged to be noisy—the paper itself notes that categories such as AR-EN arise from named entities and content mixing and 'warrant closer inspection'—so the utility of this supervision is not self-evident. Because the head is trained jointly with the MLM objective and shapes the shared encoder, the reader cannot tell whether the downstream gains come from character MLM alone, the provenance head, or their combination. Please add an MLM-only baseline trained on the same data and compare it on at least one intrinsic task and one downstream task (e.g., morphological tagging, §6.2.1).
  3. [§6.2.1, Tables 15–17] The headline claim that chDzDT achieves the 'highest accuracy' on morphological tagging rests on single-split results with no reported variance, confidence intervals, or significance tests, and each model is represented by its best-performing variant among CLS/token representations. Several margins are small (e.g., English overall 0.9545 vs BERT_cls 0.9451; Arabic Mood 0.9674 vs CANINE 0.9595). Without multiple seeds or at least a significance check, the claim of superiority is not robustly established, particularly for the smaller per-feature differences. I request standard-error reporting or a statement that differences are within noise, and a clarification of how 'best variant' was selected.
minor comments (5)
  1. [§6.2.3] The text says 'Figure 20 presents the accuracy of the models for each language,' but the results are in Table 20. Please correct the cross-reference.
  2. [§5.3 and §6.4] The ablation study reuses names such as chDzDT_4x4x32, but the ablation configurations (N=1..3, H=1..4, d=8..32) are a different model family from the main models (N=4/5, d=64/128). The text occasionally conflates their results (e.g., discussing 'd=32' ablation behavior in the context of the main chDzDT_4x4x32). Please rename the ablation variants or clearly separate the two families.
  3. [§6.3.2 and §6.3.3, Figures 12–13] The fine-tuning results for PoS and sentiment are reported only as figures, without numerical values. Key statements such as '+0.21 for chDzDT_4x4x32' in Arabizi PoS cannot be verified from the plot. Please include the underlying numbers in a table or appendix.
  4. [§6.1.4] The notation 'MPCNC' and 'MPADD' is visually confusing; consider spelling them out or using a clearer mnemonic. Also, the weighted multiplicative composition results are near 0.23 for all models; a one-sentence interpretation of why this strategy fails would help.
  5. [§5.1] The special-token names ('C', 'P', 'K', 'S', 'U') are introduced in prose; a small table listing token, BERT analogue, and function would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; central results are benchmarked against external resources.

full rationale

The paper's central claims are evaluated on independently authored benchmarks (Arramooz, Qutrub-generated morphology, MorphyNet, MorphoLex, UniMorph, Universal Dependencies treebanks, SemEval-2017, Twifil), and the reported gains on morphological tagging are measured against these external test sets using standard frozen-embedding and fine-tuning protocols. No fitted parameter is relabeled as a prediction: the pretraining objectives (character-level MLM and multi-label corpus-provenance classification) are defined in Equations 2-4, and downstream classifiers are trained and tested on disjoint splits. There is no load-bearing self-citation; the author cites external prior work (BERT, CANINE, DziriBERT, CAMeLBERT-DA, etc.) and does not invoke any uniqueness theorem or prior result by the same author to force the architecture. The untested transfer assumption from standard languages to Algerian dialect is a scope limitation explicitly acknowledged in Section 6, not a circular reduction. The high similarity between character-level embeddings and their orthographically noisy variants follows naturally from the character-input design, but the paper presents this as an empirical evaluation rather than as a prediction derived from the model; it is not the central claim. Overall, the derivation chain is self-contained and externally validated, so no circularity is identified.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard transformer/MLM machinery plus three domain assumptions: that provenance-based language labels are useful supervision (unablated), that morphology transfers from standard languages to Algerian dialect (untested with direct morphology data), and that word-isolated character MLM is sufficient. Several hand-chosen cutoffs (context window 20, length cap 30, evaluation tier thresholds) shape the reported numbers. No new entities are postulated; the c.h.dz/dt notation is decorative naming, not an entity.

free parameters (4)
  • Maximum word length cap = 30 characters
    Chosen in Section 4.3 to exclude unusually long forms, dropping only 17 words; affects the tail of the vocabulary, not the central claim.
  • Transformer context window = 20 characters
    Set in Section 5.1 to cover >99% of observed word lengths; a hand-chosen cutoff informed by the training data distribution.
  • Dataset tier thresholds (Full/Avg/Min) = e.g., Arabic derivations >=3 / >=15 / >=30 members; English inflections >=4 / >=6 / >=7
    Hand-chosen in Section 6.1.1; reported ARI and clustering numbers shift substantially between the Avg and Min tiers, and the Full tier is excluded from experiments.
  • Composition probe weights (alpha, beta, gamma, W) = trained on 60% of MorphoLex splits
    Fitted inside the evaluation probe in Section 6.1.4, evaluated on the held-out 40%; an internal evaluation parameter, not part of the model.
assumptions (4)
  • domain assumption Corpus-provenance language labels provide useful morphological supervision
    Sections 4.3 and 5.2: DZ means 'appears in Algerian YouTube comments', AR means 'appears in Arabic Wikipedia', etc. The paper acknowledges the labels are noisy ('we chose not to exclude such items') and never ablates the multi-label head, so its contribution to the central results is untested.
  • domain assumption Morphology learned from standard languages transfers to Algerian dialect and Arabizi
    Section 6 opening: 'insights gained from standard languages are expected to generalize to the Algerian dialect due to shared morphological patterns.' This is load-bearing for the abstract claim of tailoring to Algerian, while the direct Algerian evidence in the paper is mixed.
  • domain assumption Character-level MLM on isolated words encodes morphology without any context
    Core design premise (Sections 5.1-5.2): a 20-character window over a single word, with no sentence context, is sufficient for the model to learn morphological regularities that transfer to downstream tasks.
  • domain assumption Preprocessing choices preserve morphology
    Section 4.2: emoji deduplication to a maximum of two, character deduplication, diacritic removal, and spacing corrections are assumed not to destroy the morphological signal the model is trained to capture.

how reviews work

0 comments
Cite this review

Pith. "Pith review of chDzDT: Word-level morphology-aware language model for Algerian social media text." pith.science (2026). https://pith.science/paper/YRDE4FMK

@misc{pith2026250901772,
  author       = {Pith},
  title        = {Pith review of: chDzDT: Word-level morphology-aware language model for Algerian social media text},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YRDE4FMK}},
  note         = {Machine review of arXiv:2509.01772}
}
read the original abstract

Pre-trained language models (PLMs) have substantially advanced natural language processing by providing context-sensitive text representations. However, the Algerian dialect remains under-represented, with few dedicated models available. Processing this dialect is challenging due to its complex morphology, frequent code-switching, multiple scripts, and strong lexical influences from other languages. These characteristics complicate tokenization and reduce the effectiveness of conventional word- or subword-level approaches. To address this gap, we introduce chDzDT, a character-level pre-trained language model tailored for Algerian morphology. Unlike conventional PLMs that rely on token sequences, chDzDT is trained on isolated words. This design allows the model to encode morphological patterns robustly, without depending on token boundaries or standardized orthography. The training corpus draws from diverse sources, including YouTube comments, French, English, and Berber Wikipedia, as well as the Tatoeba project. It covers multiple scripts and linguistic varieties, resulting in a substantial pre-training workload. Our contributions are threefold: (i) a detailed morphological analysis of Algerian dialect using YouTube comments; (ii) the construction of a multilingual Algerian lexicon dataset; and (iii) the development and extensive evaluation of a character-level PLM as a morphology-focused encoder for downstream tasks. The proposed approach demonstrates the potential of character-level modeling for morphologically rich, low-resource dialects and lays a foundation for more inclusive and adaptable NLP systems.

Figures

Figures reproduced from arXiv: 2509.01772 by the authors.

Figure 1
Figure 1. outlines the preprocessing pipeline. For YouTube comments, an initial filtering step excluded region￾specific vocabulary using regular expressions, removing terms distinctive to neighboring dialects such as Moroc￾can, Tunisian, or Gulf Arabic. Although North African dialects share a large proportion of vocabulary, each also preserves unique expressions. For example, “ AK . @X” (/dab¯ a/, “ ¯ Now”) is primarily Moroc… view at source ↗
Figure 2
Figure 2. Distribution of word lengths in the training dataset. Lengths are grouped into five-character intervals, with both frequencies and percentages reported. To address the multilingual composition of the dataset, we applied a multi-label annotation scheme indicating the language(s) of origin for each word. Words from standard corpora such as Wikipedia and Tatoeba were an￾notated with language codes AR (Arabic), BER (Ber… view at source ↗
Figure 3
Figure 3. Pre-training architecture of chDzDT. Each word is tokenized into characters and processed by a BERT encoder. Special tokens are used for padding (“P”), masking (“K”), and word-level representation (“C”). The model jointly optimizes a character-level masked language modeling objective and a word-level multi-label classification objective. 5.1 Character tokenizer Since chDzDT processes one word at a time, the tokenize… view at source ↗
Figures from the paper (27 more)
Figure 4
Figure 4. Figure 4: Pre-training loss curves for the main models. The extended model chDzDT_4x4x32_40it was initialized from chDzDT_4x4x32_20it and trained for an additional 20 epochs. Its curve restarts from zero because training was resumed with a new scheduler. No further reduction in …
Figure 5
Figure 5. Figure 5: t-SNE visualization of embeddings from the compact French noisy-clusters dataset. The figure displays 15 word-variant clusters, each shown in a different color (best viewed in color). Ideally, embeddings from the same cluster should be compact and clearly separated fro…
Figure 6
Figure 6. Figure 6: Macro-averaged precision, recall, and F1-scores for morphemic probing in English and French. Models are ordered by descending F1-score. 6.1.4 Compositional vector arithmetic This experiment addresses the following research question: Is it possible to approximate a word…
Figure 7
Figure 7. Figure 7: Frobenius norms of projection matrices for English and French models. (a) Concatenation-based mapping (MPCNC) reports separate norms for prefix, root, and suffix components. (b) Additive mapping (MPADD) applies a single projection matrix to the sum of all morpheme comp…
Figure 8
Figure 8. Figure 8: PCA-compressed heatmaps of the learned mapping W = [Wp; Wr; Ws] for English, shown for the two best-performing models. Each block corresponds to prefix, root, and suffix mappings, respectively. Task. The goal of this task is to evaluate semantic similarity between pair…
Figure 9
Figure 9. Figure 9: illustrates the architectures: the subword-based Transformer (left) and the character-based model (right). All sentences were truncated to 60 words to account for the limitations of RNNs in handling long-distance dependencies; the same restriction was applied to subwor…
Figure 10
Figure 10. Figure 10: Polarity distribution for each dataset split. Positive (+1), neutral (0), and negative (-1) classes are reported with both percentages and absolute counts. Models. We evaluated our three character-based models alongside DziriBERT across all languages. AraBERT was used…
Figure 11
Figure 11. Figure 11: illustrates the architectures: the subword-based Transformer (left) and the character-based model (right). All models were trained with cross-entropy loss and the Adam optimizer. For fairness, training was limited to a maximum of 100 epochs, with early stopping trigge…
Figure 12
Figure 12. Figure 12: PoS tagging accuracy of frozen vs. fine-tuned models across four languages. Fine-tuning yields consistent improvements, particu￾larly for Arabizi. 6.3.3 Sentiment analysis In this experiment, we aim to address the following research question: How effectively do fine-t…
Figure 13
Figure 13. Figure 13: Sentiment analysis accuracy of frozen vs. fine-tuned models across four languages. 6.4 Ablation study The purpose of this ablation study is to identify which architectural components most significantly affect embed￾ding quality and downstream task performance. We syst…
Figure 14
Figure 14. Figure 14: Effect of Transformer depth (N) on training time and parameter count. Both increase with N, though training time grows more steeply. scaling d could restrict representational capacity. Arabic shows less decline, partly because derivational clusters combine both templa…
Figure 15
Figure 15. Figure 15: Effect of Transformer depth (N) on morphological consistency in Arabic, French, and English. Plots show derivational (Deriv) and inflectional (Infl) morphology on the “Avg” and “Min” datasets, using average cosine similarity between words and their root. Robustness to…
Figure 16
Figure 16. Figure 16: Effect of Transformer depth (N) on robustness to orthographic noise in Arabic, Arabizi, French, and English. Plots show: (a) cluster-based obfuscations, (b) tuple obfuscations with * substitutions, and (c) tuple obfuscations with visually or phonetically similar subst…
Figure 17
Figure 17. Figure 17: Effect of Transformer depth (N) on morphological tagging in Arabic, English, and French. (a) Overall accuracy across the three languages; (b) Arabic tags; (c) English tags; (d) French tags. In English, the curves for Sg and 3rdP overlap, as do those for NonFin and Imp…
Figure 18
Figure 18. Figure 18: Effect of Transformer depth (N) on PoS tagging in Arabic, Arabizi, English, and French. Results were obtained by training a BiGRU decoder on frozen chDzDT embeddings. Sentiment analysis [PITH_FULL_IMAGE:figures/full_fig_p032_18.png]
Figure 19
Figure 19. Figure 19: Effect of Transformer depth (N) on sentiment analysis in Arabic, Algerian dialect, English, and French. Results were obtained by training a BiGRU–dense layer decoder on frozen chDzDT embeddings [PITH_FULL_IMAGE:figures/full_fig_p032_19.png]
Figure 20
Figure 20. Figure 20: Effect of the number of attention heads (H) on morphological consistency in Arabic, French, and English. Plots show derivational (Deriv) and inflectional (Infl) morphology on the “Avg” and “Min” datasets, using average cosine similarity between words and their root. R…
Figure 21
Figure 21. Figure 21: Effect of the number of attention heads (H) on robustness to orthographic noise in Arabic, Arabizi, French, and English. Plots show: (a) cluster-based obfuscations, (b) tuple obfuscations with * substitutions, and (c) tuple obfuscations with visually or phonetically s…
Figure 22
Figure 22. Figure 22: Effect of the number of attention heads (H) on morphological tagging in Arabic, English, and French. (a) Overall accuracy across the three languages; (b) Arabic tags; (c) English tags; (d) French tags. In English, the curves for Sg and 3rdP overlap, as do those for No…
Figure 23
Figure 23. Figure 23: Effect of the number of attention heads (H) on PoS tagging in Arabic, Arabizi, English, and French. Results were obtained by training a BiGRU decoder on frozen chDzDT embeddings. Sentiment analysis [PITH_FULL_IMAGE:figures/full_fig_p034_23.png]
Figure 24
Figure 24. Figure 24: Effect of the number of attention heads (H) on sentiment analysis in Arabic, Algerian dialect, English, and French. Results were obtained by training a BiGRU–dense layer decoder on frozen chDzDT embeddings. Model efficiency [PITH_FULL_IMAGE:figures/full_fig_p035_24.png]
Figure 25
Figure 25. Figure 25: Effect of embedding size (d) on training time and parameter count. Morphological consistency [PITH_FULL_IMAGE:figures/full_fig_p035_25.png]
Figure 26
Figure 26. Figure 26: Effect of the embedding size (d) on morphological consistency in Arabic, French, and English. Plots show derivational (Deriv) and inflectional (Infl) morphology on the “Avg” and “Min” datasets, using average cosine similarity between words and their root. Robustness t…
Figure 27
Figure 27. Figure 27: Effect of the embedding size (d) on robustness to orthographic noise in Arabic, Arabizi, French, and English. Plots show: (a) cluster￾based obfuscations, (b) tuple obfuscations with * substitutions, and (c) tuple obfuscations with visually or phonetically similar subs…
Figure 28
Figure 28. Figure 28: Effect of the embedding size (d) on morphological tagging in Arabic, English, and French. (a) Overall accuracy across the three languages; (b) Arabic tags; (c) English tags; (d) French tags. In English, the curves for Sg and 3rdP overlap, as do those for NonFin and Im…
Figure 29
Figure 29. Figure 29: Effect of the embedding size (d) on PoS tagging in Arabic, Arabizi, English, and French. Results were obtained by training a BiGRU decoder on frozen chDzDT embeddings. Sentiment analysis [PITH_FULL_IMAGE:figures/full_fig_p036_29.png]
Figure 30
Figure 30. Figure 30: Effect of the embedding size (d) on sentiment analysis in Arabic, Algerian dialect, English, and French. Results were obtained by training a BiGRU–dense layer decoder on frozen chDzDT embeddings. 7 Conclusion This work addresses the limitations of existing PLMs in han…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 21 canonical work pages

  1. [1]

    Abdaoui, A., Berrimi, M., Oussalah, M., & Moussaoui, A. (2021). Dziribert: a pre-trained language model for the algerian dialect. CoRR , abs/2109.12346. https://arxiv.org/abs/2109.12346

    arXiv 2021

  2. [2]

    & Al-Zaidy, R

    AlYami, R. & Al-Zaidy, R. (2022). Weakly and semi-supervised learning for A rabic text classification using monodialectal language models. Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP) , 260--272. https://aclanthology.org/2022.wanlp-1.24

    work page 2022

  3. [3]

    Antoun, W., Baly, F., & Hajj, H. (2020). A ra BERT : Transformer-based model for A rabic language understanding. Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection , 9--15. https://aclanthology.org/2020.osact-1.2/

    work page 2020

  4. [4]

    Batsuren, K., Bella, G., & Giunchiglia, F. (2021). M orphy N et: a large multilingual database of derivational and inflectional morphology. Proceedings of the 18th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology , 39--48. https://doi.org/10.18653/v1/2021.sigmorphon-1.5

    doi:10.18653/v1/2021.sigmorphon-1.5 2021

  5. [5]

    Camacho-collados, J., Rezaee, K., Riahi, T., Ushio, A., Loureiro, D., Antypas, D., Boisson, J., Espinosa Anke, L., Liu, F., & Mart \'i nez C \'a mara, E. (2022). T weet NLP : Cutting-edge natural language processing for social media. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , 38--49. htt...

    doi:10.18653/v1/2022.emnlp-demos.5 2022

  6. [6]

    Chomsky, N. (1956). Three models for the description of language. IRE Transactions on Information Theory , 2(3), 113--124. https://doi.org/10.1109/TIT.1956.1056813

    work page arXiv 1956

  7. [7]

    H., Garrette, D., Turc, I., & Wieting, J

    Clark, J. H., Garrette, D., Turc, I., & Wieting, J. (2022). Canine: Pre-training an efficient tokenization-free encoder for language representation. Transactions of the Association for Computational Linguistics , 10, 73--91. https://doi.org/10.1162/tacl_a_00448

    doi:10.1162/tacl_a_00448 2022

  8. [8]

    Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzm \'a n, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020). Unsupervised cross-lingual representation learning at scale. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , 8440--8451. https://doi.org/10.18653/v1/2020.acl-main.747

    doi:10.18653/v1/2020.acl-main.747 2020

Show all 48 references
  1. [9]

    & Lagus, K

    Creutz, M. & Lagus, K. (2007). Unsupervised models for morpheme segmentation and morphology learning. ACM Trans. Speech Lang. Process. , 4(1). https://doi.org/10.1145/1187415.1187418

    2007

  2. [10]

    D., Nivre, J., & Zeman, D

    de Marneffe, M.-C., Manning, C. D., Nivre, J., & Zeman, D. (2021). Universal dependencies. Computational Linguistics , 47(2), 255--308. https://doi.org/10.1162/coli_a_00402

    2021 doi

  3. [11]

    Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT : Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Tech...

    2019 doi

  4. [12]

    El Boukkouri, H., Ferret, O., Lavergne, T., Noji, H., Zweigenbaum, P., & Tsujii, J. (2020). C haracter BERT : Reconciling ELM o and BERT for word-level open-vocabulary representations from characters. Proceedings of the 28th International Conference on Computational Linguistic...

    2020 doi

  5. [13]

    Finkelstein, L., Gabrilovich, E., Matias, Y., Rivlin, E., Solan, Z., Wolfman, G., & Ruppin, E. (2002). Placing search in context: The concept revisited. ACM Trans. Inf. Syst. , 20(1), 116–131. https://doi.org/10.1145/503104.503110

    2002

  6. [14]

    E., Handschuh, S., & Davis, B

    Freitas, A., Barzegar, S., Sales, J. E., Handschuh, S., & Davis, B. (2016). Semantic Relatedness for All (Languages): A Comparative Analysis of Multilingual Semantic Relatedness Using Machine Translation , 212--222. Springer International Publishing. https://doi.org/10.1007/97...

    2016 doi

  7. [15]

    & Palmer, A

    Ginn, M. & Palmer, A. (2024). Decomposing fusional morphemes with vector embeddings. Proceedings of the 21st SIGMORPHON workshop on Computational Research in Phonetics, Phonology, and Morphology , 57--66. https://doi.org/10.18653/v1/2024.sigmorphon-1.7

    2024 doi

  8. [16]

    Godin, F., Demuynck, K., Dambre, J., De Neve, W., & Demeester, T. (2018). Explaining character-aware neural networks for word-level prediction: Do they discover linguistic rules? Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , 3275--328...

    2018 doi

  9. [17]

    Gr \" o nroos, S., Virpioja, S., & Kurimo, M. (2020). Morfessor em+prune: Improved subword segmentation with expectation maximization and pruning. CoRR , abs/2003.03131. https://arxiv.org/abs/2003.03131

    2020 arXiv

  10. [18]

    & Ruder, S

    Howard, J. & Ruder, S. (2018). Universal language model fine-tuning for text classification. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 328--339. https://doi.org/10.18653/v1/P18-1031

    2018 doi

  11. [19]

    Hu, J., Ruder, S., Siddhant, A., Neubig, G., Firat, O., & Johnson, M. (2020). Xtreme: a massively multilingual multi-task benchmark for evaluating cross-lingual generalization. Proceedings of the 37th International Conference on Machine Learning , ICML'20

    2020

  12. [20]

    & Arabie, P

    Hubert, L. & Arabie, P. (1985). Comparing partitions. Journal of Classification , 2(1), 193--218. https://doi.org/10.1007/BF01908075

    1985 doi

  13. [21]

    Inoue, G., Alhafni, B., Baimukan, N., Bouamor, H., & Habash, N. (2021). The interplay of variant, size, and task type in A rabic pre-trained language models. Proceedings of the Sixth Arabic Natural Language Processing Workshop

    2021

  14. [22]

    Laggoun, A., Zakaria, C., & Sma \"i li, K. (2025). Knowledge Distillation for Efficient Algerian Dialect Processing: Training Compact BERT Models with DziriBERT . 7th International Conference on Advances in Signal Processing and Artificial Intelligence . https://hal.science/ha...

    2025

  15. [23]

    Le, H., Vial, L., Frej, J., Segonne, V., Coavoux, M., Lecouteux, B., Allauzen, A., Crabb \'e , B., Besacier, L., & Schwab, D. (2020). F lau BERT : Unsupervised language model pre-training for F rench. Proceedings of the Twelfth Language Resources and Evaluation Conference , 24...

    2020

  16. [24]

    Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). Roberta: A robustly optimized BERT pretraining approach. CoRR , abs/1907.11692. http://arxiv.org/abs/1907.11692

    2019 arXiv

  17. [25]

    Ma, W., Cui, Y., Si, C., Liu, T., Wang, S., & Hu, G. (2020). C har BERT : Character-aware pre-trained language model. Proceedings of the 28th International Conference on Computational Linguistics , 39--50. https://doi.org/10.18653/v1/2020.coling-main.4

    2020 doi

  18. [26]

    A., Macoir, J., Deacon, S

    Mailhot, H., Wilson, M. A., Macoir, J., Deacon, S. H., & S \'a nchez-Guti \'e rrez, C. (2020). Morpholex-fr: A derivational morphological database for 38,840 french words. Behavior Research Methods , 52(3), 1008--1025. https://doi.org/10.3758/s13428-019-01297-z

    2020 doi

  19. [27]

    Mashaabi, M., Al-Khalifa, S., & Al-Khalifa, H. (2024). A survey of large language models for arabic language and its dialects . https://arxiv.org/abs/2410.20238

    2024 arXiv

  20. [28]

    Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space . https://arxiv.org/abs/1301.3781

    2013 arXiv

  21. [29]

    & Al-Azani, S

    Mohamed, M. & Al-Azani, S. (2025). Enhancing A rabic NLP tasks through character-level models and data augmentation. Proceedings of the 31st International Conference on Computational Linguistics , 2744--2757. https://aclanthology.org/2025.coling-main.186/

    2025

  22. [30]

    Moudjari, L., Akli-Astouati, K., & Benamara, F. (2020). An A lgerian corpus and an annotation platform for opinion and emotion analysis. Proceedings of the Twelfth Language Resources and Evaluation Conference , 1202--1210. https://aclanthology.org/2020.lrec-1.151/

    2020

  23. [31]

    Pennington, J., Socher, R., & Manning, C. (2014). G lo V e: Global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing ( EMNLP ) , 1532--1543. https://doi.org/10.3115/v1/D14-1162

    2014 doi

  24. [32]

    Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018). Deep contextualized word representations. Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technolog...

    2018 doi

  25. [33]

    E., Ruder, S., & Smith, N

    Peters, M. E., Ruder, S., & Smith, N. A. (2019). To tune or not to tune? adapting pretrained representations to diverse tasks. Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019) , 7--14. https://doi.org/10.18653/v1/W19-4302

    2019 doi

  26. [34]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 21(140), 1--67. http://jmlr.org/papers/v21/20...

    2020

  27. [35]

    Rosenthal, S., Farra, N., & Nakov, P. (2017). S em E val-2017 task 4: Sentiment analysis in T witter. Proceedings of the 11th International Workshop on Semantic Evaluation ( S em E val-2017) , 502--518. https://doi.org/10.18653/v1/S17-2088

    2017 doi

  28. [36]

    Rousseeuw, P. J. (1987). Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics , 20, 53--65. https://doi.org/10.1016/0377-0427(87)90125-7

    1987 doi

  29. [37]

    Sadouki, F. (2023). Language contact in algeria: A sociolinguistic study. Argumentum , 19, 70--87. https://doi.org/10.34103/ARGUMENTUM/2023/5

    2023 doi

  30. [38]

    H., Mailhot, H., Deacon, S

    S \'a nchez-Guti \'e rrez, C. H., Mailhot, H., Deacon, S. H., & Wilson, M. A. (2018). Morpholex: A derivational morphological database for 70,000 english words. Behavior Research Methods , 50(4), 1568--1580. https://doi.org/10.3758/s13428-017-0981-8

    2018 doi

  31. [39]

    Sennrich, R., Haddow, B., & Birch, A. (2016). Neural machine translation of rare words with subword units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 1715--1725. https://doi.org/10.18653/v1/P16-1162

    2016 doi

  32. [40]

    Sparck Jones, K. (1972). A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation , 28(1), 11--21. https://doi.org/10.1108/eb026526

    1972 doi

  33. [41]

    Q., Ruder, S., Gupta, J

    Tay, Y., Tran, V. Q., Ruder, S., Gupta, J. P., Chung, H. W., Bahri, D., Qin, Z., Baumgartner, S., Yu, C., & Metzler, D. (2021). Charformer: Fast character transformers via gradient-based subword tokenization. CoRR , abs/2106.12672. https://arxiv.org/abs/2106.12672

    2021 arXiv

  34. [42]

    & Hinton, G

    van der Maaten, L. & Hinton, G. (2008). Visualizing data using t-sne. Journal of Machine Learning Research , 9(86), 2579--2605. http://jmlr.org/papers/v9/vandermaaten08a.html

    2008

  35. [43]

    N., Kaiser, L., & Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is all you need. CoRR , abs/1706.03762. http://arxiv.org/abs/1706.03762

    2017 arXiv

  36. [44]

    Wang, H., Li, J., Wu, H., Hovy, E., & Sun, Y. (2023). Pre-trained language models and their applications. Engineering , 25, 51--65. https://doi.org/10.1016/j.eng.2022.04.024

    2023 doi

  37. [45]

    Wu, S., Cotterell, R., & Hulden, M. (2021). Applying the transformer to character-level transduction. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume , 1901--1907. https://doi.org/10.18653/v1/2021.eacl-main.163

    2021 doi

  38. [46]

    Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., Klingner, J., Shah, A., Johnson, M., Liu, X., Kaiser, L., Gouws, S., Kato, Y., Kudo, T., Kazawa, H., Stevens, K., Kurian, G., Patil, N., Wang, W., Young, C., Smith...

    2016 arXiv

  39. [47]

    Xue, L., Barua, A., Constant, N., Al - Rfou, R., Narang, S., Kale, M., Roberts, A., & Raffel, C. (2021). Byt5: Towards a token-free future with pre-trained byte-to-byte models. CoRR , abs/2105.13626. https://arxiv.org/abs/2105.13626

    2021 arXiv

  40. [48]

    Zipf, G. K. (1950). Human behavior and the principle of least effort. cambridge, (mass.): Addison-wesley, 1949, pp. 573. Journal of Clinical Psychology , 6(3), 306--306. https://doi.org/10.1002/1097-4679(195007)6:3<306::AID-JCLP2270060331>3.0.CO;2-7

    1950 doi

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.