Pith. sign in

REVIEW 3 major objections 6 minor 17 references

llm-jp-modernbert: A ModernBERT Model Trained on a Large-Scale Japanese Corpus with Long Context Length

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper trains a 187M-parameter Japanese ModernBERT encoder in two stages (1024 then 8192 tokens) and claims the expansion lowers pseudo-perplexity on long sequences, while acknowledging it trails baselines on downstream tasks.

desk verdict A honest resource paper: new Japanese ModernBERT with released code, but the long-context training claim is not supported because line-by-line data likely means few long coherent sequences and the evaluation overlaps with training data. read the letter →

arxiv 2504.15544 v1 pith:HRI5GMRO submitted 2025-04-22 cs.CL

Issa Sugiura , Kouta Nakayama , Yusuke Oda This is my paper · ORCID
classification cs.CL
keywords Japaneselanguagemodelencoder-onlytransformerModernBERTlong-contextpretrainingpseudo-perplexitysentenceembeddingsmaskedmodelingtwo-stagetraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper trains llm-jp-modernbert, a 187M-parameter ModernBERT encoder, on a large public Japanese corpus using a two-stage schedule: 1024-token windows first, then 8192-token windows. Its central claim is that the context expansion in Stage 2 improves long-sequence modeling, evidenced by lower pseudo-perplexity on long Wikipedia sequences compared with the Stage 1 checkpoint. The paper is upfront that the model does not beat existing baselines on JGLUE downstream tasks, though it answers fill-mask probes well and its sentence-embedding trajectories mirror those of a same-architecture comparison model. By releasing the model, training code, and evaluation code, the authors turn the recipe and the diagnostic into something another team can reuse or test.

What carries the argument

The load-bearing objects are the two-stage schedule and the pseudo-perplexity diagnostic. Stage 1 runs 500k steps at 1024 tokens; Stage 2 runs 200k steps at 8192 tokens, with the same 30% MLM mask rate and no next-sentence objective. Pseudo-perplexity is computed as $\exp(\frac{1}{n}\sum_i \ell_i)$ over 100 sampled masked positions per sequence, on 2,000 Wikipedia sequences stratified into four length bins, following the evaluation method introduced by Breton et al. (2025). The architecture is ModernBERT-base's encoder with RoPE, alternating local-global attention, and FlashAttention; all training examples are fed line-by-line and padded to the maximum length.

What would settle it

Measure the distribution of non-padding tokens in the released Stage 2 batches: if the vast majority of examples contain fewer than a few hundred real tokens, then the pseudo-perplexity drop from Stage 1 to Stage 2 cannot be attributed to learning long-range dependencies. A complementary check is to evaluate the final checkpoint on a task that requires a distant mention within one 8192-token window and compare it against the same inputs truncated to 1024 tokens; identical scores would refute the long-context claim.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a ModernBERT encoder pretrained on an open Japanese corpus can be extended from 1024 to 8192 tokens in a second stage, and that this extension measurably lowers pseudo-perplexity on sequences beyond the first stage's window. The improvement is partial rather than complete: the final checkpoints still show rising pseudo-perplexity with sequence length and remain above a same-architecture model trained on an in-house bilingual corpus, which the authors read as undertraining on long sequences. The paper also reports two secondary observations: downstream JGLUE accuracy plateaus after roughly 50k steps even while MLM loss keeps dropping, and mean-pooled sentence embeddings drift toward anisotropy in a way that tracks architecture rather than corpus.

Load-bearing premise

The load-bearing premise is that the 8192-token Stage 2 actually trains on long, coherent text, but the line-by-line=True setting means each example is one corpus line padded to the maximum length, so if real Japanese lines are short, the model mostly sees short content with padding and the long-context improvement may be an artifact of capacity rather than learned long-range behavior.

Editorial extensions

If this is right

  • A second pretraining stage at 8192 tokens is a workable recipe for encoder pretraining: the released model, corpus, and code let others reproduce long-context Japanese BERT training without proprietary data.
  • Long-context gains in this setup appear in pseudo-perplexity rather than in JGLUE scores, so claims about context length should include length-stratified diagnostics rather than downstream accuracy alone.
  • Stage 2 improves long-sequence pseudo-perplexity but still leaves the model behind a same-architecture model trained on in-house bilingual data, indicating that corpus composition and Stage 2 duration matter as much as the window size.
  • Mean-pooled sentence embeddings drift toward anisotropy early and follow similar trajectories across same-architecture models, suggesting the behavior is architecture-driven rather than corpus-driven.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If line-by-line batching means most Stage 2 examples contain only a few hundred real tokens, the 8192-token window functions as a capacity ceiling rather than a learned skill; inspecting the real-token distribution in the released batches would settle this.
  • Pseudo-perplexity sampled at 100 random positions per sequence may hide position-dependent effects; reporting it by token position would reveal whether long-range signal is genuinely learned or position-agnostic.
  • A direct extension the authors do not run: train Stage 2 at 4096 and 16384 tokens on the same corpus and compare pseudo-perplexity curves to locate where the benefit of longer context saturates.
  • The fill-mask successes are anecdotal, so a quantitative benchmark with many masked templates would test whether the released model's lexical knowledge generalizes beyond the shown examples.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The authors pretrain llm-jp-modernbert, a 187M-parameter ModernBERT encoder, on the Japanese subset of llm-jp-corpus v4 using a two-stage schedule: a 1024-token context for 500k steps followed by an 8192-token context for 200k steps. They evaluate the model on JGLUE downstream tasks, qualitative fill-mask examples, pseudo-perplexity over long Wikipedia sequences, and alignment/uniformity of mean-pooled sentence embeddings. They report that JGLUE performance plateaus early and does not surpass strong baselines, that pseudo-perplexity on long sequences decreases from Stage 1 to Stage 2 while still showing signs of undertraining on long contexts, and that the sentence embedding geometry closely resembles that of modernbert-ja-130m. The model, training code, and evaluation code are publicly released.

Significance. If the long-context claim is established, this paper provides a useful open resource for Japanese encoder pretraining: a released model, reproducible training code, and a checkpoint-level analysis of context-length expansion. The honest reporting that the model does not outperform existing baselines on downstream tasks is a strength, as is the release of code and checkpoints. The main scientific contribution is the pseudo-perplexity analysis of context expansion, but that claim is currently not supported by the reported experimental configuration. The fill-mask evaluation is illustrative rather than quantitative.

major comments (3)
  1. [Section 3.5 / Table 1] The central claim that the pseudo-perplexity decrease from Stage 1 to Stage 2 indicates improved performance on extended contexts as a result of context-length expansion is not supported by the reported training setup. Table 1 sets Line by line to True, which in the Hugging Face run_mlm_no_trainer script used in Section 2.4 treats each line of the corpus as an independent sequence, and the paper never reports the distribution of training line lengths. If most corpus lines are shorter than 1024 tokens, then Stage 2 does not train the model on coherent sequences approaching 8192 tokens, and the observed drop at long positions could come from continued pretraining on short sequences plus the fact that Stage 1 is evaluated at positions beyond its trained context. Please report the training line-length distribution or add a control that keeps the maximum length at 1024 for the same number of additional steps before comparing pseudo-perplexity.
  2. [Section 3.5 / Section 2.2] The pseudo-perplexity evaluation uses the train split of Japanese Wikipedia, and Wikipedia is listed among the sources of llm-jp-corpus v4. Because the evaluation sequences may overlap with pretraining data, the Stage 1 to Stage 2 improvement is not cleanly attributable to context-length expansion; it may partly reflect additional exposure to the same text. Please evaluate on held-out long sequences, such as a Wikipedia validation split or a different long-document corpus, or quantify the overlap between the evaluation sequences and the training data.
  3. [Section 3.4 / Table 3] The positive fill-mask claim in the abstract is supported only by six hand-picked examples shown in Table 3, with no quantitative metric, no sample size, and no comparison of the same items across baselines. This is insufficient for the stated claim that the model achieves good results on fill-mask test evaluations. Please report a quantitative fill-mask evaluation, for example the MLM accuracy already computed on a validation set, or a set of cloze items with baseline scores.
minor comments (6)
  1. [Section 2.3] The sentence beginning 'which were selected based on and RoBERTa' is incomplete; it should likely read 'based on BERT and RoBERTa'.
  2. [Section 3.4] The sentence 'but It is useful for directly measuring the model's performance' has a capitalization error: 'It' should be 'it'.
  3. [Table 1 caption] The description of 'Line by line' as 'whether to discard the part exceeding maximum sequence length' is ambiguous; please state that line-by-line tokenization creates one sequence per line and truncates longer lines.
  4. [Section 3.2] The validation set is described as the Japanese validation subset of llm-jp-corpus-v3, while training uses llm-jp-corpus v4; please clarify whether and how this mismatch affects the reported validation curves.
  5. [Appendix A] Figure 5 shows a sequence-length distribution, but the caption does not state whether this is the 2,000 sampled sequences or the source corpus; please clarify.
  6. [Section 3.6] The phrase 'the alignment scores the majority of pairs decrease' is awkward and should be reworded.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports empirical comparisons of a released model against held-out evaluations, and no central claim reduces to a fitted input or to a self-citation.

full rationale

The paper's derivation chain is a training-and-evaluation report, not a mathematical derivation from assumptions to a predicted quantity. The model is pretrained on the llm-jp-corpus v4 using an MLM objective, and then evaluated on held-out tasks: JGLUE fine-tuning, fill-mask examples, pseudo-perplexity on held-out Wikipedia sequences of varying lengths, and alignment/uniformity on MIRACL and Wikipedia pairs. The pseudo-perplexity metric is defined as exp of the average MLM cross-entropy at sampled positions, so it shares the training objective family, but the values are computed on held-out sequences and checkpoints rather than being fit to a target result. The comparison between the Stage 1 and Stage 2 checkpoints is an empirical observation about those checkpoints, not a quantity forced by construction. No parameter is fitted so as to make the pseudo-perplexity decrease appear; no equation in the paper equates the claimed improvement to the training objective by definition. Citations to the llm-jp corpus, ModernBERT, NeoBERT, and ModernBERT-Ja are used for data provenance, architecture, and evaluation methodology, and none of these citations is invoked as an unverified uniqueness theorem or as a substitute for the reported measurements. The reviewer's concern about line_by_line=True is a possible confound in attributing the pseudo-perplexity improvement to genuine long-context learning, and it may warrant a validity caveat, but it is not circularity: it does not show that the reported result is equivalent to its inputs by construction or that a fitted parameter was renamed as a prediction. Therefore the paper shows no significant circularity and receives a score of 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical entities or fitted parameters. Its claims rest on standard MLM training and evaluation, with key domain assumptions about the corpus, the evaluation set, and the training format. The main unexamined assumptions are the overlap between the Wikipedia evaluation set and the training corpus, and whether line-by-line padding constitutes long-context training.

assumptions (3)
  • domain assumption The llm-jp-corpus v4 Japanese subset is a suitable and representative pretraining corpus for Japanese.
    The paper uses this corpus without an independent quality or bias assessment; this is a standard assumption for resource papers.
  • ad hoc to paper The Wikipedia subset used for pseudo-perplexity is an appropriate and independent evaluation set.
    Wikipedia is listed as one of the sources in llm-jp-corpus v4 (Section 2.2), so the evaluation set overlaps with the training distribution; the paper does not control for this.
  • ad hoc to paper Line-by-line training with padding to the max length provides a valid long-context training signal.
    With line_by_line=True (Table 1), each line is padded or truncated to the max length, so the model may not see coherent long documents; the paper treats 8192 as the context length used in training.

how reviews work

0 comments
Cite this review

Pith. "Pith review of llm-jp-modernbert: A ModernBERT Model Trained on a Large-Scale Japanese Corpus with Long Context Length." pith.science (2026). https://pith.science/paper/HRI5GMRO

@misc{pith2026250415544,
  author       = {Pith},
  title        = {Pith review of: llm-jp-modernbert: A ModernBERT Model Trained on a Large-Scale Japanese Corpus with Long Context Length},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HRI5GMRO}},
  note         = {Machine review of arXiv:2504.15544}
}
read the original abstract

Encoder-only transformer models like BERT are widely adopted as a pre-trained backbone for tasks like sentence classification and retrieval. However, pretraining of encoder models with large-scale corpora and long contexts has been relatively underexplored compared to decoder-only transformers. In this work, we present llm-jp-modernbert, a ModernBERT model trained on a publicly available, massive Japanese corpus with a context length of 8192 tokens. While our model does not surpass existing baselines on downstream tasks, it achieves good results on fill-mask test evaluations. We also analyze the effect of context length expansion through pseudo-perplexity experiments. Furthermore, we investigate sentence embeddings in detail, analyzing their transitions during training and comparing them with those from other existing models, confirming similar trends with models sharing the same architecture. To support reproducibility and foster the development of long-context BERT, we release our model, along with the training and evaluation code.

Figures

Figures reproduced from arXiv: 2504.15544 by the authors.

Figure 1
Figure 1. Training curves. zero-shot sentence retrieval task10. For MLM loss and accuracy, we use the Japanese validation subset of the llm-jp-corpus-v3, which includes a portion of Wikipedia as the validation dataset. For the zero-shot sentence retrieval task, we use the MIRACL dataset (Zhang et al., 2023), a benchmark for multilingual sentence retrieval11. To obtain sentence embeddings during training, we ap￾ply mean poolin… view at source ↗
Figure 2
Figure 2. Pseudo-Perplexity vs. Sequence Length. 3.4 Fill-Mask Test The fill-mask test is a task where words in a sentence are masked, and the model is required to predict the masked words. This task is tokenizer-dependent, but It is useful for directly measuring the model’s performance on the MLM task. In this experiment, we evaluate BERT models using fill-mask tests on various sentences [PITH_FULL_IMAGE:figures/full_fig_p0… view at source ↗
Figure 3
Figure 3. Alignment and uniformity. s1 and s2 represent [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Distribution of sentence similarities. and SAKURA internet Inc.’s “High Firepower PHY Service”. References Lola Le Breton, Quentin Fournier, Mariam El Mezouar, and Sarath Chandar. 2025. NeoBERT: A next￾generation bert. Preprint, arXiv:2502.19587. Alexis Conneau, Kartik…
Figure 5
Figure 5. Figure 5: The sequence length distribution of sentences with various sequence lengths ranging from 0 to 8192, [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 10 canonical work pages

  1. [3]

    How contextual are contextu- alized word representations? Comparing the geom- etry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 55–65. Alejandro Fuster Baggetto and Victor Fresno

    work page 2019

  2. [7]

    Preprint, arXiv:1907.11692

    RoBERTa: A robustly optimized bert pretraining approach. Preprint, arXiv:1907.11692. LLM-jp

    arXiv 1907

  3. [8]

    Preprint, arXiv:2407.03963

    LLM-jp: A cross-organizational project for the research and development of fully open japanese llms. Preprint, arXiv:2407.03963. NLP-Waseda

    arXiv

  4. [9]

    https://huggingface.co/nlp-waseda/ roberta-base-japanese

    nlp-waseda/roberta-base-japanese. https://huggingface.co/nlp-waseda/ roberta-base-japanese. Accessed: 2025- 03-29. Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf

    work page 2025

  5. [10]

    InThe Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track

    The FineWeb Datasets: Decanting the web for the finest text data at scale. InThe Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. Nils Reimers and Iryna Gurevych. 2019a. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. InProceedings of the 2019 Conference on Empirical Methods in Natural Langu...

    work page 2019

  6. [13]

    https://huggingface.co/tohoku-nlp/ bert-base-japanese-v3

    tohoku-nlp/bert-base-japanese- v3. https://huggingface.co/tohoku-nlp/ bert-base-japanese-v3. Accessed: 2025-03-

    work page 2025

  7. [14]

    Preprint, arXiv:2409.07737

    Ruri: Japanese General Text Embeddings. Preprint, arXiv:2409.07737. Nobuhiro Ueda

    arXiv

  8. [15]

    https://huggingface.co/ku-nlp/ deberta-v3-base-japanese

    ku-nlp/deberta-v3-base- japanese. https://huggingface.co/ku-nlp/ deberta-v3-base-japanese. Accessed: 2025- 03-29. AlexWang,AmanpreetSingh,JulianMichael,FelixHill, Omer Levy, and Samuel R. Bowman

    work page 2025

Show all 17 references
  1. [16]

    Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xi- aoguang Li, Qun Liu, Mehdi Rezagholizadeh, and JimmyLin.2023

    Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference.Preprint, arXiv:2412.13663. Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xi- aoguang Li, Qun Liu, Mehdi...

    2023 arXiv

  2. [17]

    B Details of Sentence Retrieval Task using MIRACL We used the Japanese subset of the MIRACL dataset (Zhang et al., 2023)

    0.506 0.334 Edit distance 0.289 0.198 Jaccard distance 0.031 0.021 A Distribution of Sequence Lengths Figure 5 shows the distribution of sequence lengths in the dataset used in Section 3.5. B Details of Sentence Retrieval Task using MIRACL We used the Japanese subset of the MI...

    2023

  3. [2019]

    BERT: Pre-training of deep bidirectional transformers for language under- standing. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171–...

    2019

  4. [2021]

    In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910

    SimCSE:Simplecontrastivelearningofsentenceem- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910. Gemma Team

    2021

  5. [2022]

    Tianyu Gao, Xingcheng Yao, and Danqi Chen

    Is anisotropy really the cause of BERT embeddings not being semantic? InFindings of the Association 7 for Computational Linguistics: EMNLP 2022 , pages 4271–4281. Tianyu Gao, Xingcheng Yao, and Danqi Chen

    2022

  6. [2023]

    Preprint, arXiv:2104.09864

    RoFormer: En- hanced transformer with rotary position embedding. Preprint, arXiv:2104.09864. Tohoku NLP

    arXiv

  7. [2024]

    arXiv preprint arXiv:2408.00118

    Gemma 2: Improving open lan- guage models at a practical size. arXiv preprint arXiv:2408.00118. PengchengHe,XiaodongLiu,JianfengGao,andWeizhu Chen

    arXiv

  8. [2025]

    Preprint, arXiv:2502.19587

    NeoBERT: A next- generation bert. Preprint, arXiv:2502.19587. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov

    arXiv

  9. [3992]

    Nils Reimers and Iryna Gurevych. 2019b. Sentence- BERT: Sentence embeddings using siamese bert- networks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu

    2019

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.