{"id":"1ece5265-3e39-440e-98de-b9816b60065a","arxiv_id":"2504.15544","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A 187M-parameter Japanese ModernBERT was trained on the public llm-jp corpus with 8192-token context and released with code, showing modest downstream performance but useful checkpoint-level analyses.","lead":"The authors trained and released a Japanese ModernBERT model with an 8192-token context window, along with training and evaluation code. The model does not beat existing Japanese BERT baselines on JGLUE, but the paper documents training dynamics, long-context effects, and sentence embedding behavior.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Line-by-line training means Stage 2 may never teach long-range dependencies; the pseudo-perplexity drop is not cleanly attributable to context expansion.","rationale":"The paper is a reproducibility-focused resource paper, and its central empirical claim is about long-context improvement from Stage 2. For that claim to hold, Stage 2 must actually expose the model to coherent sequences approaching 8192 tokens. The line_by_line=True setting makes this uncertain, and the paper provides no training-length distribution, so the reader's weakest assumption is well placed. This concern is load-bearing because it affects the interpretation of Figure 2 and Section 3.5: without evidence of long training sequences, the pseudo-perplexity decrease could be explained by additional training and by evaluating Stage 1 outside its 1024-token training regime. The paper itself flags related limitations, including undertraining on long sequences. However, this does not invalidate the paper's primary value as a released and documented resource with honest reporting of downstream performance, so the CONDITIONAL verdict should remain unchanged. I would ask the authors to add the training-length distribution and, ideally, a same-step 1024-token control before making the causal long-context claim.","tokens_in":8569,"tokens_out":5913,"duration_ms":60092,"concrete_test":"Instrument the released training code to record, for a random sample of Stage 2 batches, the number of non-padding tokens per sequence before masking, and report percentiles. If the 95th percentile of non-padding tokens is below 1024, the effective context in Stage 2 is not long and the Section 3.5 improvement should be attributed to continued pretraining rather than context-length expansion; if it is above 1024, the line-by-line concern is resolved and the causal attribution becomes plausible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.5's central claim requires that Stage 2 actually trained the model on coherent sequences approaching 8192 tokens. Table 1 sets Line by line to True. In the Hugging Face run_mlm_no_trainer script used (Section 2.4), line_by_line=True treats each line of the text dataset as its own sequence, truncating or padding to the maximum length; padding tokens receive loss label -100 and are masked in attention. If the lines in llm-jp-corpus v4 are typically short, and the paper never reports their length distribution, then almost every Stage 2 example has an effective context length well below 1024, the same as in Stage 1. The pseudo-perplexity comparison would then reflect continued pretraining on additional tokens plus the fact that Stage 1 is evaluated out of distribution at positions beyond 1024, rather than a learned 8192-token context. The paper's own observation that long sequences are undertrained and that Stage 2 training 'did not explicitly account for the distribution of sentence lengths in the dataset' (Section 3.5) corroborates this concern. A causal claim requires either a report of the training sequence-length distribution or a control that keeps the maximum length at 1024 for the same number of additional steps.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The authors pretrain llm-jp-modernbert, a 187M-parameter ModernBERT encoder, on the Japanese subset of llm-jp-corpus v4 using a two-stage schedule: a 1024-token context for 500k steps followed by an 8192-token context for 200k steps. They evaluate the model on JGLUE downstream tasks, qualitative fill-mask examples, pseudo-perplexity over long Wikipedia sequences, and alignment/uniformity of mean-pooled sentence embeddings. They report that JGLUE performance plateaus early and does not surpass strong baselines, that pseudo-perplexity on long sequences decreases from Stage 1 to Stage 2 while still showing signs of undertraining on long contexts, and that the sentence embedding geometry closely resembles that of modernbert-ja-130m. The model, training code, and evaluation code are publicly released.","tokens_in":8774,"tokens_out":6562,"duration_ms":57024,"significance":"If the long-context claim is established, this paper provides a useful open resource for Japanese encoder pretraining: a released model, reproducible training code, and a checkpoint-level analysis of context-length expansion. The honest reporting that the model does not outperform existing baselines on downstream tasks is a strength, as is the release of code and checkpoints. The main scientific contribution is the pseudo-perplexity analysis of context expansion, but that claim is currently not supported by the reported experimental configuration. The fill-mask evaluation is illustrative rather than quantitative.","major_comments":[{"comment":"The central claim that the pseudo-perplexity decrease from Stage 1 to Stage 2 indicates improved performance on extended contexts as a result of context-length expansion is not supported by the reported training setup. Table 1 sets Line by line to True, which in the Hugging Face run_mlm_no_trainer script used in Section 2.4 treats each line of the corpus as an independent sequence, and the paper never reports the distribution of training line lengths. If most corpus lines are shorter than 1024 tokens, then Stage 2 does not train the model on coherent sequences approaching 8192 tokens, and the observed drop at long positions could come from continued pretraining on short sequences plus the fact that Stage 1 is evaluated at positions beyond its trained context. Please report the training line-length distribution or add a control that keeps the maximum length at 1024 for the same number of additional steps before comparing pseudo-perplexity.","section":"Section 3.5 / Table 1"},{"comment":"The pseudo-perplexity evaluation uses the train split of Japanese Wikipedia, and Wikipedia is listed among the sources of llm-jp-corpus v4. Because the evaluation sequences may overlap with pretraining data, the Stage 1 to Stage 2 improvement is not cleanly attributable to context-length expansion; it may partly reflect additional exposure to the same text. Please evaluate on held-out long sequences, such as a Wikipedia validation split or a different long-document corpus, or quantify the overlap between the evaluation sequences and the training data.","section":"Section 3.5 / Section 2.2"},{"comment":"The positive fill-mask claim in the abstract is supported only by six hand-picked examples shown in Table 3, with no quantitative metric, no sample size, and no comparison of the same items across baselines. This is insufficient for the stated claim that the model achieves good results on fill-mask test evaluations. Please report a quantitative fill-mask evaluation, for example the MLM accuracy already computed on a validation set, or a set of cloze items with baseline scores.","section":"Section 3.4 / Table 3"}],"minor_comments":[{"comment":"The sentence beginning 'which were selected based on and RoBERTa' is incomplete; it should likely read 'based on BERT and RoBERTa'.","section":"Section 2.3"},{"comment":"The sentence 'but It is useful for directly measuring the model's performance' has a capitalization error: 'It' should be 'it'.","section":"Section 3.4"},{"comment":"The description of 'Line by line' as 'whether to discard the part exceeding maximum sequence length' is ambiguous; please state that line-by-line tokenization creates one sequence per line and truncates longer lines.","section":"Table 1 caption"},{"comment":"The validation set is described as the Japanese validation subset of llm-jp-corpus-v3, while training uses llm-jp-corpus v4; please clarify whether and how this mismatch affects the reported validation curves.","section":"Section 3.2"},{"comment":"Figure 5 shows a sequence-length distribution, but the caption does not state whether this is the 2,000 sampled sequences or the source corpus; please clarify.","section":"Appendix A"},{"comment":"The phrase 'the alignment scores the majority of pairs decrease' is awkward and should be reworded.","section":"Section 3.6"}],"recommendation":"major_revision","confidential_remarks":"The long-context claim is the most interesting part of the paper, but the line-by-line training setting makes it currently unverified. The stress-test concern about line_by_line=True is legitimate and should be resolved with a reported training length distribution and a 1024-length control. The fill-mask evaluation should also be made quantitative. If these points are addressed, the paper could be acceptable; the current evidence is insufficient for the central causal claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a useful resource paper, not a strong research claim. It trains and releases llm-jp-modernbert, a 187M ModernBERT on the public llm-jp-corpus v4, with two-stage 1024→8192 context, and ships training and evaluation code. That is a real contribution for anyone wanting an open Japanese encoder or a reproducible pretraining setup. The authors are honest: they report JGLUE numbers below existing baselines, and they flag the model's undertraining on long sequences.\n\nThe main positive is the release itself and the checkpoint-level analyses. The alignment/uniformity results are not new but they are clearly presented and fit with what modernbert-ja-130m shows. The MIRACL-based retrieval task appendix is a sensible sanity check.\n\nThe soft spots are about the paper's headline claim. The paper argues that pseudo-perplexity on long sequences improves from Stage 1 to Stage 2, and attributes this to the context expansion. Two things undercut that. First, Table 1 says line_by_line=True. In the HF script used, each line is its own sequence, padded/truncated to 8192. If the corpus lines are mostly short—and the paper never reports the training length distribution—the model almost never sees a coherent 8192-token sequence. Then the Stage 2 pseudo-perplexity gain mostly reflects continued training on more tokens, plus the fact that Stage 1 is being evaluated at positions beyond its 1024 training window. Second, the pseudo-perplexity evaluation samples from Wikipedia, and Wikipedia is in the training corpus, so the drop could simply be memorization of the evaluation distribution. The paper's own admission that Stage 2 'did not explicitly account for the distribution of sentence lengths' is consistent with this. This is not a fatal flaw for the resource value, but it means the title's 'long context length' claim should be reworded or better supported.\n\nOther smaller issues: the fill-mask evaluation is six hand-picked examples—illustrative, not a measurement. No error bars on the pseudo-perplexity or alignment numbers. These are minor.\n\nWho should read it? Anyone working on Japanese encoders or reproducible long-context BERT pretraining. The released model and code have practical value even if the analysis is incremental.\n\nFor peer review: I'd send it out. It is a legitimate resource paper with honest reporting. But I would ask the authors to either report the actual sequence-length distribution of the training data, run a control that keeps the max length at 1024 for the same number of extra steps, or at minimum soften the long-context claim and explicitly discuss the line-by-line confound. With that, it would be a solid contribution.","headline":"A honest resource paper: new Japanese ModernBERT with released code, but the long-context training claim is not supported because line-by-line data likely means few long coherent sequences and the evaluation overlaps with training data.","tokens_in":9312,"tokens_out":2717,"would_cite":true,"duration_ms":24010,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper trains a 187M-parameter Japanese ModernBERT encoder in two stages (1024 then 8192 tokens) and claims the expansion lowers pseudo-perplexity on long sequences, while acknowledging it trails baselines on downstream tasks.","keywords":["Japanese language model","encoder-only transformer","ModernBERT","long-context pretraining","pseudo-perplexity","sentence embeddings","masked language modeling","two-stage training"],"falsifier":"Measure the distribution of non-padding tokens in the released Stage 2 batches: if the vast majority of examples contain fewer than a few hundred real tokens, then the pseudo-perplexity drop from Stage 1 to Stage 2 cannot be attributed to learning long-range dependencies. A complementary check is to evaluate the final checkpoint on a task that requires a distant mention within one 8192-token window and compare it against the same inputs truncated to 1024 tokens; identical scores would refute the long-context claim.","tokens_in":8322,"feed_emoji":"🤖","tokens_out":10559,"duration_ms":81027,"temperature":0.7,"pith_summary":"The paper trains llm-jp-modernbert, a 187M-parameter ModernBERT encoder, on a large public Japanese corpus using a two-stage schedule: 1024-token windows first, then 8192-token windows. Its central claim is that the context expansion in Stage 2 improves long-sequence modeling, evidenced by lower pseudo-perplexity on long Wikipedia sequences compared with the Stage 1 checkpoint. The paper is upfront that the model does not beat existing baselines on JGLUE downstream tasks, though it answers fill-mask probes well and its sentence-embedding trajectories mirror those of a same-architecture comparison model. By releasing the model, training code, and evaluation code, the authors turn the recipe and the diagnostic into something another team can reuse or test.","feed_headline":"Japanese BERT at 8192 tokens lowers long-sequence perplexity","feed_subtitle":"A two-stage schedule on a public corpus makes 8192-token encoder training reproducible, with long-context gains visible in…","key_machinery":"The load-bearing objects are the two-stage schedule and the pseudo-perplexity diagnostic. Stage 1 runs 500k steps at 1024 tokens; Stage 2 runs 200k steps at 8192 tokens, with the same 30% MLM mask rate and no next-sentence objective. Pseudo-perplexity is computed as $\\exp(\\frac{1}{n}\\sum_i \\ell_i)$ over 100 sampled masked positions per sequence, on 2,000 Wikipedia sequences stratified into four length bins, following the evaluation method introduced by Breton et al. (2025). The architecture is ModernBERT-base's encoder with RoPE, alternating local-global attention, and FlashAttention; all training examples are fed line-by-line and padded to the maximum length.","core_discovery":"On the paper's own terms, the discovery is that a ModernBERT encoder pretrained on an open Japanese corpus can be extended from 1024 to 8192 tokens in a second stage, and that this extension measurably lowers pseudo-perplexity on sequences beyond the first stage's window. The improvement is partial rather than complete: the final checkpoints still show rising pseudo-perplexity with sequence length and remain above a same-architecture model trained on an in-house bilingual corpus, which the authors read as undertraining on long sequences. The paper also reports two secondary observations: downstream JGLUE accuracy plateaus after roughly 50k steps even while MLM loss keeps dropping, and mean-pooled sentence embeddings drift toward anisotropy in a way that tracks architecture rather than corpus.","pith_inferences":["If line-by-line batching means most Stage 2 examples contain only a few hundred real tokens, the 8192-token window functions as a capacity ceiling rather than a learned skill; inspecting the real-token distribution in the released batches would settle this.","Pseudo-perplexity sampled at 100 random positions per sequence may hide position-dependent effects; reporting it by token position would reveal whether long-range signal is genuinely learned or position-agnostic.","A direct extension the authors do not run: train Stage 2 at 4096 and 16384 tokens on the same corpus and compare pseudo-perplexity curves to locate where the benefit of longer context saturates.","The fill-mask successes are anecdotal, so a quantitative benchmark with many masked templates would test whether the released model's lexical knowledge generalizes beyond the shown examples."],"forward_implications":["A second pretraining stage at 8192 tokens is a workable recipe for encoder pretraining: the released model, corpus, and code let others reproduce long-context Japanese BERT training without proprietary data.","Long-context gains in this setup appear in pseudo-perplexity rather than in JGLUE scores, so claims about context length should include length-stratified diagnostics rather than downstream accuracy alone.","Stage 2 improves long-sequence pseudo-perplexity but still leaves the model behind a same-architecture model trained on in-house bilingual data, indicating that corpus composition and Stage 2 duration matter as much as the window size.","Mean-pooled sentence embeddings drift toward anisotropy early and follow similar trajectories across same-architecture models, suggesting the behavior is architecture-driven rather than corpus-driven."],"supporting_citations":[{"why":"Supplies the pseudo-perplexity method used to measure long-context performance.","marker":"(Breton et al., 2025)"},{"why":"Defines the ModernBERT architecture and training settings (30% mask rate, no next-sentence objective) that the model follows.","marker":"(Warner et al., 2024)"},{"why":"Establishes BERT-style MLM pretraining that this encoder extends.","marker":"(Devlin et al., 2019)"},{"why":"Provides the large-scale Japanese corpus used for training.","marker":"(LLM-jp, 2024)"},{"why":"Provides the same-architecture comparison model used as the long-context and embedding baseline.","marker":"(Tsukagoshi et al., 2025)"},{"why":"Supplies JGLUE, the downstream benchmark used to evaluate fine-tuned models.","marker":"(Kurihara et al., 2022)"},{"why":"Supplies RoPE, the positional encoding that makes the 8192-token window usable.","marker":"(Su et al., 2023)"},{"why":"Supplies FlashAttention, which makes long-context training computationally feasible.","marker":"(Dao et al., 2022)"}],"fun_headline_variants":["Japanese ModernBERT: 8192-token context cuts pseudo-perplexity","Long-context Japanese BERT: reproducible 8192-token training","Pseudo-perplexity drops with 8192-token Japanese encoder","Two-stage pretraining extends Japanese BERT to 8192 tokens"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 8192-token Stage 2 actually trains on long, coherent text, but the line-by-line=True setting means each example is one corpus line padded to the maximum length, so if real Japanese lines are short, the model mostly sees short content with padding and the long-context improvement may be an artifact of capacity rather than learned long-range behavior.","fun_headline_variants_meta":{"raw":{"variants":["Japanese ModernBERT: 8192-token context cuts pseudo-perplexity","Long-context Japanese BERT: reproducible 8192-token training","Pseudo-perplexity drops with 8192-token Japanese encoder","Two-stage pretraining extends Japanese BERT to 8192 tokens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000738,"raw_usage":{"total_tokens":3261,"prompt_tokens":876,"completion_tokens":2385,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":2306}},"tokens_in":492,"tokens_out":2385,"duration_ms":16808,"temperature":1.0,"reasoning_tokens":2306,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:23:41.695048+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the distribution of non-padding tokens in the released Stage 2 batches: if the vast majority of examples contain fewer than a few hundred real tokens, then the pseudo-perplexity drop from Stage 1 to Stage 2 cannot be attributed to learning long-range dependencies. A complementary check is to evaluate the final checkpoint on a task that requires a distant mention within one 8192-token window and compare it against the same inputs truncated to 1024 tokens; identical scores would refute the long-context claim.","supporting_citations":[],"review_version":1}