TiMoE: Time-Aware Mixture of Language Experts

Dongyang Fan; Martin Jaggi; Robin Faro; Tamar Alphaidze

REVIEW 4 major objections 5 minor 10 references

TiMoE: Time-Aware Mixture of Language Experts

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Six time-sliced GPT experts cut future-knowledge errors by 15%.

desk verdict A clean, modular time-sliced expert idea worth taking seriously, but the headline 15% error reduction is not in the paper's own numbers—Figure 4 implies roughly 2% relative improvement. read the letter →

arxiv 2508.08827 v1 pith:C5YFUMHE submitted 2025-08-12 cs.CL

Robin Faro , Dongyang Fan , Tamar Alphaidze , Martin Jaggi This is my paper · ORCID

classification cs.CL

keywords TiMoEtime-awarelanguagemodelscausalmaskingmixtureofexpertstemporalleakageTSQAtime-segmentedpretrainingknowledgechangeovertime

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a pretraining recipe that makes a language model's knowledge chronologically honest: rather than training one model on the entire web, it trains separate GPT-2-scale experts on two-year slices of a 2013-2024 corpus and, at inference, combines only experts whose training window ends no later than the query date. Because every expert is trained from scratch on its slice, this causal masking makes future leakage impossible by construction. On a new 10k-question benchmark (TSQA) whose wrong answers are tagged as past, future, or irrelevant, a co-adapted version of the mixture cuts future-knowledge errors by up to 15% and matches or exceeds the best single-period expert, while remaining within about 5.6% of a unified model on general benchmarks. If the recipe holds, it offers a simple, modular path to LLMs that can be updated by adding new time experts without retraining old ones.

What carries the argument

The load-bearing mechanism is the causally masked log-probability mixture (Eq. 1): for a query at year $t_q$, the aggregated log-probability is $\log \sum_{E_k \in \mathcal{E}(t_q)} w_k(x) \exp(\log P_k(x_{t+1} \mid x_{1:t}))$, where $w_k \ge 0$, $\sum w_k = 1$, and $\mathcal{E}(t_q)$ contains only experts trained on windows $k \le t_q$. A one-layer MLP router maps the input embedding to the weights $w_k$; in the best-performing variant (CoAdapt), the router and the expert matching the training document's window are jointly updated while older experts stay frozen. The mask guarantees temporal validity; the log-probability merge preserves multi-period knowledge.

What would settle it

Re-run Stage 1 and Stage 2 with the corpus partitioned by true document creation dates (from publication-date metadata) instead of crawl dates, keeping everything else fixed. If the TSQA future-knowledge error reduction of up to 15% does not persist, the causal-masking guarantee as implemented is not the source of the reported gains.

Watch

Extended reading notes

Core claim

The paper's central claim is that pre-training separate GPT-2 models on disjoint two-year slices of a 2013-2024 web corpus and aggregating their next-token log-probabilities with a causal mask—excluding any expert trained after the query's timestamp—produces a language model that is chronologically grounded by construction. On its own 10,063-question TSQA benchmark, where wrong answers are labeled past, future, or irrelevant, the co-adapted variant (TiMoE-CoAdapt) matches or exceeds the best single-period expert, beats three 3B-parameter frontier models by about 3 percentage points, and cuts future-knowledge errors by up to 15%. The same model trails a standard full-corpus GPT-2 by 5.6% on a

Load-bearing premise

The causal-masking guarantee treats a document's crawl date as the time of its knowledge, so the load-bearing premise is that a document crawled in a two-year window is representative of that window's knowledge state; if a bin is full of older or evergreen content, the 'time-specific' experts are not truly time-specific and the TSQA gains could stem from something other than temporal grounding.

Editorial extensions

If this is right

A query at any date is answered using only knowledge available at that date, making future leakage impossible by construction rather than by post-hoc filtering.
The model can be kept current by training only a new time expert on the latest slice, leaving earlier experts frozen, so there is no catastrophic forgetting and no full retraining.
Comparing expert embeddings or predictions reveals how word meanings and facts shift across periods, turning the architecture into a tool for diachronic language analysis (illustrated with "coronavirus" versus "lockdown" distances).
Time-awareness costs a bounded amount of general performance: CoAdapt trails the unified model by 5.6% on average while beating the best single expert, so the approach does not sacrifice general capability to gain temporal honesty.
TSQA gives a reusable, fine-grained diagnostic: because distractors are labeled past/future/irrelevant, future-error reductions can be measured directly rather than inferred from aggregate accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves implicit that the same causal-masking recipe could apply to continual learning and fact-update pipelines: instead of editing or merging weights, one appends a new time expert and routes by validity interval, making rollback to earlier knowledge states trivial.
Because inference cost grows with the number of past experts, the method as stated will not scale to long histories without top-k expert selection or hierarchical bins; this practical extension is not tested in the paper.
The reported TSQA advantage may be sensitive to the benchmark's construction from just three timeline datasets; re-running TSQA generation from other dated fact sources would reveal whether the 15% improvement is a property of the method or of the benchmark's label distribution.
The 5.6% general-task gap might shrink if experts shared a common backbone (e.g., time-specific adapters over a single base model) while keeping the causal mask; this hybrid is a direct, testable extension of the design.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core idea here is good: train separate GPT-2 experts on disjoint two-year web slices, then at inference mask out any expert trained on data newer than the query timestamp and merge the remaining log-probs. That is simple, modular, and has real appeal for keeping models chronologically grounded without retraining everything. The TSQA benchmark, with distractors labeled past/future/irrelevant, is also a genuinely useful diagnostic for temporal hallucination. Credit where due: the method is clearly described, the causal masking is sound, and the authors are open about the crawl-timestamp limitation in Section 6. Code and data are released.

The problem is the central quantitative claim. The abstract, conclusion, and contribution list all say TiMoE-CoAdapt "cuts future-knowledge errors by up to 15%." The only evidence on error types is Figure 4, where CoAdapt picks a future distractor in 34.5% of cases versus GPT2-Full's 35.3% — a relative reduction of about 2.3%, not 15%. No per-year or per-subset breakdown supports the larger number. This is not a minor quibble; the 15% figure is the paper's main empirical selling point, and it is contradicted by the paper's own figure. The reader's conditional verdict flagged this, and the stress-test note is right.

Other soft spots: the general-task comparison pits a 2.88B-parameter system (six 480M experts) against a single 480M GPT2-Full, and the 5.6% gap is called the "curse of time-awareness" without acknowledging the parameter disparity. No error bars. No time-aware baselines like Dhingra et al.'s timestamp conditioning or Rottger & Pierrehumbert's adapters, so it is unclear how much of the TSQA gain comes from the causal masking versus simple ensembling of six models. The crawl-timestamp issue is real but acknowledged; it is a data-quality caveat, not a load-bearing flaw.

Bottom line: the architecture is worth pursuing and the benchmark is a contribution. But as written, the paper overstates its main result. It deserves a serious referee, because the idea and the benchmark are useful, but the authors need to recompute the effect size, add matched-parameter baselines, and ideally report variance. I would send it to review with a request for major revision rather than desk reject it.

Referee Report

4 major / 5 minor

Summary. This paper proposes a time-segmented pretraining pipeline for language models. Six GPT-2-medium (480M-parameter) experts are pretrained from scratch on disjoint two-year slices of a 100B-token sample of FineWeb-Edu, with bins defined by document crawl timestamps (2013–2024). At inference, TiMoE masks any expert whose training window ends after the query timestamp and combines the log-probabilities of the remaining experts with either uniform weights (Avg), a learned router (LearnedAvg), or a router jointly co-adapted with the current time-slice expert (CoAdapt). The paper also introduces TSQA, a 10,063-question multiple-choice benchmark with distractors labeled past/future/irrelevant. Experiments on eight standard NLP tasks and TSQA report that CoAdapt matches or slightly exceeds the best single-period expert, achieves a modest TSQA accuracy gain over a GPT2-Full baseline (30.2% vs 28.7%), and the abstract and conclusion claim an 'up to 15%' reduction in future-knowledge errors. The paper also demonstrates a cross-temporal semantic-change analysis using expert embeddings.

Significance. The proposal is a clean, modular alternative to timestamp-conditioned pretraining. If the results hold, the causal-masking aggregation is an elegant way to provide a formal no-future-leakage guarantee while preserving multi-period knowledge, and the release of TSQA with error-type annotations is a useful diagnostic resource. Strengths: the approach is from-scratch and the code is open-sourced; the TSQA accuracy gain, while small, is positive and appears consistent across years (Figure 5); the comparison to 3B-parameter baselines at 480M per expert is interesting. However, the headline quantitative claim (15% future-error reduction) is inconsistent with the paper's own Figure 4, and the general-task comparison conflates model capacity (six experts vs one) with temporal segmentation. The core mechanism is plausible, but the evidence as reported is weaker than the abstract states.

major comments (4)

[Abstract, §4.4, Fig. 4, §5] The central claim that TiMoE-CoAdapt 'cuts future-knowledge errors by up to 15%' is not supported by the only reported error-type data. In Fig. 4, the future-distractor rate is 35.3% for GPT2-Full and 34.5% for TiMoE-CoAdapt, a relative reduction of about 2.3%, not 15%; combining past+future errors gives 66.5% vs 65.7%, an even smaller relative change. No per-year or per-subset table is provided that would substantiate a 15% figure. Please report the supporting breakdown (e.g., per-year future-error rates, or the exact subset that yields 15%) or revise the abstract and conclusion to match the reported aggregate numbers.
[§4.3, Table 1] The prose says TiMoE-CoAdapt 'consistently outperforms both the average and the best single-expert baselines across most tasks.' Table 1 shows CoAdapt below Year(max) on MMLU (0.282 vs 0.286), SocialIQA (0.387 vs 0.391), and Winogrande (0.513 vs 0.519). The accurate claim is 'matches or exceeds on several tasks,' not 'consistently outperforms.' Also, because CoAdapt is an ensemble of six 480M-parameter experts (2.88B total), the 5.6% gap relative to the single GPT2-Full model mixes model capacity with temporal segmentation; it should not be presented as a pure 'cost of time-awareness' without an iso-parameter control or explicit discussion.
[§3.1, §6, §4.5] The temporal bins are formed from crawl timestamps, not creation dates. As acknowledged in §6, each bin can contain older content. This limits the 'time-specific expert' narrative and the semantic-shift analysis in §4.5: the 2019–2020 expert may partly train on documents written earlier, so the coronavirus embedding distances are contaminated by bin impurity. The causal mask is conservative and does prevent future leakage, but the stronger claim that each expert represents a 'well-defined temporal context' (Section 1) needs either a creation-date-filtered analysis or a quantitative measure of bin purity.
[§4.2, §4.4] TSQA is a new benchmark, and the past/future/irrelevant labels are generated by DeepSeek-V3 with no reported human validation, agreement statistics, or quality checks. Since the paper's fine-grained claims about error-type reductions depend entirely on these labels, the absence of validation is a load-bearing gap for the TSQA contribution. Please report a validation sample, agreement with the source datasets' timelines, or at least an error analysis of the labeling step.

minor comments (5)

[§4.1] Typo: 'We choose GPT21 as our base model architecture' should be 'GPT-2'.
[§4.4] The 'approximately 3%' advantage over 3B-parameter models is imprecise: TiMoE-CoAdapt is 30.2% vs Llama 27.7%, a 2.5 percentage-point gap, and the comparison uses a 2.88B-parameter ensemble. Please use percentage points consistently and note the parameter-count mismatch.
[Figures 3 and 5] No error bars or repeated-seed results are reported. Differences among TiMoE variants (e.g., Avg 29.92, LearnedAvg 30.06, CoAdapt 30.19) are small and may be within noise; please report variance or significance tests.
[Table 1] Please clarify how 'Year(max)' is computed: is it the per-task maximum accuracy over the six single-period experts? This is implied but not stated explicitly.
[§4.2] The reference to 'DeepSeekV3' should be formatted consistently (e.g., 'DeepSeek-V3') and the prompt template in the appendix is useful but should state whether the generated dictionaries were post-processed or filtered.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TiMoE's causal masking is a design property, and the empirical claims are not forced by fitted inputs or self-citations.

full rationale

The paper's derivation chain is self-contained. The time-specific experts are pretrained on disjoint crawl-timestamp bins (Section 3.1), and TiMoE's aggregation is defined by Eq. (1) with explicit routing variants (Section 3.2), none of which are fitted to TSQA. The TSQA labels are used only for evaluation; the router and experts are trained on FineWeb-Edu, not on the benchmark (Section 4.1), so there is no fitted-input-called-prediction loop. There are no self-citations or imported uniqueness theorems: the related-work section cites external prior art, and no load-bearing step reduces to a self-citation. The claim of 'strict causal validity' is a definitional property of the expert mask, not an empirical prediction, and the empirical comparisons to single-period experts and baselines are genuine measurements. The only notable issue is internal: the abstract/conclusion claim of 'up to 15%' reduction in future-knowledge errors is not reflected in the reported Figure 4 (which shows roughly a 2% relative reduction in future distractors, 34.5% vs 35.3%), and no supporting per-subset breakdown is provided. That is a correctness/overstatement concern, not circularity, because no equation or fitted parameter makes the claimed reduction equivalent to the method's inputs by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities beyond the model components and benchmark, which are directly tested. The main analytic assumptions are about data timestamps and probability mixing, listed above.

free parameters (3)

Time-slice width (2 years)
Chosen as a compromise between temporal resolution and data availability (Section 3.1); no sweep over 1 vs 2 vs 3 years is reported, so the granularity is a hand-set hyperparameter.
Stage-2 training tokens (200M for LearnedAvg, 1B for CoAdapt) = 200M / 1B
Hand-set training budgets; no sensitivity analysis reported, so the reported gains could depend on this choice.
Router architecture and learning rates = 10e-5 / 10e-4
Training hyperparameters chosen without reported tuning; central to the co-adaptation results.

assumptions (4)

domain assumption Crawl timestamps reflect the knowledge time in documents
Section 3.1 and Section 6 state the bins are by crawl date; if crawl date is not creation date, the temporal isolation is only with respect to crawl time.
standard math Log-probability mixing in Eq. (1) produces a coherent probability distribution
The weighted geometric mean of per-token distributions is a valid distribution.
domain assumption Independently trained GPT-2 experts with the same tokenizer produce comparable log-probabilities
The method assumes that raw log-probs can be mixed without calibration; no calibration is performed.
ad hoc to paper TSQA labels are accurate and the LLM-generated distractors are valid
Section 4.2 generates the benchmark with DeepSeekV3 without reported human validation; evaluation conclusions depend on label quality.

how reviews work

0 comments

Cite this review

Pith. "Pith review of TiMoE: Time-Aware Mixture of Language Experts." pith.science (2026). https://pith.science/paper/C5YFUMHE

@misc{pith2026250808827,
  author       = {Pith},
  title        = {Pith review of: TiMoE: Time-Aware Mixture of Language Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C5YFUMHE}},
  note         = {Machine review of arXiv:2508.08827}
}

read the original abstract

Large language models (LLMs) are typically trained on fixed snapshots of the web, which means that their knowledge becomes stale and their predictions risk temporal leakage: relying on information that lies in the future relative to a query. We tackle this problem by pre-training from scratch a set of GPT-style experts on disjoint two-year slices of a 2013-2024 corpus and combining them through TiMoE, a Time-aware Mixture of Language Experts. At inference time, TiMoE masks all experts whose training window ends after the query timestamp and merges the remaining log-probabilities in a shared space, guaranteeing strict causal validity while retaining the breadth of multi-period knowledge. We also release TSQA, a 10k-question benchmark whose alternatives are explicitly labelled as past, future or irrelevant, allowing fine-grained measurement of temporal hallucinations. Experiments on eight standard NLP tasks plus TSQA show that a co-adapted TiMoE variant matches or exceeds the best single-period expert and cuts future-knowledge errors by up to 15%. Our results demonstrate that modular, time-segmented pre-training paired with causal routing is a simple yet effective path toward LLMs that stay chronologically grounded without sacrificing general performance much. We open source our code at TiMoE (Github): https://github.com/epfml/TiMoE

Figures

Figures reproduced from arXiv: 2508.08827 by the authors.

**Figure 2.** An example of our TSQA benchmark. To construct this benchmark, we start from three existing datasets of temporally grounded factual knowledge: DYKNOW (Mousavi et al., 2024), TIMESENSITIVE QA (Chen et al., 2021), and TEMPORAL ALIGNMENT QA (Zhao et al., 2024). For each event, we extract a timeline of answer variants across different years. We then prompt a DeepSeekV3 (DeepSeekAI et al., 2025) model to generate a fo… view at source ↗

**Figure 3.** Accuracies of the models on our curated TSQA benchmark [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Distribution of answer types (Correct, Temporally Past, Temporally Future, Irrele [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Yearly accuracies of the models on our curated TSQA benchmark [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: Semantic Distance from word ’Coronavirus’ in years 2013-2014 vs. 2023-2024 [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

10 extracted references · 7 canonical work pages

[1]

Choose 1 correct answer for the given year, based on the available answers from the dataset

work page
[2]

Select 3 different wrong answers by considering the other possible answers from the dataset across different years

work page
[3]

If there is not enough context to choose 3 wrong answers based on the timeline, generate the necessary ones (different from the already chosen) and ensure that the wrong answers are still coherent with the available data

work page
[4]

Use this tag when there is not enough data from the dataset's timeline to identify a valid answer

If an answer is not related to the timeline for the given question, it should be tagged as 'irrelevant'. Use this tag when there is not enough data from the dataset's timeline to identify a valid answer

work page
[5]

The 'irrelevant' tag should be used if an answer does not match the timeline for the given question or when insufficient data from the timeline exists to make the answer relevant

Tag each answer with one of the following labels: 'correct', 'past', 'future', or 'irrelevant'. The 'irrelevant' tag should be used if an answer does not match the timeline for the given question or when insufficient data from the timeline exists to make the answer relevant

work page
[6]

Ensure that it is tagged correctly as 'correct', 'past', or 'future' based on its timeline

If the correct answer also appears in the timeline as 'future' or 'past', do **not** include it as a wrong answer. Ensure that it is tagged correctly as 'correct', 'past', or 'future' based on its timeline

work page
[7]

Do not provide any additional text, symbols, or explanations

Return the question along with 4 options (1 correct and 3 wrong) in the following dictionary format: { 'question': <question>, 'options': [ {'answer': <answer 1>, 'tag': <tag 1>}, {'answer': <answer 2>, 'tag': <tag 2>}, {'answer': <answer 3>, 'tag': <tag 3>}, {'answer': <answer 4>, 'tag': <tag 4>} ] } Please give only the dictionary and ensure the format ...

work page 2013
[39]

URL https://aclanthology.org/2021.eacl-main.39/. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pe...

arXiv 2021

Show all 10 references

[381]

Bojia Zi, Xianbiao Qi, Lingzhi Wang, Jianan Wang, Kam-Fai Wong, and Lei Zhang

URL https://aclanthology.org/2025.naacl-long.381/. Bojia Zi, Xianbiao Qi, Lingzhi Wang, Jianan Wang, Kam-Fai Wong, and Lei Zhang. Delta- lora: Fine-tuning high-rank parameters with the delta of low-rank matrices, 2023. URL https://arxiv.org/abs/2309.02411. 15 Published at COLM...

2025 arXiv
[2025]

URL https://arxiv.org/abs/2502.02421. 13 Published at COLM 2025 - XTempLLMs workshop Denis Paperno, Germ ´an Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fern´andez. The lambada dataset, Aug 2016. G...

2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Choose 1 correct answer for the given year, based on the available answers from the dataset

work page

[2] [2]

Select 3 different wrong answers by considering the other possible answers from the dataset across different years

work page

[3] [3]

If there is not enough context to choose 3 wrong answers based on the timeline, generate the necessary ones (different from the already chosen) and ensure that the wrong answers are still coherent with the available data

work page

[4] [4]

Use this tag when there is not enough data from the dataset's timeline to identify a valid answer

If an answer is not related to the timeline for the given question, it should be tagged as 'irrelevant'. Use this tag when there is not enough data from the dataset's timeline to identify a valid answer

work page

[5] [5]

The 'irrelevant' tag should be used if an answer does not match the timeline for the given question or when insufficient data from the timeline exists to make the answer relevant

Tag each answer with one of the following labels: 'correct', 'past', 'future', or 'irrelevant'. The 'irrelevant' tag should be used if an answer does not match the timeline for the given question or when insufficient data from the timeline exists to make the answer relevant

work page

[6] [6]

Ensure that it is tagged correctly as 'correct', 'past', or 'future' based on its timeline

If the correct answer also appears in the timeline as 'future' or 'past', do **not** include it as a wrong answer. Ensure that it is tagged correctly as 'correct', 'past', or 'future' based on its timeline

work page

[7] [7]

Do not provide any additional text, symbols, or explanations

Return the question along with 4 options (1 correct and 3 wrong) in the following dictionary format: { 'question': <question>, 'options': [ {'answer': <answer 1>, 'tag': <tag 1>}, {'answer': <answer 2>, 'tag': <tag 2>}, {'answer': <answer 3>, 'tag': <tag 3>}, {'answer': <answer 4>, 'tag': <tag 4>} ] } Please give only the dictionary and ensure the format ...

work page 2013

[8] [39]

URL https://aclanthology.org/2021.eacl-main.39/. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pe...

arXiv 2021