Pith. sign in

REVIEW 6 major objections 3 minor 46 references

Deep Learning with Pretrained 'Internal World' Layers: A Gemma 3-Based Modular Architecture for Wildfire Prediction

T0 review · 6 major / 3 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Frozen Gemma 3 mid-layers, grafted into a lightweight wildfire network, are claimed to improve prediction by acting as a reusable 'internal world' with only ~5M trainable parameters.

desk verdict The experiment is real but the paper's own table contradicts its central claim, and the promised ablation study is missing. read the letter →

arxiv 2504.18562 v1 pith:52BY3GRR submitted 2025-04-20 cs.LG cs.AI

Ayoub Jadouli , Chaker El Amrani This is my paper · ORCID
classification cs.LGcs.AI
keywords wildfirepredictioninternalworldrepresentationsfrozentransformerlayersmodulararchitecturetabulardatatransferlearningGemma3efficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that frozen middle layers of a large pretrained transformer can serve as a reusable 'internal world' for tabular scientific prediction, specifically daily wildfire occurrence in Morocco. The proposed architecture projects 276 tabular features into a single 1152-dimensional pseudo-sequence, pushes it through two frozen Gemma 3 decoder layers, and trains only small input and output networks, about 5 million of 37.7 million total parameters. The authors report that this Internal World model achieves the highest recall (0.9433) and a competitive F1 (0.8838), and they interpret ablations as showing the frozen layers consistently improve representations. Their own Table 5 places the lightweight FFN+PosEnc baseline ahead on F1 (0.8957 vs 0.8838) and AUC (0.9516 vs 0.9344), so the claim that the internal world improves accuracy hinges on which comparison is treated as decisive. If the claim held, it would offer a data-efficient recipe for reusing billion-parameter pretrained knowledge without fine-tuning.

What carries the argument

The load-bearing object is the frozen 'internal world' slice: layers 8-9 of Gemma 3-1B's decoder, held at 14.7 million parameters with no training. The paper treats these layers as a fixed feature processor that, in its telling, injects relational and contextual knowledge from multimodal pretraining into a tabular wildfire predictor. Because the input is a length-1 pseudo-sequence, the attention heads inside these layers cannot attend across time steps; the frozen blocks therefore function as a fixed nonlinear map, effectively a pretrained MLP, applied to the 1152-dimensional projected features.

What would settle it

Run the exact same pipeline but replace layers 8-9 of Gemma 3 with randomly initialized transformer layers of the same shape, kept frozen; if validation F1 and AUC stay within noise, the pretrained weights are not the source of any gain. The paper's own Table 5 supplies a sharper check: the FFN+PosEnc baseline achieves F1 0.8957 and AUC 0.9516, while the Internal World model achieves F1 0.8838 and AUC 0.9344, so under the reported numbers the comparison does not favor the internal world on the aggregate metric the paper emphasizes.

Watch

Extended reading notes

Core claim

The paper's central claim is that frozen Transformer layers consistently contribute to better representations and that grafting Gemma 3's mid-layers into a lightweight network yields improved predictive accuracy and robustness compared with standard feed-forward and convolutional baselines. The intended discovery is a modular-reuse recipe: keep the pretrained middle layers fixed, train only the input projection and output classifier, and let the frozen 'internal world' inject relational and contextual knowledge into a task with little training data. The reported numbers show an internal-world model with 0.8760 accuracy, 0.9344 AUC, 0.9433 recall, and 0.8838 F1, with recall as its strongest metric; the paper reads this as evidence that the pretrained slice is a reusable world model for scientific tabular prediction without end-to-end fine-tuning.

Load-bearing premise

The load-bearing premise is that a single 1152-dimensional vector of wildfire features, passed as a length-1 pseudo-sequence through frozen Gemma-3 decoder layers, transfers Gemma's pretrained 'internal world' knowledge to tabular data far outside its text-and-image training distribution, even though attention at sequence length 1 cannot mix information across time and the frozen layers therefore act as a fixed MLP.

Editorial extensions

If this is right

  • If the claim holds, frozen pretrained mid-layers become drop-in feature processors for small scientific datasets, giving resource-constrained groups a way to exploit billion-parameter knowledge at a fraction of the training cost.
  • A model with only about 5 million trainable parameters would be trainable on a consumer GPU in under two hours, so the recipe could scale to other environmental tabular tasks such as flood or drought risk.
  • The reported recall gain (+6.7% over the best fully trained baseline) would mean fewer missed fire events, which matters more than raw accuracy for early-warning systems.
  • The standalone FFN+PosEnc result suggests that positional structure and per-feature embeddings, not the frozen backbone, may carry most of the performance; the paper's own best-scoring model is the one without the internal world.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the frozen slice were tested against a randomly initialized transformer of the same shape (same width, depth, and frozen status), any performance drop would isolate the pretrained knowledge from the architecture itself; the paper does not report this control.
  • Because a length-1 pseudo-sequence cannot attend across tokens, the Gemma layers are mathematically a fixed MLP in this design; a fair comparison would be a fixed randomly initialized MLP of matched capacity, which would tell whether the gain comes from pretraining or from a frozen nonlinear layer.
  • The paper does not report seed variance. Repeating the comparison several times would show whether the F1 gap between Internal World and FFN+PosEnc is stable or within noise, since both are threshold-tuned and close.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 3 minor

Summary. The paper proposes a modular architecture for wildfire occurrence prediction in Morocco. Tabular features are projected through a trainable multi-branch feed-forward network, reshaped into a length-1 pseudo-sequence, passed through frozen decoder layers of Gemma 3-1B ('internal world'), and then classified by a small trainable MLP. The authors report results on a Moroccan dataset and claim that the frozen Gemma layers consistently improve representations, reduce trainable parameters, and yield improved accuracy and robustness over feed-forward and convolutional baselines. The paper includes detailed training recipes, baseline schematics, a reproducibility checklist, and links to code and data.

Significance. If the central claim were supported, the paper would demonstrate a data-efficient transfer recipe: freezing mid-layers of a large pretrained multimodal model as a generic feature processor for tabular environmental data. The manuscript is commendably explicit about training protocol, hyperparameters, and reproducibility artifacts (public GitHub repository and Kaggle dataset). However, the headline claim is contradicted by the paper's own main results table, and the promised ablation study is absent. The significance of the contribution therefore cannot be assessed from the evidence presented; as it stands, the findings do not establish that frozen Gemma layers help, and the numerical inconsistencies further prevent independent verification.

major comments (6)
  1. [Table 5 and Section IV-A] The central claim that the Internal World model gives 'improved predictive accuracy and robustness' and that 'frozen Transformer layers consistently contribute' is contradicted by the paper's own numbers. In Table 5, FFN+PosEnc achieves F1 0.8957 and AUC 0.9516, while the Internal World model achieves F1 0.8838 and AUC 0.9344; the Internal World AUC is also below plain FFN (0.9432) and CNN (0.9418). Section IV-A's attribution of the recall gain to Gemma mid-layer reuse is not supported by any controlled comparison, because the Internal World model also includes a larger trainable input/output architecture than the baselines.
  2. [Section IV-D and abstract] The abstract and Section I promise ablation studies confirming that frozen Transformer layers consistently contribute, but no ablation experiment appears anywhere in the manuscript. Section IV-D, titled 'EFFICIENCY ANALYSIS', is empty. Without an ablation that removes or randomizes the frozen Gemma slice while keeping the input and output networks fixed, the observed recall difference cannot be attributed to the frozen layers.
  3. [Section III-A versus Figure 1 and Section III-C] The architecture description is internally inconsistent on the input tensor. Section III-A defines Xt in R^{w x d} with w=30 and d=276, while Figure 1 and the input description state a flattened input of shape (B, 276). This matters because the stated temporal windowing motivation in Section I is not implemented: the pseudo-sequence is reshaped to (B, 1, 1152), so attention within the frozen Gemma layers has sequence length 1 and cannot mix information across time steps. The frozen layers therefore act as a fixed 1152-to-1152 nonlinear projection, and the paper provides no evidence that this specific projection is beneficial for wildfire features.
  4. [Section III-C versus Section V and Figure 15] The number of frozen Gemma layers is inconsistent. Section III-C states that layers 8-9 are frozen (two blocks, 14.7M parameters), while Section V and the Appendix/Figure 15 describe a single frozen Gemma-3 decoder layer. This inconsistency prevents the reader from knowing which configuration was actually evaluated and undermines the reproducibility claim.
  5. [Table 3, Table 5, and Section V] The parameter counts are mutually inconsistent. Table 3 lists a total of 21.7M parameters with approximately 5.0M trainable; Table 5 reports 37,725,825 total parameters for the Internal World model; and Section V states that only 5.6% of the network's 37.7M parameters are trainable, which would be approximately 2.1M, not 5.0M. Since one of the paper's stated contributions is parameter efficiency, these numbers must be reconciled.
  6. [Section IV-A and Table 5] Section IV-A states that the Internal World Model achieves the highest recall (0.9433), but Table 5 reports higher recall values for CNN (0.9856) and Physics-Embedded Entropy (0.9652). Similarly, Section V claims a '+6.7% recall' gain over the best purely task-specific network, but the difference between 0.9433 and the FFN's 0.9286 is 1.5 percentage points (about 1.6% relative), and the CNN baseline has higher recall still. These misstatements of the reported results need correction.
minor comments (3)
  1. [Throughout] There are several typographical errors and formatting artifacts, including 'V aswani' in the references, 'Y e' in a citation, and the duplicated keyword 'Wildfire Prediction' in the index terms.
  2. [Figure captions and table formatting] Several figures and tables are referenced but not fully self-contained in the text; for example, Figure 15's caption mentions 'a frozen Gemma-3 decoder layer' while the main text says two layers, and the reader cannot resolve this from the figure alone.
  3. [Section II-B] The enumerated author-trajectory list is helpful for context, but some entries cite preprints with no DOI or publication venue, which makes it difficult to verify the progression described.

Circularity Check

1 steps flagged · score 4.0 of 10

Reported metrics are partly fitted to the validation set, but the core Gemma-based derivation is not self-referential; the missing ablation undermines attribution rather than proving circularity.

  1. fitted input called prediction [Section IV preamble; Section III-F.5 (Early-Stopping & Check-pointing); Section III-E (Dropout); Table 5]
    "we quote metrics on the held-out validation split, computed with the optimal threshold that maximises the harmonic mean of precision and recall (F1) score for each model. ... Training halts if the validation F1 fails to improve by ∆min = 0.001 for 10 consecutive epochs. Weights with the best F1 are preserved. ... Dropout 0.4. A grid search in {0.2, 0.3, 0.4, 0.5} selected 0.4"

    The same validation split is used to pick the classification threshold, to decide early stopping, to retain the best-F1 checkpoint, and to select dropout and learning-rate hyperparameters. Table 5 then reports metrics computed on that same split as the models' predictive accuracy. The headline numbers are therefore optimized on the very data they are claimed to predict: the threshold and checkpoint are fitted quantities, not independent predictions. This is a fitted-input-called-prediction loop even though the loop affects all compared models equally and does not by itself force the Gemma-attribution claim.

full rationale

The only genuine circularity present is the use of the validation split to tune the threshold, checkpoint, and hyperparameters (early stopping, dropout) and then reporting metrics on that same split as predictive performance. This is a fitted-input-called-prediction loop, but it applies equally to all baselines and does not by itself force the paper's central attribution claim. The Gemma weights are externally pretrained and not fit to the wildfire data, so the architecture itself is not defined in terms of the target. The more serious problem is that the stated conclusion—'frozen Transformer layers consistently contribute'—is contradicted by the paper's own Table 5, where the Internal World model (F1 0.8838) loses to FFN+PosEnc (F1 0.8957) and also trails plain FFN in AUC. That is an attribution/evidence failure, not a circularity. The missing ablation (promised in the abstract but never shown) prevents any controlled attribution, but does not make the derivation self-referential. Overall score 4: one partial circularity in the evaluation protocol, while the central architecture remains independently specified.

Assumptions & free parameters 7 free parameters · 5 assumptions · 1 invented entities

The ledger captures the modeling choices the claim rests on: the chosen Gemma layer indices, dropout, learning rates, window length, and validation-tuned thresholds are all free parameters. The central domain assumption is that a single-vector projection into Gemma's hidden space retains useful pretrained knowledge. No new physical entity is introduced; the 'internal world' is a re-labeling of frozen layers.

free parameters (7)
  • dropout_rate = 0.4
    Selected by grid search over {0.2, 0.3, 0.4, 0.5} (Section III-E).
  • projection_learning_rate = 1e-3
    Set empirically; projection adapts faster (Section III-E).
  • classifier_learning_rate = 5e-4
    Set empirically (Section III-E).
  • positive_class_weight = w1=2.0
    Chosen for class imbalance (Section III-A).
  • window_length = 30 days
    Assumed temporal context; paper notes 90 days would be better (Sections III-A, III-I).
  • frozen_layer_indices = layers 8-9
    Which Gemma layers to freeze is chosen by hand; text elsewhere says one layer (Section III-C vs V).
  • decision_threshold = 0.397 for Internal World
    Each model's threshold is optimized on validation to maximize F1 (Section IV), a post-hoc fit.
assumptions (5)
  • domain assumption Gemma-3 pretrained weights carry transferable knowledge for tabular wildfire features
    Core premise of the internal-world module (Sections II-C, III-C); no external evidence provided.
  • ad hoc to paper A length-1 sequence through frozen transformer layers preserves that knowledge
    Figure 1 reshapes to (B,1,1152); attention cannot mix tokens, so the layers become a fixed MLP.
  • domain assumption MODIS/VIIRS fire detections are reliable ground truth
    Acknowledged limitation in Section III-I(c): omission errors on cloudy days propagate to labels.
  • domain assumption Temporal split (pre-2022 train, 2022 validation) is representative of future conditions
    Section III-B; climate variability may make this stationarity assumption fragile.
  • domain assumption The 276 input columns encode the temporal dynamics of fire risk
    Section III-B includes lags, but the actual window flattening is inconsistent with the architecture's (B,276) input.
invented entities (1)
  • 'Internal world' module
    purpose: Framing device to describe frozen Gemma-3 middle layers as a reusable knowledge module
    No external falsifiable handle; the paper's own experiments show a baseline without it performs better.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Deep Learning with Pretrained 'Internal World' Layers: A Gemma 3-Based Modular Architecture for Wildfire Prediction." pith.science (2026). https://pith.science/paper/52BY3GRR

@misc{pith2026250418562,
  author       = {Pith},
  title        = {Pith review of: Deep Learning with Pretrained 'Internal World' Layers: A Gemma 3-Based Modular Architecture for Wildfire Prediction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/52BY3GRR}},
  note         = {Machine review of arXiv:2504.18562}
}
read the original abstract

Deep learning models, especially large Transformers, carry substantial "memory" in their intermediate layers -- an \emph{internal world} that encodes a wealth of relational and contextual knowledge. This work harnesses that internal world for wildfire occurrence prediction by introducing a modular architecture built upon Gemma 3, a state-of-the-art multimodal model. Rather than relying on Gemma 3's original embedding and positional encoding stacks, we develop a custom feed-forward module that transforms tabular wildfire features into the hidden dimension required by Gemma 3's mid-layer Transformer blocks. We freeze these Gemma 3 sub-layers -- thus preserving their pretrained representation power -- while training only the smaller input and output networks. This approach minimizes the number of trainable parameters and reduces the risk of overfitting on limited wildfire data, yet retains the benefits of Gemma 3's broad knowledge. Evaluations on a Moroccan wildfire dataset demonstrate improved predictive accuracy and robustness compared to standard feed-forward and convolutional baselines. Ablation studies confirm that the frozen Transformer layers consistently contribute to better representations, underscoring the feasibility of reusing large-model mid-layers as a learned internal world. Our findings suggest that strategic modular reuse of pretrained Transformers can enable more data-efficient and interpretable solutions for critical environmental applications such as wildfire risk management.

Figures

Figures reproduced from arXiv: 2504.18562 by the authors.

Figure 1
Figure 1. Text-based schematic of the Internal-World Model. Brackets mark frozen Gemma layers; solid boxes are trainable. Standardisation. Every numeric column is z-scored with µ, σ from training data only. Window assembly. For each day t we collect [xt−w+1;. . . ; xt ] giving Xt ∈ R 30×276 . C. ARCHITECTURAL DESIGN PRINCIPLES The network is decomposed into three mutable–immutable blocks: 1) Input adaptation (trainable) — fou… view at source ↗
Figure 2
Figure 2. Three-layer Feed-Forward Network (FFN-3L) baseline. With optimisation procedures, baselines, and evaluation methodology now fully specified, Section IV presents quanti￾tative results and qualitative analysis. H. BASELINE SCHEMATICS AND DETAILED PARAMETERISATION For transparency and reproducibility, we sequentially provide concise ASCII schematics of each baseline model, clearly annotated with layer dimensions and co… view at source ↗
Figure 5
Figure 5. Physics-Embedded Entropy hybrid baseline. torch.cuda. • Data Access: The dataset employed in this study, titled Morocco Wildfire Predictions: 2010–2022 ML Dataset, is openly available on Kaggle [41], with a provided SHA-256 hash to ensure data integrity. • Hardware: The model training and evaluations are con￾ducted using a single NVIDIA RTX-3060 Ti GPU (8 GB VRAM), AMD Ryzen 9 5900X CPU, and 128 GB DDR4 RAM. • Run-t… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: MLP baseline with learned per-feature embeddings and positional tokens (PE-MLP). panding this to 90 days would improve context but in￾creases attention complexity quadratically. c) Label Noise: MODIS/VIIRS omission errors, such as those arising on cloudy days, propagat…
Figure 6
Figure 6. Figure 6: Receiver–Operating-Characteristic curves for the five architectures. A larger area under the curve indicates stronger class separation; numeric AUC values appear in [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 9
Figure 9. Figure 9: Confusion matrix of the plain three-layer FFN baseline. Recall drops by ≈ 1.5% relative to the Internal World model, increasing the false-negative tally [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Confusion matrix of the 1-D CNN baseline, which attains the highest recall among baselines but nearly doubles the false-positive rate versus the FFN. for Computational Linguistics, 2019, pp. 4593–4601. [18] M. D. Zeiler and R. Fergus, ‘‘Visualizing and Understanding C…
Figure 11
Figure 11. Figure 11: Confusion matrix for the lightweight FFN + PosEnc model. Its balanced precision and recall underpin the best overall F1 score despite having the fewest parameters. ‘‘Neural Module Networks for Reasoning over Text,’’ arXiv preprint arXiv:2212.05970, 2022. [26] J. Andre…
Figure 12
Figure 12. Figure 12: Radar chart contrasting the five key discrimination metrics (accuracy, AUC, precision, recall, and F1) for every architecture. The FFN + PosEnc polygon encloses the largest area, visually confirming its superior aggregate performance [PITH_FULL_IMAGE:figures/full_fig…
Figure 13
Figure 13. Figure 13: Wall-clock training time for five epochs on a single RTX-3060 Ti. Although the Internal World model contains a frozen billion-parameter core, its effective training cost is only ≈ 10% higher than the lightest baseline thanks to the small number of trainable weights. A…
Figure 15
Figure 15. Figure 15: Block diagram of the proposed Internal World wildfire prediction network. Tabular input features are processed by four parallel linear–ReLU branches, integrated via a 3-layer feed-forward stack, projected to a 1152-D latent space, and passed through a frozen Gemma-3 d…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 37 canonical work pages

  1. [1]

    C. E. V an Wagner, ‘‘Development and structure of the Canadian forest fire weather index system,’’ Canadian Forestry Service, Technical Report 35, 1977

    work page 1977

  2. [2]

    J. E. Deeming, R. E. Burgan, and J. D. Cohen, ‘‘The National Fire-Danger Rating System—1978,’’ USDA Forest Service, General Technical Report INT-39, 1977

    work page 1978

  3. [3]

    M. A. Parisien and M. A. Moritz, ‘‘Environmental controls on the distribu- tion of wildfires at multiple spatial scales,’’ Ecol. Monogr ., vol. 79, no. 2, pp. 127–154, 2009

    work page 2009

  4. [4]

    Cortez and A

    P . Cortez and A. D. Morais, ‘‘A data mining approach to predict forest fires using meteorological data,’’ in Proc. 13th EPIA—Portuguese Conf. on Artificial Intelligence, 2007, pp. 512–523

    work page 2007

  5. [5]

    P . Jain, S. C. P . Coogan, S. G. Subramanian, M. Crowley, S. Taylor, and M. D. Flannigan, ‘‘A review of machine learning applications in wildfire science and management,’’ Environmental Reviews, vol. 28, no. 4, pp. 478–505, 2020

    work page 2020

  6. [6]

    Zhang, M

    G. Zhang, M. Wang, and K. Liu, ‘‘Forest Fire Susceptibility Modeling Using a Convolutional Neural Network for Y unnan Province of China,’’ International Journal of Disaster Risk Science , vol. 10, pp. 386–403, 2019

    work page 2019

  7. [7]

    Radke, A

    D. Radke, A. Hessler, and D. Ellsworth, ‘‘FireCast: Leveraging Deep Learning to Predict Wildfire Spread,’’ in Proc. 28th International Joint Conference on Artificial Intelligence (IJCAI) , 2019, pp. 4575–4581

    work page 2019

  8. [8]

    Lahrichi, K

    Y . Lahrichi, K. Haouari, D. Litvin, C. Mattauch, S. Gao, F. Nategh, G. Schiebinger, and K. Saenko, ‘‘Predicting Next-Day Wildfire Spread with Time Series and Attention,’’ arXiv preprint arXiv:2502.12003, 2024

    arXiv 2024

Show all 46 references
  1. [9]

    J. Zhu, S. Liu, S. D. Tran, and D. W. Scott, ‘‘Shapley-based interpretation of deep learning models for wildfire spread rate prediction: An XAI approach,’’ Fire Ecology, vol. 19, no. 1, 2023

    2023

  2. [10]

    Jadouli and C

    A. Jadouli and C. El Amrani, ‘‘Advanced Wildfire Prediction in Morocco: Developing a Deep Learning Dataset From Multisource Observations,’’ IEEE Access , vol. 12, pp. 191733–191747, 2024, doi:10.1109/ACCESS.2024.3516784. 8 VOLUME 11, 2023 Jadouli et al.: Preparation of Papers ...

    2024

  3. [11]

    Jadouli and C

    A. Jadouli and C. El Amrani, ‘‘Parallel Multi-path Feed Forward Neural Networks (PMFFNN) for Long Columnar Datasets: A Novel Approach to Complexity Reduction,’’ arXiv preprint arXiv:2411.06020, 2024

    2024 arXiv

  4. [12]

    Jadouli and C

    A. Jadouli and C. El Amrani, ‘‘Physics-Embedded Deep Learning for Wildfire Risk Assessment: Integrating Statistical Mechanics into Neural Networks for Interpretable Environmental Modeling,’’ 2025

    2025

  5. [13]

    Jadouli and C

    A. Jadouli and C. El Amrani, ‘‘Enhancing wildfire forecasting through multisource spatio-temporal data, deep learning, ensemble models and transfer learning,’’ Adv. Artif. Intell. Mach. Learn., vol. 4, no. 3, pp. 2614– 2628, Sep. 2024, doi: 10.54364/AAIML.2024.43152

    2024

  6. [14]

    Jadouli and C

    A. Jadouli and C. El Amrani, ‘‘Bridging Physical Entropy Theory and Deep Learning for Wildfire Risk Assessment: A Hybrid Pretraining and Fine-Tuning Approach with Satellite Data,’’ Preprint, 2023

    2023

  7. [15]

    Rogers, O

    A. Rogers, O. Kovaleva, and A. Rumshisky, ‘‘A Primer in BERTology: What we know about how BERT works,’’ Transactions of the Association for Computational Linguistics, vol. 8, pp. 842–866, 2020

    2020

  8. [16]

    Wallat, J

    J. Wallat, J. Singh, and A. Anwar, ‘‘BERTnesia: Investigating the capture and forgetting of knowledge in BERT,’’ arXiv preprint arXiv:2106.02902, 2021

    2021 arXiv

  9. [17]

    Tenney, D

    I. Tenney, D. Das, and E. Pavlick, ‘‘BERT Rediscovers the Classical NLP Pipeline,’’ in Proceedings of the 57th Annual Meeting of the Association Figure 9. Confusion matrix of the plain three-layer FFN baseline. Recall drops by ≈ 1.5% relative to the Internal World model, incre...

  10. [18]

    M. D. Zeiler and R. Fergus, ‘‘Visualizing and Understanding Convolutional Networks,’’ in European Conference on Computer Vision, 2014, pp. 818– 833

    2014

  11. [19]

    Raghu, T

    M. Raghu, T. Unterthiner, S. Kornblith, C. Zhang, and A. Dosovitskiy, ‘‘Do Vision Transformers See Like Convolutional Neural Networks?,’’ arXiv preprint arXiv:2108.08810, 2021

    2021 arXiv

  12. [20]

    M. Chen, H. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever, ‘‘Generative Pretraining From Pixels,’’ in International Conference on Machine Learning, 2022, pp. 1691–1703

    2022

  13. [21]

    Burns, S

    C. Burns, S. Y e, E. Singh, A. Akyürek, and J. Andreas, ‘‘Circuit Component Reuse Across Tasks in Transformer Language Models,’’ arXiv preprint arXiv:2212.10486, 2022

    2022 arXiv

  14. [22]

    B. Zoph, I. Bello, S. Kumar, N. Du, Y . Huang, J. Dean, N. Shazeer, and W. Fedus, ‘‘Designing Effective Sparse Expert Models,’’ arXiv preprint arXiv:2202.08906, 2022

    2022 arXiv

  15. [23]

    S. J. Pan and Q. Y ang, ‘‘A survey on transfer learning,’’IEEE Trans. Knowl. Data Eng., vol. 22, no. 10, pp. 1345–1359, 2010

    2010

  16. [24]

    K. He, R. Girshick, and P . Dollár, ‘‘Rethinking ImageNet Pre-training,’’ in IEEE/CVF International Conference on Computer Vision , 2019, pp. 4918– 4927

    2019

  17. [25]

    Imtiaz, S

    N. Imtiaz, S. Shrestha, S. Das, A. Wright, J. Sanchez, and A. Bhuiyan, VOLUME 11, 2023 9 Jadouli et al.: Preparation of Papers for IEEE ACCESS Figure 11. Confusion matrix for the lightweight FFN + PosEnc model. Its balanced precision and recall underpin the best overall F 1 sc...

    2023 arXiv

  18. [26]

    Andreas, M

    J. Andreas, M. Rohrbach, T. Darrell, and D. Klein, ‘‘Neural Module Net- works,’’ in IEEE Conference on Computer Vision and Pattern Recognition , 2016, pp. 39–48

    2016

  19. [27]

    Rabbani, M

    G. Rabbani, M. Shamsi, and P . Abbeel, ‘‘Transfer Learning of Tabular Data by Finetuning Large Language Models,’’ arXiv preprint arXiv:2501.06863, 2025

    2025 arXiv

  20. [28]

    Houlsby, A

    N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. de Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, ‘‘Parameter-Efficient Transfer Learning for NLP ,’’ inInternational Conference on Machine Learning, 2019, pp. 2790–2799

    2019

  21. [29]

    J. T. Abatzoglou and A. P . Williams, ‘‘Impact of anthropogenic climate change on wildfires across western US forests,’’ Proc. Natl. Acad. Sci. U.S.A., vol. 113, no. 42, pp. 11770–11775, 2016

    2016

  22. [30]

    Chuvieco, I

    E. Chuvieco, I. Aguado, M. Y ebra, H. Nieto, and J. Salas, ‘‘Development of a framework for fire risk assessment using remote sensing and geographic information system technologies,’’ Ecol. Modell., vol. 221, no. 1, pp. 46–58, 2010

    2010

  23. [31]

    Brown et al., ‘‘Language Models are Few-Shot Learners,’’ in Advances in Neural Information Processing Systems , vol

    T. Brown et al., ‘‘Language Models are Few-Shot Learners,’’ in Advances in Neural Information Processing Systems , vol. 33, 2020

    2020

  24. [32]

    Wei et al

    J. Wei et al. , ‘‘Emergent Abilities of Large Language Models,’’ arXiv preprint arXiv:2206.07682, 2022

    2022 arXiv

  25. [33]

    V aswaniet al., ‘‘Attention Is All Y ou Need,’’ inProc

    A. V aswaniet al., ‘‘Attention Is All Y ou Need,’’ inProc. Adv. Neural Inf. Process. Syst. (NeurIPS), Long Beach, CA, USA, 2017, pp. 5998–6008

    2017

  26. [34]

    Devlin, M

    J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, ‘‘BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding,’’ in Proc. NAACL–HLT, Minneapolis, MN, USA, 2019, pp. 4171–4186

    2019

  27. [35]

    Huang, J

    X. Huang, J. Gorban, and M. Rothermel, ‘‘TabTransformer: Tabular Data Modeling Using Contextual Embeddings,’’ in Proc. AAAI, vol. 35, no. 8, 2021, pp. 15025–15033

    2021

  28. [36]

    Lee et al., ‘‘Gemma 3: A Multimodal Lightweight Model with 128 k Context,’’ arXiv preprint arXiv:2503.19786, Mar

    S. Lee et al., ‘‘Gemma 3: A Multimodal Lightweight Model with 128 k Context,’’ arXiv preprint arXiv:2503.19786, Mar. 2025

    2025 arXiv

  29. [37]

    Jadouli and C

    A. Jadouli and C. El Amrani, ‘‘Detection of Human Activities in Wildlands to Prevent the Occurrence of Wildfires Using Deep Learning and Remote Sensing,’’ in Networking, Intelligent Systems and Security , Singapore: Springer, 2022, pp. 3–17, doi:10.1007/978-981-16-3637-0_1

    2022 doi

  30. [38]

    Jadouli and C

    A. Jadouli and C. El Amrani, ‘‘Hybrid Parallel Architecture Integrating FFN, 1D CNN, and LSTM for Predicting Wildfire Occurrences in Morocco,’’ in Adv. Intell. Syst. Comput. , vol. 1535, Springer, 2025, doi:10.1007/978-3- 031-88653-9_16

    2025 doi

  31. [39]

    Jadouli and C

    A. Jadouli and C. El Amrani, ‘‘Enhancing Wildfire Prediction Accuracy Through Positional Encoding and Advanced Neural Architectures,’’ Preprint, 2025

    2025

  32. [40]

    Jadouli and C

    A. Jadouli and C. El Amrani, ‘‘Gemma3-InternalWorld-WildFire: Deep Learning with Pretrained ’Internal World’ Layers for Wildfire Predic- tion,’’ GitHub repository, 2025. [Online]. Available: https://github.com/ AyoubJadouli/Gemma3-InternalWorld-WildFire

    2025

  33. [41]

    INTERNAL WORLD

    A. Jadouli and C. El Amrani, ‘‘Morocco Wildfire Predictions: 2010- 2022 ML Dataset,’’ Kaggle dataset, 2024. [Online]. DOI: 10.34740/KAG- GLE/DSV/8040722. Available: https://www.kaggle.com/dsv/8040722 AYOUB JADOULI is a Ph.D. candidate at the Fac- ulty of Sciences and Technolog...

    2010

  34. [42]

    Input tabular features – daily meteorological, vegetation, and topographic variables extracted from the Morocco Wildfire dataset (Section III-B)

  35. [43]

    Figure 14

    Multi-branch feature processing – four parallel linear– ReLU branches that map disjoint feature subsets to a shared latent dimension and concatenate the results. Figure 14. Trainable parameter budget for each architecture (log-scale). The lightweight FFN + PosEnc requires two ...

  36. [44]

    Feature integration – a 3-layer FFN followed by a projec- tion head that reshapes the integrated vector to the hidden size (1152) expected by the pretrained module

  37. [45]

    Pretrained internal world – a frozen Gemma-3 decoder layer whose multi-head attention and gated FFN inject rich, pretrained inductive bias

  38. [46]

    The entire workflow—from raw data ingestion and branch projection to probability output—forms the backbone of the experimental pipeline evaluated in Section IV

    Output classification – a lightweight MLP that produces wildfire logits, applies a sigmoid, and outputs the predicted probability. The entire workflow—from raw data ingestion and branch projection to probability output—forms the backbone of the experimental pipeline evaluated ...

    2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.