FiLLM -- A Filipino-optimized Large Language Model based on Southeast Asia Large Language Model (SEALLM)

Carlos Jude G. Maminta; Deandre Nigel Nunez; Information Sciences; Isaiah Job Enriquez; Manila); Michael B. Dela Fuente (Institution College of Computer; Polytechnic University of the Philippines; Sta. Mesa

REVIEW 4 major objections 5 minor 15 references

FiLLM -- A Filipino-optimized Large Language Model based on Southeast Asia Large Language Model (SEALLM)

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FiLLM, a LoRA-tuned SeaLLM-7B 2.5 model, reports F1 0.89 on Filipino NER and POS tagging, 0.73 on dependency parsing, and trails CalamanCy on all three tasks.

desk verdict The paper's central statistical evidence is internally impossible; the honest comparison with CalamanCy is a good instinct but the execution is too sloppy to trust. read the letter →

arxiv 2505.18995 v1 pith:JCNDMRH5 submitted 2025-05-25 cs.CL cs.AI

Carlos Jude G. Maminta , Isaiah Job Enriquez , Deandre Nigel Nunez , Michael B. Dela Fuente (Institution College of Computer , Information Sciences , Polytechnic University of the Philippines , Sta. Mesa , Manila) This is my paper · ORCID

classification cs.CLcs.AI

keywords FilipinoNLPLow-RankAdaptationSeaLLMCalamanCyNamedEntityRecognitionPart-of-SpeechTaggingDependencyParsingTextSummarization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that a Filipino-optimized large language model can be built cheaply by applying Low-Rank Adaptation (LoRA) to the SeaLLM-7B 2.5 base, and to measure whether that model is competitive with CalamanCy, an existing Filipino NLP pipeline. The authors report F1 scores of 0.89 for named entity recognition, 0.89 for part-of-speech tagging, and 0.73 for dependency parsing, against CalamanCy scores of 0.90, 0.97, and 0.97 on the same tasks. The central empirical assertion is therefore that the LoRA-tuned model is usable for entity and tagging work but materially weaker on syntax. The paper also claims a statistically significant difference between the two models under a paired t-test (p = 0.03), and presents text summarization results that show keyword overlap staying high as compression rate rises. If the comparison held up, it would tell Filipino NLP practitioners that a memory-efficient LoRA adapter does not yet close the gap to a purpose-built pipeline.

What carries the argument

Low-Rank Adaptation (LoRA), a fine-tuning scheme that keeps the pretrained SeaLLM-7B 2.5 weights frozen and learns two low-rank matrices, LoRA A of shape $D \times R$ and LoRA B of shape $R \times D$, whose product is added to the frozen weight updates. This is the mechanism that lets the paper fine-tune a seven-billion-parameter model with limited memory while adapting it to Filipino tasks. The comparison object is CalamanCy, an existing Filipino NLP pipeline, and the evaluation machinery is a paired t-test over the reported F1-style task scores.

What would settle it

Run FiLLM and CalamanCy on the same held-out split of the merged UD and TLUnified-NER data with identical tokenization and metric code, and check whether the reported F1 gaps (0.89 versus 0.97 on POS, 0.73 versus 0.97 on parsing) reproduce; a smaller or reversed gap would show the difference came from evaluation setup rather than model capability.

Watch

Extended reading notes

Core claim

FiLLM, built by LoRA fine-tuning SeaLLM-7B 2.5 on Filipino datasets, achieves average F1 scores of 0.89 for named entity recognition, 0.89 for part-of-speech tagging, and 0.73 for dependency parsing. On the same evaluation, CalamanCy scores 0.90, 0.97, and 0.97 respectively. The authors conclude that CalamanCy outperforms FiLLM, especially on syntactic tasks, and that the difference is statistically significant at the 0.05 level with p = 0.03. They also report that FiLLM's text summarization retains key information even at high compression rates, tying summarization quality to the balance between compression and keyword retention.

Load-bearing premise

The comparison assumes FiLLM and CalamanCy were evaluated under identical conditions, on the same test splits, with the same preprocessing and the same metric implementation, but the paper gives no CalamanCy configuration or evaluation code.

Editorial extensions

If this is right

FiLLM can serve as a usable Filipino NER and POS tagger at 0.89 F1 while keeping training memory low through LoRA.
For dependency parsing, FiLLM is not ready to replace CalamanCy, which scores 0.97 against FiLLM's 0.73.
If the paired t-test result holds, the gap between FiLLM and CalamanCy is too large to be dismissed as chance, at least in the paper's evaluation setting.
FiLLM's summarization behavior suggests it can compress Filipino text while retaining a substantial share of keywords, making it a candidate for extractive-style condensation.
The overall result implies that base-model choice and LoRA adaptation alone do not guarantee parity with an established low-resource-language pipeline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the reported numbers are taken at face value, the natural next experiment is to compare the two models sentence-by-sentence on the same held-out split; the dependency-parsing gap suggests FiLLM's LoRA adapter has not learned long-range syntactic structure.
The compression-rate and keyword-overlap correlation is only a proxy for summarization quality; a stricter test would score summaries against references with metrics such as ROUGE or BERTScore, which the paper does not report.
A direct replication would make the comparison airtight: fix a single held-out split, state the CalamanCy configuration, and share the evaluation script; the task-level F1 gaps could then be verified independently.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The one thing you should know: this paper's headline comparison between FiLLM and CalamanCy is unsupported by its own numbers. The paired t-test in Table 3 cannot be right. With means 83.67 vs 94.67, n=5, and the listed variances, t=0.12 and p=0.03 are arithmetically incompatible. Recomputing from the three task F1 scores (differences 1, 8, 24) gives t≈1.62, p≈0.25. The paper reports n=5 for three tasks, and the conclusion's precision/recall numbers contradict Table 2. This isn't a minor slip; the central comparative claim rests on a statistic that is impossible as reported.

What the paper does well is worth saying. The authors chose a relevant base model (SeaLLM-7B) and used standard Filipino datasets, and they were honest enough to state up front that CalamanCy beats FiLLM on most tasks. That honesty is rare and should be credited. The summarization analysis using compression rate and keyword overlap is a reasonable idea, though underdeveloped. The references cover the relevant Filipino NLP literature.

The soft spots are otherwise extensive. No evaluation code, model checkpoint, or test-split details are provided, so the comparison with CalamanCy is not reproducible. The text changes POS recall from 0.93 to 0.90, and the conclusion's 86/84 and 71 do not match the table's 89/90 and 73. The t-test interpretation is also backward: t=0.12 with a critical value of 2.776 would mean failing to reject the null, not rejecting it. These are load-bearing inconsistencies, not formatting gripes.

Who is this for? A reader hunting for a quick negative data point about LoRA-tuned SeaLLM on Filipino tasks might get a hint, but they cannot trust any specific number. The paper needs a rewrite with corrected statistics, a released artifact, and a fuller description of the evaluation setup before it is useful to anyone.

My recommendation: reject. This does not deserve referee time in its current state. If the authors fix the arithmetic, release their model and code, and redo the comparison with a proper test, it could become a modest but acceptable workshop paper. Right now it is a cautionary example of why you check the statistics before believing the conclusion.

Referee Report

4 major / 5 minor

Summary. The paper introduces FiLLM, a Filipino-language large language model obtained by LoRA fine-tuning of SeaLLM-7B 2.5, and evaluates it on Named Entity Recognition, Part-of-Speech tagging, Dependency Parsing, and Text Summarization using Filipino datasets. The authors report F1 scores of 0.89, 0.89, and 0.73 for FiLLM on NER, POS, and dependency parsing, respectively, compare these against CalamanCy, and use a paired t-test to claim that the performance difference is statistically significant. The manuscript's central claim is that FiLLM is a usable but weaker alternative to CalamanCy for Filipino NLP tasks.

Significance. If the reported results were reliable, the paper would provide a useful data point for low-resource Filipino NLP by demonstrating that memory-efficient LoRA fine-tuning of a multilingual SEA language model yields competitive NER and POS performance, and by benchmarking against an established Filipino pipeline. The authors deserve credit for assembling several Filipino datasets, for applying LoRA to a 7B-parameter model, and for attempting a statistical comparison against an existing system. However, the central empirical claim is not supported as reported: the statistical test in Table 3 is arithmetically impossible, the sample-size and task-count values do not match, and the precision/recall/F1 numbers differ across the table, text, and conclusion. These internal inconsistencies make the current evidence inadequate to support the paper's conclusions, so the potential significance cannot be realized without a complete re-analysis and re-reporting of the experiments.

major comments (4)

[Table 3 and §2.5] The paired t-test results in Table 3 are internally impossible. With the reported means 83.67 and 94.67, the mean difference is 11. Across the three F1 scores reported in Figure 3/Table 2, the paired differences are 1, 8, and 24, yielding a paired t-statistic of approximately 1.62 with 2 degrees of freedom and a two-tailed p-value of about 0.25. The reported t-statistic of 0.12 with p = 0.03 cannot occur for any consistent dataset; t = 0.12 would give p ≈ 0.91, and p = 0.03 with df = 4 would require t ≈ 2.78. Since the conclusion "we reject the null hypothesis" follows from the p-value, the test as presented is self-contradictory and does not support the claimed significant difference.
[Table 3] Table 3 reports 5 observations for both FiLLM and CalamanCy, but the paper only presents three task-level F1 scores for each model (NER, POS, dependency parsing). No fourth or fifth observation is identified anywhere. The degrees of freedom (4) and the critical value (2.776) are consistent with n = 5, but no actual data with n = 5 is described, so the statistical analysis is not reproducible from the manuscript.
[§3, Table 2, and Conclusion] The reported metric values are not self-consistent across the paper. Table 2 gives FiLLM POS recall as 0.90, but the text in §3 states a Recall of 0.93 for POS; the conclusion reports POS precision 86% and recall 84%, and dependency parsing precision/recall of 71%, whereas Table 2 and the text report 0.73/0.74 and 0.73. The NER precision also changes from 0.86 in Table 2 to 86% in the conclusion, which is consistent numerically, but the POS and dependency parsing numbers are not. Because the paper never identifies which numbers are authoritative, the central results cannot be verified.
[§2.2 and §3 (Figure 3 comparison)] The comparison against CalamanCy is not adequately specified. The paper does not state which version of CalamanCy was used, how it was configured, whether it was evaluated on the exact same 80-20 test splits as FiLLM, which preprocessing steps were applied, or how the F1 metrics were computed for each system. Without identical evaluation conditions, the relative F1 scores in Figure 3 have no clear interpretation, and the central claim that CalamanCy outperforms FiLLM is not properly supported.

minor comments (5)

[§2.5] The text says "This formula represents the t-statistic for a paired t-test" but the formula itself does not appear in the manuscript; please include the actual equation with proper notation.
[Table 1 and Figure 1] Table 1 lists datasets for NER, dependency parsing, POS, and summarization, but Figure 1's architecture description additionally mentions the NewsPH-NLI dataset, which is not listed in Table 1; the dataset list should be reconciled.
[Various] There are numerous spelling and formatting inconsistencies, including "Calamancy" vs. "CalamanCy", "Name Entity Recogition" in Table 1, "DEPPOSSUM" as a model label in Table 2, and inconsistent capitalization throughout.
[Figure 2] Figure 2 is not described in enough detail for the reader to interpret the summarization results; the axes are labeled but no numerical results, sample sizes, or evaluation protocol are given in the text.
[References] Several references are incomplete or inconsistently formatted, including missing venue information for [5] and truncated entries in [6]; full bibliographic details should be supplied.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular reasoning: FiLLM's empirical comparison to CalamanCy is externally anchored, with no derivation or fitted-parameter prediction that reduces to its inputs.

full rationale

FiLLM is a LoRA fine-tune of SeaLLM-7B, and the paper's claims are empirical benchmark scores compared against the external CalamanCy pipeline. There is no mathematical derivation, no fitted parameter renamed as a prediction, and no load-bearing self-citation: the base model [7], datasets [1,4,6,8,10], and CalamanCy are independent external artifacts. The t-test in Table 3 is arithmetically inconsistent with the reported F1 scores and sample size, and the body text disagrees with Table 2 (e.g., POS recall 0.93 vs 0.90), but those are correctness/reproducibility problems, not circularity. The paper's own caveats about data variability do not close a definitional loop. Accordingly, no step in the derivation chain is equivalent to its input by construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entity. Its empirical claims rest entirely on unstated training settings, an unverified evaluation baseline, and an inconsistent statistical test. The axiom ledger records the assumptions a replication would need to satisfy.

free parameters (2)

LoRA rank R and alpha = Not reported
The LoRA decomposition dimensions control adaptation capacity and are chosen by hand; without them the fine-tuning cannot be reproduced.
Training hyperparameters (learning rate, batch size, epochs) = Not reported
Section 2.4 lists libraries but no numeric settings, so the performance numbers are not reproducible.

assumptions (4)

domain assumption SeaLLM-7B is an appropriate frozen base model for Filipino adaptation
Section 2.3 assumes the pretrained model's knowledge is suitable and that freezing it preserves capabilities.
domain assumption The named datasets and their 80-20 splits form valid training and test sets for Filipino NLP tasks
Section 2.2 describes datasets but provides no stratification, no dataset versions, and no verification of label quality.
domain assumption CalamanCy was evaluated under identical conditions with the same preprocessing and metric code
Figure 3 and Section 3 compare the models without describing CalamanCy's configuration or evaluation protocol.
standard math The paired t-test's distributional assumptions hold for the three task-level F1 scores
Section 2.5 invokes a paired t-test with n=5, a sample too small and task-heterogeneous for the assumption to be credible.

how reviews work

0 comments

Cite this review

Pith. "Pith review of FiLLM -- A Filipino-optimized Large Language Model based on Southeast Asia Large Language Model (SEALLM)." pith.science (2026). https://pith.science/paper/JCNDMRH5

@misc{pith2026250518995,
  author       = {Pith},
  title        = {Pith review of: FiLLM -- A Filipino-optimized Large Language Model based on Southeast Asia Large Language Model (SEALLM)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JCNDMRH5}},
  note         = {Machine review of arXiv:2505.18995}
}

read the original abstract

This study presents FiLLM, a Filipino-optimized large language model, designed to enhance natural language processing (NLP) capabilities in the Filipino language. Built upon the SeaLLM-7B 2.5 model, FiLLM leverages Low-Rank Adaptation (LoRA) fine-tuning to optimize memory efficiency while maintaining task-specific performance. The model was trained and evaluated on diverse Filipino datasets to address key NLP tasks, including Named Entity Recognition (NER), Part-of-Speech (POS) tagging, Dependency Parsing, and Text Summarization. Performance comparisons with the CalamanCy model were conducted using F1 Score, Precision, Recall, Compression Rate, and Keyword Overlap metrics. Results indicate that Calamancy outperforms FILLM in several aspects, demonstrating its effectiveness in processing Filipino text with improved linguistic comprehension and adaptability. This research contributes to the advancement of Filipino NLP applications by providing an optimized, efficient, and scalable language model tailored for local linguistic needs.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 13 canonical work pages

[1]

INTRODUCTION The Philippines is a linguistically diverse country with over 175 languages spoken nationwide [3]. Despite the increasing adoption of artificial intelligence (AI) in natural language processing (NLP), the development of large language models (LLMs) for Filipino remains limited due to the scarcity of labeled datasets and computational resource...

work page
[2]

The SeaLLM-7B 2.5 model serves as the base model, which was fine - tuned using LoRA to adapt to Filipino linguistic characteristics [7]

METHODOLOGY 2.1 Research Design This study employs an experimental approach, comparing the performance of FiLLM and CalamanCy on various NLP tasks. The SeaLLM-7B 2.5 model serves as the base model, which was fine - tuned using LoRA to adapt to Filipino linguistic characteristics [7]. 2.2 Datasets The following datasets were utilized: Filipino Hatespeech D...

work page 2023
[3]

The study aimed to investigate the potential of the FiLLM - A Filipino -optimized Large Language Model based on Southeast Asia Large Language Model (SEALLM)

RESULTS AND DISCUSSION This chapter presents the analysis and discussion of data gathered through the implementation of the proposed tool. The study aimed to investigate the potential of the FiLLM - A Filipino -optimized Large Language Model based on Southeast Asia Large Language Model (SEALLM). Task Model Precision Recall F1- Score (Name Entity Recogniti...

work page
[4]

These results underscore FiLLM’s reliability in fundamental NLP tasks, making it a valuable tool for text processing applications

CONCLUSION The study on FiLLM (Filipino-optimized Large Language Model) demonstrates its strengths in Part-of-Speech (POS) Tagging and Named Entity Recognition (NER), achieving 86% precision and 84% recall in POS tagging, and 86% precision and 93% recall in NER. These results underscore FiLLM’s reliability in fundamental NLP tasks, making it a valuable to...

work page
[5]

Without their guidance and aid during the creation of this research, it would not have been finished

ACKNOWLEDGMENTS We would like to express our deepest gratitude and appreciation to everyone who helped make this paper a success. Without their guidance and aid during the creation of this research, it would not have been finished. We thank our Almighty God first and foremost for directing us to this text and for providing us with wisdom, knowledge, stren...

work page
[6]

Angelina Aquino and Franz De Leon. 2020. Parsing in the absence of related languages: Evaluating low - resource dependency parsers on Tagalog

work page 2020
[7]

Jan Christian, Blaise Cruz, and Charibeth Cheng. 2022. Improving Large-scale Language Models and Resources for Filipino. Retrieved from https://elrc-share.eu/

work page 2022
[8]

Jan Christian Blaise Cruz and Charibeth Cheng. 2019. Evaluating Language Model Finetuning Techniques for Low-resource Languages. (June 2019). https://doi.org/10.13140/RG.2.2.23028.40322

work page arXiv 2019

Show all 15 references

[9]

Jan Christian Blaise Cruz and Charibeth Cheng. 2020. Establishing Baselines for Text Classification in Low - Resource Languages. (May 2020). Retrieved from http://arxiv.org/abs/2005.02068

2020 arXiv
[10]

Adegoke Israel, Daniel Izenyi, Adegoke A Israel, Daniel A Izenyi, and Lwasinam L Dilli. 2024. Efficiently Fine- tuning Large Language Model: LoRA Approach. (2024). https://doi.org/10.5281/zenodo.11312792

2024 doi
[11]

IEEE ICA 2018 : 2018 IEEE International Conference on Agents : July 28 -31, 2018, Nanyang Technological University, Singapore

Nanyang Technological University, IEEE Computer Society, Institute of Electrical and Electronics Engineers., International Conference on Crowd Science and Engineering (3rd : 2018 : Singapore), and International Conference on Ageless Aging (6th : 2018 : Singapore). IEEE ICA 201...

2018
[12]

Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, Hang Zhang, and Lidong Bing. 2023. SeaLLMs -- Large Language Models for Southeast Asia. (December 2023). Retrieved from http://arxiv.org/abs/2312.00738

2023 arXiv
[13]

Hammam Riza, Michael Purwoadi, Teduh Uliniansyah Badan Pengkajian dan Penerapan Teknologi, Indonesia Aw Ai Ti, Sharifah Mahani Aljunied, Luong Chi Mai, Vu Tat Thang, Nguyen Phuong Thái, Rapid Sun, Vichet Chea, Khin Mar Soe, Khin Thandar Nwet, Masao Utiyama, and Chenchen Ding. ...

2016
[14]

Ria Ambrocio Sagum. 2021. Filipino Native Language Identification using Markov Chain Model and Maximum Likelihood Decision Rule

2021
[15]

Neil P Vicente Cabasag, Vicente C Raphael Chan, Sean Y Christian Lim, Mark M Edward Gonzales, and Charibeth K Cheng. 2019. Hate Speech in Philippine Election-Related Tweets: Automatic Detection and Classification Using Natural Language Processing

2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

INTRODUCTION The Philippines is a linguistically diverse country with over 175 languages spoken nationwide [3]. Despite the increasing adoption of artificial intelligence (AI) in natural language processing (NLP), the development of large language models (LLMs) for Filipino remains limited due to the scarcity of labeled datasets and computational resource...

work page

[2] [2]

The SeaLLM-7B 2.5 model serves as the base model, which was fine - tuned using LoRA to adapt to Filipino linguistic characteristics [7]

METHODOLOGY 2.1 Research Design This study employs an experimental approach, comparing the performance of FiLLM and CalamanCy on various NLP tasks. The SeaLLM-7B 2.5 model serves as the base model, which was fine - tuned using LoRA to adapt to Filipino linguistic characteristics [7]. 2.2 Datasets The following datasets were utilized: Filipino Hatespeech D...

work page 2023

[3] [3]

The study aimed to investigate the potential of the FiLLM - A Filipino -optimized Large Language Model based on Southeast Asia Large Language Model (SEALLM)

RESULTS AND DISCUSSION This chapter presents the analysis and discussion of data gathered through the implementation of the proposed tool. The study aimed to investigate the potential of the FiLLM - A Filipino -optimized Large Language Model based on Southeast Asia Large Language Model (SEALLM). Task Model Precision Recall F1- Score (Name Entity Recogniti...

work page

[4] [4]

These results underscore FiLLM’s reliability in fundamental NLP tasks, making it a valuable tool for text processing applications

CONCLUSION The study on FiLLM (Filipino-optimized Large Language Model) demonstrates its strengths in Part-of-Speech (POS) Tagging and Named Entity Recognition (NER), achieving 86% precision and 84% recall in POS tagging, and 86% precision and 93% recall in NER. These results underscore FiLLM’s reliability in fundamental NLP tasks, making it a valuable to...

work page

[5] [5]

Without their guidance and aid during the creation of this research, it would not have been finished

ACKNOWLEDGMENTS We would like to express our deepest gratitude and appreciation to everyone who helped make this paper a success. Without their guidance and aid during the creation of this research, it would not have been finished. We thank our Almighty God first and foremost for directing us to this text and for providing us with wisdom, knowledge, stren...

work page

[6] [6]

Angelina Aquino and Franz De Leon. 2020. Parsing in the absence of related languages: Evaluating low - resource dependency parsers on Tagalog

work page 2020

[7] [7]

Jan Christian, Blaise Cruz, and Charibeth Cheng. 2022. Improving Large-scale Language Models and Resources for Filipino. Retrieved from https://elrc-share.eu/

work page 2022

[8] [8]

Jan Christian Blaise Cruz and Charibeth Cheng. 2019. Evaluating Language Model Finetuning Techniques for Low-resource Languages. (June 2019). https://doi.org/10.13140/RG.2.2.23028.40322

work page arXiv 2019