Fine-tuning open-source LLMs outperforms prompting for POS tagging on medieval Occitan, French, and Spanish, and pooling Romance training data helps the most under-resourced texts.
The Importance of Context in Very Low Resource Language Modeling
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
This paper investigates very low resource language model pretraining, when less than 100 thousand sentences are available. We find that, in very low resource scenarios, statistical n-gram language models outperform state-of-the-art neural models. Our experiments show that this is mainly due to the focus of the former on a local context. As such, we introduce three methods to improve a neural model's performance in the low-resource setting, finding that limiting the model's self-attention is the most effective one, improving on downstream tasks such as NLI and POS tagging by up to 5% for the languages we test on: English, Hindi, and Turkish.
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Unveiling Factors for Enhanced POS Tagging: A Study of Low-Resource Medieval Romance Languages
Fine-tuning open-source LLMs outperforms prompting for POS tagging on medieval Occitan, French, and Spanish, and pooling Romance training data helps the most under-resourced texts.