Localizing AI: Evaluating Open-Weight Language Models for Languages of Baltic States

Jurgita Kapo\v{c}i\=ut\.e-Dzikien\.e; M\=arcis Pinnis; Toms Bergmanis

REVIEW 3 major objections 5 minor 18 references

Localizing AI: Evaluating Open-Weight Language Models for Languages of Baltic States

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Open-weight LLMs can nearly match commercial AI for Baltic languages, but every open model still makes at least one lexical error per 20 words in generated text.

desk verdict Solid benchmark of open-weight LLMs for Baltic languages, but the '1 in 20 words' floor is overstated and should be fixed before publication. read the letter →

arxiv 2501.03952 v1 pith:QEVVU7DJ submitted 2025-01-07 cs.CL cs.AI

Jurgita Kapočiūtė-Dzikienė , Toms Bergmanis , Mārcis Pinnis This is my paper · ORCID

classification cs.CLcs.AI

keywords open-weightLLMsBalticlanguagesmachinetranslationmultiple-choicequestionansweringlexicalhallucinationmodelquantizationlow-resourceGemma2

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether open-weight large language models, which can be deployed on local hardware, are good enough for Lithuanian, Latvian, and Estonian to be used in privacy-sensitive public-sector applications. It benchmarks Llama 3, Gemma 2, Phi 3, and NeMo against commercial cloud services on machine translation, multiple-choice question answering, and free-form generation, using FLORES-200, Belebele, and a human error-counting protocol. The answer is yes for Gemma 2 27B at 4-bit precision, which matches GPT-3.5 Turbo and comes close to GPT-4o and DeepL on translation and reading comprehension. The paper also reports that every open-weight multilingual model it tested produces at least one lexical error per 20 words in free-form answers, while a language-specific fine-tuned model drops that rate to about one percent. A sympathetic reading is that local, sovereign AI for Baltic languages is within reach, but only if applications can tolerate or mitigate a steady trickle of invented and grammatically wrong words.

What carries the argument

The load-bearing mechanism is the three-part evaluation protocol. Translation quality is scored with COMET on the FLORES-200 devtest set; comprehension is scored as accuracy on the Belebele multiple-choice benchmark; and generation quality is measured by having two native-speaker linguists count, per language, the number of words that are grammatically incorrect, wrongly inflected, invented, or syntactically misplaced in answers to ten open-ended prompts. The lexical hallucination claim comes from the last of these: the per-word error rate converts directly to the '1 in 20 words' figure. The other key object is the 4-bit quantization variant, which the paper uses to show that Gemma 2's architecture loses almost nothing when compressed, while Llama's does not.

What would settle it

Run the same ten prompts plus ninety new ones through each open-weight model at default settings, and have three independent native-speaker linguists mark errors; the 'at least 1 in 20 words' claim fails if any non-fine-tuned model's error rate falls below 5 percent with a confidence interval excluding 5 percent.

Watch

Extended reading notes

Core claim

Stated on the paper's own terms, the discovery is that Gemma 2 27B, run at 4-bit quantization, performs close to the top commercially available models across all three Baltic languages: it reaches an average COMET score of 0.89 on FLORES-200 translation (versus 0.90 for GPT-4o and 0.91 for DeepL) and an average Belebele MCQA accuracy of 0.912 (versus 0.944 for GPT-4o and 0.797 for GPT-3.5 Turbo). At the same time, all evaluated open-weight multilingual models, including Gemma 2, show error rates of roughly 4 to 19 percent in free-form text generation in Lithuanian and Latvian as judged by native-speaker linguists, meaning at least one in every 20 words is grammatically incorrect, wrongly inflected, syntactically misplaced, or invented. The exception is a Lithuanian-specific fine-tune of Llama 2, whose error rate is about 1 percent with no invented words. The paper also demonstrates that quantization affects models differently: Gemma 2 suffers negligible drops from 16-bit to 4-bit, whereas Llama 3 loses measurable accuracy, with larger losses for the Baltic languages than for English or Czech.

Load-bearing premise

The binding assumption is that ten free-form questions per language, scored by two native-speaker linguists without reported agreement statistics, produce error rates representative of each model's general output in that language.

Editorial extensions

If this is right

Gemma 2 27B at 4-bit is a practical local replacement for cloud APIs on Baltic-language machine translation and multiple-choice QA, at a fraction of the memory and no data-leakage risk.
Applications that generate free-form Baltic text should not go live with non-fine-tuned open models; the measured lexical error rate of 4 to 19 percent is too high for customer-facing or legal text.
Language-specific fine-tuning, as demonstrated by Lt-Llama 2, can reduce lexical errors to about 1 percent, so building high-quality Baltic-language corpora is the clear path to sovereign AI.
Quantization-friendly architectures like Gemma 2 make local deployment much cheaper; Llama's sensitivity to 4-bit means local Llama deployments should prefer 8-bit or full precision if quality matters.
The consistent gap between Czech and Baltic results suggests training-data volume, not linguistic structure, is the limiting factor for open models in smaller European languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the 1-in-20 word error rate is roughly stable across domains, even 'good' open models need a hallucination-detection layer before use in government documents; the paper's pilot is too small to certify stability, but that is the natural reading of the reported rates.
The Czech–Baltic gap implies a data-volume threshold: once a language crosses a certain number of training tokens, open-model quality jumps; this predicts that future models trained on more Baltic web data will show disproportionate gains on exactly the tasks tested here.
Because the two annotators were linguists, the error counts may be stricter than what ordinary users notice; a user study measuring perceived acceptability could show whether the 1-in-20 bound overstates practical harm.
The quantization-robustness difference between Gemma and Llama suggests internal representation redundancy differs across model families, which could guide future model selection for low-resource multilingual deployment.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Worth a look if you care about how open-weight models actually behave on smaller European languages. The core empirical work is solid: they run four model families across MT and MCQA on Lithuanian, Latvian, and Estonian, using FLORES-200 devtest and Belebele, with COMET for MT. The tables are consistent and the main finding—Gemma 2 27B is the best open-weight family, close to GPT-3.5/4o, and robust to 4-bit quantization—is a genuinely new and useful data point for anyone thinking about sovereign AI deployments in the Baltics. The comparison across model sizes and quantization levels is well done.

The soft spot is exactly where the abstract makes its most striking claim. The statement that open-weight multilingual LLMs produce 'errors in at least 1 in 20 words' does not hold up against their own Table 4: Gemma 2 27B shows 4.08% errors on Lithuanian, which is below the 5% threshold. That table also comes from only ten free-form questions per language, judged by two native speakers, with no inter-annotator agreement, no confidence intervals, and uneven denominators across models. The paper itself admits the factual-accuracy results are a pilot lacking statistical significance, and the same human-eval data underlies the word-error rates. On top of that, the label 'lexical hallucinations' is too broad—annotators counted grammatical and syntactic errors, while the separately reported invented-word rates are much lower (e.g., 0.39 per 100 words for Gemma 2 27B in Lithuanian). So the '1 in 20 words' claim should be treated as preliminary, not a floor.

That said, the problem is confined to the human-eval generalization. The MT and MCQA results, based on standard benchmarks with hundreds of items, are trustworthy and support the ranking. I don't see any circularity or invented results; the citation pattern is appropriate, and the paper is honest about its limitations.

Who is this for? Practitioners in the Baltic states choosing local models, and researchers working on low-resource multilingual evaluation. It's not a methodological breakthrough, but it's a competent benchmark that fills a gap. I'd send it to peer review, but the authors should be required to soften the abstract claim, report the human eval with proper caveats (or expand it), and ideally release the annotation data.

Referee Report

3 major / 5 minor

Summary. This paper reports a systematic evaluation of locally deployable open-weight LLMs (Llama 3/3.1/3.2, Gemma 2, Phi 3, NeMo) for Lithuanian, Latvian, and Estonian, with Czech and English as comparison languages. Using FLORES-200 devtest (1,012 sentences) for machine translation with COMET scores, Belebele (900 questions) for multiple-choice question answering, and a small human evaluation of free-form answers by two native-speaker linguists per language, it compares the open models against GPT-3.5 Turbo, GPT-4o, and DeepL. The main findings are that Gemma 2 27B approaches commercial systems in MT and MCQA, that quantization degrades Gemma 2 less than Llama models, and that open-weight multilingual models remain prone to frequent word-level errors in generated text, summarized in the abstract as 'errors in at least 1 in 20 words for all open-weight multilingual LLMs.'

Significance. The MT and MCQA results are valuable and generally credible: they use standard benchmarks, compare a coherent set of model families and precisions, and consistently identify Gemma 2 as the strongest open-weight family for these languages. The quantization comparison is a useful practical contribution, and the inclusion of Lt-Llama 2 fine-tuned models provides a concrete baseline for language specialization. The paper does not present a new method, and its main novelty is the empirical coverage of three under-resourced languages. The headline lexical-hallucination claim, however, is not supported with the same rigor as the benchmark results, because it rests on a small pilot annotation study with no confidence intervals or inter-annotator agreement. The practical implications for sovereign AI deployments are real if the findings hold, but the paper should state them with appropriate uncertainty.

major comments (3)

[Abstract; §3, Table 4] The universal claim that open-weight multilingual LLMs produce lexical hallucinations with 'errors in at least 1 in 20 words' is not established by Table 4. Gemma 2 27B has a 4.08% error rate on Lithuanian, below the 5% threshold, and all rates derive from only ten free-form questions per language. The paper itself labels the parallel factual-accuracy results a pilot lacking statistical significance; the word-error rates come from the same small sample, yet no confidence intervals, per-question error distributions, or inter-annotator agreement are reported. The denominators also vary substantially across rows (e.g., 300 vs. 1,273 words for Lt-Llama 2 vs. Gemma 2 in Lithuanian), so the reported percentages are sensitive to a few long answers. I recommend reporting confidence intervals or at least per-question ranges and restricting the conclusion to the models actually evaluated.
[§2 (Text Quality) and §4] The term 'lexical hallucinations' overstates what was measured. The evaluation counted grammatically incorrect words, incorrect inflections, invented words, and words in syntactically incorrect structures, but the separately reported invented-word rates are much lower than the total error rates (e.g., Gemma 2 27B: 0.39/100 invented vs. 4.08/100 total in Lithuanian; 1.45 vs. 5.98 in Latvian). The abstract and conclusions should either use a more neutral term such as 'word-level linguistic errors' or restrict the hallucination claim to the invented-word counts.
[§2 and §3 (Table 4)] The generalization from the human evaluation to the Baltic states as a whole is too broad. Table 4 contains no Estonian rows, so the claim about all three languages is unsupported for Estonian, and only three open-weight multilingual models (Llama 3.1 8B/70B and Gemma 2 27B) appear in the table. The conclusion that 'most multilingual models are still surprisingly prone to lexical hallucinations' should be qualified to the evaluated languages and model families, or additional Estonian annotation and more model families should be added.

minor comments (5)

[§3] In the sentence 'As a result, Gamma 2 models show little performance degradation,' 'Gamma 2' should be 'Gemma 2'.
[Table 1] The header contains 'OpenaAI GPT 3.5-Turbo'; correct it to 'OpenAI GPT-3.5 Turbo'.
[§3] The claim that Gemma 2 shows a 'statistically insignificant drop' in MT performance is not backed by a reported significance test; please provide the test or phrase it as an observation about effect size.
[§2] The precision labels '4bit', '8bit', and '16bit' are used inconsistently; consider using '4-bit', '8-bit', and '16-bit' throughout.
[Author affiliations] The affiliation contains 'Lithu ania' with a spurious space; correct it.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports empirical benchmark and human-evaluation measurements against external datasets, not a derivation that reduces to its own inputs.

full rationale

This paper is an empirical evaluation, not a derivation. Its headline results are measurements on external benchmarks (FLORES-200 devtest for machine translation, Belebele for multiple-choice question answering) and manual human annotation of free-form outputs; no parameter is fitted to a target and then re-predicted, and no load-bearing claim is justified by a self-citation chain. The only authorial choices are model selection, prompts, quantization levels, and annotation criteria, which are experimental design decisions rather than circular reasoning. The lexical-hallucination summary is a direct aggregation of Table 4, and its fragility (ten questions per language, no confidence intervals, and a Gemma 2 Lithuanian error rate of 4.08% that falls below the stated 5% floor) is a validity and generalization concern, not a circularity concern. Because the central claims remain independently checkable against the cited benchmarks and reported human judgments, the appropriate finding is no significant circularity, with score 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities, forces, or fitted constants. Its claims rest on standard public benchmarks and on human annotation, so the axiom ledger captures the main domain assumptions about metric validity and sample representativeness.

assumptions (4)

domain assumption COMET-22 scores are valid for ranking translation quality in Lithuanian, Latvian, and Estonian.
Used as the primary MT metric without language-specific validation; COMET is trained largely on high-resource languages and the paper does not test correlation with human judgments for these languages.
domain assumption Word-level error rates measured by two annotators on 10 free-form questions generalize to typical model output for each language.
The paper itself cautions that factual accuracy results lack statistical significance due to small sample size; the lexical error rates come from the same small pilot and no inter-annotator agreement is reported.
domain assumption Belebele multiple-choice questions are equally fair across languages and the single-answer labels are correct.
MCQA accuracy is interpreted directly as language competence; the paper does not analyze whether answer format failures (e.g., Phi models) reflect comprehension or prompt compliance.
domain assumption Ollama default sampling parameters represent a reasonable deployment configuration.
Inference uses Ollama defaults without reporting temperature, top-p, or max tokens; results may vary under different decoding settings.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Localizing AI: Evaluating Open-Weight Language Models for Languages of Baltic States." pith.science (2026). https://pith.science/paper/QEVVU7DJ

@misc{pith2026250103952,
  author       = {Pith},
  title        = {Pith review of: Localizing AI: Evaluating Open-Weight Language Models for Languages of Baltic States},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QEVVU7DJ}},
  note         = {Machine review of arXiv:2501.03952}
}

read the original abstract

Although large language models (LLMs) have transformed our expectations of modern language technologies, concerns over data privacy often restrict the use of commercially available LLMs hosted outside of EU jurisdictions. This limits their application in governmental, defence, and other data-sensitive sectors. In this work, we evaluate the extent to which locally deployable open-weight LLMs support lesser-spoken languages such as Lithuanian, Latvian, and Estonian. We examine various size and precision variants of the top-performing multilingual open-weight models, Llama~3, Gemma~2, Phi, and NeMo, on machine translation, multiple-choice question answering, and free-form text generation. The results indicate that while certain models like Gemma~2 perform close to the top commercially available models, many LLMs struggle with these languages. Most surprisingly, however, we find that these models, while showing close to state-of-the-art translation performance, are still prone to lexical hallucinations with errors in at least 1 in 20 words for all open-weight multilingual LLMs.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 2 canonical work pages

[1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matt...

arXiv 2024
[4]

Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. https://aclanthology.org/2024.acl-long.44 The belebele benchmark: a parallel reading comprehension dataset in 122 language variants . In Proceedings of the 62nd Annual Meeting of the ...

work page 2024
[5]

Costa jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Y

Marta R. Costa jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Y. Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loïc Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon...

arXiv 2022
[6]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany...

arXiv 2024
[7]

Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc'Aurelio Ranzato, Francisco Guzm\' a n, and Angela Fan. 2021. The flores-101 evaluation benchmark for low-resource and multilingual machine translation

2021
[8]

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. http://arxiv.org/abs/2310.06...

arXiv 2023

Show all 18 references

[9]

Tom Kocmi, Vil \'e m Zouhar, Christian Federmann, and Matt Post. 2024. https://doi.org/10.18653/v1/2024.acl-long.110 Navigating the metrics maze: Reconciling score magnitudes and accuracies . In Proceedings of the 62nd Annual Meeting of the Association for Computational Lingui...

2024 doi
[10]

Gemma Team Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, L. Sifre, Morgane Riviere, Mihir Kale, J Christopher Love, Pouya Dehghani Tafti, L'eonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambros...

2024 arXiv
[11]

Kelleher, and Andy Way

Yasmin Moslem, Rejwanul Haque, John D. Kelleher, and Andy Way. 2023. https://aclanthology.org/2023.eamt-1.22 Adaptive machine translation with large language models . In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 227--2...

2023
[12]

Artūras Nakvosas, Povilas Daniušis, and Vytas Mulevičius. 2024. http://arxiv.org/abs/2408.12963 Open llama2 model for the lithuanian language

2024 arXiv
[13]

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

2024 arXiv
[14]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

2002
[15]

Ricardo Rei, Jos \'e G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and Andr \'e F. T. Martins. 2022. https://aclanthology.org/2022.wmt-1.52 COMET -22: Unbabel- IST 2022 submission for the metrics shared task . In Pro...

2022
[16]

Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.213 COMET : A neural framework for MT evaluation . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685--2702, ...

2020 doi
[17]

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan...

2024 arXiv
[18]

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

2023 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matt...

arXiv 2024

[4] [4]

Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. https://aclanthology.org/2024.acl-long.44 The belebele benchmark: a parallel reading comprehension dataset in 122 language variants . In Proceedings of the 62nd Annual Meeting of the ...

work page 2024

[5] [5]

Costa jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Y

Marta R. Costa jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Y. Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loïc Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon...

arXiv 2022

[6] [6]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany...

arXiv 2024

[7] [7]

Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc'Aurelio Ranzato, Francisco Guzm\' a n, and Angela Fan. 2021. The flores-101 evaluation benchmark for low-resource and multilingual machine translation

2021

[8] [8]

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. http://arxiv.org/abs/2310.06...

arXiv 2023