Pith. sign in

REVIEW 2 major objections 5 minor 25 references

FoQA: A Faroese Question-Answering Dataset

T0 review · 2 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces FoQA, a 2,000-pair extractive question-answering dataset for Faroese built with GPT-4 and native-speaker validation, plus baseline scores for nine models.

desk verdict First real Faroese QA benchmark with released code and data, but Section 5's numbers do not add up and need to be fixed before the 2,000-sample claim holds. read the letter →

arxiv 2502.07642 v1 pith:Z4MMCVHB submitted 2025-02-11 cs.CL cs.LG

Annika Simonsen , Dan Saattrup Nielsen , Hafsteinn Einarsson This is my paper · ORCID
classification cs.CLcs.LG
keywords Faroeseextractivequestionansweringlow-resourceNLPQAbenchmarkGPT-4-turbonative-speakervalidationWikipediadatasetrelease
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FoQA is an extractive question-answering dataset for Faroese, a low-resource language with few evaluation resources: it contains 2,000 question-answer pairs drawn from Faroese Wikipedia, with every answer appearing verbatim in the source article. The paper argues that this is the first QA benchmark for Faroese and that it can measure how well language models understand the language, while also demonstrating a semi-automated recipe that needs only one native-speaker annotator rather than a large annotation team. If the dataset is sound, it gives the Faroese language-technology community a shared evaluation set, gives model developers concrete baseline scores to beat, and offers other low-resource languages a low-effort path to building their own QA benchmarks. The paper reports baselines for nine models, with GPT-4-turbo at F1 77.6 and BERT-style encoders around 30, and releases both the validated pairs and the rejected samples for error analysis.

What carries the argument

The mechanism that carries the argument is the generation-rephrasing-validation pipeline. A generative model is prompted to return, for each Wikipedia article, 2-10 questions and verbatim answers in a strict JSON structure; outputs that break the format are discarded automatically. A second prompt rewrites each question using synonyms or different word order while keeping the answer fixed, to reduce surface overlap between question and source text. Then a single native Faroese annotator applies three successive checks (grammar, meaning and context, final label) to each candidate, and every Correct item is reviewed a second time by the original annotator or a second native speaker. The 241 corrected samples are included in the released benchmark so model performance on machine-generated versus corrected questions can be compared.

What would settle it

Sample, say, 200 question-answer pairs from FoQA's validated set, have two independent native Faroese speakers re-apply the paper's three-tier annotation rules, and count how often they mark a published pair as not Correct; if disagreement is substantial (around 10 percent or more), the benchmark's gold labels are too noisy for reliable model ranking. A separate check would be to run a simple word-overlap baseline that selects the source sentence most similar to the question: if it approaches GPT-4-turbo's F1, the rephrasing did not remove the surface-matching weakness the paper set out to fix.

Watch

Extended reading notes

Core claim

The paper's central contribution is FoQA, a Faroese extractive question-answering dataset of 2,000 validated samples in which every answer appears verbatim in a Faroese Wikipedia article. The samples were generated by gpt-4-turbo-2024-04-09, then a rewriting step paraphrased the questions to reduce simple text matching, and native-speaker validation filtered and corrected the results. The paper reports that on this benchmark GPT-4-turbo reaches F1 77.6 and exact match 55.6, while BERT-style encoder models score near 30, and it releases three versions of the dataset: 2,000 validated samples, all 10,001 generated samples, and 2,395 rejected samples. It also releases the open-source generation and annotation code, with the claim that the same single-annotator pipeline can create QA datasets for other low-resource languages.

Load-bearing premise

The benchmark's usefulness depends on the native-speaker reviewers having labelled the 2,000 released pairs correctly, even though the initial pass was made by a single annotator and no inter-annotator agreement was measured.

Editorial extensions

If this is right

  • Faroese QA evaluation now has a shared, SQuAD-compatible test set, so results from different models and future systems can be compared directly.
  • Published baselines give concrete targets: the strongest decoder model scores F1 77.6, while encoder models sit near 30, and any claimed improvement needs to be measured against these numbers.
  • The 2,395 rejected samples enable error analysis of LLM-generated Faroese, revealing patterns such as grammatical gender errors and Icelandic-influenced wording.
  • The 241 corrected samples offer a built-in control for measuring bias toward machine-generated question phrasing, and the paper finds the generator scored higher on corrected questions than on its own.
  • Releasing all 10,001 generated pairs lets other researchers train or fine-tune on the raw data instead of only the cleaned subset.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: because inter-annotator agreement was not measured, the 2,000 gold labels rest on one initial reviewer's judgment; re-annotating a random subset with two independent native speakers would quantify how much label noise the baselines contain.
  • Editorial extension: the paper never compared rephrased questions to the original ones, so we do not yet know whether the rephrasing step actually increased difficulty; a head-to-head evaluation on both versions would test the method's core assumption.
  • Editorial extension: if the pipeline is adopted for other languages, the corrected-question result warns that an LLM's own questions may be a hidden confound in the benchmark, so released datasets should always include rejected and corrected subsets alongside the validated one.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces FoQA, a Faroese extractive question-answering dataset constructed semi-automatically from Faroese Wikipedia using GPT-4-turbo for question and answer generation, question rephrasing, and native-speaker validation. The authors report a validated set of 2,000 QA pairs, release three dataset versions, evaluate nine models (LLMs and encoder-only models) on the benchmark, analyze question type distribution, and discuss qualitative error patterns. The central claim is that FoQA is the first Faroese QA evaluation resource and serves as a useful benchmark for Faroese language understanding.

Significance. If the central claim holds, FoQA fills a clear gap: no dedicated Faroese extractive QA benchmark currently exists, and the paper provides a reproducible semi-automated pipeline, open-source code, released datasets, and baseline scores across nine models. These are meaningful strengths for a low-resource language resource. The impact is moderate: the dataset is small (2,000 validated pairs) and the evaluation is not accompanied by human performance or inter-annotator agreement, but the resource itself is potentially useful for the NLP community.

major comments (2)
  1. [Sections 5.2 and 5.3] The reported sample counts do not reconcile. Section 5.2 reports 4,130 annotated samples with 1,759 Correct, 1,908 Incorrect, and 222 Incorrect Answer, and then says that 241 samples have the label Corrected, including 41 of the 1,759 Correct samples. If Corrected is a disjoint fourth category, the arithmetic 1,759 + 1,908 + 222 + 241 = 4,130 is consistent, but then the statement that the 41 are among the 1,759 Correct is contradictory. If the 41 overlap, then the default set described in Section 5.3 as 1,759 initially correct plus 241 corrected contains only 1,959 unique pairs, not 2,000. Section 5.3 also states that the incorrect-samples version has 2,395 rejected examples, but the manual review can account for only 1,908 + 222 = 2,130 annotated rejects (or 2,171 if the 41 double-counted samples are removed from the validated set). The composition of the released default and incorrect sets must be clarified before the central claim of a 2,000-sample validated benchmark can be accepted.
  2. [Sections 3.2, 3.3, and 5.2; Limitations] The validation protocol does not provide a quantitative measure of annotation reliability. Initial validation is performed by a single annotator, and the second pass is applied only to samples initially marked Correct; moreover, that second pass is split between the original annotator and a second annotator, so half of the Correct samples are re-reviewed by the same person who made the original judgment. The Limitations section acknowledges that traditional inter-annotator agreement measurements were not performed, but the abstract and conclusion nevertheless describe the dataset as 'validated' and as a 'valuable benchmark.' I would ask the authors to either add a small inter-annotator agreement study (for example, on a random subset of 100-200 samples) or to soften the validation claim in the abstract and conclusion so that it accurately reflects the single-annotator-plus-partial-review protocol.
minor comments (5)
  1. [Section 1] In the second contribution bullet, the phrase 'F aroeseusing' appears to be a spacing error and should read 'Faroese using.'
  2. [References] The bibliography entries for TyDi QA (Clark et al., 2020) and DeBERTa (He et al., 2021b) are truncated; the full titles should be restored.
  3. [Section 4] The text says 'with a consistent random seed (4242) to maintain reproducibility,' but it does not specify how the seed is passed to the OpenAI API; please clarify the exact API parameter used or state that reproducibility refers to the codebase rather than the model outputs.
  4. [Section 6] The description of the GPT-4-turbo comparison as 'before and after the correction' is confusing; please clarify that the model is evaluated on the original and corrected versions of the same questions, and note that the reported p-values would not survive a strict multiple-comparison correction, although this does not affect the main qualitative conclusion.
  5. [Section 5.1] The dataset format includes an answer_start character index, but the paper does not state whether the shortened contexts used in the default split were created before or after computing answer_start indices; please specify this for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FoQA is a human-validated dataset artifact, and its baselines are standard external evaluations.

full rationale

The paper's central claim is the existence and usefulness of a 2,000-sample Faroese extractive QA dataset. The construction chain is generation by GPT-4-turbo, reformulation, single-annotator filtering, and a second pass on initially correct items. None of these steps fits a parameter to a target quantity and then reports that quantity as a prediction; there are no fitted coefficients, uniqueness theorems, or ansatz rescaling. The main self-citations (Nielsen 2023; Nielsen et al. 2024; Snæbjarnarson and Einarsson 2022a, 2022b; Snæbjarnarson et al. 2023) are used for model selection, related work, or prior Faroese benchmarks; they are external, independently checkable resources and are not invoked to force the paper's conclusions. The one genuine overlap—evaluating GPT-4-turbo on questions it generated—is disclosed and directly probed: the paper reports that GPT-4-turbo scores significantly higher on human-corrected questions (p = 0.0007 for F1, p = 0.0185 for EM), which is the opposite of a self-confirming result. The Limitations section explicitly states 'we could not perform traditional inter-annotator agreement measurements,' and also acknowledges the lack of a rephrased-versus-original comparison; these are quality and validity limitations, not circularity. The reported discrepancies among annotation counts (4,130 annotated vs. 2,395 rejected vs. 2,000 validated) are internal-consistency and release-composition concerns, not cases where an output is equivalent to an input by construction. Therefore the derivation chain is self-contained and no circular step can be exhibited.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted to data: the dataset is a resource, not a model with fitted coefficients. The experimental settings (temperature 1.0, seed 4242, token limit 1,024, question-count range 2-10, length threshold 1,000 characters) are reproducible configuration choices, not parameters tuned to produce the final scores. The axioms listed are domain assumptions inherited from the method and the source corpus.

assumptions (3)
  • domain assumption Faroese Wikipedia articles over 1,000 characters are suitable source contexts for QA pairs.
    Section 4 selects only articles above the threshold; if these articles are not linguistically sound, the dataset inherits errors, as the authors acknowledge when they say Wikipedia occasionally contains ungrammatical content.
  • domain assumption GPT-4-turbo generates sufficiently fluent Faroese to be worth human editing.
    Section 4 chooses GPT-4-turbo based on ScandEval performance; the generation step assumes the model's Faroese is adequate, with human filtering meant to catch residual errors.
  • domain assumption A single native annotator, plus a partial second pass, yields correct labels.
    Sections 3.2 and 5.2 rely on this protocol; the paper explicitly notes no inter-annotator agreement could be measured (Limitations).

how reviews work

0 comments
Cite this review

Pith. "Pith review of FoQA: A Faroese Question-Answering Dataset." pith.science (2026). https://pith.science/paper/Z4MMCVHB

@misc{pith2026250207642,
  author       = {Pith},
  title        = {Pith review of: FoQA: A Faroese Question-Answering Dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z4MMCVHB}},
  note         = {Machine review of arXiv:2502.07642}
}
read the original abstract

We present FoQA, a Faroese extractive question-answering (QA) dataset with 2,000 samples, created using a semi-automated approach combining Large Language Models (LLMs) and human validation. The dataset was generated from Faroese Wikipedia articles using GPT-4-turbo for initial QA generation, followed by question rephrasing to increase complexity and native speaker validation to ensure quality. We provide baseline performance metrics for FoQA across multiple models, including LLMs and BERT, demonstrating its effectiveness in evaluating Faroese QA performance. The dataset is released in three versions: a validated set of 2,000 samples, a complete set of all 10,001 generated samples, and a set of 2,395 rejected samples for error analysis.

Figures

Figures reproduced from arXiv: 2502.07642 by the authors.

Figure 1
Figure 1. Overview of the QA dataset generation pipeline. Th [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 12 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Abubakar Abid, Ali Abdalla, Ali Abid, Dawood Khan, Abdulrahman Alfozan, and James Zou. 2019. Gradio: H assle- F ree S haring and T esting of ML M odels in the W ild. arXiv preprint arXiv:1906.02569

    arXiv 2019

  4. [4]

    Anthropic. 2024. Claude 3.5 S onnet. https://www.anthropic.com. Proprietary software, closed-source

    work page 2024

  5. [5]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    work page 2020

  6. [6]

    Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki

    Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. https://doi.org/10.1162/tacl_a_00317 Ty Di QA : A B enchmark for I nformation- S eeking Q uestion A nswering in T ypologically D iverse L anguages . Transactions of the Association for Computational Linguistics, 8:454--470

    doi:10.1162/tacl_a_00317 2020

  7. [7]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised C ross-lingual R epresentation L earning at S cale. arXiv preprint arXiv:1911.02116. *Equal contribution for first two authors

    arXiv 2019

  8. [8]

    Marta R. Costa-Jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Sp...

    arXiv 2022

Show all 25 references
  1. [9]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT : P re-training of D eep B idirectional T ransformers for L anguage U nderstanding. arXiv preprint arXiv:1810.04805

    2018 arXiv

  2. [10]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...

    2024 arXiv

  3. [11]

    Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. https://doi.org/10.18653/v1/P19-1346 ELI 5: L ong F orm Q uestion A nswering . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558-...

    2019 doi

  4. [12]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval- A ugmented G eneration for L arge L anguage M odels: A S urvey. arXiv preprint arXiv:2312.10997

    2023 arXiv

  5. [13]

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021 a . De BERT a V 3: I mproving D e BERT a using ELECTRA - S tyle P re- T raining with G radient- D isentangled E mbedding S haring. arXiv preprint arXiv:2111.09543

    2021 arXiv

  6. [14]

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021 b . https://openreview.net/forum?id=XPZIaotutsD De BERT a: D ecoding-enhanced BERT with D isentangled A ttention . In International Conference on Learning Representations

    2021

  7. [15]

    Dan Nielsen. 2023. Scandeval: A B enchmark for S candinavian N atural L anguage P rocessing. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 185--201

    2023

  8. [16]

    Dan Saattrup Nielsen, Kenneth Enevoldsen, and Peter Schneider-Kamp. 2024. Encoder vs D ecoder: C omparative A nalysis of E ncoder and D ecoder L anguage M odels on M ultilingual NLU T asks. arXiv preprint arXiv:2406.13469

    2024 arXiv

  9. [17]

    OpenAI. 2023. https://openai.com/index/new-models-and-developer-products-announced-at-devday/ New models and developer products announced at DevDay

    2023

  10. [18]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD : 100,000+ Q uestions for M achine C omprehension of T ext. In In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383--2392, Austin, Texas. Association...

    2016

  11. [19]

    Njall Skarphedinsson, Breki Gudmundsson, Steinar Smari, Marta Kristin Larusdottir, Hafsteinn Einarsson, Abuzar Khan, Eric Nyberg, and Hrafn Loftsson. 2023. https://doi.org/10.18653/v1/2023.eacl-demo.18 G ame QA : G amified M obile A pp P latform for B uilding M ultiple- D omai...

    2023 doi

  12. [20]

    V \'e steinn Sn bjarnarson and Hafsteinn Einarsson. 2022 a . https://doi.org/10.18653/v1/2022.mia-1.4 Cross- L ingual QA as a S tepping S tone for M onolingual O pen QA in I celandic . In Proceedings of the Workshop on Multilingual Information Access (MIA), pages 29--36, Seatt...

    2022 doi

  13. [21]

    V \'e steinn Sn bjarnarson and Hafsteinn Einarsson. 2022 b . https://aclanthology.org/2022.lrec-1.477 Natural Q uestions in I celandic . In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4488--4496, Marseille, France. European Language Resour...

    2022

  14. [22]

    V \'e steinn Sn bjarnarson, Annika Simonsen, Goran Glava s , and Ivan Vuli \'c . 2023. https://aclanthology.org/2023.nodalida-1.74 Transfer to a L ow- R esource L anguage via C lose R elatives: T he C ase S tudy on F aroese . In Proceedings of the 24th Nordic Conference on Com...

    2023

  15. [23]

    Akchay Srivastava and Atif Memon. 2024. Towards R obust E valuation: A C omprehensive T axonomy of D atasets and M etrics for O pen D omain Q uestion A nswering in the E ra of L arge L anguage M odels. IEEE Access

    2024

  16. [24]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention I s A ll Y ou N eed. Advances in Neural Information Processing Systems, 30

    2017

  17. [25]

    Brandon T Willard and R \'e mi Louf. 2023. Efficient G uided G eneration for L arge L anguage M odels. arXiv preprint arXiv:2307.09702

    2023 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.