REVIEW 3 major objections 5 minor 20 references
AmaSQuAD: A Benchmark for Amharic Extractive Question Answering
T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read A machine-translated Amharic version of SQuAD 2.0, with answer spans repaired by similarity and proximity, yields a usable QA benchmark and improves Amharic extractive QA when XLM-R is fine-tuned on it.
desk verdict Genuinely new Amharic QA dataset and a workable pipeline, but the benchmark's validity is undercut by circular validation and the cross-lingual gain is too small to confirm the method. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is Algorithm 1, the proximity-similarity-LCS answer extractor. For each translated question, it translates the English answer, splits the translated context into words, and slides a window of the answer's word length across the context with strides 0 through 3. Each candidate span receives a score $$s = w_1 \cdot \cos(\text{emb}(\text{span}), \text{emb}(\text{answer})) + w_2 \cdot \frac{\text{LCS}(\text{span}, \text{answer})}{\max(|\text{span}|, |\text{answer}|)}$$ with $w_1 = 2/3$ and $w_2 = 1/3$, and the best-scoring span is kept only if it is also the closest to the original English answer position. This weighted combination is what repairs the misalignment that raw machine translation introduces, and the proximity heuristic is what disambiguates multiple answer instances. A second mechanism is dataset filtering: only answers with similarity at least 0.6 are kept, and unanswerable questions are downsampled to keep the training set tractable.
What would settle it
Construct a control set of Amharic translations in which the answer has been deliberately moved far from its English position; if Algorithm 1 still selects the span nearest the English position rather than the true translated answer, the proximity prior is producing systematic label errors.
Extended reading notes
Core claim
The central claim is that answer alignment after machine translation is the real obstacle to creating QA data in low-resource languages, and that a cheap geometric proxy can overcome it. Translating SQuAD 2.0 into Amharic with machine translation does not preserve exact answer spans, so the paper's Algorithm 1 scans the translated context with a window the size of the translated answer, scores each window by $2/3$ cosine similarity (from a BERT model fine-tuned on Amharic) plus $1/3$ LCS-based string similarity, and selects the highest-scoring span whose position is closest to the English answer's position. The paper claims this proximity prior is reliable because the answer ordering in the translated Amharic context does not deviate significantly from the original English context. On the resulting filtered AmaSQuAD dataset, fine-tuning XLM-R large raises EM/F1 from 36.55/50.01 to 44.41/57.55 on AmaSQuAD development and from 52.50/67.80 to 52.66/68.80 on AmQA, which the authors take as evidence that synthetic translated data can support Amharic extractive QA.
Load-bearing premise
The pipeline assumes the answer's location in the Amharic translation stays close to its location in the original English passage, so location can guide span selection.
Editorial extensions
If this is right
- AmaSQuAD gives Amharic its first large extractive QA training set, with about 57,650 filtered answerable training questions plus downsampled unanswerable questions in SQuAD 2.0 format.
- Fine-tuning on AmaSQuAD is learnable and effective in-domain: EM jumps 7.86 points and F1 7.54 points on AmaSQuAD dev.
- The synthetic data transfers, though modestly, to human-curated questions: AmQA EM rises 0.16 points and F1 1.00 point, so training on translated data does not appear to hurt real-language QA.
- The same translate-align-filter recipe is proposed for other low-resource languages and other extractive QA source datasets.
Reading between the lines
- The proximity prior is the fragile link: for languages whose grammar moves answers further from their English positions, Algorithm 1 would likely select the wrong span even when the similarity signal is decent.
- The reported gains compare XLM-R large with and without fine-tuning on the same base; a natural extension is to benchmark mBERT or XLM-R base on the same splits to see whether the improvement is about data quality or model capacity.
- Because unanswerable questions were downsampled to 6,000/700, an evaluation set with a realistic mix of answerable and unanswerable Amharic questions would be needed to test whether AmaSQuAD actually teaches rejection of unanswerable questions.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces AmaSQuAD, an Amharic extractive question-answering benchmark created by translating SQuAD 2.0 with Google Translate. To cope with translation-induced misalignment, the authors propose Algorithm 1, which scores candidate answer spans in the translated context by a weighted combination of cosine similarity and LCS, while preferring spans near the original English answer position. The dataset is filtered to answer spans with similarity at least 0.6, and unanswerable questions are downsampled to 6,000 training and 700 development instances. The authors then fine-tune XLM-R large on this synthetic dataset and report EM/F1 improvements on AmaSQuAD dev (36.55/50.01 to 44.41/57.55) and on the human-curated AmQA dataset (52.50/67.80 to 52.66/68.80). The paper claims this demonstrates a framework for translating extractive QA datasets into low-resource languages.
Significance. If the dataset labels are of sufficient quality, AmaSQuAD would be a valuable and much-needed resource: it is, to the authors' knowledge, the first large-scale Amharic extractive QA dataset, and it is publicly released. The paper also provides a reproducible pipeline and establishes a detailed baseline on the existing AmQA dataset, which previous work had not reported. These strengths are real and should be credited. However, the significance of the central claims—that AmaSQuAD is a usable benchmark and that fine-tuning on it improves Amharic QA—depends on label quality and on the reliability of the reported gains, and the current evidence for both is incomplete.
major comments (3)
- [Section III-B, Algorithm 1, Figures 1-2] The load-bearing premise of the alignment algorithm is that the English answer position is a reliable prior for locating the Amharic answer, stated in Section III-B as the intuition that 'the answer ordering in the translated Amharic context won't deviate significantly from the answer in the original English context.' This premise is never tested against human judgment. The only quality evidence, Figures 1 and 2, plots the distribution of the same weighted cosine/LCS score that Algorithm 1 uses to select spans, so high similarity is partly by construction. Because the same automatically produced labels are used for both training and evaluation on AmaSQuAD dev, a systematic alignment failure would inflate the reported gains and invalidate the benchmark. The paper needs an independent validation of a sample of answer spans, e.g., human annotation of a few hundred instances or comparison against an alternative alignment method, to establish that the labels are correct.
- [Section IV, Table 1] All reported numbers come from single training runs with no error bars or significance testing. The improvements on the human-curated AmQA dataset are +0.16 EM and +1.00 F1, which are within the run-to-run variation typically observed when fine-tuning transformer models. Since the claim that fine-tuning on AmaSQuAD improves real Amharic QA rests on these small differences, the authors should report means and standard deviations over multiple seeds, or at least show that the difference is stable across hyperparameter settings. Without this, the AmQA improvement cannot be distinguished from noise.
- [Section IV (dataset filtering and downsampling)] The similarity filter threshold of 0.6 and the downsampling of unanswerable questions to 6,000/700 are arbitrary choices that materially change the task distribution. In SQuAD 2.0, unanswerable questions are roughly half of the dataset, whereas in AmaSQuAD they become about 9.4% of the training set and 4.8% of the development set. This affects both the difficulty of the benchmark and the meaning of the EM/F1 numbers. The paper should justify these choices, report sensitivity to the threshold, and clarify whether the development-set performance reflects the intended evaluation protocol for the released dataset.
minor comments (5)
- [Abstract and Table 1] The abstract reports the post-fine-tuning F1 on AmaSQuAD dev as 57.5%, while Table 1 reports 57.55%; these should be made consistent.
- [Algorithm 1] The pseudocode uses undefined functions such as extw, st, and prox, and the stride loop is not clearly explained. Please define these terms and clarify how the proximity prior interacts with the similarity score when multiple candidate spans tie.
- [Section VI] The conclusion contains an incomplete sentence ending with 'from 36.55' and should be finished or removed.
- [Throughout] The dataset name is spelled inconsistently as AmaSQuAD, AmaSQUAD, and AmSQuAd; please standardize it.
- [References] Reference [12] (Abedissa and Libsie) is cited in the related-work section as '[7] developed a non-factoid...' but the citation number does not match; the in-text citation appears to be incorrect.
Circularity Check
AmaSQuAD's benchmark-quality evidence is the same similarity objective that generated its answer spans, so the quality claim is partly circular; the fine-tuning claim retains some independent grounding via AmQA.
-
fitted input called prediction
[Section III-B (Algorithm 1), Section IV, Section V-A (Figures 1-2)]
"The similarity scores are used to select the answer span in the context. ... Figures 1 and 2 show the similarity distribution results ... of translated answers with the translated context span for AmaSQuAD. ... This indicates more accurate translations closely aligned with the original context in the translated context that have high similarity ranges."
Algorithm 1 selects the translated answer span by maximizing a weighted combination of cosine similarity and LCS (w1=2/3, w2=1/3), also prioritizing proximity to the original English answer position. Section IV then filters answers using a similarity threshold of 0.6. The only reported quality evidence for AmaSQuAD, Figures 1 and 2, plots the distribution of that same similarity score for the selected spans and interprets the resulting peak at 0.9-1.0 as indicating 'more accurate translations.' Because the span extractor is explicitly rewarded for producing high similarity scores, the reported distribution is a property of the optimization objective, not an independent verification of the extracted spans.
full rationale
The derivation chain for the central benchmark-quality claim is not fully self-contained. Algorithm 1 computes a weighted cosine-similarity/LCS score to find the answer span, and a similarity filter is used to keep 'higher quality' examples. The only quantitative evidence that the resulting spans are correct is the distribution of this same score, with high-scoring translations declared to be 'more accurate translations closely aligned with the original context.' That is a circular validation: the statistic offered as evidence of accuracy is the objective the label generator was designed to maximize, so elevated scores are expected for selected examples and cannot independently confirm that the spans are correct in Amharic. The paper even concedes that human feedback on the quality of the translated dataset is 'essential,' but no such human evaluation is reported. The fine-tuning results are less circular because the improvement on the human-curated AmQA dataset is an external, independent benchmark, albeit with small gains (+0.16 EM, +1.00 F1); the AmaSQuAD dev improvement, in contrast, is measured on labels produced by the same pipeline. There are no load-bearing self-citations and no uniqueness-theorem smuggling. The circularity is therefore partial: it affects the benchmark-quality evidence, a load-bearing premise, while the fine-tuning claim retains some independent grounding.
Assumptions & free parameters
free parameters (5)
- w1 (cosine similarity weight) =
2/3
- w2 (LCS weight) =
1/3
- similarity filter threshold =
0.6
- unanswerable downsample counts =
6,000 train / 700 dev
- training hyperparameters =
3 epochs, learning rate 1e-5, max sequence length 256
assumptions (5)
- domain assumption Google Translate produces Amharic translations that preserve question and context semantics well enough for extractive QA training.
- domain assumption The answer's position in the original English context is a reliable proxy for its position in the translated Amharic context.
- domain assumption Cosine similarity of fine-tuned mBERT embeddings and LCS overlap together measure whether a translated span matches the translated answer.
- domain assumption XLM-R has sufficient Amharic competence from pretraining to benefit from Amharic fine-tuning data.
- ad hoc to paper Downsampling unanswerable questions to 6,000/700 preserves a realistic QA distribution.
Cite this review
Pith. "Pith review of AmaSQuAD: A Benchmark for Amharic Extractive Question Answering." pith.science (2026). https://pith.science/paper/VVDM2POT
@misc{pith2026250202047,
author = {Pith},
title = {Pith review of: AmaSQuAD: A Benchmark for Amharic Extractive Question Answering},
year = {2026},
howpublished = {\url{https://pith.science/paper/VVDM2POT}},
note = {Machine review of arXiv:2502.02047}
}
read the original abstract
This research presents a novel framework for translating extractive question-answering datasets into low-resource languages, as demonstrated by the creation of the AmaSQuAD dataset, a translation of SQuAD 2.0 into Amharic. The methodology addresses challenges related to misalignment between translated questions and answers, as well as the presence of multiple answer instances in the translated context. For this purpose, we used cosine similarity utilizing embeddings from a fine-tuned BERT-based model for Amharic and Longest Common Subsequence (LCS). Additionally, we fine-tune the XLM-R model on the AmaSQuAD synthetic dataset for Amharic Question-Answering. The results show an improvement in baseline performance, with the fine-tuned model achieving an increase in the F1 score from 36.55% to 44.41% and 50.01% to 57.5% on the AmaSQuAD development dataset. Moreover, the model demonstrates improvement on the human-curated AmQA dataset, increasing the F1 score from 67.80% to 68.80% and the exact match score from 52.50% to 52.66%.The AmaSQuAD dataset is publicly available Datasets
Figures
Reference graph
Works this paper leans on
-
[1]
Improving part-of-speech tagging in amharic language using deep neural network,
S. Hirpassa and G. Lehal, “Improving part-of-speech tagging in amharic language using deep neural network,” Heliyon, vol. 9, no. 7, 2023
work page 2023
-
[2]
S. J. Basha, D. Veeraiah, B. V . Charan, W. S. J. Yeddu, and D. G. Babu, “Detection and comparative analysis of handwritten words of amharic language to english using cnn-based frameworks,” in 2023 International Conference on Inventive Computation Technologies (ICICT) . IEEE, 2023, pp. 422–427
work page 2023
-
[3]
Re- trieving and reading: A comprehensive survey on open-domain question answering,
F. Zhu, W. Lei, C. Wang, J. Zheng, S. Poria, and T.-S. Chua, “Re- trieving and reading: A comprehensive survey on open-domain question answering,” arXiv preprint arXiv:2101.00774 , 2021
arXiv 2021
-
[4]
Ms marco: A human-generated machine reading compre- hension dataset,
T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, and L. Deng, “Ms marco: A human-generated machine reading compre- hension dataset,” 2016
work page 2016
-
[5]
Teaching machines to read and comprehend,
K. M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Su- leyman, and P. Blunsom, “Teaching machines to read and comprehend,” Advances in neural information processing systems , vol. 28, 2015
2015
-
[6]
Know what you don’t know: Unanswerable questions for squad,
P. Rajpurkar, R. Jia, and P. Liang, “Know what you don’t know: Unanswerable questions for squad,” arXiv preprint arXiv:1806.03822 , 2018
arXiv 2018
-
[7]
Amqa: amharic question answering dataset,
T. Abedissa, R. Usbeck, and Y . Assabie, “Amqa: amharic question answering dataset,” arXiv preprint arXiv:2303.03290 , 2023
arXiv 2023
-
[8]
Unsupervised cross-lingual representation learning at scale,
A. Conneau, “Unsupervised cross-lingual representation learning at scale,” arXiv preprint arXiv:1911.02116 , 2019
arXiv 1911
Show all 20 references
-
[9]
Mlqa: Evaluating cross-lingual extractive question answering,
P. Lewis, B. O ˘guz, R. Rinott, S. Riedel, and H. Schwenk, “Mlqa: Evaluating cross-lingual extractive question answering,” arXiv preprint arXiv:1910.07475, 2019
1910 arXiv
-
[10]
Automatic span- ish translation of the squad dataset for multilingual question answering,
C. P. Carrino, M. R. Costa-Juss `a, and J. A. Fonollosa, “Automatic span- ish translation of the squad dataset for multilingual question answering,” arXiv preprint arXiv:1912.05200 , 2019
1912 arXiv
-
[11]
efmaral,
R. O. stling, “efmaral,” https://github.com/robertostling/efmaral, Oct. 2023, accessed: Dec. 10, 2023
2023
-
[12]
Amharic question answering for biography, definition, and description questions,
T. Abedissa and M. Libsie, “Amharic question answering for biography, definition, and description questions,” in Information and Communi- cation Technology for Development for Africa: Second International Conference, ICT4DA 2019, Bahir Dar , Ethiopia, May 28-30, 2019, Revised ...
2019
-
[13]
Parsquad: machine translated squad dataset for persian question answering,
N. Abadani, J. Mozafari, A. Fatemi, M. A. Nematbakhsh, and A. Kazemi, “Parsquad: machine translated squad dataset for persian question answering,” in 2021 7th International Conference on Web Research (ICWR). IEEE, 2021, pp. 163–168
2021
-
[14]
Boolq: Exploring the surprising difficulty of natural yes/no questions,
C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova, “Boolq: Exploring the surprising difficulty of natural yes/no questions,” arXiv preprint arXiv:1905.10044 , 2019
1905 arXiv
-
[15]
squad v2 · datasets at hugging face,
“squad v2 · datasets at hugging face,” https://huggingface.co/datasets/ squad v2, accessed: Dec. 19, 2023
2023
-
[16]
Davlan/bert-base-multilingual-cased-finetuned-amharic,
“Davlan/bert-base-multilingual-cased-finetuned-amharic,” https:// huggingface.co/Davlan/bert-base-multilingual-cased-finetuned-amharic, accessed: Dec. 10, 2023
2023
-
[17]
T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein, Introduction to algorithms . MIT press, 2022
2022
-
[18]
BERT: Pre- training of deep bidirectional transformers for language understanding,
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre- training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog...
2019
-
[19]
Roberta: A robustly optimized bert pretraining approach,
Y . Liu, “Roberta: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692, vol. 364, 2019
1907 arXiv
-
[20]
How multilingual is multilingual BERT?
T. Pires, E. Schlinger, and D. Garrette, “How multilingual is multilingual BERT?” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , A. Korhonen, D. Traum, and L. M `arquez, Eds. Florence, Italy: Association for Computational Linguisti...
2019
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.