{"id":"48ce2fea-6d02-4254-abb3-73e6e0e39e82","arxiv_id":"2501.09164","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Open-source LLMs mostly detect fine-grained matched versus non-matched short answers in Latvian and Lithuanian, with 70b-class models near-perfect and smaller models showing variable, model-specific weaknesses.","lead":"This paper builds new Latvian and Lithuanian datasets of short question-answer pairs where the 'right' and 'wrong' answers look very similar, and tests how well several open-source AI models can tell them apart. The authors find that the biggest models are nearly perfect at this, while smaller models are much more inconsistent, especially in Lithuanian.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central F1 results depend on class-level acceptance of dataset labels from a small manual sample; an item-level annotation audit of the released dataset is needed before the headline accuracy claims can be trusted.","rationale":"The paper makes a useful, bounded empirical contribution: it releases new Latvian/Lithuanian answer-matching benchmarks and reports F1 scores for several open LLMs. The manual verification step is a genuine strength, but it is used as a class-level filter rather than as ground-truth labeling for the released items. Since every downstream conclusion in Table 1 and the Conclusion is an accuracy claim on those labels, an audit of the actual test labels is the fastest way to determine whether the numbers are trustworthy. I do not see evidence of fabrication or of an internally inconsistent argument; the issue is that the currently provided evidence is insufficient to rule out label noise as an explanation of the smaller model differences. The reader's conditional verdict already captures this concern, so I keep the verdict unchanged.","tokens_in":18233,"tokens_out":7398,"duration_ms":77325,"concrete_test":"Release the full dataset with item-level labels and have two independent native-speaking annotators per language re-annotate a stratified random sample of at least 500 final test items per language without being told the intended label, adjudicate disagreements, and report per-rule label error rates and Cohen's kappa. Then recompute Table 1's F1 scores on the subset of items whose labels are confirmed. If label error rates exceed a few percent or correlate with generation model or alteration rule, the large-versus-small model conclusions should be re-evaluated; if errors are rare and random, the central claim stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 and Appendix A describe the only quality check on the benchmark labels: two native-speaking annotators per language reviewed sample triplets, and entire generation classes were then kept or dropped based on aggregate accept/reject counts (Table 2). This means the final dataset was never labeled at item level. For Lithuanian, all GPT-4o classes and LLaMa3 Non-Match Relat were kept after observing only 5/240 and 2/60 rejects in the sample; for Latvian, GPT-4o classes and LLaMa3 Non-Match Minor-Changes were kept, with GPT-4o MoreInfo excluded by mistake. The manual sample was drawn before duplicate removal and before this selective class filtering, so the observed reject rates do not directly estimate label error on the final test items. If a generation model produced a cluster of mislabeled items that passed the small aggregate check, all of those items enter the benchmark, and models that agree with the intended label are scored correct even when the label is wrong. The observed score differences (e.g., Mistral Nemo 12b 0.94-0.96 vs Qwen2.5 72b 0.99, or Llama3.1 8b 0.87-0.89 in zero-shot) are small enough that a few percent of systematically placed label noise could shift model rankings. No confidence intervals, per-rule item counts, or inter-annotator agreement are reported. Because the headline qualitative conclusion—that larger open models are near-perfect while smaller models vary—is read directly from these F1 numbers, label correctness is the load-bearing assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces new question-answer matching datasets for Latvian and Lithuanian, generated by applying six alteration rules to Wikipedia-derived reference answers with GPT-4o and LLaMa3, followed by a partial manual validation. It then evaluates seven open LLMs in zero-shot and few-shot settings to classify generated answers as matched or non-matched. The headline finding is that larger models (QWEN2.5 72b, LLaMa3.1 70b) achieve near-perfect F1 scores in both languages, while smaller models vary, with some benefiting from few-shot examples and others degrading. The authors release code and data on GitHub.","tokens_in":18511,"tokens_out":6539,"duration_ms":62410,"significance":"If the released datasets and labels are reliable, this is a useful resource for an under-resourced language pair: it provides multi-domain question-answer pairs with a transparent set of matched/non-matched alteration rules, and it gives a reproducible benchmark for open LLMs on a low-resource short-answer-matching task. The paper explicitly includes a manual validation component, which is a strength, and the evaluation protocol is simple enough to be replicated. The main value is empirical rather than methodological; the conclusions about model rankings and about which alteration rules are difficult are potentially useful for practitioners. However, the dataset-size arithmetic is inconsistent, and the label-validation procedure is too coarse to support the precision of the reported per-model differences. The paper needs a careful reconciliation of dataset figures and a clearer item-level audit before the accuracy claims can be accepted.","major_comments":[{"comment":"The dataset-size accounting is internally inconsistent. The text reports 3,012 Latvian triplets (1,506 matched / 1,506 non-matched) and 4,830 Lithuanian triplets (2,760 matched / 2,070 non-matched), then immediately states that the amount of matched answers is 3,697 and the amount of non-matched answers is 1,809. These two final numbers sum to 5,506, which matches neither the per-language total (7,842) nor the sum of the parenthetical matched and non-matched counts (4,266 matched / 3,576 non-matched). The paper must reconcile these figures and specify which numbers describe the final released dataset after duplicate removal and class filtering.","section":"Section 3.2"},{"comment":"The ground-truth labels are validated only at class level, not at item level. Two annotators per language reviewed 360 random triplets, and Table 2 reports aggregate accept/reject counts per generation class; the final dataset is then produced by accepting or rejecting entire classes. Because the manual sample was drawn before duplicate removal and before the selective class filtering, the observed reject rates do not directly estimate the label-error rate on the final test items. A mislabeled cluster from one generation class would enter the benchmark unexamined, and models that happen to agree with the intended label would be scored correct even when the label is wrong. The reported differences between models are small enough (e.g., Mistral Nemo 12b at 0.94–0.96 versus QWEN2.5 72b at 0.99) that a few percent of systematic label noise could shift model rankings. The paper reports no inter-annotator agreement statistic, no per-rule item counts, and no confidence intervals. An item-level audit of the released labels, or at minimum per-class precision estimates with intervals, is needed before the headline accuracy claims can be accepted.","section":"Section 3.3 / Appendix A"},{"comment":"The EuroLLM 9b zero-shot F1 scores conflate instruction-following with matching ability. The text states that in zero-shot settings EuroLLM 9b was not able to follow the output format at all and that its results were legible but impossible to parse, yet Table 1 reports F1 = 0.13 for Lithuanian ZS and F1 = 0.05 for Latvian ZS, and the discussion treats these numbers as a language-difference finding. A model whose outputs cannot be parsed should be excluded from the F1 computation or reported separately as a format-compliance failure. As written, the zero-shot comparison for EuroLLM is not interpretable as a matching-accuracy result.","section":"Section 5 / Table 1"},{"comment":"The statistical support for the model comparisons is missing. Table 1 reports only one F1 number per model/setting/language, with no sample sizes, precision/recall, or confidence intervals, and Figures 3–4 show per-rule accuracy without numerical values or error bars. The conclusions that Mistral 7b performed weaker in few-shot experiments and that QWEN2.5 7b is comparable to the larger 70b models rest on differences as small as 0.01–0.02, which may be within sampling noise given the item counts. Please provide per-cell counts and interval estimates, or explicitly soften the comparative claims.","section":"Section 5 / Table 1 and Figures 3–4"}],"minor_comments":[{"comment":"The generation model is referred to as LLaMa3 7b in Section 3.2 and as LLaMa3:7b in Table 2, while the evaluated models are LLaMa3.1 8b and 70b; the paper should clarify which model versions were used for generation and which for evaluation.","section":"Section 3.2 / Appendix A"},{"comment":"The paper uses \"accuracy\", \"F1 score\", and \"accuracy score\" interchangeably; Table 1 is explicitly labeled F1, so the text should consistently report F1 or explicitly state that the reported values are F1.","section":"Section 5 / Table 1"},{"comment":"The statement that GPT-4o Match MoreInfo \"was excluded by mistake\" for Latvian is a data-construction error that should be acknowledged as such and analyzed for its effect on the per-rule comparisons, since readers cannot tell whether the omission is intentional.","section":"Section 3.3"},{"comment":"There are several typos and grammatical slips, including \"perfromed\", \"MIstral\", \"similar the performance\", and \"flows\" for \"flaws\"; a careful proofreading pass is needed.","section":"Throughout"},{"comment":"The abstract reports 502 Latvian and 690 Lithuanian question-answer pairs, and Section 3.1 reports 502 and 690 unique questions; these match, but it would be helpful to state explicitly that these counts refer to the base QA pairs before generated triplets are added.","section":"Section 3.1 / Abstract"}],"recommendation":"major_revision","confidential_remarks":"The self-admitted \"excluded by mistake\" for Latvian MoreInfo suggests the data-selection pipeline is not fully documented, and the inconsistent triplet counts in Section 3.2 make it difficult to verify what was actually evaluated. I would ask the authors to provide the final dataset composition table, the exact filtering decisions, and an item-level label audit (or a clear statement that no item-level audit exists). The resource itself is potentially valuable for the Baltic NLP community, but the benchmark's credibility currently rests on unverified labels and unmeasured parse failures."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — the genuinely new thing here is the release, not the finding. Lat-KID and Lit-KID are the first Latvian and Lithuanian short-answer-matching benchmarks I know of: 502 and 690 Wikipedia-derived question-answer pairs with matched and non-matched generated answers under six alteration rules, plus a seven-model open-LLM benchmark. That is a useful resource for ASAG people and for anyone evaluating Baltic-language abilities of open models. The headline result — 70b models near perfect, smaller models variable, few-shot helps some and hurts Mistral 7b — is descriptive, plausible, and in line with prior multilingual work. The per-rule breakdown (Mistral Nemo weak on Lithuanian synonyms, Llama3.1 8b biased toward negative answers) is the most useful part.\n\nThe soft spots are real but not fatal. The dataset arithmetic in Section 3.2 does not add up: the text first reports 3,012 (1,506/1,506) and 4,830 (2,760/2,070) triplets, then gives 3,697 matched and 1,809 non-matched, which is a different total. The footnote contradicts Appendix A on which non-matched generation classes were kept. Those look like editorial errors, but they make it hard to know the actual test set. More substantively, the labels were validated at class level from a 360-sample check per language, not item level. If a generation batch contained systematic mislabels that slipped through the aggregate accept/reject ratio, the F1 scores — which differ by only a few points between top models — could shift. The paper would be much stronger with item-level annotation counts, inter-annotator agreement, confidence intervals, and an explicit statement of per-rule test sizes. The EuroLLM zero-shot result is also oddly presented: the model did not follow the output format, so its 0.05–0.13 F1 is not a language competence score; parsing failure is conflated with task failure. Those are fixable.\n\nThe citation pattern is fine; the paper builds on the KID pipeline and related Baltic ASAG work. I could not verify the GitHub, but the dataset is the deliverable, so code and data release matters more here than in a typical paper.\n\nBottom line: worth a proper peer review, but not as-is. A competent referee should require the dataset arithmetic fixed, the Appendix and footnote reconciled, an item-level label audit (or at least per-class counts and agreement), and error bars or per-rule item counts. If those land, the datasets become a usable benchmark. If not, the resource is still potentially useful but the accuracy claims are under-supported.","headline":"A genuinely new low-resource dataset benchmark, but class-level label validation and sloppy dataset arithmetic mean the headline F1 claims need revision before they should be trusted.","tokens_in":19085,"tokens_out":2402,"would_cite":true,"duration_ms":24593,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Large open-weights LLMs, especially QWEN2.5 72B and LLaMa3.1 70B, distinguish matched from subtly altered short answers in Latvian and Lithuanian at near-perfect accuracy.","keywords":["short answer matching","automatic short answer grading","Latvian","Lithuanian","low-resource languages","large language models","few-shot prompting","answer alteration rules"],"falsifier":"Re-run the manual annotation on all 3,012 Latvian and 4,830 Lithuanian triplets, or on a much larger random sample stratified by alteration rule. If native-speaker rejection rates on the unverified items are substantially higher than the accept ratios reported in the manual evaluation, the near-perfect model F1 scores are partly measuring template artifacts rather than genuine semantic matching.","tokens_in":18020,"feed_emoji":"🎓","tokens_out":11333,"duration_ms":99651,"temperature":0.7,"pith_summary":"This paper asks whether open-weights language models can tell matched short answers apart from subtly altered non-matched ones in Latvian and Lithuanian, two low-resource Baltic languages where automated answer grading has little data. To test this, the authors built two Wikipedia-derived datasets—502 Latvian and 690 Lithuanian question-answer pairs—and generated matched and non-matched variants with six alteration rules, yielding 3,012 and 4,830 test triplets, with a small manually verified subset. They benchmarked seven open models in zero-shot and few-shot settings. Their central finding is that the largest models, QWEN2.5 72B and LLaMa3.1 70B, classify matched versus non-matched almost perfectly in both languages under both prompting settings, while smaller models vary: some improve with few-shot examples, and Mistral Nemo 12B is noticeably weaker on Lithuanian. If this holds, automatic short-answer grading for these languages is within reach, and the released datasets give the community a way to measure it.","feed_headline":"Large LLMs ace subtle answer matching in Latvian and Lithuanian","feed_subtitle":"Seven open models and two new Baltic datasets: bigger models grade subtle changes reliably, smaller ones vary.","key_machinery":"The mechanism that carries the argument is the alteration-rule taxonomy. Four matched rules—adding question-related entities (Ents), changing words to synonyms (Synonyms), adding background information (MoreInfo), and swapping style to exclamatory (Exclamatory)—generate answers that should be accepted; two non-matched rules—incorporating minor changes (IMC) and changing domain-related information (CDRI)—generate near-identical wrong answers by swapping a key word. Every test item is a (question, reference answer, generated answer) triple labeled as matched or non-matched, and each model must start its output with True or False. The per-rule breakdown turns the aggregate F1 into a diagnostic of which kinds of semantic alertness a model has, and the language-matched few-shot prompt is what separates the small models that recover with examples from those that do not.","core_discovery":"On the paper's own terms, the discovery is that the relevant question is less whether LLMs can catch subtle answer mismatches in Latvian and Lithuanian than which model size and prompting recipe makes that reliable. In the released datasets, QWEN2.5 72B and LLaMa3.1 70B reach F1 around 0.99 on binary matched/non-matched classification in both zero-shot and few-shot conditions, with QWEN2.5 7B and Mistral 7B close behind despite their smaller size. Few-shot prompting is not uniformly helpful: it rescues EuroLLM 9B from zero-shot output that cannot even be parsed into True/False and lifts LLaMa3.1 8B, but it slightly lowers Mistral Nemo 12B and Mistral 7B. The rule-level analysis isolates the failure modes—Mistral Nemo misses synonym substitution and exclamatory-style matches in Lithuanian, while LLaMa3.1 8B and EuroLLM 9B struggle with added entities in Latvian—which is the paper's most useful output beyond aggregate accuracy.","pith_inferences":["Because the non-matched answers were generated from fixed templates, the near-perfect scores may partly reflect predictable surface patterns rather than robust semantic judgment; real student answers in Latvian and Lithuanian are likely messier and may lower accuracy.","The rule-level breakdown could be reused as a diagnostic suite: any model claiming support for these languages could be scored per rule, and synthetic data targeting the weakest rules could be used to improve small models.","The dataset design does not control for whether a model uses the question at all; swapping in an unrelated question while keeping the same reference and candidate answers would test whether the model is truly doing answer matching or just textual comparison.","Because only 360 samples per language were manually verified and the remaining labels were kept on aggregate accept ratios, the published accuracies are best read as upper bounds until the full labels are independently checked."],"forward_implications":["Large open-weights models around 70B parameters can serve as short-answer matchers in Latvian and Lithuanian without task-specific training, at essentially ceiling-level accuracy on these datasets.","Few-shot prompting is a cheap remedy for some small models—EuroLLM 9B and LLaMa3.1 8B gain substantially—but it is not a general fix, since Mistral Nemo 12B and Mistral 7B score slightly worse with examples.","Per-rule failure patterns such as synonym detection and added-entity recognition give concrete targets for evaluation and fine-tuning rather than a single pass/fail score.","The released 3,012 Latvian and 4,830 Lithuanian triplets provide a reusable benchmark for future multilingual and Baltic-language models.","A model that cannot follow a simple True/False output instruction in zero-shot is not usable as an automatic grader regardless of its underlying language ability; few-shot examples can repair this for EuroLLM 9B."],"supporting_citations":[{"why":"Supplies the Wikipedia-based knowledge-and-instruction dataset pipeline that was adapted into Latvian Lat-KID and Lithuanian Lit-KID.","marker":"Kiulian et al., 2024"},{"why":"Gemini 1.5 Pro is the generation model used to filter country-relevant Wikipedia articles and produce questions for the datasets.","marker":"Team et al., 2024"},{"why":"Defines the LLaMa3.1 8B and 70B models whose zero-shot and few-shot matching accuracy are core comparison points.","marker":"Dubey et al., 2024"},{"why":"Defines Mistral 7B, one of the smaller models that matches larger-model accuracy in zero-shot but degrades in few-shot.","marker":"Jiang et al., 2023"},{"why":"Defines EuroLLM 9B, whose inability to follow the True/False format in zero-shot and recovery in few-shot anchors the prompting findings.","marker":"Martins et al., 2024"},{"why":"Defines the QWEN2.5 family, including the 7B and 72B models that achieve the highest reported accuracies.","marker":"Team, 2024"},{"why":"Companion technical report for Qwen2; together with Team, 2024 it fixes the QWEN2.5 model configuration evaluated in the paper.","marker":"Yang et al., 2024"}],"fun_headline_variants":["Smaller LLMs rival 70B on Latvian-Lithuanian grading","Few-shot can hurt: Mistral Nemo fails subtle Baltic edits","Model size and prompting decide Baltic answer matching","7B models match 70B on subtle Baltic answer detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automatically generated match/non-match labels are correct for the whole dataset, even though only 360 random samples per language were manually verified and the rest was kept based on aggregate accept ratios.","fun_headline_variants_meta":{"raw":{"variants":["Smaller LLMs rival 70B on Latvian-Lithuanian grading","Few-shot can hurt: Mistral Nemo fails subtle Baltic edits","Model size and prompting decide Baltic answer matching","7B models match 70B on subtle Baltic answer detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000555,"raw_usage":{"total_tokens":2684,"prompt_tokens":1030,"completion_tokens":1654,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":646,"completion_tokens_details":{"reasoning_tokens":1582}},"tokens_in":646,"tokens_out":1654,"duration_ms":10363,"temperature":1.0,"reasoning_tokens":1582,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:09:54.826791+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the manual annotation on all 3,012 Latvian and 4,830 Lithuanian triplets, or on a much larger random sample stratified by alteration rule. If native-speaker rejection rates on the unverified items are substantially higher than the accept ratios reported in the manual evaluation, the near-perfect model F1 scores are partly measuring template artifacts rather than genuine semantic matching.","supporting_citations":[],"review_version":1}