Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Prompt and circumstance: A word-by-word LLM prompting approach to interlinear glossing for low-resource languages

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Retrieval-based word-by-word prompting of GPT-4 beats the BERT baseline at interlinear glossing in all seven tested languages.

desk verdict Genuinely novel instruction-generation experiment wrapped in an honest but statistically fragile empirical package; the data-leakage check is too weak to support the headline margins. read the letter →

arxiv 2502.09778 v2 pith:XV264KSG submitted 2025-02-13 cs.CL

Micha Elsner , David Liu This is my paper · ORCID
classification cs.CL
keywords interlinearglossinglow-resourcelanguagesLLMpromptingretrieval-augmentedgenerationmorphologicalsyncretismTsezSIGMORPHONhuman-in-the-loopannotation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a word-by-word retrieval prompting strategy can make large language models useful assistants for interlinear glossing, the morpheme-by-morpheme annotation used in language documentation. It shows that this approach beats the official BERT baseline in all seven SIGMORPHON 2023 languages on morpheme-level accuracy, and that when a human picks among three generated glosses, the resulting oracle beats the tuned challenge winner in five languages. It also demonstrates that automatically generated linguistic instructions can reduce confusion between syncretic Tsez tags, such as perfective converb versus past unwitnessed, by about ten percent. The larger claim is that LLMs' ability to follow natural-language instructions makes them a practical interactive tool for linguists, not just a batch predictor.

What carries the argument

The load-bearing mechanism is word-by-word retrieval prompting with k-best elicitation. Each word gets up to three exact-match sentences, up to three approximate matches sharing a four-character substring, reverse-indexed words from the metalanguage translation, and a frequency summary of the word's tags over the training corpus; the LLM must return three ordered glosses in JSON. For the syncretism case, a second machine generates contrastive instances for a confused tag pair, asks the LLM for concise syntactic rules (with a hardcoded good/bad example pattern), and then inserts those rules plus a chain-of-thought step into the glossing prompt.

What would settle it

Ask GPT-4 to produce the gold gloss for a random sample of Tsez (and other language) test sentences with no retrieval and no candidate tags; if it reliably outputs the gold glosses, then the retrieval scores are inflated by memorization. A cheaper check is to run the canary test in all seven languages asking for verbatim gloss lines rather than sentence completions.

Watch

Extended reading notes

Core claim

The central discovery is that the model's top guess is often wrong while one of its three candidates is correct, and that this near-miss behavior is what makes LLM prompting valuable in human-in-the-loop annotation. For each target word the system retrieves exact and approximate matches, reverse-matches words from the translation, and supplies a distribution of the word's common tags; the LLM then ranks three candidate glosses. A Jaccard-based oracle over those three candidates yields word-level scores above the tuned sequence-model winner in Gitksan, Lezgi, Nyangbo, Tsez, and Uspanteko, and the same oracle beats the baseline everywhere. The paper also shows that asking the LLM to write disambiguation rules from contrastive examples, then injecting those rules into the prompt, reduces the dominant Tsez error class—confusions between PFV.CVB and PST.UNW—from 102 to 74 test-set errors, raising word accuracy from 75.28 to 75.86.

Load-bearing premise

GPT-4 has not memorized the test-set glosses, so the reported scores reflect the prompting method rather than recall of the answer key.

Editorial extensions

If this is right

  • A human annotator working with a 3-best oracle could accept one of the top three glosses at rates exceeding the tuned sequence model in most languages, suggesting LLM suggestions can speed manual glossing.
  • Because word-by-word prompting uses only six retrieved examples per word, it can be cheaper and more interpretable than sentence-level prompting that retrieves up to 100 sentences.
  • The Tsez result indicates that LLMs can apply abstract grammatical instructions to concrete data when instructions are generated from contrastive examples, contrasting with repeated failures in low-resource translation.
  • The instruction-generation pipeline could be applied to other syncretic or confusable tag pairs in languages beyond Tsez, and to morphological disambiguation more broadly.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The oracle result may understate how well a real annotator would do, because the Jaccard oracle assumes perfect selection among the three options; a realistic annotator makes mistakes, so the practical benefit would likely be smaller but could still be positive.
  • If the method is applied to a truly unseen language with no training examples at all, the reverse retrieval and frequency summaries would disappear, leaving only the translation and the LLM's priors; the paper does not test this zero-example regime, but the Gitksan result (only 31 training sentences) suggests substantial degradation is likely.
  • A direct test of the memorization concern would be to run the same retrieval prompts on newly collected IGT from the same languages; if the gains vanish, the scores depend on test-set leakage rather than the method.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a word-by-word retrieval-based prompting approach to interlinear glossing, applied to the seven languages of the SIGMORPHON 2023 shared task. The system retrieves exact and approximate matches, reverse-indexed translation words, and tag-frequency summaries for each target word, and asks GPT-4o to return a 3-best list of glosses. The main claims are that the system beats the BERT baseline at morpheme-level accuracy for all seven languages, that a 3-best oracle exceeds the challenge winner's word-level scores in five languages, and that in a Tsez case study, automatically generated linguistic instructions reduce confusions between syncretic tags such as PFV.CVB and PST.UNW. The paper reports improved word-level accuracy on Tsez from 75.28 to 75.86 after adding these instructions, and a reduction in CVB-related confusions.

Significance. If the results hold, the paper provides a useful empirical data point for interactive glossing: word-by-word prompting is competitive with whole-sentence prompting, the k-best oracle quantifies a plausible upper bound for human-in-the-loop annotation, and the Tsez instruction-following case study is one of the few demonstrations that LLMs can apply abstract linguistic rules to concrete glossing decisions. The authors deserve credit for honestly stating their limitations: single runs, no significance testing, a reduced Arapaho test set, and the use of a proprietary model. The paper also makes its code and results available, which strengthens reproducibility. However, the significance is tempered by the small margins in several comparisons, the lack of any variance analysis, and an unconvincing canary test for test-set memorization; these issues directly affect the central empirical claims.

major comments (4)
  1. [Appendix A] The canary test in Appendix A is too weak to support the assumption that GPT-4o has not memorized the SIGMORPHON 2023 test glosses. It is run only for Tsez, it probes verbatim sentence completion rather than gloss recall, and the observed refusals ('I'm sorry...') are consistent with a safety policy that would apply even to perfectly memorized text. Because the claim that the system beats the BERT baseline in all seven languages at morpheme level (Section 4.2, Table 4) and the oracle comparisons in Section 5 depend on test-set novelty, this is a load-bearing gap. Please provide stronger evidence, for example a gloss-recall probe on held-out training versus test sentences, or a comparison on a newly collected evaluation set.
  2. [Section 4.2, Tables 1 and 4; Limitations] All reported scores come from single runs, as the Limitations section states, but the key margins are often smaller than any plausible run-to-run variation. For example, Gitksan morpheme accuracy is 8.68 versus the baseline's 8.54 (Table 4), and Uspanteko morpheme accuracy is 57.59 versus 57.24; the Tsez instruction improvement is 75.28 to 75.86 (Section 6.2). Without variance estimates or repeated runs, the headline claim of beating the baseline in every language is not established. I ask for at least a few repeated runs at different temperatures or seeds for the languages with the smallest margins, or an explicit error-bar analysis.
  3. [Tables 1, 2, 4, 5; Section 3] The Arapaho results in all tables are based on the first 100 test sentences only (arp*), yet they are placed next to the full-test baseline and challenge-winner scores from the shared task. A 100-sentence subset is not guaranteed to be representative, so the comparisons for arp* (e.g., word-level 66.19 vs. 71.14 and morpheme-level 52.57 vs. 44.19) are not controlled and should not be used to infer relative system quality. The paper should either run the full Arapaho test set or report the baseline and winner scores computed on the same 100-sentence subset.
  4. [Section 7, first paragraph] The conclusion states that the system 'surpass[es] the word-level test scores of the Track 1 challenge winner in Gitksan, Lezgi, Nyangbo, Tsez, and Uspanteko' without immediately qualifying that this is the 3-best oracle, not the system's 1-best output. The distinction is important because the oracle uses gold tags at test time to choose among candidates; it is a ceiling for human-in-the-loop annotation, not a deployable predictor. The abstract and Table 2 caption make the oracle status clear, but the conclusion should too.
minor comments (5)
  1. [Table 2 caption] The caption contains a broken phrase: 'Table 2 in the shows word-level scores' should read 'Table 2 shows word-level scores'.
  2. [Section 4.1] There are two typos: 'subection 4.2' and 'subection 5' should be 'subsection 4.2' and 'subsection 5'.
  3. [Table 4] In Table 4, the rows for ddo (Tsez) and usp (Uspanteko) both report 57.59 under 'Ours'; please verify that this is not a copy-paste error.
  4. [Appendix D] The JSON example in Appendix D is not valid JSON because the 'glosses' field uses triple underscores inside quotes; the formatting should be cleaned up to match the intended 'best three glosses' output.
  5. [Tables 1, 2, 4, 5] The notation 'arp*' is used in table captions but not defined at first use; please state explicitly that it refers to the first 100 sentences of the Arapaho test set.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: all scores are computed on held-out test data, retrieval and instruction generation use training/dev data only, and the oracle is explicitly an upper bound, not a fitted predictor.

full rationale

The paper's reported scores are all computed on the held-out SIGMORPHON 2023 test sets, while the retrieval contexts, tag-frequency summaries, and generated instructions are built from training and development data only. The 3-best oracle uses gold labels solely to select among the LLM's already-produced candidate glosses at evaluation time; the paper explicitly labels it an 'oracle' and does not use it to fit any parameter, so its comparison to the challenge winner is an upper-bound diagnostic rather than a prediction that reduces to its input by construction. The Tsez instruction-generation experiment similarly draws contrastive instances from the training set, and the evaluation is on the test set; the paper even documents that one generated rule is incorrect, which confirms the outcome is not forced by the setup. The self-citations to Court and Elsner (2024) and Elsner and Needle (2023) appear only in related-work context about low-resource translation and are not load-bearing for any of the paper's central quantitative claims. The Appendix A canary test is weak evidence about test-set memorization, but that is an external-validity and data-leakage concern, not a definitional circularity: no equation or score in the paper is defined in terms of the quantity it claims to predict. Thus no circular step can be exhibited, and the appropriate finding is no significant circularity.

Assumptions & free parameters 9 free parameters · 4 assumptions · 0 invented entities

The system's results depend primarily on a set of hand-chosen retrieval and prompting hyperparameters (match counts, substring length, 3-best, 32 contrastive examples, temperature), plus domain assumptions about data leakage, oracle interpretability, and subset comparability. The LLM itself is treated as a black box. There are no fitted model parameters in the usual sense; the 'free parameters' are engineering choices.

free parameters (9)
  • num_exact_matches = 3
    Number of exact-match sentences retrieved per target word (Section 4.1). Chosen during development on Tsez; affects prompt content and performance.
  • num_approximate_matches = 3
    Number of approximate-match sentences retrieved per target word (Section 4.1).
  • min_approx_substring_length = 4
    Minimum longest-common-substring length for an approximate match (Section 4.1).
  • num_best_glosses = 3
    Number of candidate glosses elicited from the LLM and used for oracle evaluation (Sections 4.1 and 5).
  • max_contrastive_examples = 32
    Maximum number of contrastive sentence pairs used to generate disambiguation instructions per tag pair (Section 6.1, Appendix E).
  • confusion_threshold = 5
    Only tag pairs with more than 5 dev-set confusions received generated instructions (Section 6.1).
  • temperature_glossing = 0
    Sampling temperature for glossing prompts (Section 4).
  • temperature_instruction_generation = 0.25
    Sampling temperature for instruction generation (Section 4).
  • arapaho_test_subset_size = 100
    Only the first 100 Arapaho test sentences were evaluated for cost reasons (Section 3); this subset is labeled arp* and may not represent the full test set.
assumptions (4)
  • domain assumption GPT-4 has not memorized the SIGMORPHON test sentences
    Section 3 and Appendix A: the authors argue the LLM has little ability to complete test sentences based on refusal behavior; all test scores depend on this.
  • domain assumption The 3-best oracle, which selects the candidate with highest Jaccard similarity to the gold tag, approximates a human annotator's ability to choose among suggestions
    Section 5: the oracle result is presented as evidence that LLM suggestions can assist human annotators, but no human selection experiment is run.
  • domain assumption The abridged Arapaho test subset (arp*) is treated as comparable to the full test set when comparing to baseline and winner scores
    Section 3 and Table 1: the authors compare their 100-sentence subset against published full-test scores, which may bias comparisons.
  • domain assumption The LLM reliably follows Leipzig glossing conventions when instructed
    Section 4.1: the system relies on instruction-following to produce well-formed glosses; Appendix D provides one prompt example but no systematic validation of formatting.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Prompt and circumstance: A word-by-word LLM prompting approach to interlinear glossing for low-resource languages." pith.science (2026). https://pith.science/paper/XV264KSG

@misc{pith2026250209778,
  author       = {Pith},
  title        = {Pith review of: Prompt and circumstance: A word-by-word LLM prompting approach to interlinear glossing for low-resource languages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XV264KSG}},
  note         = {Machine review of arXiv:2502.09778}
}
read the original abstract

Partly automated creation of interlinear glossed text (IGT) has the potential to assist in linguistic documentation. We argue that LLMs can make this process more accessible to linguists because of their capacity to follow natural-language instructions. We investigate the effectiveness of a retrieval-based LLM prompting approach to glossing, applied to the seven languages from the SIGMORPHON 2023 shared task. Our system beats the BERT-based shared task baseline for every language in the morpheme-level score category, and we show that a simple 3-best oracle has higher word-level scores than the challenge winner (a tuned sequence model) in five languages. In a case study on Tsez, we ask the LLM to automatically create and follow linguistic instructions, reducing errors on a confusing grammatical feature. Our results thus demonstrate the potential contributions which LLMs can make in interactive systems for glossing, both in making suggestions to human annotators and following directions.

Figures

Figures reproduced from arXiv: 2502.09778 by the authors.

Figure 1
Figure 1. Outline of Section 6, showing the pipeline of instruction generation and inference-time disambigua￾tion for a syncretic pair. Purple panels show LLM￾generated text. complex languages [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A POS Tier Is the Key to Automated Annotation for Low-Resource Language Documentation: Neural Interlinear Glossing for Irabu, a Southern Ryukyuan Language

    cs.CL 2026-07 conditional novelty 5.0 of 10

    For Irabu Ryukyuan, a POS tier improves neural grammatical glossing by +4.4 points with oracle POS and can more than halve data needs, but current tagger errors cancel the gain in a fully automatic pipeline.

Reference graph

Works this paper leans on

25 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [1]

    The father sent his son together with Oku

    **Tag a verb as PFV .CVB if it appears in a subordinate clause that provides background in- formation or context for the main action, often preceding the main verb.** The PFV .CVB verb typically describes an action that is completed before the action of the main verb. For example: - Sentence: *Babiya uži Okude sadaq egirno xecin.* - Gloss: dad-ERG son Oku...

    work page

  2. [2]

    When they had gone into their house, the eagle let the hunter’s son down to their house

    **Tag a verb as PST.UNW if it is the main verb of the sentence, indicating the primary action or event that is being reported.** The PST.UNW verb typically follows the PFV .CVB verb if both are present in the sentence. For example: - Sentence: *Žedu teł iduGor boqnosi, ceyä bełiqus uži žeduz GQutkoxor egirno.* - Gloss: DEM1.IPL inside home-IN.ALL III- bec...

    work page

  3. [3]

    Soon, that boy picked a beauti- ful girl from the neighboring village and married her

    **Tag a verb as PFV .CVB if it appears in a se- quence of actions where it indicates a completed action that leads to or results in the subsequent ac- tion.** The PFV .CVB verb often precedes another verb that is tagged as PST.UNW. For example: - Sentence: *Kwaxatow sida igozo Qaňay nesi žax łet’in sis bercinaw kidno yown.* - Gloss: soon-EMPH one.OBL near...

    work page

  4. [4]

    The king didn’t know that he was his own son

    **Tag a verb as PST.UNW if it is used to report a past event or action that has relevance to the nar- rative or discourse context.** The PST.UNW verb often concludes a sequence of actions or provides the main point of the sentence. For example: - Sentence: *Xaner iyn anu yeda nesis uži yałrułi.* - Gloss: khan-LAT know-PST.UNW be.NEG DEM2.ISG DEM1.ISG.OBL-...

    work page

  5. [5]

    ’If that’s the case,’ said the boy, ’I’ll wait a little, and will neither go to the mosque, nor sleep here at home

    **Tag a verb as PFV .CVB if it is used in a conditional or hypothetical clause, providing a condition or premise for the main action.** The PFV .CVB verb often appears before a PST.UNW verb that describes the outcome or result. For example: - Sentence: *Nedur resunäy ňin, e ňin užä dahawcen i ˇcin, mažgitya Gorno ik’a ˇc’in, idun kecaˇc’in.* - Gloss: so I...

    work page

  6. [6]

    **bet’erbaqi** - noun (life)

    work page

  7. [7]

    **boda** - verb (III-do-INF)

    work page

  8. [8]

    The majority of Tsez adverbial clauses have a converb as their predicate

    Subordinate clause verbs are converbs: Cor- rect. “The majority of Tsez adverbial clauses have a converb as their predicate” (Polinsky, 2014, p.296)

    work page 2014

Show all 25 references
  1. [9]

    Main verb is PST.UNW: Correct

  2. [10]

    clause chains

    Converbs in sequence of actions indicating actions that lead to the main action: Correct. Polinsky describes these as “clause chains” (p.297)

  3. [11]

    Polinsky agrees that PST.UNW generally has past ref- erence (p.298) but also states that it should be used for a 2nd/3rd person subject due to its evidential status

    PST.UNW used to report a past event relevant to the discourse: True but vague. Polinsky agrees that PST.UNW generally has past ref- erence (p.298) but also states that it should be used for a 2nd/3rd person subject due to its evidential status. Tense cannot be used to identify...

  4. [12]

    wait a little

    Converbs in hypothetical: Mostly incorrect. Tsez uses specialized converbs for hypotheti- cal constructions (p.314-315) rather than us- ing the PFV .CVB. However, PFV .CVB can appear in hypotheticals due to “linking of con- verbs, where only one converb is marked as conditiona...

  5. [13]

    There is no other verb in the sentence that would serve as the main verb, which supports the use of PST.UNW

    **PST.UNW as the main verb**: "Ayn" is likely the main verb of the sentence, as it describes the primary action of coming to a poplar forest. There is no other verb in the sentence that would serve as the main verb, which supports the use of PST.UNW. In the following case, our...

  6. [14]

    teqno" appears before the main verb

    **PFV .CVB in subordinate clauses**: The word "teqno" appears before the main verb "ik’in" (go-PST.UNW). This suggests that "teqno" could be providing background information or context for the main action, which aligns with the use of PFV .CVB

  7. [15]

    ik’in" (go-PST.UNW), which indicates the primary event

    **PST.UNW as the main verb**: The main action of the sentence is "ik’in" (go-PST.UNW), which indicates the primary event. "Teqno" is not the main verb here, which makes it less likely to be PST.UNW

  8. [16]

    Teqno" could represent a completed action that leads to the subsequent action

    **PFV .CVB in sequences of actions**: The sentence describes a sequence where the clever brother hears something and then goes behind them. "Teqno" could represent a completed action that leads to the subsequent action "ik’in" (go- PST.UNW), supporting the use of PFV .CVB. In ...

  9. [17]

    **zaman** - noun (time)

  10. [18]

    **bik’inosi** - verb (I.PL-go-ANT.CVB)

  11. [19]

    **bara** - noun (wife-ERG)

  12. [20]

    **xediw** - noun (husband)

  13. [23]

    **ik’anix** - verb (go-MSD-AD.ESS)

  14. [24]

    **muk’ur** - verb (persuade)

  15. [25]

    bik’inosi

    **PFV .CVB in subordinate clauses**: The candidate gloss suggests that "bik’inosi" (I.PL- go-ANT.CVB) is a subordinate clause providing background information. The main action seems to be the wife’s confession about going to live with her husband. "Oyn" could potentially be a ...

  16. [2009]

    In Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Lan- guage Processing, pages 36–44, Boulder, Colorado

    Evaluating automation strategies in language documentation. In Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Lan- guage Processing, pages 36–44, Boulder, Colorado. Association for Computational Linguistics. Maria Polinsky. 2014. Tsez Syntax: A Descr...

    2009 arXiv

  17. [2024]

    monkey-PL

    Teaching large language models an unseen language on the fly. Preprint, arXiv:2402.19167. Xingyuan Zhao, Satoru Ozaki, Antonios Anastasopou- los, Graham Neubig, and Lori Levin. 2020. Auto- matic interlinear glossing for under-resourced lan- guages leveraging translations. In P...

    2019 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.