{"id":"d706b34c-920e-409b-ba14-c426acc65fb7","arxiv_id":"2412.05710","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"PromptRefine uses alternating minimization over language-specific retrievers plus diversity-aware DPP fine-tuning to select cross-lingual in-context examples, improving few-shot generation in low-resource Indic languages.","lead":"This paper presents PromptRefine, a method that chooses few-shot examples for large language models by pulling from both the target low-resource Indic language and related high-resource languages, then fine-tuning retrievers to rank diverse examples. It reports consistent gains over existing retrieval baselines across question answering, translation, and summarization tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No held-out test split or variance is reported, and the final retriever is selected on validation, so PromptRefine's headline gains may be tuning artifacts; this is more load-bearing than the parameter-averaging assumption.","rationale":"The reader identifies parameter averaging in the Merge step as the weakest assumption. That is a legitimate concern, but it is partially supported by Figure 3, which shows merged-retriever validation accuracy improving over individual retrievers across iterations. The more load-bearing gap is empirical: the paper reports single-run scores, states no significance tests, and describes a pipeline that tunes the retriever selection, the auxiliary threshold, and the number of demonstrations on validation data. If the final tables are computed on that same validation set, the claimed gains could be an artifact of selection, and the paper's 'significantly outperforms' claim would not be established even if the merge step works perfectly. This is a testable, concrete concern: one needs a clear held-out split and variance estimates. Since the reader's verdict is already CONDITIONAL, my analysis does not change the verdict, but it identifies a different, more fundamental reason for the condition.","tokens_in":18753,"tokens_out":6681,"duration_ms":76826,"concrete_test":"Obtain or reconstruct the exact data split used for Tables 1-5. If the final numbers come from the same set used for Algorithm 1 lines 11/14 and for the delta/K ablations, hold out a fresh test split, re-run PromptRefine versus CEIL with at least 3 independent seeds (K=16, LLAMA-3.1-8B, Bodo XorQA and Santali Flores), and compute paired bootstrap 95% confidence intervals on the mean differences. If the CI for PromptRefine minus CEIL includes 0, or the mean gain drops substantially below the reported values, the central claim is not supported as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"PromptRefine's central empirical claim ('significantly outperforms all baselines') rests on Tables 1-5, but the pipeline selects the final retriever on the validation set: Algorithm 1 line 14 chooses rho* by arg max over iterations of Acc(rho, D_T^val), and the auxiliary threshold delta (Fig. 4), K (Fig. 5), and the 120-epoch schedule are tuned on the same benchmark. The paper never states whether Tables 1-5 report a separate held-out test split; if they report validation numbers, the +8.26 Token-F1 over CEIL and +4.85 chrF1 over CEIL could reflect selection among many configurations on the evaluation set rather than an algorithmic property. Because no error bars, seeds, or significance tests are given, the reader cannot distinguish a genuine improvement from favorable validation-based selection. The merge-step worry is real but secondary: Figure 3 provides some direct evidence that parameter averaging helps, whereas no evidence addresses the validation/selection confound.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PromptRefine, a three-stage method for selecting in-context learning (ICL) examples for low-resource Indic languages. The method first selects related high-resource Indic languages as auxiliary example banks using a cosine-similarity threshold, then trains language-specific retrievers through an alternating minimization procedure that interleaves relevance fine-tuning on each language's example bank with parameter averaging to form a shared retriever, and finally fine-tunes the merged retriever with a DPP-based diversity objective. The authors evaluate PromptRefine on four text generation tasks—cross-lingual QA, multilingual QA, machine translation, and cross-lingual summarization—using several open-weight LLMs (LLaMA-2-7B, LLaMA-3.1-8B, Qwen-2-7B, Qwen-2.5-7B) and two proprietary models (GPT-3.5, GPT-4). They report consistent gains over baselines, including EPR and CEIL, with improvements such as +8.26 Token-F1 over CEIL on Bodo cross-lingual QA and +4.85 chrF1 over CEIL on Santali-to-English translation. The paper includes ablations on the auxiliary-language selection threshold, the number of in-context examples K, the importance of the related-language choice, the diversity fine-tuning component, and the alternating-minimization merge step.","tokens_in":18973,"tokens_out":5370,"duration_ms":49102,"significance":"If the reported gains hold, PromptRefine would be a practically useful approach for improving few-shot generation in low-resource languages by leveraging related high-resource language data, an important direction for multilingual NLP. The paper has several strengths: it evaluates across four diverse generation tasks and multiple LLMs, includes code release, and provides ablations for key design decisions. In particular, Figure 3 gives some direct evidence that the parameter-averaging merge improves validation accuracy over iterations, and Figures 1-2 support the importance of related-language selection and diversity training. However, the central empirical claim rests on a missing piece: the paper never states whether the main tables report held-out test results or validation results, and no variance or significance estimates are reported anywhere. Because Algorithm 1 selects the final retriever by maximizing validation accuracy, the headline gains could in principle be artifacts of selection on the evaluation set. This must be resolved before the empirical contribution can be fully assessed.","major_comments":[{"comment":"The paper does not state whether the results in Tables 1-5 are computed on a held-out test split or on the validation set D_T^val. Algorithm 1 (lines 11-14) selects ρ* by arg max over Acc(ρ, D_T^val), and the threshold δ (Fig. 4), the number of examples K (Fig. 5), and the 120-epoch schedule (Section 5) are all tuned on the same benchmark. If the tables report validation numbers, the reported gains (e.g., +8.26 Token-F1 in Table 1, +4.85 chrF1 in Table 2) may reflect favorable selection among many configurations rather than an algorithmic property. Please clarify the split and report results on a separate held-out test set.","section":"Section 5.1 and Algorithm 1 (lines 11-14)"},{"comment":"The abstract and Section 5.1 claim that PromptRefine 'significantly outperforms' baselines, but no error bars, confidence intervals, or significance tests are reported anywhere in the paper. Since ICL is known to be sensitive to the choice and order of examples, the reported single-run differences (e.g., +3.21 Token-F1 on Maithili in Table 1, +0.88 chrF1 on Manipuri in Table 3) need variance estimates across seeds or example subsets to be interpretable. This is load-bearing for the central empirical claim.","section":"Section 5.1, Tables 1-5"},{"comment":"Algorithm 1 (line 10) assumes that parameter averaging of language-specific retrievers yields a shared representation space that enables effective cross-lingual retrieval. Figure 3 provides one validation-accuracy curve for a single task (cross-lingual QA on Manipuri), but there is no analysis of the geometry of the averaged embeddings and no demonstration on other tasks or languages. Please provide additional empirical evidence (e.g., retrieval accuracy or cross-lingual transfer curves for multiple tasks) or a theoretical justification that the average does not degrade language-specific knowledge.","section":"Section 4.1.2 and Figure 3"}],"minor_comments":[{"comment":"The description of BM25 as 'simply retrieves K examples with closest vector embedding to ϕ(x)' is inaccurate because BM25 is a lexical retrieval method and does not use the dense embedding ϕ; please correct this sentence.","section":"Section 3.2"},{"comment":"The CrossSum-In task is described in Appendix F as summarizing a non-English article into the same language, whereas Table 6 and Section 5.1 describe summarizing an English article into a low-resource language; please reconcile these descriptions.","section":"Appendix F and Table 6"},{"comment":"The notation 'ϕ_i ← min_ρ L_rel(D_i; ρ)' is ambiguous because it suggests minimizing the loss with respect to the already-fixed shared weights ρ rather than fine-tuning a retriever initialized from ρ; please rewrite to clarify the update.","section":"Algorithm 1, step 7"},{"comment":"The text states the alternating procedure runs 'until convergence,' but Algorithm 1 fixes the number of iterations I; please specify the stopping criterion or remove the convergence claim.","section":"Section 4.1.2"},{"comment":"The Limitations section states that several alternative configurations were empirically tested and the proposed approach performed best, but no details or results for these alternatives are provided; please report them or remove the claim.","section":"Section 8 (Limitations)"},{"comment":"The Figure 4 caption says the δ ablation is for cross-lingual QA on Bodo, but the text says the figure shows translation performance from three low-resource languages to English; please align the caption with the text.","section":"Figure 4 caption and surrounding text"},{"comment":"Please fix typos: 'Divsersity-induced finetuning' in Section 4.1.3, 'Conlusion' in Section 7, and the broken spacing in '2)merges' in Section 4.1.2.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The main barrier to acceptance is the missing information about the evaluation split and the absence of any variance or significance estimates. If the authors can confirm that the main tables use a held-out test split and provide multi-seed results, the paper would be substantially stronger. I would also ask the editor to encourage the authors to reconcile the CrossSum-In task description in the appendix with the main text, as the current inconsistency affects reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: PromptRefine is a plausible engineering contribution with a new cross-lingual application of alternating retriever alignment plus DPP diversity, but the empirical claim that it 'significantly outperforms' is not established by the current statistics. The paper never says whether Tables 1-5 are on a held-out test split, and the final retriever is selected on validation (Algorithm 1 line 14), with δ, K, and the 120-epoch schedule tuned on the same benchmark. Without error bars or significance tests, the headline gains could be selection artifacts.\n\nWhat's actually new: the specific recipe—specialize retrievers per language, average their parameters, repeat, then fine-tune with DPP for diversity—is new relative to EPR and CEIL. Applied to low-resource Indic languages across QA, MT, and summarization, that is a meaningful combination. The paper does decent work showing which ingredients matter: Figure 3 demonstrates that the merged retriever beats the target-only retriever across iterations, and Figure 1 shows that choosing a related auxiliary language matters more than a random one. The code is public, which is good.\n\nWhere it's soft: the evaluation confound is load-bearing. The paper reports large gains, e.g., +8.26 Token-F1 over CEIL on Bodo QA, but with no variance estimate and no explicit test split, the reader can't tell whether this is a stable algorithmic property or a lucky pick among many configurations on the evaluation set. The merge-step assumption (parameter averaging produces a shared representation space) is real but secondary, because Figure 3 provides direct evidence in its favor. The appendix ablation on δ is on the same validation set, so it doesn't resolve the concern. Also, the paper uses 'significantly' without statistical tests, and the DPP diversity ablation is limited to one translation task.\n\nIf the numbers hold under a proper held-out test set and multiple seeds, this is a useful contribution for multilingual practitioners. As written, it's a conditional accept: the authors should release the split definitions, report means and standard deviations over seeds, and show test-set numbers.\n\nI'd send it to review, but the referee should insist on those additions. My own verdict is skeptical until the evaluation is cleaned up.\n\nBest","headline":"Plausible new retriever recipe for low-resource Indic languages, but the 'significant' gains are not supported because there is no held-out test split or variance—conditional accept.","tokens_in":19492,"tokens_out":2299,"would_cite":true,"duration_ms":22642,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PromptRefine's central claim is that few-shot generation on low-resource Indic languages improves when demonstrations are drawn from merged example banks of related higher-resource languages and selected for relevance plus diversity.","keywords":["in-context learning","few-shot example selection","low-resource Indic languages","cross-lingual retrieval","retriever fine-tuning","determinantal point processes","alternating minimization","multilingual LLM evaluation"],"falsifier":"Train a control retriever on the concatenated target and auxiliary example banks with the same relevance loss and the same DPP loss, without any alternating specialize-merge loop, and compare downstream Token-F1 and chrF1 on the same four tasks. If the control matches or beats PromptRefine, the parameter-merging step is not the active ingredient. A second, direct test would be to measure cross-lingual nearest-neighbor agreement in the merged embedding space: if retrieved target-language examples for a given query do not overlap with related auxiliary-language examples, the shared-space assumption is violated.","tokens_in":18573,"feed_emoji":"🌐","tokens_out":8499,"duration_ms":76298,"temperature":0.7,"pith_summary":"Large language models perform much worse on low-resource Indic languages because their few-shot demonstrations must be built from very small labeled pools. PromptRefine is a trainable retriever that addresses this by borrowing example banks from closely related higher-resource Indic languages, merging language-specific retrievers through alternating rounds of fine-tuning and parameter averaging, and then fine-tuning the merged retriever to choose a relevant and diverse set of demonstrations. The paper evaluates this on four text generation tasks with several 7B-8B open LLMs and with GPT-3.5/GPT-4, and reports consistent gains over zero-shot prompting and over existing retrievers such as EPR and CEIL. The headline results include a +16.07 Token-F1 gain over zero-shot and +8.26 over CEIL on cross-lingual QA, and a +4.85 chrF1 gain over CEIL on Santali-to-English translation. If the method is right, low-resource languages can be served by transferring demonstrations from related languages at prompt time, without updating the LLM.","feed_headline":"Cross-lingual example banks boost low-resource Indic few-shot quality","feed_subtitle":"Averaging language-specific retrievers plus diversity selection lifts F1 by up to 16 points over zero-shot.","key_machinery":"The machinery is an alternating-minimization loop. At each round the algorithm starts from shared parameters rho, fine-tunes a separate relevance-based retriever on the target-language bank and on each selected auxiliary bank (the Specialize step), then replaces rho by the average of those retriever parameters (the Merge step). The paper treats the averaged parameter vector as a shared representation space that can retrieve relevant demonstrations across languages; the iteration with best target-language validation accuracy becomes rho, which is then fine-tuned with a determinantal point process (DPP) loss. In the DPP, the probability of selecting a subset is proportional to the determinant of a similarity kernel combining each example's relevance to the test input with pairwise repulsion, so greedy MAP inference returns a diverse but relevant set. The load-bearing operation is the parameter-average merge: it is the only mechanism that transfers knowledge from the auxiliary languages into the target retriever.","core_discovery":"On its own terms, the paper establishes that in-context example selection is the bottleneck for low-resource Indic generation, and that the bottleneck can be broken by a three-part retriever: select auxiliary high-resource languages by mean-embedding similarity above a threshold; build a shared cross-lingual representation by alternating between per-language relevance fine-tuning and parameter averaging; and add a determinantal point process loss so the retrieved subset balances relevance against redundancy. Across cross-lingual QA, multilingual QA, machine translation, and cross-lingual summarization, PromptRefine outperforms Random, BM25, Top-K, Diverse, EPR, and CEIL, with absolute gains reported up to +10.21 Token-F1 over CEIL in Manipuri cross-lingual QA and up to 2.09x relative improvement over the strongest baseline. Including auxiliary data does not help EPR or CEIL much, which the paper reads as evidence that the shared representation produced by alternating minimization is what lets auxiliary examples become useful. The paper also reports that diversity fine-tuning is necessary and that random or unrelated auxiliary languages give little gain.","pith_inferences":["Inference: If parameter averaging is the true source of cross-lingual alignment, the same alternate-specialize-average loop could transfer to other low-resource language families with a multilingual BERT-style init and related higher-resource banks; the paper only demonstrates Indic languages.","Inference: The method's reliance on mean BERT embedding similarity to pick auxiliary languages suggests script and transliteration could strongly affect which languages are deemed related, and the paper does not separate script effects from linguistic relatedness.","Inference: A direct test of the merge assumption would be to compare PromptRefine against simply training one retriever on the concatenated banks with the same relevance and DPP losses; the paper reports no such control, so the unique role of alternating minimization remains an open question.","Inference: The reported gains with 16 demonstrations suggest the selected examples partly function as cross-lingual task specification; one could test whether the same selected bank works when translated into the target language, which would isolate relevance from language identity."],"forward_implications":["PromptRefine's gains come without changing the LLM: only the demonstration subset in the prompt changes, so the approach is a drop-in addition to existing few-shot pipelines.","Related auxiliary example banks can substitute for missing target-language data; ablations show that choosing unrelated or random auxiliary languages removes most of the benefit, implying language relatedness is doing real work.","Diversity selection is not optional decoration; removing the DPP fine-tuning step lowers generation quality.","The same three-step recipe is effective across QA, translation, and summarization, so the method is task-generic within the covered generation settings.","Proprietary LLMs also improve on translation, indicating the example selection transfers beyond the open models used to train the retrievers."],"supporting_citations":[{"why":"This supplies the CEIL baseline and the DPP-based compositional exemplar selection objective that PromptRefine extends and compares against.","marker":"Ye et al. (2023)"},{"why":"This supplies EPR, the relevance-based retriever fine-tuning procedure and scorer-based candidate scoring used in the specialize step.","marker":"Rubin et al. (2021)"},{"why":"This defines the IndicGenBench tasks, the low- and high-resource language grouping, and the Token-F1 and chrF1 metrics used in every evaluation.","marker":"Singh et al. (2024)"},{"why":"This provides the greedy MAP inference routine used to build diverse subset positives during DPP fine-tuning.","marker":"Chen et al. (2018)"},{"why":"This motivates the alternating-minimization treatment of scarce multi-task data that the specialize-merge loop is built on.","marker":"Thekumparampil et al. (2021)"}],"fun_headline_variants":["PromptRefine: cross-lingual example pick boosts Indic few-shot","Auxiliary language banks improve low-resource Indic generation","Alternating minimization selects best few-shot examples for Indic","Diverse cross-lingual prompts boost Indic few-shot F1 up to 16","Related example banks: better few-shot for low-resource Indic"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that averaging the parameters of separately fine-tuned language-specific retrievers yields one shared embedding space that can retrieve useful examples across languages; the paper supports this mostly with a single validation-accuracy curve, and if the averaged space is not aligned, the auxiliary example banks would not contribute and the reported gains would shrink.","fun_headline_variants_meta":{"raw":{"variants":["PromptRefine: cross-lingual example pick boosts Indic few-shot","Auxiliary language banks improve low-resource Indic generation","Alternating minimization selects best few-shot examples for Indic","Diverse cross-lingual prompts boost Indic few-shot F1 up to 16","Related example banks: better few-shot for low-resource Indic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001225,"raw_usage":{"total_tokens":5066,"prompt_tokens":1007,"completion_tokens":4059,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":623,"completion_tokens_details":{"reasoning_tokens":3969}},"tokens_in":623,"tokens_out":4059,"duration_ms":27887,"temperature":1.0,"reasoning_tokens":3969,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:25:08.947298+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a control retriever on the concatenated target and auxiliary example banks with the same relevance loss and the same DPP loss, without any alternating specialize-merge loop, and compare downstream Token-F1 and chrF1 on the same four tasks. If the control matches or beats PromptRefine, the parameter-merging step is not the active ingredient. A second, direct test would be to measure cross-lingual nearest-neighbor agreement in the merged embedding space: if retrieved target-language examples for a given query do not overlap with related auxiliary-language examples, the shared-space assumption is violated.","supporting_citations":[{"cited_title":"Fast Greedy MAP Inference for Determinantal Point Process to Improve Recommendation Diversity","cited_arxiv_id":"1709.05135","evidence_quote":"This provides the greedy MAP inference routine used to build diverse subset positives during DPP fine-tuning."}],"review_version":1}