{"id":"6dbb33ee-c23a-4077-9a0d-cb6abd514851","arxiv_id":"2412.08955","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"The paper reports a new closed-loop XICL method with high accuracy gains, but the method and experiments are too under-specified to validate the claims.","lead":"This paper proposes a self-supervised framework for cross-lingual in-context learning by adding two losses and a reinforcement learning loop. Its claimed state-of-the-art results are not supported by concrete experimental details, code, or data.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Load-bearing concern: Eq. (4) samples example sets C from pθ, but pθ is only defined for token sequences y in Eq. (1); with no generative model over C, the loss, the RL loop, and the reported 76.1% accuracy are not well-defined and cannot be reproduced.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: Eq. (4) samples C from pθ, but pθ is never defined over C. I focused on this rather than on the encoder in Eq. (3) or the use of task accuracy in Eq. (5) because the undefined distribution over C is the first point at which the entire training procedure becomes non-executable. If the sampling distribution over C is not specified, then L_align, L_coherence, and L_RL are all formally incomplete, and the experimental comparisons cannot be interpreted. This is not a disagreement with a baseline or a matter of external consensus; it is an internal inconsistency in the method description. The concrete test I propose is the minimal check that would settle the matter: either the authors can specify a sampling procedure for C, in which case the framework can be implemented and tested, or they cannot, in which case the paper does not actually describe a method with the claimed properties. I do not see a way to rescue the central claim by charitable reading, because 'C ∼ pθ' is a type error: the model distribution is over output sequences, not over example sets, and no transition from one to the other is given. The reader's verdict of REJECT remains appropriate; the paper should not be accepted or conditionally accepted until this definition is supplied and the experiments are reproducible. I agree with the reader's assessment and do not identify an independent concern that would change the verdict.","tokens_in":7852,"tokens_out":2613,"duration_ms":29042,"concrete_test":"Ask the authors to provide the exact sampling procedure for C, including the prompt used, the decoding parameters, and whether C is sampled from a defined distribution or generated heuristically. Then implement the most charitable concrete version: define C as k examples sampled by prompting the base LLM with query x and a fixed instruction, compute Eq. (4) with this sampling distribution, and re-run Table 1 on the same dataset and base model. If the proposed loss cannot be computed because pθ has no marginal over C, or if the reported 76.1% accuracy is not reproduced under the specified sampling procedure, the central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that a self-supervised training pipeline improves cross-lingual in-context learning, but the pipeline as written is not a well-defined procedure. In Eq. (1), pθ is defined only as a conditional distribution over output token sequences y given x and an example set C: pθ(y | x, C) = ∏_t pθ(y_t | x, C, y<t). This is a distribution over y, not over C. Yet Eq. (4) writes the RL objective as L_RL = −E_{x, C∼pθ}[R(x, C)], treating pθ as a distribution over example sets. No marginalization, generative process, or surrogate distribution over C is ever specified. The same undefined dependence on C appears in Eq. (2), where the expectation is over x and C, and in the description that 'the model generates synthetic example pairs C during training.' The paper never says how C is sampled: Is C produced by prompting the model? If so, with what prompt, decoding scheme, and temperature? Is it sampled from a distribution proportional to pθ(y | x, C) marginalized over y? Such a marginal is not even defined because pθ is a conditional distribution over y, not a joint distribution over (y, C). Without a concrete sampling procedure, the training objective in Eq. (6) cannot be instantiated, gradients cannot be computed, and the entire experimental section is unmoored. Every reported result, including the 76.1% overall accuracy in Table 1 and the ablations in Table 2, is attributed to a method whose training loop is formally underspecified. This is not a stylistic gap: it is a missing definition at the core of the proposed framework. The problem is compounded by Eq. (3), which refers to 'the model's encoder' in a decoder-only LLM, and by Eq. (5), which requires task accuracy A(x, C) in a supposedly self-supervised setting. But the first and most load-bearing defect is that the distribution over C in Eq. (4) does not exist as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a closed-loop, self-supervised framework for cross-lingual in-context learning (XICL). The high-level idea is to train a large language model to generate its own in-context example sets, using a retrieval-generation alignment loss, a semantic coherence loss, and a reinforcement-learning objective that rewards task accuracy and semantic diversity. The authors report state-of-the-art results on a multilingual benchmark, an ablation study, human evaluation, and generalization to unseen tasks. However, the formal description of the training objective is underspecified and internally inconsistent: the model distribution is defined only over output sequences, the coherence loss assumes an encoder that is never introduced, and the reward requires labels in a pipeline claimed to be self-supervised. The experimental section also omits the dataset, base model, and hyperparameters, so the reported numbers cannot be verified.","tokens_in":8399,"tokens_out":7437,"duration_ms":67679,"significance":"If the proposed training pipeline were fully specified and validated, the idea of having an LLM generate and refine its own in-context examples could be a useful step toward removing external retrievers in cross-lingual settings. The paper identifies a real limitation and has a plausible high-level direction. However, as submitted, the significance cannot be assessed because the central objectives are not well-defined and the empirical support is incomplete. The reported accuracy gains, ablation trends, and human evaluation scores are all attributed to a procedure that is not reproducible from the text.","major_comments":[{"comment":"Eq. (4) defines L_RL = -E_{x,C∼pθ}[R(x,C)], treating pθ as a distribution over example sets C. Yet Eq. (1) defines pθ only as a conditional distribution over output token sequences y given x and C: pθ(y | x, C). No generative process, marginalization, or sampling scheme for C is specified anywhere in the paper; in particular, the text does not state what prompt, decoding scheme, or temperature is used to produce C, nor how a distribution over sets of pairs would be normalized. Because the RL objective and the alignment objective in Eq. (2) both depend on this undefined distribution, the overall loss in Eq. (6) cannot be instantiated, and the experimental results in Tables 1–5 are not grounded in a well-defined training procedure.","section":"§3.3, Eq. (4)"},{"comment":"Eq. (3) requires computing semantic embeddings hx and hy with 'the model's encoder,' but the model is presented in §3.1 as a generative sequence model pθ(y | x, C) with no encoder component. The paper never defines how hx and hy are obtained, what the encoder architecture is, or how its parameters relate to θ. In addition, because the loss directly minimizes the squared ℓ2 distance between hx and hy, the claimed 'cross-lingual consistency' is enforced by construction rather than discovered; a separate, non-training evaluation would be needed to support the paper's consistency claims.","section":"§3.2, Eq. (3)"},{"comment":"The reward in Eq. (5), R(x,C)=αA(x,C)+βD(C), depends on task accuracy A(x,C), which by standard definition requires ground-truth labels for the query x. This conflicts with the paper's repeated claim that the framework is self-supervised (see Abstract, §1, and §3.2). The authors must explain how A(x,C) is evaluated without labeled data, or revise the self-supervision claim. The term D(C) ('semantic diversity') is also never formally defined.","section":"§3.3, Eq. (5)"},{"comment":"The experimental section does not provide the dataset name, the base LLM, the model size, the number of languages and tasks, the train/validation splits, or the values of the hyperparameters α, β, λ, and γ. It also does not describe the optimization procedure (optimizer, learning rate, number of steps, compute). The sentence in §4.1 that all models use 'consistent hyperparameters' does not make the experiments reproducible, and it is impossible to tell whether the comparison in Tables 1–5 is controlled.","section":"§4.1"},{"comment":"The strongest baseline, XICL-RL, is used in Tables 1, 3, and 5, but it is neither defined nor cited anywhere in the paper; 'Glot500 RET' is likewise introduced without a reference or implementation description. Without a specification of these baselines, the claimed state-of-the-art improvement in Table 1 cannot be evaluated.","section":"§4.2, Table 1"},{"comment":"Tables 1–3 report only point estimates with no standard deviations, confidence intervals, or significance tests, yet the abstract and §4.2 use the word 'significantly.' The claims of significance and of robustness across language families (Table 4) are not supported by the reported statistics.","section":"Tables 1–3"}],"minor_comments":[{"comment":"The expectation E_{x,C} does not specify the distribution over x or C; even apart from the missing distribution over C, all expectations in the paper should be written with explicit probability measures.","section":"§3.2, Eq. (2)"},{"comment":"Table 1 is captioned as average accuracy, Table 5 reports F1, and the text mentions BLEU for translation tasks, but no translation experiments are shown; the metrics used in each table should be defined in the experimental setup.","section":"Tables 1 and 5"},{"comment":"Table 4 gives aggregate accuracy by language family without sample sizes or variance; the claim that the method is robust across families needs per-language or at least per-family breakdowns with counts.","section":"Table 4"},{"comment":"Table 6 is described as a text classification example, but the generated outputs and ground truth are full sentences rather than class labels; the example should be reconciled with the task or replaced.","section":"Table 6"},{"comment":"Some cited references appear unrelated to the claims they support; for example, Bellström and Kilbrink (2008), a paper on problem-based learning in programming, is cited for the cross-lingual capabilities of large language models in the Introduction. The citation list should be checked carefully.","section":"References"}],"recommendation":"reject","confidential_remarks":"The high-level idea has some intrinsic interest, but the manuscript in its current form is not close to publishable: the formalization of the training objective is incomplete, and the experimental section is too underspecified to support the central claims. I also note for the editor that the reference list contains several entries that seem unrelated to the text (e.g., Bellström & Kilbrink 2008), and the paper relies heavily on self-citations to 'Zhou et al.' without clear connections; this further reduces confidence in the care with which the manuscript was prepared."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the reader's REJECT is right, and the stress-test pinpoints the load-bearing flaw. Eq. (4) samples example sets C from pθ, but pθ is only defined for token sequences y given x and C. There is no generative model over C anywhere in the paper, no marginalization, no sampling procedure. That means the RL objective, the alignment loss, and the whole reported accuracy gain are not well-defined. This is not a stylistic gap; the method cannot be instantiated as written.\n\nTo give credit where it's due, the high-level problem is real: XICL typically relies on external retrievers or task-specific fine-tuning, and a retriever-free, self-supervised way to let the LLM internally select examples would be useful. The paper is clearly written at the level of prose, and the related work covers the relevant threads. The ablation design is sensible in spirit.\n\nBut the gaps are everywhere. Eq. (3) calls it an encoder when the model is a decoder-only LLM — a direct contradiction. Eq. (5) uses task accuracy A(x,C), which requires labels, yet the pipeline is called self-supervised. The experiments report numbers without naming datasets, models, or hyperparameters, and without error bars, so reproduction is impossible. And Table 6, labeled low-resource, shows English outputs. It reads like the experimental section was assembled to a template.\n\nWho is this for? Maybe a reader wanting a sketch of what a closed-loop XICL could look like, but not anyone wanting a working method. The citations are fine. The math is not. A serious referee could help the authors, but the paper in its current form has too many undefined core objects to justify sending out; it should go back for a complete rewrite and real experiments. If a revised version defines the sampling distribution, replaces the encoder with a decoder-compatible representation, reconciles self-supervision with task accuracy, and reports a concrete experimental setup, I'd revisit.","headline":"A self-supervised XICL framework whose headline claim is undercut by an undefined sampling distribution over example sets; the equations do not cohere.","tokens_in":8802,"tokens_out":3146,"would_cite":false,"duration_ms":28134,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A language model that generates its own in-context examples outperforms retrieval-based baselines on multilingual tasks.","keywords":["cross-lingual in-context learning","large language models","low-resource languages","retrieval-generation alignment","semantic coherence","reinforcement learning","self-supervised training","multilingual benchmarks"],"falsifier":"Run the same three-term training loss on a fixed backbone while replacing the model-generated example sets with random subsets of the training corpus. If accuracy stays near the reported 76.1%, the alignment and coherence losses alone explain the gain; if it drops toward the 69.5% baseline, the internal example-generation mechanism is doing the work. Either outcome would pin down what the closed-loop claim actually contributes.","tokens_in":7631,"feed_emoji":"🌐","tokens_out":9743,"duration_ms":85555,"temperature":0.7,"pith_summary":"The paper argues that cross-lingual in-context learning can be driven entirely by the LLM's own generative abilities, without external retrievers or task-specific fine-tuning. It proposes a closed-loop training scheme in which the model generates candidate example sets for a query, aligns retrieval and generation through a KL-divergence objective, enforces cross-lingual semantic coherence in embedding space, and refines example selection with reinforcement learning. The reported experiments put this framework ahead of the strongest baseline, XICL-RL, on average accuracy (76.1% vs. 69.5%), with larger margins in low-resource languages, and the ablation study attributes the gain to both training objectives. If correct, the result would make multilingual few-shot learning more scalable by removing the retrieval component from the pipeline.","feed_headline":"Model-written examples beat cross-lingual retrieval in LLMs","feed_subtitle":"The paper's self-supervised training loop lifts accuracy to 76.1 percent, 6.6 points above the strongest baseline.","key_machinery":"The load-bearing object is the closed-loop training objective $\\mathcal{L} = \\mathcal{L}_{\\text{align}} + \\lambda \\mathcal{L}_{\\text{coherence}} + \\gamma \\mathcal{L}_{\\text{RL}}$. $\\mathcal{L}_{\\text{align}}$ pushes the model's output distribution with selected examples toward its output distribution without them, so the chosen examples carry information rather than noise; $\\mathcal{L}_{\\text{coherence}}$ pulls the encoder representations of input and output together across languages; $\\mathcal{L}_{\\text{RL}}$ rewards example sets that improve task accuracy and semantic diversity. The design calls for the model to generate synthetic example pairs $C$ itself, evaluate them with the reward, and update $\\theta$, turning example selection into a trainable behavior.","core_discovery":"The central claim is that an LLM can be trained to internally select and exploit task-relevant examples across languages, making an external retrieval step unnecessary. Formally, the paper defines a generative model $p_\\theta(y \\mid x, C)$ for output $y$ given input $x$ and example set $C$, and optimizes three objectives: a retrieval-generation alignment loss $\\mathcal{L}_{\\text{align}} = \\mathrm{KL}(p_\\theta(y \\mid x, C) \\| p_\\theta(y \\mid x))$, a semantic coherence loss on encoder embeddings $\\|h_x - h_y\\|^2$, and a reinforcement-learning objective that maximizes a reward $R(x, C) = \\alpha A(x, C) + \\beta D(C)$ combining task accuracy and semantic diversity. The reported outcome is a consistent accuracy improvement over random sampling, SBERT retrieval, Glot500 RET, and the XICL-RL baseline, with the largest gains on low-resource and typologically distant languages; human evaluation and unseen-task tests are offered as supporting evidence.","pith_inferences":["An implicit testable extension: if closed-loop example selection is the true source of the gain, the same training scheme should also improve monolingual in-context learning, since nothing in the objective is cross-lingual-specific except the evaluation.","Because the paper never specifies how the example set $C$ is sampled, a fair replication would need to fix that procedure; the reported gains may depend on a particular generation heuristic rather than on the loss formulas alone.","A direct comparison with an oracle retriever using gold examples would show whether the internal selector matches, approaches, or falls short of external selection, clarifying what the closed-loop claim actually buys.","The diversity term in the reward predicts that example sets with broader linguistic coverage improve accuracy; this is measurable by varying $\\beta$ and watching low-resource performance."],"forward_implications":["External retrievers become optional: a single trained LLM could select its own in-context examples at inference time.","The framework should transfer to unseen tasks such as summarization and named entity recognition without fine-tuning, as the paper's generalization experiments claim.","Low-resource and typologically diverse languages stand to gain the most, since the reported margins over baselines are largest there.","Both training objectives are load-bearing: dropping either one lowers overall accuracy by roughly three to four points in the ablation.","Human-rated relevance, fluency, and semantic correctness all improve over the strongest baseline, supporting practical usability beyond accuracy."],"supporting_citations":[{"why":"MetaICL meta-trains models on diverse tasks to learn in context; the paper's self-supervised example-selection training is positioned as an extension of this idea.","marker":"Min et al., 2022"},{"why":"Frames in-context learning as implicit Bayesian inference; motivates the retrieval-generation alignment objective that keeps outputs consistent with selected examples.","marker":"Xie et al., 2022"},{"why":"Shows that many-shot in-context learning with long contexts improves performance; supports the claim that internally generated example sets can drive cross-lingual gains.","marker":"Bertsch et al., 2024"},{"why":"Analyzes how LLMs handle multilingualism and pivot-language bias; supplies the cross-lingual challenge the framework targets.","marker":"Zhao et al., 2024"}],"fun_headline_variants":["LLMs self-select examples to beat cross-lingual retrieval","Closed-loop training lifts cross-lingual in-context learning","No external retrievers: LLMs pick their own multilingual examples","Self-supervised example selection boosts XICL accuracy","Align, generate, learn: LLMs master cross-lingual ICL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework's main training loop requires the model to sample sets of example inputs and outputs from its own distribution, but the paper never explains how those example sets are produced, so the mechanism that supposedly carries the gain is left undefined.","fun_headline_variants_meta":{"raw":{"variants":["LLMs self-select examples to beat cross-lingual retrieval","Closed-loop training lifts cross-lingual in-context learning","No external retrievers: LLMs pick their own multilingual examples","Self-supervised example selection boosts XICL accuracy","Align, generate, learn: LLMs master cross-lingual ICL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000226,"raw_usage":{"total_tokens":1472,"prompt_tokens":953,"completion_tokens":519,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":432}},"tokens_in":569,"tokens_out":519,"duration_ms":5497,"temperature":1.0,"reasoning_tokens":432,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:20:43.984756+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same three-term training loss on a fixed backbone while replacing the model-generated example sets with random subsets of the training corpus. If accuracy stays near the reported 76.1%, the alignment and coherence losses alone explain the gain; if it drops toward the 69.5% baseline, the internal example-generation mechanism is doing the work. Either outcome would pin down what the closed-loop claim actually contributes.","supporting_citations":[],"review_version":1}