{"id":"a5150f7a-b9dc-48b5-9fbc-ef4fc19f0270","arxiv_id":"2506.15415","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":9,"one_line_summary":"Using Layer 2 embeddings in Lugha-Llama, the paper reports a 28% relative increase in final-layer cosine similarity for Swahili-English pairs, but the supporting layer scan contains an internal contradiction and the control set is not fully unseen.","lead":"Targeted Lexical Injection fine-tunes a Swahili-English LLM with LoRA and a contrastive loss aimed at an early layer, reporting final-layer word-pair similarity gains from about 0.32 to 0.41, including for held-out pairs. A generalist might read this as a cheap way to improve cross-lingual alignment in low-resource languages, but the paper's layer scan contradicts its own baseline and lacks baseline comparisons.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The pilot scan's Layer 31 value (0.9876, §3.2) contradicts the Pre-TLI output-layer baseline (0.3211/0.3143, §4.1 Table 1) for the same layer, model, and curated set; until resolved, the pilot's Layer 2 'latent alignment' evidence and the TLI mechanism are unsupported.","rationale":"The reader's weakest_assumption was about dimensional collapse at Layer 2, and the absence of unrelated-pair controls. That is a real concern, but the more direct and load-bearing problem is the internal contradiction between the pilot scan's Layer 31 value and the Section 4.1 baseline. The paper itself flags this discrepancy in Section 3.2, but the caveat is unsupported: the pilot is said to use the Section 3.3 curated set, and the extraction protocols are described identically. Since the paper provides no code, no per-pair list, and no clarification of the 'specific conditions or word subset,' the pilot evidence for strong latent alignment at Layer 2 cannot be trusted. That evidence is the foundation for TARGET_LAYER selection and for the mechanistic story that TLI 'unlocks latent alignment' rather than simply improving output similarity through generic fine-tuning. The statistical improvements in Table 1 might survive a re-run, but the paper's central contribution as framed, early-layer LoRA targeting an empirically identified optimal layer, would be unsupported if the pilot is invalid. This is a correctable flaw in principle, but as written it warrants rejection. A concrete reproduction of the pilot under the evaluation protocol would settle the issue; if the reproduction matches Table 1 and Layer 2 random-pair similarities are low, the paper could be substantially revised and resubmitted.","tokens_in":9423,"tokens_out":4449,"duration_ms":45843,"concrete_test":"Reproduce the pilot layer scan on the full 686-pair set using the exact Section 3.5 evaluation protocol applied to the base model: extract Layer 31 hidden states, mean-pool over attention-masked tokens, L2-normalize, and compute cosine similarity. If the Layer 31 mean is approximately 0.3211 (matching Table 1), then the pilot's 0.9876 came from a different protocol or subset and the pilot evidence is invalid; if it is approximately 0.9876, then the discrepancy with Table 1 must be explained. As a secondary control within the same run, also compute the Layer 2 cosine similarity for 100 random Swahili-English non-translation pairs; if these also exceed 0.99, the near-perfect Layer 2 values are a global geometric artifact rather than evidence of cross-lingual semantic alignment.","verdict_should_be":"REJECT","load_bearing_attack":"The central empirical chain is broken by an internal contradiction. Section 3.2 states that the pilot study used the curated set described in Section 3.3, which includes all 686 pairs, and that Layer 31 of the base model had an average cosine similarity of 0.9876. Section 3.5 and Table 1 report the Pre-TLI baseline at the same Layer 31 as 0.3211 on the 623 training pairs and 0.3143 on the 63 control pairs. The extraction protocols are described identically: mean pooling over attention-masked tokens, L2 normalization, and cosine similarity; the only stated difference is that the pilot scans all layers while evaluation uses Layer 31. For the same model, same layer, and same set, these two numbers cannot both hold. The paper's own caveat that the Layer 31 pilot value 'may reflect the specific conditions or word subset used' is ad hoc and contradicts the stated use of the Section 3.3 set. If the pilot protocol inflates Layer 31 similarity to 0.9876, then the headline pilot result, Layer 2 at 0.99998, is equally suspect. That would invalidate the selection of TARGET_LAYER and the central claim that the model 'possesses strong, latent cross-lingual lexical alignment' at Layer 2. Consequently, the proposed mechanism, that TLI preserves and propagates pre-existing early-layer alignment, is not supported. The measured Post-TLI improvements could be an artifact of LoRA fine-tuning under a different, unrelated mechanism, but without a trustworthy pilot, the paper's distinctive contribution is unsubstantiated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Targeted Lexical Injection (TLI), a LoRA-based fine-tuning method for Lugha-Llama-8B-wura that aims to improve output-level Swahili-English lexical alignment. A pilot scan of layerwise cosine similarities on a curated set of 686 word pairs identifies Layer 2 as having near-perfect average similarity (0.99998), while the final layer shows much lower similarity. TLI then trains LoRA adapters with an in-batch triplet contrastive loss applied to Layer 2 embeddings and reports that mean cosine similarity at the final layer increases from 0.3211 to 0.4113 for 623 trained pairs and from 0.3143 to 0.4033 for 63 control pairs. The paper interprets these gains as evidence that the model possesses latent early-layer alignment that TLI helps preserve and propagate to the output.","tokens_in":9817,"tokens_out":5006,"duration_ms":50448,"significance":"If the empirical claims were reliable, the paper would make a useful, parameter-efficient contribution to adapting LLMs to low-resource languages and to understanding layerwise cross-lingual representations. The core idea is clear, the quantitative reporting is detailed, and the intention to test generalization with a held-out control set is sensible. However, the central evidence currently contains an internal contradiction and a circularity in layer selection, and the proposed mechanism is not tested against plausible alternative explanations. As it stands, the paper does not establish the claimed latent-alignment phenomenon or the mechanism by which TLI is said to improve output-level alignment.","major_comments":[{"comment":"Section 3.2 reports that the pilot layer scan on the Section 3.3 curated set gives an average cosine similarity of 0.9876 at Layer 31, but Section 4.1 and Table 1 report the Pre-TLI baseline at Layer 31 as 0.3211 on the 623 trained pairs and 0.3143 on the 63 control pairs. The extraction procedures are described identically (mean pooling, L2 normalization, cosine similarity), and the pilot is stated to use the same curated set, so these numbers cannot both be correct for the same model, layer, and set. The caveat in Section 3.2 that the Layer 31 pilot value \"may reflect the specific conditions or word subset used\" is ad hoc and contradicts the stated use of the Section 3.3 set. Because the Layer 2 value of 0.99998 comes from the same scan, this contradiction undermines the central claim that the model possesses near-perfect latent alignment at Layer 2 and therefore also the choice of TARGET_LAYER.","section":"§3.2, §4.1, Table 1"},{"comment":"TARGET_LAYER=2 was selected from a pilot scan that uses \"the curated set described in Section 3.3,\" which is the full set of 686 pairs later split into 623 training pairs and 63 control pairs. Because the 63 control pairs contributed to the layer-selection decision, the claim in Sections 4.2 and 5.1 that TLI generalizes to \"unseen\" control pairs is not independent. The control pairs were unseen during LoRA training, but they were not unseen during the design step that chose the target layer. The layer selection should be rerun on the training split only, with the control split held out from all design decisions.","section":"§3.2, §3.3"},{"comment":"The pilot scan reports only similarities between Swahili-English translation pairs at each layer, with no comparison to unrelated Swahili-English pairs. The near-perfect average at Layer 2 (0.99998) is therefore not sufficient evidence of semantic alignment: if all token embeddings in early residual-stream layers occupy a narrow cone or an effectively low-dimensional subspace, translation pairs and random pairs alike could show cosine similarities close to 1. The paper needs to report non-translation-pair similarities at Layer 2, and ideally embedding norms or effective rank, before the \"latent lexical alignment\" claim can be accepted.","section":"§3.2"},{"comment":"The paper's mechanistic interpretation is that TLI works because it targets Layer 2 specifically, but no ablation compares TLI with LoRA/contrastive training on other layers (for example, Layer 31) or with a standard fine-tuning objective. Without such baselines, the observed improvement at the output layer could be a generic effect of contrastive LoRA fine-tuning rather than evidence that the model's early-layer alignment is being preserved and propagated. This should be tested before the proposed mechanism is stated as established.","section":"§5.2, §5.3"}],"minor_comments":[{"comment":"The appendix says that details of the contrastive loss implementation \"would typically be provided here\" and refers to a conceptual implementation, but no code, script, or repository link is included. Since the paper's quantitative results are central, the actual training script or a link to it should be provided.","section":"Appendix 7.1"},{"comment":"The t-SNE visualizations are described with qualitative terms such as \"loosely grouped,\" \"diffuse,\" and \"visibly much closer,\" without a quantitative link to the reported average improvements; t-SNE is stochastic and can exaggerate perceived structure, so the figures should be interpreted with caution or supplemented with quantitative embedding-space diagnostics.","section":"Figures 2 and 3"},{"comment":"The procedure for splitting the 686 curated pairs into 623 training and 63 control pairs is not described, and no randomization seed or selection criterion is reported, so the split is not reproducible.","section":"§3.3"},{"comment":"The citation for the SimCLR method is given as \"Chen, T., K. S. N. M. and Hinton, G. (2020)\"; this should be corrected to Chen, Kornblith, Norouzi, and Hinton (2020).","section":"References"},{"comment":"Equation (1) defines the triplet margin loss, but the margin value and the use of L2-normalized embeddings are mentioned only in the surrounding text; stating them directly inside or immediately below the equation would improve clarity.","section":"§3.4"}],"recommendation":"reject","confidential_remarks":"I recommend rejection. The central empirical chain is currently broken by the contradiction between the pilot Layer 31 value (0.9876) and the Table 1 baseline at the same layer and set (0.3211/0.3143), and the layer selection is circular with respect to the control pairs. A future version that resolves this contradiction, performs layer selection on the training split only, and adds non-translation-pair baselines and layer ablations could be considered afresh."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper has a genuinely new idea — a layer-wise scan of Lugha-Llama showing near-perfect Swahili-English alignment at Layer 2, then LoRA fine-tuning with a contrastive loss on those early embeddings. That recipe could be useful for low-resource language work. But the pilot study's numbers directly contradict the paper's own evaluation baseline, and that breaks the empirical chain.\n\nWhat's new: the specific observation that this model's Layer 2 carries near-1.0 cosine similarity for translation pairs, and that targeting that layer with LoRA produces output-level gains that generalize to a held-out set. That combination isn't in the cited literature. The writing is clear, and the limitations section is unusually honest.\n\nThe problem is in the numbers. Section 3.2 reports the pilot scan's Layer 31 average similarity as 0.9876 for the same curated set and extraction protocol, while Section 4.1 reports the Pre-TLI baseline at Layer 31 as 0.3211 on the training set (and 0.3143 on controls). Both cannot be true. The paper waves this away by saying the pilot's Layer 31 value 'may reflect the specific conditions or word subset used,' but the text states the pilot used the same set from Section 3.3. That caveat is ad hoc. If the pilot's Layer 31 number is inflated, the Layer 2 number (0.99998) is equally suspect.\n\nSecond, the 63 control pairs were inside the pilot scan that selected Layer 2, so they aren't truly unseen for the design choice. They are unseen during LoRA training, but they influenced the choice of TARGET_LAYER; the claim that the boost generalizes because of pre-existing alignment is thereby contaminated.\n\nThird, there's no random-pair comparison at Layer 2. Near-1.0 cosine similarity across all tokens would be a geometric artifact, not evidence of cross-lingual alignment. The paper never checks that. There's also no baseline of LoRA fine-tuning on the output layer or without contrastive loss, so the +0.09 improvement can't be attributed to the early-layer mechanism.\n\nWho's this for? People interested in layer-wise interpretability and parameter-efficient adaptation for African languages. The idea is worth pursuing. As submitted, the pilot scan is load-bearing and it contradicts the evaluation. I'd tell the authors to fix the pilot protocol, add random-pair controls, and re-select the layer without the control pairs. If they do that, I'd send it to review. Even now, I think the novelty is enough that a serious editor should get a referee's opinion rather than desk-reject outright — the review should focus on the pilot.","headline":"A novel early-layer LoRA fine-tuning idea for low-resource lexical alignment, but the pilot scan contradicts its own evaluation baseline, so the central claim doesn't hold as written.","tokens_in":10344,"tokens_out":4298,"would_cite":false,"duration_ms":40340,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning on a single early layer of Lugha-Llama with a contrastive objective raises final-layer Swahili–English word similarity by 28%, including for words never seen during training.","keywords":["Low-Resource Languages","Swahili","Large Language Models","Cross-Lingual Lexical Alignment","Fine-Tuning","LoRA","Contrastive Learning","Model Interpretability"],"falsifier":"Measure the average cosine similarity of, say, 100 randomly paired Swahili and English words that are not translations at Layer 2 of the base model. If that average is close to 1, then the pilot's 0.99998 does not demonstrate translation-specific alignment, and the paper's explanation for TLI's gains loses its foundation.","tokens_in":9186,"feed_emoji":"🧠","tokens_out":11914,"duration_ms":99340,"temperature":0.7,"pith_summary":"The paper tries to show that a Swahili-centric language model already forms near-perfect Swahili–English word equivalences inside an early transformer layer, and that a cheap fine-tuning step can make that knowledge visible at the output. It introduces Targeted Lexical Injection (TLI), which trains LoRA adapters with a contrastive loss applied to embeddings from Layer 2, the layer where the pilot scan finds average cosine similarity of 0.99998. On the paper's evaluation set, final-layer cosine similarity rises by about 28% for both the 623 trained word pairs (0.3211 to 0.4113) and the 63 unseen control pairs (0.3143 to 0.4033). If true, this matters because it suggests lexical alignment for low-resource languages can be improved with a small curated word list and few trainable parameters, rather than large parallel corpora.","feed_headline":"Targeting one early layer lifts Swahili–English word matches by 28%","feed_subtitle":"A 623-pair contrastive fine-tune also lifts unseen translations, suggesting the model already had the alignment inside.","key_machinery":"The central object is Targeted Lexical Injection (TLI): LoRA adapters of rank 16 with alpha 32 and dropout 0.05 on the query and value projections, trained with an in-batch triplet margin loss (margin 0.4) on length-normalized, mean-pooled embeddings extracted from Layer 2. The loss pulls each Swahili anchor toward its English translation and pushes it away from the hardest in-batch negative. Because the contrastive pressure is applied at the layer where alignment is already near-perfect, the LoRA updates teach the network to carry that alignment forward to the final output layer, which the paper argues is why trained and unseen pairs improve by nearly the same amount.","core_discovery":"Lugha-Llama-8B-wura already contains strong cross-lingual lexical alignment early in its computation: in the paper's pilot scan, average cosine similarity between Swahili–English translation pairs at Layer 2 is 0.99998, while the same pairs reach only around 0.32 at the final layer. The paper proposes Targeted Lexical Injection (TLI), which fine-tunes LoRA adapters using a contrastive objective on Layer 2 embeddings rather than on output embeddings. After TLI, final-layer mean cosine similarity increases from 0.3211 to 0.4113 on 623 trained pairs and from 0.3143 to 0.4033 on 63 unseen control pairs, gains of about 28% in both settings with tiny p-values. The paper interprets this as TLI strengthening the model's ability to preserve and propagate its pre-existing early-layer alignment to the output, rather than memorizing the training pairs.","pith_inferences":["Editorial inference: the optimal target layer is likely not fixed at Layer 2 across architectures and language pairs, so a layerwise similarity scan like the pilot could become a routine preliminary step for similar interventions.","Editorial inference: if the near-perfect Layer 2 similarity is partly an artifact of anisotropic embedding geometry, the 28% output gain might be explained by a global reshaping of the embedding space rather than by unlocking semantic knowledge; the method could still be useful, but for a different reason.","Editorial inference: a direct test of the generalization story would be to check whether post-TLI output embeddings approximately preserve Layer 2 angles; if the adapters truly improve propagation, a near-linear angle-preserving mapping between layers should emerge.","Editorial inference: word-pair cosine similarity is an isolated lexical measure, so the practical value of TLI would be better established by measuring downstream translation or cross-lingual retrieval before and after the intervention."],"forward_implications":["Trained-pair output alignment improves from 0.3211 to 0.4113 mean cosine similarity (+28.08%, p < 1.33 × 10⁻²⁴⁰).","Unseen control pairs improve from 0.3143 to 0.4033 (+28.32%, p < 7.17 × 10⁻²⁷), so the effect is not limited to memorized vocabulary.","The method needs only a few hundred word pairs and LoRA adapters, making it parameter-efficient and data-efficient for low-resource languages.","The paper's interpretation implies TLI refines the propagation of internal alignments, so downstream tasks that rely on lexical equivalence could benefit without retraining on parallel corpora."],"supporting_citations":[{"why":"supplies the base model Lugha-Llama-8B-wura whose layers are scanned and then fine-tuned.","marker":"Lugha Factory, 2023"},{"why":"supplies the SAWA corpus from which the 686 Swahili–English word pairs are curated.","marker":"Sim et al., 2022"},{"why":"supplies LoRA, the parameter-efficient adaptation method that TLI uses.","marker":"Hu et al., 2021"},{"why":"supplies the contrastive/triplet margin principle behind TLI's loss.","marker":"Hadsell et al., 2006"},{"why":"grounds the notion of cross-lingual representation alignment that the paper measures and targets.","marker":"Conneau et al., 2019"}],"fun_headline_variants":["Early-layer LoRA unlocks 28% better Swahili-English alignment","Targeting layer 2, not output, lifts word matches by 28%","Hidden layer held perfect alignment; TLI surfaces it to output","Contrastive LoRA on layer 2 boosts unseen pairs by 28%","Lugha-Llama's latent cross-lingual link revealed via early LoRA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the near-perfect Layer 2 cosine similarity (0.99998) reflects genuine translation-specific semantic alignment, rather than the geometric accident that all early-layer word embeddings point in nearly the same direction.","fun_headline_variants_meta":{"raw":{"variants":["Early-layer LoRA unlocks 28% better Swahili-English alignment","Targeting layer 2, not output, lifts word matches by 28%","Hidden layer held perfect alignment; TLI surfaces it to output","Contrastive LoRA on layer 2 boosts unseen pairs by 28%","Lugha-Llama's latent cross-lingual link revealed via early LoRA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"total_tokens":1575,"prompt_tokens":1100,"completion_tokens":475,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":716,"completion_tokens_details":{"reasoning_tokens":373}},"tokens_in":716,"tokens_out":475,"duration_ms":5677,"temperature":1.0,"reasoning_tokens":373,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:34:55.982310+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the average cosine similarity of, say, 100 randomly paired Swahili and English words that are not translations at Layer 2 of the base model. If that average is close to 1, then the pilot's 0.99998 does not demonstrate translation-specific alignment, and the paper's explanation for TLI's gains loses its foundation.","supporting_citations":[{"cited_title":"Lugha-llama-8b-wura","cited_arxiv_id":null,"evidence_quote":"supplies the base model Lugha-Llama-8B-wura whose layers are scanned and then fine-tuned."},{"cited_title":"H., Mazula, R.-M., Kawambwa, S","cited_arxiv_id":null,"evidence_quote":"supplies the SAWA corpus from which the 686 Swahili–English word pairs are curated."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the contrastive/triplet margin principle behind TLI's loss."}],"review_version":2}