{"id":"6336cee7-765e-47ec-982c-9139e0d51091","arxiv_id":"2508.03999","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":2,"one_line_summary":"Clustering training data by embedding similarity and jointly CP-decomposing LoRA adapters cuts multi-task merging interference: +1.4% on Phi-3 and +2.3% on Mistral-7B over SVD baselines.","lead":"This paper proposes TC-LoRA, a merging recipe that groups training data by similarity, trains one LoRA adapter per group, and factorizes the adapters jointly, reporting small accuracy gains over earlier merging methods. It targets the practical problem that merged multi-task models often get worse at every task, which limits cheap deployment of many lightweight adapters.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Aggregate gains do not establish the mechanism: without ablations, random clustering or SVD merging of per-cluster adapters may match TC-LoRA.","rationale":"The reader's verdict is UNVERDICTED because the full text is corrupted and the abstract alone cannot support verification. I agree that the load-bearing assumptions are the clustering signal and the CP disentanglement, but I frame the concern as an evidence gap rather than a demonstrated flaw: the abstract's aggregate numbers are consistent with simpler explanations (e.g., per-cluster training alone, or any low-rank merge), and no ablation evidence is available in the readable portion. The concrete test targets exactly this by isolating each mechanism. I do not see an internal inconsistency in the described method, and I am not claiming the paper is wrong; I am claiming the central causal claim is currently under-supported by the available evidence. Therefore the reader's UNVERDICTED verdict should stand, hence UNCHANGED. The duplicate '( +2.3%)' in the abstract is a likely typo but not a substantive concern.","tokens_in":8983,"tokens_out":4054,"duration_ms":56027,"concrete_test":"Obtain a clean copy of arXiv:2508.03999 and locate or rerun the controlled ablation: construct TC-LoRA variants with (a) random cluster assignment over the same samples with the same cluster sizes, and (b) per-cluster LoRA adapters merged by SVD instead of CP, holding total rank and compute budget fixed. If (a) or (b) matches TC-LoRA's +1.4%/+2.3% within noise (e.g., within 0.3%), the text-level and parameter-level mechanisms are not load-bearing; if the gains drop substantially, the mechanism is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central, load-bearing assertion is not just that TC-LoRA improves accuracy (+1.4% on Phi-3, +2.3% on Mistral-7B over SVD-based baselines), but that the improvement comes from its two named mechanisms: embedding-space sample clustering and joint CP decomposition 'disentangling' shared/task-specific factors. For that causal reading to be warranted, two conditions must hold: (1) cluster assignment by embedding similarity matters, not merely any partition of the training data; and (2) the CP joint factorization contributes beyond simply merging the per-cluster adapters with SVD. The abstract provides only aggregate accuracy, with no per-task or per-cluster breakdown, no ablation replacing embedding clustering with random partitions, and no comparison of CP merging against SVD merging of the same per-cluster LoRAs. The supplied full text is encoding-corrupted to the point that equations, tables, and ablations cannot be checked; no limitation statement or ablation is readable. If a random cluster assignment, or an SVD merge of per-cluster adapters, reproduces the reported gains within noise, then the 'input-format similarity' and 'disentanglement' narrative is not load-bearing—the gains could come from training separate adapters on smaller data subsets (implicit regularization) or from the increased total parameter budget. This is not an internal inconsistency; it is a gap between the stated mechanism and the evidence currently available.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TC-LoRA, a method for merging LoRA adapters in multi-task settings by addressing interference at two levels. At the text level, training samples are clustered in embedding space and a specialized LoRA adapter is trained per cluster. At the parameter level, a joint Canonical Polyadic (CP) decomposition is applied to the per-cluster adapters to separate shared and task-specific factors. The abstract reports +1.4% accuracy on Phi-3 and +2.3% on Mistral-7B over strong SVD-based baselines on out-of-domain zero-shot and skill-composition tasks. The submitted full text is encoding-corrupted and unreadable, so the method details, equations, tables, and ablations cannot be inspected.","tokens_in":9156,"tokens_out":6443,"duration_ms":72346,"significance":"If the reported gains are real and attributable to the two named mechanisms, this is a modest but useful contribution to parameter-efficient multi-task adaptation, connecting embedding-space data similarity with tensor-factorization-based merging. The CP decomposition approach to LoRA merging is a reasonable design point and may offer computational or storage advantages. However, the evidence as presented is insufficient: single-point aggregate improvements over unnamed baselines do not establish that embedding clustering and CP disentanglement matter. The paper appears to be an empirical recipe with no visible machine-checked proofs or reproducible code, so its significance is conditional on a full, readable experimental validation.","major_comments":[{"comment":"The supplied full text is unreadable: it consists of garbled/mojibake characters, so equations, algorithm boxes, tables, ablation results, and any limitation statements cannot be verified. This is a load-bearing defect because the paper's central claim is empirical. The authors must provide a properly encoded manuscript and explicitly identify the sections/tables that support the +1.4% and +2.3% accuracy claims.","section":"Full text (entire manuscript)"},{"comment":"The experimental claim is under-specified: it reports single-point aggregate accuracy deltas over unnamed 'strong SVD-based baselines,' with no error bars, number of seeds, significance tests, per-task results, task/dataset lists, or hyperparameters. At 1–2 percentage points, these gains are within typical run-to-run variance on zero-shot benchmarks. The grammatical fragment ('Extensive experiments ... including reasoning, question answering, and coding.') and duplicated '+2.3%' further obscure the evaluation.","section":"Abstract (experimental claim)"},{"comment":"The paper attributes the improvement to two mechanisms: embedding-space clustering and joint CP decomposition. No ablation is visible (or, due to corruption, locatable) that replaces embedding clustering with random partitions, or that uses SVD merging on the same per-cluster adapters in place of CP merging. Without these ablations, the mechanism claim is not supported; the gains could come from training separate adapters on smaller data subsets (implicit regularization) or from an increased total parameter budget. Please add ablations and a sensitivity analysis over cluster count and CP rank.","section":"Abstract (mechanism attribution)"}],"minor_comments":[{"comment":"The final sentence is a grammatical fragment; complete it and state the benchmarks, metrics, and baselines used.","section":"Abstract"},{"comment":"Duplicate '+2.3%' appears after 'Mistral-7B'; typo.","section":"Abstract"},{"comment":"'targeting to address' is awkward; consider 'addresses task interference'.","section":"Abstract"},{"comment":"Section numbering, references, and any limitation statements cannot be inspected due to the encoding corruption; ensure the final submission has a readable PDF with complete references and a limitations discussion.","section":"Full text"}],"recommendation":"major_revision","confidential_remarks":"The submitted text is unreadable, which prevents any meaningful technical review. I am treating this as the manuscript as received, per the review instructions. If the editorial office has a clean PDF, a fresh review would be needed. The paper's topic fits model merging and parameter-efficient adaptation venues; novelty relative to SVD-based merging should be checked once a readable version is available."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a sensible incremental recipe for merging multi-task LoRAs — cluster training samples by embedding similarity, train per-cluster adapters, then joint CP-decompose them — with modest reported gains over SVD-based merging. The new bit is the combination of text-level clustering and parameter-level tensor factorization. That is worth a look. But I cannot verify anything beyond the abstract: the supplied full text is encoding-garbled, with equations and tables unreadable, so my assessment is provisional.\n\nWhat the paper does well: it targets a real problem (interference when merging adapters), the two-level framing is clearly stated, and the ingredients are standard enough that the method is likely implementable. The reported +1.4/+2.3 points on Phi-3/Mistral are modest but in line with what merging methods usually deliver.\n\nThe soft spots are exactly what you'd expect from the abstract alone. The headline numbers are aggregate; no per-task breakdown, error bars, or significance tests appear in the abstract. More importantly, the causal story — that embedding clustering and CP disentanglement each contribute — is not supported by the abstract. Without ablations replacing embedding clustering with random partitions, or comparing CP merging against plain SVD merging of the same per-cluster adapters, the gains could come from training on smaller data subsets or an increased parameter budget. These ablations may exist in the full paper; I can't tell because the text is corrupted. That's a real gap in our ability to assess it, not necessarily a flaw in the paper. Also, the abstract has a grammatical fragment and a duplicated '+2.3%', which doesn't help.\n\nIf the full paper includes the ablations, this is a solid workshop-or-conference-level contribution. If not, the mechanism claims are overreach. I'd still send it to peer review — the method is plausible and the area is active — but I'd ask reviewers to insist on ablations and error bars before acceptance. For a reading group, it's a maybe: useful for people working on PEFT and model merging, less so for a general audience.","headline":"A plausible incremental method for LoRA merging, but the supplied full text is corrupted, so the evidence is currently unverifiable; the abstract's aggregate gains do not establish the mechanism.","tokens_in":9789,"tokens_out":2074,"would_cite":false,"duration_ms":25093,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TC-LoRA reduces cross-task interference in LoRA merging by clustering training samples in embedding space and jointly CP-decomposing the per-cluster adapters.","keywords":["LoRA merging","task interference","CP decomposition","embedding clustering","multi-task fine-tuning","low-rank adapters","zero-shot transfer","skill composition"],"falsifier":"Replace the embedding-space clustering with random grouping or label-based grouping while keeping the CP decomposition fixed; if out-of-domain zero-shot accuracy does not drop below the reported TC-LoRA results, then the text-level clustering is not carrying the claimed benefit.","tokens_in":8743,"feed_emoji":"🧩","tokens_out":2589,"duration_ms":29179,"temperature":0.7,"pith_summary":"This paper claims that task interference when merging multiple LoRA adapters into one large language model can be reduced by acting at two levels: grouping training samples by embedding-space similarity and training a dedicated adapter per cluster, then jointly decomposing all adapters with a canonical polyadic factorization that separates shared and task-specific factors. If this works, multi-task LoRA libraries can be merged without each task dragging down the others' performance. The reported evidence is that TC-LoRA beats SVD-based merging baselines by +1.4% accuracy on Phi-3 and +2.3% on Mistral-7B across out-of-domain zero-shot and skill-composition tasks.","feed_headline":"LoRA merging that clusters first beats SVD by up to 2.3%","feed_subtitle":"Per-embedding-cluster adapters plus joint tensor factorization cut cross-task interference in LLM merging.","key_machinery":"The central object is the joint Canonical Polyadic (CP) decomposition applied to the set of per-cluster LoRA adapters, combined with embedding-space clustering of training samples. The clustering defines the adapter library; the CP decomposition factorizes the stacked adapters into shared and task-specific rank-one components, which is what is claimed to disentangle task knowledge.","core_discovery":"The central claim is that cross-task interference in LoRA merging is addressable at both the text level and the parameter level. At the text level, clustering training samples by embedding similarity yields specialized per-cluster adapters that do not fight each other when merged. At the parameter level, a joint Canonical Polyadic (CP) decomposition of these adapters expresses them as a sum of rank-1 factor tensors, allowing shared factors to be reused while task-specific factors remain separate. The paper argues that this two-level design preserves essential task knowledge and reduces interference, and supports it with experiments showing accuracy gains over SVD-based merging baselines on P","pith_inferences":["The embedding-clustering assumption implies that interference is correlated with input-format similarity; a testable prediction is that clusters defined by output format or label distribution would perform worse.","If the CP factorization truly separates task-specific and shared factors, the shared factors should transfer to unseen tasks with minimal fine-tuning, which the paper does not test.","The reported gains might be larger when tasks are more heterogeneous, since interference grows with task diversity; the current experiments sample only reasoning, question answering, and coding."],"forward_implications":["Merging LoRA adapters for many tasks can be done without retraining the base model, making multi-task adaptation cheaper and more modular.","Embedding-space clustering provides a principled way to decide how many adapters to train, rather than one per task.","Joint factorization could allow selective pruning of task-specific factors to control which skills survive a merge.","The approach could extend to other parameter-efficient fine-tuning methods that produce low-rank delta matrices."],"supporting_citations":[],"fun_headline_variants":["Two-level LoRA merging: cluster then decompose, +2.3%","Clustered LoRA adapters + CP decomposition beat SVD merge","Stop LoRA interference: cluster samples, factor adapters","LoRA merge: text clusters + tensor factors beat SVD by 2.3%","Cluster-specialized LoRAs cut merge interference"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that training samples that interfere with each other during merging are exactly those close in embedding space, so that per-cluster specialization plus joint factorization can separate task knowledge; if embedding similarity does not track interference, the text-level step would not help.","fun_headline_variants_meta":{"raw":{"variants":["Two-level LoRA merging: cluster then decompose, +2.3%","Clustered LoRA adapters + CP decomposition beat SVD merge","Stop LoRA interference: cluster samples, factor adapters","LoRA merge: text clusters + tensor factors beat SVD by 2.3%","Cluster-specialized LoRAs cut merge interference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000447,"raw_usage":{"total_tokens":2102,"prompt_tokens":760,"completion_tokens":1342,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":1248}},"tokens_in":504,"tokens_out":1342,"duration_ms":13171,"temperature":1.0,"reasoning_tokens":1248,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T00:58:01.469426+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the embedding-space clustering with random grouping or label-based grouping while keeping the CP decomposition fixed; if out-of-domain zero-shot accuracy does not drop below the reported TC-LoRA results, then the text-level clustering is not carrying the claimed benefit.","supporting_citations":[],"review_version":1}