{"id":"9e850f0f-75d3-480e-b17e-d9064af56d9d","arxiv_id":"2505.11204","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Randomized layer shuffling and column-wise sign flips make interfering task deltas nearly orthogonal, allowing one compressed superset to retrieve near-fine-tuned accuracy for many tasks.","lead":"This paper proposes RanDeS, a method that randomizes the parameter differences, called deltas, of fine-tuned models before merging them, so that one compressed model can serve many tasks with little accuracy loss. The method stores only the base model plus one merged delta, regardless of how many task models are added.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Orthogonalization barely reduces the interference norm, so RanDeS's benefit must come from interference direction, not the Eq. (3) mechanism.","rationale":"The reader's weakest assumption correctly targets the l2-norm proxy. I agree and sharpen it: the proxy is not merely unproven; the numbers in the paper make it quantitatively implausible. With original pairwise cosines near 0.02, the norm reduction from perfect decorrelation is ~6–9%, yet accuracy improves by ~20 absolute points. So the causal story in Eq. (3) cannot be the main driver; the empirical benefit must come from the directions of the transformed interference. This is a genuine flaw in the paper's central explanation, but it does not undermine the measured accuracy or the reproducibility of the method. The experiments are broad (CLIP-ViT-B/32, CLIP-ViT-L/14, Flan-T5, GPT-2, LoRA), run three times, and the method is fully seed-defined, so the empirical core is trustworthy. The appropriate fix is to reframe the mechanism and add the norm-matched control or an equivalent analysis of interference directions. The reader already returned CONDITIONAL with instructions to fix Eq. (3) and temper 'self-cancellation'; this concern is consistent with that verdict, so I leave it unchanged.","tokens_in":17386,"tokens_out":21201,"duration_ms":234002,"concrete_test":"Implement a diagnostic control on the CLIP-ViT-B/32 8-task benchmark. For each task i, evaluate Θ_i^ctrl(λ,c_i) = Θ0 + λ∆i + λ c_i Σ_{j≠i}∆j, choosing c_i per task so that ||c_i Σ_{j≠i}∆j||2 equals the actual RanDeS-SRSF interference norm ||Σ_{j≠i} O_i^{-1}O_j∆j||2 for the same λ and seed. This keeps the un-randomized directions while matching the norm exactly. Sweep λ over {0.1,...,1.0} and report average accuracy. If the control reaches ~89.9%, the norm-reduction story explains RanDeS; if it stays near 69.8%, the benefit comes from the randomization of interference directions, and Eq. (3) must be replaced by a directional/spectral explanation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing concern is that the paper's stated mechanism—random orthogonal transformations reduce the l2 norm of the interference term (Eq. 3), and this norm reduction preserves task accuracy—cannot quantitatively explain the reported gains. In the CLIP-ViT-B/32 setup (T=8), the measured pairwise cosine between interfering deltas is already ~0.02–0.03 (Figure 3). For a fixed task i, the task-arithmetic interference norm satisfies ||Σ_{j≠i}∆j||^2 = 7||∆||^2 + 2Σ_{l<j≠i} cos(∆l,∆j)||∆l||||∆j|| ≈ (7.8–8.3)||∆||^2, so its norm is ≈2.8–2.9||∆||. Perfect decorrelation would give sqrt(7)||∆|| ≈ 2.65||∆||, a reduction of only 6–9%. Table 1, however, shows average accuracy jumping from 69.8% (task arithmetic) to 89.9% (RanDeS-SRSF). A <10% change in the norm of a perturbation that is already ~2.8× the signal norm is unlikely to yield a ~20-point accuracy gain. Furthermore, 'self-cancellation' is not what orthogonal vectors do: orthogonal vectors add in quadrature, so the interference norm grows as sqrt(T-1) even at zero cosine. The gain must therefore come from the directions of the randomized interference—its alignment or lack thereof with task-relevant parameter directions—not from its l2 norm. Figure 7 is only correlational, and Appendix C.2 explicitly concedes that 'the method of achieving orthogonality matters beyond decorrelation levels.' If direction, not norm, is the operative quantity, the design principle 'reduce pairwise cosine similarity' may be optimizing the wrong objective, and the method's success rests on uncharacterized robustness of these networks to isotropic weight perturbations. This does not invalidate the empirical accuracy claims, but it breaks the central theoretical narrative connecting Eq. (3) to the results.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reformulates multi-model compression as storing a superposition of randomly transformed task deltas and retrieving individual deltas via inverse transforms. It proposes two orthogonal-transformation implementations (layer shuffling and column-wise random sign flips) and their combination. Experiments on CLIP-ViT-B/32 (8 tasks), CLIP-ViT-L/14 (up to 20 tasks), Flan-T5-base (full and LoRA), and GPT-2 show near-fine-tuned accuracy at roughly constant memory, with memory proportional to one base model plus one delta superposition. The theoretical motivation is that random orthogonal transforms decorrelate interfering deltas, reducing the l2 norm of the interference term in Eq. (3).","tokens_in":17709,"tokens_out":9409,"duration_ms":88756,"significance":"The method is simple, training-free, and seed-based, with a public codebase; the empirical coverage is broad (vision, language, PEFT, and a 20-model large-scale setting). If the mechanism were exactly as claimed, this would be a strong contribution to multi-model serving. However, the stated l2-norm mechanism does not quantitatively account for the reported gains, and the paper itself concedes in Appendix C.2 that 'the method of achieving orthogonality matters beyond decorrelation levels.' The results are thus significant but the theoretical framing needs substantial revision.","major_comments":[{"comment":"The central claim that random orthogonal transformations reduce interference by lowering the l2 norm of the interference term is not supported by the reported numbers. For the T=8 CLIP-ViT-B/32 setup, Figure 3 reports pairwise cosine similarity among interfering deltas of approximately 0.02-0.03 before RanDeS and approximately 0.0001 after RanDeS-SRSF. Assuming comparable delta norms, Eq. (3) gives an interference norm of sqrt(7 + 42*0.025) ||Δ|| ≈ 2.84||Δ|| for task arithmetic and sqrt(7 + 42*0.0001)||Δ|| ≈ 2.65||Δ|| after RanDeS-SRSF, a reduction of about 6-7%. Yet Table 1 shows average accuracy rising from 69.8% (task arithmetic) to 89.9% (RanDeS-SRSF), a ~20-point gain that a 6-7% norm reduction is unlikely to explain. Moreover, orthogonal vectors do not 'self-cancel'; they add in quadrature, so the interference norm grows as sqrt(T-1) even at zero cosine. The gain likely comes from the direction of the randomized interference relative to task-relevant parameter directions, not from its magnitude. Please revise the theoretical justification or provide controlled experiments that isolate the l2-norm component (e.g., rescaling the interference term to match norms across conditions).","section":"Section 3, Eq. (3); Figure 3; Table 1"},{"comment":"The sentence 'the magnitude of interference is negatively correlated with the cosine similarity between deltas' misstates the direction of the relationship. The cross-term in Eq. (3) is +2 Σ cos(Δ_l, Δ_j) ||Δ_l|| ||Δ_j||, so larger cosine similarity increases the interference norm. The correct statement is that the interference magnitude is positively correlated with cosine similarity; the design goal is therefore to reduce positive cosines (or push them negative). This error should be corrected because it appears in the paper's core motivation.","section":"Section 3, sentence after Eq. (3)"},{"comment":"The paper's own analysis undermines the exclusive role of cosine-similarity reduction. Appendix C.2 reports that for GTSRB, RanDeS-S outperforms RanDeS-Shift despite having higher cosine similarity, 'indicating the method of achieving orthogonality matters beyond decorrelation levels.' Similarly, Figure 7 shows only correlational evidence (albeit with strong negative correlations) between cosine and accuracy. These observations are consistent with the concern in comment 1 that the mechanism is directional rather than norm-based. The paper should either advance a coherent directional theory or explicitly reframe the contribution as empirical, with the cosine-reduction claim demoted to a heuristic.","section":"Appendix C.2; Section 5.3"}],"minor_comments":[{"comment":"The summation bound uses an undefined symbol n; it should be T (or T-1 with the exclusion of i properly indicated).","section":"Eq. (3)"},{"comment":"The term 'self-cancellation' is misleading when applied to orthogonal vectors, which add in quadrature rather than cancel; consider using 'cross-term cancellation' or 'near-orthogonality' instead.","section":"Abstract and Section 6"},{"comment":"The RBD-vs-RD comparison is confounded because RD changes both orthogonality and the norm of each delta (entries are not ±1); please add a control with column-normalized random matrices to attribute the degradation to loss of orthogonality rather than magnitude distortion.","section":"Section 5.5, Figure 5(a)"},{"comment":"The main results are averages over three runs but no standard deviations are reported; please include error bars or state explicitly the actual variance magnitude instead of only noting that variances smaller than 0.1% are omitted.","section":"Tables 1 and 2"},{"comment":"The phrase 'three out of eight' is ambiguous; please specify which tasks are shown or state that they are representative.","section":"Figure 3 caption"},{"comment":"The column label 'Bits(Gb)' is confusing; use 'Memory (Gb)' or 'Size (Gb)' instead.","section":"Table 1 header"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a simple and apparently effective empirical method, and the public code is a plus. My main concern is the gap between the stated l2-norm mechanism and the magnitude of the reported gains; the authors need to be pressed on this in revision. The sign error in Eq. (3) should also be fixed. I do not see grounds for rejection: the empirical results are reproducible and the method is practical, but the theoretical framing as currently written overclaims the role of norm reduction. The journal scope fits; no citation issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: RanDeS works remarkably well across vision and language benchmarks, and the empirical story is solid. The paper's theoretical mechanism, however, is wrong as written, and the authors half-admit it in an appendix. The Eq. (3) sign error is a clear slip: interference magnitude grows with cosine similarity, not the reverse. More importantly, the stress-test's arithmetic is right: for 8 CLIP models with pairwise cosine ~0.02, perfect decorrelation would only reduce the interference norm from about 2.84||Δ|| to 2.65||Δ|| — a 7% change. The accuracy jump from 70% to 90% can't plausibly come from that. The real explanation must be about the direction of randomized interference, not its l2 norm. The paper even says in Appendix C.2 that 'the method of achieving orthogonality matters beyond decorrelation levels,' which undercuts its own design principle. That's a load-bearing flaw in the narrative, not a typo.\n\nWhat's genuinely new: layer shuffling as a decorrelation mechanism for delta interference, the combination with Cheung et al.'s sign flips, and the compress-and-retrieve framing. The empirical scope is impressive: up to 20 CLIP ViT-L/14 models, Flan-T5 full and LoRA, GPT-2, with near-fine-tuned accuracy at constant memory. The ablations on skip rate, context-matrix design, and layer selection are honest and informative. The method is simple enough to be reproducible, and there's a public repo.\n\nSoft spots beyond the theory: 'self-cancellation' is the wrong term — orthogonal vectors add in quadrature; they don't cancel. The 'zero additional memory' claim only holds after the initial 2x base memory, which is fine but should be phrased that way. Variance reporting is patchy (some tables show ±, most don't). A code commit hash would help.\n\nBottom line: the empirical contribution is real and the paper deserves peer review. The revision should rewrite Section 3 to say the interference direction is what matters, and remove the l2-norm argument. I'd referee it and recommend acceptance after that fix.","headline":"RanDeS is a practical multi-model compression method with strong empirical results, but the paper's stated l2-norm interference mechanism is wrong and the appendix half-admits it.","tokens_in":18346,"tokens_out":3440,"would_cite":true,"duration_ms":33670,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One compressed weight stack serves 20 fine-tuned models near full accuracy.","keywords":["model merging","delta superposition","multi-model serving","random orthogonal transformations","layer shuffling","column-wise sign flips","task interference","model compression"],"falsifier":"Take one fine-tuned model, add a second 'task' delta that is orthogonal to the first in cosine similarity but concentrated entirely on the parameters most critical to the first task's output layer, and merge them with RanDeS; if accuracy on the first task drops far more than the near-zero cosine similarity predicts, the norm proxy at the heart of the argument fails.","tokens_in":17158,"feed_emoji":"🧠","tokens_out":6225,"duration_ms":58309,"temperature":0.7,"pith_summary":"RanDeS tackles the memory cost of serving many models that share a pretrained base. Its claim is that the interference that ruins naive model merging can be suppressed by randomly transforming each model's parameter delta before summing them, then undoing that random transform when a specific task is requested. The random transforms—layer permutations and column-wise sign flips—make the interfering deltas nearly orthogonal, so their sum nearly cancels. On 8 CLIP-ViT-B/32 models, RanDeS-SRSF recovers 99.6% of fine-tuned average accuracy while using 1.58x the base model's memory, and on 20 CLIP-ViT-L/14 models it reaches 99.3% at a constant 2.87 GB. If right, this lets a serving system hold any number of fine-tunes in roughly one base model's footprint.","feed_headline":"One compressed weight stack serves 20 fine-tuned models near full accuracy","feed_subtitle":"Random layer shuffles and sign flips cancel task interference, keeping memory constant as models are added.","key_machinery":"The load-bearing object is the per-model orthogonal transformation $O_i$ applied to each delta before summation, with retrieval applying $O_i^{-1}$. In full generality it is a rotation drawn from the orthogonal group, but the paper instantiates it cheaply as a permutation of same-shaped layers inside each delta and a random diagonal matrix with $\\pm 1$ entries applied column-wise per layer. What it does is convert the interference term $\\lambda\\sum_{j\\neq i} O_i^{-1}O_j\\Delta_j$ into a sum of near-orthogonal vectors whose $\\ell^2$ norm, by Eq. (3), depends on pairwise cosine similarities; driving those cosines toward zero is what preserves task accuracy.","core_discovery":"Starting from task arithmetic, the paper rewrites model merging as compress-and-retrieve: the merged weights are the base weights plus a sum of deltas, and retrieving task $i$ should give $(1-\\lambda)\\Theta_0 + \\lambda\\Theta_i$ plus an interference term $\\lambda\\sum_{j\\neq i} O_i^{-1}O_j\\Delta_j$. The central discovery is that this interference term shrinks dramatically when each delta is first rotated by a random orthogonal transformation $O_i$, because in high dimension random rotations make the interfering deltas nearly orthogonal. The paper implements the rotations as layer shuffling within repetitive layer groups and as random column-wise sign flips per layer, both invertible and fully specified by random seeds. With both operations combined, the average pairwise cosine similarity among interfering deltas drops by orders of magnitude, and measured accuracy on vision, text-generation, and text-classification benchmarks lands near the individually fine-tuned models. The same procedure also compresses LoRA adapters.","pith_inferences":["The same decorrelation mechanism could apply to any setting where multiple learned updates to one base must be stored and selectively applied, such as continual learning checkpoints, federated round updates, or per-client adapters, since nothing in the argument depends on the task modality.","The ablation result that random orthogonal sign flips beat random non-orthogonal diagonal matrices suggests that orthogonality, not merely randomness, is the active ingredient; a cheap orthogonal transform such as a Walsh-Hadamard rotation is a natural untested extension.","The strong negative correlation between cosine similarity and accuracy in the paper's Figure 7 implies a possible operating rule: manipulate only enough layers to push interfering-delta cosine below a task-specific threshold, trading a little accuracy for less compute and memory."],"forward_implications":["A serving system can add a new fine-tuned model at effectively zero extra storage, because each transformation is stored as a seed rather than as weights.","Holding accuracy near the fine-tuned upper bound no longer requires per-task weight copies: 8 ViT-B/32 models fit in 1.58x base memory and 20 ViT-L/14 models in a constant 2.87 GB.","Models can be hot-swapped in or out of the pool without retraining or recomputing a router or task masks.","The method transfers to parameter-efficient fine-tuning: LoRA adapters on Flan-T5-base reach 99.8% normalized accuracy at 1.20 GB.","Because the transformations are random and data-agnostic, the compression ratio grows with the number of models rather than degrading."],"supporting_citations":[{"why":"Supplies task arithmetic and the task-vector decomposition that RanDeS reformulates into compress-and-retrieve.","marker":"[Ilharco et al., 2022]"},{"why":"Documents the task-interference failure in merging that motivates the method.","marker":"[Yadav et al., 2024]"},{"why":"Introduces the random binary diagonal context-matrix idea that RanDeS-RSF adapts as column-wise sign flips.","marker":"[Cheung et al., 2019]"},{"why":"Provides the SMILE low-rank routing baseline and the Flan-T5/GPT-2 fine-tuned models used in comparisons.","marker":"[Tang et al., 2024b]"},{"why":"Provides the TALL-masks baseline and the CLIP ViT-L/14 fine-tuned models used in the scalability study.","marker":"[Wang et al., 2024]"},{"why":"Provides the CLIP models and zero-shot classification protocol for the vision benchmarks.","marker":"[Radford et al., 2021]"},{"why":"Provides the Flan-T5 models and GLUE text-generation protocol for the language benchmarks.","marker":"[Longpre et al., 2023]"}],"fun_headline_variants":["Random shuffles and sign flips merge models with zero memory overhead","One weight stack, many models: random rotations cancel task clash","Deltas decorrelated: near-full accuracy for 20 merged models","No memory cost to add models: just shuffle layers and flip signs","RanDeS: random rotations shrink task interference in model merging"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the size of the interference term, measured through pairwise cosine similarity among deltas, is what determines how much task accuracy is lost; if the direction or fine-grained structure of the interference matters more than its size, the decorrelation benefit could disappear.","fun_headline_variants_meta":{"raw":{"variants":["Random shuffles and sign flips merge models with zero memory overhead","One weight stack, many models: random rotations cancel task clash","Deltas decorrelated: near-full accuracy for 20 merged models","No memory cost to add models: just shuffle layers and flip signs","RanDeS: random rotations shrink task interference in model merging"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000376,"raw_usage":{"total_tokens":1972,"prompt_tokens":883,"completion_tokens":1089,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":499,"completion_tokens_details":{"reasoning_tokens":999}},"tokens_in":499,"tokens_out":1089,"duration_ms":10341,"temperature":1.0,"reasoning_tokens":999,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:55:25.156087+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one fine-tuned model, add a second 'task' delta that is orthogonal to the first in cosine similarity but concentrated entirely on the parameters most critical to the first task's output layer, and merge them with RanDeS; if accuracy on the first task drops far more than the near-zero cosine similarity predicts, the norm proxy at the heart of the argument fails.","supporting_citations":[],"review_version":1}