{"id":"5310fbf1-5581-4894-a68b-fe66d08058c6","arxiv_id":"2505.05143","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Permuting a lottery ticket mask to align with a new random initialization's optimization basin improves sparse training accuracy and solution diversity relative to using the unpermuted mask.","lead":"This paper tests whether lottery ticket masks can be reused with new random weight initializations if the masks are first permuted to match the new model's optimization basin. On CIFAR and ImageNet benchmarks, the permuted masks train to higher accuracy than unpermuted masks, and produce more diverse ensemble members.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No random-permutation control: the reported gains could come from changing the mask rather than from basin alignment, leaving the central causal hypothesis untested.","rationale":"I read the paper as making two claims: (1) permuting an LTH mask by a dense-model permutation improves sparse training from a new init (empirical), and (2) this supports the basin-misalignment hypothesis. Claim 1 is well supported across datasets, models, widths, sparsities, and rewind points. Claim 2 is underdetermined because no control isolates the permutation's alignment from the effect of merely changing the mask. The reader's weakest assumption about pi transfer is reasonable and partially mitigated by Appendix B, but the random-permutation control is a more direct test of the causal story. I do not move the verdict: CONDITIONAL remains appropriate; the empirical contribution is credible, but the central interpretation should be presented as a hypothesis pending this control. Code availability and ImageNet error bars are secondary issues already noted by the reader.","tokens_in":25708,"tokens_out":5352,"duration_ms":59531,"concrete_test":"Reproduce the ResNet20 x {4} / CIFAR-10 setting at sparsity 0.90, rewind k=25, and train from the same B init with the following masks: naive mA, pi(mA), and 5 random permutations rho_1..rho_5 of mA (same layerwise sparsity, same seeds). Compare mean test accuracy with error bars. If random rho's accuracy is statistically indistinguishable from pi's and significantly above naive, the basin-alignment explanation is falsified. If rho is close to naive and pi beats both, the alignment-specific interpretation survives this control.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central causal claim is that misalignment of basins explains why LTH masks fail to transfer, and that permutation pi from dense activation matching realigns the mask to model B's basin. The evidence is permuted > naive across many settings. But in every comparison, the only change is mask mA versus pi(mA); there is no random-permutation baseline. A random permutation rho(mA) preserves layerwise sparsity and the permutation-group structure of the mask while destroying any alignment to B's basin. If rho(mA) also improves over naive, then the gain is not evidence for basin alignment; it is evidence that a different mask trains better from a new init. This is not an internal inconsistency; it is a missing control for the mechanism. The permutation pi is obtained from dense converged models and then applied to a sparse mask at rewind epoch k; Appendix B partially addresses the transfer of pi to early training epochs (Table 15), but it still never compares against random rho. The causal interpretation therefore rests on a single, uncontrolled comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes that lottery ticket masks fail to transfer to new random initializations because the mask is not aligned with the optimization basin of the new initialization. The authors compute a permutation pi by activation matching between two dense models trained from different initializations, apply pi to the LTH mask, and train from the new initialization with the permuted mask (permuted). They compare this against training with the original mask (naive) and against the LTH baseline, across CIFAR-10, CIFAR-100, and ImageNet, for several sparsities, rewind points, and model widths. They report that permuted consistently outperforms naive, with the gap increasing with sparsity and width, and that ensembles of permuted models are more functionally diverse. They also show, using REPAIR, that the dense solution and the LTH solution lie in the same linearly connected mode.","tokens_in":25896,"tokens_out":6853,"duration_ms":60526,"significance":"The empirical pattern is broad and internally consistent: three-run error bars on CIFAR, multiple sparsities and rewind points, four width multipliers, and a public codebase. The diversity analysis is a useful additional contribution. If the mechanism were established, the work would be a meaningful step toward sparse training from random initializations and toward connecting weight-space symmetry to lottery-ticket transfer. However, the central causal interpretation is not yet supported because every comparison differs in the mask (mA versus pi(mA)) and no random-permutation control is included. The paper also acknowledges that Sharma et al. (2024) already demonstrated mask reuse modulo permutations, so the novel element is the basin-alignment mechanism and its empirical evaluation; the novelty boundary should be sharpened.","major_comments":[{"comment":"The paper's central claim is that permuting the LTH mask with the activation-matching permutation pi improves sparse training from a new random initialization relative to using the original mask. In every comparison, the two conditions differ only in the mask: mA versus pi(mA). There is no control in which the mask is transformed by a random permutation rho(mA) that preserves layerwise sparsity but is not aligned to the new initialization's basin. If rho(mA) also outperforms naive, then the improvement is not evidence for basin alignment but simply evidence that a different mask trains better from a new initialization. Please add at least one random-permutation baseline (ideally several) for a representative subset of the main CIFAR-10 ResNet20 settings, and report whether permuted > random > naive, or permuted is approximately equal to random. Without this control, the causal hypothesis that 'misalignment of basins is the reason why LTH masks do not generalize' remains untested.","section":"Section 3, Tables 5-14"},{"comment":"The ImageNet results in Table 10 are reported as single numbers with no error bars or number of seeds, in contrast to the three-run averages used for the CIFAR experiments. The VGG11 results also omit the high-sparsity naive baselines, described in Section 4.1 as unstable. Because the abstract claims that the method works on ImageNet and on VGG11, this asymmetry weakens the generality claim. Please either provide multiple-seed error bars for the ImageNet comparisons or explicitly state that these are single-run observations, and clarify the status of the omitted VGG11 high-sparsity runs.","section":"Section 4.1 (ResNet50/ImageNet), Table 10"},{"comment":"The transfer of pi computed on dense models to the sparse masked training trajectory is a load-bearing assumption. Appendix B shows that pi computed at earlier dense epochs (t=5,20,50,100) gives similar downstream accuracy for the permuted solution, but it does not test whether pi actually aligns the sparse basin at the rewind epoch. The only evidence connecting pi to the sparse basin is the accuracy gain, which is the same evidence used for the central claim and is therefore subject to the random-permutation concern in Major Comment 1. A direct test would be to compute loss barriers between the rewound sparse initialization with the permuted mask and the final sparse solution, and to compare with the barrier under a random permutation of the mask.","section":"Appendix B (Early Matching)"}],"minor_comments":[{"comment":"The caption uses 'Jenson-Shannon'; this should be 'Jensen-Shannon'.","section":"Table 1 caption"},{"comment":"The caption contains the typo 'peroforming'; it should be 'performing'.","section":"Figure 11 caption"},{"comment":"The sentence 'The dashed (- -) line shows the dense model accuracy' is repeated verbatim in many captions; please vary or reduce the repetition.","section":"Figure captions (Figures 5-10)"},{"comment":"The permuted ensemble varies both the permutation and the random initialization, while the LTH ensemble fixes both and varies only the data order; this difference should be acknowledged when interpreting the diversity comparison, since the permuted ensemble confounds mask diversity with initialization diversity.","section":"Section 4.2"},{"comment":"Appendix D states that the alignment process 'adds a small computational overhead,' but the method requires training two dense models to convergence before the permutation is computed; the phrasing should be revised to acknowledge that the total compute of the permuted solution is substantially higher than LTH or naive.","section":"Appendix D"},{"comment":"The text says Sharma et al. (2024) 'showed that for certain neural networks, sparse mask obtained via weight rewinding can be reused modulo permutations without hurting the test performance'; the Introduction should state more explicitly how the present contribution differs from that result, beyond the specific activation-matching mechanism and the broader empirical study.","section":"Section 2 (Background & Related Work)"}],"recommendation":"major_revision","confidential_remarks":"The proposed contribution is close to the result already reported by Sharma et al. (2024), one of whose authors is a co-author of this manuscript. The paper's own literature review states that Sharma et al. 'showed that for certain neural networks, sparse mask obtained via weight rewinding can be reused modulo permutations without hurting the test performance.' The present paper's incremental contribution is the basin-alignment interpretation and the broader empirical study. The editor may wish to ask the authors to clarify the novelty boundary. Also, the missing random-permutation control is a serious gap; if it cannot be provided, the paper's main mechanistic claim should be downgraded to a descriptive observation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the central empirical claim holds up: permuting a lottery-ticket mask with a permutation found through dense activation matching consistently beats the unpermuted mask when sparse-training from a new random init. That holds across CIFAR-10/100, ImageNet, ResNet20/VGG11/ResNet50, and the gains grow with sparsity and width. Second, the explanation for why—basin misalignment—is not actually pinned down. The design compares mA with pi(mA) but never with a random permutation rho(mA) of the same mask. Without that control, the improvement could come from any mask change, not specifically from aligning to B's basin. The loss-barrier plots show pi aligns dense models, and Appendix B shows early matching works, but none of that rules out a generic permutation effect. This is a missing control for the mechanism, not an internal inconsistency; the empirical result survives regardless.\n\nWhat is actually new: applying permutation matching specifically to IMP-derived LTH masks, and measuring the functional diversity of the resulting sparse models. Sharma et al. 2024 already had mask reuse modulo permutations for weight-rewound sparse networks; the increment over that is modest, but the LTH-mask version and the diversity/ensemble analysis are new. The paper cites Sharma et al. honestly. The REPAIR-based finding that the LTH solution and the dense solution sit in the same basin once variance collapse is corrected is a nice extension of Paul et al.\n\nThe paper earns credit for three-run error bars on CIFAR, width scaling as a check on matching quality, an early-matching appendix, and a transparent statement that this method costs more than LTH, not less. Those are the right things to do.\n\nSoft spots, in proportion. ImageNet numbers (Table 10) have no error bars; VGG11 high-sparsity naive results are omitted as unstable, which is transparent but leaves the comparison incomplete; and the code link is a placeholder. None of those are fatal. The causal claim is the real gap: a random-permutation baseline is cheap and would discriminate between “alignment helps” and “any different mask helps.”\n\nWho this is for: researchers working on sparse training or LTH transfer, and model-merging people interested in how permutation symmetries interact with masks. It deserves a serious referee. My recommendation: send it out, and ask the authors to add a random-permutation control, fix the code link, and report ImageNet error bars. If the control comes back flat, the basin-alignment story is stronger; if not, the paper still has a useful empirical result, but the interpretation needs softening.","headline":"The permuted-mask result is consistent and worth taking seriously, but the basin-alignment explanation needs a random-permutation control before it can be believed.","tokens_in":26427,"tokens_out":3660,"would_cite":true,"duration_ms":40074,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that lottery-ticket masks can be reused with new random initializations by permuting the mask to align with the new optimization basin, and supports the claim with experiments on CIFAR-10, CIFAR-100, and ImageNet.","keywords":["lottery ticket hypothesis","sparse training","permutation symmetry","activation matching","linear mode connectivity","weight rewinding","functional diversity","model merging"],"falsifier":"At the rewind epoch $k$, compute the error barrier between $\\pi(w_A^k)$ and $w_B^k$ for ResNet20 on CIFAR-10; if the barrier at $k$ stays high while the converged barrier is near zero, the permutation is not transferring to the rewind point and the accuracy gain of the permuted mask would need another explanation, and a second check should compare the permuted mask against a randomly permuted mask under identical training.","tokens_in":25502,"feed_emoji":"🎟️","tokens_out":8878,"duration_ms":74079,"temperature":0.7,"pith_summary":"The paper claims that a lottery-ticket mask fails to transfer to a new random initialization not because the mask is tied to its original initialization, but because the mask's sparse optimization basin is misaligned with the new initialization's basin. The proposed fix is to permute the mask by a permutation $\\pi$ obtained by activation-matching two dense models trained to convergence, so that the mask points into the new basin. Empirically, training a fresh initialization with the permuted mask $\\pi(m_A)$ beats training it with the unpermuted mask $m_A$ across CIFAR-10, CIFAR-100, and ImageNet on VGG11 and ResNet20/50, with the gap narrowing as model width grows. If correct, this turns lottery-ticket masks into reusable assets for sparse training from arbitrary initializations rather than one-use tickets.","feed_headline":"Permuted LTH masks generalize better on new random initializations","feed_subtitle":"Activation-matching the mask to the new initialization's basin lifts accuracy on CIFAR and ImageNet.","key_machinery":"The load-bearing object is the permutation $\\pi$ computed by activation matching: for each layer, $\\pi_l = \\arg\\max_\\pi \\langle \\pi, Z_B (Z_A)^\\top \\rangle_F$, solved as a linear assignment problem with the Hungarian algorithm, which permutes the output neurons of one layer and the input neurons of the next. Applying this $\\pi$ to the sparse mask reorients the mask into the new initialization's loss basin. Two supporting devices make the argument measurable: REPAIR, which renormalizes permuted activations to remove the variance-collapse barrier, and the loss barrier $B(\\theta_1,\\theta_2) = \\sup_{\\alpha \\in [0,1]} [ L((1-\\alpha)\\theta_1+\\alpha\\theta_2) - ((1-\\alpha)L(\\theta_1)+\\alpha L(\\theta_2))]$, which quantifies whether two models share a basin.","core_discovery":"The central claim is that weight-symmetry alignment, not the original initialization, is what makes a lottery-ticket mask effective. Given a mask $m_A$ derived from model $A$ by iterative magnitude pruning, and a fresh random initialization $w_B^0$, the authors find a permutation $\\pi$ that aligns the activations of fully trained $A$ with fully trained $B$, then use $\\pi(m_A)$ as the mask for sparse training of $B$ from a rewind checkpoint $w_B^k$. Across ResNet20 widths 1, 4, 8, 16, VGG11, and ResNet50 on ImageNet, the permuted runs consistently outperform naive runs that use $m_A$ unchanged, and the advantage widens at higher sparsity and larger width. The paper also argues that the dense solution and its LTH sparse solution lie in the same linearly connected basin once REPAIR corrects variance collapse, which differs from an earlier analysis that omitted that correction.","pith_inferences":["A cheap approximation to the alignment permutation, such as matching at an early training checkpoint as the appendix tests, could make mask transfer practical; the early-matching results suggest most of the benefit survives matching at epoch 5.","If alignment is the true bottleneck, the same mechanism should allow masks to transfer across related settings, such as between different network widths or teacher-student pairs, whenever the permutation-merged model has a low loss barrier.","The paper's own loss-barrier plots predict that a better solver for the permutation problem, replacing the greedy Hungarian matching, would shrink the remaining LTH gap on ImageNet, where the barrier after permutation is highest."],"forward_implications":["Sparse training from a new random initialization can reuse an existing lottery-ticket mask by permuting it, recovering much of the LTH accuracy without reusing the mask's original initialization.","The accuracy gap between the permuted solution and the true LTH solution shrinks as network width grows, because wider models allow more accurate permutation matching.","Ensembles of models trained with different permutations of the same mask are more functionally diverse than ensembles of LTH retrains, and the permuted ensemble reaches higher accuracy than the LTH ensemble.","The dense solution and its LTH sparse solution are linearly mode connected once variance collapse is repaired, sharpening the earlier claim that sparse masks tunnel into a separate basin."],"supporting_citations":[{"why":"Introduced the lottery ticket hypothesis and the winning-ticket mask that this work reuses.","marker":"Frankle & Carbin (2019)"},{"why":"Provided the activation-matching algorithm used to find the permutation between two dense models.","marker":"Ainsworth et al. (2023)"},{"why":"Identified variance collapse and supplied REPAIR, used to show the dense and sparse solutions are in the same basin.","marker":"Jordan et al. (2023)"},{"why":"Conjectured that SGD solutions are linearly connectable modulo permutation, the premise behind mask alignment.","marker":"Entezari et al. (2022)"},{"why":"Established linear mode connectivity and weight-rewinding for lottery tickets, grounding the rewind-point training.","marker":"Frankle et al. (2020)"},{"why":"Prior analysis of sparse-dense connectivity that this paper refines by adding REPAIR.","marker":"Paul et al. (2023)"},{"why":"Showed LTH retrains the same solution; its diversity metrics and ensemble comparison are used to evaluate permuted models.","marker":"Evci et al. (2022)"},{"why":"Showed permutations align entire training trajectories and rewound sparse networks, motivating early matching.","marker":"Sharma et al. (2024)"}],"fun_headline_variants":["Permute lottery masks to match new basins for better sparse training","Weight symmetry makes lottery ticket masks transferable across initializations","Aligned masks beat originals when sparse training from a fresh init","Permutation alignment unlocks lottery ticket reuse on new seeds","Sparse training from random init: align the mask, boost accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The permutation found by matching two fully trained dense models is assumed to transfer to the much earlier rewind epoch and to the sparse masked training trajectory, so that permuting the mask by this $\\pi$ places the sparse model inside the new initialization's basin.","fun_headline_variants_meta":{"raw":{"variants":["Permute lottery masks to match new basins for better sparse training","Weight symmetry makes lottery ticket masks transferable across initializations","Aligned masks beat originals when sparse training from a fresh init","Permutation alignment unlocks lottery ticket reuse on new seeds","Sparse training from random init: align the mask, boost accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000526,"raw_usage":{"total_tokens":2533,"prompt_tokens":931,"completion_tokens":1602,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":1517}},"tokens_in":547,"tokens_out":1602,"duration_ms":10642,"temperature":1.0,"reasoning_tokens":1517,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:11:09.746112+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"At the rewind epoch $k$, compute the error barrier between $\\pi(w_A^k)$ and $w_B^k$ for ResNet20 on CIFAR-10; if the barrier at $k$ stays high while the converged barrier is near zero, the permutation is not transferring to the rewind point and the accuracy gain of the permuted mask would need another explanation, and a second check should compare the permuted mask against a randomly permuted mask under identical training.","supporting_citations":[{"cited_title":"and Carbin, M","cited_arxiv_id":null,"evidence_quote":"Introduced the lottery ticket hypothesis and the winning-ticket mask that this work reuses."},{"cited_title":"K., Hayase, J., and Srinivasa, S","cited_arxiv_id":null,"evidence_quote":"Provided the activation-matching algorithm used to find the permutation between two dense models."},{"cited_title":"REPAIR: renormalizing permuted activations for interpolation repair","cited_arxiv_id":null,"evidence_quote":"Identified variance collapse and supplied REPAIR, used to show the dense and sparse solutions are in the same basin."},{"cited_title":"The role of permutation invariance in linear mode connectivity of neural networks","cited_arxiv_id":null,"evidence_quote":"Conjectured that SGD solutions are linearly connectable modulo permutation, the premise behind mask alignment."},{"cited_title":"K., Roy, D","cited_arxiv_id":null,"evidence_quote":"Established linear mode connectivity and weight-rewinding for lottery tickets, grounding the rewind-point training."},{"cited_title":"W., Frankle, J., Ganguli, S., and Dziugaite, G","cited_arxiv_id":null,"evidence_quote":"Prior analysis of sparse-dense connectivity that this paper refines by adding REPAIR."},{"cited_title":"A., Keskin, C., and Dauphin, Y","cited_arxiv_id":null,"evidence_quote":"Showed LTH retrains the same solution; its diversity metrics and ensemble comparison are used to evaluate permuted models."},{"cited_title":"M., Rolnick, D., and Dziugaite, G","cited_arxiv_id":null,"evidence_quote":"Showed permutations align entire training trajectories and rewound sparse networks, motivating early matching."}],"review_version":1}