{"id":"a02af79f-2745-4896-b998-a50d2f2b7610","arxiv_id":"2608.11749","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MOON applies spectral-nuclear-norm geometry to multi-objective gradient manipulation and uses polar-factor updates, with O(T^-1/2) deterministic and O(T^-1/4) stochastic convergence to Pareto stationarity.","lead":"This paper proposes MOON, a multi-task training method that performs gradient manipulation under matrix spectral-nuclear geometry and updates parameters with orthonormalized (polar) factors of a momentum-smoothed weighted gradient. It claims convergence to Pareto stationarity at standard non-convex rates and reports modest but consistent improvements over twelve MOO baselines on five benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Convergence proof covers a single matrix block; the blockwise extension used in experiments is asserted without a joint stationarity measure or descent lemma, leaving the central convergence claim unestablished for real networks.","rationale":"The reader's weakest assumption identifies exactly the single-block versus blockwise gap. I agree that this is the most load-bearing concern: the central claim is convergence to Pareto stationarity for the method as applied to neural networks, yet the proof only covers a single matrix parameter. The blockwise statement in Section 'Multi-Objective Orthonormalized Updates' is an assertion, not a theorem, and the optimization literature shows that per-block smoothness and updates do not automatically imply joint descent for a product parameterization. This concern is independent of the Newton–Schulz approximation issue; even with an exact polar factor, the theory does not cover multiple blocks. The paper's empirical claims may still hold, but the formal guarantee stated in the abstract and theorems is not established for the actual algorithm. Because the gap is addressable—by adding a multi-block convergence analysis under explicit joint smoothness assumptions—the appropriate verdict remains CONDITIONAL, not REJECT. The unconditional acceptance would be premature, and there is no basis for rejection given that the single-block analysis is coherent and the experiments appear extensive. The reader's overall judgment is sound; my stress-test sharpens the justification for the conditional verdict without moving it.","tokens_in":30447,"tokens_out":11330,"duration_ms":111338,"concrete_test":"Analytically re-derive Theorem 4 for a two-block quadratic MTL loss ℓ_i(Θ_1,Θ_2) = 1/2‖Θ_1 A_i + Θ_2 B_i − C_i‖² with blockwise polar updates. Define the joint stationarity measure as min_z ‖Σ z_i (∇_{Θ_1}ℓ_i, ∇_{Θ_2}ℓ_i)‖ for a suitable joint norm (e.g., max of block spectral norms), and check whether the descent inequality corresponding to Equation (26) closes. If closing the inequality requires cross-block smoothness assumptions beyond Assumption 1, the blockwise extension is unjustified; if a numeric simulation on this two-block quadratic shows the averaged stationarity measure decreasing slower than O(1/√T), the theoretical claim for the practical algorithm fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The convergence theory is developed exclusively for a single matrix-valued parameter Θ ∈ R^{p×q}. Every descent lemma and theorem in Appendix A.3–A.4—Lemma 1, Lemma 2, Theorem 4, Theorem 6—assumes ℓ_i : R^{p×q} → R is L-smooth with respect to the spectral norm, and the parameter update is Θ_{t+1} = Θ_t − α W_t for one matrix. Algorithm 1, however, is applied to full networks with many matrix blocks, and the paper only says 'the same construction is applied blockwise' (Section 'Multi-Objective Orthonormalized Updates'). No joint norm, joint Pareto-stationarity measure, or descent lemma for the product of blocks is provided. For a blockwise parameterization, the aggregate gradient is a tuple of matrices, so the stationarity measure min_z ‖Σ z_i ∇ℓ_i(Θ)‖_{S1} is no longer well-defined; moreover, updating one block introduces cross-block terms in each objective's upper bound, which the single-block proof never controls. Without an extension, the O(T^{−1/2}) and O(T^{−1/4}) rates do not apply to the algorithm used in the experiments. This is a scope gap, not an internal contradiction, but it is load-bearing because the central claim is convergence for the network setting. Secondary issues—the contradiction between the Limitations section and Appendix D on Newton–Schulz error, and the absence of baseline error bars—further support a conditional verdict, but the blockwise gap is the most decisive.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MOON, a multi-task learning method that performs gradient manipulation under spectral--nuclear norm geometry for matrix-valued parameters. The main algorithmic update aggregates task gradients with learned simplex weights, applies momentum, and uses the polar factor of the momentum matrix as the update direction, with a finite Newton--Schulz approximation in practice. The theoretical contribution is a convergence analysis for smooth, possibly non-convex objectives: an averaged nuclear-norm Pareto-stationarity measure converges at rate O(T^{-1/2}) with exact gradients and O(T^{-1/4}) with unbiased stochastic gradients. The empirical section reports results on MultiMNIST, NYU-v2, CityScapes, QM9, and CelebA, plus toy, synthetic, and large-language-model fine-tuning experiments, claiming faster convergence and competitive or better final performance relative to Euclidean MOO baselines.","tokens_in":30717,"tokens_out":8848,"duration_ms":99534,"significance":"If the central claims hold, MOON is a principled bridge between matrix-aware optimizers such as Muon and multi-objective gradient manipulation, and the reported experiments suggest practical gains in wall-clock efficiency and final task quality. The paper has genuine strengths: the dual derivation via spectral--nuclear norm duality is elegant, the comparison inequality (Eq. 10) follows from feasibility rather than fitting, the descent lemmas in the appendix are explicit and internally consistent for the single-block setting, and the authors provide code, ablations, and a separate appendix addressing finite-step Newton--Schulz approximation. However, the convergence theory is developed only for a single matrix-valued parameter block, whereas the algorithm is applied blockwise to full networks; this is a load-bearing scope gap because the advertised rates are claimed for the network algorithm actually used in the experiments.","major_comments":[{"comment":"The convergence analysis is stated for a single matrix block Θ in R^{p×q}, with every lemma and theorem assuming ℓ_i is L-smooth with respect to the spectral norm and that the update is Θ_{t+1} = Θ_t − α W_t for one matrix. Algorithm 1, however, is applied to full networks by the sentence 'the same construction is applied blockwise', and no joint norm, joint Pareto-stationarity measure, or descent lemma for a product of blocks is provided. The stationarity measure G(Θ) = min_z ||Σ z_i ∇ℓ_i(Θ)||_{S1} is not well-defined when ∇ℓ_i is a tuple of block gradients, and cross-block interactions are not controlled by the single-block proof. Since the O(T^{-1/2}) and O(T^{-1/4}) rates are the paper's central theoretical claim, this gap must be closed by either extending the analysis to blockwise parameterizations or sharply restricting the stated claims to the single-block setting.","section":"Section 'Multi-Objective Orthonormalized Updates' and Section 'Convergence Analysis'"},{"comment":"Proposition 3 states that the dual of the minimax problem (7) is min_{z∈Δ_m} (1/2)||Σ z_i ∇ℓ_i(Θ_t)||_{S1}^2 and that the exact primal solution is recovered as C U V^T with C equal to the nuclear norm of the weighted gradient. This is a central step in the derivation, yet no proof is given in the main text or the appendix. A standard Fenchel-duality argument can establish the result, but the manuscript should include it explicitly, and it should also address the rank-deficient case rather than assuming full rank.","section":"Proposition 3, Section 'Multi-Objective Orthonormalized Updates'"},{"comment":"The Limitations section states that the Newton--Schulz approximation error is omitted from the current analysis and 'leave[s] its precise theoretical characterization to future work', but Appendix D does precisely the opposite: it introduces Assumption 4, proves Lemmas 5--7, and derives Theorem 8, concluding that a fixed number q of Newton--Schulz iterations preserves the O(T^{-1/2}) deterministic rate up to a factor 1/(1−ε_q). These two passages directly contradict each other. Moreover, the stochastic setting is dismissed with 'the same argument' after conditioning on the history, without a formal statement or proof. The authors should reconcile the limitation statement with Appendix D and provide the full stochastic Newton--Schulz analysis if Theorem 2 is to be claimed for Algorithm 1 as implemented.","section":"Section G 'Limitations' and Appendix D"},{"comment":"Theorem 2 is stated for Algorithm 1, but Algorithm 1 computes W_t by Newton--Schulz iterations while the proof of Theorem 6 assumes the exact polar factor of the momentum matrix. The stochastic proof also uses δ_t = [⟨W_t, ∇ℓ_1(Θ_t)⟩, ..., ⟨W_t, ∇ℓ_m(Θ_t)⟩] while the stochastic model assumes access only to unbiased stochastic gradients; the manuscript does not specify how δ_t is computed from stochastic gradients in the analysis. These mismatches between the theorem statement, the algorithm, and the proof assumptions should be resolved or explicitly declared as idealized analysis.","section":"Theorem 2 and Appendix A.4"}],"minor_comments":[{"comment":"The sentence 'creates a theoretical gap between their underlying assumptions, creates a mismatch between their underlying optimization principles' contains a duplicated and ungrammatical construction; please revise.","section":"Introduction"},{"comment":"The baseline results are reported without standard deviations, while Appendix F.9 reports error bars only for MOON. Given that the empirical claims rest on comparisons over three seeds, adding error bars for the baselines (or stating that they are omitted for space) would substantially strengthen the comparison.","section":"Tables 1--4"},{"comment":"Several table entries contain formatting errors, for example the SI row in Table 4 reads '135.71.00' and '4.5155.3', and Appendix F.7 uses 'Euclidian' instead of 'Euclidean'. These should be corrected.","section":"Table 4 and Appendix F"},{"comment":"The notation min_{z*_t∈Δ_m} ||∇L(Θ_t) z*_t||_{S1} is ambiguous because ∇L(Θ_t) is not defined as a matrix for a tuple of task gradients; writing Σ_{i=1}^m z_i ∇ℓ_i(Θ_t) would be clearer.","section":"Theorems 1 and 2"},{"comment":"Assumption 4 is a uniform, trajectory-dependent bound on the Newton--Schulz approximation error, but no empirical estimate of ε_q is provided for the experiments; a measurement or a justification of why the bound is reasonable for the tested architectures would be helpful.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The single-block to blockwise gap is the decisive issue: the paper's advertised convergence rates are not established for the network algorithm used in the experiments. This is fixable in principle, but it requires real mathematical work, not just a remark. The contradiction between the Limitations section and Appendix D, and the missing proof of Proposition 3, reinforce the need for a careful revision. The empirical study is broad and the code release is commendable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the core idea is genuinely new: formulate multi-task gradient manipulation under spectral–nuclear norm geometry, derive the dual as nuclear-norm minimization, and update with the polar factor. That is a clean and principled departure from the usual Euclidean flatten-and-combine. Second, the convergence theorem only covers a single matrix block, while Algorithm 1 applies the update blockwise to real networks. The gap between theory and algorithm is real and load-bearing.\n\nWhat the paper does well: the derivation is honest and the single-block analysis is internally consistent. The dual of the spectral-norm minimax is the nuclear-norm minimization, the primal recovery via the polar factor is right, and the descent lemmas in Appendix A.3 and A.4 support the stated O(T^{-1/2}) and O(T^{-1/4}) rates under the stated assumptions. The experiments are broad — MultiMNIST, NYU-v2, CityScapes, QM9, CelebA, plus a Qwen3 RL fine-tuning — and the public code is a plus. The ablations show that both the orthonormalized update and the momentum matter; the toy example in Appendix E nicely illustrates why Euclidean and matrix-geometric directions differ.\n\nNow the soft spots, in proportion. The blockwise gap is the main one. The theory assumes each loss is a function of a single matrix Θ in R^{p×q}, and every lemma and theorem is stated for that setting. Algorithm 1, however, updates a full network with many matrix blocks. The phrase \"the same construction is applied blockwise\" is not a proof: there is no joint stationarity measure for a tuple of blocks, no descent lemma that controls cross-block terms, and no argument that blockwise iteration preserves the rates. This is a scope gap rather than an internal contradiction, but it directly affects the central claim that MOON converges for the networks used in the experiments. Second, Proposition 3 — the dual representation that motivates the entire method — is stated without proof. It deserves a proof in the appendix. Third, the Limitations section says the Newton–Schulz approximation error is omitted from the analysis, but Appendix D analyzes exactly that error under Assumption 4 and shows the O(T^{-1/2}) rate is preserved up to a factor 1/(1-ε_q). Those two statements contradict each other; a referee should ask which version the authors actually stand behind. Fourth, baseline error bars are missing. The appendix gives MOON's own standard deviations, which are small, but without the same for baselines the 0.2–0.3% MultiMNIST gains are hard to judge; the CityScapes gains are larger and more convincing.\n\nWho this is for: people working on matrix-aware optimizers or multi-task gradient manipulation. The single-block theory is a useful building block even if the blockwise extension is unfinished. The citation pattern looks fine — self-citations are to directly relevant prior work.\n\nRecommendation: this deserves a serious peer review, not a desk reject. A good referee should push the authors to either prove a joint blockwise convergence result or explicitly reframe the theory as single-block with the blockwise version framed as empirical, and also to reconcile the Limitations section with Appendix D and prove Proposition 3. Those are fixable in revision, and the core idea is worth engaging with.","headline":"Genuinely new geometry for multi-task gradient manipulation, but the convergence theorem covers only a single matrix block, so the headline rates don't formally apply to the blockwise algorithm used in the experiments.","tokens_in":31315,"tokens_out":3242,"would_cite":true,"duration_ms":35428,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["90C29","90C26","65K05","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"MOON claims that multi-task gradient manipulation should happen in the spectral-nuclear norm geometry of matrix parameters, not in flattened Euclidean space, and proves Pareto-stationarity convergence at O(T^{-1/2}) deterministic and…","keywords":["multi-task learning","multi-objective optimization","gradient manipulation","spectral norm","nuclear norm","polar factor","orthonormalized updates","Pareto stationarity"],"falsifier":"Run MOON under deterministic gradients on a network with at least two interacting weight matrices (for instance, a two-layer linear network) and measure (1/T) sum_t min_z ||sum_i z_i grad_i(Theta_t)||_{S1}; if the measure does not decay at the O($T^{{-1/2}}$) rate (or climbs, or plateaus far above zero) while a single-block toy case does decay, the blockwise transfer claimed is false. Alternatively, a two-block example where each block's polar direction is a descent direction but the coupled update raises a shared loss would falsify the blockwise application.","tokens_in":30166,"feed_emoji":"📐","tokens_out":5172,"duration_ms":49578,"temperature":0.7,"pith_summary":"The paper argues that standard multi-objective optimization methods for multi-task learning flatten parameters into vectors and combine task gradients in Euclidean space, which is not the steepest descent direction when parameters are matrices. It proposes MOON, which solves a minimax problem over matrix updates under the spectral norm; the dual selects task weights by minimizing the nuclear norm of the weighted average gradient, and the update direction is the polar factor of that average. For smooth non-convex objectives, the averaged Pareto-stationarity measure converges at O($T^{{-1/2}}$) with exact gradients and O($T^{{-1/4}}$) with unbiased stochastic gradients. Experiments across five multi-task benchmarks show faster optimization and competitive or better final performance.","feed_headline":"Multi-task optimizer proves 1/sqrt(T) convergence by respecting matrix shape","feed_subtitle":"Gradient manipulation in spectral-nuclear geometry beats Euclidean flattening on multi-task benchmarks.","key_machinery":"The load-bearing object is the nuclear-norm Pareto-stationarity measure G(Theta) = min_{z in Delta_m} ||sum_i z_i grad_i(Theta)||_{S1}, the dual of the spectral-norm minimax subproblem min_W max_i [-<grad_i, W> + 1/2||W||^2_{S∞}]. Its exact solution is C * U V^T, where C is the nuclear norm and U V^T is the polar factor of the weighted aggregate gradient; the polar factor carries the argument because under spectral-norm smoothness it is the steepest descent direction, and the nuclear norm provides a matrix-geometric suboptimality certificate via duality. The proof works by telescoping weighted losses, bounding the momentum-vs-current-gradient error, and using softmax Lipschitzness to control weight drift; finite-step Newton-Schulz error is shown to alter only constants under a uniform approximation condition.","core_discovery":"MOON establishes that the correct geometry for multi-objective gradient manipulation on matrix-valued parameters is the spectral-nuclear norm pair: each objective is assumed L-smooth with respect to the spectral norm, the common descent direction is chosen by the minimax problem min_W max_i [-<grad_i, W> + 1/2||W||^2_{S∞}], whose dual minimizes the nuclear norm of the weighted aggregate gradient, and the primal solution is the polar factor times the nuclear norm. Because the polar factor discards singular-value magnitudes, the update is governed by singular subspaces rather than by the largest singular components, which the paper identifies as the reason Euclidean flattening is suboptimal under matrix geometry. The paper proves that Algorithm 1 (with momentum, Newton-Schulz polar approximation, and online softmax weight updates) drives the averaged nuclear-norm Pareto-stationarity measure to zero at the stated rates, and that at any fixed iterate the MOON weighting gives a nuclear-norm stationarity certificate no worse than Euclidean min-norm weighting.","pith_inferences":["A natural extension the paper does not develop: replacing the spectral norm by Schatten p-norms would interpolate between polar-factor updates (p = infinity) and Euclidean updates, giving a tunable family whose rates should degrade continuously.","The nuclear-norm certificate suggests a practical diagnostic: monitor min_z ||sum_i z_i grad_i(Theta)||_{S1} during training as a geometric early-stopping or task-conflict signal, independent of validation metrics.","If the blockwise convergence gap is closed, the same spectral-nuclear geometry could be composed with block preconditioners to yield multi-objective optimizers for very large models.","The two-task toy example shows the benefit is not update scaling: under an equal spectral-norm budget, the MOON direction gives strictly larger common descent, implying the geometry itself, not step size, drives the gain."],"forward_implications":["MOON attains the standard sublinear rate for smooth non-convex multi-objective optimization: O(T^{-1/2}) deterministic, O(T^{-1/4}) with unbiased bounded-variance stochastic gradients.","At any fixed iterate, the MOON dual weighting yields a nuclear-norm Pareto-stationarity certificate no larger than Euclidean min-norm (MGDA-type) weighting, so the matrix-geometric dual is never worse at that iterate.","The polar-factor update removes singular-value magnitude information, so updates are driven by singular subspaces, which the paper argues is why MOON avoids being dominated by large singular components and converges faster in practice.","The practical algorithm's finite Newton-Schulz approximation preserves the O(T^{-1/2}) rate whenever the approximation error is uniformly bounded below 1 in operator norm.","On the tested benchmarks (MultiMNIST, NYU-v2, CityScapes, QM9, CelebA), MOON reduces the average performance drop and reaches the same training loss faster in wall-clock time."],"supporting_citations":[{"why":"Supplies the spectral-norm steepest descent result (quadratic upper bound and polar-factor update) that MOON extends to the multi-objective setting.","marker":"Bernstein and Newhouse [2024]"},{"why":"Supplies the MGDA Euclidean minimax formulation and the Pareto-stationarity notion that MOON generalizes to matrix geometry.","marker":"Sener and Koltun [2018]"},{"why":"Establishes steepest descent for multi-objective optimization, grounding the minimax-descent interpretation used by both MGDA and MOON.","marker":"Fliege and Svaiter [2000]"},{"why":"PCGrad is a Euclidean gradient-manipulation baseline that MOON critiques and compares against empirically.","marker":"Yu et al. [2020]"},{"why":"FAMO provides the online single-step dual-weight update strategy (logits with softmax) that practical MOON adopts for tracking task weights.","marker":"Liu et al. [2024]"},{"why":"Muon supplies the Newton-Schulz polar-factor approximation that MOON uses in its practical implementation and motivates matrix-aware optimizers.","marker":"Jordan et al. [2024]"},{"why":"Provides the Newton-Schulz iterative orthogonalization method used to approximate the polar factor in Algorithm 1.","marker":"Björck and Bowie [1971]"}],"fun_headline_variants":["Matrix geometry beats flattening for multi-task gradient updates","Spectral-nuclear geometry improves multi-task optimization","MOON: Gradient updates that respect matrix structure","For multi-task, steepest descent is not Euclidean","Nuclear-norm geometry sharpens multi-task gradient updates"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The convergence proof assumes a single matrix block and exact polar factors, but the paper applies the construction blockwise and uses Newton-Schulz approximations in practice, so the transfer to full networks rests on approximation conditions not proven for multi-block interaction.","fun_headline_variants_meta":{"raw":{"variants":["Matrix geometry beats flattening for multi-task gradient updates","Spectral-nuclear geometry improves multi-task optimization","MOON: Gradient updates that respect matrix structure","For multi-task, steepest descent is not Euclidean","Nuclear-norm geometry sharpens multi-task gradient updates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000642,"raw_usage":{"total_tokens":2958,"prompt_tokens":955,"completion_tokens":2003,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":1929}},"tokens_in":571,"tokens_out":2003,"duration_ms":14194,"temperature":1.0,"reasoning_tokens":1929,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:29:26.452307+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run MOON under deterministic gradients on a network with at least two interacting weight matrices (for instance, a two-layer linear network) and measure (1/T) sum_t min_z ||sum_i z_i grad_i(Theta_t)||_{S1}; if the measure does not decay at the O($T^{{-1/2}}$) rate (or climbs, or plateaus far above zero) while a single-block toy case does decay, the blockwise transfer claimed is false. Alternatively, a two-block example where each block's polar direction is a descent direction but the coupled update raises a shared loss would falsify the blockwise application.","supporting_citations":[],"review_version":1}