{"id":"6f0d78c2-fe77-4c34-8f19-a760b34a5e2b","arxiv_id":"2604.09940","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":1,"one_line_summary":"Hybrid fine-tuning jointly optimizes LLMs and PEFT modules with mixed-order optimization and provides convergence guarantees under a new hybrid smoothness condition.","lead":"The paper proposes a hybrid fine-tuning method for LLMs that jointly updates full model parameters and PEFT modules using a mix of zeroth-order and first-order optimization. It introduces a hybrid smoothness condition for convergence analysis of reshuffling SGD and reports empirical gains on downstream tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Hybrid smoothness condition lacks explicit verification or bounds for LLM-PEFT landscapes","rationale":"The reader's identification of the hybrid smoothness condition as the weakest assumption aligns directly with the load-bearing theoretical step. Full-text access allows checking the derivation details, but the assumption remains the point where the 'rigorous' claim is least secured without further validation. This moves the verdict from UNVERDICTED to CONDITIONAL pending that check, while crediting the empirical component as independent support.","tokens_in":1702,"tokens_out":352,"duration_ms":27901,"concrete_test":"From the full paper, extract the precise definition and any bounding arguments for the hybrid smoothness constant (likely in the theoretical framework section); then numerically estimate it on a small-scale proxy (e.g., LLaMA-7B with LoRA on a 1k-example subset) by sampling function values and gradients at multiple points—if the estimated constant grows unbounded or exceeds the paper's implicit assumptions, recompute the convergence rate without it.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central theoretical claim rests on deriving convergence rates for reshuffling-type SGD (with multiple learning rates) under a newly introduced 'hybrid smoothness condition' that mixes first-order and zeroth-order updates across full LLM parameters and PEFT modules. This condition is posited to capture the heterogeneous landscape but is used as a black-box assumption in the analysis without deriving explicit constants, showing it holds for standard LLM architectures, or providing a counter-example check. If the condition is violated even mildly in practice (e.g., due to non-smoothness from discrete token spaces or PEFT adapter interactions), the convergence guarantees do not apply to the claimed algorithm.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript proposes a hybrid fine-tuning approach for LLMs that jointly updates full model parameters and PEFT modules via a combination of zeroth-order and first-order optimization. It introduces a hybrid smoothness condition to model the heterogeneous landscape of this joint training and derives convergence rates for a reshuffling-type SGD algorithm that employs multiple learning rates. The work is completed by empirical evaluations demonstrating performance gains on downstream tasks across different model architectures.","tokens_in":1824,"tokens_out":503,"duration_ms":26749,"significance":"If the convergence analysis is valid, the paper would supply a useful theoretical lens on mixed-order optimization for large-scale models and could guide more efficient hybrid fine-tuning strategies that outperform pure full-parameter or PEFT baselines. The empirical component suggests practical viability, yet the lack of explicit verification for the central hybrid smoothness assumption reduces the immediate strength of the contribution.","major_comments":[{"comment":"Convergence analysis section: the hybrid smoothness condition is posited as the key assumption enabling the multi-rate reshuffling SGD bounds, yet the manuscript provides neither explicit constants nor any verification (analytic or empirical) that the condition holds for standard LLM-PEFT loss landscapes. Because the stated rates are derived directly from this condition, its unverified status is load-bearing for the central theoretical claim.","section":"Convergence analysis"},{"comment":"Algorithm description and § on multiple learning rates: the analysis treats the separate learning rates for the LLM and PEFT components as free parameters without deriving or bounding their admissible ranges, which risks making the convergence result circular with respect to the newly introduced hybrid smoothness condition.","section":"Algorithm and convergence framework"}],"minor_comments":[{"comment":"The abstract and introduction would benefit from a concise statement of the precise convergence rate obtained (e.g., O(1/T) or O(1/sqrt(T))) rather than the generic claim of 'rigorous convergence analysis'.","section":"Abstract and Introduction"},{"comment":"Empirical section: standard deviations or confidence intervals are not reported for the performance tables; adding them would strengthen the claim of consistent improvement.","section":"Experiments"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be a first submission; the hybrid smoothness condition is presented without reference to prior smoothness notions in non-convex or heterogeneous optimization, which may affect perceived novelty."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive and detailed feedback on our manuscript. We address each major comment point by point below, indicating the revisions we will incorporate to strengthen the theoretical and empirical aspects of the work.","responses":[{"response":"We acknowledge that the hybrid smoothness condition is a modeling assumption central to deriving the convergence rates, and the manuscript does not include explicit constants or direct verification. Analytically computing explicit constants for general LLM-PEFT landscapes is intractable due to the scale and non-convexity of the loss surfaces. In the revision, we will add a dedicated subsection with empirical verification: we will estimate the hybrid smoothness parameters numerically on representative fine-tuning tasks using models such as LLaMA-7B with LoRA and report whether the condition approximately holds, along with sensitivity analysis. We will also make the dependence of the rates on these parameters fully explicit in the theorem statements.","revision_made":"partial","referee_comment":"[Convergence analysis] Convergence analysis section: the hybrid smoothness condition is posited as the key assumption enabling the multi-rate reshuffling SGD bounds, yet the manuscript provides neither explicit constants nor any verification (analytic or empirical) that the condition holds for standard LLM-PEFT loss landscapes. Because the stated rates are derived directly from this condition, its unverified status is load-bearing for the central theoretical claim."},{"response":"We agree that the admissible ranges for the learning rates should be stated explicitly to avoid any appearance of circularity. The original analysis selects the rates to satisfy descent inequalities involving the hybrid smoothness constants, following standard non-convex SGD practice. In the revised manuscript, we will update the algorithm description and the statement of the main convergence theorem to include precise bounds (e.g., the LLM learning rate η_full < 1/(2L_h) where L_h denotes the hybrid smoothness constant, and analogous bounds for the PEFT rate). This will be presented prior to the theorem so that the conditions are non-circular.","revision_made":"yes","referee_comment":"[Algorithm and convergence framework] Algorithm description and § on multiple learning rates: the analysis treats the separate learning rates for the LLM and PEFT components as free parameters without deriving or bounding their admissible ranges, which risks making the convergence result circular with respect to the newly introduced hybrid smoothness condition."}],"tokens_in":1305,"tokens_out":496,"duration_ms":31365,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is a hybrid fine-tuning algorithm that updates both the full LLM parameters and the PEFT modules in one pass, blending zeroth-order and first-order gradients, plus a convergence result for reshuffling SGD under multiple learning rates. The authors frame this as a practical middle ground between expensive full tuning and weaker PEFT performance. They also introduce a hybrid smoothness condition meant to handle the mixed landscape and claim rigorous rates plus empirical gains across tasks and models. That combination of algorithm design and theory is what is actually new here. The empirical section apparently shows consistent improvements, which gives the work some practical grounding even if the numbers are not detailed in the abstract. The hybrid update rule itself is a clear departure from standard PEFT literature. The soft spot is the hybrid smoothness condition. It is used as the key assumption for the convergence proof but without explicit constants, checks against real LLM-PEFT landscapes, or discussion of whether discrete token spaces or adapter interactions violate it. If the condition fails to hold even modestly, the stated rates do not apply to the algorithm as implemented. The analysis risks becoming circular if the condition is defined in terms of the very quantities the proof aims to bound. This paper is aimed at researchers working on efficient LLM adaptation who want both an algorithmic tweak and some optimization theory. Readers focused on practical fine-tuning methods could extract value from the hybrid scheme and the reported gains, while those interested in convergence analysis for non-standard landscapes would need to verify the assumption themselves. I would send it to peer review. The core idea has enough substance and the empirical side shows effort, so referees could usefully pressure the theory on the smoothness condition and the experimental details.","headline":"The paper proposes a hybrid full-plus-PEFT update with mixed zeroth/first-order steps and convergence analysis under a new hybrid smoothness condition, but that condition is introduced without verification or bounds.","tokens_in":2290,"tokens_out":415,"would_cite":false,"duration_ms":21402,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A hybrid fine-tuning method jointly optimizes all LLM parameters and PEFT adapters using both zeroth-order and first-order updates, supported by a convergence proof under hybrid smoothness.","keywords":["hybrid fine-tuning","LLMs","PEFT","zeroth-order optimization","first-order optimization","convergence analysis","reshuffling SGD","hybrid smoothness"],"falsifier":"A controlled synthetic loss surface that violates the hybrid smoothness condition, where the proposed reshuffling SGD fails to reach a stationary point at the rate stated in the convergence theorem.","tokens_in":2592,"feed_emoji":"⚙️","tokens_out":702,"duration_ms":59521,"temperature":0.7,"pith_summary":"Large language models are usually fine-tuned either by updating every parameter, which is expensive, or by updating only a small adapter set, which often fails to absorb new knowledge fully. This paper introduces a hybrid scheme that updates both sets together inside one algorithm that mixes derivative-free steps with gradient steps. A hybrid smoothness condition is introduced to capture the different curvature properties of the full model and the adapters. The authors prove that a reshuffling stochastic gradient method converges under this condition when separate learning rates are used for each parameter group. Empirical tests on multiple tasks and architectures show the joint updates deliver higher accuracy than either pure full tuning or pure PEFT alone.","feed_headline":"Hybrid updates boost LLM fine-tuning with mixed-order SGD","feed_subtitle":"Joint LLM and PEFT optimization converges under a new smoothness condition and delivers better results than either method alone.","key_machinery":"The hybrid smoothness condition that accounts for the heterogeneous optimization landscape when jointly training full LLM parameters together with PEFT modules.","core_discovery":"We propose a novel hybrid fine-tuning approach that jointly updates both LLMs and PEFT modules using a combination of zeroth-order and first-order optimization methods. To analyze our new algorithm, we develop a theoretical framework centered on the concept of hybrid smoothness condition, which accounts for the heterogeneous nature of the optimization landscape in joint LLM and PEFT training. We derive a rigorous convergence analysis for the convergence of reshuffling-type SGD algorithm under multiple learning rates and demonstrate its effectiveness through extensive empirical studies across various downstream tasks and model architectures.","pith_inferences":["The same hybrid smoothness idea could be tested on other parameter-efficient families such as prompt tuning or prefix tuning to see whether the convergence guarantees carry over.","Applying the mixed-order scheme to vision-language models would test whether the heterogeneous landscape pattern appears outside pure text tasks.","Relaxing the hybrid smoothness to allow for the discrete token-level effects common in language model losses would make the theory closer to real training runs."],"forward_implications":["The reshuffling-type SGD algorithm converges to a stationary point when separate learning rates are assigned to the LLM and PEFT parameter groups.","The hybrid method produces higher accuracy than standard full fine-tuning or PEFT on multiple downstream tasks and model sizes.","Joint updates mitigate the high compute cost of full tuning while overcoming the limited knowledge uptake of adapter-only tuning.","Multiple learning rates allow independent step-size control for the full model and the adapters inside the same training run."],"fun_headline_variants":["Hybrid fine-tuning jointly optimizes LLMs and PEFT using mixed-order SGD","Convergence of reshuffling SGD analyzed under hybrid smoothness condition","Joint zeroth-order and first-order methods for LLM and PEFT updates","Reshuffling-type SGD converges under hybrid smoothness with multiple rates"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The joint optimization landscape of LLM parameters and PEFT modules obeys a hybrid smoothness condition that separately controls the smoothness of each group.","fun_headline_variants_meta":{"raw":{"variants":["Hybrid fine-tuning jointly optimizes LLMs and PEFT using mixed-order SGD","Convergence of reshuffling SGD analyzed under hybrid smoothness condition","Joint zeroth-order and first-order methods for LLM and PEFT updates","Reshuffling-type SGD converges under hybrid smoothness with multiple rates"]},"model":"grok-4.3","cost_usd":0.010451,"raw_usage":{"total_tokens":4617,"prompt_tokens":658,"num_sources_used":0,"completion_tokens":65,"cost_in_usd_ticks":104512000,"prompt_tokens_details":{"text_tokens":658,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3894,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":658,"tokens_out":65,"duration_ms":46560,"temperature":1.0,"reasoning_tokens":3894,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-10T16:32:11.391821+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A controlled synthetic loss surface that violates the hybrid smoothness condition, where the proposed reshuffling SGD fails to reach a stationary point at the rate stated in the convergence theorem.","supporting_citations":[],"review_version":1}