{"id":"f7f370ee-de7d-4dea-b72e-a6b4aa51f0a1","arxiv_id":"2505.12242","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ZenFlow is an offloading framework that updates important gradients on the GPU and asynchronously accumulates unimportant ones on the CPU, achieving up to 5x speedups with comparable accuracy.","lead":"ZenFlow is a system for fine-tuning large language models when GPU memory is tight. It keeps the most important gradients on the GPU and sends the rest to the CPU to update asynchronously, reducing idle GPU time by over 80% and speeding up training by up to 5x.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The §3.4 convergence guarantee is asserted rather than derived, and the CPU-side update's 1/S scaling plus unmonitored growth in ρ leave the 'same as ideal SGD' and accuracy-preservation claims unsupported.","rationale":"The reader's verdict is CONDITIONAL, and I do not move it. The systems contribution is plausible and reasonably supported: speedups are consistent across model scales and two hardware configurations, the I/O model in Section 3.2 is transparent and checkable, and the locality measurements in Figures 5, 6, and 9 are real empirical evidence that a channel-level proxy has some fidelity. However, the central accuracy claim rests on two pillars: the empirical locality/retention results and the bounded-staleness convergence result in Section 3.4. The weaker pillar is the convergence result, because it is asserted rather than derived, and the stated CPU update rule with the 1/S factor is not obviously covered by standard async-SGD analyses. The paper also measures channel retention, not the gradient-energy fraction rho(t) that enters the bound; if rho grows as important coordinates converge, the staleness penalty is larger than 12% at the end of training. The admitted slight accuracy drop for OPT-350M and the absence of seeded repeats make this more than a formal quibble. A clean re-derivation of the bound is the decisive check. If the bound re-derives correctly and rho(t) is confirmed to stay small, the theoretical support is intact and the remaining issues are reproducibility and rigor; if not, the accuracy-preservation claim should be made conditional on additional measurements or a revised analysis.","tokens_in":21780,"tokens_out":10028,"duration_ms":114177,"concrete_test":"Independently re-derive the Section 3.4 convergence rate from the stated assumptions (unbiased gradients, bounded variance, L-smoothness) for the exact update rule theta_c update = alpha_t/S * sum_{i=t-S+1}^t grad_c L(theta_i), without importing the stated result. If the derivation does not yield O(sqrt((1+rho*S)/T)) with rho as the fixed 0.10 energy fraction -- for example, if it yields an extra sqrt(S) factor or requires rho(t) <= 0.1 for all t -- then the paper's 'same as ideal SGD' and 12%-penalty claims are unsupported, and the accuracy-preservation claim must rest solely on the unreleased GLUE runs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that ZenFlow 'achieves up to 5x end-to-end speedup, 2x lower PCIe traffic, and reduces GPU stalls by over 85%, all while preserving accuracy.' The load-bearing theoretical support is the bounded-staleness result in Section 3.4, stated as O(sqrt((1+rho*S)/T)) with an 18% slowdown factor for S=4, rho=0.1. This is presented as a proof, but the paper gives no theorem statement, no derivation, and no specific cited theorem that implies this exact formula. The stated update rule is also internally questionable: theta^(c) is updated only every S iterations using alpha_t * (1/S) * sum_{i=t-S+1}^t grad_c L(theta_i), i.e., a single learning-rate-scaled average every S steps. Under standard nonconvex SGD analysis, coordinates updated only every S steps with delay do not automatically inherit the synchronous 1/sqrt(T) rate; the formula as written requires an independent derivation. Moreover, the small penalty relies on rho staying near 0.10 for the whole run. As the GPU-updated important coordinates converge, their gradient energy can shrink, so the delayed coordinates' energy fraction rho can grow late in training, exactly when final validation accuracy is determined. The paper never reports rho over time; Figures 6 and 9 show channel retention and top-k density, not the gradient-energy ratio rho(t) that appears in the bound. This is not purely formal: Section 5.2 concedes that OPT-350M accuracy is slightly lower under fixed S=4, and the GLUE results are single-seed with no error bars. Thus the accuracy-preservation claim rests on an unproven bound and an unmeasured dynamic quantity.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"ZenFlow is an offloading framework for fine-tuning large language models that partitions parameters into an important subset updated in-place on the GPU and a less-important subset asynchronously accumulated and updated on the CPU. It uses a channel-level gradient-norm proxy to identify important parameters without global all-gather, exploits claimed spatial and temporal locality of important gradients, and overlaps CPU optimizer work with GPU computation using double buffering. The paper claims up to 5x end-to-end speedup, 2x lower PCIe traffic, over 85% reduction in GPU stalls, and accuracy preservation, supported by a bounded-staleness convergence rate of O(sqrt((1+rho*S)/T)) that is stated as matching ideal synchronous SGD up to a small constant.","tokens_in":22120,"tokens_out":7500,"duration_ms":79956,"significance":"If the performance and accuracy claims hold, ZenFlow addresses a practical and important bottleneck: GPU idle time caused by CPU-side optimizer updates in offloaded LLM fine-tuning. The evaluation is broad and internally consistent, covering five model families, two GPU platforms, and three CPU configurations, and the ZenFlow vs ZenFlow* ablation cleanly isolates the contribution of the pipelining design. The spatio-temporal locality observation for important gradients is a useful, falsifiable empirical finding. However, the theoretical convergence guarantee is asserted rather than derived, the key staleness parameter rho is not measured over training, and the reported I/O reduction is weaker than the abstract claims; these issues currently limit the strength of the central claims.","major_comments":[{"comment":"The convergence bound O(sqrt((1+rho*S)/T)) is stated without a theorem statement, proof, or a citation to a result that implies this exact formula. The update rule for theta^(c), which is updated only every S iterations using alpha_t/S times the sum of gradients computed at theta_i for i in [t-S+1, t], is a partial-delay scheme not directly covered by the cited async-SGD analyses [11,22,24,54]. The text says the result follows from 'common assumptions' but does not specify the Lipschitz constant, the variance bound, or the step-size schedule, so the claim that ZenFlow achieves 'the same as the ideal rate of synchronous SGD' is not checkable as written. Please provide a complete theorem and proof, or explicitly derive the bound from a cited theorem, and state clearly that the proof is for SGD while the system uses AdamW.","section":"§3.4 (bounded-staleness result)"},{"comment":"The small penalty factor sqrt(1+rho*S) depends on rho staying near 0.10 for the whole training run, but the paper never reports rho(t), the fraction of gradient energy in the delayed coordinates, over time. Figure 6b reports channel retention rate and Figure 9 reports a cumulative distribution of top-k gradients, neither of which is rho(t). As the GPU-updated important coordinates converge, their gradient energy can shrink, so the delayed coordinates' energy fraction can grow late in training, exactly when final validation accuracy is determined. The authors should report rho(t) for the evaluated workloads and either bound rho in the convergence theorem via a schedule-dependent argument or add a mechanism that provably keeps rho small.","section":"§3.4 and §5.4 (rho dynamics)"},{"comment":"The paper's own I/O formula gives an average per-iteration traffic of (S+1)*(1-k)*M/S; with the stated representative values S=4 and k=0.1 this is 5*0.9*M/4 = 1.125M, compared to 2M for ZeRO-Offload. That is a 1.78x reduction, not the '2x lower PCIe traffic' claimed in the abstract. Please either qualify the abstract and §3.2, report measured byte counts, or state that 2x is approached only for larger S.","section":"§3.2 and abstract (I/O traffic)"},{"comment":"The accuracy-preservation claim is not fully supported by the reported experiments. Section 5.2 explicitly states that OPT-350M accuracy is slightly lower under the fixed update interval S=4, which is in tension with the abstract's 'all while preserving accuracy.' In addition, Fig. 10 reports no error bars or seeded repeats for the GLUE results, so differences of 0.005-0.02 in accuracy cannot be distinguished from stochastic noise. Please report mean and standard deviation over at least three seeds, and show whether Zen-auto recovers the OPT-350M accuracy loss in the main accuracy figure.","section":"§5.2 and Fig. 10 (accuracy preservation)"}],"minor_comments":[{"comment":"The abstract contains the typo 'fulling overlapping' instead of 'fully overlapping', and the name 'Tekin Bicer' appears twice in the author list.","section":"Abstract/author list"},{"comment":"The text says 'the top 1% of gradients account for over 90% of the total gradient norm', while Fig. 4 reports 88.9% for the Qwen2.5-0.5B run; please make the numbers consistent or add the qualifier 'approximately'.","section":"§2.3 and Fig. 4"},{"comment":"The warm-up penalty example does not evaluate to the stated 0.12. With beta=0.6, tau/T=0.05, rho=0.1, and S=4, the displayed formula gives sqrt(1 + 0.4*(1 - 0.05^0.4)) ≈ 1.131, i.e., a 13.1% penalty, not 0.12; please correct the arithmetic or the formula.","section":"§3.4 (warm-up penalty)"},{"comment":"The paper says fine-tuning is evaluated on the GLUE benchmark, but only four tasks (MNLI, QNLI, QQP, SST-2) appear in Fig. 10; please state explicitly that this is a subset of GLUE or add the remaining tasks.","section":"§5.1 and Fig. 10"},{"comment":"The 'Zen-Auto' marker in Fig. 15(a) is not explained in the text; please describe how the auto-tuned configuration is obtained and how it differs from the fixed S=4 configuration in Fig. 10.","section":"Fig. 15(a)"}],"recommendation":"major_revision","confidential_remarks":"The main risk in this submission is Section 3.4: the convergence guarantee is central to the accuracy-preservation claim, but as written it is an assertion rather than a derivation. If the authors supply a rigorous proof, rho(t) measurements, and proper error bars for the accuracy comparison, the paper would be a strong systems contribution. I did not find a novelty conflict with the cited offloading or gradient-sparsity works, but the abstract and Section 3.2 should be aligned with the actual I/O reduction numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a genuine systems contribution. The new thing is the channel-level proxy for gradient importance plus the observation that important gradient channels stay stable across fine-tuning iterations. That observation is concrete and falsifiable, and it is what makes the selection cheap. The evaluation is unusually wide: five model families, A100 and H100, three CPU configurations, and it consistently shows 3.6–5× speedups over ZeRO-Offload. I believe those numbers if the system is as described.\n\nThe soft spots are real but mostly fixable. Section 3.4 presents a bounded-staleness convergence rate without a theorem statement or derivation. The rate may be true, but as written it is asserted. The stress-test note is right that rho, the delayed-gradient energy fraction, is treated as a constant around 0.10 when it could plausibly grow late in training as important coordinates converge. The paper never reports rho over time; it reports channel retention, not the energy ratio in the bound. That is a genuine missing measurement, but it is not fatal because the accuracy results are independently tested on GLUE and the main claim is throughput, not a new optimizer.\n\nSmaller issues: no error bars, no seeded repeats, no code or data release, so the speedups and the accuracy-parity claims are single-shot. The abstract says '2× lower PCIe traffic' while the paper's own formula gives 1.78×; the abstract also says 'over 85%' stall reduction, which the evaluation supports for most but not all configurations. The channel-locality validation is done on Qwen2.5-0.5B only, and the paper extrapolates to 7B and 13B models without showing the same heatmaps for them.\n\nWho is the paper for? Systems people working on offloaded training or heterogeneous CPU-GPU execution. The channel-locality observation and the double-buffered accumulation scheme are worth knowing even if the convergence proof gets fixed. It deserves a serious referee; I would not desk-reject. The referee should ask for code and data, seeded repeats, a real derivation or citation for the staleness bound, rho(t) over a full run, and a corrected abstract.","headline":"A solid systems paper with a real locality observation, a broad evaluation, and an under-built convergence proof that should be fixable in revision.","tokens_in":22710,"tokens_out":2355,"would_cite":true,"duration_ms":24569,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ZenFlow makes offloaded LLM fine-tuning up to 5x faster by updating only important gradients on the GPU.","keywords":["offloaded training","asynchronous updates","gradient importance","spatial locality","temporal locality","GPU stalls","CPU-GPU pipeline","bounded staleness"],"falsifier":"Run the system on a fine-tuning task that switches domains mid-training, and record both the retention rate of the cached top-10% channel set and final accuracy; if retention of the top-1% gradients falls below about 90% for a sustained window while the update interval is 4, the promised accuracy preservation should measurably degrade, showing that the spatial-temporal locality premise is load-bearing.","tokens_in":21566,"feed_emoji":"⚡","tokens_out":10333,"duration_ms":86220,"temperature":0.7,"pith_summary":"The paper tries to show that the main reason offloaded LLM fine-tuning is slow is that every parameter is treated alike: the full model is sent to the CPU for every update, leaving fast GPUs idle. ZenFlow instead splits parameters by learning importance, updating a small important subset in place on the GPU while a CPU process asynchronously accumulates the rest. Its core empirical claim is that important gradients cluster in a small, slowly changing set of input channels, making lightweight channel-level selection a faithful stand-in for expensive global top-k ranking. If this holds, offloaded fine-tuning can run 3.6--5x faster with about half the PCIe traffic, over 85 percent less GPU stall time, and the same accuracy. The paper backs the approach with a bounded-staleness convergence result, $O\\big(\\sqrt{(1+\\rho S)/T}\\big)$, which it says matches the ideal synchronous SGD rate in form.","feed_headline":"CPU-offloaded LLM fine-tuning runs up to 5x faster","feed_subtitle":"Updating only important gradients on the GPU hides slow CPU work and halves PCIe traffic without hurting accuracy.","key_machinery":"The load-bearing mechanism is a channel-level gradient importance proxy. Instead of gathering the full gradient matrix, which would cost tens of gigabytes per iteration in sharded training, each GPU computes per-column squared gradient norms, shares a compact vector, and caches a slowly refreshed top-10% set of important input channels; for a $4096 \\times 4096$ weight matrix this cuts selection communication by over 4,000x. On top of that proxy, a selective GPU optimizer updates only the important channels in place, while a double-buffered CPU accumulator gathers unimportant gradients and applies them every $S$ iterations, fully overlapping CPU work with GPU compute. The convergence analysis couples these pieces through the bounded-staleness rate $O\\big(\\sqrt{(1+\\rho S)/T}\\big)$, where $\\rho$ is the fraction of gradient-norm energy in delayed coordinates and $S$ is the accumulation interval.","core_discovery":"On the paper's own terms, the central discovery is that in LLM fine-tuning the top 1% of gradients carry about 90% of the total gradient norm, and those high-magnitude gradients are not scattered: they persist in a narrow set of input channels across iterations, so a fixed top-10% channel set retains over 95% of the top-1% gradients over 100 steps. ZenFlow exploits this by performing in-place optimizer updates on the GPU for important channels and offloading the remaining gradients to the CPU, where they accumulate over a few iterations before an asynchronous update. The resulting partial staleness has a bounded cost: with the default accumulation interval $S=4$ and measured CPU-side gradient-energy fraction $\\rho \\approx 0.10$, the convergence penalty is a factor of $\\sqrt{1.4} \\approx 1.18$ relative to ideal synchronous SGD, and a short synchronous warm-up reduces the effective penalty to about 0.12. Across models from 350M to 13B parameters, the paper reports 3.6--5x end-to-end speedups over the state-of-the-art offloading baseline, roughly 2x lower I/O traffic, and accuracy matching or exceeding the baseline.","pith_inferences":["A natural extension is to let the same stable channel set drive other memory-saving decisions, such as selective quantization, pruning, or gradient compression, since the paper only applies it to update scheduling and offloading.","The bounded-staleness bound assumes the CPU-side gradient-energy fraction $\\rho$ stays near 0.1; if a later training phase shifts importance to new channels faster than the cached set refreshes, the effective penalty could exceed the reported 18%, so an adaptive refresh rate may be needed for tasks with sharp distribution shifts.","The fixed $S=4$ interval is a compromise: the paper's own auto-tuned variant suggests that starting with $S=1$--$2$ and relaxing later would make the speedup-accuracy tradeoff more robust for short fine-tuning runs.","The locality claim was measured on instruction-following and benchmark tasks; a quick test would be to monitor retention of the cached channel set on a multi-task or domain-switching fine-tuning run, where the assumption is most likely to fail."],"forward_implications":["Offloaded fine-tuning of 7B--13B models would run 3.6--5x faster than current offloading systems on the same hardware, because GPU stalls are the dominant overhead.","PCIe traffic per iteration would drop from two full model copies to about 1.125 model copies for $S=4$ and a 10% important ratio, easing the communication bottleneck that limits transfer-bound training.","GPU stalls would fall by over 80% even when the CPU is heavily under-provisioned, e.g., with only 8 CPU threads, making shared GPU clusters more predictable for offloaded training.","The convergence guarantee would remain within about 18% of ideal synchronous SGD in default settings, and within about 12% with a 5% warm-up period, so accuracy preservation is argued as a formal property rather than only an empirical one.","The approach would scale to at least 13B parameters across four GPUs with model sizes comparable to conventional offloading, since the per-column proxy avoids global synchronization."],"supporting_citations":[{"why":"Supplies the baseline offloading workflow whose full-model CPU updates and transfers create the GPU stalls the paper targets.","marker":"[37]"},{"why":"Defines fully sharded training, where each GPU holds only a gradient shard and a global gradient view is unavailable, motivating the channel proxy.","marker":"[33]"},{"why":"Provides the observation that a small fraction of gradients dominates total gradient norm, the basis for importance-based selection.","marker":"[3]"},{"why":"Earlier importance-aware offloading with lossy compression that the paper contrasts with by preserving gradient fidelity.","marker":"[17]"},{"why":"The layer-wise CPU-GPU overlap baseline that shows even pipelined updates cannot hide slow CPU updates.","marker":"[43]"},{"why":"Supplies the bounded-staleness convergence framework the paper adapts to partial staleness with delayed CPU coordinates.","marker":"[11]"},{"why":"The async-SGD result that anchors the claim that the paper's rate matches ideal synchronous SGD in order.","marker":"[51]"},{"why":"The Adam optimizer used in both GPU and CPU update paths, whose adaptive moments are relevant to the staleness analysis.","marker":"[21]"}],"fun_headline_variants":["LLM fine-tuning offloads to CPU with no GPU stalls","Top gradients on GPU make CPU offload 5x faster","ZenFlow: async updates end GPU stalls in offloaded training","Key gradients stay on GPU while rest offloads: 5x speedup","CPU offload no longer stalls GPUs in fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The design assumes that a small, slowly changing set of input channels keeps carrying the truly important gradients for the whole fine-tuning run, so a cached top-10% channel set acts as a faithful proxy for global top-k importance.","fun_headline_variants_meta":{"raw":{"variants":["LLM fine-tuning offloads to CPU with no GPU stalls","Top gradients on GPU make CPU offload 5x faster","ZenFlow: async updates end GPU stalls in offloaded training","Key gradients stay on GPU while rest offloads: 5x speedup","CPU offload no longer stalls GPUs in fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000271,"raw_usage":{"total_tokens":1645,"prompt_tokens":975,"completion_tokens":670,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":580}},"tokens_in":591,"tokens_out":670,"duration_ms":6036,"temperature":1.0,"reasoning_tokens":580,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:37:59.828782+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the system on a fine-tuning task that switches domains mid-training, and record both the retention rate of the cached top-10% channel set and final accuracy; if retention of the top-1% gradients falls below about 90% for a sustained window while the update interval is 4, the promised accuracy preservation should measurably degrade, showing that the spatial-temporal locality premise is load-bearing.","supporting_citations":[{"cited_title":"2021.{Zero-offload}: Democratizing{billion-scale} model training","cited_arxiv_id":null,"evidence_quote":"Supplies the baseline offloading workflow whose full-model CPU updates and transfers create the GPU stalls the paper targets."},{"cited_title":"Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real System","cited_arxiv_id":"2403.06664","evidence_quote":"Earlier importance-aware offloading with lossy compression that the paper contrasts with by preserving gradient fidelity."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The layer-wise CPU-GPU overlap baseline that shows even pipelined updates cannot hide slow CPU updates."}],"review_version":1}