{"id":"f14279a6-0018-4e2f-81e9-2f6e21e8e845","arxiv_id":"2506.19164","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"GradualDiff-Fed transmits model deltas in federated LLM fine-tuning, but its update rule reduces exactly to FedAvg and claims significant communication savings without measuring them.","lead":"This paper proposes GradualDiff-Fed, a federated learning method in which clients send only their model weight differences to a central server. The method is tested for fine-tuning a 7B language model on a small medical chatbot dataset, but the reported gains are not measured against standard federated learning baselines.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central communication-efficiency claim is unsupported: the delta ΔLLM_i^(t) in Eq. (1) is dense and has the same bit cost as the model it replaces, and no compression or byte-level measurement is provided.","rationale":"The reader's weakest assumption identifies exactly the right soft spot: the claimed communication saving rests on the unstated and unsupported premise that a model-weight difference is smaller in bits than the model itself. For a dense vector of the same dimension and dtype, that premise is false. The paper neither specifies a compression mechanism nor measures communication volume, and the timing evidence in Table IV does not isolate communication cost. This is an internal inconsistency in the central claim, not merely a disagreement with consensus: Eq. (1) and Algorithm 2 define a transmission object whose size is identical to the object it replaces. The reader additionally notes the update rule is FedAvg in delta form; while that is a separate novelty concern, the communication-efficiency collapse is the more load-bearing issue because it invalidates the paper's stated contribution regardless of novelty. My independent reading supports the reader's REJECT verdict: the abstract, Section III, and Section IV all advance a reduction claim that the algorithm as written cannot deliver without an unprovided compression step. I therefore see no reason to adjust the verdict; if anything, the concern is sharper than the reader's phrasing because Algorithm 2 line 2 also requires a full-model broadcast to every client each round, making the total communication budget at least as large as FedAvg even before considering the client deltas. The concrete byte-accounting test would settle the matter decisively, but the burden of proof is on the paper, and the current text provides no support for the central efficiency claim.","tokens_in":7160,"tokens_out":3672,"duration_ms":39427,"concrete_test":"Instrument Algorithm 2 with byte-accounting instrumentation and rerun the 15-round medical-chatbot setup on identical hardware. For each round, record: (a) bytes broadcast by the server, (b) bytes sent by each client, (c) the shape, dtype, and sparsity fraction of the transmitted delta, and (d) cumulative uplink and downlink communication volume for GradualDiff-Fed. Run the same instrumentation under a FedAvg-with-LoRA baseline that transmits local LoRA adapter weights. If the delta is dense and same dtype as the trainable parameters and no quantization or sparsification is applied, the total communication will be equal to or greater than FedAvg-with-LoRA, directly falsifying the claimed reduction. If adapters only are transmitted, compare total bytes against FedAvg-with-LoRA to determine whether any reduction is attributable to the delta mechanism rather than to LoRA.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim, repeated in the abstract, Section III, and conclusion, is that GradualDiff-Fed 'drastically reduces communication overhead' by transmitting only ΔLLM_i^(t) = θ_i^(t) − LLM_g^(t) instead of the full local model. The load-bearing premise is that this delta is cheaper to transmit than the model. That premise is not established and, as stated, is false: ΔLLM_i^(t) is a dense vector with exactly the same dimension and typically the same dtype as θ_i^(t), so sending it costs the same number of bits as sending the local model, unless the delta is sparsified, quantized, or otherwise compressed before transmission. Algorithm 2 contains no such mechanism: line 5 computes the subtraction and sends the result directly, and the paper reports no communication volume, no sparsity ratio, and no compression step. The evaluation's 'computational timing' in Table IV is per-round wall-clock time on a single GPU, not network bytes; the reported 36% reduction (4.3 vs. 6.7 seconds) compares the federated procedure, which trains only LoRA adapters, against centralized full-model processing, and does not measure communication at all. Moreover, Algorithm 2 line 2 requires the server to send the full global LLM weights to every client each round; the total communication budget must include this broadcast plus all client deltas. If the deltas are dense and full-precision, the per-round communication is not lower than standard FedAvg, where clients send their local model updates. If, instead, only LoRA adapters are transmitted, then any saving comes from LoRA, not from the delta formulation, and the correct baseline is FedAvg with LoRA, which is absent. Thus the paper's central efficiency claim collapses under the very definition of the transmitted object in Eq. (1).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GradualDiff-Fed, a federated fine-tuning framework for large language models in which each client transmits the difference between its locally updated model and the global model, ΔLLM_i^(t) = θ_i^(t) − LLM_g^(t), instead of the full model. The server averages these deltas and adds the average to the global model. The authors claim that this 'delta parameterization' drastically reduces communication overhead while achieving performance on par with centralized training. The evaluation is conducted with a Llama-2-7B model fine-tuned with LoRA on a mental-health dataset for 15 rounds across five clients, comparing FL, centralized learning, and local-client training on training loss, perplexity, BLEU score, and per-round computation time.","tokens_in":7504,"tokens_out":4232,"duration_ms":40625,"significance":"If the communication-efficiency claim were supported, the contribution would be modest: transmitting model deltas and averaging them is algebraically equivalent to FedAvg, and real efficiency gains would require an explicit compression mechanism. The paper does not provide such a mechanism nor a byte-level measurement. Its strengths are a clear algorithmic description and a straightforward experimental setup, but no code is supplied, no FedAvg baseline is measured, and no communication volume is reported. The claimed advantage over standard federated learning is therefore not established, and the paper's central hypothesis remains untested.","major_comments":[{"comment":"The central claim that transmitting ΔLLM_i^(t) = θ_i^(t) − LLM_g^(t) 'dramatically reduces' communication is not valid as stated. ΔLLM_i^(t) is a dense vector with the same dimension and the same bit width as θ_i^(t), so transmitting it costs the same number of bits as transmitting the locally updated model. Algorithm 2 specifies no sparsification, quantization, or compression before the transmission. Table IV reports wall-clock time per training round on a single GPU, not network bytes, and the comparison there is between LoRA-based federated training and full-parameter centralized processing, so it does not measure communication savings. The communication-efficiency claim therefore lacks both a mechanism and a measurement.","section":"Section III, Eq. (1) and Algorithm 2 line 5"},{"comment":"The aggregation rule in Eq. (2) is algebraically identical to FedAvg. Because LLM_g^(t) is constant during aggregation, LLM_g^(t+1) = LLM_g^(t) + (1/K) Σ_i (θ_i^(t) − LLM_g^(t)) = (1/K) Σ_i θ_i^(t), which is exactly the FedAvg update with uniform weighting. The paper provides no comparison against standard FedAvg with LoRA, so the claimed benefits of 'delta parameterization' over the existing baseline are not demonstrated by any experiment.","section":"Section III, Eq. (2)"},{"comment":"The experimental evaluation does not measure the quantity the paper claims to improve. There is no communication-volume measurement, no FedAvg baseline, and only one dataset with a single 15-round run. The per-round time comparison (4.3 vs. 6.7 seconds) is a computational timing on one GPU and conflates the LoRA-versus-full-model training cost with communication; the 36% reduction cannot be attributed to 'reducing communication costs through weight difference updates' as stated in Section IV-C. Furthermore, the paper itself defers non-IID evaluation to future work, so the method's central motivation is not tested.","section":"Section IV-A and Table IV"},{"comment":"The server sends the global LLM weights to every client in each round. The total communication budget therefore includes this full-model broadcast plus all client deltas. Even if the client deltas were compressed, the broadcast would be a dominant cost, yet the paper's communication analysis omits this term entirely. Any statement about reduced communication must account for both directions of the exchange.","section":"Section III, Algorithm 2 line 2"}],"minor_comments":[{"comment":"The FedAvg formula as written, θ_g^(t) ← (1/N) Σ_i (n_i / Σ_j n_j) θ_i^(t), contains a spurious 1/N factor and is not the standard FedAvg aggregation; since FedAvg is the natural baseline, this should be corrected.","section":"Section II-A"},{"comment":"There are repeated typos: 'without comprising performance' in the abstract and conclusion should be 'without compromising performance'.","section":"Abstract and Section VI"},{"comment":"The phrase '4k content size' is unclear; it presumably means a 4k context size, and 'we applied LoRA to all global layers within the adapters' should be rephrased.","section":"Section IV-A"},{"comment":"Only the Central perplexity is reported with an error bar (15.36 ± 0.2), and no standard deviations are given for the other entries; the number of repeated runs should be stated.","section":"Table III"},{"comment":"The FedAvg method is cited as reference [7], but [7] is 'Adaptive federated optimization for heterogeneous data'; the original FedAvg paper [5] should be cited instead.","section":"Section II-A and References"},{"comment":"The notation 'Δθ = Δθ_b Δθ_a' and the return value 'θ′ = θ + A×B' conflate the LoRA low-rank product with the difference notation; the relationship between A, B, Δθ_b, and Δθ_a should be defined precisely.","section":"Algorithm 1"},{"comment":"The conclusion states that future work should check performance under non-IID settings, which admits that the current evaluation does not address heterogeneity; this limitation should be stated prominently in the evaluation section rather than only in the conclusion.","section":"Section IV-C and Conclusion"}],"recommendation":"reject","confidential_remarks":"The paper is a preliminary manuscript whose central claim is unsupported: the delta mechanism is a dense vector with the same communication cost as the model, no byte-level measurement is provided, and the natural FedAvg baseline is missing. In my view, addressing these issues would require substantial new experiments and analysis, effectively a different paper. I would not encourage resubmission unless the authors provide a genuine communication-reduction mechanism (e.g., sparsification or quantization), measure actual bytes exchanged including the server broadcast, and compare against standard FedAvg under heterogeneous data splits."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: this paper's central claim is false on its own terms. GradualDiff-Fed is FedAvg with full participation and equal weights, rewritten as parameter differences. Eq. (2) in Section III is algebraically identical to the FedAvg update the authors cite, and Algorithm 2 adds no compression, sparsification, or quantization. A dense delta has exactly the same bit cost as the full model, so 'transmitting only the difference' does not reduce communication. The paper even notes that the server sends the full global model to every client each round; that broadcast alone is already one model transfer per client.\n\nWhat the paper does well: the writing is clear, the FL background is accurate, and the LoRA fine-tuning setup is standard. The related work section cites the relevant federated LoRA methods (FedIT, FLoRA) and recognizes that naive LoRA averaging has known issues. The experimental setup is reproducible in principle: llama2-7b, LoRA rank 64, 15 rounds on a 100k-token mental health dataset, with training loss, perplexity, BLEU, and per-round time reported.\n\nThe soft spots are not minor. No FedAvg baseline is run, so the claimed advantage over standard FL is untested. Communication overhead is never measured in bytes; Table IV is wall-clock time on a single GPU, and the 36% reduction compares federated LoRA training against centralized full-model processing, which confounds model size, optimizer, and hardware. The loss numbers in Table II show FL beating centralized training by 26-29%, which is suspicious given the same data and should have triggered a sanity check. There is a single run, no error bars, and no code release. The paper's own conclusion admits non-IID and privacy leakage are future work.\n\nIs there a new result hiding here? No. The delta-average is a notational variant of FedAvg, and the efficiency claim collapses under the definition in Eq. (1). This is not a paper with a fixable flaw; the central contribution is a rename.\n\nWho is this for? Someone teaching FL basics might use it as a cautionary example, but it does not deserve referee time. I would desk reject.","headline":"GradualDiff-Fed is FedAvg in delta notation, and its communication-saving claim collapses because dense deltas do not reduce transmitted bits; desk reject.","tokens_in":8111,"tokens_out":2542,"would_cite":false,"duration_ms":24862,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GradualDiff-Fed's central claim is that federated LLM fine-tuning can upload only model-weight differences and still match centralized training.","keywords":["GradualDiff-Fed","federated learning","large language models","communication efficiency","model difference updates","Low-Rank Adaptation","parameter-efficient fine-tuning","privacy-preserving machine learning"],"falsifier":"Measure the bytes actually uploaded per round under GradualDiff-Fed and under standard FedAvg with the same LoRA and quantization settings; if the per-round payloads are the same size, the central claim of reduced communication overhead is falsified. An even simpler check is Algorithm 2's step 5, which sends $\\Delta \\mathrm{LLM}_i^{(t)}$ as a dense weight vector whose serialized size equals the serialized size of $\\theta_i^{(t)}$ unless an additional compression step is added.","tokens_in":6959,"feed_emoji":"🤖","tokens_out":17015,"duration_ms":150572,"temperature":0.7,"pith_summary":"GradualDiff-Fed is a federated learning scheme for fine-tuning large language models: data stays on each client, and the server coordinates training. Instead of uploading a full locally trained model, each client computes the element-wise difference between its updated weights and the current global model and uploads only that difference. The server averages the differences and adds the average back to the global model. The paper reports that on a mental-health chatbot task with a 7-billion-parameter model, this scheme matches or slightly beats centralized training on training loss and BLEU, and it claims a large reduction in communication overhead. If that claim holds, the method would let privacy-sensitive organizations collaboratively fine-tune large models without shipping raw data or full model weights.","feed_headline":"Weight-delta uploads match centralized language-model fine-tuning","feed_subtitle":"GradualDiff-Fed sends only model-weight differences and still reaches central-level loss on a 7B-parameter chatbot task.","key_machinery":"The load-bearing object is the delta parameterization: the difference between a client's locally fine-tuned Low-Rank Adaptation (LoRA) weights and the global model. Because LoRA freezes the base model and trains only low-rank adapter matrices, the delta lives on the adapter weights and is transmitted after 4-bit quantization. The server's update is a two-step operation: average the received deltas, then add the average to the global model, avoiding the averaging of full model snapshots. This delta-plus-addition rule is what the paper credits for making federated fine-tuning of billion-parameter models more scalable.","core_discovery":"The central claim is that the delta vector $\\Delta \\mathrm{LLM}_i^{(t)} = \\theta_i^{(t)} - \\mathrm{LLM}_g^{(t)}$ carries all the information the server needs, so transmitting the delta rather than the full model $\\theta_i^{(t)}$ keeps federated aggregation working while cutting communication. The proposed update extends the standard FedAvg averaging rule: the server computes $\\mathrm{LLM}_g^{(t+1)} = \\mathrm{LLM}_g^{(t)} + \\frac{1}{K}\\sum_{i=1}^K \\Delta \\mathrm{LLM}_i^{(t)}$, averaging client deltas and applying the result to the shared model. With Low-Rank Adaptation and 4-bit quantization, 15 rounds of GradualDiff-Fed reach training loss 0.22 at batch size 4 and 0.11 at batch size 8, compared with centralized training's 0.31 and 0.15; the BLEU score (n-gram overlap with reference text) is identical at 0.55, and perplexity (prediction uncertainty) is 16.78 versus central's 15.36. The paper interprets these results as showing that delta-based communication is an efficient, privacy-preserving way to fine-tune large models from distributed data without compromising performance.","pith_inferences":["The paper never measures bytes exchanged per round; because $\\Delta \\mathrm{LLM}_i^{(t)}$ is a dense vector of the same dimension as $\\theta_i^{(t)}$, the promised communication savings depend on an unstated compression or sparsification step, and measuring payload size with and without such a step would settle the claim.","The reported 36% faster per-round time is total elapsed time, not a measure of communication bytes, so it may reflect training with LoRA, 4-bit quantization, or fewer compute operations rather than the delta transmission itself.","The paper's own conclusion leaves unevenly distributed (non-IID) client data and privacy leakage through model differences as future work; both are natural stress tests, since averaged deltas could still expose information about local data."],"forward_implications":["In the tested configuration, GradualDiff-Fed reaches lower final training loss than centralized training (0.11 versus 0.15 at batch size 8) while staying within about 1.4 perplexity points of the central model.","Clients never send full model snapshots to the server, so the server's aggregation step is reduced to averaging deltas and adding them to the global model.","The framework relies on all selected clients participating each round; the paper presents this synchronous aggregation as a factor in its smooth loss reduction.","With Low-Rank Adaptation and 4-bit quantization, the method fits a 7-billion-parameter model on a single consumer GPU, making local training on ordinary hardware part of the scheme.","If the central communication claim is correct, the same delta-update pattern could be applied to other large-model fine-tuning tasks where uplink bandwidth is the bottleneck."],"supporting_citations":[{"why":"Supplies the federated learning formulation and the FedAvg update rule that GradualDiff-Fed adapts, and serves as the baseline against which the method positions itself.","marker":"[5]"},{"why":"Cited as the FedAvg aggregation algorithm whose averaging-and-update structure GradualDiff-Fed modifies.","marker":"[7]"},{"why":"Supplies Low-Rank Adaptation, the parameter-efficient fine-tuning method that lets a 7-billion-parameter model be trained on a single GPU by freezing base weights and training low-rank adapters.","marker":"[11]"}],"fun_headline_variants":["Send weight diff, not full model: federated LLM fine-tuning","Federated LLM training: only weight deltas, same accuracy","Weight-delta uploads cut communication, keep BLEU and perplexity","GradualDiff-Fed: delta-only FL matches central fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the transmitted difference $\\Delta \\mathrm{LLM}_i^{(t)}$ costs fewer bits than the full model, but the paper does not specify compression or sparsification for this dense vector, so the promised communication reduction is not demonstrated.","fun_headline_variants_meta":{"raw":{"variants":["Send weight diff, not full model: federated LLM fine-tuning","Federated LLM training: only weight deltas, same accuracy","Weight-delta uploads cut communication, keep BLEU and perplexity","GradualDiff-Fed: delta-only FL matches central fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001213,"raw_usage":{"total_tokens":5024,"prompt_tokens":1009,"completion_tokens":4015,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":3936}},"tokens_in":625,"tokens_out":4015,"duration_ms":28999,"temperature":1.0,"reasoning_tokens":3936,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:35:19.002895+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the bytes actually uploaded per round under GradualDiff-Fed and under standard FedAvg with the same LoRA and quantization settings; if the per-round payloads are the same size, the central claim of reduced communication overhead is falsified. An even simpler check is Algorithm 2's step 5, which sends $\\Delta \\mathrm{LLM}_i^{(t)}$ as a dense weight vector whose serialized size equals the serialized size of $\\theta_i^{(t)}$ unless an additional compression step is added.","supporting_citations":[{"cited_title":"Communication-efficient learning of deep networks from de- centralized data,","cited_arxiv_id":null,"evidence_quote":"Supplies the federated learning formulation and the FedAvg update rule that GradualDiff-Fed adapts, and serves as the baseline against which the method positions itself."},{"cited_title":"Testing the Robustness of a BiLSTM-based Structural Story Classifier","cited_arxiv_id":"2201.02733","evidence_quote":"Cited as the FedAvg aggregation algorithm whose averaging-and-update structure GradualDiff-Fed modifies."},{"cited_title":"Lora: Low-rank adaptation of large language models,","cited_arxiv_id":null,"evidence_quote":"Supplies Low-Rank Adaptation, the parameter-efficient fine-tuning method that lets a 7-billion-parameter model be trained on a single GPU by freezing base weights and training low-rank adapters."}],"review_version":1}