{"id":"7bca6f74-2e54-48a7-b0d8-e5dfbf552bd0","arxiv_id":"2412.13551","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A hybrid blockchain federated learning framework with Q-learning agents and LoRA-based unlearning is proposed, but the experiments do not show that model utility survives data removal.","lead":"This paper proposes a system that combines public and private blockchains, Q-learning agents, and LoRA-based unlearning to let organizations train large language models together without sharing data. A smart generalist might read it to see whether privacy-preserving collaboration can keep model quality while allowing data removal.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The unlearning experiments never measure retain-set utility, and reported final accuracies of 0.95-1.25% on IMDB contradict 'without compromising overall performance' under the natural reading.","rationale":"The reader identified Algorithm 6's unverified selective-forgetting assumption as the load-bearing weakness. I agree that is the right place to look, but the concern is sharper than 'unverified': the paper's own reported numbers appear to contradict the central claim. Under the natural reading of Section VII.A, the 'Final Accuracy' after unlearning is the model's classification accuracy, and values below 1.3% on IMDB mean the model is essentially destroyed, not preserved. Under the alternative reading that this is forget-set accuracy, the paper still omits the one measurement that would establish utility preservation: retain-set accuracy. Either way, the reported tables cannot support the abstract's assertion that the unlearning mechanism removes specific data contributions 'without compromising the model's overall performance.' The comparison with retrain-from-scratch is also unhelpful because only a single final scalar is given, not the retain-set accuracy that would make the comparison meaningful. Algorithm 6 is described at the level of a loop skeleton: it never defines the loss being maximized or minimized, never states how LoRA's low-rank subspace is targeted at forgetting, and never includes a retain-set penalty. Algorithm 7's verification criteria are undefined, and no membership-inference or privacy-leakage test is run. These are not merely missing niceties; they are the evidence needed for the paper's headline claim. I would therefore keep the reader's REJECT verdict. I am marking partial rather than full agreement because the reader framed the problem as an unverified assumption, whereas the stronger issue is that the presented experimental numbers, if taken at face value as post-unlearning model accuracy, actively falsify the claim; if they are forget-set accuracy, the paper's metric is ambiguous and the required retain-set measurement is absent.","tokens_in":1069,"tokens_out":1549,"duration_ms":51743,"concrete_test":"Re-run the IMDB experiment from Table I row 4 (r=16, alpha=4, dropout=0.2) with Algorithm 6 exactly as specified, and report three numbers on three disjoint splits: accuracy on D_forget, accuracy on a held-out retain set disjoint from D_forget, and accuracy on the original test set. Also run a membership-inference attack (e.g., LiRA) comparing D_forget against retain points, and compare against retrain-from-scratch on the same splits. If retain/test accuracy stays near the 98.7% initial value while D_forget accuracy drops toward chance, the selective-unlearning claim is supported. If retain/test accuracy collapses to the reported ~1.2% final value, the claim is falsified and the 'Final Accuracy' column in Tables I-III is not a utility-preserving metric.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires Algorithm 6 to forget D_forget while preserving utility on D \\ D_forget. The paper never measures the latter. Tables I-III report only 'Initial Accuracy' and 'Final Accuracy'; if final accuracy is the model's accuracy after unlearning, as Section VII.A's wording suggests, values of 0.95-1.25% on IMDB are catastrophic utility collapse, not 'without compromising overall performance.' If final accuracy is instead accuracy on the forget set, then it only evidences forgetting and says nothing about retain-set or held-out performance; the retrain-from-scratch comparison in Table III likewise reports only one scalar, not the required retain-set accuracy. Algorithm 6 itself has no formal objective: lines 5-7 say to compute and apply gradients 'emphasizing' D_forget, but specify neither a loss direction (ascent or descent) nor any retain-set regularization term, so there is no mechanism in the construction that guarantees preservation. The verification step (Algorithm 7) evaluates only an unspecified 'training loss and accuracy' on D_val, with undefined criteria, and no membership-inference check is performed. Thus the load-bearing premise, selective erasure with preserved utility, is not just unverified; the reported numbers actively conflict with it under the natural reading.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a hybrid blockchain-based federated learning framework for large language models, combining public and private blockchains, multi-agent Q-learning, and a LoRA-based unlearning mechanism. The stated goal is to enable transparent model-update sharing on a public chain while protecting sensitive computations in private chains, and to support selective data removal without compromising overall model performance. The manuscript presents eight algorithms describing registration, model upload, private-chain training, unlearning, verification, and aggregation, followed by experiments on IMDB and Twitter datasets that report initial and final accuracies for several LoRA configurations, a retrain-from-scratch comparison, and a blockchain overhead analysis. The central claim about unlearning is not supported by the reported results: the final accuracies collapse to near-zero values, and no retain-set or held-out utility metric is reported.","tokens_in":19719,"tokens_out":3402,"duration_ms":31942,"significance":"If the central claims were validated, the framework could be a useful blueprint for privacy-aware, trust-enhanced collaborative LLM training in regulated domains such as healthcare and education. The paper provides a fairly complete architectural description with numbered algorithms and a time-cost analysis of the hybrid blockchain, which are useful system-building blocks. However, the core novelty—LoRA-based unlearning—is specified only as a high-level gradient-adjustment loop with no formal objective, and the experimental evidence directly contradicts the claim that unlearning does not compromise overall performance. No code, machine-checked proofs, or parameter-free derivations are provided. As submitted, the significance of the contribution is substantially weakened because the key performance claim is unverified and, under the natural reading of the reported tables, refuted.","major_comments":[{"comment":"The reported final accuracies of 0.95%–1.25% on IMDB and 7.93%–9.04% on Twitter, against initial accuracies of 74%–98%, are catastrophic utility loss if 'Final Accuracy' means accuracy on the general test set after unlearning. This directly contradicts the abstract's claim that unlearning enables 'selective removal of specific data contributions without compromising the model's overall performance.' If 'Final Accuracy' instead means accuracy on the forget set, then the tables only demonstrate forgetting and provide no evidence about the model's behavior on the retained data or held-out data. The paper never defines which interpretation is intended and never reports retain-set accuracy, so the central claim is empirically unsupported.","section":"Section VII-B, Tables I and II"},{"comment":"The unlearning objective is underspecified. Lines 5–8 direct the reader to compute gradients 'emphasizing' the forget set and to adjust the LoRA adapter, but they do not specify whether the loss is minimized or maximized, nor do they include any retain-set regularization term. Consequently, there is no mechanism in the algorithm construction that guarantees preservation of performance on D \\ D_forget. The subsequent verification step in Algorithm 7, line 6, evaluates only an unspecified 'training loss and accuracy' on D_val with no defined acceptance criteria and no membership-inference test, so it cannot establish that the forget data has been selectively removed while other data remains intact.","section":"Algorithm 6, Section V-G"},{"comment":"The comparison with 'Retrain from Scratch' is not informative. The baseline final accuracies are themselves near zero (0.85% on IMDB, 7.84% on Twitter), and the table reports only a single final-accuracy scalar, not retain-set or held-out accuracy. Moreover, the initial accuracies for the baseline differ from the corresponding initial accuracies of the proposed method (e.g., 95.60% versus 97.10% on IMDB), so the comparison is not on an equal footing. These issues make it impossible to conclude that the proposed unlearning approach achieves performance comparable to retraining.","section":"Section VII-B, Table III"},{"comment":"The paper dismisses all existing baselines by stating that 'there are no direct counterparts in the current literature for comparison,' yet the related-work section discusses multiple unlearning methods for LLMs, including gradient-ascent-based approaches and selective knowledge negation. Without any comparison to these established unlearning baselines, and without the retain-set utility metric that those baselines report, the conclusion that the proposed method exhibits 'significant advantages in terms of computational efficiency, versatility, and adaptability' is not supported by the presented evidence.","section":"Section VII-A, 'Experimental Comparisons'"}],"minor_comments":[{"comment":"The case studies in Section V-J contain inconsistent algorithm references: the text cites Algorithm 5 for private-chain training and Algorithm 4 for the unlearning process, but the actual unlearning algorithm is Algorithm 6 and the private-chain training algorithm is Algorithm 4. This mismatching makes the case-study walkthrough difficult to follow.","section":"Section V-J"},{"comment":"The text contains several grammatical and typographical errors, such as 'the way computers understand and process human language, but using them effectively across different organizations remains still difficult' and 'the public data may consumed up while private data from organizations are highly needed.' These errors should be corrected throughout.","section":"Abstract and Section I"},{"comment":"In Algorithm 1, a JSON Web Token is defined as a public–secret key pair ('jwt ← Pk, Sk'), which conflates key generation with token creation; a JWT should be a signed token issued after authentication, not the key pair itself. This should be clarified.","section":"Algorithm 1"},{"comment":"The security analysis claims that the framework uses 'threshold signatures and zero-knowledge proofs,' but no cryptographic protocol or implementation detail is provided anywhere in the paper, and these mechanisms are not referenced in any algorithm. The security claims therefore outrun the presented system design.","section":"Section VI-B"},{"comment":"The captions of Figures 2–7 do not define the axes or explain what 'Accuracy' refers to, and the text does not specify whether the plotted values are forget-set accuracy, retain-set accuracy, or overall test accuracy. Additionally, the 'Setup Time' bullet in Section VII-B reports 48 seconds for hybrid blockchain setup, whereas Table IV lists 84 seconds at t=0 for the hybrid method; these numbers need to be reconciled.","section":"Figures 2–7 and Table IV"}],"recommendation":"reject","confidential_remarks":"The manuscript's experimental section is the main barrier to acceptance. The unlearning results, as reported, show complete utility collapse, and the paper does not provide the retain-set accuracy needed to evaluate the central claim. This is not a matter of presentation; the evidence is inconsistent with the paper's core assertion. A revision would require a re-definition of the evaluation protocol, new experiments measuring retain-set and held-out utility, a specification of the unlearning loss direction, and comparisons with existing unlearning baselines. Given the scope of these gaps, I cannot recommend major revision as a path within the current manuscript's framing. The high-level blockchain architecture may still be of interest, but it would need to be presented as a system description rather than as a validated solution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is the combination: hybrid public/private blockchain, multi-agent Q-learning for participation strategy, and LoRA-based unlearning in a single federated LLM framework. That integration is coherent, and the system model is laid out in enough detail to be implementable. The case studies do a reasonable job of illustrating how the pieces would fit in education and healthcare settings. Credit where it is due: the authors identify a real gap in cross-organizational LLM training—trust, privacy, and the right to be forgotten—and their architecture is a serious attempt to address all three at once.\n\nThe soft spots are not minor. The paper's central claim is selective unlearning \"without compromising the model's overall performance,\" but the experiments never measure that. Tables I and II report only Initial Accuracy and Final Accuracy. Under the natural reading that Final Accuracy is the model's accuracy after unlearning, values of 0.95–1.25% on IMDB and 7.93–9.04% on Twitter are catastrophic utility collapse, not preservation. If those numbers instead measure accuracy on the forget set, then retain-set utility is never reported, and the retrain-from-scratch comparison in Table III compares only that same undefined scalar. Either way, the evidence does not support the claim. Algorithm 6 also has no formal objective: it says to compute gradients \"emphasizing\" the forget set but specifies neither ascent nor descent, and it has no retain-set regularization term. The verification step in Algorithm 7 evaluates an undefined \"training loss and accuracy\" against unspecified criteria. There is no membership-inference check and no comparison with established unlearning baselines. There are also internal inconsistencies—the case studies cite the wrong algorithm numbers for unlearning and aggregation. These are fixable, but they add to the impression of an unfinished manuscript.\n\nThe security analysis is qualitative and largely generic (PoS game theory, ZKPs, threshold signatures) without being tied to the actual construction, so it does not rescue the evaluation. Still, the framework is not worthless: the architecture is plausible and the combination is genuinely novel in the cited literature, even if each ingredient is known. The paper would benefit from a serious referee, not a desk reject, because the authors have a real problem and a real design. But my own verdict would be reject or major revision: they need to define their metrics, report retain-set accuracy, specify the unlearning objective, and compare against baselines. As is, I would not cite it.\n\nRecommendation: send to review, expect heavy revision, and be prepared to reject if the evaluation is not substantially reworked.","headline":"A plausible systems combo that badly overclaims: the unlearning experiments never measure retain-set utility and the reported accuracies contradict the central promise.","tokens_in":20228,"tokens_out":1955,"would_cite":false,"duration_ms":18762,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a hybrid public/private blockchain federated-learning framework lets organizations co-train large language models without sharing raw data, and that a LoRA-based unlearning loop removes specific data contributions…","keywords":["federated learning","large language models","machine unlearning","blockchain","low-rank adaptation (LoRA)","multi-agent reinforcement learning","privacy","cross-organizational collaboration"],"falsifier":"Run the unlearning loop on a dataset split into forget and retained portions, then measure accuracy on the retained portion immediately afterward. If retained accuracy falls toward the reported forget-set levels (0.95–1.25% on IMDB and roughly 8% on Twitter, both near chance for these tasks), the claim that selective unlearning preserves overall performance is falsified. A follow-up probe that asks whether the model still behaves differently on the supposed-to-be-forgotten examples than on never-seen examples would settle whether the target data is actually gone.","tokens_in":19224,"feed_emoji":"🔗","tokens_out":11188,"duration_ms":92322,"temperature":0.7,"pith_summary":"Federated learning lets organizations train a shared model without handing over raw data, but it assumes honest participants and an honest aggregator, and it offers no way to honor a data-removal request once that data's influence is inside the model. The paper tries to close both gaps at once: a hybrid blockchain records every model update on a public ledger while private chains keep sensitive local computations hidden, and a LoRA-based unlearning mechanism removes selected data contributions without retraining from scratch. The intended payoff is a regulation-ready setting where hospitals, universities, or companies can co-train large language models, withdraw data on request, and still keep a model that performs well. Experiments on IMDB and Twitter sentiment tasks report final post-unlearning accuracies close to retraining from scratch, with the blockchain layers adding only seconds of overhead per round.","feed_headline":"Hybrid blockchain adds on-demand data erasure to LLM training","feed_subtitle":"Public chains record updates, private chains shield data, and LoRA removes targeted contributions without a full retrain.","key_machinery":"The argument turns on three working parts. The hybrid ledger uses a public blockchain as an immutable transaction log for model updates and unlearning submissions, and private data collections with endorsement policies restrict sensitive computations to authorized organizations. The LoRA forget adapter is a lightweight low-rank adaptation attached to the model: forward and backward passes on the forget set $D_{\\text{forget}}$ run for $E_u$ epochs, and the adapter's adjusted gradients produce a small parameter delta that stands for the unlearning and is what gets verified and uploaded. The multi-agent Q-learning layer gives each organization a Q-table over participation states and actions so resource contributions are chosen to maximize long-term reward. Smart contracts tie these together by checking JWT-based identities, aggregating private-chain models at set epochs, and applying validation criteria before any update or unlearning result is committed to the chain.","core_discovery":"The paper's central claim is that trust, privacy, and regulatory compliance can be solved together rather than traded off in cross-organizational LLM training. Its hybrid blockchain architecture keeps a public, immutable record of model updates and transactions so that contributions are transparent, while private blockchain channels keep sensitive computations and data hidden from outsiders. Each organization acts as a Q-learning agent that decides when and how much to contribute, aligning individual incentives with the collective model. When an organization requests data withdrawal, the framework avoids full retraining: it runs a short LoRA-parameterized forgetting loop on the data to be removed, extracts the resulting parameter changes, verifies them against a validation set in a smart contract, and records the unlearning transaction on the public chain. The experiments argue that this achieves forgetting comparable to retraining from scratch at much lower cost, with the hybrid ledger adding roughly six seconds of consensus overhead and about 48 seconds of one-time setup.","pith_inferences":["The reported metric is accuracy on the forget set after unlearning, which shows that something was forgotten but not that only the target data was forgotten; a retained-data accuracy column and a membership probe on the forget set would be needed to confirm selective erasure.","The same LoRA-forgetting loop could be attached to other parameter-efficient fine-tuning methods, and the smart-contract verification step could be strengthened with zero-knowledge proofs so validation does not expose the validation data.","Because the reported forget-set accuracies on IMDB, a binary task, are near chance, the decisive test is whether retained-data utility survives; adding a retain-set evaluation would turn this from a claim about forgetting into a claim about selective forgetting.","The multi-agent Q-learning layer is largely independent of the blockchain and the unlearning loop, so its contribution to trust and incentive alignment could be ablated separately; the paper does not isolate it."],"forward_implications":["Organizations with competing interests can co-train a shared LLM while keeping raw data in-house: the public chain provides tamper-evident records of every update, and the private chains restrict sensitive computations to authorized peers.","A data-removal request can be handled without a full retraining run; the LoRA forgetting loop produces a parameter delta that is verified and committed on-chain, which is the paper's mechanism for GDPR-style erasure requests.","The hybrid ledger adds only modest overhead, about 6 seconds of consensus time and 48 seconds of one-time setup, so the transparency and compliance benefits do not require sacrificing training throughput.","Because LoRA hyperparameters (rank, alpha, dropout) control how much accuracy falls on the forget set, the framework can be tuned to trade unlearning strength against retained utility.","Each organization's participation strategy is chosen by Q-learning against its own utility, aligning individual incentives with the collective goal while the ledger keeps everyone accountable."],"supporting_citations":[{"why":"Supplies the Low-Rank Adaptation method the unlearning loop uses to update only a small adapter rather than the full model.","marker":"[8]"},{"why":"Provides the blockchain-enabled federated learning survey that motivates the hybrid public/private ledger design.","marker":"[9]"},{"why":"Supplies a blockchain-enabled federated learning architecture that the paper adapts for private-chain and public-chain aggregation.","marker":"[10]"},{"why":"Establishes the GDPR right-to-be-forgotten requirement that the unlearning mechanism is designed to satisfy.","marker":"[7]"},{"why":"Provides a prior differentially private machine-unlearning method for federated learning that the proposed LoRA-based unlearning extends.","marker":"[2]"},{"why":"Defines the gradient-ascent style of LLM unlearning that the paper contrasts with its LoRA-based forgetting loop.","marker":"[17]"}],"fun_headline_variants":["Blockchain federated LLM adds on-demand unlearning","Hybrid blockchain enables selective data erasure in federated LLMs","Federated LLMs get trust, privacy, and unlearning via blockchain","LoRA unlearning on public chains cuts retraining cost in federated LLM","Q-learning agents optimize participation in blockchain federated LLM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a few rounds of gradient updates on the data to be erased, routed through a small low-rank adapter, remove exactly that data's influence while leaving the model's behavior on all other data intact; the paper reports no accuracy measurement on the data that should be kept, so this premise is assumed rather than shown.","fun_headline_variants_meta":{"raw":{"variants":["Blockchain federated LLM adds on-demand unlearning","Hybrid blockchain enables selective data erasure in federated LLMs","Federated LLMs get trust, privacy, and unlearning via blockchain","LoRA unlearning on public chains cuts retraining cost in federated LLM","Q-learning agents optimize participation in blockchain federated LLM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000448,"raw_usage":{"total_tokens":2273,"prompt_tokens":972,"completion_tokens":1301,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":1209}},"tokens_in":588,"tokens_out":1301,"duration_ms":9273,"temperature":1.0,"reasoning_tokens":1209,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:00:54.142332+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the unlearning loop on a dataset split into forget and retained portions, then measure accuracy on the retained portion immediately afterward. If retained accuracy falls toward the reported forget-set levels (0.95–1.25% on IMDB and roughly 8% on Twitter, both near chance for these tasks), the claim that selective unlearning preserves overall performance is falsified. A follow-up probe that asks whether the model still behaves differently on the supposed-to-be-forgotten examples than on never-seen examples would settle whether the target data is actually gone.","supporting_citations":[{"cited_title":"Fedtwin: Blockchain- enabled adaptive asynchronous federated learning for digital twin net- works,","cited_arxiv_id":null,"evidence_quote":"Supplies a blockchain-enabled federated learning architecture that the paper adapts for private-chain and public-chain aggregation."},{"cited_title":"General data protection regulation (gdpr),","cited_arxiv_id":null,"evidence_quote":"Establishes the GDPR right-to-be-forgotten requirement that the unlearning mechanism is designed to satisfy."}],"review_version":1}