{"id":"1f973117-364a-4cd1-9640-d5bedbd2cdbf","arxiv_id":"2412.05830","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A softmax-weighted model merging method combines multiple LLM-based link stealing attack models into one model that beats existing merging baselines across four graph datasets.","lead":"This paper describes a way for several attackers to combine their individually fine-tuned language models into one unified model that steals hidden links from graph neural network predictions, without sharing their private data. The authors report strong cross-dataset attack results and a new model merging scheme, though the scheme's weights are partly chosen using the datasets it is tested on.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lambda weights in Eq. (9) are tuned on the same datasets used for evaluation (Table IV); the out-of-domain and cross-dataset generalization claims need re-testing without target-label leakage.","rationale":"The reader's weakest_assumption identified exactly this issue, and I agree. The paper's own setting rules out target-label availability for out-of-domain data, yet Eq. (9) requires accuracy scores on those datasets. The empirical advantage in Table V is therefore not established as a property of the merging algorithm. I do not see a more load-bearing flaw: the LLM-based single-dataset result is plausible, and the Drop/Elect/Merge steps are standard magnitude/sign operations. The theoretical analysis is informal, but the empirical claim was always the main contribution. If the leakage-free recomputation preserves the gains, the paper's conditional acceptance is justified; if not, the central claim fails. Hence no change to the reader's CONDITIONAL verdict is needed.","tokens_in":23509,"tokens_out":6326,"duration_ms":60740,"concrete_test":"Re-run Section VI-G (out-of-domain) and Table V with lambda computed strictly from each attacker's own training/validation split, with no accuracy information from the target dataset. Concretely, for the Cora held-out experiment, merge the Citeseer, Pubmed, and Ogbn-arxiv models using lambda derived from the 3x3 accuracy matrix on only those three datasets (rows/columns Citeseer/Pubmed/Ogbn-arxiv), then evaluate on Cora. If the merged model no longer matches or exceeds the best individual model and the mean accuracy gap over unweighted baselines narrows, the claimed generalization is an artifact of target-label leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim--superior generalization to unseen datasets--rests on the merging weights lambda in Eq. (9). In Section VI-C, lambda is computed from Table IV, which lists each single-dataset-trained model's accuracy on all four datasets (Cora, Citeseer, Pubmed, Ogbn-arxiv), including the datasets later used for evaluation in Table V and Fig. 11. This is a train-on-test leak: the weights are chosen using the very labels whose prediction is the claimed contribution. The black-box attack setting (Section III) says attackers have no knowledge of the target model's structure or parameters, and Section VI-G calls the held-out dataset 'unknown to the attackers.' Under that premise, the accuracies in the target column are unavailable, so lambda cannot be computed as written. If the authors instead compute lambda from a 3x3 submatrix excluding the target (for Fig. 11), the paper does not say so and does not report the resulting weights or whether the advantage over unweighted Della/DARE baselines persists. Without a leakage-free weight-selection procedure, the headline 'surpassing all existing methods' (Section VI-D) may reflect test-set information rather than genuine merging ability.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a link stealing attack on graph neural networks in which multiple attackers collaboratively build a single attack model. Each attacker fine-tunes a large language model (LLM) on their own dataset, then the resulting models are merged via a novel Drop-Elect-Merge procedure. The merged model is claimed to attack multiple datasets simultaneously and to generalize to previously unseen out-of-domain datasets. Experiments on Cora, Citeseer, Pubmed, and Ogbn-arxiv compare the method against non-LLM link stealing attacks and against existing model merging baselines (Mean, Ties, DARE, Della). The main novelty is the combination of LLM-based cross-dataset attacks with weighted parameter merging, where the weight lambda in Eq. (9) is derived from each model's accuracy across datasets.","tokens_in":23765,"tokens_out":6560,"duration_ms":61204,"significance":"If the results hold, this would be a novel and practically relevant contribution: it demonstrates a realistic collaborative attack model where attackers share only parameters, not data, and it is the first to apply LLMs to link stealing across datasets. The use of known merging techniques (DARE, Della) with an additional weighting step is a reasonable design idea, and the paper includes experiments across multiple GNN and LLM architectures. However, the central empirical claim is currently undermined by a circularity in the weight-allocation scheme: the lambda weights in Eq. (9) are computed from accuracy values on the very datasets later used for evaluation. Because the reported superiority over baselines may be an artifact of fitting lambda to the test labels, the paper's core contribution is not convincingly established as written.","major_comments":[{"comment":"The merging weights lambda in Eq. (9) are computed from the accuracy matrix in Table IV, which evaluates every single-dataset model on all four datasets. These same datasets are used as the test sets for the merged model in Table V and Fig. 11. This is a train-on-test leak: the weights are fitted to the labels whose prediction is claimed as the contribution. The paper must either compute lambda on a held-out validation split that excludes all test nodes, or show that the advantage over unweighted merging and over the DARE/Della baselines persists when lambda is chosen without access to the target datasets. Without this, the headline claim of 'surpassing all existing methods' in Section VI-D is not supportable.","section":"Section VI-C, Eq. (9), Tables IV and V"},{"comment":"The out-of-domain experiment is described as merging models trained on three datasets and attacking the remaining one. However, the paper does not state whether the target dataset's column is excluded when computing the lambda weights for the three merged models. If the full Table IV matrix, including the target column, is used to compute weights, then the 'out-of-domain' evaluation still depends on target labels through Eq. (9), making the generalization claim circular. The authors should report the exact submatrix used, the resulting lambda values, and a comparison against a fixed-weight (e.g., equal-weight) merging baseline on the same held-out target.","section":"Section VI-G, Fig. 11"},{"comment":"Fine-tuning details are almost entirely absent: no optimizer, learning rate, batch size, number of epochs, LoRA/QLoRA configuration, or sequence truncation procedure is reported for any experiment. In addition, Ogbn-arxiv has 128-dimensional numeric node features (Table II), but the prompt design in Fig. 3 is based on 'Title, Abstract'; the paper never explains how the numeric features of Ogbn-arxiv are textualized or otherwise fed to the LLM. These omissions prevent replication and leave the cross-dataset mechanism unclear for a main experimental dataset.","section":"Section IV-B, Section VI-A, Table II"},{"comment":"The theoretical analysis is informal. Equations (13)-(15) only show expectation invariance for a single linear layer and do not account for the Elect masking in Eq. (8) or for the nonlinearity of Transformer blocks. The conclusion that 'Drop and Scaling maintain attack effectiveness' is therefore not established for the actual model, and the statement in Section VIII that the paper provides 'thorough theoretical analysis' overstates what is proven. A precise formal claim, or a removal of the 'thorough' characterization, is needed.","section":"Section V"}],"minor_comments":[{"comment":"The cross-entropy loss formula has mismatched parentheses: it should be written as L_CE = -[y log y_hat + (1-y) log(1-y_hat)].","section":"Eq. (5)"},{"comment":"The LLM response text 'Yes or False' should be 'Yes or No'.","section":"Fig. 3"},{"comment":"The text refers to 'as illustrated in Fig. 4' when discussing the effect of drop probability p; the correct reference is Fig. 10(a).","section":"Section VI-E1"},{"comment":"The sentence 'As shown in the table' following the cross-dataset heat map should read 'As shown in the figure' because the results are in Fig. 7, not a table.","section":"Section VI-B2"},{"comment":"The first row, labeled 'Target (Max)', is not a merging method but an upper-bound reference; it should be visually separated from the actual merging baselines to avoid confusion.","section":"Table V"},{"comment":"Reference [7] is the authors' own prior work on LLM-based link stealing; it should be explicitly discussed in Section VII-B so that the reader can see how the current paper extends it.","section":"Related Work, Section VII-B"}],"recommendation":"major_revision","confidential_remarks":"The core issue is the circularity in Eq. (9), where lambda is fitted to the evaluation datasets. This is fixable in principle by re-running the merging-weight selection on a proper validation split, but as written the experimental claims are not trustworthy. The paper also omits fine-tuning hyperparameters and code, which is a reproducibility concern for a journal submission. I also note that reference [7] appears to be a closely related prior work by the same group; the relationship should be made explicit in the revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's actual new piece is the setting: multiple attackers each fine-tune an LLM on their own graph dataset and then merge parameters to attack not only those datasets but also unseen ones. That is a reasonable extension of link stealing, and the LLM-as-universal-interface idea for cross-dataset attacks is worth taking seriously. The experiments are fairly broad – four datasets, three GNN targets, three LLM backbones, plus an out-of-domain evaluation – and the numbers are reported with means and standard deviations. Credit where due: this is more empirical work than most attack papers bother with.\n\nThe soft spot is load-bearing. The lambda weights in Eq. (9) are computed from each single-dataset model's accuracy on the very datasets that Table V later uses for evaluation. In the stated black-box setting, attackers do not know link labels on the target dataset – that is what they are trying to steal. So the weights as written require exactly the information the attack is supposed to infer. The out-of-domain experiment in Fig. 11 is less guilty, since the target dataset is excluded from merging, but the paper never says how lambda is recomputed there, and the text refers back to the same Table IV. Without a clear statement that lambda is computed only from data available to the attackers, the \"surpassing all existing methods\" claim in Table V is not established. This is fixable – use a validation split of the attackers' own data, or a small held-out portion of known links – but it has to be fixed.\n\nOther soft spots are minor by comparison. The theoretical analysis is informal: the expectation calculation in Eq. (13)-(15) only shows that dropout-style rescaling preserves the expected output of a linear layer; it does not address the elect step or the weighted merge. Fine-tuning details (learning rates, epochs, LoRA vs full fine-tune, compute) are missing. And there is a mismatch on Ogbn-arxiv: the dataset statistics say 128-dimensional numeric features, but the prompt design uses titles and abstracts. Where does the text come from? That needs a sentence or two.\n\nThe citation pattern is fine; the authors cite DARE, TIES, and Della, and their own prior LLM link stealing paper is referenced as [7]. The merging recipe itself – magnitude drop, sign election, softmax weighting – is a combination of known components, and the weighting is the only genuinely new twist.\n\nBottom line: the paper deserves a serious referee, but the referee should demand a leakage-free weight-selection procedure and a clarification of the Ogbn-arxiv text features before the central claim is accepted.","headline":"A plausible but leaky combination of LLM-based link stealing and model merging; the headline advantage comes from weights fitted on the test datasets, so the main claim needs re-testing.","tokens_in":24283,"tokens_out":1886,"would_cite":false,"duration_ms":22228,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that merging several LLM-based link stealing attack models with its Drop-Elect-Merge method yields a single attack model that outperforms prior link stealing attacks on all four tested graph datasets and remains…","keywords":["link stealing attack","graph neural networks","large language models","model merging","cross-dataset attack","privacy attacks","parameter merging","out-of-domain generalization"],"falsifier":"Re-run the four-dataset merging evaluation but compute the merging weights $\\lambda$ using only each attacker's own held-out validation split, with no labels from the other three datasets; if the merged model's mean accuracy falls to the level of the unweighted Mean or Della baselines, the reported 93.33% advantage traces to target-aware weight selection rather than to the Drop-Elect-Merge mechanism.","tokens_in":23324,"feed_emoji":"🔗","tokens_out":12186,"duration_ms":97956,"temperature":0.7,"pith_summary":"The paper tries to establish that a group of attackers with different private graph datasets can jointly build a stronger link stealing attack against graph neural networks without sharing any data. It replaces the fixed-size features used in prior attacks with a fine-tuned large language model that consumes node text and target-model posterior probabilities, which lets one model work across datasets of different sizes and feature dimensions. It then merges the attackers' LLMs with a Drop-Elect-Merge procedure that prunes redundant parameter updates, filters out conflicting updates by sign, and weights each model by the difficulty of the dataset it was trained on. The authors report that the merged model achieves 93.33% average accuracy and 93.40% F1 score on Cora, Citeseer, Pubmed, and Ogbn-arxiv, beating both single-dataset attacks and standard merging baselines, and that it keeps near-best performance on an out-of-domain dataset none of the attackers trained on. A sympathetic reader should care because this turns link stealing from a single-attacker, single-dataset game into a realistic collaborative threat that pools knowledge while preserving each party's data privacy.","feed_headline":"Merged LLMs steal GNN links at 93.3% accuracy","feed_subtitle":"Fine-tuned attack models from separate datasets merge without data sharing, and hit unseen graphs too.","key_machinery":"The central object is the delta parameter $\\delta_t = \\theta_t - \\theta$, the difference between an attacker's fine-tuned LLM parameters and the shared pretrained base. The Drop step sorts deltas by magnitude and drops each with probability inversely proportional to its magnitude, rescaling survivors by $1/(1-d_i)$ so the expected output of the merged model is unchanged; the Elect step masks out any delta whose sign disagrees with the sign of the summed deltas across all models; the Merge step forms $\\delta'_{\\text{merge}}^k = \\sum_t \\lambda_t \\delta_t'^k$ with $\\lambda_t = \\frac{\\exp(\\text{Max}(Acc_t)-\\text{Min}(Acc_t))}{\\sum_t \\exp(\\text{Max}(Acc_t)-\\text{Min}(Acc_t))}$, where the accuracy spread is measured across the datasets the models are evaluated on. The drop-rescale step removes redundant parameters, the elect step prevents conflicting updates from canceling, and the weighted merger gives more influence to models trained on datasets that are hard for other models to attack.","core_discovery":"The paper's central claim is that the knowledge of multiple attackers can be combined into a universal link stealing attack model by fine-tuning the same pretrained LLM on each attacker's dataset and then merging the models, rather than by sharing the data itself. The claim rests on the observation that an LLM's transformer attention handles variable-length node features, so a model trained on one dataset can be applied to another, and on a merging scheme that keeps only the parameter updates that survive magnitude-biased dropping and sign-consistency voting, then sums those updates with per-model weights derived from how differently each dataset behaves under attack. With this scheme, the authors assert, the merged model is more accurate than every individual single-dataset model and every prior merging baseline they test, and it generalizes to held-out graphs.","pith_inferences":["If the method transfers, the same Drop-Elect-Merge recipe could be used for other collaborative LLM-based privacy attacks on graphs, such as membership or attribute inference, wherever several parties hold disjoint private datasets.","A testable variant would compute the merging weights from each attacker's own hold-out validation set only, with no labels from the other parties' datasets; this would show whether the reported advantage over unweighted merging survives when the target datasets are truly unknown.","Because the prompts embed the target model's posterior probabilities, the merged attack is tuned to the query behavior of the GNN family it was trained against; an unstated extension is building one merged model from posteriors of several GNN architectures and testing whether it transfers across architectures.","If the attack is as effective as reported, defenses should anticipate colluding attackers who pool heterogeneous graph data, so future defenses need to be robust to cross-dataset, model-merged adversaries rather than to a single attacker on a single dataset."],"forward_implications":["A single merged attack model can target multiple victim GNNs trained on different datasets, removing the need to train a separate attack model per dataset.","No data exchange is required among attackers, so a group of parties with private graph data can pool their attack capability by sharing only model parameters.","The LLM-based attack using node text plus posterior probabilities beats the previous feature-only, posterior-only, and inductive baselines on each of the four single-dataset evaluations.","On the four merged datasets, Drop-Elect-Merge achieves higher mean accuracy and F1 than Mean, TIES, DARE, and Della merging baselines in the paper's experiments.","When the target dataset is held out from the merge, the merged model still matches the best single model in the merge, extending the threat to previously unseen graphs."],"supporting_citations":[{"why":"supplies the original link stealing attack formulation and the Feature/PP baselines that the LLM attack must beat.","marker":"[3]"},{"why":"provides the InductiveLSA baseline for inductive GNNs that the paper compares against.","marker":"[5]"},{"why":"is the authors' earlier LLM-based link stealing attack that this work extends to cross-dataset merging.","marker":"[7]"},{"why":"supplies the Vicuna-7B and Vicuna-13B LLM architectures used for fine-tuning and merging.","marker":"[20]"},{"why":"provides the GCN target model and the Cora, Citeseer, and Pubmed datasets used in the evaluation.","marker":"[25]"},{"why":"contributes DARE, the drop-and-rescale merging baseline that the Drop step builds on.","marker":"[24]"},{"why":"contributes Della, a magnitude-based sampling merging baseline that the method is compared against.","marker":"[23]"},{"why":"contributes TIES, the sign-based merging baseline that the Elect step is compared against.","marker":"[29]"}],"fun_headline_variants":["Merged LLM attacks steal GNN links across datasets","Cross-dataset link stealing: LLM merging boosts attacks","No data sharing needed: merged LLMs crack GNN links","Universal link stealer: LLM merge beats single attackers","Attackers unite: LLM merging for stronger graph link theft"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the attackers can measure how accurate each fine-tuned model is on the very datasets they later attack, because the merge weights $\\lambda$ are computed from those accuracy measurements; if they cannot obtain labeled data for those datasets, the weight formula is unusable and the claimed advantage over simple averaging is not established.","fun_headline_variants_meta":{"raw":{"variants":["Merged LLM attacks steal GNN links across datasets","Cross-dataset link stealing: LLM merging boosts attacks","No data sharing needed: merged LLMs crack GNN links","Universal link stealer: LLM merge beats single attackers","Attackers unite: LLM merging for stronger graph link theft"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000198,"raw_usage":{"total_tokens":1386,"prompt_tokens":978,"completion_tokens":408,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":324}},"tokens_in":594,"tokens_out":408,"duration_ms":4360,"temperature":1.0,"reasoning_tokens":324,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:18:20.121742+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the four-dataset merging evaluation but compute the merging weights $\\lambda$ using only each attacker's own held-out validation split, with no labels from the other three datasets; if the merged model's mean accuracy falls to the level of the unweighted Mean or Della baselines, the reported 93.33% advantage traces to target-aware weight selection rather than to the Drop-Elect-Merge mechanism.","supporting_citations":[{"cited_title":"Stealing links from graph neural networks,","cited_arxiv_id":null,"evidence_quote":"supplies the original link stealing attack formulation and the Feature/PP baselines that the LLM attack must beat."},{"cited_title":"Link stealing attacks against inductive graph neural networks,","cited_arxiv_id":null,"evidence_quote":"provides the InductiveLSA baseline for inductive GNNs that the paper compares against."},{"cited_title":"Large Language Models for Link Stealing Attacks Against Graph Neural Networks","cited_arxiv_id":"2406.16963","evidence_quote":"is the authors' earlier LLM-based link stealing attack that this work extends to cross-dataset merging."},{"cited_title":"Judging llm-as-a-judge with mt-bench and chatbot arena,","cited_arxiv_id":null,"evidence_quote":"supplies the Vicuna-7B and Vicuna-13B LLM architectures used for fine-tuning and merging."},{"cited_title":"Semi-supervised classification with graph convolutional networks,","cited_arxiv_id":null,"evidence_quote":"provides the GCN target model and the Cora, Citeseer, and Pubmed datasets used in the evaluation."},{"cited_title":"Language models are super mario: Absorbing abilities from homologous models as a free lunch,","cited_arxiv_id":null,"evidence_quote":"contributes DARE, the drop-and-rescale merging baseline that the Drop step builds on."},{"cited_title":"Ties-merging: Resolving interference when merging models,","cited_arxiv_id":null,"evidence_quote":"contributes TIES, the sign-based merging baseline that the Elect step is compared against."}],"review_version":1}