{"id":"4ba3e47c-a005-4647-8e43-8cbca748499c","arxiv_id":"2412.11983","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Locle uses active selection, LLM annotation, and GNN-based label refinement on a rewired graph to improve label-free node classification accuracy.","lead":"Locle is a new framework that combines large language models and graph neural networks to classify nodes in text-rich graphs without human labels. It claims to outperform existing methods on five benchmarks while keeping the number of LLM queries fixed, and at very low monetary cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main threat: per-dataset hyperparameters in Table 10 are reported without any validation procedure, so the claimed gains may be inflated by test-set-based tuning.","rationale":"The paper presents a rich set of experiments and the code is released, which are real assets. The active self-training pipeline with subspace clustering, disharmonicity sampling, and hybrid refinement is plausible, and the ablations support that each component contributes. However, the central quantitative claim is about outperforming SOTA under a fixed LLM budget, and the evidence for that claim rests entirely on five benchmark datasets. The hyperparameter table is the linchpin: different epsilons, budgets, and rewiring ratios per dataset. Without a described validation procedure, a standard alternative explanation exists—that the reported numbers were obtained by selecting hyperparameters that perform best on the test sets. This would invalidate the label-free setting (test labels leak through model selection) and would explain why the method's gains vary widely across datasets (9.28% on Cora but 1.29% on Pubmed). The reader's weakest assumption captures this risk exactly. I agree with that identification. The concrete test above would distinguish between a generalizable algorithmic contribution and a configuration-tuned-to-benchmarks artifact. If the validation-based comparison reproduces the gains, the CONDITIONAL verdict can be upgraded; if not, the claim requires substantial caveats. I therefore recommend keeping the reader's CONDITIONAL verdict.","tokens_in":25913,"tokens_out":5742,"duration_ms":49453,"concrete_test":"Use the released code to perform a validation-based comparison: for each dataset, randomly hold out 20% of nodes as a validation set (stratified by class), tune Locle and the strongest LLM-GNN baseline on that validation set using a modest random search over the same hyperparameter ranges (e.g., epsilon in [0.25, 0.5, 0.75], tau in [64, 128, 256], lambda in [1e-4, 1e-3, 5e-3], delta in [0, 0.05, 0.1]), then freeze the selected configurations and report mean test accuracy (+/- std) over 5 seeds on the remaining 80%. If Locle's average improvement over LLM-GNN falls below 5 percentage points on Cora and Citeseer, or if the best configurations differ substantially from Table 10, the headline claim is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 10 lists a different hyperparameter configuration for every dataset (and backbone), including epsilon, tau, lambda, delta(+), delta(-), phi, B, and alpha, yet the paper never describes a validation split, a tuning grid, or a model-selection protocol. The main text says only that results are averaged over three trials. If these configurations were chosen by comparing test accuracies across configurations, the central claim 'significantly outperforms state-of-the-art under the same query budget' would not be a valid label-free comparison: test labels would have been used indirectly to select the method's components (including B and epsilon, which directly control the LLM budget). Even under a more charitable reading, the absence of any held-out validation means we cannot rule out overfitting to these five benchmarks, and the reported 5%+ improvements may not generalize to new TAG datasets. This concern is amplified because the evaluation against 19 baselines uses the same five datasets, and the ablations in Tables 3-4 do not check whether the qualitative conclusions (e.g., subspace clustering > K-means) survive when hyperparameters are fixed a priori.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Locle, a label-free node classification framework for text-attributed graphs that uses a limited budget of LLM queries. Locle has three main components: (i) an initial active node selection stage based on subspace clustering of GNN-derived features, (ii) a multi-round self-training scheme that selects informative nodes via label entropy and a newly proposed label disharmonicity metric, and (iii) a hybrid label refinement module that combines LLM annotations with predictions from a GNN trained on a rewired graph. The manuscript reports experiments on five TAG datasets with three GNN backbones (GCN, GAT, GCNII), comparing against 19 baselines, and claims consistent accuracy improvements over the LLM-GNN baseline under the same LLM query budget, with a notable 8.08% accuracy improvement highlighted in the abstract. The paper also includes theoretical analyses connecting the proposed metrics to Dirichlet energy and spectral clustering.","tokens_in":26242,"tokens_out":6924,"duration_ms":60035,"significance":"If the central empirical claim holds, the paper makes a useful practical contribution: it addresses two well-known limitations of LLM-GNN pipelines—cost of labeling and noise in LLM-generated labels—by integrating GNN-based selection and refinement. The scope of the evaluation is broad (19 baselines, 5 datasets, 3 backbones), and the code is promised to be public. However, the current manuscript has several load-bearing inconsistencies and evaluation-protocol gaps that prevent the claims from being accepted as stated. In particular, the hyperparameter selection procedure is opaque, the reported hyperparameters conflict with the mathematical formulation in the main text, and one of the theoretical claims contains a factor error. These issues are fixable in revision, but they are central enough that the paper should not be accepted in its present form.","major_comments":[{"comment":"The hyperparameter values in Table 10 are inconsistent with the formulas that define the method. Eq. (3) uses the Neumann series H = sum_{t=0}^T (1-alpha) alpha^t Atilde^t X, which requires |alpha| < 1 for convergence; the closed form in Eq. (2) is derived from Eq. (1) under that assumption. Yet Table 10 lists alpha = 1 or alpha = 1.2 for nearly every dataset/backbone combination. If alpha = 1, Eq. (3) gives the zero matrix (since (1-alpha) = 0), and the ensemble weighting in Eq. (7) is 0/0. If alpha = 1.2, the series diverges. The authors must either correct the equations, correct the table, or explain how the implementation computes these quantities for the reported hyperparameters.","section":"Table 10 / Eq. (3), Eq. (7)"},{"comment":"The manuscript does not describe any validation split or hyperparameter selection protocol. Table 10 reports a different configuration (epsilon, tau, lambda, delta(+), delta(-), phi, B, alpha, and backbone-specific settings) for every dataset and backbone, yet the text only states that results are averaged over three trials. If these values were selected by comparing test accuracies across configurations, the label-free evaluation is compromised because test labels indirectly influence the choice of the LLM budget B and the allocation ratio epsilon. The authors should specify the tuning procedure, e.g., a held-out validation set or a fixed protocol, and report the resulting validation-based selections.","section":"Table 10 and Section 5.1"},{"comment":"The claimed identity LH(v_i) = (1/sqrt(|N(v_i)|)) * ||(L Y^(r))_i||_2 is incorrect by a factor sqrt(|N(v_i)|). Since (L Y)_i = |N(v_i)| * (Y_i - (1/|N(v_i)|) sum_{l in N(v_i)} Y_l), taking the Euclidean norm over classes yields ||(L Y)_i||_2 = |N(v_i)| * LH(v_i). The correct relation is LH(v_i) = (1/|N(v_i)|) * ||(L Y)_i||_2 (or equivalently ||(L Y)_i||_2 = |N(v_i)| * LH(v_i)). The theorem as stated should be corrected; otherwise the theoretical justification for the disharmonicity metric does not hold.","section":"Section 4.6.1 / Eq. (9)"},{"comment":"The abstract states: \"on the DBLP dataset with 14.3k nodes, Locle achieves an 8.08% improvement in accuracy over the state-of-the-art.\" Table 2 shows that the 8.08% accuracy improvement is on WikiCS with the GCN backbone, while the DBLP improvement under GCN is 2.40%. The abstract should be corrected to attribute the 8.08% figure to the correct dataset.","section":"Abstract and Table 2"},{"comment":"All reported accuracies are averages over three trials, but no standard deviations or statistical significance tests are reported. Given that the central claim is \"significantly outperforms state-of-the-art,\" the absence of variance estimates makes it impossible to assess whether the differences (some as small as 0.01% on Pubmed/GAT in Table 2) are reliable. Please report standard deviations or confidence intervals, or provide a significance test.","section":"Tables 2-4 and Section 5.1"}],"minor_comments":[{"comment":"The symbol H is used for both the node feature matrix (e.g., in Eq. (3)) and the new graph with adjacency matrix H H^T. This notation conflict makes the description of the rewiring step hard to follow. Please use different symbols, e.g., Z for features and G' for the rewired graph.","section":"Section 4.4.1"},{"comment":"The paper says \"19 baselines\" in the abstract and contributions, but Section 5.2 says \"seven groups of baseline methods, i.e., 20 competitors.\" The count should be made consistent; counting the rows in Table 2 gives 19 baseline methods.","section":"Section 5.2 and Abstract"},{"comment":"The fourth contribution bullet claims \"a consistent and remarkable improvement of at least 5% in zero-shot classification accuracy compared to the state of the art in most cases.\" Table 2 contains many improvements well below 5% (e.g., Pubmed GCN: 1.29%; DBLP GCN: 2.40%; Pubmed GAT: 0.01%). Please adjust the wording to match the actual magnitudes.","section":"Section 1 / Contribution bullet"},{"comment":"The table omits F1 scores for the LLM-as-Predictor baselines; the text explains this is because LLM outputs are invalid for F1 calculation, but it would be helpful to add a footnote in the table itself.","section":"Table 2 / LLM as Predictor"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the absence of a validation protocol for the hyperparameters in Table 10. This is not just a presentation concern: if the configurations were chosen by comparing test accuracy, the 5%+ improvements over LLM-GNN would not be label-free. Please ask the authors to clearly state the tuning procedure or to re-run the evaluation with a proper validation split. The alpha inconsistency is also critical because it suggests the implementation may not match the described Neumann-series formulas; the authors should provide a corrected formulation or clarify the meaning of alpha in Table 10. The factor error in Section 4.6.1 is smaller but should be corrected for consistency. Overall, the idea is promising and the experimental scope is appropriate, but the current manuscript needs substantial revision before its claims can be trusted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the method itself is sensible and the experiments are broad: three GNN backbones, five datasets, nineteen baselines, and ablations that back each of the three components. Second, the central claim—\"significantly outperforms under the same query budget\"—is currently not fully supported because the paper never describes how the per-dataset hyperparameters in Table 10 were chosen.\n\nWhat is actually new is the pipeline: subspace-clustering active selection, label disharmonicity plus entropy for informative sample selection, and a rewiring-based hybrid LLM/GNN label refinement. Individually these pieces exist, but this particular configuration is not in the cited prior work, and the ablations give real evidence that each piece contributes. The query-budget comparison is the right frame, and the reported gains on Cora, CiteSeer, and WikiCS are large. The code is available, which makes the work reproducible in principle.\n\nThe soft spots are real but not fatal. The biggest one is the hyperparameter issue. Table 10 lists different values for every dataset and backbone, including B and epsilon, which directly control the LLM budget, yet there is no validation split, tuning grid, or model-selection protocol in the paper. If those values were selected by comparing test accuracies, the label-free claim is compromised and the reported gains could partly reflect overfitting to these five benchmarks. I cannot rule that out from the text as written. This is a gap in the evidence, not evidence of fraud, but it needs to be fixed with a clear tuning procedure or a sensitivity analysis that shows the conclusions hold across reasonable hyperparameter choices.\n\nSecond, the abstract attributes the 8.08% improvement to DBLP; in Table 2, that number belongs to WikiCS, while DBLP shows 2.40%. Concrete factual error. Third, the theoretical connection in Section 4.6.1 is off by a factor: Eq. 9 averages neighbor labels, which gives (1/|N(v_i)|) times the Dirichlet gradient norm, not (1/sqrt(|N(v_i)|)). Minor and does not affect the method, but it should be corrected. Fourth, results are averaged over three trials with no error bars; for differences of a few percent, that matters.\n\nWho gets value from this? Anyone working on label-free or low-label node classification on text-attributed graphs, and any practitioner weighing LLM query budgets against GNN training. It is a useful engineering contribution with a credible core. It deserves a serious referee, but the referee should ask for a validation protocol, variance reporting, and the two corrections above.","headline":"Locle is a credible, well-engineered LLM+GNN pipeline whose main claim is probably right but is under-supported by the missing validation protocol and a couple of fixable presentation errors.","tokens_in":26719,"tokens_out":3009,"would_cite":true,"duration_ms":28484,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a GNN-driven active self-training loop, Locle, extracts more accuracy from a fixed LLM query budget than LLM-GNN by selecting critical nodes and refining noisy LLM labels with graph rewiring.","keywords":["label-free node classification","text-attributed graphs","large language models","graph neural networks","active learning","self-training","pseudo-labeling","graph rewiring"],"falsifier":"Run Locle on the same five datasets with all hyperparameters chosen by a documented validation split and no test-label access at any point; if the reported accuracy margins over LLM-GNN shrink or disappear, the central claim of generalizable cost-effective gain is refuted.","tokens_in":25762,"feed_emoji":"📈","tokens_out":9385,"duration_ms":78360,"temperature":0.7,"pith_summary":"Locle is a framework for classifying nodes in text-attributed graphs without any ground-truth labels, using only a small budget of queries to a large language model. The paper argues that the standard LLM-GNN pipeline wastes this budget: it annotates once, ignores graph structure during selection, and lets LLM label noise poison the downstream GNN. Locle instead runs an iterative self-training loop in which a GNN picks the most informative nodes, the LLM labels only the uncertain ones, and a rewired graph topology refines those labels. On five benchmarks the authors report consistent accuracy gains over state-of-the-art baselines under identical LLM query budgets, including an 8.08% improvement on DBLP at a cost below one cent. If correct, this would make LLM-assisted, label-free node classification substantially cheaper and more practical.","feed_headline":"8% accuracy gain for under a cent of LLM queries","feed_subtitle":"Locle pairs GNN self-training with a few LLM labels to beat label-free baselines on five text graphs.","key_machinery":"The load-bearing mechanism is the iterative interplay between two signal sources. Stage I builds a $T$-truncated graph-smoothed representation $\\mathbf{H} = \\sum_{t=0}^T (1-\\alpha)\\alpha^t \\tilde{\\mathbf{A}}^t \\mathbf{X}$ and runs subspace clustering on it; Lemma 4.1 shows that spectral clustering of the self-expressive matrix $\\mathbf{S} = \\mathbf{U}\\mathbf{U}^\\top$ equals $K$-means on the left singular vectors $\\mathbf{U}$, so the active node set is cheap to extract. In each self-training round, an ensemble of past GNN predictions is scored by label entropy $LE(v_i)$ and label disharmonicity $LH(v_i)$, the $\\ell^2$ norm of the Dirichlet-energy gradient at the node; the most confident nodes become pseudo-labels and the least confident are sent to the LLM. For those uncertain nodes, Locle rewires the graph by optimizing Dirichlet energy with respect to the Laplacian, giving a closed-form adjacency from $\\mathbf{H}\\mathbf{H}^\\top$, then removes low-weight edges, adds edges from labeled to unlabeled nodes, and blends the GNN prediction on the rewired graph with the LLM annotation by rank-based confidence. These two lemmas and the two selection metrics carry the whole argument: they convert graph structure into a label-refinement signal that compensates for LLM noise.","core_discovery":"The central claim is that label noise from LLMs, rather than query cost, is the main obstacle to label-free node classification, and that a GNN can actively manage that noise. Locle's first stage selects a small initial annotation set by subspace clustering on smoothed GNN representations; the second stage repeatedly uses label entropy and a new label disharmonicity measure to separate high-confidence nodes, whose GNN predictions become pseudo-labels, from low-confidence nodes, which are sent to the LLM, then refines the LLM outputs by comparing them with predictions on a graph rewired to suppress noisy edges. The paper reports that this pipeline outperforms LLM-GNN and its active-selection variants on Cora, Citeseer, Pubmed, WikiCS, and DBLP for GCN, GAT, and GCNII backbones. It also reports that with GPT-3.5-turbo, Locle can match or beat the same pipeline using stronger GPT-4 models, implying that with the right selection and refinement, the LLM stops being the performance bottleneck and the graph itself supplies most of the supervision.","pith_inferences":["Locle's components are not specific to LLM noise: the same entropy-plus-disharmonicity selection and rewired-graph refinement could be applied to human annotations with known error rates or to semi-supervised learning with a handful of clean labels, though the paper tests neither setting.","Because the rewiring step deliberately adds edges from labeled to unlabeled nodes, a natural extension is to measure how Locle's margin changes with graph homophily or edge density; the paper reports no such sensitivity analysis.","The per-dataset settings listed in the paper's hyperparameter table suggest that a new graph would need its own tuning, and the paper offers no label-free procedure for choosing them; transfer to a new domain would require some heuristic or validation scheme."],"forward_implications":["Under the same LLM query budget, Locle reports higher accuracy than every LLM-GNN variant on all five datasets for all three GNN backbones, with the largest headline gain being 9.28 percentage points on Cora with GCN.","Because Locle with GPT-3.5-turbo matches or beats Locle with GPT-4 on Cora and Citeseer, the framework implies that the choice of LLM matters less than the refinement loop, so users can spend less on model quality.","On Cora, once the budget reaches 350 queries, Locle's accuracy approaches that of the same pipeline trained on ground-truth labels, suggesting the label-free gap can be nearly closed.","Cost measurements in the paper put querying DBLP at under one cent and make Locle at least twice as fast as the best LLM-GNN variant, so the approach is affordable at benchmark scale."],"supporting_citations":[{"why":"Supplies the LLM-GNN baseline and the active-selection variants (FeatProp, GraphPart, RIM) that Locle must beat, and the evidence that LLM label noise degrades GNN training.","marker":"[7]"},{"why":"Provides the zero-shot LLM-as-predictor baselines and the source of raw paper texts for the benchmark datasets.","marker":"[6]"},{"why":"Produces the Sentence-BERT attribute vectors that encode node text for the graph, used in both stages of Locle.","marker":"[52]"},{"why":"Supplies the subspace clustering technique that Locle adapts to select representative active nodes for initial LLM annotation.","marker":"[63]"},{"why":"Establishes that the low-rank self-expressive matrix minimizer is $\\mathbf{S} = \\mathbf{U}\\mathbf{U}^\\top$, the identity behind Lemma 4.1.","marker":"[37]"},{"why":"Provides the spectral clustering formulation that connects subspace clustering to $K$-means in Lemma 4.1.","marker":"[64]"}],"fun_headline_variants":["Under a cent of LLM queries, 8% accuracy gain on graphs","GNN self-training with few LLM labels beats label noise","Locle: label-free node classification for under a cent","LLM label noise tamed by GNN active self-training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the per-dataset settings reported for Locle were chosen without looking at test labels; the paper gives no validation procedure, so if those values were picked to maximize test accuracy on the five benchmarks, the reported gains might not generalize.","fun_headline_variants_meta":{"raw":{"variants":["Under a cent of LLM queries, 8% accuracy gain on graphs","GNN self-training with few LLM labels beats label noise","Locle: label-free node classification for under a cent","LLM label noise tamed by GNN active self-training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000528,"raw_usage":{"total_tokens":2625,"prompt_tokens":1099,"completion_tokens":1526,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":715,"completion_tokens_details":{"reasoning_tokens":1453}},"tokens_in":715,"tokens_out":1526,"duration_ms":10007,"temperature":1.0,"reasoning_tokens":1453,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:23:52.339659+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Locle on the same five datasets with all hyperparameters chosen by a documented validation split and no test-label access at any point; if the reported accuracy margins over LLM-GNN shrink or disappear, the central claim of generalizable cost-effective gain is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LLM-GNN baseline and the active-selection variants (FeatProp, GraphPart, RIM) that Locle must beat, and the evidence that LLM label noise degrades GNN training."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the zero-shot LLM-as-predictor baselines and the source of raw paper texts for the benchmark datasets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the subspace clustering technique that Locle adapts to select representative active nodes for initial LLM annotation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes that the low-rank self-expressive matrix minimizer is $\\mathbf{S} = \\mathbf{U}\\mathbf{U}^\\top$, the identity behind Lemma 4.1."}],"review_version":1}