{"id":"2c494233-37e3-4dd2-9fb2-59d95ec48314","arxiv_id":"2501.11849","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Frozen LLMs with balanced retrieval-augmented prompting detect astroturf campaigns better than GNN baselines on a 2016 US election dataset, but the reported margins are overstated.","lead":"The paper tests whether a frozen large language model, fed with tweet text and retweet structure as text, can detect coordinated fake-news campaigns better than graph neural networks. It reports large gains when the prompt includes balanced similar and contrasting labeled examples, but the evaluation has important caveats.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains may be inflated by train/test leakage: Balanced RAG retrieves text-similar examples from training, and if the split is not claim-disjoint, near-duplicate texts with known labels make the task trivial.","rationale":"The reader identified propagation-tree reconstruction as the weakest assumption, but the more fundamental threat is data leakage through the RAG component. The central claim is that Balanced RAG plus prompt engineering yields 2x-3x improvements over GNN baselines. For that comparison to be valid, the test set must not contain texts that are near-duplicates of training texts whose labels the RAG retrieves. The paper does not describe the split, and the dataset construction (FakeNewsNet labels at article level) makes such leakage almost inevitable under a random split. This is more load-bearing than the tree-reconstruction issue because even a perfect propagation tree would not rescue the claim if the LLM is effectively reading the answer from a retrieved near-duplicate text. The paper's own ablation strengthens the concern: text alone is nearly sufficient (recall 0.7985), and removing text collapses performance, so the model is heavily text-driven. Therefore the paper must demonstrate that the split is claim-disjoint or that performance survives such a split. My recommendation remains CONDITIONAL, matching the reader, but the condition should explicitly include ruling out text-level leakage.","tokens_in":14829,"tokens_out":5008,"duration_ms":51161,"concrete_test":"Re-run the main evaluation with a claim-disjoint split: group all 10,228 propagation trees by the FakeNewsNet source article ID and place entire groups in train or test (e.g., stratified 80/20). If Balanced RAG's F1 drops from 0.5658 toward the non-RAG baselines, or the gap is substantially reduced, the reported gains are partly due to retrieving near-duplicate training texts. As a cheaper diagnostic, compute the maximum cosine similarity between the root-tweet text of each test sample and the texts of its six retrieved examples; if a large fraction of true positives have a retrieved positive with similarity above 0.9, label leakage is confirmed.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The most load-bearing assumption is hidden in the evaluation protocol. Section 3.5 (Balanced RAG) retrieves the most text-similar propagation trees from the training set and places their texts, with labels, directly into the prompt. The dataset in Section 4.1 is derived from FakeNewsNet, whose labels attach to the underlying news article or claim, not to individual tweets. The paper never describes the train/test split. If the split is random at the propagation-tree level, the same claim (and near-identical tweet text) will appear in both training and test partitions. A test tweet belonging to a fake-news article then retrieves a 'similar' positive example that is essentially the same text, and the LLM only needs to copy the label. The 2x-3x improvement over GNN baselines would then reflect retrieval leakage, not prompt engineering or graph encoding. The ablation in Table 6 supports this worry: removing text collapses recall to 0.0622, proving the model keys almost entirely on text, so any text-level leakage is decisive. The propagation-tree reconstruction concern raised by the reader is secondary: even if the tree were perfect, this leakage would invalidate the comparison.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a framework for detecting coordinated 'astroturf' political disinformation campaigns on X using a frozen large language model. The pipeline reconstructs a propagation tree from retweet data, encodes the tree structure as text, optionally retrieves labeled similar examples from a training set by embedding similarity, and queries Llama 3.1 with a prompt that combines the tweet text, graph encoding, and retrieved examples. The two main contributions are a graph-as-text prompting scheme and 'Balanced RAG,' which augments the prompt with both positive and negative retrieved examples to mitigate extreme class imbalance. On a FakeNewsNet-derived dataset of 10,228 propagation trees (9.7% positive), the best variant reports precision 0.4238, recall 0.8507, F1 0.5658, and ROC AUC 0.8602, outperforming GAT, GCN, GraphSAGE, and GraphConv baselines.","tokens_in":15060,"tokens_out":6419,"duration_ms":61526,"significance":"If the reported numbers survive a leakage-free evaluation, the framework would be a useful, training-free alternative to GNN-based fake-news detection, and the Balanced RAG idea—providing contrasting positive and negative demonstrations at inference time—is simple and potentially transferable to other class-imbalance settings. The paper ships source code, provides a full prompt template in the appendix, and uses a frozen LLM, which aids reproducibility and lowers computational barriers. However, the current evidence is not sufficient: the evaluation protocol leaves open a text-level train/test leakage route, the baselines are not class-imbalance-aware, and the headline '2x-3x' claim is not supported by the reported numbers. The core idea is worth pursuing, but the empirical claims need to be re-established under a claim-disjoint split and a fairer baseline setup.","major_comments":[{"comment":"The paper never describes the train/test split. The dataset in §4.1 is derived from FakeNewsNet, where labels are attached to underlying news articles or claims, and Balanced RAG in §3.5 retrieves the most text-similar propagation trees from the training set and places their texts and labels in the prompt. If the split is not claim-disjoint, a test tweet from a fake-news article can retrieve a near-identical positive training example and the LLM only needs to copy the label. Table 6 shows that removing text collapses recall from 0.8507 to 0.0622, so text is the decisive channel; this makes text-level leakage a decisive confound for every reported comparison. Please specify the split, ensure no claim or article overlap between train and test, and report results on a claim-disjoint split together with examples of retrieved neighbors.","section":"§4.1, §3.5, §4.3"},{"comment":"The abstract claim of '2x-3x improvements in terms of precision, recall and F1 scores' is not supported by Table 2. The best recall is 0.8507 versus 0.5500 for GAT, a 1.55x improvement; F1 and precision improvements are about 2.45x and 2.33x, respectively. The claim that the framework achieves 'state-of-the-art performance' is also unsupported because no prior LLM-based or class-imbalance-aware methods are compared. Please revise the claims to match the reported numbers and restrict comparisons to the baselines actually tested.","section":"Abstract and Table 2"},{"comment":"Equation (6) formally defines the objective as maximizing the average score over the test set using the correct answers a_i, which is an oracle-based prompt selection procedure. The paper does not describe a validation protocol for choosing the number of RAG examples, the temperature, the retrieval embedding, or the graph encoding scheme, and the hyperparameter experiments in §4.5 use Llama 3.1 8B while the main results use 70B. No repeated runs, standard deviations, or significance tests are reported. Please move prompt and component selection to a validation set and report variance across runs or seeds.","section":"§3.6, §4.5"},{"comment":"The GNN baselines are trained with ordinary cross-entropy on a dataset with 9.7% positives and are not equipped with class weights, oversampling, or threshold tuning, whereas the proposed LLM variant receives balanced positive and negative demonstrations by construction. This makes the comparison lopsided; adding imbalance-aware baselines or balanced-loss and oversampled variants is needed. Separately, the propagation-tree reconstruction in §3.3 uses a random parent chosen with probability proportional to follower count (Eq. 1) when the two conditions fail; since X provides only a star graph, the reconstructed trees may be largely noise, and the paper does not analyze sensitivity to this randomness or validate the reconstruction. Either validation or a sensitivity analysis is required before attributing gains to graph information.","section":"§3.3, §4.4"},{"comment":"The definition of Balanced RAG in §3.5 (retrieve top-n similar samples, then for each retrieve the most similar opposite-label sample) differs from the implementation described in §4.3 (retrieve three most similar positives and three most similar negatives). These procedures are not equivalent and can produce different prompts; please align the description with the code and clarify which pairs are actually fed to the LLM, including whether the query appears in the retrieved set.","section":"§3.5, §4.3"}],"minor_comments":[{"comment":"The parenthetical labels are swapped: 'positive (organic propagation trees)' and 'three negative (astroturf propagation trees)' should read 'positive (astroturf propagation trees)' and 'three negative (organic propagation trees)'.","section":"§4.3"},{"comment":"There is a typo: 'propagation tressG' should be 'propagation trees G'.","section":"§4.1"},{"comment":"The statement that the framework 'eliminates the need for labeled datasets' contradicts the use of a labeled training set for Balanced RAG and few-shot examples; please rephrase.","section":"Appendix B"},{"comment":"The captions of Tables 2, 5, and 6 duplicate the sentence 'The best results for each metric are highlighted in bold'; remove the duplicate.","section":"Tables 2, 5, 6"},{"comment":"There are minor language errors: 'astoturf' should be 'astroturf' in Appendix B, and 'represent a 1.5x-3x improvement' should be 'represents a 1.5x-3x improvement' in §4.4.","section":"Appendix B and §4.4"},{"comment":"The related-work discussion of coordinated disinformation detection describes the present framework's contributions as established literature without citations; please distinguish prior work from this paper or add appropriate attributions.","section":"§2"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the decisive issue is the missing split description; I would not proceed without a claim-disjoint split. If the authors cannot construct such a split from FakeNewsNet, the paper's central comparison should be withdrawn. The related-work section also appears to describe the present method as prior work in places, and this should be checked for citation integrity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the core idea is worth a look — using a frozen LLM with a balanced retrieval-augmented generation (RAG) component that feeds the model similar positive and negative propagation trees is a sensible twist for class imbalance. The graph-as-text encoding is not new (they cite Fatemi et al.), but applying it this way to astroturf detection is a legitimate application, and the code is on GitHub. Those are real positives.\n\nThe soft spots are serious, though. First, the paper never describes the train/test split. Balanced RAG retrieves similar propagation trees from the training set and puts their tweet text directly into the prompt. If the split is not claim-disjoint, the same news article's tweets can appear in both partitions, and the LLM merely copies the label from a near-identical retrieved example. The ablation in Table 6 makes this worse: removing text collapses recall to 0.0622, proving the model keys on text, so any text-level leakage is decisive. That is a load-bearing flaw, not a nitpick.\n\nSecond, the abstract claims 2x-3x improvements over GNN baselines, but Table 2 shows the best recall gain is 1.55x (0.8507 vs 0.5500), and precision/F1 gains are smaller. Third, there are no error bars or repeated runs, and the best prompt configuration is selected without a described validation protocol, so the numbers could be overfit to the test set. Fourth, the GNN baselines are not class-imbalance-aware, making the comparison lopsided. The related-work section also contradicts the novelty claim by saying prior work already did this, with mismatched citations.\n\nThe propagation-tree reconstruction concern (random parent selection proportional to follower count) is real but secondary — even with perfect trees, the leakage issue remains.\n\nBottom line: promising idea that deserves a serious referee, but the current evaluation does not support the claims. The authors need to specify a claim-disjoint split, add error bars, and make the baselines fair. If they do that, this could be a solid contribution. As it stands, I would not cite it.","headline":"A genuinely interesting frozen-LLM + balanced RAG idea for astroturf detection, but the evaluation likely suffers from train/test leakage and the headline gains are overstated.","tokens_in":15611,"tokens_out":2352,"would_cite":false,"duration_ms":23105,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A frozen LLM, prompted with the tweet text, a text-encoded retweet tree, and balanced similar/opposite examples, can identify coordinated astroturf campaigns without any training or fine-tuning, outperforming graph neural network…","keywords":["astroturf campaigns","disinformation detection","fake news","large language models","prompt engineering","retrieval-augmented generation","class imbalance","graph encoding"],"falsifier":"Run the best variant (Llama 3.1 70B + Balanced RAG) on the same test split but substitute the graph text with (a) the original X star graph (all retweets attached to the root) and (b) a degree-matched randomly rewired tree; if F1 stays at or above 0.55 in both cases, the reconstructed propagation-tree structure is not carrying the performance.","tokens_in":14636,"feed_emoji":"🧠","tokens_out":8885,"duration_ms":71439,"temperature":0.7,"pith_summary":"This paper claims that a frozen large language model, fed with prompt-engineered text that combines the original tweet, a textual encoding of its retweet propagation tree, and a balanced set of retrieved examples, can detect coordinated political fake-news campaigns (astroturfing) on X better than graph neural network baselines. The central novelty is Balanced RAG: instead of retrieving only the most similar examples, the method retrieves similar examples and pairs each with the most similar example of the opposite label, so the LLM sees contrastive pairs that keep it attentive to the rare positive class. If this is right, astroturf detection can work without training or fine-tuning any model, relying only on off-the-shelf LLM inference and prompt design. On a dataset of over 10,000 propagation trees from the 2016 US election, the best variant reaches about 0.42 precision, 0.85 recall, 0.57 F1, and 0.86 ROC AUC, roughly a 2x-3x improvement over the GNN baselines tested.","feed_headline":"Frozen LLMs outperform graph nets on astroturf detection by up to 3x","feed_subtitle":"Balanced RAG pairs similar positive and negative examples so a frozen LLM finds rare fake-news cascades with 0.85 recall","key_machinery":"The load-bearing pieces are three: (1) propagation-tree reconstruction, which turns X's star-shaped retweet data into a tree using two heuristic conditions and, when neither applies, a random parent selected with probability proportional to the author's follower count (Eq. 1); (2) graph-to-text encoding, which renders the edge set as strings like '(2->1), (3->1)' so a text-only LLM can reason about structure; and (3) Balanced RAG, which retrieves the top-n training propagation trees most similar to the query by cosine similarity of embeddings, then for each retrieved tree finds the most similar tree with the opposite label, forming contrastive pairs that are placed into the few-shot prompt. The interaction of these components, not any single one, is what the paper claims carries the detection performance.","core_discovery":"On the paper's own terms, the discovery is that graph structure does not need a learned encoder: a frozen decoder-only LLM can use a text encoding of a propagation tree, together with balanced few-shot examples, to identify coordinated disinformation campaigns under extreme class imbalance, outperforming GNN baselines by a factor of two to three in precision, recall, and F1. The authors assert that the combination of role prompting, chain-of-thought, and few-shot prompting, with Balanced RAG constructing pairs of similar-but-oppositely-labeled propagation trees, is what lets the LLM overcome the scarcity of positive samples without data augmentation or fine-tuning.","pith_inferences":["A likely unstated reason for the large recall gain is that Balanced RAG acts as a label-balancing prior, nudging the LLM away from the majority-class bias; a testable variant would swap the retrieved contrastive pairs for randomly sampled balanced pairs matched on text length and centrality statistics to see how much of the gain is due to the similarity-based pairing specifically.","If the reconstructed trees are largely wrong, the graph text may function mainly as a mild regularizer, with most signal coming from tweet text and the retrieved examples; replacing the graph text with the raw star graph, or with a degree-matched random tree, would reveal whether the specific edge structure matters.","The framework inherits the biases of its training labels: the 2016-election dataset's 'positive' label comes from PolitiFact verdicts, so on other topics or platforms the same prompts would need new balanced example pools, and the paper does not test cross-domain transfer."],"forward_implications":["Astroturf detection becomes a zero-training task: any decoder-only LLM can be pointed at a tweet and its retweet cascade, and the balanced few-shot prompts supply the task knowledge.","The Balanced RAG pairing recipe transfers to other imbalanced few-shot classification problems, such as fraud detection or rare-disease mention classification, wherever similar-looking items carry opposite labels.","Text-encoding graphs decouples graph analysis from GNN tooling: propagation trees can be processed by any text model, making the approach portable across LLMs and easy to update as models change.","Because the best variant reports ~0.85 recall on a dataset where only 9.7% of trees are positive, the framework is viable as a first-pass filter that flags suspected campaigns for human review, trading precision for coverage."],"supporting_citations":[{"why":"Supplies the propagation-tree reconstruction method and the GNN baselines that the framework builds on and then outperforms.","marker":"[20]"},{"why":"Provides the graph-to-text encoding guidelines used to turn the propagation tree's edge list into prompt text.","marker":"[8]"},{"why":"Foundational retrieval-augmented generation paper that Balanced RAG extends by retrieving similar examples and pairing them with opposite-label counterparts.","marker":"[16]"},{"why":"Geometric deep-learning fake-news detection work referenced alongside [20] for predicting propagation trees from retweet data.","marker":"[22]"},{"why":"Graph-based retrieval for LLMs, cited as the basis for the similarity-based sample selection in Balanced RAG.","marker":"[1]"}],"fun_headline_variants":["Frozen LLM with Balanced RAG triples astroturf detection performance","Prompt engineering helps LLM beat graph nets on astroturf by 3x","Balanced RAG lets frozen LLM spot rare astroturf cascades 3x better","LLM without fine-tuning detects coordinated disinformation 3x better","Graph-free: LLM with Balanced RAG outperforms GNNs on astroturf"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reconstructed propagation trees are assumed to faithfully reflect the true retweet cascades, even though X provides only star graphs; the paper builds trees with two heuristic rules and a follower-count-weighted random parent, so if those inferred edges are mostly wrong, the graph information in every prompt is largely noise.","fun_headline_variants_meta":{"raw":{"variants":["Frozen LLM with Balanced RAG triples astroturf detection performance","Prompt engineering helps LLM beat graph nets on astroturf by 3x","Balanced RAG lets frozen LLM spot rare astroturf cascades 3x better","LLM without fine-tuning detects coordinated disinformation 3x better","Graph-free: LLM with Balanced RAG outperforms GNNs on astroturf"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000276,"raw_usage":{"total_tokens":1666,"prompt_tokens":985,"completion_tokens":681,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":574}},"tokens_in":601,"tokens_out":681,"duration_ms":6220,"temperature":1.0,"reasoning_tokens":574,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:47:32.432981+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the best variant (Llama 3.1 70B + Balanced RAG) on the same test split but substitute the graph text with (a) the original X star graph (all retweets attached to the root) and (b) a degree-matched randomly rewired tree; if F1 stays at or above 0.55 in both cases, the reconstructed propagation-tree structure is not carrying the performance.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the propagation-tree reconstruction method and the GNN baselines that the framework builds on and then outperforms."}],"review_version":1}