{"id":"7ecfa931-c454-4bc7-9d3b-fde0e6e61664","arxiv_id":"2412.06606","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The SPECTER text-matching component used in reviewer assignment at NeurIPS and similar venues is vulnerable to collusion: archive curation plus LLM abstract edits put a colluding reviewer into the top-5 similarity ranks about 92% of the time.","lead":"This paper demonstrates that the text-matching system used to assign reviewers at top AI conferences can be gamed by a colluding author and reviewer, even without any use of the bidding system. It describes two simple manipulations, trimming the reviewer's publication list and editing the abstract with a language model, and reports that these raise a reviewer from rank 101 to the top 5 roughly 92% of the time.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 'get assigned' is not actually tested: experiments measure top-k similarity rankings, so reviewer-load, conflict, and randomization constraints could substantially lower real-world attack success.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing gap I see: the experimental success metric is top-5 similarity ranking, while the abstract and introduction claim actual assignment. The primary attack results are otherwise strong—they are evaluated out-of-sample on NeurIPS 2023 data, hyperparameters are selected on NeurIPS 2022 data, code and adversarial examples are released, and the human-subject experiment is preregistered with disclosed limitations. The main concern is not an internal inconsistency in the attack procedure; it is external validity of the headline inference from similarity rank to assignment. Because the paper itself lists assignment constraints as an unmodeled limitation and also recommends randomization as a defense, the gap is material enough to keep the verdict conditional. I would not reject the paper or lower its novelty: the vulnerability of text matching is demonstrated convincingly. I would only require that the 'get assigned' language be either verified through an assignment-level simulation or softened to 'achieve top-k similarity' in the abstract and introduction.","tokens_in":25242,"tokens_out":5501,"duration_ms":66498,"concrete_test":"Re-run the Section 5.3 fully automatic attack for the 300 natural-rank-101 colluding pairs, producing manipulated SPECTER similarity scores for all (paper, reviewer) pairs. Feed these scores into a standard TPMS/OpenReview-style assignment optimizer with realistic constraints: reviewer capacity (e.g., 4–6 papers), paper quota (3–6 reviewers), a conflict mask derived from coauthorship/affiliation, and optionally a small randomization term. Report the fraction of the 300 colluding pairs actually assigned and compare it with the reported 93% top-5 success rate. If the assigned fraction is materially lower, the abstract's 'get assigned' claim should be revised to 'achieve top-k similarity' or supplemented with an assignment-level analysis.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central advertised claim—that colluding authors and reviewers can 'get assigned their target paper'—is supported only by a ranking proxy. Experiments measure the colluding reviewer's SPECTER-similarity rank among all reviewers (top-1/3/5), not whether the assignment solver actually assigns that reviewer. The actual assignment optimization described in Section 2.1 includes reviewer and paper load limits, conflicts of interest, and potentially randomization; Section 6.2 explicitly concedes that these constraints are not factored in. The 92% figure from Tables 1 and 2 is the fraction of attacks reaching top-5 similarity, not the fraction of papers assigned to the colluding reviewer. If load balancing, fairness constraints, or randomization exclude the top-ranked reviewer, a top-5 similarity rank can fail to produce an assignment. This is especially relevant because the paper itself recommends randomization as a defense in Section 6.1, citing prior work showing that randomization provably prevents adversaries from reliably influencing assignments. The underlying technical finding—that SPECTER-based similarity scores are easily manipulable—is well supported by the out-of-sample experiments and released artifacts. The gap is the inference from manipulated rank to actual assignment. If an assignment-level simulation still assigns top-5 reviewers in the large majority of cases, the concern is minor; if not, the paper's headline claim is stronger than its evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies whether the SPECTER-based text-matching component used in automated reviewer assignment at ML/AI venues can be manipulated by colluding author-reviewer pairs. The proposed attack combines adversarial archive curation by the reviewer with LLM-based abstract modifications (IncludeThemes and InsertKeywords), in both human-in-the-loop and fully automatic modes. Experiments on NeurIPS 2022/2023 measure the colluding reviewer's similarity rank among all reviewers, reporting high top-1/3/5 success rates (e.g., 92% top-5 for natural rank 101), strong cross-year correlations, and sensitivity to archive length and pooling method. A preregistered human-subject study examines whether adversarial abstracts are detectable. The paper concludes that text matching is vulnerable and proposes defenses such as minimum archive lengths, average pooling, and randomized assignment.","tokens_in":25523,"tokens_out":5329,"duration_ms":58731,"significance":"The main technical finding—that SPECTER embeddings are susceptible to simple, LLM-assisted textual manipulation—is credible and well supported by out-of-sample evaluation: hyperparameters are tuned on NeurIPS 2022 and evaluated on held-out NeurIPS 2023 data, with standard errors reported and code, data, and adversarial examples released. The threat model is realistic, and the cross-year correlation analysis is a useful contribution. If the similarity-ranking results translate to actual assignments, the implications are substantial because the affected algorithms are used at multiple venues. The central gap is that the headline 'get assigned' claim is supported only by a ranking proxy, not by assignment-level simulation; the paper's own Section 6.2 concedes that load, conflict, and geographic constraints were not factored in. This gap must be addressed before the full significance of the claim can be accepted.","major_comments":[{"comment":"The abstract and introduction claim that colluders can 'get assigned their target paper,' but the experiments measure only the colluding reviewer's top-1/3/5 similarity rank among all reviewers, not whether the assignment optimization actually selects that reviewer. Section 5.1 defines success rates as ranking-based, Section 2.1 justifies rankings as a proxy for general applicability, and Section 6.2 explicitly states that constraints such as reviewer and paper load limits, conflicts of interest, and geographical considerations were not factored in. Thus the reported 92% figure in Tables 1 and 2 is the fraction of attacks reaching top-5 similarity, not the fraction of papers assigned to the colluding reviewer. Please either simulate the actual assignment optimization with realistic constraints, or consistently rephrase the central claim as 'achieve top-k similarity ranking' and avoid the unqualified 'get assigned' language. The current wording overstates the evidence.","section":"Abstract; §2.1; §5.1; §6.2"},{"comment":"The human-subject experiment used attack budgets and prompts that were 'erroneously tuned on the NeurIPS 2023 test data' rather than the NeurIPS 2022 training data. The manuscript argues that this should not affect the outcome because the manipulated abstracts in that experiment were edited more heavily than those in the automatic attacks. This is plausible but not demonstrated, and the detectability results are the only evidence for the paper's 'plausible deniability' conclusion. The limitations section mentions the error, but the conclusions in Section 5.7.6 should either be explicitly qualified as conditional on this tuning issue or supported by a robustness check using abstracts generated with training-data-tuned budgets.","section":"§5.7.2; §6.2"}],"minor_comments":[{"comment":"There is a typo in 'Similarily, we also sample'—it should be 'Similarly.'","section":"§5.1"},{"comment":"The 25 human-in-the-loop evaluations were not a random sample: the authors write that they 'keep the first 25 samples with paper topics we are familiar enough with.' This selection rule could bias the high success rates in Table 1; please report the number of samples excluded by this familiarity criterion and comment on the potential direction of bias.","section":"§5.2"},{"comment":"The sentence 'While [CFB+20] has only released three adversarial samples' appears to cite the wrong paper; the context indicates that the prior attack paper [EQM+23] released three samples, not the SPECTER paper [CFB+20].","section":"§3, Related Work, fourth bullet"},{"comment":"The preregistration URL 'https://aspredicted.org/HXF Z92' contains an erroneous space and should be corrected.","section":"§5.7.1"},{"comment":"There is a typo: 'reponses' should be 'responses.' Also, the rows for 'Irregularities related to IncludeThemes' and 'Irregularities related to InsertKeywords' should specify that the percentages are computed over the 67 experimental reviews, since the table currently mixes group sizes.","section":"Table 6"},{"comment":"The x-axis label 'Target Archive Length' is ambiguous; it should be 'Reviewer archive length |Qadv_r|' or 'Required archive length' to indicate that this is the number of papers the reviewer is forced to retain.","section":"Figure 4"},{"comment":"The threat model should state explicitly how the colluding pair avoids being flagged by conflict-of-interest screening; if the attack assumes colluders are not co-authors or institutional affiliates, that assumption should be stated and justified.","section":"§2.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a good fit for the venue, and the disclosure of the vulnerability plus the deployed safeguards is a strength. The main revision should focus on aligning the title and abstract with the ranking-based evidence. If the authors prefer not to run an assignment-level simulation, they should substantially soften the 'get assigned' language; given the advertised claim, I would encourage the simulation as the stronger path."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe one thing you should know: the central technical claim holds up. SPECTER text-matching, the component used by NeurIPS, ICML, ICLR, and TMLR, is easily manipulated by a colluding author-reviewer pair. The paper shows that with a curated reviewer archive and a few LLM-driven abstract modifications, a reviewer ranked 101st can be brought to top-5 similarity about 92% of the time on NeurIPS 2023 data. That part is real.\n\nWhat is actually new: prior work attacked an LDA-based system at small scale and assumed knowledge of the program committee. This paper attacks the neural SPECTER model at conference scale, introduces reviewer archive curation as an attack surface, and evaluates out-of-sample—hyperparameters tuned on NeurIPS 2022, tested on NeurIPS 2023. They release code, data, and over a thousand adversarial abstracts. The human-subject study is preregistered, IRB-approved, and finds that adversarial abstracts draw more coherence complaints but no one suspects malicious intent, which gives colluders plausible deniability. That is a meaningful, reproducible contribution.\n\nThe soft spots are real but mostly acknowledged. The headline 'get assigned' is stronger than the evidence. The experiments measure the colluding reviewer's similarity rank among all reviewers (top-1/3/5), not whether the assignment solver actually assigns that reviewer. Load limits, conflicts, fairness constraints, and randomization—which the paper itself recommends as a defense—could reduce the real-world success rate. Section 6.2 concedes this. The fix is straightforward: soften the wording, or run an assignment-level simulation with typical constraints and see how often top-5 similarity translates into assignment. If the simulation still assigns in most cases, the concern is minor; if not, the headline claim needs to change.\n\nThe human-subject experiment also has a disclosed flaw: attack budgets were tuned on the NeurIPS 2023 test set rather than the 2022 training set. The authors argue this doesn't affect the detectability outcome, only the attack strength, and that argument is plausible but not airtight. It doesn't threaten the main result.\n\nThis paper deserves serious peer review. It identifies a real vulnerability that the community assumed away, and it does so with enough rigor and transparency that the finding can be checked and built upon. I'd accept it with revisions, not desk-reject it. I would cite it for the vulnerability result, not for the assignment-level claim.","headline":"SPECTER text-matching is gameable at scale; the 'get assigned' headline outruns the ranking-based evidence.","tokens_in":26063,"tokens_out":2900,"would_cite":true,"duration_ms":29997,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Colluding author-reviewer pairs can steer text-based reviewer assignment, lifting a target reviewer from rank 101 to the top 5 about 92% of the time on NeurIPS 2023 data.","keywords":["reviewer assignment","text-matching attacks","SPECTER","collusion rings","adversarial abstracts","peer review integrity","NeurIPS","archive curation"],"falsifier":"Run the same attack on a venue's actual assignment solver, with reviewer load caps, conflict lists, and any randomization included, and measure the fraction of manipulated top-5 pairs that are actually assigned; if that fraction is far below the reported 92%, the practical consequence of the claim would fail even though the ranking shift itself is real.","tokens_in":25049,"feed_emoji":"🎯","tokens_out":5129,"duration_ms":51827,"temperature":0.7,"pith_summary":"This paper tries to establish that the text-matching component of automated reviewer assignment at major ML/AI venues is not secure against collusion. It shows that a collaborating author and reviewer, without any bid manipulation, can raise the reviewer's computed similarity to the author's paper from a natural rank of 101 to the top 5 among all reviewers roughly 92% of the time on NeurIPS 2023 data. The attack works by having the reviewer curate their profile down to one highly similar past paper and the author add theme sentences and keywords to the abstract, using an LLM. If the paper is right, venues that disabled bidding to stop collusion rings remain exposed, and text similarity cannot be treated as a trusted signal.","feed_headline":"Text matching in reviewer assignment can be gamed 92% of the time","feed_subtitle":"A colluding author-reviewer pair lifts a paper from rank 101 to top-5 similarity without any bid manipulation, on NeurIPS 2023 data.","key_machinery":"The machinery is the cosine similarity in SPECTER's embedding space between a paper's title or abstract and each paper in a reviewer's archive, aggregated by either maximum or mean and then ranked across all reviewers. The attack exploits two control surfaces: the reviewer can adversarially shrink the archive to the single paper most similar to the target paper, and the author can add abstract sentences about the reviewer's themes and greedily selected keywords, both implemented with an LLM. These operations increase the embedding similarity without obviously changing what the paper is about.","core_discovery":"The central claim is that SPECTER, the text-embedding model used for reviewer-paper matching at major ML/AI venues, is practically manipulable by a colluding author-reviewer pair. Using only the publicly available model weights, the reviewer's past-publication profile, and the previous year's reviewer pool, the pair can substantially increase the colluding reviewer's similarity ranking for the target paper. In fully automatic experiments on 3,218 NeurIPS 2023 papers and 7,900 reviewers, the attack lifts a reviewer with natural rank 101 to top-5 in 93% of cases; with human oversight and early stopping the top-5 rate is 92%. Even without touching the abstract, selecting the single most similar past paper raises the reviewer to top-5 in 41% of cases. The paper interprets these numbers as evidence that the assumption that text matching is robust to manipulation is false.","pith_inferences":["If assignment optimizers add randomization or strict load and conflict constraints, the real-world assignment-level success rate could be lower than the reported top-5 ranking success, because the paper evaluates rankings rather than final assignments.","The same abstract-plus-profile manipulation could transfer to other text-based matching systems, such as journal or grant reviewer matching, that use comparable embedding models, though success rates would need to be measured.","A defense that averages similarity and caps how few papers a reviewer may keep may only shift attacks toward more sophisticated abstract edits, since the paper's evidence suggests a trade-off between robustness and matching quality."],"forward_implications":["Venues that removed bidding to thwart collusion rings can still be attacked through text matching alone.","Reviewer archive curation is a standalone vulnerability: keeping one highly similar paper can put the colluding reviewer in the top 5 about 41% of the time.","Max-pooling aggregation is more attack-prone than mean pooling, and imposing a minimum archive size or switching aggregation reduces success rates.","Attackers can tune their modifications on publicly available prior-year reviewer pools, because manipulated rankings correlate strongly (Spearman 0.62 to 0.93) between two consecutive NeurIPS editions.","Human reviewers rarely flag the manipulated abstracts as malicious, and LLM-paraphrased benign abstracts draw similar coherence complaints, giving colluders plausible deniability."],"supporting_citations":[{"why":"Supplies the SPECTER embedding model whose cosine similarities are the target of the attack.","marker":"[CFB+20]"},{"why":"Provides the earlier text-level attack on a topic-modeling matcher, which this paper extends to neural embeddings, larger scale, and reviewer-side curation.","marker":"[EQM+23]"},{"why":"Represents the common assumption that text similarity is ground truth while bids are manipulable, the assumption the paper argues against.","marker":"[WGW+21]"},{"why":"Proposes randomized reviewer assignments, which the paper points to as the mitigation strategy robust to this class of attack.","marker":"[JZL+20]"},{"why":"Documents the scale of ML/AI conferences and the norm of 3 to 6 reviewers per paper, motivating the top-1, top-3, and top-5 success metrics.","marker":"[Sha22]"},{"why":"Describes an automated paper-reviewer matching system with user-curated archives, the profile surface the attack exploits.","marker":"[CZ13b]"},{"why":"Introduces a successor embedding model that the paper reports in preliminary experiments also appears vulnerable.","marker":"[SDC+22]"}],"fun_headline_variants":["Text matching in reviewer assignments is gameable at scale","Colluding pairs exploit text similarity to get assigned papers","Peer review text matching: 93% attack success without bids","Reviewer assignment text scoring is fragile under collusion","Text-based reviewer matching is easily hijacked, study says"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that moving the colluding reviewer into the top-1, top-3, or top-5 similarity rankings is a faithful proxy for being assigned to review the paper, since the paper does not run the full assignment optimization with load limits, conflicts of interest, and randomization.","fun_headline_variants_meta":{"raw":{"variants":["Text matching in reviewer assignments is gameable at scale","Colluding pairs exploit text similarity to get assigned papers","Peer review text matching: 93% attack success without bids","Reviewer assignment text scoring is fragile under collusion","Text-based reviewer matching is easily hijacked, study says"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000296,"raw_usage":{"total_tokens":1706,"prompt_tokens":918,"completion_tokens":788,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":709}},"tokens_in":534,"tokens_out":788,"duration_ms":8318,"temperature":1.0,"reasoning_tokens":709,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:29:08.875844+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same attack on a venue's actual assignment solver, with reviewer load caps, conflict lists, and any randomization included, and measure the fraction of manipulated top-5 pairs that are actually assigned; if that fraction is far below the reported 92%, the practical consequence of the claim would fail even though the ranking shift itself is real.","supporting_citations":[],"review_version":1}