{"id":"5fa22194-6e78-4c06-a44f-7fac32cfc699","arxiv_id":"2508.04024","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"OpenReview staff report 94 fraudulent reviewer profiles in AI conferences, used by dishonest researchers to favorably review their own papers.","lead":"An internal OpenReview investigation found 94 fake reviewer profiles used to manipulate AI conference peer review. The paper describes how attackers impersonated real researchers, often using university email aliases, and what safeguards could stop it.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 94-profile count and the identity-theft modus operandi rest entirely on an undisclosed OpenReview internal investigation; no decision rule, evidence standard, or case-level data is provided, so the central claim is currently unfalsifiable.","rationale":"The paper's central claim is a specific, high-impact number: 94 fake reviewer profiles with a described modus operandi. A reader can only evaluate this if the claim is backed by checkable evidence. The manuscript is transparent about the source ('investigated by OpenReview staff') but gives no methods, no data, and no case-level detail. The weakest point is therefore not the internal consistency of the argument (the narrative is coherent) but the empirical foundation: the 94-classification is a black-box assertion. This is exactly the reader's weakest assumption. I do not allege misconduct by the authors; an internal investigation may be perfectly sound. The issue is that the preprint currently functions as a whistleblower report rather than a scientifically assessable study. The recommendations (credential linkage, vouching, deduplication, alias monitoring) are reasonable and low-risk, and the paper's framing as an alert is legitimate. But because the headline count cannot be independently checked, the appropriate status is conditional: treat it as a credible alarm, but do not treat the count or the modus operandi as established until the evidentiary basis is disclosed or audited. My stress-test does not change the reader's conditional verdict.","tokens_in":6476,"tokens_out":4675,"duration_ms":52818,"concrete_test":"Request from OpenReview a redacted, independent-audit dataset listing, for each of the 94 alleged fake profiles: the claimed identity, the institution/email domain, the evidence category (e.g., email-alias creation log, IP/browser fingerprint match to an author account, admission by the accused, confirmation from the impersonated person), and the linked author account. Have a third-party team apply a pre-registered classification rule to independently recreate the count and the modus operandi. If the independent count is not near 94 or the evidence categories do not support impersonation of another specific researcher, the central claim is not established. A lighter-weight first step: publish a methodology appendix stating the exact decision rule and the distribution of evidence types across the 94 cases.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise is in the 'Findings of fraud' section: 'This investigation unearthed 94 reviewer (and meta-reviewer) profiles involving fake identities.' For this to support the paper's central claim, OpenReview staff must have had reliable ground truth on each profile: (1) that the profile impersonated a specific real person, and (2) that the controlling researcher was an author seeking favorable reviews. The manuscript provides neither an operational definition of 'fake identity' nor the evidence standard used. It states only that the cases 'were identified and investigated by OpenReview staff' and required 'extensive time'; no detection methodology, decision rules, case-level evidence, or inter-rater reliability is reported. The round-trip-verified email detail shows only that whoever clicked the verification link controlled the address; it does not establish that the controller was another specific researcher or that the profile was tied to an author account. If the classification was based on systematic, checkable criteria (e.g., IP overlaps, email-alias logs, admission), the count might be reproducible; if based on informal judgment, it may not be. As written, the 94 number is an assertion from an interested party with no audit trail, making the central claim unfalsifiable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports that OpenReview's internal investigation identified 94 reviewer and meta-reviewer profiles with fake identities across several AI conferences in 2024–2025. The alleged modus operandi is that dishonest researchers impersonated real researchers (including via university email aliases) to be recruited as reviewers, then bid for their own papers and wrote favorable reviews. The paper proposes identity-verification and anti-fraud measures for venues, platforms, and universities, and warns that similar vulnerabilities affect recommendation letters and other academic processes.","tokens_in":6721,"tokens_out":4033,"duration_ms":50105,"significance":"If the central factual claim were substantiated, it would document a serious, scalable attack on peer review at major AI venues and would justify the proposed safeguards. The paper's strength is that it publicizes a concrete vulnerability and offers sensible mitigations. However, the empirical core is an assertion by OpenReview-affiliated authors based on an undisclosed investigation; no methodology, detection criteria, case-level evidence, or independent verification is provided. The paper therefore functions at present as a high-level incident alert rather than a verifiable scientific report.","major_comments":[{"comment":"The central claim rests entirely on the statement that 'This investigation unearthed 94 reviewer (and meta-reviewer) profiles involving fake identities.' No operational definition of 'fake identity' is given, no detection methodology or decision rule is described, and no case-level evidence or audit trail is presented. A round-trip-verified email address only proves that someone controlled the mailbox; it does not establish that the controller was a specific dishonest researcher or that the profile impersonated a specific real person. Without these details, the 94 count and the modus operandi are unfalsifiable.","section":"Findings of fraud (94-profile count)"},{"comment":"The abstract claims that the fraudulent profiles were used 'to manipulate paper evaluations,' but the bullets only describe how a dishonest researcher 'attempts to get assigned' and, once assigned, 'provides favorable reviews.' No aggregate or case-level information is reported on how many of the 94 profiles actually received assignments to the target papers, wrote reviews, or influenced decisions. If none or few did, the severity and the 'identity theft' framing would need to be revised. This is load-bearing for the paper's central claim.","section":"Findings of fraud (assignment/review outcomes)"},{"comment":"The statement that 'In all 94 cases, the fake reviewer profiles included a round-trip-verified email' and that 92 addresses pertained to 'reputed universities' is not enough to support the conclusion that the dishonest researcher 'gained access to an email of a trusted institution' and 'created email alias(es) resembling someone else.' The investigation's method for linking an alias to a specific individual and for ruling out legitimate duplicate accounts or misconfigurations is not reported. A clear, reproducible classification protocol and redacted examples are needed.","section":"Findings of fraud (email-alias inference)"}],"minor_comments":[{"comment":"The paper lacks a Methods or Data Availability section; even the section headings are not formatted as conventional sections. Please add an explicit description of the investigation period, data sources, and anonymization procedures.","section":"Overall structure"},{"comment":"Define 'meta-reviewer' and 'reputed universities' for a general scientific audience.","section":"Terminology"},{"comment":"The recommendations, while sensible, are not prioritized or evaluated; some are said to be already implemented on OpenReview, but no before/after data are given.","section":"Recommendations"},{"comment":"References [7] and [10] are informal sources; consider citing published versions or adding access dates.","section":"References"},{"comment":"The phrase 'in all 94 cases' could be read as involving 94 emails; please clarify whether the count refers to profiles, unique individuals, or email addresses.","section":"Counting clarity"}],"recommendation":"major_revision","confidential_remarks":"This is a high-visibility claim from the platform's own leadership. Even with the best intentions, the absence of an auditable methodology makes the paper unsuitable for publication as a research report in its current form. If the authors can provide de-identified case-level evidence (e.g., redacted profiles, email-exchange logs, assignment records) to an independent auditor, or include a detailed protocol, the paper could become a valuable record. The editor may also consider whether the journal has a mechanism for 'incident reports' that do not require the same evidence standard."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead this paper if you care about peer-review integrity in AI. The headline finding — 94 reviewer/meta-reviewer profiles with fake identities, many using institutional email aliases — is new and important. It is the first platform-level documentation of this specific attack in AI conferences, and the email-alias variant is a genuinely new detail. The recommendations are sensible, and several are already implemented. The authors are candid that these are investigations by OpenReview staff, not the result of a public experiment.\n\nThe central number is as soft as it is load-bearing. The findings section states the 94 count without an operational definition of fake identity, no detection criteria, no case-level evidence, and no decision rule. The round-trip-verified email detail shows the controller clicked a link, not that the controller was another named researcher with an author account. That does not undermine the modus operandi — it is plausible and consistent with known collusion tactics — but it means the reader cannot check the classification. For an interested-party report, that is a real limitation. The paper itself does not flag this gap; it presents the count as established.\n\nTwo things keep this from being a reason to toss the paper. First, the authors are the platform, with unique access to logs, IP overlaps, and admissions that no outsider has. It is not unreasonable to give them some benefit of the doubt. Second, the practical threat model holds even if the precise 94 is off: the attack is possible, verified-emails do not protect against it, and university alias policies make it worse. The paper's value is in demonstrating a vulnerability, not in the exact numerator.\n\nWhat is missing is a methodology appendix or an independent audit that could establish the count. Even a redacted list of criteria — \"profile created via alias matching the impersonated person's name; account linked to author's IP; self-citation pattern\" — would let the community assess the evidentiary standard. Without that, the headline claim remains an assertion, though a credible one.\n\nThe citation pattern is fine. Prior collusion-ring work is directly relevant, and the authors' own survey is the right background. Self-citation is not a problem here because the cited work is real and related.\n\nBottom line: this is a solid, important report that should go through peer review with the methodology made explicit. I would send it to reviewers, not desk reject it, and I would require the authors to add a transparency appendix or explain why that is impossible. It is also worth bringing to a reading group precisely because the evidence-standard question is a good test of how we treat platform self-reports.","headline":"A credible alarm about fake reviewer identities at AI conferences, but the paper's core count is an assertion from an interested party with no audit trail.","tokens_in":7200,"tokens_out":1922,"would_cite":true,"duration_ms":22783,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper reports that platform staff uncovered 94 reviewer and meta-reviewer profiles created under fake identities to steer favorable reviews to attackers' own papers.","keywords":["peer review integrity","identity theft","fake reviewer profiles","AI conferences","reviewer fraud","email alias abuse","academic misconduct"],"falsifier":"An independent audit of the 94 flagged profiles: for each, contact the person whose identity was used and the administrators of the university email domain, and check whether the profile can be traced to an actual dishonest actor. If even a few flagged profiles turn out to be legitimate reviewers misidentified by the internal investigation, or if none of the impersonations can be verified by the affected individuals, the central claim fails.","tokens_in":6355,"feed_emoji":"🎭","tokens_out":7098,"duration_ms":85653,"temperature":0.7,"pith_summary":"The paper claims that between February and April 2024 and again in April 2025, staff of the nonprofit platform that runs reviews for many top AI conferences uncovered 94 reviewer and meta-reviewer profiles built on fake identities. According to the paper, the people behind these profiles impersonated real researchers—using their affiliations and publication histories plus email addresses at trusted universities that passed verification—to get assigned to review their own submitted papers and write favorable reviews. If the finding is correct, a verified identity-theft attack has been operating inside AI peer review, exploiting the open recruitment forms, self-created reviewer profiles, round-trip email verification, and bidding mechanisms meant to build a qualified review pool. The paper argues the vulnerability is not confined to one venue, because many venues reuse reviewer lists from previous years, so undetected impersonators can be re-invited automatically. Its purpose is to alert organizers and platforms so they can adopt the proposed safeguards.","feed_headline":"Fake reviewer identities found in 94 AI conference profiles","feed_subtitle":"Platform staff say researchers impersonated others to steer favorable reviews to their own papers.","key_machinery":"The mechanism that makes the fraud work is the unverified reviewer profile combined with a round-trip-verified email alias. The paper's account depends on open-call reviewer recruitment, where program chairs check eligibility from the applicant's self-reported seniority and publication history, and on semi-automated assignment that lets reviewers bid on papers and be matched by similarity. The fake profile supplies the right surface credentials—another researcher's papers and affiliation—while the email alias at a trusted university domain passes the platform's verification check, so the attacker enters the pool as a legitimate reviewer and can then exploit bidding or text-based matching to","core_discovery":"The central claim is that dishonest researchers have created fraudulent reviewer profiles in multiple AI conferences by taking over another researcher's identity—submitting reviewer signup forms or platform profiles with the victim's affiliation and publication record while using an email address the attacker controls. All 94 fake profiles used an email address that passed round-trip verification; 92 drew on email domains of reputable universities and two used `.edu` domains of defunct institutions, in many cases because universities allow members and visitors to generate aliases resembling other people. Once recruited, the attacker bid for or tuned their profile to be assigned to papers aut","pith_inferences":["The paper's count says nothing about how many papers received fraudulent favorable reviews; knowing that number would determine whether the reported cases are an isolated nuisance or a systematic distortion of accept/reject decisions.","A testable consequence of the email-alias claim is that university alias policies are the strongest single enabler; platforms that require a verified non-alias institutional address or real-name matching should show lower fake-profile rates, and a comparison across platforms could quantify this.","The same loophole applies to recommendation-letter workflows in admissions and hiring, where the applicant controls the recommender's contact information; this is the paper's own analogy, and it suggests the fraud pattern can be detected by checking whether the recommender's email alias and letter metadata match.","The paper proposes transparency in publicizing investigations and outcomes; if adopted, the resulting dataset of confirmed cases could be used to build automated flagging models, but this is an extension the paper does not develop."],"forward_implications":["If the 94-case finding is taken at face value, conference organizers should treat institutional email domains as weak identity evidence: in the paper's data, 92 of 94 fraudulent profiles used round-trip-verified email addresses from reputable universities.","Automated re-use of previous reviewer lists means an undetected impersonator is not a one-time event; the same fake profile can be re-invited to future editions unless profiles are deduplicated and re-verified.","The paper's proposed fixes—linking reviewer credentials to verified prior publications, requiring login through a persistent identifier system, vouching for new reviewers, and scrutinizing profiles created just before deadlines—follow directly from the observed modus operandi.","Because open-call recruitment is common outside AI, the same vulnerability plausibly affects other fields and platforms, though the paper only documents AI conferences."],"supporting_citations":[{"why":"Documents organized abuse of the peer-review system outside AI, used as precedent that such manipulation is known.","marker":"[2]"},{"why":"Prior work on identity theft in academia producing junk science; the specific harm this paper extends.","marker":"[3]"},{"why":"Early reporting of a peer-review scam, establishing that review fraud is an observed phenomenon.","marker":"[4]"},{"why":"Supplies the mechanism by which an attacker tunes paper text to increase assignment probability to their own paper.","marker":"[5]"},{"why":"Provides the bidding and randomized-assignment framework; the paper cites it both for how reviewers signal interest and for a partial mitigation.","marker":"[6]"},{"why":"Documents collusion rings in computer science, a related fraud pattern the new findings connect to.","marker":"[7]"},{"why":"Provides the account of reviewer-paper assignment and prior misconduct categories that the paper's vulnerability analysis relies on.","marker":"[9]"},{"why":"Public report of suspected organized fraud in a conference review process, another precedent for the pattern.","marker":"[10]"}],"fun_headline_variants":["94 fake reviewer profiles used to rig AI peer review","Identity theft in AI conferences: 94 fake reviewers","Researchers stole identities for 94 fake reviewer profiles","Fake reviewer IDs used to rig AI peer review in 94 cases","94 fraudulent reviewer profiles found in AI conferences"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that the platform's staff correctly classified all 94 profiles as fraudulent—each one truly created by a dishonest researcher rather than a false positive—because the paper gives no detection methodology, case-level evidence, or independent audit; if that classification errs, the count and the inferred modus operandi collapse.","fun_headline_variants_meta":{"raw":{"variants":["94 fake reviewer profiles used to rig AI peer review","Identity theft in AI conferences: 94 fake reviewers","Researchers stole identities for 94 fake reviewer profiles","Fake reviewer IDs used to rig AI peer review in 94 cases","94 fraudulent reviewer profiles found in AI conferences"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00067,"raw_usage":{"total_tokens":2799,"prompt_tokens":559,"completion_tokens":2240,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":303,"completion_tokens_details":{"reasoning_tokens":2177}},"tokens_in":303,"tokens_out":2240,"duration_ms":18096,"temperature":1.0,"reasoning_tokens":2177,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T00:55:26.512564+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An independent audit of the 94 flagged profiles: for each, contact the person whose identity was used and the administrators of the university email domain, and check whether the profile can be traced to an actual dishonest actor. If even a few flagged profiles turn out to be legitimate reviewers misidentified by the internal investigation, or if none of the impersonations can be verified by the affected individuals, the central claim fails.","supporting_citations":[{"cited_title":"and Webb, A.J., 2016","cited_arxiv_id":null,"evidence_quote":"Documents organized abuse of the peer-review system outside AI, used as precedent that such manipulation is known."},{"cited_title":"and Borchardt, G., 2018","cited_arxiv_id":null,"evidence_quote":"Prior work on identity theft in academia producing junk science; the specific harm this paper extends."},{"cited_title":"and Oransky, I., 2014","cited_arxiv_id":null,"evidence_quote":"Early reporting of a peer-review scam, establishing that review fraud is an observed phenomenon."},{"cited_title":"Vulnerability of Text-Matching in ML/AI Conference Reviewer Assignments to Collusions","cited_arxiv_id":"2412.06606","evidence_quote":"Supplies the mechanism by which an attacker tunes paper text to increase assignment probability to their own paper."},{"cited_title":"and Fang, F., 2020","cited_arxiv_id":null,"evidence_quote":"Provides the bidding and randomized-assignment framework; the paper cites it both for how reviewers signal interest and for a partial mitigation."},{"cited_title":"Collusion rings threaten the integrity of computer science research","cited_arxiv_id":null,"evidence_quote":"Documents collusion rings in computer science, a related fraud pattern the new findings connect to."},{"cited_title":"Challenges, Experiments, and Computational Solutions in Peer Review (Extended Version)","cited_arxiv_id":null,"evidence_quote":"Provides the account of reviewer-paper assignment and prior misconduct categories that the paper's vulnerability analysis relies on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Public report of suspected organized fraud in a conference review process, another precedent for the pattern."}],"review_version":1}