{"id":"1d1775b0-e564-4fd8-8511-ac95f631ad4b","arxiv_id":"2412.16382","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A sentence-level embedding perturbation attack can promote target documents into top-10 neural ranking results, but its headline success depends on a surrogate ranker despite claiming to be surrogate-free.","lead":"EMPRA is a black-box attack that rewrites target documents by nudging sentence embeddings toward query-related anchors and inserting the best rewritten sentence to boost rank. The authors report moving up to 96% of target documents from rank 51-100 into the top 10, but that headline number uses a surrogate ranker rather than the surrogate-free setting claimed in the abstract.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 96% top-10 result comes from the in-distribution surrogate, not the surrogate-agnostic setting the abstract emphasizes; the surrogate-free number is 74.3%.","rationale":"The reader's conditional verdict is well supported. The weakest assumption they name, the unspecified transformer function Pi, is a genuine reproducibility gap: Section 4.1 never identifies the model, training data, or decoding accuracy of Pi, so the iterative embedding-to-text loop cannot be independently assessed from the manuscript alone. If the released code supplies a working Pi, this concern is resolvable. However, I see an even more direct load-bearing problem for the central claim as worded: the \"almost 96%\" top-10 figure is achieved with M_S1, a surrogate trained on the victim's query distribution, while the surrogate-agnostic setting the paper advertises yields 74.3% on Easy-5 and 35.1% on Hard-5. This is not a matter of implementation detail; it is an internal inconsistency between the abstract's unqualified claims and the experimental tables. The paper still makes a meaningful contribution if the claims are restricted to the appropriate configuration, which is why I would not move the verdict to reject. The reader already flagged the headline overstatement in the rationale, so my read does not change the conditional verdict; it sharpens the specific test that would force the authors to qualify the claim.","tokens_in":26178,"tokens_out":6956,"duration_ms":64815,"concrete_test":"Use the released repository to reproduce Table 3 rows labelled M_Gbest on the 1,000-query, 10,000-document Easy-5 and Hard-5 protocol, and also report %r<=10 for every M_Gi row in Table 2. If the surrogate-free configuration yields 74.3% Easy-5 and 35.1% Hard-5 (or at most 85.2% Easy-5) rather than 95.6% and 64.9%, the abstract and Section 7 must be revised to state that \"almost 96%\" holds only when the in-distribution surrogate M_S1 is used.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract pairs \"almost 96%\" of documents ranked 51-100 moving into the top 10 with the statement that EMPRA \"does not depend on surrogate models for adversarial text generation.\" In the paper's own results, the 95.6% Easy-5 value is produced by M_S1, an in-distribution surrogate trained on 6,837 queries from the victim model's distribution (Section 5.3.2), not by a generic model. The best generic configuration highlighted in Table 3, M_Gbest, obtains 74.3% Easy-5 and 35.1% Hard-5; among all generic models in Table 2, the maximum Easy-5 %r<=10 is 85.2% (M_G1), still below 96%. Thus the two headline claims are in tension: the flagship number depends on exactly the surrogate dependence the paper claims to remove. This is an internal-consistency problem with the central claim, not a dispute with outside consensus. The reader's concern about the unspecified transformer function Pi is also real, but the more immediately falsifiable load-bearing issue is that the reported \"almost 96%\" cannot be attributed to the surrogate-agnostic configuration without qualification.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EMPRA, a black-box attack against neural ranking models. It operates in two stages: sentence embeddings of a target document are iteratively perturbed toward anchor embeddings (the query, the top-ranked document, or a similar sentence from that document) by a transporter function and decoded back to text by a transformer function; the generated texts are then inserted at all positions of the target document, and the final adversarial document is selected by an interpolated score combining BERT NSP coherence and relevance from a generic NRM. Experiments on MS MARCO V1 with the MiniLM victim ranker compare EMPRA with Query+, GPT-4, PRADA, PAT, Brittle-BERT, and IDEM, reporting the best attack metrics overall and low perplexity. The abstract claims that almost 96% of target documents originally ranked 51-100 are moved into the top 10 and that the method does not depend on surrogate models for adversarial text generation.","tokens_in":26472,"tokens_out":8200,"duration_ms":67249,"significance":"If the results are reproducible, EMPRA is a useful contribution to adversarial IR: it demonstrates a practical black-box attack that outperforms several recent baselines while maintaining readability, and the authors release code and experimental data. The significance is currently limited by two unresolved issues: the key decoding component Pi is not specified, and the headline 96% figure is produced by the in-distribution surrogate M_S1 rather than by the generic/surrogate-agnostic configuration emphasized in the abstract. These issues must be addressed before the central claims can be accepted.","major_comments":[{"comment":"The abstract and Section 1 state that EMPRA 'successfully achieves a re-ranking of almost 96% of target documents originally ranked between 51-100 to rank within the top 10' and that it 'does not depend on surrogate models.' In the paper's own results, the 95.6% Easy-5 value is obtained with M_S1, an in-distribution surrogate trained on 6,837 queries from the victim model's distribution (Section 5.3.2), not with a generic model. The best generic configuration in Table 2 is M_G4, which gives 74.3% Easy-5 and 35.1% Hard-5. The reported 96% therefore belongs to the surrogate-aware setting, not to the surrogate-agnostic setting the abstract emphasizes; the abstract should be reworded and the surrogate-agnostic results should be reported as the headline or explicitly separated.","section":"Abstract, Section 6.1, Tables 2 and 3"},{"comment":"The transformer function Pi is the load-bearing component of the method, but the paper never identifies the model, its training data, its loss, or its decoding accuracy. Eq. (4) and Algorithm 1 line 9 rely on Pi to map every perturbed embedding E(S)^(t+1) back to fluent lexical text whose own embedding matches the target. Without specifying Pi, the core generative claim cannot be checked or reproduced; the authors should provide the implementation details and a quantitative evaluation of reconstruction fidelity.","section":"Section 4.1, Eqs. (4)-(5), Algorithm 1"},{"comment":"The gradient notation in Eq. (4) is ambiguous: the derivative is written as partial derivative with respect to S while S denotes a discrete sentence, yet the quantity inside the norm is E(S)^(t) dot E(A) over the product of norms. If the gradient is taken with respect to the continuous embedding coordinates, this should be stated explicitly; if it is taken with respect to text, the expression is not well-defined. The iterative refinement of a textual hypothesis H(i) described in Section 4.1 also does not appear in the loop of Algorithm 1, so the actual decoding procedure is unclear.","section":"Eq. (4)"},{"comment":"The claim that EMPRA 'does not depend on surrogate models for adversarial text generation' is too broad. In Stage 2, Eq. (8) and Algorithm 1 lines 22-24 use a generic NRM M_G to score every candidate by query relevance and to select the final adversarial document. This is a dependence on an external ranking model even if M_G is not trained on the victim's pseudo-labels. The authors should either relax the wording or provide an analysis showing that the candidate selection does not drive the reported attack success.","section":"Section 4.2, Eq. (8), Algorithm 1"}],"minor_comments":[{"comment":"In the Hard-5 block the column header 'Readibility' is misspelled and should be 'Readability'.","section":"Table 3"},{"comment":"The text states that Brittle-BERT's boosted top-10 decreases 'from 81.3% to 33.2% and 43.4%'; Table 3 reports 43.4% for M_Gbest on Easy-5, so the 33.2 value appears to be a typo.","section":"Section 6.2 (ii)"},{"comment":"The perturbation bound epsilon and step size eta are fixed at 0.01 and 0.1 respectively without a sensitivity analysis; Section 6.4 ablates only the number of iterations and the interpolation coefficient alpha, not epsilon or eta.","section":"Section 5.3.5 and Section 6.4"},{"comment":"The captions of Figures 2 and 3 do not identify the plotted metric or the curve labels, and the text does not describe the axes; please expand the captions or explain the figures in the body.","section":"Figures 2 and 3"},{"comment":"The human evaluation uses only two annotators on the 32 Mixture documents, and the reported kappa values are very low (for example 0.01 for fluency), so the imperceptibility and fluency conclusions should be presented with appropriate caution.","section":"Section 6.5 and Table 6"},{"comment":"The ACM reference format line in the header lists the year 2018 while the submission is from 2024/2025; this should be corrected.","section":"Manuscript header"}],"recommendation":"major_revision","confidential_remarks":"The work is potentially publishable after a major revision. The two central problems are fixable: first, restate the abstract and Section 1 to distinguish the surrogate-dependent 95.6% result from the generic-model results; second, specify and evaluate the transformer function Pi. I do not see a reason to reject outright, because the experiments are extensive and the code/data release is valuable. I would also ask the authors to provide error bars or significance tests, since many ASR values are near 100% and the observed differences between methods may be small relative to variance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: EMPRA is a real advance over the prior attack baselines, and the method deserves a serious look. But the headline numbers are tied to an in-distribution surrogate, and the decoder Pi is too underspecified for the central generative loop to be verified.\n\nWhat is actually new: the anchor set (query, top-ranked document, most similar sentence) combined with iterative sentence-embedding perturbation and decoding back to text is a genuine combination not present in PRADA, PAT, Brittle-BERT, or IDEM. The insertion-stage selection using NSP coherence plus a generic ranker is thoughtful. The evaluation is wide: three victim models, Easy-5 and Hard-5 targets, six quality metrics, and the authors say they release code and data. Taken as a surrogate-dependent attack, the 95.6% Easy-5 top-10 result with M_S1 is worth taking seriously.\n\nWhere it softens: the abstract pairs \"almost 96%\" with \"does not depend on surrogate models.\" Table 2 shows that 95.6% comes from M_S1, an in-distribution surrogate trained on 6,837 queries from the same distribution as the victim. The best generic model in Table 3, M_G4, gives 74.3% on Easy-5 and 35.1% on Hard-5; even the best Table 2 generic for attack performance, M_G1 at 85.2%, is below 96%. Also, the paper's own choice to define M_Gbest by MRR@10 rather than by attack success is odd for a robustness claim. This internal inconsistency is the load-bearing issue, not an outside disagreement.\n\nThe \"surrogate-agnostic\" claim is also overstated in a subtler way: Stage 2 uses a generic ranker to score and select the inserted sentence, so the attack is not free of model dependence, just free of victim-specific surrogate training. That weaker claim is still interesting, but the abstract should say it.\n\nThe bigger reproducibility hole is Pi. Section 4.1 describes iterative refinement of a textual hypothesis, but neither Algorithm 1 nor the experimental setup identifies Pi, its training data, or its decoding accuracy. Without that, Eq. (4)'s perturb-and-decode loop is not checkable. A reader cannot tell whether EMPRA is a robust embedding-space method or a trick that relies on one unpublished decoder.\n\nMinor but real: no error bars anywhere, and the human fluency kappa in Table 6 is 0.01. Two annotators with no agreement cannot support the \"imperceptible to humans\" claim. The automatic fluency metrics are fine; just do not lean on the human assessment.\n\nBottom line: this deserves a serious referee. The missing Pi details and the abstract overclaim are fixable, and the attack idea is strong enough that the IR security community needs to know whether it holds up. I would send it to review, with requests for decoder specification, error bars, and an honest abstract that distinguishes the surrogate-dependent and generic-model results.","headline":"EMPRA is a genuinely new attack combination with strong evaluations, but the headline 96% number depends on an in-distribution surrogate and the core decoder is left underspecified.","tokens_in":26979,"tokens_out":3015,"would_cite":false,"duration_ms":28597,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A black-box attack rewrites documents to promote 96% of targets into the top 10.","keywords":["adversarial attacks","neural ranking models","black-box attack","embedding perturbation","document re-ranking","surrogate-agnostic","MS MARCO","imperceptible text"],"falsifier":"Measure the reconstruction fidelity of the transformer function on embeddings perturbed exactly as in Eq. (4): for a held-out set of sentences, push the embeddings toward anchor embeddings at the reported $\\epsilon=0.01$ and step size $\\eta=0.1$, decode with $\\Pi$, and compute the distance between the decoded text's embedding and the perturbed target embedding. If that distance is often larger than $\\epsilon$, the perturbation loop is not actually moving the decoded text toward the anchor, and the attack should fail; conversely, replacing $\\Pi$ with a deliberately weak decoder should measurably reduce the 95.6% top-10 rate reported for the surrogate setting.","tokens_in":25996,"feed_emoji":"🎯","tokens_out":7372,"duration_ms":58093,"temperature":0.7,"pith_summary":"The paper introduces EMPRA, a method for black-box adversarial attacks on neural ranking models: by rewriting a target document's sentences, it aims to make a victim ranker place that document much higher than before. The claimed mechanism is to work directly in sentence-embedding space, nudging each sentence's embedding toward the query and toward the top-ranked document, then decoding the perturbed embeddings back into natural-language text. On the MS MARCO passage collection the method reports re-ranking almost 96% of documents originally ranked 51–100 into the top 10, and about 65% of documents ranked 996–1000 into the top 10, while keeping perplexity and readability close to the original. A distinguishing claim is that no surrogate model needs to be trained against the victim: a generic out-of-distribution relevance model suffices, which the authors argue makes the attack more robust across victim rankers.","feed_headline":"Black-box attack promotes 96% of targeted documents to top 10","feed_subtitle":"Embedding-space text rewriting ranks low documents high without surrogate models, with fluent output.","key_machinery":"The machinery is the pair (transporter, transformer) acting on sentence embeddings. The transporter computes a normalized gradient of the dot product between the sentence embedding and an anchor embedding, steps along it with size $\\eta$, and clips the result to an $\\ell_\\infty$ ball of radius $\\epsilon$ around the original embedding; the transformer $\\Pi$ converts that perturbed embedding back into text (Eq. 4). Iterating $N$ times produces candidate adversarial sentences. A second stage inserts each candidate at every position of the document and scores the resulting document with an interpolated measure $\\alpha C_{\\mathrm{coh}} + (1-\\alpha) C_{\\mathrm{rel}}$, where coherence comes from a next-sentence-prediction function and relevance from a generic neural ranker; the highest-scoring candidate becomes the adversarial document. The use of a generic ranker rather than a surrogate imitation is what the paper calls 'surrogate-agnostic'.","core_discovery":"The core discovery is that an effective ranking attack can be generated without imitating the victim model: instead, the attacker perturbs sentence embeddings in a direction that points toward query-relevant anchors, and then converts those perturbed embeddings back into text. EMPRA's transporter function moves each sentence embedding iteratively toward the embeddings of anchor texts—the query, the current top-ranked document, and the sentence in the top document most similar to the target sentence—under an $\\ell_\\infty$ constraint. Its transformer function maps each intermediate embedding back to lexical form, and re-embedding the decoded text closes the loop. The rewritten sentence is inserted into the target document at the position that best satisfies a combined score of next-sentence coherence and relevance as judged by a generic neural ranker. The paper reports that this pipeline outperforms existing word-, trigger-, and prompt-based attacks on MS MARCO, with the largest gains on documents that start far from the top.","pith_inferences":["The paper leaves the transformer function $\\Pi$ unspecified; a replication study should measure how reconstruction error of $\\Pi$ grows with perturbation radius and iteration count, since that error bounds whether the gradient signal actually reaches the final text.","The same embedding-perturbation pipeline could be inverted to demote documents by moving their sentence embeddings away from the query anchors, turning EMPRA into a two-directional content manipulation tool.","Because the attack's success is measured on one dataset (MS MARCO) and one family of cross-encoder rankers, a direct test on a different corpus or on generative/retrieval-augmented rankers would reveal whether the reported 96% top-10 rate is a property of the method or of the benchmark."],"forward_implications":["Neural ranking models that rely on cross-encoders appear vulnerable to an attacker who only rewrites documents fluently, without any imitation of the target model.","Documents ranked in the long tail can be promoted into the top 10 with a single rewritten sentence insertion, implying current rankers place too much weight on the locally added context.","The attack's imperceptibility metrics suggest that fluency-based defenses (perplexity filters, linguistic-acceptability classifiers) will not catch this family of manipulations.","If EMPRA's results hold across victim models, then any public generic ranker suffices to launch the attack, so search operators cannot protect themselves by withholding their own ranking model.","Defense would need to target the embedding-space perturbation itself, for instance by training rankers that are robust to sentence-level embedding shifts, rather than detecting trigger tokens."],"supporting_citations":[{"why":"Defines the IDEM prompt-based baseline and the Easy/Hard target-document evaluation setup that EMPRA uses and outperforms.","marker":"[8]"},{"why":"Provides the PAT trigger-based attack baseline and the 'Mixture' target-document evaluation design.","marker":"[30]"},{"why":"Provides the Brittle-BERT trigger-based baseline that EMPRA compares against in attack and fluency metrics.","marker":"[59]"},{"why":"Provides the PRADA word-substitution surrogate-based attack that motivates EMPRA's surrogate-agnostic claim.","marker":"[65]"},{"why":"Supplies the MS MARCO V1 passage collection on which all attack and quality results are measured.","marker":"[36]"},{"why":"Supplies the GPT-2 language model used to compute perplexity for the fluency evaluation.","marker":"[45]"}],"fun_headline_variants":["Rank attack lifts 96% of targets into top 10","Embedding tweaks re-rank 96% of docs to top 10","No-surrogate attack boosts 96% of docs to top 10","EMPRA: stealthy rank attack without surrogate models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The attack only works if the transformer function can reliably write out fluent sentences whose embeddings match the perturbed target embeddings; if decoding perturbed embeddings is lossy, the iterative refinement loses its guiding signal and the reported rank boosts would not occur.","fun_headline_variants_meta":{"raw":{"variants":["Rank attack lifts 96% of targets into top 10","Embedding tweaks re-rank 96% of docs to top 10","No-surrogate attack boosts 96% of docs to top 10","EMPRA: stealthy rank attack without surrogate models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000931,"raw_usage":{"total_tokens":3974,"prompt_tokens":923,"completion_tokens":3051,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":2977}},"tokens_in":539,"tokens_out":3051,"duration_ms":17926,"temperature":1.0,"reasoning_tokens":2977,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:37:45.325983+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the reconstruction fidelity of the transformer function on embeddings perturbed exactly as in Eq. (4): for a held-out set of sentences, push the embeddings toward anchor embeddings at the reported $\\epsilon=0.01$ and step size $\\eta=0.1$, decode with $\\Pi$, and compute the distance between the decoded text's embedding and the perturbed target embedding. If that distance is often larger than $\\epsilon$, the perturbation loop is not actually moving the decoded text toward the anchor, and the attack should fail; conversely, replacing $\\Pi$ with a deliberately weak decoder should measurably reduce the 95.6% top-10 rate reported for the surrogate setting.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the PAT trigger-based attack baseline and the 'Mixture' target-document evaluation design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Brittle-BERT trigger-based baseline that EMPRA compares against in attack and fluency metrics."}],"review_version":1}