Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

White Hat Search Engine Optimization using Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Pith's one-line read LLM prompts fed past rankings can rewrite documents to climb search rankings.

desk verdict Useful empirical study of LLM-based document modification, but the headline significance claims are compromised by prompt selection on the same test sets; the held-out online competition gives partial redemption. read the letter →

arxiv 2502.07315 v2 pith:MPNN6TBW submitted 2025-02-11 cs.IR cs.GT

classification cs.IRcs.GT
keywords white-hatSEOsearchengineoptimizationlargelanguagemodelsprompt-baseddocumentmodificationrankingcompetitionLambdaMARTE5embeddingsScaledPromotion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes using large language models to modify documents so they rank higher in future search results, a form of white-hat search engine optimization. The central claim is that prompting the LLM with examples of past rankings for the query gives it enough implicit information about the undisclosed ranking function to make effective, targeted edits. The authors show that their best prompt configurations, called Pairwise and Listwise, outperform both human students and a strong feature-based baseline in terms of Scaled Promotion, while keeping the edited documents faithful to the original content and relevant. If this holds, document authors can use freely available LLM technology to improve rankings against rankers whose inner workings are hidden.

What carries the argument

The central mechanism is the context-specific part of the prompt, which supplies the LLM with past rankings. The Pairwise prompt gives the query along with pairs of documents and their ranks from the last three rounds, while the Listwise prompt gives the ranked list of documents from the latest and previous rounds. The general prompt instructs the LLM to edit the candidate document to improve its ranking, to focus on impactful sentences, to keep the edited length near the median corpus length, and to stay very similar to the original. This combination lets the LLM infer patterns in the undisclosed ranker and rewrite the document accordingly.

What would settle it

Run the same Pairwise and Listwise prompts on a ranking competition where the ranker is randomly replaced or heavily changed each round; if Scaled Promotion then drops to the level of a no-context prompt, the assumption of stable past rankings is what carries the method.

Watch

Extended reading notes

Core claim

The paper's central discovery is that LLM-based document modification, when the prompt includes pairs of documents and their relative ranks or full ranked lists from past rounds, can substantially improve a document's rank under an undisclosed ranking function. In offline evaluations over two ranking-competition datasets, one using LambdaMART and one using E5 embeddings, the Pairwise and Listwise bots achieved higher Scaled Promotion than both the student competitors and the SentReplace feature-based baseline. The modified documents retained high faithfulness to the original, measured by entailment-based metrics, and in an online competition the bots outperformed human students on rank promotion while producing documents of comparable or higher quality and relevance. The authors conclude that these findings attest to the clear merits of their prompt-based bots.

Load-bearing premise

The method depends on past rankings for a query being a reliable guide to what the undisclosed ranking function will do in the next round.

Editorial extensions

If this is right

  • If the central claim holds, document authors can improve their search rankings without knowing the ranking function, using only publicly observable past rankings and an LLM.
  • The success of Pairwise and Listwise prompts over Pointwise and Temporal prompts suggests that comparative ranking information is more useful for guiding document edits than single-document relevance hints.
  • Because the modified documents remain faithful and relevant by the paper's measures, white-hat SEO via LLMs could become a practical, low-cost alternative to manual content optimization.
  • The approach transfers across at least two very different rankers, one based on gradient-boosted trees with hand-crafted features and one based on dense embeddings, indicating some generality.
  • Search engines may need to consider that their ranking behavior over time can be exploited by authors who observe past rankings, even if the ranking function itself is never disclosed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not explore is whether the method still works when the ranking function drifts significantly between rounds; the assumption of stable past rankings would be violated and performance would likely degrade.
  • The authors' focus on white-hat modifications leaves open whether the same prompt-based approach could be used for manipulative black-hat SEO, which would be a direct and concerning application of the same technique.
  • One testable extension is to vary the number of past rounds and the number of example documents systematically to map how much ranking history is needed before the prompts lose their edge.
  • The paper evaluates only one LLM, GPT-4; comparing across LLMs would clarify whether the effect is a general property of LLM prompt-following or specific to that model's training.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes white-hat search engine optimization methods that use LLM prompting to modify documents so that they rank higher under an undisclosed ranking function. The prompts condition on past rankings in four context styles (Pointwise, Pairwise, Listwise, Temporal), and a configuration sweep over 192 prompt variants is used to select the best-performing Pairwise and Listwise bots. The authors evaluate the selected bots on two offline ranking-competition datasets (LambdaMARTComp and E5Comp) and in an online competition against human students, using Scaled Promotion, faithfulness to the original document, corpus faithfulness, and quality/relevance judgments. They report that the Pairwise and Listwise bots outperform students and the SentReplace feature-based baseline, and conclude that the results attest to the clear merits of the proposed methods.

Significance. If the empirical claims were fully supported, the paper would make a useful contribution to competitive search and SEO: it combines recent LLM ranking-prompt ideas with document modification, provides two offline competition datasets and a new online competition, and includes faithfulness measures and a comparison against a strong feature-based baseline. The availability of code and datasets for review is a strength, as is the effort to evaluate with human competitors who were not aware that they were playing against bots. However, the central quantitative claim is currently weakened by a model-selection issue and by several nonsignificant head-to-head comparisons, as detailed in the major comments. The contribution is therefore more incremental than the abstract suggests, and the empirical evidence needs substantial strengthening before the paper can be accepted.

major comments (4)
  1. [Section 3, 'Rank promotion performance of bots'] The selection of Pairwise and Listwise as the two best-performing prompt types is made on the same LambdaMARTComp and E5Comp data that are subsequently used in Tables 1 and 2, and the same selected bots are carried into the online evaluation of Table 3. Because the manuscript does not describe a held-out split over queries or any multiple-comparison correction for the 192-configuration sweep, the reported Scaled Promotion values and paired-permutation p-values are optimistically biased: even if every prompt configuration were useless, the maximum of 192 draws would be expected to beat the baselines on the selection data. This bias is load-bearing for the paper's central claim of 'clear merits'; I ask for a development/test split over queries (or a max-statistic permutation test), or at minimum a full report of the 192-configuration results in an appendix.
  2. [Table 1 and Table 3] In Table 1, neither the Pairwise (0.345) nor the Listwise (0.315) Scaled Promotion value is marked as significantly different from SentReplace (0.309), and in Table 3 the Listwise bot's online Scaled Promotion advantage over students (0.05 vs. -0.013) is not marked significant. These are the exact comparisons used in the abstract and conclusion to claim superiority, so the text currently overstates the statistical support; the claims should be limited to the comparisons that are significant, or the authors should provide a power analysis showing that the nonsignificant differences are within the test's resolution.
  3. [Section 3, footnote 12] The statement that SentReplace is not used 'here and after' because it was defined for sparse ranking functions conflicts with Table 1, where SentReplace is applied with LambdaMART, and with footnote 2, which says the authors replaced SVMrank with LambdaMART to improve SentReplace. This inconsistency should be resolved, and the reason SentReplace is not used in the E5Comp and online evaluations should be stated explicitly.
  4. [Section 2, opening paragraph] The method's mechanism rests on the assumption that past rankings for the query and other queries can be observed. The paper should explicitly discuss the scope of this assumption and ideally test sensitivity to ranker drift or degraded ranking history (e.g., by perturbing or omitting parts of the prompt context), since the prompts would lose their informative value if the ranking function changes substantially between rounds.
minor comments (5)
  1. [Section 3, 'Rank promotion performance of bots'] The sentence 'Actual numbers are omitted ... as they convey no additional insight' is not an adequate justification for hiding the results of the model-selection step; the numeric results of the 192-configuration sweep should be available in an appendix or repository.
  2. [Appendix A] The Listwise prompt contains a typo ('DOCUMNET') and the escaped formatting makes the templates difficult to read; a clean rendering would improve reproducibility.
  3. [Table 3 caption] The marker legend says significant differences are marked with '/u1D44F' (a superscript s), while Table 1's caption refers to 'r' and 's'; the markers should be typeset consistently and explained once.
  4. [Section 3, 'Online evaluation'] The online evaluation uses only 15 queries and three rounds; this should be reported as a limitation in Section 3, and the corresponding statistical power should be discussed.
  5. [General] Minor typos and garbled unicode appear throughout the text (e.g., 'migh t', 's earch', '/u1D451/u1D45A/u1D45C/u1D451'); the final version should be proofread.

Circularity Check

1 steps flagged · score 6.0 of 10

Offline results are selected from 192 prompt configurations on the same test sets used for evaluation, so the reported Scaled Promotion advantages are in-sample selection maxima rather than independent predictions.

  1. fitted input called prediction [Section 3, 'Rank promotion performance of bots' and Tables 1-2]
    "we found that the Pairwise bots (with random selection of document pairs) and the Listwise bots were the best performing for both the LambdaMARTComp and E5Comp datasets; the same specific instantiation of each of these two bots was always among the top-3 performing bots for both datasets. ... Actual numbers are omitted due to space considerations and as they convery no additional insight. In what follows, we present the evaluation of the two bots which posted for both datasets Scaled Promotion among the best three."

    The evaluated bot identities are not fixed in advance; they are the output of a search over 192 prompt configurations using the same LambdaMARTComp and E5Comp datasets on which Tables 1 and 2 later report Scaled Promotion. Under the null that no prompt is effective, the best of 192 trials will tend to exceed the baselines, so the reported Pairwise/Listwise advantages are in-sample selection maxima rather than independent evidence. The paper omits the sweep's numbers, saying they 'convey no additional insight', which prevents assessing the selection margin. The paired permutation tests are applied only to the selected bots, with no correction for the preceding screening.

full rationale

The paper's central derivation chain is empirical: LLM prompts modify documents, and Scaled Promotion is computed from rankings induced over modified and next-round documents. That measure is not defined in terms of the prompt, and the faithfulness measures are computed from external NLI and retrieval models, so those components are not circular. The main circularity concern is selection on the evaluation data. Section 3 reports a sweep over 192 prompt configurations, selects Pairwise and Listwise as the best performers on both benchmark datasets, and then Tables 1 and 2 report the selected bots' scores on those same datasets. The reported advantage over SentReplace and students is therefore the result of maximizing over the test data, and the significance tests do not account for this screening. This is a fitted-input-called-prediction pattern for the offline claims. However, the online competition in Table 3 uses the selected bots against students on new queries, providing some independent support for the central claim. The self-citations to prior work by the same authors for the baseline and datasets are not themselves load-bearing, since those resources are external artifacts used for comparison. Score 6 reflects partial circularity of the offline evaluation, not a full reduction of the method to its inputs.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim does not introduce new physical entities. It rests on observable past rankings, a fixed ranker, LLM output quality, and an NLI faithfulness measure. The prompt hyperparameters and temperature were selected by searching over 192 configurations on the test datasets, which is the main fitting risk and is the reason these are entered as free parameters.

free parameters (3)
  • Prompt hyperparameter configuration = Pairwise: 1 query, 3 past rounds, 1 random pair per round, temperature 0.5.
    Selected as best among 192 prompt instantiations based on test-dataset Scaled Promotion. This selection is a form of fitting to the evaluation data and carries multiple-comparison risk.
  • LLM temperature = Evaluated over {0, 0.5, 1, 1.5, 2}; best settings were 0 for Listwise and 0.5 for Pairwise.
    Temperature is a hand-tuned generation parameter chosen on the test data. It affects output drift and hence both ranking promotion and faithfulness.
  • Number of retrieved corpus documents k = 10
    Hand-set for the corpus faithfulness measurement in Section 3. It influences the reported faithfulness scores, though not the Scaled Promotion results directly.
assumptions (4)
  • domain assumption Past rankings for the query and other queries can be observed by the document author before modifying the document.
    Stated in Section 2: 'We assume that past rankings for the query and other queries can be observed.' The entire context-specific prompt design depends on this access.
  • domain assumption The undisclosed ranking function is stable across rounds, so past rankings are informative for next-round ranking.
    Implicit in the evaluation setup using round 7 of LambdaMARTComp and round 4 of E5Comp. If the ranker changed between rounds, the prompt context would lose its predictive signal.
  • domain assumption The LLM produces high-quality discourse and follows the instruction to stay faithful to the original document.
    Stated in Section 2: 'We assume that the LLM produces high quality content in terms of discourse.' The faithfulness and quality claims rely on this assumption.
  • domain assumption TrueTeacher NLI scores with a 0.5 threshold reliably indicate factual entailment and faithfulness.
    Used in Section 3 to define OrigFaith and corpus faithfulness. If this measure is miscalibrated, the reported faithfulness values are not meaningful.

how reviews work

0 comments
Cite this review

Pith. "Pith review of White Hat Search Engine Optimization using Large Language Models." pith.science (2026). https://pith.science/paper/MPNN6TBW

@misc{pith2026250207315,
  author       = {Pith},
  title        = {Pith review of: White Hat Search Engine Optimization using Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MPNN6TBW}},
  note         = {Machine review of arXiv:2502.07315}
}
read the original abstract

We present novel white-hat search engine optimization techniques based on genAI and demonstrate their empirical merits.

Figures

Figures reproduced from arXiv: 2502.07315 by the authors.

Figure 1
Figure 1. Context-specific part of the Pairwise prompt which [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Context-specific part of the Listwise prompt which [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. On the Merits of LLM-Based Corpus Enrichment

    cs.IR 2025-06 conditional novelty 5.0 of 10

    LLM-generated, query-biased documents added to a search corpus improve retrieval effectiveness, RAG answer accuracy, and answer attribution in proof-of-concept experiments that use oracle-selected source documents.

Reference graph

Works this paper leans on

24 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    Introducing Connect by CloudResearch: Advancing On- line Participant Recruitment in the Digital Age | Request PD F

    2024. Introducing Connect by CloudResearch: Advancing On- line Participant Recruitment in the Digital Age | Request PD F. https://doi.org/10.31234/osf.io/ksgyr

  2. [2]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad , Ilge Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sa m Altman, Shya- mal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

  3. [3]

    Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit , Ashwin Kalyan, Karthik Narasimhan, and Ameet Deshpande. 2024. GEO: Genera tive Engine Optimization. In Proceedings of KDD . ACM, 5–16

  4. [4]

    Christopher JC Burges. 2010. From ranknet to lambdarank to lambdamart: An overview. Learning 11, 23-581 (2010), 81

  5. [5]

    Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elki nd, and Idan Szpek- tor. 2023. TrueTeacher: Learning Factual Consistency Eval uation with Large Language Models. arXiv:2305.11171 [cs.CL]

  6. [6]

    Gregory Goren, Oren Kurland, Moshe Tennenholtz, and Fia na Raiber. 2020. Ranking-incentivized quality preserving content modifica tion. In Proceedings of SIGIR. 259–268

  7. [7]

    Gregory Goren, Oren Kurland, Moshe Tennenholtz, and Fia na Raiber. 2021. Driv- ing the Herd: Search Engines as Content Influencers. In Proceedings of CIKM . Virtual Event Queensland Australia, 586–595

  8. [8]

    Zoltán Gyöngyi and Hector Garcia-Molina. 2005. Web Spam Taxonomy. In Pro- ceedings of AIRWeb 2005 . 39–47

Show all 24 references
  1. [9]

    Aounon Kumar and Himabindu Lakkaraju. 2024. Manipulati ng Large Language Models to Increase Product Visibility. CoRR abs/2404.07981 (2024)

  2. [10]

    Oren Kurland and Moshe Tennenholtz. 2022. Competitive Search. In Proceedings of SIGIR. 2838–2849

  3. [11]

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipr as, Dilara Soylu, Michi- hiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, andAnanya Kumar

  4. [12]

    Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. 2023. Zero- shot listwise document reranking with a large language mode l. arXiv preprint arXiv:2305.02156 (2023)

  5. [13]

    Tommy Mordo, Itamar Reinman, Moshe Tennenholtz, and Or en Kurland. 2025. Search results diversification in competitive search. arXi v:2501.14922

  6. [14]

    Fredrik Nestaas, Edoardo Debenedetti, and Florian Tra mèr. 2024. Adversarial Search Engine Optimization for Large Language Models. CoRR abs/2406.18382 (2024)

  7. [15]

    Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. 2020. D ocument ranking with a pretrained sequence-to-sequence model. arXiv preprint arXiv:2003.06713 (2020)

  8. [16]

    Max Peeperkorn, Tom Kouwenhoven, Dan Brown, and Anna Jo rdanous

  9. [17]

    Samuel Pfrommer, Yatong Bai, Tanmay Gautam, and Somaye h Sojoudi. 2024. Ranking Manipulation for Conversational Search Engines. I n Proceedings of EMNLP. 9523–9552

  10. [18]

    Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, et al. 2 023. Large lan- guage models are effective text rankers with pairwise rankin g prompting. arXiv preprint arXiv:2306.17563 (2023)

  11. [19]

    Nimrod Raifer, Fiana Raiber, Moshe Tennenholtz, and Or en Kurland. 2017. In- formation retrieval meets game theory: The ranking competi tion between doc- uments’ authors. In Proceedings of SIGIR . 465–474

  12. [20]

    Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and J ason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021 . 3784–3803

  13. [22]

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Lin jun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2024. Text Embeddings by Weakly-Supervised Contrastive Pre-training. arXiv:2212.03533

  14. [23]

    Chen Wu, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yi xing Fan, and Xueqi Cheng. 2023. PRADA: Practical Black-box Adversarial Attac ks against Neural Ranking Models. ACM Trans. Inf. Syst. 41, 4 (2023), 89:1–89:27

  15. [2022]

    arXiv:arXiv :2211.09110

    Holistic evaluation of language models. arXiv:arXiv :2211.09110

  16. [2024]

    Is Temperature the Creativity Parameter of Large Lang uage Models? https://doi.org/10.48550/arXiv.2405.00492 arXiv:2405 .00492 [cs]

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.