REVIEW 4 major objections 5 minor 1 cited by
White Hat Search Engine Optimization using Large Language Models
T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read LLM prompts fed past rankings can rewrite documents to climb search rankings.
desk verdict Useful empirical study of LLM-based document modification, but the headline significance claims are compromised by prompt selection on the same test sets; the held-out online competition gives partial redemption. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the context-specific part of the prompt, which supplies the LLM with past rankings. The Pairwise prompt gives the query along with pairs of documents and their ranks from the last three rounds, while the Listwise prompt gives the ranked list of documents from the latest and previous rounds. The general prompt instructs the LLM to edit the candidate document to improve its ranking, to focus on impactful sentences, to keep the edited length near the median corpus length, and to stay very similar to the original. This combination lets the LLM infer patterns in the undisclosed ranker and rewrite the document accordingly.
What would settle it
Run the same Pairwise and Listwise prompts on a ranking competition where the ranker is randomly replaced or heavily changed each round; if Scaled Promotion then drops to the level of a no-context prompt, the assumption of stable past rankings is what carries the method.
Extended reading notes
Core claim
The paper's central discovery is that LLM-based document modification, when the prompt includes pairs of documents and their relative ranks or full ranked lists from past rounds, can substantially improve a document's rank under an undisclosed ranking function. In offline evaluations over two ranking-competition datasets, one using LambdaMART and one using E5 embeddings, the Pairwise and Listwise bots achieved higher Scaled Promotion than both the student competitors and the SentReplace feature-based baseline. The modified documents retained high faithfulness to the original, measured by entailment-based metrics, and in an online competition the bots outperformed human students on rank promotion while producing documents of comparable or higher quality and relevance. The authors conclude that these findings attest to the clear merits of their prompt-based bots.
Load-bearing premise
The method depends on past rankings for a query being a reliable guide to what the undisclosed ranking function will do in the next round.
Editorial extensions
If this is right
- If the central claim holds, document authors can improve their search rankings without knowing the ranking function, using only publicly observable past rankings and an LLM.
- The success of Pairwise and Listwise prompts over Pointwise and Temporal prompts suggests that comparative ranking information is more useful for guiding document edits than single-document relevance hints.
- Because the modified documents remain faithful and relevant by the paper's measures, white-hat SEO via LLMs could become a practical, low-cost alternative to manual content optimization.
- The approach transfers across at least two very different rankers, one based on gradient-boosted trees with hand-crafted features and one based on dense embeddings, indicating some generality.
- Search engines may need to consider that their ranking behavior over time can be exploited by authors who observe past rankings, even if the ranking function itself is never disclosed.
Reading between the lines
- A natural extension the paper does not explore is whether the method still works when the ranking function drifts significantly between rounds; the assumption of stable past rankings would be violated and performance would likely degrade.
- The authors' focus on white-hat modifications leaves open whether the same prompt-based approach could be used for manipulative black-hat SEO, which would be a direct and concerning application of the same technique.
- One testable extension is to vary the number of past rounds and the number of example documents systematically to map how much ranking history is needed before the prompts lose their edge.
- The paper evaluates only one LLM, GPT-4; comparing across LLMs would clarify whether the effect is a general property of LLM prompt-following or specific to that model's training.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes white-hat search engine optimization methods that use LLM prompting to modify documents so that they rank higher under an undisclosed ranking function. The prompts condition on past rankings in four context styles (Pointwise, Pairwise, Listwise, Temporal), and a configuration sweep over 192 prompt variants is used to select the best-performing Pairwise and Listwise bots. The authors evaluate the selected bots on two offline ranking-competition datasets (LambdaMARTComp and E5Comp) and in an online competition against human students, using Scaled Promotion, faithfulness to the original document, corpus faithfulness, and quality/relevance judgments. They report that the Pairwise and Listwise bots outperform students and the SentReplace feature-based baseline, and conclude that the results attest to the clear merits of the proposed methods.
Significance. If the empirical claims were fully supported, the paper would make a useful contribution to competitive search and SEO: it combines recent LLM ranking-prompt ideas with document modification, provides two offline competition datasets and a new online competition, and includes faithfulness measures and a comparison against a strong feature-based baseline. The availability of code and datasets for review is a strength, as is the effort to evaluate with human competitors who were not aware that they were playing against bots. However, the central quantitative claim is currently weakened by a model-selection issue and by several nonsignificant head-to-head comparisons, as detailed in the major comments. The contribution is therefore more incremental than the abstract suggests, and the empirical evidence needs substantial strengthening before the paper can be accepted.
major comments (4)
- [Section 3, 'Rank promotion performance of bots'] The selection of Pairwise and Listwise as the two best-performing prompt types is made on the same LambdaMARTComp and E5Comp data that are subsequently used in Tables 1 and 2, and the same selected bots are carried into the online evaluation of Table 3. Because the manuscript does not describe a held-out split over queries or any multiple-comparison correction for the 192-configuration sweep, the reported Scaled Promotion values and paired-permutation p-values are optimistically biased: even if every prompt configuration were useless, the maximum of 192 draws would be expected to beat the baselines on the selection data. This bias is load-bearing for the paper's central claim of 'clear merits'; I ask for a development/test split over queries (or a max-statistic permutation test), or at minimum a full report of the 192-configuration results in an appendix.
- [Table 1 and Table 3] In Table 1, neither the Pairwise (0.345) nor the Listwise (0.315) Scaled Promotion value is marked as significantly different from SentReplace (0.309), and in Table 3 the Listwise bot's online Scaled Promotion advantage over students (0.05 vs. -0.013) is not marked significant. These are the exact comparisons used in the abstract and conclusion to claim superiority, so the text currently overstates the statistical support; the claims should be limited to the comparisons that are significant, or the authors should provide a power analysis showing that the nonsignificant differences are within the test's resolution.
- [Section 3, footnote 12] The statement that SentReplace is not used 'here and after' because it was defined for sparse ranking functions conflicts with Table 1, where SentReplace is applied with LambdaMART, and with footnote 2, which says the authors replaced SVMrank with LambdaMART to improve SentReplace. This inconsistency should be resolved, and the reason SentReplace is not used in the E5Comp and online evaluations should be stated explicitly.
- [Section 2, opening paragraph] The method's mechanism rests on the assumption that past rankings for the query and other queries can be observed. The paper should explicitly discuss the scope of this assumption and ideally test sensitivity to ranker drift or degraded ranking history (e.g., by perturbing or omitting parts of the prompt context), since the prompts would lose their informative value if the ranking function changes substantially between rounds.
minor comments (5)
- [Section 3, 'Rank promotion performance of bots'] The sentence 'Actual numbers are omitted ... as they convey no additional insight' is not an adequate justification for hiding the results of the model-selection step; the numeric results of the 192-configuration sweep should be available in an appendix or repository.
- [Appendix A] The Listwise prompt contains a typo ('DOCUMNET') and the escaped formatting makes the templates difficult to read; a clean rendering would improve reproducibility.
- [Table 3 caption] The marker legend says significant differences are marked with '/u1D44F' (a superscript s), while Table 1's caption refers to 'r' and 's'; the markers should be typeset consistently and explained once.
- [Section 3, 'Online evaluation'] The online evaluation uses only 15 queries and three rounds; this should be reported as a limitation in Section 3, and the corresponding statistical power should be discussed.
- [General] Minor typos and garbled unicode appear throughout the text (e.g., 'migh t', 's earch', '/u1D451/u1D45A/u1D45C/u1D451'); the final version should be proofread.
Circularity Check
Offline results are selected from 192 prompt configurations on the same test sets used for evaluation, so the reported Scaled Promotion advantages are in-sample selection maxima rather than independent predictions.
-
fitted input called prediction
[Section 3, 'Rank promotion performance of bots' and Tables 1-2]
"we found that the Pairwise bots (with random selection of document pairs) and the Listwise bots were the best performing for both the LambdaMARTComp and E5Comp datasets; the same specific instantiation of each of these two bots was always among the top-3 performing bots for both datasets. ... Actual numbers are omitted due to space considerations and as they convery no additional insight. In what follows, we present the evaluation of the two bots which posted for both datasets Scaled Promotion among the best three."
The evaluated bot identities are not fixed in advance; they are the output of a search over 192 prompt configurations using the same LambdaMARTComp and E5Comp datasets on which Tables 1 and 2 later report Scaled Promotion. Under the null that no prompt is effective, the best of 192 trials will tend to exceed the baselines, so the reported Pairwise/Listwise advantages are in-sample selection maxima rather than independent evidence. The paper omits the sweep's numbers, saying they 'convey no additional insight', which prevents assessing the selection margin. The paired permutation tests are applied only to the selected bots, with no correction for the preceding screening.
full rationale
The paper's central derivation chain is empirical: LLM prompts modify documents, and Scaled Promotion is computed from rankings induced over modified and next-round documents. That measure is not defined in terms of the prompt, and the faithfulness measures are computed from external NLI and retrieval models, so those components are not circular. The main circularity concern is selection on the evaluation data. Section 3 reports a sweep over 192 prompt configurations, selects Pairwise and Listwise as the best performers on both benchmark datasets, and then Tables 1 and 2 report the selected bots' scores on those same datasets. The reported advantage over SentReplace and students is therefore the result of maximizing over the test data, and the significance tests do not account for this screening. This is a fitted-input-called-prediction pattern for the offline claims. However, the online competition in Table 3 uses the selected bots against students on new queries, providing some independent support for the central claim. The self-citations to prior work by the same authors for the baseline and datasets are not themselves load-bearing, since those resources are external artifacts used for comparison. Score 6 reflects partial circularity of the offline evaluation, not a full reduction of the method to its inputs.
Assumptions & free parameters
free parameters (3)
- Prompt hyperparameter configuration =
Pairwise: 1 query, 3 past rounds, 1 random pair per round, temperature 0.5.
- LLM temperature =
Evaluated over {0, 0.5, 1, 1.5, 2}; best settings were 0 for Listwise and 0.5 for Pairwise.
- Number of retrieved corpus documents k =
10
assumptions (4)
- domain assumption Past rankings for the query and other queries can be observed by the document author before modifying the document.
- domain assumption The undisclosed ranking function is stable across rounds, so past rankings are informative for next-round ranking.
- domain assumption The LLM produces high-quality discourse and follows the instruction to stay faithful to the original document.
- domain assumption TrueTeacher NLI scores with a 0.5 threshold reliably indicate factual entailment and faithfulness.
Cite this review
Pith. "Pith review of White Hat Search Engine Optimization using Large Language Models." pith.science (2026). https://pith.science/paper/MPNN6TBW
@misc{pith2026250207315,
author = {Pith},
title = {Pith review of: White Hat Search Engine Optimization using Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/MPNN6TBW}},
note = {Machine review of arXiv:2502.07315}
}
read the original abstract
We present novel white-hat search engine optimization techniques based on genAI and demonstrate their empirical merits.
Figures
Forward citations
Cited by 1 Pith paper
-
On the Merits of LLM-Based Corpus Enrichment
LLM-generated, query-biased documents added to a search corpus improve retrieval effectiveness, RAG answer accuracy, and answer attribution in proof-of-concept experiments that use oracle-selected source documents.
Reference graph
Works this paper leans on
-
[1]
2024. Introducing Connect by CloudResearch: Advancing On- line Participant Recruitment in the Digital Age | Request PD F. https://doi.org/10.31234/osf.io/ksgyr
-
[2]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad , Ilge Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sa m Altman, Shya- mal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)
arXiv 2023
-
[3]
Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit , Ashwin Kalyan, Karthik Narasimhan, and Ameet Deshpande. 2024. GEO: Genera tive Engine Optimization. In Proceedings of KDD . ACM, 5–16
work page 2024
-
[4]
Christopher JC Burges. 2010. From ranknet to lambdarank to lambdamart: An overview. Learning 11, 23-581 (2010), 81
2010
-
[5]
Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elki nd, and Idan Szpek- tor. 2023. TrueTeacher: Learning Factual Consistency Eval uation with Large Language Models. arXiv:2305.11171 [cs.CL]
arXiv 2023
-
[6]
Gregory Goren, Oren Kurland, Moshe Tennenholtz, and Fia na Raiber. 2020. Ranking-incentivized quality preserving content modifica tion. In Proceedings of SIGIR. 259–268
work page 2020
-
[7]
Gregory Goren, Oren Kurland, Moshe Tennenholtz, and Fia na Raiber. 2021. Driv- ing the Herd: Search Engines as Content Influencers. In Proceedings of CIKM . Virtual Event Queensland Australia, 586–595
work page 2021
-
[8]
Zoltán Gyöngyi and Hector Garcia-Molina. 2005. Web Spam Taxonomy. In Pro- ceedings of AIRWeb 2005 . 39–47
work page 2005
Show all 24 references
-
[9]
Aounon Kumar and Himabindu Lakkaraju. 2024. Manipulati ng Large Language Models to Increase Product Visibility. CoRR abs/2404.07981 (2024)
2024 arXiv
-
[10]
Oren Kurland and Moshe Tennenholtz. 2022. Competitive Search. In Proceedings of SIGIR. 2838–2849
2022
-
[11]
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipr as, Dilara Soylu, Michi- hiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, andAnanya Kumar
-
[12]
Xueguang Ma, Xinyu Zhang, Ronak Pradeep, and Jimmy Lin. 2023. Zero- shot listwise document reranking with a large language mode l. arXiv preprint arXiv:2305.02156 (2023)
2023 arXiv
-
[13]
Tommy Mordo, Itamar Reinman, Moshe Tennenholtz, and Or en Kurland. 2025. Search results diversification in competitive search. arXi v:2501.14922
2025 arXiv
-
[14]
Fredrik Nestaas, Edoardo Debenedetti, and Florian Tra mèr. 2024. Adversarial Search Engine Optimization for Large Language Models. CoRR abs/2406.18382 (2024)
2024 arXiv
-
[15]
Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. 2020. D ocument ranking with a pretrained sequence-to-sequence model. arXiv preprint arXiv:2003.06713 (2020)
2020 arXiv
-
[16]
Max Peeperkorn, Tom Kouwenhoven, Dan Brown, and Anna Jo rdanous
-
[17]
Samuel Pfrommer, Yatong Bai, Tanmay Gautam, and Somaye h Sojoudi. 2024. Ranking Manipulation for Conversational Search Engines. I n Proceedings of EMNLP. 9523–9552
2024
-
[18]
Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, et al. 2 023. Large lan- guage models are effective text rankers with pairwise rankin g prompting. arXiv preprint arXiv:2306.17563 (2023)
2023 arXiv
-
[19]
Nimrod Raifer, Fiana Raiber, Moshe Tennenholtz, and Or en Kurland. 2017. In- formation retrieval meets game theory: The ranking competi tion between doc- uments’ authors. In Proceedings of SIGIR . 465–474
2017
-
[20]
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and J ason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021 . 3784–3803
2021
-
[22]
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Lin jun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2024. Text Embeddings by Weakly-Supervised Contrastive Pre-training. arXiv:2212.03533
2024 arXiv
-
[23]
Chen Wu, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yi xing Fan, and Xueqi Cheng. 2023. PRADA: Practical Black-box Adversarial Attac ks against Neural Ranking Models. ACM Trans. Inf. Syst. 41, 4 (2023), 89:1–89:27
2023
- [2022]
- [2024]
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.