Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

Beyond SEO: A Transformer-Based Approach for Reinventing Web Content Optimisation

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a fine-tuned BART-base transformer, trained on 1,905 synthetic travel-page rewrites, can make web content more visible in generative search answers, reporting a 30.96% gain in position-adjusted cited word count…

desk verdict A clear, honest proof-of-concept that fine-tuning a small seq2seq model can learn GEO-style rewriting, but the headline visibility gain is not yet established because the same model family generated the labels and scored the outputs. read the letter →

arxiv 2507.03169 v1 pith:CC3PUIHI submitted 2025-07-03 stat.ML cs.LG

Florian Lüttgenau , Imar Colic , Gervasio Ramirez This is my paper · ORCID
classification stat.MLcs.LG
keywords generativeengineoptimizationGEOBARTfine-tuningwebcontentvisibilitytraveldomainsynthetictrainingdatalargelanguagemodelevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to establish that generative engine optimization is tractable with small-scale, domain-specific fine-tuning rather than requiring large black-box LLM intervention at query time. It fine-tunes BART-base on 1,905 pairs of raw travel website content and Llama-3.3-70B-generated optimized versions that add citations, statistics, and fluency improvements. In a controlled visibility test, replacing one of five unoptimized sources with the model's rewrite increases that source's cited word count by 15.63% absolutely and by 30.96% when weighted by citation position. A sympathetic reader would care because it suggests publishers with modest compute can adapt their content for AI-driven search without accessing proprietary optimization APIs.

What carries the argument

The argument is carried by supervised fine-tuning of the pre-trained BART-base model on synthetic (raw, optimized) content pairs. Labels are produced by a three-step Llama-3.3-70B pipeline that adds credible-sounding citations, improves linguistic fluency, and places statistics strategically, following the GEO strategies of Aggarwal et al. The visibility result is measured by two word-count metrics over a generative response: absolute word count of sentences citing a source, and position-adjusted word count that weights citations by a linear decay over the response, so earlier citations count more. The 30.96% headline improvement is the position-adjusted metric.

What would settle it

Take the same 50-query test set and run the identical optimized-versus-unoptimized swap experiment with a different generative engine, such as GPT-4o or Gemini, computing absolute and position-adjusted word counts. If the BART-optimized pages show no improvement or a loss in that engine, the conclusion that fine-tuning enhances visibility in generative search engines holds only for Llama-style citation behavior, not as a general result.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a standard BART-base sequence-to-sequence transformer can learn the citation-friendly rewriting style that defines GEO, rather than having to be applied as a prompt-time black-box strategy. After supervised fine-tuning, the model raises ROUGE-L from 0.226 to 0.249 and BLEU from 0.173 to 0.200 relative to baseline BART, and these textual gains translate into downstream visibility gains when Llama-3.3-70B answers queries: 15.63% more words cite the optimized source, and 30.96% more when earlier citations are weighted more heavily. The authors present this as the first end-to-end demonstration that a medium-size transformer, lightly fine-tuned on synthetic data, can boost content discoverability in generative search under modest compute.

Load-bearing premise

The load-bearing premise is that visibility measured by a 50-query controlled experiment with Llama-3.3-70B, the same model family that generated the training labels, transfers to real generative search engines and to actual user attention.

Editorial extensions

If this is right

  • Content producers in travel, and by extension other content-rich domains, can fine-tune a small open-source transformer on a few thousand synthetic pairs and measure visibility gains in LLM answers without multi-billion-parameter models or proprietary retrieval infrastructure.
  • Because the model learns to combine citations, statistics, and fluency, multi-strategy GEO can be distilled into a single offline rewrite step rather than applied per query by a large language model.
  • The larger encoder context window (384 tokens versus 256) appears to contribute to faster convergence and lower validation loss, suggesting input truncation is a meaningful design choice for content transformation.
  • The larger relative gain in position-adjusted versus absolute word count implies the rewrites do more than earn extra citations; they push those citations earlier into the answer, where user attention is higher.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the training labels and the visibility evaluator are both Llama-3.3-70B, the measured gain may partly reflect conformity to that model's citation preferences; testing with other generative engines would reveal how much of the effect generalizes.
  • The same synthetic-labelling and fine-tuning recipe could be reused per domain, and the near-doubling of the position-adjusted gain over the absolute gain suggests real click-through effects could exceed the word-count numbers if earlier citations attract more attention.
  • A natural next step, which the paper notes, is to replace the supervised imitation objective with reinforcement learning that directly optimizes the position-adjusted word-count metric; the current BART model may be leaving further visibility on the table.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a GEO (Generative Engine Optimization) approach for travel web content by fine-tuning a BART-base transformer on 1,905 synthetic pairs of raw website text and LLM-generated optimized rewrites. The rewrites are produced with Llama-3.3-70B using three strategies: credible-sounding citations, fluency improvements, and statistical evidence, with structured headings and paragraphs. The fine-tuned model is evaluated intrinsically with ROUGE-L, BLEU, perplexity, and length ratio against BART-base, and extrinsically in a controlled Llama-3.3-70B visibility experiment over 50 queries, reporting a 15.63% absolute word-count improvement and a 30.96% position-adjusted word-count improvement. The paper concludes that small-scale, domain-focused fine-tuning can meaningfully enhance content visibility in generative search engines.

Significance. If the results were robust, the paper would provide a useful proof-of-concept that a moderate-size sequence-to-sequence model can be fine-tuned with modest compute to improve content visibility in LLM-based retrieval settings, and it would extend the GEO framework of Aggarwal et al. (2024) to the travel domain. The controlled paired evaluation design and the transparent discussion of small sample size are strengths, as is the attempt to combine multiple GEO strategies in a realistic way. However, the current evidence is not yet sufficient for the central claim: the visibility evaluation uses the same model family that generated the training labels, the intrinsic comparison confounds fine-tuning with changes in context window and decoding constraints, and the visibility statistics are not accompanied by significance testing or a prespecified analysis plan. The paper is better positioned as a reproducible case study than as the 'first empirical demonstration' of fine-tuning for GEO.

major comments (4)
  1. [Section 3.3 and Section 7.2] The visibility evaluation uses Llama-3.3-70B, the same model family that generated the training labels and, via the 8B model, the queries in Section 3.1. Because the fine-tuned BART is trained to imitate Llama's rewriting style, including the citation and statistics patterns, the reported 30.96% position-adjusted gain may largely reflect the evaluator's preference for text that resembles its own output, rather than a property rewarded by generative search engines generally. This is a circularity threat to the headline claim. I recommend re-running the visibility protocol with at least one independent model family (e.g., GPT-4o, Claude, or Gemini), and ideally with human relevance judgments or user-attention proxies, and reporting results separately per model.
  2. [Section 5.2 and Section 6.1, Table 3] The intrinsic comparison between the 'Baseline' and 'Proposed' models is not an ablation of fine-tuning. The proposed model uses an encoder/decoder context window of 384 tokens while the baseline uses MAX_IN=256 and MAX_OUT=448, and the proposed decoding adds a length penalty alpha=1.1 and no-repeat-3-gram blocking. Consequently, the ROUGE-L, BLEU, and perplexity differences in Table 3 could be caused by the longer context or the decoding constraints rather than by fine-tuning. Please repeat the comparison with identical context windows and identical decoding parameters, or explicitly ablate each component.
  3. [Section 7.2] The visibility claim rests on 50 queries with no significance testing, no confidence intervals, and unspecified normalization and outlier-exclusion procedures. The paper itself acknowledges the sample is too small for statistical significance. Please report the full per-query distribution of the two metrics, a paired significance test (e.g., Wilcoxon signed-rank), the exact normalization formula, the outlier rule applied, and how many queries were excluded. In addition, clarify whether the 'randomly selected' optimized document was varied across repetitions and whether the evaluator prompt was identical in both conditions.
  4. [Section 3.3 and Section 8] The optimized content includes 'credible-sounding citations' that are not necessarily real or verifiable. If the generated citations are fabricated, then the visibility gains measured in a simulated setting may not transfer to real generative search engines that check source validity, and the proposed optimization method raises integrity concerns. Please clarify whether citations are real or synthetically generated, and discuss how fabricated citations would be treated in a deployed GEO system.
minor comments (6)
  1. [Abstract and Introduction] There are several wording and grammar errors, including 'lead by Google' (should be 'led by Google') and 'generative AI-driven search models powered such as ChatGPT Search' (missing 'by' or rephrasing).
  2. [Section 5.1 and Section 7.2] The relationship between the 'disjoint test set composed of 50 unseen queries' in Section 5.1 and the 'subset of 50 evaluation queries was created from the original set of 250' in Section 7.2 is ambiguous. Please state explicitly whether the visibility queries are exactly the 50 unseen test queries or are drawn from the full query set.
  3. [Section 7.1] The claim that 'Training loss sits above validation loss for both models because the training split contains systematically longer passages whose input side is artificially truncated' is unusual and not explained with evidence. Please clarify the mechanism or remove the claim, since training loss above validation loss typically signals a different issue.
  4. [Section 7.2] In the position-adjusted word-count formula, the notation pos(s) and |S| is not fully defined. Please specify whether |S| is the total number of sentences in the response and how sentence positions are indexed.
  5. [Section 4.8 and Section 9] The paper references local notebook paths (e.g., 'the notebook/2 Deep Learning Model/BART from scratch/OurBART.ipynb') and includes a 'Statement about individual contributions' section. For an archival paper, these should be replaced by a persistent repository link and a standard acknowledgments statement.
  6. [Figures] Figures 1, 2, and 3 are referenced but not included in the manuscript text provided; please ensure all figures are present and legible in the final version.

Circularity Check

1 steps flagged · score 5.0 of 10

Visibility gain is not independent evidence: BART is trained on Llama-3.3-70B rewrites, then scored by Llama-3.3-70B citation behavior.

  1. fitted input called prediction [Section 3.3 (Labelling) and Section 7.2 (GEO Evaluation)]
    "Our optimisation pipeline employed Llama-3.3-70B-Instruct-Turbo-Free via the together.aiAPI in a systematic three-step process. (1) Credible-sounding citations were integrated to enhance content authority. ... We queried Llama-3.3-70B via API in two conditions: (1) using only the five unoptimised sources and (2) randomly replacing one with its optimised version."

    The supervised labels defining 'optimised content' were produced by Llama-3.3-70B-Instruct-Turbo-Free, including inserted citations, statistics, and headings. The extrinsic evaluation then measures how often Llama-3.3-70B cites a source, as absolute and position-adjusted word counts. BART was trained solely to reproduce those Llama rewrites, so its output is aligned, by construction of the training data, with the evaluator's own stylistic preferences. The headline +30.96% gain thus conflates 'content that Llama prefers' with 'content that generative search engines generally reward': the label generator and the visibility scorer are the same model family.

full rationale

The paper's intrinsic results (ROUGE-L, BLEU, PPL) are computed against references that are themselves Llama-3.3-70B rewrites, so they measure fidelity to the label generator's style; this is expected for supervised fine-tuning and is not itself circular. The load-bearing external claim — that fine-tuning 'enhances web content visibility in generative search engines' — is evaluated with Llama-3.3-70B citation counts, the same model family that generated the training labels. That coupling means the 15.63% and 30.96% visibility gains can be substantially explained by stylistic self-preference rather than by properties that other engines would reward. I find no self-citation chain: the GEO framework and metrics are credited to Aggarwal et al. (2024), an external source, and no uniqueness theorem or author-imported ansatz is invoked. The n=50 sample, token constraints, and noisy website-query matches are acknowledged by the authors as limitations, but the label/evaluator overlap is not. Overall, the derivation is not circular in the strict equation-identity sense, but the central visibility prediction is partially circular because its training input and its measurement instrument are the same model family. Score 5 reflects this partial, non-forced circularity; the paper still contains independent components, including architecture choice, data cleaning, and reproducible training curves, that do not reduce to the input.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on domain assumptions about the validity of Llama-generated labels, the meaningfulness of ROUGE/BLEU and citation-based metrics, and transfer to real generative engines. The model has no fitted constants in the derivation, but decoding and evaluation choices are tuned by hand and affect the reported gains.

free parameters (5)
  • Decoding length penalty alpha = 1.1
    Chosen in Section 5.2 to counteract short outputs; this decoding constraint changes the generated content and therefore affects all downstream metrics.
  • No-repeat n-gram size = 3
    Activated in Section 5.2 to reduce copying; a decoding rule that influences output text and visibility scores.
  • Encoder/decoder context window = 384 (proposed) vs 256/448 (baseline)
    Set in Section 6.1; the proposed model uses a larger window, so intrinsic gains over baseline are confounded with this design choice.
  • Raw content truncation length = 4000 characters
    Preprocessing limit in Section 3.2; determines which content is available to the model and thus shapes all outputs.
  • Outlier exclusion and normalization criteria = Unspecified
    Sections 6.2 and 7.2 state results were normalised and outliers excluded without reporting thresholds or rules; this post-hoc selection affects the headline 30.96% figure.
assumptions (4)
  • domain assumption Llama-3.3-70B citation behavior is a valid proxy for visibility in generative search engines.
    Section 7.2 measures visibility exclusively through Llama-3.3-70B responses; generalization to ChatGPT, Perplexity, or real user attention is assumed, not demonstrated.
  • domain assumption Synthetic labels produced by Llama-3.3-70B are valid ground-truth optimized content.
    Section 3.3 treats the transformed texts as the target w-prime; if the labels do not represent genuinely better content, the fine-tuning target is arbitrary.
  • domain assumption Token-level cross-entropy on synthetic rewriting pairs is a suitable training objective for downstream visibility.
    Section 6.2 minimizes next-token cross-entropy; the link between this objective and citation-level visibility is asserted, not established.
  • domain assumption ROUGE-L and BLEU measured against synthetic references indicate meaningful content optimization.
    Section 6.3 uses these as quality metrics; they compare outputs to Llama-generated references, not to actual user or engine preferences.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond SEO: A Transformer-Based Approach for Reinventing Web Content Optimisation." pith.science (2026). https://pith.science/paper/CC3PUIHI

@misc{pith2026250703169,
  author       = {Pith},
  title        = {Pith review of: Beyond SEO: A Transformer-Based Approach for Reinventing Web Content Optimisation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CC3PUIHI}},
  note         = {Machine review of arXiv:2507.03169}
}
read the original abstract

The rise of generative AI search engines is disrupting traditional SEO, with Gartner predicting 25% reduction in conventional search usage by 2026. This necessitates new approaches for web content visibility in AI-driven search environments. We present a domain-specific fine-tuning approach for Generative Engine Optimization (GEO) that transforms web content to improve discoverability in large language model outputs. Our method fine-tunes a BART-base transformer on synthetically generated training data comprising 1,905 cleaned travel website content pairs. Each pair consists of raw website text and its GEO-optimized counterpart incorporating credible citations, statistical evidence, and improved linguistic fluency. We evaluate using intrinsic metrics (ROUGE-L, BLEU) and extrinsic visibility assessments through controlled experiments with Llama-3.3-70B. The fine-tuned model achieves significant improvements over baseline BART: ROUGE-L scores of 0.249 (vs. 0.226) and BLEU scores of 0.200 (vs. 0.173). Most importantly, optimized content demonstrates substantial visibility gains in generative search responses with 15.63% improvement in absolute word count and 30.96% improvement in position-adjusted word count metrics. This work provides the first empirical demonstration that targeted transformer fine-tuning can effectively enhance web content visibility in generative search engines with modest computational resources. Our results suggest GEO represents a tractable approach for content optimization in the AI-driven search landscape, offering concrete evidence that small-scale, domain-focused fine-tuning yields meaningful improvements in content discoverability.

Figures

Figures reproduced from arXiv: 2507.03169 by the authors.

Figure 1
Figure 1. Outline of BART’s architecture. Source: (Vaswani et al., 2023) Decoder Function: The decoder generates output se￾quences autoregressively, which means it is relying on pre￾viously generated tokens while attending to the encoder’s output (Tan et al., 2024). This cross-attention mechanism en￾ables the decoder to: (1) It uses the encoder’s full, two-way grasp of the input to guide its writing. (2) As it picks each word… view at source ↗
Figure 2
Figure 2. Training and validation loss for Proposed and Baseline models. 6.3. Evaluation metrics To evaluate the generative capabilities of the model beyond token-level likelihoods, four complementary metrics are calculated after every epoch: 1. ROUGE-L: Length of the longest shared word sequence between generated output and reference. Higher values mean better coverage of key phrases and structure. 2. BLEU: Precision of matc… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Optimizing Visibility in Generative Engines: A Critical Survey of Generative Engine Optimization (2023-2026)

    cs.IR 2026-07 conditional novelty 5.0 of 10

    A critical review of GEO research concludes that already-retrieved content can improve citation and use, but no tested technique reliably raises organic discoverability or downstream traffic across engines.

  2. Caption Injection for Optimization in Generative Search Engine

    cs.IR 2025-11 conditional novelty 5.0 of 10

    Adding VLM-generated, LLM-refined image captions into source text improves source visibility in generative search by about 1–2% relative, per the paper's G-Eval measurements on MRAMG.

Reference graph

Works this paper leans on

17 extracted references · 9 canonical work pages · cited by 2 Pith papers

  1. [1]

    Geo: Generative engine optimization

    Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., and Deshpande, A. Geo: Generative engine optimization. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’24), pp.\ 5--16, 2024. URL https://dl.acm.org/doi/10.1145/3637528.3671900

    arXiv 2024

  2. [2]

    Alokla, A., Gad, W., Nazih, W., Aref, M., and Salem, A. B. M. Pseudocode generation from source code using the bart model. Mathematics, 10 0 (21), 2022. URL https://www.mdpi.com/2227-7390/10/21/3967

    work page 2022

  3. [3]

    and Elboghdadly, T

    Amer, E. and Elboghdadly, T. The end of the search engine era and the rise of generative ai: A paradigm shift in information retrieval. pp.\ 374--379, 11 2024. URL https://ieeexplore.ieee.org/document/10783559

    work page arXiv 2024

  4. [4]

    Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization, 2017. URL https://arxiv.org/abs/1412.6980

    arXiv 2017

  5. [5]

    and Lakkaraju, H

    Kumar, A. and Lakkaraju, H. Manipulating large language models to increase product visibility, 2024. URL https://arxiv.org/abs/2404.07981

    arXiv 2024

  6. [6]

    Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension

    Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. 2019. URL https://arxiv.org/abs/1910.13461

    arXiv 2019

  7. [7]

    F., Zhang, T., and Liang, P

    Liu, N. F., Zhang, T., and Liang, P. Evaluating verifiability in generative search engines. ArXiv, 2023. URL https://api.semanticscholar.org/CorpusID:258212854

    work page 2023

  8. [8]

    and Howley, C

    LoDolce, M. and Howley, C. Gartner predicts search engine volume will drop 25\ URL https://tinyurl.com/babb9njm

    work page

Show all 17 references
  1. [9]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Sgdr: Stochastic gradient descent with warm restarts, 2017. URL https://arxiv.org/abs/1608.03983

    2017 arXiv

  2. [10]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regularization, 2019. URL https://arxiv.org/abs/1711.05101

    2019 arXiv

  3. [11]

    Teaching language models to support answers with verified quotes

    Menick, J., Trebacz, M., Mikulik, V., Aslanides, J., Song, F., Chadwick, M., Glaese, M., Young, S., Campbell-Gillingham, L., Irving, G., and McAleese, N. Teaching language models to support answers with verified quotes. 2022. URL https://api.semanticscholar.org/CorpusID:247594830

    2022

  4. [12]

    Bart for post-correction of ocr newspaper text

    Soper, E., Fujimoto, S., and Yu, Y.-Y. Bart for post-correction of ocr newspaper text. In Proceedings of the 7th Workshop on Noisy User-generated Text (W-NUT 2021), pp.\ 284--290, 2021. URL https://aclanthology.org/2021.wnut-1.31/

    2021

  5. [13]

    Qarrfsqa: Question-answer replacement and removal pretraining framework for few-shot question answering

    Tan, S., Lee, C.-P., Lim, K., Tee, C.-R., and Alqahtani, A. Qarrfsqa: Question-answer replacement and removal pretraining framework for few-shot question answering. IEEE Access, 2024. URL https://ieeexplore.ieee.org/document/10737355

    2024

  6. [14]

    N., Kaiser, L., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need, 2023. URL https://arxiv.org/abs/1706.03762

    2023 arXiv

  7. [15]

    Application of artificial intelligence (ai) in search engine optimization (seo)

    Yuniarthe, Y. Application of artificial intelligence (ai) in search engine optimization (seo). In International Conference on Soft Computing, Intelligent System and Information Technology, pp.\ 96--101, 2017. URL https://ieeexplore.ieee.org/document/8262550

    2017

  8. [16]

    and Vlachopoulou, M

    Ziakis, C. and Vlachopoulou, M. Artificial Intelligence’s Revolutionary Role in Search Engine Optimization, pp.\ 391--399. 2024. ISBN 978-3-031-51037-3. URL https://link.springer.com/chapter/10.1007/978-3-031-51038-0_43

    2024 doi

  9. [17]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.