Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

LLMs for Customized Marketing Content Generation and Evaluation at Scale

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Keyword-specific ad copy generated by a retrieval-augmented LLM outperforms template-based ads in large-scale search advertising tests, and the paper's automated judge agrees with human reviewers 89.57% of the time.

desk verdict Real deployed system with significant online engagement lifts, but the headline CTR and auto-evaluator agreement numbers are overstated and need correction. read the letter →

arxiv 2506.17863 v1 pith:TXUZBXAE submitted 2025-06-22 cs.CL

Haoran Liu , Amir Tahmasbi , Ehtesham Sam Haque , Purak Jain This is my paper · ORCID
classification cs.CL
keywords largelanguagemodelsretrieval-augmentedgenerationadcopyLLM-as-a-Judgepaidsearchadvertisinge-commercemarketingautomatedevaluationpromptrefinement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the main obstacle to effective offsite e-commerce advertising is generic template ad copy that ignores what the customer actually searched. It proposes MarketingFM, a retrieval-augmented generation system that grounds each ad in product metadata retrieved for the search keyword, and reports that in a 10,000-keyword online A/B test keyword-specific copy raised click volume and impressions and lowered cost-per-click relative to template ads. Because human review of millions of generated ads is expensive, the paper also proposes AutoEval-Main, a two-layer evaluator combining rule-based checks with LLM-as-a-Judge scoring, which it reports matches human accept/reject decisions in 89.57% of cases. A third component, AutoEval-Update, uses active sampling and a critic LLM to revise evaluation prompts as standards drift, reducing the human labor needed to keep the evaluator aligned. If these results hold, e-commerce marketers could personalize ad copy per keyword at scale while relying on cheap automated screening rather than large annotation teams.

What carries the argument

The load-bearing mechanism is retrieval-augmented generation grounded in the retailer's own search product index. A retriever maps each customer keyword to product documents, first via sentence-embedding similarity and later via the products that actually appear on the retailer's search page for that keyword, and the LLM generates headlines and descriptions from that retrieved context. A task-chaining step generates a surplus of candidates and then summarizes them into character-limited headlines. Evaluation is carried out by AutoEval-Main, which first applies rule-based rejections for safety, sale claims, and diversity, then an LLM-as-a-Judge that scores relevance and generalization on a 0-5 scale using the same retrieved context; AutoEval-Update closes the loop with active sampling of ad copies, a critic LLM that writes alignment reports from human-versus-LLM disagreements, and iterated prompt and threshold refinement.

What would settle it

Look at the raw click and impression counts behind the 10,000-keyword A/B test: if the CTR lifts remain non-significant, at about +4 to +24 basis points with p-values above 0.1, while click and impression lifts are significant, then the abstract's '9% higher CTR' would not be supported even though the engagement and cost-efficiency gains would stand.

Watch

Extended reading notes

Core claim

The central discovery is that grounding LLM ad copy in retrieval from the retailer's product search index converts a generic category template into a keyword-specific, product-focused ad, and that this conversion measurably improves paid search economics. In the paper's largest field experiment across 10,000 keywords, the treatment ads gained 8-12% more impressions and 8-9% more clicks on mobile and desktop, with modest CPC reductions; the abstract summarizes these gains as up to 9% higher CTR, 12% more impressions, and 0.38% lower CPC. For evaluation, the paper finds that an LLM judge that sees both the query and the retrieved product context and also scores how well the ad generalizes agrees with human reviewers 89.57% of the time, with most disagreements being the LLM rejecting ads humans accept. The paper further claims that a critic LLM can diagnose those disagreements and propose revised evaluation criteria, which the experiments indicate improve alignment beyond the unrefined prompt. Together these results are offered as evidence that LLM-based generation and evaluation can operate at industry scale with only minimal human oversight.

Load-bearing premise

The human accept/reject labels used as ground truth are accurate enough to serve as the benchmark for all the agreement numbers; the paper's own audit found a 3.71% weighted error rate, and if annotator noise is actually higher, the claimed 89.57% agreement and the refinement improvements are measured against an unstable target.

Editorial extensions

If this is right

  • Search-marketing teams could generate per-keyword ad copy for millions of products without hand-writing templates, because grounding in retrieved product context keeps content relevant.
  • The 89.57% agreement between AutoEval-Main and human reviewers implies that a hybrid rule-plus-LLM judge can replace most human screening; the paper reports a 200x cost reduction and 42x faster processing.
  • Because AutoEval-Main is calibrated against a large human-annotated benchmark, that same benchmark could be reused to fine-tune generation models, as the paper notes.
  • AutoEval-Update's active-sampling loop implies evaluation criteria can track shifting product catalogs and ad policies using only a few thousand human labels per refresh cycle rather than full re-annotation.
  • The lower rejection rate from search-page product context, roughly 2.79% overall versus 15% with semantic retrieval, suggests retrieval design, not model choice, is the main lever for ad relevance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The disagreement analysis shows the LLM judge is more conservative than humans, rejecting ads humans accept in 8.90% of cases while accepting in only 1.53% of the reverse; a natural extension is threshold calibration that trades a small increase in accepted-risk for a large drop in wasted rejections, a tuning decision the paper leaves open.
  • The 3.71% weighted human annotation error rate implies the 89.57% agreement is close to the practical ceiling for human-level labels, so future gains may require adjudicated or consensus labels rather than better evaluation prompts.
  • The same retrieval-plus-evaluation loop could be tested on the other marketing channels the paper names, such as social ads and outbound email, where product context and length constraints differ; whether the gains transfer is a testable prediction.
  • One could expect diminishing returns from prompt refinement once evaluator disagreements are dominated by irreducible label noise, suggesting the stopping rule in AutoEval-Update could be tied to measured annotator agreement rather than a fixed threshold.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MarketingFM, a retrieval-augmented generation system for producing keyword-specific ad copy for e-commerce paid search, and two automated evaluation components: AutoEval-Main, a hybrid rule-based/LLM-as-a-Judge evaluator, and AutoEval-Update, an iterative prompt-refinement framework that uses active sampling and a critic LLM. The authors report offline human and automated evaluations on 150,000 generated ad copies for 10,000 keywords, and online A/B tests on 3,000 and 10,000 keywords. The headline claims are that keyword-focused ad copy outperforms template-based copy (stated as 'up to 9% higher CTR' in the abstract) and that AutoEval-Main achieves 89.57% agreement with human reviewers, with AutoEval-Update further improving alignment through automated prompt refinement.

Significance. If the results held, the work would be practically significant: it addresses a real bottleneck in scaling personalized ad content and reducing human review costs. The paper reports a large human-annotated dataset, a deployed system, and online A/B testing infrastructure, all of which are relatively rare in academic work and could be useful to practitioners. However, the current manuscript contains several internal inconsistencies and missing validation details that prevent accepting the central claims as stated.

major comments (4)
  1. [Abstract / §6.2, Table 7] The abstract's claim of 'up to 9% higher CTR' is not supported by the data in Table 7. Table 7 reports CTR lift of +4 basis points on mobile (p=0.234) and +24 basis points on desktop (p=0.110), neither of which is statistically significant. The 8–9% figure in Table 7 is the clicks lift, not the CTR lift, and Section 6.2 explicitly states that 'CTR remains stable due to broader reach.' The abstract should be corrected to report the clicks lift and impression lift, or to present the CTR effect with its confidence intervals and p-values.
  2. [§6.4, Table 9] The text claims that 'All AutoEval-Update strategies improved upon the baseline' and that 'the baseline method performs the worst overall, with the highest FPR and FNR rates.' These statements are contradicted by Table 9: the diversity-based sampling strategy has accuracy 66.89% and F_beta 70.16%, both worse than the baseline (68.39% and 72.09%), and the baseline FNR (10.92%) is lower than that of random (15.52%) and diversity (22.41%) sampling. The conclusions in this section need to be revised to state that only random and uncertainty-based sampling improve over the baseline, and that no single strategy dominates on all metrics.
  3. [§6.3, Table 8] The reported 89.57% agreement between AutoEval-Main and human reviewers appears to be an in-sample artifact. Table 8 sweeps six threshold configurations over the same 150,000 human-labeled ad copies and the authors select the configuration with the lowest total disagreement (Context(2)+General(3), 10.43%). No validation split is described for AutoEval-Main, and no pre-registered thresholds are specified. The paper should report agreement on a held-out set using thresholds fixed before evaluation, or otherwise justify the threshold selection without using the test set.
  4. [§4, Table 2] The reliability of the human annotation ground truth is not established. Table 2 reports a weighted error rate of 3.71% but does not state the sample size used for manual validation or any inter-annotator agreement measure. Importantly, the reject labels have a 29% mistake rate, which directly affects the false-positive/false-negative quantities in Table 8 that AutoEval-Main is calibrated against. Appendix B.2 notes that the annotation tool logs metadata 'enabling further analysis of inter-annotator agreement,' but no such analysis is reported anywhere in the paper. The authors should report inter-annotator agreement and the number of validated samples.
minor comments (5)
  1. [§6.3] The reference to 'Figure 5' for the agreement result should be Table 5; Figure 5 is the AutoEval-Update pipeline diagram in Appendix C.
  2. [Abstract] The abstract reports '0.38% lower CPC', but Table 7 shows -0.35% (mobile) and -0.22% (desktop). Please clarify which number is being cited.
  3. [§6.3] The statement that AutoEval-Main 'reduces costs by 200× and processing time by 42×' lacks an explanation of how these factors were measured; please add a brief methodology or a reference.
  4. [§5.2] There is a typo in the sentence 'ensuring that evaluation remains adaptive and aligned with evolving data with little human involve' – 'involve' should be 'involvement'.
  5. [Table 9] The F_beta metric is not defined; please specify the beta value used.

Circularity Check

1 steps flagged · score 6.0 of 10

AutoEval-Main's 89.57% agreement is the in-sample best of six threshold fits on the same 150k human labels, not an independent prediction.

  1. fitted input called prediction [Abstract; Section 6.3 (Table 8)]
    "To evaluate the AutoEval framework, we tested its performance on 150,000 ad copies across 10,000 keywords, using a human-labeled dataset as the benchmark. ... Context (2) + General (3) ... Total Disagree. 10.43% ... AutoEval-Main achieves an agreement rate of 89.57% with human reviewers."

    The abstract's 89.57% agreement is exactly 100% minus the lowest total disagreement (10.43%) in Table 8, obtained by testing six threshold configurations on the same 150,000 human-labeled ad copies and reporting the row with the best total. No held-out split or pre-registered threshold is described for AutoEval-Main, so the claim is the selected configuration's in-sample fit, not an unbiased estimate of alignment with human judgment. The threshold is fitted to the benchmark and then the resulting agreement is presented as a system capability, i.e., a fitted parameter renamed as a prediction.

full rationale

The generation-side claim (keyword-specific RAG ad copy beating template ads) is supported by online A/B tests (Tables 6-7) and is not circular: the treatment/control comparison is external to the fitting procedure, and the 9%/12% lifts are measured engagement metrics. The AutoEval-Update results are also evaluated on contamination-free validation and golden datasets, which provides some independent check. The main circularity is confined to AutoEval-Main: its headline agreement with human reviewers is the minimum of a threshold sweep on the same human-labeled benchmark, so the 89.57% number is an in-sample selection artifact rather than a prediction. Table 2's human ground-truth validation (3.71% weighted error, no sample size or inter-annotator agreement) makes the fitted target itself weakly documented, but that is a reliability concern rather than an additional circular step. Overall, one load-bearing evaluation claim reduces to its fitting input while the central generation claim remains independent; score 6.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rest on an internal benchmark of human annotations and on assumptions that LLM scores and marketer criteria reflect ad effectiveness. Free parameters include the AutoEval-Main acceptance thresholds, the regeneration threshold, the active sampling size, and the termination condition; these are fitted to data or chosen by hand rather than derived from theory. No new physical or scientific entities are introduced.

free parameters (5)
  • AutoEval-Main relevance threshold (Context threshold X) = 2 (best configuration Context(2) + General(3))
    Table 8 sweeps Context thresholds 2, 3, 4 and General thresholds 2, 3; the 89.57% agreement is reported at (2,3), a combination selected on the same 150,000-ad human benchmark.
  • AutoEval-Main generalization threshold (Y) = 3 (best configuration Context(2) + General(3))
    Selected together with the relevance threshold to minimize total disagreement in Table 8; no held-out validation is described for this choice.
  • Generation relevance regeneration threshold = 4
    Section 6.1 states ad copies with relevance score below 4 were automatically rejected and replaced; the threshold is chosen by the authors without a sensitivity analysis.
  • Active sampling size per iteration = 1000 keywords
    Section 5.2 says from a daily digest of 20 million keywords, 1,000 keywords are selected for human review; no analysis justifies this number.
  • Termination threshold tau and max rounds N = unspecified
    Algorithm 1 requires performance threshold tau and maximum iteration rounds N, but their values are not reported in the paper.
assumptions (5)
  • domain assumption Human annotations are a reliable ground truth for ad quality
    The entire evaluation stack (97.21% generation acceptance, 89.57% agreement, AutoEval-Update refinement) is validated against human labels whose own weighted error is 3.71% (Section 4, Table 2). Annotator noise or bias would inflate all agreement metrics.
  • domain assumption LLM-as-a-Judge scores are a valid proxy for human judgments in ad evaluation
    The paper asserts alignment with humans via 89.57% agreement but does not establish what the LLM score means beyond correlation with these labels; it is used as a filter in generation and as the core of AutoEval.
  • domain assumption Search-page product context retrieved for a keyword is relevant to that keyword's ad
    Section 3.1, 'RAG using Search Page Products Context', maps keywords to products from the retailer's search result pages under the assumption that displayed products are relevant. The authors report this reduces the relevancy issue from 10% to 1%, but the mapping itself is not independently validated.
  • domain assumption Marketing criteria defined by marketers (relevance, CTA, DKI) capture ad effectiveness
    Section 6.1 and Appendix B.1 adopt marketer-defined requirements for relevance, call-to-action, and dynamic keyword insertion. The connection between these criteria and downstream metrics like CTR or CPC is asserted but not derived.
  • domain assumption The 1,000-keyword active sample is representative of the 20M-keyword distribution
    Section 5.2 reduces a daily digest of 20 million keywords to 1,000 sampled keywords for human review; representativeness of the sampling strategies is argued but not demonstrated with distribution statistics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLMs for Customized Marketing Content Generation and Evaluation at Scale." pith.science (2026). https://pith.science/paper/TXUZBXAE

@misc{pith2026250617863,
  author       = {Pith},
  title        = {Pith review of: LLMs for Customized Marketing Content Generation and Evaluation at Scale},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TXUZBXAE}},
  note         = {Machine review of arXiv:2506.17863}
}
read the original abstract

Offsite marketing is essential in e-commerce, enabling businesses to reach customers through external platforms and drive traffic to retail websites. However, most current offsite marketing content is overly generic, template-based, and poorly aligned with landing pages, limiting its effectiveness. To address these limitations, we propose MarketingFM, a retrieval-augmented system that integrates multiple data sources to generate keyword-specific ad copy with minimal human intervention. We validate MarketingFM via offline human and automated evaluations and large-scale online A/B tests. In one experiment, keyword-focused ad copy outperformed templates, achieving up to 9% higher CTR, 12% more impressions, and 0.38% lower CPC, demonstrating gains in ad ranking and cost efficiency. Despite these gains, human review of generated ads remains costly. To address this, we propose AutoEval-Main, an automated evaluation system that combines rule-based metrics with LLM-as-a-Judge techniques to ensure alignment with marketing principles. In experiments with large-scale human annotations, AutoEval-Main achieved 89.57% agreement with human reviewers. Building on this, we propose AutoEval-Update, a cost-efficient LLM-human collaborative framework to dynamically refine evaluation prompts and adapt to shifting criteria with minimal human input. By selectively sampling representative ads for human review and using a critic LLM to generate alignment reports, AutoEval-Update improves evaluation consistency while reducing manual effort. Experiments show the critic LLM suggests meaningful refinements, improving LLM-human agreement. Nonetheless, human oversight remains essential for setting thresholds and validating refinements before deployment.

Figures

Figures reproduced from arXiv: 2506.17863 by the authors.

Figure 1
Figure 1. A comparison of two paid search ads for an e [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An illustration of the proposed MarketingFM framework for marketing content generation and evaluation. The top [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. AWS GroundTruth UI for human annotation of MarketingFM generated marketing content. [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Human evaluation results for the first-round 3,000-keyword test using semantic embedding-based RAG generation. [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 6
Figure 6. Figure 6: Initial evaluation prompt before refinement by AutoEval-Update. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Refined evaluation prompt generated through AutoEval-Update. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: A heatmap between CTA and DKI counts. The number of keywords in the sample of 10,000 keywords with the number [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Frontier AI performance across the business disciplines: a case-grounded benchmark of knowledge work and analytical reasoning

    cs.CL 2026-07 conditional novelty 7.0 of 10

    On a new 615-question business-case benchmark graded by AI against instructor rubrics, frontier LLMs score 87-88% partial credit but complete only about half the questions.

Reference graph

Works this paper leans on

50 extracted references · 19 canonical work pages · cited by 1 Pith paper

  1. [1]

    Llm based generation of item-description for recommendation system

    Arkadeep Acharya, Brijraj Singh, and Naoyuki Onoe. Llm based generation of item-description for recommendation system. InProceedings of the 17th ACM Conference on Recommender Systems, RecSys ’23, page 1204–1207, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400702419. doi: 10.1145/3604915.3610647. URL https://doi.org/10.1145/36049...

    arXiv 2023

  2. [2]

    The claude 3 model family: Opus, sonnet, haiku.Claude-3 Model Card, 1, 2024

    AI Anthropic. The claude 3 model family: Opus, sonnet, haiku.Claude-3 Model Card, 1, 2024

    work page 2024

  3. [3]

    Aligning human and llm judgments: Insights from evalassist on task-specific evaluations and ai-assisted assessment strategy preferences

    Zahra Ashktorab, Michael Desmond, Qian Pan, James M Johnson, Martin Santillan Cooper, Elizabeth M Daly, Rahul Nair, Tejaswini Pedapati, Swapnaja Achintalwar, and Werner Geyer. Aligning human and llm judgments: Insights from evalassist on task-specific evaluations and ai-assisted assessment strategy preferences. arXiv preprint arXiv:2410.00873, 2024

    arXiv

  4. [4]

    Reasoningrec: Bridging personalized recommendations and human-interpretable explanations through llm reasoning.arXiv preprint arXiv:2410.23180, 2024

    Millennium Bismay, Xiangjue Dong, and James Caverlee. Reasoningrec: Bridging personalized recommendations and human-interpretable explanations through llm reasoning.arXiv preprint arXiv:2410.23180, 2024

    arXiv 2024

  5. [5]

    Language models are few-shot learners.Advances in neural infor- mation processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural infor- mation processing systems, 33:1877–1901, 2020

    1901

  6. [6]

    Evaluation of text generation: A survey.arXiv preprint arXiv: 2006.14799, 2020

    Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. Evaluation of text generation: A survey.arXiv preprint arXiv: 2006.14799, 2020

    arXiv 2006

  7. [7]

    Graphcheck: Breaking long-term text barriers with extracted knowledge graph-powered fact- checking.arXiv preprint arXiv:2502.16514, 2025

    Yingjian Chen, Haoran Liu, Yinhong Liu, Jinxiang Xie, Rui Yang, Han Yuan, Yanran Fu, Peng Yuan Zhou, Qingyu Chen, James Caverlee, et al. Graphcheck: Breaking long-term text barriers with extracted knowledge graph-powered fact- checking.arXiv preprint arXiv:2502.16514, 2025

    arXiv 2025

  8. [8]

    Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240):1–113, 2023

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, et al. Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240):1–113, 2023

    2023

Show all 50 references
  1. [9]

    John Joon Young Chung, Ece Kamar, and Saleema Amershi. Increasing diversity while maintaining accuracy: Text data generation with large language models and human interventions.Annual Meeting of the Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.acl-long.34

    2023 doi

  2. [10]

    A survey on llm inference-time self-improvement.arXiv preprint arXiv:2412.14352, 2024

    Xiangjue Dong, Maria Teleki, and James Caverlee. A survey on llm inference-time self-improvement.arXiv preprint arXiv:2412.14352, 2024

    2024 arXiv

  3. [11]

    Disclosure and mitigation of gender bias in llms.arXiv preprint arXiv:2402.11190, 2024

    Xiangjue Dong, Yibo Wang, Philip S Yu, and James Caverlee. Disclosure and mitigation of gender bias in llms.arXiv preprint arXiv:2402.11190, 2024

    2024 arXiv

  4. [12]

    The faiss library

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. 2024

    2024

  5. [13]

    all-minilm-l6-v2, 2020

    Hugging Face. all-minilm-l6-v2, 2020. URL https://huggingface.co/sentence- transformers/all-MiniLM-L6-v2

    2020

  6. [14]

    Detecting hallucinations in large language models using semantic entropy.Nature, 630 (8017):625–630, 2024

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy.Nature, 630 (8017):625–630, 2024

    2024

  7. [15]

    Retrieval-augmented generation for large language models: A survey.arXiv preprint arXiv:2312.10997, 2023

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-augmented generation for large language models: A survey.arXiv preprint arXiv:2312.10997, 2023

    2023 arXiv

  8. [16]

    A survey on llm-as-a- judge.arXiv preprint arXiv:2411.15594, 2024

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a- judge.arXiv preprint arXiv:2411.15594, 2024

    2024 arXiv

  9. [17]

    Pcr-chain: Partial code reuse assisted by hierarchical chaining of prompts on frozen copilot

    Qing Huang, Jiahui Zhu, Zhilong Li, Zhenchang Xing, Changjing Wang, and Xiwei Xu. Pcr-chain: Partial code reuse assisted by hierarchical chaining of prompts on frozen copilot. In2023 IEEE/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSE-Co...

    2023

  10. [18]

    Serp interference network and its applications in search advertising

    Purak Jain and Sandeep Appala. Serp interference network and its applications in search advertising. 2024

    2024

  11. [19]

    Copyright violations and large language models.arXiv preprint arXiv:2310.13771, 2023

    Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard. Copyright violations and large language models.arXiv preprint arXiv:2310.13771, 2023

    2023 arXiv

  12. [20]

    Studying large language model behaviors under realistic knowledge conflicts

    Evgenii Kortukov, Alexander Rubinstein, Elisa Nguyen, and Seong Joon Oh. Studying large language model behaviors under realistic knowledge conflicts. arXiv preprint arXiv:2404.16032, 2024

    2024 arXiv

  13. [21]

    Retrieval-augmented generation for knowledge- intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Se- bastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge- intensive nlp tasks. 2021

    2021

  14. [22]

    Iqa-eval: Automatic evaluation of human-model interactive question answering.arXiv preprint arXiv:2408.13545, 2024

    Ruosen Li, Barry Wang, Ruochen Li, and Xinya Du. Iqa-eval: Automatic evaluation of human-model interactive question answering.arXiv preprint arXiv:2408.13545, 2024

    2024 arXiv

  15. [23]

    Controllable text generation for large language models: A survey.arXiv preprint arXiv: 2408.12599, 2024

    Xun Liang, Hanyu Wang, Yezhaohui Wang, Shichao Song, Jiawei Yang, Simin Niu, Jie Hu, Dan Liu, Shunyu Yao, Feiyu Xiong, and Zhiyu Li. Controllable text generation for large language models: A survey.arXiv preprint arXiv: 2408.12599, 2024

    2024 arXiv

  16. [24]

    I-sheep: Self-alignment of llm from scratch through an iterative self-enhancement paradigm.arXiv preprint arXiv:2408.08072, 2024

    Yiming Liang, Ge Zhang, Xingwei Qu, Tianyu Zheng, Jiawei Guo, Xinrun Du, Zhenzhu Yang, Jiaheng Liu, Chenghua Lin, Lei Ma, et al. I-sheep: Self-alignment of llm from scratch through an iterative self-enhancement paradigm.arXiv preprint arXiv:2408.08072, 2024

    2024 arXiv

  17. [25]

    G-eval: Nlg evaluation using gpt-4 with better human alignment.arXiv preprint arXiv:2303.16634, 2023

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment.arXiv preprint arXiv:2303.16634, 2023

    2023 arXiv

  18. [26]

    Can llms follow simple rules? arXiv preprint arXiv:2311.04235, 2023

    Norman Mu, Sarah Chen, Zifan Wang, Sizhe Chen, David Karamardian, Lulwa Aljeraisy, Dan Hendrycks, and David Wagner. Can llms follow simple rules? arXiv preprint arXiv:2311.04235, 2023

    2023 arXiv

  19. [27]

    Privacy issues in large language models: A survey

    Seth Neel and Peter Chang. Privacy issues in large language models: A survey. arXiv preprint arXiv:2312.06717, 2023

    2023 arXiv

  20. [28]

    Detecting and mitigating hallucinations in multilingual summarisation.arXiv preprint arXiv:2305.13632, 2023

    Yifu Qiu, Yftah Ziser, Anna Korhonen, Edoardo M Ponti, and Shay B Cohen. Detecting and mitigating hallucinations in multilingual summarisation.arXiv preprint arXiv:2305.13632, 2023

    2023 arXiv

  21. [29]

    Applying large language models to sponsored search advertising.URL: https://www

    Martin Reisenbichler, Thomas Reutterer, and David A Schweidel. Applying large language models to sponsored search advertising.URL: https://www. msi. org/working-paper/applying-large-language-models-to-sponsored-search- advertising, 2023

    2023

  22. [30]

    Learning to plan & reason for evaluation with thinking-llm-as-a-judge

    Swarnadeep Saha, Xian Li, Marjan Ghazvininejad, Jason Weston, and Tianlu Wang. Learning to plan & reason for evaluation with thinking-llm-as-a-judge. arXiv preprint arXiv:2501.18099, 2025

    2025 arXiv

  23. [31]

    Rewarding progress: Scaling automated process verifiers for llm reasoning.arXiv preprint arXiv:2410.08146, 2024

    Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for llm reasoning.arXiv preprint arXiv:2410.08146, 2024

    arXiv

  24. [32]

    Who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences

    Shreya Shankar, JD Zamfirescu-Pereira, Bjorn Hartmann, Aditya Parameswaran, and Ian Arawjo. Who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences. InProceedings of the 37th Annual ACM Symposium on User Interface Software and Techn...

    2024

  25. [33]

    Beyond instruction following: Evalu- ating rule following of large language models.arXiv preprint arXiv:2407.08440, 2024

    Wangtao Sun, Chenxiang Zhang, Xueyou Zhang, Ziyang Huang, Haotian Xu, Pei Chen, Shizhu He, Jun Zhao, and Kang Liu. Beyond instruction following: Evalu- ating rule following of large language models.arXiv preprint arXiv:2407.08440, 2024

    2024 arXiv

  26. [34]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  27. [35]

    Can chatgpt defend its belief in truth? evaluating llm reasoning via debate.arXiv preprint arXiv:2305.13160, 2023

    Boshi Wang, Xiang Yue, and Huan Sun. Can chatgpt defend its belief in truth? evaluating llm reasoning via debate.arXiv preprint arXiv:2305.13160, 2023

    2023 arXiv

  28. [36]

    Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li

    Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li. Decodingtrust: A comprehen...

    2023

  29. [37]

    Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35: 24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35: 24824–24837, 2022

    2022

  30. [38]

    Junchao Wu, Shu Yang, Runzhe Zhan, Yulin Yuan, Lidia Sam Chao, and Derek F. Wong. A survey on llm-generated text detection: Necessity, methods, and future directions.Computational Linguistics, 2025. doi: 10.1162/coli_a_00549

    2025 doi

  31. [39]

    Kg-rank: Enhancing large language models for medical qa with knowledge graphs and ranking techniques.arXiv preprint arXiv:2403.05881, 2024

    Rui Yang, Haoran Liu, Edison Marrese-Taylor, Qingcheng Zeng, Yu He Ke, Wanxin Li, Lechao Cheng, Qingyu Chen, James Caverlee, Yutaka Matsuo, et al. Kg-rank: Enhancing large language models for medical qa with knowledge graphs and ranking techniques.arXiv preprint arXiv:2403.058...

    2024 arXiv

  32. [40]

    Retrieval- augmented multimodal language modeling, 2023

    Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. Retrieval- augmented multimodal language modeling, 2023

    2023

  33. [41]

    Self-alignment for factuality: Mitigating hallucina- tions in llms via self-evaluation.arXiv preprint arXiv:2402.09267, 2024

    Xiaoying Zhang, Baolin Peng, Ye Tian, Jingyan Zhou, Lifeng Jin, Linfeng Song, Haitao Mi, and Helen Meng. Self-alignment for factuality: Mitigating hallucina- tions in llms via self-evaluation.arXiv preprint arXiv:2402.09267, 2024

    2024 arXiv

  34. [42]

    Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    2023

  35. [43]

    Gcof: Self-iterative text generation for copywriting using large language model.arXiv preprint arXiv:2402.13667, 2024

    Jianghui Zhou, Ya Gao, Jie Liu, Xuemin Zhao, Zhaohua Yang, Yue Wu, and Lirong Shi. Gcof: Self-iterative text generation for copywriting using large language model.arXiv preprint arXiv:2402.13667, 2024

    2024 arXiv

  36. [44]

    at [e-commerce website]

    Pei Zhou, Jay Pujara, Xiang Ren, Xinyun Chen, Heng-Tze Cheng, Quoc V Le, Ed H Chi, Denny Zhou, Swaroop Mishra, and Huaixiu Steven Zheng. Self- discover: Large language models self-compose reasoning structures.arXiv preprint arXiv:2402.03620, 2024. A Prompt Examples A.1 General...

    2024 arXiv

  37. [45]

    Relevance: How closely does the ad copy relate to the keyword and context?

  38. [46]

    Specificity: Is the ad copy appropriately specific without being too narrow?

  39. [47]

    Clarity: Is the message clear and direct?

  40. [48]

    Creativity: Is the ad copy engaging without being overly abstract?

  41. [49]

    Inclusivity: Does the ad copy avoid bias or stereotypes?

  42. [50]

    Be more flexible with headlines compared to descriptions

    Accuracy: Does the ad copy avoid overclaims or exaggerations? Evaluate the following: Keyword: <keyword>{keyword}</keyword> Context: <context>{context}</context> Ad Copy: <ad_copy>{ad_copy}</ad_copy> Provide your evaluation in the following format: <ad_copy> <text>{ad_copy}</t...

    2017

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.