{"id":"138d68b5-ac2b-4275-9cbf-0f256ccdec9d","arxiv_id":"2508.18118","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A hierarchical LLM framework generates user-personalized search-ad titles using clustering, matching-based pruning, and chain-of-thought data synthesis, attaining a 0.476% online A/B lift on Douyin Search Ads.","lead":"This paper introduces HLLM-Creator, a hierarchical LLM system that writes user-personalized ad titles for Douyin Search Ads by clustering users, pruning ad candidates, and training on chain-of-thought-generated creative data. An online A/B test reported a 0.476% lift on the company's ad metric, indicating personalized ad copy can move real deployment outcomes.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"A/B lift reported as 0.476% on undefined metric 'Adss' without significance testing; the central effectiveness claim rests on an unquantified, possibly noisy result.","rationale":"The reader's weakest assumption focused on the chain-of-thought synthetic data pipeline, which is indeed a fundamental concern for the internal mechanism. I partially agree: if the synthetic data are unfaithful, the trained generator's factual consistency and personalization claims collapse. However, the single most load-bearing concern for the central claim is the online A/B result itself, because that is the only external validation presented. Without statistical significance, the 0.476% 'increase on Adss' cannot be distinguished from noise, and the paper's strongest claim of measured effectiveness is unsupported. The reader noted that statistical significance is not established, so we agree on the existence of this issue, but the reader weighted the data-pipeline concern as the weakest assumption. My emphasis shifts to the A/B uncertainty. The verdict remains CONDITIONAL (or UNCHANGED from the reader's CONDITIONAL) because the paper could satisfy the concern by supplying the missing statistical details, but currently the evidence is insufficient. I did not choose REJECT because the claimed result, if properly significant, would be a valid external outcome, and code is available for the academic dataset. I did not choose UNVERDICTED because the abstract does give a concrete number and an identifiable evaluation venue; the issue is verification of that number, not total absence of evidence.","tokens_in":3072,"tokens_out":1723,"duration_ms":23075,"concrete_test":"Request or independently inspect the A/B test report: obtain the definition of 'Adss', the sample sizes for control and treatment, the exact lift computation, and the confidence interval or p-value for the 0.476% difference. If the paper or appendix contains a significance test, recompute the standard error from the reported counts and verify the p-value. If no significance evidence can be produced, rerun the A/B with a pre-registered metric and a minimum detectable effect, or treat the 0.476% as not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim—that HLLM-Creator produces a measurable online improvement—rests entirely on the abstract's sentence: 'Online A/B test shows a 0.476% increase on Adss.' This is the only piece of external validation cited. But the abstract (the only readable portion of the paper; the full text is corrupted mojibake) provides no definition of 'Adss,' no sample size, no confidence interval, no p-value, and no indication of whether the 0.476% is relative or absolute, or whether it is statistically distinguishable from zero. In industrial A/B testing at Douyin scale, a sub-percent movement on an undefined metric is plausibly within normal traffic noise unless it is formally tested. If the lift is not statistically significant, the paper's central claim of 'effectiveness' loses its quantitative support. A secondary but related concern: the paper's own internal mechanism for factual consistency—the chain-of-thought data construction pipeline—is asserted to 'ensure factual consistency despite limited personalized data,' but no human evaluation, automated factuality metric, or consistency checks are reported in the abstract. If the synthetic data contain hallucinations or misrepresent user interests, both offline and online outcomes would be built on flawed training data. However, the A/B significance issue is more load-bearing because it directly validates the headline outcome, and without it even a correct pipeline lacks demonstrated effectiveness.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HLLM-Creator, a hierarchical LLM architecture for generating personalized ad titles for Douyin Search Ads. It combines user clustering with a user-ad matching-prediction pruning strategy to reduce inference cost, and a chain-of-thought (CoT) based data construction pipeline to synthesize user-specific training data. The abstract claims an online A/B test improvement of 0.476% on an undefined metric 'Adss' and asserts factual consistency of the generated data. As submitted, the full text is corrupted (mojibake) and cannot be read; this report therefore evaluates the abstract and the available evidence in the reader's report.","tokens_in":3226,"tokens_out":3441,"duration_ms":35976,"significance":"If substantiated, the framework would be an interesting industrial application of hierarchical LLMs for personalized creative generation, with a measurable (though tiny) online lift and open-sourced academic code. The claimed efficiency mechanism is plausible, and the CoT data augmentation addresses a real scarcity problem. However, the sole online validation lacks statistical grounding, and the internal claims about factual consistency are unverified.","major_comments":[{"comment":"The headline result, 'Online A/B test shows a 0.476% increase on Adss,' is not interpretable: 'Adss' is undefined, no confidence interval or p-value is given, and it is unclear whether 0.476% is relative or absolute. At industrial scale, a sub-percent movement on an unnamed metric is within normal traffic noise unless formally tested. This is the only external validation, so the central effectiveness claim is unsupported.","section":"Abstract"},{"comment":"The entire technical content of the manuscript is rendered as mojibake and is unreadable. None of the system architecture, equations, tables, or offline experimental results can be examined. As a reviewer I cannot verify the clustering/pruning formulation, the CoT data pipeline, or the quantitative efficiency claims. This is a load-bearing presentation failure, not a minor typo.","section":"Full text (all sections)"},{"comment":"The statement that the chain-of-thought pipeline 'ensures factual consistency despite limited personalized data' is asserted without any supporting evaluation. No human judgments, automated factuality scores, or consistency checks are reported in the abstract, and the full text is unreadable. Since the authors call this pipeline 'a critical foundation for the effectiveness of our model,' the fidelity of the synthetic data is load-bearing; an evidence-free assertion is not sufficient.","section":"Abstract"}],"minor_comments":[{"comment":"'Adss' appears to be an undefined or typo'd metric; please define it precisely and specify whether the reported increase is relative or absolute.","section":"Abstract"},{"comment":"The abstract claims 'Extensive experiments' but only one online metric is reported; the offline results are inaccessible in the corrupted full text. Please ensure a readable version of the paper is provided.","section":"Abstract"},{"comment":"The statement 'Codes for academic dataset are available' is vague; please clarify what dataset, code, and evaluation scripts are released, and under what license.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The full-text corruption may have arisen in PDF-to-text conversion, but the manuscript as submitted to me is unreadable. The editor may wish to request a clean version from the authors. Even with a clean version, the A/B result needs statistical reporting (confidence interval, p-value, sample sizes) and the factual-consistency claim needs direct evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: HLLM-Creator looks like a real industrial system with a sensible hierarchical design and released code, but the only external evidence we can read is a 0.476% online A/B lift on an undefined metric with no significance testing. That single number is doing more work than it should.\n\nWhat's actually new and good: the integration is genuinely useful. User clustering plus user-ad matching-prediction pruning to cut inference cost, and a chain-of-thought pipeline to synthesize personalized training data—these are known pieces, but putting them together for search-ad creative generation at Douyin scale is a reasonable engineering contribution. The online A/B test against real user behavior is the right kind of evidence, and releasing code for the academic dataset is concrete and checkable.\n\nSoft spots, in proportion. The big one is the headline metric. 'Adss' is undefined; we don't know if the 0.476% is relative or absolute, no confidence interval, no p-value, no sample size. At Douyin's traffic volume, a sub-percent movement can be meaningful if it clears the noise, but the abstract doesn't show that. This is a presentation gap, not evidence of fraud, but it blocks assessment. Second, the chain-of-thought pipeline is claimed to 'ensure factual consistency,' yet no human eval or automated factuality check is reported at the abstract level. Third, there is a potential circularity if the synthetic data pipeline is used to build both training data and offline evaluation data—we'd need to see whether the offline eval is independent. The online A/B is independent, so this is not a load-bearing flaw, but it matters for interpreting the offline results.\n\nOne caveat: the full text we received is corrupted mojibake, so everything above rests on the abstract and a few legible fragments. That alone doesn't make the paper bad, but it means our verdict is conditional on the full methods being coherent.\n\nWho this is for: people working in computational advertising, personalized AIGC, or large-scale LLM serving. It's not a research breakthrough, but it is a genuine deployed system with measurable (if unquantified) impact.\n\nRecommendation: send it to peer review. The statistical gaps are fixable with the authors' own logs, the code is out, and the system is real. A good referee should ask for metric definitions, confidence intervals, and a clearer separation between synthetic training data and evaluation data. I'd take it.","headline":"A plausible industrial personalization pipeline whose headline A/B lift is unquantified; worth a full look if the paper provides the missing statistical detail.","tokens_in":3874,"tokens_out":2242,"would_cite":true,"duration_ms":28322,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HLLM-Creator claims that personalized search-ad titles can be generated per user at industrial scale using a hierarchical LLM design, with a chain-of-thought data pipeline supplying factually consistent training data and an online A/B test","keywords":["personalized ad creative","hierarchical LLM","search ads","user clustering","chain-of-thought data construction","factual consistency","online A/B testing","industrial LLM inference"],"falsifier":"Take a held-out set of users with known historical click behavior; generate titles with HLLM-Creator and with a non-personalized baseline, then have human raters judge whether the HLLM-Creator title reflects the user's known interests more accurately. Additionally, audit a random sample of chain-of-thought training titles for product-fact errors; a nontrivial error rate would contradict the factual-consistency claim.","tokens_in":2817,"feed_emoji":"🎯","tokens_out":3862,"duration_ms":46014,"temperature":0.7,"pith_summary":"This paper seeks to establish that personalized ad creative generation can be made practical at industrial scale: instead of producing one generic ad title per product, HLLM-Creator generates a title tailored to each user's interests while keeping the product's factual selling points intact. The authors argue that prior AI-generated content relies on human inspiration and largely ignores who is viewing the ad. Their system combines user clustering, a user-ad matching prediction used to prune who receives LLM generation, and a chain-of-thought data-construction pipeline that creates user-specific training titles despite scarce personalized data. On Douyin Search Ads, the framework reports offline gains and an online A/B test improvement of 0.476% in Adss, supporting the claim that personalization is both effective and cheap enough to serve.","feed_headline":"A/B test: personalized LLM ad titles lift Adss 0.476%","feed_subtitle":"Hierarchical LLM clusters users, prunes candidates, and trains on chain-of-thought data for ad creatives at scale.","key_machinery":"Three components carry the argument: (1) user clustering, which groups users so interests can be modeled at cluster rather than individual level; (2) a user-ad matching-prediction pruning strategy, a learned scorer that decides which cluster-user pairs need a generated title and which can be served by cheaper defaults, reducing computational overhead; and (3) a chain-of-thought data construction pipeline, which produces training examples pairing product facts and a user profile with a reasoned, fact-consistent title. Clustering and pruning make serving feasible; the chain-of-thought pipeline supplies the personalized training signal.","core_discovery":"The central claim is that the bottleneck in personalized creative generation is not the LLM itself but the mismatch between user modeling and generation, and that this mismatch can be solved architecturally. HLLM-Creator first clusters users to identify interest groups, then uses a trained user-ad matching predictor to prune to the users most likely to respond, so only a small fraction of users trigger an expensive LLM generation. For training data, a chain-of-thought pipeline reasons from product features and user interests to produce user-specific creative titles that remain factually consistent; this synthetic data is described as the critical foundation for the model's behavior. The pape","pith_inferences":["Beyond the paper: the architecture suggests that personalization gains come more from smarter selection of when to generate than from a larger generation model; a direct test would be to freeze the generator and vary the pruning threshold.","Beyond the paper: if the chain-of-thought data is as critical as claimed, ablating it should degrade factual consistency more than fluency, a testable prediction the paper does not run.","Beyond the paper: the reported metric is a single online A/B metric; the split between revenue and engagement effects is left open.","Beyond the paper: a natural extension is to make the pruning predictor interpretable, letting advertisers see why a user was selected for a personalized title."],"forward_implications":["User-specific creative generation no longer requires one LLM call per user; cluster-level modeling plus pruning makes it deployable in search advertising.","Chain-of-thought synthetic data can substitute for scarce personalized creative data while maintaining factual constraints.","A 0.476% Adss lift, if stable, is commercially meaningful on large ad platforms.","The same hierarchical structure could be applied to other content types, such as product descriptions, images, or video captions.","Factual consistency is engineered into the training data rather than only prompted at inference, changing how factuality is controlled."],"supporting_citations":[],"fun_headline_variants":["LLM ad titles get personal via clustering and pruning","Hierarchical LLM: cluster users, prune, then generate","Chain-of-thought data trains LLM for personalized ads","0.476% Adss lift from hierarchical LLM personalization","Personalized ad creatives at scale via user clustering"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The framework's effectiveness depends on the chain-of-thought construction pipeline producing training titles that truly reflect a user's interests and preserve the product's facts; if the synthetic data is noisy or hallucinated, the personalization and factual-consistency claims are unsupported.","fun_headline_variants_meta":{"raw":{"variants":["LLM ad titles get personal via clustering and pruning","Hierarchical LLM: cluster users, prune, then generate","Chain-of-thought data trains LLM for personalized ads","0.476% Adss lift from hierarchical LLM personalization","Personalized ad creatives at scale via user clustering"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000283,"raw_usage":{"total_tokens":1549,"prompt_tokens":824,"completion_tokens":725,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":643}},"tokens_in":568,"tokens_out":725,"duration_ms":8378,"temperature":1.0,"reasoning_tokens":643,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:35:38.986848+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of users with known historical click behavior; generate titles with HLLM-Creator and with a non-personalized baseline, then have human raters judge whether the HLLM-Creator title reflects the user's known interests more accurately. Additionally, audit a random sample of chain-of-thought training titles for product-fact errors; a nontrivial error rate would contradict the factual-consistency claim.","supporting_citations":[],"review_version":1}