Pith. sign in

REVIEW 3 major objections 5 minor 44 references

A/B Agent: A Self-Evolving Agent for Strategy Iteration in Industrial A/B Testing

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read A/B Agent claims that organizing historical A/B strategies into a hierarchical experience tree and iterating on online feedback yields better recommendation strategies than flat retrieval or expert tuning alone, with a 4.829% GMV gain in…

desk verdict Plausible system, but the headline 4.829% GMV lift is the best of five sequential strategies and is not supported as a measured outcome without a confirmatory test. read the letter →

arxiv 2608.04625 v1 pith:BNAHXYA5 submitted 2026-08-05 cs.AI

classification cs.AI
keywords A/Btestingrecommendersystemsstrategyiterationretrieval-augmentedgenerationLLMagentshierarchicalknowledgetreeself-evolutione-commerce
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that industrial recommendation-strategy iteration can be turned into a closed loop run by an LLM agent. A/B Agent digests past A/B test reports into a hierarchical strategy experience tree, retrieves transferable strategies for a new optimization request through multi-path tree retrieval, generates an executable strategy with parameters, launches it, reads the online metrics, and uses parent–child and sibling comparisons to propose the next experiment. The payoff, if true, is that the labor-intensive expert loop of design–experiment–analyze–adjust is compressed into an autonomous system that also accumulates reusable knowledge. Evidence includes offline scores on 310 historical strategies across three e-commerce scenarios and an online deployment in short-video e-commerce where five tuning rounds raised cart GMV from +1.123% to +4.829% while all reported guardrail metrics ended positive.

What carries the argument

The load-bearing objects are two trees. The strategy experience tree $T_E=(V_E,E_E)$ stores atomic strategy chunks under semantic paths $\pi(c)=[d_c,s_c,l_c,o_c]$ (domain, scenario, recommendation stage, optimization objective), so retrieval can boost candidates whose tree position matches the request's context. The A/B experiment tree $T^*_A=(V^*_A,E^*_A)$ stores each tested strategy, its parameter configuration, observed metric changes, and context; edge comparisons between parent and child or between siblings tell the agent whether a modification helped. The decision rule is the node utility $U(v)=\sum_{j\in M_{\text{core}}} w_j \widetilde{\Delta y}_j - \lambda \sum_{k\in M_{\text{guard}}} w_k \max(0,-\widetilde{\Delta y}_k)$, where confidence-weighted metric changes enter with guardrail penalties, which lets the agent prefer configurations that lift core objectives without hurting ecosystem metrics.

What would settle it

Re-analyze the raw logs of the five sequential online experiments: compute confidence intervals for GMV under randomization, check prerandomization covariate balance and overlap with concurrent experiments, and test whether Strategy 5's 4.829% lift survives a multiple-comparison or sequential-testing correction. If the adjusted interval includes zero, or if a fresh deployment in another scenario fails to reproduce a positive GMV effect with non-negative guardrails, the central claim is refuted.

Watch

Extended reading notes

Core claim

The central claim is that strategy knowledge in industrial A/B testing has a natural hierarchy—domain, scenario, recommendation stage, and optimization objective—and that encoding this hierarchy changes how well retrieval-augmented generation works for strategy design. A/B Agent organizes each historical record into atomic strategy chunks, attaches each chunk to semantic paths in an experience tree, retrieves via sparse and dense paths with tree-path relevance boosting plus reranking, and generates an executable strategy judged by rules, an agent, and human reviewers. After deployment it builds an experiment tree whose node utility balances core-metric gains against guardrail penalties, and it uses parent–child and sibling comparisons to decide whether to do local parameter search or replace the mechanism. The paper reports that this loop reached +4.829% GMV in a real short-video e-commerce A/B test, with positive GPM, OPM, CVR, clicks, and orders, and it outperformed general LLMs and RAG baselines on offline correctness and novelty scores across three scenarios.

Load-bearing premise

The load-bearing premise is that the reported online A/B test is a valid causal measurement—random assignment, sufficient statistical power, no interference with concurrent experiments, and no post-hoc selection of the best of five strategies—so the 4.829% GMV gain reflects the agent's tuning rather than noise.

Editorial extensions

If this is right

  • A/B testing teams could spend their time on framing business questions and reviewing final decisions rather than designing each experiment variant by hand.
  • Strategies proven in one scenario become searchable and adaptable for other scenarios, because the experience tree indexes by domain, stage, and objective rather than raw text.
  • Guardrail knowledge accumulates: each experiment round records which parameter moves hurt metrics such as Live GMV, so future rounds start from safer configurations.
  • Across the three offline scenarios, the largest measured gap is against RAG baselines (25.0%, 31.7%, 23.5% overall-score improvements), suggesting flat retrieval is the weakest way to reuse A/B history.
  • The system's knowledge base grows with every validated experiment, so its initial strategies should improve as the company accumulates more A/B history.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 4.829% figure is a point estimate without reported uncertainty; a fair reading treats it as evidence only if the online test was a clean randomized experiment, which the paper does not fully document.
  • A natural next test is to compare A/B Agent against a non-LLM automated tuner, such as Bayesian optimization over the same parameter space, to separate the value of the knowledge tree from the value of sequential search.
  • The same closed loop could be applied to other expensive online optimization problems with guardrail constraints—search ranking, ad bidding, pricing—where historical experiments are fragmented and reuse matters.
  • Given the paper's reported error distribution, where solution mismatch accounts for 43.1% of failures and data/attribution errors for 29.2%, the most promising next improvement is fine-grained alignment of retrieved evidence with the specific scenario, pipeline stage, and available features.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes A/B Agent, a closed-loop LLM-agent framework for industrial recommendation strategy iteration. Historical A/B experiment records are decomposed into reusable strategy chunks and organized into a hierarchical experience tree; given a new request, Tree-RAG retrieves relevant experience and generates an executable strategy; after deployment, the agent analyzes online A/B metrics, iteratively tunes strategy and parameters, and writes validated outcomes back into the experience tree. The system is deployed at Kuaishou in a short-video e-commerce scenario, and the paper reports a 4.829% Cart GMV improvement in an online A/B test. Offline evaluations compare A/B Agent with general-purpose LLMs and several RAG baselines on a benchmark of 310 historical strategies across three e-commerce scenarios, with GPT-5.5 as the automatic judge.

Significance. If the reported results were properly validated, the framework would address a genuine and costly bottleneck in industrial recommender systems: the manual, knowledge-fragmented, iterative design of A/B strategies. The hierarchical experience tree, target-aware retrieval, and experiment-tree-guided self-evolution are coherent design contributions, and the construction of an industrial strategy benchmark from 310 real records is a useful asset for the community. The online deployment is also a valuable practical case study. However, the central empirical evidence is currently not statistically grounded, the offline evaluation uses the same model as judge and baseline, and the method is not reproducible from the manuscript because its core weights are undisclosed. Credit is due for the clarity of the system design, the ablation study, and the commitment to a real-world deployment, but the current evidence does not yet establish the headline claims.

major comments (3)
  1. [§5.3, §5.6, Eq. (5)] The headline 4.829% Cart GMV improvement is not supported by the statistics reported. Table 3 lists five sequential strategies, and Strategy 5 was selected after observing the outcomes of the earlier strategies; Table 5 then reports the same numbers as Strategy 5, so there is no independent confirmatory run. The paper gives no sample size, experiment duration, traffic allocation, p-values, or confidence intervals, and it does not correct for selecting the best of five strategies. The only statistical mechanism introduced, the confidence weight rho_j in Eq. (5), is never defined or estimated, and it is unclear whether it represents a formal significance test or a heuristic. Under these conditions, the 4.829% estimate cannot be distinguished from selection noise, and the claim that the online deployment demonstrates effectiveness is not established.
  2. [§5.1, §5.2, Table 2] The offline evaluation is potentially self-referential: GPT-5.5 serves as the automatic judge for correctness and novelty while also being one of the baselines in Table 2. Because the same model scores the outputs it also generates, any systematic judge preference for its own style, or against the agent's output, contaminates the comparison. No human validation, inter-annotator agreement, or alternative judge is reported. Since the offline benchmark is the main evidence for cross-scenario generalization, this issue is load-bearing for the claim that A/B Agent outperforms general-purpose LLMs and RAG baselines.
  3. [§3.2, §3.3, Eqs. (3), (4), (5), (8)] The method is not reproducible from the manuscript because all weights and thresholds in the core equations are left unspecified: alpha_m in Eq. (3), beta_l and gamma_l in Eq. (4), the confidence weights rho_j in Eq. (5), the metric weights w_j and w_k and penalty lambda in Eq. (8), and the top-K counts and reranking thresholds. These values determine retrieval rankings, utility evaluations, and the evolution decisions that produced Strategies 1 through 5. Without them, the reported results cannot be independently reproduced, and one cannot rule out that the favorable trajectory is an artifact of hand-tuned weights rather than the proposed mechanism.
minor comments (5)
  1. [§5.5, Table 4] The ablation study reports single point estimates without confidence intervals; the 0.015 overall-score drop from replacing Strategy-Tree RAG with Flat RAG is within the likely noise of an LLM-based judge, so the conclusion that hierarchical organization helps should be softened or supported by repeated evaluations.
  2. [Abstract, §1, §3] The phrase "Target-A ware" appears in the abstract and in the methodology section headers; the 'w' in "Aware" is missing and should be corrected.
  3. [Figures 1 and 2] Figure 1 contains the stray word "Tired" near "A/B Test," and Figure 2 uses abbreviated labels such as "E-Com mence"; these should be cleaned up for clarity.
  4. [§5.1] The industrial benchmark dataset is claimed as a contribution, but no data access or release statement is provided; the authors should describe how the dataset or evaluation protocol can be obtained, or clearly state that it is proprietary.
  5. [Title page] The ACM reference format still contains the placeholder "Conference acronym 'XX, Woodstock, NY" and a 2018 copyright line; these template remnants should be replaced with the actual venue information.

Circularity Check

2 steps flagged · score 4.0 of 10

Offline scores are judged by a baseline (GPT-5.5) and the headline GMV is the same number as the best step of the tuning trajectory.

  1. fitted input called prediction [Section 5.3 (Table 3) and Section 5.6 (Table 5)]
    "Strategy 5 further strengthens the primary objectives, producing the largest GMV and GPM gains of +4.829% and +4.677%, respectively, while maintaining positive guardrail performance. ... As shown in Table 5, the strategy generated by A/B Agent improves GMV by 4.829%"

    The +4.829% reported in Section 5.6 is exactly the observed GMV of Strategy 5 in Table 3, the last point of the agent's own tuning trajectory. The agent selected Strategy 5 because its GMV was the largest of the five variants, so the 'online A/B test' is not an independent confirmatory experiment; it is the maximum of the values that drove the selection. No pre-specified confirmatory run, confidence interval, or multiple-testing correction is reported, so the claimed improvement is the optimized objective itself rather than an out-of-sample prediction.

  2. other [Section 5.1 Evaluation Protocol; Section 5.2 Main Results]
    "GPT-5.5 serves as the automatic evaluator and assesses each strategy along two dimensions: Correctness, covering technical feasibility, scenario relevance, and logical consistency; and Novelty, measuring the ability to provide meaningful insights beyond direct reuse of historical experience. ... Among the standalone LLMs, GPT-5.5 obtains the highest correctness"

    Correctness is not anchored to external ground truth; it is GPT-5.5's holistic judgment, and GPT-5.5 is simultaneously one of the baselines being ranked. The claim that A/B Agent surpasses GPT-5.5 in correctness is therefore computed from GPT-5.5's own preferences, so the offline superiority comparison is closed inside the evaluator. This does not affect the external GMV measurement, but it makes the offline evaluation self-referential rather than an independent benchmark.

full rationale

The paper's mathematical pipeline (Eqs. 3-8) is not circular: retrieval, reranking, utility evaluation, and self-evolution are all genuine computations, and the online GMV lift is a measured outcome rather than a derived prediction. However, two reporting decisions create partial circularity. First, the offline correctness and novelty evaluation uses GPT-5.5 as the automatic judge while GPT-5.5 is also a baseline, so the offline superiority claims are defined by the evaluator's own ratings. Second, Section 5.6 reports the final strategy's +4.829% GMV as the online A/B test result, but this is exactly Strategy 5's row in Table 3, selected because it was the largest observed gain; there is no separate confirmatory experiment. The central online number retains external content because it was measured in production, but as presented it is the optimized maximum rather than an independent validation. These issues are evaluation-validity and selection-reporting concerns more than derivation-chain circularity, so the score is moderate rather than extreme.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The framework introduces no new physical entities, but it relies on several undisclosed configuration choices and on evaluation assumptions. The free parameters in the retrieval and utility equations are not specified, and the empirical validation depends on an LLM judge and on an online test with no statistical reporting.

free parameters (5)
  • Retrieval path weights alpha_m (Eq. 3) = Not reported
    Weights balancing sparse and dense retrieval scores; no fitting details or values given in the paper.
  • Tree relevance weights beta_l and decay rates gamma_l (Eq. 4) = Not reported
    Control hierarchy-level importance and distance-decay in tree-path boosting; values are undisclosed and likely tuned.
  • Metric weights w_j, w_k and guardrail penalty lambda (Eq. 8) = Not reported
    Define the utility used for self-evolution decisions; the paper does not specify how these weights were chosen.
  • Confidence weights rho_j (Eq. 5) = Not reported
    Adjust observed metric changes by statistical confidence; computation of rho_j is not described.
  • Top-K retrieval count and reranking thresholds = Not reported
    Number of candidates passed to the reranker and acceptance thresholds for judges are not stated.
assumptions (5)
  • domain assumption GPT-5.5 provides valid automatic scores for strategy correctness and novelty
    Section 5.2 uses GPT-5.5 as the automatic evaluator; no human-scored validation of the benchmark is reported, and GPT-5.5 is also a baseline.
  • domain assumption The online A/B test is a valid randomized experiment with adequate power
    Section 5.6 and Table 5 report lift percentages without p-values, confidence intervals, sample sizes, or duration.
  • domain assumption Historical records decompose into atomic strategy chunks with preserved causal attribution
    Section 3.1 assumes clean decomposition of experiment reports into independent modifications with aligned outcomes; no reliability study is provided.
  • domain assumption GLM-5.1 generates executable strategies from retrieved chunks
    Section 4 relies on GLM-5.1 for understanding, generation, and iterative optimization; generation reliability is only measured by the proxy scores.
  • ad hoc to paper The utility function U(v) with undisclosed weights captures the true business trade-off
    Eq. 8 defines U with weights w_j, w_k and lambda that are never specified; the self-evolution behavior depends on this choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A/B Agent: A Self-Evolving Agent for Strategy Iteration in Industrial A/B Testing." pith.science (2026). https://pith.science/paper/BNAHXYA5

@misc{pith2026260804625,
  author       = {Pith},
  title        = {Pith review of: A/B Agent: A Self-Evolving Agent for Strategy Iteration in Industrial A/B Testing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BNAHXYA5}},
  note         = {Machine review of arXiv:2608.04625}
}
read the original abstract

Industrial recommendation strategy iteration heavily relies on large-scale A/B experimentation. Traditional tuning requires experts to repeatedly design strategies, configure experiments, analyze results, and adjust parameters, making the process labor-intensive and time-consuming. Meanwhile, valuable knowledge from historical experiments is often fragmented, making systematic reuse difficult through manual expert effort alone. Existing RAG agents partially alleviate this burden by retrieving prior strategies, but typically organize experience in a flat manner, overlooking the hierarchical relationships among business scenarios, recommendation stages, optimization objectives, and experimental contexts. This often results in mismatched retrieval and limited cross-scenario transfer, while preventing agents from continuously refining strategies and parameters through sequential A/B feedback. % To address these limitations, we propose A/B Agent, a closed-loop A/B agent for industrial recommendation strategy optimization. The framework comprises three tightly coupled core components: Historical Strategy Knowledge Organization, Autonomous Target-Aware Strategy Generation, and Experiment-Guided Strategy Self-Evolution. It organizes historical strategies into a hierarchical experience tree, retrieves transferable evidence through multi-path Tree-RAG to generate executable strategies, and continuously analyzes online A/B feedback to guide autonomous tuning and update the experience tree for self-evolution. Extensive offline and online evaluations demonstrate its effectiveness, including a 4.829% improvement in GMV in a real-world short-video e-commerce recommendation system while maintaining positive gains across all guardrail metrics.

Figures

Figures reproduced from arXiv: 2608.04625 by the authors.

Figure 1
Figure 1. Motivation of A/B Agent. Traditional A/B tuning re [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of A/B Agent. The framework consists of three components: Historical Strategy Knowledge Organization, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. End-to-end workflow of A/B Agent, including autonomous strategy generation via target-aware multi-path retrieval [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Pareto-oriented tuning trajectory of A/B Agent [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Distribution of error categories for A/B Agent. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

44 extracted references · 20 canonical work pages

  1. [1]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avi Sil, and Hannaneh Hajishirzi. 2024. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In International conference on learning representations, Vol. 2024. 9112–9141

  2. [2]

    Alex Deng and Xiaolin Shi. 2016. Data-driven metric development for online controlled experiments: Seven lessons learned. InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 77– 86

  3. [3]

    Jiaxin Deng, Shiyao Wang, Kuo Cai, Lejian Ren, Qigen Hu, Weifeng Ding, Qiang Luo, and Guorui Zhou. 2025. Onerec: Unifying retrieve and rank with generative recommender and iterative preference alignment.arXiv preprint arXiv:2502.18965 (2025)

  4. [4]

    Pavel Dmitriev, Somit Gupta, Dong Woo Kim, and Garnet Vaz. 2017. A dirty dozen: Twelve common metric interpretation pitfalls in online controlled experiments. InProceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining. 1427–1436

  5. [5]

    Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. 2024. From local to global: A graph rag approach to query-focused summarization.arXiv preprint arXiv:2404.16130(2024)

  6. [6]

    Aleksander Fabijan, Pavel Dmitriev, Helena Holmström Olsson, and Jan Bosch

  7. [7]

    Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023. Precise zero- shot dense retrieval without relevance labels. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 1762–1777

  8. [8]

    Somit Gupta, Ronny Kohavi, Diane Tang, Ya Xu, Reid Andersen, Eytan Bakshy, Niall Cardin, Sumita Chandran, Nanyu Chen, Dominic Coey, et al . 2019. Top challenges from the first practical online controlled experiments summit.ACM SIGKDD Explorations Newsletter21, 1 (2019), 20–35

Show all 44 references
  1. [9]

    Jianhua Han, Yong Yu, Feng Liu, Ruiming Tang, and Yuzhou Zhang. 2019. Op- timizing ranking algorithm in recommender system via deep reinforcement learning. In2019 international conference on artificial intelligence and advanced manufacturing (AIAM). IEEE, 22–26

  2. [10]

    Chao Hao, Shuai Wang, and Kaiwen Zhou. 2025. Uncertainty-aware gui agent: Adaptive perception through component recommendation and human-in-the- loop refinement.arXiv preprint arXiv:2508.04025(2025)

  3. [11]

    Zheng Hu, Yuxin Chen, Yongsen Pan, Xu Yuan, Yuting Yin, Daoyuan Wang, Boyang Xia, Zefei Luo, Hongyang Wang, Songhao Ni, et al. 2026. Stop Treating Collisions Equally: Qualification-Aware Semantic ID Learning for Recommenda- tion at Industrial Scale.arXiv preprint arXiv:2603.00...

  4. [12]

    Haoyu Huang, Yongfeng Huang, Junjie Yang, Zhenyu Pan, Yongqiang Chen, Kaili Ma, Hongzhi Chen, and James Cheng. 2025. Retrieval-augmented generation with hierarchical knowledge.arXiv preprint arXiv:2503.10150(2025)

  5. [13]

    Xu Huang, Jianxun Lian, Yuxuan Lei, Jing Yao, Defu Lian, and Xing Xie. 2025. Recommender ai agent: Integrating large language models for interactive recom- mendations.ACM Transactions on Information Systems43, 4 (2025), 1–33

  6. [14]

    Aryan Jadon and Avinash Patil. 2024. A comprehensive survey of evaluation tech- niques for recommendation systems. InInternational Conference on Computation of Artificial Intelligence & Machine Learning. Springer, 281–304

  7. [15]

    Olivier Jeunen, Shubham Baweja, Neeti Pokharna, and Aleksei Ustimenko. 2024. Powerful A/B-testing metrics and where to find them. InProceedings of the 18th ACM Conference on Recommender Systems. 816–818

  8. [16]

    Zhuohang Jiang, Yuxin Chen, Shijie Wang, Haohao Qu, Zhou Jindong, Wenqi Fan, Li Qing, Dongxu Liang, and Jun Wang. 2026. Atomic Intent Reasoning: Bringing LLM Semantics to Industrial Cross-Domain Recommendations.arXiv preprint arXiv:2606.10357(2026)

  9. [17]

    Zhuohang Jiang, Pangjing Wu, Ziran Liang, Peter Q Chen, Xu Yuan, Ye Jia, Jiancheng Tu, Chen Li, Peter HF Ng, and Qing Li. 2025. Hibench: Benchmarking llms capability on hierarchical structure reasoning. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and...

  10. [18]

    Zhuohang Jiang, Pangjing Wu, Xu Yuan, Wenqi Fan, and Qing Li. 2025. QA- Dragon: Query-Aware Dynamic RAG System for Knowledge-Intensive Visual Question Answering.arXiv preprint arXiv:2508.05197(2025)

  11. [19]

    Zhuohang Jiang, Xu Yuan, Haohao Qu, Shanru Lin, Kanglong Liu, Wenqi Fan, and Li Qing. 2026. SUPERGLASSES: Benchmarking Vision Language Models as Intelligent Agents for AI Smart Glasses. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2165–2175

  12. [20]

    Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2021. Generation-augmented retrieval for open-domain question answering. InProceedings of the 59th Annual Meeting of the Associa- tion for Computational Linguistics and the 11th Int...

  13. [21]

    Mohammad Hossein Namaki, Avrilia Floratou, Fotis Psallidas, Subru Krishnan, Ashvin Agrawal, Yinghui Wu, Yiwen Zhu, and Markus Weimer. 2020. Vamsa: Automated provenance tracking in data science scripts. InProceedings of the 26th ACM SIGKDD international conference on knowledge ...

  14. [22]

    Liangbo Ning, Ziran Liang, Zhuohang Jiang, Haohao Qu, Yujuan Ding, Wenqi Fan, Xiao-yong Wei, Shanru Lin, Hui Liu, Philip S Yu, et al. 2025. A survey of webagents: Towards next-generation ai agents for web automation with large foundation models. InProceedings of the 31st ACM S...

  15. [23]

    Yongsen Pan, Yuxin Chen, Zheng Hu, Xu Yuan, Daoyuan Wang, Yuting Yin, Song- hao Ni, Hongyang Wang, Jun Wang, Fuji Ren, et al. 2026. Beyond Static Collision Handling: Adaptive Semantic ID Learning for Multimodal Recommendation at Industrial Scale.arXiv preprint arXiv:2604.23522(2026)

  16. [24]

    Hong Peng, Xiaoliang Jin, Qiao Huang, and Supeng Liu. 2025. E-commerce Intelli- gent Recommendation Optimization and Personalized Marketing Strategy Based on Big Model. InProceedings of the 2025 International Conference on Information Economy, Data Modeling and Cloud Computing...

  17. [25]

    Qi Pi, Guorui Zhou, Yujing Zhang, Zhe Wang, Lejian Ren, Ying Fan, Xiaoqiang Zhu, and Kun Gai. 2020. Search-based user interest modeling with lifelong sequential behavior data for click-through rate prediction. InProceedings of the 29th ACM International Conference on Informati...

  18. [26]

    Federico Quin, Danny Weyns, Matthias Galster, and Camila Costa Silva. 2024. A/B testing: A systematic literature review.Journal of Systems and Software211 (2024), 112011

  19. [27]

    Christopher Ré, Feng Niu, Pallavi Gudipati, and Charles Srisuwananukorn. 2019. Overton: A data system for monitoring and improving machine-learned products. arXiv preprint arXiv:1909.05372(2019)

  20. [28]

    Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher Manning. 2024. Raptor: Recursive abstractive processing for tree- organized retrieval. InInternational Conference on Learning Representations, Vol. 2024. 32628–32649

  21. [29]

    Marius Schlegel and Kai-Uwe Sattler. 2023. Management of machine learning lifecycle artifacts: A survey.ACM SIGMOD Record51, 4 (2023), 18–35

  22. [30]

    Xiang-Rong Sheng, Liqin Zhao, Guorui Zhou, Xinyao Ding, Binding Dai, Qiang Luo, Siran Yang, Jingshan Lv, Chi Zhang, Hongbo Deng, et al. 2021. One model to serve all: Star topology adaptive recommender for multi-domain ctr prediction. In Proceedings of the 30th ACM Internationa...

  23. [31]

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. Advances in neural information processing systems36 (2023), 8634–8652

  24. [32]

    Wenyu Tao, Xiaofen Xing, Yirong Chen, Linyi Huang, and Xiangmin Xu. 2025. Treerag: Unleashing the power of hierarchical storage for enhanced knowledge retrieval in long documents. InFindings of the Association for Computational Linguistics: ACL 2025. 356–371

  25. [33]

    Yancheng Wang, Ziyan Jiang, Zheng Chen, Fan Yang, Yingxue Zhou, Eunah Cho, Xing Fan, Yanbin Lu, Xiaojiang Huang, and Yingzhen Yang. 2024. Recmind: Large language model powered agent for recommendation. InFindings of the Association for Computational Linguistics: NAACL 2024. 4351–4364

  26. [34]

    Yuhao Wang, Xiangyu Zhao, Bo Chen, Qidong Liu, Huifeng Guo, Huanshuo Liu, Yichao Wang, Rui Zhang, and Ruiming Tang. 2023. PLATE: A prompt-enhanced paradigm for multi-scenario recommendations. InProceedings of the 46th In- ternational ACM SIGIR Conference on Research and Develo...

  27. [35]

    Pangjing Wu, Peter Q Chen, Xiaodong Li, Wenqi Fan, and Qing Li. 2026. Datamart- Agent: LLM-Driven Game-Theoretic Agent for Data Marketplace Modeling. In Findings of the Association for Computational Linguistics: ACL 2026. 32509–32531

  28. [36]

    Ya Xu, Weitao Duan, and Shaochen Huang. 2018. SQR: Balancing speed, quality and risk in online experiments. InProceedings of the 24th ACM SIGKDD Interna- tional Conference on Knowledge Discovery & Data Mining. 895–904

  29. [37]

    Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. 2024. Corrective retrieval augmented generation. (2024)

  30. [38]

    John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering.Advances in Neural Information Processing Systems37 (2024), 50528–50652

  31. [39]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629(2022)

  32. [40]

    Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. 2024. Making retrieval- augmented language models robust to irrelevant context. InInternational Confer- ence on Learning Representations, Vol. 2024. 29862–29883

  33. [41]

    Xu Yuan, Liangbo Ning, Wenqi Fan, and Qing Li. 2025. mKG-RAG: Multimodal Knowledge Graph-Enhanced RAG for Visual Question Answering.arXiv preprint arXiv:2508.05318(2025)

  34. [42]

    Yuanliang Zhang, Xiaofeng Wang, Jinxin Hu, Ke Gao, Chenyi Lei, and Fei Fang

  35. [2018]

    In2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)

    Effective online controlled experiment analysis at large scale. In2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). IEEE, 64–67

  36. [2022]

    InProceedings of the 31st ACM International Conference on Information & Knowledge Management

    Scenario-adaptive and self-supervised model for multi-scenario personal- ized recommendation. InProceedings of the 31st ACM International Conference on Information & Knowledge Management. 3674–3683. Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Zhuohang Jiang et al. ...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.