Pith. sign in

REVIEW 3 major objections 5 minor 25 references

Short-Form Video Recommendations with Multimodal Embeddings: Addressing Cold-Start and Bias Challenges

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read For short-form video feeds, CLIP retrieval beat supervised recommenders by 39% in watch-through

desk verdict A credible industry report with real production uplift evidence, but the abstract's claim of beating supervised learning is not backed by the experiments—the online gains are only vs a recency baseline. read the letter →

arxiv 2507.19346 v2 pith:4SRTQZ2T submitted 2025-07-25 cs.LG

Andrii Dzhoha , Katya Mirylenka , Egor Malykh , Marco-Andrea Buchmann , Francesca Catino This is my paper · ORCID
classification cs.LG
keywords short-formvideorecommendationsmultimodalretrievalCLIPembeddingscold-startrecommendationdurationbiaspositionvision-languagemodele-commercefeed
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Launching a short-form video feed on an e-commerce platform creates a cold-start problem and two systematic biases: immersive one-at-a-time presentation amplifies position bias, and optimizing for watch-time favors short videos. The paper argues that even after enough user-video interaction data exists, the better solution is not a conventional supervised two-tower recommender but retrieval in a shared embedding space built from a fine-tuned vision-language model (CLIP, a model that maps images and text into one space). The user is represented by a recency-weighted average of CLIP embeddings of products from their Browse and Search history, the video by the average of CLIP embeddings of sampled frames, and relevance is the dot product $e_u^\top e_v$. Online experiments raised the rate of video starts reaching 50% progress by 39% (CI 17%–61%) and produced flatter duration, popularity, and watch-time distributions, while the supervised model skewed toward short, popular videos and was not deployed. Offline ranking gains were modest and not statistically significant, so the paper's decisive evidence is the online A/B result.

What carries the argument

The load-bearing object is the shared embedding space from CLIP and the dot-product scoring function $f(u,v) = e_u^\top e_v$. Video embeddings are precomputed by averaging CLIP representations over $m$ uniformly sampled frames; user embeddings are computed online as a weighted average of CLIP product embeddings, with weights $w_k = \exp(-\lambda(t_{\mathrm{now}} - t_k))$ so recent interactions dominate. The paper's adapted CLIP, fine-tuned with few-shot learning on Browse and Search interaction data from the e-commerce catalog, transfers preference knowledge into video space. This construction avoids training on user-video interactions altogether, which is what lets the system sidestep the duration and position biases that contaminated the supervised baseline.

What would settle it

Run an online experiment comparing the CLIP-based retrieval system against a two-tower recommender trained on video interactions with propensity-score corrections for position and duration bias; if the retrieval system no longer shows a statistically significant gain in 50% watch-through, the paper's superiority claim is false. A cheaper offline check: for users with rich video histories, predict held-out video watch events from the recency-weighted product-history embedding and measure AUC; near-chance accuracy would falsify the transfer premise.

Watch

Extended reading notes

Core claim

The paper's central claim is that a multimodal retrieval system based on CLIP embeddings can outperform conventional supervised learning for short-form video recommendations, even when sufficient video interaction data is available, and can do so while avoiding the position and duration biases that plague watch-time optimization. The scoring function is $f(u,v) = e_u^\top e_v$, where $e_v$ is the mean of CLIP embeddings of $m$ uniformly sampled video frames and $e_u$ is an exponentially recency-weighted average of CLIP product embeddings from the user's interaction history; new users receive a global popular-product embedding. The paper uses a proprietary CLIP variant adapted through few-shot learning on catalog Browse and Search interactions. In online experiments, the multimodal model increased video starts reaching 25% and 50% progress by 30% and 39% respectively, and increased the number of videos watched past 25% and 50% by 41% and 50%, with all uplifts statistically significant; the conventional two-tower model instead produced high popularity and short-video skew and was not deployed.

Load-bearing premise

The method assumes that a recency-weighted average of product-image embeddings from a user's browse and search history is enough to capture which videos the user wants to watch, and the paper presents no direct test that this shopping-to-video transfer holds.

Editorial extensions

If this is right

  • Personalization for a new immersive video feed can start immediately from catalog-derived multimodal embeddings, before any user-video interaction data exists.
  • Conventional two-tower recommenders trained on watch-time labels with a 50% completion threshold tend to develop popularity and short-video skew and should be audited for these biases before deployment.
  • A retrieval-based formulation with precomputed video embeddings and online-computed user embeddings can deliver the latency needed for real-time short-form feeds.
  • Online watch-through metrics can show large, statistically significant gains even when offline ranking metrics are flat, so offline evaluation alone is insufficient for immersive-feed systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the shopping-to-video transfer is real, the same recency-weighted product-history embedding could seed recommendations for other cold-start content formats on the same platform—live sessions, editorial lookbooks, styling galleries—without retraining per format.
  • The paper's online baseline is a recency-based feed, not a debiased supervised model, so the 39% gain may conflate the value of the multimodal representation with the cost of supervised bias; comparing against propensity-debiased supervised training would disentangle these.
  • An LVLM judge could be promoted from evaluation tool to training signal: ranking losses over judge scores might encode visual coherence directly into the retrieval index.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reports an industrial experience of building a short-form video recommendation feed on an e-commerce platform. The authors describe two personalization approaches: a conventional two-tower video candidate generation model (VCG Conventional) trained on video interaction data, and a multimodal retrieval approach (VCG Multimodal) that scores videos by the dot product of a recency-weighted average of CLIP product embeddings from a user's Browse and Search history with the average of CLIP frame embeddings from a video. The offline evaluation reports engagement-based ranking metrics, visual coherence, and LVLM-as-a-judge relevance scores; the online evaluation reports statistically significant increases in watch-completion rates for VCG Multimodal relative to a recency-based baseline. The central claim is that, even with sufficient video interaction data, a retrieval system built on a fine-tuned multimodal vision-language model can outperform conventional supervised learning methods and mitigate duration and popularity biases.

Significance. If the central claim were fully supported, the paper would be a valuable industrial case study: it addresses a timely problem (cold-start and bias in immersive short-form video feeds), reports production-validated engagement uplifts over a practical recency baseline, and demonstrates a concrete use of frozen/fine-tuned CLIP embeddings and LVLM-as-a-judge evaluation. Strengths include the randomized online experiment with confidence intervals, the explicit discussion of position and duration bias in the immersive-feed setting, and the practical lesson that supervised training on biased video interactions can amplify popularity skew. However, the headline claim of superiority over conventional supervised learning is not directly established by the reported experiments, and the offline evidence in Section 4.5 is explicitly non-significant for the ranking metrics; these gaps materially affect the paper's main contribution.

major comments (3)
  1. [Abstract and Section 4.6] The abstract states that the multimodal retrieval approach 'demonstrated greater effectiveness compared to conventional supervised learning methods in online experiments,' but Section 4.6 describes no controlled online comparison between VCG Multimodal and VCG Conventional. The reported uplifts (41%, 50%, 30%, 39%) are all relative to the recency-based baseline, while VCG Conventional was tested separately, produced only modest gains, and was not deployed. Comparing two sequentially run experiments with different models and likely different traffic periods does not support a claim of superiority over the conventional supervised approach. This is a load-bearing gap; the wording of the abstract, introduction, and conclusion should be aligned with the evidence, or a head-to-head online experiment should be reported.
  2. [Section 4.5] The offline evaluation is the only place where VCG Conventional, VCG Multimodal, and the recency baseline are compared under a common protocol, and the paper states that the ranking-metric differences 'did not reach statistical significance.' The significant offline improvements come from visual coherence and LVLM-as-a-judge scores, but these are not user-engagement metrics, and the paper provides no correlation analysis linking them to online watch-time behavior. The visual coherence metric is also defined using content-based embeddings whose relationship to the CLIP embeddings used in the VCG Multimodal scoring function is not stated; if the same embedding space is used, that metric is partially self-referential as an evaluation of the method. The authors should either validate these offline metrics against online engagement or explicitly present them as auxiliary diagnostics rather than evidence of recommendation superiority.
  3. [Section 3.1 and Section 3] The user embedding e_u is a recency-weighted average of CLIP product embeddings from Browse and Search interactions, and the method's generalization claim rests on the assumption that these product-based preferences transfer to video relevance. However, the paper does not describe the few-shot fine-tuning procedure for the proprietary CLIP model, does not report any diagnostic of the transfer (e.g., whether video preference can be predicted from product embeddings in a held-out set), and does not give the values or sensitivity of the key free parameters (exponential decay factor lambda, number of sampled frames m, and the 50% watch-time threshold). Because the claimed advantage over conventional supervised learning is attributed to this transfer, the missing evidence is load-bearing; the authors should provide at least a basic transfer diagnostic or clearly reframe the claim as an empirical observation specific to their deployment.
minor comments (5)
  1. [Section 2] The sentence 'A schematic overview of the scalable two-tower-based Video Candidate Generation (VCG) architecture is provided in Section 2' appears to refer to a figure or architecture description that is not actually present in Section 2; Section 2 only discusses background, not the VCG schematic.
  2. [Section 4.4 and Figure 2] The LVLM-as-a-judge prompt in Figure 2 is truncated with '<...>', and the paper does not specify how the textual relevance categories were mapped to the 5-point scores used in Table 1; a complete prompt or mapping description would improve reproducibility.
  3. [Section 4.5] The statement 'VCG Conventional achieved moderate discriminative power with AUC scores of 0.7' would be more informative if the corresponding AUC for VCG Multimodal on the same binary classification task were also reported; otherwise the reader cannot compare the two approaches on this metric.
  4. [Section 4.6] The confidence intervals for the online uplifts are wide (e.g., 17%–61%), which is expected for video metrics, but the paper does not state the sample size or the duration of the online experiment; reporting these details would strengthen the practical significance of the result.
  5. [General] The phrase 'conventional supervised learning methods' is used in the abstract, introduction, and conclusion without a precise definition; in practice the comparison is only with the specific VCG Conventional architecture for the offline comparison and with the recency baseline for the online comparison, so the terminology should be narrowed accordingly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim rests on independent online experiments against a recency baseline, and the offline metrics are not shown to reduce to the model's own scoring function.

full rationale

This paper is an empirical systems report rather than a derivation chain. The VCG Multimodal scoring function f(u,v)=e_u^T e_v is constructed from CLIP embeddings of user product history and video frames (Section 3.1), and the reported online uplifts (Section 4.6) are measured against a recency-based production baseline with confidence intervals, so these results do not reduce to the model's inputs. The offline evaluation uses NDCG with inverse propensity scores, external LVLM-as-a-judge ratings from Qwen 2.5-VL, and a visual-coherence metric. Although the visual-coherence metric is structurally similar to the CLIP retrieval objective (dot products of product-image embeddings), the paper does not state that the content-based embeddings used in Section 4.3 are the same adapted CLIP embeddings as in Section 3.1, and this metric is secondary; the watch-time completion results are independent of it. The claim that the approach outperforms 'conventional supervised learning methods' is not backed by a direct controlled online comparison with VCG Conventional, but that is an evidence and overclaim issue, not a circularity. Self-citations (Refs. 1, 3, 6, 7) are peripheral and not load-bearing. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported, and no known result is merely renamed. The paper even reports that offline ranking-metric differences were not statistically significant, which undercuts any suggestion that the offline evaluation was rigged to favor the method. The score is therefore 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new physical or mathematical entities are introduced; the method relies on known ML components and several domain assumptions about transfer from product interactions to video preference and about the validity of external evaluation.

free parameters (3)
  • Exponential decay factor lambda = not reported
    Weights recent product interactions in the user embedding e_u; the chosen value is not disclosed.
  • Number of sampled frames m = not reported
    Number of uniformly sampled video frames averaged to form the video embedding e_v; not specified.
  • Watch-time relevance threshold = 50%
    Positive labels are defined as watch time exceeding 50% of a video's duration; used in training and offline evaluation.
assumptions (5)
  • domain assumption CLIP embeddings of products and video frames capture fashion attributes relevant to video enjoyment.
    Section 3 architecture assumes product images and text share a semantically meaningful space with video frames.
  • domain assumption Recency-weighted averaging of user product embeddings yields a valid user preference profile.
    e_u is defined as an exponential-decay-weighted average of product CLIP embeddings; no validation of this aggregation is provided.
  • domain assumption Watch time above 50% is a valid proxy for video relevance.
    Used to define positives in training and ground truth in offline evaluation.
  • domain assumption Qwen 2.5-VL relevance ratings reflect user-perceived relevance.
    LVLM-as-a-judge is used as an external evaluation signal; the paper states that quantifying its reliability is future work.
  • domain assumption Inverse propensity score weighting adequately corrects position bias in offline NDCG.
    NDCG is weighted by inverse propensity scores to account for the position of relevant items.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Short-Form Video Recommendations with Multimodal Embeddings: Addressing Cold-Start and Bias Challenges." pith.science (2026). https://pith.science/paper/4SRTQZ2T

@misc{pith2026250719346,
  author       = {Pith},
  title        = {Pith review of: Short-Form Video Recommendations with Multimodal Embeddings: Addressing Cold-Start and Bias Challenges},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4SRTQZ2T}},
  note         = {Machine review of arXiv:2507.19346}
}
read the original abstract

In recent years, social media users have spent significant amounts of time on short-form video platforms. As a result, established platforms in other domains, such as e-commerce, have begun introducing short-form video content to engage users and increase their time spent on the platform. The success of these experiences is due not only to the content itself but also to a unique UI innovation: instead of offering users a list of choices to click, platforms actively recommend content for users to watch one at a time. This creates new challenges for recommender systems, especially when launching a new video experience. Beyond the limited interaction data, immersive feed experiences introduce stronger position bias due to the UI and duration bias when optimizing for watch-time, as models tend to favor shorter videos. These issues, together with the feedback loop inherent in recommender systems, make it difficult to build effective solutions. In this paper, we highlight the challenges faced when introducing a new short-form video experience and present our experience showing that, even with sufficient video interaction data, it can be more beneficial to leverage a video retrieval system using a fine-tuned multimodal vision-language model to overcome these challenges. This approach demonstrated greater effectiveness compared to conventional supervised learning methods in online experiments conducted on our e-commerce platform.

Figures

Figures reproduced from arXiv: 2507.19346 by the authors.

Figure 1
Figure 1. Overview of the multimodal retrieval architecture. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Distribution of LVLM-as-a-judge scores for VCG [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 2
Figure 2. LLM prompt for LVLM-as-a-judge evaluation. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 2 canonical work pages

  1. [1]

    Debarun Bhattacharjya, Balaji Ganesan, Junkyu Lee, Radu Marinescu, Katsiaryna Mirylenka, Michael Glass, and Xiao Shou. 2025. SIMBA UQ: Similarity-Based Ag- gregation for Uncertainty Quantification in Large Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2025

    work page 2025

  2. [2]

    Qingpeng Cai, Shuchang Liu, Xueliang Wang, Tianyou Zuo, Wentao Xie, Bin Yang, Dong Zheng, Peng Jiang, and Kun Gai. 2023. Reinforcing User Retention in a Billion Scale Short Video Recommender System. In Companion Proceedings of the ACM Web Conference 2023 (Austin, TX, USA) (WWW ’23 Companion) . Association for Computing Machinery, New York, NY, USA, 421–42...

    arXiv 2023

  3. [3]

    Marjan Celikik, Jacek Wasilewski, Ana Peleteiro Ramallo, Alexey Kurennoy, Evgeny Labzin, Danilo Ascione, Tural Gurbanov, Géraud Le Falher, Andrii Dzhoha, and Ian Harris. 2024. Building a Scalable, Effective, and Steerable Search and Ranking Platform. arXiv:2409.02856 [cs.IR] https://arxiv.org/abs/2409.02856

    arXiv 2024

  4. [4]

    Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He

  5. [5]

    Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. In Proceedings of the 10th ACM Conference on Recommender Systems (Boston, Massachusetts, USA) (RecSys ’16). Association for Computing Machinery, New York, NY, USA, 191–198. doi:10.1145/2959100. 2959190

    doi:10.1145/2959100 2016

  6. [6]

    Andrii Dzhoha, Alexey Kurennoy, Vladimir Vlasov, and Marjan Celikik. 2024. Re- ducing Popularity Influence by Addressing Position Bias. arXiv:2412.08780 [cs.IR] https://arxiv.org/abs/2412.08780

    arXiv 2024

  7. [7]

    Andrii Dzhoha, Alisa Mironenko, Vladimir Vlasov, Maarten Versteegh, and Mar- jan Celikik. 2025. Efficient and Effective Query Context-Aware Learning-to- Rank Model for Sequential Recommendation. arXiv:2507.03789 [cs.IR] https: //arxiv.org/abs/2507.03789

    arXiv 2025

  8. [8]

    Yingqiang Ge, Shuya Zhao, Honglu Zhou, Changhua Pei, Fei Sun, Wenwu Ou, and Yongfeng Zhang. 2020. Understanding Echo Chambers in E-commerce Recommender Systems. 2261–2270. doi:10.1145/3397271.3401431

    arXiv 2020

Show all 25 references
  1. [9]

    Xudong Gong, Qinlin Feng, Yuan Zhang, Jiangling Qin, Weijie Ding, Biao Li, Peng Jiang, and Kun Gai. 2022. Real-time Short Video Recommendation on Mobile Devices. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management (Atlanta, GA, USA) (C...

    2022

  2. [10]

    Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017. Unbiased Learning-to-Rank with Biased Feedback. In Proceedings of the Tenth ACM Interna- tional Conference on Web Search and Data Mining (Cambridge, United Kingdom) (WSDM ’17). Association for Computing Machinery...

    2017

  3. [11]

    Evgeny Krivosheev, Mattia Atzeni, Katsiaryna Mirylenka, Paolo Scotton, Christoph Miksovic, and Anton Zorin. 2021. Business entity matching with siamese graph convolutional networks. In Proceedings of the AAAI Conference on Artificial Intelligence

    2021

  4. [12]

    Zihan Lin, Hui Wang, Jingshu Mao, Wayne Xin Zhao, Cheng Wang, Peng Jiang, and Ji-Rong Wen. 2022. Feature-aware Diversified Re-ranking with Disentangled Representations for Relevant Recommendation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data...

    2022

  5. [13]

    Qingyun Liu, Zhe Zhao, Liang Liu, Zhen Zhang, Junjie Shan, Yuening Li, Shuchao Bi, Lichan Hong, and Ed H. Chi. 2023. Multitask Ranking System for Immersive Feed and No More Clicks: A Case Study of Short-Form Video Recommendation. In Proceedings of the 32nd ACM International Co...

    2023 doi

  6. [14]

    Zohreh Ovaisi, Ragib Ahsan, Yifan Zhang, Kathryn Vasilaky, and Elena Zheleva

  7. [15]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  8. [16]

    Yuta Saito, Suguru Yaginuma, Yuta Nishino, Hayato Sakata, and Kazuhide Nakata

  9. [17]

    Ali Vardasbi, Harrie Oosterhuis, and Maarten de Rijke. 2020. When Inverse Propensity Scoring does not Work: Affine Corrections for Unbiased Learning to Rank. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (Virtual Event, Ireland) ...

    2020 doi

  10. [18]

    Xuanhui Wang, Michael Bendersky, Donald Metzler, and Marc Najork. 2016. Learning to Rank with Selection Bias in Personal Search. InProceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (Pisa, Italy) (SIGIR ’16). Associa...

    2016

  11. [19]

    In Proceedings of the 13th International Conference on Web Search and Data Mining (Houston, TX, USA) (WSDM ’20)

    Unbiased Recommender Learning from Missing-Not-At-Random Implicit Feedback. In Proceedings of the 13th International Conference on Web Search and Data Mining (Houston, TX, USA) (WSDM ’20). Association for Computing Machinery, New York, NY, USA, 501–509. doi:10.1145/3336191.3371783

  12. [20]

    Hongzhi Yin, Bin Cui, Jing Li, Junjie Yao, and Chen Chen. 2012. Challenging the long tail recommendation. Proc. VLDB Endow. 5, 9 (may 2012), 896–907. doi:10.14778/2311906.2311916

    2012

  13. [21]

    Ruohan Zhan, Changhua Pei, Qiang Su, Jianfeng Wen, Xueliang Wang, Guanyu Mu, Dong Zheng, Peng Jiang, and Kun Gai. 2022. Deconfounding Duration Bias in Watch-time Prediction for Video Recommendation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Da...

    2022

  14. [22]

    Ji Yang, Xinyang Yi, Derek Zhiyuan Cheng, Lichan Hong, Yang Li, Simon Xiaom- ing Wang, Taibai Xu, and Ed H. Chi. 2020. Mixed Negative Sampling for Learning Two-tower Neural Networks in Recommendations. In Companion Proceedings of the Web Conference 2020 (Taipei, Taiwan)(WWW ’2...

    2020

  15. [25]

    Yu Zheng, Chen Gao, Jingtao Ding, Lingling Yi, Depeng Jin, Yong Li, and Meng Wang. 2022. DVR: Micro-Video Recommendation Optimizing Watch-Time-Gain under Duration Bias. In Proceedings of the 30th ACM International Conference on Multimedia (Lisboa, Portugal) (MM ’22). Associati...

    2022

  16. [2020]

    InProceedings of The Web Conference 2020(Taipei, Taiwan)(WWW ’20)

    Correcting for Selection Bias in Learning-to-rank Systems. InProceedings of The Web Conference 2020(Taipei, Taiwan)(WWW ’20). Association for Computing Machinery, New York, NY, USA, 1863–1873. doi:10.1145/3366423.3380255

    2020

  17. [2023]

    ACM Trans

    Bias and Debias in Recommender System: A Survey and Future Directions. ACM Trans. Inf. Syst. 41, 3, Article 67 (feb 2023), 39 pages. doi:10.1145/3564284

    2023 doi

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.