{"id":"a142c8fb-7503-43cb-be4b-67ef6e34dcc1","arxiv_id":"2507.19346","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A CLIP-based multimodal retrieval system beat a conventional supervised two-tower model for short-form video recommendations on an e-commerce platform, with significant watch-progress gains online and less popularity and duration bias.","lead":"Zalando engineers show that a CLIP-based multimodal retrieval system, using product-view history to build user profiles, outperforms a supervised two-tower recommender in an e-commerce short-form video feed. In online tests it increased videos watched to 50% progress by 39% while reducing popularity and duration skew.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline claim of superiority over supervised learning lacks a direct online comparison, as the headline uplift is only versus recency baseline.","rationale":"The reader's verdict was CONDITIONAL with high confidence, and I agree the central claim is plausible but conditional. The reader's weakest_assumption focused on the transfer assumption of the recency-weighted CLIP user embedding as a sufficient representation of video preference. That is a real concern, but I believe the most load-bearing issue is more basic: the headline superiority claim over conventional supervised learning is not directly supported by the reported online experiments. The only online numbers for VCG Multimodal are relative to a recency baseline; the VCG Conventional model was tested separately (and not deployed) and no head-to-head comparison is reported. Offline metrics were non-significant, and the paper explicitly shifted to visual coherence and LVLM-as-a-judge metrics that are aligned with the multimodal method's objective. The LVLM-as-a-judge is itself unvalidated as a proxy for user engagement. Thus the superiority claim is not established by the evidence as reported, even though the production experience is a useful industry report. My agreement_with_reader is 'partial' because the reader identified a plausible modeling concern, but I find the missing head-to-head comparison to be the decisive issue for the paper's strongest claim. The verdict stays CONDITIONAL: the paper could be accepted conditionally on providing a direct comparison or qualifying the claim. I would not change the verdict, but I would strengthen the conditionality.","tokens_in":8059,"tokens_out":1369,"duration_ms":11151,"concrete_test":"Run a single online A/B test with three arms in the same time period and user population: recency baseline, VCG Conventional, and VCG Multimodal, reporting the same watch-time progression metrics (e.g., 25%/50% completion rates) and popularity/duration skewness for all three arms. If VCG Multimodal does not significantly beat VCG Conventional on these metrics, the headline claim of superiority over conventional supervised learning is not supported; if it does, the claim would be substantiated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The abstract and conclusion claim the multimodal retrieval approach demonstrated 'greater effectiveness compared to conventional supervised learning methods' in online experiments. However, the only online experiment reported for VCG Multimodal (Section 4.6) is against the recency-based baseline; no online A/B test with VCG Conventional is described. The VCG Conventional model was tested online but yielded only 'modest gains' and was not deployed, and the two experiments are not reported as a controlled head-to-head comparison under identical conditions. The uplifting statistics (41%, 50%, 30%, 39%) are relative to the recency baseline and say nothing directly about superiority over VCG Conventional. The claim of superiority therefore rests on an implicit comparison across different experiments, with different models, and likely different traffic periods or user populations, which is not a valid basis for the headline claim. The paper's own offline evaluation (Section 4.5) found no significant differences in ranking metrics between VCG methods and the recency baseline, and only the bias-related skewness metrics favored the multimodal approach.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an industrial experience of building a short-form video recommendation feed on an e-commerce platform. The authors describe two personalization approaches: a conventional two-tower video candidate generation model (VCG Conventional) trained on video interaction data, and a multimodal retrieval approach (VCG Multimodal) that scores videos by the dot product of a recency-weighted average of CLIP product embeddings from a user's Browse and Search history with the average of CLIP frame embeddings from a video. The offline evaluation reports engagement-based ranking metrics, visual coherence, and LVLM-as-a-judge relevance scores; the online evaluation reports statistically significant increases in watch-completion rates for VCG Multimodal relative to a recency-based baseline. The central claim is that, even with sufficient video interaction data, a retrieval system built on a fine-tuned multimodal vision-language model can outperform conventional supervised learning methods and mitigate duration and popularity biases.","tokens_in":8210,"tokens_out":2985,"duration_ms":30620,"significance":"If the central claim were fully supported, the paper would be a valuable industrial case study: it addresses a timely problem (cold-start and bias in immersive short-form video feeds), reports production-validated engagement uplifts over a practical recency baseline, and demonstrates a concrete use of frozen/fine-tuned CLIP embeddings and LVLM-as-a-judge evaluation. Strengths include the randomized online experiment with confidence intervals, the explicit discussion of position and duration bias in the immersive-feed setting, and the practical lesson that supervised training on biased video interactions can amplify popularity skew. However, the headline claim of superiority over conventional supervised learning is not directly established by the reported experiments, and the offline evidence in Section 4.5 is explicitly non-significant for the ranking metrics; these gaps materially affect the paper's main contribution.","major_comments":[{"comment":"The abstract states that the multimodal retrieval approach 'demonstrated greater effectiveness compared to conventional supervised learning methods in online experiments,' but Section 4.6 describes no controlled online comparison between VCG Multimodal and VCG Conventional. The reported uplifts (41%, 50%, 30%, 39%) are all relative to the recency-based baseline, while VCG Conventional was tested separately, produced only modest gains, and was not deployed. Comparing two sequentially run experiments with different models and likely different traffic periods does not support a claim of superiority over the conventional supervised approach. This is a load-bearing gap; the wording of the abstract, introduction, and conclusion should be aligned with the evidence, or a head-to-head online experiment should be reported.","section":"Abstract and Section 4.6"},{"comment":"The offline evaluation is the only place where VCG Conventional, VCG Multimodal, and the recency baseline are compared under a common protocol, and the paper states that the ranking-metric differences 'did not reach statistical significance.' The significant offline improvements come from visual coherence and LVLM-as-a-judge scores, but these are not user-engagement metrics, and the paper provides no correlation analysis linking them to online watch-time behavior. The visual coherence metric is also defined using content-based embeddings whose relationship to the CLIP embeddings used in the VCG Multimodal scoring function is not stated; if the same embedding space is used, that metric is partially self-referential as an evaluation of the method. The authors should either validate these offline metrics against online engagement or explicitly present them as auxiliary diagnostics rather than evidence of recommendation superiority.","section":"Section 4.5"},{"comment":"The user embedding e_u is a recency-weighted average of CLIP product embeddings from Browse and Search interactions, and the method's generalization claim rests on the assumption that these product-based preferences transfer to video relevance. However, the paper does not describe the few-shot fine-tuning procedure for the proprietary CLIP model, does not report any diagnostic of the transfer (e.g., whether video preference can be predicted from product embeddings in a held-out set), and does not give the values or sensitivity of the key free parameters (exponential decay factor lambda, number of sampled frames m, and the 50% watch-time threshold). Because the claimed advantage over conventional supervised learning is attributed to this transfer, the missing evidence is load-bearing; the authors should provide at least a basic transfer diagnostic or clearly reframe the claim as an empirical observation specific to their deployment.","section":"Section 3.1 and Section 3"}],"minor_comments":[{"comment":"The sentence 'A schematic overview of the scalable two-tower-based Video Candidate Generation (VCG) architecture is provided in Section 2' appears to refer to a figure or architecture description that is not actually present in Section 2; Section 2 only discusses background, not the VCG schematic.","section":"Section 2"},{"comment":"The LVLM-as-a-judge prompt in Figure 2 is truncated with '<...>', and the paper does not specify how the textual relevance categories were mapped to the 5-point scores used in Table 1; a complete prompt or mapping description would improve reproducibility.","section":"Section 4.4 and Figure 2"},{"comment":"The statement 'VCG Conventional achieved moderate discriminative power with AUC scores of 0.7' would be more informative if the corresponding AUC for VCG Multimodal on the same binary classification task were also reported; otherwise the reader cannot compare the two approaches on this metric.","section":"Section 4.5"},{"comment":"The confidence intervals for the online uplifts are wide (e.g., 17%–61%), which is expected for video metrics, but the paper does not state the sample size or the duration of the online experiment; reporting these details would strengthen the practical significance of the result.","section":"Section 4.6"},{"comment":"The phrase 'conventional supervised learning methods' is used in the abstract, introduction, and conclusion without a precise definition; in practice the comparison is only with the specific VCG Conventional architecture for the offline comparison and with the recency baseline for the online comparison, so the terminology should be narrowed accordingly.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is best read as an industry experience report with a credible practical result (multimodal retrieval beating the recency baseline online), but the abstract and conclusion claim superiority over supervised learning in a way that the reported experiments cannot support. The authors should either soften the claim to match the evidence or add a direct comparison with VCG Conventional under the same online protocol. Given that the claim appears in multiple prominent places, I would ask for the revision to explicitly address this discrepancy before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this if you want a candid take. The paper is a straightforward Zalando experience report on using CLIP-based retrieval for a short-form video feed. What's genuinely useful: they shipped a production system, the online gains are real (41%/50% watch-through increases vs the recency baseline, with confidence intervals), and they write honestly about position and duration bias in immersive feeds. The offline evaluation found no significant ranking gains, and they say so; the bias-skewness results are in their favor. That is a fair and useful pattern for an applied paper.\n\nThe novel part is not the algorithm—averaging CLIP frame embeddings and recency-weighted product embeddings is a known recipe. The newness is the application surface: e-commerce short-form video, with user profiles bootstrapped from catalog Browse/Search interactions. That transfer story is plausible but thin: the few-shot fine-tuning of their proprietary CLIP is not described, and there is no diagnostic of whether the recency-weighted average actually captures video preference. The LVLM-as-a-judge evaluation is interesting but unvalidated against human judgments.\n\nThe main soft spot is the headline claim. The abstract and conclusion say the approach beat 'conventional supervised learning methods' online. The experiments do not support that directly. The only online A/B is VCG Multimodal vs the recency baseline. VCG Conventional was tested separately, in a different experiment, with 'modest gains,' and was not deployed. Those are not head-to-head comparisons. The offline ranking metrics for both VCG methods vs baseline were not significant. So the superiority claim rests on an indirect comparison. It is plausible—the bias reduction is visible in the skewness metrics—but it is not established. The visual coherence metric is also partially self-referential, since it measures exactly what the retrieval method optimizes.\n\nProportionately, these are caveats about the framing rather than fatal flaws. The paper is honest about its offline null results, the production evidence is real, and the practical lessons are useful. The audience is industry practitioners building cold-start video feeds and applied researchers working on multimodal retrieval. It deserves refereeing—with the expectation that the authors either tone down the abstract or provide a direct online comparison.","headline":"A credible industry report with real production uplift evidence, but the abstract's claim of beating supervised learning is not backed by the experiments—the online gains are only vs a recency baseline.","tokens_in":8745,"tokens_out":2574,"would_cite":true,"duration_ms":25039,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For short-form video feeds, CLIP retrieval beat supervised recommenders by 39% in watch-through","keywords":["short-form video recommendations","multimodal retrieval","CLIP embeddings","cold-start recommendation","duration bias","position bias","vision-language model","e-commerce video feed"],"falsifier":"Run an online experiment comparing the CLIP-based retrieval system against a two-tower recommender trained on video interactions with propensity-score corrections for position and duration bias; if the retrieval system no longer shows a statistically significant gain in 50% watch-through, the paper's superiority claim is false. A cheaper offline check: for users with rich video histories, predict held-out video watch events from the recency-weighted product-history embedding and measure AUC; near-chance accuracy would falsify the transfer premise.","tokens_in":7866,"feed_emoji":"🎬","tokens_out":9477,"duration_ms":84511,"temperature":0.7,"pith_summary":"Launching a short-form video feed on an e-commerce platform creates a cold-start problem and two systematic biases: immersive one-at-a-time presentation amplifies position bias, and optimizing for watch-time favors short videos. The paper argues that even after enough user-video interaction data exists, the better solution is not a conventional supervised two-tower recommender but retrieval in a shared embedding space built from a fine-tuned vision-language model (CLIP, a model that maps images and text into one space). The user is represented by a recency-weighted average of CLIP embeddings of products from their Browse and Search history, the video by the average of CLIP embeddings of sampled frames, and relevance is the dot product $e_u^\\top e_v$. Online experiments raised the rate of video starts reaching 50% progress by 39% (CI 17%–61%) and produced flatter duration, popularity, and watch-time distributions, while the supervised model skewed toward short, popular videos and was not deployed. Offline ranking gains were modest and not statistically significant, so the paper's decisive evidence is the online A/B result.","feed_headline":"CLIP retrieval boosts short-form video watch-through by 39%","feed_subtitle":"A product-history embedding replaces supervised video training and removes the short-video and position bias in feeds.","key_machinery":"The load-bearing object is the shared embedding space from CLIP and the dot-product scoring function $f(u,v) = e_u^\\top e_v$. Video embeddings are precomputed by averaging CLIP representations over $m$ uniformly sampled frames; user embeddings are computed online as a weighted average of CLIP product embeddings, with weights $w_k = \\exp(-\\lambda(t_{\\mathrm{now}} - t_k))$ so recent interactions dominate. The paper's adapted CLIP, fine-tuned with few-shot learning on Browse and Search interaction data from the e-commerce catalog, transfers preference knowledge into video space. This construction avoids training on user-video interactions altogether, which is what lets the system sidestep the duration and position biases that contaminated the supervised baseline.","core_discovery":"The paper's central claim is that a multimodal retrieval system based on CLIP embeddings can outperform conventional supervised learning for short-form video recommendations, even when sufficient video interaction data is available, and can do so while avoiding the position and duration biases that plague watch-time optimization. The scoring function is $f(u,v) = e_u^\\top e_v$, where $e_v$ is the mean of CLIP embeddings of $m$ uniformly sampled video frames and $e_u$ is an exponentially recency-weighted average of CLIP product embeddings from the user's interaction history; new users receive a global popular-product embedding. The paper uses a proprietary CLIP variant adapted through few-shot learning on catalog Browse and Search interactions. In online experiments, the multimodal model increased video starts reaching 25% and 50% progress by 30% and 39% respectively, and increased the number of videos watched past 25% and 50% by 41% and 50%, with all uplifts statistically significant; the conventional two-tower model instead produced high popularity and short-video skew and was not deployed.","pith_inferences":["If the shopping-to-video transfer is real, the same recency-weighted product-history embedding could seed recommendations for other cold-start content formats on the same platform—live sessions, editorial lookbooks, styling galleries—without retraining per format.","The paper's online baseline is a recency-based feed, not a debiased supervised model, so the 39% gain may conflate the value of the multimodal representation with the cost of supervised bias; comparing against propensity-debiased supervised training would disentangle these.","An LVLM judge could be promoted from evaluation tool to training signal: ranking losses over judge scores might encode visual coherence directly into the retrieval index."],"forward_implications":["Personalization for a new immersive video feed can start immediately from catalog-derived multimodal embeddings, before any user-video interaction data exists.","Conventional two-tower recommenders trained on watch-time labels with a 50% completion threshold tend to develop popularity and short-video skew and should be audited for these biases before deployment.","A retrieval-based formulation with precomputed video embeddings and online-computed user embeddings can deliver the latency needed for real-time short-form feeds.","Online watch-through metrics can show large, statistically significant gains even when offline ranking metrics are flat, so offline evaluation alone is insufficient for immersive-feed systems."],"supporting_citations":[{"why":"Supplies CLIP, the vision-language model whose shared embedding space carries both user and video representations.","marker":"[15]"},{"why":"Supplies the two-tower deep neural architecture used as the conventional supervised baseline.","marker":"[5]"},{"why":"Documents position bias and multitask ranking in immersive short-form feeds, grounding the paper's central bias claim.","marker":"[13]"},{"why":"Identifies duration bias in watch-time prediction, the failure mode the multimodal approach is designed to avoid.","marker":"[21]"},{"why":"Formalizes duration bias for micro-video recommendation and motivates improvements in watch-time-gain under that bias.","marker":"[22]"},{"why":"Provides inverse propensity scoring, which the paper uses to weight NDCG for position in offline evaluation.","marker":"[14]"},{"why":"Surveys bias and feedback loops in recommender systems, motivating the paper's debiasing focus.","marker":"[4]"},{"why":"Supplies mixed negative sampling for training two-tower recommendation models, part of the conventional baseline's setup.","marker":"[19]"}],"fun_headline_variants":["CLIP retrieval lifts short-video watch-through 39% and cuts bias","Retrieval over supervised: CLIP embeddings fix video feed bias","How CLIP retrieval outperforms supervised video recommender models","Multimodal retrieval beats supervised learning for short-form video","CLIP-based recommender avoids bias, boosts watch-through 39%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a recency-weighted average of product-image embeddings from a user's browse and search history is enough to capture which videos the user wants to watch, and the paper presents no direct test that this shopping-to-video transfer holds.","fun_headline_variants_meta":{"raw":{"variants":["CLIP retrieval lifts short-video watch-through 39% and cuts bias","Retrieval over supervised: CLIP embeddings fix video feed bias","How CLIP retrieval outperforms supervised video recommender models","Multimodal retrieval beats supervised learning for short-form video","CLIP-based recommender avoids bias, boosts watch-through 39%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000964,"raw_usage":{"total_tokens":4125,"prompt_tokens":985,"completion_tokens":3140,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":3050}},"tokens_in":601,"tokens_out":3140,"duration_ms":21691,"temperature":1.0,"reasoning_tokens":3050,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:54:22.741196+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an online experiment comparing the CLIP-based retrieval system against a two-tower recommender trained on video interactions with propensity-score corrections for position and duration bias; if the retrieval system no longer shows a statistically significant gain in 50% watch-through, the paper's superiority claim is false. A cheaper offline check: for users with rich video histories, predict held-out video watch events from the recency-weighted product-history embedding and measure AUC; near-chance accuracy would falsify the transfer premise.","supporting_citations":[],"review_version":2}