Pith. sign in

REVIEW 3 major objections 6 minor 14 references

1$^{st}$ Place Solution of WWW 2025 EReL@MIR Workshop Multimodal CTR Prediction Challenge

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A Transformer-plus-DCNv2 CTR model that simply appends frozen multimodal embeddings to item IDs won the WWW 2025 EReL@MIR challenge with a leaderboard AUC of 0.9839.

desk verdict A reproducible 1st-place challenge write-up whose headline multimodal claim rests on a single small validation difference; still worth a serious referee for practitioners. read the letter →

arxiv 2505.03543 v1 pith:C35L7K5G submitted 2025-05-06 cs.IR

Junwei Xu , Zehao Zhao , Xiaoyu Hu , Zhenjie Song This is my paper · ORCID
classification cs.IR
keywords CTRpredictionmultimodalembeddingsembeddingconcatenationTransformerDCNv2featureinteractionsequentialrecommendationMicroLens
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is a competition report claiming that a straightforward combination of three ingredients won the multimodal click-through rate prediction task of the WWW 2025 EReL@MIR challenge. The ingredients are a Transformer over the user's recent interaction sequence, a DCNv2 cross network for feature interactions, and frozen multimodal item embeddings that are simply concatenated onto the learnable ID embedding of each item. On the final leaderboard the configuration scored an AUC of 0.9839, well above the provided DIN baseline. The authors' point is not that sophisticated fusion is needed; it is that strong sequential and interaction modeling can absorb precomputed multimodal features by concatenation alone, provided the hyperparameters are tuned well. The practical interest is that the result gives a simple, reproducible baseline for adding content embeddings to CTR models.

What carries the argument

The load-bearing mechanism is a single concatenation: the frozen multimodal vector $e_{mm}$ is appended to the learnable tabular embeddings of every item, giving $e_{item}$. Two off-the-shelf components make that concatenation useful: a Transformer encoder over the target-augmented interaction sequence, with the latest $k$ outputs plus max pooling summarizing user interest, and DCNv2, a cross network that explicitly models high-order feature interactions. The paper's ablation shows both components matter: dropping DCNv2 lowers with-multimodal validation AUC from 0.9776 to 0.9632, and dropping Transformer lowers it to 0.9688. The concatenation itself is the only multimodal-specific design choice; the rest is standard CTR modeling with tuned hyperparameters.

What would settle it

Re-run the final configuration on the challenge validation set with, say, ten random seeds, measuring AUC with and without the frozen multimodal embeddings. If the with-multimodal mean does not consistently exceed the without-multimodal mean—or the reported 0.0047 gap flips sign—the paper's claim that simply concatenating multimodal embeddings improves the model is not established, though the leaderboard finish would remain a competition result.

Watch

Extended reading notes

Core claim

The paper's central claim, on its own terms, is that appending the frozen multimodal embedding to each item embedding is sufficient to improve CTR prediction when the backbone is strong. The final item representation is $e_{item}=[e_{t_1}\|\dots\|e_{t_{|T|}}\|e_{mm}]$; each history embedding is then concatenated with the target embedding and passed through Transformer encoders, whose last-$k$ outputs and max pooling represent short- and long-term interest. A parallel DCNv2 module learns high-order feature interactions on the target, side, and sequence features before a two-layer MLP predicts the click probability. Validation AUC rises from 0.9729 without multimodal embeddings to 0.9776 with them, log loss drops from 0.2369 to 0.2358, and the leaderboard AUC is 0.9839. The paper also observes that weaker backbones (the DIN baseline and the model without Transformer) lose performance when the frozen embeddings are added, which it attributes to misalignment between those embeddings and the CTR objective.

Load-bearing premise

The load-bearing premise is that the reported validation gain from adding multimodal embeddings—0.0047 AUC, from 0.9729 to 0.9776—is real and not a single-run artifact, since no repeated trials or error bars are reported.

Editorial extensions

If this is right

  • The exact winning configuration is reproducible from the released code: two Transformer encoders, three DCNv2 cross layers, embedding dimension 64, dropout 0.2, and learning rate $5\times10^{-4}$.
  • Adding frozen multimodal embeddings by concatenation is a viable default integration strategy for CTR backbones that already have strong sequence and interaction modules.
  • Models whose backbones are too weak or incompatible—like DIN or the no-Transformer variant—can be hurt by naive multimodal concatenation, so backbone strength is a precondition for the gain.
  • Ablation tests on this dataset give a concrete expectation: removing DCNv2 is more damaging than removing the Transformer when multimodal embeddings are present.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: a learned low-rank adapter or projection inserted between the frozen embedding and the concatenation is a natural next test, since the authors identify alignment with the CTR task as the bottleneck.
  • Beyond the paper: the winning recipe suggests a production-friendly pattern for micro-video recommendation: serve frozen content embeddings as side features to a strong sequence-and-interaction model, avoiding end-to-end multimodal training cost.
  • Beyond the paper: the model's sensitivity to learning rate, including collapse at high values, implies that warmup or gradient clipping could make the configuration transfer to other datasets without re-tuning.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper reports the first-place solution for Task 2 of the WWW 2025 EReL@MIR Workshop Multimodal CTR Prediction Challenge. The proposed model combines an embedding layer that concatenates frozen multimodal embeddings with learnable item feature embeddings, a Transformer-based sequential module that incorporates the target item embedding and uses flattening plus max pooling, a DCNv2 feature interaction module, and an MLP prediction layer trained with binary cross-entropy loss. Experiments on the challenge dataset, derived from MicroLens, report a validation AUC of 0.9776 for the full model with multimodal embeddings versus 0.9729 without them, and a final leaderboard AUC of 0.9839, which the authors state ranked first. Code, configuration files, and a trained checkpoint are released.

Significance. If the result holds, the paper provides a reproducible, strong baseline for multimodal CTR prediction on the MicroLens benchmark, and the first-place leaderboard result is external evidence that the proposed architecture and hyperparameter configuration was competitive among challenge submissions. The release of code, configuration, and checkpoint is a concrete strength that should be credited. However, the paper's central claim that 'effectively applying multimodal embedding features' improves CTR rests on a single validation comparison with a small AUC difference and no statistical support, and the ablation results show that the multimodal benefit is not uniform across architectures. The scientific value of the multimodal claim is therefore currently limited, although the leaderboard achievement itself is credible.

major comments (3)
  1. [Section 3.2, Table 2] The claim that appending frozen multimodal embeddings improves performance is supported only by a single run: validation AUC 0.9776 versus 0.9729 and logloss 0.2358 versus 0.2369. No repeated trials, error bars, or significance tests are reported. With a validation set of only 10k samples, a 0.0047 AUC difference may well lie within run-to-run or sampling variability. Please report multiple seeds or a DeLong/bootstrap confidence interval for the AUC difference; without this, the multimodal-effectiveness claim is not established by the evidence as presented.
  2. [Section 3.2, Table 2] The comparison between models 'with multimodal emb.' and 'without' may be biased because the hyperparameters in Table 1 and Section 3.1.3 were tuned for the full multimodal model, while the 'w/o multimodal' variants likely inherit those same settings. If the goal is to isolate the effect of the multimodal embeddings, each configuration should be tuned separately, or the sensitivity of the ablation conclusions to the shared hyperparameters should be demonstrated. This issue is load-bearing for the paper's central multimodal claim.
  3. [Section 3.2, Table 2 and Conclusion] The ablation results show that the effect of multimodal embeddings is not uniformly positive: for the 'w/o Transformer' variant, AUC drops from 0.9741 to 0.9688 when multimodal embeddings are added, and the baseline DIN model degrades substantially (logloss 2.7697 with multimodal embeddings). Given these mixed results, the statement that the full model 'optimally integrates multimodal embeddings with Transformer and DCNv2' overstates what a single configuration pair can establish. The paper should either provide evidence that the 0.0047 gain is reproducible and not configuration-specific, or substantially temper the multimodal-effectiveness claim.
minor comments (6)
  1. [Figure 1] The figure contains typos: 'Feature Interation' and 'User Interation History' should read 'Feature Interaction' and 'User Interaction History'.
  2. [Section 2.2.4] The text says '2-layer perceptron', but the hyperparameters list hidden units [64, 32] for the prediction layer. Please clarify whether this means two hidden layers plus an output layer, or a two-layer network in total.
  3. [Section 3.1.2] FuxiCTR is mentioned as the implementation framework but is not cited in the references. Please add a citation or a URL in the reference list.
  4. [Figure 2] The y-axis label in several panels reads 'Loglosss'; this should be 'Logloss'. Some panels also lack clear axis labels, making the plot harder to read.
  5. [Equations (4)-(5)] The notation around Eq. (4)-(5) is garbled: for example, 'e𝑒𝑖𝑖𝑡𝑒𝑚' and 'f𝑒𝑁 𝑖𝑡𝑒𝑚' appear to be typographical artifacts. Please revise these equations for clarity.
  6. [Section 3.1.3] The text reports 'Numbers of cross layers and Transformer encoders are set to 3 and 2 respectively', but these values are not included in the hyperparameter grid in Table 1. Please add them for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the 0.9839 leaderboard AUC is a held-out measurement, and no claimed result reduces to fitted inputs or self-citations.

full rationale

The paper is an empirical challenge report rather than a derivation-based claim. Its central result, the 1st-place 0.9839 leaderboard AUC, is measured on the challenge's held-out test set, not computed from the paper's own fitted parameters. The model architecture combines Transformer sequential modeling (inspired by TransAct [13]) and DCNv2 feature interaction [12], and the performance comparison in Table 2 is an experimental measurement with and without multimodal embeddings. Hyperparameters in Table 1 are tuned on the validation set, which is standard practice and does not make the final leaderboard result circular. The paper contains no self-citations by the present authors: reference [2] is the workshop description by other authors, and references [8], [12], [13], [14] are external prior work. The admitted limitation that the multimodal embeddings were only concatenated without further tuning, and the lack of error bars or significance tests on the 0.0047 validation-AUC difference, are methodological or statistical weaknesses, not circularity: the ablation comparison could be confounded by hyperparameter selection, but no claim in the paper is true by definition nor does any 'prediction' reduce to a fitted input. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 10 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new entities; it uses standard network modules and existing embeddings. All listed free parameters are tuned on the validation set, which is standard for competition reports. The axioms are domain assumptions about the evaluation setup and data.

free parameters (10)
  • learning_rate = 5e-4
    Tuned by grid search on validation; affects optimization.
  • embedding_dim = 64
    Tuned by grid search; dimension of ID embeddings.
  • transformer_dropout = 0.2
    Tuned by grid search; dropout in Transformer.
  • cross_net_dropout = 0.2
    Tuned by grid search; dropout in DCNv2.
  • k in Eq.6 = 16
    Tuned by grid search; number of latest Transformer outputs used.
  • number of cross layers = 3
    Chosen by hand; not in grid search table.
  • number of Transformer encoders = 2
    Chosen by hand.
  • hidden units of DCNv2 deep net = [1024, 512, 256]
    Chosen by hand.
  • hidden units of prediction MLP = [64, 32]
    Chosen by hand.
  • batch size = 128
    Chosen by hand.
assumptions (3)
  • domain assumption The challenge's train/validation/test split is fixed and the leaderboard AUC is computed by the organizers on a hidden test set.
    The paper's evaluation relies on the organizers' protocol for the final leaderboard result.
  • domain assumption The dataset labels (clicks) are correct and the AUC metric is appropriate for the task.
    The reported performance is measured against these labels and metric.
  • domain assumption The multimodal embeddings, derived from BERT and CLIP with PCA, are frozen and not updated during training.
    Stated in Section 2.2.1: the frozen multimodal embedding is concatenated with item embeddings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of 1$^{st}$ Place Solution of WWW 2025 EReL@MIR Workshop Multimodal CTR Prediction Challenge." pith.science (2026). https://pith.science/paper/C35L7K5G

@misc{pith2026250503543,
  author       = {Pith},
  title        = {Pith review of: 1$^st$ Place Solution of WWW 2025 EReL@MIR Workshop Multimodal CTR Prediction Challenge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C35L7K5G}},
  note         = {Machine review of arXiv:2505.03543}
}
abstract

The WWW 2025 EReL@MIR Workshop Multimodal CTR Prediction Challenge focuses on effectively applying multimodal embedding features to improve click-through rate (CTR) prediction in recommender systems. This technical report presents our 1$^{st}$ place winning solution for Task 2, combining sequential modeling and feature interaction learning to effectively capture user-item interactions. For multimodal information integration, we simply append the frozen multimodal embeddings to each item embedding. Experiments on the challenge dataset demonstrate the effectiveness of our method, achieving superior performance with a 0.9839 AUC on the leaderboard, much higher than the baseline model. Code and configuration are available in our GitHub repository and the checkpoint of our model can be found in HuggingFace.

Figures

Figures reproduced from arXiv: 2505.03543 by the authors.

Figure 1
Figure 1. The overall architecture of our model. 2.2.1 Embedding Layer. The Embedding Layer converts different types of input features into dense vector representations. Let 𝑥 = {𝑡1, 𝑡2, . . . , 𝑡|𝑇 | , 𝑒𝑚𝑚} be the item features, where 𝑡𝑖 is the 𝑖-th feature of the item, |𝑇 | is the number of features and 𝑒𝑚𝑚 is the frozen multimodal embedding of the item. The embedding layer maps each feature 𝑡𝑖 to a dense vector 𝑒𝑖 using an… view at source ↗
Figure 2
Figure 2. Parameter sensitivity analysis on hyperparameters. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 2 canonical work pages

  1. [1]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Jill Burstein, Christy...

    2019

  2. [2]

    Junchen Fu, Xuri Ge, Xin Xin, Haitao Yu, Yue Feng, Alexandros Karatzoglou, Ioannis Arapakis, and Joemon M. Jose. 2025. The 1st EReLMIR Workshop on Efficient Representation Learning for Multimodal Information Retrieval. arXiv:2504.14788 [cs.IR] https://arxiv.org/abs/2504.14788

    arXiv 2025

  3. [3]

    Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. DeepFM: a factorization-machine based neural network for CTR prediction. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (Melbourne, Australia) (IJCAI’17). AAAI Press, 1725—1731

    work page 2017

  4. [4]

    Xiangnan He and Tat-Seng Chua. 2017. Neural Factorization Machines for Sparse Predictive Analytics. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (Shinjuku, Tokyo, Japan) (SIGIR ’17). Association for Computing Machinery, New York, NY, USA, 355—364. doi:10.1145/3077136.3080777

    arXiv 2017

  5. [5]

    Yuchin Juan, Yong Zhuang, Wei-Sheng Chin, and Chih-Jen Lin. 2016. Field- aware Factorization Machines for CTR Prediction. InProceedings of the 10th ACM Conference on Recommender Systems (Boston, Massachusetts, USA) (RecSys ’16). Association for Computing Machinery, New York, NY, USA, 43—50. doi:10.1145/ 2959100.2959134

    arXiv 2016

  6. [6]

    Wang-Cheng Kang and Julian McAuley. 2018. Self-Attentive Sequential Rec- ommendation . In 2018 IEEE International Conference on Data Mining (ICDM) . IEEE Computer Society, Los Alamitos, CA, USA, 197–206. doi:10.1109/ICDM. 2018.00035

    arXiv 2018

  7. [7]

    Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun. 2018. xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (London, United Kingdom) (KDD ’18). Association for Computing Machinery, Ne...

    arXiv 2018

  8. [8]

    Yongxin Ni, Yu Cheng, Xiangyan Liu, Junchen Fu, Youhua Li, Xiangnan He, Yongfeng Zhang, and Fajie Yuan. 2023. A Content-Driven Micro-Video Recom- mendation Dataset at Scale. arXiv preprint arXiv:2309.15379 (2023)

    arXiv 2023

Show all 14 references
  1. [9]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings...

    2021

  2. [10]

    Steffen Rendle. 2010. Factorization Machines. In Proceedings of the 2010 IEEE International Conference on Data Mining (ICDM ’10) . IEEE Computer Society, USA, 995—1000. doi:10.1109/ICDM.2010.127

    2010 doi

  3. [11]

    Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. 2019. AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management ...

    2019

  4. [12]

    Ruoxi Wang, Rakesh Shivanna, Derek Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed Chi. 2021. DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems. In Proceedings of the Web Conference 2021 (Ljubljana, Slovenia) (WWW ’21). Associa...

    2021

  5. [13]

    Xue Xia, Pong Eksombatchai, Nikil Pancha, Dhruvil Deven Badani, Po-Wei Wang, Neng Gu, Saurabh Vishwas Joshi, Nazanin Farahpour, Zhiyuan Zhang, and Andrew Zhai. 2023. TransAct: Transformer-based Realtime User Action Model for Recommendation at Pinterest. In Proceedings of the 2...

    2023

  6. [14]

    Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018. Deep Interest Network for Click- Through Rate Prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining...

    2018

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.