REVIEW 8 cited by
Understanding Scaling Laws for Recommendation Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Scale has been a major driving force in improving machine learning performance, and understanding scaling laws is essential for strategic planning for a sustainable model quality performance growth, long-term resource planning and developing efficient system infrastructures to support large-scale models. In this paper, we study empirical scaling laws for DLRM style recommendation models, in particular Click-Through Rate (CTR). We observe that model quality scales with power law plus constant in model size, data size and amount of compute used for training. We characterize scaling efficiency along three different resource dimensions, namely data, parameters and compute by comparing the different scaling schemes along these axes. We show that parameter scaling is out of steam for the model architecture under study, and until a higher-performing model architecture emerges, data scaling is the path forward. The key research questions addressed by this study include: Does a recommendation model scale sustainably as predicted by the scaling laws? Or are we far off from the scaling law predictions? What are the limits of scaling? What are the implications of the scaling laws on long-term hardware/system development?
Forward citations
Cited by 8 Pith papers
-
ROCS: Request-Oriented Compute Sharing for Efficient Large-Scale Recommendation
ROCS restructures recommendation models so user-side computation is shared across all candidate items, yielding up to 3x serving throughput at equal or better prediction quality.
-
Scaling Transformers for Discriminative Recommendation via Generative Pretraining
Generative pretraining plus sparse-embedding freezing makes large Transformer ranking models scale consistently, following a power law from 13K to 0.3B dense parameters.
-
Hi-SAM: A Hierarchical Structure-Aware Multi-modal Framework for Large-Scale Recommendation
Hi-SAM improves semantic-ID multimodal recommendation by disentangling shared versus modality-specific item codes and by letting transformers access history only through compressed anchor tokens.
-
FuXi-\beta: Towards a Lightweight and Fast Large-Scale Generative Recommendation Model
FuXi-β shows that removing query-key attention and using a functional relative time bias makes generative recommendation Transformers faster and, on industrial datasets, more accurate.
-
RankMixer: Scaling Up Ranking Models in Industrial Recommenders
RankMixer scales an industrial ranking model to 1B dense parameters with 10x MFU improvement and unchanged latency, gaining 1.08% in app duration in Douyin A/B tests.
-
Yambda-5B -- A Large-Scale Multi-modal Dataset for Ranking And Retrieval
A new open 4.79B-interaction music dataset from Yandex Music with an is_organic flag, audio embeddings, and a Global Temporal Split benchmark protocol.
-
WHALE: A Scalable Unified Model for Recommendation with Wukong-HSTU Architecture
Layer-wise attention fusion between Wukong-style feature crosses and HSTU-style behavior history improves recommendation quality over each backbone alone at matched FLOPs.
-
Climber: Toward Efficient Scaling Laws for Large Recommendation Models
Climber reports that splitting user sequences by behavior type, adding adaptive temperature, and co-designed batching enable more efficient Transformer scaling in recommender systems.
Discussion (0). Continue with ORCID to comment.