{"id":"4c3894b1-4810-47d0-a57e-66239488e8a4","arxiv_id":"2607.17017","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Combining Wukong feature-interaction layers with HSTU sequence layers in a per-layer attention-fused architecture improves large-scale recommendation quality over either backbone alone and is deployed online.","lead":"This paper introduces WHALE, a recommendation model that combines two existing Meta architectures—Wukong for feature interactions and HSTU for user-history sequences—and lets them exchange information at every layer. On Meta's short-form video platform it reports offline quality gains and positive online A/B results with a 5% serving-throughput cost, and the model is deployed.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FLOPs-alignment fairness is the linchpin: amortizing WHALE's K/V projections across candidates may undercount per-request compute relative to Wukong-only, so 'comparable complexity' gains may not reflect equal serving cost.","rationale":"The reader's weakest_assumption correctly identifies the FLOPs-based complexity alignment as the load-bearing condition. I reviewed the manuscript in good faith: the architecture is clearly described, the ablations support the fusion design, and the online A/B test provides real-world evidence. The explicit footnote about request-only computations and M-FALCON amortization is a sign of transparency, but it introduces an asymmetry that the paper never resolves. Wukong-only and HSTU-only baselines have very different amortization profiles: Wukong processes per-candidate non-sequence features, while HSTU processes a per-request sequence that can be shared. By reporting per-example FLOPs that include request-only computations divided across candidates, the paper may understate WHALE's true serving cost relative to Wukong-only, making the 'comparable complexity' comparison misleading. The concrete test I propose—recomputing Fig. 3 with per-request FLOPs—would settle this directly. I considered other potential concerns: the fusion is one-directional (Wukong queries HSTU but the sequence branch receives no non-sequence information), but this does not invalidate the empirical claim; the lack of error bars on offline NE is a reproducibility issue but unlikely to change the ordering at 4B samples; the absence of direct comparison to Kunlun/OneTrans affects novelty, not correctness. None of these are as central to the main claim as the FLOPs accounting. Therefore I agree with the reader and recommend keeping the CONDITIONAL verdict unchanged until the per-request FLOPs check is performed.","tokens_in":15310,"tokens_out":9619,"duration_ms":104348,"concrete_test":"Obtain from the authors a per-request FLOPs breakdown for Wukong-only, HSTU-only, and WHALE at the three operating points in Fig. 3 (8, 14, 32 GFLOPs), including the candidate count C used in the M-FALCON amortization and the exact amortization formula. Then reconstruct the comparison using total per-request FLOPs on the x-axis instead of per-example FLOPs. If WHALE's NE gain no longer stays above the baselines at equal per-request FLOPs, the 'comparable complexity' claim is not established. A complementary check: report total per-request inference FLOPs for all three models at matched per-example FLOPs; if WHALE's per-request FLOPs exceed the baselines by more than the 5% QPS regression, the complexity alignment is not serving-cost-equivalent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that WHALE 'outperforms at comparable complexity' (Fig. 3) rests entirely on the FLOPs-based complexity alignment. Footnote 1 states that reported FLOPs include request-only computations and that WHALE's K/V projections are amortized per request using M-FALCON across hundreds of candidates. However, the paper does not specify whether the same amortization is applied to the HSTU-only and Wukong-only baselines, nor does it report the candidate count C used in the amortization. Wukong-only has no sequence branch, so its per-example FLOPs are likely per-candidate and cannot be amortized; HSTU-only can amortize its sequence branch. If WHALE's per-example FLOPs divide the HSTU request computation by C while Wukong-only's are not divided, then matching per-example FLOPs does not correspond to equal per-request serving cost—and WHALE may be substantially more expensive per request. The observed 5% inference-QPS regression is consistent with this possibility. If the authors instead used inconsistent accounting—per-request for Wukong and per-example for WHALE—the comparison is unfair. Because the headline scalability advantage depends on this alignment, this is the most load-bearing concern; other issues (one-way fusion, missing hybrid baselines, no error bars) are secondary and would not by themselves overturn the empirical result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes WHALE, a stacked recommendation architecture that keeps a Wukong-style feature-interaction branch and an HSTU-style sequence branch active at every layer. Each WHALE layer contains a Wukong module, an HSTU module, and an attention-based fusion module in which non-sequence interaction representations query fine-grained behavior-history representations; the fused output is refined by a residual MLP projection and a SwiGLU FFN before being passed to the next layer. The authors also describe training and inference optimizations: a fused Triton attention kernel with shared key/value, Q-parallel and KV-parallel backward variants, a shared-gate SwiGLU FFN, mixed-precision execution, AOTInductor, and shape-hint tensors to avoid CPU-GPU synchronization. On large proprietary industrial data, WHALE reports consistent NE gains over Wukong-only and HSTU-only baselines at matched FLOPs, monotonic improvements when scaling sequence length, depth, and width, ablation results favoring attention-based progressive fusion over shallow-hybrid and average-pooling baselines, and a positive online A/B result with a 5% inference QPS regression. The central claim is that keeping both backbones active and exchanging information at every layer is a practical and scalable unification principle.","tokens_in":15608,"tokens_out":5877,"duration_ms":61315,"significance":"If the FLOPs-based complexity alignment is fair, the paper makes a meaningful practical contribution: it provides a concrete industrial architecture that unifies two complementary scaling paradigms and shows via ablations that the layer-wise attention-based exchange, rather than simply adding capacity, is responsible for the gains. The paper is honest about the serving trade-off and includes detailed system optimizations, and the deployment evidence is valuable. Its main strength is the coherence of the design: the equations are clear, the ablations isolate the proposed components, and the online A/B result shows real-world impact. The principal weakness is that the headline comparison against single-paradigm baselines depends on an underspecified FLOPs amortization rule; until that accounting is clarified, the claim of outperforming at comparable complexity is not fully established.","major_comments":[{"comment":"The headline comparison in Fig. 3 ('outperforms at comparable complexity') relies on FLOPs-based complexity alignment. Footnote 1 states that FLOPs include request-only computations and that WHALE's K/V projections are amortized via M-FALCON across 'hundreds of examples,' but it never states the candidate count C, nor whether the same amortization is applied to the Wukong-only and HSTU-only baselines. Wukong-only has no request-shared sequence branch, so if its per-example FLOPs are unamortized while WHALE's are divided by C, the matched-FLOPs curves do not represent equal per-request serving cost. The 5% inference QPS regression in Table 2 is consistent with WHALE being more expensive per request. Please report per-request FLOPs, C, QPS/latency at matched FLOPs, and use identical amortization rules across all architectures.","section":"§5.1, §5.2, Footnote 1"},{"comment":"The online A/B test compares WHALE against an unspecified 'online baseline.' It is not stated whether this baseline is Wukong-only, HSTU-only, or a different production model. Without this information, the online lift of +0.113% cannot be attributed to the unified Wukong–HSTU design; it may simply reflect an improved production system. Please identify the baseline architecture and, if available, include A/B comparisons against the single-paradigm systems used offline.","section":"§5.5, Table 2"},{"comment":"All offline results are single runs without confidence intervals or significance tests. The paper states that a 0.05% NE gain is considered noticeable; several ablation differences in Table 1 are 0.08% and 0.11%, and the Fig. 3 curves have no error bars. At these effect sizes, run-to-run variance could change the interpretation of which fusion components matter. Please provide repeated runs, standard deviations, or significance tests for at least the headline comparisons and the main ablation rows.","section":"§5.1, §5.2, Table 1"}],"minor_comments":[{"comment":"The symbol A is reused for both the attention output in Eq. (4) and the fused output after the residual MLP projection in Eq. (5). Using a distinct symbol such as F would improve readability.","section":"Eq. (5)"},{"comment":"The phrase 'FLOPs reported in this paper include request-only computations' is ambiguous with the earlier statement that FLOPs denote per-example forward-pass computation. Clarify whether per-example FLOPs are computed as (request-only FLOPs / number of candidates) plus per-candidate FLOPs, and state the candidate count used.","section":"§5.1, Footnote 1"},{"comment":"The 'Fusion MLP' is a single linear projection of concatenated vectors, not a multi-layer MLP. Consider renaming it to 'fusion projection' or describing it as a one-layer MLP.","section":"Eq. (5)"},{"comment":"The ablation baseline is WHALE itself, and all NE regressions are relative to it. Reporting the absolute NE of the reference would help readers gauge the scale of the effects.","section":"§5.4, Table 1"},{"comment":"The scaling experiments vary only one capacity axis at a time and report no error bars. It would strengthen the scaling claim to include at least one comparison against a single-paradigm baseline at the same FLOPs, especially at the largest width.","section":"§5.3, Fig. 4"}],"recommendation":"major_revision","confidential_remarks":"The central architecture is plausible and the industrial deployment is compelling, but the FLOPs-alignment issue is load-bearing for the paper's main comparative claim. Because the paper uses proprietary data and baselines from the same research ecosystem, the requested accounting clarity is especially important. This is fixable within the manuscript's scope by adding per-request FLOPs, candidate counts, and a precise amortization rule; I would not reject on the current evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should read this if you care about how industrial recommenders are actually scaling. The paper combines Wukong-style feature interaction and HSTU-style sequence modeling inside every layer, with attention-based fusion where the interaction branch queries the sequence branch. That per-layer progressive exchange is the genuinely new piece. It is incremental — Kunlun does something similar at the block level, and OneTrans unifies interaction and sequence modeling in a different way — but WHALE is the first to pair these two specific backbones, and it ships a set of systems optimizations (shared-KV attention, shared-gate SwiGLU, custom Triton kernels) that are nontrivial and well described.\n\nThe evidence is credible. The ablation table is the most convincing part: replacing attention with average pooling costs 0.23% NE, dropping the shallow-hybrid design costs 0.25%, and removing the fusion MLP or SwiGLU FFN also hurts. That tells me the fusion design is load-bearing, not decorative. The online A/B lift (+0.113% primary, with supporting metrics at +0.824% and +1.820%) is consistent with the offline direction, and a 5% QPS regression is a modest price to pay. The scaling curves show monotonic gains with sequence length, depth, and width — at least within the explored range.\n\nNow the soft spots, in proportion. The biggest one is the FLOPs accounting. Footnote 1 says reported FLOPs include request-only computations and that WHALE's K/V projections are amortized with M-FALCON across hundreds of candidates. That is fine as a serving-cost story, but it is a problem for the headline claim that WHALE beats Wukong-only and HSTU-only at \"comparable complexity.\" Wukong-only cannot amortize a sequence K/V projection because it has no sequence branch; HSTU-only can amortize but the paper does not say whether it does. Without knowing the candidate count C, and without confirming the same amortization was applied to all baselines, matching per-example FLOPs does not guarantee matching per-request serving cost. The 5% inference QPS regression actually hints that WHALE is more expensive in real serving. So the specific \"equal compute, better quality\" claim is not firmly established. The raw NE gains and the online A/B lift could still be real, but they may partly reflect more actual compute.\n\nSecond, there are no error bars on any offline NE number, and the experiments are single runs. That is common in industrial papers, but it weakens the precision of claims like \"0.23% regression.\" Third, there is no direct numerical comparison to Kunlun, OneTrans, or RankMixer, only qualitative positioning. I would have liked to see at least a table entry for a comparable unified model if one can be run on the same data.\n\nBottom line: this is a serious piece of engineering. The core architectural idea is plausible, the ablations support it, and the deployment evidence is real. The FLOPs-alignment issue should be fixed or the claim downgraded to \"better quality under similar model size\" rather than \"comparable compute.\" I would send it to peer review — an expert referee can help nail down the accounting. I would also assign it to a reading group if anyone in the group is thinking about unified ranking architectures.","headline":"Solid, deployable industrial paper: per-layer Wukong-HSTU fusion is a real engineering contribution with online gains, but the FLOPs-alignment claim is oversold unless the K/V amortization is applied symmetrically to baselines.","tokens_in":16168,"tokens_out":2477,"would_cite":true,"duration_ms":26867,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"WHALE, a layer-wise fusion of Wukong-style feature interaction and HSTU-style sequence modeling, consistently outperforms models that scale either paradigm alone, and the advantage grows as compute increases.","keywords":["recommender systems","feature interaction","sequence modeling","Wukong","HSTU","unified architecture","scalability","industrial deployment"],"falsifier":"Recompute the main quality-complexity comparison with a stricter cost model: charge every candidate its full share of the history-side K/V computation (no request-level amortization) and recompute FLOPs for all three architectures under identical accounting, or instead match the models on parameter count and layer count with amortization disabled. If the WHALE advantage disappears or reverses under that accounting, the central complexity-alignment claim is refuted; if it persists, the claim survives. A second, complementary check would be an offline experiment that evaluates WHALE on requests","tokens_in":15155,"feed_emoji":"🐋","tokens_out":11520,"duration_ms":103592,"temperature":0.7,"pith_summary":"The paper sets out to show that the next stage of scaling industrial recommenders can come from unifying two complementary architectures rather than enlarging either one. WHALE stacks layers in which a Wukong module (a factorization-machine backbone for high-order feature crosses) models user, item, and context features, an HSTU module (a sequence-transduction backbone) models the ordered behavior history, and an attention module lets each feature cross query the history and retrieve the behavior evidence most relevant to the current candidate and context. Repeating this exchange at every layer keeps both branches active and lets feature combinations refine against the history progressively. On a large industrial short-form video dataset, WHALE reports better normalized entropy (NE, lower is better) than models that scale either branch alone at matched compute, with its advantage widening as compute grows, plus monotonic gains from longer histories, more layers, and wider embeddings; online A/B tests show positive gains with a 5% throughput regression. The authors read these results as evidence that progressive cross-branch exchange is a practical, deployable scaling principle.","feed_headline":"Unifying two recommendation backbones beats scaling either one","feed_subtitle":"WHALE fuses feature crosses with long user histories at every layer, and the advantage grows as compute grows.","key_machinery":"The load-bearing object is the WHALE layer: a recursive unit containing a Wukong module (a stacked factorization-machine block plus linear compression that models high-order non-sequence feature crosses), an HSTU module (a sequential attention block with relative position-and-time biases that models long behavior histories), and an attention-based fusion module. In the fusion module, the Wukong output is normalized and projected into queries; the HSTU output is normalized and projected into shared keys and values; a softmax attention produces one history-derived vector per interaction representation; a fusion MLP concatenates that vector with the Wukong output and adds a residual; and a pre-","core_discovery":"WHALE's discovery is that keeping a feature-interaction branch and a sequence branch simultaneously active, and fusing them with attention at every layer, yields a scalable recommendation architecture whose quality improves with model size in ways neither branch achieves alone. Each WHALE layer computes a Wukong representation of non-sequence user, item, and context interactions, an HSTU representation of the ordered behavior history, and a fused representation obtained by letting every Wukong-derived vector query the HSTU-derived history, with the Wukong side providing queries and the HSTU side providing keys and values, followed by a fusion MLP and a SwiGLU feed-forward network. The fused","pith_inferences":["A testable extension the paper does not run: check whether the fusion-attention gains concentrate on candidates with sparse static features or long-tail items, where selective retrieval from history should matter most; if gains are uniform, an alternative explanation such as raw capacity increase would be more plausible.","WHALE's efficiency story depends on request-level reuse of the history computation across many candidates, which is characteristic of short-form video ranking. The same layer-wise fusion would be much more expensive on surfaces with one or few candidates per request, so the design's deployability may not transfer directly to those settings.","The shared key/value and shared-gate choices are reported as quality-neutral in this setting; a reader could probe whether the tie-breaking remains neutral with wider embeddings or with histories containing heterogeneous action types, since tying removes degrees of freedom that could matter at higher capacity.","By framing the open question as 'which fusion granularity pays off,' the architecture suggests a spectrum from one-shot summaries to per-layer attention; a natural next experiment is a controlled study of fusion frequency (e.g., every other layer) to find the minimum exchange required for the observed gains."],"forward_implications":["If WHALE is right, the scaling playbook for large recommenders shifts: adding a second, complementary paradigm and exchanging across layers buys more quality at the same FLOPs than enlarging either paradigm alone, and the benefit grows with compute.","Longer user histories, more layers, and wider embeddings all keep paying off across the tested ranges (15k history steps, 8 layers, width 512), suggesting WHALE can absorb continued capacity increases without saturating at the evaluated scales.","The ablation results imply that the attention mechanism's selective retrieval is doing the work: replacing it with average pooling over the history costs 0.23% NE, and compressing history into a shallow summary before feature interaction costs 0.25% NE, so fine-grained progressive fusion is worth a substantial fraction of the reported gains.","The deployment results imply the architectural benefit can survive serving constraints: online, the primary metric rises 0.113% and two supporting metrics rise 0.824% and 1.820%, at the price of a 5% inference QPS regression that the authors judged acceptable under the serving budget.","Because the advantage over single-paradigm baselines widens as FLOPs grow, the unified design is a scaling property rather than a one-time quality bump; this predicts WHALE will continue to pull ahead as models are scaled further."],"fun_headline_variants":["WHALE fuses feature crosses with user history at every layer","Pairing two backbones beats scaling one for recommendations","Layer-wise attention fusion of Wukong and HSTU scales further","WHALE unifies non-sequence and behavior-sequence modeling","Attention fusion of feature crosses and user history per layer"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Everything hinges on whether the FLOPs used to compare the three models count each architecture's work fairly. The paper's own footnote says the reported FLOPs include request-only computation and that WHALE computes its K/V projections once per request and amortizes them across many candidates; if that accounting favors WHALE relative to the baselines, the claim of winning at equal complexity is not settled, though the raw quality and online gains could still be real.","fun_headline_variants_meta":{"raw":{"variants":["WHALE fuses feature crosses with user history at every layer","Pairing two backbones beats scaling one for recommendations","Layer-wise attention fusion of Wukong and HSTU scales further","WHALE unifies non-sequence and behavior-sequence modeling","Attention fusion of feature crosses and user history per layer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000571,"raw_usage":{"total_tokens":2571,"prompt_tokens":813,"completion_tokens":1758,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":1676}},"tokens_in":557,"tokens_out":1758,"duration_ms":14412,"temperature":1.0,"reasoning_tokens":1676,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T04:12:11.428797+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the main quality-complexity comparison with a stricter cost model: charge every candidate its full share of the history-side K/V computation (no request-level amortization) and recompute FLOPs for all three architectures under identical accounting, or instead match the models on parameter count and layer count with amortization disabled. If the WHALE advantage disappears or reverses under that accounting, the central complexity-alignment claim is refuted; if it persists, the claim survives. A second, complementary check would be an offline experiment that evaluates WHALE on requests","supporting_citations":[],"review_version":2}