{"id":"1185f44e-065b-473b-85df-d107a5f6b339","arxiv_id":"2608.11061","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"For sampled-softmax recommender training under a fixed memory budget B=n*k, the paper recommends the largest feasible batch size and fewest negatives, n = k = sqrt(B).","lead":"This paper asks whether a fixed memory budget for training a recommender should buy more examples per batch or more sampled negative items. The authors argue, from a variance analysis, that larger batches win and recommend setting batch size equal to the square root of the budget.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.4 minimizes a variance bound outside the regime where the bound is derived (K >> k >> 1), so the k ~ 1 optimum and the n ~ B, k ~ 1 rule are not supported.","rationale":"I agree with the reader's weakest assumption. The central claim is that variance is minimized at k = O(1), which yields the n ~ B, k ~ 1 rule. But the variance formula is derived and validated only for large k (K >> k >> 1; Figure 6 starts at k = 16). Minimizing that formula at k = O(1) is extrapolation outside the derivation. The proof of Theorem 3.4 also silently drops the k >= n constraint when giving n* ~ B, k* ~ 1; the constrained statement n = k = sqrt(B) is just the boundary of the feasible set. Since this theorem is the only theoretical support for the abstract's 'fastest convergence' claim, the paper's central contribution is not established. The empirical evidence is not a substitute: Table 2 shows better AUL at larger n, but final NDCG is mixed, and in MovieLens-20M SGD the recommended (128,128) configuration has the worst NDCG@10 of the three. The practical n = k = sqrt(B) heuristic may still be reasonable, but it is not derived and not consistently validated. Hence the reader's REJECT verdict is appropriate; no adjustment is needed.","tokens_in":18098,"tokens_out":12092,"duration_ms":103348,"concrete_test":"Use the paper's synthetic protocol (Section 4.2) with B = 2^14, K = 50000, sigma_l = 1.0, drawing logits per assumptions A1-A3. For k in {1, 2, 4, 8, 16, 32, 64}, set n = B/k and compute the empirical variance of the corrected sampled-softmax gradient g_{n,k} over many independently sampled batches at a fixed parameter point, comparing with the closed-form bound from Theorem 3.3. If the variance is not monotonically decreasing as k goes below 16, or if the bound deviates from Monte Carlo variance by a large factor, the k ~ 1 minimizer is an artifact of the large-k approximation. As an analytic companion, compute the exact moments of the k = 1 case (inverse of a single lognormal) and compare them with the k >> 1 asymptotic formula in Appendix A.2.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the minimization of the variance bound in Theorem 3.4. The bound used there (Theorem 3.3, Appendix A.2, Lemma A.1) is proved under the explicit regime K >> k >> 1: the Fenton-Wilkinson log-normal approximation is applied to S = sum_{i in S_neg} h_i, the expansion (k + e^{sigma_l^2} - 1)^2/k^2 = 1 + O(1/k) is used, and Figure 6 validates Fenton-Wilkinson only for k >= 16. The unconstrained minimizer is k* = sqrt(C_hatp / C_y) = O(1), i.e., exactly the point where the k >> 1 expansion and the log-normal-sum approximation are invalid. At k = 1, the negative sum is a single log-normal and the O(1/k) corrections are O(1), so the claimed monotone dependence of the second variance term on k can reverse. Thus Theorem 3.4 does not establish that variance is minimized at k ~ 1, and the abstract's n ~ B, k ~ 1 allocation is unsupported. The practical rule n = k = sqrt(B) is only the boundary of the feasible region k >= n, not a consequence of the variance minimization. The final NDCG results in Table 2 are also mixed (e.g., MovieLens-20M SGD degrades from 0.0176 at (32,512) to 0.0093 at (128,128)), so the empirical results do not independently rescue the rule.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript studies how to split a fixed memory budget B = n k between batch size n and number of sampled negatives k in sampled-softmax training of sequential recommender systems. Section 3 develops an unbiased-correction estimator (Theorem 3.2), a variance bound for the last-layer gradient (Theorem 3.3), and an allocation rule (Theorem 3.4) claiming that n ~ B, k ~ 1, with the practical rule n = k = sqrt(B). Validation is carried out on synthetic data and on four real datasets (MovieLens-1M, Gowalla, Netflix, MovieLens-20M) using SASRec with SGD and Adam. The paper also provides public reproducibility materials.","tokens_in":18404,"tokens_out":16180,"duration_ms":136783,"significance":"The question is well motivated and practically relevant, and the paper identifies a real memory bottleneck in large-catalog recommender training. The manuscript has clear strengths: a controlled synthetic setup, real-data benchmarks across four datasets, reproducibility materials, and an explicit, actionable configuration rule. However, the central theoretical result does not support the headline allocation: the variance bound is minimized outside the regime in which it is derived, the constants in the bound contradict the minimizer computation, and the unbiasedness theorem is only approximate. The empirical evidence is also mixed, e.g., MovieLens-20M SGD NDCG@10 degrades under the recommended configuration. As it stands, the manuscript does not establish its central claim.","major_comments":[{"comment":"Theorem 3.2 is stated as an exact equality E p''_j = E p_j, but the proof only matches first moments approximately: the correction factors use (alpha + k)/(alpha + K), whereas the ratio E[p_i]/E[p'_i] equals (alpha + k - 1)/(alpha + K - 1) for the target class and K/k times this for negative classes. The proof explicitly neglects the -1 terms. This approximation is not controlled at the k ~ 1 allocations recommended later, where alpha + k need not be much larger than 1. Additionally, the theorem assumes alpha_j is the ratio of true expectations, while the practical estimator uses \\hat{alpha}_j computed from current logits and sampled negatives; the randomness of \\hat{alpha}_j is not included in the unbiasedness analysis. Since Theorem 3.1 requires a conditionally unbiased gradient estimator, the variance-to-convergence link is not established.","section":"Sec. 3.3, Appendix A.1"},{"comment":"The variance bound in Theorem 3.3 is derived under K >> k >> 1: the proof states 'we assume K >> k >> 1, and, thus, k - 1 ~ k', uses the Fenton-Wilkinson log-normal approximation for the sum S, and expands (1 + (exp(sigma_l^2) - 1)/k)^2 = 1 + O(1/k). Figure 6 validates the approximation only for k >= 16. Theorem 3.4 then minimizes this bound at k* = O(1), exactly outside the derivation regime. At k = 1 the omitted O(1/k) corrections are O(1), so the claimed monotone decrease of the class-sampling variance term in k is not controlled and can reverse. This is the load-bearing step for the n ~ B, k ~ 1 conclusion. The proof also replaces the variance of the batch average by 1/n times a single-example variance, which requires independent class sampling across examples, whereas the setup of Section 3.2 and the in-batch negative implementation use a shared set S; this dependence is not addressed.","section":"Sec. 3.4, Appendix A.2"},{"comment":"The theorem statements and the proof use inconsistent variance formulas. Theorem 3.3 states Var(g_n,k) <= C_y/n + C_hatp/k with C_y ~ O(1/(B K)) and C_hatp ~ O(1/K), while Appendix A.2 concludes Var(g_n,k) <= C_y/(K n) + C_hatp/(B K k). These are not the same up to rescaling, and they lead to different optimizers. Even using the appendix form, the proof of Theorem 3.4 obtains n* = B sqrt(C_y/C_hatp); substituting the stated magnitudes gives n* = O(sqrt(B)) and k* = O(sqrt(B)), not n* ~ B, k* ~ 1. To obtain n* ~ B one needs C_y/C_hatp = O(1), which contradicts the orders declared in Theorem 3.3. The minimizer computation is therefore internally inconsistent.","section":"Sec. 3.4 vs. Appendix A.2"},{"comment":"The practical rule n = k = sqrt(B) is the boundary of the feasibility region, not a consequence of the variance minimization. With B = n k and distinct positive classes, the constraint k >= n forces n <= sqrt(B) and k >= sqrt(B); maximizing n under this constraint gives n = k = sqrt(B). Thus, once a monotone decrease of the variance proxy in n is assumed, the rule is forced and provides no independent support for the theory. Table 2 only tests (32,512), (64,256), and (128,128) for B = 16384, i.e., the high-k side of the feasible region, and no real-data configuration with n > k is evaluated. Section 4.5 states that 'in further experiments we consider k <= n only', which is contradicted by the two rows with k > n in Table 2.","section":"Sec. 3.4, Sec. 4.5, Table 2"},{"comment":"The empirical results are mixed and do not consistently support the headline claim. In MovieLens-1M with SGD, AUL increases from 46.5 at (32,512) to 61.8 at (128,128), so the recommended configuration has worse convergence speed by the paper's own metric, since lower AUL is better. In MovieLens-20M with SGD, AUL improves from 233.1 to 61.5 while NDCG@10 drops from 0.0176 to 0.0093, contradicting the abstract's claim of 'better final recommendation quality'. In Gowalla with SGD, the best NDCG@10 is attained at (64,256), not at (128,128). The abstract's claim that the suggested configuration achieves faster convergence and better final quality is therefore not supported by the reported numbers.","section":"Table 2"}],"minor_comments":[{"comment":"The abstract contains duplicated text 'controlled synthetic and synthetic and four real sequential recommendation benchmarks' and a subject-verb disagreement in 'The suggested configuration achieve'.","section":"Abstract"},{"comment":"The name 'Felton-Wilkinson' is a misspelling of 'Fenton-Wilkinson', and the approximation is attributed to Ref. [12], a normal limit theorem by Marlow, rather than to the standard moment-matching lognormal approximation reference.","section":"Sec. 3.4"},{"comment":"The 'variance of the gradient' is not defined for a matrix-valued random variable; the paper should specify the norm (e.g., Frobenius norm) and state whether the bound is per coordinate or summed over all coordinates.","section":"Sec. 3.4, Theorem 3.3"},{"comment":"The convergence curves are plotted against the number of processed batches, but with different batch sizes each batch corresponds to a different amount of training data; the AUL definition should clarify whether Delta_t is wall-clock time, a step count, or an epoch measure, since this affects the comparability of the reported AUL values.","section":"Sec. 4.1, Figures 4-5"},{"comment":"The word 'Conseqence' in the consequence following Theorem 3.4 should be corrected to 'Consequence'.","section":"Sec. 3.4"}],"recommendation":"reject","confidential_remarks":"The paper addresses a practical and timely question and includes reproducibility materials, but the theoretical core is internally inconsistent and the empirical results do not rescue the central claim. In particular, the unbiasedness theorem is approximate, the variance bound is minimized outside its validity regime, and the constants used in the minimizer computation contradict the stated orders of magnitude. I would not be able to recommend acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper has the right question and a clean empirical trend, but the theory doesn't get to the headline rule, and the claims about final quality outrun the data. I'd send it to review with the expectation of major revision, not desk-reject it.\n\nWhat's new: the authors frame sampled-softmax training as a fixed budget B = n*k and decompose gradient variance into a mini-batch term and a class-sampling term. That decomposition is a useful way to think about the trade-off. The synthetic experiments in Figure 2 show AUL dropping as n/k increases across several B values, and the real-data results do show faster convergence (lower AUL) for the n=128,k=128 configuration in most cases. They also cite the relevant prior work, including Zhelnin et al. and Klenitskiy & Vasilev, and they don't oversell novelty relative to those empirical observations.\n\nThe problems are in the theory and in the final-quality claim. Theorem 3.2's unbiasedness proof only matches first moments approximately and drops -1 terms; that's not enough to establish unbiasedness. Theorem 3.3 derives a variance bound under K >> k >> 1, using the Fenton-Wilkinson approximation and a 1+O(1/k) expansion, and Figure 6 only validates Fenton-Wilkinson for k >= 16. Then Theorem 3.4 minimizes that bound at k* = O(1), i.e., exactly outside the regime where the bound is valid. So the n ~ B, k ~ 1 allocation is not supported. The n = k = sqrt(B) rule used in practice is just the boundary of the k >= n feasibility constraint; it does not follow from the variance minimization.\n\nThe experiments are mixed on final NDCG@10. On MovieLens-20M with SGD, (128,128) gives 0.0093 vs 0.0176 for (32,512), so larger n is substantially worse. The abstract's \"better final recommendation quality\" is too strong. The convergence speed evidence is real, but that's a weaker claim than the abstract makes. The code link wasn't accessible in the review material, so I couldn't verify the reproducibility claims.\n\nIn sum: the variance decomposition is a fair formalization, and the observation that larger batches speed up convergence under fixed B is plausible and worth knowing. But the paper's own theorem does not prove the rule, and the rule is not independently validated for final quality. A serious referee could salvage this by re-deriving the variance bound at small k or reframing the contribution as an empirical study. I would not cite it in its current form.","headline":"Good question, clean empirical trend, but the theoretical derivation doesn't reach the headline rule and the final-quality claims overreach.","tokens_in":18953,"tokens_out":2755,"would_cite":false,"duration_ms":25119,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Under a fixed memory budget, sampled-softmax recommender training converges fastest when nearly all memory goes to batch size and almost none to negative sampling.","keywords":["Sequential Recommendation","Cross-Entropy Loss Optimization","Negative Sampling","Sampled Softmax","In-Batch Negatives","Training Efficiency","Memory-Efficient Training","Gradient Variance"],"falsifier":"Measure the actual last-layer stochastic-gradient variance in the paper's synthetic logistic-regression setup at a fixed budget $B$ while sweeping $k$ from 1 up to the balanced point and beyond. If the variance minimum lands at $k$ much larger than 1, or if the variance does not follow the bound $C_y/n + C_{\\hat p}/k$, the allocation rule fails; the paper's own Figure 6 already limits the approximation's validation to $k \\ge 16$.","tokens_in":17837,"feed_emoji":"💾","tokens_out":15445,"duration_ms":115303,"temperature":0.7,"pith_summary":"This paper tries to settle a practical trade-off in large-catalog recommender training: if memory is fixed, should you use the budget to enlarge the batch or to sample more negative items for the softmax? It argues from stochastic-gradient variance that the answer is to spend almost everything on batch size, taking $n \\sim B$ and $k \\sim 1$, with $n = k = \\sqrt{B}$ as the feasible balanced point when every target class must be in the sampled set. If correct, the rule replaces heuristic grid search with a one-line configuration, which matters because the softmax layer is the dominant memory cost when catalogs reach hundreds of thousands of items. The paper supports the rule with a variance bound for a last-layer logistic-regression model and with experiments on MovieLens-1M, Gowalla, Netflix, and MovieLens-20M showing faster convergence and competitive or better final ranking quality for batch-heavy allocations.","feed_headline":"Bigger batches, fewer negatives: fastest recommender training","feed_subtitle":"In sampled-softmax training, memory buys more examples than negatives; four benchmarks agree.","key_machinery":"The carrying mechanism is a two-source variance decomposition for the corrected sampled-softmax gradient of the last layer. The proof splits the gradient into a term from sampled classes and a term from sampled examples, applies the Fenton-Wilkinson approximation (replacing the sum of log-normal exponentiated logits by one log-normal with the same mean and variance) plus the delta method, and arrives at $\\mathrm{Var}(g_{n,k}) \\le C_y/n + C_{\\hat p}/k + \\mathrm{const}$. Minimizing this bound under the product constraint $B = n k$ produces the allocation rule $n \\sim B$, $k \\sim 1$, and the balanced variant $n = k = \\sqrt{B}$ under the feasibility constraint $k \\ge n$.","core_discovery":"The paper's central claim is that the two sources of noise in sampled-softmax training—mini-batch sampling and class sampling—should not split a fixed memory budget equally. For a multiclass logistic-regression model of the final layer, under assumptions that logits are nearly Gaussian with bounded variance and that the positive class is rare among exponentiated logits, the corrected gradient variance satisfies $\\mathrm{Var}(g_{n,k}) \\le C_y/n + C_{\\hat p}/k + \\mathrm{const}$ over the constraint $n k = B$. Minimizing this bound yields $n^* \\sim B$ and $k^* \\sim 1$, and with the added requirement that the sampled class set contain every target class, $n^* = k^* = \\sqrt{B}$. The paper concludes that memory should go to batch size first, and reports that batch-heavy configurations converge faster and often reach better final NDCG@10 than negative-heavy configurations at equal memory on synthetic data and on four sequential recommendation benchmarks.","pith_inferences":["A testable extension the paper does not run is to measure end-to-end gradient variance in a full transformer recommender at $k = 1, 2, 4$; if earlier layers contribute substantial noise, the optimal allocation could shift away from extreme batch-heavy settings.","The $\\sqrt{B}$ rule depends on the feasibility constraint $k \\ge n$ that assumes each target class in a batch is distinct and must be included in the sampled set; relaxing that assumption, for example with overlapping in-batch targets, could move the optimum toward smaller $n$.","Since the variance bound contains an additive constant, the advice to maximize $n$ is asymptotic in the budget $B$; at small budgets the constant may dominate and negative-heavy configurations could win, a crossover the synthetic experiments could quantify directly.","The same variance decomposition could be applied to allocate memory between batch size and sampled tokens in language-model softmax heads, where the logit tensor is likewise the dominant memory cost."],"forward_implications":["With a fixed memory budget $B = n k$, batch-heavy allocations ($n \\sim B$, $k \\sim 1$) should converge faster than negative-heavy ones at every stage of training, not merely at the end.","The feasible balanced configuration $n = k = \\sqrt{B}$, used when each target must appear in the sampled class set, is the recommended default and should beat imbalanced alternatives at equal memory.","Because the variance structure transfers to momentum and Adam-type methods, the allocation rule applies beyond vanilla SGD.","The unbiased correction of Theorem 3.2 did not measurably change optimization dynamics, so the observed convergence gain is attributed to the $(n, k)$ allocation itself, not to the loss correction.","The same $1/n$ plus $1/k$ variance structure suggests the rule extends to other large-output softmax training settings, a broader principle the paper explicitly claims."],"supporting_citations":[{"why":"Supplies the SGD convergence bound showing that suboptimality scales linearly with gradient variance, which motivates minimizing variance.","marker":"[3]"},{"why":"Quantifies that cross-entropy logits and activations dominate memory in large-catalog sequential recommenders, justifying the $B = n k$ budget model.","marker":"[13]"},{"why":"Empirically observes the batch-size/negatives trade-off during sequential recommender training that this paper aims to explain theoretically.","marker":"[27]"},{"why":"Provides the Fenton-Wilkinson approximation used to turn the sum of log-normal exponentiated logits into a tractable variance expression.","marker":"[12]"},{"why":"Defines the SASRec architecture used for the real-data experiments and for the last-layer abstraction.","marker":"[7]"},{"why":"Shows that the loss and negative-sampling choice, not architecture, drives much of the quality difference in sequential recommendation, framing the practical relevance of $k$.","marker":"[11]"},{"why":"Introduces the logQ correction for sampled-softmax bias that the paper's Theorem 3.2 builds on and then tests as the 'Unbiased' objective.","marker":"[26]"}],"fun_headline_variants":["For fixed memory, pick batches over negatives","Memory rule: batch size beats negative count","Max out batch size, shrink negatives for faster training","In sampled softmax, spend memory on batch size","Recommender training: batch size, not negatives"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole rule rests on the variance estimate staying accurate at very small numbers of negatives, even though the derivation assumes many negatives and the numerical check of the approximation starts at 16.","fun_headline_variants_meta":{"raw":{"variants":["For fixed memory, pick batches over negatives","Memory rule: batch size beats negative count","Max out batch size, shrink negatives for faster training","In sampled softmax, spend memory on batch size","Recommender training: batch size, not negatives"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001486,"raw_usage":{"total_tokens":5998,"prompt_tokens":1008,"completion_tokens":4990,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":4919}},"tokens_in":624,"tokens_out":4990,"duration_ms":33203,"temperature":1.0,"reasoning_tokens":4919,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:07:26.432822+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the actual last-layer stochastic-gradient variance in the paper's synthetic logistic-regression setup at a fixed budget $B$ while sweeping $k$ from 1 up to the balanced point and beyond. If the variance minimum lands at $k$ much larger than 1, or if the variance does not follow the bound $C_y/n + C_{\\hat p}/k$, the allocation rule fails; the paper's own Figure 6 already limits the approximation's validation to $k \\ge 16$.","supporting_citations":[{"cited_title":"Curtis, and Jorge Nocedal","cited_arxiv_id":null,"evidence_quote":"Supplies the SGD convergence bound showing that suboptimality scales linearly with gradient variance, which motivates minimizing variance."},{"cited_title":"Scalable Cross-Entropy Loss for Sequential Recommendations with Large Item Catalogs","cited_arxiv_id":"2409.18721","evidence_quote":"Quantifies that cross-entropy logits and activations dominate memory in large-catalog sequential recommenders, justifying the $B = n k$ budget model."},{"cited_title":"Faster and Memory-Efficient Training of Sequential Recommendation Models for Large Catalogs","cited_arxiv_id":"2509.09682","evidence_quote":"Empirically observes the batch-size/negatives trade-off during sequential recommender training that this paper aims to explain theoretically."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Fenton-Wilkinson approximation used to turn the sum of log-normal exponentiated logits into a tractable variance expression."}],"review_version":1}