{"id":"a62d13fc-61be-4e16-aa84-d4218180b272","arxiv_id":"2412.21124","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Adaptive batch size schedules based on the norm test can run with FSDP-distributed Llama-style pretraining, but the reported runs do not consistently beat constant batch sizes and the convergence theorem does not cover the implemented test.","lead":"This paper applies an old norm-test idea to choose batch sizes during language model pretraining and implements it with PyTorch FSDP for distributed training. On short test runs, the method sometimes beats fixed large batches, but a fixed medium batch is better on two of three models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Own experiments contradict the abstract's central claim: on TinyLlama and OpenLlama, constant batch size 4096 reaches lower validation loss than every adaptive schedule.","rationale":"The reader's REJECT verdict is well supported, but the most load-bearing concern is not the one identified as the weakest assumption. The reader focused on the gap between the exact coordinate-wise variance test assumed by Theorem 1 and the approximate non-coordinate-wise test actually implemented. That gap is real and serious: Remark B.1 concedes the exact test is not implemented, and Algorithm 1 uses worker-gradient differences rather than per-sample gradients, so the theorem's hypotheses are not verified by the algorithm. But even if the theory were fully repaired, the paper's central empirical claim would still fail on its own reported numbers. The abstract promises that adaptive schedules outperform constant batch sizes in Llama-family pretraining, and Tables 2 and 3 show the opposite for both larger models: constant batch size 4096 attains the lowest validation loss in each table. This is an internal inconsistency, not a matter of interpretation or of a contested baseline choice. The adaptive schedules do narrow the generalization gap in the sense that, at a large average batch size near 7312-8000, they achieve validation loss closer to the small-batch constant-4096 result than to the large-batch constant-8192 result. That is a weaker and more defensible claim, but it is not the claim in the abstract. The paper could be revised to claim that adaptive schedules reduce the generalization gap while preserving large-batch efficiency, with the caveat that constant 4096 remains better in final validation loss on the two larger testbeds. As written, the central claim is contradicted by the paper's own evidence. I therefore agree with the reader's REJECT verdict, though my primary reason is the empirical contradiction rather than the theory-implementation mismatch. The public implementation and reproducible setup are credit where due, but they do not change what the tables show.","tokens_in":27057,"tokens_out":3114,"duration_ms":34177,"concrete_test":"Recompute a matched comparison from the saved checkpoints or logged validation curves in Tables 2 and 3: evaluate final validation loss at exactly 2,000,000 training sequences for constant batch 4096 and each adaptive schedule, and also at matched wall-clock times. No new training is required; the tables already show lower validation loss for constant 4096 at the same sample count. If the checkpoints/logs confirm this, the abstract's \"outperform\" claim is false as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract states that the proposed schedules \"outperform constant batch sizes and heuristic batch size warmup schedules\" for Llama-family pretraining. The paper's own tables refute this on both larger testbeds. In Table 2 (TinyLlama 1.1B), the best adaptive result is eta=0.085 with validation loss 4.256, while constant batch size 4096 achieves 3.817 and constant 8192 achieves 4.893. In Table 3 (OpenLlama 3B), the best adaptive result is eta=0.15 with validation loss 4.554, while constant 4096 achieves 3.956. The text in Section 5.3 itself concedes: \"While using a constant batch size 4096 achieves an even lower validation loss.\" This is not a disagreement with an external consensus; it is an internal contradiction in the reported numbers. Timing does not rescue the claim: for TinyLlama, constant 4096 costs 34.48h versus 32.83h for the best adaptive run (about 5% more time) while improving validation loss by roughly 0.44; for OpenLlama, constant 4096 costs 20.75h versus 19.59h (about 6% more time) while improving validation loss by roughly 0.60. All runs consume the same 2 million training sequences, so on a per-token basis constant 4096 dominates. The only testbed where an adaptive schedule beats a constant baseline is MicroLlama 300M, so the general claim about the Llama 2 family is unsupported. A separate theory gap also exists: Theorem 1 assumes an exact coordinate-wise variance norm test, while Algorithm 1 implements the non-coordinate-wise approximate DDP-Norm/FSDP-Norm test, and Appendix Remark B.1 admits the exact test is \"not implemented in practice.\" However, the empirical contradiction is sufficient on its own to undermine the central contribution.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes adaptive batch size schedules for distributed training of language models, based on a norm test that grows the global batch when the approximate gradient variance is too large relative to the batch gradient norm. The authors present DDP-Norm and FSDP-Norm implementations and a convergence theorem for Adam under a coordinate-wise exact variance norm test. Experiments are reported on MicroLlama 300M, TinyLlama 1.1B, and OpenLlama 3B pretrained on C4, with the claim that the adaptive schedules outperform constant batch sizes and heuristic batch-size warmup schedules for Llama-family models.","tokens_in":27461,"tokens_out":8468,"duration_ms":81245,"significance":"If the empirical and theoretical claims were correct, this would be a useful contribution: a theoretically motivated, practically implemented method for adjusting batch sizes during large-scale distributed LLM pretraining, with an open-source FSDP implementation and reproducible experimental configuration. The paper also gives credit for shipping code and for a seriousness of purpose in attempting to connect adaptive sampling theory with distributed training practice. However, the central empirical claim is contradicted by the paper's own tables on the two larger testbeds, and the convergence theorem does not apply to the algorithm actually run. The strengths are real but are outweighed by the failure of the two headline claims.","major_comments":[{"comment":"The abstract's claim that the proposed approaches \"outperform constant batch sizes\" in the pretraining of Llama-family models is directly contradicted by the paper's own results. On TinyLlama 1.1B (Table 2), the best adaptive run (eta=0.085) reaches validation loss 4.256, while the constant batch size 4096 baseline reaches 3.817. On OpenLlama 3B (Table 3), the best adaptive run (eta=0.15) reaches 4.554, while constant 4096 reaches 3.956. On both larger models, every adaptive schedule is worse than the constant 4096 baseline, which is an internal inconsistency with the stated contribution.","section":"§5.2 Table 2; §5.3 Table 3; Abstract"},{"comment":"The wall-clock argument does not rescue the empirical claim. The authors concede in §5.3 that \"using a constant batch size 4096 achieves an even lower validation loss,\" but the reported time savings are small: about 5% on TinyLlama (34.48h vs 32.83h) and about 6% on OpenLlama (20.75h vs 19.59h), while the validation-loss differences are large (roughly 0.44 and 0.60). Because all runs consume the same 2,000,000 training sequences, the constant batch size 4096 baseline dominates on a per-token basis, so the efficiency framing does not compensate for the worse final validation loss.","section":"§5.3"},{"comment":"There is a load-bearing mismatch between the convergence theorem and the implemented algorithm. Theorem 1 assumes that the coordinate-wise exact variance norm test, Ek[(∂iℒBk(wk)−∂iℒ(wk))2] ≤ η2(∂iℒ(wk))2, is satisfied at every iteration. Algorithm 1, however, implements the aggregate, non-coordinate-wise approximate test in Eq. (5) (DDP-Norm/FSDP-Norm) and does not verify the exact condition for the next batch. Appendix B Remark B.1 explicitly concedes that \"the exact variance test is not implemented in practice but its approximate version instead.\" Consequently, the proof does not establish convergence for the algorithm whose empirical behavior is reported.","section":"§4, Theorem 1; Algorithm 1; Appendix B, Remark B.1"},{"comment":"The stated convergence guarantee is vacuous as written. Theorem 1 bounds ∑k=1K E[‖∇ℒ(wk)‖] by ~O(K). A bound of O(K) on a sum of K terms holds trivially for any algorithm with bounded per-iteration gradient norms and does not imply convergence to a stationary point. The explicit bound in Theorem B.1 is also at least linear in K: the constant c2 contains the term 2c1∑i (log(1/√β2 v0,i) − K log β2), which grows linearly in K since log β2 < 0. Thus the average gradient norm need not decay, so the theorem does not deliver the advertised convergence guarantee.","section":"§4, Theorem 1; Appendix B, Theorem B.1 and definition of c2"}],"minor_comments":[{"comment":"The notation Vari∈B(∇ℓi(w)) is used as a vector while the displayed expression mixes norms and scalar quantities; please define the per-coordinate variance vector explicitly and distinguish it from its L1 norm.","section":"§3.1, Eq. (2)"},{"comment":"The text contains the typo \"prupose\" in the paragraph on the effect of η; it should read \"purpose.\"","section":"§5.4"},{"comment":"The batch-size panels label the horizontal axis as \"sample ×105\" or \"sample ×106\"; the unit should be \"samples\" and the exponent formatting made consistent across panels.","section":"Figures 2 and 3"},{"comment":"The experiments use AdamW with decoupled weight decay, whereas Theorem 1 concerns Adam without weight decay; the paper acknowledges this in the text, but the main-text discussion would be clearer if the limitation were stated immediately after the theorem rather than in the later discussion.","section":"§4"}],"recommendation":"reject","confidential_remarks":"The engineering component—an FSDP-based implementation with released code—is a genuine strength, and the authors are careful to include a limitations section. However, the abstract's central empirical claim is contradicted by the paper's own Tables 2 and 3, and the convergence theorem does not apply to the implemented algorithm. These are not presentation issues; they undermine both headline contributions. A future version that repositions the paper around the implementation and the single positive result on MicroLlama, and that either fixes or removes the convergence claim, could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. First, the practical piece is real: constructing a gradient-variance estimator from per-worker minibatch differences and integrating it into PyTorch FSDP is a sensible, efficient trick, and the code is released. Second, the paper's central empirical claim—that the adaptive schedules outperform constant batch sizes for Llama-family pretraining—is contradicted by its own Tables 2 and 3. On TinyLlama 1.1B, constant batch 4096 reaches 3.817 validation loss while the best adaptive run is 4.256; on OpenLlama 3B, constant 4096 gets 3.956 and the best adaptive run 4.554. You don't need external baselines to see this; it's an internal inconsistency. Only the MicroLlama 300M experiments show an adaptive win.\n\nThe distributed variance estimator DDP-Norm (Eq. 5) and the FSDP support are the genuinely new parts. Computing the norm-test statistic via differences between per-worker minibatch gradients instead of per-sample gradients is a practical contribution. The convergence theorem for Adam under the exact coordinate-wise variance norm test is a legitimate extension of Wang et al.'s proof, and it does not smuggle in its conclusion.\n\nBut the soft spots are load-bearing. Theorem 1 assumes the exact coordinate-wise norm test at every iteration; Algorithm 1 implements the non-coordinate-wise approximate test, and Remark B.1 concedes the exact test is not used in practice. So the theory does not cover the algorithm actually run. The theorem's condition on beta1,beta2 is also not met by the experiments (beta1=0.9, beta2=0.95). The paper is honest about some of this in the body—Section 5.3 explicitly says constant 4096 achieves a lower validation loss, and the limitations section says tests are proofs of concept—but the abstract and contributions overstate. That overstatement is not cosmetic; it is the main selling point.\n\nWho this is for: practitioners interested in the distributed variance estimator and the implementation details; theory readers will find the Adam proof a useful reference but should note the gap between the exact and approximate tests. A serious editor should send this to peer review—the implementation, released code, and honest body text give referees enough to work with—but my own verdict is that as currently written it should not be accepted. It needs a major revision: either fix the empirical claims and restrict the contribution to smaller-scale settings where it works, or reframe as an implementation/tool paper and move the convergence result to an appendix where the gap can be clearly stated.","headline":"Real distributed variance estimator, but the empirical claims don't survive the paper's own tables on the larger models.","tokens_in":27990,"tokens_out":3462,"would_cite":false,"duration_ms":33431,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","90C26","90C15","68W10"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that variance-based norm tests, extended to data and model parallel training, let language model pretraining grow batch sizes on demand, matching small-batch quality at large-batch speed, with Adam still provably…","keywords":["adaptive batch size","norm test","language model pretraining","distributed data parallelism","model parallelism","Adam convergence","generalization gap","fully sharded data parallel"],"falsifier":"Train one small model twice with identical hyperparameters, once using the approximate worker-level variance statistic to choose batches and once using exact per-sample gradient variance, and compare the batch-size trajectories and validation losses; if the approximate schedule consistently selects notably different batch sizes or yields worse validation loss at matched sample counts, the claim that the practical test supports the convergence guarantee is refuted.","tokens_in":26870,"feed_emoji":"📈","tokens_out":9551,"duration_ms":87015,"temperature":0.7,"pith_summary":"The paper's central claim is that the long-standing trade-off in language model training—large batches give throughput but worse generalization, small batches generalize better but train slowly—can be resolved by an adaptive batch size schedule that grows the batch only when the current gradient estimate is provably too noisy. The authors extend the norm test of adaptive sampling to distributed settings that combine data parallelism with model parallelism, implement it in a fully sharded data parallel framework, and report that it beats constant batch sizes and hand-designed warmup schedules on decoder-only models up to three billion parameters. On the theory side, they prove that Adam, the standard optimizer for pretraining, converges under these schedules for smooth nonconvex objectives if a coordinate-wise variance norm test is satisfied. If correct, this gives practitioners a way to adapt batch sizes to training dynamics rather than following fixed recipes.","feed_headline":"Norm test grows LLM batches on demand, beating fixed sizes","feed_subtitle":"A variance test shrinks the generalization gap by growing batch size only when gradients disagree.","key_machinery":"The load-bearing object is the norm test statistic: the ratio of an estimate of the per-sample gradient variance to the squared norm of the current batch gradient, compared against a threshold $\\eta^2$. In the distributed setting this statistic is computed without per-sample gradients by measuring how much the per-worker minibatch gradients differ from the global batch gradient, a version the paper calls DDP-Norm (and FSDP-Norm under model parallelism). The proof rests on a coordinate-wise version of the exact-variance norm test, which the authors show implies the coordinate-wise expected strong growth condition that Adam's iteration-complexity analysis requires.","core_discovery":"In the paper's own terms, the discovery is that the norm test—the rule that increases the next batch size to $\\lceil \\| \\operatorname{Var}_{i \\in B_k}(\\nabla \\ell_i(w_k))\\|_1 / (\\eta^2 \\|\\nabla \\mathcal{L}_{B_k}(w_k)\\|^2) \\rceil$ when the current batch gradient is too noisy—can be made practical for distributed, model-parallel training and can carry Adam to convergence. The theoretical result, Theorem 1, proves a convergence bound of order $\\sqrt{K}$ up to logarithmic factors on the cumulative expected full-gradient norm under the coordinate-wise exact-variance norm test, which implies the coordinate-wise expected strong growth condition. The empirical result is that, at equal numbers of training samples or steps, the adaptive schedules match or improve validation loss relative to constant large batches and to heuristic stagewise warmups, while using fewer steps when the batch can grow. The authors state this as a general-purpose schedule applicable beyond language models, with a particular demonstration on up-to-3-billion-parameter Llama 2 family models.","pith_inferences":["The approximation that replaces per-sample gradient variance with between-worker minibatch variance has only $J$ samples (the number of workers); when $J$ is small the batch-size schedule will inherit high variance, so smoothing the statistic over a few iterations is a natural extension the paper does not test.","A falsifiable check of the theory-practice link is to run a small model with exact per-sample gradient variance as the test and compare the resulting batch-size trajectory with the approximate worker-level test; close trajectories would certify the approximation that the proof needs.","The same norm-test machinery could be coupled with sequence-length warmup or learning-rate schedules, since both affect the gradient-noise estimate and the descent direction; the paper mentions sequence length warmup but does not combine them.","The claimed scaling-law connection between $\\eta$ and the critical batch size is left open; a direct experiment varying $\\eta$ while measuring the final batch size plateau would test whether the plateau tracks the critical batch size."],"forward_implications":["Practitioners can start training with a small batch and let the method increase it only when gradient noise is small, obtaining small-batch validation quality at large-batch throughput.","The same schedule is compatible with data and model parallelism, so memory-constrained multi-GPU setups can pretrain billion-parameter models without hand-tuning a batch warmup recipe.","Adam's convergence under the schedule is guaranteed for smooth nonconvex objectives when the coordinate-wise norm test holds, removing the need for a global variance assumption.","With comparable wall-clock time, the adaptive schedules produce validation losses closer to those of the best small constant batch while using far fewer gradient steps.","The tuning effort for pretraining shifts to the single threshold $\\eta$, which controls how aggressively the batch grows."],"supporting_citations":[{"why":"Supplies the norm test that decides when the batch gradient is too noisy and how much to grow the next batch.","marker":"[12]"},{"why":"Shows adaptive batch size schedules work for SGD in the single-device setting that this work extends to distributed training.","marker":"[13]"},{"why":"Provides the journal version of the Big Batch SGD adaptive schedule that this work extends to distributed training.","marker":"[14]"},{"why":"Establishes adaptive batch size schedules for AdaGrad, the closest prior adaptive-gradient convergence result this work extends to Adam.","marker":"[42]"},{"why":"Defines the Adam update whose convergence is analyzed in Theorem 1.","marker":"[36]"},{"why":"Provides the Adam iteration-complexity framework and technical lemmas the proof adapts.","marker":"[76]"},{"why":"Describes the fully sharded data parallel mechanism that enables the billion-parameter experiments.","marker":"[88]"},{"why":"Defines the Llama 2 family of language models used as the experimental testbed.","marker":"[73]"}],"fun_headline_variants":["Adaptive batch sizes beat fixed and warmup schedules for LLM pretraining","Batch sizes that adapt on the fly outperform static schedules in LLM training","Norm-based adaptive batches boost LLM pretraining over fixed and warmup","Adaptive batch sizes with Adam converge for large-scale distributed training","Batch size adapts via variance test, beating constant and warmup in LLM pretraining"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof assumes the exact coordinate-wise variance norm test is satisfied at every iteration, but the deployed algorithm checks a cheaper approximate statistic built from differences between workers' minibatch gradients and never verifies that the condition actually holds for the next batch.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive batch sizes beat fixed and warmup schedules for LLM pretraining","Batch sizes that adapt on the fly outperform static schedules in LLM training","Norm-based adaptive batches boost LLM pretraining over fixed and warmup","Adaptive batch sizes with Adam converge for large-scale distributed training","Batch size adapts via variance test, beating constant and warmup in LLM pretraining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000642,"raw_usage":{"total_tokens":3009,"prompt_tokens":1058,"completion_tokens":1951,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":674,"completion_tokens_details":{"reasoning_tokens":1852}},"tokens_in":674,"tokens_out":1951,"duration_ms":15311,"temperature":1.0,"reasoning_tokens":1852,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:02:47.254726+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train one small model twice with identical hyperparameters, once using the approximate worker-level variance statistic to choose batches and once using exact per-sample gradient variance, and compare the batch-size trajectories and validation losses; if the approximate schedule consistently selects notably different batch sizes or yields worse validation loss at matched sample counts, the claim that the practical test supports the convergence guarantee is refuted.","supporting_citations":[{"cited_title":"Automated inference with adaptive batches","cited_arxiv_id":null,"evidence_quote":"Provides the journal version of the Big Batch SGD adaptive schedule that this work extends to distributed training."},{"cited_title":"Kingma and Jimmy Lei Ba","cited_arxiv_id":null,"evidence_quote":"Defines the Adam update whose convergence is analyzed in Theorem 1."},{"cited_title":"Closing the gap between the upper bound and lower bound of Adam’s iteration complexity","cited_arxiv_id":null,"evidence_quote":"Provides the Adam iteration-complexity framework and technical lemmas the proof adapts."},{"cited_title":"first-order term","cited_arxiv_id":null,"evidence_quote":"Describes the fully sharded data parallel mechanism that enables the billion-parameter experiments."}],"review_version":1}