{"id":"fe30b756-ea25-4b98-82ff-a30202db81e5","arxiv_id":"2608.11746","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Selecting or generating training data to maximize epiplexity, a loss-curve measure of structural information, improves out-of-distribution accuracy on small language models.","lead":"This paper turns a mathematical measure called epiplexity, which captures how much structure a model can learn from data, into a practical rule for choosing training data and for generating new synthetic text. Language models trained with this rule performed better on tasks outside their training distribution than models trained on standard data mixes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The scaling law in Eq. 3 is validated only in-sample; EpiSelect's selection signal is a function of its extrapolated parameters, so without out-of-sample validation the reported gains may not be attributable to epiplexity maximization.","rationale":"I agree with the reader's weakest-assumption identification. The scaling law is the linchpin of EpiSelect; without out-of-sample validation, the empirical gains cannot be confidently attributed to epiplexity maximization. The concern is not that the paper is wrong, but that the evidence is insufficient for the causal claim. The reader's conditional verdict is appropriate; this concern does not change it.","tokens_in":20486,"tokens_out":5861,"duration_ms":65298,"concrete_test":"On a completed Common Pile training run, refit the scaling law (Equation 3) using only the first 30k steps, then predict per-domain losses for steps 30k-60k. At a held-out step (e.g., 40k), rank domains by predicted dS/dn_k from Equation 4 and compare this ranking to the ranking by actual epiplexity gain computed from the observed losses via Equation 2. If the rankings differ materially (e.g., Spearman < 0.8), the selection signal is not tracking epiplexity. As a further check, run EpiSelect with an oracle that reweights by the observed gains and compare final average zero-shot accuracy; if the oracle does not beat EpiSelect, the fitted law is not the source of the improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"EpiSelect's central mechanism is Equation 4: the marginal epiplexity gain dS/dn_k is computed from the cross-domain scaling law in Equation 3. The paper validates this law only in-sample, reporting a median R2 of 0.88 across Common Pile domains in Appendix C.1. At each refit, the law is used to extrapolate beyond the observed token counts to decide which domain to sample next. An in-sample fit can be accurate while the predicted ranking of dS/dn_k is wrong, especially if the parametric form misses saturation or cross-domain effects that appear only later in training. If the ranking is wrong, the selection weights pi_k do not maximize epiplexity, and the Table 1 improvements over ADO (1.5 points at 124M, 0.6 at 1.3B) could come from a different property of the reweighting, such as upweighting domains with currently steep loss curves. The authors acknowledge in the Discussion that the estimator is a tractable proxy without guarantees, so this is not an internal inconsistency, but it is the weakest link in the causal chain from epiplexity to OOD gains.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two methods that use epiplexity as an online training signal: EpiSelect, which fits a cross-domain scaling law (Eq. 3) to per-domain loss curves and reweights domain sampling by the predicted marginal epiplexity gain (Eq. 4), and EpiGen, which trains a generator with REINFORCE using the learner's loss decrease on a buffer as reward. Experiments on LLaMA-style models trained on Common Pile report improved zero-shot LM Eval Harness accuracy over Natural and ADO baselines, and GPT-2-based synthetic data training reports improved GLUE fine-tuning scores. The paper also argues that The Pile is saturated as a data-selection benchmark because training only on PileCC outperforms SOTA selection methods.","tokens_in":20730,"tokens_out":5276,"duration_ms":52547,"significance":"If the causal chain from epiplexity to OOD generalization holds, the paper supplies a principled, computationally tractable objective for curriculum learning and synthetic data generation, with derivations in Section 3.2 and Appendix D that are correct, detailed scaling-law fitting, and external-benchmark evaluations that avoid circularity. Code links are provided. However, the empirical support at present is thin: the central correlation uses five domains, scaling-law fits are only validated in-sample, and no uncertainty is reported for the main accuracy tables. The strengths are real, but the attribution of the reported gains specifically to epiplexity maximization is not yet established.","major_comments":[{"comment":"The evidence for the central claim that epiplexity predicts OOD accuracy rests on n=5 domains with Pearson r=0.88 and p=0.05, which is marginal: with five points a single influential domain (PileCC) can drive the correlation, and the Spearman ρ=0.90 is reported without a p-value. Please add more domains, a bootstrap or permutation analysis, or otherwise quantify the stability of the correlation; without this, the 'strong predictor' claim in §3.1 is not adequately supported.","section":"§3.1, Fig. 2 (left), Appendix B"},{"comment":"EpiSelect's selection signal is entirely a function of the fitted cross-domain scaling law (Eq. 3), but the law is validated only in-sample (median R²=0.88 on observed losses). Since the method uses the law to extrapolate dS/dn_k beyond observed token counts, in-sample accuracy does not establish that the induced ranking of domains is correct; a misspecified parametric form could yield the same in-sample R² while producing wrong extrapolations, for example by missing saturation or delayed cross-domain effects. Please add an out-of-sample validation: fit on the first part of the trajectory, predict later losses and the ranking of marginal epiplexity gains, and compare against measured gains. Without this, the Table 1 improvements cannot be attributed specifically to epiplexity maximization rather than to a generic upweighting of domains with currently steep loss curves. The Discussion's caveat about proxies is appropriate, but the attribution claim in the Abstract and Section 5 goes beyond what is currently demonstrated.","section":"§3.2, Eq. (4), Appendix C.1"},{"comment":"The headline comparisons are single-run results with no variance or significance. The 124M EpiSelect gain over ADO is 1.5 points on the average, the 1.3B gain is 0.6 points, and the EpiGen gain over FrozenGen on GLUE is 1.1 points; without multiple seeds or error bars these margins may be within run-to-run noise. Please report seeds, standard deviations, or a paired significance test, or state explicitly that the comparisons are single runs; as written, the 'state-of-the-art' and 'outperforms' claims are not statistically grounded.","section":"Tables 1 and 2"}],"minor_comments":[{"comment":"The temperature τ is defined in the algorithm input as τ∈(0,1), but the text fixes τ=1; this range should be τ>0. Also, the clipped momentum mixture ωπ_k+(1−ω)̅π_k is not renormalized after clipping, so it may not remain a probability distribution; please specify the normalization step.","section":"Algorithm 1, Eq. (5)"},{"comment":"The main text calls Common Pile '8TB' but Appendix A describes a token-balanced subset totaling 29.7B tokens; please make clear in the main text that the experiments use the token-balanced construction.","section":"§3.4, Appendix A"},{"comment":"The reward is defined as the difference of sums over the buffer before and after K learner steps; the text says 'difference between the learner loss on the samples in the buffer before and after training on the current batch', which is clear, but the connection to the prequential estimator of Eq. (1) would benefit from an explicit statement that the current loss replaces the final-model loss.","section":"Eq. (6) and Algorithm 2"},{"comment":"The text says 'we sample γm,k∼Dir(p) via a Dirichlet distribution'; please clarify that it is the vector γm,: that is sampled, not each entry independently, and state how the 'grid derived by the Cartesian product' interacts with this sampling.","section":"Appendix C, scaling-law initialization"}],"recommendation":"major_revision","confidential_remarks":"The manuscript relies heavily on self-citations to Finzi et al. [20], but since it extends that work with new algorithms and independent evaluations, I do not see it as a circularity problem. The main risk is attribution: the reported gains may stem from properties of the reweighting other than epiplexity. If the authors cannot provide out-of-sample validation of the scaling law and stronger statistical support for the correlation, the claims should be softened or reframed as suggestive rather than causal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about arXiv:2608.11746. First, it is the first real attempt to turn epiplexity from theory into a working training signal: EpiSelect and EpiGen are concrete algorithms with code, clean derivations, and an honest limitations section. Second, the empirical payload is thinner than the framing suggests: the central correlation between epiplexity and downstream accuracy uses five domains with p=0.05, and the scaling law that drives selection is validated in-sample only. Neither problem is fatal, but both deserve attention before the results are taken at face value.\n\nWhat is genuinely new: the operationalization of epiplexity as an online training signal via cross-domain scaling laws (Eqs. 3-4), and the EpiGen buffer-based REINFORCE reward. The Pile saturation finding—training only on PileCC beats the SOTA selector—is a real benchmark critique and a useful service to the field. The contrast with weight norm (r=0.01) strengthens the claim that the signal is specific, and the authors are transparent about using a proxy rather than true epiplexity.\n\nSoft spots, in order of severity. (1) The scaling-law extrapolation is load-bearing: EpiSelect's sampling weights come from predicted marginal gains dS/dn_k, and an in-sample R2 of 0.88 does not guarantee the ranking is correct beyond observed token counts. The authors acknowledge this in the Discussion, so it is not an internal inconsistency, but an out-of-sample check would materially raise confidence. (2) The headline EpiGen gain is 2.7 points over the pretrained baseline, but the relevant comparison is FrozenGen, where the gain is 1.1 points. Still positive, but less dramatic. (3) The switch from Pile to Common Pile is well motivated, but the direct comparison on the original benchmark with the PileCC baseline should be in the main text. (4) No error bars or repeated seeds anywhere; all results are single runs at 124M and 1.3B.\n\nThe citation pattern is fine. Heavy use of [20] is natural for a direct follow-up, and the new experiments are independent of that prior work.\n\nWho is this for: anyone working on data selection, curriculum learning, or synthetic data generation. It deserves a serious referee, not a desk reject. I would ask for out-of-sample scaling validation, error bars, and the direct PileCC comparison. I would also cite the Pile saturation result in future work.","headline":"A credible first operationalization of epiplexity for data selection and generation, with a benchmark critique that may outlast the methods.","tokens_in":21292,"tokens_out":2504,"would_cite":true,"duration_ms":25326,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that epiplexity—a measure of the structural information in data—can serve as an online training signal that improves out-of-distribution generalization.","keywords":["epiplexity","data selection","synthetic data generation","out-of-distribution generalization","curriculum learning","scaling laws","REINFORCE","language model pretraining"],"falsifier":"Take a held-out set of domains, fit the cross-domain scaling law on early checkpoints only, then compare the predicted per-domain epiplexity gains from Equation 4 with gains measured by actually training on a small addition of tokens from each domain; if the predictions are uncorrelated with measured gains, the selection signal is not tracking epiplexity.","tokens_in":20234,"feed_emoji":"🎯","tokens_out":7169,"duration_ms":65329,"temperature":0.7,"pith_summary":"This paper tries to establish that epiplexity, a measure of the structural information a compute-bounded learner can extract from data, is a practical signal for improving out-of-distribution generalization. To show this, it introduces EpiSelect, which fits cross-domain scaling laws to per-domain loss curves and reweights training batches toward domains with the largest predicted marginal epiplexity gain, and EpiGen, which uses REINFORCE to train a generator that produces synthetic text maximizing the learner's epiplexity. In experiments, EpiSelect outperforms a natural-distribution baseline and a prior adaptive selection method on zero-shot language tasks at two model sizes, and EpiGen improves GLUE fine-tuning performance over a pretrained GPT-2 baseline. The paper also finds that a simple baseline of training only on the largest Pile domain beats prior selection methods, motivating a switch to the Common Pile dataset for evaluating data selection.","feed_headline":"Pick training data by epiplexity to boost out-of-distribution accuracy","feed_subtitle":"EpiSelect reweights domains by predicted structural gain; EpiGen synthesizes data that maximizes learner epiplexity.","key_machinery":"The load-bearing machinery is the prequential estimator of epiplexity, which approximates the structural information in data as the area between the training loss curve and the final loss, and the two online estimators built on it. For selection, the paper fits a cross-domain scaling law $\\hat{L}_m(n_1,\\dots,n_K)=\\epsilon_m+\\beta_m(\\sum_k \\gamma_{m,k} n_k)^{-\\alpha_m}$ to the observed per-domain losses and takes the derivative with respect to tokens from each domain, $\\partial\\hat{S}/\\partial n_k$, as a proxy for marginal epiplexity gain. For generation, it computes the reward as the drop in learner loss on an evaluation buffer before and after a batch, and updates the generator with the REINFORCE gradient $\\nabla_{\\theta_g} J = \\mathbb{E}[(r_t-b)\\sum_{x\\in X_t}\\nabla_{\\theta_g}\\log P_{\\theta_g}(x)]$. These estimators turn epiplexity from an a posteriori quantity into a signal that can be maximized during training.","core_discovery":"The paper's central claim is that data with higher epiplexity transfers better to unanticipated downstream tasks, so maximizing epiplexity during training should improve out-of-distribution generalization. It reports that epiplexity predicts zero-shot accuracy across five Pile domains (Pearson r=0.88), while a generic checkpoint summary like weight norm does not. It then operationalizes this claim: EpiSelect maximizes a predicted epiplexity gain computed from a fitted cross-domain scaling law, and EpiGen maximizes the learner's measured loss reduction over a buffer of generated data. In both settings, the epiplexity-maximizing procedure outperforms the baselines on zero-shot and fine-tuning tasks, which the authors take as support for the hypothesis.","pith_inferences":["An implicit testable extension is whether the same epiplexity-maximizing selection transfers to multimodal or non-text data, since the loss-curve estimator does not depend on the modality.","Because the selection signal comes entirely from fitted scaling-law parameters, a reader should not assume the proxy equals true epiplexity; the paper itself flags the absence of theoretical guarantees.","The five-domain correlation is suggestive but small; a natural stronger test would measure epiplexity across many more domains and check monotonicity rather than linear correlation.","The PileCC finding hints that diversity within a large web-crawl domain may be a hidden confound; if so, epiplexity may partly be a proxy for intra-domain diversity, and explicitly controlling for diversity could separate the two."],"forward_implications":["If maximizing epiplexity is the right objective, curriculum learning becomes a matter of estimating which batch yields the next largest structural gain, rather than relying on heuristic notions of difficulty.","Epiplexity-guided selection improves zero-shot accuracy at both 124M and 1.3B scale, suggesting the signal scales with model size rather than being a small-model artifact.","Because training only on PileCC outperforms prior selectors on The Pile, earlier comparisons of data selection methods on that benchmark may reflect domain-size imbalances rather than selection quality.","Synthetic data that maximizes epiplexity improves fine-tuning performance without any new real data, and mixing it with real data gives the largest gains, indicating a complementarity between synthetic and natural text."],"supporting_citations":[{"why":"Defines epiplexity and the prequential estimator that gives the loss-curve approximation used throughout.","marker":"[20]"},{"why":"Introduces the ADO adaptive data selection method and the per-domain scaling-law approach that EpiSelect extends.","marker":"[4]"},{"why":"Provides the Common Pile dataset, the main benchmark on which EpiSelect is compared.","marker":"[24]"},{"why":"Provides The Pile dataset used to show prior selection benchmarks saturate and that training only on PileCC beats the SOTA selector.","marker":"[23]"},{"why":"Supplies the LLaMA 2 architecture used for the data selection experiments at 124M and 1.3B scale.","marker":"[36]"},{"why":"Supplies the LM Evaluation Harness tasks used to measure zero-shot downstream performance.","marker":"[37]"},{"why":"Provides the REINFORCE policy-gradient update used to train the EpiGen generator.","marker":"[22]"},{"why":"Provides the pretrained GPT-2 weights used to initialize the generator and learner in EpiGen.","marker":"[41]"},{"why":"Provides OpenWebText for measuring perplexity drift and for mixing real data into the evaluation buffer in the EpiGen ablations.","marker":"[42]"}],"fun_headline_variants":["Epiplexity picks data that transfers to new tasks","Boost OOD generalization with epiplexity-driven data","Select data by epiplexity for stronger zero-shot transfer","Epiplexity-guided data beats generic selection","Train on high-epiplexity data, transfer further"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method rests on assuming that the fitted cross-domain scaling law predicts the next marginal gain in epiplexity accurately enough to guide selection; if that parametric curve is wrong, EpiSelect is not actually selecting for epiplexity, and the observed gains could come from some other property of the reweighting.","fun_headline_variants_meta":{"raw":{"variants":["Epiplexity picks data that transfers to new tasks","Boost OOD generalization with epiplexity-driven data","Select data by epiplexity for stronger zero-shot transfer","Epiplexity-guided data beats generic selection","Train on high-epiplexity data, transfer further"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000198,"raw_usage":{"total_tokens":1358,"prompt_tokens":925,"completion_tokens":433,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":353}},"tokens_in":541,"tokens_out":433,"duration_ms":4435,"temperature":1.0,"reasoning_tokens":353,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:29:16.663734+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of domains, fit the cross-domain scaling law on early checkpoints only, then compare the predicted per-domain epiplexity gains from Equation 4 with gains measured by actually training on a small addition of tokens from each domain; if the predictions are uncorrelated with measured gains, the selection signal is not tracking epiplexity.","supporting_citations":[{"cited_title":"Llama 2: Open foundation and fine-tuned chat models,","cited_arxiv_id":null,"evidence_quote":"Supplies the LLaMA 2 architecture used for the data selection experiments at 124M and 1.3B scale."},{"cited_title":"Sutton, David A","cited_arxiv_id":null,"evidence_quote":"Provides the REINFORCE policy-gradient update used to train the EpiGen generator."},{"cited_title":"Lan- guage Models are Unsupervised Multitask Learners.OpenAI, 2019","cited_arxiv_id":null,"evidence_quote":"Provides the pretrained GPT-2 weights used to initialize the generator and learner in EpiGen."},{"cited_title":"Openwebtext corpus.http: //Skylion007.github.io/OpenWebTextCorpus, 2019","cited_arxiv_id":null,"evidence_quote":"Provides OpenWebText for measuring perplexity drift and for mixing real data into the evaluation buffer in the EpiGen ablations."}],"review_version":1}