{"id":"bcd9a06b-1f3b-4607-be08-9e5b118ee77f","arxiv_id":"2505.13398","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MDL-based regularization, which balances data fit with a network's encoding length, preserves perfect solutions on several formal-language tasks, while standard L1, L2, and no regularization degrade them.","lead":"This paper compares regularization methods for small neural networks trained on formal languages, and finds that a Minimum Description Length penalty keeps networks at perfect solutions while L1, L2, or no regularization pushes them away. The result suggests that the choice of objective, not just architecture or optimizer, may explain why networks often learn approximations instead of exact rules.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The GA comparison omits the L1/L2 regularization coefficient; without knowing λ, the claim that MDL preserves golden solutions while L1/L2 push away may be an artifact of an unstated hyperparameter.","rationale":"The reader's verdict identifies several weaknesses, and the missing λ in the GA experiments appears in the reader's rationale but not in the reader's weakest_assumption, which instead focuses on the MDL encoding and the bits-versus-nats mismatch. I consider the unspecified λ more load-bearing because it directly undermines the controlled comparison that supports the abstract's strongest claim. If the L1/L2 penalty strength was not reported, then the observed superiority of MDL could be a consequence of the chosen regularization weight rather than of MDL's inductive bias. The unit mismatch in the MDL objective is also serious, but it affects only the interpretation of the MDL term's scale; the empirical ranking could survive a correction of units, whereas an arbitrary λ could reverse the ranking entirely. The single-seed and time-capped runs are additional concerns, but they would only affect the reliability of the effect size, not the validity of the comparison. The proposed concrete test is feasible because the authors state that all code and data are public, so recovering λ and running a sensitivity analysis should settle whether the central comparison holds across reasonable regularization strengths.","tokens_in":16780,"tokens_out":6627,"duration_ms":66546,"concrete_test":"Download the public repository at github.com/taucompling/mdl-reg-approach and recover the λ value(s) used for L1/L2 in the GA experiments (Experiments 1 and 2). Then, for at least one task (e.g., Toy-English), rerun the GA with L1 and L2 over a log-spaced grid of λ values (e.g., 1e-3 to 1e3), reporting final test Δ% from the optimal score. If any λ yields test gaps comparable to MDL (within about 1 percentage point), the paper's claim that MDL outperforms standard regularization in architecture search is not established; the result would instead be a statement about a particular unstated regularization strength.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim relies on Experiments 1 and 2 (Tables 2 and 3), where a genetic algorithm optimizes networks under MDL, L1, L2, or no regularization. The L1 and L2 objectives in Section 2.5 contain an unspecified coefficient λ. Appendix D specifies λ=1 only for the gradient-descent experiments (Table 11); the GA hyperparameter table (Table 9) contains no λ, and no passage states the value used for the GA runs. This is not a cosmetic omission: the relative strength of L1/L2 versus the cross-entropy term determines whether the regularizer preserves or destroys a golden network. A large λ can collapse weights to zero, while a small λ approximates no regularization. The paper's conclusion that 'standard regularization consistently drifts away from the golden solution' is therefore conditional on an unstated choice. If the λ used in the GA was large, the failure of L1/L2 could simply reflect over-regularization rather than a structural difference from MDL. Because the same λ is not specified across the comparison, the headline result lacks the control needed to attribute the outcome to the regularization method rather than to the regularization strength.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using Minimum Description Length (MDL) regularization for neural networks and evaluates it on next-token prediction tasks over six formal languages (anbn, anbncn, Dyck-1, Dyck-2, Arithmetic Syntax, Toy-English). For each task, the authors manually construct a 'golden' RNN whose output distribution exactly matches the true grammar, initialize training from this network, and compare MDL against L1, L2, and no regularization under three settings: genetic architecture search (Experiment 1), GA with fixed architecture and weight-only mutations (Experiment 2), and gradient descent (Experiment 3). The central claim is that MDL preserves or compresses the golden solutions and achieves test cross-entropy closest to the analytically optimal value, while L1/L2/no regularization push networks away from the perfect solution and increase test cross-entropy. The abstract further claims this holds 'independently of the optimization algorithm.'","tokens_in":16999,"tokens_out":4379,"duration_ms":41185,"significance":"If the results hold, this is a valuable contribution: it provides systematic evidence that the choice of regularization can determine whether an expressive network converges to an exact symbolic solution versus a mere approximation, and it offers a concrete MDL implementation for recurrent networks. The paper has several strengths: it uses an unambiguous correctness criterion (matching the full grammar distribution, not just argmax accuracy), exhaustive test sets with analytically computed optimal scores, publicly available code, and a comparison across two optimization paradigms and multiple grammars. However, the central comparison currently rests on an unspecified L1/L2 regularization coefficient for the main genetic-algorithm experiments, the MDL objective adds bits to nats without a stated conversion, and the strong 'independently of the optimization algorithm' claim is not directly tested because MDL is absent from the gradient-descent experiments. These issues are load-bearing for the headline conclusions.","major_comments":[{"comment":"The L1 and L2 objectives in Section 2.5 contain a regularization coefficient λ, but Table 9 (the GA hyperparameter table) does not specify λ for Experiments 1 and 2; only the gradient-descent configuration in Table 11 states λ=1. The relative strength of L1/L2 versus the cross-entropy term determines whether these regularizers preserve or destroy a golden network: a large λ can collapse weights to zero, while a small λ approximates no regularization. The paper's conclusion that 'standard regularization consistently drifts away from the golden solution' is therefore conditional on an unstated hyperparameter choice. Please report the λ values used for the GA runs and provide a sensitivity analysis over a range of λ to demonstrate that the MDL advantage is not an artifact of over- or under-regularizing the L1/L2 baselines.","section":"Section 2.5, Tables 9 and 11"},{"comment":"The MDL objective is defined as |H| + |D:H|, where |H| is the encoding length of the network in bits (built from bit strings and a prefix-free scheme) and |D:H| is the cross-entropy of the data, which the paper itself identifies with log surprisal and which is conventionally measured in nats. The manuscript adds these two quantities directly without stating a conversion factor. Because the balance between complexity and data fit determines which networks minimize the objective, the lack of a unit conversion makes the MDL objective an arbitrary weighted sum rather than a well-defined information-theoretic quantity. Please either define |D:H| in bits, multiply |H| by log 2 or an equivalent factor, or otherwise justify the chosen scaling as part of the MDL principle.","section":"Section 1.1 and Section 2.5"},{"comment":"The abstract claims that 'using MDL, perfect solutions are selected over approximations, independently of the optimization algorithm,' but Experiment 3 (gradient descent) explicitly excludes MDL because it is non-differentiable (Section 3.3). The only MDL results come from the genetic algorithm, so the claimed independence from the optimization algorithm is not directly tested. At most, the results show MDL success across two GA variants. Please temper the abstract and discussion to reflect that MDL has not been demonstrated under gradient descent, or add a differentiable approximation of MDL and test it in the GD setting.","section":"Abstract and Section 3.3"},{"comment":"All experiments appear to use a single seed (Experiment seed 100, Corpus seed 100 in Tables 9 and 11) and no repeated trials or variance estimates are reported. In several comparisons the differences between MDL and L1/L2 are very small; for example, in Table 3 for anbn the test gaps are MDL 0.1%, L1 0.2%, L2 0.1%, and None 0.2%, differences that could easily arise from run-to-run variability. The claim that MDL 'consistently' outperforms the baselines needs support from multiple independent runs or an explicit justification that the single-run differences exceed expected noise levels.","section":"Section 3, Tables 2-4 and Appendix D"}],"minor_comments":[{"comment":"The evaluation section states that zero probabilities are smoothed by adding 10^-10, but the definition of |D:H| as the log surprisal does not mention this smoothing. Please state explicitly how smoothing enters the reported scores and whether it is applied uniformly to all models in the comparisons.","section":"Section 2.6"},{"comment":"The golden network for Dyck-1 was found by a GA run under MDL, which introduces some circularity into the claim that MDL preserves perfect solutions. The external test-set evaluation mitigates this, but the manuscript should acknowledge this provenance in the limitations or discussion.","section":"Section 2.4"},{"comment":"Table 9 lists 'Number of generations 25,000' but Appendix E states that runs were capped at 25,000 generations or 48 hours, whichever came first, and Table 12 marks several simulations with † for the time limit. The main text and Table 2 should note which results were obtained from time-capped runs, since these may not have converged.","section":"Appendix D.1 and Appendix E"},{"comment":"The column header 'Optim' appears split across lines in some rows; please format these tables so that column headers are unambiguous. Also, in Table 2 the 'None (Lim.|H|)' rows for Dyck-2, Arithmetic, and Toy-English show very large |H| values, which suggests these runs hit the complexity ceiling; the text should discuss whether the ceiling is a fair comparison point for L1/L2.","section":"Tables 2 and 3"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a real and timely question, and the empirical setup is thoughtful in its use of golden networks and exhaustive test sets. However, the headline claim is currently over-stated relative to the evidence: the L1/L2 λ is unspecified for the GA experiments, the MDL objective mixes bits and nats without conversion, and MDL is not tested under gradient descent. These are fixable with additional experiments or careful rewording, so I recommend major revision rather than rejection. I would also encourage the editor to ask the authors to run multiple seeds, as the single-seed results undermine the 'consistently' language."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read on 2505.13398. The central empirical claim — that MDL regularization keeps golden RNNs at the optimum while L1/L2 push them away — is plausible and mostly supported on its home turf, but the paper overstates the result. The abstract's \"independently of the optimization algorithm\" is not earned: MDL is never run with gradient descent, and the GD experiment only shows L1/L2 drifting. That is a claim-to-design gap.\n\nWhat is genuinely useful: the paper broadens MDL-for-RNNs from anbn to Dyck-2, arithmetic syntax, and toy English, compares architecture search, weight-only search, and GD, evaluates against the analytic optimum on exhaustive test sets, and ships code and data. The golden-network methodology is a clean way to ask whether the objective pushes a known-perfect solution away. The hand-built networks for arithmetic and toy English are real work and help with the circularity worry, since those were not found by MDL search.\n\nThe soft spots are empirical, not conceptual. First, the stress-test note is right: Appendix D gives λ=1 only for GD; the GA L1/L2 runs never state λ. Without a λ-sweep, the comparison could just be over-regularization. That's a missing control, not fatal, but it needs addressing. Second, seed 100 only — no variance — and some runs are time-capped at 48 hours, so the selected networks are partly an artifact of compute budget. Third, the MDL objective adds |H| in bits to |D:H| in nats without conversion, making the effective trade-off arbitrary up to a ln2 factor; it is inherited from prior work, but worth fixing or acknowledging. Fourth, the claim of optimizer independence is simply not demonstrated; MDL's non-differentiability is a real limitation, and the paper should say the result holds for evolutionary search, not across optimizers.\n\nOne more point: several golden networks come from prior MDL work or MDL-guided search, so the simplicity measure is partly self-confirming. The hand-built networks mitigate this, but an explicit discussion of how much the result depends on the specific fractional-weight encoding would help.\n\nBottom line: a serious, honest paper with a credible but not-yet-fully-supported central claim. It deserves a real referee, not a desk reject. A revision that specifies λ, adds seeds, and tones down the optimizer-independence claim would make it a solid contribution. I'd flag it maybe for reading group, but wouldn't cite it as established.","headline":"Plausible core result — MDL keeps golden RNN solutions where L1/L2 don't — but the optimizer-independence claim outruns the evidence and the GA comparison needs a λ-sweep and seeds.","tokens_in":17573,"tokens_out":2709,"would_cite":false,"duration_ms":27429,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MDL regularization keeps perfect solutions; L1 and L2 push them away.","keywords":["minimum description length","regularization","neural networks","formal languages","genetic algorithm","gradient descent","generalization","inductive bias"],"falsifier":"Run the same six-task, golden-network protocol under MDL with a different prefix-free code for weights, for example a standard binary floating-point code instead of the fractional code. If L1 or L2 then preserves the golden solution as well as or better than MDL, the reported advantage is an artifact of the chosen complexity measure rather than of the MDL principle.","tokens_in":16549,"feed_emoji":"🎯","tokens_out":8350,"duration_ms":75353,"temperature":0.7,"pith_summary":"The paper argues that the failure of neural networks to learn exact, rule-based solutions is often a failure of the objective function, not of the architecture or optimizer. To test this, the authors build 'golden' networks that perfectly match the true distribution of six formal languages (counting languages, Dyck parentheses, arithmetic syntax, and a miniature English), then train copies of these perfect networks under different regularization schemes. With no regularization, L1, or L2, training pushes the networks away from the perfect solution and raises test cross-entropy; with Minimum Description Length (MDL) regularization, which minimizes the total number of bits needed to describe the network plus the data under it, the perfect solution is preserved or compressed. The paper concludes that MDL supplies an inductive bias that standard regularizers lack, and that the choice of regularization can determine whether a model generalizes exactly or merely approximates.","feed_headline":"MDL keeps perfect network solutions; L1 and L2 push them away","feed_subtitle":"A regularization method that favors simple descriptions keeps exact rules intact, while magnitude-based penalties wreck them.","key_machinery":"The central object is the MDL score |H| + |D:H|, where |H| is the encoded length of the network and |D:H| is the model's log-loss on the data. The |H| term encodes each weight as a signed fraction using a prefix-free code, so a weight like 1/10 gets a short description while a numerically smaller but high-precision weight like 1117/50000 gets a long one; this is what lets MDL penalize information smuggling through precise weight values. The claim is that minimizing this sum creates a bias toward simple networks that still fit the data, which is why a compact golden solution is selected over a larger approximation.","core_discovery":"On the paper's own terms, the discovery is that the MDL objective |H|+|D:H| makes a perfect solution a stable minimum, while cross-entropy with L1, L2, or no regularization does not. Starting from golden networks, the genetic search under MDL stayed at or near the optimal test score across all six tasks and usually kept or reduced the network's complexity, whereas L1, L2, and no-regularization runs consistently increased test |D:H| relative to the analytic optimum, often assigning zero probability to a correct next symbol. The same pattern held when the architecture was fixed and only weights were mutated, and under gradient descent the results drifted away even when training began from the golden weights; MDL itself could not be used in gradient descent because |H| is non-differentiable. The paper reads this as evidence that the objective, not the optimization algorithm, determines whether exact generalization is reached.","pith_inferences":["The paper does not resolve the bit-versus-nat mismatch between |H| and |D:H|; checking the ranking under an explicit bits-per-nat convention would show whether the MDL advantage is robust.","The hand-built fractional code may be doing much of the work; testing MDL with a different prefix-free weight code would distinguish a property of MDL from a property of this particular simplicity measure.","The golden-network drift test could be reused as a general objective sanity check for any architecture and task with a known correct solution.","If the effect transfers to large language models, it would imply that some reasoning failures are not scale or data problems but objective-selection problems."],"forward_implications":["If the MDL objective is the right inductive bias, then L1 and L2 regularization should be expected to fail on any task that requires exact rule induction, regardless of architecture.","MDL-selected models are smaller, which should make them easier to analyze and interpret.","Because MDL preserves exact solutions even when training starts from one, it can serve as a diagnostic: if an objective drifts away from a known perfect solution, that objective is flawed.","The non-differentiability of |H| is the main practical bottleneck; differentiable MDL surrogates or faster evolutionary methods would be needed to scale the approach.","The authors propose that the same benefit could extend to Transformers and to natural-language tasks requiring compositional reasoning, though this is not yet tested."],"supporting_citations":[{"why":"Supplies the MDL-trained RNN methodology and the golden networks reused for anbn, anbncn, and Dyck-2.","marker":"[22]"},{"why":"Direct predecessor showing L1/L2 fail to reach the perfect solution on anbn in LSTMs and only MDL identifies it as an objective minimum.","marker":"[24]"},{"why":"Provides the prefix-free encoding scheme used to compute the |H| complexity term for weights and biases.","marker":"[25]"},{"why":"Foundational MDL formulation that the objective |H|+|D:H| is built on.","marker":"[33]"},{"why":"Foundational inductive-inference theory underlying the MDL principle.","marker":"[38]"},{"why":"Establishes the equivalence between data description length |D:H| and cross-entropy under Shannon-Fano coding.","marker":"[36]"}],"fun_headline_variants":["MDL regularization keeps perfect solutions; L1/L2 destroy them","MDL beats L1/L2: preserves exact network rules","Regularization matters: MDL finds perfect nets, L1/L2 don't","Perfect solutions survive MDL, die under L1/L2","MDL selects exact solutions; L1/L2 push networks astray"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results stand or fall on the hand-built complexity measure |H|: if that measure does not genuinely reflect which networks are simpler, then MDL's advantage over L1 and L2 could be an artifact of the encoding rather than a general property of MDL regularization.","fun_headline_variants_meta":{"raw":{"variants":["MDL regularization keeps perfect solutions; L1/L2 destroy them","MDL beats L1/L2: preserves exact network rules","Regularization matters: MDL finds perfect nets, L1/L2 don't","Perfect solutions survive MDL, die under L1/L2","MDL selects exact solutions; L1/L2 push networks astray"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000486,"raw_usage":{"total_tokens":2352,"prompt_tokens":857,"completion_tokens":1495,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":473,"completion_tokens_details":{"reasoning_tokens":1402}},"tokens_in":473,"tokens_out":1495,"duration_ms":8839,"temperature":1.0,"reasoning_tokens":1402,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:14:25.904315+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same six-task, golden-network protocol under MDL with a different prefix-free code for weights, for example a standard binary floating-point code instead of the fractional code. If L1 or L2 then preserves the golden solution as well as or better than MDL, the reported advantage is an artifact of the chosen complexity measure rather than of the MDL principle.","supporting_citations":[{"cited_title":"Minimum description length recurrent neural networks.Transactions of the Association for Computational Linguistics, 10: 785–799, 2022","cited_arxiv_id":null,"evidence_quote":"Supplies the MDL-trained RNN methodology and the golden networks reused for anbn, anbncn, and Dyck-2."},{"cited_title":"Bridging the Empirical-Theoretical Gap in Neural Network Formal Language Learning Using Minimum Description Length","cited_arxiv_id":"2402.10013","evidence_quote":"Direct predecessor showing L1/L2 fail to reach the perfect solution on anbn in LSTMs and only MDL identifies it as an objective minimum."},{"cited_title":"Springer, 2008","cited_arxiv_id":null,"evidence_quote":"Provides the prefix-free encoding scheme used to compute the |H| complexity term for weights and biases."},{"cited_title":"A formal theory of inductive inference","cited_arxiv_id":null,"evidence_quote":"Foundational inductive-inference theory underlying the MDL principle."}],"review_version":1}