{"id":"1b7a852f-8adb-48e0-a3a4-336bc9d681d6","arxiv_id":"2501.11587","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RPG generates full weights for models up to 200M parameters, including ConvNeXt-L and LLaMA LoRA adapters, at accuracy comparable to trained checkpoints, using recurrent token prototypes to condition a 1D diffusion model.","lead":"Researchers introduce RPG, a generative model that creates complete neural network weights up to hundreds of millions of parameters on a single GPU, using a recurrent network to coordinate parameter tokens and a diffusion model to synthesize them. If it works as claimed, weight generation scales from toy models to practical-scale vision and language networks for the first time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The inverse of the layer-wise normalization in Eq. (1) is never defined, so the denoised tokens cannot be converted into usable network weights as described; this is the key gap in the central claim.","rationale":"The reader identifies the missing inverse of layer-wise normalization as the weakest assumption, and my reading agrees. This is the most load-bearing concern because the central claim is that RPG generates complete, usable network parameters up to roughly 200M weights. If the denoised outputs are normalized tokens and no inverse transform is defined, then the method as described does not produce a deployable network, regardless of how well the diffusion loss fits the normalized training data. The concern is concrete and checkable: either the released code contains an unnormalization step (making the paper's omission a reproducibility gap, not a correctness error), or the step is genuinely absent (making the reported results unattainable from the paper alone). I do not see evidence that the main empirical claims are fraudulent or internally contradicted; the sensitivity of token size and the 8192/16384 discrepancy in the text are real but secondary. The CONDITIONAL verdict remains appropriate: the method is plausible and the experiments are suggestive, but the missing inverse transform must be resolved before the central claim can be fully verified.","tokens_in":26502,"tokens_out":7275,"duration_ms":87410,"concrete_test":"Inspect the released code (or, if none is available, reimplement the pipeline from the paper) and trace the inference path from denoised K to the weights loaded into the target model, checking specifically for an unnormalization operation that applies per-layer mean and standard deviation. Then run the ConvNeXt-L generation exactly as described in Sec. 3.1: if the raw denoised tokens are used directly as network weights, the accuracy should collapse to near chance; if the code instead uses statistics from a training checkpoint or a learned normalization, the paper must state this explicitly, because without it the claimed synthesis of full parameters is not reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. (1) standardizes each layer's weights to zero mean and unit variance before tokenization, and the diffusion loss in Eq. (5) operates on these normalized tokens K. At inference (Sec. 2.3), the paper states that the diffusion model denoises random noise into 'a final parameter set,' but it never specifies the inverse transform from normalized tokens back to raw weights, nor the per-layer mean and standard deviation to use at synthesis time. Without that inverse, the generated tensors are not the weights of a usable network; directly plugging normalized values into ConvNeXt-L or LLaMA would not reproduce the reported accuracies. This is especially acute for the unseen-task setting (Sec. 4), where no training checkpoint exists whose statistics could supply the missing normalization constants. The paper also never describes how padded token regions are located and stripped during inference, and Sec. 3.1's stated default token length of 8192 conflicts with the RPG-Large setting (token size 16384) used for the main large-model results in Tab. 13. These omissions make the central claim—that RPG synthesizes full, usable parameters—incomplete as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RPG, a recurrent diffusion framework for generating full neural-network parameter sets. Parameters are layer-wise normalized and tokenized; a recurrent model (Mamba) produces per-token prototypes that condition a 1D diffusion model. The authors report on-par or better accuracy relative to trained models on ImageNet, ADE20K, COCO, and commonsense reasoning with DoRA/LoRA on LLaMA-7B, and demonstrate generalization to unseen binary classification tasks on CIFAR-10. The main scalability claim is generation of up to roughly 200M parameters in a single pass on a single GPU.","tokens_in":26762,"tokens_out":6873,"duration_ms":66921,"significance":"If the claims hold, RPG would be a meaningful step forward in parameter generation, raising the scale by orders of magnitude and offering a practical single-GPU inference pipeline. The combination of recurrent prototypes with token-level diffusion is a sound architectural idea, and the paper includes a broad set of experiments, including held-out task generalization and ablations of the recurrent model, tokenization, and position embeddings. The main limitation is that the paper does not specify the inverse of the layer-wise normalization, without which the synthesized tensors are not demonstrably usable network weights.","major_comments":[{"comment":"The layer-wise normalization in Eq. (1) maps weights w to (w-μ)/σ, and the diffusion loss in Eq. (5) is computed on these normalized tokens K. The paper never specifies the inverse transform that maps denoised normalized tokens back to raw weights, nor the per-layer μ and σ to be used at synthesis time. For the unseen-task setting in Sec. 4, no training checkpoint exists from which those statistics could be taken. Without an explicit inverse (e.g., multiplying by σ and adding μ, and stripping padding), the generated output is a set of standardized numbers, not the parameters of a usable network; this directly affects the paper's central claim. Please add the inverse transformation and state for each experiment where the μ/σ values come from.","section":"Sec. 2.2, Eq. (1); Sec. 2.3 Inference"},{"comment":"The main text states 'The length of parameter tokens, permutation states, position embeddings, and prototypes is set to 8192' (Sec. 3.1), but Appendix B.3 states that ViT-Base, ConvNeXt-Large, ADE20K, COCO, and DoRA rank 64 used RPG-Large with token size 16384. These statements are inconsistent, making the configuration of the headline large-model results ambiguous. Please state the exact token size used for each reported model and correct the default description.","section":"Sec. 3.1 vs. Sec. B.3, Table 13"},{"comment":"The paper does not explain how padded regions are located and removed after generation, nor how the permutation state S is constructed for unseen tasks. In Sec. 4, binary embeddings are used as conditioning inputs, but the relationship between the binary embedding and S is not defined (e.g., are they concatenated, or does the embedding replace S?). A precise de-tokenization protocol and a definition of S for the unseen-task experiments are needed for the inference procedure to be reproducible.","section":"Sec. 2.2, Eq. (2); Sec. 2.3; Sec. 4"},{"comment":"It is unclear whether the 'original' accuracies in Table 1 refer to the pretrained timm models or to the fine-tuned checkpoints used to create the training set. If the latter, the on-par result is partly a reconstruction of training data; if the former, that should be stated explicitly. The paper should clarify the baseline and, if possible, add an evaluation on a truly held-out architecture checkpoint to strengthen the generalization claim.","section":"Sec. 3.1, Sec. 3.2, Table 1"}],"minor_comments":[{"comment":"The PDF contains many garbled 'uni' character sequences (e.g., on pages 1, 3, and 17), indicating an encoding problem; please regenerate the source to a clean text version.","section":"General"},{"comment":"The notation '−µ, /σ' is non-standard; rewrite the normalization as (w[i] − μ_i) / σ_i.","section":"Eq. (1)"},{"comment":"Table 8 is difficult to read: the columns and subscripts for the four baseline methods are misaligned, and the meaning of the subscript numbers is not explained in the caption.","section":"Table 8"},{"comment":"The caption refers to 'identical seen binary embeddings' and 'unseen binary embeddings' but does not define the color scheme or the number of parameters visualized; please add explicit axes and legends.","section":"Fig. 5"},{"comment":"Table 15 reports accuracy at epoch 0 for 'RPG initialization + finetune' as 94.4, which seems to correspond to the generated model's zero-shot accuracy; for clarity, distinguish the generated model's zero-shot accuracy from the accuracy after fine-tuning.","section":"Appendix C.2, Table 15"},{"comment":"The 'parameter counts' of the recurrent model for RPG-Large (3076M) exceeds the generated model size (up to 200M); this is not a flaw but should be highlighted in the efficiency discussion, since the memory and compute of the generator are not negligible.","section":"Table 13"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and the scale-oriented claims are impressive. The missing inverse normalization is a load-bearing omission that must be fixed, along with the token-size inconsistency and the ambiguity of the ImageNet baseline. The authors should also consider whether the 'first method' claim is appropriately scoped given recent concurrent work, but this is secondary to the technical points. I recommend major revision rather than rejection because the core ideas and evaluations are promising and the gaps appear addressable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Re: arXiv:2501.11587. The headline result is real as far as it goes: RPG is the first method I know that synthesizes full parameters of ConvNeXt-L and LLaMA-LoRA (roughly 200M weights) in a single pass on one GPU, with accuracies close to the trained originals on ImageNet, ADE20K, COCO, and commonsense reasoning. The design is sensible: tokenize per-layer, normalize each layer, pass the tokens through a recurrent Mamba that outputs prototypes, then condition a shared 1D diffusion on those prototypes. The ablation suite is thorough (recurrent structure, token size, tokenization, position embeddings, sampling steps), and the unseen-task experiments on CIFAR-10 binary splits give some genuinely independent evidence of generalization beyond the training set.\n\nThat said, the paper has a load-bearing omission. The layer-wise normalization in Eq. (1) is never inverted in the described pipeline. The inference section says the diffusion model denoises random noise into 'a final parameter set,' but the denoised output is a set of normalized tokens. Without specifying the per-layer means and standard deviations used to map those tokens back to raw weights, the method is incomplete. This is especially acute in the unseen-task setup, where no checkpoint exists to supply the statistics. The stress-test note is correct that this is the key gap; it needs a real fix in the text, not a sentence, because the reported accuracies depend on the missing de-normalization. The paper also never says how padded token regions are located and removed during inference, and Sec. 3.1's default token size of 8192 conflicts with the 16,384 token size used for the RPG-Large setting behind the main large-model results.\n\nTwo more moderate concerns. The ImageNet parity numbers are partly a reconstruction test: the 50 fine-tuned checkpoints used as training data are the same pool used for evaluation, so on-par accuracy is not a fully external benchmark. The unseen-task results ease this, but those are small ViT-Tiny binary models, not the 200M cases. Also, the generator itself is large: RPG-Large has over 3 billion parameters in the recurrent model alone, more than an order of magnitude larger than the 197M target it produces. That doesn't invalidate the scaling claim, but it tempers the 'lightweight footprint' language unless the reference point is training the target model from scratch.\n\nOverall, this is a serious paper worth refereeing. The central idea is new, the experiments are extensive, and the held-out results show clear thinking. But the missing inverse normalization is a fundamental completeness gap that has to be resolved before the method can be reproduced or fully trusted. Send it to review with a request for that clarification, the padding-deletion detail, the token-size consistency fix, and ideally code. If the authors supply those, this would be a solid contribution; as written, it's a conditional accept at best.","headline":"RPG is a credible scaling breakthrough for weight generation, but the missing inverse of the layer-wise normalization makes the central claim under-specified as written.","tokens_in":27279,"tokens_out":4710,"would_cite":false,"duration_ms":46419,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a recurrent-conditioned diffusion model can synthesize a network's full parameter set—up to about 200 million weights—in a single pass on one commodity GPU, matching fully trained accuracy across vision and language…","keywords":["parameter generation","diffusion models","recurrent neural networks","weight space learning","large-scale models","LoRA","neural network weights","AI generating AI"],"falsifier":"Generate a network for an unseen task while withholding all training-checkpoint statistics and using only the model's own denoised outputs to invert normalization; if accuracy collapses or the inverse transform is undefined, the single-pass generation claim is not reproducible. Concretely, compute per-layer mean and standard deviation from one held-out checkpoint and reuse them across different generated samples; if performance depends on which checkpoint's statistics are used, the pipeline is not truly synthesizing complete weights from noise.","tokens_in":26299,"feed_emoji":"🧠","tokens_out":6367,"duration_ms":63833,"temperature":0.7,"pith_summary":"The paper proposes a way to generate the complete weight set of a neural network—up to roughly 200 million parameters—with a single diffusion pass on one commodity GPU. It claims this is the first method to reach that scale, and that the generated networks perform on par with normally trained networks on ImageNet-1K classification, ADE20K segmentation, COCO detection, and commonsense reasoning with LLaMA-based LoRA adapters. The approach splits weights into per-layer tokens, uses a recurrent model to build prototypes that summarize cross-token relationships, and lets a 1D diffusion model denoise those tokens conditioned on the prototypes. If correct, it makes 'AI generating AI' practical: entire working networks could be synthesized quickly, generalized to new tasks, and even initialized from text-style prompts.","feed_headline":"Diffusion generates 200M-parameter networks on one GPU","feed_subtitle":"A recurrent-conditioned generator matches trained accuracy across ImageNet, COCO, ADE20K, and LLM adapters.","key_machinery":"The key machinery is parameter tokenization combined with recurrent-conditioned diffusion: layer-wise normalization and uniform chunking turn a network into a sequence of tokens; a recurrent model (Mamba followed by an MLP) produces per-token prototypes encoding global inter-token dependencies; and a 1D convolutional diffusion model denoises random noise into token values conditioned on those prototypes. The recurrent pass captures cross-token correlation while the diffusion pass refines each token, and this two-step decoupling is what keeps memory usage growing mildly with parameter count rather than quadratically.","core_discovery":"Full-network parameter generation is feasible at practical scale: RPG partitions parameters layer by layer into normalized, uniformly sized tokens, adds a one-hot permutation state to handle weight symmetry, and feeds the token sequence through a recurrent model whose per-token outputs condition a 1D diffusion denoiser. The paper reports that for ConvNeXt-Large (197.8M parameters) and LLaMA-7B DoRA adapters (113.1M parameters) it synthesizes complete parameter sets within minutes on a single H100 with roughly 20GB of memory, matching original accuracy; across seven ImageNet-1K architectures it stays within about 0.5 percent of the trained models; and on 1,022 CIFAR-10 binary tasks it generates valid parameters for unseen tasks without seeing their checkpoints.","pith_inferences":["If the method holds up, the natural next target is cross-architecture generalization: the paper only generates parameters for architectures seen during training, and it explicitly notes that limitation, so conditioning on architecture descriptions rather than fixed token layouts would test whether the learned structure is truly about weights as such.","The recurrent-conditioned diffusion recipe may transfer to other high-entropy structured tensors—such as pruning masks, quantization schedules, or optimizer states—where global correlation matters but memory is the bottleneck.","One can test the 'unseen task' claim more sharply by conditioning on randomized embeddings far outside the convex hull of the training distribution and measuring whether accuracy degrades gracefully; the current evaluation uses held-out embeddings from the same 1,022-task set.","The missing inverse-normalization detail is a reproducibility stake: an independent implementation must decide how per-layer mean and standard deviation are supplied at inference, and if those statistics come from training checkpoints, the claim of generating from pure noise for novel tasks is weakened."],"forward_implications":["RPG can synthesize full parameter sets of ConvNeXt-Large and ViT-Base in about one minute on a single H100 with roughly 20GB of memory, moving weight generation from academic demonstrations toward practical deployment.","Generated networks match original accuracy on ImageNet-1K, ADE20K, COCO, and seven commonsense reasoning tasks with DoRA/LoRA on LLaMA-7B, so a single trained generator can replace many trained checkpoints.","The generator generalizes to unseen CIFAR-10 binary tasks when conditioned on a 10-bit task embedding, meaning a single trained RPG can produce valid models for tasks not in its training set.","RPG-initialized parameters speed up downstream training: on an unseen task, an RPG-initialized ViT-Tiny reaches 94.4% accuracy at epoch 0 versus 50.0% for training from scratch.","Because the diffusion model is shared across tokens and tokens can be processed sequentially, generation memory can drop below 8GB, allowing deployment on smaller GPUs."],"supporting_citations":[{"why":"p-diff supplies the 1D convolutional diffusion architecture and denoising objective that RPG adapts, and is the main prior method whose memory ceiling RPG overcomes.","marker":"Wang et al., 2024"},{"why":"Mamba is the default recurrent model; its linear-time state-space design lets RPG process long token sequences within GPU memory constraints.","marker":"Gu & Dao, 2024"},{"why":"MAR provides the conditional diffusion formulation that RPG draws on to generate token values conditioned on prototypes.","marker":"Li et al., 2024a"},{"why":"G.pt is the earlier diffusion-based parameter generator whose roughly 10K-parameter ceiling motivates RPG's scalability claim.","marker":"Peebles et al., 2022"},{"why":"SKDE30 is a baseline that exhausts memory at ResNet-18 scale, grounding the claim that RPG breaks the memory barrier.","marker":"Schürholt et al., 2022a"},{"why":"SANE is the tokenization baseline that divides parameters by channel; RPG's within-layer tokenization is compared against it.","marker":"Schürholt et al., 2024"},{"why":"D2NWG is a diffusion-based baseline for weight generation that RPG outperforms in both accuracy and scale.","marker":"Soro et al., 2024"}],"fun_headline_variants":["Recurrent diffusion generates full networks matching trained accuracy","RPG: scale parameter generation to 200M on a single GPU","Generate complete networks in minutes with recurrent diffusion","Full network synthesis via RPG matches original accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the per-layer normalization used to create tokens can be inverted at generation time, so denoised normalized tokens can be turned back into real network weights; the paper does not state where the per-layer mean and standard deviation come from when generating for an unseen task.","fun_headline_variants_meta":{"raw":{"variants":["Recurrent diffusion generates full networks matching trained accuracy","RPG: scale parameter generation to 200M on a single GPU","Generate complete networks in minutes with recurrent diffusion","Full network synthesis via RPG matches original accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000933,"raw_usage":{"total_tokens":3972,"prompt_tokens":900,"completion_tokens":3072,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":3010}},"tokens_in":516,"tokens_out":3072,"duration_ms":28787,"temperature":1.0,"reasoning_tokens":3010,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:04:36.888112+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate a network for an unseen task while withholding all training-checkpoint statistics and using only the model's own denoised outputs to invert normalization; if accuracy collapses or the inverse transform is undefined, the single-pass generation claim is not reproducible. Concretely, compute per-layer mean and standard deviation from one held-out checkpoint and reuse them across different generated samples; if performance depends on which checkpoint's statistics are used, the pipeline is not truly synthesizing complete weights from noise.","supporting_citations":[],"review_version":1}