WeightCLIP: Aligning Datasets and Models for Weight Space Learning

Aron Asefaw; Damian Borth; Damian Falk; Konstantinos Tzevelekakis; L\'eo Meynent

Aligning neural-network weight representations with the datasets they were trained on yields a navigable latent space for retrieving, generating, and refining models from data prompts.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 01:39 UTC pith:YLUTT6ZC

load-bearing objection Solid empirical WSL paper: reshaping the weight latent with dataset contrastive alignment is real and useful, even if generation gains are partly head-repair and scope is narrow. the 3 major comments →

arxiv 2607.03551 v1 pith:YLUTT6ZC submitted 2026-07-03 cs.LG

WeightCLIP: Aligning Datasets and Models for Weight Space Learning

Aron Asefaw , Konstantinos Tzevelekakis , Damian Falk , L\'eo Meynent , Damian Borth This is my paper

classification cs.LG

keywords weight space learningcontrastive alignmentdataset-to-model generationmodel zooslatent refinementneural network weightshyper-representations

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Weight space learning treats the parameters of trained networks as a data modality, but pure reconstruction or property-prediction objectives leave the latent space hard to navigate because proximity has no clear semantic meaning. This paper claims that the datasets models were trained on act as natural captions for those weights, and that a contrastive objective can reshape the weight latent so models cluster by training data. Once aligned, a small set of images (a data prompt) can retrieve related models, be mapped into a full weight-space embedding that decodes into a usable network, and support short latent-space refinement that stays on the learned model manifold. Empirically, on fixed-architecture model zoos of CNNs and ResNet-18s, this improves in- and out-of-distribution retrieval, produces stronger dataset-conditioned initializations than retrieval-only or hypernetwork baselines, and can beat matched-budget fine-tuning in weight space. A sympathetic reader would care because it turns weight-space representations into something that can be prompted and adapted without training a new model from scratch for each target dataset of a known architecture family.

Core claim

Explicitly aligning weight-space embeddings with dataset embeddings via a contrastive loss reorganizes the latent manifold of neural networks by training-data identity, and that reorganization is what makes dataset-to-model retrieval, generation of strong model initializations from data prompts, and constrained latent refinement work better than reconstruction-only weight spaces or alignment used only as external retrieval or guidance.

What carries the argument

WeightCLIP: a weight autoencoder and a set-based dataset encoder trained jointly so that token-level contrastive alignment reshapes the autoencoder latent around datasets; a lightweight mapper then turns a data prompt into model tokens, which can be shell-projected and refined by backpropagating task loss through the decoder.

Load-bearing premise

The signal that identifies which dataset a model was trained on is spread evenly enough across weight tokens, and trained models sit on a hyperspherical shell that stays a valid constraint, so the same aligned space and refinement recipe transfer to held-out datasets of the same architecture.

What would settle it

On the same held-out vision datasets and fixed architecture, remove the contrastive alignment (or break the shell constraint) and check whether dataset-to-model generation and matched-budget latent refinement stop outperforming reconstruction-only latents, nearest-neighbor retrieval, and standard fine-tuning on accuracy after 0–10 adaptation steps.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Dataset prompts become a practical interface for zero-shot or few-shot model retrieval inside a same-architecture zoo.
A small image set can be mapped into decoded weights that serve as better initializations than training from scratch or retrieving the nearest zoo model.
Test-time adaptation can be done by moving latents on a hyperspherical shell rather than only fine-tuning raw weights, sometimes with higher accuracy under the same data and step budget.
Upfront cost of building the aligned space and mappers amortizes over many new target datasets for a known architecture family.
Linear interpolation of dataset prompts produces smooth trade-offs in decoded model accuracy between those datasets, making the latent navigable by data semantics.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the same alignment works beyond single-architecture vision zoos, weight hubs could be queried and specialized the way image banks are queried with text—without re-running full training for every new domain.
The residual cross-modal gap that forces a mapper suggests that pure nearest-neighbor retrieval will keep underperforming generative mapping even after strong alignment, so future systems will need both structure and a learned bridge.
Extending the idea to heterogeneous architectures would require an explicit way to encode architecture identity alongside dataset identity; otherwise clusters would mix task and structure signals.
Latent refinement outperforming fine-tuning under matched budgets hints that staying near the trained-model manifold is a useful regularizer whenever generated heads or class counts mismatch the target.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Solid empirical WSL paper: reshaping the weight latent with dataset contrastive alignment is real and useful, even if generation gains are partly head-repair and scope is narrow.

read the letter

The useful takeaway is simple: if you train a weight autoencoder with a bidirectional token-level contrastive loss against DeepSets dataset embeddings, the latent becomes organized by training data, retrieval gets cleaner, and you can map a small data prompt into a decodeable model that is a better starting point than TANS retrieval or a hypernet baseline. That is the actual new piece—not CLIP, not SANE, not TANS, but using the alignment to reshape the weight manifold itself rather than only scoring or conditioning on a frozen one.

They do the work carefully. Two zoos (CNN and ResNet-18), held-out OOD datasets, ablations of alignment (Fig. 4, Tab. 4), linear vs memory-bank mappers, nearest-neighbor vs generation (Tab. 5), and shell-constrained latent refinement vs matched-budget fine-tuning (Tabs. 6–7). Interpolation of dataset prompts trades off accuracy smoothly. App. B.2 shows dataset-ID is not localized to a few tokens. Citations are honest about TANS, D2NWG, Text2Model, and SANE. Math is standard contrastive + recon; no circular derivation. Code is promised.

Soft spots are real but proportional. Scope is vision-only, fixed architecture, modest models. The stress-test concern has some bite: Ep.0/Ep.1 gains and refinement partly reflect reconstruction quality plus classifier-head repair (they say so in Sec. 3.5). Tab. 4 still shows full alignment beats no-alignment and projection-only, and shell + alignment beats fine-tuning under the same steps/data, so the claim is not empty—but a frozen-AE / frozen-head control would have made attribution cleaner. Uniform token signal and the training-estimated hypersphere are assumptions that work here and are only partially checked. Free parameters (temp, alignment weight, shell, γ) are ordinary for the genre.

This is for people already in weight-space learning or model-zoo transfer who want a practical way to prompt and refine fixed-architecture weights. Not a field reorganizer; a clear incremental step with usable experiments. I would send it to peer review. Engage if you care about navigable weight latents; skip if you only want architecture-agnostic or multi-modal results.

Referee Report

3 major / 0 minor

Summary. The paper proposes WeightCLIP, a contrastive alignment method that reshapes a weight-space autoencoder latent (built on SANE-style tokenization) so that model embeddings and DeepSets dataset embeddings share a common space. Alignment is used for three tasks: dataset-to-model retrieval, mapping a dataset prompt to a token sequence that is decoded into network weights (linear and memory-bank mappers), and test-time latent refinement under a hyperspherical-shell constraint that is claimed to match or beat weight-space fine-tuning under matched steps and data. Experiments use two same-architecture model zoos (CNN ~11k params; ResNet-18 ~2.8M) trained on multiple image datasets, with held-out OOD datasets for evaluation, plus ablations of alignment, mapper type, nearest-neighbor vs generation, and shell/alignment for refinement.

Significance. If the results hold under stronger controls, this is a useful step for weight-space learning: it treats datasets as the natural semantic reference frame for model zoos and shows that joint contrastive reshaping of the weight latent (rather than retrieval-only alignment or frozen-latent conditioning) can improve retrieval, promptable generation, and constrained test-time adaptation. Strengths include a clear problem framing relative to TANS and D2NWG-style projection, multi-task evaluation on two architecture scales, OOD dataset splits, and concrete ablations (Tabs. 1–7, Figs. 4–5). The work is empirical rather than theorem-driven; its value is the demonstration that dataset-conditioned weight latents can serve as amortizable initializations for a fixed architecture family.

major comments (3)

Central attribution of generation gains to dataset-aligned semantics is not fully isolated. Tab. 4 shows Ep.0 average accuracy rising from 14.5 (no alignment) to 23.9 (D2NWG-style projection) to 26.6 (WeightCLIP), which supports alignment, but Sec. 3.5 explicitly states that mapped models often need classifier-head repair, and Tabs. 3 and 6 evaluate after fine-tuning/refinement that can fix heads. Without a control that freezes the autoencoder and only toggles alignment, or freezes/reinitializes only the head and reports body-only transfer, Ep.1/Ep.10 gains may partly reflect reconstruction quality plus rapid head adaptation rather than transferable task geometry in the body weights.
The load-bearing token-uniformity and shell-manifold assumptions are only weakly validated for the claimed transfer. Sec. 3.3 applies token-level contrastive loss under the hypothesis that dataset signal is uniform across tokens; App. B.2 only shows that linear probes recover dataset ID with similar accuracy across positions, not that token directions encode transferable task structure for held-out datasets. Sec. 3.5 and Tab. 7 show that the hyperspherical shell helps refinement, but c and r are estimated from training latents of the same architecture family; there is no diagnostic that the shell remains the correct manifold for OOD datasets or multi-window ResNets beyond the reported averages.
Baselines and generation protocol leave residual ambiguity about novelty of the mapped models. Tab. 5 shows mappers beat nearest-neighbor selection in the same latent (26.3 vs 30.9/31.5 Ep.0 avg), which is important, yet the memory-bank mapper is still a soft mixture of training tokens (Eqs. 6–7), and generation reports top-m of 100 subsampled prompts (Sec. 4.1). The paper should more clearly quantify how often generated models differ from the nearest zoo entry in weight space / accuracy, and how sensitive results are to the top-m selection, so that gains are not partly selection effects.

Circularity Check

0 steps flagged

Empirical method paper: alignment, mapping, and refinement are trained and measured on held-out models/datasets; no central result reduces by construction to its inputs.

full rationale

WeightCLIP’s chain is standard supervised/contrastive ML, not a first-principles derivation. The autoencoder is trained with reconstruction plus a bidirectional token–dataset contrastive loss (Eqs. 1–4); mappers are fit on training (e_D, Z) pairs (linear ridge or memory-bank soft neighborhood loss); refinement optimizes task loss through the decoder under an empirical shell prior (Eqs. 8–11). Downstream claims—OOD retrieval accuracy, decoded Ep.0/Ep.1/Ep.10 accuracy, and latent refine vs fine-tune under matched budgets—are measured on held-out model instances and held-out image datasets, so they are not forced by the fit. In-distribution retrieval (Tab. 1) is close to the training objective but still evaluates held-out models, not a tautological re-read of fitted free parameters. Self-citations (SANE/Schürholt et al. for the weight autoencoder; prior WSL geometry for the hyperspherical shell) supply infrastructure and a geometric prior; they do not uniquely force the alignment objective or the reported generation/refinement gains. No uniqueness theorem, ansatz smuggled as external fact, or renaming of a known closed-form result appears. Correctness concerns about attributing gains to alignment vs reconstruction/head repair are orthogonal to circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 3 invented entities

The central empirical claim rests on standard contrastive and autoencoder machinery plus a small set of modeling choices (token-level uniform dataset signal, hyperspherical shell prior, DeepSets prompts, same-architecture zoos). Free parameters are ordinary ML hyper-parameters and statistics estimated from the training zoo; no new physical constants. Invented entities are methodological constructs whose only evidence is the paper’s own ablations.

free parameters (5)

contrastive temperature / logit bias
Temperature fixed at 1.0 and learnable logit bias initialized at -4.0 (App. C.5); they scale the alignment loss that reshapes the latent.
alignment loss weight
Set to 0.25 in the composite autoencoder objective; controls how strongly dataset information warps reconstruction latents.
hyperspherical shell center c and radius r
Estimated as average token mean and norm from training latents (Sec. 3.5); used as hard projection for refinement initialization and constraint.
latent dimension d and token/window sizes
Chosen per architecture (128/192, token sizes 201/288, etc., Tab. 12); define the shared embedding space in which alignment and mapping occur.
refinement neighborhood penalty γ and step count
γ and number of gradient steps (e.g., 20 steps, batch 100) trade off task loss versus staying near the mapped latent; directly affect the refinement-vs-finetune comparison.

axioms (5)

domain assumption Contrastive alignment of heterogeneous modalities yields a shared space in which cosine similarity is semantically meaningful (CLIP-style).
Invoked throughout Sec. 1–3 as justification for reshaping weight latents with dataset embeddings.
ad hoc to paper Dataset identity signal is distributed roughly uniformly across weight tokens of a trained network.
Stated as hypothesis in Sec. 3.3 and checked with linear probes in App. B.2; underpins token-level (not model-level) contrastive loss.
domain assumption Trained weight embeddings concentrate near a hyperspherical shell that can be used as a manifold constraint for latent optimization.
Cited from prior WSL/contrastive geometry work and applied per-token in Sec. 3.5; load-bearing for the refinement procedure.
domain assumption DeepSets on small image subsets produces embeddings that identify dataset distribution well enough to serve as prompts.
Used for all dataset prompts; sensitivity to set size is ablated in App. B.3.
domain assumption SANE-style windowed tokenization and transformer autoencoder can reconstruct and decode useful weight sequences for the studied architectures.
Backbone of encoding/decoding; taken from Schürholt et al. 2024 and assumed adequate for generation quality.

invented entities (3)

dataset-aligned weight latent manifold (WeightCLIP space) no independent evidence
purpose: Provide a semantic reference frame so dataset prompts can retrieve, map, and refine model weights.
Defined by the joint training objective; evidence is internal clustering, retrieval, and generation tables rather than an external independent measurement.
memory-bank mapper with soft neighborhood supervision no independent evidence
purpose: Map a dataset embedding to a window of weight tokens as a convex combination of stored training latents.
Introduced in Sec. 3.4 / App. C.6; performance gains are measured only within this paper’s zoos.
shell-constrained latent refinement procedure no independent evidence
purpose: Adapt generated models at test time by optimizing latents through the decoder while staying on the learned manifold.
Sec. 3.5; compared only against fine-tuning on the same generated initializations.

pith-pipeline@v1.1.0-grok45 · 26684 in / 3613 out tokens · 35330 ms · 2026-07-12T01:39:54.214831+00:00 · methodology

0 comments

read the original abstract

Weight space learning aims to learn representations of neural network (NN) weights, enabling different downstream tasks. Existing approaches show promising performance, but lacking a way to shape these weight-space representations using information about the datasets the models were trained on, thus limiting downstream applications. We propose WeightCLIP, a method for learning a dataset-aligned latent space for neural networks, where datasets information is induced during training. The NNs are encoded as latent representations using an autoencoder, while dataset samples are encoded using a dataset encoder. The two representations are aligned using a contrastive objective, effectively reshaping the weight-space representations according to the datasets. We demonstrate that such representations can be used for different downstream tasks, including mapping dataset information to a weight-space representation that decode to strong models. In addition, we introduce a latent refinement process for generating models that outperforms standard fine-tuning. Overall, our results demonstrate that explicitly incorporating dataset information improves what can be achieved with weight-space representations across retrieval, generation, and refinement. Code will be available at https://github.com/HSG-AIML/WeightCLIP.

Figures

Figures reproduced from arXiv: 2607.03551 by Aron Asefaw, Damian Borth, Damian Falk, Konstantinos Tzevelekakis, L\'eo Meynent.

**Figure 1.** Figure 1: WeightCLIP trains a latent representation of model weights using a contrastive loss against the dataset the model weights have been trained on. Once trained, a data prompt for an unseen dataset can be used to generate model weights specifically for this dataset. they can be used for several downstream tasks such as the prediction of model properties from weights (Unterthiner et al., 2020; Eilertsen et al.,… view at source ↗

**Figure 2.** Figure 2: The proposed methods differ from other data prompting techniques in WSL. (a) While TANS (Jeong et al., 2021) trains with a contrastive objective on encoded model weights and dataset characteristics, the method aims to retrieve known nearest neighbour models. (b) In contrast, D2NWG (Bedionita et al., 2025b) trains a generative model able to sample novel weights following a conditioned projection from a data… view at source ↗

**Figure 3.** Figure 3: After training, the process to generate and refine models weights is done as following: a data encoder dθ encodes a subset of the dataset into an data embedding eD (the data prompt) into the aligned latent space. This data embedding is mapped to a sequence of model embeddings Z from which model weights W are decoded using the decoder hψ of the trained autoencoder. This newly generated NN is used to evaluat… view at source ↗

**Figure 4.** Figure 4: t-SNE projection of NN embeddings from the model zoo, obtained by mean-pooling token representations. Colors indicate the dataset each model was trained on. Left: Trained without alignment, where dataset alignment is not explicitly enforced, resulting in substantial overlap between NN embeddings. Right: Trained with the proposed alignment, where models trained on the same dataset form more compact and clea… view at source ↗

**Figure 5.** Figure 5: Latent-space navigation via dataset prompt interpolation. We linearly interpolate between dataset prompts from two distinct datasets (cactus aerial → ct images), map each interpolated prompt to a NN latent representation, and decode it. The plot shows downstream accuracy of the generated models on both datasets as a function of the interpolation coefficient t, revealing a smooth trade-off in performance be… view at source ↗

**Figure 6.** Figure 6 [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗

**Figure 7.** Figure 7: Pairwise cosine distance distributions between token embeddings with and without dataset–model alignment. Red curves denote intra-dataset distances and blue curves denote inter-dataset distances. Alignment induces a clear separation between datasets, whereas training without it yields overlapping distributions. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗

**Figure 8.** Figure 8: Model–dataset cosine similarity matrix induced by the aligned latent space after triangulation. Rows correspond to dataset embeddings and columns to the token embeddings. Strong diagonal structure indicates consistent alignment between models and their originating datasets, with limited off-diagonal confusion primarily between semantically overlapping datasets. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

27 extracted references · 3 canonical work pages

[1]

S., Willmott, D., Bair, A., Ganesh, M

Akinwande, V ., Norouzzadeh, M. S., Willmott, D., Bair, A., Ganesh, M. R., and Kolter, J. Z. Hyperclip: Adapt- ing vision-language models with hypernetworks.arXiv preprint arXiv:2412.16777,

Pith/arXiv arXiv
[2]

M., and Humayun, A

Alam, S., Reasat, T., Doha, R. M., and Humayun, A. I. NumtaDB: Assembled bengali handwritten digits.arXiv preprint arXiv:1806.02452,

Pith/arXiv arXiv
[3]

Text2model: Text-based model in- duction for zero-shot image classification

Amosy, O., V olk, T., Shapira, E., Ben-David, E., Reichart, R., and Chechik, G. Text2model: Text-based model in- duction for zero-shot image classification. InFindings of the Association for Computational Linguistics: EMNLP 2024, pp. 155–172,

2024
[4]

Bedionita, S., Andreis, B., Chong, S., and Hwang, S. J. Instruction-guided autoregressive neural network parame- ter generation.ICLR 2025 Workshop on Neural Network Weights as a New Data Modality, 2025a. Bedionita, S., Andreis, B., Lee, H., Jeong, W., Chong, S., Hutter, F., and Hwang, S. J. Diffusion-based Neural Network Weights Generation. In13th Interna...

2025
[5]

D’Orazio, A., Briglia, M

doi: 10.1109/ACCESS.2020.3010287. D’Orazio, A., Briglia, M. R., Crisostomi, D., Loi, D., Rodol`a, E., and Masi, I. Implicit inversion turns CLIP into a decoder. InThe Fourteenth International Confer- ence on Learning Representations,

doi:10.1109/access.2020.3010287 2020
[6]

Falk, D., Sch ¨urholt, K., Tzevelekakis, K., Meynent, L., and Borth, D

arXiv preprint. Falk, D., Sch ¨urholt, K., Tzevelekakis, K., Meynent, L., and Borth, D. Learning Model Representations Us- ing Publicly Available Model Hubs.arXiv preprint arXiv:2510.02096,

arXiv
[7]

Han, X., Wang, Z., Zhao, B., Zhang, B., Li, J., Borth, D., Yu, R., Maron, H., Ye, Y ., Yin, L., et al

doi: 10.1145/1015330.1015376. Han, X., Wang, Z., Zhao, B., Zhang, B., Li, J., Borth, D., Yu, R., Maron, H., Ye, Y ., Yin, L., et al. A survey of weight space learning: Understanding, representation, and generation.arXiv preprint arXiv:2603.10090,

doi:10.1145/1015330.1015376
[8]

X., and Borth, D

Hanna, J., Falk, D., Yu, S. X., and Borth, D. Geosane: Learning geospatial representations from models, not data.arXiv preprint arXiv:2603.23408,

arXiv
[9]

Ho, J., Jain, A., and Abbeel, P

doi: 10.1109/JSTARS.2019.2918242. Ho, J., Jain, A., and Abbeel, P. Denoising Diffusion Prob- abilistic Models. InAdvances in Neural Information Processing Systems (NeurIPS),

doi:10.1109/jstars.2019.2918242 2019
[10]

Version 1.3.1

doi: 10.13026/4nae-zg36. Version 1.3.1. Hssayeni, M. D., Croock, M. S., Salman, A. D., Al-khafaji, H. F., Yahya, Z. A., and Ghoraani, B. Intracranial hem- orrhage segmentation using deep convolutional model. Data, 5(1):14,

work page doi:10.13026/4nae-zg36
[11]

Jeong, W., Lee, H., Park, G., Hyung, E., Baek, J., and Hwang, S

doi: 10.3390/data5010014. Jeong, W., Lee, H., Park, G., Hyung, E., Baek, J., and Hwang, S. J. Task-adaptive neural network search with meta-contrastive learning.Advances in Neural Informa- tion Processing Systems, 34:21310–21324,

work page doi:10.3390/data5010014
[12]

Conditional lora parameter generation.arXiv preprint arXiv:2408.01415,

Jin, X., Wang, K., Tang, D., Zhao, W., Zhou, Y ., Tang, J., and You, Y . Conditional lora parameter generation.arXiv preprint arXiv:2408.01415,

Pith/arXiv arXiv
[13]

Accelerating training with neuron interaction and nowcasting networks

Knyazev, B., Moudgil, A., Lajoie, G., Belilovsky, E., and Lacoste-Julien, S. Accelerating training with neuron interaction and nowcasting networks. InInternational Conference on Learning Representations, volume 2025, pp. 29569–29588,

2025
[14]

Mahoney, M

doi: 10.1016/j.ecoinf.2019.05.005. Mahoney, M. and Martin, C. Traditional and heavy tailed self regularization in neural network models. InInterna- tional Conference on Machine Learning, pp. 4284–4293. PMLR,

work page doi:10.1016/j.ecoinf.2019.05.005 2019
[15]

A., Zhang, T., Wang, S., Wang, C., Feizi, A., Suresh, A

Masry, A., Rodriguez, J. A., Zhang, T., Wang, S., Wang, C., Feizi, A., Suresh, A. K., Puri, A., Jian, X., No ¨el, P.- A., and others. Alignvlm: Bridging vision and language 11 WeightCLIP: Aligning Datasets and Models for Weight Space Learning latent spaces for multimodal understanding. InSecond Workshop on Representational Alignment at ICLR 2025,

2025
[16]

K., and Grewe, B

Nava, E., Kobayashi, S., Yin, Y ., Katzschmann, R. K., and Grewe, B. Meta-learning via classifier (-free) diffusion guidance.Transactions on Machine Learning Research, 2023(8),

2023
[17]

Oord, A. v. d., Li, Y ., and Vinyals, O. Representation learn- ing with contrastive predictive coding.arXiv preprint arXiv:1807.03748,

Pith/arXiv arXiv
[18]

Styleclip: Text-driven manipulation of stylegan imagery

Patashnik, O., Wu, Z., Shechtman, E., Cohen-Or, D., and Lischinski, D. Styleclip: Text-driven manipulation of stylegan imagery. InProceedings of the IEEE/CVF inter- national conference on computer vision, pp. 2085–2094,

2085
[19]

Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M

doi: 10.1016/j.compbiomed.2021.104319. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3,

doi:10.1016/j.compbiomed.2021.104319 2021
[20]

2018.00011

doi: 10.1109/WACV . 2018.00011. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Om- mer, B. High-Resolution Image Synthesis with Latent Diffusion Models. InIEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR),

doi:10.1109/wacv 2018
[21]

Hyper-Representations for Pre-Training and Transfer Learning

Sch¨urholt, K., Knyazev, B., Gir ´o-i Nieto, X., and Borth, D. Hyper-Representations for Pre-Training and Transfer Learning. InFirst Workshop of Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022, 2022a. Sch¨urholt, K., Knyazev, B., Gir ´o-i Nieto, X., and Borth, D. Hyper-Representations as Generative Models: Sam- pling Unseen Neural Net...

2022
[22]

2015.2496264

doi: 10.1109/TBME. 2015.2496264. Tian, B., Chen, W., Li, Z., Lai, S., Wu, J., and Yue, Y . Text2weight: Bridging natural language and neural net- work weight spaces. InProceedings of the 33rd ACM In- ternational Conference on Multimedia, pp. 10152–10160,

doi:10.1109/tbme 2015
[23]

Predicting neural network accuracy from weights.arXiv preprint arXiv:2002.11448,

Unterthiner, T., Keysers, D., Gelly, S., Bousquet, O., and Tolstikhin, I. Predicting neural network accuracy from weights.arXiv preprint arXiv:2002.11448,

Pith/arXiv arXiv 2002
[24]

and Mahadevan, S

Wang, C. and Mahadevan, S. A General Framework for Manifold Alignment. InAAAI Conference on Artificial Intelligence (AAAI), 2009a. 12 WeightCLIP: Aligning Datasets and Models for Weight Space Learning Wang, C. and Mahadevan, S. Manifold Alignment without Correspondence. InInternational Joint Conference on Artificial Intelligence (IJCAI), 2009b. Wang, K., ...

Pith/arXiv arXiv
[25]

Additional Related Work A.1

13 WeightCLIP: Aligning Datasets and Models for Weight Space Learning A. Additional Related Work A.1. Weight-space learning Representation learning in the space of neural-network weights has grown into a broad field spanning both model analysis and weight generation. Early work focuses on predicting model properties, such as accuracy, directly from weight...

2020
[26]

Instances

meta-dataset collection, but re- stricted to 20 training datasets for the CNN zoo and 10 for the ResNet18 zoo. As in TANS, we cap datasets to at most 20 classes during zoo construction. When available, we cite the original dataset papers for the Kaggle sources, including Aerial Cactus (L´opez-Jim´enez et al., 2019), BreakHis (Span- hol et al., 2016), Numt...

2019
[27]

Since all models within each zoo share the same architecture, the topology encoding used in the original OFA-based TANS setup is uninformative and is removed

to our uniform- architecture model zoos. Since all models within each zoo share the same architecture, the topology encoding used in the original OFA-based TANS setup is uninformative and is removed. Each model is instead represented only by the functional embedding, computed by passing a fixed bank of Gaussian noise images through the model and flattenin...

2024

[1] [1]

S., Willmott, D., Bair, A., Ganesh, M

Akinwande, V ., Norouzzadeh, M. S., Willmott, D., Bair, A., Ganesh, M. R., and Kolter, J. Z. Hyperclip: Adapt- ing vision-language models with hypernetworks.arXiv preprint arXiv:2412.16777,

Pith/arXiv arXiv

[2] [2]

M., and Humayun, A

Alam, S., Reasat, T., Doha, R. M., and Humayun, A. I. NumtaDB: Assembled bengali handwritten digits.arXiv preprint arXiv:1806.02452,

Pith/arXiv arXiv

[3] [3]

Text2model: Text-based model in- duction for zero-shot image classification

Amosy, O., V olk, T., Shapira, E., Ben-David, E., Reichart, R., and Chechik, G. Text2model: Text-based model in- duction for zero-shot image classification. InFindings of the Association for Computational Linguistics: EMNLP 2024, pp. 155–172,

2024

[4] [4]

Bedionita, S., Andreis, B., Chong, S., and Hwang, S. J. Instruction-guided autoregressive neural network parame- ter generation.ICLR 2025 Workshop on Neural Network Weights as a New Data Modality, 2025a. Bedionita, S., Andreis, B., Lee, H., Jeong, W., Chong, S., Hutter, F., and Hwang, S. J. Diffusion-based Neural Network Weights Generation. In13th Interna...

2025

[5] [5]

D’Orazio, A., Briglia, M

doi: 10.1109/ACCESS.2020.3010287. D’Orazio, A., Briglia, M. R., Crisostomi, D., Loi, D., Rodol`a, E., and Masi, I. Implicit inversion turns CLIP into a decoder. InThe Fourteenth International Confer- ence on Learning Representations,

doi:10.1109/access.2020.3010287 2020

[6] [6]

Falk, D., Sch ¨urholt, K., Tzevelekakis, K., Meynent, L., and Borth, D

arXiv preprint. Falk, D., Sch ¨urholt, K., Tzevelekakis, K., Meynent, L., and Borth, D. Learning Model Representations Us- ing Publicly Available Model Hubs.arXiv preprint arXiv:2510.02096,

arXiv

[7] [7]

Han, X., Wang, Z., Zhao, B., Zhang, B., Li, J., Borth, D., Yu, R., Maron, H., Ye, Y ., Yin, L., et al

doi: 10.1145/1015330.1015376. Han, X., Wang, Z., Zhao, B., Zhang, B., Li, J., Borth, D., Yu, R., Maron, H., Ye, Y ., Yin, L., et al. A survey of weight space learning: Understanding, representation, and generation.arXiv preprint arXiv:2603.10090,

doi:10.1145/1015330.1015376

[8] [8]

X., and Borth, D

Hanna, J., Falk, D., Yu, S. X., and Borth, D. Geosane: Learning geospatial representations from models, not data.arXiv preprint arXiv:2603.23408,

arXiv

[9] [9]

Ho, J., Jain, A., and Abbeel, P

doi: 10.1109/JSTARS.2019.2918242. Ho, J., Jain, A., and Abbeel, P. Denoising Diffusion Prob- abilistic Models. InAdvances in Neural Information Processing Systems (NeurIPS),

doi:10.1109/jstars.2019.2918242 2019

[10] [10]

Version 1.3.1

doi: 10.13026/4nae-zg36. Version 1.3.1. Hssayeni, M. D., Croock, M. S., Salman, A. D., Al-khafaji, H. F., Yahya, Z. A., and Ghoraani, B. Intracranial hem- orrhage segmentation using deep convolutional model. Data, 5(1):14,

work page doi:10.13026/4nae-zg36

[11] [11]

Jeong, W., Lee, H., Park, G., Hyung, E., Baek, J., and Hwang, S

doi: 10.3390/data5010014. Jeong, W., Lee, H., Park, G., Hyung, E., Baek, J., and Hwang, S. J. Task-adaptive neural network search with meta-contrastive learning.Advances in Neural Informa- tion Processing Systems, 34:21310–21324,

work page doi:10.3390/data5010014

[12] [12]

Conditional lora parameter generation.arXiv preprint arXiv:2408.01415,

Jin, X., Wang, K., Tang, D., Zhao, W., Zhou, Y ., Tang, J., and You, Y . Conditional lora parameter generation.arXiv preprint arXiv:2408.01415,

Pith/arXiv arXiv

[13] [13]

Accelerating training with neuron interaction and nowcasting networks

Knyazev, B., Moudgil, A., Lajoie, G., Belilovsky, E., and Lacoste-Julien, S. Accelerating training with neuron interaction and nowcasting networks. InInternational Conference on Learning Representations, volume 2025, pp. 29569–29588,

2025

[14] [14]

Mahoney, M

doi: 10.1016/j.ecoinf.2019.05.005. Mahoney, M. and Martin, C. Traditional and heavy tailed self regularization in neural network models. InInterna- tional Conference on Machine Learning, pp. 4284–4293. PMLR,

work page doi:10.1016/j.ecoinf.2019.05.005 2019

[15] [15]

A., Zhang, T., Wang, S., Wang, C., Feizi, A., Suresh, A

Masry, A., Rodriguez, J. A., Zhang, T., Wang, S., Wang, C., Feizi, A., Suresh, A. K., Puri, A., Jian, X., No ¨el, P.- A., and others. Alignvlm: Bridging vision and language 11 WeightCLIP: Aligning Datasets and Models for Weight Space Learning latent spaces for multimodal understanding. InSecond Workshop on Representational Alignment at ICLR 2025,

2025

[16] [16]

K., and Grewe, B

Nava, E., Kobayashi, S., Yin, Y ., Katzschmann, R. K., and Grewe, B. Meta-learning via classifier (-free) diffusion guidance.Transactions on Machine Learning Research, 2023(8),

2023

[17] [17]

Oord, A. v. d., Li, Y ., and Vinyals, O. Representation learn- ing with contrastive predictive coding.arXiv preprint arXiv:1807.03748,

Pith/arXiv arXiv

[18] [18]

Styleclip: Text-driven manipulation of stylegan imagery

Patashnik, O., Wu, Z., Shechtman, E., Cohen-Or, D., and Lischinski, D. Styleclip: Text-driven manipulation of stylegan imagery. InProceedings of the IEEE/CVF inter- national conference on computer vision, pp. 2085–2094,

2085

[19] [19]

Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M

doi: 10.1016/j.compbiomed.2021.104319. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3,

doi:10.1016/j.compbiomed.2021.104319 2021

[20] [20]

2018.00011

doi: 10.1109/WACV . 2018.00011. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Om- mer, B. High-Resolution Image Synthesis with Latent Diffusion Models. InIEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR),

doi:10.1109/wacv 2018

[21] [21]

Hyper-Representations for Pre-Training and Transfer Learning

Sch¨urholt, K., Knyazev, B., Gir ´o-i Nieto, X., and Borth, D. Hyper-Representations for Pre-Training and Transfer Learning. InFirst Workshop of Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022, 2022a. Sch¨urholt, K., Knyazev, B., Gir ´o-i Nieto, X., and Borth, D. Hyper-Representations as Generative Models: Sam- pling Unseen Neural Net...

2022

[22] [22]

2015.2496264

doi: 10.1109/TBME. 2015.2496264. Tian, B., Chen, W., Li, Z., Lai, S., Wu, J., and Yue, Y . Text2weight: Bridging natural language and neural net- work weight spaces. InProceedings of the 33rd ACM In- ternational Conference on Multimedia, pp. 10152–10160,

doi:10.1109/tbme 2015

[23] [23]

Predicting neural network accuracy from weights.arXiv preprint arXiv:2002.11448,

Unterthiner, T., Keysers, D., Gelly, S., Bousquet, O., and Tolstikhin, I. Predicting neural network accuracy from weights.arXiv preprint arXiv:2002.11448,

Pith/arXiv arXiv 2002

[24] [24]

and Mahadevan, S

Wang, C. and Mahadevan, S. A General Framework for Manifold Alignment. InAAAI Conference on Artificial Intelligence (AAAI), 2009a. 12 WeightCLIP: Aligning Datasets and Models for Weight Space Learning Wang, C. and Mahadevan, S. Manifold Alignment without Correspondence. InInternational Joint Conference on Artificial Intelligence (IJCAI), 2009b. Wang, K., ...

Pith/arXiv arXiv

[25] [25]

Additional Related Work A.1

13 WeightCLIP: Aligning Datasets and Models for Weight Space Learning A. Additional Related Work A.1. Weight-space learning Representation learning in the space of neural-network weights has grown into a broad field spanning both model analysis and weight generation. Early work focuses on predicting model properties, such as accuracy, directly from weight...

2020

[26] [26]

Instances

meta-dataset collection, but re- stricted to 20 training datasets for the CNN zoo and 10 for the ResNet18 zoo. As in TANS, we cap datasets to at most 20 classes during zoo construction. When available, we cite the original dataset papers for the Kaggle sources, including Aerial Cactus (L´opez-Jim´enez et al., 2019), BreakHis (Span- hol et al., 2016), Numt...

2019

[27] [27]

Since all models within each zoo share the same architecture, the topology encoding used in the original OFA-based TANS setup is uninformative and is removed

to our uniform- architecture model zoos. Since all models within each zoo share the same architecture, the topology encoding used in the original OFA-based TANS setup is uninformative and is removed. Each model is instead represented only by the functional embedding, computed by passing a fixed bank of Gaussian noise images through the model and flattenin...

2024