REVIEW 1 cited by
Training and Generating Neural Networks in Compressed Weight Space
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
The inputs and/or outputs of some neural nets are weight matrices of other neural nets. Indirect encodings or end-to-end compression of weight matrices could help to scale such approaches. Our goal is to open a discussion on this topic, starting with recurrent neural networks for character-level language modelling whose weight matrices are encoded by the discrete cosine transform. Our fast weight version thereof uses a recurrent neural network to parameterise the compressed weights. We present experimental results on the enwik8 dataset.
Forward citations
Cited by 1 Pith paper
-
Revisiting LoRA through the Lens of Parameter Redundancy: Spectral Encoding Helps
SeLoRA reparameterizes LoRA updates as inverse Fourier or wavelet transforms of sparsely masked spectral coefficients, improving fine-tuning accuracy on LLaMA models with fewer trainable parameters.
Discussion (0). Continue with ORCID to comment.