REVIEW 4 cited by
A Closer Look at Memorization in Deep Networks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We examine the role of memorization in deep learning, drawing connections to capacity, generalization, and adversarial robustness. While deep networks are capable of memorizing noise data, our results suggest that they tend to prioritize learning simple patterns first. In our experiments, we expose qualitative differences in gradient-based optimization of deep neural networks (DNNs) on noise vs. real data. We also demonstrate that for appropriately tuned explicit regularization (e.g., dropout) we can degrade DNN training performance on noise datasets without compromising generalization on real data. Our analysis suggests that the notions of effective capacity which are dataset independent are unlikely to explain the generalization performance of deep networks when trained with gradient based methods because training data itself plays an important role in determining the degree of memorization.
Forward citations
Cited by 4 Pith papers
-
How much do language models memorize?
A compression-based measurement puts GPT-style model memorization capacity at roughly 3.6 bits per parameter, with membership inference success following a sigmoid in the dataset-to-capacity ratio.
-
Calibrating Pre-trained Language Classifiers on LLM-generated Noisy Labels via Iterative Refinement
SiDyP improves classifiers trained on LLM-generated noisy labels by retrieving likely true labels from embedding-space neighbors and iteratively refining them with a simplex diffusion model, reporting average gains of...
-
Feature learning is decoupled from generalization in high capacity neural networks
Current feature learning measures quantify the magnitude of representation change, which the authors argue is decoupled from the generalization benefit that neural networks show over their neural tangent kernel.
-
An overview of condensation phenomenon in deep learning
Neural networks exhibit condensation of neurons into clusters with similar outputs whose number increases monotonically during training, facilitated by small initializations or dropout, providing insights into general...
Discussion (0). Continue with ORCID to comment.