REVIEW 2 cited by
Estimating the Local Learning Coefficient at Scale
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The \textit{local learning coefficient} (LLC) is a principled way of quantifying model complexity, originally derived in the context of Bayesian statistics using singular learning theory (SLT). Several methods are known for numerically estimating the local learning coefficient, but so far these methods have not been extended to the scale of modern deep learning architectures or data sets. Using a method developed in {\tt arXiv:2308.12108 [stat.ML]} we empirically show how the LLC may be measured accurately and self-consistently for deep linear networks (DLNs) up to 100M parameters. We also show that the estimated LLC has the rescaling invariance that holds for the theoretical quantity.
Forward citations
Cited by 2 Pith papers
-
Structure Development in List-Sorting Transformers
In a list-sorting transformer, the mean and variance of gaps between adjacent sorted numbers determine whether attention heads split the vocabulary, suppress copying to calibrate, or switch off.
-
Singular leaning coefficients and efficiency in learning theory
The paper states log canonical threshold formulas for learning coefficients of deep linear, three-layer ReLU, and softmax networks, but the ReLU derivation is largely a sketch.
Discussion (0). Continue with ORCID to comment.