Not all layers are equally as important: Every Layer Counts BERT

David Samuel; Lucas Georges Gabriel Charpentier

Not all layers are equally as important: Every Layer Counts BERT

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2311.02265 v2 pith:GPSAMNHX submitted 2023-11-03 cs.CL

Lucas Georges Gabriel Charpentier , David Samuel This is my paper · ORCID

classification cs.CL

keywords layersequallyimportantlayermodificationtransformerallowsapproach

verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments

read the original abstract

This paper introduces a novel modification of the transformer architecture, tailored for the data-efficient pretraining of language models. This aspect is evaluated by participating in the BabyLM challenge, where our solution won both the strict and strict-small tracks. Our approach allows each transformer layer to select which outputs of previous layers to process. The empirical results verify the potential of this simple modification and show that not all layers are equally as important.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

AntLM: Bridging Causal and Masked Language Models
cs.CL 2024-12 conditional novelty 3.0 of 10

Alternating causal and masked language modeling objectives during pretraining yields 1 to 2 percent macro-average gains on BabyLM small-data benchmarks.

Pith tools