Pith. sign in

REVIEW 3 cited by

Pretraining with Artificial Language: Studying Transferable Knowledge in Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2203.10326 v2 pith:T7EWUEC4 submitted 2022-03-19 cs.CL

Ryokan Ri , Yoshimasa Tsuruoka This is my paper · ORCID
classification cs.CL
keywords languageknowledgenaturalartificialencoderstransferablelanguagesmodels
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We investigate what kind of structural knowledge learned in neural network encoders is transferable to processing natural language. We design artificial languages with structural properties that mimic natural language, pretrain encoders on the data, and see how much performance the encoder exhibits on downstream tasks in natural language. Our experimental results show that pretraining with an artificial language with a nesting dependency structure provides some knowledge transferable to natural language. A follow-up probing analysis indicates that its success in the transfer is related to the amount of encoded contextual information and what is transferred is the knowledge of position-aware context dependence of language. Our results provide insights into how neural network encoders process human languages and the source of cross-lingual transferability of recent multilingual language models.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Different procedural pretraining tasks create complementary, transferable structures in a transformer's attention and MLP weights, and structures from different tasks can be combined into one initialization.

  2. Procedural Pretraining: Warming Up Language Models with Abstract Data

    cs.CL 2026-01 conditional novelty 5.0 of 10

    A short warm-up on procedural data (brackets, sorting, sets) makes language models more accurate and more data-efficient on language, code, and informal math.

  3. Transfer of Structural Knowledge from Synthetic Languages

    cs.CL 2025-05 conditional novelty 5.0 of 10

    A new synthetic language, flat_shuffle, transfers more structure to English fine-tuning than earlier synthetic bracket languages, though still far short of training on English from scratch.

Pith tools