{"paper":{"title":"Investigation into In-Context Learning Capabilities of Transformers","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Transformers succeed at in-context binary classification on Gaussian mixtures under specific alignments of dimension, example count, and task diversity.","cross_cats":["cs.AI"],"primary_cat":"cs.LG","authors_text":"Arya Mazumdar, Leo Bangayan, Rushil Chandrupatla, Sebastian Leng","submitted_at":"2026-04-28T16:57:55Z","abstract_excerpt":"Transformers have demonstrated a strong ability for in-context learning (ICL), enabling models to solve previously unseen tasks using only example input output pairs provided at inference time. While prior theoretical work has established conditions under which transformers can perform linear classification in-context, the empirical scaling behavior governing when this mechanism succeeds remains insufficiently characterized.\n  In this paper, we conduct a systematic empirical study of in-context learning for Gaussian-mixture binary classification tasks. Building on the theoretical framework of "},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"Through extensive sweeps across dimensionality, sequence length, task diversity, and signal-to-noise regimes, we identify the parameter regions in which benign overfitting arises and characterize how it depends on data geometry and training exposure.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"The linear in-context classifier formulation and controlled synthetic Gaussian-mixture setup isolate the geometric conditions under which models successfully infer task structure from context alone.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"Systematic sweeps show in-context test accuracy for Gaussian-mixture classification depends on input dimension, number of examples, and pre-training task count, with benign overfitting appearing in specific geometry and noise regimes.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Transformers succeed at in-context binary classification on Gaussian mixtures under specific alignments of dimension, example count, and task diversity.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"5aa74a7984f288dbd1f5a5cd14073e1f21bf11d20907f113df9cdea929ae16f9"},"source":{"id":"2604.25858","kind":"arxiv","version":2},"verdict":{"id":"77684d2a-c43d-408e-ad21-ce101aeaf7b3","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-07T16:25:42.888449Z","strongest_claim":"Through extensive sweeps across dimensionality, sequence length, task diversity, and signal-to-noise regimes, we identify the parameter regions in which benign overfitting arises and characterize how it depends on data geometry and training exposure.","one_line_summary":"Systematic sweeps show in-context test accuracy for Gaussian-mixture classification depends on input dimension, number of examples, and pre-training task count, with benign overfitting appearing in specific geometry and noise regimes.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"The linear in-context classifier formulation and controlled synthetic Gaussian-mixture setup isolate the geometric conditions under which models successfully infer task structure from context alone.","pith_extraction_headline":"Transformers succeed at in-context binary classification on Gaussian mixtures under specific alignments of dimension, example count, and task diversity."},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2604.25858/integrity.json","findings":[],"available":true,"detectors_run":[{"name":"doi_compliance","ran_at":"2026-05-19T20:43:09.338997Z","status":"completed","version":"1.0.0","findings_count":0}],"snapshot_sha256":"acafd5c73cdedff691191455c1584ae6cea5d0f339f8d3c848563496f58bebc9"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}