{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2025:7F3HT4XHWON3TP5E7FPLKHV3OV","short_pith_number":"pith:7F3HT4XH","schema_version":"1.0","canonical_sha256":"f97679f2e7b39bb9bfa4f95eb51ebb754da148564a5076fb37f1d01fbf0faf9a","source":{"kind":"arxiv","id":"2504.13161","version":2},"attestation_state":"computed","paper":{"title":"Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Dan Su, Hongxu Yin, Jan Kautz, Markus Kliegl, Mostofa Patwary, Pavlo Molchanov, Peter Belcak, Shizhe Diao, Xin Dong, Yingyan Lin, Yonggan Fu, Yoshi Suhara, Yu Yang, Zijia Chen","submitted_at":"2025-04-17T17:58:13Z","abstract_excerpt":"Pre-training datasets are typically collected from web content and lack inherent domain divisions. For instance, widely used datasets like Common Crawl do not include explicit domain labels, while manually curating labeled datasets such as The Pile is labor-intensive. Consequently, identifying an optimal pre-training data mixture remains a challenging problem, despite its significant benefits for pre-training performance. To address these challenges, we propose CLustering-based Iterative Data Mixture Bootstrapping (Nemotron-CLIMB), an automated framework that discovers, evaluates, and refines "},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":false,"formal_links_present":false},"canonical_record":{"source":{"id":"2504.13161","kind":"arxiv","version":2},"metadata":{"license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","primary_cat":"cs.CL","submitted_at":"2025-04-17T17:58:13Z","cross_cats_sorted":[],"title_canon_sha256":"ea3e64743b38ca5e47e86c70d06b4182138b1965508bef33424aba9ba4106691","abstract_canon_sha256":"beeeb5700d355783df460d2f39a7b42479fc06e472e98e25be418b70f9bf51ec"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-06-23T20:14:13.772171Z","signature_b64":"wtqB/5aEEp9k/1JwTRu5cQFo3ihd1fL6nHROfpEXQXToRg1n9eOxryBCZKEZNmq8liiI5ByHHRssgXt5U5WuDA==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"f97679f2e7b39bb9bfa4f95eb51ebb754da148564a5076fb37f1d01fbf0faf9a","last_reissued_at":"2026-06-23T20:14:13.770636Z","signature_status":"signed_v1","first_computed_at":"2026-06-23T20:14:13.770636Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Dan Su, Hongxu Yin, Jan Kautz, Markus Kliegl, Mostofa Patwary, Pavlo Molchanov, Peter Belcak, Shizhe Diao, Xin Dong, Yingyan Lin, Yonggan Fu, Yoshi Suhara, Yu Yang, Zijia Chen","submitted_at":"2025-04-17T17:58:13Z","abstract_excerpt":"Pre-training datasets are typically collected from web content and lack inherent domain divisions. For instance, widely used datasets like Common Crawl do not include explicit domain labels, while manually curating labeled datasets such as The Pile is labor-intensive. Consequently, identifying an optimal pre-training data mixture remains a challenging problem, despite its significant benefits for pre-training performance. To address these challenges, we propose CLustering-based Iterative Data Mixture Bootstrapping (Nemotron-CLIMB), an automated framework that discovers, evaluates, and refines "},"claims":{"count":0,"items":[],"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"source":{"id":"2504.13161","kind":"arxiv","version":2},"verdict":{"id":null,"model_set":{},"created_at":null,"strongest_claim":"","one_line_summary":"","pipeline_version":null,"weakest_assumption":"","pith_extraction_headline":""},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2504.13161/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2504.13161","created_at":"2026-06-23T20:14:13.770705+00:00"},{"alias_kind":"arxiv_version","alias_value":"2504.13161v2","created_at":"2026-06-23T20:14:13.770705+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2504.13161","created_at":"2026-06-23T20:14:13.770705+00:00"},{"alias_kind":"pith_short_12","alias_value":"7F3HT4XHWON3","created_at":"2026-06-23T20:14:13.770705+00:00"},{"alias_kind":"pith_short_16","alias_value":"7F3HT4XHWON3TP5E","created_at":"2026-06-23T20:14:13.770705+00:00"},{"alias_kind":"pith_short_8","alias_value":"7F3HT4XH","created_at":"2026-06-23T20:14:13.770705+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":16,"internal_anchor_count":16,"sample":[{"citing_arxiv_id":"2607.02266","citing_title":"HERMES: A Multi-Granularity Labeling Substrate for Pre-training Data Mixtures","ref_index":4,"is_internal_anchor":true},{"citing_arxiv_id":"2606.28551","citing_title":"DataComp-VLM: Improved Open Datasets for Vision-Language Models","ref_index":59,"is_internal_anchor":true},{"citing_arxiv_id":"2606.03773","citing_title":"KletterMix: Climbing Toward High-Quality German Pretraining Data","ref_index":11,"is_internal_anchor":true},{"citing_arxiv_id":"2606.02780","citing_title":"Do Value Vectors in Deep Layers Need Context from the Residual Stream?","ref_index":72,"is_internal_anchor":true},{"citing_arxiv_id":"2606.28551","citing_title":"DataComp-VLM: Improved Open Datasets for Vision-Language Models","ref_index":59,"is_internal_anchor":true},{"citing_arxiv_id":"2605.18528","citing_title":"Scale-Invariant Neural Network Optimization: Norm Geometry and Heavy-Tailed Noise","ref_index":30,"is_internal_anchor":true},{"citing_arxiv_id":"2605.21699","citing_title":"X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillation","ref_index":12,"is_internal_anchor":true},{"citing_arxiv_id":"2605.22705","citing_title":"Tokenization with Split Trees","ref_index":40,"is_internal_anchor":true},{"citing_arxiv_id":"2605.20798","citing_title":"Most Transformer Modifications Still Do Not Transfer at 1-3B: A 2020-2026 Update to Narang et al. (2021) with Downstream Evaluation and a Noise Floor","ref_index":79,"is_internal_anchor":true},{"citing_arxiv_id":"2605.17923","citing_title":"AdaptiveLoad: Towards Efficient Video Diffusion Transformer Training","ref_index":19,"is_internal_anchor":true},{"citing_arxiv_id":"2605.12715","citing_title":"Scaling Laws for Mixture Pretraining Under Data Constraints","ref_index":9,"is_internal_anchor":true},{"citing_arxiv_id":"2507.15640","citing_title":"Data Mixing Agent: Learning to Re-weight Domains for Continual Pre-training","ref_index":6,"is_internal_anchor":true},{"citing_arxiv_id":"2506.02153","citing_title":"Small Language Models are the Future of Agentic AI","ref_index":21,"is_internal_anchor":true},{"citing_arxiv_id":"2603.08022","citing_title":"Capacity-Aware Mixture Law Enables Efficient LLM Data Optimization","ref_index":10,"is_internal_anchor":true},{"citing_arxiv_id":"2604.16380","citing_title":"Data Mixing for Large Language Models Pretraining: A Survey and Outlook","ref_index":64,"is_internal_anchor":true},{"citing_arxiv_id":"2604.08366","citing_title":"Scaling-Aware Data Selection for End-to-End Autonomous Driving Systems","ref_index":15,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":0,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/7F3HT4XHWON3TP5E7FPLKHV3OV","json":"https://pith.science/pith/7F3HT4XHWON3TP5E7FPLKHV3OV.json","graph_json":"https://pith.science/api/pith-number/7F3HT4XHWON3TP5E7FPLKHV3OV/graph.json","events_json":"https://pith.science/api/pith-number/7F3HT4XHWON3TP5E7FPLKHV3OV/events.json","paper":"https://pith.science/paper/7F3HT4XH"},"agent_actions":{"view_html":"https://pith.science/pith/7F3HT4XHWON3TP5E7FPLKHV3OV","download_json":"https://pith.science/pith/7F3HT4XHWON3TP5E7FPLKHV3OV.json","view_paper":"https://pith.science/paper/7F3HT4XH","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2504.13161&json=true","fetch_graph":"https://pith.science/api/pith-number/7F3HT4XHWON3TP5E7FPLKHV3OV/graph.json","fetch_events":"https://pith.science/api/pith-number/7F3HT4XHWON3TP5E7FPLKHV3OV/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/7F3HT4XHWON3TP5E7FPLKHV3OV/action/timestamp_anchor","attest_storage":"https://pith.science/pith/7F3HT4XHWON3TP5E7FPLKHV3OV/action/storage_attestation","attest_author":"https://pith.science/pith/7F3HT4XHWON3TP5E7FPLKHV3OV/action/author_attestation","sign_citation":"https://pith.science/pith/7F3HT4XHWON3TP5E7FPLKHV3OV/action/citation_signature","submit_replication":"https://pith.science/pith/7F3HT4XHWON3TP5E7FPLKHV3OV/action/replication_record"}},"created_at":"2026-06-23T20:14:13.770705+00:00","updated_at":"2026-06-23T20:14:13.770705+00:00"}