{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2026:Y4V6HMACAJD67ZG4JGLUWEGRKY","short_pith_number":"pith:Y4V6HMAC","schema_version":"1.0","canonical_sha256":"c72be3b0020247efe4dc49974b10d1563dc8e33e016149d2e6e10327808b9913","source":{"kind":"arxiv","id":"2605.13290","version":1},"attestation_state":"computed","paper":{"title":"What properties of reasoning supervision are associated with improved downstream model quality?","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Intrinsic metrics on reasoning data strongly predict downstream model performance in a scale-dependent way.","cross_cats":[],"primary_cat":"cs.AI","authors_text":"Dzmitry Pihulski, Jan Eliasz, Jan Koco\\'n, Maciej Piasecki, Micha{\\l} Rajkowski, Miko{\\l}aj Langner, Przemys{\\l}aw Kazienko, Teddy Ferdinan","submitted_at":"2026-05-13T10:04:38Z","abstract_excerpt":"Validating training data for reasoning models typically requires expensive trial-and-error fine-tuning cycles. In this work, we investigate whether the utility of a reasoning dataset can be reliably predicted prior to training using intrinsic data metrics. We propose a suite of quantitative measures and evaluate their predictive power by fine-tuning 8B and 11B models on semantically distinct variants of a Polish reasoning dataset. Our analysis reveals that these intrinsic metrics demonstrate strong and significant correlations with downstream model performance. Crucially, we find that the pred"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":true},"canonical_record":{"source":{"id":"2605.13290","kind":"arxiv","version":1},"metadata":{"license":"http://creativecommons.org/licenses/by/4.0/","primary_cat":"cs.AI","submitted_at":"2026-05-13T10:04:38Z","cross_cats_sorted":[],"title_canon_sha256":"953e1d82724d775a6928ad2fe96e76f53fc9a7250e47331bcf88840cc3f13822","abstract_canon_sha256":"066e71923d82a2f4dd5e76eb6faedb1a5f9f66947a96db1434281bc01d53e406"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-05-18T02:44:49.126441Z","signature_b64":"fvmq8c1Ux4ddgx6DR2b7exQPpOreY0vpzBcleGzGgBcUfIYhaEkpuHlzJYakkAs42Dy+87w2b5WDPBuAsCuyAQ==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"c72be3b0020247efe4dc49974b10d1563dc8e33e016149d2e6e10327808b9913","last_reissued_at":"2026-05-18T02:44:49.126010Z","signature_status":"signed_v1","first_computed_at":"2026-05-18T02:44:49.126010Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"What properties of reasoning supervision are associated with improved downstream model quality?","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Intrinsic metrics on reasoning data strongly predict downstream model performance in a scale-dependent way.","cross_cats":[],"primary_cat":"cs.AI","authors_text":"Dzmitry Pihulski, Jan Eliasz, Jan Koco\\'n, Maciej Piasecki, Micha{\\l} Rajkowski, Miko{\\l}aj Langner, Przemys{\\l}aw Kazienko, Teddy Ferdinan","submitted_at":"2026-05-13T10:04:38Z","abstract_excerpt":"Validating training data for reasoning models typically requires expensive trial-and-error fine-tuning cycles. In this work, we investigate whether the utility of a reasoning dataset can be reliably predicted prior to training using intrinsic data metrics. We propose a suite of quantitative measures and evaluate their predictive power by fine-tuning 8B and 11B models on semantically distinct variants of a Polish reasoning dataset. Our analysis reveals that these intrinsic metrics demonstrate strong and significant correlations with downstream model performance. Crucially, we find that the pred"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"Our analysis reveals that these intrinsic metrics demonstrate strong and significant correlations with downstream model performance. Crucially, we find that the predictors of utility are scale-dependent.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That the semantically distinct variants of a single Polish reasoning dataset are representative enough for the observed scale-dependent patterns to generalize to other languages, domains, and model families.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"Intrinsic data metrics predict reasoning dataset utility for model fine-tuning, with different predictors working best for smaller versus larger models.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Intrinsic metrics on reasoning data strongly predict downstream model performance in a scale-dependent way.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"7f0eb79a92eaab0f73b7ab1aaae158a6f28540d8356609464db8dbac00f15659"},"source":{"id":"2605.13290","kind":"arxiv","version":1},"verdict":{"id":"1fff5dc3-7360-4d54-aaab-d5224eb28b18","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-14T19:32:58.449379Z","strongest_claim":"Our analysis reveals that these intrinsic metrics demonstrate strong and significant correlations with downstream model performance. Crucially, we find that the predictors of utility are scale-dependent.","one_line_summary":"Intrinsic data metrics predict reasoning dataset utility for model fine-tuning, with different predictors working best for smaller versus larger models.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That the semantically distinct variants of a single Polish reasoning dataset are representative enough for the observed scale-dependent patterns to generalize to other languages, domains, and model families.","pith_extraction_headline":"Intrinsic metrics on reasoning data strongly predict downstream model performance in a scale-dependent way."},"references":{"count":40,"sample":[{"doi":"","year":2024,"title":"Bandarkar, L., et al.: The belebele benchmark: a parallel reading comprehension dataset in 122 language variants. In: ACL. pp. 749–775 (2024) 14 M. Langner et al","work_id":"3091917c-7714-465e-9941-18561a936ec8","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2025,"title":"Bercovich, A., et al.: Llama-nemotron: Efficient reasoning models (2025)","work_id":"79c13186-7c10-4942-a0c5-e37f4f47ee2e","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2025,"title":"A.et al.Global piqa: Evaluating physical commonsense reasoning across 100+ languages and cultures (2025)","work_id":"5856c7a6-6f3b-42c6-9514-56c1131cee21","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2024,"title":"In: Proceedings of SIGMOD","work_id":"1b11590f-9469-4da6-8e98-8553efac27bd","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2025,"title":"Reasoning Models Don't Always Say What They Think","work_id":"b9bdcbf5-9ae0-464c-b1a6-de04f85a6e33","ref_index":5,"cited_arxiv_id":"2505.05410","is_internal_anchor":true}],"resolved_work":40,"snapshot_sha256":"47b8fbb337353a491caa8c71e44764f4e4697465cb11ca88ba221d7c10c9eeba","internal_anchors":3},"formal_canon":{"evidence_count":2,"snapshot_sha256":"32ed1c712062ede0c407f0f3e6e1e84463501d7e7edcdd33fb9702a8ab84cf4b"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2605.13290","created_at":"2026-05-18T02:44:49.126077+00:00"},{"alias_kind":"arxiv_version","alias_value":"2605.13290v1","created_at":"2026-05-18T02:44:49.126077+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2605.13290","created_at":"2026-05-18T02:44:49.126077+00:00"},{"alias_kind":"pith_short_12","alias_value":"Y4V6HMACAJD6","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_16","alias_value":"Y4V6HMACAJD67ZG4","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_8","alias_value":"Y4V6HMAC","created_at":"2026-05-18T12:33:37.589309+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":0,"internal_anchor_count":0,"sample":[]},"formal_canon":{"evidence_count":2,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/Y4V6HMACAJD67ZG4JGLUWEGRKY","json":"https://pith.science/pith/Y4V6HMACAJD67ZG4JGLUWEGRKY.json","graph_json":"https://pith.science/api/pith-number/Y4V6HMACAJD67ZG4JGLUWEGRKY/graph.json","events_json":"https://pith.science/api/pith-number/Y4V6HMACAJD67ZG4JGLUWEGRKY/events.json","paper":"https://pith.science/paper/Y4V6HMAC"},"agent_actions":{"view_html":"https://pith.science/pith/Y4V6HMACAJD67ZG4JGLUWEGRKY","download_json":"https://pith.science/pith/Y4V6HMACAJD67ZG4JGLUWEGRKY.json","view_paper":"https://pith.science/paper/Y4V6HMAC","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2605.13290&json=true","fetch_graph":"https://pith.science/api/pith-number/Y4V6HMACAJD67ZG4JGLUWEGRKY/graph.json","fetch_events":"https://pith.science/api/pith-number/Y4V6HMACAJD67ZG4JGLUWEGRKY/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/Y4V6HMACAJD67ZG4JGLUWEGRKY/action/timestamp_anchor","attest_storage":"https://pith.science/pith/Y4V6HMACAJD67ZG4JGLUWEGRKY/action/storage_attestation","attest_author":"https://pith.science/pith/Y4V6HMACAJD67ZG4JGLUWEGRKY/action/author_attestation","sign_citation":"https://pith.science/pith/Y4V6HMACAJD67ZG4JGLUWEGRKY/action/citation_signature","submit_replication":"https://pith.science/pith/Y4V6HMACAJD67ZG4JGLUWEGRKY/action/replication_record"}},"created_at":"2026-05-18T02:44:49.126077+00:00","updated_at":"2026-05-18T02:44:49.126077+00:00"}