{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2021:KXAFCM2VO7PBIJTLXNHX5MACGW","short_pith_number":"pith:KXAFCM2V","schema_version":"1.0","canonical_sha256":"55c051335577de14266bbb4f7eb002359757fee86bef3164f2901ede54ddcd7f","source":{"kind":"arxiv","id":"2103.12028","version":4},"attestation_state":"computed","paper":{"title":"Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets","license":"http://creativecommons.org/licenses/by/4.0/","headline":"","cross_cats":["cs.AI"],"primary_cat":"cs.CL","authors_text":"Ahmed Baruwa, Ahsan Wahab, Alessia Battisti, Allahsera Tapo, Andr\\'e M\\\"uller, Andre Niyongabo Rubungo, Ankur Bapna, Annette Rios, Artem Sokolov, Ayanda Mnyakeni, Ayodele Awokoya, Beno\\^it Sagot, Bonaventure F. P. Dossou, Clara Rivera, Claytone Sikasote, Colin Leong, Daan van Esch, Duygu Ataman, Iroro Orife, Isaac Caswell, Isabel Papadimitriou, Israel Abebe Azime, Jamshidbek Mirzakhalov, Julia Kreutzer, Kelechi Ogueji, Lisa Wang, Mathias Jenny, Mathias M\\\"uller, Mofetoluwa Adeyemi, Monang Setyawan, Nanda Muhammad, Nasanbayar Ulzii-Orshikh, Nisansa de Silva, Nishant Subramani, Nze Lawson, Oghenefego Ahia, Orevaoghene Ahia, Orhan Firat, Pallavi Baljekar, Pedro Ortiz Suarez, Sakhile Dlamini, Sakine \\c{C}abuk Ball{\\i}, Salomey Osei, Shamsuddeen Hassan Muhammad, Sneha Kudugunta, Sokhar Samb, Stella Biderman, Supheakmungkol Sarin, Sweta Agrawal, Tapiwanashe Matangira, Toan Q. Nguyen, Yacine Jernite","submitted_at":"2021-03-22T17:30:33Z","abstract_excerpt":"With the success of large-scale pre-training and multilingual modeling in Natural Language Processing (NLP), recent years have seen a proliferation of large, web-mined text datasets covering hundreds of languages. We manually audit the quality of 205 language-specific corpora released with five major public datasets (CCAligned, ParaCrawl, WikiMatrix, OSCAR, mC4). Lower-resource corpora have systematic issues: At least 15 corpora have no usable text, and a significant fraction contains less than 50% sentences of acceptable quality. In addition, many are mislabeled or use nonstandard/ambiguous l"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":false,"formal_links_present":false},"canonical_record":{"source":{"id":"2103.12028","kind":"arxiv","version":4},"metadata":{"license":"http://creativecommons.org/licenses/by/4.0/","primary_cat":"cs.CL","submitted_at":"2021-03-22T17:30:33Z","cross_cats_sorted":["cs.AI"],"title_canon_sha256":"44b895c887010d9532b95616269de47cb9a459fb856a8674c96a0735090ee91a","abstract_canon_sha256":"fb93349e5cbaf7f4986e74e217895f8522738159273b672ee69e1ad957d8a0be"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-07-05T03:58:33.781250Z","signature_b64":"rVOlFPBFiH98VPocymmpvsuHFGnQCH+JO61YCdvxjLsZlt5FsBY+FMo28N2pRLm6e+1glDnhgN/jc/x9tB4oCA==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"55c051335577de14266bbb4f7eb002359757fee86bef3164f2901ede54ddcd7f","last_reissued_at":"2026-07-05T03:58:33.780761Z","signature_status":"signed_v1","first_computed_at":"2026-07-05T03:58:33.780761Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets","license":"http://creativecommons.org/licenses/by/4.0/","headline":"","cross_cats":["cs.AI"],"primary_cat":"cs.CL","authors_text":"Ahmed Baruwa, Ahsan Wahab, Alessia Battisti, Allahsera Tapo, Andr\\'e M\\\"uller, Andre Niyongabo Rubungo, Ankur Bapna, Annette Rios, Artem Sokolov, Ayanda Mnyakeni, Ayodele Awokoya, Beno\\^it Sagot, Bonaventure F. P. Dossou, Clara Rivera, Claytone Sikasote, Colin Leong, Daan van Esch, Duygu Ataman, Iroro Orife, Isaac Caswell, Isabel Papadimitriou, Israel Abebe Azime, Jamshidbek Mirzakhalov, Julia Kreutzer, Kelechi Ogueji, Lisa Wang, Mathias Jenny, Mathias M\\\"uller, Mofetoluwa Adeyemi, Monang Setyawan, Nanda Muhammad, Nasanbayar Ulzii-Orshikh, Nisansa de Silva, Nishant Subramani, Nze Lawson, Oghenefego Ahia, Orevaoghene Ahia, Orhan Firat, Pallavi Baljekar, Pedro Ortiz Suarez, Sakhile Dlamini, Sakine \\c{C}abuk Ball{\\i}, Salomey Osei, Shamsuddeen Hassan Muhammad, Sneha Kudugunta, Sokhar Samb, Stella Biderman, Supheakmungkol Sarin, Sweta Agrawal, Tapiwanashe Matangira, Toan Q. Nguyen, Yacine Jernite","submitted_at":"2021-03-22T17:30:33Z","abstract_excerpt":"With the success of large-scale pre-training and multilingual modeling in Natural Language Processing (NLP), recent years have seen a proliferation of large, web-mined text datasets covering hundreds of languages. We manually audit the quality of 205 language-specific corpora released with five major public datasets (CCAligned, ParaCrawl, WikiMatrix, OSCAR, mC4). Lower-resource corpora have systematic issues: At least 15 corpora have no usable text, and a significant fraction contains less than 50% sentences of acceptable quality. In addition, many are mislabeled or use nonstandard/ambiguous l"},"claims":{"count":0,"items":[],"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"source":{"id":"2103.12028","kind":"arxiv","version":4},"verdict":{"id":null,"model_set":{},"created_at":null,"strongest_claim":"","one_line_summary":"","pipeline_version":null,"weakest_assumption":"","pith_extraction_headline":""},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2103.12028/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2103.12028","created_at":"2026-07-05T03:58:33.780820+00:00"},{"alias_kind":"arxiv_version","alias_value":"2103.12028v4","created_at":"2026-07-05T03:58:33.780820+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2103.12028","created_at":"2026-07-05T03:58:33.780820+00:00"},{"alias_kind":"pith_short_12","alias_value":"KXAFCM2VO7PB","created_at":"2026-07-05T03:58:33.780820+00:00"},{"alias_kind":"pith_short_16","alias_value":"KXAFCM2VO7PBIJTL","created_at":"2026-07-05T03:58:33.780820+00:00"},{"alias_kind":"pith_short_8","alias_value":"KXAFCM2V","created_at":"2026-07-05T03:58:33.780820+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":4,"internal_anchor_count":0,"sample":[{"citing_arxiv_id":"2405.14782","citing_title":"Lessons from the Trenches on Reproducible Evaluation of Language Models","ref_index":46,"is_internal_anchor":false},{"citing_arxiv_id":"2304.01373","citing_title":"Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling","ref_index":153,"is_internal_anchor":false},{"citing_arxiv_id":"2112.04359","citing_title":"Ethical and social risks of harm from Language Models","ref_index":46,"is_internal_anchor":false},{"citing_arxiv_id":"2205.01068","citing_title":"OPT: Open Pre-trained Transformer Language Models","ref_index":251,"is_internal_anchor":false}]},"formal_canon":{"evidence_count":0,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/KXAFCM2VO7PBIJTLXNHX5MACGW","json":"https://pith.science/pith/KXAFCM2VO7PBIJTLXNHX5MACGW.json","graph_json":"https://pith.science/api/pith-number/KXAFCM2VO7PBIJTLXNHX5MACGW/graph.json","events_json":"https://pith.science/api/pith-number/KXAFCM2VO7PBIJTLXNHX5MACGW/events.json","paper":"https://pith.science/paper/KXAFCM2V"},"agent_actions":{"view_html":"https://pith.science/pith/KXAFCM2VO7PBIJTLXNHX5MACGW","download_json":"https://pith.science/pith/KXAFCM2VO7PBIJTLXNHX5MACGW.json","view_paper":"https://pith.science/paper/KXAFCM2V","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2103.12028&json=true","fetch_graph":"https://pith.science/api/pith-number/KXAFCM2VO7PBIJTLXNHX5MACGW/graph.json","fetch_events":"https://pith.science/api/pith-number/KXAFCM2VO7PBIJTLXNHX5MACGW/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/KXAFCM2VO7PBIJTLXNHX5MACGW/action/timestamp_anchor","attest_storage":"https://pith.science/pith/KXAFCM2VO7PBIJTLXNHX5MACGW/action/storage_attestation","attest_author":"https://pith.science/pith/KXAFCM2VO7PBIJTLXNHX5MACGW/action/author_attestation","sign_citation":"https://pith.science/pith/KXAFCM2VO7PBIJTLXNHX5MACGW/action/citation_signature","submit_replication":"https://pith.science/pith/KXAFCM2VO7PBIJTLXNHX5MACGW/action/replication_record"}},"created_at":"2026-07-05T03:58:33.780820+00:00","updated_at":"2026-07-05T03:58:33.780820+00:00"}