{"id":"a61a1fe7-d255-4c70-aa1e-4f325caf2da2","arxiv_id":"2506.04807","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"The paper introduces MegaHan97K, a 97,455-category Chinese character dataset and benchmark, and reports that existing OCR methods struggle with similar and zero-shot characters at this scale.","lead":"MegaHan97K is a new dataset with 97,455 Chinese character classes, built from handwritten, historical, and synthetic images, and it is the first to cover the full GB18030-2022 standard. Benchmarking nine OCR models on this scale shows closed-set accuracy around 92% at best, while zero-shot accuracy on never-seen characters falls to 45 to 79%.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The label-fidelity check is too weak relative to the central claim that all 97,455 categories are correctly labeled, and the reported accuracy numbers inherit this vulnerability.","rationale":"My review agrees with the reader's assessment: the dataset is valuable and the 97,455-category count is plausibly correct given the documented lexicon construction, but the weakest assumption is label fidelity, not category count. The strongest claim ('first dataset to fully support GB18030-2022' with 'balanced samples') depends on every image in the test set being correctly labeled. The paper's own Section 4.5 admits font synthesis errors on complex characters, and Section 3.2's 10% inspection is a weak guarantee for 3.3M synthetic images. The internal count inconsistency (Table 1 says 96,362 handwritten categories while the text says all 97,455) further supports caution, though it is addressable. I do not believe the central construction is broken; the concern is an empirically testable data-quality issue. A conditional verdict with a required label-error audit and corrected benchmark reporting is the right strength of claim for this evidence. The concrete test is a single, decisive, stratified audit that would settle whether the concern actually lands.","tokens_in":17839,"tokens_out":1397,"duration_ms":14079,"concrete_test":"Release the label-inspection protocol and run a stratified random audit: sample at least 500 images per subset (handwritten, Kangxi, M5HisDoc-reused, synthetic), oversampling complex characters (≥10 strokes) and categories with single samples, and have two independent annotators verify each image against its label. Compute per-subset label-error rates with 95% confidence intervals; if the synthetic-subset error rate or any rare/character subset exceeds 1%, re-report all benchmark accuracies with error-corrected labels and state the corrected figures.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that MegaHan97K covers 97,455 correctly labeled Chinese character categories and thereby yields trustworthy benchmark accuracies (92.32% closed-set, 79.04% zero-shot). The load-bearing premise is label fidelity across every subset: synthetically rendered glyphs must actually depict the character named by the label, the web-crawled Kangxi images must match their HanDian labels, and the volunteer handwriting must be correctly assigned to the prompted character. Section 3.2 states that only 10% of synthesized data was inspected, and Section 4.5 demonstrates that FontDiffuser produces stroke misplacement and missing details for complex characters. If label error concentrates in rare/complex categories, the benchmark accuracies are systematically inflated or deflated, and the 'fully supports GB18030-2022' claim is betrayed. The paper reports no per-subset label-error estimates and provides no release of the error-inspection protocol. This is not a mere long-tail imbalance issue; it is the fundamental data-quality guarantee. The reader's weakest_assumption identifies exactly this. The concern is concrete and testable, but the paper itself provides enough internal inconsistency (e.g., Table 1 shows 96,362 handwritten categories while the text says all 97,455 are covered) to treat it as a genuine soft spot rather than a manufactured caveat.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MegaHan97K, a dataset for Chinese character recognition containing 97,455 character categories, assembled from a handwritten subset (collected from 94 volunteers), a historical subset (M5HisDoc and web-crawled Kangxi Dictionary images), and a synthetic subset (rendered by FontDiffuser). The authors claim this is the first dataset to fully cover GB18030-2022 and the largest character-class dataset to date, and they benchmark nine methods in closed-set and zero-shot settings, reporting a best closed-set accuracy of 92.32% (HierCode) and a best zero-shot accuracy of 79.04% (CCR-CLIP). They also provide cross-validation experiments with existing datasets and an analysis of synthetic-data effects.","tokens_in":18160,"tokens_out":5305,"duration_ms":61062,"significance":"If the data-quality claims are validated, MegaHan97K would be a valuable community resource: it is the first mega-category Chinese character dataset, it covers the full GB18030-2022 standard, and its benchmark results quantify realistic challenges (storage growth, morphologically similar characters, and zero-shot difficulties) that smaller datasets cannot reveal. The paper ships a public release, benchmarks nine diverse methods, and includes cross-dataset generalization experiments, which are useful contributions. However, the central claims depend on label fidelity across 97,455 categories, and the manuscript provides only weak evidence for that fidelity while exhibiting internal inconsistencies in category counts and balance statistics. The dataset itself is promising, but the supporting evidence needs substantial strengthening before the headline claims can be accepted.","major_comments":[{"comment":"The label-fidelity check is insufficient relative to the central claim that all 97,455 categories are correctly labeled. Section 3.2 states that only 10% of the synthesized data was inspected, and Section 4.5 documents that FontDiffuser produces stroke misplacement and missing details for complex characters. Since the synthetic subset is used to train and test on the full lexicon, the reported accuracies (e.g., 92.32% in Table 8) presuppose that generated images actually depict their labels; similarly, the web-crawled Kangxi images and volunteer handwriting are assumed to match their labels. The paper reports no per-subset label-error rate, no stratification by character complexity (e.g., stroke count or IDS complexity), and no release of the inspection protocol. Please provide a label-fidelity audit: sample all three subsets, report error rates broken down by complexity, and either filter or relabel erroneous images or quantify the effect of label noise on the benchmark numbers.","section":"Section 3.2, Section 4.5"},{"comment":"The category counts are internally inconsistent. Table 1 reports Handwritten Original as 97,455 categories and Handwritten Augmented as 96,362, while Section 3.2 says the handwritten subset 'covers all 97,455 categories' and Table 5 (general test set) reports Original 96,362 and Augmented 97,455, i.e., the reverse. Table 4 lists the M5HisDoc training component with 9,369 categories but Table 5 lists 16,151 M5HisDoc test categories, and the total-category overlap structure among Original, Augmented, M5HisDoc, and Kangxi is never defined. Because the paper's first contribution is the exact number of covered categories and the claim of full GB18030-2022 coverage, these discrepancies must be reconciled and each subset's category count and overlap stated unambiguously.","section":"Tables 1, 3, 4, 5, 7 and Section 3.2"},{"comment":"The 'balanced samples' claim is contradicted by the Kangxi subset. Table 1 shows the Kangxi subset has 47,064 categories and 49,438 instances, i.e., about 1.05 images per category on average, and Table 7 shows the zero-shot test Kangxi portion has 24,476 categories and 26,111 instances. This is precisely the one- or two-sample long-tail regime that Section 2.1 criticizes in existing datasets. The long-tail analysis in Section 4.4 relies on macro accuracy, but a test set with many singleton categories can yield macro and top-1 accuracies that are close without the underlying category-frequency distribution being balanced. Please restrict the balance claim to the subsets that actually have multiple samples per category, or report per-subset category-frequency histograms for all subsets.","section":"Abstract, Section 3.3, Table 1, Table 7"},{"comment":"The dataset split is not writer-disjoint, which can inflate handwritten recognition accuracy. Section 4.1 describes splitting by character category only and does not state whether samples from the same volunteer are kept entirely in the training or test set. With 94 volunteers and roughly 900K handwritten samples, a random per-sample split will place the same writer's characters in both training and test sets. Additionally, all benchmark tables report single runs without error bars or repeated-seed statistics, so it is unclear whether the observed differences (e.g., the 31.36% gap between CCR-CLIP and SideNet-DDCM in Table 10) are significant. Please specify the writer-disjoint split, report the number of writers per split, and provide at least mean and standard deviation over multiple runs for the main methods.","section":"Section 4.1, Tables 8, 10, 12, 13"}],"minor_comments":[{"comment":"The name is misspelled as 'MetaHan97K' in the abstract and as 'MegeHan97K' in Section 3.2; please use 'MegaHan97K' consistently.","section":"Abstract, Section 3.2"},{"comment":"Section 4.2 defines morphologically similar characters as having an IDS edit distance of no more than three, while Section 4.5 says 'less than 3'; please unify the threshold.","section":"Section 4.2 vs Section 4.5"},{"comment":"Figure 3 calls HandWT-O and HandWT-A the pre-processed and post-processed versions of the handwritten subset, while Section 3.2 says the original and augmented parts are sourced from different real data; please clarify whether augmented samples are transformed copies of the original samples or independently collected real handwriting.","section":"Figure 3 caption and Section 3.2"},{"comment":"The sentence 'We extracted 10% of the synthesized data for sampling inspection' should specify whether the 10% was stratified by category and whether inspection was performed on individual images or on per-category batches.","section":"Section 3.2"},{"comment":"The data-collection year is listed as 2024 in Table 2, while the arXiv version is dated 2025; please update the year for consistency.","section":"Table 1, Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is sound and the resource is potentially important, but the load-bearing evidence on label fidelity, exact category counts, and balance needs to be materially improved before the manuscript can be accepted. The use of FontDiffuser, developed by the same group, to generate the synthetic training set is not circular by itself, but it strengthens the need for an independent label-fidelity audit. I would also encourage the editor to verify that the dataset release includes the promised images and labels if the paper is accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I've read this paper carefully. Bottom line: the scale is genuinely new—97,455 categories is a real jump from the prior 16,151, and the first full GB18030-2022 coverage is a legitimate enabling contribution. The benchmark results at this scale (92.32% closed-set best, 79.04% zero-shot best, storage growth, radical zero-shot failures) are new and useful.\n\nThe soft spots, in order of severity:\n\n1. Label fidelity is load-bearing and under-verified. Section 4.5 shows FontDiffuser produces stroke misplacement and missing details on complex characters, yet only 10% of synthetic data was inspected. Web-crawled Kangxi images and volunteer handwriting also lack per-subset error estimates. If label errors concentrate in rare/complex characters—where FontDiffuser struggles—both the coverage claim and every accuracy table are compromised. This is concrete and testable.\n\n2. Internal inconsistencies in category counts. Table 1 lists handwritten Original at 97,455 categories and Augmented at 96,362, but Table 5 lists Original at 96,362 and Augmented at 97,455. Text elsewhere says all 97,455 are covered. Not fatal, but it undermines confidence in the tables.\n\n3. No writer-disjoint split is stated for the 94 volunteers. Train/test contamination by writer would inflate handwriting accuracy.\n\n4. The balanced-sample claim is overstated. Kangxi has ~1 image per category for 47,064 categories, which is long-tailed by construction.\n\nMinor: single-run benchmarks without error bars; the \"largest in pattern recognition\" claim is unverifiable.\n\nNone of this breaks the central contribution. The dataset is likely to become a standard resource, and the issues are addressable. I'd send this to a serious referee. Before acceptance, the authors should provide a label-error audit, correct the tables, and state the writer-split policy.\n\nWould I cite it? Yes, if I worked on large-vocabulary OCR. Reading group: maybe, mainly for the benchmark findings.","headline":"A genuinely new scale for Chinese character datasets with real benchmark findings, but the label-fidelity guarantee is thinner than the central claim requires.","tokens_in":18643,"tokens_out":3163,"would_cite":true,"duration_ms":33888,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MegaHan97K provides 97,455 Chinese character categories, six times more than any previous dataset.","keywords":["Chinese character recognition","mega-category classification","GB18030-2022","zero-shot learning","optical character recognition","font synthesis","historical document analysis","dataset"],"falsifier":"Recruit expert annotators to inspect a random sample of categories from the synthetic and Kangxi subsets, comparing each image with the ground-truth character and estimating the label error rate separately for characters with many strokes; if the error rate is concentrated in rare or complex characters, the coverage claim and the reported accuracies are not reliable.","tokens_in":17647,"feed_emoji":"🀄","tokens_out":6530,"duration_ms":70857,"temperature":0.7,"pith_summary":"This paper introduces MegaHan97K, a dataset of 97,455 Chinese character categories assembled from handwritten, historical, and synthetic samples. The authors claim it is the largest such dataset, at least six times larger than any predecessor, and the first to cover the full GB18030-2022 standard of 87,887 categories plus additional Unicode characters. They argue that balanced per-category sampling removes the long-tail distribution problem that has made rare and variant characters difficult to study. Benchmarking shows state-of-the-art models reach 92.32% accuracy in closed-set recognition but only 79.04% in zero-shot recognition, which the authors present as evidence that mega-category recognition is a genuinely new challenge. A sympathetic reader would care because rare and archaic characters matter for digitizing historical documents, and no prior dataset allowed models to be trained or evaluated on this scale.","feed_headline":"A 97,455-class dataset opens mega-category Chinese OCR","feed_subtitle":"Balanced samples for 97,455 characters lift accuracy to 92.32 percent and expose zero-shot gaps.","key_machinery":"The central object is the dataset itself, organized as a balanced category-by-subset matrix. Its construction has three load-bearing components: a lexicon built from GB18030-2022 and Unicode with IDS-based deduplication; data acquisition through volunteer handwriting, historical corpora, and FontDiffuser-generated synthetic images; and train/test splits designed for both general and zero-shot recognition. FontDiffuser, a diffusion-based font generation model, supplies the synthetic samples that drive the largest accuracy gains, while the Ideographic Description Sequence representation provides the structural prior used for deduplication and similarity analysis.","core_discovery":"MegaHan97K is a dataset of 97,455 Chinese character categories, constructed by taking the union of the GB18030-2022 standard with Unicode 15.0.0 and removing ambiguous duplicate pairs through Ideographic Description Sequence comparison. It contains three subsets: a handwritten subset covering all categories, a historical subset built from M5HisDoc and Kangxi Dictionary images, and a synthetic subset of 35 FontDiffuser-generated samples per category. The authors report that the best closed-set model reaches 92.32% accuracy, the best zero-shot model reaches 79.04%, and that the dataset eliminates the long-tail problem that plagues earlier datasets. The paper's central claim is that this is the first dataset large enough to support mega-category Chinese character recognition, and that the performance gap between closed-set and zero-shot settings defines a new research problem.","pith_inferences":["A natural extension of the paper's logic is a continually expanding open-lexicon evaluation: as Unicode adds new codepoints, the zero-shot split can grow, and models can be reassessed without retraining on the new categories.","Because the authors measure confusion via IDS edit distance, the dataset could double as a glyph-similarity benchmark, letting other fine-grained recognition tasks borrow its error analysis.","The label-fidelity risk is asymmetric: if FontDiffuser silently corrupts rare characters, the coverage claim of 97,455 valid categories is at stake even though aggregate accuracy may move only slightly.","Combining MegaHan97K with text-line or page-level data, as the discussion suggests, could turn the character-level resource into a restoration tool for damaged ancient documents; that application is suggested by the paper but not implemented."],"forward_implications":["If the dataset is as claimed, researchers can train and evaluate Chinese-character recognizers on 97,455 classes, roughly six times the previous maximum, making mega-category OCR experimentally accessible.","Models trained on MegaHan97K transfer to existing handwriting and historical-document benchmarks, raising macro accuracy for rare characters on M5HisDoc, CASIA-HWDB, and CASIA-AHCDB.","The closed-set to zero-shot accuracy gap becomes a concrete benchmark target for methods that exploit radicals, strokes, or glyph-level priors.","The roughly 22-percentage-point average gain from adding synthetic data indicates that font synthesis is a viable data source, but its failures on complex and similar characters set a limit that further synthesis alone may not break."],"supporting_citations":[{"why":"Defines the 87,887-category GB18030-2022 standard that MegaHan97K claims to cover fully.","marker":"[10]"},{"why":"M5HisDoc is the largest prior Chinese-character dataset and supplies the historical subset's base samples and the long-tail comparison.","marker":"[15]"},{"why":"FontDiffuser is the diffusion model used to generate the 35 synthetic samples per category that drive the largest accuracy gains.","marker":"[42]"},{"why":"CCR-CLIP is the image-IDS alignment method that achieves the best zero-shot accuracy and is used in cross-validation.","marker":"[16]"},{"why":"HierCode is the method that achieves the best closed-set accuracy and anchors the long-tail analysis with macro accuracy.","marker":"[17]"},{"why":"CASIA-HWDB is the standard handwritten benchmark used as a cross-validation target and a comparison point for category counts.","marker":"[12]"}],"fun_headline_variants":["97,455 Chinese characters in one dataset for mega-category OCR","MegaHan97K: largest Chinese character dataset yet, 97K categories","New dataset breaks 97K Chinese character categories for OCR","Balanced 97K Chinese character dataset targets long-tail recognition","Mega-category Chinese OCR leaps to 97K classes with MegaHan97K"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every image in every category really shows the character named by its label, including the synthetic samples generated by FontDiffuser and the crawled Kangxi images; the paper checked only 10% of synthesized images and reports synthesis failures on complex characters.","fun_headline_variants_meta":{"raw":{"variants":["97,455 Chinese characters in one dataset for mega-category OCR","MegaHan97K: largest Chinese character dataset yet, 97K categories","New dataset breaks 97K Chinese character categories for OCR","Balanced 97K Chinese character dataset targets long-tail recognition","Mega-category Chinese OCR leaps to 97K classes with MegaHan97K"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000677,"raw_usage":{"total_tokens":3113,"prompt_tokens":1014,"completion_tokens":2099,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":630,"completion_tokens_details":{"reasoning_tokens":2004}},"tokens_in":630,"tokens_out":2099,"duration_ms":17836,"temperature":1.0,"reasoning_tokens":2004,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:33:19.743066+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recruit expert annotators to inspect a random sample of categories from the synthetic and Kangxi subsets, comparing each image with the ground-truth character and estimating the label error rate separately for characters with many strokes; if the error rate is concentrated in rare or complex characters, the coverage claim and the reported accuracies are not reliable.","supporting_citations":[{"cited_title":"of Industry, I","cited_arxiv_id":null,"evidence_quote":"Defines the 87,887-category GB18030-2022 standard that MegaHan97K claims to cover fully."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"M5HisDoc is the largest prior Chinese-character dataset and supplies the historical subset's base samples and the long-tail comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FontDiffuser is the diffusion model used to generate the 35 synthetic samples per category that drive the largest accuracy gains."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CCR-CLIP is the image-IDS alignment method that achieves the best zero-shot accuracy and is used in cross-validation."},{"cited_title":"Zhang, Y","cited_arxiv_id":null,"evidence_quote":"HierCode is the method that achieves the best closed-set accuracy and anchors the long-tail analysis with macro accuracy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CASIA-HWDB is the standard handwritten benchmark used as a cross-validation target and a comparison point for category counts."}],"review_version":1}