{"id":"e32712af-ca96-4026-a860-89e3a02601c5","arxiv_id":"2507.15655","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A handwritten bilingual VQA benchmark with about 1.5k pages and up to 4.8k question-answer pairs, plus baselines showing current models perform poorly, particularly with OCR-derived text.","lead":"This paper introduces HW-MLVQA, a benchmark for question answering on handwritten documents in English and Hindi, built by having volunteers copy public reading-comprehension texts by hand. It finds that current language and vision-language models perform poorly on this benchmark, especially when they must rely on OCR text.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth evidence boxes in HW-MLVQA rest on an OCR-based pipeline whose reported word accuracy is as low as 5.56%, and no reliability statistics are given for the manual correction step; the near-chance grounding result is therefore not yet established.","rationale":"The reader's weakest-assumption analysis correctly identifies the OCR-based annotation pipeline as the load-bearing risk, and I agree with the REJECT verdict. The paper's own Table 2 supplies the key damaging fact: EasyOCR word accuracy is 5.56% on English pages, so answer words are lost before bounding box generation. Even the better GoogleOCR is far below the level needed for reliable automatic matching. The manual verification step is asserted but not quantified, so the final labels remain an unverified assumption rather than demonstrated ground truth. The paper also contains unresolved internal contradictions in dataset counts (1,600 vs 1,520 pages; 2,400 vs 4,800 questions), and no download link or code is given, making the benchmark impossible to inspect. I therefore recommend keeping the reader's REJECT verdict: the central grounding claims are not yet supported by released artifacts or reliability evidence. My proposed re-annotation test would settle the concern directly if the dataset becomes available.","tokens_in":11101,"tokens_out":4130,"duration_ms":46784,"concrete_test":"Release the HW-MLVQA test split and have two independent annotators re-annotate a stratified random sample of 100 question-answer pairs per language by drawing the answer-evidence bounding box from the scanned page and question alone; compare their boxes to the released XML boxes using IoU. If annotator-annotator or annotator-released mean IoU falls below 0.5, the ground-truth boxes, and thus the near-chance mean IoU in Table 5, are not reliable evidence of a grounding failure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's central novelty is evidence-based grounded VQA (Section 3.1), and its headline empirical claim is that Qwen2VL grounds near chance (mean IoU 0.0166 English, 0.0126 Hindi, Table 5). Both claims depend on the ground-truth evidence boxes produced in Section 3.2.2. That pipeline first runs OCR, then aligns 'OCR extracted words' with expected answer words, and generates bounding boxes, with a later LabelIMG verification step. Table 2 reports EasyOCR word accuracy of only 5.56% on English and 8.29% on Hindi; GoogleOCR is higher but still only 78.64% and 67.10%. If automatic boxes are generated from words recoverable by OCR, most answer words will be missed when EasyOCR is used, and GoogleOCR will still miss roughly a quarter of words. The manual verification by five to six annotators could in principle repair every box, but no inter-annotator agreement, correction counts, or post-verification accuracy are reported. The dataset itself is also inconsistently specified: the abstract says 1,600 pages and 2,400 questions, while Section 3.4 and Table 1 say 1,520 pages and 2,400 questions per language (4,800 total). Without a released dataset or a reliability study of the final boxes, the near-zero mean IoU may reflect incorrect ground-truth boxes rather than poor model grounding.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents HW-MLVQA, a benchmark for multilingual handwritten document VQA. Contexts are taken from SQuAD and MLQA, written by hand by volunteers, and paired with question-answer sets. The authors evaluate LLaMA 3.1 and Qwen2VL in text-only, image-only, and combined image-text settings, and report a visual grounding experiment in which Qwen2VL localizes evidence with near-zero mean IoU. The stated contributions are the new bilingual handwritten benchmark, multi-modality baselines, and an evidence-grounding evaluation.","tokens_in":11405,"tokens_out":6602,"duration_ms":69323,"significance":"If the dataset and annotations were reliable, HW-MLVQA would address a genuine gap: no existing handwritten VQA benchmark combines multilingual scripts with evidence grounding. The paper's inclusion of both commercial and open-source OCR, the ANLS metric, and the explicit grounding evaluation are useful design choices. However, the benchmark's validity is not currently established: the paper gives contradictory dataset statistics, reports OCR word accuracies too low to support the automatic annotation pipeline, provides no quantitative verification of the manually corrected boxes, and no release location. The headline grounding result (mean IoU 0.0166 English, 0.0126 Hindi, Table 5) is therefore difficult to interpret. The strength of the paper is its stated problem framing; the evidence needed to support the central contribution is missing.","major_comments":[{"comment":"The dataset size is reported inconsistently: the abstract says 1,600 pages and 2,400 QA pairs, Section 3.4 says 1,520 images and 2,400 questions while also mentioning 21,600 train/val questions and 12,000 images, and Table 1 lists 2,400 questions per language (4,800 total). These discrepancies change the claimed scale by a factor of two and make it impossible to know the actual test split; please reconcile all numbers and provide page/question counts per split.","section":"Abstract, Section 3.4, Table 1"},{"comment":"Ground-truth evidence boxes are produced by OCR extraction and string matching, but Table 2 reports EasyOCR word accuracy of only 5.56% (English) and 8.29% (Hindi). Unless the manual LabelIMG verification corrected essentially every box, most answer words would not be recovered by the automatic pipeline. No inter-annotator agreement, correction counts, or post-verification accuracy are reported. Consequently, the near-zero mean IoU in Table 5 may reflect incorrect ground-truth boxes rather than poor model grounding, and the grounding claim in Section 5.2.6 is not yet established. Please report verification statistics and validate a random sample with independent annotators.","section":"Section 3.2.2, Table 2, Table 5"},{"comment":"The image-only English results in the text (EM 45.39, F1 60.13, ANLS 66) do not match Table 4 (EM 57.51, F1 71.32, ANLS 69.11). The value 22.70 that appears in the same sentence is the Hindi image-only EM from Table 4. This inconsistency affects the paper's comparison between modalities and must be corrected.","section":"Section 5.2.3, Table 4"},{"comment":"Because the QA pairs are copied from public SQuAD/MLQA contexts, models pre-trained on those texts may benefit from memorized answers, especially in the text and text+image conditions. The paper does not provide a contamination analysis or new validation questions. In addition, no URL or access information for the dataset is given, so the benchmark cannot be independently assessed or used. Please provide a data release plan and either show that test content is not in pretraining corpora or add a subset of novel handwritten questions.","section":"Section 3.2, Section 3.4"}],"minor_comments":[{"comment":"The contribution list mentions M-BERT as an evaluated model, but no M-BERT results appear anywhere in the experiments; please either add the results or remove the mention.","section":"Section 1, Section 5"},{"comment":"Equation (4) defines IoU over sets of boxes but does not specify how predicted boxes are matched to ground-truth boxes or whether the value is averaged over all evidence regions per question; please clarify the protocol.","section":"Equation (4)"},{"comment":"The text states that Fig. 12 illustrates the highest Hindi IoU and then also uses Fig. 12 as a low-IoU example; the narrative and figure captions need to be reconciled.","section":"Section 5.2.6, Figures 10-13"},{"comment":"The manuscript needs careful proofreading: examples include 'we facilitates' in the abstract, 'an sample' in Section 3.4, and inconsistent hyphenation of Qwen2VL/Qwen2-VL and HW-MLVQA/HW-ML VQA.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The manuscript's central contribution is the dataset itself, but the paper neither releases the data nor provides evidence that the automatic annotation pipeline produces reliable ground truth. The internal statistics are contradictory, and the main grounding result is uninterpretable without reliability data. I would be willing to reconsider if the authors release the dataset, correct the statistics, and add a substantive annotation-reliability study; in the absence of those, I cannot recommend publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"HW-MLVQA is the first bilingual (English/Hindi) handwritten document VQA benchmark with answer grounding, and that alone makes it worth a look. The authors hand-collected handwritten pages from SQuAD/MLQA contexts, ran a three-way evaluation (text-only, image-only, image+text) with two OCR engines, and report OCR performance honestly, including EasyOCR's terrible 5.56% word accuracy on English. Those numbers are useful on their own.\n\nThe dataset artifact is the main contribution, but the paper currently doesn't let you check it: no download link, no repository. Worse, the statistics don't add up. The abstract says 1,600 pages and 2,400 QA pairs; Section 3.4 says 1,520 images and 2,400 QA; Table 1 lists 2,400 questions per language, i.e., 4,800. The mention of 21,600 train/val questions and 12,000 images is unexplained. A benchmark paper lives or dies by its numbers and its release.\n\nThe bigger problem is the grounding annotation. The automatic boxes come from aligning OCR output with expected answer words. With EasyOCR at 5.56% word accuracy, most answer words would be missed. The manual verification step (5-6 annotators) could fix this, but no inter-annotator agreement or post-correction accuracy is reported. So the ground-truth boxes are unverified, and the headline result—Qwen2VL at near-zero mean IoU—may say more about bad boxes than about the model. That's a load-bearing flaw for the paper's central empirical claim.\n\nThe citation pattern and the math are fine; the metrics (ANLS, IoU) are standard. The novelty is real, and the core idea is sound. But as it stands, the paper is not ready to introduce a benchmark: it needs a data release, consistent statistics, and a reliability study of the ground-truth boxes. I'd send it to peer review, but with the expectation of major revision. The topic is relevant for handwriting recognition and multilingual document understanding, and the dataset, if cleaned and released, would be a useful resource. For now, I wouldn't cite it in its current form.","headline":"First bilingual handwritten VQA benchmark with grounding, but the paper needs a data release, consistent stats, and a box-reliability study before its results can be trusted.","tokens_in":11941,"tokens_out":3265,"would_cite":false,"duration_ms":32452,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper presents HW-MLVQA, the first bilingual handwritten VQA benchmark with evidence grounding, and shows current vision-language models localize answers near chance.","keywords":["HW-MLVQA","handwritten document understanding","multilingual visual question answering","evidence grounding","OCR","Hindi-English benchmark","vision-language models"],"falsifier":"Take a random sample of HW-MLVQA pages, have independent human annotators draw the exact bounding boxes of the answer spans, and compute the IoU between the benchmark's XML boxes and the human boxes; if the agreement is well below near-perfect (for example, mean IoU below 0.9), then the near-zero VLM grounding numbers partly reflect noise in the labels rather than pure model failure.","tokens_in":10913,"feed_emoji":"✍️","tokens_out":10044,"duration_ms":91609,"temperature":0.7,"pith_summary":"This paper introduces HW-MLVQA, a benchmark of 2,400 English and Hindi question-answer pairs grounded in 1,520 handwritten pages, where a correct answer must be accompanied by the bounding box of the evidence in the handwriting. The authors evaluate LLaMA 3.1 and Qwen2VL under text-only, image-only, and combined inputs, with OCR from Google and EasyOCR, and they report that a state-of-the-art vision-language model such as Qwen2VL extracts little linguistic structure from handwriting alone in zero-shot settings. The key reported result is that evidence localization is essentially unsolved: mean intersection-over-union is 0.0166 for English and 0.0126 for Hindi, near zero. If the benchmark is sound, a sympathetic reader would conclude that handwritten multilingual document understanding is an open problem and that OCR quality, especially for open-source tools, is a primary bottleneck.","feed_headline":"Handwriting VQA benchmark: models locate answers near chance","feed_subtitle":"First bilingual handwritten VQA set with evidence boxes: Qwen2VL mean IoU ~0.02.","key_machinery":"The load-bearing component is the Evidence-Based Grounded VQA (EB-GVQA) task together with the benchmark's construction pipeline. EB-GVQA formalizes outputs as an answer string plus a set of evidence pairs (bounding box, page index), scored with ANLS for the answer and mean IoU for the evidence. The dataset starts from 400 English and 400 Hindi contexts drawn from SQuAD and MLQA, splits each context into about 50-word segments, and has volunteers handwrite them onto A4 pages; an automatic annotation stage then uses Google OCR and EasyOCR to extract words, matches the OCR tokens against the expected answer words from the QA files, and writes XML boxes that a five-to-six person team manually verifies.","core_discovery":"HW-MLVQA is claimed to be the first benchmark that combines multilingual (English and Hindi) handwritten documents with evidence-based grounded VQA: each question's ground truth includes both an answer string and bounding boxes marking where in the page the answer is written. The paper's central experimental claim is that current models cannot solve this task. Under zero-shot evaluation, the paper's Table 4 reports that Qwen2VL-7B with image-only input reaches an exact-match accuracy of 57.51% on English and 22.70% on Hindi, and its evidence localization is effectively absent, with mean IoU of 0.0166 on English and 0.0126 on Hindi and variance below 0.002. The authors conclude that state-of-the-art vision-language models struggle to capture linguistic structures from handwritten images alone, and that OCR noise, especially from open-source EasyOCR, further degrades performance when transcriptions are provided.","pith_inferences":["The paper does not compare its mean IoU values against a random-box baseline; a quantitative test of the 'near chance' reading would sample random boxes on the same pages and compute their IoU distribution.","Because the ground-truth boxes are produced by an OCR-matching pipeline in which EasyOCR's word accuracy is 5.56% on English pages, a subset of evidence labels is likely imperfect; a human-annotated sample would let users separate label noise from genuine model failure.","The abstract states 1,600 handwritten pages, while Section 3.4 and Table 1 give 1,520 (834 English plus 686 Hindi); the released dataset's actual page count should be confirmed before relying on scale claims.","In Section 5.2.3 the prose reports an image-only English exact match of 45.39% for Qwen2VL, while Table 4 reports 57.51% for the same setting; readers should verify which number matches the released evaluation."],"forward_implications":["Any future system claiming to handle handwritten documents should be measured on both answer accuracy (ANLS or exact match) and evidence localization (IoU), with the benchmark's near-zero IoU numbers as the explicit target to beat.","Since Qwen2VL's exact match on English drops from 67% with ground-truth text to about 47% with GoogleOCR output, handwriting OCR quality is a direct bottleneck; improving OCR word accuracy should transfer to higher VQA scores.","The English-Hindi gap (image-only exact match 57.51% versus 22.70%) quantifies how much harder the second script is for a monolithic VLM, supporting the paper's call for language-specific adaptation.","Combined image-and-text inputs consistently outperform either modality alone (with ground truth, English exact match rises from 57.51% image-only to 68.00%), suggesting that future systems should fuse visual handwriting cues with noisy transcripts."],"supporting_citations":[{"why":"SQuAD supplies the English source contexts from which the benchmark's question-answer pairs are drawn.","marker":"[29]"},{"why":"MLQA supplies the multilingual aligned contexts and QA pairs that ground the Hindi half of the benchmark.","marker":"[12]"},{"why":"HW-SQuAD and Bentham-QA are the prior handwritten VQA datasets that this benchmark extends with multilingualism and evidence grounding.","marker":"[7]"},{"why":"DocVQA provides the ANLS evaluation metric used for answer accuracy.","marker":"[15]"},{"why":"Qwen2VL is the vision-language model whose zero-shot and multimodal results anchor the baseline evaluation.","marker":"[11]"},{"why":"LLaMA 3.1 is the language-model baseline used for text-only evaluation on ground truth and OCR transcripts.","marker":"[9]"},{"why":"The ICDAR 2024 competition situates the benchmark relative to the recent handwritten-document recognition and VQA effort.","marker":"[8]"}],"fun_headline_variants":["First handwritten VQA benchmark: models fail to ground answers","Qwen2VL near chance on Hindi handwritten VQA","HW-MLVQA: grounding accuracy at 0.02 IoU","Multilingual handwritten VQA: even Qwen2VL struggles"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The assumption that the answer-location boxes in the benchmark are accurate labels, even though they are produced by matching OCR output to answer words and EasyOCR's word accuracy on English pages is only 5.56%, so many answer boxes may be missing or misplaced.","fun_headline_variants_meta":{"raw":{"variants":["First handwritten VQA benchmark: models fail to ground answers","Qwen2VL near chance on Hindi handwritten VQA","HW-MLVQA: grounding accuracy at 0.02 IoU","Multilingual handwritten VQA: even Qwen2VL struggles"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000256,"raw_usage":{"total_tokens":1563,"prompt_tokens":923,"completion_tokens":640,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":581}},"tokens_in":539,"tokens_out":640,"duration_ms":7352,"temperature":1.0,"reasoning_tokens":581,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:26:19.135390+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of HW-MLVQA pages, have independent human annotators draw the exact bounding boxes of the answer spans, and compute the IoU between the benchmark's XML boxes and the human boxes; if the agreement is well below near-perfect (for example, mean IoU below 0.9), then the near-zero VLM grounding numbers partly reflect noise in the labels rather than pure model failure.","supporting_citations":[{"cited_title":"In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (2016) 4 18","cited_arxiv_id":null,"evidence_quote":"SQuAD supplies the English source contexts from which the benchmark's question-answer pairs are drawn."},{"cited_title":"IJDAR24(3), 235–249 (2021) 2","cited_arxiv_id":null,"evidence_quote":"HW-SQuAD and Bentham-QA are the prior handwritten VQA datasets that this benchmark extends with multilingualism and evidence grounding."},{"cited_title":"In: Barney Smith, E.H., Liwicki, M., Peng, L","cited_arxiv_id":null,"evidence_quote":"The ICDAR 2024 competition situates the benchmark relative to the recent handwritten-document recognition and VQA effort."}],"review_version":1}