{"id":"d17bd16d-ec90-4cb5-be9d-7d963f827795","arxiv_id":"2412.00846","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"GPSM4K is a new geometry QA dataset with step-by-step solutions; the paper reports that fine-tuning on it improves LVLM scores, with captioning and RAG giving smaller gains.","lead":"This paper introduces GPSM4K, a geometry dataset with step-by-step solutions for training and testing multimodal AI models. It reports that fine-tuning on this dataset improves geometry performance and that captions and retrieval help, but the data is not yet public and the evaluation relies on an LLM judge.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The unvalidated Gemini Pro auto-judge in Section 5 underpins every reported accuracy; because that same model generated much of the training data, all comparative claims need human validation before being trusted.","rationale":"The reader identified the same weakest assumption, and I agree. The central claim is comparative: fine-tuning on GPSM4K beats PGPS9K, captions help, RAG helps. All these comparisons are mediated by one unvalidated LLM judge that also generated the training content. This is not just an external-validity caveat; it creates a concrete mechanism for self-confirmation: if the judge prefers answers phrased like the Gemini-generated step-by-step solutions, models fine-tuned on GPSM4K (which contains those solutions) will receive inflated step scores even when their final answers are wrong, and PGPS9K-trained models will be penalized. A human-labelled subset would settle this. I note secondary issues—test-set size inconsistent (150 in Section 4, 200 in Table 2), dataset counts inconsistent (2157/5340 in the abstract vs 1480/4440 in Table 1 vs 1438/4440 in Section 9), and no released data/code—but those reduce reproducibility rather than directly undermining the mechanism. The judge concern is the single load-bearing one; the proposed test could validate or refute it. Hence the verdict remains conditional pending that check.","tokens_in":17544,"tokens_out":3295,"duration_ms":28550,"concrete_test":"Take a random sample of 50 predictions from the GPSM4K test set (covering models in Table 4) and 20 step-by-step evaluations from Figure 9. Have two human annotators independently judge final-answer correctness against the provided ground truth (and step labels for the 20), using the same prompts. Compute Cohen's kappa between Gemini Pro and human labels, and the human-based accuracy for each model. If kappa is below 0.8 or the accuracy for any model shifts by more than 5 absolute points, the paper's central accuracy claims are not supported. Additionally, re-run the 'Yes/No' final-answer comparison with a different judge (e.g., GPT-4) on the same 50 cases and report agreement with Gemini Pro.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that fine-tuning on GPSM4K improves geometry accuracy (e.g., Table 4: LLaVA 1.6 Vicuna 7B rises from 8% on PGPS9K to 22.66% on GPSM4K)—rests entirely on an automatic judge: Gemini Pro extracts final answers from ground truth and predictions and labels matches 'Yes'/'No' (Section 5.1, Figure 7), and also assigns binary step scores for CoT evaluation (Section 5.2, Figure 8). The same Gemini Pro was used to generate diagram descriptions, two augmented questions per original, and all regenerated step-by-step solutions (Section 3.2). The paper reports no human validation, no inter-annotator agreement, and no error analysis of the judge. If Gemini Pro is biased toward Gemini-style solutions, or errors in numeric extraction (e.g., equivalent forms like '5 cm' vs '5', or '√25' vs '5'), or over-credits steps matching its own generated templates, every accuracy number in Tables 3–7 is suspect. The step evaluation criteria ('mathematical concept, computation steps, and final answer') are subjective, and the 'pre-defined key-step template' is never described. This is load-bearing because all comparisons—dataset choice, captioning, RAG—are measured through this judge; a noisy or biased judge could invert or flatten the reported rankings.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"GPSM4K is presented as a new multimodal geometry benchmark: original question-answer pairs are manually extracted from Indian board textbooks (grades 7–12 in the abstract, grades 6–12 in Section 3.1), then augmented with Gemini Pro–generated diagram descriptions, two additional questions per original, and restructured step-by-step solutions. The authors fine-tune LLaVA and GLLaVA variants on GPSM4K, report accuracy gains relative to fine-tuning on PGPS9K (e.g., LLaVA 1.6 Vicuna 7B rises from 8% to 22.66% in Table 4), and evaluate the effects of image captions and 1-shot multimodal RAG. The evaluation of both final answers and chain-of-thought steps is performed by Gemini Pro, which extracts answer strings and compares them, and which also assigns binary step scores. The paper claims that GPSM4K fills a gap by including numerical-answer and theorem-proving questions with step-by-step solutions.","tokens_in":17828,"tokens_out":4388,"duration_ms":41325,"significance":"If the reported accuracy numbers are trustworthy, GPSM4K would be a useful resource for the geometry reasoning community: it addresses the relative scarcity of non-multiple-choice geometry benchmarks, provides step-by-step solutions in a consistent format, and includes experiments on captioning and RAG that are rarely studied on geometry data. The dataset construction is concrete, and the fine-tuning comparisons across several LLaVA/GLLaVA configurations are informative in principle. However, the empirical claims are currently not established because they rest on an unvalidated LLM judge, the dataset statistics are internally inconsistent, and the paper does not demonstrate that the test set is disjoint from the training set. The benchmark and the methodology are salvageable, but the manuscript needs substantial revision before the central claims can be accepted.","major_comments":[{"comment":"The automatic judge is unvalidated and is the sole source of every accuracy number in Tables 3–7. Gemini Pro both generates much of the training data (diagram descriptions, two augmented questions per original, and all regenerated solutions in §3.2) and evaluates the final answers and steps. The paper reports no human validation, no inter-annotator agreement, and no error analysis for the judge. Since the judge compares extracted answer strings with a simple 'Yes'/'No' label, errors in numeric normalization (e.g., '5 cm' vs. '5', or '√25' vs. '5') or a stylistic bias toward Gemini-generated solutions could systematically inflate or flatten the reported rankings. I request a human-validated subset (at least 100–150 items), a normalization-based exact-match baseline, and an error analysis of extraction and step-labeling failures.","section":"§5.1, §5.2, Figures 7 and 8"},{"comment":"The dataset size is reported inconsistently. The abstract states 2157 multimodal question-answer pairs augmented to 5340 problems, Table 1 reports 1480 original and 2960 augmented samples totaling 4440, and Section 9 says 1438 original problems augmented to 4440 entries. These are not minor rounding differences; they change the claimed scale of the dataset by hundreds of items. The authors must state one correct set of numbers and ensure the abstract, table, and limitations section agree before the benchmark can be evaluated or reproduced.","section":"Abstract, Table 1, §9"},{"comment":"The test-set size is inconsistent. Table 2 reports 200 total inference samples, while Section 4 states that 'All results are reported based on experiments conducted on a test set of 150 geometry problems' and Experiment 3 says 'we utilize images from our test set of 150 examples.' If the test set is a 150-item subset of Table 2, that split must be described; if the numbers changed during writing, the reported accuracies in Tables 3–7 cannot be reproduced without knowing which test set was used.","section":"Table 2, §4"},{"comment":"The paper does not establish that the test set is disjoint from the training set. Because question augmentation in §3.2.2 generates two new questions from each original diagram and QA pair, and because the test set appears to draw from the same textbook sources, fine-tuning on augmented questions could include near-duplicates of test items. The authors should specify the exact train/test split, provide source identifiers or diagram-level overlap statistics, and describe any deduplication procedure. Without this, the improvement from 8–8.66% to 22.66–24.66% in Table 4 could partly reflect memorization rather than improved geometric reasoning.","section":"§3.2, §4.2"},{"comment":"The comparison between models fine-tuned on PGPS9K and on GPSM4K is confounded by more than dataset quality: the two training sets differ in size, problem format (MCQ vs. numerical/theorem-proving), solution structure, and grade distribution. The claim that GPSM4K's detailed step-by-step solutions drive the improvement is therefore not directly supported. A controlled comparison (e.g., fine-tuning on a matched-size PGPS9K subset, or ablating step-by-step solutions from GPSM4K) would give the central claim much stronger support.","section":"§4.2, Table 4"}],"minor_comments":[{"comment":"There are unresolved table references: Section 4.2 refers to 'Table ??' instead of Table 4, and Section 4.4 refers to 'the results of Experiment 2 table 7' when the captioning results are in Table 5.","section":"§4.2 and §4.4"},{"comment":"The grade range is inconsistent: the abstract and title say grades 7–12, while Section 3.1 says Grades 6 through 12. Please align these statements.","section":"Abstract and §3.1"},{"comment":"The caption contains a duplicated word: 'chapter chapter-wise distribution'. The figure should also be referenced in the text before it appears.","section":"Figure 3"},{"comment":"The table header has 'Mean / Median Jaccard' and 'Mean / Median Cosine similarity' as two columns, but each cell contains two values separated by a slash; the column semantics should be made explicit (e.g., 'Jaccard (mean/median)').","section":"Table 5"},{"comment":"The keyword list contains typographical and formatting errors: 'LLMS' should be 'LLMs', 'Retrieval Augmentated' should be 'Retrieval-Augmented', and the list lacks consistent punctuation.","section":"Keywords"},{"comment":"The subsection heading contains a typo: 'Evalaution' should be 'Evaluation'. In addition, Figure 9 is placed in the appendix and is not cited in the text; it should be integrated into the step-by-step evaluation discussion.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads as an early draft: unresolved table references, inconsistent dataset and test-set sizes, and an unvalidated evaluation pipeline are too substantial for acceptance in the current form. I would ask the authors to provide a corrected dataset description, a human-validated evaluation subset, and explicit train/test separation evidence. If they cannot supply human validation of the Gemini Pro judge, I would consider the empirical claims unsupported and lean toward rejection rather than conditional acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"GPSM4K is a real contribution: a manually collected geometry QA dataset with numerical and theorem-proving questions, diagram descriptions, and step-by-step solutions in a uniform format. It fills a gap next to PGPS9K and Geometry3K, which are mostly multiple-choice. The fine-tuning experiments show consistently higher accuracy for LLaVA and GLLaVA variants when trained on GPSM4K than on PGPS9K or GeoQA++, and the captioning and 1-shot RAG results are sensible though modest.\n\nThe soft spots are real. The evaluation uses Gemini Pro as an automatic judge for both final answers and step-by-step correctness (Section 5), and the same model generated much of the training data (descriptions, two augmented questions per original, regenerated solutions). There is no human validation of the judge, no inter-annotator agreement, no error analysis. The step evaluation criteria are subjective, the 'pre-defined key-step template' is never described, and Section 5.2 contradicts itself about whether ground-truth answers are fed to the judge. If the judge is biased toward Gemini-style solutions or mishandles equivalent answer formats, every accuracy number in Tables 3–7 could shift. That is load-bearing because the paper's main claims are comparisons measured through this judge.\n\nThere are also internal inconsistencies: the abstract says 2,157 original and 5,340 total examples but Table 1 lists 1,480 original and 4,440 total; Section 4 says a test set of 150, Table 2 lists 200; several table references are placeholders. Gemini Pro's zero-shot score on GPSM4K is 44% in Table 3 and 35.33% in Table 4, with no explanation. The test set's disjointness from the augmented training data is not addressed. And the paper never fine-tunes on Geo170K, which is the closest prior dataset, so the claimed advantage over existing datasets is not fully benchmarked.\n\nStill, the core idea holds up: a geometry benchmark with diverse question types and step-by-step solutions is worth having, and the reported fine-tuning gains are plausible. The paper deserves a serious review, but the referee should require human validation of the judge, a fixed and consistent dataset description, and a comparison against Geo170K before the numbers are trusted. I'd bring it to a reading group if the authors release the data; otherwise the value is limited.","headline":"Useful new geometry dataset, but every reported number rests on an unvalidated Gemini Pro judge that also generated the training data.","tokens_in":18367,"tokens_out":3753,"would_cite":true,"duration_ms":34062,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning on GPSM4K, a new grade 7-12 geometry dataset with step-by-step solutions, raises an open vision-language model's accuracy from 8% to 22.66%, and captions and retrieval augmentation add further gains.","keywords":["multimodal large language models","geometry problem solving","GPSM4K","step-by-step solutions","image captioning","retrieval-augmented generation","automated answer evaluation","chain-of-thought scoring"],"falsifier":"Have human experts label the same test-set predictions for final-answer correctness and per-step correctness, then compare their labels with Gemini Pro's labels; strong disagreement or a systematic pattern, for example favoring responses that resemble Gemini-generated text, would invalidate the reported accuracy numbers.","tokens_in":17396,"feed_emoji":"📐","tokens_out":9275,"duration_ms":78198,"temperature":0.7,"pith_summary":"The paper tries to establish that a dataset of geometry problems with consistent step-by-step solutions is enough to make open-source vision-language models substantially better at solving geometry. It introduces GPSM4K, drawn from grades 7-12 textbooks, with numerical-answer and theorem-proving questions and two solutions per problem: the textbook original and a decomposed version generated by a multimodal model. Fine-tuning on GPSM4K raises a LLaVA 1.6 Vicuna 7B model from 8% accuracy (after fine-tuning on PGPS9K) to 22.66% on the paper's test set, with similar jumps for 13B and Mistral variants. The paper also reports that adding image captions to prompts improves results, human captions more than machine captions, and that a one-shot retrieval-augmented generation context adds a few points for Gemini and LLaMa3. The paper further argues that an automated Gemini Pro judge can extract final answers and score each reasoning step, replacing binary right/wrong evaluation with step-level labels.","feed_headline":"GPSM4K step-by-step data lifts model accuracy to 22.66%","feed_subtitle":"Fine-tuning on this geometry benchmark beats multiple-choice datasets; captions and retrieval-augmented search add gains.","key_machinery":"The central machinery is the dataset structure rather than a mathematical identity: GPSM4K provides each problem with a step-by-step solution broken into six labeled sections (data identification, problem analysis, theoretical framework, methodology development, computation, solution). Fine-tuning on this format is what the paper credits for the accuracy gains over multiple-choice-only datasets like PGPS9K. A second mechanism is the Gemini Pro pipeline that generates diagram descriptions, augments questions, regenerates solutions, and later judges model outputs. A third mechanism is a vector database with cosine similarity that retrieves one analogous solved problem as context during inference, which the paper calls retrieval-augmented generation.","core_discovery":"On its own terms, the paper's central discovery is that the structure of the training data, not just the model, determines geometry performance. GPSM4K contains numerical-answer and theorem-proving problems from grades 7 to 12, with each problem carrying both an original textbook solution and a regenerated, decomposed step-by-step solution. Fine-tuning a 7-billion-parameter LLaVA 1.6 model on GPSM4K raises its accuracy on the paper's held-out test set from 8% (fine-tuned on PGPS9K) to 22.66%, and the 13-billion-parameter variant reaches 24%. The GLLaVA 13B model improves from 22.66% on GeoQA++ to 25.33% on GPSM4K. Feeding the model a caption of the diagram during inference raises scores further, with human-written captions helping more than machine-generated ones, and one-shot retrieval-augmented generation adds about two to three points on both Gemini and LLaMa3. The paper also claims that Gemini Pro can serve as an automated judge that extracts final answers and assigns a binary score to each step of a predicted solution, enabling multistep rather than answer-only evaluation.","pith_inferences":["A testable extension beyond the paper: if the automatic judge is biased toward responses written in the style of the model that generated the solutions, then part of the reported gain from fine-tuning and RAG could be stylistic mimicry rather than improved geometry; comparing model outputs with human-graded labels would settle this.","The paper does not isolate which part of GPSM4K drives the gain, whether step-by-step solutions, diagram descriptions, or question diversity. An equal-sized control dataset with the same questions but multiple-choice-only answers would identify the mechanism.","Because the augmented questions and solutions were generated by the same model family used as judge, there is a risk of circularity in the evaluation; a human-scored subset of the test set would give an independent check.","If the approach generalizes, the same template of textbook extraction, decomposition of solutions, and step-level judging could be applied to other stepwise reasoning domains such as algebra word problems, physics exercises, or formal proofs."],"forward_implications":["Fine-tuning on a dataset with consistent step-by-step solutions can move an open vision-language model from near-chance to roughly 23% on high-school geometry, so data design is a first-order lever alongside model size.","Step-level labels make it possible to ask where models fail, whether in reading the diagram, choosing the right theorem, or computing, rather than only whether the final answer matches.","Image captions become an actionable intervention: supplying a good textual description of the diagram improves accuracy, and human captions beat automatic ones, which points to the visual encoder as a bottleneck.","Retrieval of one analogous solved problem during inference gives a small but consistent boost, offering a cheap way to improve geometry answers without additional fine-tuning."],"supporting_citations":[{"why":"Supplies the PGPS9K dataset and solution-program annotations used as the main fine-tuning baseline in the comparison.","marker":"[43]"},{"why":"Supplies Geometry3K, the prior benchmark whose lack of step-by-step solutions motivates GPSM4K.","marker":"[21]"},{"why":"Supplies the G-LLaVA architecture and Geo170K dataset, both evaluated and fine-tuned against GPSM4K.","marker":"[11]"},{"why":"Supplies GeoQA, an earlier process-annotated geometry QA dataset used as a comparison and motivation.","marker":"[6]"},{"why":"Supplies the LLaVA architecture used for the main open-source fine-tuning experiments.","marker":"[20]"},{"why":"Supplies Gemini Pro, the model that generates diagram descriptions, augmented questions, regenerated solutions, and the automatic judge scores.","marker":"[32]"}],"fun_headline_variants":["Step-by-step geometry data lifts LVLM accuracy to 22.66%","GPSM4K: detailed steps boost geometry scores from 8% to 22.66%","New benchmark enhances geometry reasoning in multimodal LLMs","Multistep geometry solutions outperform multiple-choice training","LLM judge enables automated multistep scoring in geometry"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every reported accuracy number rests on Gemini Pro's automatic judging being accurate, since it both extracts and compares final answers and assigns step-level labels, and the paper reports no human validation of that judge.","fun_headline_variants_meta":{"raw":{"variants":["Step-by-step geometry data lifts LVLM accuracy to 22.66%","GPSM4K: detailed steps boost geometry scores from 8% to 22.66%","New benchmark enhances geometry reasoning in multimodal LLMs","Multistep geometry solutions outperform multiple-choice training","LLM judge enables automated multistep scoring in geometry"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1419,"prompt_tokens":1021,"completion_tokens":398,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":637,"completion_tokens_details":{"reasoning_tokens":308}},"tokens_in":637,"tokens_out":398,"duration_ms":4020,"temperature":1.0,"reasoning_tokens":308,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:55:34.959127+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human experts label the same test-set predictions for final-answer correctness and per-step correctness, then compare their labels with Gemini Pro's labels; strong disagreement or a systematic pattern, for example favoring responses that resemble Gemini-generated text, would invalidate the reported accuracy numbers.","supporting_citations":[],"review_version":1}