{"id":"f51c3a0c-58d5-47fe-a1ba-617efef54c05","arxiv_id":"2504.12773","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A symbolic geometry engine generates step-by-step training data and verifies MLLM reasoning steps, improving accuracy on geometry benchmarks.","lead":"GeoGen generates thousands of step-by-step geometry problems by having a symbolic geometry engine solve diagrams and a large language model translate the solutions into plain language. The authors train vision-language models on this data and add a translator that checks model reasoning against symbolic rules, improving scores on geometry benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GeoExpand is built from all Geometry3K and PGPS9K images; the paper never states that test-split images were excluded, so training and benchmarking may share diagrams and inflate Tables 1–2.","rationale":"I agree with the reader's weakest assumption. The same concern is the most load-bearing: every main benchmark except GeoQA is either directly built from the same image pool (Geometry3K, PGPS9K) or draws on those datasets (MathVista-mini-GPS includes Geometry3K). The paper does not state that the 4,849 expanded images exclude test splits, and the released code makes this checkable. I do not see a different concern that outweighs it: the LLM-based completion checker and fast-mode verification are weaker but affect interpretation less directly than potential train/test overlap. The paper has real positive features—released code, a trainable GeoLogic translator, and a tree-search verification procedure—so the appropriate disposition remains conditional: require the overlap check and, if overlap exists, a clean re-evaluation before the empirical claims are accepted. Hence verdict_should_be = UNCHANGED.","tokens_in":15584,"tokens_out":4294,"duration_ms":43608,"concrete_test":"Download the released GeoExpand dataset/code from https://github.com/ycpNotFound/GeoGen, extract the list of source image IDs for the 4,849 Geometry3K/PGPS9K images, and intersect it with the official test-split image IDs/hashes of Geometry3K (601) and PGPS9K (1,000). If the intersection is non-empty, retrain the T3 configuration (Geo170K + GeoExpand + GeoSynth) on Qwen2.5-VL-7B after removing all overlapped images and rerun the GeoTest evaluation; if accuracy on Geometry3K/PGPS9K drops materially, the reported gains are contaminated by train/test overlap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 states that GeoExpand generates 45,526 QA pairs from 4,849 original images in Geometry3K and PGPS9K, but nowhere states that the official test splits of those datasets were excluded. Section 4.1.3 evaluates on the 601-image Geometry3K test set and the 1,000-image PGPS9K test set. Since Geometry3K and PGPS9K are exactly the source datasets of GeoExpand, any test image that is also a GeoExpand source image has already been shown to the model during SFT. Even when the generated question differs, the diagram itself is identical, so the model has been trained on the visual input it is later asked to solve; this can inflate both the Completion and Choice numbers in Tables 1 and 2 substantially. The paper offers no split statement, no image-level deduplication, and no evaluation on a held-out subset of Geometry3K/PGPS9K images. This is the single most load-bearing gap: the central empirical claim of consistent improvement depends on the evaluation being disjoint, and the text as written does not establish that.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents GeoGen, a pipeline that automatically generates step-by-step geometry solutions by combining random diagram synthesis, symbolic forward search with traceback, and LLM-based translation into natural language. The pipeline is used to create two training datasets: GeoExpand, with 45,526 QA pairs derived from 4,849 images from Geometry3K and PGPS9K, and GeoSynth, with 62,868 pairs from 129,230 synthesized diagrams. The authors also train GeoLogic, an LLM that translates natural language reasoning steps into formal predicates, and integrate it with a symbolic verifier in a step-level tree search during inference. Fine-tuning Qwen2.5-VL, InternVL2.5, and LLaVA-v1.6 on a mixture of Geo170K, GeoExpand, and GeoSynth yields consistent accuracy gains on GeoQA, Geometry3K, PGPS9K, and MathVista-mini-GPS. The authors report state-of-the-art results among MLLM-based geometry solvers, including 78.0% choice accuracy on GeoQA with GeoGen-SFT-7B.","tokens_in":15716,"tokens_out":5652,"duration_ms":53994,"significance":"If the reported gains are not an artifact of training/test overlap, the paper makes a useful contribution: it demonstrates a fully symbolic route to generating large volumes of chain-of-thought supervision that is guaranteed correct at the formal level, and it shows one way to use formal verification as a post-hoc check on MLLM outputs. Strengths include open-sourced code, experiments across multiple model families, manual verification of 100 translated solutions, and a GeoLogic translation accuracy of 94.16% on a custom test set. The main risk is the lack of any statement about excluding the benchmark test splits from the GeoExpand expansion; this must be resolved before the empirical claims can be accepted.","major_comments":[{"comment":"The manuscript does not establish that GeoExpand is disjoint from the evaluation sets. GeoExpand is built from 4,849 original images from Geometry3K and PGPS9K (Section 3.2), and the evaluation in Section 4.1.3 uses the standard test splits of those datasets (601 and 1,000 images). No sentence states that test-split images were excluded from the expansion, and no image-level deduplication is reported. If test images are among the GeoExpand sources, the model is trained on the exact diagrams it is later asked to solve, which alone can inflate the Completion and Choice accuracies in Tables 1 and 2 and the MathVista results, since MathVista-mini-GPS also draws from Geometry3K and GeoQA+. The authors should state the split policy, verify non-overlap at the image level, and re-run the experiments with any overlapping test images removed from the training expansion (or report results on a held-out set of images never used in GeoExpand).","section":"Section 3.2 and Section 4.1.3"},{"comment":"The Completion metric is defined through an LLM-based automatic verifier (Qwen2.5-7B) that judges whether the final three sentences of the response contain the correct numerical answer, but no validation of this verifier is reported. If the verifier is systematically lenient, or if it is affected by the same fine-tuning data, the completion numbers in Tables 1 and 2 may not be comparable across models or to the published results marked with asterisks. The paper should report exact-match or rule-based numerical extraction as a second metric and give verifier agreement on a human-annotated sample.","section":"Section 4.1.3"},{"comment":"The evidence for the inference-time search is weak and partly inconsistent. The text states that search width 1 and width 16 are comparable, but Figure 4 shows sampling widths only from 20 to 24 and exhibits small, non-monotonic differences. Moreover, the fast verification mode checks only necessary conditions, which the authors themselves note can admit many invalid steps, and the random selection among valid steps may approximate random sampling. The paper should provide the full accuracy-versus-width curve, including the stated widths 1 and 16, report confidence intervals or significance tests, and clarify whether the observed differences support the claim that verification improves accuracy.","section":"Section 4.5 and Figure 4"}],"minor_comments":[{"comment":"There are incomplete and misspelled phrases, for example 'we find that the in MLLMs with symbolic systems' and 'the symblic system'; these typographical errors should be corrected.","section":"Section 1"},{"comment":"Typographical errors: 'constructoin' should be 'construction' in Section 3.1.2, and 'evaluation ion GeoTest' should be 'evaluation on GeoTest' in Section 4.3.","section":"Sections 3.1.2 and 4.3"},{"comment":"The sentence 'as shown in Figure 1' refers to a distribution of reasoning steps, but Figure 1 is the framework diagram; no figure showing the step-length distribution is provided.","section":"Section 3.2"},{"comment":"The x-axis is labelled 'Sampling Width' with ticks 20-24, which conflicts with the text's discussion of widths 1 and 16; the label, the values, or the text should be harmonized.","section":"Figure 4"},{"comment":"The combined evaluation set is called GeoTest, but Table 2 also reports MathVista-mini-GPS without clarifying whether it is part of GeoTest; the relationship between the two should be stated explicitly.","section":"Section 4.1.3"}],"recommendation":"major_revision","confidential_remarks":"The train/test overlap concern appears substantive: the paper never states that the 4,849 source images exclude the test splits, and the absence of any split statement is unusual for a pipeline built from benchmark images. I recommend that the editor require a clear statement and, if overlap exists, re-running the experiments with test images removed from the training expansion. A related but secondary concern is that GeoQA test results may be inflated for all compared methods that train on Geo170K, since that set is derived from GeoQA+."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful data-generation pipeline with a real evaluation problem. The paper builds GeoGen, which expands public geometry images into step-by-step QA pairs via symbolic forward search and traceback, adds 62K synthetic diagrams, and trains an LLM (GeoLogic) to translate natural-language reasoning into formal triples for inference-time verification. The integration is new and the datasets are likely to be reused by others in the geometry-MLLM subfield.\n\nWhat is actually good: the forward-search-plus-traceback target selection is a sensible way to get high-quality supervision from a symbolic solver, and the paper is honest that formal data are scarce and that prompt-based generation is unreliable. GeoLogic's verification, even in fast mode, is a reasonable first step. The cross-model consistency across InternVL, Qwen, and LLaVA supports the claim that the data help on GeoQA, which is not a source of GeoExpand.\n\nNow the soft spot, and it is the big one. Section 3.2 says GeoExpand generates 45,526 QA pairs from 4,849 original images in Geometry3K and PGPS9K. Section 4.1.3 evaluates on the test sets of those same two datasets. The paper never states that test images were excluded. When you train on a diagram and later ask the model to solve a new question about that same diagram, the visual input is already familiar; that can inflate both Completion and Choice numbers substantially. I do not see a split statement, an image-level deduplication step, or an evaluation on a held-out subset that would fix this. The MathVista-mini-GPS subset also draws from Geometry3K, so that number is suspect too. This is not a minor omission; it directly affects Tables 1 and 2. It is fixable, but the authors need to rerun on disjoint test images or show an overlap analysis before the headline claims can be taken at face value.\n\nTwo smaller issues. The completion metric relies on a Qwen2.5-7B answer checker rather than exact matching, which is reasonable but should be validated on a human-checked sample. And the T0–T3 ablation conflates data volume with source diversity; a scale-matched baseline would isolate the effect of symbolic generation.\n\nWho this is for: people working on geometry MLLMs or synthetic reasoning data. It deserves a serious referee—the engineering is real and the datasets are a contribution—but the current evaluation holds it back. My recommendation: send it to review with a firm request for split clarification and a rerun of the key tables on disjoint test images.","headline":"Solid symbolic-data-generation engineering, but the headline gains on Geometry3K and PGPS9K may be inflated by train/test overlap that the paper never addresses.","tokens_in":16359,"tokens_out":2464,"would_cite":false,"duration_ms":25681,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Symbolically generated geometry data and inference-time verification make multimodal LLMs solve geometry problems more accurately and with fewer hallucinations.","keywords":["geometry problem solving","multimodal large language models","symbolic reasoning","chain-of-thought","data synthesis","inference-time verification","GeoGen","GeoLogic"],"falsifier":"Inspect the list of 4,849 source images used for GeoExpand: if any official Geometry3K or PGPS9K test image is among them, rerun the GeoGen-SFT experiments with test images excluded and compare the accuracy tables. The presence of even one test image in the expansion sources would directly inflate the reported benchmark gains.","tokens_in":15286,"feed_emoji":"📐","tokens_out":10969,"duration_ms":104092,"temperature":0.7,"pith_summary":"The paper's central claim is that the scarcity of correct step-by-step geometry solutions is the main bottleneck for multimodal large language models on geometry problems, and that a formal symbolic reasoner can remove that bottleneck. GeoGen samples geometric premises, renders diagrams, runs theorem-based forward search to find derivable conclusions, and rewrites the resulting formal proofs into natural-language question-answer pairs. From this pipeline the authors build GeoExpand (about 45k samples over existing benchmark images) and GeoSynth (about 62k samples from freshly synthesized diagrams). They then train GeoLogic, an LLM that translates natural-language reasoning steps into formal statements a symbolic system can verify, and use it to run step-level tree search over a model's candidate reasoning steps at inference. The paper reports consistent accuracy gains across several multimodal LLMs and geometry benchmarks, arguing that symbolic supervision during training and symbolic checking during inference together make geometry reasoning more reliable.","feed_headline":"Training on symbolically proved solutions lifts MLLM geometry scores","feed_subtitle":"Auto-generated step-by-step solutions and inference-time symbolic verification raise accuracy across four benchmarks.","key_machinery":"The load-bearing object is the formal dependency graph produced by GeoGen's Target Finder. Starting from a set of geometry literals, forward search applies every theorem whose premises are satisfied and records each derived literal with its premises; a chosen target is then traced backward through this graph, the subgraph is topologically sorted, and linearized into condition-theorem-conclusion triples. These triples are the common currency of the whole method: they are rewritten into natural-language chain-of-thought solutions for training, they are the supervision that trains GeoLogic to translate natural language back into formal language, and they are the unit that the symbolic system verifies during inference. The Plotter renders the diagrams from the same sampled predicates, so every training image is aligned with the formal knowledge used to solve it.","core_discovery":"On its own terms, this paper establishes that a symbolic geometry engine can serve twice in a multimodal LLM pipeline: as a generator of formally correct reasoning data and as a verifier of the model's own reasoning. GeoGen produces 45,526 question-answer pairs from 4,849 existing images and 62,868 pairs from 129,230 synthesized diagrams, each with multi-step chain-of-thought solutions. Training on this data plus an existing instruction-tuning set improves completion and choice accuracy on the GeoQA, Geometry3K, PGPS9K, and MathVista-mini-GPS benchmarks for several 2B-7B multimodal models; the best 7B model reaches 63.9% completion and 74.0% choice accuracy on MathVista-mini-GPS. GeoLogic, a 3B translator model, converts each predicted natural-language step into a formal condition-theorem-conclusion triple, and the symbolic system checks it; step-level tree search over valid candidates improves accuracy up to a point, with the paper attributing the later decline to the verifier checking necessary but not sufficient conditions.","pith_inferences":["A direct extension would be to apply the same generate-and-verify loop to other multimodal reasoning domains that have formal checkers, such as arithmetic word problems, programming, or physics, by swapping the geometry theorem set for the domain's rules.","Because the fast verifier checks only necessary visual conditions, building a sufficiency-checking verifier, or scoring candidate steps by how much they contribute to the final answer, could plausibly reverse the observed accuracy decline at larger search widths.","The 94.16% translator accuracy implies roughly one in seventeen formal steps is mistranslated, so improving GeoLogic's translation or adding self-consistency across candidate translations may raise the ceiling of verification more than simply widening the search.","If the training-time gains survive a clean train/test split, the pipeline's value may be larger as a data generator than as a verifier, since the supervised fine-tuning gains are consistent across models while the inference-time gains are modest."],"forward_implications":["Adding GeoExpand and GeoSynth to the existing instruction-tuning data improves both completion and choice accuracy across several open multimodal LLMs from 2B to 7B parameters.","The GeoGen-SFT-7B model reaches 63.9% completion and 74.0% choice accuracy on the MathVista-mini-GPS geometry subset, and 78.0% choice accuracy on GeoQA, the best among the multimodal LLM solvers compared.","GeoLogic translates natural-language steps to formal triples with 94.16% exact-match accuracy on the authors' test set, which is what makes inference-time symbolic verification feasible.","Step-level tree search with symbolic verification improves accuracy as search width grows from 1 to 4, then declines at larger widths, consistent with the fast verification mode checking necessary conditions only."],"supporting_citations":[{"why":"Supplies the Geometry3K benchmark and formal-language annotations that GeoExpand expands and that test accuracy is measured against.","marker":"[22]"},{"why":"Supplies the PGPS9K benchmark, the second source of expanded images and one of the test sets.","marker":"[39]"},{"why":"Provides the symbolic theorem-search system whose predicates, theorems, and solver GeoGen uses to generate and verify reasoning paths.","marker":"[41]"},{"why":"Motivates the GeoGen design of random premise sampling, symbolic deduction, and traceback to retrieve solution paths.","marker":"[29]"},{"why":"Provides the Geo170K instruction-tuning data used as the training baseline and as the complement to GeoExpand and GeoSynth.","marker":"[12]"},{"why":"Provides the MathVista geometry subset used to evaluate generalization beyond the three training-derived benchmarks.","marker":"[21]"},{"why":"Provides the base multimodal LLM family on which GeoGen-SFT and inference-time verification experiments are run.","marker":"[2]"}],"fun_headline_variants":["Symbolic proof data and checkers lift MLLM geometry scores","Auto-generated formal proofs boost geometry reasoning in MLLMs","Double use of symbolic tools sharpens MLLM geometry solving","GeoGen and GeoLogic pair up to cut MLLM geometry hallucinations","Symbolic-neural pipeline raises geometry accuracy across four benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the GeoExpand training examples generated from the 4,849 Geometry3K and PGPS9K images are disjoint from the test splits of those same datasets; the paper reports accuracy on those test sets but never states that test images were excluded.","fun_headline_variants_meta":{"raw":{"variants":["Symbolic proof data and checkers lift MLLM geometry scores","Auto-generated formal proofs boost geometry reasoning in MLLMs","Double use of symbolic tools sharpens MLLM geometry solving","GeoGen and GeoLogic pair up to cut MLLM geometry hallucinations","Symbolic-neural pipeline raises geometry accuracy across four benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000177,"raw_usage":{"total_tokens":1318,"prompt_tokens":992,"completion_tokens":326,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":239}},"tokens_in":608,"tokens_out":326,"duration_ms":4439,"temperature":1.0,"reasoning_tokens":239,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:22:37.249594+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the list of 4,849 source images used for GeoExpand: if any official Geometry3K or PGPS9K test image is among them, rerun the GeoGen-SFT experiments with test images excluded and compare the accuracy tables. The presence of even one test image in the expansion sources would directly inflate the reported benchmark gains.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Geometry3K benchmark and formal-language annotations that GeoExpand expands and that test accuracy is measured against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the PGPS9K benchmark, the second source of expanded images and one of the test sets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MathVista geometry subset used to evaluate generalization beyond the three training-derived benchmarks."}],"review_version":1}