{"id":"ea60c8c0-aaa5-4123-8ade-e3acb62e00a1","arxiv_id":"2412.11863","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"GeoX achieves state-of-the-art accuracy on GeoQA, UniGeo, Geometry3K, and PGPS9K geometry benchmarks using formalized vision-language pre-training with a Generator-and-Sampler Transformer.","lead":"GeoX is a multimodal AI system that solves plane geometry problems by turning diagrams and questions into formal program-like steps that a symbolic solver checks. It reports state-of-the-art results on four geometry benchmarks using geometry-specific pre-training before fine-tuning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing risk is cross-dataset leakage: the 100M-token pre-training corpus draws on the same GeoQA/UniGeo/G-LLaVA sources as the test sets, and the paper reports no cross-dataset deduplication, so the near-perfect UniGeo numbers could reflect memorization rather than geometric reasoning.","rationale":"The reader's weakest-assumption analysis and my stress-test converge on the same point: the absence of any described cross-dataset deduplication makes the benchmark results potentially uninterpretable. This is load-bearing because the paper's headline state-of-the-art claim depends on the test sets being clean, and the suspiciously high UniGeo proving scores make the risk concrete rather than hypothetical. I do not see a need to change the reader's CONDITIONAL verdict: the concern is substantial but not yet confirmed, and the paper's internal ablations and released code would provide meaningful support if a deduplication check comes back clean. I am not accusing the authors of deliberate leakage; the issue is that the described data pipeline does not rule it out. The proposed test is deliberately simple and decisive: compare the actual corpus against the actual test sets, and if overlaps exist, recompute the headline numbers on deduplicated data. Secondary issues such as zero-shot versus fine-tuned comparison and missing variance are real but less fundamental, because the specialist comparisons and ablation tables partially address the method's contribution. The main risk remains contamination of the evaluation sets through the pre-training corpus.","tokens_in":22626,"tokens_out":6054,"duration_ms":58004,"concrete_test":"Download the released pre-training corpus and every test set used in Tables 1-4 (GeoQA test, UniGeo test, Geometry3K test, PGPS9K test, MathVista-GEO). Normalize each sample by stripping translations, canonicalizing formal program tokens, and removing formatting. Then run exact-match plus near-duplicate matching (e.g., MinHash on question-plus-program strings, plus image hashing for diagrams) between every corpus document and every test example. Report overlap counts per benchmark. If any overlap exists, remove those corpus documents, retrain or re-evaluate, and recompute Tables 1-4; if the near-perfect UniGeo numbers survive deduplication, the concern is closed. A faster secondary check: verify directly whether UniGeo's calculation train/test split is identical to GeoQA's; if it is not, contamination is already present.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 4.2 is that GeoX significantly outperforms generalists and specialists on GeoQA, UniGeo, Geometry3K, and PGPS9K, with ablations (Tables 5, 6) attributing the gain to formalized alignment and GS-Former. The weakest point in this argument is the training/evaluation boundary. Appendix E says the 100M-token corpus was built \"only from the training portions from the existing geometric datasets to prevent label leakage,\" but it describes no deduplication across datasets. The corpus sources include GeoQA, UniGeo, Geometry3K, PGPS9K, PGDP5K, and G-LLaVA. UniGeo's calculation collection is the same 4,998 GeoQA problems; if UniGeo's split is not byte-identical to GeoQA's 7:1.5:1.5 split, GeoQA test items can enter pre-training under UniGeo's training split. Similarly, G-LLaVA is a GPT-generated QA set built with equation solving, value scaling, and paraphrasing; if it contains paraphrases of benchmark test items, those items also leak. The UniGeo proving results in Table 2 (97.8%-100% top-1, 99.5% top-10) are exactly the kind of near-saturation numbers that memorization of short formal proof programs would produce. The paper offers no overlap statistics, no deduplication procedure, and no release of the filtered corpus, so the state-of-the-art claim is not yet secure against this failure mode.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"GeoX is a multimodal geometry problem-solving model trained in three stages: unimodal pre-training (MAE-based Geo-ViT on 120K diagrams, auto-regressive fine-tuning of LLEMMA-7B on a 100M-token geometric corpus), geometry-language alignment with a new GS-Former (Geo-aware Query Generator and Semantics-guided Geometry Sampler), and end-to-end visual instruction tuning. The model outputs formal program sequences that are executed by a symbolic solver. Experiments on GeoQA, UniGeo, Geometry3K, PGPS9K, and MathVista-GEO report state-of-the-art results, and ablations attribute the gains to the proposed formalized alignment and to the GS-Former components.","tokens_in":22919,"tokens_out":4604,"duration_ms":38808,"significance":"If the results are robust, the paper makes a useful contribution by demonstrating that formalized (program-sequence) vision-language pre-training can outperform both generalist MLLMs and task-specific geometry solvers, and by introducing an architecture (GS-Former) with content-aware query generation and sparsification. The paper includes helpful ablations (Tables 5 and 6) that isolate the contributions of geometry-language alignment and of the GQG/SGS modules, and it releases code and data, which supports reproducibility. The headline SOTA numbers, however, are credible only if the training/evaluation boundary is contamination-free; the paper does not yet establish this, and the comparison against generalist models is confounded by fine-tuning status.","major_comments":[{"comment":"The 100M-token geometric corpus is built from the training portions of GeoQA, UniGeo, Geometry3K, PGPS9K, PGDP5K, and G-LLaVA. Since UniGeo's calculation problems are exactly the 4,998 GeoQA problems, and since G-LLaVA was generated from GeoQA problems via equation solving, value scaling, and paraphrasing, test items from GeoQA and UniGeo may appear verbatim or near-verbatim in the pre-training corpus. The paper reports no overlap statistics, no cross-dataset deduplication procedure, and no release of the filtered corpus. The near-saturated UniGeo proving results in Table 2 (97.8%-100% top-1) are exactly the pattern one would expect from memorization of short formal proof programs. To support the central claim, the authors must quantify the overlap between the corpus and each test split (exact and near-duplicate matching) and re-run the experiments on a deduplicated corpus, reporting which numbers change.","section":"Appendix E / Sec. 4.2 / Table 2"},{"comment":"The headline 'significantly outperforms' claim compares GeoX, which is fine-tuned on each benchmark's training split (hyperparameters in Appendix F, Table 11), against generalist MLLMs (GPT-4V, Qwen-VL, LLaVA-v1.5, mPLUG-Owl2) that are evaluated zero-shot. This confounds model quality with fine-tuning advantage, and the paper reports no error bars or significance tests across random seeds. A fair evaluation would either fine-tune the generalist baselines on the same instruction data or evaluate GeoX in a zero-shot/few-shot setting. Without this, the magnitude of the reported gains (e.g., 54.9% vs 43.4% on GeoQA, 58.6% vs 34.8% Completion on Geometry3K) cannot be attributed to the proposed pre-training paradigm.","section":"Sec. 4.2 / Tables 1-3 / Appendix F"},{"comment":"The paper states that to make a fair comparison with specialists, it removes Diagram GT annotations and 'replicate[s] these methods under the original conditions,' but it provides no details of this replication (training data, hyperparameters, number of runs). Since the reproduced numbers for PGPSNet and Geoformer in Table 3 underpin the SOTA claim, the authors should either describe the replication protocol or report the original published numbers with a clear statement of differences.","section":"Sec. 4.2 / Table 3"}],"minor_comments":[{"comment":"The benchmark name is written as 'PGPS9k' in the abstract and Section 1 but as 'PGPS9K' elsewhere; please unify.","section":"Abstract / Secs. 1, 4"},{"comment":"Two different code URLs are given: github.com/Alpha-Innovator/GeoX in the abstract and github.com/UniModal4Reasoning/GeoX in the appendix; please ensure the final version points to the correct repository.","section":"Abstract / Appendix G"},{"comment":"The Gumbel-Softmax reparameterization is applied to a binary mask M, but the paper does not specify whether hard or soft samples are used during training and inference; this should be clarified because it affects the optimization and the interpretation of the sparsification term.","section":"Eq. (1)"},{"comment":"GS-Former is initialized with BERT weights, but BERT is an encoder-only model while GS-Former includes cross-attention and is later used in a generation-oriented pipeline; a short explanation of how the initialization is applied and why it is appropriate would improve clarity.","section":"Sec. 3.3.2"},{"comment":"Figure 3 reports performance values but does not show numerical labels, error bars, or the definition of the y-axis; adding these would make the unimodal pre-training comparison more interpretable.","section":"Sec. 4.4 / Fig. 3"},{"comment":"The second row labels the Alignment column as '×' while the Language column is also '×'; this is ambiguous. Use explicit entries such as 'None' or 'No alignment' for readability.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The most serious concern is the lack of cross-dataset deduplication between the 100M-token pre-training corpus and the evaluation splits, especially given the known overlap between UniGeo and GeoQA. This is a fixable issue but requires substantial re-analysis. The comparison against generalist MLLMs is also confounded by fine-tuning differences. The paper's own ablations are clean and support the component-wise claims, so I see a path to acceptance, but only after the authors address these two load-bearing issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a serious look, but the main claim needs work before I'd trust the numbers. What's genuinely new: GS-Former's dynamic query generation and semantics-guided sampling are not in the cited Q-Former/Resampler line, and the formalized diagram-caption dataset (6,232 expert-annotated images) is a useful contribution. The three-stage recipe is mostly known components—MAE, LLEMMA fine-tuning, instruction tuning—but it's well executed, and Tables 5 and 6 ablations cleanly attribute gains to the new pieces. The solver-based output is a real plus: it makes answers verifiable, not just plausible.\n\nThe soft spots are proportionate but real. The headline comparisons in Tables 1–3 are asymmetric: GeoX is fine-tuned on each benchmark while GPT-4V and Qwen-VL are evaluated zero-shot. No error bars or significance tests appear anywhere. That alone would push me to \"conditional,\" not \"reject.\" The more serious issue is leakage. Appendix E says the 100M-token corpus was built only from training splits \"to prevent label leakage,\" but there is no cross-dataset deduplication described. UniGeo's calculation set is the same 4,998 GeoQA problems, so if the two datasets' train/test boundaries differ, GeoQA test items can enter pre-training through UniGeo's training split. G-LLaVA, built by GPT-assisted equation solving and paraphrasing, is another plausible contamination vector. The near-perfect UniGeo proving results (97.8–100%) are exactly what memorization of short formal programs looks like. The paper needs overlap statistics, a deduplication procedure, and release of the filtered corpus. Without those, the SOTA claim is not secure.\n\nThe Appendix A \"theoretical analysis\" is post hoc and asserts rather than proves; it can be cut or reframed as intuition. That's a minor issue.\n\nWho should read this: anyone building multimodal math or geometry reasoning systems. It's a useful baseline and the formal-language alignment idea is worth testing independently. My verdict: send it to peer review, with major revision focused on (a) fine-tuned generalist baselines with variance, (b) cross-dataset deduplication analysis, and (c) releasing the filtered corpus. If the leakage check comes back clean, this is a solid paper. If not, the headline results need to be revised substantially.","headline":"A solid three-stage geometry MLLM pipeline with genuinely new alignment components, but the headline SOTA claim is not yet trustworthy until the cross-dataset leakage risk and asymmetric baselines are addressed.","tokens_in":644,"tokens_out":1142,"would_cite":true,"duration_ms":28748,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GeoX claims that geometry problem solving by multimodal LLMs is best advanced by pre-training a vision encoder and language decoder separately on geometry data, then aligning them through formalized diagram captions before instruction…","keywords":["geometry problem solving","multimodal large language models","formalized pre-training","vision-language alignment","GS-Former","diagram understanding","symbolic solver","GeoX"],"falsifier":"Audit the overlap: extract the question text and program annotations from the GeoQA and UniGeo test splits and search for exact or near-exact matches in the 100M-token geometric corpus and in the instruction-tuning splits described in Appendix E. If any test item appears in training data, comparisons are confounded; retraining on a deduplicated corpus and re-running GeoQA, UniGeo, Geometry3K, and PGPS9K would settle whether the formalized pre-training gains survive.","tokens_in":22387,"feed_emoji":"📐","tokens_out":9007,"duration_ms":73155,"temperature":0.7,"pith_summary":"GeoX is a proposal for teaching a multimodal large model to solve plane-geometry problems by generating formal program sequences that a symbolic solver can execute and verify. The paper's central claim is that the decisive ingredient is not the base language model or the solver, but a three-stage pre-training scheme: geometry-only masked auto-encoding for the vision encoder, auto-regressive fine-tuning on a 100M-token geometry corpus for the language decoder, and a Generator-and-Sampler Transformer that aligns diagrams with formalized captions before instruction tuning. On four public benchmarks the model reports top-1 accuracy of 54.9% on GeoQA, 54.4% on UniGeo calculation, 58.6% Completion on Geometry3K, 52.7% Completion on PGPS9K, and 72.6% on MathVista-GEO, exceeding both generalist MLLMs and geometry specialists. The broader interest is that the outputs are verifiable: because the model emits formal programs rather than free-form explanations, claim and reasoning can be checked automatically.","feed_headline":"GeoX beats geometry specialists via formalized pre-training","feed_subtitle":"Three-stage recipe reports 54.9% top-1 on GeoQA and 58.6% Completion on Geometry3K.","key_machinery":"The mechanism that carries the argument is the Generator-and-Sampler Transformer (GS-Former), a cross-modal alignment module that converts a geometric diagram into a small set of discriminative queries. Its Geo-aware Query Generator builds sample-specific queries by pooling and projecting the diagram's visual features, and its Semantics-guided Geometry Sampler learns a binary mask that progressively drops uninformative patches, regularized by a sparsity term so the mask does not collapse to all-ones. GS-Former is trained on a 6,232-image dataset of formalized captions—only collinear (Line ...) and concyclic (\\odot ... lieson ...) point relations—with contrastive, matching, and caption-generation losses. The updated queries are projected into the token space of Geo-LLM-7B, which decodes a formal program that a symbolic solver executes to yield the answer.","core_discovery":"The central claim is that automatic geometry problem solving is not primarily limited by the base language model's reasoning ability but by the mismatch between natural-image and natural-text pre-training and the sparse, formal nature of geometric diagrams and symbols. GeoX therefore stages training: masked auto-encoding teaches Geo-ViT to represent diagrams; auto-regressive fine-tuning on a 100M-token geometry corpus teaches Geo-LLM to speak the formal language; GS-Former aligns the two modalities using formalized captions of collinear and concyclic point relations; and visual instruction tuning adapts the whole system to emit solver-executable program sequences. The paper reports that this sequence of choices outperforms generalists and specialists on GeoQA, UniGeo, Geometry3K, and PGPS9K, and it attributes the gains to the formalized alignment via the GS-Former rather than to the LLEMMA base or the external solver, citing ablations that drop from 58.6% to 33.1% Completion on Geometry3K when alignment is removed and from 58.6% to 55.7% when formal captions are replaced by natural-language captions.","pith_inferences":["Beyond the paper, the GS-Former sparsification idea should transfer to other diagram-heavy domains where information is sparse, such as circuit diagrams or molecular drawings, but GeoX does not test this.","The Appendix D 'random-guess correction' is a paper-specific metric; if the same correction were applied to G-LLaVA and MAVIS under exact-process scoring, the ranking in Table 8 could change.","The cleanest test the paper leaves open is cross-dataset deduplication: since UniGeo's calculation split is identical to GeoQA, an overlap audit between the pre-training corpus and test splits would separate memorization from generalization.","The recipe's dependence on the base LLM is not isolated; a controlled experiment swapping LLEMMA for another math-pretrained 7B model would show how much of the gain is attributable to the formalized pre-training alone."],"forward_implications":["Unified generalists become viable: one GeoX model reports state-of-the-art numbers on GeoQA, UniGeo, Geometry3K, and PGPS9K, so per-dataset specialist architectures are no longer the default choice for geometry problem solving.","Answers become auditable: because the model emits formal program sequences executed by a symbolic solver, any output can be checked step-by-step, and the paper's evaluation treats process errors as incorrect rather than crediting lucky final digits.","Formal language is the better alignment target: the ablation showing formal captions beat natural-language captions by 2.9 Completion points on Geometry3K implies that future diagram-caption datasets should encode structural relations like collinearity and concyclicity.","The near-perfect UniGeo proving scores (97.8–100%) suggest that proof generation for these datasets is largely solvable once the model is geometry-pretrained, conditional on the training and evaluation protocol being leak-free."],"supporting_citations":[{"why":"Supplies GeoQA, the evaluation benchmark and the NGS program-solver paradigm the paper builds on.","marker":"Chen et al. (2021)"},{"why":"Supplies UniGeo and the reformulation of proofs and calculations into unified sequence formats.","marker":"Chen et al. (2022)"},{"why":"Supplies Geometry3K and the InterGPS formal-language and symbolic-reasoning approach.","marker":"Lu et al. (2021)"},{"why":"Supplies PGPS9K, the Completion, Choice, and Top-3 metrics, and the PGPSNet specialist baseline.","marker":"Zhang et al. (2023c)"},{"why":"Supplies LLEMMA-7B, the base language model that becomes Geo-LLM after geometry fine-tuning.","marker":"Azerbayev et al. (2023)"},{"why":"Supplies the Q-Former-style alignment objectives (contrastive, matching, caption generation) that GS-Former adapts.","marker":"Li et al. (2023b)"},{"why":"Supplies the masked auto-encoding recipe used to pre-train the Geo-ViT diagram encoder.","marker":"He et al. (2022)"},{"why":"Supplies the G-LLaVA geometry dataset used in the geometric corpus and a solver-free comparison point.","marker":"Gao et al. (2023)"}],"fun_headline_variants":["GeoX: A three-stage recipe for geometry problem solving","GeoX fixes geometry AI with formalized diagram training","GeoX bridges diagram and symbol gap for geometry reasoning","GeoX: Aligning vision and language for geometry tasks","GeoX outscores specialists on geometry benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that no benchmark test question ever appears in the model's own training data through dataset overlap; the paper states that the 100M-token corpus was built only from training splits, but it does not describe cross-dataset deduplication, and because UniGeo's calculation split is the same 4,998 problems as GeoQA with both used in pre-training, instruction tuning, and evaluation, a leaked test question would make the reported gains—including near-perfect UniGeo proving scores—memorization rather than geometric reasoning.","fun_headline_variants_meta":{"raw":{"variants":["GeoX: A three-stage recipe for geometry problem solving","GeoX fixes geometry AI with formalized diagram training","GeoX bridges diagram and symbol gap for geometry reasoning","GeoX: Aligning vision and language for geometry tasks","GeoX outscores specialists on geometry benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000694,"raw_usage":{"total_tokens":3178,"prompt_tokens":1019,"completion_tokens":2159,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":2083}},"tokens_in":635,"tokens_out":2159,"duration_ms":13822,"temperature":1.0,"reasoning_tokens":2083,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:30:32.258096+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit the overlap: extract the question text and program annotations from the GeoQA and UniGeo test splits and search for exact or near-exact matches in the 100M-token geometric corpus and in the instruction-tuning splits described in Appendix E. If any test item appears in training data, comparisons are confounded; retraining on a deduplicated corpus and re-running GeoQA, UniGeo, Geometry3K, and PGPS9K would settle whether the formalized pre-training gains survive.","supporting_citations":[{"cited_title":"Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression","cited_arxiv_id":null,"evidence_quote":"Supplies UniGeo and the reformulation of proofs and calculations into unified sequence formats."},{"cited_title":"Geodrl: A self-learning framework for geometry problem solving using reinforcement learning in deductive reasoning","cited_arxiv_id":null,"evidence_quote":"Supplies the G-LLaVA geometry dataset used in the geometric corpus and a solver-free comparison point."}],"review_version":1}