REVIEW 3 major objections 5 minor 34 references
Geo-LLaVA: A Large Multi-Modal Model for Solving Geometry Math Problems with Meta In-Context Learning
T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read A multimodal model given one retrieved solved geometry example before answering attains 65.25 percent on GeoQA+ and 42.36 percent on a new solid-geometry benchmark.
desk verdict The GeoMath dataset is a real gap-filler, but the SOTA claim is contradicted by the paper's own Table 2 and the ICL evaluation leaks from the training set. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is a two-part retrieval-and-generate loop. A dual-tower retrieval network, trained with a contrastive loss, maps the question text and image into a shared space and returns the single most similar training sample (K=1). That sample's text and image are merged, with images vertically stacked into one image because the backbone accepts a single image, and wrapped in a meta-prompt together with the current question. The backbone is a 13-billion-parameter vision-language model fine-tuned with LoRA on the meta-prompted data; at inference the same retrieval supplies an in-context example before the model generates a solution and answer. The 'meta' component is that retrieval is used both in the fine-tuning stage and at test time, so the model learns from the format of solved examples.
What would settle it
Compare each test question with its retrieved neighbor by image overlap and answer text; if a large share of neighbors share the answer or are near-duplicates, the in-context-learning gain is likely leakage. A stronger test would rebuild the retrieval pool excluding every training item similar to any test question above a threshold and rerun the evaluation; if accuracy falls sharply, the benchmark numbers measure memorization rather than problem solving.
Extended reading notes
Core claim
The central claim is that meta in-context learning, training the model on pairs of a question and a retrieved similar question with its solution, then supplying the same kind of retrieved example at inference, makes a 13-billion-parameter multimodal model competitive or better on geometry QA than much larger general models. Concretely, Geo-LLaVA records 65.25% accuracy on the selected GeoQA+ test questions and 42.36% on the selected GeoMath test questions, with the ablation showing each added component (image-context data, QA data, meta-training, and inference-time ICL) increasing accuracy. The paper further claims this is the first model to handle solid geometry problems, producing picture descriptions and reasoning steps for them.
Load-bearing premise
The reported accuracy rests on treating the single retrieved training question as a helpful worked example rather than a near-copy of the test question, since the retrieval pool is the same dataset used for fine-tuning and the test split is drawn from that dataset.
Editorial extensions
If this is right
- If the reported accuracies hold, geometry problem solving no longer requires a separate symbolic engine; a retrieval-augmented multimodal model can produce answers and readable reasoning steps directly.
- Solid geometry, previously almost absent from geometry-QA benchmarks, becomes an addressable task because the new GeoMath dataset provides images, questions, and reasoning steps.
- The ablation's stepwise gains imply that image-description data, QA fine-tuning, and meta-training each contribute independently, suggesting a recipe other visual reasoning tasks could copy.
- Because the method retrieves from a training pool at inference, it can be updated with new worked examples without retraining the generator, as long as the retrieval index is refreshed.
Reading between the lines
- The reported gains could be partly an artifact of retrieval leakage: the top retrieved neighbor is drawn from the same dataset used for fine-tuning, and the test split comes from that dataset's distribution, so a neighbor may be a near-duplicate whose answer is memorized rather than reasoned. The paper does not measure question-to-neighbor similarity or hold the retrieval pool out from the test se
- A straightforward test would replace the retrieved neighbor with a random or deliberately dissimilar example; if accuracy stays high, the gain is from training rather than from the inference-time retrieval.
- The same retrieval-plus-fine-tuning loop could transfer to other diagram-heavy mathematics, such as trigonometry or vector problems, provided the retrieval pool is cleanly separated from evaluation.
- The vertical merging of retrieved and target images into one input may blur spatial relationships; models that accept multiple images separately might show different gains, a comparison the paper does not run.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Geo-LLaVA, a roughly 13B-parameter large multimodal model for geometry problem solving, built on LLaVA-1.5 with LoRA fine-tuning and a CLIP/BERT dual-tower retrieval network. The authors introduce a new solid-geometry QA dataset called GeoMath, sourced from Chinese high school education websites and machine-translated into English, and combine retrieval-augmented supervised fine-tuning with inference-time in-context learning. The abstract and conclusion claim state-of-the-art results of 65.25% on GeoQA+ and 42.36% on GeoMath on 'selected questions,' with an ablation showing incremental gains from image-context data, QA data, meta-training, and inference-time ICL.
Significance. If the reported numbers were obtained under a controlled evaluation protocol, the paper would make a useful contribution: GeoMath is a new solid-geometry QA resource, and the combination of retrieval and in-context learning for geometry LMMs is timely. The paper also provides an ablation that attempts to isolate the effect of each component, and it compares against several public and proprietary baselines. However, the experimental protocol as reported does not support the headline claims: the state-of-the-art claim is contradicted by the paper's own Table 2, the test-subset selection is never specified, and the inference-time retrieval procedure risks answer leakage from the training set. No code, model, or test-selection script is released, so the contributions cannot be independently validated in their current form.
major comments (3)
- [Abstract; §4.3, Table 2] The abstract claims state-of-the-art performance of 65.25% on GeoQA+, but Table 2 reports G-llava-13B at 67.00% on the same dataset, which is higher than Geo-LLaVA's 65.25%. If the two numbers were obtained on the same test split, the SOTA claim is false; if they were obtained on different splits, the claim is unsubstantiated because the paper never defines what 'selected questions' means, never states the selection criteria, and never re-evaluates G-llava or other baselines on the same subset. The paper must specify the exact test split, re-run all baselines on that split, and release the selection script; without this, the headline comparison is not decidable from the manuscript.
- [§3.4; §4.2.1] The retrieval protocol is both internally ambiguous and a source of potential answer leakage. Section 3.4 says K is set to 1 for retrieving the most similar training sample at inference, while Section 4.2.1 says the retrieval model selects the top 5 samples to construct the GeoMath-Meta data; the relationship between these two settings is not explained. More importantly, because the retrieval pool is the same training set used for fine-tuning and the test split is drawn from the same distribution, the nearest training neighbor may be a near-duplicate whose answer the model can copy rather than reason through. The paper does not quantify retrieval similarity, does not hold out the retrieval pool, and does not report a random-retrieval or no-retrieval control that would separate genuine in-context reasoning from memorization. A controlled comparison with the retrieval pool held out, with near-duplicates removed, and with random and oracle retrieval baselines is needed to support the claimed ICL gains.
- [§4.3] The text states that the model was fine-tuned five times to compute the mean and standard deviation of the evaluation metric, but Table 2 reports only point estimates with no standard deviations, confidence intervals, or significance tests. This matters because the central SOTA difference (Geo-LLaVA 65.25 vs. G-llava 67.00) is 1.75 percentage points, and the final ICL ablation gain is only 0.88 percentage points; without variance information, neither the SOTA claim nor the ICL benefit can be distinguished from noise. The authors should report the standard deviations and specify the number of seeds used for each configuration.
minor comments (5)
- [§4.1, Table 1] The dataset name is inconsistently spelled as 'PSDK9K' and 'PSDK-9K'; it should be PGPS9K to match the reference [33].
- [Introduction; References] GeoQA+ is cited as [11] in the introduction and as [9] in the experiments section; the reference numbering should be harmonized.
- [§3.4, §4.2.1] The term 'meta-training' is used loosely: the method concatenates retrieved examples during supervised fine-tuning rather than performing task-level meta-learning across tasks. The authors should either align the terminology with the MetaICL literature and discuss the relationship, or rename the component to avoid overclaiming novelty.
- [Figure 2] The left panel contains repeated handwritten-style text 'K Retrieval Questions' and the pipeline labels are difficult to read; a vectorized figure with clearly separated training and inference stages would improve reproducibility of the described protocol.
- [§4.2.2] The LoRA configuration is not specified: the paper states that LoRA is used but does not report rank, alpha, or which modules are adapted. These details are needed for replication.
Circularity Check
Retrieval-based ICL puts training-set answers into the prompt, so the reported benchmark gains can reduce to copying; the 'state-of-the-art' claim is also not supported by the paper's own Table 2.
-
fitted input called prediction
[Section 3.4, 'Enhancing In-Context Learning', and Table 2 (GeoQA+ / GeoMath results)]
"Contextual Retrieval: For each input sample, we retrieved the K most similar samples from the training data, ensuring that the input sample itself is excluded, where K is set to 1 in this paper. This retrieval is based on semantic similarity metrics, ensuring the context is highly relevant. 2) Concatenation and Fine-Tuning: The retrieved texts and corresponding images are concatenated with the input sample."
At inference (Table 2, 'with ICL'), the same retrieval mechanism supplies one training sample whose SOLUTION_k and ANSWER_k are inserted verbatim into the prompt (Figure 2: 'including the <QUESTIONk, SOLUTIONk, ANSWERk>'). Since the retrieval pool is the training split of the same GeoQA+/GeoMath datasets from which the test questions are drawn, and only the exact input instance is excluded, a test question whose nearest neighbor is a duplicate or near-duplicate has its correct answer already present in the input. The reported ICL gain (+2.12 on GeoQA+, +0.88 on GeoMath) is therefore not a clean measure of reasoning: for such samples the model can copy the retrieved answer.
full rationale
The paper's central derivation chain is the GeoMath dataset plus Geo-LLaVA's retrieval-augmented fine-tuning and ICL. The dataset construction is independent and not circular. The main circularity is in the evaluation protocol: the ICL retriever is trained on the same data distribution as the test set, and at inference the K=1 retrieved training sample's solution and answer are placed directly in the prompt. For test questions that are duplicates or near-duplicates of training questions, the correct answer is an input token, so the 'prediction' reduces to copying rather than reasoning. The paper explicitly excludes only the exact input sample and never quantifies retrieval similarity or performs deduplication. Separately, and not as circularity, the abstract's SOTA claim is contradicted by the paper's own Table 2, where G-llava-13B scores 67.00% on GeoQA+ versus Geo-LLaVA's 65.25%, and the term 'selected questions' is never defined; this is a missing-support problem that further weakens the headline result. No load-bearing self-citations or imported uniqueness theorems were found, so the circularity score is moderate rather than extreme.
Assumptions & free parameters
free parameters (4)
- K (number of retrieved in-context examples at inference) =
1
- Top-k for meta-training retrieval =
5
- LoRA learning rate =
2e-4
- Number of fine-tuning epochs =
5
assumptions (3)
- domain assumption ChatGPT3.5 translation from Chinese to English preserves the geometric content and reasoning steps of the dataset.
- ad hoc to paper Retrieving the most similar training sample for ICL improves reasoning without leaking the answer.
- ad hoc to paper The selected test subsets are representative and the selection bias does not favor Geo-LLaVA.
Cite this review
Pith. "Pith review of Geo-LLaVA: A Large Multi-Modal Model for Solving Geometry Math Problems with Meta In-Context Learning." pith.science (2026). https://pith.science/paper/YYDJOMVW
@misc{pith2026241210455,
author = {Pith},
title = {Pith review of: Geo-LLaVA: A Large Multi-Modal Model for Solving Geometry Math Problems with Meta In-Context Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/YYDJOMVW}},
note = {Machine review of arXiv:2412.10455}
}
read the original abstract
Geometry mathematics problems pose significant challenges for large language models (LLMs) because they involve visual elements and spatial reasoning. Current methods primarily rely on symbolic character awareness to address these problems. Considering geometry problem solving is a relatively nascent field with limited suitable datasets and currently almost no work on solid geometry problem solving, we collect a geometry question-answer dataset by sourcing geometric data from Chinese high school education websites, referred to as GeoMath. It contains solid geometry questions and answers with accurate reasoning steps as compensation for existing plane geometry datasets. Additionally, we propose a Large Multi-modal Model (LMM) framework named Geo-LLaVA, which incorporates retrieval augmentation with supervised fine-tuning (SFT) in the training stage, called meta-training, and employs in-context learning (ICL) during inference to improve performance. Our fine-tuned model with ICL attains the state-of-the-art performance of 65.25% and 42.36% on selected questions of the GeoQA dataset and GeoMath dataset respectively with proper inference steps. Notably, our model initially endows the ability to solve solid geometry problems and supports the generation of reasonable solid geometry picture descriptions and problem-solving steps. Our research sets the stage for further exploration of LLMs in multi-modal math problem-solving, particularly in geometry math problems.
Figures
Reference graph
Works this paper leans on
-
[1]
Gpt-4v(ision) system card
-
[2]
L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.Gpt-4 technical report
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)
arXiv 2023
-
[3]
Flamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems 35 (2022), 23716–23736
work page 2022
-
[4]
Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al.Palm 2 technical report. arXiv preprint arXiv:2305.10403 (2023)
arXiv 2023
-
[5]
Openflamingo: An open-source framework for training large autoregressive vision-language models
Awadalla, A., Gao, I., Gardner, J., Hessel, J., Hanafy, Y., Zhu, W., Marathe, K., Bitton, Y., Gadre, S., Sagawa, S., et al. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390 (2023)
arXiv 2023
-
[6]
Azerbayev, Z., Schoelkopf, H., Paster, K., Santos, M. D., McAleer, S., Jiang, A. Q., Deng, J., Biderman, S., and Welleck, S. Llemma: An open language model for mathematics. arXiv preprint arXiv:2310.10631 (2023)
arXiv 2023
-
[7]
Qwen-vl: A frontier large vision-language model with versatile abilities
Bai, J., Bai, S., Y ang, S., W ang, S., Tan, S., W ang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966 (2023)
arXiv 2023
-
[8]
Bordes, F., Pang, R. Y., Ajay, A., Li, A. C., Bardes, A., Petryk, S., Mañas, O., Lin, Z., Mahmoud, A., Jayaraman, B., Ibrahim, M., Hall, M., Xiong, Y., Lebensold, J., Ross, C., Jayakumar, S., Guo, C., Bouchacourt, D., Al-Tahan, H., Padthe, K., Sharma, V., Xu, H., Tan, X. E., Richards, M., Lavoie, S., Astolfi, P., Hemmat, R. A., Chen, J., Tirumala, K., Ass...
Show all 34 references
-
[9]
An augmented benchmark dataset for geometric ques- tion answering through dual parallel text encoding
Cao, J., and Xiao, J. An augmented benchmark dataset for geometric ques- tion answering through dual parallel text encoding. In Proceedings of the 29th International Conference on Computational Linguistics (2022), pp. 1511–1520
2022
-
[10]
Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression
Chen, J., Li, T., Qin, J., Lu, P., Lin, L., Chen, C., and Liang, X. Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression. arXiv preprint arXiv:2212.02746 (2022)
2022 arXiv
-
[11]
P., and Lin, L
Chen, J., Tang, J., Qin, J., Liang, X., Liu, L., Xing, E. P., and Lin, L. Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning. arXiv preprint arXiv:2105.14517 (2021)
2021 arXiv
-
[12]
Dai, W., Li, J., Li, D., Tiong, A. M. H., Zhao, J., W ang, W., Li, B., Fung, P., and Hoi, S. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023
2023
-
[13]
Bert: Pre-training of deep bidirectional transformers for language understanding
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
2018 arXiv
-
[14]
An image is worth 16x16 words: Transformers for image recognition at scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Un- terthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
2020 arXiv
-
[15]
Model-agnostic meta-learning for fast adaptation of deep networks
Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning (2017), PMLR, pp. 1126–1135
2017
-
[16]
G-llava: Solving geometric problem with multi-modal large language model
Gao, J., Pi, R., Zhang, J., Ye, J., Zhong, W., W ang, Y., Hong, L., Han, J., Xu, H., Li, Z., et al. G-llava: Solving geometric problem with multi-modal large language model. arXiv preprint arXiv:2312.11370 (2023)
2023 arXiv
-
[17]
Google bard
Google. Google bard. https://bard.google.com, 2024. Accessed: 2024-07-05
2024
-
[18]
M., and Le, Q
Ha, D., Dai, A. M., and Le, Q. V. Hypernetworks. In ICLR (2022)
2022
-
[19]
J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)
2021 arXiv
-
[20]
Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models
Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In International conference on machine learning (2023), PMLR, pp. 19730–19742
2023
-
[21]
Unimath: A foundational and multimodal mathematical reasoner
Liang, Z., Yang, T., Zhang, J., and Zhang, X. Unimath: A foundational and multimodal mathematical reasoner. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (2023), pp. 7126–7133
2023
-
[22]
Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. Advances in neural information processing systems 36 (2024)
2024
-
[23]
L., Hoqe, E., and Joty, S
Masry, A., Kavehzadeh, P., Do, X. L., Hoqe, E., and Joty, S. Unichart: A uni- versal vision-language pretrained model for chart comprehension and reasoning. arXiv preprint arXiv:2305.14761 (2023)
2023 arXiv
-
[24]
Metaicl: Learning to learn in context
Min, S., Lewis, M., Zettlemoyer, L., and Hajishirzi, H. Metaicl: Learning to learn in context. arXiv preprint arXiv:2110.15943 (2021)
2021 arXiv
-
[25]
W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning (2021), PMLR, pp. 8748–8763
2021
-
[26]
W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In ICML (2021), vol. 139 of Proceedings of Machine Lear...
2021
-
[27]
Optimization as a model for few-shot learning
Ravi, S., and Larochelle, H. Optimization as a model for few-shot learning. In ICLR (2016)
2016
-
[28]
Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-
Schmidhuber, J. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook. PhD thesis, Technische Universität München, 1987
1987
-
[29]
Llama 2: Open foundation and fine-tuned chat models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)
2023 arXiv
-
[30]
H., Wu, Y., Le, Q
Trinh, T. H., Wu, Y., Le, Q. V., He, H., and Luong, T.Solving olympiad geometry without human demonstrations. Nature 625, 7995 (2024), 476–482
2024
-
[31]
Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning
W ang, K., Ren, H., Zhou, A., Lu, Z., Luo, S., Shi, W., Zhang, R., Song, L., Zhan, M., and Li, H. Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning. arXiv preprint arXiv:2310.03731 (2023)
2023 arXiv
-
[32]
Large language models are better reasoners with self-verification
Weng, Y., Zhu, M., Xia, F., Li, B., He, S., Liu, S., Sun, B., Liu, K., and Zhao, J. Large language models are better reasoners with self-verification. arXiv preprint arXiv:2212.09561 (2022)
2022 arXiv
-
[33]
A multi-modal neural geometric solver with textual clauses parsed from diagram
Zhang, M.-L., Yin, F., and Liu, C.-L. A multi-modal neural geometric solver with textual clauses parsed from diagram. arXiv preprint arXiv:2302.11097 (2023)
2023 arXiv
-
[34]
Minigpt-4: Enhancing vision-language understanding with advanced large language models
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 (2023)
2023 arXiv
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.