Improving Multimodal LLMs Ability In Geometry Problem Solving, Reasoning, And Multistep Scoring

Abhishek Dharmadhikari; Atharva Marathe; Avinash Anand; Harshil Mital; Harsh Parimal Popat; Kritarth Prasad; Rajiv Ratn Shah; Raj Jaiswal; Roger Zimmermann

REVIEW 5 major objections 6 minor 46 references

Improving Multimodal LLMs Ability In Geometry Problem Solving, Reasoning, And Multistep Scoring

T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning on GPSM4K, a new grade 7-12 geometry dataset with step-by-step solutions, raises an open vision-language model's accuracy from 8% to 22.66%, and captions and retrieval augmentation add further gains.

desk verdict Useful new geometry dataset, but every reported number rests on an unvalidated Gemini Pro judge that also generated the training data. read the letter →

arxiv 2412.00846 v1 pith:JI4L5RVK submitted 2024-12-01 cs.AI

Avinash Anand , Raj Jaiswal , Abhishek Dharmadhikari , Atharva Marathe , Harsh Parimal Popat , Harshil Mital , Kritarth Prasad , Rajiv Ratn Shah

show 1 more author

Roger Zimmermann

This is my paper · ORCID

classification cs.AI

keywords multimodallargelanguagemodelsgeometryproblemsolvingGPSM4Kstep-by-stepsolutionsimagecaptioningretrieval-augmentedgenerationautomatedanswerevaluationchain-of-thoughtscoring

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a dataset of geometry problems with consistent step-by-step solutions is enough to make open-source vision-language models substantially better at solving geometry. It introduces GPSM4K, drawn from grades 7-12 textbooks, with numerical-answer and theorem-proving questions and two solutions per problem: the textbook original and a decomposed version generated by a multimodal model. Fine-tuning on GPSM4K raises a LLaVA 1.6 Vicuna 7B model from 8% accuracy (after fine-tuning on PGPS9K) to 22.66% on the paper's test set, with similar jumps for 13B and Mistral variants. The paper also reports that adding image captions to prompts improves results, human captions more than machine captions, and that a one-shot retrieval-augmented generation context adds a few points for Gemini and LLaMa3. The paper further argues that an automated Gemini Pro judge can extract final answers and score each reasoning step, replacing binary right/wrong evaluation with step-level labels.

What carries the argument

The central machinery is the dataset structure rather than a mathematical identity: GPSM4K provides each problem with a step-by-step solution broken into six labeled sections (data identification, problem analysis, theoretical framework, methodology development, computation, solution). Fine-tuning on this format is what the paper credits for the accuracy gains over multiple-choice-only datasets like PGPS9K. A second mechanism is the Gemini Pro pipeline that generates diagram descriptions, augments questions, regenerates solutions, and later judges model outputs. A third mechanism is a vector database with cosine similarity that retrieves one analogous solved problem as context during inference, which the paper calls retrieval-augmented generation.

What would settle it

Have human experts label the same test-set predictions for final-answer correctness and per-step correctness, then compare their labels with Gemini Pro's labels; strong disagreement or a systematic pattern, for example favoring responses that resemble Gemini-generated text, would invalidate the reported accuracy numbers.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that the structure of the training data, not just the model, determines geometry performance. GPSM4K contains numerical-answer and theorem-proving problems from grades 7 to 12, with each problem carrying both an original textbook solution and a regenerated, decomposed step-by-step solution. Fine-tuning a 7-billion-parameter LLaVA 1.6 model on GPSM4K raises its accuracy on the paper's held-out test set from 8% (fine-tuned on PGPS9K) to 22.66%, and the 13-billion-parameter variant reaches 24%. The GLLaVA 13B model improves from 22.66% on GeoQA++ to 25.33% on GPSM4K. Feeding the model a caption of the diagram during inference raises scores further, with human-written captions helping more than machine-generated ones, and one-shot retrieval-augmented generation adds about two to three points on both Gemini and LLaMa3. The paper also claims that Gemini Pro can serve as an automated judge that extracts final answers and assigns a binary score to each step of a predicted solution, enabling multistep rather than answer-only evaluation.

Load-bearing premise

Every reported accuracy number rests on Gemini Pro's automatic judging being accurate, since it both extracts and compares final answers and assigns step-level labels, and the paper reports no human validation of that judge.

Editorial extensions

If this is right

Fine-tuning on a dataset with consistent step-by-step solutions can move an open vision-language model from near-chance to roughly 23% on high-school geometry, so data design is a first-order lever alongside model size.
Step-level labels make it possible to ask where models fail, whether in reading the diagram, choosing the right theorem, or computing, rather than only whether the final answer matches.
Image captions become an actionable intervention: supplying a good textual description of the diagram improves accuracy, and human captions beat automatic ones, which points to the visual encoder as a bottleneck.
Retrieval of one analogous solved problem during inference gives a small but consistent boost, offering a cheap way to improve geometry answers without additional fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension beyond the paper: if the automatic judge is biased toward responses written in the style of the model that generated the solutions, then part of the reported gain from fine-tuning and RAG could be stylistic mimicry rather than improved geometry; comparing model outputs with human-graded labels would settle this.
The paper does not isolate which part of GPSM4K drives the gain, whether step-by-step solutions, diagram descriptions, or question diversity. An equal-sized control dataset with the same questions but multiple-choice-only answers would identify the mechanism.
Because the augmented questions and solutions were generated by the same model family used as judge, there is a risk of circularity in the evaluation; a human-scored subset of the test set would give an independent check.
If the approach generalizes, the same template of textbook extraction, decomposition of solutions, and step-level judging could be applied to other stepwise reasoning domains such as algebra word problems, physics exercises, or formal proofs.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

GPSM4K is a real contribution: a manually collected geometry QA dataset with numerical and theorem-proving questions, diagram descriptions, and step-by-step solutions in a uniform format. It fills a gap next to PGPS9K and Geometry3K, which are mostly multiple-choice. The fine-tuning experiments show consistently higher accuracy for LLaVA and GLLaVA variants when trained on GPSM4K than on PGPS9K or GeoQA++, and the captioning and 1-shot RAG results are sensible though modest.

The soft spots are real. The evaluation uses Gemini Pro as an automatic judge for both final answers and step-by-step correctness (Section 5), and the same model generated much of the training data (descriptions, two augmented questions per original, regenerated solutions). There is no human validation of the judge, no inter-annotator agreement, no error analysis. The step evaluation criteria are subjective, the 'pre-defined key-step template' is never described, and Section 5.2 contradicts itself about whether ground-truth answers are fed to the judge. If the judge is biased toward Gemini-style solutions or mishandles equivalent answer formats, every accuracy number in Tables 3–7 could shift. That is load-bearing because the paper's main claims are comparisons measured through this judge.

There are also internal inconsistencies: the abstract says 2,157 original and 5,340 total examples but Table 1 lists 1,480 original and 4,440 total; Section 4 says a test set of 150, Table 2 lists 200; several table references are placeholders. Gemini Pro's zero-shot score on GPSM4K is 44% in Table 3 and 35.33% in Table 4, with no explanation. The test set's disjointness from the augmented training data is not addressed. And the paper never fine-tunes on Geo170K, which is the closest prior dataset, so the claimed advantage over existing datasets is not fully benchmarked.

Still, the core idea holds up: a geometry benchmark with diverse question types and step-by-step solutions is worth having, and the reported fine-tuning gains are plausible. The paper deserves a serious review, but the referee should require human validation of the judge, a fixed and consistent dataset description, and a comparison against Geo170K before the numbers are trusted. I'd bring it to a reading group if the authors release the data; otherwise the value is limited.

Referee Report

5 major / 6 minor

Summary. GPSM4K is presented as a new multimodal geometry benchmark: original question-answer pairs are manually extracted from Indian board textbooks (grades 7–12 in the abstract, grades 6–12 in Section 3.1), then augmented with Gemini Pro–generated diagram descriptions, two additional questions per original, and restructured step-by-step solutions. The authors fine-tune LLaVA and GLLaVA variants on GPSM4K, report accuracy gains relative to fine-tuning on PGPS9K (e.g., LLaVA 1.6 Vicuna 7B rises from 8% to 22.66% in Table 4), and evaluate the effects of image captions and 1-shot multimodal RAG. The evaluation of both final answers and chain-of-thought steps is performed by Gemini Pro, which extracts answer strings and compares them, and which also assigns binary step scores. The paper claims that GPSM4K fills a gap by including numerical-answer and theorem-proving questions with step-by-step solutions.

Significance. If the reported accuracy numbers are trustworthy, GPSM4K would be a useful resource for the geometry reasoning community: it addresses the relative scarcity of non-multiple-choice geometry benchmarks, provides step-by-step solutions in a consistent format, and includes experiments on captioning and RAG that are rarely studied on geometry data. The dataset construction is concrete, and the fine-tuning comparisons across several LLaVA/GLLaVA configurations are informative in principle. However, the empirical claims are currently not established because they rest on an unvalidated LLM judge, the dataset statistics are internally inconsistent, and the paper does not demonstrate that the test set is disjoint from the training set. The benchmark and the methodology are salvageable, but the manuscript needs substantial revision before the central claims can be accepted.

major comments (5)

[§5.1, §5.2, Figures 7 and 8] The automatic judge is unvalidated and is the sole source of every accuracy number in Tables 3–7. Gemini Pro both generates much of the training data (diagram descriptions, two augmented questions per original, and all regenerated solutions in §3.2) and evaluates the final answers and steps. The paper reports no human validation, no inter-annotator agreement, and no error analysis for the judge. Since the judge compares extracted answer strings with a simple 'Yes'/'No' label, errors in numeric normalization (e.g., '5 cm' vs. '5', or '√25' vs. '5') or a stylistic bias toward Gemini-generated solutions could systematically inflate or flatten the reported rankings. I request a human-validated subset (at least 100–150 items), a normalization-based exact-match baseline, and an error analysis of extraction and step-labeling failures.
[Abstract, Table 1, §9] The dataset size is reported inconsistently. The abstract states 2157 multimodal question-answer pairs augmented to 5340 problems, Table 1 reports 1480 original and 2960 augmented samples totaling 4440, and Section 9 says 1438 original problems augmented to 4440 entries. These are not minor rounding differences; they change the claimed scale of the dataset by hundreds of items. The authors must state one correct set of numbers and ensure the abstract, table, and limitations section agree before the benchmark can be evaluated or reproduced.
[Table 2, §4] The test-set size is inconsistent. Table 2 reports 200 total inference samples, while Section 4 states that 'All results are reported based on experiments conducted on a test set of 150 geometry problems' and Experiment 3 says 'we utilize images from our test set of 150 examples.' If the test set is a 150-item subset of Table 2, that split must be described; if the numbers changed during writing, the reported accuracies in Tables 3–7 cannot be reproduced without knowing which test set was used.
[§3.2, §4.2] The paper does not establish that the test set is disjoint from the training set. Because question augmentation in §3.2.2 generates two new questions from each original diagram and QA pair, and because the test set appears to draw from the same textbook sources, fine-tuning on augmented questions could include near-duplicates of test items. The authors should specify the exact train/test split, provide source identifiers or diagram-level overlap statistics, and describe any deduplication procedure. Without this, the improvement from 8–8.66% to 22.66–24.66% in Table 4 could partly reflect memorization rather than improved geometric reasoning.
[§4.2, Table 4] The comparison between models fine-tuned on PGPS9K and on GPSM4K is confounded by more than dataset quality: the two training sets differ in size, problem format (MCQ vs. numerical/theorem-proving), solution structure, and grade distribution. The claim that GPSM4K's detailed step-by-step solutions drive the improvement is therefore not directly supported. A controlled comparison (e.g., fine-tuning on a matched-size PGPS9K subset, or ablating step-by-step solutions from GPSM4K) would give the central claim much stronger support.

minor comments (6)

[§4.2 and §4.4] There are unresolved table references: Section 4.2 refers to 'Table ??' instead of Table 4, and Section 4.4 refers to 'the results of Experiment 2 table 7' when the captioning results are in Table 5.
[Abstract and §3.1] The grade range is inconsistent: the abstract and title say grades 7–12, while Section 3.1 says Grades 6 through 12. Please align these statements.
[Figure 3] The caption contains a duplicated word: 'chapter chapter-wise distribution'. The figure should also be referenced in the text before it appears.
[Table 5] The table header has 'Mean / Median Jaccard' and 'Mean / Median Cosine similarity' as two columns, but each cell contains two values separated by a slash; the column semantics should be made explicit (e.g., 'Jaccard (mean/median)').
[Keywords] The keyword list contains typographical and formatting errors: 'LLMS' should be 'LLMs', 'Retrieval Augmentated' should be 'Retrieval-Augmented', and the list lacks consistent punctuation.
[§5.1] The subsection heading contains a typo: 'Evalaution' should be 'Evaluation'. In addition, Figure 9 is placed in the appendix and is not cited in the text; it should be integrated into the step-by-step evaluation discussion.

Circularity Check

1 steps flagged · score 6.0 of 10

GPSM4K's headline accuracy numbers are scored by Gemini Pro, the same model that generated the ground-truth step-by-step solutions, augmented questions, and captions; the reported 'improvements' may measure convergence to Gemini's output style rather than independently verified geometry skill.

self definitional [Introduction / Section 3.2.3 and Section 5.1 (also Section 5.2)]
"GPSM4K provides two solution versions for each problem: the textbook’s original solution and a reconstructed step-by-step version via Gemini Pro Vision. [...] Following the question generation, solutions were regenerated using Gemini Pro, a text-based LLM. [...] In our evaluation, we utilized the Gemini Pro API to assess the accuracy of our models. The evaluation of the final solution follows a two-step approach using the Gemini Pro API. In the first stage, the initial prompt examines both the ground truth solution and the predicted outcome, extracting the final answer from each of them."

The reference 'ground truth solution' used by the evaluator is itself a Gemini Pro generated reconstruction from Section 3.2.3, and the augmented questions, diagram descriptions, and captions used for fine-tuning are also Gemini Pro outputs (Sections 3.2.1-3.2.2). Section 5.1 then uses the same Gemini Pro API to extract and compare final answers, labeling a match 'Yes', while Section 5.2 has Gemini assign binary step scores against the same style of solution. Consequently, the accuracy numbers in Tables 3-7 measure agreement with Gemini-style solution text and step structure, not independently verified mathematical correctness. A model fine-tuned on Gemini-generated data can raise its score by stylistic mimicry, and no human validation or inter-annotator agreement on the judge is reported.

full rationale

The dataset construction itself is not circular: the original question-answer pairs were manually extracted from textbooks, and the fine-tuned open-source models are external to the judge. However, the evaluation protocol is load-bearing and self-referential. Gemini Pro both authored the reference step-by-step solutions used as ground truth and acts as the sole judge of whether candidate outputs match those references. Because every headline result (zero-shot Table 3, fine-tuning Table 4, captioning Table 6, RAG Table 7) passes through this same scorer, the central empirical claim is reduced in practice from 'better geometry reasoning' to 'better agreement with Gemini-generated solution conventions.' The paper reports no human validation of the Gemini judge, no inter-annotator agreement, and no error analysis of numeric extraction or step labeling, so the metric is not externally anchored. This is partial circularity: the data and models have independent content, but the measurement of the central claim is definitionally tied to the model that generated the training references.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central empirical claims rest on the reliability of the LLM judge, on the separation of train and test data, and on the fidelity of the extraction pipeline. None of these are independently verified in the paper.

assumptions (3)

domain assumption Gemini Pro automatic evaluation produces correct 'Yes/No' labels for final answers and CoT steps.
Section 5.1 and 5.2 use Gemini Pro as judge with no human validation, no inter-annotator agreement, and no error analysis.
domain assumption The 150-problem test set is disjoint from the training set.
The paper never states how the test problems were selected or whether they overlap with the original or augmented training problems.
domain assumption Mathpix conversion and human expert extraction preserve the correctness of textbook questions and solutions.
Section 3.1 describes the extraction pipeline but provides no quality checks or error rates.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Improving Multimodal LLMs Ability In Geometry Problem Solving, Reasoning, And Multistep Scoring." pith.science (2026). https://pith.science/paper/JI4L5RVK

@misc{pith2026241200846,
  author       = {Pith},
  title        = {Pith review of: Improving Multimodal LLMs Ability In Geometry Problem Solving, Reasoning, And Multistep Scoring},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JI4L5RVK}},
  note         = {Machine review of arXiv:2412.00846}
}

read the original abstract

This paper presents GPSM4K, a comprehensive geometry multimodal dataset tailored to augment the problem-solving capabilities of Large Vision Language Models (LVLMs). GPSM4K encompasses 2157 multimodal question-answer pairs manually extracted from mathematics textbooks spanning grades 7-12 and is further augmented to 5340 problems, consisting of both numerical and theorem-proving questions. In contrast to PGPS9k, Geometry3K, and Geo170K which feature only objective-type questions, GPSM4K offers detailed step-by-step solutions in a consistent format, facilitating a comprehensive evaluation of problem-solving approaches. This dataset serves as an excellent benchmark for assessing the geometric reasoning capabilities of LVLMs. Evaluation of our test set shows that there is scope for improvement needed in open-source language models in geometry problem-solving. Finetuning on our training set increases the geometry problem-solving capabilities of models. Further, We also evaluate the effectiveness of techniques such as image captioning and Retrieval Augmentation generation (RAG) on model performance. We leveraged LLM to automate the task of final answer evaluation by providing ground truth and predicted solutions. This research will help to assess and improve the geometric reasoning capabilities of LVLMs.

Figures

Figures reproduced from arXiv: 2412.00846 by the authors.

**Figure 2.** Dataset augmentation Pipeline using the Gemini-Pro API. The process starts with the original question and solution, [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Data Extraction and topic-wise distribution [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 5.** Figure 5: Image captioning example 4.4 Experiment 3 "Can adding image captions to multimodal geometry figures improve LLMs’ ability to solve complex geometry problems and theorems?" In this experiment, we explored how adding image captions affects the performance of LVLMs in so…

**Figure 6.** Figure 6: Multimodal RAG tackles this challenge by combining the strengths of images and descriptions. It analyzes diagrams [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

**Figure 7.** Figure 7: Two Prompts used in Final Solution Extraction and [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

**Figure 8.** Figure 8: Prompt used for step-by-step evaluation should involve more controlled and comparable experimental conditions and utilize multiple datasets for fine-tuning to ensure more rigorous and persuasive conclusions. REFERENCES [1] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Si…

**Figure 9.** Figure 9: The binary ‘Correct’ or ‘Incorrect’ evaluative approach of existing benchmarks is inadequate to examine the depth [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 12 canonical work pages

[1]

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection.arXiv preprint arXiv:2310.11511 (2023)

arXiv 2023
[2]

Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023 (2023)

arXiv 2023
[3]

Jie Cao and Jing Xiao. 2022. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In Proceedings of the 29th International Conference on Computational Linguistics . 1511–1520

work page 2022
[4]

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al . 2023. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology (2023)

work page 2023
[5]

Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2024. Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 17754–17762

2024
[6]

Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P Xing, and Liang Lin. 2021. GeoQA: A geometric question answering benchmark towards multimodal numerical reasoning. arXiv preprint arXiv:2105.14517 (2021)

arXiv 2021
[7]

Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William W Cohen. 2022. Murag: Multimodal retrieval-augmented generator for open question answering over images and text. arXiv preprint arXiv:2210.02928 (2022)

arXiv 2022
[8]

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 (2021)

arXiv 2021

Show all 46 references

[9]

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xi- aohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv prepri...

2020 arXiv
[10]

Jiahui Gao, Renjie Pi, Yong Lin, Hang Xu, Jiacheng Ye, Zhiyong Wu, Weizhong Zhang, Xiaodan Liang, Zhenguo Li, and Lingpeng Kong. 2022. Self-guided noise-free data generation for efficient zero-shot learning. arXiv preprint arXiv:2205.12679 (2022)

2022 arXiv
[11]

Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, et al. 2023. G-llava: Solving geometric problem with multi-modal large language model. arXiv preprint arXiv:2312.11370 (2023)

2023 arXiv
[12]

Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, Nan Duan, Weizhu Chen, et al. 2023. Tora: A tool-integrated reasoning agent for mathemati- cal problem solving. arXiv preprint arXiv:2309.17452 (2023)

2023 arXiv
[13]

Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research 24, 251 ...

2023
[14]

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, De- vendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al . 2023. Mistral 7B. arXiv preprint arXiv:2310.06825 (2023)

2023 arXiv
[15]

Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. arXiv preprint arXiv:2305.06983 (2023)

2023 arXiv
[16]

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning . PMLR, 19730–19742

2023
[17]

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning . PMLR, 12888–12900

2022
[18]

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81

2004
[19]

Bingshuai Liu, Chenyang Lyu, Zijun Min, Zhanyu Wang, Jinsong Su, and Longyue Wang. 2023. Retrieval-augmented multi-modal chain-of-thoughts reasoning for large language models. arXiv preprint arXiv:2312.01714 (2023)

2023 arXiv
[20]

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruc- tion tuning. Advances in neural information processing systems 36 (2024)

2024
[21]

Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. 2021. Inter-GPS: Interpretable geometry problem solving with formal language and symbolic reasoning. arXiv preprint arXiv:2105.04165 (2021)

2021 arXiv
[22]

Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. 2022. Generating training data with language models: Towards zero-shot language understanding. Advances in Neural Information Processing Systems 35 (2022), 462–477

2022
[23]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics . 311–318

2002
[24]

Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277 (2023)

2023 arXiv
[25]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. IMPROVING MULTIMODAL LLM’S ABILITY IN GEOMETR...

2021
[26]

Mrinmaya Sachan, Kumar Dubey, and Eric Xing. 2017. From textbooks to knowl- edge: A case study in harvesting axiomatic knowledge from textbooks to solve geometry problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing . 773–784

2017
[27]

Mrinmaya Sachan and Eric Xing. 2017. Learning to solve geometry problems from natural language demonstrations in textbooks. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (* SEM 2017) . 251–261

2017
[28]

Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm
[29]

Min Joon Seo, Hannaneh Hajishirzi, Ali Farhadi, and Oren Etzioni. 2014. Diagram understanding in geometry questions. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 28

2014
[30]

Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Xiangji Huang. 2023. A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets.arXiv e-prints (2023), arXiv–2305

2023
[31]

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Stanford alpaca: An instruction-following llama model

2023
[32]

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)

2023 arXiv
[33]

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yas- mine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhos- ale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)

2023 arXiv
[34]

Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. 2022. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.14100 (2022)

2022 arXiv
[35]

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language mod- els with self-generated instructions. arXiv preprint arXiv:2212.10560 (2022)

2022 arXiv
[36]

Jingxuan Wei, Cheng Tan, Zhangyang Gao, Linzhuang Sun, Siyuan Li, Bihui Yu, Ruifeng Guo, and Stan Z Li. 2023. Enhancing Human-like Multi-Modal Reasoning: A New Challenging Dataset and Comprehensive Framework. arXiv preprint arXiv:2307.12626 (2023)

2023 arXiv
[37]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

2022
[38]

Fangyuan Xu, Weijia Shi, and Eunsol Choi. 2023. Recomp: Improving retrieval- augmented lms with compression and selective augmentation. arXiv preprint arXiv:2310.04408 (2023)

2023 arXiv
[39]

Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. 2024. Corrective Retrieval Augmented Generation. arXiv preprint arXiv:2401.15884 (2024)

2024 arXiv
[40]

Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2022. Retrieval- augmented multimodal language modeling. arXiv preprint arXiv:2211.12561 (2022)

2022 arXiv
[41]

Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. 2022. Zerogen: Efficient zero-shot learning via dataset generation. arXiv preprint arXiv:2202.07922 (2022)

2022 arXiv
[42]

Wenhao Yu, Hongming Zhang, Xiaoman Pan, Kaixin Ma, Hongwei Wang, and Dong Yu. 2023. Chain-of-note: Enhancing robustness in retrieval-augmented language models. arXiv preprint arXiv:2311.09210 (2023)

2023 arXiv
[43]

Ming-Liang Zhang, Fei Yin, and Cheng-Lin Liu. 2023. A Multi-Modal Neural Geometric Solver with Textual Clauses Parsed from Diagram. arXiv preprint arXiv:2302.11097 (2023)

2023 arXiv
[44]

Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. 2023. Explainability for large lan- guage models: A survey. ACM Transactions on Intelligent Systems and Technology (2023)

2023
[45]

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems 36 (2024). Conference’17, Ju...

2024
[2015]

In Proceedings of the 2015 conference on empirical methods in natural language processing

Solving geometry problems: Combining text and diagram interpretation. In Proceedings of the 2015 conference on empirical methods in natural language processing. 1466–1476

2015

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection.arXiv preprint arXiv:2310.11511 (2023)

arXiv 2023

[2] [2]

Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023 (2023)

arXiv 2023

[3] [3]

Jie Cao and Jing Xiao. 2022. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In Proceedings of the 29th International Conference on Computational Linguistics . 1511–1520

work page 2022

[4] [4]

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al . 2023. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology (2023)

work page 2023

[5] [5]

Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2024. Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 17754–17762

2024

[6] [6]

Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P Xing, and Liang Lin. 2021. GeoQA: A geometric question answering benchmark towards multimodal numerical reasoning. arXiv preprint arXiv:2105.14517 (2021)

arXiv 2021

[7] [7]

Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William W Cohen. 2022. Murag: Multimodal retrieval-augmented generator for open question answering over images and text. arXiv preprint arXiv:2210.02928 (2022)

arXiv 2022

[8] [8]

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 (2021)

arXiv 2021