Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Towards Spoken Mathematical Reasoning: Benchmarking Speech-based Models over Multi-faceted Math Problems

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Spoken-MQA, a new benchmark of 2,769 spoken math problems, shows end-to-end speech LLMs lag cascade systems on arithmetic and knowledge reasoning, and that LLMs struggle with verbalized math compared to LaTeX.

desk verdict A genuinely useful new benchmark for spoken math reasoning, with a real confound between human and TTS audio that needs fixing before the headline numbers can be fully trusted. read the letter →

arxiv 2505.15000 v1 pith:RVMPW6ZQ submitted 2025-05-21 cs.CL

Chengwei Wei , Bin Wang , Jung-jae Kim , Nancy F. Chen This is my paper · ORCID
classification cs.CL
keywords spokenmathematicalreasoningspeechLLMscascadeASR-LLMmathbenchmarkarithmeticverbalizationknowledge-intensivechain-of-thought
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces Spoken-MQA, a benchmark for measuring how well speech-based AI models handle mathematical reasoning from audio alone. It argues that current end-to-end speech LLMs, which process raw audio, substantially lag behind cascade systems that first transcribe speech with ASR and then reason with a text LLM, especially on direct arithmetic and knowledge-rich problems. The benchmark also shows that LLMs are biased toward symbolic LaTeX math and falter when the same math is verbalized in natural spoken language. If these findings hold, they point to a concrete training gap: speech models need math-specific data and better handling of spoken math expressions, not just better audio understanding.

What carries the argument

The central object is the Spoken-MQA benchmark itself: a curated set of 2,769 spoken math problems organized into three tiers of difficulty (Arithmetic, Contextual Reasoning, Knowledge-Oriented Reasoning). Its load-bearing component is the verbalization-and-filtering pipeline: GPT-4o converts LaTeX-laden written problems into spoken language, human annotators and the model flag ambiguous renderings (32% of a sampled 100 MATH problems were ambiguous), and the surviving items are synthesized into Coqui TTS audio. This pipeline is what lets the paper isolate a symbolic-bias effect: identical problems in LaTeX, verbalized text, and speech produce dramatically different accuracies.

What would settle it

Re-run Spoken-MQA with the same questions read by human speakers instead of TTS, and compare accuracy on the arithmetic and knowledge subsets; a large improvement in end-to-end speech LLMs would mean the reported gap is partly an artifact of synthetic audio. A complementary test: give speech LLMs the verbalized text of the same problems as text input; if they solve them in text but not in audio, the deficit is in speech encoding rather than math reasoning.

Watch

Extended reading notes

Core claim

Spoken-MQA assembles 2,769 spoken math questions spanning pure arithmetic, single- and multi-step contextual word problems, and knowledge-oriented competition problems, all filtered for verbal ambiguity. On this benchmark, the paper finds that open-source end-to-end speech LLMs such as Qwen2-Audio, Audio-Reasoner, MERaLiON, Ultravox, and Phi-4-Multimodal underperform the best cascade systems, with the largest deficits in arithmetic (as low as 36–56% accuracy versus 61–77% for cascades) and in knowledge-oriented reasoning. It further reports that when problems are presented as verbalized text rather than LaTeX, even strong text LLMs lose 5–17 points on the knowledge category, exposing a symbolic bias. Domain-specific fine-tuning on 500k samples of spoken math instruction data lifts Phi-4-Multimodal from 38.8% to 56.4% on arithmetic and from 27.2% to 52.2% on knowledge-oriented reasoning, suggesting the gap is partly a matter of training data rather than a fundamental architectural limit.

Load-bearing premise

The evaluation assumes the synthetic speech from Coqui TTS is a faithful enough stand-in for human speech that accuracy scores reflect mathematical reasoning rather than TTS artifacts or residual ambiguities introduced during audio synthesis; the authors acknowledge the synthetic-speech bias in the Limitations section.

Editorial extensions

If this is right

  • Cascade models (ASR plus a text LLM) remain the practical high-accuracy approach for spoken math today, but any transcription error on formulas or numbers propagates into the final answer.
  • Current speech LLMs trained mostly on conversational or audio-understanding data need math-specific spoken data; fine-tuning on 500k spoken math samples materially improved arithmetic and knowledge scores.
  • LLMs' symbolic bias means spoken math input requires either better verbalization handling or training with spoken or verbalized math to close the text-to-speech gap.
  • Spoken-MQA can serve as a reusable testbed for future speech LLMs and for studying whether surface-level contextual performance reflects genuine math understanding or shallow pattern matching.
  • The digit-length analysis suggests speech LLMs' competitive contextual scores may be driven by pattern matching rather than actual computation, since they fail on short-digit direct arithmetic.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the symbolic-bias finding generalizes, improving text LLMs' ability to reason from verbalized (non-LaTeX) math would lift both cascade and end-to-end systems, since ASR transcripts and audio both present math in speech-like form.
  • The requirement to filter out ambiguous verbalizations suggests that real-world spoken math assistants will face unavoidable ambiguity that benchmarks like Spoken-MQA deliberately remove; interactive clarification may be necessary in deployment.
  • Replacing Coqui TTS with higher-quality or human audio would likely shift absolute accuracies, but the ordering between cascades and speech LLMs is probably stable unless speech encoders are substantially improved.
  • The same verbalization pipeline could be exported to other symbol-heavy domains such as physics and chemistry to build spoken reasoning benchmarks and expose analogous symbolic biases.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper introduces Spoken-MQA, a benchmark for spoken mathematical reasoning that covers three categories: Arithmetic (human recordings from Timers and Such), Contextual Reasoning (single- and multi-step word problems from AddSub, SingleOp, SVAMP, and GSM8K, synthesized with Coqui TTS), and Knowledge-Oriented Reasoning (a curated set of 500 problems from MATH, verbalized by GPT-4o, filtered by human annotators, and synthesized with Coqui TTS). The authors evaluate cascade models (Whisper ASR followed by seven text LLMs) and end-to-end speech LLMs (Qwen2-Audio, Audio-Reasoner, MERaLiON, Ultravox, Phi-4-Multimodal, GPT-4o-audio, and a fine-tuned Phi-4-Multimodal variant). Their main findings are that open speech LLMs generally lag behind cascade models, particularly on direct arithmetic and knowledge-oriented problems; that LLMs perform better on symbolic LaTeX input than on verbalized or ASR-transcribed input; and that domain-specific fine-tuning with spoken math data improves performance. The paper also analyzes the arithmetic gap by digit length and compares three input formats for cascade LLMs.

Significance. If the benchmark is reliable, Spoken-MQA fills a genuine gap: existing speech benchmarks mostly test factual comprehension or simple reasoning, while spoken mathematical reasoning is relevant to tutoring, classroom, and voice-assistant applications. The paper's strengths include the multi-category taxonomy, the combination of human-recorded and TTS speech, the breadth of evaluated models, the input-format comparisons, and the commitment to open-source the data and code. The central gap it reports, e.g., Whisper-Qwen2.5-Math at 81.0% average versus open speech LLMs at 32.0-69.0%, is large and internally consistent. However, the benchmark's core guarantee of unambiguous and faithful spoken input is currently validated only on verbalized text, not on the audio that models actually hear, and several comparisons are confounded by audio source. These issues are addressable with additional validation and controlled experiments, so the contribution is potentially valuable but not yet fully substantiated.

major comments (4)
  1. [§3.2, Table 1, Table 2] The claim that Spoken-MQA consists of 'unambiguous natural spoken language' is validated only on GPT-4o's verbalized text, not on the audio that models actually hear. The ambiguity check and human verification in Figure 2 are applied before TTS synthesis; 2,496 of the 2,769 items (Contextual and Knowledge categories) are generated by Coqui TTS, and the paper itself notes (Section 3.2) that standard TTS systems may mispronounce expressions such as '9!' as 'nine exclamation mark.' No human-listening or ASR-round-trip check is reported, so the large Knowledge-Oriented gap (e.g., Qwen2-Audio at 5.8% versus Whisper-Qwen2.5-Math at 72.4% in Table 2) could partly reflect TTS artifacts or residual prosodic ambiguity rather than mathematical reasoning. I request a listening validation on a sample of the synthesized audio, a report of mispronunciation and residual-ambiguity rates, and ideally an ASR-reconstruction check that transcribes the audio and compares it with the intended verbalized text.
  2. [§3.2, §4.2.1, Table 2] The arithmetic-versus-contextual comparison is confounded by audio source: Arithmetic items are human recordings from Timers and Such, while Contextual and Knowledge items are Coqui TTS. The central claim that speech LLMs 'struggle with direct arithmetic' (Section 5) may therefore reflect acoustic mismatch or recording conditions rather than arithmetic-specific inability. The short-digit versus long-digit analysis in Figure 3 does not remove this confound because both subsets are from the same human-recorded source. Please add a controlled comparison (e.g., TTS versions of the Arithmetic items, or human recordings of a Contextual subset), or at minimum report ASR word/character error rates per category and demonstrate that the observed gaps persist after controlling for source.
  3. [§4.1, Table 2] No error bars, significance tests, or decoding details are reported. All accuracies appear to be single-run numbers; differences such as Whisper-Llama-3.1-8B at 67.0 versus Whisper-Gemma-2-9B at 70.8 on Arithmetic are within plausible sampling noise for 273 items. The paper should report bootstrap confidence intervals (or multiple decoding runs with variance), the decoding protocol (greedy versus sampling, temperature, max tokens), and the exact answer-extraction and scoring rule (exact string match, numeric tolerance, handling of units or equivalent expressions). This is needed to support fine-grained claims such as 'Ultravox performs comparably to its cascade counterpart' in Section 4.2.3.
  4. [§4.1, Appendix C, §4.2.3] The fine-tuning experiment may be contaminated by train/test overlap. The Knowledge-Oriented subset is sampled from MATH, and FT-Phi-4-multimodal-instruct is trained on 500k samples from OpenMathInstruct-1, which contains MATH-derived instruction data. The paper does not state whether the 500 MATH problems are from the MATH test split or otherwise excluded from the OpenMathInstruct-1 samples. Without this check, the large improvement from 27.2% to 52.2% on Knowledge-Oriented Reasoning cannot be attributed to domain-specific spoken fine-tuning. Please report the exact MATH split used, confirm disjointness from the fine-tuning data, and consider an overlap-filtered evaluation.
minor comments (4)
  1. [§4.2, Table 2] The opening paragraph says 'Whisper-Qwen2.5-7B-Instruct achieves the best overall performance,' but Table 2 shows Whisper-Qwen2.5-Math-7B-Instruct at 81.0% average, not Whisper-Qwen2.5-7B-Instruct at 69.4%. Please correct the model name.
  2. [§3.2, Figure 2] The ambiguity analysis on 100 random MATH problems reports that 32 are ambiguous, but no inter-annotator agreement is provided. Reporting kappa or a second annotation round would strengthen the reliability of the filtering step.
  3. [§4.2.2, Table 3] The comparison of ASR-transcribed, verbalized, and ground-truth text inputs would benefit from reporting the actual Whisper word/character error rate per category, so that the claim that 'transcription errors can still propagate' is quantified rather than inferred from downstream accuracy.
  4. [§3.2, §4.1] GPT-4o is used both to generate and filter the verbalized problems and, in the form of GPT-4o-audio, is one of the evaluated models. Human verification mitigates the circularity, but the paper should report how many items were rejected by GPT-4o versus by human annotators and briefly discuss whether the final set is sensitive to the choice of verbalizer.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the benchmark findings are empirical evaluation results, not consequences of the construction pipeline.

full rationale

Spoken-MQA is a benchmark, not a derivation chain. The central claims (speech LLMs lag cascade models on arithmetic and knowledge-oriented tasks, and LLMs are biased toward symbolic LaTeX over verbalized math) are obtained by running held-out models on curated items and comparing accuracies in Tables 2 and 3. The benchmark construction does not fix these outcomes: items are sourced from external datasets (Timers and Such, AddSub, SingleOp, SVAMP, GSM8K, MATH), verbalized and filtered with GPT-4o plus human annotators, and synthesized with Coqui TTS. Although GPT-4o is used in construction and GPT-4o-audio is one of the evaluated models, the human ambiguity filter and the variety of evaluated models mean the measured gaps are not forced by the construction. The paper's own Limitations section concedes that TTS audio may introduce bias and that human-recorded speech is needed; this is a validity or confound concern (for example, Arithmetic uses human recordings while other categories use TTS, and fine-tuning and test audio share Coqui TTS), not a circularity in the sense of a result being equivalent to its inputs. Self-citations such as CoinMath and MERaLiON are listed as related work or evaluated models and are not load-bearing justification for the paper's conclusions. No equation or fitted parameter is renamed as a prediction, and no uniqueness claim is imported from the authors' prior work. Accordingly, the appropriate finding is no significant circularity; the mild overlaps are worth noting but do not raise the circularity score beyond 2.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark construction involves hand-chosen sizes (e.g., 500 curated MATH problems), filtering thresholds, and LoRA hyperparameters, but these are not fitted to the central results. The axioms above are the load-bearing domain assumptions about speech synthesis and evaluation fidelity.

assumptions (4)
  • domain assumption Coqui TTS-generated audio adequately represents natural spoken math for evaluation.
    Section 3.2 uses Coqui TTS for speech synthesis; the Limitations section acknowledges synthetic speech may introduce biases.
  • domain assumption The human/GPT-4o ambiguity filtering on verbalized text transfers to the synthesized audio (i.e., the audio is unambiguous).
    Ambiguity checks (Section 3.2, Appendix B) were performed on written verbalization, not on the final TTS audio.
  • domain assumption Whisper-Large-V3 is a representative state-of-the-art ASR for cascade evaluation.
    Section 4.1 selects Whisper-Large-V3 as the ASR front-end for all cascade models.
  • domain assumption Zero-shot CoT prompting with 'Please reason step by step' is a fair protocol for all models, and final-answer extraction from CoT responses is accurate.
    Section 4.1 describes the prompting and accuracy metric, but the answer-extraction procedure is not specified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Spoken Mathematical Reasoning: Benchmarking Speech-based Models over Multi-faceted Math Problems." pith.science (2026). https://pith.science/paper/RVMPW6ZQ

@misc{pith2026250515000,
  author       = {Pith},
  title        = {Pith review of: Towards Spoken Mathematical Reasoning: Benchmarking Speech-based Models over Multi-faceted Math Problems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RVMPW6ZQ}},
  note         = {Machine review of arXiv:2505.15000}
}
read the original abstract

Recent advances in large language models (LLMs) and multimodal LLMs (MLLMs) have led to strong reasoning ability across a wide range of tasks. However, their ability to perform mathematical reasoning from spoken input remains underexplored. Prior studies on speech modality have mostly focused on factual speech understanding or simple audio reasoning tasks, providing limited insight into logical step-by-step reasoning, such as that required for mathematical problem solving. To address this gap, we introduce Spoken Math Question Answering (Spoken-MQA), a new benchmark designed to evaluate the mathematical reasoning capabilities of speech-based models, including both cascade models (ASR + LLMs) and end-to-end speech LLMs. Spoken-MQA covers a diverse set of math problems, including pure arithmetic, single-step and multi-step contextual reasoning, and knowledge-oriented reasoning problems, all presented in unambiguous natural spoken language. Through extensive experiments, we find that: (1) while some speech LLMs perform competitively on contextual reasoning tasks involving basic arithmetic, they still struggle with direct arithmetic problems; (2) current LLMs exhibit a strong bias toward symbolic mathematical expressions written in LaTex and have difficulty interpreting verbalized mathematical expressions; and (3) mathematical knowledge reasoning abilities are significantly degraded in current speech LLMs.

Figures

Figures reproduced from arXiv: 2505.15000 by the authors.

Figure 1
Figure 1. Overview of Spoken-MQA. 2021), which targets grade-school level word prob￾lems, and MATH (Hendrycks et al., 2021), which comprises competition problems at the high school level. Beyond mathematics, broader reasoning eval￾uations have emerged. For example, BIG-bench (Srivastava et al., 2022), MR-Bench (Zeng et al., 2024), and ARB (Sawada et al., 2023) assess rea￾soning across diverse domains, including science, progr… view at source ↗
Figure 2
Figure 2. Pipeline for Generating and Filtering Unam [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Model Accuracy on Short vs. Long Digit Length in Arithmetic 4.2.2 Impact of Input Format on LLMs Cascade models, which transcribe speech into text via ASR systems before passing it to LLMs, demon￾strate strong performance on the Spoken-MQA benchmark. This highlights that cascade-based approaches currently offer an effective solution for speech-based instruction tasks in mathemat￾ical reasoning. To better understand … view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Instruction for generating verbalized math questions and assessing their ambiguity [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Efficient Chain-of-Modality Reasoning via Progressive Compression for Spoken Language Models

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Spoken math models that emit a 40%-compressed reasoning trace between question and answer beat full-reasoning baselines by ~3 accuracy points while using roughly one third of the text tokens.

Reference graph

Works this paper leans on

53 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. 2025. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:2503.01743

    arXiv 2025

  4. [4]

    Keyu An, Qian Chen, Chong Deng, Zhihao Du, Changfeng Gao, Zhifu Gao, Yue Gu, Ting He, Hangrui Hu, Kai Hu, et al. 2024. Funaudiollm: Voice understanding and generation foundation models for natural interaction between humans and llms. arXiv preprint arXiv:2407.04051

    arXiv 2024

  5. [5]

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. 2024. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759

    arXiv 2024

  6. [6]

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919

    arXiv 2023

  7. [7]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

  8. [8]

    Wenqian Cui, Xiaoqi Jiao, Ziqiao Meng, and Irwin King. 2025. Voxeval: Benchmarking the knowledge understanding capabilities of end-to-end spoken language models. arXiv preprint arXiv:2501.04962

    arXiv 2025

Show all 53 references
  1. [9]

    Wenqian Cui, Dianzhi Yu, Xiaoqi Jiao, Ziqiao Meng, Guangyan Zhang, Qichao Wang, Yiwen Guo, and Irwin King. 2024. Recent advances in speech language models: A survey. arXiv preprint arXiv:2410.03751

    2024 arXiv

  2. [10]

    Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. 2024. Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities. arXiv preprint arXiv:2406.11768

    2024 arXiv

  3. [11]

    Yuan Gong, Alexander H Liu, Hongyin Luo, Leonid Karlinsky, and James Glass. 2023. Joint audio and speech understanding. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1--8. IEEE

    2023

  4. [12]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  5. [13]

    Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven Hoi. 2023. From images to textual prompts: Zero-shot visual question answering with frozen large language models. In Proceedings of the IEEE/CVF conference on computer vision and pat...

    2023

  6. [14]

    Xiaotian Han, Quanzeng You, Yongfei Liu, Wentao Chen, Huangjie Zheng, Khalil Mrini, Xudong Lin, Yiqi Wang, Bohan Zhai, Jianbo Yuan, et al. 2023. Infimm-eval: Complex open-ended reasoning evaluation for multi-modal large language models. arXiv preprint arXiv:2311.11567

    2023 arXiv

  7. [15]

    Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng. 2025. Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark. arXiv preprint arXiv:2501.05444

    2025 arXiv

  8. [16]

    Yingxu He, Zhuohan Liu, Shuo Sun, Bin Wang, Wenyu Zhang, Xunlong Zou, Nancy F Chen, and Ai Ti Aw. 2024. Meralion-audiollm: Technical report. arXiv preprint arXiv:2412.09818

    2024 arXiv

  9. [17]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. NeurIPS

    2021

  10. [18]

    Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. Learning to solve arithmetic word problems with verb categorization. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 523--533

    2014

  11. [19]

    Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Feiyu Shen, Jingbei Li, Mingrui Chen, et al. 2025. Step-audio: Unified understanding and generation in intelligent speech interaction. arXiv preprint arXiv:2502.11946

    2025 arXiv

  12. [20]

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

    2024 arXiv

  13. [21]

    Identifying and mitigating vulnerabilities in llm-integrated applications

    Fengqing Jiang, Zhangchen Xu, Luyao Niu, Boxin Wang, Jinyuan Jia, Bo Li, and Radha Poovendran. Identifying and mitigating vulnerabilities in llm-integrated applications. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following

    2023

  14. [22]

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. 2022. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Syste...

    2022

  15. [23]

    Fang Liu, Yang Liu, Lin Shi, Houkun Huang, Ruifeng Wang, Zhen Yang, Li Zhang, Zhongqi Li, and Yuchi Ma. 2024. Exploring and evaluating hallucinations in llm-powered code generation. arXiv preprint arXiv:2404.00971

    2024

  16. [24]

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255

    2023 arXiv

  17. [25]

    Loren Lugosch, Piyush Papreja, Mirco Ravanelli, Abdelwahab Heba, and Titouan Parcollet. 2021. Timers and such: A practical benchmark for spoken language understanding with numbers. In 35th Conference on Neural Information Processing Systems (NeurIPS 2021) Track on Datasets and...

    2021

  18. [26]

    Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. 2022. Lila: A unified benchmark for mathematical reasoning. In Proceedings of the 2022 Conference on Empiri...

    2022

  19. [27]

    Mistral AI . 2024. Mathstral 7b v0.1. https://mistral.ai/news/mathstral. A 7B model designed for math reasoning and scientific discovery. Released under the Apache 2.0 license

    2024

  20. [28]

    Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080--2094

    2021

  21. [29]

    Jing Peng, Yucheng Wang, Yu Xi, Xu Li, Xizhuo Zhang, and Kai Yu. 2024. A survey on speech large language models. arXiv preprint arXiv:2410.18908

    2024

  22. [30]

    Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, et al. 2024. We-math: Does your large multimodal model achieve human-like mathematical reasoning? arXiv preprint arXiv:2407.01284

    2024 arXiv

  23. [31]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492--28518. PMLR

    2023

  24. [32]

    Subhro Roy and Dan Roth. 2015. Solving general arithmetic word problems. In Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, pages 1743--1752. Association for Computational Linguistics (ACL)

    2015

  25. [33]

    S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha. 2024. Mmau: A massive multi-task audio understanding and reasoning benchmark. arXiv preprint arXiv:2410.19168

    2024 arXiv

  26. [34]

    Tomohiro Sawada, Daniel Paleka, Alexander Havrilla, Pranav Tadepalli, Paula Vidas, Alexander Kranias, John J Nay, Kshitij Gupta, and Aran Komatsuzaki. 2023. Arb: Advanced reasoning benchmark for large language models. arXiv preprint arXiv:2307.13692

    2023 arXiv

  27. [35]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    2024 arXiv

  28. [36]

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri \`a Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv p...

    2022 arXiv

  29. [37]

    Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2023. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:2310.13289

    2023 arXiv

  30. [38]

    Qwen Team. 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  31. [39]

    Gemma TeamGemma, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118

    2024 arXiv

  32. [40]

    Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, et al. 2025. Llamav-o1: Rethinking step-by-step visual reasoning in llms. arXiv preprint arXiv:2501.06186

    2025 arXiv

  33. [41]

    Shubham Toshniwal, Ivan Moshkov, Sean Narenthiran, Daria Gitman, Fei Jia, and Igor Gitman. 2024. Openmathinstruct-1: A 1.8 million math instruction tuning dataset. arXiv preprint arXiv: Arxiv-2402.10176

    2024 arXiv

  34. [42]

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. 2024. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095--95169

    2024

  35. [43]

    Chengwei Wei, Bin Wang, Jung-jae Kim, Guimei Liu, and Nancy F Chen. 2024. Coinmath: Harnessing the power of coding instruction for math llms. arXiv preprint arXiv:2412.11699

    2024 arXiv

  36. [44]

    Zhifei Xie, Mingbao Lin, Zihang Liu, Pengcheng Wu, Shuicheng Yan, and Chunyan Miao. 2025. Audio-reasoner: Improving reasoning capability in large audio language models. arXiv preprint arXiv:2503.02318

    2025

  37. [45]

    Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. 2024. Llava-o1: Let vision language models reason step-by-step. arXiv preprint arXiv:2411.10440

    2024 arXiv

  38. [46]

    Weiye Xu, Jiahao Wang, Weiyun Wang, Zhe Chen, Wengang Zhou, Aijun Yang, Lewei Lu, Houqiang Li, Xiaohua Wang, Xizhou Zhu, et al. 2025. Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models. arXiv preprint arXiv:2504.15279

    2025 arXiv

  39. [48]

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. 2024 b . Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122

    2024 arXiv

  40. [49]

    Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, et al. 2024 c . Air-bench: Benchmarking large audio-language models via generative comprehension. arXiv preprint arXiv:2402.07729

    2024 arXiv

  41. [50]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conferenc...

    2024

  42. [51]

    Zhongshen Zeng, Yinhong Liu, Yingjia Wan, Jingyao Li, Pengguang Chen, Jianbo Dai, Yuxuan Yao, Rongwu Xu, Zehan Qi, Wanru Zhao, et al. 2024. Mr-ben: A meta-reasoning benchmark for evaluating system-2 thinking in llms. arXiv preprint arXiv:2406.13975

    2024 arXiv

  43. [52]

    Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2023. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15757--15773

    2023

  44. [53]

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. 2024. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169--186...

    2024

  45. [54]

    Kunhao Zheng, Juliette Decugis, Jonas Gehring, Taco Cohen, Benjamin Negrevergne, and Gabriel Synnaeve. 2024. What makes large language models reason in (multi-turn) code generation? arXiv preprint arXiv:2410.08105

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.