The reviewed record of science sign in
Pith

arxiv: 2312.11444 · v2 · pith:DJBZOKBC · submitted 2023-12-18 · cs.CL · cs.AI

An In-depth Look at Gemini's Language Abilities

Syeda Nahida Akter , Zichun Yu , Aashiq Muhamed , Tianyue Ou , Alex B\"auerle , \'Angel Alexander Cabrera , Krish Dholakia , Chenyan Xiong
show 1 more author
Graham Neubig
This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:DJBZOKBCrecord.jsonopen to challenge →

classification cs.CL cs.AI
keywords geminiabilitiescodeincludinglanguagereasoningresultsanalysis
0
0 comments X
read the original abstract

The recently released Google Gemini class of models are the first to comprehensively report results that rival the OpenAI GPT series across a wide variety of tasks. In this paper, we do an in-depth exploration of Gemini's language abilities, making two contributions. First, we provide a third-party, objective comparison of the abilities of the OpenAI GPT and Google Gemini models with reproducible code and fully transparent results. Second, we take a closer look at the results, identifying areas where one of the two model classes excels. We perform this analysis over 10 datasets testing a variety of language abilities, including reasoning, answering knowledge-based questions, solving math problems, translating between languages, generating code, and acting as instruction-following agents. From this analysis, we find that Gemini Pro achieves accuracy that is close but slightly inferior to the corresponding GPT 3.5 Turbo on all tasks that we benchmarked. We further provide explanations for some of this under-performance, including failures in mathematical reasoning with many digits, sensitivity to multiple-choice answer ordering, aggressive content filtering, and others. We also identify areas where Gemini demonstrates comparably high performance, including generation into non-English languages, and handling longer and more complex reasoning chains. Code and data for reproduction can be found at https://github.com/neulab/gemini-benchmark

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. No Universal Courtesy: A Cross-Linguistic, Multi-Model Study of Politeness Effects on LLMs Using the PLUM Corpus

    cs.CL 2026-04 unverdicted novelty 6.0

    Politeness in prompts boosts average LLM response quality by up to 11% but the benefit is language- and model-dependent, with English favoring courteous tones, Hindi deferential ones, and Spanish assertive ones.

  2. MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices

    cs.CV 2023-12 unverdicted novelty 5.0

    MobileVLM achieves on-par performance with much larger vision-language models on standard benchmarks while delivering state-of-the-art inference speeds of 21.5 tokens per second on Snapdragon 888 CPU and 65.3 on Jetso...

  3. Structured Labeling Enables Faster Vision-Language Models for End-to-End Autonomous Driving

    cs.CV 2025-06 unverdicted novelty 4.0

    Introduces structured NuScenes-S dataset and 0.9B FastDrive VLM claiming 20% higher decision accuracy and over 10x inference speedup versus larger unstructured VLMs.

  4. MobileVLM V2: Faster and Stronger Baseline for Vision Language Model

    cs.CV 2024-02 unverdicted novelty 4.0

    MobileVLM V2 shows that 1.7B and 3B parameter vision-language models can reach or exceed the performance of 3B and 7B+ models on common VLM benchmarks via targeted design and data improvements.