J-EDI QA: Benchmark for deep-sea organism-specific multimodal LLM

Daisuke Matsuoka; Daisuke Sugiyama; Shinji Tsuchida; Takero Yoshida; Yoshihiro Fujiwara; Yuikazu Ito

REVIEW 4 major objections 5 minor 21 references

J-EDI QA: Benchmark for deep-sea organism-specific multimodal LLM

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read J-EDI QA proposes that current multimodal LLMs, including OpenAI o1, cannot yet identify deep-sea organisms at expert level, scoring only 50% on a new Japanese benchmark of 100 images.

desk verdict A genuinely new but small benchmark for deep-sea organism VQA; the 50% result is plausible, but the 'below expert level' claim outruns the evidence because no expert baseline was measured. read the letter →

arxiv 2412.15574 v1 pith:NL5OPG6J submitted 2024-12-20 cs.CV

Takero Yoshida , Yuikazu Ito , Yoshihiro Fujiwara , Shinji Tsuchida , Daisuke Sugiyama , Daisuke Matsuoka This is my paper · ORCID

classification cs.CV

keywords deep-seaorganismsmultimodalLLMbenchmarkJ-EDIJapaneseQAspeciesidentification

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces J-EDI QA, a benchmark of 100 deep-sea images from the JAMSTEC archive, each paired with a four-choice identification question written in Japanese by JAMSTEC researchers. The authors evaluate two multimodal large language models, OpenAI o1 and GPT-4o, and report that o1 answers 50% of the questions correctly while GPT-4o answers 39%. They interpret this as evidence that state-of-the-art general-purpose models, as of December 2024, have not reached expert-level comprehension of deep-sea organisms. The benchmark is intended to support the development of deep-sea-specific LLMs and to serve as a test of public understanding. The central claim is that the 50% result indicates a real gap between current models and expert marine biologists.

What carries the argument

The central object is the J-EDI QA benchmark itself: 100 images selected from the J-EDI archive, each with a four-option multiple-choice question in Japanese written by JAMSTEC researchers, plus an expert commentary explaining the correct identification. The evaluation protocol uploads only the image, asks the model to choose an answer and justify it, and scores the percentage of correct choices. This machinery allows the authors to compare model performance against expert answers and against each other, and to separate identification accuracy from the ability to provide a rationale.

What would settle it

Run the identical 100 questions on the same models using images stripped of the ©JAMSTEC watermark and cropped to remove background habitat cues; if accuracy drops markedly below 50%, the original result was at least partly an artifact of contextual leakage rather than organism identification.

Watch

Extended reading notes

Core claim

The paper's central claim is that J-EDI QA measures something existing benchmarks do not: the ability to identify deep-sea organisms from survey images, and to do so in Japanese. On this benchmark, the best model tested, OpenAI o1, achieves 50% correct, with GPT-4o at 39%, while two human subjects with some knowledge of deep-sea organisms score about 40%. The authors conclude that current multimodal LLMs have only rudimentary understanding of deep-sea species and remain below the level of JAMSTEC researchers. They further observe that models sometimes rely on contextual cues such as hydrothermal deposits in the background or the ©JAMSTEC watermark, indicating that their reasoning is not purely based on the organism's features.

Load-bearing premise

A load-bearing premise is that the 100 selected images and their expert-written QA pairs are representative and correctly labeled, so that model accuracy on this set measures general deep-sea organism comprehension.

Editorial extensions

If this is right

If the claim holds, general-purpose multimodal models cannot be relied on for automated identification of deep-sea organisms from images alone.
The benchmark provides a reusable Japanese-language test for tracking progress of deep-sea-specific multimodal LLMs.
The low accuracy motivates training and retrieval-augmented generation with non-digital expert resources such as illustrated field guides.
The J-EDI archive's video data could be used to construct a video-version benchmark, testing temporal and behavioral identification.
The benchmark can also function as a public education and outreach tool for deep-sea biology.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because all images come from one archive with surveys concentrated around Japan, the 50% figure may not generalize to deep-sea biota from other regions; a geographically diverse sample would test this.
The paper's observation that models use background cues like hydrothermal deposits and the ©JAMSTEC watermark suggests that accuracy could be inflated by contextual leakage; removing those cues would likely lower scores.
The 25% random-guess baseline means 50% is clearly above chance, but the benchmark would be more informative with confidence scores or a measure of how often the model's chosen rationale matches the expert commentary.
A translated English version of the same 100 questions, which the authors mention as future work, would separate language proficiency in Japanese from biological knowledge.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

J-EDI QA is a genuine new resource: a 100-question Japanese benchmark for multimodal LLM identification of deep-sea organisms, built from JAMSTEC's J-EDI archive with QA pairs authored by domain researchers. That fills a niche that general and Japanese benchmarks like MMMU, JMMMU, and Heron-Bench don't cover. The raw results — o1 50%, GPT-4o 39% — are simple counts and I have no reason to doubt them. The paper is also refreshingly restrained about its own limitations: it says the dataset is small, notes that some images are hard to identify from the image alone, and reports that two non-expert humans scored about 40%.

The soft spots are mostly around interpretation and reproducibility. The headline claim that o1 is 'not yet at an expert level' is asserted, not demonstrated. No expert baseline was run on these 100 items, so we have no ceiling to compare against. The two human subjects were 'with some knowledge' and scored ~40%, which is below o1 — so the models may already be above non-expert humans, and possibly close to what experts achieve on deliberately tricky images. That missing control is load-bearing: the abstract's 'expert level' language needs it. Second, 100 questions is a thin sample; there are no confidence intervals or item-level statistics, so the 39 vs 50 gap between models is not clearly meaningful. Third, the selection criteria for images are not documented beyond 'chosen by JAMSTEC researchers,' and the QA pairs are only available on individual request rather than released with the paper, limiting independent verification. Fourth, the claim that this measures Japanese-language comprehension is plausible but not isolated — there is no English-parallel run.

This is a paper for people building domain-specific or Japanese-language multimodal benchmarks, and for marine biology AI. It is not a major empirical result, but the resource is a legitimate starting point. I would send it to peer review with a clear ask for revision: add an expert baseline (even on a subset), release the QA pairs and image list, report uncertainty, and recalibrate the 'expert level' wording. With those changes, the benchmark could be a useful reference point.

Referee Report

4 major / 5 minor

Summary. The paper introduces J-EDI QA, a Japanese-language multiple-choice benchmark for multimodal LLMs, built from 100 deep-sea organism images selected from the JAMSTEC J-EDI archive, with questions and answers authored by JAMSTEC researchers. The authors evaluate OpenAI o1 and GPT-4o, reporting 50% and 39% accuracy respectively, and interpret the o1 result as indicating that state-of-the-art multimodal models are not yet at an expert level for deep-sea species comprehension. The appendix lists URLs, questions, options, and answer keys for all 100 items. The paper also reports that two non-expert human subjects scored approximately 40%, and discusses model reasoning behaviors and limitations.

Significance. If the benchmark is valid and made publicly available, it would fill a specific niche: a domain-specific, Japanese-language multimodal benchmark for deep-sea imagery, which could support development and evaluation of deep-sea organism identification systems. The raw accuracy counts are simple and presumably correct, and the authors are candid about the small size of the dataset and the need for an English-translated parallel assessment. However, the headline claim that current models are 'not yet at an expert level' rests on an unmeasured expert baseline, and the benchmark itself is not yet released, so the paper's current contribution is closer to a proof-of-concept than a fully validated benchmark.

major comments (4)

[Abstract; §3.2] The central claim that o1 'is not yet at an expert level' is not supported by the evidence presented, because no expert human baseline was measured on the same 100 items. Section 3.2 reports only two non-expert human subjects at approximately 40%, and the paper itself states that some images were difficult to identify from the images alone. Without a JAMSTEC expert accuracy on these exact questions, a 50% model score cannot be interpreted as below expert level; it could in principle be at or above the expert ceiling if the image set is unusually hard. The abstract's interpretive sentence should be reworded to state what was actually measured (model accuracy on this benchmark), and the expert-level comparison should either be supported by an explicit expert baseline or explicitly deferred.
[§2.2; Data availability] The benchmark's representativeness is not established. The 100 images are described only as 'selected by JAMSTEC researchers' with no sampling criteria, no annotation protocol, and no inter-annotator agreement, and the compiled 100-image benchmark is not publicly released ('will be made available at a later date'). This makes it impossible for readers to assess whether the 50% figure reflects general deep-sea organism comprehension or a particular, possibly idiosyncratic, selection. The paper should document the selection process, release the benchmark, and ideally report agreement statistics on the expert-authored answers before claiming that the benchmark measures the target competency.
[§3.1; §3.2] All accuracy results are reported as point estimates without confidence intervals or significance tests. With n=100, the standard error of a proportion is approximately 5 percentage points, so the observed o1 versus GPT-4o gap (50% vs 39%) is not clearly distinguishable from sampling variability, and the per-category breakdowns (e.g., 14/20 vs 9/20 for crustaceans) have even wider intervals. The claims that o1 'had a higher percentage of correct answers' and that crustacean performance is 'particularly high' should be qualified with confidence intervals or significance tests.
[§1; Conclusion] The abstract and introduction describe the benchmark as assessing 'deep-sea species and Japanese terminology comprehension,' but no English-translated version was evaluated, so the Japanese-language component is confounded with visual identification ability. The conclusion acknowledges that an English parallel assessment is needed; this should be presented as an explicit limitation of the current results rather than only as future work.

minor comments (5)

[References] Reference [20] is cited for JA-VLM-Bench-In-the-Wild but the listed title is 'Evolutionary Optimization of Model Merging Recipes' by Takuya Akiba et al.; this reference appears incorrect or mismatched.
[§3.1] Typo: 'OenAI o1' should be 'OpenAI o1'.
[Appendix Table 1] The table is difficult to use because URLs are broken across lines and some rows appear to have missing or merged option cells (e.g., row 29); providing a machine-readable supplementary file with image IDs, options, and answers would greatly improve usability.
[§2.2] The phrase 'a selection question was included to facilitate the potential for erroneous responses' is unclear; please clarify what a selection question is and how it relates to the distractor design.
[Data availability] The statement that the benchmark 'will be made available at a later date, but for now they will be available on individual request' is internally inconsistent; specify a concrete release plan or repository.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: J-EDI QA is a self-contained benchmark evaluation whose expert-authored ground truth is independent of the model outputs being measured.

full rationale

The paper constructs a 100-image Japanese QA benchmark from the JAMSTEC J-EDI archive, with questions and correct answers authored by JAMSTEC researchers, and then evaluates OpenAI o1 and GPT-4o by uploading only images and prompts. Accuracy is computed against these independent expert labels, and no parameter is fitted from model responses, no prediction is derived from a model-derived input, and no load-bearing claim depends on a self-citation chain. The abstract's conclusion that deep-sea species comprehension is 'not yet at an expert level' is an interpretive claim that would be strengthened by a directly measured expert baseline, but the absence of such a baseline is an evidentiary limitation, not a circular reduction. The two non-expert human subjects, described as possessing 'some knowledge of deep-sea organisms,' are not used to define the expert ceiling. Because the benchmark is externally anchored in expert-authored ground truth and the reported scores are direct measurements, the central result is not equivalent to its inputs by construction.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No mathematical derivation or fitted parameters are present; the benchmark's validity rests on expert label accuracy, sample representativeness, and the multiple-choice format as a comprehension metric. No new physical or conceptual entities are postulated.

assumptions (3)

domain assumption Expert labels by JAMSTEC researchers are correct.
The benchmark's ground truth relies on JAMSTEC researchers' identification; no independent verification or inter-annotator agreement is reported.
ad hoc to paper The 100 selected images are representative of deep-sea organism identification difficulty.
The selection process is not described; the authors state only that researchers selected 100 images, so the sample may not generalize to J-EDI as a whole.
domain assumption Four-choice Japanese multiple-choice accuracy measures deep-sea species comprehension.
The paper uses percent correct as a proxy for species comprehension and Japanese terminology understanding, but does not validate this metric against expert judgement.

how reviews work

0 comments

Cite this review

Pith. "Pith review of J-EDI QA: Benchmark for deep-sea organism-specific multimodal LLM." pith.science (2026). https://pith.science/paper/NL5OPG6J

@misc{pith2026241215574,
  author       = {Pith},
  title        = {Pith review of: J-EDI QA: Benchmark for deep-sea organism-specific multimodal LLM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NL5OPG6J}},
  note         = {Machine review of arXiv:2412.15574}
}

read the original abstract

Japan Agency for Marine-Earth Science and Technology (JAMSTEC) has made available the JAMSTEC Earth Deep-sea Image (J-EDI), a deep-sea video and image archive (https://www.godac.jamstec.go.jp/jedi/e/index.html). This archive serves as a valuable resource for researchers and scholars interested in deep-sea imagery. The dataset comprises images and videos of deep-sea phenomena, predominantly of marine organisms, but also of the seafloor and physical processes. In this study, we propose J-EDI QA, a benchmark for understanding images of deep-sea organisms using a multimodal large language model (LLM). The benchmark is comprised of 100 images, accompanied by questions and answers with four options by JAMSTEC researchers for each image. The QA pairs are provided in Japanese, and the benchmark assesses the ability to understand deep-sea species in Japanese. In the evaluation presented in this paper, OpenAI o1 achieved a 50% correct response rate. This result indicates that even with the capabilities of state-of-the-art models as of December 2024, deep-sea species comprehension is not yet at an expert level. Further advances in deep-sea species-specific LLMs are therefore required.

Figures

Figures reproduced from arXiv: 2412.15574 by the authors.

**Figure 1.** Example image, referenced from https://www.godac.jamstec.go.jp/jedi/player/j/HPD1014HDTV0126 Answer by GPT-4o： “この画像の生物は「エドアブラザメ」です。(The creature in this image is the Heptranchias perlo.) 解説(explanation): 画像に写っているサメは、典型的なエドアブラザメの特徴を持っています。エドアブラザメは深海に生息するサメの一種で、体が比較的丸みを帯びており、頭部が大きく、鋭い目が特徴です。背びれは低く、全体的に滑らかな外観をしています。(The shark depicted in the image exhibits the typical characteristics of the species Heptranchias perl… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 21 linked inside Pith

[1]

Gemini Team, Google

Gemini 1.5 : Unlocking multimodal understanding across millions of tokens of context. Gemini Team, Google. : arXiv preprint, arXiv:2403.05530v4, 2024

arXiv 2024
[2]

Xiang Yue, et al

MMMU: A Massive Multi -discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. Xiang Yue, et al. : arXiv preprint, arXiv:2311.16502v4, 2024

arXiv 2024
[3]

Pan Lu, et al

MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts. Pan Lu, et al. : Proc. of ICLR2024, arXiv:2310.02255, 2023

arXiv 2023
[4]

Haotian Liu, et al

Improved Baselines with Visual Instruction Tuning. Haotian Liu, et al. : Proc. of CVPR2024, arXiv:2310.03744, 2024

arXiv 2024
[5]

Ji Lin, et al

VILA: On Pre-training for Visual Language Models. Ji Lin, et al. : Proc. of CVPR2024, arXiv:2312.07533, 2024

arXiv 2024
[6]

Amanpreet Singh, et al

Towards VQA Models That Can Read. Amanpreet Singh, et al. : Proc. of CVPR2019, arXiv:1904.08920, 2019

arXiv 1904
[7]

Weihao Yu, et al

MM -Vet: Evaluating Large Multimodal Models for Integrated Capabilities. Weihao Yu, et al . : arXiv preprint, arXiv:2308.02490, 2023

arXiv 2023
[8]

Bohao Li, et al

SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension. Bohao Li, et al. : arXiv preprint, arXiv:2307.16125, 2023

arXiv 2023

Show all 21 references

[9]

Chaoyou Fu, et al

MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. Chaoyou Fu, et al. : arXiv preprint, arXiv:2306.13394, 2024

2024 arXiv
[10]

GQA: A New Dataset for Real -World Visual Reasoning and Compositional Question Answering. Drew A. Hudson, Christopher D. Manning : Proc. of CVPR 2019, arXiv:1902.09506, 2019

2019 arXiv
[11]

Peng Xu, et al

LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision -Language Models. Peng Xu, et al. : arXiv preprint, arXiv:2306.09265, 2023

2023 arXiv
[12]

Ahmed Masry, et al

ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning. Ahmed Masry, et al. : Proc. of ACL 2022, arXiv:2203.10244, 2022

2022 arXiv
[13]

Grégoire Mialon, et al

GAIA: a benchmark for General AI Assistants. Grégoire Mialon, et al. : arXiv preprint, arXiv:2311.12983, 2023

2023 arXiv
[14]

Zhenfei Yin, et al

LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark. Zhenfei Yin, et al. : Proc. of NeurIPS 2023, arXiv:2306.06687, 2023

2023 arXiv
[15]

Xiang Yue, et al

MMMU -Pro: A More Robust Multi -discipline Multimodal Understanding Benchmark. Xiang Yue, et al . : arXiv preprint, arXiv:2409.02813, 2024

2024 arXiv
[16]

Xin Wang, et al

VATEX: A Large -Scale, High-Quality Multilingual Dataset for Video -and-Language Research. Xin Wang, et al. : Proc. of ICCV 2019, arXiv:1904.03493, 2020

2019 arXiv
[17]

Viorica Pătrăucean, et al

Perception Test: A Diagnostic Benchmark for Multimodal Video Models. Viorica Pătrăucean, et al. : Proc. of NeurIPS 2023, arXiv:2305.13786, 2023

2023 arXiv
[18]

Pan Lu, et al

Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering. Pan Lu, et al. : Proc. of NeurIPS 2022, arXiv:2209.09513, 2022

2022 arXiv
[19]

Yuichi Inoue, et al

Heron-Bench: A Benchmark for Evaluating Vision Language Models in Japanese. Yuichi Inoue, et al. : arXiv preprint, arXiv:2404.07824, 2024

2024 arXiv
[20]

arXiv preprint, arXiv: 2403.13187, 2024

Evolutionary Optimization of Model Merging Recipes, Takuya Akiba et al. arXiv preprint, arXiv: 2403.13187, 2024

2024 arXiv
[21]

arXiv preprint, arXiv:2410.17250, 2024

JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluation, Shota Onohara et al. arXiv preprint, arXiv:2410.17250, 2024. Appendix Table 1: Table of questions, images and answers No. URL Question and image Option A Option B Option...

2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Gemini Team, Google

Gemini 1.5 : Unlocking multimodal understanding across millions of tokens of context. Gemini Team, Google. : arXiv preprint, arXiv:2403.05530v4, 2024

arXiv 2024

[2] [2]

Xiang Yue, et al

MMMU: A Massive Multi -discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. Xiang Yue, et al. : arXiv preprint, arXiv:2311.16502v4, 2024

arXiv 2024

[3] [3]

Pan Lu, et al

MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts. Pan Lu, et al. : Proc. of ICLR2024, arXiv:2310.02255, 2023

arXiv 2023

[4] [4]

Haotian Liu, et al

Improved Baselines with Visual Instruction Tuning. Haotian Liu, et al. : Proc. of CVPR2024, arXiv:2310.03744, 2024

arXiv 2024

[5] [5]

Ji Lin, et al

VILA: On Pre-training for Visual Language Models. Ji Lin, et al. : Proc. of CVPR2024, arXiv:2312.07533, 2024

arXiv 2024

[6] [6]

Amanpreet Singh, et al

Towards VQA Models That Can Read. Amanpreet Singh, et al. : Proc. of CVPR2019, arXiv:1904.08920, 2019

arXiv 1904

[7] [7]

Weihao Yu, et al

MM -Vet: Evaluating Large Multimodal Models for Integrated Capabilities. Weihao Yu, et al . : arXiv preprint, arXiv:2308.02490, 2023

arXiv 2023

[8] [8]

Bohao Li, et al

SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension. Bohao Li, et al. : arXiv preprint, arXiv:2307.16125, 2023

arXiv 2023