REVIEW 3 major objections 5 minor 25 references
MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization
T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash
Pith's one-line read A controlled benchmark for repository-level issue localization shows that visual evidence carries real signal but that current systems use it unevenly, with the best agent reaching only 38.96 file Acc@5 and the best retriever 33.86 function
desk verdict A genuinely useful controlled benchmark for visual evidence in issue localization, with a real gold-label caveat and missing variance; send it to referees. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the benchmark's paired evaluation protocol: each instance can be run text-only, with raw images, with structured visual text (VCE), or with both, so visual evidence is an independent variable rather than a confound. Per-image annotations (7 evidence categories, 4 relevance levels, including a human-reviewed harmful-image subset) plus strict all-gold Acc@K scoring turn image use into a testable cause. The paper also contributes a controlled multimodal retriever, MM-IssueLoc-VL-Embedding, trained with a contrastive objective and a file-to-function curriculum, whose no-image ablation isolates the image contribution from the retriever's text understanding.
What would settle it
Train the paper's controlled retriever on the same training rows with all images stripped, and run it on the 343-instance function subset; if its function Acc@10 matches or exceeds the multimodal retriever's 33.86, then the benchmark's 'visual signal' is actually explainable by text cues, refuting the claim that images carry distinct localization information.
Extended reading notes
Core claim
The paper claims that visual evidence in repository issues is a measurable, and currently under-exploited, variable for code localization. It constructs a controlled benchmark of 652 real issue-PR pairs with per-image relevance and category annotations and both file- and function-level gold labels, then shows three things: no current system localizes reliably under strict all-gold Acc@K (best file Acc@5 = 38.96, best function Acc@10 = 33.86); removing images costs a controlled retriever 4.44-4.91 points of file Acc@5, so images themselves carry signal; and strong localization on text-only issue-resolving benchmarks does not transfer cleanly to multimodal issue localization. The paper's contr
Load-bearing premise
The benchmark assumes that the files and functions edited by the merged pull request are exactly the locations a good localizer should output; if real pull requests contain unrelated refactors, mass renames, or incomplete fixes, every Acc@K score measures patch alignment rather than localization skill.
Editorial extensions
If this is right
- Visual evidence is a measurable localization variable: removing images from the paper's controlled retriever drops file Acc@5 by 4.44 to 4.91 points, showing the images themselves, not just the issue text, carry signal.
- Current systems are far from reliable: the strongest agent misses all correct files in its top 5 for over 60% of instances (38.96 Acc@5), and the strongest retriever reaches only 33.86 Acc@10 at function level.
- Multi-edit issues are the main bottleneck: Acc@10 collapses from roughly 83 on easy instances to roughly 3 on hard instances for the top agent, so strict all-gold accuracy on hard issues is near zero.
- Strong text-only localization does not transfer to multimodal settings: the same agent family that scores above 90 File@5 on text-only issue-resolving benchmarks drops to around 43 on a multimodal issue-resolving benchmark, motivating dedicated multimodal localization tests.
- Structured textualization of images (VCE) can be as helpful as raw pixels for some agents, and sometimes more: several agent-framework combinations improve file Acc@5 with VCE alone, while raw-pixel residuals vary widely across models.
Reading between the lines
- The paired with/without-image protocol is a template: future 'does vision help?' questions in software engineering could reuse this design to avoid the confound where a model's patch ability masks localization failures.
- Because VCE converts images to text, the results suggest that lightweight OCR-and-structured-extraction pipelines might let existing text-only localizers gain some multimodal ability without retraining on image data.
- The benchmark's harmful-image subset, though synthetic, implies that future localizers should be evaluated not just on whether they use images but on whether they can resist misleading ones; scaling up naturally occurring harmful images is a testable extension.
- If the cross-benchmark non-transfer holds, then text-only scores on issue-resolving benchmarks are not a valid proxy for multimodal localization ability; benchmark designers should treat multimodal localization as a separate evaluation axis.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces MM-IssueLoc, a repository-level issue-localization benchmark designed to make visual evidence an explicit evaluation variable. It contains 652 real issue-PR instances (1,050 images, 23 languages), with file-level gold labels for all instances and function-level labels for 343, plus per-image category/relevance annotations and harmful-image controls. The paper evaluates LLM-based agents and retrieval-based systems under four input modes (text-only, with-image, VCE, VCE+image), including a newly trained controlled retriever MM-IssueLoc-VL-Embedding. Results show a large capability gap: the best agent achieves 38.96 file Acc@5 and 22.45 function Acc@10; the best retriever achieves 33.86 function Acc@10. The paper also reports cross-benchmark comparisons indicating that strong text-only localization performance on SWE-bench does not transfer to multimodal localization, and concludes that visual evidence can help but is used unevenly by current systems.
Significance. If the benchmark is valid, it addresses a real gap: existing SWE benchmarks either strip images or evaluate end-to-end repair, making it impossible to isolate whether visual evidence helps localization. MM-IssueLoc provides a multi-lingual, multi-granularity dataset with per-image annotations, paired input modes, harmful-image stress tests, and a controlled retriever probe. The pipeline is described in sufficient detail to be reproducible, and the authors release per-instance predictions, which is a strength. The paper also makes a falsifiable claim—current systems localize multimodal issues poorly—that seems robust to moderate label noise. However, the central validity of the benchmark rests on unvalidated PR-diff gold labels, and all headline numbers are single-run without variance or significance testing. These issues need to be addressed before the benchmark can fully support the paper's 'controlled variable' claim.
major comments (3)
- [§3.1 and §6] The gold file/function sets F*_i and G*_i are exactly the files/functions edited by the merged human PR, with no human verification that each gold entry is necessary for the fix. The paper acknowledges in §6 that 'such labels are not minimal causal explanations,' but this is load-bearing for the benchmark's central claim that it 'turns visual evidence into an explicit evaluation variable.' If merged PR diffs contain incidental edits (mass renames, lockfile updates, unrelated refactors, generated-file changes), then the strict all-gold Acc@K metric (Eq. 4) penalizes correct localization on those instances, and the measured effect of images may reflect patch-alignment skill rather than localization skill. This concern applies to every headline comparison, including the cross-benchmark transfer results. Please quantify the incidence of such incidental edits (e.g., by human review of a rando
- [§5.2, Table 3] All performance numbers are single-run, with no variance, confidence intervals, or significance tests. This is particularly problematic for the RQ2 conclusion that visual evidence is useful: the retriever gains of 4.91 and 4.44 File@5 points after removing images could be within run-to-run noise, and the agent-level Δimg values (−2.66, −0.31, +0.92) are even smaller. Similarly, in Table 2 the 1.21-point difference between MM-IssueLoc-VL-Emb-8B and Qwen3-VL-Emb-8B on Func@10 (33.86 vs 32.65) is interpreted as an advantage, but no significance is reported. Since LLM-based agents are stochastic and the benchmark size (652/343 instances) is modest, please report at least three seeds (or bootstrap intervals) for the main tables, or explicitly justify why single-run comparisons are stable for this setting.
- [§5.3, Table 4] The VCE diagnostic table shows very large variance across frameworks and backends (ρ_pix ranges from −11.69 to +4.74). The claim that 'raw images do not consistently add value once VCE is provided' is based on single-run differences that overlap substantially in magnitude with the observed framework-level variation. This may be a presentation issue if the authors add significance testing, but currently the raw-pixel residual claim is not supported by the evidence as presented. Please provide error bars or a clearer statistical treatment, and temper the conclusion accordingly.
minor comments (5)
- [Abstract vs §3.4] The abstract and Figure 1 say '23 programming languages,' while §3.4 and the 'benchmark statistics' sentence say '24 programming languages.' Please reconcile.
- [§3.2] The text says 650 repository snapshots for 652 instances; please clarify whether two instances share a snapshot or whether the counts differ for another reason.
- [Table 2 / Table 3] Model names are abbreviated inconsistently (MM-IssueLoc-VL-Emb vs MM-IssueLoc-VL-Embedding, Qwen3-VL-Emb vs Qwen3-VL-Embedding). Please standardize.
- [References] The Radford et al. reference contains a typo ('PmLR' should be 'PMLR'). Please proofread the reference list.
- [Figure 3] Panel C labels 'OpenHands harness' and 'Mini-SWE-Agent harness' are confusing: the x-axis is valid submission rate and the y-axis is File@5. Consider a clearer caption describing that these are two different agent frameworks with the same backend.
Circularity Check
No load-bearing circularity: the benchmark's central claim is externally grounded with held-out evaluations; one secondary RQ1 'difficulty collapse' finding is partly entailed by the all-gold Acc@K metric combined with the changed-file-count difficulty definition.
-
self definitional
[§5.1 (RQ1), with Eq. 4 (§4.3) and difficulty buckets (§3.2, Fig. 1)]
"performance collapses with difficulty. OpenHands GPT-5.2 drops from 83.10 Acc@10 on easy instances to 2.84 on hard instances, while MM-IssueLoc-VL-Emb-8B drops from 74.18 to 3.98, showing that multi-edit issues remain the main bottleneck."
Difficulty is defined as changed-file count (Fig. 1: Easy=1, Hard=>4), while Eq. 4 defines success as all gold files being in Top-K ('A prediction is successful only if all gold files or qualified functions appear in the top-K ranked list'). For hard instances with at least 5 gold files and K=10, the metric requires at least 5 designated files in the top 10, so any imperfect ranker's success probability falls roughly as a power of per-file recall. The reported collapse, and the conclusion that 'multi-edit issues remain the main bottleneck', is therefore substantially entailed by the metric combined with the difficulty-bucket definition rather than an independent empirical discovery about systems. The exact percentages are still empirical (a perfect ranker would score 100 on every bucket),
full rationale
The central benchmark claim — that MM-IssueLoc isolates visual evidence and that current systems remain far from reliable on multimodal localization — is not derived from the fitted retriever or from the paper's own assumptions. Gold labels F*_i/G*_i are externally sourced from merged GitHub PR diffs (§3.1), all Table 2 scores are held-out measurements under the standard strict Acc@K protocol (Eq. 4), and cross-benchmark gold for SWE-bench is derived from released repair patches by an external convention (Chen et al., 2025, cited in App. B.4). No load-bearing self-citation, imported uniqueness theorem, or ansatz-import is present; the cited baselines (LocAgent, SWERank, Qwen3-VL-Embedding, etc.) are third-party systems and conventions, not the authors' own prior results. The one exhibitable partial reduction is the RQ1 difficulty collapse described in the step above: because difficulty is defined by gold-file count and the metric requires recovering all gold files, the qualitative collapse and its 'multi-edit bottleneck' interpretation are partly constructed by the protocol. Two further points are self-referential but not circular. (a) 'MM-IssueLoc contains usable visual signal' (§5.2) is demonstrated with MM-IssueLoc-VL-Embedding, trained on instances from the same construction pipeline with the same gold definition; the instance-level disjointness (§3.4) makes this a genuine held-out generalization test and the negative delta_img in Table 3 is a measured ablation, not a fitted constant, so it is honest in-pipeline evidence. Notably, the external SWE-bench-MM ablation (Table 14) shows the image effect reverses (+0.98) for the same retriever, confirming the benefit is specific to the benchmark's own distribution, which limits how much the retriever alone establishes. (b) The paper itself concedes the gold-label limitation: 'such labels are not minimal causal explanations' (§6). PR-diff gold is a construct-validity risk — Acc@K measures recovery of full PR edit scope, so incidental edits would make scores reflect patch alignment rather than localization skill — but this affects what the benchmark numbers mean rather than making the derivation circular, since the target is defined once and measured on external artifacts. Overall, the benchmark's central contention has independent content; only a secondary finding is partially definitional, supporting a score of 2.
Assumptions & free parameters
free parameters (3)
- Contrastive temperature tau =
0.05
- Retriever max images per anchor =
2
- AI annotation gate thresholds =
admit relevance +1/+2; category confidence >=0.6; second VLM relevance diff <=1 and category match
assumptions (5)
- domain assumption Merged PR diff edited files/functions are valid gold localization targets
- domain assumption Tree-sitter reliably extracts function spans and maps edited line ranges to pre-existing functions
- domain assumption The four-stage VLM gate produces annotations accurate enough for the 202 AI-assisted instances
- domain assumption Issue-PR links and base-commit repository snapshots are complete and correct
- standard math InfoNCE/CLIP contrastive learning is a valid training objective for the controlled retriever
Cite this review
Pith. "Pith review of MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization." pith.science (2026). https://pith.science/paper/56TB5QA7
@misc{pith2026260715205,
author = {Pith},
title = {Pith review of: MM-IssueLoc: A Controlled Benchmark for Evaluating Visual Evidence in Multimodal Repository-Level Issue Localization},
year = {2026},
howpublished = {\url{https://pith.science/paper/56TB5QA7}},
note = {Machine review of arXiv:2607.15205}
}
read the original abstract
Real repository issues routinely include visual evidence such as screenshots, error dialogs, rendered UI states, and logs, yet repository-level issue localization is evaluated mostly as a text-only task. Existing multimodal SE benchmarks evaluate end-to-end repair, entangling localization with patch synthesis and obscuring whether visual input helped, hurt, or was ignored. We introduce \textbf{MM-IssueLoc}, a controlled benchmark and evaluation protocol for repository-level localization with visual evidence. MM-IssueLoc contains 652 issue-PR instances across 23 languages, with annotations for 7 image categories and 4 relevance levels. It provides file-level and function-level gold labels, paired text-only and with-image evaluation, and VCE-based diagnostics that convert images into structured textual evidence. We evaluate LLM-based and retrieval-based systems, including MM-IssueLoc-VL-Emb as a controlled multimodal retriever. Results show that existing systems remain far from reliable multimodal repository localization: the strongest agent reaches 38.96 file Acc@5 and 22.45 function Acc@10, while the strongest retriever reaches 33.86 function Acc@10. Cross-benchmark comparisons show that high localization scores on text-dominant SWE benchmarks do not transfer cleanly to multimodal issue localization. MM-IssueLoc turns visual evidence into an explicit evaluation variable, enabling future work to test whether systems improve by using visual evidence for localization, rather than by relying on text-only cues or downstream patch-generation effects.
Figures
Figures from the paper (10 more)
Reference graph
Works this paper leans on
-
[1]
The twelfth international conference on learning representations , year=
SWE-bench: Can Language Models Resolve Real-World GitHub Issues? , author=. The twelfth international conference on learning representations , year=
-
[2]
2024 , booktitle=
SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? , author=. 2024 , booktitle=
2024
-
[3]
2025 , booktitle=
Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving , author=. 2025 , booktitle=
2025
-
[4]
arXiv preprint arXiv:2407.01489 , year=
Agentless: Demystifying LLM-based Software Engineering Agents , author=. arXiv preprint arXiv:2407.01489 , year=
-
[5]
Advances in Neural Information Processing Systems , volume=
SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering , author=. Advances in Neural Information Processing Systems , volume=
-
[6]
arXiv preprint arXiv:2511.03690 , year=
The OpenHands Software Agent SDK: A Composable and Extensible Foundation for Production Agents , author=. arXiv preprint arXiv:2511.03690 , year=
-
[7]
Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
LocAgent: Graph-Guided LLM Agents for Code Localization , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=
-
[8]
2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE) , pages=
Issue Localization via LLM-Driven Iterative Code Graph Searching , author=. 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE) , pages=. 2025 , organization=
2025
Show all 25 references
-
[9]
arXiv preprint arXiv:2505.07849 , year=
SweRank: Software Issue Localization with Code Ranking , author=. arXiv preprint arXiv:2505.07849 , year=
-
[10]
arXiv e-prints , pages=
SweRank+: Multilingual, Multi-Turn Code Ranking for Software Issue Localization , author=. arXiv e-prints , pages=
-
[11]
arXiv preprint arXiv:2411.12644 , year=
CodeXEmbed: A Generalist Embedding Model Family for Multiligual and Multi-task Code Retrieval , author=. arXiv preprint arXiv:2411.12644 , year=
-
[12]
arXiv preprint arXiv:2601.04720 , year=
Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking , author=. arXiv preprint arXiv:2601.04720 , year=
-
[13]
Proceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025) , pages=
jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval , author=. Proceedings of the 5th Workshop on Multilingual Representation Learning (MRL 2025) , pages=
2025
-
[14]
arXiv preprint arXiv:2402.01935 , year=
Code Representation Learning At Scale , author=. arXiv preprint arXiv:2402.01935 , year=
-
[15]
Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=
MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems , author=. Findings of the Association for Computational Linguistics: EMNLP 2024 , pages=
2024
-
[16]
arXiv preprint arXiv:2603.17826 , year=
FailureMem: A Failure-Aware Multimodal Framework for Autonomous Software Repair , author=. arXiv preprint arXiv:2603.17826 , year=
-
[17]
arXiv preprint arXiv:2604.12890 , year=
Towards Long-horizon Agentic Multimodal Search , author=. arXiv preprint arXiv:2604.12890 , year=
-
[18]
International conference on machine learning , pages=
Learning Transferable Visual Models From Natural Language Supervision , author=. International conference on machine learning , pages=. 2021 , organization=
2021
-
[19]
arXiv preprint arXiv:1807.03748 , year=
Representation Learning with Contrastive Predictive Coding , author=. arXiv preprint arXiv:1807.03748 , year=
-
[20]
2025 , booktitle=
SWE-smith: Scaling Data for Software Engineering Agents , author=. 2025 , booktitle=
2025
-
[21]
2021 , booktitle=
Contrastive Learning with Hard Negative Samples , author=. 2021 , booktitle=
2021
-
[22]
arXiv preprint arXiv:2603.17829 , year=
CodeScout: An Effective Recipe for Reinforcement Learning of Code Search Agents , author=. arXiv preprint arXiv:2603.17829 , year=
-
[23]
arXiv preprint arXiv:2508.03012 , year=
Tool-Integrated Reinforcement Learning for Repo Deep Search , author=. arXiv preprint arXiv:2508.03012 , year=
-
[24]
2009 , publisher=
The probabilistic relevance framework: BM25 and beyond , author=. 2009 , publisher=
2009
-
[25]
arXiv preprint arXiv:2402.03216 , volume=
Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation , author=. arXiv preprint arXiv:2402.03216 , volume=
Reviewed August 1, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.