Pith. sign in

REVIEW 9 cited by

SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.03859 v1 pith:PUPJYTWH submitted 2024-10-04 cs.CL cs.AIcs.SE

John Yang , Carlos E. Jimenez , Alex L. Zhang , Kilian Lieret , Joyce Yang , Xindi Wu , Ori Press , Niklas Muennighoff
show 5 more authors
Gabriel Synnaeve Karthik R. Narasimhan Diyi Yang Sida I. Wang Ofir Press
This is my paper
classification cs.CLcs.AIcs.SE
keywords swe-benchsystemssoftwarevisualfeaturestaskabilitybugs
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Autonomous systems for software engineering are now capable of fixing bugs and developing features. These systems are commonly evaluated on SWE-bench (Jimenez et al., 2024a), which assesses their ability to solve software issues from GitHub repositories. However, SWE-bench uses only Python repositories, with problem statements presented predominantly as text and lacking visual elements such as images. This limited coverage motivates our inquiry into how existing systems might perform on unrepresented software engineering domains (e.g., front-end, game development, DevOps), which use different programming languages and paradigms. Therefore, we propose SWE-bench Multimodal (SWE-bench M), to evaluate systems on their ability to fix bugs in visual, user-facing JavaScript software. SWE-bench M features 617 task instances collected from 17 JavaScript libraries used for web interface design, diagramming, data visualization, syntax highlighting, and interactive mapping. Each SWE-bench M task instance contains at least one image in its problem statement or unit tests. Our analysis finds that top-performing SWE-bench systems struggle with SWE-bench M, revealing limitations in visual problem-solving and cross-language generalization. Lastly, we show that SWE-agent's flexible language-agnostic features enable it to substantially outperform alternatives on SWE-bench M, resolving 12% of task instances compared to 6% for the next best system.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MT-Web2Code: Benchmarking Coding Agents on Multi-Turn Regional Reconstruction and Localized Modification

    cs.CV 2026-08 conditional novelty 7.0 of 10

    MT-Web2Code is a 102-page, 16-domain multi-turn benchmark that measures how coding agents reconstruct missing web regions and fix localized defects while preserving the surrounding page.

  2. LoopsBench: From Harness Engineering to Loop Engineering in Benchmarking Coding Agent

    cs.SE 2026-07 conditional novelty 7.0 of 10

    LoopsBench exposes coding agents to dependency-DAG tasks with regression obligations; the best configuration, Opus-4.7 with Claude Code and outer continuation, resolves 25.00% of tasks.

  3. Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows

    cs.SE 2025-07 conditional novelty 7.0 of 10

    A 20-developer controlled study found that the coding agent OpenHands improved task completion by 35 percentage points and halved user effort versus GitHub Copilot, while reducing user understanding of outputs.

  4. Active-SWE: Benchmarking Coding Agents for Proactive Bug Fixing without Issue Reports

    cs.SE 2026-08 conditional novelty 6.0 of 10

    Introduces Active-SWE, a benchmark for proactive multi-bug fixing without issue reports, and shows state-of-the-art coding agents resolve few such bugs.

  5. Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering

    cs.SE 2026-06 unverdicted novelty 6.0 of 10

    Coding benchmarks misalign with agentic software engineering because they conflate model and harness, grade against single references, and provide no component-level iteration signals.

  6. Compiling Large Multi-Modal Requirement Documents into Runnable Software Systems: From an Agentic Test-Driven Perspective

    cs.SE 2026-02 conditional novelty 6.0 of 10

    ARC compiles DSL-based requirement documents into runnable web apps by generating interface tests top-down, then using those tests to gate bottom-up code generation, achieving higher GUI pass rates than baseline codin...

  7. How can we assess human-agent interactions? Case studies in software agent design

    cs.AI 2025-10 conditional novelty 6.0 of 10

    PULSE combines sparse human ratings with prediction-powered inference to cut confidence intervals by ~40% and shows LLM choice matters more than scaffolding for user satisfaction.

  8. Adversarial Bug Reports as a Security Risk in Language Model-Based Automated Program Repair

    cs.SE 2025-09 conditional novelty 6.0 of 10

    Adversarial bug reports induced attacker-desired patches in 90% of trials, while the best tested pre-repair filter caught only 47%, exposing a structural weakness in LLM-based automated program repair.

  9. Multilingual Multimodal Software Developer for Code Generation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A 7B vision-language model trained on synthetic diagram-to-code data outperforms several larger open-weight models on a new 10-language UML/flowchart code-generation benchmark.

Pith tools