REVIEW 17 cited by
SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
We introduce SWE-Lancer, a benchmark of over 1,400 freelance software engineering tasks from Upwork, valued at \$1 million USD total in real-world payouts. SWE-Lancer encompasses both independent engineering tasks--ranging from \$50 bug fixes to \$32,000 feature implementations--and managerial tasks, where models choose between technical implementation proposals. Independent tasks are graded with end-to-end tests triple-verified by experienced software engineers, while managerial decisions are assessed against the choices of the original hired engineering managers. We evaluate model performance and find that frontier models are still unable to solve the majority of tasks. To facilitate future research, we open-source a unified Docker image and a public evaluation split, SWE-Lancer Diamond (https://github.com/openai/SWELancer-Benchmark). By mapping model performance to monetary value, we hope SWE-Lancer enables greater research into the economic impact of AI model development.
Forward citations
Cited by 17 Pith papers
-
MirrorCode: AI can rebuild entire programs from behavior alone
MirrorCode benchmark shows current AI models achieving up to 56% success reimplementing 25 diverse full programs from behavior alone, including a 16,000-line bioinformatics toolkit.
-
Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity
In a randomized trial of 246 real open-source tasks, experienced developers took 19% longer when AI tools were allowed, despite forecasting 24% faster completion.
-
When Does Restricting a Coding Agent to execute_code Help? A Regime $\times$ Agent-Design Ablation
Code-only tool surfaces beat or match the cheapest tool-rich rival on cost in three of four (regime, agent) cells at matched pass rates; Claude on SWE-bench is the directional exception via failure cost.
-
Can Coding Agents Implement Missed Compiler Optimizations? Evaluating LLM Agents on LLVM Peephole Optimizations
No evaluated coding agent simultaneously matches human experts on behavioral validity and profitability when implementing real missed InstCombine optimizations from LLVM issues.
-
Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents
A reference-free LLM-judge framework measures the consistency, complexity, and policy coverage of conversational-agent benchmarks, validated with perturbations and human ratings.
-
Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering
Coding benchmarks misalign with agentic software engineering because they conflate model and harness, grade against single references, and provide no component-level iteration signals.
-
SWE-Manager: Selecting and Synthesizing Golden Proposals Before Coding
An 8B model trained with reinforcement learning selects the best among competing software-fix proposals and synthesizes a golden proposal, beating GPT-5 on SWE-Lancer Manager and improving end-to-end issue resolution ...
-
Can Vibe Coding Beat Graduate CS Students? An LLM vs. Human Coding Tournament on Market-driven Strategic Planning
Student-coded agents from a 2020 logistics course beat 40 LLM-coded agents in a bidding-and-delivery tournament, with 33 of 40 AI agents losing to a simple baseline.
-
Agentic Program Repair from Test Failures at Scale: A Neuro-symbolic approach with static analysis and test execution feedback
Meta's Engineering Agent solved 42.3% of curated internal test-failure tasks offline and landed 25.5% of generated fixes in production over three months.
-
ResearcherBench: Evaluating Deep AI Research Systems on the Frontiers of Scientific Inquiry
A new benchmark evaluates AI deep research systems on 65 frontier AI questions, finding OpenAI and Gemini lead on rubric coverage while all systems cite accurately but leave much content unsupported.
-
SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories?
A repository-level benchmark shows LLMs and agents currently achieve far smaller, statistically verified speedups than expert human patches on 140 real-world Python performance tasks.
-
Turning the Tide: Repository-based Code Reflection
A new 1,888-case, six-language benchmark and a trained model measure and improve how code LLMs edit and repair code inside multi-file repositories.
-
Establishing Best Practices for Building Rigorous Agentic Benchmarks
Agentic benchmarks frequently mis-grade agents, and the new ABC checklist helps identify and correct such errors in ten popular benchmarks.
-
ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs
A verifiable RL proxy task that asks VLMs to locate a single injected hallucination in a 200-word caption improves visual perception and transfers to math and abstract reasoning benchmarks.
-
Breakpoint: Scalable evaluation of system-level reasoning in LLM code agents
Breakpoint generates code-repair benchmarks by corrupting real GitHub functions and shows that frontier AI coding agents solve easy repairs but fail completely on tasks requiring coordinated, system-wide changes.
-
Taming System Complexity: Demystifying Software Engineering Agents in Diagnosing Linux Kernel Faults
On a new 250-bug Linux kernel benchmark, state-of-the-art LLM agents localize at most 41.6 percent of buggy files at rank 1; the proposed LinuxFL+ framework improves this to 52.4 percent.
-
Conversational AI for Rapid Scientific Prototyping: A Case Study on ESA's ELOPE Competition
One engineer paired with ChatGPT and reached second place in ESA's ELOPE competition in about one week of work; the paper draws best-practice lessons from that experience.
Discussion (0). Continue with ORCID to comment.