pith. sign in

arxiv: 2507.12367 · v2 · pith:A4PTODPVnew · submitted 2025-07-16 · 💻 cs.SE · cs.AI· cs.PL

GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilities

Diganta Misra , Nizar Islah , Victor May , Brice Rauby , Zihan Wang , Justine Gehring , Antonio Orvieto , Muawiz Chaudhary
show 4 more authors
Eilif B. Muller Irina Rish Samira Ebrahimi Kahou Massimo Caccia
This is my paper
classification 💻 cs.SE cs.AIcs.PL
keywords codegenerationgitchameleonlibrarydatasetevaluationevolutionexecution-based
0
0 comments X
read the original abstract

The rapid evolution of software libraries poses a considerable hurdle for code generation, necessitating continuous adaptation to frequent version updates while preserving backward compatibility. While existing code evolution benchmarks provide valuable insights, they typically lack execution-based evaluation for generating code compliant with specific library versions. To address this, we introduce GitChameleon 2.0, a novel, meticulously curated dataset comprising 328 Python code completion problems, each conditioned on specific library versions and accompanied by executable unit tests. GitChameleon 2.0 rigorously evaluates the capacity of contemporary large language models (LLMs), LLM-powered agents, code assistants, and RAG systems to perform version-conditioned code generation that demonstrates functional accuracy through execution. Our extensive evaluations indicate that state-of-the-art systems encounter significant challenges with this task; enterprise models achieving baseline success rates in the 48-51% range, underscoring the intricacy of the problem. By offering an execution-based benchmark emphasizing the dynamic nature of code libraries, GitChameleon 2.0 enables a clearer understanding of this challenge and helps guide the development of more adaptable and dependable AI code generation methods. We make the dataset and evaluation code publicly available at https://github.com/mrcabbage972/GitChameleonBenchmark.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ScarfBench: A Benchmark for Cross-Framework Application Migration in Enterprise Java

    cs.SE 2026-05 accept novelty 8.0

    ScarfBench supplies 204 cross-framework Java migration tasks where the best agent passes only 15.3% of focused and 12.2% of whole-application tests.

  2. LibEvoBench: Probing Temporal Knowledge Stratification in Code Generation Models

    cs.SE 2026-06 unverdicted novelty 7.0

    LibEvoBench benchmark shows LLMs are version-oblivious on evolving APIs, with documentation helping but version specification not.

  3. ScarfBench: A Benchmark for Cross-Framework Application Migration in Enterprise Java

    cs.SE 2026-05 unverdicted novelty 7.0

    ScarfBench supplies 34 Java applications yielding 204 directed cross-framework refactoring tasks and shows state-of-the-art agents achieve only 15.3% test pass on focused migrations and 12.2% on whole applications.

  4. RepoRescue: An Empirical Study of LLM Agents on Whole-Repository Compatibility Rescue

    cs.SE 2026-07 unverdicted novelty 6.0

    RepoRescue creates a benchmark of 315 repositories and shows LLM agents rescue up to 41.5% with runtime enforcement and 62.7% when combining systems, with hardest cases requiring cross-file changes.

  5. When LLMs Lag Behind: Knowledge Conflicts from Evolving APIs in Code Generation

    cs.SE 2026-04 unverdicted novelty 6.0

    LLMs produce executable code only 42.55% of the time under API evolution without full documentation, improving to 66.36% with structured docs and by 11% more with reasoning strategies, yet outdated patterns persist.