Baba Is AI: Break the Rules to Beat the Benchmark

Andrei Barbu; Christopher J. Cueva; Ignacio Cases; Meagan Jens; Michelangelo Naim; Nathan Cloos; Yen-Ling Kuo

arxiv: 2407.13729 · v2 · pith:4AJCLYU6 · submitted 2024-07-18 · cs.CL

Baba Is AI: Break the Rules to Beat the Benchmark

Nathan Cloos , Meagan Jens , Michelangelo Naim , Yen-Ling Kuo , Ignacio Cases , Andrei Barbu , Christopher J. Cueva This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 reserved pith:4AJCLYU6 record.json open to challenge →

classification cs.CL

keywords rulesgamebababenchmarkgemini-1abilitiesagentbeat

0 comments

read the original abstract

Humans solve problems by following existing rules and procedures, and also by leaps of creativity to redefine those rules and objectives. To probe these abilities, we developed a new benchmark based on the game Baba Is You where an agent manipulates both objects in the environment and rules, represented by movable tiles with words written on them, to reach a specified goal and win the game. We test three state-of-the-art multi-modal large language models (OpenAI GPT-4o, Google Gemini-1.5-Pro and Gemini-1.5-Flash) and find that they fail dramatically when generalization requires that the rules of the game must be manipulated and combined.

This paper has not been read by Pith yet.

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application
cs.CL 2026-06 unverdicted novelty 5.0

This survey categorizes agentic environments for LLMs by eight attributes and domains, introduces symbolic and neural synthesis paradigms with evaluation, and outlines four agent evolution pathways plus three environm...