{"id":"c8fcf87c-05a0-4602-a35e-218e947d1ffe","arxiv_id":"2504.15434","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"OpenAI's Computer-User Agent solves Wordle only 5.36% of the time and its color perception degrades sharply as the game progresses, showing brittleness in a simple multimodal task.","lead":"We tested OpenAI's Computer-User Agent on the New York Times Wordle game and found it solves the puzzle only 5.36% of the time, with color recognition accuracy dropping from 42% on the first guess to 6% by the fifth. The result is a concrete stress test for claims that today's AI agents are close to human-level computer use.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The color-accuracy analysis rests entirely on the agent's self-reported tool-call summaries; without a screenshot-only perception probe, the claimed 'color recognition failure' may be a guess-rationalization artifact.","rationale":"The reader's weakest assumption and my own analysis converge on the same point: the paper's central perceptual claim is inferred exclusively from the agent's self-generated observation strings, which are not a validated proxy for the visual input. The paper itself hedges this in Section 3.5 and even reports control tests where the same model passes simple color recognition, so the failure is specifically claimed to be context-dependent. That context-dependence is precisely what makes the proxy problematic: in the Wordle setting, the tool call is made after the model has formed a guess and accumulated a chain of reasoning, so the observation field can reflect confirmation bias or sycophancy toward its own actions instead of the screenshot's pixel colors. The success-rate and per-word analyses inherit this problem because every accuracy number is derived from the same self-reports. I considered other weaknesses—the post hoc exclusion of the sixth attempt, the lack of released artifacts, the small number of solved games—but none of these threatens the central causal narrative as directly as the unvalidated perception proxy. The proposed test, a screenshot-only re-query with manipulated or intact pixels, would separate low-level visual perception from in-context rationalization and would determine whether the paper documents a real perceptual limitation or an artifact of the measurement instrument. Because the reader already identified this concern and assigned CONDITIONAL, my stress-test does not move the verdict; it reinforces the need for the concrete validation before the stronger claim is accepted.","tokens_in":8453,"tokens_out":4526,"duration_ms":45562,"concrete_test":"Collect the actual screenshots from a sample of the recorded runs (e.g., 50 attempts) and crop the Wordle grid. Present each cropped grid to a fresh CUA session with a neutral, perception-only instruction—no guess word, no game history, no prior tool calls—and compare each report against both the true pixels and the in-run update_wordle_game_state observation. If the perception-only reports match the pixels far better than the in-run reports did, the in-run reports are contaminated by reasoning context and the 42%-to-6% degradation claim does not characterize visual perception. Additionally or alternatively, swap gray and green tiles in a subset of the cropped images; a model that follows the swapped colors has intact color perception.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Every color-accuracy figure (Figs. 2-5, Table 1) is computed from the observation field that CUA supplies to the update_wordle_game_state tool, which the authors' own system prompt instructs the model to use to 'record your thinking and carefully analyze the screenshot.' That field is a free-text reasoning output generated after the model has already committed to a guess, not a calibrated perception readout. A model that sees the gray tiles correctly could still report 'green' because it is rationalizing its guess, is following an optimistic RLHF prior (Section 3.5), or is merely producing a plausible-looking tool call. The authors acknowledge this exact alternative in Section 3.5 ('Alternatively, the model might just be biased toward seeing specific colors even when gray'), and their own control tests in Section 4.1 (simple color grids and Ishihara plates pass) demonstrate that low-level color vision is not broken. If the in-game reports are biased by the guess trajectory, then the headline decline from 42% (attempt 1) to 6% (attempt 5) measures the growth of that bias as the game context accumulates, not a perceptual degradation. The expected-feedback ground truth is also never described: the paper does not state how the target word was obtained or how screenshot timing was aligned to tile states, and the sixth attempt was excluded precisely because of timing issues. Since the success-rate correlation (r=0.694) and the claim that perceptual failure drives task failure are built on these unvalidated self-reports, this is the load-bearing weakness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports an empirical evaluation of OpenAI's Computer-Using Agent (CUA) on the New York Times Wordle game. Over 200 runs across 8 days, the agent solved the puzzle in only 5.36% of cases. The authors analyzed the color feedback the agent reported through the update_wordle_game_state tool and found that self-reported color-recognition accuracy declines from 42% on attempt 1 to 6% on attempt 5, with gray tiles most often misreported as yellow or green and with higher accuracy at edge positions. They report a correlation between per-word color-observation accuracy and success rate (r=0.694, p=0.056), interpret these patterns as context-dependent perceptual failure, and discuss tokenization, RLHF bias, and implications for agentic AI evaluation.","tokens_in":8741,"tokens_out":5078,"duration_ms":44966,"significance":"If the perceptual claim were supported, this would be a valuable, falsifiable counterexample to benchmark-based optimism about agentic AI, backed by a clean 200-run autonomous evaluation, explicit prompts and tool definitions, and control tests for simple color vision. The paper's strengths are the reproducible protocol skeleton, the honest acknowledgment of the alternative bias explanation in Section 3.5, and the large number of runs behind the success-rate statistic. However, the central perceptual-failure conclusion currently rests on a self-report proxy, so the significance is conditional on adding an independent perception probe or reframing the claim as a state-tracking failure.","major_comments":[{"comment":"All color-accuracy figures are computed from the observation field that CUA writes into update_wordle_game_state, which the Appendix A system prompt instructs the model to use to 'record your thinking and carefully analyze the screenshot.' This is a free-text, post-decision summary, not a calibrated perceptual readout, and the authors themselves note in §3.5 that 'the model might just be biased toward seeing specific colors even when gray.' Because the decline from 42% to 6% and the common Gray→Yellow/Gray→Green errors are measured on this self-report proxy, the headline claim of a context-dependent perceptual failure is not established; the evidence as presented supports a claim about the agent's self-reported state tracking, which could be distorted by guess-rationalization or optimistic bias. Please either add a screenshot-only perception probe that queries tile colors before a guess is committed, or reframe the central claim accordingly.","section":"§3.3–3.5, Figs. 2–4"},{"comment":"The manuscript does not state how the expected feedback (the ground-truth G/Y/B string for each guess) was obtained, nor how screenshots were aligned to tile states; the exclusion of the sixth attempt 'because at times the screenshot was not timed appropriately' (fn. 2) shows that this alignment is nontrivial. Without a documented ground-truth procedure, the accuracy numbers in Figs. 2–4 and Table 1 are not reproducible. Please provide the source of the target word, the parsing or alignment rule used to derive expected colors, and the timing criterion used to drop the sixth attempt.","section":"§3.1, Alg. 1, Appendix A"},{"comment":"With only eight word-level data points, r=0.694 and p=0.056, the evidence for the correlation between color-observation accuracy and success rate is suggestive but not significant at the conventional 5% level, and no confidence interval is reported. Calling this 'a strong validation on the hypothesis' overstates the result; please report bootstrapped confidence intervals for the correlation and temper the language to match the statistical power.","section":"§3.6, Fig. 5"}],"minor_comments":[{"comment":"The text refers to 'Fig. 4 (Left)' in §3.4 and §3.5 for content that appears in Fig. 3; please correct the figure cross-references throughout.","section":"§3.4–3.5"},{"comment":"The prompt and tool definition contain formatting artifacts such as 'Ob je ct iv e', 's u b s e q u e n t', 'minLe ng th', and 'm ax Le ng th'; please clean these up.","section":"Appendix A"},{"comment":"Table 1 gives the mean guesses per solved puzzle but not the number of solved puzzles (about 11 of 200) or a measure of spread; please report the sample size and standard deviation or range.","section":"Table 1"},{"comment":"The abstract says 'several hundred runs' while §3.1 reports 200 runs; please use the exact number or describe the additional runs explicitly.","section":"Abstract vs. §3.1"},{"comment":"The accuracy plots would benefit from error bars or confidence intervals; with self-report data, point estimates alone overstate precision.","section":"Figs. 2–4"},{"comment":"The tokenizer patch boundary in Fig. 2 (right) is admittedly 'deduced from ... documentation'; please label it as a hypothesis in the figure caption as well as in the text.","section":"Fig. 2 (right)"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The paper is a candidate for major revision rather than rejection. The 5.36% success-rate measurement is a useful data point, but the paper's central perceptual-failure claim depends on a self-report proxy and an undocumented ground-truth procedure; both are fixable within the manuscript's scope. I would also flag that the title and framing are more promotional than the evidence supports, though that is addressable during revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read: this is a clean empirical data point and a cautionary tale. OpenAI's CUA agent solves NYT Wordle only 5.36% of the time over 200 runs. That number is worth having on the record. The paper also contributes a detailed breakdown of how the agent misreports tile colors as the game progresses, including a gray-to-yellow/green optimism bias. That's a new observation, and it's presented with reasonable rigor: the authors ran 200 sessions over 8 days, report per-position/per-attempt accuracies, and include simple color-grid and Ishihara control tests showing basic color vision is intact.\n\nThe soft spot is structural. Every color-accuracy figure comes from the agent's self-annotated tool calls (the update_wordle_game_state 'observation' field), not from a direct probe of what the model actually sees. The tool prompt even asks the model to 'record your thinking and carefully analyze the screenshot,' so the string is a mixture of perception and reasoning, generated after the model has committed to a guess. A model that sees gray but rationalizes 'green' because it expects progress would produce exactly the pattern reported. The authors acknowledge this in Section 3.5, but the acknowledgment doesn't rescue the headline claim that 'perceptual abilities break down.' The control tests show the model can identify colors in isolation, so the in-game decline is at least partly a guess-rationalization artifact. That's the load-bearing weakness, and it makes the r=0.694 correlation between reported color accuracy and success less impressive: both are outputs of the same reasoning loop.\n\nOther issues are minor in comparison: the sixth attempt was excluded post hoc for timing reasons, no error bars are given, and the p=0.056 correlation is described as 'strong validation' when it would not survive a standard significance threshold. The tokenization boundary explanation is speculative. The paper generalizes to 'current AI systems' after testing one model.\n\nWho is this for? People building agent benchmarks, and anyone tracking frontier model failures. It's a useful case study with a reproducible protocol (the code is based on OpenAI's sample, though no artifact is released). It deserves peer review, because an honest, well-documented failure case is worth publishing even if the interpretation gets revised. The right outcome would be major revision: add a screenshot-only perception probe, report ground-truth alignment, provide error bars, and temper the causal language.","headline":"CUA's 5.36% Wordle success rate is a solid empirical data point; the color-perception story is weakened because the accuracy figures come from the model's own tool-call notes, not a controlled perception probe.","tokens_in":9238,"tokens_out":2290,"would_cite":true,"duration_ms":19804,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A computer-using AI agent solves Wordle only 5.36% of the time, and its reported color recognition collapses from 42% on the first guess to 6% by the fifth.","keywords":["computer-using agent","multimodal agents","Wordle","color perception","GUI interaction","benchmark evaluation","optimistic bias","visual tokenization"],"falsifier":"Run the agent on 100 pre-filled Wordle screenshots that contain no live game state, ask it to call the same annotation tool on each grid, and compare its reported green/yellow/black strings to the true tile colors: if accuracy stays near ceiling across grids that mimic attempts 1 through 5, the decay reported here is tied to acting and chain-of-thought rather than to seeing colors; if accuracy still falls on later-looking grids, the perceptual-failure reading is confirmed.","tokens_in":8248,"feed_emoji":"🟩","tokens_out":7085,"duration_ms":59241,"temperature":0.7,"pith_summary":"The paper asks whether a state-of-the-art computer-using agent can play Wordle, a deceptively simple five-letter word game in which every guess is answered with colored tiles. It reports that the agent solves the puzzle only 5.36% of the time over 200 games, and that the main failure is contextual color perception: the color feedback the agent records for its own guesses is correct 42% of the time on the first attempt and 6% of the time on the fifth. Gray tiles are frequently described as yellow or green, so the agent often announces a win that the game never granted. The authors take this as evidence that strong performance on complex computer-use benchmarks does not imply mastery of simple, multi-step perceptual-reasoning tasks, and that this gap matters for claims about approaching general intelligence.","feed_headline":"AI agent wins Wordle just 5.36% of the time","feed_subtitle":"Color recognition drops from 42% on the first guess to 6% by the fifth, derailing the game.","key_machinery":"The load-bearing mechanism is the agent's self-annotation tool, `update_wordle_game_state`, which forces the model to convert what it sees in each screenshot into a five-character string of green, yellow, and black feedback after every guess. The paper scores these observation strings against the true feedback the game should have produced, turning an invisible perceptual process into a measurable trace. A secondary mechanism is the screenshot tokenizer: each $1024 \\times 768$ image is split into a $2 \\times 2$ grid of $512 \\times 512$ patches, with the inferred boundary cutting through letter position 3 and attempt 3, the same regions where the agent's reported color accuracy is lowest. This tokenization hypothesis, plus the contrast between accurate isolated color tests and inaccurate in-game color reports, carries the paper's argument that the failure is contextual rather than a simple inability to see colors.","core_discovery":"The central finding is a context-dependent color-recognition failure in a production computer-using agent. Over 200 runs across eight days of Wordle puzzles, the agent solved the game in 5.36% of cases, averaging 3.25 guesses on the rare successful solves. When asked to record the color feedback it saw after each guess, the agent's reported accuracy fell from 42% at the first attempt to 6% at the fifth, with edge letter positions more reliable than central ones. The dominant error pattern was gray tiles reported as yellow or green, which the paper reads as an optimistic bias that lets the agent believe it has finished. Word-level success rates tracked color-observation accuracy ($r = 0.694$, $p = 0.056$), making the perceptual errors a direct driver of game failure.","pith_inferences":["A clean extension would freeze a completed Wordle grid in front of the agent and ask for annotations with no gameplay in between; if accuracy stays high, the failure belongs to acting under accumulated context, while a continued decay would point to the image tokenizer itself.","The paper's accuracy numbers inherit whatever bias the agent has in its self-reports, so an independent pixel-level comparison of the screenshots against the agent's recorded feedback would separate 'cannot see the color' from 'reports an optimistic color'.","AGI-style capability claims are usually tested on end-task success; this work suggests a cheaper audit: probe the perceptual state an agent reports before each action, because that trace can expose failures that final outcomes hide.","The same measurement pattern could be applied to other grid-based games or interfaces with deterministic visual feedback, such as Minesweeper, Sudoku, or form validators, to see whether context-dependent color or icon misreading generalizes."],"forward_implications":["High scores on complex web and computer-use benchmarks do not predict success on a simple GUI task like Wordle.","The agent's color-observation accuracy decays across attempts, so perceptual reliability should be measured as a function of context length and accumulated screen content, not just overall task success.","Because gray-to-green hallucinations make the agent declare victory early, agent evaluation should audit the agent's internal state reports, not only whether the final answer is correct.","The position-dependent accuracy pattern predicts that changing how screenshots are cropped or tokenized, for example by zooming into the grid, would change Wordle performance in a measurable way.","Word-level variation with a correlation of $r = 0.694$ between color accuracy and success suggests that the same agent can appear competent or incompetent depending on which puzzle it draws."],"supporting_citations":[{"why":"Defines the Computer-Using Agent under test and reports the strong benchmark scores that the paper contrasts with Wordle performance.","marker":"(OpenAI, 2025b)"},{"why":"Supplies OSWorld, the complex computer-use benchmark whose high CUA score is the paper's baseline for saying the agent is proficient at difficult tasks.","marker":"(Xie et al., 2024)"},{"why":"Supplies WebVoyager, a web-agent benchmark with a high success rate used as evidence of strong general web capability.","marker":"(He et al., 2024)"},{"why":"Supplies WebArena, another complex web-task benchmark used to position the agent's benchmark strength.","marker":"(Zhou et al., 2023)"},{"why":"Documents the publication and context of the Wordle game used as the test environment.","marker":"(Benveniste, 2022)"},{"why":"Supports the paper's attribution of gray-to-yellow and gray-to-green errors to an optimistic bias from reinforcement-learning post-training.","marker":"(Sharma et al., 2023)"},{"why":"Supports the tokenization hypothesis that discretizing continuous images can distort visual perception in multimodal models.","marker":"(Wang et al., 2023)"}],"fun_headline_variants":["AI agent fails Wordle 94.64% of the time","Context kills AI color reading: Wordle win rate 5.36%","Wordle exposes AI's color-blind optimism: 5.36% wins","Optimistic AI misreads gray as green, loses Wordle","AI's Wordle failure: color perception plummets to 6%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire color-accuracy analysis assumes the agent's self-reported observation strings are a faithful record of what it perceives, rather than an optimistic story it tells about its own guesses; the authors explicitly note this alternative in the paper.","fun_headline_variants_meta":{"raw":{"variants":["AI agent fails Wordle 94.64% of the time","Context kills AI color reading: Wordle win rate 5.36%","Wordle exposes AI's color-blind optimism: 5.36% wins","Optimistic AI misreads gray as green, loses Wordle","AI's Wordle failure: color perception plummets to 6%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000207,"raw_usage":{"total_tokens":1351,"prompt_tokens":850,"completion_tokens":501,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":466,"completion_tokens_details":{"reasoning_tokens":405}},"tokens_in":466,"tokens_out":501,"duration_ms":4328,"temperature":1.0,"reasoning_tokens":405,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:26:05.290345+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the agent on 100 pre-filled Wordle screenshots that contain no live game state, ask it to call the same annotation tool on each grid, and compare its reported green/yellow/black strings to the true tile colors: if accuracy stays near ceiling across grids that mimic attempts 1 through 5, the decay reported here is tied to acting and chain-of-thought rather than to seeing colors; if accuracy still falls on later-looking grids, the perceptual-failure reading is confirmed.","supporting_citations":[{"cited_title":"J., Cheng, Z., Shin, D., Lei, F., et al","cited_arxiv_id":null,"evidence_quote":"Supplies OSWorld, the complex computer-use benchmark whose high CUA score is the paper's baseline for saying the agent is proficient at difficult tasks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the publication and context of the Wordle game used as the test environment."}],"review_version":1}