{"id":"a0e5e558-95e2-4a53-a0de-9aaf1912e726","arxiv_id":"2412.08859","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ViUniT automatically generates synthetic image-answer unit tests to evaluate visual programs, improving selection, re-prompting, refusal, and reinforcement learning across VQA and image-text matching.","lead":"Visual programs that answer questions about images often give the right answer for the wrong reasons, with faulty code that happens to work. This paper proposes automatically generating synthetic image-and-answer 'unit tests' to grade candidate programs, select better ones, and train models without ground-truth labels.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The unit-test oracle is only ~75% correct (68% on GQA, with low annotator agreement), so the claimed gains may reflect selection on synthetic-test artifacts rather than verified program correctness.","rationale":"The reader's weakest assumption correctly identifies the synthetic unit-test oracle as load-bearing, and Appendix H.1 provides direct evidence that the oracle is imperfect. The proposed test is the natural sensitivity analysis: it separates noise in the scoring signal from signal. If selection is stable under removal of the known-bad tests, the oracle concern does not land and the benchmark results stand as empirical evidence. If selection is unstable, the central claim about improving logical correctness is not established, and the conditional verdict should remain or be strengthened. I am not moving to REJECT because the reported gains are consistent across models and datasets and are partially supported by the 'Most Common Answer' baseline; the concern is about mechanism and robustness, not about fabricated results. I would keep the reader's CONDITIONAL verdict pending the concrete sensitivity check.","tokens_in":40160,"tokens_out":6686,"duration_ms":75309,"concrete_test":"Using the 60 queries from Appendix H.1 (20 per dataset, 5 unit tests each), re-run the best-program-selection protocol from Table 1 with the same 5 candidate programs per query. Score each program twice: once with all 5 unit tests, and once with only the unit tests whose majority-annotator label is correct. Compare the selected program's accuracy on the real validation set between the two scorers, and report the fraction of queries where the selected program changes. If excluding incorrect tests changes selection on more than 10% of queries or reduces the reported accuracy gain materially, the 25% oracle error is load-bearing; if selections are nearly identical, the concern is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Every application in Section 3.4—best-program selection, answer refusal, re-prompting, and RL reward—uses S(p) from Eq. 6, computed against unit tests whose expected answers and images are generated without grounding. Appendix H.1 (Table 13) is the only check of this oracle: 68% correct on GQA, 75% on Winoground, 82% on SugarCREPE, with mean Cohen's κ = 0.58 (0.39 on GQA). Thus roughly one in four unit tests, and one in three on GQA, has a wrong expected answer or a synthetic image that does not support the expected answer; annotators identify 'Missing Object' as the dominant failure. If diffusion models fail systematically on the same spatial or attributive relations the benchmarks test, the scorer will prefer programs that agree with synthetic-image artifacts. A program that is correct on real images can score below one that happens to handle diffusion artifacts, so the 'right for the wrong reasons' problem may be replaced by a different, unmeasured one. The paper does not report whether selection changes when incorrect unit tests are removed, nor whether selected-program accuracy is stable under unit-test noise. The own Limitations section (I.1) concedes no formal correctness guarantee, but the central accuracy claim still depends on the oracle being informative rather than merely noisy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ViUniT, a framework that automatically generates visual unit tests—synthetic images paired with expected answers—to score candidate visual programs produced by an LLM. Unit tests are created by prompting an LLM for image captions and expected answers, selecting a diverse subset via a coverage sampler, and rendering images with text-to-image diffusion models. The unit-test score S(p) (Eq. 6) is then used for four applications: best-program selection, answer refusal, re-prompting, and an unsupervised RL reward. Experiments on GQA, Winoground, and SugarCREPE with CodeLlama-7B and CodeGemma-7B report average gains of 11.4 accuracy points over the base one-program setup, and 7.7 points over gpt-4o-mini, with a claimed 40% reduction in programs that are correct for the wrong reasons.","tokens_in":40491,"tokens_out":7836,"duration_ms":82431,"significance":"If the results hold, the paper offers a practical, unsupervised way to improve visual program synthesis without training labels: unit-test-based selection consistently helps across two program generators and three datasets, and the RL reward outperforms a supervised correctness reward. The paper deserves credit for a broad ablation of unit-test generation strategies (number of tests, sampling methods, image generators, error penalties, aggregators), for openly reporting human evaluations of both unit-test quality and program correctness, and for testing generalization of the RL policy to VQAv2 and OK-VQA. However, the significance is tempered by the paper's own evidence that the unit-test oracle is only about 75% correct (68% on GQA) with low annotator agreement, and by several internal inconsistencies in reported motivation and ablation numbers.","major_comments":[{"comment":"The motivating statistic is internally inconsistent. The abstract and §1 state that when models answer correctly they produce incorrect programs 33% of the time, and that a human evaluation of 100 CodeLlama-7B programs on GQA found only 33% correct. Appendix H.2 (Table 14) reports for the same base setup 77% 'Fully Correct (≤1)', 86% 'Partially Correct (<2)', and only 14% 'Incorrect (≥2)'. These numbers cannot both be true under the rubric described. Please reconcile the two statistics, report the exact rubric (including the overlap between 'Fully Correct' and 'Partially Correct'), and state the sample size per setup in the H.2 evaluation.","section":"§1 and Appendix H.2"},{"comment":"The unit-test oracle is the only scoring signal for all four applications, but its quality is reported as 75% correct overall and 68% on GQA, with mean Cohen's κ = 0.58 (0.39 on GQA). Because the expected answers and synthetic images are generated without grounding, a non-negligible fraction of tests is wrong, and the errors may be systematic (e.g., 'Missing Object' dominates in Fig. 17). The paper should demonstrate that program ranking is robust to oracle noise: for example, report ViUniT accuracy after excluding unit tests flagged as incorrect by annotators, add random label noise to the expected answers and measure accuracy degradation, and show that selected programs in Table 1 do not simply match synthetic-image artifacts. Without such an analysis, the claimed gains could partly reflect selection on faulty tests rather than improved program correctness.","section":"Appendix H.1, Eq. (6), and §3.4"},{"comment":"The program-correctness human evaluation supporting the 'reduces right-for-wrong-reasons by 40%' claim is reported with very low inter-annotator agreement (κ = 0.24 for base and κ = 0.30 for ViUniT; κ_bin = 0.59 and 0.40), and the manuscript does not state how many programs were annotated per setup. The 86% versus 77% difference is small relative to this disagreement. Please provide per-annotator score distributions, the exact rubric, and agreement measures for the correctness categories.","section":"Appendix H.2 and §7"}],"minor_comments":[{"comment":"Section 4.1 states testing on 502 GQA balanced-val examples and training on 1022 balanced-train examples, while Appendix A reverses these labels. Please correct the Appendix.","section":"§4.1 and Appendix A"},{"comment":"The text says a second re-prompting iteration yields a slight drop, but Table 8's averages increase for both models (CodeLlama 48.74→49.01, CodeGemma 47.38→47.49). Please correct the claim or clarify which numbers are meant.","section":"F.3.2, Table 8"},{"comment":"The 'Coverage by Answer' step is only well-defined when the number of distinct answers is at most K; if |Y| > K, Algorithm 1 would select more than K tests. Please state the intended behavior when K is smaller than the number of answers.","section":"§3.2.2, Algorithm 1"},{"comment":"The final configuration uses query-only unit tests, but Figures 1 and 3 and Section 3.2.1 define ψ(q,p) with the program as input. Please clarify in the main text that query-only is the final choice and that implementation-conditioned tests are an ablation.","section":"§5, Figure 6"},{"comment":"The individual scorer h returns negative penalties for compilation and runtime errors, so S(p) in Eq. (6) can be negative; Equation (10) then uses S(p) as a reward. Please specify how negative scores are handled in the RL reward and in the θ threshold.","section":"§3.3, Eq. (5), and §3.4"}],"recommendation":"major_revision","confidential_remarks":"The main accuracy results appear internally consistent and are promising, but the motivation numbers and the oracle-quality robustness need to be addressed before I can recommend acceptance. The issues are fixable with additional analysis and clarifications, and I do not see evidence of a load-bearing error that would force rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Top line: this is a new and useful idea. ViUniT defines a unit test as a synthetic image plus expected answer, executes candidate programs on those images, and uses the pass rate to select, refuse, repair, or train programs. That is genuinely different from PropTest, which checks output properties rather than running the program on new inputs. The four applications are a real contribution, and the reported gains are consistent across two 7B generators and three datasets (+11.4 over base, +7.7 over gpt-4o-mini). The human evaluation of program quality, 77% to 86% fully correct, supports the claim that selection improves underlying code, not just final answers.\n\nThe soft spots are real but not fatal. First, the motivation numbers do not add up. The abstract says 33% of correct answers come from incorrect programs, the intro says only 33% of programs that produced correct answers were correct, and Appendix H.2 reports 77% fully correct in the base setup. That needs to be reconciled. Second, the unit-test oracle itself is only about 75% correct (68% on GQA, with kappa = 0.39). Since every application scores against these tests, systematic errors in synthetic image generation could make the scorer prefer programs that exploit diffusion artifacts. The paper does not report how sensitive selection is to test noise, nor what happens if only the human-verified tests are used. This is the main thing I would ask for. Third, the human evaluation samples are small (300 tests, 100 programs), and no code or generated artifacts are released, so reproduction is hard. The limitations section does concede there is no formal correctness guarantee, which is honest.\n\nI would not block acceptance on the oracle issue: the accuracy gains are measured on real benchmarks, and the program human evaluation is direct evidence that selected programs are more correct on real images. But the paper should either release the test suites or run an oracle-noise robustness analysis. People working on visual programming, code generation, and robustness of reasoning will get value from this. Send it to review. It deserves a serious referee, and the required revisions are concrete.","headline":"A genuinely new way to check visual programs with synthetic unit tests, showing consistent accuracy gains, but the unit-test oracle's ~25% error rate needs a sensitivity analysis before I'd fully trust it.","tokens_in":40965,"tokens_out":2920,"would_cite":true,"duration_ms":30843,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Visual unit tests generated as image/answer pairs catch programs that are right for the wrong reasons, improving accuracy by 11.4%.","keywords":["visual programming","visual unit tests","visual question answering","image-text matching","program selection","reinforcement learning","text-to-image generation","code correctness"],"falsifier":"Take a sample of generated unit tests, have humans correct the expected answers, and re-run best-program selection on the same candidate programs; if the 11.4% accuracy advantage over the baseline disappears or reverses when the oracle is clean, then the reported gains come from the LLM-and-diffusion oracle's particular errors rather than from logical verification.","tokens_in":40022,"feed_emoji":"🧪","tokens_out":9022,"duration_ms":88674,"temperature":0.7,"pith_summary":"Visual programs can return the right answer through broken logic, and the paper's goal is to catch that by testing the program itself, not the answer. ViUniT builds a unit test as a synthetic image paired with the expected answer to the query, with captions and answers written by a language model and images rendered by text-to-image generation. The tests are chosen to cover both answer types and visual inputs, and candidate programs are scored by how many tests they pass. The paper shows this score can select better programs, refuse low-confidence answers, guide re-prompting, and serve as an unsupervised reinforcement-learning reward; the reported effect is an 11.4% accuracy improvement and a 40% reduction in correct-for-wrong-reason programs.","feed_headline":"Visual unit tests lift 7B models past gpt-4o-mini by 7.7 points","feed_subtitle":"Synthetic image-and-answer pairs verify program logic and lift frozen-LLM accuracy 11.4% across VQA and image-text tasks.","key_machinery":"The load-bearing object is the visual unit test suite $\\mathcal{T} = \\{(M(c_i), y_i)\\}$, where $c_i$ is an LLM-written caption, $y_i$ is the expected answer the LLM attaches to it, and $M$ is a text-to-image diffusion model that renders the caption. The coverage sampler $\\sigma$ keeps at least one test for every answer value in the candidate pool, then greedily adds captions whose embeddings are farthest from the already chosen ones, so the suite spans both output types and visual inputs. The scorer $S(p)$ averages per-test outcomes $h(\\hat{y}_i, y_i)$, with penalties for runtime and compilation errors, and this averaged score is the single mechanism behind program selection, refusal, re-prompting, and reward computation.","core_discovery":"The paper claims that the dominant failure mode of visual programming — correct answers produced by incorrect programs — can be detected by executing candidate programs on synthetic tests before they ever see the real image. A unit test is a generated image plus the expected answer to the query; the tests are produced without any human labels by having a language model write captions and answers, sampling for coverage, and rendering images with a diffusion model. Running each candidate program on these tests produces a score, and the highest-scoring program is the one the paper argues is most likely to be right for the right reasons. Across three benchmarks, this selection rule improves frozen-LLM accuracy by 11.4%, lets 7B open-source models outperform gpt-4o-mini by 7.7 points, and cuts the fraction of programs that are correct for the wrong reasons by 40%.","pith_inferences":["Because the unit-test oracle is only 75% accurate by the paper's own human evaluation, the reported gains likely understate what a cleaner test generator could achieve; adding a verification step that filters captions and expected answers through a vision-language model or a second LLM is a direct extension.","Because the reward signal needs no ground-truth labels, the same recipe could transfer to other settings where programs operate on images — robotics instructions, document layout reasoning, or GUI automation — as long as tests can be rendered and expected answers checked automatically.","The paper's finding that implementation-independent unit tests work better in high-resource settings mirrors classical software-testing doctrine; in practice, users should avoid over-fitting tests to one candidate program.","Diffusion artifacts, especially spatial relations, are the most plausible failure source; the layout-conditioned diffusion advantage for spatial questions suggests that controlling object positions in generated test images is the direction to harden the oracle."],"forward_implications":["Frozen-LLM accuracy on GQA, Winoground, and SugarCREPE rises by 11.4 percentage points over the single-program baseline when the best of five programs is chosen by unit-test scores.","The same scoring signal, used as an unsupervised RL reward, outperforms a supervised correctness reward by 1.3 points on average and lowers program error rates for both 7B generators.","Re-prompting with unit-test feedback beats re-prompting with error traces by 7.5 points on average, and threshold-based refusal reaches up to 0.8 F1 in declining to answer with weak programs.","Human evaluation of selected GQA programs shows fully-correct programs rising from 77% to 86% and completely incorrect programs falling from 14% to 5%.","Coverage sampling by answer-then-input and query-only unit-test generation are the configurations the paper identifies as most effective, giving a concrete recipe for future visual-programming systems."],"supporting_citations":[{"why":"Supplies the ImagePatch programming interface and execution engine that every visual program in the paper is written against and executed on.","marker":"[49]"},{"why":"Defines the visual-program self-training setup and the supervised correctness reward that ViUniT's unsupervised unit-test reward is compared against.","marker":"[23]"},{"why":"The prior property-testing baseline for visual programs; ViUniT's unit tests are positioned as verifying logical correctness beyond output-type checks.","marker":"[25]"},{"why":"One of the two 7B program generators whose single-program baseline is improved by unit-test selection.","marker":"[41]"},{"why":"The instruction-tuned LLM used to write unit-test captions and expected answers, which is the core of test generation.","marker":"[8]"},{"why":"One of the text-to-image diffusion models used to render selected captions into test images.","marker":"[40]"},{"why":"A layout-conditioned diffusion model that improves unit-test images when spatial relations matter, supporting the spatial-accuracy analysis.","marker":"[30]"},{"why":"The GQA benchmark provides the VQA data for the central accuracy, refusal, re-prompting, and reward experiments.","marker":"[20]"},{"why":"SugarCREPE is one of the two image-text-matching benchmarks used in the main comparisons.","marker":"[17]"},{"why":"Winoground is the other image-text-matching benchmark, and the full paired test set is used for evaluation.","marker":"[52]"}],"fun_headline_variants":["Unit tests let 7B models beat gpt-4o-mini by 7.7","Synthetic unit tests slash wrong-reason programs by 40%","Visual unit tests lift accuracy 11.4% on VQA and image-text","Program verification via synthetic images boosts reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a synthetic image with an LLM-written expected answer is a reliable oracle for whether a program will answer correctly on real images; the paper's own human evaluation puts unit-test accuracy at only 75%, so roughly a quarter of the scoring signal is wrong.","fun_headline_variants_meta":{"raw":{"variants":["Unit tests let 7B models beat gpt-4o-mini by 7.7","Synthetic unit tests slash wrong-reason programs by 40%","Visual unit tests lift accuracy 11.4% on VQA and image-text","Program verification via synthetic images boosts reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000594,"raw_usage":{"total_tokens":2815,"prompt_tokens":1010,"completion_tokens":1805,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":626,"completion_tokens_details":{"reasoning_tokens":1726}},"tokens_in":626,"tokens_out":1805,"duration_ms":14129,"temperature":1.0,"reasoning_tokens":1726,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:28:52.651859+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of generated unit tests, have humans correct the expected answers, and re-run best-program selection on the same candidate programs; if the 11.4% accuracy advantage over the baseline disappears or reverses when the oracle is clean, then the reported gains come from the LLM-and-diffusion oracle's particular errors rather than from logical verification.","supporting_citations":[{"cited_title":"Vipergpt: Vi- sual inference via python execution for reasoning","cited_arxiv_id":null,"evidence_quote":"Supplies the ImagePatch programming interface and execution engine that every visual program in the paper is written against and executed on."},{"cited_title":"Winoground: Probing vision and language models for visio- linguistic compositionality","cited_arxiv_id":null,"evidence_quote":"Winoground is the other image-text-matching benchmark, and the full paired test set is used for evaluation."}],"review_version":1}