Pith. sign in

REVIEW 3 major objections 5 minor 116 references

ViUniT: Visual Unit Tests for More Robust Visual Programming

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Visual unit tests generated as image/answer pairs catch programs that are right for the wrong reasons, improving accuracy by 11.4%.

desk verdict A genuinely new way to check visual programs with synthetic unit tests, showing consistent accuracy gains, but the unit-test oracle's ~25% error rate needs a sensitivity analysis before I'd fully trust it. read the letter →

arxiv 2412.08859 v1 pith:EVZ6QCZX submitted 2024-12-12 cs.CV

Artemis Panagopoulou , Honglu Zhou , Silvio Savarese , Caiming Xiong , Chris Callison-Burch , Mark Yatskar , Juan Carlos Niebles This is my paper · ORCID
classification cs.CV
keywords visualprogrammingunittestsquestionansweringimage-textmatchingprogramselectionreinforcementlearningtext-to-imagegenerationcodecorrectness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Visual programs can return the right answer through broken logic, and the paper's goal is to catch that by testing the program itself, not the answer. ViUniT builds a unit test as a synthetic image paired with the expected answer to the query, with captions and answers written by a language model and images rendered by text-to-image generation. The tests are chosen to cover both answer types and visual inputs, and candidate programs are scored by how many tests they pass. The paper shows this score can select better programs, refuse low-confidence answers, guide re-prompting, and serve as an unsupervised reinforcement-learning reward; the reported effect is an 11.4% accuracy improvement and a 40% reduction in correct-for-wrong-reason programs.

What carries the argument

The load-bearing object is the visual unit test suite $\mathcal{T} = \{(M(c_i), y_i)\}$, where $c_i$ is an LLM-written caption, $y_i$ is the expected answer the LLM attaches to it, and $M$ is a text-to-image diffusion model that renders the caption. The coverage sampler $\sigma$ keeps at least one test for every answer value in the candidate pool, then greedily adds captions whose embeddings are farthest from the already chosen ones, so the suite spans both output types and visual inputs. The scorer $S(p)$ averages per-test outcomes $h(\hat{y}_i, y_i)$, with penalties for runtime and compilation errors, and this averaged score is the single mechanism behind program selection, refusal, re-prompting, and reward computation.

What would settle it

Take a sample of generated unit tests, have humans correct the expected answers, and re-run best-program selection on the same candidate programs; if the 11.4% accuracy advantage over the baseline disappears or reverses when the oracle is clean, then the reported gains come from the LLM-and-diffusion oracle's particular errors rather than from logical verification.

Watch

Extended reading notes

Core claim

The paper claims that the dominant failure mode of visual programming — correct answers produced by incorrect programs — can be detected by executing candidate programs on synthetic tests before they ever see the real image. A unit test is a generated image plus the expected answer to the query; the tests are produced without any human labels by having a language model write captions and answers, sampling for coverage, and rendering images with a diffusion model. Running each candidate program on these tests produces a score, and the highest-scoring program is the one the paper argues is most likely to be right for the right reasons. Across three benchmarks, this selection rule improves frozen-LLM accuracy by 11.4%, lets 7B open-source models outperform gpt-4o-mini by 7.7 points, and cuts the fraction of programs that are correct for the wrong reasons by 40%.

Load-bearing premise

The load-bearing premise is that a synthetic image with an LLM-written expected answer is a reliable oracle for whether a program will answer correctly on real images; the paper's own human evaluation puts unit-test accuracy at only 75%, so roughly a quarter of the scoring signal is wrong.

Editorial extensions

If this is right

  • Frozen-LLM accuracy on GQA, Winoground, and SugarCREPE rises by 11.4 percentage points over the single-program baseline when the best of five programs is chosen by unit-test scores.
  • The same scoring signal, used as an unsupervised RL reward, outperforms a supervised correctness reward by 1.3 points on average and lowers program error rates for both 7B generators.
  • Re-prompting with unit-test feedback beats re-prompting with error traces by 7.5 points on average, and threshold-based refusal reaches up to 0.8 F1 in declining to answer with weak programs.
  • Human evaluation of selected GQA programs shows fully-correct programs rising from 77% to 86% and completely incorrect programs falling from 14% to 5%.
  • Coverage sampling by answer-then-input and query-only unit-test generation are the configurations the paper identifies as most effective, giving a concrete recipe for future visual-programming systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the unit-test oracle is only 75% accurate by the paper's own human evaluation, the reported gains likely understate what a cleaner test generator could achieve; adding a verification step that filters captions and expected answers through a vision-language model or a second LLM is a direct extension.
  • Because the reward signal needs no ground-truth labels, the same recipe could transfer to other settings where programs operate on images — robotics instructions, document layout reasoning, or GUI automation — as long as tests can be rendered and expected answers checked automatically.
  • The paper's finding that implementation-independent unit tests work better in high-resource settings mirrors classical software-testing doctrine; in practice, users should avoid over-fitting tests to one candidate program.
  • Diffusion artifacts, especially spatial relations, are the most plausible failure source; the layout-conditioned diffusion advantage for spatial questions suggests that controlling object positions in generated test images is the direction to harden the oracle.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ViUniT, a framework that automatically generates visual unit tests—synthetic images paired with expected answers—to score candidate visual programs produced by an LLM. Unit tests are created by prompting an LLM for image captions and expected answers, selecting a diverse subset via a coverage sampler, and rendering images with text-to-image diffusion models. The unit-test score S(p) (Eq. 6) is then used for four applications: best-program selection, answer refusal, re-prompting, and an unsupervised RL reward. Experiments on GQA, Winoground, and SugarCREPE with CodeLlama-7B and CodeGemma-7B report average gains of 11.4 accuracy points over the base one-program setup, and 7.7 points over gpt-4o-mini, with a claimed 40% reduction in programs that are correct for the wrong reasons.

Significance. If the results hold, the paper offers a practical, unsupervised way to improve visual program synthesis without training labels: unit-test-based selection consistently helps across two program generators and three datasets, and the RL reward outperforms a supervised correctness reward. The paper deserves credit for a broad ablation of unit-test generation strategies (number of tests, sampling methods, image generators, error penalties, aggregators), for openly reporting human evaluations of both unit-test quality and program correctness, and for testing generalization of the RL policy to VQAv2 and OK-VQA. However, the significance is tempered by the paper's own evidence that the unit-test oracle is only about 75% correct (68% on GQA) with low annotator agreement, and by several internal inconsistencies in reported motivation and ablation numbers.

major comments (3)
  1. [§1 and Appendix H.2] The motivating statistic is internally inconsistent. The abstract and §1 state that when models answer correctly they produce incorrect programs 33% of the time, and that a human evaluation of 100 CodeLlama-7B programs on GQA found only 33% correct. Appendix H.2 (Table 14) reports for the same base setup 77% 'Fully Correct (≤1)', 86% 'Partially Correct (<2)', and only 14% 'Incorrect (≥2)'. These numbers cannot both be true under the rubric described. Please reconcile the two statistics, report the exact rubric (including the overlap between 'Fully Correct' and 'Partially Correct'), and state the sample size per setup in the H.2 evaluation.
  2. [Appendix H.1, Eq. (6), and §3.4] The unit-test oracle is the only scoring signal for all four applications, but its quality is reported as 75% correct overall and 68% on GQA, with mean Cohen's κ = 0.58 (0.39 on GQA). Because the expected answers and synthetic images are generated without grounding, a non-negligible fraction of tests is wrong, and the errors may be systematic (e.g., 'Missing Object' dominates in Fig. 17). The paper should demonstrate that program ranking is robust to oracle noise: for example, report ViUniT accuracy after excluding unit tests flagged as incorrect by annotators, add random label noise to the expected answers and measure accuracy degradation, and show that selected programs in Table 1 do not simply match synthetic-image artifacts. Without such an analysis, the claimed gains could partly reflect selection on faulty tests rather than improved program correctness.
  3. [Appendix H.2 and §7] The program-correctness human evaluation supporting the 'reduces right-for-wrong-reasons by 40%' claim is reported with very low inter-annotator agreement (κ = 0.24 for base and κ = 0.30 for ViUniT; κ_bin = 0.59 and 0.40), and the manuscript does not state how many programs were annotated per setup. The 86% versus 77% difference is small relative to this disagreement. Please provide per-annotator score distributions, the exact rubric, and agreement measures for the correctness categories.
minor comments (5)
  1. [§4.1 and Appendix A] Section 4.1 states testing on 502 GQA balanced-val examples and training on 1022 balanced-train examples, while Appendix A reverses these labels. Please correct the Appendix.
  2. [F.3.2, Table 8] The text says a second re-prompting iteration yields a slight drop, but Table 8's averages increase for both models (CodeLlama 48.74→49.01, CodeGemma 47.38→47.49). Please correct the claim or clarify which numbers are meant.
  3. [§3.2.2, Algorithm 1] The 'Coverage by Answer' step is only well-defined when the number of distinct answers is at most K; if |Y| > K, Algorithm 1 would select more than K tests. Please state the intended behavior when K is smaller than the number of answers.
  4. [§5, Figure 6] The final configuration uses query-only unit tests, but Figures 1 and 3 and Section 3.2.1 define ψ(q,p) with the program as input. Please clarify in the main text that query-only is the final choice and that implementation-conditioned tests are an ablation.
  5. [§3.3, Eq. (5), and §3.4] The individual scorer h returns negative penalties for compilation and runtime errors, so S(p) in Eq. (6) can be negative; Equation (10) then uses S(p) as a reward. Please specify how negative scores are handled in the RL reward and in the θ threshold.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: ViUniT's unit tests are an independent proxy, not a fitted target; the 75%-correct oracle caveat is a validity risk, not a derivation-level circularity.

full rationale

I found no equation-level circularity. The final unit-test configuration is query-only (Section 5, Figure 6: "disregarding implementation details proves significantly more effective"), so the tests are not generated from the program being scored. Expected answers come from a separate LLM (Meta-Llama-3-8B-Instruct) via caption/answer generation (Sections 3.2.1, D.1), and images come from diffusion models (Section 3.2.3); no constant is fitted to benchmark labels. Equations 5-6 define S(p) as a unit-test proxy, and Equation 6 is used to approximate Equation 1; the reported accuracy is measured against held-out ground truth, not against S(p), so selection does not make the target equal to the proxy. The RL reward (Equation 10) is likewise an optimization signal evaluated against external labels. Hyperparameters (K, theta, eps, sampling method, image generator) are tuned on small development subsets (Section 5), not on test labels. The only self-citations ([28], [37]) appear in related-work and API-model lists and are not load-bearing for the central claim. Appendix H.1/Table 13 shows only 75% of unit tests are correct, and Limitations I.1 concedes the method "does not provide a formal guarantee of logical correctness"; these are validity/robustness concerns about the oracle, not evidence that a prediction reduces by construction to a fitted input. I therefore score at most 2 for minor non-load-bearing self-citation, with no circular steps.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The framework does not introduce new physical entities or fitted theoretical constants. Its load-bearing inputs are the LLM-generated captions and expected answers, the diffusion-generated images, and the hand-set hyperparameters; the paper's own evaluations show these inputs are only partially reliable.

free parameters (4)
  • error penalty epsilon = 0.1
    Runtime and compilation error penalties in the unit test scorer (Equation 5), chosen by hand and studied in Appendix E.3.
  • pass threshold theta = 0.7 for refusal/reprompting, 0.8 for RL
    Threshold on unit test score for refusal and re-prompting (Section 3.4), swept in Figure 9; affects reported results.
  • number of unit tests K = 5
    Selected after the Section 5 study on a smaller development set; main tables use K=5.
  • number of candidate programs N = 5
    Main tables sample 5 programs per query; more programs improve accuracy (Table 7).
assumptions (5)
  • domain assumption Synthetic images generated by diffusion models faithfully reflect the caption and the expected answer for the query
    The entire unit-test signal rests on this; human evaluation in Appendix H.1 measures only 75% accuracy, so the assumption holds only partially.
  • domain assumption LLM-generated captions and expected answers provide a correct oracle for unit tests
    Meta-Llama-3-8B-Instruct generates candidate captions and answers (Section 3.2.1); correctness is validated only on a 300-sample human eval.
  • ad hoc to paper Coverage of caption embeddings approximates coverage of visual inputs
    The sampling criterion (Equations 3-4) maximizes pairwise caption embedding distances instead of image distances, assuming textual diversity implies visual diversity.
  • ad hoc to paper Goodhart's-law regret bound applies to the proxy unit-test reward
    Section 3.4 invokes prior work [22] to claim theoretical guarantees when proxy reward declines; no proof is given for this setting.
  • domain assumption The ImagePatch execution engine behaves on synthetic images as on real images
    Programs are executed on generated images; detector or VQA failures on synthetic images would distort unit-test outcomes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ViUniT: Visual Unit Tests for More Robust Visual Programming." pith.science (2026). https://pith.science/paper/EVZ6QCZX

@misc{pith2026241208859,
  author       = {Pith},
  title        = {Pith review of: ViUniT: Visual Unit Tests for More Robust Visual Programming},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EVZ6QCZX}},
  note         = {Machine review of arXiv:2412.08859}
}
read the original abstract

Programming based approaches to reasoning tasks have substantially expanded the types of questions models can answer about visual scenes. Yet on benchmark visual reasoning data, when models answer correctly, they produce incorrect programs 33% of the time. These models are often right for the wrong reasons and risk unexpected failures on new data. Unit tests play a foundational role in ensuring code correctness and could be used to repair such failures. We propose Visual Unit Testing (ViUniT), a framework to improve the reliability of visual programs by automatically generating unit tests. In our framework, a unit test is represented as a novel image and answer pair meant to verify the logical correctness of a program produced for a given query. Our method leverages a language model to create unit tests in the form of image descriptions and expected answers and image synthesis to produce corresponding images. We conduct a comprehensive analysis of what constitutes an effective visual unit test suite, exploring unit test generation, sampling strategies, image generation methods, and varying the number of programs and unit tests. Additionally, we introduce four applications of visual unit tests: best program selection, answer refusal, re-prompting, and unsupervised reward formulations for reinforcement learning. Experiments with two models across three datasets in visual question answering and image-text matching demonstrate that ViUniT improves model performance by 11.4%. Notably, it enables 7B open-source models to outperform gpt-4o-mini by an average of 7.7% and reduces the occurrence of programs that are correct for the wrong reasons by 40%.

Figures

Figures reproduced from arXiv: 2412.08859 by the authors.

Figure 1
Figure 1. V uniT Framework Overview. Given a query q about an image, the unit test generator ψ generates a set Tcand = ψ(q, p) of M candidate pairs ti = (ci, yi), each consisting of an image caption ci and an expected answer yi (Section 3.2.1). The coverage sampler σ then subsamples K pairs from Tcand, forming the subset TK (Section 3.2.2). These captions are passed to an image generator M to create the corresponding images v… view at source ↗
Figure 2
Figure 2. Visual Unit Testing Utilization Strategies (Section [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Unit Test Examples generated by V uniT with expected answers. This approach reduces computational overhead during the preliminary stage of unit test coverage sampling, after which we generate images only for those tests that are included in the final unit test suite T . In partic￾ular, we first generate a superset of M candidate unit tests using the unit test generator ψ, which is implemented as an auto-regressive l… view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Comparison of Unit Tests Generated by Different Methods [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Accuracy across varying unit test and program counts. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 9
Figure 9. Figure 9: Refusal evaluation at different passing thresholds. [PITH_FULL_IMAGE:figures/full_fig_p007_9.png]
Figure 7
Figure 7. Figure 7: Sampling method comparison at 5 programs. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 10
Figure 10. Figure 10: Effect of sampling methods on performance across [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]
Figure 11
Figure 11. Figure 11: Performance of sampling methods across question cate [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: Effect of diffusion model on performance across varying [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: Performance of different diffusion models across ques [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 15
Figure 15. Figure 15: Effect of aggregator function on accuracy. [PITH_FULL_IMAGE:figures/full_fig_p016_15.png]
Figure 16
Figure 16. Figure 16: Accuracy and Program Pass Rate for different thereshold [PITH_FULL_IMAGE:figures/full_fig_p016_16.png]
Figure 18
Figure 18. Figure 18: Human Evaluation of Program Quality. I. Limitations and Social Ethics Impact I.1. Limitations While V uniT provides significant advancements in the logical correctness and robustness of visual programs, our framework has several limitations that present opportunities …
Figure 19
Figure 19. Figure 19: Program Selection Example 20 [PITH_FULL_IMAGE:figures/full_fig_p020_19.png]
Figure 20
Figure 20. Figure 20: Program Selection Example 21 [PITH_FULL_IMAGE:figures/full_fig_p021_20.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

116 extracted references · 66 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 2, 6

    arXiv 2023

  2. [2]

    A3test: Assertion-augmented automated test case gen- eration

    Saranya Alagarsamy, Chakkrit Tantithamthavorn, and Aldeida Aleti. A3test: Assertion-augmented automated test case gen- eration. Information and Software Technology, 176:107565,

  3. [3]

    Palm 2 technical report

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin John- son, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023. 2

    arXiv 2023

  4. [4]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. 2

    1901

  5. [5]

    Codet: Code gen- eration with generated tests

    Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code gen- eration with generated tests. In The Eleventh International Conference on Learning Representations, 2023. 1, 3

    2023

  6. [6]

    InstructBLIP: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. 3

    2023

  7. [7]

    StepCoder: Improving code generation with reinforcement learning from compiler feedback

    Shihan Dou, Yan Liu, Haoxiang Jia, Enyu Zhou, Limao Xiong, Junjie Shan, Caishuang Huang, Xiao Wang, Xiao- ran Fan, Zhiheng Xi, Yuhao Zhou, Tao Ji, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. StepCoder: Improving code generation with reinforcement learning from compiler feedback. In Proceedings of the 62nd Annual Meeting of the Association for Comput...

    2024

  8. [8]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

Show all 116 references
  1. [9]

    Clova: A closed-loop visual assistant with tool usage and update

    Zhi Gao, Yuntao Du, Xintong Zhang, Xiaojian Ma, Wenjuan Han, Song-Chun Zhu, and Qing Li. Clova: A closed-loop visual assistant with tool usage and update. Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2

    2024

  2. [10]

    Recursive visual programming

    Jiaxin Ge, Sanjay Subramanian, Baifeng Shi, Roei Herzig, and Trevor Darrell. Recursive visual programming. In Euro- pean Conference on Computer Vision, pages 1–18. Springer,

  3. [11]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017. 17

    2017

  4. [12]

    An initial investigation of chatgpt unit test generation capability

    Vitor Guilherme and Auri Vincenzi. An initial investigation of chatgpt unit test generation capability. In Proceedings of the 8th Brazilian Symposium on Systematic and Automated Software Testing, pages 15–24, 2023. 1, 3

    2023

  5. [13]

    Deepseek-coder: When the large language model meets programming–the rise of code intelligence

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196, 2024. 2

    2024 arXiv

  6. [14]

    Visual program- ming: Compositional visual reasoning without training

    Tanmay Gupta and Aniruddha Kembhavi. Visual program- ming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 14953–14962, 2023. 1, 2, 6

    2023

  7. [15]

    Image translation as diffusion visual programmers

    Cheng Han, James Chenhao Liang, Qifan Wang, MAJID RABBANI, Sohail Dianat, Raghuveer Rao, Ying Nian Wu, and Dongfang Liu. Image translation as diffusion visual programmers. In The Twelfth International Conference on Learning Representations. 1, 2

  8. [16]

    Cogagent: A visual language model for gui agents

    Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wen- meng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 14281–14290,

  9. [17]

    Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality

    Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kem- bhavi, and Ranjay Krishna. Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality. Advances in neural information processing systems, 36, 2024. 6, 12

    2024

  10. [18]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. 6

    2022

  11. [19]

    Visual program distillation: Distilling tools and programmatic reasoning into vision-language models

    Yushi Hu, Otilia Stretcu, Chun-Ta Lu, Krishnamurthy Viswanathan, Kenji Hata, Enming Luo, Ranjay Krishna, and Ariel Fuxman. Visual program distillation: Distilling tools and programmatic reasoning into vision-language models. In Proceedings of the IEEE/CVF Conference on Compute...

    2024

  12. [20]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 6700–6709, 2019. 1, 6, 12

    2019

  13. [21]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Flo- rian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825,

    arXiv

  14. [22]

    Goodhart’s law in reinforcement learning

    Jacek Karwowski, Oliver Hayman, Xingjian Bai, Klaus Kiendlhofer, Charlie Griffin, and Joar Max Viktor Skalse. Goodhart’s law in reinforcement learning. In The Twelfth International Conference on Learning Representations. 5

  15. [23]

    Self-training large language models for improved visual program synthesis with visual reinforce- ment

    Zaid Khan, Vijay Kumar BG, Samuel Schulter, Yun Fu, and Manmohan Chandraker. Self-training large language models for improved visual program synthesis with visual reinforce- ment. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 143...

  16. [24]

    Unit Testing Principles, Practices, and Patterns

    Vladimir Khorikov. Unit Testing Principles, Practices, and Patterns. Simon and Schuster, 2020. 4

    2020

  17. [25]

    PropTest: Automatic property testing for improved visual programming

    Jaywon Koo, Ziyan Yang, Paola Cascante-Bonilla, Baishakhi Ray, and Vicente Ordonez. PropTest: Automatic property testing for improved visual programming. In Findings of the Association for Computational Linguistics: EMNLP 2024 , pages 8241–8256, Miami, Florida, USA, 2024. Asso...

    2024

  18. [26]

    Efficient memory management for large lan- guage model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles,...

    2023

  19. [27]

    CodeRL: Mastering code gen- eration through pretrained models and deep reinforcement learning

    Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Hoi. CodeRL: Mastering code gen- eration through pretrained models and deep reinforcement learning. In Advances in Neural Information Processing Sys- tems, 2022. 2, 3

    2022

  20. [28]

    Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR,

  21. [29]

    Synthesize step-by-step: Tools templates and llms as data generators for reasoning-based chart vqa

    Zhuowan Li, Bhavan Jasani, Peng Tang, and Shabnam Ghadar. Synthesize step-by-step: Tools templates and llms as data generators for reasoning-based chart vqa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13613–13623, 2024. 1, 2

    2024

  22. [30]

    LLM- grounded diffusion: Enhancing prompt understanding of text- to-image diffusion models with large language models.Trans- actions on Machine Learning Research, 2024

    Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. LLM- grounded diffusion: Enhancing prompt understanding of text- to-image diffusion models with large language models.Trans- actions on Machine Learning Research, 2024. Featured Cer- tification. 5

    2024

  23. [31]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 3

    2024

  24. [32]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Con- ference on Computer Vision. Springer, 2024. 6, 12

    2024

  25. [33]

    The flan collection: Designing data and methods for effective instruction tuning

    Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. In International Conference on Machine Learning, pages 22631–22648. PM...

    2023

  26. [34]

    Chameleon: Plug-and-play compositional reasoning with large language models

    Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models. Advances in Neural Information Processing Systems, 36, 2024. 2

    2024

  27. [35]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019. 17

    2019

  28. [36]

    Xgen-7b technical report

    Erik Nijkamp, Tian Xie, Hiroaki Hayashi, Bo Pang, Congying Xia, Chen Xing, Jesse Vig, Semih Yavuz, Philippe Laban, Ben Krause, et al. Xgen-7b technical report. arXiv preprint arXiv:2309.03450, 2023. 2

    2023 arXiv

  29. [37]

    X-instructblip: A framework for aligning x-modal instruction-aware representations to llms and emergent cross-modal reasoning

    Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. X-instructblip: A framework for aligning x-modal instruction-aware representations to llms and emergent cross-modal reasoning. In Proceedi...

    2024

  30. [38]

    SDXL: Improving latent diffusion models for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M¨uller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. In The Twelfth International Conference on Learning Representations, 2024. 5

    2024

  31. [39]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  32. [40]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, 2022. 3, 5

    2022

  33. [41]

    Code llama: Open foundation models for code

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J´er´emy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023. 1, 2, 6

    2023 arXiv

  34. [42]

    Toolformer: Lan- guage models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dess `ı, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Lan- guage models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2024. 2

    2024

  35. [43]

    Squinting at vqa models: Introspecting vqa models with sub-questions

    Ramprasaath R Selvaraju, Purva Tendulkar, Devi Parikh, Eric Horvitz, Marco Tulio Ribeiro, Besmira Nushi, and Ece Ka- mar. Squinting at vqa models: Introspecting vqa models with sub-questions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,...

    2020

  36. [44]

    Pangu-coder2: Boosting large lan- guage models for code with ranking feedback

    Bo Shen, Jiaxin Zhang, Taihong Chen, Daoguang Zan, Bing Geng, An Fu, Muhan Zeng, Ailun Yu, Jichuan Ji, Jingyang Zhao, et al. Pangu-coder2: Boosting large lan- guage models for code with ranking feedback. arXiv preprint arXiv:2307.14936, 2023. 3

    2023 arXiv

  37. [45]

    Execution-based code generation using deep reinforcement learning

    Parshin Shojaee, Aneesh Jain, Sindhu Tipirneni, and Chan- dan K Reddy. Execution-based code generation using deep reinforcement learning. Transactions on Machine Learning Research. 3

  38. [46]

    Ex- ploring the effectiveness of large language models in gener- ating unit tests

    Mohammed Latif Siddiq, Joanna Santos, Ridwanul Hasan Tanvir, Noshin Ulfat, FA Rifat, and V Carvalho Lopes. Ex- ploring the effectiveness of large language models in gener- ating unit tests. arXiv preprint arXiv:2305.00418, 2023. 1, 3

    2023 arXiv

  39. [47]

    10 Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xin- lei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 10 Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, pages 8317–8326, 2019. 2

    2019

  40. [48]

    Towards truly zero-shot compositional visual reasoning with llms as programmers

    Aleksandar Stani ´c, Sergi Caelles, and Michael Tschannen. Towards truly zero-shot compositional visual reasoning with llms as programmers. Transactions on Machine Learning Research. 1, 3

  41. [49]

    Vipergpt: Vi- sual inference via python execution for reasoning

    D´ıdac Sur´ıs, Sachit Menon, and Carl V ondrick. Vipergpt: Vi- sual inference via python execution for reasoning. InProceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 11888–11898, 2023. 1, 2, 6, 12

    2023

  42. [50]

    Tdd without tears: Towards test case generation from requirements through deep rein- forcement learning

    Wannita Takerngsaksiri, Rujikorn Charakorn, Chakkrit Tan- tithamthavorn, and Yuan-Fang Li. Tdd without tears: Towards test case generation from requirements through deep rein- forcement learning. arXiv preprint arXiv:2401.07576, 2024. 1, 3

    2024 arXiv

  43. [51]

    Codegemma: Open code models based on gemma

    CodeGemma Team. Codegemma: Open code models based on gemma. arXiv preprint arXiv:2406.11409, 2024. 2, 6

    2024 arXiv

  44. [52]

    Winoground: Probing vision and language models for visio- linguistic compositionality

    Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio- linguistic compositionality. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages ...

    2022

  45. [53]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Mar- tinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023. 2

    2023 arXiv

  46. [54]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Am- jad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 2

    2023 arXiv

  47. [55]

    Adacoder: Adaptive prompt compression for programmatic visual question answering

    Mahiro Ukai, Shuhei Kurita, Atsushi Hashimoto, Yoshitaka Ushiku, and Nakamasa Inoue. Adacoder: Adaptive prompt compression for programmatic visual question answering. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 9234–9243, 2024. 2

    2024

  48. [56]

    Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33:5776– 5788, 2020. 6

    2020

  49. [57]

    Editable scene simulation for autonomous driving via collaborative llm- agents

    Yuxi Wei, Zi Wang, Yifan Lu, Chenxin Xu, Changxing Liu, Hao Zhao, Siheng Chen, and Yanfeng Wang. Editable scene simulation for autonomous driving via collaborative llm- agents. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 15077–15087,

  50. [58]

    An empirical study of gpt-3 for few-shot knowledge-based vqa

    Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. In Proceedings of the AAAI conference on artificial intelligence, pages 3081– 3089, 2022

    2022

  51. [59]

    two humans and one wheel

    Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chat- gpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381, 2023. 2 11 A. Data The three compositiona...

    2023 arXiv

  52. [66]

    A modern living room featuring sleek furniture and devoid of any pets

    Image Caption: "A modern living room featuring sleek furniture and devoid of any pets" Answer: no Query: Is there a red truck or bus in the image? Tests:

  53. [69]

    A large blue semi-truck cruising down an interstate highway

    Image Caption: "A large blue semi-truck cruising down an interstate highway" Answer: no 33

  54. [73]

    Several bright yellow school buses parked in a row at a local school

    Image Caption: "Several bright yellow school buses parked in a row at a local school" Answer: no Query: What color is the largest car in the image? Tests:

  55. [77]

    A scene with a green public bus next to a smaller blue pickup at an intersection

    Image Caption: "A scene with a green public bus next to a smaller blue pickup at an intersection" Answer: green Query: Is the vase to the left or right of the center? Tests:

  56. [80]

    A ceramic vase centrally placed on a round table surrounded by chairs

    Image Caption: "A ceramic vase centrally placed on a round table surrounded by chairs" Answer: center Query: What is the highest object in the image? Tests:

  57. [86]

    A figure standing on the peak of a grassy hill under a blue sky

    Image Caption: "A figure standing on the peak of a grassy hill under a blue sky" Answer: person Query: INSERT_QUERY_HERE Tests: Listing 7. ITM Unit Test Generation In Context Examples Query: Is the drawing of a tree on the hill, and a river that flows at the bottom of the hill? Tests:

  58. [87]

    A solitary tree stands atop a gentle hill, with a flowing river winding below it

    Image Caption: "A solitary tree stands atop a gentle hill, with a flowing river winding below it." Answer: yes

  59. [88]

    A tree on a grassy hill under a clear sky

    Image Caption: "A tree on a grassy hill under a clear sky." Answer: no

  60. [89]

    A river meandering through a dense forest of tall trees

    Image Caption: "A river meandering through a dense forest of tall trees." Answer: no

  61. [90]

    A panoramic view of rolling hills in the desert, with a river at the bottom

    Image Caption: "A panoramic view of rolling hills in the desert, with a river at the bottom." Answer: no

  62. [91]

    A vast plain with a river running through fields of wildflowers

    Image Caption: "A vast plain with a river running through fields of wildflowers." Answer: no

  63. [92]

    A hill with multiple trees and a river flowing nearby

    Image Caption: Image Caption: "A hill with multiple trees and a river flowing nearby." Answer: yes Query: Is the drawing of an airplane flying in the sky, and birds flying below it? Tests:

  64. [93]

    An airplane soars through the sky, with a flock of birds flying beneath it

    Image Caption: "An airplane soars through the sky, with a flock of birds flying beneath it." Answer: yes

  65. [94]

    Birds flying over a tranquil lake under a clear sky

    Image Caption: "Birds flying over a tranquil lake under a clear sky." Answer: no

  66. [95]

    An airplane performing aerobatic maneuvers, with birds flying above it

    Image Caption: "An airplane performing aerobatic maneuvers, with birds flying above it." Answer: no

  67. [96]

    An airplane floating in the sea with birds flying above it

    Image Caption: "An airplane floating in the sea with birds flying above it." Answer: Yes

  68. [97]

    An airplane in a clear sky

    Image Caption: "An airplane in a clear sky" Answer: no Query: Is the drawing of a girl holding an umbrella in the rain? Tests:

  69. [98]

    A girl holding an umbrella walks through a rainy street

    Image Caption: "A girl holding an umbrella walks through a rainy street." Answer: yes

  70. [99]

    A girl holds an umbrella under a bright sun in the park

    Image Caption: "A girl holds an umbrella under a bright sun in the park." Answer: no

  71. [100]

    A girl stands in the rain wearing a colorful raincoat and holding flowers

    Image Caption: "A girl stands in the rain wearing a colorful raincoat and holding flowers." Answer: no

  72. [101]

    A girl walks her dog while holding an umbrella on a rainy day

    Image Caption: "A girl walks her dog while holding an umbrella on a rainy day." Answer: yes Query: Is the drawing of a person sitting at a desk with a computer monitor in front of them? Tests:

  73. [102]

    A person sitting at a desk, writing in a notebook with a lamp beside them

    Image Caption: "A person sitting at a desk, writing in a notebook with a lamp beside them." Answer: no

  74. [104]

    Someone sitting at a desk cluttered with papers and a computer monitor

    Image Caption: "Someone sitting at a desk cluttered with papers and a computer monitor." Answer: yes

  75. [105]

    A person with a big computer screen in the background

    Image Caption: "A person with a big computer screen in the background" Answer: no Query: Is the drawing of a man riding a bicycle, and a dog running beside him? Tests:

  76. [106]

    A man cycling alone on a mountain trail surrounded by trees

    Image Caption: "A man cycling alone on a mountain trail surrounded by trees." Answer: no 34

  77. [107]

    A man rides a bicycle along the beach, his dog running beside him

    Image Caption: "A man rides a bicycle along the beach, his dog running beside him." Answer: yes

  78. [108]

    A bicycle and a dog

    Image Caption: "A bicycle and a dog" Answer: no

  79. [109]

    A dog next to a car

    Image Caption: "A dog next to a car" Answer: no

  80. [110]

    A man walking his dog

    Image Caption: "A man walking his dog" Answer: no

  81. [111]

    A man rides a bicycle down a sunny street with a dog running beside him

    Image Caption: "A man rides a bicycle down a sunny street with a dog running beside him." Answer: yes Query: INSERT_QUERY_HERE Tests: Listing 8. VQA Unit Test Generation with Implementation In-Context Examples # Query: Is there a cat or dog in the image? def execute_command(im...

  82. [112]

    A grey tabby cat peacefully napping on a plush sofa

    Image Caption: "A grey tabby cat peacefully napping on a plush sofa" Answer: yes

  83. [113]

    A lively golden retriever bounding across a grassy field in the park

    Image Caption: "A lively golden retriever bounding across a grassy field in the park" Answer: yes

  84. [114]

    Twin Siamese cats playfully swatting at a bright yellow ball

    Image Caption: "Twin Siamese cats playfully swatting at a bright yellow ball" Answer: yes

  85. [115]

    A cluster of wild horses trotting along the sandy shores of a sunlit beach

    Image Caption: "A cluster of wild horses trotting along the sandy shores of a sunlit beach" Answer: no

  86. [116]

    An orange cat and a black Labrador playfully tugging on a rope toy

    Image Caption: "An orange cat and a black Labrador playfully tugging on a rope toy" Answer: yes

  87. [117]

    A modern living room featuring sleek furniture and devoid of any pets

    Image Caption: "A modern living room featuring sleek furniture and devoid of any pets" Answer: no # Query: Is there a red truck or bus in the image? def execute_command(image) -> str: image_patch = ImagePatch(image) trucks = image_patch.find("truck") buses = image_patch.find("...

  88. [118]

    A vibrant red Ford pickup parked beside a country road

    Image Caption: "A vibrant red Ford pickup parked beside a country road" Answer: yes

  89. [119]

    A red double-decker bus navigating through a busy downtown street

    Image Caption: "A red double-decker bus navigating through a busy downtown street" Answer: yes

  90. [120]

    A large blue semi-truck cruising down an interstate highway

    Image Caption: "A large blue semi-truck cruising down an interstate highway" Answer: no

  91. [121]

    A quiet suburban street devoid of any large vehicles like buses or trucks

    Image Caption: "A quiet suburban street devoid of any large vehicles like buses or trucks" Answer: no

  92. [122]

    A shiny red Ferrari speeding on a professional race track

    Image Caption: "A shiny red Ferrari speeding on a professional race track" Answer: no

  93. [123]

    An array of red delivery trucks lined up in a distribution center parking lot

    Image Caption: "An array of red delivery trucks lined up in a distribution center parking lot" Answer: yes

  94. [124]

    Several bright yellow school buses parked in a row at a local school

    Image Caption: "Several bright yellow school buses parked in a row at a local school" Answer: no # Query: What color is the largest car in the image? def execute_command(image) -> str: image_patch = ImagePatch(image) car_patches = image_patch.find("car") if not car_patches: re...

  95. [125]

    A large blue Ford pickup truck driving on a busy highway

    Image Caption: "A large blue Ford pickup truck driving on a busy highway" Answer: blue

  96. [126]

    A city street empty of any large vehicles like buses or trucks

    Image Caption: "A city street empty of any large vehicles like buses or trucks" Answer: no answer

  97. [127]

    A row of green food trucks serving lunch in an urban park

    Image Caption: "A row of green food trucks serving lunch in an urban park" Answer: green

  98. [128]

    A scene with a green public bus next to a smaller blue pickup at an intersection

    Image Caption: "A scene with a green public bus next to a smaller blue pickup at an intersection" Answer: green # Query: Is the vase to the left or right of the center? def execute_command(image) -> str: image_patch = ImagePatch(image) vase_patches = image_patch.find("vase") i...

  99. [129]

    A delicate porcelain vase positioned on the right end of a mahogany dining table

    Image Caption: "A delicate porcelain vase positioned on the right end of a mahogany dining table" Answer: right

  100. [130]

    A tall glass vase sitting on the left side of a neatly made bed in a sunlit room

    Image Caption: "A tall glass vase sitting on the left side of a neatly made bed in a sunlit room" Answer: left

  101. [131]

    A ceramic vase centrally placed on a round table surrounded by chairs

    Image Caption: "A ceramic vase centrally placed on a round table surrounded by chairs" Answer: center # Query: What is the highest object in the image? def execute_command(image) -> str: image_patch = ImagePatch(image) possible_objects = ["car", "tree", "building", "person", "...

  102. [132]

    A massive skyscraper dominating the skyline among lower city buildings

    Image Caption: "A massive skyscraper dominating the skyline among lower city buildings" Answer: skyscraper

  103. [133]

    A lone oak tree surpassing the height of the cottage it stands next to

    Image Caption: "A lone oak tree surpassing the height of the cottage it stands next to" Answer: tree

  104. [134]

    Colorful balloons drifting above the treetops in a clear sky

    Image Caption: "Colorful balloons drifting above the treetops in a clear sky" Answer: balloons

  105. [135]

    A commercial jet flying high above the city’s tallest skyscrapers

    Image Caption: "A commercial jet flying high above the city’s tallest skyscrapers" Answer: plane

  106. [136]

    A majestic eagle soaring high above a vast canyon landscape

    Image Caption: "A majestic eagle soaring high above a vast canyon landscape" Answer: eagle

  107. [137]

    A figure standing on the peak of a grassy hill under a blue sky

    Image Caption: "A figure standing on the peak of a grassy hill under a blue sky" Answer: person Create test cases for the specified query and program using the format provided in the examples. The test cases should consist of image captions and answers to the query. The answer...

  108. [2024]

    1, 3, 5, 6, 12, 17 9

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.