CARV: A Diagnostic Benchmark for Compositional Analogical Reasoning in Multimodal LLMs

Lu Lin; Minhao Cheng; Xiaohan Zou; Yongkang Du

REVIEW 4 major objections 6 minor 12 references

CARV: A Diagnostic Benchmark for Compositional Analogical Reasoning in Multimodal LLMs

T0 review · 4 major / 6 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Multimodal AI models can compose symbolic rules but struggle to extract them from images.

desk verdict A solid benchmark with a plausible but under-validated diagnosis: the decomposition-bottleneck claim hinges on a moderate-agreement evaluator. read the letter →

arxiv 2603.27958 v2 pith:ZC4PDJS6 submitted 2026-03-30 cs.AI

Yongkang Du , Xiaohan Zou , Minhao Cheng , Lu Lin This is my paper · ORCID

classification cs.AI

keywords compositionalanalogyvisualreasoningmultimodallargelanguagemodelsbenchmarkdecompositionsetoperationsdiagnosticevaluationsymbolicrules

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CARV is a new benchmark that tests whether multimodal large language models can do compositional analogy in vision: from two image pairs, each showing a change, the model must derive atomic symbolic rules, combine them with a set operation (union, intersection, or difference), and apply the combined rule to a query image. The paper reports that even the strongest tested models reach only about 40% accuracy on the compositional task, while human participants score 100%. The central diagnostic result is that failures concentrate in the decomposition stage: models can describe what changed, but not restate those changes as discrete, general symbolic rules. When the ground-truth rules are provided in an oracle setting, the same models reach 86–99% accuracy, showing that composition and application are not the binding constraint. The authors conclude that the capability gap in scaling compositional reasoning lies in translating perception into symbolic rules, not in the logic of combining or applying them.

What carries the argument

The load-bearing machinery is the formalization of an image as a set of (property, value) pairs, which turns an analogy into arithmetic over transformation sets: T(I,I') is the set of atomic changes, and the target transformation is O(T1,T2) for O in union, intersection, difference. Around this, the paper builds a controlled visual domain (few properties, values, and spatial relations) and a four-stage diagnosis pipeline that traces each model's first failure. The oracle-decomposition condition—injecting ground-truth atomic transformations—is the instrument that isolates the decomposition bottleneck.

What would settle it

Manually score a held-out set of 120 responses at every stage with human annotators using a decomposition rubric that does not impose a required symbolic format, and compare the first-failure distribution to the automated one. If decomposition no longer dominates—or if a free-form rubric raises oracle-level performance to near-ceiling—the paper's central claim is falsified. Alternatively, a model that passes perception but consistently fails composition when given noisy rule inputs would falsify the claim that composition is easy.

Watch

Extended reading notes

Core claim

The paper's central claim is that current multimodal LLMs' weak performance on CARV comes primarily from an inability to decompose visual changes into atomic symbolic rules, not from an inability to compose or apply rules. The authors define an atomic transformation as a change of one property (such as color, number, or spatial relation) between two states, and the task as applying a set operation over the transformation sets of two image pairs. Across models, a four-step diagnosis pipeline (perception, decomposition, composition, application) shows that closed-source models fail first at decomposition, open-source models fail earlier still at perception, and failures at composition and appl

Load-bearing premise

The central result hinges on the automated evaluator's stage classification being accurate: its agreement with human judges is 0.82 accuracy with Cohen's Kappa 0.63, so if it systematically marks decomposition responses as wrong (or if model captions do not reveal internal reasoning), the decomposition bottleneck is an artifact of the measurement rather than the model.

Editorial extensions

If this is right

If decomposition is the bottleneck, then adding a rule-extraction module that turns visual changes into symbolic descriptions should lift performance more than improving low-level perception for closed-source models.
The near-perfect oracle scores imply that the remaining path to human-level compositional analogy is mostly a symbol-grounding problem, not a reasoning problem.
The different-source results imply that models rely heavily on surface similarity between source and query images; a true abstraction mechanism would not degrade when the context changes.
Complexity scaling results imply that benchmarks that only report final accuracy will hide the growing decomposition burden; stage-wise diagnostics are needed to track progress.
Interventions like prompting models to explicitly format 'property changes from A to B' improve but do not close the gap, so format alone is not the issue.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If decomposition is partly an expression or formatting problem, a more flexible symbolic interface—such as accepting free-form paraphrases or parsing rules from natural language—would likely recover some of the gap; the paper's own format ablation hints at this.
CARV's controlled domain isolates reasoning from perception, but in real images perception failures would almost certainly dominate; these results should not be extrapolated to open-world analogy without testing.
The benchmark's set operations are all binary and stateless; a natural harder extension is chaining three or more source pairs or nesting operations (e.g., union of intersections), which would test whether models can build hierarchical rule structures.
Another testable extension: give models the candidate property set (subject, number, object, color, position) and see whether failure drops—if it does, part of the decomposition failure is rule discovery rather than rule representation.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

CARV is a genuinely useful benchmark: it formalizes analogy as composing transformation sets with union/intersection/difference, extends the single-pair format from KiVA/VOILA in a clean way, and comes with a controlled dataset and a reusable four-stage diagnosis protocol. The oracle ablation (Table 4) is the strongest evidence: give models the atomic rules and they jump to 86–99%, so the hard part is extracting rules from images, not applying them. That is a real result.

The soft spot is the stage attribution. The claim that decomposition—not perception—is the bottleneck depends on GPT-5.1 judging which of four stages fails first, and the reported agreement with humans is 0.82 accuracy, Kappa 0.63. That is moderate, not 'high' as the paper says. The decomposition rubric is the most format-strict: it demands atomic rules phrased with general property names. Appendix A.2 shows a model marked wrong for writing 'Furniture changes from table to chair; items change from toilet paper to spatulas'—which is semantically correct but not formatted as separate atomic transformations. If the evaluator systematically penalizes alternative phrasing, the failure distribution in Figure 3a overstates decomposition. The paper reports global agreement but not per-stage precision/recall, so we can't tell. The oracle ablation supports 'rule inference is hard' but doesn't distinguish failure to perceive a change from failure to express it in the required form. The Without Format condition helps—it improves but stays below Oracle—so formatting alone isn't the whole story, but the cleanliness of the decomposition claim is overstated.

Other issues are smaller. The 'first diagnostic benchmark' framing doesn't engage Raven progressive matrices, which already involve composing relations; the authors should explain why CARV differs. The dataset and code aren't released (promised camera-ready), which makes independent inspection impossible. The abstract's choice of Gemini-2.5 Pro's 40.4% as the headline number is fine but a bit selective, since GPT-5.1 does better on several cells.

Overall, the benchmark is worth having and the core observation is plausible. The diagnostic conclusion needs more work before it becomes a standard reference. I'd send it to peer review with a request for per-stage evaluator validation (human labels on each stage, or a second judge) and a clearer comparison with Raven. It's a solid submission, not a finished one.

Referee Report

4 major / 6 minor

Summary. The paper introduces CARV, a 5,500-sample diagnostic benchmark for compositional analogical reasoning in vision. The task extends single-step analogy to multiple image pairs whose transformations are combined via set operations (union, intersection, difference). The dataset is built in a controlled visual domain with five pre-defined properties and a small vocabulary of values. The authors evaluate several closed- and open-source MLLMs using direct prompting, and report that even the strongest model (Gemini-2.5 Pro) achieves around 40% accuracy on the hardest setting, while humans achieve 100%. A four-stage diagnosis pipeline (perception, decomposition, composition, application) is used to attribute failures; the central claim is that closed-source models fail primarily at decomposition, i.e., translating visual changes into atomic symbolic rules, while composition and application are nearly solved when the rules are provided (Oracle ablation, Table 4). The paper also studies how performance degrades with more atomic transformations and with different source images, attributing the degradation to decomposition rather than perception.

Significance. If the diagnostic claim holds, the paper makes a valuable contribution by localizing the capability gap in multimodal compositional reasoning to the perception-to-symbolic abstraction step, rather than to rule-level logic or final application. The task formulation is novel and the controlled-domain design is a sensible way to isolate reasoning from noisy perception. The Oracle decomposition ablation (Table 4) is a strong internal control and provides credible evidence that rule inference is a major bottleneck. The benchmark itself, if released as promised, could be a useful resource for future work on compositional reasoning in MLLMs. However, the central stage-level attribution depends on an automated evaluator (GPT-5.1) whose agreement with human judges is only moderate (accuracy 0.82, Cohen's Kappa 0.63), and the validation is only reported globally, not per stage. Because the decomposition rubric is the most format-strict (Table 7), an evaluator bias at this stage could inflate the headline finding. The significance is therefore conditional on strengthening the evaluator validation and refining the interpretation of the ablation results.

major comments (4)

[§5.3, Table 7] The stage-diagnosis claim (Figure 3a: decomposition is the dominant first-failure stage for closed-source models) rests entirely on GPT-5.1's judgments against the rubric in Table 7. The validation reported is 0.82 accuracy and Cohen's Kappa 0.63 on 120 samples, which is conventionally moderate agreement, not 'high'. More importantly, only a single global agreement score is given; no per-stage precision/recall is reported. The decomposition rubric is the strictest of the four stages: it requires atomicity, general property vocabulary, and full inclusion of reference transformations. Appendix A.2 shows a model marked incorrect for a semantically correct but non-atomic phrasing. If the evaluator systematically penalizes non-atomic but semantically correct decompositions, Figure 3a will be dominated by decomposition failures regardless of the true bottleneck. Please provide a per-stage conf
[§5.5, Table 4] The Oracle ablation shows that supplying ground-truth atomic transformations improves accuracy to 86–99%, which supports the statement that rule inference is hard. However, it does not validate the attribution to the decomposition stage specifically: it does not distinguish failure to perceive the visual changes from failure to express perceived changes in the required symbolic form. The 'Without Format' condition (Table 4) is intended to separate format effects, but it still requires the model to describe the transformations and to follow the diagnosis prompt; it only removes the bracketed symbolic format. The observed gains (e.g., GPT-5.1: 60→78, Gemini-2.5 Flash: 77→90) show that formatting is a non-negligible part of the decomposition failure, and the remaining gap to Oracle could be due to perception or to inability to articulate rules even in free-form language. The conclusion that
[§5.3, Reproducibility] The paper's main deliverable is the CARV dataset, but the dataset, code, and evaluation scripts are promised only for the camera-ready version (footnote 1 and Reproducibility Statement). For a benchmark paper, this prevents reviewers and readers from verifying the central empirical claims, and from checking the data construction, the filtering step (2.4% removed), and the exact prompts. Please provide an anonymized link to the dataset and code as supplementary material for review, even if the final public release is deferred.
[§3.1 / §5.4] The conclusion in §5.4 that 'the fundamental challenge in scaling compositional reasoning is not the visual processing' is supported by Figure 3b showing that the proportion of perception failure 'decreases or remains stable' as N increases. But perception is evaluated indirectly through the model's captions and natural-language transformation descriptions. A model that perceives the changes but fails to mention them due to format or attention constraints would be counted as a perception failure (per the rubric in Table 7, perception includes the transformation description). This could undercount perception failures and overcount decomposition failures at higher N. Please clarify the criteria for distinguishing a perception failure from a decomposition failure when the model's description is incomplete but its caption is semantically correct, and provide examples at N=3 and N=4.

minor comments (6)

[Abstract / Table 2] The abstract states 'Gemini-2.5 Pro achieving only 40.4% accuracy' without specifying which setting this refers to. From Table 2, 40.4% is the Different-Source Union result; please clarify that this is the best overall or a specific column, or give a range.
[§5.1] The human test used three participants. Please report the number of samples per task, the agreement among the three participants, and whether all three achieved 100% on the same test set. A 100% human score on a small or easy sample can be misleading.
[§5.3] 'We instruct GPT-5.1 with the ground truth for each intermediate step' is unclear. The evaluator receives reference captions and reference transformation sets, but not the ground-truth failure stage. Rephrase to describe exactly what is given to the evaluator and what is withheld.
[Figure 3a] The stacked bar percentages appear to exceed 100% in some rows (e.g., Gemini-2.5 Flash: 54.0 + 73.3 + 42.6 + 47.3). This is likely because bars show the stage distribution among failures, not a decomposition of total cases. Please clarify the denominator and the normalization in the caption.
[A.1 / Appendix] There is a numbering inconsistency: Table 5 in the main text refers to neuro-symbolic prompting results, but Table 5 appears only in the appendix. Please renumber tables and refer to them consistently.
[Throughout] Some references are incomplete (e.g., 'Team, 2025' and 'Comanici et al., 2025' may refer to the same Gemini report; 'Kamath et al., 2023' is used for the image source, but the editing step with Gemini-2.5 Flash is described only via a blog reference). Please ensure all citations are complete and consistently formatted.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity; the paper is an empirical benchmark study with no derivation that reduces to its inputs.

full rationale

CARV is a constructed benchmark, and the paper's claims are empirical measurements rather than derivations. The central diagnostic claim—that decomposition is the bottleneck—comes from a staged evaluation where GPT-5.1 labels the first failed stage against ground-truth intermediate steps. This is an operational measurement, not a quantity fitted from the data it then explains. The Oracle ablation (Table 4) is an independent intervention: supplying ground-truth atomic transformations and observing near-perfect composition/application does not presuppose the decomposition finding; it is a controlled comparison. The logistic regression in Figure 4 is a post-hoc descriptive analysis, not a prediction drawn from the same fit. The paper cites two of the authors' own papers (Liu et al. 2024; Du et al. 2025) in a contextual sentence in Related Work, but these citations are not load-bearing; no uniqueness theorem or prior result is used to force the conclusions. The evaluator's human agreement (0.82 accuracy, Cohen's Kappa 0.63) is moderate and could weaken the stage attribution, but that is a validity/measurement concern, not circularity: the rubric is applied to held-out model responses, and the Without Format ablation partially controls for format-specific scoring. No equation in the paper equals its input by construction, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No parameters are fitted to data: the benchmark involves design choices (property set P, value-set sizes, sample splits 500/1500/1500/2000), which are task-design decisions, not fitted quantities; the logistic-regression weights in §5.3 are descriptive statistics, not parameters the central claim depends on. No new physical or theoretical entities are postulated: 'atomic transformation' is a definition (Eq. 1), and the ground truth is anchored to human agreement (100% on the human sample). The load-bearing assumptions are the fidelity of the synthetic images, the sufficiency of the caption vocabulary, the caption-as-reasoning proxy, and the reliability of the automated judges.

assumptions (5)

domain assumption Images generated by Gemini-2.5 Flash image editing faithfully realize exactly the intended atomic transformations, and no unintended ones.
§4.1: images are edited from Kamath et al. sources and 2.4% are manually removed for quality issues; the remaining 97.6% are assumed clean. If unintended changes (e.g., subtle spatial shifts) are present, ground-truth labels are wrong.
domain assumption The property set P = {subject, subject number, object, object color, spatial relation} and its value sets are sufficient to describe every image unambiguously.
§3.2/§4.2: states are defined as sets of (property, value) pairs; the benchmark's validity requires every image to be uniquely captioned within this vocabulary.
domain assumption Caption outputs are a faithful proxy for the model's visual reasoning: a wrong caption means failed reasoning, not failed verbalization.
§5.1/§5.5: the entire evaluation is caption-based; the format ablation (Table 4) partially tests this, showing free-form output helps a little but stays far below oracle.
domain assumption Automated judges (GPT-4o for answers, GPT-5.1 for diagnosis stages) provide valid correctness labels.
§5.1: GPT-4o judged 98% consistent with human labels on 200 samples; §5.3: the GPT-5.1 diagnosis evaluator has 0.82 accuracy / Kappa 0.63 vs humans on 120 samples — moderate-to-substantial agreement.
standard math Set operations (union, intersection, difference) on atomic-transformation sets behave as defined in Eq. (1) with unique, well-defined results given the sampled pairs.
§3.1/§3.2: the formal target T = O(T1,T2) is well-defined; no further mathematical assumptions beyond standard set algebra.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CARV: A Diagnostic Benchmark for Compositional Analogical Reasoning in Multimodal LLMs." pith.science (2026). https://pith.science/paper/ZC4PDJS6

@misc{pith2026260327958,
  author       = {Pith},
  title        = {Pith review of: CARV: A Diagnostic Benchmark for Compositional Analogical Reasoning in Multimodal LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZC4PDJS6}},
  note         = {Machine review of arXiv:2603.27958}
}

read the original abstract

Analogical reasoning tests a fundamental aspect of human cognition: mapping the relation from one pair of objects to another. Existing evaluations of this ability in multimodal large language models (MLLMs) overlook the ability to compose rules from multiple sources, a critical component of higher-order intelligence. To close this gap, we introduce CARV (Compositional Analogical Reasoning in Vision), a novel task together with a 5,500-sample dataset as the first diagnostic benchmark. We extend the analogy from a single pair to multiple pairs, which requires MLLMs to extract symbolic rules from each pair and compose new transformations. Evaluation on the state-of-the-art MLLMs reveals a striking performance gap: even Gemini-2.5 Pro achieving only 40.4% accuracy, far below human-level performance of 100%. Diagnostic analysis shows two consistent failure modes: (1) decomposing visual changes into symbolic rules, and (2) maintaining robustness under diverse or complex settings, highlighting the limitations of current MLLMs on this task.

Figures

Figures reproduced from arXiv: 2603.27958 by the authors.

**Figure 1.** Examples of single-step and compositional analogy. The single-step analogy (left) delivers one transformation (color) from input to target, while the compositional analogy (right) delivers the union of color (wood to red) and number (one to two) transformations. The development of multimodal large language models (MLLMs) (Comanici et al., 2025) has significantly advanced visual reasoning, enabling models to tac… view at source ↗

**Figure 2.** Overview of Diagnosis Pipeline. Given two image pairs, the model is instructed to [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Detailed analysis of failure distributions. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Failure Contribution by Property Combinations. For most models, combinations among subject, number, and position contribute most to the failure. In this section, we study the error distribution among subtasks and how the image contexts affect models’ performance. Ac…

**Figure 6.** Figure 6: Accuracy by Number of Atomic Transformations. The performance drops as the [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 5.** Figure 5: Error Distribution by Property Combination of Gemini-2.5 Flash. We study the most challenging combinations, the failure mainly concentrates in decomposition. Q3: Are MLLMs able to generalize in different source contexts? Our results indicate MLLMs generally struggle…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 2 linked inside Pith

[1]

Generate caption for images in image pairs (I1, I2) and (I3, I4) and describe the transforma- tions in natural language. Use the format: **Caption of (I1)**: [Concise caption of image I1] **Caption of (I2)**: [Concise caption of image I2] **Transformation of (I1, I2)**: [Describe the differences of the two images, be specific and detailed] **Caption of (I...
[2]

[Quantity] changes from two knives to one knife

**Decompose the Transformation of Each Image Pair** *Transformation T1**: [Color] changes from wooden to red. [Quantity] changes from two knives to one knife. *Transformation T2**: [Type] changes from chair to table. [Color] changes from wooden to red. [Quantity] remains the same (two knives)
[3]

[Color] changes from wooden to red.[Quantity] should be at most two knives, as a change from two to one is also possible

**Derive a Final Target Transformation Rule** *Target Transformation**: [Type] changes from chair to table. [Color] changes from wooden to red.[Quantity] should be at most two knives, as a change from two to one is also possible. ... Reason Here, T1 contains quantity transformation (from two to one) while T2 doesn’t contain quantity transformation. As we ...

2023
[4]

The final output should be a concise caption of the resulting image in JSON format

Apply this final transformation rule to the query image I5 to predict the resulting image. The final output should be a concise caption of the resulting image in JSON format. **REQUIRED OUTPUT FORMAT** {”caption”: your concise caption of the resulting image} Table 6: Template for Guided Prompting 16 Preprint. Under review. Evaluation Prompt You are an eva...
[6]

Describe what changes, and from which to which

Decompose the transformation of each image pair according to the property(s). Describe what changes, and from which to which. i.e. [texture] changes from [metal] to [glass]. Use the format: **Transformation T1**: [Your description of the transformation] **Transformation T2**: [Your description of the transformation]
[7]

URL https://aclanthology.org/2025

doi: 10.18653/v1/2025.findings-naacl.128. URL https://aclanthology.org/2025. findings-naacl.128/. Chengwei Qin, Wenhan Xia, Tan Wang, Fangkai Jiao, Yuchen Hu, Bosheng Ding, Ruirui Chen, and Shafiq Joty. Relevant or Random: Can LLMs Truly Perform Analogical Reasoning? In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.),Fin...

arXiv 2025
[8]

description When T1 or T2 contains items which indicate no change For example, T1 includes ”[property] remains the same”, and T2 includes ”[property] changes from A to B”

Derive a final target transformation rule based on the set operation operation on T1 and T2. description When T1 or T2 contains items which indicate no change For example, T1 includes ”[property] remains the same”, and T2 includes ”[property] changes from A to B”. When the operation is UNION, only consider the transformation that indicates a change for th...
[9]

Instruction: (1) whether the response is correct according to the reference captions

Perception: Check whether the model correctly perceives the visual changes in two image pairs (I1, I2) and (I3, I4). Instruction: (1) whether the response is correct according to the reference captions. The reference caption provides multiple details (color, number, position), the model’s response should reflect these details accurately. Notice, if positi...

Show all 12 references

[10]

Instruction: (1) When describing one transformation, the model must follow [property] changes from [value A] to [value B]

Decomposition: Check whether the model correctly decomposes the transformation of each image pair according to the property(s). Instruction: (1) When describing one transformation, the model must follow [property] changes from [value A] to [value B]. The property must be a gen...
[11]

Reference target transformation: Transformation T: {target transformation}

Composition: Check whether the model correctly derives the target transformation rule based on the set operation on T1 and T2. Reference target transformation: Transformation T: {target transformation}
[12]

Application: Check whether the model correctly applies this final transformation rule to the query image I5 to predict the resulting image. Query image:{caption5} **REQUIRED OUTPUT FORMAT** { ”failure stage”: 1,2,3, or 4 (indicating which evaluation task the model first failed...
[280]

Yanbo Wang, Justin Dauwels, and Yilun Du

URLhttps://aclanthology.org/2021.acl-long.280/. Yanbo Wang, Justin Dauwels, and Yilun Du. Compositional scene understanding through inverse generative modeling.arXiv preprint arXiv:2505.21780, 2025. 12 Preprint. Under review. Michihiro Yasunaga, Xinyun Chen, Yujia Li, Panupong...

2021 arXiv

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.

[1] [1]

Generate caption for images in image pairs (I1, I2) and (I3, I4) and describe the transforma- tions in natural language. Use the format: **Caption of (I1)**: [Concise caption of image I1] **Caption of (I2)**: [Concise caption of image I2] **Transformation of (I1, I2)**: [Describe the differences of the two images, be specific and detailed] **Caption of (I...

[2] [2]

[Quantity] changes from two knives to one knife

**Decompose the Transformation of Each Image Pair** *Transformation T1**: [Color] changes from wooden to red. [Quantity] changes from two knives to one knife. *Transformation T2**: [Type] changes from chair to table. [Color] changes from wooden to red. [Quantity] remains the same (two knives)

[3] [3]

[Color] changes from wooden to red.[Quantity] should be at most two knives, as a change from two to one is also possible

**Derive a Final Target Transformation Rule** *Target Transformation**: [Type] changes from chair to table. [Color] changes from wooden to red.[Quantity] should be at most two knives, as a change from two to one is also possible. ... Reason Here, T1 contains quantity transformation (from two to one) while T2 doesn’t contain quantity transformation. As we ...

2023

[4] [4]

The final output should be a concise caption of the resulting image in JSON format

Apply this final transformation rule to the query image I5 to predict the resulting image. The final output should be a concise caption of the resulting image in JSON format. **REQUIRED OUTPUT FORMAT** {”caption”: your concise caption of the resulting image} Table 6: Template for Guided Prompting 16 Preprint. Under review. Evaluation Prompt You are an eva...

[5] [6]

Describe what changes, and from which to which

Decompose the transformation of each image pair according to the property(s). Describe what changes, and from which to which. i.e. [texture] changes from [metal] to [glass]. Use the format: **Transformation T1**: [Your description of the transformation] **Transformation T2**: [Your description of the transformation]

[6] [7]

URL https://aclanthology.org/2025

doi: 10.18653/v1/2025.findings-naacl.128. URL https://aclanthology.org/2025. findings-naacl.128/. Chengwei Qin, Wenhan Xia, Tan Wang, Fangkai Jiao, Yuchen Hu, Bosheng Ding, Ruirui Chen, and Shafiq Joty. Relevant or Random: Can LLMs Truly Perform Analogical Reasoning? In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.),Fin...

arXiv 2025

[7] [8]

description When T1 or T2 contains items which indicate no change For example, T1 includes ”[property] remains the same”, and T2 includes ”[property] changes from A to B”

Derive a final target transformation rule based on the set operation operation on T1 and T2. description When T1 or T2 contains items which indicate no change For example, T1 includes ”[property] remains the same”, and T2 includes ”[property] changes from A to B”. When the operation is UNION, only consider the transformation that indicates a change for th...

[8] [9]

Instruction: (1) whether the response is correct according to the reference captions

Perception: Check whether the model correctly perceives the visual changes in two image pairs (I1, I2) and (I3, I4). Instruction: (1) whether the response is correct according to the reference captions. The reference caption provides multiple details (color, number, position), the model’s response should reflect these details accurately. Notice, if positi...