{"id":"02a0d429-9a9b-4a58-b9a8-0177b0e45134","arxiv_id":"2412.01027","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An autoregressive model with group self-attention that separates learning from applying achieves state-of-the-art few-shot image manipulation on unseen instructions.","lead":"InstaManip is a new autoregressive model that learns image-editing operations from a text description plus a couple of example image pairs, then applies the operation to a new photo. It works by splitting the 'learn from examples' and 'apply to query' steps into separate attention groups, and it reports large gains over prior diffusion-based few-shot editors.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'unseen instruction' test set is not verified to be image-disjoint from training; if query or exemplar images recur in training under other instructions, the reported in-context learning gains may partly reflect image memorization rather than learning from the exemplar pair.","rationale":"The Reader's verdict is CONDITIONAL and identifies the Eq. (2) factorization as the weakest assumption. That is a legitimate theoretical concern: if query-dependent exemplar reasoning were required, the group self-attention could discard useful information. However, the paper already provides a relevant ablation (Table 2: replacing group self-attention with vanilla causal self-attention hurts performance), and the supplementary's B.2 demonstrates that visual exemplar features (e.g., car color) are transferred, so the factorization is at least partially validated empirically. The more load-bearing threat to the paper's central empirical claim is that the 'unseen' evaluation may not be clean: the test-set construction filters instructions by keyword but does not filter images. Given the InstructPix2Pix dataset structure, image reuse across instructions is likely, and a large autoregressive model could exploit memorized query images to inflate scores. This does not prove the results are wrong, but it means the strongest claim (state-of-the-art on unseen instructions) has not yet been established under a fully disjoint split. A single concrete check -- an image-disjoint retraining and re-evaluation -- would settle whether the reported margins survive. Because this concern is additive to the Reader's conditions rather than a contradiction of the CONDITIONAL verdict, I keep the verdict unchanged and mark agreement as partial.","tokens_in":19983,"tokens_out":11912,"duration_ms":121009,"concrete_test":"Construct an image-disjoint variant of the test protocol: remove from the training set every sample whose source or target image (or a near-duplicate, e.g., same image ID or high perceptual similarity) appears in any test exemplar or query image. Retrain InstaManip and the strongest baseline(s) on this filtered corpus and re-run the in-distribution and out-of-distribution evaluations of Table 1 plus the human preference study. If the CLIP-Dir / CLIP-Vis / preference gaps shrink substantially (e.g., by more than 30 percent), image memorization is a significant confound; if the margins persist, the in-context learning claim is robust to image-level leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that InstaManip generalizes to unseen manipulation instructions through in-context learning. Section 4.1 and supplementary C.1 describe how the test set was constructed: 30 low-occurrence keywords were selected, all instructions containing them were removed from training, and the remaining test instances were manually checked. This procedure controls instruction-level leakage but says nothing about image-level overlap. The InstructPix2Pix dataset reuses source/target images across many instructions, so a test query image can plausibly appear in training samples paired with different, non-excluded instructions. Because InstaManip is a 13B autoregressive model trained for 20,000 iterations on those images, it can memorize image-specific visual features. In the in-distribution setting, where exemplar and query share image content, such memorization could produce high CLIP-Dir and CLIP-Vis scores without the model having to abstract the transformation from the exemplar pair. The out-of-distribution setting is less vulnerable, but it is not described with enough detail to rule out overlap either. Since the paper's headline numerical results and human preference study all depend on this split, the absence of an image-disjoint evaluation is a load-bearing gap. The paper's own limitation section (Sec. D) does not mention this issue, and the promised code/data release is not yet available, so the risk cannot currently be checked by third parties.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces InstaManip, a multi-modal autoregressive model (LLaMA-13B backbone with frozen Qwen-ViT image encoder and SDXL decoder) for few-shot image manipulation. The central idea is to decompose in-context learning into two stages: a learning stage that condenses the textual instruction and exemplar image pair into M=30 learnable manipulation tokens Z, and an applying stage that uses Z and the query image to generate the manipulated output. This is implemented via a group self-attention mechanism with separate causal attention groups, together with a relation regularization loss that aligns pairwise similarities of manipulation tokens with CLIP text-encoder similarities of the corresponding instructions. The model is trained on the InstructPix2Pix dataset with the 30 lowest-occurrence instruction keywords held out, and evaluated on the remaining 325 instructions/1296 samples, comparing against ImageBrush, VISII, PromptDiffusion, and InstructPix2Pix. Reported results show SOTA figures (in-distribution CLIP-Dir 19.81 vs 17.13 for PromptDiffusion; OOD 18.27 vs 15.41; human preference 40.67% vs 21.67% in-distribution), with ablations demonstrating contributions from both proposed components and a scaling study showing further gains from additional or more diverse exemplar pairs.","tokens_in":20246,"tokens_out":4634,"duration_ms":41315,"significance":"If the reported results hold, this is a meaningful advance: it is the first attempt to purpose-build an autoregressive architecture for few-shot image manipulation, and the group self-attention separation of learning and applying stages is a simple, reusable idea that could inform other in-context visual tasks. The manuscript provides a clean ablation isolating group self-attention and relation regularization, a plausible human evaluation protocol, and a qualitative analysis with failure cases. However, the evaluation has two load-bearing risks: the test set may not be image-disjoint from training (undermining the 'unseen instruction' claim), and the primary metrics CLIP-Dir/CLIP-Vis are computed in the same CLIP space used by the relation-regularization objective (part of the reported gain may be directly induced by the loss). The absence of error bars or significance tests further limits confidence in the margins.","major_comments":[{"comment":"The test-set construction excludes instructions containing 30 selected low-occurrence keywords but does not verify that query or exemplar images are disjoint from training images. Because the InstructPix2Pix dataset reuses the same source/target images across many instructions, a test example's query or exemplar image may appear in training paired with a different, non-excluded instruction. As InstaManip is a 13B autoregressive model trained for 20,000 iterations on those images, image-level memorization could inflate the reported in-distribution CLIP-Dir, CLIP-Vis, and CLIP-T scores and the human preference rates without the model actually abstracting the transformation from the exemplar pair. Please report image-overlap statistics, evaluate on a subset with no overlapping images, or provide a memorization probe such as pairing each query with a random exemplar to show that output quality is contingent on the exemplar-derived transformation.","section":"§4.1, §C.1"},{"comment":"The relation regularization in Eq. (5) explicitly matches the manipulation-token similarity matrix to the CLIP text-encoder similarity matrix, while the headline metrics CLIP-Dir and CLIP-Vis are computed in the same CLIP embedding space. Consequently, part of the reported improvement on these two metrics may be a direct consequence of the training objective rather than evidence of better manipulation fidelity. The paper should quantify this by reporting at least one direction/quality metric computed outside the CLIP space (e.g., a DINO-ViT-based directional score or a pixel-space direction metric) and by showing the relation-regularization ablation on such an independent metric.","section":"§3.3, Eq. (5); §4.1 metrics"},{"comment":"All head-to-head results are reported as point estimates without error bars, confidence intervals, or significance tests. The test set contains only 325 instructions and 1296 samples, and the user study samples 100 examples with six raters each; the claimed notable margin (e.g., CLIP-Dir 19.81 vs 17.13; human preference 40.67% vs 21.67%) could be within sampling noise. Please provide per-instruction paired bootstrap or statistical tests for the CLIP metrics and per-instance preference proportions with an appropriate test for the user study, plus inter-rater agreement.","section":"Table 1, Fig. 6, §C.4"},{"comment":"The factorization in Eq. (2) enforces that manipulation tokens Z are conditionally independent of the query image given the exemplars and text, and the group self-attention makes the exemplars and text invisible to generation tokens. This design discards any query-dependent reasoning about which parts of an exemplar to copy or how the transformation should interact with the query's content. The authors should state explicitly when this assumption holds in practice and show a qualitative or quantitative test that reveals its limitations, for example object replacement where the exemplar object's color or orientation should adapt to the query image's illumination and perspective.","section":"§3.2, Eq. (2)"}],"minor_comments":[{"comment":"Typo: 'keywrods' should be 'keywords'.","section":"§C.1"},{"comment":"Typo: 'Pleae' should be 'Please'.","section":"Table 4 caption"},{"comment":"Typo: 'CLP-T' should be 'CLIP-T'.","section":"§4.2"},{"comment":"The 'out of dist.(diverse)' setting is defined only implicitly in the caption; please define it explicitly in the main text of §4.5, since the distinction between regular and diverse out-of-distribution exemplars is central to the scaling claim.","section":"Fig. 9"},{"comment":"The prompt template text should clarify whether the placeholders surround special token boundaries and whether the template tokens are encoded as text or as learnable embeddings; this would help reproducibility of the group self-attention split.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"The image-disjointness and CLIP-metric-circularity concerns are the two main risks to the paper's central claim. Both are addressable with additional experiments, and the architecture proposal itself is interesting and likely to be useful. If the authors can show that the results survive an image-disjoint test set and an independent metric, I would be inclined to accept; without those, the reported numerical advantages are not yet convincing. The novelty relative to generic autoregressive MLLMs is also a bit thin—the comparison to Emu2 is consigned to the supplementary—so strengthening that comparison with more baselines would improve the paper's positioning."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"InstaManip is a real architectural idea with an honest evaluation, but don't trust the headline CLIP numbers until the test set is shown to be image-disjoint. The group self-attention mechanism that separates learning from applying is the genuinely new piece, and the ablation shows it buys about 1.5-2 points on CLIP-Dir and CLIP-Vis. Relation regularization adds another point. That separation is intuitive, and the empirical gain is credible. The paper is also honest about failure cases and about CLIP-I being a flawed metric, which I appreciate.\n\nThe soft spots are real but addressable. First, the test set filters out instructions containing 30 low-occurrence keywords, but nothing is said about whether the query or exemplar images appear in training under other instructions. InstructPix2Pix reuses images across many instructions, so this is not a theoretical concern. If a test image was seen during training, the in-distribution numbers and the human study could partly reflect memorization. The OOD setting is less vulnerable, but it isn't described with enough detail to rule out overlap either. The limitations section doesn't mention this, and code and data aren't out yet, so I can't check. This is the largest gap, and it is load-bearing for the 'unseen instruction' claim.\n\nSecond, the relation regularization forces manipulation-token similarities toward CLIP text similarities, and the headline metrics are computed in that same CLIP space. So part of the reported gain on CLIP-Dir and CLIP-Vis is the model being trained to be more CLIP-like. The human preference result (40.67 vs 20.00) is independent of that, which helps, but I would still want significance tests.\n\nMissing error bars, a 325-instruction test set, and a 100-example human study are minor-to-moderate concerns; they don't invalidate the central claim. The factorization in Eq. (2) is a modeling assumption, and the GSA enforces it—if real manipulations need query-dependent copying from exemplars, the architecture throws that away. But the paper's own failure cases show exactly that limit, so it's a stated trade-off, not a hidden one.\n\nBottom line: a serious referee should see this, but the revision needs an image-disjoint evaluation split and a clearer statement about the CLIP-space overlap. I'd tell the editor it's conditionally acceptable, not a desk reject.","headline":"Real architectural idea with a credible ablation, but the evaluation needs an image-disjoint test split and a CLIP-circularity check before the headline numbers can be trusted.","tokens_in":20824,"tokens_out":2260,"would_cite":true,"duration_ms":20780,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"InstaManip, an autoregressive model, claims to learn unseen image-manipulation operations from a text instruction plus a few exemplar image pairs and to apply them to new query images, outperforming prior few-shot manipulation models.","keywords":["few-shot image manipulation","in-context learning","autoregressive model","group self-attention","visual prompting","image editing","relation regularization","multi-modal model"],"falsifier":"Construct a manipulation whose correct output depends on which object in the query is selected by the exemplar, e.g., an exemplar pair turns the left person into a pirate while a query contains two people and the manipulation should apply only to the person matching the exemplar; if the group self-attention model edits the wrong person or both, the information bottleneck at $Z$ is too narrow and the factorization fails.","tokens_in":19786,"feed_emoji":"🖼️","tokens_out":7128,"duration_ms":62386,"temperature":0.7,"pith_summary":"InstaManip is a multi-modal autoregressive model built for few-shot image manipulation: given a textual instruction, a query image, and one or more exemplar source-target image pairs, it learns the underlying edit operation and applies it to the query. The paper's central claim is that this in-context learning process should be split into two stages—first abstracting a manipulation representation from the text and exemplars, then applying it to the query—and that a group self-attention mechanism can enforce this split inside an autoregressive transformer. On instructions deliberately held out of training, the model reports higher alignment with both the text and the exemplars than prior few-shot manipulation models, and human raters prefer its outputs about twice as often. The paper also claims that adding more exemplars or more diverse exemplars further improves performance.","feed_headline":"Autoregressive model learns new image edits from examples","feed_subtitle":"Two-stage attention extracts the edit rule from text and exemplars, then applies it to new query images.","key_machinery":"The central object is the group self-attention (GSA) layer with 30 trainable manipulation tokens $Z$ inserted between the exemplar pair and the query image. The layer splits the prompt into two causally masked attention groups: the first contains the textual instruction and exemplar images and produces $Z$; the second contains $Z$, the query image, and generation tokens, which produce the output. This enforces the two-stage factorization in Eq. (2) during end-to-end training. The companion mechanism is relation regularization, which aligns the pairwise similarity of manipulation-token embeddings with the text-embedding similarity of the corresponding instructions, so that $Z$ encodes the transformation rather than irrelevant image content.","core_discovery":"The paper introduces InstaManip, an autoregressive model that performs few-shot image manipulation through in-context learning rather than test-time optimization. It models the conditional distribution $P(Y|X,T,X',Y')$ as $P(Z|T,X',Y')\\cdot P(Y|X,Z)$, where $Z$ is a set of learnable manipulation tokens that are supposed to capture the transformation independently of the query image. A group self-attention layer restricts attention so that the text and exemplar pair produce $Z$ in a first group, and the query image plus $Z$ produce the manipulated image in a second group; the exemplars and text are invisible to the generation tokens. A relation regularization loss, computed with a pre-trained text encoder, pushes manipulation embeddings of semantically similar instructions together. On a held-out set built from 30 rare-instruction keywords, the model reports text-direction alignment 19.81 and visual-example alignment 32.39 in-distribution, versus 17.13 and 27.69 for the strongest prior method, and human preference rates of 40.67 percent versus 20.00 percent. The authors' interpretation is that explicit separation of learning and applying stages improves in-context reasoning for image manipulation.","pith_inferences":["The factorization implicitly assumes that the exemplars' role ends once $Z$ is formed; a manipulation that requires comparing the query with the exemplar at generation time would fall outside the model's inductive bias, so measured gains may come from easy-to-abstract edits rather than fine relational reasoning.","The text-embedding-based relation regularization couples the visual manipulation space to language semantics; for instructions whose decisive details are purely visual, this could either help or distort the representation depending on how well the text encoder captures those details.","The reported gains moving from two to three exemplars suggest that a dataset with more than four exemplars per instruction could reveal whether the benefit saturates or continues, an experiment the current data cannot run.","The same learning-and-applying decomposition could be tested as a general principle for other in-context generation problems, such as style transfer, super-resolution, or video editing, where exemplars define a mapping rather than a single task."],"forward_implications":["If the two-stage split is correct, autoregressive models can rival diffusion models on few-shot editing, expanding the design space for image-manipulation systems.","The model's improvement with more exemplars and with diverse exemplars suggests that few-shot manipulation performance is a scaling curve rather than a fixed ceiling.","Relation regularization provides a way to transfer textual semantic structure into visual manipulation embeddings, which could be reused for other instruction-following visual tasks.","Because the manipulation tokens $Z$ are the only bridge between examples and query, the same design may extend to video or multi-image manipulation by keeping the learned transformation fixed."],"supporting_citations":[{"why":"Provides the dataset of 313,010 manipulation instructions that the test set is carved from, and serves as the text-only baseline.","marker":"[7]"},{"why":"A prior CLIP-space instruction-inversion method that the paper compares against and the source of the CLIP-Vis metric.","marker":"[43]"},{"why":"A prior diffusion-based few-shot manipulation method that arranges exemplars in a 2x2 grid; the main visual-prompt baseline.","marker":"[62]"},{"why":"A prior diffusion in-context baseline that injects exemplar features through a control branch; the strongest competitor on text-direction and visual-example alignment.","marker":"[75]"},{"why":"The pre-trained text encoder whose pairwise instruction similarities define the target matrix in the relation regularization loss.","marker":"[51]"},{"why":"A generic autoregressive in-context learner used as an additional comparison to motivate task-specific architectural design.","marker":"[61]"}],"fun_headline_variants":["Autoregressive model learns image edits from few examples","Two-stage attention enables few-shot image manipulation","InstaManip: instant learning of new image edits","Few-shot image editing via in-context autoregressive learning","Autoregressive model applies learned edit rules to new images"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a fixed set of 30 manipulation tokens can carry all the information needed to specify the edit, so that once $Z$ is known, the output is independent of the original text and exemplars.","fun_headline_variants_meta":{"raw":{"variants":["Autoregressive model learns image edits from few examples","Two-stage attention enables few-shot image manipulation","InstaManip: instant learning of new image edits","Few-shot image editing via in-context autoregressive learning","Autoregressive model applies learned edit rules to new images"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000249,"raw_usage":{"total_tokens":1581,"prompt_tokens":1007,"completion_tokens":574,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":623,"completion_tokens_details":{"reasoning_tokens":497}},"tokens_in":623,"tokens_out":574,"duration_ms":5920,"temperature":1.0,"reasoning_tokens":497,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:45:27.568261+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a manipulation whose correct output depends on which object in the query is selected by the exemplar, e.g., an exemplar pair turns the left person into a pirate while a query contains two people and the manipulation should apply only to the person matching the exemplar; if the group self-attention model edits the wrong person or both, the information bottleneck at $Z$ is too narrow and the factorization fails.","supporting_citations":[{"cited_title":"Visual instruction inversion: image editing via visual prompting","cited_arxiv_id":null,"evidence_quote":"A prior CLIP-space instruction-inversion method that the paper compares against and the source of the CLIP-Vis metric."},{"cited_title":"Imagebrush: learning visual in-context instructions for exemplar-based image manipulation","cited_arxiv_id":null,"evidence_quote":"A prior diffusion-based few-shot manipulation method that arranges exemplars in a 2x2 grid; the main visual-prompt baseline."},{"cited_title":"In-context learning unlocked for diffu- sion models","cited_arxiv_id":null,"evidence_quote":"A prior diffusion in-context baseline that injects exemplar features through a control branch; the strongest competitor on text-direction and visual-example alignment."},{"cited_title":"Generative multimodal mod- els are in-context learners","cited_arxiv_id":null,"evidence_quote":"A generic autoregressive in-context learner used as an additional comparison to motivate task-specific architectural design."}],"review_version":1}