Pith

open record

sign in

arxiv: 2606.23270 · v1 · pith:WESMG3AR · submitted 2026-06-22 · cs.CV

BoxCtrl: 3D-Aware Visual Prompting for Geometric Image Editing

Feifei Wang , Shiyuan Yang , Xiaoyu Li , Jing Liao This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-26 08:42 UTCgrok-4.3pith:WESMG3ARrecord.jsonopen to challenge →

classification cs.CV
keywords visual promptinggeometric image editing3D bounding boxesdiffusion modelsreinforcement learningsynthetic dataset3D-aware editing
0
0 comments X

The pith

Projecting RGB 3D bounding boxes onto images enables precise 3D geometric edits by decoupling geometry from appearance.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to solve the challenge of precise 3D geometric image editing, such as rotating or scaling objects consistently in three dimensions, which text instructions and 2D guides often fail at. It proposes using projected 3D boxes with faces colored red, green, and blue as visual prompts that provide position, size, and orientation information in a single compact example. This design lets the model focus on geometric intent separately from the object's appearance. A two-stage process of fine-tuning on synthetic data and then reinforcement learning on real images further improves results. If successful, this would make advanced photo manipulations more reliable and intuitive for users.

Core claim

BoxCtrl is a framework that uses informative RGB 3D bounding boxes projected onto 2D images as visual prompts, where the three orthogonal faces are painted with distinct RGB colors to simultaneously encode position, size, and orientation. These boxes decouple geometric control from appearance control, allowing the model to learn consistent correspondences between same-colored faces in latent space for a precise understanding of geometric intentions and accurate editing results, achieved via supervised fine-tuning on a large synthetic dataset followed by reinforcement learning on unpaired real data.

What carries the argument

RGB 3D bounding boxes with distinctly colored orthogonal faces, acting as in-context visual examples that encode 3D geometry for the editing model.

If this is right

  • Achieves state-of-the-art results on translation, rotation, scaling, and composite geometric editing tasks.
  • Maintains photorealistic image quality while enhancing geometric accuracy.
  • Bridges the gap between synthetic training data and real-world images through the RL stage.
  • Provides an intuitive visual way to specify 3D transformations without complex text prompts.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The colored face correspondence technique might apply to other tasks requiring 3D understanding from 2D images, like object detection or pose estimation.
  • Extending the method to handle multiple objects or interactive editing sessions could broaden its utility.
  • The reward function in RL, focused on geometric accuracy and fidelity, suggests similar reward designs could improve other control-based generation models.

Load-bearing premise

That painting the box faces with distinct RGB colors allows the model to consistently map those colors to specific geometric properties across different images and edits.

What would settle it

Running the model on a test image with a known 3D transformation specified by the box and checking whether the output image shows the object moved, rotated, or scaled exactly as indicated by the box projection, with no appearance changes.

Figures

Figures reproduced from arXiv: 2606.23270 by Feifei Wang, Jing Liao, Shiyuan Yang, Xiaoyu Li.

Figure 1
Figure 1. Figure 1: Our method takes as input a source image, its corresponding RGB 3D bounding box, and a user-specified transformed RGB 3D bounding box. The [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of visual prompting for geometric editing. We contrast [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Pipeline Overview. Our method processes instructions, source images, and box pairs into conditional tokens for geometric image editing. The model is trained via SFT followed by RL, utilizing a DiT-based policy optimized using GroundingDINO, OrientAnythingV2, and CLIP as geometry reward models. Key components include: (1) Visual Prompting: 2D-projected 3D boxes with RGB faces; (2) Tokenization: Concatenated… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative comparison with state-of-the-art image editing methods. We evaluate our approach against instruction-based baselines (FLUX￾Kontext, Qwen-Image-Edit), visual prompting-based methods (FreeFine, Magic Fixup), and 3D-aware image generation methods (LooseControl, Build-A-Scene). The results span four categories: translation (Rows 1-2), scaling (Rows 3-4), rotation (Rows 5-6), and composite editing (… view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative comparison under different conditions. Left to right: source image, source RGB 3D bbox, ground truth, and ground truth RGB 3D bbox. Results are shown for: (A) text prompting only, (B) RGB 3D bbox only, and (C) RGB 3D bbox + text. Src (a) (b) (c) GT [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative comparison under different visual prompting con￾ditions. Cols (a-c): dual gray, single RGB, and dual RGB bboxes. Top rows: conditions with source (Src). Bottom: results with ground truth (GT). Src GT SFT SFT + RL [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Qualitative comparison of SFT and SFT+RL on synthetic data. Left to right: source image, ground truth, SFT, and SFT+RL. Src SFT w/o 𝑟rot w/o 𝑟ts w/o 𝑟sim Ours “Turn left 150.0 degrees, shrink by a factor of 0.90, move left 0.10.". “Turn right 120.0 degrees, enlarge by a factor of 1.20, move left 0.10." [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Qualitative comparison of reward settings. Left to right: source image, SFT only, w/o rotation, w/o translation & scaling, w/o image similarity, and ours (joint reward). Source images from PIE-Bench [Ju et al. 2024]. Source Edit (a) Large geometric transformations Source Edit (b) Reflective surfaces Source Edit (c) Complex illumination Source Edit (d) Crowded scene structures Source Edit Source Edit (e) Oc… view at source ↗
read the original abstract

As instruction-based editing models and multimodal large language models advance, diverse image editing tasks have become feasible. However, achieving precise and consistent geometric image editing, such as translating, scaling, and rotating in 3D space, remains a major challenge. In this work, we introduce BoxCtrl, a 3D-aware visual prompting framework. Unlike text-only or coarse 2D-guided approaches, our method introduces informative RGB 3D bounding boxes projected onto 2D images as visual prompts. The three orthogonal faces of each box are painted with distinct RGB colors, simultaneously encoding position, size, and orientation to provide a compact, intuitive in-context visual example. The key to BoxCtrl's success lies in these well-designed bounding boxes, which decouple geometric control from appearance control. This enables the model to learn consistent correspondences between faces of the same color in the latent space, leading to a precise understanding of geometric intentions and accurate editing results. We introduce a two-stage training paradigm: Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL). To address paired data scarcity, we construct a large-scale synthetic dataset for SFT, equipping the model with fundamental editing capabilities. To bridge the synthetic-to-real domain gap, we incorporate an online RL stage leveraging unpaired real-world data. Guided by a reward function evaluating geometric accuracy and visual fidelity, our SFT-RL strategy significantly enhances geometric precision while maintaining photorealistic quality. Extensive experiments demonstrate that BoxCtrl achieves state-of-the-art performance across translation, rotation, scaling, and composite editing tasks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The paper introduces BoxCtrl, a 3D-aware visual prompting framework for geometric image editing tasks including translation, rotation, scaling, and composites. It uses projected RGB 3D bounding boxes onto 2D images as visual prompts, with the three orthogonal faces painted in distinct RGB colors to encode position, size, and orientation while decoupling geometry from appearance via learned consistent face correspondences in latent space. Training follows a two-stage paradigm of supervised fine-tuning on a large-scale synthetic dataset followed by reinforcement learning on unpaired real-world data guided by a reward function for geometric accuracy and visual fidelity. The abstract asserts that this yields state-of-the-art performance.

Significance. If the performance claims hold with supporting evidence, the work could advance precise 3D geometric control in image editing beyond text-only or coarse 2D prompts by providing an intuitive, compact visual prompting mechanism that separates geometric intent from appearance.

major comments (2)
  1. Abstract: The central claim of state-of-the-art performance across translation, rotation, scaling, and composite editing tasks is asserted without any quantitative metrics, error bars, ablation studies, dataset details, or experimental results, rendering the performance assertions unevaluable.
  2. Abstract: The key mechanism—that distinct RGB colors on the three orthogonal faces enable learning of consistent correspondences between same-color faces in latent space, thereby decoupling geometry from appearance—is presented without ablations (e.g., single-color vs. multi-color boxes), attention visualizations, or latent-space analyses to demonstrate that the color scheme (rather than projected box geometry alone) drives the claimed geometric precision.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on the abstract. We address the two major comments point by point below. The full manuscript contains the supporting experimental details referenced in the abstract.

read point-by-point responses

  1. Referee: Abstract: The central claim of state-of-the-art performance across translation, rotation, scaling, and composite editing tasks is asserted without any quantitative metrics, error bars, ablation studies, dataset details, or experimental results, rendering the performance assertions unevaluable.

    Authors: The abstract is a high-level summary. All quantitative metrics (with error bars), ablation studies, dataset details, and experimental results are provided in the Experiments section of the full manuscript. To address the concern, we will revise the abstract to incorporate brief references to key quantitative results supporting the SOTA claim. revision: yes

  2. Referee: Abstract: The key mechanism—that distinct RGB colors on the three orthogonal faces enable learning of consistent correspondences between same-color faces in latent space, thereby decoupling geometry from appearance—is presented without ablations (e.g., single-color vs. multi-color boxes), attention visualizations, or latent-space analyses to demonstrate that the color scheme (rather than projected box geometry alone) drives the claimed geometric precision.

    Authors: The abstract describes the proposed mechanism at a summary level. Supporting ablations (including single- vs. multi-color comparisons), attention visualizations, and latent-space analyses are presented in the main body of the manuscript. We will partially revise the abstract to indicate that these analyses confirm the role of the color scheme. revision: partial

Circularity Check

0 steps flagged

No circularity; empirical framework with no derivations or self-referential reductions

full rationale

The paper describes an empirical method using projected RGB 3D bounding boxes as visual prompts, followed by SFT on synthetic data and RL on real data. No equations, first-principles derivations, fitted parameters renamed as predictions, or self-citation chains appear in the provided text. The central claim attributes performance to prompt design (distinct RGB faces enabling latent correspondences), but this is presented as a design hypothesis validated by experiments rather than a mathematical reduction to inputs. No load-bearing steps reduce by construction to the paper's own definitions or citations. The work is self-contained against external benchmarks via reported SOTA results on editing tasks.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 1 invented entities

The central claim depends on the effectiveness of the colored-box design for decoupling geometry from appearance and on the ability of the reward function to guide RL across the synthetic-to-real gap; both are domain assumptions without independent verification in the abstract.

axioms (2)
  • domain assumption Painting the three orthogonal faces with distinct RGB colors allows the model to learn consistent latent correspondences for geometric control
    Explicitly stated as the key to success in the abstract
  • domain assumption The reward function used in the RL stage accurately measures both geometric accuracy and visual fidelity on unpaired real data
    Required for the online RL stage to bridge the domain gap
invented entities (1)
  • RGB 3D bounding boxes as visual prompts no independent evidence
    purpose: To encode position, size, and orientation in a compact in-context example
    New prompting design introduced by the paper

pith-pipeline@v0.9.1-grok · 5821 in / 1457 out tokens · 31018 ms · 2026-06-26T08:42:51.424041+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 6 canonical work pages · 4 internal anchors

  1. [1]

    FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space

    FLUX. 1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space , author=. arXiv preprint arXiv:2506.15742 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Qwen-Image Technical Report

    Qwen-Image Technical Report , author=. arXiv preprint arXiv:2508.02324 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Image Sculpting: Precise Object Editing with 3D Geometry Control , author=. IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  4. [4]

    ACM Transactions on Graphics , author=

    Magic Fixup: Streamlining Photo Editing by Watching Dynamic Videos , volume=. ACM Transactions on Graphics , author=. 2025 , pages=

    2025

  5. [5]

    ACM Special Interest Group on Computer Graphics and Interactive Techniques , pages=

    3D-Fixup: Advancing Photo Editing with 3D Priors , author=. ACM Special Interest Group on Computer Graphics and Interactive Techniques , pages=

  6. [6]

    IEEE/CVF International Conference on Computer Vision , pages=

    Training-free Geometric Image Editing on Diffusion Models , author=. IEEE/CVF International Conference on Computer Vision , pages=

  7. [7]

    arXiv preprint arXiv:2506.17450 , year=

    BlenderFusion: 3D-Grounded Visual Editing and Generative Compositing , author=. arXiv preprint arXiv:2506.17450 , year=

    work page arXiv

  8. [8]

    ACM Special Interest Group on Computer Graphics and Interactive Techniques Asia , pages=

    BlobCtrl: Taming Controllable Blob for Element-level Image Editing , author =. ACM Special Interest Group on Computer Graphics and Interactive Techniques Asia , pages=

  9. [9]

    IEEE/CVF Winter Conference on Applications of Computer Vision , pages=

    GeoDiffuser: Geometry-Based Image Editing with Diffusion Models , author=. IEEE/CVF Winter Conference on Applications of Computer Vision , pages=

  10. [10]

    IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Diffusion Handles Enabling 3D Edits for Diffusion Models by Lifting Activations to 3D , author=. IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  11. [11]

    Diffusion

    Kaiwen Zheng and Huayu Chen and Haotian Ye and Haoxiang Wang and Qinsheng Zhang and Kai Jiang and Hang Su and Stefano Ermon and Jun Zhu and Ming-Yu Liu , booktitle=. Diffusion

  12. [12]

    Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection , author="Liu, Shilong and Zeng, Zhaoyang and Ren, Tianhe and Li, Feng and Zhang, Hao and Yang, Jie and Jiang, Qing and Li, Chunyuan and Yang, Jianwei and Su, Hang and Zhu, Jun and Zhang, Lei", booktitle=

  13. [13]

    Advances in Neural Information Processing Systems , pages =

    Orient Anything V2: Unifying Orientation and Rotation Understanding , author =. Advances in Neural Information Processing Systems , pages =

  14. [14]

    International Conference on Machine Learning , pages=

    Learning Transferable Visual Models From Natural Language Supervision , author=. International Conference on Machine Learning , pages=

  15. [15]

    Advances in Neural Information Processing Systems , pages =

    OBJECT 3DIT: Language-guided 3D-aware Image Editing , author =. Advances in Neural Information Processing Systems , pages =

  16. [16]

    Advances in Neural Information Processing Systems , pages =

    Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Models , author =. Advances in Neural Information Processing Systems , pages =

  17. [17]

    Advances in Neural Information Processing Systems , pages =

    ORIGEN: Zero-Shot 3D Orientation Grounding in Text-to-Image Generation , author =. Advances in Neural Information Processing Systems , pages =

  18. [18]

    IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages =

    VideoHandles: Editing 3D Object Compositions in Videos Using Video Generative Priors , author =. IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages =

  19. [19]

    IEEE/CVF International Conference on Computer Vision , pages=

    Zero-Shot Depth Aware Image Editing with Diffusion Models , author=. IEEE/CVF International Conference on Computer Vision , pages=

  20. [20]

    ACM Special Interest Group on Computer Graphics and Interactive Techniques Asia , pages=

    Customizing Text-to-Image Diffusion with Object Viewpoint Control , author =. ACM Special Interest Group on Computer Graphics and Interactive Techniques Asia , pages=

  21. [21]

    IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    Compass Control: Multi Object Orientation Control for Text-to-Image Generation , author=. IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  22. [22]

    arXiv preprint arXiv:2506.21446 , year=

    Controllable 3D Placement of Objects with Scene-Aware Diffusion Models , author=. arXiv preprint arXiv:2506.21446 , year=

    work page arXiv

  23. [23]

    ACM Special Interest Group on Computer Graphics and Interactive Techniques , pages=

    LOOSECONTROL: Lifting ControlNet for Generalized Depth Conditioning , author =. ACM Special Interest Group on Computer Graphics and Interactive Techniques , pages=

  24. [24]

    International Conference on Learning Representations , pages=

    Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation , author=. International Conference on Learning Representations , pages=

  25. [25]

    IEEE/CVF International Conference on Computer Vision , pages=

    I2V3D: Controllable Image-to-video Generation with 3D Guidance , author=. IEEE/CVF International Conference on Computer Vision , pages=

  26. [26]

    IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages =

    AnyDoor: Zero-shot Object-level Image Customization , author =. IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages =

  27. [27]

    Edward J Hu and yelong shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen , booktitle=. Lo

  28. [28]

    International Conference on Learning Representations , pages=

    Flow Matching for Generative Modeling , author=. International Conference on Learning Representations , pages=

  29. [29]

    IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages =

    Kubric: A Scalable Dataset Generator , author =. IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages =

  30. [30]

    IEEE International Conference on Robotics and Automation , pages=

    Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items , author =. IEEE International Conference on Robotics and Automation , pages=

  31. [31]

    Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities

    Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities , author=. arXiv preprint arXiv:2507.06261 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  32. [32]

    International Conference on Learning Representations , pages=

    PnP Inversion: Boosting Diffusion-based Editing with 3 Lines of Code , author=. International Conference on Learning Representations , pages=

  33. [33]

    IEEE/CVF International Conference on Computer Vision , pages=

    OminiControl: Minimal and Universal Control for Diffusion Transformer , author=. IEEE/CVF International Conference on Computer Vision , pages=

  34. [34]

    IEEE/CVF International Conference on Computer Vision , pages=

    Detect Anything 3D in the Wild , author=. IEEE/CVF International Conference on Computer Vision , pages=

  35. [35]

    IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    ObjectMover: Generative Object Movement with Video Prior , author=. IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  36. [36]

    IEEE Transactions on Image Processing , year=

    Image quality assessment: from error visibility to structural similarity , author=. IEEE Transactions on Image Processing , year=

  37. [37]

    IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

    The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , author=. IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

  38. [38]

    IEEE/CVF International Conference on Computer Vision , pages =

    Emerging Properties in Self-Supervised Vision Transformers , author =. IEEE/CVF International Conference on Computer Vision , pages =

  39. [39]

    Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks

    Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks , author=. arXiv preprint arXiv:2401.14159 , year=

    work page internal anchor Pith review Pith/arXiv arXiv