{"id":"4ebdafcb-e42b-4b3d-ac64-dae31a0a28cc","arxiv_id":"2412.00684","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"POBF paints new backgrounds around preserved objects to synthesize visual-grounding training data and filters those samples with teacher-model scores, improving accuracy by 5.83% over real-only training.","lead":"This paper presents POBF, a method that creates extra training pictures for visual grounding by keeping the object in its box and having an AI paint a new background around it, then selecting the most useful pictures. It reports consistent accuracy gains when only 1% of real labeled data is available, across four standard benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Label alignment in POBF is asserted, not verified: after repainting the background at strength 0.9, the original referring expression may no longer uniquely match the boxed object, yet no validity check is reported.","rationale":"The reader's weakest_assumption identifies exactly the condition on which the central mechanism depends: the synthetic (T, B) labels must be valid. I agree with that identification. The paper asserts the labels are valid because the box contents are preserved, but it does not supply any measurement supporting that assertion, and its own strength setting (0.9, described as controlling noise added to the base image) makes preservation questionable. Additionally, referring expressions in these datasets are often relational, so preserving the object pixels is not sufficient to preserve the meaning of T after the background is repainted. This is not an external consensus disagreement; it is an internal gap between the stated guarantee and the reported experimental configuration. A simple human/VLM annotation test on generated samples would settle whether the concern lands. The empirical gains in Table 2 are encouraging, but they are compatible with the filter selecting samples whose labels happen to remain valid, and do not by themselves prove that the generation strategy solves label misalignment. Other issues noted by the reader—missing code, inconsistent margin statements, lambda values—are reproducibility problems but do not threaten the mechanism as directly. I therefore recommend keeping the reader's conditional verdict, with the condition made explicit: the authors should report a label-validity check on the synthetic data.","tokens_in":17796,"tokens_out":9676,"duration_ms":89684,"concrete_test":"Take a random sample of 200 synthetic images generated by the POBF pipeline with the reported settings (SDXL, strength=0.9, BLIP captions). For each image, have independent annotators (or a strong VLM with verified accuracy) answer whether the original referring expression T still uniquely identifies the object in the given box B. Report the misalignment rate (fraction of samples where T is judged not to correctly and uniquely refer to B). If this rate exceeds a small threshold (e.g., 5%), the 'strictly align' claim in §4.1 is unsupported and the label-alignment mechanism is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that POBF overcomes label misalignment rests on the assertion in §4.1 that painting outside the box leaves the object inside the box 'unchanged' and thus 'strictly align[s] with the ground truth bounding box.' This assertion is not verified. The method repaints the entire outside region with an SDXL inpainting model using strength = 0.9 (§5), and the paper describes strength as 'the amount of noise added to the base image,' which controls similarity between the output and the original image. At that strength, the unmasked box region may be substantially noised and reconstructed rather than copied exactly. More importantly, even if the box pixels are preserved, referring expressions are often relational ('right cow,' 'bus in front of a building'); changing the background can remove or add the referents or distractors that make T valid, so T may no longer uniquely identify the object in B. The filter's hardness score only checks whether the teacher predicts the box; it does not check whether T is still the correct referring expression for that box. The paper reports no human or model-based verification of label validity on the generated images. If a non-trivial fraction of synthetic samples carry misaligned (T, B) labels, the measured 5.83% gain over real-only training does not demonstrate that the proposed generation strategy produces correctly labeled data, and the benefit may not transfer to real deployment.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes POBF, a framework for data-scarce visual grounding. POBF generates synthetic training images by inpainting the region outside the ground-truth bounding box while keeping the boxed object intact, and it augments text by captioning the cropped box. A teacher model trained only on the limited real data then scores each synthetic image with a hardness score (IoU of the teacher's prediction with the box), an overfitting score (1 minus IoU when the box is masked), and a penalty term (IoU with an empty text query); the highest-scoring synthetic image per real sample is kept and added to the training set. Experiments on RefCOCO, RefCOCO+, RefCOCOg, and ReferItGame report an average gain of 5.83% over real-data-only training and consistent gains across different generators, captioners, data sizes, and student architectures.","tokens_in":17995,"tokens_out":9651,"duration_ms":87763,"significance":"If the empirical claims hold, POBF is a practical and flexible data-augmentation method for visual grounding under severe data scarcity. Its strengths are that the generation pipeline uses only off-the-shelf image-caption pretrained models, the filtering scheme is compared against several data-selection baselines, and the robustness experiments span four datasets, three architectures, three generators/captioners, and three training-data sizes. The paper is also transparent about the teacher being trained on real data only and about the held-out evaluation protocol. The main unresolved risk is that the central label-alignment assumption is asserted rather than verified, and the ablation table as printed does not support all of the stated component-wise gains.","major_comments":[{"comment":"Label alignment is asserted rather than verified. The claim that painting outside the box leaves the object 'unchanged' and therefore 'strictly align[s] with the ground truth bounding box' is not demonstrated for the chosen strength of 0.9, which adds substantial noise to the base image and can alter pixels inside the box. Even if the box pixels were exactly preserved, replacing the background can invalidate relational referring expressions such as 'right cow' or 'bus in front of a building,' so the original text T may no longer uniquely refer to B in the new image. The hardness score in Eq. (1) checks only whether the teacher predicts B from T; it does not verify that T is a valid description of the boxed object in the synthetic image. I request a label-validity check (human evaluation on a sample, or an automated VQA/CLIP-based check) and a report of how often the box content changes at strength=0.9, because the 5.83% gain is attributed to a generation strategy whose labels are assumed correct.","section":"§4.1 and §5 Implementation Details"},{"comment":"The headline comparison against X-Paste, Gen2Det, and GeoDiffusion is based on a single randomly sampled subset with no error bars, while the three-subset average is reported only for the Real and POBF rows. Thus the claim of outperforming leading baselines by 2.29%–3.85% rests on one draw. Please report the three-subset average (or at least standard deviations) for the baselines as well, or provide a significance test. In addition, the header 'All methods employ the proposed filtering scheme' needs a clear statement of which hyperparameters (K, q, lambda) were shared by the baselines, since the table otherwise reads as a comparison of full pipelines rather than generation strategies under a common filter.","section":"Table 2"},{"comment":"The ablation table does not support the stated per-component gains. The text claims that S1, S2, and P give gains of 1.23%, 0.71%, and 1.14% 'compared to the variant without any filtering,' but Table 3 contains no visible row for S2 alone or P alone, and the differences between the no-filter Real+Synimg row (31.31) and the filtering rows are not 1.23, 0.71, and 1.14. The reported 'improvement of 1.96% over the baseline without filtering' for the full scheme also does not match any visible no-filter baseline; the only visible pairwise difference equal to 1.96 is with the S1-only row, which is itself a filtering variant. Please clarify the row encodings or add the missing rows so that each component's individual contribution can be verified.","section":"§5.2, Table 3"},{"comment":"The penalty term P is computed as IoU(T(I', ∅), B), but TransVG and most grounding models require a non-empty text query. The manuscript does not specify how the empty string is encoded or why a box prediction with no text measures 'prior knowledge.' Since the penalty term is one of the three components of the final score in Eq. (4), this operational detail directly affects reproducibility of the filtering scheme.","section":"§4.2, Eq. (3)"}],"minor_comments":[{"comment":"The reported margins over baselines are inconsistent: the Abstract and §5.1 state 2.29%–3.85%, while the Conclusion states 2.74%–4.35%; please correct one of these.","section":"Abstract vs. Conclusion"},{"comment":"Please specify the population over which the three scores are normalized; if the normalization is per real image (over its K synthetic variants) rather than global, the selection behavior of Eq. (4) is different and should be described.","section":"§4.2, Eq. (4)"},{"comment":"The explanation that the modest improvement on ReferIt stems from small objects and large inpainted background regions is a hypothesis; either provide supporting evidence (for example, an object-size analysis) or clearly mark it as speculative.","section":"§5.1"},{"comment":"The rows labeled 'Replace the Image Captioner with ...' do not show the default BLIP captioner row in the table; adding it would make the comparison easier to read.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"This is a solid empirical contribution with a clearly stated pipeline and a sensible set of controls. The two issues that block acceptance are both fixable: direct verification of label alignment after inpainting, and alignment of the ablation text with the numbers actually shown in Table 3. I do not see grounds for rejection, and I do not see circularity in the evaluation, since the teacher is trained only on real data and the results are measured on held-out test sets."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper inverts Gen2Det's inside-box inpainting: it leaves the object unchanged and repaints the background, then uses a teacher-based filter to pick synthetic samples. That is a genuine, simple idea, and the experiments are reasonably thorough: four datasets, ablations on each filter component, robustness across generators, data scales, and student backbones. The gains, around 5-6 points over real-only and 2-4 over baselines, are consistent across settings. I believe the central claim holds up as an empirical contribution.\n\nWhat is actually new: the paint-outside-the-box recipe plus the three-term teacher filter (hardness, overfitting, penalty). The filter builds on known data-pruning ideas, but the combination and the task-specific tailoring are new enough. The paper also frames the data-scarcity setting clearly and compares against relevant baselines.\n\nNow the soft spots. The biggest one is label alignment. Section 4.1 asserts that the object inside the box remains \"unchanged\" and thus strictly aligns with the box. But strength is set to 0.9, which the paper itself defines as the amount of noise added to the base image. That likely perturbs the box region at least to some degree. More importantly, referring expressions are often relational (\"right cow,\" \"bus in front of a building\"); repainting the background can remove the referent or add distractors, making the original text invalid for the new image. The paper reports no validity check on the synthetic labels. The filter may weed out some of these, since a teacher that cannot localize the box gets a low hardness score, but that is not the same as verifying the text-box pair. This is a real gap, though not a fatal one: the method mostly works, but the mechanism story about \"strict alignment\" is too strong.\n\nThe other issues are reporting. The main table has no error bars for the headline comparison; there is a separate three-sample average with std, but the abstract quotes the single-run result. Lambda values are tuned on validation with only two possible values, and per-dataset values are not given. The conclusion quotes a margin range (2.74-4.35) that doesn't match the abstract (2.29-3.85). The \"penalty\" term is added with a positive weight, so it is really a bonus term; the name is confusing. Code is not released, so exact reproduction is not possible from the text.\n\nWho is it for: researchers working on synthetic training data for vision-language tasks, particularly referring expressions. It deserves a serious referee; the idea is useful and the experiments are solid enough to spend referee time on. I would recommend accepting it into the review process with a request for the label-validity analysis and cleaner reporting.","headline":"A useful synthetic-data recipe for data-scarce visual grounding, with a plausible label-alignment story that needs verification.","tokens_in":18610,"tokens_out":2793,"would_cite":true,"duration_ms":26260,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"POBF repaints only the background around a target object, then filters synthetic samples with teacher scores, yielding an average 5.83% gain over real-data-only visual grounding training in data-scarce settings.","keywords":["visual grounding","data-scarce learning","synthetic training data","image inpainting","data filtering","data pruning","referring expression comprehension","diffusion models"],"falsifier":"Compare POBF's selected synthetic set against a version in which each generated image's box content is validated by an independent object detector or human annotator; if removing samples with corrupted box contents does not lower accuracy or the gain disappears, the assumed label alignment is not what drives the result.","tokens_in":17508,"feed_emoji":"🎨","tokens_out":7438,"duration_ms":64731,"temperature":0.7,"pith_summary":"The paper claims that visual grounding can be learned from very little real data if you generate extra training images by repainting everything outside the target bounding box and then keep only the synthetic samples a teacher model judges useful. The proposed framework, POBF, keeps the object inside the box untouched, so the original text-box annotation stays aligned, and selects one of several generated variants per real sample using a score that combines how easy the sample is, whether the background could cause overfitting, and a penalty for overly easy or uninformative backgrounds. Across four benchmark datasets at 1% of real training data, POBF reports an average gain of 5.83% over real-data-only training and margins of 2.29% to 3.85% over prior synthetic-data methods. The authors also show the gains persist across different generators, captioners, data sizes, and grounding architectures.","feed_headline":"Paint outside the box lifts visual grounding by 5.83%","feed_subtitle":"Regenerating the background while keeping the box intact, then filtering easy samples, beats baselines by up to 3.85%.","key_machinery":"The load-bearing object is the paint-outside-the-box inpainting step: an off-the-shelf diffusion inpainting model regenerates everything outside the bounding box at high strength while the pixels inside the box are copied unchanged, so the synthetic image is guaranteed to align with the box by construction. On top of that sits a three-term selection score computed by a teacher network trained only on the scarce real data. The hardness score $S_1$ is the teacher's IoU with the true box given the text; the overfitting score $S_2 = 1 - \\mathrm{IoU}(\\mathrm{teacher}(\\mathrm{masked\\ image}, \\mathrm{text}), \\mathrm{box})$ flags backgrounds that leak the answer; and the penalty term $P$ is the IoU the teacher gets from the image alone with an empty text prompt. Together they rank the generated variants and one winner per real sample enters the student's training set.","core_discovery":"The central claim is that the label misalignment that plagues synthetic visual grounding data comes from editing the object rather than the context, and that painting outside the box fixes it. Starting from a real image, a real text query, and its bounding box, the method captions the image, inpaints the region outside the box with high strength so the background changes substantially, and pairs the new image with the original box and text. Because the box pixels are never regenerated, the synthetic sample inherits the exact ground-truth localization. A teacher trained on the scarce real data then scores each generated image: $S_1$ measures how confidently the teacher localizes the target with the text, $S_2$ measures whether the teacher can still localize it when the box is masked (a sign the background carries unintended shortcut features), and $P$ measures localization from the image alone with an empty text prompt. The weighted sum of the three normalized scores, tuned by grid search, picks one synthetic image per real sample, and the student trains on real plus selected synthetic data, sometimes with regenerated captions.","pith_inferences":["The same paint-outside-the-box rule should transfer to other region-labeled tasks such as instance segmentation, detection, and keypoint localization: preserve the annotated region and regenerate its context, then filter by a teacher; this extension is not tested in the paper.","The per-sample 'keep one winner' rule limits synthetic expansion to 2x; a diversity-aware selection that keeps multiple high-scoring variants could use the same scores to get more benefit from a fixed generator budget.","Because the teacher is trained on the same tiny real set, the filter inherits the teacher's blind spots; averaging scores across multiple teacher seeds or architectures could make the selection more stable, but the paper's teacher set is single.","The paper's own hint that small objects gain less suggests a testable extension: constrain the inpainted background area or use an object-aware mask so the generator does not have to synthesize huge surrounding regions."],"forward_implications":["If the claim holds, dense region-text annotations are not a hard requirement for visual grounding: a small real set plus repainted backgrounds and teacher-filtered selection can lift accuracy by 5.83% on average.","Because the box pixels are never regenerated, label misalignment is avoided by construction, which is the paper's diagnosis of why prior object-editing and text-to-image baselines underperform.","The filter's three scores beat the common CLIP-similarity selection rule by an average of 1.32%, suggesting that hardness and overfitting capture complementary quality signals.","The gains are not tied to one generator or architecture: the paper reports consistent improvements across two alternative image generators, two alternative captioners, three grounding models, and three data-scarce budgets at 0.5%, 1%, and 2% of real data."],"supporting_citations":[{"why":"Supplies the transformer-based visual grounding architecture used for both teacher and student.","marker":"[12]"},{"why":"Baseline that edits the object inside the box; its label misalignment motivates painting outside the box.","marker":"[63]"},{"why":"Baseline that copy-pastes detected instances into images; strongest prior synthetic-data competitor.","marker":"[98]"},{"why":"Baseline that generates detection data from text prompts; limited generalization motivates the real-image editing approach.","marker":"[5]"},{"why":"Off-the-shelf diffusion inpainting model that produces the synthetic backgrounds.","marker":"[50]"},{"why":"Image captioner that supplies generated text descriptions for augmented samples.","marker":"[32]"},{"why":"Data pruning result used to justify prioritizing easy samples in the data-scarce regime.","marker":"[57]"},{"why":"Evidence that image backgrounds can carry spurious cues, motivating the overfitting score.","marker":"[72]"}],"fun_headline_variants":["Paint outside the box lifts visual grounding by 5.83%","Fix label misalignment by inpainting outside the box","Synthetic data selection boosts visual grounding by 5.83%","Paint-outside-box training beats baselines by up to 3.85%","Hardness and overfitting scores filter best synthetic grounding data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach assumes that inpainting the background leaves the object inside the bounding box visually unchanged and semantically aligned with the original text; if the generator alters the object or introduces overlapping artifacts, the synthetic labels are wrong even after filtering.","fun_headline_variants_meta":{"raw":{"variants":["Paint outside the box lifts visual grounding by 5.83%","Fix label misalignment by inpainting outside the box","Synthetic data selection boosts visual grounding by 5.83%","Paint-outside-box training beats baselines by up to 3.85%","Hardness and overfitting scores filter best synthetic grounding data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000674,"raw_usage":{"total_tokens":3067,"prompt_tokens":946,"completion_tokens":2121,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":2031}},"tokens_in":562,"tokens_out":2121,"duration_ms":14506,"temperature":1.0,"reasoning_tokens":2031,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:07:22.353022+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare POBF's selected synthetic set against a version in which each generated image's box content is validated by an independent object detector or human annotator; if removing samples with corrupted box contents does not lower accuracy or the gain disappears, the assumed label alignment is not what drives the result.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Baseline that edits the object inside the box; its label misalignment motivates painting outside the box."},{"cited_title":"Learning to Imagine: Visually-Augmented Natural Language Generation","cited_arxiv_id":"2305.16944","evidence_quote":"Evidence that image backgrounds can carry spurious cues, motivating the overfitting score."}],"review_version":1}