{"id":"908c0e4d-a7ed-4042-b95b-669ebf741834","arxiv_id":"2412.00122","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"An object-detection reward that checks category and count fidelity is used to fine-tune Stable Diffusion, but the headline evaluation metric is the same as the training reward.","lead":"A research team fine-tunes Stable Diffusion using a reward built from object counts detected in generated images. They report better compositional alignment scores, but their main metric is the same function they optimize, and the reward's gradients appear to vanish exactly where object counts matter.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed differentiable reward path is absent: hard thresholding, NMS, and discrete box counts make the reward gradient zero almost everywhere, and Eq. (2) does not match categories, so the reported gains cannot be attributed to the proposed SGD fine-tuning.","rationale":"The reader's weakest assumption and my primary concern coincide: the differentiability claim is central and unsubstantiated. I verified the reward construction in Section 3.2 and Algorithm 1; every quantity used in the reward is either a hard count or an indicator, so the reward function is piecewise constant in the relevant variables. This is not a missing implementation detail but a logical gap: the optimization claimed in Section 3.3 cannot be performed as described. The additional Eq. (2) defect is independent and makes the reward definition itself inconsistent with its stated purpose. The paper does provide a dataset, code link, and qualitative evidence, and the general idea of detection-based feedback is plausible, but the central quantitative claims rest on the differentiable-reward mechanism and on a CQ Score that is both the training reward and the headline metric. Because the mechanism is not supported and the metric is circular, the rejection stands; no change to the reader's verdict is needed.","tokens_in":13815,"tokens_out":8299,"duration_ms":97017,"concrete_test":"Implement a minimal PyTorch autograd trace of Algorithm 1 with the exact detector threshold (0.8), NMS (IoU 0.5), per-class counts, and Eq. (2), then backward from L_reward to the UNet parameters on a real batch and print the gradient norm; also print Aqc for the exact-match pair {person:4, skis:1}. If the reward gradient norm is zero (or backward errors) and Aqc != 1.0, the differentiable-SGD claim and the Eq. (2) quantity definition are both contradicted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing condition for the central claim is that the CQ Score reward is differentiable with respect to the diffusion model parameters (Section 3.3, Algorithm 1). The reward in Section 3.2 is computed from YOLOS outputs after hard confidence thresholding (0.8), NMS at IoU 0.5, and per-class box counts zni_b. Counts and the indicator I[zi_c in prompt labels] are discrete step functions of the image; the gradient of these terms with respect to the image is zero almost everywhere and undefined at discontinuities. The continuous confidence values pi_c are aggregated only over boxes that survive discrete decisions, so they do not provide a gradient for count changes. As written, running Algorithm 1 lines 11-18 with autograd would yield no nonzero reward gradient through the count terms; the paper gives no soft surrogate, straight-through estimator, or score-function estimator. Therefore the statement 'direct optimization like SGD can be adopted' is unsupported, and the claimed mechanism for improving quantity alignment is not established. Independently, Eq. (2) defines quantity confidence without matching detected classes to prompt classes: for the exact-match pair {person:4, skis:1}, it returns 0.625 instead of 1.0. This means even a perfectly aligned image is not maximal, and the reward cannot correctly credit category-specific counts. These are structural, testable defects, not matters of hyperparameter tuning.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a fine-tuning method for text-to-image diffusion models that uses object detection feedback to improve alignment on object categories and quantities. The method constructs a matching score, called CQ Score, from YOLOS detection outputs and a tokenized prompt, uses this score as a reward, and claims to fine-tune Stable Diffusion by backpropagating reward gradients. The authors also introduce a dataset of 1,700 prompt-image pairs and report improvements over SD v1.5, ImageReward, and DDPO on CLIP, BLIP, and CQ Score metrics.","tokens_in":14017,"tokens_out":5228,"duration_ms":44190,"significance":"If the proposed mechanism worked as described, the paper would address a genuine limitation in compositional text-to-image generation, and the release of a dataset and code would be useful to the community. The idea of using detection-based feedback for category and quantity alignment is attractive. However, the central technical claim—that the reward is differentiable and can be optimized with SGD—is not supported by the manuscript as written. The detection pipeline involves hard thresholding, non-maximum suppression, and discrete counts, which are non-differentiable operations. In addition, the primary evaluation metric is identical to the training reward, making the headline improvements circular. These issues are load-bearing rather than cosmetic, so the contribution as presented is not established.","major_comments":[{"comment":"The claim that the reward is differentiable and can be optimized with SGD is not supported. The reward r(x,z)=CQ Score is computed from YOLOS outputs after a hard confidence threshold of 0.8, NMS with IoU threshold 0.5, and integer box counts zni_b. These operations are piecewise constant functions of the image, so their gradients with respect to the diffusion model parameters are zero almost everywhere and undefined at the discrete transitions. Algorithm 1 (lines 11-18) performs the YOLOS forward pass and then computes Lreward, and line 18 applies ∇ω Lreward; however, no straight-through estimator, soft surrogate, or score-function estimator is described. Therefore the asserted learning signal for object quantity is absent, and the reported gains cannot be attributed to the proposed gradient-based fine-tuning.","section":"Section 3.3 / Algorithm 1"},{"comment":"The average quantity confidence Aqc does not check category identity. For a perfectly aligned image with detected outputs {person:4; skis:1} and the prompt 'four person and one skis', Eq. (2) gives Aqc = 0.625 rather than 1.0, because it averages min/max count ratios over all (detected class, prompt class) pairs. Consequently, a detection of a wrong category with the same count as a prompt category contributes to Aqc, and even a perfect image is not assigned the maximal score. This structural defect means the CQ Score cannot correctly credit category-specific counts and is not a valid alignment measure.","section":"Section 3.2, Eq. (2)"},{"comment":"The primary evaluation metric CQ Score is the same function used as the training reward (Section 3.2 defines r(x,z)=CQ Score). Reporting an improvement in CQ Score on a benchmark built from the same prompt distribution and evaluated with the same YOLOS detector is therefore circular. The non-CQ improvements over SD v1.5 (CLIP +7.02%, BLIP +12.93%) are comparatively small, and the paper does not report a train/test split, the number of evaluation prompts, or statistical significance. As a result, the abstract's claim of an average 11.2% improvement over SD v1.5 is not well supported by the evidence presented.","section":"Section 4.1 / Table 1"}],"minor_comments":[{"comment":"There are numerous typos and formatting errors, e.g., 'fine-turning' in the Abstract, 'sepecific' in Section 3.2, 'short for' should be 'shorted for' in Section 3.2, 'Monte Corlo' in Section 3.3, 'DDOP' on page 6, and 'bmy' in Algorithm 1 comments; these should be corrected.","section":"Throughout"},{"comment":"The cross-references are inconsistent: the text says 'As shown in Table 2' after presenting Table 1, and the caption for Figure 2 is not descriptive. In Appendix A.2, the method called 'GORS' is attributed to reference [16], but reference [16] is T2I-CompBench, not a method named GORS; the actual GORS work does not appear to be cited.","section":"Section 4.2 / Appendix A.2"},{"comment":"The dataset construction and evaluation protocol are underspecified: it is unclear whether the 1,700 image-text pairs are split into training and testing sets, and how the prompts for FID evaluation on MS-COCO are generated.","section":"Section 3.1 / Section 4.1"},{"comment":"The notation in Eq. (1) is confusing: p_i_c is defined as the sum of confidences over boxes of class i, but then p_i_c / zni_b is the average per-box confidence; the text describing 'total confidence score' should be clarified to avoid ambiguity.","section":"Section 3.2, Eq. (1)"},{"comment":"The claim that the model can serve as a metric for evaluating text-image alignment is not validated against human judgments or an independent benchmark; only a small qualitative comparison is provided.","section":"Section 4.3 / Section 5"}],"recommendation":"reject","confidential_remarks":"The stress-test concern lands: the differentiability claim is the core of the method and is invalid as written. The CQ Score circularity and the Eq. (2) category-matching error compound the problem. I recommend rejection. If the authors can reformulate the optimization with a legitimate gradient estimator and re-run experiments with an independent metric, a new submission might be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the positive: the paper proposes a genuinely new combination—use an off-the-shelf object detector to build a reward that rewards correct object categories and quantities, and fine-tune SD v1.5 with it. The dataset of 1.7K compositional prompts is a reasonable contribution, and the qualitative figures do show some improvement over ImageReward and DDPO on the examples shown. The idea is worth trying.\n\nThe soft spots are structural, not cosmetic. The paper claims the reward is differentiable and optimizes it with SGD (Section 3.3, Algorithm 1). But the reward is built from YOLOS outputs after a confidence threshold of 0.8, NMS at IoU 0.5, and discrete per-class box counts. Counts and indicator functions are step functions of the image; their gradient is zero almost everywhere. Nothing in the paper provides a surrogate, a straight-through estimator, or a score-function estimator. As written, Algorithm 1 lines 11-18 cannot produce a nonzero reward gradient through the quantity terms. So the mechanism that the paper says drives the improvement is absent from the text.\n\nSecond, the main evaluation metric, CQ Score, is exactly the reward used for training, and the benchmark was built with the same detection pipeline and the same prompt distribution. That makes the headline 11.2% improvement circular. The external CLIP/BLIP gains are small, reported without error bars, and could be explained by the regularizer or the dataset rather than the reward signal. The FID comparison is a single number.\n\nThird, Eq. (2) has an actual bug: it averages min/max over every detected class and every prompt class without aligning them by category. For a perfect match like {person:4, skis:1}, it gives 0.625, not 1.0. So the reward doesn't even credit a perfectly aligned image. That's testable and fixable, but it undermines the claim that CQ Score measures alignment.\n\nMinor: lambda, the reward weight, is not reported; the paper says code/dataset are public but no link works as of this reading; typos like 'fine-turning' and 'Tow horses' make the paper feel rushed.\n\nWho should read this: anyone working on reward-based fine-tuning for compositional text-to-image generation. The core idea deserves exploration, but this manuscript doesn't yet substantiate it. A serious referee should be engaged—the flaws are mechanical and correctable, not fatal to the problem. My recommendation: send it to review, but expect the authors to add a proper gradient estimator or a clear surrogate reward, fix Eq. (2), and report external metrics with variance on a held-out set.","headline":"A plausible detection-based reward idea, but the claimed differentiability path is absent and the main metric is the same as the training reward.","tokens_in":14641,"tokens_out":2495,"would_cite":false,"duration_ms":52926,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning diffusion models on detection-based feedback sharpens text-image alignment in categories and quantities","keywords":["text-to-image generation","diffusion models","feedback learning","reward fine-tuning","compositional generation","object detection","category-quantity alignment","CQ Score"],"falsifier":"Fine-tune the model with the quantity term Aqc removed from the reward and compare quantity accuracy against the full CQ Score; if quantity accuracy does not drop, the claimed quantity-feedback signal is not doing the work. Alternatively, compute the gradient of Aqc through the detection pipeline with respect to the denoising output and check whether it is nonzero at any sampled image.","tokens_in":13495,"feed_emoji":"🎯","tokens_out":4622,"duration_ms":38593,"temperature":0.7,"pith_summary":"The paper argues that text-to-image diffusion models can be aligned with prompts more effectively by fine-tuning on feedback that is specific to object categories and quantities, rather than on holistic similarity or human preference scores. It proposes a detection-based reward, the CQ Score, computed by comparing object detections from the generated image with quantity–category pairs parsed from the prompt, then fine-tunes Stable Diffusion v1.5 by backpropagating this reward through the last denoising step. On a new 1,700-pair compositional dataset, the method improves CLIP, BLIP, and CQ Scores over the base model by an average of 11.2%, and also improves FID. The paper further claims the reward is differentiable, enabling direct SGD optimization, and that the fine-tuned model can serve as a metric for evaluating other text-to-image generators.","feed_headline":"Detection-based feedback lifts text-image alignment by 11.2%","feed_subtitle":"A new CQ Score rewards diffusion models for getting object categories and counts right, not just overall similarity.","key_machinery":"The CQ Score is the harmonic mean of two terms: $\\text{Acc}$, the mean detection confidence per class for classes that appear in the prompt, and $\\text{Aqc}$, the mean over classes of $\\min(\\text{count}_{\\text{detected}}, \\text{count}_{\\text{prompt}}) / \\max(\\text{count}_{\\text{detected}}, \\text{count}_{\\text{prompt}})$. It is computed by running the YOLOS object detector on the generated image, filtering detections by confidence threshold 0.8 and non-maximum suppression with IoU 0.5, and pairing each detected class with a count; the prompt is parsed with the spaCy tokenizer to extract quantity–category pairs. The reward is applied only to the last denoising step, and the total loss is the pretraining denoising loss plus a reweighted reward term, so the gradient of the reward is propagated through the decoder and the last denoising step.","core_discovery":"The central claim is that a reward built from object-detection results—specifically the average category confidence and the average quantity confidence, combined through a harmonic mean—can fine-tune a text-to-image diffusion model to produce images that match prompts in both which objects appear and how many of each appear. The authors show that this focused feedback outperforms more generic rewards such as CLIP similarity, ImageReward, and DDPO on alignment metrics, with the largest gains in multi-category, multi-quantity compositions. The paper also states the reward is differentiable and can be optimized by directly backpropagating its gradient, and that the resulting model can be used to score other generators.","pith_inferences":["If the approach is right, alignment methods should shift from holistic similarity to attribute-specific feedback; the same detection-based reward could be adapted to other diffusion backbones or to other attributes such as spatial position, color, and size.","The differentiability claim is questionable given the discrete counting step; a robust implementation would need a soft or approximate differentiable count, or a policy-gradient treatment for the quantity term.","The model's utility as a metric is limited to categories detectable by the fixed detector and its COCO label set; prompts with out-of-vocabulary objects would receive zero category confidence.","The dataset's filtering by ImageReward may select images that are easier to align, so observed gains could partly reflect distribution shift rather than compositional skill."],"forward_implications":["Fine-tuning on specific detection-based feedback yields measurable alignment gains over the base Stable Diffusion v1.5 model and over ImageReward and DDPO across CLIP, BLIP, and CQ Score metrics.","The proposed model can be used as a metric to score text-image alignment of other generators, since its reward is a computable matching score.","The new 1,700-pair text-image dataset of category-quantity compositions provides a benchmark for evaluating compositional generation.","Applying the reward only at the last denoising step makes the fine-tuning memory-efficient relative to stepwise reinforcement learning approaches."],"supporting_citations":[{"why":"YOLOS is the off-the-shelf object detector used to produce class counts and confidence scores for the reward.","marker":"[9]"},{"why":"spaCy is the natural language parser that turns prompts into quantity-category pairs for matching.","marker":"[14]"},{"why":"Stable Diffusion v1.5 is the base text-to-image model being fine-tuned and the primary baseline.","marker":"[31]"},{"why":"ImageReward is used to filter the dataset and serves as a preference-based reward baseline.","marker":"[42]"},{"why":"DDPO is a reinforcement-learning baseline for fine-tuning with non-differentiable rewards.","marker":"[28]"},{"why":"CLIP score is one of the alignment metrics and represents a general similarity-based reward approach.","marker":"[13]"},{"why":"BLIP score is another alignment metric used in evaluation.","marker":"[24]"},{"why":"MS-COCO supplies the evaluation images for FID and is the training set for the detector.","marker":"[27]"}],"fun_headline_variants":["Detection-based feedback sharpens text-image alignment","New reward teaches diffusion models to count correctly","CQ score: fine-tuning with object detection feedback","Entity and quantity accuracy boost diffusion alignment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes the detection-based reward is differentiable with respect to the diffusion model's parameters, even though the count of detected objects per class is an integer produced by thresholding and non-maximum suppression, so the gradient of the quantity term is zero wherever the count does not change.","fun_headline_variants_meta":{"raw":{"variants":["Detection-based feedback sharpens text-image alignment","New reward teaches diffusion models to count correctly","CQ score: fine-tuning with object detection feedback","Entity and quantity accuracy boost diffusion alignment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000231,"raw_usage":{"total_tokens":1485,"prompt_tokens":945,"completion_tokens":540,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":484}},"tokens_in":561,"tokens_out":540,"duration_ms":5859,"temperature":1.0,"reasoning_tokens":484,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:36:43.004791+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune the model with the quantity term Aqc removed from the reward and compare quantity accuracy against the full CQ Score; if quantity accuracy does not drop, the claimed quantity-feedback signal is not doing the work. Alternatively, compute the gradient of Aqc through the detection pipeline with respect to the denoising output and check whether it is nonzero at any sampled image.","supporting_citations":[{"cited_title":"You only look at one sequence: Rethinking transformer in vision through object detection","cited_arxiv_id":null,"evidence_quote":"YOLOS is the off-the-shelf object detector used to produce class counts and confidence scores for the reward."},{"cited_title":"spaCy 2: Natural lan- guage understanding with Bloom embeddings, convolutional neural networks and incremental parsing","cited_arxiv_id":null,"evidence_quote":"spaCy is the natural language parser that turns prompts into quantity-category pairs for matching."},{"cited_title":"High-resolution image syn- thesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Stable Diffusion v1.5 is the base text-to-image model being fine-tuned and the primary baseline."},{"cited_title":"Imagere- ward: Learning and evaluating human preferences for text- to-image generation","cited_arxiv_id":null,"evidence_quote":"ImageReward is used to filter the dataset and serves as a preference-based reward baseline."},{"cited_title":"Training diffusion models towards diverse image generation with reinforcement learning","cited_arxiv_id":null,"evidence_quote":"DDPO is a reinforcement-learning baseline for fine-tuning with non-differentiable rewards."},{"cited_title":"Clipscore: A reference-free evaluation met- ric for image captioning","cited_arxiv_id":null,"evidence_quote":"CLIP score is one of the alignment metrics and represents a general similarity-based reward approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BLIP score is another alignment metric used in evaluation."},{"cited_title":"Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C","cited_arxiv_id":null,"evidence_quote":"MS-COCO supplies the evaluation images for FID and is the training set for the detector."}],"review_version":1}