{"id":"3b3e5d43-c47b-4a3b-8dda-ae116925c571","arxiv_id":"2504.19860","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CoherenDream injects MLLM-based semantic feedback into the SDS loop and improves text-3D alignment on multi-object prompts.","lead":"This paper adds feedback from a multimodal AI critic into score distillation, the optimization step that turns text prompts into 3D models, reducing missing objects and broken spatial relations. It reports higher text-faithfulness scores than prior text-to-3D generators on a small benchmark and could make 3D asset creation from natural language more reliable.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eqs. (5)-(6) substitute the cross-entropy gradient of an uncalibrated MLLM on one-step-denoised renderings for the text-alignment score ∇log p(y|x_t); no calibration or control shows this gradient points toward the user prompt at SDS noise levels.","rationale":"The reader's weakest assumption identifies the same load-bearing gap: the MLLM cross-entropy loss is used as a differentiable proxy for text-alignment score without calibration. I agree that this is the central vulnerability. The paper does provide real supporting evidence that the feedback term is doing something: the TCSD-vs-SDS ablation in Fig. 5, the guidance-task ablations in Table 4, and the qualitative comparisons all show object omission being reduced. Those results are credible engineering evidence and should count in the paper's favor. However, they do not distinguish a calibrated alignment-score correction from an uncalibrated heuristic that happens to nudge the scene toward object presence—which is exactly what TIFA's object/relation questions reward. The fixed λ values and the absence of any gradient-norm analysis mean the sign and scale of δ are not justified by the score-matching derivation. Secondary issues such as the circular GPT-4o-based critic validation and the lack of error bars on the 45-prompt subset reinforce the need for verification but are not as load-bearing as the unvalidated score substitution. Since the reader already set CONDITIONAL and our concern supports that stance rather than moving it to accept or reject, the verdict should remain unchanged pending the proposed calibration/control experiment.","tokens_in":19548,"tokens_out":8045,"duration_ms":92599,"concrete_test":"At fixed CoherenDream checkpoints and for the 45 TIFA prompts, compute x̂_0 and the TCSD correction δ(T, x̂_0) at t values spanning the SDS schedule (e.g., t=0.9, 0.7, 0.5, 0.3). Measure the average cosine similarity between δ(T, x̂_0) and the gradient of an independent, human-aligned text-image reward (e.g., ImageReward or HPS v2) on the same x̂_0. If the average cosine is below roughly 0.3, Eq. (5) is not a faithful surrogate for ∇ log p(y|x_t). Complement this with a control run in which the prompt-to-T label mapping is shuffled while keeping all hyperparameters fixed; if the TIFA gain over MVDream survives shuffling, the semantic content of δ is not the cause of the reported improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The whole TCSD construction rests on Eq. (5)-(7), but the replacement of ∇_x_t log p(y|x_t) by λ·∇_x_t Lce(T, f_cr(x̂_0)) is asserted, not derived or calibrated. Lce is the autoregressive token loss of a LLaVA fine-tuned to emit scene-graph, object, and view labels from 2×2 grids; nothing makes that loss a calibrated estimate of the probability that x_t is aligned with the user prompt. Moreover, x̂_0 is obtained by one-step denoising, while the critic's training data were generated with 5/20/50 denoising steps, so at the high t values used early in SDS, x̂_0 is far outside the critic's training distribution. The guidance weights λ_sg=1, λ_ob=0.5, λ_view=0.1 are fixed without a gradient-magnitude or λ-sensitivity ablation, so the added term could dominate or be dominated by the SDS noise prediction. The paper's own failure cases in Fig. 19 (attribute leakage, missed 'reclined position') show critic biases leaking into the output. Consequently, the central claim that TCSD corrects view-independent bias is not established by the derivation; the reported TIFA gains may reflect an uncalibrated heuristic that rewards object presence rather than a faithful alignment-score correction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CoherenDream, a text-to-3D generation framework that augments Score Distillation Sampling (SDS) with feedback from a multimodal large language model (MLLM). The central contribution is Textual Coherent Score Distillation (TCSD), which modifies the SDS noise prediction as \\hat{\\epsilon}_\\Phi = \\epsilon_\\Phi + \\lambda\\cdot\\delta(T,\\hat{x}_0), where \\delta is the gradient of an MLLM cross-entropy loss evaluated on one-step denoised renderings against scene-graph, object, and view targets. The authors fine-tune a 3DLLaVA-CRITIC model on a view-aware dataset generated with GPT-4o, and they introduce an LLM-layout initialization that warms up the NeRF density field. Experiments on a 45-prompt TIFA subset report improved TIFA, VQAScore, and CLIP scores over MVDream, DreamView, and JointDreamer, with additional ablations, comparisons to compositional methods and native 3D models, and integration experiments with JSD and VSD.","tokens_in":19941,"tokens_out":5456,"duration_ms":49275,"significance":"If the central mechanism is validated, integrating MLLM feedback directly into SDS is a promising and timely direction that could mitigate the view-independent bias accumulation in multi-object text-to-3D generation. The paper appears to be the first to embed an MLLM in the SDS optimization loop, and the view-aware data collection pipeline, the decomposition of alignment feedback into scene-graph, object, and view tasks, and the extensive qualitative comparisons are useful contributions. The compatibility of TCSD with JSD and VSD demonstrates that the approach is modular and could be adopted by other SDS variants. However, the current evidence does not establish that the added gradient is a faithful alignment-score correction: the derivation is heuristic, the critic validation is circular, and the headline quantitative evaluation is based on a small prompt subset with nonstandard aggregation and no error bars. The paper's own failure cases further show that the MLLM feedback can inject attribute biases, so the central claim requires additional calibration and controlled experiments.","major_comments":[{"comment":"The substitution of \\nabla_{\\mathbf{x}_t}\\log p(y|\\mathbf{x}_t) by \\lambda\\cdot\\nabla_{\\mathbf{x}_t} L_{\\mathrm{ce}}(T, f_{\\mathrm{cr}}(\\hat{\\mathbf{x}}_0)) is asserted rather than derived or calibrated. The cross-entropy of an autoregressive MLLM is a token-prediction loss on 2x2 grid images, not a calibrated estimate of the probability that \\mathbf{x}_t is aligned with the user prompt. Moreover, \\hat{\\mathbf{x}}_0 is obtained by one-step denoising, whereas the critic's training images were generated with 5/20/50 denoising steps (Appendix, \"View-aware Image Generation Details\"), so at the large timesteps used early in SDS the critic operates far outside its training distribution. I request either a derivation of this replacement with error bounds, or a calibration experiment showing that L_{\\mathrm{ce}} correlates with human-judged text alignment on SDS-like noisy renderings, and a control experiment comparing optimization trajectories with and without the MLLM term.","section":"Textual Coherent Score Distillation (TCSD), Eqs. (4)-(7)"},{"comment":"The headline improvement is based on a curated 45-prompt subset of TIFA v1.0 with no error bars, confidence intervals, or significance tests. The aggregation rules are nonstandard and differ across metrics: TIFA is the intersection of correct answers across 10 uniformly sampled azimuths, VQAScore is the maximum across viewpoints, and CLIP is the average across views. These choices can systematically favor different failure modes; for example, max-aggregated VQAScore rewards methods that occasionally produce a single good view, while intersection-based TIFA penalizes view inconsistency. The paper should justify these aggregation choices, report per-prompt scores and variance, and ideally evaluate on the full TIFA set or another standardized benchmark to establish that the observed 4.0-point TIFA improvement is not an artifact of prompt selection or aggregation.","section":"Quantitative Comparison, Table 1"},{"comment":"The validation of the critic is circular: GPT-4o generated the fine-tuning labels in the view-aware data collection pipeline, and GPT-4o also scores the critic's responses as correct or incorrect in the reported validation. The accuracies (T_sg 0.76, T_ob 0.89, T_view 0.87) therefore measure agreement with the same labeler that created the training data, not true accuracy on an independent ground truth. Since the critic's feedback directly steers the SDS gradient, an independent human evaluation on a sampled subset, or a held-out set annotated by a different source, is necessary to trust the TCSD results and to determine whether the critic's errors introduce biases into the generated 3D assets.","section":"Quantitative evaluation for 3DLLaVA-CRITIC, Table 2 and Appendix \"Quality of 3DLLaVA-CRITIC\""},{"comment":"The task weights \\lambda_{\\mathrm{sg}}=1, \\lambda_{\\mathrm{ob}}=0.5, and \\lambda_{\\mathrm{view}}=0.1 are fixed without a sensitivity analysis or any comparison of gradient magnitudes between the SDS noise term \\epsilon_\\Phi and the MLLM feedback term \\lambda\\cdot\\delta. The paper also notes that view classification slows training and is therefore computed only every 20 iterations, which changes the effective weight of that task across time. Without ablations that vary these weights, or at least report the norm of the feedback gradient relative to the SDS gradient during optimization, it is unclear whether the MLLM term dominates, is negligible, or is balanced at different phases of training; the reported improvements could be highly sensitive to these ad-hoc values.","section":"Guidance Task Combo, Eq. (7)"},{"comment":"The acknowledged failure cases, attribute leakage (the \"green\" from the stool influencing the monkey) and missed attributes (the pig's \"reclined position\" not rendered), show that the MLLM feedback can inject biases rather than consistently correct view-independent drift. This directly qualifies the central claim that TCSD \"steers the optimization toward a textual-consistent distribution.\" The paper should analyze why these biases arise, for example whether the scene-graph format over-weights object presence and under-weights attributes and relations, and should propose a mitigation or at least quantify the frequency of such failures across a larger prompt set.","section":"Limitation and Fig. 19"}],"minor_comments":[{"comment":"In the Appendix, \"1DreamView\" should be \"DreamView\", and in the qualitative comparison section \"our Conherent\" should be \"our CoherenDream\".","section":"Baselines Setup"},{"comment":"In \"Guidance Tasks\", \"equipe\" should be \"equip\", and in \"LLM-layout Initialization\", \"traption\" should be \"trapping\"; in Fig. 6, \"TCDC\" should be \"TCSD\".","section":"Framework of CoherenDream"},{"comment":"The notation E_{t,\\epsilon_\\Phi} is ambiguous because the sampling distributions of t and \\epsilon are not defined; please specify the noise schedule, the range of t, and the distribution of \\epsilon.","section":"Preliminaries, Eq. (2)"},{"comment":"The symbol \\lambda is used both for \\sqrt{1-\\alpha_t} in Eq. (5) and for the task weights \\lambda_{\\mathrm{sg}}, \\lambda_{\\mathrm{ob}}, \\lambda_{\\mathrm{view}}; please use a distinct symbol for the task weights to avoid confusion.","section":"Eq. (5) and Guidance Task Combo"},{"comment":"The model name appears variously as \"3DLLaV A-CRITIC\", \"3DLLaVA-CRITIC\", and \"3DLlava-CRITIC\"; please standardize, and the reference list entry for LLaVA-OneVision is incomplete.","section":"References and notation"},{"comment":"The example JSON in the layout generation prompt contains syntax errors, including double braces and a trailing comma, which would not parse; please fix the example.","section":"Fig. 16, layout prompt"},{"comment":"The paper refers to a \"curated 45-prompt subset\" of TIFA v1.0 but does not specify the curation procedure or list the prompts; adding a supplementary prompt list would aid reproducibility.","section":"Quantitative Comparison"},{"comment":"The heading \"Quantity ablation study\" should be \"Quantitative ablation study\", and the object classification prompt in Table 3 lacks an output format example, making it hard to reproduce the model's structured output.","section":"Table 4 in Appendix"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and important problem, and the idea of injecting MLLM feedback into SDS is novel and potentially impactful. However, the central gradient substitution in Eq. (5)-(7) is a heuristic that needs calibration or controlled validation, the critic evaluation is circular, and the quantitative evidence rests on a small, nonstandard evaluation. With a calibration study, error bars, human evaluation of the critic, and a sensitivity analysis of the guidance weights, the paper could be suitable for publication; in its current form the central claim is not fully established. I recommend major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nTwo things to know. The core idea—adding a fine-tuned MLLM's cross-entropy loss to the SDS gradient for 3D generation—is a real novelty in the 3D literature. And the paper backs it with solid engineering. But the gradient term is a heuristic, not a calibrated score, and the quantitative evidence is narrower than the title suggests.\n\nWhat is genuinely new: CoherenDream is the first to put MLLM feedback directly inside SDS optimization for 3D. The authors build a view-aware critic (3DLLaVA-CRITIC) by fine-tuning LLaVA-OneVision on 307k grid images that simulate SDS-like degraded renderings. The three guidance tasks—scene graph, object classification, view classification—are well chosen. The LLM-layout warm-up is a nice practical addition. Ablations are thorough: they isolate each task, show scene graph is the most important, and demonstrate that TCSD can be plugged into JSD and VSD. They also report honest failure cases.\n\nSoft spots. Equations (5)-(7) substitute λ·∇ L_CE on one-step-denoised images for ∇ log p(y|x_t). The stress-test note is correct: no calibration or control shows this gradient points at the user prompt at SDS noise levels. The fixed weights (λ_sg=1, λ_ob=0.5, λ_view=0.1) are not swept. The one-step estimate is likely out-of-distribution for the critic early in optimization. The critic validation is circular: GPT-4o generates the fine-tuning labels and also judges the critic's outputs. That does not invalidate the TIFA numbers, which use an external benchmark, but it overstates the critic's objective quality. The main comparison is 45 curated prompts, no error bars, and no code or weights released, so the 4-point TIFA gain lacks statistical grounding.\n\nThese are real weaknesses, but not fatal. The paper does not claim a rigorous derivation; it presents an engineering mechanism and demonstrates it works on several metrics. My main request to the authors would be: add a λ-sensitivity study, show the correlation between the critic loss and human judgment at different noise levels, release code, and run the evaluation on a larger prompt set.\n\nThis is a paper for people working on text-to-3D alignment and SDS variants. It deserves a serious referee and likely publication after moderate revision. Send it out.","headline":"Real novelty in MLLM-guided SDS for 3D, with solid ablations; but the gradient derivation is heuristic and the evaluation is too thin for the claims—still deserves review.","tokens_in":20453,"tokens_out":4181,"would_cite":true,"duration_ms":41389,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Feeding multimodal-LLM text-alignment feedback into score distillation as a correction term fixes object omissions and relational errors in text-to-3D generation.","keywords":["text-to-3D generation","score distillation sampling","multimodal large language models","textual coherence","scene graph generation","view classification","LLM-layout initialization","multiview consistency"],"falsifier":"Take a fixed 3D scene whose prompt and renderings are deliberately mismatched, for example a prompt naming two macaws playing chess but a scene containing one macaw and no chessboard; one-step denoise renderings across a range of noise levels and check whether the TCSD correction $\\delta(T,\\hat{x}_0)$ points in the same direction as a trusted alignment score or human judgment. Systematic sign disagreement at high noise would show the feedback proxy is unreliable there.","tokens_in":19346,"feed_emoji":"🎯","tokens_out":12135,"duration_ms":100610,"temperature":0.7,"pith_summary":"Score Distillation Sampling (SDS) turns a 2D diffusion model into a 3D generator, but when the prompt names several objects with relations, the per-view optimization drifts: objects get dropped and spatial relations go wrong. The paper tries to fix this drift by feeding semantic feedback from a multimodal large language model (MLLM) directly into the SDS gradient as a correction term. The proposed objective, Textual Coherent Score Distillation (TCSD), adds $\\lambda\\,\\delta(T,\\hat{x}_0)$ to the diffusion noise prediction, where $\\delta$ is the gradient of the critic's cross-entropy loss on one-step-denoised renderings against prompt-derived scene graphs, object lists, and view labels. The authors also fine-tune a critic they call 3DLLaVA-CRITIC on diffusion-noisy multiview grids and warm the 3D representation up with an LLM-chosen layout. On a 45-prompt TIFA subset, CoherenDream improves the TIFA score from 77.4 to 81.4 relative to the strongest multi-view SDS baseline, with gains also on VQAScore and CLIP.","feed_headline":"A multimodal critic cuts object omissions in 3D generation","feed_subtitle":"A critic that checks objects, relations, and views during optimization lifts TIFA from 77.4 to 81.4.","key_machinery":"The load-bearing object is the textual-consistent noise prediction $\\hat{\\epsilon}_\\Phi=\\epsilon_\\Phi+\\lambda\\,\\delta(T,\\hat{x}_0)$, where $\\delta(T,\\hat{x}_0)=\\nabla_{x_t}L_{\\mathrm{ce}}(T,f_{\\mathrm{cr}}(\\hat{x}_0))$ is the gradient of an MLLM critic's cross-entropy loss against prompt-derived targets on a one-step-denoised rendering. It turns a language-model judgment of which objects, relations, and views are present into a differentiable correction to the SDS update direction. The critic is decomposed into three question-answering tasks (scene graph generation, multi-label object classification, and view classification) so the feedback can target global semantics, missing objects, and geometry separately. The auxiliary LLM-layout initialization plays a supporting role by fitting the density field's occupancy to a rough box layout during the first hundreds of steps, so the diffusion prior starts from a semantically plausible arrangement.","core_discovery":"The central claim is that the accumulated view-independent bias of SDS can be corrected online by an MLLM that reads the current renderings and reports how faithfully they answer prompt-derived questions. Writing the ideal distribution as $\\hat{p}_t(x_t\\mid y)=p_t(x_t\\mid y)\\,p(y\\mid x_t)$ and using the score-function identity $\\nabla_{x_t}\\log p_t(x_t\\mid y)=-\\epsilon_\\Phi/\\sqrt{1-\\alpha_t}$, the paper replaces the unknown term $\\nabla_{x_t}\\log p(y\\mid x_t)$ with $\\lambda\\,\\nabla_{x_t}L_{\\mathrm{ce}}(T,f_{\\mathrm{cr}}(\\hat{x}_0))$ evaluated on the one-step denoising $\\hat{x}_0$, yielding the corrected noise prediction $\\hat{\\epsilon}_\\Phi=\\epsilon_\\Phi+\\lambda\\,\\delta(T,\\hat{x}_0)$ in Eq. (7). The correction is computed for three guidance tasks: scene graph generation, multi-label object classification, and view classification, and it is what lets the optimization steer toward a distribution in which all named objects and their relations appear in a holistic three-dimensional representation. The paper further claims that fine-tuning the critic on view-aware, diffusion-noisy grid images and initializing the scene from an LLM-generated layout make this feedback accurate enough to produce text-coherent results that a vanilla SDS baseline misses.","pith_inferences":["The TCSD correction is effectively a differentiable semantic reward, so the same one-step-denoising trick could transfer to image-to-3D or test-time prompt-adherence settings where a critic can be queried on intermediate outputs.","Because $\\hat{x}_0$ gets blurrier at high noise levels, the reliability of $\\delta$ should degrade there; annealing $\\lambda$ by noise level or scheduling which guidance tasks fire at which timesteps is a natural testable improvement.","Comparing the direction of the TCSD correction with a trusted reference alignment signal (such as human preference ratings) would diagnose how much the critic's loss can be trusted; the paper's own failure cases show attribute leakage when prompt supervision is weak.","The view-aware data collection pipeline consumes only rendered grids, so the same critic recipe could be reused for Gaussian-splatting or mesh-based 3D generators."],"forward_implications":["TCSD can be applied on top of existing SDS variants: integrating it with JointDreamer fixes omitted objects, and combining it with variational score distillation adds texture fidelity while keeping prompt alignment.","Because the correction acts on the noise prediction rather than replacing the diffusion model, it preserves the base model's generalization and avoids fine-tuning a diffusion model on 3D datasets.","The three-task feedback decomposition gives a practical recipe for using MLLMs as optimizers, not just evaluators, in generative 3D pipelines.","Using TCSD with a holistic NeRF representation yields more faithful object interactions than compositional methods that optimize objects separately.","CoherenDream's training time stays comparable to the baseline (about 60 minutes), so the semantic feedback adds little computational overhead."],"supporting_citations":[{"why":"Defines Score Distillation Sampling, the objective that TCSD modifies.","marker":"Poole et al. 2023"},{"why":"Provides the multi-view diffusion model used as the base SDS prior and as the main comparison baseline.","marker":"Shi et al. 2024"},{"why":"Base multimodal LLM architecture that is fine-tuned into 3DLLaVA-CRITIC.","marker":"Li et al. 2024a"},{"why":"Supplies the score-matching identity connecting noise prediction to the score of the image distribution, used in the TCSD derivation.","marker":"Song and Ermon 2019"},{"why":"Baseline for text-consistent generation whose joint score distillation TCSD is shown to improve.","marker":"Jiang et al. 2025"},{"why":"Variational score distillation baseline used to show TCSD combines with texture-fidelity-oriented SDS variants.","marker":"Wang et al. 2024"},{"why":"Inspires the LLM-layout generation used both for 3DLLaVA-CRITIC training data and for initialization.","marker":"Feng et al. 2024a"},{"why":"Provides GPT-4o, used to annotate the 307,409 grid images and to generate layout prompts.","marker":"OpenAI 2023"},{"why":"TIFA defines the question-answering evaluation protocol and the subset used for the main metrics.","marker":"Hu et al. 2023"}],"fun_headline_variants":["MLLM critic corrects SDS bias for coherent 3D","CoherenDream steers SDS with multimodal feedback","Online MLLM guidance lifts 3D text alignment","Critic feedback fixes object omissions in 3D scenes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The correction term assumes the fine-tuned MLLM's cross-entropy loss on one-step-denoised renderings is a faithful, properly scaled differentiable proxy for how well the prompt matches the image, so a miscalibrated critic could push the 3D scene away from user intent.","fun_headline_variants_meta":{"raw":{"variants":["MLLM critic corrects SDS bias for coherent 3D","CoherenDream steers SDS with multimodal feedback","Online MLLM guidance lifts 3D text alignment","Critic feedback fixes object omissions in 3D scenes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000196,"raw_usage":{"total_tokens":1431,"prompt_tokens":1087,"completion_tokens":344,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":703,"completion_tokens_details":{"reasoning_tokens":277}},"tokens_in":703,"tokens_out":344,"duration_ms":3347,"temperature":1.0,"reasoning_tokens":277,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:41:58.596384+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fixed 3D scene whose prompt and renderings are deliberately mismatched, for example a prompt naming two macaws playing chess but a scene containing one macaw and no chessboard; one-step denoise renderings across a range of noise levels and check whether the TCSD correction $\\delta(T,\\hat{x}_0)$ points in the same direction as a trusted alignment score or human judgment. Systematic sign disagreement at high noise would show the feedback proxy is unreliable there.","supporting_citations":[{"cited_title":"T.; and Mildenhall, B","cited_arxiv_id":null,"evidence_quote":"Defines Score Distillation Sampling, the objective that TCSD modifies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the multi-view diffusion model used as the base SDS prior and as the main comparison baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Baseline for text-consistent generation whose joint score distillation TCSD is shown to improve."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Variational score distillation baseline used to show TCSD combines with texture-fidelity-oriented SDS variants."}],"review_version":1}