{"id":"a5973055-ffc2-4671-a6cd-e426db277ffe","arxiv_id":"2505.00482","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"JointDiT is a Flux-based diffusion transformer that jointly generates RGB images and depth maps, and also handles depth estimation and depth-to-image generation by controlling the noise level of each branch.","lead":"JointDiT trains a diffusion transformer to generate an image and its matching depth map at the same time, building on the image-generation ability of the Flux model. The same model can also estimate depth from an image or generate an image from a depth map, by setting how much noise each branch receives.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'viable alternative to conditional generation' claim lacks a same-backbone conditional control; JointDiT's gains in Table 2 may stem from the Flux prior and joint pretraining, not from joint-distribution modeling.","rationale":"The reader's verdict is CONDITIONAL and I agree. My point of disagreement is which condition is most load-bearing. The reader's weakest_assumption points to Depth-Anything-V2 pseudo-labels (Section 4.2 / Supp. A.2). That is a genuine limitation: the depth branch cannot exceed the teacher on distribution-shifted benchmarks, and the ETH3D AbsRel gap is direct evidence. But the paper already acknowledges this and attempts a partial fix with synthetic GT fine-tuning; moreover, the pseudo-label choice is a data-supervision matter rather than a challenge to the joint-modelling mechanism. The missing conditional control, by contrast, targets the central conceptual claim directly. JointDiT outperforms conditional baselines in Table 2, but those baselines all share a weaker base architecture and different pretraining; without a same-backbone conditional arm the result is confounded. Table 6 and Figure 8 show the base prior alone has large effects on FID/IS, and Section 4.1 itself credits differences in image prior and architecture for geometric quality gains, so the same confound must be ruled out for the depth-conditioned comparison. The proposed check—a conditional-only Flux model trained on identical data—would settle whether the joint design, not the stronger prior, causes the gains. I therefore keep the conditional verdict and note that acceptance should require this control plus the artifact release already requested by the reader.","tokens_in":19993,"tokens_out":8375,"duration_ms":87055,"concrete_test":"Train a conditional depth-to-image model with the same Flux backbone and LoRA capacity as JointDiT, using depth maps concatenated or cross-attended at the same layers, on the same 16k PascalVOC split and BLIP2 prompts, then evaluate on OpenImages 6K under the Sec. 4.3 protocol. Compare FID and AbsRel against JointDiT; if the conditional-only Flux model matches or beats JointDiT, the central claim is unsupported, and if it is clearly worse, the joint mechanism is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—joint distribution modeling is a viable alternative to conditional generation—rests on JointDiT matching or beating conditional baselines on depth-conditioned image generation. Section 4.3 and Table 2 report lower FID and AbsRel than Readout-Guidance, ControlNet, and UniCon on the same PascalVOC training set and OpenImages evaluation. However, all three baselines use Stable Diffusion, while JointDiT uses Flux plus a parallel depth branch and was first joint-pretrained on 50k internal image-depth pairs before the PascalVOC fine-tuning described in Supplementary A.1. This entangles two factors: the base generative prior/architecture and the joint pretraining schedule. No experiment trains a depth-conditioned image generator with the same Flux backbone and the same LoRA budget, and no ablation removes the 50k joint-pretraining stage. Consequently, the improved numbers in Table 2 cannot be attributed to the joint-distribution modeling itself, which is the paper's headline conceptual contribution. The Depth-Anything-V2 pseudo-label issue is real but secondary: it constrains the depth-estimation demonstration and is partially acknowledged by the ETH3D results and the Ours+ft fine-tuning. The missing conditional control is the more load-bearing gap because it directly tests whether joint modeling, rather than a stronger base model, is doing the work.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes JointDiT, a diffusion transformer that models the joint distribution of RGB images and depth maps. The architecture extends a frozen Flux backbone with a parallel depth branch using LoRA adapters and joint cross-attention modules. Two techniques are introduced: adaptive scheduling weights, which modulate cross-modal information transfer according to the relative noise levels of the two branches, and an unbalanced timestep sampling strategy, which trains on both shared and independent timesteps per modality. By controlling the timesteps of each branch, the same network performs joint RGB-depth generation, depth estimation from an image, and depth-conditioned image generation. Experiments report strong joint generation quality, depth estimation comparable to diffusion-based depth estimators, and state-of-the-art depth-conditioned generation on the UniCon evaluation protocol. The central claim is that joint distribution modeling can serve as a viable alternative to conditional generation.","tokens_in":20249,"tokens_out":8163,"duration_ms":78851,"significance":"If the central claim is established, the paper offers a unified framework where one network handles several tasks through timestep control, which is a conceptually appealing direction for multi-modal diffusion models. The architecture is described in detail, and the ablation studies in Section 4.4 provide credible internal evidence that the two proposed techniques improve joint generation and depth estimation. The paper is generally clearly written and includes a limitation statement. However, the headline claim that joint distribution modeling can replace conditional generation is not supported by the present comparisons: the depth-conditioned generation baselines use a different and weaker base model (Stable Diffusion) than JointDiT (Flux), and no same-backbone conditional control is provided. In addition, the depth estimation evaluation lacks a precise description of the alignment protocol, which is essential for interpreting the AbsRel numbers. These gaps are fixable within the scope of a revision.","major_comments":[{"comment":"The claim that joint distribution modeling is a viable alternative to conditional generation is not established because the comparison in Table 2 does not control for the base generative model or the joint pretraining stage. Readout-Guidance, ControlNet, and UniCon are all based on Stable Diffusion, whereas JointDiT uses the Flux transformer backbone, adds a parallel depth branch with LoRA, and is first joint-pretrained on 50k image-depth pairs before the PascalVOC fine-tuning described in Supplementary A.1. The lower FID and AbsRel of JointDiT may therefore be due to the stronger Flux prior and the architecture rather than to joint-distribution modeling. Please add a depth-conditioned image generator with the same Flux backbone and the same LoRA budget that does not use joint pretraining, or run an ablation of JointDiT without the 50k joint-pretraining stage. Without such a control, the paper should not claim that joint modeling itself is responsible for the improvement.","section":"§4.3, Table 2; Abstract and Conclusion"},{"comment":"The depth estimation evaluation protocol is underspecified regarding the alignment of predicted depth to ground truth. JointDiT outputs normalized disparity, yet the reported AbsRel values (e.g., 5.7 on NYUv2) are only meaningful if some affine alignment (e.g., least-squares or median) to the ground truth is applied. The paper states that no ensemble technique is used but does not state whether or how alignment was performed for JointDiT or for the compared methods Marigold and GeoWizard. Without a precise statement of the alignment procedure, the numbers in Table 1 are not reproducible and the comparison to generative depth estimators is not interpretable. Please specify the exact evaluation protocol, including the alignment step, for all methods.","section":"§4.2, Table 1; Supplementary A.1"},{"comment":"The depth branch is trained on Depth-Anything-V2 pseudo-labels, and the synthetic fine-tuning data are rescaled to match DA-2's mean and standard deviation. The paper acknowledges that this teacher leads to a weak ETH3D result (AbsRel 16.5 vs 6.4 for GeoWizard), but the claim of comparable depth estimation on NYUv2, ScanNet, KITTI, and DIODE is still conditional on the teacher's quality. Because systematic errors in DA-2 predictions are likely inherited by JointDiT, the paper should discuss the potential bias of this supervision more thoroughly and, if possible, report a variant trained with a different pseudo-label source or elevate the Ours+ft results (with synthetic ground truth) as the primary evidence for the depth-estimation capability.","section":"§4.2 and Supplementary A.2"}],"minor_comments":[{"comment":"The adaptive scheduling weights wx and wy are undefined when tx + ty = 0, which corresponds to the initialization for joint generation. Please add a convention (e.g., define the value as 1/2 by limit) or add an epsilon to the denominator so the formula is well-defined for all inputs.","section":"§3.1, Eq. (6)"},{"comment":"The phrase \"With the same training dataset\" is potentially misleading because the compared methods also differ in base pretraining and in the joint-pretraining stage; please rephrase as \"with the same fine-tuning dataset\" or clarify the exact sense in which the training data are the same.","section":"§4.3, Table 2 caption"},{"comment":"In the depth-conditioned image generation ablation, the full model (both techniques) achieves FID 12.58, which is not the best among the four configurations (the \"neither\" configuration gives 11.94 and the \"unbalanced only\" configuration gives 12.51). The claim that the two techniques together are effective for this task rests primarily on the ImageReward ranking; please clarify why FID does not improve and whether human-preference ranking is the intended primary metric.","section":"Table 4"},{"comment":"The definition of f(t) is given, but g(t) is only described as f(t) with s=0.25; please write g(t) explicitly to avoid ambiguity.","section":"Supplementary A.3, Eq. (7)"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-executed and the proposed techniques are internally validated by the ablations, but the headline conceptual claim is likely to be contested by conditional-generation researchers because the comparison in Table 2 lacks a same-backbone conditional control. The missing alignment protocol for depth estimation is a reproducibility issue that should be fixed before publication. If the authors add the control experiment and clarify the evaluation, the paper could be a solid contribution to the multi-modal diffusion literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read: this is a competent, useful paper. The genuinely new part is taking a pre-trained Flux diffusion transformer, adding a parallel LoRA depth branch with joint cross-attention, and training with separate timesteps per modality, helped by two simple techniques: adaptive scheduling weights and unbalanced timestep sampling. The ablations in Tables 3 and 4 actually support both techniques; the components are individually known but the combination and the empirical finding that they improve separate-timestep joint modeling are new. The depth estimation numbers are credible for a joint model: better than JointNet and UniCon across the board, close to Marigold/GeoWizard on NYUv2 and ScanNet, and the fine-tuned version beats them on several datasets. The paper is also honest about its weakest spot: it attributes the poor ETH3D AbsRel (16.5 vs 6.4) to training on Depth-Anything-V2 pseudo-labels, and the limitation section discloses the 19.8% parameter increase and 2.9x slower sampling.\n\nWhere I part ways with the paper is the headline claim that joint distribution modeling 'can serve as a viable alternative to conditional generation.' Table 2 compares JointDiT against Readout-Guidance, ControlNet, and UniCon, all built on Stable Diffusion, while JointDiT uses Flux plus a joint pretraining stage on 50k internal pairs. That's two confounds at once. The better FID/AbsRel could come from the stronger Flux prior or the joint pretraining, not from the joint-distribution formulation itself. The paper even makes this point in Section 4.1 when explaining why its joint generation beats JointNet and LDM3D—it credits the Flux prior and architecture. You can't have it both ways. A same-backbone baseline—Flux with a ControlNet-style depth branch, or JointDiT without the 50k joint pretraining—would directly test the claim. That missing control is the load-bearing gap.\n\nThe pseudo-label issue is real but secondary. It caps the depth-estimation demonstration, and the paper says so. The synthetic fine-tuning in Supp A.2 re-scales ground truth to match DA-2's mean and std, which is a reasonable practical choice but means the 'fine-tuned' model is still anchored to the teacher's distribution.\n\nNo code or data released, which limits reproducibility, but the method is described in enough detail to reimplement.\n\nBottom line: this deserves a serious referee. I'd send it to review with a request for a same-backbone conditional baseline and a toned-down 'viable alternative' claim, or clear evidence that the joint formulation itself—not the base model—is doing the work. It's a solid submission, not a breakthrough.","headline":"Solid empirical paper extending joint RGB-depth diffusion to a Flux transformer with two simple, ablated techniques; the 'viable alternative' claim outruns the evidence because the comparison mixes base models.","tokens_in":20799,"tokens_out":1971,"would_cite":true,"duration_ms":19450,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"JointDiT models the joint distribution of RGB images and depth maps so that joint generation, depth estimation, and depth-conditioned image generation all become special cases of one flow-matching network controlled by two timesteps.","keywords":["joint distribution modeling","diffusion transformer","RGB-D","depth estimation","depth-conditioned image generation","flow matching","adaptive scheduling weights","timestep sampling"],"falsifier":"Replace the teacher-generated depth labels with ground-truth depth from the synthetic datasets at their original scale, retrain the same JointDiT pipeline, and compare absolute relative error on ETH3D; if the gap against the diffusion-based depth estimators does not shrink, then the paper's attribution of its ETH3D weakness to the teacher is not the explanation.","tokens_in":19740,"feed_emoji":"🖼️","tokens_out":8256,"duration_ms":76151,"temperature":0.7,"pith_summary":"This paper tries to establish that modeling the joint distribution of color images and depth maps is enough to solve three different tasks with a single network: generating an image together with its depth, estimating depth from an image, and generating an image from a given depth map. The proposed model, JointDiT, appends a parallel depth branch to a pre-trained diffusion transformer and trains the two branches with separate noise timesteps, so the task is selected at inference time by setting the timestep pair. Two training techniques are introduced to make that work: adaptive scheduling weights that let the noisier branch borrow structure from the cleaner branch, and an unbalanced timestep sampling strategy that covers extreme noise combinations. If the central claim is right, task-specific heads and per-task fine-tuning pipelines for depth estimation could be replaced by one shared generative model, which matters because depth estimation and depth-conditioned image generation are currently built as separate conditional models.","feed_headline":"Joint image-depth model handles three tasks by timestep alone","feed_subtitle":"JointDiT shows one frozen backbone, with timestep control, can replace task-specific heads for depth.","key_machinery":"The engine is a two-timestep flow-matching objective over paired modalities. Instead of one diffusion time, JointDiT samples independent timesteps $t_x$ and $t_y$ for the RGB and depth branches and minimizes the Joint Conditional Flow Matching loss, which forces the network to predict both components of the joint vector field at every noise-level combination. The joint connection module is a cross-attention layer between branches whose outputs are scaled by adaptive scheduling weights $w_x(t_x,t_y)$ and $w_y(t_x,t_y)$; each weight is a sigmoid of the relative noise imbalance, so the cleaner branch's structure is transferred more strongly to the noisier branch. The unbalanced timestep sampling strategy draws $t_x$ and $t_y$ independently from two different timestep distributions half of the time, and from the same distribution the other half, ensuring coverage of the extreme pairs needed for the conditional tasks.","core_discovery":"JointDiT's central claim is that a single flow-matching network can learn the joint distribution of an RGB image and its depth map over every combination of their noise levels, and that the resulting model is a viable replacement for conditional generation in depth tasks. The training objective is a joint conditional flow-matching loss with two independent timesteps, $t_x$ for the image and $t_y$ for the depth, so the vector field $v_{t_x,t_y}(x,y|x_1,y_1)$ is learned for all pairs. At inference, $(t_x,t_y)=(0,0)$ yields joint image-depth generation, $(1,0)$ gives depth estimation from a clean image, and $(0,1)$ gives depth-conditioned image generation. The architecture keeps a pre-trained diffusion transformer's image branch frozen, adds a parallel depth branch through low-rank adapters, and exchanges features between branches through joint cross-attention modules gated by adaptive scheduling weights. The paper reports high-fidelity joint generation with geometrically plausible 3D lifting, depth-estimation accuracy comparable to dedicated generative depth estimators on several benchmarks, and better depth-conditioned generation scores than the conditional baselines compared on the same training data.","pith_inferences":["Because the real-image training pairs use teacher-generated depth labels, the teacher's systematic errors set a ceiling on the model; a better teacher or a mix of ground-truth and synthetic depth should improve the conditional tasks, and the paper's own ETH3D gap is consistent with this.","The same two-timestep recipe should transfer to other paired modalities where one branch is structurally reliable, such as normals, segmentation, or albedo, and could be tested with identical adaptive scheduling and unbalanced sampling.","The joint model's ability to run at varying resolutions and without a text prompt suggests it could serve as a prior for inverse problems like depth completion or inpainting, which the paper does not explore.","The observed RGB-texture / depth-structure division of labor could be turned into a test: if joint generation is used to synthesize depth for out-of-domain illustrations, it should generalize better than monocular depth estimators, which the paper partly demonstrates but does not quantify."],"forward_implications":["One frozen image-generation backbone plus trainable adapters can serve as a unified model for joint generation, monocular depth estimation, and depth-conditioned image generation, with the task selected by timesteps instead of by network head.","Depth estimation from a single image no longer needs a task-specific architecture: with only a small fraction of trainable parameters, the joint model matches diffusion-based depth estimators on NYUv2 and ScanNet and improves on them after a short fine-tuning on synthetic data.","Jointly generated image-depth pairs lift to more coherent 3D point clouds than depth estimated from the already-generated image, and this advantage extends to stylized domains such as cartoon and pixel-art images.","In depth-conditioned image generation, the joint model provides lower FID and lower depth-consistency error than the conditional baselines compared on the same data.","Feature analysis shows the depth branch encodes structure while the RGB branch encodes texture, suggesting the joint model can serve as a geometric prior rather than only an appearance generator."],"supporting_citations":[{"why":"Supplies the frozen diffusion-transformer backbone whose image prior and global receptive field the depth branch is built on.","marker":"[7]"},{"why":"Contributes the joint cross-attention module design and serves as a baseline for conditional depth-to-image and depth estimation.","marker":"[35]"},{"why":"Provides the flow-matching formulation that the joint conditional flow matching loss extends to two timesteps.","marker":"[39]"},{"why":"Generates the depth labels used to train the depth branch from real images and as re-scaled supervision for synthetic fine-tuning.","marker":"[68]"},{"why":"Provides the low-rank adaptation method used to add the depth branch to the frozen backbone.","marker":"[27]"},{"why":"Baseline for joint image-depth generation and for depth estimation from a joint model.","marker":"[71]"},{"why":"Baseline generative depth estimator that the paper compares against on the depth benchmarks.","marker":"[32]"},{"why":"Baseline latent diffusion joint model compared for joint generation and 3D lifting quality.","marker":"[60]"},{"why":"Baseline conditional depth-to-image model compared on the OpenImages benchmark.","marker":"[72]"}],"fun_headline_variants":["One frozen backbone, three tasks, just set timesteps","JointDiT: timestep dials switch between joint, depth, and image tasks","Timesteps replace task heads in single RGB-depth diffusion model","Diffusion transformer does joint RGB-D, estimation, and conditioning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the depth maps produced by the pre-trained teacher predictor, normalized to the unit interval, are faithful enough to serve as ground truth for the depth branch, so any systematic error in those labels is inherited by JointDiT.","fun_headline_variants_meta":{"raw":{"variants":["One frozen backbone, three tasks, just set timesteps","JointDiT: timestep dials switch between joint, depth, and image tasks","Timesteps replace task heads in single RGB-depth diffusion model","Diffusion transformer does joint RGB-D, estimation, and conditioning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000254,"raw_usage":{"total_tokens":1580,"prompt_tokens":972,"completion_tokens":608,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":531}},"tokens_in":588,"tokens_out":608,"duration_ms":6185,"temperature":1.0,"reasoning_tokens":531,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:41:34.683385+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the teacher-generated depth labels with ground-truth depth from the synthetic datasets at their original scale, retrain the same JointDiT pipeline, and compare absolute relative error on ETH3D; if the gap against the diffusion-based depth estimators does not shrink, then the paper's attribution of its ETH3D weakness to the teacher is not the explanation.","supporting_citations":[{"cited_title":"Adding conditional control to text-to-image diffusion models","cited_arxiv_id":null,"evidence_quote":"Baseline conditional depth-to-image model compared on the OpenImages benchmark."},{"cited_title":"Flux.1.https://huggingface","cited_arxiv_id":null,"evidence_quote":"Supplies the frozen diffusion-transformer backbone whose image prior and global receptive field the depth branch is built on."},{"cited_title":"A sim- ple approach to unifying diffusion-based conditional gener- ation","cited_arxiv_id":null,"evidence_quote":"Contributes the joint cross-attention module design and serves as a baseline for conditional depth-to-image and depth estimation."},{"cited_title":"Depth any- thing v2.Advances in Neural Information Processing Sys- tems, 37:21875–21911, 2025","cited_arxiv_id":null,"evidence_quote":"Generates the depth labels used to train the depth branch from real images and as re-scaled supervision for synthetic fine-tuning."},{"cited_title":"Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022","cited_arxiv_id":null,"evidence_quote":"Provides the low-rank adaptation method used to add the depth branch to the frozen backbone."},{"cited_title":"Joint- net: Extending text-to-image diffusion for dense distribution modeling","cited_arxiv_id":null,"evidence_quote":"Baseline for joint image-depth generation and for depth estimation from a joint model."},{"cited_title":"Repurpos- ing diffusion-based image generators for monocular depth estimation","cited_arxiv_id":null,"evidence_quote":"Baseline generative depth estimator that the paper compares against on the depth benchmarks."}],"review_version":1}