{"id":"9365be7e-8d6c-4b7b-bfcc-ceb879156ed8","arxiv_id":"2412.06322","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLaVA-SpaceSGG is a visual instruction-tuned model that uses a new 2D+3D spatial scene graph dataset to improve open-vocabulary scene graph generation and spatial relation accuracy.","lead":"This paper trains a large multimodal model on a new dataset that adds depth-based spatial relations to scene graph generation, and reports improved recall on the PSG benchmark. It shows that synthetic spatial question-answer data can push open-vocabulary scene graph models toward better spatial reasoning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The spatial-relation benchmark is generated with the same pipeline as the training data, with a ~5-question margin over ASMv2 and a below-random ablation row; the spatial advantage is not yet independently established.","rationale":"The paper's distinctive contribution is enhanced spatial relation modeling, so the validity of the spatial relation validation set is load-bearing for the central claim as a whole. Section 4.1 explicitly reuses the Section 3.1.4 generation pipeline for the validation set, and the paper does not state that the 30 COCO-Val images are disjoint from SpaceSGG training images. The reported advantage over ASMv2 is only about 5 correct answers out of 271, well within plausible noise for a metric with no reported variance. The 1.47% accuracy for ab-train-1 on a four-choice task strongly suggests the evaluation procedure can produce meaningless scores, which further undermines the spatial benchmark as a measurement instrument. By contrast, the PSG recall improvement is measured against external PSG annotations and is a more credible result; therefore the appropriate response is to keep the conditional verdict and require an independent spatial evaluation rather than to reject the whole paper. An independent human-labeled pass on the same questions, with a training-overlap check and a significance test, would directly settle whether the 52.48% reflects spatial competence or alignment with the pipeline's output distribution.","tokens_in":18132,"tokens_out":10018,"duration_ms":105481,"concrete_test":"Take the 30 validation images, exclude any that appear in the SpaceSGG training set, and have two independent annotators answer the 271 questions from the raw images only, without seeing pipeline outputs; use their consensus labels as ground truth. Then recompute LLaVA-SpaceSGG and ASMv2 accuracies on those consensus labels and run a McNemar test. If the 52.48% vs 50.52% gap disappears, reverses, or fails significance, the spatial-relation claim is not confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 states that the spatial relation validation set is built from 30 COCO-Val-2017 images with QA generated 'using the data generation pipeline in Section 3.1.4' — the same depth/layering plus LLM question-generation pipeline used to create SpaceSGG training data. No disjointness between those 30 images and SpaceSGG training images is stated. The main spatial result is 52.48% vs ASMv2's 50.52% on 271 questions, a difference of about 5 questions, with no error bars or significance test. More troubling, Table 3's ab-train-1 (no SpaceSGG) scores 1.47% accuracy, far below the 25% random baseline, indicating the multiple-choice evaluation is not robustly measuring spatial ability, or the output parsing is broken for that model. If the human-corrected labels inherit the pipeline's depth/layering bias, or if validation images overlap training, the 52.48% mainly shows the model learned the pipeline's output distribution rather than true 3D spatial understanding. The PSG recall claim is externally grounded and less affected, but the paper's distinctive spatial-relation contribution rests on this validation set.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LLaVA-SpaceSGG, a multimodal LLM for open-vocabulary scene graph generation with enhanced spatial relations. It introduces the SpaceSGG instruction-tuning dataset, built from public datasets plus a synthetic data pipeline that combines 2D object detection, monocular depth estimation (DepthAnything), camera calibration, point-cloud layering, and LLM-generated descriptions, QA, and conversations. The model is trained with a two-stage paradigm and evaluated on the PSG validation set and on a self-constructed spatial relation validation set. The main reported results are Recall 15.43 vs. ASMv2's 14.2 and mRecall 13.23 vs. 10.3 on PSG, and 52.48% accuracy on the spatial validation set.","tokens_in":18298,"tokens_out":3158,"duration_ms":32480,"significance":"If the PSG results are reproducible, the paper makes a useful contribution: it demonstrates that adding depth-derived spatial instruction data can improve open-vocabulary triplet recall and mean recall for an MLLM-based SGG method, and it releases code, data, and models. The PSG evaluation is externally benchmarked and not circular. The main weakness is that the distinctive spatial-relation claim rests entirely on a self-constructed validation set generated by the same pipeline as the training data, with no error bars, no stated image disjointness, and a very small performance margin. The paper's central PSG claim is credible, but the spatial-relation contribution needs stronger independent evidence before the broader claims can be accepted.","major_comments":[{"comment":"The spatial relation validation set is built by randomly selecting 30 images from COCO-Val-2017 and generating QA with the same data generation pipeline (DepthAnything, camera calibration, LLM prompting) used for SpaceSGG training data. The manuscript does not state that the 30 validation images are disjoint from the SpaceSGG training images, nor does it quantify noise in the manual correction step. With only 271 questions, the reported 52.48% vs. 50.52% difference is approximately five questions, and no error bars or significance test are provided. Because this benchmark is the sole support for the spatial-relation contribution, the spatial claim is not yet independently established. Please evaluate on an external or independently human-annotated spatial benchmark, report confidence intervals or repeated runs, and state the image disjointness explicitly.","section":"§4.1, §3.1.4"},{"comment":"The ablation without the SpaceSGG dataset reports 1.47% accuracy on a four-choice spatial benchmark, which is far below the 25% random-choice baseline. This strongly suggests that the free-form answer evaluation or output parsing is not robust for that model configuration, or that the benchmark is not measuring spatial ability in a controlled way. Since the same evaluation protocol produces the headline 52.48% result, this discrepancy must be explained and the evaluation protocol must be fixed (for example, with constrained decoding or manual grading).","section":"Table 3, ab-train-1"},{"comment":"Rows ab-data-9 and ab-data-10 report Recall=0, mRecall=0, and Accuracy=0. These values are implausible as genuine model performance and are not discussed in the text. If these runs failed to converge or the evaluation failed, they cannot support the placebo-ablation conclusion; if the numbers are real, the cause must be explained with training details. As written, these rows undermine the supplementary ablation evidence.","section":"Table 6, ab-data-9 and ab-data-10"}],"minor_comments":[{"comment":"The prompt text contains a typo: 'Qeustion' should be 'Question'. Please proofread the figure text.","section":"Figure 4"},{"comment":"The algorithm title says 'Devide objects into layers'; 'Devide' should be 'Divide'. The pseudo-code also uses the phrase 'covered by' without a precise definition in the algorithm; please clarify how coverage is determined from the depth ranges and bounding boxes.","section":"Algorithm 2"},{"comment":"The model name is written inconsistently as 'LLaVA-SpaceSGG' and 'LLaV A-SpaceSGG', and the baseline is sometimes 'ASMv2' and sometimes 'AS-V2'. Please standardize the notation.","section":"Throughout"},{"comment":"The text states that results 'vary minimally' across generative models, but Table 5 shows GPT-4o giving 53.725% spatial accuracy vs. 52.48% for Llama 3, and mRecall ranging from 9.53 to 13.23. Please either provide a statistical or practical significance argument or soften the claim.","section":"§4.3.3, Table 5"},{"comment":"Some QA examples appear malformed, such as 'Is the snow in larger than the skis?' and answers that repeat the same entity names without disambiguation. If these are representative samples from the released dataset, they suggest the synthetic data may contain template errors; please clean the examples or note known artifacts.","section":"Figures 10 and 11"}],"recommendation":"major_revision","confidential_remarks":"The PSG-based comparison is externally grounded and likely publishable once the spatial evaluation is made credible. The main risk is that the paper's novel spatial contribution currently rests on a benchmark that shares its generation pipeline with the training data and on a few percent accuracy difference over 271 questions. I would encourage the editor to require an independent spatial benchmark or a clear human-annotation protocol with disjointness guarantees before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is an incremental but useful MLLM-SGG paper. The SpaceSGG dataset, built with depth-derived 3D layering and three instruction formats (Desc, QA, Conv), is a genuine new resource, and the reported PSG gains over ASMv2 are externally grounded. The spatial-specific claim, however, rests on a validation set that is partly circular and too small to carry the weight the authors put on it.\n\nWhat it does well: the dataset construction is clearly described and the authors ship code, data, and models. The PSG evaluation is independent of their pipeline, and the improvement is plausibly real: 15.43 vs 14.2 recall and 13.23 vs 10.3 mRecall. Those are modest absolute gains, but they come from adding 40K instruction samples to the ASMv2 recipe, which is a clean experimental setup. The placebo ablations, replacing SpaceSGG with LLaVA-Instruct data, are exactly the right kind of control, even if the results are messy.\n\nThe soft spots are concentrated in the spatial evaluation. The spatial relation validation set is built from 30 COCO-Val images using the same generation pipeline as the training data (Section 3.1.4), and the paper does not state that those 30 images are disjoint from SpaceSGG training images. Manual correction helps, but the model could still be learning the pipeline's output distribution rather than genuine 3D spatial understanding. More concretely, the margin over ASMv2 is about 5 questions on 271, with no error bars or significance test. The ablation tables contain red flags: ab-train-1 scores 1.47% on the spatial benchmark, far below the 25% random baseline, and Table 6 has two rows at 0 recall / 0 accuracy. These look like output-parsing failures or training divergences, not measurements, and they undercut confidence in the spatial benchmark as a tool. The authors need to explain them before the spatial accuracy claim is credible.\n\nWho is this for: people working on open-vocabulary SGG with MLLMs, and anyone who wants a depth-informed spatial instruction dataset. The PSG result is the strongest part, and the dataset is a real artifact worth having. I would send this to peer review. The dataset and external PSG result deserve referee time, but the spatial validation needs substantial revision: independent annotations, disjoint images, error bars, and an explanation of the zero-score ablations.","headline":"A useful new depth-informed SGG dataset, but the paper's signature spatial-reasoning claim leans on a small, partly circular validation set with unexplained ablation zeros; the PSG gain over ASMv2 is modest but external.","tokens_in":18918,"tokens_out":4305,"would_cite":false,"duration_ms":40405,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Depth-derived spatial instruction data raises open-vocabulary scene graph recall by 8.6% and mean recall by 28.4% over ASMv2.","keywords":["scene graph generation","open-vocabulary","spatial relations","multimodal large language model","instruction tuning","depth estimation","depth layering","PSG dataset"],"falsifier":"Re-train the identical model with the same SpaceSGG data but evaluate on an independent human-annotated spatial relation benchmark built from images the pipeline never processed; if accuracy falls to the 25% random-choice level, the depth-derived training data did not teach transferable spatial understanding.","tokens_in":17886,"feed_emoji":"🕸️","tokens_out":7156,"duration_ms":65542,"temperature":0.7,"pith_summary":"The paper sets out to show that scene graph generation, extracting (subject, predicate, object) triplets from an image, can be made open-vocabulary and spatially aware at the same time by training a multimodal large language model on synthetic data that encodes 3D layout. The authors build the SpaceSGG instruction dataset: monocular depth estimates are turned into object depth ranges, objects are assigned to near-to-far layers, and an LLM rewrites this layout with scene graph triplets into three data formats, spatial descriptions, single-turn QA, and multi-turn conversations. Training LLaVA-SpaceSGG with this mixture, they report recall of 15.43 versus 14.2 and mean recall of 13.23 versus 10.3 against the ASMv2 baseline on the PSG validation set, plus 52.48% accuracy on a spatial-relation validation set they constructed. If correct, the paper shows that explicit depth-derived spatial supervision is what an open-vocabulary SGG model needs to produce relations such as 'in front of' without regression on the open-set task.","feed_headline":"Spatial depth data lifts scene graph recall 8.6%","feed_subtitle":"A vision-language model trained with 3D layer descriptions beats prior open-vocabulary baselines on PSG.","key_machinery":"The load-bearing object is the depth-layered scene graph description produced by the SpaceSGG pipeline. A monocular depth estimator (Depth Anything) gives per-pixel depth; camera calibration converts it into a point cloud; Algorithm 1 computes each object's z-range and coverage relations; Algorithm 2 sorts objects into layers by distance and builds sub-layers for covered objects; and an LLM rewrites this structured layout into natural-language layer-by-layer descriptions. Those descriptions, paired with spatially focused QA and chain-of-thought multi-turn conversations, become instruction-tuning targets in the second stage. The mechanism works because spatial predicates are expressed not as abstract labels but as textual layer orderings, letting the MLLM reuse its language priors about 'in front of', 'behind', and 'attached to' while grounding them in geometry.","core_discovery":"The central claim is that the reason existing open-vocabulary SGG models miss spatial relations is not a lack of model capacity but a lack of spatial signal in their training text: their instruction data is mostly 2D captions and common predicates. LLaVA-SpaceSGG demonstrates the fix by injecting depth-derived layering into the instruction stream. The depth map from a monocular estimator is calibrated into a point cloud, per-object depth ranges determine which objects cover or lie behind which, and objects are sorted into layers from near to far; this layered scene layout is then narrated by an LLM as SpaceSGG-Desc, SpaceSGG-QA, and SpaceSGG-Conv. On the external PSG benchmark the resulting model improves recall by 8.6% and mean recall by 28.4% over ASMv2, and on the authors' spatial benchmark it reaches 52.48%, outperforming LLaVA-1.5-13B at 45.13% and ASMv2-13B at 50.52%.","pith_inferences":["Since the spatial benchmark shares its construction pipeline with the training data, the 52.48% figure should be read as an upper bound on genuine spatial transfer; an independent human-annotated test set would give the sharper number.","The same depth-layering-to-text pipeline could be used to synthesize spatial supervision for other structured outputs, such as referring expressions, layout-based VQA, or robot instruction data, with the same caveat that model and benchmark would share biases.","The reported gain from depth-derived layers likely depends on the choice of monocular depth estimator, and scenes with ambiguous depth such as mirrors, transparent objects, or textureless regions are where the approach would most visibly fail.","If the method generalizes, a practical outcome is that open-vocabulary scene graphs with reliable 'in front of' and 'behind' relations become cheap to produce from ordinary photographs, which would directly help image retrieval and captioning systems that consume scene graphs."],"forward_implications":["Open-vocabulary SGG can be improved by changing the training text, not the label space: adding 40K spatial instruction samples raises PSG recall from 14.2 to 15.43.","Monocular depth is sufficient to teach a model front-back and occlusion relations for SGG, avoiding the need for 3D sensors at inference time.","The same two-stage training paradigm, stage-1 image-level alignment followed by stage-2 SGG instruction tuning, is the recipe that transfers MLLM priors to structured scene graphs.","Because the SpaceSGG dataset and code are released, other models can be re-tuned on this data and compared under identical PSG evaluation.","Spatial-only data formats each contribute: the paper's ablations show removing any one of Desc, QA, or Conv lowers either PSG recall or spatial accuracy."],"supporting_citations":[{"why":"Supplies the ASMv2 baseline and the AS-V2 instruction dataset that the method extends and must beat.","marker":"[51]"},{"why":"Provides monocular depth estimates that seed the spatial layout extraction pipeline.","marker":"[57]"},{"why":"Provides the 3D reconstruction used to lift depth maps into point clouds for spatial layering.","marker":"[14]"},{"why":"Gives the PSG validation set used for the open-vocabulary recall and mean recall comparisons.","marker":"[55]"},{"why":"Provides the LLaVA-1.5 backbone that LLaVA-SpaceSGG is built on.","marker":"[33]"},{"why":"Provides the original visual instruction tuning recipe and stage-1 image-level data.","marker":"[34]"},{"why":"Supplies COCO images used for the constructed spatial relation validation set.","marker":"[30]"},{"why":"Provides the Llama 3 70B generator that creates SpaceSGG descriptions, QA, and conversations.","marker":"[2]"}],"fun_headline_variants":["Depth data boosts open-vocabulary scene graph recall by 8.6%","3D layering improves open-vocab scene graph recall by 8.6%","Depth-aware MLLM lifts scene graph recall 8.6% on PSG","Spatial depth training yields 8.6% recall gain for scene graphs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automatically generated and manually corrected spatial labels used for both training and evaluation faithfully describe true 3D layout.","fun_headline_variants_meta":{"raw":{"variants":["Depth data boosts open-vocabulary scene graph recall by 8.6%","3D layering improves open-vocab scene graph recall by 8.6%","Depth-aware MLLM lifts scene graph recall 8.6% on PSG","Spatial depth training yields 8.6% recall gain for scene graphs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001101,"raw_usage":{"total_tokens":4621,"prompt_tokens":1004,"completion_tokens":3617,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":3527}},"tokens_in":620,"tokens_out":3617,"duration_ms":26570,"temperature":1.0,"reasoning_tokens":3527,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:46:19.715859+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-train the identical model with the same SpaceSGG data but evaluate on an independent human-annotated spatial relation benchmark built from images the pipeline never processed; if accuracy falls to the 25% random-choice level, the depth-derived training data did not teach transferable spatial understanding.","supporting_citations":[{"cited_title":"Depth anything: Unleashing the power of large-scale unlabeled data","cited_arxiv_id":null,"evidence_quote":"Provides monocular depth estimates that seed the spatial layout extraction pipeline."},{"cited_title":"Deep- videomvs: Multi-view stereo on video with recurrent spatio- temporal fusion","cited_arxiv_id":null,"evidence_quote":"Provides the 3D reconstruction used to lift depth maps into point clouds for spatial layering."},{"cited_title":"Panoptic scene graph gen- eration","cited_arxiv_id":null,"evidence_quote":"Gives the PSG validation set used for the open-vocabulary recall and mean recall comparisons."},{"cited_title":"Improved baselines with visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Provides the LLaVA-1.5 backbone that LLaVA-SpaceSGG is built on."},{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Provides the original visual instruction tuning recipe and stage-1 image-level data."},{"cited_title":"Microsoft coco: Common objects in context","cited_arxiv_id":null,"evidence_quote":"Supplies COCO images used for the constructed spatial relation validation set."},{"cited_title":"Llama 3 model card","cited_arxiv_id":null,"evidence_quote":"Provides the Llama 3 70B generator that creates SpaceSGG descriptions, QA, and conversations."}],"review_version":1}