{"id":"ec3f2d88-1eea-42d6-acd7-949ad8c5e7ce","arxiv_id":"1908.02660","paper_version":2,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"SpatialSense is a new adversarial crowdsourced benchmark for spatial relation recognition where human annotators cherry-pick relations that fool simple machine predictors, and top models score close to trivial baselines on it.","lead":"This paper introduces SpatialSense, a dataset of image relations that is built to be hard to guess from simple language or image-position cues. On this benchmark, state-of-the-art visual relationship models perform no better than simple baselines, exposing a gap in current spatial reasoning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3's SOTA-vs-baseline comparison may hinge on an unspecified binary threshold and negative-example training protocol for the adapted 9-way detectors.","rationale":"The reader's identified weakest assumption—that the crowdsourcing robot is a strong proxy for all simple cues—is real but does not directly threaten the paper's headline empirical claim. Even if the robot missed cues, SpatialSense would still be a valid testbed, and the finding that SOTA models match a tuned 2D+language baseline would still suggest shallow reasoning; residual bias would if anything make that suggestion easier to support. The load-bearing condition for 'state-of-the-art models perform comparably to simple baselines' is instead that the five SOTA detectors are fairly and reproducibly adapted to the binary verification task. Section 5 and Appendix B specify architectures but omit two essentials: the training objective for negative relations (the original detectors train on positive triplet ranking, with no target for a false predicate) and the threshold used to turn a 9-way logit into a binary verdict. Without these, Table 3's numbers are not reproducible, and the 0.2% gap between DRNet and the tuned language+2D baseline could easily be reversed by a threshold sweep or a proper negative-training protocol. This is a concrete correctness risk in the central claim, not a disagreement with the value of the dataset. The paper's code is public, so the check is feasible. I would accept the paper conditional on the authors either reporting the missing training/threshold details or demonstrating that the comparison is robust under a validation-tuned threshold.","tokens_in":19181,"tokens_out":13116,"duration_ms":151471,"concrete_test":"Re-run DRNet (and one other SOTA model, e.g., VTransE) on SpatialSense using the released code, sweeping the binary decision threshold for the given-predicate logit on the validation split, and explicitly training with negative examples (e.g., binary cross-entropy on the predicate score, or a 10th none class). Compare the resulting test accuracy to the language+2D baseline; if DRNet moves by more than about 1% absolute, Table 3's conclusion is not robust.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim rests on Table 3's comparison of DRNet (71.3%) with the language+2D baseline (71.1%). Section 5 says the five relationship detectors 'generate scores for all predicates; the score for that particular predicate is the final binary classification score.' Appendix B shows raw logit outputs (e.g., DRNet 'on: 3.45'), but the paper never states how this logit is converted to a binary decision, nor how negative examples are used to train a 9-way softmax model originally designed for positive-only triplet ranking. A threshold of 0 on a softmax logit is not a calibrated 0.5 decision; the correct threshold depends on the other predicate scores, so an arbitrary threshold can suppress or inflate reported accuracy. The simple baselines, in contrast, are binary classifiers with a natural sigmoid threshold, and their 80/20 fusion weight is tuned on validation. If DRNet's threshold was not likewise tuned, the headline 'comparable to simple baselines' may be an artifact of evaluation asymmetry rather than a property of the model.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SpatialSense, a benchmark for spatial relation recognition built through adversarial crowdsourcing: annotators propose positive or negative spatial relations that must fool a robot that uses only object names and 2D coordinates. The dataset contains 17,498 relations over 9 predicates with balanced positives and negatives. The authors show reduced language and 2D bias relative to VRD and Visual Genome, verify the effect through an ablation (SpatialNaive), benchmark five visual relationship detection models, and report that the best state-of-the-art model (DRNet, 71.3%) performs on par with a tuned language+2D baseline (71.1%), while humans achieve 94.6%.","tokens_in":19380,"tokens_out":5237,"duration_ms":53382,"significance":"If the experimental claims are fully supported, this is a strong contribution: it provides a reusable benchmark for spatial reasoning, demonstrates a practical adversarial crowdsourcing protocol, and offers evidence that current visual relationship models rely on shallow cues. The dataset and code release, the SpatialNaive ablation, and the cross-dataset generalization analysis are valuable. The central quantitative conclusion, however, rests on an evaluation protocol that is currently underspecified and asymmetric, so the headline result should be regarded as preliminary until the experimental details are clarified.","major_comments":[{"comment":"The adaptation of the five relation detectors to binary classification is not fully specified. For each relation, the model outputs scores for all nine predicates, and \"the score for that particular predicate is the final binary classification score\" (§5). The paper never states how that real-valued score is converted to a binary prediction (e.g., threshold at 0, or comparison with other predicate scores), nor how negative examples are used to train models originally designed for positive triplet ranking. Because the language+2D baseline is a calibrated binary classifier with its fusion weight tuned on validation, an arbitrary or untuned threshold for the SOTA models could artificially suppress or inflate their accuracies. Please specify the exact decision rule, tune the threshold on validation (or report ROC/PR curves), and describe the negative-example training protocol so that the comparison in Table 3 is meaningful.","section":"§5 Model architectures; Table 3"},{"comment":"The evaluation is asymmetric with respect to hyperparameter tuning: the 80/20 fusion weight of the language+2D baseline is chosen by validation accuracy, while the five SOTA models are adapted off-the-shelf with no comparable validation-based selection of thresholds or hyperparameters. This is acceptable for a lower-bound demonstration, but the claim that SOTA models \"perform comparably to simple baselines\" (Abstract; §6) needs to be qualified, or the SOTA models need to be given the same tuning budget, before the conclusion can be stated as a property of the models rather than of the evaluation setup.","section":"§5 Implementation details"}],"minor_comments":[{"comment":"The claim that SpatialSense is \"less susceptible to simple cues\" should be explicitly scoped to the specific language and 2D models used; adversarial crowdsourcing guarantees hardness only against the particular robot, and a stronger language prior or geometric heuristic could conceivably still exploit regularities in the dataset.","section":"§4.1; Table 1"},{"comment":"Excluding annotators who answer \"yes\" more than 80% of the time is a reasonable quality-control step, but it can bias the human accuracy estimate upward; please report the number of excluded annotators and test the robustness of the 94.6% figure to this exclusion rule.","section":"§5 Human evaluation"},{"comment":"The table reports single accuracies without confidence intervals or significance tests; the differences between DRNet (71.3%), VTransE (69.4%), and the baseline (71.1%) are small relative to likely sampling noise, and paired bootstrap intervals or McNemar tests would strengthen the comparative claims.","section":"Table 3"},{"comment":"For the SpatialSense/SpatialNaive comparison, please state the number of relations used in the subsampled SpatialSense split and whether the same training/validation/test splits were used across the two datasets, so that the accuracy drops in Table 2 are directly interpretable.","section":"§4.2; Table 2"},{"comment":"The DRNet description notes that \"we do not perform iterative updates to the object name features\"; it would help to state explicitly whether this deviates from the original DRNet training procedure and whether the model was pretrained on any other dataset before adaptation to the binary spatial-relation task.","section":"Appendix B; Figure D"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is well-written and the dataset is likely to be a useful community resource. My main concern is the experimental protocol behind the headline claim: the missing threshold and tuning details could change the conclusion that SOTA models are on par with simple baselines. I recommend major revision rather than rejection, since the issues are fixable with additional analysis and clarification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Kaiyu, this one is worth a look. The headline result—SOTA models at roughly 70% versus humans at 94.6% on spatial relations—is the kind of number that gets attention, and the dataset is a genuine contribution. But before trusting the specific claim that DRNet (71.3%) merely matches a tuned 2D+language baseline (71.1%), check how the binary decision was made. I couldn't find it in the paper, and the stress-test note is on point: the adapted 9-way detectors output logits (e.g., 'on: 3.45'), and the paper never states how a logit becomes a yes/no prediction. If the threshold was not tuned on validation, or if negative examples were not used sensibly during training, that 0.2-point tie could move. This is a fixable reporting gap, but it is load-bearing for the paper's central comparison.\n\nWhat is genuinely new: the adversarial crowdsourcing protocol, borrowed from VQA, fits spatial relations well, and the paper shows it works. The ablation against SpatialNaive is the strongest evidence: language baseline drops from 69.2% to 56.4%, 2D from 71.3% to 65.2% when the robot is used, so the selection procedure did what it was designed to do. The balanced positives/negatives and the long-tail object classes are real improvements over VRD and Visual Genome. The cross-dataset generalization numbers are a nice bonus, and the human evaluation at 94.6% gives a clear target for future work.\n\nSoft spots, in proportion: the threshold/negative-training issue is the big one; it needs a precise description and, ideally, a sensitivity analysis showing SOTA numbers are stable across thresholds. There are no error bars anywhere, so differences of 1–2% between models are probably not meaningful. The 80/20 fusion weight for the language+2D baseline was tuned on validation while the SOTA models were not given the same treatment—that asymmetry compounds the threshold concern. The adversarial robot is a weak proxy for 'all simple cues,' but the SpatialNaive comparison mitigates that concern, so I'd call it minor.\n\nBottom line: this is a useful benchmark and the main qualitative conclusion—current models are not doing deep spatial reasoning—is probably right. It deserves a serious referee and, with a clarified evaluation protocol, it should be accepted. I'd cite it and bring it to the reading group.","headline":"A valuable spatial-relations benchmark whose headline comparison is weakened by an unspecified binary decision rule; clarify the threshold and it's a solid accept.","tokens_in":19882,"tokens_out":3090,"would_cite":true,"duration_ms":32362,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that state-of-the-art spatial relation recognition models perform no better than simple 2D and language baselines on a benchmark collected adversarially to remove easy cues.","keywords":["spatial relation recognition","visual relationship detection","adversarial crowdsourcing","dataset bias","benchmark","language priors","2D spatial cues","spatial reasoning"],"falsifier":"Train a model on SpatialSense that exceeds the language-plus-2D baseline by a wide margin (for example, above roughly 75% accuracy) without additional supervision, or show that a stronger 2D-only heuristic - using relative box area, aspect ratio, or estimated depth - reaches or beats DRNet's 71.3%. Either result would show that current models can go beyond the robot's simple cues and that the benchmark's difficulty is not the real ceiling for spatial reasoning.","tokens_in":1547,"feed_emoji":"🧩","tokens_out":4725,"duration_ms":79567,"temperature":0.7,"pith_summary":"This paper argues that state-of-the-art systems for recognizing spatial relations between objects mostly exploit shallow shortcuts instead of reasoning about the scene. To prove it, the authors built SpatialSense, a benchmark of 17,498 relations on 11,569 images, collected by asking crowd workers to find relations that fool a simple robot relying only on object names and 2D bounding-box positions. On this benchmark the best published model reaches 71.3% accuracy, essentially tied with a tuned language-plus-2D baseline at 71.1%, and the models' error patterns line up with that baseline. The paper concludes that adversarial crowdsourcing can strip out dataset bias and that progress on spatial reasoning needs benchmarks of this kind.","feed_headline":"Best spatial-relation models tie a 2D-only baseline","feed_subtitle":"An adversarial benchmark strips out easy cues, exposing the shortcut reliance of current visual-reasoning AI","key_machinery":"The load-bearing mechanism is an adversarial crowdsourcing protocol in which the 'robot' is an ensemble of a language-only model (word2vec embeddings of object names through a GRU, plus a predicate encoding) and a 2D-only model (linear layers over bounding-box coordinates). An annotator must propose a positive or negative spatial relation from a fixed nine-predicate vocabulary: the relation is kept only if the robot guesses wrong, with the robot explaining which cue gave it away. This filters out examples answerable from object names or box geometry alone, producing a testbed where simple priors fail (language accuracy 56.4% and 2D accuracy 65.2%, versus 69.2% and 71.3% on the non-adversarial ablation dataset).","core_discovery":"The central discovery is that when spatial-relation examples are selected adversarially to defeat language and 2D-geometry priors, the accuracy of state-of-the-art visual relationship models collapses to the level of simple baselines. DRNet, the best model evaluated, scores 71.3%, while a weighted combination of a language-only and 2D-only baseline scores 71.1%; the error vectors of the neural models are strongly correlated with the 2D baseline (correlations of 0.31-0.60 for most models), indicating they make the same kinds of mistakes. Humans score 94.6% on the same test data, so the ceiling is not the task itself. Adversarial crowdsourcing - human annotators propose relations that a robot ensemble cannot guess - reduces both language bias and 2D bias relative to earlier datasets, as verified by lower baseline accuracies and better cross-dataset generalization.","pith_inferences":["A stronger language-only prior than the GRU+word2vec model used in the robot might recover additional predictability from object names alone, suggesting that the residual language bias in SpatialSense could be underestimated.","Applying the same adversarial protocol to other relational tasks, such as compositional question answering or action prediction, could yield test sets where reported accuracy reflects genuine reasoning rather than corpus statistics.","The inclusion of 1,389 RGB-D images invites a direct test: if depth input lets a model substantially beat the 2D baseline, the remaining difficulty is genuinely 3D; if it does not, the bottleneck lies elsewhere.","Adversarially selected test sets could be used as a model-selection criterion during development, since they penalize shortcut reliance that biased benchmarks reward."],"forward_implications":["The best evaluated model, DRNet, scores 71.3% while the language-plus-2D baseline scores 71.1%, implying that current visual-relationship models do not deliver genuine spatial reasoning on this benchmark.","Error correlation values of 0.31-0.60 between the neural models and the 2D-only baseline indicate that these models make predictions very similar to simple geometry, not deep scene understanding.","Adversarial crowdsourcing reduces dataset bias: baseline accuracy drops from the non-adversarial SpatialNaive dataset (69.2% language, 71.3% 2D) to SpatialSense (56.4% language, 65.2% 2D).","SpatialSense's positive-only split is less biased in language and 2D than existing visual-relationship datasets: models trained on SpatialSense generalize to other datasets with an accuracy increase of 1.8% (language) and 5.2% (2D), while models trained on the other datasets drop by roughly 26-32% when evaluated across datasets.","Humans score 94.6% on the test split, so the gap between machines and humans is large: SpatialSense offers a benchmark where further progress in spatial reasoning can be measured."],"supporting_citations":[{"why":"Supplies the Beat the Machine adversarial crowdsourcing idea that the data collection protocol extends.","marker":"[2]"},{"why":"Defines the VRD dataset and the visual relationship detection task; its spatial relations are used as a biased comparison baseline.","marker":"[21]"},{"why":"Provides Visual Genome, the large-scale relation dataset whose spatial relations are shown to be predictable from language priors.","marker":"[13]"},{"why":"DRNet is the strongest state-of-the-art model benchmarked, scoring 71.3% on SpatialSense.","marker":"[5]"},{"why":"VTransE is a compared model whose error pattern correlates with both language and 2D baselines, supporting the shortcut-reliance claim.","marker":"[37]"},{"why":"Supplies the cross-dataset generalization methodology used to quantify dataset bias.","marker":"[30]"}],"fun_headline_variants":["Adversarial benchmark exposes spatial AI shortcut reliance","Spatial relation AI fails against adversarial benchmark","Humans beat top spatial models by 23% on new benchmark","New adversarial benchmark: AI spatial reasoning matches 2D heuristics","Adversarial crowdsourcing reveals spatial AI shortcuts"],"cache_read_input_tokens":22144,"weakest_assumption_plain":"The adversarial robot, an ensemble of a GRU-based language model and a linear 2D model, is assumed to catch every simple statistical cue a real model could exploit; if it misses an easy regularity, SpatialSense still contains hidden bias and the conclusion that state-of-the-art models rely on shallow cues is overstated.","fun_headline_variants_meta":{"raw":{"variants":["Adversarial benchmark exposes spatial AI shortcut reliance","Spatial relation AI fails against adversarial benchmark","Humans beat top spatial models by 23% on new benchmark","New adversarial benchmark: AI spatial reasoning matches 2D heuristics","Adversarial crowdsourcing reveals spatial AI shortcuts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00048,"raw_usage":{"total_tokens":2378,"prompt_tokens":949,"completion_tokens":1429,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":1350}},"tokens_in":565,"tokens_out":1429,"duration_ms":10787,"temperature":1.0,"reasoning_tokens":1350,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:38:03.643785+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a model on SpatialSense that exceeds the language-plus-2D baseline by a wide margin (for example, above roughly 75% accuracy) without additional supervision, or show that a stronger 2D-only heuristic - using relative box area, aspect ratio, or estimated depth - reaches or beats DRNet's 71.3%. Either result would show that current models can go beyond the robot's simple cues and that the benchmark's difficulty is not the real ceiling for spatial reasoning.","supporting_citations":[{"cited_title":"Beat the machine: Challenging workers to find the unknown unknowns","cited_arxiv_id":null,"evidence_quote":"Supplies the Beat the Machine adversarial crowdsourcing idea that the data collection protocol extends."},{"cited_title":"Visual relationship detection with language priors","cited_arxiv_id":null,"evidence_quote":"Defines the VRD dataset and the visual relationship detection task; its spatial relations are used as a biased comparison baseline."},{"cited_title":"Visual genome: Connecting language and vision using crowdsourced dense image annotations","cited_arxiv_id":null,"evidence_quote":"Provides Visual Genome, the large-scale relation dataset whose spatial relations are shown to be predictable from language priors."},{"cited_title":"Detecting visual rela- tionships with deep relational networks","cited_arxiv_id":null,"evidence_quote":"DRNet is the strongest state-of-the-art model benchmarked, scoring 71.3% on SpatialSense."},{"cited_title":"Visual translation embedding network for visual relation detection","cited_arxiv_id":null,"evidence_quote":"VTransE is a compared model whose error pattern correlates with both language and 2D baselines, supporting the shortcut-reliance claim."},{"cited_title":"Unbiased look at dataset bias","cited_arxiv_id":null,"evidence_quote":"Supplies the cross-dataset generalization methodology used to quantify dataset bias."}],"review_version":1}