{"id":"85567970-8cc7-43a8-8d1e-b698063ab0d4","arxiv_id":"2506.16805","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Co-VisiON is a new sparse-view co-visibility graph benchmark on Gibson and HM3D, where all tested vision models, including the proposed Covis, fall short of human performance.","lead":"The paper introduces Co-VisiON, a benchmark that asks vision models to decide which images in a sparse set of indoor photos show overlapping parts of the same scene. On it, the best models (GPT-4o at 63% graph IoU, the new Covis at 61%) still trail humans (72%), suggesting current systems lack human-like spatial reasoning from few views.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Human 'upper bound' is computed against a different co-visibility criterion than the automatic ground truth, so the headline human-vs-model gap may be a label-mismatch artifact rather than a measured human advantage.","rationale":"The reader's weakest assumption correctly identifies the criterion mismatch between human annotations and the automatic geometric ground truth as the key vulnerability. My stress-test reading confirms this is the single most load-bearing issue: the abstract and Section 5.1 state that all models fall significantly short of human performance, with the human score treated as an upper bound. But the human annotator instructions in Appendix B.1.2 explicitly exclude featureless overlaps, while the automatic ground truth in Section 3.1 includes any non-zero shared surface. If humans are effectively labeling a subset of the automatic GT, their IoU of 0.72 may simply reflect the fraction of automatic-GT edges with distinguishable features, not a ceiling on human reasoning ability. This is a correctness risk rather than a stylistic complaint: the headline quantitative comparison is not measuring the same task for both sides. The proposed check of recomputing human and model IoUs against a common human-rule GT would settle the issue. The reader's additional concerns about the small human sample, missing error bars, and unclear overlap with the test split are real but secondary; they would be mitigated by the same criterion-aligned re-scoring plus standard statistical reporting. No adjustment to the reader's CONDITIONAL verdict is needed, because the benchmark itself remains useful and the proposed Covis method and its ablations are plausible; the conditional status appropriately requires the authors to address the criterion mismatch and the statistical gaps before the human-comparison claim is stated as a headline result.","tokens_in":18555,"tokens_out":5224,"duration_ms":58843,"concrete_test":"Re-score the six human-annotated Gibson scenes against a human-rule ground truth: start from the automatic per-pixel overlap masks of Section 3.1, flip connected edges to 0 when the overlapping region is featureless (e.g., using a texture-energy threshold calibrated on plain-wall regions), and recompute both human and model Graph IoUs on exactly this same target. If human IoU rises to near 1.0 while model IoUs remain near 0.6, the reported 0.72 human score is a criterion artifact; if human IoU stays near 0.72, the criterion mismatch is not the explanation and the existing comparison is more defensible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark definition in Section 3.1 and Fig. 1 treats co-visibility as any non-zero shared visible surface: A∩B ≠ ∅ ⇒ G(A,B)=1. Appendix B.1.2, however, instructs human annotators to label a pair as not connected when the overlapping region is featureless, e.g., a plain wall. This yields a human-annotated graph H that is a stricter, semantic graph rather than the geometric graph G used for automatic ground truth. If H ⊆ G, then the reported human Graph IoU against G simplifies to |H|/|G|, so the observed 0.72 can be explained by humans omitting roughly 28% of automatic-GT edges whose overlap is featureless. In that case the number is not a measure of human co-visibility reasoning on the benchmark's own task definition; it is a measure of agreement between two different labeling criteria. Comparing this 0.72 to model IoUs computed against G is therefore not an apples-to-apples comparison, and the claim that human annotation represents an upper bound is unsupported. The human sample is also small (six Gibson scenes, arbitrary selection, no confidence intervals, unclear overlap with the model test split), which amplifies the problem, but the criterion mismatch alone is sufficient to call the central quantitative claim into question.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Co-VisiON, a benchmark for co-visibility reasoning from sparse sets of indoor images. The task is to predict, for every image pair in a scenario, whether the two images share any visible 3D surface, producing a binary co-visibility graph. The benchmark is built from Gibson and HM3D scenes rendered in Habitat-sim, with automatic ground-truth graphs computed from camera poses and depth, and with a small human-annotation subset on six Gibson scenes. The authors benchmark feature matching, contrastive learning, pairwise classifiers, 3D reconstruction methods, and several vision-language models, and they propose Covis, a multi-view masked classification baseline. The headline results are that GPT-4o reaches 0.63 Graph IoU on Gibson, Covis-freeze reaches 0.61, and human annotators reach 0.72, leading to the claim that all models fall significantly short of human performance. The paper also demonstrates downstream uses of predicted co-visibility graphs for DUSt3R-based reconstruction and CroCo-based training-pair selection.","tokens_in":18707,"tokens_out":3844,"duration_ms":42293,"significance":"If the human comparison were valid, this would be a useful and underexplored benchmark: co-visibility reasoning on sparse image sets is relevant to image matching, place recognition, SLAM, and scene reconstruction, and the paper provides a broad and fairly organized set of baselines. The automatic ground-truth generation from simulated depth and poses is clean and reproducible, and the authors release data and code. The proposed Covis model with mask supervision is a sensible baseline and the ablation isolating the effect of masking is informative. However, the central quantitative claim that humans outperform all models depends on a comparison between two different labeling criteria, as detailed in the major comments. The benchmark infrastructure and the Covis contribution remain valuable, but the headline human-vs-model gap is not currently supported by the evidence as presented.","major_comments":[{"comment":"The automatic ground truth defines co-visibility as any non-zero shared visible surface, while the human annotation instructions in Appendix B.1.2 explicitly say that a pair whose overlap is a featureless region such as a plain wall is labeled as not connected. The human-annotated graph is therefore a stricter, semantic graph rather than the geometric graph used for all automatic and model evaluations. If the human graph H is a subset of the automatic graph G, then the reported human Graph IoU of 0.72 against G simplifies to |H|/|G|, so the value may simply reflect the fraction of automatically connected pairs whose overlap humans judged featureless. This means the human score is not measuring performance on the benchmark's own task definition, and Table 4's statement that \"Human annotation performance represents the upper bound\" is not supported. The comparison must be made consistent, either by collecting human annotations under the geometric criterion, by evaluating models against a human-style semantic ground truth, or by substantially qualifying the human-vs-model claim.","section":"Sec. 3.1 and Appendix B.1.2"},{"comment":"The human baseline is derived from only six Gibson scenes chosen arbitrarily, with no report of the number of annotators per scene, the number of annotated pairs, inter-annotator agreement, per-scene variance, or confidence intervals. It is also not stated whether these six scenes are part of the model test split or the training split. With this sample size and these omissions, the reported 0.72 human IoU cannot support the strong claim that \"all models fall significantly short of human performance\" or that the human score is a reliable upper bound. The authors should report per-scene human IoU, bootstrapped confidence intervals, and explicit details about how the human-annotated scenes relate to the model evaluation split.","section":"Appendix B.1.2 and Table 4"},{"comment":"The table note claims that for 3D reconstruction, VLM, and human annotation categories \"AUC values are equivalent to IoU since they are not affected by thresholding.\" This is not justified by the AUC definition in Eq. (5), which averages Graph IoU over thresholds. For any method that outputs a binary graph directly, varying the threshold changes the binarized prediction at the threshold crossing, so the AUC is not generally equal to the IoU at a single operating point. The manuscript should clarify how AUC is computed for methods that do not produce continuous scores, or should report only IoU for those methods; as written, the AUC column mixes incompatible definitions across methods.","section":"Table 4 and Eq. (5)"}],"minor_comments":[{"comment":"The sentence \"Modern vision models have shown achieved remarkable progress\" contains a grammatical error and should be rewritten.","section":"Sec. 1"},{"comment":"The description of Covis says the backbone is CroCo v2 with a BCE loss, but the relationship between the pairwise and multi-view variants, the exact freezing procedure, and the training hyperparameters are only partially specified; please provide full training details in the appendix or supplement.","section":"Sec. 4.3"},{"comment":"The row \"Applied GT Mask\" reports 0.74/0.72 Gibson IoU/AUC and is substantially above all learned models; this is an oracle upper bound that should be explicitly interpreted as such in the main text, since it also indicates the headroom available with perfect masking.","section":"Table 2"},{"comment":"The dataset generation constrains pairwise IoU to the range 5%--30%, which may strongly influence the difficulty distribution and the reported \"hard\" cases in Table 3; please state whether this constraint applies to all scenarios and discuss its effect on the generality of the benchmark.","section":"Sec. 3.3"},{"comment":"The scoring function uses alpha=0.9 and beta=0.1, but there is no sensitivity analysis for these parameters; a sentence acknowledging this would help.","section":"Appendix B.1.1"},{"comment":"The IoU formula uses A and A-hat in the text but the displayed equation uses Aij and Aij with hats missing in one place; please fix the notation for clarity.","section":"Appendix D, Eq. (4)"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and the Covis baseline are potentially valuable contributions, and the automatic annotation pipeline is a solid engineering contribution. The main risk is that the headline claim about human superiority is built on a comparison of incompatible labeling criteria and a very small human sample. I would encourage the editor to require the authors to address the criterion mismatch and the statistical robustness of the human baseline before considering the paper for publication; the benchmark itself, however, does not need to be rejected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the benchmark is a genuinely useful contribution, but the headline claim that humans beat all models is not supported, because the human annotations obey a different co-visibility rule than the automatic ground truth.\n\nWhat's new: Co-VisiON formalizes sparse-set co-visibility graph prediction, pairs it with dense co-visibility masks from two simulators, and evaluates with graph IoU/AUC. That's not covered by Rau et al.'s pairwise overlap embeddings or EC-SfM's use of co-visibility. The baseline sweep is broad—feature matchers, contrastive, supervised classifiers, DUSt3R, VLMs—and the proposed Covis (CroCo/MV-DUSt3R with a learnable mask) is a reasonable extension. The Sim2Real appendix is a nice addition, and the authors release data and code.\n\nThe soft spot is the human comparison. Section 3.1 defines co-visibility as any non-zero shared visible surface. Appendix B.1.2 tells annotators to label a pair as 'not connected' when the overlap is featureless, e.g. a plain wall. So the human graph H is, by construction, a subset of the geometric ground truth G: humans who follow the instructions will omit exactly the featureless-overlap edges. Then human Graph IoU against G is just |H|/|G|, not a measure of human performance on the same task as the models. The reported 0.72 versus best model 0.63 could be entirely explained by this label mismatch. The paper also gives no error bars, no inter-annotator agreement, and uses only six Gibson scenes, so the 'upper bound' claim is doubly unsupported.\n\nThat said, the benchmark itself rests on the automatic geometric ground truth, which is clean in simulation, and the models are all scored consistently against that ground truth. The problem is confined to the human-vs-model comparison, not the entire benchmark. The ablation of Covis and the downstream SfM/completion experiments stand on their own.\n\nWho it's for: anyone working on multi-view understanding, sparse-view geometry, or embodied perception will find the dataset and evaluation protocol useful. A serious referee should engage with it, but should require the authors to either (a) align the human criteria with the geometric definition, (b) re-run the human study under the same rule, or (c) downgrade the human number to a separate 'semantic co-visibility' baseline rather than an upper bound. They should also report variance across runs and across scenes.\n\nRecommendation: accept for peer review with major revision on the human comparison.","headline":"Useful benchmark, but the human-vs-model gap is likely a labeling-criterion artifact and should be fixed before the headline claim is trusted.","tokens_in":19378,"tokens_out":2783,"would_cite":false,"duration_ms":25569,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Co-VisiON argues that deciding which sparse indoor photos see the same 3D surface is a distinct skill current models lack: humans hit 0.72 graph IoU, GPT-4o 0.63, best vision-only 0.61.","keywords":["co-visibility reasoning","sparse-view images","co-visibility graph","graph IoU","indoor scenes","multi-view learning","vision-language models","3D reconstruction"],"falsifier":"Re-run the human annotation on the Gibson subset with one change to the instructions: count any pixel-level overlap of visible 3D surface as connected, including plain walls, and recompute human graph IoU against the automatic ground truth. If the human score stays at or near 0.72, the reported human-model gap is robust; if it rises substantially, the gap is in part an artifact of mismatched annotation criteria rather than a pure measure of human reasoning.","tokens_in":18263,"feed_emoji":"🏠","tokens_out":10255,"duration_ms":101162,"temperature":0.7,"pith_summary":"Co-VisiON asks a deceptively simple question: given a handful of photos taken at scattered viewpoints inside an indoor scene, which pairs show the same 3D surfaces? The paper's central claim is that answering this 'co-visibility' question reliably is a genuine spatial-reasoning ability — not a byproduct of low-level feature matching — and that current models, from classic feature matchers to GPT-4o, fall measurably short of humans on it. On the proposed benchmark, human annotators reach 0.72 graph IoU on Gibson, the strongest large vision-language model reaches 0.63, and the best pure-vision model (the paper's own Covis-freeze) reaches 0.61. The paper also argues that a predicted co-visibility graph, even when imperfect, is a useful substitute for expensive ground-truth graphs in 3D reconstruction and in automatically labeling training pairs for cross-view completion. The reason to care is practical: robots, SLAM, and 3D reconstruction pipelines need to know which views are linked before they can fuse them, especially when observations are sparse.","feed_headline":"Humans outscore GPT-4o and all vision models on new 3D test","feed_subtitle":"Co-VisiON benchmark: humans 0.72 graph IoU, GPT-4o 0.63, best vision-only 0.61.","key_machinery":"The object that carries the argument is the co-visibility graph: a binary graph whose nodes are images and whose edges connect exactly those pairs that share a non-zero visible surface, as determined by projecting rendered depth into a common point cloud. Evaluation reduces the whole task to computing graph intersection over union between predicted and ground-truth adjacency matrices, with AUC averaging graph IoU over thresholds. For the model side, the load-bearing mechanism is the learnable co-visibility mask in Covis: pixel-wise features are projected from the decoder tokens, a mask $M_i = \\mathrm{conv}(X_i)$ is predicted for each view, features are filtered by $X' = X \\odot M$, and the mask is supervised by ground-truth co-visible regions; this filtering prevents non-co-visible information from contaminating the representation and is what allows the multi-view variant to beat pairwise inputs. The multi-view setting itself uses cross-view attention in an encoder-decoder, following the MV-DUSt3R pattern.","core_discovery":"The central claim, stated on the paper's own terms, is that co-visibility reasoning under sparse views — determining whether two images share any non-zero visible surface area — is a well-posed task that existing vision systems do not solve, and that the gap to human performance is large and consistent across method families. The paper establishes this by building a 1,000-plus scenario benchmark from photorealistic indoor simulations with pixel-level ground-truth co-visibility, by collecting human annotations as an upper bound, and by showing that feature matching, contrastive learning, pairwise classification, 3D reconstruction, and vision-language prompting all lag behind humans; the best result, GPT-4o at 0.63 graph IoU on Gibson, still trails the human score of 0.72. A secondary claim is that a multi-view classifier with an explicit learnable co-visible-region mask — Covis — is the strongest pure-vision baseline, outperforming its pairwise counterpart by about three IoU points and approaching the proprietary VLM, which the paper reads as evidence that multi-view context plus localized co-visibility supervision captures more of the spatial reasoning humans perform.","pith_inferences":["The human annotation rules and the automatic ground truth disagree on featureless overlaps (humans were told plain walls do not count as co-visible; the geometric ground truth counts any shared surface). Re-annotating Gibson with the geometric rule would likely move the human score and could shrink or expand the reported gap; the paper does not quantify this disagreement.","Because the co-visibility graph transfers to DUSt3R and CroCo with little loss, the same graph could plausibly drive keyframing and view-selection policies in online SLAM and NeRF-style reconstruction, a use the paper mentions only implicitly.","The sim2real result (Covis zero-shot AUC 0.61 on the AVD real-world set, in the appendix) suggests the benchmark's synthetic ground truth is not the only source of the human-model gap; a testable extension would be to collect human annotations directly on real sparse photo sets and compare.","A model that combines Covis-style masked multi-view features with a language-model prior might exceed GPT-4o's 0.63; the paper's own analysis implies such a hybrid is the natural next step rather than a purely geometric refinement."],"forward_implications":["If the benchmark measures what it claims, then any practical sparse-view system — a robot exploring a home, a structure-from-motion pipeline — should treat co-visibility as a first-class prediction target rather than a by-product of feature matching.","The downstream experiments imply that a predicted co-visibility graph can replace the complete or ground-truth graph in DUSt3R-style reconstruction: the co-visibility graph gives an average pose error of 1.70 m versus 1.55 m for ground truth, while cutting memory from 10.2 GB to 6.7 GB per scene.","For training-set labeling, the sparse co-visibility graph generates CroCo cross-view completion training pairs that reach PSNR 16.34, essentially matching the 16.32 of a high-overlap graph, which implies strict high-overlap filtering is unnecessary for effective self-supervision.","The consistent ordering — humans, then a prompt-based VLM, then the best supervised vision model — implies that spatial-semantic integration, not just more training data on the same features, is the direction needed to close the gap.","Hard-case breakdowns (image overlap under 10%, scene overlap under 4%) show all models collapse to near-chance, so the corollary is that improvements on easy cases should not be read as progress on the actual hard sparse-view regime."],"supporting_citations":[{"why":"Gibson Env supplies the photorealistic virtual scenes used for one half of the Co-VisiON dataset and for the human-annotation benchmark.","marker":"[61]"},{"why":"HM3D supplies the second set of virtual indoor scenes and the larger-scale training/test portion of the dataset.","marker":"[41]"},{"why":"DUSt3R provides a 3D-reconstruction baseline and the confidence-inspired masking idea that Covis adapts into its learnable mask.","marker":"[57]"},{"why":"MV-DUSt3R+ provides the multi-view encoder-decoder backbone that Covis builds on for cross-view information exchange.","marker":"[54]"},{"why":"CroCo v2 is the backbone for both pairwise and multi-view Covis and for the binary classification baselines.","marker":"[59]"},{"why":"SuperGlue serves as the representative deep feature-matching baseline that fails on sparse views.","marker":"[43]"},{"why":"GPT-4o is the strongest vision-language baseline whose 0.63 graph IoU defines the closest model-to-human gap.","marker":"[38]"},{"why":"NetVLAD serves as the visual-place-recognition baseline showing that retrieval-style global descriptors are insufficient for co-visibility reasoning.","marker":"[5]"},{"why":"The multiview scene graph work supplies the graph IoU metric used to score all predicted co-visibility graphs.","marker":"[67]"}],"fun_headline_variants":["Humans beat GPT-4o and all vision models on 3D co-visibility test","New 3D co-visibility benchmark: humans still rule, AI lags","Sparse-view 3D test: humans 0.72, best AI 0.63","AI trails humans by 9 points on new 3D reasoning test","Co-visibility: humans outthink all AI models in sparse indoor scenes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's headline human-versus-model comparison assumes that the automatically computed ground truth (any non-zero shared visible surface, including featureless walls) measures the same co-visibility ability that humans were instructed to judge, yet the human annotators were explicitly told that plain-wall overlaps do not count as connected.","fun_headline_variants_meta":{"raw":{"variants":["Humans beat GPT-4o and all vision models on 3D co-visibility test","New 3D co-visibility benchmark: humans still rule, AI lags","Sparse-view 3D test: humans 0.72, best AI 0.63","AI trails humans by 9 points on new 3D reasoning test","Co-visibility: humans outthink all AI models in sparse indoor scenes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001024,"raw_usage":{"total_tokens":4374,"prompt_tokens":1055,"completion_tokens":3319,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":671,"completion_tokens_details":{"reasoning_tokens":3210}},"tokens_in":671,"tokens_out":3319,"duration_ms":20807,"temperature":1.0,"reasoning_tokens":3210,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:18:06.224242+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the human annotation on the Gibson subset with one change to the instructions: count any pixel-level overlap of visible 3D surface as connected, including plain walls, and recompute human graph IoU against the automatic ground truth. If the human score stays at or near 0.72, the reported human-model gap is robust; if it rises substantially, the gap is in part an artifact of mismatched annotation criteria rather than a pure measure of human reasoning.","supporting_citations":[{"cited_title":"Croco v2: Improved cross-view completion pre- training for stereo matching and optical flow","cited_arxiv_id":null,"evidence_quote":"CroCo v2 is the backbone for both pairwise and multi-view Covis and for the binary classification baselines."},{"cited_title":"Superglue: Learning feature matching with graph neural networks","cited_arxiv_id":null,"evidence_quote":"SuperGlue serves as the representative deep feature-matching baseline that fails on sparse views."},{"cited_title":"Gpt-4 technical report, 2023","cited_arxiv_id":null,"evidence_quote":"GPT-4o is the strongest vision-language baseline whose 0.63 graph IoU defines the closest model-to-human gap."},{"cited_title":"Netvlad: Cnn architecture for weakly supervised place recognition","cited_arxiv_id":null,"evidence_quote":"NetVLAD serves as the visual-place-recognition baseline showing that retrieval-style global descriptors are insufficient for co-visibility reasoning."},{"cited_title":"#$# \"&'#()*#≥,!","cited_arxiv_id":null,"evidence_quote":"The multiview scene graph work supplies the graph IoU metric used to score all predicted co-visibility graphs."}],"review_version":2}