{"id":"5fee12a3-dcec-47a1-bfee-08498fdf7ac8","arxiv_id":"2412.13569","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A new synthetic multi-view dataset and baseline model for predicting voxel-level pedestrian occupancy and panoptic labels in dense urban scenes.","lead":"The authors introduce MVP-Occ, a synthetic dataset of five large urban scenes with multi-view images and voxel-level pedestrian occupancy, panoptic, depth, and pose labels. They also present OmniOcc, a baseline model that predicts 2D and 3D pedestrian occupancy from multiple cameras and transfers from synthetic training to real surveillance data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The real-world 3D occupancy claim rests on a 2D segmentation proxy: Table 6 reports view-level 2D scores from ray-marched voxels, not voxel-level 3D occupancy, so it cannot by itself certify 3D transfer.","rationale":"The paper's central contribution is a new dataset plus a baseline demonstrating multi-view pedestrian occupancy prediction, including synthetic-to-real transfer. The dataset itself is plausible: it provides voxel-level semantic, instance, and panoptic labels for dense pedestrian scenes, which existing multi-view pedestrian datasets do not offer. The same-scene 3D results (Table 4) are measured on actual voxel ground truth and, while high, are consistent with a model trained and tested on the same synthetic scenes. The fragile step is the synthetic-to-real 3D evaluation. The paper explicitly discloses the proxy in the supplementary, which is transparent, but the disclosure does not convert view-level 2D segmentation metrics into 3D occupancy evidence. The central claim of predicting both the voxel occupancy state and panoptic labels on real data is only supported by these proxies. The proposed test, a 2D-only segmentation baseline trained on the same synthetic data, would settle whether the proxy is informative. If a 2D model matches OmniOcc, then the real-world 3D claim is not established; if not, the ray-marched 3D evaluation has some validity. The secondary concern about unequal supervision is real but less central to the dataset; it mainly weakens the architectural-superiority wording. I therefore do not change the reader's conditional verdict: the dataset may be accepted with conditions, but the real-world 3D claims should be relabeled or verified with standard 3D ground truth or a stronger proxy test.","tokens_in":18480,"tokens_out":6501,"duration_ms":57863,"concrete_test":"Train a standard 2D segmentation model (e.g., DeepLabV3 or PSPNet with ResNet-18 backbone, matched to OmniOcc's image resolution) on per-view 2D semantic and instance masks rendered from MVP-Occ ground-truth voxel occupancy, using the same training scenes and camera poses. Evaluate it on the manually annotated WildTrack 2D masks under exactly the same view-level metrics and thresholds as Table 6. If the 2D-only model achieves comparable mIoU/AP/PQ to OmniOcc's ray-marched predictions, then the reported '3D occupancy' numbers are explained by 2D segmentation ability and are not evidence of 3D accuracy. If OmniOcc clearly outperforms, the proxy has some 3D discriminative value.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 6 is titled '3D occupancy prediction under synthetic-to-real evaluation on WildTrack,' but the supplementary section 'Details about Synthetic-to-Real Evaluation' states that because WildTrack lacks 3D ground truth, 'we adopted an alternative approach by gauging 3D occupancy prediction performance with 2D segmentation labels.' Predicted voxels are ray-marched into per-view 2D masks and compared against manually annotated 2D semantic, instance, and panoptic masks, with metrics adapted from voxel-level IoU/AP/PQ to view-level segmentation (thresholds T={0.25,...,0.70}). The numbers in Table 6 are therefore 2D view-level segmentation scores, not 3D occupancy metrics. A model that solves 2D segmentation per view, without forming a geometrically consistent 3D volume, could in principle score well on this protocol, so the table does not establish accurate 3D occupancy on real data. A secondary issue is that Tables 3 and 5 compare OmniOcc, trained with voxel-level supervision, against ground-plane-only baselines; the margin does not isolate architectural superiority. The dataset contribution is credible; the unresolved point is the real-world 3D claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MVP-Occ, a synthetic multi-view dataset of five CARLA urban scenes with dense pedestrian crowds, providing voxel-level semantic occupancy labels (Free, Pedestrian, Ground, Wall, Others) as well as panoptic, instance, depth, and segmentation annotations. The authors also propose OmniOcc, a baseline model that predicts 2D bird's-eye-view pedestrian occupancy and 3D semantic, instance, and panoptic occupancy from multiple posed images. Experiments report same-scene results on MVP-Occ and synthetic-to-real transfer to WildTrack and CityStreet, and the paper claims state-of-the-art 2D pedestrian occupancy accuracy and strong 3D occupancy transfer performance.","tokens_in":18703,"tokens_out":5560,"duration_ms":51098,"significance":"If the dataset is released, it fills a real gap: existing multi-view pedestrian datasets such as WildTrack, MultiviewX, and GMVD provide only ground-plane locations, while MVP-Occ offers dense voxel-level occupancy and panoptic labels for large outdoor scenes. The dataset construction is described in unusual detail, including the fusion of monocular depth point clouds, the use of additional cameras to handle occlusion, and the labeling protocol. The OmniOcc baseline is simple and well ablated, and the authors provide multiple ablation tables (loss functions, semantic scene understanding, pedestrian instance grouping, location decoder, voxel encoder, backbone). The inclusion of external real-data tests on WildTrack and CityStreet is a positive feature that goes beyond self-evaluation. However, the paper's central claims of synthetic-to-real 3D occupancy performance and of superiority over prior detectors are weakened by two evaluation issues: the real-world 3D metrics are actually 2D segmentation proxies after ray marching, and the comparisons against prior methods are not controlled for the additional voxel-level supervision available to OmniOcc.","major_comments":[{"comment":"","section":"Supplementary, 'Details about Synthetic-to-Real Evaluation'; Table 6"},{"comment":"","section":"Tables 3 and 5; 'Same-Scene Evaluation' and 'Synthetic-to-Real Evaluation'"}],"minor_comments":[{"comment":"","section":"References"},{"comment":"","section":"Supplementary, 'Training on all scenes'"},{"comment":"","section":"Table 3 footnote"},{"comment":"","section":"Table 6 caption"},{"comment":"","section":"Supplementary, 'Details about Synthetic-to-Real Evaluation'"}],"recommendation":"major_revision","confidential_remarks":"The dataset contribution is solid and likely useful to the community. The main revision needed is to align the paper's claims with the actual evaluation protocol: the real-world 3D occupancy claim should be explicitly framed as a 2D-rendered proxy evaluation, and the comparisons against prior methods should be reframed as being with additional voxel supervision. These are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nWorth your attention: this paper's main contribution is the MVP-Occ dataset, not the OmniOcc model. The dataset provides voxel-level semantic, instance, and panoptic occupancy labels for dense multi-view pedestrian scenes, with five CARLA scenes, 12.5k frames, up to 100 pedestrians, plus depth, point clouds, and poses. That's a genuinely new resource compared to WildTrack, MultiviewX, and GMVD, which only give ground-plane locations. If the dataset is released, it could become a useful benchmark for surveillance and crowd analysis.\n\nThe model is a competent baseline: ImVoxelNet-style ray projection, 3D U-Net, weighted CE plus Lovasz and affinity losses, and instance grouping by distance to detected locations. Same-scene results on MVP-Occ are strong (mIoU 91.8–93.7, Pedestrian IoU ~63–70). They also run synthetic-to-real transfer to WildTrack and CityStreet, showing large gains in 2D detection F1 over prior methods.\n\nThe soft spot is real, and the paper's own supplement confirms it: Table 6 is titled \"3D occupancy prediction under synthetic-to-real evaluation on WildTrack,\" but the evaluation actually ray-marches predicted voxels into per-view 2D masks and compares those to manually annotated 2D segmentation. Those are view-level 2D segmentation scores, not voxel-level 3D occupancy metrics. So the table does not certify 3D transfer. The authors are transparent about this, but the framing in the main text is misleading. A secondary issue: OmniOcc trains with voxel supervision while baselines only see ground-plane points, so the headline gains in Tables 3 and 5 do not isolate architecture; they partly reflect a supervision advantage. The authors note this in captions, but it weakens the SOTA claim.\n\nThe dataset contribution stands. The proxy evaluation is a known limitation, not a hidden flaw. I'd send this to a serious reviewer with the expectation that the real-world 3D claims get relabeled as 2D proxy metrics or replaced with clear qualifiers, and that same-supervision baselines be added if they want to claim architectural gains. The paper is honest and well-structured. I'd cite the dataset if I worked in this area.\n\nMy take: accept with major revision, or conditionally if the venue allows. It's a solid dataset paper with an honest but overstated evaluation section.","headline":"MVP-Occ is a genuinely useful new dataset for dense multi-view pedestrian occupancy; the OmniOcc baseline is competent, but the real-world 3D numbers are 2D proxy scores, so treat that claim with caution.","tokens_in":19268,"tokens_out":2344,"would_cite":true,"duration_ms":20666,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MVP-Occ is a synthetic multi-view dataset with voxel-level pedestrian occupancy and panoptic labels, and OmniOcc predicts both 2D and 3D occupancy from camera images, transferring to real surveillance footage.","keywords":["multi-view pedestrian detection","occupancy prediction","voxel occupancy","panoptic segmentation","synthetic dataset","synthetic-to-real transfer","semantic scene understanding","benchmark dataset"],"falsifier":"Train OmniOcc on MVP-Occ and evaluate it on a real multi-camera scene that has genuine 3D occupancy ground truth, for example voxelized LiDAR or dense multi-view reconstructions, then compare voxel IoU with the paper's ray-marched mask metric; if the voxel IoU is much lower than the mask-based score, the proxy overstates 3D accuracy.","tokens_in":18261,"feed_emoji":"🧍","tokens_out":9555,"duration_ms":76447,"temperature":0.7,"pith_summary":"This paper proposes a new task target for multi-view pedestrian understanding: instead of only predicting locations on the ground plane, predict the occupancy state of every voxel in a large scene, together with semantic, instance, and panoptic labels. To make that possible, it introduces MVP-Occ, a CARLA-generated synthetic dataset of five dense urban scenes with 10 cm voxel annotations (Free, Pedestrian, Ground, Wall, Others), depth, point clouds, and per-person pose and segmentation labels. The paper also builds OmniOcc, a baseline that projects multi-view image features into a voxel grid, processes them with a 3D U-Net, and outputs 2D ground-plane occupancy plus 3D semantic, instance, and panoptic occupancy. On same-scene evaluation OmniOcc reports high voxel IoU and detection scores, and on synthetic-to-real transfer to WildTrack and CityStreet it reports substantially higher F1 than ground-plane-only detectors. The practical stake is a benchmark that lets occupancy prediction and scene understanding be evaluated together in crowded, camera-surveillance-like environments.","feed_headline":"New voxel-level dataset predicts pedestrian occupancy from cameras","feed_subtitle":"A five-scene synthetic benchmark with voxel labels, plus a baseline that transfers to real surveillance footage.","key_machinery":"The central mechanism is a non-parametric view transformer that distributes each 2D image feature along the camera ray into a voxel grid, averaging the feature samples from all views that see the voxel; this avoids learned cross-view attention and keeps the model compatible with variable camera counts and scene sizes. A 3D U-Net then refines the voxel feature volume, a two-layer MLP classifies each voxel into Free, Pedestrian, Ground, Wall, or Others, and a separate head averages along the vertical axis and applies one convolution to produce the 2D ground-plane occupancy heatmap. Pedestrian instance grouping assigns each pedestrian-labeled voxel to the nearest detected location within 50 cm, turning the semantic volume into instance and panoptic occupancy. The accompanying weighted cross-entropy, Lovász-Softmax, and scene-class affinity losses balance the dominant Free voxels and sharpen pedestrian boundaries, and the ablation results show that these pieces jointly carry the reported accuracy.","core_discovery":"In the paper's own terms, the discovery is that voxel-level pedestrian occupancy can be predicted from multiple posed cameras with high accuracy, and that doing so transfers to real scenes much better than ground-plane localization alone. MVP-Occ is claimed to be the first multi-view pedestrian dataset to provide voxel occupancy and panoptic annotations for large-scale dense scenes: 12,500 frames across five scenes with up to 100 pedestrians per scene and 10 cm voxels. OmniOcc is claimed to outperform the prior multi-view detection baselines MVDet, SHOT, GMVD, and MVFP on the 2D occupancy benchmark, reaching an average F1 of 96.8 in same-scene evaluation, and to exceed them on synthetic-to-real transfer, reaching 87.5 F1 from the Facade scene to WildTrack and 89.7 F1 when trained on all MVP-Occ scenes. The paper further argues that semantic scene understanding, trained through the voxel labels, is a load-bearing component: removing it drops 2D pedestrian F1 from 87.5 to 81.6 and instance AP from 40.2 to 38.2 in the WildTrack transfer setting.","pith_inferences":["Editorial inference: the same voxel-occupancy formulation should extend to other dynamic classes such as vehicles or cyclists in multi-camera surveillance, since only the instance-grouping step is pedestrian-specific; a direct test would be annotating a multi-class version of a scene and retraining the baseline.","Editorial inference: if voxel supervision is what enables transfer, then densifying real-world labels through LiDAR voxelization or multi-view reconstruction, rather than manual 2D masks, should improve measured 3D accuracy further; the paper does not test this.","Editorial inference: the paper's own future-work mention of neural-rendering supervision implies a path where 3D occupancy is learned from 2D labels alone; a testable consequence is that OmniOcc-style models could train without any voxel ground truth, which the current experiments do not isolate.","Editorial inference: because the manual 2D-mask proxy may be insensitive to small voxel errors, an independent benchmark with true voxel ground truth would likely rank methods differently than the reported mask-based metrics."],"forward_implications":["MVP-Occ gives the multi-view pedestrian community a common benchmark with voxel-level semantic, instance, and panoptic labels, so future methods can be compared on scene understanding and occupancy, not only ground-plane locations.","If OmniOcc's transfer results hold, synthetic training with voxel supervision can substitute for expensive real-world 3D labels in dense pedestrian surveillance.","The Facade scene, designed to resemble WildTrack, achieves 87.5 F1 in direct transfer, and training on all five scenes raises this to 89.7, implying that scene replication plus diversity is a practical recipe.","OmniOcc's simple components—averaging instead of learned compression, a single convolutional head instead of dilated or LKR heads—suggest that extra architectural complexity is not what drives the gains.","Because the paper reports 57.3 MODA and 78.0 F1 on CityStreet after training on MVP-Occ, the dataset generalizes beyond its closest WildTrack analog to a larger, noisier real scene."],"supporting_citations":[{"why":"Real-world test bed for synthetic-to-real transfer; the paper adds manual 2D segmentation masks to evaluate occupancy because WildTrack lacks 3D labels.","marker":"Chavdarova et al. 2018"},{"why":"Provides the MVDet baseline and the MODA/MODP/F1 evaluation metrics reused for 2D pedestrian occupancy.","marker":"Hou, Zheng, and Gould 2020"},{"why":"Provides the GMVD baseline and dataset; the paper shows it fails to transfer to WildTrack, motivating scene-aware occupancy.","marker":"Vora et al. 2023"},{"why":"Provides the MVFP baseline that uses 3D feature pulling, the LKR module compared in the location-decoder ablation, and a cross-scene comparison point.","marker":"Aung et al. 2024"},{"why":"Supplies Human-M3, a multi-scene outdoor multi-view dataset that motivates the MVP-Occ scene design and cross-scene evaluation.","marker":"Fan et al. 2023"},{"why":"Simulator used to render the five MVP-Occ scenes, the RGB/depth/semantic/instance sensors, and pedestrian behavior.","marker":"Dosovitskiy et al. 2017"},{"why":"Supplies the non-parametric image-to-voxel projection used by the view transformer, letting OmniOcc handle variable camera configurations.","marker":"Rukhovich, Vorontsova, and Konushin 2022"},{"why":"Supplies the scene-class affinity loss and the semantic-scene-completion evaluation conventions adapted to occupancy evaluation.","marker":"Cao and de Charette 2022"},{"why":"Defines the panoptic quality metric (PQ/SQ/RQ) used for 3D panoptic occupancy evaluation.","marker":"Kirillov et al. 2019"}],"fun_headline_variants":["OmniOcc predicts voxel pedestrian occupancy from multi-view cameras","MVP-Occ: first voxel-level multi-view pedestrian dataset","Voxel occupancy prediction beats ground-plane baselines in transfer","New dataset and model for dense pedestrian occupancy from cameras","Multi-view voxel occupancy: synthetic training transfers to real scenes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation of 3D occupancy on real data assumes that agreement between ray-marched 2D segmentation masks and manually annotated masks faithfully measures 3D voxel correctness; if that proxy is not faithful, the reported synthetic-to-real 3D metrics do not establish the 3D occupancy claim.","fun_headline_variants_meta":{"raw":{"variants":["OmniOcc predicts voxel pedestrian occupancy from multi-view cameras","MVP-Occ: first voxel-level multi-view pedestrian dataset","Voxel occupancy prediction beats ground-plane baselines in transfer","New dataset and model for dense pedestrian occupancy from cameras","Multi-view voxel occupancy: synthetic training transfers to real scenes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000154,"raw_usage":{"total_tokens":1183,"prompt_tokens":891,"completion_tokens":292,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":208}},"tokens_in":507,"tokens_out":292,"duration_ms":2972,"temperature":1.0,"reasoning_tokens":208,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:59:51.797576+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train OmniOcc on MVP-Occ and evaluate it on a real multi-camera scene that has genuine 3D occupancy ground truth, for example voxelized LiDAR or dense multi-view reconstructions, then compare voxel IoU with the paper's ray-marched mask metric; if the voxel IoU is much lower than the mask-based score, the proxy overstates 3D accuracy.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Real-world test bed for synthetic-to-real transfer; the paper adds manual 2D segmentation masks to evaluate occupancy because WildTrack lacks 3D labels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MVDet baseline and the MODA/MODP/F1 evaluation metrics reused for 2D pedestrian occupancy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the GMVD baseline and dataset; the paper shows it fails to transfer to WildTrack, motivating scene-aware occupancy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MVFP baseline that uses 3D feature pulling, the LKR module compared in the location-decoder ablation, and a cross-scene comparison point."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the non-parametric image-to-voxel projection used by the view transformer, letting OmniOcc handle variable camera configurations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the scene-class affinity loss and the semantic-scene-completion evaluation conventions adapted to occupancy evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the panoptic quality metric (PQ/SQ/RQ) used for 3D panoptic occupancy evaluation."}],"review_version":1}