{"id":"5caa4cc9-97b4-45df-bd20-d1a01ee7b1de","arxiv_id":"2412.01147","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A2VIS integrates amodal, full-shape masks into video instance segmentation via global prototypes and a spatiotemporal-prior mask head, improving occlusion-robust tracking on synthetic benchmarks.","lead":"A computer vision framework, A2VIS, adds amodal segmentation (full-object masks including hidden parts) to video instance segmentation, with a spatiotemporal-prior masked attention head to keep track of objects through occlusion. On the FISHBOWL and SAIL-VOS synthetic benchmarks, it reports better tracking and segmentation scores than several prior VIS and MOT methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Amodal ground truth is frame-confined and begins only at first visible frame, so the experiments never test full disappearance/reappearance or out-of-frame occlusion; the central claim of robustness to complete occlusion overreaches.","rationale":"The reader's weakest assumption correctly identifies that the amodal supervision definition excludes out-of-frame and disappear/reappear cases, and that this narrows the scope of the claimed occlusion robustness. My stress-test concurs and sharpens the concern: the same definition also truncates the amodal target for objects that exit the frame, so the model is actively trained to ignore out-of-view extent. This is not an external disagreement with consensus; it is an internal mismatch between the paper's stated contribution ('partially or completely hidden from view') and the evidence it can possibly provide. The paper has real independent support: the SAMH ablation (Table 8) and the VSPM/ASPM ablation (Table 4) show consistent gains, and the comparison against amodal baselines (Table 2) is appropriate. These support the claim that the proposed architecture helps under the tested in-frame occlusion regime. They do not, however, establish robustness to full occlusion with reappearance, which is the regime most relevant to real-world tracking failures. Because the limitation is acknowledged by the authors and the reader's verdict is already CONDITIONAL, I do not recommend changing the verdict; I recommend keeping it conditional on evidence that the method generalizes beyond the truncated amodal supervision.","tokens_in":16868,"tokens_out":4226,"duration_ms":43664,"concrete_test":"Construct a subset of OVIS (or of the SAIL-VOS test set, if annotations permit) consisting only of object tracks that have at least one gap of 5 or more consecutive frames with no visible mask, followed by reappearance of the same identity. Run the released A2VIS and GenVIS checkpoints on this subset and compare HOTA, IDF1, and IDs. If A2VIS does not retain its IDF1/IDs advantage over GenVIS on this subset, the claim that amodal awareness confers robustness to complete occlusion is unsupported. A complementary synthetic check is to generate FISHBOWL variants where objects are translated out of frame for whole clips, with amodal ground truth extended beyond the frame boundary, and retrain/evaluate on those variants.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that amodal awareness makes tracking robust when objects are 'partially or completely hidden from view.' The load-bearing assumption is that the amodal supervision used to train and evaluate A2VIS actually captures complete object shape under occlusion. Section 3.1 violates this in two ways: (i) Ai[t'] = ∅ whenever the instance has not been visibly present from the start of the video, and (ii) the amodal mask is confined to the frame size. Thus for an object that exits the frame or appears only after the video starts, the 'amodal' target is truncated or absent: the model is never asked to predict the full shape, only the in-frame portion after first visible appearance. Consequently, FISHBOWL and SAIL-VOS evaluations in Tables 1, 2, 3, and 9 exercise only in-frame partial occlusion, not the full occlusion and reappearance failure mode that the abstract and introduction emphasize. The paper's own Limitation section confirms this: A2VIS 'does not explicitly account for objects that are occluded by being partly or completely out of the frame or disappear in one frame and reappear in another.' The reported IDF1/IDs gains may be genuine for in-frame occlusion, but they do not support the broader claim of 'reliable and comprehensive understanding' of occluded objects in real-world tracking, where exit/re-entry is a dominant failure mode.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces A2VIS, a video instance segmentation framework that augments instance prototypes with amodal masks, using a Spatiotemporal-Prior Amodal Mask Head (SAMH) that combines short-range visible priors (VSPM) and long-range amodal priors (ASPM) via masked attention. The method is trained with both visible and amodal mask losses and evaluated on FISHBOWL and SAIL-VOS for VIS, amodal VIS, and MOT, plus a zero-shot OVIS evaluation. The authors report consistent improvements over published baselines and ablations showing the contribution of both spatiotemporal priors.","tokens_in":17158,"tokens_out":8251,"duration_ms":67518,"significance":"If the reported results are reliable, the paper demonstrates a useful inductive bias: incorporating amodal segmentation into instance prototypes improves occlusion robustness in video instance segmentation and tracking. The method is technically novel in combining spatiotemporal masked attention for amodal prediction with global instance prototypes, and the ablations support the role of the two priors. The paper also introduces a new set of amodal-VIS baselines. However, the evaluation scope is narrower than the abstract claims, and the lack of statistical rigor and validation-based hyperparameter selection weakens the strength of the conclusions; the absence of code is an additional reproducibility concern.","major_comments":[{"comment":"The amodal supervision is defined as frame-confined and empty until first visible appearance (§3.1), and the Limitation explicitly states that out-of-frame occlusion and disappear-and-reappear cases are not handled. Yet the Abstract and Introduction claim a 'reliable and comprehensive understanding' of objects that are 'partially or completely hidden', and Figure 1 promises 'complete trajectory and shape'. As a result, the experimental evidence in Tables 1-3 and 9 supports improvements for in-frame partial and full occlusion by other objects, but not for exit/re-entry or out-of-frame occlusion, which are dominant real-world tracking failure modes. Please either temper the claims to in-frame occlusion or provide experiments that cover complete disappearance and reappearance.","section":"§3.1, Limitation, Abstract"},{"comment":"All main results are reported as single runs, with no error bars, confidence intervals, or significance tests, making the phrase 'significant performance gap' in §4.4.1 unsupported. Moreover, the hyperparameters Nc, L, and the number of convolution layers are chosen based on ablations in Tables 5-7, yet there is no mention of a validation split; if the ablations were performed on the test set, the reported numbers are optimistically biased. Please report mean and variance over multiple seeds and clearly state the protocol for model selection.","section":"§4.4.1, Tables 1-8"},{"comment":"The amodal baselines (Mask2Former-Amodal, VITA-Amodal, GenVIS-Amodal) are trained with amodal supervision alone, whereas A2VIS is trained with both visible and amodal losses (Eq. 4). This gives A2VIS additional supervision and makes the comparison in Table 2 not an apples-to-apples test of the proposed SAMH. To isolate the contribution of the amodal head, the authors should include multi-task baselines that are also trained with both visible and amodal masks, or otherwise match the supervision available to each method.","section":"§4.3, Table 2"}],"minor_comments":[{"comment":"Algorithm 1 uses ρ(Fk) for the Amodal Feature Extraction, while the text refers to Ω; please unify the notation.","section":"§3.5, Algorithm 1"},{"comment":"The description of the amodal baseline training is confusing: 'all the models are pretrained with frame-level FISHBOWL and SAIL VOS datasets on FISHBOWL dataset with amodal segmentation ground truth' seems to contain a typo and should be rewritten.","section":"§4.3"},{"comment":"The MOT baseline list mentions TrackFormer and MOTR, but Table 3 reports MOTRv2; please add MOTRv2 to the baseline description for consistency.","section":"§4.3, Table 3"},{"comment":"The caption contains the typo 'quatitatively'; it should read 'qualitatively'.","section":"§4.4.1, Figure 6 caption"},{"comment":"The implementation details state that a frame-level Mask2Former is pretrained and then video-level training is performed; please explain why this two-stage procedure is needed and how it interacts with the amodal supervision.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The core idea is promising and the experimental trend is consistent, but the paper needs stronger experimental methodology (validation-based hyperparameter selection, error bars, fair amodal baselines) and a more careful alignment between claims and evaluation scope. The absence of code is also a concern for reproducibility, though not fatal by itself."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: A2VIS is a solid incremental extension of VITA/GenVIS that adds amodal supervision to VIS, and the SAMH design is sensible and ablated reasonably. The track-and-segmentation gains are consistent on two synthetic datasets. The weakness is not the method but the claim: the evaluation only ever exercises partial, in-frame occlusion, so the abstract's 'completely hidden' language oversells it. The paper's own Limitation section admits this, but the framing elsewhere doesn't.\n\nWhat's actually new: first end-to-end multi-object VIS that predicts amodal masks as part of the tracking pipeline, using global instance prototypes to carry amodal info across clips. The spatiotemporal masked attention (VSPM+ASPM) is a neat way to use visible parts of near frames plus previous amodal predictions. Ablations in Tables 4 and 8 support both components. The zero-shot OVIS result is a nice extra, though it only evaluates visible tracking under two categories and no amodal GT.\n\nSoft spots, in order of severity.\n\n1. The central claim about robustness to 'completely hidden' or reappearing objects is not what is measured. The amodal GT is frame-confined and only starts at first visible frame, so full disappearing/reappearing or out-of-frame behavior is excluded. The stress-test note is right: the paper's limitation section says this explicitly. That is an honest limitation, but it means the headline 'reliable and comprehensive understanding' of occluded objects is only supported for partial in-frame occlusion. That's still useful for autonomous driving and surveillance, but not the complete claim.\n\n2. No code or released models. For a method built directly on VITA/GenVIS, that makes reproduction unnecessarily hard.\n\n3. No error bars or significance tests on the main tables. With 1,000 FISHBOWL test videos the differences are probably real, but AP 41.77 vs 40.04 is a modest gap and could move.\n\n4. Hyperparameters (clip length, L, conv layers) are selected by running ablations on the same benchmarks they report. That's mild overfitting to the test set; the paper doesn't mention holding out a validation split.\n\nI'm less bothered by the novelty concern the reader raised: yes, it's prototype-based VIS plus an amodal head, but that's exactly what 'first' claims usually look like, and the authors position it fairly.\n\nWho is this for? People working on occlusion-aware tracking or amodal segmentation. It deserves a serious referee — the method is well described, the experiments are internally consistent, and the limitation is disclosed, so a reviewer could demand corrections to the framing rather than a rejection. If I were editing, I'd send it out and ask for: code or at least detailed training configs, error bars, and a toned-down abstract.","headline":"Solid but narrower than claimed: the gains are real for partial in-frame occlusion, not for full disappearance/reappearance.","tokens_in":17689,"tokens_out":2784,"would_cite":true,"duration_ms":24155,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A2VIS claims that predicting full object shapes, including occluded parts, makes video instance tracking more stable and cuts identity switches on occlusion-heavy benchmarks.","keywords":["amodal segmentation","video instance segmentation","multiple object tracking","occlusion handling","global instance prototype","spatiotemporal masked attention","FISHBOWL","SAIL VOS"],"falsifier":"Run A2VIS and the GenVIS-style baselines on a benchmark where objects are frequently fully occluded by other objects or leave the frame entirely and then return, with identity annotations for reappearance; if A2VIS's identity-switch advantage disappears or reverses, the claim that amodal awareness stabilizes tracking beyond in-frame partial occlusion fails. A cheaper test is to add out-of-frame amodal annotations to FISHBOWL or SAIL VOS and check whether training on them changes the reported gaps.","tokens_in":16658,"feed_emoji":"👁️","tokens_out":6898,"duration_ms":56421,"temperature":0.7,"pith_summary":"The paper introduces A2VIS, a video instance segmentation framework that predicts, for each object, both its visible mask and its amodal mask, meaning the full shape including parts hidden by occlusion. The central claim is that feeding amodal masks into the tracking process makes instance identity more stable over time, because an object's full shape changes less abruptly than its visible outline during occlusion. A2VIS encodes this by compressing each object into a global instance prototype that accumulates amodal information across the whole video, and by using a dedicated head, SAMH, that draws on both nearby visible frames and long-range amodal knowledge. On FISHBOWL and SAIL VOS the authors report gains over prior VIS and amodal-VIS baselines, including higher IDF1 and fewer identity switches. If the claim holds, amodal awareness is a practical inductive bias for occlusion-robust tracking, not just a way to render completed masks.","feed_headline":"Amodal masks cut identity switches in video tracking","feed_subtitle":"A2VIS predicts full object shapes and beats GenVIS on FISHBOWL and SAIL VOS tracking metrics.","key_machinery":"The load-bearing object is the Spatiotemporal-prior Amodal Mask Head (SAMH), a mask-decoding module that predicts amodal masks by attending to two spatiotemporal-prior masks: VSPM, built from visible segmentations inside the current clip, and ASPM, built from amodal segmentations accumulated in the global instance prototypes across clips. The attention operation is $\\mathbf{p}^G_l = \\mathrm{softmax}(T^k + QK^\\top)V + \\mathbf{p}^G_{l-1}$, where $T^k$ is the union of VSPM and ASPM, so the head can only look at locations where the instance is known to be visible locally or amodally globally. Alongside SAMH, the global instance prototypes act as a compressed, video-wide memory that keeps each object's identity and full-shape information, updated clip by clip through cross-attention. Together they carry the paper's argument that amodal information stabilizes tracking: SAMH supplies the full-shape predictions, and the prototypes propagate that information across time.","core_discovery":"The paper's discovery candidate is that amodal segmentation is a useful representation for video instance tracking, not merely an output task. A2VIS treats the video as a sequence of clips, maintains a set of global instance prototypes that represent every object across the whole video, and decodes both visible and amodal masks from those prototypes. The amodal masks are produced by the Spatiotemporal-prior Amodal Mask Head, whose masked attention uses a visible spatiotemporal-prior mask (short-range, from visible segments in nearby frames) and an amodal spatiotemporal-prior mask (long-range, from accumulated amodal knowledge in the global prototypes). Empirically, the paper reports that A2VIS surpasses the prior GenVIS baseline on FISHBOWL and SAIL VOS across segmentation and bounding-box tracking metrics, and that the amodal-aware design reduces identity switches; ablation studies attribute the gain to the combination of the two spatiotemporal priors. The paper also introduces amodal extensions of VIS baselines and shows A2VIS ahead of them. The reported gains are confined to in-frame occlusions, because the amodal annotations in FISHBOWL and SAIL VOS clip masks to the frame and do not cover objects that disappear and reappear.","pith_inferences":["If the reported gains come from amodal masks being more temporally stable than visible masks, then the same mechanism should help any tracking-by-query method, not only prototype-based VIS; a direct test is to add SAMH to MOTR-style query trackers and compare identity-switch rates.","The paper leaves out-of-frame occlusion untested; a natural extension is to let amodal masks extend beyond the frame boundary using an external memory of object appearance, which would address the main real-world failure mode.","Because the authors report results on synthetic and game-derived datasets plus a zero-shot real-world probe, the strongest validation would come from a real-world dataset with dense amodal annotations; until such data exists, the practical benefit under realistic heavy occlusion remains an open question.","The ablation showing SAMH improves even visible-only metrics suggests amodal supervision acts as a regularizer; one could test whether weaker forms of full-shape supervision, such as amodal bounding-box expansion, give part of the same tracking gain at lower annotation cost."],"forward_implications":["On FISHBOWL with ResNet-50, A2VIS raises AP from 40.04 to 41.77 and IDF1 from 50.08 to 52.14 while cutting identity switches from 3480 to 3392 relative to GenVIS.","On SAIL VOS with ResNet-50, A2VIS improves AP to 23.12, HOTA to 30.04, and IDF1 to 25.94 relative to GenVIS, with fewer identity switches.","Under occlusion rates above 50%, A2VIS keeps an advantage over GenVIS-Amodal on FISHBOWL, with AP 33.14 versus 29.78.","Ablations show that both the visible spatiotemporal-prior mask and the amodal spatiotemporal-prior mask contribute to the gain; removing either lowers amodal VIS performance.","Adding SAMH improves even visible-only VIS metrics, with FISHBOWL AP rising from 39.94 to 41.77, suggesting amodal supervision helps ordinary tracking, not just amodal mask prediction."],"supporting_citations":[{"why":"supplies the inter-clip association baseline that A2VIS extends and outperforms","marker":"[14]"},{"why":"provides the object-token association used to build clip-based instance prototypes","marker":"[15]"},{"why":"introduces the SAIL VOS amodal video dataset that provides amodal ground truth for training and evaluation","marker":"[16]"},{"why":"provides the FISHBOWL synthetic dataset used for training and evaluation","marker":"[35]"},{"why":"introduces spatiotemporal-prior amodal video segmentation that motivates the SAMH design","marker":"[43]"},{"why":"supplies the transformer-based amodal mask head design that SAMH adapts for feature extraction","marker":"[37]"},{"why":"provides the masked-attention mask transformer used for visible segmentation and frame-level pretraining","marker":"[7]"},{"why":"supplies the OVIS occlusion benchmark used for the zero-shot real-world evaluation","marker":"[30]"}],"fun_headline_variants":["Amodal masks keep video identities stable","Full-shape predictions cut tracking identity swaps","Occlusion-proof instance tracking with amodal masks","Predict occluded shapes to stop ID switches","Amodal awareness reduces tracking identity flips"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the amodal masks in the training data correctly represent full object shape under the occlusions that matter; the datasets only annotate amodal masks inside the frame and never for objects that leave the frame or disappear for a stretch, so the method's claimed robustness to occlusion is demonstrated only for objects that remain partially visible within the frame.","fun_headline_variants_meta":{"raw":{"variants":["Amodal masks keep video identities stable","Full-shape predictions cut tracking identity swaps","Occlusion-proof instance tracking with amodal masks","Predict occluded shapes to stop ID switches","Amodal awareness reduces tracking identity flips"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000375,"raw_usage":{"total_tokens":2021,"prompt_tokens":985,"completion_tokens":1036,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":969}},"tokens_in":601,"tokens_out":1036,"duration_ms":8421,"temperature":1.0,"reasoning_tokens":969,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:38:21.088395+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run A2VIS and the GenVIS-style baselines on a benchmark where objects are frequently fully occluded by other objects or leave the frame entirely and then return, with identity annotations for reappearance; if A2VIS's identity-switch advantage disappears or reverses, the claim that amodal awareness stabilizes tracking beyond in-frame partial occlusion fails. A cheaper test is to add out-of-frame amodal annotations to FISHBOWL or SAIL VOS and check whether training on them changes the reported gaps.","supporting_citations":[{"cited_title":"A generalized framework for video instance segmentation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp","cited_arxiv_id":null,"evidence_quote":"supplies the inter-clip association baseline that A2VIS extends and outperforms"},{"cited_title":"Vita: Video instance segmentation via object token association","cited_arxiv_id":null,"evidence_quote":"provides the object-token association used to build clip-based instance prototypes"},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"introduces the SAIL VOS amodal video dataset that provides amodal ground truth for training and evaluation"},{"cited_title":"Self-supervised amodal video object segmentation","cited_arxiv_id":null,"evidence_quote":"introduces spatiotemporal-prior amodal video segmentation that motivates the SAMH design"},{"cited_title":"Occluded video instance segmentation: A benchmark","cited_arxiv_id":null,"evidence_quote":"supplies the OVIS occlusion benchmark used for the zero-shot real-world evaluation"}],"review_version":1}