{"id":"6e5586a8-6e28-4078-89d4-4813413757a2","arxiv_id":"2411.13056","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"E-MAC, with density-guided masked modeling and optical-flow temporal fusion, achieves state-of-the-art MAE on four video counting benchmarks, including the new DroneBird bird dataset.","lead":"The authors propose E-MAC, a video object counting model that learns from both image frames and density maps inside a masked autoencoder, and they release DroneBird, a new drone-captured bird counting benchmark with millions of point annotations. The model reports the best counting error on Mall, FDST, VSCrowd, and the new dataset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 baselines are not protocol-matched: E-MAC uses fixed sigma=6 density maps (Sec 5.1) while cited baselines used their own kernels and preprocessing, so the claimed SOTA margins on Mall, FDST, and VSCrowd may be evaluation artifacts.","rationale":"The reader's weakest assumption is the most load-bearing concern: the comparison to prior results is not controlled for the density-map generation and evaluation protocol. The paper explicitly uses a single fixed Gaussian kernel (sigma=6) across datasets, while most earlier methods cited in Table 1 report numbers from their own protocols. Since the metric is computed against the ground-truth density map, the baseline numbers are not directly comparable. This is especially critical for the small margins on Mall (4%) and FDST (7%), where a protocol shift could easily invert the ranking. The paper does provide some supporting evidence: ablations show each component contributes, and Table 11 reports seed variance on FDST (1.30 +/- 0.022). However, those ablations do not address the protocol mismatch, and the variance is only reported for FDST. The DroneBird results are less affected because the authors evaluated baselines on their own dataset, but they still do not report error bars for those baselines. A secondary concern is the train/inference mismatch in SAM: during training the encoder sees a subset of image tokens (selected by density), but at inference it sees all tokens (N_ret_I = NI). This could make inference out-of-distribution, though it does not by itself invalidate the reported numbers; it mainly affects the method's soundness and reproducibility. The most direct path to confirming or refuting the central claim is protocol-matched retraining of baselines. The reader's CONDITIONAL verdict is appropriate: the paper is promising but the headline result is not yet robustly established.","tokens_in":22053,"tokens_out":7473,"duration_ms":71267,"concrete_test":"Retrain or re-evaluate the main baselines (at least FRVCC, STGN, MAN, PET, and CSRNet) on Mall, FDST, and VSCrowd using the same fixed sigma=6 density-map generation, the same evaluation script, and the same train/test splits as E-MAC; then recompute Table 1. If any baseline's MAE drops below E-MAC's, or if the margins shrink to within noise, the SOTA claim fails. Releasing the evaluation code and baseline checkpoints would make this check straightforward.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that E-MAC achieves state-of-the-art MAE/RMSE on all four benchmarks rests on Table 1, which mixes numbers from the authors' runs (DroneBird) with numbers copied from prior papers (Mall, FDST, VSCrowd). Section 5.1 states: 'We use a fixed Gaussian kernel (sigma = 6) to generate the ground-truth density map on these datasets.' However, the cited baselines were originally evaluated with their own density-map generation protocols (e.g., geometry-adaptive kernels in CSRNet, MCNN, and MAN) and potentially different evaluation resolutions. Because MAE/RMSE are computed against ground-truth density maps, changing the kernel changes the metric even for identical predictions. The reported gains on Mall (1.35 vs 1.41, 4%) and FDST (1.29 vs 1.38, 7%) are small enough that a protocol shift could reverse them. The paper does not retrain any baseline under the sigma=6 protocol, and it does not report per-dataset error bars for baselines. Without protocol-matched baselines, the 'significant advantages' claim is not settled.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes an Efficient Masked Autoencoder Counting (E-MAC) framework for video object counting. The method combines three components: density-embedded masked modeling (DEMO), which treats the density map as an auxiliary modality during masked autoencoding; spatial adaptive masking (SAM), which prunes background tokens based on the density map; and temporal collaborative fusion (TCF), which uses optical flow to warp the previous frame's predicted density map and compute temporal residuals. The authors also introduce DroneBird, a large drone-captured video dataset with 3.69M bird annotations and trajectory annotations. Experiments on Mall, FDST, VSCrowd and DroneBird report state-of-the-art MAE/RMSE, with the best MAE of 1.35, 1.29, 6.0 and 38.72, respectively.","tokens_in":22292,"tokens_out":4733,"duration_ms":41458,"significance":"The dataset contribution is substantial: DroneBird appears to be the first drone-view video bird counting dataset with both point and trajectory annotations, containing 3.69M annotations, and the authors make code and data available. The methodological idea of using the density map as a second modality in masked autoencoding is novel and well motivated by the fore-background imbalance problem. The ablation study is systematic and includes a random-seed robustness check on FDST. If the reported gains survive protocol-matched evaluation, the work would be a meaningful advance for video object counting. However, the current empirical evidence for state-of-the-art performance is weakened by the evaluation protocol mismatch described below.","major_comments":[{"comment":"Section 5.1 states that 'We use a fixed Gaussian kernel (sigma = 6) to generate the ground-truth density map on these datasets,' yet the baseline MAE/RMSE values for Mall, FDST and VSCrowd in Table 1 are quoted from earlier papers that used their own density-map generation protocols (e.g., geometry-adaptive kernels in CSRNet/MAN). Because MAE and RMSE are computed against ground-truth density maps, changing the kernel changes the regression target and the metric even for identical predictions. The margins on Mall (1.35 vs 1.41) and FDST (1.29 vs 1.38) are small enough that a kernel/protocol shift could plausibly reverse them. Please retrain or re-evaluate all baselines under the exact sigma=6 protocol (and the same input resolutions and preprocessing), or present the comparisons only for protocol-matched runs.","section":"Section 5.1, Table 1"},{"comment":"The hyperparameters P, the mask ratio, and the loss weights {lambda_1..lambda_4} are tuned via experiments on FDST in Section 5.4 and Figure 5, after which FDST is reported as a headline result in Table 1. If these experiments used the FDST test frames, the reported 1.29/1.69 is a test-set-tuned estimate and is not an unbiased measure of generalization. Please clarify whether the tuning was done on a held-out validation split; if not, report results with hyperparameters fixed a priori or tuned only on validation data.","section":"Section 5.4, Tables 2-3"},{"comment":"The paper does not state which baseline numbers were produced by the authors' own re-runs and which were transcribed from prior publications. The DroneBird rows presumably come from the authors' implementations, whereas Mall/FDST/VSCrowd rows appear to be copied from earlier papers with heterogeneous protocols. This mixed provenance makes it impossible for the reader to separate algorithmic gains from evaluation-configuration gains. Please provide a source statement for every number in Table 1, along with the exact evaluation code and, where feasible, standard deviations over at least three runs.","section":"Table 1, Section 5.2"}],"minor_comments":[{"comment":"The RMSE value for the proposed method is written as '42,92'; it should be '42.92'.","section":"Table 12"},{"comment":"Subfigures (c)-(f) lack axis labels identifying which loss weight is being swept, and the caption does not state the fixed values of the other weights; please label each subfigure and state the fixed settings.","section":"Figure 5"},{"comment":"The number of retained density tokens N_ret^D and its relationship to N_ret^I (including how the Dirichlet distribution determines them) are mentioned in the appendix but not defined in Section 4.3; please define both quantities in the main text.","section":"Algorithm 2, Section 4.3"},{"comment":"The dataset name is spelled 'VScrowd' in the text (e.g., 'VScrowd collected more videos') but 'VSCrowd' elsewhere; please unify the spelling.","section":"Section 5.2"},{"comment":"Equation (6) introduces lambda_1..lambda_4 without mapping them explicitly to L_fuse, L_cur, L_opt, and L_TV; a one-sentence mapping would improve readability.","section":"Equation (6)"}],"recommendation":"major_revision","confidential_remarks":"The main risk is that the state-of-the-art claim rests on protocol-mismatched baselines; this is fixable with additional experiments or by softening the claim. The dataset and code release are valuable independently of the benchmark numbers. The FDST hyperparameter-tuning issue should be clarified before publication. I would support acceptance after a major revision that provides matched baseline evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the DroneBird dataset is the strongest part of this paper. The E-MAC counting framework is a coherent combination of known ideas, and the SOTA claims are real but weaker than the reported margins suggest because the baselines on Mall, FDST, and VSCrowd are borrowed numbers from different density-map protocols.\n\nWhat's genuinely new: DroneBird, the first video bird counting dataset from a drone, with 3.6M point annotations, trajectory labels, attribute breakdowns, and documented small-target statistics. That is a valuable resource. The method itself extends MultiMAE by treating density as a second modality, adds a density-guided spatial adaptive mask that retains foreground tokens with a background retention probability, and uses optical-flow warping plus cross-attention for temporal fusion. Ablations on FDST are clean, and the appendix reports multi-seed variance (1.30±0.022 MAE). That is honest, reproducible work.\n\nThe soft spot is the comparison table. Section 5.1 fixes a Gaussian kernel with sigma=6 for the paper's own density maps. The baseline numbers on Mall, FDST, and VSCrowd come from the original papers, which used their own kernels and preprocessing. MAE and RMSE are computed against these density maps, so a different kernel changes the metric even for identical predictions. The gains on Mall (4%) and FDST (7%) are small enough that a protocol mismatch could flip them. On DroneBird the baselines appear to have been run by the authors, so those comparisons are more trustworthy—but it's their own benchmark, so it's not an independent test.\n\nThere is also mild circularity: the mask ratio, background retention probability, and loss weights were tuned on the FDST test set, and FDST is then reported as a headline result. The seed variance table helps, but it doesn't address the baseline protocol issue.\n\nIf I were refereeing, I would ask for protocol-matched baselines on the existing datasets, or at least a clear statement that the borrowed numbers are reported rather than directly comparable. The method is solid, the writing is clear, and the dataset alone justifies the paper. This deserves a serious peer review, and the likely outcome after revision is acceptance.","headline":"Useful new video bird benchmark and a coherent masked-autoencoder counting framework, but the SOTA numbers rest on baselines that don't share the same density-map protocol.","tokens_in":22837,"tokens_out":3139,"would_cite":true,"duration_ms":27517,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A masked autoencoder that treats density maps as a second modality sets new accuracy marks on four video counting benchmarks while introducing a drone-captured bird-counting dataset.","keywords":["video object counting","masked autoencoder","density map regression","crowd counting","bird counting dataset","optical flow temporal fusion","self-supervised representation learning","spatial adaptive masking"],"falsifier":"Run each video-counting baseline on Mall, FDST, VSCrowd, and DroneBird under the paper's exact protocol (fixed Gaussian sigma=6, same input sizes, same standardization), then recompute MAE/RMSE; if the best baselines match E-MAC within noise, the claimed state-of-the-art advantage would not survive.","tokens_in":21854,"feed_emoji":"🦅","tokens_out":4976,"duration_ms":42361,"temperature":0.7,"pith_summary":"Video frames of crowds and animals are often sparse, with most pixels belonging to empty background, which makes density-counting models over- or under-count. The paper proposes E-MAC, a masked-autoencoder framework that treats the ground-truth density map as a second input modality, reconstructs masked density patches, and uses the density distribution to decide which image patches to keep, so the model spends its capacity on foreground objects. An optical-flow-based fusion module then warps the previous frame's predicted density map into the current frame and adds residual corrections. The authors report state-of-the-art mean absolute error on all four tested benchmarks (Mall, FDST, VSCrowd, and their new DroneBird), and they introduce DroneBird, a drone-captured video dataset of migratory birds. If the results hold, the framework gives video counting a reusable recipe: let the density map itself supervise representation learning rather than only serving as the regression target.","feed_headline":"Density-aware masked autoencoder tops four video counting benchmarks","feed_subtitle":"It adds a drone-shot bird dataset and cuts counting error by up to 58 percent on existing benchmarks.","key_machinery":"The load-bearing object is the density map used as an auxiliary modality inside a masked autoencoder: instead of only being the regression target, the density map also selects which image tokens survive masking and provides the reconstruction signal. This transforms counting into a self-supervised cross-modal reconstruction problem. The supporting mechanisms are SAM (spatial adaptive masking, which sorts image tokens by the object count in their corresponding density patch and keeps the top foreground tokens with probability 1-P) and TCF (temporal collaborative fusion, which warps the previous frame's density prediction to the current frame using optical flow and adds a cross-attention residual).","core_discovery":"The paper's central claim is that density-embedded masked modeling solves the foreground-background imbalance that hurts video object counting. In DEMO, image patches and density-map patches are tokenized together; a density-guided Spatial Adaptive Masking (SAM) keeps image tokens whose corresponding density patches contain many objects (with a small background-retention probability), while density-map tokens are randomly masked. The decoder reconstructs the full density map from the surviving tokens, so counting is learned as a self-supervised reconstruction task. A Temporal Collaborative Fusion (TCF) module computes optical flow between adjacent frames, warps the previous predicted density map, and uses cross-attention to produce residual corrections that are added to the current prediction. The authors report the lowest MAE and RMSE on Mall, FDST, and VSCrowd, and on the new DroneBird dataset (e.g., MAE 1.35/1.29/6.0/38.72 respectively), with the largest gains on DroneBird where the MAE is claimed 58% below STGN.","pith_inferences":["The same density-embedded masked modeling recipe likely transfers to other sparse-object dense-regression tasks (cell counting, vehicle counting from drones), since the underlying mechanism of density-guided token selection inside masked reconstruction does not assume human targets.","Because the mask ratio and background-retention probability (P=0.2) are tuned on FDST, applying E-MAC to new domains will probably require re-tuning these two hyperparameters; the paper's own experiments show both have non-monotonic effects.","A natural next test is whether the density map used for masking can be replaced by a coarse predicted density at inference, which would relax the need for point annotations during deployment and potentially enable self-supervised adaptation to unseen scenes."],"forward_implications":["Counting accuracy improves by using the density map as a second modality during masked modeling, not just as the training target; the ablation shows DEMO alone cuts FDST MAE by roughly 27% over the baseline.","Spatial adaptive masking concentrates computation on foreground patches, reducing redundant background learning; it contributes the largest single ablation gain (about 32% MAE on FDST).","Temporal collaborative fusion with optical flow combines adjacent-frame density predictions and lowers error on video benchmarks (about 5 to 16 percent on FDST in the ablation).","The DroneBird dataset supports evaluation on small, fast-moving, sparsely distributed targets in natural scenes; transferring a crowd-trained model to DroneBird fails (MAE 183.31 vs. 38.72 when trained on target), showing the domain gap."],"supporting_citations":[{"why":"Supplies the masked autoencoder paradigm for self-supervised representation learning that DEMO extends.","marker":"(He et al., 2021)"},{"why":"Provides the pre-trained ViT-B encoder and the multi-modal masked-modeling setup (including symmetric Dirichlet token counts) on which E-MAC is built.","marker":"(Bachmann et al., 2022)"},{"why":"PWCNet supplies the optical flow used by Temporal Collaborative Fusion to warp the previous frame's predicted density map.","marker":"(Sun et al., 2018)"},{"why":"Defines the FDST video crowd counting benchmark used for the main comparisons and most ablations.","marker":"(Fang et al., 2019)"},{"why":"Provides the VSCrowd video crowd dataset and the GNANet baseline that the paper competes against.","marker":"(Li et al., 2022)"},{"why":"STGN is the strongest video-counting baseline on FDST and DroneBird; the paper compares against its reported MAE/RMSE.","marker":"(Wu et al., 2023)"},{"why":"FRVCC is the previous best method on Mall and is used as the runner-up for improvement claims.","marker":"(Hou et al., 2023)"},{"why":"CSRNet defines the density-regression baseline and protocol conventions (Gaussian density maps) that E-MAC follows.","marker":"(Li et al., 2018)"}],"fun_headline_variants":["Density-aware mask curing video counting imbalance","New DroneBird dataset slashes video counting error","Density-embedded masking improves video object counting","DEMO: density-guided masking for accurate video counting","DroneBird benchmark and DEMO cut counting error by 58%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparisons with prior methods assume those methods were evaluated under the same density-map generation and evaluation protocol (a fixed Gaussian kernel with sigma=6 in this paper); if the published baseline numbers used different kernels or preprocessing, the reported performance gaps may partly reflect protocol differences rather than counting accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Density-aware mask curing video counting imbalance","New DroneBird dataset slashes video counting error","Density-embedded masking improves video object counting","DEMO: density-guided masking for accurate video counting","DroneBird benchmark and DEMO cut counting error by 58%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00081,"raw_usage":{"total_tokens":3606,"prompt_tokens":1049,"completion_tokens":2557,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":665,"completion_tokens_details":{"reasoning_tokens":2480}},"tokens_in":665,"tokens_out":2557,"duration_ms":17655,"temperature":1.0,"reasoning_tokens":2480,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:52:36.519165+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run each video-counting baseline on Mall, FDST, VSCrowd, and DroneBird under the paper's exact protocol (fixed Gaussian sigma=6, same input sizes, same standardization), then recompute MAE/RMSE; if the best baselines match E-MAC within noise, the claimed state-of-the-art advantage would not survive.","supporting_citations":[{"cited_title":"MultiMAE : Multi-modal multi-task masked autoencoders","cited_arxiv_id":null,"evidence_quote":"Provides the pre-trained ViT-B encoder and the multi-modal masked-modeling setup (including symmetric Dirichlet token counts) on which E-MAC is built."},{"cited_title":"PWC-Net : CNNs for optical flow using pyramid, warping, and cost volume","cited_arxiv_id":null,"evidence_quote":"PWCNet supplies the optical flow used by Temporal Collaborative Fusion to warp the previous frame's predicted density map."}],"review_version":1}