{"id":"257cec10-0271-4324-9ef8-8c9cd44148a5","arxiv_id":"2501.01040","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An event camera action recognition framework pretrains a masked autoencoder on raw (x,y,t) event points and uses plane-fitting inlier selection to form cleaner event patches.","lead":"This paper adapts masked autoencoder pretraining, originally made for 3D point clouds, to raw event camera data for action recognition. It reports state-of-the-art accuracy on several gesture benchmarks and adds a denoising patch-generation step based on plane fitting.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ShapeNet pretraining confounds the SOTA comparison; reported gains may come from external point-cloud transfer, not the proposed event MAE.","rationale":"I identified the use of external ShapeNet pretraining as the single most load-bearing concern because it directly affects the validity of the headline claim. The paper's central assertion is that the proposed method outperforms SOTA, and the experimental evidence is the accuracy numbers in Tables I–III. However, the proposed model alone benefits from a prior trained on ShapeNet, a large external point-cloud dataset, whereas the baselines do not. This confound is acknowledged by the authors in Sec. III-A, where they state that the ShapeNet-pretrained initialization yields lower loss. Therefore, the observed accuracy gains could be entirely due to transfer learning, unrelated to the event-specific contributions (inlier-based patch selection and masked autoencoding). This is not a minor reproducibility issue; it undermines the causal attribution of the performance improvement. The reader's identified planarity assumption is a potential weakness, but it is secondary because even if the planarity assumption holds perfectly, the SOTA claim would still be confounded. The proposed concrete test—training from scratch—would settle whether the method genuinely advances event-based action recognition or merely inherits strength from point-cloud pretraining. Based on this, I maintain the conditional verdict: the paper should be accepted only if the authors provide a controlled comparison that isolates the effect of their contributions.","tokens_in":7493,"tokens_out":7130,"duration_ms":66336,"concrete_test":"Retrain the proposed Event MAE from scratch (random initialization, no ShapeNet pretraining) on the same training splits and report accuracy on DVS128-Gesture 11-class and SL-Animals-DVS S3. If the from-scratch accuracy drops below EvT's 96.2% and 87.45%, respectively, the SOTA claim is not supported by the current protocol. Additionally, fine-tune EvT from a ShapeNet-pretrained checkpoint (or an equivalently pre-trained point-cloud model) to see if the advantage persists under a controlled comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that Event MAE 'outperforms the state-of-the-art' rests on an unfair comparison: the model is initialized from a ShapeNet-pretrained checkpoint (Sec. III-A), while the baselines in Tables I–III (EvT, PointNet++, TORE, etc.) are evaluated without such external pretraining. The authors explicitly state that the ShapeNet-pretrained model is better than training only on event data, so part of the performance gap must be attributed to transfer learning from a large point-cloud dataset. Since the proposed contributions are the event inlier patch generation and masked autoencoding on event streams, the incremental gain over baselines cannot be isolated unless all methods are trained under the same initialization protocol. Without a from-scratch comparison, the claimed SOTA status is not established; the improvement could be a generic effect of large-scale pretraining rather than the event-specific mechanism. This concern is more load-bearing than the planarity assumption because even a perfect inlier model would show inflated numbers under this protocol.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Event Masked Autoencoder (Event-MAE), which applies Point-MAE-style masked autoencoding to DVS event streams treated as 3D point clouds (x, y, t). The main novelties are an event patch generation algorithm that uses plane-fitting inlier selection (Eqs. 2-4) with KNN grouping and an inlier threshold H, and a PointNet-based patch embedding for transformer models. The model is pre-trained on ShapeNet and then on event datasets, and fine-tuned for classification on DVS128-Gesture, SL-Animals-DVS, and DVS Action. The paper reports state-of-the-art accuracy on several benchmarks (Table I: 98.54/97.75; Table II S3: 88.23; Table III: 93.9) and provides ablations of the threshold and sampling method.","tokens_in":7691,"tokens_out":4260,"duration_ms":35671,"significance":"If the reported results are reproducible and the comparison is fair, the paper demonstrates that masked autoencoding on raw event point clouds is viable and can outperform frame-based and point-based baselines, opening a promising direction for unified point/event pre-training. The paper is well aligned with current MAE literature, and the ablation of patch generation methods provides useful evidence that FPS is poorly suited to event data. However, the strength of the empirical claim is substantially weakened by an internal contradiction on DVS Action, a confounded pre-training comparison, test-set hyperparameter selection, and a lack of variance estimates, so the central 'state-of-the-art' claim is not yet established.","major_comments":[{"comment":"The text states: 'DVS Action is a dataset smaller than DVS128 Gesture Dataset. So we do not fine-tune our model on this dataset,' yet Table III reports 'Our 93.9' for DVS Action. This is internally inconsistent and makes the DVS Action result uninterpretable. The authors must state exactly how the 93.9% figure was obtained (fine-tuning protocol, which pretrained checkpoint was used, evaluation setup) or remove the result from the comparison.","section":"Section III-B / Table III"},{"comment":"The model is initialized from a ShapeNet-pretrained Point-MAE checkpoint, and the authors report that this reduces the loss by 2.3e-4 compared with event-only training. The baselines in Tables I-III (EvT, PointNet++, TORE, etc.) are trained on event data without such external pretraining. Because the claimed contribution is the event-specific patch generation and masked modeling, the reported gains (0.14-0.78 percentage points over EvT) are confounded by transfer from point-cloud data. To support the SOTA claim, provide an ablation with the same backbone trained from scratch or with event-only pretraining, and/or pretrain the baselines under the same initialization protocol.","section":"Section III-A / Tables I-III"},{"comment":"The inlier threshold H is selected based on DVS128-Gesture classification accuracy (the best value, H=0.85, is identified in Table IV), and the same dataset's accuracy is then reported as the headline result in Table I. This is selection on the test set; without a separated validation split or repeated cross-validation, the reported accuracy is optimistically biased. Please report validation-based selection or show that the result is stable across H values and across multiple runs.","section":"Section III-C / Table IV"},{"comment":"The reported margins over the strongest baseline EvT are 0.14, 0.22, and 0.78 percentage points on DVS128-Gesture 10/11 classes and SL-Animals S3, and on SL-Animals S4 the proposed method (87.46) is below EvT (88.12). No error bars, standard deviations, or numbers of seeds are reported anywhere in the tables. With these margins, the claim that the method 'outperforms the state-of-the-art' is not statistically supported; add repeated-run statistics or at least report the number of runs for each result.","section":"Tables I-III (overall)"}],"minor_comments":[{"comment":"The normalization equation is written as 't = tn−t0 / tmax−t0', which is ambiguous; it should be t = (tn−t0)/(tmax−t0).","section":"Section II-A"},{"comment":"The definition of erj is unclear: the expression uses a bare 'P' symbol without a defined summation, and the division by n is not visually attached to the sum. Please rewrite as a standard average, e.g., erj = (1/n) * sum_i (Delta t_i - Delta t̂_i).","section":"Equation (4)"},{"comment":"The phrase 'K-Nearest Neighborhood' should be 'K-Nearest Neighbor', and the notation P = KNN(E, C_I) is confusing because P is later used for patches; please define all variables consistently.","section":"Section II-B"},{"comment":"The value of the inlier threshold H used for the main results is not stated in Section III-A; it appears only in the ablation. Please report the default H used for Tables I-III.","section":"Section III-A / Table IV"},{"comment":"The caption says 'we contact two event data streams'; this is likely a typo for 'concatenate'. Please correct the wording.","section":"Figure 5 caption"},{"comment":"The position embedding is said to follow [20], which is the BERT paper; the relevant reference for point-cloud patch position embedding is more likely Point-MAE [11] or Point-BERT [14]. Please cite the appropriate source.","section":"Reference [20]"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central idea is plausible and the ablation framework is a good start, but the empirical claims are currently not reliable: the DVS Action contradiction, the ShapeNet-pretraining confound, and the test-set selection of H are load-bearing issues. I recommend major revision rather than rejection because these issues can, in principle, be fixed with additional experiments and clarification. I also note that the paper does not mention code availability or detailed training hyperparameters, which would be important for reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's the quick take on arXiv:2501.01040. The paper is the first to apply masked autoencoding to raw event camera point streams. That is a real contribution, and the plane-fitting inlier selection for patch centers is a sensible adaptation of Point-MAE to event noise. The ablation showing their patch generation cuts reconstruction loss sharply versus FPS and random sampling is credible and useful.\n\nThe trouble is the SOTA claim. The model is initialized from a ShapeNet-pretrained checkpoint, while the baselines are not. The authors admit ShapeNet pretraining helps. So the margin over EvT on DVS128 (0.14 and 0.22 points) and the S3 win on SL-Animals cannot be attributed to the event-specific mechanism. For all we know, generic point-cloud transfer is doing the work. This is load-bearing, not a quibble. Also, the S4 result is below EvT, the inlier threshold H is tuned on DVS128-Gesture and then reported on the same benchmark, there are no error bars or run counts, and Section III-B says DVS Action is not fine-tuned while Table III lists a number for it. That last one is an internal contradiction.\n\nThe core idea still deserves scrutiny. The plane fit is simple and the implementation is straightforward, but the empirical case needs a from-scratch comparison, an error-bar study, and a clean explanation of the DVS Action line. I'd send this to review, because the novelty is real and the community should get a corrected version into the record. But I would not take the SOTA claim at face value until the pretraining confound is resolved.","headline":"First to apply masked autoencoding to raw event streams, but the SOTA claim is confounded by ShapeNet pretraining and the empirical support is thinner than the tables suggest.","tokens_in":8226,"tokens_out":1784,"would_cite":true,"duration_ms":16764,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Masked autoencoding of raw event points achieves state-of-the-art action recognition on DVS benchmarks while preserving temporal structure.","keywords":["event camera","action recognition","masked autoencoder","self-supervised learning","point cloud","event inlier model","dynamic vision sensor","transformer"],"falsifier":"Train the same model on a synthetic event dataset with controlled non-linear acceleration while varying the threshold H, and compare against a random-sampling baseline; if random sampling matches or beats the inlier method as acceleration grows, the planarity assumption is the binding limit. Alternatively, measure classification accuracy on the SL-Animals-DVS S4 noisy split against a version with no plane filtering; if removing the filter does not lower accuracy, the denoising step is not carrying the claimed benefit.","tokens_in":7269,"feed_emoji":"⚡","tokens_out":5042,"duration_ms":40243,"temperature":0.7,"pith_summary":"The paper tries to establish that self-supervised masked reconstruction, previously successful for images and point clouds, works on raw event streams from dynamic vision sensors. It treats events as point clouds and pretrains a transformer by masking event patches and reconstructing them, using a plane-fitting inlier model to select clean patch centers. If correct, event-camera action recognition can skip lossy frame-based conversions and still beat prior methods. The authors report top accuracy on DVS128-Gesture (98.54 percent, 10 classes), DVS Action (93.9 percent), and SL-Animals-DVS S3 (88.23 percent), while lagging slightly on noise-heavy S4 (87.46 versus EvT's 88.12).","feed_headline":"Masked autoencoder sets best DVS action recognition yet","feed_subtitle":"Pretraining on raw event points beats prior methods on DVS action benchmarks.","key_machinery":"Event patch generation via inlier filtering: each candidate patch center is accepted only if its K-nearest neighbors fit a plane t = ax + by + c with average residual below a threshold H. The chosen centers define event patches; PointNet embeds each patch into a token, a standard transformer encoder sees only visible tokens, and a decoder reconstructs masked patches under Chamfer Distance loss. Pretraining starts from ShapeNet point-cloud weights, then continues on event data, followed by fine-tuning for classification.","core_discovery":"The central claim is that masked autoencoding applied directly to raw event points—with an event inlier model that rejects noise when selecting patch centers—learns representations that outperform existing frame-based and point-based action recognition models on several public benchmarks. The paper shows that Farthest Point Sampling, standard for point clouds, degrades on event data because the time axis does not encode shape and because noise biases FPS toward outliers. Replacing FPS with least-squares plane fitting over local neighborhoods selects centers whose groups lie on near-planar structures in (x,y,t), lowering pretraining Chamfer Distance by roughly fourfold and improving downstream classification. The method's weaker result on the noisier SL-Animals-DVS S4 split is attributed to limits of the plane-fitting denoiser.","pith_inferences":["If the local-planarity assumption holds, the same patch selection could be adapted to event-based optical flow or low-latency tracking, where denoising raw events matters more than classification accuracy.","The S4 shortfall suggests a testable boundary: on datasets with fast non-linear motion or strong sensor noise, an adaptive threshold that scales with local event density might outperform the fixed hyperparameter H.","The 0.5-second sliding window choice implies a latency ceiling; shrinking the window would trade accuracy for responsiveness in real-time robotics applications.","A direct comparison against a random-sampling MAE without plane fitting would isolate how much of the gain comes from denoising versus from masked pretraining itself."],"forward_implications":["Masked modeling is shown to be effective for raw event stream data, not just images and point clouds.","Pre-training on ShapeNet point clouds transfers to event data, reducing reconstruction loss by 2.3 × 10^-4 compared to training from scratch.","The inlier-based patch center sampler cuts pretraining Chamfer Distance roughly fourfold versus FPS and also improves on random sampling.","A single masked-autoencoder backbone may later unify event data, point clouds, and images for multi-modal learning."],"supporting_citations":[{"why":"Supplies the Point-MAE base structure that the event MAE adapts, including masking and reconstruction with Chamfer Distance.","marker":"[11]"},{"why":"Inspires the event inlier model used to select denoised patch centers.","marker":"[13]"},{"why":"Provides the point resampling augmentation applied to event training data.","marker":"[14]"},{"why":"Defines the DVS128-Gesture benchmark and the event stream used for pretraining and evaluation.","marker":"[9]"},{"why":"Event Transformer, the primary frame-based baseline the method compares against.","marker":"[10]"},{"why":"ShapeNet point cloud dataset used for initial pretraining, improving downstream event reconstruction.","marker":"[25]"},{"why":"Space-time event cloud PointNet++ baseline for point-based gesture recognition.","marker":"[6]"}],"fun_headline_variants":["Event MAE ditches FPS for plane fitting, tops action benchmarks","First masked autoencoder on raw event points wins action recognition","Plane-fitting patch centers boost event point MAE accuracy","Raw event point pretraining sets new DVS action recognition record","Event point MAE: plane fitting beats farthest point sampling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The plane-fitting center selection assumes that, within a short time window, object motion is locally linear so nearby events lie on a plane in (x,y,t); when motion is highly non-linear or noise is severe, this assumption filters out informative events.","fun_headline_variants_meta":{"raw":{"variants":["Event MAE ditches FPS for plane fitting, tops action benchmarks","First masked autoencoder on raw event points wins action recognition","Plane-fitting patch centers boost event point MAE accuracy","Raw event point pretraining sets new DVS action recognition record","Event point MAE: plane fitting beats farthest point sampling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000717,"raw_usage":{"total_tokens":3198,"prompt_tokens":900,"completion_tokens":2298,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":2223}},"tokens_in":516,"tokens_out":2298,"duration_ms":13648,"temperature":1.0,"reasoning_tokens":2223,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:36:06.469402+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same model on a synthetic event dataset with controlled non-linear acceleration while varying the threshold H, and compare against a random-sampling baseline; if random sampling matches or beats the inlier method as acceleration grows, the planarity assumption is the binding limit. Alternatively, measure classification accuracy on the SL-Animals-DVS S4 noisy split against a version with no plane filtering; if removing the filter does not lower accuracy, the denoising step is not carrying the claimed benefit.","supporting_citations":[{"cited_title":"Masked autoencoders for point cloud self-supervised learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the Point-MAE base structure that the event MAE adapts, including masking and reconstruction with Chamfer Distance."},{"cited_title":"Self-supervised learning of inlier events for event-based optical flow,","cited_arxiv_id":null,"evidence_quote":"Inspires the event inlier model used to select denoised patch centers."},{"cited_title":"A low power, fully event-based gesture recognition system,","cited_arxiv_id":null,"evidence_quote":"Defines the DVS128-Gesture benchmark and the event stream used for pretraining and evaluation."},{"cited_title":"Event transformer. a sparse-aware solution for efficient event data processing,","cited_arxiv_id":null,"evidence_quote":"Event Transformer, the primary frame-based baseline the method compares against."},{"cited_title":"Space-time event clouds for gesture recognition: From rgb cameras to event cameras,","cited_arxiv_id":null,"evidence_quote":"Space-time event cloud PointNet++ baseline for point-based gesture recognition."}],"review_version":1}