{"id":"4e1efe7a-6d72-45e6-970e-4e55cc5ff34e","arxiv_id":"2501.13805","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"mmEgoHand fuses head-mounted mmWave radar with IMU data to predict 3D hand keypoints and recognize eight interaction gestures, reaching 90.8% accuracy on its own 10-subject dataset.","lead":"A head-mounted millimeter-wave radar paired with an IMU estimates 3D hand joint positions and recognizes eight VR interaction gestures without using a camera during inference. A smart generalist might read it because it points toward private, occlusion-robust hand tracking for wearable devices that works while users stand, sit, or lie down.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 72.73 mm MPJPE depends on an undefined conversion from MediaPipe's normalized 3D hand landmarks to metric millimeters; without a disclosed scale or calibration procedure, the central quantitative pose result is not interpretable.","rationale":"The reader's weakest assumption identified the same load-bearing concern: MediaPipe-derived 3D keypoints are treated as metric ground truth without a described calibration procedure. My reading of the manuscript confirms this is the most serious threat to the central quantitative claim. Section 4.1(3) describes label generation using MediaPipe, Section 4.2(1) computes MPJPE in millimeters, and Table 4 reports 72.73 mm, but no step defines how MediaPipe's normalized coordinate frame is mapped to physical millimeters. This is not merely an implementation detail; it directly affects whether the reported error, the claimed improvement from IMU fusion, and the comparison to mm4Arm are scientifically meaningful. The paper also includes an explicit admission that MediaPipe keypoints \"may not offer high-precision tracking,\" reinforcing the concern. The manuscript has real independent support: a public dataset and code release, a plausible multimodal architecture, and a reasonable experimental design for a proof-of-concept. However, the released artifacts cannot by themselves validate the metric conversion without inspection of the preprocessing source. The reader's CONDITIONAL verdict already captures this uncertainty, so my stress-test does not change the verdict; it sharpens the condition by identifying a concrete, testable calibration check. Secondary issues such as selective filtering of fast-motion samples and missing error bars are real but less fundamental, and they do not alter the conclusion. I am not raising this as an accusation of misconduct; the concern is about the completeness and interpretability of the experimental reporting, and it can be settled by examining the code and running the proposed calibration check.","tokens_in":15736,"tokens_out":2847,"duration_ms":27723,"concrete_test":"Inspect the released code and dataset at https://github.com/WhisperYi/mmVR to locate the exact code path that converts MediaPipe hand landmarks into the ground-truth coordinates used in Eq. (6). Then run the following check: using the same camera and a ruler or known-size object in view, record a hand with a measured palm width, run MediaPipe on the frames, and compute the ratio between MediaPipe landmark spacing and the physical width in millimeters. Apply this measured scale to the test-set ground-truth coordinates and recompute the 72.73 mm MPJPE. If the conversion in the released code is missing, uses an arbitrary normalization, or yields a corrected MPJPE that differs by more than 10% from the reported value, the central pose-estimation claim in Table 4 is not established in metric units.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest quantitative claim, a 72.73 mm MPJPE after fusing mmWave and IMU, rests on ground truth produced by Google MediaPipe Hand Landmark SDK (Section 4.1(3)) and evaluated by the L2 norm in Section 4.2, Eq. (6), with units reported as millimeters. MediaPipe's 3D hand landmarks are normalized to hand scale and are not, without additional calibration, expressed in metric coordinates. The manuscript never describes how normalized MediaPipe keypoints are converted to millimeters: no camera calibration, no known hand-size scale, no depth sensor, and no physical reference object are mentioned. In fact, Section 4.1(3) acknowledges that \"MediaPipe's 3D keypoints may not offer high-precision tracking,\" yet no correction or uncertainty is propagated into the MPJPE computation. If the conversion is absent or uses an arbitrary scale, then the reported 72.73 mm value, the mmWave-only baseline of 96.42 mm, and the claimed 24.6% improvement are not meaningful in metric units, and the pose-training signal itself is ill-defined. This is a load-bearing issue because the downstream gesture-recognition argument is built on the premise that the estimated hand poses are accurate enough to serve as intermediate features. A secondary but related concern is that samples with more than 40% of frames missing MediaPipe keypoints are discarded (Section 4.1(3)), preferentially removing fast-motion samples; this could inflate both pose and gesture metrics, but the undefined metric conversion is the more fundamental threat to the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents mmEgoHand, a head-mounted sensing system that combines a 60-64 GHz TI IWR6843 mmWave radar with a co-located IMU to estimate 3D hand keypoints for egocentric interaction. The architecture is a Transformer-based encoder-decoder with a pose decoder and a 30-block context decoder, adapted from the Person-in-WiFi 3D multi-person pose framework to output a variable number of hands (0-2) per frame. The reported main results are an MPJPE of 72.73 mm with mmWave+IMU versus 96.42 mm with radar only and 109.92 mm without the context decoder, and a two-stage gesture-recognition pipeline (estimated poses fed into ResNet50, LSTM, GCN, or ViT) that achieves 90.80% accuracy on an eight-gesture VR-interaction dataset collected from 10 subjects in three postures and three scenes. The paper additionally reports few-shot cross-person, cross-posture, and cross-hand evaluations, and states that the 26 GB dataset and code will be released.","tokens_in":16077,"tokens_out":16291,"duration_ms":135369,"significance":"If the pose and gesture results hold, this is a useful contribution to egocentric radio-frequency sensing: it is the first head-mounted mmWave system demonstrated for fine-grained one- and two-hand pose estimation of which I am aware, and the IMU-based compensation of head-motion artifacts is a sensible design choice that is cleanly ablated (96.42 mm to 72.73 mm). The paper has concrete strengths: the repetition-based held-out split is methodologically sound; the context-decoder ablation (109.92 mm to 72.73 mm) and the comparison of two-stage pose-based gesture recognition against one-stage baselines (90.80% versus 60-85%) are informative; the reported inference budget of 42.03 ms for 30 frames supports the real-time claim; and the few-shot section is commendably honest, reporting zero-shot cross-person accuracy near 62-63% and documenting the benefit of one/two-shot calibration. The promised dataset and code releases, if verified, would be a valuable community resource.","major_comments":[{"comment":"The headline pose result is not interpretable as a metric error as written. The ground-truth keypoints come from the Google MediaPipe Hand Landmark SDK (Section 4.1(3)), whose 3D coordinates are normalized with respect to the image or to hand pose scale and are not expressed in millimeters without an additional scale assumption; the paper never describes such a conversion, mentioning no camera calibration, hand-size measurement, reference object, or use of MediaPipe's metric world landmarks. As a result, the L2 norm in Eq. (6), reported as millimeters in Table 4 (72.73 mm fused, 96.42 mm radar-only), is not well-defined, and the claimed 24.6% improvement inherits the same ambiguity; the coordinate frame of the network output relative to the ground truth is also unspecified. Please disclose the exact conversion and frame alignment, or re-report all pose errors in the native normalized units (for example, normalized per-hand units or PCK-style metrics) and restate the relative improvements in those units.","section":"4.1(3), 4.2(1), Eqs. (6)-(7)"},{"comment":"The sample-filtering rule biases the evaluation toward slow motions. Section 4.1(3) states that MediaPipe frequently fails under rapid hand motion and that samples with more than 40% of frames missing keypoints are discarded, which removes 554 of the 5,760 recorded samples (about 9.6%); because the failure mode is correlated with fast motion, both pose and gesture metrics are computed on an easier, filtered distribution. Please quantify the dependence of the discarded samples on gesture class and posture (in particular whether the two-handed gestures are affected more), and report how the Table 4 numbers change if the filter is relaxed or if missing frames are imputed.","section":"4.1(3), Table 3"},{"comment":"The MediaPipe-derived labels are never validated against any independent reference, so the absolute accuracy of the pose numbers is unverifiable. The paper itself acknowledges that \"MediaPipe's 3D keypoints may not offer high-precision tracking\" (Section 4.1(3)), yet the same labels serve as both the training target and the evaluation ground truth in Eqs. (6)-(7); label noise is therefore baked into the reported 72.73 mm value, and the paper offers no estimate of its magnitude. I would ask for a small validation subset (for example, a few hundred frames with manual or depth-based annotation, or a known-size hand reference placed in the scene) to establish the label floor, or at minimum a quantitative sensitivity discussion of how label noise propagates into MPJPE.","section":"4.1(3), 4.3(1)"},{"comment":"The mm4Arm comparison is under-specified and appears not to be evaluated on the same test set. Because mm4Arm is closed-source, the authors reimplemented it, but give no protocol details (training epochs, optimizer, input representation, hyperparameters), and the footnote says it is \"trained and evaluated solely on single-hand data,\" whereas the mmEgoHand figures in the same table are computed on the full mixed single- and two-hand test set; the comparison thus conflates algorithmic quality with differences in training data and test data. Please report the reimplementation protocol, evaluate mm4Arm on the same test set used for the mmEgoHand rows (or state clearly why that is infeasible and weaken the claim accordingly), and, if feasible, add a two-stage baseline that consumes MediaPipe ground-truth poses to separate pose-estimation error from gesture-classification error.","section":"4.3(1), Table 4"}],"minor_comments":[{"comment":"The context-decoder ablation is reported as 109.02 mm in Table 4 but as 109.92 mm in the text; please reconcile the discrepancy.","section":"Table 4 vs Section 4.3(1)"},{"comment":"Please fix typos: \"unseern persons\" and \"unseern hands\" in the captions of Tables 6 and 8, \"we use apply Doppler FFT\" in Section 3.1, and \"lying in in scene\" in the Table 3 caption.","section":"Captions and typos"},{"comment":"Section 4.1(1) asserts \"precise temporal synchronization\" among the radar, IMU, and camera, but the clock-alignment mechanism is never described; given that the fusion in Section 3.2 depends on cross-modal temporal alignment, please add a sentence on how the 20 fps, 20 Hz, and 20 fps streams are synchronized.","section":"4.1(1)"},{"comment":"The loss formulation should spell out the Hungarian matching cost (for example, L1 or L2 on keypoints, and the penalty for unmatched candidates) and how zero-hand and one-hand sequences are treated during training.","section":"3.2, Eq. (5)"},{"comment":"The input description is internally inconsistent: a 2-second window at 20 frames per second contains 40 radar frames, but the input is specified as x_mm in R^{30×256×128} with \"30 uniform patches\"; please clarify whether radar heatmaps are binned or averaged into 30 patches, or whether a 1.5-second window is used.","section":"3.2"},{"comment":"The abstract's \"90.8% recognition accuracy\" is obtained under the repetition-based split of Table 3, in which all subjects appear in both training and test; since the cross-person zero-shot results in Table 6 are 61.96-63.11%, the abstract should state explicitly that the headline accuracy is subject-dependent.","section":"Abstract, Table 6"},{"comment":"Tables 4 and 5 report point estimates without variance; given the 12 subject/hand settings in Table 2, reporting the mean and standard deviation across subjects would strengthen the comparison claims.","section":"Tables 4-5"},{"comment":"The dataset URL (github.com/WhisperYi/mmVR) should be verified at acceptance, and the paper should report the per-gesture and per-posture sample counts after filtering, since class imbalance could affect the per-class numbers in Table 5.","section":"Abstract, 4.1(3)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits a systems-oriented sensing venue, and the open dataset is likely the most durable asset. The authors disclose their adaptation of Ref. [29], which is from the same group; the novelty therefore rests primarily on the new sensor modality, the IMU-fusion design, and the context decoder, and the paper should be assessed on those terms. I recommend verifying the GitHub repository contents at acceptance, since the dataset release is a listed contribution. The undefined millimeter conversion is the main correctness risk and is, in my view, fixable; if the authors cannot reconstruct a metric ground truth, the relative ablations and the gesture-recognition results still support the core design claims, which is why I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a real step forward for egocentric mmWave sensing. First system I've seen that mounts a commodity radar on the head and estimates 1–2 hand poses with an IMU-based ego-motion compensation, and they release a 26 GB dataset with 10 subjects, 8 VR gestures, 3 postures. The dual-decoder adaptation of Person-in-WiFi 3D to variable hand counts is a sensible engineering contribution, and the ablations show the IMU and Context Decoder each matter. The few-shot cross-person/posture/hand results are a nice addition because they honestly show the zero-shot limits and the value of calibration. Credit also for acknowledging that MediaPipe keypoints are not high-precision tracking.\n\nNow the soft spots, in proportion. The big one is the metric conversion. MediaPipe 3D hand landmarks are normalized to hand scale; the paper never says how they become millimeters. No camera calibration, no known hand size, no depth sensor. Equation (6) just takes an L2 norm and calls it mm. That makes 72.73 mm, the 96.42 mm baseline, and the 24.6% improvement uninterpretable as metric quantities. The ranking of ablations survives if the scale is constant, but the absolute numbers don't mean what the paper implies. This is the kind of thing an editor or referee should catch. The filtering of samples where >40% of frames lost MediaPipe keypoints is a secondary concern; it likely removes fast motion and could inflate both pose and gesture numbers, but it's disclosed and not fatal. The gesture recognition comparison to one-stage baselines is a bit apples-to-oranges—they use a two-stage pipeline with pose as intermediate features, which is a different task setup—but the comparison is transparent and the gains are large enough to be plausible.\n\nThe central claim—that head-mounted radar plus IMU can produce usable egocentric hand poses and that pose-as-feature helps gesture recognition—holds up as a proof of concept. The paper is worth a serious referee and will be useful to anyone in mmWave sensing, wearable interaction, or HCI. My advice: send it to review, but the authors must either disclose a calibration procedure for converting MediaPipe keypoints to metric coordinates or reframe the pose error in scale-normalized units. Without that, the headline number is paperwork.","headline":"A genuinely new egocentric hand-pose/gesture dataset and system, but the headline 72.73 mm MPJPE rests on an undefined MediaPipe-to-millimeter conversion that needs fixing before the numbers are interpretable.","tokens_in":16607,"tokens_out":606,"would_cite":true,"duration_ms":7283,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper aims to show that a radar worn on the head, combined with an inertial measurement unit, can reconstruct the 3D pose of one or two hands, and that these poses make a strong feature for recognizing VR gestures.","keywords":["egocentric sensing","hand pose estimation","mmWave radar","IMU fusion","gesture recognition","Transformer","virtual reality interaction","multi-modal sensing"],"falsifier":"Run the same data collection protocol but label ground truth with a motion-capture system or multi-view stereo reconstruction, then recompute the mean per-joint position error; if the error against that ground truth is substantially larger than 72.73 mm, the pose estimates are not as accurate as reported. A simpler check: place the hand at known distances and orientations relative to the radar and verify that the predicted keypoint scale matches the measured millimeter scale.","tokens_in":15574,"feed_emoji":"📡","tokens_out":5759,"duration_ms":44258,"temperature":0.7,"pith_summary":"This paper tries to establish that a person's hand pose can be reconstructed in 3D from a radar worn on the head, as long as head motion is measured by an inertial sensor and fused into the model. If true, it would let VR and teleoperation systems track hands without cameras, preserving privacy and allowing users to move freely rather than stand in front of a fixed sensor. The paper reports a mean per-joint position error of 72.73 mm with the fusion, versus 96.42 mm with radar alone, and shows that feeding the estimated poses into a simple classifier recognizes eight interaction gestures at 90.80% accuracy. The authors argue that the estimated hand pose, not raw radar heatmaps, is the more informative representation for downstream tasks.","feed_headline":"Head-worn radar plus IMU tracks hand poses at 72.73mm error","feed_subtitle":"Fusing head-motion data cuts pose error by 24.6% and pushes VR gesture recognition to 90.8% accuracy.","key_machinery":"The load-bearing components are two. First, the dual-decoder Transformer: the Pose Decoder attends to fused radar-plus-IMU embeddings to localize joints spatially, while the Context Decoder, stacked to 30 blocks, carries refined pose features across frames to enforce temporal coherence. Second, the IMU fusion itself: acceleration and angular velocity from a sensor on the radar bracket are temporally patched and concatenated with radar embeddings, so that signal changes caused by head movement are actively compensated rather than treated as noise. A set-based Hungarian matching loss lets the model output a variable number of hands, which is what lets one framework handle both one-handed and two-handed gestures.","core_discovery":"The central claim is that egocentric 3D hand pose estimation is achievable with a head-mounted mmWave radar, and that the addition of IMU data is what makes it work under natural head movement. Using a 2-second window of radar range-Doppler and range-angle heatmaps together with synchronized 6-axis IMU samples, a Transformer with two decoders regresses 3D keypoints for zero, one, or two hands, with a set-based Hungarian matching loss. The reported mean per-joint position error is 72.73 mm, a 24.6% improvement over the radar-only baseline, and the pose sequence fed into a ResNet classifier yields 90.80% accuracy on an eight-gesture VR interaction task. The paper also reports that removing the context decoder degrades pose error to about 109 mm, and that a single-hand baseline trained on the same data reaches only 165.19 mm, supporting the claim that the fused architecture is doing real work.","pith_inferences":["Because ground truth comes from RGB-derived keypoints, the 72.73 mm error is only meaningful relative to that source; a true metric benchmark such as motion capture or multi-view stereo could shift the absolute number substantially.","The reported cross-hand gap, with zero-shot accuracy near 25 to 28 percent jumping to 73 to 77 percent after one or two calibration samples, suggests the model learns hand-specific patterns; a hand-symmetric augmentation or canonicalization could close much of this gap without per-user calibration.","The pose-as-intermediate-feature recipe could transfer to other downstream tasks the paper lists but does not evaluate, such as skill digitization or robotic teleoperation, where a structured hand state is the natural control variable.","Radar privacy cuts both ways: the conclusion notes that the same hardware can recover respiration and heart rate, so limiting range and field of view is not just an engineering detail but a privacy control."],"forward_implications":["A head-mounted radar plus IMU can serve as a hands-free, privacy-preserving input device for VR and mixed reality, without requiring a fixed sensing zone.","Using estimated 3D hand poses as intermediate features outperforms classifying raw mmWave heatmaps directly by 20 to 30 percentage points in the reported setting.","IMU fusion is the main source of the pose accuracy gain: removing it raises the mean per-joint position error from 72.73 mm to 96.42 mm.","The system runs in real time at 42 ms per 30-frame window on an RTX 3090, so the architecture is deployable in interactive settings."],"supporting_citations":[{"why":"Provides the head-mounted radar plus IMU hardware configuration and the egocentric sensing precedent that mmEgoHand builds on.","marker":"[23]"},{"why":"Supplies the end-to-end set-based decoder architecture and Hungarian matching scheme adapted here for variable hand counts.","marker":"[29]"},{"why":"Provides the radar heatmap preprocessing pipeline (range, Doppler, and angle FFT) that turns sparse point clouds into the network input.","marker":"[7]"},{"why":"Defines the Hungarian assignment algorithm used to match predicted hand candidates to ground-truth keypoints.","marker":"[30]"},{"why":"Demonstrates egocentric head-mounted radar tracking in a multi-view add-on, serving as a direct comparative precursor for wearable mmWave sensing.","marker":"[24]"},{"why":"Is the single-hand mmWave pose baseline that mmEgoHand reimplements and outperforms on mean per-joint position error.","marker":"[27]"},{"why":"Is a state-of-the-art environment-independent mmWave gesture recognition baseline used for comparison in the downstream task.","marker":"[9]"},{"why":"Is a semi-supervised mmWave gesture recognition baseline used for comparison in the downstream task.","marker":"[31]"}],"fun_headline_variants":["Head-mounted radar plus IMU tracks hand poses to 72.73mm","Egocentric hand pose from mmWave radar and IMU hits 72.73mm error","IMU fusion lifts head-mounted radar hand pose accuracy by 24.6%","mmWave radar on head achieves 90.8% VR gesture accuracy with IMU","Head-worn radar+IMU system cuts hand pose error to 72.73mm"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results assume that hand keypoints automatically extracted from RGB video are accurate enough to serve as metric ground truth and that they can be converted to millimeter coordinates, even though the paper describes no calibration procedure for that conversion.","fun_headline_variants_meta":{"raw":{"variants":["Head-mounted radar plus IMU tracks hand poses to 72.73mm","Egocentric hand pose from mmWave radar and IMU hits 72.73mm error","IMU fusion lifts head-mounted radar hand pose accuracy by 24.6%","mmWave radar on head achieves 90.8% VR gesture accuracy with IMU","Head-worn radar+IMU system cuts hand pose error to 72.73mm"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000668,"raw_usage":{"total_tokens":3081,"prompt_tokens":1016,"completion_tokens":2065,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":1955}},"tokens_in":632,"tokens_out":2065,"duration_ms":12818,"temperature":1.0,"reasoning_tokens":1955,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:35:29.381484+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same data collection protocol but label ground truth with a motion-capture system or multi-view stereo reconstruction, then recompute the mean per-joint position error; if the error against that ground truth is substantially larger than 72.73 mm, the pose estimates are not as accurate as reported. A simpler check: place the hand at known distances and orientations relative to the radar and verify that the predicted keypoint scale matches the measured millimeter scale.","supporting_citations":[{"cited_title":"Egocentric human pose estimation using head-mounted mmwave radar,","cited_arxiv_id":null,"evidence_quote":"Provides the head-mounted radar plus IMU hardware configuration and the egocentric sensing precedent that mmEgoHand builds on."},{"cited_title":"Person-in- wifi 3d: End-to-end multi-person 3d pose estimation with wi-fi,","cited_arxiv_id":null,"evidence_quote":"Supplies the end-to-end set-based decoder architecture and Hungarian matching scheme adapted here for variable hand counts."},{"cited_title":"Real-time arm gesture recognition in smart home scenarios via millimeter wave sensing,","cited_arxiv_id":null,"evidence_quote":"Provides the radar heatmap preprocessing pipeline (range, Doppler, and angle FFT) that turns sparse point clouds into the network input."},{"cited_title":"The hungarian method for the assignment prob- lem,","cited_arxiv_id":null,"evidence_quote":"Defines the Hungarian assignment algorithm used to match predicted hand candidates to ground-truth keypoints."},{"cited_title":"Argus: Multi-View Egocentric Human Mesh Reconstruction Based on Stripped-Down Wearable mmWave Add-on","cited_arxiv_id":"2411.00419","evidence_quote":"Demonstrates egocentric head-mounted radar tracking in a multi-view add-on, serving as a direct comparative precursor for wearable mmWave sensing."},{"cited_title":"Leveraging the properties of mmwave signals for 3d finger motion tracking for in- teractive iot applications,","cited_arxiv_id":null,"evidence_quote":"Is the single-hand mmWave pose baseline that mmEgoHand reimplements and outperforms on mean per-joint position error."},{"cited_title":"mtranssee: Enabling environment-independent mmwave sensing based gesture recognition via transfer learning,","cited_arxiv_id":null,"evidence_quote":"Is a state-of-the-art environment-independent mmWave gesture recognition baseline used for comparison in the downstream task."},{"cited_title":"mmges- ture: Semi-supervised gesture recognition system using mmwave radar,","cited_arxiv_id":null,"evidence_quote":"Is a semi-supervised mmWave gesture recognition baseline used for comparison in the downstream task."}],"review_version":1}