{"id":"4996c220-c7e5-4eb6-b26e-0aa0481edb3b","arxiv_id":"2606.22986","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":1,"one_line_summary":"Extra Trees outperforms neural open-set methods for unknown-subject rejection in hand landmark identification under LOSO evaluation on the ML2HP dataset using enriched geometric features.","lead":"The paper evaluates methods for identifying unknown subjects using hand landmark data from the Leap Motion Controller 2 on the ML2HP dataset under a leave-one-subject-out protocol with inner validation for rejection thresholds. It finds that a tree ensemble outperforms neural open-set approaches, suggesting utility for compact landmark descriptors in contactless unknown-subject rejection on small cohorts.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Inner threshold from withheld enrolled subject may not match true unknown score distribution, risking biased rejection performance.","rationale":"The reader's weakest assumption is precisely the load-bearing point; the full protocol description confirms the inner-validation design but supplies no empirical check that the proxy threshold generalizes. This leaves the headline claim about Extra Trees being strongest conditional on an untested distributional equivalence. No other internal inconsistency appears in the stated protocol.","tokens_in":1746,"tokens_out":316,"duration_ms":15055,"concrete_test":"For each outer LOSO fold, recompute all method scores using both the inner-validation threshold and the threshold that would be optimal if the true outer unknown were available for tuning; report whether method rankings or absolute metrics (e.g., accuracy or EER) change by more than 5 percentage points.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The protocol withholds one enrolled subject solely for threshold selection, treating its gallery similarity scores as a proxy for unknown rejection. Because this subject is drawn from the same enrolled population (identical acquisition conditions and subject pool), its impostor-score distribution can differ systematically from that of the true outer unknown subject, whose hand geometry and motion patterns are entirely unseen. Any mismatch shifts the operating point for all methods, directly determining whether Extra Trees outperforms the embedding and OpenMax baselines on the reported metrics. The abstract states the inner step is used 'to avoid tuning on the true outer unknown,' yet provides no verification that the resulting threshold remains valid for the actual test unknown.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper studies subject-level unknown-identity identification from Leap Motion Controller 2 hand landmark data on the ML2HP dataset. It enriches geometric landmarks with fingertip-to-palm distances and palm-normalized inter-finger angles, then evaluates under a nested Leave-One-Subject-Out protocol: an outer LOSO fold holds out one subject as unknown, while an inner validation step withholds one enrolled subject solely to select the unknown-rejection threshold. Three methods are compared (Extra Trees on the descriptors, a learned embedding with centroid matching and cosine rejection, and MLP+OpenMax); the abstract states that Extra Trees is strongest overall, implying the key challenge is robust known/unknown score separation rather than enrolled discrimination alone.","tokens_in":1881,"tokens_out":581,"duration_ms":14266,"significance":"If the empirical comparison holds under a validated protocol, the work demonstrates that compact, interpretable landmark descriptors can support contactless open-set hand identification on small-cohort data. The nested validation protocol is a strength, as it explicitly avoids direct tuning on the true outer unknown. Reproducible evaluation on a public dataset with multiple baselines is also a positive.","major_comments":[{"comment":"Abstract and Evaluation Protocol: the central claim that 'Extra Trees remains the strongest overall method' is presented without any quantitative results, error bars, dataset statistics, or per-method metrics in the abstract; the full results section must supply these numbers (including the inner-validation threshold values) to make the comparison verifiable and to confirm that the reported superiority is not an artifact of the specific operating point.","section":"Abstract / Evaluation Protocol"},{"comment":"Evaluation Protocol (inner validation step): the threshold is chosen by withholding one enrolled subject and treating its gallery similarity scores as a proxy for unknown rejection. No analysis or diagnostic is provided showing that the impostor-score distribution of this inner withheld subject matches the distribution of the true outer unknown subject (different hand geometry and motion patterns). Because the operating point directly determines all reported metrics, any systematic mismatch undermines the claim that Extra Trees outperforms the embedding and OpenMax baselines on robust score separation.","section":"Evaluation Protocol"}],"minor_comments":[{"comment":"The abstract states the protocol 'avoids tuning on the true outer unknown' but does not name the concrete threshold-selection criterion (e.g., maximum F1 on the inner fold, fixed false-positive rate); this should be stated explicitly.","section":"Abstract"},{"comment":"Dataset statistics (number of subjects, samples per subject, acquisition conditions) are referenced but not quantified; these belong in a dedicated 'Dataset' subsection or table for reproducibility.","section":"Dataset"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback on our manuscript. We respond point-by-point to the major comments below.","responses":[{"response":"We agree that the abstract states the overall finding without supporting numbers. The results section already reports the full per-method metrics under the nested LOSO protocol. We will revise the manuscript to explicitly list the inner-validation threshold values selected for each method (Extra Trees, embedding baseline, and MLP+OpenMax) and ensure all quantitative results, error bars, and dataset statistics are clearly tabulated for verifiability.","revision_made":"yes","referee_comment":"[Abstract / Evaluation Protocol] Abstract and Evaluation Protocol: the central claim that 'Extra Trees remains the strongest overall method' is presented without any quantitative results, error bars, dataset statistics, or per-method metrics in the abstract; the full results section must supply these numbers (including the inner-validation threshold values) to make the comparison verifiable and to confirm that the reported superiority is not an artifact of the specific operating point."},{"response":"The nested protocol deliberately avoids any exposure to the true outer unknown during threshold selection. We acknowledge that the original submission contains no explicit diagnostic comparing impostor-score distributions between the inner proxy and outer unknowns. We will add this analysis (e.g., distribution plots and summary statistics across folds) in the revision to allow readers to evaluate the proxy's suitability.","revision_made":"yes","referee_comment":"[Evaluation Protocol] Evaluation Protocol (inner validation step): the threshold is chosen by withholding one enrolled subject and treating its gallery similarity scores as a proxy for unknown rejection. No analysis or diagnostic is provided showing that the impostor-score distribution of this inner withheld subject matches the distribution of the true outer unknown subject (different hand geometry and motion patterns). Because the operating point directly determines all reported metrics, any systematic mismatch undermines the claim that Extra Trees outperforms the embedding and OpenMax baselines on robust score separation."}],"tokens_in":1476,"tokens_out":423,"duration_ms":29729,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The core of this work is applying standard open-set tools to Leap Motion hand landmarks on the ML2HP dataset. They keep the raw landmarks, add fingertip-to-palm distances and palm-normalized inter-finger angles, then run a Leave-One-Subject-Out test where one subject is treated as unknown. Extra Trees comes out ahead of a centroid embedding and an MLP+OpenMax model.\n\nWhat stands out is the nested validation: for each outer fold they hold out one enrolled subject only to pick the rejection threshold, keeping the real unknown untouched. That is a reasonable attempt to avoid direct tuning on the test unknown.\n\nThe soft spot is exactly the one the stress-test flags. The inner held-out subject still comes from the enrolled pool and the same acquisition setup, so its impostor scores may not match the distribution of a genuinely new subject. If the score separation shifts, the operating point moves and the ranking of methods could change. The abstract gives no numbers, no error bars, and no dataset size details, so there is no way to check how large the gap is or how stable the threshold choice proves to be.\n\nThis is a narrow, applied study on a small-cohort sensor-specific task. It will interest people working on contactless hand biometrics or small-data open-set problems. The methods are off-the-shelf, so the value sits in the feature choices and the evaluation protocol rather than new theory.\n\nI would bring the full paper to a reading group focused on applied biometrics or open-set evaluation. I would not cite it in my own work unless the numbers turn out strong and the threshold bias is checked. It deserves peer review because the protocol is explicit and the comparison is there; the missing results and the potential distribution mismatch are exactly the kind of issues referees can press on.","headline":"The paper adds geometric hand descriptors and a nested LOSO protocol for unknown-subject rejection on Leap Motion data, but the inner threshold step may not proxy the true unknown well and no numbers are shown.","tokens_in":2387,"tokens_out":452,"would_cite":false,"duration_ms":15452,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Extra Trees with enriched hand landmarks outperforms neural models at identifying known subjects and rejecting unknowns under a nested leave-one-subject-out protocol.","keywords":["hand landmarks","unknown identity identification","open-set recognition","Extra Trees","leave-one-subject-out","geometric descriptors","contactless biometrics"],"falsifier":"Applying the inner-validation threshold to the outer unknown subject produces markedly lower rejection rates than an oracle threshold tuned directly on that unknown subject.","tokens_in":2650,"feed_emoji":"✋","tokens_out":660,"duration_ms":25840,"temperature":0.7,"pith_summary":"The paper evaluates contactless subject recognition from Leap Motion hand landmark data under a protocol that holds out one subject entirely as unknown at test time. An inner validation loop withholds a different enrolled subject only to set the rejection threshold, keeping the true unknown unseen during tuning. Geometric landmarks are augmented with fingertip-to-palm distances and palm-normalized inter-finger angles before feeding tree ensembles, centroid embeddings, and an MLP with OpenMax. Extra Trees yields the strongest combined identification and rejection performance, showing that separating unknown probes from known ones is harder than discriminating among enrolled subjects alone. The results indicate that compact, interpretable landmark descriptors can support unknown-subject rejection and identification on small-cohort datasets.","feed_headline":"Extra Trees rejects unknown hand identities better than neural models","feed_subtitle":"Enriched Leap Motion landmarks plus inner validation let the tree method separate known from unknown probes on small data.","key_machinery":"The nested leave-one-subject-out protocol that uses an inner validation subject solely for threshold selection to avoid bias from the true outer unknown.","core_discovery":"Under the leave-one-subject-out protocol with inner validation for threshold estimation, an Extra Trees classifier on the enriched landmark descriptors achieves the best overall balance of known-subject identification accuracy and unknown-probe rejection, outperforming both a learned embedding baseline and an MLP+OpenMax open-set model; this indicates that the dominant difficulty lies in robust score separation between known and unknown probes rather than in enrolled-subject discrimination.","pith_inferences":["The same protocol could be applied to other landmark-based hand trackers to test device independence.","Adding temporal consistency checks across multiple frames might further improve rejection of unknowns.","Demographic variation in hand geometry could be measured by repeating the protocol on more diverse subject groups.","Fusion with additional modalities such as depth or color could be tested as a direct extension of the current landmark pipeline."],"forward_implications":["Enriched geometric descriptors suffice for the task without requiring large-scale neural training.","Score separation between known and unknown probes remains the primary performance bottleneck.","Tree ensembles can deliver stronger unknown rejection than embedding or OpenMax alternatives under this protocol.","Inner validation enables threshold selection that preserves the unknown-subject guarantee.","Compact landmark methods become practical for contactless unknown-identity scenarios on small cohorts."],"fun_headline_variants":["Extra Trees tops neural models on enriched LMC2 unknown rejection","Leap landmarks show tree edge in subject-level unknown separation","Inner validation aids Extra Trees known-unknown probe distinction","Tree ensemble beats embeddings for LOSO hand identity tasks","Enriched descriptors favor Extra Trees over MLP open-set models"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The rejection threshold estimated from an inner enrolled subject generalizes without bias to the true unknown subject held out in the outer test fold.","fun_headline_variants_meta":{"raw":{"variants":["Extra Trees tops neural models on enriched LMC2 unknown rejection","Leap landmarks show tree edge in subject-level unknown separation","Inner validation aids Extra Trees known-unknown probe distinction","Tree ensemble beats embeddings for LOSO hand identity tasks","Enriched descriptors favor Extra Trees over MLP open-set models"]},"model":"grok-4.3","cost_usd":0.005573,"raw_usage":{"total_tokens":2602,"prompt_tokens":692,"num_sources_used":0,"completion_tokens":76,"cost_in_usd_ticks":55728000,"prompt_tokens_details":{"text_tokens":692,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1834,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":692,"tokens_out":76,"duration_ms":13969,"temperature":1.0,"reasoning_tokens":1834,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T08:50:26.797731+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Applying the inner-validation threshold to the outer unknown subject produces markedly lower rejection rates than an oracle threshold tuned directly on that unknown subject.","supporting_citations":[],"review_version":1}