{"id":"3a192c33-e5ab-43b2-a6f8-6c2b6bcc5638","arxiv_id":"2411.11288","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Neuron uses multi-turn LLM descriptions and evolving spatial-temporal micro-prototypes to improve zero-shot skeleton action recognition.","lead":"This paper presents Neuron, a framework for zero-shot skeleton action recognition that combines multi-turn LLM descriptions with evolving spatial-temporal prototypes. It reports state-of-the-art accuracy on three benchmarks, but the inference rule and several hyperparameters are under-specified.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Set-valued final prediction makes 'Top-1' accuracy a two-chance evaluation; reported SOTA margins may be an artifact of the metric.","rationale":"The reader's stated weakest assumption was that CLIP embeddings of multi-turn LLM descriptions are discriminative enough to separate fine-grained skeleton actions; that is a plausible concern about attribution, but it is downstream of a more direct problem. The paper's own equations define a set-valued prediction and then measure accuracy as membership in that set, which is not comparable to the single-label accuracy used by the baselines. This challenge is internal to the manuscript, not a matter of consensus, and it affects every reported performance number, including the central SOTA claim. The reader's rationale did mention the ambiguous final prediction rule, so there is partial overlap, but the reader did not identify it as the weakest assumption. The concern is concrete and testable: if a valid single-label fusion rule is provided and the tables are recomputed, the empirical claim can be assessed. I therefore keep the reader's CONDITIONAL verdict as unchanged, because the manuscript needs this clarification and re-evaluation before its central claim can be accepted.","tokens_in":13975,"tokens_out":7119,"duration_ms":72070,"concrete_test":"Rerun Tables 2-4 with a single fused prediction rule, for example y* = argmax_{y in Y^u/Y} [rho_s(X_s^3)^T phi(Z_hat_{s,y}^3) + rho_t(X_t^3)^T phi(Z_hat_{t,y}^3)], or any other explicitly stated combination of the two streams, and recompute Acc and H. If the single-label numbers fall below the reported values by more than a small margin (e.g., >1 point), or no longer exceed STAR, the SOTA claim is an artifact of the set-valued metric rather than of the proposed method.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.5 defines the final prediction for each test sample as the two-element set y* = {y_s, y_t}, where y_s and y_t are obtained independently from the spatial and temporal streams (Eqs. 9-10). Section 4.2 then defines Top-1 accuracy as Acc = (1/N) sum_i I[y_i in y*]. Because y* is a set, a sample is counted correct if either stream happens to identify the true class. This is not the single-label Top-1 accuracy used by RelationNet, STAR, or the other baselines in Tables 2-4, so the reported gains (e.g., +5.5 ZSL on NTU60 Xsub 55/5 over STAR) are not measured on the same basis. No fusion rule is supplied that would collapse the two stream predictions into one label (e.g., summing their similarities or a learned confidence gate). Until that rule is specified and the tables are recomputed, the central SOTA claim cannot be evaluated from the manuscript as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Neuron, a zero-shot skeleton action recognition framework that combines multi-turn LLM-generated \"context-aware side information\" with spatial and temporal micro-prototypes that evolve over several phases. The spatial stream uses a compression mechanism to retain only top-α joint-prototype similarities, while the temporal stream splits the sequence into three phases and uses a memory mechanism to combine recall and new information. Training aligns pooled prototype features with CLIP text embeddings of the generated semantics at each phase. The paper reports state-of-the-art ZSL and GZSL results on NTU RGB+D 60, NTU RGB+D 120, and PKU-MMD, together with ablations and qualitative visualizations.","tokens_in":14240,"tokens_out":4618,"duration_ms":47891,"significance":"The core idea—using multi-turn LLM descriptions to guide progressively refined micro-prototypes—is a reasonable and potentially useful extension of recent fine-grained zero-shot skeleton recognition work, and the paper provides a substantial set of experiments, ablations, and code. If the results are correctly measured, the observed margins over STAR and other baselines would be notable. However, the central state-of-the-art claim is compromised as written by a nonstandard evaluation rule and by unresolved inconsistencies in the objective and hyperparameters. The significance of the empirical contribution therefore cannot be confirmed from the current manuscript.","major_comments":[{"comment":"The final prediction is defined as the two-element set \\hat{y}^* = {\\hat{y}_s, \\hat{y}_t}, and the reported Top-1 accuracy is Acc = (1/N) \\sum_i I[y_i \\in \\hat{y}^*]. This is not the standard single-label Top-1 accuracy used by RelationNet, STAR, and the other baselines in Tables 2-4: a test sample is counted correct if either stream independently produces the right class, giving the method two chances per sample. Consequently the reported SOTA gains, such as +5.5 ZSL on NTU60 Xsub 55/5 over STAR, are not measured on the same basis as the baselines. The authors need to supply a deterministic fusion rule that collapses the two stream predictions into one label (for example, summing the two similarity scores, or a learned confidence gate) and recompute all tables under that single-label rule.","section":"Sections 3.5 and 4.2, Eqs. (9)-(10)"},{"comment":"The implementation details state that \"the hyperparameter \\lambda_s and \\lambda_t are set to 0.0003 and 0.0002\", but the overall objective in Eq. (8) contains no such coefficients: it is a plain sum of L_s^e and L_t^e. Either the equations are missing weighting terms, or the text is describing unused parameters. In addition, the spatial compression threshold α is described only as \"incrementally increased\" without a schedule or actual values. Both issues block exact reproduction of the reported results and should be corrected.","section":"Section 4.3 and Section 3.4, Eq. (8)"},{"comment":"All reported comparisons are single-run results without error bars or standard deviations. Table 5 says three different seen-unseen partitions were averaged \"to minimize variance\", but no variance is reported. Given that the central claim is state-of-the-art performance, the authors should report mean and standard deviation over multiple runs (at least with different seeds) for the main tables, and for the averaged settings in Table 5.","section":"Tables 2-4, Section 4.4, and Table 5"},{"comment":"The context-aware side information is a central contribution, but the generation process is not reproducible from the paper: no multi-turn prompt templates are provided, no example generated descriptions are shown, and the number of descriptions N_a per phase is never specified. Since the method's separation and claimed benefit depend on these LLM-generated semantics, the authors should include the exact prompting protocol and sample outputs (or a link to them) so that reviewers and readers can judge the semantic quality and reproduce the experiments.","section":"Section 3.2"}],"minor_comments":[{"comment":"The row for the proposed method contains the typo \"Neruon\"; it should be \"Neuron\".","section":"Table 1"},{"comment":"The header contains \"A cc\" in several places; this should be \"Acc\".","section":"Table 2"},{"comment":"The first column header reads \"Xusb\"; it should be \"Xsub\".","section":"Table 3"},{"comment":"The sentence \"In the inference stage, we can contain the spatial-temporal features\" should be \"we can obtain\".","section":"Section 3.5"},{"comment":"The temporal feature F_t^0 is not defined: Section 3.3 defines temporal segments as F_t = {F_t^1, ..., F_t^{N_e}}, so Eq. (3) should use F_t^1 (and similarly in Eq. (4)).","section":"Section 3.3, Eq. (3)"},{"comment":"The inference equations introduce functions \\rho_s, \\rho_t, and \\phi that are not defined in the training objective, where the corresponding projection functions are \\phi_s, \\phi_t, and \\psi. The notation should be unified or explicitly distinguished.","section":"Eqs. (6)-(7) vs. Eqs. (9)-(10)"},{"comment":"The paragraphs end with the literal text \"[Summary]:\"; these appear to be leftover drafting placeholders and should be removed or converted into normal prose.","section":"Sections 2.1 and 2.2"},{"comment":"The text states that the best performance is obtained with N_s around 100 and N_t around 50, yet the implementation sets both N_s and N_t to 80. Please clarify which values were used for the reported results.","section":"Section 4.3 and Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a serious empirical contribution, and the core framework is reasonable. The main concern is not the idea but the evaluation: the set-valued prediction rule changes the metric relative to all baselines and makes the headline SOTA claim unverifiable as written. The missing \\lambda_s/\\lambda_t terms and the unspecified \\alpha schedule are additional reproducibility problems. These are all fixable within the scope of a revision, so I recommend major revision rather than rejection. I would also encourage the editor to ask for the prompt templates and generated side-information examples, since the method's central claim depends on them."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: Neuron extends your own STAR line with multi-turn LLM side information, spatial compression via top-alpha selection, and temporal memory with recall/remember gating. The ablations show each piece helps. But the headline SOTA numbers are currently not comparable to any baselines, because the paper changes the evaluation rule without saying so. The stress-test note is right, and it is load-bearing.\n\nSection 3.5 defines the final prediction as the set y* = {y_s, y_t}, where y_s and y_t come from independent argmaxes across the spatial and temporal streams. Section 4.2 then defines Top-1 accuracy as I[y_i ∈ y*]. That is a two-chance oracle: a sample counts correct if either stream happens to guess the class. STAR, RelationNet, and the other baselines report single-label Top-1. So the +5.5 ZSL improvement over STAR on NTU60 Xsub 55/5 could be an artifact of this set-membership rule. No fusion rule is supplied that would collapse the two stream predictions into one label. Until that is specified and the tables are recomputed on the same basis, the central claim cannot be evaluated from the manuscript.\n\nWhat is genuinely new is reasonable: context-aware side information from multi-turn prompting is a sensible upgrade over one-turn homogeneous descriptions, and the evolving micro-prototypes with spatial compression and temporal memory are legitimate mechanisms. The paper would be worth reviewing if the evaluation were fixed. The minor soft spots: λ_s and λ_t are listed in the implementation but never appear in the objective, and the α schedule is only described as \"incrementally increased\" without specifics. No error bars are reported, and the 13-point jump on PKU-MMD (76.3 to 89.2) deserves extra scrutiny.\n\nI don't think this is a takedown of the method; the mechanisms could still work. But as written, the reported SOTA is on a different basis than the baselines. The authors should clarify the inference rule, release code with a commit hash, specify all hyperparameters, and report variance. A good reviewer would catch this, and the paper should not be desk-rejected. It deserves peer review, but the next version had better make the evaluation apples-to-apples.","headline":"The evaluation metric makes the reported SOTA gains non-comparable to baselines, but the mechanisms are worth a serious look.","tokens_in":14727,"tokens_out":2128,"would_cite":false,"duration_ms":22102,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Neuron learns evolving skeleton prototypes to recognize unseen actions from text descriptions.","keywords":["zero-shot skeleton action recognition","generalized zero-shot learning","context-aware side information","micro-prototypes","spatial compression","temporal memory","LLM-generated action descriptions","cross-modal alignment"],"falsifier":"Take the context-aware descriptions generated for a confusable pair such as 'walking apart' versus 'walking towards' or 'sneeze/cough' versus 'nausea/vomiting', embed them with the same CLIP text encoder, and compute their cosine similarity; if the similarity is near 1.0 at all three phases, the semantic space cannot support the claimed fine-grained separation, and the method's reported gains would have to come from elsewhere. A complementary test is to rerun the full pipeline with the multi-turn contextual descriptions replaced by one-turn single descriptions while keeping everything else fixed; a small accuracy drop on the hardest split would indicate that the context-aware side information is not actually load-bearing.","tokens_in":13789,"feed_emoji":"🦴","tokens_out":3716,"duration_ms":38262,"temperature":0.7,"pith_summary":"The paper aims to improve zero-shot skeleton action recognition by replacing static, one-line semantic descriptions with context-rich side information generated through multi-turn LLM prompting, and by aligning skeleton features to this semantics through learnable spatial and temporal micro-prototypes that grow iteratively. It claims this dynamic, controllable alignment from micro to macro perspectives captures fine-grained cross-modal correspondences more effectively than prior global or static alignment schemes. If correct, the method should transfer knowledge from seen to unseen action categories more reliably, reducing the confusion between visually similar actions. The paper reports state-of-the-art accuracy on NTU RGB+D, NTU RGB+D 120, and PKU-MMD under both zero-shot and generalized zero-shot protocols.","feed_headline":"Evolving prototypes lift zero-shot skeleton recognition past prior best","feed_subtitle":"Multi-turn LLM descriptions plus growing spatial-temporal prototypes narrow the gap on NTU and PKU-MMD benchmarks.","key_machinery":"The central object is the spatial-temporal micro-prototype pair: learnable attribute vectors initialized randomly and updated over three phases, intended to capture structure-related spatial patterns and regularity-dependent temporal patterns. Spatial prototypes are refined by a spatial compression mechanism that thresholds joint-prototype similarity scores with a gradually increasing retention ratio alpha, while temporal prototypes are updated by a temporal memory mechanism with sigmoid-gated recall and remember branches to prevent forgetting across phases. These prototypes are guided at each phase by context-aware side information, produced via multi-turn LLM prompting and embedded with a frozen CLIP text encoder, and aligned to skeleton features through per-phase contrastive losses, providing the micro-to-macro progressive alignment that carries the generalization claim.","core_discovery":"The paper's central claim is that zero-shot skeleton action recognition improves when semantics are both context-aware and evolving, and when skeleton features are aligned to them through growing prototypes rather than through a single static embedding. Concretely, the proposed Neuron framework first queries an LLM in multiple turns to generate spatial descriptions at coarse, mid, and fine granularities and temporal descriptions at start, mid, and end phases, producing heterogeneous, context-rich side information for each action class. It then constructs spatial and temporal micro-prototypes: the spatial prototype grows through a spatial compression mechanism that keeps only the top-alpha joint-prototype similarity scores per phase (alpha increasing), while the temporal prototype grows through a temporal memory mechanism that combines recall of previous phases with absorption of new temporal patterns via sigmoid-gated updates. At each phase, the prototypes are pooled into a skeleton representation and aligned to the corresponding CLIP-embedded semantics through contrastive losses, progressively refining the cross-modal alignment. The paper argues that this stepwise, controllable alignment avoids shortcut feature learning and knowledge oblivion, and the reported experiments show accuracy gains over previous methods on three benchmarks.","pith_inferences":["Since the method relies on a frozen CLIP text encoder, a direct test of its load-bearing premise would be to compute pairwise cosine similarities of the CLIP embeddings of generated descriptions for confusable action pairs; if those similarities remain very high at all phases, the semantic space alone cannot support the reported fine-grained alignment.","The paper does not independently validate the semantic quality of the multi-turn LLM descriptions; replacing the LLM with a weaker model, or shuffling the descriptions across classes, would provide a clean diagnosis of how much of the gain comes from the context-aware side information versus the evolving prototype alignment.","The micro-to-macro prototype growth idea may transfer to other skeleton-language tasks such as text-based motion retrieval or instruction-conditioned action generation, where stepwise semantic guidance could similarly help control alignment.","The reported failure on 'sneeze/cough' versus 'nausea/vomiting' suggests that the current context-aware descriptions still carry homogeneous semantics for extremely similar abstract actions; further refining the prompting strategy or using complementary visual descriptions might be a testable extension."],"forward_implications":["For zero-shot skeleton action recognition, richer context-aware side information improves accuracy on both seen and unseen classes, reducing the domain bias problem in generalized zero-shot settings.","The spatial compression mechanism, by discarding least relevant prototype attributes per phase, reduces shortcut feature learning and improves generalization, as shown in the component ablation.","The temporal memory mechanism mitigates knowledge oblivion across phases, which the paper links to better recognition of temporally similar actions such as 'walking apart' versus 'walking towards'.","Performance remains stable across a range of prototype counts, suggesting the framework is not highly sensitive to the number of spatial or temporal micro-prototypes chosen."],"supporting_citations":[{"why":"The main baseline STAR, which uses dual prompts and fine-grained side information; the paper compares directly against it on all benchmarks.","marker":"[6]"},{"why":"Provides the frozen CLIP text encoder that embeds the multi-turn LLM semantic descriptions, the semantic space used throughout the alignment.","marker":"[28]"},{"why":"PURLS, a part-aware unified representation method with LLM-generated part descriptions, which the paper extends by making semantics context-aware and dynamic.","marker":"[41]"},{"why":"SA-DAVAE, a recent disentangled variational autoencoder baseline for zero-shot skeleton recognition, used for comparison in generalized zero-shot learning.","marker":"[21]"},{"why":"SMIE, a mutual-information alignment baseline with a temporal constraint, representing a prior approach to skeleton-semantic alignment.","marker":"[39]"},{"why":"RelationNet, a pioneering global-alignment method for zero-shot skeleton action recognition, illustrating the older local-alignment paradigm the paper contrasts with.","marker":"[18]"},{"why":"Calibrated stacking, used in the GZSL prediction step to mitigate the domain shift between seen and unseen categories.","marker":"[4]"},{"why":"Shift-GCN, the skeleton encoder pretrained on seen categories whose features the micro-prototypes refine.","marker":"[8]"}],"fun_headline_variants":["Evolving context-aware prototypes sharpen zero-shot skeleton recognition","Neuron's dual prototypes evolve with context for zero-shot skeleton","LLM-driven prototype growth improves zero-shot skeleton action recognition","Evolving prototypes with context lift zero-shot skeleton past baselines","Micro-prototypes grow with side info to boost zero-shot skeleton recognition"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the frozen CLIP text encoder, when fed multi-turn LLM-generated spatial and temporal action descriptions, produces semantic embeddings that are discriminative enough to separate fine-grained skeleton actions, so that the per-phase contrastive alignment can succeed.","fun_headline_variants_meta":{"raw":{"variants":["Evolving context-aware prototypes sharpen zero-shot skeleton recognition","Neuron's dual prototypes evolve with context for zero-shot skeleton","LLM-driven prototype growth improves zero-shot skeleton action recognition","Evolving prototypes with context lift zero-shot skeleton past baselines","Micro-prototypes grow with side info to boost zero-shot skeleton recognition"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001057,"raw_usage":{"total_tokens":4461,"prompt_tokens":993,"completion_tokens":3468,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":3384}},"tokens_in":609,"tokens_out":3468,"duration_ms":24782,"temperature":1.0,"reasoning_tokens":3384,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:41:55.533874+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the context-aware descriptions generated for a confusable pair such as 'walking apart' versus 'walking towards' or 'sneeze/cough' versus 'nausea/vomiting', embed them with the same CLIP text encoder, and compute their cosine similarity; if the similarity is near 1.0 at all three phases, the semantic space cannot support the claimed fine-grained separation, and the method's reported gains would have to come from elsewhere. A complementary test is to rerun the full pipeline with the multi-turn contextual descriptions replaced by one-turn single descriptions while keeping everything else fixed; a small accuracy drop on the hardest split would indicate that the context-aware side information is not actually load-bearing.","supporting_citations":[{"cited_title":"Fine-grained side information guided dual-prompts for zero-shot skeleton action recognition","cited_arxiv_id":null,"evidence_quote":"The main baseline STAR, which uses dual prompts and fine-grained side information; the paper compares directly against it on all benchmarks."},{"cited_title":"Part-aware unified representation of language and skele- ton for zero-shot action recognition","cited_arxiv_id":null,"evidence_quote":"PURLS, a part-aware unified representation method with LLM-generated part descriptions, which the paper extends by making semantics context-aware and dynamic."},{"cited_title":"Sa-dvae: Improv- ing zero-shot skeleton-based action recognition by disentan- gled variational autoencoders","cited_arxiv_id":null,"evidence_quote":"SA-DAVAE, a recent disentangled variational autoencoder baseline for zero-shot skeleton recognition, used for comparison in generalized zero-shot learning."},{"cited_title":"Zero-shot skeleton-based action recogni- tion via mutual information estimation and maximization","cited_arxiv_id":null,"evidence_quote":"SMIE, a mutual-information alignment baseline with a temporal constraint, representing a prior approach to skeleton-semantic alignment."},{"cited_title":"An empirical study and analysis of generalized zero- shot learning for object recognition in the wild","cited_arxiv_id":null,"evidence_quote":"Calibrated stacking, used in the GZSL prediction step to mitigate the domain shift between seen and unseen categories."},{"cited_title":"Skeleton-based action recognition with shift graph convolutional network","cited_arxiv_id":null,"evidence_quote":"Shift-GCN, the skeleton encoder pretrained on seen categories whose features the micro-prototypes refine."}],"review_version":1}