{"id":"7ca350c8-3dad-44e4-ba6b-43bb52c4b3b5","arxiv_id":"2412.16698","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SocialEgoNet jointly forecasts interaction intent, attitude, and actions from one second of egocentric skeleton video on a newly labeled version of the JPL-Interaction dataset.","lead":"This paper introduces SocialEgoNet, a model that watches one second of egocentric video and predicts whether a person wants to interact with the agent, their attitude, and which action they will perform. The work also adds new person-level labels to an existing interaction dataset and reports real-time performance, though a baseline model actually scores higher on accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Attitude labels are a deterministic function of action classes (Section III), so the reported 91.11% attitude accuracy does not demonstrate forecasting of an independent social signal.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: attitude is defined by action class, so attitude forecasts are not an independent social signal. I agree this is the most fundamental issue because it affects the construct validity of one of the three headline tasks and the interpretation of the hierarchical classifier's gains, rather than merely statistical precision or baseline selection. The lack of error bars and the absence of released code/data are also real limitations, but they are secondary to the deterministic label dependency: even with perfect experimental reporting, the attitude accuracy would still be a re-expression of action-group discrimination. The paper still contributes a real-time skeleton-based action and intent forecasting pipeline and a new annotated dataset, so a full rejection is not warranted. The CONDITIONAL verdict already flags this concern and is the appropriate level: the authors should either re-annotate attitude independently of action classes, or reframe the contribution to avoid claiming attitude as a separate forecasted signal. No change to the reader's verdict is needed.","tokens_in":8738,"tokens_out":5008,"duration_ms":46031,"concrete_test":"Run the following check on the same train/test split used for Table II: take SocialEgoNet's action predictions and map them to attitude predictions using the annotation rule in Section III (punching/throwing -> negative, all other actions -> positive). If the resulting attitude accuracy matches or exceeds the 91.11% reported by the dedicated attitude head, then the attitude head provides no independent predictive information. Also print the contingency table of action vs attitude labels in JPL-Social; if it is exactly a block matrix with no within-action variation, the attitude labels are deterministic and the joint-forecasting claim for attitude is not supported by the data.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the construction of the attitude labels in Section III: 'Punching or Throwing an object to the observer are labelled as having a negative attitude while the rest are considered as having a positive attitude.' This makes attitude an exact deterministic relabeling of the 10-class action taxonomy (2 negative versus 8 positive classes). As a result, (i) the attitude task carries zero information beyond distinguishing punching/throwing from the other actions; (ii) the 91.11% attitude accuracy in Table II is not evidence that SocialEgoNet forecasts a separate social attitude, only that it can partially discriminate those action groups; and (iii) the Chain classifier (Sec. IV-C), which explicitly treats attitude as the parent class of actions, encodes this annotation rule into the architecture, so any accuracy gain attributed to hierarchical multitask learning may simply reflect the labels' deterministic structure. The headline average accuracy of 83.15% is therefore inflated by a task that is not independent of the action task, weakening the paper's central claim of jointly forecasting three distinct social signals. This is a construct-validity problem with the dataset, not a mere implementation flaw.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SocialEgoNet, a graph-based spatiotemporal model that jointly forecasts a person's intent to interact, attitude, and action from one second of egocentric whole-body skeleton video. The authors also introduce JPL-Social, an augmented version of the JPL-Interaction dataset with person-level labels for these three tasks. The model uses separate GCNs for body, face, and hand keypoints, fuses them with multi-head self-attention, and models temporal dependencies with a Bi-LSTM before passing features to a hierarchical classifier. On 290 person tracks, the model achieves an average accuracy of 83.15%, outperforming R3D-18 and ST-GCN but not MS-G3D, while being smaller and faster. The paper reports ablations on body-part combinations, hierarchy designs, and observation window size.","tokens_in":9066,"tokens_out":3931,"duration_ms":38341,"significance":"If the three tasks were genuinely independent social signals, the paper would present a practical real-time framework and a useful dataset contribution. The strengths of the work are the whole-body keypoint design, the real-time inference claim, the body-part and hierarchy ablations, and the planned release of annotations and code. However, the construct validity of the attitude and intent labels is a central issue: attitude is defined directly from the action taxonomy, and the interested/not-interested distinction for non-interacting persons is essentially a gaze/no-response label. As a result, the reported accuracy numbers overstate the degree to which the model forecasts three distinct social cues. The empirical comparisons are plausible but are supported by a single run with no error bars or significance tests.","major_comments":[{"comment":"The attitude ground truth is a deterministic function of the action class: 'Punching or Throwing an object to the observer are labelled as having a negative attitude while the rest are considered as having a positive attitude.' Therefore the 91.11% attitude accuracy in Table II does not measure an independent social attitude; it measures the model's ability to separate two action classes (punching and throwing) from the other eight. The paper should either collect independent attitude annotations (e.g., from human judges) or explicitly reframe attitude as a label derived from the action taxonomy rather than a separate social signal.","section":"Section III, attitude annotation"},{"comment":"The intent-to-interact labels for non-interacting persons are also nearly deterministic: a person is 'interested' when constantly looking at the observer, and the new action class 'gaze' is defined as looking at the observer, while 'no response' is defined as ignoring the observer. Thus the interested/not-interested split is essentially a relabeling of the gaze/no-response action classes. The paper should clarify the exact mapping between intent classes and action classes and discuss the implication that intent, attitude, and action are not three independent annotation dimensions.","section":"Section III, intent annotation"},{"comment":"The experimental protocol is under-specified: the paper does not state the train/test split, whether the split is at the video level or the person-track level, the number of runs, or any variance statistics. With only 290 person-level samples and strong data augmentation, the reported margins over ST-GCN (2.33% average accuracy) and the hierarchy gains in Table IV may not be statistically reliable. The authors should report repeated runs with means and standard deviations, or at minimum a confidence interval, and describe the split protocol precisely.","section":"Section V, Table II"},{"comment":"The Chain classifier explicitly encodes the annotation rule that attitude is the parent class of actions, because attitude is defined by whether the action is punching/throwing or not. The observed improvement of Chain over Parallel and Tree may therefore simply reflect the deterministic group structure of the labels, not a learned semantic hierarchy. To support the claim that the hierarchical structure learns complementary information between tasks, the paper needs a control condition (e.g., a hierarchy based on a random grouping of actions) or independent attitude labels.","section":"Section IV-C and Table IV"}],"minor_comments":[{"comment":"The text says the model outperforms 'several competitive baselines,' but MS-G3D achieves the best accuracy on every task. The paper should acknowledge this explicitly and frame the contribution as a favorable accuracy/speed/parameter trade-off rather than overall state-of-the-art accuracy.","section":"Section V-A and Table II"},{"comment":"The latency comparison is not fully apples-to-apples: graph-based methods include pose extraction time (Delta), while R3D-18 does not. The paper should state the timing comparison conditions more clearly, including whether pose extraction is included for all methods.","section":"Table II caption and Section V-A"},{"comment":"The text says every person track is annotated with three labels each, but later states that attitude is applicable only to interacting subjects. This is contradictory; the paper should clarify how attitude is assigned and evaluated for non-interacting tracks.","section":"Section III, 'annotated with three labels each'"},{"comment":"The class distribution figure is difficult to read in the text; providing the exact per-class counts in a table or in the caption would improve reproducibility and transparency.","section":"Fig. 3"},{"comment":"The dimension notation 'z(t) = Concat(...) in R^{n x 16}' is unclear because n is not defined; please clarify the shape of the concatenated feature.","section":"Algorithm 1, line 5"},{"comment":"The name 'MS-G3D Net' in Section V-A and 'MS-G3D' in Table II should be made consistent.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The central empirical results may be sound as action-forecasting numbers, but the paper's framing of three independent social signals is not supported by the label construction. This is a fixable issue if the authors either collect independent attitude/intent annotations or substantially reframe the tasks as derived labels. I would also encourage the editor to request the statistical details (split protocol, repeated runs) before any acceptance decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is real: jointly forecasting intent to interact, attitude, and action from one second of egocentric skeleton video, with person-level annotations (JPL-Social) on the JPL dataset. That task formulation and the annotation effort are worth having. The model itself is a sensible assembly of known pieces—AlphaPose, separate GCNs for face/body/hands, attention, Bi-LSTM, hierarchical heads—and the paper is honest enough to report that MS-G3D beats it on every metric, arguing instead for smaller size and faster inference. The hierarchy ablation (Table IV) and the body-part study (Table III) are useful and informative, and the real-time claim is credible.\n\nThe soft spot, and it is a load-bearing one, is the attitude label. Section III says punching and throwing are negative and everything else is positive, making attitude a deterministic function of the 10 action classes. So the reported 91.11% attitude accuracy is not evidence that the model forecasts an independent social attitude; it mostly measures whether the model can tell punching/throwing apart from the other eight actions. The Chain classifier explicitly treats attitude as the parent class of actions, which mirrors this annotation rule, so the hierarchy gains may just reflect the label structure rather than genuine multitask learning. The paper should either re-annotate attitude independently or clearly frame it as a derived attribute and stop claiming three distinct social signals.\n\nSmaller issues: 290 person tracks from 200 lab videos is tiny; there are no error bars or significance tests, just a single run; and the code and data are not released, which limits independent verification. The \"superior performance\" phrasing overreaches when MS-G3D is better on all metrics; \"competitive with a smaller model\" is the fair claim. None of this is fatal, and the intent and action forecasting parts stand on their own.\n\nWho should read this: people building social robots or working on egocentric action anticipation. It supplies a new benchmark and a reasonable baseline, even if the attitude task needs rethinking. I would send it to peer review—conditional on the authors fixing the attitude construct-validity problem or downgrading the claim, adding error bars, and releasing the annotations.","headline":"A genuinely new joint forecasting task and a useful annotated egocentric dataset, but the attitude label is a deterministic relabeling of the action taxonomy, so the headline accuracy partly measures action discrimination rather than an independent social signal.","tokens_in":9475,"tokens_out":1210,"would_cite":true,"duration_ms":16091,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes that a compact graph network, SocialEgoNet, can jointly forecast a person's intent to interact, their attitude, and their next action from one second of egocentric skeleton video.","keywords":["egocentric vision","human-robot interaction","intent prediction","skeleton-based action recognition","multitask learning","graph convolutional network","social interaction forecasting"],"falsifier":"Remove the attitude head from SocialEgoNet and, on the held-out test split, map the predicted action labels to attitude labels through the same annotation rule; if the resulting accuracy is statistically indistinguishable from the reported 91.11%, then the attitude task adds no predictive information beyond action classification. A complementary test: have independent annotators label attitude from facial expression and body language without seeing the action label, and check whether SocialEgoNet trained on those labels performs above chance.","tokens_in":8503,"feed_emoji":"🤖","tokens_out":5795,"duration_ms":46307,"temperature":0.7,"pith_summary":"This paper proposes a new task for social robotics: jointly forecasting, from one second of egocentric video, whether a person intends to interact, what attitude they have toward the observer, and which of ten actions they will perform next. To solve it, the authors build SocialEgoNet, a graph-based spatiotemporal network that reads whole-body skeleton keypoints from the face, hands, and body and uses a hierarchical classifier to exploit dependencies among the three tasks. On the augmented JPL-Social dataset, SocialEgoNet attains 83.15% average accuracy across all tasks with real-time inference, outperforming R3D-18 and ST-GCN on every task while being smaller and faster. The work also provides person-level annotations that turn the JPL egocentric interaction dataset into a benchmark for joint social forecasting.","feed_headline":"Joint forecast of intent, attitude, action hits 83.15%","feed_subtitle":"Robots can read intent, attitude, and next action from one second of skeleton video.","key_machinery":"The central machinery is SocialEgoNet, a three-stage pipeline: (1) AlphaPose extracts whole-body keypoints (68 face, 23 body, 42 hand) and constructs a pose graph for each part; (2) three three-layer GCNs produce spatial embeddings that are concatenated, passed through multi-head self-attention, and fed into a three-layer Bi-LSTM to capture temporal dynamics across the 30-frame window; (3) a hierarchical classifier with a chain structure first predicts intent, then attitude, then action, mimicking the order in which a person sizes up an approaching human. The JPL-Social dataset—the other supporting artifact—adds person-level tracks, bounding boxes, and new labels for gaze and no-response actions to the original JPL videos, yielding 290 annotated person tracks.","core_discovery":"The central claim is that a lightweight graph network operating purely on whole-body keypoints can jointly forecast a person's intent to interact (interacting, interested, not interested), their attitude (positive or negative), and their upcoming action (ten classes) from the first second of an egocentric interaction, and that doing so jointly with a chain-structured hierarchical classifier outperforms treating the tasks independently. SocialEgoNet achieves 88.10% intent accuracy, 91.11% attitude accuracy, and 70.24% action accuracy, for an average of 83.15%, with 3.18 million parameters and sub-millisecond graph inference speed, beating R3D-18 and ST-GCN; the larger MS-G3D obtains higher accuracy (85.08%) but is about eight times slower. The paper further claims that each body part contributes complementary information: face keypoints dominate intent and attitude prediction, while body and hand keypoints matter most for action prediction.","pith_inferences":["Since attitude labels in JPL-Social are defined entirely by the action class (punching and throwing negative, everything else positive), the reported 91.11% attitude accuracy should be read as measuring action discrimination; a genuinely independent attitude signal would need human-annotated valence labels not derived from the action taxonomy.","The same architecture could be extended to multi-person scenes by running the person-track module on all detected tracks and adding a ranking head that predicts which person is most likely to initiate interaction; this would test whether the intent-to-interact superclass idea scales to social selection.","Because the model uses only keypoint coordinates and confidence scores, its transfer to new environments is likely limited by pose-estimation quality on unseen camera angles, occlusions, and unusual clothing; a stress test with noisy or absent keypoints would bound its robustness.","The hierarchy assumption that intent is a superclass of attitude and action could be tested causally by intervening on the training labels (shuffling attitude labels) and measuring whether the action head's accuracy drops; that would reveal how much the chain structure truly carries information."],"forward_implications":["If SocialEgoNet's results hold, robots and embodied agents can use one-second skeleton input to prepare a response before a human reaches them, without using raw RGB frames, preserving privacy and saving compute.","The chain hierarchy shows a measurable gain over parallel or single-task heads (82.33% vs 80.13% and 78.93% average F1), supporting the design principle that intent and attitude are useful superclasses for action forecasting.","Because face keypoints carry most information for intent and attitude, downstream systems could focus camera resolution on the face region when only those two tasks matter.","Opening the observation window from 8 to 30 frames improves attitude and action accuracy more than intent accuracy, suggesting that early intent is readable sooner than valence or specific action."],"supporting_citations":[{"why":"Supplies the original JPL-Interaction egocentric videos with eight action classes that this paper augments into JPL-Social.","marker":"[4]"},{"why":"Extends the same dataset toward robot-centric activity prediction, providing the forecasting motivation and additional video clips.","marker":"[5]"},{"why":"AlphaPose whole-body pose estimator that generates the face, body, and hand keypoints used as SocialEgoNet's input.","marker":"[27]"},{"why":"ST-GCN, the skeleton-based spatiotemporal graph baseline that SocialEgoNet is compared against and which motivates the graph modeling approach.","marker":"[7]"},{"why":"R3D-18, the RGB 3D-CNN baseline that SocialEgoNet outperforms, showing the advantage of keypoint-based input.","marker":"[6]"},{"why":"MS-G3D, the stronger graph baseline that achieves the highest accuracy but with a much larger model and higher latency.","marker":"[33]"},{"why":"COCO keypoint definition and pretraining format that fixes the adjacency matrices connecting the whole-body keypoints.","marker":"[28]"},{"why":"Graph convolutional network layer used to extract spatial representations from each body part's pose graph.","marker":"[29]"},{"why":"Bi-LSTM used to model temporal dependencies over the frame sequence.","marker":"[31]"},{"why":"Multi-head self-attention used to fuse and weight keypoint importance across body parts.","marker":"[30]"}],"fun_headline_variants":["SocialEgoNet jointly forecasts intent, attitude, action at 83.15%","1-second skeleton video predicts intent, attitude, and action in real time","SocialEgoNet: joint intent, attitude, action forecast at 83.15%","Robots read intent, attitude, action from one second of skeleton video"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The attitude label is defined by the action class: punching and throwing count as negative, every other action as positive, so attitude forecasting collapses into action recognition and the high attitude accuracy is not evidence of an independent social signal.","fun_headline_variants_meta":{"raw":{"variants":["SocialEgoNet jointly forecasts intent, attitude, action at 83.15%","1-second skeleton video predicts intent, attitude, and action in real time","SocialEgoNet: joint intent, attitude, action forecast at 83.15%","Robots read intent, attitude, action from one second of skeleton video"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000587,"raw_usage":{"total_tokens":2745,"prompt_tokens":919,"completion_tokens":1826,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":1740}},"tokens_in":535,"tokens_out":1826,"duration_ms":12683,"temperature":1.0,"reasoning_tokens":1740,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:20:43.287684+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Remove the attitude head from SocialEgoNet and, on the held-out test split, map the predicted action labels to attitude labels through the same annotation rule; if the resulting accuracy is statistically indistinguishable from the reported 91.11%, then the attitude task adds no predictive information beyond action classification. A complementary test: have independent annotators label attitude from facial expression and body language without seeing the action label, and check whether SocialEgoNet trained on those labels performs above chance.","supporting_citations":[{"cited_title":"First-person activity recognition: What are they doing to me?,","cited_arxiv_id":null,"evidence_quote":"Supplies the original JPL-Interaction egocentric videos with eight action classes that this paper augments into JPL-Social."},{"cited_title":"Robot-centric activity prediction from first-person videos: What will they do to me?,","cited_arxiv_id":null,"evidence_quote":"Extends the same dataset toward robot-centric activity prediction, providing the forecasting motivation and additional video clips."},{"cited_title":"Alphapose: Whole-body regional multi-person pose estimation and tracking in real-time,","cited_arxiv_id":null,"evidence_quote":"AlphaPose whole-body pose estimator that generates the face, body, and hand keypoints used as SocialEgoNet's input."},{"cited_title":"Spatial temporal graph convolutional networks for skeleton-based action recognition,","cited_arxiv_id":null,"evidence_quote":"ST-GCN, the skeleton-based spatiotemporal graph baseline that SocialEgoNet is compared against and which motivates the graph modeling approach."},{"cited_title":"A closer look at spatiotemporal convolutions for action recognition,","cited_arxiv_id":null,"evidence_quote":"R3D-18, the RGB 3D-CNN baseline that SocialEgoNet outperforms, showing the advantage of keypoint-based input."},{"cited_title":"Disentangling and unifying graph convolutions for skeleton- based action recognition,","cited_arxiv_id":null,"evidence_quote":"MS-G3D, the stronger graph baseline that achieves the highest accuracy but with a much larger model and higher latency."},{"cited_title":"Microsoft coco: Common objects in context,","cited_arxiv_id":null,"evidence_quote":"COCO keypoint definition and pretraining format that fixes the adjacency matrices connecting the whole-body keypoints."},{"cited_title":"Semi-supervised classification with graph convolutional networks,","cited_arxiv_id":null,"evidence_quote":"Graph convolutional network layer used to extract spatial representations from each body part's pose graph."},{"cited_title":"Framewise phoneme classifi- cation with bidirectional lstm networks,","cited_arxiv_id":null,"evidence_quote":"Bi-LSTM used to model temporal dependencies over the frame sequence."},{"cited_title":"Attention is all you need,","cited_arxiv_id":null,"evidence_quote":"Multi-head self-attention used to fuse and weight keypoint importance across body parts."}],"review_version":1}