{"id":"c94c6c58-f8a0-46d6-8b91-dafbd8c62c4d","arxiv_id":"2607.10287","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A first large multimodal 4D human–dog interaction dataset (6.8M frames) plus an autoregressive model that generates dog motion from human body/hand gestures and audio.","lead":"InterPet4D is a new 6.8-million-frame multimodal dataset of humans and dogs interacting, with multi-view video, egocentric views, audio, and 3D poses for both. The authors also train InterPetMoGen, a MotionGPT-style model that generates dog motion from human gestures and speech, beating simple baselines on FID and user ratings.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.5","headline":"Reconstruction quality of dog 3D motion is unquantified, so generation metrics may largely reflect reconstruction artifacts rather than true interaction modeling.","rationale":"The reader correctly isolates the unvalidated dog reconstruction pipeline as the weakest link supporting the strongest claim. The generation architecture itself is a standard MotionGPT-style system; its numerical superiority is only as trustworthy as the 3D dog labels it is trained and scored against. Because no quantitative reconstruction metrics appear anywhere in the main text or supplement, and because the FID feature extractor is trained on the same reconstructed data, the reported FID 11.21 and user-study preference cannot yet be taken as independent evidence of realistic interaction modeling. The concern is therefore load-bearing for the generation half of the claim, while the dataset-resource half remains valuable even if reconstruction is imperfect. This keeps the verdict at CONDITIONAL (pending public data plus reconstruction validation) and does not require a harsher REJECT. No other internal inconsistency or circular derivation is present; the issue is simply missing measurement of the quantity the model is asked to learn.","tokens_in":17415,"tokens_out":597,"duration_ms":7109,"concrete_test":"On a held-out multi-view subset, recompute dog 3D joints with an independent method (e.g., multi-view BARC/BITE or manual keypoint annotation on 200 frames spanning petting and commanding), report MPJPE and bone-length error against the paper’s SMAL fits, and re-evaluate Table 3 FID after replacing the training targets with the independent reconstructions; if FID rises >30% or ranking vs. baselines collapses, the generation claim is reconstruction-limited.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that InterPetMoGen learns a meaningful p(P|H,A) and that FID 11.21 demonstrates realistic human-pet interaction modeling rests on the assumption that the dog 3D targets (Section 4.2) are accurate. That pipeline triangulates HRNet 2D keypoints after Mask-R-CNN human-mask filtering, applies RTS smoothing, then fits SMAL with shape parameters frozen from a single manually chosen canonical frame. No MPJPE, PCK, multi-view reprojection error, or contact-consistency numbers are reported for the dog reconstructions, especially under the close-range petting/command occlusions the paper itself flags as severe. Because the same reconstructed sequences are used both to train the PetVAE/GPT and to train the dog-identity classifier that supplies FID features (supplementary F), any systematic reconstruction bias is shared by training targets, model, and metric. The reported gains over Seq2Seq/DiT and the user-study scores (Table 6) therefore cannot yet be cleanly attributed to interaction modeling rather than to the model simply reproducing the reconstruction pipeline’s own regularities.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper introduces InterPet4D, claimed as the first large-scale multimodal 4D dataset of human–dog interactions (6.8M frames, 12 multi-view + egocentric RGB, audio, 2D/3D keypoints, SMPL-X/MANO and SMAL meshes) collected from 23 humans and 13 dogs of 11 breeds under a four-category protocol (petting, commanding, calling, free-form). It further proposes InterPetMoGen (IPMG): separate VQ-VAE tokenizers for body, hand, pet pose (PetVAE with bone-length auxiliary loss), and root translation, plus MERT audio features, fed to a prefix-LM GPT with modality-aware attention (bidirectional human conditioning, causal coarse-to-fine translation/pet decoding). On a fixed 80/20 per-dog split, IPMG reports FID_k/s of 11.21/12.96 and higher R-Precision/diversity than Seq2Seq-Transformer and DiT baselines (Table 3), with supporting modality/PetVAE ablations (Tables 4–5, Supp. E), qualitative sequences (Fig. 4), and a 12-participant Likert study with ANOVA (Table 6).","tokens_in":17682,"tokens_out":926,"duration_ms":9596,"significance":"If the reconstructions are sufficiently accurate, InterPet4D fills a genuine gap: existing interaction datasets are human–human or human–object, and animal datasets lack synchronized human motion and audio. A public multimodal 4D resource with a clear taxonomy and a reproducible generation baseline would enable cross-species interaction modeling for animation, robotics, and behavioral analysis. Strengths include the multi-sensor capture design (lower cameras for pets, egocentric glasses), the hierarchical MAA design that matches the interaction structure, consistent gains across FID, R-Precision, diversity, and a small but statistically tested user study, and the planned Hugging Face release. The work is therefore of clear community value provided reconstruction fidelity is demonstrated.","major_comments":[{"comment":"Section 4.2 (and the central claim that FID 11.21 demonstrates realistic interaction modeling): dog 3D targets are obtained by multi-view HRNet triangulation after Mask-R-CNN human-mask filtering, RTS smoothing, and SMAL fitting with shape parameters frozen from a single manually chosen canonical frame. No MPJPE, PCK, multi-view reprojection error, or contact-consistency numbers are reported, despite the paper’s own emphasis on severe close-range occlusion. Because the same sequences train PetVAE/GPT and the dog-identity classifier used for FID features (Supp. F), systematic reconstruction bias is shared by targets, model, and metric. Quantitative reconstruction validation (at least multi-view reprojection error stratified by interaction category, and preferably a small marker-based or multi-view consistency check) is required before the generation numbers can be attributed to interactio","section":null},{"comment":"Table 3 / Supp. F: the FID feature extractor is a 12-class dog-identity ResNet trained on the same reconstructed motion data. No external validation (e.g., correlation with MPJPE, velocity error, or human preference) is provided, so the large reported FID reductions (47% vs Seq2Seq, 83% vs DiT) remain only partially interpretable. Either validate the feature space against independent motion quality measures or report additional metrics (e.g., bone-length error, foot-skate, command-response latency) that do not rely on this classifier.","section":null},{"comment":"Section 6.1 and Table 1: total training motion after 30 fps downsampling and 10 s windowing is only ~87.5 minutes, with 200 train / 40 val clips and all dogs appearing in both splits. While the per-dog split avoids complete identity leakage, the absolute scale is small for a 28–30 M parameter autoregressive model and for claims of “large-scale” diversity across 11 breeds. The manuscript should discuss overfitting risk more carefully and, if possible, report leave-one-dog-out or leave-one-breed-out numbers to show that the learned p(P|H,A) generalizes beyond the reconstruction idiosyncrasies of the training animals.","section":null}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The punchline is the dataset. InterPet4D is the first public-scale multimodal 4D capture of natural human–dog interaction: 6.8 M frames, 12-view + egocentric video, audio commands, and reconstructed 3D for both agents across 23 people and 13 dogs of 11 breeds. Table 2 makes the gap clear; nothing prior jointly supplies synchronized multi-view, ego, audio, and dual 3D motion. That alone is useful for animal pose, animation, and socially aware robotics.\n\nThey did the hard engineering well. Capture protocol (petting/commanding/calling/free-form), lower-camera placement for hands and pets, Ray-Ban ego sync, ASR + LLM captions, and the full annotation stack (SMPL-X/MANO + SMAL) are carefully described. The generation piece, InterPetMoGen, is a straightforward MotionGPT-style system (PetVAE + prefix-LM + cross-attention + their modality-aware attention mask). Gains over Seq2Seq and DiT are large and consistent on FID, R-Precision, diversity, and a 12-person Likert study with ANOVA. Ablations on body/hand/audio and the bone-length term are clean.\n\nThe soft spot is real but proportionate. Dog 3D comes from multi-view HRNet triangulation after human-mask filtering, RTS smoothing, then SMAL with shape frozen from one canonical frame. No MPJPE, PCK, or multi-view reprojection numbers appear, especially under the close-range occlusions they themselves flag. Because the same reconstructions train both the model and the dog-identity classifier used for FID, some of the reported “realism” could be the model learning the reconstruction regularities. All dogs appear in both train and val, and full data/code release is still pending. These are standard first-dataset caveats, not load-bearing circularity.\n\nThis is for people who need animal interaction data or cross-species motion generation. The math and citations are solid; the free parameters are ordinary. I would bring it to reading group, cite the dataset, and send it to peer review. A referee can push for reconstruction metrics and public release without killing the contribution.","headline":"First real multimodal 4D human–dog interaction dataset at multi-million-frame scale; generation model is competent but secondary, and dog reconstruction quality remains unquantified.","tokens_in":18353,"tokens_out":561,"would_cite":true,"duration_ms":12464,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"The first large multimodal 4D human–dog dataset enables models that generate realistic pet motion from human gestures and voice.","keywords":["human-pet interaction","4D dataset","dog motion generation","multimodal capture","MotionGPT","PetVAE","SMAL","gesture-conditioned generation"],"falsifier":"A controlled hold-out of interaction sequences whose 3D dog reconstructions are independently verified by multi-view marker or high-precision optical systems, then checking whether InterPetMoGen’s FID and user-study scores remain superior to the same baselines when trained only on the verified subset.","tokens_in":18288,"feed_emoji":"🐕","tokens_out":622,"duration_ms":7419,"temperature":0.7,"pith_summary":"Human–pet interaction has lagged behind human–human and human–object work because no large, synchronized 3D dataset existed. This paper fills that gap with InterPet4D: 6.8 million frames of 13 dogs of 11 breeds interacting with 23 people under a controlled multi-view and egocentric capture setup, with full annotations for video, audio, 2D/3D keypoints, and parametric meshes. On top of the data the authors introduce InterPetMoGen, a MotionGPT-style model that tokenizes human body, hand, and audio signals and autoregressively generates dog motion. The model reaches an FID of 11.21 and clearly beats Seq2Seq and diffusion baselines, showing that the dataset is large and rich enough to learn realistic cross-species responses.","feed_headline":"First large 4D human–dog dataset drives realistic pet motion","feed_subtitle":"6.8 M frames and InterPetMoGen beat Seq2Seq and diffusion baselines at FID 11.21","key_machinery":"InterPetMoGen (IPMG): a PetVAE tokenizer that discretizes dog pose (with bone-length consistency) plus a modality-aware attention (MAA) prefix-LM that conditions autoregressive pet-motion tokens on bidirectional human body/hand tokens and cross-attended audio.","core_discovery":"InterPet4D is the first large-scale multimodal 4D dataset of natural human–dog interactions (6.8 M frames, 13 dogs, 23 humans), and the InterPetMoGen model trained on it can generate plausible 3D dog motion conditioned on human body/hand gestures and audio, achieving FID 11.21 and outperforming Seq2Seq and DiT baselines.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["InterPet4D: first multimodal 4D dog–human dataset at 6.8 M frames","6.8 M frames of human–dog tasks power InterPetMoGen’s FID 11.21","Natural dog–human 4D capture beats Seq2Seq and DiT at FID 11.21","InterPetMoGen generates dog motion from human cues on InterPet4D","13 dogs, 23 people: multimodal 4D set for pet motion generation"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The multi-view keypoint triangulation and SMAL fitting pipeline, which freezes dog shape from one frame and masks out human-occluded keypoints, must produce accurate enough 3D dog motion that the learned responses are real rather than reconstruction artifacts.","fun_headline_variants_meta":{"raw":{"variants":["InterPet4D: first multimodal 4D dog–human dataset at 6.8 M frames","6.8 M frames of human–dog tasks power InterPetMoGen’s FID 11.21","Natural dog–human 4D capture beats Seq2Seq and DiT at FID 11.21","InterPetMoGen generates dog motion from human cues on InterPet4D","13 dogs, 23 people: multimodal 4D set for pet motion generation"]},"model":"grok-4.5","effort":"low","cost_usd":0.001792,"raw_usage":{"total_tokens":844,"prompt_tokens":722,"num_sources_used":0,"completion_tokens":122,"cost_in_usd_ticks":17920000,"prompt_tokens_details":{"text_tokens":722,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":0,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":722,"tokens_out":122,"duration_ms":1384,"temperature":1.0,"reasoning_tokens":0,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T12:53:39.978941+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"A controlled hold-out of interaction sequences whose 3D dog reconstructions are independently verified by multi-view marker or high-precision optical systems, then checking whether InterPetMoGen’s FID and user-study scores remain superior to the same baselines when trained only on the verified subset.","supporting_citations":[],"review_version":1}