Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

SIMS: Simulating Stylized Human-Scene Interactions with Retrieval-Augmented Script Generation

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Physics-based characters can act out long, stylized stories from a text theme

desk verdict A solid system paper with a useful new dataset, but the text-to-style control claim lacks direct evidence and the abstract overreaches. read the letter →

arxiv 2411.19921 v2 pith:V7XZNP5M submitted 2024-11-29 cs.CV cs.AIcs.CLcs.GR

Wenjia Wang , Liang Pan , Zhiyang Dou , Jidong Mei , Zhouyingcheng Liao , Yuke Lou , Yifan Wu , Lei Yang
show 2 more authors
Jingbo Wang Taku Komura
This is my paper · ORCID
classification cs.CVcs.AIcs.CLcs.GR
keywords human-sceneinteractionphysics-basedcharactercontrolretrieval-augmentedgenerationtext-conditionedpolicystylediversitylong-termmotionplanningfinitestatemachinecapturedataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a simulated human character can follow a long, everyday narrative—walking to a sofa, sitting sadly, carrying a box, getting up—while moving in a believable style and staying physically plausible in a cluttered 3D room. It proposes a two-level system: a language model that plans the story as a sequence of short, retrievable scripts, and a physics-based controller that executes each scripted action with a style drawn from text descriptions. The claim is that combining retrieval-augmented script planning with a multi-condition control policy solves a problem earlier long-horizon interaction methods left open, namely producing both diverse style and stable physical contact in one pipeline. If the claim holds, character animation and embodied simulation can move from short, neutral motion clips toward long, emotionally expressive performances without hand-authored choreography.

What carries the argument

The load-bearing mechanism is the pairing of Retrieval-Augmented Script Generation (RASG) with a multi-condition physics control policy. The short-script database converts open-ended narratives into executable keyframe tuples, each specifying skill, target object, caption, and style label; the retrieval step keeps the story grounded in actions the controller can actually perform. At the low level, the policy is conditioned on proprioception, an egocentric heightmap, a task goal, and a style embedding, and is trained with a text-conditioned adversarial reward, so a desired style is injected through language rather than through dense reference motion.

What would settle it

Run the trained Sit or Carry policy many times under a fixed scene and fixed task goal with two text conditions that are semantically opposite (for example, 'sad' versus 'excited') and measure the distribution of generated joint angles. If the two distributions overlap almost completely, the claim that text embeddings provide style control collapses; diversity alone would not separate conditions.

Watch

Extended reading notes

Core claim

SIMS is a hierarchical framework that separates what to do from how to do it. High-level planning is handled by retrieval-augmented script generation: a library of short scripts, each a few keyframes with skill, object, caption, and emotion label, is built once; given a user theme, the planner retrieves the most semantically similar short scripts and asks a language model to concatenate them into a coherent long script. Low-level execution is handled by a set of physics-based policies that observe an egocentric heightmap of the scene, a task goal, and a language embedding of the desired style, and are trained with a text-conditioned adversarial discriminator plus task rewards. A finite state machine switches policies at keyframe boundaries. The central discovery is that this division makes stylized, long-term human-scene interaction trainable and controllable: style comes through text, scene geometry comes through the heightmap, and physical plausibility comes through reinforcement learning in simulation.

Load-bearing premise

The paper assumes that a text-conditioned adversarial style reward trained with aligned text and motion embeddings reliably turns a language description into the intended motion style; if that mapping is weak, the reported diversity is just unlabeled variety rather than controllable style.

Editorial extensions

If this is right

  • A user can type a theme such as "a person gets fired and drinks at home" and receive a physically simulated animation that chains walking, carrying, sitting, and lying with matching emotional styles.
  • Because styles enter through language embeddings, new styles can be added by collecting or captioning more motion data and retraining the relevant policy, without rewriting the planner.
  • The egocentric heightmap lets policies trained on one furniture set generalize to unseen objects, so the same controllers can be dropped into new room layouts.
  • The ViconStyle dataset plus the short-script database gives later work a common benchmark for stylized long-horizon interaction, not just single-action generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A sharper test of the style mechanism than the paper reports would compare repeated rollouts under the same text embedding: if within-condition diversity is nearly as high as across-condition diversity, then much of the reported diversity could come from task randomness rather than from style control.
  • The same retrieval-augmented planning structure could be reused for other embodied agents, such as robot arms or virtual humans with finger articulation, by swapping the skill set and the low-level policy, since the planner only reasons about keyframe tuples.
  • The short-script database could become a lightweight planning benchmark independent of any physics simulator, allowing language planners to be evaluated on narrative coherence and executability before motion control is trained.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents SIMS, a hierarchical framework for generating long-term stylized human-scene interactions in a physics simulator. A high-level LLM-based planner with retrieval-augmented generation (RASG) creates coherent scripts with keyframes; a low-level multi-condition control policy executes these keyframes, using CLIP text embeddings as style conditions, egocentric heightmaps for scene awareness, and task-specific goal conditions. The authors contribute a short-script database, a new ViconStyle motion capture dataset, and quantitative evaluations reporting improvements in FID, APD, success rate, and contact error over several baselines.

Significance. If the claims hold, SIMS advances physics-based character animation by integrating language-driven planning with controllable low-level policies and by releasing a reusable stylized motion dataset. The system is end-to-end automatic, and the reported FID/APD improvements over UniHSI and InterPhys are substantial for several skills. The paper also demonstrates scalability to new skills and generalization to unseen objects. However, the central claim of text-conditioned style control is not directly verified: the current evidence is partially consistent with the alternative that language conditioning mostly increases stochastic diversity rather than aligning with the specified style. The dataset release is a valuable community contribution.

major comments (4)
  1. [Sec. 3.3, Sec. 4.3.2, Tables 4 and 11] The paper's core claim is that the language embedding z conveys stylistic cues to the low-level policy, but no experiment directly verifies that a given text condition produces the corresponding motion style. Table 4's APD is averaged over randomly sampled text conditions, so it measures diversity, not text-style alignment. The ablation in Table 11 shows that removing text changes APD for Sit and Lie by amounts within the reported error bars (16.52±0.47 vs. 16.29±0.22 for Sit; 16.99±1.28 vs. 16.59±0.28 for Lie), while only Carry shows a clear drop (14.92±0.23 vs. 12.41±0.19). The qualitative examples in Fig. 4 are suggestive but not quantified. Please add a direct per-style evaluation: for instance, train a style classifier on held-out stylized motion, report per-style FID against style-specific references, or run a forced-choice user study in which raters select which text label matches a generated motion clip with the script/story context removed. Without such evidence, the 'stylized' component of the central contribution is not established.
  2. [Abstract and Table 3] The abstract states that SIMS 'significantly outperforms previous methods,' but Table 3 shows that SIMS's Reach success rate (95.2) is below UniHSI's (97.5) and its Carry contact error (0.099) is worse than InterPhys's (0.08). Although the text in Sec. 4.3.1 acknowledges that these two results are slightly lower, the abstract and the contribution list present an unqualified dominance claim. Please qualify the claim to reflect the subset of metrics and skills where the advantage holds, or extend the experiments/training data to close these gaps.
  3. [Sec. 4.4.4, Table 11] The ablation study that supports the text-conditioning claim is under-specified. The row labeled 'SIMS(ours)' reports success rates (96.9 for Sit, 89.7 for Lie) that match the dataset-augmented results in Table 10, while the 'w/o text' row may have been trained on a different data mixture; the paper does not state the training data for each row. Since additional data changes both success rate and APD (Tables 8-10), the difference between 'w/o text' and 'SIMS(ours)' may conflate data volume with text conditioning. Please report the exact training set for each ablation row and, ideally, add error bars over multiple seeds for success rates as well as APD.
  4. [Sec. 4.2, Table 4] The FID evaluation in Table 4 uses SAMP as the reference distribution for Sit and Lie, but the generated motions for UniHSI are conditioned on the ScenePlan chain of contacts, not on SAMP-style motions; the comparison may be biased because SIMS is trained on SAMP while UniHSI is not. Please specify the reference motion set used for each method and skill, and consider reporting FID against a common held-out motion set.
minor comments (4)
  1. [Sec. 1, Introduction] The word 'Smultating' appears to be a typo for 'Simulating' in the introduction.
  2. [Sec. 6, heading] The heading 'Furture Work' should be 'Future Work'.
  3. [Table 1 caption and layout] The caption contains 'Comparision' and 'Physical-Plausibe' typos; the table layout appears misaligned, with the skill columns (Walk, Sit, Lie, GetUp, Reach, Idle, Carry) running together with the property columns, making the checkmarks difficult to interpret.
  4. [Sec. 4.4.3, Tables 8-10] The dataset abbreviations in the captions are inconsistent: for example, '100S' is defined as '100Style' but the table only reports two rows, and the text does not state whether 'A+100S' includes all of AMASS or a subset. Please clarify the exact data composition in each ablation row.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the derivation chain is self-contained and evaluated against external data and benchmarks.

full rationale

The paper's derivation chain is not circular. The high-level planner (RASG) retrieves pre-generated short scripts using fixed CLIP embeddings and GPT-4, neither of which is fitted to the target output; the low-level policy is trained with RL plus an AMP-style text-conditioned discriminator on external motion data (SAMP, AMASS, 100Style, COUCH, and the newly captured ViconStyle), and the evaluation metrics (FID, APD, success rate, contact error, user study) are computed on generated motions against real reference distributions. The short-script database is LLM-generated, but the LLM is not given the target motions or metrics to optimize, so this is not a fitted input renamed as a prediction. Self-citations (e.g., InterScene's FSM, UniHSI's initialization and pointcloud handling, C·ASE's conditional discriminator) are implementation choices citing prior work and are not used to justify the central claim by an unverified uniqueness or authority argument. The reader's concern that style fidelity is not directly measured—APD mainly measures diversity and the user study rates emotional resonance—is an evaluation gap, but it does not make any claimed result equivalent to its inputs by construction. No equation or step reduces a predicted quantity to a fitted parameter or to a definition, so the paper warrants a zero circularity score.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The system relies on standard RL and generative modeling assumptions. The key domain assumptions are that CLIP text embeddings capture motion style, that a 12x12 heightmap suffices for scene generalization, that LLM-generated scripts are executable, and that the FSM switching rules produce coherent sequences. No new physical entities are introduced.

free parameters (1)
  • Reward coefficients λ_style and λ_task
    Chosen by hand; values not reported. They control the trade-off between style and task success and could influence the diversity vs. success results.
assumptions (4)
  • domain assumption CLIP text embeddings capture motion style sufficiently for conditioning.
    Sec 3.3 uses CLIP embeddings as the style condition for the policy and the discriminator; if this mapping is weak, style control fails.
  • domain assumption The egocentric 12x12 heightmap is a sufficient scene representation for generalization.
    Sec 3.3 uses this heightmap as the only geometric observation; the PartNet results suggest it works, but it is still a simplifying assumption.
  • domain assumption LLM-generated short scripts are executable by the low-level policies.
    Sec 3.2 assumes the retrieved short scripts can be parsed into keyframes the policies can execute; no verification of executability is provided after generation.
  • domain assumption FSM rule-based transitions (overlap threshold) yield coherent long-term execution.
    Sec 3.3 relies on a simple threshold rule to switch skills; the paper does not test failure modes of this scheduling.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SIMS: Simulating Stylized Human-Scene Interactions with Retrieval-Augmented Script Generation." pith.science (2026). https://pith.science/paper/V7XZNP5M

@misc{pith2026241119921,
  author       = {Pith},
  title        = {Pith review of: SIMS: Simulating Stylized Human-Scene Interactions with Retrieval-Augmented Script Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V7XZNP5M}},
  note         = {Machine review of arXiv:2411.19921}
}
read the original abstract

Simulating stylized human-scene interactions (HSI) in physical environments is a challenging yet fascinating task. Prior works emphasize long-term execution but fall short in achieving both diverse style and physical plausibility. To tackle this challenge, we introduce a novel hierarchical framework named SIMS that seamlessly bridges highlevel script-driven intent with a low-level control policy, enabling more expressive and diverse human-scene interactions. Specifically, we employ Large Language Models with Retrieval-Augmented Generation (RAG) to generate coherent and diverse long-form scripts, providing a rich foundation for motion planning. A versatile multicondition physics-based control policy is also developed, which leverages text embeddings from the generated scripts to encode stylistic cues, simultaneously perceiving environmental geometries and accomplishing task goals. By integrating the retrieval-augmented script generation with the multi-condition controller, our approach provides a unified solution for generating stylized HSI motions. We further introduce a comprehensive planning dataset produced by RAG and a stylized motion dataset featuring diverse locomotions and interactions. Extensive experiments demonstrate SIMS's effectiveness in executing various tasks and generalizing across different scenarios, significantly outperforming previous methods.

Figures

Figures reproduced from arXiv: 2411.19921 by the authors.

Figure 1
Figure 1. SIMS enables physically simulated characters to perform diverse skills within complex 3D scenes given long-term daily narratives [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. (a) Our main pipeline. We prompt LLMs to generate new short scripts following their emotion and interaction logic. The retrieval [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Long-term scripts with detailed keyframes and vivid final stories in two complex 3D scenes generated by our complete system. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Qualitative results for skills with different text condi [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: ViconStyle demos. Locomotion StaticHSI SIMSTask DynamicHSI Walk Idle Sit Lie GetUp Reach New Skill ? Short Script Database relaxed sad happy angry Step1: Train policy Step2: Add Script Data Motion Caption Style Objects tired stressed drunk others Step3: Inference, run …
Figure 7
Figure 7. Figure 7: Our re-implemented MotionClip. in [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 6
Figure 6. Figure 6: Scalability on new skills. pre-trained CLIP text encoder [31]. Given a motion clip mˆ = (qˆ1, . . . , qˆn), a motion encoder z = Encm(mˆ ) maps the motion to an embedding z. The embedding is normal￾ized to lie on a unit sphere ∥z∥ = 1. We set the embed￾ding size z to 6…
Figure 8
Figure 8. Figure 8: The motion capture environment of Vicon optical motion [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MoRAE: Flow-Friendly Self-Supervised Latents for Text-to-Motion Generation

    cs.CV 2026-07 conditional novelty 7.0 of 10

    Distilling frozen Motion-JEPA features into a compact 32-D latent whose geometry is coupled to the decoder lets a standard non-autoregressive flow-matching DiT reach state-of-the-art text-to-motion quality on HumanML3...

  2. Absolute Coordinates Make Motion Generation Easy

    cs.CV 2025-05 conditional novelty 7.0 of 10

    Using absolute 3D joint coordinates with a plain Transformer and velocity-prediction diffusion outperforms the standard local-relative motion representation, improving fidelity and enabling direct control.

  3. CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objects

    cs.GR 2025-05 conditional novelty 6.0 of 10

    CoDA generates coordinated whole-body articulated-object manipulation by optimizing the noise of three decoupled diffusion models, guided by BPS-based end-effector and object trajectories.

Reference graph

Works this paper leans on

60 extracted references · 46 canonical work pages · cited by 3 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Circle: Capture in rich contextual environ- ments

    Joao Pedro Ara ´ujo, Jiaman Li, Karthik Vetrivel, Rishi Agar- wal, Jiajun Wu, Deepak Gopinath, Alexander William Clegg, and Karen Liu. Circle: Capture in rich contextual environ- ments. In CVPR, 2023. 6

    work page 2023

  3. [3]

    Laserhuman: language-guided scene-aware hu- man motion generation in free environment

    Peishan Cong, Ziyi Wang, Zhiyang Dou, Yiming Ren, Wei Yin, Kai Cheng, Yujing Sun, Xiaoxiao Long, Xinge Zhu, and Yuexin Ma. Laserhuman: language-guided scene-aware hu- man motion generation in free environment. arXiv preprint arXiv:2403.13307, 2024. 3

    arXiv 2024

  4. [4]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. 2

    arXiv 2018

  5. [5]

    C· ase: Learning conditional adversar- ial skill embeddings for physics-based characters

    Zhiyang Dou, Xuelin Chen, Qingnan Fan, Taku Komura, and Wenping Wang. C· ase: Learning conditional adversar- ial skill embeddings for physics-based characters. In SIG- GRAPH 2023, 2023. 3, 5, 6

    work page 2023

  6. [6]

    Tore: Token reduction for efficient human mesh re- covery with transformer

    Zhiyang Dou, Qingxuan Wu, Cheng Lin, Zeyu Cao, Qiangqiang Wu, Weilin Wan, Taku Komura, and Wenping Wang. Tore: Token reduction for efficient human mesh re- covery with transformer. In ICCV, 2023. 3

    work page 2023

  7. [7]

    3d-front: 3d furnished rooms with layouts and semantics

    Huan Fu, Bowen Cai, Lin Gao, Ling-Xiao Zhang, Jiaming Wang, Cao Li, Qixun Zeng, Chengyue Sun, Rongfei Jia, Bin- qiang Zhao, et al. 3d-front: 3d furnished rooms with layouts and semantics. In ICCV, 2021. 5, 6, 8

    work page 2021

  8. [8]

    Coohoi: Learning cooperative human-object interac- tion with manipulated object dynamics

    Jiawei Gao, Ziqin Wang, Zeqi Xiao, Jingbo Wang, Tai Wang, Jinkun Cao, Xiaolin Hu, Si Liu, Jifeng Dai, and Jiangmiao Pang. Coohoi: Learning cooperative human-object interac- tion with manipulated object dynamics. Advances in Neural Information Processing Systems, 37, 2024. 5

    work page 2024

Show all 60 references
  1. [9]

    3d human reconstruction in the wild with synthetic data using generative models

    Yongtao Ge, Wenjia Wang, Yongfan Chen, Hao Chen, and Chunhua Shen. 3d human reconstruction in the wild with synthetic data using generative models. arXiv preprint arXiv:2403.11111, 2024. 3

    2024

  2. [10]

    Nima Ghorbani and Michael J. Black. Soma: Solving optical marker-based mocap automatically. In ICCV, 2021. 3

    2021

  3. [11]

    Generating diverse and natural 3d human motions from text

    Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In CVPR, 2022. 3

    2022

  4. [12]

    Stochastic scene-aware motion prediction

    Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael J Black. Stochastic scene-aware motion prediction. In ICCV, 2021. 1, 2, 3, 6, 7, 8

    2021

  5. [13]

    Synthesizing physi- cal character-scene interactions

    Mohamed Hassan, Yunrong Guo, Tingwu Wang, Michael Black, Sanja Fidler, and Xue Bin Peng. Synthesizing physi- cal character-scene interactions. In SIGGRAPH 2023, 2023. 2, 3, 5, 6, 7

    2023

  6. [14]

    Mod- skill: Physical character skill modularization

    Yiming Huang, Zhiyang Dou, and Lingjie Liu. Mod- skill: Physical character skill modularization. arXiv preprint arXiv:2502.14140, 2025. 3

    2025 arXiv

  7. [15]

    Motiongpt: Human motion as a foreign language

    Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign language. NeuraIPS, 36, 2023. 3

    2023

  8. [16]

    Scaling up dynamic human-scene interaction mod- eling

    Nan Jiang, Zhiyuan Zhang, Hongjie Li, Xiaoxuan Ma, Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, and Siyuan Huang. Scaling up dynamic human-scene interaction mod- eling. In CVPR, 2024. 3

    2024

  9. [17]

    Padl: Language-directed physics-based character con- trol

    Jordan Juravsky, Yunrong Guo, Sanja Fidler, and Xue Bin Peng. Padl: Language-directed physics-based character con- trol. In SIGGRAPH Asia 2022 Conference Papers, 2022. 2, 1

    2022

  10. [18]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K¨uttler, Mike Lewis, Wen-tau Yih, Tim Rockt ¨aschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. NeuraIPS, 33, 2020. 2, 4

    2020

  11. [19]

    Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. Smpl: a skinned multi- person linear model. TOG, 34(6), 2015. 3

    2015

  12. [20]

    Scamo: Exploring the scaling law in autoregressive motion generation model

    Shunlin Lu, Jingbo Wang, Zeyu Lu, Ling-Hao Chen, Wenxun Dai, Junting Dong, Zhiyang Dou, Bo Dai, and Ruimao Zhang. Scamo: Exploring the scaling law in autoregressive motion generation model. arXiv preprint arXiv:2412.14559, 2024. 3

    2024 arXiv

  13. [21]

    Perpetual humanoid control for real-time simulated avatars

    Zhengyi Luo, Jinkun Cao, Kris Kitani, Weipeng Xu, et al. Perpetual humanoid control for real-time simulated avatars. In ICCV, 2023. 5

    2023

  14. [22]

    Amass: Archive of motion capture as surface shapes

    Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Ger- ard Pons-Moll, and Michael J Black. Amass: Archive of motion capture as surface shapes. In ICCV, 2019. 6

    2019

  15. [23]

    Isaac gym: High performance gpu-based physics simulation for robot learning

    Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2108.10470, 2021. 2

    2021 arXiv

  16. [24]

    Real-time style modelling of human locomotion via feature-wise trans- formations and local motion phases

    Ian Mason, Sebastian Starke, and Taku Komura. Real-time style modelling of human locomotion via feature-wise trans- formations and local motion phases. Proceedings of the ACM on Computer Graphics and Interactive Techniques , 5 (1), 2022. 6

    2022

  17. [25]

    Partnet: A large- scale benchmark for fine-grained and hierarchical part-level 3d object understanding

    Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas J Guibas, and Hao Su. Partnet: A large- scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In CVPR, 2019. 8

    2019

  18. [26]

    Synthesizing phys- ically plausible human motions in 3d scenes

    Liang Pan, Jingbo Wang, Buzhen Huang, Junyu Zhang, Hao- fan Wang, Xu Tang, and Yangang Wang. Synthesizing phys- ically plausible human motions in 3d scenes. In 3DV, 2024. 2, 3, 5, 6

    2024

  19. [28]

    Deepmimic: Example-guided deep reinforce- ment learning of physics-based character skills

    Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. Deepmimic: Example-guided deep reinforce- ment learning of physics-based character skills. TOG, 37(4),

  20. [29]

    Amp: Adversarial motion priors for styl- ized physics-based character control

    Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adversarial motion priors for styl- ized physics-based character control. TOG, 2021. 2, 4, 6 10

    2021

  21. [30]

    Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters

    Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters. TOG, 41 (4), 2022. 3

    2022

  22. [31]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, 2021. 2, 3, 5

    2021

  23. [32]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019. 6, 8

    1908 arXiv

  24. [33]

    Neural state machine for character-scene interactions

    Sebastian Starke, He Zhang, Taku Komura, and Jun Saito. Neural state machine for character-scene interactions. TOG, 38(6), 2019. 1, 2, 3, 5

    2019

  25. [34]

    Prompt, plan, perform: Llm- based humanoid control via quantized imitation learning

    Jingkai Sun, Qiang Zhang, Yiqun Duan, Xiaoyang Jiang, Chong Cheng, and Renjing Xu. Prompt, plan, perform: Llm- based humanoid control via quantized imitation learning. In ICRA, 2024. 2

    2024

  26. [35]

    Aios: All-in-one-stage expressive human pose and shape estimation

    Qingping Sun, Yanjun Wang, Ailing Zeng, Wanqi Yin, Chen Wei, Wenjia Wang, Haiyi Mei, Chi-Sing Leung, Ziwei Liu, Lei Yang, et al. Aios: All-in-one-stage expressive human pose and shape estimation. In CVPR, 2024. 3

    2024

  27. [36]

    Calm: Conditional adversar- ial latent models for directable virtual characters

    Chen Tessler, Yoni Kasten, Yunrong Guo, Shie Mannor, Gal Chechik, and Xue Bin Peng. Calm: Conditional adversar- ial latent models for directable virtual characters. In SIG- GRAPH 2023, 2023. 5

    2023

  28. [37]

    Maskedmimic: Unified physics-based char- acter control through masked motion inpainting

    Chen Tessler, Yunrong Guo, Ofir Nabati, Gal Chechik, and Xue Bin Peng. Maskedmimic: Unified physics-based char- acter control through masked motion inpainting. TOG, 43 (6), 2024. 5

    2024

  29. [38]

    Motionclip: Exposing human motion generation to clip space

    Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. Motionclip: Exposing human motion generation to clip space. In ECCV, 2022. 3, 1, 2

    2022

  30. [39]

    Human motion diffu- sion model

    Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffu- sion model. In ICLR, 2023. 6

    2023

  31. [40]

    Tlcontrol: Trajectory and language control for human motion synthesis

    Weilin Wan, Zhiyang Dou, Taku Komura, Wenping Wang, Dinesh Jayaraman, and Lingjie Liu. Tlcontrol: Trajectory and language control for human motion synthesis. arXiv preprint arXiv:2311.17135, 2023. 3

    2023 arXiv

  32. [41]

    Towards diverse and natural scene-aware 3d human motion synthesis

    Jingbo Wang, Yu Rong, Jingyuan Liu, Sijie Yan, Dahua Lin, and Bo Dai. Towards diverse and natural scene-aware 3d human motion synthesis. In CVPR, 2022. 1, 3, 5, 6

    2022

  33. [42]

    Disentangled clothed avatar generation from text descriptions

    Jionghao Wang, Yuan Liu, Zhiyang Dou, Zhengming Yu, Yongqing Liang, Cheng Lin, Xin Li, Wenping Wang, Rong Xie, and Li Song. Disentangled clothed avatar generation from text descriptions. arXiv preprint arXiv:2312.05295 ,

    arXiv

  34. [43]

    Zolly: Zoom focal length correctly for perspective- distorted human mesh reconstruction

    Wenjia Wang, Yongtao Ge, Haiyi Mei, Zhongang Cai, Qing- ping Sun, Yanjun Wang, Chunhua Shen, Lei Yang, and Taku Komura. Zolly: Zoom focal length correctly for perspective- distorted human mesh reconstruction. In ICCV, 2023. 3

    2023

  35. [44]

    Humanise: Language-conditioned hu- man motion generation in 3d scenes

    Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. Humanise: Language-conditioned hu- man motion generation in 3d scenes. NeuraIPS, 35, 2022. 1, 2, 3

    2022

  36. [45]

    Move as you say interact as you can: Language-guided human motion generation with scene af- fordance

    Zan Wang, Yixin Chen, Baoxiong Jia, Puhao Li, Jinlu Zhang, Jingze Zhang, Tengyu Liu, Yixin Zhu, Wei Liang, and Siyuan Huang. Move as you say interact as you can: Language-guided human motion generation with scene af- fordance. In CVPR, 2024. 2, 3

    2024

  37. [46]

    Physics-based character controllers using conditional vaes

    Jungdam Won, Deepak Gopinath, and Jessica Hodgins. Physics-based character controllers using conditional vaes. TOG, 41(4), 2022. 5

    2022

  38. [47]

    Unified human-scene interaction via prompted chain-of-contacts

    Zeqi Xiao, Tai Wang, Jingbo Wang, Jinkun Cao, Wenwei Zhang, Bo Dai, Dahua Lin, and Jiangmiao Pang. Unified human-scene interaction via prompted chain-of-contacts. In ICLR, 2024. 2, 3, 4, 5, 6, 7, 1

    2024

  39. [48]

    Hierarchical planning and control for box loco-manipulation

    Zhaoming Xie, Jonathan Tseng, Sebastian Starke, Michiel van de Panne, and C Karen Liu. Hierarchical planning and control for box loco-manipulation. Proceedings of the ACM on Computer Graphics and Interactive Techniques , 6 (3), 2023. 2

    2023

  40. [49]

    Intermimic: Towards universal whole-body control for physics-based human-object interactions

    Sirui Xu, Hung Yu Ling, Yu-Xiong Wang, and Liang-Yan Gui. Intermimic: Towards universal whole-body control for physics-based human-object interactions. arXiv preprint arXiv:2502.20390, 2025. 5

    2025

  41. [50]

    Black, Xue Bin Peng, and Davis Rempe

    Hongwei Yi, Justus Thies, Michael J. Black, Xue Bin Peng, and Davis Rempe. Generating human interaction motions in scenes with text control. ECCV, 2024. 2, 3

    2024

  42. [51]

    Physdiff: Physics-guided human motion diffusion model

    Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, and Jan Kautz. Physdiff: Physics-guided human motion diffusion model. In ICCV, 2023. 2

    2023

  43. [52]

    Motiondif- fuse: Text-driven human motion generation with diffusion model

    Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondif- fuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208.15001, 2022. 3

    2022 arXiv

  44. [53]

    Roam: Robust and object-aware motion genera- tion using neural pose descriptors

    Wanyue Zhang, Rishabh Dabral, Thomas Leimk ¨uhler, Vladislav Golyanik, Marc Habermann, and Christian Theobalt. Roam: Robust and object-aware motion genera- tion using neural pose descriptors. In 3DV, 2024. 3

    2024

  45. [54]

    Couch: Towards controllable human-chair interactions

    Xiaohan Zhang, Bharat Lal Bhatnagar, Sebastian Starke, Vladimir Guzov, and Gerard Pons-Moll. Couch: Towards controllable human-chair interactions. In ECCV, 2022. 1, 3, 6, 8

    2022

  46. [55]

    Synthesizing diverse human motions in 3d indoor scenes

    Kaifeng Zhao, Yan Zhang, Shaofei Wang, Thabo Beeler, and Siyu Tang. Synthesizing diverse human motions in 3d indoor scenes. In ICCV, 2023. 1

    2023

  47. [56]

    Emdm: Efficient motion diffusion model for fast and high-quality motion generation

    Wenyang Zhou, Zhiyang Dou, Zeyu Cao, Zhouyingcheng Liao, Jingbo Wang, Wenjia Wang, Yuan Liu, Taku Komura, Wenping Wang, and Lingjie Liu. Emdm: Efficient motion diffusion model for fast and high-quality motion generation. In ECCV, 2025. 3 11 SIMS: Simulating Stylized Human-Scen...

    2025

  48. [57]

    • Loco Reward

    Reward Templates In this section, we introduce the reward functions in 3 parts: locomotion (Loco), human-scene interaction (HSI), and dy- namic object interaction (DOI). • Loco Reward. The locomotion reward is defined in Equation 1. The overall reward comprises the far rf ar t...

  49. [58]

    Sitting loosely with legs crossed

    Re-implemented MotionCLIP To control the policy language constraints, we aim to con- struct an embedding space fed into the policy network, where the embedding aligns motion representation with their corresponding natural language descriptions. To do this, we follow [17, 38], ...

  50. [59]

    6, we show the easy scalability of our framework

    New Skill Scalability In Fig. 6, we show the easy scalability of our framework. When new skills of new styles come, we need to train the corresponding skill based on the 3 kinds of templates, and expand the scripts database following the instruction of Sec. 3.1

  51. [60]

    ViconStyle Dataset We propose a comprehensive motion dataset called Vicon- Style, in which well-labeled reconstructed motion clips with diverse styles and multiple skills are provided. 10.1. Capture Setting The motion clips are captured with Vicon, an optical motion capture sy...

  52. [61]

    13 for all the emo- tions/styles we use

    Short Script Examples We show some vivid examples in Tab. 13 for all the emo- tions/styles we use. Please check the skills, style label, ob- ject type, and captions, which are essential for FSM con- trol. 3 Summary: The character enjoys arelaxedafternoon in the living room. sk...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.