Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

FineMotion: A Dataset and Benchmark with both Spatial and Temporal Annotation for Fine-grained Motion Generation and Editing

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Adding temporally aligned body-part text to coarse captions improves text-to-motion generation and enables zero-shot text-based motion editing.

desk verdict Useful dataset and a genuinely interesting annotation pipeline, but the headline +15.3% gain is an apples-to-oranges comparison that needs a controlled re-run. read the letter →

arxiv 2507.19850 v1 pith:BSV3EF2M submitted 2025-07-26 cs.CV

Bizhu Wu , Jinheng Xie , Meidan Ding , Zhe Kong , Jianfeng Ren , Ruibin Bai , Rong Qu , Linlin Shen This is my paper · ORCID
classification cs.CV
keywords FineMotiondatasettext-to-motiongenerationfine-grainededitingbodypartmovementdescriptiontemporalalignmentmotion-languagezero-shot
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper builds a motion-language dataset in which every short 0.5-second snippet of a human motion sequence carries a sentence describing exactly what the body parts do during that interval, and a whole-sequence paragraph assembled from those sentences. The central aim is to show that this temporally anchored, body-part-level text, used together with the existing coarse caption, makes text-to-motion models generate motions closer to the ground truth: the adapted MDM model's Top-3 retrieval accuracy rises from 0.606 to 0.759, a 15.3-point gain, with similar improvements reported for two other generation architectures. The same structure supports a zero-shot editing pipeline in which a user edits or deletes a snippet description to change a specific body part at a specific time, or to insert and delete time intervals. The broader payoff, if the claim holds, is a scalable way to produce detailed motion annotations without manual timing annotation, since the descriptions are generated per fixed interval and then merged.

What carries the argument

The load-bearing object is the body-part movement snippet description (BPMSD): an imperative sentence generated for each 0.5-second snippet from the pose transition between the snippet's first and last frames. A correctional text-generation model trained to describe pose differences produces these sentences; human annotators corrected about five percent of them; and a large language model then merges them, in order, into whole-sequence body-part movement paragraphs (BPMPs). Adapted baseline models encode the coarse caption and the concatenated detailed text separately with a T5 text encoder and concatenate their pooled embeddings, which the authors show outperforms concatenating the raw texts. Temporal augmentation of training clips, obtained by cropping along the snippet timeline, is what lets the same model perform temporal edits such as inserting or deleting intervals.

What would settle it

Train the adapted MDM model on FineMotion using only automatically generated snippet descriptions, with the human-annotated five percent excluded, and compare Top-3 R-Precision on the test set against the coarse-only baseline; if the gain shrinks to zero or reverses, the claimed value of the scalable pipeline rests on the small human subset rather than on the automatic annotations.

Watch

Extended reading notes

Core claim

The central discovery is that fine-grained text can be made strictly aligned to motion by construction rather than by asking a language model to imagine details from a coarse caption. Segmenting a motion into fixed 0.5-second snippets and describing the transition from each snippet's start pose to its end pose yields sentences whose content is anchored to what actually happens at that time. When these snippet descriptions are fed to a text-to-motion model together with the coarse caption, the model learns to condition on both global semantics and local body-part and timing detail, and retrieval-based evaluation shows it selects motions closer to the ground truth. The paper further demonstrates that editing the snippet text and then re-generating the motion from the modified description yields spatial and temporal edits with no dedicated editing network and no additional training.

Load-bearing premise

The paper assumes that the automatically generated snippet descriptions, which make up about 95 percent of the dataset, accurately describe the motion in each 0.5-second interval, and this is supported only indirectly through a small human-annotation comparison and downstream task improvements.

Editorial extensions

If this is right

  • Fine-grained, temporally aligned text is a drop-in conditioning signal: three different generation architectures, a diffusion model, an autoregressive GPT-style model, and a masked generation model, all improve in R-Precision and FID when detailed snippet text is added to the coarse caption.
  • Editing becomes a text-editing operation: deleting or altering a snippet description changes only the corresponding body-part interval, while inserting time steps is handled by the temporally augmented model, without training an edit-specific network.
  • Because descriptions are generated per fixed interval and then merged, the annotation pipeline can be re-run on other motion corpora, giving scalable fine-grained annotation for new datasets.
  • The fine-grained condition reduces generated motion diversity: models trained on the detailed text score lower on diversity and multimodality metrics, a direct consequence of the stronger constraint.
  • Temporal editing is more accurate than spatial editing in the reported user study, a result the paper attributes to the temporally augmented training data used by the model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported gains could come substantially from the fixed 0.5-second segmentation and template-driven concatenation, which inject temporal structure even before considering the semantic content of the generated sentences; ablating with placeholder text per interval would separate those factors.
  • Because BPMSDs provide dense per-interval language supervision, the dataset could serve as a benchmark for motion-to-text generation and for directly measuring temporal alignment between language and motion, a direction the paper does not develop.
  • The zero-shot editing pipeline's fidelity is bounded by the initial text-to-motion synthesis, so a stronger evaluation would compare edited outputs against ground-truth edited motions or use automatic metrics rather than only user preference.
  • If the automatic description generator fails on rare poses or large pose changes, scaling to more diverse motion data would require retraining or augmenting that generator; the 0.5-second cap on snippet duration is a direct mitigation, and its necessity could be tested by measuring generator accuracy on longer snippets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces FineMotion, a dataset built on HumanML3D motions that augments coarse captions with fine-grained, body-part-level descriptions: over 442k snippet-level descriptions (BPMSD) generated automatically via PoseFix plus 21k human-annotated ones, and about 94k sequence-level paragraphs (BPMP) produced by prompting Gemini to stitch snippet descriptions into coherent text. The authors adapt MDM, T2M-GPT, and MoMask by replacing CLIP with T5-Base and concatenating coarse and detailed text embeddings, and report benchmark results including a headline +15.3% improvement in Top-3 R-Precision for MDM. They also propose a zero-shot fine-grained motion editing pipeline and evaluate it with a small user study.

Significance. If the annotation pipeline is reliable and the evaluation is not confounded, FineMotion would be a valuable and scalable resource for fine-grained text-to-motion, addressing a real gap in existing datasets such as HumanML3D and KIT-ML. The paper ships the dataset, code, and adapted baselines, and the idea of using pose-difference descriptions for snippets is novel and potentially useful. However, the central quantitative claim is currently undermined by an asymmetric evaluation setup and possible information leakage, so the significance as presented is conditional.

major comments (3)
  1. [Sec. 4.5, Table 3] The headline +15.3% Top-3 R-Precision improvement for MDM compares (T&DT)-MDM, which is conditioned on both the coarse caption and the detailed paragraph of the target ground-truth motion, against MDM conditioned on the coarse caption alone. This is not a controlled comparison of text informativeness: the fine-conditioned model receives strictly more information at test time, and the detailed paragraph is a deterministic function of the same ground-truth motion used as the retrieval target. Under R-Precision, where the generated motion must retrieve the correct text among distractors, a more discriminative and target-specific text makes the task easier regardless of motion quality. The paper needs a matched-information control, such as a coarse-caption-plus-length-matched-paraphrase condition, or should demote the fine-conditioned results and lead with the coarse-only transfer results in Table 4.
  2. [Sec. 3.1.2 and Sec. 4.4] The validity of the automatically generated snippet descriptions is load-bearing because 95% of the dataset's snippet annotations are produced by PoseFix on fixed 0.5s snippets from AMASS/HumanAct12, yet the only quantitative validation is BERTScore between the auto descriptions and human descriptions that were themselves produced with the auto descriptions as the starting point (Sec. 3.1.2). This does not establish that the descriptions accurately match the actual body-part movements in arbitrary snippets. A direct alignment test is needed, e.g., a snippet-to-text retrieval experiment on a set of independently written human descriptions, or a human rating of alignment on a random sample of snippets, to support the claim that the dataset is 'strictly aligned'.
  3. [Sec. 12 and Table 4] The paper acknowledges the asymmetry between coarse-only and coarse-plus-fine inputs (Sec. 12) but does not provide the appropriate experiment to isolate the effect. Table 4, which evaluates on the coarse-only test set, shows that training with fine-grained text improves coarse-only generation for MDM (R-Top3 0.746 vs 0.606) and MoMask (R-Top3 0.827 vs 0.753), but for T2M-GPT FID worsens (0.154 vs 0.123) while R-Top3 stays flat. This mixed evidence should be reported and discussed in the main text, and the claim that FineMotion improves fine-grained generation should be separated from the claim that it helps coarse-only generation.
minor comments (6)
  1. [Abstract and Sec. 4.5] The abstract states '+15.3% improvement in Top-3 accuracy', but the table reports an absolute increase from 0.606 to 0.759 (15.3 percentage points), not a 15.3% relative improvement; please clarify to avoid misleading readers.
  2. [Table 3 and References] Citation numbers are inconsistent: FineMoGen is correctly cited as [32] in the text but appears as [28] in Table 3, while FG-T2M is also listed as [28]; please correct these to avoid confusion.
  3. [Figure 3] The figure contains the unspaced string 'PoseFixcorrectional text generation model' which appears to be a typographical artifact.
  4. [Sec. 10 (Supplementary)] The text says 'casual self-attention' but should read 'causal self-attention'.
  5. [Sec. 4.6] The pipeline is called 'zero-shot fine-grained motion editing', but the model is explicitly trained with temporally augmented data (Sec. 4.6), so it is not zero-shot in the standard sense; a more precise term such as 'training-free at edit time' would be clearer.
  6. [Sec. 14, Table 5] The TDT-MoMask results (e.g., FID 8.328 for BPMSD) are dramatically worse than even the coarse-only MoMask baseline (FID 0.249), which suggests possible training failure or overflow rather than a simple encoding disadvantage; the authors should verify this baseline and comment on the instability.

Circularity Check

2 steps flagged · score 6.0 of 10

Headline +15.3% R-Precision gain is partly forced by conditioning on target-derived PoseFix text; human-annotation BERTScore is also partly self-referential, though Table 4 transfer keeps the central claim partially independent.

  1. self definitional [Sec. 3.1.2 (Collection of Human Annotations) and Sec. 4.4 (Evaluation of the Textual Annotation Pipeline)]
    "Then, for each motion sequence, annotators were provided with its automatically generated annotations to ease their workload. If those annotations were inappropriate, they were instructed to select suitable sentences from the corpus. This ensured that the manually annotated descriptions closely match the style of the automatically generated ones. ... Furthermore, the BERTScore between human annotations and automatically generated ones is 0.89, comparable to the scores achieved by translation models such as Transformer-big on WMT14 En-De (0.86) and En-Fr (0.89) [33]."

    The human annotations were constructed from the automatic annotations: annotators were given the auto-generated descriptions and a corpus 'derived from the automatically generated annotations for all snippets' (same section), and were told to pick replacement sentences from that corpus. The manually annotated 'ground truth' is therefore deliberately matched to the automatic output in style and vocabulary. Using the BERTScore between the two as proof of automatic-annotation quality measures agreement with a reference that was itself built from the automatic annotation, so the validation is partly by construction rather than an independent check on annotation accuracy.

  2. self definitional [Sec. 3.1.2 (Snippet Description Generation) and Sec. 4.5 (Benchmarking FineMotion / Table 3)]
    "We believe that the correctional text describing the transition between start and end poses of a snippet effectively captures the body part movements within it, and can thus be naturally regarded as the detailed BPM description for this snippet. ... All our variants exhibit performance improvements, with (T&DT)-MDM showing a notable +15.3% increase in Top-3 retrieval accuracy."

    In Table 3, the headline gain compares (T&DT)-MDM, whose condition at both training and test is the concatenation of the coarse caption and a detailed description DT, with MDM, which receives only the coarse caption. DT is generated by PoseFix from the very motion snippet that is later used as the ground-truth target of R-Precision, so the test condition contains a deterministic function of the target motion. R-Precision then partly measures how well the model re-identifies the motion from which its own detailed input was generated, and FID is similarly reduced because the target description is part of the conditioning signal.

full rationale

The central benchmarking claim is partially circular: the detailed text (DT) is produced by PoseFix from the same motion snippet used as the R-Precision ground truth, and the (T&DT) variants receive this target-derived text at test time while their coarse-only baselines do not. The +15.3% Top-3 gain therefore partly reflects the model retrieving the motion from which its own conditioning text was generated, i.e., an inversion of the annotation pipeline, rather than a fully external measure of fine-grained language understanding. The BERTScore validation is also partly self-referential because the human annotations were seeded from, and restricted to, the style of the automatic annotations. However, the paper is not built on a self-citation chain: PoseFix and Gemini are external tools, and no uniqueness claim is imported from the authors' own prior work. Moreover, Appendix Table 4 reports a transfer setting where models trained with fine-grained text are tested on coarse captions only, and MDM and MoMask improve there, giving the central claim some independent content. Because one headline 'prediction' reduces by construction to the input asymmetry, but not all evidence reduces that way, the appropriate score is 6 rather than 8.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The dataset construction relies on two external models (PoseFix and Gemini) and one hand-set duration Ts. No new theoretical entities are introduced; the central empirical claims depend on the accuracy of these upstream models and the chosen temporal granularity.

free parameters (1)
  • Snippet duration Ts = 0.5s
    Chosen by hand using two guiding principles: minimize cosine similarity between start and end poses, and do not exceed PoseFix's 0.5s maximum pose-pair time difference (Sec 3.1.1). This is a design decision, not fitted to the evaluation metrics, but it determines the temporal granularity of all annotations.
assumptions (4)
  • domain assumption PoseFix generates accurate correctional text describing the transition between two poses for arbitrary pose pairs.
    Assumed in Sec 3.1.2; the entire automatic annotation pipeline uses PoseFix's model, and its accuracy on AMASS/HumanAct12 snippets is not independently verified beyond BERTScore against human annotations (Sec 4.4).
  • domain assumption Gemini LLM follows the prompt and organizes snippet descriptions into a coherent paragraph without adding body-part movements not present in the input list.
    Assumed in Sec 3.1.3; prompt requirement (4) says 'make sure not to add any extra body part movement', but no verification is provided that Gemini complies on all 94k paragraphs.
  • domain assumption Motions in the dataset generally progress slowly, so a fixed 0.5s snippet duration avoids redundancy and captures meaningful movement transitions.
    Empirical claim from sampled snippets in Fig 4 (Sec 3.1.1); if some motion classes are fast, 0.5s snippets may be misaligned with movement phases.
  • domain assumption T5-Base with mean pooling adequately represents long detailed texts for conditioning.
    Used in all adapted baselines (Sec 10, Eq 1-2); the paper does not compare with other text encoders or pooling methods.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FineMotion: A Dataset and Benchmark with both Spatial and Temporal Annotation for Fine-grained Motion Generation and Editing." pith.science (2026). https://pith.science/paper/BSV3EF2M

@misc{pith2026250719850,
  author       = {Pith},
  title        = {Pith review of: FineMotion: A Dataset and Benchmark with both Spatial and Temporal Annotation for Fine-grained Motion Generation and Editing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BSV3EF2M}},
  note         = {Machine review of arXiv:2507.19850}
}
read the original abstract

Generating realistic human motions from textual descriptions has undergone significant advancements. However, existing methods often overlook specific body part movements and their timing. In this paper, we address this issue by enriching the textual description with more details. Specifically, we propose the FineMotion dataset, which contains over 442,000 human motion snippets - short segments of human motion sequences - and their corresponding detailed descriptions of human body part movements. Additionally, the dataset includes about 95k detailed paragraphs describing the movements of human body parts of entire motion sequences. Experimental results demonstrate the significance of our dataset on the text-driven finegrained human motion generation task, especially with a remarkable +15.3% improvement in Top-3 accuracy for the MDM model. Notably, we further support a zero-shot pipeline of fine-grained motion editing, which focuses on detailed editing in both spatial and temporal dimensions via text. Dataset and code available at: CVI-SZU/FineMotion

Figures

Figures reproduced from arXiv: 2507.19850 by the authors.

Figure 1
Figure 1. Illustration of (a) Existing Motion-Language Datasets are manually annotated, with textual descriptions that are coarse and lack detail. (b) Existing textual enhancement works obtained more detailed descriptions of a motion phrase or caption via large language models, but failed to align with the actual motion sequence. (c) Our FineMotion dataset features strictly aligned and fine-grained descriptions of human body … view at source ↗
Figure 2
Figure 2. Examples from the FineMotion dataset. Top: Human-annotated BPM snippet descriptions and paragraphs. Bottom: Auto￾matically generated BPM snippet descriptions and paragraphs. Colored text in paragraphs links to the corresponding snippet descriptions. 2. Related Work Text-Driven Human Motion Generation. Motion gener￾ation can be generated from various conditions, including text [12, 18, 25, 30, 34], action classes [11… view at source ↗
Figure 4
Figure 4. Mean and 95% confidence interval of the cosine similar [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (7 more)
Figure 3
Figure 3. Figure 3: The construction pipeline of our FineMotion dataset. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png]
Figure 5
Figure 5. Figure 5: Visualizations of the 200 most frequent words in our textual descriptions. From left to right are the word clouds of automatically-generated BPMSD, automatically-generated BPMP, human-annotated BPMSD, and human-annotated BPMP. 5 [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Pipeline for zero-shot fine-grained motion editing. To edit human motion with fine granularity, users first provide a coarse textual description of the desired motion. An initial motion is generated using any text-to-motion (T2M) model. This motion is then processed th…
Figure 7
Figure 7. Figure 7: The statistical results of the user study. The left figure displays the average preference score for the spatial editing results (3 cases) of three models, with a score of 3 for the best model, 2 for the second, and 1 for the last. The middle one shows the score for th…
Figure 9
Figure 9. Figure 9: More examples of human-annotated body part movement snippet descriptions ( [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: More examples of automatically generated body part movement snippet descriptions ( [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: Overview of the baseline network, (T&DT)2M-GPT. It generates motions that strictly follow the fine-grained descrip￾tion Xdetail and the coarse-grained caption Xcoarse. It consists of a motion VQ-VAE for discretizing the motion into tokens and a GPT for generating moti…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Interactive Generative Motion Editing via Scheduled Inpainting

    cs.GR 2026-07 conditional novelty 6.0 of 10

    Scheduled inpainting blends a base motion clip into a diffusion model's denoising process via a user-controlled schedule and spatiotemporal mask, enabling interactive editing of existing animations without retraining.

Reference graph

Works this paper leans on

47 extracted references · 37 canonical work pages · cited by 1 Pith paper

  1. [1]

    Sinc: Spatial composition of 3d human motions for simultaneous action generation supplementary material

    Nikos Athanasiou, Mathis Petrovich, Michael J Black, and G¨ul Varol. Sinc: Spatial composition of 3d human motions for simultaneous action generation supplementary material. 2

    work page

  2. [2]

    Posescript: 3d human poses from natural language

    Ginger Delmas, Philippe Weinzaepfel, Thomas Lucas, Francesc Moreno-Noguer, and Gr ´egory Rogez. Posescript: 3d human poses from natural language. In European Con- ference on Computer Vision, pages 346–362. Springer, 2022. 4, 1

    work page 2022

  3. [3]

    Posefix: Correcting 3d hu- man poses with natural language

    Ginger Delmas, Philippe Weinzaepfel, Francesc Moreno- Noguer, and Gr ´egory Rogez. Posefix: Correcting 3d hu- man poses with natural language. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15018–15028, 2023. 4, 1

    work page 2023

  4. [4]

    https://www.grammarly.com

    Grammarly. https://www.grammarly.com. 5

    work page

  5. [5]

    Ac- tion2motion: Conditioned generation of 3d human motions

    Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, Qingyao Sun, Annan Deng, Minglun Gong, and Li Cheng. Ac- tion2motion: Conditioned generation of 3d human motions. In Proceedings of the 28th ACM International Conference on Multimedia, pages 2021–2029, 2020. 3

    work page 2021

  6. [6]

    Generating diverse and natural 3d human motions from text

    Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5152–5161, 2022. 2, 3, 6, 7

    work page 2022

  7. [7]

    Tm2t: Stochastic and tokenized modeling for the reciprocal genera- tion of 3d human motions and texts

    Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. Tm2t: Stochastic and tokenized modeling for the reciprocal genera- tion of 3d human motions and texts. InEuropean Conference on Computer Vision, pages 580–597. Springer, 2022. 7

    work page 2022

  8. [8]

    Momask: Generative masked model- ing of 3d human motions

    Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Generative masked model- ing of 3d human motions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1900–1910, 2024. 3, 6, 7, 4

    work page 1900

Show all 47 references
  1. [9]

    Semanticboost: Elevating motion generation with augmented textual cues

    Xin He, Shaoli Huang, Xiaohang Zhan, Chao Wen, and Ying Shan. Semanticboost: Elevating motion generation with augmented textual cues. arXiv preprint arXiv:2310.20323 ,

    arXiv

  2. [10]

    Motiongpt: Human motion as a foreign language

    Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign language. Ad- vances in Neural Information Processing Systems, 36, 2024. 3

    2024

  3. [11]

    Action-gpt: Leveraging large-scale language models for improved and generalized action gen- eration

    Sai Shashank Kalakonda, Shubh Maheshwari, and Ravi Ki- ran Sarvadevabhatla. Action-gpt: Leveraging large-scale language models for improved and generalized action gen- eration. In 2023 IEEE International Conference on Multi- media and Expo (ICME) , pages 31–36. IEEE, 2023. 2, 3, 5

    2023

  4. [12]

    Flame: Free- form language-based motion synthesis & editing

    Jihoon Kim, Jiseob Kim, and Sungjoon Choi. Flame: Free- form language-based motion synthesis & editing. In Pro- ceedings of the AAAI Conference on Artificial Intelligence , pages 8255–8263, 2023. 2, 3, 8

    2023

  5. [13]

    Dance- former: Music conditioned 3d dance generation with para- metric motion transformer

    Buyu Li, Yongchi Zhao, Shi Zhelun, and Lu Sheng. Dance- former: Music conditioned 3d dance generation with para- metric motion transformer. In Proceedings of the AAAI Con- ference on Artificial Intelligence, pages 1272–1279, 2022. 3

    2022

  6. [14]

    Exploring multi-modal control in music- driven dance generation

    Ronghui Li, Yuqin Dai, Yachao Zhang, Jun Li, Jian Yang, Jie Guo, and Xiu Li. Exploring multi-modal control in music- driven dance generation. arXiv preprint arXiv:2401.01382,

    arXiv

  7. [15]

    Smpl: A skinned multi- person linear model

    Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi- person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 851–866. 2023. 3

    2023

  8. [16]

    Amass: Archive of motion capture as surface shapes

    Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Ger- ard Pons-Moll, and Michael J Black. Amass: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5442–5451, 2019. 3

    2019

  9. [17]

    Action- conditioned 3d human motion synthesis with transformer vae

    Mathis Petrovich, Michael J Black, and G ¨ul Varol. Action- conditioned 3d human motion synthesis with transformer vae. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 10985–10995, 2021. 3

    2021

  10. [18]

    Temos: Generating diverse human motions from textual descriptions

    Mathis Petrovich, Michael J Black, and G ¨ul Varol. Temos: Generating diverse human motions from textual descriptions. In European Conference on Computer Vision , pages 480–

  11. [19]

    The kit motion-language dataset

    Matthias Plappert, Christian Mandery, and Tamim Asfour. The kit motion-language dataset. Big data, 4(4):236–252,

  12. [20]

    Modi: Un- conditional motion synthesis from diverse data

    Sigal Raab, Inbal Leibovitch, Peizhuo Li, Kfir Aberman, Olga Sorkine-Hornung, and Daniel Cohen-Or. Modi: Un- conditional motion synthesis from diverse data. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13873–13883, 2023. 3

    2023

  13. [21]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  14. [22]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. 6, 7, 3, 4

    2020

  15. [23]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 5 9

    2023 arXiv

  16. [24]

    Motionclip: Exposing human motion generation to clip space

    Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. Motionclip: Exposing human motion generation to clip space. In European Conference on Com- puter Vision, pages 358–374. Springer, 2022. 3

    2022

  17. [25]

    Human motion dif- fusion model

    Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion dif- fusion model. arXiv preprint arXiv:2209.14916, 2022. 2, 3, 6, 7, 8, 4

    2022 arXiv

  18. [26]

    Edge: Editable dance generation from music

    Jonathan Tseng, Rodrigo Castellon, and Karen Liu. Edge: Editable dance generation from music. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 448–458, 2023. 3

    2023

  19. [27]

    Neural discrete representation learning

    Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information pro- cessing systems, 30, 2017. 3

    2017

  20. [28]

    Fg-t2m: Fine-grained text-driven human motion generation via diffusion model

    Yin Wang, Zhiying Leng, Frederick WB Li, Shun-Cheng Wu, and Xiaohui Liang. Fg-t2m: Fine-grained text-driven human motion generation via diffusion model. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 22035–22044, 2023. 7

    2023

  21. [29]

    Motionscript: Natural language descriptions for expressive 3d human motions

    Payam Jome Yazdian, Eric Liu, Li Cheng, and Angelica Lim. Motionscript: Natural language descriptions for expressive 3d human motions. arXiv preprint arXiv:2312.12634, 2023. 3

    2023

  22. [30]

    Generating human motion from textual descrip- tions with discrete representations

    Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descrip- tions with discrete representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, ...

    2023

  23. [31]

    Motiondif- fuse: Text-driven human motion generation with diffusion model

    Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondif- fuse: Text-driven human motion generation with diffusion model. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 3, 7

    2024

  24. [32]

    Finemogen: Fine-grained spatio- temporal motion generation and editing

    Mingyuan Zhang, Huirong Li, Zhongang Cai, Jiawei Ren, Lei Yang, and Ziwei Liu. Finemogen: Fine-grained spatio- temporal motion generation and editing. Advances in Neural Information Processing Systems, 36, 2024. 2, 3

    2024

  25. [33]

    Bertscore: Evaluating text genera- tion with bert

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein- berger, and Yoav Artzi. Bertscore: Evaluating text genera- tion with bert. arXiv preprint arXiv:1904.09675, 2019. 7

    1904 arXiv

  26. [34]

    Mo- tiongpt: Finetuned llms are general-purpose motion genera- tors

    Yaqi Zhang, Di Huang, Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli Ouyang. Mo- tiongpt: Finetuned llms are general-purpose motion genera- tors. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7368–7376, 2024. 3 10 Fine...

    2024

  27. [36]

    The textual descriptions in our FineMotion dataset are under the CC BY 4.0 International license

    License The license for human motion sequences in this dataset fol- lows the term specified at HumanML3D and AMASS. The textual descriptions in our FineMotion dataset are under the CC BY 4.0 International license. For detailed license infor- mation, please refer to https://cre...

  28. [37]

    As shown in Fig

    Discussion on Selecting of optimal Ts To determine the optimal snippet duration Ts, we propose two guiding principles to help researchers tailor this value to their own datasets. As shown in Fig. 4, we randomly sample 1,000 snippets with varying durations from all mo- tion seq...

  29. [38]

    Then, he slightly turns his upper body to the right

    Data Format Examples The data format example for all the detailed human body part snippet descriptions (BPMSDs) in a whole human mo- tion sequence is shown below: { ”000314”: # name of motion sequence [ ””, # 0.0s-0.5s’ BPMSD ”Bend your elbows and raise your hands up to your h...

  30. [39]

    9 and 10

    More Dataset Examples We display more examples of body part movement descriptions for motion snippet ( i.e., BPMSD) and for whole motion sequence (i.e., BPMP) of our FineMotion dataset in Fig. 9 and 10. Initially, the individual advances the right leg while moving the right ar...

  31. [40]

    • (T&DT)-MDM builds from MDM [25]

    Baseline Model Details This section outlines the network architecture and the im- plementation of three variants of motion generation meth- ods, including MDM [25], T2M-GPT [30], and Mo- Mask [8] on our dataset, and denoted them as (T&DT)- MDM, (T&DT)2M-GPT, and (T&DT)-MoMask,...

  32. [41]

    Notably, since we replace the text encoder with that of T5, the dimension of output text embedding turns to 768 rather than that of the CLIP text encoder, 512

    More Implementation Details The architecture and training hyperparameters of our base- line models strictly follow those in the original paper [8, 25, 30]. Notably, since we replace the text encoder with that of T5, the dimension of output text embedding turns to 768 rather th...

  33. [42]

    Fine-grained text captures detailed body part movements and timing, while coarse text supplements global motion semantics, both crucial for pre- cise motion generation

    More Discussion on Motion Generation with Fine-grained Texts Only We did not evaluate this setting because it will lead to am- biguity in motion generation. Fine-grained text captures detailed body part movements and timing, while coarse text supplements global motion semantic...

  34. [43]

    More Discussion on Table 2 One may notice that when (T&DT)2M-GPT— i.e., Rows (2)-(5) in Table 2—generates motions using only coarse de- scriptions (Test Set: T2M), it shows a slight performance drop, compared to our implementation of T2M-GPT trained solely on the T2M task, Row...

  35. [44]

    Specifically, we de- note the strategy of connecting the coarse text (T) and de- tailed text (DT) into a single text and feeding it into the text encoder as ‘TDT’

    Ablation Study on Baseline Model Design Here, we conduct an ablation study on different strategies for encoding coarse and detailed texts. Specifically, we de- note the strategy of connecting the coarse text (T) and de- tailed text (DT) into a single text and feeding it into t...

  36. [45]

    Metrics and Results for Temporal Align- ment Currently, there is no metric that directly evaluates the pre- cision of temporal alignment between detailed texts and generated motion sequences. Given that our detailed texts are strictly aligned with ground-truth motions over tim...

  37. [46]

    Consequently, future work will fo- cus on developing effective methods for spatial human mo- tion editing

    Limitations and Future Work Since we use temporally augmented data to train the text- to-motion models, editing motions along the temporal di- mension becomes more straightforward and accurate com- pared to spatial editing. Consequently, future work will fo- cus on developing ...

  38. [47]

    original text: a person lifts their left wrist towards their face as if to look at a watch editing requirement: Lift your left hand to the head

    User Study Case 1: Add the body part movements Spatially. original text: a person lifts their left wrist towards their face as if to look at a watch editing requirement: Lift your left hand to the head. Original Text-to-Motion After Editing Reference Snapshot (0.5s per figure)...

  39. [497]

    Springer, 2022. 2, 3, 7

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.