Pith. sign in

REVIEW 3 major objections 5 minor 51 references

Hearing Hands: Generating Sounds from Physical Interactions in 3D Scenes

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper shows that a 3D scene reconstruction can be queried with a hand trajectory to produce a sound that humans often cannot distinguish from a real recording.

desk verdict Useful task and dataset; the 'indistinguishable' claim doesn't survive contact with the human study, and the static-scene assumption bounds the method more than the abstract admits. read the letter →

arxiv 2506.09989 v1 pith:KWPNX4IN submitted 2025-06-11 cs.CV

Yiming Dou , Wonseok Oh , Yuqing Luo , Antonio Loquercio , Andrew Owens This is my paper
classification cs.CV
keywords soundgeneration3Dscenereconstructionrectifiedflowhandposeaction-conditionedaudioGaussiansplattingmultimodalinteraction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a static 3D reconstruction of a scene can be made interactive in the auditory domain: given a user-specified trajectory of a hand, can the model synthesize the sound that action would produce in that scene? To test this, the authors collect 9.1 hours of paired hand-action and sound data across 24 indoor and outdoor scenes, reconstructing each scene with Gaussian splatting and registering 3D hand poses into it. They train a rectified flow generative model that maps a sequence of hand poses and rendered video frames to a mel-spectrogram, and they report that the synthesized sounds convey material properties and action timing, with human listeners misclassifying them as real about 47% of the time in a forced-choice test, near the 50% chance level. If true, the work provides a way to query a reconstructed scene for plausible impact sounds of never-performed hand actions.

What carries the argument

The load-bearing component is an action-conditioned audio estimator, implemented as a rectified flow matching generative model operating in the latent space of a pretrained audio autoencoder. The model is conditioned on two synchronized streams: CLIP embeddings of video frames rendered from the Gaussian splatting reconstruction (including global views and local close-ups of the contact region), and the 3D hand pose trajectory, which is sampled at 30 Hz and upsampled to the spectrogram's temporal resolution. The two conditioning vectors are normalized and summed, then concatenated to the input noise for the flow model. Removing the visual stream degrades material accuracy, while removing the hand-pose stream degrades temporal synchronization, which is the pattern the paper uses to argue that both streams carry complementary information.

What would settle it

Take a soft, deformable object, such as a crumpled plastic bag, reconstruct the scene, record the true sound of patting it, and ask the model to synthesize the same action. Because the bag changes shape during contact, a model that only sees static appearance and hand motion should produce a sound that misses the deformation-dependent low-frequency components; if the generated sound is audibly incorrect for such objects while correct for rigid ones, the static-scene assumption is the limiting factor.

Watch

Extended reading notes

Core claim

The paper's central claim is that the sound of a hand physically interacting with a scene can be generated from the action alone, specified as a 3D hand trajectory, together with visual content rendered from a static 3D reconstruction. The authors record videos of human hands hitting, scratching, patting, and rubbing objects, lift the hands into the same coordinate frame as a Gaussian-splatting reconstruction, and use the resulting action-sound pairs to train a latent rectified flow model. At test time the model accepts any new trajectory of hand poses, renders a video of that interaction from the static scene, and outputs a mel-spectrogram (a time-frequency image of the sound) that can be converted to a waveform. The paper argues that the generated sounds are often indistinguishable from real recordings: in a two-alternative forced-choice study, participants selected the generated sound as real roughly 47% of the time, and automated CLAP-based zero-shot labeling shows that the sounds carry both action and material identity.

Load-bearing premise

The model assumes that objects in the scene stay still and do not deform when touched, so the sound depends only on the static appearance of the scene and the hand trajectory.

Editorial extensions

If this is right

  • A user can query any new 3D hand trajectory in a reconstructed scene and receive a plausible audio signal, without physically performing the action.
  • The synthesized audio preserves material and action identity well enough that a zero-shot CLAP model assigns the same action-material label to generated and ground-truth sounds at rates well above the ablated baselines.
  • Real-or-fake discrimination by human listeners is near chance for the full model, with a misclassification rate of about 47 percent, indicating the generated sounds are often judged as realistic as actual recordings.
  • The synthetic-view data augmentation, which renders the same interaction from different camera poses, improves generation quality and does not harm audio-video alignment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same conditioning scheme could plausibly transfer to other contact-based actuators, such as robot grippers or drumsticks, as long as the contact trajectory is expressed in the same 3D coordinates; the paper does not test this extension.
  • If the model truly captures material properties from appearance, the generated audio could serve as a weak supervisory signal for material classification or physics estimation in scenes, a use the paper leaves implicit.
  • A direct stress test would be to query the model with the same hand trajectory on two scenes that look alike but are made of different materials; the paper's claim predicts that the model should produce different sounds, and this could be measured with CLAP accuracy or human listening.
  • The near-chance human result may partly reflect that background noise in real recordings hurts perceived realism, which the authors noted qualitatively; this suggests that denoising real reference sounds could change the outcome of future real-or-fake evaluations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper addresses the task of generating plausible audio for hand-based physical interactions with 3D scene reconstructions. The authors collect a new dataset of 24 indoor/outdoor scenes with over 9 hours of synchronized hand-interaction videos and audio, reconstruct each scene with Gaussian Splatting, estimate 3D hand poses with HaMeR, and render simulated interaction videos from multiple viewpoints. They train a rectified-flow generative model (built on Frieren) that conditions on CLIP features of the rendered video and a sequence of 3D hand poses to synthesize a mel-spectrogram, which is converted to a waveform with a vocoder. Experiments report automated metrics (STFT, envelope, CDPAM, FID/IS, CLAP accuracy) and a human 2AFC real-or-fake study, where the full model achieves a misclassification rate of approximately 47%, which the authors interpret as near-indistinguishable from real sounds. The paper also ablates the contribution of CLIP features, hand poses, and synthetic-view augmentation.

Significance. If the central claims hold, this is a valuable contribution to interactive 3D reconstruction and audio-visual generation: it provides a large dataset of aligned 3D hand-action–sound pairs, a novel conditioning scheme combining static scene appearance and hand trajectories, and a plausible use of rectified flow matching for this task. The inclusion of a human perceptual study and explicit ablations are strengths. However, the paper's strongest claim—that generated sounds are often indistinguishable from real ones—rests on a human study that is statistically indistinguishable from chance, and the evaluation protocol has a potential train/test leakage risk. The acknowledged limitation regarding object deformation is not quantified. These issues must be addressed before the claims can be accepted.

major comments (3)
  1. [4 (Implementation Details)] The training/test split is performed on camera views rather than on distinct interaction recordings: 'We use the knowledge of each video’s camera pose to ensure that none of the camera views in the test set overlap with the ones in the training and validation set.' Since the data augmentation in Sec. 3.1 synthesizes multiple views (global, local, top, side) from the same original recording, the same physical interaction and its audio can appear under different rendered viewpoints in both training and test. This risk of train/test contamination could inflate all reported metrics, including the human study, because the model may memorize audio associated with a specific interaction rather than generalizing to new actions. Please clarify whether test clips are drawn from entirely distinct original videos, and if not, re-run the evaluation with a split by recording session.
  2. [5.1 (Real-or-fake study) and Table 1] The full model's misclassification rate is reported as approximately 47% (Table 1: 47.18±2.66). This is statistically indistinguishable from the 50% chance level, yet the abstract claims the sounds are 'often indistinguishable to human observers from real sounds.' The paper does not report a significance test or confidence interval for the comparison to 50%; with the reported uncertainty, the 95% confidence interval likely includes 50%, so the result does not support the claim of indistinguishability. Please report the appropriate confidence interval and a test against chance, and interpret the result accordingly (e.g., as evidence that raters cannot reliably distinguish, if the CI excludes a meaningful difference, or as inconclusive otherwise).
  3. [6 (Limitations) and Sec. 3 (Method)] The method conditions on a video rendered from a static Gaussian Splatting reconstruction (Sec. 3.1) and explicitly assumes objects do not move or deform when manipulated, as stated in Sec. 6. For deformable objects such as snow, plastic bags, or cloth, the conditioning signal does not contain the deformation information that substantially shapes the impact sound, so the mapping from (v,a) to sound is not identifiable from the inputs. The evaluation, however, averages all metrics over the test set and never isolates performance on rigid versus deformable interactions; the breakdown in Fig. 7 by softness/smoothness does not address this failure mode. Given that the limitation is acknowledged to be 'often violated,' the paper should report results separately for interactions where the assumption holds versus those where it is violated, or otherwise quantify the impact of this limitation on the central claim.
minor comments (5)
  1. [Table 1] Please clarify what the ± values represent (standard deviation, standard error, or 95% confidence interval) and report confidence intervals for the real-or-fake study so readers can assess the comparison to chance.
  2. [Table 1] The column heading 'Labeled real (%)' is ambiguous; the text describes the value as a misclassification rate. Please reconcile the heading with the description to avoid confusion about what is being reported.
  3. [Figure 7] The categories 'softness' and 'smoothness' in Figure 7 are not defined. Please specify how these material properties were assigned to test clips, since this breakdown is used to support claims about material-dependent performance.
  4. [3.1 (Dataset)] The paper states approximately 1,400 seconds of video per scene over 24 scenes, which sums to about 9.33 hours, close to but not exactly the reported 9.1 hours. Please check the arithmetic or clarify whether some scenes have shorter recordings.
  5. [5.1 (CLAP accuracy)] The CLAP accuracy metric is newly introduced and used as a central quantitative measure. Please provide a validation of this metric (e.g., correlation with human judgments) so readers can interpret what it captures.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central claim is supported by held-out test data and an external human study; self-citations are not load-bearing.

full rationale

This paper is an empirical learning pipeline rather than an analytic derivation, so the standard circularity patterns (self-definitional equations, fitting a parameter and then predicting the same quantity) do not apply. The central claim that generated sounds convey material properties and actions is evaluated on a held-out test split and by a real-or-fake study with 59 participants choosing between ground-truth and generated audio; that is an independent benchmark, not a quantity forced by construction. The CLAP-accuracy metric uses an off-the-shelf audio-language model, and although the generator conditions on CLIP visual features, the metric is not the training objective and is not identical to the model's inputs. The paper's own Sec. 6 limitation (objects assumed not to move or deform when manipulated) is a genuine correctness and generalization risk for deformable objects, but it is not circularity: it means the conditioning signal may be underdetermined in some interactions, not that the output reduces to the input by definition. Self-citations such as [8] (Tactile-augmented radiance fields) and [13] (ObjectFolder benchmark) appear only in related-work comparisons or metric conventions and do not carry a load-bearing argument. No equation, fitted parameter, or metric is renamed as a prediction, and no specific circular step can be exhibited. Therefore the score is 1.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The paper introduces no new physical entities. Its central claim depends on standard ML background (rectified flow), pretrained components (Gaussian Splatting, HaMeR, CLIP, CLAP, BigVGAN), the learnability assumption, and a set of hand-chosen hyperparameters. The dataset is a new artifact but not an invented entity in the sense of a new force or particle.

free parameters (4)
  • Neural network weights of the audio generator F_phi = trained on 9.1 hours of author-collected data
    The central claim of realistic generation rests on these fitted weights; the evaluation is in-distribution.
  • Guidance scale = 4.5
    Inference hyperparameter chosen by the authors, affects sample quality and diversity; no sensitivity analysis.
  • Sampling steps = 26
    Inference hyperparameter for the rectified flow ODE solver; chosen by the authors.
  • Training schedule (40 epochs, batch size 128, LR 1e-5 to 4e-4) = specified in Section 4.1
    Hyperparameters chosen by hand; no systematic search reported.
assumptions (6)
  • standard math Rectified flow matching provides a valid generative model for continuous distributions.
    The model is built on rectified flow (Liu et al., 2022) and Frieren (Wang et al., 2024); the paper assumes this background theory is correct.
  • domain assumption Gaussian Splatting reconstructions accurately represent scene appearance for the rendered conditioning video.
    The rendered global and local views are generated from the Splatfacto reconstruction; errors here propagate into the conditioning signal.
  • domain assumption HaMeR 3D hand pose estimates are sufficiently accurate and synchronized with the recorded audio.
    The action condition a is the HaMeR output; inaccurate poses would misalign the trajectory with the target sound.
  • domain assumption Sound is a function of the static scene appearance and hand trajectory, with object motion and deformation negligible.
    The paper states this assumption in the Conclusion and notes it is often violated for small objects.
  • domain assumption CLIP image embeddings carry the material and action information needed to predict audio.
    The model conditions on CLIP features of rendered frames; the paper found CLIP worked better than CAVP, but this is an empirical choice.
  • domain assumption The pretrained CLAP model is a valid automatic proxy for whether generated sounds have the correct action and material label.
    The new CLAP-acc metric relies on CLAP zero-shot classification; the paper gives no calibration against human judgments for this classifier.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hearing Hands: Generating Sounds from Physical Interactions in 3D Scenes." pith.science (2026). https://pith.science/paper/KWPNX4IN

@misc{pith2026250609989,
  author       = {Pith},
  title        = {Pith review of: Hearing Hands: Generating Sounds from Physical Interactions in 3D Scenes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KWPNX4IN}},
  note         = {Machine review of arXiv:2506.09989}
}
read the original abstract

We study the problem of making 3D scene reconstructions interactive by asking the following question: can we predict the sounds of human hands physically interacting with a scene? First, we record a video of a human manipulating objects within a 3D scene using their hands. We then use these action-sound pairs to train a rectified flow model to map 3D hand trajectories to their corresponding audio. At test time, a user can query the model for other actions, parameterized as sequences of hand poses, to estimate their corresponding sounds. In our experiments, we find that our generated sounds accurately convey material properties and actions, and that they are often indistinguishable to human observers from real sounds. Project page: https://www.yimingdou.com/hearing_hands/

Figures

Figures reproduced from arXiv: 2506.09989 by the authors.

Figure 1
Figure 1. What sound does this object make when you strike it with your hand? We capture a 3D scene representation that can be used to simulate the sound that would result from a given hand motion. We reconstruct the scene Gaussian Splatting [20], then manipulate objects in the scene with hands, obtaining a sparse set of action-sound pairs. We use these examples to train a rectified flow model to map 3D hand trajectories at g… view at source ↗
Figure 2
Figure 2. Sound generation. We train a rectified flow model [45] to generate a sound spectrogram from a sequence of 3D hand po￾sitions and video frames generated from a 3D reconstruction of a scene. The sound can subsequently be converted into a waveform using a vocoder. Frieren [45] uses rectified flow matching [27] for better gen￾eration quality and efficiency. Our audio generation module is based on the Frieren’s rectified… view at source ↗
Figure 3
Figure 3. Data capturing pipeline. In the original video, a human collector interacts with the scene by performing various actions with their hands. We lift the annotator’s hands to the same 3D space of the scene reconstruction. We render a video of the interaction by projecting 3D hands on multiple viewpoints of the scene. All rendered videos are synchronized with the sounds made by the hand actions [PITH_FULL_IMAGE:figures… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Representative examples from the dataset. Our dataset is collected in 24 scenes, including offices, outdoor trees, bedrooms, etc. We show six such scenes in the figure above, with examples of action-generated sounds. Our dataset covers a wide range of actions (hitting,…
Figure 5
Figure 5. Figure 5: Qualitative results. We show the generation results on five interactions. Generally, the predictions match the ground-truth in both motion synchronization and material properties. Note that when the hand is less visible in the video or the motion is ambiguous (e.g., th…
Figure 6
Figure 6. Figure 6: Ablation study results. We show the spectrogram predictions from our full model and three ablations. We notice that removing CLIP features softens impact sounds while removing hand pose features results in poor audio-video synchronization. Similar to quantitative resul…
Figure 7
Figure 7. Figure 7: Results of real-or-fake study. We show the ratio of humans being fooled by different variants of our model. We break down our results into three categories: softness, smoothness, and average over all samples. The error bars show 95% confidence intervals. We find that o…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

51 extracted references · 37 canonical work pages

  1. [1]

    Look, listen and learn

    Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. InProceedings of the IEEE international conference on computer vision, pages 609–617, 2017. 2

    work page 2017

  2. [2]

    Novel-view acoustic synthesis

    Changan Chen, Alexander Richard, Roman Shapovalov, Vamsi Krishna Ithapu, Natalia Neverova, Kristen Grauman, and Andrea Vedaldi. Novel-view acoustic synthesis. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. 3

    work page 2023

  3. [3]

    Vggsound: A large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. InICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 721–725. IEEE, 2020. 7

    work page 2020

  4. [4]

    Generating visually aligned sound from videos.IEEE Transactions on Image Processing, 29:8292–8302, 2020

    Peihao Chen, Yang Zhang, Mingkui Tan, Hongdong Xiao, Deng Huang, and Chuang Gan. Generating visually aligned sound from videos.IEEE Transactions on Image Processing, 29:8292–8302, 2020. 2

    work page 2020

  5. [5]

    Real acoustic fields: An audio-visual room acous- tics dataset and benchmark

    Ziyang Chen, Israel D Gebru, Christian Richardt, Anurag Kumar, William Laney, Andrew Owens, and Alexander Richard. Real acoustic fields: An audio-visual room acous- tics dataset and benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21886–21896, 2024. 3

    work page 2024

  6. [6]

    The visual microphone: Passive recovery of sound from video

    Abe Davis, Michael Rubinstein, Neal Wadhwa, Gautham J Mysore, Fredo Durand, and William T Freeman. The visual microphone: Passive recovery of sound from video. 2014. 2

    work page 2014

  7. [7]

    Image-space modal bases for plausible manipulation of objects in video

    Abe Davis, Justin G Chen, and Frédo Durand. Image-space modal bases for plausible manipulation of objects in video. ACM Transactions on Graphics (TOG), 34(6):1–7, 2015. 1, 2

    work page 2015

  8. [8]

    Tactile-augmented radiance fields

    Yiming Dou, Fengyu Yang, Yi Liu, Antonio Loquercio, and Andrew Owens. Tactile-augmented radiance fields. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26529–26539, 2024. 2

    work page 2024

Show all 51 references
  1. [9]

    Learning signal-agnostic manifolds of neural fields

    Yilun Du, Katie Collins, Josh Tenenbaum, and Vincent Sitz- mann. Learning signal-agnostic manifolds of neural fields. Advances in Neural Information Processing Systems, 2021. 3

    2021

  2. [10]

    Conditional generation of audio from video via foley analogies

    Yuexi Du, Ziyang Chen, Justin Salamon, Bryan Russell, and Andrew Owens. Conditional generation of audio from video via foley analogies. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 2426–2436, 2023. 2

    2023

  3. [11]

    Objectfolder: A dataset of objects with implicit visual, auditory, and tactile representations.arXiv preprint arXiv:2109.07991, 2021

    Ruohan Gao, Yen-Yu Chang, Shivani Mall, Li Fei-Fei, and Jiajun Wu. Objectfolder: A dataset of objects with implicit visual, auditory, and tactile representations.arXiv preprint arXiv:2109.07991, 2021. 2

    2021 arXiv

  4. [12]

    Ob- jectfolder 2.0: A multisensory object dataset for sim2real transfer

    Ruohan Gao, Zilin Si, Yen-Yu Chang, Samuel Clarke, Jean- nette Bohg, Li Fei-Fei, Wenzhen Yuan, and Jiajun Wu. Ob- jectfolder 2.0: A multisensory object dataset for sim2real transfer. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 105...

    2022

  5. [13]

    The ob- jectfolder benchmark: Multisensory learning with neural and real objects

    Ruohan Gao, Yiming Dou, Hao Li, Tanmay Agarwal, Jean- nette Bohg, Yunzhu Li, Li Fei-Fei, and Jiajun Wu. The ob- jectfolder benchmark: Multisensory learning with neural and real objects. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages ...

    2023

  6. [14]

    Learning joint reconstruction of hands and manipulated ob- jects

    Yana Hasson, Gul Varol, Dimitrios Tzionas, Igor Kale- vatykh, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning joint reconstruction of hands and manipulated ob- jects. InProceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pages 11807–11816,

  7. [15]

    Modeling and driving human body soundfields through acoustic primitives.arXiv preprint arXiv:2407.13083, 2024

    Chao Huan, Dejan Markovic, Chenliang Xu, and Alexan- der Richard. Modeling and driving human body soundfields through acoustic primitives.arXiv preprint arXiv:2407.13083, 2024. 3

    2024 arXiv

  8. [16]

    Taming visually guided sound generation

    Vladimir Iashin and Esa Rahtu. Taming visually guided sound generation. InBritish Machine Vision Conference (BMVC), 2021. 6

    2021

  9. [17]

    Taming visually guided sound generation.arXiv preprint arXiv:2110.08791, 2021

    Vladimir Iashin and Esa Rahtu. Taming visually guided sound generation.arXiv preprint arXiv:2110.08791, 2021. 2

    2021 arXiv

  10. [18]

    Vr-gs: A physical dynamics- aware interactive gaussian splatting system in virtual reality

    Ying Jiang, Chang Yu, Tianyi Xie, Xuan Li, Yutao Feng, Huamin Wang, Minchen Li, Henry Lau, Feng Gao, Yin Yang, and Chenfanfu Jiang. Vr-gs: A physical dynamics- aware interactive gaussian splatting system in virtual reality. arXiv preprint arXiv:2401.16663, 2024. 1, 2

    2024 arXiv

  11. [19]

    Can one hear the shape of a drum?The american mathematical monthly, 73(4P2):1–23, 1966

    Mark Kac. Can one hear the shape of a drum?The american mathematical monthly, 73(4P2):1–23, 1966. 2

    1966

  12. [20]

    3d gaussian splatting for real-time radiance field rendering.ACM Transactions on Graphics, 42 (4), 2023

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering.ACM Transactions on Graphics, 42 (4), 2023. 1, 2, 3

    2023

  13. [21]

    Lerf: Language embedded radiance fields

    Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. InProceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 19729–19739,

  14. [22]

    Robot see robot do: Imitating articulated object manipulation with monocular 4d reconstruction

    Justin Kerr, Chung Min Kim, Mingxuan Wu, Brent Yi, Qian- qian Wang, Ken Goldberg, and Angjoo Kanazawa. Robot see robot do: Imitating articulated object manipulation with monocular 4d reconstruction. In8th Annual Conference on Robot Learning, 2024. 1

    2024

  15. [23]

    Garfield: Group anything with radiance fields

    Chung Min* Kim, Mingxuan* Wu, Justin* Kerr, Matthew Tancik, Ken Goldberg, and Angjoo Kanazawa. Garfield: Group anything with radiance fields. InConference on Com- puter Vision and Pattern Recognition (CVPR), 2024. 1

    2024

  16. [24]

    Adam: A method for stochastic opti- mization.arXiv preprint arXiv:1412.6980, 2014

    Diederik P Kingma. Adam: A method for stochastic opti- mization.arXiv preprint arXiv:1412.6980, 2014. 4, 5

    2014 arXiv

  17. [25]

    Bigvgan: A universal neural vocoder with large-scale training.arXiv preprint arXiv:2206.04658,

    Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. Bigvgan: A universal neural vocoder with large-scale training.arXiv preprint arXiv:2206.04658,

    arXiv

  18. [26]

    Av-nerf: Learning neural fields for real-world audio-visual scene synthesis.arXiv preprint arXiv:2302.02088, 2023

    Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Av-nerf: Learning neural fields for real-world audio-visual scene synthesis.arXiv preprint arXiv:2302.02088, 2023. 3

    2023 arXiv

  19. [27]

    Flow straight and fast: Learning to generate and transfer data with rectified flow.arXiv preprint arXiv:2209.03003, 2022

    Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow.arXiv preprint arXiv:2209.03003, 2022. 2, 3, 4 9

    2022 arXiv

  20. [28]

    Diff-foley: Synchronized video-to-audio synthesis with la- tent diffusion models.Advances in Neural Information Pro- cessing Systems, 36, 2024

    Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. Diff-foley: Synchronized video-to-audio synthesis with la- tent diffusion models.Advances in Neural Information Pro- cessing Systems, 36, 2024. 2, 4, 5

    2024

  21. [29]

    Few-shot audio-visual learning of environ- ment acoustics.Advances in Neural Information Processing Systems, 2022

    Sagnik Majumder, Changan Chen, Ziad Al-Halah, and Kris- ten Grauman. Few-shot audio-visual learning of environ- ment acoustics.Advances in Neural Information Processing Systems, 2022. 3

    2022

  22. [30]

    CDPAM: Contrastive learning for perceptual au- dio similarity

    Pranay Manocha, Zeyu Jin, Richard Zhang, and Adam Finkelstein. CDPAM: Contrastive learning for perceptual au- dio similarity. InICASSP 2021, To Appear, 2021. 6

    2021

  23. [31]

    Nerf: Representing scenes as neural radiance fields for view syn- thesis.Communications of the ACM, 65(1):99–106, 2021

    Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view syn- thesis.Communications of the ACM, 65(1):99–106, 2021. 1, 2

    2021

  24. [32]

    Multimodal deep learn- ing

    Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, Andrew Y Ng, et al. Multimodal deep learn- ing. InICML, 2011. 2

    2011

  25. [33]

    Sound synthesis and evaluation of interactive footsteps and envi- ronmental sounds rendering for virtual reality applications

    Rolf Nordahl, Luca Turchet, and Stefania Serafin. Sound synthesis and evaluation of interactive footsteps and envi- ronmental sounds rendering for virtual reality applications. IEEE transactions on visualization and computer graphics,

  26. [34]

    Audio-visual scene analysis with self-supervised multisensory features

    Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. InPro- ceedings of the European conference on computer vision (ECCV), pages 631–648, 2018. 2

    2018

  27. [35]

    Visually indicated sounds

    Andrew Owens, Phillip Isola, Josh McDermott, Antonio Tor- ralba, Edward H Adelson, and William T Freeman. Visually indicated sounds. InProceedings of the IEEE conference on computer vision and pattern recognition, 2016. 2, 7

    2016

  28. [36]

    Reconstruct- ing hands in 3D with transformers

    Georgios Pavlakos, Dandan Shan, Ilija Radosavovic, Angjoo Kanazawa, David Fouhey, and Jitendra Malik. Reconstruct- ing hands in 3D with transformers. InCVPR, 2024. 2, 3

    2024

  29. [37]

    D-nerf: Neural radiance fields for dynamic scenes

    Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 10318–10327, 2021. 2

    2021

  30. [38]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. InInternational conference on machine learning, p...

    2021

  31. [39]

    Language embedded radiance fields for zero-shot task-oriented grasping

    Adam Rashid, Satvik Sharma, Chung Min Kim, Justin Kerr, Lawrence Yunliang Chen, Angjoo Kanazawa, and Ken Gold- berg. Language embedded radiance fields for zero-shot task-oriented grasping. In7th Annual Conference on Robot Learning, 2023. 2

    2023

  32. [40]

    Structure-from-motion revisited

    Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-motion revisited. InConference on Com- puter Vision and Pattern Recognition (CVPR), 2016. 3

    2016

  33. [41]

    Understanding human hands in contact at inter- net scale

    Dandan Shan, Jiaqi Geng, Michelle Shu, and David F Fouhey. Understanding human hands in contact at inter- net scale. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9869–9878,

  34. [42]

    Inras: Implicit neural representation for audio scenes.Advances in Neural Information Processing Systems, 2022

    Kun Su, Mingfei Chen, and Eli Shlizerman. Inras: Implicit neural representation for audio scenes.Advances in Neural Information Processing Systems, 2022. 3

    2022

  35. [43]

    Nerfstudio: A modular framework for neural radiance field development

    Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristof- fersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. Nerfstudio: A modular framework for neural radiance field development. InACM SIG...

    2023

  36. [44]

    Reconciling reality through simulation: A real-to-sim-to-real approach for robust manipulation.arXiv preprint arXiv:2403.03949,

    Marcel Torne, Anthony Simeonov, Zechu Li, April Chan, Tao Chen, Abhishek Gupta, and Pulkit Agrawal. Reconciling reality through simulation: A real-to-sim-to-real approach for robust manipulation.arXiv preprint arXiv:2403.03949,

    arXiv

  37. [45]

    Frieren: Efficient video-to-audio generation with rectified flow matching.arXiv preprint arXiv:2406.00320, 2024

    Yongqi Wang, Wenxiang Guo, Rongjie Huang, Jiawei Huang, Zehan Wang, Fuming You, Ruiqi Li, and Zhou Zhao. Frieren: Efficient video-to-audio generation with rectified flow matching.arXiv preprint arXiv:2406.00320, 2024. 2, 3, 4, 5, 6

    2024 arXiv

  38. [46]

    Large-scale con- trastive language-audio pretraining with feature fusion and keyword-to-caption augmentation

    Yusong Wu*, Ke Chen*, Tianyu Zhang*, Yuchen Hui*, Tay- lor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale con- trastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. InIEEE International Conference on Acoustics, Speech and Signal Process...

    2023

  39. [47]

    Physgaussian: Physics- integrated 3d gaussians for generative dynamics

    Tianyi Xie, Zeshun Zong, Yuxing Qiu, Xuan Li, Yutao Feng, Yin Yang, and Chenfanfu Jiang. Physgaussian: Physics- integrated 3d gaussians for generative dynamics. InProceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4389–4398, 2024. 1, 2, 4

    2024

  40. [48]

    Sounding bod- ies: modeling 3d spatial sound of humans using body pose and audio.Advances in Neural Information Processing Sys- tems, 36, 2024

    Xudong Xu, Dejan Markovic, Jacob Sandakly, Todd Kee- bler, Steven Krenn, and Alexander Richard. Sounding bod- ies: modeling 3d spatial sound of humans using body pose and audio.Advances in Neural Information Processing Sys- tems, 36, 2024. 3

    2024

  41. [49]

    Llm- grounder: Open-vocabulary 3d visual grounding with large language model as an agent

    Jianing Yang, Xuweiyi Chen, Shengyi Qian, Nikhil Madaan, Madhavan Iyengar, David F Fouhey, and Joyce Chai. Llm- grounder: Open-vocabulary 3d visual grounding with large language model as an agent. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 76...

    2024

  42. [50]

    Shape and material from sound.Advances in Neural Information Processing Systems, 30, 2017

    Zhoutong Zhang, Qiujia Li, Zhengjia Huang, Jiajun Wu, Josh Tenenbaum, and Bill Freeman. Shape and material from sound.Advances in Neural Information Processing Systems, 30, 2017. 2

    2017

  43. [51]

    Visual to sound: Generating natural sound for videos in the wild

    Yipin Zhou, Zhaowen Wang, Chen Fang, Trung Bui, and Tamara L Berg. Visual to sound: Generating natural sound for videos in the wild. InProceedings of the IEEE con- ference on computer vision and pattern recognition, pages 3550–3558, 2018. 2 10

    2018

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.