Search Results for author:
Found 25 papers, 10 papers with code
The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspective
We propose a unified multi-modal framework -- Audio-Visual Conversational Attention (AV-CONV), for the joint prediction of conversation behaviors -- speaking and listening -- for both the camera wearer as well as all other social partners present in the egocentric video.
SoundCam: A Dataset for Finding Humans Using Room Acoustics
A room's acoustic properties are a product of the room's geometry, the objects within the room, and their specific positions.
NOIR: Neural Signal Operated Intelligent Robots for Everyday Activities
We present Neural Signal Operated Intelligent Robots (NOIR), a general-purpose, intelligent brain-robot interface system that enables humans to command robots to perform everyday activities through brain signals.
RealImpact: A Dataset of Impact Sound Fields for Real Objects
Objects make unique sounds under different perturbations, environment conditions, and poses relative to the listener.
Sonicverse: A Multisensory Simulation Platform for Embodied Household Agents that See and Hear
We introduce Sonicverse, a multisensory simulation platform with integrated audio-visual simulation for training household agents that can both see and hear.
The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objects
We introduce the ObjectFolder Benchmark, a benchmark suite of 10 tasks for multisensory object-centric learning, centered around object recognition, reconstruction, and manipulation with sight, sound, and touch.
An Extensible Multimodal Multi-task Object Dataset with Materials
For example, we can train a model to predict the object category from the listing text, or the mass and price from the product listing image.
Learning Object-Centric Neural Scattering Functions for Free-Viewpoint Relighting and Scene Composition
We propose Object-Centric Neural Scattering Functions (OSFs) for learning to reconstruct object appearance from only images.
See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation
Humans use all of their senses to accomplish different tasks in everyday activities.
Differentiable Physics Simulation of Dynamics-Augmented Neural Objects
A robot can use this simulation to optimize grasps and manipulation trajectories of neural objects, or to improve the neural object models through gradient-based real-to-simulation transfer.
ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfer
We present ObjectFolder 2. 0, a large-scale, multisensory dataset of common household objects in the form of implicit neural representations that significantly enhances ObjectFolder 1. 0 in three aspects.
Visual Acoustic Matching
We introduce the visual acoustic matching task, in which an audio clip is transformed to sound like it was recorded in a target environment.
Geometry-Aware Multi-Task Learning for Binaural Audio Generation from Video
Binaural audio provides human listeners with an immersive spatial sound experience, but most existing videos lack binaural audio recordings.
ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations
Multisensory object-centric perception, reasoning, and interaction have been a key research topic in recent years.
VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency
Given a video, the goal is to extract the speech associated with a face in spite of simultaneous background sounds and/or other human speakers.
Learning to Set Waypoints for Audio-Visual Navigation
In audio-visual navigation, an agent intelligently travels through a complex, unmapped 3D environment using both sights and sounds to find a sound source (e. g., a phone ringing in another room).
VisualEchoes: Spatial Image Representation Learning through Echolocation
Several animal species (e. g., bats, dolphins, and whales) and even visually impaired humans have the remarkable ability to perform echolocation: a biological sonar used to perceive spatial layout and locate objects in the world.
Listen to Look: Action Recognition by Previewing Audio
In the face of the video data deluge, today's expensive clip-level classifiers are increasingly impractical.
Ranked #8 on
Action Recognition
on ActivityNet
Co-Separating Sounds of Visual Objects
Learning how objects sound from video is challenging, since they often heavily overlap in a single audio channel.
Ranked #1 on
Audio Denoising
on AV-Bench - Wooden Horse
2.5D Visual Sound
We devise a deep convolutional neural network that learns to decode the monaural (single-channel) soundtrack into its binaural counterpart by injecting visual information about object and scene configurations.
Learning to Separate Object Sounds by Watching Unlabeled Video
Our work is the first to learn audio source separation from large-scale "in the wild" videos containing multiple audio sources per video.
Im2Flow: Motion Hallucination from Static Images for Action Recognition
Second, we show the power of hallucinated flow for recognition, successfully transferring the learned motion into a standard two-stream network for activity recognition.
ShapeCodes: Self-Supervised Feature Learning by Lifting Views to Viewgrids
We introduce an unsupervised feature learning approach that embeds 3D shape information into a single-view image representation.
On-Demand Learning for Deep Image Restoration
While machine learning approaches to image restoration offer great promise, current methods risk training models fixated on performing well only for image corruption of a particular level of difficulty---such as a certain level of noise or blur.
Object-Centric Representation Learning from Unlabeled Videos
Compared to existing temporal coherence methods, our idea has the advantage of lightweight preprocessing of the unlabeled video (no tracking required) while still being able to extract object-level regions from which to learn invariances.
