Visual Speech Recognition
40 papers with code • 2 benchmarks • 5 datasets
Benchmarks
These leaderboards are used to track progress in Visual Speech Recognition
| Trend | Dataset | Best Model | Paper | Code | Compare |
|---|---|---|---|---|---|
|
|||||
|
LRS2
LRS3-TED
CAS-VSR-W1k (LRW-1000)
GLips
AV Digits Database
Latest papers
MAVD: The First Open Large-Scale Mandarin Audio-Visual Dataset with Depth Information
Audio-visual speech recognition (AVSR) gains increasing attention from researchers as an important part of human-computer interaction.
Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization
We investigate the emergent abilities of the recently proposed web-scale speech model Whisper, by adapting it to unseen tasks with prompt engineering.
Cross-Modal Global Interaction and Local Alignment for Audio-Visual Speech Recognition
yuchen005/gila
•
•
However, most existing AVSR approaches simply fuse the audio and visual features by concatenation, without explicit interactions to capture the deep correlations between them, which results in sub-optimal multimodal representations for downstream speech recognition task.
Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels
mpc001/auto_avsr
•
•
Recently, the performance of automatic, visual, and audio-visual speech recognition (ASR, VSR, and AV-ASR, respectively) has been substantially improved, mainly due to the use of larger models and training sets.
Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoring
Thus, we firstly analyze that the previous AVSR models are not indeed robust to the corruption of multimodal input streams, the audio and the visual inputs, compared to uni-modal models.
MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and Recognition
rongjiehuang/transpeech
•
•
ICCV 2023
However, despite researchers exploring cross-lingual translation techniques such as machine translation and audio speech translation to overcome language barriers, there is still a shortage of cross-lingual studies on visual speech.
MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation
We introduce MuAViC, a multilingual audio-visual corpus for robust speech recognition and robust speech-to-text translation providing 1200 hours of audio-visual speech in 9 languages.
OLKAVS: An Open Large-Scale Korean Audio-Visual Speech Dataset
iip-sogang/olkavs-avspeech
•
•
Inspired by humans comprehending speech in a multi-modal manner, various audio-visual datasets have been constructed.
Jointly Learning Visual and Auditory Speech Representations from Raw Data
ahaliassos/raven
•
•
We observe strong results in low- and high-resource labelled data settings when fine-tuning the visual and auditory encoders resulting from a single pre-training stage, in which the encoders are jointly trained.
Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognition
ms-dot-k/AVSR
•
•
The enhanced audio features are fused with the visual features and taken to an encoder-decoder model composed of Conformer and Transformer for speech recognition.
