Visual Speech Recognition
40 papers with code • 2 benchmarks • 5 datasets
Benchmarks
These leaderboards are used to track progress in Visual Speech Recognition
| Trend | Dataset | Best Model | Paper | Code | Compare |
|---|---|---|---|---|---|
|
|||||
|
LRS2
LRS3-TED
CAS-VSR-W1k (LRW-1000)
GLips
AV Digits Database
Latest papers
A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition
In this paper, we investigate this contrasting phenomenon from the perspective of modality bias and reveal that an excessive modality bias on the audio caused by dropout is the underlying reason.
Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processing
sally-sh/vsp-llm
•
•
In visual speech processing, context modeling capability is one of the most important requirements due to the ambiguous nature of lip movements.
The NPU-ASLP-LiAuto System Description for Visual Speech Recognition in CNVSRC 2023
mkt-dataoceanai/cnvsrc2023baseline
•
•
This paper delineates the visual speech recognition (VSR) system introduced by the NPU-ASLP-LiAuto (Team 237) in the first Chinese Continuous Visual Speech Recognition Challenge (CNVSRC) 2023, engaging in the fixed and open tracks of Single-Speaker VSR Task, and the open track of Multi-Speaker VSR Task.
Multichannel AV-wav2vec2: A Framework for Learning Multichannel Multi-Modal Speech Representation
zqs01/multi-channel-wav2vec2
•
•
Considering that visual information helps to improve speech recognition performance in noisy scenes, in this work we propose a multichannel multi-modal speech self-supervised learning framework AV-wav2vec2, which utilizes video and multichannel audio data as inputs.
Do VSR Models Generalize Beyond LRS3?
yasserdahouml/vsr_test_set
•
•
The Lip Reading Sentences-3 (LRS3) benchmark has primarily been the focus of intense research in visual speech recognition (VSR) during the last few years.
LIP-RTVE: An Audiovisual Database for Continuous Spanish in the Wild
Speech is considered as a multi-modal process where hearing and vision are two fundamentals pillars.
Improving Audio-Visual Speech Recognition by Lip-Subword Correlation Based Visual Pre-training and Cross-Modal Fusion Encoder
mispchallenge/misp-icme-avsr
•
•
In this paper, we propose two novel techniques to improve audio-visual speech recognition (AVSR) under a pre-training and fine-tuning training framework.
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition
In this paper, we aim to learn the shared representations across modalities to bridge their gap.
Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognition
yuchen005/univpm
•
•
In this work, we investigate the noise-invariant visual modality to strengthen robustness of AVSR, which can adapt to any testing noises while without dependence on noisy training data, a. k. a., unsupervised noise adaptation.
OpenSR: Open-Modality Speech Recognition via Maintaining Multi-Modality Alignment
exgc/opensr
•
•
We demonstrate that OpenSR enables modality transfer from one to any in three different settings (zero-, few- and full-shot), and achieves highly competitive zero-shot performance compared to the existing few-shot and full-shot lip-reading methods.
