Search Results for author:
Found 8 papers, 2 papers with code
Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processing
In visual speech processing, context modeling capability is one of the most important requirements due to the ambiguous nature of lip movements.
Ranked #4 on
Lipreading
on LRS3-TED
(using extra training data)
Multilingual Visual Speech Recognition with a Single Model by Learning with Discrete Visual Speech Units
By using the visual speech units as the inputs of our system, we pre-train the model to predict corresponding text outputs on massive multilingual data constructed by merging several VSR databases.
Towards Practical and Efficient Image-to-Speech Captioning with Vision-Language Pre-training and Multi-modal Tokens
To this end, we start with importing the rich knowledge related to image comprehension and language modeling from a large-scale pre-trained vision-language model into Im2Sp.
Visual Speech Recognition for Languages with Limited Labeled Data using Automatic Labels from Whisper
Different from previous methods that tried to improve the VSR performance for the target language by using knowledge learned from other languages, we explore whether we can increase the amount of training data itself for the different languages without human intervention.
Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledge
In order to mitigate the challenge, we try to learn general speech knowledge, the ability to model lip movements, from a high-resource language through the prediction of speech units.
AKVSR: Audio Knowledge Empowered Visual Speech Recognition by Compressing Audio Knowledge of a Pretrained Model
Visual Speech Recognition (VSR) is the task of predicting spoken words from silent lip movements.
Multi-Temporal Lip-Audio Memory for Visual Speech Recognition
Visual Speech Recognition (VSR) is a task to predict a sentence or word from lip movements.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Reading
With the multi-head key memories, MVM extracts possible candidate audio features from the memory, which allows the lip reading model to consider the possibility of which pronunciations can be represented from the input lip movement.
Ranked #2 on
Lipreading
on CAS-VSR-W1k (LRW-1000)
