Multimodal learning
Robust Audio-Visual Speech Recognition via Multimodal Feature Fusion
An end-to-end system that combines acoustic and lip-motion features to stay accurate in noisy environments.
- Period
- Oct 2025 – Nov 2026
- Stack
- Python
- PyTorch
- TensorFlow/Keras
- MediaPipe
- OpenCV
01 · Problem
Listen, and read lips.
Speech recognition loses accuracy quickly in noise. This project designs and implements an end-to-end audio-visual system that combines acoustic and lip-motion features, so recognition holds up when the audio gets noisy.
02 · Audio branch
MFCC, written from scratch.
The audio branch uses MFCC feature extraction implemented from scratch, feeding a temporal CNN model.
03 · Visual branch
Tracking the mouth with face-mesh landmarks.
The visual branch tracks the mouth region with MediaPipe face-mesh landmarks and extracts 2D-DCT and lip-shape features, also modelled with a temporal CNN.
04 · Fusion
Early, late — and adaptive.
Feature-level early fusion and decision-level late fusion were designed and compared. Entropy-based adaptive weighting favours audio when video quality drops.
05 · Results
Tested across noise types and SNR levels.
Comparative experiments ran across noise types and signal-to-noise ratios. Multi-condition training raised audio recognition accuracy at 0 dB from 30.5% to 77.7%. The models are implemented in both PyTorch and Keras and exported to ONNX/TFLite.
Key result
77.7%
Audio recognition accuracy at 0 dB after multi-condition training, up from 30.5%.