‹ All projects

Multimodal learning

Robust Audio-Visual Speech Recognition via Multimodal Feature Fusion

An end-to-end system that combines acoustic and lip-motion features to stay accurate in noisy environments.

Period
Oct 2025 – Nov 2026
Stack
  • Python
  • PyTorch
  • TensorFlow/Keras
  • MediaPipe
  • OpenCV
01 / 05Problem
  1. 01 · Problem

    Listen, and read lips.

    Speech recognition loses accuracy quickly in noise. This project designs and implements an end-to-end audio-visual system that combines acoustic and lip-motion features, so recognition holds up when the audio gets noisy.

  2. 02 · Audio branch

    MFCC, written from scratch.

    The audio branch uses MFCC feature extraction implemented from scratch, feeding a temporal CNN model.

  3. 03 · Visual branch

    Tracking the mouth with face-mesh landmarks.

    The visual branch tracks the mouth region with MediaPipe face-mesh landmarks and extracts 2D-DCT and lip-shape features, also modelled with a temporal CNN.

  4. 04 · Fusion

    Early, late — and adaptive.

    Feature-level early fusion and decision-level late fusion were designed and compared. Entropy-based adaptive weighting favours audio when video quality drops.

  5. 05 · Results

    Tested across noise types and SNR levels.

    Comparative experiments ran across noise types and signal-to-noise ratios. Multi-condition training raised audio recognition accuracy at 0 dB from 30.5% to 77.7%. The models are implemented in both PyTorch and Keras and exported to ONNX/TFLite.

Key result

77.7%

Audio recognition accuracy at 0 dB after multi-condition training, up from 30.5%.