Exploring the architecture of dual-encoder models that align audio waveforms with video frames. This note details the implementation of a contrastive loss function (InfoNCE) to create a shared embedding space, allowing the model to accurately determine if a spoken audio clip temporally matches lip movements in a video snippet.
A deep dive into optimizing NeRF architectures for real-time rendering. This blog covers the mathematics behind volumetric rendering and how techniques like hash-grid encoding (Instant NGP) drastically reduce the training time required to synthesize novel views from sparse 2D image sets.
Micro-expressions are involuntary facial movements lasting fractions of a second. This post details a custom pipeline that extracts facial landmarks over sequential video frames and uses Graph Convolutional Networks (ST-GCN) to model the structural dynamics of facial muscles for accurate emotion classification.
Analyzing methods to separate speaker identity (timbre) from linguistic content (phonemes) in raw audio. The post breaks down the use of Vector Quantized Variational Autoencoders (VQ-VAE) to compress speech, allowing us to synthesize a target voice without requiring any paired training data.
Human emotion is inherently multimodal. This research note investigates early-fusion vs. late-fusion strategies for combining CNN-based facial expression features with RNN-processed speech acoustics (pitch, energy, mel-spectrograms) to build a more robust, context-aware emotion classifier.