Speech, Multimodal, and Embodied Language

Covers image patching, patch embeddings, ViT architecture, ViT pre-training.

Covers CLIP architecture, CLIP training objective, CLIP zero-shot classification, CLIP embeddings.

Covers ViT variants for VLMs, SigLIP improvements, image resolution handling, encoder selection.

Covers linear projection, MLP projection, Q-Former approach, projection training.

Covers LLaVA design, two-stage training, visual conversation, LLaVA variants.

Covers cross-attention to images, gated cross-attention, few-shot visual learning, Flamingo training.

Covers image-text pairs, interleaved documents, visual instruction data, data quality.

Covers VQA benchmarks, multimodal understanding benchmarks, multimodal generation evaluation.

Covers multimodal learning principles, cross-modal alignment, joint embedding spaces, multimodal challenges.

Covers visual question answering, image captioning, visual grounding, scene understanding.

Covers diffusion models, text-to-image generation, image editing, generation evaluation.

Covers document understanding, medical imaging, video understanding, multimodal reasoning tasks.

Covers mel spectrograms, mel filterbanks, feature normalization, audio preprocessing.

Covers Whisper encoder-decoder, multitask training, language tokens, timestamp prediction.

Covers Whisper training data, weak supervision, multilingual training, Whisper capabilities.

Covers speech encoder + LLM, audio tokens, speech-to-text-to-LLM vs end-to-end, speech LLM architectures.

Covers TTS architecture overview, vocoder role, TTS quality metrics, neural TTS approaches.

Modality encoders, shared token spaces, fusion, generation, and interference in omni models.

Frame sampling, event localization, temporal memory, causal reasoning, and long-video evaluation.

Continuous perception, interruption, turn-taking, proactive responses, latency, and duplex generation.

OCR, layout, charts, diagrams, screenshots, grounding, and visually situated tool use.

Direct speech interaction, prosody, speaker identity, emotion, latency, and voice-specific safety.

Grounded language for robotics, action tokenization, policy learning, embodiment, and simulation-to-reality gaps.

Cross-modal injection, privacy, deepfakes, modality imbalance, interactive benchmarks, and human evaluation.