AI & Machine Learning Systems Engineering
A mathematically rigorous and systems-focused journey into modern machine learning. Go beyond calling APIs to understanding tensor operations, gradient descent backpropagation, architecture design, and low-latency deployment.
Mathematical Foundations & Computational Tensors
Linear algebra, calculus of vector spaces, and probability distributions that power neural networks.
What you're experiencing: "Linear Algebra & Gradients"
You expected to build neural networks on day 1, but you find yourself staring at matrix decompositions and partial derivatives. You wonder: "Do I really need this math?"
Your brain is establishing spatial intuition for high-dimensional geometric hyperplanes. Without this, modern loss landscapes and embedding spaces will always feel like black magic.
- Skipping matrix dimensions and struggling endlessly later with tensor broadcast errors.
- Memorizing proofs instead of understanding geometric transformations (rotation, scaling, projection).
Linear Algebra for Machine Learning
Vector spaces, matrix multiplications, dot products, projection, and SVD decomposition.
Vector Calculus & Automatic Differentiation
Jacobians, Hessians, partial derivatives, and computational reverse-mode graphs.
NumPy Tensor Computations & Vectorization
Strides, memory broadcasting, contiguous buffers, and vectorized SIMD execution.
Capstone: Autograd Engine from Pure Python
Build a miniature scalar autograd engine (similar to Andrej Karpathy micrograd) implementing forward and backward passes, dynamic computational DAG, and training an MLP classifier from scratch without PyTorch.
Deep Neural Networks & PyTorch Internals
Convolutions, Recurrence, Regularization, and debugging PyTorch training loops on GPU.
What you're experiencing: "PyTorch & Backpropagation"
After the initial struggle with tensor shapes, things click! You train your first model and watch the loss curve decay steadily. You feel a massive wave of capability.
You have transitioned from passive theory into active experimental science. You learn to read loss curves like an EKG and understand bias-variance trade-offs.
- Treating deep learning like magic hyperparameters without checking gradient norm explosions.
- Forgetting model.eval() and torch.no_grad() during inference, causing silent test evaluation leakage.
PyTorch Architecture: Tensors, Autograd & Modules
nn.Module, Dataset/DataLoader workers, custom loss functions, and CUDA device transfers.
Optimization Dynamics: AdamW, Learning Rate Schedules & Norms
Momentum, RMSprop, AdamW weight decay, Batch Normalization, Layer Normalization, and Dropout.
Capstone: Residual Neural Network with Custom GPU DataLoader
Implement a ResNet architecture with skip connections, custom convolution blocks, mixed precision (torch.cuda.amp), and achieve >92% test accuracy on CIFAR-10 in under 15 minutes of training.
Attention Mechanisms, Transformers & Modern LLMs
Self-attention, Multi-Head Attention, RoPE embeddings, KV-cache, and decoder-only architectures.
What you're experiencing: "Transformer Architecture & KV-Cache"
You move from consumer of AI to builder of AI systems. You dismantle the transformer black-box and see that attention is simply query-key dot products and softmax routing.
You have achieved true engineering literacy. Instead of depending on third-party SaaS wrappers, you understand memory footprints, sequence lengths, and context windows at the byte level.
- Confusing training attention (parallel causal mask) with autoregressive inference (sequential generation with KV cache).
- Ignoring tokenization quirks (BPE merges, byte fallbacks, and whitespace handling).
Scaled Dot-Product & Multi-Head Attention
Attention Is All You Need: queries, keys, values, causal masking, and rotary position embeddings (RoPE).
LLM Inference Optimization: KV-Cache, PagedAttention & Quantization
Understanding the memory-bandwidth wall, flash-attention, INT8/INT4 quantization, and speculative decoding.
Capstone: Mini-Llama Decoder with KV-Cache & Streaming API
Build and train a 50M parameter decoder-only transformer with RoPE, SwiGLU activation, and RMSNorm. Implement autoregressive generation with KV-cache and wrap it in an asynchronous FastAPI streaming server.