Advanced Machine Learning Theory (Optimization)

The subject includes stochastic gradient descent, momentum methods, adaptive optimizers, Hessian approximations, and convergence analysis. It focuses on why training algorithms behave as they do, not just how to call them in a software library.

The Hessian describes the curvature of a loss surface through second derivatives. It helps explain why Newton-style methods can move quickly near a solution, while also showing why storing or computing it is difficult for modern neural networks.

Full-batch gradient descent uses the entire training set for each update, while stochastic or mini-batch methods estimate the gradient from a smaller sample. The noisy estimate can make training faster and sometimes helps the optimizer move through flat or irregular regions.

A convergence proof states conditions under which an algorithm approaches a stationary point or reaches a bounded error after enough updates. The assumptions may involve the learning rate, smoothness, convexity, or noise, and they may not hold exactly for deep neural networks.