« The Elements of Differentiable Programming

TinyAD: Automatic Differentiation in Geometry Processing Made Simple »

Matrix Calculus for Machine Learning and Beyond

Course Description

We all know that calculus courses such as 18.01 Single Variable Calculus and 18.02 Multivariable Calculus cover univariate and vector calculus, respectively. Modern applications such as machine learning and large-scale optimization require the next big step, “matrix calculus” and calculus on arbitrary vector spaces.

This class covers a coherent approach to matrix calculus showing techniques that allow you to think of a matrix holistically (not just as an array of scalars), generalize and compute derivatives of important matrix factorizations and many other complicated-looking operations, and understand how differentiation formulas must be reimagined in large-scale computing. We will discuss reverse/adjoint/backpropagation differentiation, custom vector-Jacobian products, and how modern automatic differentiation is more computer science than calculus (it is neither symbolic formulas nor finite differences).

Course Link

Topics:

Here are some of the topics covered:

Derivatives as linear operators and linear approximation on arbitrary vector spaces: beyond gradients and Jacobians.
Derivatives of functions with matrix inputs and/or outputs (e.g. matrix inverses and determinants). Kronecker products and matrix “vectorization.”
Derivatives of matrix factorizations (e.g. eigenvalues/SVD) and derivatives with constraints (e.g. orthogonal matrices).
Multidimensional chain rules, and the significance of right-to-left (“forward”) vs. left-to-right (“reverse”) composition. Chain rules on computational graphs (e.g. neural networks).
Forward- and reverse-mode manual and automatic multivariate differentiation.
Adjoint methods (vJp/pullback rules) for derivatives of solutions of linear, nonlinear, and differential equations.
Application to nonlinear root-finding and optimization. Multidimensional Newton and steepest–descent methods.
Applications in engineering/scientific optimization and machine learning.
Second derivatives, Hessian matrices, quadratic approximations, and quasi-Newton methods.

math, optimization, Derivatives, automatic-differentiation, algorithmic-differentiation, ML, Alan-Edelman, Steven-Johnson, MIT, mit-opencourseware, 2023, course

Discussion

Real name:

E-Mail:

Website:

Enter your comment. Wiki syntax is allowed:

Please fill all the letters into the box to prove you're human.

Please keep this field empty:

Subscribe to comments

𝐶𝑎𝑟𝑎𝑚𝑖𝑧𝑎𝑟𝑢'𝑠 𝐵𝑙𝑜𝑔

Sidebar

Matrix Calculus for Machine Learning and Beyond

Discussion

𝐶𝑎𝑟𝑎𝑚𝑖𝑧𝑎𝑟𝑢'𝑠 𝐵𝑙𝑜𝑔

User Tools

Site Tools

Sidebar

Matrix Calculus for Machine Learning and Beyond

Discussion

Page Tools