Understand

Advanced · after gradient descent and evaluation

Deep Learning & Neural Networks

Beta This course is a beta version. Lessons, exercises and translations are still being reviewed and may change. Report anything that looks wrong with the feedback button.

You write the mechanisms before using the framework: reverse-mode differentiation, optimisers, batch normalisation, convolution, attention, checked in your browser against numerical gradients. Then PyTorch, on your machine, with the habits that make training debuggable. PyTorch doesn't run in this site's browser sandbox, so those chapters say plainly what must run locally.

Modules
10
Lessons
22
Time
~37 hours
Projects
2
Get started

What you'll be able to do

  • Implement reverse-mode automatic differentiation and verify it numerically
  • Implement optimisers, normalisation and regularisation, and explain their effects
  • Diagnose failed training from evidence
  • Write correct PyTorch training loops on CPU or GPU
  • Implement convolution and attention, and explain transformers from the mechanism up
  • Deliver a trained, evaluated and documented network

Who it's for

People who know gradient descent, overfitting and train/validation/test, for instance from Applied Machine Learning.

What you need

Python and NumPy; derivatives, matrix products and basic probability. PyTorch chapters need a local installation, which works on a CPU. A GPU helps, and is never required.

What you'll build

  • Digit Classifier and Failure Report
  • Neural Network Capstone

Curriculum

MechanismsAutodiff, networks and optimisers, by hand. 3 modules · 7 lessons
  1. 1 Tensors and Automatic DifferentiationBuild the engine underneath every framework.
  2. 2 Neural Networks from ScratchLogistic units, backpropagation and a gradient check, so that PyTorch holds no mysteries.
  3. 3 Optimisers and InitialisationWhy plain gradient descent is rarely enough.
Training in PracticeRegularisation, debugging and PyTorch. 3 modules · 4 lessons
  1. 4 Regularisation and NormalisationDropout and batch norm, including their train/eval difference.
  2. 5 Diagnosing Failed TrainingA method for when the loss won't fall.
  3. 6 PyTorch on Your MachineThe framework, now that you know what it does.
ArchitecturesConvolution, sequences, attention, transformers. 2 modules · 7 lessons
  1. 7 Convolutional NetworksLocal patterns, shared weights.
  2. 8 Sequences, Attention, TransformersFrom a recurrent step to the block inside modern language models.
DeliveryTransfer, evaluation, optimisation and capstone. 2 modules · 4 lessons
  1. 9 Smaller ModelsPruning and distillation, measured rather than assumed.
  2. 10 CapstoneA network of your own, on your own machine.