Advanced machine learning - autumn 2021

Course information

This course covers advanced topics in machine learning. The course is intended for Master's students in physics and mathematics. This course is the follow-up of the course Machine Learning

Format: The course will be weekly sessions. Emphasis is on learning the material through mathematical derivation and computer exercises.

Course material: The course uses the following material:

Week Topic Material Exercises
1 45 Monte Carlo Methods (1)
Sampling means and variance, uniform sampling, sampling from multi variate Gaussian, importance sampling, rejection sampling
MK chapter 29.1-29.5
BRML chapter 27
BRML exercise 27.1. Box-Muller method (3)
MK exercise 29.3. Show diffusion scales as sqrt(T) (1)
MK exercise 29.13. Importance sampling of one Gaussian with another (only the computer simulations, reproducing figs 29.20) (3)
2 46 Monte Carlo Methods (2)
Markov processes, ergodicity, Metropolis Hasting algorithm, Gibbs sampling, Hamilton Monte Carlo
MK chapter 29.6
MK chapter 30.1, 30.3
handouts chapter 1
MK exercise 29.15. Gibbs sampling of posterior over mu, sigma given data (5). Hint: It is recommended to sample beta = 1/sigma^2 rather than sigma^2. But be aware that such a transformation affects the prior that you assume. For instance, if you assume a flat prior over sigma^2, this transforms to a non-flat prior over beta. For this exercise choose the prior over beta as 1/beta. This choice corresponds to a so-called non-informative prior that is flat in the log(sigma) domain. See also slides lecture 3 where we consider the variational approximation for this problem. (5)
MCMC exercises (10)
3 47 The Ising model
Phase transitions, critical slowing down, frustration, transfer matrix method
discrete optimization with simulated annealing.
MK chapter 31
Further reading: HKP appendix A,
Sandvik 2018 section 5
SoKal 1999: Critical slowing down
Aarts and Korst, Simulated Annealing and Boltzmann Machines 1989
MK exercise 31.1. Relation entropy and free energy (2)
Simulated annealing exercise on spin glass (10)
4 48 Deterministic approximate inference for Bayesian posterior
Laplace approximation
Variational approximation
MK chapter 33.1, 33.4, 33.5
Further reading: Barber Bishop 1998, Ensemble Learning in Bayesian Neural Networks
Consider again the perceptron learning problem of Mackay chapter 39 and 41, for which we computed the posterior by sampling in week 2. This time, compute p(t=1|x,D,alpha) using the Laplace approximation and reproduce Mackay figure 41.11b.(7)
5 49 Deterministic approximate inference for the Ising model
Mean field approximation
Linear response correction, TAP
SK model
Belief propagation
MK 33.2, 33.3, BRML chapter 28.7
Further reading: Kappen, Spanjers, Mean field theory for asymmetric neural networks (1999), Physical Review E, 61:56585663.
MF and BP in Ising model exercise (10)
6 50 Deterministic approximate inference for the Ising model
Convergence of BP, Factor graph version of BP, max-product BP, Applications of BP for compressed sensing and clustering

Further reading: Mooij, Kappen. Sufficient conditions for convergence of the sum-product algorithm (2007). IEEE Information Theory, 53:44224437
7 51 The statistical physics approach to machine learning
The replica symmetric solution for the SK model
The cavity method
Analysis of compressed sensing and random satisfiability using replica method and message passing algorithms

Further reading: Sherrington, D. and Kirkpatrick, S. (1975). Solvable model of Spin-Glass. Physical review letters, 35:17921796.
Kappen, H.J. An introduction to stochastic neural networks, in: Handbook of biological physics 2001, 517-552. pdf
Reproduce the phase plot of the SK model in the replica symmetric approximation replica_SK.pdf (7)
8 5 The Boltzmann Machine MK chapter 43
handouts chapter 1-2
Boltzmann Machine Learning (10)
Here are the
salamander retina data.
9 6 Student presentations
Dirk, Ilja, Luc.
Generative Adversarial Networks
10 7 Student presentations
Wenhao, Yangchu, Casper.
Autoregessive variational inference
11 8 Student presentations
Andrea, Santiago
Markov processes on speech recognition and biological sequences
Lauren, Caroline, Kasper
Recurrent neural networks for speech recognition
12 10 Student presentations
Steffen, Rao, Stijn
Dimension reduction methods
Niels, Tomas, Thalia
Gaussian processes
13 11 Student presentations
Dirren, Olivier, Matthijns
Dimension reduction methods
Maksim, Mats, Bruno
Model comparison
14 12 Student presentations
Shakeeb, Andre, Leo
Neural data analysis and anaestesia
Research overview
Quantum Machine Learning
Atomic BMs
Approximate quantum inference
Medical expert system (Promedas)
Disaster Victim Identification (Bonaparte)

There will be no final examination. The grade will be based entirely on the exercises and the student presentation of a research paper. You are expected to work in groups of 3 persons and you will be graded as a group. The final grade for each student is his group grade.

Each exercise counts for a number of points, indicated between brackets. The total number of points for the exercises is 68. For each exercise, hand in the code that can be run stand-alone. For the large (10 point) exercises, write a report:

All assignments of lectures 1-6 should be handed in before the end of January 2022. The assignments of lecture 7-9 should be handed in before end March 2022.

In addition to the exercises, your group should prepare a 45 minute presentation on a modern topic in machine learning. There are 8 suggestions for
student presentations but your group can also suggest another topic. The topic of your group presentation and preferred time slot (max 4 presentations per week, in weeks 5 to 10 in 2022) should be sent to me before the end of the year. The student presentation counts for 20 points.