This course covers advanced topics in machine learning. The course is intended for Master's students in physics and mathematics. This course is the followup of the course Machine Learning
Format: The course will be weekly sessions. Emphasis is on learning the material through mathematical derivation and computer exercises.
Course material: The course uses the following material:
Week  Topic  Material  Exercises  
1  45  Monte Carlo Methods (1) Sampling means and variance, uniform sampling, sampling from multi variate Gaussian, importance sampling, rejection sampling 
MK chapter 29.129.5 BRML chapter 27 
BRML exercise 27.1. BoxMuller method (3) MK exercise 29.3. Show diffusion scales as sqrt(T) (1) MK exercise 29.13. Importance sampling of one Gaussian with another (only the computer simulations, reproducing figs 29.20) (3) 
2  46 
Monte Carlo Methods (2) Markov processes, ergodicity, Metropolis Hasting algorithm, Gibbs sampling, Hamilton Monte Carlo 
MK chapter 29.6 MK chapter 30.1, 30.3 handouts chapter 1 
MK exercise 29.15. Gibbs sampling of posterior over mu, sigma given data (5).
Hint: It is recommended to sample beta = 1/sigma^2 rather than
sigma^2. But be aware that such a transformation affects the prior that you assume. For instance,
if you assume a flat prior over sigma^2, this transforms to a nonflat prior over beta. For this
exercise choose the prior over beta as 1/beta. This choice corresponds to a socalled
noninformative prior that is flat in the log(sigma) domain. See also slides lecture 3 where we
consider the variational approximation for this problem. (5)
MCMC exercises (10) 
47  No lecture  
3  48  The Ising model Phase transitions, critical slowing down, frustration, transfer matrix method discrete optimization with simulated annealing. 
MK chapter 31 Further reading: HKP appendix A, Sandvik 2018 section 5 SoKal 1999: Critical slowing down Aarts and Korst, Simulated Annealing and Boltzmann Machines 1989 
MK exercise 31.1. Relation entropy and free energy (2) Simulated annealing exercise on spin glass (10) 
4  49  Deterministic approximate inference for Bayesian posterior Laplace approximation Variational approximation 
MK chapter 33.1, 33.4, 33.5 Further reading: Barber Bishop 1998, Ensemble Learning in Bayesian Neural Networks 
Consider again the perceptron learning problem of Mackay chapter 39 and 41,
for which we computed the posterior by
sampling in week 2. This time, compute p(t=1x,D,alpha) using the Laplace approximation and
reproduce Mackay figure 41.11b.(7) 
5  50  Deterministic approximate inference for the Ising model Mean field approximation Linear response correction, TAP SK model Belief propagation 
MK 33.2, 33.3, BRML chapter 28.7 Further reading: Kappen, Spanjers, Mean field theory for asymmetric neural networks (1999), Physical Review E, 61:56585663. 
MF and BP in Ising model exercise (10) 
6  51  Deterministic approximate inference for the Ising model Convergence of BP, Factor graph version of BP, maxproduct BP, Applications of BP for compressed sensing and clustering 
Further reading: Mooij, Kappen. Sufficient conditions for convergence of the sumproduct algorithm (2007). IEEE Information Theory, 53:44224437 

7  5  The statistical physics approach to machine learning The replica symmetric solution for the SK model The cavity method Analysis of compressed sensing and random satisfiability using replica method and message passing algorithms 
Further reading: Sherrington, D. and Kirkpatrick, S. (1975). Solvable model of SpinGlass. Physical review letters, 35:17921796. Kappen, H.J. An introduction to stochastic neural networks, in: Handbook of biological physics 2001, 517552. pdf 
Reproduce the phase plot of the SK model in the replica symmetric approximation replica_SK.pdf (7) 
8  6  The Boltzmann Machine Quantum machine learning, Quantum Boltzmann machine  MK chapter 43 handouts chapter 12 
Boltzmann Machine Learning (10) Here are the salamander retina data. 
7  No class  
9  8  Transformers  Modern AI is rapidly changing society. Write a 12 page essay
on what you think will be the implications of AI for our future. Your essay
should express your personal point of view and based on rational
arguments. Substantiate
your views by reviewing existing
literature, where you contrast different views. You may consider
the following questions:
PS1: This assignment is in principle a group effort, ie. one essay per group. But if you have diverging views, you may also hand in your personal essay. PS2: Please do not send me ChatGPT generated documents, because those I can generate myself. They will be discarted. (15) 

10  9  Student presentations Marko Vinklbilobrk. Sparse regression Lisa Seijger, Max Eberhard and Jeroen van der Poel. Replicated Monte Carlo  
11  10  Student presentations Jasper Pieterse, Daria Mihaila, Benedetta, Felici. Emulating small quantum systems using pbits. Lefteris Papadopoulos, Wicher van Bree, Haoze Zhu. Nonnegative matrix factorization  
12  11  Student presentations Brandon ChinALien, Felix Dörr, Inga Schöyen. Barren plateaus in quantum circuits. Ruben Versteeg, Bas Drost en Onur Ozdemir. Hidden Markov models 
Examination:
There will be no final examination. The grade will be based entirely on the
exercises and the student presentation of a research paper.
You are expected to work in groups of 3 persons and you will be graded as a group.
The final grade for each student is his group grade.