Short course on control theory and dynamic programming - Madrid, October 2010

The course provides an introduction to stochastic optimal control theory. The course is in part based on a tutorial given by me and Marc Toussaint at ICML 2008 and on some selected material from the book Dynamic programming and optimal control by Dimitri Bertsekas.

Course information

Course material:

Date Topic Material Recommended exercises
1 Feb 8
11-13 hours
Discrete time control
dynamic programming
Bellman equation
Bertsekas 2-5, 13-14, 18, 21-32 Bertsekas 1.1 a and b, 1.2
2 Feb 9
11-13 hours
Continuous time control
Hamilton-Jacobi-Bellman Equation
Pontryagin Minimum Principle
Stochastic optimal control
Kappen ICML tutorial 1.2, 1.3, 1.4
extra exercise 1, 2a,b
Bertsekas 3.2
3 Feb 21
10-13 hours
Dual control: the problem of joint inference and control
Path integral control theory
Kappen ICML tutorial 1.5,1.6, 1.7
extra exercise 2c, 3
4 Feb 22
10-13 hours
Stochastic optimal control
Path integral control theory
Kappen ICML tutorial 1.7
extra exercise 4,5 Matlab code for n joint problem
Here is a directory of matlab files, which allows you to run and inspect the variational approximation for the n joint stochastic control problem as discussed in the tutorial text section 1.6.7. Type tar xvf njoints.tar to unpack the directory and simply run file1.m. In file1.m you can select demo1 (3 joint arm) or demo2 (10 joint arm). You can also try larger n but be sure to adjust eta for the smoothing of the variational fixed point equations. You can compare the results with exact cmputation (only recommendable for 2 joints) by setting METHOD='exact'. There is also an implementation of importance sampling (does not work very well) and Metropolis Hastings sampling (works nice, but not as stable as the variational approximation).