Arindam Banerjee

[intermediate/advanced] Deep Generative and Dynamical Models

Summary

Recent years have seen considerable advances in generative models, which learn distributions from data and also generate new data instances from the learned distribution; and dynamical models, which model systems with a dynamical or temporal component. Both of these developments have been leveraging advances in deep learning. The course will cover key advances in generative and dynamical models, including variational auto-encoders, normalizing flows, generative adversarial networks, neural differential equations, diffusion models, and scored based models.

Syllabus

• Background, including basics of probabilistic models
• Overview of Markov Chain Monte Carlo (MCMC) and Variational Inference (VI)
• Variational auto-encoders
• Normalizing flows
• Generative Adversarial Networks
• Neural differential equations
• Diffusion models, Score based models

References

A. Banerjee, Deep Generative and Dynamical Models, Course: CS 598, UIUC, Fall, 2021.

D. Kingma, M. Welling, Auto-Encoding Variational Bayes, ICLR, 2014.

D. Kingma, M. Welling, An Introduction to Variational Autoencoders, FTML 2019.

D. Kingma, T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, M. Welling. Improved Variational Inference with Inverse Autoregressive Flow, NeurIPS, 2016.

C. Sonderby, T. Raiko, L. Maaloe, S. Sonderby, O. Winther. Ladder Variational Autoencoders, NeurIPS, 2016.

R. Child. Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images, ICLR, 2021.

L. Dinh, D. Krueger, Y. Bengio. NICE: Non-linear Independent Components Estimation, ICLR Workshop, 2015.

D. Rezende, S. Mohamed, Variational Inference with Normalizing Flows, ICML, 2015.

L. Dinh, J. Sohl-Dickstein, S. Bengio. Density Estimation using Real NVP, ICLR, 2017.

D. P. Kingma, P. Dhariwal. Glow: Generative flow with invertible 1×1 convolutions, NeurIPS, 2018.

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative Adversarial Networks, NeurIPS, 2014.

M. Arjovsky, S. Chintala, L. Bottou. Wasserstein GAN, ICML, 2017.

I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, A. C. Courville. Improved training of Wasserstein GANs, NeurIPS, 2017.

A. Radford, L. Metz, S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.

J. Zhu, T. Park, P. Isola, A. Efros. Unpaired image-to image translation using cycle-consistent adversarial networks, ICCV, 2017.

T. Karras, S. Laine, T. Aila, A Style-Based Generator Architecture for Generative Adversarial Networks, CVPR, 2019.

R. Chen, Y. Rubanova, J. Bettencourt, D. Duvenaud. Neural ordinary differential equations, NeurIPS, 2018.

C. Finlay, J. Jacobsen, L. Nurbekyan, A. Oberman. How to train your neural ODE, ICML, 2020.

J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics, ICML, 2015.

J. Ho, A. Jain, P. Abbeel. Denoising diffusion probabilistic models, NeurIPS, 2020.

A. Hyvarinen. Estimation of non-normalized statistical models by score matching, JMLR, 2005.

Y. Song, S. Ermon. Generative modeling by estimating gradients of the data distribution, NeurIPS, 2019.

Pre-requisites

Linear algebra, multi-variate calculus – advanced undergrad level. Probability and statistics – advanced undergrad level. Basics of optimization, e.g. gradient descent. Basic deep learning models, such as feedforward networks, CNNs, RNN.

Short bio

Arindam Banerjee is a Founder Professor at the Department of Computer Science, University of Illinois Urbana-Champaign. His research interests are in machine learning and data mining, especially on problems involving geometry and randomness. His current research focuses on computational and statistical aspects of deep learning, generative and dynamical models, spatial and temporal data analysis, and sequential decision making problems. His work also focuses on applications in complex real-world problems in different areas including climate science, ecology, recommendation systems, and finance, among others. He has won several awards, including the NSF CAREER award (2010), the IBM Faculty Award (2013), and six best paper awards in top-tier venues.