DeepLearn 2023 Winter
8th International School
on Deep Learning
Bournemouth, UK · January 16-20, 2023
Registration
Downloads
  • Call DeepLearn 2023 Winter
  • Poster DeepLearn 2023 Winter
  • Lecture Materials
  • Home
  • Schedule
  • Lecturers
  • News
  • Accommodation
  • Info
    • Travel from London to Bournemouth
    • Sponsoring
    • Code of conduct
    • Visa
    • Testimonials
  • Home
  • Schedule
  • Lecturers
  • News
  • Accommodation
  • Info
    • Travel from London to Bournemouth
    • Sponsoring
    • Code of conduct
    • Visa
    • Testimonials
deeplearn-speaker-david-mcallerster

David McAllester

Toyota Technological Institute at Chicago

[intermediate/advanced] Information Theory for Deep Learning

Summary

Information theory is central to deep learning. Most fundamental is the cross entropy loss used in training classifiers. But generative adversarial networks (GANS) and variational autoencoders (VAES) are also defined by information-theoretic loss functions. Recently methods based contrastive predictive coding (CPC), motivated by the maximization of mutual information, has proved extremely effective in self-supervised training of image features. Even more recently more direct maximization of mutual information has proved more effective. This course will explore information theory form the perspective of current deep learning architectures with an emphasis on recent empirical results.

Syllabus

  • The definition of information and Shanon’s source coding theorem.
  • Generative Adversarial Networks (GANS) and Variational Auto Encoders (VAEs) from an information theoretic perspective.
  • Perils of differential entropy and the rise of discrete VAEs.
  • Mutual Information and self-supervised learning.
  • Formal limitations on the measurement of mutual information.

References

  • Cover and Thomas, Elements of Information Theory, Wiley Press
  • Goodfellow et al., Generative Adversarial Nets, NeurIPS, 2014
  • Kingma and Welling, Auto-encoding Variational Bayes, arXiv:1312.6114, 2013
  • Oord and Viyals, Representation Learning with Contrastive Predictive Coding, arXiv:1807.03748, 2018
  • Poole et al, On Variational Bounds of Mutual Information, arXiv:1905.06922, 2019
  • McAllester and Stratos, Formal Limitations on the Measurement of Mutual Information, AISTATS 2020

Pre-requisites

Vector Calculus. Familiarity with convex functions and Jensen’s inequality.

Short bio

David A. McAllester is Professor and former chief academic officer at the TTIC (the Toyota Technological Institute at Chicago). He received his B.S., M.S. and Ph.D. degrees from the Massachusetts Institute of Technology and has served on the faculties of Cornell and MIT. He was a member of technical staff at AT&T Labs-Research from 1995 to 2002 and has been a fellow of the American Association of Artificial Intelligence since 1997. He has written over 100 refereed publications. McAllester’s research areas include machine learning theory, the theory of programming languages, automated reasoning, AI planning, computer game playing (computer chess) and computational linguistics. A 1991 paper on AI planning proved to be one of the most influential papers of the decade in that area. A 1993 paper on computer game algorithms[6] influenced the design of the algorithms used in the Deep Blue chess system that defeated Garry Kasparov. A 1998 paper on machine learning theory introduced PAC-Bayesian theorems which combine Bayesian and non-Bayesian methods. He was a co-author of the deformable part model that dominated object detection in computer vision from 2008 to 2012. He has been teaching fundamentals of deep learning since 2016.

Other Courses

deeplearn-speaker-yi-maYi Ma
Daphna WeinshallDaphna Weinshall
Eric P. XingEric P. Xing
Matias Carrasco KindMatias Carrasco Kind
deeplearn-speaker-nitesh-chawlaNitesh Chawla
Sumit ChopraSumit Chopra
Luc De RaedtLuc De Raedt
Marco DuarteMarco Duarte
Joao GamaJoão Gama
Claus HornClaus Horn
Zhiting Hu & Eric P. XingZhiting Hu & Eric P. Xing
deeplearn-speaker-nathalie-japkowiczNathalie Japkowicz
deeplearn-speaker-gregor-kasieczkaGregor Kasieczka
Karen LivescuKaren Livescu
deeplearn-speaker-dhabaleswar-k-pandaDhabaleswar K. Panda
Fabio RoliFabio Roli
Bracha ShapiraBracha Shapira
deeplearn-speaker-kunal-tawarKunal Talwar
Tinne TuytelaarsTinne Tuytelaars
deeplearn-speaker-lyle-ungarLyle Ungar
speakers-bram-van-ginnekenBram van Ginneken
deeplearn-speaker-yudong-zhangYu-Dong Zhang

DeepLearn 2022 Winter

CO-ORGANIZERS

Bournemouth University
Department of Computing and Informatics

Universitat Rovira i Virgili, Tarragona

Institute for Research Development, Training and Advice – IRDTA, Brussels/London

Active links
  • DeepLearn 2023 Summer – 10th International Gran Canaria School on Deep Learning
  • BigDat 2023 Summer – 7th International School on Big Data
  • DeepLearn 2023 Spring – 9th International School on Deep Learning
Past links
  • DeepLearn 2022 Autumn
  • DeepLearn 2022 Summer
  • DeepLearn 2022 Spring
  • DeepLearn 2021 Summer
  • DeepLearn 2019
  • DeepLearn 2018
  • DeepLearn 2017
© IRDTA 2022. All Rights Reserved.
We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept All”, you consent to the use of ALL the cookies. However, you may visit "Cookie Settings" to provide a controlled consent.
Cookie SettingsAccept All
Manage consent

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
CookieDurationDescription
cookielawinfo-checkbox-advertisement1 yearThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Advertisement".
cookielawinfo-checkbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
PHPSESSIDsessionThis cookie is native to PHP applications. The cookie is used to store and identify a users' unique session ID for the purpose of managing user session on the website. The cookie is a session cookies and is deleted when all the browser windows are closed.
viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
Functional
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
Performance
Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
Analytics
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
CookieDurationDescription
_ga2 yearsThis cookie is installed by Google Analytics. The cookie is used to calculate visitor, session, campaign data and keep track of site usage for the site's analytics report. The cookies store information anonymously and assign a randomly generated number to identify unique visitors.
_gat_gtag_UA_74880351_91 minuteThis cookie is set by Google and is used to distinguish users.
_gid1 dayThis cookie is installed by Google Analytics. The cookie is used to store information of how visitors use a website and helps in creating an analytics report of how the website is doing. The data collected including the number visitors, the source where they have come from, and the pages visted in an anonymous form.
Advertisement
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
Others
Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.
SAVE & ACCEPT
Powered by CookieYes Logo