DeepLearn 2024
11th International School on Deep Learning
(and the Future of Artificial Intelligence)
Porto - Maia, Portugal · July 15-19, 2024
Registration
Downloads
  • Call DeepLearn 2024
  • Poster DeepLearn 2024
  • Lecture Materials
  • Home
  • Lecturers
  • Schedule
  • Sponsoring
  • News
  • Info
    • Travel
    • Accommodation
    • UMaia / UP students and staff
    • Visa
    • Code of conduct
    • Testimonials
  • Home
  • Lecturers
  • Schedule
  • Sponsoring
  • News
  • Info
    • Travel
    • Accommodation
    • UMaia / UP students and staff
    • Visa
    • Code of conduct
    • Testimonials
Frank Hutter

Frank Hutter

University of Freiburg

[intermediate/advanced] Automated Machine Learning, or Deep Learning 2.0: AI that Builds and Improves AI

Summary

Throughout the history of AI, there is a clear pattern that manual elements of AI methods are eventually replaced by better-performing automatically-found ones; for example, deep learning (DL) replaced manual feature engineering with learned representations. The logical next step in representation learning is to also (meta-)learn the best architectures for these representations, as well as the best algorithms & hyperparameters for learning them. In this short course, I will discuss various works towards this aim from the area of AutoML — AI that builds and improves AI. Specifically, I will show various advances demonstrating that AutoML can be efficient and arguing for an emphasis on multi-objective AutoML to also account for the various dimensions of trustworthiness (such as algorithmic fairness, robustness, and uncertainty calibration). The first lecture will focus on basics and the second lecture on speedup techniques. Finally, in the third lecture, I will focus on AI that builds AI, with a deep-dive into a novel approach that learns an entire classification algorithm for small tabular datasets that achieves a new state of the art at the cost of a single forward pass. Throughout, I will illustrate the methods with Colab notebook demos.

Syllabus

Lecture 1: The Basics

  • The Big Picture
  • Hyperparameter Optimization (HPO)
  • Bayesian optimization for HPO + demo
  • Multi-objective HPO

Lecture 2: Advanced HPO

  • Speedup Techniques for HPO + demo
  • AutoML for Foundation Models: QuickTune + demo

Lecture 3: Metalearning entire algorithms

  • Prior-Fitted Networks (PFNs)
  • TabPFN v1
  • TabPFN v2 + demo

References

Deep Learning 2.0:

https://www.automl.org/deep-learning-2-0-extending-the-power-of-deep-learning-to-the-meta-level/

HPO:

  • Survey: https://www.automl.org/wp-content/uploads/2019/05/AutoML_Book_Chapter1.pdf
  • PriorBand: https://openreview.net/forum?id=uoiwugtpCH
  • ifBO: https://openreview.net/forum?id=VyoY3Wh9Wd

QuickTune:

https://openreview.net/forum?id=tqh1zdXIra

PFNs:

  • https://openreview.net/forum?id=KSugKcbNf9
  • https://openreview.net/forum?id=eu9fVjVasr4

Pre-requisites

Basic knowledge of deep learning.

Short bio

Frank Hutter is a Full Professor for Machine Learning at the University of Freiburg (Germany), currently on leave as a Hector Endowed Fellow at the ELLIS Institute Tübingen. Before taking up a faculty job at the University of Freiburg in 2013, he did a PhD (2004-2009) and postdoc (2009-2013) at the University of British Columbia (UBC) in Canada. He received the 2010 CAIAC doctoral dissertation award for the best thesis in AI in Canada, as well as several best paper awards and prizes in international ML competitions. He is a Fellow of ELLIS and EurAI, Director of the ELLIS unit Freiburg, and the recipient of 3 ERC grants. Frank is best known for his research on automated machine learning (AutoML), including neural architecture search, efficient hyperparameter optimization, and meta-learning. He co-authored the first book on AutoML and the prominent AutoML tools Auto-WEKA, Auto-sklearn and Auto-PyTorch, won the first two AutoML challenges with his team, is co-teaching the first MOOC on AutoML, co-organized 15 AutoML-related workshops at ICML, NeurIPS and ICLR, and founded the AutoML conference as general chair in 2022 and 2023. In recent years, his focus has been on the intersection of foundation models and AutoML, including the first foundation model for tabular data, TabPFN, and improving pretraining and fine-tuning with AutoML.

Other Courses

deeeplearn-speakers-hanJiawei Han
Katia SycaraKatia Sycara
deeeplearn-speakers-beniniLuca Benini
Gustau Camps-VallsGustau Camps-Valls
Nitesh ChawlaNitesh Chawla
Daniel CremersDaniel Cremers
deeeplearn-speakers-cuiPeng Cui
deeeplearn-speakers-gleyzerSergei V. Gleyzer
deeeplearn-speakers-heYulan He
deeeplearn-speakers-karypisGeorge Karypis
deeeplearn-speakers-neyHermann Ney
Massimiliano PontilMassimiliano Pontil
Elisa RicciElisa Ricci
Wojciech SamekWojciech Samek
Xinghua Mindy ShiXinghua Mindy Shi
Michalis VazirgiannisMichalis Vazirgiannis
James ZouJames Zou

DeepLearn 2023 Summer

CO-ORGANIZERS

University of Maia

Institute for Research Development, Training and Advice – IRDTA, Brussels/London

Active links
    Past links
    • DeepLearn 2023 Summer
    • DeepLearn 2023 Spring
    • DeepLearn 2023 Winter
    • DeepLearn 2022 Autumn
    • DeepLearn 2022 Summer
    • DeepLearn 2022 Spring
    • DeepLearn 2021 Summer
    • DeepLearn 2019
    • DeepLearn 2018
    • DeepLearn 2017
    © IRDTA 2023. All Rights Reserved.
    We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept All”, you consent to the use of ALL the cookies. However, you may visit "Cookie Settings" to provide a controlled consent.
    Cookie SettingsAccept All
    Manage consent

    Privacy Overview

    This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
    Necessary
    Always Enabled
    Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
    CookieDurationDescription
    cookielawinfo-checkbox-advertisement1 yearThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Advertisement".
    cookielawinfo-checkbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
    cookielawinfo-checkbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
    cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
    cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
    cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
    PHPSESSIDsessionThis cookie is native to PHP applications. The cookie is used to store and identify a users' unique session ID for the purpose of managing user session on the website. The cookie is a session cookies and is deleted when all the browser windows are closed.
    viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
    Functional
    Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
    Performance
    Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
    Analytics
    Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
    CookieDurationDescription
    _ga2 yearsThis cookie is installed by Google Analytics. The cookie is used to calculate visitor, session, campaign data and keep track of site usage for the site's analytics report. The cookies store information anonymously and assign a randomly generated number to identify unique visitors.
    _gat_gtag_UA_74880351_91 minuteThis cookie is set by Google and is used to distinguish users.
    _gid1 dayThis cookie is installed by Google Analytics. The cookie is used to store information of how visitors use a website and helps in creating an analytics report of how the website is doing. The data collected including the number visitors, the source where they have come from, and the pages visted in an anonymous form.
    Advertisement
    Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
    Others
    Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.
    SAVE & ACCEPT
    Powered by CookieYes Logo