Chen Change Loy

[introductory/intermediate] Image and Video Restoration

Summary

The lecture will provide an in-depth overview of using deep learning techniques for image and video restoration tasks. We will begin with an introduction to the fundamental concepts and problem formulation.

We will then discuss popular deep network architectures in image restoration. Specifically, we will cover classic networks such as SRCNN, VDSR, SwinIR and ESRGAN. We will also examine video restoration architectures, including BasicVSR and BasicVSR++ for improved video super-resolution. We will learn the applications of deep generative prior for versatile image restoration and manipulation. The discussion will extend to recent techniques such as GLEAN for generative latent bank for image super-resolution and CodeFormer for robust blind face restoration.

Throughout the lecture, we will provide real-world examples and case studies to demonstrate the practical applications of these techniques. Attendees will gain a thorough understanding of deep learning-based approaches for image and video restoration, including the underlying principles, recent research advancements, and potential future directions.

Syllabus

• Problem formulation
• Basic deep network architectures
• Training losses
• The use of generative adversarial networks
• Conditional super-resolution
• Generative priors
• Face restoration

References

Embedding Fourier for Ultra-High-Definition Low-Light Image Enhancement
C. Li, C. Guo, M. Zhou, Z. Liang, S. Zhou, R. Feng, C. C. Loy
International Conference on Learning Representations, 2023 (ICLR, Oral).

Towards Robust Blind Face Restoration with Codebook Lookup Transformer
S. Zhou, K. C. K. Chan, C. Li, C. C. Loy
in Proceedings of Neural Information Processing Systems, 2022 (NeurIPS).

GLEAN: Generative Latent Bank for Image Super-Resolution and Beyond
K. C. K. Chan, X. Wang, X. Xu, J. Gu, C. C. Loy
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022 (TPAMI).

LEDNet: Joint Low-light Enhancement and Deblurring in the Dark
S. Zhou, C. Li, C. C. Loy
European Conference on Computer Vision, 2022 (ECCV).

BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment
K. C. K. Chan, S. Zhou, X. Xu, C. C. Loy
in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR).

Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation
X. Pan, X. Zhan, B. Dai, D. Lin, C. C. Loy, P. Luo
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021 (TPAMI).

BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond
K. C. K. Chan, X. Wang, K. Yu, C. Dong, C. C. Loy
in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2021 (CVPR).

ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks
X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, C. C. Loy
in Workshop Proceedings of European Conference on Computer Vision, 2018 (ECCVW).

Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform
X. Wang, K. Yu, C. Dong, C. C. Loy
in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2018 (CVPR).

Accelerating the Super-Resolution Convolutional Neural Network
C. Dong, C. C. Loy, X. Tang
in Proceedings of European Conference on Computer Vision, 2016 (ECCV).

Learning a Deep Convolutional Network for Image Super-Resolution
C. Dong, C. C. Loy, K. He, X. Tang
in Proceedings of European Conference on Computer Vision, pp 184-199, 2014 (ECCV).

Pre-requisites

Basic machine learning and computer vision knowledge. Elementary concepts of linear algebra. Some knowledge about deep networks is a plus.

Short bio

Chen Change Loy is a Nanyang Associate Professor with the School of Computer Science and Engineering, Nanyang Technological University (NTU), Singapore. He serves as the Co-Associate Director for S-Lab, NTU and Director of MMLab@NTU. He is also an Adjunct Associate Professor at the Chinese University of Hong Kong. He received his PhD (2010) in Computer Science from the Queen Mary University of London. Before joining NTU, he served as a Research Assistant Professor at the MMLab of the Chinese University of Hong Kong, from 2013 to 2018. He is the recipient of 2019 Nanyang Associate Professorship (Early Career Award) from Nanyang Technological University. He is recognized as one of the 100 most influential scholars in computer vision by AMiner. His research interests include computer vision and deep learning with a focus on image/video restoration, enhancement, and content creation. He serves as an Associate Editor of the IJCV and TPAMI. He also serves/served as the Area Chair of top conferences such as CVPR, ICCV, and ECCV. He is a senior member of IEEE.