Chen Change Loy

[intermediate/advanced] Harnessing Prior for Content Enhancement and Creation

Summary

The lectures will provide an in-depth overview of using deep learning techniques for image and video restoration tasks. We will begin with an introduction to the fundamental concepts and problem formulation.

We will then discuss popular deep network architectures in image restoration. Specifically, we will cover classic networks such as SRCNN, VDSR, and ESRGAN. We will learn the applications of deep generative prior for versatile image restoration and manipulation. The discussion will extend to recent techniques based on variational autoencoders and diffusion models, such as CodeFormer for robust blind face restoration, StableSR for image super-resolution, and Upscale-A-Video for video super-resolution.

Throughout the lecture, we will provide real-world examples and case studies to demonstrate the practical applications of these techniques. Attendees will gain a thorough understanding of deep learning-based approaches for image and video restoration, including the underlying principles, recent research advancements, and potential future directions.

Syllabus

• Problem formulation
• Basic deep network architectures
• Training losses
• The use of generative adversarial networks
• Conditional super-resolution
• Face restoration
• Generative priors

References

Efficient Diffusion Model for Image Restoration by Residual Shifting
Z. Yue, J. Wang, C. C. Loy
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024 (TPAMI)

DifFace: Blind Face Restoration with Diffused Error Contraction
Z. Yue, C. C. Loy
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024 (TPAMI)

Kalman-Inspired Feature Propagation for Video Face Super-Resolution
R. Feng, C. Li, C. C. Loy
European Conference on Computer Vision, 2024 (ECCV)

Exploiting Diffusion Prior for Real-World Image Super-Resolution
J. Wang, Z. Yue, S. Zhou, K. C. K. Chan, C. C. Loy
International Journal of Computer Vision, 2024 (IJCV)

Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution
S. Zhou, P. Yang, J. Wang, Y. Luo, C. C. Loy
in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2024 (CVPR, Highlight)

PGDiff: Guiding Diffusion Models for Versatile Face Restoration via Partial Guidance
P. Yang, S. Zhou, Q. Tao, C. C. Loy
in Proceedings of Neural Information Processing Systems, 2023 (NeurIPS)

ResShift: Efficient Diffusion Model for Image Super-resolution by Residual Shifting
Z. Yue, J. Wang, C. C. Loy
in Proceedings of Neural Information Processing Systems, 2023 (NeurIPS, Spotlight)

Learning Generative Structure Prior for Blind Text Image Super-resolution
X. Li, W. Zuo, C. C. Loy
in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2023 (CVPR)

Towards Robust Blind Face Restoration with Codebook Lookup Transformer
S. Zhou, K. C. K. Chan, C. Li, C. C. Loy
in Proceedings of Neural Information Processing Systems, 2022 (NeurIPS)

GLEAN: Generative Latent Bank for Image Super-Resolution and Beyond
K. C. K. Chan, X. Wang, X. Xu, J. Gu, C. C. Loy
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022 (TPAMI)

BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment
K. C. K. Chan, S. Zhou, X. Xu, C. C. Loy
in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2022 (CVPR)

Exploiting Deep Generative Prior for Versatile Image Restoration and Manipulation
X. Pan, X. Zhan, B. Dai, D. Lin, C. C. Loy, P. Luo
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021 (TPAMI)

BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond
K. C. K. Chan, X. Wang, K. Yu, C. Dong, C. C. Loy
in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2021 (CVPR)

ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks
X. Wang, K. Yu, S. Wu, J. Gu, Y. Liu, C. Dong, Y. Qiao, C. C. Loy
in Workshop Proceedings of European Conference on Computer Vision, 2018 (ECCVW)

Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform
X. Wang, K. Yu, C. Dong, C. C. Loy
in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2018 (CVPR)

Accelerating the Super-Resolution Convolutional Neural Network
C. Dong, C. C. Loy, X. Tang
in Proceedings of European Conference on Computer Vision, 2016 (ECCV)

Learning a Deep Convolutional Network for Image Super-Resolution
C. Dong, C. C. Loy, K. He, X. Tang
in Proceedings of European Conference on Computer Vision, pp. 184-199, 2014 (ECCV)

Pre-requisites

Basic machine learning and computer vision knowledge. Elementary concepts of linear algebra. Some knowledge about deep networks is a plus.

Short bio

Chen Change Loy is a President’s Chair Professor with the College of Computing and Data Science, Nanyang Technological University, Singapore. He is the Lab Director of MMLab@NTU and Co-associate Director of S-Lab. He received his Ph.D. (2010) in Computer Science from Queen Mary University of London. Prior to joining NTU, he served as a Research Assistant Professor at the MMLab of The Chinese University of Hong Kong, from 2013 to 2018. His research interests include computer vision and deep learning with a focus on image/video restoration and enhancement, generative tasks, and representation learning. His accolades include the CCF-CV Test of Time Award, Nanyang Research Award, and the Singapore Open Research Award. He served/serves as an Associate Editor of the International Journal of Computer Vision (IJCV), IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) and Computer Vision and Image Understanding (CVIU). He also serves/served as an Area Chair of top conferences such as ICCV, CVPR, ECCV, ICLR, and NeurIPS. He will serve as the Program Co-Chair of CVPR 2026.