A. Murat Tekalp

[intermediate/advanced] Deep Learning for Image/Video Restoration and Compression

Summary

Deep learning has made a significant impact not only on computer vision and natural language processing but also on classical signal processing problems such as image/video restoration/SR and compression. As a result, learning-based methods have emerged as a promising nonlinear signal-processing framework and the state of the art in image/video restoration, SR, and compression is getting redefined. Performance of learned restoration and compression methods are affected by architectures, loss functions, and training data/procedures. This short course presents the state-of-the-art deep network architectures and training methods for image/video restoration/SR and compression, including advances in learned models for temporal correlations/motion in video.

Recent advances in architectures and training methods resulted in significant improvements in the performance of learned image/video restoration and SR compared to traditional model-based methods. We can consider learned image restoration/SR as learning either a regression mapping from the space of degraded images to ideal images based on the universal approximation theorem, or a generative model that captures the probability distribution of ideal images. An important benefit of data-driven deep learning approach is that neural models can be optimized for any differentiable loss function, including visual perceptual loss functions, leading to perceptual video restoration and SR, which cannot be easily handled by traditional model-based methods. I will discuss loss functions and evaluation criteria for image/video restoration/SR, including fidelity and perceptual criteria, and the relation between them via perception vs. fidelity (distortion) trade-off. I will also discuss the generalization issue, namely, problems with overfitting image priors and overfitting the degradation model when applying supervised training using synthetically generated degraded (LR) and ground-truth (GT) image/video pairs to real-life restoration/SR, and some possible ways to deal with these problems.

In learned image/video compression, end-to-end optimized variational auto-encoders are replacing transforms such as discrete cosine transform and wavelet transform as learned image-dependent transforms or generative codecs that are trained to synthesize decoded images without artefacts. Recent work in learned image/video compression focus on jointly learning variational auto-encoders with models of the latent variable probability distributions. Adding side information (“hyper priors”) to allow the probability models adapt to local properties of images/video resulted in learned models that outperform traditional image codecs (e.g., BPG) and video codecs (e.g., x265, x266, SVT-HEVC) in both PSNR and MS-SSIM. I will discuss end-to-end optimization of state-of-the-art learned image compression models for PSNR and visual criteria. I will also discuss architectures and training methods for end-to-end optimized learned low-delay and random-access video codecs and compare their performances with those of the traditional video codecs.

Syllabus

This short course is organized into three parts:

PART 1: Introduction to Learned Image Restoration and Superresolution (90 mins)

Introduce ill-posed problems and limitations of traditional model-based solutions. Discuss the state-of-the-art neural network architectures and learned models for image/video restoration and super-resolution (SR), and evaluation of the results of these models. I will cover:

• Definition of ill-posed problems and traditional vs. data-driven regularization strategies
• Convolutional networks (ConvNets), residual and dense ConvNet architectures
• Successful image restoration/SR models, such as EDSR, RCAN, and RRDB
• Adversarial loss, perceptual image quality measures, and perception-distortion trade-off

PART 2: Learned Image/Video Restoration and Superresolution (90 mins)

Discuss the ability of image restoration/SR models to generalize to real-world applications. Discuss different video restoration problems, the state-of-the-art network architectures and learned models for video restoration and super-resolution (SR). I will cover:

• Generalization of ConvNets to real-world image restoration/SR applications
• Video restoration/SR models, such as EDVR, video deinterlacing and frame interpolation
• Should you prefer CNN or RNN architecture for video processing?
• Perceptual video restoration/SR
• Discussion of results of NTIRE 2017/2018/2019/2020 and AIM 2019/2020 challenges

PART 3: Learned Image/Video Compression (90 mins)

Discuss learned models as image/video codec optimization tools, such as pre- and/or post-processing filters. Discuss end-to-end optimized image/video compression. Recent research indicates that learned transforms models can be highly effective for uni-directional (low-delay) and bi-directional (random access) video compression. I will cover:

• Learned image/video compression tools vs. end-to-end optimized solutions
• Variational auto-encoders, learned probability models and hyper-priors for latents
• Perceptual image/video compression and Perception-Rate-Distortion optimization
• Uni-directional and bi-directional end-to-end learned video compression approaches
• Can learned frame prediction compete with traditional motion-compensation?
• Discussion of results of Challenge on Learned Image Compression CLIC 2018/2019/2020

References

• Special issue of IEEE Journal on Special Topics in Signal Processing on Deep Learning for Image/Video Restoration and Compression, Guest editors: A. M. Tekalp (lead GE), M. Covell, R. Timofte, C. Dong, vol. 15, no. 2, February 2021.
• O. Keleş, A. M. Tekalp, J. Malik, and S. Kıranyaz, “Self-organized residual blocks for image super-resolution,” IEEE Int. Conf. on Image Processing, Anchorage, Alaska, USA, Sept. 2021.
• O. Keleş, M. A. Yılmaz, H. Güven, A. M. Tekalp, J. Malik, and S. Kıranyaz, “Self-organized variational autoencoders (Self-VAE) for learned image compression,” IEEE Int. Conf. on Image Processing, Anchorage, Alaska, USA, Sept. 2021.
• M. A. Yılmaz and A. M. Tekalp, “DFPN: Deformable Frame Prediction Network,” IEEE Int. Conf. on Image Processing, Anchorage, Alaska, USA, Sept. 2021.
• C. Korkmaz, A. M. Tekalp and Z. Dogan, “Two-stage domain adapted training for better generalization in real-world image restoration and super-resolution,” IEEE Int. Conf. on Image Processing, Anchorage, Alaska, USA, Sept. 2021.
• S. Sulun and A. M. Tekalp, “Can learned frame-prediction compete with block-motion compensation for video coding?” Signal, Image and Video Processing (SIViP), Springer, vol. 15, pp. 401-410, 2021.
• O. Kırmemiş, and A. M. Tekalp, “Shrinkage as activation for learned image compression,” IEEE Int. Conf. on Image Processing, Abu Dhabi, UAE, Nov. 2020.
• M. A. Yılmaz and A. M. Tekalp, “End-to-end rate-distortion optimization for bi-directional learned video compression,” IEEE Int. Conf. on Image Processing, Abu Dhabi, UAE, Nov. 2020.
• M. A. Yilmaz and A. M. Tekalp, “Effect of architectures and training methods on the performance of learned video frame prediction,” IEEE Int. Conf. Image Proc. (ICIP), Taipei, Taiwan, Sept. 2019.
• S. Nah, et al., “NTIRE 2019 Challenge on Video Super-Resolution: Methods and Results,” IEEE CVPR Workshop, Long Beach, CA, USA, June. 2019.
• O. Kirmemis and A. M. Tekalp, “Effect of training and test datasets on image restoration and super-resolution by deep learning,” Proc. of EUSIPCO, Rome, Italy, Sept. 2018.
• O. Kirmemis, G. Bakar, and A. M. Tekalp, “Learned compression artifact removal by deep residual networks,” IEEE CVPR Workshop Challenge on Learned Image Compression, Salt Lake City, Utah, USA, June 2018.

Pre-requisites

This short course presents material that every scientist and engineer who is interested in image and video processing must know today. Background in traditional signal/image processing concepts and some familiarity with convolutional and recurrent neural networks is expected.

Short bio

A. Murat Tekalp received Ph.D. degree in Electrical, Computer, and Systems Engineering from Rensselaer Polytechnic Institute (RPI), Troy, New York, in 1984, He has been with Eastman Kodak Company, Rochester, New York, from 1984 to 1987, and with the University of Rochester, Rochester, New York, from 1987 to 2005, where he was promoted to Distinguished University Professor. He is currently Professor at Koc University, Istanbul, Turkey. He served as Dean of Engineering between 2010-2013. His research interests are in digital image and video processing, including video compression and streaming, video networking, and image/video processing using deep learning.

He is a Fellow of IEEE and a member of Academia Europaea. He served as an Associate Editor for IEEE Trans. on Signal Processing (1990-1992) and IEEE Trans. on Image Processing (1994-1996). He chaired the IEEE Signal Processing Society Technical Committee on Image and Multidimensional Signal Processing (Jan. 1996 – Dec. 1997). He was the Editor-in-Chief of the EURASIP journal Signal Processing: Image Communication published by Elsevier between 1999-2010. He was appointed as the General Chair of IEEE Int. Conf. on Image Processing (ICIP) at Rochester, NY in 2002, and as Technical Program Co-Chair of ICIP 2020, which was held as an on-line event. He was on the Editorial Board of the IEEE Signal Processing Magazine (2007-2010) and the Proceedings of the IEEE (2014-2020). He is now serving in the Editorial Board of Wiley-IEEE Press (since 2018). He served in the ERC Advanced Grants Evaluation Panel (2009-2015). Dr. Tekalp has authored the Prentice Hall book Digital Video Processing (1995), a completely rewritten second edition of which is published in 2015.