Daphna Weinshall
Curriculum Learning in Deep Networks
Summary
Deep learning protocols typically involve the random sampling of training examples. In curriculum learning, on the other hand, we employ the gradual exposure of learners to examples in a meaningful order (from easy to hard), in a manner reminiscent of human teaching. Accordingly, I will start my talk by describing curriculum learning algorithms, in which data ranking is obtained by one of two methods: transfer learning from some competitive “teacher” network, and self ranking. Both methods show similar benefits in terms of increased learning speed and improved accuracy. I will then briefly describe our theoretical results, investigating the two simple models of linear regression and binary classification by hinge loss minimization. In both cases we can prove that the expected convergence rate decreases monotonically with some ideal difficulty score, thus showing that curriculum learning can be expected to speed up convergence under ideal circumstances. The opposite result can be shown when the difficulty score is replaced by the current loss at each point. Interestingly, I will discuss how these two results bring to terms two apparently contradicting heuristics: curriculum learning on the one hand, and hard data mining (or active learning) on the other.
Short bio
Daphna Weinshall received the PhD degree from Tel-Aviv University in 1986, working on models of evolution and population genetics. Her current research interests lie in the areas of computer vision and machine learning, including human vision and human learning. Her recent research interests focus on advancing deep learning by addressing characteristics where it differs most from humans, including curriculum learning, learning from very few labels, and ethical issues that arise from statistical biases in data. Professionally, Dr Weinshall published about 135 papers in journals and peer reviewed conferences, and served as area chair in most of the leading conferences in computer vision and machine learning.
