Kunal Talwar
[introductory/intermediate] Foundations of Differentially Private Learning
Summary
Machine Learning models can memorize training data and have been shown to allow extraction of information found in individual training examples. Differential Privacy is a standard definition of privacy in statistical settings and can allow training of models that capture global trends while provably preventing learning information specific to one or a few training examples. Recent advances have shown that one can in many settings, one can learn with differential privacy with little or no impact on accuracy. This includes both theoretical results showing that provably one can learn convex models with little impact on accuracy for a large range of parameters, and practical results showing that one can train neural network models with accuracy close to non-private methods. Similarly, learning popular items from a discrete set can be done privately at a small cost to accuracy. This course will study the foundational techniques that allow for solving these problems.
Syllabus
- Differential Privacy definition, properties
- Basic Algorithms, Composition
- Convex Optimization with Differential Privacy
- Privacy Amplification, Moments Accountant, Differentially Private SGD
- Frequency Estimation, Shuffling, Compression
- Principal Component Analysis
References
Cynthia Dwork, Aaron Roth. The Algorithmic Foundations of Differential Privacy. https://www.cis.upenn.edu/~aaroth/privacybook.html
Pre-requisites
Basic understanding of probabilities and statistics, calculus, linear algebra, and machine learning.
Short bio
Kunal Talwar is a Research Scientist at Apple, leading a research group focusing on foundations of ML and Private Data Analysis. His research interests span various aspects of Computer Science including Differential Privacy, Machine Learning, Algorithms and Data Structures. He got his B.Tech. from IIT Delhi (2000) and his PhD from UC Berkeley (2004). Prior to joining Apple, he worked at Microsoft Research in Silicon Valley from 2005 to 2014, and at Google Brain from 2014 to 2019. He has made major contributions to Differential Privacy, Metric Embeddings and Discrepancy Theory. His work has been recognized by the Privacy Enhancing Technologies award in 2009 and the ICLR Best Paper award in 2017.
