Lyle Ungar
[intermediate] Natural Language Processing using Deep Learning
Summary
We will cover current deep learning methods for natural language processing (NLP), including the use of context-oblivious embeddings, RNNs and LSTMs, and transformers for tasks ranging from recognizing the sentiment and emotion of a sentence to automatically generating responses to natural language prompts.
Syllabus
(1) NLP tasks and vector embeddings
- Recognize NLP tasks: IR/search, Question Answering/text completion, machine translation
- Distributional similarity on words and context
- Context-oblivious embeddings (word2vec, glove, fastText), including multilingual embeddings
(2) Deep learning architectures for NLP
- RNNs and LSTMs
- Attention
- Context-sensitive embeddings: BERT and transformers (masking and self-attention)
- Fine-tuning language embeddings
(3) Natural language generation (NLG)
- Big language models: GPT-3 and friends
- Where they work, where they fail
References
- Speech and Language Processing, Jurafsky and Martin (https://web.stanford.edu/~jurafsky/slp3/) Chapter 6
- Dive into Deep Learning (https://d2l.ai) Chapters 8-10, 14 and 15
- Language Models are Few-Shot Learners, Brown et al. (The GPT3 paper)
Code to implement the concepts we will cover
- https://github.com/CIS-522/course-content/tree/main/tutorials/W09_RNN
- https://github.com/CIS-522/course-content/tree/main/tutorials/W10_NLP
Pre-requisites
Basics of deep learning. No NLP experience required.
Short bio
Dr. Lyle Ungar is a Professor of Computer and Information Science at the University of Pennsylvania, where he also holds appointments in multiple departments in the Schools of Business, Medicine, Arts and Sciences, and Engineering and Applied Science. He has published over 300 articles, supervised two dozen Ph.D. students, and is co-inventor on ten patents. His current research focuses on developing scalable machine learning methods for data mining and text mining, including deep learning methods for natural language processing, and analysis of cell phone and social media to better understand the drivers of physical and mental well-being.
