Eneko Agirre

[introductory/intermediate] Natural Language Processing in the Pretrained Language Model Era

Summary

Deep Learning models have been successfully applied to natural language processing, and are now changing radically how we interact with machines (Siri, Alexa, Google Home, Translate and Search Engine). These models are able to infer a continuous representation for text, instead of using hand-engineered features as in other machine learning approaches. The course will explain the main deep learning models used in NLP, including pre-trained language models such as BERT, XLM-R and GPT, and how they are applied to obtain the best NLP results by fine-tuning and prompting. Attendants will be able to understand and play with implementations in Tensorflow Keras.

Syllabus

• Introduction to NLP and DL
• Multilayer Perceptron and Transformer
• Pretrained Language Models, Fine-Tuning and Prompting

References

Deep Learning. Ian Goodfellow, Yoshua Bengio and Aaron Courville. 2015.

Neural Network Methods for Natural Language Processing. Yoav Goldberg. 2017.

Deep Learning with Python. Francois Chollet. 2017.

Liu et al. (2021). Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586.

Min et al. (2021). Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey. arXiv preprint arXiv:2111.01243.

Hugging Face Course: https://huggingface.co/course

Pre-requisites

Addressed to professionals, researchers and students who want to understand and apply deep learning techniques to text. The practical part requires basic programming experience.

Short bio

Eneko Agirre is full profesor in the Computer Science department of the University of the Basque Country (UPV/EHU). He has published over 150 international peer-reviewed articles and conference papers in NLP. He has been secretary and president of the ACL SIGLEX, member of the editorial board of Computational Linguistics, Transactions of the ACL and Journal of Artificial Intelligence Research. He is co-founder of the Joint Conference on Lexical and Computational Semantics (*SEM), now in it’s ninth edition. He is a usual reviewer for top international journals, a regular area chair and member of the program committees for top international conferences. He has coordinated several National and European projects. He has received three Google Research Awards in 2016, 2018 and 2019, and five best paper awards and nominations. Several dissertations under his supervision have received the SEPLN, Spanish Computer Science and EurAI 2021 best PhD awards. He received the Spanish Computer Science Research Award in 2021.