Yulan He

[introductory/intermediate] Machine Reading Comprehension with Large Language Models

Summary

Large Language Models (LLMs) have demonstrated their impressive capabilities across various tasks, including content generation, code writing, and human-like conversation. They have transformed the approaches to Machine Reading Comprehension (MRC). In MRC, an AI system is trained to read and comprehend text, and generate answers to the questions posed. MRC tasks can vary in complexity, ranging from simple fact-based extractive Question-Answering (QA) where answers can be directly extracted from text, to more complex questions that require situational awareness, reasoning and inference. MRC can find applications in many real-world scenarios, such as helping in understanding lengthy narratives, facilitating customer support chatbots, and enhancing educational assessments.

This tutorial on MRC with LLMs offers a comprehensive exploration of the subject matter. It will begin by delving into the fundamentals of MRC, including a discussion of the evolution of LLMs, the core architecture of LLMs, and some prominent LLM examples. Afterwards, it will cover techniques for parameter-efficient fine-tuning, prompt engineering, and in-context learning. Subsequently, it will shift the focus to present the use cases related to MRC, such as narrative understanding, long-range question-answering, automated student answer scoring, and claim veracity assessment. Students will also be introduced to a validation framework tailored for evaluating LLMs’ performance in MRC tasks. Finally, the tutorial will explore the aspects of explainability of LLMs and future trends in the field. This tutorial will guide students to navigate the evolving landscape of MRC with LLMs, preparing them to address real-world language understanding challenges.

Syllabus

• Fundamentals of Machine Reading Comprehension (MRC): brief history and evolution of Large Language Models (LLMs); architecture of LLMs; examples of LLMs.
• Learning of LLMs: parameter-efficient fine-tuning; prompt engineering; in-context learning.
• Case studies of MRC: narrative understanding; long range QA; automated student answer scoring; claim veracity assessment.
• Evaluation: validation framework for LLMs.
• Explainability: understanding the emergent capabilities of LLMs; uncertainty interpretation of LLMs.
• Future trends: limitations and challenges in the field; emerging trends in MRC.

References

Vaswani, A., Shazeer, N.M., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., & Polosukhin, I. (2017). Attention is All you Need. Neural Information Processing Systems.

Devlin, J., Chang, M., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. North American Chapter of the Association for Computational Linguistics.

Raffel, C., Shazeer, N.M., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P.J. (2019). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. J. Mach. Learn. Res., 21, 140:1-140:67.

Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., et al. (2020). Language Models are Few-Shot Learners. ArXiv, abs/2005.14165.

Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021). Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. ACM Computing Surveys, 55, 1 – 35.

Hu, J.E., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., & Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models. ArXiv, abs/2106.09685.

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E.H., Xia, F., Le, Q., & Zhou, D. (2022). Chain of Thought Prompting Elicits Reasoning in Large Language Models. ArXiv, abs/2201.11903.

Kadavath, S., Conerly, T., Askell, A., Henighan, T.J., Drain, D., Perez, E., Schiefer, N., et al. (2022). Language Models (Mostly) Know What They Know. ArXiv, abs/2207.05221.

Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., et al. (2023). Holistic Evaluation of Language Models. Annals of the New York Academy of Sciences, 1525, 140 – 146.

Guo, Z., Jin, R., Liu, C., Huang, Y., Shi, D., Supryadi, Yu, L., Liu, Y., Li, J., Xiong, B., & Xiong, D. (2023). Evaluating Large Language Models: A Comprehensive Survey. ArXiv, abs/2310.19736.

Pre-requisites

Participants should have prior knowledge on machine learning and deep learning.

Short bio

Yulan He is a Professor at the Department of Informatics in King’s College London. She is currently holding a prestigious 5-year UKRI Turing AI Fellowship. Yulan’s research interests lie in the integration of machine learning and natural language processing for text analytics. She has published over 200 papers on topics including natural language understanding, model interpretability, rumour veracity assessment, question-answering, sentiment analysis, topic and event extraction, and biomedical text mining. She has received several prizes and awards, including a SWSA Ten-Year Award, a CIKM 2020 Test-of-Time Award, and AI 2020 Most Influential Scholar Honourable Mention by AMiner. She has served as the General Chair for AACL-IJCNLP 2022, a Program Co-Chair for EMNLP 2020, and as an Action Editor for Transactions of the ACL and an Associate Editor for the Royal Society Open Science journal.