Rex Ying

[intermediate/advanced] Multimodal Foundation Models for Graph-Structured Data: Framework and Scientific Applications

Summary

Foundation models have demonstrated remarkable capabilities across a diverse range of tasks and modalities such as natural language, images, time series, graphs and tabular data. They leverage abundant unsupervised data to map them into an embedding space (usually Euclidean), such that the embeddings can be easily used for a variety of generative and predictive downstream tasks with little demonstration or label. However, the complex structural properties of these data modalities often exhibit non-Euclidean characteristics including hierarchy, power-law distribution and clustering. In this course, we will demonstrate the underlying rationale for non-Euclidean representation learning. We will further introduce research breakthroughs in leveraging non-Euclidean geometry, such as hyperbolic geometry, as a powerful embedding technique to be integrated in powerful large language models, multimodal foundation models and graph foundation models.

Syllabus

• Non-Euclidean Representation Learning
• Hyperbolic Deep Learning Architectures
• Non-Euclidean Training and Fine-tuning of Foundation Models
• Tools, Benchmarks, and Applications

References

Nickel, Maximillian, and Douwe Kiela. “Poincaré embeddings for learning hierarchical representations.” Advances in Neural Information Processing Systems 30 (2017).

Gu, Albert, et al. “Learning mixed-curvature representations in product spaces.” International Conference on Learning Representations. 2018.

Chami, Ines, et al. “Hyperbolic graph convolutional neural networks.” Advances in Neural Information Processing Systems 32 (2019).

Desai, Karan, et al. “Hyperbolic image-text representations.” International Conference on Machine Learning. PMLR, 2023.

Yang, Menglin, et al. “Hypformer: Exploring efficient transformer fully in hyperbolic space.” Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2024.

Yang, Menglin, et al. “Hyperbolic Fine-tuning for Large Language Models.” arXiv:2410.04010 (2024).

Pre-requisites

Deep learning and representation learning basics. Foundation models.

Short bio

Rex Ying is an assistant professor in the Department of Computer Science at Yale University. His research focus includes algorithms for foundation models, multimodal models, graph neural networks, geometric deep learning, and trustworthy deep learning. He is interested in the use of graphs and geometry to empower representation learning in expressiveness and trustworthiness in large-scale settings. Rex has built multi-modal foundation models in engineering, natural science, social science and financial domains. He won the best dissertation award at KDD 2022, and the Amazon Research Award in 2024. His research is in part supported by National Science Foundation, Gordon and Betty Moore Foundation, Snap Research and Amazon Research Award.