Furong Huang
[advanced] Generative AI Agents
Summary
This advanced course introduces the foundations and frontiers of generative AI agents: autonomous systems built on large language and vision models that can reason, plan, act, use tools, and adapt in complex environments. The course connects theoretical foundations with emerging practice, covering alignment and post-training, reasoning and self-improvement, multi-agent workflows, web and code agents, and agent safety. The emphasis is on understanding the algorithmic building blocks of modern agent systems, how they are evaluated, and where the key research challenges remain.
Syllabus
- Foundations of generative AI agents: agent architectures, memory, planning, action, and tool use
- Alignment and post-training for agents: RL/RLHF, preference optimization, and test-time alignment
- Reasoning and self-improvement: chain-of-thought, search-based reasoning, reflection, and ensemble strategies
- Agentive workflows and multi-agent systems: design patterns, communication graphs, and role allocation
- Web and code agents: tool-using agents, software engineering agents, and benchmark environments
- Robustness, safety, and evaluation: adversarial vulnerabilities, benchmarking, and AI-generated content detection/watermarking
- Beyond digital environments: world models and agents for robotics and simulation
References
Sutton, R.S. and Barto, A.G. Reinforcement Learning: An Introduction. Second edition, 2018. https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf
Wang, L. et al. A Survey on Large Language Model based Autonomous Agents. 2024. https://arxiv.org/abs/2308.11432
Rafailov, R. et al. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. 2023. https://arxiv.org/abs/2305.18290
Yao, S. et al. ReAct: Synergizing Reasoning and Acting in Language Models. 2023. https://arxiv.org/abs/2210.03629
Schick, T. et al. Toolformer: Language Models Can Teach Themselves to Use Tools. 2023. https://arxiv.org/abs/2302.04761
Wu, Q. et al. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. 2023. https://arxiv.org/abs/2308.08155
Zhou, S. et al. WebArena: A Realistic Web Environment for Building Autonomous Agents. 2024. https://arxiv.org/abs/2307.13854
Jimenez, C.E. et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? 2024. https://arxiv.org/abs/2310.06770
Pre-requisites
Students should have a solid foundation in machine learning and deep learning, including neural networks and basic reinforcement learning. They should also be comfortable with linear algebra, calculus, probability, optimization, and Python-based implementation (e.g., PyTorch or TensorFlow). Familiarity with large language models is helpful, but not strictly required.
Short bio
Furong Huang is an Associate Professor in the Department of Computer Science at the University of Maryland, College Park, with affiliations in UMIACS, the Maryland Robotics Center, AMSC, and ECE. Her research bridges trustworthy machine learning, sequential decision-making, and generative AI, with a strong emphasis on building reliable, interpretable, and aligned foundation models for robotics and autonomous systems. Her work combines theoretical rigor with practical impact in robust, adaptable, and safe intelligent systems.
