Event Details

Abstract: AI agents today are mostly siloed – they either retrieve and reason over vast amount of digital information and knowledge obtained online; or interact with the physical world through embodied perception, planning and action – but rarely both. This separation limits their ability to solve tasks that require integrated physical and digital intelligence, such as cooking from online recipes, navigating with dynamic map data, or interpreting real-world landmarks using web knowledge. We introduce Embodied Web Agents, a novel paradigm for AI agents that fluidly bridge embodiment and web-scale reasoning. To operationalize this concept, we first develop the Embodied Web Agents task environments, a unified simulation platform that tightly integrates realistic 3D indoor and outdoor environments with functional web interfaces. Building upon this platform, we construct and release the Embodied Web Agents Benchmark, which encompasses a diverse suite of tasks including cooking, navigation, shopping, tourism, and geolocation – all requiring coordinated reasoning across physical and digital realms for systematic assessment of cross-domain intelligence. Experimental results reveal significant performance gaps between state-of-the-art AI systems and human capabilities, establishing both challenges and opportunities at the intersection of embodied cognition and web-scale knowledge access.

Zoom link
Zoom password: 2025

Host: DJ Ashok
POC: Maura Covaci

Speaker Bio

Rui Sun is a second-year Ph.D. student in Computer Science at the University of California, Los Angeles, advised by Professor Kai-Wei Chang. His research focuses on multimodal language agents, including embodied agents, computer-use agents, and multimodal deep research agents. He is an incoming research intern at Apple AI/ML Foundation Model Team. Previously, he was a research intern at Microsoft Research, where he worked with Dr. Jianfeng Gao, and a research assistant at the MIT–IBM Watson AI Lab, advised by Professor Chuang Gan. Prior to joining UCLA, he received his master’s degree from Columbia University, advised by Professor Shih-Fu Chang.