Event Details

Speaker: Chenglei Si, Stanford University
Location: Conference Room 1135 + 1137 and online.

November 08, 2024

Join Zoom Meeting: https://usc.zoom.us/j/94409584905?pwd=Sm5LVkd0bndUdEluM3piK0NWTUQrUT09

Zoom meeting ID: 944 0958 4905
Passcode: 822247

Hosted by: Abel Salinas
POC: Justina Gilleland

Abstract: Recent advancements in large language models (LLMs) have sparked excitement about their potential to accelerate scientific discovery, particularly in automating research. However, no prior evaluations have demonstrated that LLMs can generate novel, expert-level ideas. In this talk, we present a human study that evaluates research idea generation, controlling for confounders, and conduct the first direct comparison between LLMs and expert NLP researchers. By recruiting over 100 NLP researchers to generate ideas and review them blindly, we find that LLM-generated ideas are judged as more novel (p < 0.05) than human-generated ones, though slightly weaker in feasibility. Our study highlights critical challenges in building and evaluating research agents, such as LLMs’ self-evaluation failures and lack of diversity in generation. We also discuss limitations of the current work and outline future directions to address them.

This event will be recorded.

It will be posted on our USC/ISI YouTube page within 1-2 business days: http://www.youtube.com/user/USCISI.

Speaker Bio

Chenglei Si is a second-year PhD student at Stanford NLP where he is co-advised by Tatsu Hashimoto and Diyi Yang. His research lies at the intersection of NLP and HCI. These days, he is most interested in how LLMs can transform scientific research, and new forms of human-AI co-intelligence. Towards this end, he splits his time between working with LLMs and running human studies.

Subscribe here to learn more about upcoming seminars: https://www.isi.edu/events/