Event Details

In-Person Presentation: Friday, November 15, 2024

Speaker: Yiorgos (Georgios) Chochlakis, Research Assistant, USC

Location: ISI Marina del Rey, Conference Room 1135/37. In-person attendance for USC-ISI faculty, staff, and students only. Open to the public virtually via Zoom.

Zoom Link: https://usc.zoom.us/j/92176495278?pwd=Ty0fTlOzy3YXyDK6uz3rtez5SN8Lco.1
Meeting ID: 921 7649 5278
Meeting Password: 926780

Hosted by: Eric Boxer
POC: Karen Lake

This event will be recorded.

It will be posted on our USC/ISI YouTube page within 1-2 business days: https://www.youtube.com/user/USCISI.

Abstract: In-Context Learning (ICL) has rapidly become the leading approach for performing natural language tasks with Large Language Models (LLMs), due to the ability to leverage background knowledge. ICL’s few-shot adaptability shows promise but relies heavily on retrieving task priors from that background knowledge. This is especially challenging in domains where human judgements vary, causing LLMs to default to priors even when evidence conflicts with them, which can limit performance on complex, subjective tasks like emotion and morality recognition. Augmenting ICL with Chain-of-Thought (CoT) prompting, intended to explicitly include reasoning, does not overcome these limitations, as it instead retrieves reasoning priors that remain fixed despite task-specific prompts. We design experiments to measure the influence of LLM priors on their posteriors and find that the effect intensifies with model size, leading to posterior collapse. Additionally, we explore how aggregating human judgments in subjective tasks introduces artifacts and biases that may confound LLM performance. Our findings suggest a need for caution when deploying larger LLMs for subjective tasks and underscore the value of annotator-level modeling over aggregated datasets in these contexts.

Speaker Bio

Yiorgos (Georgios) Chochlakis is a 4th year PhD fellow at the University of Southern California, advised by professor Shrikanth Narayanan in the Signal Analysis and Interpretation Laboratory. Previously, he earned his joint degree in Electrical and Computer Engineering from the National Technical University of Athens advised by Alexandros Potamianos and interned twice as an Applied Scientist at AWS.

His contemporary research is focused on modeling subjective natural language tasks, namely settings where different annotators can reasonably disagree about their semantic interpretations of the same stimulus, or rather can use distinct yet valid reasoning paths to arrive at different valid answers. His work has quantified the relationship between the prior and the posterior of Large Language Models on these tasks under various settings and conditions, including In-Context Learning, Chain-of-Thought Prompting, and Instruction Finetuning.