Using Data Science to Understand “Entertainment Choices” at Netflix: quasi-experimentation for advertising evaluation and beyond * note conference room change

Friday, March 3, 2017, 11:00 am - 12:00 pm PDTiCal
6th floor large conference room
This event is open to the public.
Jeremy Glick and Kelly Uphoff

Netflix begins to use a new advertising platform, and within a few days, aggregate signups increase.  What, if anything, should we conclude?  Netflix ceases to use the new platform, and signups decrease.  Now what?  How can we estimate the causal effectiveness of advertising when we don't even know who saw the ad?  At Netflix, we performed a series of studies to learn about paid search advertising.  The lessons from these early quasi-experiments, methodological successes and failures, continue to guide a broad range of Netflix experimentation as we continue to blend the worlds of entertainment and tech. 

In this talk, we will describe a specific class of quasi-experimental design that uses region-scale interventions, and which can be measured only with highly aggregated time series data.  We will introduce a structured multivariate time series model which allows us to make business-relevant decisions on such quasi-experiments, and which appropriately captures our uncertainty about the results.  Finally, we will discuss the lessons for quasi-experimental design which allow this model to be useful in practice.  This type of experiment has broad applicability to many problem spaces. More recently, we intend to use this class of experimentation to better understand the societal and market-level network and other effects that define and cause “Word of Mouth” - organic, social buzz about entertainment properties that has the potential to create outsized positive impact for the business. 

Jeremy Glick received his Ph.D. in cognitive psychology at Stanford University, working with Dr. James McClelland.  His dissertation contrasts neural network and structured Bayesian models of human semantic cognition.  He now focuses on the design and statistical analysis of human behavioral experiments, quasi-experiments, and observational studies, inside a broader framework of causal inference.  He can be reached at



Kelly Uphoff is the Director of Growth Data Science at Netflix, focused on building teams that use experiments, statistical models, and algorithms to grow the Netflix member base and audiences for its titles worldwide. Kelly started at Netflix six years ago as a statistician and prior to her time at Netflix, built predictive models to understand customer retention at GEICO. She also served in the Peace Corps in the Republic of Georgia, where she helped women entrepreneurs start and manage small businesses.

« Return to Upcoming Events