Intelligent Systems Division

The impact of sources of uncertainty on forecasts in various expert domains

Friday, November 17, 2017, 11:00am - 12:00pm PSTiCal
11th floor large conference room
This event is open to the public.
AI Seminar - Interview talk
Daniel Benjamin

In two lines of research, I explore how experts make and stakeholders react to judgmental forecasts. Key stakeholders often rely on forecasts from multiple sources to make decisions about uncertain events. Previous research found that individuals are driven to reduce conflict before imprecision for gains. However, conflict and imprecision are rarely well-differentiated and are often correlated. I examine the joint and distinct effects of conflict and imprecision focusing on how the configuration of forecast sets influences decision-makers. First, I describe how individuals react to multiple models projecting specific climate change impacts. Results suggest that reactions to multiple forecasts are driven by the structure and balance of the forecast set. The impact of these factor varies between quantitative estimation and comparative rating tasks. I then extend these these findings to less partisan domains and more complex sets.

The accuracy of investigator judgments about risk and benefit is vital to efficient research, but little is known about the accuracy of investigator judgments. I collected subjective probability forecasts to quantify oncology experts’ predictive judgments about clinical trials in their specialty. Experts were somewhat accurate about the base rates of oncology, but underperformed several naive forecasting strategies suggesting trials are launched under heightened uncertainty. Scientists’ judgments about whether a finding will reproduce determine how scientists design their own experiments and the pace at which science self-corrects. I collected expert researcher forecasts to test whether experts could judge if several preclinical cancer studies would reproduce original effects. Experts tended to be overconfident about the likelihood of reproducing experiments meaning investigators find published effects more credible than they actually are, and/or they underappreciate the difficulty in replicating experiments in biological systems.


Dr. Benjamin is postdoctoral fellow in decision sciences in the Biomedical Ethics Unit working on forecasting the success of translational studies via expert judgments. He earned his PhD in Psychometrics and Quantitative Psychology at Fordham University. His research interests include topics in judgment and decision-making focusing on 1) forecasting and elicitation of expert judgments, 2) sources of uncertainty and decision environments, and 3) the communication of uncertain and ambiguous information.

« Return to Events