Abstract:

Decision support systems for classification tasks are predominantly designed to predict the value of the ground truth labels. However, these systems also need to make human experts understand when and how to use these predictions to update their own predictions. Unfortunately, this has been proven challenging. In this talk, I will introduce an alternative type of decision support systems that circumvent this challenge by design. Rather than providing a single label prediction, these systems provide a set of label prediction values, namely a prediction set, and ask experts to predict a label value from the prediction set. Moreover, I will discuss how to use conformal prediction, online learning and counterfactual inference to efficiently construct prediction sets that optimize experts’ performance, provably. Further, I will present the results of a large-scale human subject study, which show that, for decision support systems based on prediction sets, limiting experts level of agency leads to greater performance than allowing experts to always exercise their own agency.