Active evaluation of predictive models

Más información sobre el libro

The field of machine learning focuses on algorithms that create predictive models from data, which are essential for tasks like spam filtering and personalized recommendations. To effectively deploy these models, understanding their performance is vital, requiring a set of labeled test instances from the same distribution they will encounter in real-world applications. However, labeling unlabeled instances can be time-consuming and costly, often necessitating human expertise. This thesis tackles the challenge of accurately evaluating predictive models with minimal labeling effort through an active model evaluation process that selectively queries labels based on an instrumental sampling distribution. We derive distributions aimed at minimizing estimation error for various performance metrics, leading to confidence intervals that reflect the precision of error estimations. Additionally, we consider the varying labeling costs associated with different data characteristics and extend the sampling distribution accordingly. Our empirical studies demonstrate that active evaluation can outperform standard estimation methods. We also explore comparing risks between predictive models, developing a sampling procedure that enhances statistical test power. Furthermore, we apply these concepts to ranking functions, deriving optimal sampling distributions for key performance measures, with experiments showing significant labeling cost reduc