In the context of decision making in robotics, the use of a classification framework which produces scores with inappropriate confidences will ultimately lead to the robot making dangerous decisions. In order to select a framework which will make the best decisions, we should pay careful attention to the ways in which it generates scores.
Precision and recall have been widely adopted as canonical metrics to quantify the performance of learning algorithms, but for robotics applications involving mission-critical decision making, good performance in relation to these metrics is insufficient. We introduce and motivate the importance of a classifier’s introspective capacity: the ability to associate an appropriate assessment of confidence with any particular classification.
We compare the introspective capacities of a number of commonly used classification frameworks in the context of visual perception for autonomous driving, in both classification and detection tasks. We show that better introspection leads to improved decision-making in the context of tasks such as autonomous driving or semantic map generation.