Psychophysical Evaluation of Deep Re-Identification Models

Pedestrian re-identification (ReID) is the task of continuously recognising the sameindividual across time and camera views. Researchers of pedestrian ReID and theirGPUs spend enormous energy producing novel algorithms, challenging datasets,and readily accessible tools to successfully improve results on standard metrics.Yet practitioners in biometrics, surveillance, and autonomous driving have not re-alized benefits that reflect these metrics. Different detections, slight occlusions,changes in perspective, and other banal perturbations render the best neural net-works virtually useless. This work makes two contributions. First, we introducethe ReID community to a budding area of computer vision research in model eval-uation. By adapting established principles of psychophysical evaluation from psy-chology, we can quantify the performance degradation and begin research thatwill improve the utility of pedestrian ReID models; not just their performance ontest sets. Second, we introduce NuscenesReID, a challenging new ReID datasetdesigned to reflect the real world autonomous vehicle conditions in which ReIDalgorithms are used. We show that, despite performing well on existing ReIDdatasets, most models are not robust to synthetic augmentations or to the morerealistic NuscenesReID data.