Towards Seamless Adaptation of Pre-trained Models for Visual Place Recognition
Recent studies show that vision models pre-trained in generic visual learning tasks with large-scale data can provide useful feature representations for a wide range of visual perception problems. However, few attempts have been made to exploit pre-trained foundation models in visual place recognition (VPR). Due to the inherent difference in training objectives and data between the tasks of model pre-training and VPR, how to bridge the gap and fully unleash the capability of pre-trained models for VPR is still a key issue to address. To this end, we propose a novel method to realize seamless adaptation of pre-trained models for VPR. Specifically, to obtain both global and local features that focus on salient landmarks for discriminating places, we design a hybrid adaptation method to achieve both global and local adaptation efficiently, in which only lightweight adapters are tuned without adjusting the pre-trained model. Besides, to guide effective adaptation, we propose a mutual nearest neighbor local feature loss, which ensures proper dense local features are produced for local matching and avoids time-consuming spatial verification in re-ranking. Experimental results show that our method outperforms the state-of-the-art methods with less training data and training time, and uses about only 3% retrieval runtime of the two-stage VPR methods with RANSAC-based spatial verification. It ranks 1st on the MSLS challenge leaderboard (at the time of submission). The code is released at https://github.com/Lu-Feng/SelaVPR.
PDF AbstractCode
Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Benchmark |
---|---|---|---|---|---|---|
Visual Place Recognition | Mapillary test | SelaVPR | Recall@1 | 73.5 | # 2 | |
Recall@5 | 87.5 | # 2 | ||||
Recall@10 | 90.6 | # 2 | ||||
Visual Place Recognition | Mapillary val | SelaVPR | Recall@1 | 90.8 | # 2 | |
Recall@5 | 96.4 | # 1 | ||||
Recall@10 | 97.2 | # 1 | ||||
Visual Place Recognition | Nordland | SelaVPR | Recall@1 | 86.6 | # 1 | |
Recall@5 | 94.0 | # 2 | ||||
Recall@10 | 95.9 | # 1 | ||||
Visual Place Recognition | Pittsburgh-250k-test | SelaVPR | Recall@1 | 95.7 | # 1 | |
Recall@5 | 99.2 | # 1 | ||||
Recall@10 | 98.8 | # 3 | ||||
Visual Place Recognition | Pittsburgh-30k-test | SelaVPR | Recall@1 | 92.8 | # 1 | |
Recall@5 | 97.7 | # 1 | ||||
Recall@10 | 96.8 | # 1 | ||||
Visual Place Recognition | St Lucia | SelaVPR | Recall@1 | 99.8 | # 1 | |
Visual Place Recognition | Tokyo247 | SelaVPR | Recall@1 | 94.0 | # 1 | |
Recall@5 | 97.5 | # 1 | ||||
Recall@10 | 96.8 | # 1 |