PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

Point cloud is an important type of geometric data structure. Due to its irregular format, most researchers transform such data to regular 3D voxel grids or collections of images. This, however, renders data unnecessarily voluminous and causes issues. In this paper, we design a novel type of neural network that directly consumes point clouds and well respects the permutation invariance of points in the input. Our network, named PointNet, provides a unified architecture for applications ranging from object classification, part segmentation, to scene semantic parsing. Though simple, PointNet is highly efficient and effective. Empirically, it shows strong performance on par or even better than state of the art. Theoretically, we provide analysis towards understanding of what the network has learnt and why the network is robust with respect to input perturbation and corruption.

PDF Abstract CVPR 2017 PDF CVPR 2017 Abstract

Results from the Paper


 Ranked #1 on Semantic Segmentation on S3DIS (Number of params metric)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Skeleton Based Action Recognition CAD-120 PointNet (5-shot) Accuracy 69.1% # 7
3D Semantic Segmentation KITTI-360 PointNet miou 13.07 # 4
mIoU Category 30.42 # 4
Model size N/A # 1
3D Point Cloud Classification ModelNet40 PointNet Overall Accuracy 89.2 # 93
Mean Accuracy 86.0 # 33
Number of params 3.47M # 94
Few-Shot 3D Point Cloud Classification ModelNet40 10-way (10-shot) PointNet Overall Accuracy 46.60 # 23
Standard Deviation 13.5 # 24
Few-Shot 3D Point Cloud Classification ModelNet40 10-way (20-shot) PointNet Overall Accuracy 35.20 # 24
Standard Deviation 13.5 # 24
Few-Shot 3D Point Cloud Classification ModelNet40 5-way (10-shot) PointNet Overall Accuracy 51.97 # 23
Standard Deviation 12.1 # 23
Few-Shot 3D Point Cloud Classification ModelNet40 5-way (20-shot) PointNet Overall Accuracy 57.81 # 23
Standard Deviation 15.5 # 24
3D Point Cloud Classification ModelNet40-C PointNet Error Rate 0.283 # 12
Point Cloud Segmentation PointCloud-C PointNet mean Corruption Error (mCE) 1.178 # 11
Point Cloud Classification PointCloud-C PointNet mean Corruption Error (mCE) 1.422 # 23
Semantic Segmentation S3DIS PointNet mAcc 66.2 # 27
Number of params N/A # 1
Scene Segmentation ScanNet PointNet++ Average Accuracy 60.2% # 2
3D Point Cloud Classification ScanObjectNN PointNet Overall Accuracy 68.2 # 59
Mean Accuracy 63.4 # 28
3D Semantic Segmentation SemanticKITTI PointNet test mIoU 14.6% # 39
3D Part Segmentation ShapeNet-Part PointNet Instance Average IoU 83.7 # 58

Results from Other Papers


Task Dataset Model Metric Name Metric Value Rank Source Paper Compare
3D Point Cloud Classification IntrA PointNet F1 score (5-fold) 0.684 # 12
3D Part Segmentation IntrA PointNet IoU (V) 75.23 # 7
DSC (V) 85.00 # 7
IoU (A) 37.75 # 7
DSC (A) 49.59 # 7
Semantic Segmentation S3DIS Area5 PointNet mAcc 49.0 # 36
Number of params N/A # 2

Methods