Deep Video Super-Resolution using HR Optical Flow Estimation

6 Jan 2020  ·  Longguang Wang, Yulan Guo, Li Liu, Zaiping Lin, Xinpu Deng, Wei An ·

Video super-resolution (SR) aims at generating a sequence of high-resolution (HR) frames with plausible and temporally consistent details from their low-resolution (LR) counterparts. The key challenge for video SR lies in the effective exploitation of temporal dependency between consecutive frames. Existing deep learning based methods commonly estimate optical flows between LR frames to provide temporal dependency. However, the resolution conflict between LR optical flows and HR outputs hinders the recovery of fine details. In this paper, we propose an end-to-end video SR network to super-resolve both optical flows and images. Optical flow SR from LR frames provides accurate temporal dependency and ultimately improves video SR performance. Specifically, we first propose an optical flow reconstruction network (OFRnet) to infer HR optical flows in a coarse-to-fine manner. Then, motion compensation is performed using HR optical flows to encode temporal dependency. Finally, compensated LR inputs are fed to a super-resolution network (SRnet) to generate SR results. Extensive experiments have been conducted to demonstrate the effectiveness of HR optical flows for SR performance improvement. Comparative results on the Vid4 and DAVIS-10 datasets show that our network achieves the state-of-the-art performance.

PDF Abstract

Results from the Paper
Edit

Ranked #6 on Video Super-Resolution on MSU Super-Resolution for Video Compression (BSQ-rate over ERQA metric)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Video Super-Resolution MSU Super-Resolution for Video Compression SOF-VSR-BD + x264 BSQ-rate over ERQA 1.544 # 6
Compare
BSQ-rate over VMAF 1.213 # 20
Compare
BSQ-rate over PSNR 2.763 # 16
Compare
BSQ-rate over MS-SSIM 0.843 # 14
Compare
BSQ-rate over LPIPS 1.262 # 14
Compare
Video Super-Resolution MSU Super-Resolution for Video Compression SOF-VSR-BD + uavs3e BSQ-rate over ERQA 11.458 # 48
Compare
BSQ-rate over VMAF 6.596 # 70
Compare
BSQ-rate over PSNR 8.658 # 47
Compare
BSQ-rate over MS-SSIM 3.566 # 48
Compare
BSQ-rate over LPIPS 4.007 # 32
Compare
Video Super-Resolution MSU Super-Resolution for Video Compression SOF-VSR-BI + x264 BSQ-rate over ERQA 4.981 # 24
Compare
BSQ-rate over Subjective Score 1.273 # 18
Compare
BSQ-rate over VMAF 1.083 # 19
Compare
BSQ-rate over PSNR 6.058 # 30
Compare
BSQ-rate over MS-SSIM 0.764 # 11
Compare
BSQ-rate over LPIPS 1.26 # 13
Compare
Video Super-Resolution MSU Super-Resolution for Video Compression SOF-VSR-BI + x265 BSQ-rate over ERQA 18.545 # 77
Compare
BSQ-rate over Subjective Score 2.244 # 31
Compare
BSQ-rate over VMAF 3.565 # 57
Compare
BSQ-rate over PSNR 9.07 # 49
Compare
BSQ-rate over MS-SSIM 4.558 # 54
Compare
BSQ-rate over LPIPS 11.236 # 58
Compare
Video Super-Resolution MSU Super-Resolution for Video Compression SOF-VSR-BI + uavs3e BSQ-rate over ERQA 5.299 # 25
Compare
BSQ-rate over Subjective Score 3.196 # 45
Compare
BSQ-rate over VMAF 5.361 # 63
Compare
BSQ-rate over PSNR 10.917 # 59
Compare
BSQ-rate over MS-SSIM 6.82 # 72
Compare
BSQ-rate over LPIPS 4.23 # 34
Compare
Video Super-Resolution MSU Super-Resolution for Video Compression SOF-VSR-BI + aomenc BSQ-rate over ERQA 12.808 # 51
Compare
BSQ-rate over Subjective Score 2.84 # 41
Compare
BSQ-rate over VMAF 5.398 # 64
Compare
BSQ-rate over PSNR 11.314 # 61
Compare
BSQ-rate over MS-SSIM 6.833 # 73
Compare
BSQ-rate over LPIPS 4.82 # 41
Compare
Video Super-Resolution MSU Super-Resolution for Video Compression SOF-VSR-BI + vvenc BSQ-rate over ERQA 18.844 # 78
Compare
BSQ-rate over Subjective Score 2.822 # 40
Compare
BSQ-rate over VMAF 4.527 # 61
Compare
BSQ-rate over PSNR 9.245 # 50
Compare
BSQ-rate over MS-SSIM 4.882 # 59
Compare
BSQ-rate over LPIPS 11.273 # 59
Compare
Video Super-Resolution MSU Super-Resolution for Video Compression SOF-VSR-BD + vvenc BSQ-rate over ERQA 15.958 # 68
Compare
BSQ-rate over VMAF 6.41 # 66
Compare
BSQ-rate over PSNR 8.027 # 43
Compare
BSQ-rate over MS-SSIM 2.112 # 38
Compare
BSQ-rate over LPIPS 13.494 # 77
Compare
Video Super-Resolution MSU Super-Resolution for Video Compression SOF-VSR-BD + aomenc BSQ-rate over ERQA 15.11 # 67
Compare
BSQ-rate over VMAF 7.464 # 73
Compare
BSQ-rate over PSNR 13.076 # 67
Compare
BSQ-rate over MS-SSIM 7.546 # 76
Compare
BSQ-rate over LPIPS 4.034 # 33
Compare
Video Super-Resolution MSU Super-Resolution for Video Compression SOF-VSR-BD + x265 BSQ-rate over ERQA 13.098 # 53
Compare
BSQ-rate over VMAF 4.346 # 60
Compare
BSQ-rate over PSNR 3.274 # 17
Compare
BSQ-rate over MS-SSIM 1.825 # 36
Compare
BSQ-rate over LPIPS 13.141 # 71
Compare
Video Super-Resolution MSU Video Super Resolution Benchmark: Detail Restoration SOF-VSR-BI Subjective score 4.805 # 22
Compare
ERQAv1.0 0.66 # 17
Compare
QRCRv1.0 0.557 # 10
Compare
SSIM 0.872 # 10
Compare
PSNR 29.381 # 10
Compare
FPS 0.571 # 21
Compare
1 - LPIPS 0.904 # 9
Compare
Video Super-Resolution MSU Video Super Resolution Benchmark: Detail Restoration SOF-VSR-BD Subjective score 4.863 # 21
Compare
ERQAv1.0 0.647 # 19
Compare
QRCRv1.0 0.557 # 10
Compare
SSIM 0.831 # 16
Compare
PSNR 25.986 # 22
Compare
FPS 0.699 # 18
Compare
1 - LPIPS 0.895 # 11
Compare
Video Super-Resolution MSU Video Upscalers: Quality Enhancement SOF-VSR PSNR 27.14 # 38
Compare
SSIM 0.937 # 45
Compare
VMAF 56.45 # 9
Compare
Video Super-Resolution Vid4 - 4x upscaling SOF-VSR PSNR 26 # 12
Compare
SSIM 0.772 # 12
Compare

Methods
Add Remove

No methods listed for this paper. Add relevant methods here
Contact us on: hello@paperswithcode.com . Papers With Code is a free resource with all data licensed under CC-BY-SA.
Terms Data policy Cookies policy