Video Prediction
183 papers with code • 19 benchmarks • 24 datasets
Video Prediction is the task of predicting future frames given past video frames.
Gif credit: MAGVIT
Source: Photo-Realistic Video Prediction on Natural Videos of Largely Changing Frames
Benchmarks
These leaderboards are used to track progress in Video Prediction
| Trend | Dataset | Best Model | Paper | Code | Compare |
|---|---|---|---|---|---|
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
|||||
|
Libraries
Use these libraries to find Video Prediction models and implementationsDatasets
MNIST
Cityscapes
KITTI
Kinetics
Human3.6M
DAVIS
DAVIS 2017
KTH
Something-Something V2
Vimeo90K
Most implemented papers
PredNet and Predictive Coding: A Critical Review
We fill in the gap by evaluating PredNet both as an implementation of the predictive coding theory and as a self-supervised video prediction model using a challenging video action classification dataset.
Disentangling Physical Dynamics from Unknown Factors for Unsupervised Video Prediction
vincent-leguen/PhyDNet
•
•
CVPR 2020
Leveraging physical knowledge described by partial differential equations (PDEs) is an appealing way to improve unsupervised video prediction methods.
PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning
This paper models these structures by presenting PredRNN, a new recurrent network, in which a pair of memory cells are explicitly decoupled, operate in nearly independent transition manners, and finally form unified representations of the complex environment.
Video Diffusion Models
lucidrains/make-a-video-pytorch
•
•
Generating temporally coherent high fidelity video is an important milestone in generative modeling research.
SimVP: Simpler yet Better Video Prediction
gaozhangyang/simvp-simpler-yet-better-video-prediction
•
•
CVPR 2022
From CNN, RNN, to ViT, we have witnessed remarkable advancements in video prediction, incorporating auxiliary inputs, elaborate neural architectures, and sophisticated training strategies.
Video Prediction Models as Rewards for Reinforcement Learning
alescontrela/viper
•
•
NeurIPS 2023
A promising approach is to extract preferences for behaviors from unlabeled videos, which are widely available on the internet.
Expert Gate: Lifelong Learning with a Network of Experts
ContinualAI/avalanche
•
•
CVPR 2017
Further, the autoencoders inherently capture the relatedness of one task to another, based on which the most relevant prior model to be used for training a new expert, with finetuning or learning without-forgetting, can be selected.
Predicting Deeper into the Future of Semantic Segmentation
facebookresearch/SegmPred
•
•
ICCV 2017
The ability to predict and therefore to anticipate the future is an important attribute of intelligence.
Learning to Generate Long-term Future via Hierarchical Prediction
rubenvillegas/icml2017hierchvid
•
•
ICML 2017
To avoid inherent compounding errors in recursive pixel-level prediction, we propose to first estimate high-level structure in the input frames, then predict how that structure evolves in the future, and finally by observing a single frame from the past and the predicted high-level structure, we construct the future frames without having to observe any of the pixel-level predictions.
Prediction Under Uncertainty with Error-Encoding Networks
mbhenaff/EEN
•
•
In this work we introduce a new framework for performing temporal predictions in the presence of uncertainty.
