Bayesian Learning from Sequential Data using Gaussian Processes with Signature Covariances

ICML 2020  ·  Csaba Toth, Harald Oberhauser ·

We develop a Bayesian approach to learning from sequential data by using Gaussian processes (GPs) with so-called signature kernels as covariance functions. This allows to make sequences of different length comparable and to rely on strong theoretical results from stochastic analysis... Signatures capture sequential structure with tensors that can scale unfavourably in sequence length and state space dimension. To deal with this, we introduce a sparse variational approach with inducing tensors. We then combine the resulting GP with LSTMs and GRUs to build larger models that leverage the strengths of each of these approaches and benchmark the resulting GPs on multivariate time series (TS) classification datasets. Code available at https://github.com/tgcsaba/GPSig. read more

PDF Abstract ICML 2020 PDF

Datasets


Task Dataset Model Metric Name Metric Value Global Rank Benchmark
Time Series Classification ArabicDigits GP-LSTM Accuracy 0.985 # 6
NLL 0.082 # 6
Time Series Classification ArabicDigits GP-GRU Accuracy 0.986 # 5
NLL 0.066 # 4
Time Series Classification ArabicDigits GP-Sig-LSTM Accuracy 0.992 # 3
NLL 0.047 # 2
Time Series Classification ArabicDigits GP-Sig-GRU Accuracy 0.994 # 1
NLL 0.023 # 1
Time Series Classification ArabicDigits GP-Sig Accuracy 0.979 # 8
NLL 0.071 # 5
Time Series Classification ArabicDigits GP-KConv1D Accuracy 0.984 # 7
NLL 0.050 # 3
Time Series Classification AUSLAN GP-Sig-GRU Accuracy 0.978 # 3
NLL 0.123 # 2
Time Series Classification AUSLAN GP-KConv1D Accuracy 0.784 # 9
NLL 1.900 # 6
Time Series Classification AUSLAN GP-LSTM Accuracy 0.880 # 8
NLL 0.650 # 5
Time Series Classification AUSLAN GP-Sig-LSTM Accuracy 0.983 # 2
NLL 0.106 # 1
Time Series Classification AUSLAN GP-Sig Accuracy 0.925 # 7
NLL 0.550 # 4
Time Series Classification AUSLAN GP-GRU Accuracy 0.949 # 6
NLL 0.248 # 3
Time Series Classification CharacterTrajectories GP-KConv1D Accuracy 0.941 # 6
NLL 0.409 # 4
Time Series Classification CharacterTrajectories GP-GRU Accuracy 0.114 # 9
NLL 3.523 # 6
Time Series Classification CharacterTrajectories GP-Sig-LSTM Accuracy 0.991 # 3
NLL 0.031 # 1
Time Series Classification CharacterTrajectories GP-Sig Accuracy 0.979 # 4
NLL 0.108 # 2
Time Series Classification CharacterTrajectories GP-LSTM Accuracy 0.233 # 8
NLL 2.506 # 5
Time Series Classification CharacterTrajectories GP-Sig-GRU Accuracy 0.925 # 7
NLL 0.258 # 3
Time Series Classification CMUsubject16 GP-Sig-LSTM Accuracy 1.000 # 1
NLL 0.088 # 2
Time Series Classification CMUsubject16 GP-Sig-GRU Accuracy 1.000 # 1
NLL 0.040 # 1
Time Series Classification CMUsubject16 GP-Sig Accuracy 0.979 # 7
NLL 0.089 # 3
Time Series Classification CMUsubject16 GP-LSTM Accuracy 0.924 # 8
NLL 0.270 # 6
Time Series Classification CMUsubject16 GP-GRU Accuracy 0.993 # 6
NLL 0.089 # 3
Time Series Classification CMUsubject16 GP-KConv1D Accuracy 0.897 # 9
NLL 0.255 # 5
Time Series Classification DigitShapes GP-LSTM Accuracy 1.000 # 1
NLL 0.013 # 2
Time Series Classification DigitShapes GP-KConv1D Accuracy 1.000 # 1
NLL 0.035 # 4
Time Series Classification DigitShapes GP-Sig-GRU Accuracy 1.000 # 1
NLL 0.035 # 4
Time Series Classification DigitShapes GP-GRU Accuracy 0.812 # 9
NLL 0.727 # 6
Time Series Classification DigitShapes GP-Sig Accuracy 1.000 # 1
NLL 0.021 # 3
Time Series Classification DigitShapes GP-Sig-LSTM Accuracy 1.000 # 1
NLL 0.008 # 1
Time Series Classification ECG GP-LSTM Accuracy 0.782 # 7
NLL 0.496 # 4
Time Series Classification ECG GP-Sig Accuracy 0.848 # 3
NLL 0.356 # 1
Time Series Classification ECG GP-Sig-GRU Accuracy 0.832 # 5
NLL 0.431 # 3
Time Series Classification ECG GP-Sig-LSTM Accuracy 0.816 # 6
NLL 0.402 # 2
Time Series Classification ECG GP-GRU Accuracy 0.734 # 9
NLL 0.601 # 6
Time Series Classification ECG GP-KConv1D Accuracy 0.760 # 8
NLL 0.543 # 5
Time Series Classification JapaneseVowels GP-GRU Accuracy 0.986 # 2
NLL 0.052 # 1
Time Series Classification JapaneseVowels GP-LSTM Accuracy 0.982 # 5
NLL 0.061 # 3
Time Series Classification JapaneseVowels GP-KConv1D Accuracy 0.986 # 2
NLL 0.067 # 4
Time Series Classification JapaneseVowels GP-Sig-LSTM Accuracy 0.981 # 7
NLL 0.080 # 6
Time Series Classification JapaneseVowels GP-Sig Accuracy 0.982 # 5
NLL 0.069 # 5
Time Series Classification JapaneseVowels GP-Sig-GRU Accuracy 0.985 # 4
NLL 0.053 # 2
Time Series Classification KickvsPunch GP-GRU Accuracy 0.600 # 9
NLL 0.674 # 5
Time Series Classification KickvsPunch GP-Sig-LSTM Accuracy 0.900 # 4
NLL 0.301 # 2
Time Series Classification KickvsPunch GP-Sig-GRU Accuracy 0.820 # 6
NLL 0.493 # 3
Time Series Classification KickvsPunch GP-Sig Accuracy 0.900 # 4
NLL 0.224 # 1
Time Series Classification KickvsPunch GP-LSTM Accuracy 0.620 # 8
NLL 0.696 # 6
Time Series Classification KickvsPunch GP-KConv1D Accuracy 0.700 # 7
NLL 0.662 # 4
Time Series Classification Libras GP-Sig-LSTM Accuracy 0.921 # 4
NLL 0.320 # 2
Time Series Classification Libras GP-KConv1D Accuracy 0.698 # 9
NLL 1.608 # 6
Time Series Classification Libras GP-LSTM Accuracy 0.776 # 6
NLL 0.911 # 4
Time Series Classification Libras GP-Sig Accuracy 0.923 # 3
NLL 0.259 # 1
Time Series Classification Libras GP-Sig-GRU Accuracy 0.899 # 5
NLL 0.346 # 3
Time Series Classification Libras GP-GRU Accuracy 0.742 # 8
NLL 1.110 # 5
Time Series Classification NetFlow GP-Sig Accuracy 0.937 # 4
NLL 0.189 # 2
Time Series Classification NetFlow GP-Sig-GRU Accuracy 0.921 # 8
NLL 0.259 # 6
Time Series Classification NetFlow GP-GRU Accuracy 0.926 # 7
NLL 0.194 # 3
Time Series Classification NetFlow GP-LSTM Accuracy 0.928 # 6
NLL 0.251 # 5
Time Series Classification NetFlow GP-KConv1D Accuracy 0.945 # 3
NLL 0.168 # 1
Time Series Classification NetFlow GP-Sig-LSTM Accuracy 0.931 # 5
NLL 0.218 # 4
Time Series Classification PEMS GP-Sig Accuracy 0.820 # 2
NLL 0.520 # 1
Time Series Classification PEMS GP-Sig-LSTM Accuracy 0.763 # 6
NLL 0.704 # 3
Time Series Classification PEMS GP-KConv1D Accuracy 0.794 # 3
NLL 0.537 # 2
Time Series Classification PEMS GP-Sig-GRU Accuracy 0.775 # 4
NLL 1.100 # 5
Time Series Classification PEMS GP-LSTM Accuracy 0.745 # 8
NLL 1.194 # 6
Time Series Classification PEMS GP-GRU Accuracy 0.769 # 5
NLL 0.784 # 4
Time Series Classification PenDigits GP-GRU Accuracy 0.951 # 6
NLL 0.187 # 4
Time Series Classification PenDigits GP-KConv1D Accuracy 0.946 # 7
NLL 0.181 # 2
Time Series Classification PenDigits GP-LSTM Accuracy 0.953 # 4
NLL 0.185 # 3
Time Series Classification PenDigits GP-Sig Accuracy 0.955 # 2
NLL 0.146 # 1
Time Series Classification PenDigits GP-Sig-LSTM Accuracy 0.928 # 8
NLL 0.289 # 5
Time Series Classification PenDigits GP-Sig-GRU Accuracy 0.902 # 9
NLL 0.399 # 6
Time Series Classification Shapes GP-Sig-GRU Accuracy 1.000 # 1
NLL 0.012 # 2
Time Series Classification Shapes GP-Sig Accuracy 1.000 # 1
NLL 0.011 # 1
Time Series Classification Shapes GP-LSTM Accuracy 1.000 # 1
NLL 0.016 # 5
Time Series Classification Shapes GP-GRU Accuracy 0.867 # 9
NLL 0.168 # 6
Time Series Classification Shapes GP-Sig-LSTM Accuracy 1.000 # 1
NLL 0.014 # 4
Time Series Classification Shapes GP-KConv1D Accuracy 1.000 # 1
NLL 0.012 # 2
Time Series Classification UWave GP-Sig-GRU Accuracy 0.968 # 4
NLL 0.121 # 2
Time Series Classification UWave GP-Sig-LSTM Accuracy 0.970 # 2
NLL 0.113 # 1
Time Series Classification UWave GP-KConv1D Accuracy 0.947 # 6
NLL 0.189 # 4
Time Series Classification UWave GP-LSTM Accuracy 0.870 # 8
NLL 0.745 # 5
Time Series Classification UWave GP-GRU Accuracy 0.763 # 9
NLL 1.168 # 6
Time Series Classification UWave GP-Sig Accuracy 0.964 # 5
NLL 0.140 # 3
Time Series Classification Wafer GP-GRU Accuracy 0.994 # 1
NLL 0.029 # 1
Time Series Classification Wafer GP-Sig-LSTM Accuracy 0.988 # 4
NLL 0.048 # 2
Time Series Classification Wafer GP-Sig Accuracy 0.965 # 9
NLL 0.105 # 5
Time Series Classification Wafer GP-Sig-GRU Accuracy 0.978 # 7
NLL 0.081 # 3
Time Series Classification Wafer GP-LSTM Accuracy 0.966 # 8
NLL 0.105 # 5
Time Series Classification Wafer GP-KConv1D Accuracy 0.984 # 5
NLL 0.085 # 4
Time Series Classification WalkvsRun GP-GRU Accuracy 1.000 # 1
NLL 0.028 # 2
Time Series Classification WalkvsRun GP-Sig-GRU Accuracy 1.000 # 1
NLL 0.030 # 3
Time Series Classification WalkvsRun GP-Sig Accuracy 1.000 # 1
NLL 0.023 # 1
Time Series Classification WalkvsRun GP-Sig-LSTM Accuracy 1.000 # 1
NLL 0.030 # 3
Time Series Classification WalkvsRun GP-LSTM Accuracy 1.000 # 1
NLL 0.048 # 5
Time Series Classification WalkvsRun GP-KConv1D Accuracy 1.000 # 1
NLL 0.066 # 6

Methods