SKEP

SKEP is a self-supervised pre-training method for sentiment analysis. With the help of automatically-mined knowledge, SKEP conducts sentiment masking and constructs three sentiment knowledge prediction objectives, so as to embed sentiment information at the word, polarity and aspect level into pre-trained sentiment representation. In particular, the prediction of aspect-sentiment pairs is converted into multi-label classification, aiming to capture the dependency between words in a pair.

SKEP contains two parts: (1) Sentiment masking recognizes the sentiment information of an input sequence based on automatically-mined sentiment knowledge, and produces a corrupted version by removing these informations. (2) Sentiment pre-training objectives require the transformer to recover the removed information from the corrupted version. The three prediction objectives on top are jointly optimized: Sentiment Word (SW) prediction (on $\left.\mathrm{x}_{9}\right)$, Word Polarity (SP) prediction (on $\mathrm{x}_{6}$ and $\mathbf{x}_{9}$ ), Aspect-Sentiment pairs (AP) prediction (on $\mathbf{x}_{1}$ ). Here, the smiley denotes positive polarity. Notably, on $\mathrm{x}_{6}$, only SP is calculated without SW, as its original word has been predicted in the pair prediction on $\mathbf{x}_{1}$.