Multiplicative RNN

A Multiplicative RNN (mRNN) is a type of recurrent neural network with multiplicative connections. In a standard RNN, the current input $x_{t}$ is first transformed via the visible-to-hidden weight matrix $W_{hx}$ and then contributes additively to the input for the current hidden state. An mRNN allows the current input (a character in the original example) to affect the hidden state dynamics by determining the entire hidden-to-hidden matrix (which defines the non-linear dynamics) in addition to providing an additive bias.

To achieve this goal, the authors modify the RNN so that its hidden-to-hidden weight matrix is a (learned) function of the current input $x_{t}$:

$$ h_{t} = \tanh\left(W_{hx}x_{t} + W_{hh}^{\left(x_{y}\right)}h_{t-1} + b_{h}\right)$$

$$ o_{t} = W_{oh}h_{t} + b_{o} $$

This is the same as the equations for a standard RNN, except that $W_{hh}$ is replaced with $W^{(xt)}_{hh}$. allowing each input (character) to specify a different hidden-to-hidden weight matrix.