Fast weights, the recent past, and transformers

Jimmy Ba, Geoffrey Hinton, Volodymyt Mnih, Joel Z. Leibo, Catalin Ionescu, Using Fast Weights to Attend to the Recent Past, arXiv:1610.06258v3 [stat.ML] 5 Dec 2016

Abstract: Until recently, research on artificial neural networks was largely restricted to systems with only two types of variable: Neural activities that represent the current or recent input and weights that learn to capture regularities among inputs, outputs and payoffs. There is no good reason for this restriction. Synapses have dynamics at many different time-scales and this suggests that artificial neural networks might benefit from variables that change slower ...