Complex synapses for continual learning

Complex Synapse

Most of our existing plasticity models are essentially a mathematical description of how to move a single parameter w, the synaptic weight, of a synapse up and down. We are good at casting this into a differential equation framework that works for both rate-based and spiking neurons and can be fitted to data from STDP experiments.

Zynapse double well potential

However, chemical synapses are complicated dynamical systems in their own right with a potentially high-dimensional state space and possibly complex latent dynamics. But what is the role if this synaptic complexity? Is it just an epiphenomenon of how biology implements LTP and LTD or do the complicated temporal dynamics do something essential for learning (Redondo and Morris, 2011; Ziegler et al. 2015; Zenke et al. 2015)? Unlikely. A body of theoretical work suggests that synaptic complexity may be crucial for memory retention (Fusi et al., 2005; Lahiri and Ganguli, 2013; Benna and Fusi, 2016).

Until recently, most existing theoretical work was focused on memory retention at the single synapse level. However, just a while ago we have shown that “synaptic intelligence”, i.e. specific complex synaptic dynamics are can, in fact, be sufficient to avoid catastrophic forgetting in deep neural networks. However, this only gives us a glimpse of what complex synaptic dynamics could in principle be capable of and opens the door to many exciting future research directions.

Cifar10 Cifar100 benchmark


Bibliography