February 16th

14:30, Shannon amphitheatre (building 660) (see location):

Corentin Tallec (TAO team, LRI)

Title: Unbiased Online Recurrent Optimization

Abstract:

The novel Unbiased Online Recurrent Optimization (UORO) algorithm allows for
online learning of general recurrent computational graphs such as recurrent
network models. It avoids backtracking through the history of past activations and
inputs. UORO is a modification of NoBackTrack (NBT) that makes implementation
on complex architectures easy with current deep learning frameworks. UORO is as
computationally heavy as Truncated Backpropagation Through Time (TBPTT).
Contrary to TBPTT, UORO is guaranteed to provide an unbiased estimate of the gradient.

Performance is tested on tasks where TBPTT fails and on synthetic text
prediction. On the former, UORO overcomes TBPTT deficiencies. On the
latter, UORO succeeds in learning medium range temporal dependencies in
reasonable time. It competes with or outperforms TBPTT when inherent
dependencies are beyond the truncation range.



Contact: guillaume.charpiat at inria.fr