NICE learns a composition of invertible neural-network layers that transform data into independent latent variables, enabling exact log-likelihood training and sampling for density estimation.
How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation
10 Pith papers cite this work. Polarity classification is still indexing.
10
Pith papers citing it
abstract
We propose to exploit {\em reconstruction} as a layer-local training signal for deep learning. Reconstructions can be propagated in a form of target propagation playing a role similar to back-propagation but helping to reduce the reliance on derivatives in order to perform credit assignment across many levels of possibly strong non-linearities (which is difficult for back-propagation). A regularized auto-encoder tends produce a reconstruction that is a more likely version of its input, i.e., a small move in the direction of higher likelihood. By generalizing gradients, target propagation may also allow to train deep networks with discrete hidden units. If the auto-encoder takes both a representation of input and target (or of any side information) in input, then its reconstruction of input representation provides a target towards a representation that is more likely, conditioned on all the side information. A deep auto-encoder decoding path generalizes gradient propagation in a learned way that can could thus handle not just infinitesimal changes but larger, discrete changes, hopefully allowing credit assignment through a long chain of non-linear operations. In addition to each layer being a good auto-encoder, the encoder also learns to please the upper layers by transforming the data into a space where it is easier to model by them, flattening manifolds and disentangling factors. The motivations and theoretical justifications for this approach are laid down in this paper, along with conjectures that will have to be verified either mathematically or experimentally, including a hypothesis stating that such auto-encoder mediated target propagation could play in brains the role of credit assignment through many non-linear, noisy and discrete transformations.
citation-role summary
background 1
citation-polarity summary
roles
background 1polarities
background 1representative citing papers
DTG-FF reaches 91.8% on CIFAR-10 and 49.4% on ImageNet-100 224x224 but BP baselines beat it by 2.4-5.93 pp with gaps widening by class count on real data while reversing the synthetic trend.
FFR adapts Forward-Forward learning to regression via ordinal competitive goodness, stratified ladder layers, and hierarchical uncertainty-aware prediction, recovering 98.6% of backpropagation accuracy with substantially lower peak memory.
PC-ALM uses dual ascent on an augmented Lagrangian to achieve exact backpropagation gradients via layer-local updates in linear networks and matching performance in nonlinear networks up to depth 128.
SBD unifies broadcast credit assignment for losses including cross-entropy via output-score orthogonality to hidden activations and reports performance gains on CIFAR-10 and Tiny ImageNet.
Highway error propagation augments predictive coding with feedback matrices V to deliver depth-independent error corrections, allowing effective training of 128-layer MLPs while preserving local synaptic updates.
Learnable channel-class assignment and adaptive layer weighting allow forward-only CNNs to reach new state-of-the-art results among FF models on CIFAR-10, CIFAR-100, and Tiny-ImageNet.
HCL-FF augments the Forward-Forward algorithm with hierarchical learning and contrastive objectives to reach new state-of-the-art accuracies among FF methods on CIFAR-10 (+5.46%), CIFAR-100 (+17.00%), and Tiny-ImageNet (+12.51%).
SAL is a spike-timing-based local learning rule that aligns feedback weights to forward weights in spiking networks by exploiting noise and Hebbian/anti-Hebbian plasticity to recover the true gradient.
Covariance-aware goodness and auxiliary modules let Forward-Forward training scale to 16-layer networks, achieving 73.01% on ImageNet-100 and 50.30% on Tiny-ImageNet with roughly half the peak memory of backpropagation.
citing papers explorer
-
NICE: Non-linear Independent Components Estimation
NICE learns a composition of invertible neural-network layers that transform data into independent latent variables, enabling exact log-likelihood training and sampling for density estimation.