BAYESIAN TRAINING OF BACKPROPAGATION NETWORKS
BY THE HYBRID MONTE CARLO METHOD
Radford M. Neal
Department of Computer Science
University of Toronto
10 April 1992
It is shown that Bayesian training of backpropagation neural
networks can feasibly be performed by the "Hybrid Monte Carlo"
method. This approach allows the true predictive distribution
for a test case given a set of training cases to be approximated
arbitrarily closely, in contrast to previous approaches which
approximate the posterior weight distribution by a Gaussian. In
this work, the Hybrid Monte Carlo method is implemented in
conjunction with simulated annealing, in order to speed relaxation
to a good region of parameter space. The method has been applied
to a test problem, demonstrating that it can produce good
predictions, as well as an indication of the uncertainty of these
predictions. Appropriate weight scaling factors are found
automatically. By applying known techniques for calculation of
"free energy" differences, it should also be possible to compare
the merits of different network architectures. The work described
here should also be applicable to a wide variety of statistical
models other than neural networks.