O. Winther, B. Lautrup and J-B. Zhang,
Optimal Learning in Multilayer Neural Networks.
Tech. Rep. 95-200, CERN, Theory Division, 1211 Genève 23, Switzerland, August 1995.
O. Winther, B. Lautrup and J-B. Zhang,
Optimal Learning in Multilayer Neural Networks.
Tech. Rep. 95-200, CERN, Theory Division, 1211 Genève 23, Switzerland, August 1995.
The generalization performance of two learning algorithms, Bayes algorithm and the ``optimal learning'' algorithm on two classification tasks is studied theoretically. In the first example the task is defined by a restricted two-layer network, a committee machine, and in the second the task is defined by the so-called prototype problem. The architecture of the learning machine is in both cases defined to be a committee machine. For both tasks the optimal learning algorithm, which is optimal when the solution is restricted to a specific architecture, performs worse than the overall optimal Bayes algorithm. However, both algorithms perform far better than the conventional stochastic Gibbs algorithm, showing that using prior knowledge about the rule helps to avoid overfitting.