In this paper we show how neural networks can be formulated in terms of various parameterised connection models which explicitly encode desired properties of the target system. Such a modeling approach to neural networks raises issues about their relationships to other technologies such as Adaptive Filtering and Principal Components Analysis. The benefits of this approach can be a significant decrease in the parameter space, improved generalisation, and a learning procedure which guarantees a priori specified invariance constraints.