The artificial neural network is a computing system inspired by the biological neural networks that constitute animal brains. Such systems learn (progressively improve performance) to do tasks by considering examples, generally without task-specific programming.
They use an elaboration architecture similar to the human one, which is made of the following components:
- Strong parallelism;
- Strong interconnection;
- Fast communication;
- Simple elaboration units;
- Large memory space (in the connections);
- Naturally associative;
- Not programmed, but teached to an automatic learning.
Analogy with the biological neuron
The neuron is a cell (sometimes called neuron cell) with a membrane and a central nucleus. The typical neuron has an axon, and more dendrites. It receive the input from the dendrites, then elaborate them into the some (the central body) and then it put the output towards the axon. More linked neurons compose a neural network.
The historical computer architecture is called Von Neumann and was developed by this sir during the 40s of the XX° century. This computer architecture is intrinsically sequential and is composed by a central CPU, a memory associated, and access this is very slow.
The neural network approach is different and can be simulated by computer with Von Neumann architectures. But the real advantage can came out only with an hardware architecture based on NN.
General Model of Neural Network
To describe comprehensively a neural network we can use a bunch of components that are common for all the several different models:
- elaboration units set
- activation state
- output function f
- connections among units
- propagation rule
- activation function F
- learning rule
Learning Rules and Hebb Rule
Almost all the learning rules depend on the Hebb rule:
If a unit i riceves an input from another unit j, and if both of them are strongly-active, the weight w(ji) from j to i has to be strengthened.
The different models of NN are:
- Linear Pattern Association
- Hopfield Networks
- Multi Layer Perceptron
- Kohonen Self Organizing Maps
- Boltzmann Machine
Linear Pattern Association (PA)
Before dive into the digression of this section you need to know two concepts of linear algebra:
- inner product of two vectors
- outer product of two vectors
Once, learned the meaning of this two mathematical operations and discovered how to compute them, let us continue.
Suppose we want to store the association between an input patter ip and an output pattern op. We can describe it, mathematically, as: w = € * op * ip^T. The Pattern Association will learn the weigth of an association through the outer product.
A PA is able to store more asssociations of the same network, and in certain cases it is impossible to store all the associations. The hopfiled network are an improvement of the PA.
A Hopfield network is a form of recurrent artificial neural network popularized by John Hopfield in 1982.
Hopfield networks serve as content-addressable memory systems with binary threshold nodes. They are guaranteed to converge to a local minimum, but will sometimes converge to a false pattern (wrong local minimum) rather than the stored pattern (expected local minimum). Hopfield networks also provide a model for understanding human memory.
The perceptron is an algorithm for supervised learning of binary classifiers. It is a type of linear classifier, i.e. a classification algorithm that makes its predictions based on a linear predictor function combining a set of weights with the feature vector.
MLP (Multi Layer Perceptron)
Its multiple layers and non-linear activation distinguish MLP from a linear perceptron.
Kohonen Self-Organizing Maps (SOM)
The Kohonen SOM provide a way of representing multidimensional data in much lower dimensional spaces – usually one or two dimensions. This process, of reducing the dimensionality of vectors, is essentially a data compression technique known as vector quantisation. In addition, the Kohonen technique creates a network that stores information in such a way that any topological relationships within the training set are maintained.
A Boltzmann machine is a type of stochastic recurrent neural network (RNN).
They can be seen as the stochastic, generative counterpart of Hopfield networks.