Neuro Evolution

Neuroevolution is a machine learning technique that utilizes evolutionary algorithms (e.g. genetic algorithms) to evolve artificial neural networks. The idea is to utilize the power and flexibility (as discussed earlier) of evolutionary algorithms to optimize for optimal neural network weights and structures. Neuroevolution are commonly used to solve reinforcement learning tasks [46]. Even though it lacks the evaluation of direct interactions between agents and environment as required by typical

26

reinforcement learning algorithms, but with carefully designed fitness functions neuroevolution can perform as well [3] as reinforcement learning techniques [2].

In principle, any metaheuristic optimization algorithms can be used to optimize neural networks (as long as appropriate search operators are implemented), but neuroevolution is mostly associated with evolutionary algorithms, this association is intuitive since both neural networks and evolutionary algorithms takes inspiration from nature. Nevertheless it is worth to keep in mind that other optimization techniques (or a combination of techniques) can be used to evolve neural networks, e.g. [47].

To put this into perspective, this thesis considers neuroevolution to be made up of two components:

1. A metaheuristic (usually population based such as GA) optimization algorithm, used to evolve a population of neural networks (solutions).

2. Neural networks and related encoding and manipulation operators (e.g.

mutation and crossover) used by the optimization algorithm.

Besides the two distinct components mentioned above, problem dependent fitness functions for different optimization scenarios are required as well.

Notice that the above interpretation of neuroevolution is quite ambiguous as it does not narrow on any specific algorithms, because the purpose is to attempt to simplify the understanding of the modular components in neuroevolution on a macroscopic level.

Encoding of neural networks is about representing the structure of a neural network in such a way that it is possible to apply manipulation operator while maintaining the functionality of neural networks as a whole. For example fixed neural network structures can be encodes as a vector of weights [48], and evolution of each vector (network) is driven by randomly mutating the weight values within the vector. A population of weight vectors will be evaluated in each generation using a fitness function to assign corresponding fitness to each network.

According to a review by Yao [49], some more complex encoding schemes can be used to encode not only the weights of but also the topology and transfer functions of a network, other techniques to encode the transfer functions to evolve

heterogeneous networks had also been studied [50]. What is to be encoded for a neural network usually depends on the objective of the application or experiment, but for some problems it is worth considering whether to use indirect or direct encoding scheme.

Direct encoding is an encoding scheme where the relevant structures of a neural network can be directly mapped to the encoding and vice versa (isomorphic). The mapping assumes that whatever that is encoded are all that is needed to directly represent a network. For example a direct encoding of a fixed topology network may

27

be a fixed length vector of weight values corresponding to the different connections in the network, meaning that it is possible to directly translate a network between encoding and network structure consistently.

A problem with direct encoding is the length of the encoding string will grow in proportion to the number of encoded elements in a network. This is not a problem for small neural networks but can quickly become an issue when neural networks have the ability to grow bigger (i.e. changing topology). For certain task it is

fundamentally required that the neural networks used must be big, e.g. consider the work by Matthew et al. [3] where raw screen pixels are fed into the inputs of a neural network that is to be evolved for playing Atari games. This is where indirect

encoding start to show some promising properties.

Indirect encoding alleviates the issue with direct encoding where the encoded string can grow and become too big that can cause performance problems when mutation and crossover are applied. The idea to indirect encoding is that instead of

representing a network structure as exact as possible, it is possible to be represented by a set of rules that can be used to construct a neural network. These rules can be used to generate any relevant part of an ANN, i.e. the weights, topology and transfer functions.

For example the ES-HyperNEAT algorithm [51] indirectly encodes not just the connection weights but also the density and connections within a complex neural network called a substrate. This algorithm is based in HyperNEAT [52] and therefore evolves convolutional neural networks (CNNs) to generate structural patterns of large scale ANNs. The encoding of the CNNs utilizes direct encoding while the behavior of the CNNs indirectly encodes the actual ANN structures that are to be evaluated.

Once an encoding scheme has been decided then the next step is to design evolving mechanisms. These mechanisms are manipulation operators that takes a genome and apply modifications to it to create new genomes. Manipulation can be on the

weights, topology, transfer functions or even other properties such as learning rules for dynamic neural plasticity [53][54]. In genetic algorithms for example,

manipulation operators are the mutation and crossover operators that were previously discussed.

As evolution carries on and crossovers are applied, one of the questionable issues that arises with neuroevolution is what is known as the competing convention problem [55] also known as the permutations problem [56].

The competing convention problem refers to a problem that occurs when a naïve recombination operator is applied to genomes, e.g. by using single point crossover.

This is because some recombination operators do not take into consideration the topological ordering of neural network structures. What this means is that e.g. the single point crossover operator assumes that each gene in a genome uniquely