Network motifs

6.2 Network motifs

6.2.1 Architecture

In the case of a conventional reservoir computing setup, where the reservoir consists of a random network of nonlinear nodes, the input is fed in parallel to all nodes of the reservoir. While in terms of simplicity this concept suffers from the large number of nodes that need to be implemented, in terms of speed it benefits from its huge parallelism. On the other hand, the delayed feedback approach is far more resource efficient, but that implies that all the input data need to be serialized. The processing speed is limited by the delay length and can only be improved by using a faster sampling rate.

It is reasonable to look for a compromise between these two extremes by using a few nonlinear elements instead of only one. This opens a wide range of possible configurations of which only one example is studied here. The dynamics of small networks of delay-coupled nonlinear nodes have already been discussed in several studies [64, 113, 114].

In Fig. 6.6 the situation of two bidirectionally coupled nodes is shown. They are separated from each other by half the delay time and are both driven by a masked input. To fully benefit from all available delay induced dimensions the mask is chosen to be different for both nodes. In the case of both nodes being identical and receiving exactly the same masked input, their output would be identical as well. Hence the virtual node states in the two delay lines would be redundant. When choosing a different mask we benefit from all dimensions in a total delay time of τ. Moreover, the time to process one input sample has now become τ /2, only half of the single node delay system. The more nodes are added, the faster the processing can take place, because less serializing is needed. In the limit we reach a network of many nodes, similar to conventional neural networks. A trade-off has to be made between speed, simplicity of implementation and other factors such as power consumption.

6.2.2 Numerically obtained performance for NARMA10

In Fig. 6.7 the memory capacity of a bidirectionally coupled system of two Mackey-Glass nodes is depicted. Both nodes receives exactly the same input, but with a different mask. The parameters are identical for both nodes. The exponent is taken 1 in Fig. 6.7a and 7 in Fig. 6.7b.

116 6 System modifications

NL NL

τ

τ

Input Input

Fig. 6.6: Network of two bidirectionally coupled nodes. The two nodes are equally spaced along the delay line and receive the same input, but with a different masking function for each.

MC

Fig. 6.7: Memory capacity for two coupled nodes. The memory capacity is plotted in color code for a network motif consisting of 2 bidirectionally coupled nodes, while scanning the feedback strength and the input scaling. Both nodes receive identical inputs, but preprocessed with a different mask. The two nodes are equally spaced within the τ-interval and each drive 200 virtual nodes with a separation of 0.2.

(a)p= 1, (b)p= 7.

6.2 Network motifs 117

Fig. 6.8: Computational ability for two coupled nodes. The com-putational ability is plotted in color code for a network motif consisting of 2 bidirectionally coupled nodes, while scanning the feedback strength and the input scaling. Both nodes receive identical inputs, but prepro-cessed with a different mask. The two nodes are equally spaced within the τ-interval and each drive 200 virtual nodes with a separation of 0.2. (a)p= 1, (b)p= 7.

Next to memory capacity, we study the computational ability of the bidirec-tionally coupled system. The color coded plots in Fig. 6.8 shows the rank corresponding to the computational ability as defined in Eq.(5.1).

The scanned parameter region is identical to the one studied in detail for the single delayed-feedback situation. Also in terms of results a clear similarity can be seen. Although the parameter scan uses larger steps in this case, the same trends are observed. For a low exponent the memory is highest for small values ofγ and a total degradation of memory is observed when crossing the bifurcation point. For thep= 7 the memory is higher in general (for 0< γ <

1) and the drastic degradation within the scanned interval disappears. When looking at the performance of the system on the NARMA10 benchmark (see Fig. 6.9), we observe forp= 1 (Fig. 6.9a) that the region of good performance has shifted towards higher values ofγ. NRMSE values of 0.15 can be found for γ as high as 1. This relaxes the strict conditions that were imposed for good performance with the single delay element. For an exponent of p = 7 the system never outperforms a linear shift register and all errors are higher than 0.4, as shown in Fig. 6.9b.

118 6 System modifications

Fig. 6.9: Two coupled nodes, NARMA10. The NARMA10 perfor-mance expressed as an NRMSE is plotted in color code for a network motif consisting of 2 bidirectionally coupled nodes, while scanning the feedback strength and the input scaling. Both nodes receive identical inputs, but preprocessed with a different mask. The two nodes are equally spaced within the τ-interval and each node drives 200 virtual nodes with a separation of 0.2. (a) p= 1, the region of good perfor-mance - NRMSE≈0.15 - reaches much higher values ofγ compared to the single delayed-feedback case (b) p = 7,the performance never reaches better than what can be achieved by a linear shift register.