Chapter 2 lays the foundation for the theoretical preliminaries. The chapter starts with in-troducing concepts from linear algebra, particularly norms. Secondly, neural networks and deep learning is introduced, covering neural network architectures, the universal approxi-mation theorem and neural network optimisation. The concept of neural network robust-ness is also treated in detail together with common tools for attempting to robustify them, such as`2norm regularisation. The concept of persistency of excitation is also introduced and discussed in relation to neural networks. Lastly, nonlinear stability theory concepts are presented, particularly input-to-state stability
Chapter 3 presents the main contributions (theoretical methodology). The regular LSTM neural network equations are altered to account for the principle of persistency of excitation.
Two training procedures acting on the altered equations are presented. Lastly, an input-to-state stability analysis on the altered LSTM equations is presented.
Chapter 4 describes the practical methodology. The dataset generation process is de-scribed together with the resulting dataset. Neural network configuration aspects are dis-cussed. Lastly, four experiments are described, rooted in the objectives of the thesis.
Chapter 5 presents results and discusses the findings.
Chapter 6 concludes the thesis and provide recommendations for further work.
Chapter 2
Preliminaries: linear algebra, deep
learning, robust optimisation and stability theory
Chapter 2 will provide the theoretical framework necessary for discussing robustness and stability of deep learning algorithms, particularly RNNs. Initially, some important results from linear algebra will be presented in Section2.1. ANNs and an overview of the most im-portant components in the optimisation of a neural network is given in Section 2.2. The concept of robustness in neural networks will be treated in Section 2.3. Following, persis-tency of excitation in light of neural networks, with its relevance to robustness, is introduced in Section2.4. The chapter ends with nonlinear stability theory in Section2.5.
Please note that parts of this chapter stem from a specialisation project [12] conducted by the author during autumn 2020. If the reader is familiar with the work, these parts may be skipped. The relevant subsections are repeated in this thesis with some minor changes due to their relevance to the work in this master thesis. The subsections that are included from [12], will be clearly marked with a dagger symbol (†) in each subsection title and a double dagger (‡) symbol at the end of each subsection.
2.1 Linear Algebra
This chapter will introduce some results from linear algebra, particularly norm functions.
For a more in-depth review of the fundamentals of linear algebra, we refer to [25, Chapter 2], which is the main reference for the material in Section2.1.
2.1.1 Vector norms
The reference for the vector norm material is [25, Chapter 2]. A vector norm is a function on a vector space that provide a distance measure. They serve the same purpose as the absolute
9
value in the scalar case. The real coordinate spaceRntogether with a norm onRnconstitute a metric space
Definition
A vector norm is defined in Definition2.1.1. As we see, three conditions ought to be satisfied:
positivity, triangle inequality/subadditivity and homogeneity.
Definition 2.1.1. A vector norm onRnis a functionkk:Rn7→Rsatisfying the following prop-erties: Special class of norms: p-norms
Different norm classes are distinguished by subscripts onk·k. A useful class of vector norms that will be used extensively are the p-norms. The p-norms are defined in eq. (2.1)-(2.2),
kxkp=(
Some frequently used p-norms are the 1-, 2- and∞-norms, given in eq. (2.3), (2.4) and (2.2), kxk1= |x1| + |x2| + · · · + |xn| (2.3) kxk2=(|x1|2+ |x2|2+ · · · + |xn|2)12 =(xTx)12 (2.4) Vector norm properties
Hölder inequality is a fundamental inequality concerning p-norms, given in eq. (2.5),
°
By choosing p=q=1, we get a special case of this inequality calledCauchy-Schwarz inequality.
This is given in eq. (2.6),
There exists constantsk1,k2such that all norms onRnare equivalent. Assumek·kαand k·kβare norms onRn. In such a case, the inequality given in eq. (2.7) holds for allx∈Rn,
k1kxkα≤ kxkβ≤k2kxkα (2.7)
2.1. LINEAR ALGEBRA 11
2.1.2 Matrix norms
Matrix norms are important in analysing algorithms or systems involving matrices. For ex-ample in linear system theory, evaluating how sensitive a system is to noise/data error, the matrix norm (more specifically, the spectral norm, which will be defined in the coming sec-tion) of the state matrix, provides a measure of the amplification of noise. Similar to vector norms, matrix norms provide a measure of distance on matrix space. We will adapt the nota-tion of [25] and denote vector with lowercase symbols, and matrices with uppercase symbols.
Definition
The definition of a matrix norm ought to be equivalent to the definition of a vector norm.
This is due toRm×nis isomorphic, i.e. identical in structure, toRmn[25, Chapter 2.3].
Definition 2.1.2. A vector norm onRm×nis a functionkk:Rm×n7→Rsatisfying the following properties:
kAk ≥0 ∀A∈Rm×n kA+Bk ≥ kAk + kBk ∀A,B∈Rm×n
°
°k A°
°=°
°k°
°kAk ∀k∈R,A∈Rm×n Matrix p-norms
Just as with vector norms, we use subscripts to identify different classes of matrix norms. A important class of matrix norms are the p-norms, defined in Definiton2.1.3
Definition 2.1.3 (p-norm). The p-norm of a matrix A ∈Rm×n is the p-norm of the largest vector that is produced by applying the matrix Ato a unit p-norm vector x∈Rn:
kAkp=sup
x6=0
kAxkp
kxkp
= max
kxkp=1kAxkp (2.8)
The spectral norm is defined by letting p=2 in Definition2.1.3. An alternative formulation of the spectral norm is given in eq. (2.9).
kAk2=
qλmax(ATA)=σ1(A) (2.9)
whereσ1(A) denotes the largest singular value of the matrix A. A drawback of the 2-norm compared to the 1-norm and∞-norms, is that it is more convoluted to compute. A common method is to apply the singular value decomposition to the matrix. Applying singular value decomposition is computationally heavy, and thus in practice, the largest singular value is most often estimated using the power iteration method. We refer to [25] for a description of the method.
2.1.3 Hadamard product
The Hadamard product [25, Chapter 12] appears frequently in machine learning and denotes a pointwise product between two matrices.
Definition 2.1.4. Given two matrices with the same dimension A∈Rm×n and B∈Rm×n, the Hadamard product (A¯B)∈Rm×n is a matrix with the same dimensions as its operands, where each element is given by the product given by eq.(2.10),
(A¯B)=(A)i j(B)i j (2.10)
where(A)i jdenotes the element at column i and row j for matrix A, and likewise for matrix B.
Note that the Hadamard product is represented by a wide number of symbols:¯,◦,⊗. In this thesis, the symbol¯will be used. Example2.1.1shows the Hadamard product between two 2×2 matrices.
It is sometimes useful to get rid of the Hadamard product operator when doing vector and matrix operations. Indeed, the relation in eq. (2.12) is true:
a¯b=