This section will present the approach taken in order to identify the optimal machine learning model. Namely the investigation of a superior feature set and classifier. Finding any one of these in an early stage will help to ease the computational complexity. The approach is shown in figure 3.12. For every classifier and for every feature set, the steps in figure 3.13 will be taken. In more detail, the steps are:
1. For every classifier:
Decision tree, AdaBoost and SVM, all mentioned in section 2.3.2.
2. For every feature set:
Feature set one to five, all mentioned in section 3.3.
3. For all combinations of two classes:
Baseline - Adrenaline, Baseline - Esmolol etc.
(a) Produce a result:
That is, proceed through the steps in figure 3.13 where the feature set, the classifier, and the raw data all have been specified in the above steps.
The investigation will be conducted using two dysfunctions at a time. This is the easiest classification task and is expected to yield the best results.
With five dysfunctions and two dysfunctions at a time, this will in total yield ten different combinations. Having a vast number of results makes any optimal classifier, and feature set, if any, evident. Consequently, the underachieving alternatives can be discarded. Additionally, the graphs in section 3.2.4, shows the overlap between the dysfunctions. Hence, the 5-class 5-classification task is expected to be challenging. The focus is therefore on the easier two-class classification task.
Furthermore, there are some combinations that are clinically more impor-tant than others. This is because certain cardiac functions are expected dur-ing or after surgery. The interestdur-ing combinations are the ones involvdur-ing baseline versus the others as this is either the initial or desirable heart func-tion. Also, esmolol versus ischemia is clinically interesting because patients
3.6. APPROACH
can be given esmolol to decrease their heart rate, but given too much es-molol would cause the heart to be ischemic. A two-class machine learning model trained on esmolol and ischemia examples can assess if the heart is in the “esmolol-state” or is ischemic.
In addition, an assessment of whether or not feature scaling aids the clas-sification will be done. The training dataset will be standardized, meaning having zero mean and unit variance. The mean and variance of the training dataset will then be applied to the testing set in the following manner:
x= x−µ
σ (3.1)
whereµandσare the mean and variance, respectively.
Having possibly identified the optimal classifier and feature set, the pro-cess in figure 3.12 will be left. The next step will be to experiment with the best model in an attempt to further improved them. Lastly, the classes will be added to the best machine learning models. The results are expected to drop for each addition of a class as the feature space of the classes is somewhat to overlapping. This is seen in section 3.2.4.
Start
For all classifiers
For all feature
sets
For all combinations of
2 classes
Produce result as shown in
fig. 3.13
Stop
Decision tree/AdaBoost/SVM
Feature sets 1, 2, ..., 5
Baseline and Adrenaline/Baseline and Esmolol/
Ischemia and Volume/ ...
Figure 3.12: Figure showing the 2-dysfunction-classification task which aims to find the best feature set and classifier. The diamond boxes represent loops while the produce result block is specified in figure 3.13.
The parallelograms represents an input and output.
3.6. APPROACH
Raw data
Extract cycles Extract features
Feature selection
Train and test split
Training set
Train classifier
Model Test
classifier
Partial result
Loop
Average results
Final result Produce
result
Test set
Figure 3.13: Figure showing the steps in creating a model and achieving a result. Rectangles represents a process that takes an input and produces an output. The dashed rectangle is a process that is sometimes skipped.
White ellipses represents a result from a process. The parallelograms represents an input and output.
Details of produce result
1. Raw data The dataset as it were when received/obtained, described in 3.2.1
2. Extract cycles This step will extract cycles from the raw data as described and reasoned in section 3.2.2.
3. Extract features This step will create one of the six feature sets mentioned in section 3.3.
4. Feature scaling This step takes all features as input, and standardizes the features as specified previously in this section and reasoned in section 2.4.3.
5. Feature selection Feature selection will choose the best subset of features. It is only applied to feature set four and five.
6. Loop The following steps will be repeated:
(a) Train and test split Here, the dataset will be split into a training set and testing set as described in section 3.4. That is, the heartbeats from one animal that has not previously used as a testing test in the current realization, will be used as such, and the rest will be used in the training set.
(b) Train classifier This steps takes the training set as input and trains a classifier.
(c) Model The training phase outputs the final model. That is, a model that now can make predictions on new motion data (d) Test classifier This step takes the model and testing set as inputs
and uses the model to predict the classes in the testing set.
(e) Partial result The true dysfunction is known, hence the result from the previous step is stored. If every animal has been used in the testing set, the next step is number five. If not, the loop starts over again from the (a) step.
7. Average result All partial results stored in the step above is averaged to produce a final result which is also the output.
8. Final result The final results consist of several measures, specified in section 2.6.1.
Chapter 4
Experiments and results
This chapter intends to present the results of all the models described in the previous chapter. The results of the two-class classification task will be investigated first, in section 4.1. The best results will be used to further improve the best models, described in section 4.2. The results of the best performing models will then be presented in section 4.3. These model will be used when adding classes to the classification task, starting from section 4.5.
4.1 Two-class experiments and results
Table 4.1 shows the results achieved when carrying out the approach shown in figures 3.12 and 3.13.
Feature set Classifier Direction Accuracy
Decision tree circ. 0.66 0.66
SVM circ. 0.54 0.62
Table 4.1: Table showing the results after following the approach shown in figures 3.12 and 3.13. The three best results are marked with bold text. The various feature sets are specified in section 3.3.1.
4.1. TWO-CLASS EXPERIMENTS AND RESULTS
Note that the accuracy is the average accuracy of the combinations of two classes (Baseline versus adrenaline, baseline versus esmolol etc.). All experiments were done twice, both with and without feature scaling. The three best results are marked with bold text. The results create a basis on what to pursue next.
4.1.1 Analysis
Initially, it is seen that almost all models perform better than chance (50%).
This implies there is difference in motion, which to some degree can be dis-criminated by certain machine learning models. Though most of the mod-els performed only slightly better than chance, there are some modmod-els that should be further investigated. How the models are tried to be improved is described in section 4.2. An analysis of the initial results are given in the paragraphs below.
Regarding feature scaling, it had an effect only on the SVM classifier. de-cision tree, and AdaBoost which uses dede-cision trees as its classifier does in contrast to the SVM, not rely on the distance between feature vectors. In-stead, the rely on the intrinsic values of the features, hence feature scaling did not affect the results and will be discarded in future experiments.
Regarding the three different directions, the circumferential direction per-formed better than the longitudinal direction (P < 0.05), but were not sig-nificantly better than the radial direction (P = 0.066). However, in order to limit the number of models created in further experiments, features in the longitudinal and radial direction will still be discarded.
Furthermore, the three models with highest accuracy all used feature se-lection. This shows that feature selection aids the models. Feature selection will thus be further inspected.
It is worth noticing that feature set one which uses features from the midsystolic phase, were amongst the highest performing feature sets. This supports the cited articles in section 3.3.1 and 2.2.3, which favors the mo-tion in the same phase.
There were some differences in the performance when comparing the clas-sifiers. The mean accuracy of decision tree, SVM and AdaBoost were 66, 61 and 63 respectively. Both decision tree and AdaBoost performed signifi-cantly better than SVM, (P < 0.005 on both). Further experiments will there-fore not use SVM as the classifier. The results using AdaBoost and decision tree are similar (P = 0.83), but AdaBoost is considerably more computation-ally expensive due to its iterative behavior. Decision tree will therefore be used in further experiments, and AdaBoost tested in the final experiments on only the best models.
Experiments were not conducted with features from different directions or
the raw position data. This is further addressed in section 4.2.3.