Mean Error with Gaussian Noise and Mean Shift Clustering
5.6 Verification of the Best Result
The best result from evaluations of A + PL and AlexNet were further evaluated. In the previous experiments they were run once over a training set combining images from all the rats. In order to verify the results, the best structures will be evaluated with leave-one-out cross validation, which is illustrated in Figure 5.10.
Rat NN11 Rat NN15 Rat NN23 Rat NN24 Run:
1
Rat NN11 Rat NN15 Rat NN23 Rat NN24
2
Rat NN11 Rat NN15 Rat NN23 Rat NN24
4
Test Train
Figure 5.10: Leave-one-out cross validation as conducted in the experiment. The purple boxes corre-sponds to training data and the grey are used as test set.
This is done by leaving one rat out as test set and train on the combined images of the other rats.
This is repeated until all rats have been in the test set. The performance is then evaluated by the combined results.
Type/Model: NN11: NN15: NN23: NN24 Total:
A+PL/138 1.652×10−2 1.387×10−2 1.845×10−2 1.429×10−2 1.611×10−2 A+PL/112 1.652×10−2 1.387×10−2 1.845×10−2 1.429×10−2 1.610×10−2 AlexNet/8 1.873×10−2 1.743×10−2 2.116×10−2 1.781×10−2 1.860×10−2
Table 5.10:Results of leave-one-out cross validation.
The results shows that the A + PL structure number 112 has the best overall performance.
According to the results the A + PL is also expected to yield better results in general than a pre-trained AlexNet network.
The top performing structure is implemented in the system and produces predicted positions as displayed in Figure 5.11.
(a)Reconstructed points from rat NN11. (b)Reconstructed points from rat NN15.
(c)Reconstructed points from rat NN23. (d)Reconstructed points from rat NN24.
Figure 5.11:Reconstructed image points, where blue points are the original vesicle positions and where purple points corresponds to the predicted positions.
The predicted synaptic vesicle positions from Figure 5.11 is not inside the image patch. Instead the visible non-zero elements are close to the location of the observation axes. In Figure 5.11a, the reconstructed images are more noisy than the rest. This is not because of overfitting, but because the biggest rat dataset is used for testing. This means that there are few samples left for training, and the model never learns to approximate the zeros of the signed distances in the encoded signal. Oddly enough, this does not put the cost of the test set into bigger numbers.
This may be because of the noisy points acting more similar to the random projected encoded signed distances.
Chapter 6
Discussion
In this chapter the results of the experiments are summarized and further discussed. The overall performance relies on every part of the system functioning in harmony. Addressing which parts that can be held responsible for good or bad performance is a step to improve a future similar system.
6.1 Dataset
A problem concerning most machine learning applications is the amount of available data. In the experiments conducted with neural networks, the training and validation curves are experi-encing high variances. This is a indication that the amount of training data is insufficient. Also the data that is annotated only originates from 7 cylindrical cuts from 4 different rat brains. In Figure 3.2 two consecutive images from the same cut is displayed. The consecutive images have small differences because of the brain structure thickness. This can make it difficult for a neural network to map the coherence between changes in the encoded signals to the changes in the images. For processing the images as 2D data the diversity should be greater to make it easier for the neural network to learn the coherence.
If the images from each cut were to be processed as 3D data only the diversity between the cuts would have mattered. But since the synaptic vesicles has a smaller diameter than the thickness between each image they won’t appear in two consecutive images. Therefore, 3D processing of the data is not preferable over 2D processing.
6.1.1 Pre-Processing
The goal with the pre-processing was to optimize the data for the neural network. The data was bias corrected to remove the additive bias field. With histogram equalization the synaptic vesicles becomes more distinct in the images. The Gaussian smoothing is added because of the
existing noise in the images.
6.1.2 Data Augmentation
After the image processing, the images were split into128×128image patches with and with-out overlapping. The original images was in different shapes, in order to process them in a CNN with FC layers the images either had to be resized or split into equal dimensions during aug-mentation. The unlabelled images was split into non-overlapping tiles and further augmented with one random rotation each. The non-overlapping ensured that an increase in similarity be-tween the already similar image patches was avoided. The annotated images were augmented with overlapping due to the insufficient number of images. Due to the fact that only some parts of the images had annotated synaptic vesicles, only the image patches containing annotations were kept. It was proposed to use overlapping of99.2%or127pixels at each translation which would correspond to almost4.5million image patches. But as all of these images would origi-nate from the same68images the small diversity would not have defended the amount of data.
Therefore, overlapping with87.5%or112pixels at each translation was used resulting in23223 image patches instead.
6.2 Compressed Sensing
Both the encoding and decoding of CS was shown to not generate error in the reconstructed image points with a compression rate up to2.0. In the decoding process the basis pursuit was the applied minimization problem for the recovery algorithm. However, other methods such as Lassocould have been applied but where not looked into in this thesis[34].
6.3 Neural Networks
From the results, illustrated by the figures in Figure 5.11, it is clear that the best neural network model is not able to learn the signed distances in the encoded signal. A great variety of struc-tures were evaluated, but none were able to learn the actual synaptic vesicle positions. This is reflected in the training and validation curves in figures 5.8 and 5.9, where the models struggles to converge.
6.3.1 Autoencoder
Traditional autoencoders are looking for every feature that can be helpful to reconstruct the input and later to classify the total image. In the CNNCS model the goal is not to classify the whole image but to generate predictions on positions of a few elements in it. An autoencoder should therefore be used with caution. When the encoder parts compress the image data with
75%there is a chance that valuable features contained about the synaptic vesicles gets removed in the optimization process of the autoencoder.
6.3.2 Processing Layers
The performance of the processing layers depends on the ability to learn the random projection caused by CS combined with the ability of extracting information of the synaptic vesicles from the autoencoders encoder part. The best model structure has a test cost corresponding to1.610× 10−2, which through experiments conducted on the framework of the system is not satisfactory.
In total;252different structures were tested. Training without encoder part from an autoencoder quickly turns into overfitting. This indicates that for small datasets it is beneficial to use an autoencoder to prevent the aforementioned problem.
6.3.3 AlexNet
The use of a pre-trained AlexNet with tuned structure was first suggested by Xue and Ray on cell detection using CNNCS[8]. Due to the similarity between their cell images and the synaptic vesicle images it was reasonable that the same structure would be able to predict the synaptic vesicle positions as well. But in reality, with the dataset available it actually performs worse than the A + PL structure. That none of the models are able to accurately predict the encoded signals suggests that the problem is not with the neural network, but with the training data.