4.3 Experiments regarding Privacy Issues in Federated Learning
4.3.2 Memorization in Federated and Centralized Learning
This section will describe an experiment performed to observe the occurrence of memorization in fed-erated and centralized learning. This experiment was inspired by the fact that information leakage is a well-known issue in machine learning models, often due to memorization as described in Section 2.5. We wanted to explore if the issue of memorization posed the same threat in federated learning as in central-ized learning. In addition, we wanted to see how the size of the model used in training would affect the degree of memorization. To obtain results about memorization in federated and centralized learning, we decided to train both learning approaches with the artificial neural network and the 1D convolutional neu-ral network. The experiment would only provide accurate results if we were to make sure that one type of the data would only occur a few times in the dataset. In order to achieve this, we manipulated the dataset
by reducing theUnknown class from 20000 examples to 100 examples. The remaining classes contained 20000 examples each.
Federated Learning
Table 25 describes the training configuration used in the memorization experiment using federated learn-ing.
Training Configuration
Learning algorithm: Federated
Aggregation method: Federated Averaging (FedAvg)
Data distribution: Non-IID
Epochs: 15
Client Epochs: 10
Total number of clients: 10
Number of participating clients per round: 10
Server optimizer: SGD
Server learning rate: 1.0
Client optimizer: SGD
Client learning rate: 0.02
Loss function: Categorical Cross-Entropy
Table 25: Training configuration for the memorization experiment using the FedAvg algorithm.
4.3.2.1 FedAvg with ANN
This section will present the results achieved when the ANN model was trained with the FedAvg algo-rithm while forcing memorization.
Metrics
Test Accuracy: 92.9%
Training Accuracy: 98.3%
Test Loss: 0.29
Training Loss: 0.05
Training Time: 281 s
Table 26: Accuracy, loss and training time for the memorization experiment using the FedAvg algorithm with the ANN model. The metrics illustrated in this table describe a well-performing model with a rela-tively short training time.
Classification Report
Class Precision Recall F1-Score Support
Normal 0.98 0.95 0.96 18118
Supra Ventricular 0.44 0.83 0.57 556
Ventricular 0.79 0.94 0.86 1448
Fusion 0.41 0.90 0.56 162
Unknown 0.99 0.71 0.83 1608
Table 27: Classification report for the memorization experiment using the FedAvg algorithm with the ANN model. The table describes the precision, recall, F1-Score and support values for the experiment.
From the F1-scores, one can observe that the model performed well on theUnknownclass despite the model only having seen training examples of the class a hundred times.
N S V F U
Predicted label N
S V F U
True label
0.95 0.03 0.01 0.01 0.00
0.15 0.83 0.02 0.01 0.00
0.02 0.01 0.94 0.03 0.00
0.05 0.01 0.05 0.90 0.00
0.16 0.03 0.10 0.00 0.71
0.0 0.2 0.4 0.6 0.8
Figure 49: Confusion matrix for the memorization experiment using the FedAvg algorithm with the ANN model. The confusion matrix illustrates a clear diagonal, indicating a high number of true positives and true negatives. One can also observe that the model mostly classified correctly for theUnknownclass de-spite it not having seen more than a hundred training examples of the class.
4.3.2.2 FedAvg with CNN
This section will present the results achieved while training the 1D CNN model with the FedAvg algo-rithm.
Metrics
Table 28: Accuracy, loss and training time for the memorization experiment using the FedAvg algorithm with the CNN model. The metrics shown in this table describe a well-performing model with a relatively long training time.
Table 29: Classification report for the memorization experiment using the FedAvg algorithm with the CNN model. The table describes the precision, recall, F1-Score and support values for the experiment.
From the F1-scores, one can observe that the model performed well on theUnknownclass despite it only having seen training examples of the class a hundred times. The model did also have high F1-scores for the other class, indicating a high true positive rate and a high true negative rate.
N S V F U
Figure 50: Confusion matrix for the memorization experiment using the FedAvg algorithm with the CNN model. The confusion matrix shows a clear diagonal. One can observe that the model largely classified correctly for theUnknownclass despite it only having seen a hundred training examples of the class.
Centralized Learning
Table 30 describes the training configuration used in the memorization experiment using centralized learn-ing.
Training Configuration Learning algorithm: Centralized
Epochs: 15
Server optimizer: SGD
Learning rate: 0.01
Loss function: Categorical Cross-Entropy
Table 30: Training configuration for the memorization experiment using centralized learning.
4.3.2.3 Centralized Learning with ANN
This section will present the results achieved while training the ANN model with centralized learning.
Metrics
Test Accuracy: 92.7%
Training Accuracy: 98.2%
Test Loss: 0.27
Training Loss: 0.05
Training Time: 190 s
Table 31: Accuracy, loss and training time for the memorization experiment using centralized learning with the ANN model. The metrics illustrated in this table describe a well-performing model with a low training time.
Classification Report
Class Precision Recall F1-Score Support
Normal 0.96 0.96 0.96 18118
Supra Ventricular 0.55 0.78 0.65 556
Ventricular 0.83 0.92 0.87 1448
Fusion 0.33 0.88 0.48 162
Unknown 1.00 0.59 0.74 1608
Table 32: Classification report for the memorization experiment using centralized learning with the ANN model. The table describes the precision, recall, F1-Score and support values for the experiment. From the F1-scores one can observe that the model performed well on theUnknown class despite it only hav-ing seen trainhav-ing examples of the class a hundred times. However, the model performed worse on the Un-known class compared to the memorization experiment using FedAvg described in Table 27.
N S V F U Predicted label
N S V F U
True label
0.96 0.02 0.01 0.01 0.00
0.19 0.78 0.01 0.01 0.00
0.04 0.01 0.92 0.03 0.00
0.07 0.01 0.04 0.88 0.00
0.31 0.02 0.08 0.01 0.59
0.0 0.2 0.4 0.6 0.8
Figure 51: Confusion matrix for the memorization experiment using the FedAvg algorithm with the ANN model. The confusion matrix shows a relatively clear diagonal. One can observe that the model classified correctly for theUnknownclass far less than for the other classes, but it still performed decently on this class despite only having seen training examples of the class a hundred times.
4.3.2.4 Centralized Learning with CNN
This section will present the results achieved while training the 1D CNN model with centralized learning.
Metrics
Test Accuracy: 94.2%
Training Accuracy: 99.2%
Test Loss: 0.30
Training Loss: 0.02
Training Time: 403 s
Table 33: Accuracy, loss and training time for the memorization experiment using centralized learning with the CNN model. The metrics illustrated in this table describe a well-performing model that had a decent training time.
Classification Report
Table 34: Classification report for the memorization experiment using centralized learning with the CNN model. The table describes the precision, recall, F1-Score and support values for the experiment. From the F1-scores one can observe that the model performed extremely well on theUnknown class, despite only having seen training examples of this class a hundred times. The F1-score for theUnknown class is slightly higher for this experiment compared to the memorization experiment using FedAvg with the CNN model described in Table 29.
Figure 52: Confusion matrix for the memorization experiment using centralized learning with the CNN model. The confusion matrix shows a clear diagonal. One can observe a high rate of true positives and true negatives for theUnknown class despite it only having seen training examples of this class a hundred times.