Possible applications of dataset

One way to apply this dataset is for experiments similar to what we do in the later chapters of this thesis. As stated in earlier sections, our intention is to apply this dataset to the training of emotional intelligent machines using reinforcement learning. Our general approach is to train a neural network to reproduce the signals recorded from humans, on the basis of the current game state. The reproduced signals can then be incorporated into the reward function of a reinforcement learning agent and, given a strong emotional correlation, could be used to mimic human emotional responses, creating machines with a certain kind of emotional intelligence. The idea is that this kind of emotional intelligent machine might aid the pure logic of reinforcement learning models and produce improved learning, as well as possibly reveal new insights into how both machines and humans learn.

They might also be a step towards machines with even more complex emotional intelligence, like the ability to recognize, express, and respond to human emotions. The Toadstool dataset, all though limited to the SMB context, can be a great starting point for certain avenues of such work.

We hope that the many channels of information provided in the Toadstool dataset can facilitate a range of different, and ideally complementary, approaches to this problem, contributing to the future of this field.

Furthermore, we believe there are many other types of scenarios and research, for which the dataset could be useful. One possibility is to use it to detect relationships between the player sentiment and the current gameplay state. This could be done solely based on the gameplay frames

and collected sensor data, or could also be combined with the player’s facial expression for further analysis. The uncovered relationships could be interesting when studying how players get invested in video games, and what typical scenarios contribute to a strong reaction from players.

Another example could be predicting a person’s facial expression based on a given game state or degree of progress in a game or level.

This could also be expanded to the sensor data, as one could predict the sensory output of the wristband using the game state and facial expressions as input, like photoplethysmography [56] does with the heart rate, but connected to the current game state the player is in. These two problems could be modeled as either a regression or classification problem, depending on the application.

The dataset could also be useful in the field of game design. Under-standing the correlations between game progress, game state, the input of players, and the emotions and sentiment of the players could potentially enable a new approach to experience-based game design. With the possi-bility to use this as a method for playtesting, game designers can evaluate when and how their crafted experience is (or is not) invoked in the players.

Moreover, games can be built around the concept of self-adapting challenge and difficulty by counter-acting unnecessary frustration, boredom, or an-noyance by adapting the game to the player’s emotion and sentiment. Last but not least, the correlation between the game, emotions, sentiment, and game engagement, especially flow (the state of being fully immersed in an activity while enjoying it), can be investigated [39, 71]. This could lead to a better understanding of what makes games enjoyable and impact fields like game and media studies, psychology, game engineering, game design, and educational games.

Chapter 4

Simulating Emotion using the Blood Volume Pulse

After having created the necessary dataset, the next step was to use the data collected, and find a way to train an algorithm to predict a signal that correlates to the recorded emotional reactions. As we have seen in the previous chapters, the Toadstool dataset [70] provides a variety of data that can be used, independently or in combination, to attempt to achieve this. As our project must maintain a limited scope, we do not have the opportunity to investigate the many different possible implementations here. This is, however, an interesting available path for future work.

Instead, we rely upon what has been shown to be effective in the past.

Once again, we draw inspiration from the work in the Visceral Machines project [46]. As discussed in section 2.7.4, this work used a deep neural network to predict the BVP in a self-driving system. Our approach will be to apply the same principle in our own context of playing Mario, following the same process as closely as we reasonably can. In this chapter we will briefly explore the methods and history of the BVP measurement, as well as describe in detail the procedure we used to obtain, process, and train neural network models to predict, the BVP signal amplitudes. We also perform some evaluations of the performance and behaviour of the trained models.

4.1 Photoplethysmography, Fight or Flight, and Lever-aging the Blood Volume Pulse

In this section we will look at the measurement of photoplethysmography (PPG) and how we can use it to obtain the BVP. We will also discuss how the BVP is related to visceral emotional reactions in the sympathetic nervous system, and how these can be leveraged for reinforcement learning.