Game session setup and capture

An important requirement for the study was the ability to capture the relevant data from the game sessions of the participants. The gym-super-mario-bros framework uses the nes-py emulator¹, which is based on the

1https://github.com/Kautenja/nes-py

SimpleNES emulator². By making some adjustments to the script used by the nes-py emulator to play as a human from the keyboard in the gym-super-mario-bros environment, we were able to modify the system so that we could play in the environment as a human while recording the information we needed from the play-session.

To record the game session itself, it was sufficient to store a list of each action, or lack of action, performed at each step of the game-loop. As the SMB environment is deterministic, such a list of actions can be used to recreate a game session at a later time. This was deemed preferable to extracting the frames during the play session itself, as images of each frame can then be extracted later at our convenience, without disturbing the processing of the game session with heavy disk operations. Another benefit of doing this is that it drastically reduces the size of the resulting dataset, as the total size of all the frames from a game session can be quite large.

The reduced size makes storage, transfer and sharing of the dataset much more convenient. Furthermore, gym-super-mario-bros offer four different graphic environments to play the game in. These include the standard graphics, as well as three different down-sampled versions (down-sample, pixel, and rectangle). While the data in our dataset is collected in the standard environment, since the actions are recorded, sessions may be replayed in any of the other environments if preferable. The beginning and end of each session was marked with a UNIX timestamp stored with the actions, so that it could later be synchronized with data captured through other channels.

The experience of playing in the gym-super-mario-bros environment has some distinct differences from playing in the originalSuper Mario Bros.

game released by the Japanese gaming company Nintendo in 1985. In the gym environment there is no music, nor is there any of the sound effects that usually accompany events in the game. All game-freezing animations and cut-scenes, such as transitions between levels, traveling through pipes etc., have also been removed from the game. While the gym-super-mario-bros framework offers setups for playing that are designed for reinforcement learning contexts, like replaying from a single level, over and over, with only a few lives, these are not very suitable for the kind of human play sessions we needed for the data collection. Therefore, we built a custom setup for playing the game that seemed more appropriate for our purposes by making some additional modifications to the code.

Specifically, we used the gym-super-mario-bros option for playing a single level at a time as a base and created custom methods for transitioning from one level to the next.

In the session, players are tasked with completing as many levels as possible within the time given, with no limit on lives. If they die in the game, they are transported to the beginning of the level. If they complete the level they go to the next one and so on.To ensure that players would not end up stuck on a single level they were unable to complete, we implemented a time limit so that if a player died after spending a certain

2https://github.com/amhndu/SimpleNES

Figure 3.1: Frames are taken from each of the 32 levels contained within Super Mario Bros. Note that each image is taken from the very first frame of each level. Levels inSuper Mario Bros. are organized in groups of four and called worlds, so the first level is world 1-1, the second level is world 1-2, the fifth level is world 2-1, etc.

amount of time (about 4 minutes and 30 seconds) on the same level, they will restart on the next one.

The varying skill level of players meant that we could not know in advance how many levels each participant would be able to complete, and therefore play, during the course of their session, beyond the seven or so levels that is required by our setup due to the time restrictions described above. Therefore, in order to ensure that we were able to collect adequate samples from various situations in the game, the levels played were set up in a specific order so that the players would be more likely to encounter a broad range of situations during their playthrough. The custom order is somewhat similar to that of the original game, but with some key differences.

The SMB game contains 8 worlds, each consisting of four levels, for a total of 32 standard levels. Examples of frames taken from each of these levels can be seen in Figure 3.1. There are several types of levels;

overworld, underground, underwater, athletic (platforms) and castle, each with different situations and characteristics. The first five levels the players face are therefore each a representation of one of these types of levels. For simplicity these are just the first examples of such levels that appear in the game.

There are also some rare enemies in the game, namely; Hammer-Bros, that throws hammers in arcs, and Lakitu, which flies around the top of the screen, throwing spiky enemies at Mario. The 6th and 7th level in the ordered setup were therefore set to the first levels where these enemies show up in the game. The subsequent levels given are the skipped levels up to this point, followed by the rest of the game’s levels, in their sequential order.

Order: World Stage

1 1 1

2 1 2

3 1 3

4 2 2

5 1 4

6 3 1

7 4 1

8 2 1

9 2 3

10 2 4

11 3 2

12 3 3

13 3 4

14 4 2

15 4 3

16 4 4

. . .

. . .

32 8 4

Table 3.1: The custom stage order used in the data collection.

The customized order of the levels that was used for the data collection can be seen in Table 3.1. By ordering the levels in this specific sequence we hope to increase our chances of collecting the appropriate data for all the different, relevant game-states.

Other significant features to note in the final setup the are;

1. Participants play for a set amount of time 2. There is no limit on lives in the game 3. The levels are played in the custom order

4. Power-ups do not carry over to subsequent levels

5. Warp zones are off-limits (players must be informed of this if necessary)

We also adjusted the parameters of the image viewer so that players would get a more appropriately sized game-window.