On the Acquisition of Human Emotions in Space and Time

(1)

F. Biljecki and V. Tourre (Editors)

On the acquisition of human emotions in space and time

Julius Schöning and Corinna E. Bonhage

Institute of Cognitive Science, University of Osnabrück, Germany

Abstract

Emotion assessments traditionally take place in academic or medical settings. The impact of environmental factors such as e.g. the characteristics of the place, the weather, or the time of day on the emotional state of inhabitants are often dismissed, thus emotions as a dynamic feature cannot be used for city planning or development. To estimate emotions as spatio-temporal data, a growing research community focused on emotion extraction from social network platforms such as Twitter or Facebook. The quality and the reliability of the resulting emotion evaluations are not comparable to direct emotion assessments, e.g. because of possible biases of emotional statements in social communities. In order to bridge this gap, we designed a web-based survey service that allows for the acquisition of geo-located emotion ratings of a huge group of participants in real-world environments, which in turn can be related to numerous environmental variables. In this paper, we present the architecture of this web-based service along with first results of a pilot study. In the end, we discuss possible use cases for spatio-temporal emotion data, the limitations of our web-based service, and present an outlook on possible future projects.

Categories and Subject Descriptors (according to ACM CCS): H.2.8 [Database Management]: Database Applications—Spatial databases and GIS

1. Introduction

How does the current location influence the emotion and the behavior of people—and can this data be visualized in a way that improves our understanding of our surround- ings? Those were the two initial questions inspiring us to develop a tool for spatio-temporal emotion acquisition. Cur- rently the most prominent way of acquiring emotion data is their extraction from a social network platform such as Twitter[BPM09,TSA14]. These approaches mainly use sentiment analysis based on trained keywords and phrases or emoticons, and they work reasonably well on large datasets.

Why might this approach not be suitable for smaller cities in Germany? In Germany, the number of unique visitors (number of unique people requesting a homepage) ofTwit- ter decreased from 4.1 million to 3.6 million from 2012 to 2014 [Sta15], respectively. In less populated areas and smaller cities like Osnabrück, the amount of available social network data is insufficient to apply data-mining approaches.

A different approach to assessing emotion data suggested here is to take advantage of the heavy smartphone use in Germany: According to a study on mobile internet usage in Germany, in 2014 over 69% of internet user accessed the internet on mobile devices [Ini14]. In order to obtain the emo-

tion of people in time and space, we developed a web-based service for geo-located assessments.

In the following sections, we will start by briefly outlin- ing the state of the art in spatio-temporal/geo-located surveys, emotion mining and environmental effects on wellbeing. This theoretical introduction is followed by a descrip- tion of our pilot project, in which we created the architecture of our web service and piloted its functionality in Os- nabrück. After presenting first pilot data results, we discuss possible application cases for spatio-temporal emotion assessments, highlighting opportunities, as well as limitations, of our web-based service in order to improve our service.

2. Geo-located services and emotion assessment Traditionally, spatio-temporal or geo-located surveys are often conducted in form of an interview at a certain place (e.g., a pedestrian mall or city park): Subjects are randomly selected to answer a questionnaire. Another form of geo- located surveys is to ask people “where” questions. To map happiness, MacKerron and Mourato [Mac15] created the iPhoneapplication “mappiness”. This application randomly asks the participants to answer a question regarding their happiness at their current location. The answer is saved to-

c

The Eurographics Association 2015.

(2)

social network platforms usually provide not only the text messages transmitted or received by a user, but also the current location of the user and additional user-related information (e.g. sex or age). Examining the text messages, sentiment analyses allows for extraction of the current emotional state of the user [BPM09,TSA14,BBMS12]. Based on this, data emotions can be mapped to countries, regions, or locations [LBB^∗15,IP13].

A different line of research focuses on the impact of the environment on our emotional state. For example, there is a growing body of literature on environmental effects on wellbeing [BDLG15,HCMN03,HH89,Kap95]. Unfor- tunately, most of the studies are conducted in artificial labo- ratory settings (e.g. presenting participants with pictures of more or less natural scenes and measuring their psychological wellbeing). In sum, comprehensive emotion studies for real-world environments with a sufficient number of participants are largely missing; one approach that might be able to bridge this gap is conducting geo-located emotion surveys on mobile devices, as proposed in the present paper.

3. Geo-located emotion surveys

Our present spatio-temporal web-based service enabled us to track the emotions and appraisal processes of people in their natural environment. For this pilot study, we were especially interested in whether or not

1. the self-rated naturalness of a scene influences the the level of relaxation (calm-excited), and

2. the current weather influences the emotional state (feeling positive or safe).

The pilot study was conducted in Osnabrück, a small Ger- man city (∼119.8km²) with a population of about 156.000 inhabitants. Osnabrück is a medium density area, character- ized by small buildings and mansion districts. In the following paragraphs, we will first (i) explicate the hypotheses regarding the emotion-environment interactions before (ii) de- scribing the details of the present web-based service for the emotion data acquisition.

3.1. Hypothesis

We hypothesized that (1) the greener/more natural a scene is, the more calm and relaxed the participants would feel. More- over, (2) we supposed that the current cloudiness (which is

Figure 1:Screenshot of spatio-temporal questionnaire for assessment of emotional status and scenic attributes. The current location, incl. its accuracy, is estimated using the HTML5 geolocation API [Wor15] and stored in the database.

partially accompanied by rain, thunder, or lighting) might reduce feelings of positivity and safety.

To evaluate these questions, we asked 30 participants (19 participants aged between 18−25 years, 11 participants aged between 26−35 years; 15 female) to rate places in Os- nabrück within a period of 10 days, resulting in 133 ratings of locations regarding six attributes (cf. Figure4(a)): Three emotions that these places elicited in the participant (i.e., how safe, positive, and relaxed the participant felt) plus three judgements regarding the place (i.e., how natural, beautiful and inviting the respective place was rated). To avoid neu- tral votes, an even number of points is chosen for the Lik- ert scale. Additionally, our web-based service automatically stored not only the time stamp and the rated location, but also extracted current weather data (e.g., cloudiness, pressure,

(3)

Figure 2:Architecture of the web-based service for acquiring spatio-temporal emotions. Front-end: The mobile device dis- played the spatio-temporal questionnaire (cf. Fig.1) and received the emotional ratings and the current location of the user.

Back-end: The back end received the acquired data via mobile internet and stored this data into separate databases (cf. Fig.3).

temperature) at the current location from the OpenWeath- erMap [Ope15] application programming interface (API). In sum, the present web-based service enables us to estimate how weather, scenic attributes, and emotions are related to each other.

3.2. Web-based service

In order to assess geo-located emotion data, similar to MacKerron and Mourato [Mac15], we benefitted from the circumstance that almost every smart or mobile device is equipped with a localization receiver based onGPS,Galileo or GLONASS. The accuracy of the location information provided by these devices via the geolocation API mostly depends on the number of satellites in the range of the respective mobile device. For example, the average accuracy of consumerGPS devices is approximately two me- ters [MKtH06], which is sufficient for geo-located surveys.

However, in contrast to MacKerron and Mourato, we de- cided to create a web-based service (instead of a smartphone application) that can be accessed with any mobile internet device and thus is not limited to a special brand or operating system.

Figure2illustrates the architecture of our web-based service: The front-end provides the user interface and is exe- cuted on the users’ device. Therefore, the front-end is able to use the device hardware—in our case, to access the localization receiver to get hold of the current geographical location.

In contrast, the back-end runs on the server and manages the database. It also provides an opportunity to handle the users’

sessions, thereby allowing for identification of the logged- in user and provide user-specific context on the website. In the following paragraphs, we describe the characteristics of front- and back-end in detail before reporting the results of the pilot study.

3.2.1. Front-end

The front-end design is based on the fifth version of the hy- pertext markup language (HTML) standard [LS11,Pil10], because HTML5 offers a standardized JavaScript interface

for geolocation. This provides universal access to the device position without requiring a specific code for each browser and device [Wor15,Ant12]. Furthermore, HTML5 imple- ments the third version of the cascading style sheets (CSS) that allow for the same layout in any browser only depending on the CSS file [Ant12].

The user (also referred to as participant) can visit the web- based service like any website via an URL (website of the pilot study:https://ikw.uos.de/~emotionmap). On the web, the user is provided with a map of Osnabrück that illustrates current places that have been rated. If one would like to participate, they can register. Once the user is registered, they can log in via username and password or using their in- dividual log-in link. The process of emoting—answering the survey questionnaire—is simple, intuitive, and user friendly:

Via visual sliders, the users indicate their emotions and rate the scene characteristics on a 6-point Likert scale (i.e., from 1 [very negative] to 6 [very positive]). The short survey questionnaire used in the present pilot study is depicted in Fig- ure1. After filling in the questionnaire, the user receives a rating summary and information on automatically collected data (e.g., a weather icon and the current emoted location on a map).

In order to provide the best possible visualization of graphics and images for all kinds of browsers, we used au- tomatic browser detection and adapted the CSS file accord- ingly.

3.2.2. Back-end

The basic back-end functionality was realized in the server- scripting language PHP. PHP is used to adjust our website dynamically and alter the HTML5 pages dependent on the respective user. In order to achieve this, PHP also provides a log-in mechanism and a secure session between the users’

devices and the server. The access of the internal and the external database was also implemented in PHP. In conse- quence, the web-based service is able to identify, for example, the first login of the user after registration, to de- tect whether or not an email address already exists in the database, and execute an inquiry request on other services.

(4)

Figure 3: Data base tables USER and SPA- TIO_TEMPORAL_EMOTION with their attributes. Each table is stored in a separate database. The spatio-temporal emotions are referred to by a foreign key to one unique user. All bold printed attributes are obligatory. The daily mood of the participant is stored in an additional table (not visualized in this figure).

These options allowed for implementation of a daily assessment of the participants’ general mood, which might influence the subsequent ratings of current emotions and locations. To ensure that participants answer the questionnaire and do not continue without rating, the slider positions were randomly set to the most extreme negative/positive values.

In order to receive the current weather data at the respective location, a PHP function triggers the API of the OpenWeath- erMap [Ope15], as shown in Figure 2. The OpenWeath- erMap service answers the request and sends all weather information as JavaScript object notation to our service, where PHP splits the information for further processing. Apart from this specific feature, PHP is used for any type of data processing on the server.

The data is stored in two encrypted SQLite tables, as illustrated in Figure 3. For security reasons and to com- ply with the German data security laws, each table is stored separately in a database at different locations. An entry in the tableUSER represents one unique user and is added to the table after a successful registration of a new user. As primary key for this table, the index attribute UserID is defined to avoid insertion anomalies. Each spatio-temporal questionnaire results in one entry in the ta- bleSPATIO_TEMPORAL_EMOTION. Additionally, in order to avoid insertion anomalies an index attribute is defined as well. The assigning ofnulltonentries of theSPA- TIO_TEMPORAL_EMOTIONtable is realized using the at- tributeUser_IDas foreign key.

ing how to enable the devices localization receiver and how to allow our service to read out the device position. In order to solve this issue we developed a detailed manual, “How to enableGPS”, and added it to the frequently asked questions page. As users mentioned they were interested in the results of the pilot study, we provided daily updates of maps with emoted locations.

4. Emotion Extraction and Mapping 4.1. Results

As mentioned above, the present pilot study was designed to investigate how scenic attributes (such as naturalness) or the current weather are related to the emotional status of human beings. In order to capture both the intra- and the interindi- vidual variance in the data, we estimated general linear models (linear mixed models algorithm implemented inSPSS22) for each of our hypotheses.

In order to compare scenes with lower naturalness scores (i.e., rather urban scenes) to more natural/green scenes, in a first step we split the ratings into two groups (according to the median of the naturalness ratings). This factor “naturalness” was then fed into the mixed models analysis in order to investigate whether or not naturalness impacts the dependent variable “relaxation status” (on a scale from one to six).

As illustrated in Figure4, urban scenes were associated with significantly decreased relaxation of the participants com- pared to natural scenes (F(1,131) =7.67,p=.006). This finding is in line with previous literature on restorative effects of green places [HCMN03,Kap95].

Our second hypothesis was concerned with the relation of the current weather and the emotional status of the participants. We thus split the cloudiness data into two groups (i.e., cloudy vs. clear sky, median split) and fed the cloudiness factor into the mixed models analysis in order to see whether the cloudiness could (partially) explain the feelings of safety and being in a positive mood. Indeed, a rather clear sky was associated with a significantly increased feeling of safety (F(1,131) =5.72,p=.018; see also Figure5(a)-(b)).

Moreover, in line with our expectation, less clouds were accompanied by a more positive feelings than a clouded sky (F(1,131) =4.65,p=.033; see also Figure5(c)-(d)).

4.2. Visualization

There are numerous approaches to visualization of georeferenced data [MBP98,HPRD10,FL78]. Nonetheless, we

(5)

(a) (b) (c)

Figure 4:How calm does this place make you feel?(a)and(b)show piloting data mapped to the city Osnabrück.(a)visualized the degree of relaxation from calm (red) to excited (blue);(b)visualized the impression of naturalness from urban (red) to natural (blue).(c)compares relaxation of the participants to natural scenes (F(1,131) =7.67, p=.006)

used a rather simple visualizing technique for the present pilot data due to two main reasons: First, the number of data points was rather small and thus, not sufficient for creating a full map layer of Osnabrück. Moreover, in this first step we focused on the functionality of the assessment tool and did not yet test sophisticated hypotheses regarding the visualized emotions. Thus, we mapped each survey question with theMATLABR v4 griddata interpolation method on the city map of Osnabrück, as illustrated in Figures4(a)-(b), 5(a) and5(c). In order to account not only for the spatial, but also for the temporal dimension of our preliminary data set, we created animated images in the GIF file format, which are also available online(cf. spatio-temporal imageshttp:

//ikw.uos.de/~emotionmap/mvotes.php). The temporal resolution of the animated images is one hour; the animated images included weather information as well.

5. Discussion

Overall, the results of the present pilot study are encour- aging: Although we only acquired few data points from a small sample of participants from a small geographic area, we already see how the environment interacts with the emotional status of people. In future, from a psychological perspective, systems like the one presented here might be key for investigating human beings in their natural environments, be it for the purpose of a general understanding of human-environment interaction or for comparing groups (e.g., healthy and clinical populations) by tracking people and their emotions in space, thereby generating a “foot- print” of psychological wellbeing in real environments.

From a technical perspective, overall the web-based service for the acquisition of spatio-temporal emotions or related spatio-temporal data works reasonably well. A main issue—apart from the fact that some participants exhib- ited problems with activating the location receiver on their device—was the overall motivation of the participants. We recognized a decreasing number of ratings over time. As a reason, participants pointed out that they forgot to rate be-

cause of a missing reminder functionality of the web-based service.

Moreover, due to the fact that the localization receiver cannot receive the satellite signals inside buildings or other shielded environments, the spatio- or geolocation is only available in outdoor environments. This circumstance some- times leads to insufficient position data; this issue cannot be solved with the present approach.

6. Conclusion and Future Work

Along with the first pilot study, we present our web-based tool for spatio-temporal emotion assessments. A comprehensive spatio-temporal mapping of emotion in cities will open up numerous new opportunities: For example, navigation devices that provide the not only the fastest, but also thesafestor themost beautifulroute depending on the time of day and the weather. Moreover, inhabitants, hikers, and tourists might use an emotion map to find emotion-specific places in a city, with an emotional compass [IP13] depicting for example where people are relaxed. Moreover, one can imagine improving city planning and development by con- sidering people’s emotions in design, in illumination as well as in the number of surveillance cameras. The possibilities of spatio-temporal emotion data are limitless, but in order to evaluate the representative nature of data extracted from social networks, social network information should be related to direct spatio-temporal emotion assessment.

In principle, the current web-based service is usable all over the world to acquire data, although the visualization of emotion data is currently limited to Osnabrück.

To overcome the drawback that our service cannot remind the participants to rate the current location, we are planning to develop a cross-platform mobile application that uses a similar database structure as our web-based service, which offers a bundle of additional opportunities like photographs or microphone recordings of the location for the acquisition of spatio-temporal data.

(6)

Figure 5:(a)and(b)How safe does this place make you feel?;(c)and(d)How positive does this place make you feel?(a)and (c)illustrate the pilot data mapped on the city Osnabrück.(a)visualizes the feeling of safety from unsafe (red) to safe (blue);(c) visualizes the emotional valence from negative (red) to positive (blue);(b)relates the current cloudiness to participants’ feeling of safety (F(1,131) =5.72, p=.018);(d)illustrates the relation of cloudiness and the emotional valence of the participants (F(1,131) =4.65, p=.033).

7. Acknowledgment

This project would not have been possible without the amaz- ing work by Lisa Freise, Renato Garita Figueiredo, Nisha Jagannathan, Jürgen Kröger, Michał Ljubljanac, Aishwarya Nair, Vivien Radtke, Maria Carolina Vargas Rivero, and Maximilian Alexander Wächter, who participated in the course "Emotion in Space" and helped to realize the idea and the pilot study.

References

[Ant12] ANTHES G.: HTML5 leads a web revolution.

Communications of the ACM 55, 7 (Jul 2012), 16. URL:

http://dx.doi.org/10.1145/2209249.2209256, doi:10.1145/2209249.2209256.3

[BBMS12] BALABANTARAY R. C., BHUBANESWAR I., MO- HAMMADM., SHARMAN.: N.: Multi-class twitter emotion classification: A new approach.International Journal of Applied Information Systems(2012), 48–53.2

[BDLG15] BRATMANG. N., DAILYG. C., LEVYB. J., GROSS J. J.: The benefits of nature experience: Improved affect and cognition. Landscape and Urban Planning 138(Jun 2015), 41–

50.doi:10.1016/j.landurbplan.2015.02.005.2 [BPM09] BOLLENJ., PEPEA., MAOH.: Modeling public mood

and emotion: Twitter sentiment and socio-economic phenomena.

arXiv preprint arXiv:0911.1583(2009).1,2

[FL78] FRANKLINR., LEWISH. R.: 3-D graphic display of dis- crete spatial data by prism maps.Proceedings of the 5th annual conference on Computer graphics and interactive techniques - SIGGRAPH(1978).doi:10.1145/800248.807373.4 [HCMN03] HERZOG T. R., COLLEEN, MAGUIRE P., NEBEL

M. B.: Assessing the restorative components of environments.

Journal of Environmental Psychology 23, 2 (Jun 2003), 159–170.

doi:10.1016/s0272-4944(02)00113-5.2,4 [HH89] HULL R. B., HARVEY A.: Explaining the emo-

tion people experience in suburban parks. Environment and Behavior 21, 3 (May 1989), 323–345. doi:10.1177/

0013916589213005.2

[HPRD10] HENNIGB., PRITCHARDJ., RAMSDENM., DOR- LINGD.: Remapping the world’s population: visualizing data using cartograms.ArcUser(2010), 66–69.4

[Ini14] INITIATIVE D21 E.V.: Mobile internetnutzung 2014, Dec. 2014. URL: http://www.

initiatived21.de/wp-content/uploads/2014/

12/Mobile-Internetnutzung-2014_WEB.pdf.1

[IP13] IACONESIS., PERSICOO.: An emotional compass.Emo- tion and Sentiment in Social and Expressive Media(2013), 181–

200.2,5

[Kap95] KAPLAN S.: The restorative benefits of nature: To- ward an integrative framework. Journal of Environmental Psychology 15, 3 (Sep 1995), 169–182. doi:10.1016/

0272-4944(95)90001-2.2,4

[LBB^∗15] LARSEN M., BOONSTRA T., BATTERHAM P., O’DEA B., PARIS C., CHRISTENSEN H.: We feel: Map- ping emotion on twitter. IEEE Journal of Biomedical and Health Informatics (2015), 1–7. doi:10.1109/jbhi.

2015.2403839.2

[LS11] LAWSONB., SHARPR.:Introducing HTML5. New Rid- ers, 2011.3

[LWPS13] LEETARU K., WANG S., PADMANABHAN A., SHOOK E.: Mapping the global twitter heartbeat: The geography of twitter. FM 18, 5 (May 2013). URL:

http://dx.doi.org/10.5210/fm.v18i5.4366, doi:10.5210/fm.v18i5.4366.2

[Mac15] MACKERRONG.: Mappiness. org. uk, Jul. 2015. URL:

http://eprints.lse.ac.uk/36319/1/Landscape_

MacKerron.pdf.1,3

[MBP98] MACEACHREN A. M., BREWER C. A., PICKLE L. W.: Visualizing georeferenced data: representing reliability of health statistics.Environment and Planning 30, 9 (1998), 1547–

1561.4

[MKtH06] MODSCHING M., KRAMER R., TEN HAGEN K.:

Field trial on gps accuracy in a medium size city: The influence of built-up. In3rd Workshop on Positioning, Navigation and Com- munication(2006), pp. 209–218.3

[Ope15] OPENWEATHERMAPINC. : Weather api, Jul. 2015.

URL:http://openweathermap.org/api.2,4 [Pil10] PILGRIMM.:HTML5: up and running. O’Reilly Media,

Inc., 2010.3

[Sta15] STATISTA: Anzahl der besucher von twitter in deutschland in den jahren 2011 bis 2014, Jul. 2015. URL:

http://de.statista.com/statistik/daten/

studie/223174/umfrage/.1

[TSA14] TORKILDSON M. K., STARBIRD K., ARAGON C.:

Analysis and visualization of sentiment and emotion on cri- sis tweets. Lecture Notes in Computer Science(2014), 64–67.

doi:10.1007/978-3-319-10831-5_9.1,2

[Wor15] WORLDWIDEWEBCONSORTIUMW3C: Geolocation API specification, Jul. 2015. URL:http://dev.w3.org/

geo/api/spec-source.html.2,3