Not just seeing, but also feeling art: Mid-air haptic experiences integrated in a multisensory art exhibition

ContentslistsavailableatScienceDirect

International Journal of Human-Computer Studies

journalhomepage:www.elsevier.com/locate/ijhcs

Not just seeing, but also feeling art: Mid-air haptic experiences integrated in a multisensory art exhibition

Chi Thanh Vi

, Damien Ablart

, Elia Gatti

, Carlos Velasco

, Marianna Obrist

aSCHI ‘sky ’ Lab, Creative Technology Research Group, School of Engineering and Informatics, University of Sussex, UK

bDepartment of Marketing, BI Norwegian Business School, Oslo, Norway

a r t i c le i n f o

Keywords:

Multisensory experience Museum

Mid-air haptic feedback Taste

Smell Sound Emotion User experience Art gallery Art exhibition

a b s t r a ct

TheuseofthesensesofvisionandauditionasinteractivemeanshasdominatedthefieldofHuman-Computer Interaction(HCI)fordecades,eventhoughnaturehasprovideduswithmanymoresensesforperceivingand interactingwiththeworldaroundus.Thatsaid,ithasbecomeattractiveforHCIresearchersanddesignersto harnesstouch,taste,andsmellininteractivetasksandexperiencedesign.Inthispaper,wepresentresearchand designinsightsgainedthroughoutaninterdisciplinarycollaborationonasix-weekmultisensorydisplay– Tate Sensorium– exhibitedattheTateBritainartgalleryinLondon,UK.Thisisauniqueandfirsttimecasestudy onhowtodesignartexperienceswhilstconsideringallthesenses(i.e.,vision,sound,touch,smell,andtaste),in particulartouch,whichweexploitedbycapitalizingonanovelhaptictechnology,namely,mid-airhaptics.We firstdescribetheoverallsetupofTateSensoriumandthenmoveontodescribingindetailthedesignprocess ofthemid-airhapticfeedbackanditsintegrationwithsoundfortheFullStoppaintingbyJohnLatham(1961).

Thiswasthefirsttimethatmid-airhaptictechnologywasusedinamuseumcontextoveraprolongedperiodof timeandintegratedwithsoundtoenhancetheexperienceofvisualart.Aspartofaninterdisciplinaryteamof curators,sensorydesigners,soundartists,weselectedatotalofthreevariationsofthemid-airhapticexperience (i.e.,hapticpatterns),whichwerealternatedatdedicatedtimesthroughoutthesix-weekexhibition.Wecollected questionnaire-basedfeedbackfrom2500visitorsandconducted50interviewstogainquantitativeandqualitative insightsonvisitors’ experiencesandemotionalreactions.Whilstthequestionnaireresultsaregenerallyvery positivewithonlyasmallvariationofthevisitors’ arousalratingsacrossthethreetactileexperiencesdesigned fortheFullStoppainting,theinterviewdatashedlightonthedifferencesinthevisitors’ subjectiveexperiences.

Ourfindingssuggestmultisensorydesignersandartcuratorscanensureabalancebetweensurprisingexperiences versusthepossibilityoffreeexplorationforvisitors.Inaddition,participantsexpressedthatexperiencingartwith thecombinationofmid-airhapticandsoundwasimmersiveandprovidedanup-liftingexperienceoftouching withouttouch.Weareconvincedthattheinsightsgainedfromthislarge-scaleandreal-worldfieldexplorationof multisensoryexperiencedesignexploitinganewandemergingtechnologyprovideasolidstartingpointforthe HCIcommunity,creativeindustries,andartcuratorstothinkbeyondconventionalartexperiences.Specifically, ourworkdemonstrateshownovelmid-airtechnologycanmakeartmoreemotionallyengagingandstimulating, especiallyabstractartthatisoftenopentointerpretation.

© 2017TheAuthors.PublishedbyElsevierLtd.

ThisisanopenaccessarticleundertheCCBYlicense.(http://creativecommons.org/licenses/by/4.0/)

1. Introduction

Humansareequippedwithmultiplesensestoperceiveandinteract withtheirenvironment.However,inHCI,visionandhearinghavebeen thedominantsenses,andoursenseoftouch,taste,andsmellhaveoften beendescribedassecondary,asthelowersenses(Spence,2011).HCI researchersandpractitionersarehoweverincreasinglyfascinatedbythe opportunitiesthattouch,smell,andtastecanoffertoenrichHCI.Re-

∗Corresponding author.

E-mail addresses: c.vi@sussex.ac.uk , vichithanh@gmail.com (C.T. Vi), da292@sussex.ac.uk (D. Ablart), e.gatti@sussex.ac.uk (E. Gatti), carlos.velasco@bi.no (C. Velasco), m.obrist@sussex.ac.uk (M. Obrist).

centexamplesofsuchexperiencesincludethenovelolfactorydisplayby Seahetal.(2014),taste-basedgamingbyMureretal.(2013),olfactory in-carinteractionbyDmitrenkoetal.(2016),digitalflavourexperiences byRanasingheetal.(2014),andtheaddedvalueofhapticfeedbackfor audio-visualcontentbyMaggionietal.(2017).Inparticular,therehas beenagrowinginterestinuncoveringthespecificitiesofhapticexperi- encedesign(Schneideretal.,2017)andtheuniquefeaturesofhaptic

http://dx.doi.org/10.1016/j.ijhcs.2017.06.004

Received 24 May 2016; Received in revised form 20 June 2017; Accepted 22 June 2017 Available online 24 June 2017

1071-5819/© 2017 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license. ( http://creativecommons.org/licenses/by/4.0/ )

stimulationthatwouldallowthecreationofemotionallyengagingand meaningfulexperiences(Gattietal.,2013;Seifi andMacLean,2017).

Withtheadventofnoveltouchlesstechnologiesthatenablethecre- ationoftactilestimuliwithoutphysicalcontact(e.g.,(Carteretal.,2013;

Hamadaetal.,2014;Hoshietal.,2010;Longetal.,2014;Sodhietal., 2013),anoveldesignspacefortactileexperienceshasbeenopeningup (Obristetal.,2013).Mostnotably,ithasbeendemonstratedthatmid-air hapticstimulationcanbeusedtoconveyemotionstotheuser(Obristet al.,2015).Thisresearchhasmotivatedfurtherinvestigationsofthede- signpossibilitiesforcreatingnovelmid-airhapticsexperiences(Ablart etal.,2017).Hereweextendtheuseofmid-airhapticsstimulationin thecontextofamuseum,movingbeyondacontrolledlaboratoryenvi- ronmenttoinvestigatetheeffectofmultisensorystimulationonusers’ experienceofart.

Museumsandartgallerieshavealwaysbeenintheforefrontofinte- gratingandstimulatingmultiplehumansenses,notonlytoexplorenew waysofrepresentingarts,butalsotoincreasethewiderpublicinterest intheartifactsbeingdisplayed.Harveyetal.(1998)showedthattheuse oftouchspecimens,sounds,andsmellstocomplementtheobjectalong withinteractivecomponents(e.g.,roleplayinginductiondevice)and dynamicdisplayscanhaveastronginfluenceonvisitors’ experiences, especiallycreatingastrongsenseofflow– beingfullyimmersedandfo- cusedinatask(Csikszentmihalyi,1997).Anotherintriguingworkthat relatestomultisensorymuseumexperiencesistheJorvikVikingCentre (Jorvik,2017),wheremultisensorystimuliwereusedtoenrichtheex- perienceofatourconcerningtheVikingpastofthecityofYork.This experienceallowedvisitorstotouchhistoricalobjects(VikingAgearte- facts),tastetheunsalted,driedcodoftheVikingdiet,smellthearoma ofthecorrespondingdisplayedobjects,seetheanimalsandinhabitants oftheVikingcity,andlistentotheVikingsagas.Morefocusedonthe senseoftouch,Loscosetal.(2004)presentedhowvisitorscouldseeand feelvirtual3Dartworks(e.g.,statues)usingahapticdevicethatwas connectedtotheuser’srightindexfingertoprovidehapticfeedback.

Thisuseoftechnologyenableduserstotouchandfeelthecontoursand stiffnessoftheartwork.

Despitetheincreasinginterestinthedifferentsensesasinteraction modalitiesinHCIandrelateddisciplinesandprofessions(e.g.,artcura- tors,sensorydesigners),thereisonlyalimitedunderstandingofhowto systematicallydesignmultisensoryartexperiencesthatareemotionally stimulating.Moreover,therealsoseemstobealackofunderstanding onhowtointegratedifferentsensorystimuliinameaningfulwayto enrichuserexperienceswithtechnology(Velascoetal.,2016),includ- ingartpieces.Carbon(2017)replicatedtheworkofSmithandSmith (2001)andpointedoutthemismatchesintheamountoftimeandspace peoplespentinviewingartworksinalaboratoryversusamuseumcon- text.Specifically,museumvisitors hadlongerviewingtimethanwas mostlyrealizedinlabcontexts,aswellaslongerviewingtimewhen attendingingroupsofpeople.Additionally,thisworkuncoveredaposi- tivecorrelationbetweensizeofartworkandtheviewingdistance.These findingsemphasizethefactthatthereisaneedtocarryoutmuseum relatedinvestigations in theactual environment of a museum.Only throughanin-situapproach,theintendeduserswhohaveanintuitive interestandknowledgeaboutartenvironments,arereachedandcan providevaluablefeedbackonthemultisensorydesignandintegration efforts.

Buildingonthesepriorworks,inthispaper,wepresentresearchand designeffortscarriedoutaspartofasix-weekmultisensoryartdisplay – TateSensorium– inanactualmuseumenvironment(i.e.,TateBritain artgallery).Forthefirsttime,mid-airhaptictechnologywasusedina museumcontexttoenhancetheexperienceofapainting(i.e.,theFull StopbyJohnLatham)throughitsintegrationwithsound.Themultisen- soryintegrationoftouchandsoundaimedtoaidthecommunicationof emotionsandmeaninghiddeninthepainting:alargecircularblackspot intheapproximatecentreofanunprimedcanvas(seeFig.2b).

Incollaborationwithacreativeteamofartcuratorsandsensoryde- signers,thespecificexperiencefortheFullStoppaintingwascreated.

Atotalofthreevariationsoftheexperiencewerecreated,keepingthe soundthesamebutchangingthemid-airhapticpatterntoinvestigate theeffectofthesenseoftouchonthevisitors’ artexperience(seeillus- trated in Fig.6anddescribed in Section3.3).We hypothesizedthat museumvisitors wouldenjoy moreexperience involvingthepattern specificallydesignedforTateSensorium(Tatepattern,themostsophisti- catedandpurposefuldesignedexperience),followedbytheexperience involvingtheCirclepattern(congruentwiththevisualappearanceof thepainting)andfinallytheLinepattern(incongruentwiththevisual appearanceofthepainting).Visitors’ experienceswereassessedthrough ashortquestionnaireattheendoftheTateSensoriumexperienceand throughinterviewstodeepenourunderstandingonthesubjectivedif- ferencesofsensoryenhancedartexperiences.

Inthefollowingsections,wefirstprovideareviewofrelatedwork onmultisensoryresearchanddesigninmuseums,followedbyagen- eraloverviewonthemultisensoryartdisplay– TateSensoriuminthe TateBritainartgallery.Weincludethedescriptionoftheexhibitedart piecesandsensorydesignspace.Wethenfocusontheworkaroundthe FullStoppaintingandthedesignanddevelopmentofthemid-airhaptic patternsaspartofthespecifictouch-soundintegration.Weprovideade- taileddescriptionofthedatacollectionprocessandtheinsightsfromthe analysisof2500questionnairesand50interviews.Weconcludewitha discussionofourfindingswithrespecttothelessonslearnt,limitations andfutureopportunitiesfordesigningmultisensoryexperiencesoutside theboundaryofalaboratoryenvironment.

2. Relatedwork

Museumsarepublicplacesthatcontainacollectionofartifactsthat holdvaluesin artistic,historical,andculturalcontexts(Alexanderet al.,2008).Importantly,museumsoffer“amulti-layeredjourneythatis proprioceptive,sensory,intellectual,aestheticandsocial” (Leventand Pascual-Leone,2014).Giventheexperientialaspectofmuseums,they (andexhibitors) havealwaysbeenlookingfornewwaystodiversify andenrichtheexperiencesthattheydelivertothevisitors.Therefore, therehavebeenexamplesandeffortsofenhancingartobjectsthrough sensorystimulitoengagevisitorsandconveymeaning.

2.1. Multisensoryinteractioninthemuseum

Museumsareaforerunnerin harnessingnew waysof interacting with publicusers.Therefore,they arerecognizedwithin thefieldof HCI asrelevantplacesfordesigninginteractivesystemstoreachout tothepublic.AnexampleisTranscendingBoundaries(PACE,2017),an exhibitionthatexploredthetranscendbetweenphysicalandconceptual boundaries(e.g.,elementsfromoneworkcanfluidlyinteractwithand influenceelementsoftheotherworksexhibitedinthesamespace)via visual,auditory,andtactileinteractions.Inaddition,therearevarious casesinwhichtheintegrationofmultiplesenseshasbeenexploredin museums.Forexample,Lai(2015)exploredthe“UniversalScentBlack- box”,anartworkcomposedofboxesemittingfivesmells:grass,baby powder,whiskeytobacco,darkchocolate,andleather.Visitorstothe installationcouldtriggeranodouremissioninanotherareaforothervis- itorsandvice-versa.Thisolfactoryinteractionattractedmuchinterest fromthevisitorsandbecameaninspirationalprobeforexploringolfac- toryinterfacesforcommunication.Basedonthosepriorexplorations, ithasbeensuggestedthatmultisensorydesignin amuseummayen- hance therichness, andeventhememorability,ofthevisitor’sexpe- rience(Eardleyetal.,2016;LehmannandMurray,2005),due tothe emphasisonthemultisensorynatureofoureverydaylifeexperiences.

WorkbyTeramoto etal.(2012)hasshownthat auditoryandvisual modalitiesmutuallyinfluenceeachotherduringmotionprocessingof externaleventssothatthebrainobtainsthebestestimatesofsuchevents (Teramotoetal.,2012).WithinHCI,wecanadditionallyobservevar- iouseffortsofintegratinginteractivetechnologies(e.g.,touchscreens,

multi-touchtabletop,see (Correiaetal.,2010;Dijketal.,2012;Hor- necker,2008;Maetal.,2015)intoamuseumcontexttomakeartworks moreaccessibleandenjoyable.Inparticular,Correiaetal.(2010)used amulti-touchtabletopformultimediainteractioninmuseums,allowing visitorstoaccessartworks’ detailsandtoassigntagstoartworks.

Amongtheimplementationsof multisensoryintegration inmuse- ums,theintegrationoftouch,togetherwithvisionandhearing,arethe mostfrequentsensestobe stimulated.Forexample,theVictoriaand AlbertMuseuminLondon(VAM,2017) providedvisitors “touchob- jects” (e.g.,awiseowlsupervisingtheSculptureGalleriesandcarved examplesofdifferentwoodstypes)toexperiencethedisplayedartifacts.

Visitorswerealsoabletopressabuttonnexttoanobjecttohearrelated audiodescriptions.AnotherexampleisCiolfi andBannon(2002)who presentedasandboxusedinanarchaeologyworkshoptorecreatean archaeologicalscenefortheattendingchildrentoenjoy“playingthear- chaeologist”.Harleyetal.(2016)designedthreeinteractiveprototypes ofprayer-nutsinanefforttoconveyandcontextualizethehistorical, sensory,anditsembodiedinformation.These3Dprintedtangiblepro- totypesofferedvisitorssensoryinteractionsofsmell,touch,andsound withvisualandaudiofeedback,whichwasrelevanttothehistorical, social,andculturalcontextoftheartifact.Loscosetal.(2004)createda virtualenvironmentwherevisitorscouldseevirtual3Dartworks(e.g., statues)andexperiencedanassociatedhapticfeedback.Atwo-contact- pointhapticdevicewaslinkedtotherightindexfingerofeachvisitor enablingthemtotouchandfeelthecontoursandstiffnessoftheart- worksthroughhapticfeedback.However,theauthorsalsopointedout thataskingvisitorstowearanexoskeleton,toenablethehapticfeed- back,iscontradictorytotheideaoffreeexplorationinamuseum.Thus, anydevicesdesignedformuseumvisitorsshouldbeaslittleinvasiveas possible.

Fromtheartisticside,newtechnologieshavebeenusedasinnovative meansforcreatingartpieces.Forexample,Yoshidaetal.(2004)created aninterface fordrawingusingastylusthatprovideddifferenthaptic feedbacksdependingonthecoloursusedtopaint(e.g.,participantsex- perienceddarkcoloursasheavyinweightandlightcoloursaslightin weight).Inthiswork,theattachmentofvibrotactilefeedbackstodif- ferentcolourscreatedanovelexperienceforthecreatorsofthosedig- ital/mediaartworks.However,theauthorsdidnotinvestigatefurther thevisitor’suserexperienceoncepresentedwiththeseartworks.An- otherworkexploredthecreationprocessofartintegratingvisionand touch(Azhetal.,2016).Theauthorsranone-on-oneguideddesignses- sionswherevisualartistscreatedtactiledesignprototypesaugmenting anexistingworkintheirportfolioasavisualcontext.Theyanalysed thecreationfollowingtworationales:(1)thetactileconstruct(asetof attributesthatdefineitsphysicalcharacteristics)and(2)thetactilein- tent(thevarietyofmeaningassignedtoatactilefeature).Thisanalysis providesinsightsonhowtodesigncreativitytoolsforartists,butdoes notfurtherinvestigatethemuseumvisitors’ experience.

Theaboveexamplesshowtheinterestandgrowingattentionfrom variousstakeholdersinexploitingthehumansensesintheexperience ofartwork.Inparticular,theproliferationofhaptictechnologiescreates anewspaceforexperimentationsforbothresearchersandartistsalike.

Allpriorworkaroundthesenseoftouchishoweversofarlimitedtoac- tualphysicalcontactbetweenvisitorsandtheartifacts.Consequently,it doesnotyetexploittheuseofnovelcontactlesstechnology.Thisconse- quentlyraisesthequestionofwhatuserexperiencesaroundartcanbe createdthroughtheuseandintegrationofmid-airhapticfeedbackin amuseumcontext,inparticulargivenrecentevidencesuggestingthat mid-airhapticfeedbackcanconveyemotions(Obristetal.,2015).

2.2. Hapticsasanaidincommunicatingemotions

Recentdevelopmentsofnovelhaptictechnology,suchasfocusedul- trasound(Carteretal.,2013;Hoshietal.,2010),airvortex(Sodhietal., 2013),andPinPad(Jungetal.,2017),aimtocreatenewformsoftactile experiences.Theseworkshighlightthedesignopportunityofcreating

tactilesensationsinmid-air,without requiringtheusertophysically touchanobject,asurfaceorwearanattachmentsuchasagloveorex- oskeleton.Suchexperiencesareofgreatinterestwhenitcomestoaug- mentingtheexperienceofartworks,whichareoftenfragileandwould decaythroughmultipleexposuretohumantouch.Yet,thesenewhaptic technologiesareintriguingtoengagepeoplewithartemotionally,and toinspireartisticexplorationsandcreatememorableexperiences.

Herewefocusoncommunicatingandmediatingemotionsthrough touchasaresearchareathatallowsthedesignofnewemotion-related interactions(Obristetal.,2015;Petrecaetal.,2013).Thisisdemon- stratedinarecentworkofParketal.(2013)ontheintegrationoftouch duringphoneconversationsinordertoenhanceemotionalexpressive- nessinlong-distancerelationships.Moreover,thereisagrowingnum- berofwearablesystemsthatallowdifferenttypesofsocialtouchand anincreasingnumberofstudiesdemonstratingtherichexpressiveness oftactilesensationsderivedfromnovelhapticsystems(Hertensteinet al., 2009;HuismanandFrederiks,2013; Jungetal., 2014;Leetal., 2014;SmithandMacLean,2007;Wilsonetal.,2016).Previouswork hasshowedthatparticipantsusedweaktouchesforpositiveemotions, andhard,fast,andcontinuoustouchesfornegativeemotions(Parket al.,2013).Othersidentifieddifferenttypesoftouchforeachemotion (e.g.,strokingforlove,squeezingforfear),butalsoreportedpartici- pants’ difficultyindifferentiatingtheintensityoftheexpressionswhen appliedthroughawearablesystemontheforearm(HuismanandFred- eriks,2013).Altogether,theseresultspromotethepotentialforcommu- nicatingaffectiveinformationthroughtouch.

Mostrecently,thispotentialhasbeenestablishedformid-airhaptic technologyusingahapticdevicethatusesfocusedultrasoundtocreate oneormultiplefocalpointsonthehumanhand.Afocalpointiscreated usingafixedpressure(physicalintensity)inmid-airusing40kHzul- trasoundwavesandbyapplyingthecorrectphasedelaystoanarrayof ultrasoundtransducers(Carteretal.,2013).Thisfocalpointofpressure canthenbefeltwhenmodulatingtheultrasoundwaveswithinthefre- quencyrangeofthemechanoreceptorsofthehumanhand(i.e.,Meissner corpuscleandPaciniancorpuscle(Obristetal.,2013).Usingthismid-air hapticdevice,Obristetal.(2015)createdhapticemotionaldescriptions andidentifieda specificsetofparameters (combiningspatial, direc- tional,andhapticcharacteristics)withrespecttothetwo-dimensional emotionframeworkofvalenceandarousal.Basedonthis,theauthors concludedthatitispossibletocommunicateemotionsthroughmid-air tactilestimulationinanon-arbitrarymannerfromoneusertoanother.

Thisworkwasamajorinspirationfortheteamofpractitioners,curators, andresearchersworkingontheTateSensorium.

3. TateSensorium

Tate Sensorium was a six-weeks multisensory exhibition in Tate Britain,aninternationallyrecognizedartgalleryinLondon,UK.Inthis section,weprovideageneraloverviewandbackgroundontheproject, theoverallambition,andthespecificaimsforthemultisensoryaugmen- tationofartworkthroughtheuseofmid-airhaptictechnology.

TateSensorium wasthewinningprojectofthe2015TateBritain IKPrizeawardthatisspecificallydesignedbyTatetosupportinnova- tiveinstallationsusingcutting-edgetechnologiesthatenablethepublic todiscover,explore,andenjoyartinnewways.TheambitionofTate Sensorium was toenablemuseumvisitors toexperienceartthrough all senses(vision, sound,touch,smell,andtaste).Thiswasachieved throughthejoint effortsofacross-disciplinaryteam ofcollaborators from the art gallery, creative industries, sensory designers, and re- searchers(seedetailsintheAcknowledgments).FlyingObject(Object, 2017),acreativestudiobasedinLondon,ledtheprojectandcoordi- natedtheactivitiesacrossthevariousstakeholders.

BelowwewillfirstdescribethesetupofTateSensoriumintheTate Britaingallery(foranoverview).Wethenprovidethedetailson the artworkselectionprocessandthedesignofthesensorystimuliforthe finallyselected artpieces (i.e.,fourpaintings,see Fig.2),theirinte-

Wall

Artwork / Paintings Headphones (cabled) Speaker

Diffuser Anglepoise lighting Ultrahaptics Measurement area Scent unit

Chocolates Desk / Table Plinth

Small lighting for an object Main lighting Practical lighting (for operators)

ENTER

EXIT ₁

2

3a 3b

4

Fig. 1. Room setup of Tate Sensorium split into different sub-spaces (design by flying object): visitors enter on the right, where they receive the headphones (1). Then they move to the room (2) to see the first painting Interior II alongside olfactory and sound stimuli. After that, they move to either (3a) to experience the Full Stop painting alongside mid-air haptic and sound or (3b) to see the painting In the Hold through olfactory and sound stimuli. After swapping, visitors move to the last station (4) to experience taste sensations for the Figure in a Landscape painting.

grationanddeploymentin themuseum,sothatvisitorswereableto experiencethedifferentartpiecesinanovelway.Wewilldescribein evenmoredetailthedesignofthehapticfeedbackusingmid-airhaptic technologyandthescientificapproachtocollectuserfeedback(both ledbytheresearchteamattheUniversityofSussex).

3.1. Overviewonthesetupinthemuseum

AlargededicatedroominsidetheTateBritainartgallerywasused forTateSensorium.Fig.1showsthelayoutoftheroomdividedintofour areasspecifyingthefinalsetupforthefourselectedpaintingsincluding detailsonthepaintinglocations,lighting,sensesused,etc.Eachpainting hadadedicatedspaceandwashungonawallineachsectionofthe room(marked2,3a,3b,4).

Visitorsfirstenteredtheroomandwerewelcomedjustinsidethe entrance(infrontofthepointmarked1inFig.1).Atthatpoint,visitors putonheadphonesandlistenedtoawelcomemessage,whichbrieflyin- troducedtheeventandgavesomegeneralinstructions.Visitorsentered inagroupoffouratatimeandviewedonepaintingatatimeduring thetour.Afterviewingthefirstpainting,thegroupoffourpeoplesplit whenreachingthesecondpainting,sothattwopeoplecontinuedwith thesecondpaintingandtheothertwowenttothethirdpainting.These groupsswappedafterwards,beforemovingforwardalltogethertothe fourthpainting.Thesplitwasnecessaryduetothesetupofthemid-air

haptictechnologyforthesecondpainting,whichcouldonlybeusedby twopeopleatatime.

3.2. Artworkselectionandsensorydesign

Theselectionoftheartworkswasacollaborativeprocessbetween galleryprofessionalsandexternalexpertsfromdifferentfields(atFlying Object,UniversityofSussex,andotherindependentsensoryexperts).

At first,not onlypaintingsbutalso sculptureswerepartofthepool of potentialartworks.Thelistofpotentialartworkswascompiledby FlyingObjectandincludedsuggestionsfromtheteamatTateBritain aswell.Thisresultedinaninitialpoolofpotentialartworksconsisting of60paintings.Theselectioncriteriaforthepaintingsfocusedonnon- representational(orabstract)paintings,asitwasagreedthattheywould leavemoreroomforviewerinterpretation.Inotherwords,withoutany clearvisualidentityofobjectswithinthepainting,thenon-visualstim- uliwouldpotentiallyhaveastrongerimpactonhowtheartworkwould beperceived.Additionally,thenot-so-clearvisualidentitywouldgive roomforothersensorystimulitoguidetheinterpretationoftheexperi- ence,giventhatsensoryinformationcanprimespecificnotionsinusers (SmeetsandDijksterhuis,2014).

Theavailabilityoftheartworkfortheexhibitionandtheprepara- tionphase(∼2months)wasalsoakeycriterionconsideredintheselec- tionprocess.Thefinaldecisionastowhatartworkstoselectwasmade bythecreativeprojectteamledbyFlyingObject,withsign-off byTate Britain’smanagement,inJune2015.TateBritain’sstaff providedadvice ontheselectionofartworks,basedontheiravailabilityandsuitability forinclusion(intermsofconservation,safety,andotherartisticconsid- erations).Furtherguidanceondevelopingcontent(selectingappropri- ateinterpretive/contextualinformationrelatingtoeachwork)forthe display,eventuallytranslatedinto“sensoryform” (e.g.audiomaterial), wasprovidedbyTate.

Fourpaintingswereselectedbasedontheirpotentialforinterpreta- tionthroughdifferentsenses,aswellastheiravailabilityatthemuseum forthedurationofthedisplayinAugustandSeptember.

Thefourselectedpaintingswere:

1. InteriorIIbyRichardHamilton¹ 2. FullStopbyJohnLatham² 3. IntheHoldbyDavidBomberg³ 4. FigureinaLandscapebyFrancisBacon⁴

Fig.2showstheillustrationshotsofaparticipantexperiencingthe fourselectedpaintings.Originalcopiesofthepaintingscanbeaccessed viatheTateBritainwebsite.⁵Thedetailsofeachpaintingareinthenext sectionalongsidethedescriptionofthesensorystimuli.

Thesuitabilityofthesensorystimuliwasdecidedbyconsideringthe literatureonmultisensoryperceptionandexperiences(bytheuniversity researchteam),suggestionsfromsensoryprofessionals,andbasedonan iterativecreativeprocess.Todothis,anon-sitevisittotheartgalleryby thewholeteamwasarranged.Duringthevisit,theteamexperimented withthedifferentsensesinfrontoftheartwork(e.g.,usingscentedpa- perstrips),aswellasexperiencingthemid-airhaptictechnologyatthe Universitywiththeprojectteam.

Themethodologyfordesigningthesensorystimuliwasasfollows:

(1)Theteam(ofallpeopleintheproject)generatedideasforeachof thefourpaintingsselected,aswellasafifthreservedpainting,prototyp- ingthemwherepossible(i.e.selectingactualscentsorfoodingredients, creatingaudiosamples).(2)Theteamassignedaleadingsensetoeach painting,alongwithasecondarysense(inthecaseofthepaintingFig- ureinaLandscapebyFrancisBacon,atertiarysensetoaccompanythe

1http://www.tate.org.uk/art/artworks/hamilton-interior-ii-t00912 .

2http://www.tate.org.uk/art/artworks/latham-full-stop-t11968 .

3http://www.tate.org.uk/art/artworks/bomberg-in-the-hold-t00913 .

4http://www.tate.org.uk/art/artworks/bacon-figure-in-a-landscape-n05941 .

5http://www.tate.org.uk/visit/tate-britain .

Fig. 2. Tate Sensorium exhibition at Tate Britain in 2015. (a) Installation shot of Interior II (1964) by Richard Hamilton. Photo: Tate. Illustration shows a participant experiencing the first painting, combining vision, audition, and smell. (b) Installation shot of Full Stop (1961) by John Latham © John Latham Estate. Photo: Tate. Illustration of a participant experiencing the second painting combining vision, auditory, and haptic (with the haptic pattern projected on the user’s right hand). (c) Installation shot of In the Hold (c. 1913–4) by David Bomberg.

Photo: Tate. Illustration of a user experiencing the third painting combining vision, auditory, and smell (by holding a 3D printed scent object close to her nose). (d) Installation shot of Figure in a Landscape (1945) by Francis Bacon. Photo: Tate. Illustration of a user experiencing the fourth painting combining vision, audition, and taste (by eating a piece of chocolate with multiple ingredients, namely, charcoal, sea salt, cacao nibs and smoky Lapsang Souchong tea).

taste).(3)Thedesignersofeachofthosesensesformed,withFlyingOb- ject,sub-teamstocollaborateontheexperienceforeachpainting.(4) Throughiterativediscussionswithexpertsandprofessionalsbetween theteams, thesesensoryideaswererefined.Below,wepresentade- taileddescriptionofthe“FullStop”,whichwasselectedforthepresent study,whereweutilizedmid-airhapticstodesigntheexperienceofsuch apainting.

3.3. Sensorydesignforthe“FullStop” painting

Hereweprovidedetailsonthespecificdesignforthesecondpaint- ing(FullStopbyJohnLatham),whichwasaugmentedthroughthein- tegrationofsoundwithmid-airhapticstimuliusingthemid-airhaptic devicedescribedby(Carteretal.,2013)anddevelopedbyUltraHaptics (2017a).

3.3.1. Backgroundaboutthepainting

TheFullStoppaintingbyJohnLathamisanacrylicpaintoncanvas from1961,withthesize3015×2580×40mm.Itwaspresentedinthe roommarked3ainFig.1andcanbedescribedthus:

“FullStopisamonumentalpaintingcomprisingalargecircularblack spotintheapproximatecentreofanunprimedcanvas.The spotwascre- atedbyrepeatedactionwithaspraygun,itscurvedelineatedusingweighted sheets ofnewspaper cut to the correctshape and, as a result, traces of rectangularformsarefaintlyvisibleoutsidethecircumference.Thecircle’s edgesareblurred, particularlyonthe leftsidewhereasprinkling oftiny andslightlylargerdotsemergefromthedenseblackofthelargespot.The semi-mechanicalprocessofmakingthespot,inwhichmanydotsareapplied tothecanvasatthesametime,suggeststhemechanicalprocessofprinting ratherthanthemoretraditionalpaintingprocessesnormallyassociatedwith acanvas.Thepainting’s canvasisunstretchedandisdisplayedpinnedto thewallinthemannerofawall-hangingevokingsignageandheraldry.The title,FullStop,referstotext,andevokestheprintedword.Atthesametime, theblurrededgesofthespotandtheslighthalosaroundsomeofthelarger dotsatitscircumferencerecallasolareclipse,ablackholeorthenegative ofphotographsoflightreflectingoff planetsinthedarkgalaxy”.

(QuotedinArtafterPhysics,p.106.)

3.3.2. Sensoryaugmentation

Participants experiencedthis painting through the integration of soundandtouchfeatures.Thesoundwaspresentedviaheadphonessup- pliedbyPolarAudio(manufacturedbyBeyerDynamic)andwhichwere wornbyparticipantswhileintheroom(seeFig.3).Thesoundwascre- atedbyasoundexpertaccentuatingtheinterplaybetweenthepositive andnegativespacein theartwork,especiallyemphasizingthepaint- ing’sdualityofblackandwhite.Theaudiowasalsodesignedtocreate

Fig. 3. Detailed setup of the space for the painting, Full Stop (left), with the specifications of the setup on the right.

Fig. 4. Tate Sensorium exhibition at Tate Britain in 2015, installation shot of Full Stop (1961) by John Latham © John Latham Estate. Photo: Tate. Illustration of a participant experiencing the second painting combining vision, auditory, and haptic.

asenseofscale,ofroundnessandreferencetoLatham’suseofspray paint,whichwasresembledinthemid-airhapticfeedback.

Participantsstoodinfrontofaplinthboxandputonehand,with the palmfacing down,inside thetop partof theplinthtohave the hapticfeedbackdeliveredtotheirpalm(seeFig.4).Thehapticdevice wasplacedinsidetheplinth,withthespecificationsshowninFig.5.A speakergauzewasplaced50mmabovethedevicetopreventpartici- pantstouchingthedevice.Thehapticfeedbackwaspresentedthrough thegauzewhenparticipantsputtheirhandontopofit(Carteretal.,

Fig. 5. The plinth created for the haptic stimulus for the Latham painting using mid-air haptic technology, the UltraHaptics device (design by flying object).

Fig. 6. Haptic patterns for the Full Stop painting. Main Tate Sensorium pattern (left), and two alternating haptic patterns (middle ‘simple circle ’ and right ‘line ’). In the Main Tate Sensorium pattern, there is a circle shape composed of 16 points of varying size (having an increase/decrease in diameter of the formed circle), synchronized with the rain pattern.

2013).Theheightoftheplinthwascalculatedsothatitfittedcomfort- ablywithadults,children,anddisabledvisitorsinwheelchairs.

3.3.3. Mid-airhapticpatterndesign

Synchronizationbetweenthesoundandthemid-airhapticsensa- tionwashandledbyself-developedsoftware thatcouldreadMusical InstrumentDigitalInterface(MIDI)inputs(usingRtMidi2.1).Thus,the mid-airhapticpatternscouldbe synchronizedautomaticallywiththe soundscreatedbythesounddesigner.Inotherwords,thesoundde- signercouldcontrolthemid-airhapticpatterns(frequency,intensity, andmovementpaths)tocreateadesiredexperiencefortheFullStop painting.Thefinalversionofthesoundfilealsosynchronizedwiththe desiredmid-airhapticfeedbacksensation(asdepictedinFig.6,left).

Thissensationhadthe“Changeablecirclesizeswithraindropsensations” featuretoenhancethevisitor’sexperienceofthepainting.Specifically, itwascreatedbyaround-shapehapticsensationsynchronizedwiththe sound.Thecircleshapewascomposedof16pointsofvaryingsize(hav- inganincrease/decreaseindiameter),andwasintegratedwiththerain patterncreatedbyusingonepointatrandompositionsonthewhole hand.

Importantly,wefurtherinvestigatedtheimpactofthemid-airhaptic stimulationonvisitor’sexperiences.Todoso,wecreatedasetofseven alternativehapticexperiencesusingthreesourcesofinspiration:(1)the paintingitself,tryingtoemphasizeitsvisualproperties(rounded),(2) contradictingthevisualappearanceofthepainting(notrounded)and (3)emotionalhapticstimulibased onthefindingsfromObrist etal.

(2015).Thesesevenpatternswere:

• Pattern1:Acirclewithnosizevariation.

• Pattern2:Asimplefocalpointinthemiddleofthepalm.

• Pattern3:Onepointmovingfromlefttoright.

• Pattern4&5:Twopointsmovinginacircleclockwiseorcounter- clockwise.

• Pattern6&7:Twopatternsdesignedbasedonthespatialanddi- rectionalparametersidentifiedbyObristetal.(2015)torepresent positiveandnegativeemotions(positive:onepointmovingfromthe edgeofthefingerstothewristinapredictableway;negative:one pointmovingaround6locationson thepalm creatinganunpre- dictablepath).

Eight participants volunteered to evaluate these seven patterns alongsidethemainhapticpattern.Participantsexperiencedeachhap- ticpatterninacounterbalancedorder,andthenratedboththevalence andarousalofeachpatternonaLikertscale(1–9).Participantswere alsoencouragedtodescribewhattheyfelt andhowmeaningfulthey perceivedthesensoryintegrationfortheFullStoppainting(whichwas representedbyanA3posteronthewall).

Theresultsshowedthat“Circle” (pattern#4)and“Line” (pattern

#3)patternswerethemostdistinctiveonesfortheFullStoppaintingin termsofvalanceandarousal,accordingly.Inspecific,theCirclepattern hadthehighestvalenceratings(6.43±2.15)amongallthepatterns(av- eraged5.02±0.65)andanarousalaverageratingof4.14(±2.48).The Linepatternhadthehighestarousalrating(5.86±2.48)amongallthe patterns(averaged 5.11±0.59)anda valenceaverageratingof 5.71 (±2.48).Notably,theLinepatternhasacontradictingshapewiththe painting(showingacircleshape).Therefore,itwasexpectedtohave lowerratingsinvalenceandlikingaswellduringthesciencedays.The twopatternschosenaredescribedbelow:

• The“AlternativeCircle” patternhadacircleshapebutwasonlycom- posedof2pointsinsteadof16,rotatingonafixedpositionandof constantsize(10cmofdiameter)onthepalm.

• The“AlternativeLine” patternhadalineshapeandwascomposed ofonepointmovingfromlefttoright.Whenreachingtheendofthe line,thepointstartedagainfromtheleftsideandmovedtotheright tomakethewholeline(10cm).

Thethreepatterns(namedTate,Circle,andLine)werealternated duringtheScience daysbefore closingtheexhibition(seeFig.7).In contrast,ontheotherdaysoftheexhibition,onlytheTatepatternwas shown.

4. Procedureandmethod

Inthissection,weprovideadetaileddescriptionofhowtheTate Sensorium visitors experiencedthemultisensoryinstallation andour methodforcapturingtheirexperiencesthroughquestionnairesandin- terviews.Additionally,weexplainthedifferencebetweenStandarddays

2015 Feb

START 2015 Aug 27^th

Public exhibition

2015 Oct 4^th CLOSE

Preparation and setup process (e.g. artwork selection & experience design)

Pilot (1 week)

Public exhibition

& collecting data

Science days (5 days) Standard days

(34 days)

Fig. 7. Overview of the Tate Sensorium project timeline with a six-month preparation and design period, followed by a six-week (four weeks + two weeks extension) public ex- hibition and data collection period.

Fig. 8. Tate Sensorium exhibition at Tate Britain in 2015. Tate illustration shot of a par- ticipant’s first stop point, after entering the room, where they hear a short introduction about Tate Sensorium.

andSciencedays(asdepictedinFig.7).Overall,theexhibitionopened tothepublicfor1monthand8days.

Asmentionedbefore,thepurposeofSciencedayswastoinvestigate theimpactofdifferentparametersofmid-airhapticstimulationonvis- itors’ experience.Thethreepatternswerealternatedatdifferenttimes oneachScienceday(ontheotherdaysoftheexhibition,onlytheTate patternwasshown).Additionally,onScience days,wecollectedvisi- tors’ perceptionsthroughquestionnairesontherelativeimportanceof eachsense(vision,auditory,smell,touch,andtaste)whenexperiencing thepaintingsatTateSensorium.Onthefinaldayofthedisplay,visitors werealsoaskedtotakepartinashortaudio-recordedinterviewlasting for10minutes(seebelow).

4.1. Step-by-stepprocedure

ParticipantsenteredTateSensoriumingroupsoffour.Thisgroup sizewastoallowTateSensoriumvisitorsatrulyimmersivemultisensory experience,aswellastoseparatevisitorstoattenddifferentpaintingsin asmoothtraffic.Anotherpurposewastomimicacommongroupvisit toamuseum.Moreover,agroupoffourpeoplewasamanageablegroup persession(15min)allowingeachparticipanttoenjoytheartworkwith themultisensoryexperience.Afterenteringthemaindoor,participants werewelcomedandthenguidedbya memberof staff untiltheend ofthetour.First,participantsstoppedatthepointmarked1inFig.1. Herethey wereinstructed toputontheheadphonestohear ashort introductionaboutTateSensorium(seeFig.8),asfollows:

Ineachroomwewantyoutofocusonthepaintingandletyoursenses dotherest.

Maybethe sensorystimuli willinspirethoughts, ormemories. Maybe they’llsuggestdetailsinthepaintings,orbringoutshapeorcolour.Each

ofthemhasbeenmadeinresponsetotheartworks,thinkingaboutwhatthey depict,andhowandwhentheyweremade.

Wewantyoutofindyourowninterpretationofeachartwork,andwe hopethesestimuliwillhelp.

Additionalaudioguidanceforeachpaintingwasprovided,giving somedetailsaboutthepaintingitself(bywhomitwaspainted),andthe accompanyingmultisensorystimulation(e.g.,walkaroundtheroomto explorethedifferentsmells).Participantsalsoreceivedawristbandto capturetheirskinconductanceresponse,whichwasused tocreatea personalizedprintoutattheendofthetour.Thisdataisnotincluded inthispaperasitwasnotthefocusofthestudyledbytheUniversity team.

Aftertheshortintroduction,participantsremovedtheirheadphones andcontinuedwalkingtothefirstpainting(InteriorIIbyRichardHamil- ton,asmarked2inFig.1).Here,theystoodinfrontofthepainting andwereinstructed(throughthespeakersintheroom)toexperience itasnaturallyaspossible,andtomovearoundtheroomtoexplorethe threedifferentscents(seeFig.2a).Threeminutesweregiventoallfour participantstoexperiencethepainting.Afterthat,participantswerein- structedbythestaff toseparateintotwopairsof twoparticipantsto continuetothenextpainting.

Pair#1wenttotheroommarked3ainFig.1andviewtheFullStop painting.Participantswereaskedtoputontheheadphonesprovided.

Followingtheaudioguidance,eachparticipantwasaskedtoputtheir handintotheemptyspaceintheplinthtoexperiencethemid-airhaptic feedbacks(seeFig.4foranexampleandFig.5fortheplinthspecifi- cations).Themid-airhapticfeedbackwasprovidedontheparticipant’s palm,andwassynchronizedwiththesoundprovidedthroughthehead- phones.Afterthesound-hapticstimulusfinished(1minute),thesecond participanttookaturninexperiencingthemid-airhapticstimulusfor theFullStoppainting.Participantswereinstructedtoenjoyviewingthe paintingwhileexperiencingthesoundandtouchintegration.Thetotal durationgivenforparticipantstobeinthisroomwas3minutes.

Pair#2wenttotheroommarked3binFig.1andviewedtheIn theHoldpainting.Thereweretwoplinthsinthisroom.Ontopofeach plintharetwo3Dprintedscentobjects.Participantswereencouraged toexperiencethepaintingandthescentsbypickingupthescentedob- jectandsmellingit(seeFig.2c).Participantsweregiven3minutesto explorethepaintinginassociationwiththesoundandsmellstimuliin thisroom.

After,Pair #1finishedexperiencingRoom3a, andPair#2went throughroom3b,theyswitchedroles.Pair#1nowmovedontoroom 3b andPair#2movedtoroom3a,followingthesameprocedure as describedaboveforeachofthetwopaintings.

OncebothpairscompletedRoom3aand3b,allfourparticipants movedtothefinalroom(marked4inFig.1).Here,eachparticipant putontheheadphonesagain.TheyallstoodinfrontoftheFigureina Landscapepaintingwithaplinthinbetween.Ontopoftheplinthwasa boxwith4piecesofchocolate.Participantswereencouragedtopickup apieceofchocolateandeatit(seeFig.2d).Threeminutesweregiven toparticipantstoexperiencethepaintinganditsassociatedtasteand sound.

4.2. Methodsused:questionnaireandinterview

Onceparticipantshadfinishedvisitingallfourrooms,theywerere- questedtomovetotheexitpoint.Justbeforeexiting,participantswere encouragedtocompleteashortquestionnaireabouttheirexperienceof TateSensorium.Thequestionnaireconsistedofthreequestionsforeach painting:(1)visualliking(ofthepaintingitself);(2)multisensoryexpe- rienceliking(thesensorystimuliintegratedintothepainting);and(3) emotionalreaction(arousal)(seeFig.9foranillustration).Theseques- tionswereusedtoquantifytheaddedvaluesofthedesignedsensory augmentationaddedtotheexperienceofthepaintings.

Participants answeredusing 5-point Likertscales(where5 isthe highestrating(Beelietal.,2005).Participantswerealsoaskedtore-

How much do you like this painng?

How much did you like the mul-sensory experience created for this painng?

How intense was the mul-sensory experience created for this painng

Fig. 9. Questionnaire about Visual Liking/ Multisensory Experience Liking / Arousal.

spondtosomedemographicquestions(i.e.,age,gender),andtoreport whethertheywouldbeinterestedinvisitingsuchamultisensoryexperi- enceagaininthefuture(yes/no/maybe).Thisinformationwasusedin theanalysistoexploredifferencesbetweentheexperienceratingsand users’ personalbackgrounds.Moreover,thecuratorofTateSensorium wasinterestedintheageandgenderdistributionattractedbythemul- tisensorydisplayandifpeoplewouldbeinterestedinfutureevents.

ForthededicatedSciencedays,participantshadanadditionalques- tionontheimportanceofeachindividualsense(seeFig.10).Partici- pantssignedaconsentformbeforeansweringthequestionnaires.

Onthelastdayofthedisplay,visitorsofTateSensoriumwerealso invitedtotakepartinashortaudio-recordedinterviewlastingabout10 minutes.Theinterviewsaimedtoexplore:(i)theoverallexperienceof themultisensorydisplay,and(ii)gainspecificinsightsontheexperience createdfortheFullStoppainting,whichintegratedmid-airhapticfeed- backwithsound.Here,wewereparticularlyinterestedinunderstanding anyqualitativedifferencesintheperceptionofthethreehapticpatterns (theTateSensorium,Circle,andLinepatternsasillustratedinFig.6), whichwerealternatedbetweengroupsofparticipants.

Aninterviewguidewasdefinedbasedonthosetwomainareasof interestandincludedthefollowingeightquestionsforeachinterview session:

1. HowwouldyoudescribeyourTateSensoriumexperience?

2. WhatdoyouthinkparticularlyaboutyourexperienceoftheFullStop painting?

3. Howwouldyoudescribethehapticexperienceyoureceivedonyour hand?

4. Howmeaningfulwasitforyou?Why?

5. Howdidthehapticexperiencematchyourperceptionofthepaint- ing?

6. Whatqualitiesof thepaintingweresupportedthroughthehaptic experience?

7. Wouldyouhaveexpectedsomethingelse,ifatall?

8. Anythingelseyouwouldliketoshareorsayabouttheexperience ofthisartinstallation?

Ineachinterviewsession,betweentwoandfourusersparticipatedat atime.Eachparticipantwasencouragedtoexpressher/hisopinionone

Fig. 10. Questionnaire about the importance of each individual sense.

afteranother,aswellastoreacttoeachother’sresponsestoallowsome discussionandreflectiononthemultisensoryexperiences.Thiscould helptoobtainfurtherinsightaboutthevisitorexperiencesintheirown words.

Participantssignedaconsentformbeforetakingpartinthestudy, whichwasapprovedbytheUniversityofSussexScienceandTechnology ethicscommittee.

5. Results

Intotal,wecollecteddatafrom 2500participants(1700 females, 800males,meanage36.00SD16.11).Weanalysedparticipants’ visual liking,multisensoryexperienceliking,andemotionalreaction(arousal) ratingsusingamixedeffectdesign,ANOVA,wherepaintingwasconsid- eredawithin-participantsfactor,andgenderwereconsideredbetween- participantfactors.Weusedagetoinvestigatehowdifferentagegroups perceivedthesensoryaugmentationofthepaintingsandtocalculate correlationswiththeparticipant’sratings.Weadded‘hapticpatterns’ asbetweenfactorintheanalysisinordertoinvestigateanydifferences acrossthethreehapticpatternsusedinrelationtotheparticipant’srat- ings.

FullinteractionswereconsideredineachANOVA modelweused.

Overall,ANOVA’sassumptionsweretestedonallthecombinationsof betweenandwithinfactors.TheSaphiro-Wilktestindicatedthenormal distributionofthedata(p>0.05inallcases),Mauchly’stestofsphericity wasusedtoassessthesphericityofthedata(again,p>0.05inallcases), andLevene’stestthehomogeneityofthedata(p>0.05inallcases).

WhenANOVAsshowedsignificance,Bonferroni-correctedpairwise comparisonswereperformed.Moreover,giventhehighnumberofpar- ticipants,Cohen’sdwasusedoneachsignificantcomparisonasanin- dexoftheeffectsize.Notethattheeffectsizewasnotcomputedatthe ANOVA level,giventhefactthatthepoweranalysisof multipleway mixedeffectexperimentaldesignscanleadtonegativevaluesanddiffi-

Table 1

Selected paintings and their associated sense designs.

Paintings

# 1 Interior II by Richard Hamilton √ √ √

# 2 Full Stop by John Latham √ √ √

# 3 In the Hold by David Bomberg √ √ √

# 4 Figure in a Landscape by Francis Bacon √ √ √

Table 2

Overview on the results for the three mid-air haptic patterns created for the Full Stop painting, based on number of participants and ratings on visual liking, multisensory experience liking and experienced arousal.

#1: Tate #2: Circle #3: Line

Number of participants 1889 133 152

Visual liking 3.99 ± 1.04 4.05 ± 1.03 3.97 ± 1.00 Multisensory experience liking 4.13 ± 0.97 4.14 ± 1.00 3.98 ± 0.99 Arousal 3.77 ± 1.04 3.90 ± 0.97 3.50 ± 1.13

cultinterpretation,anditisstillanactivefieldofresearch(Robertsand Monaco,2006).

Inadditiontothequestionnairedata,wecollectedqualitativedata from50participantsthroughconductinginterviewsonthelastdayof themultisensorydisplay.Alltheinterviewsweretranscribedandanal- ysedbyoneresearcher(whoconductedtheinterviews)basedonthe mainareasofinterestdefinedabove(seeSection6).

Basedonrepeatedreadingsofthetranscriptsanddiscussionsinthe group, we clusteredthe findings into three mainthemes, which we presentin thefollowingsectionsafterthequantitativeresultsgained fromthequestionnaire.

5.1. Effectofthedifferentmid-airhapticpatterns

Withtheaimofinvestigating theadd-valuesofmid-airhapticin amuseumcontext,we wereparticularlyinterested inevaluating the effectofmid-airhapticfeedbackonparticipants’ experiences.Forthat purpose,threevariationsofhapticpatternswerecreatedfortheFullStop paintingandalternatedduringthededicatedSciencedays(seeFig.6for illustrationsofthehapticpatterns).Table1.

Table2summarizesthenumbersofparticipantsthatexperiencedthe differentmid-airhapticpatterns(Tate,Circle,andLine).Pleasenotethat thealternationbetweenpatternswasconstrainttothededicatedScience days,hencethereisadifferentnumberofparticipantsexperiencingeach pattern.

Theexpectationwasthatparticipantswouldlikethemainpattern purposelydesignedforTatemost,followedbytheCirclepattern,and theLinepatternbeingtheleastlikedduetoitsincongruencewiththe visualappearanceofthepainting(roundedshapeoftheFullStopona largecanvas).

Totestthis hypothesis(thatis:whetherthedifferent patternsin- fluencedtheratingsoftheparticipants),threemultiplewayANOVAs wereusedtoanalysethevisualliking,multisensoryexperienceliking, andarousalratings,havingasindependentvariablestheageofthepar- ticipants,theviewingorderof thepaintings,andthedifferenthaptic patternsintothemodel.

Theanalysisshowedthatthedifferentmid-airhapticpatternsonly hadaneffecton thereportedarousal (F=4.129,p<0.01). Nostatis- ticallysignificantinteractionwasobserved(p>0.05inallcases).Fig.

11showstheaveragedratingsforeachpattern.Pairwisecomparisons, usingtheBonferronicorrection,showedthatpattern1andpattern2 (Tate3.77±1.04andCircle3.90±0.96)werefoundtobemorearous- ingcomparedtopattern3(Line3.50±1.13,Cohen’sdtotheclosest value=0.38).TheseresultsareinlinewithourexpectationoftheLine

Fig. 11. (top) Ratings of arousal, visual liking, and multisensory experience liking for the different haptic patterns (with standard deviation, on a Likert scale of 1 to 5). (bottom) The schematic representation of the pattern on participant’s hand: (1) Tate custom made;

(2) circle; and (3) line.

Table 3

Summary of visitor ratings for each sense (with standard de- viations) for the Full Stop painting (associated with mid-air haptic patterns).

Sight Sound Touch Scent Taste Mean 4.40 4.23 4.15 1.53 1.49

SD 0.91 1.03 1.15 0.96 0.95

patternbeingtheleastappropriatesensationinmid-airasitdoesnot resembletheroundedcharacteristicofthepainting.

5.2. Importanceofhapticexperience

SpecifictotheSciencedays(asdescribedaboveandshowninFig.7), participantswereaskedoneadditionalquestiondesignedtoassessthe perceivedimportanceofeachsenseineachofthemultisensoryexperi- ences(e.g.,Ratetheimportanceofeachofyoursensesinthisexperience).

Thiswasinspiredbypreviousworkassessingtherelativeimportance, topeople,ofthefivesensesinagivenexperience(AdankandWarell, 2006).

Table3andFig.12showtheaverageparticipants’ ratings(withstan- darddeviation)oftheimportanceofhapticfortheFullStoppainting.

ArepeatedmeasureANOVAandpost-hocpairwisecomparisonswith Bonferronicorrectionwereusedtoassesswhichsenseswereconsidered moreimportantforthepainting.

Wefoundthatratingsoftouchasratedsignificantlymoreimportant (p<0.001)comparedtotheratingsofscentandtaste.Thisisasexpected forthispaintingasitwasdesignedwiththemid-airhaptic(thesenseof touch).

Fig. 12. The reported importance of haptic sense in the multisensory experience for the painting “Full Stop ”. Each sense is represented by a vertex of the pentagon, while each scale (from 1 - centre to 5 - vertex) are represented by the line and the points connecting the centre of the pentagon to the vertex. The solid black line represents the mean; the dotted lines represent standard deviation.

MultiplewayANOVAswerealsoconductedtoassessanydifferences ingender,hapticpatterns,ontherelativeimportanceofthedifferent sensesintheirexperience.Nosignificanteffectofanyofthesefactors wasfound(p>0.05inallcases).Thatmeansthatparticipantsratedthe addedexperiencesoftheassociatedsensesimilarly,regardlessoftheir genderandhapticpatterns.

6. Interviewfindings

Asmentionedbefore,theaimoftheinterviewswastogainmore insightsintoparticipants’ overallexperienceofthemultisensoryinstal- lation,andmorespecificallytoobtainqualitativefeedbackontheirex- periencefortheFullStoppainting.Belowwesummarisethemainfind- ings,furtherillustratedthroughquotesfromparticipants(n=50).We firstpresentthequalitativefindingsoftheoverallexperienceofthemul- tisensoryexhibition(Section6.1and6.2),followedbythefindingsthat focusontheexperiencesoftheFullStoppainting,withthemid-airhaptic feedback(Section6.3and6.4).

6.1. Overallmulti-facetedexperiences:immersivevsdistracting

ParticipantsdescribedtheirexperienceofTateSensoriumas“stimu- lating”,“interesting”,“mindblowing”,“incredible,Ireallyenjoyedit”,

“somethingnew,unusual”.Whiletheirfeedbackwasoverwhelmingly positive– whichalsofitsthequantitativeresults– therewerealsosome morecriticalvoices.Thesecriticsweremainlybasedondifferentexpec- tations,suchasthoseexpressedbysomeparticipantsas“I’dsayitwasn’t asstrongasIthoughtitwouldbe”,and“Iexpectedsomethingdifferent,like somethinginvolvingmywholebodymaybe,butIdidlikethatIfeltthings verydifferentineverypainting.”Someparticipantsliterallyexpecteda completefullbodyimmersioninthepaintingthroughthestimulation ofallsenses.Oneparticipantwasevenreadytotakeoff theirshoesin expectationtobestimulatedonthefeet.

All participants strongly acknowledged that stimulating all the sensesaddedanotherlayer,dimension,andperspectivetotheexperi- enceofthepaintingsandthusopenednewwaysofthinkingandinter- pretingart,inparticularabstractart,whichsometimesleavespeople wonderinghowtointerpretthework.Oneparticipantsaid:“Ithelped createlikeastoryforeachpaintingbecausesomeofthesepaintingsarequite abstract,sothenwiththesoundsorthesmellsyoukindofbegintostartcre- atinganideaofwhat’sactuallygoingoninthepaintingorwhatthestory is.”Themajorityofparticipantsstatedthatadditionalsensorystimuli didnotchangetheirinitiallikingoftheartwork.However,somepar- ticipantshighlightedthepotentialofmultisensorystimulitoturntheir

attentiontowardpainting.“Itmademefeelreallydifferent.TheFullStop andthereasonIlikeditisIwouldneverbeveryimpressedwithanimage likethatnormallybutthesound,itwasreallyawesome.”Theinterviews broughttotheforethegeneralfeelingthatsensoryaugmentationcan awakenamuseumvisitor’simagination,makethevisittothemuseum orartgallerymoreengaging,andhastheabilitytoelicitstrongreac- tions,establishaconnectionto,andbuildanarrativearoundtheart.

Themultisensorylayersontopofthevisualappearanceofthepaint- ingswasdescribedtoallowstrongeremotionalreactions,suchasempa- thy,beingimmersed,orevenscaredinfrontoftheartwork.Onepartic- ipantdescribeditasfollows:“Inawaythatgavethepaintinganarrative havingthatchocolate,youcouldbuildupastorylikemaybeyou’rewalking onthefield.[…]andyoucouldalmostpullthemoodfromthesunshineas well.”FortheFullStoppainting,thesensoryexperiencewasdescribedas veryintenseduetotheintegrationofmid-airhapticsandsound.While oneparticipantstatedthat“Ilovedthesoundofthatone.Itwaskindof scary”,anotherparticipantfocusedonthesensationonthehand“Itwas strange,itfreakedmeoutbecauseIwantedtopullmyhandout[fromthe plinth]butIdidn’twanttobecauseIwantedtocarryonandseewhatitwas like.”

Inaddition,participantshighlightedtheopportunityanddangerof multisensorystimuli.Forexample,itcouldeither‘helpfocus’onthepar- ticularitiesofanartworkor‘distract’fromtheartworkitself.Involving allthesenses,whenexperiencinganartworkforthefirsttimeinsuch asettingcouldcausedistraction,whichwas,however,notalwaysde- scribedasnegativedistraction.Instead,itwassometimesawelcomed distraction,asthefollowingstatementsrepresent:“Ilikedthepainting andIwaskindofdisturbedbythestrongsound”versus“It’safunnything butherethevisualpartwasdistracting.Iwasclosingmyeyesandtryingto listentothesoundandtouchingandimaginingbecauseIhadthepaintingin frontofmeevenifIclosemyeyes.”FortheFullStoppainting,onepar- ticipantpointedtothepositiveemphasisofthehapticstimulusonthe handwhichmadehernoticetheparticularitiesoftheartwork:“Icould kindofseeitbecauseofthespray,Inoticeditatthestart,Ithinkonthe righthandcorneritlookslikeit’speteringoutabitanditmademeseethat becauseIwasimaginingsmalldropletsandIsawthatwhereasIhadn’tseen it… [withoutthefeelingonthehand]”.

6.2. Balanceinsensorydesign:curatedvs.explorative

Theimpactofthesensorystimulioneachindividual’sexperience wasnotalwaysstraightforwardandsometimesbipolarinthesensethat multisensoryaugmentationofartcaneitheropenupopportunitiesfor interpretation,butcanalsonarrowdownthevisitor’sperspective.

Ontheonehand,participantsdescribedthemultisensoryexperience assupportiveinunderstandingart,creatingastory,elevatingthevisual experience throughtouch, taste,andsmellandsound. Whileon the otherhand,theexperiencewasdescribedastooprescriptive,orches- trated,andshepherded.Oneparticipantstated:“Ifeltlikeitwasleading yousomewherebecauseitwasalreadyachoice,itwasanotherchoicefrom someoneelse,soIfeltlikeIwasbeingdraggedintosomeoneelse’s”.Another participantmadethefollowingstatement:“Ithinkitwasinterestingto viewthepaintingsinadifferentwaybutIthinkitwasalittlebittoocon- ducted,especiallythefirstone.Youseethispaintingandyousmellthesmell andyouknow,itwastooobviousineveryoneofthem.Thesoundismatching perfectlythepaintingandthesmellwasmatchingperfectlythepaintingand thefeelingofthehandwasmatchingperfectlytolittledotsandthespray.” Thereseemedtoemerge,althoughonlyfromahandfulofparticipants, afeelingofnotbeingincontrol,andmaybenotbeingabletofollow theirownexplorationofthesensesalongsidetheart,butthenagainbe- ingexcitedaboutthenoveltyoftheengagement.Thisleavesspacefor otherwaysofdesigningfuturemultisensoryexperiencesandcreating aninteractivesettinginamuseumservingthevaryingexpectationsof visitors:beingguidedorallowingforsurprise.