DISSERTATION
The Personal Interaction Panel
a two-handed Interface for Augmented Reality
ausgeführt zum Zwecke der Erlangung des akademischen Grades eines Doktors der technischen Wissenschaften unter Leitung von
Ao. Univ. Prof. Dipl-Ing. Dr.techn. Michael Gervautz E 186/2
Institut für Computergraphik
eingereicht an der Technischen Universität Wien Technisch-Naturwissenschaftliche Fakultät
von
Dipl.-Ing. SZALAVÁRI Zsolt Matr. Nr. 9326205
Hernalser Hauptstraße 168/18, A-1170 Wien
Wien, im September 1999
3$*(,,
$EVWUDFW
7KH ZD\ ZH SHUFHLYH WKH UDSLGO\ H[SDQGLQJ LQIRUPDWLRQ HQYLURQPHQW ZLOO FKDQJH GUDPDWLFDOO\ RYHU WKH QH[W \HDUV $XJPHQWHG 5HDOLW\ UHVHDUFK RIIHUV D VPRRWK LPPHUVLRQLQWRWKLVSDUDOOHOZRUOGE\RYHUOD\LQJVSDWLDOO\DOLJQHGWKUHHGLPHQVLRQDO FRPSXWHUJHQHUDWHGLQIRUPDWLRQRQWRDKXPDQ·VYLHZ+RZHYHUWRPDQLSXODWHWKH SHUFHLYHG LQIRUPDWLRQ QHZ W\SHV RI LQWHUIDFHV DUH QHHGHG WKDW DUH VPRRWKO\
LQWHJUDWHGDWWKHERUGHURIRXUUHDOZRUOGDQGWKHLQIRUPDWLRQVSDFH
)RUWKHLQWHUDFWLRQZLWKWKLVYLUWXDOFRQWHQWWKH3HUVRQDO,QWHUDFWLRQ3DQHO3,3D WZRKDQGHG LQWHUIDFH IRU $XJPHQWHG 5HDOLW\ LV LQWURGXFHG 7KH XQGHUO\LQJ GHVLJQ LGHDHPSOR\VH[LVWLQJKXPDQVNLOOVIRUWKHLQWHUDFWLRQLQDQHZHQYLURQPHQWVRWKDW WKHLQWHUDFWLRQWDVNLWVHOIGRHVQRWLQGXFHDGGLWLRQDOFRJQLWLYHORDGRQWKHXVHU7KH FDUHIXOFRQFHSWLRQRIWKH3,3LQWHUIDFHHQVXUHVWKDWLWFRQIRUPVWRSUHYLRXVUHVXOWVLQ ELPDQXDODFWLRQDQGEHQHILWVIURPWKHREVHUYDWLRQVRIWKLVUHVHDUFKILHOG
7KHUHVXOWVRIWKHDQDO\VLVRIH[LVWLQJPHWKRGVWRLQWHUDFWLQ$XJPHQWHG5HDOLW\DUH VXPPDUL]HG DV %DVLF 'HVLJQ *XLGHOLQHV 7KHVH JXLGHOLQHV DUH WKH EDVLV IRU WKH WKHRUHWLFDOIRUPXODWLRQDQGFRQFHSWLRQRIWKHSURSRVHGLQWHUIDFH$ZLGHVSHFWUXP RQGLIIHUHQWPHWDSKRUVLVSUHVHQWHGWKDWEHQHILWIURPWKHXQGHUO\LQJLQWHUIDFHGHVLJQ DQGWKXVLPSURYHVWKHXVHU·VLQWHUDFWLRQDELOLWLHV
7KHLPSOHPHQWDWLRQRIWKHGHVFULEHGWKHRUHWLFDOUHVHDUFKDQGWKHILHOGWHVWLQJ LQ D QXPEHU RI DSSOLFDWLRQ VFHQDULRV KHOSHG WR SURYH WKH GHVFULEHG LGHDV 8VLQJ WKH H[SHULHQFH JDLQHG IURP WKLV ZRUN WKH LQWHUIDFH ZDV DSSOLHG WR GLIIHUHQW HQYLURQPHQWVZLWKGLYHUVGHPDQGV7KHLQIOXHQFHVIURPWKHVHH[SHULPHQWVKHOSHGWR UHFRJQL]HWKHXQLYHUVDOFKDUDFWHURIWKH3HUVRQDO,QWHUDFWLRQ3DQHODQGWRFRQFOXGH WKDW WKH 3,3 LQWHUIDFH FDQ EH D JHQHUDO WRRO IRU LQWHUDFWLQJ ZLWK YLUWXDO FRQWHQW LQ PDQ\GLIIHUHQWHQYLURQPHQWV
7+(3(5621$/,17(5$&7,213$1(/²$7:2+$1'(',17(5)$&()25$8*0(17('5($/,7<
3$*(,,,
.XU]IDVVXQJGHU'LVVHUWDWLRQ
'LH$UWXQG:HLVHZLHZLUPLWGHPVWlQGLJZDFKVHQGHP,QIRUPDWLRQVUDXPGHUXQV XPJLEW XPJHKHQ ZLUG VLFK LQ GHQ QlFKVWHQ -DKUHQ GUDPDWLVFK lQGHUQ 'LH )RUVFKXQJDXIGHP*HELHWGHU$XJPHQWHG5HDOLW\ELHWHWHLQHVDQIWH0HWKRGHVLFK GLHVHU SDUDOOHOHQ :HOW ]X QlKHUQ 6LH EHUODJHUW GHQ PHQVFKOLFKHQ %OLFN PLW GUHLGLPHQVLRQDOHUFRPSXWHUJHQHULHUWHU,QIRUPDWLRQGLHUlXPOLFKHQ%H]XJKDW8P GLHVHQ ZDKUJHQRPPHQHQ ,QIRUPDWLRQVUDXP EHHLQIOXVVHQ ]X N|QQHQ ZHUGHQ QHXH 6FKQLWWVWHOOHQ JHEUDXFKW GLH DQ GHU *UHQ]H ]ZLVFKHQ UHDOHU XQG YLUWXHOOHU :HOW DQJHVLHGHOWVLQG
'LHVH $UEHLW WUlJW ]X GLHVHU 3UREOHPVWHOOXQJ PLW GHU (LQIKUXQJ GHV 3HUVRQDO ,QWHUDFWLRQ3DQHOVEHL'HP]ZHLKlQGLJHQ,QWHUIDFHOLHJWGLH,GHH]X*UXQGHGDVV H[LVWLHUHQGH PHQVFKOLFKH )lKLJNHLWHQ YHUZHQGHW ZHUGHQ XP DXI HLQH QHXH 8PJHEXQJHLQ]XZLUNHQ=LHOLVWDXFKGDVVGLH0DQLSXODWLRQVHOEVWNHLQHNRJQLWLYH
%HODVWXQJ GDUVWHOOW VRGDVV VLFK GHU %HQXW]HU DXI GLH HLJHQWOLFKH $XIJDEH NRQ]HQWULHUHQ NDQQ 'DV VRUJIlOWLJH .RQ]HSW VLFKHUW GDVV GLH YRUJHVWHOOWH 6FKQLWWVWHOOH YRUDQJHJDQJHQHQ (UNHQQWQLVVHQ DXV GHP %HUHLFK GHU EHLGKlQGLJHQ ,QWHUDNWLRQHQWVSULFKWXQGYRQGHVVHQ5HVXOWDWHQSURILWLHUW
'LH (UJHEQLVVH GHU 8QWHUVXFKXQJ DQGHUHU ,QWHUDNWLRQVPHWKRGHQ LQ $XJPHQWHG 5HDOLW\ 8PJHEXQJHQ ZXUGHQ DOV /HLWIDGHQ ]XP (QWZXUI QHXHU 6FKQLWWVWHOOHQ ]XVDPPHQJHIDVVW'LHVH5LFKWOLQLHQOLHJWDXFKGHUWKHRUHWLVFKHQ)RUPXOLHUXQJXQG
$XVDUEHLWXQJ GHU YRUJHVWHOOWHQ 6FKQLWWVWHOOH ]X *UXQGH (LQ EUHLWHV 6SHNWUXP DQ
$QZHQGXQJVP|JOLFKNHLWHQ ZLUG YRUJHVWHOOW 'LHVH SURILWLHUHQ YRP ,QWHUIDFHGHVLJQ XQGHUZHLWHUQVRGLH,QWHUDNWLRQVIlKLJNHLWHQGHV%HQXW]HUV
'LHEHVFKULHEHQHQ,GHHQNRQQWHQPLW+LOIHYRQYHUVFKLHGHQHQ,PSOHPHQWLHUXQJHQ XQGGHU(USUREXQJLPSUDNWLVFKHQ(LQVDW]EHOHJWZHUGHQ0LW+LOIHGHUJHZRQQHQHQ (UIDKUXQJHQ NRQQWH GLH 6FKQLWWVWHOOH DXFK LQ DQGHUHQ YLUWXHOOHQ 8PJHEXQJHQ HLQJHVHW]W ZHUGHQ 'LH (UNHQQWQLVVH GLHVHU ([SHULPHQWH YHUKDOIHQ ]XP (UNHQQHQ GHU 8QLYHUVDOLWlW GHV 3HUVRQDO ,QWHUDFWLRQ 3DQHOV 'DUDXV UHVXOWLHUHQG ZXUGH IHVWJHVWHOOWGDVVGLH3,36FKQLWWVWHOOHDOVDOOJHPHLQHV:HUN]HXJIUGLH+DQGKDEXQJ YRQYLUWXHOOHQ,QKDOWHQLQYHUVFKLHGHQVWHQ6]HQDULHQEHWUDFKWHWZHUGHQNDQQ
3$*(,9
$FNQRZOHGJPHQWV
7KLVSDJHLVGHGLFDWHGWRDOOSHRSOHKHOSLQJWKLVWKHVLVWRLWVH[LVWHQFH
)LUVW , ZDQW WR H[SUHVV P\ JUDWLWXGH WR P\ VXSHUYLVRU 0LFKDHO *HUYDXW] ZKR VXSSRUWHGP\WKHVLVLQDOOSRVVLEOHZD\VDQGLQVSLUHGPHWRWKLVLPDJLQDWLRQ,OLNHWR WKDQN:HUQHU3XUJDWKRIHUIRUDGRSWLQJPHWRKLVJURXSLQ9LHQQD
$VDFRKDELWDQWRI6WXGLHUVWXEH,WKDQN'LHWHU6FKPDOVWLHJDQG$QWRQ)XKUPDQQ'LHWHU PRWLYDWHGPHLQVHYHUDOGLVFXVVLRQVDQGRSHQHGRXUH\HVIRUWUDQVSDUHQF\7RQLVHW XSRXUFRPPRQVQXJJHU\LQHYHU\PHDQ
,DOVRZDQWWRWKDQNP\FROOHDJXHV(ULN(FNVWHLQ5HLQKDUG6DLQLW]HU+HUEHUW%XFKHJJHU
&KULVWLDQ)DLVVWQDXHUDQGWKHVWXGHQWV+HUPDQQ:XUQLJDQG:ROIJDQJ*UHLPHOIRUWKHLU JUHDWZRUNLQLPSOHPHQWLQJ3,3DSSOLFDWLRQV,UHQGHUWKDQNVWR5REHUW7REOHUZKR JDYH PH KLV GLJLWDO 1LNRQ WR PDNH VRPH RI WKH SKRWRV IRU WKLV WKHVLV DQG +HOZLJ /|IIHOPDQQIRUFRQWLQXRXVO\SLFNLQJDWPHWRZULWHGRZQP\WKHVLV
, SDUWLFXODUO\ WKDQN P\SDUHQWV IRU JURZLQJ PH XS LQ ORYH DQG WR JURXQG PH LQ VWUXJJOLQJ
/DVWEXWQRWOHDVWP\VSHFLDOWKDQNVJRWRP\ZLIH9HURQLNDIRUKHUSDWLHQFHZLWKPH DQGP\OLWWOH=VyILZKRP,PLVVHGWRVHHZKHQVWDQGLQJXSWKHILUVWWLPHEHFDXVHRI ILQLVKLQJWKLVWKHVLV²,KRSH\RXZLOOIRUJLYHPH
7+(3(5621$/,17(5$&7,213$1(/²$7:2+$1'(',17(5)$&()25$8*0(17('5($/,7<
3$*(9
7DEOHRI&RQWHQWV
&+$37(521(,1752'8&7,21
&+$37(57:267$7(2)7+($57 ,1752'8&7,21
6859(<2)$8*0(17('5($/,7<
9LUWXDO5HDOLW\DQG$XJPHQWHG5HDOLW\
$XJPHQWHG5HDOLW\6\VWHPV
,17(5$&7,21,1$8*0(17('5($/,7<
,QWHUDFWLQJZLWK,QIRUPDWLRQ ,QWHUDFWLRQ0HWDSKRUVIRU$XJPHQWHG5HDOLW\
%,0$18$/,17(5$&7,21
3V\FKRORJ\
%LPDQXDO,QWHUIDFHVIRU',QWHUDFWLRQ 7ZR+DQGHG,QWHUDFWLRQLQ,PPHUVLYH(QYLURQPHQWV 6XPPDU\RI$56\VWHPVDQG,QWHUDFWLRQ0HWDSKRUV %DVLF'HVLJQ*XLGHOLQHVIRU$5,QWHUIDFHV
&+$37(57+5((&21&(37,21
)81'$0(17$/'(6,*1*2$/
7+(3(5621$/,17(5$&7,213$1(/
3,39$5,$176,17+('(6,*1352&(66 3UHVVXUHVHQVLWLYH)ODW3DQHO'LVSOD\ZLWK3HQ2EVHUYHGZLWK6WHUHRVFRSLF*ODVVHV 3UHVVXUHVHQVLWLYH)ODW3DQHOZLWK3HQDQGD6HHWKURXJK+HDG0RXQWHG'LVSOD\
´'XPEµ3DQHODQG3HQLQ&RPELQDWLRQZLWKD6HHWKURXJK+HDG0RXQWHG'LVSOD\
)($785(62)7+(3+<6,&$/,17(5)$&(
´'LQ'µ $V\PPHWULF%LPDQXDO,QWHUDFWLRQ +DSWLFV
0(7$3+256)25,17(5$&7,217$6.6
2EMHFW0DQLSXODWLRQ7DVNV 1DYLJDWLRQ7DVNV $SSOLFDWLRQ&RQWURO7DVNV 3UHFLVLRQ(QKDQFLQJ7HFKQLTXHV 6SDWLDOO\&RQWUROOHG6HPDQWLFV 3ULYDF\DQG3XEOLFLW\
$SSOLFDWLRQ6SHFLILF0HWDSKRUV 2UJDQL]DWLRQRI,QWHUDFWLRQ(OHPHQWV 6XPPDU\
&+$37(5)285,03/(0(17$7,21
+$5':$5(6(783
*HQHUDWLRQ *HQHUDWLRQ *HQHUDWLRQ
'(6,*12)7+(9,578$/3$57
*8,'HVLJQ ,QWHUDFWLRQ(OHPHQWV
3$*(9,
&+$37(5),9($33/,&$7,21
2%-(&70$1,38/$7,216&(1('(6,*1
'UDJDQG'URS 3UHFLVHDQG&RQVWUDLQHG0DQLSXODWLRQ 2EMHFW%URZVHU 5RRP'HVLJQHU
9,(:32,170$1,38/$7,21
0DS 3KRWR&DPHUD 6HDPV
6<67(0&21752/
7DEVDQG6KHHWV $SSOLFDWLRQ/RDGHU
6&,(17,),&9,68$/,=$7,21
6WXGLHUVWXEH )ORZ9LVXDOL]DWLRQ 5|QWJHQ &2//$%25$7,9(*$0,1*
:K\*DPLQJ"
3DOHWWHRI$XJPHQWHG*DPHV *DPH7HVWEHG 3ULYDF\0DQDJHPHQW 0DK-RQJJ
&+$37(56,;75$16/2&$7,21
3,3,1'(6.72395$33/,&$7,216
950&6D6SDFHFUDIW0RQLWRULQJDQG&RQWURO(QYLURQPHQW
3,3)257+(&$9((19,5210(17
3,3)257$%/((19,5210(176
352326(')857+(5$33/,&$7,21$5($6$1'(19,5210(176
&+$37(56(9(1&21&/86,216 &21&/8',1*5(0$5.6 &216,'(5$7,216)257+()8785(
&+$37(5(,*+75()(5(1&(6
&+$37(51,1($33(1',&(6 &855,&8/809,7$(
5(/$7('38%/,&$7,216
Chapter One - Introduction
6FLHQFH KHDGV WKH JRDO WR GHVFULEH WKH XQLYHUVH DV GHWDLOHG DV SRVVLEOH LQ HYHU\
GLVFLSOLQHWRRSHQWKHNQRZQXQLYHUVHDWHDFKVFDOH (YHQ WKRXJK VRPH SKLORVRSKLF JURXSVGHQ\HYHQWKHH[LVWHQFHRIDXQLYHUVHWKHUHVWRIPDQNLQGDOZD\VZDQWHGWR DPSOLI\ WKH RZQ VHQVHV WR JHW EHKLQG WKH GLUHFWO\ SHUFHLYHG UHDOLW\ ([DPSOHV RI PHWKRGV IRU WKLV DPSOLILFDWLRQ RI WKH VHQVHG UHDOLW\ UHDFK IURP WKH WDNLQJ RI KDOOXFLQRJHQVWKHLQYHQWLRQRIWKHWHOHVFRSHWKHGHVLJQRIVSHFWURPHWHUVWRWKHKXJH DFFHOHUDWRUVIRUSLFNLQJWKHEXLOGLQJEORFNVRIPDWWHULQWRSLHFHV&RPPRQLQWKHVH DSSURDFKHVLVWKHHIIRUWWRVKRZWKHLQYLVLEOHSDUWRIUHDOLW\ZKLFKZDVQRWSUHVHQWWR WKHVHQVHVEHIRUH
7RVKRZDQLQYLVLEOHSDUWRIVXUURXQGLQJVLVDOVRWKHPRVWLPSRUWDQWREMHFWLYHRIWKH PHWKRGFDOOHG$XJPHQWHG5HDOLW\$5
7KRW²(J\SWLDQJRGRIZULWLQJDQGVFLHQFH a%&
&+$37(521(²3$*(
7KLVPHWKRGIRUYLVXDOLPSURYHPHQWRUHQULFKPHQWRIWKHVXUURXQGLQJHQYLURQPHQWE\RYHUOD\LQJ VSDWLDOO\DOLJQHGWKUHHGLPHQVLRQDOFRPSXWHUJHQHUDWHGLQIRUPDWLRQRQWRDKXPDQ·VYLHZSUHVHQWVD V\QWKHWLFUHDOLW\GHVLJQHGWRDPSOLI\RXUSHUFHSWLRQ$UWLILFLDOO\JHQHUDWHGREMHFWVRU DQ\NLQGRIDEVWUDFWLQIRUPDWLRQDSSHDUVWRWKHREVHUYHUEOHQGHGZLWKWKHHYHU\GD\
UHDOLW\LQWKUHHGLPHQVLRQV
:K\LV$5DQLQWHUHVWLQJWRSLFDQGZLOOEHDXVHIXOWRROLQWKHIXWXUH"7KHDELOLW\WR UHSUHVHQWQRWGLUHFWO\SHUFHSWLEOHLQIRUPDWLRQZLWKYLUWXDOREMHFWVPDNHVWDVNVHDVLHU WRSHUIRUPIRUKXPDQV7KLVYLVXDOFRPSXWHUVXSSRUWLVDVSHFLILFH[DPSOHRIZKDW )UHG%URRNVGHVFULEHVLQJHQHUDODVLQWHOOLJHQFHDPSOLILFDWLRQ>%URRNV@
0RUHWKDQMXVWVLPSOHYLVXDOHQKDQFHPHQWDQGVXSSRUWLQWHUDFWLYHDXJPHQWHGUHDOLW\
DSSOLFDWLRQVZLOOFKDQJHWKHZD\RILQWHUDFWLRQZLWKDEVWUDFWLQIRUPDWLRQLQIXWXUH%\
OHWWLQJXVHUVRI$5V\VWHPVWRPDQLSXODWHDXJPHQWHGFRQWHQWWKHEOHQGRIUHDODQG YLUWXDOEHFRPHV\HWPRUHHIIHFWLYHDVIOH[LELOLW\LVLQWURGXFHGLQWRWKHV\VWHP7KLV SRZHUIXO WRRO KDV D JUHDW SRWHQWLDO IRU D EURDG UDQJH RI DSSOLFDWLRQV LQFOXGLQJ PRELOHFRQWH[WVHQVLWLYHLQIRUPDWLRQV\VWHPVVFLHQWLILFYLVXDOL]DWLRQLQSODFHGLVSOD\
RI PHDVXUHPHQW GDWD PHGLFLQH DQG VXUJLFDO SODQQLQJ HGXFDWLRQ WUDLQLQJ DQG HQWHUWDLQPHQW
7R FRPH XS WR WKH DSSOLFDWLRQ H[SHFWDWLRQV XVHU LQWHUDFWLRQ ZLWK WKH VXSSRUWLQJ FRPSXWHUKDVWRXQGHUJRGUDPDWLFFKDQJHVFRPSDUHGWRWRGD\·VLQSXWGHYLFHVZKLFK FRQFHQWUDWH RQ D WH[W EDVHG LQSXW RU WZRGLPHQVLRQDO VHOHFWLRQ :H DUH DW WKH EHJLQQLQJRIDQHZHUDRILQWHUDFWLRQSDUDGLJPVWRKDQGOHWKUHHGLPHQVLRQDOFRQWHQW LQ D QDWXUDO DQG IOH[LEOH ZD\ 3UHVHQW ZRUN FRQWULEXWHV WR WKLV HYROXWLRQ ZLWK WKH LQWURGXFWLRQRIWKH3HUVRQDO,QWHUDFWLRQ3DQHO²WKH3,3²DWZRKDQGHGLQWHUIDFHIRU DXJPHQWHGUHDOLW\
Chapter Two - State Of the Art
,QWURGXFWLRQ
6LQFH WKH EHJLQQLQJ RI WLPH PDQNLQG ZDV DOZD\V IRQG RI XVLQJ LPDJHV WR H[SUHVV YDULRXV WKRXJKWV &DUYHG LQWR WKH ZDOO RI D FDYH SDLQWHG RQWR SDS\UXV RU VLPSO\
GHVFULEHGZLWKZRUGVWRIRUPDPHQWDOLPDJH3DUWQHUVLQDGLVFXVVLRQHQYLVLRQLGHDV RU SODQV WR VKDUH D FRPPRQ NQRZOHGJH RQ DQ DEVWUDFW WRSLF &RPPXQLFDWLQJ WKHRUHWLFDO FRQWHQW XVLQJ LPDJHV PDNHV FRPSUHKHQVLRQ PXFK IDVWHU DV KXPDQV DFTXLUHWKHPRVWLQIRUPDWLRQWKURXJKWKHLUH\HV
7KH LQIRUPDWLRQ DJH FUHDWHG SRZHUIXO WRROV WR FUHDWH WUDQVSRUW DQG VKRZ LPDJHV 6WDWLFSLFWXUHVILJXUHVDQGSKRWRJUDSKVHYROYHGLQWRPRYLQJSLFWXUHVDQGDQLPDWLRQ 6WRUHG HDUOLHU RQ FHOOXORLG QRZDGD\V RQ PDJQHWLF PHGLD RU RSWLFDO GLVFV DQG GLVWULEXWHGYLDUDGLRZDYHVDQGFDEOHVG\QDPLFLPDJHVDUHDEDVLFDQGLQWHJUDOSDUWRI RXUHYHU\GD\OLIH1RZZHDUHDERXWWRFURVVWKHQH[WIURQWLHURILQWHUDFWLYHLPDJHV 0XOWLPHGLDDQGUDQGRPDFFHVVYLGHRPHGLDDQGV\VWHPVDOORZE\SDVVLQJWKHOLQHDULW\
(DGZLQ²0RQNDWWKH&DQWHUEXU\$EEH\
&+$37(57:2²3$*(
RI WLPH DQG KLVWRU\ RI D PRYLQJ SLFWXUH (DFK WLPH H[SHULHQFHG WKH FRQWHQW LV SUHVHQWHG LQ D GLIIHUHQW DQG QHZ ZD\ /HDUQLQJ DQG XQGHUVWDQGLQJ EUHDN ZLWK WUDGLWLRQ IURP QRZ RQ DFTXLULQJ NQRZOHGJH EHFRPHV D QRQOLQHDU SURFHVV RI LQWHUDFWLYHLQIRUPDWLRQH[SORUDWLRQ
7KHQH[WIURQWLHUWRFURVVZLOOEHWRH[SORUHFUHDWHDQGFRPPXQLFDWHLQIRUPDWLRQ GLUHFWO\LQWKUHHGLPHQVLRQV7KHH[SHULHQFHZLOOEHLQWHUDFWLYHLQDFRPIRUWDEOHDQG QDWXUDOZD\5HDOLW\DQGV\QWKHWLFFRQWHQWZLOOEHVHDPOHVVO\LQWHJUDWHGLQDPL[WXUH WKDW DPSOLILHV KXPDQ LQWHOOLJHQFH 7R ZRUN ZLWK WKLV V\QWKHWLF FRQWHQW FRPSOHWHO\
QHZ LQWHUIDFHV DUH QHHGHG ,Q IDFW LQWHUIDFHV RU LQWHUDFWLRQ GHYLFHV ZLOO EH IXQGDPHQWDOSDUWRIWKHVHDXJPHQWHGHQYLURQPHQWV
:KHUHDUHZHQRZ"+RZIDULVWKLVQH[WIURQWLHUDQGZKDWFDQWRGD\·VWHFKQRORJ\
SURYLGHWRPDNHWKHILUVWVPDOOVWHSVWRZDUGVWKLVIXWXUH"7KHQH[WVHFWLRQWULHVWR VXPPDUL]HWKLVDQGJLYHDQRYHUYLHZRIWKLVILHOG
6XUYH\RI$XJPHQWHG5HDOLW\
2.2.1 Virtual Reality and Augmented Reality
$VDKLJKO\LQWHUGLVFLSOLQDU\ILHOGVFLHQWLILFYLVXDOL]DWLRQIUHTXHQWO\UHTXLUHVH[SHUWV ZLWK GLIIHUHQW EDFNJURXQG WR FRRSHUDWH &ROODERUDWRUV PD\ KDYH GLIIHUHQW SUHIHUHQFHVFRQFHUQLQJWKHFKRVHQYLVXDOUHSUHVHQWDWLRQRIWKHGDWDRUWKH\PD\EH LQWHUHVWHG LQ GLIIHUHQW DVSHFWV $Q HIILFLHQW FROODERUDWLRQ UHTXLUHV WKDW HDFK RI WKH UHVHDUFKHUVKDYHDFXVWRPL]HGYLHZRIWKHGDWDVHW$WWKHVDPHWLPHSUHVHQFHLQWKH VDPHURRPLVSUHIHUUHGEHFDXVHRIWKHQDWXUDOLQWHUDFWLRQGXULQJDGLVFXVVLRQ7KHVH UHTXLUHPHQWVFDQXQLTXHO\EHIXOILOOHGLQDQDXJPHQWHGUHDOLW\V\VWHPWKDWFRPELQHV UHDO ZRUOG H[SHULHQFH RI WKH FROODERUDWRUV DQG SK\VLFDO HTXLSPHQW ZLWK WKH YLVXDOL]DWLRQRIWKHV\QWKHWLFGDWD
:K\LV$5'LIIHUHQW"
&RPSDUHGWRYLVXDOL]DWLRQLQLPPHUVLYHYLUWXDOUHDOLW\DXJPHQWHGUHDOLW\DOORZVWKH XVH RI GHWDLOHG SK\VLFDO PRGHOV WKH SURSHUWLHV RI ZKLFK FDQQRW EH PHW E\ WKHLU YLUWXDOFRXQWHUSDUWVDUELWUDULO\GHWDLOHGYLVXDOUHSUHVHQWDWLRQQRYLVXDORUWHPSRUDO DUWLIDFWVDQGIRUFHIHHGEDFNIRUIUHH2QO\WKRVHDVSHFWVRIWKHPRGHOWKDWFDQQRWEH
7+(3(5621$/,17(5$&7,213$1(/²$7:2+$1'(',17(5)$&()25$8*0(17('5($/,7<
&+$37(57:2²3$*(
VHHQLQUHDOLW\KDYHWREHDGGHGE\WKHFRPSXWHUV\VWHP)RUH[DPSOHRQHFRXOGWDNH WKHSK\VLFDOPRGHORIDQDLUSODQHRUDLUSODQHZLQJWRLQYHVWLJDWHWKHIORZDURXQGWKLV REMHFWZKLFKLVVLPXODWHGE\FRPSXWHUDQGDGGHGWRWKHGLVSOD\0DQLSXODWLRQRIWKH UHDOZRUOGPRGHOHJLWVRULHQWDWLRQLVPRUHLQWXLWLYHDQGVLPSOHUWRVXSSRUWWKDQD SXUHO\YLUWXDOHQYLURQPHQW$UHODWHGH[DPSOHZRXOGEHWKHXVHRIDKXPDQRLGWRUVR RUSXSSHWWKDWLVRYHUODLGZLWKPHGLFDOLQIRUPDWLRQIURPLQVLGHWKHKXPDQERG\LQ WKHVW\OHRI>%DMXUD@
7KLVFRPELQDWLRQRIFRQYHQWLRQDOH[SHULPHQWDOZRUNZLWKVFLHQWLILFYLVXDOL]DWLRQDQG DXJPHQWHGUHDOLW\WHFKQRORJ\OHDGVWRWKHFRQFHSWRIDQDXJPHQWHGODERUDWRU\ZKLFK ZRXOG SURYLGH D VXSHULRU UHVHDUFK HQYLURQPHQW LQ ZKLFK WR FRQGXFW H[SHULPHQWV WKDWDUHH[HFXWHGVROHO\LQVLGHWKHFRPSXWHUZKLOHPDLQWDLQLQJDFRQYHQWLRQDODQG IDPLOLDUZRUNVHWXS
7KH6WXGLHUVWXEH DSSURDFK FRQFHQWUDWHV RQ WKH VHDPOHVV FRPELQDWLRQ RI D SK\VLFDO ZRUOG ZRUNVSDFH DQG DQ DXJPHQWHG HQYLURQPHQW IRU PXOWLSOH XVHUV LQ WKUHH GLPHQVLRQVZLWKXQDIIHFWHGVRFLDOFRPPXQLFDWLRQFKDQQHOVDQGDQDXJPHQWHGXVHU LQWHUIDFHWKDWVXSSRUWVQDWXUDOKDQGOLQJRIFRPSOH[GDWDDWLQWHUDFWLYHUDWHV,QWKLV W\SH RI GLVWULEXWHG PXOWLXVHU V\VWHPV DGHTXDWH FRPPXQLFDWLRQ VWUDWHJLHV IRU FRQWLQXRXVV\QFKURQL]DWLRQDQG UHDOWLPH SHUIRUPDQFH DUH UHTXLUHG WKDW DOVR DOORZ WKH LQWHUDFWLRQ ZLWK D VKDUHG JHRPHWULF GDWDEDVH6WXGLHUVWXEH LV GHVFULEHG LQ PRUH GHWDLOLQFKDSWHU
'HVSLWHWKHH[WUDRUGLQDU\UDSLGGHYHORSPHQWRIFRPSXWHUVDQGVRIWZDUHIRUYLUWXDO UHDOLW\ WKH DFFHSWDQFH RI IXOO LPPHUVLYH V\VWHPV IROORZV WKHVH JURZWK UDWHV UDWKHU KDUG7KHDLPWRFUHDWHWKHIHHOLQJRILPPHUVLRQE\SUHVHQWLQJFRQYLQFLQJVWLPXOLWR WKH XVHU LV VWLOO QRW UHDOO\ VDWLVI\LQJ ZH DUH IDU IURP KLJKILGHOLW\ LQ YLUWXDO UHDOLW\
(VSHFLDOO\ VRFLDO DVSHFWV RI IXOO LPPHUVLRQ PD\ SOD\ D VXEVWDQWLDO UROH LQ GHQ\LQJ XVDJHE\VRPHSHRSOH
$XJPHQWHG UHDOLW\ RIIHUV D VPRRWK LPPHUVLRQ E\ OHDYLQJ WKH FRQQHFWLRQ WR UHDO ZRUOG HQYLURQPHQWV EDVLFDOO\ XQWRXFKHG DQG VXSHULPSRVLQJ FRPSXWHU JHQHUDWHG LPDJHU\ RQWR UHDO DUWLIDFWV 6RFLDO FRPPXQLFDWLRQ FKDQQHOV DV QDWXUDO VSHHFK DQG SDUDODQJXDJH DUH QRW EORFNHG EUHDNLQJ GRZQ PHQWDO EDUULHUV RI DSSO\LQJ YLUWXDO UHDOLW\ WHFKQRORJ\ WR D VSHFLILF SUREOHP 1RW XQH[SHFWHGO\ D ORW RI GLIIHUHQW SUREOHPV DULVH LQ WKH LQYHVWLJDWLRQ RI DXJPHQWHG UHDOLW\ OLNH UHJLVWUDWLRQ DQG
&+$37(57:2²3$*(
RFFOXVLRQRIUHDOZRUOGREVWDFOHVEXWPXFKUHVHDUFKLVFRQFHQWUDWLQJRQWKHVHWRSLFV HJ>%DMXUD@DQG>6WDWHD@7UDFNLQJSUHFLVLRQDQGWUDFNLQJUDQJHLQODUJHU HQYLURQPHQWV HVSHFLDOO\ IRU RXWVLGH DSSOLFDWLRQV DUH VWLOO XQVDWLVI\LQJ DQG QHHG IXUWKHULQYHVWLJDWLRQKRZHYHUVRPHVROXWLRQVDUHSURPLVLQJOLNHSUHVHQWHGLQ>$XHU
@DQG>)HLQHU@
2.2.2 Augmented Reality Systems
(YROXWLRQ
7KHHYROXWLRQRIDXJPHQWHGUHDOLW\VWDUWHGLQWKHHDUO\GD\VRIFRPSXWHUJUDSKLFV ZKHQ 6XWKHUODQG SLRQHHUHG UHVHDUFK RQ KHDGPRXQWHG GLVSOD\V >6XWKHUODQG @ +LVZRUNVWLOOLQVSLUHVWKHYLUWXDOUHDOLW\UHVHDUFKFRPPXQLW\RIWRGD\$OWKRXJKRQO\
FDSDEOHRIVLPSOHYHFWRUGUDZLQJVKLVSURWRW\SHKHDGPRXQWHGGLVSOD\ZDVWKHILUVW ELQRFXODUVHHWKURXJKV\VWHPHIIHFWLYHO\WKHILUVWDXJPHQWHGUHDOLW\V\VWHP
)HLQHU HW DO GHVFULEHG LQ >)HLQHU @ DQG >)HLQHU @ D NQRZOHGJH EDVHG DXJPHQWHGUHDOLW\V\VWHP$VDGHPRQVWUDWLRQWKH\FKRRVHWRFRQILJXUHWKHV\VWHP WRVXSSRUWSHRSOHZLWKWKHPDLQWHQDQFHRIODVHUSULQWHUV)LJXUH+RZHYHUDORW RIHIIRUWLVUHTXLUHGWRJHQHUDWHDFFXUDWHPRGHOVDQGH[WUHPHO\SUHFLVHUHJLVWUDWLRQ
%DMXUD HW DO >%DMXUD @ GHVFULEHG D PHGLFDO YLVXDOL]DWLRQ V\VWHP EDVHG RQ DXJPHQWHG UHDOLW\ WHFKQLTXHV $ VHHWKURXJK KHDG PRXQWHG GLVSOD\ +0' DOVR GHYHORSHGDW81&>+ROPJUHQ@DOORZVJHRPHWULFDOO\FRUUHFWVXSHUSRVLWLRQRI XOWUDVRXQGGDWDRIWKHXQERUQRQWRWKHEHOO\RIWKHPRWKHUWREHVRWKHJ\QHFRORJLVW FDQ H[DPLQH WKH SRVLWLRQ RI WKH XQERUQ ZLWKLQ WKH PRWKHU $QRWKHU PHGLFDO DSSOLFDWLRQRI$5KDVEHHQSUHVHQWHGE\6WDWHHW DO >6WDWH E@ IRU XOWUDVRXQG JXLGHGQHHGOHELRSV\RIWKHEUHDVW6KDUPDDQG0ROLQHURV>6KDUPD@SUHVHQWD V\VWHP IRU PHFKDQLFDO DVVHPEO\ JXLGDQFH XVLQJ DQQRWDWLRQV DWWDFKHG WR UHDO ZRUOG VFHQHU\
6FLHQWLILF YLVXDOL]DWLRQ LQ YLUWXDO UHDOLW\ EHFRPHV LQFUHDVLQJO\ D ILHOG RI LQWHUHVW IRU PDQ\UHVHDUFKHUV,QWKHHDUO\LHVWKH*523(SURMHFWJURXSDURXQG)UHG%URRNV SURGXFHGDKDSWLFDUPOLNHGHYLFHDQGDODUJHVWHUHRGLVSOD\IRUWKHYLVXDOL]DWLRQDQG PDQLSXODWLRQRIFKHPLFDOGDWD>%URRNV@7KHLUQDQR0DQLSXODWRU >7D\ORU @ VHHDOVR)LJXUHDOORZVSUHFLVHPDQLSXODWLRQRIDVFDQQLQJWXQQHOLQJPLFURVFRSH DQG ZRUNV DOVR ZLWK IRUFH IHHGEDFN $QRWKHU LPSRUWDQW PLOHVWRQH IRU WKH
7+(3(5621$/,17(5$&7,213$1(/²$7:2+$1'(',17(5)$&()25$8*0(17('5($/,7<
&+$37(57:2²3$*(
FRPELQDWLRQ RI 95 DQG VFLHQWLILF YLVXDOL]DWLRQ ZDV WKH GHYHORSPHQW RI WKH YLUWXDO ZLQGWXQQHODW1$6$$0(6E\6WHYH%U\VRQ8VLQJD%RRPGHYLFHDQGDGDWDJORYH DVLQWHUDFWLRQWRRO>%U\VRQ@VFLHQWLVWVZHUHDEOHWRVHHDQGLQWHUDFWZLWKWUXH VWHUHRVFRSLFLPDJHVRIDIORZILHOGYLVXDOL]DWLRQ$IROORZXSSURMHFWWKHGLVWULEXWHG ZLQG WXQQHO >%U\VRQ @ ZDV GHYHORSHG ZKLFK GLYLGHG FRPSXWDWLRQ LQ D GLVWULEXWHGV\VWHPIRUEHWWHUHIILFLHQF\DQGDOORZHGPXOWLSOHXVHUVWRH[SHULHQFHWKH VLPXODWLRQDWWKHVDPHWLPH&ROODERUDWLRQLQDGLVWULEXWHGYLUWXDOHQYLURQPHQWQRW QHFHVVDULO\ OLPLWHG WR VFLHQWLILF YLVXDOL]DWLRQ KDV EHHQ SURSRVHG E\ )DKOpQ HW DO
>)DKOpQ@
&DWHJRUL]DWLRQ
)URP WKHGHVLJQ SULQFLSOHV SRLQW RI YLHZ WRGD\·V DXJPHQWHG UHDOLW\ V\VWHPV FDQ EH FDWHJRUL]HG LQWR WZR PDMRU JURXSV 'HSHQGLQJ RQ WKH DSSOLHG WHFKQRORJ\ ZH GLIIHUHQWLDWHEHWZHHQRSWLFDOYVYLGHREDVHG$5V\VWHPV(DFKKDVSDUWLFXODUDGYDQWDJHV DQGGLVDGYDQWDJHV%RWKGHVLJQSULQFLSOHVKDYHWKHLUVSHFLILFDSSOLFDWLRQILHOGVZKHUH GHFLVLRQ LV HDV\ WR DFFRPSOLVK +RZHYHU LQ PDQ\ FDVHV RQH KDV WR WUDGH LQ VRPH EHQHILWRIDV\VWHPIRUDGUDZEDFNRIWKHRWKHUWRJDLQPD[LPXPVXLWDELOLW\
)URP WKHGLVSOD\ GHYLFHV SRLQW RI YLHZ ZH FDQ GLIIHUHQWLDWH EHWZHHQKHDGPRXQWHG GLVSOD\V KDQGKHOG GLVSOD\V PRQLWRUV DQG SURMHFWLRQ GHYLFHV 6RPH UHVHDUFKHUV GHILQH DXJPHQWHG UHDOLW\ LQ FRPELQDWLRQ ZLWK KHDGPRXQWHG GLVSOD\V 7KH IROORZLQJ H[DPSOHVKRZVWKDWDGHILQLWLRQIL[HGWRRQHGLVSOD\V\VWHPGRHVQRWFRYHUDOOILHOGV RIWKLVHYROYLQJWHFKQRORJ\7KHJURXSRIKDQGKHOGGLVSOD\EDVHG$5V\VWHPVUHSUHVHQWV DYHU\LQWHUHVWLQJGRPDLQIRUWKHQH[WJHQHUDWLRQRIKDQGKHOGGHYLFHVOLNHSDOPWRS FRPSXWHUV SHUVRQDO GLJLWDO DVVLVWDQWV 3'$V DQG PRELOH FRPPXQLFDWLRQ GHYLFHV )ULW]PDXULFHDQG5HNLPRWRSUHVHQWHGLQ>)ULW]PDXULFH@DQG>5HNLPRWR@
V\VWHPV ZKLFK DXJPHQW UHDO VXUURXQGLQJV ZLWK DGGLWLRQDO LQIRUPDWLRQ EDVHG RQ KDQGKHOGGLVSOD\V
7DEOH VKRZV DQ RYHUYLHZ RI H[LVWLQJ DXJPHQWHG UHDOLW\ V\VWHPV %DVHG RQ WKH DERYHFDWHJRUL]DWLRQV\VWHPVFDQEHLGHQWLILHGZLWKRQHURZFROXPQFRPELQDWLRQ
$GGLWLRQDO WR WUXH DXJPHQWHG V\VWHPV WKH WDEOH DOVR FRQWDLQV 95 V\VWHPV QDPHO\
&$9( DQG WKH5HVSRQVLYH :RUNEHQFK WKDW GR QRW DXJPHQW UHDOLW\ EXW SURYLGH DQ H[SHULHQFHIRUDJURXSRIXVHUV7KLVPXOWLXVHUIHDWXUHDVWKLUGGLPHQVLRQRI7DEOH LVGLVFXVVHGEHORZLQFRPSDULVRQWRPXOWLXVHU$5V\VWHPVOLNH6WXGLHUVWXEH
&+$37(57:2²3$*(
'LVSOD\GHYLFH 'HVLJQSULQFLSOH
YLGHREDVHG$5V\VWHP 2SWLFDOVHHWKURXJK
$5V\VWHP PXOWLXVHU95V\VWHP KHDGPRXQWHG
GLVSOD\ >%DMXUD@
>$XHU@ >)HLQHU@
>6FKPDOVWLHJ@
>)XKUPDQQ@
>6]DODYiULD@
KDQGKHOG
GLVSOD\ >)ULW]PDXULFH@
>5HNLPRWR@
PRQLWRU >3HXFKRW@
>6HUUD@ >&DUOVVRQ@
SURMHFWLRQ >'DUHOO@
>3HWWD@ >.(2@
>6FKPDOVWLHJD@ >.UJHU@
>&UX]1HLUDD@
7DEOH &DWHJRUL]DWLRQRI$XJPHQWHG5HDOLW\V\VWHPV
9LGHREDVHG$5V\VWHPVFRPELQHUHDODQGYLUWXDOLPDJHRIDQDXJPHQWHGHQYLURQPHQW E\XVLQJFDPHUDVIRUWKHXVHU·VYLHZRIWKHUHDOZRUOG)LJXUHIURP>$]XPD@
&RPELQDWLRQRIERWKLPDJHVLVFRPSOHWHGLQWKHODVWVWHSRIWKHUHQGHULQJSLSHOLQHRI WKHXQGHUO\LQJFRPSXWHUV\VWHP7KLVKHOSVWRFRUUHFWWLPLQJODJVJLYHVIXOOFRQWURO RYHUWKHWUDQVSDUHQF\RIDYLUWXDOREMHFWWKHFRQWUDVWUDWLRRIERWKLPDJHVDQGWKH DFTXLUHGYLGHRLPDJHFDQEHXVHGIRUDGGLWLRQDOWUDFNLQJVWUDWHJLHV$OLVWRIYLGHR EDVHG $5 V\VWHPV FDQ EH IRXQG LQ >$]XPD @ >(GZDUGV @ JLYHV D GHVFULSWLRQIRUPRGLI\LQJDQ+0'WRDYLGHRVHHWURXJK+0'
)LJXUH &RQFHSWXDO GLDJUDP RI YLGHREDVHG DXJPHQWHG UHDOLW\ V\VWHPV
>$]XPD@
7+(3(5621$/,17(5$&7,213$1(/²$7:2+$1'(',17(5)$&()25$8*0(17('5($/,7<
&+$37(57:2²3$*(
)LJXUH &RQFHSWXDO GLDJUDP RI VHHWKURXJK DXJPHQWHG UHDOLW\ V\VWHPV
>$]XPD@
2SWLFDO VHHWKURXJK $5 V\VWHPV XWLOL]H VSHFLDO RSWLFV WR FRPELQH WKH LPDJHV RI UHDO VXUURXQGLQJV UHDFKLQJ WKH H\H ZLWK FRPSXWHU JHQHUDWHG RYHUOD\V )LJXUH 7KHVH V\VWHPV WHQG WR EH PXFK VLPSOHU LQ GHVLJQ FRPSDUHG WR YLGHR EDVHG VROXWLRQV
$GYDQWDJHV RI VXFK V\VWHPV DUH VLPSOLFLW\ DQG VDIHW\ LQ FRQFHUQ RI V\VWHP EUHDNGRZQV)XUWKHUPRUHWKHUHDOLPDJHNHHSVWKHRULJLQDOKLJKUHVROXWLRQDQGQR H\HRIIVHWKDVWREHWDNHQFDUHRIDVLQYLGHREDVHGVROXWLRQV8SWRWKHSUHVHQWGD\
DOO RSWLFDO VHHWKURXJK V\VWHPV HPSOR\ VHHWKURXJK +0'V OLNH GHVFULEHG LQ
>6]DODYiUL@>%LOOLQJKXUVW@>)HLQHU@DQG>)HLQHU@WKRXJKWKHVH V\VWHPV GHSHQG PDLQO\ RQ WKH DYDLODELOLW\ RI WKHVH GHYLFHV &XUUHQWO\ 621<·V
*ODVVWURQ >*ODVVWURQ @ VHULHV RIIHUV D JRRG SULFHSHUIRUPDQFH UDWLR IRU VHH WKURXJK+0'DSSOLFDWLRQV6FKPDOVWLHJSUHVHQWVLQ>6FKPDOVWLHJD@WKHLGHDRI DXJPHQWHG95DVDPL[WXUHRIDXJPHQWHGDQGYLUWXDOHQYLURQPHQWV
0XOWLXVHUV\VWHPV
0RVW H[LVWLQJ DXJPHQWHG DSSOLFDWLRQV DUH VLQJOH XVHU VHWXSV RU GR QRW H[SORLW WKH PXOWLXVHUFKDUDFWHURIWKHLUV\VWHPV6RPHRIWKHIHZH[FHSWLRQVDUHGHVFULEHGKHUH LQ GHWDLO DV SUHVHQW ZRUN FRQWULEXWHV WR WKHVH VHWXSV WKH &$9( V\VWHP WKH 5HVSRQVLYH:RUNEHQFKWKH7UDQV9LVLRQVHWXSWKH6KDUHG6SDFHDQG6WXGLHUVWXEH
,Q WKH &$9( V\VWHP >&UX]1HLUD D@ DQG >&UX]1HLUD E@ XVHUV VHH VWHUHRVFRSLF ' VFHQHV ZLWK /&'VKXWWHU JODVVHV RQ ODUJH SURMHFWLRQ ZDOOV VXUURXQGLQJWKHP)LJXUH3RVLWLRQDQGRULHQWDWLRQRIRQHXVHU·VKHDGLVWUDFNHG VRWKDWWKHLPDJHVRQDOOZDOOVFRUUHVSRQGWRWKLVYLHZHU·VSRVLWLRQ7KHYLHZHUVKDYH
&+$37(57:2²3$*(
WKHLPSUHVVLRQWREHVXUURXQGHGE\D'YLUWXDOVFHQH+RZHYHUWKH&$9(LVQRW DQ$5VHWXSLQWKHFRPPRQVHQVHGXHWRLWVVL]HLWFDQLQFRUSRUDWHUHDOPRFNXSV DQGWKXVPL[UHDOLW\DQGYLUWXDOLW\$GLVDGYDQWDJHRIWKLVV\VWHPLVWKDWWKHSUHVHQWHG LPDJHVILWWRWKHKHDGSRVLWLRQRQO\IRURQHYLHZHUQRWLFHDEOHYLVXDODUWLIDFWVH[LVW IRUDOORWKHUYLHZHUV,QWHUDFWLRQZLWKWKHDSSOLFDWLRQFDQWDNHSODFHLQWZRGLIIHUHQW ZD\V )LUVW WKH &$9( V\VWHP KDV LQ PRVW LQVWDOODWLRQV D FRQWURO GHVN ZKHUH DQ RSHUDWRU FDQ FRQWURO DQ\ DSSOLFDWLRQ LQWHUIDFH IURP D FRQVROH ,QVLGH WKH &$9( D ZDQGDVVKRZQLQ)LJXUHLVXVHGWRFRQWUROYLHZSRLQWDQGWRDFFRPSOLVKDSSOLFDWLRQ VSHFLILF FRQWURO WDVNV 7KLV ZDQG LV D KDQGOH ZLWK EXWWRQV DQG LV PDJQHWLFDOO\
WUDFNHGLQSRVLWLRQDQGRULHQWDWLRQ
)LJXUH 7KH&$9(V\VWHP7KHVHWXSILJXUHVKRZVWKHSURMHFWLRQGHYLFHV DQGJLYHVDQLPSUHVVLRQRIWKHQHFHVVDU\HIIRUWIRUWKHLQVWDOODWLRQ RIWKLVV\VWHP
)LJXUH 7KHZDQGXVHGLQPRVW&$9(LQVWDOODWLRQV
7+(3(5621$/,17(5$&7,213$1(/²$7:2+$1'(',17(5)$&()25$8*0(17('5($/,7<
&+$37(57:2²3$*(
7KH5HVSRQVLYH:RUNEHQFK>.UJHU@XVHVRQHGLVSOD\DUHDZKLFKLVEXLOWLQWRD WDEOHWRS$&57SURMHFWRULVSURMHFWLQJWKHFRPSXWHUJHQHUDWHGLPDJHVIURPEHKLQG WKHSURMHFWLRQVFUHHQ/LNHLQWKH&$9(YLHZHUVZHDU/&'VKXWWHUJODVVHVDQGRQO\
WZRXVHUVFDQVHHWKHREMHFWVLQFRUUHFWVWHUHRVFRS\DWDPD[LPXPDVSUHVHQWHGLQ
>$JUDZDOD @ )XUWKHUPRUH D UHODWLYHO\ VWHHS YLHZLQJ DQJOH LV QHFHVVDU\ WR DFKLHYHDJRRG'LPSUHVVLRQLHWKHYLHZHUVKDYHWRVWD\FORVHWRWKHWDEOH9HU\
VLPLODU V\VWHPV KDYH EHHQ SUHVHQWHG RI GLIIHUHQW JURXSV LQ WLJKW FRRSHUDWLRQ ZLWK PDQXIDFWXUHUV RI WKHVH V\VWHPV,PPHUVD'HVN >3\UDPLG @,PPHUVLYH :RUNEHQFK
>)DNH6SDFHD@%DURQ7DEOH>%DUFR@)LJXUHWUDGHPDUNVFRYHUWKHVDPH PHWDSKRUIRUDWDEOHPRXQWHGSURMHFWLRQV\VWHPVZLWKVOLJKWGLIIHUHQFHVLQWLOWLQJRI WKHWDEOHRUWKHPRELOLW\RIWKHZKROHDSSDUDWXV+RZHYHUQRLQWHUDFWLRQGHYLFHKDV EHHQ GLUHFWO\ PDWHG ZLWK WDEOH OLNH GLVSOD\ V\VWHPV PRVW DSSOLFDWLRQV XVH VR IDU D
&\EHU*ORYH >9LUWH[ @ RU 3,1&+ *ORYHV >)DNH6SDFH E@ $ PRGLILHG WUDQVSDUHQWYHUVLRQRIWKH3HUVRQDO,QWHUDFWLRQ3DQHOKDVEHHQVXFFHVVIXOO\LPSOHPHQWHG DQGWHVWHGZLWKDQXUEDQGHVLJQDSSOLFDWLRQDVSUHVHQWHGODWHULQFKDSWHU
)LJXUH 7KH%DURQ7DEOHDWDEOHOLNHGLVSOD\V\VWHP
&+$37(57:2²3$*(
)LJXUH 7KH7UDQV9LVLRQ V\VWHP LQ DFWLRQ ZKLOH WR VFLHQWLVWV LQWHUDFW ZLWK YLUWXDOREMHFWV
&RPPRQWRERWKSUHVHQWHGV\VWHPVLVWKDWLPDJHJHQHUDWLRQLVSHUIRUPHG E\ RQH GLVSOD\ V\VWHP IRU DOO SDUWLFLSDQWV RI WKH H[SHULHQFH 7KLV IDFW OLPLWV WKH UDQJH RI DSSOLFDWLRQV WR D VXEVHW ZKHUH FRUUHFW VWHUHRVFRSLF YLHZ IRU DOO SDUWLFLSDQWV LV QRW FUXFLDO7RDFFRPSOLVKWKLV5HNLPRWRSUHVHQWHGLQ>5HNLPRWR@WKH7UDQV9LVLRQ V\VWHPDPXOWLXVHUVHWXSXVLQJKDQGKHOGGLVSOD\V)LJXUH,QWKLVV\VWHPLPDJH JHQHUDWLRQ PRYHG IURP RQH FRPPRQ GLVSOD\ XQLW WR SHUVRQDO SDOPWRS VFUHHQV KRZHYHU WKH SUHVHQWHG DXJPHQWDWLRQ LV RQO\ D PRQRVFRSLF LPDJH $Q LQWHUHVWLQJ IHDWXUHRIWKLVV\VWHPLVWKDWWKHGLVSOD\GHYLFHLWVHOIUHSUHVHQWVDQLQWHUDFWLRQGHYLFH
&KDSWHUGHVFULEHVWKLVLQWHUDFWLRQPHWDSKRULQPRUHGHWDLO
&ORVHVW WR RXU ZRUN6WXGLHUVWXEH V\VWHP LV WKH SURWRW\SH LPSOHPHQWDWLRQ RI6KDUHG 6SDFH>%LOOLQJKXUVW @ VKRZQ LQ )LJXUH 8VHUV ZHDULQJ KHDGPRXQWHG VHH WKURXJKGLVSOD\VFDQGLVFXVVVKDUHGLQIRUPDWLRQLQWKUHHGLPHQVLRQVIORDWLQJDURXQG WKHPLQVSDFHDQGLQWHUDFWXVLQJJHVWXUHVDQGVSHHFKFRPPDQGV$VWKHIRFXVRIWKLV ZRUNLVRQXELTXLWRXVFRPSXWLQJDQGQRWLQVLWXFRRSHUDWLYHZRUNGLVWULEXWLRQRI GDWD LQIRUPDWLRQ VKDULQJ DQG LQWHUDFWLRQ WHFKQLTXHV IDFH GLIIHUHQW SUREOHPV DV SUHVHQWHGLQRXUZRUN
7+(3(5621$/,17(5$&7,213$1(/²$7:2+$1'(',17(5)$&()25$8*0(17('5($/,7<
&+$37(57:2²3$*(
)LJXUH 7ZRXVHUVFRPPXQLFDWHXVLQJWKH6KDUHG6SDFHV\VWHP
6WXGLHUVWXEH
7KH ,QVWLWXWH RI &RPSXWHU *UDSKLFV RI WKH 9LHQQD 8QLYHUVLW\ RI 7HFKQRORJ\
GHYHORSVLQFRRSHUDWLRQZLWKWKH&RPSXWHU9LVLRQ*URXSRIWKH*UD]8QLYHUVLW\RI 7HFKQRORJ\DPXOWLXVHUDXJPHQWHGUHDOLW\V\VWHPFDOOHG6WXGLHUVWXEH7KHSURMHFWZDV QDPHVHOHFWHGDIWHUWKHSOD\)DXVWE\-RKDQQ:ROIJDQJYRQ*RHWKHLQZKLFKWKH OHDGLQJ FKDUDFWHU XVHV D VWXG\ URRP IRU SHUIRUPLQJ UHVHDUFK DQG SKLORVRSK\ WKH 6WXGLHUVWXEH)LJXUH
)LJXUH 6WXGLHUVWXEH²XQYHLOLQJKLGGHQVHFUHWV
,Q >*HUYDXW] @ >6FKPDOVWLHJ @ >)XKUPDQQ @ DQG PRUH GHWDLOHG LQ
>6]DODYiULD@ZHSURSRVHGDV\VWHPFDSDEOHRIYLVXDOL]DWLRQRIWKUHHGLPHQVLRQDO VFLHQWLILF GDWD IRU PXOWLSOH VLPXOWDQHRXV YLHZHUV ZLWKLQ RQH URRP LQ FRQWUDVW WR RXWVLGH $5 DSSOLFDWLRQV 7KH FKRLFH RI WKLV VHWWLQJ OLPLWV WKH FRPSOH[LW\ RI WKH SUREOHP DV WKH ´UHDO ZRUOGµ LV OLPLWHG WR D URRP ZKLFK LV FRPSOHPHQWHG E\ WKH
&+$37(57:2²3$*(
´YLUWXDO ZRUOGµ (DFK YLHZHU ZHDUV PDJQHWLFDOO\ WUDFNHG VHHWKURXJK +0'V SURYLGLQJ D VWHUHRVFRSLF UHDOWLPH GLVSOD\ DQG FDQ IUHHO\ ZDON DURXQG LQ RUGHU WR REVHUYHWKHDXJPHQWHGHQYLURQPHQWIURPGLIIHUHQWYLHZSRLQWV)LJXUH
)LJXUH 6WXGLHUVWXEHLQDFRQFHSWXDOGUDIW²7KUHHSHRSOHZHDULQJVHHWURXJK JODVVHVDWDPHHWLQJYLHZLQJDYLUWXDOJOREH1RWHWKDWWKHWDEOHLVDQ REMHFWLQWKHUHDOZRUOGWKHJOREHMXVWDQLPDJHRYHUODLGRQWKHUHDO VXUURXQGLQJVE\WKHKHDGVHW
7KHPL[WXUHEHWZHHQUHDODQGYLUWXDOYLVXDOH[SHULHQFHFUHDWHGLQRXUV\VWHPE\VHH WURXJK+0'VLVDNH\IHDWXUHRIRXUV\VWHP7KXVLWLVSRVVLEOHWRPRYHDURXQG IUHHO\ZLWKRXWIHDUWREXPSLQWRREVWDFOHVDVRSSRVHGWRIXOO\LPPHUVLYHGLVSOD\V ZKHUHRQO\YLUWXDOREMHFWVFDQEHSHUFHLYHG7KLVHQDEOHVDZRUNJURXSWRGLVFXVVWKH YLHZHG REMHFW EHFDXVH WKH SDUWLFLSDQWV DUH VHHLQJ RQH DQRWKHU DQG FDQ WKHUHIRUH FRPPXQLFDWHLQWKHXVXDOZD\)LJXUH
)LJXUH 7ZRXVHUVLQYHVWLJDWHWKH5|VVOHUDWWUDFWRULQ6WXGLHUVWXEH
7+(3(5621$/,17(5$&7,213$1(/²$7:2+$1'(',17(5)$&()25$8*0(17('5($/,7<
&+$37(57:2²3$*(
7KHIROORZLQJNH\SURSHUWLHVVXPPDUL]HWKHDWWULEXWHVRIWKHV\VWHP
◊ 9LUWXDOLW\²9LHZLQJDQGH[DPLQLQJRIREMHFWVWKDWDUHQRWDFFHVVLEOHGLUHFWO\RU WKDWGRQRWH[LVWLQWKHUHDOZRUOGFDQEHFDUULHGRXWLQWKLVHQYLURQPHQW
◊ $XJPHQWDWLRQ ² 5HDOZRUOG REMHFWV FDQ EH DXJPHQWHG ZLWK VSDWLDOO\ DOLJQHG LQIRUPDWLRQ7KLVDOORZVVPRRWKH[WHQVLRQRIUHDOREMHFWVZLWKYLUWXDOSURSHUWLHV LQGHVLJQSURFHVVHVOLNHYDULDWLRQVRIQHZSDUWVIRUDQH[LVWLQJV\VWHP
◊ 0XOWLXVHU VXSSRUW ² 0XFK UHVHDUFK KDV EHHQ GHYRWHG WR WKH TXHVWLRQ KRZ FRQYHQWLRQDOVRIWZDUHDQGGHVNWRSFRPSXWHUVFDQEHHQKDQFHGZLWKPHDVXUHVWR VXSSRUWHIIHFWLYHJURXSLQWHUDFWLRQ)RUWXQDWHO\DEHQHILWRIDXJPHQWHGUHDOLW\LV WKDWVRSKLVWLFDWHGJURXSZDUHPHFKDQLVPVDUHQRWUHDOO\QHHGHGWRSHUIRUPUHDO ZRUN1RUPDOKXPDQLQWHUDFWLRQVYHUEDOJHVWXUHVHWFDUHHDVLO\SRVVLEOHLQDQ DXJPHQWHG UHDOLW\ VHWXS DQG WKH\ DUH SUREDEO\ ULFKHU WKDQ DQ\ FRPSXWHU JRYHUQHGLQWHUDFWLRQFDQHYHUEH
◊ ,QGHSHQGHQFH ² 8QOLNH WKH&$9( DQG WKH5HVSRQVLYH :RUNEHQFK FRQWURO LV QRW OLPLWHGWRDJXLGLQJSHUVRQZKLOHRWKHUXVHUVDFWDVSDVVLYHREVHUYHUV(DFKXVHU KDVWKHRSWLRQWRPRYHIUHHO\DQGLQGHSHQGHQWO\RIWKHRWKHUXVHUV,QSDUWLFXODU HDFK XVHU PD\ IUHHO\ FKRRVH D YLHZSRLQW ZLWK VWHUHRVFRS\ IRU FRUUHFW GHSWK SHUFHSWLRQ %XW QRW RQO\ LV REVHUYDWLRQ LQGHSHQGHQW LQWHUDFWLRQ FDQ DOVR EH SHUIRUPHGRQDSHUVRQDOEDVH
◊ 6KDULQJYV,QGLYLGXDOLW\²,QYHVWLJDWHGREMHFWVDUHLQJHQHUDOVKDUHGDPRQJXVHUVLQ WKHVHQVHRIYLVLELOLW\WKLVPHDQVWKDWDOOSDUWLFLSDQWVFDQVHHWKHVDPHFRKHUHQW PRGHOFRQVLVWHQWLQLWVVWDWHRYHUWLPH%\SUHVHQWLQJWKHYLVXDOVHQVDWLRQGLUHFWO\
WRHDFKXVHUZLWKWKHOLJKWZHLJKWVHHWURXJK+0'VWKHGLVSOD\HGGDWDVHWFDQ DOVREHGLIIHUHQWIRUHDFKYLHZHUDVUHTXLUHGE\WKHDSSOLFDWLRQ·VQHHGVDQGWKH LQGLYLGXDO·VFKRLFH3HUVRQDOSUHIHUHQFHVRQGLIIHUHQWOD\HUVRILQIRUPDWLRQFDQ EH VZLWFKHG RQ DQG RII DV NQRZQ IURP &$' SDFNDJHV $ XQLTXH 6($0 PHFKDQLVP >6FKPDOVWLHJ E@ HQDEOHV LQWHUDFWLYH UHYHDOLQJ DQG KLGLQJ RI LQIRUPDWLRQLQGLIIHUHQWOD\HUVLQWKUHHGLPHQVLRQV
◊ ,QWHUDFWLRQDQG,QWHUDFWLYLW\²:LWKWKHVXSSRUWRIDXJPHQWHGWRROVOLNHWKHSURSRVHG 3HUVRQDO ,QWHUDFWLRQ 3DQHO YLVXDOL]HG GDWD FDQ EH H[SORUHG LQWHUDFWLYHO\ &KDQJHV LQKHUHQWLQWKHVFLHQWLILFVLPXODWLRQFDQEHYLHZHGLPPHGLDWHO\
&+$37(57:2²3$*(
)LJXUH 3,3DSSOLFDWLRQLQ6WXGLHUVWXEH
&XUUHQW ZRUN IRFXVHV RQ WKH GHVLJQ RI WKH3HUVRQDO ,QWHUDFWLRQ 3DQHO )LJXUH +RZHYHUWKLVGHVLJQSURFHVVLVWLJKWO\FRQQHFWHGWRWKHGHYHORSPHQWRI6WXGLHUVWXEH DW WKH ,QVWLWXWH RI &RPSXWHU *UDSKLFV RI 9LHQQD 8QLYHUVLW\ RI 7HFKQRORJ\
7KHUHIRUH WKURXJKRXW WKLV ZRUN 6WXGLHUVWXEH ZLOO EH RIWHQ UHIHUHQFHG DV LPSOHPHQWDWLRQSODWIRUPRUWHVWEHGIRU3,3IHDWXUHVDQGDSSOLFDWLRQV
,QWHUDFWLRQLQ$XJPHQWHG5HDOLW\
2.3.1 Interacting with Information
,Q UHDO HQYLURQPHQWV SHRSOH DUH XVHG WR LQWHUDFW ZLWK REMHFWV 2XU HYHU\GD\
H[SHULHQFH LV WR JUDE WKLQJV PDQLSXODWH DQG SXW WKHP DZD\ ,Q WKH ODVW GHFDGHV
´QRQUHDOµHOHPHQWVSRSXODWHRXUHQYLURQPHQWZHLQWHUDFWPRUHDQGPRUHZLWKSXUH LQIRUPDWLRQ &RPSXWHUV DUH FXUUHQWO\ WKH PRVW ZLGHVSUHDG ZD\ WR LQWHUDFW GLUHFWO\
ZLWK LQIRUPDWLRQ 6DGO\ QRW FRPSXWHUV GLG HYROYH WR LQWHUIDFH KXPDQV DV EHVW DV SRVVLEOH
7+(3(5621$/,17(5$&7,213$1(/²$7:2+$1'(',17(5)$&()25$8*0(17('5($/,7<
&+$37(57:2²3$*(
)LJXUH &ROOHFWLRQ RI WUDGLWLRQDO ' LQSXW GHYLFHV PRXVH NH\ERDUG DQG WDEOHW
(QJLQHHUVGHYHORSHGDWWKHFXUUHQWOHYHORIWHFKQRORJ\GHYLFHVDQGPHWDSKRUVWKDW ZHUH HDV\ WR XQGHUVWDQG DQG FKHDS WR LPSOHPHQW LQ PDVVHV )LJXUH 0RVW RI WKHVHGHYLFHVKDGRQHLQFRPPRQ7KH\PRUHRUOHVVPDLQO\VXSSRUWHGWKHWHFKQLFDO QHHGV RI FRPSXWHUV 8VHUV KDG WR DGDSW WR WKH LQSXW GHYLFHV DQG WKH LQWHUDFWLRQ PHWDSKRUV
1HZFRPSXWHUJHQHUDWLRQVLQFRUSRUDWLQJJUDSKLFVKDUGZDUHDOORZHGWKHYLVXDOL]DWLRQ RI WKUHHGLPHQVLRQDO SUREOHPV DW LQWHUDFWLYH UDWHV 7KH LQWURGXFWLRQ RI 6WHUHRJUDSKLFV·V&U\VWDO(\HVJODVVHVLQIRU'RXWSXWRQZRUNVWDWLRQPRQLWRUV DOORZHGVHHLQJWKHVHYLVXDOL]DWLRQVHYHQLQVWHUHRVFRSLF'IRUDEURDGFRPPXQLW\
7KUHHGLPHQVLRQDODSSOLFDWLRQVHPHUJLQJRQWKHVHPDFKLQHVUHTXLUHGDQHZZD\RI LQWHUDFWLRQ ZLWK WKH FRPSXWHU 5HVHDUFKHU ILUVW HPSOR\HG H[LVWLQJ ' KDUGZDUH LQWHUIDFHVWRLPSOHPHQWQHZLQWHUDFWLRQPHWDSKRUV>&KHQ@DQG>1LHOVRQ@
JLYH DQ H[DPSOH RI XVLQJ ' GHYLFHV IRU ' URWDWLRQ DQG PDQLSXODWLRQ +RZHYHU PDSSLQJ'LQSXWWR'DFWLRQVSOD\VHYHQWRGD\DVLJQLILFDQWUROHLQ'HVNWRS95 DSSOLFDWLRQVWKLVW\SHRILQWHUDFWLRQODFNVGLUHFWFRUUHVSRQGHQFHRIUHDODQGYLUWXDO VSDFH
5HPRYLQJ WKH QHFHVVLW\ RI PHQWDO DEVWUDFWLRQ RI VSDFH D QHZ JHQHUDWLRQ RI LQSXW GHYLFHV ZDV FUHDWHG 7KHVH ZHUH PDLQO\ FRQVWUXFWHG DURXQG VSHFLILF WDVNV OLNH ' PRGHOLQJVFHQHFRPSRVLWLRQOD\RXWSODQLQJHWF%HFDXVHRIWKLVDZLGHYDULHW\RI FRPSOHWHO\ GLIIHUHQW ' LQSXW GHYLFHV KDYH EHHQ LQWURGXFHG &KULV +DQG JLYHV LQ
>+DQG@DQG>+DQG@DJRRGRYHUYLHZRQWKLVGHYHORSPHQWRI'LQSXW GHYLFHVIURPWKHHDUO\EHJLQQLQJVLQPLG·V
&+$37(57:2²3$*(
:KHUHDVFRQYHQWLRQDOGHVNWRSLQSXWGHYLFHVHJNH\ERDUGDQGPRXVHKDYHUHDFKHG D KLJK GHJUHH RI VSHFLDOL]DWLRQ GXULQJ D V\QWKHWLF HYROXWLRQDU\ SURFHVV ' LQSXW GHYLFHV KDYH VWLOO VLJQLILFDQW GLVDGYDQWDJHV +LJK DFFXUDF\ PHFKDQLFDO GHYLFHV DUH VRPHZKDW EXON\ RU ERXQG WR FHUWDLQ DSSOLFDWLRQV DQG GR QRW VXSSRUW JHQHUDOL]HG LQWHUDFWLRQWHFKQLTXHV>7D\ORU@)LJXUH6L[GHJUHHRIIUHHGRP'2)PLFH OLNHWKH6SDFHEDOO)/;>/DEWHF@LQ)LJXUHDQGGDWDJORYHV>97,@
>1LQWHQGR@LQ)LJXUHH[WHQGWKHSRVVLEOHVHWRILQWHUDFWLRQVE\DGGLQJQHDUO\
XQFRQVWUDLQHG WKUHH GLPHQVLRQDO PRYHPHQW DQG FDSWXULQJ GR]HQV RI SRVLWLRQ DQG RULHQWDWLRQ GDWD +RZHYHU WKHVH GHYLFHV VXIIHU HLWKHU IURP XQVDWLVI\LQJ ORZ LQWHUDFWLRQ EDQGZLGWK RU DQ RYHUORDGHG PHWDSKRU OLNH FRPSOH[ JHVWXUH ODQJXDJHV
>6WXUPDQ@6LQFHQRGLUHFWWDFWLOHIHHGEDFNRIWKHYLUWXDOREMHFWVLVSURYLGHGLQ PRVWV\VWHPVLQH[SHULHQFHGXVHUVIHHOGLVRULHQWHGDQGILQGLWUDWKHUGLIILFXOWWRZRUN ZLWK IO\LQJ EXWWRQV PHQXV ´' ZLGJHWVµ >&RQQHU @ DQG VLPLODU PHWDSKRUV IORDWLQJDURXQGWKHP)LQDOO\3LHUFHHWDOUHSRUWLQ>3LHUFH@RQLPDJHSODQH LQWHUDFWLRQWHFKQLTXHVIRULPPHUVLYHYLUWXDOHQYLURQPHQWVDQGOHWXVHUVLQWHUDFWZLWK ' SURMHFWLRQV RI ' REMHFWV DQ DSSURDFK WKDW VXSSRUWV PDQLSXODWLRQ RI GLVWDQW REMHFWVLQ9(
)LJXUH 7KHQDQR0DQLSXODWRU
7+(3(5621$/,17(5$&7,213$1(/²$7:2+$1'(',17(5)$&()25$8*0(17('5($/,7<
&+$37(57:2²3$*(
)LJXUH 6SDFHEDOO)/;GHJUHHRIIUHHGRPLQSXWGHYLFH
)LJXUH 97,&\EHU*ORYHDQG0DWWHO1LQWHQGR3RZHUJORYH
7RRYHUFRPHPDQ\GUDZEDFNVRIRWKHUGHYLFHV:ORNDSUHVHQWHGLQ>:ORND@
WKH9LUWXDO7ULFRUGHU7KLVGHYLFHUHVHPEOHVDMR\VWLFNOLNHKDQGOHZLWKEXWWRQVRQWKH WRS7KHWULFRUGHULVUHSUHVHQWHGLQWKH95V\VWHPZLWKDYLVXDOFORQHPDNLQJLWHDV\
WRLGHQWLI\$GGLWLRQDOO\WKHYLVXDOVLPLODULW\KHOSVWRVXVSHQGGLVEHOLHI$SSOLFDWLRQ FRQWUROLVHDV\WRIXOILOZLWKDPHQXDSSHDULQJDWWKHWLSRIWKHWULFRUGHU7KLVPHQXLV DWWDFKHG WR WKH LQSXW GHYLFH VR WDUJHW DFTXLVLWLRQ LV RQO\ UHODWLYH HQKDQFLQJ WKH SHUIRUPDQFHRILQWHUDFWLRQ)O\LQJPHQXVDUHERXQGWRDSK\VLFDOSURSPDNLQJWKHP HDV\WRXVH
$VHOHFWLRQSURFHVVVKRXOGQRZIROORZWKLVHDUO\SKDVHRIHYROXWLRQZKLFKJHQHUDWHG DQ HQRUPRXV QXPEHU RI GLIIHUHQW DSSURDFKHV +LJKO\ VSHFLDOL]HG WRROV SHUIRUP SRVVLEO\PXFKEHWWHULQFHUWDLQDSSOLFDWLRQVFHQDULRVEXWIDLOLQPRUHJHQHUDOWDVNV
&+$37(57:2²3$*(
+RZHYHUWKLVPLJKWNHHSWKHVHPHWDSKRUVDOLYHLQWKHLUDSSOLFDWLRQILHOGLWKLQGHUV WKHLUJHQHUDODFFHSWDQFH*HQHUDOL]DWLRQZDVRIWHQGLVDSSURYHGDWWKHEHJLQQLQJRI WKH HYROXWLRQ SURFHVV RI LQSXW GHYLFHV >%X[WRQ @ EORFNLQJ SRVVLEOH QHZ DQG GLIIHUHQWJDGJHWV7RGD\ZHFDQORRNEDFNDQGWHOOWKDWPDQ\GHYHORSPHQWVGLGQRW VXUYLYHG WKH VWUXJJOH IRU DFFHSWDQFH $ IHZ KDUGZDUH YHQGRUV OLNH 3ROKHPXV /RJLWHFKDQG93/OLFHQVHGWKHUDWKHUVXFFHVVIXOGHYLFHVRWKHUUHPDLQHGSURWRW\SHV RUFXVWRPVROXWLRQVIRUHYHU:HFDQOHDUQIURPWKHLUGHVLJQIODZVH[WUDFWWKHJRRG LGHDVDQGGHGXFWJHQHUDOL]HGLQWHUIDFHVOLNHWKHSURSRVHG3HUVRQDO,QWHUDFWLRQ3DQHO
2.3.2 Interaction Metaphors for Augmented Reality
,QWHUDFWLRQPHWDSKRUVIRU$XJPHQWHG5HDOLW\V\VWHPVZHQWWKURXJKDVLPLODUSURFHVV RIGHYHORSPHQW+RZHYHUWKHRYHUDOOQXPEHURILQWHUIDFHVLVPXFKVPDOOHUWKDQLQ FRPPRQ95DSSOLFDWLRQVGXHWRWKHGLYHUVLW\RI$5V\VWHPVLQJHQHUDOLQWHUIDFHVDUH GLVWULEXWHGDORQJDEURDGHUVSHFWUXP
0DQ\HDUO\$5DSSOLFDWLRQVIRFXVVHGRQWKHDXJPHQWDWLRQRIUHDOHQYLURQPHQWVZLWK DGGLWLRQDOSDVVLYH GDWD WKDW FRXOG EH EURZVHG LQ DQ LQWHUDFWLYH PDQQHU $PVHOHP
>$PVHOHP @ )LW]PDXULFH >)LW]PDXULFH @ DQG 5HNLPRWR >5HNLPRWR @ VKRZRQHKDQGHGLQWHUDFWLRQPHWDSKRUVXVLQJDKDQGKHOGGLVSOD\++'GHYLFHIRU WKHH[SORUDWLRQRIVSDWLDOO\DOLJQHGGDWDEDVHV 7KH PDJQHWLF WUDFNHG GLVSOD\ FDQ EH FDUULHGDURXQGVKRZLQJVLWXDWHGJUDSKLFVDXJPHQWHGRYHUWKHUHDOHQYLURQPHQW8VHU LQWHUDFWLRQFRYHUVPDLQO\DEURZVLQJWDVNRIODUJHVSDWLDOO\GLVWULEXWHGPXOWLPHGLD GDWDEDVH EXW LPSOHPHQWDWLRQV DOVR LQFOXGH IHDWXUHV OLNH ]RRPLQJ OD\HUV DQG EURZVLQJRIUHPRWHHQYLURQPHQWV
)HLQHU·VZRUNJRHVDVWHSIXUWKHULQLQWHUDFWLRQZLWKERWKWKHUHDODQGWKHYLUWXDOSDUW RIDQDXJPHQWHGHQYLURQPHQW,Q>)HLQHU@DQG>)HLQHU@KHGHVFULEHVDQ
$5 V\VWHP IRU WKH PDLQWHQDQFH RI ODVHU SULQWHUV ,Q WKLV V\VWHP DGGLWLRQDO LQIRUPDWLRQLVRYHUODLGRYHUWKHUHDOSDUWIRUVXSSRUWLQJWKHPDLQWHQDQFHWDVN)LJXUH :LWK WKH $5&V\VWHP VKRZQ LQ )LJXUH KH H[WHQGV WKLV DSSURDFK IRU D FRQVWUXFWLRQ WDVN LQ >:HEVWHU @ 8VHUV DUH JLYHQ WR D EDUFRGH UHDGHU IRU LGHQWLI\LQJ SDUWV XVHG WR FRQVWUXFW WKH DOXPLQXP FRQVWUXFWLRQ IUDPH 8VLQJ DXJPHQWHGRYHUOD\VWKHV\VWHPVKRZV'SRVLWLRQRIWKHQH[WSDUWWREHLQVWDOOHG 7KLV DSSURDFK GRHV DOVR QRW DOWHU WKH VWRUHG LQIRUPDWLRQ KRZHYHU LW PRGLILHV WKH
7+(3(5621$/,17(5$&7,213$1(/²$7:2+$1'(',17(5)$&()25$8*0(17('5($/,7<
&+$37(57:2²3$*(
GLVSOD\HG LQIRUPDWLRQ VWHSE\VWHS 7KH7RXULQJ 0DFKLQH DSSURDFK LQ >)HLQHU @ H[WHQGV WKHVH GHYHORSPHQW IRU RXWVLGH VFHQDULRV FRPELQLQJ DQ +0' IRU VLWXDWHG LQIRUPDWLRQ DQG D KDQGKHOG SHQ EDVHG SHUVRQDO GLJLWDO DVVLVWDQW 3'$ IRU LQWHUDFWLRQDQGDGGLWLRQDOLQIRUPDWLRQUHWULHYDO)LJXUH,QIRUPDWLRQGLVSOD\HGLQ WKH+0'FDQEHFRQILJXUHGZLWKWKH3'$7KHXVHULVDOVRYLVXDOO\QRWLILHGLQWKH +0'ZKHQHYHUDGGLWLRQDOGDWDLVDYDLODEOHDQGFRXOGEHUHWULHYHGRQWKH3'$
)LJXUH .$50$WKHNQRZOHGJHEDVHG$5V\VWHPSURYLGHVVLWXDWHGYLVXDO KHOSLQIRUPDWLRQ
)LJXUH $5& ² &RQVWUXFWLQJ DOXPLQXP IUDPH V\VWHPV XVLQJ DXJPHQWHG KHOSLQIRUPDWLRQ$PDJQHWLFDOO\WUDFNHGEDUFRGHUHDGHUSURYLGHV LQWHUDFWLRQZLWKWKHV\VWHP
)LJXUH 0$56 ² 7KH 7RXULQJ 0DFKLQH DOORZV EURZVLQJ VLWXDWHG LQIRUPDWLRQLQRXWVLGHVFHQDULRVHJWKH&ROXPELD8QLYHUVLW\
&+$37(57:2²3$*(
)LJXUH 7UDQV9LVLRQ ² %\ SXVKLQJ WKH REMHFW VHOHFWRU EXWWRQ D EHDP LV HPLWWHG SHUSHQGLFXODU WR WKH VXUIDFH RI WKH KDQGKHOG GLVSOD\
GHYLFH 7KH ILUVW REMHFW KLW E\ WKH EHDP LV VHOHFWHG IRU IXUWKHU PDQLSXDOWLRQ
)LQDOO\ WKH HDUOLHU SUHVHQWHG7UDQV9LVLRQ >5HNLPRWR @ V\VWHP LQ )LJXUH HPSOR\V DOVR LQWHUDFWLYH PDQLSXODWLRQ RI ' GDWD LQ DQ $5V\VWHP 7KH GLVSOD\
GHYLFHLWVHOIUHSUHVHQWVDQLQWHUDFWLRQGHYLFHWKDWLVFRQWUROOHGZLWKWZREXWWRQV2QH LVXVHGIRUREMHFWVHOHFWLRQDQGPDQLSXODWLRQ%\SXVKLQJWKLVEXWWRQDYLUWXDOEHDP LVHPLWWHGSHUSHQGLFXODUWRWKHVXUIDFHRIWKHGLVSOD\GHYLFH7KHILUVWREMHFWKLWE\
WKH EHDP LV VHOHFWHG DQG FDQ EH PDQLSXODWHG $ VLPLODU DSSURDFK LV XVHG LQ
>)ULW]PDXULFH @ WR PDNH REMHFW VHOHFWLRQV %\ SXVKLQJ WKH RWKHU ² PHQX ² EXWWRQ D SLH PHQX DSSHDUV RQ WKH GLVSOD\ PHQX LWHPV DUH VHOHFWHG E\ WLOWLQJ WKH GLVSOD\DQGUHOHDVLQJWKHEXWWRQ7KLVXQLTXHWHFKQLTXHILWVYHU\ZHOOWRWKLVW\SHRI GHYLFH KRZHYHU WKH RWKHU KDQG RI WKH XVHU LV QRW LQYROYHG LQ LQWHUDFWLRQ 7ZR KDQGHGLQWHUDFWLRQLQWKLVFRQWH[WFRXOGLPSURYHWKHTXDOLW\RILQWHUDFWLRQ7RVKRZ KRZ WKLV FDQ EH SURYHQ WKH IROORZLQJ FKDSWHU VXPPDUL]HV H[LVWLQJ ZRUN RQ ELPDQXDOLQWHUDFWLRQZLWKQRQUHDOFRQWHQW
7+(3(5621$/,17(5$&7,213$1(/²$7:2+$1'(',17(5)$&()25$8*0(17('5($/,7<
&+$37(57:2²3$*(
%LPDQXDO,QWHUDFWLRQ
2.4.1 Psychology
7ZRKDQGHGLQSXWKDVRIWHQEHHQYLHZHGDVDWHFKQLTXHWRLPSURYHWKHHIILFLHQF\RI KXPDQFRPSXWHU LQWHUDFWLRQ E\ HQDEOLQJ WKH XVHU WR SHUIRUP WZR VXEWDVNV LQ SDUDOOHO>%X[WRQ@UDWKHUWKDQVHTXHQWLDOO\VHOHFWHGPRGHV:KHQLQWHUDFWLQJLQ WKUHHGLPHQVLRQV+LQFNOH\IRXQGWKDWXVLQJWZRKDQGVQRWRQO\LPSURYHVHIILFLHQF\
EXWFDQDOVRKHOSWRPDNHVSDWLDOLQSXWFRPSUHKHQVLEOHWRWKHXVHU>+LQFNOH\D@
(QDEOLQJ WKH XVH RI ERWK KDQGV FDQ DOORZ XVHUV WR JURXQG WKHPVHOYHV LQ WKH LQWHUDFWLRQVSDFHLQHVVHQFHWKHXVHU·VRZQERG\EHFRPHVDVSDWLDOUHIHUHQFH0LQH H[SORLWV WKLV DELOLW\ FDOOHG SURSULRFHSWLRQ WR HQKDQFH WKH SUHFLVLRQ RI UHPRWH PDQLSXODWLRQ RI REMHFWV ZKLFK DUH RXW RI UHDFK RI WKH XVHU >0LQH @ 6DFKV REVHUYHG DQ HYHQ PRUH LQWHUHVWLQJ SKHQRPHQRQ ´WKH VLPXOWDQHRXV XVH RI WZR VSDWLDO LQSXW VHQVRUV WDNHV DGYDQWDJH RI SHRSOH·V LQQDWH DELOLW\ NQRZLQJ SUHFLVHO\
ZKHUHWKHLUKDQGVDUHUHODWLYHWRHDFKRWKHUµ>6DFKV@7KLVDELOLW\DULVHVIURP WKHNLQDHVWKHWLFIHHGEDFN>+DQG@WKDWDOORZVXVWRNQRZWKHSRVLWLRQRIRXUOLPEV UHODWLYHWRRXUERG\+LQFNOH\3DXVFKDQG*REOHGRFXPHQWHGLQ>+LQFNOH\E@
DQG>*REOH@WKHLUREVHUYDWLRQZLWKVHYHUDOKXQGUHGWHVWXVHUVRIDWZRKDQGHG VSDWLDOLQWHUIDFHIRUQHXURVXUJLFDOYLVXDOL]DWLRQ7KLVLQIRUPDOVWXG\UHDIILUPHGDQG VWUHQJWKHQHG 6DFKV·V REVHUYDWLRQ PRVW WHVW XVHUV FDQ RSHUDWH WKLV LQWHUIDFH HIIHFWLYHO\ZLWKLQWKHLUILUVWPLQXWHRIXVHZLWKOLWWOHRUQRWUDLQLQJDWDOO)LQGLQJVRI
%X[WRQ>%X[WRQ@DQG.DEEDVK>.DEEDVK@FRXOGEHUHLQIRUFHGWKDWXVHUV DUH DEOH WR WUDQVIRUP HYHU\GD\ VNLOOV IRU PDQLSXODWLQJ WRROV ZLWK WZR KDQGV WR KXPDQFRPSXWHU LQWHUIDFHV 1HZHU VWXGLHV LQ >6KDZ @ >+LQFNOH\ D@ DQG
>=HOH]QLN @ DJDLQ YDOLGDWHG WKHVH UHVXOWV ZKLOH SHUIRUPLQJ GLIIHUHQW WHVWV LQ YDU\LQJ VRIWZDUH HQYLURQPHQWV DQG DSSOLFDWLRQ VFHQDULRV ,Q >6KDZ @ 6KDZ DQDO\]HGSDLQDQGIDWLJXHLQD'HVNWRS95HQYLURQPHQW5HVXOWVVKRZWKDWDZHOO GHVLJQHGWZRKDQGHGPHWDSKRUFDQQRWRQO\EHPRUHHIILFLHQWLQIXOILOOLQJ'WDVNV EXWDOVR²EHLQJOHVVIDWLJXLQJWKDQDRQHKDQGHGLQWHUIDFH²EHDPRUHFRPIRUWDEOH KXPDQFRPSXWHULQWHUIDFHUHSODFLQJRWKHUPHWDSKRUV
0XFKRIWKHDERYHFLWHGZRUNURRWVEDFNLQ*XLDUG·VHDUO\DQDO\VLVRIKXPDQVNLOOHG ELPDQXDODFWLRQ>*XLDUG@+HSURYLGHVDQLQVLJKWIXOWKHRUHWLFDOIUDPHZRUNIRU K\SRWKHVL]LQJ ZKLFK FODVVHV RI WZRKDQGHG LQWHUIDFHV PLJKW LPSURYH SHUIRUPDQFH
&+$37(57:2²3$*(
ZLWKRXW LQGXFLQJ DGGLWLRQDO FRJQLWLYH ORDG *XLDUG KDV SURSRVHG WKH IROORZLQJ SULQFLSOHVEDVHGRQKLVREVHUYDWLRQRIULJKWKDQGHGVXEMHFWV
◊ ´0RWLRQRIWKHULJKWKDQGW\SLFDOO\ILQGVLWVVSDWLDOUHIHUHQFHVLQWKHUHVXOWVRI PRWLRQRIWKHOHIWKDQGµ)RUH[DPSOHZKHQZULWLQJWKHOHIWKDQGFRQWUROVWKH SRVLWLRQ DQG RULHQWDWLRQ RI WKH SDJH ZKLOH WKH ULJKW KDQG SHUIRUPV WKH DFWXDO ZULWLQJE\PRYLQJWKHSHQUHODWLYHWRWKHOHIWKDQG
◊ 7KH ULJKW DQG OHIW KDQGV DUH LQYROYHG LQ DV\PPHWULF WHPSRUDOVSDWLDO VFDOHV RI PRWLRQ ,Q WKH ZULWLQJ WDVN IRU H[DPSOH WKH PRYHPHQWV RI WKH OHIW KDQG DGMXVWLQJ WKH SDJH DUH ORZ LQ WHPSRUDO DQG VSDWLDO IUHTXHQF\ FRPSDUHG WR WKH KLJKIUHTXHQF\GHWDLOHGZRUNGRQHE\WKHULJKWKDQG
◊ ´7KH FRQWULEXWLRQ RI WKH OHIW KDQG WR JOREDO PDQXDO SHUIRUPDQFH VWDUWV HDUOLHU WKDQWKDWRIWKHULJKWµ7KHOHIWKDQGILUVWSRVLWLRQVWKHSDSHUWKHQWKHULJKWKDQG EHJLQVWRZULWH
7KURXJKRXWWKLVWKHVLVWKHSKUDVHVGRPLQDQWKDQGDQGQRQGRPLQDQWKDQGZLOOEHXVHG IURPQRZRQWRGHVFULEHULJKWDQGOHIWKDQGIRUULJKWKDQGHGVXEMHFWVDQGVLPLODUOHIW DQGULJKWKDQGIRUOHIWKDQGHGVXEMHFWVDVDFFHSWHGLQKXPDQFRPSXWHUOLWHUDWXUH
*XLDUG·V FRQFOXVLRQV LPSO\ WKDW ELPDQXDO LQWHUDFWLRQ FDQ ULVH RYHUDOO SHUIRUPDQFH HVSHFLDOO\ LQ FDVHV ZKHUH DV\PPHWULF GLYLVLRQ RI ODERU LV DSSOLHG WR WKH KDQGV 1HYHUWKHOHVVWKHDSSOLFDWLRQRIDELPDQXDOLQWHUIDFHIRUDVSHFLILFWDVNQHHGVFDUHIXO DQDO\VLVVLQFHWKHHQULFKHGLQWHUIDFHPD\SRVVLEO\GHJUDGHWKHTXDOLW\RILQWHUDFWLRQ LQVRPHFDVHVOLNHLQ>.DEEDVK@ZKHUHWKHWZRKDQGHGPHWDSKRUFRPSOLHGWR DERYHJXLGHOLQHVEXWLQVRPHFDVHVLQGXFHGDQDGGLWLRQDOFRJQLWLYHORDG
2.4.2 Bimanual Interfaces for 3D Interaction
/RQJEHIRUHWKHQHFHVVDU\FRPSXWHUKDUGZDUHZDVUHDOO\DYDLODEOHWKHYLVLRQDU\LGHD RIDPRELOHSHQEDVHGFRPSXWHUZDVSXEOLVKHGLQWZRSDSHUV$JURXSRIUHVHDUFKHUV DW WKH 8QLYHUVLW\ RI ,OOLQRLV GHVFULEHG LQ >0HO @ WKHLU FRQFHSW RI7DEOHW ² WKH SHUVRQDO FRPSXWHU LQ WKH \HDU ,W LV D QRWHERRN VL]HG /&' GLVSOD\ ZLWK SHQ EDVHG LQSXW LQFRUSRUDWLQJ ZLUHOHVV FRPPXQLFDWLRQ RYHU LQIUDUHG DQG UDGLR ZDYH FRQQHFWLRQV 7KH DXWKRUV SUHGLFW WKDW E\ WKH \HDU WKLV GHYLFH FRXOG EH D FRPSDQLRQWKURXJKRXWRXUHYHU\GD\OLIH
7+(3(5621$/,17(5$&7,213$1(/²$7:2+$1'(',17(5)$&()25$8*0(17('5($/,7<
&+$37(57:2²3$*(
0DUN:HLVHUVKRZVLQ>:HLVHU@VRPH\HDUVODWHUSURWRW\SHVRIDVLPLODUFRQFHSW GHYHORSHGDW;HUR[3DOR$OWR5HVHDUFK&HQWHU$OWKRXJKWKHSUHGLFWLRQGLGQRWKROG LQHYHU\GHWDLOPRVWRILWZDVUHDOL]HGZKHQORRNLQJDWFRPPHUFLDOO\DYDLODEOH3HUVRQDO 'LJLWDO $VVLVWDQWV 3'$V OLNH WKH HDUOLHU $SSOH·V 1HZWRQ 0HVVDJH 3DG RU &RP·V 3DOP9,,)LJXUH
:KHUHLVWKHFRQQHFWLRQWRDXJPHQWHGUHDOLW\RQHPLJKWDVN"7KHVHHDUO\YLVLRQVRI WZRKDQGHGGHYLFHVLQIOXHQFHGWKHGHYHORSPHQWRIPDQ\IROORZLQJLQWHUIDFHV8VLQJ
$5WHFKQRORJ\PDQ\DVSHFWVRIWKHLGHDVWKDWZHUHVHWDWWKDWWLPHFRXOGEHDFKLHYHG LQ WKH ODVW GHFDGH %XW UHVHDUFK HYHQ H[FHHGHG DOO H[SHFWDWLRQV OLNH WKH IROORZLQJ H[DPSOHVDQGFXUUHQWZRUNVKRZ
6DFKVHWDOFDUULHGRXWDQHDUO\H[DPSOHRIELPDQXDO'LQWHUDFWLRQUHVHDUFKLQQRQ LPPHUVLYHGHVNWRSDSSOLFDWLRQV7KH\VKRZLQWKHLUSDSHU'UDZ>6DFKV@KRZ LQVXIILFLHQWVRSKLVWLFDWHG&$'WRROVVHUYHVKDSHGHVLJQ%HLQJXQDEOHWRURXJKRXW LQLWLDOLGHDVGLUHFWO\ZLWKWKHVHLQWHUDFWLRQWHFKQLTXHVGHVLJQHUVSUHIHUWRXVHSHQDQG SDSHULQVWHDG$VXIILFLHQWO\LQWXLWLYHDQGHDV\WRXVHDSSURDFKRI´GHVLJQGLUHFWO\LQ 'µ XVLQJ WZR KDQGKHOG VL[GHJUHHRIIUHHGRP VHQVRUV LQ IRUP RI D VW\OXV DQG SDOHWWHKDVEHHQSURSRVHG)LJXUH7KHSDOHWWHLVXVHGWRGHILQHDUHIHUHQFHIUDPH WRZKLFKREMHFWVEHLQJGUDZQDUHDWWDFKHGJLYLQJWKHSRVVLELOLW\RIPRYLQJWKHREMHFW LQVWDQWO\LQWRWKHULJKWDQJOHIRUYLHZLQJRQWKHPRQLWRU:KLOHJLYLQJWKHSDOHWWHD VHFRQGDU\SUREOHPWKHSHQLVHPSOR\HGWRGUDZDQGHGLWIUHHIRUPFXUYHVGLUHFWO\LQ '+RZHYHUWKHV\VWHPZDVQRWDQLPPHUVLYHDQGKHDGWUDFNHGDSSOLFDWLRQLWZDV VXLWDEOHIRU&$'VKDSHGHVLJQ
)LJXUH 6FUDWFKSDGVDQG3'$7KHOHIWSLFWXUHVKRZVDFRQFHSWSURWRW\SH PRGHOIURP;HUR[3$5&LQ7KHULJKWSLFWXUHVKRZV&RP·V 3DOP 9,, DYDLODEOH IRU 86' LQ PDMRU FRPSXWHU VWRUHV LQ
&+$37(57:2²3$*(
)LJXUH 'HVLJQHUVNHWFKLQJDXWRPRELOHSDUWVZLWKWKH'UDZLQWHUIDFHDQ HDUO\ H[DPSOH RI WZRKDQGHG 'LQWHUDFWLRQ GHYLFH IRU GHVNWRS DSSOLFDWLRQV
*REOHDQGDJURXSRIUHVHDUFKHUVDWWKH8QLYHUVLW\RI9LUJLQLDGHYHORSHG D KLJKO\
VSHFLDOL]HGWZRKDQGHGLQWHUIDFHIRUQHXURVXUJLFDOSODQQLQJ>*REOH@7KHVHWXS FRQVLVWHGRIPDJQHWLFDOO\WUDFNHGSURSVLQIRUPRIDGROO·VKHDGIRUKHDGYLHZLQJD FURVVVHFWLRQLQJSURSWRDGMXVWFXWWLQJSODQHVDQGDWUDMHFWRU\VHOHFWLRQSURSWRLQGLFDWHWKH GHVLUHGDSSURDFKWRDVXUJLFDOWDUJHWLQWKHYLUWXDOKHDG)LJXUH7KHSUHVHQWHG 'LPDJHRIWKHSDWLHQW·VKHDGRQWKHPRQLWRUFRXOGEHURWDWHGE\URWDWLQJWKHGROO·V KHDG KHOG LQ WKH QRQGRPLQDQW KDQG 8VLQJ WKH GRPLQDQW KDQG WKH VXUJHRQ FDQ GHILQH HLWKHU FXWWLQJ SODQHV IRU WKH YLVXDOL]DWLRQ RI WKH ' GDWD RU GHWHUPLQH WUDMHFWRULHVIRUDWDUJHW,QFRPSOLDQFHZLWK*XLDUG·VREVHUYDWLRQVWKHQRQGRPLQDQW KDQG IXOILOOV WKH URXJK ORZ IUHTXHQF\ WDVN RI SRVLWLRQLQJ ILUVW WKHQ WKH GRPLQDQW KDQGDFFRPSOLVKHVWKHILQHJUDLQHGKLJKIUHTXHQF\WDVNV$PDMRUUHVXOWRIWKLVZRUN ZDVDOVRWKHLQWURGXFWLRQRI LQVWUXPHQWHG SURSV VWDQGLQV DV D JHQHUDO PHWDSKRU IRU LQWHUDFWLRQ GHYLFHV WKDW FRUUHVSRQG LQ SK\VLFDO SURSHUWLHV ZLWK WKH YLUWXDO FRXQWHUSDUWIRUHDVLHUDFFHSWDQFHDQGLPSURYHGSHUIRUPDQFH
7+(3(5621$/,17(5$&7,213$1(/²$7:2+$1'(',17(5)$&()25$8*0(17('5($/,7<
&+$37(57:2²3$*(
)LJXUH 1HXURVXUJLFDO SODQQLQJ ZLWK SURSV 8VLQJ WKH GROO·V KHDG DQG WKH SODQH D FXWWLQJ SODQH FDQ EH GHILQHG XSSHU URZ 8VLQJ WKH WUDMHFWRU\WRRODWUDMHFWRU\LVVHWLQWKHORZHUURZ
%LOOLQJKXUVW SUHVHQWV LQ >%LOOLQJKXUVW @ WKH' 3DOHWWH D YLUWXDOUHDOLW\ FRQWHQW FUHDWLRQWRRO7KHSK\VLFDOLQWHUIDFHLVVLPLODUWRWKH'UDZLQWHUIDFHKRZHYHUWKH SDOHWWHLVDSUHVVXUHVHQVLWLYH:DFRPWDEOHWDQGWKHXVHUZHDUV&U\VWDO(\HVVKXWWHU JODVVHV IRU VWHUHRVFRSLF RXWSXW RQ D PRQLWRU &RPPDQGLQJ LV VXSSRUWHG E\ YRLFH LQSXWIRUVLQJOHVSRNHQSKUDVHV
6KDZ GHVFULEHV LQ >6KDZ @ DQ LQWHUIDFH XVLQJ WZR PRGLILHG PDJQHWLF WUDFNLQJ VHQVRUV &RQWUDU\ WR WKH DERYH SUHVHQWHG DSSURDFKHV 6KDZ GRHV QRW XVH SURSV DV LQWHUDFWLRQGHYLFHV,QIDFWWKHSUHVHQWHG7+5('V\VWHPXVHVWZRFXVWRPPDGH EXWWRQ 3ROKHPXV UHFHLYHUV FDOOHGEDWV 7KH XVHU LQWHUDFWV XVLQJ WKH EDWV LQ ERWK KDQGVVZLWFKLQJEHWZHHQGLIIHUHQWFRQWH[WVXVLQJWKHEXWWRQV7KHGLUHFWPDQQHURI REMHFWPDQLSXODWLRQLPSURYHVXVHUSHUIRUPDQFHHVSHFLDOO\ZKHQSHUIRUPLQJFRPSOH[
'RSHUDWLRQV
0XOWL*HQ·V6PDUW6FHQHSURGXFWDVSXEOLVKHGLQ>0DSHV@XVHVWZRPDJQHWLFDOO\
WUDFNHGGDWDJORYHVLQVWHDGRIEDWV8VLQJWKHJORYHVWKHXVHUFDQSHUIRUPDYDULHW\RI 'RSHUDWLRQVDQGJHVWXUHV,WFRQFHQWUDWHVRQV\PPHWULFWZRKDQGHGWHFKQLTXHVIRU VFDOLQJURWDWLQJDQGVWUHWFKLQJREMHFWVDQGQDYLJDWLQJWKURXJKRXWWKHVFHQH8VHUV FDQDOVRDOLJQREMHFWVZLWKERWKKDQGVYLDDQFKRUVDQGFRQVWUDLQWV
&+$37(57:2²3$*(
&XWOHUHWDOGHVFULEHLQ>&XWOHU@DZLGHYDULHW\RIGLIIHUHQWREMHFWPDQLSXODWLRQ WRROVWKDWFDQEHXVHGZLWKWKH5HVSRQVLYH:RUNEHQFKVHWXS7KHEDVLFEXLOGLQJEORFNV IRUWKHLQWHUIDFHDUHWZR)DNHVSDFH3,1&+JORYHVDQGDVW\OXVDOOWUDFNHGZLWKD '2)3ROKHPXVVHQVRU8VHU·VZHUHDOORZHGWRXVHRQHRUWZRGHYLFHVZKHUHDVLQ FDVHRIWZR LQSXW GHYLFHV ERWK FRPELQDWLRQV RI JORYHJORYH DQG JORYHVW\OXV ZHUH HQFRXUDJHG 7KH FDWHJRUL]DWLRQ RI WKH ORQJ OLVW RI WRROV LV RULHQWHG RQ *XLDUG·V FODVVLILFDWLRQRIPDQXDODFWLYLWLHVLQWRXQLPDQXDOELPDQXDOV\PPHWULFDQGELPDQXDO DV\PPHWULFDFWLRQV,QWKLVZRUNUHVXOWVQRWRQO\UHLQIRUFH*XLDUG·VREVHUYDWLRQVRQ DV\PPHWULFGLYLVLRQRIODERUEXWDOVRQRWLFHGWKDWXVHUVSHUIRUPHGLQFRPSDULVRQWR D V\PPHWULF SDLU RI JORYHV EHWWHU ZKHQ WKH\ XVHG WKH DV\PPHWULF FRPELQDWLRQ RI JORYHDQGVW\OXV
$FRPSOHWHO\GLIIHUHQWDSSURDFKWR'LQWHUDFWLRQLVGLVFXVVHGLQ>=HOH]QLN@
=HOH]QLN HW DO SUHVHQW D PHWDSKRU XVLQJ WZR HVVHQWLDOO\ ' LQSXW GHYLFHV VLPXOWDQHRXVO\ IRU ' REMHFW PDQLSXODWLRQ DQG FDPHUD FRQWURO 7KHLU FRQFOXVLRQ VKRZ WKDW D FDUHIXO GHVLJQHG PDSSLQJ RI ' SDUDPHWHUV RQWR ' GHYLFHV FDQ LPSURYHHIILFLHQF\RIFRPSOH[PDQLSXODWLRQWDVNVZKHQHYHUWKHPHWDSKRUXVHGLV FRPPRQHQRXJKWRXVHUV8VLQJHQWLUHO\LQGHSHQGHQWGHJUHHVRIIUHHGRPRQWKHWZR GHYLFHVUHVXOWVLQFRQIXVLQJVHWXSVDQGXVHUVIHHOGLVRULHQWHG$JDLQWKHREVHUYDWLRQV RI*XLDUGVHHPWRKROGWKHDXWKRUVUHSRUW´7KHEHVWFKRLFHRIPDSSLQJVVHHPWR EHWKHRQHVWKDWKDYHWKHVWURQJHVWSK\VLFDODQDORJV7KXVWHFKQLTXHVLQZKLFKRQH KDQGSLQVGRZQDSRLQWLQWKHVFHQHDQGWKHRWKHUKDQGPDQLSXODWHVUHODWLYHWRWKDW SRLQWVHHPWRZRUNSDUWLFXODUO\ZHOOµ
2.4.3 Two-Handed Interaction in Immersive Environments
'HVNWRSEDVHG ELPDQXDO LQWHUIDFHV VKRZHG KRZ DV\PPHWULF GLYLVLRQ RI ODERU FDQ LPSURYHSHUIRUPDQFHEXWDOVRSRLQWHGRXWWKHIXQGDPHQWDOLPSRUWDQFHRIFDUHIXOO\
GHVLJQHG LQWHUIDFHV ,PPHUVLYH HQYLURQPHQWV SODFH HYHQ PRUH UHTXLUHPHQWV RQ LQWHUIDFHV3DUWLFLSDQWVRILPPHUVLYHYLUWXDOHQYLURQPHQWVDUHYLVXDOO\FXWRIIRIWKH UHDOVXUURXQGLQJV8VHULQWHUIDFHVIRUVXFKVFHQDULRVKDYHWRFRQVLGHUWKDWUHORFDWLRQ RIGHYLFHVLVKDUGO\SRVVLEOHZLWKRXWDJRRGYLUWXDOUHSUHVHQWDWLRQ3K\VLFDOSURSHUW\
RIWKHKDUGZDUHSOD\VDVLJQLILFDQWUROHLQWKHVXVSHQVLRQRIGLVEHOLHIRIWKHSUHVHQWHG YLUWXDOHQYLURQPHQW
7+(3(5621$/,17(5$&7,213$1(/²$7:2+$1'(',17(5)$&()25$8*0(17('5($/,7<
&+$37(57:2²3$*(
)LJXUH 9LUWXDO5HDOLW\RQWKH:,0/HIWLPDJHVKRZVWKHDFWXDOLQWHUIDFH 5LJKWLPDJHLVSUHVHQWHGWRWKHXVHULQ+0'
6RZL]UDODQG$QJXVVKRZLQ>6RZL]UDO@DQG>$QJXV@KRZWRDGGUHVVWKLV SUREOHPLQDQLPPHUVLYHVHWXSDWWKH%RHLQJ95JURXSIRUDPDLQWHQDQFHWDVN7KH\
XVHDEXWWRQPRXVHHTXLSSHGZLWKRQH3ROKHPXV)DVWUDNUHFHLYHUIRULQWHUDFWLQJ LQ'7KLV9LUWXDO7ULFRUGHULVWKHKDQGOHRIDSDGGOHOLNHLQWHUIDFHZKHUHDVTXDUH GLVSOD\VXUIDFHLVDXJPHQWHGLQDGGLWLRQWRWKHUHDOSURS7KXVDGHYLFHZDVFUHDWHG WKDWJLYHVVXIILFLHQWWDFWLOHIHHGEDFNIRUKROGLQJWKHGHYLFHDQGLVIOH[LEOHHQRXJKWR IXOILOGLIIHUHQWLQWHUDFWLRQWDVNV$SSOLFDWLRQVDUHFRQWUROOHGHLWKHUZLWKWKHEXWWRQV RQWKHKDQGOHRUZLWKWKHRWKHUKDQG7KHLQGH[ILQJHURQWKHXVHU·VRWKHUKDQGRUD VW\OXV KHOG LQ WKDW KDQG LV DOVR SRVLWLRQ WUDFNHG 7KH DSSOLFDWLRQ HQFRXQWHUV LQWHUVHFWLRQRIWKHYLUWXDOGLVSOD\DQGWKHSRLQWLQJGHYLFHVR'DSSOLFDWLRQFRQWURO FDQEHSHUIRUPHGIURPZLWKLQWKHYLUWXDOHQYLURQPHQW+RZHYHUWKHGUDZEDFNRI WKLV DSSURDFK LV WKDW WKH YLUWXDO GLVSOD\ VXUIDFH JLYHV QR KDSWLF IHHGEDFN RI WKH SHUIRUPHGRSHUDWLRQ
$GLIIHUHQWSUREOHPLVDGGUHVVHGE\3DXVFKHWDO>3DXVFK@>6WRDNOH\@
1DYLJDWLRQLQLPPHUVHGYLUWXDOHQYLURQPHQWVPD\EHFRPHDGLIILFXOWWDVNIRUVRPH XVHUVVLQFHDGDSWDWLRQWRQHZPHWDSKRUVUHTXLUHVDQLQWURGXFWRU\SKDVH,QDGGLWLRQ WR WKH ILUVW SHUVRQ YLHZ WKH GHYHORSHU RI WKH:RUOG LQ 0LQLDWXUH :,0 PHWDSKRU VXSSO\D*RG·VH\HYLHZRIWKHOLIHVL]HGVXUURXQGLQJVSDFHRQDKDQGKHOGFOLSERDUG )LJXUH 1DYLJDWLRQ ORFRPRWLRQ DQG REMHFW PDQLSXODWLRQ FDQ EH DFKLHYHG DW GLIIHUHQWVFDOHVE\GLUHFWO\PDQLSXODWLQJREMHFWVRQWKH:,0ZLWKDEXWWRQEDOOKHOG LQWKHXVHU·VRWKHUKDQGDQGJHWWLQJIHHGEDFNRQWKHVFDOHRILPPHUVLRQ(GZDUGV DQG +DQG >(GZDUGV @ GHVFULEH VLPLODU DSSURDFKHV LQ WKHLU ZRUN DERXW WKH SURWRW\SH RI WKHLU XVHU LQWHUIDFH0D36 IRU QDYLJDWLRQ SODQQLQJ DQG YLHZSRLQW PDQLSXODWLRQ ZKLFK WKH\ LPSOHPHQWHG DV DQ H[WHQVLRQ WR LPPHUVLYH 950/
>+DUWPDQ@EURZVHUV,QWKHLULPSOHPHQWDWLRQWKHXVHULQWHUIDFHLVSDUWRIWKH
&+$37(57:2²3$*(
YLUWXDOHQYLURQPHQWLWVHOIDQGVXSSRUWVDZLGHYDULHW\RIQDYLJDWLRQPHWDSKRUV7KH HPSOR\HG 9LUWXDO)OH[RU LQWHUIDFH GHYLFH LV YHU\ VLPLODU WR WKH 9LUWXDO 7ULFRUGHU DSSURDFKLQ>6RZL]UDO@>$QJXV@DQG>:ORND@LWDOVRH[WHQGVWKH UHDO GHYLFH ZLWK YLUWXDO PDSV $ PRUH UHFHQW ZRUN RI 3RXS\UHY SUHVHQWV LQ
>3RXS\UHY@WKH9LUWXDO1RWHSDGDWZRKDQGHGFRPELQDWLRQRIDVPDOOSUHVVXUH VHQVLWLYHWDEOHWDQGDGDWDJORYH)LJXUH7KHZRUNIRFXVHVRQWDNLQJKDQGZULWWHQ QRWHVLQDQLPPHUVLYHHQYLURQPHQWIRUDQQRWDWLRQ$SSOLFDWLRQFRQWUROLVDFKLHYHG XVLQJRQHOHWWHUFRPPDQGVZULWWHQGLUHFWO\RQWKHVXUIDFHRIWKHSDQHO
)LJXUH 9LUWXDO1RWHSDG²DOORZVWDNLQJQRWHVLQDQLPPHUVLYHHQYLURQPHQW
2.4.4 Summary of AR Systems and Interaction Metaphors
$V UHVXOW RI WKLV VXUYH\ RQ DXJPHQWHG UHDOLW\ V\VWHPV DQG WZRKDQGHG LQWHUDFWLRQ WHFKQLTXHVDVXPPDU\VKHHWFRXOGEHFUHDWHGDVVKRZQLQ7DEOH6RPHWLPHVLWLV KDUGWRGLVWLQJXLVKEHWZHHQDZKROHV\VWHPDQGDVWDQGDORQHLQWHUDFWLRQPHWDSKRU
%HFDXVHRIWKLVWKHQDPHLQWKHILUVWFROXPQRIWKHVKHHWUHIHUVWRV\VWHPVDQGRUWR LQWHUDFWLRQPHWDSKRUV,QRQHFDVH9LUWXDO7ULFRUGHUWKHUHLVDOVRDGRXEOHQDPLQJ LQ OLWHUDWXUH :KHQ D PHWDSKRU LV WZRKDQGHG WKH PDUN IRU WKH RQHKDQGHG LQWHUDFWLRQ W\SH ZDV RQO\ VHW LI DXWKRUV H[SOLFLWO\ VWDWHG WKLV 7KH FDWHJRUL]DWLRQ RI LQWHUDFWLRQ WDVNV SHUIRUPHG ZLWK WKH GHYLFH LV GLYLGHG LQWR REMHFW PDQLSXODWLRQ YLHZSRLQWPDQLSXODWLRQDQGV\VWHPFRQWURODFFRUGLQJWR>+DQG@+RZHYHULQ VRPHFDVHV²ZKHQWKHGHYLFHLVXVHGLQGLIIHUHQWDSSOLFDWLRQV²DIILOLDWLRQWRRQHRU WKH RWKHU JURXS LV DSSOLFDWLRQ GHSHQGHQW ,Q WKLV FDVH WKH WDEOH WULHV WR FRYHU DOO SRVVLELOLWLHVRIWKHGHYLFH
