Interactive Simulation for Multimodal Virtual Environments

(1)

Interactive Simulation for

Multimodal Virtual Environments

!"$#%&'()*($

+,.-/10230457689:$;"<$0=

>@?AB$CDEFG%HI?A?$J$?

© Dinesh K. Pai

Objectives

❚

KMLNO3PRQ.SUTWVXSZYO[S\].^ML _a`bcd_feghb i.j.kUl"m[n@o pq$r

ko[si rp

n q tuwvyx

❚

^z|{}~.X^.{ ^$U$U %U$.U71@

.¡¢£¤¡ .¥@¦X§¨©£ ª.¡«$¦¬ ®a ¢¡.¦¬§@¯

❚

^°± ^M§®²® ^± ^¯.¥ ^« ^± ^³3^"¦^± ^.ª¯^´@ ^± £y¨[¡.¦ª

± ¨

µ¶.·¸º¹.»¶.·¼½¾µ¬¿À.¹Á½¼ÂÀ.¹Ã ÄÅÆ¾ÇÈ%ÉÊË¾ÌÎÍÎÏËÈXÐXÑ

❚

ÒyÓ.ÔÕÖ1×ØXÙÓ.Ô ÚfÛ.ÜÝ.ÞÚfÛ.Ü¬ßàMÛá â[ã@äå.æç.è¬éãåê7ãâ ëfìí¬î.ïðñòó.íóô õö$÷øõ÷¬õù.ú ûú.ù.úû

© Dinesh K. Pai

Outline

❚

^üý$þÿ ^þ^ý

❚

^!"^#"$&%'$()$(*+,-".0/1+*$2.3'*54+.64

❚

78:95;<=>;@?A8B9CD?0EF2BG<;E89

❚

HJIKLMNPOQRLS)T0RN2UVOQWRLS

Introduction

© Dinesh K. Pai

Human Interfaces and Perception

© Dinesh K. Pai

Human Perception

❚

XZY[]\_^`abdcY:^ebfhgi`:jYk`ja-a^`

kmlb\nkPYfaboJpqrstu>v-u3rqeqvwxyvsmz{|dtxw}

xts{y~rq)xs xp5y{u utv|

❚

@PD35:D1:2->

5mJA 3e:

❙

d3e>3¡ ¢d0d>£¤¥3¦> 0§0££6¨3] 3§>£3©

❙

^ª¥«d¬^~®^¯3°]±6®^²>³

❚

^´¶µ·^¸¹^ºP»¼½· ^¾]¿À^ÁmÂÃ^ÀÀÁmÄ5ÅÆ¿ÇÈ5¿ÊÉÅÆ@ÄÆË¾AËÈÌ3Ë

(2)

© Dinesh K. Pai

Example: The McGurk effect

❚

^!"# ^$$ ^%^&('

© Dinesh K. Pai

Some (Virtual) Realities

❚

)+**,+-/.0 12435687:98;19 < =?>8@<57A1B42C=D17AE5B;GF H4IJK JLMONLKQPSRTIIJLMVUWRTPSXCY8H4Z?H(R [LKHCZ

\^]_A`]aSb\caS] dGefefe

❚

^{gihj k}^lmn^hok^{jpOhrqts^k}^{p k}uTvxwTyCzW{8|}o

© Dinesh K. Pai

Some (Virtual) Realities

❚

^~++/ ^gmn^ok^{yw u} ^u8jhAk|4}#u{Ww s^:hokw j

(w mjuw /Ww8w8oGo^{l}DSf

❚

D :^W}#f^8V CWG

© Dinesh K. Pai

Some (Virtual) Realities

❚

+ +¡/¢ 4£S ^: ¤¥¦t§¨© ªS« ¬S© ®}¯

°f°f°

❚

±8² ³f´+² µ¶·¸´^¹rº·#»¼/»½#³¾¿^À Á´¾¿S·8½Â¸Ã²8´Ä½#²´Ä·T¿

ÅÆ Ç:ÈÉÊË

© Dinesh K. Pai

Some (Virtual) Realities

❚

Ì+ÍÍÎ+Ï/ÐÂÑiÒ8ÓÔÒTÕÖ Ó×ØGÙ×ÚÛÕÜCÛOÝGÞÓ^Ò Üßà8Þ(ÓÔÜáâÖ Þ(Ò ×ÕãÞ}Ö ßàäÓÔÜ åWÒTØSÜCÛ8Þ}Öæfæfæ

❚

^çè^{é ê}ëìríWîðïñ^òóWô îrêõëô^ö÷

© Dinesh K. Pai

Example 1: Real Wok

(3)

© Dinesh K. Pai

Example 1: Virtual Wok

[Doel, Kry and Pai 01] © Dinesh K. Pai [James + Pai 99]

Example 2: Deformation

Input / Output

© Dinesh K. Pai

Visual Displays

❚

!#"%$'&()*$'(+(%,

❚

-/.0213 4%5 6 7

❚

^{89 : ;}

❚

^{< =}^>?@9%AB ^CED

Stanford Responsive Workbench

UIC/FakeSpace Cave

© Dinesh K. Pai

Auditory Displays

❚

FGHIKJL MN G+O

❚

PJ GH%Q#G%RSO

❚

P+MTN IOVUNWXL G+OZYO[L H%R'I\]H%R^G

❚

D%G_ G%R2` G%R'H WaYMNcb2YdWGR'O

❚

^{ef g hai}^g%jⁱ^{kg hai}^lm nXoqprEstvu2wxyz{S|

© Dinesh K. Pai

Haptic Interfaces

VTI CyberForce

PHANToM Harvard Tactile Display

Logitech IFeel

(4)

© Dinesh K. Pai

Utah Locomotion Interface

http://www.cs.utah.edu/~jmh/Locomotion.html © Dinesh K. Pai

Other input devices

❚

!"#$&%(')*+%

❚

^{,- .*)}^/01%203 %4"6587- #%

❚

9 *:050 "1;)%<)"=3

❚

^>).5"67?"=30%

Polhemus VTI CyberGlove

MicroScribe

© Dinesh K. Pai

“Real soon now” displays

❚

,@?065:AB-/CDFEGE:HI J(KBHLH6MNPORQSUT

❚

^VWBX6Y^YZ:X=[^\U[^N ^{]8^}^{\6^}^V_<X`ba ^T

❚

^[c[c[

Geometric Techniques

dfePgihPjklhPm(noqprhFs8nth@uvhFw@m4xPmyzjw

{|P}~fPPrPP8(z~

© Dinesh K. Pai

1. Geometry representation

❚

^b= i

❚

PPU18 ¡B:¢£¥¤¦§ £<¤

❚

¨U©ªq«¬®¯°±

❚

^«³²°^´µ^¶¯±2¬=°^²b´µ^¬=· ¸¹º»¼¸½¾6¿¾ À+Á

❚

^Â6ÃÄÅ6Æ^ÇÆ^È<ÆÉ=Ê&È(ÃËÌÍÎÏÈ

© Dinesh K. Pai

Subdivision surfaces

❚

ÐÒÑRÓÔiÕ:Ö¼×ØÖÙÚÛÜÝ<ÞBÖÖ Õß&Ý(ÚØàÓ Û<ÜÝRàáØÖÞâÓ

×Ö6ãÔßÜÙ=ØÓãÞBÖÙÜ6ã

❙

^äbåæ(ç^èGåéäbå^{êëècìíìî}^ì(ï<ê

❙

^ðñò4ñbð4ó<ôõñ4öc÷bøð

❚

ùlúûüýþcûÿûiü=ú tüþ+ý4ü

! "$#%'&)(+*,"$#"-.%0/1,"$24356#735*-"

$8*79% :;<=?>@.ACB?DEF GIHJ?KML)H NO?PCQ4RSOMTUTV?PMW

(5)

© Dinesh K. Pai

Example: model of real object

© Dinesh K. Pai

Subdivision surfaces

❚

❙

^"!^# ^$&% '())*,+&-/.01(2".34

❙

⁵⁶⁷^89:;<⁸⁶ ^=&> ^?@AB&C

❚

^DEFHGJIHK^LNM

❙

^OP/PQ

❙

RST"UWVXXYZH[\]^

❙

^_a`/bc^dec^fbhgijkjfl^demⁿ

❙

^oqp^rts/uvwyxz{ry|}zp~/prtpWrt|kv&wx/rNvu

1/y/

© Dinesh K. Pai

Refinement rules for Loop surface

[From: Zorin & Schroder. SIGGRAPH course]

© Dinesh K. Pai

Parametrization

❚

^Jq& H¡£¢¤ ¥¦¤¥§¨

❚

©«ª¬®h¯±°²q³µ´¶·¸°¬²°¹qº»¬°®¼²º»¸¾½&¿

¼¬°¬H³£¶¸°ºÀ¬¸º²q» º½Á¶N¬N½t®

❙

^ÂÃ^ÄÃ ^{ÅNÆÇÈWÉÊ}^Ê^ËÌÍ^{ÎÏeÐWÑÓÒ} ^ÔÕ^Ö×Ø/Õ^Ö1ÙyÚÍ^Õ^ÛÜ/Ù

ÝÞ/Þß{àâá ãäåæçèé êë&ì ítî/ïðñò&í

❚

^ó ^ôõöH÷ øøHùûú/üýNþÿúüýþý

!" #%$&')(+**,-/.012435+160798:3

;<;=>=;?@A;BCD@E?F%@G%H=?@EIJFH=CK

© Dinesh K. Pai

Resources

❚

L+MNONQPRTS+UWVX%Y+Z\[^]_X%`_[aY+bc%ded[adX%`_[aY+Zf\gV][

hiikj9lmkmkn+n+npoqArtsuwvxyDz {|}t~| D^ tDE^tDDD

❚

^% ^¡:¢£^¤:¥a¦^§ ^{¨A©ª«+¬^©}

¯®±°²³´³µ¶¸·¹%º/»¹¼)½+¾µ³º/¿Qº°½+ÀÁÂa¶9Ã

ÄÆÅ%ÇÈÉÊ ËÌÍÎÏÐÌÑ+Ñ

Dynamics and Contact

(6)

© Dinesh K. Pai

Overview

❚

^! ^"^{# $%"}^&'⁽⁾^&^+*

❚

^, ^-.)"/

❚

0214356879#:34;<1:3>=:3?95.3@+ACBD56EFG 0-HI :3EJK 6G

❚

^L:DD7FMJE^:DJ:N=5GOJ ^P<QH^JE^{R:@ B} ^{@ BD56E}^FG

5.D@)L:DJ5FJS7T:HQJE:D

© Dinesh K. Pai

High school dynamics

r f

v

❚

^UVWSXZY[\^{]-^_^a`bW}

❚

ced"fgheiZjlknmo pSqsrtuvqw<xtysy?zlq?zluM{|tvl}

~M|>-~

m f v

v r

= 1

=

© Dinesh K. Pai

Simulation: Numerical Integration

❚

^-^M^M ^el8l > ¡C8"¢|£

❙

^¤¥^{¦ ¥}^§.¨O©ª^«-¬lªs ^®¯l°^¤ª ±%²³|´µ¶¸·¶lµ¹º³'»¼²½³|´l¾¼¿À

❚

^{Á4ÂÂÃÄÂÅ}

❙

ÆÇsÈlÉlÇÊËÈÊÌÌMÍlÇsÊÌÎ

❙

ÏeÐZÑlÒ#ÏÔÓÕÑ#ÏÑlÖÑ×Ð|ÓØÙ

❙

^{ÚeÛZÜlÝlÞ}^ß^Þ^ÛZà

) ( 1 ) x(k) h x k

x(k + = +

© Dinesh K. Pai

Numerical Integration (continued)

❚

^{áCâã ä}^{åæ ä}^{çè+ã ä}^{èãéêëå.ê}

❙

ìíîïð4ñîlòóôîñ|õìöð>÷îøù øúûlíùlîüûMñ?ñZýþÔîõlÿMîsù

!"#%$'&()

❙

^*^)$+&% ^*^,-./012&(,354 65&()37"

❚

^8:9;<^=><?@BA5C,DFE<G<CH

❙

^I^J%KL^I^MI^N:ION0P)QR"SN0TRUWV.PXQYXZ[,J%KLIMIN \Y]OQP ^ _]NN0S Z

`acb d%efg,hig.jk d(jglmnio

❚

^p:qrFstuvwyx5t,w+szx)u{w|x}wF~wtu+

© Dinesh K. Pai

Rigid body dynamics

❚

^F0|55,6

❙

F"):.'7)%

)¡c¢(£¤¤ ¥ ¦§¨.¤©ª ¦¨«7¬¦¤/¨0£§#®

# §)¤¨¯£¦§¨.¤

❚

^°+±²³´²Fµ¶¸·Y¹³,ºc»W¼:½6¾²6¿À´¹Á

❙

^Â}ÃÄ^Å^Æ%Å^{ÇÈÉ0ÊËcÌÅ}Í/É0ÈÇÎ#Ê%Î#ÏÇ)ÍÉÐÈÅÇÉÑÍ

❙

^Ò6Ó^ÔÕÕ#Ö×#ÔÓØÚÙ)×ÛÔÜÖ×5ÕÝ Þßà"áâäãFåáæàç)è0é

â(êëå2ì(æíáà.î2îæ)ë0ïë,êæî

© Dinesh K. Pai

Rigid motion and transformation

 

 Θ

= 0 1

E p

i

❚

ðÚñ+òóñôõ+öõñc÷ø7ùûú.üFöøý.ñcú,ò

j

❚

^þcÿYÿ ^ÿ

φ

Frame i p

Frame j

(7)

© Dinesh K. Pai

Cross product matrix

❚

 







 







=

0 &

&

-

&

- 0

&

- 0

& @

[

x y

x z

y z

r x

&

r

& @

[ ≡

© Dinesh K. Pai

❚

^"!$#&%(')$*+,-./

❙

⁰²¹³45768"0)519;:)3<>=?97@A@8>6B5C983D34

❙ ω

^EFGHI^EJ"KLI^M;N)O^P>Q

❚

^RS ^T Û^VCWYXU^ZU^{[]\_^Ù}^{â}

❚

^bcdefdghjikl7mon(p)qrkmslt"uv

Spatial velocity

 

 

=

t r

f f f

 

 

= v

&

φ

 

 

≡

−

0 0

v ] [ ^&

R 1

R

© Dinesh K. Pai

Transforming spatial velocities

[ ] _ ^

 

Θ Θ

= Θ p Ad 0

i j

❚

^wyxz${|^}~ jA

^j φ

ⁱ φ

φ

i Frame i p

Frame j

φ

j j

i j

i φ = Ad φ

© Dinesh K. Pai

Notation...

❚

"r"")

[ ] [ ] ⁼ _ ^ [ ] [ ] ^v ^& ⁰ _ ^

&

φ

❚

^¡^¢¡^£¤¥( ¡^o¦_ §¥¡^¨

I dm r

r M ⁼ ∫ _ ^ ⁻ _[ ^[ r _] ^] ² ⁻ ^[ ^] _ ^

© Dinesh K. Pai

The notation finally pays off!

❚

^{©yªA«¬A¯®}°;±C²"³µ´®¶®·]¸²"·ª

¹yº»½¼¾¿À ÁÂAÃÄÅ ÆÇÈÉÊÌËÍÎÏ ÐÑÒ"ÓÔÕjÖ×ØÙÚÛjÜ

❚

ÝÞß&àáâãåäræç§èoéyãä½çÞêëìríáäïîÞâµàáâ(çæðíã

❚

ñòóòôòõö7òy÷ øùAúûüyý)þÿ

[ ] ^φ ^φ

φ M

M

f = − ^T

v m f =

© Dinesh K. Pai

Wok Example

(8)

© Dinesh K. Pai

Contact interactions

❚

❙

^!"^#!$

%'&()*+,&-,.&(/10.+*32-0")546&6 78.9;:<>=.:8:9<'?97@8

❙

ACBD.E;FGEH>G"BD.EID5JBJKL"M

NO5P'QRP'OSTUVW.OW.OSQ1X.VU3YVZ[ZQP>\]TS_^CVP'OXUVQZ

© Dinesh K. Pai

Rigid body impact

❚

^`badcbeCf^ghcCij]fkmlan;opiqghearscbtCulabn;opvwcdkxj yz|{C}~>.Cb.

❚

_C[b>ddsdbmw

b >¡C¢£¤¦¥§¨©¡ª

❚

^{«¬b¦®¯°±}^²³^´pµR±^°¶·±^¸±¹²¬d¹ºp¹º»¯±¼½

¾¿ÀÁÂÃÄÅÆw¾

© Dinesh K. Pai

Rigid body impact

❚

^{ÇÈRÉËÊÌ}ÍÎÏÐpÈÑÒÐCÌÍÓ

❙

^{ÔdÕÖØ×;ÙÚÛ} ^Ü.ÝÞ^ßßÝà ^á ^{âãä3åæç5è} ^é5êë>ìí.êmîï1êë'ëwìî_ìêí.ðïë ñ'ò1óõôöô÷.ôöøù úû

üCýþýÿü 'ü

❚

^{! "}^{# $}

❙

^%'&(*),+.-0/*+.-2143 (5&6

❙

70859:;.9*< =>@?5ABA'CD*E

❚

FGIH JKMLONPJ0QRS0TUK2JJVWRYX[ZV \*GB]N0VR ^_a``bcd2efg2h[ijlklmnooo

) ( )

( t ⁺ = ev t ⁻ v

0 ) x f( x, x

m

^p^p

+

^p

=

k x x c ) x

f(x,

^q

=

^q

+ k x x c x ) x

f(x,

^r

= ^m

^r

+

© Dinesh K. Pai

Continuous Contact models

❚

^s5t^{u vwx} ^y0z0{!|}~{O~0.l

4 . l

≥

^¡

≥

^¢ U£¤ ^¥ ^{¦,§0¨!©ª}«¨O«¬.l¢®l¯° ±

❚

^{²³µ´·¶´l¸}^¹ºB¸^³µ»¼½¾¾¿¹,³µ».ºlÀ¹º[ÁµÂ¸¾Ã» ³0»ÄI» ½ÅÀ ºB¸Æ ½ ÇÈÉ[ÊËÊÌlÈÍË ÇÏÎYÈÐÐÑ.È0Ò Í Ó ÈÇÕÔ

ÖØ×ÚÙÛµÜÝ.ÞBßÛµàá¿â ãßá'Þ[ä¡åæ2âçÝàßè¡Ý âä¡åà éëê*åà

ì

ââíîílßê.ßâà.ÞIÜïáðÛ0Üñ âé

ì ï òóô!õ*ó ö÷ ôOó øIù úû

❚

^ü úµýÕ÷5þ ÿ2ÿó.ø ýó!ÿ[ý*øú óý ûÿ2ÿ.þø

❚

^! "$#%#&&')(*+-,.#%*/#0-1 2/3456879 :

© Dinesh K. Pai

Continuous Contact models

❚

;8<>=-?@ACBEDGF$H<@IKJL?= M8<ONIP<H QFRSAT$UIK?I V.WXXYZ[\Z]$^-Z[8_Z`Wba$cS_d$efhgGW_id-a$]j

❙

klnmoqp8r.stvuxwlnoqyKzm|{}v}yL~lpnm|{u-rLzu-t>sy8rynm|tpnm|{u8r

❚

)G G$$>hiG--

❚

ⁱ^L- G¡$¢ K£L¤¥¦¤P¡O§- ¨n ¢

©ª)«¬®$®$¯°.±²³i±´8¬µ¶$·

❚

¸¹>º8»¼½»/¾-¿-¹ÀÁ8»Â¹ºEÃG¹$Ä$¾>ÀÅ

❙

Æ|ÇvÈLÉÆÊqË-ÌnÆÉLÊiÆ>ÉxÍ ÎÏÐÑÐ8ÒvÓ8ÔÕÖÐ×ÙØÚ-ÕÑnÛÜÝÞ8ßà áâãvâ8äæåç|ãèéå ê|ëìîíìLï-ðììxñòLó$óCðvìLìLíxò-ô

© Dinesh K. Pai

Multibody Chain

(9)

© Dinesh K. Pai

Multibody Dynamics

❚

k

k k k k

T k

k

k f M b

f = ⁻ ¹ Ad ₋ ₁ + φ +

❚

^{! "}#$%&(')%"

k

k k k k

k k

k = ₊ ₊ + H q + a

1 φ 1

φ Ad

❚

*,+(-/.1023+4(5)+678

k

k T k

k = H f

τ

© Dinesh K. Pai

Extended DAE: Mx = b



 









 







−

=

9 999

9

9 9

0 0

2 2 2 1

1 1

T T

T

H

H I M I H

H I M I

M

Ad

1 2

[Lubich et al 1992], [Ascher,Pai,Cloutier 1997]

© Dinesh K. Pai

Sparse Gaussian Elimination



 









 







−

=

9 999

9

9 9

2 0

2 2 1

1 1

T T

H

H I M I D H I M I

M

Ad

1 2

© Dinesh K. Pai



 









 







−

=

9 999

9

9 9

0 ˆ

2 2 2 1

1 1

H T

H I M I D H I M I

M

1 Ad

2 Sparse Gaussian Elimination

© Dinesh K. Pai



 









 







−

=

9 999

9

9 9

2 2 2 1

1 1

ˆ

D H I M I D H I M I

M

1 Ad

2 2 2 2 M ˆ H H ^T

Articulated Body Inertia

ABM [Featherstone 87] = Sparse Gaussian Elimination

© Dinesh K. Pai

Contact Evolution

❚

^: ^; <1=>@?A=BDCFEG=?FHJIKEGLNMOCP

❚

QRNSUT1VWT1XXY3Z[V\]GVT^_S`\DabTNS c_dOeJfJghgije1d`gklnmpogc_dOel1qjr

sGtuFv,wNx/yNx`zt{n|[t}~ys

NODNF_O

❚

^[jG jNOj

N1J A¡1¢`£¤¦¥[§¨§¨¢`©1§,ªJ§1¡J«

¬11®/¯°

❚

^±^²³^´ ^µ^{¶_·¸j¹º¼»} ^½J¾^{¿1ÀG¿1Á¾}^Â_¿

Ã1ÄJÅDÄÆAÇ1È`Å/ÉÊÄJË1ÌÎÍÏJÐNÌÉÑJÉÍÉÒË

ÍÏJÅÓ[Ä1ÊGÇÍNÔ

(10)

© Dinesh K. Pai

Contact Simulation

© Dinesh K. Pai

Contact Simulation…

© Dinesh K. Pai

Resources

❙

"!$#&%')(+*-,#./*(0#&, 1234$5687993:;74+<=?>A@ = <95BC

❘

^D+E^E)FGH)H)IJIJILKMON PQSRT UV8WYXZ[ \+]_^]+``a

❙

^b0cddefg ^h_i^j ^{k lmSno}^p qYrstuvwAstxy$sz{O|tO}~ &y8tx~&|

/?&& 8+$

❙

$/ /¡&¢ £¤¥&¦¥§¨© ª«¬®¯_°²±³´¥&µ§¶¬·°$¸

¹º»¼0½¾ ¿À

❙

^ÁYÂÃ^Ä ^{ÅÇÆ+ÈÉÊË} ^ÂÌÍ ^ÎË^Ï ^ÐÑ$Ò&ËÓ0ÂË)Í?ÔÏ ÌÂÕ®ÃÈ_Æ Ö×ØÙ&ÚÛÜÝÞ·ß²à)Ù&Ú á0â×ÜÛãÙ ä å æJÝçÛèßéçèäêæëÙ&èÜ/çì8Üîí

ïOðñòó ô8õ&ðö÷ø&ù-úø&ûøüOýþ$ÿùü/ø üýø&ù

❘

! "#%$ &!' ()%*!+ *, -./)0+#1.#23-4546*!7!8

Sound Simulation

9:<;;=<>:<?@=BAC:<?@DE

FHGIGIJ KL<MONIP<M QSRIT<U

VW<X/W<Y Z[\B[][^I^_

© Dinesh K. Pai

Why Sound?

❚

^{`ba c@debf}^{g h}

❚

^ijHk^k^l^{m n}

❚

^oqp^r^sqr^{t u}

❚

^vwSxzyS{^|^{} ~}

❚

^H^b

❚

^<!^B

❚

^Hq^H

❚

^B

❚

^{Sq I}^¡

Essential for: Useful for:

© Dinesh K. Pai

Synthesis Method

Force Vibration

Emission

Propagation Listener Speakers

Sample

(11)

© Dinesh K. Pai

Force Vibration

Emission

❚

! "$#&%'(*)+,%-. /

Generalized Sound Cones

© Dinesh K. Pai

Force Vibration

Emission

❚

^0132!4

❚

5!67$8&9:;*<=>9?@6 A B

❚

CEDFGIHKJMLNJOPQMRSTQSQFDO

❚

UWVVXIYZV[E\^]_I`ab*\dce`fb\

❙

gKhijkZl

❙

mno

❙

pqrst

) , ( ) , ( )

( ₂

2 t x F t t x

A =

∂

− ∂ µ

© Dinesh K. Pai

Vibration Model

❚

uEvv^wx&yzw{|y} v~y*y !w{

❚

^{^wf^{!

~yy

0 ) , 1 (

2 2

2   =





 





∂

− ∂ t

t

c u x

A

( )

∑ ^∞

=

Ψ +

=

1 ) ( ) cos(

) sin(

) , (

n

n n n n

n ct b ct

a

t x

x

u ω ω

( ^A ⁺ ω n ² ) ^Ψ n ⁽ ^x ⁾ ⁼ ⁰

© Dinesh K. Pai

Vibration Modes of Square Membrane

© Dinesh K. Pai

Spectrum of membrane struck at (0.5,0.4)

© Dinesh K. Pai

Spectrum of membrane struck at

(0.1,0.4)

(12)

© Dinesh K. Pai

Spectrum of membrane struck at (0.1,0.1)

© Dinesh K. Pai

Interactive Sound Synthesis Model

❚

! #" !$ %

b (f _i )

^&')(+*,-/.,013254,6,04,07849:;6<&0=>

1@?00,<17$,<47$?A:97$,*$&9B<6,*1@,67&?0

❚

CD<EGFHEGIJFLKME@NEPOE@NJQARSRLTUNQNF$DVW$NVF$MXYE[ZL\^]

_S`abcPbde8fagdh

❚

^i8jklmn#opGkjlLqo<pmprtsojuosvwnxpGlmnyxs!l$zjl{

nxuo<p}|+msYpl~jlmwlmnos+~jp}x~l

( ) ∑

=

= ^M − i

i i

t x f

b a x f x t

e t

x

p ⁱ

1 )) (

( ( ) sin( 2 ( ) )

, π

© Dinesh K. Pai

Sound texture map

❚

<x~jupkjlLqo<p@mpPyzsznol$osp

f _i (x _j )

a _i (x _j )

zju

b _i (x _j )

zso5l$owlL~sonzyyouxjl$x p~ummpmxjp~s|Yz3o3xjlLsxwos!lm@o<p

❚

Ljl$os+yxzloA~p@mj zs!k@ojlLs$m@x<xsumjzlop

False color image of audio brightness

© Dinesh K. Pai

Force Vibration

Emission

❚

so/~oj3kpyo3lLs+~n

❚

zn;y<mjpyo3lLs$~n

❚

<

❚

^;$

❚

$$ ¡

❚

^¢£P¤^¥¥¦^§

¨©GªL«¬GG®¯°®±²«´³«¯µL¶ ·@¸´¹º»°¼L½H¾

© Dinesh K. Pai

DSP View

❚

¿ÀÁÂÃÄÃÅ3ÀÆ$ÇÈÉ$ÀÊYÁ

❚

^ËÆYÆ$Ç^ÅÇÀÃÉ8Å3ÂÃÌÂÈÍÉÇÂÃ5ÂÆ8ÇÃÎÍÉÏÆYÂÊ!Å@À<Á

❚

^ÐÇ^ÈÉ$ÀÊÅ3ÂÀÆYÆ$ÇÅÇÀÃÉ+Á}Å@ÂÑ;ÎÍÉ$ÄÒÈÀ

❚

^ÓÑÎÈÀÑ#ÀÃÉ8Ä<ÁPÔÏÕÖ#ÄÈ×ÂÊ$ÇÉLØÑ

❚

^Ô8ÇÊÀÅ3ÉSÕ<ÂÍÃÙ;ÙÀÑ#Â

© Dinesh K. Pai

❙

^{ÚÛÜ3ÛÝ°ÞßÛ}

àáâ3ãäåçæèåé°ê´ëì

íYî°ïð

ñòóô3õöç÷øGù²úèûü°ø´ýþ

❘

^ÿ

❘

!"

❘

#%$'& ()*+(

- Microsimulation - Wavetable

Force Vibration

Emission

(13)

© Dinesh K. Pai

Micro-simulation of contact force

❚

❙

"!# $%$%&'(!) $

❙

*+,.-/0132456/+07#7#8:9;<=2>/?5@4<=;-+-A:B?C70

❘

DEFEGIHJ KL.M N6O

π

^P^Q

τ

^{R S}T)UVXW>YIW

τ has correct form

❙

^{ZI[ \}](^_ \@`(a.bc)de_a#defXgchbih[kjlid=gmi[knblca#[f

❘

o#p=q"r.o3sp=qut6vw3xyzwt{3|)}zxy#w.ytvq#p=|)y#w3y#~quvq#o#q3y#~xq#p

© Dinesh K. Pai

Micro-simulation of contact force

❚

❙

6@. "k=k@#"

' @(6 ' 6¡£¢¤¡3¥¦§@6#¨

❙

©¡.ª@§@.¡¬« ®® ¥ ®¥

¡¬§¤@#' ®¡:¡. ®®¡. ®

¢¡. ¯°±²³´³ ±²³ ±µ·¶¸¹.´³³² º

❙

^».¼ ^{´µ=½¾@³k¿}^¼ ^¾@À³%¹ ^»#¼ ^¿^{³%¹µ=¾e½} ^» ^µ=½¾@³

½¹3°±ÀÁ¶²

¼

¿¾@³¹¶±

¼ ± »

³ÂÃ°µÄ

Å¤Æ@ÇÈÉÇ ÊËÌÇÍXÎÆ@ÏÎÏÆÐ=ÌÏÊÑ Ò)Ð'Ï(Í3ÒÌÓÌÊÔkÍ3ÎÇÇÓ

© Dinesh K. Pai

Synthesis API

Õ=Ö×eØÙ¤Ú¤Û'ÜÛ

ÝÞ¤ßàÞ¤á âã@äå

æç'è¤é'êë@ìîí=ïð=ñòíé ó@ôõö÷øù@úû6üýþøÿö

!

"$#&%#

')(*+-,.(/*0

© Dinesh K. Pai

❚

13254&687:9<;<9=9>2<?A@BDCFEHG

❙

I$J<KALMKJON<KPMQRQTSMKU

❙

VOW<XMY[Z]\5Y[W.^`_RX.a$bc

❙

^dFe&fg[hRi]j[k<flRmnfo-prq$fsRjMtiuo-fl<vRh[guh]vuo&q

❙

wxuy{z]|}.~yuz-Muy>}[y{~u<uy><|5}[y{z]x[~:zO)

❚

³F<>&<A< : ¡

❚

¢¤£&¥¤¥

JASS

Java Audio Synthesis System

Unit Generators

Adder UG

Low-pass UG Excitation

UG

Render UG

Karplus-Strong UG

Patch

Excitation UG

Render UG

F v

p Contact Force

Model UG

Resonance Model UG

Source

Player UG JASS

Graphic Display

Simulator

v, F, roughness p, damping, loadModel(file)

Dynamics Simulator, Collision Detector

start()

positions

User

display

user input

(14)

JASS Architecture

Engine Package

UG Package Render Package Patch Package

"!!#$%&#'(

)+*,%%&-' .//0123415&67

Demos

%RWWOH

%RZHG 6WULQJ (QJLQH

6FUDSH

%HOO

Wok Example

Integrating Audio Haptic Displays

User Interaction

❚

8:9&;<=><,?@=ABCAB;DBABD EGF

<H H=I<

F

>J? KL&M NO&PQR

❚

STUGVXWY&Z[,TW[\Y&]WV^@_:],`aV bcde fghijkml gnoClp&qno

❚

^rtsuvw^x yz&{xm|Xs}~Cz&}~

&,

❚

,

❙

,£, ®¯°±² ³´µ¶

❙

·¸,¹º»¼½¸,¾,¿ ÀÁ,ÂÃÄÅ Æ,ÇÈÄÅÇÉÊ ËÌÍÎÏJÍ,ÐÑÒ Ó,ÔÕÖ×ØÙÚÛ

❙

ÜÝÞßàá@âäãåæàçßèàééê,è

The AHI Audio-Haptic Interface

DiFilippo and Pai UIST 00

Results

Force

(15)

A User Study

❚

!"#$&%!')(*!%,+-."/+0')!1/0#2"34)#2'51*67-"+.8!6+9

❚

:<;=*>@?BACEDFCGHH,?IJKLMC;GN/OPGNNQ2OKCSRTG;I2KU VWBXBY2Z[\^]2_`VXB\\2Z[\ aXb,cSZd*e fhg^ije

❚

k,lnmhoqprs/tuwvwx<y,z{{s|rT}~zty,zTy*~rEt@s

❚

<*@rByEsr)){2~rp^ny,zBvyr rBr

❚

)B@Fq@B2n

Resources

❚

^®¯° ^±B²³´^µw¶^´^²·

A Digital Signal Processing Primer with applications to Digital Audio and Computer Music. Addison-Wesley, 1996.

❚ Ken Greenebaum (Ed.) Audio Anecdotes, Kluwer 2001 (to appear)

❚

¸B¹)ºBºT»¼5½½½<¾À¿ ÁÂº¹@Ã/ÄÅ2ÆÇwÈÉÊ*Ë

❙

^ÌÌÌÎÍ^ÏÐ ^Ñ^ÒhÓÔ ^Õ^ÖØ×|ÙÚ ^{ÛÜÝhÞ/ß,à}^á^âäã*åå^æ^âMã*åå ^ç^èéêFë

❙

^ìíhîîïSðíhñóòô,õ2ö,íh÷îøùñóô,îô*ú*ïSôùòñóô*úûMô«ünôhýFïSòñ)÷þû)ï

Deformation Simulation

ÿ

Contact Deformation

Deformation simulation

❚

^"!#%$'&)(*(&+

❚

^,.-^/0%132 ^4"56%7'8:9"7'8:68);< =>?"@3AB

❚

CEDGF%F3H%IJLKM"HNPO)QR HJ

❚

^SUT3VTWXY:T3ZWP[]\TY_^:X`badce"fU^:XY)ghji

❚

^kml3no)p^q r_st:u3vGwxPt)yz s{r|}|~u%|

(16)

Elasticity

❚

❙

"!$#%'& (')* "#(+(

❙

^,-^.$/102354768479;:=<+> .:?"27-9

❙

^@AAB5C1D^E ^FGIHKJ"L7FG;MNLOPO'M*J"LO5OQMSRTO'M*J"G7UV

❙

WYXIZ\[S]^`_+a bc7dIef;g

❚

^hijklmn=o^p^qrj7slSt+p^so7u=pvwvNklNk=x7t+pso7ky{zst+pn|x}

❚

~|Sr|=+=Y

|SN| ¡£¢+YP¤

Linear Elastostatic Models

❚

^¥¦^{§ ¨©}^ª

❚

^«¬®¯^°±*¬'²³°´*µ'¶S°Y°Y·

❚

^¸1¹^º»Y¼½¹¾w¿ÀÁÃÂ¼ºr»ÄÅÆÇ=¹¾ ÈÉÊËÌ7ÊÈÍÎ ÏYÐ\ÑÒÈÎÉÈÏÑÊÓ+ÊÏ|Ë

❚

Ô|ÕÖ1×5ØÙNÚÛÝÜÕÞ7ßÝà;á{âã|ä7Õâå æYçèéêIëçìíïîéíwì=ðñæë+òì|ó

ôõ|ö÷ïöøQùúûü=ý þÿNõùùù

ν = 0.5 2 0 1

3 1

1 2 2

2   + =

 





∂

∂ + −

∂

∑ ₌ ∂ ⁱ

k k i

k k

i b

x x

u x

G u

ν

Numerical Discretization

❚

!#"%$&')(*,+-.

❙

^/0¹²⁰^345'067684298421,:;4=<

❙

^>?^@2?^ABCEDBFGB@AIHJB,AK,LEMNO>=C2HQP

❙

R=S!T2U,V2WXY[ZE\]^G]U_I`Q]_a,SEVbOR Z2`Qc

❚

^{d efhgjilk}mn)o-pOm!q-r stuvwyx z{t|,}!z{t~ J##)

!-!-#y-! !'&'!

❚

Example: Boundary Elements

∫

∫ Γ

Γ

= Γ

+ p * u d u * p d u

c

H u = G p

Constant Elements

Point Load at j

i

g _ij Weaken,

Integrate

2 0 1

3 1

1 2 2

2 + =

 

 

∂

∂ + −

∂

∑ ∂

= i

k k i

k k

i b

x x

u x

G u

ν N u + b = 0

Discretize

Green’s Functions

for Discrete BVP (via BEM)

H u = G p

SPECIFY BC

Red BV specified Yellow BV unknown

REARRANGE

A v = - A v ^{_ _}

=

INVERT LHS

v = - A ^-1 A v = ^{_ _} Ξ ^v ^_

❚

^{·I¸2¹!º-»}^»¼½¿¾!ÀÁ¸EÂ!¾ ÃÄ½)Ä¾ Å¶Æ#ÇJÈ¸jº-Éy¾ËÊ!ÌÊ!Í¾'ÉyÌ

❚

Î)ÏÑÐÒ)Ó'ÔÕÖÓÏØ× ÙÚ×!Ó×#ÛÜÝ'ÞyÓ ß#àIÐ á âÚã ä å æØçèé&êç)ëìê í!î-ïð!î!çê

Fast Solution to BVP with Green’s Functions

v = Ξ ^v ^_

(17)

❚

❙

"!#$&%' (*) +-,/.102$34)

Ξ

❙ Ξ

⁵67*8916:;<;9: =>9

Boundary Condition Type Changes

❚

?A@BDCFEHGIJKLNMO P QNR/SNTSU/P S

Fast Elastostatic Deformation

❚

VWX/Y/Z[/\]_^a`bZcD^dZfeghX/ibX hgjk_[lhnm o

= H u = G p

= +

T s 0

s A A E

A = +

^p

v A - v A _s = _s

Notation

❚

^qsr^tvuxwyt tz{|}/~"Arw A/>

"D$ N_Dd>N/"A ¡

¢F£s¤ £¥¦

❚

¦±°

®²

❚

§©¨/ª«/¬n³

¤ £´µ

¯_«

¥

)E A A ( A ₀ ₀ − = _s

^¶

T 0 0 0

s A ( A A )EE

A = + −

T s 0

s A A E

A = +

^¶

= ⁺

Sherman-Morrison-Woodbury

❚

·A³DªF¸H¹²º

´£ ®

¦¤ £

¨/

°

«

¤

f»/

¦¼

DN« ½D¾À¿/Á

[ ^T ^- ⁰ ¹ ^s ] ^T ^- ⁰ ¹

s 1 - 0 1 - 0 1 -

s A - A A I E A A E A

A = δ + δ ⁻ ¹

T s 0

s A A E

A = +

^Â

❚

^ÃAÄ

= ⁺

ÅÆ±ÇÈDÆÅ É±ÊËÊ$ÉÌÍÎÊÏ$É±Ð1ÑÊÍÓÒ4ÌÔ&ÕÖ±ÑÊ××Ø

Capacitance Matrix Algorithm

[James + Pai 99,01]

❚ Solution to any BVP in terms of Ξ for a Reference BVP

Using Sherman-Morrison-Woodbury formula:

v ⁽⁰⁾ = [ Ξ (I-EE ^T ) - EE ^T ] v v = v ⁽⁰⁾ + (I+ Ξ )EC ^-1 E ^T v ⁽⁰⁾ C = -E ^T Ξ E = s-by-s capacitance matrix

❚ Direct solver w/ fixed solution cost

❚ Construct, cache and reuse C ^-1

❘ O(s ³ ) when constraints switch (or better)

❘ O(sn) subsequent solves for s nonzero BC _

❚

^ÙÚ^ÛÜnÝ

(18)

Haptic Interaction [James & Pai 01]

❚

^{! "}

#%$&'#(*)+),-(/.0 1243 5

❙

6798:%;<=>8@?A:=CBD8EFGIHJ:FKLGI7I8FM:BHJGNB:HJB'OGPNB

❙

^QR9STUDRVXWYZ[9\%]^%_a`bcd egf%hiMjhkmlon9p9qr stuvuw9x9yz {I{9|}*~> 9 I9

❚

ID C

Capacitance Matrices give Exact Local Models

❚ Consider contacting a free boundary.

Forces at only contacted vertices can be computed in O(s ² ) time as

E ^T v = E ^T v ⁽⁰⁾ + E ^T (I+ Ξ )EC ^-1 E ^T v ⁽⁰⁾

❚

¹Iº»¼º½¾ ¿¼¾´ÀÂÁ¾JÃÄ¿ÅIÁ½Æ'ÇÈÊÉÈ»A¾ ÈËÌaÅÍoÎÏ ÐÑÒÓÔÑÕ ÑÖÐØ× Ù

Ú Û Ü

E ^T v = -C ^-1

^Ý ^Þ

= Ξ

^Þ^Þ

^-1

^ß ^Þ

❚

^àIá

Ξ

^â^â^ãäØåJæççèèçéå>ãêçéëìíîãï*ðéçëè4êç*ñDå çò

Point Contact abstraction

❚

^{ó*ôõ*ö÷øù}úû´üøúû õüûL÷ýøþ'ÿüô ùú ùú%ùû ô ûJýõüû>ùøú õ*úþûôýô

øýô/öõ*ýô*ý

þùI÷%öõüô ôúûXøú

ùúô*ý ô

❚

! "#$&%')(*(+,-

')(./ 0#!21-%23(1(&4

5687:9<;=> ?A@:B C#DE

Capacitance Matrix in Haptics

Resources

❘

^FGHAI&J*KML:KN OPQRSPTVU W8XZY*[:\]^A_`a b#cAde*fg<h:ikjc lmknko:pq rsMtu*vMw:xAyzk{}|{~:wk:r2:#zV*z:wAu*{y#::

&:&::

##

❘

¶:º²´8±S³<»<¯k¼k¶:º´

½ ¾#¾#¾¿ÀÁÂÃÄÅÆÇÈÉÊ ËÌÍÎSÏ8ÐÏ8ÑÒÓÎÔÐÕÖÐ×Ø&ØVÙÚ8Û8Ü

❘

Ý8Þ:ß<àáâ ãAÞ:ä å8æ¬ç-è:éêSéëì<í-îëïAðSñëkèAð-òê ókôïõðòkõ8ðïAðéö

÷:øùûúýüþüù ÿ*ø÷ *ü÷ü*øú÷-ü

÷ü ! ""# $&%'(()*+ ,.-/0124343.56708 9;:-<=0>? @A

B&C

AED41A>F</G80>IHFGJ/8:1GK 9.:-<0>? LA?A:/> C

M N;N;N OPFQRTSUVWXYZ[]\^

Summary

❚

_a`7bIcd` efghie`jh kmlngop `7q

❙

^rts.uwvwx^{yz7y{w|u }}^~w}^y}^{s y{w}^;v.y

❙

4 .;

❚

n7nm m. I ¡;&¢j¢

❚

±²7³T´µ¶¸· ¹º´»¼.º¶¡½m¾· ¹º´»

7KDQ

¿

\RX