simulation in virtual environments
RolandStean 1
andTorstenKuhlen 2
1
VirtualRealityCenterAachen
c/oInstituteofTechnicalComputerScience,AachenUniversityofTechnology
2
(correspondingauthor)Virtual RealityCenterAachen
c/oComputingCenter,AachenUniversityofTechnology
Abstract. WeintroduceMAESTRO,aVirtualRealitybasedassembly
simulation toolthat comprisesphysically-based modeling,haptic feed-
back and articialsupport mechanisms. Thefocus of this paper is on
the developed support mechanisms and the evaluation of the system.
Theexperimentalresultsshowthatallthreefeatures{haptics,physics,
andarticialsupport{considerablyimproveuserperformanceanduser
acceptanceduringthecompletionofassemblytasksinavirtualenviron-
ment.
1 Introduction
Duringthelast years,Virtual Reality(VR)hasprovenitspotentialfor thevi-
sualizationandmanipulationofcomplexdatalike3-Dgeometriesgeneratedby
meansofCADapplications.Aninterestingandpromisingareaofapplicationfor
virtualenvironmentsisassembly simulation. Thoughvirtualenvironmentscan
alreadyprotably beintegrated into engineers' dailywork,there are stillalot
of disadvantages in existing VR-based assembly simulation systems, requiring
trainedengineers forperforminginteractivesimulations.Especially,the lackof
satisfactoryfeedbackmechanismsinexistingVR-basedsimulationenvironments
complicatesexecutionof interactivemanipulationsof virtualobjects.Indetail,
suitable modeling of user-objectinteractions, realistic simulationof object be-
haviour,and intuitive presentation of informationare missing.Duringthe last
2years,wehavebeendevelopingasoftwarecalledMAESTRO(MultimodalIn-
teractionTechniquesforAssemblySimulationinVirtualEnvironments),where,
forthersttime,thefollowingfeaturesarecombinedinasingle,comprehensive
tool:
{ Realistic,physically-basedbehaviourofvirtualobjects,includingautomatic
calculationoftheobjects'inherentmechanicalcharacteristics
{ Integrationofforcefeedbackintotheinteractionwithvirtualobjects,oper-
atingonauniquescenegraphforboth,graphicsandhaptics
{ Articial support mechanisms like sensitive polygons, virtual magnetism,
oftheMAESTROfeaturesin detail,butinsteadtogiveinsightintotheoverall
MAESTROfunctionality,and todemonstratebymeansofasystemevaluation
thatthecombinationofallfeaturescancontributesignicantlytoabetteruser
performance anduser acceptanceduring thesimulationof assembly tasksin a
virtualenvironment(VE).
The remainderof the paperwill start with a short surveyof existing VR-
basedassemblysimulationtools.Insection3,ourapproachiscomparedtoother
simulationsystems.Furthermorethephysically-basedmodelingandthehaptics
component of MAESTRO are briey described here. Section 4 illustrates the
developedarticialsupportmechanisms,andsection5givesanoverviewofthe
MAESTROhardwareand software. Finally, section 6evaluatesthe systemby
meansoftwoexperiments.Thepaperendssomeremarksaboutfuturework.
2 A Brief Survey of Assembly Simulation in VE
Basically,available systemsfor interactiveassembly simulationin virtualenvi-
ronments can besubdivided into knowledge/rule-based approaches and physi-
cally-basedapproaches.
R.Heger[1]introducedaknowledge-basedsystemthatallowsaninteractive
executionofmanualassemblytasksinavirtualenvironment.Thesystemworks
withtheconceptofreferencepoints,i.e.,singleverticeswithinthepolygonmod-
elswhichcanbeplacedatconnectionelements(e.g.,screws)ortargetpositions.
Forthesinglereferencepoints,assembly-specicobjectcharacteristicsarestored
and takecarethat objectsareplacedautomaticallyand exactlywhentheuser
approachessuchareferencepoint.
A similar approach was chosen by M. Grafe [2] for a virtual construction
system that uses basic elements. Functional nodes can be attached to single
surfacesoftheseelementscausing,e.g.,asnap-inwhentwocorrespondingnodes
areapproaching.
B. Jung [3] describes a knowledge-based system that allowsan interactive
assembly of virtualbasicelementsto complexmodules. Here,apolygonaland
alogicaldescription arestored in twoseparate knowledge databases.Therst
one contains object characteristics which are relevant to assembly, and which
arerepresentedbysocalledports.Thesecondoneservestodescribeaconstruc-
tion goal, i.e., akind of plan howto create modules from singleobjects. This
knowledge allows the system to recognize assembled objects as modules and,
by inferencefrom the knowledgedatabase, to deducethe specic useof single
elementswithin amodule.
All three knowledge-based systems are using a tracked instrumented glove
and3-Dmices asinteractiondevices anddonotprovideforce feedbackmecha-
nisms.IntheVADEsystemdevelopedbyS.Jayaram[4]andanassemblysystem
introducedbyM.Buck[5],bimanualinteractionisincludedbymeansoftwoin-
strumentedgloves.Bothsystemscontainaphysical,constraint-basedapproach
intersect,theobjectsaresimultaneouslypresentedtotheuserattheirphysically
plausiblepositionsand,inawireframerepresentation,attheirintersectedposi-
tions.Whilein thesystemofBuckreactionforcesarevisualizedasvectors,the
VADEsystemshouldprovidearealforce feedbackbymeansof anexoskeleton
glove.
R. Guptaet al. [6]introduce an assembly simulation systemthat provides
forcefeedbackbymeansoftwoPHANToMHapticDevices[7].Here,thegraph-
ical representation of the scene and the physically, constraint-basedmodeling
are completelyseparated. Incontrastto all other systemsmentionedhere,the
systemofGuptaisrestrictedtotwodimensionsanddoesnotsupporttheuseof
immersivedisplays.
3 The MAESTRO Concept
MAESTRO combines a physically-based and a knowledge-based approach in
order to prot from the advantages of bothstrategies, i.e., realisticobject be-
haviour on the one hand and eective manipulation of virtual objects on the
other hand. The inexibility of rule- and knowledge-based systems is avoided
here,becausemostof thenecessaryknowledgeis generatedautomatically, and
becauseonlyafewassembly-relevantrulesareapplied.Physically-basedmodel-
ing(PBM)andarticialsupportmechanisms,basedonprioriknowledgeabout
the assembly goal, are completed by haptic feedback functionality. Our ap-
proachesfor PBMand hapticsare only briey introduced in theremainder of
this section, whereasthe support mechanismswill be described in moredetail
in thenextsection.
Physically-Based Modeling Toachievea realistic behaviourof virtual ob-
jectsbymeansofphysically-basedmodeling,mass,frictioncoeÆcient,centreof
gravity,andinertiatensorofthevirtualobjectsmustbeknown.InMAESTRO,
these inherentmechanicalcharacteristicsarecalculatedautomatically fromthe
polygonaldescriptionofthemodelgeometry.Sincethegeometriescannormally
beeasilyexportedfrom aCAD application,MAESTROpossessesahigh exi-
bility.Inprinciple,theproblemcanbereducedtovolumedeterminationofthe
polygonalmodels.Here,wehaveimplementedandcompareddierentexactand
approximativealgorithms[8].
Starting from themechanical object characteristics calculated in step one,
we have, in contrast to other systems (see section 2), preferred an impulse-
basedapproachtoaconstraint-basedapproach,becauseitallowsforaconsistent
methodical treatment of all kinds of contact between virtual objects. Most of
thealgorithmswehaveintegratedhere,arewelldocumentedinthepublications
of B.Mirtich and D. Bara (see, e.g.,[9{11]). Whenever possible, reactionsto
collisionsarecalculatedalgebraicallyinordertofullltherealtimerequirements
asimple,rule-basedapproachwasfound.
Haptic Feedback A severedecitoftoday'ssystemsforinteractiveassembly
simulationisthelackofmultimodal,especiallyforcefeedback.Thefewexisting
systems likethose of Jayaram or Gupta (see section 2) operate with aredun-
dant haptic scene graphin order to provide stability of the haptic rendering.
In contrast to these systems, our approach is based on a single scene graph
for graphics aswell ashaptics,thus dramatically reducingmodelingcosts and
gainingahighersystemexibility. Inorder to achievestability evenwithouta
hapticdescriptionofthescene,weareoperatingwithaninterimrepresentation
of contact situations.Wewill decribethis approachin detail in aforthcoming
paper.
4 Articial Support Mechanisms
In order to compensate for the problems which inevitably exist in VR-based
assembly operations,arisingfrom thenon-exactmodeling ofgeometry and be-
haviour of virtual objects as well as from inadequacy of available interaction
devices, wedevelopedand integratedarticial support mechanismsinto MAE-
STRO.Figure1illustrateshowtheimplementedmechanisms{guidingsleeves,
sensitivepolygons, virtualmagnetism, and snapin { areassigned to the three
classicalphasesofanassemblyprocess.
Fig.1. Assignment of the integrated support mechanisms to the single interaction
prioriknowledgeaboutpossibletarget positions andorientationsof objects.In
thecongurationphaseofanassemblysimulationwithMAESTRO,theguiding
sleeves are generated in form of a scaled copy or a scaled bounding box of
the objects that are to be assembled, and then placed as invisible items at
thecorrespondingtargetlocation.Whenaninteractivelyguidedobjectcollides
with an adequate guiding sleeve, the motion path necessary to complete the
assemblytaskisanimatedasawireframerepresentation.Inaddition,thetarget
positionandorientationisvisualizedbymeansofasemi-transparentcopyofthe
manipulated object(seeFig.2).
Fig.2. Guiding sleeves supportthe user during the transport phase of anassembly
taskbymeansofawireframeanimationoftheassemblypath
SensitivePolygons Sensitivepolygonsarecreatedbeforesimulationstartand
positionedatadequatepositionsonobjectsurfaces.A functioncanbeassigned
to everysensitivepolygon, which is automaticallycalled when aninteractively
guided element collideswith thesensitive polygon. Wehave designedsensitive
polygonsprimarily to support theuser during thecoarse positioning phase of
an assembly task. Among others, the polygons can be used to constraint the
degrees of freedom for interactively guided objects. For instance, a sensitive
polygonpositioned attheportof aholecanrestrictthemotionofapintothe
axialdirectionoftheholeandthusconsiderablyfacilitatetheassemblytask.In
case of this classical peg in hole assembly task, MAESTRO creates a copy of
theguidedobject,orientatesthiscopyintothedirectionof thehole'saxis,and
switchestheoriginalobjectintoasemi-transparentrepresentation(seeFig.3).
Virtual Magnetism Inassembly procedures,itisoftennecessaryto position
objectsexactly,i.e.,paralleltoeachotheratarbitrarylocations,withoutleaving
anyspacebetweenthem. Sincein avirtualenvironmentthistaskis nearlyim-
possibletoaccomplishwithoutanyarticialsupportmechanisms,wedeveloped
theprincipleofvirtualmagnetism.TheleftpartofFig.4illustratestheeectof
virtualmagnetismonmovableobjects.Aswithsensitivepolygons,virtualmag-
netismisanessentialcomponentofthetheknowledge-basedmodelingofvirtual
objects'behaviour. In contrast to PBM and the other support mechanisms,a
Fig.4.Eectofvirtual magnetism(left), andforcecomponentsbasedonobjectdis-
tanceandobjectsurfaces
Beforesimulationstart,anobject'sinuence radiusis calculatedasafunc-
tion of its volume. Whenever two of such inuence areas overlap during the
simulation,anattractionforceisgeneratedthatconsistsoftwocomponents(see
Fig. 4). The rst component is calculated from the distance betweenthe two
objects'centers of gravity,and thesecond component isafunction ofpolygon
surfaces. Tocalculate thesecond component, asearch beamstarts from every
polygonmidpointintodirectionofthesurfacenormal.Ifsuchabeamintersects
with a polygon of the other object, a force directed to the surface normal is
created,whichisproportionaltothedistancebetweenthepolygons.Finally,the
sum ofthe to force componentsis applied to movethe smallerobjecttowards
thelargerone.
Snap-In AsallotherVR-basedassemblysimulationtools(seesection2),MAE-
STROprovidesasnap-inmechanismforthenalphaseofanassemblyprocess.
The snap-inis activated when position and orientation dierence betweenthe
guidedobjectandthetargetlocationfallsbelowaspecicthreshold.
5 The MAESTRO Prototype
Hardware Recently, we have accomplished a rst prototype of MAESTRO,
in which the force feedback system is a crucial component. We are using the
PHANToM 1.5fromSensableTechnolgies[7],allowingasixdegreesoffreedom
input andasimple force vectorasoutput. Due to therather small interaction
volumeof thePHANToM, we hadto choosean indirectinteraction paradigm,
where the pencil-like manipulator("gimbal") of thePHANToM isrepresented
data to and receivesforce data from the simulation computervia a 100MHz
FastEthernet.
Inprinciple,MAESTROcanworktogetherwithanyimmersivedisplaytech-
nology.IncombinationwiththePHANToM,atable-likedisplayismoresuitable
than a HMD or a fully immersive display like the CAVE. To accomplish the
evaluationofthesystem(seenextsection),wevisualizedthesceneonadisplay
consistingof ahorizontal andaverticalprojectionsurface("TAN HoloBench",
seeFig.5).
Tofacilitateinteraction, wehavefurthermoreintegrated speech recognition
thatallowstheusertopickupandreleasevirtualobjects,ortoactivatesupport
mechanismslikesnap-inorvirtualmagnetism,byafewsimplespeechcommands.
Sofar, MAESTROhas been implementedon aSun MicrosystemsE450 (2
Expert3D graphics boards) and on a SGI Onyx2 (1 Innite Reality graphics
pipe) workstation,bothcapableofdrivingaHoloBenchinstereomode.
4
5 1
2
3
6 7
2 6
9
8
Fig.5. Hardware setup of the MAESTRO assembly simulation system 1: Shutter
glasseswithreceiverofelectro-magnetictrackingsystem,2:Infraredemitterforshut-
tersynchronisation,3:PHANToMHapticDevice,4:Immersivedisplay"HoloBench",
5:PC withhaptics serverand speechrecognitionsoftware, 6:Stereoloudspeakers,7:
Microphoneforspeechrecognition8:Gimbal9:Graphicalrepresentationofthegimbal
withattachedvirtualobject
Software MAESTROisbasedonthecrossplatformVRsoftwareViSTA(Vir-
tualReality SoftwareUniversityof Technology Aachen[12]) and issubdivided
intothreeparallelprocesses:Thegraphicalprocessisresponsibleforprocessing
speechcommands,applyinginteractionmechanisms,andrenderingofthescene.
It is typically running at a frequency of 30 Hz. The PBM process processes
250Hz. Both processesrunondierentprocessorsof thesimulationcomputer
andcommunicatewith eachother viasharedmemory.Finally, thehapticspro-
cess runsonthePC atanupdate rateof1200 HzandcommunicatesviaUDP
protocolwiththesimulationcomputer.
6 Evaluation
InordertoevaluatetheMAESTROprototype,wehavecarriedoutexperiments
that are documentedin detailin [8]. Twoof these experimentsshall be intro-
duced and discussed here, with regard to eectiveness and user acceptance of
the articialsupport mechanisms,force feedback, and physically-basedmodel-
ing. 12 right-handed subjectsin theageof 24to 31participated at theexper-
iments. They were asked to complete the required manipulation tasks quickly
andaccurately.
6.1 Experiment1
The setup of experiment 1is shown in Fig. 6. Two large and two small nails
mustbebroughtintothecorrespondingholesofavirtualblock.Theexperiment
consistsof3phases:Inphase1,subjectshadtocompletethetaskina"native"
virtualenvironment,i.e.,basedonthe3-Dvisualizationofthescenewithoutany
furthermodelingorfeedbacktechniques.Inphases2and3theyweresupported
byguidingsleevesandsensitivepolygons,respectively.Everyphasewascarried
outtwice.Duringthersttrial,objectssnappedinasfarasthesubjectplaced
themintothetargetpositionbelowacertaintolerance.Thetrialendedwhenall
objectssnappedin,andthetimesubjectsneededforcompletionofthetaskwas
measured.Thesecondtrialwas interruptedautomaticallyafter axedamount
oftime, andthemismatchesbetweentheactualand therequiredobjecttarget
positionswere documented.
Results Theresultsofexperiment1showasignicanttimebenetwithguiding
sleeves in comparison to a native VE, and in turn a signicant benet when
using sensitive polygons instead of guiding sleeves.In trial 2, the positions of
the nails are signicantly more exact for guiding sleeves than for native VEs
(see Fig.7). The subjects' judgementof task diÆculty and quality of theuser
interface conrm thequantitativeresults(see Fig. 8). The snap-inmechanism
has great importance with regard to the judgement of task diÆculty. In case
thesnap-inisnotavailable,e.g.,becausethetargetpositionofanobjectisnot
known, subjects judge the manipulation task as morediÆcult. In such cases,
testpersonsregardthemechanismsofsensitivepolygonsandvirtualmagnetism
Fig.7.Quantitativeresultsof experiment1{ placingvirtualnailsintoablock.The
left diagram shows the average amount of time needed until snap-in, the right dia-
gramshows theaverage distanceto thetarget positionsixseconds afterstart of the
experiment.
Figure9depictsthesetupofexperiment2.Here,threevirtualelementsinformof
thelettersI,L,andT,havetobepositionedintoaframe.Again,theexperiment
consists of three phases. Inthe rst phase, subjects had to complete the task
in a native VE, where collisions between a letter and the frame are optically
signaled by a wireframe presentation of the letter. During the second phase,
thesubjectsweresupportedbyhapticfeedback.Finally,inthethirdphase,the
subjectswereaskedtomovethelettersabovetheirtargetpositionsandletthem
fallintotheframe.Thefallingprocedure,aswellasthebehaviouroftheobjects
whentouchingtheframe,havebeenphysicallymodeled.
Fig.9.Setupof experiment 2{start situation(left) of virtualletters(I, L, T), and
targetposition(right)inavirtualframe
Results A major result of experiment 2 is that the test persons had severe
diÆculties to complete the task in a native VE. They needed a dramatically
largeramountoftimethanwithsupportofhapticsorPBMtopositionletterI.
ForlettersLandT,itwasevenimpossibletocompletethetask(seeFig.10).An
interviewwiththesubjectsrevealedthat inanativeVE,theywerenotableto
comprehendwhyandwhereexactlyacollisionappeared.Asaconsequence,they
didnotchangetheirassembly strategyin awell directedmannerand, instead,
followed a trial and error principle, resulting in signicant longer interaction
times.
Fig.10.Quantitativeresultsofexperiment2{placingvirtualobjectsI,L,andTinto
sults.Both,hapticfeedbackaswellasPBM,gotapositiverating,althoughthe
testpersonsclearly preferredthe hapticsupporttocomplete thetask(seeFig.
11).Nearly allsubjectspraisedtheintuitivehapticrepresentationofcollisions.
Furthermore, they pointed out that the simulated gravitational forces had a
stabilizingeect ontotheir hand-arm-system.In adetailed judgement,haptics
andPBMwerenearlyidenticallyconsideredasimportant,realistic,helpful,and
non-distracting(seeFig.12).
Fig.11.AveragejudgementoftaskdiÆcultyanduserinterfaceforexperiment2
Fig.12.Averagejudgementofhapticfeedbackandphysically-basedmodeling(PBM)
7 Future Work
SinceMAESTROstartsfromthepolygonalrepresentationofascenewhichcan
easilybeexportedfrommostCAD systems,andsincemostparametersneeded
for objectbehaviour,haptics, andarticial supportmechanismsarecalculated
automatically before simulation start, MAESTRO is more exible than most
otherassemblysimulationsystems.Thisbenetofahighexibilitycomesalong
with the drawback, that MAESTRO canonly handle virtual scenes of rather
low complexity. Thecollisiondetection update rateis crucial to the quality of
physically-based modeling and the stability of haptics. Since in MAESTRO,
collisiondetectionandallmodelingarebasedonthepolygonalobjectrepresen-
tation,thesystemperformanceisverysensitivetothenumberofpolygonsinthe
scene.InordertomakeMAESTROapplicabletoassemblysimulationscenarios
thatarerelevantforindustrialapplications,weareworkingontheintegrationof
improvedcollisiondetection algorithmsandon acomprehensiveparallelization
oftheMAESTROsoftwarearchitecture.
TheMAESTROprototypemakesuseofaforcefeedbackdevicethatmerely
produces a force vector. However,for many manipulation tasks it is desirable
devicesareavailableandshouldbeintegratedintotheprototype,whichproduce
sixdegreesoffreedomforces.
Besidesgraphics and haptics, 3-Dacoustics could furthermoreimprovethe
realismofaninteractiveassemblyprocessandleadtoabetteruserperformance
andacceptance.Therefore,wejuststartedtoworkonanintegrationofbinaural
acousticstechnology into MAESTRO. Here,wefollowan innovative approach
thatisbasedononlytwoloudspeakers,producingastablebinauralsoundeven
whentheusermovesin frontoftheHoloBench.
A considerable extension of MAESTRO'sfunctionality is still necessary to
makeitattractiveforreal lifeindustrial applications. Future versionsof MAE-
STRO should allow bimanual interaction, the use of virtual tools like screw-
driversand wrenches,andprovideareal-time,realisticmodelingofdeformable
objects.Furthermore,itis highlydesirableto establishabidirectionalinterface
between MAESTROand product data managementsystems (PDMs)in order
to integrateVR-basedassemblysimulationintotheoverallproductionprocess.
References
1. Heger,R.:EntwicklungeinesSystemszurinteraktivenGestaltungundAuswertung
vonmanuellenMontagetatigkeitenindervirtuellenRealitat.Springer-Verlag,Berlin
(1998)
2. Gausemeier, J.,Grafe, M., Wortmann, R.:InteractivePlanning ofManufacturing
SystemswithVirtualConstructionSets.24thAnnualConferenceoftheIEEEIndus-
trialElectronicsSocietyIECON'98,Aachen(1998)2146{2151
3. Jung, B., Lenzmann,B., Wachsmuth,I.:InteraktiveMontage-Simulationmitwis-
sensbasierterGrak.ASIMMitteilungen49(1995)20{32
4. Jayaram, S., Jayaram, U., Wang, Y., Tirumali, H., Lyons, K., Hart, P.: VADE:
A VirtualAssembyDesignEnvironment.IEEEComputerGraphics&Applications
19(6)(1999)44{50
5. Buck, M.: Simulation interaktiv bewegter Objekte mit Hinderniskontakten.PhD
thesis,UniversitatdesSaarlandes(1999)
6. Gupta, R., Sheridan,T., Whitney,D.: Experiments onusingMultimodal Virtual
EnvironmentsforDesignforAssemblyAnalysis.Presence 6(3)(1997)318{338
7. Massie, T.H., Salisbury, J.:ThePHANToM HapticInterface:A Device for Prob-
ing VitualObjects. Symposium onHaptic Interfacesfor Virtual Environment and
TeleoperatorSystems,Chicago(1994)295{302
8. Stean, R.: Multimodale Interaktionstechniken fur die Simulation von Mon-
tagevorgangen in virtuellen Umgebungen.PhD thesis, AachenUniversity of Tech-
nology (2001)
9. Bara,D.:DynamicSimulationofNon-PenetratingRigidBodies.PhDthesis,Cor-
nell University(1992)
10. Bara,D.,Witkin,A.:Physically-BasedModeling:PrinciplesandPractice.ACM
SIGGRAPH1997CourseNotes,LosAngeles(1997)
11. Mirtich,B.:Impulse-BasedDynamicSimulationofRigidBodySystems.PhDthe-
sis, UniversityofCalifornia,Berkeley(1996)
12. vanReimersdahl,T., Kuhlen,T., Gerndt,A.,Henrichs,J.,Bischof,C.: ViSTA{
a multimodal, platform-independant VR-toolkit based on WTK, VTK, and MPI.
