Inhabited Virtual Heritage

(1)

Inhabited Virtual Heritage

Nadia Magnenat-Thalmann - U. of Geneva Alan Chalmers - U. of Bristol

Daniel Thalmann - EPFL

Synopsis

• Inhabited Virtual Cultural Heritage is a novel way of conservation, preservation and interpretation of cultural history. By simulating an ancient community within the virtual reconstructions of a habitat, the public can better grasp and understand the culture of that community.

The course will present the following concepts:

– Reconstruction technology – Computer Animation technology – Interaction technology

• Three case studies will be shown: the simulation of the Xian Terra Cotta Army, the representation of Geneva in 1602 and the reconstruction of Aya Sofia church in Turkey.

(2)

www.miralab.unige.ch

www.miralab.unige.ch University of GenevaUniversity of Geneva

MIRALab Presentation University of Geneva

Professor

Nadia Magnenat-Thalmann

Generating Animatable 3D Virtual Humans from

Photographs

Nadia Magnenat-Thalmann Won-Sook Lee

Jin Gu

MIRALab MIRALab

Where Research means Creativity Inhabited Virtual Heritage

Eurographics2001

Introduction

• Two techniques depending on the interest

– accuracy and precision of the obtained object model shapes,

• CAD systems, medical application.

– visual realism and speed for animation of the reconstructed models,

• internet applications

• Virtual Reality applications.

MIRALab MIRALab

Eurographics2001

Virtual humans for real-time applications

• What’s the components to consider?

– acquisition of human shape data – realistic high-resolution texture data

– functional information for animation of the human (both face and body)

(3)

State of the Art - Face

• Features on photographs (organized) and a generic model

– Modeling used for getting the individualized face using a few points

• [Kurihara 91] [Akimoto 93] [Ip 96]

– Modeling used for expression database

• [Pighin 98]

State of the Art - comparison

Problems for hairy parts

Better to catch non-characteristic points Difficult to catch non-characteristic points

Often noisy to catch characteristic points Easy to catch characteristic points

Usually low resolution of texture mapping Usually high resolution of texture mapping

Output: Numerous points Special equipment Very general equipment

Expensive Cheaper

Laser Scanner Photography

MIRALab MIRALab

Eurographics2001

Modification of a Generic Model Feature Detection Orthogonal

photographs Camera

with Feature Points

Texture Generation

Texture Fitting

Facial Animation

Generic model with animation

structure

Expression Database Other features detection

Key feature detection

Automatic Interaction Only once

DFFD coordinate calculation

Face Cloning

• Input

– photograph

– generic head & animation

• Method

– Feature based

• Output

– Animatable virtual human

MIRALab MIRALab

Eurographics2001

Head shapes from photos

• Features on front (and side) view

– eyes, nose, lips, hair and face outlines, etc.

• Semiautomatic structured feature detection

– piecewise affine mapping

– structured snake to keep structure of points

(4)

Head shapes from photos in 3D rather than in 2D

• Generation of ( x, y, z ) from ( x, y

^f

) and ( y

^s, z ) – criteria for giving more importance on the front view – robust even though the input photographs are not perfectly

orthogonal

• Dirichlet FFD (DFFD)

– the convex hull of a set of control points in general position

Head shapes from photos

• Feature points < control points

MIRALab MIRALab

Eurographics2001

Texture mapping

• Texture Generation

– One texture image from two images

• Geometrical deformation

• Multi-Resolution techniques

• Texture Mapping

– Projection to three planes – Transformation to several spaces

MIRALab MIRALab

Eurographics2001

Seamless texture mapping

• Texture generation

– Image deformation

Front Side

( right, left )

Deformed side ( right, left )

(5)

Seamless texture mapping

• Texture generation

– Multiresolution image mosaic

• Rotation in 360 degree

Results

MIRALab MIRALab

Eurographics2001

Results

• Several ethnic group from one generic model

MIRALab MIRALab

Eurographics2001

Results - shape texture separation

Photograph set for shape

Features for image Features for

shape

Photograph set for image

Modification of the

generic model Texture Mapping Modification of a generic model

Animation in a Virtual World

(6)

• Visual comparison

• 3D- distance measurement : 2.84306 %

Results - Validation

Face Front photo

Face Side photo

Body Side photo Body Front

photo

Body Back photo

Feature points (x, y, z) Feature points

(x, y, z)

Front view rough shape

Back view rough shape Automatic edge

detection

Front view fine shape

Back view fine shape

Integrated virtual human (VRML H-anim)

Back view skeleton

interactive automatic

(x,y) (y, z) (x,y)

Front view texture mapping

Back view texture mapping Animatable

face

Animatable body

input data Front view

skeleton Face Cloning Generic

face and body (skeleton,

skin)

texture blending

Posture correction

Body Cloning

• Input

– three photographs – H-Anim 1.1 generic body

• Feature - edge based

• Output

– animatable vitual human

MIRALab MIRALab

Eurographics2001

Body Cloning - Generic body

• Continuous mesh humanoids

– MPEG-4 compatible H-Anim

1.1 formats [http:H-Anim]

– 94 skeleton joints & 12 skin parts (different from the face with only skin)

MIRALab MIRALab

Eurographics2001

Body Cloning - Generic body

• H-Anim joints related to skin parts

– the local coordinates of the skin part i to global coordinate by 4x4 matrix M_i.

Skullbase (head) vc4 (neck) l_shoulder (left_upper_arm)

l_elbow (left_lower_arm)

Sacroiliac (hip) l_hip (left_upper_leg)

l_wrist (left_hand) l_knee (left_lower_leg)

l_ankle (left_foot) vl5 (front_torso, back_torso)

(7)

Body Cloning - Generic body

• Skin has grid structure

– each skin part has several slices

– each slice on the skin part has the same number of points – Share the same 3D coordinates between different skin part

• Resulting seamlessly continuous skin envelope

Body Features and skeleton

• Features and skeleton adjust

Feature points on images

Modify the movable skeleton joints

Modify other skeleton joints

MIRALab MIRALab

Eurographics2001

Body rough skin adjustment

• Feature points -> Control points -> skin modification

Left most Right most

Left most

Right most Front most

Back most

Front most Back most Right most

Left most

Right most

Left most Right most

Front most Front most

Front most

Up most (end-shouler pt)

Left most

Down most (Armpit pt) Up most (mid-shoulder pt)

Right most Down most

Top-slice (Arm hole) Shoulder-slice

Bottom-slice (d)

(a) (b)

(c)

MIRALab MIRALab

Eurographics2001

• Feature driven edge extraction

• Canny edge detector

• Each feature segment indicates the vicinity and approximate direction of the boundary to be found

• evaluate the “goodness” of the potential connection

Body fine skin adjustment

(8)

• Front and side views are used

– Deform body and texture for each side separately

• Texture blending

– Problem caused by digitization and illumination

– Linear blending following corresponding edges on the front and back views

Body Cloning - Texture mapping

Body and Face together

• Automatic connection with own face from face cloning system

– use features on face and body

• Neck adjustment

– bridge to connect the face and body smoothly and seamlessly

MIRALab MIRALab

Eurographics2001

Body Results

• H-Anim 1.1 format

– visualized by web browsers – Animatable

MIRALab MIRALab

Eurographics2001

Body Cloning - Results

• Sometimes postcorrection needed

– Skeleton correction from skin envolope

• Elbow skeleton correction

– H-Anim & Vicon (optical motion capture system) posture

• length and angle coordinate

• adjust angles for arms and legs

(9)

• Animation with cloned body

– Comparison with real motion

Animation result with motion capture

Conclusion

• Easy input like photographs is the first priority to build the system

• A complete integration of whole face and body parts from five photographs

• Continuous mesh for generic body

– real-time animation without texture problems

MIRALab MIRALab

Eurographics2001

Conclusion

• Several problems are solved

– efficient and robust semi-automatic feature detection method – 3D-deformation approaches rather than in 2D resulting error

resistance for input images

– more robust 3D deformation using DFFD

– fully automatic generation of seamless texture mapping

MIRALab MIRALab

Eurographics2001

Measurement based body creation

Nadia Magnenat-Thalmann

HyeWon Seo

(10)

Overview

Initial model

∑ H-Anim skeleton

∑ Skin mesh

Skeleton deformation

Volume deformation

Surface optimization

H-Anim exportation Final body model in H-Anim

Measurer Body measurement

data Skin

Attachment

Initial model

HumanoidRoot : sacrum

sacroiliac : pelvis

l_hip: l_thigh l_knee: l_calf l_ankle:

l_hindfoot l_subtalar : l_midproxima l l_midtarsal : l_middistal l_metatarsal : l_forefoot

l_hip: l_thigh l_knee: l_calf l_ankle:

l_hindfoot l_subtalar : l_midproxima l l_midtarsal : l_middistal l_metatarsal : l_forefoot

vl5 : l5 vl3 : l3 vl1 : l1 vt10 : t10 vt6 : t6 vt1 : t1 vc4 : c4 vc2 : c2 skullbase : skull vl5 : l5 vl3 : l3 vl1 : l1 vt10 : t10 vt6 : t6 vt1 : t1 vc4 : c4 vc2 : c2 skullbase : skull l_sternoclavicular : l_clavicle l_acromioclavicular : l_scapula

l_elbow : l_forearm l_shoulder : l_upperarm

l_wrist : l_hand l_sternoclavicular : l_clavicle l_acromioclavicular : l_scapula

l_elbow : l_forearm l_shoulder : l_upperarm

l_wrist : l_hand r_hip: r_thigh

r_knee: r_calf r_ankle:

r_hindfoot r_subtalar : r_midproximal r_midtarsal : r_middistal r_metatarsal : r_forefoot

r_hip: r_thigh r_knee: r_calf r_ankle:

r_hindfoot r_subtalar : r_midproximal r_midtarsal : r_middistal r_metatarsal : r_forefoot

r_sternoclavicular : r_clavicle r_acromioclavicular : r_scapula

r_elbow : r_forearm r_shoulder : r_upperarm

r_wrist : r_hand r_sternoclavicular : r_clavicle r_acromioclavicular : r_scapula

r_elbow : r_forearm r_shoulder : r_upperarm

r_wrist : r_hand

MIRALab MIRALab

Eurographics2001

Skeletal deformation

current desired

T

= T s

Skin attachment to bones Skeletal deformation

Scale factor

MIRALab MIRALab

Eurographics2001

Volumetric deformation – breast example

= +

+

• Breast

– Grid structure (20 x 23).

– Parametric curves for preserving the round

aspect.

(11)

Volumetric deformation – other parts

• Waist

• Similar to the breast but with the use of simpler(Bézier) curve.

• Hips

• Deformation based on

FFD(Free Form Deformation).

90 cm 82 cm

Volumetric deformation – results

63 cm 85 cm

80 cm

68 cm

MIRALab MIRALab

Eurographics2001

Facial Animation From Facial Mesh to Expressive Talking Faces

Nadia Magnenat-Thalmann Sumedha Kshirsagar

MIRALab MIRALab

Eurographics2001

Overview

• Hierarchy in Facial Animation

• Definition of Static Expressions

• From Expressions to Animation

• Speech Animation Overview

(12)

Hierarchy in Facial Animation

• Static Expressions : Deformation of this mesh controlled by parameters

• Animation : Varying the static expressions with time

• Face Object : Collection of mesh vertices and topology

Defining Static Expressions

Need of Parameterization to define static expressions MPEG-4 Facial Animation Parameters

Feature Points defined on the Specific locations of the face Animation defined by the displacements of these Feature Points

MIRALab MIRALab

Eurographics2001

Designing Facial Expressions

Facial Animation Parameters divided into three groups

•Lips : Lower inner midlip, Stretch corner lip etc.

•Eyes : Close right eyelid, Raise left eyebrow etc.

•Other : Puff right cheek, Roll head etc.

MIRALab MIRALab

Eurographics2001

Different Time Envelopes for Expressions

Simple (Triangular)

Attack-Decay- Sustain-Release

Multipoint Articulation

Quick Transition

Spline Interpolation Linear Interpolation

(13)

Building Animations

Possibility to add different expression envelopes at different time instants

Different animation tracks enables the designer to design head movements, facial expressions, eyebrow movements independently

From Speech to Animation

Temporized phonemes

Audio signal

Facial animation parameters

(MPEG-4 FAP)

Synchronization

Animatable face model Speech -

synthetic/real

MIRALab MIRALab

Eurographics2001

From Natural Speech to Visemes

Speech signal

Extracting parameters from speech that are related to mouth shapes

Speech Processing Parameters : LPC, pitch, zero crossing

Reference Parameter Database

Estimation of Phonemes Parameters

MPEG-4 Mapping from phonemes to visemes

MIRALab MIRALab

Eurographics2001

Mechanical Simulation of Deformable Surfaces

for Animation of Synthetic Garments

Nadia Magnenat-Thalmann Pascal Volino

Marlène Arévalo

(14)

Cloth Simulation Techniques

• Geometrical Models

– Reproduction of the geometrical deformations of the cloth.

• Mechanical Models

– Simulation of the cloth deformations using equations derived from the mechanical behavior of fabrics.

MIRALab History

• Lafleur, Thalmann, 1991:

– Simple viscoelastic surfaces using Lagrange equations.

• Carignan, Yang, Werner, Thalmann, 1991-92-93:

– Modified Terzopoulos model with octree collision detection and avanced pattern-

seaming garment design.

MIRALab MIRALab

Eurographics2001

MIRALab History

• Volino, Courschesne, Thalmann, 1995-96:

– Viscoelastic surfaces simulated with particle systems and constraint based collision response.

• Volino, Thalmann, 1997-98:

– Fast and optimized spring mass model computed with Runge- Kutta integration and new design tools for creating garments.

MIRALab MIRALab

Eurographics2001

MIRALab History

• Volino, Thalmann, 2000-01:

– Fast and accurate model

simulating dynamically

complete viscoelasticity

parameters using advanced

implicit integration methods.

(15)

Mechanical Parameters

• Internal Forces (From surface deformations)

– Elasticity (metric, curvature).

– Viscosity (internal dissipation).

– Plasticity (behavior curve hysteresis).

• External Forces (From environment interaction)

– Gravity, Aerodynamic effects.

– Contact reaction, Friction.

– Miscellaneous external interactions.

Parameter Measurements

• Kawabata Evaluation System

– Normalized procedure and equipment for measuring elasticity parameters.

MIRALab MIRALab

Eurographics2001

Parameter Modeling

• Spring-Mass Systems

– Discrete representation of the surface as a mesh of punctual masses, parameters represented as springs creating

viscoelastic forces between them.

MIRALab MIRALab

Eurographics2001

Parameter Modeling

• Spring-Mass Systems

– Simple to implement.

– Flexible for adaptation to geometrical constraints.

– Inaccurate representation of parameters (surface anisotropy and bending).

– Mainly used in fast simulation models for

computer graphics.

(16)

Parameter Modeling

• Continuum Mechanics

– Expression of the surface energy and

forces exerted on surface elements derived from surface deformation (Lagrange

equations), and integration using finite difference discretization.

Parameter Modeling

• Continuum Mechanics

– Accurate Modeling of material properties.

– Complex implementation.

– Slow computation.

– Difficulties for integrating nonlinear models and geometrical constraints.

– Mainly used for precise computation of simple and situations (Draping).

MIRALab MIRALab

Eurographics2001

Parameter Modeling

Cloth Surface

Discrete Representation

∫ ^{f(x) dx}

Continuum Mechanical Model

∑ ^{f(x) ∆x}

Discrete Mechanical Model

Particle Systems

Continuum Mechanics

MIRALab MIRALab

Eurographics2001

Parameter Modeling

• Finite Elements

– Particular formulation of continuum mechanics model where high-order

elements are used to represent accurately deformations with adequate degrees of freedom and advanced energy

minimization techniques compute the

actual system evolution.

(17)

MIRALab Model

• First-Order Finite Element

representation integrated using state-of the art particle systems methods.

– Combines the advantage of accurate

parameter representation with the flexibility of particle systems (choice of integration methods and collision response

integration).

MIRALab Model

• First-Order Finite Elements

– Degrees of freedom = Mesh vertex positions and speeds.

– Accurate representation of metric elasticity (Anisotropic Weft-Warp and Shear

elasticity curves, Poisson coefficient, viscosity curves) within elements.

– Additional inter-element equations for modeling Weft-Warp bending forces.

MIRALab MIRALab

Eurographics2001

MIRALab Model

• Integration Methods

– Explicit Runge-Kutta integration

• Slow and precise high-order integration that ensures high accuracy level through controlled numerical error evaluation.

– Implicit Euler and Midpoint integration

• Fast and efficient integration that allows controlled approximations to highly speed up computations without instability problems related to explicit methods.

MIRALab MIRALab

Eurographics2001

MIRALab Model

• Efficient and Accurate Simulations

– Accurate evaluations of energy evolutions of the cloth during animations.

Energy Evolution (Undamped)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

1 2 6 5 1 7 6 1 0 1 1 2 6 1 5 1 1 7 6 2 0 1 2 2 6 2 5 1 2 7 6 3 0 1 3 2 6 3 5 1

Time (1/50 s)

Energy (J)

E Bending E Planar E Kinetic E Gravity

(18)

MIRALab Model

– Accurate representation of internal viscosity and damping parameters.

• Important for producing realistic animations, not only draping on static bodies.

– Accurate representation of collision reaction and friction.

• Allows garments to be maintained on the animated body mechanically through their own friction, without artificial “attachment points”.

Collision Detection

• Numerical Complexity

– Arises from the high number of polygons that the object meshes have (cloth and body, several thousands of polygons), and how to extract the colliding polygons quickly.

MIRALab MIRALab

Eurographics2001

Collision Detection

• Detection Techniques

– Space subdivision: Only detect collision of objects sharing a same space region.

• Grid subdivision (voxels).

• Hierarchical subdivision (octree).

– Object subdivision: Subdivide the object into geometrically localized sub-objects.

• Hierarchical bounding-box subdivision.

MIRALab MIRALab

Eurographics2001

Collision Detection

• Detection Techniques

– Space subdivision: Mostly used when dealing with numerous independent objects.

– Object subdivision: Efficient when a constant structure can be identified between the colliding elements.

• Adapted for the detecting collisions between

mesh elements of a deformable cloth.

(19)

Self-Collision Detection

• Self-Collision Adjacency Problem

– Avoid detection of

“colliding” adjacent

polygons though inclusion of curvature evaluation.

• No self-collisions occur within a region with not enough curvature.

Self-Collision Detection

Detection Between No Detection Between

Detection Within

No Detection Between No Detection Within

Detection Between

Self-Collisions

Inter-Collisions (Adjacent)

Inter-Collisions (Non-Adjacent)

– Detection Within and Between Regions

• Use of “curvature boxes” within regions and between adjacent regions, regular bounding boxes between non adjacent regions.

MIRALab MIRALab

Eurographics2001

Self-Collision Detection

– Efficiency of self-collision detection is not the limiting factor of detection anymore.

• Detection focused only in colliding regions.

MIRALab MIRALab

Eurographics2001

Collision Response

– Collision effect distributed on the vertices of the colliding mesh elements using

mechanical momentum conservation laws.

Pc+ddddPc Pc ddddPc

(20)

Collision Response

– Geometric constraint

enforcement using combined correction of system state.

• Position Correction: Obtaining desired position at current frame.

• Speed Correction: Obtaining desired position at next frame.

• Acceleration Correction: Obtaining desired position and speed at two next frames.

State

t t+dt t+2dt Time

Corrected acceleration Goal state

t t+dt Time

State

Corrected speed Goal state

t Time

State

Corrected position Goal state

Designing Garments

• 3D Pattern Assembly Using Simulation

MIRALab MIRALab

Eurographics2001

Animating Garments

• Mechanical Computation on Animated Body.

MIRALab MIRALab

Eurographics2001

Virtual Fashion Design

(21)

Creative Simulation

The Terra-Cotta Soldiers

Nadia Magnenat-Thalmann Marlène Arévalo

Gaël Sannier

MIRALab MIRALab

Eurographics2001

The Xian Project

• Excavation of the grave complex of the Ch'in emperor Shi Huang Ti in Xian in the 1970s has revealed a field of statues depicting soldiers, servants, and horses, estimated to total 6’000 pieces. The figures were modeled after the emperor's real army, and each face is different.

• The Xian project in 1997 is intended to recreate and give again life to this army using computer-generated techniques.

Discovery of the statues

MIRALab MIRALab

Eurographics2001

Sculpting the Soldiers' Faces (I)

• The real soldier faces are all different and have details.

• We use a method similar to the modeling of clay; It consists of adding or eliminating parts of the material, and turning around the object.

• The steps of the first head modeling (I):

– We apply scaling deformations on a sphere to obtain an egg shape aspect.

– We move regions selected with triangles & also lift or move vertices.

– We split in half in order to work more efficiently.

Creation of a soldier head from a sphere (I)

(22)

Sculpting the Soldiers' Faces (II)

• The steps of the modeling (II):

– We model specific regions (nose, jaws, eyes, etc) by sculpting and pushing back and forth vertices and regions.

– We obtain an half face of the soldier to which we apply a reversed scaling on X axis to produce the other half.

– The two sides are merged together which finally give us our first soldier's face.

Creation of a soldier head from a sphere (II)

Texture-fitting (I)

• To increase realism, we apply texture fitting to objects. We map a picture onto the object, in a way that allows the user to specify some matching points between the texture and the object:

– We can see the texture while fitting it to the object.

– Some interesting vertices are selected, suitable for circumscribe the area and fitting the texture to some specific features of the model. All these marked vertices are projected to the texture image.

– We move each projected vertex to its right position on the 2D texture. The 3D object is mapped in real-time in the 3D window using the information given by the position of these marked vertices on the texture image.

Adjusting features upon the texture image

Result of the fitting in real-time in 3D

MIRALab MIRALab

Eurographics2001

Texture-fitting (II)

• As we only have a single photo of each soldier face to model from, we create a global texture using this photo, so that this texture can be mapped around the whole head.

1- Photo of a real soldier 2- Texture image 3- 3D model 1- 2- 3-

MIRALab MIRALab

Eurographics2001

Creating the Soldier Bodies

• Our goal is to make realistic and efficient human modeling and deformation capabilities for many different bodies. So we use the metaball technique as it is inherent to interactive design.

• The metaballs hierarchy is taken from a standard model we have, we then modify the metaballs positions and shapes to fit soldiers anatomy.

• The head, hands and feet are attached to our body envelope.

(23)

• Scenario:

– We see first a scene with the 3D terra-cotta soldiers inside the earth.

– It is dark with a starry sky.

– The day is coming so more and more light is appearing. This suddenly awakes one terra cotta soldier. He is extremely astonished to see the scene around himself…

The Film (I)

The Film (II)

– He notices the presence of a soldier near him and also his head which is on the ground. He took the head and put it on the next soldier's body…

– This latter start to live again. They look at each other, and all the army is slowly coming to life. They start to walk again, but the first soldiers decide to let them go...

MIRALab MIRALab

Eurographics2001

Flashback to the Future

Nadia Magnenat-Thalmann Marlène Arévalo

MIRALab MIRALab

Eurographics2001

The Project

• A virtual reality experience developed in the MIRALab research laboratories of the University of Geneva. This real-time adventure, with 3D glasses, has been experienced at Palexpo in October l999, during Telecom’99.

(24)

The Project

• To illustrate telecommunications, the show communicates in real time with three distant booths, one located in Palexpo, the second one in the Uni Dufour Hall and the third one at the Geneva Airport.

Booth at Palexpo

The Project

Booths at the University and at the Airport

MIRALab MIRALab

Eurographics2001

The Project

• Real people are being cloned, and their virtual counterparts take part in 3D scenes from the past and the future.

• To do the virtual double of each person, we use a procedure based on two photographs, that can reconstruct the faces of individuals in 3D.

Face Cloning

MIRALab MIRALab

Eurographics2001

The Project

• This world première illustrates the face-to-face interaction within the virtual scene of individuals who in reality are situated at a distance from each other, like you and I.

• It is also a first for the reconstruction of the Vieille Ville by computer and for the appearance of a virtual Mère Royaume.

The Vieille Ville of Geneva in real The Vieille Ville of Geneva in virtual

(25)

1602

• Escalade: soldiers from Haute Savoie tried to invade Geneva and were stopped by the Geneva inhabitants and more particularly the “Mere Royaume”, who spilled the content of her cauldron over the invaders.

The Mère Royaume and 2 soldiers

1602: The Mère Royaume

MIRALab MIRALab

Eurographics2001

Nadia Magnenat-Thalmann Allessandro Foni Grégoire L’Hoste Georgios Papagiannakis

The making of the SS.

Sergius and Bacchus edifice

MIRALab MIRALab

Eurographics2001

The CAHRISMA project (I)

• Main objective of the CAHRISMA project (Conservation of the Acoustical Heritage by the Revival and Identification of the Sinans Mosques) is to innovate the concept of hybrid architectural heritage.

• Hybrid architectural heritage is a new way of identification that covers acoustical characteristics besides visual peculiarities.

• It states that, for the spaces, having acoustical importance, architectural heritage concept should be upgraded covering acoustical and visual properties. The effects of this improvement will reflect to actual implementation of conservation and

restoration.

(26)

The CAHRISMA project (II)

• MIRALab's involvement:

– Real-time visualisation of selected spaces.

– Creation of people (virtual bodies, faces and cloth textures).

– Animation of virtual humans.

– Integration of visual and acoustical models into a virtual 3D interactive system.

• One of the monuments selected for this project is SS. Sergius and Bacchus edifice in Istanbul.

SS. Sergius and Bacchus church

• The church of the SS. Sergius and Bacchus, a landmark in Byzantine ecclesiastical architecture, was founded by Justinian probably in 527, the first year of his reign.

• The church of the SS. Sergius and Bacchus known to this day as “the Little Hagia Sophia”, because the general principles of its architecture are comparable with those of the Great Church.

• Sometime between 1506 and 1512, the church of the SS. Sergius and Bacchus was converted into a mosque. The atrium was replaced by a peristyle, surviving to this day, and a courtyard where the medrese (religious school) stands today.

MIRALab MIRALab

Eurographics2001

Reconstruction of the edifice 3D model (I)

• The 3D model of the SS. Sergius and Bacchus edifice is reconstructed from the available architectural plans and the visual data resulted from the data collection process performed by UNIGE and EPFL teams.

MIRALab MIRALab

Eurographics2001

Reconstruction of the edifice 3D model (II)

• The whole edifice is reconstructed in three dimensions using polygonal method of 3D Studio Max software.

• During the modelling phase special consideration are taken to keep the number of polygons as low as possible, so that the final model would be optimised for real-time visualisation.

View of the mesh model from 3D Studio Max

(27)

Texturing the 3D model

• The texture are created from 2D photographs, they are used as texture image maps to improve the visual details of the 3D model. A special care is taken to correct for the perspective of the picture and to enhance the aspect of the texture.

Actual picture Texture extracted from the picture Textured 3D model

Lighting the 3D model

• The lighting of the 3D model is done with Lightscape software, as it allows for realistic lighting effects.

• The techniques used are physical based model of global illumination, such as radiosity and ray-tracing.

Distribution of light on the surfaces of the 3D model

MIRALab MIRALab

Eurographics2001

Use of light maps for realistic visualisation

• The creation and use of light-maps, from the lights generated in Lightscape, allows the real-time visualisation of the realistic lighting.

Light-maps applied on the 3D model

MIRALab MIRALab

Eurographics2001

Visualisation of the 3D model

• Both textured and light-mapped models are exported in VRML and merged together for real-time visualisation on MIRALab’s real-time rendering engine or on the World Wide Web.

Textures

Light-maps Final model of the edifice

(28)

The results (I)

The results (II)

MIRALab MIRALab

Eurographics2001

• Virtual humans are also modelled using the polygon method.

The clothes of the model are realised with MIRALab cloth plug-in, according to picture of ancient time people.

• Virtual human are then converted to h-anim standard format and animated with Vicon motion capture data.

Creation of virtual humans

Historical model Patterns of the clothes

MIRALab MIRALab

Eurographics2001

Real-time visualisation of the 3D model

• Both model of the edifice and of the virtual human are loaded in

MIRALab’s real-time rendering engine. User can walk inside the

3D model and examine it interactively.

(29)

2000/M.GrossandF.R.A.Hopgoodditors) Volume19(2000),Number3

Generating Animatable 3 D V irtual Humans fr om Photographs

WonSook.Lee,JinGu,andNadiaMagnenat-Thalmann

MIRALab,CUI,UniversityofGeneva,SwitzerlandWeb:http://www.miralab.unige.chE-mail:{wslee,gu,thalmann}@cui.unige.ch

stract

epresentaneasy,practicalandefficientfullbodycloningmethodology.Thissystemutilizesphotostakenfromthefront,sideandbackofapersoninanygivenimagingenvironmentwithoutrequiringaspecialbackgroundorcontrolledilluminatingcondition.AseamlessgenericbodyspecifiedintheVRMLH-Anim1.1formatisusedtogenerateanindividualizedvirtualhuman.Thesystemiscomposedoftwomajorcomponents:face-cloningandcloning.Theface-cloningcomponentusesfeaturepointsonfrontandsideimagesandthenappliesDFFDforshapemodification.Nextafullyautomaticseamlesstexturemappingisgeneratedfor360 ocoloringonapolygonalmodel.Thebody-cloningcomponenthastwosteps:(i)featurepointsspecification,whichenablesomaticsilhouettedetectioninanarbitrarybackground(ii)two-stagebodymodificationbyusingfeaturepointsbodysilhouetterespectively.Thefinalintegratedhumanmodelhasphoto-realisticanimatableface,hands,feetandbody.TheresultcanbevisualizedinanyVRMLcompliantbrowser.

oduction

years,modelingvirtualhumanbodyhasattractedmoreattentionfromboththeresearchandin-community.Itisnolongerfantasytoimaginethatseeherself/himselfinavirtualenvironmentmov-ingandinteractingwithothervirtualfiguresorevenhumans.Byadvancesinalgorithmsandnewdevel-sinthesupportinghardwarethisfantasyhasbecome.

issuesinvolvedinmodelingavirtualhumanmodelllows:

ofhumanfaceandbodyshapedatahigh-resolutiontextureionalinformationforanimationofthehumanface

esshowtoacquireananimatablehumanbodyrealisticappearance.Itisourgoaltodevelopatech-atenablesaneasyacquisitionoftheavatarmodeltheabilitytobeanimatedwellandproducedatalowearetwobasictypesoftechniquesforobtainingmodels,accordingtothedifferentrequirements forthemodels.Thefirsttypeoftechniquefocusesontheac-curacyandprecisionoftheobtainedobjectmodels,suchasthoseusedinCADsystemsandindustrialapplications.Thesecondtypeoftechniquesconcentratesontheshapeandvi-sualrealismofthereconstructedmodels,suchasthoseusedinvirtualrealityapplications.

Whenwehavetoplaceimportanceontheaccuracyoftheshape,therearevariousapproachestothereconstructionofafaceeitherusingasculptor8,alaserscanner22,astereoscopiccamera21,anactivelightstripper24,videostream16;9.Inre-centyears,bodycloninghasalsobecomeanincreasinglyhottopic.Similarly,therearemanymethodsthatconcernpre-cisionandaccuracy16;11;1;7;10;13.Generally,thesesystemsareeitherexpensiveorrequireexpertiseknowledgeinusingthemandneedaspecialenvironmentsetting.Thus,mostofthemhavelimitationswhencomparedpracticallytoacom-mercialproduct(suchasacamera)fortheinputofdataforreconstructionandfinallyanimation.

Ontheotherhand,systemsusingthesecondtypeoftech-niquesaremuchcheaperandeasiertouse.Thesetechniquesareusuallymodel-based.Thereareseveralapproachesto

ographicsAssociationandBlackwellPublishers2000.PublishedbyBlackwell

108CowleyRoad,OxfordOX41JF,UKand350MainStreet,Malden,MA

(30)

Lee,GuandThalmann/AnimatableVirtualHumansfromPhotographs ctionofeitheraface2;15;17;20orabody12fromTheseapproachesconcernmainlytheindivid-eandvisualrealismusingahighqualityim-Forexample,Hiltonetal.12proposedamethodvirtualpeople.Agenerichumanmodelistakenionextractedfromphotosisusedtomodifythe.Theapproachissimpleandefficient.How-hoddoesnotgiveagoodreconstructionandfortheface.Inaddition,theirgenericmodeliswhichmeanstheregionsaroundcertainskele-donothavethesmoothlyconnectedsurface,sotexturedmodelhassomemismatchingprob-weanimatethejoints.Italsolackstheflexibilitytheimagingenvironmentsinceitrequiresaspe-edbackgroundandproperlycontrolledlightingaretaken.

oach,whichbelongstothesecondtype,ad-followingquestionsandsuggeststhesolutions.

producefromwhat?

arealisticandanimatablewholebodyinclud-handsandbodyfromphotodata.Everybodyhlyconnectedandtextured.Photographsofbodycannotprovidesufficientfacialinformationconstructagoodfacemodelandfurtherfacialan-erefore,wetaketwoadditionalphotosthatfocusonly,besidesthethreewholebodyphotos.

istheenvironmenttogettheinput?

simplesnapshotswithcommercialcameraswith-environment.Insteadofseekingasolutionecialenvironment,weprovideauser-friendlyin-challowsnon-expertusertointeractivelyhinttocertainimportantinformationaboutthehumanway,withalittleamountofuserinteraction,wereflexibilityinusingthesystem.

tomaticistheprocessingforusers?

ideanautomaticsystemexceptforafewinterac-beginningasshowninFigure1.

hcanweanimate?

vidualizedvirtualhumaninheritesthefunctionalthegenerichumanwithanimationcapacityandbody.

isittovisualizewithotherapplications?

MLHumanoidAnimationWorkingGroup(H-stsforthemainpurposeofcreatingastandardesentationforhumanoid.Ourgenericbodyisinim1.1format14andtheresultingbodycanbebywebbrowsers,suchasNetscapeandanimatedprogram.

eofthealgorithmisshowninFigure1.Section Face Front

photo Face Side

photo

Rough feature points(x, y, z)

Face shape(DFFD) Fine feature points(x, y, z) Body Side

photo Body Front photo Body Back

photo

(x,y)(y, z)

Feature points(x, y, z) Feature points(x, y, z) Front viewrough shape Back viewrough shape Automatic edge

detection

Front view fine

shape Back view fine

shape

Integrated virtualhuman

(VRML H-anim) Back viewskeleton

interactiveautomatic (x,y)(y, z)(x,y)

Texture generationand fitting

Front view

texture mapping Back view

texture mapping

AnimatablefaceAnimatablebody Generic faceand body(skeleton,

skin)

input data Front viewskeleton

Figure1:Overflowoffaceandbodycloning

2isdevotedtotheface-cloningprogramwhileSection3ex-plainsthebodycloning.TheresultsareshowninSection4andareconcludedinSection5.

2.Facecloning

2.1.ShapemodelingInthissection,wepresentawaytoreconstructaphoto-realisticheadforanimationfromorthogonalpictures.First,weprepareagenericheadwithananimationstructureandtwoorthogonalpicturesofthefrontandsideviews.Thegenericheadhasefficienttriangulation,withfinertrianglesoverthehighlycurvedand/orhighlyarticulatedregionsofthefaceandlargertriangleselsewhere.Italsoincludeseye-ballsandteeth.

Themainideatogetanindividualizedhead,istodetectfeaturepoints(eyes,nose,lips,andsoon)onthetwoim-agesandthenobtainthe3Dpositionofthefeaturepointstomodifyagenericheadusingageometricaldeformation.Thefeaturedetectionisprocessedinasemi-automaticway.Theusersetsaveryfewfeaturepoints(keypoints)andtheotherfeaturepointsarefittedusingapiecewiseaffinetrans-formationfirstandthensnakemethods.Thestructuresnakemethodwithsomeanchorfunctionalityisdescribedinan-otherpaper19.Then,two2Dpositioncoordinatesinthefrontandsideviews,whicharetheXYandtheZYplanes,arecom-binedtobea3Dpoint.Afterusingaglobaltransformationtomovethe3Dfeaturepointstothespaceforagenerichead,DirichletFreeFormDeformations(DFFD)23areusedtogetnewgeometricalcoordinatesofagenericheadadaptingtothedetectedfeaturepoints.ThecontrolpointsfortheDFFD

c

TheEurographicsAssociationandBlackwellPublishers2000.

(31)

Lee,GuandThalmann/AnimatableVirtualHumansfromPhotographs

2:(a)normalizationandfeatures.(b)Modificationcheadwithfeaturepoints

pointsdetectedfromtheimages.Thentheshapeseyesandteethareseparatelyadaptedtothenewheadationandscalingfromthegenericmodel.Figurethestepsforheadmodificationfromphotos.

exturemapping

mappingisusefulnotonlytocovertheroughshape,asheretheshapeisobtainedonlybyfea-intmatching,butalsotogetamorerealisticcolorful

nideaoftexturemappingistogetanimagebyingtwoorthogonalpicturesinaproperwaytogettheresolutionforthemostdetailedparts.Thedetectedpointsdataisusedforautomatictexturegenerationiningtwoviews(actuallythreeviewsbycreatingviewbyflippingtherightview).Wefirstconnectreswithapredefinedindexforfeaturelinesusingdeformation(seeFigure3(a))andamulti-tiontechnique6forremovingboundariesbetweendif-source(seeFigure3(b)).Theeyesandteethim-addedautomaticallyontopofanimage,andthesefortheanimationoftheeyesandmouthre-

veapropercoordinateonacombinedimageforev-onahead,wefirstprojectanindividualized3Dthreeplanessuchasthefront(XY),right(ZY)and)directions.Withtheinformationofthepredefinedfeaturelines,whichareusedforimagemergingwedecideonwhichplaneapointona3DheadisThenprojectedpointsononeofthreeplanesareredtoeitherthefrontfeaturepointsspaceorthesidepointsspacein2D.Finally,atransformontheimageprocessedtoobtainthetexturecoordinates.Moreefoundinthepaper20. Figure3:(a)Ageometricaldeformationforthesideviewstoconnecttothefrontview(b)beforeandaftermulti-resolutiontechniques.

Figure4*showsseveralviewsofthefinalreconstructedheadoutoftwopicturesinFigure2(a).Whenweconnectthisheadwithabody,weremovetheneck(seethesecondlastfaceinFigure4*)sincetheneckisfromthebodyduetothebodyskeletonanimationforfacerotation.ThefaceanimationisimmediatelypossibleasbeinginheritedfromthegenericheadasshowninthelastfaceinFigure4*.

Figure4:snapshotsofareconstructedheadinseveralviewsandanimationontheface

3.Bodycloning

Ourbodycloningisamodel-basedmethod.Weusetwomaininputs.Thefirstinputisthegenericbody.Thesecondisstillphotosofapersontobecloned.Weassumetheper-sonwearstrousersandnottoolooseclothes.Wedeformthegenericbodytoadapttotheindividualizedbody.

3.1.Genericbodystructure

ThegenericbodyisinMPEG-4compatibleH-Anim1.1formats14.TheskeletonandseveralskinpartsdisplayedwithseveralcolorsareshowninFigure5wheretheskinpartsaresmoothlyconnected.Ithas94skeletonjointsand15skinpartsincludinghead,right_hand,left_hand,right_footandleft_foot.Thefirstversionofgenericbodyweareusingiscollectedfromapublicdomain3andmodifiedforourusage.EachskinpartissavedinlocalcoordinatesandisrelatedtoaskeletonjointasshowninFigure6,wheretheskele-tonlocationisindicatedbyarrowsandrelatedskinpartsare

ographicsAssociationandBlackwellPublishers2000.