A Million Polygons, a Million Pixels:Which is Heavier?

(1)

A million pixels, a million polygons.

Which is heavier?

François X. Sillion François X. Sillion

iMAGIS*

Grenoble, France Grenoble, France

*A joint research project of CNRS, INRIA, INPG and UJF

MAGIS MAGIS

(2)

MAGIS 1997 MAGIS 1997

Why this question?

Evolution of processing power and Evolution of processing power and

architectures architectures

New applications, demands and markets New applications, demands and markets

––Giant databases (digital mock up)Giant databases (digital mock up) ––virtual reality, games...virtual reality, games...

Image-based graphics:

––current state and trendscurrent state and trends ––potentialitiespotentialities

(3)

A million polygons

(4)

Who needs a million polygons?

Assemblies of CAD models Assemblies of CAD models

Integrated design/manufacturing Integrated design/manufacturing

Digital mock-ups Digital mock-ups

(5)

A million pixels

(6)

Rendering in Computer Graphics Rendering in Computer Graphics

Models for 3D geometry, light reflection Models for 3D geometry, light reflection

Global illumination simulation Global illumination simulation

Real-time rendering Real-time rendering

All of these requirements All of these requirements present difficult challenges ! present difficult challenges !

(7)

Subtle illumination effects

(8)

Real-time rendering for dynamic scenes

(9)

Image-based rendering (IBR) Image-based rendering (IBR)

Avoid expensive/difficult 3D model Avoid expensive/difficult 3D model

Start from a set of images Start from a set of images

Manipulate pixels to create new image Manipulate pixels to create new image

With real images, elaborate lighting With real images, elaborate lighting

effects are “free”

QuicktimeVR [Chen95], [Laveau], QuicktimeVR [Chen95], [Laveau],

[McMillan95,97],...

(10)

What's an image?

array of RGB (

array of RGB (αα) samples) samples

add depth sample add depth sample

add multiple depths, normals...

(Layered Depth Image, LDI) (Layered Depth Image, LDI)

(11)

Tour into the picture [Horry 97]

Use a single image Use a single image

Manually define simple perspective Manually define simple perspective

Manually create layers with selected Manually create layers with selected

portions of the image portions of the image

See http://www-syntim.inria.fr/~horry/images/s97slide.html

(12)

Layered depth images [Gortler97]

Gather multiple depth Gather multiple depth samples for each pixel samples for each pixel

See http://www.research.microsoft.com/research/graphics/cohen/SIG_97_IBR/index.htm

(13)

Layered depth images Layered depth images

Reproject all samples in Reproject all samples in

new image new image

––no need for depth no need for depth comparisons

comparisons

––splatting techniquesplatting technique

(14)

Rendering from a million polygons?

Transform 1-3M Transform 1-3M

vertices vertices

Lighting Lighting

Texturing Texturing

Memory bandwidth Memory bandwidth

Raster engine, Raster engine,

z-buffering z-buffering

20 M flop 20 M flop 10 M flop 10 M flop 15 M flop 15 M flop 100 Mb 100 Mb

??

(15)

Rendering from a million pixels?

Transform 1M Transform 1M

points (coherence) points (coherence)

No lighting No lighting

No z-buffering No z-buffering

Memory bandwith Memory bandwith

(coherent access) (coherent access)

6 M flop 6 M flop

8 Mb8 Mb

(16)

Rendering performance considerations Rendering performance considerations

3D rendering reaches the consumer market 3D rendering reaches the consumer market

––thousands of lit,textured polygons / second.thousands of lit,textured polygons / second.

––specialized boards require careful design for specialized boards require careful design for efficient integration.

efficient integration.

Image processing subsystems Image processing subsystems

––video (analog/digital), video (analog/digital), ––texture (games),texture (games),

–– multimedia extensions multimedia extensions

(17)

Generating and obtaining IBR models Generating and obtaining IBR models

From synthetic images From synthetic images

––Ray tracingRay tracing

––Range images, LDIs, LumigraphsRange images, LDIs, Lumigraphs

From real images From real images

––use panoramic views, vision techniquesuse panoramic views, vision techniques ––feature matching (difficult)feature matching (difficult)

––Lumigraphs (no depth)Lumigraphs (no depth)

(18)

Link with vision Link with vision

Image based modeling (IBM...) Image based modeling (IBM...)

Use images + parameters Use images + parameters

––avoid WYSIAYGavoid WYSIAYG

––object class informationobject class information

––interactive modeling (facade)interactive modeling (facade)

(19)

IBR = sampling + reconstruction IBR = sampling + reconstruction

Operate without geometry Operate without geometry

More complete representations

More complete representations (higher dimensionality)

(higher dimensionality)

Simplified representations Simplified representations

(adding simplified 3D model) (adding simplified 3D model)

(20)

Light field - Lumigraph Light field - Lumigraph

Levoy, Hanrahan 96 Levoy, Hanrahan 96

Gortler et al 96 Gortler et al 96

Slide used with permission (M. Levoy, Pat Hanrahan) See http://www-graphics.stanford.edu/projects/lightfield

(21)

Light field - Lumigraph sampling Light field - Lumigraph sampling

Slide used with permission (M. Levoy, Pat Hanrahan)

(22)

Impostors Impostors

Create textured 3D model from images Create textured 3D model from images

––simplifiedsimplified representation representation ––rendered as 3D geometryrendered as 3D geometry

Planar polygons

Planar polygons [Maciel95, Schaufler96, Shade96][Maciel95, Schaufler96, Shade96]

3D meshes from range images 3D meshes from range images

[Pulli 97, Darsa 97, Sillion 97]

(23)

Textured 3D mesh from a range image Textured 3D mesh from a range image

Pulli 97 Pulli 97

Slide used with permission (K. Pulli et al.)

(24)

Blending required to combine views Blending required to combine views

without blending without blending

(z-buffer) (z-buffer)

with blending with blending

[Pulli 97]

Images used with permission (K. Pulli et al.) See http://www.cs.washington.edu/homes/kapu

(25)

Principles of our approach: example

(26)

Local model (3D objects)

(27)

Distant model (3D objects)

(28)

Impostor (Textured 3D mesh)

(29)

Combined model (local+impostor)

(30)

Combined model (local+impostor)

(31)

Deforming impostors Deforming impostors

Talisman

Talisman [Torborg 96][Torborg 96]

––Render spritesRender sprites ––Layered modelLayered model ––Affine transformsAffine transforms

[Lengyel97]

Impostor transition Impostor transition

Slide used with permission (J. Lengyel et al, Microsoft research.) See http://research.microsoft.com/~jedl

(32)

Applications for IBR Applications for IBR

Walkthrough / view synthesis Walkthrough / view synthesis

Stereo synthesis Stereo synthesis

Interpolation/extrapolation Interpolation/extrapolation

––Latency compensationLatency compensation ––Frame rate equalizationFrame rate equalization ––Network transmissionNetwork transmission

––Leverage expensive renderingLeverage expensive rendering

(33)

Polygons Pixels Polygons Pixels

Continuous Continuous

Modeling Modeling

Animation Animation

Level of detail Level of detail

Discrete Discrete Capture Capture

Video streams Video streams

Filtering Filtering

(34)

Pixels Pixels

Discrete, regular nature Discrete, regular nature

––easy to filter: adaptation to user easy to filter: adaptation to user perceptual limitations

perceptual limitations

Work with real images Work with real images

––Easy to captureEasy to capture

––Let nature do the modeling/lightingLet nature do the modeling/lighting ––Work from existing images Work from existing images

(historical, legal, forensic applications...) (historical, legal, forensic applications...)

WYSIAYG WYSIAYG

(35)

Polygons Polygons

Complete 3D model Complete 3D model

––solid modelingsolid modeling

––global illuminationglobal illumination

––path planning, assembly checking,path planning, assembly checking, collision detection

collision detection

Common denominator for many Common denominator for many

modeling systems modeling systems

Can be simplified but it's hard to Can be simplified but it's hard to

keep the model consistent keep the model consistent

(36)

Extended notion of image-based models Extended notion of image-based models

Use Use bothboth images and 3D data images and 3D data

Combine a simplified model with images Combine a simplified model with images

model can be extracted from images or model can be extracted from images or

other information other information

(37)

IBR and availability of 3D models IBR and availability of 3D models

Complete 3D model Complete 3D model

––IBR as graphics subsystemIBR as graphics subsystem

No 3D model No 3D model

––QTVR, plenoptic renderingQTVR, plenoptic rendering ––The model isThe model is the image(s) the image(s)

Range data available Range data available

––Scanned data is huge: need to simplifyScanned data is huge: need to simplify

(38)

Problems with current algorithms Problems with current algorithms

Holes in reconstructed images Holes in reconstructed images

Image deformation (impostors) Image deformation (impostors)

Volume of data Volume of data

Sampling/filtering artifacts Sampling/filtering artifacts

(39)

Can we expect hardware advances?

view interpolator view interpolator

soft z-buffering and blending soft z-buffering and blending

multiple or view-dependent textures multiple or view-dependent textures

decompression decompression

memory bandwith memory bandwith

(40)

Limitations of IBR Limitations of IBR

Specularities Specularities

Lighting/geometry/reflectance changes are hard Lighting/geometry/reflectance changes are hard

Computer Vision issues: model building Computer Vision issues: model building

Images may not be available!

(41)

Marketability Marketability

QTVR, panoramic images QTVR, panoramic images

Image-based modeling Image-based modeling

Image-based rendering architectures Image-based rendering architectures

Image caching, impostors Image caching, impostors

Network applications (QoS) Network applications (QoS)

Light field Light field

(42)

...and now?

Simulation of global illumination Simulation of global illumination

Visibility calculations Visibility calculations

View-dependent texture mapping View-dependent texture mapping

––disparity/depthdisparity/depth

––specularity/shadingspecularity/shading ––re-lightingre-lighting

Compression of depth values Compression of depth values

(43)

Computer-augmented reality Computer-augmented reality

Drettakis 97 Drettakis 97

computed solution computed solution input panorama

input panorama

(44)

Conclusions Conclusions

IBR offers useful advances IBR offers useful advances

––leverage cost of high-quality renderingleverage cost of high-quality rendering ––fast extension via specialized subsystemfast extension via specialized subsystem

Vision issues limit applicability of “pure”

IBR for real images IBR for real images

Use combined 3D models and images Use combined 3D models and images

Polygons are still useful!

(45)

Acknowledgements Acknowledgements

Yann Argotti, George Drettakis, Frédo Yann Argotti, George Drettakis, Frédo

Durand, Peter Kipfer, Céline Loscos, Durand, Peter Kipfer, Céline Loscos,

Stéphane Moreau, Cyril Soler Stéphane Moreau, Cyril Soler

Michael F. Cohen, Steven Gortler Michael F. Cohen, Steven Gortler

Marc Levoy, Pat Hanrahan, Kari Pulli, Marc Levoy, Pat Hanrahan, Kari Pulli,

Jed Lengyel, Youichi Horry, Ken-ichi Jed Lengyel, Youichi Horry, Ken-ichi

Anjyo, Kiyoshi Arai Anjyo, Kiyoshi Arai