X-Dimensional Display:
Superimposing 2D Cross-Sectional Image inside 3D Wireframe Aerial Image
Y. Furuyama1, Y. Makino1 and H. Shinoda1
1The University of Tokyo
Abstract
In this paper, we propose a new interactive volumetric display which simultaneously shows a floating 3D image and its 2D cross sectional image inside it. This display enables users to see an arbitrary cross sectional image by inserting a hand-held semi-transparent screen into the 3D image floating in the mid-air. This system has following possibilities:
doctors easily see MRI images inside the 3D body image, and designers check the internal structure of a 3D object in CAD. We construct a prototype device to demonstrate our proposed method by combining a 3D display and an aerial imaging technology. The spinning LED array constructs the volumetric image, and which is optically mirrored to appear in the free-space by the transmissive mirror called Aerial Imaging Plate (AIP). We find that a fine meshed silk screen can be inserted into the aerial image without disturbing the shape of it. Cross sectional images can be projected onto it together with the 3D image. We demonstrate how a silk screen diffuses a 3D aerial image and a projected 2D image.
Categories and Subject Descriptors (according to ACM CCS): I.3.3 [Computer Graphics]: Display algorithm.
1. Introduction
In this paper, we propose a new display system which con- structs an aerial three-dimensional (3D) wireframe image and a two-dimensional (2D) image superimposed inside it at the same time. This system renders a 3D image in the air, and then projects a 2D image on a semi-transparent screen which is inserted into the 3D image. Our technique allows users to slice a wireframe image by intuitively inserting a screen into a 3D image to see its sliced cross-sectional image.
The cross-sectional images of 3D objects are used in sev- eral fields such as MRI images in a medical field, 3D design works, and numerical simulations of physics. Currently, us- ers select a certain cross-sectional plane by inputting its co- ordinate values or moving the section with a mouse. There are some researches that enable users to slice a 3D image intuitively. VolumeSlicingDisplay [CI09] renders 2D cross- sectional images corresponding to the posture of the 2D screen device held by a user. Users can see sequential 2D images by moving the screen device. In this system, some- times it is difficult to search the exact plane since users can- not find the 3D position to be sliced. Moreover, it is also difficult to imagine the whole shape and size of the 3D ob- ject by using the sets of 2D cross-section images. In contrast,
our system renders both the 3D wireframe and 2D cross-sec- tional image on the same physical coordinates. The ad- vantage of our display is that users can find a particular cross-sectional image using a 3D wireframe as a hint for de- termining a slicing location.
To achieve our purpose, we construct an interactive 3D display combining a 3D display of rotating 2D LED array and an aerial imaging technology. First, rotating 64 x 64 LEDs create a light pattern in 3D space based on a persis- tence of vision. Since interaction with this 3D image by fin- gers or any tools is impossible due to the rotating mechanical parts, we use the special transmissive mirror: the Aerial Im- aging Plate (AIP) [Asu11] manufactured by Asukanet Co.
Ltd. This mirror is composed of many tiny cornered mirrors.
The AIP optically reconstructs the touchable aerial image of a real object which is set at the other side of the AIP. Details are given in 3.1.
We superimpose a cross-sectional image inside the 3D floating image by using a semi-transparent screen and a pro- jector. We use a fine mesh screen called a silk screen as a semi-transparent projection screen. The silk screen put in- side the 3D aerial image does not diffuse its shape since light rays can pass through the small mesh of the screen to form the 3D image. Combining the screen posture detection and the rear-projection, we realize a new display system which M. Imura, P. Figueroa, and B. Mohler (Editors)
c The Eurographics Association 2015.
DOI: 10.2312/egve.20151307
enables us to view the cross sectional 2D image in the 3D image simultaneously.
Our 3D display has about 9000 voxels with its refresh rate of 15.5 Hz. By calibrating the floating image position with projector’s coordinate, the 3D display with high precision but low resolution is easily created and controlled. Many other research have developed higher resolution 3D displays than ours. We can easily replace our rotating display with those high-quality 3D displays. Details of the other 3D dis- plays are described in next section.
2. Related works
3D image display systems are roughly categorized into three types. The first one is to display two different 2D im- ages to each eye with parallax. Observers feel depth of the image based on the parallax. Some of them have used polar- ization filtered glasses, Head-Mounted Display (HMD) [Sut68] and so on. In these cases, users have to wear a device on them. Some of them have used special equipment on the display surface such as a lenticular lens, parallax barriers [KOS*01], and Fresnel lens [UIS*14]. In these methods, us- ers have to see the display within an appropriate viewing an- gle.
The second one is to create light distribution pattern in the mid-air. Some of them have used small floating objects to project image on them such as fog screen [Fog02] [Hel01]
and small particles trapped by ultrasound [OTR14]. Others have utilized the persistence of vision. A rotating LED array or a rotating projection screen produces 3D volumetric im- ages [JK84] [KOF9393] [LBB*02]. Laser induced plasma is another method to add light source in the air [KUY06].
The third one is to reconstruct light field. A famous exam- ple is holography, which records and reconstructs phase in- formation of light waves as well as its intensity [sHBL*90]
[HMK91]. A concave mirror can also be used to reproduce light field in the air [BHI*11]. Projection on a rotating mir- ror creates a light field [JMY*07]. Recently a novel optical device: AIP has been proposed [Asu11]. The AIP is com- posed of tiny mirrors and reproduces an object image in mid- air (details are given in next section). There is a similar light field reconstruction method based on retro-reflective mate- rial [YS13].
In this paper, we combine the second and the third princi- ples. A rotating volumetric display observed via the AIP pre- sents 3D volumetric floating image in the mid-air.
Figure 1: System configuration.
Figure 2: Prototype device.
3. System Configuration
We show our system configuration in Figure 1 and the prototyped device in Figure 2. This system is composed of the volumetric display, the AIP to reconstruct the image in the mid-air, the silk screen to be projected cross sectional images on it, the webcam to detect the screen location, and the projector.
Figure 3: 3D Volumetric Display. (Left)Top view.
(Right)Bottom view.
3.1 3D Volumetric Display
Figure 3 shows the 3D volumetric display. The spinning LED array constructs a volumetric image based on persis- tence of vision. Two sets of 64-LEDs arrays mounted bilat- eral symmetrically on the AC motor shaft (BLM5120-A2,
Projector
AIP
Webcam
Visolator AC motor
LED Array
9V battery Bluetooth
Module 40 mm
Microcontroller
Shaft holder
40 mm
LED array
Oriental Motor) which rotates at 930 rpm. Each LED is con- trolled by the microcontroller (LPC1768, NXP) correspond- ing to the rotation angle at intervals of every 2.5 degrees (144 steps in total). This display can generate 9216 voxels image in the cylindrical area whose radius is 4 cm and height is 4 cm and its volumetric refresh rate is 15.5 Hz. 3D light patterns are send from PC via Bluetooth communication.
Figure 4: Aerial Imaging Plate.
3.2 Aerial Imaging Plate
The Aerial Imaging Plate (AIP) is a commercially-availa- ble optical device that reconstructs light patterns in the air.
The basic principle of the AIP is shown in Figure 4. Orthog- onal tiny mirrors are arrayed two dimensionally. When the AIP is tilted 45 degrees from the horizontal plane, where an object locates, a floating image of it appears before the AIP.
This image is viewable without using any special glasses.
We use an octagonal shape of the AIP, 10 cm each side, which is located 30 cm above the spinning volumetric dis- play. In this configuration, the 3D image floats 20 cm before the AIP.
Figure 5: The whole views and closeups of the three types of silk screens. (Left)T100 whose mesh interval is 260 m.
(Center)T150 whose mesh interval is 160 m. (Right) T180 whose mesh interval is 140 m.
3.3 Silk Screen
We used a silk screen as a semi-transparent projection screen which consisted of densely knitted fibers shown in Figure 5. We tried to understand the effect of the screen in- sertion. We used different mesh pitch silk screens whose in- tervals of 140, 160, and 260 m, respectively. Since the size
of the mesh is large enough compared to the light wave- length, the effect of the diffraction is small. The main effect of the insertion of a silk screen is occlusion and diffusion at the silk string. The light rays mainly go through small meshes keeping its directional information, therefore a vol- umetric image is reconstructed even when the silk screen is inserted into the image. We evaluated this effect in section 5.
3.4 Camera and Rear-projector
The position and orientation of a silk screen is estimated by using a webcam. Four markers located on each corner of the silk screen is captured by the webcam and processed by OpenCV library. A cross-sectional image is calculated based on the position and orientation of the screen and rendered onto the silk screen using OpenGL. The image is projected from the projector (PK320, Optoma) set above the AIP as shown in Figure 2.
4. Performance of the proposed system 4.1 3D wireframe image
Figure 6: The demonstration setup. (Left) Constructed 3D cuboid blue image. The silk screen was located as it sliced the yellow region. (Center) Camera setup from top
view. (Right) Camera setup from right view.
We show the performance of our prototype. Figure 6 (Left) shows a 3D cuboid model, we prepared as a sample 3D wireframe, whose size is 50 mm in width, 40 mm in depth, and 20 mm in height and rotated 15 degrees around the Y-axis. Figure 8 shows the aerial images by our system, captured in setup shown in Figure 6 (Center and Right).
These captured images are changed according to the view angles.
Figure 9 shows the 3D images passing through the T180 silk screen which is located to slice the 3D image shown in Figure 6 (Left). Each LED point is blurred and its intensity is weaken by the insertion of the screen, however the whole shape of the 3D image is clearly kept.
260 m 160 m 140 m
50 mm 40 mm
20 mm X
Z Y
Z
X 15 degree
10 degree
±10 degree
Z Y Object
Reconstructed AIP Image
Orthogonal mirrors
c The Eurographics Association 2015.
Figure 7: Projected brain image. (Left) Raw image.
(Right) Projected image on the T180 silk screen. (Bottom) Zoomed image.
4.2 2D projection image
Figure 7 shows the 40 x 40 mm image of the cross-sec- tional human brain image projected on the silk screen. No volumetric image is superimposed in this case. The resolu- tion of the projected image on the silk screen depends on both the mesh pitch of the screen and the projector perfor- mance. Since the silk screen has small holes the projected image is seen as a meshed image when zoomed (Figure 7 bottom-right). However, each hose size is several hundred
m and the details of the brain image is sufficiently visible.
4.3 2D projection image superimposed inside the 3D image
Figure 10 shows the cross-sectional 2D image projected in the cuboid wireframe. The 2D projection image and the 3D image are simultaneously visible. 3D wireframe beyond the 2D projection image is also visible.
The Figure 11 shows another example of our display. We constructed a 3D model of a human head facing left, whose size is 30 mm in width, 30 mm in depth, and 40 mm in height.
Same as the cuboid example, it appears that the 3D image kept its shape when the silk screen is inserted into it, and the cross-sectional brain image coexisted with the 3D wireframe image.
Figure 8: The 3D cuboid wireframe image (50 mm in width, 40 mm in depth, and 20 mm in height and rotated 15 degrees around Y-axis shown in Figure 6.) (Left) View from 10 degree left. (Center left) View from 5 degree left. (Center) View from
the center. (Center right) View from 5 degree right (Right) View from 10 degree right.
Figure 9: The 3D image that the silk screen was located as it sliced the yellow region in Figure 6.
Figure 11: The 3D human head model image and its 2D cross-sectional image projected on the silk screen. (Left) 3D hu- man model image composed of a cuboid, a leftward arrow, and a pentagon. (Center Left) Constructed 3D wireframe image
from front view. (Center) A 3D image in which the silk screen is inserted. (Center Right) Superimposed 2D cross-sectional blue image of the wireframe. (Right) Superimposed 2D brain image inside the 3D head image.
5. Basic Experiment
5.1 Experiment 1: Evaluation of the 3D aerial image and 2D projection image
Figure 12: Experiment 1 Setup.
5.1.1 Method
We conducted the experiment to evaluate the effect of the insertion of a silk screen. We evaluated 2 points, how an aer- ial image was diffused by the silk screen and how clear the projected image was on the silk screen.
The experimental setup is shown in Figure 12. We used a 2D red-squared image on LCD as a light source of the aerial image. A paper was used for projecting the red square at the focal location, parallel to YZ plane. The silk screen was in- serted between the paper screen and the AIP. The insertion of the silk screen can disturb the clarity of the red square image on the paper screen. We captured the edge region of the red square image on the paper screen by 20.0 m/pixel while changing the silk screen pitch (i.e. T100, T150, T180 and without a silk screen). We calculated the x-directional image intensity and its power spectrum from
𝐼(x)= ∑𝑦𝐻(𝑥, 𝑦) (1) P(u) = 𝐹𝐹𝑇[𝐼(𝑥)] (2)
where H(x, y) is a pixel value at (x, y) and u is a spatial fre- quency along x axis.
Then, we projected a plain red image onto the silk screen to evaluate how the silk screen can be used as a projection surface. When the mesh interval is large the projected image becomes rough and dark. We captured the image on the silk screen at 20.0 µm/pixel and calculate its x-dimensional power spectra from the equation (2).
Figure 13: The mesh pitch effects on aerial image. (Top) Captured images. (Middle) The x-directional pixel value.
(Bottom) Logarithm power spectra.
5.1.2 Result
Aerial images on the paper screen and their logarithm power spectra are shown in Figure 13. Compared to the re- sult of “without a silk screen” condition, the aerial image is blurred by the insertion of the silk screen. It also suppresses the light intensity. The finer silk screen blurs the edge of the aerial image more. The change of the pitch of the silk screen does not affect to averaged light intensities.
T100 T150 T180 without a silk screen
-10 -5 0 5 10
0 0.5 1
position x [mm]
I(x) [a.u.]
0 1 2 3 4 5
0 0.5 1
spatial frequency u [cycles/mm]
log10 P(u) [a.u.]
T100 T150 T180
without a silk screen
c The Eurographics Association 2015.
Projected plain images on the silk screen and their loga- rithm power spectra are shown in Figure 14. As shown in the raw data, the projected image is affected by the mesh pitch.
The peak power of the spectra is found at the spatial fre- quency corresponding to the each mesh pitch. The projected image on the fine mesh silk screen is clear and sharp. These results suggests the fine mesh screen suits for 2D image pro- jection, though it does not appropriate for keeping shape of the 3D image.
Figure 14: The mesh pitch effects on the projection im- age. (Top) Captured image. (Middle) The x-directional
pixel value. (Bottom) Logarithm power spectra.
Figure 15: Experiment 2 setup.
5.2 Experiment 2: User study 5.2.1 Method
The purpose of this experiment is to demonstrate that us- ers can see the volumetric floating image as 3 dimensional even when a silk screen is inserted. To achieve our concept,
the depth of the low resolution wireframe image must be rec- ognized as 3D. Additionally, The fine meshed silk screen must not block the 3D image. Since the diffusion of the 3D image depends on both the mesh pitch and slicing location, we evaluate the user perception while changing these condi- tions.
The experimental setup is shown in Figure 15. This exper- iment is conducted in following procedure. Eight subjects stood in front of our device. They closed their eyes as an initial condition. A silk screen is inserted inside the aerial cuboid wireframe image (6 cm in width, 6 cm in depth, and 2 cm in height) parallel to the YZ plane as shown in Figure 15. Then, they were asked to open their eyes and to move the pointer along X axis to indicate the nearest position of the 3D image. The difference between the pointer location and the true one is measured by the camera. This sequence is repeated while changing the silk screen conditions. We changed two parameters: two screen positions (at the middle and the behind of the 3D cuboid image) and two screen pitches (T100 and T180). For comparison, we also experi- mented in no silk screen condition. Thus there were 5 con- ditions: 2 positions x 2 pitches + no screen. Each condition was repeated 5 times in random order. In total, 25 trials were conducted to one subject. Eight male examinees, whose ages were 20-25 years old, participated in this experiment.
Figure 16: The mean distance between pointer location and actual one. Error bar means a standard deviation.
5.2.2 Result
The result is shown in Figure 16. Each bar represents the mean distance between the indicated position and the actual one. The positive value means that examinee set the pointer nearer to their view point. Each error bar means the standard deviation. T-tests are conducted between the result of with- out a silk screen and other 4 conditions. There were no sig- nificant differences among the most results between with/without a silk screen except for Subject 7. Most exam- inees located the pointer within 10 mm from the actual posi- tion. Thus, users can see the aerial image as 3D. The inser- tion of the silk screen does not affect to the 3D shape recog- nition.
In this experiment, we receive many qualitative com- ments from examinees. Some felt easy to recognize the 3D
T100 T150 T180 paper screen
-5 0 5
0 0.5 1
position x [mm]
I(x) [a.u.]
0 2 4 6 8 10
0 0.5 1
spatial frequency u [cycles/mm]
log10 P(u) [a.u.]
T100 T150 T180 paper screen
shape and some did not. One examinee could not perceive the depth of the 3D image when he approached to the 3D image. He said the viewing distance was important to recog- nize 3D structure. This comment suggests the view point is important.
6. Limitations
We discuss the limitations about 3D volumetric display and the AIP.
6.1 3D Volumetric Display
The spatio-temporal resolution of the rotating volumetric display depends on the resolution and refresh rate of the screen and its rotating speed. The resolution of our volumet- ric display is 9216 voxels and the volumetric refresh rate is 15 Hz. We construct this prototype to show the basic princi- ple of our concept. Our system can be easily combined with another high resolution 3D display and we have to choose the appropriate one depending on the requirement.
6.2 Aerial Imaging Plate
The AIP is composed of an array of tiny corner reflectors.
The array pitch defines the resolution of reconstructed aerial images. Currently the interval of the mirrors is about 0.5 mm.
For achieving fine volumetric images, higher resolution ver- sion of the AIP is desirable, though the current AIP has suf- ficiently high spatial resolutions.
The visible position of the 3D virtual image depends on the sizes of the AIP and the reconstructed objects. The virtual image appeared 15.5 cm from the plate in this time. Assum- ing that the size of the object is about 5 cm, the visible angle will be about ±16 degrees when we used the AIP of 23.5 cm square. To realize wider visible range, the AIP should be larger.
6.3 Necessity of 2D projection
When we assume an ideal high-resolution 3D volumetric display, we do not need to have a projector for projecting 2D image. The ideal 3D display can show both the 3D image and its cross-sectional image at the same time.
We used a projector in the current study because there is no such high-resolution 3D display yet. For example, Jones et al. realized the high-quality 3D display [JMY*07] that has 768 x 768 voxels updated by 20 Hz. They demonstrated high quality 3D volumetric images. In contrast, a commercially available small projector allows 1280 x 800 resolution 2D images with 60 fps. In order to achieve high resolution and interactive 2D image superimposed into the 3D image, we thought it was better to use a projector. The situation can be changed depending on the development of 3D displays.
7. Applications
Our system can superimpose a 2D image onto a 3D volu- metric image. This enables users to interact with 2D and 3D images at the same coordinates. The main application of this
interaction is intuitive handling of 3D CAD data. Usually people see 3D CAD data by projecting it onto a 2D plane such as elevation, bird’s-eye, and cross section views. Some- times it is difficult to set those cross section planes intui- tively. Our method allows users to insert a cross section plane at arbitrary location and angles. This visualizing method will be useful for comprehending 3D analytical re- sults of structures, fluids behaviors, electromagnetic fields and so on.
This technology also can be applied for a medical field.
Currently, medical doctors measure 3D information of hu- man body with the MRI or some other visualizing technolo- gies. Although such data is usually presented on a conven- tional 2D display, our system allows doctors to see both 3D and 2D images at the same time. They can observe arbitrary cross section images intuitively. Our system would be one of the useful interfaces for those purposes.
8. Conclusion
In this paper we proposed a novel volumetric display that can superimpose 2D images inside it. We used two key tech- nologies: a rotating volumetric display and the Aerial Imag- ing Plate (AIP). Combination of these technologies produces interactive volumetric images in the air. Images can be seen as 3D even when a semi-transparent silk screen is inserted into it. Thus users can interact with volumetric images su- perimposing their cross sectional images by projecting it on the silk screen. In this paper we demonstrated a basic con- figuration of the system and experimentally showed that the insertion of the silk screen did not affect to the perception of 3D image. Examinees could point out a nearest point of a produced 3D image independent on the insertion of the silk screen. This technology can be used for manipulating 3D data intuitively such as MRI images in medical field.
Reference
[Asu11] ASUKANET CO.LTD:AIPLATE (2011) http://aerialimaging.tv/index.php
[BHI*11] BUTLER,A.,HILLIGES,O.,IZADI,S.,HODGES,S., MOLYNEAUX,D.,KIM,D., AND KONG,D.: Vermeer: direct interaction with a 360° viewable 3D display. In Proceed- ings of the 24th annual ACM symposium on User inter- face software and technology (2011), New York, NY, USA, 569-576.
[CI09] CASSINELLI, A., AND ISHIKAWA M.: Volume Slicing Display, SIGGRAPH ASIA ‘09 (2009), Emerging Technologies
[Fog02] FOGSCREEN (2002) http://www.fogscreen.com/
[Hel01] HELIODISPLAY (2001) http://www.io2technology.com/
[HMK91] HASHIMOTO,N.,MOROKAWA,S., AND
c The Eurographics Association 2015.
KITAMURA K.: Real-time holography using the high-reso- lution LCTV-SLM, Proc. SPIE 1461, Practical Holog- raphy V, 291 (1991).
[IHT08] IZADI,S.,HODGES,S.,TAYLOR,S.,ROSENFELD, D.,VILLAR,N.,BUTLER,A., AND WESTHUES J.: Going Be- yond the Display: A Surface Technology with an Elec- tronically Switchable Diffuser, In Proceedings of the 21st annual ACM symposium on User interface software and technology (UIST’08) (2008), 269-278.
[JK84] JANSSON,D.G. AND KOSOWSKY R.P.: Display of Moving Volumetric Images", Proc. SPIE 0507, Pro- cessing and Display of Three-Dimensional Data II, 82 (1984).
[JMY*07] JONES,A.,MCDOWALL,I.,YAMADA,H.,BOLAS, M., AND DEBEVEC,P.: Rendering for an interactive 360°
light field display. In ACM SIGGRAPH 2007 papers (2007), New York, NY, USA, Article 40.
[KOF93] KAMEYAMA,K.,OHTOMI,K., AND FUKUI,Y.: In- teractive 3-D Volume Scanning Display with Optical Re- lay System and Multi-Dimensional Input Device, Stereo- scopic Displays and Applications IV, vol. 1905 (1993).
[KOS*01] KUNITA,Y.,OGAWA,N.,SAKUMA,A.,INAMI, M.,MAEDA,T. AND TACHI,S.:Immersive Autostereo- scopic Display for Mutual Telexistence: TWISTER I (Telexistence Wide-angle Immersive STEReoscope model I). In Proceedings of IEEE VR 2001, 31-36.
[KUY06] KIMURA,H.,UCHIYAMA,T., AND YOSHIKAWA, H.: Laser produced 3D display in the air. In Proceedings of ACM SIGGRAPH 2006 Emerging Technologies (2006).
[LBB*02] LANGHANS,K.,BAHR,D.,BEZECNY,D., HOMANN,D.,OLTMANN,K.,OLTMANN,K.,GUILL,C., RIEPER,E., AND ARDEY,G.: FELIX 3D display: an inter- active tool for volumetric imaging, Proc. SPIE 4660, Ste- reoscopic Displays and Virtual Reality Systems IX, 176 (2002).
[OTR14] OCHIAI, Y.,TAKAYUKI,H., AND REKIMOTO, J.:
Pixie dust: graphics generated by levitated and animated objects in computational acoustic-potential field. ACM Trans. Graph. 33, 4, Article 85 (July 2014), 13 pages.
[SHBL*90] ST-HILAIRE,P.,BENTON,S.A.,LUCENTE,M.
E.,JEPSEN,M.L.,KOLLIN,J.,YOSHIKAWA,H., AND
UNDERKOFFLER J.S.: Electronic display system for com- putational holography, Proc. SPIE 1212, Practical Ho- lography IV, 174 (1990).
[Sut68] SUTHERLAND,I.E.: A head-mounted three dimen- sional display. In Proceedings of the December 9-11, 1968, fall joint computer conference, part I (AFIPS '68 (Fall, part I)), 757-764.
[UIS*14] UEDA,Y.,IWAZAKI,K.,SHIBASAKI,M., MIZUSHINA,Y.,FURUKAWA,M.,NII,H.,MINAMIZAWA, K., AND TACHI,S.: HaptoMIRAGE: mid-air autostereo- scopic display for seamless interaction with mixed reality
environments. In ACM SIGGRAPH 2014 Emerging Technologies
[YS13] YAMAMOTO,H., AND SUYAMA,S.: Aerial Imaging by Retro-Reflection (AIRR). SID Symposium Digest of Technical Papers (2013), Vol.44, No.1, pp.895-897.
