ELMO: An Enhanced Optical See-Through Display Using an LCD Panel for Mutual Occlusion Kiyoshi Kiyokawa*, Yoshinori Kurata**, Hiroyuki Ohno* *Communications Research Laboratory (CRL) 4-2-1 Nukui-Kitamachi, Koganei, Tokyo, 184-8795, Japan **Topcon Co. 75-1 Hasunuma-cho, Itabashi-ku, Tokyo, 174-8580, Japan
[email protected],
[email protected],
[email protected]
Abstract To attack a well-known occlusion problem of conventional optical see-through displays, we have been developing ELMO (an Enhanced optical seethrough display using an LCD panel for Mutual Occlusion). The latest prototype display under construction has a real-time rangefinder as well as an embedded light-blocking mechanism, thereby presents a mixed reality environment with correct mutual occlusion phenomena in real-time. Keywords: Mixed reality, mutual occlusion, HMD.
1 Introduction
Erecting Prism
CG Image LCD Panel
f
Half Silvered Mirror
f Objective Lens
Eyepiece
Figure 1 Basic design of the optics. Mirror 2
Mirror 1
Prism 1 Prism 3 Prism 4 Prism 2
Figure 2 ELMO-2 optics.
In a mixed reality system, mutual occlusion of real and virtual objects highly enhances the user’ s sense of reality. Though video see-through displays are suitable for mutual occlusion, they degrade real images in terms of spatial and temporal resolution. On the other hand, conventional optical see-through displays cannot present mutual occlusion correctly since the synthetic objects always appear as semitransparent ghosts. Previous efforts to attack this problem required special environmental settings such as a dark room [1]. This paper describes our optical see-through display named ELMO (An Enhanced optical see-through display using an LCD panel for Mutual Occlusion), which has the capability of handling true mutual occlusion optically.
2 ELMO-1 and -2 ELMO uses an LCD panel to block any rays coming from outside, and two convex lenses to keep the image on the panel in focus (see Figure 1). By opening and closing pixels on the LCD panel where real objects should appear and disappear, mutual occlusion can be presented optically. We first built a bench-top prototype display ELMO-1 to confirm the strategy [2]. Then we built ELMO-2 shown in Figures 2 and 3, which has smaller optics and shorter viewpoint offset. The outer
Right Fuse Cross-eyed
Left
Figure 3 ELMO-2 with a link support.
Right Fuse Wall-eyed
Figure 4 ELMO-2 with a link support. frame is made to fit a 10.4-inch LCD panel, though the core optics is much smaller. Since ELMO does not affect the real environment nor require any additional environmental settings, it can be used anywhere in any place, e.g., outdoor. Besides, since ELMO can surely block any rays coming from outside scenery, virtual images keep their original intended color, e.g., black. Specification of ELMO-2 is shown in Table 1.
Figure 5 ELMO-3 blueprint. Figure 4 shows a few examples of composed stereo images that are captured by a pair of small cameras (Toshiba IK-CU43). In this case, a cross-shaped virtual object is superimposed onto the real white box. You can fuse these stereo pairs and confirm that ELMO-2 presents stereoscopic images with mutual occlusion correctly. We also confirmed this scene is correctly seen stereoscopically by the naked eyes.
3 ELMO-3 To present mutual occlusion for unknown environments, we need to have depth information of the real scene in real-time as well. Based on the ELMO-2 optics, ELMO-3 is being developed to incorporate a depth acquisition mechanism. Among a number of realtime depth acquisition mechanisms, a passive approach would be suitable for ELMO to keep the advantage that ELMO does not affect the real environment. Considering resolution of the depth map and processing speed, we decided to use a multi-camera real-time stereovision system FZ930 of Komatsu, which can produce thirty 280x200 depth maps per second. We use five cameras that have a focal length of 6 [mm] and a baseline of 79 [mm]. The blueprint of ELMO-3 is shown in Figure 5 and a camera head of ELMO-3 in Figure 6. Depth resolution varies in proportion to the square of the distance. At the minimum detectable Table 1 Specification of ELMO-2 and -3. Eyepiece/Objective lenses Focal length Effective aperture Center thickness Weight (for each) Erecting prism Window shape Refractive index Weight Effective field of view Exit pupil aperture Eye relief Viewpoint offsets Horizontal Vertical
75 [mm] 38 [mm] 15.4 [mm] 56 [g] 40 x 40 [mm] 1.516 ~ 1.834 161 ~ 284 [g] > 25 [degree] > 5 [mm] > 60 [mm] 216.9 [mm] 0 [mm]
Figure 6 ELMO-3 camera head. distance (831 [mm]), depth resolution is 5.5 [mm]. Horizontal and vertical fields of view are about 40 and 30 degrees, respectively, that are larger than those of ELMO optics. Another new feature of ELMO-3 is that there is no need to use a normal see-through HMD any more to produce color CG, by using beam splitters and color LCD panels instead of prisms No.2 shown in Figure 2. ELMO-3 will be the first optical see-through display in the world that has all necessary elements for mutual occlusion, a real-time depth acquisition mechanism, a light-blocking mechanism, and color display modules inside it.
4 Conclusions Our display ELMO can be used anywhere including outdoor, and it enhances color fidelity of virtual images. Through the experimental usage, we confirmed that our display design surely solved the occlusion problem. Though current ELMO has problems of bulky body, slow response and light attenuation, all of them can be improved by replacing the LCD panel with a better one, which is smaller, faster and brighter. This display is especially useful when users want to avoid any degradation of real scene. Medical and outdoor applications will suit this display.
5 Acknowledgements We thank Komatsu Co. for technical support.
6 References [1] Shinichi Noda, Yoshihiro Ban, Kosuke Sato and Kunihiro Chihara, “ An Optical See-Through Mixed Reality Display with a Realtime Rangefinder and an Active Pattern Light Source,” Trans. of the VRSJ, Vol.4, No.4, pp.665-670, 1999 (in Japanese). [2] Kiyoshi Kiyokawa, Yoshinori Kurata and Hiroyuki Ohno, “ An Optical See-through Display for Mutual Occlusion of Real and Virtual Environments,” Proc. of the IEEE & ACM ISAR 2000, pp.60-67, 2000.
