Interactive Holographic Application using Augmented

Download PDF
0 downloads 0 Views 1MB Size Report
Comment
learning and teaching [1], as the 3D images allow the student to understand certain topic easily ..... engine provides the scripting application programming interface (API) for the software ... JavaScript or C# language can be used for the Unity. ..... https://docs.unity3d.com/Manual/class-AnimatorController.html. [28. August 2017].
Interactive Holographic Application using Augmented Reality EduCard and 3D Holographic Pyramid for Interactive and Immersive Learning 1st Chan Vei Siang Faculty of Computing Universiti Teknologi Malaysia Skudai, Malaysia [email protected] 4th Yusman Azimi Yusoff Faculty of Computing Universiti Teknologi Malaysia Skudai, Malaysia [email protected]

2nd Muhammad Ismail Mat Isham Faculty of Computing Universiti Teknologi Malaysia Skudai, Malaysia [email protected]

5th Mohd Khalid Mokhtar Faculty of Computing Universiti Teknologi Malaysia Skudai, Malaysia [email protected]

Abstract—The advancement of augmented reality (AR) and holographic display technologies have a great potential to support and improve teaching and learning process, because the 3D images give new perspectives to the students to understand certain topic easily and intuitively. In this paper, an Interactive Holographic Display is introduced, which is aimed to ease the teachers to deliver the knowledge to the students as well as to provide self-learning for the students. This proposed solution is implemented using the AR tracking technique and fused with 3D holographic pyramid display. This makes the virtual objects can display in thin air like a real object and makes the holographic effect more realistic and interactive, as the user can interact with the virtual objects using an image target. This paper explains how the system is physically realized in term of hardware configuration and software design. Keywords—Feature-based augmented reality (AR), holographic projection, 3D holographic pyramid, human-computer interaction.

I. INTRODUCTION The advancement in computer graphics and multimedia technologies have changed the way human view and interact with the virtual world, such as the augmented reality (AR) and the hologram display. The usage of AR display devices, such as smartphones and smart glasses, allow the user to receive additional information, which is in the form of informative graphics based on his or her field of view through the devices, for example, the street’s name, navigation arrow to lead the user to the destination, etc. On the other hand, the use of holographic pyramid prism can produce the holographic results that displayed the 3D objects in the real world environment, by letting the user to look at different perspective of these holograms when viewing from different angles. Thus, these multimedia technologies have a great potential to support learning and teaching [1], as the 3D images allow the student to understand certain topic easily and intuitively [2] [3]. In

3rd Farhan Mohamed Faculty of Computing Universiti Teknologi Malaysia Skudai, Malaysia [email protected]

6th Bazli Tomi Faculty of Computing Universiti Teknologi Malaysia Skudai, Malaysia [email protected]

7th Ali Selamat Faculty of Computing Universiti Teknologi Malaysia Skudai, Malaysia [email protected]

addition, the use of holographic pyramid let the digital content to be shared in a group of people easily without the use of glasses and wearable devices, therefore improve the presentation of teaching material in classroom. However, there are also some issues and challenges found in the current scenario. The current AR application is displayed in a small screen of smartphone where not many information can be displayed on the screen which caused difficulty for the user to view the content on the screen [2]. Users also need to always hold their smartphones or tablets and target their devices to an image target, in order to view the virtual objects present on the device’s screen [2]. Besides that, these AR technologies are only available for only one user to view the AR object at a time, socially separated user experience [4]. These issue makes human-computer interaction difficult and hard to share the digital content. Furthermore, recently the application of holographic pyramid is static and support for exhibition displays only, where user cannot interact with the holographic display, such as in [5]. Although there is some effort being made to improve the interactivity of holographic display, for example the addition of tracking sensor that allow the user to interact with the 3D objects using hand gestures [3] [6]. Beside that in [7], smartphones are used for the interaction. This paper tends to provide an alternative yet novelty option to allow the user to interact with the holographic display. Hence, the Interactive Holographic Display is the proposed solution for the stated problem as well as to improve the interactivity in the student’s learning process. This proposed solution is implemented using the AR tracking technique and fused with 3D holographic pyramid display. This makes the virtual objects can display in thin air like a real object and makes the holographic effect more realistic and interactive. User can also interact and view various 3D objects in just one holographic pyramid by using a specially designed AR image target – the EduCard. The displayed content consists of

This paper is supported by Ministry of Science, Technology & Innovation under the ScienceFund grant vote 4S135 and Universiti Teknologi Malaysia Research University Grant vote 11H81. © © 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.C. V. Siang et al., "Interactive holographic application using augmented reality EduCard and 3D holographic pyramid for interactive and immersive learning," 2017 IEEE Conference on e-Learning, e-Management and e-Services (IC3e), Miri, 2017, pp. 73-78.doi: 10.1109/IC3e.2017.8409241. URL: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8409241&isnumber=8409225

animation and also the audio explanation about a certain topic, such as the Earth, etc., that will eased the teachers to deliver the knowledge to the students as well as to provide self-learning for the students. Furthermore, the resources used to build the holographic pyramid are easy to gather and cheap, such as the Perspex glass, thus provides a low-cost solution for the holographic display setup. This paper is structured as follows. Section 2 presents the literature review that covers the AR technology, holographic pyramid display as well as related applications and projects. In Section 3, the system overview of the proposed solution is explained. Section 4 explains the project methodology. Section 5 details out the implementation process in three important modules, which are the AR tracking technique, the holographic display configuration and the software development. Section 6 provides and discusses the results of this project. And lastly, Section 7 gives the conclusion of the proposed solution and its future works. II. LITERATURE REVIEW A. Augmented Reality (AR) Thomas P. Caudell and David W. Mizell coined the term “augmented reality” in 1990, both are Boeing researcher, to describe a digital display used by aircraft electricians which superimposed virtual graphics onto a physical reality [8]. Based on Azuma [9], the AR is a variation of virtual environment where it allows the user to see the real world, with virtual objects superimposed upon or composited with the real world. The AR system consists of these three characteristics, which are, the combination of real and virtual, is interactive in real time and is registered in 3D [9]. The AR can be categorized as marker-based AR and marker-less based AR [10]. The marker-less based AR can be further classified into global positioning system (GPS) based AR and the natural features based AR. The marker-based AR perform interesting point detection, or fiducial markers detection by using the image processing methods and then the virtual object is positioned and oriented on the output screen based on the marker [11] [12]. The fiducial markers are capture by the camera in order to feed the input images for image processing. The GPS based AR uses the advantage of GPS function in the user’s smartphone, by finding the geographical location of the user [2] so that it can detect the interesting place around user which is also known as the Point of Interest (POI). Once a POI is founded, user will get additional information on top of the physical environment through the use of this application, in the form of graphics, videos, audios, texts, 2D or 3D objects, etc. [13]. The natural based AR can recognize dynamic objects in our real world environment, such as sights, buildings, as well as living beings, without using any fiducial markers and GPS data [14]. It is heavily relying on the object recognition process [14]: image processing process, feature extraction process, and classification process. B. Holographic Display The holographic pyramid is a reflective prism display that allows the viewer to view images of computer-generated object

from different perspectives through the multi-side of prism [15]. Technically it is not an actual hologram, although it provides the similar effect as hologram [7]. It uses an illusion method called the Pepper’s Ghost technique, which uses glass or any reflective material to reflect light from projection in order to create a holographic effect [15]. The display device will project four distorted views (front, back, left, right) of a 3D objects or environment that are reflected on the glass to create an illusion of 3D objects floating in the middle of the pyramid, where the reflective glass is placed at 45 degrees with screens and lighting. The Musion Eyeliner is a 3D holographic projection system built and patented by Uwe Maass, and it is considered a modern commercial variation of Pepper’s Ghost [17], where it is used in many applications such as in teleconferencing, mixed-media entertainment, as well as educational event. III. SYSTEM OVERVIEW The Interactive Holographic Display offers a solution to display the learning content in the form of hologram in a fun and interactive way, which consists of 3D animation and the audio explanation of a topic. This proposed system utilized the AR tracking technique to determine what content to be displayed in the holographic pyramid by using the Vuforia software development toolkit (SDK). On the other hand, the holographic pyramid acted as the output for the system to display the holographic 3D images from the multi-view projection, where the users can view the images from different perspectives, such as front, back, left and right. The aim of this system is to improve the learning process of the student as well as provides an effective sharing of digital content. The workflow of the system is as follows. The users need to place the EduCard – a specially designed AR image target, within the webcam’s range of view. If an input marker is detected, the AR tracking technique is used to determine which and where the virtual object should display in the virtual world according to the EduCard. Next, the four cameras are setup within the virtual world that represent the front view, back view, left view and right view of the displayed virtual object. In order to display the 3D content inside the holographic pyramid, the system used the render texture method, where the four cameras’ viewport are rendered onto the four image textures which also represent the front view, back view, left view and right view in relative with the cameras’ position. Ultimately, all the image textures are projected on the screen and then are projected on the holographic pyramid according to the Pepper’s Ghost technique. Figure 1 shows the framework of the proposed system. As metioned beforehand, the users interact with the system using the specially designed EduCard. The position and rotation of the displayed 3D content in the holographic pyramid is based on the transformation of the EduCard within the webcam’s range of view. If the EduCard is placed near the left side of the view range, the displayed 3D content is projected near the left side where the viewer on the left able to view the bigger size content while the viewer at the right view a smaller size content. Furthermore, each EduCard represents a certain topic and hence, has unique animation and audio explanation. Thus, the users can switch the displayed content

The webcam will capture the video frame to allow the Vuforia SDK to search for the input image markers or image targets. If an input marker is detected, the SDK identify the marker features that are found in the image itself and then comparing these tracked features with a known target resource database. Once the image target is detected, the SDK will perform the viewport calculation to position and orient the virtual object and lastly display that virtual object on the output screen. The SDK will continue track the input marker as long as it is at least partially in the camera’s field of view.

Fig. 1. The framework of proposed system.

by placing different EduCard within the webcam’s range of view. IV. METHODOLOGY This section is separated into three important modules, which are the AR tracking technique, the holographic display configuration and the software development. The Vuforia software development toolkit (SDK) is used to detect and track the EduCard. The system is developed by using the Unity 3D game engine. A. Feature-based Tracking Technique The proposed system used the Vuforia SDK, as it has provided many utilities to develop the AR application. The tracking technique was chosen for this project is the featurebased tracking, where it required the AR image target or image marker to allow the Vuforia SDK to detect and track. The workflow of Vuforia SDK is as shown in Figure 2.

Fig. 2. The workflow of Vuforia SDK.

B. Holographic Display Configuration The texture facet serves as the image source which projects different views of the virtual environment onto each surface of the pyramid which is also known as facet, through different viewport, with the use of monitor. In order to display the 3D object in the holographic pyramid, the texture facet rendering in the Unity 3D is performed by placing four cameras in the Unity 3D scene to render the four viewports, which represent front view, back view, left view and right view, of the virtual object to be projected onto each side of the pyramid [15]. Ultimately, the render textures are actively updated in real-time where any updates feed from the cameras are rendered onto the render textures as well as acted as the final output to be projected on the monitor screen. The holographic display is created by implementing the Pepper’s Ghost technique. This technique requires a combination of two main hardware, which are the holographic pyramid with four reflective sides at the bottom and a monitor that is placed on top of the holographic pyramid. The monitor displays the texture facet rendered in the Unity scene and the image in the scene is reflected on the reflective surface at an angle of 45 degrees. Therefore, it appears natural on the holographic display when users are viewing through the Pyramid Prism in single viewpoint. C. Software Development Unity 3D has provided many important functions to develop the software. The Unity provides the camera asset that are used to capture and display the virtual world to the output device. There can be unlimited number of cameras in a Unity scene, where each camera is set at any place in Unity scene, as well as can be rendered at any place on the output screen [17]. The cameras are essential to render the texture facet as stated in the previous subsection. The Unity also has a prefab asset type that allows the developer to store a game object complete with components, properties as well as C# scripts. The prefab acts as a template where a new object instance can be created in the scene. Any edits made to a prefab asset is immediately reflected in all instances produced from it, but also can override components and settings for each instance individually [18]. Besides that, the animator controller asset in Unity allows the developer to arrange and maintain a set of animations for a game object by creating his or her own state machine, which is a visual programming language [19]. The animator controller is important for display the correct content when certain event is raised. Furthermore, the Unity engine provides the scripting application programming interface (API) for the software developers to customize the attributes and behaviors of game

object to achieve many kinds of imaginable effect. Either JavaScript or C# language can be used for the Unity. V. IMPLEMENTATION This section explained the implementation of AR and holographic display. A. Implementing AR Tracking Technique The EduCard, which is an image target or AR marker to allow the Vuforia SDK to detect and track is required in this project. There are two types of EduCard, which are the base card type and extra content card type. The base card type contains the basic introduction of a certain topic, while the extra content card type contains a specific content based on the base topic. In other word, the extra content card type cannot work on its own and is required the base card type to be presented within the webcam’s range of view and is detected by the SDK. Only one base card type can be detected by the SDK. If there are more than one base card type within the webcam’s range of view, the first base EduCard placed in the range is choosen to be detected. Figure 3 shows the example of EduCards.

are rendered onto four render textures, by set the “Target Texture” field in the inspection tab of the camera object, such that the “TopCam” is rendered onto the “Top” texture; the “BottomCam” is rendered onto the “Bottom” texture; the “LeftCam” is rendered onto the “Left” texture; the “RightCam” is rendered onto the “Right” texture. The render textures are created using the “Raw Image” object from the UI in the Unity 3D, while the triangular mesh is created in the Autodesk Maya 2017. After import the triangle mesh into the Unity, assign the render texture to the triangle mesh as its material and arranged as shown in Figure 4. The final output for texture facet rendering is shown in Figure 5.

The configuration for Vuforia in Unity is essential to actualize the tracking technique for this project. Firstly, from the “Vuforia Configuration” found in the “ARCamera” game object, the “Max Simultaneous Tracked Image” field is set to “2”, which means that the Vuforia can only detected and tracked maximum two image targets at the same time when there are two EduCards presented in the webcam’s range of view. Besides that, the “AR Camera” has fixed position and orientation, while the target objects can move around the virtual world according to the position of the EduCard. This is essential to the holographic display setup, which is explained in the next subsection. B. Holographic Display Configuration In order to setup the render texture facet to display on the monitor’s screen, four cameras are placed in the Unity 3D scene to render the four viewports, which represent front view, back view, left view and right view, of the virtual object to be projected onto each side of the pyramid. All four cameras are placed as follows, the “LeftCam” rotates by 90 degrees, the “BottomCam” rotates by 180 degrees, the “RightCam” rotates by 270 degrees, and the “TopCam” remains at 0 degrees. The “ARCamera” game object is placed at center in order to capture by the cameras. Figure 4 shows the four-sided camera setup in the Unity 3D scene. Next, the four cameras’ viewport

(a)

(b)

Fig. 4. The configuration of hologram camera and texture facet setup.

(c)

Fig. 3. The example of EduCards. (a) is the base EduCard (namely

EarthAR), while (b) and (c) are the extra content EduCard (namely MoonAR and AtmosphereAR respectively).

Fig. 5. The texture facet displayed in the monitor screen.

After texture facet rendering process and the output is displayed on the monitor screen, the holographic pyramid is required to set up in order to display the holographic object in the real-world environment. The list of hardware that were used to build the holographic pyramid beforehand are a monitor, a monitor stand, a reflective pyramid prism, a card platform, a webcam and a laptop. In order to determine the size of holographic pyramid, firstly have to specify the screen size to be projected. In this project, the screen size is 34cm. Next, using the trigonometry to find the height of the pyramid. Then, using the trigonometry and the height of the pyramid to calculate the height of each triangle prism, so that can get the measurement of Perspex glasses. After that, the Perspex glasses are assembled together to form the pyramid. Figure 6 shows the steps to calculate the size of triangle face for each side of pyramid. Figure 7 shows the pyramid prism that is used in this project. The pyramid prism has the height of 17cm, with a square base of 34cm at each side. The angle between the incline plane of pyramid prism wall and the horizontal table or projection monitor screen is 45 degrees. Figure 8 shows the hardware setup of the system.

Fig. 6. The steps to calculate the size of triangle face for each side of pyramid

C. Software Development The execution of the system is arranged in different states as shown in the finite state machine in Figure 9. When the system is in the Idle state, a 3D UTM logo is shown floating in the holographic pyramid. It is also spinning slowly in clockwise direction. This state is ready to detect the present of the EduCard. If a base educard is presented in the webcam’s range of view and is detected by the Vuforia SDK, the system changed its state into Base Content state. When the system is in the Base Content state, the 3D content of the EduCard is displayed in the holographic pyramid while the 3D UTM logo is hided. The audio explanation about the content is played. If the Vuforia SDK lost detect the base EduCard, the system changed its state back to Idle state. If an extra content EduCard that is related to the base EduCard is detected by Vuforia, the system changed its state into Extra Content state. During this state, if another base Educard is presented within the webcam’s range of view, the system continues displayed the content of the first base EduCard.

Fig. 7. The pyramid prism.

Fig. 8. The hardware setup of holographic pyramid.

When the system is in the Extra Content state, the extra 3D content of the EduCard is displayed in the holographic pyramid. The audio explanation from the Base Content state is stopped if the audio is not finished playing, while the audio explanation about the extra content is played. If the Vuforia SDK lost detect the base EduCard, the system changed its state back to Idle state. If the Vuforia SDK lost detect the extra content EduCard, the system changed its state back to Base Content state only if the audio explanation of extra content is finished playing. During this state, if another extra content Educard, that is either related to the base EduCard or not, is presented within the webcam’s range of view, the system continues displayed the content of the first extra content EduCard. Fig. 9. The finite state machine of the system.

VI. RESULT AND DISCUSSION

ACKNOWLEDGMENT

The result of the Interactive Holographic Display is a standalone PC application that integrating EduCard with augmented reality (AR) tracking and holographic pyramid display. The main contents of this system are information about the planet Earth, such as its relationship with the moon, its atmosphere and its movement. The EduCards also came in bilingual, which are Malay version and English version. The user can interact with the game by placing the base type EduCard onto the card platform, which is within the viewing range of the webcam to track the image target. Once the image target is detected, the 3D model or contents of the base type EduCard is displayed in the holographic pyramid and the audio explanantion about that content is played. Besides that, the user can also perform some action to invoke the extra content in the base type EduCard. If there are two EduCards tracked by the application where each card has type base card and extra content card (that are related to the base card) respectively, the system displayed the extra content in the base card. For example, placing the “Earth” card together with the “Moon” card, the 3D animation showed that the moon circulated the Earth with audio explanation. Figure 10 shows the result of the proposed system.

This paper is supported by Ministry of Science, Technology & Innovation under the ScienceFund grant vote 4S135 and Universiti Teknologi Malaysia Research University Grant vote 11H81. The authors would like to acknowledge Universiti Teknologi Malaysia (UTM) for providing the research facilities to work as well as the experts to guide.

REFERENCES [1]

[2]

[3] [4]

[5] [6]

VII. CONCLUSION The Interactive Holographic Display is one of the starting point to revolutionize the holographic application with user interaction using AR tracking technique, such as can improve the enjoyment of the student toward learning and allows them to actively participate in the classroom. However, there are still have some limitation observed during the development of this project, such as the use of small monitor that results in small pyramid prism and hence, reduce the size of the holographic object generated on the pyramid prism as well as reduced the space that can display necessary information, such as graphical user interface (GUI). In the future research, the usability evaluation of the proposed system is planned to conduct in order to collect the users’ experience, especially the students and the teachers, in order to improve the 3D holographic display technology using AR in the education field.

[7] [8] [9] [10] [11]

[12]

[13] [14]

(a)

(b)

[15]

[16]

(c)

(d)

Fig. 10. The result of the system: (a) EarthAR card is presented; (b) MoonAR card is presented with EarthAR card, then a moon is shown beside Earth; (c) AtmosphereAR card is presented with EarthAR card, then the atmosphere is hightlighted; (d) MovementAR card is presented with EarthAR card, then the rotation axis is shown.

[17] [18] [19]

Wu, H. K., Lee, S. W. Y., Chang, H. Y., & Liang, J. C. (2013). Current status, opportunities and challenges of augmented reality in education. Computers & Education, 62, pp. 41-49. Nazri, N. I. A. M., & Rambli, D. R. A. (2014). Current limitations and opportunities in mobile augmented reality applications. In Computer and Information Sciences (ICCOINS), 2014 International Conference on (pp. 1-4). IEEE. Perozo, J., Lo Leung, M., & Ramírez, E. (2016). HoloMed: A Low-Cost Gesture-Based Holographic. arXiv preprint arXiv:1607.05812. Sutter, J. D. (2010). New phone apps seek to 'augment' reality. CNN. Available from: http://edition.cnn.com/2009/TECH/10/24/tech.augmented.reality.apps/in dex.html?_s=PM:TECH. [7 June 2017]. Youlalight (2017). 3D Holographic Pyramid. Available from: http://youlalight.com/en/3dpyramid. [7 June, 2017]. Bovier, F., Caggianese, G., De Pietro, G., Gallo, L., & Neroni, P. (2016). An Interactive 3D Holographic Pyramid for Museum Exhibition. In Signal-Image Technology & Internet-Based Systems (SITIS), 2016 12th International Conference (pp. 428 – 434). IEEE. Holus (2017). The World’s First Interactive, Tabletop Holographic Display. Available from: https://hplustech.com/. [11 June 2017]. Lee, K. (2012). Augmented reality in education and training. TechTrends, 56(2), 13-21. Azuma, R. T. (1997). A Survey of Augmented Reality. Presence: Teleoperators and virtual environments. 6(4), pp. 355-385. AR Lab (2017). What is Augmented Reality?. Available from: http://www.arlab.nl/content/what-augmented-reality. [10 June 2017]. Hirota, G., Chen, D.T., Garrett, W.F., & Livingston, M.A. (1996). Superior augmented reality registration by integrating landmark tracking and magnetic tracking. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques (pp. 429-438). ACM. Soares, A.B., Júnior, E.A.L., de Oliveira Andrade, A., & Cardosa, A. (2012). Virtual and Augmented Reality: A New Approach to Aid Users of Myoelectric Prostheses. In Computational Intelligence in Electromyography Analysis - A Perspective on Current Applications and Future Challenges. pp. 409-426. Höllerer, T., & Feiner, S. (2004). Telegeoinformatics: Location-Based Computing and Services. Taylor and Francis Books Ltd, London. Weng, E. N. G., Khan, R. U., Adruce, S. A. Z., & Onn, Y. B. (2013). Object Tracking from Natural Features in Mobile Augmented Reality. Procedia-Social and Behavorial Sciences. 97, pp. 753-760. Muhammad, A. B. A., Ismail, N. A. F., & Sunar, M. S. (2016). Reflective Prism Display Using Pepper’s Ghost Technique Software Toolkit Plugin for Unity 3D. Jurnal Teknologi (Sciences & Engineering). 78(12–3), pp. 189-196. Maass, U. (1999). U.S. Patent No. 5,865,519. Washington, DC: U.S. Patent and Trademark Office. Unity Manual (2017). Camera. Available from: https://docs.unity3d.com/Manual/class-Camera.html. [28 August 2017] Unity Manual (2017). Prefabs. Available from: https://docs.unity3d.com/Manual/Prefabs.html. [28 August 2017] Unity Manual (2017). Animator Controller. Available from: https://docs.unity3d.com/Manual/class-AnimatorController.html. [28 August 2017]