2004. 2. Doug A. Bowman, Ernst Kruijff, Joseph J. Laviola, JR, Ivan Poupyrev, "3D User Inter- ... Mario Gutierrez, Daniel Thalmann, Frederic Vexo. "Semantic ...
Semantic Integration for Virtual Devices Manipulation using Ontologies Daniela Calderón12, Sylvia Irawati12, Heedong Ko12 1
Department of Human Computer Interaction and Robotics, University of Science and Technology 2
Imaging Media Research Center, Korea Institute of Science and Technology {dcalderon,sylvi,ko}@imrc.kist.re.kr
Abstract. This paper describes a framework that uses ontology in processing the input speech commands and grounding their utterances to the corresponding manipulation commands in a context-aware environment. The semantic integration of the components of this framework is done using ontology. The ontology includes information of the application world and the virtual devices in it. The user inputs are semantically integrated using the device related information of the ontology, along with the current context, in order interpret them as manipulations commands the system can understands and effect the changes to render the scene.
1 Introduction Semantic interpretation of user’s input is an important task in virtual environments. Speech as the system’s input requires parsing the spoken words and giving semantic meaning to the utterances in order to compose manipulation commands that the system can understand. To generate an accurate semantic meaning from the recognized raw text, other input modalities, and system’s context have to be known. To address this issue, in this paper we present a framework that semantically integrate the user’s inputs and context using ontologies. The ontologies describe the virtual devices and the application environment. This information is used to process what is intended by the user’s actions. For the system to understand the intentions of the spoken commands and other inputs, the interaction manager will make the semantic integration for the interaction between the user and the system. This paper is arranged as follows. First we present previous works related to ontologies used in object manipulation. Afterwards the multimodal interaction framework is explained. Next we present the interaction manager responsible of the semantic integration with the help of ontologies. The implementation and results of using our methodology as part of the multimodal framework in context-aware virtual environments is then presented. Finally, we draw our conclusions and present future works.
2 Related Works A number of researches have improved multimodal object manipulation using ontologies. Each of them has used different kinds of procedures in order to enhance the best performance of their applications. Gutierrez et al. [4] proposed an object ontology definition for complementing semantic models. This methodology maps the output of the interaction device to a particular virtual entity. In this work they use object ontology to express the relationships between interaction devices and virtual entities in virtual environment. In another of their works [5], they describe the semantic representation of the functions, characteristics and relationships between virtual objects, with the aim of having adaptive entities to reutilize them in a variety of contexts without re-implementing the application. They turn the objects in the virtual environment into autonomous and reusable entities. In our approach we use ontologies to process the user’s input and context semantic integration in order to manipulate virtual devices in a virtual environment. We use a domain independent device conceptualization as part of the ontology describing general virtual entities of virtual devices in combination with a domain dependent world application conceptualization that describes the application environment. This ontology in collaboration with the context information will compute the semantic integration.
3 Multimodal Interaction Framework A multimodal interaction framework for virtual device manipulation is intended to be used for developing multimodal interaction in virtual environments. Here is explained the main components of the framework involve in the semantic integration process during the interaction.
3.1 Speech Server and Speech Manager One particular case of modality is the use of speech. For using speech we need a speech server. The speech server uses a utility grammar file. The grammar file has the information about the possible recognized utterances and what are the semantics of the raw recognized text. The grammar file is automatically generated in an offline process using the ontology. The ontology will provide the information of the possible recognized lexicons and the corresponding semantic interpretation. For example the ontology has the information about all the virtual devices that can be manipulated by the user, and the information about the current design of the environment. With this information the grammar file is constructed and so the possible recognized speech will know the possible virtual devices within the defined environment the user may want to interact with via speech.
After having the interpretation result the speech manager sends those values to the interaction manager to integrate with other inputs and context for further processing. 3.2 Device Server and Device Manager The interaction devices are connected to the device server. The device server is responsible for capturing the user input and sending feedback to output devices. Interaction devices are such as joystick, tracker, wand, spidar; or output devices like displays. The device server sends the received values to the device manager. In the device manager this values get semantic meaning depending on the current application domain. For this study, the interaction devices control de user’s position within the virtual environment. The user’s position affects the camera position when rendering the scene and his physical context. These values are then passed to the interaction manager for further processing. 3.3 Context Manager In order to create an immersive interaction in a virtual environment, the system has to be aware of the current context. Context-aware systems are controlled with interpreted human actions. The context can refer to how we are, what we are doing, where are we doing it, why we are doing it, etc. in order to simplify the control of the systems. If seen from the context-aware interaction it can have different points of view such as computational, physical, anthropological, cultural, etc. The context used in this system can be classified as user context or environment context. When talking about the user context we refer as variables such as user current location and available devices in the location, mainly physical context, in order to know what his intentions are. The environment context refers to the current simulated behaviors or actions in the environment, changing of a device state as effect of user interaction, or time of the day, etc. Depending on the context the semantic integration will have different effects in the virtual environment. The context information is part of the ontology. In this case the ontology will act as the context repository to retrieve information of the current state of the world. The context manager is in charge of sending this information to the interaction manager. For example if the user interact with a device, the changes of that device will be reflected in the context repository, this information will be send to the interaction manager and the changes will effect in the next scene rendered. And if the user refers to an device that appears in many locations of the environment, the physical context will help the system complete the manipulation command. 3.4 Device Simulator In order to interact with the virtual devices, we create a device simulator. The device
simulator is in charge of starting, managing and stopping the virtual devices. The simulator has all the information about the devices interacting in the system, their services and current status. The device simulator changes the status of the devices depending on what the interaction manager commands it to execute. The device simulator receives the manipulation commands from the interaction manager after the semantic integration and updates the information in the context repository via the context manager, information that will be used posterior to render the scene.
4 Interaction Manager The role of the interaction manager is to process the values of the user’s commands via speech and/or interaction devices and the current context and integrate them into a manipulation command in order to manipulate the virtual devices. The interaction manager makes the semantic integration using the information in the ontology. The ontology has the information about the virtual devices and the world state. In addition to this the interaction manager also retrieves the system current world state in order to compute the correct manipulation command or the system’ s autonomous reactions.
Fig. 1. Multimodal Interaction Framework 4.1 Virtual Device Conceptualization The virtual device concepts in the ontology describe all the information about the devices involved in the application. This part of the ontology describes each virtual device in detail, including services, actions, and state variables. The information of the state variables may change as the interaction is developed. This information is created based on the functionality of the device so the system can integrate the semantics of the user’s inputs referring to particular devices. During the semantic integration process executed in the interaction manager these information helps in the grounding of the utterances of the spoken words and/or the user inputs via interaction devices, to the corresponding commands that are send to the device simulator for further execution. In Figure 2 we present a summarize example of a device information in the ontology.
Fig. 2. Device Information in the Ontology The device ontology is domain independent. The devices defined can be reused from application to application. The design of the environment will determined the differences in the application. This is explained below. 4.1 World Conceptualization The ontology describes the system’s environment to define is the purpose of the application. The overall ontology, world and devices, help in the semantic integration to process the understandable manipulation command for the system. This part of the ontology contains the description of the environment, how the virtual objects are distributed, and their purposes. This information is used to resolve the physical context within the interaction. This information in the ontology is domain dependent due to the fact that it will change from application to application depending on the objects used in the different applications and the purpose the applications were made for.
5
Implementation
We have implemented a prototype application based on the presented framework. This application simulates a house composed with virtual devices. These devices are house hold devices such as television, lights, air-conditioner, etc. The purposed of the application is to simulate a real intelligent house environment in which the user can manipulate the devices in his house, and also the devices interact by themselves depending on the current context. Our implementation is based on NAVERLib [8], microkernel architecture in the distributed network environment. It provides libraries for a variety of interactions, interfaces, and virtual contents than can be composed in the VR system. In order to communicate with the interaction device, we use VRPN library [9] which consists of a set of classes within library and a set of servers that implement a device-independent, network transparent interface between application programs, and the set of physical devices (trackers, buttons, etc.). The device server, in which the
interaction devices are connected, works as VRPN server, whereas the device manager works as a VRPN client. The device manager communicates with the device server to get the device values or send the reaction feedback to the output device. The virtual devices are simulated using a device simulator. This device simulator create virtual devices such as the ones mention before, and gives the possibility to receive commands from the application in order to operate the devices or retrieve information from them. The context manager and device simulator are based on CAIM [12]. The ontology is design using Protégé [10][11]. We used Microsoft Speech API 5.1 for speech engine. The grammar is defined in Speech API text grammar format. It is used to define the phrases recognized by speech recognition engine and their semantic interpretation. The current implemented grammar can recognize command for the devices such as “Power the light on”, and so on in each of the devices and their particular services provided. 5.1 Interaction Loop Before the system is initialized, the speech grammar file is created from the ontology. Then the device simulator is initialized using the world information for the current application from the ontology. Refer to Figure 3, step 0. After this the interaction loop begins. At the interaction loop, first the speech server, using the speech grammar definition file, recognizes the raw speech input. At the same time the device server gets the input values of the joystick. Refer to Figure 3, step 1. At this point the device server, including the speech server, will send all the collected raw data together to the device and speech manager for further processing. Refer to Figure 3, step 2. In the device manager the input values from the joystick are mapped semantic variables that will be used in the interaction manager. The speech manager sends the speech interpretation result to the interaction manager. This interpretation result is the semantic interpretation of the raw speech. This result is constructed using the speech grammar file created from the ontologies. In the interaction manager the input values from the speech server and device server are used in collaboration with the information coming from the context manager and the ontologies to integrate into a semantic meaning. Refer to Figure 3, step 3. The semantic integration is used to construct the manipulation command that is send to the virtual device simulator to execute the interaction. Refer to Figure 3, step 4.
Fig. 3. Interaction Loop The device simulator executes the command and updates the values in of the context, which are after process in order update new world state when rendering the scene. Refer to Figure 3, step 5. The context manager continuously updates the world state in the ontology. This information is used in every interaction loop to integrate all inputs and expose the changes of the new scene in the virtual application. When this new scene is rendered, parallel processing can be triggered depending on the new context. The interaction loop will continue it course a long all the interactions.
6
Results
We have tested our framework using ontology containing information about virtual devices and environment configuration. The ontology is used in combination with the context manager for the semantic integration in the multimodal object manipulation for virtual devices in a virtual environment to create an immerse feeling to the user of been in a context-aware space for interacting with virtual devices. This application has been tested in a CAVE Environment. CAVE system is suitable to give users a wide angle of field of view and a fully immersive feeling. The joystick controls the current user position inside the virtual environment, in this particular case, in the house. The system also provides a stereo graphics which is seen through the stereoscopic glasses. The speech grammar file is automatically generated by the system in accordance to the ontology. The application is conform of various rooms, each with different virtual devices. Among these devices we have lights, television, air-conditioner, computer, oven, etc. And each of the devices provided services that can be manipulated via speech in the application. Also depending on the current context the devices might change automatically their own properties to beneficiate the user. This application integrates context-awareness and multimodality to create an immersive space in the virtual reality environment to simulate an intelligent house. The user can manipulate the virtual devices using speech and can navigate around the house using the joystick.
7
Conclusions and Future Work
We have present a context-aware multimodal interaction framework that combines information form ontologies to integrate the semantic meaning of the user inputs and map them to the corresponding virtual devices in order to create the correct manipulation command in order to succeed in achieving the user intentions in the virtual environment. The key point of this framework is how to manage the information provided from the ontologies in order to create the semantic interpretation of the user interactions with the system. In the future, extending the usage of this methodology can be done. For this more interaction devices have to be added to the device ontology and more application ontologies have to be created to test the framework in different domains.
Acknowledgment This research is supported as part of the Ubiquitous Autonomic computer and Network Project of the Ministry of Information and Communication (MIC) 21st Century Frontier R&D Program in Korea.
References 1.
2. 3. 4.
5.
6.
7. 8.
9.
Christopoulou, Eleni et al. An Ontology-based Conceptual Model for Composing Context-Aware Applications. Research Academic Computer Technology Institute. Patras, Greece. 2004. Doug A. Bowman, Ernst Kruijff, Joseph J. Laviola, JR, Ivan Poupyrev, "3D User Interfaces Theory and Practice", Addison-Wesley, Person Education, Inc., Boston, 2004. G. Smith, W. Stuerzlinger, "Integration of Constraints into a VR Environment", VRIC, Virtual Reality International Conference, 2001. Mario Gutierrez, Daniel Thalmann, Frederic Vexo. "Semantic Virtual Environments with Adaptive Multimodal Interfaces," mmm, pp. 277-283, 11th International Multimedia Modelling Conference (MMM'05), 2005. Mario Gutierrez, Frederic Vexo, Daniel Thalman. “Semantic-based representation of virtual environments”. International Journal of Computer Applications in Technology 2005 – Vol. 23, No. 2/3/4 pp. 229-238. G. Heumer, M. Schilling, and M. E. Latoschik, "Automatic data exchange and synchronization for knowledge-based intelligent virtual environments", In Proceedings of the IEEE VR2005, pp. 43-50, Bonn, Germany, 2005. OpenCyc, http://www.opencyc.org/ Park, C.H, Ko, H., and Kim, T, "NAVER: Networked and Augmented Virtual Environment Architecture; design and implementation of VR framework for Gyeongju VR Theater", Computers & Graphics, 27, pp. 223-230, 2003. Russell M. Taylor II, Thomas C. Hudson, Adam Seeger, Hans Weber, Jeffrey Juliano, Aron T. Helser, "VRPN: A Device-Independent, Network-Transparent VR Peripheral System", ACM Symposium on Virtual Reality Software and Technology, pp. 56-61, 2001.
10. Salber, D., Abowd, G. The Design and Use of a Generic Context Server, Technical Report GIT-GVU-98-32, Georgia Institute of Technology, 1998.N. Noy & D. L. McGuinness. Ontology Development 101: A Guide to Creating Your First Ontology. 2001. 11. Dameron, N. F. Noy, H. Knublauch, & M. A. Musen. Accessing and manipulating ontologies using web services.. Third International Semantic Web Conference (ISWC2004), Hiroshima, Japan, 2004. 12. Sangchul Ahn et al. The Interaction Framework of Context Aware Systems. 2005.
