computer screen. On the screen, there is a game which is made by our proposed algorithm. In real time, subjects are able to observe gathered brain wave data ...
Fifth International Conference on Software Engineering Research, Management and Applications
Implementation of a 3-Dimensional Game for developing balanced Brainwave Beom-Soo Shim, Sung-Wook Lee and Jeong-Hoon Shin Dept. of Computer Information & Communication Engineering, Catholic University of Daegu, Korea
1. Introduction
Abstract In a ubiquitous condition, HCI(Human Computer Interaction) is popularly used to meet users’ needs for high quality service. In a condition with HCI, there are a lot of interfaces that are used to satisfy the needs of the users. As a result of these changes, various types of effective human computer interface methods have been developed. In many studies, researchers have focused on using brain-wave interface, BCI(Brain-wave computer interaction). BCI is one of the research areas in HCI. Nowadays, studies which are related to BCI are under way to find effective methods of controlling and collecting brain-wave. Most researches related to BCI are not centralized and not systematic. BCI research uses brain wave which is gathered from the subjects’ scalp. The brain wave is affected easily in an experimental condition. Moreover, subjects do not control their brain well when they attend the research. These problems bring about ineffective results research. In most research to related in HCI and BCI, that is to say – pattern recognition, the most important foundation of the research is to set a standard about subjects’ mindset and attitude in research. For this factor, in this paper, we propose the Implementation of a 3-dimensional game for developing balanced brain wave. Apart from other kinds of pattern recognition, In BCI, it is difficult to gather the specific brain wave researchers want. Researchers do not know effective methods for gathering the brain wave they want. Subjects also do not know how to release the brain wave researchers expect. To solve these kinds of problems, we propose a novel game for developing balanced brain wave. The game works by balancing the power of the brainwave values from the left cerebral hemisphere and right cerebral hemisphere. Subjects look at a computer screen. On the screen, there is a game which is made by our proposed algorithm. In real time, subjects are able to observe gathered brain wave data and therefore teach themselves to produce the appropriate brain wave. To verify the effectiveness of our proposed system, we analyzed the difference of brain wave gathered form the left and right cerebral hemisphere. On the basis of the balanced left and right cerebral hemisphere analysis, we propose the Implementation of a 3-Dimensional game for stably developing balanced brain-wave.
0-7695-2867-8/07 $25.00 © 2007 IEEE DOI 10.1109/SERA.2007.94
Until recently, controlling computers by human thought was science fiction, but it’s rapidly becoming science fact. The last decade has witnessed a rapidlygrowing body of research and technique development involving detecting human brain responses and putting these techniques to appropriate uses to help people with debilitating diseases or who are disabled in some way. The definition of BCI reflects the principal reason for proving new augmentative communication technology to patients with neuromuscular impairments who are paralyzed or have other severe movement deficits. Also, Brain Computer Interaction (BCI) stems from a need for alternative communication and control options for individuals with severe disabilities( e.g amyotropic lateral sclerosis), though its potential uses extend to rehabilitation of neurological disorders, brain-state monitoring and game. The most practical and widely applicable BCI solutions are those based on noninvasive electroencephalogram (EEG) measurements recorded from the scalp. In general, BCI offers the possibility of communication for people who are paralyzed to input letters on a device or give order to the smallest embedded system. In addition, the BCI in conjunction with exciting multimedia applications, e.g., a dexterity discovery, could define a new level of control possibilities for patients as well to control information directly from their brains, as reflected in EEG signals which are recorded non-invasively from the scalp. But, there are some limits using BCI. Brain wave has a very weak signal, affected a lot by the surroundings, and many errors happen according to the electrode pole stability when brain wave is gathered, which makes it difficult for the researcher to gather necessary brain-wave. It is common that a researcher wants to gather brain-wave in a correct and an efficient way while avoiding the irrelevant brainwave this can be time-consuming. It is difficult to gather sable and meaningful brain wave data from subjects. To solve this problem we propose a 3dimensional game that will help to developing balanced brain wave. Without brain wave gathering training, researchers can not gather the brain wave data which they expect. Also, it is difficult for subjects to concentrate on BCI research because of the absence of standards for subjects who are in the
751
research. Many studies proposed an efficient way to get the brain wave researchers expected from subjects. Some developed an intelligent algorithm for gathering brain wave from the scalp and introduced effective method expected to gather brain wave. However, there are some limits in the ways. So, we proposed the implementation of a 3-Dimensional game for which can train subjects’ to use their brain in a stable and controlled way. Moreover, most of the research focuses on a standard method to make subjects concentrate on the experiment which works independently and also depends on the platform that is used. The trend research way of making subjects concentrate on the experiment is inefficient and non systematic. There are no standard for research environment. Consequently, our proposed training game suggests a standard way to control a subjects’ concentration. In BCI research, it is important for subjects to being gathered for accuracy and reliability of the brain wave. Being gathered in BCI research, limits encountered are the analysis of brain wave. Our proposed training game could improve gathering expected and useful brain wave. The brain wave is affected easily in an external condition. It is necessary for researchers to set the standard concentration condition method which other researchers agree to use. In order to agree with the standard concentration condition, we propose the implementation of 3-Dimensional game for developing balanced brain wave. Presented training game is the performance of the real-time BCI game “Training Brain balance” when played by normal subjects. The design of the Training Brain balance game was split into two parts. The first part is the analysis of left and right cerebral hemisphere’s power value. The training game works through the analysis of left and right cerebral hemisphere power value. The second part is analysis of brain wave device which makes it possible subject for to see the power values easily in real time. The game also contains visual and sound effects so that subjects can concentrate on producing brain wave easily. The goals of the proposed game focus on gathering the stable and meaningful brain wave. In chapter 2, we introduce some studies which are related to the brain wave training device and some game that use brain wave. In chapter 3, we will explain our brain wave training device and 3D game using brain wave we proposed. In chapter 4, we will introduce the experiment result and its further study.
2.1 Brain Computer Interface based on the Steady-State VEP for Immersive Gaming Control. [1] Lalor E, Kelly S.P., Finucane C., Burke R., Reilly R.B., McDarby G. 2.1.1 Objective In this paper the authors wish to address the application of the SSVEP(Steady State Visual Evoked Potential)-based BCI design to a real-time gaming framework. It is proposed that performance on the BCI game detailed below will be sensitive to neurological disorders such as Attention Deficit/Hyperactivity Disorder and thus may aid in its rehabilitation. Presented here is the performance of the real-time BCI game “MindBalance” when played by normal subjects.
2.1.2 Configuration MindBalance – the Game : The object of the MindBalance game is to control the balance of an animated character on a tightrope using only the player’s EEG. A checkerboard is positioned on either side of character. A screen-shot of the game can be seen in Figure 1.
Figure 1. The character loses balance during the game Signal Processing and the C# Engine: In order to carry out this study, a programming engine and platform were required, capable of rendering detailed 3-D graphics while processing continuous EEG data to control a sprite within the game at the same time. This was accomplished using a combined graphics, signal processing and network communications engine implemented in C#.
2. Related Works
2.1.3 Experimental Results
HCI using brain wave is a popular topic of research. Also, it is a popular topic for game using BCI. We gathered some related issues about training and feedback game using BCI. The following subsections describe these related studies.
Results of the study indicate that successful binary control using Steady State Visual Evoked Potentials is possible in an uncontrolled environment and is
752
resilient to any ill effects potentially incurred by a rich detailed visual environment as in the MindBalance game. The authors also propose to extend the results of the preliminary trials of this study to covert visual attention, in which subjects direct attention to one of two bilateral stimuli without eye movement.
2.2 Learning to Control Brain Rhythms : Making A Brain-computer Interface Possible [2] J. A. Pineda, D. S.Silverman,A.Vankov,and J.Hestenes. 2.2.1 Objective The authors hypothesize that a more efficient method for training subjects to control mu rhythm, while still meeting most of game conditions, involves visual representation of the signal. Hence, the goal of their study is to examine the mu rhythm and to determine the effects on learning while using a complex visual representation of the brain signal. In this case, the signal was mapped to navigational movements (i.e., left or right) within a 3-dimensional (3-D) first person shooter video game. Unlike the simple visual feedback conditions used by previous studies, this study involved training in a stimulus-rich, realistic, and motivationally engaging environment.
Figure 2. A view of the 3-D first-person shooter game. Subjects controlled the movement of the scenery either to the left by producing low mu or to the right by producing high mu.
2.2.3 Experimental Results The results of this study indicate that subjects learn to control levels of mu very quickly, but especially when this learning involves producing similar mu levels (whether high or low) over each hemisphere. They are able to maintain that level of control across ten training sessions. In contrast, subjects show an almost linear increase in learning across training sessions when a difference over each hemisphere in mu levels is necessary to achieve control.
2.2.2 Configuration Subjects were placed in a soundproof chamber and asked to look straight ahead at a computer monitor that displayed a high-resolution 3-D first-person shooter video game, as shown in Fig. 2. During the free running period, subjects were asked to explore the game by pressing the “s” key on the keyboard to move forward and the “x” key to move back. Right and left movements were controlled by “high” and “low” mu respectively. At the end of the free-running period, subjects began either the left- or right- movement period. Subjects were instructed not to touch the keyboard (thus keeping the environment on the screen stationary), but to attempt to rotate it left or right by producing “low” or “high” mu respectively. For a left movement, the subject was told to focus on rotating the environment only to the left(Thus making counter-clockwise circles). Similarly, for a right movement, the subject was told to focus on rotating the environment only to the right (thus making clockwise circles). When the subject completed the three periods of training, the session for that day ended.
2.3 Quantifying Mental Relaxation with EEG for use in Computer Games [3] T. A. Lin, L. R. John. 2.3.1 Objective The aim of this study was to investigate methods for the implementation of EEG based measurement of mental relaxation, and to demonstrate the potential of the interface with a simple game, where a simulated ball is controlled to move left or right based on player’s mental relaxation level. The Bulk of this research concerns investigating what frequency components of the EEG signals or the combinations of the frequency components best measures the user’s mental relaxation level, as well as how viable it is to be measured in real-time, so that it could be implemented in a game.
2.3.2 Configuration The EEG data gathered was large (+500 timewindows per test, per subject), thus parametric methods were used for analyzing the data. A three-
753
the Modular EEG device. The game application was also designed to be easily extended if other programmers wanted to create different neuro feedback games.
way ANOVA with repeated measures on three factors (tests, indices, and time-windows) was performed to analyze the data in MATLAB, in order to compare the difference between the three factors in the EEG data. Turkey HSD post-hoc tests were carried out on the selected significant factors. For all statistical analysis, a significance level of 0.05 was chosen. The game was implemented in JAVA, using open source libraries from Brainathon, and Sun’s COMMAPI library for the serial connection from EEG device to the PC. A simulated ball moves left or right along a horizontal axis controlled by the player’s EEG signals relating to level of mental relaxation. Two players can compete simultaneously to determine who is more relaxed by comparing the relative position of the ball. The controllability of the ball was assessed by subject’s perception of relaxation. See Figure 3 for a screenshot of the game interface.
2.4.2 Configuration The Brainathlon game consists of three consecutive mini-games called courses. Each course monitors and rewards activity in configurable brainwave ranges. The game can be played by one player or by two players simultaneously. When played in two-player mode, the player who wins the most events will win the game. When the game is played by two players, the screen is split in half vertically with player 1’s game board on the left and player 2’s on the right, as shown in Figure 4.1. When a single player is playing, only one game board appears on the screen, as shown in Figure 4.2. During all of the courses, a small display of current brainwave activity in the target frequency band is displayed at the bottom of the screen. This display provides additional feedback to the players and assists them in identifying increases and decreases in brain activity.
Figure 3. A screenshot of the game interface
2.3.3 Experimental Results This is the first study that attempted to measure mental relaxation state using one channel (Fp1-Fp2) EEG for game implementation. The EEG results indicate that the sum alpha + theta, and sum of alpha + beta + theta are good indices for the measurement of neurological relaxation. Game testing also reflects that these indices have the capability to measure the basic level of relaxation in at least half of the players.
Figure 4.1. Two-player mode
2.4 BRAINATHLON: Enhancing Brainwave Control Through Brain-Controlled Game Play [4] Palke, Amy. 2.4.1 Objective The goal of this project was to create a different type of application for the Modular EEG device. In the spirit of Brain ball, a competitive neuro feedback game was designed and developed. In addition to a software game, the aim was to build a reusable library of EEG acquisition and analysis components that could be used to build other applications for use with
Figure 4.2. One-player mode Each course has an accompanying XML configuration file where users can set the frequency
754
ranges, time limits, and other course variables. The configuration is read into the application at runtime, so changing the game to suit different players’ training interests and skill levels is easy.
As shown in figure 5, we constructed a game for brain training by comparing the power value of left and right cerebral hemisphere. Subjects can train the way they use their brain stably with the help of this game. The merit of this algorithm is it will help researchers gather brain wave data which is stable and balanced. The proposed game gathers raw data from the subject’s scalp. After gathering the brain wave, we process raw data of the left and right cerebral hemisphere using FFT filtering. The value of FFT level can be used to play games or end games by comparing the balance of the left and right cerebral hemisphere. Through this process, subjects can practice producing appropriate brain wave by playing this game systematically. The proposed game includes a real time analysis device which shows the power value of the left and right cerebral hemisphere. It is possible for subjects to practice creating brain wave and therefore use their brain wave to control a function. The brain game is played by aproposed algorithm which will be presented in the indicator. it makes it easy for the subjects to understand the brain wave data, that is being collected, in real time.
2.4.3 Experimental Results The data from one-player and two-player games was also analyzed separately to determine if either mode was more effective in increasing alpha activity. Although the player's average alpha amplitude was higher for those who played alone, both modes resulted in increased alpha activity with repeated game play.
3. Implementation of a 3-Dimensional Game for developing balanced Brain wave In our research, we proposed the implementation of a 3-dimensional game for developing balanced brain wave. It is an analysis game which analyzes raw brain wave gathered. By comparing the power value of the left and right cerebral hemisphere, the game is played. The proposed game, which was used in an experimental environment by the researchers, was designed carefully to provide data needed. The resource of the proposed game is in the analysis of the power value of the left and right cerebral hemisphere which is made from FFT (Fast Fourier Transform) algorithm. It can be used to develop subjects’ ability to control the left and right cerebral hemisphere in a stable way. Through this training, it is possible that the subject’s will be able to provide reliable and meaningful brain wave for the researchers. The subjects consist of 5 men and 5 women from Dae-Gu Catholic University.
3.1 Entire system The 3-dimensional game is used by comparing the power value of left and right cerebral hemisphere. Figure 6. Falling the rope by break balance left and right cerebral hemisphere
3.2 Algorithm of the system The proposed game works by using the FFT power value of the left and right cerebral hemisphere which is under in the selective environment. In order for subjects to release brain wave related to BCI research, subjects need to practice controlling their mind as much as possible. We use sound and visual effect, and the character in the game acts in a funny way to make subjects concentrate better. We set the game like this: if the subjects releases 120% more power than the average value of normal brain wave, the character in the game can move forward on the rope. 120% power value is the threshold for the character to move forward. In case when the brain wave is not balanced in power value of the left and right cerebral hemisphere, the character will fall from the rope.
Figure 5. Move forward by balanced power value of left and right cerebral hemisphere in proposed game 3D game
755
Considering the error of measuring equipment, we set a rule that its balance is broken if the power value of the left and right cerebral hemisphere is more than 10 units. The proposed algorithm enables the subjects to practice using and controlling brain wave by playing this game. Figure 7 shows processing FFT algorithm of 3-D game.
values from the left and right cerebral hemispheres stably. Figure 8 presents the architecture of the proposed training device which consists of three indicators. These indicators display brain wave power.
Figure 7. Algorithm of processing FFT in 3D game
Figure 8. Brain wave training device Among the three indicators, the left one presents the power of brain wave that comes from the left cerebral hemisphere, which the subjects can see easily in real time. The right one presents the power of brain wave that comes from the right cerebral hemisphere. The one in the middle presents the value (power value of left brain wave – power value of right brain wave). As in the picture shown above, with balanced brain-wave value the indicator will indicates ‘0 value’ in middle indicator. If it moves to the left, the power value of the left brain wave is stronger than the right brain wave, which means that the left cerebral hemisphere is more activated. If it moves to right, the power value of the right brain wave is stronger than left brain wave, which means that the right cerebral hemisphere activated. If subjects use either the left cerebral hemisphere or the right one more during the training, the balance of left and right cerebral hemisphere will be broken. Subjects will be able to notice this immediately by looking at screen with this training device. Then subjects will try to adjust the balance of the left and right cerebral hemisphere. The subject will be able to keep balance. The interaction of training device leads on improving the power value of the left and right cerebral hemispheres stably.
3.3 Features of system The important goal of our proposed 3D game is to train subjects to practice using brain wave systematically. Our 3D game provides more reliable brain wave. Our proposed 3D game improves data analyzed of a subjects’ brain wave by comparing the power value of left and right cerebral hemisphere. The results of experimental value are processed using FFT in real time. Among the processed value of FFT, the value of 2~8 Hz is θ (theta) value and the value of 12~20 Hz is SMR(lowβ (beta)) + middle- β (beta). They can be displayed as (SMR + middlebeta)/ (theta value) which is processed in each channel. The power value of the left cerebral hemisphere is input from channels 1 and 3 and the power value of the right cerebral hemisphere is input in channels 2 and 4. We call (SMR + middle-beta) / (theta value) the degree of concentration. It is used for the threshold value for moving forward on the rope and comparison value of the left and right cerebral hemisphere’s is used to control balance. In Figure 7 the entire process of analyzing brain wave data by using the FFT filtering is presented. Also, the degree of concentration can eliminate unnecessary band and provide the necessary the power value for researchers. By using this system, it is possible for us to study in a systematical way using trained subjects in BCI research. This processed brain wave data makes it possible for researchers to apply it to the related BCI research and to a lot of meaningful applications in a ubiquitous system.
4. Experimental Evaluation Our proposed 3D game is played by the power of brain wave which is more than the threshold value and by the analysis of left and right brain wave. This 3D game improves subjects’ ability to use their own brain wave. To end the 3D game, subjects should release stable and balanced brain wave. The programming language visual C++ was used in the development of this game.
3.4 Real-time brain wave analysis device As mentioned, we also proposed a real time brain wave analysis device. It is a training device that improves subjects’ ability to release balanced power
756
Figure 11. Results of play the 3D game The first time, subjects play the game it is easy for them. But, at second training, subjects have trouble when subjects play the game. Subjects release a high power brain wave so it is possible to move forward. But, the left and right cerebral hemisphere balance is broken by untrained subjects using brain wave. It ends the game because the character falls off the rope. However, when subjects try to play the game more and more, the results are better than each previous trail. In the final trail, we can find out that subjects release balanced and stable brain wave. In order to increase the difficulty of the game, researcher can set the value of threshold and balanced level. The result of the research is illustrated in Figure 12 and Figure 13. Through this research we can see that it is possible for the human’s brain wave to develop with brain wave training. Moreover, we adopted the use of real time system to allow subjects to notice the value of the brain wave according to the change of time and situation. Figure 12 presents training results of elapsed time before the end of the game.
Figure 9. System development environment using visual c++ Measuring brain wave equipment is QEEG-4 system (lxe3204). QEEQ-4 is made of a junction box, plate electrode and paste. Figure 10 shows specification of equipment.
Figure 10. Specification of brain wave measuring equipment
4.1 Results The proposed game system analyses the power value of the left and right cerebral hemispheres. It is possible for subjects to release balanced and stable brain wave of from the data left and right cerebral hemisphere. It gives us useful brain wave from raw brain wave. In Figure 11, we present the result of training 10 participants in brain wave research. In Figure 11, we see the optimized result that human’s brain wave can be controlled with brain wave training. Based on the graphical result, we can find out that the brain wave training has many possibilities in BCI. After brain wave training, most subjects release stable and reliable brain wave.
Figure 12. Average elapsed time (success) Figure 13 presents number of failures, which is when the character falls off the rope, therefore ending the game.
757
6. References [1] Lalor E, Kelly S.P., Finucane C., Burke R., Reilly R.B., McDarby G.: Brain Computer Interface based on the Steady-state VEP for Immersive Gaming Control, Biomedizinsche Tecknik, (2004)pp. 63-64 [2] J. A. Pineda, D. S.Silverman,A.Vankov,andJ.Hestenes.: Learning to Control Brain Rhythms: Making a BrainComputer Interface Possible, IEEE, (2003)pp. 181 [3] T. A. Lin, L. R. John.: Quantifying Mental Relaxation with EEG for use in Computer Games, International Conference on Internet Computing, (2006)pp. 409-415 [4] Palke, Amy.: Brainathlon, Enhancing brainwave control through brain-controlled game play, Master thesis, Mills College, (2004) [5] Kulman, W.N.: EEG feedback training: enhancement of somatosensory cortical activity. Electroencephalography and Clinical Neurophysiology, (1978)pp. 290-294 [6] Vankov, A. Instantaneous evaluation of EEG rhythms by variable epoch frequency decomposition. Society for Neuroscience Abstracts, (2000)pp. 840.2 [7] Mu¨ ller, K.-R., Anderson, C., Birch, B.: Linear and nonlinear methods for brain computer interfaces, IEEE Transactions on Neural Systems and Rehabilitation Engineering, (2003)pp. 165–169 [8] Harel, D., Carmel, L., Lancet, D.: Towards an odor communication system, Computational Biology and Chemistry 27, (2003)pp.121–133. [9] Peters, B.O., Pfurtscheller, G., Flyvbjerg, H.: Automatic differentiation of multichannel eeg signals, IEEE Transactions on Biomedical Engineering (2001)pp. 111– 116. [10] Delorme, A., Makeig, S. EEGLAB: an open source toolbox for analysis of single-trial EEG Dynamics, Journal of Neuroscience Methods, (2004)pp, 134:9-21 [11] Ming, C., Xiaorong, G., Shangkai, G., and Dingfeng, X.: Design and implementation of a brain-computer interface with high transfer rates, Biomedical Engineering, IEEE Transactions on, Vol. 49, No. 10, (2002)pp. 1181-1186 [12] Wolpaw, J.R., Birbaumer, N., McFarland, D.J., Pfurtscheller, G., Vaughan, T.M.: Brain–computer interfaces for communication and control, Clinical Neurophysiology 113, (2002)pp. 767–791.
Figure 13. Average elapsed times (fail) As a result of this research, we know that most subjects improve their ability to use brain wave stably. We know the proposed 3D game and training device system provides researchers with more capable, versatile, stable, and reliable data collected form trained subjects than trained subjects.
5. Conclusions The BCI is emerging in important research because it is easy to use in any special environment. This research will help to improve services for the disabled as it can be operated 24 hour a day. Brain wave signals are very weak and are affected by their surroundings a lot. Besides a lot of errors come out according to the electrode stability. Accordingly, we developed brain wave training device and made a 3D game using brain wave to produce balanced and stable brain wave of the left and right cerebral hemisphere By using this game consistently users will produce brain wave signal as a means of HCI more accurately through this training. In this paper, we proposed and implemented a 3dimensional game for developing balanced brain wave. In this experiment we found the fact that the possibility of winning the game increases as the brain wave training continues and the time to finish the game shortens. Through this we know that a human’s brain wave can be developed with brain wave training and it is possible to use it in the field of HCI. Subjects can perform very complicated functions if it is performed consistently. We tried to develop very effective brain wave learning devices and make very interesting games. The result implies that using our proposed 3D game can increase the reliability and stability of gathered brain wave and improve subjects using brain wave. This study will be used in many application researches and applied to the real life. The future BCI research will be focused on high quality of life with the development of various BCI techniques. The paralyzed patients and workers who work in a particular environment can control some robotic machines like controlling their own body by using our proposed 3D game training, which can ease the complaints made by researchers in BCI.
758
