To appear in Niklasson, L., Bodén, M., and Ziemke, T. (Eds), ICANN 98,
Perspectives in Neural Computing, Springer-Verlag
Neural Control of a Virtual Prosthesis Lars Eriksson Dept. of Electrical Measurement, Lund Institute of Technology Lund, Sweden
[email protected]
Fredrik Sebelius Dept. of Electrical Measurement, Lund Institute of Technology Lund, Sweden
[email protected]
Christian Balkenius Lund University Cognitive Science Lund, Sweden
[email protected]
1 Introduction The abilities of the currently existing hand prostheses are typically limited to opening or closing the hand. This limits the usefulness of the prosthesis considerably compared to the many degrees of freedom in an intact hand. In order to develop more advanced hand prostheses two main problems have to be solved. The first is to develop more advanced mechanical solutions that allows for more degrees of freedom. The second, that we address below, is to devise a way of controlling the additional dexterity of such a prosthesis. Before the second problem is solved, the development of more advanced prostheses will be severely hindered. The most natural way to control a prosthesis would be in the same way as the hand is controlled: by neural control mediated by the nerves intended for the amputated hand or arm. The system we present below is an implementation of this type of control that exploits that the muscle activity of the remaining part of the arm is correlated with the desired action of the prosthesis. By recording from several surface mounted electrodes on the arm, we are able to predict the corresponding motion of the hand using an artificial neural network. Figure 1 shows the principle of the system. Eight electrodes record EMG signals from the limb. These are subsequently fed into a preprocessing stage and then into a variant of the self-organizing feature map that learns to recognize categories of muscle patterns. These categories are then associated with the appropriate motor action recorded from a data glove. The signal processing and classification methods are based on our earlier work with signals recorded from rats [1, 2]. At the current state of the project, we are using surface mounted electrodes because they do not require surgery and are easy to manage. While the surface mounted electrodes seem to provide enough information, implanted electrodes have better signal conditions and would therefore enhance performance. The benefits of surface mounted electrodes would however come about to the expense of more complex filtering and pre-processing. Since no prosthesis with the desired mechanical properties exist, we use a simulated hand generated on a computer screen
in an OpenGL environment. This hand shows the movement of the virtual prosthesis corresponding to the EMG patterns sent to the neural network. The following sections describe the processing of the EMG signals and the architecture of the neural network used. In the last sections we describe preliminary results with a patient with a congenital amputation of the left hand. After training the neural network, the patient was able to control the virtual prosthesis by will. Virtual Prosthesis on Computer Screen
Output
EMG Input ANN
DataGlove Target
Figure 1: A virtual prosthesis can be controlled by EMG signals that are interpreted by a neural network.
2 Recording of EMG Signals The EMG signals were recorded using sixteen electrodes and sent to a specially built eight channel bipolar amplifier. The amplifier uses a switched notch filter at 50Hz. The signals were then digitized at 8 kHz on a standard analogue to digital converter card on a PC. Figure 2 (top) illustrates the signals picked up from electrodes mounted on a patients lower limb. The index finger was moved back and forth two times during the measurement. We only want to retrieve the important information in the muscle signals. If this is not achieved then the network will, of course, give poor results. Thus, we reduce the data as much as possible to create a smooth function with a small time delay (Figure 2 bottom).
x(t)
0.6
d(t) 0 0
t
3.82 s
Figure 2: (top) One of the raw muscle signal as recorded from the patient’s left limb. From left to right, the picture shows the measured signals derived when moving the finger twice. (bottom) The signal after pre-processing.
3 Preprocessing In order to reveal the true signals, we have employed a custom made preprocessing methods. These are in order (figure 3): (1) rectification of the signal, (2) correlation between individual channels are used to remove inter-channel crosstalk, (3) a special band-pass filter that uses time versus amplitude differences in the signal, (4) downsampling to remove redundant information and reduce memory storage. These methods are applied to each of the eight channels. The following sections describe them in detail.
Mean
+
From other channels
Σ
LP-filter
f(t)
Interchannel
yi(t)
L1(t)
m(t)
x(t)
Delay Line
L2(t)
–
h(t)
Rectification
s(t)
1000
d(t)
Downsampling
LP-filter
Figure 3: The preprocessing of the EMG signals
3.1 Rectification To rectify the recorded signals, we let a floating value represent the mean of the signal, and then mirror the activity round it. This is necessary since the muscle signals contain both a positive and a negative peak [3]. With this method we achieved an adequate amplitude for the signal. The formula is: f (t ) = x(t )− m(t )
(3.1)
where x(t) is the sampled value and m(t) is the floating mean of the original signal.
3.2 Correlation Between Channels In order to inhibit inter-channel cross talk we used the following algorithm: K
g(t ) = (1− α )g(t − 1) + α
∑f
k (t
k =0
K hk (t ) = f k (t ) f k (t) k =0
∑
−1)
(3.2)
−1
g(t )
(3.3)
where K is the number of channels, α is a constant and g(t) represents the floating mean of the total activity of the signal.
3.3 Band-Pass Filtering The next step is to band-pass filter the signals. This is done using two third order low-pass filters as follows: L0 (t ) = h(t ) Li+1 (t) = γLi (t ) + (1− γ ) Li (t −1)
(3.4)
The signal is divided into two parts with different parameters in the filters. The difference between the outputs from the two low-pass filters are then calculated: s (t ) = L3γ 1 (t) − L3γ 2 (t )
(3.5)
The two time constants γ1 and γ2 generate differences in the two low pass filters and we thus achieved a band pass filter. In order to decrease the storage space needed it is now possible to down-sample the data set without losing any important information. N −1
d(t) =
1 N
∑ s(tN + i)
(3.6)
i= 0
Figure 2 (bottom) shows the outcome of the preprocessing stage. Notice how the essentials of the signal are revealed. We can see that a great clarification has been achieved through the amplitude differentiation algorithm. Because of the dramatic decrease in density, the storage space required is also far less than that of the original signal, which in turn boosts network performance. Finally we normalize the signals before we send them to a delay line which constructs the input vectors for the categorization network.
4 Classification of EMG Patterns For classification, we use the self-organizing feature map [4] modified with the conscience mechanism suggested by DeSieno [5]. Additionally, we bias each input signal to the network with the estimated variance of that signal in such a way that a signal with small variance is allowed to make a larger contribution to the category. The variance is calculated locally at each node for the signals that are categorized as belonging to its class. The basis for the conscience mechanism is that all nodes in the feature maps should be used. To make this possible, the selection of the winning node is biased by how much it has been previously activated. Nodes that have not been activated very much gain an advantage in the competition. To accomplish this, we calculate an estimate, f, of how much a node has won the competition, f i (t + 1) = f i (t )(1− β ) + βzi (t)
(4.1)
where zi (t) is 1 when node i wins at time t, and 0 otherwise. The parameter β sets the time horizon for the conscience mechanism. This estimate is used to calculate a bias for node i as,
1 bias i (t) = λ − f i (t +1) N
(4.2)
N is the number of nodes in the network and λ is a constant. The calculation of the distance to the winning node is biased in the following way [5], n−1
dist i (t) =
∑ ( x (t ) − w (t )) i
i
2
(1− vi (t ) ) − biasi (t )
i= 0
(4.3)
where vi is a floating variance estimate, v i (t +1) = γ ( xi − wi ) + (1− γ )vi (t) 2
(4.4)
In this way each node determine how substantial each input is. Changes to the weights are subsequently done in the standard way [4]. Finally, each node in the network is associated with the corresponding hand posture recorded by the data glove. On recall, these stored hand signals are used to predict the movement of the virtual prosthesis. The inclusion of the conscience and the variance mechanisms improves the learning speed and the correctness of the classification compared to the standard SOFM-network [1]. Predicted
max Index finger flexion 0
Actual
Error
0
t
3.82 s
Figure 4: The actual finger position recorded from the data glove (black) compared to the predicted finger position (gray) generated by the neural network and the difference between actual and predicted position.
5 Experiment and Results A 24 year old patient who was born without a left hand tried to imagine moving her left hand in the same way as the intact right hand. The position of the intact right hand was recorded by a data glove simultaneously with EMG signals measured at her left limb. Eight bipolar surface mounted electrodes were used. The patient performed five different movement patterns with her hand. Each finger was flexed individually and each movement was performed twice. In all, signals were recorded in four sets at two different occasions. Data from the first session were used to train the network, and data from the second session were used to verify the performance of the network.
Figure 4 shows the behavior of the system for the index finger during two movements. It can be seen that the neural network closely follows the actual movement of the finger. Similar results were obtained for the other fingers and for other movements. The difference between actual and predicted finger position is less than what is required to control a prosthesis. Also note that in this experiment, the patient is not allowed to learn the behavior of the virtual prostheses. If this were allowed, even better performance could be expected.
6 Conclusion Our experiment shows that neural networks are feasible for categorizing patterns of EMG signals. The signals recorded by the surface electrodes are sufficient to control the movements of a virtual prosthesis. The presented method offers great potential for the development of future hand prostheses.
Acknowledgements This research was performed at and partly supported by the Dept. of Hand Surgery at Malmö University Hospital. We would like to thank Göran Lundborg for initiating and promoting the project, Styrbjörn Lindberg for the effort he put into constructing the measurement hardware, and Birgitta Rosén for help with the measurements. Earlier stages of this project were achieved with the kind help of Ingmar Rosén, Dept. of Clinical Neurophysiology, Lund University and Lars Montelius, Dept. of Solid State Physics, Lund University. We also thank Nissho Electronics, Japan, for lending us the SuperGlove used in the experiments.
References [1] [2]
[3] [4] [5]
Eriksson L. Sebelius F. Pattern recognition of nerve signals using artificial neural networks. M.Sc. thesis, Department of Solid State Physics, Lund University, 1996 Montelius L, Sebelius F, Eriksson L, Holmberg H, Schouenbourg J, Danielsen N, Wallman L, Laurell T, Balkenius C. (1996). Pattern recognition of nerve signals using an artificial neural network. Proceedings of the 18th Annual International Conference of the IEEE Engineering in Medicine and Biology Society , 1996 Deutch, S. and Deutch, A. Understanding the Nervous System. IEEE Press, New York, 1992 Kohonen, T. Self-organization and associative memory. Springer-Verlag, Berlin, 1984 DeSieno, D. Adding a conscience to competitive learning. IEEE International Conference on Neural Networks, IEEE Press, New York, 1988, vol. 1, pp 117--124
