Using the Human Body Field as a Medium for Natural Interaction Andreas Braun Darmstadt University of Technology Karolinenplatz 5
Pascal Hamisu Fraunhofer-IGD
64289 Darmstadt, Germany
Fraunhoferstr. 5 64283 Darmstadt, Germany
[email protected]
[email protected]
ABSTRACT In this paper we present a novel technique for integration in assistive environments, using capacitive proximity sensing to detect the presence of a human body, thus creating a medium for natural, deliberate or unaware interaction. As the world’s population ages, we witness a growing number of health-related issues and the need to simplify interaction with technologies that are getting ever more complex. We present implementation of hardware and software prototypes of this versatile and cheap technology that can be easily and unobtrusively integrated into ambient assisted living environments.
Keywords Human-centered computing, Proximity Sensors, filters and Interaction techniques
1. ITRODUCTIO In the past few years societies of industrialized countries have been confronted with the issues of an aging population. Modern seniors want to maintain a self-managed, socially included and healthy lifestyle in their familiar surroundings. However as health deteriorates, it becomes necessary to monitor body functions and take precautions to automatically call for emergency in case of accidents. Another challenging aspect for many elder people is the increasing complexity of technology and the necessity to interact with it. Capacitive sensing technology is commonly used for either touchsensing devices in consumer electronics or short distance (~1cm) detection in industrial applications. Our idea is using capacitive sensor devices to acquire proximity values and use the acquired data to interact with a computer system. This interaction can happen unaware for the user by relying on passive sensor arrays. Alternatively it can be used to directly interact with the system using hand gestures. We have developed a prototype based on generic hardware to demonstrate the conversion of sensor data to software events. The focus of this work is the generic description and generation of those events and their evaluation in several demo applications. We conclude with an outlook for several possible implementations in Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. PETRA’09, June 09-13, 2009, Corfu, Greece. Copyright 2009 ACM ISBN 978-1-60558-409-6...$5.00
ambient assisted living environments that rely on the described events.
2. RELATED WORKS Capacitive Sensing Thomas et al [12] investigated a non-contact sensor, based on electric field sensing for human computer interface. In their work they discuss two types of sensing mechanisms. The first is called Human shunt mode, where the electric field lines are intercepted and grounded by a person acting as a charge reservoir. This decreases the amount of displacement current recorded at the receiver. The other method, called Human Transmitter Mode, abstracts the person as capacitively coupled to parts of the circuitry, therefore acting as an electric field emitter. Baxter [1] uses capacitive sensing theory to provide a theoretical analysis of capacitive sensors, their properties and the design of applications using those. Wimmer et al [14] present a toolkit designed to enable rapid prototyping of capacitive sensing applications for human-computer interaction in pervasive computing systems. Rekimoto et al [10] describe the architecture to create interactive surfaces that are sensitive to human hand and finger gestures using capacitive sensing theory. The sensors track the position of the user’s hands on the surface, enabling multitouch interaction. Ambient Assisted Living Raisinghani et al [9] present an evaluation of developments to date in ambient intelligence, the driving forces behind digital information technology and the obstacles to implementing ambient intelligence on a large scale in real world scenarios. In their paper O’Flynn et al [8] present a platform designed to test a miniaturized wireless, wearable sensor for blood pressure, pulse rate and ECG monitoring. Cheng et al [5] investigate the suitability of body worn capacitive sensors for the measurement of physiological parameters. They employ a conductive textile electrode in a measurement setup to obtain signal recorded during breathing.
3. CAPACITIVE PROXIMITY SESIG Capacitive sensing is the technology of detecting the change in capacitance of either a single condenser or a system of those. It can be used for detecting grounded conductors, by using a single electrode as sensor and measuring the capacitance between sensor and target. Detecting isolators is possible as well, using two electrodes. The dielectric will cause a change of capacitance if brought between those two.
4. SETTIG UP THE PROXIMITY IPUT DEVICE 4.1 HARDWARE
Figure 1: Human body influencing capacity between sensor and electronics Human Body Field The human body produces a subtle electric fields generated by cell activity and ionic currents of the nervous system [3]. Therefore if the sensor electronics are sensitive enough we can detect the presence of a body part if its proximity is close enough to the sensor. As shown in Figure 1 this can be roughly approximated as a series circuit of capacitors where CB is dependent on body proximity.
Sensor Value in Percent
Capacity Measurement As capacity can’t be evaluated directly the sensor is constantly charged and discharged. Within each cycle the electronics calculates capacity by using the required time to discharge the sensor.
As this work is trying to implement a proof-of-concept device, focusing on creating a software application that generates events from a capacitive proximity sensor, it was decided to use generic devices and modify the setup for optimized results. It was decided to use the ProxDet CY-3235 DemoKit from Cypress Microsystems. The package contains two boards. The first is an I2C module containing the capacitive proximity sensor and connector for the antenna. The second board is the USB-I2C bridge, which connects to the sensor and is detectable by an OS using the USB-HID standard. I2C is a simple bus system created for short distance communication between devices on the same board. USB-HID is an extension of the USB bus system to simplify communication with devices used for human-computer-interaction. We are using wire antennae as sensing electrodes, as they are easy to connect and can be easily modified in shape and length. The provided wire was de-soldered and replaced with a longer one which provided better results (see Section 5).
1,2 1 0,8 0,6 0,4 0,2 0 0,00
20,00
40,00
60,00
(a) Prototype used for this paper
Distance to Sensor in cm
Figure 2: Characteristic curves of several sensor configurations Proximity Evaluation Capacitance is proportioanl to the distance between the electrodes. However gathering absolute distance values from sensor data is difficult. Difference in setup, humidity, temperature can affect the measurement. The transfer function is continuous but nonlinear. A few examples can be seen in Figure 2. This issue can’t be easily solved and consequently we are eithher working with relative proximity values that quantize the transfer functions into several threshold levels or relative distance values. Signal-to-oise Ratio Any measurement of electric values is prone to noise, the sources of which are mainly thermal noise and shot noise [6][13]. This problem is system-inherent and precautions have to be taken to allow a good ratio between signal and noise. A classical method is implementing a low-pass filter that cuts off signals with high frequency. Signals carrying low frequencies pass undisturbed.
(b) Wireless Sensor Array Prototype
Figure 3: Prototype device supporting antenna wire and demo boards. Various prototypes have been built. For this paper we settled with the design in Figure 3a with a hidden antenna of approximately 80cm length that provided good results. As an outlook to future works with wireless sensor arrays our first working prototype is displayed in Figure 3b. So far it uses two sensors with 80cm antennae but will be improved in the near future.
4.2 Software The demo kit is showing as HID, thus it can generally be controlled by any OS supporting the USB-HID standard. For Windows systems the manufacturer provides an ActiveX-control, which allows easy access to the bridge board using high level functions. It needs to be installed as a proprietary driver, which was deemed reasonable for a proof-of-concept device. It was written in C# using the .NET 3.0 framework. The components are visualized in Figure 4.
fragments. While one fragment is sufficient for event generation it is also possible to combine multiple or ignore them (Figure 5). Another important DOF that is shared by all devices is time. Timestamps are used to evaluate duration of fragments which can be subsequently used in the event generation.
Figure 5: Pipeline from state changes to event generation To apply this theory to capacitive proximity sensing we have to evaluate available inputs, their possible states and how they can be used to generate events. Focus of this paper is the generation of events from raw sensor data and providing a series of applications that allow instant testing and debugging of those.
Figure 4: Software structure visualizing data flow, modules and IO Via user interface the polling rate is set, which determines the data transfer interval from the connected HID. This interval is referred to as tick. To simplify processing of the sensor values we generate a zero level of the undisturbed sensor input. Consequent calculations with sensor measurements are based on the difference of the current input data to this value. Upon first connection to the bridge we take a variable amount of samples and determine the zero level through a simple mean calculation. In the next step that difference is forwarded to a gesture analyzer. This module generates specific events using multiple difference values and a number of ticks, which cause different behavior within the GUI. A more detailed explanation is following in the next section. The application optionally enables an average low-pass filtering of the input values to increase the signal-to-noise ratio albeit at the cost of causing a delay in the signal. The mean of the n-1 previous samples and the current raw value is stored.
Fragment Detection We have two degrees of freedom in our setup, sensor values and time. As it is our intention to outsource signal processing to the microcontroller in the future we do not take the raw data but calculate the difference to the zero level described earlier. This value is either used directly or quantized using several threshold levels as shown in Figure 6. Again quantization can be done on the microcontroller. The time-DOF is simply the number of passed ticks. Consequently the actual time passed will vary according to the polling rate.
(a)
Direct usage of difference values
(b)
Usage of quantized thresholds
5. EVET GEERATIO VIA CAPACITIVE PROXIMITY SESIG In this section we describe the generalized pipeline that processes events from generic input data. Computer science defines events as activities from either hardware or software that trigger a predefined behavior in an application. This paradigm can be extended to any input device, where state changes are detected. Methodically any device is a system that has several inputs or degrees of freedom (DOF). Each of these is uniquely identified and can forward its state to the device. These states can be described by either discrete or continuous functions. The smallest possible activity that might trigger an event is the change of the state in a single input. In our methodic we name these minimal activities
Figure 6: Usage modes of difference values in fragment detection Gesture Analysis The gesture module stores a number of detected fragments. Depending on the intended application we combine those fragments freely to create a number of gestures. The module
is taking the function of the fragment analyzer. If a detected fragment is used and the order and number of fragments that determine a gesture is decided here. In our case we use a different set of gestures for each application to evaluate the versatility of our setup.
Figure 7: Three-fragment example gesture based on quantized thresholds and wait ticks Event Generation The successful detection of a gesture can be directly used to generate an event. A simple three-fragment-gesture is shown in Figure 7. Input Device Modeling The number of events we generate for a given demo application is modeling an input device. As the fragment analysis occurs within the software we can emulate several different devices by simply modifying the gesture module.
6. DEMOS FOR EXPLICIT ITERACTIO To evaluate and debug the fragment analysis, several simple applications have been developed. The focus is on demonstrating the general feasibility of interaction using capacitive proximity sensing devices. Specific applications in assisted living environments that either have been demonstrated already or are currently in development will be described in Section 7.
6.1 Proximity Image Viewer The proximity image viewer is used to manipulate a photo slideshow. The viewer is realized using only one proximity sensor with and antenna shown in Figure 3a. Again we are using thresholds but this time combined with a tick counter to create more gestures. Figure 8 shows the combinations of fragments and the events they will generate.
Figure 8: Detected Fragments and their combinations that will cause events in the proximity image viewer By using two proximity sensors, the interaction is augmented to feel even more natural with a simple wave of the hand. This could be very useful for users with disabilities that limit them from using a mouse or a remote control device. In this setup both sensors easily realize the effect of a normal slider as shown in Figure 9.
Figure 9: Sensor Array demonstration using two sensors detecting gestures
6.2 Interaction with Music Instruments The Theremin is an electrical instrument using proximity of the hands relative to two antennae to determine pitch and volume of a sine wave [4]. This application tries to evaluate usage of a continuous input. Having only one antenna available we modulate pitch only. Each tick we take the current difference measurement and adjust the frequency of the sine wave according to it, shown in Figure 10. Interesting sound effects can be generated if the tick interval is changed, which causes a quantization of the sound output.
Figure 10: Fragments and events used in the Theremin mode The Theremin mode is sensible to sensor noise as it is directly audible. To allow stable sounds the implemented low-pass filter should be activated.
6.3 Drums This demonstration defines two one-fragment threshold gestures. Exceeding the lower threshold will trigger an event playing a tomtom drum. Exceeding the higher threshold causes playback of a hihat. Thresholds are set in a way that the device has to be touched with either one or several fingers (Figure 11).
Figure 11: Used fragments and events in drum mode
7. SETUP FOR APPLICATIOS I AAL EVIROMETS
7.1 Fall Detection An important aspect of monitoring elderly people is detecting breakdowns and eventually calling for emergency. Current systems either rely on accelerometer based systems that have to be worn by the victim [2], camera-based solutions that face acceptance
problems or infrared detectors that have limitations of line-of-sight requirement [10]. We can use a single sensor box containing several capacitive proximity sensors based on wire antennae. This setup is illustrated in Figure 12. Each sensor then serves as an input channel for measuring capacitance changes caused by the presence of a human body on the floor of a room.
Figure 12: Sensor array setup reading data from six endpoints in a room and transmitting to control PC Those wires are spread below carpet or floor boards and can be several meters of length each. While the sensitivity is reduced when using these long wires it remains decent enough for detecting a lying person. It is difficult to determine a formula calculating the resulting detected capacitance for a person due to morphological differences between people and the sensor array only sparsely being spread through the floor. However in early experiments we experienced that increasing the contact area between body and wire causes an according growth in sensor amplitude, touch of one or two hands can be easily distinguished. Capacitive sensor arrays using antennae wire consume little power. Therefore, while it would be easiest to integrate the system into new furniture, we propose a battery powered sensor box, which connects to multiple endpoints that are basically coiled, insulated wire. It connects to the controlling computer via common wireless technology. This system (Figure 12) can be fixed to existing furniture, e.g. a living room table.
sensors. Power consumption is reduced by using conservative polling and transmission intervals of several seconds.
7.2 Preventive Feedback Scenarios In our initial experiments we realized that the simple configuration depicted in Figure 9, could be hidden in wooden shelves or incorporated in furniture and electronic appliances (since the sensor does not require a direct line of sight). In a simple scenario for implementing a medication reminder, this sensor could be hidden inside the mediation box to register information such as: when and if the user had accessed the medication box at a given time. An ICT application processing such events could either alert the user or generate a feedback loop to a caregiver if the user for example, forgot to take his medication at a given time. However, accessing the box does not necessarily mean the user had taken his mediation. In order to obtain this addiction information, this sensor input can be combined with information from an RFID reader or a weight differentiator. In another scenario, this sensor could be embedded in regions around hot plates to register and generate warning alerts and feedback information to users who suffer from varying levels of dementia and easily forget that they did not turn off the hot plate. A similar situation is applicable for users with declining perceptual abilities as their sensory motors no longer react accordingly. In such cases, this sensor can greatly help to prevent body contact with hot plates.
7.3 Indoor Localization Based on the preliminary setup (in Figure 12), we quickly noticed that the setup could be extrapolated to cover a wider surface area using wireless, battery powered sensor placed in the same configuration at the edges of a room in a home to realize an indoor localization system. By using a weighted average algorithm: (x = Σi=1m xi*Ci / Σi=1mCi), where xi is the distance of each sensor to the current location of the user and Ci is the value registered at each wireless node in the proximity sensor array, the exact position of the user can be averagely weighted. Our goal is to use the acquired location and distance data to generate schemes for simple implicit interactions and feedback loops with senior citizens.
7.4 Respiration Monitoring
Figure 13: Sensor wires spread on the floor. Left side shows the silhouette a human body creates, right side the silhouette of a dog Based on this observation we summarize sensor values of all wires. This sum increases accordingly to the area of a body currently in close proximity to the wires. We intend to detect a human body currently lying on the floor and differentiate it from other objects (e.g. pets – Figure 13) and standing/walking persons by performing several calibration tests and determining suitable threshold levels. This setup can also be implemented with wireless, battery powered
High quality capacitive proximity sensors can be sensitive enough to register torso movement associated with respiration [7] without physical connection. A sensor unit placed below a patients bed should be sufficient to detect disturbances in lung function and call for emergency. However the sensors we are using thus far still have to be fine-tuned to register enough samples upon which evaluations of the technology can be performed to proof its validity in such scenarios.
8. DISCUSSIO Our initial focus of using capacitive proximity sensors as simple input devices was expanded quickly, as we explored the versatility of this sensor type. Experimenting with different electrodes we settled on a relatively long insulated wire that could be easily coiled and geometrically adapted. Our prototype is able to detect a human hand in a distance of approximately 40cm under good
conditions. Having been interested in public reception of touch-less input devices, we have been using our initial prototype on several public demonstrations showcasing the Proximity Image Viewer and Theremin demo applications. A short introduction to the gestures and a few minutes of training proved sufficient for most uninvolved people to successfully control the applications.
[3]
[4] After shifting our focus to using capacitive proximity sensor devices in AAL environments we have successfully established sensor arrays that allow for more advanced applications as described in the last section. Our initial experiments using hidden sensors and long wires have been reassuring. We are confident that our research in this area will show valuable results in the near future and that we can successfully implement the proposed systems.
9. LIMITATIOS Technical limitations Our focus was on software implementation. The proof-of-concept prototype is based on generic hardware and simple components. Custom designs provide much better detection distance and signal-to-noise ratio. Furthermore the microchips on the sensor boards are merely used to acquire raw data. Signal processing is happening in the application but could be offloaded. Disturbances in measurement Capacity metering can be unreliable depending on the environmental situation and stability of the setup. Other factors disturbing the measurements are grounded conductors within detection distance and to a lesser extent temperature and humidity. Consequently threshold levels need to be clearly separated as the actual distance they apply might vary on each use. Zero Level Calculation The method determining the zero level is a very basic approach. It is a static one-time calculation that will not detect a changed zero level while running. Dynamic algorithms exist and will be implemented in the future either at microcontroller or software level.
[5]
[6]
[7] [8]
[9]
[10]
[11]
10. COCLUSIO AD FUTURE WORKS We presented a methodical approach to generate input devices based on capacitive proximity sensors and evaluated the usability in software applications. Furthermore we proposed several possible viable applications in assisted living environments, displaying the versatility of our concept. As future work, we intend to further explore the proposed applications. This will include the usage of wireless proximity arrays, signal processing applied on microcontrollers, softwaredevelopment and real-world evaluation.
[12]
[13]
11. REFERECES [1] Baxter, L. 1997. Capacitive Sensors IEEE Press Series on Electronics Technology, Piscataway N.J., 1997. ISBN 078035351X, 9780780353510 [2] Chen, J., Kwong, K., Chang, D., Luk, J., Bajcsy, R. 2005. Wearable Sensors for Reliable Fall Detection. Engineering in Medicine and Biology Society, 2005. IEEE-EMBS 2005. 27th
[14]
Annual International Conference of the. (Shanghai). pp. 35513554. DOI=http://doi.acm.org/10.1109/IEMBS.2005.1617246 Cohen, D. 1967. Magnetic Fields around the Torso: Production by Electrical Activity of the Human Heart. Science 5 May 1967: Vol. 156. no. 3775, pp. 652 - 654 DOI=http://doi.acm.org/10.1126/science.156.3775.652 Garner, L. 1967 For That Different Sound, Music a la Theremin. Popular Electronics (November 1967). http://www.uneeda-audio.com . pp. 29-33,102-103 Jingyuan, C., Lukowicz, P., 2008. Towards wearable capacitive sensing of physiological parameters. Second International Conference on Pervasive Computing Technologies for Healthcare, PervasiveHealth 2008 (Tampere, Finland, Jan. 30 2008-Feb. 1 2008). pp. 272-273. DOI=http://10.1109/PCTHEALTH.2008.4571087 Johnson, J. B. 1928. Thermal Agitation of Electricity in Conductors. Physical Review 32, pp. 97-109. DOI= http://doi.acm.org/10.1103/PhysRev.32.97 MacLachlan, R. 2004. Spread Spectrum Capacitive Sensor http://www.cs.cmu.edu/~ram/capsense/ O'Flynn, B., Angove, P., Barton, J., Gonzalez, A., O'Donoghue J., Herbert, J. 2006. Wireless Biomonitor for Ambient Assisted Living. Oral presentation at ICSES 2006 (International Conference on Signals and Electronic Systems). (Lodz, Poland, 17-20 September 2006) Raisinghani, M. S., Benoit, A., Ding, J., Gomez, M., Gupta, K., Gusila, V., Power, D., Schmedding, O. 2004. Ambient Intelligence: Changing Forms of Human-Computer Interaction and their Social Implications. Journal of Digital Information. Vol. 5. no. 4. Article No. 271 Rekimoto, 2002 SmartSkin: An infrastructure for freehand manipulation on interactive surfaces. In Proceedings of the SIGCHI conference on Human factors in computing systems (Minneapolis, Minnesota, USA) CHI’02 ACM Press/AddisonWesley Publishing Co. New York, NY, 280 – 287. DOI= http://doi.acm.org/10.1145/503376.503397 Sixsmith, A., Johnson, N. 2004. A smart sensor to detect the falls of the elderly. Pervasive Computing, IEEE. Vol. 3. no. 2. pp 42- 47. DOI=http://doi.acm.org/10.1109/MPRV.2004.1316817 Thomas Z., Joshua S, Joseph P., David A and Neil G. 1995. Applying Electric Field Sensing to Human-Computer Interfaces. In Proceedings of SIGCHI Conference on Human Factors in Computing Systems (Denver, Colorado, United States, 1995). ACM Press/Addison-Wesley Publishing Co. New York, NY, pp. 280 – 287. DOI= http://doi.acm.org/10.1145/223904.223940 Van der Ziel, A. 1955. Theory of Shot Noise in Junction Diodes and Junction Transistors. Proceedings of the IRE. Vol. 43, no. 11. pp. 1639-1646. DOI=http://doi.acm.org/0.1109/JRPROC.1955.277990 Wimmer R., Schmidt A., Holleis P. and Kranz M. 2007 A Capacitive Sensing Toolkit for Pervasive Activity Detection and Recognition. In Proceedings of the 5th IEEE International Conference on Pervasive Computing and Communications (19-23 March 2007) PerCom ’07, 171-180. DOI=http://doi.acm.org/10.1109/PERCOM.2007.1
