Personal Mobile Hub - CiteSeerX

Personal Mobile Hub Dirk Husemann, Chandra Narayanaswami, Michael Nidd IBM Research Division [email protected], [email protected], [email protected]

Abstract As advances are made in wearable computing, there is a need for a personal mobile hub that can manage interactions between the wearable devices and act as a proxy for these devices. In this paper we describe why this is a good model, why the interfaces need to be open, and why different parties in the value chain, such as wireless service providers and device manufacturers, may prefer such an architecture. Our personal mobile hub supports multiple wireless protocols, some short range and some wide area so that the power of the internet is available even to body worn sensors. We describe custom hardware that we built for this purpose and also the software necessary to make this concept work. We have tested out this architecture with an end to end application. The working system was demonstrated at the annual IBM Stockholders meeting in 2004 and is also available for customers to see at the IBM Industry Solutions Lab in Zurich.

1. Introduction Steady advances of all aspects of wearable computing have been made over the past several years. It is now possible for us to wear multiple devices and benefit from them without being considered “strange” by people around us. Examples of such wearable devices include biomedical sensors in vests[7] or arm bands [7], wristwatches, eyeglass mounted displays, belt computers, sensor equipped shoes, etc. Recently, video capsule endoscopy [1] based on an ingestible capsule measuring 11x23mm is making it possible for gastroenterologists to view the entire small intestine without pain or sedation. The capsule contains a color camera, six light emitting diodes, a radio transmitter, an antenna and two batteries that last for eight hours. Images from the capsule are transmitted to a sensor array and stored in a data recorder worn on the patient’s waist. Clearly as the number of devices worn increases more issues with inter-device communications, security, costs, and device maintenance arise. There are more batteries to charge, more chargers to carry, more batteries to replace, and more manuals to read. On the positive side, the multiple devices can interact and provide more value to the users. The above possibilities led us to develop the concept of the Personal Mobile Hub (PMH). This hub serves as the focal point for all the devices worn by the user. It supports multiple wireless protocols and helps convert from one protocol to another and thus allows two devices that support two different protocols to communicate. This hub also has

wide area network connectivity and so it can connect to the internet and get and send information on behalf of the other worn devices. In effect the hub is a router and a gateway for the multiple devices that are worn. The above decoupling leads to several advantages. The individual devices now no longer need to communicate directly with the internet and therefore can implement wireless technologies with shorter ranges and thus save power and costs. In addition, since the range is reduced, for some applications the encryption requirements can possibly be less stringent than the case where ranges are larger. The PMH can also serve as the display device and interaction center for other worn devices that do not support a display. Such a split allows manufacturers to reduce the costs or to shift costs to other features such as ergonomics, style, etc. In addition, since the various devices are likely to be manufactured by different vendors, the PMH can transcode between published protocols, and allow interaction among devices that would not have been able to communicate natively. We have built a customized PMH, built the necessary middleware and have tested it in a health monitoring application. We have shown our prototype at leading trade shows including CeBIT.

2. PMH Concept

GSM/GPRS

Mobile

Server

Bluetooth ZigBee 802.11 UWB

Old: 2-Tier

accessory

GSM/GPRS

Server

Hub

appliance

sensor

New: 3-Tier

Figure 1: Two tier vs three tier architectures A significant change induced by the personal mobile hub concept is that we move from a conventional two tier architecture to a more flexible three tier architecture shown in Figure 1.

Proceedings of the Eighth International Symposium on Wearable Computers (ISWC’04) 1530-0811/04 $ 20.00 IEEE

With the conventional two tier wireless architecture the mobile phone was the endpoint of all data traffic - and due to its limited capabilities very little data traffic was (and still is) generated. With the three tier PMH architecture the mobile device becomes a data hub: by using inexpensive short-range wireless technologies such as Bluetooth® (currently) and ZigBee we can enable applications and services that so far have not been possible. Flexible textiles[12] could be used to connect the sensors to the PMH as well, but our general experience has been that people prefer systems without wires. Thus, as shown in Figure 2, the concept of personal mobile hub is a rather powerful one. In its simplest form a PMH assumes the role of a mobile internet router and serves as an access point for the user’s devices, regardless of whether they are worn or not, such as laptops, PDAs, game consoles, jukeboxes, medical sensors, and so forth. The PMH in this case works a normal internet router would work: the only difference being that it is a mobile device and would provide firewall services, sensor data encryption, Virtual Private Network (VPN) functionality, accounting support, and traffic shaping for devices that may not be capable of including this function due to constraints on size and power. A more advanced version of the PMH as a mobile internet router might include additional functionality such as instant messaging support, mail filtering and IMAP (Internet Message Access Protocol) support.

app 1 app 2 appliance loyalty loyalty beacon

3rd parties

JVM

EvtSpace

Wev Svc

Dev Mgmt

loyalty

WebSphere

heartrate

heartrate watchdog

heartrate sensor

Tivoli

OS

game device

PMH HW

Eclipse etc

PPC CSR MC45 SIM

Tier-1: Backend

Tier-2: Hub

preferences, and so forth. The personal repository can be replicated in secure storage in the back end as well so that the data is safe even if the PMH or one of the sensors were to get corrupted and lose data. The PMH can also provide the user interface necessary to interact with personal sensors. For example, voice recognition software could be used to request a sensor to send data. The PMH may also include traditional GUIs to enter commands that can then be conveyed to the sensors. It can also accept input from other wearable input devices such as a should pad insert [19]. In addition, the display on the PMH can be used to receive alerts from the sensors. The sensors do not need to include a display. Perhaps the most interesting role, from a business point of view, is that the PMH can serve as an aggregator and service extender. Here the PMH serves as an open service platform interacting not only with the user but also with sensors and actuators in the personal sphere of the user. Application areas range from maintenance to augmented reality to medical and healthcare applications and more. To make a concept such as the PMH an attractive proposition and get it off the ground, it is necessary to design it from the start as an open and extensible platform: PMH is by its very nature a platform that depends on third party support; a proprietary solution would make it very difficult to “sell” the idea of a PMH. An open platform that relies on open standards as far as possible makes it easy to implement applications and create an attractive eco-system around it. Several providers in the value chain like this model: cellular phone companies and WiFi service providers can get more traffic in their networks, small device manufacturers can concentrate on functionality and can offer back end services once monitoring data gets back to them. Service providers can study logs in the backend and proactively look for patterns in the data and take necessary action.

3. Related work OS

Tier-3: Accessories

Figure 2: Three tier architecture Another role that the PMH could take is that of a personal agent. Here the PMH serves as a control point for tickets, vouchers, loyalty applications, and other ecommerce applications. It might even serve as a consulting agent through the use of pervasive collaborative filtering techniques. We have investigated the use of personal web services[3] for this task. Tied to the role of personal agent is the role of a personal repository where we not only store the usual address book and calendar on the PMH but also other information: lists of books we have bought, recommendations we received from friends and colleagues, places we have been to, personal vital signs, electronic health records, voice memos, personal

The concept of personal hubs and servers has been explored earlier by several efforts at some level. However, much of the work has been focused on hardware or on point applications. IXI has produced a hub telephone device, called the “Personal Mobile Gateway,” that can connect several peripherals via Bluetooth. The peripherals are either manufactured by IXI or include a licensed software stack to connect with this hub. The physical connection is via Bluetooth, and can link both audio and data devices to the hub. The link from the hub to the outside is via GSM/GPRS (the cellular network). RTX has produced base stations for the home that use Bluetooth to connect with various peripheral devices. Peripherals incorporate a stack licensed from RTX, and the hub links to the internet either via GPRS/GSM or PSTN (the fixed line network).


Both of the above could be considered similar in concept, but their proprietary nature means they appeal to a different set of customers. Our open architecture is intended to encourage a broad range of device manufacturers to build into the local wireless health device space, providing a flexible, general application platform. The MIThril hardware platform [6] combines body-worn computation, sensing, and networking that can integrate into clothes. The MIThril body bus connects several peripherals and sensors (devices which do not require Ethernet networking) to the BSEV core. The MIThril Body Network and MIThril Body Bus simplify the physical on-body networking problem by providing a reliable single-cable power/data connection among all devices on the body. Some sensors that attach to the MIThril body bus include a threeaxis accelerometer, an IR Active tag reader, a USB microphone and headphone, and a USB CCD camera. More recently the MIThril researchers have started using the system for proactive health care monitoring - LiveNet [18]. The WearARM [10,11] team is using their low power modular core for context aware computing. Several health monitoring devices, including watches, are available today. Many of them come with their own user interfaces and come as sealed units, i.e., they do not allow the data collected to leave the watch. Some watches do support interfaces that allow them to connect them to exercise equipment and to other computing devices. BodyMedia has health monitoring devices that store lifestyle data during the day, and allow it to be downloaded (by cable or RF) at home for later analysis. Other recent wearable health care devices include LifeGuard, MedicTouch, etc. The idea of a wearable server has been described before. For example, at the Bluetooth Developers Conference in Dec 2000, IBM showed a Bluetooth enabled MicroDrive that served data to a Bluetooth equipped player (see Figure 3).

PMH in contrast works with a personal set of devices, and so important problems of device disambiguation, authentication, etc., are simplified. Also the Intel Personal Server does not support multiple wireless protocols. The IBM® Linux Watch [13] prototypes (Figure 4) pack a fair amount of hardware and software into a watch form factor. The watches run a standard operating system and support short range wireless communication via Bluetooth and Infrared technologies. They also include a jog dial and a touch screen for user interaction. These watches could clearly serve as displays for other worn devices that do not have a display. However, the watch has no internet connectivity. A concept related to the Personal Mobile Hub was introduced in [14] and was called the Parasitic Computing Enabler.

Figure 4: IBM Linux Watch with OLED Display The IBM® MetaPad packages a fully functional computer into a device that fits in a large shirt pocket. It can connect to a small mobile display with a touch screen or to a docking station connected to a large display and a keyboard. The most recent version also includes 802.11 connectivity. The MetaPad allows users to suspend their computation state in one place and resume it at another. In some ways the MetaPad can be considered to be a portable personal server since it has several GB of storage and offers wireless connectivity. OQO and Antelope Technologies have shown similar concepts. In contrast to some of the earlier work, we have put more emphasis on building an open software architecture and testing it with an end to end application.

4. PMH Hardware and Software Architecture

Figure 3: Wearable Data - Served from a MicroDrive The Intel® Personal Server™ [20] includes a processor and storage and aims to communicate with other devices in the environment using wireless means. One significant feature in addition to serving data to other devices is the elimination of the display, thereby saving power consumed by displays, and using displays in the environment. The

Our first implementation of the PMH, shown in Figure 5, consisted of three main components: a MC45 cellular module, a MPP310 processor module and an interposer module code named Opal. The basic function of the PMH was to allow cellular phone connectivity to a personal area network (PAN) over Bluetooth. The following lists the main components of the PMH:


Message Terminal

Computer

Handset

GPRS/ GSM

Bio-Monitor PDA/ MP3

PSTN/ Internet

PWG

Headset

Bluetooth PAN

Entertainment/ Telematics

Figure 5: The IBM Personal Mobile Hub Figure 6: Roles for the IBM Personal Mobile Hub x x

x

x x x x

MC45 Cellular Engine x Complete standalone tri-band GSM/GPRS module MPP-310 Module x PPC405LP Processor running up to 233Mhz. x 64MB SDRAM, 16MB FLASH x 10/100 Ether net (Not used) x RTC x Voltage and Thermal monitor Interposer card (Opal) x Mitsumi WML-C10 BlueTooth module x CPLD x 1.0v/1.8v DC/DC Regulator x Battery Charger x USB 1.1 Slave interface x SIM, Power, Vibrator, Headphone/Mic, Micro USB Slave, Display Connectors x Power Button x Panic Button x Speaker x CODEC 1050mAhr Lithium Ion Battery Pack Vibrator Tri-band Cellular Antenna 64x128 LCD Display

This design worked and addresses several of the traditional challenges in wearable computing [16,17], but had two non-technical drawbacks. First, the business people interested in new hardware are not the same people who are interested in new software, and vice versa, so doing both means everyone you deal with has no sympathy for at least half of your project. Second, customers are more comfortable trying new software on a platform that has already been proven and available in large quantities. With these arguments in mind, we moved to a platform that delivered most of the capabilities we had on the original PMH: the Sony Ericsson P800 (later, the P900). The software architecture of the hub device is centered around an event engine. The event engine provides a programming model similar to the “Blackboard” communication model [15], in that any object may add an event to the pool, and any number (zero, one, or more) can receive notification of the event. Events are not, however, stored for any significant amount of time. Objects connect to the event engine, then interact with it in either or both of two general behaviors: registering interest in events, or inserting events. Inserting events is just a matter of formatting an event object, and passing it to the engine. Registering an interest passes a callback reference to the engine, along with an optional mask that describes the classes of events expected, so the engine can notify the object of any events that are received. If multiple objects are interested in the same event, then they will all receive a copy of the event. Once all interested objects have been notified, the event is dropped from the engine. If history is required, then a logging object can be connected, but the engine itself deals with events as they arrive. One of our demonstrations simply forwards all medical events to a server that then acts as a web portal to allow remote viewing, as shown in Figure 7.


successor to pJava) will not offer JNI, so if we want to continue allowing applications in Java, the engine will have to be in Java. Fortunately, the other significant development is the inclusion of JSR-82 (the Bluetooth API) in the P900 MIDP 2.0 support. The net result is that, while we must replace the technical architecture of using an Application Engine, the communications ability is in place to make this possible.

5. PMH Use : IBM mobile health toolkit

Figure 7: Sample Data Log To avoid allowing slow applications to block the engine, notifications are sent as simple alerts, to which the application responds by issuing a read request to the engine. Events are stored in a circular buffer, and replaced as new events arrive. If a read request is sent after the event has been overwritten, a timeout error is returned. The current event engine is implemented as a Symbian Application Engine. This is similar to the address book in that the object is loaded from a DLL and instantiated the first time a client application tries to connect to it. Future connection attempts will connect to the already-instantiated object, creating connection contexts for each client. In this model, the peripheral-specific Bluetooth connections can be established with very small (~11 KB) applications, loaded separately to the phone. These applications are aware of the device communication protocols (command formats, etc.) and also the format of the various event types that can be produced by that device. Events are usually grouped into an envelope so, for example, a Blood Pressure device may produce envelopes containing three events (blood pressure, original time of measurement, and pulse). The envelope itself is timestamped with the time of day and an event serial number when it is inserted into the event engine. Our event taxonomy consists of application domains (e.g., a medical domain, a system domain, and so forth). An individual event type is addressable through a tuple notation, providing an extensible and future-proof taxonomy, and also introducing sensor device independence. The event engine, and current Bluetooth device connectors are written in native Symbian C++. A JNI (Java Native Interface) bridge allows Personal Java applications to use native Symbian code to connect to the event engine and send or receive events. This design was initially chosen for various reasons, including the unavailability of Bluetooth to Java applications and the difficulty of communicating between separately installed Java applications (i.e., Application Engines must be in native Symbian). In response to recent developments, this design is being updated. One of the most significant reasons is that Personal Java has entered its end of life process, and MIDP (Mobile Information Device Profile – the preferred

One of the very early demonstrators that we built based on the PMH concept was a heartrate monitor using a normal off-the-shelf chest-strap monitor for tracking your heartrate during exercises. We built an RF analog-to-Bluetooth relay device about the size of a large pen that would convert the analog RF burst emitted by the chest-strap monitor for each detected heart beat into an “X” character that was then transmitted to the PMH device via the Bluetooth RFCOMM protocol. On the PMH device we ran a simple tracking application that would measure the time between received heart beat indications and calculate the current pulse. Whenever the calculated value exceeded a predetermined threshold, the application would trigger an alarm SMS text message to be sent to a mobile phone. While we did build other demonstrators as well, this rather simple application was the one application that everybody understood right away. Following an article in the New York Times[7] about this demonstrator we were contacted not only by companies operating in the pharmaceutical and healthcare industries, but also by mobile phone operators who were interested in having lifestyle-oriented and well-being offerings for their customers. Triggered by the overwhelming response, we decided to concentrate on the healthcare and pharmaceutical industries first, and created the IBM mobile health toolkit to address those spaces. The toolkit is not a ready-to-deploy solution, but rather a set of building blocks that can be used to build custom tailored applications and solutions.


Figure 8: Blood pressure cuff and PMH

As shown in Figure 8 first prototype of the IBM mobile health toolkit includes a Bluetooth enabled blood pressure cuff, a medication compliance monitoring device from Bang & Olufsen Medicom, and the Sony Ericsson P900 smart phone as the mobile hub device. Already this combination of devices allows us to address a wide range of healthcare applications. One of the most interesting ones being those that address patient non-compliance: 55% of all long term patients do not take their medication at all or at least 14 hours too late, resulting in a cost of at least USD 100 billion a year to the US tax payer --- and accounting for about 12% of the UK hospital admissions. With the B&O Medicom medication device we can now track patient compliance in real-time. The medication schedule is sent to the mobile hub device (see Figure 9) and from there to the B&O IDAS II medication device. At the same time we can provide alerting rules to a correlation engine running on the mobile: when the patient forgets to take his medication at the appointed time, the correlation engine will notice the absence of the medication event and will generate a local alert event that will be displayed to the patient. If the patient still fails to comply with the medication event, a network alert event is generated and transmitted via the mobile phone network to a medical help desk. Likewise, every medication event that did occur is relayed into a database at the healthcare service provider.

Figure 9: Medication Compliance application Pharmaceutical applications of the IBM mobile health toolkit address clinical trials: currently most of the data from a clinical trial is captured on paper, generating a rather impressive paper trial and a not so impressive delay in getting at the data. Also, due to the repeated transcription of the data, the data quality tends to be less than what might be desirable. Clearly, with the IBM mobile health toolkit it becomes possible to increase the data quality dramatically, lower the delay drastically, and react in a timely fashion to any adverse reactions that might occur. Furthermore, it allows for the future possibility of modifying the trial protocol in case the drug under evaluation proves to be of such a good quality that it becomes unethical to withhold it from the placebo group.

6. Other applications We are investigating other applications that could relay context information about the user to the enterprise. One example is determining the current activity of the user and then propagating that information to the back end infrastructure. For example additional roles can be added to an instant messaging client, such as the person is in the car, or that the person is walking, etc. The instant messaging partner may then use this information to tailor the message so that it does not require the level of user attention commensurate with a user in his office. This concept extends to other applications as well. We are also working on analyzing sensor data that can be collected in the backend to look for patterns so that proactive actions can be taken.

7. Conclusion In this paper we presented the concept of a Personal Mobile Hub and illustrated the value that a three tier architecture could provide. We built a customized hardware version of the PMH to understand some of the hardware challenges in realizing such a concept. We also developed an architecture for logging and subscribing to events occurring in the system and implemented it on a Sony Ericsson P900 smart phone. A correlation engine on the PMH is able to look for events and generate alerts. This architecture has been tested on a health care application. Our work thus far has attracted the attention of pharmaceutical, health care and mobile device manufacturing companies. We believe that other possible applications include supporting remote workers (capturing measurement information, providing context dependent feedback, monitoring their well-being in case of dangerous work environments) as well as applications in the container shipping industry (secure trade lanes, container checking, etc.).

8. Acknowledgments IBM, IBM(logo) are trademarks of International Business Machines Corporation in the United States, other countries, or both. Intel, Intel Personal Server are trademarks of Intel Corporation in the United States, other countries, or both. Java and all Java-based trademarks are trademarks of Sun Microsystems Inc. in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others.

9. References 1.


A. Ali, J. M. Santisi, J. Vargo, Video Capsule endoscopy: A voyage beyond the end of the scope, Cleveland Clinic Journal of Medicine, Vol 71., No. 5., pp. 415-426.

2. 3.

4. 5. 6.

7. 8.

9.

10. 11. 12.

13. 14.

15.

L. J. Bass, C. Kasabach, R. Martin, D. P. Siewiorek, A. Smailagic, J. Stivoric: The Design of a Wearable Computer. Proceedings of CHI 1997: 139-146. S. Berger, S. McFaddin, C. Narayanaswami, M. Raghunath, Web Services on Mobile Devices Implementation and Experience, Proceedings of the Fifth IEEE Workshop on Mobile Computing Systems & Applications 2003, pp 100-109. M. Billinghurst, T. Starner: Wearable Devices: New Ways to Manage Information. IEEE Computer 32(1): 57-64 (1999). BodyMedia SenseWear 2, http://www.bodymedia.com/pdf/sm_pro_spec_10_11.p df R. DeVaul, M. Sung, J. Gips, A. Pentland MIThril 2003: Applications and Architecture, Proceedings of the Seventh IEEE International Symposium on Wearable Computers, 2003 pp 4-11. A. Eisenberg, When the Athlete's Heart Falters, a Monitor Dials for Help, Circuits Section, New York Times, January 9, 2003. P. Grossman, The Lifeshirt: A Multi-Function Ambulatory System That Monitors Health, Disease, and Medical Intervention in the Real World, New Generation of Wearable Systems for Health: Towards a Revolution of Citizens' Health and Life Style Management? Pisa, Italy, Dec 12 - 14, 2003. N. Kamijoh, T. Inoue, C. M. Olsen, M. Raghunath, C. Narayanaswami, Energy Trade-offs in the IBM Wristwatch Computer, Proceedings of the Fifth IEEE International Symposium on Wearable Computers: 133140 (2001). P. Lukowicz, U. Anliker, G. Tröster, S. J. Schwartz, R. W. DeVaul: The WearARM Modular, Low-Power Computing Core. IEEE Micro 21(3): 16-28 (2001). P. Lukowicz, H. Junker, M. Stäger, T. von Büren, G. Tröster: WearNET: A Distributed Multi-sensor System for Context Aware Wearables. Ubicomp 2002: 361-370 T. Martin, M. Jones, J. Edmison, R. Shenoy, Towards a design framework for wearable electronic textiles, Proceedings of the Seventh IEEE International Symposium on Wearable Computers, 2003 pp 190-199. C. Narayanaswami et al., IBM's Linux Watch: The Challenge of Miniaturization, IEEE Computer, 35(1): 33-41, Jan 2002. C. Narayanaswami, M. Raghunath, N. Kamijoh, T. Inoue, What Would You Do with a Hundred MIPS on Your Wrist? IBM Research Report RC22057, January 2001. A. Newell, Some problems of the basic organization in problem-solving programs. In Proceedings of the Second Conference on Self-Organizing Systems. Yovits, M.C., Jacobi, G.T., and Goldstein, G.D. (eds), pp. 393-423 (referenced in J.A. Kaplan, J.W. McManus, and W.L. Bynum, Automated Concurrent Blackboard System Generation in C++, National Aeronautics and Space Administration report NASA/TM-1999-209128)

16. T. Starner: The Challenges of Wearable Computing: Part 1. IEEE Micro 21(4): 44-52 (2001). 17. T. Starner: The Challenges of Wearable Computing: Part 2. IEEE Micro 21(4): 54-67 (2001). 18. M. Sung and A. S. Pentland, MIThril LiveNet: Health and Lifestyle Networking", Submitted to the Workshop on Applications of Mobile Embedded Systems (WAMES'04), Boston, MA, June, 2004 19. A. Toney, L. Dunne, B. H. Thomas, S. P. Ashdown, A Shoulder Pad Insert Vibrotactile Display, Proceedings of the Seventh IEEE International Symposium on Wearable Computers, 2003 pp 35-44. 20. R. Want, T. Pering, G. Danneels, M. Kumar, M. Sundar, J. Light, The Personal Server: Changing the Way We Think about Ubiquitous Computing. In Proceedings of Ubicomp 2002: 194-209.
