RemoteUI: A High-Performance Remote User Interface System for ...

RemoteUI: A High-Performance Remote User Interface System for Mobile Consumer Electronic Devices Daniel Thommes, Student Member, IEEE, Ansgar Gerlicher, Member, IEEE Qi Wang, Member, IEEE, Christos Grecos, Senior Member, IEEE

Abstract — Modern consumer electronic devices can easily integrate into distributed computing environments. By applying network-based user interface systems to control these devices, innovative applications become feasible. However, several requirements like high responsiveness and low bandwidth consumption need be satisfied. Particularly, to be applicable in wireless and mobile local and wide area networks, they must circumvent low available data rates and high network latency. We propose a new solution called RemoteUI, which works with abstract user interface (UI) descriptions and their prioritized, incremental remote replication. The system combines different approaches from thin client computing and widget based UI systems and is further optimized in the context of wireless networks and mobile consumer devices. To demonstrate the advantages of our solution, we experimentally compare the proposed RemoteUI System with Virtual Network Computing (VNC) implementations and an HTML-based solution1. Index Terms — Remote User Interfaces, Mobile Consumer Electronic Devices, VNC, Web4CE, HTML

I. INTRODUCTION A growing number of consumer electronic devices can be integrated into IP networks allowing them to use and offer pervasive services. Particularly, the transfer of user interfaces (UIs) to remote control devices is an essential technique to facilitate such services in a user-friendly and locationindependent manner. Scheduling the next TV-recording on a TV set or setting the temperature at home with the help of a smart climate control device can be carried out from remote locations. Versatile mobile consumer electronic devices like smartphones or tablet computers are today's remote controls "to go". They are connected to the Internet and capable of presenting rich user interfaces for a comprehensive number of applications. In this way they become a desirable amendment 1 Daniel Thommes is with the Institute of Applied Science, Stuttgart Media University, 70569 Stuttgart, Germany (e-mail: [email protected]). Ansgar Gerlicher is with the Institute of Applied Science, Stuttgart Media University, 70569 Stuttgart, Germany (e-mail: [email protected]). Qi Wang is with the Audio Visual Communications and Networks Group, School of Computing, University of the West of Scotland, Paisley, PA1 2BE, United Kingdom (e-mail: [email protected]). Christos Grecos is with the Audio Visual Communications and Networks Group, School of Computing, University of the West of Scotland, Paisley, PA1 2BE, United Kingdom (e-mail: [email protected]).

Fig. 1. Using mobile devices to remotely control other consumer electronic devices within LAN or WAN environments.

or even replacement for traditional remote controls. They even allow remote control use cases that have not been possible before. Fig. 1 illustrates two of these use cases where a smartphone is used to control consumer electronic hardware from a local and a wide area network, respectively. Furthermore, the mentioned mobile devices are able to offer innovative network services themselves. For example, they can make personal information management functionality or rich media applications available to other network devices. There are situations where more convenient or better-suited input equipment for the control of these services are either preferable or even mandatory. Fig. 2 shows such a scenario in which the driver of a car controls the smartphone through the car’s head unit, typically an onboard “thin client” computer with a larger screen. The latter mirrors the mobile phone’s user interface, transmitted via a wireless personal or local area network (WPAN or WLAN), and allows controlling it with touch gestures. This solution provides a safer and more

Fig. 2. Using specialized input devices to control the mobile device in a WLAN or WPAN environment.

convenient way for the driver to enjoy the applications running on the mobile phone. To realize the described use cases, a remote user interface system is needed that fulfills the following requirements: • The complexity of both client and server software should be low, because in a consumer electronic environment the majority of devices have reduced processing power. • The responsiveness of the user interface, although being transferred over a possibly slow network connection, should be high for good user experience. • The remote control protocol’s required bandwidth should be reduced to accomplish the desired fast reaction times in slow networks and lower the network traffic in scenarios where users are charged on a traffic amount basis. • The transferred remote user interface must be adaptable to the client device, e.g. allowing high-quality client-side scaling and rotation-specific layouts. • To achieve high user acceptance, the system should enable the usage of native UI-elements on client side to represent a system-specific design and behavior. • The server component of the system must be portable to a variety of platforms including platforms that do not natively support graphics rendering (for example due to the absence of a display). • There must be a possibility for server-initiated updates to allow asynchronous events occurring on the server to be propagated to the client. The aim of the RemoteUI Project is to develop technologies for realizing a high-performance, portable remote user interface system satisfying these requirements. In this paper, we propose the RemoteUI System and present the results of an in-depth comparison with existing well-known solutions. The remainder of this paper is structured as follows. In Section II we discuss existing work in the areas of thin client computing and remote control solutions. In Section III the architecture of the RemoteUI System and the functionality of its protocol are defined. For our experimental evaluation we implemented prototypes for the proposed system, and two existing representative solutions, as described in Section IV. With these prototypes we accomplished different experimental evaluations introduced in Section V. In Section VI we describe the setup and the results of the bandwidth and traffic volume comparison tests. In Section VII the test bed of the latency tests and experiment results are discussed. We summarize our observations and conclude in Section VIII. II. RELATED WORK Transferring graphical user interfaces from a server to a client via network connections can be realized through two major approaches. The first approach originated from the thin client concept, and typically employs pixel information or low-level graphic primitives for this purpose. For instance, systems like Virtual Network Computing (VNC) [1], Muse [2] and the system proposed by Simoens et al. [13] retrieve the required bitmap information from the framebuffer at the server. Systems like the Remote Desktop Protocol (RDP,

based on ITU-T T.128 [3]), THINC [4] and BASS [5] work as a mirror graphic driver and intercept graphic commands. XServer [6] introduces a graphic abstraction layer and Joveski et al. [12] intercept the latter for accessing graphic primitives. Information gathered from these sources is encoded with special run-length codecs like ZRLE or TRLE [7] or existing codecs like PNG, JPEG, H.264 or Theora. Client and server are usually connected persistently over TCP or RTP. These systems have various advantages. Typically the (thin) client software has relatively low complexity and displays a pixelaccurate reproduction of the server's UI. Employing optimized codecs, these systems perform adequately when transferring video streams. However disadvantages exist regarding client adaptation, for example client-side scaling is possible but often not optimal on mobile devices. Additionally the bandwidth requirements of these solutions are relatively high and the responsiveness in high-latency environments suffers from the necessity for full client-server round trips. There are various applications of thin client systems especially VNC-based solutions in consumer electronic environments. Haraikawa et al. propose the µVNC system [8], which transfers the UI of consumer electronic hardware through VNC. In this way a video recorder can be controlled with the help of a regular VNC client on a PC. To lower the complexity of the server software and supersede the necessity of a real framebuffer, µVNC uses pre-rendered images. Mainly for the realization of remote video servers, the RVU Protocol [9] uses an approach similar to RDP to transfer user interfaces. Videos can be embedded into these UIs using specialized video codecs. Consumer electronic devices implementing such technology, which shall support special mobile-based UI delivery options in the near future, have already been shipped. To mirror the user interface of a mobile consumer electronic device to a car's head-unit, Bose et al. [10] describe a solution that extends VNC with special carrelated features. The device can be connected via USB, WLAN or WPAN. The European project Mobithin [11] has focused on the thin client computing with mobile devices. Different proposed systems [12]-[14] use parts of the MPEG4 standard to accomplish high performance. The other major approach utilizes widget descriptions to transfer user interfaces, allowing a high-quality client-side rendering and scaling. These descriptions are often formulated in XML languages supporting specialized elements for widgets like “ToggleButton” or “Progressbar”. Examples are HAViML [15], Verhoeven and Dees’ protocol [16], LRDP [17] and QML [18]. A similar approach explores HTML to realize and transfer rich user interfaces. Sonnenberg [19] proposes to utilize HTML for the integration of mobile devices into cars. Following the Web4CE standard [20], a consumer electronic device can offer a UPnP Remote UI service [21] in the form of an XHTML 1.0 page. The Web4CE standard also addresses a common problem of HTML- and XML-based solutions that use HTTP as the underlying protocol. A so-called NotifSocket object is defined that allows the server to send updates to the client through a persistent

TCP connection. Using HTTP, this can only be realized with client-side polling. The same problem is addressed with Websockets [22] that have been already implemented in most current web browsers, in contrary to NotifSocket. However, both solutions do not practically support efficient transfer of binary data to HTML-based clients, which is highly desirable for the transmission of bitmaps. Standardized JavaScript [23] clients at the moment can only receive data from these sockets in text form. Therefore, binary data have to be encoded e.g. with Base64 [24] before the transfer. Base64 introduces approximately 33% overhead as 6 input bits are encoded with 8 output bits, which adversely influences the bandwidth requirements of such a system. According to Nielsen [25] the user interface of a software system should react within 100 milliseconds to inform the user about direct object manipulations, e.g. changing a button’s color to express its pressed state. If the presentation of results in response to user input takes longer than 1.0 second, additional feedback is necessary. While network round-trip times (RTTs) in local area networks are typically below 100 ms, investigations show that RTTs in wireless wide-are networks (WWANs) such as UMTS can become considerably high. For a moving client sending 64-byte packets Wewetzer et al. [26] determined a mean RTT of about 600 ms. Systems working with widget descriptions in such scenarios have the advantage that no round-trip with the server is required to render the widget behavior needed for instant user feedback. In the proposed RemoteUI System, we leverage the advantageous features in existing approaches such as a persistent connection and efficient widget descriptions to form a high-performance UI system. Furthermore, we propose a range of techniques to minimize the required bandwidth and system latency. To evaluate the effects of these solutions we compare our system with two representative systems, VNCand HTML-based respectively. Both systems are most widely used and available on many platforms. This paper has substantially extended its preliminary version [27] by expounding the system’s design. Furthermore, the performance evaluations have been detailed through extensive experiments. The proposed RemoteUI system is compared with VNC- and HTML-based systems in terms of traffic volume and latency wherever appropriate. III. THE REMOTEUI SYSTEM The proposed RemoteUI System (RUI) consists of at least one RemoteUI Server that delivers user interface information. One or more RemoteUI Clients can connect to this server to retrieve and display its user interface. In Fig. 3 the components of the RemoteUI System are shown. We distinguish between server and client components. The Connection components establish a connection between the server and the client to enable the display of the user interface (client) and the delivery of the user interface (server) respectively. Different implementations for the Connection component are possible, e.g. WLAN- or WPAN-based. This component provides a connection object for the overlaying

Fig. 3. Components of the RemoteUI System (yellow/light) and application specific components (blue/dark)

tier. To establish a connection, an optional Discovery component can be used to find devices offering the RemoteUI functionality. The Event System uses the established connection to realize an asynchronous transfer of user interface descriptions, input events and system commands between the server and the client. The RUI-Protocol component is a class library describing the protocol entities. The Controller and Input components use it to create protocol objects at runtime. The Event System is responsible for their serialization, transmission, retrieval and deserialization. We use a platformindependent, binary serialization protocol derived from HBWSP [28]. This protocol is able to describe arbitrary objects including the efficient representation of binary data as required for bitmap transfer. On the server side, a Controller connects the existing Business Logic and the RemoteUI System. Its implementation highly depends on the scenario and the execution environment in which the RemoteUI System is used. In order to control a smartphone via RemoteUI, the Controller has to be implemented as a bridge to the platform’s UI components. On the client side, the Rendering component is responsible for displaying the received UI descriptions and commands. The Input component transfers input events originating from user interaction through the Event System to the server. A. UI-Tree and Tree Manipulation The RemoteUI System uses widget descriptions and graphic primitives to describe and transfer the user interface. These graphical objects build the so-called UI-Tree, which is replicated between server and client. Fig. 4 shows a UI-Tree (b on the left) and its rendered representation (a). While the contained image in this example must be transferred as pixel information, an abstract description of the displayed text and buttons is sufficient to create a client-side representation of the user interface. With this method, the required amount of data can be significantly reduced, as we will show in Section VI. When a new screen is opened, the server initially transfers its UI-Tree. The client receives, renders and saves a local copy of the tree. From then on, the client renders simple widget behavior like button presses or list scrolling locally. If serverside event-listeners have been registered on a widget, the client transfers the corresponding events to the server. The processing of these events usually produces visual results that

(a) Fig. 4. (a) The left UI-Tree being rendered on the phone, (b) UI-Tree manipulation

shall be displayed through UI-Tree manipulations, which are triggered by server commands. The client receives and instantly executes the commands, which includes rendering the resultant tree. To allow the manipulation, each node in the UI-Tree carries a unique ID that is used for addressing. Fig. 4 (b) exemplifies this process. The left tree is transformed into the right by replacing text and image information in different nodes. For certain widgets a round trip with the server is necessary to determine their appearance. The server for instance defines the state of a toggle button after it is pressed. During the following round-trip the button is in a “pending” state. If the round-trip time is high, this new state should be presented to the user. We call this phenomenon the “additional-state problem” that can generally occur in remote UI systems. The RemoteUI Client in future versions will provide special modified widgets that are capable of displaying these additional states. B. Incremental Transfer and Rendering Although the amount of data required to express a user interface with the RemoteUI Protocol is relatively small, there are situations where the transfer of a complete UI-Tree takes a long time. This can be the case in scenarios with low available bandwidth or the UI containing many images. To provide fast user feedback, the RemoteUI-Server incrementally sends the UI-Tree as soon as the first nodes become ready for transfer. The RemoteUI-Client incrementally receives, builds up and renders the local copy of the tree. This results in a row-by-row display of the user interface. If the first row of the layout can be displayed within less than a second, we assume this as sufficient for the required timely user feedback. Additionally, a less conspicuous progress indicator can be part of the interface to indicate a pending receive process. C. Prioritization The RemoteUI Protocol offers a mechanism to prioritize different user interface and command objects. The server processes and transmits objects with high priority first, which intentionally causes a delay for lower prioritized objects especially in low bandwidth scenarios. Because the client is able to incrementally render these objects, this technology reduces the user interface latency. In a preliminary approach,

(b)

widgets displaying pixel data (ImageViews) are separated from the UI-Tree and transferred as lower-prioritized SetData commands. Part of our research is the development of algorithms to determine the optimal priority from input parameters such as visibility, traffic volume, operational importance, available bandwidth and network latency. By continuously applying these algorithms at runtime, the system will adapt to changing environment conditions to offer the highest possible performance. D. Caching and Prefetching Theoretically a UI-Tree does not need to be transferred more than once. Although there are screen switches that are realized by replacement of the tree, it is likely that a tree will be reused once loaded. Avoiding the retransfer of resources like pixel-images used in a UI-Tree can reduce the required traffic volume significantly. To profit from this fact, the RemoteUI System caches resources on the client. Additionally, during idle-times, a low-priority process prefetches resources that are likely to be required later on. A tree that contains a resource that has already been fetched only needs to contain a reference to it, which reduces the user interface latency. IV. PROTOTYPIC IMPLEMENTATIONS As platform for all prototypic implementations we chose the Linux-based operating system described by McLoughlin [29] later on referred to as “target platform”. This system is available for different kinds of mobile consumer electronic devices like smartphones, tablets and multimedia hardware. Additionally a specialized version is available for TV systems predestining the platform for our prototypic implementations. A. RemoteUI System Prototype We realized the RemoteUI System with the components described in Section III for the above mentioned target platform. Fig. 5 shows the applications that have been developed by us. While the RUI Client was realized as a stand-alone-application, the server functionality was divided into different applications. It was an important requirement for us to avoid manipulations of the operating system or the necessity of gaining root privileges to run our applications. Access to the user interface on the target platform is allowed

Fig. 5. Deployment of the RemoteUI applications

from the owning application only. Therefore in our system an application must be enhanced with RemoteUI functionality in order to become remote controllable. A standard application project can easily be extended with RemoteUI functionality by weaving the required function calls and properties into the application’s bytecode. We developed three prototypic Remote Controllable Applications. A simple Calculator, a MediaPlayer and a Demo App. Fig. 6 shows Screenshots of the Calculator, and a special List Demo Activity that is part of the Demo App, where the number of list items can be configured. We used these UIs for our evaluations described in Sections VI and VII. On startup a Remote Controllable Application connects locally to the RUI Server application by inter-process communication (IPC). The RUI Server in the following works as relay for the communication with a connected RUI Client. The implementation uses a Connection module working with TCP-Sockets and a Zeroconf-based Discovery [30]. Images are transmitted in display size, encoded with full color PNG [31]. B. HTML System Prototype To compare the RemoteUI System with an HTML-based approach, we additionally implemented a prototypic dynamic web server for the target platform. The server uses HTTP 1.1 [32] to serve static and dynamic content, using a new TCP connection for each request. It serves static content like HTML files from the SD card handling each request in a new thread. Additionally it supports simple listeners (servlets) that

(a)

(b)

Fig. 6. Screenshots of the applications that we used for the performance comparisons: (a) Calculator, (b) List Demo

dynamically serve content when being registered for certain URLs. In contrast to the RemoteUI System's server, this solution is realized as a library and must be directly integrated into each application. Inter-process communication is not required in this way. Also this system does not automatically translate the native UI into HTML. We expected some performance benefits in terms of latency from these facts. We integrated the server into a version of the Calculator application. Following the Web4CE standard, this Calculator can be controlled via a static XHTML 1.0 web page shown in Fig. 7 (a). The page contains JavaScript code that sends an XMLHttpRequest [33] to a calculation servlet on each button press, carrying the corresponding button value as the GET parameter. The servlet on request modifies or evaluates the equation and updates the original Calculator's text view with the result. The updated text value is then sent to the requesting script in a JSON [34] response. The script writes this value into the respective HTML text area. Thus, the prototype works similarly to the RemoteUI System, keeping the business logic itself on the server. A second web page (Fig. 7 (b)) is an HTML version of the Demo List view. It shows 100*100 pixels thumbnails that are dynamically converted by a servlet from the originally 300*300 pixels application resources. Therefore the thumbnails are encoded and transferred in the same format as with the RemoteUI System. The list itself has no further functionality so the HTML page does not contain any JavaScript code. To measure the performance of the HTML system on the client, we implemented a small browser allowing us to process all events during page loads and XMLHttpRequests. This browser is based on the target system’s WebView widget that uses the same native browser library as the built-in system browser. C. VNC Server and Client We used a VNC server application running natively on the target platform. The software requires root privileges to get access to the device’s framebuffer. We modified the device accordingly to grant this access. On client-side we used a modified version of the open-source VNC viewer also used by Yu et al. [2] where we additionally enabled the portrait modus for controlled devices and introduced logging for our profiler.

(a)

(b)

Fig. 7. Screenshots of the web browser displaying XHTML versions of the test applications: (a) Calculator, (b) List Demo

D. Profiler Software For the latency comparisons described in Section VII we implemented special profiler software for the target platform. The profiler is able to measure the duration of arbitrary processes in a distributed environment. The logged events are stored in a process-local in-memory list. This list is written into a CSV-file when the profiling is finished. The profiling can be controlled through a special application that functions as profiler frontend on the device itself. We integrated the profiler library into all involved components.

Fig. 8. Bandwidth comparison for the Calculator

V. EXPERIMENTAL SETUP We carried out different experiments to compare the required bandwidth and latency of the RemoteUI System with those of HTML and VNC. For all tests we used a device with a resolution of 540*960 pixels, a dual-core 1.2 GHz RISC CPU and 768 MB RAM to execute the server implementations (RemoteUI Server, web server and VNC server). The client device offered a resolution of 600*1024 pixels, a 1 GHz RISC CPU and 512 MB RAM. The RemoteUI Client, VNC client and our HTML browser were installed on this device. On both devices we installed the profiler frontend. VI. BANDWIDTH AND TRAFFIC VOLUME COMPARISON A. Experimental Setup To compare the bandwidth and traffic volume we used the Calculator applications and the List Demos described in Section IV. In one experiment we configured the List Demo to contain 50 items to analyze the effect of scrolling (List Demo 50). In a second experiment the List Demo was displaying only 9 items (List Demo 9); in all remote systems the 9 items could be displayed without scrolling. The latter experiment was required to determine the traffic volume produced during the latency tests described in Section VII. With capture software running on the notebook we recorded the TCP traffic when executing the test applications. For the VNC connection we chose the full color depth (32 bit per pixel) and the ZRLE encoding. RemoteUI and HTML also transferred images in full color (PNG) so the display quality was comparable. Both test applications were started using the respective remote clients. A start button on a remote screen was touched, which opened either the respective view to be transferred (RemoteUI and VNC) or the web page containing the screen's replica. After being loaded, the Calculator was used on the client side to carry out a set of simple calculations. The List Demo 50 was used to scroll down the list of items. B. Experimental Results The results of the bandwidth consumption and data measurements are shown in Fig. 8, Fig. 9 and Table I. The bandwidth plots for the Calculator experiment (Fig. 8) show that after the initial transfer of the UI, RemoteUI and HTML required only few data to transfer button clicks and the corresponding text area updates (seconds 10-40). VNC in contrast uses pixel area updates to transfer the changes in the text area, which caused considerably more traffic. RemoteUI

Fig. 9. Bandwidth comparison for the List Demo 50 with scrolling TABLE I ABSOLUTE TRAFFIC VOLUME OF REMOTEUI, HTML AND VNC Calculator List Demo 50 List Demo 9 (with scrolling) (no scrolling) VNC (kB) 203.428 4523.162 261.046 HTML (kB) 25.966 1118.706 202.542 RemoteUI (kB) 19.202 684.143 205.712

in this case reduced the required traffic volume compared to VNC by over 90%. HTML was very efficient as well, with a reduction of about 87%. The HTML version of the Calculator uses nearly no styling information and contains only a few lines of JavaScript code. Nevertheless this simple implementation already caused about 25% more network traffic than the RemoteUI System that is able to render the UI in the system's native style. It is expected that a full-featured HTML version would require much more data. A similar result emerged for the List Demo 50 experiment. While RemoteUI required about 684 kB for the transfer, HTML without any JavaScript code and minimal styling used over 1.1 MB. However, Fig. 9 shows that both systems load all list items in advance so that no traffic is generated from client-side scrolling events. This is a weak point of VNC where every scrolling action produced a considerable amount of pixel data transfer (seconds 5-60). These high data requirements also degrade the system's responsiveness, making scrolling a difficult business for the user. We terminated the experiment after 60 seconds with resulting 4.5 MB of VNC traffic. Under these circumstances, VNC does not satisfy our requirements for scenarios with low bandwidth or for users being charged on a traffic volume basis. Transferring the 9-item list without scrolling in all three systems led to only the initial transfer of the UI. RemoteUI and HTML required about 200 kB. VNC, with 260 kB, also had modest traffic volume requirements. For this configuration mainly the latency of the systems is interesting. This is influenced by the systems' designs and implementations, as examined in the following section.

VII. LATENCY COMPARISON Equally important to the amount of data transfer is the system’s latency. Evidently, the faster the reaction times of the system, the better the user’s experience. Data minimization can improve the latency of such a system especially in low bandwidth scenarios. However, other parameters such as computation cycles, device performance and parallelism of the implementation highly affect the latency. A. Experimental Setup For the latency analysis we conducted 18 test series with 50 single test runs each. A test run consisted of the List Demo 9 (described in Section IV.A) being started by clicking a button on a client-side start screen. With the help of our profiler we measured the time between the touch-up event and the first and last drawing events of the client when rendering the received UIs. We refer to these times as “First Draw” and “Full Load” RTTs. To connect the client and the server, we selected a WLANrouter with IEEE 802.11g support and an extendable Linuxbased firmware. We installed tools for traffic shaping onto this device. For our WLAN configuration we chose the free channel 7 and WPA-2 encryption. We conducted all tests with three different maximum available bandwidths and two different network latencies, resulting in six test scenarios. At first we used the WLAN's native data rate and later simulated a 2.1 Mb/s and a 384 kb/s connection with the traffic shaping functionality of the router. We chose these bandwidths to obtain an impression of the systems’ behavior under lower-speed wireless networking conditions, and typical WPAN or WWAN data rate environments. It is noted that we are aware that different radio access technologies have different protocol stacks and operation conditions. Thus, the results of that subset of the tests can only be rough indicators for the behavior in WPANs or WWANs. The network latency in the first three test series was the default latency of the WLAN configuration. With a ping command we measured an average network RTT of 16 ms. In the other series with the help of the router firmware we introduced an additional latency of 292 ms. The resultant network RTT of 600 ms simulates a moving client in a UMTS environment as described by Wewetzer et al. [26]. B. Experimental Results Fig. 10 and Table II show the results of the measurements conducted with a network RTT of 16 ms. The times for a first draw event taking place (“First Draw” RTT) are represented by dark colors. The VNC client lacks an incremental rendering feature, thus only the complete load times have been considered. It can be seen that RemoteUI performed significantly better than VNC: the RTTs for the complete loads were 40 to 50% faster. Additionally, RemoteUI was able to present a first rendered result in under a second in all scenarios. This accelerated rendering is important for the user feedback and supersedes the necessity for additional feedback during load times.

Fig. 10. Mean round trip times with std. deviation indicators for starting the List Demo 9 with a given network RTT of 16 ms TABLE II MEAN RTT [MS] FOR STARTING THE LIST DEMO 9 WITH A GIVEN NETWORK RTT OF 16 MS WLAN 2.1 Mb/s 384 kb/s HTML First Draw 332.83 335.22 334.04 Complete Load 909.93 1263.61 4002.24 RemoteUI First Draw 966.25 945.24 910.70 Complete Load 1513.57 1578.86 5559.33 VNC Complete Load 2875.44 3825.09 9074.79

Even faster than RemoteUI was the tested HTML solution. In all scenarios, the browser presented a first rendering after about 330 ms. The complete load times varied depending on network bandwidth but in all scenarios were faster than the ones of RemoteUI. In the WLAN scenario, the system was able to present a completely rendered result within less than a second. There are different reasons for HTML being faster than RemoteUI in this scenario. We used a static HTML page with minimal style sheets to represent the list. The time for converting the server’s UI into a UI protocol as done by RemoteUI and VNC is therefore unaccounted in the HTMLbased system. Additionally the HTML client used the browser library built into the operating system of our target platform. This library is highly optimized and runs natively on the processor while the RemoteUI Client and the VNC client are implemented in code executed by a virtual machine. To achieve optimal results for the client-side behavior of HTML we also had to implement special JavaScript code. A typical browser on devices with touch screen waits after a single touch for a possible second touch forming a double touch event. This gesture is used for zooming and therefore must be considered by the browser. The wait time is typically 300 ms, during which the standard click event is not yet triggered. This behavior can only be overridden by introducing non-standard listeners in JavaScript, which operate on the mobile-browser specific touchEnd event. This approach however introduces problems to scrolling gestures. Nevertheless, for this test we implemented a touchEnd-based trigger to achieve a low RTT. The results otherwise would have contained the additional 300 ms reaction time.

Fig. 11. Mean Round Trip Times for starting the List Demo 9 with a given network RTT of 600 ms TABLE III MEAN RTT [MS] FOR STARTING THE LIST DEMO 9 WITH A GIVEN NETWORK RTT OF 600 MS WLAN 2.1 Mb/s 384 kb/s HTML First Draw 919.53 923.32 922.14 Complete Load 2121.72 2442.01 5182.64 RemoteUI First Draw 1550.85 1530.54 1498.30 Complete Load 2075.02 2165.26 6148.93 VNC Complete Load 3462.54 4420.19 9664.19

Fig. 11 and Table III show the results for the experiments conducted with a given network RTT of 600 ms. Under these conditions the additional network RTT has more or less been added to the RTTs of RemoteUI and VNC. RemoteUI in this scenario rendered a first result in about 1.5 seconds whilst HTML still achieved a first user feedback in under a second. However, the complete load times for the HTML page now exceeded the ones of RemoteUI in the WLAN and 2.1 Mb/s scenarios. The reason lies in the system design of HTML. The images’ URLs are contained in the HTML code, hence the browser can only retrieve the images after loading the HTML document. This introduces a second round trip, causing an effective network delay of about 1.2 seconds in this scenario. RemoteUI does not require the latter because the server pushes the images directly after transferring the UI-Tree. VIII. CONCLUSIONS We presented a new system for remote user interfaces that is applicable for a variety of usage scenarios related to consumer electronic devices. The system allows the realization of remote control applications that can efficiently be used in local and wide area networks accordingly. In order to achieve high performance regarding required bandwidth and system responsiveness, we combined different approaches from thin client computing and widget based UI systems. These include a persistent connection to realize server-side updates and the usage of abstract view descriptions to enable efficient client-side scaling and overall bandwidth reduction. We further optimized the system taking into account the context of wireless networks and mobile devices. In contrast to pixel-based thin clients, our solution is able to render basic UI behaviors on the client. This technique substantially

increases the system's responsiveness in high-latency network environments. Additionally our client implementation is much thinner than traditional web browsers, because it does not require a JavaScript engine. The latter is indispensable if HTML is used to realize rich UI functionality Our experimental results show that the proposed RemoteUI System outperforms the pixel-based approach represented by VNC substantially regarding bandwidth, traffic volume and latency. Compared with a simple HTML solution, RemoteUI's required bandwidth is equal or lower. In some latency test scenarios, HTML showed better responsiveness when loading the web pages representing the user interfaces. However, in our simplistic implementation, the HTML solution lacks several features such as real dynamic UI-generation, native UI behavior and rich capabilities for user interaction. These features could theoretically be integrated into a specialized HTML client. The required JavaScript libraries, style sheets and images would significantly increase the traffic volume. In our latency tests we demonstrated the positive effect of our proposed prioritization mechanism. The images in our example were transferred with lower priority, enabling the RemoteUI Client to partially render a preliminary UI. The server automatically pushes the images subsequent to the UITree. This design prevents additional network round trips, leading to RemoteUI being faster than HTML under high latency conditions. In our future work we will further investigate mechanisms for the proposed prioritization scheme. Future versions of the RemoteUI System shall e.g. realize automatic adaptation to changing network conditions. The prioritization scheme will be designed to incorporate this information. Other planned actions include the application of more efficient image encodings and the examination of optimized underlying transport protocols as a possible replacement for TCP. We also assume that there is a relation between required bandwidth, network behavior and energy consumption. Analyzing this relationship is useful to give evidence to the applicability of the system in situations with limited energy. In scenarios with unavoidable high network latencies we plan the introduction of special client-only view variants as a potential solution for the additional-state problem. REFERENCES [1] [2]

[3] [4] [5] [6]

T. Richardson, Q. Stafford-Fraser, K. Wood, and A. Hopper, “Virtual network computing,” IEEE Internet Computing, vol. 2, no. 1, pp. 33-38, 2002. W. Yu, J. Li, C. Hu, and L. Zhong, “Muse: a multimedia streaming enabled remote interactivity system for mobile devices,” Proc. 10th International Conference on Mobile and Ubiquitous Multimedia, pp. 216–225, 2011. ITU-T Rec. T.128, “Multipoint application sharing,” Jun. 2008. R. A. Baratto, L. N. Kim, and J. Nieh, “THINC: a virtual display architecture for thin-client computing,” Proc. 20th ACM Symposium on Operating Systems Principles, pp. 277–290, 2005. O. Boyaci and H. Schulzrinne, “BASS Application Sharing System,” Proc. 10th IEEE International Symposium on Multimedia, pp. 277–290, 15-17 Dec. 2008. R. W. Scheifler and J. Gettys, “The X window system,” ACM Transactions on Graphics, vol. 5, no. 2, pp. 79–109, Apr. 1986.

[7] [8]

[9] [10]

[11]

[12]

[13]

[14] [15] [16] [17] [18]

[19] [20] [21] [22] [23] [24] [25] [26]

[27]

[28]

[29]

[30]

T. Richardson and J. Levine, “The Remote Framebuffer Protocol”, IETF RFC 6143, Mar. 2011. T. Haraikawa, T. Sakamoto, T. Hase, T. Mizuno, and A. Togashi, “µVNC: a proposal for Internet connectivity and interconnectivity of home appliances based on remote display framework,” IEEE Transactions on Consumer Electronics, vol. 47, no. 3, pp. 512–519, Aug. 2001. RVU Alliance, “RVU Protocol Specification 1.0,” 03 Dec. 2009. R. Bose, J. Brakensiek, and K. Y. Park, “Terminal mode: transforming mobile devices into automotive application platforms,” Proc. 2nd International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 148–155, 2010. B. Vankeirsbilck, P. Simoens, L. Deboosere, B. Dhoedt, F. Westphal, T. Plantier, B. Lécroart, F. Prêteux, A. Dejonghe, and J. Point, “Bringing thin clients to the mobile world,” Proc. NEM Summit Towards Future Media Internet, 2008. B. Joveski, P. Simoens, L. Gardenghi, J. Marshall, M. Mitrea, B. Vankeirsbilck, F. Preteux, and B. Dhoed, “Towards a multimedia remote viewer for mobile thin clients,” Proc. of the SPIE, the International Society for Optical Engineering, vol. 7881, 2011. P. Simoens, P. Praet, B. Vankeirsbilck, J. De Wachter, L. Deboosere, F. De Turck, B. Dhoedt, and P. Demeester, “Design and implementation of a hybrid remote display protocol to optimize multimedia experience on thin client devices,” Proc. IEEE Australasian Telecommunication Networks and Applications Conference, pp. 391–396, 2008. B. Joveski, M. Mitrea, and F. Preteux, “MPEG-4 LASeR - based thin client remote viewer,” Proc. 2nd European Workshop on Visual Information Processing (EUVIP), pp. 125–128, 2010. R. G. Wendorft, R. T. Udink, and M. P. Bodlaender, “Remote execution of HAVi applications on Internet-enabled devices,” IEEE Transactions on Consumer Electronics, vol. 47, no. 3, pp. 485–495, Aug. 2001. R. Verhoeven and W. Dees, “Defining services for mobile terminals using remote user interfaces,” IEEE Transactions on Consumer Electronics, vol. 50, no. 2, pp. 535– 542, May 2004. V. Stirbu, P. Shrubsole, and J. Costa-Requena, “LRDP: The Lightweight Remote Display Protocol,” IETF Internet draft, 11 Feb. 2005. V. Stirbu and J. Savolainen, “Hypermedia-Driven Framework for Scalable and Adaptive Application Sharing,” REST: From Research to Practice, E. Wilde and C. Pautasso, Eds. Springer New York, pp. 199– 219, 2011. J. Sonnenberg, “Connecting in-vehicle entertainment with CE devices via Multi-Platform Web Applications,” Proc. IEEE International Conference on Consumer Electronics, pp. 515–516, 2011. CEA R7 Home Network Committee, “Web-based Protocol and Framework for Remote User Interface on UPnP Networks and the Internet (Web4CE),” Jan. 2011. UPnP Forum, “RemoteUIServer:1 Service Template Version 1.01.” 02 Sep. 2004. I. Fette and A. Melnikov, “The WebSocket Protocol,” IETF Internet draft, 30 Sep. 2011. Ecma International, “Standard ECMA-262 - ECMAScript Language Specification.” Jun. 2011. S. J. Josefsson “The Base16, Base32, and Base64 Data Encodings.” IETF RFC 4648, Oct. 2006. J. Nielsen, Usability engineering, Morgan Kaufmann, p. 135, 1993. C. Wewetzer, M. Caliskan, K. Meier, and A. Luebke, “Experimental Evaluation of UMTS and Wireless LAN for Inter-Vehicle Communication,” Proc. 7th International Conference on ITS Telecommunications, pp. 1–6, 2007. D. Thommes, Q. Wang, A. Gerlicher, and C. Grecos, “RemoteUI: A High-Performance Remote User Interface System for Mobile Consumer Electronic Devices”, Proc. IEEE International Conference on Consumer Electronics, pp. 670-671, Jan. 2012. (Best Poster Paper Award Winner) P. Lin and Q. Chen, “A Distributed Platform Based on HBWSP & XML for Net Resources Sharing,” Proc. IEEE International Seminar on Future Information Technology and Management Engineering, pp. 242 –245, 2008. I. V. McLoughlin, “Open-Source and Consumer Electronics - The Back Door to World Domination: Why Reinvent the Wheel (Especially When Nice People Are Giving Wheels Away for Free)?,” IEEE Consumer Electronics Magazine, vol. 1, no. 2, pp. 53 –58, Apr. 2012. B. Aboba, E. Guttman, and S. Cheshire, “Dynamic Configuration of IPv4 Link-Local Addresses.” IETF RFC 3927, May 2005.

[31] W3C, “Portable Network Graphics (PNG) Specification (Second Edition),“ W3C Recommendation. 10 Nov. 2003. [32] P. J. Leach, T. Berners-Lee, J. C. Mogul, L. Masinter, R. T. Fielding, and J. Gettys, “Hypertext Transfer Protocol – HTTP/1.1,” IETF RFC 2616, Jun. 1999. [33] W3C, “XMLHttpRequest Level 2,” W3C Working Draft. 17 Jan. 2012. [34] D. Crockford, “The application/json Media Type for JavaScript Object Notation (JSON),” IETF RFC 4627, Jul. 2006. BIOGRAPHIES Daniel Thommes (S’11) is a PhD student in a collaborative research program of the University of the West of Scotland (UWS), UK, and the Stuttgart Media University, Germany. Since 2010 he is Research Assistant and Lecturer with the Stuttgart Media University. Previously, he worked for several years as a Research Assistant and Consulting Software Architect. His research interests include mobile computing, wireless/mobile networks, web technologies, software architecture and usability engineering. He received the Diplom-degree in Media Technology Engineering from the Ilmenau University of Technology, Germany, in 2006. Since 2012 Daniel is member of the IEEE Consumer Electronics Society. Ansgar Gerlicher (M’06) Prof Ansgar Gerlicher is a Professor in Mobile Applications and Director of Research in Mobile Applications & Security with the Institute of Applied Science, Stuttgart Media University, Germany. Previously, he worked for several years as a Software Architect and Project Manager in the Telecommunication and Automotive Industries. His research interests include integration of consumer electronic devices in vehicles and mobile and embedded software architectures, frameworks and mobile security. He received his PhD degree in real-time collaboration systems from the London College of Communication, UArts, UK. Since 2012 he is a member of the IEEE Consumer Electronics Society. Qi Wang (S’02-M’06) Dr Qi Wang is a Lecturer in Computer Networking with the University of the West of Scotland (UWS), UK. Previously, he was a Postdoctoral Research Fellow with the University of Strathclyde, UK, and a Telecommunications engineer with the State Grid Corporation of China. He received his PhD in mobile networking from the University of Plymouth, UK, and his MEng and BEng degrees in electronic and communication systems from Dalian Maritime University, China. His research interests include Internet Protocol networks, wireless/mobile networks, and video networking. Recently, he has been involved in various international and national projects such as the European Union FP6 MULTINET project and the UK EPSRC DIAS project. He is the Principal Investigator of the UK EPSRC project entitled Enabler for NextGeneration Mobile Video Applications, and was a recipient of England ORS Award. He is on the technical programme committees of numerous international conferences. Christos Grecos (M’01-SM’06) Prof Christos Grecos is a Professor in Visual Communications Standards, and Head of School of Computing at the University of the West of Scotland (UWS), UK. He leads the AudioVisual Communications and Networks Research Group (AVCN) with UWS, and his research interests include image/video compression standards, image/video processing and analysis, image/video networking and computer vision. He has published many research papers in top-tier international publications including a number of IEEE transactions on these topics. He is on the editorial board or served as guest editor for many international journals, and he has been invited to give talks in various international conferences. He was the Principal Investigator for several national or international projects funded by UK EPSRC or EU. He received his PhD degree in Image/Video Coding Algorithms from the University of Glamorgan, UK.