An architecture for QoS data replication in network virtual environments

An Architecture for QoS Data Replication in Network Virtual Environments George V. Popescu and Christopher F. Codella IBM T.J. Watson Research Center {popescu, codella}@ us.ibm.com Abstract Predicted availability of broadband access will enable deployment of network Virtual Environments over the Internet. Online virtual worlds will require efficient runtime data replication solutions. This paper presents a Quality of Service (QoS) architecture for just-in-time data replication in network Virtual Environments. Quality of service is achieved by predicting the load and adapting to network traffic variations. Data is prefetched at the client based on network traffic estimates and viewpoint navigation prediction. QoS negotiation allows the server to control the network resources allocated per client. Experimental results show that QoS data replication can be implemented with reasonably small network and server overload.

1. Introduction Network Virtual Environment architectures have been successfully deployed over local area networks (see [14] for a review of networked Virtual Environments). With the advance of graphics capabilities on client platforms (PCs, game consoles) and availability of broadband Internet access (DSL, Cable Modem, etc.) networked Virtual Environment (netVE) technology is ready for deployment over the Internet. However the Internet is a much more heterogeneous environment, requiring new architectural solutions. A central problem in netVE design is geometry data replication. Replicating the whole database is not needed and may be prohibitive in case of very large Virtual Worlds. Therefore data needs to be replicated at run-time, while the user navigate the virtual world. Data replication policies for netVEs are designed to download data just-intime for being displayed. This is achieved by tracking the viewpoint and prefetching geometry data in Client's cache. Just-in-time data replication in netVEs has particular network traffic patterns. It does not have the "hard" realtime requirement characteristic to videostreaming, but imposes timeliness constraints on data download. The data replication methods proposed so far for netVEs ([4], [13]) use best-effort policies, guided by display optimization

Proceedings of the IEEE Virtual Reality 2002 (VR’02) 1087-8270/02 $17.00 © 2002 IEEE

criteria. When deployed over the Internet, such methods will lead to unexpected variations of VE quality determined by fluctuations in the Web serving infrastructure. Latencies, bandwidth limitations, network congestion, and server load variations can reduce the quality of service perceived by end users. In consequence data replication in netVEs requires a QoS approach. Previous application level QoS research was motivated by real-time applications such as video streaming, voiceover-IP, etc. Several QoS architectures were developed for distributed multimedia systems (see [2], [8] for a review). QoS data replication in netVEs, while imposing temporal constraints on data downloading does not require real-time response. Quality of service for netVE data replication can be achieved by predicting the load and adapting to traffic variations. We present in the following a QoS architecture for predictive data downloading in netVEs. The proposed architecture implements a flexible solution, which allows customization of serving, download, and caching policies. It allows the serving module to control the load and the client to negotiate a level of service that corresponds to its application profile. The architecture provides adaptation to varying network condition and serving load.

2. Related work Reducing Web latencies motivated the early research on predictive prefetching. Padmanaban and Mogul [12] showed that significant reduction of average access time of WWW documents could be achieved at the expense of an increase in network traffic by a similar fraction. In [5] prefetching was used to reduce latencies over a lowbandwidth connection between client and a caching proxy. A significant latency reduction was achieved without generating additional Internet traffic in the backbone. Prefetching was used in distributed virtual environments for geometry streaming [4] [9] [13]. Multiresolution geometries are stored at the server and sent to the client just-in-time for being rendered. This allows better resource utilization for very large geometry databases with complex models. Chim et al. [4] also propose a multiresolution caching mechanism optimized for Virtual Environment navigation.

A general framework for task-specific asset prioritization in distributed virtual environments has been proposed in [3]. The framework performs resource management based on quality, importance and cost (QUICK) ratings of each component of the Virtual Environment. The approach attaches to the scenegraph a set of QUICK annotations, used by resource managers to decide how to optimize the scene at run-time. The framework has been implemented in a prototype display and a distributed cache management system. These previous prefetch techniques can be characterized as “best effort”, as they do not attempt to provide QoS for the data replication mechanism. The prefetch policies are based on display optimization criteria, ignoring network-related aspects. Our approach uses load prediction and adaptation to provide QoS data replication. The method can be used in conjunction with client-side techniques to provide end-to-end QoS in netVEs.

3. Client server interaction for on-demand data replication The replication model assumed here uses scenegraph architecture [16] to model VE data. Client-server interaction is controlled based on the user navigation model. Two popular navigation models are continuous navigation and portal navigation. In continuous navigation the viewpoint moves incrementally between successive frames, controlled by an input device or by an animation sequence. Portal-based navigation consists of switching to another 3D universe. We discuss in the following a control model for continuous navigation. There are two choices for controlling data replication: 1) Pull model: data flow is controlled by the client. This requires that the client has acquired a description of the Virtual Environment before the navigation process starts. A practical implementation of this model uses a scenegraph with "NULL" geometry nodes. These nodes contain data indicating the URL of the geometry node and the size of the geometry file. The client application switches the NULL nodes with geometries as required by a data visibility mechanism provided that they are available in the client’s cache. 2) Push model: the server keeps track of client viewpoint movement through frequent updates and pushes new content into the client’s disk cache as needed. This model does not require any additional annotations in the scenegraph, but requires a constant client/server data flow and additional server CPU load per client. Client controlled data replication is well suited for browsing 3D Web content while netVEs supporting large numbers of dynamic objects (such as multi-player 3D games) could be implemented more efficiently with server


control. We discuss here client controlled data replication of static virtual environments. The replication model considered here places the database at a server (or caching proxy) in the WAN. The client (viewer) imports the scenegraph description and a minimum set of objects when the VE application is launched. Subsequently data replication is controlled based on the client’s data visibility model. The controller uses a three-level data visibility model: visible data, area of interest (AOI) data and unknown data. Visible data contains all objects rendered in a particular VE frame. Limiting the visibility of data is necessary in very large Virtual Environments in order to achieve interactive frame rates. The shape of the visible region was chosen circular for simplicity. In general, it can be an arbitrary shape, obtained at run-time using visibility culling methods [6]. Visible data changes continuously as the viewpoint navigates the Virtual Environment. As a consequence new data is constantly loaded in the main memory at run-time from the local cache. Every time a scenegraph node becomes visible, a new thread is dispatched to load the geometry data, which replaces the null geometry node. The node is now labeled visible data for the time it stays in the memory. When the node moves out of the visible region, it is labeled AOI data, its geometry is discarded from memory, and the null node is switched back in. AOI is a well-known technique for managing interactions between participants in a network Virtual Environment [7] [10]. In our model the area of interest (AOI) specifies a surveillance region used by the client for prefetch planning. AOI has a circular shape concentric with the visible data region. A collision detection mechanism signals when a null geometry node enters the area of interest. The null geometry nodes in the area of interest are placed in a queue for being downloaded in the cache according to the download policy. The rest of the scenegraph, containing null geometries, is classified as unknown data. The data accessibility model is presented in Fig. 1. We also show the cached area in order to contrast it with the AOI area (note that the shape of the cache area varies at run-time, being determined by the caching and expiry policy). AOI and the cached area partially overlap. Part of data in the AOI could be in the local cache if downloaded by the prefetch module. Depending on the size of the cache, some geometry data might still be in the cache while not in the area of interest. An important measure is the average data density of the virtual world. This is calculated as the size of the data contained in the virtual world over the extent of the virtual world. Using the data density and the speed of the viewpoint we can calculate an average bit rate at which objects enter the visibility area. The bit rate can be expressed as:

BR = 2 * DD * v * Rvisible _ area

(bps)

(1)

where DD is the average data density, v is the speed of the viewpoint and Rvisible _ area is the radius of the visible region. The data rate is obtained by considering the area displaced during an incremental movement of the viewpoint. This is used as the lower bound of the bit rate necessary for just-in-time replication. The rate is one of the parameters used in the QoS policy negotiations between client and server. Visible geometry Prefetched geometry Null geometry

Data Flow Volume Based on Movement in a Virtual Space

BR = 2dvR visible area density = d cache v Ai visible area

area of interest

Figure 1. Geometry prefetching model

4. The QoS Architecture The QoS model proposed here combines the data replication prediction model with network performance information. The download time is estimated based on the data size and the available bandwidth. This estimate is combined with the user navigation model to construct the data download prediction model. Requested data is prefetched just-in-time, following the client download policy. The replication process has four stages: Queue, Predict, Prefetch and Cache. Each fragment of data replicated at the client is initially placed in a priority queue. A predictor is used to trigger the replication request. The data fragment enters then the prefetch phase,


which ends when the data is replicated in the client’s cache. The predictive data download architecture has three components: Download and Cache modules on the client (receiving) side and a Serving module on the server side. A QoS agent, included in both Download and Serving modules, assures just-in-time data download. The symmetry of the QoS model allows easy configuration of several network topologies such as client server (push and pull models) and peer to peer. The architecture for client/server pull model is shown in Figure 2. The Data Management layer controls the data loading from local files or cache on disk and place requests in the download queue. The layer uses the data visibility model described in the previous section. A request is specified as of the file to be downloaded. The download queue buffers the requests of all objects in the Area of Interest. Buffering is also used in streaming applications. The difference is that while streaming applications transfer data in a sequential order, netVEs data replication uses a priority queue, where the order in which data is requested is decided by a Download Predictor component. The Download component is responsible for a just-intime delivery of the requested data. The download parameters are specified in the Download Policy component. The Download Predictor estimates the time variables required for just-in-time download, as shown in section 4.3. All objects in the download queue are continuously evaluated until they are prefetched. Prediction uses a parametric model combining viewpoint motion estimation, data size and network information. The component in charge of file transfers is the Download Scheduler. The scheduler starts file download based on previous step estimates and the policy specified by the application. The download threads place data on disk while adding an entry in the File Cache. In addition the download scheduler collect network statistics, such as bandwidth used per download thread. The Cache Module contains a File Cache - a hash table on disk data structure - and a Cache Manager. The Cache Manager handles the requests from data management layer and implements a replacement mechanism specified in the Cache Policy. The replacement mechanism for netVE data replication is described in section 4.4. The Serving Module has two components: a Load Adapter and a QoS agent. The Load Adapter controls bandwidth usage per download thread. Based on the feedback from the QoS Agent, the file server can block or redirect a file request if resources are scarce. The serving policy details how the server adjusts to overload and how it implements differentiation of services for download applications. The serving policy determines the resource allocation on the server and the adaptation to overload.

Application Data Management

C C ll ii ee nn tt

Download Module Application Profile

Download Policy

Download Predictor

QoS Agent Resource Resource Controller Monitor

Resource Broker

Request Queue

Cache Module Cache Manager

Cache Policy

File Cache

Download Scheduler

Network Interface

S S ee rr vv ee rr

Network Network Interface QoS Agent Resource Broker Server Policy

Resource Resource Controller Monitor

Serving Module

Load Adaptor

DB

Figure 2. Predictive data replication architecture

The QoS Agent performs negotiation and selection of serving resources and adaptation to variable network and server performance. The agent monitors and controls resources on the local machine and communicates with other agents in order to provide quality of service data serving. Due to its symmetric design (relative to negotiation and adaptation), the same agent structure can be embedded in download, serving or peer-to-peer modules. The QoS Agent has three components: Resource Broker, Resource Controller and Resource Monitor. The Resource Broker negotiates the download parameters with another QoS Agent located at the serving module. These are the reference parameters until a re-negotiation occurs due to changes in server load or network conditions. The negotiation follows a protocol detailed in section 4.2. At the Client site the Resource Broker uses a profile provided by the application. The profile contains the QoS-tuple: a set of parameters, which specify application QoS requirements. At the server site several classes of clients are specified by the serving policy. These classes contain limits for QoS-tuples and per class negotiation and adaptation policies. The Resource Controller implements resource allocation and adaptation at a fine grain level (per


download thread). This component is implemented if elements of network and OS reservation mechanisms (real-time scheduler, network bandwidth allocation, etc.) are available. The Resource Monitor periodically collects network statistic. The statistic is obtained through application level (download thread) measurements. Using network data, the effective bandwidth available for the download application is estimated. The estimated bandwidth is fed to Download Predictor (at the client) and Load Adapter (at server) to provide adaptive behavior to QoS parameter fluctuations.

4.1. QoS mapping QoS mapping translates netVE data replication requirements into QoS parameters. In our model, the mapping relates the data density of the virtual world to the download bandwidth. In order to support QoS mapping, the client internal scenegraph is extended with a data size field. This additional field is added to any online node, such as geometry, texture, etc. At the server, this requires either extension of the scenegraph modeling language (VRML) to accommodate the data size labeling, or

querying at run-time to obtain the data size of online nodes. Querying incur an additional CPU cost but scales better as it does not require the modification of VRML files stored at the server every time the online geometry files are changed. Therefore it is the preferred solution for the pull implementation. The virtual world parameters necessary for QoS mapping are: center (Cx, Cy, Cz) extents (Ex, Ey, Ez) and data weight (W[KB] ) (Figure 3). These parameters are calculated at the client obtained using a graph traversal algorithm, which computes the center, extent, weight (CEW) at each node in the scenegraph. The CEW of root is used to calculate the average data density of the virtual world: DD = Wroot ( Exroot * Ez root )

(2)

for the planar viewpoint motion. A volumetric data density can be calculated similarly for three-dimesional viewpoint motion. The CEW of each node in the hierarchy can be similarly used to determine local average densities. This would allow data replication at different levels of granularity (groups of objects such as game levels, etc.). Using equation (1) (for the maximum speed of the viewpoint), the average bit rate for the entire VE is calculated. The average bit rate Q_vis is the main QoS parameter for data download. In our model this is used as the minimum bit rate for QoS data replication. Using the same formula for the AOI we calculate the data rate for a replication policy that download all data in the area of interest without using prediction. This is used as the maximum bit rate – Q_aoi. The QoS negotiation interval is then [Q_vis, Q_aoi].

CEW=Center,Extent, Weight

Virtual World

Loaded geometry

CEW

Null geometry Compute CEW CEW

CEW

CEW

CEW

CEW

CEW

Figure 3. Scenegraph parameter computation


4.2. The QoS protocol QoS data replication requires client-server negotiation of download bandwidth. The negotiation of download bandwidth is intended to allow the server to control the network resources allocated per client. By using negotiation, QoS application parameters are shaped according to the available network resources. The QoS-tuple has the format . The first parameter represents the average download bandwidth available for the connection. The second term is the maximum allowable sever response time for a client request. The last term indicates the priority class, allowing differentiated treatment of the clients. The protocol starts with a preliminary step when the client is probing the network resources. This includes latency measurements and bandwidth availability on the routing path to the server. A ping-like utility is used to determine network latencies. Active bandwidth measurements are performed next. After collecting these data, the client set the QoS target based on the network measurements. The target should be in the QoS interval calculated in the previous section. Then the client starts the QoS negotiation process. The protocol consist of three steps: 1. Request: QoS client requirements (target and QoS interval) are announced to the server. 2. Adjustment: The server adjusts the QoS-tuple based on its QoS serving policy and available resources. The parameters can be accepted without modification, be degraded or rejected. The QoS-tuple is returned to the client. 3. Confirmation: The client accepts the adjusted parameters if in the QoS interval and set the download model parameters. If rejected, the client restarts the negotiation process. While the protocol does not make network reservations, we assume that some form of network policing (such as [11]) is in effect between the client and the edge of the network.

4.3. Prediction and Prefetching The prefetching model uses the client negotiated download bandwidth for just-in-time data replication. The prefetch manager keeps a table with time penalties, Tdownload, (calculated for the estimated client network bandwidth) for all null nodes in the AOI. The size of the AOI is selected so that the largest (in terms of data size) VE object can be downloaded in time for being rendered. The time when the geometry becomes visible - Tvisible - is also estimated. Two methods were used to calculate this parameter; the first one uses the magnitude of viewpoint

speed vector, while the second one take into consideration the direction of the viewpoint motion. Whenever the time to get to a node is less than the download time plus buffer time (to account for variation in connection bandwidth), a download thread is dispatched to get the geometry into client’s cache: Tdownload = Size(object ) Bwdownload

(3)

Tbuffer = Tresponse + T param

(4)

I. Tvisible =  VC obj − Rvisible _ area  v (5)   II. Tvisible =  VC obj − Rvisible _ area  v , VC obj VC obj (6)   where v is the speed of the viewpoint and VC obj is the vector viewpoint-object. The replication policy is: Tvisible < Tdownload + Tbuffer : start object replication (7)

Speed estimation used in the second Tvisible estimate employs a moving average predictor. This predictor gives good performance for a smooth, continuous viewpoint motion. Other predictors were shown to perform slightly better for various viewpoint motion patterns [4]. The buffering time is composed of the response time of the server and a parameter proportional to standard deviation of instant traffic measurements. Figure 4 illustrates the just-in-time download mechanism for the first replication method. The replication condition was represented using circles proportional to the time labels in equations (3) to (5). The download of geometry starts when its circle overlaps the viewpoint circle. Using the second Tvisible equation the prefetch is further optimized to reduce the wasted bandwidth. Minimizing wasted resources is especially important for the server, as the prefetch mechanism requires significant sever bandwidth and CPU load.

v*T_download(B) B

v v*T_buffer

A v*T_download(A)

start download

Figure 4. Prediction for just-in-time data download


4.4. Cache management Cache management is performed based on a replacement policy specific to netVE data replication. The goal of replacement policy is to keep in the cache as long as possible the nodes closer to the viewpoint. That requires continuous evaluation of distances from the viewpoint of the cached nodes. Using a standard Last Recently Used algorithm will not give good performance unless viewpoint motion is unidirectional. Therefore a different cache management policy needs to be considered. To increase the cache hit ratio (or saved data requests), the replacement mechanism needs to be driven by viewpoint motion. The policy proposed here uses the same timetable as the prefetching mechanism. Each node in the cache is labeled with: Treplace = Tvisible − Tdownload

(8)

According to this formula, the nodes with the biggest Treplace are the ones further away from the viewpoint and which incur a smaller penalty to be downloaded again. Therefore the expiry policy replaces the nodes in the decreasing order of their Treplace label. These labels are evaluated every time a new data request requires more space than is available in the cache. When removed from the cache, such node is again labeled unknown data and requires a new prefetch if becomes of interest.

5. Implementation and Results A prototype of the QoS architecture was implemented using Java (Client/Server interaction) and Java3D [15] (3D Client). The client ran on a Pentium III 866 MHz PC with a GEForce2 graphics card. The geometry data was placed at a server in a WAN. The average bandwidth measured between client and server was 1 Mbps. The VE consists of 1000 static 3D objects [1] arranged randomly on a terrain. The terrain map is 4000 by 4000 units. Each object has a distinct URL. Data size of geometry objects varies between 200 KB and 1 MB; the corresponding size of the geometry database is between 200 MB and 1 GB. Doom style navigation was used for viewpoint motion. The maximum speed of the viewpoint is 1 unit per frame. The frame rate was limited to 20 fps. The first experiment recorded the replication bandwidth during a user controlled VE navigation session. The data rate from equation (1) is 800 Kbps for this experiment. The QoS negotiated rate is 1.0 Mbps. The data collected from passive monitoring of bandwidth threads is shown in Figure 5. The bandwidth was sampled at the end of each download thread. The required bit rate (800 Kbps) is shown as a dashed line. The result shows the end-to-end performance of the system. Performance

variation was small, allowing all geometry data to be replicated in time at the client.

Figure 5. Passive monitoring of data replication bandwidth

We designed a set of experiments to measure the prefetching performance. The performance metric chosen is wasted bandwidth - the sum of bytes prefetched but not used at the client. This metrics is critical for such prefetch systems. While prefetching adds a percentile load to the client, it can put a heavy load on the servers and network backbone. Wasted bandwidth and CPU cycles add up with the number of connected clients. Therefore maintaining a small waste while satisfying the timeliness requirements of the interactive application is key to large-scale deployment of such architecture. The prefetching method described in section 4.3 was compared with a fixed AOI-based method, with no estimate of download time. Three cases were considered: 1) AOI 1: all data in the AOI is prefetched to the client without using prediction; 2) AOI 2: the prefetch method of section 4.3 which uses the maximum speed of the viewpoint; 3) AOI 3: the prefetch method of section 4.3 using the second method of Tvisible estimate. The experiment measured the wasted bandwidth during continuous navigation in the VE. The speed of the viewpoint was adjusted in each experiment so that the required data rate estimated by (1) was kept constant at 800 Kbps. Wasted bandwidth was expressed as a percentage of the total bandwidth used by the application. Viewpoint motion was linear, to record the largest waste during VE navigation. The results are shown in Figure 6. The three methods provide almost constant performance for varying average data density of the VE. The second and third methods outperform twice and respectively four times the fixed size AOI. The result


shows that QoS data replication in netVEs can be implemented with reasonably small network and server overload.

Figure 6. Prefetch performance: wasted bandwidth vs. VE data density.

6. Conclusions and future work We presented in this paper a QoS approach to data replication in network Virtual Environments. The proposed QoS architecture implements a flexible solution, which allow customization of serving, download, and caching policies. Quality of service is achieved by predicting the load and adapting to network traffic variations. The negotiation of download bandwidth allows the server to control the network resources allocated per client. A simple data visibility model was used for prototyping. Data is prefetched just in time for being displayed, based on an estimate of download time and viewpoint motion prediction. The prefetch method is optimized in order to reduce the wasted bandwidth. Experimental results suggest that the QoS data replication architecture can be deployed efficiently over broadband links. Additional work is required to evaluate the performance of the data replication architecture. Experimental work to test the behavior of the server with a large number of connected clients is ongoing. Additional server adaptation policies need to be explored. Such policies should allow the server to downgrade the quality of the content served to the clients (depending on their priority class) to reduce congestion. Another direction of work is to implement QoS adaptation by varying the resolution of the geometry objects. The resolution of replicated objects is changed to respond to small variations in traffic shape from a negotiated average bit rate between client and server.

References [1] 3D Café. 2000. 3D models database. Online at: http://www.3dcafe.org/ [2] Aurrecoechea, C., A. Campbell, and L. Hauw, “A Survey of QoS Architectures” Multimedia Systems Journal, Special Issue on QoS Architectures, 6(3), pp. 138-151, May 1998. [3] Capps, M. “The QUICK Framework for Task-Specific Asset Prioritization in Distributed Virtual Environments,” Proceedings of IEEE VR2000, pp. 143-150, 2000. [4] Chim, J., M. Green, R. Lau, H. V. Leong, and A. Si, “On caching and prefetching of virtual objects in distributed Virtual Environments,” In Proceedings of ACM Multimedia '98, ACM Press, pp. 171-180, 1998. [5] Fan, L., P. Cao, W. Lin, Q. Jacobson. "Web prefetching between low-bandwidth clients and proxies: potential and performance." Proc. of ACM SIGMETRICS Conference, pp. 178-187, 1999. [6] Funkhouser, T., “RING: A Client-Server System for MultiUser Virtual Environments,” ACM SIGGRAPH Special Issue on 1995 Symposium on Interactive 3D Graphics, Monterey, CA, pp. 85-92, 1995. [7] Greenhalgh, C., and S. Benford, “MASSIVE: a Distributed Virtual Reality System Incorporating Spatial Trading,” Proc. 15th IEEE International Conference on Distributed Computing Systems (ICDCS'95), Vancouver, Canada, May 30-June 2, IEEE Computer Society, pp. 27-34, 1995.


[8] Hafid, A., G. von Bochman, and R. Dssouli, “Distributed multimedia application and quality of service: a review”, Electronic Journal on Networks and Distributed Processing, 2(6), pp. 1-50, 1998. [9] Hesina, G., and D. Schmalstieg, “A network architecture for remote rendering”, Proceedings of Second International Workshop on Distributed Interactive Simulation and Real-Time Applications, pp. 88-91, 1998. [10] Macedonia, M., M. Zyda, D. Pratt, P. Barham, and S. Zeswitz. “NPSNET: A Network Software Architecture For Large Scale Virtual Environments.” In Presence: Teleoperators and Virtual Environments, 3(4):265-287, 1994. [11] K. Nichols, V. Jacobson and L. Zhang. “A Two Bit Differentiated Services Architecture For The Internet,” RFC 2638, July 1999. [12] Padmanabhan, V. N., and J. C. Mogul, "Using Predictive prefetching to improve World-Wide Web Latency." ACM Computer Communication Review, pp.22-36, v.27, n3, July 1996. [13] Schmalstieg, D., and M. Gervautz. "Demand-Driven Geometry Transmission for Distributed Virtual Environments." In Proceedings of Eurographics '96, pp. 421-432, 1996. [14] Singhal, S., and M. Zyda. Networked Virtual Environments, New York, ACM Press, Addison-Wesley, 1999. [15] Sowizral, H., K. Rushforth, and M. Deering, The Java 3D API Specification. Addison-Wesley, 1997. [16] Walsh, A., and M. Bourges-Sevenier, Core Web3D. Prentice-Hall. 2001.