Performance Evaluation Framework for Video Applications in Mobile

2009 Second International Conference on Advances in Mesh Networks

Performance Evaluation Framework for Video Applications in Mobile Networks Alexander Klein and Jirka Klaue EADS Innovation Works Munich, Germany [email protected]

Abstract

is also interest by the military to enable distributed wireless video communication between different kind of mobile units.

We present a framework for performance evaluation and optimization of video applications in mobile multi hop networks. We focus on the perceived video quality depending on the simulated mobility model, traffic pattern, retransmission strategy, and the configuration of the routing protocol. Moreover, the functionality of the framework is demonstrated by evaluating the performance of a video surveillance application with multiple mobile sources and sinks. The application is designed for exchanging multimedia content, e.g. audio and video streams, in wireless networks where all nodes are highly mobile.

Due to the high mobility, the used applications and routing protocols have to deal with frequent link breaks and topology changes. The protocols have to find new routes within a short amount of time in order to achieve a high video quality while limiting the routing overhead to the absolutely necessary. Another optimization problem is represented by the trade-off between frame loss and end-to-end delay. Lost frames cause quality degradations of the video while retransmissions of lost packets result in additional delay which is unacceptable for real-time applications. Usually a play-out buffer is used at the video sink. The size of the buffer depends on the delay requirements of the application which also limits the number of possible retransmissions. Typical video players buffer up to several seconds of video data depending on the application. For that reason, the retransmission strategy of the video application and the capability of the routing protocol to repair broken routes are the two major key elements in mobile wireless networks.

1. Introduction The interest in wireless mesh networks for transporting real-time video content has grown, since more and more mobile devices come with high data rate interfaces e.g. UMTS and IEEE 802.11. Many of these devices have a large display which is quite sufficient for watching videos. Furthermore, they provide enough computational power to operate as a router in wireless networks. If the number of these devices keeps increasing, it will be possible in the near future to cover large areas with cheap internet access by using these devices to build a wireless mesh network. Besides video applications for entertainment, applications for surveillance are of great interest. In most of the currently installed wireless networks for video surveillance, the sources send the video content to a single data sink which evaluates the received streams. Both, the sources and the sink are usually not mobile. Therefore, the topology of these networks can be calculated in advance. If the topology is known, optimized routes can be calculated without much effort. However, the next generation of video surveillance applications will consist of mobile and fix nodes whereas the fix nodes build an infrastructure mesh backbone [1]. A large number of applications can benefit from high data rate wireless networks. Police patrol cars and stationary cameras can build a wireless mesh network which could significantly improve their surveillance possibilities. In addition, the broadcasting of large outdoor sport events, e.g. Tour de France [2] or Paris Dakar, could be improved if wireless mesh nodes would be attached to the bicycles or the support cars. Finally, there

978-0-7695-3667-5/09 $25.00 © 2009 IEEE DOI 10.1109/MESH.2009.15

The modular framework that is presented in this paper can be divided into two parts. The first part is implemented using the OPNET Modeler 1 discrete event simulator and is responsible for the simulation of the wireless mesh network. It consists of several smaller modules which can be replaced or modified without much effort to meet given requirements. We implemented new models for the following tasks: traffic generation, application, routing, MAC, mobility, energy consumption, overhead calculation, and visualization. The second part is represented by the video quality evaluation framework EvalVid [3] which calculates the Peak Signal to Noise Ratio (PSNR) and the Mean Opinion Score (MOS) values of the received video in order to get a better impression of the perceived video quality. We demonstrate the functionality of our framework by evaluating the performance of a video application with a varying number of traffic sources. The parameters of the routing protocol and the video application are optimized in respect to the MOS of the perceived video.

1. OPNET Modeler, University Program, http://www.opnet.com/services/ university/

43

Authorized licensed use limited to: Technische Universitaet Berlin. Downloaded on November 10, 2009 at 05:59 from IEEE Xplore. Restrictions apply.

2. Simulation & Video Quality Framework The framework that is presented in this paper consists of two parts. The first one is represented by a modular simulation framework which is implemented with the OPNET Modeler and is responsible for the simulation of the wireless communication. The second part is called EvalVid and is used to generate and evaluate the simulated video traces. Both parts are introduced in more detail in this section after a short overview of related evaluation tools. There are several commercial video quality evaluation tools available e.g. [4] and [5]. Both tools mainly focus on the evaluation metrics and do not offer an interface to connect them directly to network simulators, e.g. OPNET or ns-2. Another well known tool is represented by the Video Quality Metric (VQM) Software [6] which is developed by Wolf and Pinson. The tool offers a large number of different metrics in order to evaluate and compare the quality of videos. However, just as its commercial counterparts it does neither offer the functionality to create trace files nor an interface to interact with network simulation tools for performance studies. The video quality evaluation software Aquavit [7] was developed by Kasai and Nilsson. It offers almost the same functionality but it is not further developed. Therefore, it does not support state-of-the-art codecs like H.264. In one of our previous works we extended the basic Evalvid tool to simulate rate adaptive MPEG-4 transmission [8] with ns-2. The ns-2 extension only supports the MPEG-4 transmission and is based on an older version of EvalVid. Thus, we decided to use our OPNET simulation framework since it offers more flexibility due to its modular structure. The OPNET simulation framework consists of different process models that are able to communicate with each other. Figure 1 shows the configuration of the framework that is used in this paper. The arrows represent traffic streams which can be used to pass data packets from one process model to another. However, process models can also exchange information without using streams. This kind of information exchange is used by the Data Sink model and the Overhead Module to pass the collected statistics to the Statistic Module for further evaluation. Traffic Module The Traffic Module is based on the OPNET standard traffic generation process, but has advanced features. The module offers the possibility to generate single packets and data bursts. In addition, trace files captured by Wireshark 2 or tcpdump can be used for traffic generation. Thus, we use the Traffic Module to generate data packets according to a previously recorded video trace. Application Module The Application Module is used to modify the incoming and outgoing data packets to simulate the behavior of different applications, like the buffer of a

Figure 1. OPNET Framework

video application or data aggregation. We added different kind of retransmission strategies in order to find out which strategy offers the best performance in our scenario. Routing Module AODV, OLSR, GBR, and SBR are currently part of the framework. In fact, AODV and OLSR are already included in the OPNET Modeler library. However, we re-implemented them to speed up the simulation since most of their features, like multiple gateway support, are not required in most of the scenarios. Overhead Module The Overhead Module can be placed in different positions inside the framework to count the ingoing and outgoing overhead, e.g. application, MAC, and routing overhead. The collected statistics are forwarded to the Statistic Module at the end of the simulation. Mobility Module The Mobility Module includes several standard mobility patterns, like Random Waypoint, Random Walk, Random Direction, Manhattan Mobility, and Random Group Mobility. The nodes in the simulation can use different mobility patterns which allows the generation of more complex and realistic movement. In addition, the shape of the movement area can be chosen (square or circle). This feature is necessary since some mobility patterns, like Random Direction, do not generate suitable traces for all kinds of movement areas (e.g. high density in corners). WLAN Modules The models beneath the line in Figure 1 are taken from the OPNET Modeler library. The models are used to simulate the IEEE 802.11 (a,b,g) MAC and physical layer. We modified the signal propagation such that packets can only be exchanged between nodes that are less than 150 meters away from each other which results in a disc model. However, nodes that are further away will recognize noise on the radio channel during the transmission.

2. Network Protocol Analyzer, http://www.wireshark.org/

44


Table 1. ITU-R quality and impairment Scale 5 4 3 2 1

Quality Excellent Good Fair Poor Bad

Impairment Imperceptible Perceptible Slightly annoying Annoying Very annoying

Table 2. PSNR to MOS conversion PSNR [dB] MOS > 37 > 5 (Excellent) 31 - 37 4 (Good) 25 - 31 3 (Fair) 20 - 25 2 (Poor) < 20 < 1 (Bad)

Figure 2. Simulation Scenario

and type of each video packet transmitted over RTP, and use this trace to generate packets in the OPNET simulation. The Data Sink modules of the OPNET framework write a trace file of the received packets at the end of each simulation run. These traces are used by EvalVid to calculate packet/frame loss and delay figures as well as reconstructing the received (possibly distorted) video files. The received videos are then decoded using FFmpeg 3 to be able to calculate the PSNR and MOS figures for the video quality evaluation.

Before we start to describe the used evaluation tool, we want to introduce the two standard methods that are used to evaluate the video quality. One method to assess the performance of video transmission systems is to calculate the PSNR between the source and the received (possibly distorted) video sequence. It is a differential metric which is calculated image-wise and very similar to the wellknown SNR but correlating better with the human quality perception [9]. Thus, this metric is only meaningful if the quality of the original image sequence is high in terms of human perception which is not necessarily the case. For instance, if the video sequence is passed through a state-ofthe-art video encoder to reduce the bit-rate, the compressed video will be already distorted since modern video-codecs, like MPEG-4 or H.264, are usually lossy. Loss of packets will lead to decoding errors at the decoder/player while delay can cause buffer under-runs. Both will ultimately result in the loss of images at the player which results in a low video quality. For a better illustration of the meaning of quality measures for non-experts, the ITU-R developed a quality indication scale which is tied to the quality impression of human observers [10]. This scale is shown in Table 1. BT.500 further describes a methodology to gain these quality indicators by subjective assessment series (by a group of humans). Such a scale is often called Mean Opinion Score and used in several quality assessment systems. Ohm [11] gives a heuristic mapping of PSNR to MOS values which can be used to roughly estimate the human quality perception for videos with relatively low motion (like for instance videos from surveillance cameras). This mapping from PSNR to MOS is shown in Table 2. The tool that is used to evaluate the video quality is called EvalVid [3]. It is used here to compare the quality of the source (encoded and already slightly distorted video) with the received video quality in order to evaluate the performance of the simulated mobile wireless network. We recorded a trace of the original video file, containing size

3. Scenario 105 fix nodes are placed along a manhattan grid within a square of 2000 by 2000 meters as shown in Figure 2. The distance between two nodes along the grid is 125 meters resulting in an edge length of 375 meters. In addition, 16 mobile nodes are placed in the center of the scenario. Communication The standard OPNET IEEE 802.11b (ad hoc) mac and physical layer models are used. We modified the physical layer such that the transmission range of the nodes is limited to 150 meters. Nodes which are further away will only recognize a busy radio channel, but are not able to communicate with each other. Mobility The mobile nodes use the Random Group Mobility [16] mobility model. One of these 16 nodes is randomly selected as group leader. The group leader moves according to the Random Walk mobility model. The other 15 fellow nodes follow the group leader. Their movement in relation to the group leader is similar to the Random Waypoint model. The only difference is that the absolute speed of the fellow nodes is correlated to the speed of the group leader. Furthermore, their movement is limited such that they can not move further than 200 meters away from the group leader. The speed of the group leader is randomly chosen between 5 m/s and 20 m/s which represents the typical speed 3. FFmpeg - Multimedia Framework,http://ffmpeg.org

45


Table 3. SBR - Configuration Mode

3.00 s

Hello Message Interval

0.80 s

Decrease Routing Interval

0.80 s

Hello Message TTL Maximum Routing Value Routing Value Increase Function Routing Value Decrease Function

(a) Sample image 40 448 35 30 25

256

20 Bit-rate (avg = 258)

192

PSNR [dB]

Average Bit-rate [kbit/s]

384 320

15

128

10

64

5

0

0 0

10

20

30

40

50

60

70

32 20 Fast Divide

boundary of the manhattan grid, in order to minimize side effects caused by inhomogeneous traffic load distribution. Application Wireless links are usually lossy, especially in the case of a high node density and a high traffic load. However, packets are not only lost due to bit errors or interference. Packet loss is often the consequence of link breaks in mobile networks. Depending on the used routing protocol it may take up to several seconds until a new route is found. During this time interval, no communication between nodes that are using this link, is possible. For that reason, many applications use waiting queues to buffer data packets, if no valid route is available to the destination. Some applications use retransmissions and acknowledgements to guarantee a loss-free data exchange. The video surveillance application in our simulation has certain requirements regarding the maximum end-to-end delay. In fact, a maximum end-to-end delay of up to two seconds is still acceptable since there is no direct interaction between the source and the destination. Therefore, our example application uses selective end-toend acknowledgements and retransmits packets up to three times. A packet is retransmitted if no acknowledgement is received within 0.6 seconds. Thus, the maximum expected end-to-end delay will be slightly above two seconds which meets the given requirements. Routing The Statistic-Based-Routing (SBR) [14] protocol is used due to the fact that it is able to detect link breaks within a short amount of time. Furthermore, the protocol uses a delay based forwarding mechanism of routing messages and a continuous cumulative metric which results in a load balancing effect. The protocol is based on the concept of Directed Diffusion [15] which is a popular data-centric approach in wireless sensor networks. However, the higher data rate of wireless mesh networks allows a more frequent transmission of routing messages compared to sensor networks. The higher transmission frequency allows a fast detection of broken links. Nonetheless, the amount of routing overhead is approximately two percent of the transmitted video data which is on a quite acceptable level. A more detailed description of the routing protocol is given in [14] and [16]. The configuration of the SBR protocol is shown in Table 3.

512

PSNR (avg = 37.7)

Hybrid

Active Route Timeout

80

Time [s]

(b) Video profile & quality

Figure 3. Profile of “highway” video clip, (a) sample image (b) data rate & PSNR profile

of cars within the city area. A new direction and speed are chosen every 10 seconds. Video We selected a standard video sequence called ”Highway” which is used by a large number of video encoding and transmission studies, like the Video Quality Experts Group [12]. The sequence consists of 2000 frames with CIF resolution (352 x 288) and a frame rate of 25 Hz. The resulting 80 seconds long video was encoded with the stateof-the-art H.264 video encoder x264 [13] using an average target bit-rate of 256 kbit/s. A good balance between coding efficiency and error recovery capabilities was achieved by encoding a key-frame at least every 10 seconds. Figure 3 shows a snapshot from the original video sequence and the bit-rate profile in combination with the PSNR of the encoded video. The average PSNR of 37.7 dB reflects an excellent video quality. The video file is transmitted in 2205 IP packets with a maximum size of 1500 bytes which results in an average rate of ≈ 27.5 packets per second. Traffic The video sources are randomly selected among the 16 mobile nodes. In contrast to the sources, the sinks are chosen in a point symmetric way among the fix nodes at the

46


1

1

0.95

0.9

End−to−End Reliability

0.9 Probability P(T