Fault-Tolerant Middleware Based on Multistream

The 7th International Conference for Internet Technology and Secured Transactions (ICITST-2012)

Fault-Tolerant Middleware Based on Multistream Pipeline for Private Storage Services J. L. Gonzalez, Victor Sosa-Sosa Cd. Valles Institute of Technology, Information Technology Laboratory Center of Research CINVESTAV Ciudad Victoria, Mexico

Abstract- This paper presents a fault-tolerant middleware for private storage services based on a client-server model. A c1ient

side

API

split

files

encoded/decoded,

into

redundant

anonymized,

and

chunks,

which

distributed

to

are

different

storage locations in parallel by using multistream pipeline. A chunk placement engine on the server-side API receives/delivers chunks from/to the client side API. In this model, the user preserves

her

files

in-house

and

achieves

data

availability

anytime from anywhere even in site failure scenarios while the organizations preserves the control of their storage resources. The

experimental

evaluation

reveals

the

middleware

can

compensate the overhead produced on the client side by taking advantage of multiple cores commonly found in user's devices. The

users

observe

that

the

middleware

produces

similar

performance to the private and public file hosting systems tested in

the

evaluation.

The

organizations

observe

a

significant

reduction of work in their servers. In site disaster scenarios, the middleware yields even better performance than fault-tolerant online distributed solutions tested in the evaluation.

Keywords- Optimization; Storage Services; Privacy; Fault tolerance; Virtualization;

I.

INTRODUCTION

The volume of data that is produced, stored, exchanged, and accessed on the Web has increased dramatically[l], by year 2013, predictions place the percentage of the content delivered digitally to more than 50% of all content [2]. This has led to increased interest on online storage systems [3]. Storage services such as content synchronization among different devices[4] (e.g. tablets, phones, computers), online backup[5] and File Hosting Service (FHS) to remote file access via web dashboards[6] have become a common solution for the large majority of users - individuals or organizations that need to store or retrieve their files any time from anywhere. There are, however, concerns with this kind of solutions because both user and organizations have no control and no assurance on the procedures related to data management [7]. In addition, there is also a strong concern about the information vulnerability, in case that the storage provider were compromised [8],[9],[10]. The problem is not to store files by using a public provider but also to store them without applying fault-tolerant and data assurance strategies to the files on the client-side. This is quite important because a user rejects her legitimate expectation to

978-1-908320-08/7/$25.00©2012 IEEE

Borja Bergua, Luis Miguel Sanchez, Jesus Carretero Computer Architecture and Technology Area, Computer Science. Department UNIVERSLDAD CARLOS III Madrid, Spain

privacy when she voluntarily relegating private content to a third-party (the public provider) [11],[12]. This fact not only raises issues of de-anonimyzing personal records [13] mining private information [14], and in particular meta-data[15] but also makes roles and responsibilities unclear for an error, or rending a service unavailable[16]. A naive solution is to encrypt the files before sending them to the provider [17],[18]. Nevertheless, this approach not solving the problem of service unavailability [16],[9]. In addition, regulations in different countries require organizations to guarantee the integrity and confidentiality of their files during large periods of time such as six years in the case of medical contents [19] or three for financial information [20]. This is enough time for the data volume to grow exponentially and increase the costs of outsourcing storage services in the long term. On the one hand, organizations are currently considering the possibility to build their own in-house storage capacities as a promising alternative to keep in control of this valuable resource [21]. On the other hand, studies show that users prefer local storage solutions for managing sensitive data [22],[23], and favor one time costs for personal storage rather than monthly charges[2]. We propose a fault-tolerant middleware for the data dispersion over private storage services. The middleware is based on a client-server model in which the fault tolerant and data assurance strategies are applied to the files on the client-side while the storage resource administration is performed on the server-side. The contributions of this work are as follows: • Multi-pipeline API: A client-side API has been implemented to split files into redundant portions called chunks by using the Information Dispersal Algorithm or IDA [24]. We have designed and implemented a multithreading version of the IDA algorithm, in which the Encode/decode processing is performed in parallel by taking advantage of multiple cores commonly found in the user's devices. The API produces HTTP multi-streams for sending/retrieving anonymized chunks in pipeline to/from private cloud storage locations. This API can recover files even when some of these locations are unavailable.

548


Cliem-Side

•

Chunk Placement Engine: A server-side API has been proposed to deliver/receive chunks to/from the pipeline processing created by the client-side APT. This engine creates one temporal path or relative URL for each chunk stream produced by the client-side APT. Once the chunks have been read or written, the engine moves them from the relative URLs to secure disk partitions in the organization servers. To complete the storage process, the engine registers the locations assigned to the chunks in a metadata system. This allows the client-side API to perform GET operations of these chunks. In this solution, the user is managing files while the organization is only managing anonymized chunks or encoded portions of files. We have implemented the middleware in an online storage service prototype. The performance of that prototype was compared with fault-tolerant distributed web storage as well as public and private File Hosting Services (FHS in the rest of the paper). The experimental evaluation reveals the middleware can compensate the overhead produced on the client-side by taking advantage of multiple cores commonly found in user's devices. The users observe the middleware produces similar performance to private and public hosting file systems. The organizations observe a significant reduction of the work of their servers. In site disaster scenarios, the middleware yields even better performance than fault-tolerant online distributed solutions. II.

RELATED

WORK

Before you Web storage service is a popular approach to perform incremental online backup of files in which the user sends a copy of a given file to the cloud storage[25],[26] or file hosting system (FHS)[4],[25] by using any of the web browsers, synchronizer based on HTTP streams or (S)FTP applications. Tn this approach, the user/organization is outsourcing the storage service by following a pay-as-you-go pricing mode, which running costs down during idle periods. However, they are also outsourcing the control and assurance of the data management [7]. The private distributed systems are cost-effectiveness storage approaches commonly chosen by organizations to preserve the control and assurance of their information [27],[21]. Tn order to avoid the user to suffer from system failures, in this kind of approaches the servers split the files into chunks [28] by applying a given codification [29] and they distributes them according a given fault-tolerant strategy. Nevertheless, the users and the organizations still sending the whole files to the storage services, which could produce concerns about the way privacy data is managed. Tn addition, the encoding/decoding data and its distribution both represent an extra work to be performed by the organizations servers. P2P storage backup is a traditional approach in which the users send a given file to a shared storage. In this method, some applications even allow the users to send either files [30] or chunks [31]. The user must share her hard disk with a set of (un)known users when using this kind of applications.

97B-1-90B320-0B/7/$25.00©2012 IEEE

Client-Side

�

, Vulneravility Win

!

Figure 1. Traditional Online Incremental Backup Approach

However, this seems not to be a feasible option for current Organizations [2]. Tn addition, our work is focused on scenarios in which organizations are expressed its willingness to provide their users with both private infrastructure and platform storage [21]. The distribution of data on several providers have been proposed to avoiding cloud vendor lock-in[32],[33]. Nevertheless, this kind of solutions are only available for public providers and solutions taking both the user and organization concerns into account are currently required by organizations. This is the case of the storage middleware proposed in this work. Ill.

FAULT-TOLERANT MLDDLEWARE FOR PRIVATE STORAGE SERVICES

Tn this section, we describe the traditional online storage and our Fault-Tolerant middleware for Private Storage Services. A.

Online Storage Overview

The traditional approach for online storage is depicted in Fig. I, which shows the work flow applied when a given user is storing (left) and retrieving (right) the file IFI. Fig. I also shows how a user sends a replica (lFil) of the file IFI to the cloud storage by using a HTTP stream produced on the client-side. Once the provider has received the replica, it split IFI into redundant chunks C{1..5) (Creating chunks stage) that are distributed to different places in the provider storage network (Distributing chunks stage). This is represents a backup used by the provider to recover IFil when the location in which it was stored is not available. The recovery procedure is quite simple, the provider reconstructs lFil (Reconstructing IFil stage) by using the chunks C{1..5} (Retrieving chunks stage). Once IFil has been recovered, it is returned to the user as IFI (Locating stage). Unless the costumer had paid for strong redundancy, there is a vulnerability window during the time in which the provider decides to perform the backup of this file. Tn this vulnerability window any failure could represent lost data [34] (See Fig. I). Tn addition, when the user delivers IFI to a third party, she could not expect privacy. When IFI has been encrypted in advance, the above risk is reduced but the user can not access either IFil or its chunks (C{1..5}) when the provider service is not available.

549


Client-Side

Client-Side CII cn" .. - 0) (1),-: ,,"0'" .c

s·c .0) en > "3

8

""0) �cr

FiQure 2. Work flow of the Fault- Tolerant Middleware based on Multistream Pipeline.

B.

Fault- Tolerant Pipelines

Middleware

Based

on

Multistream

We propose a fault-tolerant middleware for the data dispersion over private storage services in which the stages of the traditional online storage are based on continuous pipelines. This middleware is depicted in Fig. 2, which shows a multipipeline API on the client-side and a chunk placement engine on the server-side. Fig. 2 also shows the work flow when a user storing (left) and retrieving (right) the file IFI by using an online private storage service. As we can see, the middleware conducts the online storage process as a pipeline including four stages: the acquisition stage in which the API reads the file IFI from the local hard disk or partition, the transformation stage in which the API split IFI into n anonymized chunks, the transportation stage in which the API creates n parallel HTTP(s) streams to write/read the chunks inlfrom the cloud storage locations and the storage stage in which the chunk engine determines the places to GETIPUT the chunks being delivered/received by the streams. The stages are detailed in the following sections: I) Data Acquisition Stage: This phase is carried out sequentially reading the data via either cloud storage locations or a disk partition by using the Inputfilter class Data Acquisition Stage: This phase is carried out sequentially reading the data via either cloud storage locations or a disk partition by using the Inputfilter class. When a user performing a PUT operation. The Inputfilter class reads the file IFI stored in the user storage device and handles the messages generated with the input data to be processed and stored in next phases. In the case of GET operations, the Inputfilter class sends a request for the file IFI to the next stages. Once the file IFI has been returned by the pipeline to the Inputfilter class, it writes the file IFI in the hard disk on the client-side. 2) Transformation Stage: In this phase, the multi-pipeline API applies the fault-tolerant strategy to the files by using Information Dispersal strategy or IDA [24]. Information Dispersal Algorithm: This algorithm is part of the Reed-Solomon family of error-correcting codes and it is commonly used to provide information integrity.

97B-1-90B320-0B/7/$25.00©2012 IEEE

TABLE I.

n Server n=2

IDA PARAMETERS COMBINATION

m (Dispersals required for recovering files) m=3 m=4 m=5 m=t m=2

m=6

100%

n=3

200%

50%

n=4

300%

100%

33.3%

n=5

400%

150%

66.70%

25%

n=6

500%

200%

100%

50%

20%

n=7

600%

250%

133.3%

75%

40%

16.7%

When a user performing a PUT operation, the algorithm split the file IFI into n redundant chunks (C{1..5}) as shown in Fig. 2. The redundant chunks are distributed to n different storage locations to guarantee the effectiveness of the fault-tolerant strategy. When a user performs a GET operation, the algorithm requires m minimum number of available storage locations to recover IFI. It is granted that the original file can be reconstructed by multi-pipeline APlwhen n > m. The IDA codification of a given file comes at a price because the size of each resultant chunk is IFllm, which means a percentage excess of redundancy equal to (n- m)/m. Let us consider an IDA implementation with parameters (n=5;m=3), in this case the file IFI is transformed into five chunks and it can be reconstructed by retrieving at least three chunks from three different locations. This means the combination of parameters (n,m) tolerates up to (n -m) unavailable locations and the system spends an excess of 66.7% on redundancy overhead. Table I shows the amount of extra capacity spent for n servers when requiring m chunks (at least) to support fault tolerance. IDA Optimization with Threading Building Blocks: We designed and implemented the Transformfilter class to perform IDA encode/decode processing by using techniques of thread level parallelism. This is done by using the Intel Threading Building Blocks library (TBB) [35]. The TBB library allows the Transfornifilter class to create chunks in a standard parallel pattern pipeline [36].

550


I w.ant my FUll! 1"1 bad;

��

allocate/locating chunks in the cloud storage. In this metadata scheme, one web storage server is appointed to be metadata controller while the rest of servers are considered as storage nodes if any. TABLE II.

.•. -

THE FEATURES OF THE STORAGE ASSETS OF TWO ORGANIZATIONS

. GET Streaning

--.PUT Streaming

Figure 3. Information Assurance Policy applied to Federated

Private Clouds

This pattern creates an application by linking tasks (I/O and compute) coordinated with each other, which will get exploit the parallelism of T/O and computing, as well as better manage computing resources at all times trying to use the full potential of processors [37], [38]. The communication between the Transfornifilter processes arranged in the pipeline is done through message passing. The implementation of Transformfilter allows the multi pipeline API to defme the amount of cores to be used in this stage by including the input parameter i. When I parameter is not included, the Transformfilter class assumes all cores found in the user device can be used in the transformation stage. 3) Transportation Stage: A class called Outputfilter is in charge to collect all messages processed in the previous phases as well as to store the output data in the transportation stage to transport the chunks gradually created by the Transformfilter class by using HTTP streams. This avoids the multi-pipeline APT to write chunks in local hard disk, which also reduces the time spent in the whole process. In this phase, the data assurance strategy is applied to the files by deciding the amount of chunks that the multi-pipeline API must store as well as the enough storage locations for guaranteeing that only the user is able to reconstruct her files. The number of streams is n when a user performing a PUT operation while this number is m for GET operations. The user can choose these parameters to establish a given information assurance strategy. Fig. 3 depicts the distribution of chunks of a n=5,m=3 IDA configuration, in which an information assurance policy has been applied to the transportation of the chunks Stage. In this policy the user establishes that only she can reconstruct her files. The middleware then distributes two chunks to the cloud storage of organizations Org I and Org2 while one chunk is distributed to organization Org3. As a result, the files cannot be reconstructed on the server-side because m = 3 chunks are required for that procedure. Fig. 3 also shows how the user can retrieve her file (right) even when cloud storage Orgl is not available. We have included HTTP transfer library in OutputFilter class abased on a multi-protocol file transfer library called libcurl [39]. 4) Chunk Storage Stage: The placement engine has been designed to be installed in one or a set of web servers. This engine includes a reduced metadata scheme to

97B-1-90B320-0B/7/$25.00©2012 IEEE

UC3M-Cloud

Region and City Number and Type of Nodes

4

Madrid-Spain Leganes Cloud Instances

Operating System

OpenStack

Virtualized Capacity t

HB

Network

GigabitEthernet

UC3M-Colme Madrid-Spain Colmenarejo 2 15 and 1 17 Ubuntu

HB GigabitEthernet

The function of the metadata controller is to generate as many relative URLs as chunks generated/requested by the multipipeline API. The web server in charge to attend a given request must contact the metadata controller for validating it. Once the request has been authorized by the controller, the server response with a ServiceKey string, which is an array of relative URLs that guarantees the effectiveness of the IDA configuration. The relative URLs are constructed by using random names and they can be accessed one single time. When the chunk arrives to its corresponding relative URL, the web server moves it to a directory in a secure disk partition. The final step of this procedure is to delete the URL. TV.

FAUL T-TOLERANT MIDDLEWARE PROTOTYPE

Tn this section, we describe a prototype of online storage based on FHS (file hosting system), in which we have implemented the fault-tolerant middleware. We consider a scenario where two organizations placed in different cities require the building of an in-house online storage service. This is a quite common scenario when each participant is either a business unit of a larger enterprise. A.

Storage Infrastructure

Each organization that takes part of this shared storage service has preserved an in-house infrastructure, which includes a set of storage nodes or one of them at least. A storage node is either a private cloud computing instance or a physical PC, which has been configured to serve I/O operations such as PUT, GET, DELETE and LIST any time from anywhere. Each storage node is in charge of a set of partitions, which are used for storing the files sent by the users. The organizations are represented by the following acronyms UC3M-Colme and UC3M-Cloud. Table II shows the features of storage infrastructure shared by each organization. B.

Storage Platform

FHS that enables users to upload/download files, create work groups, add users to their work groups and share their files to other users or work groups.

551


TABLEll CPU ARCHITECTURE TESTBED CPU Specification

Sockets

Cores per Socket

Hyperthreading

Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz Intel(R) Xeon(R) CPU E7-4807 @ 1.87GHz

1

4

Yes Yes

FHS has been configured to gather and coordinate the storage capacities of the two organizations. This service is configured as a hot standby scheme in which each organization cooperates to withstand the site disaster of other one. The service is accessed by users of both organizations as single domain. Based on this unified interface, any user of any organization is supported even when the servers of one organization are not available. Tn the FHS we have included both an access control scheme to protect against any of the unauthorized viewing, modification, or copying of data and a multi-tenancy system to isolate the users of a given organization from other users and organizations. C. Multi-Pipeline Data Dispersion Service The users were provided with and APT of folder synchronization that is based on multi-thread Java applet. Each time the user either stores or retrieves files, the synchronizer runs the Multipipeline API to PUT/GET chunks to/from the FHS. Fault- Tolerant Strategy: The Multi-pipeline API was configured to apply a IDA storage scheme to the files, which is based on (n = 5;m = 3) parameters. information Assurance Policy: The chunks are distributed according to an information assurance strategy in which the user keeps one chunk in her device and two of them are distributed to each organization. This means the user achieves data availability even in a blackout of one organization site or when her tablet or notebook is crashed. Tn this type of policy, the client-side only requires to retrieve two chunks to reconstruct a file and the organizations have not enough number of chunks to reconstruct it. Storage Resource Management: The chunk placement engine was installed in each storage node and each organization has chosen one node to be automatically configured as metadata controller. This controller only accepts queries from the IPs of UC3M-Colme and UC3M-Cloud nodes and it only delivers relative URLs to Multi-pipeline APis by using public keys. In order to easily adapt the fault-tolerant middleware either to File Hosting Services or storage platforms, the multipipeline and the chunk placement engine have both been implemented as RESTful APTs. v.

DESTGN OF EXPERIMENTS

We implemented an 1/0 load injection API called 10 Launcher to test the storage services by producing PUT/GET operations as well as user credentials and sending them to the FHS. We have designed a workload scenario to test the middleware prototype in which TO Launcher sends an incremental load by duplicating the file size from 512KB to 1 GB. FHS assumes this artificial load come from real and valid users.

97B-1-90B320-0B/7/$25.00©2012 IEEE

6

4

A.

Memory (GB ) 8

128

Configurations to Measure the Codification Performance.

To evaluate the feasibility of the middleware, a preliminary assessment has been conducted by using files arranged in a storage device. The following configurations were tested for determining the influence of the device availability on the performance observed on the client-side. Serial: This configuration represents the implementation of the traditional IDA algorithm. Tbb: This configuration represents the IDA Optimization in pipeline by using threading building blocks. It has been taken into account two different situations: cool mode in which cache is not used (tbb) and hot mode in which input data is allocated in cache (tbb-cache). Table TIT shows the CPU features of all the computer architectures included m the evaluation of above configurations. Both architectures represent two different models: UMA (i7) and NUMA (Xeon). B.

Configurations to Measure the Codification Performance.

The following configurations were tested for showing the performance observed in the online storage service: • Phoenix: Tn this configuration, the user stores and retrieves files by using an Online Distributed Web Storage System called Phoenix[27]. We implemented Phoenix in the UC3M-Colme and UC3M-Cloud organizations and it has been configured to apply a fault tolerant strategy based on IDA storage schemes to the files. • Private FHS: In this configuration, the user stores and retrieves files by using a Private Web File Hosting System. We have implemented a Private FHS in UC3MCoime organization in which the files are stored without producing and distributing redundancy. • Amazon: Tn this configuration, the user stores and retrieves files by using an AWS Amazon [6] by using the standard redundancy associated to a free account. • Multi-Pipeline: We implemented the fault-tolerant middleware in the UC3M-Colme and UC3M-Cloud organizations. This configuration allows us to measure all the stages of the pipeline produced by the middleware. In this scenario, only Phoenix and Multi-Pipeline configurations can withstand the failure of a whole site. C.

Metrics

The following measurements were obtained when testing the configurations: • Codification Time: This represents the times obtained either in the generation of redundant chunks or in the reconstruction of files.

552


Amazon(Standard Redundancy) --+- Effort RedUndanCY ---X--Phoenix(IDA Redundancy) ... .. . IJ Multi-Pipeline(IDA

Private FHS(Best

200 U Q) .!!!.. Q)

E i=

Q)