The Implementation of Cloud Storage System Based on OpenStack Swift Duan Zhiying1, a, Cao Yizhen2, b 1
School of Computer, Communication University of China, Beijing, 100024, China
2
School of Computer, Communication University of China, Beijing, 100024, China a
[email protected],
[email protected]
Keywords: OpenStack Object Storage; Swift; Swauth; S3; Cyberduck
Abstract. This paper describes the implementation of a fast and efficient method to build a cloud storage environment. In this method, we base on the OpenStack Object Storage (code-named swift), use Swauth as the authentication system, and build a S3-compatible cloud storage platform. Then do a second development of open source software Cyberduck and use it as the cloud storage Client. Introduction The term "cloud" originates from the world of telecommunications when providers began using virtual private network services for data communications [1]. As smart portable devices such as: IPad, IPhone, Android Tablet, etc. are widely used, and the amount of data, especially unstructured data (such as images, audio, video, etc.) is growing sharply, traditional storage technologies are confronted with series of problems, such as high costs, complex operation and maintenance, and limited scalability. In the context, cloud storage technology is gaining more and more attention. Cloud storage is the storage part of cloud computing, and it allows users to store their data and access them anytime anywhere via Internet [2]. Several years earlier, we were advocating the use of FTP service to manage documents and files to solve safety problems of traditional file sharing [3]. Now we are exploring to use object-based storage to replace file storage and block storage. Flexible to expand capacity, safe, reliable and low cost data storage are now the focus of enterprises. Overview of OpenStack Object Storage OpenStack is a cloud computing platform developed by Rackspace and NASA, aims to implement a cloud infrastructure-as-a-service (IaaS), similar to Amazon EC2 and S3, for service providers and enterprises. What makes it different is that OpenStack is a free software and has open source projects. Core components of Swift are as follows: Proxy servers: the public interface of Object Storage, used to handle all of the incoming API requests. Rings: map logical names of data to their physical locations with consisting hashing algorithm. Zones: isolate failure boundaries. Each data replica resides in a separate zone. Accounts, containers and objects: basic organization units of data storage. Partitions: a collection of stored data, including account databases, container databases and objects. Realization of the Cloud Storage System A. Build OpenStack Swift Environment Ubuntu 12.04.3 (32 bit) is used as the experimental operating system. Ubuntu is the world’s most popular operating system for OpenStack. The release schedules of Ubuntu and OpenStack are synchronized. To carry out the experiment, we do Swift development on a virtual machine, run all Swift service
on a single server and emulate 4-nodes swift cluster. We build the developing environment according to Swift’s developer documentation [4]. Then, set up scripts for running Swift, use a simple command like “startmain” to start Swift daemon processes, finally we will get the results shown in figure 1. root@dzy-virtual-machine:~# startmain Starting proxy-server...(/etc/swift/proxy-server.conf) Starting container-server...(/etc/swift/container-server/1.conf) Starting container-server...(/etc/swift/container-server/2.conf) Starting container-server...(/etc/swift/container-server/3.conf) Starting container-server...(/etc/swift/container-server/4.conf) Starting account-server...(/etc/swift/account-server/1.conf) Starting account-server...(/etc/swift/account-server/2.conf) Starting account-server...(/etc/swift/account-server/3.conf) Starting account-server...(/etc/swift/account-server/4.conf) Starting object-server...(/etc/swift/object-server/1.conf) Starting object-server...(/etc/swift/object-server/2.conf) Starting object-server...(/etc/swift/object-server/3.conf) Starting object-server...(/etc/swift/object-server/4.conf)
.
└── sdb1 ├── 1 │ └── node │ └── sdb1 │ ├── accounts │ ├── async_pending │ ├── containers │ ├── objects │ ├── quarantined │ └── tmp ├── 2 ├── 3 └── 4
Figure 2. Folder Structure Figure 1. Start the Swift Daemon Processes We mount the partition to the directory “/mnt/sdb1”, so there are four nodes in this directory. The folders, “/mnt/sdb1/1”, “/mnt/sdb1/2”, “/mnt/sdb1/3” and “/mnt/sdb1/4”, have the same structure, which is shown in figure 2. Each account and container is an individual SQLite database distributed across the cluster. Here, the account and container database is separately stored, by default, three times, in the accounts and containers directory in different nodes. The table “account_stat” in account database holds the information about accounts, and its schema is shown in Table1. Likewise, Table 2 shows the schema of table “container_stat” in container database. Unlike accounts and containers, there is a “.data” file in “objects” folder which stores the object metadata. Table 1. account_stat Table 2. container_stat Name
Type
Name
Type
account created_at put_timestamp delete_timestamp container_count object_count bytes_used hash id status status_changed_at metadata
TEXT TEXT TEXT TEXT INTEGER INTEGER INTEGER TEXT TEXT TEXT TEXT TEXT
account container created_at put_timestamp delete_timestamp object_count bytes_used reported_put_timestamp reported_delete_timestamp reported_object_count reported_bytes_used hash id status status_changed_at metadata
TEXT TEXT TEXT TEXT TEXT INTEGER INTEGER TEXT TEXT INTEGER INTEGER TEXT TEXT TEXT TEXT TEXT
B. Authentication System – Swauth Before granting access to the requested resources, the control process invokes the authentication system to validate user’s identity. The authentication part can be an external system or a subsystem running within swift as WSGI middleware. So enterprises can create their own authentication subsystem with proxy server middleware. The following three authentication methods are mostly used: TempAuth, Swauth and Keystone. TempAuth is Swift’s default authentication. TempAuth stores username and password in plain text format. By default, each user in the system possesses read access to the file where super user credential is located [5]. So, TempAuth is generally not considered in the production deployment. In this experimental environment, we use Swauth as the authentication system. Unlike tempAuth, Swauth possesses properly configured access rights to secure password data. However, Keystone is
widely used in the real developing environment [6] [7]. Keystone is the identity management service of OpenStack, with a better and more secure way of storing passwords in database. C. Amazon Simple Storage Service Amazon S3 is a public service, it provides storage through web services interfaces like REST [8]. The broad application of S3 has given rise to competing services based on the S3 API. Following we describe the method to support S3 API in OpenStack Swift. We just need to install Swift3 middleware from the website “https://github.com/fujita/swift3”, and then modify the file “proxy-server.conf ”. Thus we can access to Swift via Amazon S3 API. In this system, we use a graphical interface to access to Swift. You can use other programming languages to operate S3 API. For example, if you use Python, you need to install boto, the Python package of AWS. D. Client – Cyberduck Swift can be connected through REST API or a client supporting object storage API. Here, we use Cyberduck, a popular open source storage browser for several cloud storage platforms. It works with third-party providers using either the S3 or OpenStack Swift APIs. For Cyberduck is an open source software, second development is possible. The latest version of Cyberduck is 4.4.5, which we can build in Windows and Mac. Here we mainly talk about Windows operating system, with Microsoft Windows SDK and .NET Framework 4.0 installed. To build Cyberduck, following steps are needed: a) Install the latest Java SDK and Apache Ant, and remember to set environment variables. b) Download the IKVM binary and modify “ikvm.home” in “tools.properties” from Cyberduck installation directory. c) Download and install Subversion, find the “build-windows.xml” in the Cyberduck directory and modify “svn.revision” in it. d) Install “MSBuild Community Tasks” and “Bonjour SDK for Windows”. After building Cyberduck, we can open and debug it in Microsoft Visual Studio 2013. Cyberduck is a Windows Form program, whose underlying is mainly composed of a Java implementation. Generally, we need to add users before logging in OpenStack Swift, which can be achieved with cURL, a command line tool supporting a range of common internet protocols including HTTP. To implement users’ self register, we access and modify the Swift database. We provide three more functions: username existence check, user register and password changing through web service. And then, add Register and ChangePassword forms in Cyberduck shown in figure 4, which makes the whole system more fulfilled.
Figure 3. Cyberduck Login Interface
Figure 4. Added Forms
Figure 3 shows the Cyberduck login interface, we can use either S3 or Swift to access the cloud system we built. Summary and Future Work This paper proposes an effective method to build a cloud storage system based on OpenStack Swift. Detailed implementation steps were given from four aspects: building Swift environment, choosing authentication system, supporting Amazon S3 API and second development of Cyberduck as the client. The final results indicate the system available. The above Swift development is conducted in an experimental environment of a single computer. Actually, Swift is usually deployed in clusters in real practices, where needs more safety and reliability. In my future research more focus will be given on that. Acknowledgement This paper is supported by “The technology research and application demonstration of integrated network music production and broadcasting” (2013BAH66F02) and “The technical research on digital production and broadcasting platform” (3132013XNG1316). References [1] Kaufman L.M. Data Security in the World of Cloud Computing [J]. Security & Privacy, IEEE, 2009, 7 (4), 61-64. [2] J.Y. Wu, L.D. Ping. Cloud Storage as the Infrastructure of Cloud Computing [C]. 2010 International Conference on Intelligent Computing and Cognitive Informatics, 2010. [3] Z.Y. Duan, Y.Z. Cao. Serv-U based Audio Data Transmitting Management System on Broadcasting Stations [C]. International Conference on Electrical Insulating Materials and Electrical Engineering, 2013. [4] Information on http://docs.OpenStack.org/developer/swift/development_saio.html [5] Primoz Cigoj. Security Aspects of OpenStack [D]. Slovenija: Mednarodna Podiplomska Sola Jozefa Stefana, 2012. [6] Sitaram Dinkar, H.L. Phalachandra. Keystone Federated Security [C]. International Conference for Internet Technology and Secured Transactions, 2013. [7] David W. Chadwick, Kristy Siu. Adding Federated Identity Management to OpenStack [J]. Journal of Grid Computing, 2014, 12 (1), 3-27. [8] Roy Tomas Fielding. Architectual Styles and the Design of Network-based Software Architectures [D]. America: University of California, Irvine, 2000.
