Ubiquity: Extensible Persistence as a Service for

Download PDF
4 downloads 0 Views 539KB Size Report
Comment
Keywords: Persistence, CloudFoundry, Kubernetes, Docker ... alization mechanism where the application running inside the container shares the kernel with the .... As shown in Figure 1, Ubiquity is made up of different loosely coupled com-.
Ubiquity: Extensible Persistence as a Service for Heterogeneous Container-based Frameworks Mohamed Mohamed1 , Amit Warke1 , Dean Hildebrand1 , Robert Engel1 , Heiko Ludwig1 and Nagapramod Mandagere2 1

Almaden Research Center, IBM Research, San Jose, CA, USA mmohamed,hludwig,aswarke,dhildeb,[email protected] 2 Commulytics Inc, CA, USA [email protected]

Abstract. Over the last few years, micro-services are becoming the dominant architectural approach in designing software applications environments. This new paradigm is based on developing small, independently deployable services communicating using lightweight mechanisms and being consumed by other services or end user applications. Containers have proven to be a convenient runtime platform with container frameworks enabling elastic horizontal scaling if container workloads are designed as so-called 12 factor applications, foregoing file or block storage use for persistence. Persistence services such as databases are typically deployed in traditional runtime platforms such as virtual machines or bare metal servers. To use container frameworks to manage stateful services in a scalable way is not well supported in terms of manageability, security and performance, in particular in scenarios where state is accessed from workloads in different deployment platforms. In this paper, we present the Ubiquity framework, which provides seamless access to persistent storage across different container orchestrators (CloudFoundry, Openshift, Kubernetes, Docker and Mesos). Ubiquity is extensible to other container frameworks and different types of file and block storage systems, which can be managed independently of the container orchestrator. Ubiquity makes it easy to onboard stateful services in containerized and hybrid environments, extending the efficiency gains of containerization. Keywords: Persistence, CloudFoundry, Kubernetes, Docker

1

Introduction

In the last few years micro-services became the new trend for designing software applications. This paradigm consists of developing software applications as a set of independent components that focus on small functionalities, can be deployed separately, and use some lightweight communication mechanism such as REST, gRPC, etc. This trend changed how IT system developers perceive the software life-cycle, from the design and going forward to the testing, deployment and release management. These different phases of the software life-cycle have shrunk

2

M. Mohamed et al.

from months to weeks, days or hours giving more opportunities to innovate. The designers, architects and developers can now focus on smaller code bases and make the changes faster. They can easily isolate any fault that might occur in the overall system, fix it and redeploy only the affected part. Another trend that goes hand to hand with micro-services adoption is the usage of containers. Containers are a lightweight operating system level virtualization mechanism where the application running inside the container shares the kernel with the host operating system but has its own root file system [1]. Containers consume less resources than operating system based on hardware virtualization [16]. They exploit the cgroup concept in order to manage resources quota (CPU, Memory, Storage and Network) and offer the needed isolation to run the application without interfering with other containers. Many platforms and orchestration systems are based on container concepts to offer an agile way of building micro-service based software such as Cloudfoundry, Docker, Kubernetes, Mesos, etc. While using these container orchestrators (COs) is beneficial for large scale deployments, there is still a lively discussion as to which type of applications they might be best suited for. Most of these COs favor stateless micro-services due to the challenges of managing state in concurrency situations. Onboarding stateful applications into these COs in a scalable way is not well supported in terms of manageability, security and performance, in particular in scenarios where state is accessed from workloads in different deployment platforms. In these scenarios, even though we can have the same persistence backend used to maintain the state, the frameworks that are being used have different ways of consuming persistence resources. These resources are offered to the platform as volumes or persistent volumes that are specific to each CO. For example, CloudFoundry uses persistent volumes through its persistence drivers and service brokers, whereas docker uses docker plugins to create and manage these volumes. Kubernetes has a third way of doing it by having a dynamic provisioner responsible of the creation and deletion of volumes and a volume plugin responsible of the other consumption functions (e.g., attach/detach, mount/unmount). One other challenge in this context, is adding new storage backends to offer their resources as persistent volumes to be consumed by containers within these different COs. In order to do that, the storage provider needs to understand the details of every CO and how it consumes persistent storage as well as the different deployment models that it supports. This is a time consuming task for each CO that comes with adding all the management facilities for security, multi-tenancy, high-availability, etc... In this paper, we propose the Ubiquity framework that enables seamless access to storage for heterogeneous COs. Ubiquity is agnostic to the CO since it has the needed mechanisms to integrate with the widely used orchestrators (CloudFoundry, Kubernetes, Docker, Mesos, Openshift). Moreover, the design of Ubiquity makes it easy to be extended to support new COs while maintaining all the advantages that it already offers (Security, High Availability, Scalability). It also offers an easy way of adding new storage backends without the need to understand the specificities of the COs that are supported. Ubiquity supports

Ubiquity

3

also different deployment models to accommodate the different users preferences. Our close collaboration with the leaders of most of the COs storage teams allowed us to continuously adapt Ubiquity to the newly supported functionalities in an agile way. Hereafter, we will detail how Ubiquity can be integrated and used with the different COs and how a new storage backend can be added easily. The remainder of this paper is structured as follows: in Section 2 we give some motivating examples and detail the different problems that we address. In Section 3, we present Ubiquity and show how it solves these problems. Subsequently, we present the implementation details of Ubiquity in Section 4. Then, we give an overview of the state of the art in Section 5. Finally, we conclude the paper and give some future directions in Section 6.

2

Motivation and Problem description

The motivating applications that drove the adoption of micro-services had few storage requirements. For example, Web and application servers, load balancers, and monitoring services are mostly stateless and the little state they do retain can typically be lost without any major consequence. Recently though, the COs that enabled micro-services are being used for a much broader set of applications, many of which have extensive persistent storage requirements. Key-value stores, SQL and NoSQL databases, OLTP applications, machine and deep learning analytic applications, and just about every other enterprise application is now being proposed to run inside CO environments. This demand for storage has exposed a gap in the COs that desire to run these applications. With the rise of CO in enterprise data centers, there is a desire to shift day-to-day storage management over to the CO administrator. This avoids the need to interact with the storage administrator for each and every application deployment. But there is an ever expanding set of on-premise and cloud-based file and block storage solutions, each with their own set of unique features (e.g., encryption, storage tiering, DR). It is impractical for every CO to support each and every storage system (and all of their unique features). Other similar cloud middleware frameworks such as OpenStack simply limited the set of supported features through standardized APIs (e.g., Cinder), but this presented problems for storage vendors by limiting their ability to provide value. COs chose instead to allow storage vendors to expose their features directly to the CO administrator. This additional flexibility comes at the price of additional complexity. Given that each CO has a different set of APIs and storage features (e.g., Kubernetes is the only CO that supports storage lifecycle management), the burden is on the storage vendor to integrate into each CO. No middleware exists to glue together the diverse set of CO and storage systems. Ubiquity resolves this complexity by providing a pluggable framework that implements each of the CO APIs and provides a simple API that allows any storage vendor to integrate without loss of differentiation. Given that most enterprise data centers use a variety of storage systems, and increasingly the use of more than one CO, Ubiquity greatly simplifies adoption CO in diverse environments.

4

M. Mohamed et al.

When a CO uses Ubiquity to access file or block-based storage systems, an applications dataset can now be referred to by its Volume Name (VN). Developers can simply refer to the VN in all uses of their application, avoiding the need for remembering complex block volume ids or long file paths.

3

Ubiquity

In this section we will go into the details of Ubiquity framework. We will start with giving an overview of our framework in Section 3.1. Then, we will detail how Ubiquity is able to support the existing COs in Section 3.2. Afterwards, in Section 3.3 we will describe the different deployment models. Finally, we will talk about the future of the storage plugins in Section 3.4. 3.1

Overview of Ubiquity Framework

As discussed in Section 2, accessing persistent storage is a necessity for many scenarios. For the discussed scenarios and many others, the absence of persistent storage support in container frameworks is hindering the applications from strangling the monolith and moving to a more scalable and agile life-cycle based on containers. There are parallel efforts going on in order to add support for efficiently accessing persistent volumes and offering them to applications running inside containers. Meanwhile, the continuous change of the APIs used by the different COs makes it difficult to cope with their support for storage. More specifically, if the concerned applications need to span over different COs, this makes it really complicated to handle all the changes at once. Ubiquity framework allows to promote persistent storage usage within one or more COs without any need to understand the intrinsic details and mechanisms of the targeted CO. As shown in Figure 1, Ubiquity is made up of different loosely coupled components that are easy to extend or replace. These components are the following: – Ubiquity Volume Service: a scalable service that links COs to storage systems. – CO Plugins: a peace of software that allows a CO to offer a persistent volume to a container inside a host (i.e., a node). – Storage System: a module per provider that maps the CO requirements to storage resources. – HA Database Service: a database that maintains the state of the system and back it with high resiliency and availability. Ubiquity Volume Service: This service is the main component of Ubiquity playing the role of the mediator between the COs and the storage backends. It is offering southbound interfaces to be consumed by COs to allow them to create persistent volumes and manage them. The management operations are continuously evolving based on the functionalities supported by the COs. These operations include attaching/detaching volumes to nodes, setting quota of volumes, maintaining

Ubiquity

5

Fig. 1. Ubiquity Architecture Overview

the coherence of the attachment of volumes. These operations come with multitenancy support by maintaining the meta-data about the context of the creation of a volume and prohibiting the unauthorized access to data in volumes. Other operations are being progressively included like snapshotting and dynamic resizing of volumes. Ubiquity Volume Service is respecting the 12 factors application recommendations. It can be easily scaled up or down when needed to handle more queries. It supports also concurrent access by implementing a rigorous locking mechanism. It is noteworthy that this service is maintaining its state in a highly available database service to get more resiliency and high availability.

Container Orchestrators Plugins: Ubiquity framework has support for different COs. Most of these latter have a specific way of consuming storage via storage plugins. These plugins generally live in all the nodes managed by the CO and they allow to make host side operations to make the persistent volume ready to consumption by the containers. These operations include mounting and unmounting the volume to the container. In the context of containers, mounting means making the persistent volume visible in a specific location inside the container file system and ready to be used. In some cases, the mount operation requires other preparations before presenting the volume to the container. More specifically, if the persistent volume is based on a block device, there is a need to create a file system on top of that device in order to be able to mount it. If the persistent volume is based on a shared file system, then there is no need to do any further operations other then mounting. We will give more details on the

6

M. Mohamed et al.

supported COs in Section 3.2 where we describe the different ways COs use the plugins in order to create and manage persistent storage. Storage Backends: these are the mechanisms needed from the storage provider perspective to make their storage portfolio ready to COs.These components implement the needed mapping between Ubiquity API and the specific provider API to allow the creation and management of persistence storage. Since the APIs might change from one provider to another, the mapping is not always straightforward. In some cases, we needed to integrate one or two functionalities from the provider side in order to map one functionality in Ubiquity. In other cases, we made a functionality as a no operation one because it is not relevant in the context of a particular provider. Database Service: This component is playing a critical role in our architecture. In fact, it is critical for any 12 factor application to externalize its status to an external service to facilitate its scalability. Moreover, the Database Service is playing an essential role in our locking mechanism, since the locks are created within the database and accessible from the different instances of Ubiquity. Whenever we have a concurrent access to a resource, Ubiquity consults this service to check the status on the lock granted to the needed resource. This service is also our tool to accomplish high availability and consistency. When we have different instances of Ubiquity running with a load balancer in front of them, all instances check if there is at least one that is currently playing the role of the leader in processing the calls. If the leader is not active (i.e., the leader did not update his status for a given period in the database), a new leader is elected and takes over the leadership responsibility. 3.2

Supported Container Orchestrators

Ubiquity framework plays the role of a mediator between COs and storage providers. For instance, our framework have support for any CO based on the open source PaaS CloudFoundry (e.g., IBM Bluemix, Pivotal CloudFoundry, SAP Cloud Platform, etc.). It has also support for any solution based on Kubernetes (IBM Container Service, Google Container Engine, Openshift, etc.). Finally, Ubiquity integrates very well with solutions based on Docker API (Docker, Mesos, etc.). CloudFoundry (CF): This Platform as a Service (PaaS) is among the most successful PaaS offerings. Support for persistent storage was introduced to the platform late 2016. The persistent storage support is based on three major components namely the service broker, the volume driver and the Diego Rep volume manager (volman): – Service Broker: compliant with the Open Service Broker API [11]. It is responsible of creating persistent volumes based on the backend offered by the storage provider. It sends back a blob of information determining how to access the created volume.

Ubiquity

7

– Volume Driver: this driver is compliant with Docker Volume API [8]. It allows attaching volumes to the host and mounting them to the application container. – Volman: is the manager that consumes the blob of information sent back by the service broker and uses it to communicate with the volume driver in order to attach and mount volumes. The service broker and volume driver are the responsibility of the storage provider while volman is already embedded in the platform itself. Ubiquity has support for CF by implementing the docker API for the driver and by offering an implementation of the service broker with different storage backends (Spectrum Scale, OpenStack Manilla, Softlayer Storage). Kubernetes: It is ’an open-source system for automating deployment, scaling, and management of containerized applications’ [15]. This CO is one of the most popular open source COs with a huge base of contributors from all over the world. Kubernetes have support for persistent volumes using two main components: dynamic provisioner and volume plugins. – Dynamic Provisioner: it is a unique feature for kubernetes that allows storage volumes to be created on-demand [13]. This feature is based on defining storage classes to present different flavors or configurations of persistence storage. Each storage class needs to specify a dynamic provisioner responsible of creating and deleting persistent volumes. An end user can specify his requirements in term of persistent volumes in a so called Persistent Volume Claim (PVC). Whenever a new PVC is created, the dynamic provisioner will take it, create a persistent volume and bind it to it. – Volume Plugins: they are pieces of software needed to consume an existing persistent volume [14]. They offer attach/detach and mount/unmount functionalities in each node of the kubernetes cluster. These plugins can be in-tree (released with kubernetes base code) or out-of-tree (released as a separate code that can be used with kubernetes). These plugins work with the assumption that the persistent volume was already created (manually or through a dynamic provisioner). If an application requires a given type of persistent volumes, its associated volume plugin should be available on each and every host in order to be able to schedule containers on this host. Ubiquity offers a dynamic provisioner and an out-of-tree volume plugin that are agnostic to the provided storage backend. In order to consume any persistent storage for the list that we are supporting today (Spectrum Scale, Openstack Manilla and Softlayer Storage), one can specify the backend name in the storage class and Ubiquity will take care of the rest. Docker: Since 2013, Docker [7] became the de facto way of deploying applications on containers. It is an open platform for developers and sysadmins to build, ship, and run distributed applications, whether on laptops, data center VMs, or

8

M. Mohamed et al.

the cloud. Docker uses volume plugins [8] to integrate with external storage backends and allow containers to persist their data beyond the lifetime of a container. Any provider can integrate with docker by implementing a volume plugin compliant with the defined API that allows the creation of volumes and making them available to the container. Ubiquity comes with a docker plugin that is agnostic to the storage backend. A Docker user can create volumes using this plugin by specifying the backend that he desires. The plugin will just map the Docker calls to Ubiquity calls and get back the volume to be consumed. 3.3

Deployment Model

Ubiquity framework is made up of different components that could be deployed in different ways: Ubiquity Service: Since this service is stateless, we have different options to deploy it. The easiest way to do that and since we are working in container-based environments, is to run the service as a container. This option woks for all the COs as far as we have the right plugin and right endpoint of the container running ubiquity. The second option takes advantage of the concept of deployment proposed by Kubernetes where we can describe our deployment as a pod of containers and we can specify how many instances of that container should be running at any given time. Using this option, we can benefit from kubernetes strategy of offering high availability and fault tolerance. We can also scale up or down the number of running instances as much as we need. The third option is to run the code as a standalone binary application. Finally, we can run the service as a systemd script that starts with the startup of our machine. CO plugins: These components are specific to each CO, so we are limited in term of deployment options. For CF, the plugin should be deployed as part of the runtime (diego release [3]) using the so called bosh system [2]. However, for Docker, the plugin needs just to publish a JSON file in a given directory discoverable by docker daemon. This file indicates how to consume the plugin API. Kubernetes comes with a different deployment model where the plugins must be in a given location on each node of the cluster while the dynamic provisioner can run as a container or a standalone web server. The dynamic provisioner can also run as systemd script as well. Storage Backends: these latter are the responsibility of the storage providers. Some providers offer a kubernetes friendly deployment of their systems, some others offer cloud native storage and a third category have their own specific tools to deploy and manage their storage systems. Database Service: No matter how this is deployed, it must respect the constraints that we cited before. The solution that we are using is based on kubernetes cluster that ensures high availability and fault tolerance as well as a high resiliency.

Ubiquity

3.4

9

Towards a Unified Plugin Interface

Within the last few months, there is a commitment among the different COs to come up with a unified way to consume storage by having a unified API for the plugins. The effort is entitled Container Storage Interface and CSI for short. The objective of this effort is to come up with an industry standard that allows storage providers to develop a plugin once and use it across the different COs [18]. We are continuously participating in this effort and will be among the first to adopt it. This will reduce our management burden in term of maintaining different plugins. Since we will need just to maintain one plugin that will work with the different COs adhering to the standard.

4

Implementation

In this section we will provide the implementation details of each of the components of Ubiquity Framework. These components are shown in Figure 2. We will start by describing Ubiquity Service in Section 4.1. Afterwards, we will detail the implementations of each CO plugin that we support in Section 4.2. Then, we will talk about the different backends that we implemented in Section 4.3. Finally we will give some indications about the database service that we are using in Section 4.4.

Fig. 2. Ubiquity Support for CloudFoundry/Docker/Kubernetes: Overview

10

M. Mohamed et al.

4.1

Ubiquity Service

Ubiquity Service3 is implemented as go4 language project. The project has a web server that offers REST interfaces to manage persistent volumes. As shown in Figure 2, the framework allows creation, deletion and listing of volumes. The basic interfaces offered by the project are depicted in Figure 4.1 where the Storage Client interface describes the different functions that need to be executed from the server side. These functions do not need to be executed in the node/ host that will host the container. The needed functions to be executed from the host side are described in the Mounter interface. These functions allow presenting the persistent volume and making it available inside the container to be consumed by the application. Ubiquity Service is extensible and can support new COs and new storage backends. In order to add a new CO, we can easily reuse the remote client code that we implemented. The remote client is playing the role of a proxy to invoke the Ubiquity interfaces. Moreover, in order to add a new storage backend, we need just to implement the two interfaces shown in Figure 4.1 to be able to manage the specific type of storage provided by the related backend.

//StorageClient interface to create/delete volumes and attach/detach type StorageClient interface { Activate(ActivateRequest) error CreateVolume(CreateVolumeRequest) error RemoveVolume(RemoveVolumeRequest) error ListVolumes(ListVolumesRequest) ([]Volume, error) GetVolume(GetVolumeRequest) (Volume, error) GetVolumeConfig(GetVolumeConfigRequest) (Config, error) Attach(AttachRequest) (string, error) Detach(DetachRequest) error } //Mounter interface to do host specific operations mount/unmount type Mounter interface { Mount(MountRequest) (string, error) Unmount(UnmountRequest) error } Fig. 3. Ubiquity StorageClient and Mounter interfaces

4.2

Supported CO Plugins

CloudFoundry Support: Ubiquity provides persistent filesystem storage to CloudFoundry (CF) applications through its implementation of the Open Service Broker API [11] and the CF Volume Services API specification [5], as well 3 4

https://github.com/IBM/ubiquity https://golang.org/

Ubiquity

11

as by providing a volume driver plugin for CF cells (hosts) to be used by the CF runtime called Diego. The available storage backends are exposed as CF volume services through the Ubiquity Service Broker’s service catalog to support provisioning of volumes. Attaching and mounting of volumes to CF cells is supported through the Ubiquity Service Broker in conjunction with the Ubiquity CF Volume Driver Plugin. The relationships between the various involved components are illustrated in Fig. 2 and described in detail in the following. Provisioning: Ubiquity volumes can be created for consumption by CF applications by creating instances of the services exposed by the Ubiquity Service Broker API (i.e., through cf create-service-broker and cf create service). Internally the Ubiquity Service translates these Service Broker calls into ”native” Ubiquity API calls. Ubiquity/CF administrators can configure these volumes (e.g., with regard to mounting method, quotas, QoS parameters, etc.) either by (i) configuring a number of service plans in the Ubiquity Service Broker services catalog that encapsulate different preset configuration settings; or by (ii) passing arbitrary parameters through the cf create-service interface which are internally passed on by Ubiquity to the storage backend associated with the CF service. When a CF service representing a Ubiquity volume is instantiated through the Service Broker, the corresponding storage backend is eventually responsible for creating the volume. Attaching/Mounting: Attaching to and mounting Ubiquity volumes on individual CF cells is arranged for by the Diego Rep Volume Manager component of the CF installation in conjunction with the Ubiquity CF Volume Driver Plugin residing on the individual CF cells. On binding a Ubiquity service instance to a CF application (i.e., through cf bind-service), the Diego Rep volume manager (volman) instructs the Ubiquity CF Volume Driver Plugin on all affected cells to attach to the Ubiquity volume represented by the service that is being bound to the application, and subsequently mounts the volume. The mounting method (e.g., NFS, native Spectrum Scale mount, etc.) and mounting details (e.g., remote IP address of an NFS share) are chosen based on configuration details provided by the Ubiquity Service Broker in the service binding response and using pluggable logic in the Ubiquity CF Volume Driver Plugin for different storage backends. Note that if NFS is chosen as a mounting method, the CF community NFS plugin [4] may be employed in lieu of the volume driver plugin provided by Ubiquity.

Kubernetes Support: We implemented support for kubernetes persistent volumes5 following the community recommendations. Ubiquity framework offers an implementation of the dynamic provisioner API [13] and an implementation of the flexvolume plugin [10]. 5

https://github.com/IBM/ubiquity-k8s

12

M. Mohamed et al.

Provisioning: For provisioning functions, Ubiquity implements a dynamic provisioner as go language project that runs as a web server. In the context of kubernetes, a dynamic provisioner must advertise its name. This name will be used by a storage class that represents a storage flavor for kubernetes. For example, since we have support for Spectrum Scale storage, we defined a storage class named goldFileset that refers to filesets created within the gold filesystem from Spectrum Scale. The storage class refers to our dynamic provisioner to specify that any query related to this storage class should be forwarded to our running dynamic provisioner server. The dynamic provisioner will then handle the request, create a new Spectrum Scale fileset and bind it to the request (PVC). Attaching/Mounting: Attaching and Mounting a volume is a host side action that is done by Kubelet component of kubernetes. We implemented a flexvolume plugin that communicates with Ubiquity Service in order to run its functionalities. Whenever a container that needs any volume provisioned by ubiquity is to be deployed, kubernetes manager will check which nodes dispose of a Ubiquity flexvolume plugin. Then it will schedule the container to run on a specific node. An attach call is sent to ubiquity in order to make the volume available to that specific host. Afterwards, kubelet will call the flexvolume plugin to mount the volume and make it available to the container. Docker Support: Ubiquity Docker Plugin 6 is an external volume plugin that works in conjunction with Ubiquity Service to provide persistent storage to containers orchestrated by Docker. Docker communicates with Ubiquity Docker Plugin via Dockers Volume Plugin Protocol [8], a REST based API, to provide persistence storage as Docker volumes to containers orchestrated by Docker. Ubiquity Docker plugin must be run on the same host as the Docker engine. The Plugin registers itself with the Docker daemon by placing a JSON specification file in the plugin directory, which the daemon scans to discover plugins. The plugin is activated when the Docker daemon discovers the specification file in the plugin directory and activates it with a handshake API. The Docker daemon will refer to the plugin whenever a user or container refers to the plugin by name. Ubiquity Docker plugin communicates with Ubiquity Volume Service using a REST based API, to perform Volume Management operations (Create, Remove, Path, Mount, Unmount, Activate, Get, List, Capabilities) requested by Docker daemon for a specific storage backend supported by Ubiquity Volume Service. The Plugin also implements the Mounter interface for each storage backend supported by the Ubiquity Volume Service. The Mounter interface performs storage backend specific host-side operations (Mount / Unmount) so that volume created using Ubiquity volume service can be made available as a path on the Docker hosts filesystem. For example, an IBM Spectrum Scale Mounter would set user permissions on a volume made available on the host if the volume was created with user, group permissions. Similarly, an NFS mounter would mount 6

https://github.com/IBM/ubiquity-docker-plugin

Ubiquity

13

and unmount NFS shares on the host. Docker daemon makes these volumes available to the containers by bind-mounting this path into the right path inside the containers. In the Docker Swarm Mode, which consists of a cluster of Docker hosts, Ubiquity Docker Plugin must be running on all the worker nodes and all of instances of the plugin must be configured to communicate with the running Ubiquity Service. 4.3

Supported Backends

As shown in Figure 2, we have support for different storage backends. The list of supported backends is continuously growing. The first backend that we started with is Spectrum Scale backend that offers two flavors of volumes. The first one is based on Spectrum Scale filesets while the second one is based on Spectrum Scale lightweight volumes. Fileset based volumes benefit from the possibility of fixing quota on them while lightweight based ones benefit from the fact that they don’t have any number limit. So according to the use case the user can choose the most suitable flavor. We also have support for spectrum scale NFS volumes that could be consumed using our plugins or by any default NFS plugin developed by the community. Moreover, we added support for Openstack Manilla to enable creating persistent volumes out of the offered NFS shares. All these backends are based on shared filesystems. The work in progress that we are doing now is to add support for block devices to support more use cases in the future. 4.4

Database Service

In our implementation we are using etcd key value store [17]. This is an opensource distributed key value store. It allows to share configurations and service discovery for Container Linux clusters. We used etcd to enforce the high availability of Ubiquity by handling the leader election and the loss of the current leader. We are also using it for all locking mechanisms and concurrency management scenarios.

5

Related work

Persistence for COs is relatively a new topic that goes back to the last three years. The issues and challenges have not been addressed widely by the research community. Meanwhile, in the industrial context we see that there are different efforts to resolve this problem. These efforts are mainly led by different storage system providers who want to support the trend of their customers moving towards containers. For example, EMC proposes Rex-Ray [6] as a storage integration framework for Container Orchestrator and provides plugins for their storage backends. Rex-Ray focuses on Docker, Kubernetes and Mesos, as publicly available at the time of writing. In the same context, NetApp proposes Trident [12] as a solution for Kubernetes-based systems. CoreOS proposes their

14

M. Mohamed et al.

solution called Torus [9], which is targeting persistent storage to containers orchestrated by Kubernetes. Compared to the existing efforts, Ubiquity presents an easy way to bring together the different existing COs and storage providers in a open way. It also supports the paradigm of PaaS as well as regular container orchestration by providing service-oriented pathways into storage life-cycle management, as articulated in the implementation of the Cloud Foundry service broker API. This support enables the integration not only with systems based on Docker and Kubernetes but also with PaaS like CloudFoundry. Ubiquity comes also with an easy way to add support for new storage providers without the need to understand all the specificities of the existing COs. These storage providers can be on-site systems or plugins to Cloud storage providers where this makes sense.

6

Conclusions

With the rise of Container Orchestrators in enterprise data centers, demand arises to run not only stateless applications but also stateful applications in containers. This demand has exposed a gap in these COs that desire to run these applications in term of persistent storage support. Moreover, given that each CO has a different set of APIs and storage features, the burden is on the storage vendor to integrate into each CO. So far there are little efforts being done to integrate this diverse set of COs and storage systems. In this paper, we proposed the Ubiquity framework that addresses the complexity of bringing together the different COs and Storage providers in the context of sharing persistent state across COs. Ubiquity offers an easy way to plug in the support of new - versions of - COs on the one hand and, on the other hand, to enable the support for new storage backends independently of the specific CO. Ubiquity is an open source project that is continuously growing by adding support of new functionalities, new COs and new storage backends. While multiple storage vendors offer persistence solutions for some COs for their storage systems, Ubiquity supports a large spectrum as well as multiple orchestration paradigms, including Platform-as-aService, providing additional flexibility for managing the storage life-cycle for a wider range of systems. Due to its open source nature, the Ubiquity community edition is also an ideal platform for academic experimentation. In future work we will look at emerging container interface standards such as the Container Storage Interface being discussed by the Cloud Native Computing Foundation and adding new, storage related functionality. In addition, we are evaluating storage management performance of COs and storage system combinations in the context of different workloads.

References 1. Azab, A.: Enabling docker containers for high-performance and many-task computing. In: 2017 IEEE International Conference on Cloud Engineering (IC2E). pp. 279–285 (April 2017)

Ubiquity

15

2. bosh.io: bosh. https://bosh.io/docs, retrieved on June 19, 2017 (2017) 3. CloudFoundry Foundation.: Diego architecture. https://docs.cloudfoundry.org/concepts/diego/diego-architecture.html, retrieved on June 19, 2017 (2017) 4. CloudFoundry Foundation.: Nfsv3 volume driver. https://github.com/cloudfoundry/nfsv3driver, retrieved on June 18, 2017 (2017) 5. cloudfoundry.org: Volume services. http://docs.cloudfoundry.org/services/volumeservices.html, retrieved on June 18, 2017 (June 2017) 6. Dell EMC: Rex-ray openly serious about storage. https://rexray.readthedocs.io/en/stable/, retrieved on June 20, 2017 (2017) 7. Docker Inc.: A better way to build apps. https://www.docker.com/, retrieved on June 19, 2017 (2017) 8. Docker Inc.: Volume plugins. https://docs.docker.com/engine/extend/plugins volume/, retrieved on June 18, 2017 (2017) 9. Michener, B.: Presenting torus: A modern distributed storage system by coreos. https://coreos.com/blog/torus-distributed-storage-by-coreos.html, retrieved on June 20, 2017 (2017) 10. Nelluri, C.: Flexvolume explored. https://www.diamanti.com/blog/flexvolumeexplored/, retrieved on June 20, 2017 (2017) 11. Open Service Broker: Open service broker api specification. https://github.com/openservicebrokerapi/servicebroker/blob/v2.12/spec.md, retrieved on June 18, 2017 (2017) 12. Sullivan, A.: Introducing trident: A dynamic persistent volume provisioner for kubernetes. http://netapp.io/2016/12/23/introducing-trident-dynamic-persistentvolume-provisioner-kubernetes/, retrieved on June 20, 2017 (2017) 13. The Kubernetes Authors: Dynamic provisioning and storage classes in kubernetes. http://blog.kubernetes.io/2016/10/dynamic-provisioning-and-storage-inkubernetes.html, retrieved on June 19, 2017 (2017) 14. The Kubernetes Authors: Persistent volumes. https://kubernetes.io/docs/concepts/storage/persistent-volumes/, retrieved on June 19, 2017 (2017) 15. The Kubernetes Authors: Production-grade container orchestration. https://kubernetes.io/, retrieved on June 19, 2017 (2017) 16. Uehara, M.: Performance evaluations of lxc based educational cloud in amazon ec2. In: 2016 30th International Conference on Advanced Information Networking and Applications Workshops (WAINA). pp. 638–643 (March 2016) 17. Walker-Morgan, D.: etcd introduced. https://www.compose.com/articles/etcdintroduced/, retrieved on June 20, 2017 (2017) 18. Yu, J., Ali, S., DeFelice, J.: Container storage interface (csi). https://github.com/container-storage-interface/spec/blob/master/spec.md, retrieved on June 19, 2017 (2017)