Guideline for Service Deployment
In order to help ENVRIplus service providers and RIs have clear ideas on how to deploy services on e-Infrastructures, WP9 works on a guideline explaining service deployment approaches. EGI is one of the e-Infrastructures supporting the ENVRIplus project and working closely with WP9. Several use cases (such as IC_14 and TC_2) and WP7 use EGI Federated Cloud, which is a typical e-Infrastructure service allowing to deploy and operate services. In this guideline, we use this e-Infrastructure setting as a concrete example to discuss deployment approaches.
EGI Federated Cloud
EGI Federated Cloud (FedCloud) is a grid of clouds with a harmonised operational behaviour.
EGI FedCloud federates institutional Resources Providers (RP) offering an Infrastructure-as-a-Service (IaaS) solution composed of 22 providers form 14 National Grid Initiatives (NGI) running different Cloud Middleware Frameworks (CMF), 2/3 are OpenStack, 1/3 are OpenNebula and there is one Synnefo site. Around 6000 cores are available.
In the federation, the Clouds and their interconnections are based on open technologies such as the OpenNebula, OpenStack and Synnefo Cloud Middleware Frameworks (CMF) and on open Standards such as the Open Cloud Computing Interface (OCCI) for Virtual Machine (VM) management and the Cloud Data Management Interface (CDMI) for object storage. FedCloud is also offering native access to OpenStack sites. A common authentication and authorization layer using x509 and VOMS is used and OpenID Connect integration is ongoing. Operational tools for accounting, monitoring and ticketing are operated centrally.
The EGI FedCloud value proposition is the capability of instantiating virtual machine images across heterogeneous cloud providers in a uniform way, giving the possibility to compute and store data across a combination of public and community clouds of choice and bringing computing to data.
Multiple tools are available to interact with the Federated Cloud. Firstly, the EGI Application Database is an application catalogue for Virtual Appliances (VAs). Secondly, it is possible to use low level Command Line Interfaces (CLI) and Application Programming Interfaces (API). Finally, some high level tools such as the EGI AppDB VMops dashboard are built on those APIs to offer more user friendly and automated interfaces to FedCloud.
The EGI Application Database (EGI AppDB)
Figure 1: The EGI AppDB showing EGI Ubuntu Docker Virtual Appliance
The EGI AppDB is an application catalogue based on a user-friendly web portal for sharing:
- EGI-endorsed Virtual Appliances (VAs)
- Software packages, tarballs and scripts
- Sites and Virtual Organisation (VO) information, including VA availability at a sites for each VO
- Information about people
Using the EGI AppDB, users can share and search VAs and software. It is possible to publish metadata about the VA such as a description or hardware requirements. For the packages, software repositories are automatically created for debs and RPMs simplifying their usage. The VO-specific Virtual Appliances are synchronized on every FedCloud sites supporting them.
Deploying using Command Line Interface (CLI) and Application Programming Interface (API)
The lowest level of interaction with EGI FedCloud is using the supported CLIs and APIs.
Currently, FedCloud is accessible using two realms: OpenStack and OCCI. OCCI can be used to access OpenStack, OpenNebula and Synnefo sites. It is also possible with OpenStack sites to use the OpenStack native tools.
Virtual Machine contextualization has to be done using cloud-init, it can be used to allow remote connection using ssh or to bootstrap a complete service.
Deploying using Complementary Services and tools
In addition to the IaaS cloud service, some other services are offered to the users, some of which are provided by EGI while others are just higher level services that can ease and automatize the usage of a IaaS solution.
For the long tail of science users, an Application On Demand service was created, offering an easy on-demand deployment of application specific clusters (implemented using the EC3 tool) as well as some ready to use and pre-configured Science Gateways including various tools.
Some other third party applications are also able to access FedCloud. They can be grouped in two broad categories: Virtual Research Environments (VRE) and orchestrators. For example, the VREs d4science.org and WS-PGRADE are both able to use FedCloud resources and allow to create user-friendly and community-based Science Gateways simplifying the use of the cloud resources.
Orchestrators are components that can automatically deploy a complete virtual infrastructure on one or multiple cloud sites, sometimes even among different cloud providers. This category includes UPV’s Infrastructure Manager (IM) offering a web interface, a Command Line Interface client and multiple APIs allowing to easily deploy and update complex infrastructures; the Dynamic Realtime Infrastructure Planner (DRIP) allowing to orchestrate deployments based on time constraints; and the INDIGO Orchestrator, the central component of the INDIGO Platfrom-as-a-Service solution. Those services can deployed infrastructure that are specified in TOSCA and interact with the sites using OCCI.
Deploying using EGI AppDB VM Operations Dashboard (EGI VMops dashboard)
The EGI VMops dashboard is a graphical user friendly interface allowing users to create and deploy topologies on FedCloud. A topology is composed of one or multiple instances of a Virtual Appliance and can include contextualization and automatic mounting of additional storage.
The EGI VMops dashboard is a web portal that is operated centrally by EGI and is built on top of the EGI services. It is integrated with the AppDB, thus allowing users to see the list of Virtual Appliances available for a specific VO and it will allow to deploy the topology on one of the available sites supporting this VO as it is also integrated with EGI Information System and monitoring. Due to the integration with the EGI CheckIn service (EGI’s Authentication and Authorization Infrastructure (AAI)) there is no need for a X509 certificate for the end user. Technically the deployment orchestration is managed by the Infrastructure Manager.
The EGI VMops dashboard is an example of high-level services offering high value to users and that can be built using the standard APIs supported by FedCloud.
Figure 2: The EGI AppDB VMops dashboard
EGI Federated Cloud is currently focused on delivering Infrastructure-as-a-Service (IaaS) solutions, but leveraging orchestrators work on Platform-as-a-Service (PaaS) and Software-as-a-Service (SaaS) solutions is ongoing. The Service orchestration will allow the ENVRIplus community to more easily deploy and manage a complete virtual infrastructure that can be tailored to the specific need of each Research Infrastructures. Another important effort is the OpenID Connect integration, which will simplify the access and usage of the API and CLIs by removing the need for a X509 certificate. Being used also by a number of other tools, OpenID Connect will ease integration with other solutions, such as ENVRIplus services supporting OpenID Connect, thus allowing to use a federated Authentication and Authorization Infrastructure (AAI) supporting Single sign on across all services. FedCloud monitoring probes are being extended to improve the reliability of the infrastructure and exhaustively monitor it. Finally, in order to go beyond the Virtual Appliances usage and follow the growing usage of containers, an EGI DockerHub of trusted containers will be created. It will offer access to selected tools useful for the various ENVRIplus RIs that use containers to easily package and deploy their user-community specific applications will be able to propose them in order to easily present and share them among the EGI community.