Best Practices for Developing connectors to Cloud Platforms/Services
Cloud Service Providers like Amazon Web Services, Microsoft Azure, Google Compute Engine, IBM Bluemix, Rackspace, Digital Ocean, Oracle Cloud, etc. and the Cloud Platforms like Openstack, Cloudstack, vCloud, Onapp etc. are increasingly programmable through API or Web Services. To consume these API/Web Services, we require to develop a connector. We have experience developing API connectors to almost all Cloud platforms or Services. This blog is to share our experience and provide some of the best practices to be followed for developing Web Service connectors to Cloud Services or Platforms
Why API Connectors
The API Connectors to Cloud platforms/Services are required for the following reasons,
- Discover cloud resources
- Manage resources
- Automate operations
- Orchestrate resources
- Integrate with tools/products
- Report summary of resources and its utilization
Steps for developing API Connectors
The following are the sequence of steps to develop efficient and effective connectors to Cloud Platforms/Services
- Understand API Characteristics
- Analyse API support provided by Cloud platform/Services
- Identify API Operations
- Validate API End Points
- Verify API Authorization
- Check Quota
- Analyse Cloud Resource Pricing
- Design considerations for API Connector
The API Characteristics includes API Type, Authentication mechanism and Request/Response type.
Typically, APIs are exposed either through REST (Representational State Transfer) / SOAP (Simple Object Access Protocol). REST is becoming the standard and replacing some of the old SOAP API. This is very evident as per the data in Table-1.
Every Cloud Platform use different types of authentication mechanism to access the API and it is important to understand the Authentication Mechanism. The typical API Authentication mechanism are
- Basic Authentication
- Token Based Authentication
- SSL Authentication
- Multi Factor Authentication
The basic authentication uses the combination of username and password encoded with the base64 which is supplied in the Authorization HTTP Header.
Headers: Content-type: application/xml
Authorization: Basic dG9ib3RyYXM6cTE=
IPaddress | host - api.xyzcloud.com
PortNo – 4465
Path – paci/v1.0/ve
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ve-info subscription-id='100001' name='web' state='created' description='Web Server'/>
Token Based Authentication (X-Auth-Token, Oauth Token)
Token based authentication provides temporary token based on the user credentials for Authentication. A resource required to refresh temporary tokens when they expire. An internal authentication handler authenticates based on the provided tokens in the header.
Headers: Content-type : application/json
Secure Sockets Layer is a standard security technology for establishing an encrypted link between a server and a client – typically a web server (website) and a browser. The SSL Authentication requires SSL Certificate to be uploaded to the platform under the subscription. The API end point need to be authenticated though SSL Certificate
Multi factor Authentication
Multi-Factor Authentication (MFA) adds an extra layer of protection on top of user name and password. MFA Supported API require user name and password for the first factor and authentication code from MFA device as second factor. These multiple factors provide increased security for API Endpoint.
API Response Type
The request and response type of the API need to be looked at for feeding input and consuming the output. The API Request and Response are either XML or JSON. The connector need to convert the response as per its interface requirement
API connectors for the Cloud Platform/Services can be developed by the following options
- Consume REST/SOAP API Directly using programming language of your choice like Python, Java, Dotnet, Ruby, GO, Node.Js, etc.
- Some of the Cloud platform/services offer SDKs which is a wrapper on top of the API to make it easy for developers Consume programmable SDK specific to Python, Java, Dotnet, Ruby etc. provided by the platforms/services
The table below shows API support for some of the leading Cloud service providers and platforms..
|Cloud platform/Services||API Type||SDK Support||Authentication Type||API Response||API Documentation|
|AWS||REST, SOAP, QUERY, REST-QUERY (Types based on Service)||boto(python),|
|Token Based & Multi Factor||JSON,XML||Click Here|
|Microsoft Azure||REST||Azure-sdk-for-python, azure-sdk-for-net, azure-sdk-for-node, azure-sdk-for-ruby||SSL Authentication||JOSN, XML||Click Here|
|Rackspace||REST||Pyrax (python), fog (ruby), java,go, .net, php, node.js||Token Based||JSON, XML||Click Here|
|Openstack||REST||Python-openstackclient, apache-libcloud, pyrax, Openstack_SDK-DotNet, jstack,|
js-openclient, golang-client etc.
|Token Based||JSON||Click Here|
|Cloudstack||REST||Apache-libcloud||Token Based||JSON, XML||Click Here|
|VMWare vCloud||REST||Pyvcloud (python)||Token Based||XML||Click Here|
|OnApp||REST||Basic HTTP or API key||JSON, XML||Click Here|
|OnApp||REST||Basic HTTP or API key||JSON, XML||Click Here|
|Lunacloud||REST||Lunacloud-sdk-java||Basic HTTP||XML, JSON||Click Here|
|DigitalOcean||REST||https://developers.digitalocean.com/libraries/||Token Based||JSON||Click Here|
|Oracle Cloud||REST||Bmcs-python-sdk, bmcs-java-sdk||Basic HTTP||JSON||Click Here|
|Internap||REST||HAPI-python, hAPI-java, hAPI-perl, hAPI-php, hAPI-js||Token Based||JSON, XML||Click Here|
Note: Cloud Service API Details is available in Google Sheets. Encourage the service providers and other readers to update the API endpoint of the cloud so that it acts as easy reference for developers.
Understand the API Operations supported by the platform by going through the API Documentation and identify the operations that you would like to consume. It is better to perform the operations through the Management Portal or Dashboard to get the sense of how it works before start consuming it through the API. The first thing that you need to do to consume the API is Authentication and then you can try basic Read Operations before performing create option.
Validate API End Point
API End points are not the same as Cloud Platform Management URL. API Endpoint typically includes host, port and path. It includes Access key and Secret key if it is REST API. Validate the accessibility of API Endpoints for the platforms or services using any of the tools like POSTMAN, RESTClient etc. In case of Token based authentication, we need to generate the token and feed the token in RESTClient.
After API is authenticated using the Authentication mechanism, we need to know the authorization for the given user in the Cloud platform or Service. Example: AWS Identity and Access Management (IAM), we may have successfully authenticated but we can perform only the action that we are authorized in the IAM.
Cloud platform/services impose quota for resources that can be consumed by user account. It is better understand the quota limits upfront. Example, AWS limits allocation of Elastic IP to 5 for the account. However this can be increased by raising request. Openstack admin can define limits to resources in each Project which is consumed by the user.
Analyse Cloud Resource Pricing
It is very important to check Pricing for the resources by the Cloud service provider. Cloud Service providers charge resources on monthly or Hourly or Minute basis. It is important to know these pricing before consumption otherwise we will have surprises in the billing. It is also important to understand the free tier offered by the service provider in detail so that we don’t have any surprises.
Design consideration for API Connector development
- It is ideal to use SDK provided by the platform if you are developing connector to only one platform.
- If the platform do not provide SDK for the required language, there are tools like APIMatic, AWS API Gateway etc. which helps in generating SDK for the API Endpoint. Use of SDK in connector development reduce development effort.
- If you are looking to develop connectors across multiple cloud you can consider using Third Party SDK as it helps in accelerating the development however if you want the connector to be dynamic and you require to evolve with the platform or services, it is better to go with the SDK provided by platform or services as the Third party SDK support for some of the new releases may take time.
- Understand the API Rate limit (No. of API request that can be made to the API endpoint in a period by user) set by some of the providers and platforms as it gives an indication of how frequent we can make calls to an end point.
- For some of the asynchronous API (in cases the API response is not immediate), the responses are provided through PUSH or retrieved through POLL. The ‘Push’ model expects a call back end point to which it send the response when it is available. In ‘POLL’ model the requestor to call an API repeatedly to check for an update in status. When you have to poll or retry an API request, we recommend using an exponential backoff algorithm to calculate the sleep interval between API calls. The idea behind exponential backoff is to use progressively longer waits between retries for consecutive error responses.
- Some of the cloud service providers/platforms exposes different endpoints for each services to be consumed. It is recommended to maintain a service catalog with API End point for using proper end point.
- Sometimes the endpoint vary based on the Sub accounts of the Cloud platform or Services, ensure that the endpoints are concatenated as per the requirement before making a call
Hope this helps. Have fun developing API Connector…