In our previous blog we have discussed about Orchestration and the evolution of Domain Specific Languages in orchestrating the workflows. We also mentioned that DSL brings in higher level of abstraction to simplify the complexity of provisioning the entire cloud stack with limited customization and we mentioned few Cloud Orchestration tools based on DSL and organizations which provide standards for DSL. Now in this blog let us see how the organizations follow the process of provisioning Cloud Stack using DSL’s and orchestrate the same. Before going to details let us know
What is Domain Specific Language?
Have you ever heard about a Domain specific language! If you have worked on development of a webpage or a service used by a webpage then you have already came across DSL’s, or Domain Specific Languages. DSL’s are small, expressive programming languages custom designed for specific tasks. To say it in simple and understandable way
- A keyword input file to an application that receives input data is a DSL.
- A configuration file is a DSL.
- A makefile is a DSL which is used to define rules and dependencies for building an application.
If you have written any of these then you have already started using DSL’s.
Advantages of DSL’s with an example
DSL’s are simple and concise, simplicity is key for success of a DSL. A person familiar with the languages domain can easily understand DSL. DSL is easy to create. A DSL is highly expressive, simple and concise at the same time. This can help the user of an application more productive.
Let us take an example, if you need to create a DSL for provisioning an instance in IaaS platform then we need to specify particular set of requirements for provisioning, actually requirements should be gathered from the customer, so the DSl we create for them should be in the vocabulary which customer understands and is generally used (like using the keywords Cpu,Ram,Disk). You want them to use the syntax you provide, but it should seem to them that they are merely specifying some discrete rules. And they should be able to do so without getting an impression that they are really programming or even using some kind of language. Syntax below is an DSL which is written in YAML
If you observe you don’t find it difficult as a programming language, more over its readable and seems
like a text file.
Types of DSL’s
A DSL may be classified as internal or external depending on how it is designed and implemented.
Languages created inside other languages are called internal Domain-Specific languages. These are also called as Domain-Specific embedded languages. An internal DSL follows the syntax of host language which reduces the effort of designing the syntax of DSL. The new language will be syntactically compatible with its host language so it can be executed by the same infrastructure i.e. compiled by the same compiler, interpreted by the same interpreter.
As a result new Domain specific concepts are made available for the host languages which use it and create a new language that mixes a subset of the host language syntax with the new domain specific concepts.
Some of the internal DSL’s are Rake by ruby, Css , Gant written using groovy,YAML used in TOSCA
External DSL’s have their own syntax, they are not dependent on the other language syntaxes. You can design the syntax of the external DSL’s as you like. You need to define the grammar for you language. In the same way you need to create a compiler to parse and process the syntax. An external DSL gives you a lot of flexibility but you have to take the time to do the hard work of compiling it.
Some of the external DSL’s are makefile, ant and Xtext from eclipse which is an open-source project helps to implement your own DSL with appropriate IDE support.
So which DSL is better?
It’s difficult to say which one is better as each one has their own advantage, if your project is already developed in Ruby, node or java you can try using internal DSL’s which are easy to create and adopt. If you don’t want to use some general purpose language then you have some external DSL’s with IDE available in market like Xtext, ant, makefile etc. So here choice is yours, only thing to say is that DSL’s are emerging in providing cloud orchestration solutions and you have the necessity to know about them if you are working on cloud technologies.
Demanding cloud Stack using DSL and its Orchestration
Now let us see an example of an Internal DSL which is the fastest emerging one and is used by many organization’s to orchestrate cloud stack.
In an enterprise which specially has innovation projects and prototyping, requirements may change all the time and the iteration cycle should be short. For an organization beside project portals supporting an agile methodology, there is often a need to also simply provide innovation projects with a platform of readily available server infrastructure with software and services installed in it. Manual process of doing these things may consume time as well as man power, so here is the need of automating things which save both man power as well as time .
Let us see a model in which cloud stack id deployed through orchestration tool that uses templates designed with DSL’s with some standards like TOSCA YAML
Figure above gives an overview of DSL approach to create an instance, deploy software in it and have a monitoring service like NAGIOS to monitor both the server as well as services in it.
Let us go to the technical details on how this approach works and the process involved in it.
Initiating the process
Left bottom part is the DSL template that contains the requirement of the infrastructure like security groups, key pairs, volumes, Server Ports. Template also contains the details of the software and services like Docker word press, apache, MySQL that needs to be deployed and installed in the instance created. There are certain grammar rules that need to be followed to specify the requirements in DSL, the grammar rules may change according to the languages like ruby, python, node.js you are using.
Generally the grammar for infrastructure is defined as FWRule, src, dst, volumes, servers, images, flavor, cpu, ram, disk, kp, Sg etc. System may consider the default values for firewall rules, key pairs and security groups if they are not mentioned in the DSL.
Once the template is ready to use, the left upper part is the script that reads the values from the template and calls the IaaS provider API’s through MQ software with the information needed to provision the instance. We can have these scripts written in python, ruby, node.js etc.
Processing the workflow and executing it
In middle layer we can see Message queue (MQ) software that acts as a message broker between the services and pushes the data to each service continuously during the provisioning of instance. When we call the API of the IaaS provider to provision the instance, we should get the details on the status of the provisioning like creating, created, failed etc. To continuously pool the data between the services we call API’s through MQ software, so that it will keep us updated on the status and help to follow the workflow. RabbitMQ is one of that kind.
Once after the provisioning of instance we immediately need to set the platform for the software installation, to set the platform we need to install Apache (HTTP) and MySQL (DB server). To automate this installation process we have Puppet and Chef which takes care of installing the software’s if we have the scripts ready.
On the right side of the figure we can see the VM is provisioned and the software is deployed in the platforms mentioned in the template. Is it completed? What if there is problem in the server and the service hosted in it? So the real maintenance starts once after the setup is done with the infrastructure and the software deployed in it is ready to use.
Monitoring a service and a server plays an important role once after the deployment. This is where we need monitoring services like NAGIOS which alerts us whenever there is a problem either in service or in server. We can configure different types of alerts using NAGIOS. Type of alerts that can be configured in service is warning, error, timeout etc. Using NAGIOS we can check the status of the server or service by setting up the auto ping option which pings the service by using its port and the server by using its IP. We can even configure an Email or SMS alert when there is a problem.
We can even use Open Stack Mistral which is a workflow service that helps in executing the steps in particular order if we describe the process as set of tasks and upload the task relationship descriptions to it. By using Mistral we can set the workflow by using the steps defined above.
So within minutes we have defined a template in DSL, executed it with appropriate programming language supported and provisioned instance and deployed software as well as service in it. Had a monitoring service on top to keep the server and services alive and check the status.
Mentioned above is the heat template defined in YAML. Here you can find the instance details mentioned under properties section and the software installation details mentioned under user data section. Installation of Apache and MySQL will be taken care by CHEF and the details need not be mentioned in the template