Developers Club geek daily blog

1 year, 6 months ago
SparkleFormation

If you seriously use AWS (Amazon Web Services), then for certain know about an opportunity to describe infrastructure by means of JSON templates. In AWS this service is called CloudFormation. In fact this solution allows you to describe a desirable status of any resources available in AWS (instansa, layers of opsworks, ELB, security groups, etc.). The set of resources is called a stack. After loading of CloudFormation of a template the system itself or will create necessary resources in a stack if they are absent yet, or will try to update existing to a desirable status.

It well works if you have a small amount of resources, but as soon as the infrastructure expands problems appear:
  • In JSON there is no opportunity to use cycles and for similar resources it is necessary to repeat the same parameters and in case of change too (not DRY)
  • For the configuration record for cloud-init double escaping is necessary
  • In JSON there are no comments and it has a bad cheloveko-chitayemmost

To avoid similar problems engineers from Heavy Water wrote on ruby DSL and CLI for generation and work with these templates under the name SparkleFormation (github).

DRY


When I came to the current project we had CloudFormation the template containing about 1500 lines of the description of resources and about 0th lines of comments. After use of SparkleFormation the template began to hold 300 places, many of which comments. How we it achieved? For a start we will look as CloudFormation works, the typical description of a resource looks so:
Creation of ELB
    "AppsElb": {
      "Type": "AWS::ElasticLoadBalancing::LoadBalancer",
      "Properties": {
        "Scheme": "internal",
        "Subnets": [
          {"Ref": "Subnet1"},
          {"Ref": "Subnet2"}
        ],
        "SecurityGroups": [
          {"Ref": "SG"}
        ],
        "HealthCheck": {
          "HealthyThreshold": "2",
          "Interval": "5",
          "Target": "TCP:80",
          "Timeout": "2",
          "UnhealthyThreshold": "2"
        },
        "Listeners": [
          {
            "InstancePort": "80",
            "LoadBalancerPort": "80",
            "Protocol": "TCP",
            "InstanceProtocol": "TCP"
          },
          {
            "InstancePort": "22",
            "LoadBalancerPort": "2222",
            "Protocol": "TCP",
            "InstanceProtocol": "TCP"
          },
          {
            "InstancePort": "8500",
            "LoadBalancerPort": "8500",
            "Protocol": "TCP",
            "InstanceProtocol": "TCP"
          }
        ]
      }
    }


As SparkleFormation allows to use normal ruby a code in DSL, it is possible to rewrite it so:
Creation of ELB in SparkleFormation
  resources(:AppsElb) do
    type 'AWS::ElasticLoadBalancing::LoadBalancer'
    properties do
      scheme 'internal'
      subnets [PARAMS[:Subnet1], PARAMS[:Subnet2]]
      security_groups [ref!(:SG)]
      # port mapping 80->80, 22 -> 2222, etc.
      listeners = { :'80' => 80, :'2222' => 22, :'8500' => 8500 }.map do |k, v|
        { 'LoadBalancerPort' => k.to_s,
          'InstancePort' => v,
          'Protocol' => 'TCP',
          'InstanceProtocol' => 'TCP' }
      end
      listeners listeners
      health_check do
        target 'TCP:80'
        healthy_threshold '2'
        unhealthy_threshold '2'
        interval '5'
        timeout '2'
      end
    end
  end


As it is possible to notice we we do not repeat in the description of each port any more and adding of new will take at us only one place. Moreover if at us it is necessary to create many almost same resources but differing in 1-2 parameters, SparkleFormation provides such entity as dynamics where you can describe an abstract resource to which transfer parameters:
Example from documentation
# dynamics/node.rb
SparkleFormation.dynamic(:node) do |_name, _config={}|
  unless(_config[:ssh_key])
    parameters.set!("#{_name}_ssh_key".to_sym) do
      type 'String'
    end
  end
  dynamic!(:ec2_instance, _name).properties do
    key_name _config[:ssh_key] ? _config[:ssh_key] : ref!("#{_name}_ssh_key".to_sym)
  end
end

And then we can cause this abstract resource in a template:
SparkleFormation.new(:node_stack) do
  dynamic!(:node, :fubar)
  dynamic!(:node, :foobar, :ssh_key => 'default')
end


Thus we can reuse resources necessary to us and in need of change to change everything in 1 place.

Cloud-init


We often use an opportunity to transfer an instansa when loading cloud-init a config in the form of the yaml-file and by means of it to execute installation of packets, a configuration of CoreOS, separate services and other settings. The problem is that yaml has to transfer an instansa in user-data in CloudFormation a template and it looked approximately so:
Mad escaping
        "UserData": {
          "Fn::Base64": {
            "Fn::Join": [
              "",
              [
                "#cloud-config\n",
                "\n",
                "coreos:\n",
                "  etcd:\n",
                "    discovery: ", {"Ref": "AppDiscoveryURL"}, "\n",
                "    addr: $private_ipv4:4001\n",
                "    peer-addr: $private_ipv4:7001\n",
                "  etcd2:\n", 
...


As it is possible to see it absolutely not readably, ugly and it is badly supported, that it is possible to forget about syntax highlighting. Thanks to the fact that in DSL it is possible to use ruby a code, all yaml can be taken out in the separate file and just to cause:
user_data Base64.encode64(IO.read('files/cloud-init.yml'))

Apparently it is much more pleasant than to edit it in JSON. Instead of IO.read it is possible to use also HTTP a challenge for any parameters if you need it.

CLI


At ourselves in the project we use own wrapper for management of templates, but the same command provides CLI (Command Line Interface) for management of templates, the called sfn. By means of it it is possible to load, delete and update CloudFormation stacks, the sfn create, sfn destroy and sfn update commands. There integration with knife is also implemented.

In the whole ambassador of 4 months of use of SparkleFormation I am happy with it and I hope I will not return on plain JSON for the description of infrastructure any more. In plans to try all workflow, including sfn offered by the Heavy Water command.

This article is a translation of the original post at habrahabr.ru/post/269501/
If you have any questions regarding the material covered in the article above, please, contact the original author of the post.
If you have any complaints about this article or you want this article to be deleted, please, drop an email here: sysmagazine.com@gmail.com.

We believe that the knowledge, which is available at the most popular Russian IT blog habrahabr.ru, should be accessed by everyone, even though it is poorly translated.
Shared knowledge makes the world better.
Best wishes.

comments powered by Disqus