knife-bs

Description

Declarative cloud infrastructure management tool build on top of opscode/knife-ec2.

Allows declaraction of top level organizations, environments (production, staging). Declare instance profiles and combine them to form stacks. Stacks and instances can be deployed to subnets within an ec2 region and optionally a vpc.

Supports custom code execution for fine-tune deployment through the use of mixins

Integrates with opscode/chef

Installation

Bundler

Add the following to your Gemfile:

gem 'chef'
gem 'knife-bs'

Otherwise

gem install knife-bs

Configuration

You have the option of passing your AWS API Key/Secret into the individual knife subcommands using the -A (or --aws-access-key-id) -K (or --aws-secret-access-key) command options

Chef/knife looks for the config file in ~/.chef/knife.rb. It contains connection info for AWS (keys, etc.). It can be generated by running:

knife configure --initial [--defaults]

In order for knife-bs to work, you must add some additional information:

...
knife[:yaml]            = '<PATH TO YOUR bs-atlas.yaml>'
knife[:mixins]          = '<DIRECTORY WITH CUSTOM USER MIXINS>'

** Make sure knife[:mixins] directory exists **

For more details about configuring chef/knife, click here

bs-atlas.yaml sample

Take a look at the schema to get a feeling for the structure, or use the sample below to start

organizations:
  bitsight:
    domain: bitsighttech.com
    # Only domain is a reserved tag at this level.
    mixin:
      chef:
        # Recipe(s) to apply to each instance in this organization
        run_list:
          - recipe[bs-base]
      volume:
        # Every instance will have its instance storage RAID-ed
        # and mounted to /media/eph
        ephemeral:
          raid:
            name: ephemeral_raid
            level: 0
            mount: /media/eph
      # Every instance within this organization will have
      # an environment variable ORGDOMAIN with value
      # bitsighttech.com
      var:
        ORGDOMAIN: bitsighttech.com
regions:
  east:
    id: us-east-1
    vpc:
      ame1:
        vpcid: vpc-XXXXXXXX
        mixin:
          ami:
            # Sets base AMI to use for instances in this region. Prefix
            # is added to AMIs generated within this region
            base: ami-XXXXXXXX
            prefix: bs
          # VPC environment variable for all instances belonging
          var:
            VPC: ame1
        subnets:
          prd1:
            network: xxx.xxx.xxx.xxx/24
            id: prd1
            env: production
            mixin:
              # Availability Zone to use
              az: us-east-1c
          stg1:
            network: xxx.xxx.xxx.xxx/24
            id: stg1
            env: staging
# Environment listing
env:
  staging:
    mixin:
      # Tag the instance with the environment
      tag:
        environment: staging
  production:
    mixin:
      tag:
        environment: prod
stacks:
  # A hadoop cluster with 5 slaves and one master node
  cluster:
    profiles:
      master:
      slave:
        count: 5
profiles:
  master:
    # The last IP octet if used in a subnet
    ipaddress: 16
    # [Instance flavor](http://www.ec2instances.info/)
    type: m2.xlarge
    hostname: ms101
    mixin:
      chef:
        # In addition to base recipe, run master
        run_list:
          - recipe[bs-master]
      tag:
        instance: master
      volume: &EBSMOUNT
        ebs:
          /dev/sdf:
            format: ext4
            mount: /ebs
  slave:
    type: m1.xlarge
    # Hostnames for slaves are rs101, rs102, ... rs199
    hostname: rs1%02d
    mixin:
      ami:
        suffix: -slave
      chef:
        # In addition to base recipe, run slave
        run_list:
          - recipe[bs-slave]
      tag:
        instance: slave
      # Use the same volume configuration, except with
      # explicit volume size
      volume:
        <<: *EBSMOUNT
        size: 200 #GB

Usage

To see the list of commands available type knife at the command line. It shows all the commands available under knife category. Check the BS COMMANDS category for our commands. Each command has its own set of options, which can be viewed by typing --help at the end of the command. Use -V or -VV to get more verbose output while running any command.

For BS commands, most of the options are picked up from the bs-atlas.yaml file. This file describes the current state of our AWS infrastructure as shown above

knife bs server create

This command is used to spin up instances. Multiple instances can be created using the --count NUMBER flag. Specify ebs volume sizes to attach to the instance(s) using --ebs SIZE[,SIZE,...]. To mock all calls to AWS, use --mock. Omit chef using --nochef. If a price tag is available in the YAML, spot instances will be created by default. To use on-demand instances, use --nospot

See the help text for other options/flags.

knife bs server delete

Terminate instances in AWS using wildcards and subnet/environment specification.

knife bs server list

Use this command to view what instances are running in a subnet or environment. See help text for usage

knife bs stack create

Used to bring up multiple nodes of different types. Uses stack definitions inside the YAML to pick up stack options. Brings up nodes in parallel. Allows per-profile overrides using --each.

See banner in ruby code or help message for more details

knife bs ami create

** Work in Progress **

Uses information from YAML to create new AMIs.

The ami id is calculated automatically. Chef takes care of installing all the packages and the configuration necessary. For each type of ami we want to create, the run_list must be specified in the yaml or else the instance is simply spun up and registered as an AMI. By default a spot instance is created for AMI generation but it can be overridden by specifying a command line parameter --nospot. This creates an on-demand instance for AMI generation, when sometimes we don't want to wait.

knife bs bootstrap

** Work in Progress **

Bootstraps an instance by installing Chef on a running machine and bringing it under the control of Chef server.

knife bs ebs

** Work in Progress **

Manage EBS volumes (create, attach, mount)

knife bs ebs delete

Delete/detach EBS volumes

Changelog