8-bit internet rodent

Gravatar of porcupo


| Comments


I’ve recently been putting a more thought into test-driven infrastructure. http://serverspec.org/
has been on my radar for a while. Based on RSpec in the ruby world, serverspec provides a DSL for
spec testing servers.

The DSL is pretty minimal, but conveniently extendable.

Why Test Driven Infrastructure?

Test driven infrastructure allows you to specify outcomes, rather than the procedures that are supposed to
produce those outcomes.

In the below example for nginx, we’ll have to consider what the final outcome should be.

Example: nginx

In ones typical sysadmin world, you might get a request along these lines:

install web server (now!!)

From here, we proceed to install nginx, enable service, and start daemon.[1]
We’re pretending chef, puppet, et al. do not exist, and we’re running this by hand or via some sweet bash scripts over ssh with a for loop.

>:3 sudo apt-get install nginx
>:3 sudo update-rc.d nginx enable
>:3 sudo service nginx start

Note that we’re assuming a lot from the requester here.

This is all well and good, and most likely will produce the desired (or at least assumed) outcome. But
let’s say this request comes up a lot. You have a script that installs everything without interaction.
Works great! So far every request from the dev team for an nginx install went as expected.

If you’ve ever closed a ticket at this point, you know there’s a 33% chance this will all end in tears.

  • The requester meant nginx hosted port port 80 with various virtual hosts when he said “nginx”
  • The requester is not in engineering, or is working on a side project, etc., that does not conform to
    any recognizable standard you have
  • The requester assumed they would have sudo access and could configure nginx themselves
  • and so on and so on

Another TDI benefit? get the answers to these questions from the requester, removing even more ambiguity
from the request.

  • What port should nginx run on?
  • what should be in sites-enabled?
  • Should this service start on boot?

…and whatever other outcomes they might like.

nginx spec

I’ll start out with a simple example, and then dig into the details of setting everything up.

# spec/httpd_spec.rb
require 'spec_helper'

describe package('nginx') do
  it { should be_installed }

describe service('nginx') do
  it { should be_enabled  }
  it { should be_running  }

describe port(80) do
  it { should be_listening }

describe file('/etc/nginx/sites-enabled/porcupo') do
  it { should be_file }
  its(:content) { should match /server_name porcupo.net \*\.porcupo.net localhost/ }

Pretty self explanatory! Basically you assert the following:

  • nginx is installed
  • nginx service is running
  • nginx service is enabled on boot
  • something should be listening on port 80 (we’ll assume that’s nginx here)
  • the /etc/nginx/sites-enabled/porcupo file has a line matching the regexp server_name porcupo.net \*\.porcupo.net localhost
# run spec test
>:3 rake spec:kaiju.porcupo.net
Package "nginx"
should be installed

Service "nginx"
should be enabled
should be running

Port "80"
should be listening

File "/etc/nginx/sites-enabled/porcupo"
should be file
should match /server_name porcupo.net \*\.porcupo.net localhost/

Finished in 1.28 seconds (files took 6.03 seconds to load)
6 examples, 0 failures

The nitty- gritty

There are some dependencies required for all this:

>:3 tree .
├── Gemfile
├── Gemfile.lock
├── Rakefile
└── spec
    ├── kaiju.porcupo.net
    │   ├── base_spec.rb
    │   └── httpd_spec.rb
    └── spec_helper.rb

2 directories, 6 files

spec_helper.rb contains the test-independent glue for making all of this happen. Here’s what I’m using
in this instance.

# spec_helper.rb
require 'serverspec'
require 'net/ssh'
require 'highline/import'

options = Net::SSH::Config.for(host)
options[:user] ||= Etc.getlogin

set :host,        options[:host_name] || host
set :ssh_options, options
set :backend, :ssh
set :disable_sudo, true
set :env, :LANG => 'en_US.UTF-8'

note on sudo

sudo is enabled by default, and you can have serverspec prompt for or read a password. This
can be spotty, as you won’t quite know exactly when it’s using sudo along the way.
Preferably, you would enable this per feature:

describe command('whoami'), :sudo => true do
  it { should return_stdout 'root' }

Your Rakefile automates the processes

# Rakefile
require 'rake'
require 'rspec/core/rake_task'

task :spec    => 'spec:all'
task :default => :spec

namespace :spec do
  targets = []
  Dir.glob('./spec/*').each do |dir|
    next unless File.directory?(dir)
    targets << File.basename(dir)

  task :all     => targets
  task :default => :all

  targets.each do |target|
    desc "Run serverspec tests to #{target}"
    RSpec::Core::RakeTask.new(target.to_sym) do |t|
      ENV['TARGET_HOST'] = target
      t.pattern = "spec/#{target}/*_spec.rb"

Here’s the Gemfile:

~~~ ruby Gemfile
source “https://rubygems.org”

gem “serverspec”
gem “rake”
gem “rspec”
gem “highline”

Example: base system

A simple example for basic Ubuntu setup

The hot thing to do with rspec now is to use expect over should. I’ll try that below.

# base_spec.rb
require 'spec_helper'

describe "Hostname" do
  this_host = host("porcupo.net")

  it "should resolve to" do
    expect(this_host.ipaddress).to eq ''

  it "should be reachable on ports 22, 80" do
    expect(this_host).to be_reachable.with(:port => 22)
    expect(this_host).to be_reachable.with(:port => 80)

  it "should NOT be reachable on ports 25, 1234" do
    expect(this_host).not_to be_reachable.with(:port => 25)
    expect(this_host).not_to be_reachable.with(:port => 1234)

describe "OS" do
  it "should be Ubuntu 14.04 x86_64" do
    expect(os[:family]).to match /^ubuntu$/
    expect(os[:release]).to match /^14.04$/
    expect(os[:arch]).to match /^x86_64$/

This is only the beginning! I’ll be sure to add anything interesting.

Additional ideas

  • Use chef, puppet, fabric, saltstack, etc. to push files and configuring, using serverspec to test.
  • Use md5 sum of config files when checking
  • Have failure tests (apache should not be installed, etc.)
  • functional tests (hit API URL and expect a correct answer)
  • a more complex, role-based database of hosts to test.
  • more packages and package configs

Code also available on github