Intelligent Vagrant and Ansible files

I use both Vagrant and Ansible to run and provision development virtual machines for testing work locally. This provides an easy to build environment as close to production as possible that all developers can easily create from the source code repository. A simple vagrant up and the associated Ansible scripts will handle all of the configuration and package installation for the VM.

This is unbelievably handy and it really helps to reduce the kind of bugs that are difficult to track down - “it works on my machine!”

Shared configuration

Recently, though I got to thinking about how the configuration is bound up in a rather unhelpful Vagrantfile, which is a Ruby script underneath in reality. The same configuration details need for the Vagrantfile will likely also be required by your provisioning scripts.

There are at least two ways to achieve this - each with their respective advantages and pitfalls. You can use a central file or pass the information as arguments to Ansible from the Vagrant provision commands. If you need to support machines that Ansible cannot run on then you’ll prefer the central configuration file as otherwise you need to pass the parameters in two locations. Using a bash script to support Windows machines is discussed further on.

Central configuration file

One way to work around this is to use a universal configuration file that both your provisioning scripts (Ansible, etc) and the Vagrantfile can read. The common thread between Ansible and Ruby (of course) is that they both parse YAML so a central config file is going to be the ticket. I am calling this file vagrant.yml and I have it sat at the same level as Vagrantfile in my projects.

In vagrant.yml you can have a structure like:

---
ip_address: 192.168.33.66
vm_name: example
server_domain: example.dev

From the Ruby script in Vagrantfile it is possible to parse the vagrant.yml configuration file and set the values against internal Vagrant options.

require 'yaml'
settings = YAML.load_file 'vagrant.yml'

Vagrant.configure("2") do |config|
  config.vm.network :private_network, ip: settings['ip_address']
end

You can also use these configuration details from Ansible project by loading it in a vars_files: directive. The variables will then become available in the global space. In the example code you can see the variables in use to define servername:.

---
- hosts: all
  sudo: true
  vars_files:
    - ../vagrant.yml
    - vars/common.yml
  vars:
    servername: "{{ server_domain }} www.{{ server_domain }} {{ ip_address }}"
    timezone: Europe/London
  roles:
    - init

Note that my Ansible configuration is in a subfolder hence the need to call the shared configuration with ../vagrant.yml.

Passed as arguments

Another way of having shared configuration between Vagrant and Ansible is to pass arguments from Vagrant into Ansible at provision time. This is done using the ansible API in your Vagrantfile and specifically the extra_vars property.

The sample code below illustrates how this might look in a simple Ansible backed Vagrant setup.

Vagrant.configure("2") do |config|
  ansible_inventory_dir = "ansible/hosts"

  config.vm.provision "ansible" do |ansible|
    ansible.playbook = "ansible/playbook.yml"
    ansible.inventory_path = "#{ansible_inventory_dir}/vagrant"
    ansible.limit = 'all'
    ansible.extra_vars = {
        vm_cores: cpus,
        vm_memory: mem,
        server_domain: servers['server_domain'],
        ip_address: servers['ip_address'],
        additional_server_domain_aliases: servers['additional_server_domain_aliases'],
        vm_user: settings['vm_user']
    }
  end
end

Just like the shared configuration these variables can be accessed in the global space of Ansible.

---
- hosts: all
  sudo: true
  vars_files:
    - vars/common.yml
  vars:
    servername: "{{ server_domain }} www.{{ server_domain }} {{ ip_address }}"
    timezone: Europe/London
  roles:
    - init

Dynamically create the Ansible inventory file

One aspect of projects that can be annoying to maintain or see committed into the project is the Ansible inventory file. Thankfully this can easily be automated from the Vagrantfile and the path dynamically set against Vagrant’s configuration.

In the code below the Ansible directory is set to a variable and then Ansible is set as the provisioning setup for Vagrant. This is all pretty much standard, but then the code moves onto handle the actual inventory file creation.

Vagrant.configure("2") do |config|
  ansible_inventory_dir = "ansible/hosts"

  config.vm.provision "ansible" do |ansible|
    ansible.playbook = "ansible/playbook.yml"
    ansible.inventory_path = "#{ansible_inventory_dir}/vagrant"
    ansible.limit = 'all'
  end

  # setup the ansible inventory file
  Dir.mkdir(ansible_inventory_dir) unless Dir.exist?(ansible_inventory_dir)
  File.open("#{ansible_inventory_dir}/vagrant" ,'w') do |f|
    f.write "[#{settings['vm_name']}]\n"
    f.write "#{settings['ip_address']}\n"
  end
end

It simply creates the directory if it doesn’t already exist and then opens the inventory file for writing whereupon it puts the machine name and IP address into the file. This is a simple way to save yourself a little work when creating new Ansible backed Vagrant projects.

Give the box all virtual cores and a quarter of the systems memory

Another tip I have picked up is from Stefan Wrobel’s article How to make Vagrant performance not suck. He suggests an automatic method for determining the number of CPU cores available on your host machine and then giving the Vagrant box access to all of them. To further increase performance you can also have the Vagrantfile calculate and assign a quarter of available host system memory.

The Ruby code to perform this is reasonably self explanatory and uses command line to establish the system resources.

Vagrant.configure("2") do |config|
  config.vm.provider :virtualbox do |v|
    v.name = settings['vm_name']

    # taken from http://www.stefanwrobel.com/how-to-make-vagrant-performance-not-suck#toc_1
    # assigns all available CPU cores and 1/4 of the host systems memory to the vm
    host = RbConfig::CONFIG['host_os']

    # Give VM 1/4 system memory & access to all cpu cores on the host
    if host =~ /darwin/
      cpus = `sysctl -n hw.ncpu`.to_i
      # sysctl returns Bytes and we need to convert to MB
      mem = `sysctl -n hw.memsize`.to_i / 1024 / 1024 / 4
    elsif host =~ /linux/
      cpus = `nproc`.to_i
      # meminfo shows KB and we need to convert to MB
      mem = `grep 'MemTotal' /proc/meminfo | sed -e 's/MemTotal://' -e 's/ kB//'`.to_i / 1024 / 4
    else # sorry Windows folks, I can't help you
      cpus = 2
      mem = 1024
    end

    v.customize ["modifyvm", :id, "--memory", mem]
    v.customize ["modifyvm", :id, "--cpus", cpus]
  end
end

Finally, using v.customize the values are set against the Vagrant configuration.

Provision without Ansible installed

It is possible to provision a Vagrant box on a system that doesn’t have Ansible installed by using a small shell script. This is the approach that phansible.com has taken and with some slight modification I have adopted.

The first step is two write some Ruby in the Vagrantfile that determines if Ansible is installed in the user’s path. If it is not then we should use the shell script as the provisioner.

# Check to determine whether we're on a windows or linux/os-x host,
# later on we use this to launch ansible in the supported way
# source: https://stackoverflow.com/questions/2108727/which-in-ruby-checking-if-program-exists-in-path-from-ruby
def which(cmd)
  exts = ENV['PATHEXT'] ? ENV['PATHEXT'].split(';') : ['']
  ENV['PATH'].split(File::PATH_SEPARATOR).each do |path|
    exts.each { |ext|
      exe = File.join(path, "#{cmd}#{ext}")
      return exe if File.executable? exe
    }
  end
  return nil
end

Vagrant.configure("2") do |config|
  if which('ansible-playbook')
    config.vm.provision "ansible" do |ansible|
      ansible.playbook = "ansible/playbook.yml"
      ansible.inventory_path = "#{ansible_inventory_dir}/vagrant"
      ansible.limit = 'all'
    end
  else
    config.vm.provision :shell, path: "ansible/windows.sh"
  end
end

This shell script will handle the base setup of the box before Ansible can run - so installing Ansible dependencies, then Ansible, setup SSH keys, link the Ansible inventory and finally running the playbook locally on the box. This script targets Ubuntu/Debian Vagrant boxes, but it could be adapted for other POSIX systems.

#!/usr/bin/env bash
sudo apt-get update
sudo apt-get install -y python-software-properties
sudo add-apt-repository -y ppa:ansible/ansible
sudo apt-get update
sudo apt-get install -y ansible
cp /vagrant/ansible/hosts/vagrant /etc/ansible/hosts -f
chmod 666 /etc/ansible/hosts
cat /vagrant/ansible/files/authorized_keys >> /home/vagrant/.ssh/authorized_keys
sudo ansible-playbook /vagrant/ansible/playbook.yml --connection=local

Some non-config related tips

Handy plugins

In most of the configurations I prepare I also make use of vagrant-cachier and vagrant-hostsupdater. The former aims to prevent duplicate package downloads for a given Vagrant box so that subsequent provisions are faster. Hostsupdater will automatically add the IP address and host name of the project to your hosts file so that you don’t have to. Both of their configurations are pretty straight forward and dealt with on their respective project pages so I won’t duplicate effort here.

Moving the Vagrant and VirtualBox VMs to an external HDD

It is rare for a computer not to contain an SSD drive of some sort these days and they’re often set as the primary drive for the machine. This means that both Vagrant and Ansible will be storing their large VMs and files on your limited capacity (unless you’re ultra lucky) SSD. To free up space a USB 3.0 external HDD can really help without slowing down performance too much.

If you’ve already got a few boxes and/or VirtualBox VMs setup then this process can take some time as you will be copying large files - you may want to leave it over night rather than a cup of coffee! It is a pretty simple process though.

For the sake of this example I am going to assume the external HDD is mounted at /media/simon/mydrive/ and you’ll need to substitute this for your drive as you follow along.

The first step is to move the Vagrant home directory to a new location on your external hard drive.

rsync -av ~/.vagrant.d/ /media/simon/mydrive/.vagrant.d/
echo 'export VAGRANT_HOME="/media/simon/mydrive/.vagrant.d"' >> ~/.bash_profile

We’ve also added the new location to your .bash_profile so that it will automatically available when you boot your machine.

With that out of the way the bulk of the copying is still to come! Open the VirtualBox application and then in Preferences set the Default Machine Folder to /media/simon/mydrive/VirtualBox VMs. Now to move your current VMs to the new location.

rsync -av ~/VirtualBox VMs/ /media/simon/mydrive/VirtualBox VMs/

The next step maybe unnecessary, but you can then re-open VirtualBox and remove any VMs that are showing as inaccessible. To re-add them it you can simply run

find /media/simon/mydrive/VirtualBox VMs/ -iname *.vbox -exec vboxmanage registervm '{}' \;

Finally, you can now move your actual project directories to the external harddrive too and they’ll use the new locations for storage and access. They could also stay where they are as they are only small so up to you!