Incite Code

Blog of a Rails Developer

Setting Up Chef-solo With Berkshelf

In the early days of chef, there was only what’s now referred to as the ‘monolithic chef repo’ pattern, in which your chef-repo was a single source controlled repository, and contained all the chef cookbooks necessary to perform all the tasks you might want to use. Several years back, berkshelf came on the scene, as a dependency manager for cookbooks, in a very similar pattern as language top level dependency managers such as bundler for ruby, and npm for node. This has in my opinion been a huge help for cookbook management, as without it, it’s far to easy to get stuck not upgrading cookbooks, being unable to remember where you sourced some less common cookbooks from, and generally falling into a lack of maintainability.

Using berkshelf

To get started using berkshelf, newcomers are recommened to install the chef development kit which contains berkshelf, and other tools to get you up and running – those familiar with ruby may wish to install the gem directly, which lets you stay on the latest version. To use berkshelf, then create a Berksfile in your repository root, and place entries corresponding to cookbooks that you wish to use in it. There are 3 main source types supported:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# Berksfile
source "https://supermarket.chef.io" # Chef and community cookbooks are published to here

# Install the latest compatible version of the apache2 cookbook from the official repository
cookbook "apache2"

# Install the cookbook from the official repository matching the given version constraints.
cookbook "apache2", "~> 2.0"

# Install the latest unstable version from a git repository
cookbook "mysql", git: "git@github.com:chef-cookbooks/mysql.git"

# Install the cookbook at a specific commit to the repostitory
cookbook "mysql", git: "git@github.com:chef-cookbooks/mysql.git", ref: "47645e1974fa6ea9e774b1f6595a015f86454705"

# Install the cookbook from the given path on disk
cookbook "myapp", path: "site-cookbooks/myapp"

You can then run berks install, and the cookbooks, along with their dependencies are resolved, and downloaded to ~/.berkshelf. A Berksfile.lock will also be created detailing the exact versions used. As is usually the case with 3rd party dependencies, you should check both these files into source control for reproducability.

Monolithic vs. Separate cookbook repositories

Here-in lies somewhat of a trade-off in having a well polished and maintained setup, and a beginner friendly setup. The traditional monolithic setup is far easier for beginners to understand – all code that is run as part of your repository is a directory wide search away, and can generally be changed right where it sits. Of course, this is what causes the bulk of the issues in long-term maintainability in the first place. The difficulty for beginners in working with berkshelf, is in knowing where the third party code lies, and how to add and manage cookbook usage. A well written README in your chef-repo root in how to perform basic tasks expected goes a long way to helping your co-workers who perhaps aren’t touching the chef setup that frequently.

I generally chose to take a somewhat hybrid approach, and use two cookbook sources. A site-cookbooks directory in the chef repo root, as well as upstream dependencies via berkshelf. Third party cookbooks providing community recipes (e.g. apt, mysql, apache, nginx etc) are likely to be the bulk of this, and are populated in the berksfile. Any cookbooks which I feel are suitable to make in a generic fashion, and release as open source (e.g. unattended-upgrades) or complicated enough to be maintained as a separate private cookbook (such as internal auditing and logging tools) – I create as a separate cookbook repository, and manage them as their own software project, and add to the berksfile as a git source. The site-cookbooks folder I leave for private in-house recipes that I expect developers to have to be able to modify frequently, or those that are pretty simple in nature. Given that berkshelf manages the entire dependency chain, those cookbooks in site-cookbooks should also be added to the berksfile via the path option as shown above. I’ve found that this is the best middle ground to the multi-repository flow that berkshelf heavily suggests and promotes, and keeping necessary items accessible.

chef-solo deployment

The final piece of the puzzle – is that as chef-solo runs your chef recipes locally on the machine you are setting up – there isn’t actually an out of the box solution for how you choose to deploy the chef-repo you develop on your local workstation onto the machine you wish to configure. You can vendor your berkshelf cookbooks, and chef-solo can take a URL containing a zip archive – some places use this to CI on a custom jenkins setup, with passing builds automatically vendored and published into an artifact store. We’ll look at a much simpler method though, whick is the knife-solo plugin. This plugin is designed to take care of the chef-solo deployment, and while simply copying a folder wouldn’t be much to write home about, it’s the integration with the rest of the ecosystem that really makes it valuable. As noted previously, with berkshelf, the actual cookbook files live outsite your repo folder, and knife-solo integrates with berkshelf to ensure those dependencies are resolved, and copied across as well.

Testing

Given berkshelf’s push to managing your cookbooks as individual software projects, it’s recommended that those cookbooks themselves should be tested with combinations of chef-spec and test-kitchen as appropriate – of course, I would still recommend testing your entire run-list as a single unit to ensure there isn’t any interference and things work together as expected. The ‘deployment’ problem of resolving your berkshelf cookbooks and installing them on the test stack still remains. Vagrant + the vagrant-berkshelf plugin are a pretty basic solution to this, and the first tool I’d recommend to look at for this. I’ll be aiming to cover this in more detail in a future post, as I’ve disliked the default behaviour of vagrant-berkshelf to the point of uninstalling it within an hour everytime I’ve tried it, and recently come up with an alternate custom solution.

Comments