Here at Shutl HQ we are big fans of Chef, using it to provision our boxes from bare OS up to running application. Chef handles all of our application deployments, which happen many times a day across all of our environments. However, while Chef is excellent at managing the configuration of each machine, it is lacking when it comes to orchestration of deployments across clusters of machines.
This post is about how we addressed our release orchestration headaches with Rundeck, and built a fully automated Continuous Delivery pipeline using Rundeck to integrate our CI platform, releases, and Chef together.
Our release flow
Applications are deployed from a Git repo. Chef is configured to checkout the code, run any necessary build steps and database migrations before deploying the new code.
In staging, we always deploy master from Git. When code is pushed to master, a build is triggered in Semaphore, our hosted CI platform. When the build passes, we can log into the staging server and run the Chef client. This will check out and deploy master. This is the simplest workflow case, but there is a problem. Someone needs to watch for the build to turn green and then log into each staging machine in turn and run the deployment.
Rundeck is an open source tool for running jobs on remote machines. It can run predefined or ad-hoc jobs, and it can run many jobs concurrently. It is Java based, and can be easily extended with plugins to perform tasks such as posting a notification in your Slack channel when a job completes. It has a very nice Web UI as well as a fully featured API.
Rundeck executes remote tasks via ssh, as the “rundeck” user by default. So no special agent needs to be running on each machine, you simply need to ensure that the rundeck user has the required permissions to perform the task. Predefined jobs can be configured with options which are passed from the user at run time. This allows jobs to be quite flexible in the tasks they perform. Each job consists of a sequence of steps which can also include other Rundeck jobs.
How we use it
We created a job called “Deploy Application”, which takes three options:
- Name of the application to deploy
- Environment to deploy in
- Version of the app to deploy
The revision is optional, and does not need to be supplied for staging deployments, where we always deploy master. Rundeck uses the application and environment options to find the nodes (machines) to deploy to. In order to find the nodes and tie Rundeck into Chef, we use rundeck-chef, a simple Sinatra app which talks to the Chef API and presents the list of nodes to Rundeck. The Chef environment is visible on each node, and each Chef role becomes a tag on the nodes in Rundeck. We have a role for each application, which allows Rundeck to find the target nodes by searching for the tag which matches the application name.
When it has found the nodes, Rundeck simply executes “sudo chef-client” on them to perform the deployment.
Continuous Delivery (in staging)
In our staging environment, we take this a step further and do fully automated deployments. When the build passes in the CI, a simple script is run which uses the Rundeck API to trigger the deployment:
#!/bin/bash APP=admin_portal ENVIRONMENT=staging JOB_ID=02abce51-a001-46fc-9c4f-080a12e18cb1 TOKEN=XXXX curl --data-urlencode "argString=-application $APP -environment $ENVIRONMENT" "https://RUNDECK_HOST/api/10/job/$JOB_ID/run?authtoken=$TOKEN"
This allows us to go from code push to having the code running in staging. To let us know when deployments happen, we wrote a simple Slack notifier plugin for Rundeck which posts a message in a Slack channel.
In production, we trigger the Rundeck job manually. However, things are slightly more complicated because we are deploying a tagged release from Git. This means we need a way to pass the tag from Rundeck to Chef.
To solve this, we wrote a simple Sinatra app which we call Deployment Service. This has a simple API which can be used to store and retrieve the tag for each application, in each environment. We then wrote a java plugin for Rundeck to post the tag to the Deployment Service. When chef-client runs on the target node, it calls the Deployment Service API to get the tag to deploy.
Bonus Points – run deployments from the command line
Although Rundeck has a very nice Web UI, sometimes a command line tool is more convenient. So we wrote a knife plugin to trigger the Rundeck job (knife is a command-line client for Chef). This has the advantage of being able to trigger concurrent deployments all our production-like environments with one command:
$ bundle exec knife roller deploy -A admin_portal -E production -T 2.98.14 Environment: production Running job 4553 - https://rundeck.shutl.co.uk/execution/follow/4553 Environment: production-us-west Running job 4554 - https://rundeck.shutl.co.uk/execution/follow/4554 Environment: sandbox-us-west Running job 4555 - https://rundeck.shutl.co.uk/execution/follow/4555
This uses an HTTP call to the Rundeck API, just like the CI deploy script mentioned above but via Ruby.
(CC Image by Rubin)