April 19, 2015

Automate everything with CircleCI

Software has many moving parts. Even a small open source project expects you to maintain the codebase, keep the documentation in sync, push snapshots and releases, update the changelog and maybe showcase a demo application. All of this is a distraction from your original scope: writing code that matters. A solution is to outsource as much as possible to continuous integrations services. When asked for a snapshot of my library I want to answer: "The last commit is always available as a snapshot". When a snapshot version looks good I want promote a release from my smartphone while on the move. The less commands I run from my laptop, the happier I am.

You won't believe how much time I wasted on this

Obligatory xkcd quote

Automating requires a lot of trial and error, and there's always room for improvement. The following is my current setup using Leiningen and CircleCI. Hopefully it will give other devs an easy recipe to start automating their builds while avoiding the most common pitfalls.

So point your browser at circleci.com and let's get started.

CircleCI, our new friend

This post is mainly targeted at open source project maintainers. If you're rolling your own CI service (Jeknkins, TeamCity) for you own private repo, some of the things I mention won't apply to you. You can just check in your passwords (wait, don't!) or set up the CI machines with all the credentials you need (yeah, that's better). But for my free, open source projects I'd like to use a free, hosted CI server. I've been a long time Travis user but I recently switched to CircleCI because:

  • CircleCI is written in Clojure. On top of that it's made by a fantastic team that is doing good for the Clojure community (check out their blog, Typed Clojure support and open source frontend!)
  • You can SSH into their remote machines. Oh, the joy! A quick ssh into the suspicious build, edit some files and retry the failing command! Oh, so much time saved! Now, if only you could run emacs on those machines...
  • It's very well documented and has a clean, simple approach to configuring your build environment.

To be completely fair, here are the things I think they could do better:

  • The UX needs a bit of love. As a comparison in Travis in each screen you have access to exactly the actions you need. Navigating through projects and settings is clean and effective. Circle can be confusing and cluttered at times.
  • There's no way to remove or download the log files. That is very annoying if you -cough- accidentally leak sensitive information. You wil need to send a polite email to their customer service that potentially takes a lot of time to be actionable (at least for us on the European time zone).
  • Pull request builds can be activated, but they do not hide secure environment variables. It would make more sense to me to enable pull request by default and add a switch to enable/disable secure variables for them. As they stand now they are unusable.

CircleCI UI

CircleCI interface

A continuous integration server cannot perform many interesting tasks if it doesn't have the credentials to do so. CircleCI allows you to set up hidden environmental variables for each project. This way your scripts can access passwords, api tokens, email addresses and other sensible information. I found this approach really hard scale if you have over 4-5 projects: I like to change my credentials from time to time (and so should you) so keeping everything is sync is too much work.

In a better world Circle would provide you a way to setup a configuration that you'd like to run for every machine. This configuration would be stored securely, never displayed in the logs, and only executed when you're the author of the commit. We don't live in that world yet, but we could achieve something pretty similar.

All the common configuration for our CI machines can be extracted and pushed to a private repo. Assuming you have private repos on Github account, before running your build steps you just tell Circle to execute:

- git clone {{CONFIG_REPO_URL}} ~/dotfiles && . ~/dotfiles/init.sh

Note the dot preprendend before running init.sh: this makes any variable you export inside your script globaly accessible.

This is what my init script looks like:

echo "Init env..."
git config --global user.name "NAME"
git config --global user.email "EMAIL"
mv ~/dotfiles/credentials.clj.gpg ~/.lein/credentials.clj.gpg
echo -e "use-agent\ndefault-key EMAIL\npassphrase PASSPHRASE" >> ~/.gnupg/gpg.conf
gpg --allow-secret-key-import --quiet --import  ~/dotfiles/secring.gpg
echo "Done."

This repo contains the files init.sh, credentials.clj.gpg and secring.gpg. After setting up git, the init script imports the clojars credentials and the gpg key so the remote machine can perfom the same actions as my local machine. It is also adding the passphrase and the default identity to gpg.conf so Leiningen would not complain when it tries to sign things.

If you don't have a Github private repo (like me) you can always resort to Bitbucket. Since CircleCI doesn't integrate with Bitbucket there's no easy way to just download your private repo from it. You have to use a Bitbucket API token, which requires you to set up a team, which means you need to push your repo under your team name. Everything is explained here and it's not as complicated as it sounds. Once you have a token you can clone your repo using a url like https://TOKEN@bitbucket.org/TEAM/REPO.git. Protect this url with great care otherwise anybody can access all your credentials. Add this repo url as a Circle hidden environment variable so it is accessible in your build script:

- git clone $BITBUCKET_URL ~/dotfiles && . ~/dotfiles/init.sh

This script is part of the circle.yml file that sits at the root of project and tells Circle how build it. You can execute any bash command but seriously, who wants to write build steps in bash? It's much more fun to use Clojure, of course. To do so we're gonna call a Leiningen tasks that performs our deploys after the tests are successfully completed. Here's the entire circle.yml that I use for every project.

test:
  pre:
   - git clone $DOTFILES ~/dotfiles && . ~/dotfiles/init.sh
  post:
    - lein circle

Leiningen, our trusted old pal

I know the cool kids are moving to boot (and with good reasons), but Leiningen remains strictly superior when it comes to deploy logic. I'm looking forward to see if this changes in the near future.

Not everybody might that Leiningen can do a fairly good job as a scripting tool. If a .lein-classpath file is added at the root of your project and it contains the path to a folder, let's say build, then under build/leiningen/circle.clj you can write a custom circle task that performs any build logic. Here are some examples.

Automate snapshots and releases deploy

(ns leiningen.circle
  (:require [leiningen
             [core.eval :as eval]
             [release :as release]
             [deploy :as deploy]]))

(defn env [s]
  (System/getenv s))

(defn circle [project & args]
  (let [branch (env "CIRCLE_BRANCH")]
    (condp re-find branch
      #"master"
      (deploy/deploy project "clojars")

      #"(?i)release"
      (do
        (eval/sh "git" "reset" "--hard" "origin/master")
        (release/release project)
        (eval/sh "git" "push" "origin" "--delete" branch)))))

This is the basic task for most of my projects. Every push to master triggers a snapshot release to clojars. When I want to cut a release I go to Github and creatae a new branch called 'Release'. Boom! Circle will build that branch and the lein task will switch to master, deploy the release, and delete the temporary branch. You can control the actions performed by the release task adding these lines to your project.clj:

:release-tasks [["vcs" "assert-committed"]
                ["change" "version" "leiningen.release/bump-version" "release"]
                ["vcs" "commit"]
                ["vcs" "tag" "v"]
                ["deploy" "clojars"]
                ["change" "version" "leiningen.release/bump-version"]
                ["vcs" "commit"]
                ["vcs" "push"]]
Since gpg and clojars credentials are available on the CI machine, artefacts signing and deploy should work seamlessly.

Automate gh-pages documentation and heroku deploy

Github pages are a great way to host HTML documentation for your project. The output generated by the great codox plugin can be pushed to a gh-pages branch so it's readily available online (and versioned!).

To syncronise a folder in your project structure (in this case doc) to a remote branch (in this case gh-pages) we can add another little utility under build/leiningen/rsync.clj:

(ns leiningen.rsync
  (:require [clojure.string :as str]
            [leiningen.core [eval :as eval]])
  (:import [java.nio.file Files]
           [java.nio.file.attribute FileAttribute]))

(defn tmp-dir []
  (.toString (Files/createTempDirectory nil (into-array FileAttribute []))))

(defn rsync [project dir branch]
  (let [tmp-dir (tmp-dir)
        msg (with-out-str (eval/sh "git" "log" "-1" "--pretty=%B"))
        url (-> (eval/sh "git" "config" "--get" "remote.origin.url")
                with-out-str (str/replace "\n" ""))]
    (eval/sh "git" "clone" "-b" branch url tmp-dir)
    (eval/sh "rsync" "-a" "--exclude=checkouts" dir tmp-dir)
    (binding [eval/*dir* tmp-dir]
      (eval/sh "git" "add" "--all")
      (eval/sh "git" "commit" "-m" msg)
      (eval/sh "git" "push" "origin" branch))))

By no means perfect, this task still gets the job done by checking the remote branch in a temporary directory, adding the generated documentations to it, commit and push.

Let's add it to our release tasks just after the doc generation:

:release-tasks [["vcs" "assert-committed"]
                ["change" "version" "leiningen.release/bump-version" "release"]
                ["doc"]                     ;; <-
                ["rsync" "doc/" "gh-pages"] ;; <-
                ["vcs" "commit"]
                ["vcs" "tag" "v"]
                ["deploy" "clojars"]
                ["change" "version" "leiningen.release/bump-version"]
                ["vcs" "commit"]
                ["vcs" "push"]]

And voilĂ , documentation is taken care of.

We can reuse the same task to push a demo application (in my case under a directory called "sample") to a branch called "heroku". When the heroku branch gets built, the demo application will be sent to Heroku.

Simply add an rsync call to your master branch build:


;; ...

(defn circle [project & args]
  (let [branch (env "CIRCLE_BRANCH")]
    (condp re-find branch
      #"master"
      (do
        (deploy/deploy project "clojars")
        (rsync/rsync "sample/" "heroku")) ;; <-

      ;; ...
      )))

Done! Congratulations, there's nothing for you left to do.

Tropicana

Now sit back, relax and enjoy your automated setup

Do you build better?

You can look at the full configuration at pedestal-swagger. I've used the same strategy to build a cljx project (in linked), a multiproject with lein sub (in icepick) and a Cryogen static website (in frankiesardo.github.io). If you'd like to share or discuss ideas on how can we make our open source builds better, leave a comment or ping me on twitter @frankiesardo.

Tags: continuous integration clojure leiningen circleci