Automated Deployments

Posted On: 2018-11-05

By Mark

Over the past week, I have gotten automated deployments working for this site. For those unfamiliar with the term, an automated deployment is a system that allows you to make changes in a safe environment (such as your local computer), and then, once you're happy with them, click a button to have them be automatically deployed (that is, copied to/installed on) to any number of sites. Often, an automated deployment will also run automated tests, to verify that the site works correctly after deployment. My current tests are a bit limited (makes sure only a couple of the pages are working), but I have a good framework on which to easily continue improving it (in the meantime, it's running with metaphorical training wheels on: after a deployment to a non-production server, I manually look it over and "approve" it to be deployed to the production site.) A delightful benefit of this approach is that updates to the site (no matter how dramatic) only have a few seconds of downtime (as opposed to a manual deployment, which could leave the site offline for over an hour during major updates.) For those that are curious or looking into doing it themselves, I'll detail how I was able to accomplish this below:

Prior to any automation I had a few things in place already:

I used Git as a version control system. (I particularly like that Git can work without a remote server, as it allows me to start versioning things immediately, and decide later if it is worth versioning remotely.)
I used Azure Dev Ops (previously called VSTS but was recently rebranded) as a remote version control and automated build system.
The particular host I use provides the ability to store code in a Git repository and run a (pre-written) script to deploy that to the server. Unfortunately, these deployments require the Git resource to be structured in a very specific way, so I can't use it as a remote on my existing Git setup.
I don't have user uploads or a database for my site. This makes deployments substantially simpler than they otherwise would be.

Knowing what I had already, I came up with a simple plan for how to deploy it: I would take the output of my automated builds, deploy that to the remote Git that my host provides and then use ssh to run the installation script. Having limited experience working with Git (I usually use a GUI for it so I hadn't learned the command line commands) and absolutely no experience working with SSH, this simple plan ended up taking the better part of the week, and turned into a pretty great learning experience.

Although at first I wanted to work on the Git side of things (and save the SSH for later) it became clear that I actually needed to understand SSH as a prerequisite for using Git here. This was due to the fact that Git requires credentials, and using SSH (with certificates) is the recommended way of providing those credentials in any kind of automated setup. It turns out that SSH uses private+public key pairs for its authentication, so I already understood the basics for how they worked. Interestingly, both the Host and the Client have to provide public keys to each-other, so I ended both having to generate my own private+public key pair as well as get the correct public key for the server I was trying to connect to. After getting the correct keys, I was able to use the "Install SSH Key" task in Azure Dev Ops to load them so that later tasks could use them. The "Known Hosts Entry" needed the pairing of server name and public key, while the rest of the fields are related to the client's SSH credentials. I also had to setup the host side of things to know it could trust my public key.

Once I had the authentication working with my host's Git remote, I set about getting the release agent to update it with the build output. The first task will get the git history, but without any of the files. To achieve this, I used clone and then manually deleted the files (I tried a number of other approaches, but none of them worked. I expect there must be a better way, so I will likely come back to it at a later time.) Since cloning also downloads the files, and I don't want the existing files (keeping existing files would leave old files on the server, even if I deleted them from the source) I had to delete all the files (besides the .Git folder). For simplicity, I am using the variable $(StagingPath) to designate the folder that I will be using for all the work. Here's the script I am using for this:

git clone --separate-git-dir "gitfolder" git+ssh://$(GitRemote)/$(gitName) "$(StagingPath)"
cd "$(StagingPath)"
find -not -name ".git" -delete

Once the $(StagingPath) is initialized with Git, I can go about preparing and committing the code. I use a simple Extract Files to extract the build artifact (a zip of the exactly the code needed to run the site) into the $(StagingPath). After that, I use another Bash task to update the repository on the remote host. This requires a combination of telling git to add the new files and delete any that are missing, committing all that to the local git, and then pushing the changes to the remote. I also configure the git identity at this point (since it's required for any commits). Here's the code for this section:

 cd "$(StagingPath)"
git add -A
git config user.name "Automated Deployment"
git config user.email "bogus@example.com"
git commit -m "Automated deployment of $(Release.Artifacts.HedbergGamesBuild.BuildNumber)"
git push

Finally, once all that is done, I can use ssh to execute the actual code deployment command: ssh $(GitRemote) deploy $(gitName). I use a separate task for this so that it's easy to disable (if, for example, I want to do a dry-run of a change to the release.) I can't really say that much about this, since the actual deploy script is something my host provides and maintains, but it seems to work, and it deploys much faster (especially compared to the trans-continental data transfers I used to do.)

I also run a couple automated tests as well, but they are pretty rudimentary. They make use of the curl operation to call pages, and only pipe failures to stderr (I setup the build to fail on any stderr in this task). The curl command is probably going to change/get improved, so I put it in a variable so that I can easily iterate on it. Here's the complete curl command: curl --retry 5 --fail --silent --show-error --head and the actual bash task uses it to automate testing a few pages:

$(SmokeTestBashCommand) http://$(Domain)/
$(SmokeTestBashCommand) http://$(Domain)/about
$(SmokeTestBashCommand) http://$(Domain)/blog
$(SmokeTestBashCommand) http://$(Domain)/prototypes

Now that I've got all this setup, it runs regularly with every commit I push to my remote repository. I've been really happy with the results, since it's allowed me to make big changes safely (I've made several major updates to the underlying code since adding this, including a couple security improvements.) I am optimistic that this will give me a good framework to make high impact changes at low cost (time/effort), so that I can keep implementing improvements for the site.

Hopefully this window into managing a website is interesting and/or useful. If you have any thoughts or feedback about any of this, please let me know.