Rob SherlingHi! In this blog post, I'll show you in detail how to create daily/nightly builds in CircleCI with AWS Lambda, and take the results of those builds and create a Github pull request with the changes. Everything in this tutorial is 100% free.
For this project, I am assuming that you already have your project on CircleCI and Github. If you don't use Github, the Lambda and CircleCI parts will still be useful to you, just ignore the part at the very end where we create pull requests. If you don't have CircleCI, this project unfortunately has almost nothing for you unless you want to see an example of what a Github API request looks like.
CAVEAT: In the bash script, we use the line git add apples.txt to add our newest files. I would recommend writing git add <whatever your files are expected to be> instead of some catch all expression like git add .. For example, if you ran bundle update, I would recommend running only git add Gemfile.lock. This is because CircleCI can mess with your repo in crazy ways when it sets up for things. Example: Before it gets to your code, it may run something like bundle install to get all of your project dependencies. This means that you will suddenly have tons of files related to gems in vendor/bundle, and if you didn't gitignore that file, you are looking at a huge diff. It can also edit database.yml, so there's that.
Along the way, I'll be explaining gotchas or odd lines of code. Let's begin.
The way that nightly builds work is that first, we'll make a bash script to run all the code we want to execute on the build machine (in this case, make a file). This could be anything from rails t to bundle update to a bash script that checks the weather around your Singapore server (if you write that, please show me, I would be super interested). The end of that script will have us push any changes we made to Github as a pull request.
Then, we'll put that script in our CircleCI File to make it activate when we trigger our builds via API. This is relatively simple, a few lines.
Finally, we'll build a lambda and slap a cron job on there to make it proc our build at whatever time and frequency you please. Note: Lambda cron is limited to a rate of no greater than 1/minute (source).
First, the bash script. Create a file with a name that describes what the code will do. I wrote a script called bundle_update.sh, because we're gonna be updating bundles. Don't forget to chmod +x !
# First check out the branch you want to work code magic on. This is our base.
git checkout 'master'
# Force our branch to be in alignment with remote.
git reset --hard origin/master
# Make sure the commit is made with the correct credentials
git config --global user.email "<Your email here>"
git config --global user.name "circle-ci"
# Check to see if our work branch exists - $? command will return 0 if branch exists
# of course, this branch name could be anything you like, it's just important that it is unique to this script.
git rev-parse --verify circle_ci/make_apples
# If we have no branch, make one. Else, switch to it.
if ! [ 0 == $? ]; then
git checkout -b 'circle_ci/make_apples'
else
git checkout 'circle_ci/make_apples'
fi
# Run your custom code here. We'll just make a file for ease of use.
# This code can be anything you want to do, bundle update, nginx ip configs, etc.
touch apples.txt
# If we have changes, make a PR for those changes
if [[ `git status --porcelain` ]]; then
# Ensure that our branch is correct. The last thing you want is to force push into the wrong branch
if [ `git rev-parse --abbrev-ref HEAD` == 'circle_ci/make_apples' ]; then
git add 'apples.txt'
git commit -m 'My commit message here'
# -f Necessary to possibly overwrite any existing remote branch code and force this to be correct
# -u sets upstream for circleci
git push -f -u origin circle_ci/make_apples
# Head = Code to merge, base = branch to merge into
# $GITHUB_ACCESS_TOKEN is an environment variable we set on CircleCI, but you must set it on your local machine if you want to test there as well
# Example Github URL: https://api.github.com/repos/Rob117/circletest/pulls?access_token=$GITHUB_ACCESS_TOKEN
curl -H "Content-Type: application/json" -X POST -d '{"title":"Title Of Pull Request Here","body":"Body of pull request here", "head": "circle_ci/make_apples", "base":"master"}' https://api.github.com/repos/<Organization or User Name>/<Project Name>/pulls?access_token=$GITHUB_ACCESS_TOKEN
fi
fi
Before I explain the gotchas of the above code, go onto Github and create a token. That token should have the repo checkbox ticked so it has Full control of private repositories. Save that token. Switch over to CircleCI, and on your project settings page under Environment Variables, create a variable with the name GITHUB_ACCESS_TOKEN and a value of whatever your token string was.
Also ensure that your project has ssh permissions for Github set as well, or we can't push code from within our scripts. You can check by going to Checkout SSH Keys in the project settings and ensuring that you have a key that is permitted to ssh to your git repo there. If not (i.e., you don't have a user key there), click add user key and authorize the application to use your key. After authorizing, click on the 'create and add user key' button to get your ssh key on the machines so they can push code.
If you want to test the code on your local machine, go ahead and export GITHUB_ACCESS_TOKEN=<string here>.
Okay, now that we can actually run this code, let's break it down.
Most of the comments explain the code clearly. Remember to replace your email in the git config section, and your Github API URL organziation name and project name in the last line. The gotchas are:
We have to reset our base branch to whatever it is on remote with git reset --hard origin/develop. The reason for this is because as a byproduct of using CircleCI, a lot of things happen to our build machine repository before we execute a single line of code, and they could taint the final results. Syncing develop with its remote counterpart ensures that we have a pristine copy, so it's just good practice anyway.
These two lines:
git rev-parse --verify circle_ci/make_apples
if ! [ 0 == $? ]; then
The first line checks to see if the git branch exists. If it does, it will show 0 when you run the command $?, else it will show something else. This tells us if the branch exists so we know whether to make one or not.
if [[ `git status --porcelain` ]];
Simply runs git status and outputs nothing if there is no change. If this outputs text, we want to make a pull request.
if [ `git rev-parse --abbrev-ref HEAD` == 'circle_ci/make_apples' ]; then
Checks if our branch was actually created correctly. We do this so that we are sure we aren't going to force push to the wrong branch.
curl -H "Content-Type: application/json" -X POST -d '{"title":"Title Of Pull Request Here","body":"Body of pull request here", "head": "circle_ci/make_apples", "base":"master"}' https://api.github.com/repos/Rob117/circletest/pulls?access_token=$GITHUB_ACCESS_TOKEN
This is the magic. Base is the branch we started with in the beginning, the one that we want to reflect our code changes post-merge. Head is the branch we created in this script to do all of our work. We take the changes and make a pull request using our github access token that is stored in our environment variable.
Okay! We've broken down the bash script. Remember- just change the name of the base branch, created branch, and the file creation part to use the script in your own project. Running this script on your local environment - assuming you set your environment GITHUB_ACCESS_TOKEN var - should work smoothly and create a pull request on github.
Next, let's edit our circle CI. It's just a few lines of code added to your normal circle.yml:
# Other Circle.yml code
test:
post:
- >
if [ -n "${RUN_NIGHTLY_BUILD}" ]; then
# If you put the script in your scripts folder, the path is ./scripts/<name>
./bundle_update.sh
# As many scripts as you want here
fi
# More circle CI code
This code says 'if we get a post request that has variable called RUN_NIGHTLY_BUILD, execute the following scripts'.
That's it! Push this code to the branch that you want to call it from. For example, maybe you want to keep all of your scripts in your builds branch. Simply merge the above script and yml file into a branch called builds, and push that branch. You're all set to trigger your builds! You could also use a staging branch, develop branch, etc.
SUPER IMPORTANT NOTE: If you have more than one script set to run and you only store the scripts in a single branch - say, builds - and you change branches within a script, you MUST run git checkout builds between script runs / at the end of each script. If you don't, Circle CI may fail to run all your scripts because you'll have changed to a branch where they don't exist!
Okay, next let's set up our lambda and be done with this.
First, hop on over to Circle CI, go to project settings, then go to API Permissions, then click Create Token. The scope of the token should be All, and the label can be whatever you want. I chose Lambda Execution.
Save that token, and open up your AWS Control Panel. Switch over to Lambda.
Click Create Lambda Function, then click Blank Function. Next, click the empty dotted box to the left of the word Lambda and click Cloudwatch Events as the trigger. For the rule, create new rule. Rule name is Nightly-Build and the schedule expression is cron(0 0 * * ? *) -run at 12 UTC every day. (Source)
Check Enable Trigger, then move to the next screen.
Name the function 'NightlyBuild'. Skip the code for now.
Set the environment variable CircleCIToken to .
Set the role to a basic lambda execution role.
Under advanced settings, set the timeout to 15 seconds.
Finally, the lambda code:
// Based on: https://github.com/godfreyhobbs/circleci-cron-serverless/blob/master/handler.js
'use strict';
var http = require("https");
exports.handler = (event, context, callback) => {
var options = {
"method": "POST",
"hostname": "circleci.com",
"port": null,
// example: "/api/v1.1/project/github/RobUserName117/secret-project/tree/builds"
"path": "/api/v1.1/project/<github or bitbucket>/<organization or user name>/<project name>/tree/<branch name with scripts>" +
"?circle-token=" + process.env.CircleCIToken,
"headers": {
"content-type": "application/json",
"cache-control": "no-cache"
}
};
// Make request, write out all of response
var req = http.request(options, function (res) {
var chunks = [];
res.on("data", function (chunk) {
chunks.push(chunk);
});
res.on("end", function () {
var body = Buffer.concat(chunks);
const response = body.toString();
console.log(response);
callback(null, response);
});
});
req.write(JSON.stringify({
"build_parameters": {
"RUN_NIGHTLY_BUILD": "true"
}
}));
req.end();
};
Most of this code is self explanatory. The line that I would like to direct your attention to is the "path" part of your options object. According to the Circle CI docs, the URL that you post to is /api/v1.1/project/vcs system/organization or user name/project name/tree/branch
So if you wanted to push to the staging branch of the blogly project of the Google organization, stored in github, you would use:
/api/v1.1/project/github/google/blogly/tree/staging
The second thing is that we are calling our builds by sending a JSON with the RUN_NIGHTLY_BUILD parameter set to true. You could set or send as many variables as you please - see the docs for more details.
When this code runs successfully (click 'TEST'), you should get a response that says something like "Last build was blah blah blah". If you see a response that looks like CircleCI Webpage HTML, you made a configuration error somewhere.
Switch over to CircleCI and watch your build happen, then switch to Github and witness your glorious pull request!
Finished! Thanks for reading to the end. If you have any questions, please message me directly or leave a comment.
