Last week, I posted a tutorial on How to Test a GitHub Action with GitHub Actions. The audience for that post, as well as this one, are maintainers of GitHub Actions. In that post, I explained my automated approach to testing GitHub Actions, including unit testing and integration testing, all within GitHub Actions itself. I assumed in that post that the Action you are testing is implemented in Python, although most of the post is applicable more generally, or can easily be adapted to other languages.
This week's post is a bonus tip related to testing a GitHub Action within GitHub Actions. In last week's post, I assumed that it was possible to fully automate the validation of the results of the Action we are testing. In particular, I had a step that used Python's unittest module to run a set of tests that verified correctness of any files produced or modified by the action. But what if the action produces something that cannot be verified automatically? For example, what if it produces an image or something else that must be visually inspected for correctness?
Table of Contents: The rest of this post is organized as follows:
I'm assuming that you have unit tests and already have of a workflow that runs your unit tests, such as something like my example from last week. Now let's consider a case where my previous approach to integration testing isn't feasible, such as if the output of the action itself is an image or something else that must be visually inspected. We can replace the integration testing steps of last week's workflow with the following step (or add this as an additional step if you can partially automate validating your integration test results). In this example step, I'm assuming your action produces an svg file that you want to inspect, but this can work with anything.
-name:Upload files produced by action as a workflow artifact for inspection if necessaryuses:actions/upload-artifact@v3with:name:name-for-the-artifactpath:path/to/file/filename.svg
Or perhaps your action produces an entire directory of files, then you can use something like:
-name:Upload files produced by action as a workflow artifact for inspection if necessaryuses:actions/upload-artifact@v3with:name:name-for-the-artifactpath:path/to/directory
In both of the above cases, I've used the actions/upload-artifact action to upload either a file or an entire directory of files as a workflow artifact. In both cases, that action will zip up what the path input points to into a zip file with name based on the name input, such as name-for-the-artifact.zip in this example. It is then attached to the run of the workflow that produced it.
In this way, you can download from the Actions tab of your repository, and inspect the results. Just go to the Actions tab, and find the relevant workflow run, such as from a PR or push.
Real Example
I use this approach in the user-statistician, which generates an SVG with a detailed summary of your GitHub activity, such as contributions, language statistics, etc. I have unit tests implemented in Python with the unittest module. But, the SVG as a whole can only really be validated visually, such as to inspect that content fits appropriately, image dimensions are correct based on content, etc.
The last step of my workflow that runs the tests for that project is the following:
-name:Upload generated SVG as a workflow artifact for inspection if necessaryuses:actions/upload-artifact@v3with:name:generated-imagepath:images/userstats.svg
After the workflow runs, such as while reviewing a PR, I can navigate to the Actions tab of the repository, find the workflow run, and I'll see the following (screenshot of an actual run).
Be aware that if you are using this approach in a private repository that workflow artifacts count against your storage limit. In public repositories, they do not count against your quota, but they are removed after some time period. The default is 90 days.
The complete workflow for this project is found at: build.yml. The repository itself is:
The cicirello/user-statistician GitHub
Action generates a detailed visual summary of your activity on GitHub in the form of an SVG
suitable to display on
your GitHub Profile README
Although the intended use-case is to generate an SVG image for your GitHub Profile README
you can also potentially link to the image from a personal website, or from anywhere else
where you'd like to share a summary of your activity on GitHub. The SVG that the action
generates includes statistics for the repositories that
you own, your contribution statistics (e.g., commits, issues, PRs, etc), as well as
the distribution of languages within public repositories that you own
The user stats image can be customized, including the colors such as with one
of the built-in themes or your own set of custom…
Here's last week's full example workflow for automated testing, but with the example artifact upload step added at the end.
name:buildon:push:branches:[main]pull_request:branches:[main]jobs:build:runs-on:ubuntu-lateststeps:-uses:actions/checkout@v3-name:Setup Pythonuses:actions/setup-python@v4with:python-version:'3.10'-name:Run Python unit testsrun:python3 -u -m unittest tests/tests.py-name:Verify that the Docker image for the action buildsrun:docker build . --file Dockerfile-name:Integration test 1uses:./with:input-one:somethinginput-two:true-name:Integration test 2uses:./with:input-one:something elseinput-two:false-name:Verify integration test resultsrun:python3 -u -m unittest tests/integration.py-name:Upload files produced by action as a workflow artifact for inspection if necessaryuses:actions/upload-artifact@v3with:name:name-for-the-artifactpath:path/to/file/filename.svg
Vincent A. Cicirello - Professor of Computer Science at Stockton University - is a
researcher in artificial intelligence, evolutionary computation, swarm intelligence,
and computational intelligence, with a Ph.D. in Robotics from Carnegie Mellon
University. He is an ACM Senior Member, IEEE Senior Member, AAAI Life Member,
EAI Distinguished Member, and SIAM Member.