Harden your GitHub Actions Workflows with zizmor, dependency pinning, and dependency cooldowns¶
It's been a crazy week (or weeks?) for a lot of people with several supply chain attacks.
They all seem to have originated from the compromise of trivy (ironically, trivy is a security scanner).
I wanted to understand how they initially gained access to the secrets used in their GitHub organization and found that there was an earlier attack that targeted GitHub Actions workflows of open source projects.
Looking at the details of how the secrets were extracted I noticed that they all used similar techniques.
And, unless I missed something, it is through template injection and unsafe pull_request_target triggers.
This could be avoided because all of those vulnerabilities (and more) can be found by zizmor, a static analysis tool for GitHub Actions!
The problem is that, unfortunately, GitHub Actions is NOT secure by default1.
One would think that when following the official documentation you end up with workflows that are secure and can not be exploited.
Last year, I came across zizmor and upon checking my workflows it pointed out several problems that I was quite surprised to find out about.
Of course, GitHub should make Actions more secure by default2. And it seems that the latest attacks have finally helped to make some progress in that direction. GitHub have published a security roadmap for GitHub Actions and are looking for feedback from the community.
Until that happens, what can you do right now to harden your GitHub Actions workflows?
Add zizmor to your project¶
Start by running it locally to address findings:
You can find alternative ways to install and run zizmor in the documentation.
At the same time, zizmor should run as a pre-commit hook and in CI as well so that future changes to your workflows don't introduce vulnerabilities again.
You can run the pre-commit hooks in CI (which is a good thing to ensure consistency across local development and CI):
- uses: j178/prek-action@79f765515bd648eb4d6bb1b17277b7cb22cb6468 # v2.0.0 (1)
with:
prek-version: <version>
- At the time of writing, this is the current version, please update it if there's a newer one and pin to the commit SHA.
Or, zizmor also provides a handy GitHub Action that you can integrate into your workflow.
See the integrations documentation for more ways to integrate zizmor.
Use actionlint in addition to zizmor
At the same time, I also suggest to use actionlint with the shellcheck integration.
It provides a lot of checks that complement zizmor.
In particular, it has a shellcheck integration.
You can add it as a pre-commit hook as well, either using the container image, or via the Go module:
- repo: https://github.com/rhysd/actionlint
rev: <version>
hooks:
- id: actionlint-docker
The container image includes shellcheck which is run by default.
- repo: https://github.com/rhysd/actionlint
rev: <version>
hooks:
- id: actionlint
language: golang
additional_dependencies:
# see also: https://github.com/rhysd/actionlint/pull/482
- "github.com/wasilibs/go-shellcheck/cmd/shellcheck@<version>"
See my blog post about renovating additional hook dependencies to ensure that the shellcheck dependency also receives dependency updates.
Note that there is no official GitHub Action provided for actionlint.
Only some third-party ones.
Hardening GitHub Actions workflows is something that especially open source projects should do.
And this would have certainly helped trivy and others not to have their secrets stolen and malicious new versions published.
But what if you were just a user of trivy.
How could you have avoided (or delayed) getting the malicious version?
The answer is: Dependency pinning.
Dependency Pinning¶
Dependency pinning is absolutely essential to get reproducible builds, development environments etc. Whenever you install dependencies or run a build, you know exactly which version of direct and transitive (via lock files) dependency versions you get. Renovate has a guide on dependency pinning that I recommend.
If you are still not convinced, look at what happened with the litellm compromise.
Based on analysis by futuresearch, in the 46 minutes that the two malicious versions were available on PyPI, litellm was downloaded over 46000 times!
Only 9% out of the over 2000 packages they analyzed pinned to an exact version.
88% of dependants would have received a malicious versions if they installed litellm during the attack window.
The trivy attack is similar.
In addition, the attacker also force-pushed tags of their actions to malicious versions.
It required pinning to a commit SHA hash to be unaffected since existing version tags were changed by the malicious actor.
Another attack with an action happened last year with tj-actions/changed-files.
So pinning to an exact version is not sufficient. You need to pin to a commit SHA.
Note
Maintainers should enable immutable releases on their repository/organization to prevent tags from being changed. Unfortunately, this is not enabled by default.
No one likes updating dependencies manually, and you don't have to. Thankfully, there are fantastic tools like Renovate available to use (and free and open source!).
Renovate provides a helper preset helpers:pinGitHubActionDigestsToSemver that can pin an action to a digest (commit hash) of a semantic version.
It also includes the version in a comment which is very helpful.
Here is an example with actions/checkout (probably the most used action out there?):
When a new version of actions/checkout is published, Renovate will create a PR for you with a diff like this:
steps:
- name: Checkout repository
- uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
+ uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
zizmor actually has an unpinned-uses rule which, since version v1.20.0, ensures you use hash-pinning.
For all other dependencies, Renovate provides the :pinAllExceptPeerDependencies.
Dependency cooldowns¶
With all this, it is of course still possible to be quick and merge the dependency update PR to update to a new version right away.
William Woodruff, the author of zizmor, made the case for dependency cooldowns.
And Andrew Nesbitt followed this up with a post looking at support for cooldowns in package managers.
Sticking with the Renovate example of this post3, we can make use of Renovate's minimumReleaseAge feature4:
With the above configuration, every dependency needs to have been released at least 7 days before.
Lock files
Renovate currently does not use minimumReleaseAge to restrict transitive dependencies in lock files.
However, there is an open issue to use minimum release age for package managers.
To do it manually, refer to the comparison of support for a cooldown across package managers.
Doing all this will give you hardened workflows and prevent you from unwillingly installing malicious versions (or at least decrease the probability of this happening quite a bit). Finally, if you are a maintainer of a package, please enable immutable releases, and use trusted publishing.
Hope this helps! Do you know if anything else that can be done? Please let me know.
-
Look through the audit rules of
zizmorto get an idea. ↩ -
See a LinkedIn post by Dan Lorenc (the CEO of Chainguard) on what GitHub would need to do to make actions more secure by default. ↩
-
Dependabot has a cooldown feature which
zizmortells you about. ↩ -
Renovate also provides a preset specific to npm that sets
minimumReleaseAgeto 3 days. ↩