The Air Gap Does Not Remove the Supply Chain
What changes when your CI/CD infrastructure has no route to the internet
I closed my last post by telling you to inventory what actually flows through your pipelines. Which base images you use, where they come from, which actions run in your workflows, and whether any of them are pinned. Then I spent a month doing exactly that inside a GitHub Enterprise Server appliance in a customer's airgapped environment with no outbound route to the internet, and found that half of what I'd written only works if the runner can reach the internet.
The tj-actions compromise and the Shai-Hulud worm made the case that CI/CD is a supply chain. That case is settled. What nobody writes about is what happens to the remediation advice when you take it somewhere the internet does not reach, or very few sites are reachable. Air-gapping does not remove your CI/CD supply chain or magically just make it "safe." It removes the vendor tooling that manages it, and you must manually re-create much of it.
Everything Public Assumes github.com
Start with the most basic line in any workflow file. Nearly every example on the public internet opens like this:
jobs:
build:
runs-on: ubuntu-latest
There is no ubuntu-latest on an air-gapped appliance. GitHub-hosted runners are not available on GitHub Enterprise Server, so every job runs only on a self-hosted runner you built and labeled yourself. A developer copies a workflow from a public README, commits it, and the job does not fail. It sits in pending. Forever. No error in the run log, nothing in the checks tab, no notification. The scheduler is waiting for a runner that will never register.
Container jobs behave the same way:
container:
image: python:3.12-slim
The runner has no route to Docker Hub, so the image never pulls and the job stalls in the same quiet way. In a disconnected environment, a missing dependency does not raise an error. It hangs, and that turns out to be true everywhere in this post.
My favorite example of this had nothing to do with the network. An organization got renamed, which left the old name sitting in the runner's .runner registration file. Every job assigned to that runner went to the “pending” state. Nothing reported the runner as unhealthy, because it was not unhealthy. It was registered, running, and reporting in, to an organization name that no longer existed.
Your Curated Allowlist Is a Text File
In the first post in this series I recommended a governed internal mirror with frozen snapshots as the answer to package manager sprawl. The equivalent for GitHub Actions on an internal appliance is github/actions-sync. Normally an appliance gets public actions through GitHub Connect, which links it to github.com and needs an outbound connection, so it stays off in an airgapped environment. GitHub's documentation says the tool [is intended for systems where GitHub Connect is not enabled], which is exactly the disconnected case. You run it from a machine that can reach both the open internet and your internal network, or you split it into pull and push if you can only reach one at a time, and move the results to your airgapped machine.
What the documentation does not convey is what this looks like in practice. The curated allowlist that the governance literature recommends turns out to be a text file of action references that a human maintains by hand, fed to a sync script that another human remembers to run. The tool only pulls from public repositories, and every destination organization must already exist on the appliance, or the sync has nowhere to land. Synced actions land in a single internal org as private repositories by default, and someone has to manually flip each one to visible before another team's workflow can reference it.
CodeQL, GitHub's code-scanning engine, does not come along for the ride. It needs its own separate bundle sync, through a different tool, on its own cadence.
When a developer adds a new uses: line to a workflow, nothing tells you. No warning, no failed check, no mail. The job pends, someone eventually asks why their build has been queued since Tuesday, and you go re-run the sync. I got into the habit of running it before every demo, which is a sentence that should bother you as much as it bothers me.
The Pinning Advice Partially Breaks
Pinning to a SHA is still worth doing, but the appliance decides which versions you get to pin:
- uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608 #
v4.1.0
- uses: actions/upload-artifact@v3.2.2
The second line looks like an oversight, but it's deliberate. The upload-artifact README states directly why: v4 and later are not supported on GitHub Enterprise Server, and GHES users must run v3.2.2. The v4 line depends on an artifact backend the appliance does not implement, so the newer, better-maintained version of the action returns a not-supported error and the older one is the only thing that works.
I want to leave that sitting there rather than resolve it. Security guidance tells you to run the current version of everything. The appliance tells you to run the version it supports. In disconnected environments those are frequently different instructions, and pretending otherwise just means your pipeline is broken and your policy document is happy.
Safety Controls That Need Unsafe Permissions
The R and Quarto toolchain made this tension explicit. The setup-r and setup-quarto actions install system packages as part of provisioning, which means the runner service account needs passwordless sudo before any of it works. To be clear, this is not a hack. It is the documented setup: the runner account is expected to have password-free sudo, and GitHub-hosted runners ship that way by default, which is exactly why workflows written against github.com assume it. Packages themselves came from an internal Posit Package Manager snapshot, which is the governed mirror from post one doing its job.
It still feels weird to type. You spend the whole engagement narrowing what the pipeline is allowed to reach, and then you hand the account that runs arbitrary workflow code the ability to become root without a prompt, because that is what the instructions say to do. Being documented is not the same as being comfortable, and the fact that nobody blinks at this step is worth noticing on its own.
The R toolchain was not the only place this tension showed up. Container jobs produced the same shape of problem from a different direction. They failed at the checkout step on a certificate error, because the internal CA is installed on the runner host but not inside the container image. The fix is to inject the CA into every container job:
cp /usr/local/share/ca-certificates/internal-ca.crt /etc/ssl/certs/
update-ca-certificates
Both of these are the security posture getting worse in service of the security tooling, and both are the documented way to do it: standing sudo on the runner, and a trust store you now modify at job runtime in every containerized build. I do not have a tidy fix. The honest version is that you take the trade deliberately, write down why, and make sure the next person knows it was a decision rather than an accident.
Enabled Is Not Running
A security scanning feature cost me the most time. It was checked in the management console, the configuration was applied, and the console reported it as on. It was not running, and nothing in the interface said so.
I spent hours in that configuration before the answer turned out not to be in the configuration. Advanced Security is licensed separately from base user seats, billed against active committers rather than included with the appliance, and a licensing gap does not show up as a licensing gap. It presents as a checkbox that is checked, leaving you with false confidence that it's running. That is the sentence worth taking away from all of this: a control that is enabled is not the same as a control that is running, and on an appliance you have to verify the second one yourself.
None of this is an argument against air gaps. For an insurer running regulated workloads, the disconnected appliance is the correct call and I would make it again. The argument is narrower than that. The cost of the air gap is real, it is paid almost entirely in human labor, and I have never once seen it budgeted for. The vendor tooling that quietly maintains your supply chain on github.com does not have an offline equivalent. You are the offline equivalent.
So here is the Monday morning version. Open one workflow file, list every uses: line in it, and for each one answer three questions:
Where does this come from on our network, and who put it there?
Who refreshes it, and how would we know if they stopped?
What happens the day it goes stale, and would it look like an error or like nothing?
If the answer to the third question is "like nothing," you have found the same problem I did.
None of the three questions above get answered by a tool you install once. Someone has to own the allowlist, own the sync cadence, and own checking that a licensed feature is actually running rather than just showing a checkbox, on a schedule tight enough that a stale sync shows up before a developer's build has been pending since Tuesday. Keep a name next to each of those three. If you can't, that is your first finding and problem to solve. That is the kind of runbook work we help clients at Lander Analytics build out: turning what GitHub.com gives you for free into a maintained internal process for the environments where it never shipped.
Travis Knoche
Senior Data Scientist
Lander Analytics
Subscribe to our Substack and below to our monthly emails for practical AI strategies for your organization: what to build, what to avoid, and how to make systems reliable in the real world.
Work with us: If you want help identifying the right first workflow, building a permissioned knowledge base, or training your team to ship responsibly, reach out at info@landeranalytics.com.
About the author: Travis Knoche is a Senior Data Scientist at Lander Analytics, where he designs, deploys, and maintains data science infrastructure in a wide variety of environments and constraints, and bridges the gap between backend infrastructure and frontend data science development work.
