Enterprise Github repo sync with LFS - Getting beyond the HTTP 400s during checkout

Disclaimer

This post is tech-heavy. The topic is very specific about a particular technical issue. If you don't have this issue, you should probably skip this post. You have been warned.

Summary

If your project's repository has binary files, it is useful to use git-lfs (Large File Storage), so its size doesn't grow with each binary file edit. (Please note, if you need to version your binary assets, git-lfs is not the right choice.) With git-lfs, the binary files are stored in a different location, and only the source code and other text assets are versioned. Only a reference, a small text file, to the binary files are stored, to save space in the code repository.

However, if you need to access the actual binary files as part of an Action, perhaps to synchronize your repository with another one, then you need the Action to checkout/pull the actual binary files, and not just the references. What seems like something that should work out of the box actually doesn't. At least not with the combination of Enterprise Github, a self-hosted Action Runner, and LFS. This post explains how to make it work.

The Setup

Currently if you are using Enterprise Github, a self-hosted action-runner is your only option. (Even if that changes, you still might want a self-hosted action-runner.) Either way, if your Action's YAML is configured to checkout your repo, it might look like the following:

When this runs, the checkout fails due to HTTP 400 errors. There doesn't seem to be a workaround for this using the standard checkout action. Including using SSH configuration as part of the “with:” information.

The Cause

The Github checkout action sets up a basic authorization token by adding a section to git-config named http.<your.domain.com>.extraheader (where your.domain.com is the domain where your Enterprise Github instance is hosted.) While this works great with the normal git-checkout, it fails when pulling LFS files due to a second, more specific authorization header, being injected into the transaction flow.

Git-lfs rejects the transactions with HTTP 400s - “Bad Requests” due to the two authorization headers being present as follows:

The log snippet above was attained by setting the following environment variables:

Solution

There may be other ways to solve this problem. For example, if your LFS storage is on a different subdomain than your code repository, you can turn on subdomain isolation within your network configuration. In our particular case, LFS storage is on the same subdomain, so a different method was required.

By removing LFS from the initial checkout, the checkout will complete successfully, but it configures the environment in a way that prevents LFS from working. At this point we have our action run a shell script that reconfigures the environment, and then does the git-lfs pull. The following steps describe how to make that work.

Step 1 - Remove LFS from the initial checkout part of the action's YML file, run a shell script, and set up an Action secret for a private SSH key (more on that in step 2)

This triggers a checkout of the source code and the LFS (text-file) references. Keeping the initial non-LFS checkout in the Action's YML file allows changes to be made to the shell script that ultimately needs to run to pull the actual binary files. (entrypoint.sh in this example)

If the Action just runs entrypoint.sh without a checkout first, there isn't a good way to update entrypoint.sh within the action-runner.

Because LFS file transfers are only done over HTTPS, and authorization is required for the initial connection, an SSH user had to be set up for the Git-specific access. Otherwise it would use HTTPS and the Basic Authorization header, which will then break the git-lfs pull. (A public/private keypair was generated with ssh-keygen.)

An Enterprise Github build user, named “builder”, was created, and its public key was added to its user-configuration as an SSH key under Enterprise Github's Settings. The build user has read access to the repository. (It is set up as a separate user, so any activity can be monitored from a security perspective, and not tied to a particular employee account.)

Then in the repository settings on Enterprise Github, the private key was configured as a “secret”, and named “SSH_LFS_PULL_KEY”

The default Action-checkout script(s) configure things in a way that prevents SSH from working, at least when used in conjunction with LFS. The custom shell script, entrypoint.sh, reconfigures the environment, then does the git lfs pull to get the actual binary files.

The first thing that needs to happen is to remove the Basic Authorization token from git-config with the following command:

Since basic authorization was removed above, the remote.orgin.url needs to be replaced. The following command configures git to try and use SSH instead of HTTPS.

For SSH to work correctly, the private SSH pull key needs to be in place, and be readable.

The following starts the ssh-agent, and adds the SSH key to it. (If you are running on a persistent VM, you'll want to check and see if it is already running before trying to start it.)

The known_hosts file might need to be updated too. You lose the protection of a persistent known hosts list if you recreate it each time, so you may prefer for this to fail if it is wrong. However, if you are running in a fresh container or VM each time, your known_hosts may not exist. The following is a generic example that will create known_hosts if necessary, and add your entry if it isn't there:

Finally, set the git user and email to the “build user” created earlier, assign the GIT_SSH_COMMAND environment variable, and pull the LFS files from your branch of choice.

Once the LFS pull completes, the actual files, not their references, will exist alongside the source code on the action-runner. All of the files can then be synced with a different remote repository loosely following this existing pattern: https://github.com/cpina/github-action-push-to-another-repository

The Github Action example linked above was my starting point to synchronize two repositories. It did not work out of the box when using Enterprise Github, a self-hosted runner, and LFS. I suggest you start by looking at that project, then apply the changes described above as needed.