Making a private Github Runner

Making a private Github Runner

First see Adding Github Actions to a repo

Installation

You will need an on-prem machine and service account to run the runner. Either you or someone from ESCP will need root privilege to configure the service. Please contact escp-ticket@tufts.edu to create these.

Note that Github maintains a matrix of requirements for self-hosted runners. When your OS version goes EOL, your runner will soon follow and may encounter difficulties starting the runner service. You’ll want to ensure that you migrate/upgrade your runner OS ahead of the EOL schedule of your OS. See:About self-hosted runners - GitHub Docs

After the service account is created, someone with root privileges should do this:

  • If you are not a member of ESCP, document what you’re doing however you document things.

    • If you are a member of ESCP, document in the node.yaml file:

      # See https://tuftswork.atlassian.net/wiki/spaces/ESPTS/pages/499220519/Making+a+private+Github+Runner # Git Runner installed as user 'foobaruser' connected to repo [URL of github repo] # Note: This runner will die after the OS is EOL. For RHEL 8 this is May 31, 2029 # Note: This runner will die after the OS is EOL. For RHEL 9 this is May 31, 2032
  • In your web browser, browse to your repository.

  • Go to Settings > Actions > Runners > New Self-Hosted Runner.

  • Now you have a choice. You can do the Manual process below, or the Scripted process.

  • Manual:

    • If you just follow GitHub’s instructions copying & pasting the commands they give you, there are a few issues. (1) When someone logs into the linux machine and looks in the runner directory, there is no way to know what repo it’s connected to. (2) When you login to github and look at the repo, there is no way to know what host or username the runner is running under. (3) At the end of github’s installation process, the runner is running, but it’s not configured to start as a service on the machine.

    • The instructions here help you manually install the runner, while addressing all those issues.

    • On the linux host, become the service user.

      # become root sudo su - # become the service user su - exampleuser1
    • Copy/paste the commands from github, up to and including extracting the tarball. Don’t run the config line yet.

    • When you run the config line, it will prompt you interactively:

      • Runner group: (just press enter to leave it blank)

      • Name of the runner: Enter something of the form hostname_username so you can easily identify where the runner runs just by looking at the runner in github

      • Labels: In most cases just press enter, but if you want any extra labels, type them here

      • Work folder: (just press enter)

    • Don’t run the last line that github provides you, run.sh. That would launch it once and it would not persist a reboot. Instead, continue as follows:

    • Since there is no record on the linux host, which github repo this runner is connected to, go to the service user’s home dir, and create a text file

      cd # Edit the following. Put the URL of the github repo echo url > actions-runner-url.txt
    • Logout as the service user. Become root.

    • See notes below about selinux GHA Runner service and systemd+selinux. After fixing the selinux context come back here.

    • Edit the following. Substitute something appropriate for ${user_home_dir} and $service_user

      # Do this as root # Substitute something appropriate for ${user_home_dir} and $service_user cd ${user_home_dir}/actions-runner ./svc.sh install $service_user service_name=$(./svc.sh status | grep '^/etc/.*service$' | head -n 1 | sed 's/.*\///') systemctl enable --now $service_name
  • Scripted:

    • It will give you a series of commands to paste into the terminal. Instead of pasting them into the terminal, do this:

      • Find the line that says "./config.sh --url=..."

      • Get the URL and the token.

      • Read the URL to ensure it's a specific repository, not the organization or whatever.

      • Run this script as root. The script installs the runner, creates the service, and launches the service as the service account. It records the URL of which repository it's working for, in the service account's home directory under "actions-runner-url.txt":

        # Optionally, if you need your runner to have any custom label, you may specify --labels= /root/bin/create-gitrunner.sh username GithubRepositoryURL token [--labels=foobarlabel]
    • Browse back to Actions > Runners, and confirm the new runner appears there, with a green Status, and Idle.

GHA Runner service and systemd+selinux

The OOTB experience when installing the runner in a separate user’s home directory is that the runner will not operate correctly on any system with selinux enabled, such as onprem RHEL9 builds.

For generic notes on selinux, see: SELinux

While the runner installer does attempt to set things properly:

Creating launch runner in /etc/systemd/system/actions.runner.Tufts-Technology-Services-runner-name.service Run as user: username Run as uid: nnn gid: nnn Relabeled /etc/systemd/system/actions.runner.Tufts-Technology-Services-runner-name.service from unconfined_u:object_r:user_home_t:s0 to unconfined_u:object_r:systemd_unit_file_t:s0 Created symlink /etc/systemd/system/multi-user.target.wants/actions.runner.Tufts-Technology-Services-runner-name.service → /etc/systemd/system/actions.runner.Tufts-Technology-Services-runner-name.service. Finished. Service is running.

this only changes the context of the service itself. The service will try to run a shell script inside the user directory and this will fail:

# example outputs from systemctl/journalctl date hostname systemd[1092642]: actions.runner.Tufts-Technology-Services-runner-name.service: Failed at step EXEC spawning /home/username/actions-runner/runsvc.sh: P> date hostname systemd[1]: actions.runner.Tufts-Technology-Services-runner-name.service: Main process exited, code=exited, status=203/EXEC # error if you look at selinux messages type=AVC msg=audit(1724335071.274:92964): avc: denied { execute } for pid=242535 comm="(unsvc.sh)" name="runsvc.sh" dev="dm-5" ino=407846 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclas s=file permissive=0

The fix is to update the selinux context:

# Note that you could probably skip the chcon command, run the # semanage command, and then use restorecon to reapply the new changes. # note that this change is not persistent across relabels chcon -R -t bin_t /home/username/actions-runner # this change preserves the context semanage fcontext -a -t bin_t "/home/username/actions-runner(/.*)"

Removing a private runner

If you installed a runner as above and need to remove it:

  • Login as root

    cd ${service_user}/actions-runner ./svc.sh uninstall rm -rf ${service_user}/actions-runner
  • Look in ${service_user}/actions-runner-url.txt

    • Browse to that repository > Settings > Actions > Runners, and force-remove the runner.

  • Finally rm ${service_user}/actions-runner-url.txt

If you need separate CI/CD runners on Dev & Prod

  • Install a runner on dev, and another on prod, using --labels=develop and --labels=main or whatever. It is customary to match your branch names.

  • Note: you should set up an ssh deploy key, and for the “first run,” you’ll need to clone your repo onto the system into your working-directory manually

  • Create two separate workflows in your repository, like this:

    • .github/workflows/autodeploy-develop.yml

      name: autodeploy-develop on: push: branches: - develop jobs: autodeploy: runs-on: [ self-hosted, develop ] steps: - working-directory: /path/to/repository run: git pull
    • .github/workflows/autodeploy-main.yml

      name: autodeploy-main on: push: branches: - main jobs: autodeploy: runs-on: [ self-hosted, main ] steps: - working-directory: /path/to/repository run: git pull