This solution automates Gitlab Runner deployment and administrative tasks on Amazon EC2 through Infrastructure-as-Code (IaC).
The solution is published in this blog post.
NOTE: The solution presented in this post illustrates a possible way of implementing Gitlab runner autoscaling. Support for this solution is limited. A vendor supported solution is available here: GitLab HA Scaling Runner Vending Machine for AWS EC2 ASG. As a result of this breaking change, the solution's token registration process has been be updated on 04/22/2024. The previsou code has been tagged as v1.0.
The following diagram displays the solution architecture. It also shows the workflow of deploying the Gitlab Runner and registering it to Gitlab projects.

-
We use AWS CloudFormation to describe the infrastructure that is hosting the Gitlab Runner. The user runs a deploy script in order to deploy the CloudFormation template. The template is parameterized, and the parameters are defined in a properties file. The properties file specifies the infrastructure configuration, as well as the environment in which to deploy the template.
-
The deploy script calls CloudFormation CreateStack API to create a Gitlab Runner stack in the specified environment.
-
During stack creation, an EC2 autoscaling group is created with the desired number of EC2 instances. Each instance is launched via a launch template, which is created with values from the properties file. An IAM role is created and attached to the EC2 instance. The role contains permissions required for the Gitlab Runner to execute pipeline jobs. A lifecycle hook is attached to the autoscaling group on instance termination events. This ensures graceful instance termination.
-
During instance launch, CloudFormation uses a cfn-init helper script to install and configure the Gitlab Runner:
- (4a) cfn-init installs the Gitlab Runner software on the EC2 instance.
- (4b) cfn-init configures the Gitlab Runner as a docker executor using a pre-defined docker image in the Gitlab Container Registry. The docker executor implementation lets the Gitlab Runner run each build in a separate and isolated container. The docker image contains the software required to run the pipeline workloads, thereby eliminating the need to install these packages during each build.
- (4c) cfn-init registers the Gitlab Runner to Gitlab projects specified in the properties file, so that these projects can utilize the Gitlab Runner to run pipelines.
-
The user may repeat the same steps to deploy Gitlab Runner into another environment.
Run the interactive setup script from the root of this repository:
./setup-runner.sh
It checks your local tools and AWS credentials, and infers the GitLab project and stack name from the git remote. Then it:
- creates the GitLab runner (with a fine-grained personal access token that has Project: Read and Runner: Create) or tells you how to create it and asks for the token;
- builds and pushes the executor image for linux/amd64, logging in with a project deploy token that has the write_registry scope (fine-grained tokens can't push images). Each build is tagged with the git commit (plus a timestamp if there are uncommitted changes) as well as
:latest, and the properties file points at the commit tag. Runners use the pull policyif-not-present,always(DockerPullPolicy), which reuses an image already on the instance whenever its tag matches, so a new tag is how a new image reaches the runners; - lets you pick a VPC and subnets (public or private with NAT);
- creates the Lambda S3 bucket if needed;
- writes
<stack-name>.propertiesand runsdeploy-runner.sh.
Re-running it for an existing stack reuses that properties file. Generated properties files are ignored by git. Runner tokens are never written to them; they are stored as an SSM SecureString at /<stack-name>/runner-tokens.
The sections below describe the same steps done by hand. Every script (setup-runner.sh, deploy-runner.sh, cycle-runner.sh) prints its usage with --help, -h or -?.
For this walkthrough, you need the following:
- A Gitlab account (all tiers including Gitlab Free self-managed, Gitlab Free SaaS, and higher tiers). This demo uses gitlab.com free tier.
- A Gitlab Container Registry.
- A Git client to clone the source code provided.
- An AWS account with local credentials properly configured (typically under ~/.aws/credentials) with these permissions:
- AmazonEC2FullAccess
- AutoScalingFullAccess
- AmazonS3FullAccess
- AmazonSSMFullAccess
- AmazonEventBridgeFullAccess
- AWSCloudFormationFullAccess
- AWSLambda_FullAccess
- IAMFullAccess
- AmazonECS_FullAccess
- AmazonEC2ContainerRegistryPowerUser
- The latest version of the AWS CLI. For more information, see Installing, updating, and uninstalling the AWS CLI.
- Docker is installed and running on the localhost/laptop.
- Nodejs and npm installed on the localhost/laptop.
- A VPC with 2 subnets that allow outbound internet traffic. Either private subnets behind a NAT gateway, or public subnets with
AssociatePublicIpAddress=truein the properties file (no NAT gateway needed). - The following IAM service-linked role created in the AWS account: AWSServiceRoleForAutoScaling
- An S3 bucket for storing Lambda deployment packages.
- Familiarity with Git, Gitlab CI/CD, Docker, EC2, and CloudFormation.
The Gitlab Runner in this solution is implemented as docker executor. The Docker executor connects to Docker Engine and runs each build in a separate and isolated container via a predefined docker image. The first step in deploying the Gitlab Runner is building a docker executor image. We provided a simple Dockerfile in order to build this image. You may customize the Dockerfile to install your own requirements.
- Create a directory where we will store our demo code. From your terminal run:
mkdir demo-repos && cd demo-repos
- Clone the source code repository found in the following location:
git clone <URL_OF_THIS_REPO>
- Create a new project on your Gitlab server. Name the project any name you like.
- Clone your newly created repo to your laptop. Ignore the warning about cloning an empty repository.
git clone <URL_OF_YOUR_REPO>
- Copy the demo repo files into your newly created repo on your laptop, and push it to your Gitlab repository. You may customize the Dockerfile before pushing it to Gitlab.
cp -r amazon-ec2-gitlab-runner/* <your-repo-dir>
cd <your-repo-dir>
git add .
git commit -m “Initial commit”
git push
- On the Gitlab console, go to your repository’s Package & Registries -> Container Registry. Follow the instructions provided on the Container Registry page in order to build and push a docker image to your repository’s container registry.
Once the docker executor image has been pushed to the Gitlab Container Registry, we can deploy the Gitlab Runner. The Gitlab Runner infrastructure is described in the Cloudformation template gitlab-runner.yaml. Its configuration is stored in a properties file called sample-runner.properties. A launch template is created with the values in the properties file. Then it is used to launch instances. This architecture lets you deploy Gitlab Runner to as many environments as you like by utilizing the configurations provided in the appropriate properties files.
- Obtain the runner registration tokens of the Gitlab projects that you want registered to the Gitlab Runner. Obtain the token by selecting the project’s Settings > CI/CD and expand the Runners section.
- Update the sample-runner.properties file parameters according to your own environment. Refer to the gitlab-runner.yaml file for a description of these parameters. Rename the file if you like. You may also create an additional properties file for deploying into other environments.
- Run the deploy script to deploy the runner:
cd <your-repo-dir>
./deploy-runner.sh <properties-file> [<region>] [<aws-profile>] <stack-name>
<properties-file> is the name of the properties file.
<region> is optional: the region where you want to deploy the stack. Leave it out to use AWS_REGION, AWS_DEFAULT_REGION or the region configured for the AWS CLI profile; the script asks for one if none is set. The other scripts take <region> the same way. Given only one of <region> and <aws-profile>, a value that looks like a region (us-east-1) is the region and anything else the profile.
<aws-profile> is optional: the name of the CLI profile you set up in the prerequisites section. Leave it out to use the AWS CLI's default credentials (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY environment variables, AWS_PROFILE, or the default profile).
<stack-name> is the name you chose for the CloudFormation stack.
For example:
./deploy-runner.sh sample-runner.properties us-east-1 dev gitlab-runner-demo
After the stack is deployed successfully, you will see the Gitlab Runner autoscaling group created in the EC2 console.
Now go to your Gitlab project Settings->CICD->Runners->Available specific runners, you will see the fully configured Gitlab Runner. The green circle indicates that the Gitlab Runner is ready for use.
There are times when you would want to update the Gitlab Runner. For example, updating the instance VolumeSize in order to resolve a disk space issue, or picking up a new AMI. The ImageId property is an SSM public parameter name (by default the latest Amazon Linux 2023 AMI), so re-running the deploy script resolves it to the newest AMI.
Utilizing the properties file and launch template makes it easy to update the Gitlab Runner. Simply update the Gitlab Runner configuration parameters in the properties file. Then, run the deploy script to udpate the Gitlab Runner stack.
Below is an example of updating the Gitlab Runner instance type.
- Update the “InstanceType” parameter in the properties file.
InstanceType=t3a.large
- Run the deploy-runner.sh script to update the CloudFormation stack:
cd <your-repo-dir>
./deploy-runner.sh <properties-file> [<region>] [<aws-profile>] <stack-name>
When the runner stops picking up jobs, collect its logs and state with:
./collect-runner-diagnostics.sh <stack-name> [<region>] [<aws-profile>] [--hours <n>]
It writes diagnostics/<stack-name>-<timestamp>/ and a .tar.gz of it, with the stack, autoscaling group, Lambda logs and metrics from AWS, and from each instance (through SSM Run Command) the gitlab-runner journal, redacted config.toml, gitlab-runner verify, a goroutine dump, docker state, memory, disk and OOM kills. Set GITLAB_TOKEN to also fetch the runners' status and recent jobs from GitLab; both this script and watch-runner-jobs.sh also read it from a .env file in the repository root (GITLAB_TOKEN=<token>, git-ignored) when it isn't already set. Tokens are redacted. For instances terminated during the window, the CPU, memory, swap and memory pressure they published are in metrics/, and aws/lifecycle-shutdowns.txt shows whether each one's graceful shutdown completed and how long it waited for jobs. SSM reports most of those shutdowns as Undeliverable even when they worked, because the instance terminates before the SSM agent reports back.
To watch the stack live from your machine:
GITLAB_TOKEN=<token> ./watch-runner-jobs.sh <stack-name> [<region>] [<aws-profile>] [--interval <seconds>] [--runner-id <id>]
It refreshes the autoscaling group's capacity and each instance's state, health, CPU, memory and swap use, memory pressure and running job count from CloudWatch (an instance whose count stops updating is shown as stale). Each instance publishes these every 30 seconds. Memory pressure (PSI) is the share of the last 10 seconds in which every task on the instance was waiting for memory: above about 10% the instance is thrashing and jobs slow down. With GITLAB_TOKEN set to a token that can read the runner, it also lists the jobs running right now, with the project, stage, how long each has run and which instance runs it. Without --runner-id, the runner ID is read once from an instance through SSM. --once prints the view a single time.
The cache keyword in .gitlab-ci.yml is stored in an S3 bucket created by the stack (the RunnerCacheBucket output), shared by all the runner instances, so a cache saved by a job on one instance is found by jobs on the others and survives instances being replaced. The runner signs the cache requests with the instance role, so no access keys are stored. Cache archives are deleted CacheExpirationDays (default 14) after they were last saved. The bucket is kept when the stack is deleted; empty and delete it by hand.
There are times when an autoscaling group instance must be terminated. For example, during an autoscaling scale-in event, or when the instance is being replaced by a new instance during a stack update, as seen previously. When terminating an instance, you must ensure that the Gitlab Runner finishes executing any running jobs before the instance is terminated, otherwise your environment could be left in an inconsistent state. Also, we want to ensure that the terminated Gitlab Runner is removed from the Gitlab project. We utilize an autoscaling lifecycle hook to achieve these goals.
The lifecycle hook works like this: A CloudWatch event rule actively listens for the EC2 Instance-terminate events. When one is detected, the event rule triggers a Lambda function. The Lambda function calls SSM Run Command to run a series of commands on the EC2 instances, via a SSM Document. The commands include stopping the Gitlab Runner gracefully when all running jobs are finished, de-registering the runner from Gitlab projects, and signaling the autoscaling group to terminate the instance.
There are also times when you want to terminate an instance manually. For example, when an instance is suspected to not be functioning properly. To terminate an instance from the Gitlab Runner autoscaling group, use the following command:
aws autoscaling terminate-instance-in-auto-scaling-group \
--instance-id="${InstanceId}" \
--no-should-decrement-desired-capacity \
--region="${region}" \
--profile="${profile}"
The above command terminates the instance. The lifecycle hook ensures that the cleanup steps are conducted properly, and the autoscaling group launches another new instance to replace the old one.
Note that if you terminate the instance by using the "ec2 terminate-instance" command, then the autoscaling lifecycle hook actions will not be triggered.
As new projects are added to your enterprise, you may want to register them to the Gitlab Runner, so that those projects can utilize the Gitlab Runner to run pipelines. On the other hand, you would want to remove the Gitlab Runner from a project if it no longer wants to utilize the Gitlab Runner, or if it qualifies to utilize the Gitlab Runner. For example, if a project is no longer allowed to deploy to an environment configured by the Gitlab Runner. Our architecture offers a simple way to add and remove projects from the Gitlab Runner. To add new projects to the Gitlab Runner, set the RunnersToken parameter in the properties file, and then rerun the deploy script to update the Gitlab Runner stack.
- Set the RunnersToken parameter in the properties file to all of the runner authentication tokens, including the existing ones. For example:
RunnersToken=glrt-exampletoken1,glrt-exampletoken2
- Update the Gitlab Runner stack. This overwrites the SecureString SSM parameter
/<stack-name>/runner-tokensthat stores the tokens. If RunnersToken is empty, the stored tokens are kept.
cd <your-repo-dir>
./deploy-runner.sh <properties-file> [<region>] [<aws-profile>] <stack-name>
- Relaunch the instances in the Gitlab Runner autoscaling group. The new instances will use the new RunnersToken value. Run the following command to relaunch the instances:
./cycle-runner.sh <runner-autoscaling-group-name> [<region>] [<aws-profile>]
To remove projects from the Gitlab Runner, follow the steps described above, with just one difference. Instead of adding new tokens to the RunnersToken parameter, remove the token(s) of the project that you want to dissociate from the runner.
Each Gitlab Runner can be configured to handle a fixed number of concurrent jobs. Once this capacity is reached for every runner, any new jobs will be in a Queued/Waiting status until the current jobs complete, which would be a poor experience for our team. Setting the number of concurrent jobs too high on our runners would also result in a poor experience, because all jobs leverage the same CPU, memory, and storage in order to conduct the builds.
In this solution, we utilize a scheduled Lambda function that runs every minute in order to inspect the number of jobs running on every runner. Each runner reads its job count from its local Prometheus Metrics endpoint and publishes it to CloudWatch every 30 seconds, and the Lambda function reads those metrics, so it does not need to run inside the VPC. A runner that has not published recently is treated as unavailable. If we approach the concurrent build limit of the group, then we increase the Autoscaling Group size so that it can take on more work. As the number of concurrent jobs decreases, then the scheduled Lambda function will scale the Autoscaling Group back in an effort to minimize cost. The Scaling-Up operation will ignore the Autoscaling Group’s cooldown period, which will help ensure that our team is not waiting on a new instance, whereas the Scale-Down operation will obey the group’s cooldown period. Before scaling, the function protects instances that are running jobs from scale-in and removes the protection once they are idle, so a scale-in terminates idle instances first. It also keeps the oldest healthy instances warm, as many as MinSize: they stay protected while idle, so a scale-in after a burst removes the newer instances and the warm one keeps the images it has pulled. A warm instance is replaced only when a problem is found on it. One that stops publishing its job count, or fails its EC2 status checks, is replaced right away. One launched from an out of date launch template (after a deploy that changed it), or whose root volume is 85% full, is replaced once it is idle. The function also keeps the group's AZRebalance process suspended, since evening out the zones after a scale-in would launch and terminate instances and restart the scale-in cooldown. Among idle instances that aren't warm, a scale-in terminates one from an out of date launch template, then the newest. Instances are not replaced for their age unless MaxInstanceLifetime is set. cycle-runner.sh replaces every instance, the warm one included. An instance that picks up a job just before it is chosen still finishes the job through the termination lifecycle hook.
The target capacity is enough instances for the running jobs plus CountOfNewJobsBeforeScaling free job slots (default 2), so new jobs start without waiting for an instance to boot. With 2 concurrent jobs per instance, 2 free slots adds a second instance as soon as one job starts; 1 adds it only once the first instance is full, which launches and terminates instances less often.
A runner only sees a job once it takes it, so on running jobs alone a burst of jobs scales out one instance at a time: each new instance boots, fills up, and only then is the next one added. The monitor can also count the jobs waiting in GitLab and scale out for them in one step. Set GitlabRunnerIds in the properties file to the IDs of the project runners the stack runs (setup-runner.sh records it when it creates the runner), and pass a token that can read those runners and their projects' jobs when deploying, for example a project access token with the Maintainer role and read_api scope:
GitlabApiToken=<token> ./deploy-runner.sh <properties-file> [<region>] [<aws-profile>] <stack-name>
The token is stored as the SSM SecureString /<stack-name>/gitlab-api-token, so later deploys don't need it. GitlabApiToken can also go in a .env file in the repository root. If it isn't set but GITLAB_TOKEN is (from the environment or .env), deploy-runner.sh asks before storing that token for the monitor, unless it is already the stored one. A waiting job counts when the runner has all of its tags, or when it has none and the runner runs untagged jobs. This assumes no other runner takes those jobs, such as a GitLab.com shared runner for untagged jobs. Group and instance runners aren't counted. If the GitLab API can't be reached or the token is rejected (for example once it expires), the monitor logs why and scales on running jobs alone.
Here is the logical sequence diagram for the work:
For operational monitoring, the Lambda function also publishes custom CloudWatch Metrics for the count of active jobs and, when it can count them, waiting jobs, along with the target and actual capacities of the Autoscaling group. We can utilize this information to validate that the system is working properly and determine if we need to modify any of our autoscaling parameters.
Problem: I deployed the CloudFormation template, but no runner is listed in my repository.
Possible Cause: Errors have been encountered during cfn-init, causing runner registration to fail. Connect to your runner EC2 instance, and check /var/log/cfn-*.log files.
See CONTRIBUTING for more information.
This library is licensed under the MIT-0 License. See the LICENSE file.




