Skip to content

Module - Multi runner

This module replaces the top-level module to make it easy to create with one deployment multiple type of runners.

This module creates many runners with a single GitHub app. The module utilizes the internal modules and deploys parts of the stack for each runner defined.

The module takes a configuration as input containing a matcher for the labels. The webhook lambda is using the configuration to delegate events based on the labels in the workflow job and sent them to a dedicated queue based on the configuration. Events on each queue are processed by a dedicated lambda per configuration to scale runners.

For each configuration:

  • When enabled, the distribution syncer is deployed for each unique combination of OS and architecture.
  • For each configuration a queue is created and runner module is deployed

Matching

Matching of the configuration is done based on the labels specified in labelMatchers configuration. The webhook is processing the workflow_job event and match the labels against the labels specified in labelMatchers configuration in the order of configuration with exact-match true first, followed by all exact matches false.

The catch

Controlling which event is taken up by which runner is not to this module. It is completely done by GitHub. This means when potentially different runners can run the same job there is nothing that can be done to guarantee a certain runner will take up the job.

An example, given you have two runners one with the labels. self-hosted, linux, x64, large and one with the labels self-hosted, linux, x64, small. Once you define a subset of the labels in the workflow, for example self-hosted, linux, x64. Both runners can take the job potentially. You can define to scale one of the runners for the event, but still there is no guarantee that the scaled runner takes the job. The workflow with subset of labels (self-hosted, linux, x64) can take up runner with specific labels (self-hosted, linux, x64, large) and leave the workflow with labels (self-hosted, linux, x64, large) be without the runner. The only mitigation that is available right now is to use a small pool of runners. Pool instances can also exist for a short amount of time and only created once in x time based on a cron expression.

Jobs not defining all all labels but for example only [self-hosted, linux] could be matched to potentially different runners. The matcher scales the first runner that matches. With the attribute priority the order of matchers can be defined.

Usages

A complete example is available in the examples, see the multi-runner example for actual implementation.

module "multi-runner" {
  prefix = "multi-runner"

  github_app = {
    # app details
  }

  multi_runner_config = {
    "linux-arm" = {
      matcherConfig : {
        labelMatchers = [["self-hosted", "linux", "arm64", "arm"]]
        exactMatch    = true
      }
      runner_config = {
        runner_os                      = "linux"
        runner_architecture            = "arm64"
        runner_extra_labels            = "arm"
        enable_ssm_on_runners          = true
        instance_types                 = ["t4g.large", "c6g.large"]
        ...
      }
      ...
    },
    "linux-x64" = {
      matcherConfig : {
        labelMatchers = [["self-hosted", "linux", "x64"]]
        exactMatch    = false
      }
      runner_config = {
        runner_os                       = "linux"
        runner_architecture             = "x64"
        instance_types                  = ["m5ad.large", "m5a.large"]
        enable_ephemeral_runners        = true
        delay_webhook_event = 0
        ...
      }
      ...
    }
  }

}

Requirements

Name Version
terraform >= 1.3
aws >= 6.33
random ~> 3.0

Providers

Name Version
aws >= 6.33
random ~> 3.0

Modules

Name Source Version
ami_housekeeper ../ami-housekeeper n/a
instance_termination_watcher ../termination-watcher n/a
runner_binaries ../runner-binaries-syncer n/a
runners ../runners n/a
ssm ../ssm n/a
webhook ../webhook n/a

Resources

Name Type
aws_sqs_queue.queued_builds resource
aws_sqs_queue.queued_builds_dlq resource
aws_sqs_queue_policy.build_queue_dlq_policy resource
aws_sqs_queue_policy.build_queue_policy resource
random_string.random resource
aws_iam_policy_document.deny_insecure_transport data source

Inputs

Name Description Type Default Required
ami_housekeeper_cleanup_config Configuration for AMI cleanup. 
object({
    maxItems       = optional(number)
    minimumDaysOld = optional(number)
    amiFilters = optional(list(object({
      Name   = string
      Values = list(string)
    })))
    launchTemplateNames = optional(list(string))
    ssmParameterNames   = optional(list(string))
    dryRun              = optional(bool)
  })
 {} no
ami_housekeeper_lambda_memory_size Memory size limit in MB of the lambda. number 256 no
ami_housekeeper_lambda_s3_key S3 key for syncer lambda function. Required if using S3 bucket to specify lambdas. string null no
ami_housekeeper_lambda_s3_object_version S3 object version for syncer lambda function. Useful if S3 versioning is enabled on source bucket. string null no
ami_housekeeper_lambda_schedule_expression Scheduler expression for action runner binary syncer. string "cron(11 7 * * ? *)" no
ami_housekeeper_lambda_timeout Time out of the lambda in seconds. number 300 no
ami_housekeeper_lambda_zip File location of the lambda zip file. string null no
associate_public_ipv4_address Associate public IPv4 with the runner. Only tested with IPv4 bool false no
aws_partition (optiona) partition in the arn namespace to use if not 'aws' string "aws" no
aws_region AWS region. string n/a yes
cloudwatch_config (optional) Replaces the module default cloudwatch log config. See https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-Configuration-File-Details.html for details. string null no
enable_ami_housekeeper Option to disable the lambda to clean up old AMIs. bool false no
enable_dynamic_labels Experimental! Can be removed / changed without trigger a major release. Enable dynamic labels with 'ghr-' prefix. When enabled, jobs can use 'ghr-ec2-:' labels to dynamically configure EC2 instances (e.g., 'ghr-ec2-instance-type:t3.large') and 'ghr-run- bool false no
enable_managed_runner_security_group Enabling the default managed security group creation. Unmanaged security groups can be specified via runner_additional_security_group_ids. bool true no
eventbridge Enable the use of EventBridge by the module. By enabling this feature events will be put on the EventBridge by the webhook instead of directly dispatching to queues for scaling. 
object({
    enable        = optional(bool, true)
    accept_events = optional(list(string), [])
  })
 {} no
ghes_ssl_verify GitHub Enterprise SSL verification. Set to 'false' when custom certificate (chains) is used for GitHub Enterprise Server (insecure). bool true no
ghes_url GitHub Enterprise Server URL. Example: https://github.internal.co - DO NOT SET IF USING PUBLIC GITHUB. .However if you are using GitHub Enterprise Cloud with data-residency (ghe.com), set the endpoint here. Example - https://companyname.ghe.com| string null no
github_app GitHub app parameters, see your github app.
You can optionally create the SSM parameters yourself and provide the ARN and name here, through the *_ssm attributes.
If you chose to provide the configuration values directly here,
please ensure the key is the base64-encoded .pem file (the output of base64 app.private-key.pem, not the content of private-key.pem).
Note: the provided SSM parameters arn and name have a precedence over the actual value (i.e key_base64_ssm has a precedence over key_base64 etc).		 
object({
    key_base64 = optional(string)
    key_base64_ssm = optional(object({
      arn  = string
      name = string
    }))
    id = optional(string)
    id_ssm = optional(object({
      arn  = string
      name = string
    }))
    webhook_secret = optional(string)
    webhook_secret_ssm = optional(object({
      arn  = string
      name = string
    }))
  })
 n/a yes
iam_overrides This map provides the possibility to override some IAM defaults. The following attributes are supported: instance_profile_name overrides the instance profile name used in the launch template. runner_role_arn overrides the IAM role ARN used for the runner instances. 
object({
    override_instance_profile = optional(bool, null)
    instance_profile_name     = optional(string, null)
    override_runner_role      = optional(bool, null)
    runner_role_arn           = optional(string, null)
  })
 
{
  "instance_profile_name": null,
  "override_instance_profile": false,
  "override_runner_role": false,
  "runner_role_arn": null
}
 no
instance_profile_path The path that will be added to the instance_profile, if not set the environment name will be used. string null no
instance_termination_watcher Configuration for the spot termination watcher lambda function. This feature is Beta, changes will not trigger a major release as long in beta.

enable: Enable or disable the spot termination watcher.
memory_size: Memory size limit in MB of the lambda.
s3_key: S3 key for syncer lambda function. Required if using S3 bucket to specify lambdas.
s3_object_version: S3 object version for syncer lambda function. Useful if S3 versioning is enabled on source bucket.
timeout: Time out of the lambda in seconds.
zip: File location of the lambda zip file.		 
object({
    enable = optional(bool, false)
    features = optional(object({
      enable_spot_termination_handler              = optional(bool, true)
      enable_spot_termination_notification_watcher = optional(bool, true)
    }), {})
    memory_size       = optional(number, null)
    s3_key            = optional(string, null)
    s3_object_version = optional(string, null)
    timeout           = optional(number, null)
    zip               = optional(string, null)
  })
 {} no
key_name Key pair name string null no
kms_key_arn Optional CMK Key ARN to be used for Parameter Store. string null no
lambda_architecture AWS Lambda architecture. Lambda functions using Graviton processors ('arm64') tend to have better price/performance than 'x86_64' functions. string "arm64" no
lambda_event_source_mapping_batch_size Maximum number of records to pass to the lambda function in a single batch for the event source mapping. When not set, the AWS default of 10 events will be used. number 10 no
lambda_event_source_mapping_maximum_batching_window_in_seconds Maximum amount of time to gather records before invoking the lambda function, in seconds. AWS requires this to be greater than 0 if batch_size is greater than 10. Defaults to 0. number 0 no
lambda_principals (Optional) add extra principals to the role created for execution of the lambda, e.g. for local testing. 
list(object({
    type        = string
    identifiers = list(string)
  }))
 [] no
lambda_runtime AWS Lambda runtime. string "nodejs24.x" no
lambda_s3_bucket S3 bucket from which to specify lambda functions. This is an alternative to providing local files directly. string null no
lambda_security_group_ids List of security group IDs associated with the Lambda function. list(string) [] no
lambda_subnet_ids List of subnets in which the action runners will be launched, the subnets needs to be subnets in the vpc_id. list(string) [] no
lambda_tags Map of tags that will be added to all the lambda function resources. Note these are additional tags to the default tags. map(string) {} no
log_class The log class of the CloudWatch log groups. Valid values are STANDARD or INFREQUENT_ACCESS. string "STANDARD" no
log_level Logging level for lambda logging. Valid values are 'silly', 'trace', 'debug', 'info', 'warn', 'error', 'fatal'. string "info" no
logging_kms_key_id Specifies the kms key id to encrypt the logs with string null no
logging_retention_in_days Specifies the number of days you want to retain log events for the lambda log group. Possible values are: 0, 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, and 3653. number 180 no
matcher_config_parameter_store_tier The tier of the parameter store for the matcher configuration. Valid values are Standard, and Advanced. string "Standard" no
metrics Configuration for metrics created by the module, by default metrics are disabled to avoid additional costs. When metrics are enable all metrics are created unless explicit configured otherwise. 
object({
    enable    = optional(bool, false)
    namespace = optional(string, "GitHub Runners")
    metric = optional(object({
      enable_github_app_rate_limit    = optional(bool, true)
      enable_job_retry                = optional(bool, true)
      enable_spot_termination_warning = optional(bool, true)
    }), {})
  })
 {} no
multi_runner_config multi_runner_config = {
runner_config: {
runner_os: "The EC2 Operating System type to use for action runner instances (linux, osx, windows)."
runner_architecture: "The platform architecture of the runner instance_type."
runner_metadata_options: "(Optional) Metadata options for the ec2 runner instances."
ami: "(Optional) AMI configuration for the action runner instances. This object allows you to specify all AMI-related settings in one place."
create_service_linked_role_spot: (Optional) create the serviced linked role for spot instances that is required by the scale-up lambda.
credit_specification: "(Optional) The credit specification of the runner instance_type. Can be unset, standard or unlimited.
delay_webhook_event: "The number of seconds the event accepted by the webhook is invisible on the queue before the scale up lambda will receive the event."
disable_runner_autoupdate: "Disable the auto update of the github runner agent. Be aware there is a grace period of 30 days, see also the GitHub article"
ebs_optimized: "The EC2 EBS optimized configuration."
enable_ephemeral_runners: "Enable ephemeral runners, runners will only be used once."
enable_dynamic_labels: "Experimental! Can be removed / changed without trigger a major release. Enable dynamic labels with 'ghr-' prefix. When enabled, jobs can use 'ghr-ec2-:' labels to dynamically configure EC2 instances (e.g., 'ghr-ec2-instance-type:t3.large') and 'ghr-run-		 
map(object({
    runner_config = object({
      runner_os           = string
      runner_architecture = string
      runner_metadata_options = optional(map(any), {
        instance_metadata_tags      = "enabled"
        http_endpoint               = "enabled"
        http_tokens                 = "required"
        http_put_response_hop_limit = 1
      })
      ami = optional(object({
        filter               = optional(map(list(string)), { state = ["available"] })
        owners               = optional(list(string), ["amazon"])
        id_ssm_parameter_arn = optional(string, null)
        kms_key_arn          = optional(string, null)
      }), null)
      create_service_linked_role_spot      = optional(bool, false)
      credit_specification                 = optional(string, null)
      delay_webhook_event                  = optional(number, 30)
      disable_runner_autoupdate            = optional(bool, false)
      ebs_optimized                        = optional(bool, false)
      enable_ephemeral_runners             = optional(bool, false)
      enable_job_queued_check              = optional(bool, null)
      enable_on_demand_failover_for_errors = optional(list(string), [])
      scale_errors = optional(list(string), [
        "UnfulfillableCapacity",
        "MaxSpotInstanceCountExceeded",
        "TargetCapacityLimitExceededException",
        "RequestLimitExceeded",
        "ResourceLimitExceeded",
        "MaxSpotInstanceCountExceeded",
        "MaxSpotFleetRequestCountExceeded",
        "InsufficientInstanceCapacity",
        "InsufficientCapacityOnHost",
      ])
      enable_organization_runners             = optional(bool, false)
      enable_runner_binaries_syncer           = optional(bool, true)
      enable_ssm_on_runners                   = optional(bool, false)
      enable_userdata                         = optional(bool, true)
      instance_allocation_strategy            = optional(string, "lowest-price")
      instance_max_spot_price                 = optional(string, null)
      instance_target_capacity_type           = optional(string, "spot")
      instance_types                          = list(string)
      job_queue_retention_in_seconds          = optional(number, 86400)
      minimum_running_time_in_minutes         = optional(number, null)
      pool_runner_owner                       = optional(string, null)
      runner_as_root                          = optional(bool, false)
      runner_boot_time_in_minutes             = optional(number, 5)
      runner_disable_default_labels           = optional(bool, false)
      runner_extra_labels                     = optional(list(string), [])
      runner_group_name                       = optional(string, "Default")
      runner_name_prefix                      = optional(string, "")
      runner_run_as                           = optional(string, "ec2-user")
      runners_maximum_count                   = number
      runner_additional_security_group_ids    = optional(list(string), [])
      scale_down_schedule_expression          = optional(string, "cron(/5 * * * ? )")
      scale_up_reserved_concurrent_executions = optional(number, 1)
      userdata_template                       = optional(string, null)
      userdata_content                        = optional(string, null)
      enable_jit_config                       = optional(bool, null)
      enable_runner_detailed_monitoring       = optional(bool, false)
      enable_cloudwatch_agent                 = optional(bool, true)
      cloudwatch_config                       = optional(string, null)
      userdata_pre_install                    = optional(string, "")
      userdata_post_install                   = optional(string, "")
      runner_hook_job_started                 = optional(string, "")
      runner_hook_job_completed               = optional(string, "")
      runner_ec2_tags                         = optional(map(string), {})
      runner_iam_role_managed_policy_arns     = optional(list(string), [])
      vpc_id                                  = optional(string, null)
      subnet_ids                              = optional(list(string), null)
      idle_config = optional(list(object({
        cron             = string
        timeZone         = string
        idleCount        = number
        evictionStrategy = optional(string, "oldest_first")
      })), [])
      cpu_options = optional(object({
        core_count            = optional(number)
        threads_per_core      = optional(number)
        amd_sev_snp           = optional(string)
        nested_virtualization = optional(string)
      }), null)
      placement = optional(object({
        affinity                = optional(string)
        availability_zone       = optional(string)
        group_id                = optional(string)
        group_name              = optional(string)
        host_id                 = optional(string)
        host_resource_group_arn = optional(string)
        spread_domain           = optional(string)
        tenancy                 = optional(string)
        partition_number        = optional(number)
      }), null)
      license_specifications = optional(list(object({
        license_configuration_arn = string
      })), [])
      use_dedicated_host = optional(bool, false)
      runner_log_files = optional(list(object({
        log_group_name   = string
        prefix_log_group = bool
        file_path        = string
        log_stream_name  = string
        log_class        = optional(string, "STANDARD")
      })), null)
      block_device_mappings = optional(list(object({
        delete_on_termination = optional(bool, true)
        device_name           = optional(string, "/dev/xvda")
        encrypted             = optional(bool, true)
        iops                  = optional(number)
        kms_key_id            = optional(string)
        snapshot_id           = optional(string)
        throughput            = optional(number)
        volume_size           = number
        volume_type           = optional(string, "gp3")
        })), [{
        volume_size = 30
      }])
      pool_config = optional(list(object({
        schedule_expression          = string
        schedule_expression_timezone = optional(string)
        size                         = number
      })), [])
      job_retry = optional(object({
        enable             = optional(bool, false)
        delay_in_seconds   = optional(number, 300)
        delay_backoff      = optional(number, 2)
        lambda_memory_size = optional(number, 256)
        lambda_timeout     = optional(number, 30)
        max_attempts       = optional(number, 1)
      }), {})
      iam_overrides = optional(object({
        override_instance_profile = optional(bool, null)
        instance_profile_name     = optional(string, null)
        override_runner_role      = optional(bool, null)
        runner_role_arn           = optional(string, null)
        }), {
        override_instance_profile = false
        instance_profile_name     = null
        override_runner_role      = false
        runner_role_arn           = null
      })
    })
    matcherConfig = object({
      labelMatchers = list(list(string))
      exactMatch    = optional(bool, false)
      priority      = optional(number, 999)
    })
    redrive_build_queue = optional(object({
      enabled         = bool
      maxReceiveCount = number
      }), {
      enabled         = false
      maxReceiveCount = null
    })
  }))
 n/a yes
parameter_store_tags Map of tags that will be added to all the SSM Parameter Store parameters created by the Lambda function. map(string) {} no
pool_lambda_reserved_concurrent_executions Amount of reserved concurrent executions for the scale-up lambda function. A value of 0 disables lambda from being triggered and -1 removes any concurrency limitations. number 1 no
pool_lambda_timeout Time out for the pool lambda in seconds. number 60 no
prefix The prefix used for naming resources string "github-actions" no
queue_encryption Configure how data on queues managed by the modules in ecrypted at REST. Options are encrypted via SSE, non encrypted and via KMSS. By default encryptes via SSE is enabled. See for more details the Terraform aws_sqs_queue resource https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue. 
object({
    kms_data_key_reuse_period_seconds = number
    kms_master_key_id                 = string
    sqs_managed_sse_enabled           = bool
  })
 
{
  "kms_data_key_reuse_period_seconds": null,
  "kms_master_key_id": null,
  "sqs_managed_sse_enabled": true
}
 no
repository_white_list List of github repository full names (owner/repo_name) that will be allowed to use the github app. Leave empty for no filtering. list(string) [] no
role_path The path that will be added to the role; if not set, the environment name will be used. string null no
role_permissions_boundary Permissions boundary that will be added to the created role for the lambda. string null no
runner_additional_security_group_ids (optional) List of additional security groups IDs to apply to the runner list(string) [] no
runner_binaries_s3_sse_configuration Map containing server-side encryption configuration for runner-binaries S3 bucket. any 
{
  "rule": {
    "apply_server_side_encryption_by_default": {
      "sse_algorithm": "AES256"
    }
  }
}
 no
runner_binaries_s3_tags Map of tags that will be added to the S3 bucket. Note these are additional tags to the default tags. map(string) {} no
runner_binaries_s3_versioning Status of S3 versioning for runner-binaries S3 bucket. Once set to Enabled the change cannot be reverted via Terraform! string "Disabled" no
runner_binaries_syncer_lambda_timeout Time out of the binaries sync lambda in seconds. number 300 no
runner_binaries_syncer_lambda_zip File location of the binaries sync lambda zip file. string null no
runner_binaries_syncer_memory_size Memory size limit in MB for binary syncer lambda. number 256 no
runner_egress_rules List of egress rules for the GitHub runner instances. 
list(object({
    cidr_blocks      = list(string)
    ipv6_cidr_blocks = list(string)
    prefix_list_ids  = list(string)
    from_port        = number
    protocol         = string
    security_groups  = list(string)
    self             = bool
    to_port          = number
    description      = string
  }))
 
[
  {
    "cidr_blocks": [
      "0.0.0.0/0"
    ],
    "description": null,
    "from_port": 0,
    "ipv6_cidr_blocks": [
      "::/0"
    ],
    "prefix_list_ids": null,
    "protocol": "-1",
    "security_groups": null,
    "self": null,
    "to_port": 0
  }
]
 no
runners_lambda_s3_key S3 key for runners lambda function. Required if using S3 bucket to specify lambdas. string null no
runners_lambda_s3_object_version S3 object version for runners lambda function. Useful if S3 versioning is enabled on source bucket. string null no
runners_lambda_zip File location of the lambda zip file for scaling runners. string null no
runners_scale_down_lambda_timeout Time out for the scale down lambda in seconds. number 60 no
runners_scale_up_lambda_timeout Time out for the scale up lambda in seconds. number 30 no
runners_ssm_housekeeper Configuration for the SSM housekeeper lambda. This lambda deletes token / JIT config from SSM.

schedule_expression: is used to configure the schedule for the lambda.
enabled: enable or disable the lambda trigger via the EventBridge.
lambda_memory_size: lambda memory size limit.
lambda_timeout: timeout for the lambda in seconds.
config: configuration for the lambda function. Token path will be read by default from the module.		 
object({
    schedule_expression = optional(string, "rate(1 day)")
    enabled             = optional(bool, true)
    lambda_memory_size  = optional(number, 512)
    lambda_timeout      = optional(number, 60)
    config = object({
      tokenPath      = optional(string)
      minimumDaysOld = optional(number, 1)
      dryRun         = optional(bool, false)
    })
  })
 
{
  "config": {}
}
 no
scale_down_lambda_memory_size Memory size limit in MB for scale down. number 512 no
scale_up_lambda_memory_size Memory size limit in MB for scale_up lambda. number 512 no
ssm_paths The root path used in SSM to store configuration and secrets. 
object({
    root    = optional(string, "github-action-runners")
    app     = optional(string, "app")
    runners = optional(string, "runners")
    webhook = optional(string, "webhook")
  })
 {} no
state_event_rule_binaries_syncer Option to disable EventBridge Lambda trigger for the binary syncer, useful to stop automatic updates of binary distribution string "ENABLED" no
subnet_ids List of subnets in which the action runners will be launched, the subnets needs to be subnets in the vpc_id. list(string) n/a yes
syncer_lambda_s3_key S3 key for syncer lambda function. Required if using S3 bucket to specify lambdas. string null no
syncer_lambda_s3_object_version S3 object version for syncer lambda function. Useful if S3 versioning is enabled on source bucket. string null no
tags Map of tags that will be added to created resources. By default resources will be tagged with name and environment. map(string) {} no
tracing_config Configuration for lambda tracing. 
object({
    mode                  = optional(string, null)
    capture_http_requests = optional(bool, false)
    capture_error         = optional(bool, false)
  })
 {} no
user_agent User agent used for API calls by lambda functions. string "github-aws-runners" no
vpc_id The VPC for security groups of the action runners. string n/a yes
webhook_lambda_apigateway_access_log_settings Access log settings for webhook API gateway. 
object({
    destination_arn = string
    format          = string
  })
 null no
webhook_lambda_memory_size Memory size limit in MB for webhook lambda. number 256 no
webhook_lambda_s3_key S3 key for webhook lambda function. Required if using S3 bucket to specify lambdas. string null no
webhook_lambda_s3_object_version S3 object version for webhook lambda function. Useful if S3 versioning is enabled on source bucket. string null no
webhook_lambda_timeout Time out of the lambda in seconds. number 10 no
webhook_lambda_zip File location of the webhook lambda zip file. string null no

Outputs

Name Description
binaries_syncer_map n/a
instance_termination_handler n/a
instance_termination_watcher n/a
runners_map n/a
ssm_parameters n/a
webhook n/a