feat(control-plane)!: add support for handling multiple events in a single invocation

[iainlane]

Currently we restrict the scale-up Lambda to only handle a single event at a time. In very busy environments this can prove to be a bottleneck: there are calls to GitHub and AWS APIs that happen each time, and they can end up taking long enough that we can't process job queued events faster than they arrive.

In our environment we are also using a pool, and typically we have responded to the alerts generated by this (SQS queue length growing) by expanding the size of the pool. This helps because we will more frequently find that we don't need to scale up, which allows the lambdas to exit a bit earlier, so we can get through the queue faster. But it makes the environment much less responsive to changes in usage patterns.

At its core, this Lambda's task is to construct an EC2 CreateFleet call to create instances, after working out how many are needed. This is a job that can be batched. We can take any number of events, calculate the diff between our current state and the number of jobs we have, capping at the maximum, and then issue a single call.

The thing to be careful about is how to handle partial failures, if EC2 creates some of the instances we wanted but not all of them. Lambda has a configurable function response type which can be set to ReportBatchItemFailures. In this mode, we return a list of failed messages from our handler and those are retried. We can make use of this to give back as many events as we failed to process.

Now we're potentially processing multiple events in a single Lambda, one thing we should optimise for is not recreating GitHub API clients. We need one client for the app itself, which we use to find out installation IDs, and then one client for each installation which is relevant to the batch of events we are processing. This is done by creating a new client the first time we see an event for a given installation.

We also remove the same batch_size = 1 constraint from the job-retry Lambda. This Lambda is used to retry events that previously failed. However, instead of reporting failures to be retried, here we maintain the pre-existing fault-tolerant behaviour where errors are logged but explicitly do not cause message retries, avoiding infinite loops from persistent GitHub API issues or malformed events.

Tests are added for all of this.

Tests in a private repo (sorry) look good. This was running ephemeral runners with no pool, SSM high throughput enabled, the job queued check _dis_abled, batch size of 200, wait time of 10 seconds. The workflow runs are each a matrix with 250 jobs.

[iainlane]

I was getting warnings about mixing this with persistentKeys. The function says

    /**
     * @deprecated This method is deprecated and will be removed in the future major versions, please use {@link appendPersistentKeys() `appendPersistentKeys()`} instead.
     */
    addPersistentLogAttributes(attributes: LogKeys): void;

so I've switched it!

[Copilot]

Pull Request Overview

This PR adds support for handling multiple events in a single Lambda invocation to improve throughput in busy environments. Previously, the scale-up Lambda was restricted to processing only one event at a time, which could create bottlenecks when GitHub and AWS API calls took too long.

• Event batching: Enables processing multiple SQS messages in a single Lambda invocation with configurable batch sizes
• Partial failure handling: Implements ReportBatchItemFailures to retry only failed messages rather than entire batches
• GitHub client optimization: Reuses GitHub API clients across events for the same installation to reduce API overhead

Reviewed Changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
variables.tf	Adds new variables for configuring Lambda event source mapping batch size and batching window
modules/runners/scale-up.tf	Updates Lambda event source mapping to support batch processing with ReportBatchItemFailures
lambdas/functions/control-plane/src/scale-runners/scale-up.ts	Major refactor to handle batched events, optimize GitHub client usage, and return failed message IDs
lambdas/functions/control-plane/src/lambda.ts	Updates Lambda handler to process SQS batches and return batch item failures
lambdas/functions/control-plane/src/scale-runners/ScaleError.ts	Enhanced to support batch failure reporting with failed instance counts

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[npalm]

Tested with multi runner setup and pools. Looks all good.

[iainlane]

@iainlane @guicaulada what you think. Since the SQS message event is changed the PR can create a tiny deisruption during an update. For that reason I think we should mark it as a breaking change. What do you think?

Yeh not a bad idea. It’s a big and still a bit of a risky change. Could be an idea to do a beta / prerelease and let our orgs try it in prod for a bit before making a stable.

[npalm]

@iainlane @guicaulada what you think. Since the SQS message event is changed the PR can create a tiny deisruption during an update. For that reason I think we should mark it as a breaking change. What do you think?

Yeh not a bad idea. It’s a big and still a bit of a risky change. Could be an idea to do a beta / prerelease and let our orgs try it in prod for a bit before making a stable.

I have no setup for a beta stream. So in case you have tested on your side, I think we could merge it. In case of any problems we could set the message batch size to 1. And it should work as before.

[guicaulada]

@iainlane @guicaulada what you think. Since the SQS message event is changed the PR can create a tiny deisruption during an update. For that reason I think we should mark it as a breaking change. What do you think?

Yeh not a bad idea. It’s a big and still a bit of a risky change. Could be an idea to do a beta / prerelease and let our orgs try it in prod for a bit before making a stable.

I have no setup for a beta stream. So in case you have tested on your side, I think we could merge it. In case of any problems we could set the message batch size to 1. And it should work as before.

I agree. I think merging it as a major release and marking it as a breaking change is the way to go if there's no beta stream. We should start running this in large scale soon and if there's any new issues we would be quick to address them. Anyone updating to the new major release should be aware of this change and take the necessary measures to ensure minimal disruption.

I have tested with several parallel workflows of 200 jobs each, the only issue I found is when I hit VcpuLimitExceeded which would cause repeated failures, eventually messages would get dropped and jobs would be left hanging in the queue. I assumed this would be the expected behavior given the retry_job is also fault-tolerant. Once the quota was increased, all jobs would finish successfully, it felt more like an issue related to AWS Account limits than the new scale-up logic.

We don't have redrive_build_queue or retry_job enabled, maybe those could help prevent these kinds of issues.

[guicaulada]

I believe this is ready for a release.

We have been running this on production for nearly 48h now without any issues, we run 100s of thousands of workflows every day over ~2000 different repositories.

I've noticed queue wait times significantly reduced (by over 50%) and the whole system is able to handle peak demand and keep pace with everyday workflows much more efficiently.

Our SQS looks much healthier as well, here's some metrics from it.

Max # of messages on queue spiking up to ~1600 before the change:

Max # of messages staying low (under 50) after the change:

Oldest message age reaching ~1200 seconds before the change:

Oldest message age capped around 300 seconds after the change:

[Brend-Smits]

It seems after rebasing there's a test failing. Any idea what is going on here?

[iainlane]

Have rebased, squashed the old commits into one and fixed the failure 👍

[npalm]

@guicaulada @iainlane changed the target branch to next, will use this branch to bundle a few PRs for a breaking release.

[npalm]

@iainlane @guicaulada Thanks a lot!!!