[libc++] Store the premerge runner images in the monorepo

[ldionne]

We need one canonical place to store the image used by the various sets of libc++ CI runners. This is needed so that our run-buildbot-container script can stay up-to-date, and for the pre-merge infrastructure to stay up-to-date.

Previously, the images used by the premerge infrastructure were stored in llvm-zorg, which makes it less discoverable and more complicated to update and keep synchronized.

[llvmbot]

@llvm/pr-subscribers-libcxx

Author: Louis Dionne (ldionne)

Changes

We need one canonical place to store the image used by the various sets of libc++ CI runners. This is needed so that our run-buildbot-container script can stay up-to-date, and for the pre-merge infrastructure to stay up-to-date.

Previously, the images used by the premerge infrastructure were stored in llvm-zorg, which makes it less discoverable and more complicated to update and keep synchronized.

---

Full diff: https://github.com/llvm/llvm-project/pull/171443.diff

5 Files Affected:

• (modified) libcxx/docs/Contributing.rst (+11-31)
• (added) libcxx/utils/ci/images/libcxx_next_runners.txt (+1)
• (added) libcxx/utils/ci/images/libcxx_release_runners.txt (+1)
• (added) libcxx/utils/ci/images/libcxx_runners.txt (+1)
• (modified) libcxx/utils/ci/run-buildbot-container (+3-2)

diff --git a/libcxx/docs/Contributing.rst b/libcxx/docs/Contributing.rst
index e660daeba7e5b..84bbe1478c512 100644
--- a/libcxx/docs/Contributing.rst
+++ b/libcxx/docs/Contributing.rst
@@ -271,31 +271,18 @@ Updating the CI testing container images
 The libcxx linux premerge testing can run on one of three sets of runner
 groups. The three runner group names are ``llvm-premerge-libcxx-runners``,
 ``llvm-premerge-libcxx-release-runners`` and ``llvm-premerge-libcxx-next-runners``.
-The runner set to use is controlled by the contents of
+The runner set currently in use is controlled by the contents of
 https://github.com/llvm/llvm-project/blob/main/.github/workflows/libcxx-build-and-test.yaml.
 By default, it uses ``llvm-premerge-libcxx-runners``. To switch to one of the
 other runner sets, just replace all uses of ``llvm-premerge-libcxx-runners`` in
 the yaml file with the desired runner set.
 
-Which container image is used by these three runner sets is controlled
-and set by the variable values in
-https://github.com/llvm/llvm-zorg/blob/main/premerge/premerge_resources/variables.tf.
-The table below shows the variable names and
-the runner sets to which they correspond. To see their values, follow the
-link above (to ``variables.tf`` in llvm-zorg).
-
-+------------------------------------+---------------------------+
-|Runner Set                          |Variable                   |
-+====================================+===========================+
-|llvm-premerge-libcxx-runners        |libcxx_runner_image        |
-+------------------------------------+---------------------------+
-|llvm-premerge-libcxx-release-runners|libcxx_release_runner_image|
-+------------------------------------+---------------------------+
-|llvm-premerge-libcxx-next-runners   |libcxx_next_runner_image   |
-+------------------------------------+---------------------------+
-
-
-When updating the container image you can either update just the runner binary (the part
+The container image used by these three runner sets is controlled by the contents
+of the corresponding text files in ``libcxx/utils/ci/images``. The content of these
+files is read by the `Terraform configuration in llvm-zorg
+<https://github.com/llvm/llvm-zorg/blob/main/premerge/premerge_resources/main.tf>`__.
+
+When updating the container image, you can either update just the runner binary (the part
 that connects to Github), or you can update everything (tools, etc.). To update the runner
 binary, bump the value of ``GITHUB_RUNNER_VERSION`` in ``libcxx/utils/ci/docker/docker-compose.yml``.
 To update all of the tools, bump ``BASE_IMAGE_VERSION`` to a newer version of the ``libcxx-linux-builder-base``
@@ -304,16 +291,9 @@ image. You can see all versions of that image at https://github.com/llvm/llvm-pr
 On push to ``main``, a new version of both the ``libcxx-linux-builder`` and the ``libcxx-android-builder``
 images will be built and pushed to https://github.com/llvm/llvm-project/packages.
 
-You can then update the image used by the actual runners by changing the sha associated
-to ``libcxx_runner_image``, ``libcxx_release_runner_image`` or ``libcxx_next_runner_image``
-in `the Terraform configuration file <https://github.com/llvm/llvm-zorg/blob/main/premerge/premerge_resources/variables.tf>`_.
-To do so, you will need to create a PR in the llvm-zorg repository and wait for it to be
-merged. Once that change has been merged, an LLVM premerge maintainer (a Google employee)
-must use terraform to apply the change to the running GKE cluster.
-
-.. note:: When you update the ``libcxx_runner_image``, also make sure to update the
-          ``libcxx/utils/ci/run-buildbot-container`` script to contain the new image.
-
+You can then update the image used by the actual runners by changing the image encoded in
+``libcxx/utils/ci/images`` and asking an LLVM premerge maintainer (a Google employee) to
+actually deploy the changes to the GKE cluster via Terraform.
 
 Monitoring premerge testing performance
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -336,7 +316,7 @@ your system.
 run-buildbot
 ~~~~~~~~~~~~
 
-Contains the build script executed on Buildkite. This script can be executed
+This is the script executed by the CI runners. This script can be executed
 locally or inside ``run-buildbot-container``. The script must be called with
 the target to test. For example, ``run-buildbot generic-cxx20`` will build
 libc++ and test it using C++20.
diff --git a/libcxx/utils/ci/images/libcxx_next_runners.txt b/libcxx/utils/ci/images/libcxx_next_runners.txt
new file mode 100644
index 0000000000000..eafa0fe5ff6d8
--- /dev/null
+++ b/libcxx/utils/ci/images/libcxx_next_runners.txt
@@ -0,0 +1 @@
+ghcr.io/llvm/libcxx-linux-builder:55f634cc7f345a8708f2497c0274144ab1eb3d91
\ No newline at end of file
diff --git a/libcxx/utils/ci/images/libcxx_release_runners.txt b/libcxx/utils/ci/images/libcxx_release_runners.txt
new file mode 100644
index 0000000000000..58f0718c1ce48
--- /dev/null
+++ b/libcxx/utils/ci/images/libcxx_release_runners.txt
@@ -0,0 +1 @@
+ghcr.io/llvm/libcxx-linux-builder:05b5090e961f6b45ba9144d4d846e0f174d0aedf
\ No newline at end of file
diff --git a/libcxx/utils/ci/images/libcxx_runners.txt b/libcxx/utils/ci/images/libcxx_runners.txt
new file mode 100644
index 0000000000000..4389aaa73866f
--- /dev/null
+++ b/libcxx/utils/ci/images/libcxx_runners.txt
@@ -0,0 +1 @@
+ghcr.io/llvm/libcxx-linux-builder:d6b22a347f813cf4a9832627323a43074f57bbcf
\ No newline at end of file
diff --git a/libcxx/utils/ci/run-buildbot-container b/libcxx/utils/ci/run-buildbot-container
index fa83d1db4f40f..2e5dac92a066b 100755
--- a/libcxx/utils/ci/run-buildbot-container
+++ b/libcxx/utils/ci/run-buildbot-container
@@ -26,6 +26,7 @@ if [[ ! -d "${MONOREPO_ROOT}/libcxx/utils/ci" ]]; then
     echo "Was unable to find the root of the LLVM monorepo; are you running from within the monorepo?"
     exit 1
 fi
-docker pull ghcr.io/llvm/libcxx-linux-builder:d6b22a347f813cf4a9832627323a43074f57bbcf
-docker run -it --volume "${MONOREPO_ROOT}:/llvm" --workdir "/llvm" --cap-add=SYS_PTRACE ghcr.io/llvm/libcxx-linux-builder:d6b22a347f813cf4a9832627323a43074f57bbcf \
+image="$(cat ${MONOREPO_ROOT}/libcxx/utils/ci/images/libcxx_runners.txt)"
+docker pull ${image}
+docker run -it --volume "${MONOREPO_ROOT}:/llvm" --workdir "/llvm" --cap-add=SYS_PTRACE ${image} \
     bash -c 'git config --global --add safe.directory /llvm ; exec bash'

[ldionne]

Along with llvm/llvm-zorg#670, this gives us a canonical place for storing the Docker image used by various pieces of libc++ infrastructure.