[CI] Tweak wording for builds with passing tests and build errors

[DavidSpickett]

"All tests passed" is too easily interpreted as every possible test was run and was fine. A lot of the time it means all the tests that didn't fail to build ran and were fine.

Maybe the wording is still too subtle but at least it hints to the idea that the tests run might be fewer than if the build had no compilation errors.

[llvmbot]

@llvm/pr-subscribers-infrastructure

Author: David Spickett (DavidSpickett)

Changes

"All tests passed" is too easily read as every possible test was run and was fine. A lot of the time it means all the tests that didn't fail to build ran and were fine.

Maybe the wording is still too subtle but at least it hints to the idea that the tests run might be fewer than if the build had no compilation errors.

---

Full diff: https://github.com/llvm/llvm-project/pull/171436.diff

2 Files Affected:

• (modified) .ci/generate_test_report_lib.py (+2-2)
• (modified) .ci/generate_test_report_lib_test.py (+4-4)

diff --git a/.ci/generate_test_report_lib.py b/.ci/generate_test_report_lib.py
index 9a4fc6030d5ec..5edde254eb73d 100644
--- a/.ci/generate_test_report_lib.py
+++ b/.ci/generate_test_report_lib.py
@@ -267,7 +267,7 @@ def plural(num_tests):
             report.extend(
                 [
                     "",
-                    "All tests passed but another part of the build **failed**. "
+                    "All executed tests passed, but another part of the build **failed**. "
                     "Information about the build failure could not be automatically "
                     "obtained.",
                     "",
@@ -278,7 +278,7 @@ def plural(num_tests):
             report.extend(
                 [
                     "",
-                    "All tests passed but another part of the build **failed**. Click on "
+                    "All executed tests passed, but another part of the build **failed**. Click on "
                     "a failure below to see the details.",
                     "",
                 ]
diff --git a/.ci/generate_test_report_lib_test.py b/.ci/generate_test_report_lib_test.py
index b9e992e0f798b..06279d672f3c3 100644
--- a/.ci/generate_test_report_lib_test.py
+++ b/.ci/generate_test_report_lib_test.py
@@ -343,7 +343,7 @@ def test_no_failures_build_failed(self):
 
               * 1 test passed
 
-              All tests passed but another part of the build **failed**. Information about the build failure could not be automatically obtained.
+              All executed tests passed, but another part of the build **failed**. Information about the build failure could not be automatically obtained.
 
               Download the build's log file to see the details.
               
@@ -390,7 +390,7 @@ def test_no_failures_build_failed_ninja_log(self):
 
                     * 1 test passed
 
-                    All tests passed but another part of the build **failed**. Click on a failure below to see the details.
+                    All executed tests passed, but another part of the build **failed**. Click on a failure below to see the details.
 
                     <details>
                     <summary>test/4.stamp</summary>
@@ -476,7 +476,7 @@ def test_no_failures_multiple_build_failed_ninja_log(self):
 
                     * 1 test passed
 
-                    All tests passed but another part of the build **failed**. Click on a failure below to see the details.
+                    All executed tests passed, but another part of the build **failed**. Click on a failure below to see the details.
 
                     <details>
                     <summary>touch test/2.stamp</summary>
@@ -978,7 +978,7 @@ def test_generate_report_end_to_end(self):
 
                     * 1 test passed
 
-                    All tests passed but another part of the build **failed**. Click on a failure below to see the details.
+                    All executed tests passed, but another part of the build **failed**. Click on a failure below to see the details.
 
                     <details>
                     <summary>test/4.stamp</summary>

[DavidSpickett]

I don't mind what it says as long as "all tests passed" is not literally in the text. It makes sense if you look more closely and think about it, but at a glance it's misleading. For example this one I saw today:

🪟 Windows x64 Test Results

• 2993 tests passed
• 30 tests skipped

All tests passed but another part of the build failed. Click on a failure below to see the details.

[boomanaiden154]

LGTM. Thanks.