Known Test Repository Issue Sometimes Misses Test Failures #6411
Description
Build
Build leg reported
Microsoft.DotNet.Cli.New.IntegrationTests.DotnetNewSearchTests.BasicTest
Pull Request
Known issue core information
Fill out the known issue JSON section by following the step by step documentation on how to create a known issue
{
"ErrorMessage" : "The SSL connection",
"BuildRetry": false,
"ErrorPattern": "",
"ExcludeConsoleLog": false
}So, I believe there is an issue with the known issue flagger, where it is failing to mark certain failures as known, even if they are a known issue. Please let me know if I am mistaken.
Here's the issue:
Looking at this image, this test was not considered a known issue:
https://github.com/dotnet/sdk/pull/51591/checks?check_run_id=54740778719
I believe that, because it's showing as a test failure, and other similar test failures are counted in the above section:
Furthermore, it appeared to be blocking build-analysis from being green.
Here's such an example of 1 of several 'known issues' that flag this issue:
#5015
dotnet/sdk#51574
#6410
As you can see in, #51574 looks for the string "The SSL connection could not be established" as a ErrorMessage which is supposed to use String.Contains. The test, however, that I think was not flagged as a 'known issue', did indeed have that error message string. It looks like some were flagged, while others were not. Here's the output of that 'unknown' test failure:
Expected command to exit with 0 but it did not.\nFile Name: /Users/runner/work/1/s/artifacts/bin/redist/Release/dotnet/dotnet\nArguments: new console --search --debug:custom-hive /Users/runner/work/1/s/artifacts/tmp/Release/dotnet-new.IntegrationTests/SharedHomeDirectory/20251106211951401\nExit Code: 103\nStdOut:\nWarning: use of 'dotnet new --search' is deprecated. Use 'dotnet new search' instead.\nFor more information, run: \n dotnet new search -h\n\nSearching for the templates...\nStdErr:\nMatches from template source: NuGet.org\nSearch failed: Failed to update search cache. (The SSL connection could not be established, see inner exception.) (The SSL connection could not be established, see inner exception.)\n\nFor details on the exit code, refer to https://aka.ms/templating-exit-codes#103\n [[Full log]](https://dev.azure.com/dnceng-public/public/_build/results?buildId=1196220&view=ms.vss-test-web.build-test-results-tab&runId=33106418&resultId=100166&paneView=debug)
While there may be a confounding variable in that there are now several known issues which are tracking substrings of each other, or may overlap, the whole reason there are multiple in the first place is because the first existing known test repository issue did not flag all of the tests which failed with the known issue.
Could this alternatively be a limitation of:
- A limit of how many tests can be flagged as a known issue for the same known issue
- A different 'failing configuration' which causes issues to be missed?
- A misunderstanding of which strings get searched? Note that I believe the error is also in the 'helix logs'.
- A failure of my own accord to file the known issue? @dotnet/dnceng
Release Note Category
- Feature changes/additions
- Bug fixes
- Internal Infrastructure Improvements
Release Note Description
Additional information about the issue reported
No response
Known issue validation
Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1196220
Error message validated: [The SSL Connection C]
Result validation: ❌ Known issue did not match with the provided build.
Validation performed at: 11/6/2025 10:39:11 PM UTC
Known issue validation
Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1196220
Error message validated: [The SSL connection]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 11/6/2025 10:42:09 PM UTC
Report
| Build | Definition | Step Name | Console log | Pull Request |
|---|---|---|---|---|
| 1196423 | dotnet/machinelearning | Build | Log | dotnet/machinelearning#7543 |
| 1196398 | dotnet/aspnetcore | Run build.sh | Log | dotnet/aspnetcore#64262 |
| 1196235 | dotnet/runtime | Send tests to Helix (Unix) | Log | dotnet/runtime#121356 |
| 1196186 | dotnet/runtime | Send to Helix (Unix) | Log | dotnet/runtime#121417 |
| 1196168 | dotnet/runtime | Send tests to Helix (Unix) | Log |
Summary
| 24-Hour Hit Count | 7-Day Hit Count | 1-Month Count |
|---|---|---|
| 7 | 7 | 7 |
Report
| Build | Definition | Step Name | Console log | Pull Request |
|---|---|---|---|---|
| 1233693 | dotnet/maui | 🛠️ Build Microsoft.Maui.sln | Log | |
| 1229346 | dotnet/maui | Run Helix Tests | Log | dotnet/maui#33109 |
| 1226023 | dotnet/sdk | 🟣 Build | Log | dotnet/sdk#52038 |
| 2852422 | dotnet-dotnet | Build | Log |
Summary
| 24-Hour Hit Count | 7-Day Hit Count | 1-Month Count |
|---|---|---|
| 0 | 4 | 64 |