[E2E Test] Add retry logic for transient pcluster CLI errors in test_build_image #7186
+102
−22
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add retry mechanism to handle transient CI environment issues in build-image and get-image-stack-events commands. These errors include: - Package metadata issues (error message contains only 'aws-parallelcluster') - Infrastructure creation failures with package loading problems - ModuleNotFoundError for pcluster modules - pyenv command not found errors Retries up to 3 times with 30 second delays between attempts.
hehe7318
added
the
skip-changelog-update
gmarciani
reviewed
Jan 7, 2026
| TRANSIENT_RETRY_DELAY_SECONDS = 30 | ||
|
|
||
|
|
||
| def _is_transient_pcluster_error(stdout: str = None, stderr: str = None) -> bool: |
Contributor
There was a problem hiding this comment.
Let's first understand the root cause of this transient failure before covering it with a retry. A retry could mask an issue that could have impacts on other parts of our CICD infra.
Contributor
Author
There was a problem hiding this comment.
The retry is only for build-image, and only for specific error pattern. It will not mask issues on other parts. And we only noticed these kind of issues on build-image. I think we can make it stable first and in parallel dive deep to the root cause. We already have enough test samples for our research. What do you think?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes
Add retry mechanism to handle transient CI environment issues in build-image and get-image-stack-events commands. These errors include:
Retries up to 3 times with 30 second delays between attempts.
Tests
Checklist
developadd the branch name as prefix in the PR title (e.g.[release-3.6]).Please review the guidelines for contributing and Pull Request Instructions.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.