Skip to content

Maestro: Add detail to error flows #7407

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
malmstein wants to merge 3 commits into
base: develop
Choose a base branch
from
Draft

Maestro: Add detail to error flows #7407

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
610f1d4
Maestro: Add detail to error flows
malmstein Dec 19, 2025
415c429
added task creation for all flows
malmstein Dec 19, 2025
171057e
lint issues
malmstein Dec 19, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 0 additions & 79 deletions .github/workflows/ads-end-to-end.yml
View file
Open in desktop

This file was deleted.

11 changes: 11 additions & 0 deletions .github/workflows/design-system-composable-pr-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ jobs:
run: find . -name "*internal-release*.apk" -exec mv '{}' apk/internal.apk \;

- name: Run Design System Maestro Tests
id: maestro-run
uses: mobile-dev-inc/action-maestro-cloud@v1.9.8
timeout-minutes: 30
with:
Expand All @@ -67,6 +68,16 @@ jobs:
workspace: .maestro
include-tags: androidDesignSystemTest

- name: Analyze Maestro Flow Results
if: always()
id: analyze-maestro
uses: ./.github/actions/maestro-flow-analyzer
with:
flow_results_json: ${{ steps.maestro-run.outputs.MAESTRO_CLOUD_FLOW_RESULTS }}
console_url: ${{ steps.maestro-run.outputs.MAESTRO_CLOUD_CONSOLE_URL }}
upload_status: ${{ steps.maestro-run.outputs.MAESTRO_CLOUD_UPLOAD_STATUS }}
app_binary_id: ${{ steps.maestro-run.outputs.MAESTRO_CLOUD_APP_BINARY_ID }}

- name: Comment on PR
if: failure()
uses: actions/github-script@v8
Expand Down
111 changes: 111 additions & 0 deletions .github/workflows/e2e-nightly-android-design-system.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
name: 'End to End Tests - Android Design System (Nightly)'

on:
schedule:
- cron: '0 5 * * *' # run at 5 AM UTC
workflow_dispatch:

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
instrumentation_tests:
runs-on: ubuntu-latest
name: End-to-End tests

steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
submodules: recursive

- name: Set up JDK version
uses: actions/setup-java@v4
with:
java-version-file: .github/.java-version
distribution: 'adopt'

- name: Create folder
if: always()
run: mkdir apk

- name: Decode keys
uses: davidSchuppa/base64Secret-toFile-action@v2
with:
secret: ${{ secrets.FAKE_RELEASE_PROPERTIES }}
fileName: ddg_android_build.properties
destination-path: $HOME/jenkins_static/com.duckduckgo.mobile.android/

- name: Decode key file
uses: davidSchuppa/base64Secret-toFile-action@v2
with:
secret: ${{ secrets.FAKE_RELEASE_KEY }}
fileName: android
destination-path: $HOME/jenkins_static/com.duckduckgo.mobile.android/

- name: Setup Gradle
uses: gradle/actions/setup-gradle@v3

- name: Assemble the project
run: ./gradlew assembleInternalRelease -Pforce-default-variant

- name: Move APK to new folder
if: always()
run: find . -name "*.apk" -exec mv '{}' apk/release.apk \;

- name: ADS Preview Flows
id: maestro-run
uses: mobile-dev-inc/action-maestro-cloud@v1.9.8
timeout-minutes: 120
with:
api-key: ${{ secrets.ROBIN_API_KEY }}
project-id: ${{ vars.ROBIN_ANDROID_PROJECT_ID }}
name: androidDesignSystem_${{ github.sha }}
timeout: ${{ vars.ROBIN_TIMEOUT_MINUTES }}
app-file: apk/release.apk
android-api-level: 30
workspace: .maestro
include-tags: androidDesignSystemTest

- name: Analyze Maestro Flow Results
if: always()
id: analyze-maestro
uses: ./.github/actions/maestro-flow-analyzer
with:
flow_results_json: ${{ steps.maestro-run.outputs.MAESTRO_CLOUD_FLOW_RESULTS }}
console_url: ${{ steps.maestro-run.outputs.MAESTRO_CLOUD_CONSOLE_URL }}
upload_status: ${{ steps.maestro-run.outputs.MAESTRO_CLOUD_UPLOAD_STATUS }}
app_binary_id: ${{ steps.maestro-run.outputs.MAESTRO_CLOUD_APP_BINARY_ID }}

- name: Show analyzer output
if: always()
run: echo "flow_summary_status=${{ steps.analyze-maestro.outputs.flow_summary_status }}"

- name: Create Asana task when Maestro tests failed
if: always() && steps.analyze-maestro.outputs.flow_summary_status == 'failure'
id: create-maestro-web-failure-task
uses: duckduckgo/native-github-asana-sync@v2.0
with:
asana-pat: ${{ secrets.ASANA_ACCESS_TOKEN }}
asana-project: ${{ vars.GH_ANDROID_APP_PROJECT_ID }}
asana-section: ${{ vars.GH_ANDROID_APP_INCOMING_SECTION_ID }}
asana-task-name: E2E Tests Failure - Android Design System
asana-task-description: |
Run: https://github.com/duckduckgo/Android/actions/runs/${{ github.run_id }}
Console: ${{ steps.maestro-run.outputs.MAESTRO_CLOUD_CONSOLE_URL }}
Summary:
${{ steps.analyze-maestro.outputs.flow_summary_message }}
action: 'create-asana-task'

- name: Create Asana task when workflow failed
if: ${{ failure() && steps.analyze-maestro.outputs.flow_summary_status != 'failure' }}
id: create-failure-task
uses: duckduckgo/native-github-asana-sync@v2.0
with:
asana-pat: ${{ secrets.ASANA_ACCESS_TOKEN }}
asana-project: ${{ vars.GH_ANDROID_APP_PROJECT_ID }}
asana-section: ${{ vars.GH_ANDROID_APP_INCOMING_SECTION_ID }}
asana-task-name: GH Workflow Failure - Android Design System End to End Tests (Nightly)
asana-task-description: Workflow execution has failed. See https://github.com/duckduckgo/Android/actions/runs/${{ github.run_id }}
action: 'create-asana-task'
70 changes: 66 additions & 4 deletions .github/workflows/e2e-nightly-autofill.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: Autofill Feature Critical Path End-to-End tests
name: 'End to End Tests - Autofill Critical Path (Nightly)'

on:
schedule:
Expand Down Expand Up @@ -55,6 +55,7 @@ jobs:
run: find . -name "*.apk" -exec mv '{}' apk/release.apk \;

- name: Autofill Critical Path E2E Flows
id: maestro-critical-path
uses: mobile-dev-inc/action-maestro-cloud@v1.9.8
timeout-minutes: 120
with:
Expand All @@ -67,7 +68,38 @@ jobs:
workspace: .maestro
include-tags: autofillNoAuthTests

- name: Analyze Maestro Flow Results (Critical Path)
if: always()
id: analyze-maestro-critical-path
uses: ./.github/actions/maestro-flow-analyzer
with:
flow_results_json: ${{ steps.maestro-critical-path.outputs.MAESTRO_CLOUD_FLOW_RESULTS }}
console_url: ${{ steps.maestro-critical-path.outputs.MAESTRO_CLOUD_CONSOLE_URL }}
upload_status: ${{ steps.maestro-critical-path.outputs.MAESTRO_CLOUD_UPLOAD_STATUS }}
app_binary_id: ${{ steps.maestro-critical-path.outputs.MAESTRO_CLOUD_APP_BINARY_ID }}

- name: Show analyzer output
if: always()
run: echo "flow_summary_status=${{ steps.analyze-maestro-critical-path.outputs.flow_summary_status }}"

- name: Create Asana task when Maestro tests failed (Critical Path)
if: always() && steps.analyze-maestro-critical-path.outputs.flow_summary_status == 'failure'
id: create-maestro-critical-failure-task
uses: duckduckgo/native-github-asana-sync@v2.0
with:
asana-pat: ${{ secrets.ASANA_ACCESS_TOKEN }}
asana-project: ${{ vars.GH_ANDROID_APP_PROJECT_ID }}
asana-section: ${{ vars.GH_ANDROID_APP_INCOMING_SECTION_ID }}
asana-task-name: E2E Tests Failure - Autofill Critical Path
asana-task-description: |
Run: https://github.com/duckduckgo/Android/actions/runs/${{ github.run_id }}
Console: ${{ steps.maestro-critical-path.outputs.MAESTRO_CLOUD_CONSOLE_URL }}
Summary:
${{ steps.analyze-maestro-critical-path.outputs.flow_summary_message }}
action: 'create-asana-task'

- name: Autofill web flows
id: maestro-web-flows
uses: mobile-dev-inc/action-maestro-cloud@v1.9.8
timeout-minutes: 120
with:
Expand All @@ -80,14 +112,44 @@ jobs:
workspace: .maestro
include-tags: autofillBackfillingUsername,autofillPasswordGeneration

- name: Analyze Maestro Flow Results (Web Flows)
if: always()
id: analyze-maestro-web-flows
uses: ./.github/actions/maestro-flow-analyzer
with:
flow_results_json: ${{ steps.maestro-web-flows.outputs.MAESTRO_CLOUD_FLOW_RESULTS }}
console_url: ${{ steps.maestro-web-flows.outputs.MAESTRO_CLOUD_CONSOLE_URL }}
upload_status: ${{ steps.maestro-web-flows.outputs.MAESTRO_CLOUD_UPLOAD_STATUS }}
app_binary_id: ${{ steps.maestro-web-flows.outputs.MAESTRO_CLOUD_APP_BINARY_ID }}

- name: Show analyzer output
if: always()
run: echo "flow_summary_status=${{ steps.analyze-maestro-web-flows.outputs.flow_summary_status }}"

- name: Create Asana task when Maestro tests failed (Web Flows)
if: always() && steps.analyze-maestro-web-flows.outputs.flow_summary_status == 'failure'
id: create-maestro-web-failure-task
uses: duckduckgo/native-github-asana-sync@v2.0
with:
asana-pat: ${{ secrets.ASANA_ACCESS_TOKEN }}
asana-project: ${{ vars.GH_ANDROID_APP_PROJECT_ID }}
asana-section: ${{ vars.GH_ANDROID_APP_INCOMING_SECTION_ID }}
asana-task-name: E2E Tests Failure - Autofill Web Flows
asana-task-description: |
Run: https://github.com/duckduckgo/Android/actions/runs/${{ github.run_id }}
Console: ${{ steps.maestro-web-flows.outputs.MAESTRO_CLOUD_CONSOLE_URL }}
Summary:
${{ steps.analyze-maestro-web-flows.outputs.flow_summary_message }}
action: 'create-asana-task'

- name: Create Asana task when workflow failed
if: ${{ failure() }}
if: ${{ failure() && steps.analyze-maestro-critical-path.outputs.flow_summary_status != 'failure' && steps.analyze-maestro-web-flows.outputs.flow_summary_status != 'failure' }}
id: create-failure-task
uses: duckduckgo/native-github-asana-sync@v2.0
with:
asana-pat: ${{ secrets.ASANA_ACCESS_TOKEN }}
asana-project: ${{ vars.GH_ANDROID_APP_PROJECT_ID }}
asana-section: ${{ vars.GH_ANDROID_APP_INCOMING_SECTION_ID }}
asana-task-name: GH Workflow Failure - Autofill E2E Flows (Robin)
asana-task-description: Autofill tests have failed. See https://github.com/duckduckgo/Android/actions/runs/${{ github.run_id }}
asana-task-name: GH Workflow Failure - Autofill Critical Path End to End Tests (Nightly)
asana-task-description: Workflow execution has failed. See https://github.com/duckduckgo/Android/actions/runs/${{ github.run_id }}
action: 'create-asana-task'
45 changes: 41 additions & 4 deletions .github/workflows/custom-tabs-nightly.yml → ...hub/workflows/e2e-nightly-custom-tabs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: Custom Tabs nightly tests
name: 'End to End Tests - Custom Tabs (Nightly)'

on:
schedule:
Expand Down Expand Up @@ -55,6 +55,7 @@ jobs:
run: find . -name "*.apk" -exec mv '{}' apk/release.apk \;

- name: Custom Tabs Flows
id: maestro-run
uses: mobile-dev-inc/action-maestro-cloud@v1.9.8
timeout-minutes: 120
with:
Expand All @@ -67,16 +68,52 @@ jobs:
workspace: .maestro
include-tags: customTabsTest

- name: Analyze Maestro Flow Results
if: always()
id: analyze-maestro
uses: ./.github/actions/maestro-flow-analyzer
with:
flow_results_json: ${{ steps.maestro-run.outputs.MAESTRO_CLOUD_FLOW_RESULTS }}
console_url: ${{ steps.maestro-run.outputs.MAESTRO_CLOUD_CONSOLE_URL }}
upload_status: ${{ steps.maestro-run.outputs.MAESTRO_CLOUD_UPLOAD_STATUS }}
app_binary_id: ${{ steps.maestro-run.outputs.MAESTRO_CLOUD_APP_BINARY_ID }}

- name: Create Asana task when Maestro tests failed
if: always() && steps.analyze-maestro.outputs.flow_summary_status == 'failure'
id: create-maestro-failure-task
uses: duckduckgo/native-github-asana-sync@v2.0
with:
asana-pat: ${{ secrets.ASANA_ACCESS_TOKEN }}
asana-project: ${{ vars.GH_ANDROID_APP_PROJECT_ID }}
asana-section: ${{ vars.GH_ANDROID_APP_INCOMING_SECTION_ID }}
asana-task-name: E2E Tests Failure - Custom Tabs
asana-task-description: |
Run: https://github.com/duckduckgo/Android/actions/runs/${{ github.run_id }}
Console: ${{ steps.maestro-run.outputs.MAESTRO_CLOUD_CONSOLE_URL }}
Summary:
${{ steps.analyze-maestro.outputs.flow_summary_message }}
action: 'create-asana-task'

- name: Adding as a release blocker
if: ${{ steps.create-maestro-failure-task.outputs.taskId != '' }}
uses: duckduckgo/native-github-asana-sync@v2.0
with:
asana-pat: ${{ secrets.ASANA_ACCESS_TOKEN }}
asana-project: ${{ vars.GH_ANDROID_APP_RELEASES_PROJECT_ID }}
asana-section: ${{ vars.GH_ANDROID_APP_RELEASES_PROJECT_BLOCKER_SECTION_ID }}
asana-task-id: ${{ steps.create-maestro-failure-task.outputs.taskId }}
action: 'add-task-asana-project'

- name: Create Asana task when workflow failed
if: ${{ failure() }}
if: ${{ failure() && steps.analyze-maestro.outputs.flow_summary_status != 'failure' }}
id: create-failure-task
uses: duckduckgo/native-github-asana-sync@v2.0
with:
asana-pat: ${{ secrets.ASANA_ACCESS_TOKEN }}
asana-project: ${{ vars.GH_ANDROID_APP_PROJECT_ID }}
asana-section: ${{ vars.GH_ANDROID_APP_INCOMING_SECTION_ID }}
asana-task-name: GH Workflow Failure - Custom Tabs Flows (Robin)
asana-task-description: The Custom Tabs nightly workflow has failed. See https://github.com/duckduckgo/Android/actions/runs/${{ github.run_id }}
asana-task-name: GH Workflow Failure - Custom Tabs End to End Tests (Nightly)
asana-task-description: Workflow execution has failed. See https://github.com/duckduckgo/Android/actions/runs/${{ github.run_id }}
action: 'create-asana-task'

- name: Adding as a release blocker
Expand Down
Loading