Add failure summary to ConsoleOutputRecorder #1420

tienquocbui · 2025-11-13T22:13:22Z

Overview

This PR implements a failure summary section for ConsoleOutputRecorder to help users quickly locate and review test failures in large test suites.

Closes: #1355

Architectural Changes Implemented

State Management

Uses existing HumanReadableOutputRecorder state with single Locked<Context>
Fits naturally with existing Graph traversal logic
No additional state in ConsoleOutputRecorder (stateless design preserved)

Data Structure

Added lightweight TestData.IssueInfo struct with: sourceLocation, description, isKnown, severity
Issues stored as array in TestData.issues (each issue has its own source location)
Test metadata (displayName, testCaseArguments) stored once per test in TestData
Existing issueCount dictionary preserved for efficient parallel tracking

New TestRunSummary Type

Dedicated type for failure summary generation
Breaks down logic into focused methods: header(), formatFailedTest(), fullyQualifiedName(), formatIssue()
Collects failures from Graph and formats them independently
fileprivate initializer uses HumanReadableOutputRecorder.Context types

API Improvements

generateFailureSummary() returns Optional<String> (nil when no failures)
Uses expandedDescription() for normal output, expandedDebugDescription() when verbose
ConsoleOutputRecorder calls this at runEnded event with blank line spacing

Output Features

Fully qualified suite/test paths (addresses Add failure summary section to console output for easier debugging #1355)
Custom display names shown in quotes (e.g., "Custom Display Name")
Parameterized test arguments displayed once per test (e.g., (value → 7))
Each issue listed with indentation, description, and source location
Visual separation with blank lines before and after summary

Example Output

Regular Test Failure:

✗  TestingTests/FailureSummaryDemoTests/MathTests/"Division fails"
  - (result: Swift.Int → 3) == (4: Swift.Int → 4)
    at TestingTests/FailureSummaryDemoTests.swift:26

Parameterized Test Failure (shows which argument failed):

✗  TestingTests/FailureSummaryDemoTests/ParameterizedTests/testEvenNumbers(value:)/"Check even numbers"
  (value → 7)
  - (value % 2: Swift.Int → 1) == (0: Swift.Int → 0)
    at TestingTests/FailureSummaryDemoTests.swift:72

Test with Custom Display Name:

✗  TestingTests/FailureSummaryDemoTests/ParameterizedTests/"This is a custom display name"
  - (value: Swift.Int → 42) == (100: Swift.Int → 100)
    at TestingTests/FailureSummaryDemoTests.swift:83

Complete Test Output

Click to expand full test run output

(base) buitienquoc@Kelvins-MacBook-Pro swift-testing % swift test --filter "FailureSummaryDemoTests" --disable-xctest
[1/1] Planning build
Building for debugging...
[257/257] Linking swift-testingPackageTests
Build complete! (39.83s)
􀟈  Test run started.
􀄵  Testing Library Version: 6.3-dev (d8b140d780dc2da - modified)
􀟈  Suite "Failure Summary Demo" started.
􀟈  Suite "Math Operations" started.
􀟈  Suite "String Operations" started.
􀟈  Test "Addition works correctly" started.
􀟈  Test "String comparison passes" started.
􀟈  Test "Multiple string failures" started.
􀟈  Test "String concatenation fails" started.
􀟈  Suite "Array Operations" started.
􀟈  Test "Division fails" started.
􀟈  Test "Array equality" started.
􀟈  Suite "Parameterized Tests" started.
􁁛  Test "Addition works correctly" passed after 0.001 seconds.
􁁛  Test "String comparison passes" passed after 0.001 seconds.
􁁛  Test "Array equality" passed after 0.001 seconds.
􀟈  Test "Array contains element" started.
􀟈  Test "This is a custom display name" started.
􀟈  Test "Check even numbers" started.
􀟈  Test "String length validation" started.
􀟈  Test case passing 1 argument value → 4 to "Check even numbers" started.
􀟈  Test case passing 1 argument value → 6 to "Check even numbers" started.
􀟈  Test case passing 1 argument value → 7 to "Check even numbers" started.
􀟈  Test case passing 1 argument value → 10 to "Check even numbers" started.
􀟈  Test case passing 1 argument text → "hello" to "String length validation" started.
􀟈  Test case passing 1 argument text → "world" to "String length validation" started.
􀟈  Test case passing 1 argument text → "a" to "String length validation" started.
􀟈  Test case passing 1 argument value → 2 to "Check even numbers" started.
􀢄  Test "Multiple string failures" recorded an issue at FailureSummaryDemoTests.swift:46:7: Expectation failed: "a" == "b"
􀢄  Test "String concatenation fails" recorded an issue at FailureSummaryDemoTests.swift:36:7: Expectation failed: (greeting + name → "HelloWorld") == "Hello World"
􀢄  Test "This is a custom display name" recorded an issue at FailureSummaryDemoTests.swift:83:7: Expectation failed: (value → 42) == 100
􀢄  Test "String length validation" recorded an issue with 1 argument text → "a" at FailureSummaryDemoTests.swift:77:7: Expectation failed: (text.count → 1) >= 5
􀢄  Test "Array contains element" recorded an issue at FailureSummaryDemoTests.swift:57:7: Expectation failed: (numbers → [1, 2, 3, 4, 5]).contains(10)
􀢄  Test "Multiple string failures" recorded an issue at FailureSummaryDemoTests.swift:47:7: Expectation failed: ("hello".count → 5) == 10
􀢄  Test "Division fails" recorded an issue at FailureSummaryDemoTests.swift:26:7: Expectation failed: (result → 3) == 4
􀢄  Test "Check even numbers" recorded an issue with 1 argument value → 7 at FailureSummaryDemoTests.swift:72:7: Expectation failed: (value % 2 → 1) == 0
􀢄  Test "This is a custom display name" failed after 0.003 seconds with 1 issue.
􀢄  Test "Array contains element" failed after 0.004 seconds with 1 issue.
􀟈  Test case passing 1 argument text → "swift" to "String length validation" started.
􀢄  Test "Multiple string failures" recorded an issue at FailureSummaryDemoTests.swift:48:7: Expectation failed: ("swift".uppercased() → "SWIFT") == "swift"
􀢄  Test "String concatenation fails" failed after 0.005 seconds with 1 issue.
􀢄  Suite "Array Operations" failed after 0.005 seconds with 1 issue.
􀢄  Test "Division fails" failed after 0.005 seconds with 1 issue.
􀢄  Test "String length validation" with 4 test cases failed after 0.004 seconds with 1 issue.
􀢄  Test "Check even numbers" with 5 test cases failed after 0.004 seconds with 1 issue.
􀢄  Suite "Math Operations" failed after 0.005 seconds with 1 issue.
􀢄  Test "Multiple string failures" failed after 0.005 seconds with 3 issues.
􀢄  Suite "Parameterized Tests" failed after 0.005 seconds with 3 issues.
􀢄  Suite "String Operations" failed after 0.005 seconds with 4 issues.
􀢄  Suite "Failure Summary Demo" failed after 0.007 seconds with 9 issues.
􀄵  /// Demo tests showcasing the refactored failure summary feature.

Test run had 7 failed tests with 9 issues:
􀢄  TestingTests/FailureSummaryDemoTests/MathTests/"Division fails"
  - (result: Swift.Int → 3) == (4: Swift.Int → 4)
    at TestingTests/FailureSummaryDemoTests.swift:26
􀢄  TestingTests/FailureSummaryDemoTests/ArrayTests/"Array contains element"
  - (numbers: Swift.Array<Swift.Int> → [1, 2, 3, 4, 5]).contains(10: Swift.Int → 10)
    at TestingTests/FailureSummaryDemoTests.swift:57
􀢄  TestingTests/FailureSummaryDemoTests/ParameterizedTests/testEvenNumbers(value:)/"Check even numbers"
  (value → 7)
  - (value % 2: Swift.Int → 1) == (0: Swift.Int → 0)
    at TestingTests/FailureSummaryDemoTests.swift:72
􀢄  TestingTests/FailureSummaryDemoTests/ParameterizedTests/testStringLength(text:)/"String length validation"
  (text → "a")
  - (text.count: Swift.Int → 1) >= (5: Swift.Int → 5)
    at TestingTests/FailureSummaryDemoTests.swift:77
􀢄  TestingTests/FailureSummaryDemoTests/ParameterizedTests/"This is a custom display name"
  - (value: Swift.Int → 42) == (100: Swift.Int → 100)
    at TestingTests/FailureSummaryDemoTests.swift:83
􀢄  TestingTests/FailureSummaryDemoTests/StringTests/"Multiple string failures"
  - ("a": Swift.String → "a") == ("b": Swift.String → "b")
    at TestingTests/FailureSummaryDemoTests.swift:46
  - ("hello".count: Swift.Int → 5) == (10: Swift.Int → 10)
    at TestingTests/FailureSummaryDemoTests.swift:47
  - ("swift".uppercased(): Swift.String → "SWIFT") == ("swift": Swift.String → "swift")
    at TestingTests/FailureSummaryDemoTests.swift:48
􀢄  TestingTests/FailureSummaryDemoTests/StringTests/"String concatenation fails"
  - (greeting + name: Swift.String → "HelloWorld") == ("Hello World": Swift.String → "Hello World")
    at TestingTests/FailureSummaryDemoTests.swift:36

􀢄  Test run with 10 tests in 5 suites failed after 0.007 seconds with 9 issues.
(base) buitienquoc@Kelvins-MacBook-Pro swift-testing %

…failure summary using expandedDebugDescription for detailed output, with support for custom display names and parameterized tests.

grynspan · 2025-11-14T04:45:37Z

Would it make sense to factor the new code into a dedicated TestRunSummary type? (I'm not saying you must, just asking.)

Sources/Testing/Events/Recorder/Event.HumanReadableOutputRecorder.swift

…logic into TestRunSummary type

tienquocbui · 2025-11-18T14:53:52Z

@grynspan thanks for the suggestion! I discussed that with Stuart and I've refactored the failure summary logic into a dedicated TestRunSummary type. Would you mind taking another look when you get a chance? Thanks!

…y is shown, with the default behavior being enabled

… message so that it uses the 'test(s) which recorded N issue(s)" terminology

…tringConvertible conformance instead of fileID:line

stmontgomery

We met about this offline, and I made a handful of commits of my own which I've just pushed to the PR branch after discussion. Beyond that, there is one problem in how the fully-qualified name is formed for parameterized test functions, and cases within them, in the failure summary section, and @tienquocbui it sounds like you will address that (thanks!)

stmontgomery · 2025-12-02T17:41:04Z

Sources/Testing/Events/Recorder/Event.HumanReadableOutputRecorder.swift

+          // Convert Context.TestData.IssueInfo to TestRunSummary.IssueInfo
+          let issues = testData.issues.map { issue in


Context.TestData.IssueInfo appears to be identical to TestRunSummary.IssueInfo, can we get rid of the latter and use the former in both places? You could include a typealias if it's helpful

Done! I've removed the duplicate TestRunSummary.IssueInfo struct and now use Context.TestData.IssueInfo directly throughout TestRunSummary.

… with proper indentation hierarchy

tienquocbui · 2025-12-03T16:48:32Z

@stmontgomery I've addressed the parameterized test display issue. Here's what changed:

Before:

✗ FailureSummaryDemoTests/ParameterizedTests/testEvenNumbers(value:)/"Check even numbers"
(value → 7)
   - (value % 2 → 1) == 0
        at FailureSummaryDemoTests.swift:70:13

After:

✗ FailureSummaryDemoTests/ParameterizedTests/"Check even numbers"
(value → 7)
   - (value % 2 → 1) == 0
        at FailureSummaryDemoTests.swift:70:13

Add failure summary to ConsoleOutputRecorder. Implemented end-of-run …

73f87ae

…failure summary using expandedDebugDescription for detailed output, with support for custom display names and parameterized tests.

tienquocbui requested review from briancroom, grynspan, jerryjrchen and stmontgomery as code owners November 13, 2025 22:13

grynspan reviewed Nov 14, 2025

View reviewed changes

Sources/Testing/Events/Recorder/Event.HumanReadableOutputRecorder.swift Show resolved Hide resolved

aluco100 reviewed Nov 15, 2025

View reviewed changes

Sources/Testing/Events/Recorder/Event.HumanReadableOutputRecorder.swift Outdated Show resolved Hide resolved

stmontgomery reviewed Nov 17, 2025

View reviewed changes

Add severity tracking to IssueInfo struct and factor failure summary …

29c448f

…logic into TestRunSummary type

tienquocbui force-pushed the gsoc-development4 branch from d533995 to 29c448f Compare November 18, 2025 14:47

tienquocbui requested review from grynspan and stmontgomery November 18, 2025 14:54

stmontgomery assigned tienquocbui Nov 18, 2025

stmontgomery added enhancement New feature or request issue-handling Related to Issue handling within the testing library command-line experience ⌨️ enhancements to the command line interface gsoc ☀️ Google Summer of Code contributions labels Nov 18, 2025

stmontgomery added this to the Swift 6.3.0 milestone Nov 18, 2025

stmontgomery added this to [Swift Testing] Console Experience Nov 18, 2025

stmontgomery added 9 commits December 2, 2025 10:02

nit: whitespace

893529d

Fix warning about unnecessary var

416bdf4

Include an environment variable to control whether the failure summar…

42d582d

…y is shown, with the default behavior being enabled

TestRunSummary doesn't need to be @_spi public, it can be private

de83b87

Use the counting(_:) utility to pluralize nouns

69ac56e

Correct comment about "debug" description, specifically

da879e8

Fix error in my earlier change which adopted 'counting', and rephrase…

8bcee7a

… message so that it uses the 'test(s) which recorded N issue(s)" terminology

fixup: whitespace

21e91c5

Simplify by beginning variable with initial value

fe58065

stmontgomery added 6 commits December 2, 2025 11:13

Omit the module name on complete paths for tests

c83ae9b

Indent source location of issues one level deeper and use the CustomS…

a46b1c5

…tringConvertible conformance instead of fileID:line

Use severity >= .error instead of == .error

e6558c5

Refactor to simplify and consolidate some logic in fullyQualifiedName()

b3ba640

whitespace

46c077b

Remove unnecessary 'public' access level

3acb802

stmontgomery reviewed Dec 2, 2025

View reviewed changes

Fix parameterized test display and group test cases under parent test…

44e71d4

… with proper indentation hierarchy

tienquocbui requested a review from stmontgomery December 3, 2025 16:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add failure summary to ConsoleOutputRecorder #1420

Add failure summary to ConsoleOutputRecorder #1420

Uh oh!

tienquocbui commented Nov 13, 2025 •

edited

Loading

Uh oh!

grynspan commented Nov 14, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tienquocbui commented Nov 18, 2025

Uh oh!

stmontgomery left a comment

Uh oh!

stmontgomery Dec 2, 2025

Uh oh!

tienquocbui Dec 3, 2025

Uh oh!

tienquocbui commented Dec 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		// Convert Context.TestData.IssueInfo to TestRunSummary.IssueInfo
		let issues = testData.issues.map { issue in

Add failure summary to ConsoleOutputRecorder #1420

Are you sure you want to change the base?

Add failure summary to ConsoleOutputRecorder #1420

Uh oh!

Conversation

tienquocbui commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Architectural Changes Implemented

Example Output

Complete Test Output

Uh oh!

grynspan commented Nov 14, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tienquocbui commented Nov 18, 2025

Uh oh!

stmontgomery left a comment

Choose a reason for hiding this comment

Uh oh!

stmontgomery Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

tienquocbui Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

tienquocbui commented Dec 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tienquocbui commented Nov 13, 2025 •

edited

Loading