Clarify behaviour

In hybrid mode only hashes can affect result. Clarify what `--mpl-results-always` does and when it's used.

Author: ConorMacBride

README.rst

@@ -75,8 +75,8 @@ directly in the right directory.
 With a Hash Library
 ^^^^^^^^^^^^^^^^^^^
 
-Instead of comparing to baseline images, you can instead compare against a json
-library of sha256 hashes. This has the advantage of not having to check baseline
+Instead of comparing to baseline images, you can instead compare against a JSON
+library of SHA-256 hashes. This has the advantage of not having to check baseline
 images into the repository with the tests, or download them from a remote
 source.
 
@@ -91,8 +91,11 @@ Hybrid Mode: Hashes and Images
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 It is possible to configure both hashes and baseline images. In this scenario
-the hashes will be compared first, with the baseline images only used if the hash
-comparison fails.
+only the hash comparison can determine the test result. If the hash comparison
+fails, the test will fail, however a comparison to the baseline image will be
+carried out so the actual difference can be seen. If the hash comparison passes,
+the comparison to the baseline image is skipped (unless **results always** is
+configured).
 
 This is especially useful if the baseline images are external to the repository
 containing the tests, and are accessed via HTTP. In this situation, if the hashes
@@ -104,7 +107,7 @@ without having to modify the external images.
 Running Tests
 ^^^^^^^^^^^^^
 
-Once tests are written with either baseline images or a hash library to compare
+Once tests are written with baseline images, a hash library, or both to compare
 against, the tests can be run with::
 
     pytest --mpl
@@ -118,12 +121,15 @@ Generating a Test Summary
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 
 By specifying the ``--mpl-generate-summary=html`` CLI argument, a HTML summary
-page will be generated showing the result, log entry and RMS of each test,
-and the hashes if configured. The baseline, diff and result image for each
-failing test will be shown. If **Results always** is configured
-(see section below), images for passing tests will also be shown.
-If no baseline images are configured, just the result images will
-be displayed.
+page will be generated showing the test result, log entry and generated result
+image. When in the (default) image comparison mode, the baseline image, diff
+image and RMS (if any), and tolerance of each test will also be shown.
+When in the hash comparison mode, the baseline hash and result hash will
+also be shown. When in hybrid mode, all of these are included.
+
+When generating a HTML summary, the ``--mpl-results-always`` option is
+automatically applied (see section below). Therefore images for passing
+tests will also be shown.
 
 +---------------+---------------+---------------+
 | |html all|    | |html filter| | |html result| |
@@ -199,20 +205,25 @@ decorator takes precedence.
 Results always
 ^^^^^^^^^^^^^^
 
-By default, result images are only generated for tests that fail.
+By default, result images are only saved for tests that fail.
 Passing ``--mpl-results-always`` to pytest will force result images
-to be generated for all tests, even for tests that pass.
-If a baseline image exists, a diff image will also be generated.
-All of these images will be shown in the summary (if requested).
+to be saved for all tests, even for tests that pass.
+
+When in **hybrid mode**, even if a test passes hash comparison,
+a comparison to the baseline image will also be carried out,
+with the baseline image and diff image (if image comparison fails)
+saved for all tests. This secondary comparison will not affect
+the success status of the test.
 
-This option is useful for always *comparing* the result images again
+This option is useful for always *comparing* the result images against
 the baseline images, while only *assessing* the tests against the
 hash library.
 If you only update your baseline images after merging a PR, this
 option means that the generated summary will always show how the
 PR affects the baseline images, with the success status of each
 test (based on the hash library) also shown in the generated
-summary.
+summary. This option is applied automatically when generating
+a HTML summary.
 
 Base style
 ^^^^^^^^^^

pytest_mpl/plugin.py

@@ -140,7 +140,7 @@ def pytest_addoption(parser):
     group.addoption('--mpl-results-path', help=results_path_help, action='store')
     parser.addini('mpl-results-path', help=results_path_help)
 
-    results_always_help = ("Always generate result images, not just for failed tests. "
+    results_always_help = ("Always compare to baseline images and save result images, even for passing tests. "
                            "This option is automatically applied when generating a HTML summary.")
     group.addoption('--mpl-results-always', action='store_true',
                     help=results_always_help)
@@ -272,7 +272,7 @@ def __init__(self,
             if len(unsupported_formats) > 0:
                 raise ValueError(f"The mpl summary type(s) '{sorted(unsupported_formats)}' "
                                  "are not supported.")
-            # Ignore `results_always` and always save result images for HTML output
+            # When generating HTML always apply `results_always`
             if generate_summary & {'html', 'basic-html'}:
                 results_always = True
         self.generate_summary = generate_summary