Changeset 213918 in webkit


Ignore:
Timestamp:
Mar 14, 2017 11:22:06 AM (7 years ago)
Author:
commit-queue@webkit.org
Message:

Nwtr ignores ImageDiff's errors for ref tests
https://bugs.webkit.org/show_bug.cgi?id=168033

Patch by Fujii Hironori <Fujii Hironori> on 2017-03-14
Reviewed by Alexey Proskuryakov.

Nwtr checks ImageDiff's errors only for pixel tests, but for ref
tests. Those errors of ref tests also should be checked.

In the current implementation of expected mismatch ref tests,
diff_image was called if the image hashes match. This is useless
because two images are ensured identical in that case. Calling
image_hash is considered unnecessary for expected mismatch ref
tests. Do not call diff_image for them.

As the result, check the error only for expected match ref tests.

  • Scripts/webkitpy/layout_tests/controllers/single_test_runner.py:

(SingleTestRunner._compare_image): Rename a variable 'err_str' to 'error_string'.
(SingleTestRunner._compare_output_with_reference): Do not call
diff_image for expected mismatch ref tests. Check the error and
marked the test failed for expected match ref tests.

Location:
trunk/Tools
Files:
2 edited

Legend:

Unmodified
Added
Removed
  • trunk/Tools/ChangeLog

    r213916 r213918  
     12017-03-14  Fujii Hironori  <Hironori.Fujii@sony.com>
     2
     3        Nwtr ignores ImageDiff's errors for ref tests
     4        https://bugs.webkit.org/show_bug.cgi?id=168033
     5
     6        Reviewed by Alexey Proskuryakov.
     7
     8        Nwtr checks ImageDiff's errors only for pixel tests, but for ref
     9        tests. Those errors of ref tests also should be checked.
     10
     11        In the current implementation of expected mismatch ref tests,
     12        diff_image was called if the image hashes match. This is useless
     13        because two images are ensured identical in that case. Calling
     14        image_hash is considered unnecessary for expected mismatch ref
     15        tests. Do not call diff_image for them.
     16
     17        As the result, check the error only for expected match ref tests.
     18
     19        * Scripts/webkitpy/layout_tests/controllers/single_test_runner.py:
     20        (SingleTestRunner._compare_image): Rename a variable 'err_str' to 'error_string'.
     21        (SingleTestRunner._compare_output_with_reference): Do not call
     22        diff_image for expected mismatch ref tests. Check the error and
     23        marked the test failed for expected match ref tests.
     24
    1252017-03-14  Brady Eidson  <beidson@apple.com>
    226
  • trunk/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py

    r212284 r213918  
    274274        elif driver_output.image_hash != expected_driver_output.image_hash:
    275275            diff_result = self._port.diff_image(expected_driver_output.image, driver_output.image)
    276             err_str = diff_result[2]
    277             if err_str:
    278                 _log.warning('  %s : %s' % (self._test_name, err_str))
     276            error_string = diff_result[2]
     277            if error_string:
     278                _log.warning('  %s : %s' % (self._test_name, error_string))
    279279                failures.append(test_failures.FailureImageHashMismatch())
    280                 driver_output.error = (driver_output.error or '') + err_str
     280                driver_output.error = (driver_output.error or '') + error_string
    281281            else:
    282282                driver_output.image_diff = diff_result[0]
     
    332332            failures.append(test_failures.FailureReftestNoImagesGenerated(reference_filename))
    333333        elif mismatch:
     334            # Calling image_hash is considered unnecessary for expected mismatch ref tests.
    334335            if reference_driver_output.image_hash == actual_driver_output.image_hash:
    335                 diff_result = self._port.diff_image(reference_driver_output.image, actual_driver_output.image, tolerance=0)
    336                 if not diff_result[0]:
    337                     failures.append(test_failures.FailureReftestMismatchDidNotOccur(reference_filename))
    338                 else:
    339                     _log.warning("  %s -> ref test hashes matched but diff failed" % self._test_name)
    340 
     336                failures.append(test_failures.FailureReftestMismatchDidNotOccur(reference_filename))
    341337        elif reference_driver_output.image_hash != actual_driver_output.image_hash:
     338            # ImageDiff has a hard coded color distance threshold even though tolerance=0 is specified.
    342339            diff_result = self._port.diff_image(reference_driver_output.image, actual_driver_output.image, tolerance=0)
    343             if diff_result[0]:
     340            error_string = diff_result[2]
     341            if error_string:
     342                _log.warning('  %s : %s' % (self._test_name, error_string))
     343                failures.append(test_failures.FailureReftestMismatch(reference_filename))
     344                actual_driver_output.error = (actual_driver_output.error or '') + error_string
     345            elif diff_result[0]:
    344346                failures.append(test_failures.FailureReftestMismatch(reference_filename))
    345347            else:
Note: See TracChangeset for help on using the changeset viewer.