Changeset 212237 in webkit

Timestamp:

Feb 13, 2017 11:58:38 AM (7 years ago)

Author:

commit-queue@webkit.org

Message:

Nwtr unexpectedly passes mismatch ref test if the hashes doesn't match but no diff
​https://bugs.webkit.org/show_bug.cgi?id=168221

Patch by Fujii Hironori <Fujii Hironori> on 2017-02-13
Reviewed by Ryosuke Niwa.

There is a logic error in comparing mismatch ref test images. In
mismatch ref tests, it should be failed if two images has no diff.
But, if the hashes are different, nwtr unexpectedly pass the
mismatch ref test.

• Scripts/webkitpy/layout_tests/controllers/single_test_runner.py:

(SingleTestRunner._compare_output_with_reference): Fail the
mismatch ref test if hashes of two images are equal. Invoke
ImageDiff if the hashes don't match.

Location:

trunk/Tools

Files:

• ChangeLog (modified) (1 diff)
• Scripts/webkitpy/layout_tests/controllers/single_test_runner.py (modified) (1 diff)

trunk/Tools/ChangeLog

@@ +1 @@
+2017-02-13  Fujii Hironori  <Hironori.Fujii@sony.com>
+
+        Nwtr unexpectedly passes mismatch ref test if the hashes doesn't match but no diff
+        https://bugs.webkit.org/show_bug.cgi?id=168221
+
+        Reviewed by Ryosuke Niwa.
+
+        There is a logic error in comparing mismatch ref test images. In
+        mismatch ref tests, it should be failed if two images has no diff.
+        But, if the hashes are different, nwtr unexpectedly pass the
+        mismatch ref test.
+
+        * Scripts/webkitpy/layout_tests/controllers/single_test_runner.py:
+        (SingleTestRunner._compare_output_with_reference): Fail the
+        mismatch ref test if hashes of two images are equal. Invoke
+        ImageDiff if the hashes don't match.
+
 2017-02-13  Myles C. Maxfield  <mmaxfield@apple.com>
 

trunk/Tools/Scripts/webkitpy/layout_tests/controllers/single_test_runner.py

@@ -332 +332 @@
             failures.append(test_failures.FailureReftestNoImagesGenerated(reference_filename))
         elif mismatch:
-            if reference_driver_output.image_hash == actual_driver_output.image_hash:
+            if reference_driver_output.image_hash != actual_driver_output.image_hash:
                 diff_result = self._port.diff_image(reference_driver_output.image, actual_driver_output.image, tolerance=0)
                 if not diff_result[0]:
                     failures.append(test_failures.FailureReftestMismatchDidNotOccur(reference_filename))
-                else:
-                    _log.warning("  %s -> ref test hashes matched but diff failed" % self._test_name)
+            else:
+                failures.append(test_failures.FailureReftestMismatchDidNotOccur(reference_filename))
 
         elif reference_driver_output.image_hash != actual_driver_output.image_hash: