Changeset 277406 in webkit

Timestamp:

May 12, 2021, 4:18:49 PM (5 years ago)

Author:

mmaxfield@apple.com

Message:

[MotionMark] Temporarily remove the focus test
​https://bugs.webkit.org/show_bug.cgi?id=225714

Reviewed by Geoffrey Garen.

The focus test is causing unacceptably high variance on test results. Initial investigation
indicates that this is related to secondary and tertiary interactions between the browser's
runloop and the compositor's runloop. Not only that, but the variable cost of the focus test
seems to affect the variance on the subtests which run after it.

This patch temporarily removes the focus test, just so we can get a stronger signal of
browsers' true performance, while we do a deeper analysis of the focus test.

Because the focus test's score is lower than the average score of the subtests, removing it
causes scores in all browsers to increase:

| WebKit | Chrome | Firefox

================================================
Score with Focus | 1020 | 550 | 962
Score without Focus | 1623 | 874 | 1101
Stddev with Focus | 9.3% | 2.3% | 2.9%
Stddev without Focus | 0.7% | 0.5% | 7.9%

• MotionMark/resources/runner/tests.js:

Location:

trunk/PerformanceTests

Files:

• ChangeLog (modified) (1 diff)
• MotionMark/resources/runner/tests.js (modified) (1 diff)

trunk/PerformanceTests/ChangeLog

@@ +1 @@
+2021-05-12  Myles C. Maxfield  <mmaxfield@apple.com>
+
+        [MotionMark] Temporarily remove the focus test
+        https://bugs.webkit.org/show_bug.cgi?id=225714
+
+        Reviewed by Geoffrey Garen.
+
+        The focus test is causing unacceptably high variance on test results. Initial investigation
+        indicates that this is related to secondary and tertiary interactions between the browser's
+        runloop and the compositor's runloop. Not only that, but the variable cost of the focus test
+        seems to affect the variance on the subtests which run after it.
+
+        This patch temporarily removes the focus test, just so we can get a stronger signal of
+        browsers' true performance, while we do a deeper analysis of the focus test.
+
+        Because the focus test's score is lower than the average score of the subtests, removing it
+        causes scores in all browsers to increase:
+
+                             | WebKit | Chrome | Firefox
+        ================================================
+        Score with Focus     |   1020 |    550 |    962
+        Score without Focus  |   1623 |    874 |   1101
+        Stddev with Focus    |   9.3% |   2.3% |   2.9%
+        Stddev without Focus |   0.7% |   0.5% |   7.9%
+
+        * MotionMark/resources/runner/tests.js:
+
 2021-04-19  Kimmo Kinnunen  <kkinnunen@apple.com>
 

trunk/PerformanceTests/MotionMark/resources/runner/tests.js

@@ -53 +53 @@
         },
         {
-            url: "master/focus.html",
-            name: "Focus"
-        },
-        {
             url: "master/image-data.html",
             name: "Images"