Changeset 230295 in webkit

Timestamp:

Apr 4, 2018 9:49:21 PM (6 years ago)

Author:

Dewei Zhu

Message:

Added UI to show potential regressions in chart with t-testing against segmentations.
​https://bugs.webkit.org/show_bug.cgi?id=184131

Reviewed by Ryosuke Niwa.

Added UI in the chart-pane so that user can use new option in trendline which not only
shows the segmentation, but also t-test against potential changes indicated by segmentation.

Fix a bug in AnalysisTaskPage that chart is not updated when change type of task changes.

• public/shared/statistics.js: Added a function to t-test certain range based on segmentation results.

(Statistics.supportedOneSideTTestProbabilities):
(Statistics.findRangesForChangeDetectionsWithWelchsTTest): The argument segmentations, every 2 items in the list defines
segmentation, that is why the index incremental is 2 in this funcion.

• public/v3/components/chart-pane-base.js: Will select the range if user clicks on a suggested annotation.

(ChartPaneBase.prototype.configure):
(ChartPaneBase.prototype._didClickAnnotation):

• public/v3/components/chart-styles.js:

(ChartStyles.annotationFillStyleForTask): Added 'annotationFillStyleForTask' to determine the fillStyle for annotation based on change type of a analysis task.

• public/v3/components/interactive-time-series-chart.js:

(InteractiveTimeSeriesChart.prototype._findAnnotation): Also need to search among suggested annotaions.

• public/v3/components/time-series-chart.js: Introduced 'suggested annotaion' which does not have an existing task and is suggested by t-test based on segmentation.

(TimeSeriesChart):
(TimeSeriesChart.prototype.setSuggestedAnnotations):
(TimeSeriesChart.prototype.allAnnotations): Returns both annotations with and without analysis task.
(TimeSeriesChart.prototype._layoutAnnotationBars): Should take all annotations in the calculation.

• public/v3/models/measurement-set.js:

(MeasurementSet.prototype.metricId): Returns metric id.

• public/v3/models/metric.js:

(Metric.prototype.summarizeForValues): Added helper function to summarize a given value

• public/v3/models/test-group.js:

(TestGroup.prototype.compareTestResults): Adapted to use 'Metric.summarizeForValues'.

• public/v3/pages/chart-pane.js: Added 'Segmentation with t-test analysis' to 'ChartTrendLineTypes'.

(ChartPane.prototype._renderTrendLinePopover):
(ChartPane.prototype.async._updateTrendLine): make it an async function.

• unit-tests/statistics-tests.js: Added unit tests for 'findRangesForChangeDetectionsWithWelchsTTest'.

Location:

trunk/Websites/perf.webkit.org

Files:

• ChangeLog (modified) (1 diff)
• browser-tests/interactive-time-series-chart-tests.js (modified) (2 diffs)
• browser-tests/time-series-chart-tests.js (modified) (2 diffs)
• public/shared/statistics.js (modified) (2 diffs)
• public/v3/components/chart-pane-base.js (modified) (8 diffs)
• public/v3/components/chart-styles.js (modified) (1 diff)
• public/v3/components/interactive-time-series-chart.js (modified) (1 diff)
• public/v3/models/measurement-set.js (modified) (2 diffs)
• public/v3/models/metric.js (modified) (1 diff)
• public/v3/models/test-group.js (modified) (1 diff)
• public/v3/pages/analysis-task-page.js (modified) (1 diff)
• public/v3/pages/chart-pane.js (modified) (5 diffs)
• unit-tests/statistics-tests.js (modified) (1 diff)

trunk/Websites/perf.webkit.org/ChangeLog

@@ +1 @@
+2018-03-29  Dewei Zhu  <dewei_zhu@apple.com>
+
+        Added UI to show potential regressions in chart with t-testing against segmentations.
+        https://bugs.webkit.org/show_bug.cgi?id=184131
+
+        Reviewed by Ryosuke Niwa.
+
+        Added UI in the chart-pane so that user can use new option in trendline which not only
+        shows the segmentation, but also t-test against potential changes indicated by segmentation.
+
+        Fix a bug in AnalysisTaskPage that chart is not updated when change type of task changes.
+
+        * browser-tests/interactive-time-series-chart-tests.js: Fix a unit tests.
+        * browser-tests/time-series-chart-tests.js: Fix a unit tests.
+        * public/shared/statistics.js: Added a function to t-test certain range based on segmentation results.
+        (Statistics.supportedOneSideTTestProbabilities):
+        (Statistics.findRangesForChangeDetectionsWithWelchsTTest): The argument `segmentations`, every 2 items in the list defines 
+        segmentation, that is why the index incremental is 2 in this funcion.
+        * public/v3/components/chart-pane-base.js: Will select the range if user clicks on a suggested annotation.
+        (ChartPaneBase.prototype.configure):
+        (ChartPaneBase.prototype._didClickAnnotation):
+        * public/v3/components/chart-styles.js:
+        (ChartStyles.annotationFillStyleForTask): Added 'annotationFillStyleForTask' to determine the fillStyle for annotation based on change type of a analysis task.
+        * public/v3/components/interactive-time-series-chart.js:
+        (InteractiveTimeSeriesChart.prototype._findAnnotation): Also need to search among suggested annotaions.
+        * public/v3/components/time-series-chart.js: Introduced 'suggested annotaion' which does not have an existing task and is suggested by t-test based on segmentation.
+        (TimeSeriesChart):
+        (TimeSeriesChart.prototype.setSuggestedAnnotations):
+        (TimeSeriesChart.prototype.allAnnotations): Returns both annotations with and without analysis task.
+        (TimeSeriesChart.prototype._layoutAnnotationBars): Should take all annotations in the calculation.
+        * public/v3/models/measurement-set.js:
+        (MeasurementSet.prototype.metricId): Returns metric id.
+        * public/v3/models/metric.js:
+        (Metric.prototype.summarizeForValues): Added helper function to summarize a given value
+        * public/v3/models/test-group.js:
+        (TestGroup.prototype.compareTestResults): Adapted to use 'Metric.summarizeForValues'.
+        * public/v3/pages/chart-pane.js: Added 'Segmentation with t-test analysis' to 'ChartTrendLineTypes'.
+        (ChartPane.prototype._renderTrendLinePopover):
+        (ChartPane.prototype.async._updateTrendLine): make it an async function.
+        * unit-tests/statistics-tests.js: Added unit tests for 'findRangesForChangeDetectionsWithWelchsTTest'.
+
 2018-04-02  Aakash Jain  <aakash_jain@apple.com>
 

trunk/Websites/perf.webkit.org/browser-tests/interactive-time-series-chart-tests.js

@@ -495 +495 @@
         return ChartTest.importChartScripts(context).then(() => {
             const chart = ChartTest.createInteractiveChartWithSampleCluster(context, null,
-                {annotations: { textStyle: '#000', textBackground: '#fff', minWidth: 3, barHeight: 10, barSpacing: 1}});
+                {annotations: { textStyle: '#000', textBackground: '#fff', minWidth: 3, barHeight: 10, barSpacing: 1, fillStyle: '#ccc'}});
 
             chart.setDomain(ChartTest.sampleCluster.startTime, ChartTest.sampleCluster.endTime);
@@ -506 +506 @@
                 endTime: ChartTest.sampleCluster.endTime - diff / 4,
                 label: 'hello, world',
-                fillStyle: 'rgb(0, 0, 255)',
-            }]
+            }];
             chart.setAnnotations(annotations);
 

trunk/Websites/perf.webkit.org/browser-tests/time-series-chart-tests.js

@@ -801 +801 @@
             const context = new BrowsingContext();
             return ChartTest.importChartScripts(context).then(() => {
-                const options = {annotations: { textStyle: '#000', textBackground: '#fff', minWidth: 3, barHeight: 7, barSpacing: 2}};
+                const options = {annotations: { textStyle: '#000', textBackground: '#fff', minWidth: 3, barHeight: 7, barSpacing: 2, fillStyle: '#ccc'}};
                 const chartWithoutAnnotations = ChartTest.createChartWithSampleCluster(context, null, options);
                 const chartWithAnnotations = ChartTest.createChartWithSampleCluster(context, null, options);
@@ -819 +819 @@
                         endTime: ChartTest.sampleCluster.startTime + diff / 2,
                         label: 'hello, world',
-                        fillStyle: 'rgb(0, 0, 255)',
                     }]);
 

trunk/Websites/perf.webkit.org/public/shared/statistics.js

@@ -38 +38 @@
             supportedProbabilities.push(oneSidedToTwoSidedProbability(probability).toFixed(2));
         return supportedProbabilities
+    }
+
+    this.supportedOneSideTTestProbabilities = function () {
+        return Object.keys(tDistributionByOneSidedProbability);
     }
 
@@ -106 +110 @@
         };
     }
+
+    this.findRangesForChangeDetectionsWithWelchsTTest = function (values, segmentations, oneSidedPossibility) {
+        if (!values.length)
+            return [];
+
+        const selectedRanges = [];
+        const twoSidedFromOneSidedPossibility = 2 * oneSidedPossibility - 1;
+
+        for (let i = 1; i + 2 < segmentations.length; i += 2) {
+            let found = false;
+            const previousMean = segmentations[i].value;
+            const currentMean = segmentations[i + 1].value;
+            console.assert(currentMean != previousMean);
+            const currentChangePoint = segmentations[i].seriesIndex;
+            const start = segmentations[i - 1].seriesIndex;
+            const end = segmentations[i + 2].seriesIndex;
+
+            for (let leftEdge = currentChangePoint - 2, rightEdge = currentChangePoint + 2; leftEdge >= start && rightEdge <= end; leftEdge--, rightEdge++) {
+                const result = this.computeWelchsT(values, leftEdge, currentChangePoint - leftEdge, values, currentChangePoint, rightEdge - currentChangePoint, twoSidedFromOneSidedPossibility);
+                if (result.significantlyDifferent) {
+                    selectedRanges.push({
+                        startIndex: leftEdge,
+                        endIndex: rightEdge - 1,
+                        segmentationStartValue: previousMean,
+                        segmentationEndValue: currentMean
+                    });
+                    found = true;
+                    break;
+                }
+            }
+            if (!found && Statistics.debuggingTestingRangeNomination)
+                console.log('Failed to find a testing range at', currentChangePoint, 'changing from', previousMean, 'to', currentMean);
+        }
+
+        return selectedRanges;
+    };
 
     function sampleMeanAndVarianceForValues(values, startIndex, length) {

trunk/Websites/perf.webkit.org/public/v3/components/chart-pane-base.js

@@ -20 +20 @@
         this._commitLogViewer = null;
         this._tasksForAnnotations = null;
-        this._renderedAnnotations = false;
+        this._detectedAnnotations = null;
+        this._renderAnnotationsLazily = new LazilyEvaluatedFunction(this._renderAnnotations.bind(this));
     }
 
@@ -52 +53 @@
         this._mainChart.listenToAction('selectionChange', this._mainSelectionDidChange.bind(this));
         this._mainChart.listenToAction('zoom', this._mainSelectionDidZoom.bind(this));
-        this._mainChart.listenToAction('annotationClick', this._openAnalysisTask.bind(this));
+        this._mainChart.listenToAction('annotationClick', this._didClickAnnotation.bind(this));
         this.renderReplace(this.content().querySelector('.chart-pane-main'), this._mainChart);
 
@@ -98 +99 @@
         AnalysisTask.fetchByPlatformAndMetric(this._platformId, this._metricId, noCache).then(function (tasks) {
             self._tasksForAnnotations = tasks;
-            self._renderedAnnotations = false;
             self.enqueueToRender();
         });
@@ -106 +106 @@
     didUpdateAnnotations()
     {
-        this._renderedAnnotations = false;
+        this._tasksForAnnotations = [...this._tasksForAnnotations];
         this.enqueueToRender();
     }
@@ -170 +170 @@
         this._commitLogViewer.view(this._openRepository, range.from, range.to);
         this.enqueueToRender();
+    }
+
+    _didClickAnnotation(annotation)
+    {
+        if (annotation.task)
+            this._openAnalysisTask(annotation);
+        else {
+            const newSelection = [annotation.startTime, annotation.endTime];
+            this._mainChart.setSelection(newSelection);
+            this._overviewChart.setSelection(newSelection, this);
+            this.enqueueToRender();
+        }
     }
 
@@ -247 +259 @@
             this._overviewChart.enqueueToRender();
 
-        if (this._mainChart)
+        if (this._mainChart) {
             this._mainChart.enqueueToRender();
+            this._renderAnnotationsLazily.evaluate(this._tasksForAnnotations, this._detectedAnnotations);
+        }
 
         if (this._errorMessage) {
@@ -255 +269 @@
         }
 
-        this._renderAnnotations();
 
         if (this._mainChartStatus)
@@ -269 +282 @@
     }
 
-    _renderAnnotations()
-    {
-        if (!this._tasksForAnnotations || this._renderedAnnotations)
-            return;
-        this._renderedAnnotations = true;
-
-        var annotations = this._tasksForAnnotations.map(function (task) {
-            var fillStyle = '#fc6';
-            switch (task.changeType()) {
-            case 'inconclusive':
-                fillStyle = '#fcc';
-                break;
-            case 'progression':
-                fillStyle = '#39f';
-                break;
-            case 'regression':
-                fillStyle = '#c60';
-                break;
-            case 'unchanged':
-                fillStyle = '#ccc';
-                break;
-            }
-
+    _renderAnnotations(taskForAnnotations, detectedAnnotations)
+    {
+        let annotations = (taskForAnnotations || []).map((task) => {
             return {
-                task: task,
+                task,
+                fillStyle: ChartStyles.annotationFillStyleForTask(task),
                 startTime: task.startTime(),
                 endTime: task.endTime(),
-                label: task.label(),
-                fillStyle: fillStyle,
+                label: task.label()
             };
         });
+
+        annotations = annotations.concat(detectedAnnotations || []);
+
         this._mainChart.setAnnotations(annotations);
     }

trunk/Websites/perf.webkit.org/public/v3/components/chart-styles.js

@@ -153 +153 @@
             minWidth: 3,
             barHeight: 7,
-            barSpacing: 2,
+            barSpacing: 2
         };
         return options;
     }
+
+    static annotationFillStyleForTask(task) {
+        if (!task)
+            return '#888';
+
+        switch (task.changeType()) {
+            case 'inconclusive':
+                return '#fcc';
+            case 'progression':
+                return '#39f';
+            case 'regression':
+                return '#c60';
+            case 'unchanged':
+                return '#ccc';
+        }
+        return '#fc6';
+
+    }
 }

trunk/Websites/perf.webkit.org/public/v3/components/interactive-time-series-chart.js

@@ -342 +342 @@
             return null;
 
-        for (var item of this._annotations) {
+        for (const item of this._annotations) {
             if (item.x <= cursorLocation.x && cursorLocation.x <= item.x + item.width
                 && item.y <= cursorLocation.y && cursorLocation.y <= item.y + item.height)

trunk/Websites/perf.webkit.org/public/v3/models/measurement-set.js

@@ -23 +23 @@
 
     platformId() { return this._platformId; }
+    metricId() { return this._metricId; }
 
     static findSet(platformId, metricId, lastModified)
@@ -226 +227 @@
             var addSegment = function (startingPoint, endingPoint) {
                 var value = Statistics.mean(timeSeries.valuesBetweenRange(startingPoint.seriesIndex, endingPoint.seriesIndex));
-                segmentationSeries.push({value: value, time: startingPoint.time, interval: function () { return null; }});
-                segmentationSeries.push({value: value, time: endingPoint.time, interval: function () { return null; }});
+                segmentationSeries.push({value: value, time: startingPoint.time, seriesIndex: startingPoint.seriesIndex, interval: function () { return null; }});
+                segmentationSeries.push({value: value, time: endingPoint.time, seriesIndex: endingPoint.seriesIndex, interval: function () { return null; }});
             };
 

trunk/Websites/perf.webkit.org/public/v3/models/metric.js

@@ -86 +86 @@
     }
 
+    labelForDifference(beforeValue, afterValue, progressionTypeName, regressionTypeName)
+    {
+        const diff = afterValue - beforeValue;
+        const changeType = diff < 0 == this.isSmallerBetter() ? progressionTypeName : regressionTypeName;
+        const relativeChange = diff / beforeValue * 100;
+        const changeLabel = Math.abs(relativeChange).toFixed(2) + '% ' + changeType;
+        return {changeType, relativeChange, changeLabel};
+    }
+
     makeFormatter(sigFig, alwaysShowSign) { return Metric.makeFormatter(this.unit(), sigFig, alwaysShowSign); }
 

trunk/Websites/perf.webkit.org/public/v3/models/test-group.js

@@ -145 +145 @@
 
         if (beforeValues.length && afterValues.length) {
-            var diff = afterMean - beforeMean;
-            var smallerIsBetter = metric.isSmallerBetter();
-            var changeType = diff < 0 == smallerIsBetter ? 'better' : 'worse';
-            var changeLabel = Math.abs(diff / beforeMean * 100).toFixed(2) + '% ' + changeType;
+            const summary = metric.labelForDifference(beforeMean, afterMean, 'better', 'worse');
+            result.changeType = summary.changeType;
+            result.changeLabel = summary.changeLabel;
             var isSignificant = Statistics.testWelchsT(beforeValues, afterValues);
             var significanceLabel = isSignificant ? 'significant' : 'insignificant';
 
-            result.changeType = changeType;
-            result.label = changeLabel;
             if (hasCompleted)
                 result.status = isSignificant ? result.changeType : 'unchanged';

trunk/Websites/perf.webkit.org/public/v3/pages/analysis-task-page.js

@@ -752 +752 @@
 
         const updateRendering = () => {
-            this._chartPane.didUpdateAnnotations();
+            this.part('chart-pane').didUpdateAnnotations();
             this.enqueueToRender();
         };

trunk/Websites/perf.webkit.org/public/v3/pages/chart-pane.js

@@ -40 +40 @@
     },
     {
+        id: 6,
+        label: 'Segmentation with Welch\'s t-test change detection',
+        execute: async function (source, parameters) {
+            const segmentation =  await source.measurementSet.fetchSegmentation('segmentTimeSeriesByMaximizingSchwarzCriterion', parameters,
+                source.type, source.includeOutliers, source.extendToFuture);
+            if (!segmentation)
+                return segmentation;
+
+            const metric = Metric.findById(source.measurementSet.metricId());
+            const timeSeries = source.measurementSet.fetchedTimeSeries(source.type, source.includeOutliers, source.extendToFuture);
+            segmentation.analysisAnnotations = Statistics.findRangesForChangeDetectionsWithWelchsTTest(timeSeries.values(),
+                segmentation, parameters[parameters.length - 1]).map((range) => {
+                const startPoint = timeSeries.findPointByIndex(range.startIndex);
+                const endPoint = timeSeries.findPointByIndex(range.endIndex);
+                const summary = metric.labelForDifference(range.segmentationStartValue, range.segmentationEndValue, 'progression', 'regression');
+                return {
+                    task: null,
+                    fillStyle: ChartStyles.annotationFillStyleForTask(null),
+                    startTime: startPoint.time,
+                    endTime: endPoint.time,
+                    label: `Potential ${summary.changeLabel}`,
+                };
+            });
+            return segmentation;
+        },
+        parameterList: [
+            {label: "Segment count weight", value: 2.5, min: 0.01, max: 10, step: 0.01},
+            {label: "Grid size", value: 500, min: 100, max: 10000, step: 10},
+            {label: "t-test significance", value: 0.99, options: Statistics.supportedOneSideTTestProbabilities()},
+        ]
+    },
+    {
         id: 1,
         label: 'Simple Moving Average',
@@ -418 +450 @@
                 element('h3', 'Parameters'),
                 element('ul', this._trendLineType.parameterList.map(function (parameter, index) {
+                    if (parameter.options) {
+                        const select = element('select', parameter.options.map((option) =>
+                            element('option', {value: option, selected: option == parameter.value}, option)));
+                        select.onchange = self._trendLineParameterDidChange.bind(self);
+                        select.parameterIndex = index;
+                        return element('li', element('label', [parameter.label + ': ', select]));
+                    }
+
                     var attributes = {type: 'number'};
                     for (var name in parameter)
                         attributes[name] = parameter[name];
+
                     attributes.value = configuredParameters[index];
-                    var input = element('input', attributes);
+                    const input = element('input', attributes);
                     input.parameterIndex = index;
                     input.oninput = self._trendLineParameterDidChange.bind(self);
@@ -476 +517 @@
     }
 
-    _updateTrendLine()
+    async _updateTrendLine()
     {
         if (!this._mainChart.sourceList())
@@ -485 +526 @@
         var currentTrendLineParameters = this._trendLineParameters || this._defaultParametersForTrendLine(currentTrendLineType);
         var currentTrendLineVersion = this._trendLineVersion;
-        var self = this;
         var sourceList = this._mainChart.sourceList();
 
@@ -493 +533 @@
         } else {
             // Wait for all trendlines to be ready. Otherwise we might see FOC when the domain is expanded.
-            Promise.all(sourceList.map(function (source, sourceIndex) {
-                return currentTrendLineType.execute.call(null, source, currentTrendLineParameters).then(function (trendlineSeries) {
-                    if (self._trendLineVersion == currentTrendLineVersion)
-                        self._mainChart.setTrendLine(sourceIndex, trendlineSeries);
-                });
-            })).then(function () {
-                self.enqueueToRender();
-            });
+            await Promise.all(sourceList.map(async (source, sourceIndex) => {
+                const trendlineSeries = await currentTrendLineType.execute.call(null, source, currentTrendLineParameters);
+                if (this._trendLineVersion == currentTrendLineVersion)
+                    this._mainChart.setTrendLine(sourceIndex, trendlineSeries);
+
+                if (trendlineSeries && trendlineSeries.analysisAnnotations)
+                    this._detectedAnnotations = trendlineSeries.analysisAnnotations;
+                else
+                    this._detectedAnnotations = null;
+            }));
+            this.enqueueToRender();
         }
     }

trunk/Websites/perf.webkit.org/unit-tests/statistics-tests.js

@@ -389 +389 @@
         });
     });
+
+    describe('findRangesForChangeDetectionsWithWelchsTTest', () => {
+        it('should return an empty array if the value is empty list', () => {
+            assert.deepEqual(Statistics.findRangesForChangeDetectionsWithWelchsTTest([], [], 0.975), []);
+        });
+
+        it('should return an empty array if segmentation is empty list', () => {
+            assert.deepEqual(Statistics.findRangesForChangeDetectionsWithWelchsTTest([1,2,3], [], 0.975), []);
+        });
+
+        it('should return the range if computeWelchsT shows a significant change', () => {
+            const values = [
+                747.30337423744,
+                731.47392585276,
+                743.66763513161,
+                738.02055323487,
+                738.25426340842,
+                742.38680046471,
+                733.13921703284,
+                739.22069966147,
+                735.69295749633,
+                743.01705472504,
+                745.45778145306,
+                731.04841157169,
+                729.4372674973,
+                735.4497416527,
+                739.0230668644,
+                730.91782989909,
+                722.18725411279,
+                731.96223451728,
+                730.04119216192,
+                730.78087646284,
+                729.63155210365,
+                730.17585200878,
+                733.93766054706,
+                740.74920717197,
+                752.14718023647,
+                764.49990164847,
+                766.36100828473,
+                756.2291883252,
+                750.14522451097,
+                749.57595092266,
+                748.03624881866,
+                769.41522176386,
+                744.04660430456,
+                751.17927808265,
+                753.29996854062,
+                757.01813756936,
+                746.62413820741,
+                742.64420062736,
+                758.12726352772,
+                778.2278439089,
+                775.11818554541,
+                775.11818554541];
+            const segmentation = [{
+                    seriesIndex: 0,
+                    time: 1505176030671,
+                    value: 736.5366704896555,
+                    x: 370.4571789404566,
+                    y: 185.52613334520248,
+                },
+                {
+                    seriesIndex: 18,
+                    time: 1515074391534,
+                    value: 736.5366704896555,
+                    x: 919.4183852714947,
+                    y: 185.52613334520248
+                },
+                {
+                    seriesIndex: 18,
+                    time: 1515074391534,
+                    value: 750.3483428383142,
+                    x: 919.4183852714947,
+                    y: 177.9710953409673,
+                },
+                {
+                    seriesIndex: 41,
+                    time: 1553851695869,
+                    value: 750.3483428383142,
+                    x: 3070.000290764446,
+                    y: 177.9710953409673,
+                }];
+            assert.deepEqual(Statistics.findRangesForChangeDetectionsWithWelchsTTest(values, segmentation, 0.975), [
+                {
+                  "endIndex": 29,
+                  "segmentationEndValue": 750.3483428383142,
+                  "segmentationStartValue": 736.5366704896555,
+                  "startIndex": 6
+                }
+            ]);
+        })
+    });
 });