Changeset 196440 in webkit


Ignore:
Timestamp:
Feb 11, 2016 2:17:55 PM (8 years ago)
Author:
rniwa@webkit.org
Message:

Perf dashboard should have UI to retry A/B testing
https://bugs.webkit.org/show_bug.cgi?id=154090

Reviewed by Chris Dumez.

Added a button to re-try an existing A/B testing group with a custom repetition count. The same button functions
as a way of confirming the progression/regression when there have been no A/B testing scheduled in the task.

Also fixed the bug that A/B testing groups that have been waiting for other test groups will be shown as "running".

  • public/v3/components/results-table.js:

(ResultsTable.cssTemplate): Don't pad the list of extra repositories when it's empty.

  • public/v3/components/test-group-results-table.js:

(TestGroupResultsTable.prototype.buildRowGroups): Use TestGroup.labelForRootSet instead of manually
computing the letter for each configuration set.

  • public/v3/models/build-request.js:

(BuildRequest.prototype.hasStarted): Added.

  • public/v3/models/data-model.js:

(DataModelObject.ensureSingleton): Added.
(DataModelObject.cachedFetch): Added noCache option. This is used when re-fetching the test groups after
creating one.

  • public/v3/models/measurement-cluster.js:

(MeasurementCluster.prototype.startTime): Added.

  • public/v3/models/measurement-set.js:

(MeasurementSet.prototype.hasFetchedRange): Added. Returns true only if there are no "holes" (cluster
yet to be fetched) between the specified time range. This was added to fix a bug in AnalysisTaskPage's
_didFetchMeasurement.

  • public/v3/models/test-group.js:

(TestGroup): Added this._rootSetToLabel.
(TestGroup.prototype.addBuildRequest): Reset this._rootSetToLabel along with this._requestedRootSets.
(TestGroup.prototype.repetitionCount): Added. Returns the number of iterations executed per set. We assume that
every root set in the test group shares a single repetition count.
(TestGroup.prototype.requestedRootSets): Now populates this._rootSetToLabel for labelForRootSet.
(TestGroup.prototype.labelForRootSet): Added.
(TestGroup.prototype.hasStarted): Added.
(TestGroup.prototype.compareTestResults): Use 'running' and 'pending' to differentiate test groups that are waiting
for other groups to finish running from the ones that are actually running ('incomplete' before this patch).
(TestGroup.fetchByTask):
(TestGroup.createAndRefetchTestGroups): Added. Creates a new test group using the privileged-api/create-test-group
and fetches the list of test groups for the specified analysis task.
(TestGroup._createModelsFromFetchedTestGroups): Extracted from TestGroup.fetchByTask.

  • public/v3/pages/analysis-task-page.js:

(AnalysisTaskPage): Initialize _renderedCurrentTestGroup to undefined so that we'd always can differentiate
the initial call to AnalysisTaskPage.render and subsequent calls in which it's identical to _currentTestGroup.
(AnalysisTaskPage.prototype._didFetchMeasurement): Fixed a bug that we don't exit early even when some
clusters in between startPoint and endPoint are still being fetched via newly added hasFetchedRange.
(AnalysisTaskPage.prototype.render): Update the default repetition count based on the current test group.
Also update the label of the button to "Confirm the change" if there is no A/B testing in this task.
(AnalysisTaskPage.prototype._retryCurrentTestGroup): Added. Re-triggers an existing A/B testing group or creates
the A/B testing for the entire range of the analysis task.
(AnalysisTaskPage.prototype._hasDuplicateTestGroupName): Added.
(AnalysisTaskPage.prototype._createRetryNameForTestGroup): Added.
(AnalysisTaskPage.htmlTemplate): Added form controls to re-trigger A/B testing.
(AnalysisTaskPage.cssTemplate): Updated the style.

Location:
trunk/Websites/perf.webkit.org
Files:
9 edited

Legend:

Unmodified
Added
Removed
  • trunk/Websites/perf.webkit.org/ChangeLog

    r196390 r196440  
     12016-02-10  Ryosuke Niwa  <rniwa@webkit.org>
     2
     3        Perf dashboard should have UI to retry A/B testing
     4        https://bugs.webkit.org/show_bug.cgi?id=154090
     5
     6        Reviewed by Chris Dumez.
     7
     8        Added a button to re-try an existing A/B testing group with a custom repetition count. The same button functions
     9        as a way of confirming the progression/regression when there have been no A/B testing scheduled in the task.
     10
     11        Also fixed the bug that A/B testing groups that have been waiting for other test groups will be shown as "running".
     12
     13        * public/v3/components/results-table.js:
     14        (ResultsTable.cssTemplate): Don't pad the list of extra repositories when it's empty.
     15
     16        * public/v3/components/test-group-results-table.js:
     17        (TestGroupResultsTable.prototype.buildRowGroups): Use TestGroup.labelForRootSet instead of manually
     18        computing the letter for each configuration set.
     19
     20        * public/v3/models/build-request.js:
     21        (BuildRequest.prototype.hasStarted): Added.
     22
     23        * public/v3/models/data-model.js:
     24        (DataModelObject.ensureSingleton): Added.
     25        (DataModelObject.cachedFetch): Added noCache option. This is used when re-fetching the test groups after
     26        creating one.
     27
     28        * public/v3/models/measurement-cluster.js:
     29        (MeasurementCluster.prototype.startTime): Added.
     30
     31        * public/v3/models/measurement-set.js:
     32        (MeasurementSet.prototype.hasFetchedRange): Added. Returns true only if there are no "holes" (cluster
     33        yet to be fetched) between the specified time range. This was added to fix a bug in AnalysisTaskPage's
     34        _didFetchMeasurement.
     35
     36        * public/v3/models/test-group.js:
     37        (TestGroup): Added this._rootSetToLabel.
     38        (TestGroup.prototype.addBuildRequest): Reset this._rootSetToLabel along with this._requestedRootSets.
     39        (TestGroup.prototype.repetitionCount): Added. Returns the number of iterations executed per set. We assume that
     40        every root set in the test group shares a single repetition count.
     41        (TestGroup.prototype.requestedRootSets): Now populates this._rootSetToLabel for labelForRootSet.
     42        (TestGroup.prototype.labelForRootSet): Added.
     43        (TestGroup.prototype.hasStarted): Added.
     44        (TestGroup.prototype.compareTestResults): Use 'running' and 'pending' to differentiate test groups that are waiting
     45        for other groups to finish running from the ones that are actually running ('incomplete' before this patch).
     46        (TestGroup.fetchByTask):
     47        (TestGroup.createAndRefetchTestGroups): Added. Creates a new test group using the privileged-api/create-test-group
     48        and fetches the list of test groups for the specified analysis task.
     49        (TestGroup._createModelsFromFetchedTestGroups): Extracted from TestGroup.fetchByTask.
     50
     51        * public/v3/pages/analysis-task-page.js:
     52        (AnalysisTaskPage): Initialize _renderedCurrentTestGroup to undefined so that we'd always can differentiate
     53        the initial call to AnalysisTaskPage.render and subsequent calls in which it's identical to _currentTestGroup.
     54        (AnalysisTaskPage.prototype._didFetchMeasurement): Fixed a bug that we don't exit early even when some
     55        clusters in between startPoint and endPoint are still being fetched via newly added hasFetchedRange.
     56        (AnalysisTaskPage.prototype.render): Update the default repetition count based on the current test group.
     57        Also update the label of the button to "Confirm the change" if there is no A/B testing in this task.
     58        (AnalysisTaskPage.prototype._retryCurrentTestGroup): Added. Re-triggers an existing A/B testing group or creates
     59        the A/B testing for the entire range of the analysis task.
     60        (AnalysisTaskPage.prototype._hasDuplicateTestGroupName): Added.
     61        (AnalysisTaskPage.prototype._createRetryNameForTestGroup): Added.
     62        (AnalysisTaskPage.htmlTemplate): Added form controls to re-trigger A/B testing.
     63        (AnalysisTaskPage.cssTemplate): Updated the style.
     64
    1652016-02-10  Ryosuke Niwa  <rniwa@webkit.org>
    266
  • trunk/Websites/perf.webkit.org/public/v3/components/results-table.js

    r194788 r196440  
    227227            }
    228228
     229            .results-table-extra-repositories:empty {
     230                padding: 0;
     231            }
     232
    229233            .results-table-extra-repositories li {
    230234                display: inline;
  • trunk/Websites/perf.webkit.org/public/v3/components/test-group-results-table.js

    r194788 r196440  
    3131
    3232        var rootSets = this._testGroup.requestedRootSets();
    33         var groups = rootSets.map(function (rootSet, setIndex) {
     33        var groups = rootSets.map(function (rootSet) {
    3434            var rows = [new ResultsTableRow('Mean', rootSet)];
    3535            var results = [];
     
    5252                rows[0].setResult(aggregatedResult);
    5353
    54             return {heading: String.fromCharCode('A'.charCodeAt(0) + setIndex), rows:rows};
     54            return {heading: testGroup.labelForRootSet(rootSet), rows:rows};
    5555        });
    5656
     
    5858        for (var i = 0; i < rootSets.length; i++) {
    5959            for (var j = i + 1; j < rootSets.length; j++) {
    60                 var startConfig = String.fromCharCode('A'.charCodeAt(0) + i);
    61                 var endConfig = String.fromCharCode('A'.charCodeAt(0) + j);
     60                var startConfig = testGroup.labelForRootSet(rootSets[i]);
     61                var endConfig = testGroup.labelForRootSet(rootSets[j]);
    6262
    6363                var result = this._testGroup.compareTestResults(rootSets[i], rootSets[j]);
    64                 if (result.status == 'incomplete' || result.status == 'failed')
     64                if (result.status == 'pending' || result.status == 'running' || result.status == 'failed')
    6565                    continue;
    6666
     
    7171        }
    7272
    73         groups.push({heading: '', rows: comparisonRows});
     73        groups.unshift({heading: '', rows: comparisonRows});
    7474
    7575        return groups;
  • trunk/Websites/perf.webkit.org/public/v3/models/build-request.js

    r194788 r196440  
    2222
    2323    hasCompleted() { return this._status == 'failed' || this._status == 'completed'; }
     24    hasStarted() { return this._status != 'pending'; }
    2425    statusLabel()
    2526    {
  • trunk/Websites/perf.webkit.org/public/v3/models/data-model.js

    r194618 r196440  
    77    }
    88    id() { return this._id; }
     9
     10    static ensureSingleton(id, object)
     11    {
     12        var singleton = this.findById(id);
     13        if (singleton)
     14            return singleton;
     15        return new (this)(id, object);
     16    }
    917
    1018    static namedStaticMap(name)
     
    4452    }
    4553
    46     static cachedFetch(path, params)
     54    static cachedFetch(path, params, noCache)
    4755    {
    4856        var query = [];
     
    5361        if (query.length)
    5462            path += '?' + query.join('&');
     63
     64        if (noCache)
     65            return getJSONWithStatus(path);
    5566
    5667        var cacheMap = this.ensureNamedStaticMap(DataModelObject.CacheMapSymbol);
  • trunk/Websites/perf.webkit.org/public/v3/models/measurement-cluster.js

    r194618 r196440  
    88
    99    startTime() { return this._response['startTime']; }
     10    endTime() { return this._response['endTime']; }
    1011
    1112    addToSeries(series, configType, includeOutliers, idMap)
  • trunk/Websites/perf.webkit.org/public/v3/models/measurement-set.js

    r194664 r196440  
    192192    }
    193193
     194    hasFetchedRange(startTime, endTime)
     195    {
     196        console.assert(startTime < endTime);
     197        var hasHole = false;
     198        var previousEndTime = null;
     199        for (var cluster of this._sortedClusters) {
     200            if (cluster.startTime() < startTime && startTime < cluster.endTime())
     201                hasHole = false;
     202            if (previousEndTime !== null && previousEndTime != cluster.startTime())
     203                hasHole = true;
     204            if (cluster.startTime() < endTime && endTime < cluster.endTime())
     205                break;
     206            previousEndTime = cluster.endTime();
     207        }
     208        return !hasHole;
     209    }
     210
    194211    fetchedTimeSeries(configType, includeOutliers, extendToFuture)
    195212    {
  • trunk/Websites/perf.webkit.org/public/v3/models/test-group.js

    r196333 r196440  
    1212        this._repositories = null;
    1313        this._requestedRootSets = null;
     14        this._rootSetToLabel = new Map;
    1415        this._allRootSets = null;
    1516        console.assert(!object.platform || object.platform instanceof Platform);
     
    2425        this._requestsAreInOrder = false;
    2526        this._requestedRootSets = null;
     27        this._rootSetToLabel = null;
     28    }
     29
     30    repetitionCount()
     31    {
     32        if (!this._buildRequests.length)
     33            return 0;
     34        var rootSet = this._buildRequests[0].rootSet();
     35        var count = 0;
     36        for (var request of this._buildRequests) {
     37            if (request.rootSet() == rootSet)
     38                count++;
     39        }
     40        return count;
    2641    }
    2742
     
    3752            }
    3853            this._requestedRootSets.sort(function (a, b) { return a.latestCommitTime() - b.latestCommitTime(); });
     54            var setIndex = 0;
     55            for (var set of this._requestedRootSets) {
     56                this._rootSetToLabel.set(set, String.fromCharCode('A'.charCodeAt(0) + setIndex));
     57                setIndex++;
     58            }
     59
    3960        }
    4061        return this._requestedRootSets;
     
    4566        this._orderBuildRequests();
    4667        return this._buildRequests.filter(function (request) { return request.rootSet() == rootSet; });
     68    }
     69
     70    labelForRootSet(rootSet)
     71    {
     72        console.assert(this._requestedRootSets);
     73        return this._rootSetToLabel.get(rootSet);
    4774    }
    4875
     
    6390    {
    6491        return this._buildRequests.every(function (request) { return request.hasCompleted(); });
     92    }
     93
     94    hasStarted()
     95    {
     96        return this._buildRequests.some(function (request) { return request.hasStarted(); });
    6597    }
    6698
     
    86118
    87119        if (!this.hasCompleted()) {
    88             result.status = 'incomplete';
    89             result.label = 'Running';
    90             result.fullLabel = 'Running';
     120            if (this.hasStarted()) {
     121                result.status = 'running';
     122                result.label = 'Running';
     123                result.fullLabel = 'Running';
     124            } else {
     125                result.status = 'pending';
     126                result.label = 'Pending';
     127                result.fullLabel = 'Pending';
     128            }
    91129        } else if (result.changeType) {
    92130            var significance = result.isStatisticallySignificant ? 'significant' : 'insignificant';
     
    108146    }
    109147
    110     static fetchByTask(taskId)
     148    static createAndRefetchTestGroups(task, name, repetitionCount, rootSets)
    111149    {
    112         return this.cachedFetch('../api/test-groups', {task: taskId}).then(function (data) {
    113             var testGroups = data['testGroups'].map(function (row) {
    114                 row.platform = Platform.findById(row.platform);
    115                 return new TestGroup(row.id, row);
    116             });
    117 
    118             var rootIdMap = {};
    119             for (var root of data['roots'])
    120                 rootIdMap[root.id] = root;
    121 
    122             var rootSets = data['rootSets'].map(function (row) {
    123                 row.roots = row.roots.map(function (rootId) { return rootIdMap[rootId]; });
    124                 row.testGroup = RootSet.findById(row.testGroup);
    125                 return new RootSet(row.id, row);
    126             });
    127 
    128             var buildRequests = data['buildRequests'].map(function (rawData) {
    129                 rawData.testGroup = TestGroup.findById(rawData.testGroup);
    130                 rawData.rootSet = RootSet.findById(rawData.rootSet);
    131                 return new BuildRequest(rawData.id, rawData);
    132             });
    133 
    134             return testGroups;
     150        var self = this;
     151        return PrivilegedAPI.sendRequest('create-test-group', {
     152            task: task.id(),
     153            name: name,
     154            repetitionCount: repetitionCount,
     155            rootSets: rootSets,
     156        }).then(function (data) {
     157            return self.cachedFetch('../api/test-groups', {task: task.id()}, true).then(self._createModelsFromFetchedTestGroups.bind(self));
    135158        });
    136159    }
    137160
     161    static fetchByTask(taskId)
     162    {
     163        return this.cachedFetch('../api/test-groups', {task: taskId}).then(this._createModelsFromFetchedTestGroups.bind(this));
     164    }
     165
     166    static _createModelsFromFetchedTestGroups(data)
     167    {
     168        var testGroups = data['testGroups'].map(function (row) {
     169            row.platform = Platform.findById(row.platform);
     170            return TestGroup.ensureSingleton(row.id, row);
     171        });
     172
     173        var rootIdMap = {};
     174        for (var root of data['roots'])
     175            rootIdMap[root.id] = root;
     176
     177        var rootSets = data['rootSets'].map(function (row) {
     178            row.roots = row.roots.map(function (rootId) { return rootIdMap[rootId]; });
     179            row.testGroup = RootSet.findById(row.testGroup);
     180            return RootSet.ensureSingleton(row.id, row);
     181        });
     182
     183        var buildRequests = data['buildRequests'].map(function (rawData) {
     184            rawData.testGroup = TestGroup.findById(rawData.testGroup);
     185            rawData.rootSet = RootSet.findById(rawData.rootSet);
     186            return BuildRequest.ensureSingleton(rawData.id, rawData);
     187        });
     188
     189        return testGroups;
     190    }
    138191}
  • trunk/Websites/perf.webkit.org/public/v3/pages/analysis-task-page.js

    r196387 r196440  
    1414        this._testGroups = null;
    1515        this._renderedTestGroups = null;
    16         this._renderedCurrentTestGroup = null;
     16        this._renderedCurrentTestGroup = undefined;
    1717        this._analysisResults = null;
    1818        this._measurementSet = null;
     
    2525        this._analysisResultsViewer.setTestGroupCallback(this._showTestGroup.bind(this));
    2626        this._testGroupResultsTable = this.content().querySelector('test-group-results-table').component();
     27
     28        this.content().querySelector('.test-group-retry-form').onsubmit = this._retryCurrentTestGroup.bind(this);
    2729    }
    2830
     
    8688        var startPoint = series.findById(this._task.startMeasurementId());
    8789        var endPoint = series.findById(this._task.endMeasurementId());
    88         if (!startPoint || !endPoint)
     90        if (!startPoint || !endPoint || !this._measurementSet.hasFetchedRange(startPoint.time, endPoint.time))
    8991            return;
    9092
     
    140142        var v2URL = `/v2/#/analysis/task/${this._taskId}`;
    141143        this.content().querySelector('.error-message').innerHTML +=
    142             `<p>This page is read only for now. To schedule a new A/B testing job, use <a href="${v2URL}">v2 page</a>.</p>`;
     144            `<p>To schedule a custom A/B testing, use <a href="${v2URL}">v2 UI</a>.</p>`;
    143145
    144146         this._chartPane.render();
     
    169171            this._renderedCurrentTestGroup = null;
    170172        }
    171         if (this._renderedCurrentTestGroup != this._currentTestGroup) {
     173
     174        if (this._renderedCurrentTestGroup !== this._currentTestGroup) {
    172175            if (this._renderedCurrentTestGroup) {
    173176                var element = this.content().querySelector('.test-group-list-' + this._renderedCurrentTestGroup.id());
     
    180183                    element.classList.add('selected');
    181184            }
     185
     186            this.content().querySelector('.test-group-retry-button').textContent = this._currentTestGroup ? 'Retry' : 'Confirm the change';
     187
     188            var repetitionCount = this._currentTestGroup ? this._currentTestGroup.repetitionCount() : 4;
     189            var repetitionCountController = this.content().querySelector('.test-group-retry-repetition-count');
     190            repetitionCountController.value = repetitionCount;
     191
    182192            this._renderedCurrentTestGroup = this._currentTestGroup;
    183193        }
     194
     195        this.content().querySelector('.test-group-retry-button').disabled = !(this._currentTestGroup || this._startPoint);
    184196
    185197        this._testGroupResultsTable.render();
     
    193205        this._testGroupResultsTable.setTestGroup(this._currentTestGroup);
    194206        this.render();
     207    }
     208
     209    _retryCurrentTestGroup(event)
     210    {
     211        event.preventDefault();
     212        console.assert(this._currentTestGroup || this._startPoint);
     213
     214        var testGroupName;
     215        var rootSetList;
     216        var rootSetLabels;
     217
     218        if (this._currentTestGroup) {
     219            var testGroup = this._currentTestGroup;
     220            testGroupName = this._createRetryNameForTestGroup(testGroup.name());
     221            rootSetList = testGroup.requestedRootSets();
     222            rootSetLabels = rootSetList.map(function (rootSet) { return testGroup.labelForRootSet(rootSet); });
     223        } else {
     224            testGroupName = 'Confirming the change';
     225            rootSetList = [this._startPoint.rootSet(), this._endPoint.rootSet()];
     226            rootSetLabels = ['Point 0', `Point ${this._endPoint.seriesIndex - this._startPoint.seriesIndex}`];
     227        }
     228
     229        var rootSetsByName = {};
     230        for (var repository of rootSetList[0].repositories())
     231            rootSetsByName[repository.name()] = [];
     232
     233        var setIndex = 0;
     234        for (var rootSet of rootSetList) {
     235            for (var repository of rootSet.repositories()) {
     236                var list = rootSetsByName[repository.name()];
     237                if (!list) {
     238                    alert(`Set ${rootSetLabels[setIndex]} specifies ${repository.label()} but set ${rootSetLabels[0]} does not.`);
     239                    return null;
     240                }
     241                list.push(rootSet.commitForRepository(repository).revision());
     242            }
     243            setIndex++;
     244            for (var name in rootSetsByName) {
     245                var list = rootSetsByName[name];
     246                if (list.length < setIndex) {
     247                    alert(`Set ${rootSetLabels[0]} specifies ${repository.label()} but set ${rootSetLabels[setIndex]} does not.`);
     248                    return null;
     249                }
     250            }
     251        }
     252
     253        var repetitionCount = this.content().querySelector('.test-group-retry-repetition-count').value;
     254
     255        TestGroup.createAndRefetchTestGroups(this._task, testGroupName, repetitionCount, rootSetsByName)
     256            .then(this._didFetchTestGroups.bind(this), function (error) {
     257            alert('Failed to create a new test group: ' + error);
     258        });
     259    }
     260
     261    _createRetryNameForTestGroup(name)
     262    {
     263        var nameWithNumberMatch = name.match(/(.+?)\s*\(\s*(\d+)\s*\)\s*$/);
     264        var number = 1;
     265        if (nameWithNumberMatch) {
     266            name = nameWithNumberMatch[1];
     267            number = parseInt(nameWithNumberMatch[2]);
     268        }
     269
     270        var newName;
     271        do {
     272            number++;
     273            newName = `${name} (${number})`;
     274        } while (this._hasDuplicateTestGroupName(newName));
     275
     276        return newName;
     277    }
     278
     279    _hasDuplicateTestGroupName(name)
     280    {
     281        console.assert(this._testGroups);
     282        for (var group of this._testGroups) {
     283            if (group.name() == name)
     284                return true;
     285        }
     286        return false;
    195287    }
    196288
     
    209301                <section class="test-group-view">
    210302                    <ul class="test-group-list"></ul>
    211                     <div class="test-group-details"><test-group-results-table></test-group-results-table></div>
     303                    <div class="test-group-details">
     304                        <test-group-results-table></test-group-results-table>
     305                        <form class="test-group-retry-form">
     306                            <button class="test-group-retry-button" type="submit">Retry</button>
     307                            with
     308                            <select class="test-group-retry-repetition-count">
     309                                <option>1</option>
     310                                <option>2</option>
     311                                <option>3</option>
     312                                <option>4</option>
     313                                <option>5</option>
     314                                <option>6</option>
     315                                <option>7</option>
     316                                <option>8</option>
     317                                <option>9</option>
     318                                <option>10</option>
     319                            </select>
     320                            iterations per set
     321                        </form>
     322                    </div>
    212323                </section>
    213324            </div>
     
    269380                display: table;
    270381                margin: 0 1rem;
     382                margin-bottom: 2rem;
    271383            }
    272384
     
    274386                display: table-cell;
    275387                margin-bottom: 1rem;
     388                padding: 0;
     389                margin: 0;
     390            }
     391
     392            .test-group-retry-form {
     393                padding: 0;
     394                margin: 0.5rem;
    276395            }
    277396
    278397            .test-group-list {
    279398                display: table-cell;
    280             }
    281 
    282             .test-group-list:not(:empty) {
    283399                margin: 0;
    284400                padding: 0.2rem 0;
     
    286402                border-right: solid 1px #ccc;
    287403                white-space: nowrap;
     404            }
     405
     406            .test-group-list:empty {
     407                margin: 0;
     408                padding: 0;
     409                border-right: none;
    288410            }
    289411
Note: See TracChangeset for help on using the changeset viewer.