Changeset 180000 in webkit


Ignore:
Timestamp:
Feb 12, 2015 10:50:53 AM (9 years ago)
Author:
rniwa@webkit.org
Message:

Perf dashboard should show the results of A/B testing
https://bugs.webkit.org/show_bug.cgi?id=141500

Reviewed by Chris Dumez.

Added the support for fetching test_runs for a specific test group in /api/runs/, and used it in the
analysis task page to fetch results for each test group.

Merged App.createChartData into App.Manifest.fetchRunsWithPlatformAndMetric so that App.BuildRequest
can use the formatter.

  • public/api/runs.php:

(fetch_runs_for_config_and_test_group): Added.
(fetch_runs_for_config): Just return the fetched rows since main will format them with RunsGenerator.
(main): Use fetch_runs_for_config_and_test_group to fetch rows when a test group id is specified. Also
use RunsGenerator to format results.
(RunsGenerator): Added.
(RunsGenerator::construct): Added.
(RunsGenerator::add_runs): Added.
(RunsGenerator::format_run): Moved.
(RunsGenerator::parse_revisions_array): Moved.

  • public/v2/analysis.js:

(App.TestGroup): Fixed a typo. The property on a test group that refers to an analysis task is "task".
(App.TestGroup._fetchChartData): Added. Fetches all A/B testing results for this group.
(App.BuildRequest.configLetter): Renamed from config since this returns a letter that identifies the
configuration associated with this build request such as "A" and "B".
(App.BuildRequest.statusLabel): Added the missing label for failed build requests.
(App.BuildRequest.url): Added. Returns the URL associated with this build request.
(App.BuildRequest._meanFetched): Added. Retrieve the mean and the build number for this request via
_fetchChartData.

  • public/v2/app.js:

(App.Pane._fetch): Set chartData directly here.
(App.Pane._updateMovingAverageAndEnvelope): Renamed from _computeChartData. No longer sets chartData
now that it's done in App.Pane._fetch.
(App.AnalysisTaskController._fetchedRuns): Updated per createChartData merge.

  • public/v2/data.js:

(Measurement.prototype.buildId): Added.
(TimeSeries.prototype.findPointByBuild): Added.

  • public/v2/index.html: Fixed a bug that build status URL was broken. We can't use link-to helper since

url is not an Ember routed path.

  • public/v2/manifest.js:

(App.Manifest.fetchRunsWithPlatformAndMetric): Takes testGroupId as the third argument. Merged
App.createChartData here so that App.BuildRequest can use the formatter

Location:
trunk/Websites/perf.webkit.org
Files:
7 edited

Legend:

Unmodified
Added
Removed
  • trunk/Websites/perf.webkit.org/ChangeLog

    r179989 r180000  
     12015-02-12  Ryosuke Niwa  <rniwa@webkit.org>
     2
     3        Perf dashboard should show the results of A/B testing
     4        https://bugs.webkit.org/show_bug.cgi?id=141500
     5
     6        Reviewed by Chris Dumez.
     7
     8        Added the support for fetching test_runs for a specific test group in /api/runs/, and used it in the
     9        analysis task page to fetch results for each test group.
     10
     11        Merged App.createChartData into App.Manifest.fetchRunsWithPlatformAndMetric so that App.BuildRequest
     12        can use the formatter.
     13
     14        * public/api/runs.php:
     15        (fetch_runs_for_config_and_test_group): Added.
     16        (fetch_runs_for_config): Just return the fetched rows since main will format them with RunsGenerator.
     17        (main): Use fetch_runs_for_config_and_test_group to fetch rows when a test group id is specified. Also
     18        use RunsGenerator to format results.
     19        (RunsGenerator): Added.
     20        (RunsGenerator::__construct): Added.
     21        (RunsGenerator::add_runs): Added.
     22        (RunsGenerator::format_run): Moved.
     23        (RunsGenerator::parse_revisions_array): Moved.
     24
     25        * public/v2/analysis.js:
     26        (App.TestGroup): Fixed a typo. The property on a test group that refers to an analysis task is "task".
     27        (App.TestGroup._fetchChartData): Added. Fetches all A/B testing results for this group.
     28        (App.BuildRequest.configLetter): Renamed from config since this returns a letter that identifies the
     29        configuration associated with this build request such as "A" and "B".
     30        (App.BuildRequest.statusLabel): Added the missing label for failed build requests.
     31        (App.BuildRequest.url): Added. Returns the URL associated with this build request.
     32        (App.BuildRequest._meanFetched): Added. Retrieve the mean and the build number for this request via
     33        _fetchChartData.
     34
     35        * public/v2/app.js:
     36        (App.Pane._fetch): Set chartData directly here.
     37        (App.Pane._updateMovingAverageAndEnvelope): Renamed from _computeChartData. No longer sets chartData
     38        now that it's done in App.Pane._fetch.
     39        (App.AnalysisTaskController._fetchedRuns): Updated per createChartData merge.
     40
     41        * public/v2/data.js:
     42        (Measurement.prototype.buildId): Added.
     43        (TimeSeries.prototype.findPointByBuild): Added.
     44
     45        * public/v2/index.html: Fixed a bug that build status URL was broken. We can't use link-to helper since
     46        url is not an Ember routed path.
     47
     48        * public/v2/manifest.js:
     49        (App.Manifest.fetchRunsWithPlatformAndMetric): Takes testGroupId as the third argument. Merged
     50        App.createChartData here so that App.BuildRequest can use the formatter
     51
    1522015-02-12  Ryosuke Niwa  <rniwa@webkit.org>
    253
  • trunk/Websites/perf.webkit.org/public/api/runs.php

    r179591 r180000  
    33require('../include/json-header.php');
    44
     5function fetch_runs_for_config_and_test_group($db, $config, $test_group_id) {
     6    return $db->query_and_fetch_all('
     7        SELECT test_runs.*, builds.*, array_agg((commit_repository, commit_revision, commit_time)) AS revisions
     8            FROM builds
     9                LEFT OUTER JOIN build_commits ON commit_build = build_id
     10                LEFT OUTER JOIN commits ON build_commit = commit_id,
     11                test_runs, build_requests, analysis_test_groups
     12            WHERE run_build = build_id AND run_config = $1 AND request_build = build_id AND request_group = $2
     13            GROUP BY build_id, run_id', array($config['config_id'], $test_group_id));
     14}
     15
    516function fetch_runs_for_config($db, $config) {
    6     $raw_runs = $db->query_and_fetch_all('
     17    return $db->query_and_fetch_all('
    718        SELECT test_runs.*, builds.*, array_agg((commit_repository, commit_revision, commit_time)) AS revisions
    819            FROM builds
     
    1122            WHERE run_build = build_id AND run_config = $1 AND NOT EXISTS (SELECT * FROM build_requests WHERE request_build = build_id)
    1223            GROUP BY build_id, run_id', array($config['config_id']));
    13 
    14     $formatted_runs = array();
    15     if (!$raw_runs)
    16         return $formatted_runs;
    17 
    18     foreach ($raw_runs as $run)
    19         array_push($formatted_runs, format_run($run));
    20 
    21     return $formatted_runs;
    22 }
    23 
    24 function parse_revisions_array($postgres_array) {
    25     // e.g. {"(WebKit,131456,\"2012-10-16 14:53:00\")","(Chromium,162004,)"}
    26     $outer_array = json_decode('[' . trim($postgres_array, '{}') . ']');
    27     $revisions = array();
    28     foreach ($outer_array as $item) {
    29         $name_and_revision = explode(',', trim($item, '()'));
    30         if (!$name_and_revision[0])
    31             continue;
    32         $time = strtotime(trim($name_and_revision[2], '"')) * 1000;
    33         $revisions[trim($name_and_revision[0], '"')] = array(trim($name_and_revision[1], '"'), $time);
    34     }
    35     return $revisions;
    36 }
    37 
    38 function format_run($run) {
    39     return array(
    40         'id' => intval($run['run_id']),
    41         'mean' => floatval($run['run_mean_cache']),
    42         'iterationCount' => intval($run['run_iteration_count_cache']),
    43         'sum' => floatval($run['run_sum_cache']),
    44         'squareSum' => floatval($run['run_square_sum_cache']),
    45         'revisions' => parse_revisions_array($run['revisions']),
    46         'buildTime' => strtotime($run['build_time']) * 1000,
    47         'buildNumber' => intval($run['build_number']),
    48         'builder' => $run['build_builder']);
    4924}
    5025
     
    6136        exit_with_error('DatabaseConnectionFailure');
    6237
    63     // FIXME: We should support revalication as well as caching results in the server side.
    64     $maxage = config('jsonCacheMaxAge');
    65     header('Expires: ' . gmdate('D, d M Y H:i:s', time() + $maxage) . ' GMT');
    66     header("Cache-Control: maxage=$maxage");
    67 
    6838    $platform_id = intval($parts[0]);
    6939    $metric_id = intval($parts[1]);
     
    7343        exit_with_error('ConfigurationNotFound');
    7444
    75     $results = array();
    76     foreach ($config_rows as $config) {
    77         if ($runs = fetch_runs_for_config($db, $config))
    78             $results[$config['config_type']] = $runs;
     45    $test_group_id = array_get($_GET, 'testGroup');
     46    if ($test_group_id)
     47        $test_group_id = intval($test_group_id);
     48    else {
     49        // FIXME: We should support revalication as well as caching results in the server side.
     50        $maxage = config('jsonCacheMaxAge');
     51        header('Expires: ' . gmdate('D, d M Y H:i:s', time() + $maxage) . ' GMT');
     52        header("Cache-Control: maxage=$maxage");
    7953    }
    8054
    81     exit_with_success($results);
     55    $generator = new RunsGenerator();
     56
     57    foreach ($config_rows as $config) {
     58        if ($test_group_id)
     59            $raw_runs = fetch_runs_for_config_and_test_group($db, $config, $test_group_id);
     60        else
     61            $raw_runs = fetch_runs_for_config($db, $config);
     62        $generator->add_runs($config['config_type'], $raw_runs);
     63    }
     64
     65    exit_with_success($generator->results());
     66}
     67
     68class RunsGenerator {
     69    function __construct() {
     70        $this->results = array();
     71    }
     72
     73    function &results() { return $this->results; }
     74
     75    function add_runs($name, $raw_runs) {
     76        $formatted_runs = array();
     77        if ($raw_runs) {
     78            foreach ($raw_runs as $run)
     79                array_push($formatted_runs, self::format_run($run));
     80        }
     81        $this->results[$name] = $formatted_runs;
     82        return $formatted_runs;
     83    }
     84
     85    private static function format_run($run) {
     86        return array(
     87            'id' => intval($run['run_id']),
     88            'mean' => floatval($run['run_mean_cache']),
     89            'iterationCount' => intval($run['run_iteration_count_cache']),
     90            'sum' => floatval($run['run_sum_cache']),
     91            'squareSum' => floatval($run['run_square_sum_cache']),
     92            'revisions' => self::parse_revisions_array($run['revisions']),
     93            'build' => $run['build_id'],
     94            'buildTime' => strtotime($run['build_time']) * 1000,
     95            'buildNumber' => intval($run['build_number']),
     96            'builder' => $run['build_builder']);
     97    }
     98
     99    private static function parse_revisions_array($postgres_array) {
     100        // e.g. {"(WebKit,131456,\"2012-10-16 14:53:00\")","(Chromium,162004,)"}
     101        $outer_array = json_decode('[' . trim($postgres_array, '{}') . ']');
     102        $revisions = array();
     103        foreach ($outer_array as $item) {
     104            $name_and_revision = explode(',', trim($item, '()'));
     105            if (!$name_and_revision[0])
     106                continue;
     107            $time = strtotime(trim($name_and_revision[2], '"')) * 1000;
     108            $revisions[trim($name_and_revision[0], '"')] = array(trim($name_and_revision[1], '"'), $time);
     109        }
     110        return $revisions;
     111    }
    82112}
    83113
  • trunk/Websites/perf.webkit.org/public/v2/analysis.js

    r178234 r180000  
    6767
    6868App.TestGroup = App.NameLabelModel.extend({
    69     analysisTask: DS.belongsTo('analysisTask'),
     69    task: DS.belongsTo('analysisTask'),
    7070    author: DS.attr('string'),
    7171    createdAt: DS.attr('date'),
     
    8181        return rootSetIds;
    8282    }.property('buildRequests'),
     83    _fetchChartData: function ()
     84    {
     85        var task = this.get('task');
     86        if (!task)
     87            return null;
     88        var self = this;
     89        return App.Manifest.fetchRunsWithPlatformAndMetric(this.store,
     90            task.get('platform').get('id'), task.get('metric').get('id'), this.get('id')).then(
     91            function (result) { self.set('chartData', result.data); },
     92            function (error) {
     93                // FIXME: Somehow this never gets called.
     94                alert('Failed to fetch the results:' + error);
     95                return null;
     96            });
     97    }.observes('task', 'task.platform', 'task.metric').on('init'),
    8398});
    8499
     
    131146    }.property('order'),
    132147    rootSet: DS.attr('number'),
    133     config: function ()
     148    configLetter: function ()
    134149    {
    135150        var rootSets = this.get('testGroup').get('rootSets');
     
    147162        case 'running':
    148163            return 'Running';
     164        case 'failed':
     165            return 'Failed';
    149166        case 'completed':
    150167            return 'Finished';
    151168        }
    152169    }.property('status'),
     170    url: DS.attr('string'),
    153171    build: DS.attr('number'),
     172    _fetchMean: function ()
     173    {
     174        var testGroup = this.get('testGroup');
     175        if (!testGroup)
     176            return;
     177        var chartData = testGroup.get('chartData');
     178        if (!chartData)
     179            return;
     180
     181        var point = chartData.current.findPointByBuild(this.get('build'));
     182        if (!point)
     183            return;
     184        this.set('mean', chartData.formatter(point.value) + (chartData.unit ? ' ' + chartData.unit : ''));
     185        this.set('buildNumber', point.measurement.buildNumber());
     186    }.observes('build', 'testGroup', 'testGroup.chartData').on('init'),
    154187});
  • trunk/Websites/perf.webkit.org/public/v2/app.js

    r179965 r180000  
    353353                self.set('platform', result.platform);
    354354                self.set('metric', result.metric);
    355                 self.set('fetchedData', result);
    356                 self._computeChartData();
     355                self.set('chartData', result.data);
     356                self._updateMovingAverageAndEnvelope();
    357357            }, function (result) {
    358358                if (!result || typeof(result) === "string")
     
    472472        return chosenStrategy;
    473473    },
    474     _computeChartData: function ()
    475     {
    476         if (!this.get('fetchedData'))
    477             return;
    478 
    479         var chartData = App.createChartData(this.get('fetchedData'));
     474    _updateMovingAverageAndEnvelope: function ()
     475    {
     476        var chartData = this.get('chartData');
     477        if (!chartData)
     478            return;
    480479
    481480        var movingAverageStrategy = this.get('chosenMovingAverageStrategy');
     
    486485
    487486        chartData.movingAverage = this._computeMovingAverageAndOutliers(chartData, movingAverageStrategy, envelopingStrategy);
    488 
    489         this.set('chartData', chartData);
    490487    }.observes('chosenMovingAverageStrategy', 'chosenMovingAverageStrategy.parameterList.@each.value',
    491488        'chosenEnvelopingStrategy', 'chosenEnvelopingStrategy.parameterList.@each.value'),
     
    543540    },
    544541});
    545 
    546 App.createChartData = function (data)
    547 {
    548     var runs = data.runs;
    549     return {
    550         current: runs.current.timeSeriesByCommitTime(),
    551         baseline: runs.baseline ? runs.baseline.timeSeriesByCommitTime() : null,
    552         target: runs.target ? runs.target.timeSeriesByCommitTime() : null,
    553         unit: data.unit,
    554         formatter: data.useSI ? d3.format('.4s') : d3.format('.4g'),
    555         deltaFormatter: data.useSI ? d3.format('+.2s') : d3.format('+.2g'),
    556         smallerIsBetter: data.smallerIsBetter,
    557     };
    558 }
    559542
    560543App.encodePrettifiedJSON = function (plain)
     
    10281011        }));
    10291012    },
    1030     _fetchedRuns: function (data)
    1031     {
    1032         var runs = data.runs;
    1033 
    1034         var currentTimeSeries = runs.current.timeSeriesByCommitTime();
     1013    _fetchedRuns: function (result)
     1014    {
     1015        var chartData = result.data;
     1016        var currentTimeSeries = chartData.current;
    10351017        if (!currentTimeSeries)
    10361018            return; // FIXME: Report an error.
     
    10451027        highlightedItems[end.measurement.id()] = true;
    10461028
    1047         var chartData = App.createChartData(data);
    10481029        var formatedPoints = currentTimeSeries.seriesBetweenPoints(start, end).map(function (point, index) {
    10491030            return {
     
    10511032                measurement: point.measurement,
    10521033                label: 'Point ' + (index + 1),
    1053                 value: chartData.formatter(point.value) + (data.unit ? ' ' + data.unit : ''),
     1034                value: chartData.formatter(point.value) + (chartData.unit ? ' ' + chartData.unit : ''),
    10541035            };
    10551036        });
  • trunk/Websites/perf.webkit.org/public/v2/data.js

    r179965 r180000  
    243243}
    244244
     245Measurement.prototype.buildId = function()
     246{
     247    return this._raw['build'];
     248}
     249
    245250Measurement.prototype.buildNumber = function ()
    246251{
     
    316321// FIXME: We need to devise a way to fetch runs in multiple chunks so that
    317322// we don't have to fetch the entire time series to just show the last 3 days.
    318 RunsData.fetchRuns = function (platformId, metricId)
     323RunsData.fetchRuns = function (platformId, metricId, testGroupId)
    319324{
    320325    var filename = platformId + '-' + metricId + '.json';
     326
     327    if (testGroupId)
     328        filename += '?testGroup=' + testGroupId;
    321329
    322330    return new Ember.RSVP.Promise(function (resolve, reject) {
     
    356364}
    357365
     366TimeSeries.prototype.findPointByBuild = function (buildId)
     367{
     368    return this._series.find(function (point) { return point.measurement.buildId() == buildId; })
     369}
     370
    358371TimeSeries.prototype.findPointByMeasurementId = function (measurementId)
    359372{
  • trunk/Websites/perf.webkit.org/public/v2/index.html

    r179913 r180000  
    581581                                <tr>
    582582                                    <td>{{orderLabel}}</td>
    583                                     <td>{{config}}</td>
    584                                     <td>{{#if url}}{{#link-to url}}{{statusLabel}}{{/link-to}}{{else}}{{statusLabel}}{{/if}}</td>
    585                                     <td>{{build}}</td>
     583                                    <td>{{configLetter}}</td>
     584                                    <td><a {{bind-attr href=url}}>{{statusLabel}}</a></td>
     585                                    <td>{{buildNumber}}</td>
    586586                                    <td>{{mean}}</td>
    587587                                </tr>
  • trunk/Websites/perf.webkit.org/public/v2/manifest.js

    r179763 r180000  
    280280        this._defaultDashboardName = dashboards.length ? dashboards[0].get('name') : null;
    281281    },
    282     fetchRunsWithPlatformAndMetric: function (store, platformId, metricId)
     282    fetchRunsWithPlatformAndMetric: function (store, platformId, metricId, testGroupId)
    283283    {
    284284        return Ember.RSVP.all([
    285             RunsData.fetchRuns(platformId, metricId),
     285            RunsData.fetchRuns(platformId, metricId, testGroupId),
    286286            this.fetch(store),
    287287        ]).then(function (values) {
     
    302302            var smallerIsBetter = unit != 'fps' && unit != '/s'; // Assume smaller is better for unit-less metrics.
    303303
    304             return {platform: platform, metric: metric, runs: runs, unit: unit, useSI: unit == 'bytes', smallerIsBetter: smallerIsBetter};
     304            var useSI = unit == 'bytes';
     305            return {
     306                platform: platform,
     307                metric: metric,
     308                data: {
     309                    current: runs.current.timeSeriesByCommitTime(),
     310                    baseline: runs.baseline ? runs.baseline.timeSeriesByCommitTime() : null,
     311                    target: runs.target ? runs.target.timeSeriesByCommitTime() : null,
     312                    unit: unit,
     313                    formatter: useSI ? d3.format('.4s') : d3.format('.4g'),
     314                    deltaFormatter: useSI ? d3.format('+.2s') : d3.format('+.2g'),
     315                    smallerIsBetter: smallerIsBetter,
     316                }
     317            };
    305318        });
    306319    },
Note: See TracChangeset for help on using the changeset viewer.