Changeset 109190 in webkit


Ignore:
Timestamp:
Feb 28, 2012 10:06:55 PM (12 years ago)
Author:
rniwa@webkit.org
Message:

perf-o-matic: generate dashboard images using Google Chart Tools
​https://bugs.webkit.org/show_bug.cgi?id=79838

Reviewed by Hajime Morita.

Rename RunsJSONGenerator to Runs and added an ability to generate parameters for Google chart tool.
Also added RunsChartHandler to make url-fetches these images and DashboardImageHandler to serve them.
The image is stored in DashboardImage model.

We can't enable flip the switch to use images yet because we don't create images on fly (they're
generated when runs are updated; i.e. bots upload new results). We should be able to flip the switch
once this patch lands and all perf bots cycle.

We probably make way too many calls to Google chart tool's server with this preliminary design but we
can easily move this task into the backend and run it via a cron job once we know it works.

  • Websites/webkit-perf.appspot.com/controller.py:

(schedule_runs_update):
(RunsUpdateHandler.post):
(RunsChartHandler):
(RunsChartHandler.get):
(RunsChartHandler.post):
(DashboardImageHandler):
(DashboardImageHandler.get):
(schedule_report_process):

  • Websites/webkit-perf.appspot.com/json_generators.py:

(ManifestJSONGenerator.value):
(Runs):
(Runs.init):
(Runs.value):
(Runs.chart_params):

  • Websites/webkit-perf.appspot.com/json_generators_unittest.py:

(RunsTest):
(RunsTest._create_results):
(RunsTest.test_generate_runs):
(RunsTest.test_value_without_results):
(RunsTest.test_value_with_results):
(RunsTest.test_run_from_build_and_result):
(RunsTest.test_chart_params_with_value):
(RunsTest.test_chart_params_with_value.split_as_int):

  • Websites/webkit-perf.appspot.com/main.py:
  • Websites/webkit-perf.appspot.com/models.py:

(PersistentCache.get_cache):
(DashboardImage):
(DashboardImage.key_name):

Location:
trunk
Files:
6 edited

Legend:

Unmodified
Added
Removed

  • trunk/ChangeLog

    r109180 r109190  
     12012-02-28  Ryosuke Niwa  <rniwa@webkit.org>
     2
     3        perf-o-matic: generate dashboard images using Google Chart Tools
     4        https://bugs.webkit.org/show_bug.cgi?id=79838
     5
     6        Reviewed by Hajime Morita.
     7
     8        Rename RunsJSONGenerator to Runs and added an ability to generate parameters for Google chart tool.
     9        Also added RunsChartHandler to make url-fetches these images and DashboardImageHandler to serve them.
     10        The image is stored in DashboardImage model.
     11
     12        We can't enable flip the switch to use images yet because we don't create images on fly (they're
     13        generated when runs are updated; i.e. bots upload new results). We should be able to flip the switch
     14        once this patch lands and all perf bots cycle.
     15
     16        We probably make way too many calls to Google chart tool's server with this preliminary design but we
     17        can easily move this task into the backend and run it via a cron job once we know it works.
     18
     19        * Websites/webkit-perf.appspot.com/controller.py:
     20        (schedule_runs_update):
     21        (RunsUpdateHandler.post):
     22        (RunsChartHandler):
     23        (RunsChartHandler.get):
     24        (RunsChartHandler.post):
     25        (DashboardImageHandler):
     26        (DashboardImageHandler.get):
     27        (schedule_report_process):
     28        * Websites/webkit-perf.appspot.com/json_generators.py:
     29        (ManifestJSONGenerator.value):
     30        (Runs):
     31        (Runs.__init__):
     32        (Runs.value):
     33        (Runs.chart_params):
     34        * Websites/webkit-perf.appspot.com/json_generators_unittest.py:
     35        (RunsTest):
     36        (RunsTest._create_results):
     37        (RunsTest.test_generate_runs):
     38        (RunsTest.test_value_without_results):
     39        (RunsTest.test_value_with_results):
     40        (RunsTest.test_run_from_build_and_result):
     41        (RunsTest.test_chart_params_with_value):
     42        (RunsTest.test_chart_params_with_value.split_as_int):
     43        * Websites/webkit-perf.appspot.com/main.py:
     44        * Websites/webkit-perf.appspot.com/models.py:
     45        (PersistentCache.get_cache):
     46        (DashboardImage):
     47        (DashboardImage.key_name):
     48
    1492012-02-28  Dave Tu  <dtu@chromium.org>
    250

  • trunk/Websites/webkit-perf.appspot.com/controller.py

    r108917 r109190  
    2828# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
    2929
     30import urllib
    3031import webapp2
    3132from google.appengine.api import taskqueue
    … …  
    3435from json_generators import DashboardJSONGenerator
    3536from json_generators import ManifestJSONGenerator
    36 from json_generators import RunsJSONGenerator
     37from json_generators import Runs
    3738from models import Branch
     39from models import DashboardImage
    3840from models import Platform
    3941from models import Test
    … …  
    98100def schedule_runs_update(test_id, branch_id, platform_id):
    99101    taskqueue.add(url='/api/test/runs/update', params={'id': test_id, 'branchid': branch_id, 'platformid': platform_id})
     102    taskqueue.add(url='/api/test/runs/chart', params={'id': test_id, 'branchid': branch_id, 'platformid': platform_id})
    100103
    101104
    … …  
    126129        assert test
    127130
    128         cache_runs(test_id, branch_id, platform_id, RunsJSONGenerator(branch, platform, test.name).to_json())
     131        cache_runs(test_id, branch_id, platform_id, Runs(branch, platform, test.name).to_json())
    129132        self.response.out.write('OK')
    130133
    … …  
    142145
    143146
     147class RunsChartHandler(webapp2.RequestHandler):
     148    def get(self):
     149        self.post()
     150
     151    def post(self):
     152        self.response.headers['Content-Type'] = 'text/plain; charset=utf-8'
     153        test_id, branch_id, platform_id = _get_test_branch_platform_ids(self)
     154
     155        branch = model_from_numeric_id(branch_id, Branch)
     156        platform = model_from_numeric_id(platform_id, Platform)
     157        test = model_from_numeric_id(test_id, Test)
     158        display_days = int(self.request.get('displayDays'))
     159        assert branch
     160        assert platform
     161        assert test
     162
     163        params = Runs(branch, platform, test.name).chart_params(display_days)
     164        dashboard_chart_file = urllib.urlopen('http://chart.googleapis.com/chart', urllib.urlencode(params))
     165
     166        DashboardImage(key_name=DashboardImage.key_name(branch.id, platform.id, test.id, display_days),
     167            image=dashboard_chart_file.read()).put()
     168
     169        self.response.out.write('Fetched http://chart.googleapis.com/chart?%s' % urllib.urlencode(params))
     170
     171
     172class DashboardImageHandler(webapp2.RequestHandler):
     173    def get(self, test_id, branch_id, platform_id, display_days):
     174        try:
     175            branch_id = int(branch_id)
     176            platform_id = int(platform_id)
     177            test_id = int(test_id)
     178            display_days = int(display_days)
     179        except ValueError:
     180            self.response.headers['Content-Type'] = 'text/plain; charset=utf-8'
     181            self.response.out.write('Failed')
     182
     183        self.response.headers['Content-Type'] = 'image/png'
     184        image = DashboardImage.get_by_key_name(DashboardImage.key_name(branch_id, platform_id, test_id, display_days))
     185        if image:
     186            self.response.out.write(image.image)
     187
     188
    144189def schedule_report_process(log):
     190    self.response.headers['Content-Type'] = 'application/json'
    145191    taskqueue.add(url='/api/test/report/process', params={'id': log.key().id()})

  • trunk/Websites/webkit-perf.appspot.com/json_generators.py

    r108399 r109190  
    2929
    3030import json
     31from datetime import datetime
     32from datetime import timedelta
    3133from time import mktime
    3234
    … …  
    119121
    120122
    121 class RunsJSONGenerator(JSONGeneratorBase):
    122     def __init__(self, branch, platform, test):
    123         self._test_runs = []
    124         self._averages = {}
    125         values = []
    126 
    127         for build, result in RunsJSONGenerator._generate_runs(branch, platform, test):
    128             self._test_runs.append(RunsJSONGenerator._entry_from_build_and_result(build, result))
    129             # FIXME: Calculate the average. In practice, we wouldn't have more than one value for a given revision.
    130             self._averages[build.revision] = result.value
    131             values.append(result.value)
    132 
    133         self._min = min(values) if values else None
    134         self._max = max(values) if values else None
     123# FIXME: This isn't a JSON generator anymore. We should move it elsewhere or rename the file.
     124class Runs(JSONGeneratorBase):
     125    def __init__(self, branch, platform, test_name):
     126        self._branch = branch
     127        self._platform = platform
     128        self._test_name = test_name
    135129
    136130    @staticmethod
    … …  
    168162
    169163    def value(self):
     164        _test_runs = []
     165        _averages = {}
     166        values = []
     167
     168        for build, result in Runs._generate_runs(self._branch, self._platform, self._test_name):
     169            _test_runs.append(Runs._entry_from_build_and_result(build, result))
     170            # FIXME: Calculate the average. In practice, we wouldn't have more than one value for a given revision.
     171            _averages[build.revision] = result.value
     172            values.append(result.value)
     173
     174        _min = min(values) if values else None
     175        _max = max(values) if values else None
     176
    170177        return {
    171             'test_runs': self._test_runs,
    172             'averages': self._averages,
    173             'min': self._min,
    174             'max': self._max,
     178            'test_runs': _test_runs,
     179            'averages': _averages,
     180            'min': _min,
     181            'max': _max,
    175182            'date_range': None,  # Never used by common.js.
    176183            'stat': 'ok'}
     184
     185    def chart_params(self, display_days, now=datetime.now()):
     186        chart_data_x = []
     187        chart_data_y = []
     188        end_time = now
     189        start_timestamp = mktime((end_time - timedelta(display_days)).timetuple())
     190        end_timestamp = mktime(end_time.timetuple())
     191
     192        for build, result in self._generate_runs(self._branch, self._platform, self._test_name):
     193            timestamp = mktime(build.timestamp.timetuple())
     194            if timestamp < start_timestamp or timestamp > end_timestamp:
     195                continue
     196            chart_data_x.append(timestamp)
     197            chart_data_y.append(result.value)
     198
     199        dates = [end_time + timedelta(day - display_days) for day in range(0, display_days + 1)]
     200
     201        y_max = max(chart_data_y) * 1.1
     202        y_grid_step = y_max / 5
     203        y_axis_label_step = int(y_grid_step + 0.5)  # This won't work for decimal numbers
     204
     205        return {
     206            'cht': 'lxy',  # Specify with X and Y coordinates
     207            'chxt': 'x,y',  # Display both X and Y axies
     208            'chxl': '0:|' + '|'.join([date.strftime('%b %d') for date in dates]),  # X-axis labels
     209            'chxr': '1,0,%f,%f' % (int(y_max + 0.5), y_axis_label_step),  # Y-axis range: min=0, max, step
     210            'chds': '%f,%f,%f,%f' % (start_timestamp, end_timestamp, 0, y_max),  # X, Y data range
     211            'chxs': '1,676767,11.167,0,l,676767',  # Y-axis label: 1,color,font-size,centerd on tick,axis line/no ticks, tick color
     212            'chs': '360x240',  # Image size: 360px by 240px
     213            'chco': 'ff0000',  # Plot line color
     214            'chg': '%f,%f,0,0' % (100 / (len(dates) - 1), y_grid_step),  # X, Y grid line step sizes - max for X is 100.
     215            'chls': '3',  # Line thickness
     216            'chf': 'bg,s,eff6fd',  # Transparent background
     217            'chd': 't:' + ','.join([str(x) for x in chart_data_x]) + '|' + ','.join([str(y) for y in chart_data_y]),  # X, Y data
     218        }

  • trunk/Websites/webkit-perf.appspot.com/json_generators_unittest.py

    r108399 r109190  
    3434from google.appengine.ext import testbed
    3535from datetime import datetime
     36from datetime import timedelta
    3637from json_generators import JSONGeneratorBase
    3738from json_generators import DashboardJSONGenerator
    3839from json_generators import ManifestJSONGenerator
    39 from json_generators import RunsJSONGenerator
     40from json_generators import Runs
    4041from models_unittest import DataStoreTestsBase
    4142from models import Branch
    … …  
    186187
    187188
    188 class RunsJSONGeneratorTest(DataStoreTestsBase):
    189     def _create_results(self, branch, platform, builder, test_name, values):
     189class RunsTest(DataStoreTestsBase):
     190    def _create_results(self, branch, platform, builder, test_name, values, timestamps=None):
    190191        results = []
    191192        for i, value in enumerate(values):
    192193            build = Build(branch=branch, platform=platform, builder=builder,
    193                 buildNumber=i, revision=100 + i, timestamp=datetime.now())
     194                buildNumber=i, revision=100 + i, timestamp=timestamps[i] if timestamps else datetime.now())
    194195            build.put()
    195196            result = TestResult(name=test_name, build=build, value=value)
    … …  
    205206        results = self._create_results(some_branch, some_platform, some_builder, 'some-test', [50.0, 51.0, 52.0, 49.0, 48.0])
    206207        last_i = 0
    207         for i, (build, result) in enumerate(RunsJSONGenerator._generate_runs(some_branch, some_platform, "some-test")):
     208        for i, (build, result) in enumerate(Runs._generate_runs(some_branch, some_platform, "some-test")):
    208209            self.assertEqual(build.buildNumber, i)
    209210            self.assertEqual(build.revision, 100 + i)
    … …  
    218219        self.assertThereIsNoInstanceOf(Test)
    219220        self.assertThereIsNoInstanceOf(TestResult)
    220         self.assertEqual(RunsJSONGenerator(some_branch, some_platform, 'some-test').value(), {
     221        self.assertEqual(Runs(some_branch, some_platform, 'some-test').value(), {
    221222            'test_runs': [],
    222223            'averages': {},
    … …  
    232233        results = self._create_results(some_branch, some_platform, some_builder, 'some-test', [50.0, 51.0, 52.0, 49.0, 48.0])
    233234
    234         value = RunsJSONGenerator(some_branch, some_platform, 'some-test').value()
     235        value = Runs(some_branch, some_platform, 'some-test').value()
    235236        self.assertEqualUnorderedList(value.keys(), ['test_runs', 'averages', 'min', 'max', 'date_range', 'stat'])
    236237        self.assertEqual(value['stat'], 'ok')
    … …  
    275276        result = TestResult(name=test_name, value=123.0, build=build)
    276277        result.put()
    277         self._assert_entry(RunsJSONGenerator._entry_from_build_and_result(build, result), build, result, 123.0)
     278        self._assert_entry(Runs._entry_from_build_and_result(build, result), build, result, 123.0)
    278279
    279280        build = create_build(2, 102)
    280281        result = TestResult(name=test_name, value=456.0, valueMedian=789.0, build=build)
    281282        result.put()
    282         self._assert_entry(RunsJSONGenerator._entry_from_build_and_result(build, result), build, result, 456.0)
     283        self._assert_entry(Runs._entry_from_build_and_result(build, result), build, result, 456.0)
    283284
    284285        result.valueStdev = 7.0
    285286        result.put()
    286         self._assert_entry(RunsJSONGenerator._entry_from_build_and_result(build, result), build, result, 456.0)
     287        self._assert_entry(Runs._entry_from_build_and_result(build, result), build, result, 456.0)
    287288
    288289        result.valueStdev = None
    … …  
    290291        result.valueMax = 789.0
    291292        result.put()
    292         self._assert_entry(RunsJSONGenerator._entry_from_build_and_result(build, result), build, result, 456.0)
     293        self._assert_entry(Runs._entry_from_build_and_result(build, result), build, result, 456.0)
    293294
    294295        result.valueStdev = 8.0
    … …  
    296297        result.valueMax = 789.0
    297298        result.put()
    298         self._assert_entry(RunsJSONGenerator._entry_from_build_and_result(build, result), build, result, 456.0,
     299        self._assert_entry(Runs._entry_from_build_and_result(build, result), build, result, 456.0,
    299300            statistics={'stdev': 8.0, 'min': 123.0, 'max': 789.0})
    300301
    … …  
    304305        result.valueMax = 789.0
    305306        result.put()
    306         self._assert_entry(RunsJSONGenerator._entry_from_build_and_result(build, result), build, result, 456.0,
     307        self._assert_entry(Runs._entry_from_build_and_result(build, result), build, result, 456.0,
    307308            statistics={'stdev': 8.0, 'min': 123.0, 'max': 789.0})
     309
     310    def test_chart_params_with_value(self):
     311        some_branch = Branch.create_if_possible('some-branch', 'Some Branch')
     312        some_platform = Platform.create_if_possible('some-platform', 'Some Platform')
     313        some_builder = Builder.get(Builder.create('some-builder', 'Some Builder'))
     314
     315        start_time = datetime(2011, 2, 21, 12, 0, 0)
     316        end_time = datetime(2011, 2, 28, 12, 0, 0)
     317        results = self._create_results(some_branch, some_platform, some_builder, 'some-test',
     318            [50.0, 51.0, 52.0, 49.0, 48.0, 51.9, 50.7, 51.1],
     319            [start_time + timedelta(day) for day in range(0, 8)])
     320
     321        # Use int despite of its impreciseness since tests may fail due to rounding errors otherwise.
     322        def split_as_int(string):
     323            return [int(float(value)) for value in string.split(',')]
     324
     325        params = Runs(some_branch, some_platform, 'some-test').chart_params(7, end_time)
     326        self.assertEqual(params['chxl'], '0:|Feb 21|Feb 22|Feb 23|Feb 24|Feb 25|Feb 26|Feb 27|Feb 28')
     327        self.assertEqual(split_as_int(params['chxr']), [1, 0, 57, int(52 * 1.1 / 5 + 0.5)])
     328        x_min, x_max, y_min, y_max = split_as_int(params['chds'])
     329        self.assertEqual(datetime.fromtimestamp(x_min), start_time)
     330        self.assertEqual(datetime.fromtimestamp(x_max), end_time)
     331        self.assertEqual(y_min, 0)
     332        self.assertEqual(y_max, int(52 * 1.1))
     333        self.assertEqual(split_as_int(params['chg']), [int(100 / 7), int(52 * 1.1 / 5), 0, 0])
     334
    308335
    309336

  • trunk/Websites/webkit-perf.appspot.com/main.py

    r109057 r109190  
    2727from controller import CachedManifestHandler
    2828from controller import CachedRunsHandler
     29from controller import DashboardImageHandler
    2930from controller import DashboardUpdateHandler
    3031from controller import ManifestUpdateHandler
     32from controller import RunsChartHandler
    3133from controller import RunsUpdateHandler
    3234from create_handler import CreateHandler
    … …  
    4244    ('/admin/create/(.*)', CreateHandler),
    4345    (r'/admin/([A-Za-z\-]*)', AdminDashboardHandler),
     46
    4447    ('/api/user/is-admin', IsAdminHandler),
    4548    ('/api/test/?', CachedManifestHandler),
    … …  
    4952    ('/api/test/runs/?', CachedRunsHandler),
    5053    ('/api/test/runs/update', RunsUpdateHandler),
     54    ('/api/test/runs/chart', RunsChartHandler),
    5155    ('/api/test/dashboard/?', CachedDashboardHandler),
    5256    ('/api/test/dashboard/update', DashboardUpdateHandler),
    53 ]
     57
     58    ('/images/dashboard/flot-(\d+)-(\d+)-(\d+)_(\d+).png', DashboardImageHandler)]
    5459
    5560

  • trunk/Websites/webkit-perf.appspot.com/models.py

    r109057 r109190  
    323323        memcache.set(name, cache.value)
    324324        return cache.value
     325
     326
     327class DashboardImage(db.Model):
     328    image = db.BlobProperty(required=True)
     329
     330    @staticmethod
     331    def key_name(branch_id, platform_id, test_id, display_days):
     332        return '%d:%d:%d:%d' % (branch_id, platform_id, test_id, display_days)
Note: See TracChangeset for help on using the changeset viewer.