Changeset 107274 in webkit


Ignore:
Timestamp:
Feb 9, 2012 12:20:41 PM (12 years ago)
Author:
rniwa@webkit.org
Message:

Perf-o-matic should update memcache in taskqueue
​https://bugs.webkit.org/show_bug.cgi?id=78209

Reviewed by Adam Barth.

Update dashboard, manifest, and runs memcaches in taskqueue.
Also centralized the management of caches in controller.py.

  • Websites/webkit-perf.appspot.com/app.yaml:
  • Websites/webkit-perf.appspot.com/controller.py: Added.

(cache_manifest):
(schedule_manifest_update):
(CachedManifestHandler):
(CachedManifestHandler.get):
(cache_dashboard):
(schedule_dashboard_update):
(CachedDashboardHandler):
(CachedDashboardHandler.get):
(cache_runs):
(schedule_runs_update):
(CachedRunsHandler):
(CachedRunsHandler.get):

  • Websites/webkit-perf.appspot.com/create_handler.py:

(CreateHandler.post):

  • Websites/webkit-perf.appspot.com/dashboard_handler.py:

(DashboardHandler.post):

  • Websites/webkit-perf.appspot.com/main.py:
  • Websites/webkit-perf.appspot.com/manifest_handler.py:

(ManifestHandler.post):

  • Websites/webkit-perf.appspot.com/merge_tests_handler.py:

(MergeTestsHandler.post):

  • Websites/webkit-perf.appspot.com/report_handler.py:

(ReportHandler.post):

  • Websites/webkit-perf.appspot.com/runs_handler.py:

(RunsHandler.post):

Location:
trunk
Files:
1 added
9 edited

Legend:

Unmodified
Added
Removed

  • trunk/ChangeLog

    r107250 r107274  
     12012-02-09  Ryosuke Niwa  <rniwa@webkit.org>
     2
     3        Perf-o-matic should update memcache in taskqueue
     4        https://bugs.webkit.org/show_bug.cgi?id=78209
     5
     6        Reviewed by Adam Barth.
     7
     8        Update dashboard, manifest, and runs memcaches in taskqueue.
     9        Also centralized the management of caches in controller.py.
     10
     11        * Websites/webkit-perf.appspot.com/app.yaml:
     12        * Websites/webkit-perf.appspot.com/controller.py: Added.
     13        (cache_manifest):
     14        (schedule_manifest_update):
     15        (CachedManifestHandler):
     16        (CachedManifestHandler.get):
     17        (cache_dashboard):
     18        (schedule_dashboard_update):
     19        (CachedDashboardHandler):
     20        (CachedDashboardHandler.get):
     21        (cache_runs):
     22        (schedule_runs_update):
     23        (CachedRunsHandler):
     24        (CachedRunsHandler.get):
     25        * Websites/webkit-perf.appspot.com/create_handler.py:
     26        (CreateHandler.post):
     27        * Websites/webkit-perf.appspot.com/dashboard_handler.py:
     28        (DashboardHandler.post):
     29        * Websites/webkit-perf.appspot.com/main.py:
     30        * Websites/webkit-perf.appspot.com/manifest_handler.py:
     31        (ManifestHandler.post):
     32        * Websites/webkit-perf.appspot.com/merge_tests_handler.py:
     33        (MergeTestsHandler.post):
     34        * Websites/webkit-perf.appspot.com/report_handler.py:
     35        (ReportHandler.post):
     36        * Websites/webkit-perf.appspot.com/runs_handler.py:
     37        (RunsHandler.post):
     38
    1392012-02-09  Carlos Garcia Campos  <cgarcia@igalia.com>
    240

  • trunk/Websites/webkit-perf.appspot.com/app.yaml

    r106890 r107274  
    11application: webkit-perf
    2 version: 11
     2version: 12
    33runtime: python27
    44api_version: 1

  • trunk/Websites/webkit-perf.appspot.com/create_handler.py

    r106321 r107274  
    2929
    3030import webapp2
    31 from google.appengine.api import memcache
    3231from google.appengine.ext import db
    3332
    3433import json
    3534
     35from controller import schedule_dashboard_update
    3636from models import Builder
    3737from models import Branch
    … …  
    4343class CreateHandler(webapp2.RequestHandler):
    4444    def post(self, model):
    45         self.response.headers['Content-Type'] = 'text/plain; charset=utf-8';
     45        self.response.headers['Content-Type'] = 'text/plain; charset=utf-8'
    4646
    4747        try:
    … …  
    6464
    6565        # No need to clear manifest or runs since they only contain ones with test results
    66         memcache.delete('dashboard')
     66        schedule_dashboard_update()
    6767        self.response.out.write(error + '\n' if error else 'OK')
    6868

  • trunk/Websites/webkit-perf.appspot.com/dashboard_handler.py

    r106321 r107274  
    2929
    3030import webapp2
    31 from google.appengine.api import memcache
    32 
    3331import json
    3432
     33from controller import cache_dashboard
    3534from models import Builder
    3635from models import Branch
    … …  
    4039
    4140class DashboardHandler(webapp2.RequestHandler):
    42     def get(self):
    43         self.response.headers['Content-Type'] = 'application/json; charset=utf-8';
    44         cache = memcache.get('dashboard')
    45         if cache:
    46             self.response.out.write(cache)
     41    def post(self):
     42        self.response.headers['Content-Type'] = 'text/plain; charset=utf-8'
     43        webkit_trunk = Branch.get_by_key_name('webkit-trunk')
     44        if not webkit_trunk:
     45            self.response.out.write("BAD: no webkit-trunk")
    4746            return
    48 
    49         webkit_trunk = Branch.get_by_key_name('webkit-trunk')
    5047
    5148        # FIXME: Determine popular branches, platforms, and tests
    … …  
    6360            dashboard['testToId'][test.name] = test.id
    6461
    65         result = json.dumps(dashboard)
    66         self.response.out.write(result)
    67         memcache.add('dashboard', result)
     62        cache_dashboard(json.dumps(dashboard))
     63        self.response.out.write('OK')

  • trunk/Websites/webkit-perf.appspot.com/main.py

    r106292 r107274  
    2121import json
    2222
     23from controller import CachedDashboardHandler
     24from controller import CachedManifestHandler
     25from controller import CachedRunsHandler
    2326from create_handler import CreateHandler
    2427from dashboard_handler import DashboardHandler
    … …  
    3336    ('/admin/merge-tests/?', MergeTestsHandler),
    3437    ('/admin/create/(.*)', CreateHandler),
    35     ('/api/test/?', ManifestHandler),
     38    ('/api/test/?', CachedManifestHandler),
     39    ('/api/test/update', ManifestHandler),
    3640    ('/api/test/report/?', ReportHandler),
    37     ('/api/test/runs/?', RunsHandler),
    38     ('/api/test/dashboard/?', DashboardHandler),
     41    ('/api/test/runs/?', CachedRunsHandler),
     42    ('/api/test/runs/update', RunsHandler),
     43    ('/api/test/dashboard/?', CachedDashboardHandler),
     44    ('/api/test/dashboard/update', DashboardHandler),
    3945]
    4046

  • trunk/Websites/webkit-perf.appspot.com/manifest_handler.py

    r106334 r107274  
    2929
    3030import webapp2
    31 from google.appengine.api import memcache
    3231
    3332import json
    3433
     34from controller import cache_manifest
    3535from models import Builder
    3636from models import Branch
    … …  
    4040
    4141class ManifestHandler(webapp2.RequestHandler):
    42     def get(self):
    43         self.response.headers['Content-Type'] = 'text/plain; charset=utf-8';
    44         cache = memcache.get('manifest')
    45         if cache:
    46             self.response.out.write(cache)
    47             return
     42    def post(self):
     43        self.response.headers['Content-Type'] = 'text/plain; charset=utf-8'
    4844
    4945        test_map = {}
    … …  
    8985            }
    9086
    91         result = json.dumps({'testMap': test_map, 'platformMap': platform_map, 'branchMap': branch_map})
    92         self.response.out.write(result)
    93         memcache.add('manifest', result)
     87        cache_manifest(json.dumps({'testMap': test_map, 'platformMap': platform_map, 'branchMap': branch_map}))
     88        self.response.out.write('OK')

  • trunk/Websites/webkit-perf.appspot.com/merge_tests_handler.py

    r106321 r107274  
    2929
    3030import webapp2
    31 from google.appengine.api import memcache
    3231from google.appengine.ext.webapp import template
    3332
    3433import os
    3534
     35from controller import schedule_runs_update
     36from controller import schedule_dashboard_update
     37from controller import schedule_manifest_update
    3638from models import Test
    3739from models import TestResult
    … …  
    5456        merged_results = TestResult.all()
    5557        merged_results.filter('name =', merge.name)
     58        branches_and_platforms_to_update = set()
    5659        for result in merged_results:
     60            branches_and_platforms_to_update.add((result.build.branch.id, result.build.platform.id))
    5761            result.name = into.name
    5862            result.put()
    5963
    60         # Just flush everyting since we rarely merge tests and we need to flush
    61         # dashboard, manifest, and all runs for this test here.
    62         memcache.flush_all()
     64        for branch_id, platform_id in branches_and_platforms_to_update:
     65            schedule_runs_update(into.id, branch_id, platform_id)
     66
     67        schedule_dashboard_update()
     68        schedule_manifest_update()
    6369
    6470        delete_model_with_numeric_id_holder(merge)

  • trunk/Websites/webkit-perf.appspot.com/report_handler.py

    r106687 r107274  
    2929
    3030import webapp2
    31 from google.appengine.api import memcache
    3231from google.appengine.ext import db
    3332
    … …  
    3736from datetime import datetime
    3837
     38from controller import schedule_runs_update
     39from controller import schedule_dashboard_update
     40from controller import schedule_manifest_update
    3941from models import Builder
    4042from models import Branch
    … …  
    9698        for test_name, result in self._body['results'].iteritems():
    9799            test = self._add_test_if_needed(test_name, branch, platform)
    98             memcache.delete(Test.cache_key(test.id, branch.id, platform.id))
     100            schedule_runs_update(test.id, branch.id, platform.id)
    99101            if isinstance(result, dict):
    100102                TestResult(name=test_name, build=build, value=float(result['avg']), valueMedian=_float_or_none(result, 'median'),
    … …  
    107109
    108110        # We need to update dashboard and manifest because they are affected by the existance of test results
    109         memcache.delete('dashboard')
    110         memcache.delete('manifest')
     111        schedule_dashboard_update()
     112        schedule_manifest_update()
    111113
    112114        return self._output('OK')

  • trunk/Websites/webkit-perf.appspot.com/runs_handler.py

    r106890 r107274  
    2929
    3030import webapp2
    31 from google.appengine.api import memcache
    3231
    3332import json
    … …  
    3534from datetime import datetime
    3635
     36from controller import cache_runs
    3737from models import Build
    3838from models import Builder
    … …  
    4646
    4747class RunsHandler(webapp2.RequestHandler):
    48     def get(self):
    49         self.response.headers['Content-Type'] = 'application/json; charset=utf-8'
     48    def post(self):
     49        self.response.headers['Content-Type'] = 'text/plain; charset=utf-8'
    5050
    5151        try:
    … …  
    6161        # FIXME: Just fetch builds specified by "days"
    6262        # days = self.request.get('days', 365)
    63 
    64         cache_key = Test.cache_key(test_id, branch_id, platform_id)
    65         cache = memcache.get(cache_key)
    66         if cache:
    67             self.response.out.write(cache)
    68             return
    6963
    7064        builds = Build.all()
    … …  
    106100            'date_range': [min(timestamps), max(timestamps)] if timestamps else None,
    107101            'stat': 'ok'})
    108         self.response.out.write(result)
    109         memcache.add(cache_key, result)
     102        cache_runs(test_id, branch_id, platform_id, result)
     103        self.response.out.write('OK')
Note: See TracChangeset for help on using the changeset viewer.