Changeset 190817 in webkit


Ignore:
Timestamp:
Oct 9, 2015 2:00:27 PM (8 years ago)
Author:
rniwa@webkit.org
Message:

Perf dashboard erroneously shows an old OS build in A/B testing range
https://bugs.webkit.org/show_bug.cgi?id=149942

Reviewed by Darin Adler.

Ordering OS builds lexicologically turned out be a bad idea since 15A25 falls between 15A242 and 15A251.
Use a fake/synthetic timestamp to force the commonly understood total order instead.

Refactored pull-os-versions.py to share the server config JSON with other scripts. Also made the script
support pulling multiple sources; e.g. both OS X and iOS.

Also removed superfluous feature to submit results in chunks. The perf dashboard can handle thousands of
revisions being submitted at once just fine.

  • public/api/commits.php:

(main): A partial revert of r185574 since we no longer need to order builds lexicologically.

  • tools/pull-os-versions.py:

(main): Takes --os-config-json, --server-config-json, and --seconds-to-sleep as arguments instead of
a single --config argument to share the server config JSON with other scripts.
(OSBuildFetcher): Extracted out of main. This class is instantiated for each OS kind (e.g. OS X).
(OSBuildFetcher.init): Added.
(OSBuildFetcher._fetch_available_builds): Extracted out of main. Fetches available builds from a website
or custom commands.
(OSBuildFetcher.fetch_and_report_new_builds): Extracted out of main. Submits the fetched builds after
filtering out the ones we've already reported.
(OSBuildFetcher._assign_fake_timestamps): Creates a fake timestamp to establish a total order amongst each
OS X / iOS style build number such as 12A3456b.

Location:
trunk/Websites/perf.webkit.org
Files:
3 edited

Legend:

Unmodified
Added
Removed
  • trunk/Websites/perf.webkit.org/ChangeLog

    r190764 r190817  
     12015-10-08  Ryosuke Niwa  <rniwa@webkit.org>
     2
     3        Perf dashboard erroneously shows an old OS build in A/B testing range
     4        https://bugs.webkit.org/show_bug.cgi?id=149942
     5
     6        Reviewed by Darin Adler.
     7
     8        Ordering OS builds lexicologically turned out be a bad idea since 15A25 falls between 15A242 and 15A251.
     9        Use a fake/synthetic timestamp to force the commonly understood total order instead.
     10
     11        Refactored pull-os-versions.py to share the server config JSON with other scripts. Also made the script
     12        support pulling multiple sources; e.g. both OS X and iOS.
     13
     14        Also removed superfluous feature to submit results in chunks. The perf dashboard can handle thousands of
     15        revisions being submitted at once just fine.
     16
     17        * public/api/commits.php:
     18        (main): A partial revert of r185574 since we no longer need to order builds lexicologically.
     19
     20        * tools/pull-os-versions.py:
     21        (main): Takes --os-config-json, --server-config-json, and --seconds-to-sleep as arguments instead of
     22        a single --config argument to share the server config JSON with other scripts.
     23        (OSBuildFetcher): Extracted out of main. This class is instantiated for each OS kind (e.g. OS X).
     24        (OSBuildFetcher.__init__): Added.
     25        (OSBuildFetcher._fetch_available_builds): Extracted out of main. Fetches available builds from a website
     26        or custom commands.
     27        (OSBuildFetcher.fetch_and_report_new_builds): Extracted out of main. Submits the fetched builds after
     28        filtering out the ones we've already reported.
     29        (OSBuildFetcher._assign_fake_timestamps): Creates a fake timestamp to establish a total order amongst each
     30        OS X / iOS style build number such as 12A3456b.
     31
    1322015-10-08  Ryosuke Niwa  <rniwa@webkit.org>
    233
  • trunk/Websites/perf.webkit.org/public/api/commits.php

    r185574 r190817  
    8080        $first = $first_commit['commit_time'];
    8181        $second = $second_commit['commit_time'];
    82         $column = 'commit_time';
    83         if (!$first && !$second) {
    84             $first = $first_commit['commit_revision'];
    85             $second = $second_commit['commit_revision'];
    86             $column = 'commit_revision';
    87         }
    8882        $in_order = $first < $second;
    8983        array_push($values, $in_order ? $first : $second);
    90         $statements .= ' AND ' . $column . ' >= $' . count($values);
     84        $statements .= ' AND commit_time >= $' . count($values);
    9185        array_push($values, $in_order ? $second : $first);
    92         $statements .= ' AND ' . $column . ' <= $' . count($values);
     86        $statements .= ' AND commit_time <= $' . count($values);
    9387    }
    9488
  • trunk/Websites/perf.webkit.org/tools/pull-os-versions.py

    r186033 r190817  
    33import argparse
    44import json
     5import operator
    56import re
    67import sys
     
    1011import urllib2
    1112
     13from datetime import datetime
    1214from xml.dom.minidom import parseString as parseXmlString
    1315from util import submit_commits
     
    1820def main(argv):
    1921    parser = argparse.ArgumentParser()
    20     parser.add_argument('--config', required=True, help='Path to a config JSON file')
     22    parser.add_argument('--os-config-json', required=True, help='The path to a JSON that specifies how to fetch OS build information')
     23    parser.add_argument('--server-config-json', required=True, help='The path to a JSON file that specifies the perf dashboard')
     24    parser.add_argument('--seconds-to-sleep', type=float, default=43200, help='The seconds to sleep between iterations')
    2125    args = parser.parse_args()
    2226
    23     with open(args.config) as config_file:
    24         config = json.load(config_file)
     27    with open(args.os_config_json) as os_config_json:
     28        os_config_list = json.load(os_config_json)
    2529
    26     setup_auth(config['server'])
     30    with open(args.server_config_json) as server_config_json:
     31        server_config = json.load(server_config_json)
     32        setup_auth(server_config['server'])
    2733
    28     submission_size = config['submissionSize']
    29     reported_revisions = set()
     34    fetchers = [OSBuildFetcher(os_config) for os_config in os_config_list]
    3035
    3136    while True:
     37        for fetcher in fetchers:
     38            fetcher.fetch_and_report_new_builds(server_config)
     39        print "Sleeping for %d seconds" % args.seconds_to_sleep
     40        time.sleep(args.seconds_to_sleep)
     41
     42
     43# FIXME: Move other static functions into this class.
     44class OSBuildFetcher:
     45    def __init__(self, os_config):
     46        self._os_config = os_config
     47        self._reported_revisions = set()
     48
     49    def _fetch_available_builds(self):
     50        config = self._os_config
     51        repository_name = self._os_config['name']
     52
    3253        if 'customCommands' in config:
    3354            available_builds = []
    3455            for command in config['customCommands']:
    3556                print "Executing", ' '.join(command['command'])
    36                 available_builds += available_builds_from_command(config['repositoryName'], command['command'], command['linesToIgnore'])
    37                 print "Got %d builds" % len(available_builds)
     57                available_builds += available_builds_from_command(repository_name, command['command'], command['linesToIgnore'])
    3858        else:
    3959            url = config['buildSourceURL']
    4060            print "Fetching available builds from", url
    41             available_builds = fetch_available_builds(config['repositoryName'], url, config['trainVersionMap'])
     61            available_builds = fetch_available_builds(repository_name, url, config['trainVersionMap'])
     62        return available_builds
     63
     64    def fetch_and_report_new_builds(self, server_config):
     65        available_builds = self._fetch_available_builds()
     66        reported_revisions = self._reported_revisions
     67        print 'Found %d builds' % len(available_builds)
    4268
    4369        available_builds = filter(lambda commit: commit['revision'] not in reported_revisions, available_builds)
    44         print "%d builds available" % len(available_builds)
     70        self._assign_fake_timestamps(available_builds)
    4571
    46         while available_builds:
    47             commits_to_submit = available_builds[:submission_size]
    48             revisions_to_report = map(lambda commit: commit['revision'], commits_to_submit)
    49             print "Submitting builds (%d remaining):" % len(available_builds), json.dumps(revisions_to_report)
    50             available_builds = available_builds[submission_size:]
     72        print "Submitting %d builds" % len(available_builds)
     73        submit_commits(available_builds, server_config['server']['url'], server_config['slave']['name'], server_config['slave']['password'])
     74        reported_revisions |= set(map(lambda commit: commit['revision'], available_builds))
    5175
    52             submit_commits(commits_to_submit, config['server']['url'], config['slave']['name'], config['slave']['password'])
    53             reported_revisions |= set(revisions_to_report)
    54 
    55             time.sleep(config['submissionInterval'])
    56 
    57         print "Sleeping for %d seconds" % config['fetchInterval']
    58         time.sleep(config['fetchInterval'])
     76    @staticmethod
     77    def _assign_fake_timestamps(builds):
     78        build_name_regex = re.compile(r'(?P<major>\d+)(?P<kind>\w)(?P<minor>\d+)(?P<variant>\w*)')
     79        for commit in builds:
     80            match = build_name_regex.search(commit['revision'])
     81            major = int(match.group('major'))
     82            kind = ord(match.group('kind').upper()) - ord('A')
     83            minor = int(match.group('minor'))
     84            variant = ord(match.group('variant').upper()) - ord('A') + 1 if match.group('variant') else 0
     85            # These fake times won't conflict with real commit time since even 99Z9999z is still in Feb 1973
     86            fake_time = datetime.utcfromtimestamp((major * 100 + kind) * 10000 + minor + float(variant) / 100)
     87            commit['time'] = fake_time.isoformat()
    5988
    6089
Note: See TracChangeset for help on using the changeset viewer.