⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 271211 in webkit


Ignore:
Timestamp:
Jan 6, 2021, 12:31:23 PM (6 years ago)
Author:
ap@apple.com
Message:

Assorted fixes for bisect-builds
​https://bugs.webkit.org/show_bug.cgi?id=220158

Reviewed by Jonathan Bedard.

Switched to Python 3.
Many trivial Python style changes.
Simplified arguments, and cleaned up output.
Switched from run-safari to run-minibrowser on macOS, as run-safari currently
doesn't work on regular macOS installations.

  • Scripts/bisect-builds:

(QueueDescriptor): New class that holds a description of a queue, or a "platform",
as coming from a string like mac-catalina or mac-catalina-x86_64-release. Used for
matching arguments to an existing archive directory on the server.
(trac_link): Added. The tool now prints a trac link for seeing where the regression
was introduced.
(bisect_builds): Fixed arithmetic mistakes. Got rid of an unnecessary while-true loop,
as the function was already recursive.
(host_platform_name): The script now defaults to current platform, so -p argument
is mostly to select simulator.
(main): Moved code from "if name == 'main'" block into main function for
consistency.

Location:
trunk/Tools
Files:
2 edited

Legend:

Unmodified
Added
Removed

  • trunk/Tools/ChangeLog

    r271207 r271211  
     12021-01-06  Alexey Proskuryakov  <ap@apple.com>
     2
     3        Assorted fixes for bisect-builds
     4        https://bugs.webkit.org/show_bug.cgi?id=220158
     5
     6        Reviewed by Jonathan Bedard.
     7
     8        Switched to Python 3.
     9        Many trivial Python style changes.
     10        Simplified arguments, and cleaned up output.
     11        Switched from run-safari to run-minibrowser on macOS, as run-safari currently
     12        doesn't work on regular macOS installations.
     13
     14        * Scripts/bisect-builds:
     15        (QueueDescriptor): New class that holds a description of a queue, or a "platform",
     16        as coming from a string like mac-catalina or mac-catalina-x86_64-release. Used for
     17        matching arguments to an existing archive directory on the server.
     18        (trac_link): Added. The tool now prints a trac link for seeing where the regression
     19        was introduced.
     20        (bisect_builds): Fixed arithmetic mistakes. Got rid of an unnecessary while-true loop,
     21        as the function was already recursive.
     22        (host_platform_name): The script now defaults to current platform, so -p argument
     23        is mostly to select simulator.
     24        (main): Moved code from "if __name__ == '__main__'" block into main function for
     25        consistency.
     26
    1272021-01-06  Alex Christensen  <achristensen@webkit.org>
    228

  • trunk/Tools/Scripts/bisect-builds

    r269052 r271211  
    1 #!/usr/bin/env python
    2 
    3 # Copyright (C) 2017 Apple Inc. All rights reserved.
     1#!/usr/bin/env python3
     2
     3# Copyright (C) 2017, 2020 Apple Inc. All rights reserved.
    44#
    55# Redistribution and use in source and binary forms, with or without
    … …  
    2727# THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
    2828
     29# A webkitpy import needs to go first for autoinstaller to work with subsequent imports.
     30from webkitpy.common.memoized import memoized
     31from webkitpy.common.system.systemhost import SystemHost
     32from webkitpy.common.version_name_map import VersionNameMap
     33from webkitpy.tool.grammar import pluralize
     34
    2935import argparse
    3036import bisect
    … …  
    3642import sys
    3743import tempfile
    38 import urllib2
    39 import urlparse
    40 from webkitpy.common.memoized import memoized
     44import urllib
     45
    4146
    4247REST_API_URL = 'https://q1tzqfy48e.execute-api.us-west-2.amazonaws.com/v2_2/'
    … …  
    4752
    4853
     54class QueueDescriptor(object):
     55    def __init__(self, descriptor_string):
     56        self.platform_name = None
     57        self.version = None
     58        self.architectures = set()
     59        self.configuration = None
     60
     61        if descriptor_string.startswith('mac-'):
     62            platform_name_end_index = descriptor_string.find('-')
     63            version_start_index = platform_name_end_index + 1
     64            version_end_index = descriptor_string.find('-', version_start_index)
     65            self.platform_name = descriptor_string[:platform_name_end_index]
     66            if version_end_index == -1:
     67                self.version = descriptor_string[version_start_index:]
     68                return
     69            self.version = descriptor_string[version_start_index:version_end_index]
     70            architectures_and_configuration = descriptor_string[version_end_index + 1:]
     71        elif descriptor_string.startswith('ios-simulator-'):
     72            platform_name_end_index = descriptor_string.find('-', descriptor_string.find('-') + 1)
     73            version_start_index = platform_name_end_index + 1
     74            version_end_index = descriptor_string.find('-', version_start_index)
     75            self.platform_name = descriptor_string[:platform_name_end_index]
     76            if version_end_index == -1:
     77                self.version = descriptor_string[version_start_index:]
     78                return
     79            self.version = descriptor_string[version_start_index:version_end_index]
     80            architectures_and_configuration = descriptor_string[version_end_index + 1:]
     81        else:
     82            platform_name_end_index = descriptor_string.find('-')
     83            if platform_name_end_index == -1:
     84                self.platform_name = descriptor_string
     85                return
     86            self.platform_name = descriptor_string[:platform_name_end_index]
     87            architectures_and_configuration = descriptor_string[platform_name_end_index + 1:]
     88        architectures_end_index = architectures_and_configuration.find('-')
     89        if architectures_end_index == -1:
     90            self.architectures = set(architectures_and_configuration.split(' '))
     91            return
     92        configuration_start_index = architectures_end_index + 1
     93        self.architectures = set(architectures_and_configuration[:architectures_end_index].split(' '))
     94        self.configuration = architectures_and_configuration[configuration_start_index:]
     95
     96    def pretty_string(self):
     97        result = self.platform_name
     98        if self.version:
     99            result += '-' + self.version
     100        result += ' (' + ' '.join(self.architectures)
     101        result += ', ' + self.configuration + ')'
     102        return result
     103
     104
     105def trac_link(start_revision, end_revision):
     106    if start_revision + 1 == end_revision:
     107        return 'https://trac.webkit.org/r{}'.format(end_revision)
     108    else:
     109        return 'https://trac.webkit.org/log/trunk/?mode=follow_copy&rev={}&stop_rev={}'.format(end_revision, start_revision + 1)
     110
     111
    49112def bisect_builds(revision_list, start_index, end_index, options):
    50     while True:
    51         index_to_test = pick_next_build(revision_list, start_index, end_index)
    52         if index_to_test == None:
    53             print('No more builds to test...')
    54             exit(1)
    55         download_archive(options, revision_list[index_to_test])
    56         extract_archive(options)
    57         reproduces = test_archive(options, revision_list[index_to_test])
    58 
    59         if reproduces:          # bisect left
    60             index_to_test -= 1  # We can remove this from the end of the list of builds to test
    61             bisect_builds(revision_list, start_index, index_to_test, options)
    62         if not reproduces:      # bisect right
    63             index_to_test += 1  # We can remove this from the start of the list of builds to test
    64             bisect_builds(revision_list, index_to_test, end_index, options)
    65 
    66 
     113    index_to_test = pick_next_build(revision_list, start_index, end_index)
     114    if index_to_test is None:
     115        print('\nWorks: r{}'.format(revision_list[start_index]))
     116        print('Fails: r{}'.format(revision_list[end_index]))
     117        print(trac_link(revision_list[start_index], revision_list[end_index]))
     118        exit(0)
     119
     120    archive_count = end_index - start_index - 1
     121    print('Bisecting between r{} and r{}, {} in the range.'.format(revision_list[start_index], revision_list[end_index], pluralize(archive_count, 'archive')))
     122    reproduces = test_revision(options, revision_list[index_to_test])
     123
     124    if reproduces:
     125        bisect_builds(revision_list, start_index, index_to_test, options)
     126    else:
     127        bisect_builds(revision_list, index_to_test, end_index, options)
     128
     129
     130# download-built-product and built-product-archive implicitly use WebKitBuild directory for downloaded archive.
     131# FIXME: Modifying the WebKitBuild directory makes no sense here, find a way to use a temporary directory for the archive.
    67132def download_archive(options, revision):
    68133    api_url = get_api_archive_url(options)
    69134    s3_url = get_s3_location_for_revision(api_url, revision)
    70     print('Archive URL: {}'.format(s3_url))
     135    print('Downloading r{}: {}'.format(revision, s3_url))
    71136    command = ['python', '../CISupport/download-built-product', '--{}'.format(options.configuration), '--platform', options.platform, s3_url]
    72     print('Downloading revision: {}'.format(revision))
    73137    subprocess.check_call(command)
    74138
    75139
    76140def extract_archive(options):
    77     command = ['python', '../CISupport/built-product-archive', '--platform', options.platform, '--%s' % options.configuration, 'extract']
     141    command = ['python', '../CISupport/built-product-archive', '--{}'.format(options.configuration), '--platform', options.platform, 'extract']
    78142    subprocess.check_call(command)
    79143
    … …  
    97161
    98162
    99 def get_api_archive_url(options, LastEvaluatedKey=None):
     163def get_api_archive_url(options, last_evaluated_key=None):
    100164    if options.full:
    101         base_url = urlparse.urljoin(REST_API_URL, REST_API_ARCHIVE_ENDPOINT)
    102     else:
    103         base_url = urlparse.urljoin(REST_API_URL, REST_API_MINIFIED_ARCHIVE_ENDPOINT)
    104 
    105     api_url = urlparse.urljoin(base_url, '-'.join([options.platform, options.architecture, options.configuration]))
    106     if LastEvaluatedKey:
    107         querystring = urllib2.quote(json.dumps(LastEvaluatedKey))
     165        base_url = urllib.parse.urljoin(REST_API_URL, REST_API_ARCHIVE_ENDPOINT)
     166    else:
     167        base_url = urllib.parse.urljoin(REST_API_URL, REST_API_MINIFIED_ARCHIVE_ENDPOINT)
     168
     169    api_url = urllib.parse.urljoin(base_url, urllib.parse.quote(options.queue))
     170    if last_evaluated_key:
     171        querystring = urllib.parse.quote(json.dumps(last_evaluated_key))
    108172        api_url += '?ExclusiveStartKey=' + querystring
    109        
     173
    110174    return api_url
    111175
    … …  
    113177def get_indices_from_revisions(revision_list, start_revision, end_revision):
    114178    if start_revision is None:
    115         print('WARNING: No starting revision was given, defaulting to first available for this configuration')
     179        print('WARNING: No starting revision was given, defaulting to first available for this configuration.')
    116180        start_index = 0
    117181    else:
    … …  
    119183
    120184    if end_revision is None:
    121         print('WARNING: No ending revision was given, defaulting to last avialable for this configuration')
     185        print('WARNING: No ending revision was given, defaulting to last available for this configuration.')
    122186        end_index = len(revision_list) - 1
    123187    else:
    … …  
    130194    revisions = [int(item['revision']['N']) for item in revisions_dict['revisions']['Items']]
    131195    return sorted(revisions)
    132    
     196
    133197
    134198def get_s3_location_for_revision(url, revision):
    135199    url = '/'.join([url, str(revision)])
    136     r = urllib2.urlopen(url)
     200    r = urllib.request.urlopen(url)
    137201    data = json.load(r)
    138    
     202
    139203    for archive in data['archive']:
    140204        s3_url = archive['s3_url']
    … …  
    142206
    143207
     208def host_platform_name():
     209    platform = SystemHost().platform
     210    version_name = VersionNameMap.strip_name_formatting(platform.os_version_name())
     211    if version_name is None:
     212        return platform.os_name
     213    return platform.os_name + '-' + version_name
     214
     215
    144216def parse_args(args):
    145     helptext = 'bisect-builds is designed to help pinpoint regressions to specific code changes. It does this by bisecting across archives produced by build.webkit.org. Full and "minified" archives are available. Minified archives are significantly smaller, as they have been stripped of dSYMs and other non-essential components.'
     217    helptext = 'bisect-builds helps pinpoint regressions to specific code changes. It does this by bisecting across archives produced by build.webkit.org. Full and "minified" archives are available. Minified archives are significantly smaller, as they have been stripped of dSYMs and other non-essential components.'
    146218    parser = argparse.ArgumentParser(description=helptext)
    147     parser.add_argument('-c', '--configuration', default='release', help='The configuration to query [release | debug]')
    148     parser.add_argument('-a', '--architecture', default='x86_64', help='The architecture to query [x86_64 | i386]')
    149     parser.add_argument('-p', '--platform', default='None', required=True, help='The platform to query [mac-sierra | gtk | ios-simulator | win]')
    150     parser.add_argument('-f', '--full', action='store_true', default=False, help='Use full archives containing debug symbols. These are significantly larger files!')
    151     parser.add_argument('-s', '--start', default=None, type=int, help='The starting revision to bisect.')
    152     parser.add_argument('-e', '--end', default=None, type=int, help='The ending revision to bisect')
    153     parser.add_argument('-l', '--list', action='store_true', default=False, help='Display a list of platforms and revisions')
     219    parser.add_argument('-c', '--configuration', default='release', help='the configuration to query [release | debug]')
     220    parser.add_argument('-a', '--architecture', help='the architecture to query, e.g. x86_64, default is no preference')
     221    parser.add_argument('-p', '--platform', default=host_platform_name(), help='the platform to query, e.g. mac-bigsur, gtk, ios-simulator-14, win, default is current host platform.')
     222    parser.add_argument('-f', '--full', action='store_true', default=False, help='use full archives containing debug symbols, which are significantly larger files')
     223    parser.add_argument('-s', '--start', default=None, type=int, help='the starting revision to bisect')
     224    parser.add_argument('-e', '--end', default=None, type=int, help='the ending revision to bisect')
     225    parser.add_argument('--sanity-check', action='store_true', default=False, help='verify both starting and ending revisions before bisecting')
     226    parser.add_argument('-l', '--list', action='store_true', default=False, help='display a list of platforms and revisions')
    154227    return parser.parse_args(args)
    155228
    156229
    157230def pick_next_build(revision_list, start_index, end_index):
    158     revisions_remaining = (end_index - start_index) + 1
    159     print('Found {} revisions in this range to test...'.format(revisions_remaining))
    160 
    161     if start_index >= end_index:
    162         print('No archives available between {} and {}'.format(revision_list[end_index], revision_list[start_index]))
     231    if start_index + 1 >= end_index:
     232        print('No archives available between r{} and r{}.'.format(revision_list[start_index], revision_list[end_index]))
    163233        return None
    164234
    165     middleIndex = (start_index + end_index) / 2
    166     return int(math.ceil(middleIndex))
     235    middle_index = (start_index + end_index) / 2
     236    return int(math.ceil(middle_index))
    167237
    168238
    169239def prompt_did_reproduce():
    170     var = raw_input('\nDid the error reproduce? [y/n]: ')
     240    var = input('\nDid the error reproduce? [y/n]: ')
    171241    var = var.lower()
    172242    if 'y' in var:
    … …  
    176246    else:
    177247        prompt_did_reproduce()
    178    
     248
    179249
    180250def set_webkit_output_dir(temp_dir):
    181     print('Setting environment variable WEBKIT_OUTPUTDIR to {}'.format(temp_dir))
     251    print('Archives will be extracted to {}'.format(temp_dir))
    182252    os.environ['WEBKIT_OUTPUTDIR'] = temp_dir
    183253
    184254
    185 def test_archive(options, revision):
    186     print('Testing revision {}...'.format(revision))
    187     command = []
    188     if 'mac' in options.platform:
    189         command = ['./run-safari']
    190     elif 'ios' in options.platform:
    191         command = ['./run-safari', '--simulator']
    192     else:
    193         print('Default test behavior for this platform is not implemented...'.format(options.platform))
     255def test_revision(options, revision):
     256    download_archive(options, revision)
     257    extract_archive(options)
     258    if options.platform.startswith('ios-simulator'):
     259        command = ['./run-safari', '--iphone-simulator', '--{}'.format(options.configuration)]
     260    else:
     261        command = ['./run-minibrowser', '--{}'.format(options.configuration)]
    194262
    195263    if command:
    196         subprocess.call(command)
     264        subprocess.call(command, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
    197265    return prompt_did_reproduce()
    198266
    199267
    200268def get_platforms(endpoint):
    201     platform_url = urlparse.urljoin(REST_API_URL, endpoint)
    202     r = urllib2.urlopen(platform_url)
     269    platform_url = urllib.parse.urljoin(REST_API_URL, endpoint)
     270    r = urllib.request.urlopen(platform_url)
    203271    data = json.load(r)
    204272    platforms = []
    205273    for platform in data.get('Items'):
    206274        platforms.append(str(platform['identifier']['S']))
    207    
     275
    208276    return platforms
    209      
     277
     278
    210279@memoized
    211280def minified_platforms():
    212281    return get_platforms(REST_API_MINIFIED_PLATFORM_ENDPOINT)
    213  
     282
     283
    214284@memoized
    215285def unminified_platforms():
    216286    return get_platforms(REST_API_PLATFORM_ENDPOINT)
    217  
    218 def is_supported_platform(options):
    219     platform = '-'.join([options.platform, options.architecture, options.configuration])   
     287
     288
     289def queue_for(options):
    220290    if options.full:
    221         return platform in unminified_platforms()
    222     return platform in minified_platforms()
     291        platform_list = unminified_platforms()
     292    else:
     293        platform_list = minified_platforms()
     294
     295    descriptor_from_options = QueueDescriptor(options.platform)
     296
     297    if not descriptor_from_options.architectures:
     298        if options.architecture:
     299            descriptor_from_options.architectures = set(options.architecture)
     300    elif options.architecture is not None and descriptor_from_options.architectures != {options.architecture}:
     301        return None
     302
     303    if not descriptor_from_options.configuration:
     304        if options.configuration:
     305            descriptor_from_options.configuration = options.configuration
     306    elif options.configuration is not None and descriptor_from_options.configuration != options.configuration:
     307        return None
     308
     309    for platform_name in platform_list:
     310        available_platform = QueueDescriptor(platform_name)
     311        if descriptor_from_options.platform_name != available_platform.platform_name:
     312            continue
     313        if descriptor_from_options.version and descriptor_from_options.version != available_platform.version:
     314            continue
     315        if not descriptor_from_options.architectures.issubset(available_platform.architectures):
     316            continue
     317        if descriptor_from_options.configuration and descriptor_from_options.configuration != available_platform.configuration:
     318            continue
     319        return platform_name
     320    return None
     321
     322
     323def print_platforms(platforms):
     324    platform_strings = ['    {}'.format(QueueDescriptor(queue_name).pretty_string()) for queue_name in platforms]
     325    print('\n'.join(sorted(platform_strings)))
    223326
    224327
    225328def validate_options(options):
    226     if not is_supported_platform(options):
    227         print('Unsupported platform combination: [{}], exiting...'.format('-'.join([options.platform, options.architecture, options.configuration])))
     329    options.queue = queue_for(options)  # Resolve and cache for future use.
     330    if options.queue is None:
     331        print('Unsupported platform combination, exiting.')
    228332        if options.full:
    229             print('Available Unminified platforms: {}'.format(unminified_platforms()))
     333            print('Available unminified platforms:')
     334            print_platforms(unminified_platforms())
    230335        else:
    231             print('Available Minified platforms: {}'.format(minified_platforms()))
    232             print('INFO: pass --full to try against full archives')
     336            print('Available minified platforms:')
     337            print_platforms(minified_platforms())
    233338        exit(1)
    234339
     340
    235341def print_list_and_exit(revision_list, options):
    236         print('Supported minified platforms: {}'.format(minified_platforms()))
    237         print('Supported unminified platforms: {}'.format(unminified_platforms()))
    238         print('{} revisions available for supplied platform: {}-{}-{}:'.format(len(revision_list), options.platform, options.architecture, options.configuration))
    239         print(revision_list)
    240         exit(0)
    241 
    242 def fetch_revision_list(options, LastEvaluatedKey=None):
    243     url = get_api_archive_url(options, LastEvaluatedKey)
    244     r = urllib2.urlopen(url)
     342    print('Supported minified platforms:')
     343    print_platforms(minified_platforms())
     344    print('Supported unminified platforms:')
     345    print_platforms(unminified_platforms())
     346    print('{} revisions available for {}:'.format(len(revision_list), options.queue))
     347    print(revision_list)
     348    exit(0)
     349
     350
     351def fetch_revision_list(options, last_evaluated_key=None):
     352    url = get_api_archive_url(options, last_evaluated_key)
     353    r = urllib.request.urlopen(url)
    245354    data = json.load(r)
    246355    revision_list = get_sorted_revisions(data)
    247356
    248357    if 'LastEvaluatedKey' in data['revisions']:
    249         LastEvaluatedKey = data['revisions']['LastEvaluatedKey']
    250         revision_list += fetch_revision_list(options, LastEvaluatedKey)
    251        
     358        last_evaluated_key = data['revisions']['LastEvaluatedKey']
     359        revision_list += fetch_revision_list(options, last_evaluated_key)
     360
    252361    return revision_list
    253    
    254 def main(options):
    255     validate_options(options)
    256     revision_list = fetch_revision_list(options)
    257    
    258     if options.list:
    259         print_list_and_exit(revision_list, options)
    260 
    261     start_index, end_index = get_indices_from_revisions(revision_list, options.start, options.end)
    262     print('Bisecting between {} and {}'.format(revision_list[start_index], revision_list[end_index]))
    263    
    264     # from here forward, use indices instead of revisions
    265     bisect_builds(revision_list, start_index, end_index, options)
    266 
    267 
    268 if __name__ == '__main__':
     362
     363
     364def main():
    269365    options = parse_args(sys.argv[1:])
    270366    script_path = os.path.abspath(__file__)
    … …  
    272368    os.chdir(script_directory)
    273369    webkit_output_dir = tempfile.mkdtemp()
     370
     371    validate_options(options)
     372    revision_list = fetch_revision_list(options)
     373
     374    if options.list:
     375        print_list_and_exit(revision_list, options)
     376
     377    if not revision_list:
     378        print('No archives found for {}.'.format(options.queue))
     379        exit(1)
     380    start_index, end_index = get_indices_from_revisions(revision_list, options.start, options.end)
     381
    274382    set_webkit_output_dir(webkit_output_dir)
     383
     384    # From here forward, use indices instead of revisions.
    275385    try:
    276         main(options)   
     386        if options.sanity_check:
     387            if test_revision(options, revision_list[start_index]):
     388                print('Issue reproduced with the first revision in the range, cannot bisect.')
     389                exit(1)
     390            if not test_revision(options, revision_list[end_index]):
     391                print('Issue did not reproduce with the last revision in the range, cannot bisect.')
     392                exit(1)
     393
     394        bisect_builds(revision_list, start_index, end_index, options)
    277395    except KeyboardInterrupt:
    278         exit("Aborting.")
     396        exit(1)
    279397    finally:
    280398        shutil.rmtree(webkit_output_dir, ignore_errors=True)
     399
     400
     401if __name__ == '__main__':
     402    main()
Note: See TracChangeset for help on using the changeset viewer.