Changeset 270447 in webkit


Ignore:
Timestamp:
Dec 4, 2020 12:03:53 PM (3 years ago)
Author:
Jonathan Bedard
Message:

[git-webkit] Use contributors.json
​https://bugs.webkit.org/show_bug.cgi?id=217732
<rdar://problem/70309518>

Reviewed by Dewei Zhu.

The interaction between Git, Svn and old commits means that our canonical record of
commit authors is somewhat incomplete. contributors.json has most of the information
we are missing, we should rely on it to map non-standard author names to their canonical
names and email addresses.

Additionally, making the record of Contributors owned by the repository instead of being global
to the entire process.

  • Scripts/git-webkit: Parse contributors.json and add it to a Contributor.Mapping.
  • Scripts/libraries/webkitscmpy/webkitscmpy/init.py: Bump version.
  • Scripts/libraries/webkitscmpy/webkitscmpy/commit.py:

(Commit): Contributor class object no longer owns record of contributors.

  • Scripts/libraries/webkitscmpy/webkitscmpy/contributor.py:

(Contributor):
(Contributor.Mapping): Dictionary mapping Contributors and their potential aliases.
(Contributor.Mapping.init):
(Contributor.Mapping.add): Add Contributor to mapping.
(Contributor.Mapping.create): Find or create a contributor with the specified name and email
addresses and bind it to the record of contributors.
(Contributor.from_scm_log): Leverage Contributor.Mapping provided by caller.
(Contributor.clear): Deleted.

  • Scripts/libraries/webkitscmpy/webkitscmpy/local/git.py:

(Git.init): Instantiate repository with existing Contributor.Mapping.
(Git.commit): Repository now owns the record of contributors.

  • Scripts/libraries/webkitscmpy/webkitscmpy/local/scm.py:

(Scm.from_path): Instantiate repository with existing Contributor.Mapping.
(Scm.init): Ditto.

  • Scripts/libraries/webkitscmpy/webkitscmpy/local/svn.py:

(Svn.init): Instantiate repository with existing Contributor.Mapping.
(Svn.commit): Repository now owns the record of contributors.

  • Scripts/libraries/webkitscmpy/webkitscmpy/program.py:

(main): Allow caller to pass an existing record of contributors.

  • Scripts/libraries/webkitscmpy/webkitscmpy/remote/scm.py:

(Scm.from_url): Instantiate repository with existing Contributor.Mapping.
(Scm.init): Ditto.

  • Scripts/libraries/webkitscmpy/webkitscmpy/remote/svn.py:

(Svn.init): Instantiate repository with existing Contributor.Mapping.
(Svn.commit): Repository now owns the record of contributors.

  • Scripts/libraries/webkitscmpy/webkitscmpy/scm_base.py:

(ScmBase.init): Instantiate repository with existing Contributor.Mapping.

  • Scripts/libraries/webkitscmpy/webkitscmpy/test/commit_unittest.py:
  • Scripts/libraries/webkitscmpy/webkitscmpy/test/contributor_unittest.py:

(TestContributor.test_git_log): No more global contributor record.
(TestContributor.test_git_svn_log): Ditto.
(TestContributor.test_git_no_author): Ditto.
(TestContributor.test_git_svn_no_author): Ditto.
(TestContributor.test_svn_log): Ditto.
(TestContributor.test_short_svn_log): Ditto.
(TestContributor.test_svn_patch_by_log): Ditto.
(TestContributor.test_author_mapping): Contributor record is owned by the caller of the caller
of Contributor management.
(TestContributor.test_email_mapping): Ditto.
(TestContributor.test_invalid_log): No more global contributor record.

Location:
trunk/Tools
Files:
14 edited

Legend:

Unmodified
Added
Removed

  • trunk/Tools/ChangeLog

    r270446 r270447  
     12020-12-04  Jonathan Bedard  <jbedard@apple.com>
     2
     3        [git-webkit] Use contributors.json
     4        https://bugs.webkit.org/show_bug.cgi?id=217732
     5        <rdar://problem/70309518>
     6
     7        Reviewed by Dewei Zhu.
     8
     9        The interaction between Git, Svn and old commits means that our canonical record of
     10        commit authors is somewhat incomplete. contributors.json has most of the information
     11        we are missing, we should rely on it to map non-standard author names to their canonical
     12        names and email addresses.
     13
     14        Additionally, making the record of Contributors owned by the repository instead of being global
     15        to the entire process.
     16
     17        * Scripts/git-webkit: Parse contributors.json and add it to a Contributor.Mapping.
     18        * Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py: Bump version.
     19        * Scripts/libraries/webkitscmpy/webkitscmpy/commit.py:
     20        (Commit): Contributor class object no longer owns record of contributors.
     21        * Scripts/libraries/webkitscmpy/webkitscmpy/contributor.py:
     22        (Contributor):
     23        (Contributor.Mapping): Dictionary mapping Contributors and their potential aliases.
     24        (Contributor.Mapping.__init__):
     25        (Contributor.Mapping.add): Add Contributor to mapping.
     26        (Contributor.Mapping.create): Find or create a contributor with the specified name and email
     27        addresses and bind it to the record of contributors.
     28        (Contributor.from_scm_log): Leverage Contributor.Mapping provided by caller.
     29        (Contributor.clear): Deleted.
     30        * Scripts/libraries/webkitscmpy/webkitscmpy/local/git.py:
     31        (Git.__init__): Instantiate repository with existing Contributor.Mapping.
     32        (Git.commit): Repository now owns the record of contributors.
     33        * Scripts/libraries/webkitscmpy/webkitscmpy/local/scm.py:
     34        (Scm.from_path): Instantiate repository with existing Contributor.Mapping.
     35        (Scm.__init__): Ditto.
     36        * Scripts/libraries/webkitscmpy/webkitscmpy/local/svn.py:
     37        (Svn.__init__): Instantiate repository with existing Contributor.Mapping.
     38        (Svn.commit): Repository now owns the record of contributors.
     39        * Scripts/libraries/webkitscmpy/webkitscmpy/program.py:
     40        (main): Allow caller to pass an existing record of contributors.
     41        * Scripts/libraries/webkitscmpy/webkitscmpy/remote/scm.py:
     42        (Scm.from_url): Instantiate repository with existing Contributor.Mapping.
     43        (Scm.__init__): Ditto.
     44        * Scripts/libraries/webkitscmpy/webkitscmpy/remote/svn.py:
     45        (Svn.__init__): Instantiate repository with existing Contributor.Mapping.
     46        (Svn.commit): Repository now owns the record of contributors.
     47        * Scripts/libraries/webkitscmpy/webkitscmpy/scm_base.py:
     48        (ScmBase.__init__): Instantiate repository with existing Contributor.Mapping.
     49        * Scripts/libraries/webkitscmpy/webkitscmpy/test/commit_unittest.py:
     50        * Scripts/libraries/webkitscmpy/webkitscmpy/test/contributor_unittest.py:
     51        (TestContributor.test_git_log): No more global contributor record.
     52        (TestContributor.test_git_svn_log): Ditto.
     53        (TestContributor.test_git_no_author): Ditto.
     54        (TestContributor.test_git_svn_no_author): Ditto.
     55        (TestContributor.test_svn_log): Ditto.
     56        (TestContributor.test_short_svn_log): Ditto.
     57        (TestContributor.test_svn_patch_by_log): Ditto.
     58        (TestContributor.test_author_mapping): Contributor record is owned by the caller of the caller
     59        of Contributor management.
     60        (TestContributor.test_email_mapping): Ditto.
     61        (TestContributor.test_invalid_log): No more global contributor record.
     62
    1632020-12-04  Kate Cheney  <katherine_cheney@apple.com>
    264

  • trunk/Tools/Scripts/git-webkit

    r268433 r270447  
    2424
    2525import os
    26 import webkitpy
    2726import sys
    2827
    29 from webkitscmpy import program
     28from webkitpy.common.config.committers import CommitterList
     29from webkitscmpy import program, Contributor
    3030
    31 sys.exit(program.main(path=os.path.dirname(__file__)))
    3231
     32if '__main__' == __name__:
     33    contributors = Contributor.Mapping()
     34    for contributor in CommitterList().contributors():
     35        c = contributors.create(contributor.full_name, *contributor.emails)
     36        if not c:
     37            continue
     38        for alias in contributor.aliases or []:
     39            if alias not in contributors:
     40                contributors[alias] = c
     41        for nick in contributor.irc_nicknames or []:
     42            if nick not in contributors:
     43                contributors[nick] = c
     44
     45    sys.exit(program.main(path=os.path.dirname(__file__), contributors=contributors))
     46

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py

    r270412 r270447  
    4747    )
    4848
    49 version = Version(0, 4, 5)
     49version = Version(0, 5, 0)
    5050
    5151AutoInstall.register(Package('dateutil', Version(2, 8, 1), pypi_name='python-dateutil'))

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/commit.py

    r270365 r270447  
    188188        self.timestamp = timestamp
    189189
    190         if author and isinstance(author, six.string_types):
    191             self.author = Contributor.by_email.get(
    192                 author,
    193                 Contributor.by_name.get(author),
    194             )
    195             if not self.author:
    196                 raise ValueError("'{}' does not match a known contributor")
    197         elif author and isinstance(author, dict) and author.get('name'):
     190        if author and isinstance(author, dict) and author.get('name'):
    198191            self.author = Contributor(author.get('name'), author.get('emails'))
    199192        elif author and not isinstance(author, Contributor):

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/contributor.py

    r270365 r270447  
    2424import re
    2525
     26from collections import defaultdict
    2627from webkitcorepy import string_utils
    2728
    … …  
    3233    SVN_AUTHOR_RE = re.compile(r'r\d+ \| (?P<email>.*) \| (?P<date>.*) \| \d+ lines?')
    3334    SVN_PATCH_FROM_RE = re.compile(r'Patch by (?P<author>.*) <(?P<email>.*)> on \d+-\d+-\d+')
    34 
    35     by_email = dict()
    36     by_name = dict()
    3735
    3836    class Encoder(json.JSONEncoder):
    … …  
    4846            return result
    4947
    50     @classmethod
    51     def clear(cls):
    52         cls.by_email = dict()
    53         cls.by_name = dict()
     48    class Mapping(defaultdict):
     49        def __init__(self):
     50            super(Contributor.Mapping, self).__init__(lambda: None)
     51
     52        def add(self, contributor):
     53            if not isinstance(contributor, Contributor):
     54                raise ValueError("'{}' is not a Contributor object".format(type(contributor)))
     55            return self.create(name=contributor.name, *contributor.emails)
     56
     57        def create(self, name=None, *emails):
     58            emails = [email for email in emails or []]
     59            if not name and not emails:
     60                return None
     61
     62            contributor = None
     63            for argument in [name] + (emails or []):
     64                contributor = self[argument]
     65                if contributor:
     66                    break
     67
     68            if contributor:
     69                for email in emails or []:
     70                    if email not in contributor.emails:
     71                        contributor.emails.append(email)
     72                if contributor.name in contributor.emails and name:
     73                    contributor.name = name
     74            else:
     75                contributor = Contributor(name or emails[0], emails=emails)
     76
     77            self[contributor.name] = contributor
     78            for email in contributor.emails or []:
     79                self[email] = contributor
     80            return contributor
     81
    5482
    5583    @classmethod
    56     def from_scm_log(cls, line):
     84    def from_scm_log(cls, line, contributors=None):
    5785        email = None
    5886        author = None
    … …  
    76104            return None
    77105
    78         contributor = cls.by_name.get(author or email)
    79         if not contributor:
    80             contributor = cls.by_email.get(email)
    81 
    82         if not contributor:
    83             contributor = cls(author or email, emails=[email])
    84             cls.by_name[contributor.name] = contributor
    85         elif email not in contributor.emails:
    86             contributor.emails.append(email)
    87 
    88         cls.by_name[email] = contributor
    89         return contributor
     106        if contributors is not None:
     107            return contributors.create(author, email)
     108        return cls(author or email, emails=[email])
    90109
    91110    def __init__(self, name, emails=None):

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/git.py

    r270358 r270447  
    4343        return run([cls.executable(), 'rev-parse', '--show-toplevel'], cwd=path, capture_output=True).returncode == 0
    4444
    45     def __init__(self, path, dev_branches=None, prod_branches=None):
    46         super(Git, self).__init__(path, dev_branches=dev_branches, prod_branches=prod_branches)
     45    def __init__(self, path, dev_branches=None, prod_branches=None, contributors=None):
     46        super(Git, self).__init__(path, dev_branches=dev_branches, prod_branches=prod_branches, contributors=contributors)
    4747        if not self.root_path:
    4848            raise OSError('Provided path {} is not a git repository'.format(path))
    … …  
    266266            branch=branch,
    267267            timestamp=int(commit_time.stdout.lstrip()),
    268             author=Contributor.from_scm_log(log.stdout.splitlines()[1]),
     268            author=Contributor.from_scm_log(log.stdout.splitlines()[1], self.contributors),
    269269            message='\n'.join(line[4:] for line in log.stdout.splitlines()[4:]) if include_log else None,
    270270        )

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/scm.py

    r269891 r270447  
    4949
    5050    @classmethod
    51     def from_path(cls, path):
     51    def from_path(cls, path,  contributors=None):
    5252        from webkitscmpy import local
    5353
    5454        if local.Git.is_checkout(path):
    55             return local.Git(path)
     55            return local.Git(path, contributors=contributors)
    5656        if local.Svn.is_checkout(path):
    57             return local.Svn(path)
     57            return local.Svn(path, contributors=contributors)
    5858        raise OSError("'{}' is not a known SCM type".format(path))
    5959
    60     def __init__(self, path, dev_branches=None, prod_branches=None):
    61         super(Scm, self).__init__(dev_branches=dev_branches, prod_branches=prod_branches)
     60    def __init__(self, path, dev_branches=None, prod_branches=None, contributors=None):
     61        super(Scm, self).__init__(dev_branches=dev_branches, prod_branches=prod_branches, contributors=contributors)
    6262
    6363        if not isinstance(path, six.string_types):

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/svn.py

    r270358 r270447  
    5151        return run([cls.executable(), 'info'], cwd=path, capture_output=True).returncode == 0
    5252
    53     def __init__(self, path, dev_branches=None, prod_branches=None):
    54         super(Svn, self).__init__(path, dev_branches=dev_branches, prod_branches=prod_branches)
     53    def __init__(self, path, dev_branches=None, prod_branches=None, contributors=None):
     54        super(Svn, self).__init__(path, dev_branches=dev_branches, prod_branches=prod_branches, contributors=contributors)
    5555
    5656        self._root_path = self.path
    … …  
    371371                    break
    372372
    373             author = Contributor.from_scm_log(author_line)
     373            author = Contributor.from_scm_log(author_line, self.contributors)
    374374            message = '\n'.join(split_log[3:-1])
    375375        else:
    … …  
    377377                self.log('Failed to connect to remote, cannot compute commit message')
    378378            email = info.get('Last Changed Author')
    379             author = Contributor.by_email.get(
    380                 email,
    381                 Contributor.by_name.get(
    382                     email,
    383                     Contributor(name=email, emails=[email]),
    384                 ),
    385             ) if email else None
     379            author = self.contributors.create(email, email) if '@' in email else self.contributors.create(email)
    386380            message = None
    387381

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/program.py

    r270358 r270447  
    163163
    164164
    165 def main(args=None, path=None, loggers=None):
     165def main(args=None, path=None, loggers=None, contributors=None):
    166166    logging.basicConfig(level=logging.WARNING)
    167167
    … …  
    192192
    193193    if parsed.repository.startswith(('https://', 'http://')):
    194         repository = remote.Scm.from_url(parsed.repository)
     194        repository = remote.Scm.from_url(parsed.repository, contributors=contributors)
    195195    else:
    196         repository = local.Scm.from_path(path=parsed.repository)
     196        repository = local.Scm.from_path(path=parsed.repository, contributors=contributors)
    197197
    198198    return parsed.main(args=parsed, repository=repository)

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/scm.py

    r270038 r270447  
    2828class Scm(ScmBase):
    2929    @classmethod
    30     def from_url(cls, url):
     30    def from_url(cls, url, contributors=None):
    3131        from webkitscmpy import remote
    3232
    3333        if remote.Svn.is_webserver(url):
    34             return remote.Svn(url)
     34            return remote.Svn(url, contributors=contributors)
    3535        raise OSError("'{}' is not a known SCM server".format(url))
    3636
    37     def __init__(self, url, dev_branches=None, prod_branches=None):
    38         super(Scm, self).__init__(dev_branches=dev_branches, prod_branches=prod_branches)
     37    def __init__(self, url, dev_branches=None, prod_branches=None, contributors=None):
     38        super(Scm, self).__init__(dev_branches=dev_branches, prod_branches=prod_branches, contributors=contributors)
    3939
    4040        if not isinstance(url, six.string_types):

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/svn.py

    r270412 r270447  
    4747        return True if cls.URL_RE.match(url) else False
    4848
    49     def __init__(self, url, dev_branches=None, prod_branches=None):
     49    def __init__(self, url, dev_branches=None, prod_branches=None, contributors=None):
    5050        if url[-1] != '/':
    5151            url += '/'
    5252        if not self.is_webserver(url):
    5353            raise self.Exception("'{}' is not a valid SVN webserver".format(url))
    54         super(Svn, self).__init__(url, dev_branches=dev_branches, prod_branches=prod_branches)
     54        super(Svn, self).__init__(url, dev_branches=dev_branches, prod_branches=prod_branches, contributors=contributors)
    5555
    5656        if os.path.exists(self._cache_path):
    … …  
    427427            name = info.get('Last Changed Author')
    428428
    429         author = Contributor.by_email.get(
    430             name,
    431             Contributor.by_name.get(
    432                 name,
    433                 Contributor(name=name, emails=[name] if '@' in name else []),
    434             ),
    435         ) if name else None
     429        author = self.contributors.create(name, name) if name and '@' in name else self.contributors.create(name)
    436430
    437431        return Commit(

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/scm_base.py

    r270358 r270447  
    2727
    2828from logging import NullHandler
    29 from webkitscmpy import Commit, log
     29from webkitscmpy import Commit, Contributor, log
    3030
    3131
    … …  
    3939    PROD_BRANCHES = re.compile(r'\S+-[\d+\.]+-branch')
    4040
    41     def __init__(self, dev_branches=None, prod_branches=None):
     41    def __init__(self, dev_branches=None, prod_branches=None, contributors=None):
    4242        self.dev_branches = dev_branches or self.DEV_BRANCHES
    4343        self.prod_branches = prod_branches or self.PROD_BRANCHES
    4444        self.path = None
     45        self.contributors = Contributor.Mapping() if contributors is None else contributors
    4546
    4647    @property

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/test/commit_unittest.py

    r270365 r270447  
    196196
    197197    def test_contributor(self):
    198         Contributor.clear()
    199198        contributor = Contributor.from_scm_log('Author: Jonathan Bedard <jbedard@apple.com>')
    200199
    … …  
    202201        self.assertEqual(commit.author, contributor)
    203202
    204         commit = Commit(revision=1, identifier=1, author='Jonathan Bedard')
     203        commit = Commit(revision=1, identifier=1, author=Contributor.Encoder().default(contributor))
    205204        self.assertEqual(commit.author, contributor)
    206205
    207         commit = Commit(revision=1, identifier=1, author='jbedard@apple.com')
    208         self.assertEqual(commit.author, contributor)
    209 
    210206    def test_invalid_contributor(self):
    211         Contributor.clear()
    212         with self.assertRaises(ValueError):
     207        with self.assertRaises(TypeError):
    213208            Commit(revision=1, identifier=1, author='Jonathan Bedard')
    214209
    … …  
    219214
    220215    def test_json_encode(self):
    221         Contributor.clear()
    222216        contributor = Contributor.from_scm_log('Author: Jonathan Bedard <jbedard@apple.com>')
    223217
    … …  
    244238
    245239    def test_json_decode(self):
    246         Contributor.clear()
    247240        contributor = Contributor.from_scm_log('Author: Jonathan Bedard <jbedard@apple.com>')
    248241
    … …  
    252245            identifier='1@main',
    253246            timestamp=1000,
    254             author=contributor,
     247            author=Contributor.Encoder().default(contributor),
    255248            message='Message'
    256249        )

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/test/contributor_unittest.py

    r270365 r270447  
    3030class TestContributor(unittest.TestCase):
    3131    def test_git_log(self):
    32         Contributor.clear()
    3332        contributor = Contributor.from_scm_log('Author: Jonathan Bedard <jbedard@apple.com>')
    3433
    … …  
    3736
    3837    def test_git_svn_log(self):
    39         Contributor.clear()
    4038        contributor = Contributor.from_scm_log('Author: Jonathan Bedard <jbedard@apple.com@268f45cc-cd09-0410-ab3c-d52691b4dbfc>')
    4139
    … …  
    4442
    4543    def test_git_no_author(self):
    46         Contributor.clear()
    4744        contributor = Contributor.from_scm_log('Author: Automated Checkin <devnull>')
    4845        self.assertIsNone(contributor)
    4946
    5047    def test_git_svn_no_author(self):
    51         Contributor.clear()
    5248        contributor = Contributor.from_scm_log('Author: (no author) <(no author)@268f45cc-cd09-0410-ab3c-d52691b4dbfc>')
    5349        self.assertIsNone(contributor)
    5450
    5551    def test_svn_log(self):
    56         Contributor.clear()
    5752        contributor = Contributor.from_scm_log('r266751 | jbedard@apple.com | 2020-09-08 14:33:42 -0700 (Tue, 08 Sep 2020) | 10 lines')
    5853
    … …  
    6156
    6257    def test_short_svn_log(self):
    63         Contributor.clear()
    6458        contributor = Contributor.from_scm_log('r266751 | jbedard@apple.com | 2020-09-08 14:33:42 -0700 (Tue, 08 Sep 2020) | 1 line')
    6559
    … …  
    6862
    6963    def test_svn_patch_by_log(self):
    70         Contributor.clear()
    7164        contributor = Contributor.from_scm_log('Patch by Jonathan Bedard <jbedard@apple.com> on 2020-09-10')
    7265
    … …  
    7568
    7669    def test_author_mapping(self):
    77         Contributor.clear()
    78         Contributor.from_scm_log('Author: Jonathan Bedard <jbedard@apple.com>')
    79         contributor = Contributor.from_scm_log('Author: Jonathan Bedard <jbedard@webkit.org>')
     70        contributors = Contributor.Mapping()
     71        Contributor.from_scm_log('Author: Jonathan Bedard <jbedard@apple.com>', contributors)
     72        contributor = Contributor.from_scm_log('Author: Jonathan Bedard <jbedard@webkit.org>', contributors)
    8073
    8174        self.assertEqual(contributor.name, 'Jonathan Bedard')
    … …  
    8376
    8477    def test_email_mapping(self):
    85         Contributor.clear()
    86         Contributor.from_scm_log('Author: Jonathan Bedard <jbedard@apple.com>')
    87         contributor = Contributor.from_scm_log('r266751 | jbedard@apple.com | 2020-09-08 14:33:42 -0700 (Tue, 08 Sep 2020) | 10 lines')
     78        contributors = Contributor.Mapping()
     79        Contributor.from_scm_log('Author: Jonathan Bedard <jbedard@apple.com>', contributors)
     80        contributor = Contributor.from_scm_log('r266751 | jbedard@apple.com | 2020-09-08 14:33:42 -0700 (Tue, 08 Sep 2020) | 10 lines', contributors)
    8881
    8982        self.assertEqual(contributor.name, 'Jonathan Bedard')
    … …  
    9184
    9285    def test_invalid_log(self):
    93         Contributor.clear()
    9486        with self.assertRaises(ValueError):
    9587            Contributor.from_scm_log('Jonathan Bedard <jbedard@apple.com>')
Note: See TracChangeset for help on using the changeset viewer.