Changeset 277102 in webkit


Ignore:
Timestamp:
May 6, 2021 11:32:40 AM (3 years ago)
Author:
Jonathan Bedard
Message:

[webkitcorepy] Add API to efficiently create a sequence of commits
​https://bugs.webkit.org/show_bug.cgi?id=224890
<rdar://problem/76975733>

Rubber-stamped by Aakash Jain.

While it is possible to simple iterate through a range of commits to define them,
every API we use to define commits has much more efficient techniques.

  • Scripts/libraries/webkitscmpy/setup.py: Bump version.
  • Scripts/libraries/webkitscmpy/webkitscmpy/init.py: Ditto.
  • Scripts/libraries/webkitscmpy/webkitscmpy/contributor.py:

(Contributor): Add revision to SVN_AUTHOR_RE and add regex without lines.
(Contributor.from_scm_log): Strip leading whitespace from author.

  • Scripts/libraries/webkitscmpy/webkitscmpy/local/git.py:

(Git._args_from_content):
(Git.commits): Use git log to efficiently compute a range of commits.

  • Scripts/libraries/webkitscmpy/webkitscmpy/local/svn.py:

(Svn._args_from_content):
(Svn.commits): Use svn log to efficiently compute a range of commits.

  • Scripts/libraries/webkitscmpy/webkitscmpy/mocks/local/git.py:

(Git.init): Add git log mock.

  • Scripts/libraries/webkitscmpy/webkitscmpy/mocks/local/svn.py:

(Svn.init): Add svn log mock and more explicit svn info mock.
(Svn._log_range):

  • Scripts/libraries/webkitscmpy/webkitscmpy/mocks/remote/git_hub.py:

(GitHub._commits_response): Return all parent commits to provided ref.
(GitHub.request):

  • Scripts/libraries/webkitscmpy/webkitscmpy/mocks/remote/svn.py:

(Svn.range): More efficiently compute the range.
(Svn.request):

  • Scripts/libraries/webkitscmpy/webkitscmpy/remote/git_hub.py:

(GitHub.request): Allow caller to disable pagination.
(GitHub.commit): Reduce number of requests required to compute order.
(GitHub.commits): Using the commits endpoint, more efficiently
compute a range of commits.

  • Scripts/libraries/webkitscmpy/webkitscmpy/remote/svn.py:

(Svn): Generalize HISTORY_RE to match any single-line SVN XML response.
(Svn._cache_revisions): Replace HISTORY_RE with DATA_RE.
(Svn.commits): Use svn/rvr to efficiently compute a range of commits.

  • Scripts/libraries/webkitscmpy/webkitscmpy/scm_base.py:

(ScmBase._commit_range): Return a pair of commits representing the range
the caller is requesting, and preform some basic sanity checks.
(ScmBase.commits): Declare function implemented by decedents.

  • Scripts/libraries/webkitscmpy/webkitscmpy/test/find_unittest.py:
  • Scripts/libraries/webkitscmpy/webkitscmpy/test/git_unittest.py:

(TestGit.test_commits):
(TestGit.test_commits_branch):
(TestGitHub.test_commits):
(TestGitHub.test_commits_branch):

  • Scripts/libraries/webkitscmpy/webkitscmpy/test/svn_unittest.py:

(TestLocalSvn.test_commits):
(TestLocalSvn.test_commits_branch):
(TestRemoteSvn.test_commits):
(TestRemoteSvn.test_commits_branch):

Location:
trunk/Tools
Files:
16 edited

Legend:

Unmodified
Added
Removed

  • trunk/Tools/ChangeLog

    r277101 r277102  
     12021-05-06  Jonathan Bedard  <jbedard@apple.com>
     2
     3        [webkitcorepy] Add API to efficiently create a sequence of commits
     4        https://bugs.webkit.org/show_bug.cgi?id=224890
     5        <rdar://problem/76975733>
     6
     7        Rubber-stamped by Aakash Jain.
     8
     9        While it is possible to simple iterate through a range of commits to define them,
     10        every API we use to define commits has much more efficient techniques.
     11
     12        * Scripts/libraries/webkitscmpy/setup.py: Bump version.
     13        * Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py: Ditto.
     14        * Scripts/libraries/webkitscmpy/webkitscmpy/contributor.py:
     15        (Contributor): Add revision to SVN_AUTHOR_RE and add regex without lines.
     16        (Contributor.from_scm_log): Strip leading whitespace from author.
     17        * Scripts/libraries/webkitscmpy/webkitscmpy/local/git.py:
     18        (Git._args_from_content):
     19        (Git.commits): Use `git log` to efficiently compute a range of commits.
     20        * Scripts/libraries/webkitscmpy/webkitscmpy/local/svn.py:
     21        (Svn._args_from_content):
     22        (Svn.commits): Use `svn log` to efficiently compute a range of commits.
     23        * Scripts/libraries/webkitscmpy/webkitscmpy/mocks/local/git.py:
     24        (Git.__init__): Add `git log` mock.
     25        * Scripts/libraries/webkitscmpy/webkitscmpy/mocks/local/svn.py:
     26        (Svn.__init__): Add `svn log` mock and more explicit `svn info` mock.
     27        (Svn._log_range):
     28        * Scripts/libraries/webkitscmpy/webkitscmpy/mocks/remote/git_hub.py:
     29        (GitHub._commits_response): Return all parent commits to provided ref.
     30        (GitHub.request):
     31        * Scripts/libraries/webkitscmpy/webkitscmpy/mocks/remote/svn.py:
     32        (Svn.range): More efficiently compute the range.
     33        (Svn.request):
     34        * Scripts/libraries/webkitscmpy/webkitscmpy/remote/git_hub.py:
     35        (GitHub.request): Allow caller to disable pagination.
     36        (GitHub.commit): Reduce number of requests required to compute order.
     37        (GitHub.commits): Using the `commits` endpoint, more efficiently
     38        compute a range of commits.
     39        * Scripts/libraries/webkitscmpy/webkitscmpy/remote/svn.py:
     40        (Svn): Generalize HISTORY_RE to match any single-line SVN XML response.
     41        (Svn._cache_revisions): Replace HISTORY_RE with DATA_RE.
     42        (Svn.commits): Use svn/rvr to efficiently compute a range of commits.
     43        * Scripts/libraries/webkitscmpy/webkitscmpy/scm_base.py:
     44        (ScmBase._commit_range): Return a pair of commits representing the range
     45        the caller is requesting, and preform some basic sanity checks.
     46        (ScmBase.commits): Declare function implemented by decedents.
     47        * Scripts/libraries/webkitscmpy/webkitscmpy/test/find_unittest.py:
     48        * Scripts/libraries/webkitscmpy/webkitscmpy/test/git_unittest.py:
     49        (TestGit.test_commits):
     50        (TestGit.test_commits_branch):
     51        (TestGitHub.test_commits):
     52        (TestGitHub.test_commits_branch):
     53        * Scripts/libraries/webkitscmpy/webkitscmpy/test/svn_unittest.py:
     54        (TestLocalSvn.test_commits):
     55        (TestLocalSvn.test_commits_branch):
     56        (TestRemoteSvn.test_commits):
     57        (TestRemoteSvn.test_commits_branch):
     58
    1592021-05-06  Chris Dumez  <cdumez@apple.com>
    260

  • trunk/Tools/Scripts/libraries/webkitscmpy/setup.py

    r275635 r277102  
    3030setup(
    3131    name='webkitscmpy',
    32     version='0.13.9',
     32    version='0.14.0',
    3333    description='Library designed to interact with git and svn repositories.',
    3434    long_description=readme(),

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py

    r275635 r277102  
    4747    )
    4848
    49 version = Version(0, 13, 9)
     49version = Version(0, 14, 0)
    5050
    5151AutoInstall.register(Package('fasteners', Version(0, 15, 0)))

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/contributor.py

    r275352 r277102  
    3333    UNKNOWN_AUTHOR = re.compile(r'Author: (?P<author>.*) <None>')
    3434    EMPTY_AUTHOR = re.compile(r'Author: (?P<author>.*) <>')
    35     SVN_AUTHOR_RE = re.compile(r'r\d+ \| (?P<email>.*) \| (?P<date>.*) \| \d+ lines?')
     35    SVN_AUTHOR_RE = re.compile(r'r(?P<revision>\d+) \| (?P<email>.*) \| (?P<date>.*) \| \d+ lines?')
     36    SVN_AUTHOR_Q_RE = re.compile(r'r(?P<revision>\d+) \| (?P<email>.*) \| (?P<date>.*)')
    3637    SVN_PATCH_FROM_RE = re.compile(r'Patch by (?P<author>.*) <(?P<email>.*)> on \d+-\d+-\d+')
    3738
    … …  
    116117        author = None
    117118
    118         for expression in [cls.GIT_AUTHOR_RE, cls.SVN_AUTHOR_RE, cls.SVN_PATCH_FROM_RE, cls.AUTOMATED_CHECKIN_RE, cls.UNKNOWN_AUTHOR, cls.EMPTY_AUTHOR]:
     119        for expression in [
     120            cls.GIT_AUTHOR_RE,
     121            cls.SVN_AUTHOR_RE,
     122            cls.SVN_PATCH_FROM_RE,
     123            cls.AUTOMATED_CHECKIN_RE,
     124            cls.UNKNOWN_AUTHOR,
     125            cls.EMPTY_AUTHOR,
     126            cls.SVN_AUTHOR_Q_RE,
     127        ]:
    119128            match = expression.match(line)
    120129            if match:
    121130                if 'author' in expression.groupindex:
    122                     author = match.group('author')
     131                    author = match.group('author').lstrip()
    123132                    if '(no author)' in author or 'Automated Checkin' in author or 'Unknown' in author:
    124133                        author = None

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/git.py

    r276856 r277102  
    2121# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
    2222
     23import calendar
    2324import logging
    2425import os
    2526import re
    2627import six
    27 
    28 from webkitcorepy import run, decorators, TimeoutExpired
     28import subprocess
     29import sys
     30import time
     31
     32from datetime import datetime, timedelta
     33
     34from webkitcorepy import run, decorators
    2935from webkitscmpy.local import Scm
    3036from webkitscmpy import Commit, Contributor, log
    … …  
    301307        )
    302308
     309    def _args_from_content(self, content, include_log=True):
     310        author = None
     311        timestamp = None
     312
     313        for line in content.splitlines()[:4]:
     314            split = line.split(': ')
     315            if split[0] == 'Author':
     316                author = Contributor.from_scm_log(line.lstrip(), self.contributors)
     317            elif split[0] == 'CommitDate':
     318                tz_diff = line.split(' ')[-1]
     319                date = datetime.strptime(split[1].lstrip()[:-len(tz_diff)], '%a %b %d %H:%M:%S %Y ')
     320                date += timedelta(
     321                    hours=int(tz_diff[1:3]),
     322                    minutes=int(tz_diff[3:5]),
     323                ) * (1 if tz_diff[0] == '-' else -1)
     324                timestamp = int(calendar.timegm(date.timetuple())) - time.timezone
     325
     326        message = ''
     327        for line in content.splitlines()[5:]:
     328            message += line[4:] + '\n'
     329        matches = self.GIT_SVN_REVISION.findall(message)
     330
     331        return dict(
     332            revision=int(matches[-1].split('@')[0]) if matches else None,
     333            author=author,
     334            timestamp=timestamp,
     335            message=message.rstrip() if include_log else None,
     336        )
     337
     338    def commits(self, begin=None, end=None, include_log=True, include_identifier=True):
     339        begin, end = self._commit_range(begin=begin, end=end, include_identifier=include_identifier)
     340
     341        try:
     342            log = None
     343            log = subprocess.Popen(
     344                [self.executable(), 'log', '--format=fuller', '{}...{}'.format(end.hash, begin.hash)],
     345                cwd=self.root_path,
     346                stdout=subprocess.PIPE,
     347                stderr=subprocess.PIPE,
     348                **(dict(encoding='utf-8') if sys.version_info > (3, 0) else dict())
     349            )
     350            if log.poll():
     351                raise self.Exception("Failed to construct history for '{}'".format(end.branch))
     352
     353            line = log.stdout.readline()
     354            previous = [end]
     355            while line:
     356                if not line.startswith('commit '):
     357                    raise OSError('Failed to parse `git log` format')
     358                branch_point = previous[0].branch_point
     359                identifier = previous[0].identifier
     360                hash = line.split(' ')[-1].rstrip()
     361                if hash != previous[0].hash:
     362                    identifier -= 1
     363
     364                if not identifier:
     365                    identifier = branch_point
     366                    branch_point = None
     367
     368                content = ''
     369                line = log.stdout.readline()
     370                while line and not line.startswith('commit '):
     371                    content += line
     372                    line = log.stdout.readline()
     373
     374                commit = Commit(
     375                    repository_id=self.id,
     376                    hash=hash,
     377                    branch=end.branch if identifier and branch_point else self.default_branch,
     378                    identifier=identifier if include_identifier else None,
     379                    branch_point=branch_point if include_identifier else None,
     380                    order=0,
     381                    **self._args_from_content(content, include_log=include_log)
     382                )
     383
     384                # Ensure that we don't duplicate the first and last commits
     385                if commit.hash == previous[0].hash:
     386                    previous[0] = commit
     387
     388                # If we share a timestamp with the previous commit, that means that this commit has an order
     389                # less than the set of commits cached in previous
     390                elif commit.timestamp == previous[0].timestamp:
     391                    for cached in previous:
     392                        cached.order += 1
     393                    previous.append(commit)
     394
     395                # If we don't share a timestamp with the previous set of commits, we should return all commits
     396                # cached in previous.
     397                else:
     398                    for cached in previous:
     399                        yield cached
     400                    previous = [commit]
     401
     402            for cached in previous:
     403                cached.order += begin.order
     404                yield cached
     405        finally:
     406            if log:
     407                log.kill()
     408
    303409    def find(self, argument, include_log=True, include_identifier=True):
    304410        if not isinstance(argument, six.string_types):

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/local/svn.py

    r276856 r277102  
    402402        )
    403403
     404    def _args_from_content(self, content, include_log=True):
     405        leading = content.splitlines()[0]
     406        match = Contributor.SVN_AUTHOR_RE.match(leading) or Contributor.SVN_AUTHOR_Q_RE.match(leading)
     407        if not match:
     408            return {}
     409
     410        tz_diff = match.group('date').split(' ', 2)[-1]
     411        date = datetime.strptime(match.group('date')[:-len(tz_diff)], '%Y-%m-%d %H:%M:%S ')
     412        date += timedelta(
     413            hours=int(tz_diff[1:3]),
     414            minutes=int(tz_diff[3:5]),
     415        ) * (1 if tz_diff[0] == '-' else -1)
     416
     417        return dict(
     418            revision=int(match.group('revision')),
     419            timestamp=int(calendar.timegm(date.timetuple())),
     420            author=Contributor.from_scm_log(leading, self.contributors),
     421            message='\n'.join(content.splitlines()[2:]).rstrip() if include_log else None,
     422        )
     423
     424
     425    def commits(self, begin=None, end=None, include_log=True, include_identifier=True):
     426        begin, end = self._commit_range(begin=begin, end=end, include_identifier=include_identifier)
     427        previous = end
     428        if end.branch == self.default_branch or '/' in end.branch:
     429            branch_arg = '^/{}'.format(end.branch)
     430        else:
     431            branch_arg = '^/branches/{}'.format(end.branch)
     432
     433        try:
     434            log = None
     435            log = subprocess.Popen(
     436                [self.executable(), 'log', '-r', '{}:{}'.format(
     437                    end.revision, begin.revision,
     438                ), branch_arg] + ([] if include_log else ['-q']),
     439                cwd=self.root_path,
     440                stdout=subprocess.PIPE,
     441                stderr=subprocess.PIPE,
     442                **(dict(encoding='utf-8') if sys.version_info > (3, 0) else dict())
     443            )
     444            if log.poll():
     445                raise self.Exception('Failed to find commits between {} and {} on {}'.format(begin, end, branch_arg))
     446
     447            content = ''
     448            line = log.stdout.readline()
     449            divider = '-' * 72
     450            while True:
     451                if line and line.rstrip() != divider:
     452                    content += line
     453                    line = log.stdout.readline()
     454                    continue
     455
     456                if not content:
     457                    line = log.stdout.readline()
     458                    continue
     459
     460                branch_point = previous.branch_point if include_identifier else None
     461                identifier = previous.identifier if include_identifier else None
     462
     463                args = self._args_from_content(content, include_log=include_log)
     464                if args['revision'] != previous.revision:
     465                    yield previous
     466                    identifier -= 1
     467                if not identifier:
     468                    identifier = branch_point
     469                    branch_point = None
     470
     471                previous = Commit(
     472                    repository_id=self.id,
     473                    branch=end.branch if branch_point else self.default_branch,
     474                    identifier=identifier,
     475                    branch_point=branch_point,
     476                    **args
     477                )
     478                content = ''
     479                if not line:
     480                    break
     481                line = log.stdout.readline()
     482
     483            yield previous
     484
     485        finally:
     486            if log:
     487                log.kill()
     488
    404489    def checkout(self, argument):
    405490        commit = self.find(argument)

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/mocks/local/git.py

    r275353 r277102  
    2424import os
    2525import re
     26import time
    2627
    2728from datetime import datetime
    … …  
    229230                            author=self.find(args[2]).author.name,
    230231                            email=self.find(args[2]).author.email,
    231                             date=datetime.fromtimestamp(self.find(args[2]).timestamp).strftime('%a %b %d %H:%M:%S %Y'),
     232                            date=datetime.utcfromtimestamp(self.find(args[2]).timestamp + time.timezone).strftime('%a %b %d %H:%M:%S %Y +0000'),
    232233                            log='\n'.join([
    233234                                    ('    ' + line) if line else '' for line in self.find(args[2]).message.splitlines()
    … …  
    240241                        ),
    241242                ) if self.find(args[2]) else mocks.ProcessCompletion(returncode=128),
     243            ), mocks.Subprocess.Route(
     244                self.executable, 'log', '--format=fuller', re.compile(r'.+\.\.\..+'),
     245                cwd=self.path,
     246                generator=lambda *args, **kwargs: mocks.ProcessCompletion(
     247                    returncode=0,
     248                    stdout='\n'.join([
     249                        'commit {hash}\n'
     250                        'Author:     {author} <{email}>\n'
     251                        'AuthorDate: {date}\n'
     252                        'Commit:     {author} <{email}>\n'
     253                        'CommitDate: {date}\n'
     254                        '\n{log}'.format(
     255                            hash=commit.hash,
     256                            author=commit.author.name,
     257                            email=commit.author.email,
     258                            date=datetime.utcfromtimestamp(commit.timestamp + time.timezone).strftime('%a %b %d %H:%M:%S %Y +0000'),
     259                            log='\n'.join([
     260                                ('    ' + line) if line else '' for line in commit.message.splitlines()
     261                            ] + ([
     262                                '    git-svn-id: https://svn.{}/repository/{}/trunk@{} 268f45cc-cd09-0410-ab3c-d52691b4dbfc'.format(
     263                                    self.remote.split('@')[-1].split(':')[0],
     264                                    os.path.basename(path),
     265                                   commit.revision,
     266                            )] if git_svn else []),
     267                        )) for commit in self.commits_in_range(args[3].split('...')[-1], args[3].split('...')[0])
     268                    ])
     269                )
    242270            ), mocks.Subprocess.Route(
    243271                self.executable, 'rev-list', '--count', '--no-merges', re.compile(r'.+'),
    … …  
    464492            stdout=stdout.getvalue(),
    465493        )
     494
     495    def commits_in_range(self, begin, end):
     496        branches = [self.default_branch]
     497        for branch, commits in self.commits.items():
     498            if branch == self.default_branch:
     499                continue
     500            for commit in commits:
     501                if commit.hash == end:
     502                    branches.insert(0, branch)
     503                    break
     504            if len(branches) > 1:
     505                break
     506
     507        in_range = False
     508        previous = None
     509        for branch in branches:
     510            for commit in reversed(self.commits[branch]):
     511                if commit.hash == end:
     512                    in_range = True
     513                if in_range and (not previous or commit.hash != previous.hash):
     514                    yield commit
     515                previous = commit
     516                if commit.hash == begin:
     517                    in_range = False
     518            in_range = False
     519            if not previous or branch == self.default_branch:
     520                continue
     521
     522            for commit in reversed(self.commits[self.default_branch]):
     523                if previous.branch_point == commit.identifier:
     524                    end = commit.hash

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/mocks/local/svn.py

    r272172 r277102  
    6666        super(Svn, self).__init__(
    6767            mocks.Subprocess.Route(
     68                self.executable, 'info', self.BRANCH_RE,
     69                cwd=self.path,
     70                generator=lambda *args, **kwargs: self._info(branch=self.BRANCH_RE.match(args[2]).group('branch'), cwd=kwargs.get('cwd', ''))
     71            ), mocks.Subprocess.Route(
     72                self.executable, 'info', '-r', re.compile(r'\d+'),
     73                cwd=self.path,
     74                generator=lambda *args, **kwargs: self._info(revision=int(args[3]), cwd=kwargs.get('cwd', ''))
     75            ), mocks.Subprocess.Route(
    6876                self.executable, 'info',
    6977                cwd=self.path,
    7078                generator=lambda *args, **kwargs: self._info(cwd=kwargs.get('cwd', ''))
    71             ), mocks.Subprocess.Route(
    72                 self.executable, 'info', self.BRANCH_RE,
    73                 cwd=self.path,
    74                 generator=lambda *args, **kwargs: self._info(branch=self.BRANCH_RE.match(args[2]).group('branch'), cwd=kwargs.get('cwd', ''))
    7579            ), mocks.Subprocess.Route(
    7680                self.executable, 'list', '^/branches',
    … …  
    118122                    branch=self.BRANCH_RE.match(args[6]).group('branch'),
    119123                    revision=args[5],
     124                ) if self.connected else mocks.ProcessCompletion(returncode=1)
     125            ), mocks.Subprocess.Route(
     126                self.executable, 'log', '-r', re.compile(r'\d+:\d+'), self.BRANCH_RE,
     127                cwd=self.path,
     128                generator=lambda *args, **kwargs: self._log_range(
     129                    branch=self.BRANCH_RE.match(args[4]).group('branch'),
     130                    end=int(args[3].split(':')[0]),
     131                    begin=int(args[3].split(':')[-1]),
    120132                ) if self.connected else mocks.ProcessCompletion(returncode=1)
    121133            ), mocks.Subprocess.Route(
    … …  
    205217        )
    206218
     219    def _log_range(self, branch=None, end=None, begin=None):
     220        if end < begin:
     221            return mocks.ProcessCompletion(returncode=1)
     222
     223        output = ''
     224        previous = None
     225        for b in [branch, 'trunk']:
     226            for candidate in reversed(self.commits.get(b, [])):
     227                if candidate.revision > end or candidate.revision < begin:
     228                    continue
     229                if previous and previous.revision <= candidate.revision:
     230                    continue
     231                previous = candidate
     232                output += ('------------------------------------------------------------------------\n'
     233                    '{line} | {lines} lines\n\n'
     234                    '{log}\n').format(
     235                        line=self.log_line(candidate),
     236                        lines=len(candidate.message.splitlines()),
     237                        log=candidate.message
     238                )
     239        return mocks.ProcessCompletion(returncode=0, stdout=output)
     240
    207241    def find(self, branch=None, revision=None):
    208242        if not branch and not revision:

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/mocks/remote/git_hub.py

    r275195 r277102  
    133133        ], url=url)
    134134
     135    def _commits_response(self, url, ref):
     136        from datetime import datetime, timedelta
     137
     138        base = self.commit(ref)
     139        if not base:
     140            return mocks.Response(
     141                status_code=404,
     142                url=url,
     143                text=json.dumps(dict(message='No commit found for SHA: {}'.format(ref))),
     144            )
     145
     146        response = []
     147        for branch in [self.default_branch] if base.branch == self.default_branch else [base.branch, self.default_branch]:
     148            in_range = False
     149            previous = None
     150            for commit in reversed(self.commits[branch]):
     151                if commit.hash == ref:
     152                    in_range = True
     153                if not in_range:
     154                    continue
     155                previous = commit
     156                response.append({
     157                    'sha': commit.hash,
     158                    'commit': {
     159                        'author': {
     160                            'name': commit.author.name,
     161                            'email': commit.author.email,
     162                            'date': datetime.utcfromtimestamp(commit.timestamp - timedelta(hours=7).seconds).strftime('%Y-%m-%dT%H:%M:%SZ'),
     163                        }, 'committer': {
     164                            'name': commit.author.name,
     165                            'email': commit.author.email,
     166                            'date': datetime.utcfromtimestamp(commit.timestamp - timedelta(hours=7).seconds).strftime('%Y-%m-%dT%H:%M:%SZ'),
     167                        }, 'message': commit.message + ('\ngit-svn-id: https://svn.example.org/repository/webkit/{}@{} 268f45cc-cd09-0410-ab3c-d52691b4dbfc\n'.format(
     168                            'trunk' if commit.branch == self.default_branch else commit.branch, commit.revision,
     169                        ) if commit.revision else ''),
     170                        'url': 'https://{}/git/commits/{}'.format(self.api_remote, commit.hash),
     171                    }, 'url': 'https://{}/commits/{}'.format(self.api_remote, commit.hash),
     172                    'html_url': 'https://{}/commit/{}'.format(self.remote, commit.hash),
     173                })
     174            if branch != self.default_branch:
     175                for commit in reversed(self.commits[self.default_branch]):
     176                    if previous.branch_point == commit.identifier:
     177                        ref = commit.hash
     178
     179        return mocks.Response.fromJson(response, url=url)
     180
    135181    def _commit_response(self, url, ref):
    136182        from datetime import datetime, timedelta
    … …  
    154200                    'email': commit.author.email,
    155201                    'date': datetime.utcfromtimestamp(commit.timestamp - timedelta(hours=7).seconds).strftime('%Y-%m-%dT%H:%M:%SZ'),
    156                 }, 'message': commit.message + '\ngit-svn-id: https://svn.example.org/repository/webkit/{}@{} 268f45cc-cd09-0410-ab3c-d52691b4dbfc\n'.format(
     202                }, 'message': commit.message + ('\ngit-svn-id: https://svn.example.org/repository/webkit/{}@{} 268f45cc-cd09-0410-ab3c-d52691b4dbfc\n'.format(
    157203                    'trunk' if commit.branch == self.default_branch else commit.branch, commit.revision,
    158                 ) if commit.revision else '',
     204                ) if commit.revision else ''),
    159205                'url': 'https://{}/git/commits/{}'.format(self.api_remote, commit.hash),
    160206            }, 'url': 'https://{}/commits/{}'.format(self.api_remote, commit.hash),
    … …  
    234280        )
    235281
    236     def request(self, method, url, data=None, **kwargs):
     282    def request(self, method, url, data=None, params=None, **kwargs):
    237283        if not url.startswith('http://') and not url.startswith('https://'):
    238284            return mocks.Response.create404(url)
    239285
     286        params = params or {}
    240287        stripped_url = url.split('://')[-1]
    241288
    … …  
    247294        if stripped_url in ['{}/branches'.format(self.api_remote), '{}/tags'.format(self.api_remote)]:
    248295            return self._list_refs_response(url=url, type=stripped_url.split('/')[-1])
     296
     297        # Return a commit and it's parents
     298        if stripped_url == '{}/commits'.format(self.api_remote) and params.get('sha'):
     299            return self._commits_response(url=url, ref=params['sha'])
    249300
    250301        # Extract single commit

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/mocks/remote/svn.py

    r274366 r277102  
    107107        if category and category.startswith('branches/'):
    108108            category = category.split('/')[-1]
    109 
    110         if not category:
    111             for commits in self.commits.values():
    112                 for commit in commits:
    113                     if commit.revision == start:
    114                         category = commit.branch
    115                         break
    116 
    117         if not category:
    118             return []
    119 
    120         result = [commit for commit in reversed(self.commits[category])]
    121         if self.commits[category][0].branch_point:
    122             result += [commit for commit in reversed(self.commits['trunk'][:self.commits[category][0].branch_point])]
    123 
    124         for index in reversed(range(len(result))):
    125             if result[index].revision < end:
    126                 result = result[:index]
    127                 continue
    128             if result[index].revision > start:
    129                 result = result[index:]
    130                 break
    131 
    132         return result
     109        category = [category] if category else self.branches(start) + self.tags(start)
     110
     111        previous = None
     112        for b in category + ['trunk']:
     113            for candidate in reversed(self.commits.get(b, [])):
     114                if candidate.revision > start or candidate.revision < end:
     115                    continue
     116                if previous and previous.revision <= candidate.revision:
     117                    continue
     118                previous = candidate
     119                yield candidate
    133120
    134121    def request(self, method, url, data=None, **kwargs):
    … …  
    267254        # Log for commit
    268255        if method == 'REPORT' and stripped_url.startswith('{}!'.format(self.remote)) and match and data.get('S:log-report'):
    269             commits = self.range(
     256            commits = list(self.range(
    270257                category=match.group('category'),
    271258                start=int(data['S:log-report']['S:start-revision']),
    272259                end=int(data['S:log-report']['S:end-revision']),
    273             )
     260            ))
    274261
    275262            limit = int(data['S:log-report'].get('S:limit', 0))

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/git_hub.py

    r276856 r277102  
    7676        return True
    7777
    78     def request(self, path=None, params=None, headers=None, authenticated=None):
     78    def request(self, path=None, params=None, headers=None, authenticated=None, paginate=True):
    7979        headers = {key: value for key, value in headers.items()} if headers else dict()
    8080        headers['Accept'] = headers.get('Accept', 'application/vnd.github.v3+json')
    … …  
    106106        result = response.json()
    107107
    108         while isinstance(response.json(), list) and len(response.json()) == params['per_page']:
     108        while paginate and isinstance(response.json(), list) and len(response.json()) == params['per_page']:
    109109            params['page'] += 1
    110110            response = requests.get(url, params=params, headers=headers, auth=auth)
    … …  
    288288        # zero-indexed "order" within it's timestamp.
    289289        order = 0
    290         while not identifier or order + 1 < identifier + (branch_point or 0):
    291             response = self.request('commits/{}'.format('{}~{}'.format(commit_data['sha'], order + 1)))
    292             if not response:
     290        lhash = commit_data['sha']
     291        while lhash:
     292            response = self.request('commits', paginate=False, params=dict(sha=lhash, per_page=20))
     293            if len(response) <= 1:
    293294                break
    294             parent_timestamp = int(calendar.timegm(datetime.strptime(
    295                 response['commit']['committer']['date'], '%Y-%m-%dT%H:%M:%SZ',
    296             ).timetuple()))
    297             if parent_timestamp != timestamp:
    298                 break
    299             order += 1
     295            for c in response:
     296                if lhash == c['sha']:
     297                    continue
     298                parent_timestamp = int(calendar.timegm(datetime.strptime(
     299                    c['commit']['committer']['date'], '%Y-%m-%dT%H:%M:%SZ',
     300                ).timetuple()))
     301                if parent_timestamp != timestamp:
     302                    lhash = None
     303                    break
     304                lhash = c['sha']
     305                order += 1
    300306
    301307        return Commit(
    … …  
    314320        )
    315321
     322    def commits(self, begin=None, end=None, include_log=True, include_identifier=True):
     323        begin, end = self._commit_range(begin=begin, end=end, include_identifier=include_identifier)
     324
     325        previous = end
     326        cached = [previous]
     327        while previous:
     328            response = self.request('commits', paginate=False, params=dict(sha=previous.hash))
     329            if not response:
     330                break
     331            for commit_data in response:
     332                branch_point = previous.branch_point
     333                identifier = previous.identifier
     334                if commit_data['sha'] == previous.hash:
     335                    cached = cached[:-1]
     336                else:
     337                    identifier -= 1
     338
     339                if not identifier:
     340                    identifier = branch_point
     341                    branch_point = None
     342
     343                matches = self.GIT_SVN_REVISION.findall(commit_data['commit']['message'])
     344                revision = int(matches[-1].split('@')[0]) if matches else None
     345
     346                email_match = self.EMAIL_RE.match(commit_data['commit']['author']['email'])
     347                timestamp = int(calendar.timegm(datetime.strptime(
     348                    commit_data['commit']['committer']['date'], '%Y-%m-%dT%H:%M:%SZ',
     349                ).timetuple()))
     350
     351                previous = Commit(
     352                    repository_id=self.id,
     353                    hash=commit_data['sha'],
     354                    revision=revision,
     355                    branch=end.branch if identifier and branch_point else self.default_branch,
     356                    identifier=identifier if include_identifier else None,
     357                    branch_point=branch_point if include_identifier else None,
     358                    timestamp=timestamp,
     359                    author=self.contributors.create(
     360                        commit_data['commit']['author']['name'],
     361                        email_match.group('email') if email_match else None,
     362                    ), order=0,
     363                    message=commit_data['commit']['message'] if include_log else None,
     364                )
     365                if not cached or cached[0].timestamp != previous.timestamp:
     366                    for c in cached:
     367                        yield c
     368                    cached = [previous]
     369                else:
     370                    for c in cached:
     371                        c.order += 1
     372                    cached.append(previous)
     373
     374                if previous.hash == begin.hash or previous.timestamp < begin.timestamp:
     375                    previous = None
     376                    break
     377
     378        for c in cached:
     379            c.order += begin.order
     380            yield c
     381
     382
    316383    def find(self, argument, include_log=True, include_identifier=True):
    317384        if not isinstance(argument, six.string_types):

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/remote/svn.py

    r275635 r277102  
    3333from datetime import datetime
    3434
    35 from webkitcorepy import log, run, decorators
     35from webkitcorepy import decorators, string_utils
    3636from webkitscmpy.remote.scm import Scm
    37 from webkitscmpy import Commit, Contributor, Version
     37from webkitscmpy import Commit, Version
    3838
    3939
    4040class Svn(Scm):
    4141    URL_RE = re.compile(r'\Ahttps?://svn.(?P<host>\S+)/repository/\S+\Z')
    42     HISTORY_RE = re.compile(b'<D:version-name>(?P<revision>\d+)</D:version-name>')
     42    DATA_RE = re.compile(b'<[SD]:(?P<tag>\S+)>(?P<content>.*)</[SD]:.+>')
    4343    CACHE_VERSION = Version(1)
    4444
    … …  
    245245            default_count = 0
    246246            for line in response.iter_lines():
    247                 match = self.HISTORY_RE.match(line)
    248                 if not match:
     247                match = self.DATA_RE.match(line)
     248                if not match or match.group('tag') != b'version-name':
    249249                    continue
    250250
    … …  
    255255                        did_warn = True
    256256
    257                 revision = int(match.group('revision'))
     257                revision = int(match.group('content'))
    258258                if pos > 0 and self._metadata_cache[branch][pos - 1] == revision:
    259259                    break
    … …  
    456456            message=message,
    457457        )
     458
     459    def _args_from_content(self, content, include_log=True):
     460        xml = xmltodict.parse(content)
     461        date = datetime.strptime(string_utils.decode(xml['S:log-item']['S:date']).split('.')[0], '%Y-%m-%dT%H:%M:%S')
     462        name = string_utils.decode(xml['S:log-item']['D:creator-displayname'])
     463
     464        return dict(
     465            revision=int(xml['S:log-item']['D:version-name']),
     466            author=self.contributors.create(name, name) if name and '@' in name else self.contributors.create(name),
     467            timestamp=int(calendar.timegm(date.timetuple())),
     468            message=string_utils.decode(xml['S:log-item']['D:comment']) if include_log else None,
     469        )
     470
     471    def commits(self, begin=None, end=None, include_log=True, include_identifier=True):
     472        begin, end = self._commit_range(begin=begin, end=end, include_identifier=include_identifier)
     473        previous = end
     474
     475        content = b''
     476        with requests.request(
     477                method='REPORT',
     478                url='{}!svn/rvr/{}/{}'.format(
     479                    self.url,
     480                    end.revision,
     481                    end.branch if end.branch == self.default_branch or '/' in end.branch else 'branches/{}'.format(end.branch),
     482                ), stream=True,
     483                headers={
     484                    'Content-Type': 'text/xml',
     485                    'Accept-Encoding': 'gzip',
     486                    'DEPTH': '1',
     487                }, data='<S:log-report xmlns:S="svn:">\n'
     488                        '<S:start-revision>{end}</S:start-revision>\n'
     489                        '<S:end-revision>{begin}</S:end-revision>\n'
     490                        '<S:path></S:path>\n'
     491                        '</S:log-report>\n'.format(end=end.revision, begin=begin.revision),
     492        ) as response:
     493            if response.status_code != 200:
     494                raise self.Exception("Failed to construct branch history for '{}'".format(branch))
     495            for line in response.iter_lines():
     496                if line == b'<S:log-item>':
     497                    content = line + b'\n'
     498                else:
     499                    content += line + b'\n'
     500                if line != b'</S:log-item>':
     501                    continue
     502
     503                args = self._args_from_content(content, include_log=include_log)
     504
     505                branch_point = previous.branch_point if include_identifier else None
     506                identifier = previous.identifier if include_identifier else None
     507                if args['revision'] != previous.revision:
     508                    identifier -= 1
     509                if not identifier:
     510                    identifier = branch_point
     511                    branch_point = None
     512
     513                previous = Commit(
     514                    repository_id=self.id,
     515                    branch=end.branch if branch_point else self.default_branch,
     516                    identifier=identifier,
     517                    branch_point=branch_point,
     518                    **args
     519                )
     520                yield previous
     521                content = b''

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/scm_base.py

    r275353 r277102  
    7272
    7373    def commit(self, hash=None, revision=None, identifier=None, branch=None, tag=None, include_log=True, include_identifier=True):
     74        raise NotImplementedError()
     75
     76    def _commit_range(self, begin=None, end=None, include_log=False, include_identifier=True):
     77        begin_args = begin or dict()
     78        end_args = end or dict()
     79
     80        if not begin_args:
     81            raise TypeError("_commit_range() missing required 'begin' arguments")
     82        if not end_args:
     83            raise TypeError("_commit_range() missing required 'end' arguments")
     84
     85        if list(begin_args.keys()) == ['argument']:
     86            begin_result = self.find(include_log=include_log, include_identifier=False, **begin_args)
     87        else:
     88            begin_result = self.commit(include_log=include_log, include_identifier=False, **begin_args)
     89
     90        if list(end_args.keys()) == ['argument']:
     91            end_result = self.find(include_log=include_log, include_identifier=include_identifier, **end_args)
     92        else:
     93            end_result = self.commit(include_log=include_log, include_identifier=include_identifier, **end_args)
     94
     95        if not begin_result:
     96            raise TypeError("'{}' failed to define begin in _commit_range()".format(begin_args))
     97        if not end_result:
     98            raise TypeError("'{}' failed to define begin in _commit_range()".format(end_args))
     99        if begin_result.timestamp > end_result.timestamp:
     100            raise TypeError("'{}' pre-dates '{}' in _commit_range()".format(begin_result, end_result))
     101        if end_result.branch == self.default_branch and begin_result.branch != self.default_branch:
     102            raise TypeError("'{}' and '{}' do not share linear history".format(begin_result, end_result))
     103        return begin_result, end_result
     104
     105    def commits(self, begin=None, end=None, include_log=True, include_identifier=True):
    74106        raise NotImplementedError()
    75107

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/test/find_unittest.py

    r275353 r277102  
    232232Revision: 4
    233233Identifier: 3@trunk
    234 '''.format(datetime.fromtimestamp(1601686700).strftime('%a %b %d %H:%M:%S %Y')),
     234'''.format(datetime.fromtimestamp(1601684700).strftime('%a %b %d %H:%M:%S %Y')),
    235235        )
    236236

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/test/git_unittest.py

    r274830 r277102  
    2929from webkitcorepy import LoggerCapture, OutputCapture
    3030from webkitcorepy.mocks import Time as MockTime
    31 from webkitscmpy import local, mocks, remote
     31from webkitscmpy import Commit, local, mocks, remote
    3232
    3333
    … …  
    284284                self.assertEqual(1, local.Git(self.path).commit(hash='d8bce26fa65c').order)
    285285
     286    def test_commits(self):
     287        for mock in [mocks.local.Git(self.path), mocks.local.Git(self.path, git_svn=True)]:
     288            with mock:
     289                git = local.Git(self.path)
     290                self.assertEqual(Commit.Encoder().default([
     291                    git.commit(hash='bae5d1e9'),
     292                    git.commit(hash='1abe25b4'),
     293                    git.commit(hash='fff83bb2'),
     294                    git.commit(hash='9b8311f2'),
     295                ]), Commit.Encoder().default(list(git.commits(begin=dict(hash='9b8311f2'), end=dict(hash='bae5d1e9')))))
     296
     297    def test_commits_branch(self):
     298        for mock in [mocks.local.Git(self.path), mocks.local.Git(self.path, git_svn=True)]:
     299            with mock:
     300                git = local.Git(self.path)
     301                self.assertEqual(Commit.Encoder().default([
     302                    git.commit(hash='621652ad'),
     303                    git.commit(hash='a30ce849'),
     304                    git.commit(hash='fff83bb2'),
     305                    git.commit(hash='9b8311f2'),
     306                ]), Commit.Encoder().default(list(git.commits(begin=dict(argument='9b8311f2'), end=dict(argument='621652ad')))))
     307
    286308
    287309class TestGitHub(unittest.TestCase):
    … …  
    416438        self.assertEqual(remote.GitHub(self.remote).id, 'webkit')
    417439
     440    def test_commits(self):
     441        with mocks.remote.GitHub():
     442            git = remote.GitHub(self.remote)
     443            self.assertEqual(Commit.Encoder().default([
     444                git.commit(hash='bae5d1e9'),
     445                git.commit(hash='1abe25b4'),
     446                git.commit(hash='fff83bb2'),
     447                git.commit(hash='9b8311f2'),
     448            ]), Commit.Encoder().default(list(git.commits(begin=dict(hash='9b8311f2'), end=dict(hash='bae5d1e9')))))
     449
     450    def test_commits_branch(self):
     451        with mocks.remote.GitHub():
     452            git = remote.GitHub(self.remote)
     453            self.assertEqual(Commit.Encoder().default([
     454                git.commit(hash='621652ad'),
     455                git.commit(hash='a30ce849'),
     456                git.commit(hash='fff83bb2'),
     457                git.commit(hash='9b8311f2'),
     458            ]), Commit.Encoder().default(list(git.commits(begin=dict(argument='9b8311f2'), end=dict(argument='621652ad')))))
     459
     460
    418461
    419462class TestBitBucket(unittest.TestCase):

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/test/svn_unittest.py

    r275604 r277102  
    2828from datetime import datetime, timedelta
    2929from webkitcorepy import OutputCapture
    30 from webkitscmpy import local, mocks, remote
     30from webkitscmpy import Commit, local, mocks, remote
    3131
    3232
    … …  
    232232            self.assertIsNone(local.Svn(self.path).find('trunk', include_identifier=False).identifier)
    233233
     234    def test_commits(self):
     235        with mocks.local.Svn(self.path), OutputCapture():
     236            svn = local.Svn(self.path)
     237            self.assertEqual(Commit.Encoder().default([
     238                svn.commit(revision='r6'),
     239                svn.commit(revision='r4'),
     240                svn.commit(revision='r2'),
     241                svn.commit(revision='r1'),
     242            ]), Commit.Encoder().default(list(svn.commits(begin=dict(revision='r1'), end=dict(revision='r6')))))
     243
     244    def test_commits_branch(self):
     245        with mocks.local.Svn(self.path), OutputCapture():
     246            svn = local.Svn(self.path)
     247            self.assertEqual(Commit.Encoder().default([
     248                svn.commit(revision='r7'),
     249                svn.commit(revision='r3'),
     250                svn.commit(revision='r2'),
     251                svn.commit(revision='r1'),
     252            ]), Commit.Encoder().default(list(svn.commits(begin=dict(argument='r1'), end=dict(argument='r7')))))
     253
    234254
    235255class TestRemoteSvn(unittest.TestCase):
    … …  
    332352    def test_id(self):
    333353        self.assertEqual(remote.Svn(self.remote).id, 'webkit')
     354
     355    def test_commits(self):
     356        self.maxDiff = None
     357        with mocks.remote.Svn():
     358            svn = remote.Svn(self.remote)
     359            self.assertEqual(Commit.Encoder().default([
     360                svn.commit(revision='r6'),
     361                svn.commit(revision='r4'),
     362                svn.commit(revision='r2'),
     363                svn.commit(revision='r1'),
     364            ]), Commit.Encoder().default(list(svn.commits(begin=dict(revision='r1'), end=dict(revision='r6')))))
     365
     366    def test_commits_branch(self):
     367        with mocks.remote.Svn(), OutputCapture():
     368            svn = remote.Svn(self.remote)
     369            self.assertEqual(Commit.Encoder().default([
     370                svn.commit(revision='r7'),
     371                svn.commit(revision='r3'),
     372                svn.commit(revision='r2'),
     373                svn.commit(revision='r1'),
     374            ]), Commit.Encoder().default(list(svn.commits(begin=dict(argument='r1'), end=dict(argument='r7')))))
Note: See TracChangeset for help on using the changeset viewer.