⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 271182 in webkit


Ignore:
Timestamp:
Jan 5, 2021, 4:35:52 PM (6 years ago)
Author:
Jonathan Bedard
Message:

[webkitscmpy] Add command to canonicalize unpushed commits
​https://bugs.webkit.org/show_bug.cgi?id=219982
<rdar://problem/72427536>

Reviewed by Dewei Zhu.

  • Scripts/git-webkit: Specify web-service for canonical identifier translation.
  • Scripts/libraries/webkitscmpy/setup.py: Add canonicalize directory to package.
  • Scripts/libraries/webkitscmpy/webkitscmpy/init.py: Bump version.
  • Scripts/libraries/webkitscmpy/webkitscmpy/canonicalize: Added.
  • Scripts/libraries/webkitscmpy/webkitscmpy/canonicalize/init.py: Added.

(Canonicalize): Command to edit history of unpushed commits on a branch.
(Canonicalize.parser):
(Canonicalize.main): Call git filter-branch to edit commit message and authorship
of unpushed commits.

  • Scripts/libraries/webkitscmpy/webkitscmpy/canonicalize/committer.py: Added.

(canonicalize): Given a name, email and a contributor mapping, return a canonical name
and email for the given author or committer.
(main): Print out the canonical author and committer to be parsed by git filter-branch.

  • Scripts/libraries/webkitscmpy/webkitscmpy/canonicalize/message.py: Added.

(main): Add the canonical identifier to a commit message.

  • Scripts/libraries/webkitscmpy/webkitscmpy/contributor.py:

(Contributor): Add unknown author regex.
(Contributor.from_scm_log):

  • Scripts/libraries/webkitscmpy/webkitscmpy/mocks/local/git.py:

(Git.init): Add git filter-branch mock. Note that this mock makes an effort to test
message.py and contributor.py, but not the shell script passed to the command.

  • Scripts/libraries/webkitscmpy/webkitscmpy/program.py:

(Command.main): Accept arbitrary kwargs.
(Find.main): Ditto.
(Checkout.main): Ditto.
(main): Allow caller to specify a string template for canonical identifiers in commit messages.

  • Scripts/libraries/webkitscmpy/webkitscmpy/test/canonicalize_unittest.py: Added.

(TestCanonicalize):
(TestCanonicalize.test_invalid):
(TestCanonicalize.test_no_commits):
(TestCanonicalize.test_formated_identifier):
(TestCanonicalize.test_git_svn):
(TestCanonicalize.test_branch_commits):

Location:
trunk/Tools
Files:
5 added
7 edited

Legend:

Unmodified
Added
Removed

  • trunk/Tools/ChangeLog

    r271179 r271182  
     12021-01-05  Jonathan Bedard  <jbedard@apple.com>
     2
     3        [webkitscmpy] Add command to canonicalize unpushed commits
     4        https://bugs.webkit.org/show_bug.cgi?id=219982
     5        <rdar://problem/72427536>
     6
     7        Reviewed by Dewei Zhu.
     8
     9        * Scripts/git-webkit: Specify web-service for canonical identifier translation.
     10        * Scripts/libraries/webkitscmpy/setup.py: Add canonicalize directory to package.
     11        * Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py: Bump version.
     12        * Scripts/libraries/webkitscmpy/webkitscmpy/canonicalize: Added.
     13        * Scripts/libraries/webkitscmpy/webkitscmpy/canonicalize/__init__.py: Added.
     14        (Canonicalize): Command to edit history of unpushed commits on a branch.
     15        (Canonicalize.parser):
     16        (Canonicalize.main): Call `git filter-branch` to edit commit message and authorship
     17        of unpushed commits.
     18        * Scripts/libraries/webkitscmpy/webkitscmpy/canonicalize/committer.py: Added.
     19        (canonicalize): Given a name, email and a contributor mapping, return a canonical name
     20        and email for the given author or committer.
     21        (main): Print out the canonical author and committer to be parsed by `git filter-branch`.
     22        * Scripts/libraries/webkitscmpy/webkitscmpy/canonicalize/message.py: Added.
     23        (main): Add the canonical identifier to a commit message.
     24        * Scripts/libraries/webkitscmpy/webkitscmpy/contributor.py:
     25        (Contributor): Add unknown author regex.
     26        (Contributor.from_scm_log):
     27        * Scripts/libraries/webkitscmpy/webkitscmpy/mocks/local/git.py:
     28        (Git.__init__): Add `git filter-branch` mock. Note that this mock makes an effort to test
     29        message.py and contributor.py, but not the shell script passed to the command.
     30        * Scripts/libraries/webkitscmpy/webkitscmpy/program.py:
     31        (Command.main): Accept arbitrary **kwargs.
     32        (Find.main): Ditto.
     33        (Checkout.main): Ditto.
     34        (main): Allow caller to specify a string template for canonical identifiers in commit messages.
     35        * Scripts/libraries/webkitscmpy/webkitscmpy/test/canonicalize_unittest.py: Added.
     36        (TestCanonicalize):
     37        (TestCanonicalize.test_invalid):
     38        (TestCanonicalize.test_no_commits):
     39        (TestCanonicalize.test_formated_identifier):
     40        (TestCanonicalize.test_git_svn):
     41        (TestCanonicalize.test_branch_commits):
     42
    1432021-01-05  Angelos Oikonomopoulos  <angelos@igalia.com>
    244

  • trunk/Tools/Scripts/git-webkit

    r270447 r271182  
    4343                contributors[nick] = c
    4444
    45     sys.exit(program.main(path=os.path.dirname(__file__), contributors=contributors))
     45    sys.exit(program.main(
     46        path=os.path.dirname(__file__),
     47        contributors=contributors,
     48        identifier_template='Canonical link: https://commits.webkit.org/{}',
     49    ))
    4650

  • trunk/Tools/Scripts/libraries/webkitscmpy/setup.py

    r270893 r271182  
    5151    packages=[
    5252        'webkitscmpy',
     53        'webkitscmpy.canonicalize',
    5354        'webkitscmpy.local',
    5455        'webkitscmpy.mocks',

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/__init__.py

    r270956 r271182  
    4747    )
    4848
    49 version = Version(0, 6, 4)
     49version = Version(0, 7, 0)
    5050
    5151AutoInstall.register(Package('fasteners', Version(0, 15, 0)))

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/contributor.py

    r270858 r271182  
    3131    GIT_AUTHOR_RE = re.compile(r'Author: (?P<author>.*) <(?P<email>[^@]+@[^@]+)(@.*)?>')
    3232    AUTOMATED_CHECKIN_RE = re.compile(r'Author: (?P<author>.*) <devnull>')
     33    UNKNOWN_AUTHOR = re.compile(r'Author: (?P<author>.*) <None>')
    3334    SVN_AUTHOR_RE = re.compile(r'r\d+ \| (?P<email>.*) \| (?P<date>.*) \| \d+ lines?')
    3435    SVN_PATCH_FROM_RE = re.compile(r'Patch by (?P<author>.*) <(?P<email>.*)> on \d+-\d+-\d+')
    … …  
    112113        author = None
    113114
    114         for expression in [cls.GIT_AUTHOR_RE, cls.SVN_AUTHOR_RE, cls.SVN_PATCH_FROM_RE, cls.AUTOMATED_CHECKIN_RE]:
     115        for expression in [cls.GIT_AUTHOR_RE, cls.SVN_AUTHOR_RE, cls.SVN_PATCH_FROM_RE, cls.AUTOMATED_CHECKIN_RE, cls.UNKNOWN_AUTHOR]:
    115116            match = expression.match(line)
    116117            if match:
    117118                if 'author' in expression.groupindex:
    118119                    author = match.group('author')
    119                     if '(no author)' in author or 'Automated Checkin' in author:
     120                    if '(no author)' in author or 'Automated Checkin' in author or 'Unknown' in author:
    120121                        author = None
    121122                if 'email' in expression.groupindex:

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/mocks/local/git.py

    r270956 r271182  
    2626
    2727from datetime import datetime
    28 from webkitcorepy import mocks
     28from webkitcorepy import mocks, OutputCapture, StringIO
    2929from webkitscmpy import local, Commit, Contributor
     30from webkitscmpy.canonicalize.committer import main as committer_main
     31from webkitscmpy.canonicalize.message import main as message_main
    3032
    3133
    … …  
    5961
    6062        self.head = self.commits[self.default_branch][-1]
     63        self.remotes = {'origin/{}'.format(branch): commits[-1] for branch, commits in self.commits.items()}
    6164        self.tags = {}
    6265
    … …  
    234237                    mocks.ProcessCompletion(returncode=0) if self.checkout(args[2]) else mocks.ProcessCompletion(returncode=1)
    235238            ), mocks.Subprocess.Route(
     239                self.executable, 'filter-branch', '-f',
     240                cwd=self.path,
     241                generator=lambda *args, **kwargs: self.filter_branch(
     242                    args[-1],
     243                    identifier_template=args[-2].split("'")[-2] if args[-3] == '--msg-filter' else None,
     244                    environment_shell=args[4] if args[3] == '--env-filter' and args[4] else None,
     245                )
     246            ), mocks.Subprocess.Route(
    236247                self.executable,
    237248                cwd=self.path,
    … …  
    266277                return None
    267278
     279        something = str(something)
     280        if '..' in something:
     281            a, b = something.split('..')
     282            a = self.find(a)
     283            b = self.find(b)
     284            return b if a and b else None
     285
    268286        if something == 'HEAD':
    269287            return self.head
    … …  
    272290        if something in self.tags.keys():
    273291            return self.tags[something]
    274 
    275         something = str(something)
    276         if '..' in something:
    277             something = something.split('..')[1]
     292        if something in self.remotes.keys():
     293            return self.remotes[something]
     294
    278295        for branch, commits in self.commits.items():
    279296            if branch == something:
    … …  
    287304
    288305    def count(self, something):
    289         match = self.find(something)
    290         if '..' in something or not match.branch_point:
    291             return match.identifier
    292         return match.branch_point + match.identifier
     306        if '..' not in something:
     307            match = self.find(something)
     308            return (match.branch_point or 0) + match.identifier
     309
     310        a, b = something.split('..')
     311        a = self.find(a)
     312        b = self.find(b)
     313        if a.branch_point == b.branch_point:
     314            return abs(b.identifier - a.identifier)
     315        return b.identifier
    293316
    294317    def branches_on(self, hash):
    … …  
    314337            self.detached = something not in self.commits.keys()
    315338        return True if commit else False
     339
     340    def filter_branch(self, range, identifier_template=None, environment_shell=None):
     341        # We can't effectively mock the bash script in the command, but we can mock the python code that
     342        # script calls, which is where the program logic is.
     343        head, start = range.split('...')
     344        head = self.find(head)
     345        start = self.find(start)
     346
     347        commits_to_edit = []
     348        for commit in reversed(self.commits[head.branch]):
     349            if commit.branch == start.branch and commit.identifier <= start.identifier:
     350                break
     351            commits_to_edit.insert(0, commit)
     352        if head.branch != self.default_branch:
     353            for commit in reversed(self.commits[self.default_branch][:head.branch_point]):
     354                if commit.identifier <= start.identifier:
     355                    break
     356                commits_to_edit.insert(0, commit)
     357
     358        stdout = StringIO()
     359        original_env = {key: os.environ.get('OLDPWD') for key in [
     360            'OLDPWD', 'GIT_COMMIT',
     361            'GIT_AUTHOR_NAME', 'GIT_AUTHOR_EMAIL',
     362            'GIT_COMMITTER_NAME', 'GIT_COMMITTER_EMAIL',
     363        ]}
     364
     365        try:
     366            count = 0
     367            os.environ['OLDPWD'] = self.path
     368            for commit in commits_to_edit:
     369                count += 1
     370                os.environ['GIT_COMMIT'] = commit.hash
     371                os.environ['GIT_AUTHOR_NAME'] = commit.author.name
     372                os.environ['GIT_AUTHOR_EMAIL'] = commit.author.email
     373                os.environ['GIT_COMMITTER_NAME'] = commit.author.name
     374                os.environ['GIT_COMMITTER_EMAIL'] = commit.author.email
     375
     376                stdout.write(
     377                    'Rewrite {hash} ({count}/{total}) (--- seconds passed, remaining --- predicted)\n'.format(
     378                        hash=commit.hash,
     379                        count=count,
     380                        total=len(commits_to_edit),
     381                    ))
     382
     383                if identifier_template:
     384                    messagefile = StringIO()
     385                    messagefile.write(commit.message)
     386                    messagefile.seek(0)
     387                    with OutputCapture() as captured:
     388                        message_main(messagefile, identifier_template)
     389                    lines = captured.stdout.getvalue().splitlines()
     390                    if lines[-1].startswith('git-svn-id: https://svn'):
     391                        lines.pop(-1)
     392                    commit.message = '\n'.join(lines)
     393
     394                if not environment_shell:
     395                    continue
     396                if re.search(r'echo "Overwriting', environment_shell):
     397                    stdout.write('Overwriting {}\n'.format(commit.hash))
     398
     399                match = re.search(r'(?P<json>\S+\.json)', environment_shell)
     400                if match:
     401                    with OutputCapture() as captured:
     402                        committer_main(match.group('json'))
     403                    captured.stdout.seek(0)
     404                    for line in captured.stdout.readlines():
     405                        line = line.rstrip()
     406                        os.environ[line.split(' ')[0]] = ' '.join(line.split(' ')[1:])
     407
     408                commit.author = Contributor(name=os.environ['GIT_AUTHOR_NAME'], emails=[os.environ['GIT_AUTHOR_EMAIL']])
     409
     410                if re.search(r'echo "\s+', environment_shell):
     411                    for key in ['GIT_AUTHOR_NAME', 'GIT_AUTHOR_EMAIL', 'GIT_COMMITTER_NAME', 'GIT_COMMITTER_EMAIL']:
     412                        stdout.write('    {}={}\n'.format(key, os.environ[key]))
     413
     414        finally:
     415            for key, value in original_env.items():
     416                if value is not None:
     417                    os.environ[key] = value
     418                else:
     419                    del os.environ[key]
     420
     421        return mocks.ProcessCompletion(
     422            returncode=0,
     423            stdout=stdout.getvalue(),
     424        )

  • trunk/Tools/Scripts/libraries/webkitscmpy/webkitscmpy/program.py

    r270447 r271182  
    4444
    4545    @classmethod
    46     def main(cls, args, repository):
     46    def main(cls, args, repository, **kwargs):
    4747        sys.stderr.write('No command specified\n')
    4848        return -1
    … …  
    8282
    8383    @classmethod
    84     def main(cls, args, repository):
     84    def main(cls, args, repository, **kwargs):
    8585        try:
    8686            commit = repository.find(args.argument[0], include_log=args.include_log)
    … …  
    144144
    145145    @classmethod
    146     def main(cls, args, repository):
     146    def main(cls, args, repository, **kwargs):
    147147        if not repository.path:
    148148            sys.stderr.write("Cannot checkout on remote repository")
    … …  
    163163
    164164
    165 def main(args=None, path=None, loggers=None, contributors=None):
     165def main(args=None, path=None, loggers=None, contributors=None, identifier_template=None):
    166166    logging.basicConfig(level=logging.WARNING)
    167167
    … …  
    184184    subparsers = parser.add_subparsers(help='sub-command help')
    185185
    186     for program in [Find, Checkout]:
     186    from webkitscmpy.canonicalize import Canonicalize
     187
     188    for program in [Find, Checkout, Canonicalize]:
    187189        subparser = subparsers.add_parser(program.name, help=program.help)
    188190        subparser.set_defaults(main=program.main)
    … …  
    196198        repository = local.Scm.from_path(path=parsed.repository, contributors=contributors)
    197199
    198     return parsed.main(args=parsed, repository=repository)
     200    return parsed.main(args=parsed, repository=repository, identifier_template=identifier_template)
Note: See TracChangeset for help on using the changeset viewer.