⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 281184 in webkit


Ignore:
Timestamp:
Aug 18, 2021, 2:13:47 AM (5 years ago)
Author:
Martin Robinson
Message:

[Flatpak SDK] Increase the startup performance of webkit-flatpak
​https://bugs.webkit.org/show_bug.cgi?id=229185

Reviewed by Philippe Normand.

Improve flatpakutils.py by ensuring that we don't make redundant
calls to flatpak during initialization of the data structures. This
saves 5 seconds on every call to build-webkit and run-webkit-tests.

Before:

$ time ./Tools/Scripts/webkit-flatpak -c true
real 0m6,297s
user 0m0,786s
sys 0m0,513s

After:

$ time ./Tools/Scripts/webkit-flatpak -c true
real 0m1,243s
user 0m0,375s
sys 0m0,162s

  • flatpak/flatpakutils.py:

(FlatpakPackages.init): Separate the update into another
method so that it can be called directly when a package is installed.
Add new packages directly to self.packages.
(FlatpakRepos.init): Ensure that we only update our repositories
and package list once we've initialized our list of repositories.
(FlatpakRepos.add): Return True if this method actually changed anything
and accept a parameter determining whether an update is done to
the repository and package list.
(FlatpakRepos.is_package_installed): Added this method which replaces
FlatpakRepo.is_app_installed.
(FlatpakRepo.init): Remove the app registry because it is duplicated
by FlatpakPackages.
(FlatpakPackage.is_installed): Now call FlatpakRepos.is_package_installed.
(WebkitFlatpak._reset_repository): Set up both repositories here and
pass then as part of the FlatpakRepos constructor.
(WebkitFlatpak.main): Separate out the check which deletes the Flatpak
repository from the one that checks if the packages are installed. This
makes sure we don't install packages and then immediately delete them.
(WebkitFlatpak.check_installed_packages): rename _get_packages to _get_dependency_packages
in order to make it clear that it doesn't return installed packages.
(WebkitFlatpak._get_dependency_packages): Ditto.
(WebkitFlatpak.install_all): Ditto.
(FlatpakRepo.is_app_installed): Deleted.
(WebkitFlatpak._get_packages): Deleted.

Location:
trunk/Tools
Files:
2 edited

Legend:

Unmodified
Added
Removed

  • trunk/Tools/ChangeLog

    r281167 r281184  
     12021-08-18  Martin Robinson  <mrobinson@webkit.org>
     2
     3        [Flatpak SDK] Increase the startup performance of webkit-flatpak
     4        https://bugs.webkit.org/show_bug.cgi?id=229185
     5
     6        Reviewed by Philippe Normand.
     7
     8        Improve flatpakutils.py by ensuring that we don't make redundant
     9        calls to flatpak during initialization of the data structures. This
     10        saves 5 seconds on every call to build-webkit and run-webkit-tests.
     11
     12        Before:
     13            $ time ./Tools/Scripts/webkit-flatpak -c true
     14            real    0m6,297s
     15            user    0m0,786s
     16            sys     0m0,513s
     17
     18        After:
     19            $ time ./Tools/Scripts/webkit-flatpak -c true
     20            real    0m1,243s
     21            user    0m0,375s
     22            sys     0m0,162s
     23
     24        * flatpak/flatpakutils.py:
     25        (FlatpakPackages.__init__): Separate the update into another
     26        method so that it can be called directly when a package is installed.
     27        Add new packages directly to self.packages.
     28        (FlatpakRepos.__init__): Ensure that we only update our repositories
     29        and package list once we've initialized our list of repositories.
     30        (FlatpakRepos.add): Return True if this method actually changed anything
     31        and accept a parameter determining whether an update is done to
     32        the repository and package list.
     33        (FlatpakRepos.is_package_installed): Added this method which replaces
     34        FlatpakRepo.is_app_installed.
     35        (FlatpakRepo.__init__): Remove the app registry because it is duplicated
     36        by FlatpakPackages.
     37        (FlatpakPackage.is_installed): Now call FlatpakRepos.is_package_installed.
     38        (WebkitFlatpak._reset_repository): Set up both repositories here and
     39        pass then as part of the FlatpakRepos constructor.
     40        (WebkitFlatpak.main): Separate out the check which deletes the Flatpak
     41        repository from the one that checks if the packages are installed. This
     42        makes sure we don't install packages and then immediately delete them.
     43        (WebkitFlatpak.check_installed_packages): rename _get_packages to _get_dependency_packages
     44        in order to make it clear that it doesn't return installed packages.
     45        (WebkitFlatpak._get_dependency_packages): Ditto.
     46        (WebkitFlatpak.install_all): Ditto.
     47        (FlatpakRepo.is_app_installed): Deleted.
     48        (WebkitFlatpak._get_packages): Deleted.
     49
    1502021-08-17  Jonathan Bedard  <jbedard@apple.com>
    251

  • trunk/Tools/flatpak/flatpakutils.py

    r280172 r281184  
    229229    def __init__(self, repos, user=True):
    230230        FlatpakObject.__init__(self, user=user)
    231 
    232231        self.repos = repos
    233 
    234         packs = []
     232        self.update()
     233
     234    def update(self):
     235        self.packages = []
    235236        out = self.flatpak("list", "--columns=application,arch,branch,origin", "-a", gather_output=True)
    236237        package_defs = [line for line in out.split("\n") if line]
    … …  
    240241            # If installed from a file, the package is in no repo
    241242            repo = self.repos.repos.get(origin)
    242 
    243             packs.append(FlatpakPackage(name, branch, repo, arch))
    244 
    245         self.packages = packs
     243            self.packages.append(FlatpakPackage(name, branch, repo, arch))
    246244
    247245    def __iter__(self):
    … …  
    252250class FlatpakRepos(FlatpakObject):
    253251
    254     def __init__(self, user=True):
     252    def __init__(self, repos, user=True):
    255253        FlatpakObject.__init__(self, user=user)
    256         self.repos = {}
    257254        self.update()
     255
     256        updated = False
     257        for repo in repos:
     258            updated = self.add(repo, update=False) or updated
     259
     260        # Only fetch the remote and package list again if we updated anything.
     261        if updated:
     262            self.update()
    258263
    259264    def update(self):
    … …  
    272277        self.packages = FlatpakPackages(self)
    273278
    274     def add(self, repo, override=True):
     279    def add(self, repo, update=True, override=True):
    275280        try:
     281            repo.repos = self
     282
    276283            same_name = None
    277284            for name, tmprepo in self.repos.items():
    278285                if repo.url == tmprepo.url:
    279                     return tmprepo
     286                    return False
    280287                elif repo.name == name:
    281288                    same_name = tmprepo
    282289
    283290            if same_name:
    284                 if override:
     291                if override and same_name.url != repo.url:
    285292                    self.flatpak("remote-modify", repo.name, "--url=" + repo.url)
    286293                    same_name.url = repo.url
    287 
    288                     return same_name
     294                    return True
    289295                else:
    290                     return None
     296                    return False
     297
     298            args = ["remote-add", repo.name, "--if-not-exists"]
     299            if repo.repo_file:
     300                args.extend(["--from", repo.repo_file.name])
    291301            else:
    292                 args = ["remote-add", repo.name, "--if-not-exists"]
    293                 if repo.repo_file:
    294                     args.extend(["--from", repo.repo_file.name])
    295                 else:
    296                     args.extend(["--no-gpg-verify", repo.url])
    297                 self.flatpak(*args, comment="Adding repo %s" % repo.name)
    298 
    299             repo.repos = self
    300             return repo
     302                args.extend(["--no-gpg-verify", repo.url])
     303            self.flatpak(*args, comment="Adding repo %s" % repo.name)
     304            return True
    301305        finally:
    302             self.update()
     306            if update:
     307                self.packages = FlatpakPackages(self)
     308
     309    def is_package_installed(self, name, branch=None, arch=None):
     310        for package in self.packages:
     311            if name != package.name:
     312                continue
     313            if branch and branch != package.branch:
     314                continue
     315            if arch and arch != package.arch:
     316                continue
     317            return True
     318        return False
    303319
    304320
    … …  
    326342            assert url
    327343
    328         self._app_registry = {}
    329         output = self.flatpak("list", "--columns=application,branch", "-a", gather_output=True)
    330         for line in output.splitlines():
    331             name, branch = line.split("\t")
    332             self._app_registry[name] = branch
    333 
    334     def is_app_installed(self, name, branch=None):
    335         if branch:
    336             try:
    337                 return self._app_registry[name] == branch
    338             except KeyError:
    339                 return False
    340         else:
    341             return name in self._app_registry.keys()
    342 
    343344    @property
    344345    def repo_file(self):
    … …  
    374375
    375376    def is_installed(self, branch):
     377        # Bundles installed from files do not have repositories.
    376378        if not self.repo:
    377             # Bundle installed from file
    378379            return True
    379 
    380         for package in self.repo.repos.packages:
    381             if package.name == self.name and package.branch == branch and package.arch == self.arch:
    382                 return True
    383 
    384         return False
     380        return self.repo.repos.is_package_installed(self.name, branch, self.arch)
    385381
    386382    def install(self):
    … …  
    392388        comment = "Installing from " + self.repo.name + " " + self.name + " " + self.arch + " " + branch
    393389        self.flatpak(*args, comment=comment)
     390        self.repo.repos.packages.update()
    394391
    395392    def update(self):
    … …  
    604601
    605602    def _reset_repository(self):
    606         self.repos = FlatpakRepos()
    607603        url = "https://software.igalia.com/webkit-sdk-repo/"
    608604        repo_file = "https://software.igalia.com/flatpak-refs/webkit-sdk.flatpakrepo"
    … …  
    610606            url = "file://%s" % self.user_repo
    611607            repo_file = None
    612         self.sdk_repo = self.repos.add(FlatpakRepo("webkit-sdk", url=url, repo_file=repo_file))
     608
     609        self.sdk_repo = FlatpakRepo("webkit-sdk", url=url, repo_file=repo_file)
     610        self.flathub_repo = FlatpakRepo("flathub", url="https://dl.flathub.org/repo/",
     611                                        repo_file="https://dl.flathub.org/repo/flathub.flatpakrepo")
     612        self.repos = FlatpakRepos([self.sdk_repo, self. flathub_repo])
    613613
    614614    def setup_builddir(self):
    … …  
    987987            regenerate_toolchains = (repo.version("org.webkit.Sdk") != version_before_update) or not self.check_toolchains_generated()
    988988
    989             for package in self._get_packages():
    990                 if package.name.startswith("org.webkit") and repo.is_app_installed(package.name) \
    991                    and not repo.is_app_installed(package.name, branch=self.sdk_branch):
     989            # If we have an out-of-date package, simply remove the entire flatpak directory and start over.
     990            for package in self._get_dependency_packages():
     991                if package.name.startswith("org.webkit") \
     992                   and self.repos.is_package_installed(package.name) \
     993                   and not self.repos.is_package_installed(package.name, branch=self.sdk_branch):
    992994                    Console.message("New SDK version available, removing local UserFlatpak directory before switching to new version")
    993995                    shutil.rmtree(self.flatpak_build_path)
    994996                    self._reset_repository()
    995                     regenerate_toolchains = True
    996997                    break
    997                 elif not repo.is_app_installed(package.name):
     998
     999            for package in self._get_dependency_packages():
     1000                if not self.repos.is_package_installed(package.name):
    9981001                    package.install()
    9991002                    regenerate_toolchains = True
     1003
    10001004        else:
    10011005            regenerate_toolchains = self.regenerate_toolchains
    … …  
    11011105
    11021106    def check_installed_packages(self):
    1103         for package in self._get_packages():
     1107        for package in self._get_dependency_packages():
    11041108            if package.name.startswith("org.webkit") and not package.is_installed(self.sdk_branch):
    11051109                Console.error_message("Flatpak package %s not installed. Please update your SDK: Tools/Scripts/update-webkit-flatpak", package)
    … …  
    11291133        return 0
    11301134
    1131     def _get_packages(self):
     1135    def _get_dependency_packages(self):
    11321136        arch = platform.machine()
    11331137        self.runtime = FlatpakPackage("org.webkit.Platform", self.sdk_branch,
    … …  
    11391143                                       self.sdk_repo, arch))
    11401144
    1141         self.flathub_repo = self.repos.add(FlatpakRepo("flathub", url="https://dl.flathub.org/repo/",
    1142                                                        repo_file="https://dl.flathub.org/repo/flathub.flatpakrepo"))
    1143 
    11441145        fdo_branch = "20.08"
    11451146        extensions = ("rust-stable", "llvm11")
    … …  
    11541155            return
    11551156        Console.message("Installing %s dependencies in %s", self.build_type, self.flatpak_build_path)
    1156         for package in self._get_packages():
     1157        for package in self._get_dependency_packages():
    11571158            if not package.is_installed(self.sdk_branch):
    11581159                package.install()
Note: See TracChangeset for help on using the changeset viewer.