Changeset 279984 in webkit


Ignore:
Timestamp:
Jul 16, 2021, 7:25:26 AM (4 years ago)
Author:
Angelos Oikonomopoulos
Message:

Bundle libraries for remote execution in run-jsc-benchmarks
​https://bugs.webkit.org/show_bug.cgi?id=227579

Reviewed by Carlos Alberto Lopez Perez.

Similarly to run-jsc-stress-tests, introduce a bundling step that
ships all library dependencies to the remote system when using
run-jsc-benchmarks --remote.

This patch factors out the code to

  • lookup the ELF interpreter/libraries and to
  • strip the rpath and create a wrapper script

from generate-bundle and places it webkitpy/binary_bundling.

It also introduces a simpler script that only bundles a single
binary (bundle-binary) and switches run-jsc-benchmarks and
run-jsc-stress-tests to use it.

It also updates run-jsc-benchmark to propagate any environment
variables intended for consumption by the JSC binary.

  • Scripts/generate-bundle: Factor out reusable bundling code.
  • Scripts/run-jsc-benchmarks: Do bundling and variable propagation.
  • Scripts/run-jsc-stress-tests: Switch to bundle-binary.
  • Scripts/webkitpy/binary_bundling/init.py: Added.
  • Scripts/webkitpy/binary_bundling/bundle.py: Added.

(BinaryBundler):
(BinaryBundler.init):
(BinaryBundler.destination_dir):
(BinaryBundler.copy_and_remove_rpath):
(BinaryBundler.generate_wrapper_script):

  • Scripts/webkitpy/binary_bundling/ldd.py: Added.

(SharedObjectResolver):
(SharedObjectResolver.init):
(SharedObjectResolver._run_cmd_and_get_output):
(SharedObjectResolver._get_interpreter_objname):
(SharedObjectResolver._get_libs_and_interpreter):
(SharedObjectResolver._ldd_recursive_get_libs_and_interpreter):
(SharedObjectResolver.get_libs_and_interpreter):

  • Scripts/bundle-binary: Added.
  • Scripts/generate-bundle:
  • Scripts/run-jsc-benchmarks:
  • Scripts/run-jsc-stress-tests:
  • Scripts/webkitpy/binary_bundling/init.py: Added.
  • Scripts/webkitpy/binary_bundling/bundle.py: Added.

(BinaryBundler):
(BinaryBundler.init):
(BinaryBundler.destination_dir):
(BinaryBundler.copy_and_remove_rpath):
(BinaryBundler.generate_wrapper_script):

  • Scripts/webkitpy/binary_bundling/ldd.py: Added.

(SharedObjectResolver):
(SharedObjectResolver.init):
(SharedObjectResolver._run_cmd_and_get_output):
(SharedObjectResolver._get_interpreter_objname):
(SharedObjectResolver._get_libs_and_interpreter):
(SharedObjectResolver._ldd_recursive_get_libs_and_interpreter):
(SharedObjectResolver.get_libs_and_interpreter):

Location:
trunk/Tools
Files:
5 added
4 edited

Legend:

Unmodified
Added
Removed

  • trunk/Tools/ChangeLog

    r279982 r279984  
     12021-07-16  Angelos Oikonomopoulos  <angelos@igalia.com>
     2
     3        Bundle libraries for remote execution in run-jsc-benchmarks
     4        https://bugs.webkit.org/show_bug.cgi?id=227579
     5
     6        Reviewed by Carlos Alberto Lopez Perez.
     7
     8        Similarly to run-jsc-stress-tests, introduce a bundling step that
     9        ships all library dependencies to the remote system when using
     10        run-jsc-benchmarks --remote.
     11
     12        This patch factors out the code to
     13        - lookup the ELF interpreter/libraries and to
     14        - strip the rpath and create a wrapper script
     15        from generate-bundle and places it webkitpy/binary_bundling.
     16
     17        It also introduces a simpler script that only bundles a single
     18        binary (bundle-binary) and switches run-jsc-benchmarks and
     19        run-jsc-stress-tests to use it.
     20
     21        It also updates run-jsc-benchmark to propagate any environment
     22        variables intended for consumption by the JSC binary.
     23
     24        * Scripts/generate-bundle: Factor out reusable bundling code.
     25        * Scripts/run-jsc-benchmarks: Do bundling and variable propagation.
     26        * Scripts/run-jsc-stress-tests: Switch to bundle-binary.
     27        * Scripts/webkitpy/binary_bundling/__init__.py: Added.
     28        * Scripts/webkitpy/binary_bundling/bundle.py: Added.
     29        (BinaryBundler):
     30        (BinaryBundler.__init__):
     31        (BinaryBundler.destination_dir):
     32        (BinaryBundler.copy_and_remove_rpath):
     33        (BinaryBundler.generate_wrapper_script):
     34        * Scripts/webkitpy/binary_bundling/ldd.py: Added.
     35        (SharedObjectResolver):
     36        (SharedObjectResolver.__init__):
     37        (SharedObjectResolver._run_cmd_and_get_output):
     38        (SharedObjectResolver._get_interpreter_objname):
     39        (SharedObjectResolver._get_libs_and_interpreter):
     40        (SharedObjectResolver._ldd_recursive_get_libs_and_interpreter):
     41        (SharedObjectResolver.get_libs_and_interpreter):
     42
     43        * Scripts/bundle-binary: Added.
     44        * Scripts/generate-bundle:
     45        * Scripts/run-jsc-benchmarks:
     46        * Scripts/run-jsc-stress-tests:
     47        * Scripts/webkitpy/binary_bundling/__init__.py: Added.
     48        * Scripts/webkitpy/binary_bundling/bundle.py: Added.
     49        (BinaryBundler):
     50        (BinaryBundler.__init__):
     51        (BinaryBundler.destination_dir):
     52        (BinaryBundler.copy_and_remove_rpath):
     53        (BinaryBundler.generate_wrapper_script):
     54        * Scripts/webkitpy/binary_bundling/ldd.py: Added.
     55        (SharedObjectResolver):
     56        (SharedObjectResolver.__init__):
     57        (SharedObjectResolver._run_cmd_and_get_output):
     58        (SharedObjectResolver._get_interpreter_objname):
     59        (SharedObjectResolver._get_libs_and_interpreter):
     60        (SharedObjectResolver._ldd_recursive_get_libs_and_interpreter):
     61        (SharedObjectResolver.get_libs_and_interpreter):
     62
    1632021-07-16  Philippe Normand  <pnormand@igalia.com>
    264

  • trunk/Tools/Scripts/generate-bundle

    r270828 r279984  
    4141sys.path.insert(0, os.path.join(top_level_directory, 'Tools', 'flatpak'))
    4242sys.path.insert(0, os.path.join(top_level_directory, 'Tools', 'jhbuild'))
     43sys.path.insert(0, os.path.join(top_level_directory, 'Tools', 'Scripts', 'webkitpy'))
    4344import jhbuildutils
    4445import flatpakutils
    45 
     46from binary_bundling.ldd import SharedObjectResolver
     47from binary_bundling.bundle import BinaryBundler
    4648
    4749INSTALL_DEPS_SCRIPT_TEMPLATE = """\
    … …  
    128130        self._buildername = builder_name
    129131        self._syslibs = syslibs
    130         self._ldd = ldd
     132        self._shared_object_resolver = SharedObjectResolver(ldd)
    131133        self._should_strip_objects = should_strip_objects
    132134        self._compression_type = compression_type
    … …  
    152154        return tempfile.mkdtemp()
    153155
     156
    154157    def _run_cmd_and_get_output(self, command):
    155         command_process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, encoding='utf-8')
     158        _log.debug("EXEC %s" % command)
     159        command_process = subprocess.Popen(command,
     160                                           stdout=subprocess.PIPE,
     161                                           stderr=subprocess.STDOUT,
     162                                           encoding='utf-8')
    156163        stdout, stderr = command_process.communicate()
    157164        return command_process.returncode, stdout, stderr
    158 
    159 
    160     def _get_interpreter_objname(self, object):
    161         # Note: we use patchelf to get the object name (not the path!)
    162         # of the interpreter because this works regardless of the
    163         # architecture of the ELF file.
    164         retcode, stdout, stderr = self._run_cmd_and_get_output(['patchelf', '--print-interpreter', object])
    165         if retcode != 0:
    166             _log.debug("patchelf stdout:\n%s\nPatchelf stderr:\n%s" % (stdout, stderr))
    167             if 'cannot find section' in stdout:
    168                 # This is fine; we only expect an interpreter in the main binary.
    169                 return None
    170             raise RuntimeError('The patchelf command returned non-zero status for object %s' % object)
    171         interpreter_path = PurePath(stdout.strip())
    172         return interpreter_path.name
    173 
    174     def _get_libs_and_interpreter(self, object):
    175         interpreter = None
    176         retcode, stdout, stderr = self._run_cmd_and_get_output([self._ldd, object])
    177         _log.debug("ldd stdout:\n%s" % stdout)
    178         if retcode != 0:
    179             raise RuntimeError('The %s command returned non-zero status for object %s' % (self._ldd, object))
    180         libs = []
    181         for line in stdout.splitlines():
    182             line = line.strip()
    183             if '=>' in line:
    184                 line = line.split('=>')[1].strip()
    185                 if 'not found' in line:
    186                     raise RuntimeError('ldd can not resolve all dependencies for object %s.' % object)
    187                 line = line.split(' ')[0].strip()
    188                 if os.path.isfile(line):
    189                     libs.append(line)
    190             else:
    191                 line = line.split(' ')[0].strip()
    192                 if os.path.isfile(line):
    193                     interpreter = line
    194         if interpreter is None:
    195             # This is the case for non-native binaries. For those, we
    196             # can use a cross-ldd (xldd), but then the interpreter
    197             # looks like any other shared object in the output of
    198             # ldd. Try to identify it by looking at the object name
    199             # from the interpreter string.
    200             interpreter_objname = self._get_interpreter_objname(object)
    201             for lib in libs:
    202                 if PurePath(lib).name == interpreter_objname:
    203                     interpreter = lib
    204                     break
    205             # If we found an interpreter, remove it from the libs.
    206             libs = [lib for lib in libs if lib != interpreter]
    207         return libs, interpreter
    208 
    209 
    210     def _ldd_recursive_get_libs_and_interpreter(self, object, already_checked_libs = []):
    211         libs, interpreter = self._get_libs_and_interpreter(object)
    212         if libs:
    213             for lib in libs:
    214                 if lib in already_checked_libs:
    215                     continue
    216                 # avoid recursion loops (libfreetype.so.6 <-> libharfbuzz.so.0)
    217                 already_checked_libs.append(lib)
    218                 sub_libs, sub_interpreter = self._ldd_recursive_get_libs_and_interpreter(lib, already_checked_libs)
    219                 libs.extend(sub_libs)
    220                 if sub_interpreter and interpreter and sub_interpreter != interpreter:
    221                     raise RuntimeError('library %s has interpreter %s but object %s has interpreter %s' % (lib, sub_interpreter, object, interpreter))
    222         return list(set(libs)), interpreter
    223 
    224165
    225166    def _get_osprettyname(self):
    … …  
    259200
    260201    def _generate_wrapper_script(self, interpreter, binary_to_wrap):
    261         if not os.path.isfile(os.path.join(self._tmpdir, 'bin', binary_to_wrap)):
    262             raise RuntimeError('Can not find binary to wrap for %s' % binary_to_wrap)
     202        variables = dict()
     203        mydir = self._bundler.VAR_MYDIR
     204        if os.path.isdir(os.path.join(self._bundler.destination_dir(), 'gio')):
     205            gio_var = 'GIO_MODULE_DIR' if self._syslibs == 'bundle-all' else 'GIO_EXTRA_MODULES'
     206            variables[gio_var] = "${%s}/gio" % mydir
     207
     208        if os.path.isdir(os.path.join(self._bundler.destination_dir(), 'gst')):
     209            gst_var = 'GST_PLUGIN_SYSTEM_PATH_1_0' if self._syslibs == 'bundle-all' else 'GST_PLUGIN_PATH_1_0'
     210            variables[gst_var] = "${%s}/gst" % mydir
     211            variables['GST_REGISTRY_1_0'] = "${%s}/gst/gstreamer-1.0.registry" % mydir
     212        if binary_to_wrap != "jsc":
     213            variables['WEBKIT_EXEC_PATH'] = "${%s}/bin" % mydir
     214            variables['WEBKIT_INJECTED_BUNDLE_PATH'] = "${%s}/lib" % mydir
     215        if self._syslibs == 'bundle-all':
     216            if binary_to_wrap != "jsc":
     217                for var in ['WEB_PROCESS_CMD_PREFIX',
     218                            'PLUGIN_PROCESS_CMD_PREFIX',
     219                            'NETWORK_PROCESS_CMD_PREFIX',
     220                            'GPU_PROCESS_CMD_PREFIX']:
     221                    variables[var] = "${%s}" % self._bundler.VAR_INTERPRETER
     222        else:
     223            interpreter = None
     224        self._bundler.generate_wrapper_script(interpreter, binary_to_wrap, variables)
    263225        self._wrapper_scripts.append(binary_to_wrap)
    264         _log.info('Generate wrapper script %s' % binary_to_wrap)
    265         script_file = os.path.join(self._tmpdir, binary_to_wrap)
    266 
    267         with open(script_file, 'w') as script_handle:
    268             script_handle.write('#!/bin/sh\n')
    269             script_handle.write('MYDIR="$(dirname $(readlink -f $0))"\n')
    270             script_handle.write('export LD_LIBRARY_PATH="${MYDIR}/lib"\n')
    271             if os.path.isdir(os.path.join(self._tmpdir, 'gio')):
    272                 gio_var = 'GIO_MODULE_DIR' if self._syslibs == 'bundle-all' else 'GIO_EXTRA_MODULES'
    273                 script_handle.write('export %s="${MYDIR}/gio"\n' % gio_var)
    274             if os.path.isdir(os.path.join(self._tmpdir, 'gst')):
    275                 gst_var = 'GST_PLUGIN_SYSTEM_PATH_1_0' if self._syslibs == 'bundle-all' else 'GST_PLUGIN_PATH_1_0'
    276                 script_handle.write('export %s="${MYDIR}/gst"\n' % gst_var)
    277                 script_handle.write('export GST_REGISTRY_1_0="${MYDIR}/gst/gstreamer-1.0.registry"\n')
    278             if binary_to_wrap != "jsc":
    279                 script_handle.write('export WEBKIT_EXEC_PATH="${MYDIR}/bin"\n')
    280                 script_handle.write('export WEBKIT_INJECTED_BUNDLE_PATH="${MYDIR}/lib"\n')
    281             if self._syslibs == 'bundle-all':
    282                 script_handle.write('INTERPRETER="${MYDIR}/lib/%s"\n' % os.path.basename(interpreter))
    283                 if binary_to_wrap != "jsc":
    284                     script_handle.write('export WEB_PROCESS_CMD_PREFIX="${INTERPRETER}"\n')
    285                     script_handle.write('export PLUGIN_PROCESS_CMD_PREFIX="${INTERPRETER}"\n')
    286                     script_handle.write('export NETWORK_PROCESS_CMD_PREFIX="${INTERPRETER}"\n')
    287                     script_handle.write('export GPU_PROCESS_CMD_PREFIX="${INTERPRETER}"\n')
    288                 script_handle.write('exec "${INTERPRETER}" "${MYDIR}/bin/%s" "$@"\n' % binary_to_wrap)
    289             else:
    290                 script_handle.write('exec "${MYDIR}/bin/%s" "$@"\n' % binary_to_wrap)
    291         os.chmod(script_file, 0o755)
    292226
    293227    def _generate_install_deps_script(self, system_packages_needed):
    … …  
    313247        os.chmod(installdeps_file, 0o755)
    314248
    315 
    316     def _copy_and_remove_rpath(self, orig_file, type='bin', destination_dir=None):
    317         if not destination_dir:
    318             dir_suffix = 'lib' if type == 'interpreter' else type
    319             destination_dir = os.path.join(self._tmpdir, dir_suffix)
    320         if not os.path.isdir(destination_dir):
    321             os.makedirs(destination_dir)
    322 
    323         if not os.path.isfile(orig_file):
    324             raise ValueError('Can not find file %s' % orig_file)
    325 
    326         _log.info('Add to bundle [%s]: %s' % (type, orig_file))
    327         shutil.copy(orig_file, destination_dir)
    328 
    329         if shutil.which('patchelf'):
    330             patch_elf_command = ['patchelf', '--remove-rpath', os.path.join(destination_dir, os.path.basename(orig_file))]
    331             if subprocess.call(patch_elf_command) != 0:
    332                 _log.warning('The patchelf command returned non-zero status')
    333         else:
    334                 _log.warning('patchelf not found. Not modifying rpath')
    335 
    336         if self._should_strip_objects:
    337             if shutil.which('strip'):
    338                 strip_command = ['strip', '--strip-unneeded', os.path.join(destination_dir, os.path.basename(orig_file))]
    339                 if subprocess.call(strip_command) != 0:
    340                     _log.warning('The strip command returned non-zero status')
    341             else:
    342                 _log.warning('strip not found. Not stripping object')
    343 
    344249    def _remove_tempdir(self):
    345250        if not self._tmpdir:
    … …  
    350255    def create(self):
    351256        self._tmpdir = self._create_tempdir(self._buildpath)
     257        self._bundler = BinaryBundler(self._tmpdir, self._should_strip_objects)
    352258
    353259        if os.path.isfile(self._bundle_file_path):
    … …  
    457363        provided_by_system_package = None
    458364        if self._syslibs == 'bundle-all':
    459             self._copy_and_remove_rpath(object, type=object_type)
     365            self._bundler.copy_and_remove_rpath(object, type=object_type)
    460366        else:
    461367            provided_by_system_package = self._get_system_package_name(object)
    462368            if not provided_by_system_package:
    463                 self._copy_and_remove_rpath(object, type=object_type)
     369                self._bundler.copy_and_remove_rpath(object, type=object_type)
    464370        return provided_by_system_package
    465371
    … …  
    507413                    system_packages_needed.add(system_package)
    508414            elif object.endswith('.so'):
    509                 self._copy_and_remove_rpath(object, type='lib')
     415                self._bundler.copy_and_remove_rpath(object, type='lib')
    510416            else:
    511                 self._copy_and_remove_rpath(object, type='bin')
     417                self._bundler.copy_and_remove_rpath(object, type='bin')
    512418            # There is no need to examine the libraries linked with objects coming from a system package,
    513419            # because system packages already declare dependencies between them.
    … …  
    515421            # and everything will be examined and bundled as we don't account for system packages in that case.
    516422            if not system_package:
    517                 libraries, interpreter = self._ldd_recursive_get_libs_and_interpreter(object)
     423                libraries, interpreter = self._shared_object_resolver.get_libs_and_interpreter(object)
    518424                if interpreter is None:
    519425                    raise RuntimeError("Could not determine interpreter for binary %s" % object)
    520426                if copied_interpreter is None:
    521427                    if self._syslibs == 'bundle-all':
    522                         self._copy_and_remove_rpath(interpreter, type='interpreter')
     428                        self._bundler.copy_and_remove_rpath(interpreter, type='interpreter')
    523429                    copied_interpreter = interpreter
    524430                elif copied_interpreter != interpreter:
    … …  
    660566
    661567    handler = LogHandler(sys.stdout)
    662     logger = logging.getLogger(__name__)
     568    logger = logging.getLogger()
    663569    logger.addHandler(handler)
    664570    logger.setLevel(log_level)

  • trunk/Tools/Scripts/run-jsc-benchmarks

    r276765 r279984  
    238238$includeBigIntBench = false
    239239$includePrivateFieldsBench = false
     240$ldd=nil
    240241$measureGC=false
    241242$benchmarkPattern=nil
    … …  
    14141415    @@extraEnvSet[key] = true
    14151416  end
    1416  
     1417
     1418  def copyIntoBenchPathBundle(basename, outputdir)
     1419    bundle_binary = (SCRIPT_PATH.dirname + 'bundle-binary').realpath
     1420    cmd = Shellwords.join([
     1421                            bundle_binary,
     1422                            '--dest-dir', outputdir,
     1423                            @path
     1424                          ])
     1425    $stderr.puts ">> #{cmd}" if $verbosity>=2
     1426    raise unless system(cmd)
     1427    @path = "#{basename}/jsc"
     1428  end
     1429
    14171430  def copyIntoBenchPath
    14181431    raise unless canCopyIntoBenchPath
    14191432    basename, filename = Benchfile.uniqueFilename("vm")
    14201433    raise unless Dir.mkdir(filename)
     1434    if not $remoteHosts.empty? and `uname` == 'Linux'
     1435      copyIntoBenchPathBundle(basename, Shellwords.shellescape(filename.to_s))
     1436      return
     1437    end
     1438
    14211439    @libPath.each {
    14221440      | libPathPart |
    … …  
    14251443      raise unless system(cmd)
    14261444    }
    1427     @path = "#{basename}/#{@relativeBinPath}"
     1445    cmd = "cp -a #{@path} #{Shellwords.shellescape(filename.to_s)}"
     1446    $stderr.puts ">> #{cmd}" if $verbosity>=2
     1447    raise unless system(cmd)
     1448    @path = "#{basename}/jsc"
    14281449    @libPath = [basename]
    14291450  end
    … …  
    23902411  result + curLine + "\n"
    23912412end
    2392  
     2413
     2414def exportJSCEnvironmentVariables
     2415  buf = ""
     2416  ENV.each {
     2417    | name, value |
     2418    next unless name.start_with?("JSC_")
     2419    buf << "export #{name}=#{Shellwords.shellescape(value)}\n"
     2420  }
     2421  buf
     2422end
     2423
    23932424def runAndGetResults
    23942425  results = nil
    … …  
    30563087    when '--config'
    30573088      $configPath = Pathname.new(arg)
     3089    when '--ldd'
     3090      $ldd = arg
    30583091    when '--help'
    30593092      usage
    … …  
    36123645    File.open("#{BENCH_DATA_PATH}/runscript", "w") {
    36133646      | file |
     3647      file.puts exportJSCEnvironmentVariables
    36143648      file.puts "echo -e \"HOSTNAME:\\c\""
    36153649      file.puts "hostname"

  • trunk/Tools/Scripts/run-jsc-stress-tests

    r279916 r279984  
    19201920
    19211921            if $remote and $hostOS == "linux"
    1922                 generate_bundle = (Pathname.new(THIS_SCRIPT_PATH).dirname + 'generate-bundle').realpath
     1922                bundle_binary = (Pathname.new(THIS_SCRIPT_PATH).dirname + 'bundle-binary').realpath
    19231923                Dir.mktmpdir {
    19241924                    | tmpdir |
    … …  
    19271927                    # (it's the only zip file there).
    19281928                    cmdline = [
    1929                         generate_bundle.to_s,
    1930                         "--platform=gtk",
    1931                         "--bundle=jsc",
    1932                         "--syslibs=bundle-all",
    1933                         "--no-strip",
    1934                         "--compression=tar.xz",
    1935                         ($buildType == "release") ? "--release" : "--debug",
    1936                         "--destination=#{tmpdir}"
     1929                        bundle_binary.to_s,
     1930                        "--dest-dir=#{$jscPath.dirname}",
     1931                        "--log-level=debug",
     1932                        $jscPath.to_s
    19371933                    ]
    19381934                    if not $ldd.nil?
    … …  
    19401936                    end
    19411937                    mysys(cmdline)
    1942                     archives = Dir.glob("#{tmpdir}/*.tar.xz")
    1943                     if archives.size != 1
    1944                         raise "Expected exactly one entry in tmpdir, not #{archives}"
    1945                     end
    1946                     # Note: we overwrite 'jsc'. This obviously conflicts with
    1947                     # !copyVM but, then gain, so does $remote.
    1948                     mysys(["tar",
    1949                            "-C",
    1950                            $jscPath.dirname.to_s,
    1951                            "-xf",
    1952                            archives[0]])
    19531938                }
    19541939            end
Note: See TracChangeset for help on using the changeset viewer.