Changeset 88995 in webkit


Ignore:
Timestamp:
Jun 15, 2011 7:47:49 PM (13 years ago)
Author:
dpranke@chromium.org
Message:

2011-06-15 Dirk Pranke <​dpranke@chromium.org>

Reviewed by Tony Chang.

nrwt: should clean up stale server processes from a previous run
​https://bugs.webkit.org/show_bug.cgi?id=62615

This change fixes cases where NRWT might start a server and then
exit, leaving the server still running. Now NRWT should detect
this case and clean up on the next run, without shutting down
any processes it didn't start.

This change also fixes a bug in executive.kill_process on UNIX
where zombies weren't being handled properly.

  • Scripts/webkitpy/common/system/executive.py:
  • Scripts/webkitpy/layout_tests/port/apache_http_server.py:
  • Scripts/webkitpy/layout_tests/port/base.py:
  • Scripts/webkitpy/layout_tests/port/http_server.py:
  • Scripts/webkitpy/layout_tests/port/http_server_base.py:
  • Scripts/webkitpy/layout_tests/port/port_testcase.py:
  • Scripts/webkitpy/layout_tests/port/websocket_server.py:
Location:
trunk/Tools
Files:
1 added
9 edited

Legend:

Unmodified
Added
Removed

  • trunk/Tools/ChangeLog

    r88994 r88995  
     12011-06-15  Dirk Pranke  <dpranke@chromium.org>
     2
     3        Reviewed by Tony Chang.
     4
     5        nrwt: should clean up stale server processes from a previous run
     6        https://bugs.webkit.org/show_bug.cgi?id=62615
     7
     8        This change fixes cases where NRWT might start a server and then
     9        exit, leaving the server still running. Now NRWT should detect
     10        this case and clean up on the next run, without shutting down
     11        any processes it didn't start.
     12
     13        This change also fixes a bug in executive.kill_process on UNIX
     14        where zombies weren't being handled properly.
     15
     16        * Scripts/webkitpy/common/system/executive.py:
     17        * Scripts/webkitpy/layout_tests/port/apache_http_server.py:
     18        * Scripts/webkitpy/layout_tests/port/base.py:
     19        * Scripts/webkitpy/layout_tests/port/http_server.py:
     20        * Scripts/webkitpy/layout_tests/port/http_server_base.py:
     21        * Scripts/webkitpy/layout_tests/port/port_testcase.py:
     22        * Scripts/webkitpy/layout_tests/port/websocket_server.py:
     23
     242011-06-15  Dirk Pranke  <dpranke@chromium.org>
     25
     26        Reviewed by Tony Chang.
     27
     28        nrwt: fix http, websocket server startup, shutdown
     29        https://bugs.webkit.org/show_bug.cgi?id=62180
     30
     31        This change refactors all of the server-related code to
     32        have consistent semantics. It pushes the start()/stop()
     33        logic into the base class and adds four much smaller routines
     34        for specialization:
     35        _prepare_config() - for optionally writing config files etc.
     36            to disk prior to starting the server
     37        _remove_stale_log_files()
     38        _spawn_process() - the actual launch of the subprocess
     39        _cleanup_after_stop() - and custom cleanup code
     40
     41        Prior to this change the three server implementations were
     42        fairly inconsistent, leading to lots of weirdness on the bots
     43        with servers being left around, etc.
     44
     45        * Scripts/webkitpy/layout_tests/port/apache_http_server.py:
     46        * Scripts/webkitpy/layout_tests/port/http_server.py:
     47        * Scripts/webkitpy/layout_tests/port/http_server_base.py:
     48        * Scripts/webkitpy/layout_tests/port/websocket_server.py:
     49
     502011-06-15  Dirk Pranke  <dpranke@chromium.org>
     51
     52        Reviewed by Tony Chang.
     53
     54        webkitpy: clean up code prior to functional changes for server startup/shutdown
     55        https://bugs.webkit.org/show_bug.cgi?id=62256
     56
     57        This patch does a bunch of minor cleanup of the code, but there
     58        should be no functional changes except that all of the modules
     59        will now use the same exception type. Other changes include
     60        removing unused code paths and functions, pushing 'name' and
     61        'pid_file' into the base class of the server objects, and using
     62        the Filesystem object instead of the codecs, tempfile, and
     63        shutils modules.
     64
     65        * Scripts/webkitpy/layout_tests/port/apache_http_server.py:
     66        * Scripts/webkitpy/layout_tests/port/http_server.py:
     67        * Scripts/webkitpy/layout_tests/port/http_server_base.py:
     68        * Scripts/webkitpy/layout_tests/port/websocket_server.py:
     69
     702011-06-15  Dirk Pranke  <dpranke@chromium.org>
     71
     72        Reviewed by Tony Chang.
     73
     74        webkitpy: add integration tests for new-run-webkit-httpd, stop calling shut_down_http_server
     75        https://bugs.webkit.org/show_bug.cgi?id=62251
     76
     77        shut_down_http_server() was a total hack that was only used by
     78        new-run-webkit-httpd, so I've moved the code there and switched
     79        to using executive.kill_process() for the common case. The
     80        method itself will be removed in the patch on bug 59993.
     81
     82        * Scripts/new-run-webkit-httpd:
     83        * Scripts/webkitpy/layout_tests/port/apache_http_server.py:
     84        * Scripts/webkitpy/layout_tests/port/http_server.py:
     85        * Scripts/webkitpy/layout_tests/port/http_server_integrationtest.py: Added.
     86        * Scripts/webkitpy/layout_tests/port/websocket_server.py:
     87
    1882011-06-15  Dirk Pranke  <dpranke@chromium.org>
    289

  • trunk/Tools/Scripts/new-run-webkit-httpd

    r88680 r88995  
    7171            httpd.start()
    7272        else:
    73             httpd.stop(force=True)
    74 
     73            # FIXME: We use _shut_down_http_server() instead of stop_http_server()
     74            # because stop() only works if start() was called previously in the
     75            # same process. _shut_down() is too coarse and may kill unrelated
     76            # web servers, so this is a hack. We should change the interface so we
     77            # can rely on pid files or something.
     78            port_obj._shut_down_http_server(None)
    7579
    7680def main():

  • trunk/Tools/Scripts/webkitpy/common/system/executive.py

    r88287 r88995  
    228228                retries_left -= 1
    229229                os.kill(pid, signal.SIGKILL)
     230                _ = os.waitpid(pid, os.WNOHANG)
    230231            except OSError, e:
    231232                if e.errno == errno.EAGAIN:

  • trunk/Tools/Scripts/webkitpy/layout_tests/port/apache_http_server.py

    r88680 r88995  
    11#!/usr/bin/env python
    2 # Copyright (C) 2010 Google Inc. All rights reserved.
     2# Copyright (C) 2011 Google Inc. All rights reserved.
    33#
    44# Redistribution and use in source and binary forms, with or without
    … …  
    3131
    3232
    33 from __future__ import with_statement
    34 
    35 import codecs
    3633import logging
    37 import optparse
    3834import os
    3935import re
    40 import subprocess
    4136import sys
    4237
    43 import http_server_base
     38from webkitpy.layout_tests.port import http_server_base
    4439
    4540_log = logging.getLogger("webkitpy.layout_tests.port.apache_http_server")
    … …  
    5449        """
    5550        http_server_base.HttpServerBase.__init__(self, port_obj)
     51        self._name = 'apache'
     52        self._mappings = [{'port': 8000},
     53                          {'port': 8080},
     54                          {'port': 8081},
     55                          {'port': 8443, 'sslcert': True}]
    5656        self._output_dir = output_dir
    57         self._httpd_proc = None
     57        self._name = 'apache'
    5858        port_obj.maybe_make_directory(output_dir)
    59 
    60         self.mappings = [{'port': 8000},
    61                          {'port': 8080},
    62                          {'port': 8081},
    63                          {'port': 8443, 'sslcert': True}]
    64 
    65         # The upstream .conf file assumed the existence of /tmp/WebKit for
    66         # placing apache files like the lock file there.
    67         self._runtime_path = os.path.join("/tmp", "WebKit")
    68         port_obj.maybe_make_directory(self._runtime_path)
    6959
    7060        # The PID returned when Apache is started goes away (due to dropping
    7161        # privileges?). The proper controlling PID is written to a file in the
    7262        # apache runtime directory.
    73         self._pid_file = os.path.join(self._runtime_path, 'httpd.pid')
     63        self._pid_file = self._filesystem.join(self._runtime_path, '%s.pid' % self._name)
    7464
    7565        test_dir = self._port_obj.layout_tests_dir()
    … …  
    158148        httpd_config = self._port_obj._path_to_apache_config_file()
    159149        httpd_config_copy = os.path.join(output_dir, "httpd.conf")
    160         # httpd.conf is always utf-8 according to http://archive.apache.org/gnats/10125
    161         with codecs.open(httpd_config, "r", "utf-8") as httpd_config_file:
    162             httpd_conf = httpd_config_file.read()
     150        httpd_conf = self._filesystem.read_text_file(httpd_config)
    163151        if self._is_cygwin():
    164152            # This is a gross hack, but it lets us use the upstream .conf file
    … …  
    173161                'ServerRoot "%s"' % self._get_cygwin_path(cygusr))
    174162
    175         with codecs.open(httpd_config_copy, "w", "utf-8") as file:
    176             file.write(httpd_conf)
     163        self._filesystem.write_text_file(httpd_config_copy, httpd_conf)
    177164
    178165        if self._is_cygwin():
    … …  
    180167        return httpd_config_copy
    181168
    182     def _get_virtual_host_config(self, document_root, port, ssl=False):
    183         """Returns a <VirtualHost> directive block for an httpd.conf file.
    184         It will listen to 127.0.0.1 on each of the given port.
    185         """
    186         return '\n'.join(('<VirtualHost 127.0.0.1:%s>' % port,
    187                           'DocumentRoot "%s"' % document_root,
    188                           ssl and 'SSLEngine On' or '',
    189                           '</VirtualHost>', ''))
    190 
    191     def _start_httpd_process(self):
    192         """Starts the httpd process and returns whether there were errors."""
     169    def _spawn_process(self):
    193170        # Use shell=True because we join the arguments into a string for
    194171        # the sake of Window/Cygwin and it needs quoting that breaks
    … …  
    197174        # shell=True is a trail of tears.
    198175        # Note: Not thread safe: http://bugs.python.org/issue2320
    199         _log.debug('Starting http server, cmd="%s"' % str(self._start_cmd))
    200         self._httpd_proc = subprocess.Popen(self._start_cmd,
    201                                             stderr=subprocess.PIPE,
    202             shell=True)
    203         err = self._httpd_proc.stderr.read()
     176        _log.debug('Starting %s server, cmd="%s"' % (self._name, str(self._start_cmd)))
     177        process = self._executive.popen(self._start_cmd, shell=True, stderr=self._executive.PIPE)
     178        err = process.stderr.read()
    204179        if len(err):
    205180            _log.debug(err)
    206             return False
    207         return True
    208 
    209     def start(self):
    210         """Starts the apache http server."""
    211         # Stop any currently running servers.
    212         self.stop()
    213 
    214         _log.debug("Starting apache http server")
    215         server_started = self.wait_for_action(self._start_httpd_process)
    216         if server_started:
    217             _log.debug("Apache started. Testing ports")
    218             server_started = self.wait_for_action(
    219                 self.is_server_running_on_all_ports)
    220 
    221         if server_started:
    222             _log.debug("Server successfully started")
    223         else:
    224             raise Exception('Failed to start http server')
    225 
    226     def stop(self):
    227         """Stops the apache http server."""
    228         _log.debug("Shutting down any running http servers")
    229         httpd_pid = None
    230         if os.path.exists(self._pid_file):
    231             httpd_pid = int(open(self._pid_file).readline())
    232         # FIXME: We shouldn't be calling a protected method of _port_obj!
    233         self._port_obj._shut_down_http_server(httpd_pid)
     181            self._executive.kill_process(process.pid)
     182            process.wait()
     183            raise ServerError('Failed to start %s: %s' % (self._name, err))
     184        return process

  • trunk/Tools/Scripts/webkitpy/layout_tests/port/base.py

    r88986 r88995  
    647647
    648648        Ports can stub this out if they don't need a web server to be running."""
     649        assert not self._http_server, 'Already running an http server.'
     650
    649651        if self.get_option('use_apache'):
    650652            server = apache_http_server.LayoutTestApacheHttpd(self, self.results_directory())
    651653        else:
    652654            server = http_server.Lighttpd(self, self.results_directory())
     655
    653656        server.start()
    654657        self._http_server = server
    … …  
    658661
    659662        Ports can stub this out if they don't need a websocket server to be running."""
     663        assert not self._websocket_server, 'Already running a websocket server.'
     664
    660665        server = websocket_server.PyWebSocket(self, self.results_directory())
    661666        server.start()
    … …  
    676681        if self._http_server:
    677682            self._http_server.stop()
     683            self._http_server = None
    678684
    679685    def stop_websocket_server(self):
    … …  
    681687        if self._websocket_server:
    682688            self._websocket_server.stop()
     689            self._websocket_server = None
    683690
    684691    def release_http_lock(self):

  • trunk/Tools/Scripts/webkitpy/layout_tests/port/http_server.py

    r88680 r88995  
    11#!/usr/bin/env python
    2 # Copyright (C) 2010 Google Inc. All rights reserved.
     2# Copyright (C) 2011 Google Inc. All rights reserved.
    33#
    44# Redistribution and use in source and binary forms, with or without
    … …  
    3030"""A class to help start/stop the lighttpd server used by layout tests."""
    3131
    32 from __future__ import with_statement
     32import logging
     33import os
     34import sys
     35import time
    3336
    34 import codecs
    35 import logging
    36 import optparse
    37 import os
    38 import shutil
    39 import subprocess
    40 import sys
    41 import tempfile
    42 import time
    43 import urllib
    44 
    45 import factory
    46 import http_server_base
     37from webkitpy.layout_tests.port import http_server_base
    4738
    4839_log = logging.getLogger("webkitpy.layout_tests.port.http_server")
    49 
    5040
    5141class Lighttpd(http_server_base.HttpServerBase):
    … …  
    5848        # Webkit tests
    5949        http_server_base.HttpServerBase.__init__(self, port_obj)
     50        self._name = 'lighttpd'
    6051        self._output_dir = output_dir
    61         self._process = None
    6252        self._port = port
    6353        self._root = root
    6454        self._run_background = run_background
    6555        self._layout_tests_dir = layout_tests_dir
     56
     57        self._pid_file = self._filesystem.join(self._runtime_path, '%s.pid' % self._name)
    6658
    6759        if self._port:
    … …  
    7163            self._layout_tests_dir = self._port_obj.layout_tests_dir()
    7264
    73         try:
    74             self._webkit_tests = os.path.join(
    75                 self._layout_tests_dir, 'http', 'tests')
    76             self._js_test_resource = os.path.join(
    77                 self._layout_tests_dir, 'fast', 'js', 'resources')
    78             self._media_resource = os.path.join(
    79                 self._layout_tests_dir, 'media')
    80 
    81         except:
    82             self._webkit_tests = None
    83             self._js_test_resource = None
    84             self._media_resource = None
     65        self._webkit_tests = os.path.join(self._layout_tests_dir, 'http', 'tests')
     66        self._js_test_resource = os.path.join(self._layout_tests_dir, 'fast', 'js', 'resources')
     67        self._media_resource = os.path.join(self._layout_tests_dir, 'media')
    8568
    8669        # Self generated certificate for SSL server (for client cert get
    … …  
    10083                 'sslcert': self._pem_file}])
    10184
    102     def is_running(self):
    103         return self._process != None
    104 
    105     def start(self):
    106         if self.is_running():
    107             raise 'Lighttpd already running'
    108 
     85    def _prepare_config(self):
    10986        base_conf_file = self._port_obj.path_from_webkit_base('Tools',
    11087            'Scripts', 'webkitpy', 'layout_tests', 'port', 'lighttpd.conf')
    … …  
    11693        error_log = os.path.join(self._output_dir, log_file_name)
    11794
    118         # Remove old log files. We only need to keep the last ones.
    119         self.remove_log_files(self._output_dir, "access.log-")
    120         self.remove_log_files(self._output_dir, "error.log-")
    121 
    12295        # Write out the config
    123         with codecs.open(base_conf_file, "r", "utf-8") as file:
    124             base_conf = file.read()
     96        base_conf = self._filesystem.read_text_file(base_conf_file)
    12597
    12698        # FIXME: This should be re-worked so that this block can
    12799        # use with open() instead of a manual file.close() call.
    128         # lighttpd.conf files seem to be UTF-8 without BOM:
    129         # http://redmine.lighttpd.net/issues/992
    130         f = codecs.open(out_conf_file, "w", "utf-8")
     100        f = self._filesystem.open_text_file_for_writing(out_conf_file)
    131101        f.write(base_conf)
    132102
    … …  
    205175                os.makedirs(tmp_module_path)
    206176            lib_file = 'liblightcomp.dylib'
    207             shutil.copyfile(os.path.join(module_path, lib_file),
    208                             os.path.join(tmp_module_path, lib_file))
     177            self._filesystem.copyfile(os.path.join(module_path, lib_file),
     178                                      os.path.join(tmp_module_path, lib_file))
    209179
    210         env = self._port_obj.setup_environ_for_server()
    211         _log.debug('Starting http server, cmd="%s"' % str(start_cmd))
    212         # FIXME: Should use Executive.run_command
    213         self._process = subprocess.Popen(start_cmd, env=env, stdin=subprocess.PIPE)
     180        self._start_cmd = start_cmd
     181        self._env = self._port_obj.setup_environ_for_server()
     182        self._mappings = mappings
    214183
    215         # Wait for server to start.
    216         self.mappings = mappings
    217         server_started = self.wait_for_action(
    218             self.is_server_running_on_all_ports)
     184    def _remove_stale_logs(self):
     185        # Sometimes logs are open in other processes but they should clear eventually.
     186        for log_prefix in ('access.log-', 'error.log-'):
     187            try:
     188                self._remove_log_files(self._output_dir, log_prefix)
     189            except OSError, e:
     190                _log.warning('Failed to remove old %s %s files' % self._name, log_prefix)
    219191
    220         # Our process terminated already
    221         if not server_started or self._process.returncode != None:
    222             raise Exception('Failed to start httpd.')
    223 
    224         _log.debug("Server successfully started")
    225 
    226     # TODO(deanm): Find a nicer way to shutdown cleanly.  Our log files are
    227     # probably not being flushed, etc... why doesn't our python have os.kill ?
    228 
    229     def stop(self, force=False):
    230         if not force and not self.is_running():
    231             return
    232 
    233         httpd_pid = None
    234         if self._process:
    235             httpd_pid = self._process.pid
    236         self._port_obj._shut_down_http_server(httpd_pid)
    237 
    238         if self._process:
    239             # wait() is not threadsafe and can throw OSError due to:
    240             # http://bugs.python.org/issue1731717
    241             self._process.wait()
    242             self._process = None
     192    def _spawn_process(self):
     193        _log.debug('starting %s, cmd="%s"' % (self._name, self._start_cmd))
     194        return self._executive.popen(self._start_cmd, env=self._env, shell=False, stderr=self._executive.PIPE)

  • trunk/Tools/Scripts/webkitpy/layout_tests/port/http_server_base.py

    r88680 r88995  
    11#!/usr/bin/env python
    2 # Copyright (C) 2010 Google Inc. All rights reserved.
     2# Copyright (C) 2011 Google Inc. All rights reserved.
    33#
    44# Redistribution and use in source and binary forms, with or without
    … …  
    2828# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
    2929
    30 """Base class with common routines between the Apache and Lighttpd servers."""
    31 
     30"""Base class with common routines between the Apache, Lighttpd, and websocket servers."""
     31
     32import errno
    3233import logging
    33 import os
     34import socket
     35import sys
     36import tempfile
    3437import time
    35 import urllib
    36 
    37 from webkitpy.common.system import filesystem
    3838
    3939_log = logging.getLogger("webkitpy.layout_tests.port.http_server_base")
    4040
    4141
     42class ServerError(Exception):
     43    pass
     44
     45
    4246class HttpServerBase(object):
     47    """A skeleton class for starting and stopping servers used by the layout tests."""
    4348
    4449    def __init__(self, port_obj):
     50        self._executive = port_obj._executive
     51        self._filesystem = port_obj._filesystem
     52        self._name = '<virtual>'
     53        self._mappings = {}
     54        self._pid_file = None
    4555        self._port_obj = port_obj
    46 
    47     def wait_for_action(self, action):
     56        self._process = None
     57
     58        # We need a non-checkout-dependent place to put lock files, etc. We
     59        # don't use the Python default on the Mac because it defaults to a
     60        # randomly-generated directory under /var/folders and no one would ever
     61        # look there.
     62        tmpdir = tempfile.gettempdir()
     63        if sys.platform == 'darwin':
     64            tmpdir = '/tmp'
     65
     66        self._runtime_path = self._filesystem.join(tmpdir, "WebKit")
     67        port_obj.maybe_make_directory(self._runtime_path)
     68
     69    def start(self):
     70        """Starts the server."""
     71        assert not self._process, '%s server is already running.'
     72
     73        self._remove_stale_processes()
     74        self._remove_stale_logs()
     75        self._prepare_config()
     76        self._check_that_all_ports_are_available()
     77        self._process = self._spawn_process()
     78        _log.debug("%s started. Testing ports" % self._name)
     79        server_started = self._wait_for_action(self._is_server_running_on_all_ports)
     80        if server_started:
     81            _log.debug("%s successfully started" % self._name)
     82        else:
     83            raise ServerError('Failed to start %s server' % self._name)
     84
     85        if self._pid_file:
     86            self._filesystem.write_text_file(self._pid_file, str(self._process.pid))
     87
     88    def stop(self):
     89        _log.debug("Shutting down any running %s servers" % self._name)
     90        pid = None
     91        if self._process:
     92            pid = self._process.pid
     93        elif self._pid_file and self._filesystem.exists(self._pid_file):
     94            pid = int(self._filesystem.read_text_file(self._pid_file))
     95        else:
     96            return
     97
     98        self._port_obj._executive.kill_process(pid)
     99        if self._process:
     100            # wait() is not threadsafe and can throw OSError due to:
     101            # http://bugs.python.org/issue1731717
     102            try:
     103                self._process.wait()
     104            except OSError, e:
     105                if e.errno != errno.ECHILD:
     106                    raise
     107            self._process = None
     108        if self._pid_file and self._filesystem.exists(self._pid_file):
     109            self._filesystem.remove(self._pid_file)
     110
     111        self._cleanup_after_stop()
     112
     113    def _prepare_config(self):
     114        """This routine should be implemented by subclasses to do any sort
     115        of initialization required prior to starting the server that may fail."""
     116        pass
     117
     118    def _remove_stale_processes(self):
     119        """Remove any instances of the server left over from a previous run.
     120        There should be at most one of these because of the global http lock."""
     121        def kill_and_check(name, pid):
     122            _log.warning('Previously started %s is still running as pid %d, trying to kill it.' % (name, pid))
     123            self._executive.kill_process(pid)
     124            return not self._executive.check_running_pid(pid)
     125
     126        if self._pid_file and self._filesystem.exists(self._pid_file):
     127            pid = int(self._filesystem.read_text_file(self._pid_file))
     128            if not self._executive.check_running_pid(pid) or self._wait_for_action(lambda: kill_and_check(self._name, pid)):
     129                self._filesystem.remove(self._pid_file)
     130
     131    def _remove_stale_logs(self):
     132        """This routine should be implemented by subclasses to try and remove logs
     133        left over from a prior run. This routine should log warnings if the
     134        files cannot be deleted, but should not fail unless failure to
     135        delete the logs will actually cause start() to fail."""
     136        pass
     137
     138    def _spawn_process(self):
     139        """This routine mst be implemented by subclasses to actually
     140        launch the subprocess and return the process object. The method
     141        does not need to ensure that the process started correctly (start()
     142        will verify that itself)."""
     143        raise NotImplementedError
     144
     145    def _cleanup_after_stop(self):
     146        """This routine may be implemented in subclasses to clean up any state
     147        as necessary after the server process is stopped."""
     148        pass
     149
     150    def _remove_log_files(self, folder, starts_with):
     151        files = self._filesystem.listdir(folder)
     152        for file in files:
     153            if file.startswith(starts_with):
     154                full_path = self._filesystem.join(folder, file)
     155                self._filesystem.remove(full_path)
     156
     157    def _wait_for_action(self, action):
    48158        """Repeat the action for 20 seconds or until it succeeds. Returns
    49159        whether it succeeded."""
    … …  
    57167        return False
    58168
    59     def is_server_running_on_all_ports(self):
     169    def _is_server_running_on_all_ports(self):
    60170        """Returns whether the server is running on all the desired ports."""
    61         for mapping in self.mappings:
    62             if 'sslcert' in mapping:
    63                 http_suffix = 's'
    64             else:
    65                 http_suffix = ''
    66 
    67             url = 'http%s://127.0.0.1:%d/' % (http_suffix, mapping['port'])
    68 
     171        if not self._executive.check_running_pid(self._process.pid):
     172            _log.debug("Server isn't running at all")
     173            raise ServerError("Server exited")
     174
     175        for mapping in self._mappings:
     176            s = socket.socket()
     177            port = mapping['port']
    69178            try:
    70                 response = urllib.urlopen(url, proxies={})
    71                 _log.debug("Server running at %s" % url)
     179                s.connect(('localhost', port))
     180                _log.debug("Server running on %d" % port)
    72181            except IOError, e:
    73                 _log.debug("Server NOT running at %s: %s" % (url, e))
     182                if e.errno not in (errno.ECONNREFUSED, errno.ECONNRESET):
     183                    raise
     184                _log.debug("Server NOT running on %d: %s" % (port, e))
    74185                return False
    75 
     186            finally:
     187                s.close()
    76188        return True
    77189
    78     def remove_log_files(self, folder, starts_with):
    79         files = os.listdir(folder)
    80         for file in files:
    81             if file.startswith(starts_with):
    82                 full_path = os.path.join(folder, file)
    83                 filesystem.FileSystem().remove(full_path)
     190    def _check_that_all_ports_are_available(self):
     191        for mapping in self._mappings:
     192            s = socket.socket()
     193            port = mapping['port']
     194            try:
     195                s.bind(('localhost', port))
     196            except IOError, e:
     197                if e.errno in (errno.EALREADY, errno.EADDRINUSE):
     198                    raise ServerError('Port %d is already in use.' % port)
     199                else:
     200                    raise
     201            finally:
     202                s.close()

  • trunk/Tools/Scripts/webkitpy/layout_tests/port/port_testcase.py

    r88680 r88995  
    4949class PortTestCase(unittest.TestCase):
    5050    """Tests that all Port implementations must pass."""
    51 
    52     HTTP_PORTS = [8000, 8080, 8443]
    53     WEBSOCKET_PORTS = [8880]
     51    HTTP_PORTS = (8000, 8080, 8443)
     52    WEBSOCKET_PORTS = (8880,)
    5453
    5554    def port_maker(self, platform):
    … …  
    9392        for port in ports:
    9493            try:
    95                 s = socket.socket()
    96                 s.connect((host, port))
     94                test_socket = socket.socket()
     95                test_socket.connect((host, port))
    9796                self.fail()
    9897            except IOError, e:
    9998                self.assertTrue(e.errno in (errno.ECONNREFUSED, errno.ECONNRESET))
    10099            finally:
    101                 s.close()
     100                test_socket.close()
    102101
    103102    def assert_servers_are_up(self, host, ports):
    104103        for port in ports:
    105104            try:
    106                 s = socket.socket()
    107                 s.connect((host, port))
     105                test_socket = socket.socket()
     106                test_socket.connect((host, port))
    108107            except IOError, e:
    109108                self.fail('failed to connect to %s:%d' % (host, port))
    110109            finally:
    111                 s.close()
     110                test_socket.close()
    112111
    113112    def integration_test_http_lock(self):
    … …  
    138137        if not port:
    139138            return
    140 
    141139        self.assert_servers_are_down('localhost', self.HTTP_PORTS)
    142140        port.start_http_server()
    … …  
    149147        if not port:
    150148            return
    151 
    152149        # Test that if a port isn't available, the call fails.
    153150        for port_number in self.HTTP_PORTS:
    154             s = socket.socket()
     151            test_socket = socket.socket()
    155152            try:
    156153                try:
    157                     s.bind(('localhost', port_number))
     154                    test_socket.bind(('localhost', port_number))
    158155                except socket.error, e:
    159156                    if e.errno in (errno.EADDRINUSE, errno.EALREADY):
    … …  
    167164            finally:
    168165                port.stop_http_server()
    169                 s.close()
     166                test_socket.close()
    170167
    171168        # Test that calling start() twice fails.
    172169        try:
    173170            port.start_http_server()
    174             try:
    175                 port.start_http_server()
    176                 self.fail('calling port.start_http_server() twice worked')
    177             except http_server_base.ServerError, e:
    178                 pass
     171            self.assertRaises(AssertionError, port.start_http_server)
    179172        finally:
    180173            port.stop_http_server()
    181174
    182         # Test that calling stop() twice is harmless.
    183         port.stop_http_server()
     175        # Test that calling start() on two different ports causes the
     176        # first port to be treated as stale and killed.
     177        port.start_http_server()
     178        new_port = self.make_port()
     179        try:
     180            new_port.start_http_server()
     181
     182            # Check that the first server was killed.
     183            self.assertFalse(port._executive.check_running_pid(port._http_server._process.pid))
     184
     185            # Check that there is something running.
     186            self.assert_servers_are_up('localhost', self.HTTP_PORTS)
     187
     188            # Test that calling stop() on a killed server is harmless.
     189            port.stop_http_server()
     190        finally:
     191            port.stop_http_server()
     192            new_port.stop_http_server()
     193
     194            # Test that calling stop() twice is harmless.
     195            new_port.stop_http_server()
    184196
    185197    def integration_test_image_diff(self):
    … …  
    261273        # Test that start() fails if a port isn't available.
    262274        for port_number in self.WEBSOCKET_PORTS:
    263             s = socket.socket()
     275            test_socket = socket.socket()
    264276            try:
    265                 s.bind(('localhost', port_number))
     277                test_socket.bind(('localhost', port_number))
    266278                try:
    267279                    port.start_websocket_server()
    … …  
    271283            finally:
    272284                port.stop_websocket_server()
    273                 s.close()
     285                test_socket.close()
    274286
    275287        # Test that calling start() twice fails.
    276288        try:
    277289            port.start_websocket_server()
    278             try:
    279                 port.start_websocket_server()
    280                 self.fail('calling port.start_websocket_server() twice worked')
    281             except http_server_base.ServerError, e:
    282                 pass
     290            self.assertRaises(AssertionError, port.start_websocket_server)
    283291        finally:
    284292            port.stop_websocket_server()
    285293
    286         # Test that calling stop() twice is harmless.
    287         port.stop_websocket_server()
     294        # Test that calling start() on two different ports causes the
     295        # first port to be treated as stale and killed.
     296        port.start_websocket_server()
     297        new_port = self.make_port()
     298        try:
     299            new_port.start_websocket_server()
     300
     301            # Check that the first server was killed.
     302            self.assertFalse(port._executive.check_running_pid(port._websocket_server._process.pid))
     303
     304            # Check that there is something running.
     305            self.assert_servers_are_up('localhost', self.WEBSOCKET_PORTS)
     306
     307            # Test that calling stop() on a killed server is harmless.
     308            port.stop_websocket_server()
     309        finally:
     310            port.stop_websocket_server()
     311            new_port.stop_websocket_server()
     312
     313            # Test that calling stop() twice is harmless.
     314            new_port.stop_websocket_server()
    288315
    289316    def test_test_configuration(self):

  • trunk/Tools/Scripts/webkitpy/layout_tests/port/websocket_server.py

    r88680 r88995  
    11#!/usr/bin/env python
    2 # Copyright (C) 2010 Google Inc. All rights reserved.
     2# Copyright (C) 2011 Google Inc. All rights reserved.
    33#
    44# Redistribution and use in source and binary forms, with or without
    … …  
    3030"""A class to help start/stop the PyWebSocket server used by layout tests."""
    3131
     32import logging
     33import os
     34import sys
     35import time
    3236
    33 from __future__ import with_statement
    34 
    35 import codecs
    36 import logging
    37 import optparse
    38 import os
    39 import subprocess
    40 import sys
    41 import tempfile
    42 import time
    43 import urllib
    44 
    45 import factory
    46 import http_server
    47 
    48 from webkitpy.common.system.executive import Executive
    49 
     37from webkitpy.layout_tests.port import http_server
     38from webkitpy.layout_tests.port import http_server_base
    5039
    5140_log = logging.getLogger("webkitpy.layout_tests.port.websocket_server")
    … …  
    5847
    5948
    60 def url_is_alive(url):
    61     """Checks to see if we get an http response from |url|.
    62     We poll the url 20 times with a 0.5 second delay.  If we don't
    63     get a reply in that time, we give up and assume the httpd
    64     didn't start properly.
    65 
    66     Args:
    67       url: The URL to check.
    68     Return:
    69       True if the url is alive.
    70     """
    71     sleep_time = 0.5
    72     wait_time = 10
    73     while wait_time > 0:
    74         try:
    75             response = urllib.urlopen(url, proxies={})
    76             # Server is up and responding.
    77             return True
    78         except IOError:
    79             pass
    80         # Wait for sleep_time before trying again.
    81         wait_time -= sleep_time
    82         time.sleep(sleep_time)
    83 
    84     return False
    85 
    86 
    87 class PyWebSocketNotStarted(Exception):
    88     pass
    89 
    90 
    91 class PyWebSocketNotFound(Exception):
    92     pass
    93 
    94 
    9549class PyWebSocket(http_server.Lighttpd):
    96 
    9750    def __init__(self, port_obj, output_dir, port=_DEFAULT_WS_PORT,
    9851                 root=None, use_tls=False,
    … …  
    10558                                      root=root)
    10659        self._output_dir = output_dir
     60        self._pid_file = pidfile
    10761        self._process = None
     62
    10863        self._port = port
    10964        self._root = root
    11065        self._use_tls = use_tls
     66
     67        self._name = 'pywebsocket'
     68        if self._use_tls:
     69            self._name = 'pywebsocket_secure'
     70
    11171        self._private_key = self._pem_file
    11272        self._certificate = self._pem_file
    11373        if self._port:
    11474            self._port = int(self._port)
    115         if self._use_tls:
    116             self._server_name = 'PyWebSocket(Secure)'
    117         else:
    118             self._server_name = 'PyWebSocket'
    119         self._pidfile = pidfile
     75        self._wsin = None
    12076        self._wsout = None
     77        self._mappings = [{'port': self._port}]
     78
     79        if not self._pid_file:
     80            self._pid_file = self._filesystem.join(self._runtime_path, '%s.pid' % self._name)
    12181
    12282        # Webkit tests
    … …  
    13494                self._web_socket_tests = None
    13595
    136     def start(self):
    137         if not self._web_socket_tests:
    138             _log.info('No need to start %s server.' % self._server_name)
    139             return
    140         if self.is_running():
    141             raise PyWebSocketNotStarted('%s is already running.' %
    142                                         self._server_name)
     96        if self._use_tls:
     97            self._log_prefix = _WSS_LOG_PREFIX
     98        else:
     99            self._log_prefix = _WS_LOG_PREFIX
    143100
     101    def _prepare_config(self):
    144102        time_str = time.strftime('%d%b%Y-%H%M%S')
    145         if self._use_tls:
    146             log_prefix = _WSS_LOG_PREFIX
    147         else:
    148             log_prefix = _WS_LOG_PREFIX
    149         log_file_name = log_prefix + time_str
    150 
    151         # Remove old log files. We only need to keep the last ones.
    152         self.remove_log_files(self._output_dir, log_prefix)
     103        log_file_name = self._log_prefix + time_str
     104        self._wsin = open(os.devnull, 'r')
    153105
    154106        error_log = os.path.join(self._output_dir, log_file_name + "-err.txt")
    155107
    156108        output_log = os.path.join(self._output_dir, log_file_name + "-out.txt")
    157         self._wsout = codecs.open(output_log, "w", "utf-8")
     109        self._wsout = self._filesystem.open_text_file_for_writing(output_log)
    158110
    159111        from webkitpy.thirdparty.autoinstalled.pywebsocket import mod_pywebsocket
    … …  
    188140                              '-c', self._certificate])
    189141
    190         env = self._port_obj.setup_environ_for_server()
    191         env['PYTHONPATH'] = (pywebsocket_base + os.path.pathsep +
    192                              env.get('PYTHONPATH', ''))
     142        self._start_cmd = start_cmd
     143        self._env = self._port_obj.setup_environ_for_server()
     144        self._env['PYTHONPATH'] = (pywebsocket_base + os.path.pathsep + self._env.get('PYTHONPATH', ''))
    193145
    194         _log.debug('Starting %s server on %d.' % (
    195                    self._server_name, self._port))
    196         _log.debug('cmdline: %s' % ' '.join(start_cmd))
    197         # FIXME: We should direct this call through Executive for testing.
    198         # Note: Not thread safe: http://bugs.python.org/issue2320
    199         self._process = subprocess.Popen(start_cmd,
    200                                          stdin=open(os.devnull, 'r'),
    201                                          stdout=self._wsout,
    202                                          stderr=subprocess.STDOUT,
    203                                          env=env)
     146    def _remove_stale_logs(self):
     147        try:
     148            self._remove_log_files(self._output_dir, self._log_prefix)
     149        except OSError, e:
     150            _log.warning('Failed to remove stale %s log files: %s' % (self._name, str(e)))
    204151
    205         if self._use_tls:
    206             url = 'https'
    207         else:
    208             url = 'http'
    209         url = url + '://127.0.0.1:%d/' % self._port
    210         if not url_is_alive(url):
    211             if self._process.returncode == None:
    212                 # FIXME: We should use a non-static Executive for easier
    213                 # testing.
    214                 Executive().kill_process(self._process.pid)
    215             with codecs.open(output_log, "r", "utf-8") as fp:
    216                 for line in fp:
    217                     _log.error(line)
    218             raise PyWebSocketNotStarted(
    219                 'Failed to start %s server on port %s.' %
    220                     (self._server_name, self._port))
     152    def _spawn_process(self):
     153        _log.debug('starting %s, cmd="%s"' % (self._name, self._start_cmd))
     154        return self._executive.popen(self._start_cmd, env=self._env, shell=False, stdin=self._wsin, stdout=self._wsout, stderr=self._executive.STDOUT)
    221155
    222         # Our process terminated already
    223         if self._process.returncode != None:
    224             raise PyWebSocketNotStarted(
    225                 'Failed to start %s server.' % self._server_name)
    226         if self._pidfile:
    227             with codecs.open(self._pidfile, "w", "ascii") as file:
    228                 file.write("%d" % self._process.pid)
    229 
    230     def stop(self, force=False):
    231         if not force and not self.is_running():
    232             return
    233 
    234         pid = None
    235         if self._process:
    236             pid = self._process.pid
    237         elif self._pidfile:
    238             with codecs.open(self._pidfile, "r", "ascii") as file:
    239                 pid = int(file.read().strip())
    240 
    241         if not pid:
    242             raise PyWebSocketNotFound(
    243                 'Failed to find %s server pid.' % self._server_name)
    244 
    245         _log.debug('Shutting down %s server %d.' % (self._server_name, pid))
    246         # FIXME: We should use a non-static Executive for easier testing.
    247         Executive().kill_process(pid)
    248 
    249         if self._process:
    250             # wait() is not threadsafe and can throw OSError due to:
    251             # http://bugs.python.org/issue1731717
    252             self._process.wait()
    253             self._process = None
    254 
     156    def _cleanup_after_stop(self):
     157        if self._wsin:
     158            self._wsin.close()
     159            self._wsin = None
    255160        if self._wsout:
    256161            self._wsout.close()
Note: See TracChangeset for help on using the changeset viewer.