Changeset 128399 in webkit


Ignore:
Timestamp:
Sep 12, 2012 9:18:43 PM (12 years ago)
Author:
Csaba Osztrogonác
Message:

Add SVN mirror handling feature to build.webkit.org
​https://bugs.webkit.org/show_bug.cgi?id=85887

Reviewed by Dirk Pranke.

  • BuildSlaveSupport/build.webkit.org-config/config.json: Add SVN mirrors for Qt buildslaves hosted in Szeged.
  • BuildSlaveSupport/build.webkit.org-config/master.cfg:

(ConfigureBuild.init): Add SVNMirror property to be able to watch which slaves use which mirror on build.webkit.org.
(ConfigureBuild.start):
(CheckOutSource.init): Set baseURL to the SVNMirror or the default ​http://svn.webkit.org/repository/webkit/
(WaitForSVNServer): Add new buildstep for waiting the SVNMirror to be in sync with ​http://svn.webkit.org/repository/webkit/
(Factory.init):
(BuildFactory.init):
(TestFactory.init):
(BuildAndTestFactory.init):
(BuildAndPerfTestFactory.init):
(BuildAndPerfTestWebKit2Factory.init):
(DownloadAndPerfTestFactory.init):
(DownloadAndPerfTestWebKit2Factory.init):
(loadBuilderConfig): Use kwargs instead of args to be able to add the optional SVNMirror factory argument.

  • BuildSlaveSupport/build.webkit.org-config/mastercfg_unittest.py:

(SVNMirrorTest):
(SVNMirrorTest.setUp): Load config.json once at startup.
(SVNMirrorTest.get_SVNMirrorFromConfig): Get the SVNMirror from config.json for a given buildslave.
(SVNMirrorTest.test_CheckOutSource): Compare CheckOutSource.baseURL with SVNMirror in config.json for all builders

  • BuildSlaveSupport/build.webkit.org-config/wait-for-SVN-server.py: Added.

(getLatestSVNRevision): Get the latest SVN revison from the given server.
(waitForSVNRevision): Wait until the given SVN revision is committed to the given server. It doesn't wait if the

SVN revision is empty (force build triggered without revision) or the server is unavailable.

Location:
trunk/Tools
Files:
1 added
4 edited

Legend:

Unmodified
Added
Removed

  • trunk/Tools/BuildSlaveSupport/build.webkit.org-config/config.json

    r128381 r128399  
    201201                      "name": "Qt Linux Release", "type": "BuildAndTest", "builddir": "qt-linux-release",
    202202                      "platform": "qt-4.8", "configuration": "release", "architectures": ["i386"],
     203                      "SVNMirror": "svn://rain.inf.u-szeged.hu/",
    203204                      "slavenames": ["szeged-linux-1"]
    204205                    },
    205206                    {
    206207                      "name": "Qt Linux Release minimal", "type": "Build", "builddir": "qt-linux-release-minimal",
     208                      "SVNMirror": "svn://rain.inf.u-szeged.hu/",
    207209                      "platform": "qt-4.8", "configuration": "release", "architectures": ["i386"],
    208210                      "slavenames": ["szeged-linux-2"]
    … …  
    210212                    {
    211213                      "name": "Qt Linux 64-bit Release (Perf)", "type": "BuildAndPerfTest", "builddir": "qt-linux-64-release-perf-tests",
     214                      "SVNMirror": "svn://rain.inf.u-szeged.hu/",
    212215                      "platform": "qt-5.0-wk1", "configuration": "release", "architectures": ["x86_64"],
    213216                      "slavenames": ["szeged-linux-3"]
    … …  
    215218                    {
    216219                      "name": "Qt Linux 64-bit Release (WebKit2 Perf)", "type": "BuildAndPerfTestWebKit2", "builddir": "qt-linux-64-release-wk2-perf-tests",
     220                      "SVNMirror": "svn://rain.inf.u-szeged.hu/",
    217221                      "platform": "qt-5.0-wk2", "configuration": "release", "architectures": ["x86_64"],
    218222                      "slavenames": ["szeged-linux-5"]
    … …  
    220224                    {
    221225                      "name": "Qt Linux ARMv7 Release", "type": "Build", "builddir": "qt-linux-armv7-release",
     226                      "SVNMirror": "svn://rain.inf.u-szeged.hu:3389/",
    222227                      "platform": "qt-arm", "configuration": "release", "architectures": ["armv7"],
    223228                      "slavenames": ["szeged-linux-4"]
    … …  
    225230                    {
    226231                      "name": "Qt Windows 32-bit Release", "type": "Build", "builddir": "qt-windows-32bit-release",
     232                      "SVNMirror": "svn://rain.inf.u-szeged.hu/",
    227233                      "platform": "qt-win", "configuration": "release", "architectures": ["i386"],
    228234                      "slavenames": ["szeged-windows-1"]
    … …  
    230236                    {
    231237                      "name": "Qt Windows 32-bit Debug", "type": "Build", "builddir": "qt-windows-32bit-debug",
     238                      "SVNMirror": "svn://rain.inf.u-szeged.hu/",
    232239                      "platform": "qt-win", "configuration": "debug", "architectures": ["i386"],
    233240                      "slavenames": ["szeged-windows-2"]

  • trunk/Tools/BuildSlaveSupport/build.webkit.org-config/master.cfg

    r127104 r128399  
    9393    description = ["configuring build"]
    9494    descriptionDone = ["configured build"]
    95     def __init__(self, platform, configuration, architecture, buildOnly, *args, **kwargs):
     95    def __init__(self, platform, configuration, architecture, buildOnly, SVNMirror, *args, **kwargs):
    9696        buildstep.BuildStep.__init__(self, *args, **kwargs)
    9797        self.platform = platform.split('-', 1)[0]
    … …  
    100100        self.architecture = architecture
    101101        self.buildOnly = buildOnly
    102         self.addFactoryArguments(platform=platform, configuration=configuration, architecture=architecture, buildOnly=buildOnly)
     102        self.SVNMirror = SVNMirror
     103        self.addFactoryArguments(platform=platform, configuration=configuration, architecture=architecture, buildOnly=buildOnly, SVNMirror=SVNMirror)
    103104
    104105    def start(self):
    … …  
    109110        self.setProperty("buildOnly", self.buildOnly)
    110111        self.setProperty("shouldAbortEarly", True)
     112        self.setProperty("SVNMirror", self.SVNMirror)
    111113        self.finished(SUCCESS)
    112114        return defer.succeed(None)
    … …  
    114116
    115117class CheckOutSource(source.SVN):
    116     baseURL = "http://svn.webkit.org/repository/webkit/"
    117118    mode = "update"
    118     def __init__(self, **kwargs):
    119         kwargs['baseURL'] = self.baseURL
     119    def __init__(self, SVNMirror, **kwargs):
     120        kwargs['baseURL'] = SVNMirror or "http://svn.webkit.org/repository/webkit/"
    120121        kwargs['defaultBranch'] = "trunk"
    121122        kwargs['mode'] = self.mode
    122123        source.SVN.__init__(self, **kwargs)
    123 
    124 
     124        self.addFactoryArguments(SVNMirror=SVNMirror)
     125
     126class WaitForSVNServer(shell.ShellCommand):
     127    name = "wait-for-svn-server"
     128    command = ["python", "./Tools/BuildSlaveSupport/wait-for-SVN-server.py", "-r", WithProperties("%(revision)s"), "-s", WithProperties("%(SVNMirror)s")]
     129    description = ["waiting for SVN server"]
     130    descriptionDone = ["SVN server is ready"]
     131    haltOnFailure = True
    125132
    126133class InstallWin32Dependencies(shell.Compile):
    … …  
    698705
    699706class Factory(factory.BuildFactory):
    700     def __init__(self, platform, configuration, architectures, buildOnly):
     707    def __init__(self, platform, configuration, architectures, buildOnly, SVNMirror):
    701708        factory.BuildFactory.__init__(self)
    702         self.addStep(ConfigureBuild(platform=platform, configuration=configuration, architecture=" ".join(architectures), buildOnly=buildOnly))
    703         self.addStep(CheckOutSource())
     709        self.addStep(ConfigureBuild(platform=platform, configuration=configuration, architecture=" ".join(architectures), buildOnly=buildOnly, SVNMirror=SVNMirror))
     710        if SVNMirror:
     711            self.addStep(WaitForSVNServer())
     712        self.addStep(CheckOutSource(SVNMirror=SVNMirror))
    704713        # There are multiple Qt slaves running on same machines, so buildslaves shouldn't kill the processes of other slaves.
    705714        if not platform.startswith("qt"):
    … …  
    716725
    717726class BuildFactory(Factory):
    718     def __init__(self, platform, configuration, architectures, triggers=None):
    719         Factory.__init__(self, platform, configuration, architectures, True)
     727    def __init__(self, platform, configuration, architectures, triggers=None, SVNMirror=None):
     728        Factory.__init__(self, platform, configuration, architectures, True, SVNMirror)
    720729        self.addStep(CompileWebKit())
    721730        if triggers:
    … …  
    735744    TestClass = RunWebKitTests
    736745    ExtractTestResultsClass = ExtractTestResults
    737     def __init__(self, platform, configuration, architectures):
    738         Factory.__init__(self, platform, configuration, architectures, False)
     746    def __init__(self, platform, configuration, architectures, SVNMirror=None):
     747        Factory.__init__(self, platform, configuration, architectures, False, SVNMirror)
    739748        if platform.startswith("chromium"):
    740749            self.addStep(CleanupChromiumCrashLogs())
    … …  
    768777    TestClass = RunWebKitTests
    769778    ExtractTestResultsClass = ExtractTestResults
    770     def __init__(self, platform, configuration, architectures, triggers=None, **kwargs):
    771         Factory.__init__(self, platform, configuration, architectures, False, **kwargs)
     779    def __init__(self, platform, configuration, architectures, triggers=None, SVNMirror=None, **kwargs):
     780        Factory.__init__(self, platform, configuration, architectures, False, SVNMirror, **kwargs)
    772781        if platform.startswith("chromium"):
    773782            self.addStep(CleanupChromiumCrashLogs)
    … …  
    812821
    813822class BuildAndPerfTestFactory(Factory):
    814     def __init__(self, platform, configuration, architectures, **kwargs):
    815         Factory.__init__(self, platform, configuration, architectures, False, **kwargs)
     823    def __init__(self, platform, configuration, architectures, SVNMirror=None, **kwargs):
     824        Factory.__init__(self, platform, configuration, architectures, False, SVNMirror, **kwargs)
    816825        if platform.startswith("chromium"):
    817826            self.addStep(CleanupChromiumCrashLogs)
    … …  
    820829
    821830class BuildAndPerfTestWebKit2Factory(Factory):
    822     def __init__(self, platform, configuration, architectures, **kwargs):
    823         Factory.__init__(self, platform, configuration, architectures, False, **kwargs)
     831    def __init__(self, platform, configuration, architectures, SVNMirror=None, **kwargs):
     832        Factory.__init__(self, platform, configuration, architectures, False, SVNMirror, **kwargs)
    824833        if platform.startswith("chromium"):
    825834            self.addStep(CleanupChromiumCrashLogs)
    … …  
    828837
    829838class DownloadAndPerfTestFactory(Factory):
    830     def __init__(self, platform, configuration, architectures, **kwargs):
    831         Factory.__init__(self, platform, configuration, architectures, False, **kwargs)
     839    def __init__(self, platform, configuration, architectures, SVNMirror=None, **kwargs):
     840        Factory.__init__(self, platform, configuration, architectures, False, SVNMirror, **kwargs)
    832841        self.addStep(DownloadBuiltProduct())
    833842        self.addStep(ExtractBuiltProduct())
    … …  
    835844
    836845class DownloadAndPerfTestWebKit2Factory(Factory):
    837     def __init__(self, platform, configuration, architectures, **kwargs):
    838         Factory.__init__(self, platform, configuration, architectures, False, **kwargs)
     846    def __init__(self, platform, configuration, architectures, SVNMirror=None, **kwargs):
     847        Factory.__init__(self, platform, configuration, architectures, False, SVNMirror, **kwargs)
    839848        self.addStep(DownloadBuiltProduct)
    840849        self.addStep(ExtractBuiltProduct)
    … …  
    903912        builderType = builder.pop('type')
    904913        factory = globals()["%sFactory" % builderType]
    905         factoryArgs = []
    906         for key in "platform", "configuration", "architectures", "triggers":
     914        factorykwargs = {}
     915        for key in "platform", "configuration", "architectures", "triggers", "SVNMirror":
    907916            value = builder.pop(key, None)
    908917            if value:
    909                 factoryArgs.append(value)
    910 
    911         builder["factory"] = factory(*factoryArgs)
     918                factorykwargs[key] = value
     919
     920        builder["factory"] = factory(**factorykwargs)
    912921
    913922        if platform.startswith('chromium'):

  • trunk/Tools/BuildSlaveSupport/build.webkit.org-config/mastercfg_unittest.py

    r125372 r128399  
    333333
    334334
     335class SVNMirrorTest(unittest.TestCase):
     336    def setUp(self):
     337        self.config = json.load(open('config.json'))
     338
     339    def get_SVNMirrorFromConfig(self, builderName):
     340        SVNMirror = None
     341        for builder in self.config['builders']:
     342            if builder['name'] == builderName:
     343                SVNMirror = builder.pop('SVNMirror', 'http://svn.webkit.org/repository/webkit/')
     344        return SVNMirror
     345
     346    def test_CheckOutSource(self):
     347        # SVN mirror feature isn't unittestable now with source.oldsource.SVN(==source.SVN) , only with source.svn.SVN(==SVN)
     348        # https://bugs.webkit.org/show_bug.cgi?id=85887
     349        if issubclass(CheckOutSource, source.SVN):
     350            return
     351
     352        # Compare CheckOutSource.baseURL with SVNMirror (or with the default URL) in config.json for all builders
     353        for builder in c['builders']:
     354            for buildStepFactory, kwargs in builder['factory'].steps:
     355                if str(buildStepFactory).split('.')[-1] == 'CheckOutSource':
     356                        CheckOutSourceInstance = buildStepFactory(**kwargs)
     357                        self.assertEquals(CheckOutSourceInstance.baseURL, self.get_SVNMirrorFromConfig(builder['name']))
     358
     359
    335360class BuildStepsConstructorTest(unittest.TestCase):
    336361    # "Passing a BuildStep subclass to factory.addStep is deprecated. Please pass a BuildStep instance instead.  Support will be dropped in v0.8.7."

  • trunk/Tools/ChangeLog

    r128395 r128399  
     12012-09-12  Csaba Osztrogonác  <ossy@webkit.org>
     2
     3        Add SVN mirror handling feature to build.webkit.org
     4        https://bugs.webkit.org/show_bug.cgi?id=85887
     5
     6        Reviewed by Dirk Pranke.
     7
     8        * BuildSlaveSupport/build.webkit.org-config/config.json: Add SVN mirrors for Qt buildslaves hosted in Szeged.
     9        * BuildSlaveSupport/build.webkit.org-config/master.cfg:
     10        (ConfigureBuild.__init__): Add SVNMirror property to be able to watch which slaves use which mirror on build.webkit.org.
     11        (ConfigureBuild.start):
     12        (CheckOutSource.__init__): Set baseURL to the SVNMirror or the default http://svn.webkit.org/repository/webkit/
     13        (WaitForSVNServer): Add new buildstep for waiting the SVNMirror to be in sync with http://svn.webkit.org/repository/webkit/
     14        (Factory.__init__):
     15        (BuildFactory.__init__):
     16        (TestFactory.__init__):
     17        (BuildAndTestFactory.__init__):
     18        (BuildAndPerfTestFactory.__init__):
     19        (BuildAndPerfTestWebKit2Factory.__init__):
     20        (DownloadAndPerfTestFactory.__init__):
     21        (DownloadAndPerfTestWebKit2Factory.__init__):
     22        (loadBuilderConfig): Use kwargs instead of args to be able to add the optional SVNMirror factory argument.
     23        * BuildSlaveSupport/build.webkit.org-config/mastercfg_unittest.py:
     24        (SVNMirrorTest):
     25        (SVNMirrorTest.setUp): Load config.json once at startup.
     26        (SVNMirrorTest.get_SVNMirrorFromConfig): Get the SVNMirror from config.json for a given buildslave.
     27        (SVNMirrorTest.test_CheckOutSource): Compare CheckOutSource.baseURL with SVNMirror in config.json for all builders
     28        * BuildSlaveSupport/build.webkit.org-config/wait-for-SVN-server.py: Added.
     29        (getLatestSVNRevision): Get the latest SVN revison from the given server.
     30        (waitForSVNRevision): Wait until the given SVN revision is committed to the given server. It doesn't wait if the
     31                              SVN revision is empty (force build triggered without revision) or the server is unavailable.
     32
    1332012-09-12  Kangil Han  <kangil.han@samsung.com>
    234
Note: See TracChangeset for help on using the changeset viewer.