Changeset 248979 in webkit


Ignore:
Timestamp:
Aug 21, 2019 5:21:50 PM (5 years ago)
Author:
Jonathan Bedard
Message:

results.webkit.org: Add database table to save zip archives to
​https://bugs.webkit.org/show_bug.cgi?id=200718

Rubber-stamped by Aakash Jain.

  • resultsdbpy/resultsdbpy/model/archive_context.py: Added.

(ArchiveContext):
(ArchiveContext.ArchivesByCommit): Store archives to be retrieved by commit and configuration.
(ArchiveContext.ArchivesByCommit.unpack):
(ArchiveContext.assert_zipfile):
(ArchiveContext.open_zipfile):
(ArchiveContext.init):
(ArchiveContext.enter):
(ArchiveContext.exit):
(ArchiveContext.register): Save the provided archive to Cassandra.
(ArchiveContext.find_archive): Find all archives matching the specified criteria.
(ArchiveContext._files_for_archive):
(ArchiveContext.ls): List all files in the matching archives.
(ArchiveContext.file): Extract a file from the matching archives.

  • resultsdbpy/resultsdbpy/model/archive_context_unittest.py: Added.

(ArchiveContextTest):

  • resultsdbpy/resultsdbpy/model/mock_model_factory.py:

(MockModelFactory):
(MockModelFactory.process_results):
(MockModelFactory.add_mock_archives):

  • resultsdbpy/resultsdbpy/model/model.py:

(Model.init): Add an ArchiveContext to the model.

Location:
trunk/Tools
Files:
2 added
3 edited

Legend:

Unmodified
Added
Removed

  • trunk/Tools/ChangeLog

    r248976 r248979  
     12019-08-21  Jonathan Bedard  <jbedard@apple.com>
     2
     3        results.webkit.org: Add database table to save zip archives to
     4        https://bugs.webkit.org/show_bug.cgi?id=200718
     5
     6        Rubber-stamped by Aakash Jain.
     7
     8        * resultsdbpy/resultsdbpy/model/archive_context.py: Added.
     9        (ArchiveContext):
     10        (ArchiveContext.ArchivesByCommit): Store archives to be retrieved by commit and configuration.
     11        (ArchiveContext.ArchivesByCommit.unpack):
     12        (ArchiveContext.assert_zipfile):
     13        (ArchiveContext.open_zipfile):
     14        (ArchiveContext.__init__):
     15        (ArchiveContext.__enter__):
     16        (ArchiveContext.__exit__):
     17        (ArchiveContext.register): Save the provided archive to Cassandra.
     18        (ArchiveContext.find_archive): Find all archives matching the specified criteria.
     19        (ArchiveContext._files_for_archive):
     20        (ArchiveContext.ls): List all files in the matching archives.
     21        (ArchiveContext.file): Extract a file from the matching archives.
     22        * resultsdbpy/resultsdbpy/model/archive_context_unittest.py: Added.
     23        (ArchiveContextTest):
     24        * resultsdbpy/resultsdbpy/model/mock_model_factory.py:
     25        (MockModelFactory):
     26        (MockModelFactory.process_results):
     27        (MockModelFactory.add_mock_archives):
     28        * resultsdbpy/resultsdbpy/model/model.py:
     29        (Model.__init__): Add an ArchiveContext to the model.
     30
    1312019-08-21  Aakash Jain  <aakash_jain@apple.com>
    232

  • trunk/Tools/resultsdbpy/resultsdbpy/model/mock_model_factory.py

    r247628 r248979  
    2121# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
    2222
     23import base64
     24import io
    2325import time
    2426
    … …  
    3133
    3234class MockModelFactory(object):
     35    ARCHIVE_ZIP = """UEsDBAoAAAAAAAtSBU8AAAAAAAAAAAAAAAAIABAAYXJjaGl2ZS9VWAwAZ2RIXWZkSF31ARQAUEsDBBQACAAIAA9SBU8AAAAAAAAAAAAAAAAQABAAYXJjaGl2ZS9maWxlLnR4dFVYDABovU1d
     36    bmRIXfUBFABLSSxJBABQSwcIY/PzrQYAAAAEAAAAUEsDBAoAAAAAABRdCU8AAAAAAAAAAAAAAAAJABAAX19NQUNPU1gvVVgMACi+TV0ovk1d9QEUAFBLAwQKAAAAAAAUXQlPAAAAAAAA
     37    AAAAAAAAEQAQAF9fTUFDT1NYL2FyY2hpdmUvVVgMACi+TV0ovk1d9QEUAFBLAwQUAAgACAAPUgVPAAAAAAAAAAAAAAAAGwAQAF9fTUFDT1NYL2FyY2hpdmUvLl9maWxlLnR4dFVYDABo
     38    vU1dbmRIXfUBFABjYBVjZ2BiYPBNTFbwD1aIUIACkBgDJxAbMTAwegFpIJ+xhoEo4BgSEgRhgXXcAeIFaEqYoeICDAxSyfm5eokFBTmpejmJxSWlxakpKYklqcoBwVC1b4DYg4GBH6Eu
     39    NzE5B2K+CUROFCFXWJpYlJhXkpmXypCd4hELUsUaKK4AVs0w95H9l352x+37375yVmg4n0+cf9BBob6BgYWxtWmKSUpSipGxtWNRckZmWWpMhZFBaElmTmZJpbWBs6GzkbOzpa6FpamF
     40    romRm6Wuk7mFi66FqZuxiamLhauriSsDAFBLBwjEE3dr4AAAAHwBAABQSwMEFAAIAAgAzFwJTwAAAAAAAAAAAAAAABIAEABhcmNoaXZlL2luZGV4Lmh0bWxVWAwAor1NXaC9TV31ARQA
     41    tVNdb+IwEHz3r9j2qZUCvfbt7hCSSQxYCnHOdsrxmBK3tRRilJj2+u9vbajKfejeDgliMruzM7PJ5GI0IpC6/Vtvn549XKXXcPfp9jPQ/b41wLvtGGjbQkQH6M1g+hfTjAkBaRo7+N4+
     42    HLx1HdRdA4fBgO1gcId+a+KdB9vV/Rs8un43JPBq/TO4Pl7dwRPYucY+2m0dGBKoewN70++s96aBfe9ebIMH/1x7/DHI0rbu1XZPsHVdY0PTQGLXzvgvBG7Hv4kawD2+q9m6BusOg0cT
     43    vkaVgbF+cC8BOtkngJ/Oebs1CeJ2gBbZAsnHwGjrVzU4ctvWdmf6MYG7P0XgsLMc3kWgv+aAwv6HDjj6izyN2x52pvP1+5pucAMO0R52tTe9rdvhI+y4okB7biGsWy+5AiXmek0lAzyX
     44    UtzzjGUw2wAtyxxvFukYLqlC9BJokeF3Q4B9LyVTCoQEvipzjh1IIWmhOVNJaMqrjBeLBGaVhkJoyPmKayzTIsGxjPylD8QcVkymS/xLZzznehMnzrkuwrA5TqNQUql5WuVUEigrWQrF
     45    IKjPuEpzylcsGwMKwKHA7lmhQS1pnp+7EdiZikJLjuKEVDBjKI/OEI8jig2SSZbqYOTjlGIwKCxPQJUs5XgIOTC0QeUmCVEgqWLfKqxCFDK6ogt0dfXvNEgIPa0kWwWxGIGqZkpzXWkG
     46    CyGyGLJi8p6nTH2FXKgYVKVYgiM0TaIf5MCYEMfiWaV4DIwXmklZlZqL4hqWYo2BoEqKvVlMVhTRLe5DSNwq0oYcYvIJrJcMARmyjGnREIPC1FJ9XoYDMURNznxCwRY5X7AiZQEWgWbN
     47    FbvGRXEVCvhx8JpuQFTRNdYET+R4Pnssk7hG4HOg2T0Pyk/VuHnFT49JjC1dnjIfk9FoSsjk2e/aKV5M3Zh+OvHWt2Zqu8b8GAdocnO8M7k5VZDJg2vepvENWxp8A+HV9W1zQSY3RwAr
     48    A+VPUEsHCPbdMMviAgAAYQUAAFBLAwQUAAgACADMXAlPAAAAAAAAAAAAAAAAHQAQAF9fTUFDT1NYL2FyY2hpdmUvLl9pbmRleC5odG1sVVgMAKK9TV2gvU1d9QEUAGNgFWNnYGJg8E1M
     49    VvAPVohQgAKQGAMnEBsxMDB6AWkgn7GGgSjgGBISBGGBddwB4gVoSpih4gIMDFLJ+bl6iQUFOal6OYnFJaXFqSkpiSWpygHBULVvgNiDgYEfoS43MTkHYr4JRE4UIVdYmliUmFeSmZfK
     50    UL/XNxak6qLfEiGwaoa5j+y/9LM7bt//9pWzQsP5fOL8gw4K9Q0MLIytTVNMUpJSjIytHYuSMzLLUmMqjAxCSzJzMksqrQ2cDZ2NnJ0tdS0sTS10TYzcLHWdzC1cdC1M3YxNTF0sXF1N
     51    XBkAUEsHCLRBGwrgAAAAfAEAAFBLAwQUAAgACAALUgVPAAAAAAAAAAAAAAAAEgAQAF9fTUFDT1NYLy5fYXJjaGl2ZVVYDABnZEhdZmRIXfUBFABjYBVjZ2BiYPBNTFbwD1aIUIACkBgD
     52    JxAbMTAwCgFpIJ/RhYEo4BgSEgRhgXVsAeIJaEqYoOIeDAz8yfm5eokFBTmpermJyTkQ+T8QOVGEXGFpYlFiXklmXioDI0Ntye3fifMcHKZ8fXTEZauLLSPD3Ef2X/rZHbfvf/vKWaHh
     53    fD4x7izUNzCwMLY2gAJrx6LkjMyy1JgKI4PQksyczJJKawNnQ2cjZ2dLXQtLUwtdEyM3S10ncwsXXQtTN2MTUxcLV1cTVwYAUEsHCAAolTbHAAAARAEAAFBLAQIVAwoAAAAAAAtSBU8A
     54    AAAAAAAAAAAAAAAIAAwAAAAAAAAAAEDtQQAAAABhcmNoaXZlL1VYCABnZEhdZmRIXVBLAQIVAxQACAAIAA9SBU9j8/OtBgAAAAQAAAAQAAwAAAAAAAAAAECkgTYAAABhcmNoaXZlL2Zp
     55    bGUudHh0VVgIAGi9TV1uZEhdUEsBAhUDCgAAAAAAFF0JTwAAAAAAAAAAAAAAAAkADAAAAAAAAAAAQP1BigAAAF9fTUFDT1NYL1VYCAAovk1dKL5NXVBLAQIVAwoAAAAAABRdCU8AAAAA
     56    AAAAAAAAAAARAAwAAAAAAAAAAED9QcEAAABfX01BQ09TWC9hcmNoaXZlL1VYCAAovk1dKL5NXVBLAQIVAxQACAAIAA9SBU/EE3dr4AAAAHwBAAAbAAwAAAAAAAAAAECkgQABAABfX01B
     57    Q09TWC9hcmNoaXZlLy5fZmlsZS50eHRVWAgAaL1NXW5kSF1QSwECFQMUAAgACADMXAlP9t0wy+ICAABhBQAAEgAMAAAAAAAAAABApIE5AgAAYXJjaGl2ZS9pbmRleC5odG1sVVgIAKK9
     58    TV2gvU1dUEsBAhUDFAAIAAgAzFwJT7RBGwrgAAAAfAEAAB0ADAAAAAAAAAAAQKSBawUAAF9fTUFDT1NYL2FyY2hpdmUvLl9pbmRleC5odG1sVVgIAKK9TV2gvU1dUEsBAhUDFAAIAAgA
     59    C1IFTwAolTbHAAAARAEAABIADAAAAAAAAAAAQKSBpgYAAF9fTUFDT1NYLy5fYXJjaGl2ZVVYCABnZEhdZmRIXVBLBQYAAAAACAAIAF4CAAC9BwAAAAA="""
     60    THREE_WEEKS = 60 * 60 * 24 * 21
    3361
    3462    @classmethod
    … …  
    117145        with model.upload_context:
    118146            current = time.time()
    119             old = current - 60 * 60 * 24 * 21
     147            old = current - cls.THREE_WEEKS
    120148            for complete_configuration in configurations:
    121149                if complete_configuration != configuration:
    … …  
    130158
    131159    @classmethod
    132     def process_results(self, model, configuration=Configuration(), suite='layout-tests'):
     160    def process_results(cls, model, configuration=Configuration(), suite='layout-tests'):
    133161        configurations = [configuration] if configuration.is_complete() else ConfigurationContextTest.CONFIGURATIONS
    134162
    … …  
    148176                                test_results=result['test_results'], timestamp=result['timestamp'],
    149177                            )
     178
     179    @classmethod
     180    def add_mock_archives(cls, model, configuration=Configuration(), suite='layout-tests', archive=None):
     181        archive = archive or io.BytesIO(base64.b64decode(cls.ARCHIVE_ZIP))
     182        configurations = [configuration] if configuration.is_complete() else ConfigurationContextTest.CONFIGURATIONS
     183
     184        with model.upload_context:
     185            current = time.time()
     186            old = current - cls.THREE_WEEKS
     187            for complete_configuration in configurations:
     188                if complete_configuration != configuration:
     189                    continue
     190
     191                timestamp_to_use = current
     192                if (complete_configuration.platform == 'Mac' and complete_configuration.version <= Configuration.version_to_integer('10.13')) \
     193                   or (complete_configuration.platform == 'iOS' and complete_configuration.version <= Configuration.version_to_integer('11')):
     194                    timestamp_to_use = old
     195
     196                cls.iterate_all_commits(model, lambda commits: model.archive_context.register(archive, complete_configuration, commits, suite=suite, timestamp=timestamp_to_use))

  • trunk/Tools/resultsdbpy/resultsdbpy/model/model.py

    r247628 r248979  
    2424import sys
    2525
     26from resultsdbpy.model.archive_context import ArchiveContext
    2627from resultsdbpy.model.ci_context import CIContext
    2728from resultsdbpy.model.commit_context import CommitContext
    … …  
    7576            self.upload_context.register_upload_callback(context.name, context.register)
    7677
     78        self.archive_context = ArchiveContext(
     79            configuration_context=self.configuration_context,
     80            commit_context=self.commit_context,
     81            ttl_seconds=self.default_ttl_seconds,
     82        )
     83
    7784    def do_work(self):
    7885        if not self._async_processing:
Note: See TracChangeset for help on using the changeset viewer.