Changeset 64687 in webkit


Ignore:
Timestamp:
Aug 4, 2010 4:50:58 PM (14 years ago)
Author:
victorw@chromium.org
Message:

2010-08-04 Victor Wang <​victorw@chromium.org>

Reviewed by Ojan Vafai.

-. Add result.json incremental merging functionality to test results app engine.
-. blobstore is not suitable for AE data merging and there is no API to

programatically edit existing blob or write a new one yet, so replace blobstore
with datastore. If file is oversize (>1000*1000 bytes), store file data in
multiple datastore entries.

-. Fix styles.

Test: jsonresults_unittest to test merging logics.

​https://bugs.webkit.org/show_bug.cgi?id=38599

  • TestResultServer/handlers/dashboardhandler.py:
  • TestResultServer/handlers/menu.py:
  • TestResultServer/handlers/testfilehandler.py:
  • TestResultServer/main.py:
  • TestResultServer/model/dashboardfile.py:
  • TestResultServer/model/datastorefile.py: Added.
  • TestResultServer/model/jsonresults.py: Added.
  • TestResultServer/model/jsonresults_unittest.py: Added.
  • TestResultServer/model/testfile.py:
  • TestResultServer/templates/uploadform.html:
Location:
trunk/WebKitTools
Files:
3 added
8 edited

Legend:

Unmodified
Added
Removed

  • trunk/WebKitTools/ChangeLog

    r64673 r64687  
     12010-08-04  Victor Wang  <victorw@chromium.org>
     2
     3        Reviewed by Ojan Vafai.
     4
     5        -. Add result.json incremental merging functionality to test results app engine.
     6        -. blobstore is not suitable for AE data merging and there is no API to
     7           programatically edit existing blob or write a new one yet, so replace blobstore
     8           with datastore. If file is oversize (>1000*1000 bytes), store file data in
     9           multiple datastore entries.
     10        -. Fix styles.
     11
     12        Test: jsonresults_unittest to test merging logics.
     13
     14        https://bugs.webkit.org/show_bug.cgi?id=38599
     15
     16        * TestResultServer/handlers/dashboardhandler.py:
     17        * TestResultServer/handlers/menu.py:
     18        * TestResultServer/handlers/testfilehandler.py:
     19        * TestResultServer/main.py:
     20        * TestResultServer/model/dashboardfile.py:
     21        * TestResultServer/model/datastorefile.py: Added.
     22        * TestResultServer/model/jsonresults.py: Added.
     23        * TestResultServer/model/jsonresults_unittest.py: Added.
     24        * TestResultServer/model/testfile.py:
     25        * TestResultServer/templates/uploadform.html:
     26
    1272010-08-04  Antonio Gomes  <tonikitoo@webkit.org>
    228

  • trunk/WebKitTools/TestResultServer/handlers/dashboardhandler.py

    r56635 r64687  
    44# modification, are permitted provided that the following conditions are
    55# met:
    6 # 
     6#
    77#     * Redistributions of source code must retain the above copyright
    88# notice, this list of conditions and the following disclaimer.
    … …  
    1414# contributors may be used to endorse or promote products derived from
    1515# this software without specific prior written permission.
    16 # 
     16#
    1717# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
    1818# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
    … …  
    3838
    3939PARAM_FILE = "file"
     40
    4041
    4142def get_content_type(filename):

  • trunk/WebKitTools/TestResultServer/handlers/menu.py

    r56753 r64687  
    44# modification, are permitted provided that the following conditions are
    55# met:
    6 # 
     6#
    77#     * Redistributions of source code must retain the above copyright
    88# notice, this list of conditions and the following disclaimer.
    … …  
    1414# contributors may be used to endorse or promote products derived from
    1515# this software without specific prior written permission.
    16 # 
     16#
    1717# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
    1818# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
    … …  
    6262        self.response.out.write(
    6363            template.render("templates/menu.html", template_values))
    64 

  • trunk/WebKitTools/TestResultServer/handlers/testfilehandler.py

    r56635 r64687  
    44# modification, are permitted provided that the following conditions are
    55# met:
    6 # 
     6#
    77#     * Redistributions of source code must retain the above copyright
    88# notice, this list of conditions and the following disclaimer.
    … …  
    1414# contributors may be used to endorse or promote products derived from
    1515# this software without specific prior written permission.
    16 # 
     16#
    1717# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
    1818# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
    … …  
    3131
    3232from google.appengine.api import users
    33 from google.appengine.ext import blobstore
    3433from google.appengine.ext import webapp
    35 from google.appengine.ext.webapp import blobstore_handlers
    3634from google.appengine.ext.webapp import template
    3735
     36from model.jsonresults import JsonResults
    3837from model.testfile import TestFile
    3938
    … …  
    4443PARAM_KEY = "key"
    4544PARAM_TEST_TYPE = "testtype"
     45PARAM_INCREMENTAL = "incremental"
    4646
    4747
    4848class DeleteFile(webapp.RequestHandler):
    49     """Delete test file for a given builder and name from datastore (metadata) and blobstore (file data)."""
     49    """Delete test file for a given builder and name from datastore."""
    5050
    5151    def get(self):
    … …  
    5656
    5757        logging.debug(
    58             "Deleting File, builder: %s, test_type: %s, name: %s, blob key: %s.",
     58            "Deleting File, builder: %s, test_type: %s, name: %s, key: %s.",
    5959            builder, test_type, name, key)
    6060
    … …  
    6666
    6767
    68 class GetFile(blobstore_handlers.BlobstoreDownloadHandler):
     68class GetFile(webapp.RequestHandler):
    6969    """Get file content or list of files for given builder and name."""
    7070
    … …  
    7878        """
    7979
    80         files = TestFile.get_files(builder, test_type, name, 100)
     80        files = TestFile.get_files(
     81            builder, test_type, name, load_data=False, limit=100)
    8182        if not files:
    8283            logging.info("File not found, builder: %s, test_type: %s, name: %s.",
    … …  
    104105        """
    105106
    106         files = TestFile.get_files(builder, test_type, name, 1)
     107        files = TestFile.get_files(
     108            builder, test_type, name, load_data=True, limit=1)
    107109        if not files:
    108110            logging.info("File not found, builder: %s, test_type: %s, name: %s.",
    … …  
    110112            return
    111113
    112         blob_key = files[0].blob_key
    113         blob_info = blobstore.get(blob_key)
    114         if blob_info:
    115             self.send_blob(blob_info, "text/plain")
     114        self.response.headers["Content-Type"] = "text/plain; charset=utf-8"
     115        self.response.out.write(files[0].data)
    116116
    117117    def get(self):
    … …  
    134134
    135135
    136 class GetUploadUrl(webapp.RequestHandler):
    137     """Get an url for uploading file to blobstore. A special url is required for each blobsotre upload."""
    138 
    139     def get(self):
    140         upload_url = blobstore.create_upload_url("/testfile/upload")
    141         logging.info("Getting upload url: %s.", upload_url)
    142         self.response.out.write(upload_url)
    143 
    144 
    145 class Upload(blobstore_handlers.BlobstoreUploadHandler):
    146     """Upload file to blobstore."""
     136class Upload(webapp.RequestHandler):
     137    """Upload test results file to datastore."""
    147138
    148139    def post(self):
    149         uploaded_files = self.get_uploads("file")
    150         if not uploaded_files:
    151             return self._upload_done([("Missing upload file field.")])
     140        file_params = self.request.POST.getall(PARAM_FILE)
     141        if not file_params:
     142            self.response.out.write("FAIL: missing upload file field.")
     143            return
    152144
    153145        builder = self.request.get(PARAM_BUILDER)
    154146        if not builder:
    155             for blob_info in uploaded_files:
    156                 blob_info.delete()
    157    
    158             return self._upload_done([("Missing builder parameter in upload request.")])
     147            self.response.out.write("FAIL: missing builder parameter.")
     148            return
    159149
    160150        test_type = self.request.get(PARAM_TEST_TYPE)
     151        incremental = self.request.get(PARAM_INCREMENTAL)
    161152
    162153        logging.debug(
    … …  
    164155            builder, test_type)
    165156
     157        # There are two possible types of each file_params in the request:
     158        # one file item or a list of file items.
     159        # Normalize file_params to a file item list.
     160        files = []
     161        logging.debug("test: %s, type:%s", file_params, type(file_params))
     162        for item in file_params:
     163            if not isinstance(item, list) and not isinstance(item, tuple):
     164                item = [item]
     165            files.extend(item)
     166
    166167        errors = []
    167         for blob_info in uploaded_files:
    168             tf = TestFile.update_file(builder, test_type, blob_info)
    169             if not tf:
     168        for file in files:
     169            filename = file.filename.lower()
     170            if ((incremental and filename == "results.json") or
     171                (filename == "incremental_results.json")):
     172                # Merge incremental json results.
     173                saved_file = JsonResults.update(builder, test_type, file.value)
     174            else:
     175                saved_file = TestFile.update(
     176                    builder, test_type, file.filename, file.value)
     177
     178            if not saved_file:
    170179                errors.append(
    171180                    "Upload failed, builder: %s, test_type: %s, name: %s." %
    172                     (builder, test_type, blob_info.filename))
    173                 blob_info.delete()
    174 
    175         return self._upload_done(errors)
    176 
    177     def _upload_done(self, errors):
    178         logging.info("upload done.")
    179 
    180         error_messages = []
    181         for error in errors:
    182             logging.info(error)
    183             error_messages.append("error=%s" % urllib.quote(error))
    184 
    185         if error_messages:
    186             redirect_url = "/uploadfail?%s" % "&".join(error_messages)
     181                    (builder, test_type, file.filename))
     182
     183        if errors:
     184            messages = "FAIL: " + "; ".join(errors)
     185            logging.warning(messages)
     186            self.response.set_status(500, messages)
     187            self.response.out.write("FAIL")
    187188        else:
    188             redirect_url = "/uploadsuccess"
    189 
    190         logging.info(redirect_url)
    191         # BlobstoreUploadHandler requires redirect at the end.
    192         self.redirect(redirect_url)
     189            self.response.set_status(200)
     190            self.response.out.write("OK")
    193191
    194192
    195193class UploadForm(webapp.RequestHandler):
    196     """Show a form so user can submit a file to blobstore."""
     194    """Show a form so user can upload a file."""
    197195
    198196    def get(self):
    199         upload_url = blobstore.create_upload_url("/testfile/upload")
    200197        template_values = {
    201             "upload_url": upload_url,
     198            "upload_url": "/testfile/upload",
    202199        }
    203200        self.response.out.write(template.render("templates/uploadform.html",
    204201                                                template_values))
    205 
    206 class UploadStatus(webapp.RequestHandler):
    207     """Return status of file uploading"""
    208 
    209     def get(self):
    210         logging.debug("Update status")
    211 
    212         if self.request.path == "/uploadsuccess":
    213             self.response.set_status(200)
    214             self.response.out.write("OK")
    215         else:
    216             errors = self.request.params.getall("error")
    217             if errors:
    218                 messages = "FAIL: " + "; ".join(errors)
    219                 logging.warning(messages)
    220                 self.response.set_status(500, messages)
    221                 self.response.out.write("FAIL")

  • trunk/WebKitTools/TestResultServer/main.py

    r56635 r64687  
    44# modification, are permitted provided that the following conditions are
    55# met:
    6 # 
     6#
    77#     * Redistributions of source code must retain the above copyright
    88# notice, this list of conditions and the following disclaimer.
    … …  
    1414# contributors may be used to endorse or promote products derived from
    1515# this software without specific prior written permission.
    16 # 
     16#
    1717# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
    1818# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
    … …  
    4343    ('/dashboards/([^?]+)?', dashboardhandler.GetDashboardFile),
    4444    ('/testfile/delete', testfilehandler.DeleteFile),
    45     ('/testfile/uploadurl', testfilehandler.GetUploadUrl),
    4645    ('/testfile/upload', testfilehandler.Upload),
    4746    ('/testfile/uploadform', testfilehandler.UploadForm),
    4847    ('/testfile/?', testfilehandler.GetFile),
    49     ('/uploadfail', testfilehandler.UploadStatus),
    50     ('/uploadsuccess', testfilehandler.UploadStatus),
    5148    ('/*|/menu', menu.Menu),
    5249]
    5350
    5451application = webapp.WSGIApplication(routes, debug=True)
     52
    5553
    5654def main():

  • trunk/WebKitTools/TestResultServer/model/dashboardfile.py

    r56635 r64687  
    44# modification, are permitted provided that the following conditions are
    55# met:
    6 # 
     6#
    77#     * Redistributions of source code must retain the above copyright
    88# notice, this list of conditions and the following disclaimer.
    … …  
    1414# contributors may be used to endorse or promote products derived from
    1515# this software without specific prior written permission.
    16 # 
     16#
    1717# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
    1818# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
    … …  
    3636SVN_PATH_DASHBOARD = ("http://src.chromium.org/viewvc/chrome/trunk/tools/"
    3737    "dashboards/")
     38
    3839
    3940class DashboardFile(db.Model):
    … …  
    9394            logging.info("No existing file, added as new file.")
    9495            return cls.add_file(name, data)
    95        
     96
    9697        logging.debug("Updating existing file.")
    9798        file = files[0]

  • trunk/WebKitTools/TestResultServer/model/testfile.py

    r56635 r64687  
    44# modification, are permitted provided that the following conditions are
    55# met:
    6 # 
     6#
    77#     * Redistributions of source code must retain the above copyright
    88# notice, this list of conditions and the following disclaimer.
    … …  
    1414# contributors may be used to endorse or promote products derived from
    1515# this software without specific prior written permission.
    16 # 
     16#
    1717# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
    1818# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
    … …  
    3030import logging
    3131
    32 from google.appengine.ext import blobstore
    3332from google.appengine.ext import db
    3433
     34from model.datastorefile import DataStoreFile
    3535
    36 class TestFile(db.Model):
     36
     37class TestFile(DataStoreFile):
    3738    builder = db.StringProperty()
    38     name = db.StringProperty()
    3939    test_type = db.StringProperty()
    40     blob_key = db.StringProperty()
    41     date = db.DateTimeProperty(auto_now_add=True)
    4240
    4341    @classmethod
    … …  
    6462
    6563    @classmethod
    66     def get_files(cls, builder, test_type, name, limit):
     64    def get_files(cls, builder, test_type, name, load_data=True, limit=1):
    6765        query = TestFile.all()
    6866        if builder:
    … …  
    7371            query = query.filter("name =", name)
    7472
    75         return query.order("-date").fetch(limit)
     73        files = query.order("-date").fetch(limit)
     74        if load_data:
     75            for file in files:
     76                file.load_data()
     77
     78        return files
    7679
    7780    @classmethod
    78     def add_file(cls, builder, test_type, blob_info):
     81    def add_file(cls, builder, test_type, name, data):
    7982        file = TestFile()
    8083        file.builder = builder
    8184        file.test_type = test_type
    82         file.name = blob_info.filename
    83         file.blob_key = str(blob_info.key())
    84         file.put()
     85        file.name = name
     86
     87        if not file.save(data):
     88            return None
    8589
    8690        logging.info(
    87             "File saved, builder: %s, test_type: %s, name: %s, blob key: %s.",
    88             builder, test_type, file.name, file.blob_key)
     91            "File saved, builder: %s, test_type: %s, name: %s, key: %s.",
     92            builder, test_type, file.name, str(file.data_keys))
    8993
    9094        return file
    9195
    9296    @classmethod
    93     def update_file(cls, builder, test_type, blob_info):
    94         files = cls.get_files(builder, test_type, blob_info.filename, 1)
     97    def update(cls, builder, test_type, name, data):
     98        files = cls.get_files(builder, test_type, name)
    9599        if not files:
    96             return cls.add_file(builder, test_type, blob_info)
     100            return cls.add_file(builder, test_type, name, data)
    97101
    98102        file = files[0]
    99         old_blob_info = blobstore.BlobInfo.get(file.blob_key)
    100         if old_blob_info:
    101             old_blob_info.delete()
    102 
    103         file.builder = builder
    104         file.test_type = test_type
    105         file.name = blob_info.filename
    106         file.blob_key = str(blob_info.key())
    107         file.date = datetime.now()
    108         file.put()
     103        if not file.save(data):
     104            return None
    109105
    110106        logging.info(
    111             "File replaced, builder: %s, test_type: %s, name: %s, blob key: %s.",
    112             builder, test_type, file.name, file.blob_key)
     107            "File replaced, builder: %s, test_type: %s, name: %s, data key: %s.",
     108            builder, test_type, file.name, str(file.data_keys))
    113109
    114110        return file
    115111
     112    def save(self, data):
     113        if not self.save_data(data):
     114            return False
     115
     116        self.date = datetime.now()
     117        self.put()
     118
     119        return True
     120
    116121    def _delete_all(self):
    117         if self.blob_key:
    118             blob_info = blobstore.BlobInfo.get(self.blob_key)
    119             if blob_info:
    120                 blob_info.delete()
    121 
     122        self.delete_data()
    122123        self.delete()

  • trunk/WebKitTools/TestResultServer/templates/uploadform.html

    r56635 r64687  
    1919    </tr>
    2020    </table>
     21    <br>
     22    <div><input class=button type="checkbox" name="incremental">Incremental results, merge with server file.</div>
     23    <br>
    2124    <div><input class=button type="file" name="file" multiple></div>
    2225    <br>
Note: See TracChangeset for help on using the changeset viewer.