Changeset 140513 in webkit


Ignore:
Timestamp:
Jan 22, 2013 11:36:10 PM (11 years ago)
Author:
commit-queue@webkit.org
Message:

Add monitoring of patches and queues to the QueueStatusServer
​https://bugs.webkit.org/show_bug.cgi?id=107612

Patch by Alan Cutter <​alancutter@chromium.org> on 2013-01-22
Reviewed by Adam Barth.

Created classes for recording events into the datastore and integrated them into the existing handlers.
Code for presenting the recorded data will come in a separate patch.

  • QueueStatusServer/app.yaml:
  • QueueStatusServer/config/init.py: Added.
  • QueueStatusServer/config/logging.py: Copied from Tools/QueueStatusServer/model/queuestatus.py.
  • QueueStatusServer/config/messages.py: Copied from Tools/QueueStatusServer/model/queuestatus.py.
  • QueueStatusServer/handlers/nextpatch.py:

(NextPatch.get):
(NextPatch._assign_patch):

  • QueueStatusServer/handlers/releasepatch.py:

(ReleasePatch.post):

  • QueueStatusServer/handlers/updatestatus.py:

(UpdateStatus.post):

  • QueueStatusServer/handlers/updateworkitems.py:

(UpdateWorkItems._parse_work_items_string):
(UpdateWorkItems):
(UpdateWorkItems._update_work_items_from_request):
(UpdateWorkItems._queue_from_request):
(UpdateWorkItems.post):

  • QueueStatusServer/loggers/init.py: Added.
  • QueueStatusServer/loggers/recordbotevent.py: Copied from Tools/QueueStatusServer/model/queuestatus.py.

(RecordBotEvent):
(RecordBotEvent.activity):

  • QueueStatusServer/loggers/recordpatchevent.py: Added.

(RecordPatchEvent):
(RecordPatchEvent.added):
(RecordPatchEvent.retrying):
(RecordPatchEvent.started):
(RecordPatchEvent.stopped):
(RecordPatchEvent.updated):
(RecordPatchEvent._get_patches_waiting):

  • QueueStatusServer/model/patchlog.py: Copied from Tools/QueueStatusServer/model/queuestatus.py.

(PatchLog):
(PatchLog.lookup):
(PatchLog.calculate_wait_duration):
(PatchLog.calculate_process_duration):

  • QueueStatusServer/model/queuelog.py: Copied from Tools/QueueStatusServer/handlers/nextpatch.py.

(QueueLog):
(QueueLog.get_current):
(QueueLog.create_key):

  • QueueStatusServer/model/queuestatus.py:

(QueueStatus.is_retry_request):

Location:
trunk/Tools
Files:
5 added
7 edited
5 copied

Legend:

Unmodified
Added
Removed

  • trunk/Tools/ChangeLog

    r140511 r140513  
     12013-01-22  Alan Cutter  <alancutter@chromium.org>
     2
     3        Add monitoring of patches and queues to the QueueStatusServer
     4        https://bugs.webkit.org/show_bug.cgi?id=107612
     5
     6        Reviewed by Adam Barth.
     7
     8        Created classes for recording events into the datastore and integrated them into the existing handlers.
     9        Code for presenting the recorded data will come in a separate patch.
     10
     11        * QueueStatusServer/app.yaml:
     12        * QueueStatusServer/config/__init__.py: Added.
     13        * QueueStatusServer/config/logging.py: Copied from Tools/QueueStatusServer/model/queuestatus.py.
     14        * QueueStatusServer/config/messages.py: Copied from Tools/QueueStatusServer/model/queuestatus.py.
     15        * QueueStatusServer/handlers/nextpatch.py:
     16        (NextPatch.get):
     17        (NextPatch._assign_patch):
     18        * QueueStatusServer/handlers/releasepatch.py:
     19        (ReleasePatch.post):
     20        * QueueStatusServer/handlers/updatestatus.py:
     21        (UpdateStatus.post):
     22        * QueueStatusServer/handlers/updateworkitems.py:
     23        (UpdateWorkItems._parse_work_items_string):
     24        (UpdateWorkItems):
     25        (UpdateWorkItems._update_work_items_from_request):
     26        (UpdateWorkItems._queue_from_request):
     27        (UpdateWorkItems.post):
     28        * QueueStatusServer/loggers/__init__.py: Added.
     29        * QueueStatusServer/loggers/recordbotevent.py: Copied from Tools/QueueStatusServer/model/queuestatus.py.
     30        (RecordBotEvent):
     31        (RecordBotEvent.activity):
     32        * QueueStatusServer/loggers/recordpatchevent.py: Added.
     33        (RecordPatchEvent):
     34        (RecordPatchEvent.added):
     35        (RecordPatchEvent.retrying):
     36        (RecordPatchEvent.started):
     37        (RecordPatchEvent.stopped):
     38        (RecordPatchEvent.updated):
     39        (RecordPatchEvent._get_patches_waiting):
     40        * QueueStatusServer/model/patchlog.py: Copied from Tools/QueueStatusServer/model/queuestatus.py.
     41        (PatchLog):
     42        (PatchLog.lookup):
     43        (PatchLog.calculate_wait_duration):
     44        (PatchLog.calculate_process_duration):
     45        * QueueStatusServer/model/queuelog.py: Copied from Tools/QueueStatusServer/handlers/nextpatch.py.
     46        (QueueLog):
     47        (QueueLog.get_current):
     48        (QueueLog.create_key):
     49        * QueueStatusServer/model/queuestatus.py:
     50        (QueueStatus.is_retry_request):
     51
    1522013-01-22  Timothy Loh  <timloh@chromium.com>
    253

  • trunk/Tools/QueueStatusServer/app.yaml

    r70028 r140513  
    11application: webkit-commit-queue
    2 version: 1
     2version: 107612 # Bugzilla bug ID of last major change
    33runtime: python
    44api_version: 1

  • trunk/Tools/QueueStatusServer/config/logging.py

    r140512 r140513  
    1 # Copyright (C) 2009 Google Inc. All rights reserved.
     1# Copyright (C) 2013 Google Inc. All rights reserved.
    22#
    33# Redistribution and use in source and binary forms, with or without
    44# modification, are permitted provided that the following conditions are
    55# met:
    6 # 
     6#
    77#     * Redistributions of source code must retain the above copyright
    88# notice, this list of conditions and the following disclaimer.
    … …  
    1414# contributors may be used to endorse or promote products derived from
    1515# this software without specific prior written permission.
    16 # 
     16#
    1717# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
    1818# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
    … …  
    2727# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
    2828
    29 from google.appengine.ext import db
    30 from model.queuepropertymixin import QueuePropertyMixin
    31 
    32 
    33 class QueueStatus(db.Model, QueuePropertyMixin):
    34     author = db.UserProperty()
    35     queue_name = db.StringProperty()
    36     bot_id = db.StringProperty()
    37     active_bug_id = db.IntegerProperty()
    38     active_patch_id = db.IntegerProperty()
    39     message = db.StringProperty(multiline=True)
    40     date = db.DateTimeProperty(auto_now_add=True)
    41     results_file = db.BlobProperty()
    42 
    43     def is_retry_request(self):
    44         return self.message == "Retry"  # From AbstractQueue._retry_status
     29# Specified in seconds
     30queue_log_duration = 60 * 60

  • trunk/Tools/QueueStatusServer/config/messages.py

    r140512 r140513  
    1 # Copyright (C) 2009 Google Inc. All rights reserved.
     1# Copyright (C) 2013 Google Inc. All rights reserved.
    22#
    33# Redistribution and use in source and binary forms, with or without
    44# modification, are permitted provided that the following conditions are
    55# met:
    6 # 
     6#
    77#     * Redistributions of source code must retain the above copyright
    88# notice, this list of conditions and the following disclaimer.
    … …  
    1414# contributors may be used to endorse or promote products derived from
    1515# this software without specific prior written permission.
    16 # 
     16#
    1717# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
    1818# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
    … …  
    2727# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
    2828
    29 from google.appengine.ext import db
    30 from model.queuepropertymixin import QueuePropertyMixin
    31 
    32 
    33 class QueueStatus(db.Model, QueuePropertyMixin):
    34     author = db.UserProperty()
    35     queue_name = db.StringProperty()
    36     bot_id = db.StringProperty()
    37     active_bug_id = db.IntegerProperty()
    38     active_patch_id = db.IntegerProperty()
    39     message = db.StringProperty(multiline=True)
    40     date = db.DateTimeProperty(auto_now_add=True)
    41     results_file = db.BlobProperty()
    42 
    43     def is_retry_request(self):
    44         return self.message == "Retry"  # From AbstractQueue._retry_status
     29# These must be in sync with webkit-patch's AbstractQueue.
     30pass_status = "Pass"
     31fail_status = "Fail"
     32retry_status = "Retry"
     33error_status = "Error"

  • trunk/Tools/QueueStatusServer/handlers/nextpatch.py

    r70088 r140513  
    1 # Copyright (C) 2010 Google Inc. All rights reserved.
     1# Copyright (C) 2013 Google Inc. All rights reserved.
    22#
    33# Redistribution and use in source and binary forms, with or without
    … …  
    3232from google.appengine.ext import webapp
    3333
     34from loggers.recordpatchevent import RecordPatchEvent
    3435from model.queues import Queue
    3536
    … …  
    4849            self.error(404)
    4950            return
     51        RecordPatchEvent.started(patch_id, queue_name)
    5052        self.response.out.write(patch_id)
    5153
    5254    @staticmethod
    5355    def _assign_patch(key, work_item_ids):
    54         now = datetime.now()
     56        now = datetime.utcnow()
    5557        active_work_items = db.get(key)
    5658        active_work_items.deactivate_expired(now)

  • trunk/Tools/QueueStatusServer/handlers/releasepatch.py

    r70092 r140513  
    1 # Copyright (C) 2010 Google Inc. All rights reserved.
     1# Copyright (C) 2013 Google Inc. All rights reserved.
    22#
    33# Redistribution and use in source and binary forms, with or without
    … …  
    3131
    3232from handlers.updatebase import UpdateBase
     33from loggers.recordpatchevent import RecordPatchEvent
    3334from model.attachment import Attachment
    3435from model.queues import Queue
    … …  
    5859        if not last_status or not last_status.is_retry_request():
    5960            queue.work_items().remove_work_item(attachment_id)
     61            RecordPatchEvent.stopped(attachment_id, queue_name)
     62        else:
     63            RecordPatchEvent.retrying(attachment_id, queue_name)
    6064
    6165        # Always release the lock on the item.

  • trunk/Tools/QueueStatusServer/handlers/updatestatus.py

    r70088 r140513  
    1 # Copyright (C) 2009 Google Inc. All rights reserved.
     1# Copyright (C) 2013 Google Inc. All rights reserved.
    22#
    33# Redistribution and use in source and binary forms, with or without
    … …  
    3232
    3333from handlers.updatebase import UpdateBase
     34from loggers.recordbotevent import RecordBotEvent
     35from loggers.recordpatchevent import RecordPatchEvent
    3436from model.attachment import Attachment
    3537from model.queuestatus import QueueStatus
    … …  
    6365        queue_status = self._queue_status_from_request()
    6466        queue_status.put()
     67        RecordBotEvent.record_activity(queue_status.queue_name, queue_status.bot_id)
     68        if queue_status.active_patch_id:
     69            RecordPatchEvent.updated(queue_status.active_patch_id, queue_status.queue_name, queue_status.bot_id)
    6570        Attachment.dirty(queue_status.active_patch_id)
    6671        self.response.out.write(queue_status.key().id())

  • trunk/Tools/QueueStatusServer/handlers/updateworkitems.py

    r70028 r140513  
    1 # Copyright (C) 2010 Google Inc. All rights reserved.
     1# Copyright (C) 2013 Google Inc. All rights reserved.
    22#
    33# Redistribution and use in source and binary forms, with or without
    … …  
    3131
    3232from handlers.updatebase import UpdateBase
     33from loggers.recordpatchevent import RecordPatchEvent
    3334from model.queues import Queue
    3435from model.workitems import WorkItems
    … …  
    4243
    4344    def _parse_work_items_string(self, items_string):
    44         # Our parsing could be much more robust.
    45         item_strings = items_string.split(" ") if items_string else []
    46         return map(int, item_strings)
     45        try:
     46            item_strings = items_string.split(" ") if items_string else []
     47            return map(int, item_strings)
     48        except ValueError:
     49            return None
    4750
    48     def _work_items_from_request(self):
     51    def _update_work_items_from_request(self, work_items):
     52        items_string = self.request.get("work_items")
     53        new_work_items = self._parse_work_items_string(items_string)
     54        if new_work_items == None:
     55            self.response.out.write("Failed to parse work items: %s" % items_string)
     56            return False
     57        work_items.item_ids = new_work_items
     58        work_items.date = datetime.utcnow()
     59        return True
     60
     61    def _queue_from_request(self):
    4962        queue_name = self.request.get("queue_name")
    5063        queue = Queue.queue_with_name(queue_name)
    … …  
    5265            self.response.out.write("\"%s\" is not in queues %s" % (queue_name, Queue.all()))
    5366            return None
    54 
    55         items_string = self.request.get("work_items")
    56         work_items = queue.work_items()
    57         work_items.item_ids = self._parse_work_items_string(items_string)
    58         work_items.date = datetime.now()
    59         return work_items
     67        return queue
    6068
    6169    def post(self):
    62         work_items = self._work_items_from_request()
    63         if not work_items:
     70        queue = self._queue_from_request()
     71        if not queue:
    6472            self.response.set_status(500)
    6573            return
     74        work_items = queue.work_items()
     75        old_items = set(work_items.item_ids)
     76
     77        success = self._update_work_items_from_request(work_items)
     78        if not success:
     79            self.response.set_status(500)
     80            return
     81        new_items = set(work_items.item_ids)
    6682        work_items.put()
     83
     84        for work_item in new_items - old_items:
     85            RecordPatchEvent.added(work_item, queue.name())
     86        for work_item in old_items - new_items:
     87            RecordPatchEvent.stopped(work_item, queue.name())

  • trunk/Tools/QueueStatusServer/loggers/recordbotevent.py

    r140512 r140513  
    1 # Copyright (C) 2009 Google Inc. All rights reserved.
     1# Copyright (C) 2013 Google Inc. All rights reserved.
    22#
    33# Redistribution and use in source and binary forms, with or without
    44# modification, are permitted provided that the following conditions are
    55# met:
    6 # 
     6#
    77#     * Redistributions of source code must retain the above copyright
    88# notice, this list of conditions and the following disclaimer.
    … …  
    1414# contributors may be used to endorse or promote products derived from
    1515# this software without specific prior written permission.
    16 # 
     16#
    1717# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
    1818# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
    … …  
    2727# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
    2828
    29 from google.appengine.ext import db
    30 from model.queuepropertymixin import QueuePropertyMixin
     29from config.logging import queue_log_duration
     30from model.queuelog import QueueLog
    3131
    3232
    33 class QueueStatus(db.Model, QueuePropertyMixin):
    34     author = db.UserProperty()
    35     queue_name = db.StringProperty()
    36     bot_id = db.StringProperty()
    37     active_bug_id = db.IntegerProperty()
    38     active_patch_id = db.IntegerProperty()
    39     message = db.StringProperty(multiline=True)
    40     date = db.DateTimeProperty(auto_now_add=True)
    41     results_file = db.BlobProperty()
    42 
    43     def is_retry_request(self):
    44         return self.message == "Retry"  # From AbstractQueue._retry_status
     33class RecordBotEvent(object):
     34    @classmethod
     35    def record_activity(cls, queue_name, bot_id):
     36        queue_log = QueueLog.get_current(queue_name, queue_log_duration)
     37        if queue_log and bot_id not in queue_log.bot_ids_seen:
     38            queue_log.bot_ids_seen.append(bot_id)
     39            queue_log.put()

  • trunk/Tools/QueueStatusServer/model/patchlog.py

    r140512 r140513  
    1 # Copyright (C) 2009 Google Inc. All rights reserved.
     1# Copyright (C) 2013 Google Inc. All rights reserved.
    22#
    33# Redistribution and use in source and binary forms, with or without
    44# modification, are permitted provided that the following conditions are
    55# met:
    6 # 
     6#
    77#     * Redistributions of source code must retain the above copyright
    88# notice, this list of conditions and the following disclaimer.
    … …  
    1414# contributors may be used to endorse or promote products derived from
    1515# this software without specific prior written permission.
    16 # 
     16#
    1717# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
    1818# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
    … …  
    2727# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
    2828
     29from datetime import datetime
     30
    2931from google.appengine.ext import db
    30 from model.queuepropertymixin import QueuePropertyMixin
    3132
    3233
    33 class QueueStatus(db.Model, QueuePropertyMixin):
    34     author = db.UserProperty()
     34class PatchLog(db.Model):
     35    attachment_id = db.IntegerProperty()
    3536    queue_name = db.StringProperty()
     37    date = db.DateTimeProperty(auto_now_add=True)
    3638    bot_id = db.StringProperty()
    37     active_bug_id = db.IntegerProperty()
    38     active_patch_id = db.IntegerProperty()
    39     message = db.StringProperty(multiline=True)
    40     date = db.DateTimeProperty(auto_now_add=True)
    41     results_file = db.BlobProperty()
     39    retry_count = db.IntegerProperty(default=0)
     40    status_update_count = db.IntegerProperty(default=0)
     41    finished = db.BooleanProperty(default=False)
     42    wait_duration = db.IntegerProperty()
     43    process_duration = db.IntegerProperty()
    4244
    43     def is_retry_request(self):
    44         return self.message == "Retry"  # From AbstractQueue._retry_status
     45    @classmethod
     46    def lookup(cls, attachment_id, queue_name):
     47        key = "%s-%s" % (attachment_id, queue_name)
     48        return cls.get_or_insert(key, attachment_id=attachment_id, queue_name=queue_name)
     49
     50    def calculate_wait_duration(self):
     51        time_delta = datetime.utcnow() - self.date
     52        self.wait_duration = int(time_delta.total_seconds())
     53
     54    def calculate_process_duration(self):
     55        if self.wait_duration:
     56            time_delta = datetime.utcnow() - self.date
     57            self.process_duration = int(time_delta.total_seconds()) - self.wait_duration

  • trunk/Tools/QueueStatusServer/model/queuelog.py

    r140512 r140513  
    1 # Copyright (C) 2010 Google Inc. All rights reserved.
     1# Copyright (C) 2013 Google Inc. All rights reserved.
    22#
    33# Redistribution and use in source and binary forms, with or without
    … …  
    2727# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
    2828
     29from time import time
    2930from datetime import datetime
    3031
    3132from google.appengine.ext import db
    32 from google.appengine.ext import webapp
    33 
    34 from model.queues import Queue
    3533
    3634
    37 class NextPatch(webapp.RequestHandler):
    38     # FIXME: This should probably be a post, or an explict lock_patch
    39     # since GET requests shouldn't really modify the datastore.
    40     def get(self, queue_name):
    41         queue = Queue.queue_with_name(queue_name)
    42         if not queue:
    43             self.error(404)
    44             return
    45         # FIXME: Patch assignment should probably move into Queue.
    46         patch_id = db.run_in_transaction(self._assign_patch, queue.active_work_items().key(), queue.work_items().item_ids)
    47         if not patch_id:
    48             self.error(404)
    49             return
    50         self.response.out.write(patch_id)
     35class QueueLog(db.Model):
     36    date = db.DateTimeProperty()
     37    # duration specifies in seconds the time period these log values apply to.
     38    duration = db.IntegerProperty()
     39    queue_name = db.StringProperty()
     40    bot_ids_seen = db.StringListProperty()
     41    max_patches_waiting = db.IntegerProperty(default=0)
     42    patch_wait_durations = db.ListProperty(int)
     43    patch_process_durations = db.ListProperty(int)
     44    patch_retry_count = db.IntegerProperty(default=0)
     45    status_update_count = db.IntegerProperty(default=0)
     46
     47    @classmethod
     48    def get_current(cls, queue_name, duration):
     49        timestamp_now = time()
     50        timestamp = int(timestamp_now / duration) * duration
     51        date = datetime.utcfromtimestamp(timestamp)
     52        key = cls.create_key(queue_name, duration, timestamp)
     53        return cls.get_or_insert(key, date=date, duration=duration, queue_name=queue_name)
    5154
    5255    @staticmethod
    53     def _assign_patch(key, work_item_ids):
    54         now = datetime.now()
    55         active_work_items = db.get(key)
    56         active_work_items.deactivate_expired(now)
    57         next_item = active_work_items.next_item(work_item_ids, now)
    58         active_work_items.put()
    59         return next_item
     56    def create_key(queue_name, duration, timestamp):
     57        return "%s-%s-%s" % (queue_name, duration, timestamp)

  • trunk/Tools/QueueStatusServer/model/queuestatus.py

    r70019 r140513  
    1 # Copyright (C) 2009 Google Inc. All rights reserved.
     1# Copyright (C) 2013 Google Inc. All rights reserved.
    22#
    33# Redistribution and use in source and binary forms, with or without
    … …  
    2727# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
    2828
     29from config import messages
    2930from google.appengine.ext import db
    3031from model.queuepropertymixin import QueuePropertyMixin
    … …  
    4243
    4344    def is_retry_request(self):
    44         return self.message == "Retry"  # From AbstractQueue._retry_status
     45        return self.message == messages.retry_status
Note: See TracChangeset for help on using the changeset viewer.