Changeset 173979 in webkit


Ignore:
Timestamp:
Sep 25, 2014 4:16:57 PM (10 years ago)
Author:
ap@apple.com
Message:

EWS only repeats its cycle every two hours
https://bugs.webkit.org/show_bug.cgi?id=137129

Reviewed by Ryosuke Niwa.

  • QueueStatusServer/app.yaml: Updated version.
  • QueueStatusServer/config/queues.py: Moved timeout from activeworkitems.py to configuration.
  • QueueStatusServer/handlers/releaselock.py: Added. Releases the lock without removing

the patch from work items.

  • QueueStatusServer/index.yaml: No real change, just let AppEngine have its way with entry order.
  • QueueStatusServer/main.py: Added release-lock.
  • QueueStatusServer/model/activeworkitems.py:

(ActiveWorkItems.deactivate_expired): Use timeout from configuration.

  • QueueStatusServer/model/workitems.py: Added move_to_end. When we unlock a patch,

we don't want it to be immediately picked up again, it's better to give other patches
a chance.

  • QueueStatusServer/templates/releaselock.html: Added. Not sure why all commands have

these interactive versions, but OK.

  • Scripts/webkitpy/tool/commands/earlywarningsystem.py: (AbstractEarlyWarningSystem.review_patch):

Unlock the patch when a non-final failure occurs (e.g. can't build even without the patch,
or svn is down).

  • Scripts/webkitpy/tool/commands/queues.py:

(AbstractReviewQueue.process_work_item): Do not try/catch ScriptError around review_patch.
Style queue never raises these, and EWS already calls _did_fail, before re-throwing,
meaning that these handlers could never do the right thing. We'd either get a duplicate
_did_fail, or try to unlock an already unlocked patch.
(StyleQueue.review_patch): Unlock the patch on transient failure (such as svn failure),
making it eligible for retry immediately.

  • Scripts/webkitpy/common/net/statusserver.py:

(StatusServer._post_release_lock):
(StatusServer.release_lock):
Added a call to release-lock.

Location:
trunk/Tools
Files:
2 added
10 edited

Legend:

Unmodified
Added
Removed
  • trunk/Tools/ChangeLog

    r173976 r173979  
     12014-09-25  Alexey Proskuryakov  <ap@apple.com>
     2
     3        EWS only repeats its cycle every two hours
     4        https://bugs.webkit.org/show_bug.cgi?id=137129
     5
     6        Reviewed by Ryosuke Niwa.
     7
     8        * QueueStatusServer/app.yaml: Updated version.
     9
     10        * QueueStatusServer/config/queues.py: Moved timeout from activeworkitems.py to configuration.
     11
     12        * QueueStatusServer/handlers/releaselock.py: Added. Releases the lock without removing
     13        the patch from work items.
     14
     15        * QueueStatusServer/index.yaml: No real change, just let AppEngine have its way with entry order.
     16
     17        * QueueStatusServer/main.py: Added release-lock.
     18
     19        * QueueStatusServer/model/activeworkitems.py:
     20        (ActiveWorkItems.deactivate_expired): Use timeout from configuration.
     21
     22        * QueueStatusServer/model/workitems.py: Added move_to_end. When we unlock a patch,
     23        we don't want it to be immediately picked up again, it's better to give other patches
     24        a chance.
     25
     26        * QueueStatusServer/templates/releaselock.html: Added. Not sure why all commands have
     27        these interactive versions, but OK.
     28
     29        * Scripts/webkitpy/tool/commands/earlywarningsystem.py: (AbstractEarlyWarningSystem.review_patch):
     30        Unlock the patch when a non-final failure occurs (e.g. can't build even without the patch,
     31        or svn is down).
     32
     33        * Scripts/webkitpy/tool/commands/queues.py:
     34        (AbstractReviewQueue.process_work_item): Do not try/catch ScriptError around review_patch.
     35        Style queue never raises these, and EWS already calls _did_fail, before re-throwing,
     36        meaning that these handlers could never do the right thing. We'd either get a duplicate
     37        _did_fail, or try to unlock an already unlocked patch.
     38        (StyleQueue.review_patch): Unlock the patch on transient failure (such as svn failure),
     39        making it eligible for retry immediately.
     40
     41        * Scripts/webkitpy/common/net/statusserver.py:
     42        (StatusServer._post_release_lock):
     43        (StatusServer.release_lock):
     44        Added a call to release-lock.
     45
    1462014-09-25  Roger Fong  <roger_fong@apple.com>
    247
  • trunk/Tools/QueueStatusServer/app.yaml

    r173587 r173979  
    11application: webkit-queues
    2 version: 173587 # Bugzilla bug ID of last major change
     2version: 173979 # Bugzilla bug ID of last major change
    33runtime: python
    44api_version: 1
  • trunk/Tools/QueueStatusServer/config/queues.py

    r170172 r173979  
     1# Copyright (C) 2014 Apple Inc. All rights reserved.
    12# Copyright (C) 2013 Google Inc. All rights reserved.
    23#
     
    2728# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
    2829
     30from datetime import timedelta
     31
    2932# Eventually the list of queues may be stored in the data store.
    3033all_queue_names = [
     
    3740    "efl-wk2-ews",
    3841]
     42
     43# If the patch is still active after this much time, then a bot must have frozen or rebooted,
     44# and dropped the patch on the floor. We will ignore the lock in this case, and let another bot pick up.
     45work_item_lock_timeout = timedelta(minutes=120)
  • trunk/Tools/QueueStatusServer/index.yaml

    r161062 r173979  
    2626  - name: active_patch_id
    2727  - name: queue_name
     28  - name: date
     29    direction: desc
     30
     31- kind: QueueStatus
     32  properties:
     33  - name: bot_id
    2834  - name: date
    2935    direction: desc
     
    6470- kind: QueueStatus
    6571  properties:
    66   - name: bot_id
    67   - name: date
    68     direction: desc
    69 
    70 - kind: QueueStatus
    71   properties:
    7272  - name: queue_name
    7373  - name: date
  • trunk/Tools/QueueStatusServer/main.py

    r162358 r173979  
    4646from handlers.recentstatus import QueuesOverview
    4747from handlers.releasepatch import ReleasePatch
     48from handlers.releaselock import ReleaseLock
    4849from handlers.showresults import ShowResults
    4950from handlers.statusbubble import StatusBubble
     
    7677    (r'/next-patch/(.*)', NextPatch),
    7778    (r'/release-patch', ReleasePatch),
     79    (r'/release-lock', ReleaseLock),
    7880    ('/update-status', UpdateStatus),
    7981    ('/update-work-items', UpdateWorkItems),
  • trunk/Tools/QueueStatusServer/model/activeworkitems.py

    r173587 r173979  
    3232import time
    3333
     34from config.queues import work_item_lock_timeout
    3435from model.queuepropertymixin import QueuePropertyMixin
    3536
     
    8283        # If the patch is still active after this much time, then a bot must have frozen or rebooted,
    8384        # and dropped the patch on the floor. Let another bot pick it up.
    84         two_hours_ago = time.mktime((now - timedelta(minutes=120)).timetuple())
    85         nonexpired_pairs = [pair for pair in self._item_time_pairs() if pair[1] > two_hours_ago]
     85        cutoff_time = time.mktime((now - work_item_lock_timeout).timetuple())
     86        nonexpired_pairs = [pair for pair in self._item_time_pairs() if pair[1] > cutoff_time]
    8687        self._set_item_time_pairs(nonexpired_pairs)
    8788
  • trunk/Tools/QueueStatusServer/model/workitems.py

    r84216 r173979  
    7676    def remove_work_item(self, attachment_id):
    7777        db.run_in_transaction(self._unguarded_remove, self.key(), attachment_id)
     78
     79    @staticmethod
     80    def _unguarded_move_to_end(key, attachment_id):
     81        work_items = db.get(key)
     82        if attachment_id in work_items.item_ids:
     83            # We should never have more than one entry for a work item, so we only need remove the first.
     84            work_items.item_ids.remove(attachment_id)
     85            work_items.item_ids.append(attachment_id)
     86        work_items.put()
     87
     88    def move_to_end(self, attachment_id):
     89        db.run_in_transaction(self._unguarded_move_to_end, self.key(), attachment_id)
  • trunk/Tools/Scripts/webkitpy/common/net/statusserver.py

    r162357 r173979  
    138138        return NetworkTransaction(convert_404_to_None=True).run(lambda: self._post_release_work_item(queue_name, patch))
    139139
     140    def _post_release_lock(self, queue_name, patch):
     141        release_lock_url = "%s/release-lock" % (self.url)
     142        self._browser.open(release_lock_url)
     143        self._browser.select_form(name="release_lock")
     144        self._browser["queue_name"] = queue_name
     145        self._browser["attachment_id"] = unicode(patch.id())
     146        self._browser.submit()
     147
     148    def release_lock(self, queue_name, patch):
     149        _log.info("Releasing lock for work item %s from %s" % (patch.id(), queue_name))
     150        return NetworkTransaction(convert_404_to_None=True).run(lambda: self._post_release_lock(queue_name, patch))
     151
    140152    def update_work_items(self, queue_name, work_items):
    141153        _log.debug("Recording work items: %s for %s" % (work_items, queue_name))
  • trunk/Tools/Scripts/webkitpy/tool/commands/earlywarningsystem.py

    r162159 r173979  
    8686            return False
    8787        try:
    88             return task.run()
     88            succeeded = task.run()
     89            if not succeeded:
     90                # Caller unlocks when review_patch returns True, so we only need to unlock on transient failure.
     91                self._unlock_patch(patch)
     92            return succeeded
    8993        except UnableToApplyPatch, e:
    9094            self._did_error(patch, "%s unable to apply patch." % self.name)
  • trunk/Tools/Scripts/webkitpy/tool/commands/queues.py

    r162159 r173979  
    249249        self._release_work_item(patch)
    250250
     251    def _unlock_patch(self, patch):
     252        self._tool.status_server.release_lock(self.name, patch)
     253
    251254    def work_item_log_path(self, patch):
    252255        return os.path.join(self._log_directory(), "%s.log" % patch.bug_id())
     
    426429
    427430    def process_work_item(self, patch):
    428         try:
    429             if not self.review_patch(patch):
    430                 return False
     431        passed = self.review_patch(patch)
     432        if passed:
    431433            self._did_pass(patch)
    432             return True
    433         except ScriptError, e:
    434             if e.exit_code != QueueEngine.handled_error_code:
    435                 self._did_fail(patch)
    436             else:
    437                 # The subprocess handled the error, but won't have released the patch, so we do.
    438                 # FIXME: We need to simplify the rules by which _release_work_item is called.
    439                 self._release_work_item(patch)
    440             raise e
     434        return passed
    441435
    442436    def handle_unexpected_error(self, patch, message):
     
    462456            return False
    463457        try:
    464             return task.run()
     458            style_check_succeeded = task.run()
     459            if not style_check_succeeded:
     460                # Caller unlocks when review_patch returns True, so we only need to unlock on transient failure.
     461                self._unlock_patch(patch)
     462            return style_check_succeeded
    465463        except UnableToApplyPatch, e:
    466464            self._did_error(patch, "%s unable to apply patch." % self.name)
Note: See TracChangeset for help on using the changeset viewer.