Changeset 90823 in webkit


Ignore:
Timestamp:
Jul 12, 2011 10:49:27 AM (13 years ago)
Author:
abarth@webkit.org
Message:

cr-linux-ews complains about tests that aren't actually failing
https://bugs.webkit.org/show_bug.cgi?id=64367

The underlying problem here is that full_results.json doesn't have
enough information to determine whether a given test result was
expected because whether an actual result is expected depends on
whether full_results.json was generated duing a test run that included
pixel tests.

The right long-term solution is to make full_results.json a complete
description of what happened durning a test run. In this patch
however, to stop the spam, this patch makes the jsonresultsparser
ignore pixel failures.

(I'm landing this patch unreviewed in order to stop the bots from
spamming. I'll happily address any review comments in a follow-up
patch.)

  • Scripts/webkitpy/common/net/resultsjsonparser.py:
  • Scripts/webkitpy/common/net/resultsjsonparser_unittest.py:
Location:
trunk/Tools
Files:
4 edited

Legend:

Unmodified
Added
Removed
  • trunk/Tools/ChangeLog

    r90822 r90823  
     12011-07-12  Adam Barth  <abarth@webkit.org>
     2
     3        cr-linux-ews complains about tests that aren't actually failing
     4        https://bugs.webkit.org/show_bug.cgi?id=64367
     5
     6        The underlying problem here is that full_results.json doesn't have
     7        enough information to determine whether a given test result was
     8        expected because whether an actual result is expected depends on
     9        whether full_results.json was generated duing a test run that included
     10        pixel tests.
     11
     12        The right long-term solution is to make full_results.json a complete
     13        description of what happened durning a test run.  In this patch
     14        however, to stop the spam, this patch makes the jsonresultsparser
     15        ignore pixel failures.
     16
     17        (I'm landing this patch unreviewed in order to stop the bots from
     18        spamming.  I'll happily address any review comments in a follow-up
     19        patch.)
     20
     21        * Scripts/webkitpy/common/net/resultsjsonparser.py:
     22        * Scripts/webkitpy/common/net/resultsjsonparser_unittest.py:
     23
    1242011-07-12  Adam Barth  <abarth@webkit.org>
    225
  • trunk/Tools/Scripts/webkitpy/common/net/resultsjsonparser.py

    r90539 r90823  
    7474        actual_results = self._actual_as_tokens()
    7575        expected_results = self._expected_as_tokens()
     76        # FIXME: We should only remove_pixel_failures this JSONResult can from a run without pixel tests!
     77        if not test_expectations.has_pixel_failures(actual_results):
     78            expected_results = test_expectations.remove_pixel_failures(expected_results)
    7679        for actual_result in actual_results:
    7780            if not test_expectations.result_was_expected(actual_result, expected_results, False, False):
     
    8386        if None in tokens:
    8487            log("Unrecognized result in %s" % results_string)
    85         return tokens
     88        return set(tokens)
    8689
    8790    @memoized
  • trunk/Tools/Scripts/webkitpy/common/net/resultsjsonparser_unittest.py

    r90546 r90823  
    5959                    "expected": "PASS",
    6060                    "actual": "TEXT PASS"
     61                },
     62                "prototype-peach.html": {
     63                    "expected": "IMAGE+TEXT",
     64                    "actual": "TEXT"
    6165                }
    6266            }
  • trunk/Tools/Scripts/webkitpy/layout_tests/models/test_expectations.py

    r90547 r90823  
    8585
    8686
     87def has_pixel_failures(actual_results):
     88    return IMAGE in actual_results or IMAGE_PLUS_TEXT in actual_results
     89
     90
    8791def strip_comments(line):
    8892    """Strips comments from a line and return None if the line is empty
Note: See TracChangeset for help on using the changeset viewer.