Changeset 191374 in webkit


Ignore:
Timestamp:
Oct 20, 2015 10:36:49 PM (9 years ago)
Author:
commit-queue@webkit.org
Message:

run-webkit-tests does not copy all crash logs for layout test failures on iOS
​https://bugs.webkit.org/show_bug.cgi?id=150056
Tools:

<rdar://problem/9280656>

Patch by Aakash Jain <​aakash_jain@apple.com> on 2015-10-20
Reviewed by Alexey Proskuryakov.

  • Scripts/webkitpy/common/system/crashlogs.py:

(CrashLogs.find_all_logs): Generic method to find all crash logs.
(CrashLogs._find_all_logs_darwin): Darwin based method to find all crash logs.
It iterates through log directory and returns all the logs based on timestamp.

  • Scripts/webkitpy/common/system/crashlogs_unittest.py:

(CrashLogsTest.create_crash_logs_darwin): Creates sample crash logs and verify them.
(CrashLogsTest.test_find_all_log_darwin): Testcase for above find_all_logs method
(CrashLogsTest.test_find_log_darwin): Restructured to share code with other methods.

  • Scripts/webkitpy/layout_tests/controllers/manager.py:

(Manager.run): Modified start_time to start counting before simulator launch
so that we can capture crashes during simualator launch.
(Manager._look_for_new_crash_logs): Browse through list of crashes and append
any test which is not already marked as CRASH to the run_results.

  • Scripts/webkitpy/layout_tests/models/test_expectations.py:

(TestExpectationsModel.get_expectations_string): return PASS in case there
are no expectations defined for this test.

  • Scripts/webkitpy/layout_tests/models/test_run_results.py:

(summarize_results): Add other_crashes in a separte category in full_results.json.

  • Scripts/webkitpy/port/ios.py:

(IOSSimulatorPort._merge_crash_logs): Merge unique crash logs from two dictionaries.
(IOSSimulatorPort._look_for_all_crash_logs_in_log_dir): Get the crash logs
from the log directory.
(IOSSimulatorPort.look_for_new_crash_logs): Uses above method to get crash
logs from log directory and merge them with the list of already crashed tests.

LayoutTests:

<rdar://problem/22239750>

Patch by Aakash Jain <​aakash_jain@apple.com> on 2015-10-20
Reviewed by Alexey Proskuryakov.

  • fast/harness/results.html: Added the column for Other crashes, this contain

all the newly find crashes from the crash-log directory. Added method forOtherCrashes
which processes othre_crashes section from full_results.json. Also fixed the method
splitExtension to handle the case when there is no extension.

Location:
trunk
Files:
10 edited

Legend:

Unmodified
Added
Removed

  • trunk/LayoutTests/ChangeLog

    r191364 r191374  
     12015-10-20  Aakash Jain  <aakash_jain@apple.com>
     2
     3        run-webkit-tests does not copy all crash logs for layout test failures on iOS
     4        https://bugs.webkit.org/show_bug.cgi?id=150056
     5        <rdar://problem/22239750>
     6
     7        Reviewed by Alexey Proskuryakov.
     8
     9        * fast/harness/results.html: Added the column for Other crashes, this contain
     10        all the newly find crashes from the crash-log directory. Added method forOtherCrashes
     11        which processes othre_crashes section from full_results.json. Also fixed the method
     12        splitExtension to handle the case when there is no extension.
     13
    1142015-10-20  Mark Lam  <mark.lam@apple.com>
    215

  • trunk/LayoutTests/fast/harness/results.html

    r187605 r191374  
    236236        g_state = {
    237237            crashTests: [],
     238            crashOther: [],
    238239            flakyPassTests: [],
    239240            hasHttpTests: false,
    … …  
    264265{
    265266    var index = test.lastIndexOf('.');
     267    if (index == -1) {
     268        return [test, ""];
     269    }
    266270    return [test.substring(0, index), test.substring(index + 1)];
    267271}
    … …  
    702706        } else
    703707            forEachTest(handler, tree[key], newPrefix);
     708    }
     709}
     710
     711function forOtherCrashes()
     712{
     713    var tree = globalState().results.other_crashes;
     714    for (var key in tree) {
     715            var testObject = tree[key];
     716            testObject.name = key;
     717            globalState().crashOther.push(testObject);
    704718    }
    705719}
    … …  
    13621376{
    13631377    forEachTest(processGlobalStateFor);
     1378    forOtherCrashes();
    13641379
    13651380    var html = "";
    … …  
    13701385    if (globalState().crashTests.length)
    13711386        html += testList(globalState().crashTests, 'Tests that crashed', 'crash-tests-table');
     1387
     1388    if (globalState().crashOther.length)
     1389        html += testList(globalState().crashOther, 'Other Crashes', 'crash-tests-table');
    13721390
    13731391    html += failingTestsTable(globalState().failingTests,

  • trunk/Tools/ChangeLog

    r191372 r191374  
     12015-10-20  Aakash Jain  <aakash_jain@apple.com>
     2
     3        run-webkit-tests does not copy all crash logs for layout test failures on iOS
     4        https://bugs.webkit.org/show_bug.cgi?id=150056
     5        <rdar://problem/9280656>
     6
     7        Reviewed by Alexey Proskuryakov.
     8
     9        * Scripts/webkitpy/common/system/crashlogs.py:
     10        (CrashLogs.find_all_logs): Generic method to find all crash logs.
     11        (CrashLogs._find_all_logs_darwin): Darwin based method to find all crash logs.
     12        It iterates through log directory and returns all the logs based on timestamp.
     13        * Scripts/webkitpy/common/system/crashlogs_unittest.py:
     14        (CrashLogsTest.create_crash_logs_darwin): Creates sample crash logs and verify them.
     15        (CrashLogsTest.test_find_all_log_darwin): Testcase for above find_all_logs method
     16        (CrashLogsTest.test_find_log_darwin): Restructured to share code with other methods.
     17        * Scripts/webkitpy/layout_tests/controllers/manager.py:
     18        (Manager.run): Modified start_time to start counting before simulator launch
     19        so that we can capture crashes during simualator launch.
     20        (Manager._look_for_new_crash_logs): Browse through list of crashes and append
     21        any test which is not already marked as CRASH to the run_results.
     22        * Scripts/webkitpy/layout_tests/models/test_expectations.py:
     23        (TestExpectationsModel.get_expectations_string): return PASS in case there
     24        are no expectations defined for this test.
     25        * Scripts/webkitpy/layout_tests/models/test_run_results.py:
     26        (summarize_results): Add other_crashes in a separte category in full_results.json.
     27        * Scripts/webkitpy/port/ios.py:
     28        (IOSSimulatorPort._merge_crash_logs): Merge unique crash logs from two dictionaries.
     29        (IOSSimulatorPort._look_for_all_crash_logs_in_log_dir): Get the crash logs
     30        from the log directory.
     31        (IOSSimulatorPort.look_for_new_crash_logs): Uses above method to get crash
     32        logs from log directory and merge them with the list of already crashed tests.
     33
    1342015-10-20  Dana Burkart  <dburkart@apple.com>
    235

  • trunk/Tools/Scripts/webkitpy/common/system/crashlogs.py

    r191018 r191374  
    4646        elif self._host.platform.is_win():
    4747            return self._find_newest_log_win(process_name, pid, include_errors, newer_than)
     48        return None
     49
     50    def find_all_logs(self, include_errors=False, newer_than=None):
     51        if self._host.platform.is_mac():
     52            return self._find_all_logs_darwin(include_errors, newer_than)
    4853        return None
    4954
    … …  
    119124            return errors
    120125        return None
     126
     127    def _find_all_logs_darwin(self, include_errors, newer_than):
     128        def is_crash_log(fs, dirpath, basename):
     129            return basename.endswith(".crash")
     130
     131        log_directory = self._log_directory_darwin()
     132        logs = self._host.filesystem.files_under(log_directory, file_filter=is_crash_log)
     133        first_line_regex = re.compile(r'^Process:\s+(?P<process_name>.*) \[(?P<pid>\d+)\]$')
     134        errors = ''
     135        crash_logs = {}
     136        for path in reversed(sorted(logs)):
     137            try:
     138                if not newer_than or self._host.filesystem.mtime(path) > newer_than:
     139                    result_name = "Unknown"
     140                    pid = 0
     141                    log_contents = self._host.filesystem.read_text_file(path)
     142                    match = first_line_regex.match(log_contents[0:log_contents.find('\n')])
     143                    if match:
     144                        process_name = match.group('process_name')
     145                        pid = str(match.group('pid'))
     146                        result_name = process_name + "-" + pid
     147
     148                    while result_name in crash_logs:
     149                        result_name = result_name + "-1"
     150                    crash_logs[result_name] = errors + log_contents
     151            except IOError, e:
     152                if include_errors:
     153                    errors += "ERROR: Failed to read '%s': %s\n" % (path, str(e))
     154            except OSError, e:
     155                if include_errors:
     156                    errors += "ERROR: Failed to read '%s': %s\n" % (path, str(e))
     157
     158        if include_errors and errors and len(crash_logs) == 0:
     159            return errors
     160        return crash_logs

  • trunk/Tools/Scripts/webkitpy/common/system/crashlogs_unittest.py

    r174136 r191374  
    234234
    235235class CrashLogsTest(unittest.TestCase):
    236     def test_find_log_darwin(self):
     236    def create_crash_logs_darwin(self):
    237237        if not SystemHost().platform.is_mac():
    238238            return
    239239
    240         older_mock_crash_report = make_mock_crash_report_darwin('DumpRenderTree', 28528)
    241         mock_crash_report = make_mock_crash_report_darwin('DumpRenderTree', 28530)
    242         newer_mock_crash_report = make_mock_crash_report_darwin('DumpRenderTree', 28529)
    243         other_process_mock_crash_report = make_mock_crash_report_darwin('FooProcess', 28527)
    244         misformatted_mock_crash_report = 'Junk that should not appear in a crash report' + make_mock_crash_report_darwin('DumpRenderTree', 28526)[200:]
    245         files = {}
    246         files['/Users/mock/Library/Logs/DiagnosticReports/DumpRenderTree_2011-06-13-150718_quadzen.crash'] = older_mock_crash_report
    247         files['/Users/mock/Library/Logs/DiagnosticReports/DumpRenderTree_2011-06-13-150719_quadzen.crash'] = mock_crash_report
    248         files['/Users/mock/Library/Logs/DiagnosticReports/DumpRenderTree_2011-06-13-150720_quadzen.crash'] = newer_mock_crash_report
    249         files['/Users/mock/Library/Logs/DiagnosticReports/DumpRenderTree_2011-06-13-150721_quadzen.crash'] = None
    250         files['/Users/mock/Library/Logs/DiagnosticReports/DumpRenderTree_2011-06-13-150722_quadzen.crash'] = other_process_mock_crash_report
    251         files['/Users/mock/Library/Logs/DiagnosticReports/DumpRenderTree_2011-06-13-150723_quadzen.crash'] = misformatted_mock_crash_report
    252         filesystem = MockFileSystem(files)
    253         crash_logs = CrashLogs(MockSystemHost(filesystem=filesystem))
     240        self.older_mock_crash_report = make_mock_crash_report_darwin('DumpRenderTree', 28528)
     241        self.mock_crash_report = make_mock_crash_report_darwin('DumpRenderTree', 28530)
     242        self.newer_mock_crash_report = make_mock_crash_report_darwin('DumpRenderTree', 28529)
     243        self.other_process_mock_crash_report = make_mock_crash_report_darwin('FooProcess', 28527)
     244        self.misformatted_mock_crash_report = 'Junk that should not appear in a crash report' + make_mock_crash_report_darwin('DumpRenderTree', 28526)[200:]
     245        self.files = {}
     246        self.files['/Users/mock/Library/Logs/DiagnosticReports/DumpRenderTree_2011-06-13-150718_quadzen.crash'] = self.older_mock_crash_report
     247        self.files['/Users/mock/Library/Logs/DiagnosticReports/DumpRenderTree_2011-06-13-150719_quadzen.crash'] = self.mock_crash_report
     248        self.files['/Users/mock/Library/Logs/DiagnosticReports/DumpRenderTree_2011-06-13-150720_quadzen.crash'] = self.newer_mock_crash_report
     249        self.files['/Users/mock/Library/Logs/DiagnosticReports/DumpRenderTree_2011-06-13-150721_quadzen.crash'] = None
     250        self.files['/Users/mock/Library/Logs/DiagnosticReports/DumpRenderTree_2011-06-13-150722_quadzen.crash'] = self.other_process_mock_crash_report
     251        self.files['/Users/mock/Library/Logs/DiagnosticReports/DumpRenderTree_2011-06-13-150723_quadzen.crash'] = self.misformatted_mock_crash_report
     252        self.filesystem = MockFileSystem(self.files)
     253        crash_logs = CrashLogs(MockSystemHost(filesystem=self.filesystem))
     254        logs = self.filesystem.files_under('/Users/mock/Library/Logs/DiagnosticReports/')
     255        for path in reversed(sorted(logs)):
     256            self.assertTrue(path in self.files.keys())
     257        return crash_logs
     258
     259    def test_find_all_log_darwin(self):
     260        crash_logs = self.create_crash_logs_darwin()
     261        all_logs = crash_logs.find_all_logs()
     262        self.assertEqual(len(all_logs), 5)
     263
     264        for test, crash_log in all_logs.iteritems():
     265            self.assertTrue(crash_log in self.files.values())
     266            self.assertTrue(test == "Unknown" or int(test.split("-")[1]) in range(28527, 28531))
     267
     268    def test_find_log_darwin(self):
     269        crash_logs = self.create_crash_logs_darwin()
    254270        log = crash_logs.find_newest_log("DumpRenderTree")
    255         self.assertMultiLineEqual(log, newer_mock_crash_report)
     271        self.assertMultiLineEqual(log, self.newer_mock_crash_report)
    256272        log = crash_logs.find_newest_log("DumpRenderTree", 28529)
    257         self.assertMultiLineEqual(log, newer_mock_crash_report)
     273        self.assertMultiLineEqual(log, self.newer_mock_crash_report)
    258274        log = crash_logs.find_newest_log("DumpRenderTree", 28530)
    259         self.assertMultiLineEqual(log, mock_crash_report)
     275        self.assertMultiLineEqual(log, self.mock_crash_report)
    260276        log = crash_logs.find_newest_log("DumpRenderTree", 28531)
    261277        self.assertIsNone(log)
    … …  
    269285            raise OSError('OSError: No such file or directory')
    270286
    271         filesystem.read_text_file = bad_read
     287        self.filesystem.read_text_file = bad_read
    272288        log = crash_logs.find_newest_log("DumpRenderTree", 28531, include_errors=True)
    273289        self.assertIn('IOError: No such file or directory', log)
    274290
    275         filesystem = MockFileSystem(files)
    276         crash_logs = CrashLogs(MockSystemHost(filesystem=filesystem))
    277         filesystem.mtime = bad_mtime
     291        self.filesystem = MockFileSystem(self.files)
     292        crash_logs = CrashLogs(MockSystemHost(filesystem=self.filesystem))
     293        self.filesystem.mtime = bad_mtime
    278294        log = crash_logs.find_newest_log("DumpRenderTree", newer_than=1.0, include_errors=True)
    279295        self.assertIn('OSError: No such file or directory', log)

  • trunk/Tools/Scripts/webkitpy/layout_tests/controllers/manager.py

    r184382 r191374  
    5050from webkitpy.layout_tests.models import test_expectations
    5151from webkitpy.layout_tests.models import test_failures
     52from webkitpy.layout_tests.models import test_results
    5253from webkitpy.layout_tests.models import test_run_results
    5354from webkitpy.layout_tests.models.test_input import TestInput
    … …  
    180181        tests_to_run, tests_to_skip = self._prepare_lists(paths, test_names)
    181182        self._printer.print_found(len(test_names), len(tests_to_run), self._options.repeat_each, self._options.iterations)
     183        start_time = time.time()
    182184
    183185        # Check to make sure we're not skipping every test.
    … …  
    189191            return test_run_results.RunDetails(exit_code=-1)
    190192
    191         start_time = time.time()
    192193        enabled_pixel_tests_in_retry = False
    193194        try:
    … …  
    306307                writer.write_crash_log(crash_log)
    307308
     309                # Check if this crashing 'test' is already in list of crashed_processes, if not add it to the run_results
     310                if not any(process[0] == test for process in crashed_processes):
     311                    result = test_results.TestResult(test)
     312                    result.type = test_expectations.CRASH
     313                    result.is_other_crash = True
     314                    run_results.add(result, expected=False, test_is_slow=False)
     315                    _log.debug("Adding results for other crash: " + str(test))
     316
    308317    def _clobber_old_results(self):
    309318        # Just clobber the actual test results directories since the other

  • trunk/Tools/Scripts/webkitpy/layout_tests/models/test_expectations.py

    r189137 r191374  
    562562        """Returns the expectatons for the given test as an uppercase string.
    563563        If there are no expectations for the test, then "PASS" is returned."""
    564         expectations = self.get_expectations(test)
     564        try:
     565            expectations = self.get_expectations(test)
     566        except:
     567            return "PASS"
    565568        retval = []
    566569

  • trunk/Tools/Scripts/webkitpy/layout_tests/models/test_results.py

    r175367 r191374  
    5656        self.total_run_time = 0  # The time taken to run the test plus any references, compute diffs, etc.
    5757        self.test_number = None
     58        self.is_other_crash = False
    5859
    5960    def __eq__(self, other):

  • trunk/Tools/Scripts/webkitpy/layout_tests/models/test_run_results.py

    r184382 r191374  
    134134    """
    135135    results = {}
    136     results['version'] = 3
     136    results['version'] = 4
    137137
    138138    tbe = initial_results.tests_by_expectation
    … …  
    153153
    154154    tests = {}
     155    other_crashes_dict = {}
    155156
    156157    for test_name, result in initial_results.results_by_name.iteritems():
    … …  
    163164
    164165        if result_type == test_expectations.SKIP:
     166            continue
     167
     168        if result.is_other_crash:
     169            other_crashes_dict[test_name] = {}
    165170            continue
    166171
    … …  
    257262    results['has_pretty_patch'] = port_obj.pretty_patch.pretty_patch_available()
    258263    results['pixel_tests_enabled'] = port_obj.get_option('pixel_tests')
     264    results['other_crashes'] = other_crashes_dict
    259265
    260266    try:

  • trunk/Tools/Scripts/webkitpy/port/ios.py

    r190519 r191374  
    329329        return Simulator().lookup_or_create_device(self.simulator_device_type.name + ' WebKit Tester', self.simulator_device_type, self.simulator_runtime)
    330330
     331    def _merge_crash_logs(self, logs, new_logs, crashed_processes):
     332        for test, crash_log in new_logs.iteritems():
     333            try:
     334                process_name = test.split("-")[0]
     335                pid = int(test.split("-")[1])
     336            except IndexError:
     337                continue
     338            if not any(entry[1] == process_name and entry[2] == pid for entry in crashed_processes):
     339                # if this is a new crash, then append the logs
     340                logs[test] = crash_log
     341        return logs
     342
     343    def _look_for_all_crash_logs_in_log_dir(self, newer_than):
     344        crash_log = CrashLogs(self.host)
     345        return crash_log.find_all_logs(include_errors=True, newer_than=newer_than)
     346
    331347    def look_for_new_crash_logs(self, crashed_processes, start_time):
    332348        crash_logs = {}
    … …  
    338354                continue
    339355            crash_logs[test_name] = crash_log
    340         return crash_logs
     356        all_crash_log = self._look_for_all_crash_logs_in_log_dir(start_time)
     357        return self._merge_crash_logs(crash_logs, all_crash_log, crashed_processes)
    341358
    342359    def look_for_new_samples(self, unresponsive_processes, start_time):
Note: See TracChangeset for help on using the changeset viewer.