Changeset 57907 in webkit


Ignore:
Timestamp:
Apr 20, 2010 12:54:46 PM (14 years ago)
Author:
eric@webkit.org
Message:

2010-04-20 Eric Seidel <​eric@webkit.org>

Reviewed by Adam Barth.

REGRESSION(57531): the commit-queue still hates Tor Arne Vestbø
​https://bugs.webkit.org/show_bug.cgi?id=37765

I fixed the queue to not ignore Tor as a reviwer in r57531,
but instead it throws an exception every time his name is in a patch.

This fixes our Executive.run_command code to work around a Popen
bug ​http://bugs.python.org/issue5290 whereby python versions before 2.6
do not correctly handle unicode objects as input or output to
Popen.communicate.

Following the advice of:
​http://farmdev.com/talks/unicode/
I'm attempting to take the python unicode plunge and use unicode()
objects as strings instead of str() objects everywhere in webkitpy.

We do not have to use u"" instead of "" because u"a" == "a" as expected
in Python. Python will generate a warning to the console in cases where
a unicode() == str() operation cannot be performed.

I also cleaned up the input handling in run_command a little by adding
a new _compute_input() method which can return early instead of having
such a long/cluttered if-block.

Executive.run* now correctly accept and return unicode() objects.
I attempted to fix all the places that we call .write() to make sure we
encode any unicode() objects into utf-8.

All places which use StringIO need to be sure to pass StringIO a
pre-encoded byte-array (str object) instead of unicode so that
clients which read from the StringIO don't have encoding exceptions.
To make this easier, I removed the patch_file_object support from
add_patch_to_bug, and changed the 4 places which previously used
StringIO to create a fake patch file.

I attempted to document any places where we are not correctly converting
to/from bytes (str() objects) to strings (unicode() objects).

  • Scripts/webkitpy/common/checkout/api_unittest.py:
    • Read/write utf-8 files instead of ascii.
    • Update the tests to use test for proper unicode() handling.
  • Scripts/webkitpy/common/checkout/changelog_unittest.py:
    • Use unicode() strings instead of str() byte arrays.
  • Scripts/webkitpy/common/checkout/scm.py:
    • Remove use of str().
  • Scripts/webkitpy/common/checkout/scm_unittest.py:
    • Read/write utf-8 files and use unicode() strings in testing.
  • Scripts/webkitpy/common/config/committers.py:
    • Use \u instead of \x to make slightly clearer what we're doing.
  • Scripts/webkitpy/common/net/bugzilla.py:
    • Add a new _string_contents() method and explain why we have to call unicode() on the result of soup.string and why it's safe to do so w/o needing to pass a codec name.
    • Remove the (unused) support for passing a file object to add_patch_to_bug().
  • Scripts/webkitpy/common/net/buildbot.py:
    • Use unicode() instead of str() when needing to coax a NavigableString object into a unicode() object.
  • Scripts/webkitpy/common/net/statusserver.py:
    • Remove use of str()
  • Scripts/webkitpy/common/prettypatch.py:
    • Write out the patch file as utf-8.
  • Scripts/webkitpy/common/system/autoinstall.py:
    • Add a FIXME about encoding.
  • Scripts/webkitpy/common/system/deprecated_logging.py:
    • Document that tee() works on bytes, not strings.
  • Scripts/webkitpy/common/system/executive.py:
    • Make run* properly take and return unicode() objects.
  • Scripts/webkitpy/common/system/executive_unittest.py:
    • Added a unit test to make sure we don't break Tor again!
  • Scripts/webkitpy/layout_tests/layout_package/dump_render_tree_thread.py:
    • Write out the test list as utf-8.
  • Scripts/webkitpy/layout_tests/layout_package/json_results_generator.py:
    • Write out json files as utf-8.
  • Scripts/webkitpy/layout_tests/layout_package/metered_stream.py:
    • Add FIXME about encoding handling.
  • Scripts/webkitpy/tool/commands/upload.py:
    • Pass the diff directly to add_patch_to_bug instead of creating a StringIO file wrapper.
  • Scripts/webkitpy/tool/mocktool.py:
    • Rename add_patch_to_bug argument to match bugzilla.py
  • Scripts/webkitpy/tool/steps/postdiff.py:
    • Pass the diff directly to add_patch_to_bug instead of creating a StringIO file wrapper.
  • Scripts/webkitpy/tool/steps/postdiffforcommit.py: ditto.
  • Scripts/webkitpy/tool/steps/postdiffforrevert.py: ditto.
  • Scripts/webkitpy/tool/steps/steps_unittest.py:
    • Fixed spurious logging seen when running test-webkitpy
Location:
trunk/WebKitTools
Files:
32 edited

Legend:

Unmodified
Added
Removed

  • trunk/WebKitTools/ChangeLog

    r57905 r57907  
     12010-04-20  Eric Seidel  <eric@webkit.org>
     2
     3        Reviewed by Adam Barth.
     4
     5        REGRESSION(57531): the commit-queue still hates Tor Arne Vestbø
     6        https://bugs.webkit.org/show_bug.cgi?id=37765
     7
     8        I fixed the queue to not ignore Tor as a reviwer in r57531,
     9        but instead it throws an exception every time his name is in a patch.
     10
     11        This fixes our Executive.run_command code to work around a Popen
     12        bug http://bugs.python.org/issue5290 whereby python versions before 2.6
     13        do not correctly handle unicode objects as input or output to
     14        Popen.communicate.
     15
     16        Following the advice of:
     17        http://farmdev.com/talks/unicode/
     18        I'm attempting to take the python unicode plunge and use unicode()
     19        objects as strings instead of str() objects everywhere in webkitpy.
     20
     21        We do not have to use u"" instead of "" because u"a" == "a" as expected
     22        in Python.  Python will generate a warning to the console in cases where
     23        a unicode() == str() operation cannot be performed.
     24
     25        I also cleaned up the input handling in run_command a little by adding
     26        a new _compute_input() method which can return early instead of having
     27        such a long/cluttered if-block.
     28
     29        Executive.run* now correctly accept and return unicode() objects.
     30        I attempted to fix all the places that we call .write() to make sure we
     31        encode any unicode() objects into utf-8.
     32
     33        All places which use StringIO need to be sure to pass StringIO a
     34        pre-encoded byte-array (str object) instead of unicode so that
     35        clients which read from the StringIO don't have encoding exceptions.
     36        To make this easier, I removed the patch_file_object support from
     37        add_patch_to_bug, and changed the 4 places which previously used
     38        StringIO to create a fake patch file.
     39
     40        I attempted to document any places where we are not correctly converting
     41        to/from bytes (str() objects) to strings (unicode() objects).
     42
     43        * Scripts/webkitpy/common/checkout/api_unittest.py:
     44         - Read/write utf-8 files instead of ascii.
     45         - Update the tests to use test for proper unicode() handling.
     46        * Scripts/webkitpy/common/checkout/changelog_unittest.py:
     47         - Use unicode() strings instead of str() byte arrays.
     48        * Scripts/webkitpy/common/checkout/scm.py:
     49         - Remove use of str().
     50        * Scripts/webkitpy/common/checkout/scm_unittest.py:
     51         - Read/write utf-8 files and use unicode() strings in testing.
     52        * Scripts/webkitpy/common/config/committers.py:
     53         - Use \u instead of \x to make slightly clearer what we're doing.
     54        * Scripts/webkitpy/common/net/bugzilla.py:
     55         - Add a new _string_contents() method and explain why
     56           we have to call unicode() on the result of soup.string
     57           and why it's safe to do so w/o needing to pass a codec name.
     58         - Remove the (unused) support for passing a file object to add_patch_to_bug().
     59        * Scripts/webkitpy/common/net/buildbot.py:
     60         - Use unicode() instead of str() when needing to coax a
     61           NavigableString object into a unicode() object.
     62        * Scripts/webkitpy/common/net/statusserver.py:
     63         - Remove use of str()
     64        * Scripts/webkitpy/common/prettypatch.py:
     65         - Write out the patch file as utf-8.
     66        * Scripts/webkitpy/common/system/autoinstall.py:
     67         - Add a FIXME about encoding.
     68        * Scripts/webkitpy/common/system/deprecated_logging.py:
     69         - Document that tee() works on bytes, not strings.
     70        * Scripts/webkitpy/common/system/executive.py:
     71         - Make run* properly take and return unicode() objects.
     72        * Scripts/webkitpy/common/system/executive_unittest.py:
     73         - Added a unit test to make sure we don't break Tor again!
     74        * Scripts/webkitpy/layout_tests/layout_package/dump_render_tree_thread.py:
     75         - Write out the test list as utf-8.
     76        * Scripts/webkitpy/layout_tests/layout_package/json_results_generator.py:
     77         - Write out json files as utf-8.
     78        * Scripts/webkitpy/layout_tests/layout_package/metered_stream.py:
     79         - Add FIXME about encoding handling.
     80        * Scripts/webkitpy/tool/commands/upload.py:
     81         - Pass the diff directly to add_patch_to_bug instead of creating a StringIO file wrapper.
     82        * Scripts/webkitpy/tool/mocktool.py:
     83         - Rename add_patch_to_bug argument to match bugzilla.py
     84        * Scripts/webkitpy/tool/steps/postdiff.py:
     85         - Pass the diff directly to add_patch_to_bug instead of creating a StringIO file wrapper.
     86        * Scripts/webkitpy/tool/steps/postdiffforcommit.py: ditto.
     87        * Scripts/webkitpy/tool/steps/postdiffforrevert.py: ditto.
     88        * Scripts/webkitpy/tool/steps/steps_unittest.py:
     89         - Fixed spurious logging seen when running test-webkitpy
     90
    1912010-04-20  Chris Jerdonek  <cjerdonek@webkit.org>
    292

  • trunk/WebKitTools/Scripts/webkitpy/common/checkout/api.py

    r57572 r57907  
    5151    def _latest_entry_for_changelog_at_revision(self, changelog_path, revision):
    5252        changelog_contents = self._scm.contents_at_revision(changelog_path, revision)
    53         return ChangeLog.parse_latest_entry_from_file(StringIO.StringIO(changelog_contents))
     53        fake_file = StringIO.StringIO(changelog_contents.encode("utf-8"))
     54        return ChangeLog.parse_latest_entry_from_file(fake_file)
    5455
    5556    def changelog_entries_for_revision(self, revision):

  • trunk/WebKitTools/Scripts/webkitpy/common/checkout/api_unittest.py

    r57572 r57907  
    2727# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
    2828
     29import codecs
    2930import os
    3031import shutil
    … …  
    4041# FIXME: Copied from scm_unittest.py
    4142def write_into_file_at_path(file_path, contents):
    42     new_file = open(file_path, 'w')
     43    new_file = codecs.open(file_path, "w", "utf-8")
    4344    new_file.write(contents)
    4445    new_file.close()
    4546
    4647
    47 _changelog1entry1 = """2010-03-25  Eric Seidel  <eric@webkit.org>
     48_changelog1entry1 = u"""2010-03-25  Tor Arne Vestb\u00f8  <vestbo@webkit.org>
    4849
    4950        Unreviewed build fix to un-break webkit-patch land.
    … …  
    5455        * Scripts/webkitpy/common/checkout/api.py: import scm.CommitMessage
    5556"""
    56 _changelog1entry2 = """2010-03-25  Adam Barth  <abarth@webkit.org>
     57_changelog1entry2 = u"""2010-03-25  Adam Barth  <abarth@webkit.org>
    5758
    5859        Reviewed by Eric Seidel.
    … …  
    6364        * Scripts/webkitpy/common/checkout/api.py:
    6465"""
    65 _changelog1 = "\n".join([_changelog1entry1, _changelog1entry2])
    66 _changelog2 = """2010-03-25  Eric Seidel  <eric@webkit.org>
     66_changelog1 = u"\n".join([_changelog1entry1, _changelog1entry2])
     67_changelog2 = u"""2010-03-25  Tor Arne Vestb\u00f8  <vestbo@webkit.org>
    6768
    6869        Unreviewed build fix to un-break webkit-patch land.
    … …  
    8081
    8182class CommitMessageForThisCommitTest(unittest.TestCase):
    82     expected_commit_message = """2010-03-25  Eric Seidel  <eric@webkit.org>
     83    expected_commit_message = u"""2010-03-25  Tor Arne Vestb\u00f8  <vestbo@webkit.org>
    8384
    8485        Unreviewed build fix to un-break webkit-patch land.
    … …  
    8889
    8990        * Scripts/webkitpy/common/checkout/api.py: import scm.CommitMessage
    90 2010-03-25  Eric Seidel  <eric@webkit.org>
     912010-03-25  Tor Arne Vestb\u00f8  <vestbo@webkit.org>
    9192
    9293        Unreviewed build fix to un-break webkit-patch land.
    … …  
    138139        commitinfo = checkout.commit_info_for_revision(4)
    139140        self.assertEqual(commitinfo.bug_id(), 36629)
    140         self.assertEqual(commitinfo.author_name(), "Eric Seidel")
    141         self.assertEqual(commitinfo.author_email(), "eric@webkit.org")
     141        self.assertEqual(commitinfo.author_name(), u"Tor Arne Vestb\u00f8")
     142        self.assertEqual(commitinfo.author_email(), "vestbo@webkit.org")
    142143        self.assertEqual(commitinfo.reviewer_text(), None)
    143144        self.assertEqual(commitinfo.reviewer(), None)

  • trunk/WebKitTools/Scripts/webkitpy/common/checkout/changelog.py

    r57531 r57907  
    103103        entry_lines = []
    104104        # The first line should be a date line.
    105         first_line = changelog_file.readline()
     105        first_line = unicode(changelog_file.readline(), "utf-8")
    106106        if not date_line_regexp.match(first_line):
    107107            return None
    … …  
    109109
    110110        for line in changelog_file:
     111            line = unicode(line, "utf-8")
    111112            # If we've hit the next entry, return.
    112113            if date_line_regexp.match(line):
    … …  
    118119    def latest_entry(self):
    119120        # ChangeLog files are always UTF-8, we read them in as such to support Reviewers with unicode in their names.
    120         changelog_file = codecs.open(self.path, "r", "utf-8")
     121        # We don't use codecs.open here to make the api for parse_latest_entry_from_file clearer.
     122        # If we did, then it would be unclear as to whos reponsibility decoding of the file should be.
     123        changelog_file = open(self.path, "r")
    121124        try:
    122125            return self.parse_latest_entry_from_file(changelog_file)

  • trunk/WebKitTools/Scripts/webkitpy/common/checkout/changelog_unittest.py

    r57531 r57907  
    8888
    8989    def test_latest_entry_parse(self):
    90         changelog_contents = "%s\n%s" % (self._example_entry, self._example_changelog)
    91         changelog_file = StringIO(changelog_contents)
     90        changelog_contents = u"%s\n%s" % (self._example_entry, self._example_changelog)
     91        changelog_file = StringIO(changelog_contents.encode("utf-8"))
    9292        latest_entry = ChangeLog.parse_latest_entry_from_file(changelog_file)
    9393        self.assertEquals(latest_entry.contents(), self._example_entry)
    … …  
    122122
    123123    def test_set_reviewer(self):
    124         changelog_contents = "%s\n%s" % (self._new_entry_boilerplate, self._example_changelog)
     124        changelog_contents = u"%s\n%s" % (self._new_entry_boilerplate, self._example_changelog)
    125125        changelog_path = self._write_tmp_file_with_contents(changelog_contents)
    126126        reviewer_name = 'Test Reviewer'
    … …  
    170170
    171171    def _assert_update_for_revert_output(self, args, expected_entry):
    172         changelog_contents = "%s\n%s" % (self._new_entry_boilerplate, self._example_changelog)
     172        changelog_contents = u"%s\n%s" % (self._new_entry_boilerplate, self._example_changelog)
    173173        changelog_path = self._write_tmp_file_with_contents(changelog_contents)
    174174        changelog = ChangeLog(changelog_path)

  • trunk/WebKitTools/Scripts/webkitpy/common/checkout/commitinfo.py

    r57195 r57907  
    2828#
    2929# WebKit's python module for holding information on a commit
    30 
    31 import StringIO
    3230
    3331from webkitpy.common.checkout.changelog import view_source_url

  • trunk/WebKitTools/Scripts/webkitpy/common/checkout/scm.py

    r57439 r57907  
    146146
    147147    def strip_r_from_svn_revision(self, svn_revision):
    148         match = re.match("^r(?P<svn_revision>\d+)", svn_revision)
     148        match = re.match("^r(?P<svn_revision>\d+)", unicode(svn_revision))
    149149        if (match):
    150150            return match.group('svn_revision')
    … …  
    345345    def changed_files_for_revision(self, revision):
    346346        # As far as I can tell svn diff --summarize output looks just like svn status output.
    347         status_command = ["svn", "diff", "--summarize", "-c", str(revision)]
     347        status_command = ["svn", "diff", "--summarize", "-c", revision]
    348348        return self.run_status_and_extract_filenames(status_command, self._status_regexp("ACDMR"))
    349349
    … …  
    365365
    366366    def committer_email_for_revision(self, revision):
    367         return run_command(["svn", "propget", "svn:author", "--revprop", "-r", str(revision)]).rstrip()
     367        return run_command(["svn", "propget", "svn:author", "--revprop", "-r", revision]).rstrip()
    368368
    369369    def contents_at_revision(self, path, revision):
    370370        remote_path = "%s/%s" % (self._repository_url(), path)
    371         return run_command(["svn", "cat", "-r", str(revision), remote_path])
     371        return run_command(["svn", "cat", "-r", revision, remote_path])
    372372
    373373    def diff_for_revision(self, revision):
    374374        # FIXME: This should probably use cwd=self.checkout_root
    375         return run_command(['svn', 'diff', '-c', str(revision)])
     375        return run_command(['svn', 'diff', '-c', revision])
    376376
    377377    def _repository_url(self):
    … …  
    406406
    407407    def svn_commit_log(self, svn_revision):
    408         svn_revision = self.strip_r_from_svn_revision(str(svn_revision))
     408        svn_revision = self.strip_r_from_svn_revision(svn_revision)
    409409        return run_command(['svn', 'log', '--non-interactive', '--revision', svn_revision]);
    410410

  • trunk/WebKitTools/Scripts/webkitpy/common/checkout/scm_unittest.py

    r57439 r57907  
    2929
    3030import base64
     31import codecs
    3132import getpass
    3233import os
    … …  
    5859
    5960def write_into_file_at_path(file_path, contents):
    60     file = open(file_path, 'w')
     61    file = codecs.open(file_path, "w", "utf-8")
    6162    file.write(contents)
    6263    file.close()
    6364
    6465def read_from_path(file_path):
    65     file = open(file_path, 'r')
     66    file = codecs.open(file_path, "r", "utf-8")
    6667    contents = file.read()
    6768    file.close()

  • trunk/WebKitTools/Scripts/webkitpy/common/config/committers.py

    r57898 r57907  
    240240    Reviewer("Tim Omernick", "timo@apple.com"),
    241241    Reviewer("Timothy Hatcher", ["timothy@hatcher.name", "timothy@apple.com"], "xenon"),
    242     Reviewer(u'Tor Arne Vestb\xf8', "vestbo@webkit.org", "torarne"),
     242    Reviewer(u"Tor Arne Vestb\u00f8", "vestbo@webkit.org", "torarne"),
    243243    Reviewer("Vicki Murley", "vicki@apple.com"),
    244244    Reviewer("Xan Lopez", ["xan.lopez@gmail.com", "xan@gnome.org", "xan@webkit.org"], "xan"),

  • trunk/WebKitTools/Scripts/webkitpy/common/net/bugzilla.py

    r57869 r57907  
    3333import os.path
    3434import re
     35import StringIO
    3536import subprocess
    3637
    … …  
    412413                attachment[result_key] = flag['setter']
    413414
     415    def _string_contents(self, soup):
     416        # WebKit's bugzilla instance uses UTF-8.
     417        # BeautifulSoup always returns Unicode strings, however
     418        # the .string method returns a (unicode) NavigableString.
     419        # NavigableString can confuse other parts of the code, so we
     420        # convert from NavigableString to a real unicode() object using unicode().
     421        return unicode(soup.string)
     422
    414423    def _parse_attachment_element(self, element, bug_id):
     424
    415425        attachment = {}
    416426        attachment['bug_id'] = bug_id
    … …  
    420430        # FIXME: No need to parse out the url here.
    421431        attachment['url'] = self.attachment_url_for_id(attachment['id'])
    422         attachment['name'] = unicode(element.find('desc').string)
    423         attachment['attacher_email'] = str(element.find('attacher').string)
    424         attachment['type'] = str(element.find('type').string)
     432        attachment['name'] = self._string_contents(element.find('desc'))
     433        attachment['attacher_email'] = self._string_contents(element.find('attacher'))
     434        attachment['type'] = self._string_contents(element.find('type'))
    425435        self._parse_attachment_flag(
    426436                element, 'review', attachment, 'reviewer_email')
    … …  
    433443        bug = {}
    434444        bug["id"] = int(soup.find("bug_id").string)
    435         bug["title"] = unicode(soup.find("short_desc").string)
    436         bug["reporter_email"] = str(soup.find("reporter").string)
    437         bug["assigned_to_email"] = str(soup.find("assigned_to").string)
    438         bug["cc_emails"] = [str(element.string)
     445        bug["title"] = self._string_contents(soup.find("short_desc"))
     446        bug["reporter_email"] = self._string_contents(soup.find("reporter"))
     447        bug["assigned_to_email"] = self._string_contents(soup.find("assigned_to"))
     448        bug["cc_emails"] = [self._string_contents(element)
    439449                            for element in soup.findAll('cc')]
    440450        bug["attachments"] = [self._parse_attachment_element(element, bug["id"]) for element in soup.findAll('attachment')]
    … …  
    533543    def _fill_attachment_form(self,
    534544                              description,
    535                               patch_file_object,
     545                              diff,
    536546                              comment_text=None,
    537547                              mark_for_review=False,
    … …  
    553563        else:
    554564            patch_name ="%s.patch" % timestamp()
     565
     566        # ClientForm expects a file-like object
     567        patch_file_object = StringIO.StringIO(diff.encode("utf-8"))
    555568        self.browser.add_file(patch_file_object,
    556569                              "text/plain",
    … …  
    560573    def add_patch_to_bug(self,
    561574                         bug_id,
    562                          patch_file_object,
     575                         diff,
    563576                         description,
    564577                         comment_text=None,
    … …  
    579592                          self.bug_server_url, bug_id))
    580593        self.browser.select_form(name="entryform")
     594
    581595        self._fill_attachment_form(description,
    582                                    patch_file_object,
     596                                   diff,
    583597                                   mark_for_review=mark_for_review,
    584598                                   mark_for_commit_queue=mark_for_commit_queue,
    … …  
    613627                   bug_description,
    614628                   component=None,
    615                    patch_file_object=None,
     629                   diff=None,
    616630                   patch_description=None,
    617631                   cc=None,
    … …  
    638652            self.browser["cc"] = cc
    639653        if blocked:
    640             self.browser["blocked"] = str(blocked)
     654            self.browser["blocked"] = unicode(blocked)
    641655        self.browser["short_desc"] = bug_title
    642656        self.browser["comment"] = bug_description
    643657
    644         if patch_file_object:
     658        if diff:
    645659            self._fill_attachment_form(
    646660                    patch_description,
    647                     patch_file_object,
     661                    diff,
    648662                    mark_for_review=mark_for_review,
    649663                    mark_for_commit_queue=mark_for_commit_queue)

  • trunk/WebKitTools/Scripts/webkitpy/common/net/buildbot.py

    r57805 r57907  
    4545class Builder(object):
    4646    def __init__(self, name, buildbot):
    47         self._name = unicode(name)
     47        self._name = name
    4848        self._buildbot = buildbot
    4949        self._builds_cache = {}
    … …  
    224224        tables = BeautifulSoup(page).findAll("table")
    225225        for table in tables:
    226             table_title = table.findPreviousSibling("p").string
     226            table_title = unicode(table.findPreviousSibling("p").string)
    227227            if table_title not in cls.expected_keys:
    228228                # This Exception should only ever be hit if run-webkit-tests changes its results.html format.
    229                 raise Exception("Unhandled title: %s" % str(table_title))
     229                raise Exception("Unhandled title: %s" % table_title)
    230230            # We might want to translate table titles into identifiers before storing.
    231             parsed_results[table_title] = [row.find("a").string for row in table.findAll("tr")]
     231            parsed_results[table_title] = [unicode(row.find("a").string) for row in table.findAll("tr")]
    232232
    233233        return parsed_results
    … …  
    362362        # First cell is the name
    363363        name_link = status_cells[0].find('a')
    364         builder["name"] = name_link.string
     364        builder["name"] = unicode(name_link.string)
    365365
    366366        self._parse_last_build_cell(builder, status_cells[1])
    … …  
    411411
    412412    def _parse_twisted_file_row(self, file_row):
    413         string_or_empty = lambda string: str(string) if string else ""
     413        string_or_empty = lambda soup: unicode(soup.string) if soup.string else u""
    414414        file_cells = file_row.findAll('td')
    415415        return {
    416             "filename" : string_or_empty(file_cells[0].find("a").string),
    417             "size" : string_or_empty(file_cells[1].string),
    418             "type" : string_or_empty(file_cells[2].string),
    419             "encoding" : string_or_empty(file_cells[3].string),
     416            "filename": string_or_empty(file_cells[0].find("a")),
     417            "size": string_or_empty(file_cells[1]),
     418            "type": string_or_empty(file_cells[2]),
     419            "encoding": string_or_empty(file_cells[3]),
    420420        }
    421421

  • trunk/WebKitTools/Scripts/webkitpy/common/net/buildbot_unittest.py

    r57795 r57907  
    5252    def setUp(self):
    5353        self.buildbot = BuildBot()
    54         self.builder = Builder("Test Builder", self.buildbot)
     54        self.builder = Builder(u"Test Builder \u2661", self.buildbot)
    5555        self._install_fetch_build(lambda build_number: ["test1", "test2"])
    5656

  • trunk/WebKitTools/Scripts/webkitpy/common/net/statusserver.py

    r57195 r57907  
    5353            return
    5454        if patch.bug_id():
    55             self.browser["bug_id"] = str(patch.bug_id())
     55            self.browser["bug_id"] = unicode(patch.bug_id())
    5656        if patch.id():
    57             self.browser["patch_id"] = str(patch.id())
     57            self.browser["patch_id"] = unicode(patch.id())
    5858
    5959    def _add_results_file(self, results_file):
    … …  
    8080        self.browser.open(update_svn_revision_url)
    8181        self.browser.select_form(name="update_svn_revision")
    82         self.browser["number"] = str(svn_revision_number)
     82        self.browser["number"] = unicode(svn_revision_number)
    8383        self.browser["broken_bot"] = broken_bot
    8484        return self.browser.submit().read()

  • trunk/WebKitTools/Scripts/webkitpy/common/prettypatch.py

    r57587 r57907  
    3939        pretty_diff = self.pretty_diff(diff)
    4040        diff_file = tempfile.NamedTemporaryFile(suffix=".html")
    41         diff_file.write(pretty_diff)
     41        diff_file.write(pretty_diff.encode("utf-8"))
    4242        diff_file.flush()
    4343        return diff_file

  • trunk/WebKitTools/Scripts/webkitpy/common/system/autoinstall.py

    r56897 r57907  
    125125        file = open(path, "w")
    126126        try:
    127             file.write(text)
     127            file.write(text)  # FIXME: What's the encoding of this file?
    128128        finally:
    129129            file.close()

  • trunk/WebKitTools/Scripts/webkitpy/common/system/deprecated_logging.py

    r56544 r57907  
    4646        self.files = files
    4747
    48     def write(self, string):
     48    # Callers should pass an already encoded string for writing.
     49    def write(self, bytes):
    4950        for file in self.files:
    50             file.write(string)
     51            file.write(bytes)
    5152
    5253class OutputTee:

  • trunk/WebKitTools/Scripts/webkitpy/common/system/executive.py

    r57550 r57907  
    8989
    9090    def _run_command_with_teed_output(self, args, teed_output):
     91        args = map(unicode, args)  # Popen will throw an exception if args are non-strings (like int())
    9192        child_process = subprocess.Popen(args,
    9293                                         stdout=subprocess.PIPE,
    … …  
    100101            if output_line == "" and child_process.poll() != None:
    101102                return child_process.poll()
     103            # We assume that the child process wrote to us in utf-8,
     104            # so no re-encoding is necessary before writing here.
    102105            teed_output.write(output_line)
    103106
    104     def run_and_throw_if_fail(self, args, quiet=False):
     107    # FIXME: Remove this deprecated method and move callers to run_command.
     108    # FIXME: This method is a hack to allow running command which both
     109    # capture their output and print out to stdin.  Useful for things
     110    # like "build-webkit" where we want to display to the user that we're building
     111    # but still have the output to stuff into a log file.
     112    def run_and_throw_if_fail(self, args, quiet=False, decode_output=True):
    105113        # Cache the child's output locally so it can be used for error reports.
    106114        child_out_file = StringIO.StringIO()
    … …  
    116124        child_output = child_out_file.getvalue()
    117125        child_out_file.close()
     126
     127        # We assume the child process output utf-8
     128        if decode_output:
     129            child_output = child_output.decode("utf-8")
    118130
    119131        if exit_code:
    … …  
    146158            # using Cygwin, it worked fine.  We should investigate whether
    147159            # we need this platform specific code here.
    148             subprocess.call(('taskkill.exe', '/f', '/pid', str(pid)),
     160            subprocess.call(('taskkill.exe', '/f', '/pid', unicode(pid)),
    149161                            stdin=open(os.devnull, 'r'),
    150162                            stdout=subprocess.PIPE,
    … …  
    164176        pass
    165177
    166     # FIXME: This should be merged with run_and_throw_if_fail
    167 
     178    def _compute_stdin(self, input):
     179        """Returns (stdin, string_to_communicate)"""
     180        if not input:
     181            return (None, None)
     182        if hasattr(input, "read"):  # Check if the input is a file.
     183            return (input, None)  # Assume the file is in the right encoding.
     184
     185        # Popen in Python 2.5 and before does not automatically encode unicode objects.
     186        # http://bugs.python.org/issue5290
     187        # See https://bugs.webkit.org/show_bug.cgi?id=37528
     188        # for an example of a regresion caused by passing a unicode string directly.
     189        # FIXME: We may need to encode differently on different platforms.
     190        if isinstance(input, unicode):
     191            input = input.encode("utf-8")
     192        return (subprocess.PIPE, input)
     193
     194    # FIXME: run_and_throw_if_fail should be merged into this method.
    168195    def run_command(self,
    169196                    args,
    … …  
    172199                    error_handler=None,
    173200                    return_exit_code=False,
    174                     return_stderr=True):
    175         if hasattr(input, 'read'): # Check if the input is a file.
    176             stdin = input
    177             string_to_communicate = None
    178         else:
    179             stdin = None
    180             if input:
    181                 stdin = subprocess.PIPE
    182             # string_to_communicate seems to need to be a str for proper
    183             # communication with shell commands.
    184             # See https://bugs.webkit.org/show_bug.cgi?id=37528
    185             # For an example of a regresion caused by passing a unicode string through.
    186             string_to_communicate = str(input)
    187         if return_stderr:
    188             stderr = subprocess.STDOUT
    189         else:
    190             stderr = None
     201                    return_stderr=True,
     202                    decode_output=True):
     203        args = map(unicode, args)  # Popen will throw an exception if args are non-strings (like int())
     204        stdin, string_to_communicate = self._compute_stdin(input)
     205        stderr = subprocess.STDOUT if return_stderr else None
    191206
    192207        process = subprocess.Popen(args,
    … …  
    196211                                   cwd=cwd)
    197212        output = process.communicate(string_to_communicate)[0]
     213        # run_command automatically decodes to unicode() unless explicitly told not to.
     214        if decode_output:
     215            output = output.decode("utf-8")
    198216        exit_code = process.wait()
    199217

  • trunk/WebKitTools/Scripts/webkitpy/common/system/executive_unittest.py

    r56517 r57907  
    1 # Copyright (C) 2009 Google Inc. All rights reserved.
     1# Copyright (C) 2010 Google Inc. All rights reserved.
    22# Copyright (C) 2009 Daniel Bates (dbates@intudata.com). All rights reserved.
    33#
    … …  
    3232from webkitpy.common.system.executive import Executive, run_command
    3333
     34
    3435class ExecutiveTest(unittest.TestCase):
    3536
    … …  
    3940        self.failUnlessRaises(OSError, run_bad_command)
    4041
    41 if __name__ == '__main__':
    42     unittest.main()
     42    def test_run_command_with_unicode(self):
     43        """Validate that it is safe to pass unicode() objects
     44        to Executive.run* methods, and they will return unicode()
     45        objects by default unless decode_output=False"""
     46        executive = Executive()
     47        unicode_tor = u"WebKit \u2661 Tor Arne Vestb\u00F8!"
     48        utf8_tor = unicode_tor.encode("utf-8")
     49
     50        output = executive.run_command(["cat"], input=unicode_tor)
     51        self.assertEquals(output, unicode_tor)
     52
     53        output = executive.run_command(["echo", "-n", unicode_tor])
     54        self.assertEquals(output, unicode_tor)
     55
     56        output = executive.run_command(["echo", "-n", unicode_tor], decode_output=False)
     57        self.assertEquals(output, utf8_tor)
     58
     59        # FIXME: We should only have one run* method to test
     60        output = executive.run_and_throw_if_fail(["echo", "-n", unicode_tor], quiet=True)
     61        self.assertEquals(output, unicode_tor)
     62
     63        output = executive.run_and_throw_if_fail(["echo", "-n", unicode_tor], quiet=True, decode_output=False)
     64        self.assertEquals(output, utf8_tor)

  • trunk/WebKitTools/Scripts/webkitpy/layout_tests/layout_package/dump_render_tree_thread.py

    r57446 r57907  
    3636"""
    3737
     38import codecs
    3839import copy
    3940import logging
    … …  
    287288        tests_run_filename = os.path.join(self._options.results_directory,
    288289                                          "tests_run.txt")
    289         tests_run_file = open(tests_run_filename, "a")
     290        tests_run_file = codecs.open(tests_run_filename, "a", "utf-8")
    290291
    291292        while True:

  • trunk/WebKitTools/Scripts/webkitpy/layout_tests/layout_package/json_results_generator.py

    r57800 r57907  
    2828# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
    2929
     30import codecs
    3031import logging
    3132import os
    … …  
    119120        json = self._get_json()
    120121        if json:
    121             results_file = open(self._results_file_path, "w")
     122            results_file = codecs.open(self._results_file_path, "w", "utf-8")
    122123            results_file.write(json)
    123124            results_file.close()

  • trunk/WebKitTools/Scripts/webkitpy/layout_tests/layout_package/metered_stream.py

    r57841 r57907  
    6161        self._last_update = ""
    6262
     63    # FIXME: Does this take a string (unicode) or an array of bytes (str)?
     64    # If it takes a string, it needs to call txt.encode("utf-8")
    6365    def write(self, txt):
    6466        """Write to the stream, overwriting and resetting the meter."""
    … …  
    8789        self._write(str)
    8890
     91    # FIXME: Does this take a string (unicode) or an array of bytes (str)?
     92    # If it takes a string, it needs to call txt.encode("utf-8")
    8993    def update(self, str):
    9094        """

  • trunk/WebKitTools/Scripts/webkitpy/layout_tests/port/mac_unittest.py

    r54830 r57907  
    4141        self.assertEqual(relative_paths, ['LayoutTests/platform/mac-leopard/Skipped', 'LayoutTests/platform/mac/Skipped'])
    4242
    43     example_skipped_file = """
     43    example_skipped_file = u"""
    4444# <rdar://problem/5647952> fast/events/mouseout-on-window.html needs mac DRT to issue mouse out events
    4545fast/events/mouseout-on-window.html
    … …  
    5959    def test_skipped_file_paths(self):
    6060        port = mac.MacPort()
    61         skipped_file = StringIO.StringIO(self.example_skipped_file)
     61        skipped_file = StringIO.StringIO(self.example_skipped_file.encode("utf-8"))
    6262        self.assertEqual(port._tests_from_skipped_file(skipped_file), self.example_skipped_tests)
    6363

  • trunk/WebKitTools/Scripts/webkitpy/tool/commands/earlywarningsystem.py

    r57778 r57907  
    2626# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
    2727# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
    28 
    29 from StringIO import StringIO
    3028
    3129from webkitpy.tool.commands.queues import AbstractReviewQueue

  • trunk/WebKitTools/Scripts/webkitpy/tool/commands/queues.py

    r57451 r57907  
    7272        webkit_patch_args = [self.tool.path()]
    7373        # FIXME: This is a hack, we should have a more general way to pass global options.
    74         webkit_patch_args += ["--status-host=%s" % self.tool.status_server.host]
    75         webkit_patch_args += map(str, args)
     74        webkit_patch_args.append("--status-host")
     75        webkit_patch_args.append(self.tool.status_server.host)
     76        webkit_patch_args.extend(args)
    7677        return self.tool.executive.run_and_throw_if_fail(webkit_patch_args)
    7778
    … …  
    124125            message = "Error: %s" % message
    125126        output = script_error.message_with_output(output_limit=1024*1024) # 1MB
    126         return tool.status_server.update_status(cls.name, message, state["patch"], StringIO(output))
     127        return tool.status_server.update_status(cls.name, message, state["patch"], StringIO(output.encode("utf-8")))
    127128
    128129

  • trunk/WebKitTools/Scripts/webkitpy/tool/commands/queues_unittest.py

    r57788 r57907  
    7676
    7777        queue.run_webkit_patch(run_args)
    78         expected_run_args = ["echo", "--status-host=example.com"] + map(str, run_args)
     78        expected_run_args = ["echo", "--status-host", "example.com"] + run_args
    7979        tool.executive.run_and_throw_if_fail.assert_called_with(expected_run_args)
    8080
    … …  
    1511511 patch in commit-queue [106]
    152152""",
    153             "process_work_item": "MOCK run_and_throw_if_fail: ['echo', '--status-host=example.com', 'land-attachment', '--force-clean', '--build', '--test', '--non-interactive', '--ignore-builders', '--build-style=both', '--quiet', '76543']\n",
     153            "process_work_item": "MOCK run_and_throw_if_fail: ['echo', '--status-host', 'example.com', 'land-attachment', '--force-clean', '--build', '--test', '--non-interactive', '--ignore-builders', '--build-style=both', '--quiet', 76543]\n",
    154154        }
    155155        self.assert_queue_outputs(CommitQueue(), tool=tool, work_item=rollout_patch, expected_stderr=expected_stderr)

  • trunk/WebKitTools/Scripts/webkitpy/tool/commands/upload.py

    r57788 r57907  
    3131import os
    3232import re
    33 import StringIO
    3433import sys
    3534
    … …  
    261260        return comment_text
    262261
    263     def _diff_file_for_commit(self, tool, commit_id):
    264         diff = tool.scm().create_patch_from_local_commit(commit_id)
    265         return StringIO.StringIO(diff) # add_patch_to_bug expects a file-like object
    266 
    267262    def execute(self, options, args, tool):
    268263        commit_ids = tool.scm().commit_ids_from_commitish_arguments(args)
    … …  
    285280                have_obsoleted_patches.add(bug_id)
    286281
    287             diff_file = self._diff_file_for_commit(tool, commit_id)
     282            diff = tool.scm().create_patch_from_local_commit(commit_id)
    288283            description = options.description or commit_message.description(lstrip=True, strip_url=True)
    289284            comment_text = self._comment_text_for_commit(options, commit_message, tool, commit_id)
    290             tool.bugs.add_patch_to_bug(bug_id, diff_file, description, comment_text, mark_for_review=options.review, mark_for_commit_queue=options.request_commit)
     285            tool.bugs.add_patch_to_bug(bug_id, diff, description, comment_text, mark_for_review=options.review, mark_for_commit_queue=options.request_commit)
    291286
    292287
    … …  
    405400
    406401        diff = tool.scm().create_patch_from_local_commit(commit_id)
    407         diff_file = StringIO.StringIO(diff) # create_bug expects a file-like object
    408         bug_id = tool.bugs.create_bug(bug_title, comment_text, options.component, diff_file, "Patch", cc=options.cc, mark_for_review=options.review, mark_for_commit_queue=options.request_commit)
     402        bug_id = tool.bugs.create_bug(bug_title, comment_text, options.component, diff, "Patch", cc=options.cc, mark_for_review=options.review, mark_for_commit_queue=options.request_commit)
    409403
    410404        if bug_id and len(commit_ids) > 1:
    … …  
    425419
    426420        diff = tool.scm().create_patch()
    427         diff_file = StringIO.StringIO(diff) # create_bug expects a file-like object
    428         bug_id = tool.bugs.create_bug(bug_title, comment_text, options.component, diff_file, "Patch", cc=options.cc, mark_for_review=options.review, mark_for_commit_queue=options.request_commit)
     421        bug_id = tool.bugs.create_bug(bug_title, comment_text, options.component, diff, "Patch", cc=options.cc, mark_for_review=options.review, mark_for_commit_queue=options.request_commit)
    429422
    430423    def prompt_for_bug_title_and_comment(self):

  • trunk/WebKitTools/Scripts/webkitpy/tool/mocktool.py

    r57869 r57907  
    261261                   bug_description,
    262262                   component=None,
    263                    patch_file_object=None,
     263                   diff=None,
    264264                   patch_description=None,
    265265                   cc=None,
    … …  
    300300    def add_patch_to_bug(self,
    301301                         bug_id,
    302                          patch_file_object,
     302                         diff,
    303303                         description,
    304304                         comment_text=None,

  • trunk/WebKitTools/Scripts/webkitpy/tool/steps/abstractstep.py

    r56601 r57907  
    4040        log("Running %s" % script_name)
    4141        # FIXME: This should use self.port()
    42         self._tool.executive.run_and_throw_if_fail(port.script_path(script_name), quiet)
     42        self._tool.executive.run_and_throw_if_fail([port.script_path(script_name)], quiet)
    4343
    4444    # FIXME: The port should live on the tool.

  • trunk/WebKitTools/Scripts/webkitpy/tool/steps/postdiff.py

    r57679 r57907  
    2727# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
    2828
    29 import StringIO
    30 
    3129from webkitpy.tool.steps.abstractstep import AbstractStep
    3230from webkitpy.tool.steps.options import Options
    … …  
    4543    def run(self, state):
    4644        diff = self.cached_lookup(state, "diff")
    47         diff_file = StringIO.StringIO(diff) # add_patch_to_bug expects a file-like object
    4845        description = self._options.description or "Patch"
    4946        comment_text = None
    … …  
    5350        if codereview_issue:
    5451            description += "-%s" % state["codereview_issue"]
    55         self._tool.bugs.add_patch_to_bug(state["bug_id"], diff_file, description, comment_text=comment_text, mark_for_review=self._options.review, mark_for_commit_queue=self._options.request_commit)
     52        self._tool.bugs.add_patch_to_bug(state["bug_id"], diff, description, comment_text=comment_text, mark_for_review=self._options.review, mark_for_commit_queue=self._options.request_commit)
    5653        if self._options.open_bug:
    5754            self._tool.user.open_url(self._tool.bugs.bug_url_for_bug_id(state["bug_id"]))

  • trunk/WebKitTools/Scripts/webkitpy/tool/steps/postdiffforcommit.py

    r56497 r57907  
    2727# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
    2828
    29 import StringIO
    30 
    3129from webkitpy.tool.steps.abstractstep import AbstractStep
    3230
    … …  
    3634        self._tool.bugs.add_patch_to_bug(
    3735            state["bug_id"],
    38             StringIO.StringIO(self.cached_lookup(state, "diff")),
     36            self.cached_lookup(state, "diff"),
    3937            "Patch for landing",
    4038            mark_for_review=False,

  • trunk/WebKitTools/Scripts/webkitpy/tool/steps/postdiffforrevert.py

    r57462 r57907  
    2727# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
    2828
    29 import StringIO
    30 
    3129from webkitpy.common.net.bugzilla import Attachment
    3230from webkitpy.tool.steps.abstractstep import AbstractStep
    … …  
    4543        self._tool.bugs.add_patch_to_bug(
    4644            state["bug_id"],
    47             StringIO.StringIO(self.cached_lookup(state, "diff")),
     45            self.cached_lookup(state, "diff"),
    4846            "%s%s" % (Attachment.rollout_preamble, state["revision"]),
    4947            comment_text=comment_text,

  • trunk/WebKitTools/Scripts/webkitpy/tool/steps/steps_unittest.py

    r57788 r57907  
    4949        options = Mock()
    5050        options.update = True
    51         self._run_step(Update, options)
     51        expected_stderr = "Updating working directory\n"
     52        OutputCapture().assert_outputs(self, self._run_step, [Update, options], expected_stderr=expected_stderr)
    5253
    5354    def test_prompt_for_bug_or_title_step(self):
Note: See TracChangeset for help on using the changeset viewer.