⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 244828 in webkit


Ignore:
Timestamp:
May 1, 2019, 10:33:03 AM (7 years ago)
Author:
Ryan Haddad
Message:

WebKit has too much of its own UTF-8 code and should rely more on ICU's UTF-8 support
​https://bugs.webkit.org/show_bug.cgi?id=195535

Patch by Darin Adler <Darin Adler> on 2019-05-01
Reviewed by Alexey Proskuryakov.

LayoutTests/imported/w3c:

  • web-platform-tests/encoding/textdecoder-utf16-surrogates-expected.txt:

Updated expected results to have the Unicode replacement character in cases where the
text contains unpaired surrogates. The tests are still doing the same operations, and
still getting the same results, but the text output no longer includes illegal UTF-8.

Source/JavaScriptCore:

  • API/JSClassRef.cpp: Removed uneeded include of UTF8Conversion.h.
  • API/JSStringRef.cpp:

(JSStringCreateWithUTF8CString): Updated for changes to convertUTF8ToUTF16.
(JSStringGetUTF8CString): Updated for changes to convertLatin1ToUTF8.
Removed unneeded "true" to get the strict version of convertUTF16ToUTF8,
since that is the default. Also updated for changes to CompletionResult.

  • runtime/JSGlobalObjectFunctions.cpp:

(JSC::decode): Stop using UTF8SequenceLength, and instead use U8_COUNT_TRAIL_BYTES
and U8_MAX_LENGTH. Instead of decodeUTF8Sequence, use U8_NEXT. Also use U_IS_BMP,
U_IS_SUPPLEMENTARY, U16_LEAD, U16_TRAIL, and U_IS_SURROGATE instead of our own
equivalents, since these macros from ICU are correct and efficient.

  • wasm/WasmParser.h:

(JSC::Wasm::Parser<SuccessType>::consumeUTF8String): Updated for changes to
convertUTF8ToUTF16.

Source/WebCore:

  • platform/SharedBuffer.cpp:

(WebCore::utf8Buffer): Removed unnecessary "strict" argument to convertUTF16ToUTF8 since
that is the default behavior. Also updated for changes to return values.

  • xml/XSLTProcessorLibxslt.cpp:

(WebCore::writeToStringBuilder): Removed unnecessary use of StringBuffer for a temporary
buffer for characters. Rewrote to use U8_NEXT and U16_APPEND directly.

  • xml/parser/XMLDocumentParserLibxml2.cpp:

(WebCore::convertUTF16EntityToUTF8): Updated for changes to CompletionResult.

Source/WebKit:

  • Shared/API/APIString.h: Removed uneeded includes and also switched to #pragma once.
  • Shared/API/c/WKString.cpp: Moved include of UTF8Conversion.h here.

(WKStringGetUTF8CStringImpl): Updated for changes to return values.

Source/WTF:

  • wtf/text/AtomicString.cpp:

(WTF::AtomicString::fromUTF8Internal): Added code to compute string length when the
end is nullptr; this behavior used to be implemented inside the
calculateStringHashAndLengthFromUTF8MaskingTop8Bits function.

  • wtf/text/AtomicStringImpl.cpp:

(WTF::HashAndUTF8CharactersTranslator::translate): Updated for change to
convertUTF8ToUTF16.

  • wtf/text/AtomicStringImpl.h: Took the WTF_EXPORT_PRIVATE off of the

AtomicStringImpl::addUTF8 function. This is used only inside a non-inlined function in
the AtomicString class and its behavior changed subtly in this patch; it's helpful
to document that it's not exported.

  • wtf/text/StringImpl.cpp:

(WTF::StringImpl::utf8Impl): Don't pass "true" for strictness to convertUTF16ToUTF8
since strict is the default. Also updated for changes to ConversionResult.
(WTF::StringImpl::utf8ForCharacters): Updated for change to convertLatin1ToUTF8.
(WTF::StringImpl::tryGetUtf8ForRange const): Ditto.

  • wtf/text/StringView.cpp: Removed uneeded include of UTF8Conversion.h.
  • wtf/text/WTFString.cpp:

(WTF::String::fromUTF8): Updated for change to convertUTF8ToUTF16.

  • wtf/unicode/UTF8Conversion.cpp:

(WTF::Unicode::inlineUTF8SequenceLengthNonASCII): Deleted.
(WTF::Unicode::inlineUTF8SequenceLength): Deleted.
(WTF::Unicode::UTF8SequenceLength): Deleted.
(WTF::Unicode::decodeUTF8Sequence): Deleted.
(WTF::Unicode::convertLatin1ToUTF8): Use U8_APPEND, enabling us to remove
almost everything in the function. Also changed resturn value to be a boolean
to indicate success since there is only one possible failure (target exhausted).
There is room for further simplification, since most callers have lengths rather
than end pointers for the source buffer, and all but one caller supplies a buffer
size known to be sufficient, so those don't need a return value, nor do they need
to pass an end of buffer pointer.
(WTF::Unicode::convertUTF16ToUTF8): Use U_IS_LEAD, U_IS_TRAIL,
U16_GET_SUPPLEMENTARY, U_IS_SURROGATE, and U8_APPEND. Also changed behavior
for non-strict mode so that unpaired surrogates will be turned into the
replacement character instead of invalid UTF-8 sequences, because U8_APPEND
won't create an invalid UTF-8 sequence, and because we don't need to do that
for any good reason at any call site.
(WTF::Unicode::isLegalUTF8): Deleted.
(WTF::Unicode::readUTF8Sequence): Deleted.
(WTF::Unicode::convertUTF8ToUTF16): Use U8_NEXT instead of
inlineUTF8SequenceLength, isLegalUTF8, and readUTF8Sequence. Use
U16_APPEND instead of lots of code that does the same thing. There is
room for further simplification since most callers don't need the "all ASCII"
feature and could probably pass the arguments in a more natural way.
(WTF::Unicode::calculateStringHashAndLengthFromUTF8MaskingTop8Bits):
Use U8_NEXT instead of isLegalUTF8, readUTF8Sequence, and various
error handling checks for things that are handled by U8_NEXT. Also removed
support for passing nullptr for end to specify a null-terminated string.
(WTF::Unicode::equalUTF16WithUTF8): Ditto.

  • wtf/unicode/UTF8Conversion.h: Removed UTF8SequenceLength and

decodeUTF8Sequence. Changed the ConversionResult to match WebKit coding
style, with an eye toward perhaps removing it in the future. Changed
the convertUTF8ToUTF16 return value to a boolean and removed the "strict"
argument since no caller was passing false. Changed the convertLatin1ToUTF8
return value to a boolean. Tweaked comments.

LayoutTests:

  • css3/escape-dom-api-expected.txt:
  • fast/text/dangling-surrogates-expected.txt:
  • js/dom/webidl-type-mapping-expected.txt:
  • js/invalid-utf8-in-syntax-error-expected.txt:

Updated expected results to have the Unicode replacement character in cases where the
text contains unpaired surrogates. The tests are still doing the same operations, and
still getting the same results, but the text output no longer includes illegal UTF-8.

  • js/invalid-utf8-in-syntax-error.html: Added. Before adding this, the test was

run, but unlike the rest of the tests in this directory, was only run as part of
run-javascriptcore-tests. There are two reasons for adding this. One is to be
consistent with the rest of the tests here and run a second time as part of the
broader WebKit tests. The second is that we can now use "--reset-results" to generate
new expected results, something that run-webkit-tests has but run-javascriptcore-tests
does not have.

Location:
trunk
Files:
1 added
30 edited

Legend:

Unmodified
Added
Removed

  • trunk/LayoutTests/ChangeLog

    r244827 r244828  
     12019-05-01  Darin Adler  <darin@apple.com>
     2
     3        WebKit has too much of its own UTF-8 code and should rely more on ICU's UTF-8 support
     4        https://bugs.webkit.org/show_bug.cgi?id=195535
     5
     6        Reviewed by Alexey Proskuryakov.
     7
     8        * css3/escape-dom-api-expected.txt:
     9        * fast/text/dangling-surrogates-expected.txt:
     10        * js/dom/webidl-type-mapping-expected.txt:
     11        * js/invalid-utf8-in-syntax-error-expected.txt:
     12        Updated expected results to have the Unicode replacement character in cases where the
     13        text contains unpaired surrogates. The tests are still doing the same operations, and
     14        still getting the same results, but the text output no longer includes illegal UTF-8.
     15
     16        * js/invalid-utf8-in-syntax-error.html: Added. Before adding this, the test was
     17        run, but unlike the rest of the tests in this directory, was only run as part of
     18        run-javascriptcore-tests. There are two reasons for adding this. One is to be
     19        consistent with the rest of the tests here and run a second time as part of the
     20        broader WebKit tests. The second is that we can now use "--reset-results" to generate
     21        new expected results, something that run-webkit-tests has but run-javascriptcore-tests
     22        does not have.
     23
    1242019-05-01  Shawn Roberts  <sroberts@apple.com>
    225

  • trunk/LayoutTests/css3/escape-dom-api-expected.txt

    r244827 r244828  
    55
    66PASS CSS.escape.length is 1
    7 PASS CSS.escape('\0') is "�"
    8 PASS CSS.escape('a\0') is "a�"
    9 PASS CSS.escape('\0b') is "�b"
    10 PASS CSS.escape('a\0b') is "a�b"
    11 PASS CSS.escape('�') is "�"
    12 PASS CSS.escape('a�') is "a�"
    13 PASS CSS.escape('�b') is "�b"
    14 PASS CSS.escape('a�b') is "a�b"
     7PASS CSS.escape('\0') is "�"
     8PASS CSS.escape('a\0') is "a�"
     9PASS CSS.escape('\0b') is "�b"
     10PASS CSS.escape('a\0b') is "a�b"
     11PASS CSS.escape('�') is "�"
     12PASS CSS.escape('a�') is "a�"
     13PASS CSS.escape('�b') is "�b"
     14PASS CSS.escape('a�b') is "a�b"
    1515PASS CSS.escape() threw exception TypeError: Not enough arguments.
    1616PASS CSS.escape(undefined) is "undefined"
    … …  
    5555PASS CSS.escape('--') is "--"
    5656PASS CSS.escape('--a') is "--a"
    57 PASS CSS.escape('€-_©') is "€-_©"
    58 PASS CSS.escape('€‚ƒ„Â
    59 Â†Â‡ÂˆÂ‰ÂŠÂ‹ÂŒÂÂŽÂÂÂ‘’“”•–—˜™š›œžŸ') is "\\7f €‚ƒ„Â
    60 Â†Â‡ÂˆÂ‰ÂŠÂ‹ÂŒÂÂŽÂÂÂ‘’“”•–—˜™š›œžŸ"
    61 PASS CSS.escape(' ¡¢') is " ¡¢"
     57PASS CSS.escape('€-_©') is "€-_©"
     58PASS CSS.escape('€‚ƒ„
     59†‡ˆ‰Š‹ŒŽ‘’“”•–—˜™š›œžŸ') is "\\7f €‚ƒ„
     60†‡ˆ‰Š‹ŒŽ‘’“”•–—˜™š›œžŸ"
     61PASS CSS.escape(' ¡¢') is " ¡¢"
    6262PASS CSS.escape('a0123456789b') is "a0123456789b"
    6363PASS CSS.escape('abcdefghijklmnopqrstuvwxyz') is "abcdefghijklmnopqrstuvwxyz"
    6464PASS CSS.escape('ABCDEFGHIJKLMNOPQRSTUVWXYZ') is "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
    6565PASS CSS.escape(' !xy') is "\\ \\!xy"
    66 PASS CSS.escape('𝌆') is "𝌆"
    67 PASS CSS.escape('í¼†') is "\udf06"
    68 PASS CSS.escape('í ´') is "\ud834"
     66PASS CSS.escape('𝌆') is "𝌆"
     67PASS CSS.escape('�') is "\udf06"
     68PASS CSS.escape('�') is "\ud834"
    6969PASS successfullyParsed is true
    7070

  • trunk/LayoutTests/fast/text/dangling-surrogates-expected.txt

    r244827 r244828  
    44
    55
    6 PASS danglingFirst is "í ƒ"
    7 PASS danglingSecond is "í°"
     6PASS danglingFirst is "�"
     7PASS danglingSecond is "�"
    88PASS successfullyParsed is true
    99

  • trunk/LayoutTests/imported/w3c/ChangeLog

    r244827 r244828  
     12019-05-01  Darin Adler  <darin@apple.com>
     2
     3        WebKit has too much of its own UTF-8 code and should rely more on ICU's UTF-8 support
     4        https://bugs.webkit.org/show_bug.cgi?id=195535
     5
     6        Reviewed by Alexey Proskuryakov.
     7
     8        * web-platform-tests/encoding/textdecoder-utf16-surrogates-expected.txt:
     9        Updated expected results to have the Unicode replacement character in cases where the
     10        text contains unpaired surrogates. The tests are still doing the same operations, and
     11        still getting the same results, but the text output no longer includes illegal UTF-8.
     12
    1132019-05-01  Shawn Roberts  <sroberts@apple.com>
    214

  • trunk/LayoutTests/imported/w3c/web-platform-tests/encoding/textdecoder-utf16-surrogates-expected.txt

    r244827 r244828  
    11
    2 FAIL utf-16le - lone surrogate lead assert_equals: expected "\ufffd" but got "í €"
     2FAIL utf-16le - lone surrogate lead assert_equals: expected "\ufffd" but got "�"
    33FAIL utf-16le - lone surrogate lead (fatal flag set) assert_throws: function "function () {
    44            new TextDecoder(t.encoding, {fatal: true}).decode(new Uint8Array(t.input))
    55        }" did not throw
    6 FAIL utf-16le - lone surrogate trail assert_equals: expected "\ufffd" but got "í°€"
     6FAIL utf-16le - lone surrogate trail assert_equals: expected "\ufffd" but got "�"
    77FAIL utf-16le - lone surrogate trail (fatal flag set) assert_throws: function "function () {
    88            new TextDecoder(t.encoding, {fatal: true}).decode(new Uint8Array(t.input))
    99        }" did not throw
    10 FAIL utf-16le - unmatched surrogate lead assert_equals: expected "\ufffd\0" but got "í €\0"
     10FAIL utf-16le - unmatched surrogate lead assert_equals: expected "\ufffd\0" but got "�\0"
    1111FAIL utf-16le - unmatched surrogate lead (fatal flag set) assert_throws: function "function () {
    1212            new TextDecoder(t.encoding, {fatal: true}).decode(new Uint8Array(t.input))
    1313        }" did not throw
    14 FAIL utf-16le - unmatched surrogate trail assert_equals: expected "\ufffd\0" but got "í°€\0"
     14FAIL utf-16le - unmatched surrogate trail assert_equals: expected "\ufffd\0" but got "�\0"
    1515FAIL utf-16le - unmatched surrogate trail (fatal flag set) assert_throws: function "function () {
    1616            new TextDecoder(t.encoding, {fatal: true}).decode(new Uint8Array(t.input))
    1717        }" did not throw
    18 FAIL utf-16le - swapped surrogate pair assert_equals: expected "\ufffd\ufffd" but got "í°€í €"
     18FAIL utf-16le - swapped surrogate pair assert_equals: expected "\ufffd\ufffd" but got "��"
    1919FAIL utf-16le - swapped surrogate pair (fatal flag set) assert_throws: function "function () {
    2020            new TextDecoder(t.encoding, {fatal: true}).decode(new Uint8Array(t.input))

  • trunk/LayoutTests/js/dom/webidl-type-mapping-expected.txt

    r244827 r244828  
    10101010converter.testUSVString = '!@#123ABCabc\x00\x80\xFF\r\n\t'
    10111011converter.testString = '!@#123ABCabc\x00\x80\xFF\r\n\t'
    1012 PASS converter.testUSVString is "!@#123ABCabc\u0000€ÿ\r\n\t"
    1013 PASS converter.testString is "!@#123ABCabc\u0000€ÿ\r\n\t"
     1012PASS converter.testUSVString is "!@#123ABCabc\u0000€ÿ\r\n\t"
     1013PASS converter.testString is "!@#123ABCabc\u0000€ÿ\r\n\t"
    10141014converter.testUSVString = '\u0100'
    10151015converter.testString = '\u0100'
    1016 PASS converter.testUSVString is "Ā"
    1017 PASS converter.testString is "Ā"
     1016PASS converter.testUSVString is "Ā"
     1017PASS converter.testString is "Ā"
    10181018PASS converter.testUSVString = {toString: function() { throw Error(); }} threw exception Error.
    10191019PASS converter.testString = {toString: function() { throw Error(); }} threw exception Error.
    1020 PASS converter.testUSVString is "Ā"
    1021 PASS converter.testString is "Ā"
     1020PASS converter.testUSVString is "Ā"
     1021PASS converter.testString is "Ā"
    10221022converter.testUSVString = "\ud800"
    10231023converter.testString = "\ud800"
    1024 PASS converter.testUSVString is "�"
     1024PASS converter.testUSVString is "�"
    10251025PASS converter.testString is "\ud800"
    10261026converter.testUSVString = "\udc00"
    10271027converter.testString = "\udc00"
    1028 PASS converter.testUSVString is "�"
     1028PASS converter.testUSVString is "�"
    10291029PASS converter.testString is "\udc00"
    10301030converter.testUSVString = "\ud800\u0000"
    10311031converter.testString = "\ud800\u0000"
    1032 PASS converter.testUSVString is "�\u0000"
     1032PASS converter.testUSVString is "�\u0000"
    10331033PASS converter.testString is "\ud800\u0000"
    10341034converter.testUSVString = "\udc00\u0000"
    10351035converter.testString = "\udc00\u0000"
    1036 PASS converter.testUSVString is "�\u0000"
     1036PASS converter.testUSVString is "�\u0000"
    10371037PASS converter.testString is "\udc00\u0000"
    10381038converter.testUSVString = "\udc00\ud800"
    10391039converter.testString = "\udc00\ud800"
    1040 PASS converter.testUSVString is "��"
     1040PASS converter.testUSVString is "��"
    10411041PASS converter.testString is "\udc00\ud800"
    1042 converter.testUSVString = "𝄞"
    1043 converter.testString = "𝄞"
    1044 PASS converter.testUSVString is "𝄞"
    1045 PASS converter.testString is "𝄞"
     1042converter.testUSVString = "𝄞"
     1043converter.testString = "𝄞"
     1044PASS converter.testUSVString is "𝄞"
     1045PASS converter.testString is "𝄞"
    10461046converter.testByteString = '!@#123ABCabc\x00\x80\xFF\r\n\t'
    1047 PASS converter.testByteString is "!@#123ABCabc\u0000€ÿ\r\n\t"
     1047PASS converter.testByteString is "!@#123ABCabc\u0000€ÿ\r\n\t"
    10481048converter.testByteString = '\u00FF'
    1049 PASS converter.testByteString is "ÿ"
     1049PASS converter.testByteString is "ÿ"
    10501050PASS converter.testByteString = '\u0100' threw exception TypeError: Type error.
    1051 PASS converter.testByteString is "ÿ"
     1051PASS converter.testByteString is "ÿ"
    10521052PASS converter.testByteString = {toString: function() { throw Error(); }} threw exception Error.
    1053 PASS converter.testByteString is "ÿ"
     1053PASS converter.testByteString is "ÿ"
    10541054converter.testUSVString = true
    10551055converter.testString = true
    … …  
    11811181PASS converter.testNodeRecord()['key2'] is document.documentElement
    11821182PASS converter.setTestNodeRecord({ key: 'hello' }) threw exception TypeError: Type error.
    1183 converter.setTestLongRecord({'í €': 1 })
    1184 PASS converter.testLongRecord()['í €'] is 1
    1185 converter.setTestNodeRecord({'í €': document })
    1186 PASS converter.testNodeRecord()['�'] is document
    1187 converter.setTestLongRecord({'í°€': 1 })
    1188 PASS converter.testLongRecord()['í°€'] is 1
    1189 converter.setTestNodeRecord({'í°€': document })
    1190 PASS converter.testNodeRecord()['�'] is document
    1191 converter.setTestLongRecord({'í €': 1 })
    1192 PASS converter.testLongRecord()['í €\0'] is 1
    1193 converter.setTestNodeRecord({'í €': document })
    1194 PASS converter.testNodeRecord()['�\0'] is document
    1195 converter.setTestLongRecord({'í°€': 1 })
    1196 PASS converter.testLongRecord()['í°€\0'] is 1
    1197 converter.setTestNodeRecord({'í°€': document })
    1198 PASS converter.testNodeRecord()['�\0'] is document
    1199 converter.setTestLongRecord({'í°€í €': 1 })
    1200 PASS converter.testLongRecord()['í°€í €'] is 1
    1201 converter.setTestNodeRecord({'í°€í €': document })
    1202 PASS converter.testNodeRecord()['��'] is document
    1203 converter.setTestLongRecord({'𝄞': 1 })
    1204 PASS converter.testLongRecord()['𝄞'] is 1
    1205 converter.setTestNodeRecord({'𝄞': document })
    1206 PASS converter.testNodeRecord()['𝄞'] is document
     1183converter.setTestLongRecord({'�': 1 })
     1184PASS converter.testLongRecord()['�'] is 1
     1185converter.setTestNodeRecord({'�': document })
     1186PASS converter.testNodeRecord()['�'] is document
     1187converter.setTestLongRecord({'�': 1 })
     1188PASS converter.testLongRecord()['�'] is 1
     1189converter.setTestNodeRecord({'�': document })
     1190PASS converter.testNodeRecord()['�'] is document
     1191converter.setTestLongRecord({'�': 1 })
     1192PASS converter.testLongRecord()['�\0'] is 1
     1193converter.setTestNodeRecord({'�': document })
     1194PASS converter.testNodeRecord()['�\0'] is document
     1195converter.setTestLongRecord({'�': 1 })
     1196PASS converter.testLongRecord()['�\0'] is 1
     1197converter.setTestNodeRecord({'�': document })
     1198PASS converter.testNodeRecord()['�\0'] is document
     1199converter.setTestLongRecord({'��': 1 })
     1200PASS converter.testLongRecord()['��'] is 1
     1201converter.setTestNodeRecord({'��': document })
     1202PASS converter.testNodeRecord()['��'] is document
     1203converter.setTestLongRecord({'𝄞': 1 })
     1204PASS converter.testLongRecord()['𝄞'] is 1
     1205converter.setTestNodeRecord({'𝄞': document })
     1206PASS converter.testNodeRecord()['𝄞'] is document
    12071207converter.setTestSequenceRecord({ key: ['value', 'other value'] })
    12081208PASS converter.testSequenceRecord().hasOwnProperty('key') is true
    12091209PASS 'key' in converter.testSequenceRecord() is true
    12101210PASS converter.testSequenceRecord()['key'] is ['value', 'other value']
    1211 PASS converter.setTestSequenceRecord({ 'Ā': ['value'] }) threw exception TypeError: Type error.
    1212 converter.setTestSequenceRecord({ 'ÿ': ['value'] })
    1213 PASS converter.testSequenceRecord()['ÿ'] is ['value']
     1211PASS converter.setTestSequenceRecord({ 'Ā': ['value'] }) threw exception TypeError: Type error.
     1212converter.setTestSequenceRecord({ 'ÿ': ['value'] })
     1213PASS converter.testSequenceRecord()['ÿ'] is ['value']
    12141214PASS converter.testImpureNaNUnrestrictedDouble is NaN
    12151215PASS converter.testImpureNaN2UnrestrictedDouble is NaN

  • trunk/LayoutTests/js/invalid-utf8-in-syntax-error-expected.txt

    r244827 r244828  
    44
    55
    6 PASS ({f("íº­")}) threw exception SyntaxError: Unexpected string literal "íº­". Expected a parameter pattern or a ')' in parameter list..
     6PASS ({f("�")}) threw exception SyntaxError: Unexpected string literal "�". Expected a parameter pattern or a ')' in parameter list..
    77PASS successfullyParsed is true
    88

  • trunk/Source/JavaScriptCore/API/JSClassRef.cpp

    r244827 r244828  
    3636#include "JSCInlines.h"
    3737#include <wtf/text/StringHash.h>
    38 #include <wtf/unicode/UTF8Conversion.h>
    3938
    4039using namespace JSC;
    41 using namespace WTF::Unicode;
    4240
    4341const JSClassDefinition kJSClassDefinitionEmpty = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 };

  • trunk/Source/JavaScriptCore/API/JSStringRef.cpp

    r244827 r244828  
    5050        bool sourceIsAllASCII;
    5151        const LChar* stringStart = reinterpret_cast<const LChar*>(string);
    52         if (conversionOK == convertUTF8ToUTF16(&string, string + length, &p, p + length, &sourceIsAllASCII)) {
     52        if (convertUTF8ToUTF16(string, string + length, &p, p + length, &sourceIsAllASCII)) {
    5353            if (sourceIsAllASCII)
    5454                return &OpaqueJSString::create(stringStart, length).leakRef();
    … …  
    103103
    104104    char* destination = buffer;
    105     ConversionResult result;
     105    bool failed = false;
    106106    if (string->is8Bit()) {
    107107        const LChar* source = string->characters8();
    108         result = convertLatin1ToUTF8(&source, source + string->length(), &destination, destination + bufferSize - 1);
     108        convertLatin1ToUTF8(&source, source + string->length(), &destination, destination + bufferSize - 1);
    109109    } else {
    110110        const UChar* source = string->characters16();
    111         result = convertUTF16ToUTF8(&source, source + string->length(), &destination, destination + bufferSize - 1, true);
     111        auto result = convertUTF16ToUTF8(&source, source + string->length(), &destination, destination + bufferSize - 1);
     112        failed = result != ConversionOK && result != TargetExhausted;
    112113    }
    113114
    114115    *destination++ = '\0';
    115     if (result != conversionOK && result != targetExhausted)
    116         return 0;
    117 
    118     return destination - buffer;
     116    return failed ? 0 : destination - buffer;
    119117}
    120118

  • trunk/Source/JavaScriptCore/ChangeLog

    r244827 r244828  
     12019-05-01  Darin Adler  <darin@apple.com>
     2
     3        WebKit has too much of its own UTF-8 code and should rely more on ICU's UTF-8 support
     4        https://bugs.webkit.org/show_bug.cgi?id=195535
     5
     6        Reviewed by Alexey Proskuryakov.
     7
     8        * API/JSClassRef.cpp: Removed uneeded include of UTF8Conversion.h.
     9
     10        * API/JSStringRef.cpp:
     11        (JSStringCreateWithUTF8CString): Updated for changes to convertUTF8ToUTF16.
     12        (JSStringGetUTF8CString): Updated for changes to convertLatin1ToUTF8.
     13        Removed unneeded "true" to get the strict version of convertUTF16ToUTF8,
     14        since that is the default. Also updated for changes to CompletionResult.
     15
     16        * runtime/JSGlobalObjectFunctions.cpp:
     17        (JSC::decode): Stop using UTF8SequenceLength, and instead use U8_COUNT_TRAIL_BYTES
     18        and U8_MAX_LENGTH. Instead of decodeUTF8Sequence, use U8_NEXT. Also use U_IS_BMP,
     19        U_IS_SUPPLEMENTARY, U16_LEAD, U16_TRAIL, and U_IS_SURROGATE instead of our own
     20        equivalents, since these macros from ICU are correct and efficient.
     21
     22        * wasm/WasmParser.h:
     23        (JSC::Wasm::Parser<SuccessType>::consumeUTF8String): Updated for changes to
     24        convertUTF8ToUTF16.
     25
    1262019-05-01  Shawn Roberts  <sroberts@apple.com>
    227

  • trunk/Source/JavaScriptCore/runtime/JSGlobalObjectFunctions.cpp

    r244827 r244828  
    5959#include <wtf/dtoa.h>
    6060#include <wtf/text/StringBuilder.h>
    61 #include <wtf/unicode/UTF8Conversion.h>
    6261
    6362namespace JSC {
    64 
    65 using namespace WTF::Unicode;
    6663
    6764const ASCIILiteral ObjectProtoCalledOnNullOrUndefinedError { "Object.prototype.__proto__ called on null or undefined"_s };
    … …  
    185182            if (k <= length - 3 && isASCIIHexDigit(p[1]) && isASCIIHexDigit(p[2])) {
    186183                const char b0 = Lexer<CharType>::convertHex(p[1], p[2]);
    187                 const int sequenceLen = UTF8SequenceLength(b0);
    188                 if (sequenceLen && k <= length - sequenceLen * 3) {
     184                const int sequenceLen = 1 + U8_COUNT_TRAIL_BYTES(b0);
     185                if (k <= length - sequenceLen * 3) {
    189186                    charLen = sequenceLen * 3;
    190                     char sequence[5];
     187                    uint8_t sequence[U8_MAX_LENGTH];
    191188                    sequence[0] = b0;
    192189                    for (int i = 1; i < sequenceLen; ++i) {
    … …  
    200197                    }
    201198                    if (charLen != 0) {
    202                         sequence[sequenceLen] = 0;
    203                         const int character = decodeUTF8Sequence(sequence);
    204                         if (character < 0 || character >= 0x110000)
     199                        UChar32 character;
     200                        int32_t offset = 0;
     201                        U8_NEXT(sequence, offset, sequenceLen, character);
     202                        if (character < 0)
    205203                            charLen = 0;
    206                         else if (character >= 0x10000) {
     204                        else if (!U_IS_BMP(character)) {
    207205                            // Convert to surrogate pair.
    208                             builder.append(static_cast<UChar>(0xD800 | ((character - 0x10000) >> 10)));
    209                             u = static_cast<UChar>(0xDC00 | ((character - 0x10000) & 0x3FF));
    210                         } else
     206                            ASSERT(U_IS_SUPPLEMENTARY(character));
     207                            builder.append(U16_LEAD(character));
     208                            u = U16_TRAIL(character);
     209                        } else {
     210                            ASSERT(!U_IS_SURROGATE(character));
    211211                            u = static_cast<UChar>(character);
     212                        }
    212213                    }
    213214                }

  • trunk/Source/JavaScriptCore/wasm/WasmParser.h

    r244827 r244828  
    163163        UChar* bufferCurrent = bufferStart;
    164164        const char* stringCurrent = reinterpret_cast<const char*>(stringStart);
    165         if (WTF::Unicode::convertUTF8ToUTF16(&stringCurrent, reinterpret_cast<const char *>(stringStart + stringLength), &bufferCurrent, bufferCurrent + buffer.size()) != WTF::Unicode::conversionOK)
     165        if (!WTF::Unicode::convertUTF8ToUTF16(stringCurrent, reinterpret_cast<const char *>(stringStart + stringLength), &bufferCurrent, bufferCurrent + buffer.size()))
    166166            return false;
    167167    }

  • trunk/Source/WTF/ChangeLog

    r244827 r244828  
     12019-05-01  Darin Adler  <darin@apple.com>
     2
     3        WebKit has too much of its own UTF-8 code and should rely more on ICU's UTF-8 support
     4        https://bugs.webkit.org/show_bug.cgi?id=195535
     5
     6        Reviewed by Alexey Proskuryakov.
     7
     8        * wtf/text/AtomicString.cpp:
     9        (WTF::AtomicString::fromUTF8Internal): Added code to compute string length when the
     10        end is nullptr; this behavior used to be implemented inside the
     11        calculateStringHashAndLengthFromUTF8MaskingTop8Bits function.
     12
     13        * wtf/text/AtomicStringImpl.cpp:
     14        (WTF::HashAndUTF8CharactersTranslator::translate): Updated for change to
     15        convertUTF8ToUTF16.
     16
     17        * wtf/text/AtomicStringImpl.h: Took the WTF_EXPORT_PRIVATE off of the
     18        AtomicStringImpl::addUTF8 function. This is used only inside a non-inlined function in
     19        the AtomicString class and its behavior changed subtly in this patch; it's helpful
     20        to document that it's not exported.
     21
     22        * wtf/text/StringImpl.cpp:
     23        (WTF::StringImpl::utf8Impl): Don't pass "true" for strictness to convertUTF16ToUTF8
     24        since strict is the default. Also updated for changes to ConversionResult.
     25        (WTF::StringImpl::utf8ForCharacters): Updated for change to convertLatin1ToUTF8.
     26        (WTF::StringImpl::tryGetUtf8ForRange const): Ditto.
     27
     28        * wtf/text/StringView.cpp: Removed uneeded include of UTF8Conversion.h.
     29
     30        * wtf/text/WTFString.cpp:
     31        (WTF::String::fromUTF8): Updated for change to convertUTF8ToUTF16.
     32
     33        * wtf/unicode/UTF8Conversion.cpp:
     34        (WTF::Unicode::inlineUTF8SequenceLengthNonASCII): Deleted.
     35        (WTF::Unicode::inlineUTF8SequenceLength): Deleted.
     36        (WTF::Unicode::UTF8SequenceLength): Deleted.
     37        (WTF::Unicode::decodeUTF8Sequence): Deleted.
     38        (WTF::Unicode::convertLatin1ToUTF8): Use U8_APPEND, enabling us to remove
     39        almost everything in the function. Also changed resturn value to be a boolean
     40        to indicate success since there is only one possible failure (target exhausted).
     41        There is room for further simplification, since most callers have lengths rather
     42        than end pointers for the source buffer, and all but one caller supplies a buffer
     43        size known to be sufficient, so those don't need a return value, nor do they need
     44        to pass an end of buffer pointer.
     45        (WTF::Unicode::convertUTF16ToUTF8): Use U_IS_LEAD, U_IS_TRAIL,
     46        U16_GET_SUPPLEMENTARY, U_IS_SURROGATE, and U8_APPEND. Also changed behavior
     47        for non-strict mode so that unpaired surrogates will be turned into the
     48        replacement character instead of invalid UTF-8 sequences, because U8_APPEND
     49        won't create an invalid UTF-8 sequence, and because we don't need to do that
     50        for any good reason at any call site.
     51        (WTF::Unicode::isLegalUTF8): Deleted.
     52        (WTF::Unicode::readUTF8Sequence): Deleted.
     53        (WTF::Unicode::convertUTF8ToUTF16): Use U8_NEXT instead of
     54        inlineUTF8SequenceLength, isLegalUTF8, and readUTF8Sequence. Use
     55        U16_APPEND instead of lots of code that does the same thing. There is
     56        room for further simplification since most callers don't need the "all ASCII"
     57        feature and could probably pass the arguments in a more natural way.
     58        (WTF::Unicode::calculateStringHashAndLengthFromUTF8MaskingTop8Bits):
     59        Use U8_NEXT instead of isLegalUTF8, readUTF8Sequence, and various
     60        error handling checks for things that are handled by U8_NEXT. Also removed
     61        support for passing nullptr for end to specify a null-terminated string.
     62        (WTF::Unicode::equalUTF16WithUTF8): Ditto.
     63
     64        * wtf/unicode/UTF8Conversion.h: Removed UTF8SequenceLength and
     65        decodeUTF8Sequence. Changed the ConversionResult to match WebKit coding
     66        style, with an eye toward perhaps removing it in the future. Changed
     67        the convertUTF8ToUTF16 return value to a boolean and removed the "strict"
     68        argument since no caller was passing false. Changed the convertLatin1ToUTF8
     69        return value to a boolean. Tweaked comments.
     70
    1712019-05-01  Shawn Roberts  <sroberts@apple.com>
    272

  • trunk/Source/WTF/wtf/text/AtomicString.cpp

    r244827 r244828  
    114114}
    115115
    116 AtomicString AtomicString::fromUTF8Internal(const char* charactersStart, const char* charactersEnd)
     116AtomicString AtomicString::fromUTF8Internal(const char* start, const char* end)
    117117{
    118     auto impl = AtomicStringImpl::addUTF8(charactersStart, charactersEnd);
    119     if (!impl)
    120         return nullAtom();
    121     return impl.get();
     118    ASSERT(start);
     119
     120    // Caller needs to handle empty string.
     121    ASSERT(!end || end > start);
     122    ASSERT(end || start[0]);
     123
     124    return AtomicStringImpl::addUTF8(start, end ? end : start + std::strlen(start));
    122125}
    123126
    124127#ifndef NDEBUG
     128
    125129void AtomicString::show() const
    126130{
    127131    m_string.show();
    128132}
     133
    129134#endif
    130135

  • trunk/Source/WTF/wtf/text/AtomicStringImpl.cpp

    r244827 r244828  
    220220        bool isAllASCII;
    221221        const char* source = buffer.characters;
    222         if (convertUTF8ToUTF16(&source, source + buffer.length, &target, target + buffer.utf16Length, &isAllASCII) != conversionOK)
     222        if (!convertUTF8ToUTF16(source, source + buffer.length, &target, target + buffer.utf16Length, &isAllASCII))
    223223            ASSERT_NOT_REACHED();
    224224

  • trunk/Source/WTF/wtf/text/AtomicStringImpl.h

    r244827 r244828  
    5757
    5858    // Returns null if the input data contains an invalid UTF-8 sequence.
    59     WTF_EXPORT_PRIVATE static RefPtr<AtomicStringImpl> addUTF8(const char* start, const char* end);
     59    static RefPtr<AtomicStringImpl> addUTF8(const char* start, const char* end);
     60
    6061#if USE(CF)
    6162    WTF_EXPORT_PRIVATE static RefPtr<AtomicStringImpl> add(CFStringRef);

  • trunk/Source/WTF/wtf/text/StringImpl.cpp

    r244827 r244828  
    17571757        while (characters < charactersEnd) {
    17581758            // Use strict conversion to detect unpaired surrogates.
    1759             ConversionResult result = convertUTF16ToUTF8(&characters, charactersEnd, &buffer, bufferEnd, true);
    1760             ASSERT(result != targetExhausted);
     1759            auto result = convertUTF16ToUTF8(&characters, charactersEnd, &buffer, bufferEnd);
     1760            ASSERT(result != TargetExhausted);
    17611761            // Conversion fails when there is an unpaired surrogate.
    17621762            // Put replacement character (U+FFFD) instead of the unpaired surrogate.
    1763             if (result != conversionOK) {
     1763            if (result != ConversionOK) {
    17641764                ASSERT((0xD800 <= *characters && *characters <= 0xDFFF));
    17651765                // There should be room left, since one UChar hasn't been converted.
    … …  
    17721772        bool strict = mode == StrictConversion;
    17731773        const UChar* originalCharacters = characters;
    1774         ConversionResult result = convertUTF16ToUTF8(&characters, characters + length, &buffer, buffer + bufferSize, strict);
    1775         ASSERT(result != targetExhausted); // (length * 3) should be sufficient for any conversion
     1774        auto result = convertUTF16ToUTF8(&characters, characters + length, &buffer, buffer + bufferSize, strict);
     1775        ASSERT(result != TargetExhausted); // (length * 3) should be sufficient for any conversion
    17761776
    17771777        // Only produced from strict conversion.
    1778         if (result == sourceIllegal) {
     1778        if (result == SourceIllegal) {
    17791779            ASSERT(strict);
    17801780            return UTF8ConversionError::IllegalSource;
    … …  
    17821782
    17831783        // Check for an unconverted high surrogate.
    1784         if (result == sourceExhausted) {
     1784        if (result == SourceExhausted) {
    17851785            if (strict)
    17861786                return UTF8ConversionError::SourceExhausted;
    … …  
    18101810    char* buffer = bufferVector.data();
    18111811    const LChar* source = characters;
    1812     ConversionResult result = convertLatin1ToUTF8(&source, source + length, &buffer, buffer + bufferVector.size());
    1813     ASSERT_UNUSED(result, result != targetExhausted); // (length * 3) should be sufficient for any conversion
     1812    bool success = convertLatin1ToUTF8(&source, source + length, &buffer, buffer + bufferVector.size());
     1813    ASSERT_UNUSED(success, success); // (length * 3) should be sufficient for any conversion
    18141814    return CString(bufferVector.data(), buffer - bufferVector.data());
    18151815}
    … …  
    18551855    if (is8Bit()) {
    18561856        const LChar* characters = this->characters8() + offset;
    1857 
    1858         ConversionResult result = convertLatin1ToUTF8(&characters, characters + length, &buffer, buffer + bufferVector.size());
    1859         ASSERT_UNUSED(result, result != targetExhausted); // (length * 3) should be sufficient for any conversion
     1857        auto success = convertLatin1ToUTF8(&characters, characters + length, &buffer, buffer + bufferVector.size());
     1858        ASSERT_UNUSED(success, success); // (length * 3) should be sufficient for any conversion
    18601859    } else {
    18611860        UTF8ConversionError error = utf8Impl(this->characters16() + offset, length, buffer, bufferVector.size(), mode);

  • trunk/Source/WTF/wtf/text/StringView.cpp

    r244827 r244828  
    3636#include <wtf/Optional.h>
    3737#include <wtf/text/TextBreakIterator.h>
    38 #include <wtf/unicode/UTF8Conversion.h>
    3938
    4039namespace WTF {
    41 
    42 using namespace Unicode;
    4340
    4441bool StringView::containsIgnoringASCIICase(const StringView& matchString) const

  • trunk/Source/WTF/wtf/text/WTFString.cpp

    r244827 r244828  
    860860    UChar* bufferCurrent = bufferStart;
    861861    const char* stringCurrent = reinterpret_cast<const char*>(stringStart);
    862     if (convertUTF8ToUTF16(&stringCurrent, reinterpret_cast<const char *>(stringStart + length), &bufferCurrent, bufferCurrent + buffer.size()) != conversionOK)
     862    if (!convertUTF8ToUTF16(stringCurrent, reinterpret_cast<const char *>(stringStart + length), &bufferCurrent, bufferCurrent + buffer.size()))
    863863        return String();
    864864

  • trunk/Source/WTF/wtf/unicode/UTF8Conversion.cpp

    r244827 r244828  
    11/*
    2  * Copyright (C) 2007, 2014 Apple Inc. All rights reserved.
     2 * Copyright (C) 2007-2019 Apple Inc. All rights reserved.
    33 * Copyright (C) 2010 Patrick Gansterer <paroga@paroga.com>
    44 *
    … …  
    3535namespace Unicode {
    3636
    37 inline int inlineUTF8SequenceLengthNonASCII(char b0)
     37bool convertLatin1ToUTF8(const LChar** sourceStart, const LChar* sourceEnd, char** targetStart, char* targetEnd)
    3838{
    39     if ((b0 & 0xC0) != 0xC0)
    40         return 0;
    41     if ((b0 & 0xE0) == 0xC0)
    42         return 2;
    43     if ((b0 & 0xF0) == 0xE0)
    44         return 3;
    45     if ((b0 & 0xF8) == 0xF0)
    46         return 4;
    47     return 0;
     39    const LChar* source;
     40    char* target = *targetStart;
     41    int i = 0;
     42    for (source = *sourceStart; source < sourceEnd; ++source) {
     43        UBool sawError = false;
     44        // Work around bug in either Windows compiler or old version of ICU, where passing a uint8_t to
     45        // U8_APPEND warns, by converting from uint8_t to a wider type.
     46        UChar32 character = *source;
     47        U8_APPEND(reinterpret_cast<uint8_t*>(target), i, targetEnd - *targetStart, character, sawError);
     48        if (sawError)
     49            return false;
     50    }
     51    *sourceStart = source;
     52    *targetStart = target + i;
     53    return true;
    4854}
    4955
    50 inline int inlineUTF8SequenceLength(char b0)
     56ConversionResult convertUTF16ToUTF8(const UChar** sourceStart, const UChar* sourceEnd, char** targetStart, char* targetEnd, bool strict)
    5157{
    52     return isASCII(b0) ? 1 : inlineUTF8SequenceLengthNonASCII(b0);
    53 }
    54 
    55 int UTF8SequenceLength(char b0)
    56 {
    57     return isASCII(b0) ? 1 : inlineUTF8SequenceLengthNonASCII(b0);
    58 }
    59 
    60 int decodeUTF8Sequence(const char* sequence)
    61 {
    62     // Handle 0-byte sequences (never valid).
    63     const unsigned char b0 = sequence[0];
    64     const int length = inlineUTF8SequenceLength(b0);
    65     if (length == 0)
    66         return -1;
    67 
    68     // Handle 1-byte sequences (plain ASCII).
    69     const unsigned char b1 = sequence[1];
    70     if (length == 1) {
    71         if (b1)
    72             return -1;
    73         return b0;
    74     }
    75 
    76     // Handle 2-byte sequences.
    77     if ((b1 & 0xC0) != 0x80)
    78         return -1;
    79     const unsigned char b2 = sequence[2];
    80     if (length == 2) {
    81         if (b2)
    82             return -1;
    83         const int c = ((b0 & 0x1F) << 6) | (b1 & 0x3F);
    84         if (c < 0x80)
    85             return -1;
    86         return c;
    87     }
    88 
    89     // Handle 3-byte sequences.
    90     if ((b2 & 0xC0) != 0x80)
    91         return -1;
    92     const unsigned char b3 = sequence[3];
    93     if (length == 3) {
    94         if (b3)
    95             return -1;
    96         const int c = ((b0 & 0xF) << 12) | ((b1 & 0x3F) << 6) | (b2 & 0x3F);
    97         if (c < 0x800)
    98             return -1;
    99         // UTF-16 surrogates should never appear in UTF-8 data.
    100         if (c >= 0xD800 && c <= 0xDFFF)
    101             return -1;
    102         return c;
    103     }
    104 
    105     // Handle 4-byte sequences.
    106     if ((b3 & 0xC0) != 0x80)
    107         return -1;
    108     const unsigned char b4 = sequence[4];
    109     if (length == 4) {
    110         if (b4)
    111             return -1;
    112         const int c = ((b0 & 0x7) << 18) | ((b1 & 0x3F) << 12) | ((b2 & 0x3F) << 6) | (b3 & 0x3F);
    113         if (c < 0x10000 || c > 0x10FFFF)
    114             return -1;
    115         return c;
    116     }
    117 
    118     return -1;
    119 }
    120 
    121 // Once the bits are split out into bytes of UTF-8, this is a mask OR-ed
    122 // into the first byte, depending on how many bytes follow.  There are
    123 // as many entries in this table as there are UTF-8 sequence types.
    124 // (I.e., one byte sequence, two byte... etc.). Remember that sequencs
    125 // for *legal* UTF-8 will be 4 or fewer bytes total.
    126 static const unsigned char firstByteMark[7] = { 0x00, 0x00, 0xC0, 0xE0, 0xF0, 0xF8, 0xFC };
    127 
    128 ConversionResult convertLatin1ToUTF8(
    129     const LChar** sourceStart, const LChar* sourceEnd,
    130     char** targetStart, char* targetEnd)
    131 {
    132     ConversionResult result = conversionOK;
    133     const LChar* source = *sourceStart;
     58    ConversionResult result = ConversionOK;
     59    const UChar* source = *sourceStart;
    13460    char* target = *targetStart;
     61    UBool sawError = false;
     62    int i = 0;
    13563    while (source < sourceEnd) {
    13664        UChar32 ch;
    137         unsigned short bytesToWrite = 0;
    138         const UChar32 byteMask = 0xBF;
    139         const UChar32 byteMark = 0x80;
    140         const LChar* oldSource = source; // In case we have to back up because of target overflow.
    141         ch = static_cast<unsigned short>(*source++);
    142 
    143         // Figure out how many bytes the result will require
    144         if (ch < (UChar32)0x80)
    145             bytesToWrite = 1;
    146         else
    147             bytesToWrite = 2;
    148 
    149         target += bytesToWrite;
    150         if (target > targetEnd) {
    151             source = oldSource; // Back up source pointer!
    152             target -= bytesToWrite;
    153             result = targetExhausted;
     65        int j = 0;
     66        U16_NEXT(source, j, sourceEnd - source, ch);
     67        if (U_IS_SURROGATE(ch)) {
     68            if (source + j == sourceEnd && U_IS_SURROGATE_LEAD(ch)) {
     69                result = SourceExhausted;
     70                break;
     71            }
     72            if (strict) {
     73                result = SourceIllegal;
     74                break;
     75            }
     76            ch = replacementCharacter;
     77        }
     78        U8_APPEND(reinterpret_cast<uint8_t*>(target), i, targetEnd - target, ch, sawError);
     79        if (sawError) {
     80            result = TargetExhausted;
    15481            break;
    15582        }
    156         switch (bytesToWrite) { // note: everything falls through.
    157         case 2:
    158             *--target = (char)((ch | byteMark) & byteMask);
    159             ch >>= 6;
    160             FALLTHROUGH;
    161         case 1:
    162             *--target =  (char)(ch | firstByteMark[bytesToWrite]);
    163         }
    164         target += bytesToWrite;
     83        source += j;
    16584    }
    16685    *sourceStart = source;
    167     *targetStart = target;
     86    *targetStart = target + i;
    16887    return result;
    16988}
    17089
    171 ConversionResult convertUTF16ToUTF8(
    172     const UChar** sourceStart, const UChar* sourceEnd,
    173     char** targetStart, char* targetEnd, bool strict)
     90bool convertUTF8ToUTF16(const char* source, const char* sourceEnd, UChar** targetStart, UChar* targetEnd, bool* sourceAllASCII)
    17491{
    175     ConversionResult result = conversionOK;
    176     const UChar* source = *sourceStart;
    177     char* target = *targetStart;
    178     while (source < sourceEnd) {
    179         UChar32 ch;
    180         unsigned short bytesToWrite = 0;
    181         const UChar32 byteMask = 0xBF;
    182         const UChar32 byteMark = 0x80;
    183         const UChar* oldSource = source; // In case we have to back up because of target overflow.
    184         ch = static_cast<unsigned short>(*source++);
    185         // If we have a surrogate pair, convert to UChar32 first.
    186         if (ch >= 0xD800 && ch <= 0xDBFF) {
    187             // If the 16 bits following the high surrogate are in the source buffer...
    188             if (source < sourceEnd) {
    189                 UChar32 ch2 = static_cast<unsigned short>(*source);
    190                 // If it's a low surrogate, convert to UChar32.
    191                 if (ch2 >= 0xDC00 && ch2 <= 0xDFFF) {
    192                     ch = ((ch - 0xD800) << 10) + (ch2 - 0xDC00) + 0x0010000;
    193                     ++source;
    194                 } else if (strict) { // it's an unpaired high surrogate
    195                     --source; // return to the illegal value itself
    196                     result = sourceIllegal;
    197                     break;
    198                 }
    199             } else { // We don't have the 16 bits following the high surrogate.
    200                 --source; // return to the high surrogate
    201                 result = sourceExhausted;
    202                 break;
    203             }
    204         } else if (strict) {
    205             // UTF-16 surrogate values are illegal in UTF-32
    206             if (ch >= 0xDC00 && ch <= 0xDFFF) {
    207                 --source; // return to the illegal value itself
    208                 result = sourceIllegal;
    209                 break;
    210             }
    211         }
    212         // Figure out how many bytes the result will require
    213         if (ch < (UChar32)0x80) {
    214             bytesToWrite = 1;
    215         } else if (ch < (UChar32)0x800) {
    216             bytesToWrite = 2;
    217         } else if (ch < (UChar32)0x10000) {
    218             bytesToWrite = 3;
    219         } else if (ch < (UChar32)0x110000) {
    220             bytesToWrite = 4;
    221         } else {
    222             bytesToWrite = 3;
    223             ch = replacementCharacter;
    224         }
    225 
    226         target += bytesToWrite;
    227         if (target > targetEnd) {
    228             source = oldSource; // Back up source pointer!
    229             target -= bytesToWrite;
    230             result = targetExhausted;
    231             break;
    232         }
    233         switch (bytesToWrite) { // note: everything falls through.
    234             case 4: *--target = (char)((ch | byteMark) & byteMask); ch >>= 6; FALLTHROUGH;
    235             case 3: *--target = (char)((ch | byteMark) & byteMask); ch >>= 6; FALLTHROUGH;
    236             case 2: *--target = (char)((ch | byteMark) & byteMask); ch >>= 6; FALLTHROUGH;
    237             case 1: *--target =  (char)(ch | firstByteMark[bytesToWrite]);
    238         }
    239         target += bytesToWrite;
     92    RELEASE_ASSERT(sourceEnd - source <= std::numeric_limits<int>::max());
     93    UBool error = false;
     94    UChar* target = *targetStart;
     95    UChar32 orAllData = 0;
     96    int targetOffset = 0;
     97    for (int sourceOffset = 0; sourceOffset < sourceEnd - source; ) {
     98        UChar32 character;
     99        U8_NEXT(reinterpret_cast<const uint8_t*>(source), sourceOffset, sourceEnd - source, character);
     100        if (character < 0)
     101            return false;
     102        U16_APPEND(target, targetOffset, targetEnd - target, character, error);
     103        if (error)
     104            return false;
     105        orAllData |= character;
    240106    }
    241     *sourceStart = source;
    242     *targetStart = target;
    243     return result;
    244 }
    245 
    246 // This must be called with the length pre-determined by the first byte.
    247 // If presented with a length > 4, this returns false.  The Unicode
    248 // definition of UTF-8 goes up to 4-byte sequences.
    249 static bool isLegalUTF8(const unsigned char* source, int length)
    250 {
    251     unsigned char a;
    252     const unsigned char* srcptr = source + length;
    253     switch (length) {
    254         default: return false;
    255         // Everything else falls through when "true"...
    256         case 4: if ((a = (*--srcptr)) < 0x80 || a > 0xBF) return false; FALLTHROUGH;
    257         case 3: if ((a = (*--srcptr)) < 0x80 || a > 0xBF) return false; FALLTHROUGH;
    258         case 2: if ((a = (*--srcptr)) > 0xBF) return false;
    259 
    260         switch (*source) {
    261             // no fall-through in this inner switch
    262             case 0xE0: if (a < 0xA0) return false; break;
    263             case 0xED: if (a > 0x9F) return false; break;
    264             case 0xF0: if (a < 0x90) return false; break;
    265             case 0xF4: if (a > 0x8F) return false; break;
    266             default:   if (a < 0x80) return false;
    267         }
    268         FALLTHROUGH;
    269 
    270         case 1: if (*source >= 0x80 && *source < 0xC2) return false;
    271     }
    272     if (*source > 0xF4)
    273         return false;
     107    *targetStart = target + targetOffset;
     108    if (sourceAllASCII)
     109        *sourceAllASCII = isASCII(orAllData);
    274110    return true;
    275 }
    276 
    277 // Magic values subtracted from a buffer value during UTF8 conversion.
    278 // This table contains as many values as there might be trailing bytes
    279 // in a UTF-8 sequence.
    280 static const UChar32 offsetsFromUTF8[6] = { 0x00000000UL, 0x00003080UL, 0x000E2080UL, 0x03C82080UL, static_cast<UChar32>(0xFA082080UL), static_cast<UChar32>(0x82082080UL) };
    281 
    282 static inline UChar32 readUTF8Sequence(const char*& sequence, unsigned length)
    283 {
    284     UChar32 character = 0;
    285 
    286     // The cases all fall through.
    287     switch (length) {
    288         case 6: character += static_cast<unsigned char>(*sequence++); character <<= 6; FALLTHROUGH;
    289         case 5: character += static_cast<unsigned char>(*sequence++); character <<= 6; FALLTHROUGH;
    290         case 4: character += static_cast<unsigned char>(*sequence++); character <<= 6; FALLTHROUGH;
    291         case 3: character += static_cast<unsigned char>(*sequence++); character <<= 6; FALLTHROUGH;
    292         case 2: character += static_cast<unsigned char>(*sequence++); character <<= 6; FALLTHROUGH;
    293         case 1: character += static_cast<unsigned char>(*sequence++);
    294     }
    295 
    296     return character - offsetsFromUTF8[length - 1];
    297 }
    298 
    299 ConversionResult convertUTF8ToUTF16(
    300     const char** sourceStart, const char* sourceEnd,
    301     UChar** targetStart, UChar* targetEnd, bool* sourceAllASCII, bool strict)
    302 {
    303     ConversionResult result = conversionOK;
    304     const char* source = *sourceStart;
    305     UChar* target = *targetStart;
    306     UChar orAllData = 0;
    307     while (source < sourceEnd) {
    308         int utf8SequenceLength = inlineUTF8SequenceLength(*source);
    309         if (sourceEnd - source < utf8SequenceLength)  {
    310             result = sourceExhausted;
    311             break;
    312         }
    313         // Do this check whether lenient or strict
    314         if (!isLegalUTF8(reinterpret_cast<const unsigned char*>(source), utf8SequenceLength)) {
    315             result = sourceIllegal;
    316             break;
    317         }
    318 
    319         UChar32 character = readUTF8Sequence(source, utf8SequenceLength);
    320 
    321         if (target >= targetEnd) {
    322             source -= utf8SequenceLength; // Back up source pointer!
    323             result = targetExhausted;
    324             break;
    325         }
    326 
    327         if (U_IS_BMP(character)) {
    328             // UTF-16 surrogate values are illegal in UTF-32
    329             if (U_IS_SURROGATE(character)) {
    330                 if (strict) {
    331                     source -= utf8SequenceLength; // return to the illegal value itself
    332                     result = sourceIllegal;
    333                     break;
    334                 } else {
    335                     *target++ = replacementCharacter;
    336                     orAllData |= replacementCharacter;
    337                 }
    338             } else {
    339                 *target++ = character; // normal case
    340                 orAllData |= character;
    341             }
    342         } else if (U_IS_SUPPLEMENTARY(character)) {
    343             // target is a character in range 0xFFFF - 0x10FFFF
    344             if (target + 1 >= targetEnd) {
    345                 source -= utf8SequenceLength; // Back up source pointer!
    346                 result = targetExhausted;
    347                 break;
    348             }
    349             *target++ = U16_LEAD(character);
    350             *target++ = U16_TRAIL(character);
    351             orAllData = 0xffff;
    352         } else {
    353             if (strict) {
    354                 source -= utf8SequenceLength; // return to the start
    355                 result = sourceIllegal;
    356                 break; // Bail out; shouldn't continue
    357             } else {
    358                 *target++ = replacementCharacter;
    359                 orAllData |= replacementCharacter;
    360             }
    361         }
    362     }
    363     *sourceStart = source;
    364     *targetStart = target;
    365 
    366     if (sourceAllASCII)
    367         *sourceAllASCII = !(orAllData & ~0x7f);
    368 
    369     return result;
    370111}
    371112
    372113unsigned calculateStringHashAndLengthFromUTF8MaskingTop8Bits(const char* data, const char* dataEnd, unsigned& dataLength, unsigned& utf16Length)
    373114{
    374     if (!data)
    375         return 0;
    376 
    377115    StringHasher stringHasher;
    378     dataLength = 0;
    379116    utf16Length = 0;
    380117
    381     while (data < dataEnd || (!dataEnd && *data)) {
    382         if (isASCII(*data)) {
    383             stringHasher.addCharacter(*data++);
    384             dataLength++;
    385             utf16Length++;
    386             continue;
    387         }
    388 
    389         int utf8SequenceLength = inlineUTF8SequenceLengthNonASCII(*data);
    390         dataLength += utf8SequenceLength;
    391 
    392         if (!dataEnd) {
    393             for (int i = 1; i < utf8SequenceLength; ++i) {
    394                 if (!data[i])
    395                     return 0;
    396             }
    397         } else if (dataEnd - data < utf8SequenceLength)
     118    int inputOffset = 0;
     119    int inputLength = dataEnd - data;
     120    while (inputOffset < inputLength) {
     121        UChar32 character;
     122        U8_NEXT(reinterpret_cast<const uint8_t*>(data), inputOffset, inputLength, character);
     123        if (character < 0)
    398124            return 0;
    399125
    400         if (!isLegalUTF8(reinterpret_cast<const unsigned char*>(data), utf8SequenceLength))
    401             return 0;
    402 
    403         UChar32 character = readUTF8Sequence(data, utf8SequenceLength);
    404         ASSERT(!isASCII(character));
    405 
    406126        if (U_IS_BMP(character)) {
    407             // UTF-16 surrogate values are illegal in UTF-32
    408             if (U_IS_SURROGATE(character))
    409                 return 0;
    410             stringHasher.addCharacter(static_cast<UChar>(character)); // normal case
     127            ASSERT(!U_IS_SURROGATE(character));
     128            stringHasher.addCharacter(character);
    411129            utf16Length++;
    412         } else if (U_IS_SUPPLEMENTARY(character)) {
    413             stringHasher.addCharacters(static_cast<UChar>(U16_LEAD(character)),
    414                                        static_cast<UChar>(U16_TRAIL(character)));
     130        } else {
     131            ASSERT(U_IS_SUPPLEMENTARY(character));
     132            stringHasher.addCharacters(U16_LEAD(character), U16_TRAIL(character));
    415133            utf16Length += 2;
    416         } else
    417             return 0;
     134        }
    418135    }
    419136
     137    dataLength = inputOffset;
    420138    return stringHasher.hashWithTop8BitsMasked();
    421139}
    … …  
    424142{
    425143    while (b < bEnd) {
    426         if (isASCII(*a) || isASCII(*b)) {
    427             if (*a++ != *b++)
    428                 return false;
    429             continue;
    430         }
    431 
    432         int utf8SequenceLength = inlineUTF8SequenceLengthNonASCII(*b);
    433 
    434         if (bEnd - b < utf8SequenceLength)
     144        int offset = 0;
     145        UChar32 character;
     146        U8_NEXT(reinterpret_cast<const uint8_t*>(b), offset, bEnd - b, character);
     147        if (character < 0)
    435148            return false;
    436 
    437         if (!isLegalUTF8(reinterpret_cast<const unsigned char*>(b), utf8SequenceLength))
    438             return false;
    439 
    440         UChar32 character = readUTF8Sequence(b, utf8SequenceLength);
    441         ASSERT(!isASCII(character));
     149        b += offset;
    442150
    443151        if (U_IS_BMP(character)) {
    444             // UTF-16 surrogate values are illegal in UTF-32
    445             if (U_IS_SURROGATE(character))
    446                 return false;
     152            ASSERT(!U_IS_SURROGATE(character));
    447153            if (*a++ != character)
    448154                return false;
    449         } else if (U_IS_SUPPLEMENTARY(character)) {
     155        } else {
     156            ASSERT(U_IS_SUPPLEMENTARY(character));
    450157            if (*a++ != U16_LEAD(character))
    451158                return false;
    452159            if (*a++ != U16_TRAIL(character))
    453160                return false;
    454         } else
    455             return false;
     161        }
    456162    }
    457163

  • trunk/Source/WTF/wtf/unicode/UTF8Conversion.h

    r244827 r244828  
    11/*
    2  * Copyright (C) 2007 Apple Inc. All rights reserved.
     2 * Copyright (C) 2007-2019 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    3232namespace Unicode {
    3333
    34     // Given a first byte, gives the length of the UTF-8 sequence it begins.
    35     // Returns 0 for bytes that are not legal starts of UTF-8 sequences.
    36     // Only allows sequences of up to 4 bytes, since that works for all Unicode characters (U-00000000 to U-0010FFFF).
    37     WTF_EXPORT_PRIVATE int UTF8SequenceLength(char);
     34enum ConversionResult {
     35    ConversionOK, // conversion successful
     36    SourceExhausted, // partial character in source, but hit end
     37    TargetExhausted, // insufficient room in target for conversion
     38    SourceIllegal // source sequence is illegal/malformed
     39};
    3840
    39     // Takes a null-terminated C-style string with a UTF-8 sequence in it and converts it to a character.
    40     // Only allows Unicode characters (U-00000000 to U-0010FFFF).
    41     // Returns -1 if the sequence is not valid (including presence of extra bytes).
    42     WTF_EXPORT_PRIVATE int decodeUTF8Sequence(const char*);
     41// Conversion functions are strict, except for convertUTF16ToUTF8, which takes
     42// "strict" argument. When strict, both illegal sequences and unpaired surrogates
     43// will cause an error. When not, illegal sequences and unpaired surrogates are
     44// converted to the replacement character, except for an unpaired lead surrogate
     45// at the end of the source, which will instead cause a SourceExhausted error.
    4346
    44     typedef enum {
    45             conversionOK,       // conversion successful
    46             sourceExhausted,    // partial character in source, but hit end
    47             targetExhausted,    // insuff. room in target for conversion
    48             sourceIllegal       // source sequence is illegal/malformed
    49     } ConversionResult;
     47WTF_EXPORT_PRIVATE bool convertUTF8ToUTF16(const char* sourceStart, const char* sourceEnd, UChar** targetStart, UChar* targetEnd, bool* isSourceAllASCII = nullptr);
     48WTF_EXPORT_PRIVATE bool convertLatin1ToUTF8(const LChar** sourceStart, const LChar* sourceEnd, char** targetStart, char* targetEnd);
     49WTF_EXPORT_PRIVATE ConversionResult convertUTF16ToUTF8(const UChar** sourceStart, const UChar* sourceEnd, char** targetStart, char* targetEnd, bool strict = true);
    5050
    51     // These conversion functions take a "strict" argument. When this
    52     // flag is set to strict, both irregular sequences and isolated surrogates
    53     // will cause an error.  When the flag is set to lenient, both irregular
    54     // sequences and isolated surrogates are converted.
    55     //
    56     // Whether the flag is strict or lenient, all illegal sequences will cause
    57     // an error return. This includes sequences such as: <F4 90 80 80>, <C0 80>,
    58     // or <A0> in UTF-8, and values above 0x10FFFF in UTF-32. Conformant code
    59     // must check for illegal sequences.
    60     //
    61     // When the flag is set to lenient, characters over 0x10FFFF are converted
    62     // to the replacement character; otherwise (when the flag is set to strict)
    63     // they constitute an error.
     51WTF_EXPORT_PRIVATE unsigned calculateStringHashAndLengthFromUTF8MaskingTop8Bits(const char* data, const char* dataEnd, unsigned& dataLength, unsigned& utf16Length);
    6452
    65     WTF_EXPORT_PRIVATE ConversionResult convertUTF8ToUTF16(
    66                     const char** sourceStart, const char* sourceEnd,
    67                     UChar** targetStart, UChar* targetEnd, bool* isSourceAllASCII = 0, bool strict = true);
    68 
    69     WTF_EXPORT_PRIVATE ConversionResult convertLatin1ToUTF8(
    70                     const LChar** sourceStart, const LChar* sourceEnd,
    71                     char** targetStart, char* targetEnd);
    72 
    73     WTF_EXPORT_PRIVATE ConversionResult convertUTF16ToUTF8(
    74                     const UChar** sourceStart, const UChar* sourceEnd,
    75                     char** targetStart, char* targetEnd, bool strict = true);
    76 
    77     WTF_EXPORT_PRIVATE unsigned calculateStringHashAndLengthFromUTF8MaskingTop8Bits(const char* data, const char* dataEnd, unsigned& dataLength, unsigned& utf16Length);
    78 
    79     // The caller of these functions already knows that the lengths are the same, so we omit an end argument for UTF-16 and Latin-1.
    80     bool equalUTF16WithUTF8(const UChar* stringInUTF16, const char* stringInUTF8, const char* stringInUTF8End);
    81     bool equalLatin1WithUTF8(const LChar* stringInLatin1, const char* stringInUTF8, const char* stringInUTF8End);
     53// Callers of these functions must check that the lengths are the same; accordingly we omit an end argument for UTF-16 and Latin-1.
     54bool equalUTF16WithUTF8(const UChar* stringInUTF16, const char* stringInUTF8, const char* stringInUTF8End);
     55bool equalLatin1WithUTF8(const LChar* stringInLatin1, const char* stringInUTF8, const char* stringInUTF8End);
    8256
    8357} // namespace Unicode

  • trunk/Source/WebCore/ChangeLog

    r244827 r244828  
     12019-05-01  Darin Adler  <darin@apple.com>
     2
     3        WebKit has too much of its own UTF-8 code and should rely more on ICU's UTF-8 support
     4        https://bugs.webkit.org/show_bug.cgi?id=195535
     5
     6        Reviewed by Alexey Proskuryakov.
     7
     8        * platform/SharedBuffer.cpp:
     9        (WebCore::utf8Buffer): Removed unnecessary "strict" argument to convertUTF16ToUTF8 since
     10        that is the default behavior. Also updated for changes to return values.
     11
     12        * xml/XSLTProcessorLibxslt.cpp:
     13        (WebCore::writeToStringBuilder): Removed unnecessary use of StringBuffer for a temporary
     14        buffer for characters. Rewrote to use U8_NEXT and U16_APPEND directly.
     15
     16        * xml/parser/XMLDocumentParserLibxml2.cpp:
     17        (WebCore::convertUTF16EntityToUTF8): Updated for changes to CompletionResult.
     18
    1192019-05-01  Shawn Roberts  <sroberts@apple.com>
    220

  • trunk/Source/WebCore/ChangeLog-2010-12-06

    r244826 r244828  
    9519195191        https://bugs.webkit.org/show_bug.cgi?id=42768
    9519295192
    95193         Normalize path separators for the script API by convering \'s to /'s.
     95193        Normalize path separators for the script API by converting \'s to /'s.
    9519495194
    9519595195        Covered by input-file-directory-upload.html

  • trunk/Source/WebCore/ChangeLog-2013-04-24

    r244826 r244828  
    4251142511        * platform/graphics/chromium/GraphicsLayerChromium.cpp:
    4251242512        (WebCore::transformToSkMatrix44):
    42513           Utility for convering a WebCore::TransformationMatrix to an SkMatrix44.
     42513          Utility for converting a WebCore::TransformationMatrix to an SkMatrix44.
    4251442514          Will move to a more common location once it gets more callers.
    4251542515        (WebCore::GraphicsLayerChromium::updateTransform):

  • trunk/Source/WebCore/platform/SharedBuffer.cpp

    r244827 r244828  
    335335    // Convert to runs of 8-bit characters.
    336336    char* p = buffer.data();
    337     WTF::Unicode::ConversionResult result;
    338337    if (length) {
    339338        if (string.is8Bit()) {
    340339            const LChar* d = string.characters8();
    341             result = WTF::Unicode::convertLatin1ToUTF8(&d, d + length, &p, p + buffer.size());
     340            if (!WTF::Unicode::convertLatin1ToUTF8(&d, d + length, &p, p + buffer.size()))
     341                return nullptr;
    342342        } else {
    343343            const UChar* d = string.characters16();
    344             result = WTF::Unicode::convertUTF16ToUTF8(&d, d + length, &p, p + buffer.size(), true);
     344            if (WTF::Unicode::convertUTF16ToUTF8(&d, d + length, &p, p + buffer.size()) != WTF::Unicode::ConversionOK)
     345                return nullptr;
    345346        }
    346         if (result != WTF::Unicode::conversionOK)
    347             return nullptr;
    348347    }
    349348

  • trunk/Source/WebCore/xml/XSLTProcessorLibxslt.cpp

    r244827 r244828  
    4949#include <libxslt/xsltutils.h>
    5050#include <wtf/Assertions.h>
    51 #include <wtf/text/StringBuffer.h>
    52 #include <wtf/unicode/UTF8Conversion.h>
    5351
    5452#if OS(DARWIN) && !PLATFORM(GTK)
    … …  
    160158}
    161159
    162 static int writeToStringBuilder(void* context, const char* buffer, int len)
     160static int writeToStringBuilder(void* context, const char* buffer, int length)
    163161{
    164162    StringBuilder& resultOutput = *static_cast<StringBuilder*>(context);
    165163
    166     if (!len)
    167         return 0;
    168 
    169     StringBuffer<UChar> stringBuffer(len);
    170     UChar* bufferUChar = stringBuffer.characters();
    171     UChar* bufferUCharEnd = bufferUChar + len;
    172 
    173     const char* stringCurrent = buffer;
    174     WTF::Unicode::ConversionResult result = WTF::Unicode::convertUTF8ToUTF16(&stringCurrent, buffer + len, &bufferUChar, bufferUCharEnd);
    175     if (result != WTF::Unicode::conversionOK && result != WTF::Unicode::sourceExhausted) {
    176         ASSERT_NOT_REACHED();
    177         return -1;
    178     }
    179 
    180     int utf16Length = bufferUChar - stringBuffer.characters();
    181     resultOutput.append(stringBuffer.characters(), utf16Length);
    182     return stringCurrent - buffer;
     164    // FIXME: Consider ways to make this more efficient by moving it into a
     165    // StringBuilder::appendUTF8 function, and then optimizing to not need a
     166    // Vector<UChar> and possibly optimize cases that can produce 8-bit Latin-1
     167    // strings, but that would need to be sophisticated about not processing
     168    // trailing incomplete sequences and communicating that to the caller.
     169
     170    Vector<UChar> outputBuffer(length);
     171
     172    UBool error = false;
     173    int inputOffset = 0;
     174    int outputOffset = 0;
     175    while (inputOffset < length) {
     176        UChar32 character;
     177        int nextInputOffset = inputOffset;
     178        U8_NEXT(reinterpret_cast<const uint8_t*>(buffer), nextInputOffset, length, character);
     179        if (character < 0) {
     180            if (nextInputOffset == length)
     181                break;
     182            ASSERT_NOT_REACHED();
     183            return -1;
     184        }
     185        inputOffset = nextInputOffset;
     186        U16_APPEND(outputBuffer.data(), outputOffset, length, character, error);
     187        if (error) {
     188            ASSERT_NOT_REACHED();
     189            return -1;
     190        }
     191    }
     192
     193    resultOutput.append(outputBuffer.data(), outputOffset);
     194    return inputOffset;
    183195}
    184196

  • trunk/Source/WebCore/xml/parser/XMLDocumentParserLibxml2.cpp

    r244827 r244828  
    11551155    const char* originalTarget = target;
    11561156    auto conversionResult = WTF::Unicode::convertUTF16ToUTF8(&utf16Entity, utf16Entity + numberOfCodeUnits, &target, target + targetSize);
    1157     if (conversionResult != WTF::Unicode::conversionOK)
     1157    if (conversionResult != WTF::Unicode::ConversionOK)
    11581158        return 0;
    11591159

  • trunk/Source/WebKit/ChangeLog

    r244827 r244828  
     12019-05-01  Darin Adler  <darin@apple.com>
     2
     3        WebKit has too much of its own UTF-8 code and should rely more on ICU's UTF-8 support
     4        https://bugs.webkit.org/show_bug.cgi?id=195535
     5
     6        Reviewed by Alexey Proskuryakov.
     7
     8        * Shared/API/APIString.h: Removed uneeded includes and also switched to #pragma once.
     9
     10        * Shared/API/c/WKString.cpp: Moved include of UTF8Conversion.h here.
     11        (WKStringGetUTF8CStringImpl): Updated for changes to return values.
     12
    1132019-05-01  Shawn Roberts  <sroberts@apple.com>
    214

  • trunk/Source/WebKit/Shared/API/APIString.h

    r244827 r244828  
    2424 */
    2525
    26 #ifndef APIString_h
    27 #define APIString_h
     26#pragma once
    2827
    2928#include "APIObject.h"
    30 #include <wtf/Ref.h>
    3129#include <wtf/text/StringView.h>
    32 #include <wtf/text/WTFString.h>
    33 #include <wtf/unicode/UTF8Conversion.h>
    3430
    3531namespace API {
    … …  
    7672
    7773} // namespace WebKit
    78 
    79 #endif // APIString_h

  • trunk/Source/WebKit/Shared/API/c/WKString.cpp

    r244827 r244828  
    3131#include <JavaScriptCore/InitializeThreading.h>
    3232#include <JavaScriptCore/OpaqueJSString.h>
     33#include <wtf/unicode/UTF8Conversion.h>
    3334
    3435WKTypeID WKStringGetTypeID()
    … …  
    7980
    8081    char* p = buffer;
    81     WTF::Unicode::ConversionResult result;
    8282
    8383    if (stringView.is8Bit()) {
    8484        const LChar* characters = stringView.characters8();
    85         result = WTF::Unicode::convertLatin1ToUTF8(&characters, characters + stringView.length(), &p, p + bufferSize - 1);
     85        if (!WTF::Unicode::convertLatin1ToUTF8(&characters, characters + stringView.length(), &p, p + bufferSize - 1))
     86            return 0;
    8687    } else {
    8788        const UChar* characters = stringView.characters16();
    88         result = WTF::Unicode::convertUTF16ToUTF8(&characters, characters + stringView.length(), &p, p + bufferSize - 1, strict);
     89        auto result = WTF::Unicode::convertUTF16ToUTF8(&characters, characters + stringView.length(), &p, p + bufferSize - 1, strict);
     90        if (result != WTF::Unicode::ConversionOK && result != WTF::Unicode::TargetExhausted)
     91            return 0;
    8992    }
    90 
    91     if (result != WTF::Unicode::conversionOK && result != WTF::Unicode::targetExhausted)
    92         return 0;
    9393
    9494    *p++ = '\0';
Note: See TracChangeset for help on using the changeset viewer.