⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 243049 in webkit


Ignore:
Timestamp:
Mar 16, 2019, 7:20:52 PM (7 years ago)
Author:
Darin Adler
Message:

Improve normalization code, including moving from unorm.h to unorm2.h
​https://bugs.webkit.org/show_bug.cgi?id=195330

Reviewed by Michael Catanzaro.

Source/JavaScriptCore:

  • runtime/JSString.h: Move StringViewWithUnderlyingString to StringView.h.
  • runtime/StringPrototype.cpp: Include unorm2.h instead of unorm.h.

(JSC::normalizer): Added. Function to create normalizer object given
enumeration value indicating which is selected. Simplified because we
know the function will not fail and so we don't need error handling code.
(JSC::normalize): Changed this function to take a JSString* so we can
optimize the case where no normalization is needed. Added an early exit
if the string is stored as 8-bit and another if the string is already
normalized, using unorm2_isNormalized. Changed error handling to only
check cases that can actually fail in practice. Also did other small
optimizations like passing VM rather than ExecState.
(JSC::stringProtoFuncNormalize): Used smaller enumeration names that are
identical to the names used in the API and normalization parlance rather
than longer ones that expand the acronyms. Updated to pass JSString* to
the normalize function, so we can optimize 8-bit and already-normalized
cases, rather than callling the expensive String::upconvertedCharacters
function. Use throwVMRangeError.

Source/WebCore:

  • editing/TextIterator.cpp: Include unorm2.h.

(WebCore::normalizeCharacters): Rewrote to use unorm2_normalize rather than
unorm_normalize, but left the logic otherwise the same.

  • platform/graphics/SurrogatePairAwareTextIterator.cpp: Include unorm2.h.

(WebCore::SurrogatePairAwareTextIterator::normalizeVoicingMarks):
Use unorm2_composePair instead of unorm_normalize.

  • platform/graphics/cairo/FontCairoHarfbuzzNG.cpp:

(characterSequenceIsEmoji): Changed to use existing SurrogatePairAwareTextIterator.
(FontCascade::fontForCombiningCharacterSequence): Use normalizedNFC instead of
calling unorm2_normalize directly.

  • WebCore/platform/graphics/freetype/SimpleFontDataFreeType.cpp:

Removed unneeded include of <unicode/normlzr.h>.

  • platform/text/TextEncoding.cpp:

(WebCore::TextEncoding::encode const): Use normalizedNFC instead of the
code that was here. The normalizedNFC function is better in multiple ways,
but primarily it handles 8-bit strings and other already-normalized
strings much more efficiently.

Source/WTF:

  • wtf/URLHelpers.cpp: Removed unneeded include of unorm.h since the

normalization code is now in StringView.cpp.
(WTF::URLHelpers::escapeUnsafeCharacters): Renamed from
createStringWithEscapedUnsafeCharacters since it now only creates
a new string if one is needed. Use unsigned for string lengths, since
that's what WTF::String uses, not size_t. Added a first loop so that
we can return the string unmodified if no lookalike characters are
found. Removed unnecessary round trip from UTF-16 and then back in
the case where the character is not a lookalike.
(WTF::URLHelpers::toNormalizationFormC): Deleted. Moved this logic
into the WTF::normalizedNFC function in StringView.cpp.
(WTF::URLHelpers::userVisibleURL): Call escapeUnsafeCharacters and
normalizedNFC. The normalizedNFC function is better in multiple ways,
but primarily it handles 8-bit strings and other already-normalized
strings much more efficiently.

  • wtf/text/StringView.cpp:

(WTF::normalizedNFC): Added. This has two overloads. One is for when
we already have a String, and want to re-use it if no normalization
is needed, and another is when we only have a StringView, and may need
to allocate a String to hold the result. Includes a fast special case
for 8-bit and already-normalized strings, and uses the same strategy
that JSC::normalize was already using: calls unorm2_normalize twice,
first just to determine the length.

  • wtf/text/StringView.h: Added normalizedNFC, which can be called with

either a StringView or a String. Also moved StringViewWithUnderlyingString
here from JSString.h, here for use as the return value of normalizedNFC;
it is used for a similar purpose in the JavaScriptCore rope implementation.
Also removed an inaccurate comment.

Location:
trunk/Source
Files:
13 edited

Legend:

Unmodified
Added
Removed

  • trunk/Source/JavaScriptCore/ChangeLog

    r243032 r243049  
     12019-03-16  Darin Adler  <darin@apple.com>
     2
     3        Improve normalization code, including moving from unorm.h to unorm2.h
     4        https://bugs.webkit.org/show_bug.cgi?id=195330
     5
     6        Reviewed by Michael Catanzaro.
     7
     8        * runtime/JSString.h: Move StringViewWithUnderlyingString to StringView.h.
     9
     10        * runtime/StringPrototype.cpp: Include unorm2.h instead of unorm.h.
     11        (JSC::normalizer): Added. Function to create normalizer object given
     12        enumeration value indicating which is selected. Simplified because we
     13        know the function will not fail and so we don't need error handling code.
     14        (JSC::normalize): Changed this function to take a JSString* so we can
     15        optimize the case where no normalization is needed. Added an early exit
     16        if the string is stored as 8-bit and another if the string is already
     17        normalized, using unorm2_isNormalized. Changed error handling to only
     18        check cases that can actually fail in practice. Also did other small
     19        optimizations like passing VM rather than ExecState.
     20        (JSC::stringProtoFuncNormalize): Used smaller enumeration names that are
     21        identical to the names used in the API and normalization parlance rather
     22        than longer ones that expand the acronyms. Updated to pass JSString* to
     23        the normalize function, so we can optimize 8-bit and already-normalized
     24        cases, rather than callling the expensive String::upconvertedCharacters
     25        function. Use throwVMRangeError.
     26
    1272019-03-15  Mark Lam  <mark.lam@apple.com>
    228

  • trunk/Source/JavaScriptCore/runtime/JSString.h

    r242519 r243049  
    6767bool isJSString(JSValue);
    6868JSString* asString(JSValue);
    69 
    70 struct StringViewWithUnderlyingString {
    71     StringView view;
    72     String underlyingString;
    73 };
    74 
    7569
    7670// In 64bit architecture, JSString and JSRopeString have the following memory layout to make sizeof(JSString) == 16 and sizeof(JSRopeString) == 32.

  • trunk/Source/JavaScriptCore/runtime/StringPrototype.cpp

    r242252 r243049  
    5050#include <algorithm>
    5151#include <unicode/uconfig.h>
    52 #include <unicode/unorm.h>
     52#include <unicode/unorm2.h>
    5353#include <unicode/ustring.h>
    5454#include <wtf/ASCIICType.h>
    … …  
    18061806}
    18071807
    1808 enum class NormalizationForm {
    1809     CanonicalComposition,
    1810     CanonicalDecomposition,
    1811     CompatibilityComposition,
    1812     CompatibilityDecomposition
    1813 };
    1814 
    1815 static JSValue normalize(ExecState* exec, const UChar* source, size_t sourceLength, NormalizationForm form)
    1816 {
    1817     VM& vm = exec->vm();
    1818     auto scope = DECLARE_THROW_SCOPE(vm);
    1819 
     1808enum class NormalizationForm { NFC, NFD, NFKC, NFKD };
     1809
     1810static constexpr bool normalizationAffects8Bit(NormalizationForm form)
     1811{
     1812    switch (form) {
     1813    case NormalizationForm::NFC:
     1814        return false;
     1815    case NormalizationForm::NFD:
     1816        return true;
     1817    case NormalizationForm::NFKC:
     1818        return false;
     1819    case NormalizationForm::NFKD:
     1820        return true;
     1821    }
     1822    ASSERT_NOT_REACHED();
     1823    return true;
     1824}
     1825
     1826static const UNormalizer2* normalizer(NormalizationForm form)
     1827{
    18201828    UErrorCode status = U_ZERO_ERROR;
    1821     // unorm2_get*Instance() documentation says: "Returns an unmodifiable singleton instance. Do not delete it."
    18221829    const UNormalizer2* normalizer = nullptr;
    18231830    switch (form) {
    1824     case NormalizationForm::CanonicalComposition:
     1831    case NormalizationForm::NFC:
    18251832        normalizer = unorm2_getNFCInstance(&status);
    18261833        break;
    1827     case NormalizationForm::CanonicalDecomposition:
     1834    case NormalizationForm::NFD:
    18281835        normalizer = unorm2_getNFDInstance(&status);
    18291836        break;
    1830     case NormalizationForm::CompatibilityComposition:
     1837    case NormalizationForm::NFKC:
    18311838        normalizer = unorm2_getNFKCInstance(&status);
    18321839        break;
    1833     case NormalizationForm::CompatibilityDecomposition:
     1840    case NormalizationForm::NFKD:
    18341841        normalizer = unorm2_getNFKDInstance(&status);
    18351842        break;
    18361843    }
    1837 
    1838     if (!normalizer || U_FAILURE(status))
    1839         return throwTypeError(exec, scope);
    1840 
    1841     int32_t normalizedStringLength = unorm2_normalize(normalizer, source, sourceLength, nullptr, 0, &status);
    1842 
    1843     if (U_FAILURE(status) && status != U_BUFFER_OVERFLOW_ERROR) {
    1844         // The behavior is not specified when normalize fails.
    1845         // Now we throw a type error since it seems that the contents of the string are invalid.
    1846         return throwTypeError(exec, scope);
    1847     }
    1848 
    1849     UChar* buffer = nullptr;
    1850     auto impl = StringImpl::tryCreateUninitialized(normalizedStringLength, buffer);
    1851     if (!impl)
     1844    ASSERT(normalizer);
     1845    ASSERT(U_SUCCESS(status));
     1846    return normalizer;
     1847}
     1848
     1849static JSValue normalize(ExecState* exec, JSString* string, NormalizationForm form)
     1850{
     1851    VM& vm = exec->vm();
     1852    auto scope = DECLARE_THROW_SCOPE(vm);
     1853
     1854    auto viewWithString = string->viewWithUnderlyingString(exec);
     1855    RETURN_IF_EXCEPTION(scope, { });
     1856
     1857    StringView view = viewWithString.view;
     1858    if (view.is8Bit() && (!normalizationAffects8Bit(form) || charactersAreAllASCII(view.characters8(), view.length())))
     1859        RELEASE_AND_RETURN(scope, string);
     1860
     1861    const UNormalizer2* normalizer = JSC::normalizer(form);
     1862
     1863    // Since ICU does not offer functions that can perform normalization or check for
     1864    // normalization with input that is Latin-1, we need to upconvert to UTF-16 at this point.
     1865    auto characters = view.upconvertedCharacters();
     1866
     1867    UErrorCode status = U_ZERO_ERROR;
     1868    UBool isNormalized = unorm2_isNormalized(normalizer, characters, view.length(), &status);
     1869    ASSERT(U_SUCCESS(status));
     1870    if (isNormalized)
     1871        RELEASE_AND_RETURN(scope, string);
     1872
     1873    int32_t normalizedStringLength = unorm2_normalize(normalizer, characters, view.length(), nullptr, 0, &status);
     1874    ASSERT(status == U_BUFFER_OVERFLOW_ERROR);
     1875
     1876    UChar* buffer;
     1877    auto result = StringImpl::tryCreateUninitialized(normalizedStringLength, buffer);
     1878    if (!result)
    18521879        return throwOutOfMemoryError(exec, scope);
    18531880
    18541881    status = U_ZERO_ERROR;
    1855     unorm2_normalize(normalizer, source, sourceLength, buffer, normalizedStringLength, &status);
    1856     if (U_FAILURE(status))
    1857         return throwTypeError(exec, scope);
    1858 
    1859     RELEASE_AND_RETURN(scope, jsString(exec, WTFMove(impl)));
     1882    unorm2_normalize(normalizer, characters, view.length(), buffer, normalizedStringLength, &status);
     1883    ASSERT(U_SUCCESS(status));
     1884
     1885    RELEASE_AND_RETURN(scope, jsString(&vm, WTFMove(result)));
    18601886}
    18611887
    … …  
    18681894    if (!checkObjectCoercible(thisValue))
    18691895        return throwVMTypeError(exec, scope);
    1870     auto viewWithString = thisValue.toString(exec)->viewWithUnderlyingString(exec);
    1871     RETURN_IF_EXCEPTION(scope, encodedJSValue());
    1872     StringView view = viewWithString.view;
    1873 
    1874     NormalizationForm form = NormalizationForm::CanonicalComposition;
    1875     // Verify that the argument is provided and is not undefined.
    1876     if (!exec->argument(0).isUndefined()) {
    1877         String formString = exec->uncheckedArgument(0).toWTFString(exec);
    1878         RETURN_IF_EXCEPTION(scope, encodedJSValue());
     1896    JSString* string = thisValue.toString(exec);
     1897    RETURN_IF_EXCEPTION(scope, { });
     1898
     1899    auto form = NormalizationForm::NFC;
     1900    JSValue formValue = exec->argument(0);
     1901    if (!formValue.isUndefined()) {
     1902        String formString = formValue.toWTFString(exec);
     1903        RETURN_IF_EXCEPTION(scope, { });
    18791904
    18801905        if (formString == "NFC")
    1881             form = NormalizationForm::CanonicalComposition;
     1906            form = NormalizationForm::NFC;
    18821907        else if (formString == "NFD")
    1883             form = NormalizationForm::CanonicalDecomposition;
     1908            form = NormalizationForm::NFD;
    18841909        else if (formString == "NFKC")
    1885             form = NormalizationForm::CompatibilityComposition;
     1910            form = NormalizationForm::NFKC;
    18861911        else if (formString == "NFKD")
    1887             form = NormalizationForm::CompatibilityDecomposition;
     1912            form = NormalizationForm::NFKD;
    18881913        else
    1889             return throwVMError(exec, scope, createRangeError(exec, "argument does not match any normalization form"_s));
    1890     }
    1891 
    1892     RELEASE_AND_RETURN(scope, JSValue::encode(normalize(exec, view.upconvertedCharacters(), view.length(), form)));
     1914            return throwVMRangeError(exec, scope, "argument does not match any normalization form"_s);
     1915    }
     1916
     1917    RELEASE_AND_RETURN(scope, JSValue::encode(normalize(exec, string, form)));
    18931918}
    18941919

  • trunk/Source/WTF/ChangeLog

    r243040 r243049  
     12019-03-16  Darin Adler  <darin@apple.com>
     2
     3        Improve normalization code, including moving from unorm.h to unorm2.h
     4        https://bugs.webkit.org/show_bug.cgi?id=195330
     5
     6        Reviewed by Michael Catanzaro.
     7
     8        * wtf/URLHelpers.cpp: Removed unneeded include of unorm.h since the
     9        normalization code is now in StringView.cpp.
     10        (WTF::URLHelpers::escapeUnsafeCharacters): Renamed from
     11        createStringWithEscapedUnsafeCharacters since it now only creates
     12        a new string if one is needed. Use unsigned for string lengths, since
     13        that's what WTF::String uses, not size_t. Added a first loop so that
     14        we can return the string unmodified if no lookalike characters are
     15        found. Removed unnecessary round trip from UTF-16 and then back in
     16        the case where the character is not a lookalike.
     17        (WTF::URLHelpers::toNormalizationFormC): Deleted. Moved this logic
     18        into the WTF::normalizedNFC function in StringView.cpp.
     19        (WTF::URLHelpers::userVisibleURL): Call escapeUnsafeCharacters and
     20        normalizedNFC. The normalizedNFC function is better in multiple ways,
     21        but primarily it handles 8-bit strings and other already-normalized
     22        strings much more efficiently.
     23
     24        * wtf/text/StringView.cpp:
     25        (WTF::normalizedNFC): Added. This has two overloads. One is for when
     26        we already have a String, and want to re-use it if no normalization
     27        is needed, and another is when we only have a StringView, and may need
     28        to allocate a String to hold the result. Includes a fast special case
     29        for 8-bit and already-normalized strings, and uses the same strategy
     30        that JSC::normalize was already using: calls unorm2_normalize twice,
     31        first just to determine the length.
     32
     33        * wtf/text/StringView.h: Added normalizedNFC, which can be called with
     34        either a StringView or a String. Also moved StringViewWithUnderlyingString
     35        here from JSString.h, here for use as the return value of normalizedNFC;
     36        it is used for a similar purpose in the JavaScriptCore rope implementation.
     37        Also removed an inaccurate comment.
     38
    1392019-03-16  Diego Pino Garcia  <dpino@igalia.com>
    240

  • trunk/Source/WTF/wtf/URLHelpers.cpp

    r242776 r243049  
    11/*
    2  * Copyright (C) 2005, 2007, 2014 Apple Inc. All rights reserved.
     2 * Copyright (C) 2005-2019 Apple Inc. All rights reserved.
    33 * Copyright (C) 2018 Igalia S.L.
    44 *
    … …  
    3434#include <mutex>
    3535#include <unicode/uidna.h>
    36 #include <unicode/unorm.h>
    3736#include <unicode/uscript.h>
    3837#include <wtf/Optional.h>
    … …  
    738737}
    739738
    740 static String createStringWithEscapedUnsafeCharacters(const String& sourceBuffer)
    741 {
     739static String escapeUnsafeCharacters(const String& sourceBuffer)
     740{
     741    unsigned length = sourceBuffer.length();
     742
     743    Optional<UChar32> previousCodePoint;
     744
     745    unsigned i;
     746    for (i = 0; i < length; ) {
     747        UChar32 c = sourceBuffer.characterStartingAt(i);
     748        if (isLookalikeCharacter(previousCodePoint, sourceBuffer.characterStartingAt(i)))
     749            break;
     750        previousCodePoint = c;
     751        i += U16_LENGTH(c);
     752    }
     753
     754    if (i == length)
     755        return sourceBuffer;
     756
    742757    Vector<UChar, urlBytesBufferLength> outBuffer;
    743758
    744     const size_t length = sourceBuffer.length();
    745 
    746     Optional<UChar32> previousCodePoint;
    747     size_t i = 0;
    748     while (i < length) {
     759    outBuffer.grow(i);
     760    if (sourceBuffer.is8Bit())
     761        StringImpl::copyCharacters(outBuffer.data(), sourceBuffer.characters8(), i);
     762    else
     763        StringImpl::copyCharacters(outBuffer.data(), sourceBuffer.characters16(), i);
     764
     765    for (; i < length; ) {
    749766        UChar32 c = sourceBuffer.characterStartingAt(i);
    750 
     767        unsigned characterLength = U16_LENGTH(c);
    751768        if (isLookalikeCharacter(previousCodePoint, c)) {
    752769            uint8_t utf8Buffer[4];
    … …  
    755772            U8_APPEND(utf8Buffer, offset, 4, c, failure)
    756773            ASSERT(!failure);
    757            
     774
    758775            for (size_t j = 0; j < offset; ++j) {
    759776                outBuffer.append('%');
    … …  
    762779            }
    763780        } else {
    764             UChar utf16Buffer[2];
    765             size_t offset = 0;
    766             UBool failure = false;
    767             U16_APPEND(utf16Buffer, offset, 2, c, failure)
    768             ASSERT(!failure);
    769             for (size_t j = 0; j < offset; ++j)
    770                 outBuffer.append(utf16Buffer[j]);
     781            for (unsigned j = 0; j < characterLength; ++j)
     782                outBuffer.append(sourceBuffer[i + j]);
    771783        }
    772784        previousCodePoint = c;
    773         i += U16_LENGTH(c);
    774     }
     785        i += characterLength;
     786    }
     787
    775788    return String::adopt(WTFMove(outBuffer));
    776 }
    777 
    778 static String toNormalizationFormC(const String& string)
    779 {
    780     Vector<UChar> sourceBuffer = string.charactersWithNullTermination();
    781     ASSERT(sourceBuffer.last() == '\0');
    782     sourceBuffer.removeLast();
    783 
    784     UErrorCode uerror = U_ZERO_ERROR;
    785     const UNormalizer2* normalizer = unorm2_getNFCInstance(&uerror);
    786     if (U_FAILURE(uerror))
    787         return { };
    788 
    789     UNormalizationCheckResult checkResult = unorm2_quickCheck(normalizer, sourceBuffer.data(), sourceBuffer.size(), &uerror);
    790     if (U_FAILURE(uerror))
    791         return { };
    792 
    793     // No need to normalize if already normalized.
    794     if (checkResult == UNORM_YES)
    795         return string;
    796 
    797     Vector<UChar, urlBytesBufferLength> normalizedCharacters(sourceBuffer.size());
    798     auto normalizedLength = unorm2_normalize(normalizer, sourceBuffer.data(), sourceBuffer.size(), normalizedCharacters.data(), normalizedCharacters.size(), &uerror);
    799     if (uerror == U_BUFFER_OVERFLOW_ERROR) {
    800         uerror = U_ZERO_ERROR;
    801         normalizedCharacters.resize(normalizedLength);
    802         normalizedLength = unorm2_normalize(normalizer, sourceBuffer.data(), sourceBuffer.size(), normalizedCharacters.data(), normalizedLength, &uerror);
    803     }
    804     if (U_FAILURE(uerror))
    805         return { };
    806 
    807     return String(normalizedCharacters.data(), normalizedLength);
    808789}
    809790
    … …  
    892873    }
    893874
    894     auto normalized = toNormalizationFormC(result);
    895     return createStringWithEscapedUnsafeCharacters(normalized);
     875    return escapeUnsafeCharacters(normalizedNFC(result));
    896876}
    897877

  • trunk/Source/WTF/wtf/text/StringView.cpp

    r241183 r243049  
    11/*
    22
    3 Copyright (C) 2014-2017 Apple Inc. All rights reserved.
     3Copyright (C) 2014-2019 Apple Inc. All rights reserved.
    44
    55Redistribution and use in source and binary forms, with or without
    … …  
    3030#include <mutex>
    3131#include <unicode/ubrk.h>
     32#include <unicode/unorm2.h>
    3233#include <wtf/HashMap.h>
    3334#include <wtf/Lock.h>
    … …  
    239240        return convertASCIICase<ASCIICase::Upper>(static_cast<const LChar*>(m_characters), m_length);
    240241    return convertASCIICase<ASCIICase::Upper>(static_cast<const UChar*>(m_characters), m_length);
     242}
     243
     244StringViewWithUnderlyingString normalizedNFC(StringView string)
     245{
     246    // Latin-1 characters are unaffected by normalization.
     247    if (string.is8Bit())
     248        return { string, { } };
     249
     250    UErrorCode status = U_ZERO_ERROR;
     251    const UNormalizer2* normalizer = unorm2_getNFCInstance(&status);
     252    ASSERT(U_SUCCESS(status));
     253
     254    // No need to normalize if already normalized.
     255    UBool checkResult = unorm2_isNormalized(normalizer, string.characters16(), string.length(), &status);
     256    if (checkResult)
     257        return { string, { } };
     258
     259    unsigned normalizedLength = unorm2_normalize(normalizer, string.characters16(), string.length(), nullptr, 0, &status);
     260    ASSERT(status == U_BUFFER_OVERFLOW_ERROR);
     261
     262    UChar* characters;
     263    String result = String::createUninitialized(normalizedLength, characters);
     264
     265    status = U_ZERO_ERROR;
     266    unorm2_normalize(normalizer, string.characters16(), string.length(), characters, normalizedLength, &status);
     267    ASSERT(U_SUCCESS(status));
     268
     269    StringView view { result };
     270    return { view, WTFMove(result) };
     271}
     272
     273String normalizedNFC(const String& string)
     274{
     275    auto result = normalizedNFC(StringView { string });
     276    if (result.underlyingString.isNull())
     277        return string;
     278    return result.underlyingString;
    241279}
    242280

  • trunk/Source/WTF/wtf/text/StringView.h

    r242308 r243049  
    11/*
    2  * Copyright (C) 2014-2017 Apple Inc. All rights reserved.
     2 * Copyright (C) 2014-2019 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    212212inline bool operator!=(const char*a, StringView b) { return !equal(b, a); }
    213213
     214struct StringViewWithUnderlyingString;
     215
     216// This returns a StringView of the normalized result, and a String that is either
     217// null, if the input was already normalized, or contains the normalized result
     218// and needs to be kept around so the StringView remains valid. Typically the
     219// easiest way to use it correctly is to put it into a local and use the StringView.
     220WTF_EXPORT_PRIVATE StringViewWithUnderlyingString normalizedNFC(StringView);
     221
     222WTF_EXPORT_PRIVATE String normalizedNFC(const String&);
     223
    214224}
    215225
    … …  
    219229namespace WTF {
    220230
     231struct StringViewWithUnderlyingString {
     232    StringView view;
     233    String underlyingString;
     234};
     235
    221236inline StringView::StringView()
    222237{
    223     // FIXME: It's peculiar that null strings are 16-bit and empty strings return 8-bit (according to the is8Bit function).
    224238}
    225239
    226240#if CHECK_STRINGVIEW_LIFETIME
     241
    227242inline StringView::~StringView()
    228243{
    … …  
    281296    return *this;
    282297}
     298
    283299#endif // CHECK_STRINGVIEW_LIFETIME
    284300
    … …  
    9971013using WTF::equal;
    9981014using WTF::StringView;
     1015using WTF::StringViewWithUnderlyingString;

  • trunk/Source/WebCore/ChangeLog

    r243048 r243049  
     12019-03-16  Darin Adler  <darin@apple.com>
     2
     3        Improve normalization code, including moving from unorm.h to unorm2.h
     4        https://bugs.webkit.org/show_bug.cgi?id=195330
     5
     6        Reviewed by Michael Catanzaro.
     7
     8        * editing/TextIterator.cpp: Include unorm2.h.
     9        (WebCore::normalizeCharacters): Rewrote to use unorm2_normalize rather than
     10        unorm_normalize, but left the logic otherwise the same.
     11
     12        * platform/graphics/SurrogatePairAwareTextIterator.cpp: Include unorm2.h.
     13        (WebCore::SurrogatePairAwareTextIterator::normalizeVoicingMarks):
     14        Use unorm2_composePair instead of unorm_normalize.
     15
     16        * platform/graphics/cairo/FontCairoHarfbuzzNG.cpp:
     17        (characterSequenceIsEmoji): Changed to use existing SurrogatePairAwareTextIterator.
     18        (FontCascade::fontForCombiningCharacterSequence): Use normalizedNFC instead of
     19        calling unorm2_normalize directly.
     20
     21        * WebCore/platform/graphics/freetype/SimpleFontDataFreeType.cpp:
     22        Removed unneeded include of <unicode/normlzr.h>.
     23
     24        * platform/text/TextEncoding.cpp:
     25        (WebCore::TextEncoding::encode const): Use normalizedNFC instead of the
     26        code that was here. The normalizedNFC function is better in multiple ways,
     27        but primarily it handles 8-bit strings and other already-normalized
     28        strings much more efficiently.
     29
    1302019-03-16  Jer Noble  <jer.noble@apple.com>
    231

  • trunk/Source/WebCore/editing/TextIterator.cpp

    r238693 r243049  
    11/*
    2  * Copyright (C) 2004-2017 Apple Inc. All rights reserved.
     2 * Copyright (C) 2004-2019 Apple Inc. All rights reserved.
    33 * Copyright (C) 2005 Alexey Proskuryakov.
    44 *
    … …  
    6161#include "VisiblePosition.h"
    6262#include "VisibleUnits.h"
     63#include <unicode/unorm2.h>
    6364#include <wtf/Function.h>
    6465#include <wtf/text/CString.h>
    … …  
    7273#endif
    7374
    74 
    7575namespace WebCore {
     76
    7677using namespace WTF::Unicode;
    77 
    7878using namespace HTMLNames;
    7979
    … …  
    20152015}
    20162016
    2017 ALLOW_DEPRECATED_DECLARATIONS_BEGIN
    2018 // NOTE: ICU's unorm_normalize function is deprecated.
    2019 
    20202017static void normalizeCharacters(const UChar* characters, unsigned length, Vector<UChar>& buffer)
    20212018{
    2022     ASSERT(length);
     2019    UErrorCode status = U_ZERO_ERROR;
     2020    const UNormalizer2* normalizer = unorm2_getNFCInstance(&status);
     2021    ASSERT(U_SUCCESS(status));
    20232022
    20242023    buffer.resize(length);
    20252024
    2026     UErrorCode status = U_ZERO_ERROR;
    2027     size_t bufferSize = unorm_normalize(characters, length, UNORM_NFC, 0, buffer.data(), length, &status);
    2028     ASSERT(status == U_ZERO_ERROR || status == U_STRING_NOT_TERMINATED_WARNING || status == U_BUFFER_OVERFLOW_ERROR);
    2029     ASSERT(bufferSize);
    2030 
    2031     buffer.resize(bufferSize);
    2032 
    2033     if (status == U_ZERO_ERROR || status == U_STRING_NOT_TERMINATED_WARNING)
     2025    auto normalizedLength = unorm2_normalize(normalizer, characters, length, buffer.data(), length, &status);
     2026    ASSERT(U_SUCCESS(status) || status == U_BUFFER_OVERFLOW_ERROR);
     2027
     2028    buffer.resize(normalizedLength);
     2029
     2030    if (U_SUCCESS(status))
    20342031        return;
    20352032
    20362033    status = U_ZERO_ERROR;
    2037     unorm_normalize(characters, length, UNORM_NFC, 0, buffer.data(), bufferSize, &status);
    2038     ASSERT(status == U_STRING_NOT_TERMINATED_WARNING);
    2039 }
    2040 
    2041 ALLOW_DEPRECATED_DECLARATIONS_END
     2034    unorm2_normalize(normalizer, characters, length, buffer.data(), length, &status);
     2035    ASSERT(U_SUCCESS(status));
     2036}
    20422037
    20432038static bool isNonLatin1Separator(UChar32 character)

  • trunk/Source/WebCore/platform/graphics/SurrogatePairAwareTextIterator.cpp

    r235935 r243049  
    11/*
    2  * Copyright (C) 2003, 2006, 2008, 2009, 2010, 2011 Apple Inc. All rights reserved.
     2 * Copyright (C) 2003-2019 Apple Inc. All rights reserved.
    33 * Copyright (C) 2008 Holger Hans Peter Freyther
    44 * Copyright (C) Research In Motion Limited 2011. All rights reserved.
    … …  
    2424#include "SurrogatePairAwareTextIterator.h"
    2525
    26 #include <unicode/unorm.h>
     26#include <unicode/unorm2.h>
    2727
    2828namespace WebCore {
    … …  
    7070}
    7171
    72 ALLOW_DEPRECATED_DECLARATIONS_BEGIN
    73 // NOTE: ICU's unorm_normalize function is deprecated.
    74 
    7572UChar32 SurrogatePairAwareTextIterator::normalizeVoicingMarks()
    7673{
    7774    // According to http://www.unicode.org/Public/UNIDATA/UCD.html#Canonical_Combining_Class_Values
    78     static const uint8_t hiraganaKatakanaVoicingMarksCombiningClass = 8;
     75    static constexpr uint8_t hiraganaKatakanaVoicingMarksCombiningClass = 8;
    7976
    8077    if (m_currentIndex + 1 >= m_endIndex)
    … …  
    8279
    8380    if (u_getCombiningClass(m_characters[1]) == hiraganaKatakanaVoicingMarksCombiningClass) {
    84         // Normalize into composed form using 3.2 rules.
    85         UChar normalizedCharacters[2] = { 0, 0 };
    86         UErrorCode uStatus = U_ZERO_ERROR; 
    87         int32_t resultLength = unorm_normalize(m_characters, 2, UNORM_NFC, UNORM_UNICODE_3_2, &normalizedCharacters[0], 2, &uStatus);
    88         if (resultLength == 1 && !uStatus)
    89             return normalizedCharacters[0];
     81        UErrorCode status = U_ZERO_ERROR;
     82        const UNormalizer2* normalizer = unorm2_getNFCInstance(&status);
     83        ASSERT(U_SUCCESS(status));
     84        auto composedCharacter = unorm2_composePair(normalizer, m_characters[0], m_characters[1]);
     85        if (composedCharacter > 0)
     86            return composedCharacter;
    9087    }
    9188
    … …  
    9390}
    9491
    95 ALLOW_DEPRECATED_DECLARATIONS_END
    96 
    9792}

  • trunk/Source/WebCore/platform/graphics/cairo/FontCairoHarfbuzzNG.cpp

    r241402 r243049  
    3333#include "FontCache.h"
    3434#include "SurrogatePairAwareTextIterator.h"
    35 #include <unicode/normlzr.h>
    3635
    3736namespace WebCore {
    … …  
    4746}
    4847
    49 static bool characterSequenceIsEmoji(const Vector<UChar, 4>& normalizedCharacters, int32_t normalizedLength)
     48static bool characterSequenceIsEmoji(SurrogatePairAwareTextIterator& iterator, UChar32 firstCharacter, unsigned firstClusterLength)
    5049{
    51     UChar32 character;
    52     unsigned clusterLength = 0;
    53     SurrogatePairAwareTextIterator iterator(normalizedCharacters.data(), 0, normalizedLength, normalizedLength);
     50    UChar32 character = firstCharacter;
     51    unsigned clusterLength = firstClusterLength;
    5452    if (!iterator.consume(character, clusterLength))
    5553        return false;
    … …  
    10199}
    102100
    103 const Font* FontCascade::fontForCombiningCharacterSequence(const UChar* characters, size_t length) const
     101const Font* FontCascade::fontForCombiningCharacterSequence(const UChar* originalCharacters, size_t originalLength) const
    104102{
    105     UErrorCode error = U_ZERO_ERROR;
    106     Vector<UChar, 4> normalizedCharacters(length);
    107     const auto* normalizer = unorm2_getNFCInstance(&error);
    108     if (U_FAILURE(error))
    109         return nullptr;
    110     int32_t normalizedLength = unorm2_normalize(normalizer, characters, length, normalizedCharacters.data(), length, &error);
    111     if (U_FAILURE(error)) {
    112         if (error != U_BUFFER_OVERFLOW_ERROR)
    113             return nullptr;
     103    auto normalizedString = normalizedNFC(StringView { originalCharacters, static_cast<unsigned>(originalLength) });
    114104
    115         error = U_ZERO_ERROR;
    116         normalizedCharacters.resize(normalizedLength);
    117         normalizedLength = unorm2_normalize(normalizer, characters, length, normalizedCharacters.data(), normalizedLength, &error);
    118         if (U_FAILURE(error))
    119             return nullptr;
    120     }
     105    // Code below relies on normalizedNFC never narrowing a 16-bit input string into an 8-bit output string.
     106    // At the time of this writing, the function never does this, but in theory a future version could, and
     107    // we would then need to add code paths here for the simpler 8-bit case.
     108    auto characters = normalizedString.view.characters16();
     109    auto length = normalizedString.view.length();
    121110
    122111    UChar32 character;
    123112    unsigned clusterLength = 0;
    124     SurrogatePairAwareTextIterator iterator(normalizedCharacters.data(), 0, normalizedLength, normalizedLength);
     113    SurrogatePairAwareTextIterator iterator(characters, 0, length, length);
    125114    if (!iterator.consume(character, clusterLength))
    126115        return nullptr;
    127116
    128     bool isEmoji = characterSequenceIsEmoji(normalizedCharacters, normalizedLength);
     117    bool isEmoji = characterSequenceIsEmoji(iterator, character, clusterLength);
    129118
    130119    const Font* baseFont = glyphDataForCharacter(character, false, NormalVariant).font;
    131120    if (baseFont
    132         && (static_cast<int32_t>(clusterLength) == normalizedLength || baseFont->canRenderCombiningCharacterSequence(characters, length))
     121        && (clusterLength == length || baseFont->canRenderCombiningCharacterSequence(characters, length))
    133122        && (!isEmoji || baseFont->platformData().isColorBitmapFont()))
    134123        return baseFont;

  • trunk/Source/WebCore/platform/graphics/freetype/SimpleFontDataFreeType.cpp

    r239822 r243049  
    5151#include FT_TRUETYPE_TABLES_H
    5252#include FT_TRUETYPE_TAGS_H
    53 #include <unicode/normlzr.h>
    5453#include <wtf/MathExtras.h>
    5554

  • trunk/Source/WebCore/platform/text/TextEncoding.cpp

    r236674 r243049  
    11/*
    2  * Copyright (C) 2004-2017 Apple Inc. All rights reserved.
     2 * Copyright (C) 2004-2019 Apple Inc. All rights reserved.
    33 * Copyright (C) 2006 Alexey Proskuryakov <ap@nypop.com>
    44 * Copyright (C) 2007-2009 Torch Mobile, Inc.
    … …  
    3232#include "TextCodec.h"
    3333#include "TextEncodingRegistry.h"
    34 #include <unicode/unorm.h>
    3534#include <wtf/NeverDestroyed.h>
    3635#include <wtf/StdLibExtras.h>
    37 #include <wtf/text/CString.h>
    3836#include <wtf/text/StringView.h>
    3937
    … …  
    7270}
    7371
    74 ALLOW_DEPRECATED_DECLARATIONS_BEGIN
    75 // NOTE: ICU's unorm_quickCheck and unorm_normalize functions are deprecated.
    76 
    77 Vector<uint8_t> TextEncoding::encode(StringView text, UnencodableHandling handling) const
     72Vector<uint8_t> TextEncoding::encode(StringView string, UnencodableHandling handling) const
    7873{
    79     if (!m_name || text.isEmpty())
     74    if (!m_name || string.isEmpty())
    8075        return { };
    81 
    82     // FIXME: Consider adding a fast case for ASCII.
    8376
    8477    // FIXME: What's the right place to do normalization?
    8578    // It's a little strange to do it inside the encode function.
    8679    // Perhaps normalization should be an explicit step done before calling encode.
    87 
    88     auto upconvertedCharacters = text.upconvertedCharacters();
    89 
    90     const UChar* source = upconvertedCharacters;
    91     unsigned sourceLength = text.length();
    92 
    93     Vector<UChar> normalizedCharacters;
    94 
    95     UErrorCode err = U_ZERO_ERROR;
    96     if (unorm_quickCheck(source, sourceLength, UNORM_NFC, &err) != UNORM_YES) {
    97         // First try using the length of the original string, since normalization to NFC rarely increases length.
    98         normalizedCharacters.grow(sourceLength);
    99         int32_t normalizedLength = unorm_normalize(source, sourceLength, UNORM_NFC, 0, normalizedCharacters.data(), sourceLength, &err);
    100         if (err == U_BUFFER_OVERFLOW_ERROR) {
    101             err = U_ZERO_ERROR;
    102             normalizedCharacters.resize(normalizedLength);
    103             normalizedLength = unorm_normalize(source, sourceLength, UNORM_NFC, 0, normalizedCharacters.data(), normalizedLength, &err);
    104         }
    105         ASSERT(U_SUCCESS(err));
    106 
    107         source = normalizedCharacters.data();
    108         sourceLength = normalizedLength;
    109     }
    110 
    111     return newTextCodec(*this)->encode(StringView { source, sourceLength }, handling);
     80    auto normalizedString = normalizedNFC(string);
     81    return newTextCodec(*this)->encode(normalizedString.view, handling);
    11282}
    113 
    114 ALLOW_DEPRECATED_DECLARATIONS_END
    11583
    11684const char* TextEncoding::domName() const
Note: See TracChangeset for help on using the changeset viewer.