⚠ Archived content — this site is no longer maintained.   Current WebKit documentation is at docs.webkit.org.

Changeset 176473 in webkit


Ignore:
Timestamp:
Nov 21, 2014, 2:33:17 PM (12 years ago)
Author:
mmaxfield@apple.com
Message:

Source/WebCore:
Add support to -webkit-line-break property for CSS3 Text line-break property values and semantics.
​https://bugs.webkit.org/show_bug.cgi?id=89235

Reviewed by Eric Seidel and Dave Hyatt.

This patch adds semantic support for the CSS3 line-break property (qua -webkit-line-break),
and enables testing on (apple) mac ports. Follow on patches will enable these tests on
other ports as they are incrementally verified.

See also wiki documentation at:
[1] ​http://trac.webkit.org/wiki/LineBreaking
[2] ​http://trac.webkit.org/wiki/LineBreakingCSS3Mapping

Tests: css3/line-break/line-break-auto-centered.html

css3/line-break/line-break-auto-half-kana.html
css3/line-break/line-break-auto-hyphens.html
css3/line-break/line-break-auto-inseparables.html
css3/line-break/line-break-auto-iteration-marks.html
css3/line-break/line-break-auto-postfixes.html
css3/line-break/line-break-auto-prefixes.html
css3/line-break/line-break-auto-sound-marks.html
css3/line-break/line-break-loose-centered.html
css3/line-break/line-break-loose-half-kana.html
css3/line-break/line-break-loose-hyphens.html
css3/line-break/line-break-loose-inseparables.html
css3/line-break/line-break-loose-iteration-marks.html
css3/line-break/line-break-loose-postfixes.html
css3/line-break/line-break-loose-prefixes.html
css3/line-break/line-break-loose-sound-marks.html
css3/line-break/line-break-normal-centered.html
css3/line-break/line-break-normal-half-kana.html
css3/line-break/line-break-normal-hyphens.html
css3/line-break/line-break-normal-inseparables.html
css3/line-break/line-break-normal-iteration-marks.html
css3/line-break/line-break-normal-postfixes.html
css3/line-break/line-break-normal-prefixes.html
css3/line-break/line-break-normal-sound-marks.html
css3/line-break/line-break-strict-centered.html
css3/line-break/line-break-strict-half-kana.html
css3/line-break/line-break-strict-hyphens.html
css3/line-break/line-break-strict-inseparables.html
css3/line-break/line-break-strict-iteration-marks.html
css3/line-break/line-break-strict-postfixes.html
css3/line-break/line-break-strict-prefixes.html
css3/line-break/line-break-strict-sound-marks.html

These tests were previously added in ​http://trac.webkit.org/changeset/143378, but skipped
in generic TestExpectations. In this patch, they are marked as Pass for the (apple) mac ports.

  • platform/text/LineBreakIteratorPoolICU.h:

(WebCore::LineBreakIteratorPool::makeLocaleWithBreakKeyword):
Add static function to construct ICU locale argument (also used as pool key) with additional
break keyword.
(LineBreakIteratorPool):
(WebCore::LineBreakIteratorPool::take):
(WebCore::LineBreakIteratorPool::put):
Remove direct dependency from ICU library (and types), moving that dependency into
new {open,close}LineBreakIterator() functions (defined in TextBreakIteratorICU.cpp).
Update to take line break mode into account.
Create (and cache) different break iterators depending on line break mode (in addition to locale),
which entails expanding pool entry key format to optionally append "@break=" +
"loose"|"normal"|"strict" keyword to locale string.

  • platform/text/TextBreakIterator.h:

(WebCore::LazyLineBreakIterator::LazyLineBreakIterator):
(WebCore::LazyLineBreakIterator::isLooseCJKMode):
(WebCore::LazyLineBreakIterator::get):
(WebCore::LazyLineBreakIterator::reset):
(LazyLineBreakIterator):
Define LineBreakIteratorMode enumeration for use in TextBreakIterator et al.
Add state member to indicate line break mode.

  • platform/text/TextBreakIteratorICU.cpp:

(WebCore::acquireLineBreakIterator):
Use new line break mode when making iterator from pool.
Handle change of return type of LineBreakIteratorPool::take() to non-ICU type,
i.e., TextBreakIterator* instead of ICU's UBreakIterator*.
(WebCore::releaseLineBreakIterator):
Handle change of parameter type of LineBreakIteratorPool::put() to non-ICU type,
i.e., TextBreakIterator* instead of ICU's UBreakIterator*.
(WebCore):
(WebCore::isCJKLocale):
New functions for determining if CJK rules apply.
(WebCore::openLineBreakIterator):
New function for abstracting opening of ICU style line break iterator. This is now
used in LineBreakIteratorPoolICU.h rather than having direct ICU API dependency there.
This function also takes into account the line break mode.

Note that this function only calls ubrk_openRules() when the author has opted-in via
using the -webkit-line-break CSS property. Eventually, we would like to be able to
customize the rules that ICU's line breaking algorithm uses (especially for CJK text);
however, ubrk_openRules() currently parses its input string to create a DFA and is
therefore very slow. In fact, it's so slow that increasing our cache size in
LineBreakIteratorPool doesn't actually help enough. Also note that the default value
for the line-break CSS property is 'auto'.
(WebCore::closeLineBreakIterator):
(WebCore::mapLineIteratorModeToRules):
New function for abstracting closing of ICU style line break iterator. This is now
used in LineBreakIteratorPoolICU.h rather than having direct ICU API dependency there.

  • rendering/RenderBlockLineLayout.cpp:

(WebCore::RenderBlock::LineBreaker::nextSegmentBreak):
Pass line break iterator mode flag when reseting LazyLineBreakIterator.
Add looseMode local variable to prevent need for computing under isBreakable().

  • rendering/RenderText.cpp:

(WebCore::mapLineBreakToIteratorMode):
Add implementation for mapLineBreakToIteratorMode(), used by both RenderText::computePreferredLogicalWidths
and RenderBlock::LineBreaker::nextLineBreak.
(WebCore):
(WebCore::RenderText::computePreferredLogicalWidths):
Ensure (lazy line) breakIterator is initialized for line break mode.
Ensure isBreakable() is passed loose mode flag to match behavior in RenderBlock::LineBreaker::nextLineBreak.

  • rendering/RenderText.h:

(WebCore):
Add declaration for mapLineBreakToIteratorMode(), used by both RenderText::computePreferredLogicalWidths
and RenderBlock::LineBreaker::nextLineBreak.

  • rendering/break_lines.cpp:

(WebCore):
Introduce (local) enum NBSPBehavior for expanding template on nextBreakablePosition.
(WebCore::isBreakableSpace):
Add externally specified loose mode parameter to prevent need to invoke line break iterator
accessor method on each invocation. Use new loose mode flavors off NBP functions.
(WebCore::needsLineBreakIterator):
Use enum NBSP behavior template parameter rather than boolean.
(WebCore::nextBreakablePositionNonLoosely):
Extend name to distinguish from loose flavor of this function.
(WebCore::nextBreakablePositionLoosely):
Add loose flavor of NBP invoked only when loose mode applies, in which case ASCII shortcut
table cannot be used.
(WebCore::nextBreakablePosition):
(WebCore::nextBreakablePositionIgnoringNBSP):
Use (renamed) non-loose flavor of NBP.
(WebCore::nextBreakablePositionLoose):
(WebCore::nextBreakablePositionIgnoringNBSPLoose):
Introduce loose flavor of NBP template expansions.

  • rendering/break_lines.h:

(WebCore):
(WebCore::isBreakable):
Add externally specified loose mode parameter to prevent need to invoke line break iterator
accessor method on each invocation.

LayoutTests:
CSS3: line-break property support
​https://bugs.webkit.org/show_bug.cgi?id=89235

Reviewed by Eric Seidel and Dave Hyatt.

  • platform/mac/TestExpectations: Mark css3/line-break tests as passing.
Location:
trunk
Files:
13 edited

Legend:

Unmodified
Added
Removed

  • trunk/LayoutTests/ChangeLog

    r176470 r176473  
     12014-11-21  Glenn Adams  <glenn@skynav.com> and Myles C. Maxfield  <mmaxfield@apple.com>
     2
     3        CSS3: line-break property support
     4        https://bugs.webkit.org/show_bug.cgi?id=89235
     5
     6        Reviewed by Eric Seidel and Dave Hyatt.
     7
     8        * platform/mac/TestExpectations: Mark css3/line-break tests as passing.
     9
    1102014-11-21  Zalan Bujtas  <zalan@apple.com>
    211

  • trunk/LayoutTests/platform/mac/TestExpectations

    r176416 r176473  
    14301430
    14311431webkit.org/b/82980 http/tests/navigation/back-twice-without-commit.html [ Timeout ]
     1432
     1433# Verified passing, so override generic skip
     1434webkit.org/b/89235 css3/line-break [ Pass ]
     1435webkit.org/b/138115 css3/line-break/line-break-auto-hyphens.html [ ImageOnlyFailure ]
     1436webkit.org/b/138115 css3/line-break/line-break-auto-sound-marks.html [ ImageOnlyFailure ]

  • trunk/Source/WebCore/ChangeLog

    r176472 r176473  
     12014-11-21  Glenn Adams  <glenn@skynav.com> and Myles C. Maxfield  <mmaxfield@apple.com>
     2
     3        Add support to -webkit-line-break property for CSS3 Text line-break property values and semantics.
     4        https://bugs.webkit.org/show_bug.cgi?id=89235
     5
     6        Reviewed by Eric Seidel and Dave Hyatt.
     7
     8        This patch adds semantic support for the CSS3 line-break property (qua -webkit-line-break),
     9        and enables testing on (apple) mac ports. Follow on patches will enable these tests on
     10        other ports as they are incrementally verified.
     11
     12        See also wiki documentation at:
     13        [1] http://trac.webkit.org/wiki/LineBreaking
     14        [2] http://trac.webkit.org/wiki/LineBreakingCSS3Mapping
     15
     16        Tests: css3/line-break/line-break-auto-centered.html
     17               css3/line-break/line-break-auto-half-kana.html
     18               css3/line-break/line-break-auto-hyphens.html
     19               css3/line-break/line-break-auto-inseparables.html
     20               css3/line-break/line-break-auto-iteration-marks.html
     21               css3/line-break/line-break-auto-postfixes.html
     22               css3/line-break/line-break-auto-prefixes.html
     23               css3/line-break/line-break-auto-sound-marks.html
     24               css3/line-break/line-break-loose-centered.html
     25               css3/line-break/line-break-loose-half-kana.html
     26               css3/line-break/line-break-loose-hyphens.html
     27               css3/line-break/line-break-loose-inseparables.html
     28               css3/line-break/line-break-loose-iteration-marks.html
     29               css3/line-break/line-break-loose-postfixes.html
     30               css3/line-break/line-break-loose-prefixes.html
     31               css3/line-break/line-break-loose-sound-marks.html
     32               css3/line-break/line-break-normal-centered.html
     33               css3/line-break/line-break-normal-half-kana.html
     34               css3/line-break/line-break-normal-hyphens.html
     35               css3/line-break/line-break-normal-inseparables.html
     36               css3/line-break/line-break-normal-iteration-marks.html
     37               css3/line-break/line-break-normal-postfixes.html
     38               css3/line-break/line-break-normal-prefixes.html
     39               css3/line-break/line-break-normal-sound-marks.html
     40               css3/line-break/line-break-strict-centered.html
     41               css3/line-break/line-break-strict-half-kana.html
     42               css3/line-break/line-break-strict-hyphens.html
     43               css3/line-break/line-break-strict-inseparables.html
     44               css3/line-break/line-break-strict-iteration-marks.html
     45               css3/line-break/line-break-strict-postfixes.html
     46               css3/line-break/line-break-strict-prefixes.html
     47               css3/line-break/line-break-strict-sound-marks.html
     48
     49        These tests were previously added in http://trac.webkit.org/changeset/143378, but skipped
     50        in generic TestExpectations. In this patch, they are marked as Pass for the (apple) mac ports.
     51
     52        * platform/text/LineBreakIteratorPoolICU.h:
     53        (WebCore::LineBreakIteratorPool::makeLocaleWithBreakKeyword):
     54        Add static function to construct ICU locale argument (also used as pool key) with additional
     55        break keyword.
     56        (LineBreakIteratorPool):
     57        (WebCore::LineBreakIteratorPool::take):
     58        (WebCore::LineBreakIteratorPool::put):
     59        Remove direct dependency from ICU library (and types), moving that dependency into
     60        new {open,close}LineBreakIterator() functions (defined in TextBreakIteratorICU.cpp).
     61        Update to take line break mode into account.
     62        Create (and cache) different break iterators depending on line break mode (in addition to locale),
     63        which entails expanding pool entry key format to optionally append "@break=" +
     64        "loose"|"normal"|"strict" keyword to locale string.
     65
     66        * platform/text/TextBreakIterator.h:
     67        (WebCore::LazyLineBreakIterator::LazyLineBreakIterator):
     68        (WebCore::LazyLineBreakIterator::isLooseCJKMode):
     69        (WebCore::LazyLineBreakIterator::get):
     70        (WebCore::LazyLineBreakIterator::reset):
     71        (LazyLineBreakIterator):
     72        Define LineBreakIteratorMode enumeration for use in TextBreakIterator et al.
     73        Add state member to indicate line break mode.
     74
     75        * platform/text/TextBreakIteratorICU.cpp:
     76        (WebCore::acquireLineBreakIterator):
     77        Use new line break mode when making iterator from pool.
     78        Handle change of return type of LineBreakIteratorPool::take() to non-ICU type,
     79        i.e., TextBreakIterator* instead of ICU's UBreakIterator*.
     80        (WebCore::releaseLineBreakIterator):
     81        Handle change of parameter type of LineBreakIteratorPool::put() to non-ICU type,
     82        i.e., TextBreakIterator* instead of ICU's UBreakIterator*.
     83        (WebCore):
     84        (WebCore::isCJKLocale):
     85        New functions for determining if CJK rules apply.
     86        (WebCore::openLineBreakIterator):
     87        New function for abstracting opening of ICU style line break iterator. This is now
     88        used in LineBreakIteratorPoolICU.h rather than having direct ICU API dependency there.
     89        This function also takes into account the line break mode.
     90
     91        Note that this function only calls ubrk_openRules() when the author has opted-in via
     92        using the -webkit-line-break CSS property. Eventually, we would like to be able to
     93        customize the rules that ICU's line breaking algorithm uses (especially for CJK text);
     94        however, ubrk_openRules() currently parses its input string to create a DFA and is
     95        therefore very slow. In fact, it's so slow that increasing our cache size in
     96        LineBreakIteratorPool doesn't actually help enough. Also note that the default value
     97        for the line-break CSS property is 'auto'.
     98        (WebCore::closeLineBreakIterator):
     99        (WebCore::mapLineIteratorModeToRules):
     100        New function for abstracting closing of ICU style line break iterator. This is now
     101        used in LineBreakIteratorPoolICU.h rather than having direct ICU API dependency there.
     102
     103        * rendering/RenderBlockLineLayout.cpp:
     104        (WebCore::RenderBlock::LineBreaker::nextSegmentBreak):
     105        Pass line break iterator mode flag when reseting LazyLineBreakIterator.
     106        Add looseMode local variable to prevent need for computing under isBreakable().
     107
     108        * rendering/RenderText.cpp:
     109        (WebCore::mapLineBreakToIteratorMode):
     110        Add implementation for mapLineBreakToIteratorMode(), used by both RenderText::computePreferredLogicalWidths
     111        and RenderBlock::LineBreaker::nextLineBreak.
     112        (WebCore):
     113        (WebCore::RenderText::computePreferredLogicalWidths):
     114        Ensure (lazy line) breakIterator is initialized for line break mode.
     115        Ensure isBreakable() is passed loose mode flag to match behavior in RenderBlock::LineBreaker::nextLineBreak.
     116
     117        * rendering/RenderText.h:
     118        (WebCore):
     119        Add declaration for mapLineBreakToIteratorMode(), used by both RenderText::computePreferredLogicalWidths
     120        and RenderBlock::LineBreaker::nextLineBreak.
     121
     122        * rendering/break_lines.cpp:
     123        (WebCore):
     124        Introduce (local) enum NBSPBehavior for expanding template on nextBreakablePosition.
     125        (WebCore::isBreakableSpace):
     126        Add externally specified loose mode parameter to prevent need to invoke line break iterator
     127        accessor method on each invocation. Use new loose mode flavors off NBP functions.
     128        (WebCore::needsLineBreakIterator):
     129        Use enum NBSP behavior template parameter rather than boolean.
     130        (WebCore::nextBreakablePositionNonLoosely):
     131        Extend name to distinguish from loose flavor of this function.
     132        (WebCore::nextBreakablePositionLoosely):
     133        Add loose flavor of NBP invoked only when loose mode applies, in which case ASCII shortcut
     134        table cannot be used.
     135        (WebCore::nextBreakablePosition):
     136        (WebCore::nextBreakablePositionIgnoringNBSP):
     137        Use (renamed) non-loose flavor of NBP.
     138        (WebCore::nextBreakablePositionLoose):
     139        (WebCore::nextBreakablePositionIgnoringNBSPLoose):
     140        Introduce loose flavor of NBP template expansions.
     141
     142        * rendering/break_lines.h:
     143        (WebCore):
     144        (WebCore::isBreakable):
     145        Add externally specified loose mode parameter to prevent need to invoke line break iterator
     146        accessor method on each invocation.
     147
    11482014-11-21  Anders Carlsson  <andersca@apple.com>
    2149

  • trunk/Source/WebCore/WebCore.exp.in

    r176462 r176473  
    25942594__ZN7WebCore11MemoryCache18pruneDeadResourcesEv
    25952595__ZN7WebCore11MemoryCache18pruneLiveResourcesEb
     2596__ZN7WebCore11isCJKLocaleERKN3WTF12AtomicStringE
    25962597__ZN7WebCore11isEndOfLineERKNS_15VisiblePositionE
    25972598__ZN7WebCore11prefetchDNSERKN3WTF6StringE
    … …  
    27062707__ZN7WebCore24FloatingPointEnvironment25saveMainThreadEnvironmentEv
    27072708__ZN7WebCore24FloatingPointEnvironment6sharedEv
    2708 __ZN7WebCore24acquireLineBreakIteratorEN3WTF10StringViewERKNS0_12AtomicStringEPKtj
     2709__ZN7WebCore24acquireLineBreakIteratorEN3WTF10StringViewERKNS0_12AtomicStringEPKtjNS_21LineBreakIteratorModeEb
    27092710__ZN7WebCore24charactersAroundPositionERKNS_15VisiblePositionERiS3_S3_
    27102711__ZN7WebCore24createTemporaryDirectoryEP8NSString

  • trunk/Source/WebCore/platform/text/LineBreakIteratorPoolICU.h

    r161309 r176473  
    2727#define LineBreakIteratorPoolICU_h
    2828
     29#include "TextBreakIterator.h"
    2930#include "TextBreakIteratorInternalICU.h"
    3031#include <unicode/ubrk.h>
    … …  
    3536#include <wtf/text/AtomicString.h>
    3637#include <wtf/text/CString.h>
     38#include <wtf/text/StringBuilder.h>
    3739
    3840namespace WebCore {
    … …  
    4951    static PassOwnPtr<LineBreakIteratorPool> create() { return adoptPtr(new LineBreakIteratorPool); }
    5052
    51     UBreakIterator* take(const AtomicString& locale)
     53    static String makeLocaleWithBreakKeyword(const AtomicString& locale, LineBreakIteratorMode mode)
    5254    {
    53         UBreakIterator* iterator = 0;
     55        StringBuilder localeWithKeyword;
     56        localeWithKeyword.append(locale);
     57        localeWithKeyword.appendLiteral("@break=");
     58        switch (mode) {
     59        case LineBreakIteratorModeUAX14:
     60            ASSERT_NOT_REACHED();
     61            break;
     62        case LineBreakIteratorModeUAX14Loose:
     63            localeWithKeyword.appendLiteral("loose");
     64            break;
     65        case LineBreakIteratorModeUAX14Normal:
     66            localeWithKeyword.appendLiteral("normal");
     67            break;
     68        case LineBreakIteratorModeUAX14Strict:
     69            localeWithKeyword.appendLiteral("strict");
     70            break;
     71        }
     72        return localeWithKeyword.toString();
     73    }
     74
     75    TextBreakIterator* take(const AtomicString& locale, LineBreakIteratorMode mode, bool isCJK)
     76    {
     77        AtomicString localeWithOptionalBreakKeyword;
     78        if (mode == LineBreakIteratorModeUAX14)
     79            localeWithOptionalBreakKeyword = locale;
     80        else
     81            localeWithOptionalBreakKeyword = makeLocaleWithBreakKeyword(locale, mode);
     82
     83        TextBreakIterator* iterator = 0;
    5484        for (size_t i = 0; i < m_pool.size(); ++i) {
    55             if (m_pool[i].first == locale) {
     85            if (m_pool[i].first == localeWithOptionalBreakKeyword) {
    5686                iterator = m_pool[i].second;
    5787                m_pool.remove(i);
    … …  
    6191
    6292        if (!iterator) {
    63             UErrorCode openStatus = U_ZERO_ERROR;
    64             bool localeIsEmpty = locale.isEmpty();
    65             iterator = ubrk_open(UBRK_LINE, localeIsEmpty ? currentTextBreakLocaleID() : locale.string().utf8().data(), 0, 0, &openStatus);
    66             // locale comes from a web page and it can be invalid, leading ICU
    67             // to fail, in which case we fall back to the default locale.
    68             if (!localeIsEmpty && U_FAILURE(openStatus)) {
    69                 openStatus = U_ZERO_ERROR;
    70                 iterator = ubrk_open(UBRK_LINE, currentTextBreakLocaleID(), 0, 0, &openStatus);
    71             }
    72                
    73             if (U_FAILURE(openStatus)) {
    74                 LOG_ERROR("ubrk_open failed with status %d", openStatus);
     93            iterator = openLineBreakIterator(localeWithOptionalBreakKeyword, mode, isCJK);
     94            if (!iterator)
    7595                return 0;
    76             }
    7796        }
    7897
    7998        ASSERT(!m_vendedIterators.contains(iterator));
    80         m_vendedIterators.set(iterator, locale);
     99        m_vendedIterators.set(iterator, localeWithOptionalBreakKeyword);
    81100        return iterator;
    82101    }
    83102
    84     void put(UBreakIterator* iterator)
     103    void put(TextBreakIterator* iterator)
    85104    {
    86105        ASSERT_ARG(iterator, m_vendedIterators.contains(iterator));
    87106
    88107        if (m_pool.size() == capacity) {
    89             ubrk_close(m_pool[0].second);
     108            closeLineBreakIterator(m_pool[0].second);
    90109            m_pool.remove(0);
    91110        }
    … …  
    99118    static const size_t capacity = 4;
    100119
    101     typedef std::pair<AtomicString, UBreakIterator*> Entry;
     120    typedef std::pair<AtomicString, TextBreakIterator*> Entry;
    102121    typedef Vector<Entry, capacity> Pool;
    103122    Pool m_pool;
    104     HashMap<UBreakIterator*, AtomicString> m_vendedIterators;
     123    HashMap<TextBreakIterator*, AtomicString> m_vendedIterators;
    105124
    106125    friend WTF::ThreadSpecific<LineBreakIteratorPool>::operator LineBreakIteratorPool*();

  • trunk/Source/WebCore/platform/text/TextBreakIterator.cpp

    r172513 r176473  
    257257}
    258258
    259 TextBreakIterator* acquireLineBreakIterator(StringView string, const AtomicString& locale, const UChar* priorContext, unsigned priorContextLength)
    260 {
    261     TextBreakIterator* iterator = reinterpret_cast<TextBreakIterator*>(LineBreakIteratorPool::sharedPool().take(locale));
     259TextBreakIterator* acquireLineBreakIterator(StringView string, const AtomicString& locale, const UChar* priorContext, unsigned priorContextLength, LineBreakIteratorMode mode, bool isCJK)
     260{
     261    TextBreakIterator* iterator = LineBreakIteratorPool::sharedPool().take(locale, mode, isCJK);
    262262    if (!iterator)
    263263        return nullptr;
    … …  
    270270    ASSERT_ARG(iterator, iterator);
    271271
    272     LineBreakIteratorPool::sharedPool().put(reinterpret_cast<UBreakIterator*>(iterator));
     272    LineBreakIteratorPool::sharedPool().put(iterator);
     273}
     274
     275static const char* uax14Prologue =
     276    "!!chain;"
     277    "!!LBCMNoChain;"
     278    "!!lookAheadHardBreak;";
     279
     280static const char* uax14AssignmentsBefore =
     281    // explicitly enumerate $CJ since ICU versions prior to 49 don't support :LineBreak=Conditional_Japanese_Starter:
     282    "$CJ = ["
     283#if (U_ICU_VERSION_MAJOR_NUM >= 4) && (U_ICU_VERSION_MINOR_NUM >= 9)
     284    ":LineBreak=Conditional_Japanese_Starter:"
     285#else
     286    "\\u3041\\u3043\\u3045\\u3047\\u3049\\u3063\\u3083\\u3085\\u3087\\u308E\\u3095\\u3096\\u30A1\\u30A3\\u30A5\\u30A7"
     287    "\\u30A9\\u30C3\\u30E3\\u30E5\\u30E7\\u30EE\\u30F5\\u30F6\\u30FC"
     288    "\\u31F0\\u31F1\\u31F2\\u31F3\\u31F4\\u31F5\\u31F6\\u31F7\\u31F8\\u31F9\\u31FA\\u31FB\\u31FC\\u31FD\\u31FE\\u31FF"
     289    "\\uFF67\\uFF68\\uFF69\\uFF6A\\uFF6B\\uFF6C\\uFF6D\\uFF6E\\uFF6F\\uFF70"
     290#endif
     291    "];";
     292
     293static const char* uax14AssignmentsCustomLooseCJK =
     294    "$BA_SUB = [\\u2010\\u2013];"
     295    "$EX_SUB = [\\u0021\\u003F\\uFF01\\uFF1F];"
     296    "$ID_SUB = '';"
     297    "$IN_SUB = [\\u2025\\u2026];"
     298    "$IS_SUB = [\\u003A\\u003B];"
     299    "$NS_SUB = [\\u203C\\u2047\\u2048\\u2049\\u3005\\u301C\\u303B\\u309D\\u309E\\u30A0\\u30FB\\u30FD\\u30FE\\uFF1A\\uFF1B\\uFF65];"
     300    "$PO_SUB = [\\u0025\\u00A2\\u00B0\\u2030\\u2032\\u2033\\u2103\\uFF05\\uFFE0];"
     301    "$PR_SUB = [\\u0024\\u00A3\\u00A5\\u20AC\\u2116\\uFF04\\uFFE1\\uFFE5];"
     302    "$ID_ADD = [$CJ $BA_SUB $EX_SUB $IN_SUB $IS_SUB $NS_SUB $PO_SUB $PR_SUB];"
     303    "$NS_ADD = '';";
     304
     305static const char* uax14AssignmentsCustomLooseNonCJK =
     306    "$BA_SUB = '';"
     307    "$EX_SUB = '';"
     308    "$ID_SUB = '';"
     309    "$IN_SUB = [\\u2025\\u2026];"
     310    "$IS_SUB = '';"
     311    "$NS_SUB = [\\u3005\\u303B\\u309D\\u309E\\u30FD\\u30FE];"
     312    "$PO_SUB = '';"
     313    "$PR_SUB = '';"
     314    "$ID_ADD = [$CJ $IN_SUB $NS_SUB];"
     315    "$NS_ADD = '';";
     316
     317static const char* uax14AssignmentsCustomNormalCJK =
     318    "$BA_SUB = [\\u2010\\u2013];"
     319    "$EX_SUB = '';"
     320    "$IN_SUB = '';"
     321    "$ID_SUB = '';"
     322    "$IS_SUB = '';"
     323    "$NS_SUB = [\\u301C\\u30A0];"
     324    "$PO_SUB = '';"
     325    "$PR_SUB = '';"
     326    "$ID_ADD = [$CJ $BA_SUB $NS_SUB];"
     327    "$NS_ADD = '';";
     328
     329static const char* uax14AssignmentsCustomNormalNonCJK =
     330    "$BA_SUB = '';"
     331    "$EX_SUB = '';"
     332    "$ID_SUB = '';"
     333    "$IN_SUB = '';"
     334    "$IS_SUB = '';"
     335    "$NS_SUB = '';"
     336    "$PO_SUB = '';"
     337    "$PR_SUB = '';"
     338    "$ID_ADD = [$CJ];"
     339    "$NS_ADD = '';";
     340
     341static const char* uax14AssignmentsCustomStrictCJK =
     342    "$BA_SUB = '';"
     343    "$EX_SUB = '';"
     344    "$ID_SUB = '';"
     345    "$IN_SUB = '';"
     346    "$IS_SUB = '';"
     347    "$NS_SUB = '';"
     348    "$PO_SUB = '';"
     349    "$PR_SUB = '';"
     350    "$ID_ADD = '';"
     351    "$NS_ADD = [$CJ];";
     352
     353#define uax14AssignmentsCustomStrictNonCJK      uax14AssignmentsCustomStrictCJK
     354#define uax14AssignmentsCustomDefaultCJK        uax14AssignmentsCustomNormalCJK
     355#define uax14AssignmentsCustomDefaultNonCJK     uax14AssignmentsCustomStrictNonCJK
     356
     357static const char* uax14AssignmentsAfter =
     358    "$AI = [:LineBreak = Ambiguous:];"
     359    "$AL = [:LineBreak = Alphabetic:];"
     360    "$BA = [[:LineBreak = Break_After:] - $BA_SUB];"
     361    "$BB = [:LineBreak = Break_Before:];"
     362    "$BK = [:LineBreak = Mandatory_Break:];"
     363    "$B2 = [:LineBreak = Break_Both:];"
     364    "$CB = [:LineBreak = Contingent_Break:];"
     365    "$CL = [:LineBreak = Close_Punctuation:];"
     366    "$CM = [:LineBreak = Combining_Mark:];"
     367    "$CP = [:LineBreak = Close_Parenthesis:];"
     368    "$CR = [:LineBreak = Carriage_Return:];"
     369    "$EX = [[:LineBreak = Exclamation:] - $EX_SUB];"
     370    "$GL = [:LineBreak = Glue:];"
     371#if (U_ICU_VERSION_MAJOR_NUM >= 4) && (U_ICU_VERSION_MINOR_NUM >= 9)
     372    "$HL = [:LineBreak = Hebrew_Letter:];"
     373#else
     374    "$HL = [[:Hebrew:] & [:Letter:]];"
     375#endif
     376    "$HY = [:LineBreak = Hyphen:];"
     377    "$H2 = [:LineBreak = H2:];"
     378    "$H3 = [:LineBreak = H3:];"
     379    "$ID = [[[[:LineBreak = Ideographic:] - $CJ] $ID_ADD] - $ID_SUB];"
     380    "$IN = [[:LineBreak = Inseparable:] - $IN_SUB];"
     381    "$IS = [[:LineBreak = Infix_Numeric:] - $IS_SUB];"
     382    "$JL = [:LineBreak = JL:];"
     383    "$JV = [:LineBreak = JV:];"
     384    "$JT = [:LineBreak = JT:];"
     385    "$LF = [:LineBreak = Line_Feed:];"
     386    "$NL = [:LineBreak = Next_Line:];"
     387    "$NS = [[[[:LineBreak = Nonstarter:] - $CJ] $NS_ADD] - $NS_SUB];"
     388    "$NU = [:LineBreak = Numeric:];"
     389    "$OP = [:LineBreak = Open_Punctuation:];"
     390    "$PO = [[:LineBreak = Postfix_Numeric:] - $PO_SUB];"
     391    "$PR = [[:LineBreak = Prefix_Numeric:] - $PR_SUB];"
     392    "$QU = [:LineBreak = Quotation:];"
     393    "$SA = [:LineBreak = Complex_Context:];"
     394    "$SG = [:LineBreak = Surrogate:];"
     395    "$SP = [:LineBreak = Space:];"
     396    "$SY = [:LineBreak = Break_Symbols:];"
     397    "$WJ = [:LineBreak = Word_Joiner:];"
     398    "$XX = [:LineBreak = Unknown:];"
     399    "$ZW = [:LineBreak = ZWSpace:];"
     400    "$dictionary = [:LineBreak = Complex_Context:];"
     401    "$ALPlus = [$AL $AI $SA $SG $XX];"
     402    "$ALcm = $ALPlus $CM*;"
     403    "$BAcm = $BA $CM*;"
     404    "$BBcm = $BB $CM*;"
     405    "$B2cm = $B2 $CM*;"
     406    "$CLcm = $CL $CM*;"
     407    "$CPcm = $CP $CM*;"
     408    "$EXcm = $EX $CM*;"
     409    "$GLcm = $GL $CM*;"
     410    "$HLcm = $HL $CM*;"
     411    "$HYcm = $HY $CM*;"
     412    "$H2cm = $H2 $CM*;"
     413    "$H3cm = $H3 $CM*;"
     414    "$IDcm = $ID $CM*;"
     415    "$INcm = $IN $CM*;"
     416    "$IScm = $IS $CM*;"
     417    "$JLcm = $JL $CM*;"
     418    "$JVcm = $JV $CM*;"
     419    "$JTcm = $JT $CM*;"
     420    "$NScm = $NS $CM*;"
     421    "$NUcm = $NU $CM*;"
     422    "$OPcm = $OP $CM*;"
     423    "$POcm = $PO $CM*;"
     424    "$PRcm = $PR $CM*;"
     425    "$QUcm = $QU $CM*;"
     426    "$SYcm = $SY $CM*;"
     427    "$WJcm = $WJ $CM*;";
     428
     429static const char* uax14Forward =
     430    "!!forward;"
     431    "$CAN_CM = [^$SP $BK $CR $LF $NL $ZW $CM];"
     432    "$CANT_CM = [$SP $BK $CR $LF $NL $ZW $CM];"
     433    "$AL_FOLLOW_NOCM = [$BK $CR $LF $NL $ZW $SP];"
     434    "$AL_FOLLOW_CM = [$CL $CP $EX $HL $IS $SY $WJ $GL $OP $QU $BA $HY $NS $IN $NU $ALPlus];"
     435    "$AL_FOLLOW = [$AL_FOLLOW_NOCM $AL_FOLLOW_CM];"
     436    "$LB4Breaks = [$BK $CR $LF $NL];"
     437    "$LB4NonBreaks = [^$BK $CR $LF $NL];"
     438    "$LB8Breaks = [$LB4Breaks $ZW];"
     439    "$LB8NonBreaks = [[$LB4NonBreaks] - [$ZW]];"
     440    "$LB18NonBreaks = [$LB8NonBreaks - [$SP]];"
     441    "$LB18Breaks = [$LB8Breaks $SP];"
     442    "$LB20NonBreaks = [$LB18NonBreaks - $CB];"
     443    "$ALPlus $CM+;"
     444    "$BA $CM+;"
     445    "$BB $CM+;"
     446    "$B2 $CM+;"
     447    "$CL $CM+;"
     448    "$CP $CM+;"
     449    "$EX $CM+;"
     450    "$GL $CM+;"
     451    "$HL $CM+;"
     452    "$HY $CM+;"
     453    "$H2 $CM+;"
     454    "$H3 $CM+;"
     455    "$ID $CM+;"
     456    "$IN $CM+;"
     457    "$IS $CM+;"
     458    "$JL $CM+;"
     459    "$JV $CM+;"
     460    "$JT $CM+;"
     461    "$NS $CM+;"
     462    "$NU $CM+;"
     463    "$OP $CM+;"
     464    "$PO $CM+;"
     465    "$PR $CM+;"
     466    "$QU $CM+;"
     467    "$SY $CM+;"
     468    "$WJ $CM+;"
     469    "$CR $LF {100};"
     470    "$LB4NonBreaks? $LB4Breaks {100};"
     471    "$CAN_CM $CM* $LB4Breaks {100};"
     472    "$CM+ $LB4Breaks {100};"
     473    "$LB4NonBreaks [$SP $ZW];"
     474    "$CAN_CM $CM* [$SP $ZW];"
     475    "$CM+ [$SP $ZW];"
     476    "$CAN_CM $CM+;"
     477    "$CM+;"
     478    "$CAN_CM $CM* $WJcm;"
     479    "$LB8NonBreaks $WJcm;"
     480    "$CM+ $WJcm;"
     481    "$WJcm $CANT_CM;"
     482    "$WJcm $CAN_CM $CM*;"
     483    "$GLcm $CAN_CM $CM*;"
     484    "$GLcm $CANT_CM;"
     485    "[[$LB8NonBreaks] - [$SP $BA $HY]] $CM* $GLcm;"
     486    "$CM+ GLcm;"
     487    "$LB8NonBreaks $CL;"
     488    "$CAN_CM $CM* $CL;"
     489    "$CM+ $CL;"
     490    "$LB8NonBreaks $CP;"
     491    "$CAN_CM $CM* $CP;"
     492    "$CM+ $CP;"
     493    "$LB8NonBreaks $EX;"
     494    "$CAN_CM $CM* $EX;"
     495    "$CM+ $EX;"
     496    "$LB8NonBreaks $IS;"
     497    "$CAN_CM $CM* $IS;"
     498    "$CM+ $IS;"
     499    "$LB8NonBreaks $SY;"
     500    "$CAN_CM $CM* $SY;"
     501    "$CM+ $SY;"
     502    "$OPcm $SP* $CAN_CM $CM*;"
     503    "$OPcm $SP* $CANT_CM;"
     504    "$OPcm $SP+ $CM+ $AL_FOLLOW?;"
     505    "$QUcm $SP* $OPcm;"
     506    "($CLcm | $CPcm) $SP* $NScm;"
     507    "$B2cm $SP* $B2cm;"
     508    "$LB18NonBreaks $CM* $QUcm;"
     509    "$CM+ $QUcm;"
     510    "$QUcm .?;"
     511    "$QUcm $LB18NonBreaks $CM*;"
     512    "$LB20NonBreaks $CM* ($BAcm | $HYcm | $NScm); "
     513    "$BBcm [^$CB];"
     514    "$BBcm $LB20NonBreaks $CM*;"
     515    "$HLcm ($HYcm | $BAcm) [^$CB]?;"
     516    "($ALcm | $HLcm) $INcm;"
     517    "$CM+ $INcm;"
     518    "$IDcm $INcm;"
     519    "$INcm $INcm;"
     520    "$NUcm $INcm;"
     521    "$IDcm $POcm;"
     522    "$ALcm $NUcm;"
     523    "$HLcm $NUcm;"
     524    "$CM+ $NUcm;"
     525    "$NUcm $ALcm;"
     526    "$NUcm $HLcm;"
     527    "$PRcm $IDcm;"
     528    "$PRcm ($ALcm | $HLcm);"
     529    "$POcm ($ALcm | $HLcm);"
     530    "($PRcm | $POcm)? ($OPcm | $HYcm)? $NUcm ($NUcm | $SYcm | $IScm)* ($CLcm | $CPcm)? ($PRcm | $POcm)?;"
     531    "$JLcm ($JLcm | $JVcm | $H2cm | $H3cm);"
     532    "($JVcm | $H2cm) ($JVcm | $JTcm);"
     533    "($JTcm | $H3cm) $JTcm;"
     534    "($JLcm | $JVcm | $JTcm | $H2cm | $H3cm) $INcm;"
     535    "($JLcm | $JVcm | $JTcm | $H2cm | $H3cm) $POcm;"
     536    "$PRcm ($JLcm | $JVcm | $JTcm | $H2cm | $H3cm);"
     537    "($ALcm | $HLcm) ($ALcm | $HLcm);"
     538    "$CM+ ($ALcm | $HLcm);"
     539    "$IScm ($ALcm | $HLcm);"
     540    "($ALcm | $HLcm | $NUcm) $OPcm;"
     541    "$CM+ $OPcm;"
     542    "$CPcm ($ALcm | $HLcm | $NUcm);";
     543
     544static const char* uax14Reverse =
     545    "!!reverse;"
     546    "$CM+ $ALPlus;"
     547    "$CM+ $BA;"
     548    "$CM+ $BB;"
     549    "$CM+ $B2;"
     550    "$CM+ $CL;"
     551    "$CM+ $CP;"
     552    "$CM+ $EX;"
     553    "$CM+ $GL;"
     554    "$CM+ $HL;"
     555    "$CM+ $HY;"
     556    "$CM+ $H2;"
     557    "$CM+ $H3;"
     558    "$CM+ $ID;"
     559    "$CM+ $IN;"
     560    "$CM+ $IS;"
     561    "$CM+ $JL;"
     562    "$CM+ $JV;"
     563    "$CM+ $JT;"
     564    "$CM+ $NS;"
     565    "$CM+ $NU;"
     566    "$CM+ $OP;"
     567    "$CM+ $PO;"
     568    "$CM+ $PR;"
     569    "$CM+ $QU;"
     570    "$CM+ $SY;"
     571    "$CM+ $WJ;"
     572    "$CM+;"
     573    "$AL_FOLLOW $CM+ / ([$BK $CR $LF $NL $ZW {eof}] | $SP+ $CM+ $SP | $SP+ $CM* ([^$OP $CM $SP] | [$AL {eof}]));"
     574    "[$PR] / $CM+ [$BK $CR $LF $NL $ZW $SP {eof}];"
     575    "$LB4Breaks [$LB4NonBreaks-$CM];"
     576    "$LB4Breaks $CM+ $CAN_CM;"
     577    "$LF $CR;"
     578    "[$SP $ZW] [$LB4NonBreaks-$CM];"
     579    "[$SP $ZW] $CM+ $CAN_CM;"
     580    "$CM+ $CAN_CM;"
     581    "$CM* $WJ $CM* $CAN_CM;"
     582    "$CM* $WJ [$LB8NonBreaks-$CM];"
     583    "$CANT_CM $CM* $WJ;"
     584    "$CM* $CAN_CM $CM* $WJ;"
     585    "$CM* $GL $CM* [$LB8NonBreaks-[$CM $SP $BA $HY]];"
     586    "$CANT_CM $CM* $GL;"
     587    "$CM* $CAN_CM $CM* $GL;"
     588    "$CL $CM+ $CAN_CM;"
     589    "$CP $CM+ $CAN_CM;"
     590    "$EX $CM+ $CAN_CM;"
     591    "$IS $CM+ $CAN_CM;"
     592    "$SY $CM+ $CAN_CM;"
     593    "$CL [$LB8NonBreaks-$CM];"
     594    "$CP [$LB8NonBreaks-$CM];"
     595    "$EX [$LB8NonBreaks-$CM];"
     596    "$IS [$LB8NonBreaks-$CM];"
     597    "$SY [$LB8NonBreaks-$CM];"
     598    "[$CL $CP $EX $IS $SY] $CM+ $SP+ $CM* $OP; "
     599    "$CM* $CAN_CM $SP* $CM* $OP;"
     600    "$CANT_CM $SP* $CM* $OP;"
     601    "$AL_FOLLOW? $CM+ $SP $SP* $CM* $OP;"
     602    "$AL_FOLLOW_NOCM $CM+ $SP+ $CM* $OP;"
     603    "$CM* $AL_FOLLOW_CM $CM+ $SP+ $CM* $OP;"
     604    "$SY $CM $SP+ $OP;"
     605    "$CM* $OP $SP* $CM* $QU;"
     606    "$CM* $NS $SP* $CM* ($CL | $CP);"
     607    "$CM* $B2 $SP* $CM* $B2;"
     608    "$CM* $QU $CM* $CAN_CM;"
     609    "$CM* $QU $LB18NonBreaks;"
     610    "$CM* $CAN_CM $CM* $QU;"
     611    "$CANT_CM $CM* $QU;"
     612    "$CM* ($BA | $HY | $NS) $CM* [$LB20NonBreaks-$CM];"
     613    "$CM* [$LB20NonBreaks-$CM] $CM* $BB;"
     614    "[^$CB] $CM* $BB;"
     615    "[^$CB] $CM* ($HY | $BA) $CM* $HL;"
     616    "$CM* $IN $CM* ($ALPlus | $HL);"
     617    "$CM* $IN $CM* $ID;"
     618    "$CM* $IN $CM* $IN;"
     619    "$CM* $IN $CM* $NU;"
     620    "$CM* $PO $CM* $ID;"
     621    "$CM* $NU $CM* ($ALPlus | $HL);"
     622    "$CM* ($ALPlus | $HL) $CM* $NU;"
     623    "$CM* $ID $CM* $PR;"
     624    "$CM* ($ALPlus | $HL) $CM* $PR;"
     625    "$CM* ($ALPlus | $HL) $CM* $PO;"
     626    "($CM* ($PR | $PO))? ($CM* ($CL | $CP))? ($CM* ($NU | $IS | $SY))* $CM* $NU ($CM* ($OP | $HY))? ($CM* ($PR | $PO))?;"
     627    "$CM* ($H3 | $H2 | $JV | $JL) $CM* $JL;"
     628    "$CM* ($JT | $JV) $CM* ($H2 | $JV);"
     629    "$CM* $JT $CM* ($H3 | $JT);"
     630    "$CM* $IN $CM* ($H3 | $H2 | $JT | $JV | $JL);"
     631    "$CM* $PO $CM* ($H3 | $H2 | $JT | $JV | $JL);"
     632    "$CM* ($H3 | $H2 | $JT | $JV | $JL) $CM* $PR;"
     633    "$CM* ($ALPlus | $HL) $CM* ($ALPlus | $HL);"
     634    "$CM* ($ALPlus | $HL) $CM* $IS;"
     635    "$CM* $OP $CM* ($ALPlus | $HL | $NU);"
     636    "$CM* ($ALPlus | $HL | $NU) $CM* $CP;";
     637
     638static const char* uax14SafeForward =
     639    "!!safe_forward;"
     640    "[$CM $OP $QU $CL $CP $B2 $PR $HY $BA $SP $dictionary]+ [^$CM $OP $QU $CL $CP $B2 $PR $HY $BA $dictionary];"
     641    "$dictionary $dictionary;";
     642
     643static const char* uax14SafeReverse =
     644    "!!safe_reverse;"
     645    "$CM+ [^$CM $BK $CR $LF $NL $ZW $SP];"
     646    "$CM+ $SP / .;"
     647    "$SP+ $CM* $OP;"
     648    "$SP+ $CM* $QU;"
     649    "$SP+ $CM* ($CL | $CP);"
     650    "$SP+ $CM* $B2;"
     651    "$CM* ($HY | $BA) $CM* $HL;"
     652    "($CM* ($IS | $SY))+ $CM* $NU;"
     653    "($CL | $CP) $CM* ($NU | $IS | $SY);"
     654    "$dictionary $dictionary;";
     655
     656static String mapLineIteratorModeToRules(LineBreakIteratorMode mode, bool isCJK)
     657{
     658    StringBuilder rulesBuilder;
     659    rulesBuilder.append(uax14Prologue);
     660    rulesBuilder.append(uax14AssignmentsBefore);
     661    switch (mode) {
     662    case LineBreakIteratorModeUAX14:
     663        rulesBuilder.append(isCJK ? uax14AssignmentsCustomDefaultCJK : uax14AssignmentsCustomDefaultNonCJK);
     664        break;
     665    case LineBreakIteratorModeUAX14Loose:
     666        rulesBuilder.append(isCJK ? uax14AssignmentsCustomLooseCJK : uax14AssignmentsCustomLooseNonCJK);
     667        break;
     668    case LineBreakIteratorModeUAX14Normal:
     669        rulesBuilder.append(isCJK ? uax14AssignmentsCustomNormalCJK : uax14AssignmentsCustomNormalNonCJK);
     670        break;
     671    case LineBreakIteratorModeUAX14Strict:
     672        rulesBuilder.append(isCJK ? uax14AssignmentsCustomStrictCJK : uax14AssignmentsCustomStrictNonCJK);
     673        break;
     674    }
     675    rulesBuilder.append(uax14AssignmentsAfter);
     676    rulesBuilder.append(uax14Forward);
     677    rulesBuilder.append(uax14Reverse);
     678    rulesBuilder.append(uax14SafeForward);
     679    rulesBuilder.append(uax14SafeReverse);
     680    return rulesBuilder.toString();
     681}
     682
     683// Recognize BCP47 compliant primary language values of 'zh', 'ja', 'ko'
     684// (in any combination of case), optionally followed by subtags. Don't
     685// recognize 3-letter variants 'chi'/'zho', 'jpn', or 'kor' since BCP47
     686// requires use of shortest language tag.
     687bool isCJKLocale(const AtomicString& locale)
     688{
     689    size_t length = locale.length();
     690    if (length < 2)
     691        return false;
     692    auto c1 = locale[0];
     693    auto c2 = locale[1];
     694    auto c3 = length == 2 ? 0 : locale[2];
     695    if (!c3 || c3 == '-' || c3 == '_' || c3 == '@') {
     696        if (c1 == 'z' || c1 == 'Z')
     697            return c2 == 'h' || c2 == 'H';
     698        if (c1 == 'j' || c1 == 'J')
     699            return c2 == 'a' || c2 == 'A';
     700        if (c1 == 'k' || c1 == 'K')
     701            return c2 == 'o' || c2 == 'O';
     702    }
     703    return false;
     704}
     705
     706TextBreakIterator* openLineBreakIterator(const AtomicString& locale, LineBreakIteratorMode mode, bool isCJK)
     707{
     708    UBreakIterator* ubrkIter;
     709    UErrorCode openStatus = U_ZERO_ERROR;
     710    bool localeIsEmpty = locale.isEmpty();
     711    if (mode == LineBreakIteratorModeUAX14)
     712        ubrkIter = ubrk_open(UBRK_LINE, localeIsEmpty ? currentTextBreakLocaleID() : locale.string().utf8().data(), 0, 0, &openStatus);
     713    else {
     714        UParseError parseStatus;
     715        auto rules = mapLineIteratorModeToRules(mode, isCJK);
     716        ubrkIter = ubrk_openRules(StringView(rules).upconvertedCharacters(), rules.length(), 0, 0, &parseStatus, &openStatus);
     717    }
     718    // locale comes from a web page and it can be invalid, leading ICU
     719    // to fail, in which case we fall back to the default locale.
     720    if (!localeIsEmpty && U_FAILURE(openStatus)) {
     721        openStatus = U_ZERO_ERROR;
     722        ubrkIter = ubrk_open(UBRK_LINE, currentTextBreakLocaleID(), 0, 0, &openStatus);
     723    }
     724
     725    if (U_FAILURE(openStatus)) {
     726        LOG_ERROR("ubrk_open failed with status %d", openStatus);
     727        return nullptr;
     728    }
     729
     730    return reinterpret_cast<TextBreakIterator*>(ubrkIter);
     731}
     732
     733void closeLineBreakIterator(TextBreakIterator*& iterator)
     734{
     735    UBreakIterator* ubrkIter = reinterpret_cast<UBreakIterator*>(iterator);
     736    ASSERT(ubrkIter);
     737    ubrk_close(ubrkIter);
     738    iterator = nullptr;
    273739}
    274740

  • trunk/Source/WebCore/platform/text/TextBreakIterator.h

    r172862 r176473  
    3131
    3232// Note: The returned iterator is good only until you get another iterator, with the exception of acquireLineBreakIterator.
     33
     34enum LineBreakIteratorMode {
     35    LineBreakIteratorModeUAX14,
     36    LineBreakIteratorModeUAX14Loose,
     37    LineBreakIteratorModeUAX14Normal,
     38    LineBreakIteratorModeUAX14Strict,
     39};
    3340
    3441// This is similar to character break iterator in most cases, but is subject to
    … …  
    4148TextBreakIterator* sentenceBreakIterator(StringView);
    4249
    43 WEBCORE_EXPORT TextBreakIterator* acquireLineBreakIterator(StringView, const AtomicString& locale, const UChar* priorContext, unsigned priorContextLength);
     50WEBCORE_EXPORT TextBreakIterator* acquireLineBreakIterator(StringView, const AtomicString& locale, const UChar* priorContext, unsigned priorContextLength, LineBreakIteratorMode, bool isCJK);
    4451WEBCORE_EXPORT void releaseLineBreakIterator(TextBreakIterator*);
     52TextBreakIterator* openLineBreakIterator(const AtomicString& locale, LineBreakIteratorMode, bool isCJK);
     53void closeLineBreakIterator(TextBreakIterator*&);
    4554
    4655int textBreakFirst(TextBreakIterator*);
    … …  
    5665const int TextBreakDone = -1;
    5766
     67bool isCJKLocale(const AtomicString&);
     68
    5869class LazyLineBreakIterator {
    5970public:
    6071    LazyLineBreakIterator()
    61         : m_iterator(0)
    62         , m_cachedPriorContext(0)
     72        : m_iterator(nullptr)
     73        , m_cachedPriorContext(nullptr)
     74        , m_mode(LineBreakIteratorModeUAX14)
    6375        , m_cachedPriorContextLength(0)
     76        , m_isCJK(false)
    6477    {
    6578        resetPriorContext();
    6679    }
    6780
    68     LazyLineBreakIterator(String string, const AtomicString& locale = AtomicString())
     81    LazyLineBreakIterator(String string, const AtomicString& locale = AtomicString(), LineBreakIteratorMode mode = LineBreakIteratorModeUAX14)
    6982        : m_string(string)
    7083        , m_locale(locale)
    71         , m_iterator(0)
    72         , m_cachedPriorContext(0)
     84        , m_iterator(nullptr)
     85        , m_cachedPriorContext(nullptr)
     86        , m_mode(mode)
    7387        , m_cachedPriorContextLength(0)
    7488    {
    7589        resetPriorContext();
     90        m_isCJK = isCJKLocale(locale);
    7691    }
    7792
    … …  
    8398
    8499    String string() const { return m_string; }
     100    bool isLooseCJKMode() const { return m_isCJK && m_mode == LineBreakIteratorModeUAX14Loose; }
    85101
    86102    UChar lastCharacter() const
    … …  
    89105        return m_priorContext[1];
    90106    }
     107
    91108    UChar secondToLastCharacter() const
    92109    {
    … …  
    94111        return m_priorContext[0];
    95112    }
     113
    96114    void setPriorContext(UChar last, UChar secondToLast)
    97115    {
    … …  
    100118        m_priorContext[1] = last;
    101119    }
     120
    102121    void updatePriorContext(UChar last)
    103122    {
    … …  
    106125        m_priorContext[1] = last;
    107126    }
     127
    108128    void resetPriorContext()
    109129    {
    … …  
    112132        m_priorContext[1] = 0;
    113133    }
     134
    114135    unsigned priorContextLength() const
    115136    {
    … …  
    123144        return priorContextLength;
    124145    }
     146
    125147    // Obtain text break iterator, possibly previously cached, where this iterator is (or has been)
    126148    // initialized to use the previously stored string as the primary breaking context and using
    … …  
    131153        const UChar* priorContext = priorContextLength ? &m_priorContext[priorContextCapacity - priorContextLength] : 0;
    132154        if (!m_iterator) {
    133             m_iterator = acquireLineBreakIterator(m_string, m_locale, priorContext, priorContextLength);
     155            m_iterator = acquireLineBreakIterator(m_string, m_locale, priorContext, priorContextLength, m_mode, m_isCJK);
    134156            m_cachedPriorContext = priorContext;
    135157            m_cachedPriorContextLength = priorContextLength;
    136158        } else if (priorContext != m_cachedPriorContext || priorContextLength != m_cachedPriorContextLength) {
    137             this->resetStringAndReleaseIterator(m_string, m_locale);
     159            resetStringAndReleaseIterator(m_string, m_locale, m_mode);
    138160            return this->get(priorContextLength);
    139161        }
    140162        return m_iterator;
    141163    }
    142     void resetStringAndReleaseIterator(String string, const AtomicString& locale)
     164
     165    void resetStringAndReleaseIterator(String string, const AtomicString& locale, LineBreakIteratorMode mode)
    143166    {
    144167        if (m_iterator)
    … …  
    146169        m_string = string;
    147170        m_locale = locale;
    148         m_iterator = 0;
    149         m_cachedPriorContext = 0;
     171        m_iterator = nullptr;
     172        m_cachedPriorContext = nullptr;
     173        m_mode = mode;
     174        m_isCJK = isCJKLocale(locale);
    150175        m_cachedPriorContextLength = 0;
    151176    }
    … …  
    156181    AtomicString m_locale;
    157182    TextBreakIterator* m_iterator;
     183    const UChar* m_cachedPriorContext;
     184    LineBreakIteratorMode m_mode;
     185    unsigned m_cachedPriorContextLength;
    158186    UChar m_priorContext[priorContextCapacity];
    159     const UChar* m_cachedPriorContext;
    160     unsigned m_cachedPriorContextLength;
     187    bool m_isCJK;
    161188};
    162189

  • trunk/Source/WebCore/rendering/RenderText.cpp

    r174875 r176473  
    598598}
    599599
     600LineBreakIteratorMode mapLineBreakToIteratorMode(LineBreak lineBreak)
     601{
     602    switch (lineBreak) {
     603    case LineBreakAuto:
     604    case LineBreakAfterWhiteSpace:
     605        return LineBreakIteratorModeUAX14;
     606    case LineBreakLoose:
     607        return LineBreakIteratorModeUAX14Loose;
     608    case LineBreakNormal:
     609        return LineBreakIteratorModeUAX14Normal;
     610    case LineBreakStrict:
     611        return LineBreakIteratorModeUAX14Strict;
     612    }
     613    return LineBreakIteratorModeUAX14;
     614}
     615
    600616void RenderText::computePreferredLogicalWidths(float leadWidth)
    601617{
    … …  
    674690    float wordSpacing = font.wordSpacing();
    675691    int len = textLength();
    676     LazyLineBreakIterator breakIterator(m_text, style.locale());
     692    LazyLineBreakIterator breakIterator(m_text, style.locale(), mapLineBreakToIteratorMode(style.lineBreak()));
    677693    bool needsWordSpacing = false;
    678694    bool ignoringSpaces = false;
    … …  
    709725    bool breakNBSP = style.autoWrap() && style.nbspMode() == SPACE;
    710726    bool breakAll = (style.wordBreak() == BreakAllWordBreak || style.wordBreak() == BreakWordBreak) && style.autoWrap();
     727    bool isLooseCJKMode = breakIterator.isLooseCJKMode();
    711728
    712729    for (int i = 0; i < len; i++) {
    … …  
    756773        }
    757774
    758         bool hasBreak = breakAll || isBreakable(breakIterator, i, nextBreakable, breakNBSP);
     775        bool hasBreak = breakAll || isBreakable(breakIterator, i, nextBreakable, breakNBSP, isLooseCJKMode);
    759776        bool betweenWords = true;
    760777        int j = i;
    … …  
    764781                break;
    765782            c = uncheckedCharacterAt(j);
    766             if (isBreakable(breakIterator, j, nextBreakable, breakNBSP) && characterAt(j - 1) != softHyphen)
     783            if (isBreakable(breakIterator, j, nextBreakable, breakNBSP, isLooseCJKMode) && characterAt(j - 1) != softHyphen)
    767784                break;
    768785            if (breakAll) {

  • trunk/Source/WebCore/rendering/RenderText.h

    r176365 r176473  
    2828#include "SimpleLineLayout.h"
    2929#include "Text.h"
     30#include "TextBreakIterator.h"
    3031#include <wtf/Forward.h>
    3132
    … …  
    275276void applyTextTransform(const RenderStyle&, String&, UChar);
    276277void makeCapitalized(String*, UChar previous);
     278LineBreakIteratorMode mapLineBreakToIteratorMode(LineBreak);
    277279   
    278280inline RenderText* Text::renderer() const

  • trunk/Source/WebCore/rendering/SimpleLineLayout.cpp

    r176401 r176473  
    161161        return false;
    162162    if (style.borderFit() == BorderFitLines)
     163        return false;
     164    if (style.lineBreak() != LineBreakAuto)
    163165        return false;
    164166    const RenderText& textRenderer = downcast<RenderText>(*flow.firstChild());

  • trunk/Source/WebCore/rendering/SimpleLineLayoutFlowContents.cpp

    r176470 r176473  
    5454{
    5555    String string = m_lineBreakIterator.string();
    56     unsigned breakablePosition = nextBreakablePosition<LChar, false>(m_lineBreakIterator, string.characters8(), string.length(), position);
     56    unsigned breakablePosition = nextBreakablePositionNonLoosely<LChar, NBSPBehavior::IgnoreNBSP>(m_lineBreakIterator, string.characters8(), string.length(), position);
    5757    if (appendNextRendererContentIfNeeded(breakablePosition))
    5858        return findNextBreakablePosition(position);
    … …  
    158158
    159159    ++m_lastRendererIndex;
    160     m_lineBreakIterator.resetStringAndReleaseIterator(string + String(nextRenderer->text()), m_flow.style().locale());
     160    m_lineBreakIterator.resetStringAndReleaseIterator(string + String(nextRenderer->text()), m_flow.style().locale(), LineBreakIteratorModeUAX14);
    161161    return true;
    162162}

  • trunk/Source/WebCore/rendering/break_lines.h

    r163310 r176473  
    3636extern const unsigned char asciiLineBreakTable[][asciiLineBreakTableColumnCount];
    3737
    38 int nextBreakablePositionIgnoringNBSP(LazyLineBreakIterator&, int pos);
    39 int nextBreakablePosition(LazyLineBreakIterator&, int pos);
    40 
    41 template<bool treatNoBreakSpaceAsBreak>
     38enum class NBSPBehavior {
     39    IgnoreNBSP,
     40    TreatNBSPAsBreak,
     41};
     42
     43template<NBSPBehavior nbspBehavior>
    4244static inline bool isBreakableSpace(UChar ch)
    4345{
    … …  
    4850        return true;
    4951    case noBreakSpace:
    50         return treatNoBreakSpaceAsBreak;
     52        return nbspBehavior == NBSPBehavior::TreatNBSPAsBreak;
    5153    default:
    5254        return false;
    … …  
    7274}
    7375
    74 template<bool treatNoBreakSpaceAsBreak>
     76template<NBSPBehavior nbspBehavior>
    7577inline bool needsLineBreakIterator(UChar ch)
    7678{
    77     if (treatNoBreakSpaceAsBreak)
     79    if (nbspBehavior == NBSPBehavior::TreatNBSPAsBreak)
    7880        return ch > asciiLineBreakTableLastChar;
    7981    return ch > asciiLineBreakTableLastChar && ch != noBreakSpace;
    8082}
    8183
    82 template<typename CharacterType, bool treatNoBreakSpaceAsBreak>
    83 inline int nextBreakablePosition(LazyLineBreakIterator& lazyBreakIterator, const CharacterType* str, unsigned length, int pos)
     84// When in non-loose mode, we can use the ASCII shortcut table.
     85template<typename CharacterType, NBSPBehavior nbspBehavior>
     86inline int nextBreakablePositionNonLoosely(LazyLineBreakIterator& lazyBreakIterator, const CharacterType* str, unsigned length, int pos)
    8487{
    8588    int len = static_cast<int>(length);
    … …  
    9295        CharacterType ch = str[i];
    9396
    94         if (isBreakableSpace<treatNoBreakSpaceAsBreak>(ch) || shouldBreakAfter(lastLastCh, lastCh, ch))
     97        // Non-loose mode, so use ASCII shortcut (shouldBreakAfter) if not breakable space.
     98        if (isBreakableSpace<nbspBehavior>(ch) || shouldBreakAfter(lastLastCh, lastCh, ch))
    9599            return i;
    96100
    97         if (needsLineBreakIterator<treatNoBreakSpaceAsBreak>(ch) || needsLineBreakIterator<treatNoBreakSpaceAsBreak>(lastCh)) {
     101        // Non-loose mode, so conditionally use break iterator.
     102        if (needsLineBreakIterator<nbspBehavior>(ch) || needsLineBreakIterator<nbspBehavior>(lastCh)) {
    98103            if (nextBreak < i) {
    99104                // Don't break if positioned at start of primary context and there is no prior context.
    … …  
    107112                }
    108113            }
    109             if (i == nextBreak && !isBreakableSpace<treatNoBreakSpaceAsBreak>(lastCh))
     114            if (i == nextBreak && !isBreakableSpace<nbspBehavior>(lastCh))
    110115                return i;
    111116        }
    … …  
    118123}
    119124
     125// When in loose mode, we can't use the ASCII shortcut table since loose mode allows "$100" to break after '$' in content marked as CJK.
     126// N.B. It should be possible to combine the following with the non-loose version above by adding a LooseBehavior template parameter;
     127// however, when doing this, a 10% performance regression appeared on chromium-win (https://bugs.webkit.org/show_bug.cgi?id=89235#c112).
     128template<typename CharacterType, NBSPBehavior nbspBehavior>
     129static inline int nextBreakablePositionLoosely(LazyLineBreakIterator& lazyBreakIterator, const CharacterType* str, unsigned length, int pos)
     130{
     131    int len = static_cast<int>(length);
     132    int nextBreak = -1;
     133
     134    CharacterType lastCh = pos > 0 ? str[pos - 1] : static_cast<CharacterType>(lazyBreakIterator.lastCharacter());
     135    unsigned priorContextLength = lazyBreakIterator.priorContextLength();
     136    for (int i = pos; i < len; i++) {
     137        CharacterType ch = str[i];
     138
     139        // Always loose mode, so don't use ASCII shortcut (shouldBreakAfter).
     140        if (isBreakableSpace<nbspBehavior>(ch))
     141            return i;
     142
     143        // Always use line break iterator in loose mode.
     144        if (nextBreak < i) {
     145            // Don't break if positioned at start of primary context and there is no prior context.
     146            if (i || priorContextLength) {
     147                TextBreakIterator* breakIterator = lazyBreakIterator.get(priorContextLength);
     148                if (breakIterator) {
     149                    nextBreak = textBreakFollowing(breakIterator, i - 1 + priorContextLength);
     150                    if (nextBreak >= 0)
     151                        nextBreak -= priorContextLength;
     152                }
     153            }
     154        }
     155        if (i == nextBreak && !isBreakableSpace<nbspBehavior>(lastCh))
     156            return i;
     157
     158        lastCh = ch;
     159    }
     160
     161    return len;
     162}
     163
     164inline int nextBreakablePosition(LazyLineBreakIterator& lazyBreakIterator, int pos)
     165{
     166    String string = lazyBreakIterator.string();
     167    if (string.is8Bit())
     168        return nextBreakablePositionNonLoosely<LChar, NBSPBehavior::TreatNBSPAsBreak>(lazyBreakIterator, string.characters8(), string.length(), pos);
     169    return nextBreakablePositionNonLoosely<UChar, NBSPBehavior::TreatNBSPAsBreak>(lazyBreakIterator, string.characters16(), string.length(), pos);
     170}
     171
    120172inline int nextBreakablePositionIgnoringNBSP(LazyLineBreakIterator& lazyBreakIterator, int pos)
    121173{
    122174    String string = lazyBreakIterator.string();
    123175    if (string.is8Bit())
    124         return nextBreakablePosition<LChar, false>(lazyBreakIterator, string.characters8(), string.length(), pos);
    125     return nextBreakablePosition<UChar, false>(lazyBreakIterator, string.characters16(), string.length(), pos);
    126 }
    127 
    128 inline int nextBreakablePosition(LazyLineBreakIterator& lazyBreakIterator, int pos)
    129 {
    130     String string = lazyBreakIterator.string();
    131     if (string.is8Bit())
    132         return nextBreakablePosition<LChar, true>(lazyBreakIterator, string.characters8(), string.length(), pos);
    133     return nextBreakablePosition<UChar, true>(lazyBreakIterator, string.characters16(), string.length(), pos);
    134 }
    135 
    136 inline bool isBreakable(LazyLineBreakIterator& lazyBreakIterator, int pos, int& nextBreakable, bool breakNBSP)
    137 {
    138     if (pos > nextBreakable) {
     176        return nextBreakablePositionNonLoosely<LChar, NBSPBehavior::IgnoreNBSP>(lazyBreakIterator, string.characters8(), string.length(), pos);
     177    return nextBreakablePositionNonLoosely<UChar, NBSPBehavior::IgnoreNBSP>(lazyBreakIterator, string.characters16(), string.length(), pos);
     178}
     179
     180inline int nextBreakablePositionLoose(LazyLineBreakIterator& lazyBreakIterator, int pos)
     181{
     182    String string = lazyBreakIterator.string();
     183    if (string.is8Bit())
     184        return nextBreakablePositionLoosely<LChar, NBSPBehavior::TreatNBSPAsBreak>(lazyBreakIterator, string.characters8(), string.length(), pos);
     185    return nextBreakablePositionLoosely<UChar, NBSPBehavior::TreatNBSPAsBreak>(lazyBreakIterator, string.characters16(), string.length(), pos);
     186}
     187
     188inline int nextBreakablePositionIgnoringNBSPLoose(LazyLineBreakIterator& lazyBreakIterator, int pos)
     189{
     190    String string = lazyBreakIterator.string();
     191    if (string.is8Bit())
     192        return nextBreakablePositionLoosely<LChar, NBSPBehavior::IgnoreNBSP>(lazyBreakIterator, string.characters8(), string.length(), pos);
     193    return nextBreakablePositionLoosely<UChar, NBSPBehavior::IgnoreNBSP>(lazyBreakIterator, string.characters16(), string.length(), pos);
     194}
     195
     196inline bool isBreakable(LazyLineBreakIterator& lazyBreakIterator, int pos, int& nextBreakable, bool breakNBSP, bool isLooseMode)
     197{
     198    if (pos <= nextBreakable)
     199        return pos == nextBreakable;
     200
     201    if (isLooseMode) {
     202        if (breakNBSP)
     203            nextBreakable = nextBreakablePositionLoose(lazyBreakIterator, pos);
     204        else
     205            nextBreakable = nextBreakablePositionIgnoringNBSPLoose(lazyBreakIterator, pos);
     206    } else {
    139207        if (breakNBSP)
    140208            nextBreakable = nextBreakablePosition(lazyBreakIterator, pos);

  • trunk/Source/WebCore/rendering/line/BreakingContextInlineHeaders.h

    r176287 r176473  
    625625    bool breakAll = m_currentStyle->wordBreak() == BreakAllWordBreak && m_autoWrap;
    626626    float hyphenWidth = 0;
     627    bool isLooseCJKMode = false;
    627628
    628629    if (isSVGText) {
    … …  
    636637        m_renderTextInfo.m_font = &font;
    637638        m_renderTextInfo.m_layout = font.createLayout(&renderText, m_width.currentWidth(), m_collapseWhiteSpace);
    638         m_renderTextInfo.m_lineBreakIterator.resetStringAndReleaseIterator(renderText.text(), style.locale());
     639        m_renderTextInfo.m_lineBreakIterator.resetStringAndReleaseIterator(renderText.text(), style.locale(), mapLineBreakToIteratorMode(m_blockStyle.lineBreak()));
     640        isLooseCJKMode = m_renderTextInfo.m_lineBreakIterator.isLooseCJKMode();
    639641    } else if (m_renderTextInfo.m_layout && m_renderTextInfo.m_font != &font) {
    640642        m_renderTextInfo.m_font = &font;
    … …  
    677679
    678680        int nextBreakablePosition = m_current.nextBreakablePosition();
    679         bool betweenWords = c == '\n' || (m_currWS != PRE && !m_atStart && isBreakable(m_renderTextInfo.m_lineBreakIterator, m_current.offset(), nextBreakablePosition, breakNBSP)
     681        bool betweenWords = c == '\n' || (m_currWS != PRE && !m_atStart && isBreakable(m_renderTextInfo.m_lineBreakIterator, m_current.offset(), nextBreakablePosition, breakNBSP, isLooseCJKMode)
    680682            && (style.hyphens() != HyphensNone || (m_current.previousInSameNode() != softHyphen)));
    681683        m_current.setNextBreakablePosition(nextBreakablePosition);
Note: See TracChangeset for help on using the changeset viewer.