Changeset 211224 in webkit


Ignore:
Timestamp:
Jan 26, 2017 11:52:35 AM (7 years ago)
Author:
jfbastien@apple.com
Message:

OSR entry: delay outer-loop compilation when at inner-loop
​https://bugs.webkit.org/show_bug.cgi?id=167149

Reviewed by Filip Pizlo.

JSTests:

Try to be mean to OSR entry by using nested loops, and having
non-int32 types or truly varying types.

Mandelbrot currently never tiers up to FTL because it exits too
many times before this. That shouldn't happen because it's just
numbers and int32s. I'll file a bug to fix this.

  • microbenchmarks/mandelbrot.js: Added.

(mandelbrot):
(printable):

  • microbenchmarks/nonude.js: Added.

(Array.prototype.remove):
(const.u):
(const.load):
(const.scan):
(const.main):

Source/JavaScriptCore:

As of ​https://bugs.webkit.org/show_bug.cgi?id=155217 OSR
compilation can be kicked off for an entry into an outer-loop,
while executing an inner-loop. This is desirable because often the
codegen from an inner-entry isn't as good as the codegen from an
outer-entry, but execution from an inner-loop is often pretty hot
and likely to kick off compilation. This approach provided nice
speedups on Kraken because we'd select to enter to the outer-loop
very reliably, which reduces variability (the inner-loop was
selected roughly 1/5 times from my unscientific measurements).

When compilation starts we take a snapshot of the JSValues at the
current execution state using OSR's recovery mechanism. These
values are passed to the compiler and are used as way to perform
type profiling, and could be used to observe cell types as well as
to perform predictions such as through constant propagation.

It's therefore desired to enter from the outer-loop when we can,
but we need to be executing from that location to capture the
right JSValues, otherwise we're confusing the compiler and giving
it inaccurate JSValues which can lead it to predict the wrong
things, leading to suboptimal code or recompilation due to
misprediction, or in super-corner-cases a crash.

These effects are pretty hard to measure: Fil points out that
marsalis-osr-entry really needs mustHandleValues (the JSValues
from the point of execution) because right now it just happens to
correctly guess int32. I tried removing mustHandleValues entirely
and saw no slowdowns, but our benchmarks probably aren't
sufficient to reliably find issues, sometimes because we happen to
have sufficient mitigations.

DFG tier-up was added here:
​https://bugs.webkit.org/show_bug.cgi?id=112838

  • JavaScriptCore.xcodeproj/project.pbxproj:
  • dfg/DFGJITCode.h:
  • dfg/DFGJITCompiler.cpp:

(JSC::DFG::JITCompiler::JITCompiler):

  • dfg/DFGOSREntry.cpp:

(JSC::DFG::prepareOSREntry):

  • dfg/DFGOSREntry.h:

(JSC::DFG::prepareOSREntry):

  • dfg/DFGOperations.cpp:
  • dfg/DFGOperations.h:
  • dfg/DFGSpeculativeJIT64.cpp:

(JSC::DFG::SpeculativeJIT::compile):

  • dfg/DFGTierUpEntryTrigger.h: Copied from Source/JavaScriptCore/ftl/FTLOSREntry.h.
  • dfg/DFGToFTLForOSREntryDeferredCompilationCallback.cpp:

(JSC::DFG::ToFTLForOSREntryDeferredCompilationCallback::ToFTLForOSREntryDeferredCompilationCallback):
(JSC::DFG::Ref<ToFTLForOSREntryDeferredCompilationCallback>ToFTLForOSREntryDeferredCompilationCallback::create):
(JSC::DFG::ToFTLForOSREntryDeferredCompilationCallback::compilationDidBecomeReadyAsynchronously):
(JSC::DFG::ToFTLForOSREntryDeferredCompilationCallback::compilationDidComplete):

  • dfg/DFGToFTLForOSREntryDeferredCompilationCallback.h:
  • ftl/FTLOSREntry.cpp:

(JSC::FTL::prepareOSREntry):

  • ftl/FTLOSREntry.h:
  • jit/JITOperations.cpp:
Location:
trunk
Files:
2 added
15 edited
1 copied

Legend:

Unmodified
Added
Removed

  • trunk/JSTests/ChangeLog

    r211195 r211224  
     12017-01-26  JF Bastien  <jfbastien@apple.com>
     2
     3        OSR entry: delay outer-loop compilation when at inner-loop
     4        https://bugs.webkit.org/show_bug.cgi?id=167149
     5
     6        Reviewed by Filip Pizlo.
     7
     8        Try to be mean to OSR entry by using nested loops, and having
     9        non-int32 types or truly varying types.
     10
     11        Mandelbrot currently never tiers up to FTL because it exits too
     12        many times before this. That shouldn't happen because it's just
     13        numbers and int32s. I'll file a bug to fix this.
     14
     15        * microbenchmarks/mandelbrot.js: Added.
     16        (mandelbrot):
     17        (printable):
     18        * microbenchmarks/nonude.js: Added.
     19        (Array.prototype.remove):
     20        (const.u):
     21        (const.load):
     22        (const.scan):
     23        (const.main):
     24
    1252017-01-25  Saam Barati  <sbarati@apple.com>
    226

  • trunk/Source/JavaScriptCore/ChangeLog

    r211195 r211224  
     12017-01-26  JF Bastien  <jfbastien@apple.com>
     2
     3        OSR entry: delay outer-loop compilation when at inner-loop
     4        https://bugs.webkit.org/show_bug.cgi?id=167149
     5
     6        Reviewed by Filip Pizlo.
     7
     8        As of https://bugs.webkit.org/show_bug.cgi?id=155217 OSR
     9        compilation can be kicked off for an entry into an outer-loop,
     10        while executing an inner-loop. This is desirable because often the
     11        codegen from an inner-entry isn't as good as the codegen from an
     12        outer-entry, but execution from an inner-loop is often pretty hot
     13        and likely to kick off compilation. This approach provided nice
     14        speedups on Kraken because we'd select to enter to the outer-loop
     15        very reliably, which reduces variability (the inner-loop was
     16        selected roughly 1/5 times from my unscientific measurements).
     17
     18        When compilation starts we take a snapshot of the JSValues at the
     19        current execution state using OSR's recovery mechanism. These
     20        values are passed to the compiler and are used as way to perform
     21        type profiling, and could be used to observe cell types as well as
     22        to perform predictions such as through constant propagation.
     23
     24        It's therefore desired to enter from the outer-loop when we can,
     25        but we need to be executing from that location to capture the
     26        right JSValues, otherwise we're confusing the compiler and giving
     27        it inaccurate JSValues which can lead it to predict the wrong
     28        things, leading to suboptimal code or recompilation due to
     29        misprediction, or in super-corner-cases a crash.
     30
     31        These effects are pretty hard to measure: Fil points out that
     32        marsalis-osr-entry really needs mustHandleValues (the JSValues
     33        from the point of execution) because right now it just happens to
     34        correctly guess int32. I tried removing mustHandleValues entirely
     35        and saw no slowdowns, but our benchmarks probably aren't
     36        sufficient to reliably find issues, sometimes because we happen to
     37        have sufficient mitigations.
     38
     39        DFG tier-up was added here:
     40        https://bugs.webkit.org/show_bug.cgi?id=112838
     41
     42        * JavaScriptCore.xcodeproj/project.pbxproj:
     43        * dfg/DFGJITCode.h:
     44        * dfg/DFGJITCompiler.cpp:
     45        (JSC::DFG::JITCompiler::JITCompiler):
     46        * dfg/DFGOSREntry.cpp:
     47        (JSC::DFG::prepareOSREntry):
     48        * dfg/DFGOSREntry.h:
     49        (JSC::DFG::prepareOSREntry):
     50        * dfg/DFGOperations.cpp:
     51        * dfg/DFGOperations.h:
     52        * dfg/DFGSpeculativeJIT64.cpp:
     53        (JSC::DFG::SpeculativeJIT::compile):
     54        * dfg/DFGTierUpEntryTrigger.h: Copied from Source/JavaScriptCore/ftl/FTLOSREntry.h.
     55        * dfg/DFGToFTLForOSREntryDeferredCompilationCallback.cpp:
     56        (JSC::DFG::ToFTLForOSREntryDeferredCompilationCallback::ToFTLForOSREntryDeferredCompilationCallback):
     57        (JSC::DFG::Ref<ToFTLForOSREntryDeferredCompilationCallback>ToFTLForOSREntryDeferredCompilationCallback::create):
     58        (JSC::DFG::ToFTLForOSREntryDeferredCompilationCallback::compilationDidBecomeReadyAsynchronously):
     59        (JSC::DFG::ToFTLForOSREntryDeferredCompilationCallback::compilationDidComplete):
     60        * dfg/DFGToFTLForOSREntryDeferredCompilationCallback.h:
     61        * ftl/FTLOSREntry.cpp:
     62        (JSC::FTL::prepareOSREntry):
     63        * ftl/FTLOSREntry.h:
     64        * jit/JITOperations.cpp:
     65
    1662017-01-25  Saam Barati  <sbarati@apple.com>
    267

  • trunk/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj

    r211069 r211224  
    20602060                ADE8029C1E08F1DE0058DE78 /* WebAssemblyLinkErrorPrototype.h in Headers */ = {isa = PBXBuildFile; fileRef = ADE802971E08F1C90058DE78 /* WebAssemblyLinkErrorPrototype.h */; settings = {ATTRIBUTES = (Private, ); }; };
    20612061                ADE8029E1E08F2280058DE78 /* WebAssemblyLinkErrorConstructor.cpp in Sources */ = {isa = PBXBuildFile; fileRef = ADE8029D1E08F2260058DE78 /* WebAssemblyLinkErrorConstructor.cpp */; };
     2062                ADFF2F701E319DE3001EA54E /* DFGTierUpEntryTrigger.h in Headers */ = {isa = PBXBuildFile; fileRef = ADFF2F6F1E319DD0001EA54E /* DFGTierUpEntryTrigger.h */; };
    20622063                B59F89391891F29F00D5CCDC /* UnlinkedInstructionStream.cpp in Sources */ = {isa = PBXBuildFile; fileRef = B59F89381891ADB500D5CCDC /* UnlinkedInstructionStream.cpp */; };
    20632064                BC02E90D0E1839DB000F9297 /* ErrorConstructor.h in Headers */ = {isa = PBXBuildFile; fileRef = BC02E9050E1839DB000F9297 /* ErrorConstructor.h */; };
    … …  
    45724573                ADE802971E08F1C90058DE78 /* WebAssemblyLinkErrorPrototype.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = WebAssemblyLinkErrorPrototype.h; path = js/WebAssemblyLinkErrorPrototype.h; sourceTree = "<group>"; };
    45734574                ADE8029D1E08F2260058DE78 /* WebAssemblyLinkErrorConstructor.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = WebAssemblyLinkErrorConstructor.cpp; path = js/WebAssemblyLinkErrorConstructor.cpp; sourceTree = "<group>"; };
     4575                ADFF2F6F1E319DD0001EA54E /* DFGTierUpEntryTrigger.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = DFGTierUpEntryTrigger.h; path = dfg/DFGTierUpEntryTrigger.h; sourceTree = "<group>"; };
    45744576                B59F89371891AD3300D5CCDC /* UnlinkedInstructionStream.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = UnlinkedInstructionStream.h; sourceTree = "<group>"; };
    45754577                B59F89381891ADB500D5CCDC /* UnlinkedInstructionStream.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = UnlinkedInstructionStream.cpp; sourceTree = "<group>"; };
    … …  
    71307132                                0FD8A31F17D51F5700CA2C40 /* DFGTierUpCheckInjectionPhase.cpp */,
    71317133                                0FD8A32017D51F5700CA2C40 /* DFGTierUpCheckInjectionPhase.h */,
     7134                                ADFF2F6F1E319DD0001EA54E /* DFGTierUpEntryTrigger.h */,
    71327135                                0FD8A32117D51F5700CA2C40 /* DFGToFTLDeferredCompilationCallback.cpp */,
    71337136                                0FD8A32217D51F5700CA2C40 /* DFGToFTLDeferredCompilationCallback.h */,
    … …  
    88128815                                A1A009C01831A22D00CF8711 /* MacroAssemblerARM64.h in Headers */,
    88138816                                86ADD1460FDDEA980006EEC2 /* MacroAssemblerARMv7.h in Headers */,
     8817                                ADFF2F701E319DE3001EA54E /* DFGTierUpEntryTrigger.h in Headers */,
    88148818                                863B23E00FC6118900703AA4 /* MacroAssemblerCodeRef.h in Headers */,
    88158819                                E32AB2441DCD75F400D7533A /* MacroAssemblerHelpers.h in Headers */,

  • trunk/Source/JavaScriptCore/dfg/DFGJITCode.h

    r208985 r211224  
    3434#include "DFGOSREntry.h"
    3535#include "DFGOSRExit.h"
     36#include "DFGTierUpEntryTrigger.h"
    3637#include "DFGVariableEventStream.h"
    3738#include "ExecutionCounter.h"
    … …  
    155156    // This can never be modified after it has been initialized since the addresses of the triggers
    156157    // are used by the JIT.
    157     HashMap<unsigned, uint8_t> tierUpEntryTriggers;
     158    HashMap<unsigned, TierUpEntryTrigger> tierUpEntryTriggers;
    158159
    159160    // Set of bytecode that were the target of a TierUp operation.

  • trunk/Source/JavaScriptCore/dfg/DFGJITCompiler.cpp

    r210023 r211224  
    6161    m_jitCode->tierUpInLoopHierarchy = WTFMove(m_graph.m_plan.tierUpInLoopHierarchy);
    6262    for (unsigned tierUpBytecode : m_graph.m_plan.tierUpAndOSREnterBytecodes)
    63         m_jitCode->tierUpEntryTriggers.add(tierUpBytecode, 0);
     63        m_jitCode->tierUpEntryTriggers.add(tierUpBytecode, TierUpEntryTrigger::None);
    6464#endif
    6565}

  • trunk/Source/JavaScriptCore/dfg/DFGOSREntry.cpp

    r203356 r211224  
    9292
    9393SUPPRESS_ASAN
    94 void* prepareOSREntry(ExecState* exec, CodeBlock* codeBlock, unsigned bytecodeIndex)
     94Expected<void*, OSREntryFail> prepareOSREntry(ExecState* exec, CodeBlock* codeBlock, unsigned bytecodeIndex)
    9595{
    9696    ASSERT(JITCode::isOptimizingJIT(codeBlock->jitType()));
    … …  
    100100
    101101    if (!Options::useOSREntryToDFG())
    102         return 0;
     102        return makeUnexpected(OSREntryFail::Disabled);
    103103
    104104    if (Options::verboseOSR()) {
    … …  
    137137        if (Options::verboseOSR())
    138138            dataLog("    OSR failed because the target code block is not DFG.\n");
    139         return 0;
     139        return makeUnexpected(OSREntryFail::TargetNotDFG);
    140140    }
    141141   
    … …  
    146146        if (Options::verboseOSR())
    147147            dataLogF("    OSR failed because the entrypoint was optimized out.\n");
    148         return 0;
     148        return makeUnexpected(OSREntryFail::TargetOptimizedOut);
    149149    }
    150150   
    … …  
    182182                dataLogF(".\n");
    183183            }
    184             return 0;
     184            return makeUnexpected(OSREntryFail::BadPrediction);
    185185        }
    186186       
    … …  
    197197                    ", expected ", entry->m_expectedValues.argument(argument), ".\n");
    198198            }
    199             return 0;
     199            return makeUnexpected(OSREntryFail::BadPrediction);
    200200        }
    201201    }
    … …  
    210210                        exec->registers()[localOffset].asanUnsafeJSValue(), ", expected number.\n");
    211211                }
    212                 return 0;
     212                return makeUnexpected(OSREntryFail::BadPrediction);
    213213            }
    214214            continue;
    … …  
    222222                        "machine int.\n");
    223223                }
    224                 return 0;
     224                return makeUnexpected(OSREntryFail::BadPrediction);
    225225            }
    226226            continue;
    … …  
    233233                    entry->m_expectedValues.local(local), ".\n");
    234234            }
    235             return 0;
     235            return makeUnexpected(OSREntryFail::BadPrediction);
    236236        }
    237237    }
    … …  
    248248        if (Options::verboseOSR())
    249249            dataLogF("    OSR failed because stack growth failed.\n");
    250         return 0;
     250        return makeUnexpected(OSREntryFail::StackGrowthFailed);
    251251    }
    252252   

  • trunk/Source/JavaScriptCore/dfg/DFGOSREntry.h

    r206525 r211224  
    2929#include "Operands.h"
    3030#include <wtf/BitVector.h>
     31#include <wtf/Expected.h>
    3132
    3233namespace JSC {
    … …  
    3637
    3738namespace DFG {
     39
     40enum class OSREntryFail {
     41    Disabled,
     42    TargetNotDFG,
     43    TargetOptimizedOut,
     44    BadPrediction,
     45    StackGrowthFailed,
     46};
    3847
    3948#if ENABLE(DFG_JIT)
    … …  
    7079}
    7180
    72 // Returns a pointer to a data buffer that the OSR entry thunk will recognize and
    73 // parse. If this returns null, it means
    74 void* prepareOSREntry(ExecState*, CodeBlock*, unsigned bytecodeIndex);
     81// Returns a pointer to a data buffer that the OSR entry thunk will recognize and parse.
     82Expected<void*, OSREntryFail> prepareOSREntry(ExecState*, CodeBlock*, unsigned bytecodeIndex);
    7583#else
    76 inline void* prepareOSREntry(ExecState*, CodeBlock*, unsigned) { return 0; }
     84inline Expected<void*, OSREntryFail> prepareOSREntry(ExecState*, CodeBlock*, unsigned) { return makeUnexpected(OSREntryFail::Disabled); }
    7785#endif
    7886

  • trunk/Source/JavaScriptCore/dfg/DFGOperations.cpp

    r211110 r211224  
    23152315    }
    23162316
     2317    // This updates the execution counter.
    23172318    if (shouldTriggerFTLCompile(codeBlock, jitCode))
    23182319        triggerFTLReplacementCompile(vm, codeBlock, jitCode);
    … …  
    23382339}
    23392340
    2340 static char* tierUpCommon(ExecState* exec, unsigned originBytecodeIndex, unsigned osrEntryBytecodeIndex)
    2341 {
    2342     VM* vm = &exec->vm();
    2343     CodeBlock* codeBlock = exec->codeBlock();
     2341enum class EntryReason {
     2342    Spurious,
     2343    CheckingUpOnHowCompilationIsGoing,
     2344    HaveOSREntryReady,
     2345    ShouldStartCompiling,
     2346    ShouldStartCompilingRightNow,
     2347    CompilationFailed,
     2348};
     2349
     2350static EntryReason whatHaveYouDoneAndWhyAmIHere(VM* vm, CodeBlock* codeBlock, JITCode* jitCode, unsigned bytecodeIndex, CodeBlock*& osrEntryBlock)
     2351{
     2352    // Gather facts about why we could be here.
    23442353
    23452354    // Resolve any pending plan for OSR Enter on this function.
    23462355    Worklist::State worklistState;
    2347     if (Worklist* worklist = existingGlobalFTLWorklistOrNull()) {
    2348         worklistState = worklist->completeAllReadyPlansForVM(
    2349             *vm, CompilationKey(codeBlock->baselineVersion(), FTLForOSREntryMode));
    2350     } else
     2356    if (Worklist* worklist = existingGlobalFTLWorklistOrNull())
     2357        worklistState = worklist->completeAllReadyPlansForVM(*vm, CompilationKey(codeBlock->baselineVersion(), FTLForOSREntryMode));
     2358    else
    23512359        worklistState = Worklist::NotKnown;
    23522360
    2353     JITCode* jitCode = codeBlock->jitCode()->dfg();
    2354     if (worklistState == Worklist::Compiling) {
     2361    osrEntryBlock = jitCode->osrEntryBlock();
     2362
     2363    // Was the tier-up entry trigger set to slow-path?
     2364    bool triggeredSlowPath = false;
     2365    auto tierUpEntryTriggers = jitCode->tierUpEntryTriggers.find(bytecodeIndex);
     2366    if (tierUpEntryTriggers != jitCode->tierUpEntryTriggers.end()) {
     2367        if (tierUpEntryTriggers->value == TierUpEntryTrigger::TakeSlowPath) {
     2368            // We were asked to enter as soon as possible. Unset this trigger so we don't continually enter.
     2369            if (Options::verboseOSR())
     2370                dataLog("EntryTrigger for ", *codeBlock, " forced slow-path.\n");
     2371            triggeredSlowPath = true;
     2372            tierUpEntryTriggers->value = TierUpEntryTrigger::None;
     2373        }
     2374    }
     2375
     2376    // Put those facts together to make a final determination.
     2377
     2378    switch (worklistState) {
     2379    case Worklist::NotKnown:
     2380        if (osrEntryBlock)
     2381            return EntryReason::HaveOSREntryReady;
     2382        if (triggeredSlowPath) {
     2383            // Someone went through the trouble of forcing us into slow-path, they really wanted us to compile.
     2384            return EntryReason::ShouldStartCompilingRightNow;
     2385        }
     2386        return EntryReason::ShouldStartCompiling;
     2387
     2388    case Worklist::Compiling:
     2389        if (triggeredSlowPath) {
     2390            // We're already compiling, and can't OSR enter. We therefore must
     2391            // have set our slow-path trigger as well as another slow path
     2392            // trigger, hoping one of these would kick off a compilation. Sure
     2393            // enough another bytecode location did kick off a compilation
     2394            // before this one got a chance to do so. There's nothing for us to
     2395            // do but wait.
     2396            return EntryReason::Spurious;
     2397        }
     2398        return EntryReason::CheckingUpOnHowCompilationIsGoing;
     2399
     2400    case Worklist::Compiled:
     2401        return EntryReason::CompilationFailed;
     2402    }
     2403
     2404    RELEASE_ASSERT_NOT_REACHED();
     2405    return EntryReason::Spurious;
     2406}
     2407
     2408enum class CanOSREnterFromHere {
     2409    No,
     2410    Yes,
     2411};
     2412
     2413static char* tierUpCommon(VM* vm, ExecState* exec, CodeBlock* codeBlock, JITCode* jitCode, unsigned bytecodeIndex, CanOSREnterFromHere canOSREnterFromHere)
     2414{
     2415    CodeBlock* osrEntryBlock;
     2416    EntryReason entryReason = whatHaveYouDoneAndWhyAmIHere(vm, codeBlock, jitCode, bytecodeIndex, osrEntryBlock);
     2417
     2418    switch (entryReason) {
     2419    case EntryReason::CheckingUpOnHowCompilationIsGoing:
    23552420        CODEBLOCK_LOG_EVENT(codeBlock, "delayFTLCompile", ("still compiling"));
    23562421        jitCode->setOptimizationThresholdBasedOnCompilationResult(
    23572422            codeBlock, CompilationDeferred);
    23582423        return nullptr;
    2359     }
    2360 
    2361     if (worklistState == Worklist::Compiled) {
     2424
     2425    case EntryReason::CompilationFailed:
    23622426        CODEBLOCK_LOG_EVENT(codeBlock, "delayFTLCompile", ("compiled and failed"));
    2363         // This means that compilation failed and we already set the thresholds.
     2427        // We've already set the thresholds.
    23642428        if (Options::verboseOSR())
    23652429            dataLog("Code block ", *codeBlock, " was compiled but it doesn't have an optimized replacement.\n");
    23662430        return nullptr;
    2367     }
    2368 
    2369     // If we can OSR Enter, do it right away.
    2370     if (originBytecodeIndex == osrEntryBytecodeIndex) {
    2371         unsigned streamIndex = jitCode->bytecodeIndexToStreamIndex.get(originBytecodeIndex);
    2372         if (CodeBlock* entryBlock = jitCode->osrEntryBlock()) {
    2373             if (void* address = FTL::prepareOSREntry(exec, codeBlock, entryBlock, originBytecodeIndex, streamIndex)) {
    2374                 CODEBLOCK_LOG_EVENT(entryBlock, "osrEntry", ("at bc#", originBytecodeIndex));
     2431
     2432    case EntryReason::HaveOSREntryReady:
     2433        if (canOSREnterFromHere == CanOSREnterFromHere::No)
     2434            break; // OSR entry is for another location.
     2435
     2436        {
     2437            // If we can OSR Enter, do it right away.
     2438            unsigned streamIndex = jitCode->bytecodeIndexToStreamIndex.get(bytecodeIndex);
     2439            auto osrEntryPreparation = FTL::prepareOSREntry(exec, codeBlock, osrEntryBlock, bytecodeIndex, streamIndex);
     2440            if (osrEntryPreparation) {
     2441                CODEBLOCK_LOG_EVENT(osrEntryBlock, "osrEntry", ("at bc#", bytecodeIndex));
     2442                void* address = osrEntryPreparation.value();
     2443                ASSERT(address);
    23752444                return static_cast<char*>(address);
    23762445            }
     2446            switch (osrEntryPreparation.error()) {
     2447            case FTL::OSREntryFail::StackGrowthFailed:
     2448                break;
     2449            case FTL::OSREntryFail::WrongBytecode:
     2450                // Above we checked that an entry was possible from the current
     2451                // location, but we didn't know whether the compiled OSR entry
     2452                // was for this location. Now we know it's not.
     2453                break;
     2454            }
     2455
     2456            break;
    23772457        }
     2458
     2459    case EntryReason::ShouldStartCompiling:
     2460    case EntryReason::ShouldStartCompilingRightNow:
     2461        // We'll take care of that below.
     2462        break;
     2463
     2464    case EntryReason::Spurious:
     2465        if (Options::verboseOSR())
     2466            dataLog("Code block ", *codeBlock, " was spuriously woken up, compilation is already under way.\n");
     2467        jitCode->checkIfOptimizationThresholdReached(codeBlock);
     2468        return nullptr;
    23782469    }
    23792470
    … …  
    23822473    // - If we couldn't enter for a while, then trigger OSR entry.
    23832474
     2475    // This updates the execution counter.
    23842476    if (!shouldTriggerFTLCompile(codeBlock, jitCode))
    23852477        return nullptr;
    … …  
    23992491        CODEBLOCK_LOG_EVENT(codeBlock, "delayFTLCompile", ("avoiding replacement compile"));
    24002492
    2401     // It's time to try to compile code for OSR entry.
    2402     if (CodeBlock* entryBlock = jitCode->osrEntryBlock()) {
     2493    if (osrEntryBlock) {
     2494        // We have a compiled OSR entry for this function, but didn't enter.
     2495
    24032496        if (jitCode->osrEntryRetry < Options::ftlOSREntryRetryThreshold()) {
    24042497            CODEBLOCK_LOG_EVENT(codeBlock, "delayFTLCompile", ("OSR entry failed, OSR entry threshold not met"));
    … …  
    24092502        }
    24102503
    2411         FTL::ForOSREntryJITCode* entryCode = entryBlock->jitCode()->ftlForOSREntry();
     2504        FTL::ForOSREntryJITCode* entryCode = osrEntryBlock->jitCode()->ftlForOSREntry();
    24122505        entryCode->countEntryFailure();
    24132506        if (entryCode->entryFailureCount() <
    … …  
    24222515        // without exponential backoff and we only do this for the entry code block.
    24232516        CODEBLOCK_LOG_EVENT(codeBlock, "delayFTLCompile", ("OSR entry failed too many times"));
    2424         unsigned osrEntryBytecode = entryBlock->jitCode()->ftlForOSREntry()->bytecodeIndex();
     2517        unsigned osrEntryBytecode = osrEntryBlock->jitCode()->ftlForOSREntry()->bytecodeIndex();
    24252518        jitCode->clearOSREntryBlock();
    24262519        jitCode->osrEntryRetry = 0;
    2427         jitCode->tierUpEntryTriggers.set(osrEntryBytecode, 0);
     2520        jitCode->tierUpEntryTriggers.set(osrEntryBytecode, TierUpEntryTrigger::None);
    24282521        jitCode->setOptimizationThresholdBasedOnCompilationResult(
    24292522            codeBlock, CompilationDeferred);
    … …  
    24312524    }
    24322525
    2433     unsigned streamIndex = jitCode->bytecodeIndexToStreamIndex.get(osrEntryBytecodeIndex);
    2434     auto tierUpHierarchyEntry = jitCode->tierUpInLoopHierarchy.find(osrEntryBytecodeIndex);
    2435     if (tierUpHierarchyEntry != jitCode->tierUpInLoopHierarchy.end()) {
    2436         for (unsigned osrEntryCandidate : tierUpHierarchyEntry->value) {
    2437             if (jitCode->tierUpEntrySeen.contains(osrEntryCandidate)) {
    2438                 osrEntryBytecodeIndex = osrEntryCandidate;
    2439                 streamIndex = jitCode->bytecodeIndexToStreamIndex.get(osrEntryBytecodeIndex);
     2526    // We aren't compiling and haven't compiled anything for OSR entry. So, try to compile something.
     2527
     2528    if (entryReason != EntryReason::ShouldStartCompilingRightNow) {
     2529        // We entered because our threshold was set, not because someone is forcing us to compile.
     2530        // Try to see if there would be a better place to compile from than where we currently are.
     2531
     2532        // Compiling an outer-loop for OSR entry often generates better code than an inner-loop because
     2533        // the entry is less disruptive. If we're at an inner-loop be smart and mark the outer-loop as
     2534        // needing compilation ASAP. Once execution reaches the outer-loop compilation will trigger.
     2535        auto tierUpHierarchyEntry = jitCode->tierUpInLoopHierarchy.find(bytecodeIndex);
     2536        if (tierUpHierarchyEntry != jitCode->tierUpInLoopHierarchy.end() && !tierUpHierarchyEntry->value.isEmpty()) {
     2537            // Traverse the loop hierarchy from the outer-most loop, to the inner-most one.
     2538            for (auto iterator = tierUpHierarchyEntry->value.rbegin(), end = tierUpHierarchyEntry->value.rend(); iterator != end; ++iterator) {
     2539                unsigned osrEntryCandidate = *iterator;
     2540                if (jitCode->tierUpEntrySeen.contains(osrEntryCandidate)) {
     2541                    unsigned outerLoopOsrEntryBytecodeIndex = osrEntryCandidate;
     2542
     2543                    // We found an outer-loop which would be a better OSR entry
     2544                    // than the current location:
     2545                    //  - Set its trigger so it takes the slow-path ASAP;
     2546                    //  - Tell the codeblock to stop its counter slow-path for a
     2547                    //    while, because it should wait for the outer-loop to
     2548                    //    trigger.
     2549                    //
     2550                    // If we slow-path again then one of these is true:
     2551                    //  1. We enter from the outer-loop because of its trigger,
     2552                    //     it un-sets its trigger, kicks off a compile,
     2553                    //     everything is good;
     2554                    //  2. We enter from anywhere and a compile is under way,
     2555                    //     just chill as usual;
     2556                    //  3. We never got to that outer-loop and the counter
     2557                    //     tripped again, bummer!
     2558                    //
     2559                    // We can detect 3. because the outer-loop's trigger is
     2560                    // set. It's still a great place to enter, so leave its
     2561                    // trigger set, but also:
     2562                    //  - Set the trigger for the next loop in, hope that one triggers;
     2563                    //  - Backoff the counter as before.
     2564                    // That is, unless all the inner-loop's parent triggers are
     2565                    // set. In that case just optimize the innermost-loop: it
     2566                    // won't generate as good code, but the outer-loops aren't
     2567                    // triggering so we may as we tier up where we can.
     2568                    auto outerLoopTierUpEntryTriggers = jitCode->tierUpEntryTriggers.find(outerLoopOsrEntryBytecodeIndex);
     2569                    ASSERT(outerLoopTierUpEntryTriggers != jitCode->tierUpEntryTriggers.end());
     2570
     2571                    if (outerLoopTierUpEntryTriggers->value != TierUpEntryTrigger::TakeSlowPath) {
     2572                        if (Options::verboseOSR())
     2573                            dataLog("Forcibly FTL-optimize outer-loop bc#", outerLoopOsrEntryBytecodeIndex, " in ", *codeBlock, "by setting its trigger.\n");
     2574
     2575                        CODEBLOCK_LOG_EVENT(codeBlock, "delayFTLCompile", ("OSR entry request from inner-loop for outer-loop"));
     2576                        jitCode->tierUpEntryTriggers.set(outerLoopOsrEntryBytecodeIndex, TierUpEntryTrigger::TakeSlowPath);
     2577                        jitCode->optimizeSoon(codeBlock);
     2578                        return nullptr;
     2579                    }
     2580
     2581                    if (Options::verboseOSR())
     2582                        dataLog("Trying to forcibly FTL-optimize outer-loop bc#", outerLoopOsrEntryBytecodeIndex, " in ", *codeBlock, ", but trigger is already set.\n");
     2583                }
    24402584            }
     2585
     2586            // All the outer-loops have their trigger set, but none have been entered.
     2587
     2588            if (canOSREnterFromHere == CanOSREnterFromHere::No) {
     2589                // We just can't enter from here, we've asked everyone we know
     2590                // to please optimize ASAP, nobody has heeded our call. Keep
     2591                // hoping one of the outer loops triggers get to their slow
     2592                // path. Stop the counter, there's nothing we can do.
     2593                jitCode->dontOptimizeAnytimeSoon(codeBlock);
     2594                return nullptr;
     2595            }
     2596
     2597            if (Options::verboseOSR())
     2598                dataLog("Tried to forcibly-optimize outer-loop, but falling back to bc#", bytecodeIndex, " in ", *codeBlock, ", outer-loop triggers didn't work.\n");
    24412599        }
    24422600    }
    24432601
    2444     // We aren't compiling and haven't compiled anything for OSR entry. So, try to compile
    2445     // something.
    2446     auto triggerIterator = jitCode->tierUpEntryTriggers.find(osrEntryBytecodeIndex);
     2602    RELEASE_ASSERT(canOSREnterFromHere == CanOSREnterFromHere::Yes);
     2603
     2604    unsigned streamIndex = jitCode->bytecodeIndexToStreamIndex.get(bytecodeIndex);
     2605
     2606    // We're not trying to kick off a compile of an outer-loop from within an
     2607    // inner-loop. We can compile right here, right now.
     2608    auto triggerIterator = jitCode->tierUpEntryTriggers.find(bytecodeIndex);
    24472609    RELEASE_ASSERT(triggerIterator != jitCode->tierUpEntryTriggers.end());
    2448     uint8_t* triggerAddress = &(triggerIterator->value);
    2449 
     2610    TierUpEntryTrigger* triggerAddress = &(triggerIterator->value);
     2611
     2612    // Use OSR reconstruction to generate a snapshot of JSValues at the current
     2613    // location of execution. The optimizer is happy when it can look at real
     2614    // and live values, as opposed to mere type traces.
    24502615    Operands<JSValue> mustHandleValues;
    24512616    jitCode->reconstruct(
    2452         exec, codeBlock, CodeOrigin(osrEntryBytecodeIndex), streamIndex, mustHandleValues);
     2617        exec, codeBlock, CodeOrigin(bytecodeIndex), streamIndex, mustHandleValues);
    24532618    CodeBlock* replacementCodeBlock = codeBlock->newReplacement();
    24542619
    24552620    CODEBLOCK_LOG_EVENT(codeBlock, "triggerFTLOSR", ());
    24562621    CompilationResult forEntryResult = compile(
    2457         *vm, replacementCodeBlock, codeBlock, FTLForOSREntryMode, osrEntryBytecodeIndex,
     2622        *vm, replacementCodeBlock, codeBlock, FTLForOSREntryMode, bytecodeIndex,
    24582623        mustHandleValues, ToFTLForOSREntryDeferredCompilationCallback::create(triggerAddress));
    24592624
    … …  
    24672632        return nullptr;
    24682633    }
    2469    
    2470     CODEBLOCK_LOG_EVENT(jitCode->osrEntryBlock(), "osrEntry", ("at bc#", originBytecodeIndex));
    2471     // It's possible that the for-entry compile already succeeded. In that case OSR
    2472     // entry will succeed unless we ran out of stack. It's not clear what we should do.
    2473     // We signal to try again after a while if that happens.
    2474     void* address = FTL::prepareOSREntry(
    2475         exec, codeBlock, jitCode->osrEntryBlock(), originBytecodeIndex, streamIndex);
    2476     return static_cast<char*>(address);
     2634
     2635    // It's possible that the for-entry compile already succeeded. In that case
     2636    // OSR entry will succeed unless we ran out of stack.
     2637    auto osrEntryPreparation = FTL::prepareOSREntry(exec, codeBlock, jitCode->osrEntryBlock(), bytecodeIndex, streamIndex);
     2638    if (osrEntryPreparation) {
     2639        CODEBLOCK_LOG_EVENT(jitCode->osrEntryBlock(), "osrEntry", ("at bc#", bytecodeIndex));
     2640        void* address = osrEntryPreparation.value();
     2641        ASSERT(address);
     2642        return static_cast<char*>(address);
     2643    }
     2644    switch (osrEntryPreparation.error()) {
     2645    case FTL::OSREntryFail::StackGrowthFailed:
     2646        // It's not clear what we should do. We signal to try again after a
     2647        // while if that happens.
     2648        return nullptr;
     2649    case FTL::OSREntryFail::WrongBytecode:
     2650        RELEASE_ASSERT_NOT_REACHED();
     2651    }
     2652
     2653    RELEASE_ASSERT_NOT_REACHED();
     2654    return nullptr;
    24772655}
    24782656
    … …  
    24972675    }
    24982676
     2677    // It's impossible to OSR enter from the current bytecode index: CheckTierUpInLoop is only even generated for non-OSR-entry bytecodes.
     2678    // It's nonetheless a good spot to request that a nearby loop get optimized the next time it's entered.
     2679
    24992680    auto tierUpHierarchyEntry = jitCode->tierUpInLoopHierarchy.find(bytecodeIndex);
    25002681    if (tierUpHierarchyEntry != jitCode->tierUpInLoopHierarchy.end()
    25012682        && !tierUpHierarchyEntry->value.isEmpty()) {
    2502         tierUpCommon(exec, bytecodeIndex, tierUpHierarchyEntry->value.first());
    2503     } else if (shouldTriggerFTLCompile(codeBlock, jitCode))
     2683        // There is a suitable loop to OSR enter from.
     2684        tierUpCommon(vm, exec, codeBlock, jitCode, bytecodeIndex, CanOSREnterFromHere::No);
     2685    } else if (shouldTriggerFTLCompile(codeBlock, jitCode)) // This updates the execution counter.
    25042686        triggerFTLReplacementCompile(vm, codeBlock, jitCode);
    25052687
    … …  
    25112693}
    25122694
    2513 char* JIT_OPERATION triggerOSREntryNow(ExecState* exec, unsigned bytecodeIndex)
     2695char* JIT_OPERATION checkTierUpAndOSREnterNow(ExecState* exec, unsigned bytecodeIndex)
    25142696{
    25152697    VM* vm = &exec->vm();
    … …  
    25242706
    25252707    JITCode* jitCode = codeBlock->jitCode()->dfg();
    2526     jitCode->tierUpEntrySeen.add(bytecodeIndex);
    25272708
    25282709    if (Options::verboseOSR()) {
    … …  
    25322713    }
    25332714
    2534     return tierUpCommon(exec, bytecodeIndex, bytecodeIndex);
     2715    jitCode->tierUpEntrySeen.add(bytecodeIndex);
     2716    return tierUpCommon(vm, exec, codeBlock, jitCode, bytecodeIndex, CanOSREnterFromHere::Yes);
    25352717}
    25362718

  • trunk/Source/JavaScriptCore/dfg/DFGOperations.h

    r209638 r211224  
    221221void JIT_OPERATION triggerTierUpNow(ExecState*) WTF_INTERNAL;
    222222void JIT_OPERATION triggerTierUpNowInLoop(ExecState*, unsigned bytecodeIndex) WTF_INTERNAL;
    223 char* JIT_OPERATION triggerOSREntryNow(ExecState*, unsigned bytecodeIndex) WTF_INTERNAL;
     223char* JIT_OPERATION checkTierUpAndOSREnterNow(ExecState*, unsigned bytecodeIndex) WTF_INTERNAL;
    224224#endif // ENABLE(FTL_JIT)
    225225

  • trunk/Source/JavaScriptCore/dfg/DFGSpeculativeJIT64.cpp

    r210844 r211224  
    57825782        auto triggerIterator = m_jit.jitCode()->tierUpEntryTriggers.find(bytecodeIndex);
    57835783        DFG_ASSERT(m_jit.graph(), node, triggerIterator != m_jit.jitCode()->tierUpEntryTriggers.end());
    5784         uint8_t* forceEntryTrigger = &(m_jit.jitCode()->tierUpEntryTriggers.find(bytecodeIndex)->value);
    5785 
     5784        TierUpEntryTrigger* forceEntryTrigger = &(m_jit.jitCode()->tierUpEntryTriggers.find(bytecodeIndex)->value);
     5785
     5786        static_assert(sizeof(TierUpEntryTrigger) == 1, "8-bit load is generated below");
     5787        static_assert(!static_cast<uint8_t>(TierUpEntryTrigger::None), "NonZero test below depends on this");
    57865788        MacroAssembler::Jump forceOSREntry = m_jit.branchTest8(MacroAssembler::NonZero, MacroAssembler::AbsoluteAddress(forceEntryTrigger));
    57875789        MacroAssembler::Jump overflowedCounter = m_jit.branchAdd32(
    … …  
    58035805            silentSpill(savePlans);
    58045806            m_jit.setupArgumentsWithExecState(TrustedImm32(bytecodeIndex));
    5805             appendCallSetResult(triggerOSREntryNow, tempGPR);
     5807            appendCallSetResult(checkTierUpAndOSREnterNow, tempGPR);
    58065808
    58075809            if (savePlans.isEmpty())

  • trunk/Source/JavaScriptCore/dfg/DFGTierUpEntryTrigger.h

    r211223 r211224  
    11/*
    2  * Copyright (C) 2013 Apple Inc. All rights reserved.
     2 * Copyright (C) 2017 Apple Inc. All rights reserved.
    33 *
    44 * Redistribution and use in source and binary forms, with or without
    … …  
    2626#pragma once
    2727
    28 #if ENABLE(FTL_JIT)
     28#if ENABLE(DFG_JIT)
    2929
    30 namespace JSC {
     30namespace JSC { namespace DFG {
    3131
    32 class CodeBlock;
    33 class ExecState;
     32enum class TierUpEntryTrigger : uint8_t {
     33    None = 0, // Must stay zero: JIT-compiled code takes the triggerOSREntryNow slow-path when non-zero.
     34    TakeSlowPath, // The slow path can be taken because the compilation needs to be started, or it's ready and we should OSR enter.
     35};
    3436
    35 namespace FTL {
     37} } // namespace JSC::DFG
    3638
    37 void* prepareOSREntry(
    38     ExecState*, CodeBlock* dfgCodeBlock, CodeBlock* entryCodeBlock, unsigned bytecodeIndex,
    39     unsigned streamIndex);
    40 
    41 } } // namespace JSC::FTL
    42 
    43 #endif // ENABLE(FTL_JIT)
     39#endif // ENABLE(DFG_JIT)

  • trunk/Source/JavaScriptCore/dfg/DFGToFTLForOSREntryDeferredCompilationCallback.cpp

    r208063 r211224  
    3636namespace JSC { namespace DFG {
    3737
    38 ToFTLForOSREntryDeferredCompilationCallback::ToFTLForOSREntryDeferredCompilationCallback(uint8_t* forcedOSREntryTrigger)
     38ToFTLForOSREntryDeferredCompilationCallback::ToFTLForOSREntryDeferredCompilationCallback(TierUpEntryTrigger* forcedOSREntryTrigger)
    3939    : m_forcedOSREntryTrigger(forcedOSREntryTrigger)
    4040{
    … …  
    4545}
    4646
    47 Ref<ToFTLForOSREntryDeferredCompilationCallback>ToFTLForOSREntryDeferredCompilationCallback::create(uint8_t* forcedOSREntryTrigger)
     47Ref<ToFTLForOSREntryDeferredCompilationCallback>ToFTLForOSREntryDeferredCompilationCallback::create(TierUpEntryTrigger* forcedOSREntryTrigger)
    4848{
    4949    return adoptRef(*new ToFTLForOSREntryDeferredCompilationCallback(forcedOSREntryTrigger));
    … …  
    5959    }
    6060
    61     *m_forcedOSREntryTrigger = 1;
     61    *m_forcedOSREntryTrigger = TierUpEntryTrigger::TakeSlowPath;
    6262}
    6363
    … …  
    7777        jitCode->setOSREntryBlock(*codeBlock->vm(), profiledDFGCodeBlock, codeBlock);
    7878        unsigned osrEntryBytecode = codeBlock->jitCode()->ftlForOSREntry()->bytecodeIndex();
    79         jitCode->tierUpEntryTriggers.set(osrEntryBytecode, 1);
     79        jitCode->tierUpEntryTriggers.set(osrEntryBytecode, TierUpEntryTrigger::TakeSlowPath);
    8080        break;
    8181    }

  • trunk/Source/JavaScriptCore/dfg/DFGToFTLForOSREntryDeferredCompilationCallback.h

    r206525 r211224  
    2828#if ENABLE(FTL_JIT)
    2929
     30#include "DFGTierUpEntryTrigger.h"
    3031#include "DeferredCompilationCallback.h"
    3132#include <wtf/RefPtr.h>
    … …  
    3940class ToFTLForOSREntryDeferredCompilationCallback : public DeferredCompilationCallback {
    4041protected:
    41     ToFTLForOSREntryDeferredCompilationCallback(uint8_t* forcedOSREntryTrigger);
     42    ToFTLForOSREntryDeferredCompilationCallback(TierUpEntryTrigger* forcedOSREntryTrigger);
    4243
    4344public:
    4445    virtual ~ToFTLForOSREntryDeferredCompilationCallback();
    4546
    46     static Ref<ToFTLForOSREntryDeferredCompilationCallback> create(uint8_t* forcedOSREntryTrigger);
     47    static Ref<ToFTLForOSREntryDeferredCompilationCallback> create(TierUpEntryTrigger* forcedOSREntryTrigger);
    4748   
    4849    virtual void compilationDidBecomeReadyAsynchronously(CodeBlock*, CodeBlock* profiledDFGCodeBlock);
    … …  
    5051
    5152private:
    52     uint8_t* m_forcedOSREntryTrigger;
     53    TierUpEntryTrigger* m_forcedOSREntryTrigger;
    5354};
    5455

  • trunk/Source/JavaScriptCore/ftl/FTLOSREntry.cpp

    r209764 r211224  
    4040
    4141SUPPRESS_ASAN
    42 void* prepareOSREntry(
     42Expected<void*, OSREntryFail> prepareOSREntry(
    4343    ExecState* exec, CodeBlock* dfgCodeBlock, CodeBlock* entryCodeBlock,
    4444    unsigned bytecodeIndex, unsigned streamIndex)
    … …  
    6262        if (Options::verboseOSR())
    6363            dataLog("    OSR failed because we don't have an entrypoint for bc#", bytecodeIndex, "; ours is for bc#", entryCode->bytecodeIndex(), "\n");
    64         return 0;
     64        return makeUnexpected(OSREntryFail::WrongBytecode);
    6565    }
    6666   
    … …  
    9696        if (Options::verboseOSR())
    9797            dataLog("    OSR failed because stack growth failed.\n");
    98         return 0;
     98        return makeUnexpected(OSREntryFail::StackGrowthFailed);
    9999    }
    100100   

  • trunk/Source/JavaScriptCore/ftl/FTLOSREntry.h

    r206525 r211224  
    2828#if ENABLE(FTL_JIT)
    2929
     30#include <wtf/Expected.h>
     31
    3032namespace JSC {
    3133
    … …  
    3537namespace FTL {
    3638
    37 void* prepareOSREntry(
     39enum class OSREntryFail {
     40    WrongBytecode,
     41    StackGrowthFailed,
     42};
     43
     44Expected<void*, OSREntryFail> prepareOSREntry(
    3845    ExecState*, CodeBlock* dfgCodeBlock, CodeBlock* entryCodeBlock, unsigned bytecodeIndex,
    3946    unsigned streamIndex);

  • trunk/Source/JavaScriptCore/jit/JITOperations.cpp

    r210565 r211224  
    14241424    ASSERT(JITCode::isOptimizingJIT(optimizedCodeBlock->jitType()));
    14251425   
    1426     if (void* dataBuffer = DFG::prepareOSREntry(exec, optimizedCodeBlock, bytecodeIndex)) {
     1426    if (auto osrEntryPreparation = DFG::prepareOSREntry(exec, optimizedCodeBlock, bytecodeIndex)) {
     1427        void* dataBuffer = osrEntryPreparation.value();
    14271428        CODEBLOCK_LOG_EVENT(optimizedCodeBlock, "osrEntry", ("at bc#", bytecodeIndex));
    14281429        if (Options::verboseOSR()) {
Note: See TracChangeset for help on using the changeset viewer.