Changeset 194331 in webkit

Timestamp:

Dec 21, 2015, 8:16:01 AM (11 years ago)

Author:

fpizlo@apple.com

Message:

B3->Air lowering incorrectly copy-propagates over ZExt32's
​https://bugs.webkit.org/show_bug.cgi?id=152365

Reviewed by Benjamin Poulain.

The instruction selector thinks that Value's that return Int32's are going to always be lowered
to instructions that zero-extend the destination. But this isn't actually true. If you have an
Add32 with a destination on the stack (i.e. spilled) then it only writes 4 bytes. Then, the
filler will load 8 bytes from the stack at the point of use. So, the use of the Add32 will see
garbage in the high bits.

The fact that the spiller chose to use 8 bytes for a Tmp that gets defined by an Add32 is a
pretty sad bug, but:

• It's entirely up to the spiller to decide how many bytes to use for a Tmp, since we do not ascribe a type to Tmps. We could ascribe types to Tmps, but then coalescing would become harder. Our goal is to fix the bug while still enabling coalescing in cases like "a[i]" where "i" is a 32-bit integer that is computed using operations that already do zero-extension.

• More broadly, it's strange that the instruction selector decides whether a Value will be lowered to something that zero-extends. That's too constraining, since the most optimal instruction selection might involve something that doesn't zero-extend in cases of spilling, so the zero-extension should only happen if it's actually needed. This means that we need to understand which Air instructions cause zero-extensions.

• If we know which Air instructions cause zero-extensions, then we don't need the instruction selector to copy-propagate ZExt32's. We have copy-propagation in Air thanks to the register allocator.

In fact, the register allocator is exactly where all of the pieces come together. It's there that
we want to know which operations zero-extend and which don't. It also wants to know how many bits
of a Tmp each instruction reads. Armed with that information, the register allocator can emit
more optimal spill code, use less stack space for spill slots, and coalesce Move32's. As a bonus,
on X86, it replaces Move's with Move32's whenever it can. On X86, Move32 is cheaper.

This fixes a crash bug in V8/encrypt. After fixing this, I only needed two minor fixes to get
V8/encrypt to run. We're about 10% behind LLVM on steady state throughput on this test. It
appears to be mostly due to excessive spilling caused by CCall slow paths. That's fixable: we
could make CCalls on slow paths use a variant of CCallSpecial that promises not to clobber any
registers, and then have it emit spill code around the call itself. LLVM probably gets this
optimization from its live range splitting.

I tried writing a regression test. The problem is that you need garbage on the stack for this to
work, and I didn't feel like writing a flaky test. It appears that running V8/encrypt will cover
this, so we do have coverage.

• CMakeLists.txt:
• JavaScriptCore.xcodeproj/project.pbxproj:
• assembler/AbstractMacroAssembler.h:

(JSC::isX86):
(JSC::isX86_64):
(JSC::optimizeForARMv7IDIVSupported):
(JSC::optimizeForX86):
(JSC::optimizeForX86_64):

• b3/B3LowerToAir.cpp:

(JSC::B3::Air::LowerToAir::highBitsAreZero):
(JSC::B3::Air::LowerToAir::shouldCopyPropagate):
(JSC::B3::Air::LowerToAir::lower):

• b3/B3PatchpointSpecial.cpp:

(JSC::B3::PatchpointSpecial::forEachArg):

• b3/B3StackmapSpecial.cpp:

(JSC::B3::StackmapSpecial::forEachArgImpl):

• b3/B3Value.h:
• b3/air/AirAllocateStack.cpp:

(JSC::B3::Air::allocateStack):

• b3/air/AirArg.cpp:

(WTF::printInternal):

• b3/air/AirArg.h:

(JSC::B3::Air::Arg::pointerWidth):
(JSC::B3::Air::Arg::isAnyUse):
(JSC::B3::Air::Arg::isColdUse):
(JSC::B3::Air::Arg::isEarlyUse):
(JSC::B3::Air::Arg::isDef):
(JSC::B3::Air::Arg::isZDef):
(JSC::B3::Air::Arg::widthForB3Type):
(JSC::B3::Air::Arg::conservativeWidth):
(JSC::B3::Air::Arg::minimumWidth):
(JSC::B3::Air::Arg::bytes):
(JSC::B3::Air::Arg::widthForBytes):
(JSC::B3::Air::Arg::Arg):
(JSC::B3::Air::Arg::forEachTmp):

• b3/air/AirCCallSpecial.cpp:

(JSC::B3::Air::CCallSpecial::forEachArg):

• b3/air/AirEliminateDeadCode.cpp:

(JSC::B3::Air::eliminateDeadCode):

• b3/air/AirFixPartialRegisterStalls.cpp:

(JSC::B3::Air::fixPartialRegisterStalls):

• b3/air/AirInst.cpp:

(JSC::B3::Air::Inst::hasArgEffects):

• b3/air/AirInst.h:

(JSC::B3::Air::Inst::forEachTmpFast):
(JSC::B3::Air::Inst::forEachTmp):

• b3/air/AirInstInlines.h:

(JSC::B3::Air::Inst::forEachTmpWithExtraClobberedRegs):

• b3/air/AirIteratedRegisterCoalescing.cpp:
• b3/air/AirLiveness.h:

(JSC::B3::Air::AbstractLiveness::AbstractLiveness):
(JSC::B3::Air::AbstractLiveness::LocalCalc::execute):

• b3/air/AirOpcode.opcodes:
• b3/air/AirSpillEverything.cpp:

(JSC::B3::Air::spillEverything):

• b3/air/AirTmpWidth.cpp: Added.

(JSC::B3::Air::TmpWidth::TmpWidth):
(JSC::B3::Air::TmpWidth::~TmpWidth):

• b3/air/AirTmpWidth.h: Added.

(JSC::B3::Air::TmpWidth::width):
(JSC::B3::Air::TmpWidth::defWidth):
(JSC::B3::Air::TmpWidth::useWidth):
(JSC::B3::Air::TmpWidth::Widths::Widths):

• b3/air/AirUseCounts.h:

(JSC::B3::Air::UseCounts::UseCounts):

• b3/air/opcode_generator.rb:
• b3/testb3.cpp:

(JSC::B3::testCheckMegaCombo):
(JSC::B3::testCheckTrickyMegaCombo):
(JSC::B3::testCheckTwoMegaCombos):
(JSC::B3::run):

Location:

trunk/Source/JavaScriptCore

Files:

• CMakeLists.txt (modified) (1 diff)
• ChangeLog (modified) (1 diff)
• JavaScriptCore.xcodeproj/project.pbxproj (modified) (5 diffs)
• assembler/AbstractMacroAssembler.h (modified) (2 diffs)
• b3/B3LowerToAir.cpp (modified) (3 diffs)
• b3/B3PatchpointSpecial.cpp (modified) (1 diff)
• b3/B3StackmapSpecial.cpp (modified) (1 diff)
• b3/B3Value.h (modified) (1 diff)
• b3/air/AirAllocateStack.cpp (modified) (2 diffs)
• b3/air/AirArg.cpp (modified) (1 diff)
• b3/air/AirArg.h (modified) (8 diffs)
• b3/air/AirCCallSpecial.cpp (modified) (1 diff)
• b3/air/AirEliminateDeadCode.cpp (modified) (1 diff)
• b3/air/AirFixPartialRegisterStalls.cpp (modified) (2 diffs)
• b3/air/AirInst.cpp (modified) (1 diff)
• b3/air/AirInst.h (modified) (1 diff)
• b3/air/AirInstInlines.h (modified) (3 diffs)
• b3/air/AirIteratedRegisterCoalescing.cpp (modified) (19 diffs)
• b3/air/AirLiveness.h (modified) (4 diffs)
• b3/air/AirOpcode.opcodes (modified) (20 diffs)
• b3/air/AirOptimizeBlockOrder.cpp (modified) (1 diff)
• b3/air/AirSpillEverything.cpp (modified) (4 diffs)
• b3/air/AirTmpWidth.cpp (added)
• b3/air/AirTmpWidth.h (added)
• b3/air/AirUseCounts.h (modified) (1 diff)
• b3/air/opcode_generator.rb (modified) (8 diffs)
• b3/testb3.cpp (modified) (2 diffs)
• ftl/FTLLowerDFGToLLVM.cpp (modified) (1 diff)
• ftl/FTLOSRExitHandle.cpp (modified) (1 diff)

trunk/Source/JavaScriptCore/CMakeLists.txt

@@ -91 +91 @@
     b3/air/AirStackSlot.cpp
     b3/air/AirTmp.cpp
+    b3/air/AirTmpWidth.cpp
     b3/air/AirValidate.cpp
 

trunk/Source/JavaScriptCore/ChangeLog

@@ +1 @@
+2015-12-21  Filip Pizlo  <fpizlo@apple.com>
+
+        B3->Air lowering incorrectly copy-propagates over ZExt32's
+        https://bugs.webkit.org/show_bug.cgi?id=152365
+
+        Reviewed by Benjamin Poulain.
+
+        The instruction selector thinks that Value's that return Int32's are going to always be lowered
+        to instructions that zero-extend the destination. But this isn't actually true. If you have an
+        Add32 with a destination on the stack (i.e. spilled) then it only writes 4 bytes. Then, the
+        filler will load 8 bytes from the stack at the point of use. So, the use of the Add32 will see
+        garbage in the high bits.
+
+        The fact that the spiller chose to use 8 bytes for a Tmp that gets defined by an Add32 is a
+        pretty sad bug, but:
+
+        - It's entirely up to the spiller to decide how many bytes to use for a Tmp, since we do not
+          ascribe a type to Tmps. We could ascribe types to Tmps, but then coalescing would become
+          harder. Our goal is to fix the bug while still enabling coalescing in cases like "a[i]" where
+          "i" is a 32-bit integer that is computed using operations that already do zero-extension.
+
+        - More broadly, it's strange that the instruction selector decides whether a Value will be
+          lowered to something that zero-extends. That's too constraining, since the most optimal
+          instruction selection might involve something that doesn't zero-extend in cases of spilling, so
+          the zero-extension should only happen if it's actually needed. This means that we need to
+          understand which Air instructions cause zero-extensions.
+
+        - If we know which Air instructions cause zero-extensions, then we don't need the instruction
+          selector to copy-propagate ZExt32's. We have copy-propagation in Air thanks to the register
+          allocator.
+
+        In fact, the register allocator is exactly where all of the pieces come together. It's there that
+        we want to know which operations zero-extend and which don't. It also wants to know how many bits
+        of a Tmp each instruction reads. Armed with that information, the register allocator can emit
+        more optimal spill code, use less stack space for spill slots, and coalesce Move32's. As a bonus,
+        on X86, it replaces Move's with Move32's whenever it can. On X86, Move32 is cheaper.
+
+        This fixes a crash bug in V8/encrypt. After fixing this, I only needed two minor fixes to get
+        V8/encrypt to run. We're about 10% behind LLVM on steady state throughput on this test. It
+        appears to be mostly due to excessive spilling caused by CCall slow paths. That's fixable: we
+        could make CCalls on slow paths use a variant of CCallSpecial that promises not to clobber any
+        registers, and then have it emit spill code around the call itself. LLVM probably gets this
+        optimization from its live range splitting.
+
+        I tried writing a regression test. The problem is that you need garbage on the stack for this to
+        work, and I didn't feel like writing a flaky test. It appears that running V8/encrypt will cover
+        this, so we do have coverage.
+
+        * CMakeLists.txt:
+        * JavaScriptCore.xcodeproj/project.pbxproj:
+        * assembler/AbstractMacroAssembler.h:
+        (JSC::isX86):
+        (JSC::isX86_64):
+        (JSC::optimizeForARMv7IDIVSupported):
+        (JSC::optimizeForX86):
+        (JSC::optimizeForX86_64):
+        * b3/B3LowerToAir.cpp:
+        (JSC::B3::Air::LowerToAir::highBitsAreZero):
+        (JSC::B3::Air::LowerToAir::shouldCopyPropagate):
+        (JSC::B3::Air::LowerToAir::lower):
+        * b3/B3PatchpointSpecial.cpp:
+        (JSC::B3::PatchpointSpecial::forEachArg):
+        * b3/B3StackmapSpecial.cpp:
+        (JSC::B3::StackmapSpecial::forEachArgImpl):
+        * b3/B3Value.h:
+        * b3/air/AirAllocateStack.cpp:
+        (JSC::B3::Air::allocateStack):
+        * b3/air/AirArg.cpp:
+        (WTF::printInternal):
+        * b3/air/AirArg.h:
+        (JSC::B3::Air::Arg::pointerWidth):
+        (JSC::B3::Air::Arg::isAnyUse):
+        (JSC::B3::Air::Arg::isColdUse):
+        (JSC::B3::Air::Arg::isEarlyUse):
+        (JSC::B3::Air::Arg::isDef):
+        (JSC::B3::Air::Arg::isZDef):
+        (JSC::B3::Air::Arg::widthForB3Type):
+        (JSC::B3::Air::Arg::conservativeWidth):
+        (JSC::B3::Air::Arg::minimumWidth):
+        (JSC::B3::Air::Arg::bytes):
+        (JSC::B3::Air::Arg::widthForBytes):
+        (JSC::B3::Air::Arg::Arg):
+        (JSC::B3::Air::Arg::forEachTmp):
+        * b3/air/AirCCallSpecial.cpp:
+        (JSC::B3::Air::CCallSpecial::forEachArg):
+        * b3/air/AirEliminateDeadCode.cpp:
+        (JSC::B3::Air::eliminateDeadCode):
+        * b3/air/AirFixPartialRegisterStalls.cpp:
+        (JSC::B3::Air::fixPartialRegisterStalls):
+        * b3/air/AirInst.cpp:
+        (JSC::B3::Air::Inst::hasArgEffects):
+        * b3/air/AirInst.h:
+        (JSC::B3::Air::Inst::forEachTmpFast):
+        (JSC::B3::Air::Inst::forEachTmp):
+        * b3/air/AirInstInlines.h:
+        (JSC::B3::Air::Inst::forEachTmpWithExtraClobberedRegs):
+        * b3/air/AirIteratedRegisterCoalescing.cpp:
+        * b3/air/AirLiveness.h:
+        (JSC::B3::Air::AbstractLiveness::AbstractLiveness):
+        (JSC::B3::Air::AbstractLiveness::LocalCalc::execute):
+        * b3/air/AirOpcode.opcodes:
+        * b3/air/AirSpillEverything.cpp:
+        (JSC::B3::Air::spillEverything):
+        * b3/air/AirTmpWidth.cpp: Added.
+        (JSC::B3::Air::TmpWidth::TmpWidth):
+        (JSC::B3::Air::TmpWidth::~TmpWidth):
+        * b3/air/AirTmpWidth.h: Added.
+        (JSC::B3::Air::TmpWidth::width):
+        (JSC::B3::Air::TmpWidth::defWidth):
+        (JSC::B3::Air::TmpWidth::useWidth):
+        (JSC::B3::Air::TmpWidth::Widths::Widths):
+        * b3/air/AirUseCounts.h:
+        (JSC::B3::Air::UseCounts::UseCounts):
+        * b3/air/opcode_generator.rb:
+        * b3/testb3.cpp:
+        (JSC::B3::testCheckMegaCombo):
+        (JSC::B3::testCheckTrickyMegaCombo):
+        (JSC::B3::testCheckTwoMegaCombos):
+        (JSC::B3::run):
+
 2015-12-21  Andy VanWagoner  <thetalecrafter@gmail.com>
 

trunk/Source/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj

@@ -696 +696 @@
                 0FE0502D1AA9095600D33B33 /* VarOffset.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FE050241AA9095600D33B33 /* VarOffset.h */; settings = {ATTRIBUTES = (Private, ); }; };
                 0FE0502F1AAA806900D33B33 /* ScopedArgumentsTable.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FE0502E1AAA806900D33B33 /* ScopedArgumentsTable.cpp */; };
+                0FE0E4AD1C24C94A002E17B6 /* AirTmpWidth.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FE0E4AB1C24C94A002E17B6 /* AirTmpWidth.cpp */; };
+                0FE0E4AE1C24C94A002E17B6 /* AirTmpWidth.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FE0E4AC1C24C94A002E17B6 /* AirTmpWidth.h */; };
                 0FE228ED1436AB2700196C48 /* Options.h in Headers */ = {isa = PBXBuildFile; fileRef = 0FE228EB1436AB2300196C48 /* Options.h */; settings = {ATTRIBUTES = (Private, ); }; };
                 0FE228EE1436AB2C00196C48 /* Options.cpp in Sources */ = {isa = PBXBuildFile; fileRef = 0FE228EA1436AB2300196C48 /* Options.cpp */; };
@@ -2836 +2838 @@
                 0FE050241AA9095600D33B33 /* VarOffset.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = VarOffset.h; sourceTree = "<group>"; };
                 0FE0502E1AAA806900D33B33 /* ScopedArgumentsTable.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ScopedArgumentsTable.cpp; sourceTree = "<group>"; };
+                0FE0E4AB1C24C94A002E17B6 /* AirTmpWidth.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; name = AirTmpWidth.cpp; path = b3/air/AirTmpWidth.cpp; sourceTree = "<group>"; };
+                0FE0E4AC1C24C94A002E17B6 /* AirTmpWidth.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; name = AirTmpWidth.h; path = b3/air/AirTmpWidth.h; sourceTree = "<group>"; };
                 0FE228EA1436AB2300196C48 /* Options.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = Options.cpp; sourceTree = "<group>"; };
                 0FE228EB1436AB2300196C48 /* Options.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = Options.h; sourceTree = "<group>"; };
@@ -4801 +4805 @@
                                 0FEC85691BDACDC70080FF74 /* AirTmp.h */,
                                 0FEC856A1BDACDC70080FF74 /* AirTmpInlines.h */,
+                                0FE0E4AB1C24C94A002E17B6 /* AirTmpWidth.cpp */,
+                                0FE0E4AC1C24C94A002E17B6 /* AirTmpWidth.h */,
                                 0F3730921C0D67EE00052BFA /* AirUseCounts.h */,
                                 0FEC856B1BDACDC70080FF74 /* AirValidate.cpp */,
@@ -7297 +7303 @@
                                 0F2B9CF719D0BAC100B1D1B5 /* FTLExitTimeObjectMaterialization.h in Headers */,
                                 0F235BDB17178E1C00690C7F /* FTLExitValue.h in Headers */,
+                                0FE0E4AE1C24C94A002E17B6 /* AirTmpWidth.h in Headers */,
                                 A7F2996C17A0BB670010417A /* FTLFail.h in Headers */,
                                 0FEA0A2C170B661900BB722C /* FTLFormattedValue.h in Headers */,
@@ -9230 +9237 @@
                                 BCCF0D0C0EF0B8A500413C8F /* StructureStubInfo.cpp in Sources */,
                                 705B41AB1A6E501E00716757 /* Symbol.cpp in Sources */,
+                                0FE0E4AD1C24C94A002E17B6 /* AirTmpWidth.cpp in Sources */,
                                 705B41AD1A6E501E00716757 /* SymbolConstructor.cpp in Sources */,
                                 705B41AF1A6E501E00716757 /* SymbolObject.cpp in Sources */,

trunk/Source/JavaScriptCore/assembler/AbstractMacroAssembler.h

@@ -68 +68 @@
 }
 
+inline bool isX86_64()
+{
+#if CPU(X86_64)
+    return true;
+#else
+    return false;
+#endif
+}
+
 inline bool optimizeForARMv7IDIVSupported()
 {
@@ -81 +90 @@
 {
     return isX86() && Options::useArchitectureSpecificOptimizations();
+}
+
+inline bool optimizeForX86_64()
+{
+    return isX86_64() && Options::useArchitectureSpecificOptimizations();
 }
 

trunk/Source/JavaScriptCore/b3/B3LowerToAir.cpp

@@ -159 +159 @@
     }
 
-    // NOTE: This entire mechanism could be done over Air, if we felt that this would be fast enough.
-    // For now we're assuming that it's faster to do this here, since analyzing B3 is so cheap.
     bool shouldCopyPropagate(Value* value)
     {
@@ -167 +165 @@
         case Identity:
             return true;
-        case ZExt32:
-            return highBitsAreZero(value->child(0));
         default:
             return false;
@@ -1776 +1772 @@
 
         case ZExt32: {
-            if (highBitsAreZero(m_value->child(0))) {
-                ASSERT(tmp(m_value->child(0)) == tmp(m_value));
-                return;
-            }
-
             appendUnOp<Move32, Air::Oops>(m_value->child(0));
             return;

trunk/Source/JavaScriptCore/b3/B3PatchpointSpecial.cpp

@@ -55 +55 @@
     }
 
-    callback(inst.args[1], Arg::Def, inst.origin->airType());
+    callback(inst.args[1], Arg::Def, inst.origin->airType(), inst.origin->airWidth());
     forEachArgImpl(0, 2, inst, SameAsRep, callback);
 }

trunk/Source/JavaScriptCore/b3/B3StackmapSpecial.cpp

@@ -112 +112 @@
             break;
         }
-        
-        callback(arg, role, Arg::typeForB3Type(child.value()->type()));
+
+        Type type = child.value()->type();
+        callback(arg, role, Arg::typeForB3Type(type), Arg::widthForB3Type(type));
     }
 }

trunk/Source/JavaScriptCore/b3/B3Value.h

@@ -77 +77 @@
     // This is useful when lowering. Note that this is only valid for non-void values.
     Air::Arg::Type airType() const { return Air::Arg::typeForB3Type(type()); }
+    Air::Arg::Width airWidth() const { return Air::Arg::widthForB3Type(type()); }
 
     AdjacencyList& children() { return m_children; } 

trunk/Source/JavaScriptCore/b3/air/AirAllocateStack.cpp

@@ -105 +105 @@
         for (Inst& inst : *block) {
             inst.forEachArg(
-                [&] (Arg& arg, Arg::Role role, Arg::Type) {
+                [&] (Arg& arg, Arg::Role role, Arg::Type, Arg::Width) {
                     if (role == Arg::UseAddr && arg.isStack())
                         escapingStackSlots.add(arg.stackSlot());
@@ -149 +149 @@
 
             inst.forEachArg(
-                [&] (Arg& arg, Arg::Role role, Arg::Type) {
+                [&] (Arg& arg, Arg::Role role, Arg::Type, Arg::Width) {
                     if (!Arg::isDef(role))
                         return;

trunk/Source/JavaScriptCore/b3/air/AirArg.cpp

@@ -179 +179 @@
         out.print("UseDef");
         return;
+    case Arg::ZDef:
+        out.print("ZDef");
+        return;
+    case Arg::UseZDef:
+        out.print("UseZDef");
+        return;
     case Arg::UseAddr:
         out.print("UseAddr");

trunk/Source/JavaScriptCore/b3/air/AirArg.h

@@ -102 +102 @@
         Def,
 
+        // This is a special variant of Def that implies that the upper bits of the target register are
+        // zero-filled. Specifically, if the Width of a ZDef is less than the largest possible width of
+        // the argument (for example, we're on a 64-bit machine and we have a Width32 ZDef of a GPR) then
+        // this has different implications for the upper bits (i.e. the top 32 bits in our example)
+        // depending on the kind of the argument:
+        //
+        // For register: the upper bits are zero-filled.
+        // For address: the upper bits are not touched (i.e. we do a 32-bit store in our example).
+        // For tmp: either the upper bits are not touched or they are zero-filled, and we won't know
+        // which until we lower the tmp to either a StackSlot or a Reg.
+        //
+        // The behavior of ZDef is consistent with what happens when you perform 32-bit operations on a
+        // 64-bit GPR. It's not consistent with what happens with 8-bit or 16-bit Defs on x86 GPRs, or
+        // what happens with float Defs in ARM NEON or X86 SSE. Hence why we have both Def and ZDef.
+        ZDef,
+
         // This is a combined Use and Def. It means that both things happen.
         UseDef,
+
+        // This is a combined Use and ZDef. It means that both things happen.
+        UseZDef,
 
         // This is a special kind of use that is only valid for addresses. It means that the
@@ -127 +146 @@
     };
 
+    static Width pointerWidth()
+    {
+        if (sizeof(void*) == 8)
+            return Width64;
+        return Width32;
+    }
+
     enum Signedness : int8_t {
         Signed,
@@ -140 +166 @@
         case ColdUse:
         case UseDef:
+        case UseZDef:
         case LateUse:
             return true;
         case Def:
+        case ZDef:
         case UseAddr:
             return false;
@@ -156 +184 @@
         case Use:
         case UseDef:
+        case UseZDef:
         case Def:
+        case ZDef:
         case UseAddr:
             return false;
@@ -174 +204 @@
         case ColdUse:
         case UseDef:
+        case UseZDef:
             return true;
         case Def:
+        case ZDef:
         case UseAddr:
         case LateUse:
@@ -199 +231 @@
         case Def:
         case UseDef:
+        case ZDef:
+        case UseZDef:
+            return true;
+        }
+    }
+
+    // Returns true if the Role implies that the Inst will ZDef the Arg.
+    static bool isZDef(Role role)
+    {
+        switch (role) {
+        case Use:
+        case ColdUse:
+        case UseAddr:
+        case LateUse:
+        case Def:
+        case UseDef:
+            return false;
+        case ZDef:
+        case UseZDef:
             return true;
         }
@@ -231 +282 @@
         case Int64:
         case Double:
+            return Width64;
+        }
+    }
+
+    static Width conservativeWidth(Type type)
+    {
+        return type == GP ? pointerWidth() : Width64;
+    }
+
+    static Width minimumWidth(Type type)
+    {
+        return type == GP ? Width8 : Width32;
+    }
+
+    static unsigned bytes(Width width)
+    {
+        return 1 << width;
+    }
+
+    static Width widthForBytes(unsigned bytes)
+    {
+        switch (bytes) {
+        case 0:
+        case 1:
+            return Width8;
+        case 2:
+            return Width16;
+        case 3:
+        case 4:
+            return Width32;
+        default:
             return Width64;
         }
@@ -718 +800 @@
     // This defs (%rcx) but uses %rcx.
     template<typename Functor>
-    void forEachTmp(Role argRole, Type argType, const Functor& functor)
+    void forEachTmp(Role argRole, Type argType, Width argWidth, const Functor& functor)
     {
         switch (m_kind) {
         case Tmp:
             ASSERT(isAnyUse(argRole) || isDef(argRole));
-            functor(m_base, argRole, argType);
+            functor(m_base, argRole, argType, argWidth);
             break;
         case Addr:
-            functor(m_base, Use, GP);
+            functor(m_base, Use, GP, argRole == UseAddr ? argWidth : pointerWidth());
             break;
         case Index:
-            functor(m_base, Use, GP);
-            functor(m_index, Use, GP);
+            functor(m_base, Use, GP, argRole == UseAddr ? argWidth : pointerWidth());
+            functor(m_index, Use, GP, argRole == UseAddr ? argWidth : pointerWidth());
             break;
         default:

trunk/Source/JavaScriptCore/b3/air/AirCCallSpecial.cpp

@@ -46 +46 @@
 {
     for (unsigned i = 0; i < numCalleeArgs; ++i)
-        callback(inst.args[calleeArgOffset + i], Arg::Use, Arg::GP);
+        callback(inst.args[calleeArgOffset + i], Arg::Use, Arg::GP, Arg::pointerWidth());
     for (unsigned i = 0; i < numReturnGPArgs; ++i)
-        callback(inst.args[returnGPArgOffset + i], Arg::Def, Arg::GP);
+        callback(inst.args[returnGPArgOffset + i], Arg::Def, Arg::GP, Arg::pointerWidth());
     for (unsigned i = 0; i < numReturnFPArgs; ++i)
-        callback(inst.args[returnFPArgOffset + i], Arg::Def, Arg::FP);
+        callback(inst.args[returnFPArgOffset + i], Arg::Def, Arg::FP, Arg::Width64);
     
     for (unsigned i = argArgOffset; i < inst.args.size(); ++i) {
         // For the type, we can just query the arg's type. The arg will have a type, because we
         // require these args to be argument registers.
-        callback(inst.args[i], Arg::Use, inst.args[i].type());
+        Arg::Type type = inst.args[i].type();
+        callback(inst.args[i], Arg::Use, type, Arg::conservativeWidth(type));
     }
 }

trunk/Source/JavaScriptCore/b3/air/AirEliminateDeadCode.cpp

@@ -81 +81 @@
         bool storesToLive = false;
         inst.forEachArg(
-            [&] (Arg& arg, Arg::Role role, Arg::Type) {
+            [&] (Arg& arg, Arg::Role role, Arg::Type, Arg::Width) {
                 if (!Arg::isDef(role))
                     return;

trunk/Source/JavaScriptCore/b3/air/AirFixPartialRegisterStalls.cpp

@@ -114 +114 @@
     }
 
-    inst.forEachTmp([&] (Tmp& tmp, Arg::Role role, Arg::Type) {
+    inst.forEachTmp([&] (Tmp& tmp, Arg::Role role, Arg::Type, Arg::Width) {
         ASSERT_WITH_MESSAGE(tmp.isReg(), "This phase must be run after register allocation.");
 
@@ -204 +204 @@
                 RegisterSet defs;
                 RegisterSet uses;
-                inst.forEachTmp([&] (Tmp& tmp, Arg::Role role, Arg::Type) {
+                inst.forEachTmp([&] (Tmp& tmp, Arg::Role role, Arg::Type, Arg::Width) {
                     if (tmp.isFPR()) {
                         if (Arg::isDef(role))

trunk/Source/JavaScriptCore/b3/air/AirInst.cpp

@@ -39 +39 @@
     bool result = false;
     forEachArg(
-        [&] (Arg&, Arg::Role role, Arg::Type) {
+        [&] (Arg&, Arg::Role role, Arg::Type, Arg::Width) {
             if (Arg::isDef(role))
                 result = true;

trunk/Source/JavaScriptCore/b3/air/AirInst.h

@@ -99 +99 @@
     }
 
-    typedef void EachArgCallback(Arg&, Arg::Role, Arg::Type);
+    typedef void EachArgCallback(Arg&, Arg::Role, Arg::Type, Arg::Width);
     
-    // Calls the functor with (arg, role, type). This function is auto-generated by
+    // Calls the functor with (arg, role, type, width). This function is auto-generated by
     // opcode_generator.rb.
     template<typename Functor>
     void forEachArg(const Functor&);
 
-    // Calls the functor with (tmp, role, type).
+    // Calls the functor with (tmp, role, type, width).
     template<typename Functor>
     void forEachTmp(const Functor& functor)
     {
         forEachArg(
-            [&] (Arg& arg, Arg::Role role, Arg::Type type) {
-                arg.forEachTmp(role, type, functor);
+            [&] (Arg& arg, Arg::Role role, Arg::Type type, Arg::Width width) {
+                arg.forEachTmp(role, type, width, functor);
             });
     }

trunk/Source/JavaScriptCore/b3/air/AirInstInlines.h

@@ -56 +56 @@
     {
         inst.forEachArg(
-            [&] (Arg& arg, Arg::Role role, Arg::Type type) {
+            [&] (Arg& arg, Arg::Role role, Arg::Type type, Arg::Width width) {
                 if (!arg.isStack())
                     return;
@@ -67 +67 @@
                 // https://bugs.webkit.org/show_bug.cgi?id=151128
                 
-                functor(stackSlot, role, type);
+                functor(stackSlot, role, type, width);
                 arg = Arg::stack(stackSlot, arg.offset());
             });
@@ -100 +100 @@
 {
     forEachTmp(
-        [&] (Tmp& tmpArg, Arg::Role role, Arg::Type argType) {
-            functor(tmpArg, role, argType);
+        [&] (Tmp& tmpArg, Arg::Role role, Arg::Type argType, Arg::Width argWidth) {
+            functor(tmpArg, role, argType, argWidth);
         });
 
     auto reportReg = [&] (Reg reg) {
-        functor(Tmp(reg), Arg::Def, reg.isGPR() ? Arg::GP : Arg::FP);
+        Arg::Type type = reg.isGPR() ? Arg::GP : Arg::FP;
+        functor(Tmp(reg), Arg::Def, type, Arg::conservativeWidth(type));
     };
 

trunk/Source/JavaScriptCore/b3/air/AirIteratedRegisterCoalescing.cpp

@@ -36 +36 @@
 #include "AirRegisterPriority.h"
 #include "AirTmpInlines.h"
+#include "AirTmpWidth.h"
 #include "AirUseCounts.h"
 #include <wtf/ListDump.h>
@@ -631 +632 @@
 class ColoringAllocator : public AbstractColoringAllocator<unsigned> {
 public:
-    ColoringAllocator(Code& code, const UseCounts<Tmp>& useCounts, const HashSet<unsigned>& unspillableTmp)
+    ColoringAllocator(Code& code, TmpWidth& tmpWidth, const UseCounts<Tmp>& useCounts, const HashSet<unsigned>& unspillableTmp)
         : AbstractColoringAllocator<unsigned>(regsInPriorityOrder(type), AbsoluteTmpMapper<type>::lastMachineRegisterIndex(), tmpArraySize(code))
         , m_code(code)
+        , m_tmpWidth(tmpWidth)
         , m_useCounts(useCounts)
         , m_unspillableTmps(unspillableTmp)
@@ -647 +649 @@
     }
 
+    // This tells you if a Move will be coalescable if the src and dst end up matching. This method
+    // relies on an analysis that is invalidated by register allocation, so you it's only meaningful to
+    // call this *before* replacing the Tmp's in this Inst with registers or spill slots.
+    bool mayBeCoalescable(const Inst& inst) const
+    {
+        return mayBeCoalescableImpl(inst, &m_tmpWidth);
+    }
+
     bool isUselessMove(const Inst& inst) const
     {
-        return mayBeCoalescable(inst) && inst.args[0].tmp() == inst.args[1].tmp();
+        return mayBeCoalescableImpl(inst, nullptr) && inst.args[0].tmp() == inst.args[1].tmp();
     }
 
@@ -771 +781 @@
         inst.forEachTmpWithExtraClobberedRegs(
             nextInst,
-            [&] (const Tmp& arg, Arg::Role role, Arg::Type argType) {
+            [&] (const Tmp& arg, Arg::Role role, Arg::Type argType, Arg::Width) {
                 if (!Arg::isDef(role) || argType != type)
                     return;
@@ -778 +788 @@
                 // We should not use forEachDefAndExtraClobberedTmp() here since colored Tmps
                 // do not need interference edges in our implementation.
-                inst.forEachTmp([&] (Tmp& otherArg, Arg::Role role, Arg::Type argType) {
+                inst.forEachTmp([&] (Tmp& otherArg, Arg::Role role, Arg::Type argType, Arg::Width) {
                     if (!Arg::isDef(role) || argType != type)
                         return;
@@ -792 +802 @@
             Tmp defTmp;
             Tmp useTmp;
-            inst.forEachTmp([&defTmp, &useTmp] (Tmp& argTmp, Arg::Role role, Arg::Type) {
+            inst.forEachTmp([&defTmp, &useTmp] (Tmp& argTmp, Arg::Role role, Arg::Type, Arg::Width) {
                 if (Arg::isDef(role))
                     defTmp = argTmp;
@@ -840 +850 @@
         inst.forEachTmpWithExtraClobberedRegs(
             nextInst,
-            [&] (const Tmp& arg, Arg::Role role, Arg::Type argType) {
+            [&] (const Tmp& arg, Arg::Role role, Arg::Type argType, Arg::Width) {
                 if (!Arg::isDef(role) || argType != type)
                     return;
@@ -858 +868 @@
     }
 
-    bool mayBeCoalescable(const Inst& inst) const
+    // Calling this without a tmpWidth will perform a more conservative coalescing analysis that assumes
+    // that Move32's are not coalescable.
+    static bool mayBeCoalescableImpl(const Inst& inst, TmpWidth* tmpWidth)
     {
         switch (type) {
@@ -864 +876 @@
             switch (inst.opcode) {
             case Move:
+            case Move32:
                 break;
             default:
@@ -888 +901 @@
         ASSERT(inst.args[1].type() == type);
 
+        // We can coalesce a Move32 so long as either of the following holds:
+        // - The input is already zero-filled.
+        // - The output only cares about the low 32 bits.
+        //
+        // Note that the input property requires an analysis over ZDef's, so it's only valid so long
+        // as the input gets a register. We don't know if the input gets a register, but we do know
+        // that if it doesn't get a register then we will still emit this Move32.
+        if (inst.opcode == Move32) {
+            if (!tmpWidth)
+                return false;
+
+            if (tmpWidth->defWidth(inst.args[0].tmp()) > Arg::Width32
+                && tmpWidth->useWidth(inst.args[1].tmp()) > Arg::Width32)
+                return false;
+        }
+        
         return true;
     }
@@ -1025 +1054 @@
 
     Code& m_code;
+    TmpWidth& m_tmpWidth;
     // FIXME: spilling should not type specific. It is only a side effect of using UseCounts.
     const UseCounts<Tmp>& m_useCounts;
@@ -1054 +1084 @@
         while (true) {
             ++m_numIterations;
-            ColoringAllocator<type> allocator(m_code, m_useCounts, unspillableTmps);
+
+            // FIXME: One way to optimize this code is to remove the recomputation inside the fixpoint.
+            // We need to recompute because spilling adds tmps, but we could just update tmpWidth when we
+            // add those tmps. Note that one easy way to remove the recomputation is to make any newly
+            // added Tmps get the same use/def widths that the original Tmp got. But, this may hurt the
+            // spill code we emit. Since we currently recompute TmpWidth after spilling, the newly
+            // created Tmps may get narrower use/def widths. On the other hand, the spiller already
+            // selects which move instruction to use based on the original Tmp's widths, so it may not
+            // matter than a subsequent iteration sees a coservative width for the new Tmps. Also, the
+            // recomputation may not actually be a performance problem; it's likely that a better way to
+            // improve performance of TmpWidth is to replace its HashMap with something else. It's
+            // possible that most of the TmpWidth overhead is from queries of TmpWidth rather than the
+            // recomputation, in which case speeding up the lookup would be a bigger win.
+            // https://bugs.webkit.org/show_bug.cgi?id=152478
+            m_tmpWidth.recompute(m_code);
+            
+            ColoringAllocator<type> allocator(m_code, m_tmpWidth, m_useCounts, unspillableTmps);
             if (!allocator.requiresSpilling()) {
                 assignRegistersToTmp(allocator);
@@ -1070 +1116 @@
             for (unsigned instIndex = 0; instIndex < block->size(); ++instIndex) {
                 Inst& inst = block->at(instIndex);
+
+                // The mayBeCoalescable() method will change its mind for some operations after we
+                // complete register allocation. So, we record this before starting.
+                bool mayBeCoalescable = allocator.mayBeCoalescable(inst);
+
+                // On X86_64, Move32 is cheaper if we know that it's equivalent to a Move. It's
+                // equivalent if the destination's high bits are not observable or if the source's high
+                // bits are all zero. Note that we don't have the opposite optimization for other
+                // architectures, which may prefer Move over Move32, because Move is canonical already.
+                if (type == Arg::GP && optimizeForX86_64() && inst.opcode == Move
+                    && inst.args[0].isTmp() && inst.args[1].isTmp()) {
+                    if (m_tmpWidth.useWidth(inst.args[1].tmp()) <= Arg::Width32
+                        || m_tmpWidth.defWidth(inst.args[0].tmp()) <= Arg::Width32)
+                        inst.opcode = Move32;
+                }
+
                 inst.forEachTmpFast([&] (Tmp& tmp) {
                     if (tmp.isReg() || tmp.isGP() == (type != Arg::GP))
@@ -1086 +1148 @@
                     tmp = assignedTmp;
                 });
+
+                if (mayBeCoalescable && inst.args[0].isTmp() && inst.args[1].isTmp()
+                    && inst.args[0].tmp() == inst.args[1].tmp())
+                    inst = Inst();
             }
 
             // Remove all the useless moves we created in this block.
             block->insts().removeAllMatching([&] (const Inst& inst) {
-                return allocator.isUselessMove(inst);
+                return !inst;
             });
         }
@@ -1104 +1170 @@
 
             // Allocate stack slot for each spilled value.
-            bool isNewTmp = stackSlots.add(tmp, m_code.addStackSlot(8, StackSlotKind::Anonymous)).isNewEntry;
+            StackSlot* stackSlot = m_code.addStackSlot(
+                m_tmpWidth.width(tmp) <= Arg::Width32 ? 4 : 8, StackSlotKind::Anonymous);
+            bool isNewTmp = stackSlots.add(tmp, stackSlot).isNewEntry;
             ASSERT_UNUSED(isNewTmp, isNewTmp);
         }
@@ -1115 +1183 @@
             for (unsigned instIndex = 0; instIndex < block->size(); ++instIndex) {
                 Inst& inst = block->at(instIndex);
+
+                // The TmpWidth analysis will say that a Move only stores 32 bits into the destination,
+                // if the source only had 32 bits worth of non-zero bits. Same for the source: it will
+                // only claim to read 32 bits from the source if only 32 bits of the destination are
+                // read. Note that we only apply this logic if this turns into a load or store, since
+                // Move is the canonical way to move data between GPRs.
+                bool forceMove32IfDidSpill = false;
+                bool didSpill = false;
+                if (type == Arg::GP && inst.opcode == Move) {
+                    if (m_tmpWidth.defWidth(inst.args[0].tmp()) <= Arg::Width32
+                        || m_tmpWidth.useWidth(inst.args[1].tmp()) <= Arg::Width32)
+                        forceMove32IfDidSpill = true;
+                }
 
                 // Try to replace the register use by memory use when possible.
@@ -1121 +1202 @@
                     if (arg.isTmp() && arg.type() == type && !arg.isReg()) {
                         auto stackSlotEntry = stackSlots.find(arg.tmp());
-                        if (stackSlotEntry != stackSlots.end() && inst.admitsStack(i))
+                        if (stackSlotEntry != stackSlots.end() && inst.admitsStack(i)) {
                             arg = Arg::stack(stackSlotEntry->value);
+                            didSpill = true;
+                        }
                     }
                 }
 
+                if (didSpill && forceMove32IfDidSpill)
+                    inst.opcode = Move32;
+
                 // For every other case, add Load/Store as needed.
-                inst.forEachTmp([&] (Tmp& tmp, Arg::Role role, Arg::Type argType) {
+                inst.forEachTmp([&] (Tmp& tmp, Arg::Role role, Arg::Type argType, Arg::Width) {
                     if (tmp.isReg() || argType != type)
                         return;
@@ -1142 +1228 @@
 
                     Arg arg = Arg::stack(stackSlotEntry->value);
-                    Opcode move = type == Arg::GP ? Move : MoveDouble;
+                    Opcode move = Oops;
+                    switch (stackSlotEntry->value->byteSize()) {
+                    case 4:
+                        move = type == Arg::GP ? Move32 : MoveFloat;
+                        break;
+                    case 8:
+                        move = type == Arg::GP ? Move : MoveDouble;
+                        break;
+                    default:
+                        RELEASE_ASSERT_NOT_REACHED();
+                        break;
+                    }
 
                     if (Arg::isAnyUse(role)) {
@@ -1167 +1264 @@
 
     Code& m_code;
+    TmpWidth m_tmpWidth;
     UseCounts<Tmp> m_useCounts;
     unsigned m_numIterations { 0 };

trunk/Source/JavaScriptCore/b3/air/AirLiveness.h

@@ -91 +91 @@
 
             block->last().forEach<typename Adapter::Thing>(
-                [&] (typename Adapter::Thing& thing, Arg::Role role, Arg::Type type) {
+                [&] (typename Adapter::Thing& thing, Arg::Role role, Arg::Type type, Arg::Width) {
                     if (Arg::isLateUse(role) && Adapter::acceptsType(type))
                         liveAtTail.add(Adapter::valueToIndex(thing));
@@ -217 +217 @@
             // First handle def's.
             inst.forEach<typename Adapter::Thing>(
-                [&] (typename Adapter::Thing& thing, Arg::Role role, Arg::Type type) {
+                [&] (typename Adapter::Thing& thing, Arg::Role role, Arg::Type type, Arg::Width) {
                     if (Arg::isDef(role) && Adapter::acceptsType(type))
                         workset.remove(Adapter::valueToIndex(thing));
@@ -224 +224 @@
             // Then handle use's.
             inst.forEach<typename Adapter::Thing>(
-                [&] (typename Adapter::Thing& thing, Arg::Role role, Arg::Type type) {
+                [&] (typename Adapter::Thing& thing, Arg::Role role, Arg::Type type, Arg::Width) {
                     if (Arg::isEarlyUse(role) && Adapter::acceptsType(type))
                         workset.add(Adapter::valueToIndex(thing));
@@ -233 +233 @@
                 Inst& prevInst = m_block->at(instIndex - 1);
                 prevInst.forEach<typename Adapter::Thing>(
-                    [&] (typename Adapter::Thing& thing, Arg::Role role, Arg::Type type) {
+                    [&] (typename Adapter::Thing& thing, Arg::Role role, Arg::Type type, Arg::Width) {
                         if (Arg::isLateUse(role) && Adapter::acceptsType(type))
                             workset.add(Adapter::valueToIndex(thing));

trunk/Source/JavaScriptCore/b3/air/AirOpcode.opcodes

@@ -24 +24 @@
 # Syllabus:
 #
-# Roles and types:
-# U:G => use of a general-purpose register or value
-# D:G => def of a general-purpose register or value
-# UD:G => use and def of a general-purpose register or value
-# UA:G => UseAddr (see comment in Arg.h)
-# U:F => use of a float register or value
-# D:F => def of a float register or value
-# UD:F => use and def of a float register or value
+# Examples of some roles, types, and widths:
+# U:G:32 => use of the low 32 bits of a general-purpose register or value
+# D:G:32 => def of the low 32 bits of a general-purpose register or value
+# UD:G:32 => use and def of the low 32 bits of a general-purpose register or value
+# U:G:64 => use of the low 64 bits of a general-purpose register or value
+# ZD:G:32 => def of all bits of a general-purpose register, where all but the low 32 bits are guaranteed to be zeroed.
+# UA:G:Ptr => UseAddr (see comment in Arg.h)
+# U:F:32 => use of a float register or value
+# U:F:64 => use of a double register or value
+# D:F:32 => def of a float register or value
+# UD:F:32 => use and def of a float register or value
 #
 # Argument kinds:
@@ -45 +48 @@
 # even newlines are meaningful to the parser. For example, you could write:
 #
-# Foo42 U:G, UD:F Imm, Tmp Addr, Tmp
+# Foo42 U:G:32, UD:F:32 Imm, Tmp Addr, Tmp
 #
 # And the parser would know that this is the same as:
 #
-# Foo42 U:G, UD:F
+# Foo42 U:G:32, UD:F:32
 #     Imm, Tmp
 #     Addr, Tmp
@@ -59 +62 @@
 # opcode only available on x86_64:
 #
-# x86_64: Fuzz UD:G, D:G
+# x86_64: Fuzz UD:G:64, D:G:64
 #     Tmp, Tmp
 #     Tmp, Addr
@@ -65 +68 @@
 # But this only restricts the two-operand form, the other form is allowed on all architectures:
 #
-# x86_64: Fuzz UD:G, D:G
+# x86_64: Fuzz UD:G:64, D:G:64
 #     Tmp, Tmp
 #     Tmp, Addr
-# Fuzz UD:G, D:G, U:F
+# Fuzz UD:G:Ptr, D:G:Ptr, U:F:Ptr
 #     Tmp, Tmp, Tmp
 #     Tmp, Addr, Tmp
@@ -74 +77 @@
 # And you can also restrict individual forms:
 #
-# Thingy UD:G, D:G
+# Thingy UD:G:32, D:G:32
 #     Tmp, Tmp
 #     arm64: Tmp, Addr
@@ -82 +85 @@
 # versions are available on armv7 or x86_64:
 #
-# x86_64 armv7: Buzz U:G, UD:F
+# x86_64 armv7: Buzz U:G:32, UD:F:32
 #     Tmp, Tmp
 #     Imm, Tmp
@@ -104 +107 @@
 Nop
 
-Add32 U:G, UD:G
-    Tmp, Tmp
-    x86: Imm, Addr
-    Imm, Tmp
-    x86: Addr, Tmp
-    x86: Tmp, Addr
-
-Add32 U:G, U:G, D:G
+Add32 U:G:32, UZD:G:32
+    Tmp, Tmp
+    x86: Imm, Addr
+    Imm, Tmp
+    x86: Addr, Tmp
+    x86: Tmp, Addr
+
+Add32 U:G:32, U:G:32, ZD:G:32
     Imm, Tmp, Tmp
     Tmp, Tmp, Tmp
 
-64: Add64 U:G, UD:G
-    Tmp, Tmp
-    x86: Imm, Addr
-    Imm, Tmp
-    x86: Addr, Tmp
-    x86: Tmp, Addr
-
-64: Add64 U:G, U:G, D:G
+64: Add64 U:G:64, UD:G:64
+    Tmp, Tmp
+    x86: Imm, Addr
+    Imm, Tmp
+    x86: Addr, Tmp
+    x86: Tmp, Addr
+
+64: Add64 U:G:64, U:G:64, D:G:64
     Imm, Tmp, Tmp
     Tmp, Tmp, Tmp
 
-AddDouble U:F, UD:F
-    Tmp, Tmp
-    x86: Addr, Tmp
-
-AddFloat U:F, UD:F
-    Tmp, Tmp
-    x86: Addr, Tmp
-
-Sub32 U:G, UD:G
-    Tmp, Tmp
-    x86: Imm, Addr
-    Imm, Tmp
-    x86: Addr, Tmp
-    x86: Tmp, Addr
-
-64: Sub64 U:G, UD:G
-    Tmp, Tmp
-    x86: Imm, Addr
-    Imm, Tmp
-    x86: Addr, Tmp
-    x86: Tmp, Addr
-
-SubDouble U:F, UD:F
-    Tmp, Tmp
-    x86: Addr, Tmp
-
-SubFloat U:F, UD:F
-    Tmp, Tmp
-    x86: Addr, Tmp
-
-Neg32 UD:G
+AddDouble U:F:64, UD:F:64
+    Tmp, Tmp
+    x86: Addr, Tmp
+
+AddFloat U:F:32, UD:F:32
+    Tmp, Tmp
+    x86: Addr, Tmp
+
+Sub32 U:G:32, UZD:G:32
+    Tmp, Tmp
+    x86: Imm, Addr
+    Imm, Tmp
+    x86: Addr, Tmp
+    x86: Tmp, Addr
+
+64: Sub64 U:G:64, UD:G:64
+    Tmp, Tmp
+    x86: Imm, Addr
+    Imm, Tmp
+    x86: Addr, Tmp
+    x86: Tmp, Addr
+
+SubDouble U:F:64, UD:F:64
+    Tmp, Tmp
+    x86: Addr, Tmp
+
+SubFloat U:F:32, UD:F:32
+    Tmp, Tmp
+    x86: Addr, Tmp
+
+Neg32 UZD:G:32
     Tmp
     Addr
 
-64: Neg64 UD:G
+64: Neg64 UD:G:64
     Tmp
 
-Mul32 U:G, UD:G
-    Tmp, Tmp
-    x86: Addr, Tmp
-
-Mul32 U:G, U:G, D:G
+Mul32 U:G:32, UZD:G:32
+    Tmp, Tmp
+    x86: Addr, Tmp
+
+Mul32 U:G:32, U:G:32, ZD:G:32
     Imm, Tmp, Tmp
 
-64: Mul64 U:G, UD:G
-    Tmp, Tmp
-
-MulDouble U:F, UD:F
-    Tmp, Tmp
-    x86: Addr, Tmp
-
-MulFloat U:F, UD:F
-    Tmp, Tmp
-    x86: Addr, Tmp
-
-DivDouble U:F, UD:F
-    Tmp, Tmp
-    x86: Addr, Tmp
-
-DivFloat U:F, UD:F
-    Tmp, Tmp
-    x86: Addr, Tmp
-
-x86: X86ConvertToDoubleWord32 U:G, D:G
+64: Mul64 U:G:64, UD:G:64
+    Tmp, Tmp
+
+MulDouble U:F:64, UD:F:64
+    Tmp, Tmp
+    x86: Addr, Tmp
+
+MulFloat U:F:32, UD:F:32
+    Tmp, Tmp
+    x86: Addr, Tmp
+
+DivDouble U:F:32, UD:F:32
+    Tmp, Tmp
+    x86: Addr, Tmp
+
+DivFloat U:F:32, UD:F:32
+    Tmp, Tmp
+    x86: Addr, Tmp
+
+x86: X86ConvertToDoubleWord32 U:G:32, ZD:G:32
     Tmp*, Tmp*
 
-x86_64: X86ConvertToQuadWord64 U:G, D:G
+x86_64: X86ConvertToQuadWord64 U:G:64, D:G:64
     Tmp*, Tmp*
 
-x86: X86Div32 UD:G, UD:G, U:G
+x86: X86Div32 UZD:G:32, UZD:G:32, U:G:32
     Tmp*, Tmp*, Tmp
 
-x86_64: X86Div64 UD:G, UD:G, U:G
+x86_64: X86Div64 UZD:G:64, UZD:G:64, U:G:64
     Tmp*, Tmp*, Tmp
 
-Lea UA:G, D:G
-    Addr, Tmp
-
-And32 U:G, UD:G
-    Tmp, Tmp
-    Imm, Tmp
-    x86: Tmp, Addr
-    x86: Addr, Tmp
-    x86: Imm, Addr
-
-64: And64 U:G, UD:G
-    Tmp, Tmp
-    Imm, Tmp
-
-AndDouble U:F, UD:F
-    Tmp, Tmp
-
-AndFloat U:F, UD:F
-    Tmp, Tmp
-
-Lshift32 U:G, UD:G
+Lea UA:G:Ptr, D:G:Ptr
+    Addr, Tmp
+
+And32 U:G:32, UZD:G:32
+    Tmp, Tmp
+    Imm, Tmp
+    x86: Tmp, Addr
+    x86: Addr, Tmp
+    x86: Imm, Addr
+
+64: And64 U:G:64, UD:G:64
+    Tmp, Tmp
+    Imm, Tmp
+
+AndDouble U:F:64, UD:F:64
+    Tmp, Tmp
+
+AndFloat U:F:32, UD:F:32
+    Tmp, Tmp
+
+Lshift32 U:G:32, UZD:G:32
     Tmp*, Tmp
     Imm, Tmp
 
-64: Lshift64 U:G, UD:G
+64: Lshift64 U:G:64, UD:G:64
     Tmp*, Tmp
     Imm, Tmp
 
-Rshift32 U:G, UD:G
+Rshift32 U:G:32, UZD:G:32
     Tmp*, Tmp
     Imm, Tmp
 
-64: Rshift64 U:G, UD:G
+64: Rshift64 U:G:64, UD:G:64
     Tmp*, Tmp
     Imm, Tmp
 
-Urshift32 U:G, UD:G
+Urshift32 U:G:32, UZD:G:32
     Tmp*, Tmp
     Imm, Tmp
 
-64: Urshift64 U:G, UD:G
+64: Urshift64 U:G:64, UD:G:64
     Tmp*, Tmp
     Imm, Tmp
 
-Or32 U:G, UD:G
-    Tmp, Tmp
-    Imm, Tmp
-    x86: Tmp, Addr
-    x86: Addr, Tmp
-    x86: Imm, Addr
-
-64: Or64 U:G, UD:G
-    Tmp, Tmp
-    Imm, Tmp
-
-Xor32 U:G, UD:G
-    Tmp, Tmp
-    Imm, Tmp
-    x86: Tmp, Addr
-    x86: Addr, Tmp
-    x86: Imm, Addr
-
-64: Xor64 U:G, UD:G
-    Tmp, Tmp
-    x86: Tmp, Addr
-    Imm, Tmp
-
-Not32 UD:G
+Or32 U:G:32, UZD:G:32
+    Tmp, Tmp
+    Imm, Tmp
+    x86: Tmp, Addr
+    x86: Addr, Tmp
+    x86: Imm, Addr
+
+64: Or64 U:G:64, UD:G:64
+    Tmp, Tmp
+    Imm, Tmp
+
+Xor32 U:G:32, UZD:G:32
+    Tmp, Tmp
+    Imm, Tmp
+    x86: Tmp, Addr
+    x86: Addr, Tmp
+    x86: Imm, Addr
+
+64: Xor64 U:G:64, UD:G:64
+    Tmp, Tmp
+    x86: Tmp, Addr
+    Imm, Tmp
+
+Not32 UZD:G:32
     Tmp
     x86: Addr
 
-64: Not64 UD:G
+64: Not64 UD:G:64
     Tmp
     x86: Addr
 
-CeilDouble U:F, UD:F
-    Tmp, Tmp
-    Addr, Tmp
-
-CeilFloat U:F, UD:F
-    Tmp, Tmp
-    Addr, Tmp
-
-SqrtDouble U:F, UD:F
-    Tmp, Tmp
-    x86: Addr, Tmp
-
-SqrtFloat U:F, UD:F
-    Tmp, Tmp
-    x86: Addr, Tmp
-
-ConvertInt32ToDouble U:G, D:F
-    Tmp, Tmp
-    x86: Addr, Tmp
-
-64: ConvertInt64ToDouble U:G, D:F
-    Tmp, Tmp
-
-CountLeadingZeros32 U:G, D:G
-    Tmp, Tmp
-    x86: Addr, Tmp
-
-64: CountLeadingZeros64 U:G, D:G
-    Tmp, Tmp
-    x86: Addr, Tmp
-
-ConvertDoubleToFloat U:F, D:F
-    Tmp, Tmp
-    x86: Addr, Tmp
-
-ConvertFloatToDouble U:F, D:F
+CeilDouble U:F:64, UD:F:64
+    Tmp, Tmp
+    Addr, Tmp
+
+CeilFloat U:F:32, UD:F:32
+    Tmp, Tmp
+    Addr, Tmp
+
+SqrtDouble U:F:64, UD:F:64
+    Tmp, Tmp
+    x86: Addr, Tmp
+
+SqrtFloat U:F:32, UD:F:32
+    Tmp, Tmp
+    x86: Addr, Tmp
+
+ConvertInt32ToDouble U:G:32, D:F:64
+    Tmp, Tmp
+    x86: Addr, Tmp
+
+64: ConvertInt64ToDouble U:G:64, D:F:64
+    Tmp, Tmp
+
+CountLeadingZeros32 U:G:32, ZD:G:32
+    Tmp, Tmp
+    x86: Addr, Tmp
+
+64: CountLeadingZeros64 U:G:64, D:G:64
+    Tmp, Tmp
+    x86: Addr, Tmp
+
+ConvertDoubleToFloat U:F:64, D:F:32
+    Tmp, Tmp
+    x86: Addr, Tmp
+
+ConvertFloatToDouble U:F:32, D:F:64
     Tmp, Tmp
     x86: Addr, Tmp
@@ -319 +322 @@
 # Move64 instruction. OTOH, our MacroAssemblers already have this notion of "move()" that basically
 # means movePtr.
-Move U:G, D:G
+Move U:G:Ptr, D:G:Ptr
     Tmp, Tmp
     Imm, Tmp as signExtend32ToPtr
@@ -329 +332 @@
     Imm, Addr as storePtr
 
-Move32 U:G, D:G
+Move32 U:G:32, ZD:G:32
     Tmp, Tmp as zeroExtend32ToPtr
     Addr, Tmp as load32
@@ -338 +341 @@
     Imm, Index as store32
 
-SignExtend32ToPtr U:G, D:G
-    Tmp, Tmp
-
-ZeroExtend8To32 U:G, D:G
+SignExtend32ToPtr U:G:32, D:G:Ptr
+    Tmp, Tmp
+
+ZeroExtend8To32 U:G:8, ZD:G:32
     Tmp, Tmp
     Addr, Tmp as load8
     Index, Tmp as load8
 
-SignExtend8To32 U:G, D:G
+SignExtend8To32 U:G:8, ZD:G:32
     Tmp, Tmp
     x86: Addr, Tmp as load8SignedExtendTo32
     Index, Tmp as load8SignedExtendTo32
 
-ZeroExtend16To32 U:G, D:G
+ZeroExtend16To32 U:G:16, ZD:G:32
     Tmp, Tmp
     Addr, Tmp as load16
     Index, Tmp as load16
 
-SignExtend16To32 U:G, D:G
+SignExtend16To32 U:G:16, ZD:G:32
     Tmp, Tmp
     Addr, Tmp as load16SignedExtendTo32
     Index, Tmp as load16SignedExtendTo32
 
-MoveFloat U:F, D:F
+MoveFloat U:F:32, D:F:32
     Tmp, Tmp as moveDouble
     Addr, Tmp as loadFloat
@@ -368 +371 @@
     Tmp, Index as storeFloat
 
-MoveDouble U:F, D:F
+MoveDouble U:F:64, D:F:64
     Tmp, Tmp
     Addr, Tmp as loadDouble
@@ -375 +378 @@
     Tmp, Index as storeDouble
 
-MoveZeroToDouble D:F
+MoveZeroToDouble D:F:64
     Tmp
 
-64: Move64ToDouble U:G, D:F
+64: Move64ToDouble U:G:64, D:F:64
     Tmp, Tmp
     Addr, Tmp as loadDouble
     Index, Tmp as loadDouble
 
-MoveInt32ToPacked U:G, D:F
+MoveInt32ToPacked U:G:32, D:F:32
     Tmp, Tmp
     Addr, Tmp as loadFloat
     Index, Tmp as loadFloat
 
-64: MoveDoubleTo64 U:F, D:G
+64: MoveDoubleTo64 U:F:64, D:G:64
     Tmp, Tmp
     Addr, Tmp as load64
     Index, Tmp as load64
 
-MovePackedToInt32 U:F, D:G
+MovePackedToInt32 U:F:32, D:G:32
     Tmp, Tmp
     Addr, Tmp as load32
     Index, Tmp as load32
 
-Load8 U:G, D:G
+Load8 U:G:8, ZD:G:32
     Addr, Tmp
     Index, Tmp
 
-Store8 U:G, D:G
+Store8 U:G:8, D:G:8
     Tmp, Index
     Tmp, Addr
@@ -408 +411 @@
     Imm, Addr
 
-Load8SignedExtendTo32 U:G, D:G
+Load8SignedExtendTo32 U:G:8, ZD:G:32
     Addr, Tmp
     Index, Tmp
 
-Load16 U:G, D:G
+Load16 U:G:16, ZD:G:32
     Addr, Tmp
     Index, Tmp
 
-Load16SignedExtendTo32 U:G, D:G
+Load16SignedExtendTo32 U:G:16, ZD:G:32
     Addr, Tmp
     Index, Tmp
 
-Compare32 U:G, U:G, U:G, D:G
+Compare32 U:G:32, U:G:32, U:G:32, ZD:G:32
     RelCond, Tmp, Tmp, Tmp
     RelCond, Tmp, Imm, Tmp
 
-64: Compare64 U:G, U:G, U:G, D:G
+64: Compare64 U:G:32, U:G:64, U:G:64, ZD:G:32
     RelCond, Tmp, Imm, Tmp
     RelCond, Tmp, Tmp, Tmp
 
-Test32 U:G, U:G, U:G, D:G
+Test32 U:G:32, U:G:32, U:G:32, ZD:G:32
     x86: ResCond, Addr, Imm, Tmp
     ResCond, Tmp, Tmp, Tmp
 
-64: Test64 U:G, U:G, U:G, D:G
+64: Test64 U:G:32, U:G:64, U:G:64, ZD:G:32
     ResCond, Tmp, Imm, Tmp
     ResCond, Tmp, Tmp, Tmp
 
-CompareDouble U:G, U:F, U:F, D:G
+CompareDouble U:G:32, U:F:64, U:F:64, ZD:G:32
     DoubleCond, Tmp, Tmp, Tmp
 
-CompareFloat U:G, U:F, U:F, D:G
+CompareFloat U:G:32, U:F:32, U:F:32, ZD:G:32
     DoubleCond, Tmp, Tmp, Tmp
 
@@ -445 +448 @@
 # you opt them into the block order optimizations.
 
-Branch8 U:G, U:G, U:G /branch
+Branch8 U:G:32, U:G:8, U:G:8 /branch
     x86: RelCond, Addr, Imm
     x86: RelCond, Index, Imm
 
-Branch32 U:G, U:G, U:G /branch
+Branch32 U:G:32, U:G:32, U:G:32 /branch
     x86: RelCond, Addr, Imm
     RelCond, Tmp, Tmp
@@ -457 +460 @@
     x86: RelCond, Index, Imm
 
-64: Branch64 U:G, U:G, U:G /branch
+64: Branch64 U:G:32, U:G:64, U:G:64 /branch
     RelCond, Tmp, Tmp
     x86: RelCond, Tmp, Addr
@@ -463 +466 @@
     x86: RelCond, Index, Tmp
 
-BranchTest8 U:G, U:G, U:G /branch
+BranchTest8 U:G:32, U:G:8, U:G:8 /branch
     x86: ResCond, Addr, Imm
     x86: ResCond, Index, Imm
 
-BranchTest32 U:G, U:G, U:G /branch
+BranchTest32 U:G:32, U:G:32, U:G:32 /branch
     ResCond, Tmp, Tmp
     ResCond, Tmp, Imm
@@ -475 +478 @@
 # Warning: forms that take an immediate will sign-extend their immediate. You probably want
 # BranchTest32 in most cases where you use an immediate.
-64: BranchTest64 U:G, U:G, U:G /branch
+64: BranchTest64 U:G:32, U:G:64, U:G:64 /branch
     ResCond, Tmp, Tmp
     ResCond, Tmp, Imm
@@ -482 +485 @@
     x86: ResCond, Index, Imm
 
-BranchDouble U:G, U:F, U:F /branch
+BranchDouble U:G:32, U:F:64, U:F:64 /branch
     DoubleCond, Tmp, Tmp
 
-BranchFloat U:G, U:F, U:F /branch
+BranchFloat U:G:32, U:F:32, U:F:32 /branch
     DoubleCond, Tmp, Tmp
 
-BranchAdd32 U:G, U:G, UD:G /branch
+BranchAdd32 U:G:32, U:G:32, UZD:G:32 /branch
     ResCond, Tmp, Tmp
     ResCond, Imm, Tmp
@@ -495 +498 @@
     x86: ResCond, Addr, Tmp
 
-64: BranchAdd64 U:G, U:G, UD:G /branch
+64: BranchAdd64 U:G:32, U:G:64, UD:G:64 /branch
     ResCond, Imm, Tmp
     ResCond, Tmp, Tmp
 
-BranchMul32 U:G, U:G, UD:G /branch
+BranchMul32 U:G:32, U:G:32, UZD:G:32 /branch
     ResCond, Tmp, Tmp
     x86: ResCond, Addr, Tmp
 
-BranchMul32 U:G, U:G, U:G, D:G /branch
+BranchMul32 U:G:32, U:G:32, U:G:32, ZD:G:32 /branch
     ResCond, Tmp, Imm, Tmp
 
-64: BranchMul64 U:G, U:G, UD:G /branch
-    ResCond, Tmp, Tmp
-
-BranchSub32 U:G, U:G, UD:G /branch
+64: BranchMul64 U:G:32, U:G:64, UZD:G:64 /branch
+    ResCond, Tmp, Tmp
+
+BranchSub32 U:G:32, U:G:32, UZD:G:32 /branch
     ResCond, Tmp, Tmp
     ResCond, Imm, Tmp
@@ -516 +519 @@
     x86: ResCond, Addr, Tmp
 
-64: BranchSub64 U:G, U:G, UD:G /branch
+64: BranchSub64 U:G:32, U:G:64, UD:G:64 /branch
     ResCond, Imm, Tmp
     ResCond, Tmp, Tmp
 
-BranchNeg32 U:G, UD:G /branch
+BranchNeg32 U:G:32, UZD:G:32 /branch
     ResCond, Tmp
 
-64: BranchNeg64 U:G, UD:G /branch
+64: BranchNeg64 U:G:32, UZD:G:64 /branch
     ResCond, Tmp
 
-MoveConditionally32 U:G, U:G, U:G, U:G, UD:G
+MoveConditionally32 U:G:32, U:G:32, U:G:32, U:G:Ptr, UD:G:Ptr
     RelCond, Tmp, Tmp, Tmp, Tmp
 
-64: MoveConditionally64 U:G, U:G, U:G, U:G, UD:G
+64: MoveConditionally64 U:G:32, U:G:64, U:G:64, U:G:Ptr, UD:G:Ptr
     RelCond, Tmp, Tmp, Tmp, Tmp
 
-MoveConditionallyTest32 U:G, U:G, U:G, U:G, UD:G
+MoveConditionallyTest32 U:G:32, U:G:32, U:G:32, U:G:Ptr, UD:G:Ptr
     ResCond, Tmp, Tmp, Tmp, Tmp
     ResCond, Tmp, Imm, Tmp, Tmp
 
-64: MoveConditionallyTest64 U:G, U:G, U:G, U:G, UD:G
+64: MoveConditionallyTest64 U:G:32, U:G:64, U:G:64, U:G:Ptr, UD:G:Ptr
     ResCond, Tmp, Tmp, Tmp, Tmp
     ResCond, Tmp, Imm, Tmp, Tmp
 
-MoveConditionallyDouble U:G, U:F, U:F, U:G, UD:G
+MoveConditionallyDouble U:G:32, U:F:64, U:F:64, U:G:Ptr, UD:G:Ptr
     DoubleCond, Tmp, Tmp, Tmp, Tmp
 
-MoveConditionallyFloat U:G, U:F, U:F, U:G, UD:G
+MoveConditionallyFloat U:G:32, U:F:32, U:F:32, U:G:Ptr, UD:G:Ptr
     DoubleCond, Tmp, Tmp, Tmp, Tmp
 
-MoveDoubleConditionally32 U:G, U:G, U:G, U:F, UD:F
+MoveDoubleConditionally32 U:G:32, U:G:32, U:G:32, U:F:64, UD:F:64
     RelCond, Tmp, Tmp, Tmp, Tmp
 
-64: MoveDoubleConditionally64 U:G, U:G, U:G, U:F, UD:F
+64: MoveDoubleConditionally64 U:G:32, U:G:64, U:G:64, U:F:64, UD:F:64
     RelCond, Tmp, Tmp, Tmp, Tmp
 
-MoveDoubleConditionallyTest32 U:G, U:G, U:G, U:F, UD:F
+MoveDoubleConditionallyTest32 U:G:32, U:G:32, U:G:32, U:F:64, UD:F:64
     ResCond, Tmp, Tmp, Tmp, Tmp
     ResCond, Tmp, Imm, Tmp, Tmp
 
-64: MoveDoubleConditionallyTest64 U:G, U:G, U:G, U:F, UD:F
+64: MoveDoubleConditionallyTest64 U:G:32, U:G:64, U:G:64, U:F:64, UD:F:64
     ResCond, Tmp, Tmp, Tmp, Tmp
     ResCond, Tmp, Imm, Tmp, Tmp
 
-MoveDoubleConditionallyDouble U:G, U:F, U:F, U:F, UD:F
+MoveDoubleConditionallyDouble U:G:32, U:F:64, U:F:64, U:F:64, UD:F:64
     DoubleCond, Tmp, Tmp, Tmp, Tmp
 
-MoveDoubleConditionallyFloat U:G, U:F, U:F, U:F, UD:F
+MoveDoubleConditionallyFloat U:G:32, U:F:32, U:F:32, U:F:64, UD:F:64
     DoubleCond, Tmp, Tmp, Tmp, Tmp
 

trunk/Source/JavaScriptCore/b3/air/AirOptimizeBlockOrder.cpp

@@ -149 +149 @@
         case BranchTest32:
         case BranchTest64:
+        case BranchFloat:
         case BranchDouble:
         case BranchAdd32:

trunk/Source/JavaScriptCore/b3/air/AirSpillEverything.cpp

@@ -68 +68 @@
             inst.forEachTmpWithExtraClobberedRegs(
                 index < block->size() ? &block->at(index) : nullptr,
-                [&registerSet] (const Tmp& tmp, Arg::Role role, Arg::Type) {
+                [&registerSet] (const Tmp& tmp, Arg::Role role, Arg::Type, Arg::Width) {
                     if (tmp.isReg() && Arg::isDef(role))
                         registerSet.set(tmp.reg());
@@ -120 +120 @@
             // Now fall back on spilling using separate Move's to load/store the tmp.
             inst.forEachTmp(
-                [&] (Tmp& tmp, Arg::Role role, Arg::Type type) {
+                [&] (Tmp& tmp, Arg::Role role, Arg::Type type, Arg::Width) {
                     if (tmp.isReg())
                         return;
@@ -141 +141 @@
                         break;
                     case Arg::Def:
+                    case Arg::ZDef:
                         for (Reg reg : regsInPriorityOrder(type)) {
                             if (!setAfter.get(reg)) {
@@ -150 +151 @@
                         break;
                     case Arg::UseDef:
+                    case Arg::UseZDef:
                     case Arg::LateUse:
                         for (Reg reg : regsInPriorityOrder(type)) {

trunk/Source/JavaScriptCore/b3/air/AirUseCounts.h

@@ -78 +78 @@
             for (Inst& inst : *block) {
                 inst.forEach<Thing>(
-                    [&] (Thing& arg, Arg::Role role, Arg::Type) {
+                    [&] (Thing& arg, Arg::Role role, Arg::Type, Arg::Width) {
                         Counts& counts = m_counts.add(arg, Counts()).iterator->value;
 

trunk/Source/JavaScriptCore/b3/air/opcode_generator.rb

@@ -45 +45 @@
 
 class Arg
-    attr_reader :role, :type
-
-    def initialize(role, type)
+    attr_reader :role, :type, :width
+
+    def initialize(role, type, width)
         @role = role
         @type = type
+        @width = width
     end
 end
@@ -174 +175 @@
 
 def isUD(token)
-    token =~ /\A((U)|(D)|(UD)|(UA))\Z/
+    token =~ /\A((U)|(D)|(UD)|(ZD)|(UZD)|(UA))\Z/
 end
 
@@ -189 +190 @@
 end
 
+def isWidth(token)
+    token =~ /\A((8)|(16)|(32)|(64)|(Ptr))\Z/
+end
+
 def isKeyword(token)
-    isUD(token) or isGF(token) or isKind(token) or isArch(token) or
+    isUD(token) or isGF(token) or isKind(token) or isArch(token) or isWidth(token) or
         token == "special" or token == "as"
 end
@@ -253 +258 @@
         result = token.string
         parseError("Expected kind (Imm, Imm64, Tmp, Addr, Index, RelCond, ResCond, or DoubleCond)") unless isKind(result)
+        advance
+        result
+    end
+
+    def consumeWidth
+        result = token.string
+        parseError("Expected width (8, 16, 32, or 64)") unless isWidth(result)
         advance
         result
@@ -351 +363 @@
                         consume(":")
                         type = consumeType
+                        consume(":")
+                        width = consumeWidth
                         
-                        signature << Arg.new(role, type)
+                        signature << Arg.new(role, type, width)
                         
                         break unless token == ","
@@ -607 +621 @@
         | opcode, overload |
         if opcode.special
-            outp.puts "functor(args[0], Arg::Use, Arg::GP); // This is basically bogus, but it works f analyses model Special as an immediate."
+            outp.puts "functor(args[0], Arg::Use, Arg::GP, Arg::pointerWidth()); // This is basically bogus, but it works for analyses that model Special as an immediate."
             outp.puts "args[0].special()->forEachArg(*this, scopedLambda<EachArgCallback>(functor));"
         else
             overload.signature.each_with_index {
                 | arg, index |
+                
                 role = nil
                 case arg.role
@@ -618 +633 @@
                 when "D"
                     role = "Def"
+                when "ZD"
+                    role = "ZDef"
                 when "UD"
                     role = "UseDef"
+                when "UZD"
+                    role = "UseZDef"
                 when "UA"
                     role = "UseAddr"
@@ -625 +644 @@
                     raise
                 end
+
+                if arg.width == "Ptr"
+                    width = "Arg::pointerWidth()"
+                else
+                    width = "Arg::Width#{arg.width}"
+                end
                 
-                outp.puts "functor(args[#{index}], Arg::#{role}, Arg::#{arg.type}P);"
+                outp.puts "functor(args[#{index}], Arg::#{role}, Arg::#{arg.type}P, #{width});"
             }
         end

trunk/Source/JavaScriptCore/b3/testb3.cpp

@@ -6367 +6367 @@
     value = -1;
     CHECK(invoke<int>(*code, &value - 2, 1) == 42);
+}
+
+void testCheckTrickyMegaCombo()
+{
+    Procedure proc;
+    BasicBlock* root = proc.addBlock();
+    Value* base = root->appendNew<ArgumentRegValue>(proc, Origin(), GPRInfo::argumentGPR0);
+    Value* index = root->appendNew<Value>(
+        proc, ZExt32, Origin(),
+        root->appendNew<Value>(
+            proc, Add, Origin(),
+            root->appendNew<Value>(
+                proc, Trunc, Origin(),
+                root->appendNew<ArgumentRegValue>(proc, Origin(), GPRInfo::argumentGPR1)),
+            root->appendNew<Const32Value>(proc, Origin(), 1)));
+
+    Value* ptr = root->appendNew<Value>(
+        proc, Add, Origin(), base,
+        root->appendNew<Value>(
+            proc, Shl, Origin(), index,
+            root->appendNew<Const32Value>(proc, Origin(), 1)));
+    
+    CheckValue* check = root->appendNew<CheckValue>(
+        proc, Check, Origin(),
+        root->appendNew<Value>(
+            proc, LessThan, Origin(),
+            root->appendNew<MemoryValue>(proc, Load8S, Origin(), ptr),
+            root->appendNew<Const32Value>(proc, Origin(), 42)));
+    check->setGenerator(
+        [&] (CCallHelpers& jit, const StackmapGenerationParams& params) {
+            AllowMacroScratchRegisterUsage allowScratch(jit);
+            CHECK(!params.size());
+
+            // This should always work because a function this simple should never have callee
+            // saves.
+            jit.move(CCallHelpers::TrustedImm32(42), GPRInfo::returnValueGPR);
+            jit.emitFunctionEpilogue();
+            jit.ret();
+        });
+    root->appendNew<ControlValue>(
+        proc, Return, Origin(), root->appendNew<Const32Value>(proc, Origin(), 0));
+
+    auto code = compile(proc);
+
+    int8_t value;
+    value = 42;
+    CHECK(invoke<int>(*code, &value - 2, 0) == 0);
+    value = 127;
+    CHECK(invoke<int>(*code, &value - 2, 0) == 0);
+    value = 41;
+    CHECK(invoke<int>(*code, &value - 2, 0) == 42);
+    value = 0;
+    CHECK(invoke<int>(*code, &value - 2, 0) == 42);
+    value = -1;
+    CHECK(invoke<int>(*code, &value - 2, 0) == 42);
 }
 
@@ -9475 +9530 @@
     RUN(testCheckLessThan());
     RUN(testCheckMegaCombo());
+    RUN(testCheckTrickyMegaCombo());
     RUN(testCheckTwoMegaCombos());
     RUN(testCheckTwoNonRedundantMegaCombos());

trunk/Source/JavaScriptCore/ftl/FTLLowerDFGToLLVM.cpp

@@ -4219 +4219 @@
         for (unsigned i = 1; i < numKids; ++i) {
             flags = m_out.bitAnd(flags, m_out.load32(kids[i], m_heaps.JSString_flags));
+#if FTL_USES_B3
+            B3::CheckValue* lengthCheck = m_out.speculateAdd(
+                length, m_out.load32(kids[i], m_heaps.JSString_length));
+            blessSpeculation(lengthCheck, Uncountable, noValue(), nullptr, m_origin);
+            length = lengthCheck;
+#else // FTL_USES_B3            
             LValue lengthAndOverflow = m_out.addWithOverflow32(
                 length, m_out.load32(kids[i], m_heaps.JSString_length));
             speculate(Uncountable, noValue(), 0, m_out.extractValue(lengthAndOverflow, 1));
             length = m_out.extractValue(lengthAndOverflow, 0);
+#endif // FTL_USES_B3
         }
         m_out.store32(

trunk/Source/JavaScriptCore/ftl/FTLOSRExitHandle.cpp

@@ -40 +40 @@
     jit.pushToSaveImmediateWithoutTouchingRegisters(CCallHelpers::TrustedImm32(index));
     CCallHelpers::PatchableJump jump = jit.patchableJump();
+    RefPtr<OSRExitHandle> self = this;
     jit.addLinkTask(
-        [this, jump] (LinkBuffer& linkBuffer) {
-            exit.m_patchableJump = CodeLocationJump(linkBuffer.locationOf(jump));
+        [self, jump] (LinkBuffer& linkBuffer) {
+            self->exit.m_patchableJump = CodeLocationJump(linkBuffer.locationOf(jump));
 
             linkBuffer.link(