Changeset 65042 in webkit

Timestamp:

Aug 9, 2010 8:19:19 PM (14 years ago)

Author:

oliver@apple.com

Message:

2010-08-09 Oliver Hunt <​oliver@apple.com>

Reviewed by Gavin Barraclough.

Allow an assembler/macroassembler to compact branches to more concise forms when linking
​https://bugs.webkit.org/show_bug.cgi?id=43745

This patch makes it possible for an assembler to convert jumps into a different
(presumably more efficient) form at link time. Currently implemented in the
ARMv7 JIT as that already had logic to delay linking of jumps until the end of
compilation already. The ARMv7 JIT chooses between either a 4 byte short jump
or a full 32-bit offset (and rewrites ITTT instructions as appropriate), so does
not yet produce the most compact form possible. The general design of the linker
should make it relatively simple to introduce new branch types with little effort,
as the linker has no knowledge of the exact form of any of the branches.

• JavaScriptCore.xcodeproj/project.pbxproj:
• assembler/ARMv7Assembler.cpp: Added. (JSC::): Record jump sizes

• assembler/ARMv7Assembler.h: (JSC::ARMv7Assembler::LinkRecord::LinkRecord): (JSC::ARMv7Assembler::LinkRecord::from): (JSC::ARMv7Assembler::LinkRecord::setFrom): (JSC::ARMv7Assembler::LinkRecord::to): (JSC::ARMv7Assembler::LinkRecord::type): (JSC::ARMv7Assembler::LinkRecord::linkType): (JSC::ARMv7Assembler::LinkRecord::setLinkType): Encapsulate LinkRecord fields so we can compress the values somewhat

(JSC::ARMv7Assembler::JmpSrc::JmpSrc):

Need to record the jump type now

(JSC::ARMv7Assembler::b):
(JSC::ARMv7Assembler::blx):
(JSC::ARMv7Assembler::bx):

Need to pass the jump types

(JSC::ARMv7Assembler::executableOffsetFor):
(JSC::ARMv7Assembler::jumpSizeDelta):
(JSC::ARMv7Assembler::linkRecordSourceComparator):
(JSC::ARMv7Assembler::computeJumpType):
(JSC::ARMv7Assembler::convertJumpTo):
(JSC::ARMv7Assembler::recordLinkOffsets):
(JSC::ARMv7Assembler::jumpsToLink):
(JSC::ARMv7Assembler::link):
(JSC::ARMv7Assembler::unlinkedCode):

Helper functions for the linker

(JSC::ARMv7Assembler::linkJump):
(JSC::ARMv7Assembler::canBeShortJump):
(JSC::ARMv7Assembler::linkLongJump):
(JSC::ARMv7Assembler::linkShortJump):
(JSC::ARMv7Assembler::linkJumpAbsolute):

Moving code around for the various jump linking functions

• assembler/AbstractMacroAssembler.h: (JSC::AbstractMacroAssembler::beginUninterruptedSequence): (JSC::AbstractMacroAssembler::endUninterruptedSequence): We have to track uninterrupted sequences in any assembler that compacts branches as that's not something we're allowed to do in such sequences. AbstractMacroAssembler has a nop version of these functions as it makes the code elsewhere nicer.

• assembler/LinkBuffer.h: (JSC::LinkBuffer::LinkBuffer): (JSC::LinkBuffer::link): (JSC::LinkBuffer::patch): (JSC::LinkBuffer::locationOf): (JSC::LinkBuffer::locationOfNearCall): (JSC::LinkBuffer::returnAddressOffset): (JSC::LinkBuffer::trampolineAt): Updated these functions to adjust for any changed offsets in the linked code

(JSC::LinkBuffer::applyOffset):

A helper function to deal with the now potentially moved labels

(JSC::LinkBuffer::linkCode):

The new and mighty linker function

• assembler/MacroAssemblerARMv7.h: (JSC::MacroAssemblerARMv7::MacroAssemblerARMv7): (JSC::MacroAssemblerARMv7::beginUninterruptedSequence): (JSC::MacroAssemblerARMv7::endUninterruptedSequence): (JSC::MacroAssemblerARMv7::jumpsToLink): (JSC::MacroAssemblerARMv7::unlinkedCode): (JSC::MacroAssemblerARMv7::computeJumpType): (JSC::MacroAssemblerARMv7::convertJumpTo): (JSC::MacroAssemblerARMv7::recordLinkOffsets): (JSC::MacroAssemblerARMv7::jumpSizeDelta): (JSC::MacroAssemblerARMv7::link): (JSC::MacroAssemblerARMv7::jump): (JSC::MacroAssemblerARMv7::branchMul32): (JSC::MacroAssemblerARMv7::breakpoint): (JSC::MacroAssemblerARMv7::nearCall): (JSC::MacroAssemblerARMv7::call): (JSC::MacroAssemblerARMv7::ret): (JSC::MacroAssemblerARMv7::tailRecursiveCall): (JSC::MacroAssemblerARMv7::executableOffsetFor): (JSC::MacroAssemblerARMv7::inUninterruptedSequence): (JSC::MacroAssemblerARMv7::makeJump): (JSC::MacroAssemblerARMv7::makeBranch):

All branches need to pass on their type now

• jit/ExecutableAllocator.h: (JSC::ExecutablePool::returnLastBytes):

We can't know ahead of time how much space will be necessary to
hold the linked code if we're compacting branches, this new
function allows us to return the unused bytes at the end of linking

• jit/JIT.cpp: (JSC::JIT::JIT): (JSC::JIT::privateCompile):
• jit/JIT.h: (JSC::JIT::compile):

The JIT class now needs to take a linker offset so that recompilation
can generate the same jumps when using branch compaction.

• jit/JITArithmetic32_64.cpp: (JSC::JIT::emitSlow_op_mod):
• jit/JITOpcodes.cpp: (JSC::JIT::privateCompileCTIMachineTrampolines):
• jit/JITOpcodes32_64.cpp: (JSC::JIT::privateCompileCTIMachineTrampolines): (JSC::JIT::privateCompileCTINativeCall): Update for new trampolineAt changes

• wtf/FastMalloc.cpp: (WTF::TCMallocStats::):
• wtf/Platform.h:

Location:

trunk/JavaScriptCore

Files:

• ChangeLog (modified) (1 diff)
• JavaScriptCore.xcodeproj/project.pbxproj (modified) (4 diffs)
• assembler/ARMv7Assembler.cpp (added)
• assembler/ARMv7Assembler.h (modified) (16 diffs)
• assembler/AbstractMacroAssembler.h (modified) (2 diffs)
• assembler/LinkBuffer.h (modified) (13 diffs)
• assembler/MacroAssemblerARMv7.h (modified) (13 diffs)
• jit/ExecutableAllocator.h (modified) (1 diff)
• jit/JIT.cpp (modified) (3 diffs)
• jit/JIT.h (modified) (7 diffs)
• jit/JITArithmetic32_64.cpp (modified) (1 diff)
• jit/JITOpcodes.cpp (modified) (2 diffs)
• jit/JITOpcodes32_64.cpp (modified) (4 diffs)
• jit/JITPropertyAccess.cpp (modified) (8 diffs)
• jit/JITPropertyAccess32_64.cpp (modified) (8 diffs)
• jit/SpecializedThunkJIT.h (modified) (1 diff)
• runtime/Executable.cpp (modified) (2 diffs)
• wtf/FastMalloc.cpp (modified) (1 diff)
• wtf/Platform.h (modified) (1 diff)
• yarr/RegexJIT.cpp (modified) (1 diff)

trunk/JavaScriptCore/ChangeLog

@@ +1 @@
+2010-08-09  Oliver Hunt  <oliver@apple.com>
+
+        Reviewed by Gavin Barraclough.
+
+        Allow an assembler/macroassembler to compact branches to more concise forms when linking
+        https://bugs.webkit.org/show_bug.cgi?id=43745
+
+        This patch makes it possible for an assembler to convert jumps into a different
+        (presumably more efficient) form at link time.  Currently implemented in the
+        ARMv7 JIT as that already had logic to delay linking of jumps until the end of
+        compilation already.  The ARMv7 JIT chooses between either a 4 byte short jump
+        or a full 32-bit offset (and rewrites ITTT instructions as appropriate), so does
+        not yet produce the most compact form possible.  The general design of the linker
+        should make it relatively simple to introduce new branch types with little effort,
+        as the linker has no knowledge of the exact form of any of the branches.
+
+        * JavaScriptCore.xcodeproj/project.pbxproj:
+        * assembler/ARMv7Assembler.cpp: Added.
+        (JSC::):
+          Record jump sizes
+
+        * assembler/ARMv7Assembler.h:
+        (JSC::ARMv7Assembler::LinkRecord::LinkRecord):
+        (JSC::ARMv7Assembler::LinkRecord::from):
+        (JSC::ARMv7Assembler::LinkRecord::setFrom):
+        (JSC::ARMv7Assembler::LinkRecord::to):
+        (JSC::ARMv7Assembler::LinkRecord::type):
+        (JSC::ARMv7Assembler::LinkRecord::linkType):
+        (JSC::ARMv7Assembler::LinkRecord::setLinkType):
+          Encapsulate LinkRecord fields so we can compress the values somewhat
+
+        (JSC::ARMv7Assembler::JmpSrc::JmpSrc):
+          Need to record the jump type now
+
+        (JSC::ARMv7Assembler::b):
+        (JSC::ARMv7Assembler::blx):
+        (JSC::ARMv7Assembler::bx):
+          Need to pass the jump types
+
+        (JSC::ARMv7Assembler::executableOffsetFor):
+        (JSC::ARMv7Assembler::jumpSizeDelta):
+        (JSC::ARMv7Assembler::linkRecordSourceComparator):
+        (JSC::ARMv7Assembler::computeJumpType):
+        (JSC::ARMv7Assembler::convertJumpTo):
+        (JSC::ARMv7Assembler::recordLinkOffsets):
+        (JSC::ARMv7Assembler::jumpsToLink):
+        (JSC::ARMv7Assembler::link):
+        (JSC::ARMv7Assembler::unlinkedCode):
+          Helper functions for the linker
+
+        (JSC::ARMv7Assembler::linkJump):
+        (JSC::ARMv7Assembler::canBeShortJump):
+        (JSC::ARMv7Assembler::linkLongJump):
+        (JSC::ARMv7Assembler::linkShortJump):
+        (JSC::ARMv7Assembler::linkJumpAbsolute):
+           Moving code around for the various jump linking functions
+
+        * assembler/AbstractMacroAssembler.h:
+        (JSC::AbstractMacroAssembler::beginUninterruptedSequence):
+        (JSC::AbstractMacroAssembler::endUninterruptedSequence):
+          We have to track uninterrupted sequences in any assembler that compacts
+          branches as that's not something we're allowed to do in such sequences.
+          AbstractMacroAssembler has a nop version of these functions as it makes the
+          code elsewhere nicer.
+
+        * assembler/LinkBuffer.h:
+        (JSC::LinkBuffer::LinkBuffer):
+        (JSC::LinkBuffer::link):
+        (JSC::LinkBuffer::patch):
+        (JSC::LinkBuffer::locationOf):
+        (JSC::LinkBuffer::locationOfNearCall):
+        (JSC::LinkBuffer::returnAddressOffset):
+        (JSC::LinkBuffer::trampolineAt):
+          Updated these functions to adjust for any changed offsets in the linked code
+
+        (JSC::LinkBuffer::applyOffset):
+          A helper function to deal with the now potentially moved labels
+
+        (JSC::LinkBuffer::linkCode):
+          The new and mighty linker function
+
+        * assembler/MacroAssemblerARMv7.h:
+        (JSC::MacroAssemblerARMv7::MacroAssemblerARMv7):
+        (JSC::MacroAssemblerARMv7::beginUninterruptedSequence):
+        (JSC::MacroAssemblerARMv7::endUninterruptedSequence):
+        (JSC::MacroAssemblerARMv7::jumpsToLink):
+        (JSC::MacroAssemblerARMv7::unlinkedCode):
+        (JSC::MacroAssemblerARMv7::computeJumpType):
+        (JSC::MacroAssemblerARMv7::convertJumpTo):
+        (JSC::MacroAssemblerARMv7::recordLinkOffsets):
+        (JSC::MacroAssemblerARMv7::jumpSizeDelta):
+        (JSC::MacroAssemblerARMv7::link):
+        (JSC::MacroAssemblerARMv7::jump):
+        (JSC::MacroAssemblerARMv7::branchMul32):
+        (JSC::MacroAssemblerARMv7::breakpoint):
+        (JSC::MacroAssemblerARMv7::nearCall):
+        (JSC::MacroAssemblerARMv7::call):
+        (JSC::MacroAssemblerARMv7::ret):
+        (JSC::MacroAssemblerARMv7::tailRecursiveCall):
+        (JSC::MacroAssemblerARMv7::executableOffsetFor):
+        (JSC::MacroAssemblerARMv7::inUninterruptedSequence):
+        (JSC::MacroAssemblerARMv7::makeJump):
+        (JSC::MacroAssemblerARMv7::makeBranch):
+           All branches need to pass on their type now
+
+        * jit/ExecutableAllocator.h:
+        (JSC::ExecutablePool::returnLastBytes):
+           We can't know ahead of time how much space will be necessary to
+           hold the linked code if we're compacting branches, this new
+           function allows us to return the unused bytes at the end of linking
+
+        * jit/JIT.cpp:
+        (JSC::JIT::JIT):
+        (JSC::JIT::privateCompile):
+        * jit/JIT.h:
+        (JSC::JIT::compile):
+           The JIT class now needs to take a linker offset so that recompilation
+           can generate the same jumps when using branch compaction.
+        * jit/JITArithmetic32_64.cpp:
+        (JSC::JIT::emitSlow_op_mod):
+        * jit/JITOpcodes.cpp:
+        (JSC::JIT::privateCompileCTIMachineTrampolines):
+        * jit/JITOpcodes32_64.cpp:
+        (JSC::JIT::privateCompileCTIMachineTrampolines):
+        (JSC::JIT::privateCompileCTINativeCall):
+          Update for new trampolineAt changes
+
+        * wtf/FastMalloc.cpp:
+        (WTF::TCMallocStats::):
+        * wtf/Platform.h:
+
 2010-08-09  Gavin Barraclough  <barraclough@apple.com>
 

trunk/JavaScriptCore/JavaScriptCore.xcodeproj/project.pbxproj

@@ -312 +312 @@
                 A7482E93116A7CAD003B0712 /* JSWeakObjectMapRefInternal.h in Headers */ = {isa = PBXBuildFile; fileRef = A7482E37116A697B003B0712 /* JSWeakObjectMapRefInternal.h */; settings = {ATTRIBUTES = (Private, ); }; };
                 A74B3499102A5F8E0032AB98 /* MarkStack.cpp in Sources */ = {isa = PBXBuildFile; fileRef = A74B3498102A5F8E0032AB98 /* MarkStack.cpp */; };
+                A74DE1D0120B875600D40D5B /* ARMv7Assembler.cpp in Sources */ = {isa = PBXBuildFile; fileRef = A74DE1CB120B86D600D40D5B /* ARMv7Assembler.cpp */; };
                 A75706DE118A2BCF0057F88F /* JITArithmetic32_64.cpp in Sources */ = {isa = PBXBuildFile; fileRef = A75706DD118A2BCF0057F88F /* JITArithmetic32_64.cpp */; };
                 A766B44F0EE8DCD1009518CA /* ExecutableAllocator.h in Headers */ = {isa = PBXBuildFile; fileRef = A7B48DB50EE74CFC00DCBDB6 /* ExecutableAllocator.h */; settings = {ATTRIBUTES = (Private, ); }; };
@@ -928 +929 @@
                 A7482E37116A697B003B0712 /* JSWeakObjectMapRefInternal.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = JSWeakObjectMapRefInternal.h; sourceTree = "<group>"; };
                 A74B3498102A5F8E0032AB98 /* MarkStack.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = MarkStack.cpp; sourceTree = "<group>"; };
+                A74DE1CB120B86D600D40D5B /* ARMv7Assembler.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = ARMv7Assembler.cpp; sourceTree = "<group>"; };
                 A75706DD118A2BCF0057F88F /* JITArithmetic32_64.cpp */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.cpp.cpp; path = JITArithmetic32_64.cpp; sourceTree = "<group>"; };
                 A76C51741182748D00715B05 /* JSInterfaceJIT.h */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.h; path = JSInterfaceJIT.h; sourceTree = "<group>"; };
@@ -1849 +1851 @@
                                 86D3B2C010156BDE002865E7 /* ARMAssembler.h */,
                                 86ADD1430FDDEA980006EEC2 /* ARMv7Assembler.h */,
+                                A74DE1CB120B86D600D40D5B /* ARMv7Assembler.cpp */,
                                 9688CB130ED12B4E001D649F /* AssemblerBuffer.h */,
                                 86D3B2C110156BDE002865E7 /* AssemblerBufferWithConstantPool.h */,
@@ -2678 +2681 @@
                                 8627E5EB11F1281900A313B5 /* PageAllocation.cpp in Sources */,
                                 DDE82AD71209D955005C1756 /* GCHandle.cpp in Sources */,
+                                A74DE1D0120B875600D40D5B /* ARMv7Assembler.cpp in Sources */,
                         );
                         runOnlyForDeploymentPostprocessing = 0;

trunk/JavaScriptCore/assembler/ARMv7Assembler.h

@@ -382 +382 @@
         u.d = d;
 
-        int sign = (u.i >> 63);
-        int exponent = (u.i >> 52) & 0x7ff;
+        int sign = static_cast<int>(u.i >> 63);
+        int exponent = static_cast<int>(u.i >> 52) & 0x7ff;
         uint64_t mantissa = u.i & 0x000fffffffffffffull;
 
@@ -445 +445 @@
     } m_u;
 };
-
 
 class ARMv7Assembler {
@@ -477 +476 @@
         ConditionLE,
         ConditionAL,
-
+        
         ConditionCS = ConditionHS,
         ConditionCC = ConditionLO,
     } Condition;
 
+    enum JumpType { JumpNoCondition, JumpCondition, JumpFullSize };
+    enum JumpLinkType { LinkInvalid, LinkShortJump, LinkConditionalShortJump, LinkLongJump, JumpTypeCount };
+    static const int JumpSizes[JumpTypeCount];
+    enum { JumpPaddingSize = 5 * sizeof(uint16_t) };
+    class LinkRecord {
+    public:
+        LinkRecord(intptr_t from, intptr_t to, JumpType type, Condition condition)
+            : m_from(from)
+            , m_to(to)
+            , m_type(type)
+            , m_linkType(LinkInvalid)
+            , m_condition(condition)
+        {
+        }
+        intptr_t from() const { return m_from; }
+        void setFrom(intptr_t from) { m_from = from; }
+        intptr_t to() const { return m_to; }
+        JumpType type() const { return m_type; }
+        JumpLinkType linkType() const { return m_linkType; }
+        void setLinkType(JumpLinkType linkType) { ASSERT(m_linkType == LinkInvalid); m_linkType = linkType; }
+        Condition condition() const { return m_condition; }
+    private:
+        intptr_t m_from : 31;
+        intptr_t m_to : 31;
+        JumpType m_type : 2;
+        JumpLinkType m_linkType : 3;
+        Condition m_condition : 16;
+    };
+    
     class JmpSrc {
         friend class ARMv7Assembler;
         friend class ARMInstructionFormatter;
+        friend class LinkBuffer;
     public:
         JmpSrc()
@@ -492 +521 @@
 
     private:
-        JmpSrc(int offset)
+        JmpSrc(int offset, JumpType type)
             : m_offset(offset)
-        {
+            , m_condition(0xffff)
+            , m_type(type)
+        {
+            ASSERT(m_type != JumpCondition);
+        }
+
+        JmpSrc(int offset, JumpType type, Condition condition)
+            : m_offset(offset)
+            , m_condition(condition)
+            , m_type(type)
+        {
+            ASSERT(m_type == JumpCondition || m_type == JumpFullSize);
         }
 
         int m_offset;
+        Condition m_condition : 16;
+        JumpType m_type : 16;
+        
     };
     
@@ -503 +546 @@
         friend class ARMv7Assembler;
         friend class ARMInstructionFormatter;
+        friend class LinkBuffer;
     public:
         JmpDst()
@@ -525 +569 @@
 
 private:
-
-    struct LinkRecord {
-        LinkRecord(intptr_t from, intptr_t to)
-            : from(from)
-            , to(to)
-        {
-        }
-
-        intptr_t from;
-        intptr_t to;
-    };
 
     // ARMv7, Appx-A.6.3
@@ -740 +773 @@
 
 public:
-
+    
     void add(RegisterID rd, RegisterID rn, ARMThumbImmediate imm)
     {
@@ -879 +912 @@
         m_formatter.twoWordOp12Reg4FourFours(OP_ASR_reg_T2, rn, FourFours(0xf, rd, 0, rm));
     }
-
+    
     // Only allowed in IT (if then) block if last instruction.
-    JmpSrc b()
+    JmpSrc b(JumpType type)
     {
         m_formatter.twoWordOp16Op16(OP_B_T4a, OP_B_T4b);
-        return JmpSrc(m_formatter.size());
+        return JmpSrc(m_formatter.size(), type);
     }
     
     // Only allowed in IT (if then) block if last instruction.
-    JmpSrc blx(RegisterID rm)
+    JmpSrc blx(RegisterID rm, JumpType type)
     {
         ASSERT(rm != ARMRegisters::pc);
         m_formatter.oneWordOp8RegReg143(OP_BLX, rm, (RegisterID)8);
-        return JmpSrc(m_formatter.size());
+        return JmpSrc(m_formatter.size(), type);
     }
 
     // Only allowed in IT (if then) block if last instruction.
-    JmpSrc bx(RegisterID rm)
+    JmpSrc bx(RegisterID rm, JumpType type, Condition condition)
     {
         m_formatter.oneWordOp8RegReg143(OP_BX, rm, (RegisterID)0);
-        return JmpSrc(m_formatter.size());
+        return JmpSrc(m_formatter.size(), type, condition);
+    }
+
+    JmpSrc bx(RegisterID rm, JumpType type)
+    {
+        m_formatter.oneWordOp8RegReg143(OP_BX, rm, (RegisterID)0);
+        return JmpSrc(m_formatter.size(), type);
     }
 
@@ -1618 +1657 @@
         return dst.m_offset - src.m_offset;
     }
+
+    int executableOffsetFor(int location)
+    {
+        if (!location)
+            return 0;
+        return static_cast<int32_t*>(m_formatter.data())[location / sizeof(int32_t) - 1];
+    }
+    
+    int jumpSizeDelta(JumpLinkType jumpLinkType) { return JumpPaddingSize - JumpSizes[jumpLinkType]; }
     
     // Assembler admin methods:
@@ -1626 +1674 @@
     }
 
-    void* executableCopy(ExecutablePool* allocator)
-    {
-        void* copy = m_formatter.executableCopy(allocator);
-        if (!copy)
-            return 0;
-
-        unsigned jumpCount = m_jumpsToLink.size();
-        for (unsigned i = 0; i < jumpCount; ++i) {
-            uint16_t* location = reinterpret_cast<uint16_t*>(reinterpret_cast<intptr_t>(copy) + m_jumpsToLink[i].from);
-            uint16_t* target = reinterpret_cast<uint16_t*>(reinterpret_cast<intptr_t>(copy) + m_jumpsToLink[i].to);
-            linkJumpAbsolute(location, target);
-        }
-        m_jumpsToLink.clear();
-
-        return copy;
-    }
-
+    static bool linkRecordSourceComparator(const LinkRecord& a, const LinkRecord& b)
+    {
+        return a.from() < b.from();
+    }
+
+    JumpLinkType computeJumpType(LinkRecord& record, const uint8_t* from, const uint8_t* to)
+    {
+        if (record.type() >= JumpFullSize) {
+            record.setLinkType(LinkLongJump);
+            return LinkLongJump;
+        }
+        bool mayTriggerErrata = false;
+        const uint16_t* shortJumpLocation = reinterpret_cast<const uint16_t*>(from  - (JumpPaddingSize - JumpSizes[LinkShortJump]));
+        if (!canBeShortJump(shortJumpLocation, to, mayTriggerErrata)) {
+            record.setLinkType(LinkLongJump);
+            return LinkLongJump;
+        }
+        if (mayTriggerErrata) {
+            record.setLinkType(LinkLongJump);
+            return LinkLongJump;
+        }
+        if (record.type() == JumpCondition) {
+            record.setLinkType(LinkConditionalShortJump);
+            return LinkConditionalShortJump;
+        }
+        record.setLinkType(LinkShortJump);
+        return LinkShortJump;
+    }
+
+    void recordLinkOffsets(int32_t regionStart, int32_t regionEnd, int32_t offset)
+    {
+        int32_t ptr = regionStart / sizeof(int32_t);
+        const int32_t end = regionEnd / sizeof(int32_t);
+        int32_t* offsets = static_cast<int32_t*>(m_formatter.data());
+        while (ptr < end)
+            offsets[ptr++] = offset;
+    }
+    
+    Vector<LinkRecord>& jumpsToLink()
+    {
+        std::sort(m_jumpsToLink.begin(), m_jumpsToLink.end(), linkRecordSourceComparator);
+        return m_jumpsToLink;
+    }
+
+    void link(LinkRecord& record, uint8_t* from, uint8_t* to)
+    {
+        uint16_t* itttLocation;
+        if (record.linkType() == LinkConditionalShortJump) {
+            itttLocation = reinterpret_cast<uint16_t*>(from - JumpSizes[LinkConditionalShortJump] - 2);
+            itttLocation[0] = ifThenElse(record.condition()) | OP_IT;
+        }
+        ASSERT(record.linkType() != LinkInvalid);
+        if (record.linkType() != LinkLongJump)
+            linkShortJump(reinterpret_cast<uint16_t*>(from), to);
+        else
+            linkLongJump(reinterpret_cast<uint16_t*>(from), to);
+    }
+
+    void* unlinkedCode() { return m_formatter.data(); }
+    
     static unsigned getCallReturnOffset(JmpSrc call)
     {
@@ -1661 +1752 @@
         ASSERT(to.m_offset != -1);
         ASSERT(from.m_offset != -1);
-        m_jumpsToLink.append(LinkRecord(from.m_offset, to.m_offset));
+        m_jumpsToLink.append(LinkRecord(from.m_offset, to.m_offset, from.m_type, from.m_condition));
     }
 
@@ -1864 +1955 @@
     }
 
-    static void linkJumpAbsolute(uint16_t* instruction, void* target)
-    {
-        // FIMXE: this should be up in the MacroAssembler layer. :-(
-        const uint16_t JUMP_TEMPORARY_REGISTER = ARMRegisters::ip;
-
+    static bool canBeShortJump(const uint16_t* instruction, const void* target, bool& mayTriggerErrata)
+    {
         ASSERT(!(reinterpret_cast<intptr_t>(instruction) & 1));
         ASSERT(!(reinterpret_cast<intptr_t>(target) & 1));
-
-        ASSERT( (isMOV_imm_T3(instruction - 5) && isMOVT(instruction - 3) && isBX(instruction - 1))
-            || (isNOP_T1(instruction - 5) && isNOP_T2(instruction - 4) && isB(instruction - 2)) );
-
+        
         intptr_t relative = reinterpret_cast<intptr_t>(target) - (reinterpret_cast<intptr_t>(instruction));
-
         // From Cortex-A8 errata:
         // If the 32-bit Thumb-2 branch instruction spans two 4KiB regions and
@@ -1885 +1969 @@
         // The instruction is spanning two pages if it ends at an address ending 0x002
         bool spansTwo4K = ((reinterpret_cast<intptr_t>(instruction) & 0xfff) == 0x002);
+        mayTriggerErrata = spansTwo4K;
         // The target is in the first page if the jump branch back by [3..0x1002] bytes
         bool targetInFirstPage = (relative >= -0x1002) && (relative < -2);
         bool wouldTriggerA8Errata = spansTwo4K && targetInFirstPage;
-
-        if (((relative << 7) >> 7) == relative && !wouldTriggerA8Errata) {
+        return ((relative << 7) >> 7) == relative && !wouldTriggerA8Errata;
+    }
+
+    static void linkLongJump(uint16_t* instruction, void* target)
+    {
+        linkJumpAbsolute(instruction, target);
+    }
+    
+    static void linkShortJump(uint16_t* instruction, void* target)
+    {
+        // FIMXE: this should be up in the MacroAssembler layer. :-(        
+        ASSERT(!(reinterpret_cast<intptr_t>(instruction) & 1));
+        ASSERT(!(reinterpret_cast<intptr_t>(target) & 1));
+        
+        intptr_t relative = reinterpret_cast<intptr_t>(target) - (reinterpret_cast<intptr_t>(instruction));
+        bool scratch;
+        UNUSED_PARAM(scratch);
+        ASSERT(canBeShortJump(instruction, target, scratch));
+        // ARM encoding for the top two bits below the sign bit is 'peculiar'.
+        if (relative >= 0)
+            relative ^= 0xC00000;
+
+        // All branch offsets should be an even distance.
+        ASSERT(!(relative & 1));
+        instruction[-2] = OP_B_T4a | ((relative & 0x1000000) >> 14) | ((relative & 0x3ff000) >> 12);
+        instruction[-1] = OP_B_T4b | ((relative & 0x800000) >> 10) | ((relative & 0x400000) >> 11) | ((relative & 0xffe) >> 1);
+    }
+
+    static void linkJumpAbsolute(uint16_t* instruction, void* target)
+    {
+        // FIMXE: this should be up in the MacroAssembler layer. :-(
+        ASSERT(!(reinterpret_cast<intptr_t>(instruction) & 1));
+        ASSERT(!(reinterpret_cast<intptr_t>(target) & 1));
+
+        ASSERT((isMOV_imm_T3(instruction - 5) && isMOVT(instruction - 3) && isBX(instruction - 1))
+            || (isNOP_T1(instruction - 5) && isNOP_T2(instruction - 4) && isB(instruction - 2)));
+
+        intptr_t relative = reinterpret_cast<intptr_t>(target) - (reinterpret_cast<intptr_t>(instruction));
+        bool scratch;
+        if (canBeShortJump(instruction, target, scratch)) {
             // ARM encoding for the top two bits below the sign bit is 'peculiar'.
             if (relative >= 0)
@@ -1907 +2030 @@
             instruction[-1] = OP_B_T4b | ((relative & 0x800000) >> 10) | ((relative & 0x400000) >> 11) | ((relative & 0xffe) >> 1);
         } else {
+            const uint16_t JUMP_TEMPORARY_REGISTER = ARMRegisters::ip;
             ARMThumbImmediate lo16 = ARMThumbImmediate::makeUInt16(static_cast<uint16_t>(reinterpret_cast<uint32_t>(target) + 1));
             ARMThumbImmediate hi16 = ARMThumbImmediate::makeUInt16(static_cast<uint16_t>(reinterpret_cast<uint32_t>(target) >> 16));
@@ -1921 +2045 @@
         return op | (imm.m_value.i << 10) | imm.m_value.imm4;
     }
+
     static uint16_t twoWordOp5i6Imm4Reg4EncodedImmSecond(uint16_t rd, ARMThumbImmediate imm)
     {
@@ -2037 +2162 @@
 
     Vector<LinkRecord> m_jumpsToLink;
+    Vector<int32_t> m_offsets;
 };
 

trunk/JavaScriptCore/assembler/AbstractMacroAssembler.h

@@ -419 +419 @@
 
     // Section 3: Misc admin methods
-
-    static CodePtr trampolineAt(CodeRef ref, Label label)
-    {
-        return CodePtr(AssemblerType::getRelocatedAddress(ref.m_code.dataLocation(), label.m_label));
-    }
-
     size_t size()
     {
@@ -480 +474 @@
         return AssemblerType::getDifferenceBetweenLabels(from.m_label, to.m_jmp);
     }
+    
+    void beginUninterruptedSequence() { }
+    void endUninterruptedSequence() { }
 
 protected:

trunk/JavaScriptCore/assembler/LinkBuffer.h

@@ -50 +50 @@
 class LinkBuffer : public Noncopyable {
     typedef MacroAssemblerCodeRef CodeRef;
+    typedef MacroAssemblerCodePtr CodePtr;
     typedef MacroAssembler::Label Label;
     typedef MacroAssembler::Jump Jump;
@@ -56 +57 @@
     typedef MacroAssembler::DataLabel32 DataLabel32;
     typedef MacroAssembler::DataLabelPtr DataLabelPtr;
+    typedef MacroAssembler::JmpDst JmpDst;
+#if ENABLE(BRANCH_COMPACTION)
+    typedef MacroAssembler::LinkRecord LinkRecord;
+    typedef MacroAssembler::JumpLinkType JumpLinkType;
+#endif
 
     enum LinkBufferState {
@@ -67 +73 @@
     //       First, executablePool is copied into m_executablePool, then the initialization of
     //       m_code uses m_executablePool, *not* executablePool, since this is no longer valid.
-    LinkBuffer(MacroAssembler* masm, PassRefPtr<ExecutablePool> executablePool)
+    // The linkOffset parameter should only be non-null when recompiling for exception info
+    LinkBuffer(MacroAssembler* masm, PassRefPtr<ExecutablePool> executablePool, void* linkOffset)
         : m_executablePool(executablePool)
-        , m_code(masm->m_assembler.executableCopy(m_executablePool.get()))
-        , m_size(masm->m_assembler.size())
+        , m_size(0)
+        , m_code(0)
+        , m_assembler(masm)
 #ifndef NDEBUG
         , m_state(StateInit)
 #endif
     {
+        linkCode(linkOffset);
     }
 
@@ -98 +107 @@
     {
         ASSERT(call.isFlagSet(Call::Linkable));
+        call.m_jmp = applyOffset(call.m_jmp);
         MacroAssembler::linkCall(code(), call, function);
     }
@@ -103 +113 @@
     void link(Jump jump, CodeLocationLabel label)
     {
+        jump.m_jmp = applyOffset(jump.m_jmp);
         MacroAssembler::linkJump(code(), jump, label);
     }
@@ -109 +120 @@
     {
         for (unsigned i = 0; i < list.m_jumps.size(); ++i)
-            MacroAssembler::linkJump(code(), list.m_jumps[i], label);
+            link(list.m_jumps[i], label);
     }
 
     void patch(DataLabelPtr label, void* value)
     {
-        MacroAssembler::linkPointer(code(), label.m_label, value);
+        JmpDst target = applyOffset(label.m_label);
+        MacroAssembler::linkPointer(code(), target, value);
     }
 
     void patch(DataLabelPtr label, CodeLocationLabel value)
     {
-        MacroAssembler::linkPointer(code(), label.m_label, value.executableAddress());
+        JmpDst target = applyOffset(label.m_label);
+        MacroAssembler::linkPointer(code(), target, value.executableAddress());
     }
 
@@ -128 +141 @@
         ASSERT(call.isFlagSet(Call::Linkable));
         ASSERT(!call.isFlagSet(Call::Near));
-        return CodeLocationCall(MacroAssembler::getLinkerAddress(code(), call.m_jmp));
+        return CodeLocationCall(MacroAssembler::getLinkerAddress(code(), applyOffset(call.m_jmp)));
     }
 
@@ -135 +148 @@
         ASSERT(call.isFlagSet(Call::Linkable));
         ASSERT(call.isFlagSet(Call::Near));
-        return CodeLocationNearCall(MacroAssembler::getLinkerAddress(code(), call.m_jmp));
+        return CodeLocationNearCall(MacroAssembler::getLinkerAddress(code(), applyOffset(call.m_jmp)));
     }
 
     CodeLocationLabel locationOf(Label label)
     {
-        return CodeLocationLabel(MacroAssembler::getLinkerAddress(code(), label.m_label));
+        return CodeLocationLabel(MacroAssembler::getLinkerAddress(code(), applyOffset(label.m_label)));
     }
 
     CodeLocationDataLabelPtr locationOf(DataLabelPtr label)
     {
-        return CodeLocationDataLabelPtr(MacroAssembler::getLinkerAddress(code(), label.m_label));
+        return CodeLocationDataLabelPtr(MacroAssembler::getLinkerAddress(code(), applyOffset(label.m_label)));
     }
 
     CodeLocationDataLabel32 locationOf(DataLabel32 label)
     {
-        return CodeLocationDataLabel32(MacroAssembler::getLinkerAddress(code(), label.m_label));
+        return CodeLocationDataLabel32(MacroAssembler::getLinkerAddress(code(), applyOffset(label.m_label)));
     }
 
@@ -157 +170 @@
     unsigned returnAddressOffset(Call call)
     {
+        call.m_jmp = applyOffset(call.m_jmp);
         return MacroAssembler::getLinkerCallReturnOffset(call);
     }
@@ -170 +184 @@
         return CodeRef(m_code, m_executablePool, m_size);
     }
+
     CodeLocationLabel finalizeCodeAddendum()
     {
@@ -177 +192 @@
     }
 
+    CodePtr trampolineAt(Label label)
+    {
+        return CodePtr(MacroAssembler::AssemblerType_T::getRelocatedAddress(code(), applyOffset(label.m_label)));
+    }
+
 private:
+    template <typename T> T applyOffset(T src)
+    {
+#if ENABLE(BRANCH_COMPACTION)
+        src.m_offset -= m_assembler->executableOffsetFor(src.m_offset);
+#endif
+        return src;
+    }
+    
     // Keep this private! - the underlying code should only be obtained externally via 
     // finalizeCode() or finalizeCodeAddendum().
@@ -185 +213 @@
     }
 
+    void linkCode(void* linkOffset)
+    {
+        UNUSED_PARAM(linkOffset);
+        ASSERT(!m_code);
+#if !ENABLE(BRANCH_COMPACTION)
+        m_code = m_assembler->m_assembler.executableCopy(m_executablePool.get());
+        m_size = m_assembler->size();
+#else
+        size_t initialSize = m_assembler->size();
+        m_code = (uint8_t*)m_executablePool->alloc(initialSize);
+        if (!m_code)
+            return;
+        ExecutableAllocator::makeWritable(m_code, m_assembler->size());
+        uint8_t* inData = (uint8_t*)m_assembler->unlinkedCode();
+        uint8_t* outData = reinterpret_cast<uint8_t*>(m_code);
+        const uint8_t* linkBase = linkOffset ? reinterpret_cast<uint8_t*>(linkOffset) : outData;
+        int readPtr = 0;
+        int writePtr = 0;
+        Vector<LinkRecord>& jumpsToLink = m_assembler->jumpsToLink();
+        unsigned jumpCount = jumpsToLink.size();
+        for (unsigned i = 0; i < jumpCount; ++i) {
+            int offset = readPtr - writePtr;
+            ASSERT(!(offset & 1));
+            
+            // Copy the instructions from the last jump to the current one.
+            size_t regionSize = jumpsToLink[i].from() - readPtr;
+            memcpy(outData + writePtr, inData + readPtr, regionSize);
+            m_assembler->recordLinkOffsets(readPtr, jumpsToLink[i].from(), offset);
+            readPtr += regionSize;
+            writePtr += regionSize;
+            
+            // Calculate absolute address of the jump target, in the case of backwards
+            // branches we need to be precise, forward branches we are pessimistic
+            const uint8_t* target;
+            if (jumpsToLink[i].to() >= jumpsToLink[i].from())
+                target = linkBase + jumpsToLink[i].to() - offset; // Compensate for what we have collapsed so far
+            else
+                target = linkBase + jumpsToLink[i].to() - m_assembler->executableOffsetFor(jumpsToLink[i].to());
+            
+            JumpLinkType jumpLinkType = m_assembler->computeJumpType(jumpsToLink[i], linkBase + writePtr, target);
+
+            // Step back in the write stream
+            int32_t delta = m_assembler->jumpSizeDelta(jumpLinkType);
+            if (delta) {
+                writePtr -= delta;
+                m_assembler->recordLinkOffsets(jumpsToLink[i].from() - delta, readPtr, readPtr - writePtr);
+            }
+            jumpsToLink[i].setFrom(writePtr);
+        }
+        // Copy everything after the last jump
+        memcpy(outData + writePtr, inData + readPtr, m_assembler->size() - readPtr);
+        m_assembler->recordLinkOffsets(readPtr, m_assembler->size(), readPtr - writePtr);
+        
+        // Actually link everything (don't link if we've be given a linkoffset as it's a
+        // waste of time: linkOffset is used for recompiling to get exception info)
+        if (!linkOffset) {
+            for (unsigned i = 0; i < jumpCount; ++i) {
+                uint8_t* location = outData + jumpsToLink[i].from();
+                uint8_t* target = outData + jumpsToLink[i].to() - m_assembler->executableOffsetFor(jumpsToLink[i].to());
+                m_assembler->link(jumpsToLink[i], location, target);
+            }
+        }
+
+        jumpsToLink.clear();
+        m_size = writePtr + m_assembler->size() - readPtr;
+        m_executablePool->returnLastBytes(initialSize - m_size);
+#endif
+    }
+
     void performFinalization()
     {
@@ -197 +294 @@
 
     RefPtr<ExecutablePool> m_executablePool;
+    size_t m_size;
     void* m_code;
-    size_t m_size;
+    MacroAssembler* m_assembler;
 #ifndef NDEBUG
     LinkBufferState m_state;

trunk/JavaScriptCore/assembler/MacroAssemblerARMv7.h

@@ -46 +46 @@
 
 public:
+    typedef ARMv7Assembler::LinkRecord LinkRecord;
+    typedef ARMv7Assembler::JumpLinkType JumpLinkType;
+
+    MacroAssemblerARMv7()
+        : m_inUninterruptedSequence(false)
+    {
+    }
+    
+    void beginUninterruptedSequence() { m_inUninterruptedSequence = true; }
+    void endUninterruptedSequence() { m_inUninterruptedSequence = false; }
+    Vector<LinkRecord>& jumpsToLink() { return m_assembler.jumpsToLink(); }
+    void* unlinkedCode() { return m_assembler.unlinkedCode(); }
+    JumpLinkType computeJumpType(LinkRecord& record, const uint8_t* from, const uint8_t* to) { return m_assembler.computeJumpType(record, from, to); }
+    void recordLinkOffsets(int32_t regionStart, int32_t regionEnd, int32_t offset) {return m_assembler.recordLinkOffsets(regionStart, regionEnd, offset); }
+    int jumpSizeDelta(JumpLinkType jumpLinkType) { return m_assembler.jumpSizeDelta(jumpLinkType); }
+    void link(LinkRecord& record, uint8_t* from, uint8_t* to) { return m_assembler.link(record, from, to); }
+
     struct ArmAddress {
         enum AddressType {
@@ -970 +987 @@
     void jump(RegisterID target)
     {
-        m_assembler.bx(target);
+        m_assembler.bx(target, inUninterruptedSequence() ? ARMv7Assembler::JumpFullSize : ARMv7Assembler::JumpNoCondition);
     }
 
@@ -977 +994 @@
     {
         load32(address, dataTempRegister);
-        m_assembler.bx(dataTempRegister);
+        m_assembler.bx(dataTempRegister, inUninterruptedSequence() ? ARMv7Assembler::JumpFullSize : ARMv7Assembler::JumpNoCondition);
     }
 
@@ -1013 +1030 @@
     Jump branchMul32(Condition cond, RegisterID src, RegisterID dest)
     {
-        ASSERT(cond == Overflow);
+        ASSERT_UNUSED(cond, cond == Overflow);
         m_assembler.smull(dest, dataTempRegister, dest, src);
         m_assembler.asr(addressTempRegister, dest, 31);
@@ -1021 +1038 @@
     Jump branchMul32(Condition cond, Imm32 imm, RegisterID src, RegisterID dest)
     {
-        ASSERT(cond == Overflow);
+        ASSERT_UNUSED(cond, cond == Overflow);
         move(imm, dataTempRegister);
         m_assembler.smull(dest, dataTempRegister, src, dataTempRegister);
@@ -1060 +1077 @@
     void breakpoint()
     {
-        m_assembler.bkpt();
+        m_assembler.bkpt(0);
     }
 
@@ -1066 +1083 @@
     {
         moveFixedWidthEncoding(Imm32(0), dataTempRegister);
-        return Call(m_assembler.blx(dataTempRegister), Call::LinkableNear);
+        return Call(m_assembler.blx(dataTempRegister, ARMv7Assembler::JumpFullSize), Call::LinkableNear);
     }
 
@@ -1072 +1089 @@
     {
         moveFixedWidthEncoding(Imm32(0), dataTempRegister);
-        return Call(m_assembler.blx(dataTempRegister), Call::Linkable);
+        return Call(m_assembler.blx(dataTempRegister, ARMv7Assembler::JumpFullSize), Call::Linkable);
     }
 
     Call call(RegisterID target)
     {
-        return Call(m_assembler.blx(target), Call::None);
+        return Call(m_assembler.blx(target, ARMv7Assembler::JumpFullSize), Call::None);
     }
 
@@ -1083 +1100 @@
     {
         load32(address, dataTempRegister);
-        return Call(m_assembler.blx(dataTempRegister), Call::None);
+        return Call(m_assembler.blx(dataTempRegister, ARMv7Assembler::JumpFullSize), Call::None);
     }
 
     void ret()
     {
-        m_assembler.bx(linkRegister);
+        m_assembler.bx(linkRegister, ARMv7Assembler::JumpFullSize);
     }
 
@@ -1188 +1205 @@
         // Like a normal call, but don't link.
         moveFixedWidthEncoding(Imm32(0), dataTempRegister);
-        return Call(m_assembler.bx(dataTempRegister), Call::Linkable);
+        return Call(m_assembler.bx(dataTempRegister, ARMv7Assembler::JumpFullSize), Call::Linkable);
     }
 
@@ -1197 +1214 @@
     }
 
+    
+    int executableOffsetFor(int location)
+    {
+        return m_assembler.executableOffsetFor(location);
+    }
 
 protected:
+    bool inUninterruptedSequence()
+    {
+        return m_inUninterruptedSequence;
+    }
+
     ARMv7Assembler::JmpSrc makeJump()
     {
         moveFixedWidthEncoding(Imm32(0), dataTempRegister);
-        return m_assembler.bx(dataTempRegister);
+        return m_assembler.bx(dataTempRegister, inUninterruptedSequence() ? ARMv7Assembler::JumpFullSize : ARMv7Assembler::JumpNoCondition);
     }
 
@@ -1209 +1236 @@
         m_assembler.it(cond, true, true);
         moveFixedWidthEncoding(Imm32(0), dataTempRegister);
-        return m_assembler.bx(dataTempRegister);
+        return m_assembler.bx(dataTempRegister, inUninterruptedSequence() ? ARMv7Assembler::JumpFullSize : ARMv7Assembler::JumpCondition, cond);
     }
     ARMv7Assembler::JmpSrc makeBranch(Condition cond) { return makeBranch(armV7Condition(cond)); }
@@ -1299 +1326 @@
         ARMv7Assembler::relinkCall(call.dataLocation(), destination.executableAddress());
     }
+    
+    bool m_inUninterruptedSequence;
 };
 

trunk/JavaScriptCore/jit/ExecutableAllocator.h

@@ -129 +129 @@
     }
     
+    void returnLastBytes(size_t count)
+    {
+        m_freePtr -= count;
+    }
+
     ~ExecutablePool()
     {

trunk/JavaScriptCore/jit/JIT.cpp

@@ -72 +72 @@
 }
 
-JIT::JIT(JSGlobalData* globalData, CodeBlock* codeBlock)
+JIT::JIT(JSGlobalData* globalData, CodeBlock* codeBlock, void* linkerOffset)
     : m_interpreter(globalData->interpreter)
     , m_globalData(globalData)
@@ -90 +90 @@
     , m_jumpTargetsPosition(0)
 #endif
+    , m_linkerOffset(linkerOffset)
 {
 }
@@ -512 +513 @@
     if (!executablePool)
         return JITCode();
-    LinkBuffer patchBuffer(this, executablePool.release());
+    LinkBuffer patchBuffer(this, executablePool.release(), m_linkerOffset);
     if (!patchBuffer.allocationSuccessful())
         return JITCode();

trunk/JavaScriptCore/jit/JIT.h

@@ -179 +179 @@
 
     public:
-        static JITCode compile(JSGlobalData* globalData, CodeBlock* codeBlock, CodePtr* functionEntryArityCheck = 0)
-        {
-            return JIT(globalData, codeBlock).privateCompile(functionEntryArityCheck);
+        static JITCode compile(JSGlobalData* globalData, CodeBlock* codeBlock, CodePtr* functionEntryArityCheck = 0, void* offsetBase = 0)
+        {
+            return JIT(globalData, codeBlock, offsetBase).privateCompile(functionEntryArityCheck);
         }
 
@@ -222 +222 @@
             if (!globalData->canUseJIT())
                 return;
-            JIT jit(globalData);
+            JIT jit(globalData, 0, 0);
             jit.privateCompileCTIMachineTrampolines(executablePool, globalData, trampolines);
         }
@@ -230 +230 @@
             if (!globalData->canUseJIT())
                 return CodePtr();
-            JIT jit(globalData);
+            JIT jit(globalData, 0, 0);
             return jit.privateCompileCTINativeCall(executablePool, globalData, func);
         }
@@ -260 +260 @@
         };
 
-        JIT(JSGlobalData*, CodeBlock* = 0);
+        JIT(JSGlobalData*, CodeBlock* = 0, void* = 0);
 
         void privateCompileMainPass();
@@ -667 +667 @@
 #endif // USE(JSVALUE32_64)
 
-#if defined(ASSEMBLER_HAS_CONSTANT_POOL) && ASSEMBLER_HAS_CONSTANT_POOL
-#define BEGIN_UNINTERRUPTED_SEQUENCE(name) beginUninterruptedSequence(name ## InstructionSpace, name ## ConstantSpace)
-#define END_UNINTERRUPTED_SEQUENCE(name) endUninterruptedSequence(name ## InstructionSpace, name ## ConstantSpace)
+#if (defined(ASSEMBLER_HAS_CONSTANT_POOL) && ASSEMBLER_HAS_CONSTANT_POOL)
+#define BEGIN_UNINTERRUPTED_SEQUENCE(name) do { beginUninterruptedSequence(); beginUninterruptedSequence(name ## InstructionSpace, name ## ConstantSpace); } while (false)
+#define END_UNINTERRUPTED_SEQUENCE(name) do { endUninterruptedSequence(name ## InstructionSpace, name ## ConstantSpace); endUninterruptedSequence(); } while (false)
 
         void beginUninterruptedSequence(int, int);
@@ -675 +675 @@
 
 #else
-#define BEGIN_UNINTERRUPTED_SEQUENCE(name)
-#define END_UNINTERRUPTED_SEQUENCE(name)
+#define BEGIN_UNINTERRUPTED_SEQUENCE(name)  do { beginUninterruptedSequence(); } while (false)
+#define END_UNINTERRUPTED_SEQUENCE(name)  do { endUninterruptedSequence(); } while (false)
 #endif
 
@@ -941 +941 @@
 #endif
 #endif
+        void* m_linkerOffset;
         static CodePtr stringGetByValStubGenerator(JSGlobalData* globalData, ExecutablePool* pool);
     } JIT_CLASS_ALIGNMENT;

trunk/JavaScriptCore/jit/JITArithmetic32_64.cpp

@@ -1384 +1384 @@
 void JIT::emitSlow_op_mod(Instruction* currentInstruction, Vector<SlowCaseEntry>::iterator& iter)
 {
+    UNUSED_PARAM(currentInstruction);
+    UNUSED_PARAM(iter);
 #if ENABLE(JIT_USE_SOFT_MODULO)
     unsigned result = currentInstruction[1].u.operand;

trunk/JavaScriptCore/jit/JITOpcodes.cpp

@@ -166 +166 @@
     if (!*executablePool)
         CRASH();
-    LinkBuffer patchBuffer(this, *executablePool);
+    LinkBuffer patchBuffer(this, *executablePool, 0);
     // We can't run without the JIT trampolines!
     if (!patchBuffer.allocationSuccessful())
@@ -185 +185 @@
     CodeRef finalCode = patchBuffer.finalizeCode();
 
-    trampolines->ctiVirtualCallLink = trampolineAt(finalCode, virtualCallLinkBegin);
-    trampolines->ctiVirtualConstructLink = trampolineAt(finalCode, virtualConstructLinkBegin);
-    trampolines->ctiVirtualCall = trampolineAt(finalCode, virtualCallBegin);
-    trampolines->ctiVirtualConstruct = trampolineAt(finalCode, virtualConstructBegin);
-    trampolines->ctiNativeCall = trampolineAt(finalCode, nativeCallThunk);
-    trampolines->ctiNativeConstruct = trampolineAt(finalCode, nativeConstructThunk);
+    trampolines->ctiVirtualCallLink = patchBuffer.trampolineAt(virtualCallLinkBegin);
+    trampolines->ctiVirtualConstructLink = patchBuffer.trampolineAt(virtualConstructLinkBegin);
+    trampolines->ctiVirtualCall = patchBuffer.trampolineAt(virtualCallBegin);
+    trampolines->ctiVirtualConstruct = patchBuffer.trampolineAt(virtualConstructBegin);
+    trampolines->ctiNativeCall = patchBuffer.trampolineAt(nativeCallThunk);
+    trampolines->ctiNativeConstruct = patchBuffer.trampolineAt(nativeConstructThunk);
 #if ENABLE(JIT_USE_SOFT_MODULO)
-    trampolines->ctiSoftModulo = trampolineAt(finalCode, softModBegin);
+    trampolines->ctiSoftModulo = patchBuffer.trampolineAt(softModBegin);
 #endif
 #if ENABLE(JIT_OPTIMIZE_PROPERTY_ACCESS)
-    trampolines->ctiStringLengthTrampoline = trampolineAt(finalCode, stringLengthBegin);
+    trampolines->ctiStringLengthTrampoline = patchBuffer.trampolineAt(stringLengthBegin);
 #endif
 }

trunk/JavaScriptCore/jit/JITOpcodes32_64.cpp

@@ -164 +164 @@
     if (!*executablePool)
         CRASH();
-    LinkBuffer patchBuffer(this, *executablePool);
+    LinkBuffer patchBuffer(this, *executablePool, 0);
     // We can't run without the JIT trampolines!
     if (!patchBuffer.allocationSuccessful())
@@ -183 +183 @@
     CodeRef finalCode = patchBuffer.finalizeCode();
 
-    trampolines->ctiVirtualCall = trampolineAt(finalCode, virtualCallBegin);
-    trampolines->ctiVirtualConstruct = trampolineAt(finalCode, virtualConstructBegin);
-    trampolines->ctiNativeCall = trampolineAt(finalCode, nativeCallThunk);
-    trampolines->ctiNativeConstruct = trampolineAt(finalCode, nativeConstructThunk);
+    trampolines->ctiVirtualCall = patchBuffer.trampolineAt(virtualCallBegin);
+    trampolines->ctiVirtualConstruct = patchBuffer.trampolineAt(virtualConstructBegin);
+    trampolines->ctiNativeCall = patchBuffer.trampolineAt(nativeCallThunk);
+    trampolines->ctiNativeConstruct = patchBuffer.trampolineAt(nativeConstructThunk);
 #if ENABLE(JIT_OPTIMIZE_PROPERTY_ACCESS)
-    trampolines->ctiStringLengthTrampoline = trampolineAt(finalCode, stringLengthBegin);
+    trampolines->ctiStringLengthTrampoline = patchBuffer.trampolineAt(stringLengthBegin);
 #endif
 #if ENABLE(JIT_OPTIMIZE_CALL)
-    trampolines->ctiVirtualCallLink = trampolineAt(finalCode, virtualCallLinkBegin);
-    trampolines->ctiVirtualConstructLink = trampolineAt(finalCode, virtualConstructLinkBegin);
+    trampolines->ctiVirtualCallLink = patchBuffer.trampolineAt(virtualCallLinkBegin);
+    trampolines->ctiVirtualConstructLink = patchBuffer.trampolineAt(virtualConstructLinkBegin);
 #endif
 #if ENABLE(JIT_USE_SOFT_MODULO)
-    trampolines->ctiSoftModulo = trampolineAt(finalCode, softModBegin);
+    trampolines->ctiSoftModulo = patchBuffer.trampolineAt(softModBegin);
 #endif
 }
@@ -363 +363 @@
 
     // All trampolines constructed! copy the code, link up calls, and set the pointers on the Machine object.
-    LinkBuffer patchBuffer(this, executablePool);
+    LinkBuffer patchBuffer(this, executablePool, 0);
     // We can't continue if we can't call a function!
     if (!patchBuffer.allocationSuccessful())
@@ -369 +369 @@
 
     patchBuffer.link(nativeCall, FunctionPtr(func));
-
-    CodeRef finalCode = patchBuffer.finalizeCode();
-    return trampolineAt(finalCode, nativeCallThunk);
+    patchBuffer.finalizeCode();
+
+    return patchBuffer.trampolineAt(nativeCallThunk);
 }
 

trunk/JavaScriptCore/jit/JITPropertyAccess.cpp

@@ -78 +78 @@
     jit.ret();
     
-    LinkBuffer patchBuffer(&jit, pool);
+    LinkBuffer patchBuffer(&jit, pool, 0);
     // We can't run without the JIT trampolines!
     if (!patchBuffer.allocationSuccessful())
@@ -653 +653 @@
     Call failureCall = tailRecursiveCall();
 
-    LinkBuffer patchBuffer(this, m_codeBlock->executablePool());
+    LinkBuffer patchBuffer(this, m_codeBlock->executablePool(), 0);
     if (!patchBuffer.allocationSuccessful())
         return false;
@@ -744 +744 @@
     Jump success = jump();
 
-    LinkBuffer patchBuffer(this, m_codeBlock->executablePool());
+    LinkBuffer patchBuffer(this, m_codeBlock->executablePool(), 0);
     if (!patchBuffer.allocationSuccessful())
         return false;
@@ -810 +810 @@
         compileGetDirectOffset(protoObject, regT1, regT0, cachedOffset);
     Jump success = jump();
-    LinkBuffer patchBuffer(this, m_codeBlock->executablePool());
+    LinkBuffer patchBuffer(this, m_codeBlock->executablePool(), 0);
     if (!patchBuffer.allocationSuccessful())
         return false;
@@ -873 +873 @@
     Jump success = jump();
 
-    LinkBuffer patchBuffer(this, m_codeBlock->executablePool());
+    LinkBuffer patchBuffer(this, m_codeBlock->executablePool(), 0);
     if (!patchBuffer.allocationSuccessful())
         return false;
@@ -951 +951 @@
     Jump success = jump();
 
-    LinkBuffer patchBuffer(this, m_codeBlock->executablePool());
+    LinkBuffer patchBuffer(this, m_codeBlock->executablePool(), 0);
     if (!patchBuffer.allocationSuccessful())
         return false;
@@ -1027 +1027 @@
     Jump success = jump();
 
-    LinkBuffer patchBuffer(this, m_codeBlock->executablePool());
+    LinkBuffer patchBuffer(this, m_codeBlock->executablePool(), 0);
     if (!patchBuffer.allocationSuccessful())
         return false;
@@ -1101 +1101 @@
     Jump success = jump();
 
-    LinkBuffer patchBuffer(this, m_codeBlock->executablePool());
+    LinkBuffer patchBuffer(this, m_codeBlock->executablePool(), 0);
     if (!patchBuffer.allocationSuccessful())
         return false;

trunk/JavaScriptCore/jit/JITPropertyAccess32_64.cpp

@@ -296 +296 @@
     jit.ret();
     
-    LinkBuffer patchBuffer(&jit, pool);
+    LinkBuffer patchBuffer(&jit, pool, 0);
     // We can't run without the JIT trampolines!
     if (!patchBuffer.allocationSuccessful())
@@ -657 +657 @@
     Call failureCall = tailRecursiveCall();
     
-    LinkBuffer patchBuffer(this, m_codeBlock->executablePool());
+    LinkBuffer patchBuffer(this, m_codeBlock->executablePool(), 0);
     if (!patchBuffer.allocationSuccessful())
         return false;
@@ -753 +753 @@
     Jump success = jump();
     
-    LinkBuffer patchBuffer(this, m_codeBlock->executablePool());
+    LinkBuffer patchBuffer(this, m_codeBlock->executablePool(), 0);
     if (!patchBuffer.allocationSuccessful())
         return false;
@@ -820 +820 @@
     Jump success = jump();
     
-    LinkBuffer patchBuffer(this, m_codeBlock->executablePool());
+    LinkBuffer patchBuffer(this, m_codeBlock->executablePool(), 0);
     if (!patchBuffer.allocationSuccessful())
         return false;
@@ -887 +887 @@
     Jump success = jump();
     
-    LinkBuffer patchBuffer(this, m_codeBlock->executablePool());
+    LinkBuffer patchBuffer(this, m_codeBlock->executablePool(), 0);
     if (!patchBuffer.allocationSuccessful())
         return false;
@@ -965 +965 @@
     Jump success = jump();
     
-    LinkBuffer patchBuffer(this, m_codeBlock->executablePool());
+    LinkBuffer patchBuffer(this, m_codeBlock->executablePool(), 0);
     if (!patchBuffer.allocationSuccessful())
         return false;
@@ -1042 +1042 @@
     Jump success = jump();
     
-    LinkBuffer patchBuffer(this, m_codeBlock->executablePool());
+    LinkBuffer patchBuffer(this, m_codeBlock->executablePool(), 0);
     if (!patchBuffer.allocationSuccessful())
         return false;
@@ -1116 +1116 @@
     Jump success = jump();
     
-    LinkBuffer patchBuffer(this, m_codeBlock->executablePool());
+    LinkBuffer patchBuffer(this, m_codeBlock->executablePool(), 0);
     if (!patchBuffer.allocationSuccessful())
         return false;

trunk/JavaScriptCore/jit/SpecializedThunkJIT.h

@@ -130 +130 @@
         MacroAssemblerCodePtr finalize(MacroAssemblerCodePtr fallback)
         {
-            LinkBuffer patchBuffer(this, m_pool.get());
+            LinkBuffer patchBuffer(this, m_pool.get(), 0);
             // We can't continue if we can't call a function!
             if (!patchBuffer.allocationSuccessful())

trunk/JavaScriptCore/runtime/Executable.cpp

@@ -305 +305 @@
 #if ENABLE(JIT)
     if (globalData->canUseJIT()) {
-        JITCode newJITCode = JIT::compile(globalData, newCodeBlock.get());
+        JITCode newJITCode = JIT::compile(globalData, newCodeBlock.get(), 0, codeBlock->m_isConstructor ? generatedJITCodeForConstruct().start() : generatedJITCodeForCall().start());
         if (!newJITCode) {
             globalData->functionCodeBlockBeingReparsed = 0;
@@ -338 +338 @@
 #if ENABLE(JIT)
     if (globalData->canUseJIT()) {
-        JITCode newJITCode = JIT::compile(globalData, newCodeBlock.get());
+        JITCode newJITCode = JIT::compile(globalData, newCodeBlock.get(), 0, generatedJITCodeForCall().start());
         if (!newJITCode) {
             globalData->functionCodeBlockBeingReparsed = 0;

trunk/JavaScriptCore/wtf/FastMalloc.cpp

@@ -4455 +4455 @@
     &FastMallocZone::log, &FastMallocZone::forceLock, &FastMallocZone::forceUnlock, &FastMallocZone::statistics
 
-#if !defined(BUILDING_ON_TIGER) && !defined(BUILDING_ON_LEOPARD) && !OS(IOS)
+#if !defined(BUILDING_ON_TIGER) && !defined(BUILDING_ON_LEOPARD)
     , 0 // zone_locked will not be called on the zone unless it advertises itself as version five or higher.
 #endif
-#if !defined(BUILDING_ON_TIGER) && !defined(BUILDING_ON_LEOPARD) && !defined(BUILDING_ON_SNOW_LEOPARD) && !OS(IOS)
+#if !defined(BUILDING_ON_TIGER) && !defined(BUILDING_ON_LEOPARD) && !defined(BUILDING_ON_SNOW_LEOPARD)
     , 0, 0, 0, 0 // These members will not be used unless the zone advertises itself as version seven or higher.
 #endif

trunk/JavaScriptCore/wtf/Platform.h

@@ -1095 +1095 @@
 #endif
 
+#if CPU(ARM_THUMB2)
+#define ENABLE_BRANCH_COMPACTION 1
+#endif
+
 #endif /* WTF_Platform_h */

trunk/JavaScriptCore/yarr/RegexJIT.cpp

@@ -1473 +1473 @@
         }
 
-        LinkBuffer patchBuffer(this, executablePool.release());
+        LinkBuffer patchBuffer(this, executablePool.release(), 0);
         if (!patchBuffer.allocationSuccessful()) {
             m_shouldFallBack = true;