Investigating Leaks and Bloat

by Simon Fraser

• Or… a bunch of half-baked memory tools
• Or… don’t be afraid to change code to collect data
  • Don’t be scared to do this

• Starting problem
• “WebContent physical footprint doesn’t go back down significantly after going from nytimes.com to a simple page (about:blank)”
• When we say memory use is too high we look at Physical Footprint metric
• When measuring memory: Beware of caches
• Proper steps
  • Load about:blank
  • Get physical footprint with vmmap
  • Load nytimes.com
  • Go back to about:blank
  • Fire memory warning
  • Get physical footprint with vmmap
• “Something is not going away”
  • Leaks?
• Tool called “leaks”: leaks <pid>
• Adding Quick & Easy “State of the world” Data Gathering
• Handy too on Mac
  • notifyutil -p
  • Broadcasts notifications to processes, processes can register handlers
  • Handles:
    • org.WebKit.lowMemory
    • com.apple.WebKit.fullGC
    • com.apple.WebKit.deleteAllCode
    • com.apple.WebKit.Cache.dump
    • com.apple.WebKit.showAllDocuments*
    • com.apple.WebKit.showMemoryCache*
    • com.apple.WebKit.showPageCache*
    • * added in last 6 months
• notifyutil -p com.apple.WebKit.showAllDocuments
• Why aren’t they leaks?
  • Leaks tool looks for rooted graphs of objects
• This is really “Abandonment” or “leaks”
• So what objects are hanging around?
  • Object Counters
• Is this a JS heap issue, or a C++ ref-counting issue?
• Which world is the issue happening in?
  • Not always easy to tell
  • Do the JS* objects go away?
  • Did all the doc nodes get destroyed?
    • notifyutil -p com.apple.WebKit.showAllDocuments
    • Look at referencingNodeCount
• JS Heap
  • GC
  • Visist s every object that JS knows about (as JSCell*), starting at GC roots
    • JSC::Heap, JSC::SlotVisitor, visistChildren() methods
  • Unvisisted objects can be GC
  • Doesn’t care about unrooted cycles
• GC Heap Inspector
  • Heap Snapshots in Web Inspector
  • Not quite enough info, no C++ pointer objects
• notifyutil -p com.apple.WebKit.dumpGCHeap
• Checked-in: ​https://trac.webkit.org/r235271
• GC-related Document Leaks
  • DFG Scratch buffer
  • Exception backtraces
  • VM lastException
  • JSPerformanceObserverCallback: Strong<Function>
    • Strong references indicate GC problems
• C++ leaks
  • Unbalanced ref()/deref(), retain reference cycles
  • refCount 1 indicates a C++ leak
• Ref tracking
  • Collate matching ref() and deref() via “tokens”
• Ref Token Tracking
  • Fix Ref<> and RefPtr<> to store tokens
    • All ref() and deref() via Ref<> and RefPtr<> are tracked
• RefTracker class example
• Call stack records the ref
• StackShot is your friend
  • Wrapper for WTFGetBacktrace()
  • Hashable
  • Great for “accumulate all the calls stacks that did this thing”
  • Controllable stack depth
• TrackedRefCounted<>
• Dumping Unmatched Refs
  • RefTracker::dumpRemainingReferences()
• RefTracker is powerful!
  • Sometimes dumps the only unmatched ref() callstack!
  • Easy to patch up manual ref()/deref() code paths by tracking RefTrackingTokens
• webkit.org/b.186269 (reviewz plz)
  • Make it possible to track unbalanced ref()/deref()
• RefTracker on Node
  • Give Node a RefTracker
  • Tracks Nodes, Elements, Documents
  • Now we can track Document refs
  • Hack the “com.apple.WebKit.showAllDocuments” callback to dump ref stacks for live Documents
  • You can turn it on and its not a huge perf hit
• Making leaks not happen again
  • Used GCHeap Inspector
  • Fixed a leak
• “Finding a regression in EWS is 1000x more efficient than finding it via a perf benchmark regression” — Simon Fraser
• webkit.org/b/186214
• Isn’t this like the old “world leaks”?
  • Sort of is…
  • Leaks of world-level objects
• Checking for Leaks in Testing
  • ./Tools/Scripts/run-webkit-tests --world-leaks
• How it Works
  • After each test
    • WKBundleGetLiveDocumentURLs()
• Implications for run-webkit-tests
  • New failure type: “LEAK”
  • “LEAK” only relevant if you ran with --world-leaks
  • Leak checking is post-hoc
  • A test can pass, and then become a leak failure
• Remaining Work
  • Add LEAK expectations for all platforms
  • Fix the causes of most common document leaks
  • Fix causes of false Leaks
  • Turn on by default
• C++ Document Leaks
  • Web Animations
  • Document::removeFocusNavigationNodeofSubtree()
  • Content Filtering
  • IndexedDB
  • SVG text tests
  • WebFullScreenManager
• C++ Document Lifetime Extenders
  • NavigationAction
  • Editor/Selection
  • ServicesOverlayController
  • DragController
• To Do
  • When WebKitTestRunner detects a leak:
    • Dump unmatched refs when tests detect a leak
    • Dump a GC heap
  • Some auto analysis?
• Halp
  • webkit.org/b/186214
• Now the memory problems are solved, right?
  • Not really, still some issues in vmmap
  • MALLOC zones
  • Dirty size column is interesting, WebKit MALLOC (411 MB)
• “Where does all the memory go?”
  • webkit.org/b/186422 Create lots of different malloc zones for easier accounting of memory use
  • Created custom MALLOC zones for WebKit
  • Big ones
  • Vector capacity
  • HashTable
  • webkit.org/b/186698 Make it possible to track all sites that waste container capacity
• Container Capacity Tracking
  • Give the Vector/HashTable an ID (can’t track by address)
  • Record allocation with a StackShot
  • On deallocation remove the record
  • Dump living capacity:
    • Aggregate by StackShot
    • Sort by most to least wasteful
• Consider not using as many HashTables?
  • Maybe use different types of tables?
• Memory Reduction
  • bmalloc space efficiency
    • Maybe as good as system malloc? Problem if using both, using one is fine
  • JSC optimizations for space efficiency (even with the JIT)
  • WebCore optimizations for space efficiency
  • Review Vector and HashTable growth policies
    • Resiable hashes (Robin Hood hashing?)
  • Vector shrinking
  • Convert from HashMaps and HashSets to other data structures
  • Adjust caching policies
• Better Low Memory Handler
  • Don’t dirty more pages than you release
    • Organize data structures to release memory without visiting pages
  • More use of purgeable memory
• WebProcess watermark (big 2.5 GB?)
• Space-Efficient Class Layout
  • Micro-optimization?
  • dump-class-layout -c Release WebCore FillLayer
  • Look for <Padding: n bytes> entries
  • Also look at <UNUSED BITS: n bits>
• std::optional causes these to get big
  • There’s been work on different optional types
  • Optional with a magic value (doesn’t waste any space)
  • Patch for a set of optionals in a bit field
• Beware of enums
  • For enums, you should specify size
• Big Stuff
  • Don’t just use all available memory
  • Manage cache sizes
    • Cache budgets
    • JS Heap size
• Don’t Cause Leaks
  • Be careful with JS bindings code
    • JSC::strong<> is almost always wrong
  • Be careful with holding refs to Documents, elements
    • Avoid cycles
    • Avoid extending object lifetimes
• Don’t write floaty classes
  • Shrink vectors, avoid unused inline capacity on heap-allocated objects
  • Do you really need to use HashTable?
  • Beware enums and std::optionals
• Look for Leaks
• Tools for investigating Leaks
  • cCache dumping
  • GC heap inspector
  • Ref token tracking