= Investigating Leaks and Bloat = ''by Simon Fraser'' - Or… a bunch of half-baked memory tools - Or… don’t be afraid to change code to collect data - Don’t be scared to do this - Starting problem - “WebContent physical footprint doesn’t go back down significantly after going from nytimes.com to a simple page (about:blank)” - When we say memory use is too high we look at Physical Footprint metric - When measuring memory: Beware of caches - Proper steps - Load about:blank - Get physical footprint with `vmmap` - Load nytimes.com - Go back to about:blank - Fire memory warning - Get physical footprint with `vmmap` - “Something is not going away” - Leaks? - Tool called “leaks”: `leaks ` - Adding Quick & Easy “State of the world” Data Gathering - Handy too on Mac - `notifyutil -p` - Broadcasts notifications to processes, processes can register handlers - Handles: - org.WebKit.lowMemory - com.apple.WebKit.fullGC - com.apple.WebKit.deleteAllCode - com.apple.WebKit.Cache.dump - com.apple.WebKit.showAllDocuments* - com.apple.WebKit.showMemoryCache* - com.apple.WebKit.showPageCache* - * added in last 6 months - `notifyutil -p com.apple.WebKit.showAllDocuments` - Why aren’t they leaks? - Leaks tool looks for rooted graphs of objects - This is really “Abandonment” or “leaks” - So what objects are hanging around? - Object Counters - Is this a JS heap issue, or a C++ ref-counting issue? - Which world is the issue happening in? - Not always easy to tell - Do the JS* objects go away? - Did all the doc nodes get destroyed? - `notifyutil -p com.apple.WebKit.showAllDocuments` - Look at referencingNodeCount - JS Heap - GC - Visist s every object that JS knows about (as JSCell*), starting at GC roots - JSC::Heap, JSC::SlotVisitor, visistChildren() methods - Unvisisted objects can be GC - Doesn’t care about unrooted cycles - GC Heap Inspector - Heap Snapshots in Web Inspector - Not quite enough info, no C++ pointer objects - `notifyutil -p com.apple.WebKit.dumpGCHeap` - Checked-in: https://trac.webkit.org/r235271 - GC-related Document Leaks - DFG Scratch buffer - Exception backtraces - VM lastException - JSPerformanceObserverCallback: Strong - Strong references indicate GC problems - C++ leaks - Unbalanced ref()/deref(), retain reference cycles - `refCount 1` indicates a C++ leak - Ref tracking - Collate matching ref() and deref() via “tokens” - Ref Token Tracking - Fix Ref<> and RefPtr<> to store tokens - All ref() and deref() via Ref<> and RefPtr<> are tracked - RefTracker class example - Call stack records the ref - StackShot is your friend - Wrapper for WTFGetBacktrace() - Hashable - Great for “accumulate all the calls stacks that did this thing” - Controllable stack depth - TrackedRefCounted<> - Dumping Unmatched Refs - `RefTracker::dumpRemainingReferences()` - RefTracker is powerful! - Sometimes dumps the only unmatched `ref()` callstack! - Easy to patch up manual `ref()`/`deref()` code paths by tracking RefTrackingTokens - webkit.org/b.186269 (reviewz plz) - Make it possible to track unbalanced ref()/deref() - RefTracker on Node - Give Node a RefTracker - Tracks Nodes, Elements, Documents - Now we can track Document refs - Hack the “com.apple.WebKit.showAllDocuments” callback to dump ref stacks for live Documents - You can turn it on and its not a huge perf hit - Making leaks not happen again - Used GCHeap Inspector - Fixed a leak - “Finding a regression in EWS is 1000x more efficient than finding it via a perf benchmark regression” — Simon Fraser - webkit.org/b/186214 - Isn’t this like the old “world leaks”? - Sort of is… - Leaks of world-level objects - Checking for Leaks in Testing - `./Tools/Scripts/run-webkit-tests --world-leaks` - How it Works - After each test - WKBundleGetLiveDocumentURLs() - Implications for run-webkit-tests - New failure type: “LEAK” - “LEAK” only relevant if you ran with `--world-leaks` - Leak checking is post-hoc - A test can pass, and then become a leak failure - Remaining Work - Add LEAK expectations for all platforms - Fix the causes of most common document leaks - Fix causes of false Leaks - Turn on by default - C++ Document Leaks - Web Animations - Document::removeFocusNavigationNodeofSubtree() - Content Filtering - IndexedDB - SVG text tests - WebFullScreenManager - C++ Document Lifetime Extenders - NavigationAction - Editor/Selection - ServicesOverlayController - DragController - To Do - When WebKitTestRunner detects a leak: - Dump unmatched refs when tests detect a leak - Dump a GC heap - Some auto analysis? - Halp - webkit.org/b/186214 - Now the memory problems are solved, right? - Not really, still some issues in vmmap - MALLOC zones - Dirty size column is interesting, WebKit MALLOC (411 MB) - “Where does all the memory go?” - webkit.org/b/186422 Create lots of different malloc zones for easier accounting of memory use - Created custom MALLOC zones for WebKit - Big ones - Vector capacity - HashTable - webkit.org/b/186698 Make it possible to track all sites that waste container capacity - Container Capacity Tracking - Give the Vector/HashTable an ID (can’t track by address) - Record allocation with a StackShot - On deallocation remove the record - Dump living capacity: - Aggregate by StackShot - Sort by most to least wasteful - Consider not using as many HashTables? - Maybe use different types of tables? - Memory Reduction - bmalloc space efficiency - Maybe as good as system malloc? Problem if using both, using one is fine - JSC optimizations for space efficiency (even with the JIT) - WebCore optimizations for space efficiency - Review Vector and HashTable growth policies - Resiable hashes (Robin Hood hashing?) - Vector shrinking - Convert from HashMaps and HashSets to other data structures - Adjust caching policies - Better Low Memory Handler - Don’t dirty more pages than you release - Organize data structures to release memory without visiting pages - More use of purgeable memory - WebProcess watermark (big 2.5 GB?) - Space-Efficient Class Layout - Micro-optimization? - `dump-class-layout -c Release WebCore FillLayer` - Look for `` entries - Also look at `` - std::optional causes these to get big - There’s been work on different optional types - Optional with a magic value (doesn’t waste any space) - Patch for a set of optionals in a bit field - Beware of enums - For enums, you should specify size - Big Stuff - Don’t just use all available memory - Manage cache sizes - Cache budgets - JS Heap size - Don’t Cause Leaks - Be careful with JS bindings code - JSC::strong<> is almost always wrong - Be careful with holding refs to Documents, elements - Avoid cycles - Avoid extending object lifetimes - Don’t write floaty classes - Shrink vectors, avoid unused inline capacity on heap-allocated objects - Do you really need to use HashTable? - Beware enums and std::optionals - Look for Leaks - Tools for investigating Leaks - cCache dumping - GC heap inspector - Ref token tracking