Improving Interop with WPT

by Philip Jägenstedt & Stephen McGruer

​Slides

• Why does interop matter?
  • Users, developers, browser engines
  • Users
    • Go to website on favorite browser, but broken
    • Go find a browser it’s meant to work on
    • All the things we do for interop make our lives easier
  • Developers
    • MDN ran largest survey ever
    • Top pain points cross-browser compat/interop/testing
    • What to do?
• A shared test suite, for all the web
  • 20 commits from WebKit
  • Believe it is infrastructure
• High Level View
  • Chrome + Firefox + Safari
  • 0/3 ~ 5K tests
  • 1/3 ~ 6K tests
  • 2/3 ~ 6K tests
  • 3/3 ~ 20K tests
  • Moving to target the browser specific failures (2/3)
  • Interop is steady
• Browser specific failures in WPT
  • Gap is growing
  • Methodology for data for this presentation
  • WPT splits into tests and sub-tests
    • Sub-tests are within the test file
    • Some tests with too many subtests
    • Encoding tests have absurd
    • Normalized to test level
• Runs on Azure pipelines
  • STP runs every 3 hours, takes < 2 hours
  • Edge Dev and Canary too
  • Thanks Microsoft and @mustjab
• Runs on Taskcluster
  • Chrome + Firefox on every commit
  • WebKitGTK ally
  • Thanks Mozilla and @jgraham
• Room for Improvement
  • STP updates
  • Reliability
  • Thanks Apple and @burg
  • ​https://github.com/web-platform-tests/wpt/projects/4
• Epiphany / WebKit
  • Also run subtests on Linux for a second opinion
• WebKit on Linux
  • Google partnering with Igalia
  • WebKitGTK stable+nightly runs
  • Fixing noise in the data
• 2-way Sync
  • Script for WebKit exists
• 2-way Sync in Chromium
  • Update impl+tests in single review
  • Use of WPT by Chromium devs up ~220% (*j.mp/wpt-yes)
  • ~55% of impl+test changes now use WPT
    • Chromium & Gecko
      • Automatic import
      • Automatic export AFTER changes land
    • WebKit
      • Manual import (?)
        • Yes it is
      • Manual (?) export BEFORE changes land
  • Room for improvement / collaboration?
  • Issue on export PRs?
• A Wishlist
  • Collaborate on wit.fyi
    • Wish there was a way to browser-specific failures in the same way computed for the graph
  • Running web-driver to internal builds
  • Collaborate on running WPT on iOS
  • Collaborate on MDN survey: developers say interop is a big thing

• WPT Right approach
  • Importing all the tests could slow us down
  • Maybe keep them separate?
  • New tests added don’t look like regressions
  • WPT: Tracking WPT as a sub-module is not compatible

• Please don't scrape wpt.fyi, use the ​API