Importing third-party tests

See ​slides

Mozilla's practice

Mozilla has submit and recieved directories sumbitted directory is automtically synced to the Mozilla directory on the W3C repository.

W3C process Each organization submits tests into their own directories, they're reviewed by editors, then merged into approved directory

Downstreaming Planning to get tests from the approved director after a review. Can't be automated because Gecko may fail on some of new tests. Mozilla's ref test framework allows annotations inside reftest.list such as some tests only match or fail on Windows, Mac, etc...

The import process may work every week or every month because it requires a manual review.

If there's a bug in Gecko that causes a test to fail, they file a bug then update reftest.list. If there is a bug in tests, then try to fix it upstream. No changes are made to tests regardless of the failure.

When adding reference files to non-reference tests, submit patches upstream first to avoid making Mozilla's copy and W3C's copy out of sync.

Microsoft is now open to adding refenreces for their tests now that CSS2.1 has been REC'ed.

We probably don't want to import tests that can't be automated. But it's probably valuable to run tests even if we didn't necessary pass.

There appears to be conflicting goals: Importing tests without making modifications to the imported tests If tests are mostly good, then we would like tests to be running instead of waiting for W3C process We want to identify which test directories contain imported tests We also need to deal with platform-differences

WebKit has a desire to run tests even if results are incorrect as regression tests.

We need to make sure test expectations are right so that they run cleanly on bots.

Adding test expectations in WebKit side would cause things to get out of sync. So we'll need update expectations in the next syncing process.

Maybe each reviewer should judge what kind of review process is needed instead of creating a global policy. We probably don't need to review "tests". It's a matter of how to import them and to where.

One way to deal with failing tests is to process them at the importation time like Mozilla does. Another option is to make incremental progress like we do for any other tests.

Mozilla has the same problem. They're planning to document those importation process.

For each directory, we need to know which harness, etc...

Maybe we should add a change log in each imported test directory so that we can document dates we imported tests and the changes we made. There will be less noise if it's separate from LayoutTests/ChangeLog.

LayoutTests are licensed as BSD so it's compatible with W3C's test harness.

For upstreaming, we need to auto-create linked elements. Could be done via automated directory analsysis.

We need to identify which tests are valuable for upstreaming. Maybe we can create a "submitted" directory for each W3C test suite.

We want to avoid duplicating tests in "submitted" and "approved" directories.

We also don't want to submit pixel tests. Maybe we can learn quickly once we start the conversion. For future tests, using testharness.js and ref tests would make the upstream process easier.

It's okay to not upstream the old tests as long as new tests are upstreamed because then the number of tests that are not shared is capped.

We can't just mechanically convert WebKit's js-tests to testharness.js because of some subtle differenes (e.g. testharness.js doesn't use eval) but they provide very similar mechanisms. Maybe testharness.js is too verbose?

How frequently should we update? We don't normally make centralized decision about these things.

What kind of directory structures should we use? We need to distingush imported tests and our own tests.

How can we automate this process? EWS doesn't run tests at the moment so automating the importation will be hard.

There might be tests removed from newer versions of imported tests that we find valuable. Where should we put them?

How about importing from other vendors? e.g. Mozilla. We don't want multiple ways of doing the same thing. e.g. bidi-isolate tests from Mozilla would have been helpful but they were not approved by W3C just yet.

But we could use that pressure as the leverage to push other vendors to submit all tests to W3C, and always import from W3C.

Maybe we should revisit using link element vs. reftest.list. This is fine for CSS tests that have valid parsers but it may not work for some tests that test parsers.