Improved DevTools release/regression testing #1209
Description
Regression testing is an important part of updating and releasing DevTools, but it has gotten increasingly difficult as we've added advanced features to React (e.g. async, suspense, hooks). Even some of the recent tag constant changes (e.g. #1103, #1190) complicate this testing process, as they introduce potentially significant differences between minor releases of React that require explicit testing.
- Improved automated tests
Despite its large feature set, DevTools currently has only 4 test suites containing a total of 16 tests. We should invest some more effort here. Specifically, I'd like to see us add tests that protect against regressions. I think we could use a similar approach as with reactjs/react-lifecycles-compat, where we run tests against multiple versions of React.
Specifically, I think there might be some low-effort value to be gained by writing a simple test for each version of React that feeds DevTools an element tree and then verifyies the expected return values (e.g. name, type, props, state).
- Added manual test harnesses for React 16.0-16.7 #1210: Improved manual test harness
Our "plain test shell" example app can be great for testing a change to the DevTools UI, but it isn't all that helpful in terms of regression testing. The fact that it requires compilation, and that the compiled source code is committed to the Git repo, is also irritating.
I think we could create a couple of similar test harnesses, perhaps one for each React version 14, 15, and 16, don't require compilation. Instead these test harnesses could load a UMD version of react and react-dom (maybe selectable via a drop-down that controls a URL parameter). Each test harness could progressively enhance its features based on the semver minor tag (e.g. 16.6.x+ would have a test for .memo).
This way we'd have a centralized place to sanity check changes prior to releasing a new version.