Add Jest unit tests and Detox end to end tests #415

AndrewGable · 2020-09-04T23:50:59Z

@tgolen @marcaaron - Will you please review?

Adds test two frameworks to RNC repo, jest unit tests and detox e2e tests. While the tests are sparse at the moment, I hope they enable the ability to add future tests that will grow into a valuable addition.

__mocks__/@react-native-community/netinfo.js

tgolen · 2020-09-08T14:16:47Z

__mocks__/react-native-config.js

@@ -0,0 +1,10 @@
+export default {


Rather than using a mock file for this, could a .env file be used instead like .env.travis?

This is how the library recommended to do it: https://github.com/luggit/react-native-config#jest

We might be able to create a env.travis file, then import it in this specific file (I think the file name is the most important), then use the variables?

OK, thanks for that info! What we could do is use dotenv (lib which we already use) to read the .env.travis file and export the config values in the same format being done here. Like this:

const travisConfig = dotenv('.env.travis'); export default { REPORT_IDS: travisConfig.REPORT_IDS, };

or maybe even just:

const travisConfig = dotenv('.env.travis'); export default travisConfig;

…all correctly.

AndrewGable · 2020-09-08T21:08:55Z

Ok so now the default Jest tests pass that were originally included with the project, but now I need to dig into https://reactnative.dev/docs/testing-overview to really understand what type of tests we'd like to add. Any ideas @tgolen?

AndrewGable · 2020-09-08T22:52:20Z

Ok so after reading and discussing with @marcaaron, in addition to the code added here, I think we should explore a simple E2E style test on all platforms to verify that the app at minimum boots as we expect. I believe this is the highest value test we could add to verify we aren't breaking production with our continuous deploy changes. The two instances that we should be testing for are: 1. Bad style props that cause native clients not to boot 2. bad import statements that cause the build to break

It looks like detox is the recommended library from the React Native documentation. It has iOS/Android support out of the box and a plugin for web. Let me know your thoughts, cc @cead22 / @Jag96

tgolen · 2020-09-08T22:54:18Z

Lots of good stuff in that article! The few things standing out to me are: 1. We probably don't need end-to-end tests, that's what Applause is for 1. Historically, the only valuable unit tests we have in JS have been around code that is ripe for regressions (like working with multi-tag levels). I think we should use some common sense to not "test everything" but to test the stuff that is valuable 1. That article says that it won't be able to test any platform-specific code, so we need to rely on Applause uncovering platform bugs 1. We could approach a bug mentality for testing, where a bug has to be reproduced in a test first before it can be fixed. While this is an interesting idea, I don't think it's very practical and I don't know that it would be too valuable except for catching regressions.

1. Bad style props that cause native clients not to boot 2. bad import

statements that cause the build to break I agree with having a test that verifies both of those! +1

…

On Tue, Sep 8, 2020 at 3:09 PM Andrew Gable ***@***.***> wrote: Ok so now the default Jest tests pass that were originally included with the project, but now I need to dig into https://reactnative.dev/docs/testing-overview to really understand what type of tests we'd like to add. Any ideas @tgolen <https://github.com/tgolen>? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#415 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJMABYJZTXJ6C4R7OHYYXLSE2MPNANCNFSM4QZVWF6Q> .

AndrewGable · 2020-09-08T23:13:10Z

So from reading the docs, I couldn't figure out if we can test for those two cases using just Unit tests, so I assumed that we needed a basic E2E test that in the end is just making sure the specific platform layer booted successfully.

I agree that we probably shouldn't write super extensive automated tests that run on the device, but what does everyone think about having a boot test (that is an E2E test using detox)?

tgolen · 2020-09-08T23:36:30Z

Using detox seems overkill for just doing a boot test. I agree that the boot test is valuable, but is there another way to accomplish that without the overhead of detox? (also, using something other than detox would help to avoid people wanting to add a bunch of E2E tests just for the sake of having them, which puts us down the road of maintaining them, etc. etc. flash back to selenium...).

AndrewGable · 2020-09-08T23:46:08Z

I think we could verify we built and booted a simulator, but I don't think we could confirm that we are on the sign in page after we booted without detox, which is when we've had these error pop up. They appear to be run time errors seen when launching the app, for example:

tgolen · 2020-09-09T18:48:20Z

OK, I guess it's fine to see where this takes us.

…

On Tue, Sep 8, 2020 at 5:46 PM Andrew Gable ***@***.***> wrote: I think we could verify we built and booted a simulator, but I don't think we could confirm that we are on the sign in page after we booted without detox, which is when we've had these error pop up. They appear to be run time errors seen when launching the app, for example: [image: redbox] <https://user-images.githubusercontent.com/2838819/92538134-c3f1ba00-f1f2-11ea-826a-f62b165b9a32.png> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#415 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJMABZOJA6XCZNAJLWUIFLSE2645ANCNFSM4QZVWF6Q> .

AndrewGable · 2020-09-18T17:32:29Z

Detox tests are passing locally:

andrew ➜ ReactNativeChat detox test
detox[35438] INFO:  [test.js] useCustomLogger=true DETOX_START_TIMESTAMP=1600450165158 reportSpecs=true jest --config e2e/config.json --testNamePattern '^((?!:android:).)*$' --maxWorkers 1 e2e
detox[35536] INFO:  [DetoxServer.js] server listening on localhost:52054...
detox[35536] INFO:  [AppleSimUtils.js] com.chat.expensify.chat launched. To watch simulator logs, run:
        /usr/bin/xcrun simctl spawn 74AAF2C0-2EF1-4D3F-93D4-51A1CF53D622 log stream --level debug --style compact --predicate 'process == "Chat"'
detox[35536] INFO:  Example is assigned to 74AAF2C0-2EF1-4D3F-93D4-51A1CF53D622 {"type":"iPhone 11"}
detox[35536] INFO:  Example: should have Log In screen
detox[35536] INFO:  Example: should have Log In screen [OK]

 PASS  e2e/firstTest.e2e.js (45.972 s)
  Example
    ✓ should have Log In screen (74 ms)

But are failing to start on GitHub actions:

detox[15946] INFO:  [test.js] useCustomLogger=true DETOX_START_TIMESTAMP=1600449883137 reportSpecs=true jest --config e2e/config.json --testNamePattern '^((?!:android:).)*$' --maxWorkers 1 e2e
detox[15947] INFO:  [DetoxServer.js] server listening on localhost:49575...
detox[15947] INFO:  [AppleSimUtils.js] com.chat.expensify.chat launched. To watch simulator logs, run:
        /usr/bin/xcrun simctl spawn 19824C9C-6254-4F7F-9B6C-6E2EC51AF1A6 log stream --level debug --style compact --predicate 'process == "Chat"'
detox[15947] INFO:  Example is assigned to 19824C9C-6254-4F7F-9B6C-6E2EC51AF1A6 {"type":"iPhone 11"}
detox[15947] INFO:  Example: should have Log In screen
detox[15947] WARN:  [Client.js/PENDING_REQUESTS] App has not responded to the network requests below:
  (id = -1000) reactNativeReload: {}
That might be the reason why the test "Example should have Log In screen" has timed out.

detox[15947] INFO:  Example: should have Log In screen [FAIL]
FAIL e2e/firstTest.e2e.js (170.557 s)
  Example
    ✕ should have Log In screen (120073 ms)

  ● Example › should have Log In screen

    thrown: "Exceeded timeout of 120000 ms for a hook.
    Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."

      1 | describe('Example', () => {
    > 2 |     beforeEach(async () => {
        |     ^
      3 |         await device.reloadReactNative();
      4 |     });
      5 | 

      at firstTest.e2e.js:2:5
      at Object.<anonymous> (firstTest.e2e.js:1:1)

detox[15946] ERROR: [cli.js] Error: Command failed: jest --config e2e/config.json --testNamePattern '^((?!:android:).)*$' --maxWorkers 1 e2e

tgolen · 2020-10-01T15:45:47Z

Any ideas on how we can get this working?

AndrewGable · 2020-10-01T16:37:27Z

Yeah good question, I need to put some more time into figuring out why the simulator is failing to launch on GitHub Actions.

AndrewGable · 2020-10-05T21:36:16Z

Ok circling back to this, it looks like detox has a guide on this: https://github.com/wix/Detox/blob/master/docs/Guide.RunningOnCI.md

I'll check it out and try to get things running

AndrewGable · 2020-10-05T23:07:35Z

Updated with recommended solution and locally they pass, on GitHub they fail still 😞

Local:

andrew ➜ ReactNativeChat detox test --configuration ios.sim.release --cleanup
detox[73908] INFO:  [test.js] configuration="ios.sim.release" cleanup=true useCustomLogger=true DETOX_START_TIMESTAMP=1601939073439 reportSpecs=true jest --config e2e/config.json --testNamePattern '^((?!:android:).)*$' --maxWorkers 1 e2e
detox[73996] INFO:  [DetoxServer.js] server listening on localhost:53632...
detox[73996] INFO:  [AppleSimUtils.js] com.chat.expensify.chat launched. To watch simulator logs, run:
        /usr/bin/xcrun simctl spawn 91775C90-306D-4C36-A1D6-5EB50FF77778 log stream --level debug --style compact --predicate 'process == "Chat"'
detox[73996] INFO:  Example is assigned to 91775C90-306D-4C36-A1D6-5EB50FF77778 {"type":"iPhone 11"}
detox[73996] INFO:  Example: should have Log In screen
detox[73996] INFO:  Example: should have Log In screen [OK]

 PASS  e2e/firstTest.e2e.js (44.05 s)
  Example
    ✓ should have Log In screen (313 ms)

andrew ➜ ReactNativeChat

GitHub:

Run detox test --configuration ios.sim.release --cleanup
detox[18724] INFO:  [test.js] configuration="ios.sim.release" cleanup=true useCustomLogger=true DETOX_START_TIMESTAMP=1601936246354 reportSpecs=true jest --config e2e/config.json --testNamePattern '^((?!:android:).)*$' --maxWorkers 1 e2e
detox[18725] INFO:  [DetoxServer.js] server listening on localhost:49779...
detox[18725] INFO:  [AppleSimUtils.js] com.chat.expensify.chat launched. To watch simulator logs, run:
        /usr/bin/xcrun simctl spawn 6C1CB0D1-8C8C-493D-94B2-3198E44CF86F log stream --level debug --style compact --predicate 'process == "Chat"'
detox[18725] ERROR: Signal 6 was raised
(
	0   Detox                               0x000000010e91ba31 __DTXHandleSignal + 59
	1   libsystem_platform.dylib            0x00007fff522f95fd _sigtramp + 29
	2   libsystem_c.dylib                   0x00007fff521edf39 itoa64 + 2793
	3   libsystem_c.dylib                   0x00007fff521e9b7c abort + 120
	4   libc++abi.dylib                     0x00007fff500ef858 abort_message + 231
	5   libc++abi.dylib                     0x00007fff500e0cbf demangling_unexpected_handler() + 0
	6   libobjc.A.dylib                     0x00007fff512a1c0b _objc_terminate() + 96
	7   Detox                               0x000000010e91bf41 __dtx_terminate() + 157
	8   libc++abi.dylib                     0x00007fff500eec87 std::__terminate(void (*)()) + 8
	9   libc++abi.dylib                     0x00007fff500eec29 std::terminate() + 41
	10  libdispatch.dylib                   0x00007fff520dc8df _dispatch_client_callout + 28
	11  libdispatch.dylib                   0x00007fff520e260c _dispatch_lane_serial_drain + 707
	12  libdispatch.dylib                   0x00007fff520e3044 _dispatch_lane_invoke + 388
	13  libdispatch.dylib                   0x00007fff520ed0c4 _dispatch_workloop_worker_thread + 626
	14  libsystem_pthread.dylib             0x00007fff52301a3d _pthread_wqthread + 290
	15  libsystem_pthread.dylib             0x00007fff52300b77 start_wqthread + 15
)
detox[18725] INFO:  Example is assigned to undefined
detox[18725] INFO:  Example: should have Log In screen

detox[18725] INFO:  Example: should have Log In screen [SKIPPED]
detox[18724] ERROR: [cli.js] Error: Command failed: jest --config e2e/config.json --testNamePattern '^((?!:android:).)*$' --maxWorkers 1 e2e

AndrewGable · 2020-11-04T23:49:28Z

Tests are passing, I'd love a review, thanks!

marcaaron

Looks awesome!!

marcaaron · 2020-11-05T00:21:22Z

.github/workflows/lint.yml

@@ -1,4 +1,4 @@
-name: eslint
+name: ESLint JavaScript Lint


NAB, not sure why this format is changing but the extra Lint seems extra maybe just Lint JavaScript

Just a name change, I thought that being a bit more descriptive was better, but I like the suggestion 👍

marcaaron · 2020-11-05T00:22:33Z

.github/workflows/test.yml

+            - uses: actions/checkout@v2
+            - name: Use Node.js
+              uses: actions/setup-node@v1
+            - uses: webfactory/ssh-agent@v0.4.0


Do we need to update this guy re: that other PR?

Yes, good catch, I will update.

marcaaron · 2020-11-05T00:22:57Z

babel.config.js

    plugins: [],
 };

+


NAB extra new line

marcaaron · 2020-11-05T00:25:11Z

tests/e2e/environment.js

+        super(config);
+
+        // Can be safely removed, if you are content with the default value (=300000ms)
+        this.initTimeout = 300000;


Are we content?

Yes, we are, I shall remove this value 👍

tests/unit/__snapshots__/loginTest.js.snap

sketchydroide

LGTM
In all honesty this is hard to understand, but from I could understand it looks good.

marcaaron · 2020-11-05T21:21:27Z

Got a conflict but ready for merging :)

marcaaron

Looks great!

@tgolen

@tgolen everything looks good to me but feel free to take a look when you get back from OOO

Julesssss · 2020-11-09T14:52:18Z

Since this PR has been merged 1.0.1-171, I'm unable to run the app on mobile. I'm not certain, but I believe there is some issue with this PR or an additional step I need to do locally to fix the issue. It's as if a native file is missing after an update to the nanopb/decode lib -- but I can see that the file does exist

error: Error: Unable to resolve module `./maps/decode.json` from `node_modules/entities/lib/decode_codepoint.js`:

None of these files exist:
  * decode.json
  * node_modules/entities/lib/maps/decode.json/index(.native|.android.jsx|.native.jsx|.jsx|.android.js|.native.js|.js|.android.ts|.native.ts|.ts|.android

Any ideas what I'm missing, or does anyone remember successfully testing this PR on mobile? (to rule out an issue on this PR)

I've tried clearing NPM cache, npm install, ect. Additional info on this Slack thread

Julesssss · 2020-11-09T15:07:27Z

Well, please ignore the above. Restarting my machine fixed this issue... No idea what the actual issue is, and why restarting works but I'll add it to my RN troubleshooting steps 😞

tgolen · 2020-11-09T17:13:35Z

tests/e2e/loginTest.e2e.js

+jest.setTimeout(120000 * 10);
+
+describe('Test login page', () => {
+    beforeEach(async () => {


From looking at the Jest documentation, using promises should be pretty straight forward. Can you please switch these to be promises?

tgolen · 2020-11-09T17:16:18Z

tests/unit/mocks/react-native-config.js

@@ -0,0 +1,10 @@
+export default {


Can this file just import and parse the .env file so that this stuff doesn't need to be defined in more than one place?

If .env was a JS file, maybe. But this is how they recommend setting it up: https://github.com/luggit/react-native-config#testing

Even without the .env file being JS, it can still be easily parsed using the dotenv package that we already have installed and are using elsewhere. That will DRY this config up and make sure that all of our configs are only defined in a single place.

Ah I see, let me see if I can get it to work in the tests 👍

tgolen · 2020-11-09T17:17:06Z

README.md


 ## Running the tests 🎰
-* To run the **Jest Unit Tests**: `npm run test`
+### Unit tests


Could you please add more info in this Readme about the difference between unit tests and e2e tests and when you would use one instead of the other and the benefits of either of them?

sketchydroide

LGTM

Tests are noisy, but are passing locally!

48a9931

tgolen previously requested changes Sep 8, 2020

View reviewed changes

AndrewGable added 3 commits September 8, 2020 10:28

Clean up workflow and job names for GitHub actions

d18bf2f

Move tests and mocks into a tests/ folder, configure Jest to find it …

4db0827

…all correctly.

Continue to configure.. Tests are silent on my machine

16f9fb0

Continue to explore unit tests

6b92a1f

AndrewGable added 7 commits September 10, 2020 09:57

Start to add detox

0914a43

Merge branch 'master' into andrew-test

fd79338

Tweak configs

7ee502c

Continue config of detox

08d384b

Add passing tests

e54bd90

Install xcpretty from gem

6c49e8d

Tweak sim used

276e2c7

AndrewGable added 2 commits September 18, 2020 10:59

Add more logs

5b95625

Increase timeout

b0432aa

AndrewGable added 3 commits October 5, 2020 14:37

Merge branch 'master' into andrew-test

ca94b4a

Try to use release configs

6e32bdf

Debug -> Release

daf8312

AndrewGable added 2 commits November 4, 2020 12:46

Use Xcode 11.5

8819cb1

Merge branch 'master' into andrew-test

e86066d

AndrewGable requested review from marcaaron and tgolen November 4, 2020 23:49

AndrewGable marked this pull request as ready for review November 4, 2020 23:49

AndrewGable requested a review from a team as a code owner November 4, 2020 23:49

botify requested review from sketchydroide and removed request for a team November 4, 2020 23:49

marcaaron previously approved these changes Nov 5, 2020

View reviewed changes

sketchydroide previously approved these changes Nov 5, 2020

View reviewed changes

AndrewGable added 2 commits November 5, 2020 10:51

Apply peer review changes

d42ef6e

Update readme and add two new scripts

618b789

AndrewGable dismissed stale reviews from sketchydroide and marcaaron via 618b789 November 5, 2020 19:04

AndrewGable changed the title ~~Tests are noisy, but are passing locally!~~ Add Jest unit tests and Detox end to end tests Nov 5, 2020

AndrewGable requested a review from marcaaron November 5, 2020 19:16

Merge branch 'master' into andrew-test

1efe05c

marcaaron approved these changes Nov 5, 2020

View reviewed changes

marcaaron merged commit b856025 into master Nov 5, 2020

marcaaron deleted the andrew-test branch November 5, 2020 22:40

tgolen reviewed Nov 9, 2020

View reviewed changes

AndrewGable mentioned this pull request Nov 17, 2020

Apply test review changes #801

Merged

sketchydroide reviewed Nov 17, 2020

View reviewed changes

atef1995 mentioned this pull request Oct 23, 2023

[Snyk] Upgrade react-native-safe-area-context from 4.4.1 to 4.7.2 atef1995/Expensify#3

Open

chiragsalian mentioned this pull request Oct 29, 2023

[Snyk] Upgrade react-native-safe-area-context from 4.4.1 to 4.7.2 #30555

Closed

Add Jest unit tests and Detox end to end tests #415

Add Jest unit tests and Detox end to end tests #415

Uh oh!

Conversation

AndrewGable commented Sep 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AndrewGable commented Sep 8, 2020

Uh oh!

AndrewGable commented Sep 8, 2020

Uh oh!

tgolen commented Sep 8, 2020 via email

Uh oh!

AndrewGable commented Sep 8, 2020

Uh oh!

tgolen commented Sep 8, 2020

Uh oh!

AndrewGable commented Sep 8, 2020

Uh oh!

tgolen commented Sep 9, 2020 via email

Uh oh!

AndrewGable commented Sep 18, 2020

Uh oh!

tgolen commented Oct 1, 2020

Uh oh!

AndrewGable commented Oct 1, 2020

Uh oh!

AndrewGable commented Oct 5, 2020

Uh oh!

AndrewGable commented Oct 5, 2020

Uh oh!

AndrewGable commented Nov 4, 2020

Uh oh!

marcaaron left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sketchydroide left a comment

Choose a reason for hiding this comment

Uh oh!

marcaaron commented Nov 5, 2020

Uh oh!

marcaaron left a comment

Choose a reason for hiding this comment

Uh oh!

Julesssss commented Nov 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Julesssss commented Nov 9, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

AndrewGable commented Sep 4, 2020 •

edited

Loading

Julesssss commented Nov 9, 2020 •

edited

Loading