how bad is the reliability of tests these days?

[tomholub]

@ioanmo226 @sosnovsky is it still a real PITA these days to ensure that tests all pass due to flakiness? Or has that somewhat improved? Got worse?

[sosnovsky]

I think currently we have issues only with live tests, mostly when testing drafts functionality, but @rrrooommmaaa works on #5069 which includes some test improvements.

And there is also another PR for live tests - #5088, which should be re-checked after finishing #5069

[tomholub]

Got it, so mocked tests are mostly fine?

[sosnovsky]

Yes, I checked our latest test runs - fails happen only for live tests

[rrrooommmaaa]

There are some intermittent glitches with the whole test setup, like today:

npm WARN EBADENGINE Unsupported engine {00:01
npm WARN EBADENGINE   package: 'pdfjs-dist@3.7.107',00:01
npm WARN EBADENGINE   required: { node: '>=18' },00:01
npm WARN EBADENGINE   current: { node: 'v16.17.1', npm: '8.15.0' }00:01
npm WARN EBADENGINE }

https://flowcrypt.semaphoreci.com/jobs/d3912c1d-0885-4885-a047-0b4316fc2eec
Suddenly, there was an old version of Node for consumer mock - unit tests.
Other tests have passed.
Perhaps, we should explicitly set Node version for Semaphore?
@sosnovsky ?

[sosnovsky]

Suddenly, there was an old version of Node for consumer mock - unit tests.
Other tests have passed.
Perhaps, we should explicitly set Node version for Semaphore?
@sosnovsky ?

For sure, we should use node 18 on semaphore, I checked semaphore.yml and it currently explicitly sets version 16.
Interesting that this error didn't appear in master branch, tests successfully passed there today.