test: e2e replay #3993

vaind · 2024-08-05T20:03:40Z

📢 Type of change

Bugfix
New feature
Enhancement
Refactoring

📜 Description

Tests that a replay is part of the error event when session replay options are enabled.

💡 Motivation and Context

💚 How did you test it?

📝 Checklist

I reviewed submitted code
I added tests to verify changes
No new PII added or SDK only sends newly added PII if sendDefaultPII is enabled
All tests passing
No breaking changes

🔮 Next steps

Co-authored-by: Krystof Woldrich <31292499+krystofwoldrich@users.noreply.github.com>

github-actions · 2024-08-05T20:31:39Z

iOS (legacy) Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	1225.48 ms	1237.58 ms	12.10 ms
Size	2.36 MiB	3.08 MiB	733.94 KiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`dadc233`+dirty	1223.20 ms	1236.88 ms	13.68 ms
`5571a20`+dirty	1203.57 ms	1204.57 ms	1.00 ms
`70e6261`+dirty	1220.09 ms	1230.04 ms	9.95 ms
`62a750b`+dirty	1216.60 ms	1229.14 ms	12.54 ms
`acadc0f`+dirty	1264.38 ms	1290.06 ms	25.68 ms
`6e8584e`+dirty	1274.50 ms	1296.82 ms	22.32 ms
`c398f67`+dirty	1219.67 ms	1225.66 ms	5.99 ms
`86d6d2c`+dirty	1267.55 ms	1286.21 ms	18.66 ms
`e5c9b8b`+dirty	1258.57 ms	1267.32 ms	8.75 ms
`31fcca2`+dirty	1209.17 ms	1216.21 ms	7.04 ms

App size

Revision	Plain	With Sentry	Diff
`dadc233`+dirty	2.36 MiB	2.84 MiB	486.85 KiB
`5571a20`+dirty	2.36 MiB	2.92 MiB	569.93 KiB
`70e6261`+dirty	2.36 MiB	3.03 MiB	680.42 KiB
`62a750b`+dirty	2.36 MiB	2.92 MiB	570.00 KiB
`acadc0f`+dirty	2.36 MiB	2.83 MiB	480.37 KiB
`6e8584e`+dirty	2.36 MiB	2.88 MiB	533.17 KiB
`c398f67`+dirty	2.36 MiB	3.04 MiB	696.27 KiB
`86d6d2c`+dirty	2.36 MiB	2.82 MiB	462.82 KiB
`e5c9b8b`+dirty	2.36 MiB	2.87 MiB	520.43 KiB
`31fcca2`+dirty	2.36 MiB	2.90 MiB	552.95 KiB

github-actions · 2024-08-05T20:37:32Z

iOS (new) Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	1246.06 ms	1244.00 ms	-2.06 ms
Size	2.92 MiB	3.64 MiB	740.01 KiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`dadc233`+dirty	1266.52 ms	1282.55 ms	16.03 ms
`5571a20`+dirty	1228.09 ms	1233.45 ms	5.36 ms
`70e6261`+dirty	1224.90 ms	1231.02 ms	6.12 ms
`62a750b`+dirty	1228.12 ms	1230.53 ms	2.41 ms
`acadc0f`+dirty	1271.12 ms	1272.28 ms	1.16 ms
`6e8584e`+dirty	1271.71 ms	1281.26 ms	9.55 ms
`c398f67`+dirty	1227.31 ms	1230.00 ms	2.69 ms
`86d6d2c`+dirty	1291.62 ms	1296.80 ms	5.18 ms
`e5c9b8b`+dirty	1276.90 ms	1280.92 ms	4.02 ms
`31fcca2`+dirty	1222.04 ms	1226.51 ms	4.47 ms

App size

Revision	Plain	With Sentry	Diff
`dadc233`+dirty	2.92 MiB	3.40 MiB	492.53 KiB
`5571a20`+dirty	2.92 MiB	3.48 MiB	575.54 KiB
`70e6261`+dirty	2.92 MiB	3.59 MiB	686.11 KiB
`62a750b`+dirty	2.92 MiB	3.48 MiB	575.59 KiB
`acadc0f`+dirty	2.92 MiB	3.39 MiB	487.34 KiB
`6e8584e`+dirty	2.92 MiB	3.44 MiB	536.52 KiB
`c398f67`+dirty	2.92 MiB	3.60 MiB	701.89 KiB
`86d6d2c`+dirty	2.92 MiB	3.37 MiB	464.31 KiB
`e5c9b8b`+dirty	2.92 MiB	3.43 MiB	524.50 KiB
`31fcca2`+dirty	2.92 MiB	3.46 MiB	557.31 KiB

github-actions · 2024-08-20T08:59:44Z

Android (new) Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	645.61 ms	724.43 ms	78.82 ms
Size	7.15 MiB	8.34 MiB	1.19 MiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
`52a8031`+dirty	330.72 ms	358.76 ms	28.03 ms
`86d6d2c`+dirty	267.21 ms	325.24 ms	58.04 ms
`2534337`+dirty	597.14 ms	665.04 ms	67.90 ms
`22e31b6`+dirty	295.75 ms	346.73 ms	50.98 ms
`dadc233`+dirty	363.19 ms	370.37 ms	7.18 ms
`148f924`+dirty	347.36 ms	389.13 ms	41.77 ms
`c398f67`+dirty	315.08 ms	345.60 ms	30.52 ms
`5bb8d5f`+dirty	356.71 ms	389.65 ms	32.94 ms
`d197b5c`+dirty	258.75 ms	313.61 ms	54.86 ms
`15c80ab`+dirty	276.38 ms	327.54 ms	51.17 ms

App size

Revision	Plain	With Sentry	Diff
`52a8031`+dirty	7.15 MiB	8.09 MiB	965.95 KiB
`86d6d2c`+dirty	7.15 MiB	8.09 MiB	962.69 KiB
`2534337`+dirty	7.15 MiB	8.11 MiB	988.68 KiB
`22e31b6`+dirty	7.15 MiB	8.10 MiB	981.29 KiB
`dadc233`+dirty	7.15 MiB	8.04 MiB	910.84 KiB
`148f924`+dirty	7.15 MiB	8.21 MiB	1.07 MiB
`c398f67`+dirty	7.15 MiB	8.21 MiB	1.07 MiB
`5bb8d5f`+dirty	7.15 MiB	8.21 MiB	1.06 MiB
`d197b5c`+dirty	7.15 MiB	8.09 MiB	962.72 KiB
`15c80ab`+dirty	7.15 MiB	8.09 MiB	966.13 KiB

krystofwoldrich · 2024-09-03T11:46:55Z

Session replays uploaded from this PR CI

test/react-native/rn.patch.app.js

krystofwoldrich · 2024-09-03T12:04:23Z

test/e2e/test/e2e.test.ts

+    expect(replay).toBeDefined();
+    expect(replay.replay_id.length).toBe(32);
+
+    const replayInfo = await fetchReplay(replay.replay_id);


I thought possibly the Android replays are uploaded due to a short time limit, but if I understand the code correctly, this fetch will wait up to 10 minutes, checking the endpoint every 1s.

Is that right? @vaind

yeah, nothing to do with timeouts. Replay capture doesn't work at all on android emulator in GHA at the moment

Also no screenshots are captured by appium, if you look into an artifact for any of the Android jobs:

From what I can tell it's an old/recurring issue of the emulator on GHA, see actions/runner-images#6814

test/e2e/test/e2e.test.ts

bruno-garcia · 2024-11-12T16:04:41Z

ping

vaind · 2024-11-17T16:32:34Z

replaced by #4277

vaind and others added 30 commits July 25, 2024 11:16

wip: extract e2e test script from yaml

fe7ab45

update e2e test script

a393435

move more stuff to e2e.js

5c13f0d

script args

a477d6e

more stuff moved to e2e.js

fc61db5

run tests from e2e script

857e6a8

remove e2e build from ci

53e825f

fix ruby/setup-ruby

dd8abf5

update actions

b0a2ca2

more script fixes

273a376

fixup script on CI

7b7a0c2

cleanup

e368e4a

fix CI

fd4c50a

fix CI

b6601a3

linter issues

655300d

Merge branch 'main' into test/e2e-refactor

2dc8ebc

post-merge script update

3f65502

Update scripts/e2e.mjs

6e94861

Co-authored-by: Krystof Woldrich <31292499+krystofwoldrich@users.noreply.github.com>

Update scripts/e2e.mjs

612813d

Co-authored-by: Krystof Woldrich <31292499+krystofwoldrich@users.noreply.github.com>

Update scripts/e2e.mjs

85e6afb

Co-authored-by: Krystof Woldrich <31292499+krystofwoldrich@users.noreply.github.com>

Update scripts/e2e.mjs

efd1f9a

Co-authored-by: Krystof Woldrich <31292499+krystofwoldrich@users.noreply.github.com>

cleanup

883f9fb

move webdriveragent build to test

5701c7c

cache webdriveragent build

030049d

Merge branch 'main' into test/e2e-refactor

1739957

use bundler

d0b1f42

Update e2e.yml

8becfea

fix e2e test run

b26695b

try to fix ci

009864d

Update e2e.yml

d2cd560

Base automatically changed from test/e2e-refactor to main August 12, 2024 08:16

vaind and others added 9 commits August 12, 2024 20:25

Merge branch 'main' into test/e2e-replay

89bb744

test: replay video segments data

ae69622

Merge branch 'main' into test/e2e-replay

931c9e1

linter issue

37f2f8d

Merge branch 'main' into test/e2e-replay

9029815

enable debug logging in e2e tests

6dee84a

show emulator window to enable replay recording

61e5700

no no-window doesn't work

ed27926

Merge branch 'main' into test/e2e-replay

8f4b251

try to fix android screenshots

1863e62

krystofwoldrich reviewed Sep 3, 2024

View reviewed changes

test/react-native/rn.patch.app.js Outdated Show resolved Hide resolved

krystofwoldrich reviewed Sep 3, 2024

View reviewed changes

vaind added 3 commits September 3, 2024 19:42

don't run e2e replay test on android

f2ef368

disable session replay in tests

8bf63c7

Merge branch 'main' into test/e2e-replay

4ee29f2

vaind marked this pull request as ready for review September 5, 2024 17:38

vaind requested a review from lucas-zimerman as a code owner September 5, 2024 17:38

lucas-zimerman reviewed Sep 10, 2024

View reviewed changes

test/e2e/test/e2e.test.ts Show resolved Hide resolved

lucas-zimerman reviewed Sep 10, 2024

View reviewed changes

test/e2e/test/e2e.test.ts Show resolved Hide resolved

Merge branch 'main' into test/e2e-replay

9c4de41

vaind self-assigned this Nov 12, 2024

vaind mentioned this pull request Nov 17, 2024

test: e2e replay test using Maestro (js) #4277

Merged

9 tasks

vaind closed this Nov 17, 2024

Uh oh!

test: e2e replay #3993

test: e2e replay #3993

Uh oh!

Conversation

vaind commented Aug 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📢 Type of change

📜 Description

💡 Motivation and Context

💚 How did you test it?

📝 Checklist

🔮 Next steps

Uh oh!

github-actions bot commented Aug 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

iOS (legacy) Performance metrics 🚀

Baseline results on branch: main

Startup times

App size

Uh oh!

github-actions bot commented Aug 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

iOS (new) Performance metrics 🚀

Baseline results on branch: main

Startup times

App size

Uh oh!

github-actions bot commented Aug 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Android (new) Performance metrics 🚀

Baseline results on branch: main

Startup times

App size

Uh oh!

krystofwoldrich commented Sep 3, 2024

Session replays uploaded from this PR CI

Uh oh!

Uh oh!

krystofwoldrich Sep 3, 2024

Choose a reason for hiding this comment

Uh oh!

vaind Sep 3, 2024

Choose a reason for hiding this comment

Uh oh!

vaind Sep 3, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bruno-garcia commented Nov 12, 2024

Uh oh!

vaind commented Nov 17, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

vaind commented Aug 5, 2024 •

edited

Loading

github-actions bot commented Aug 5, 2024 •

edited

Loading

github-actions bot commented Aug 5, 2024 •

edited

Loading

github-actions bot commented Aug 20, 2024 •

edited

Loading