Onyx File handling - preparation by kidroca · Pull Request #4908 · Expensify/App

kidroca · 2021-08-30T13:48:08Z

Details

This is a proof of concept work that showcases file handling in Onyx as per the linked issue
Onyx was updated with peer dependency to localforage
Onyx PR: Expensify/react-native-onyx#102

Deleted listenToStorageEvents lib - this is now handled using IndexedDb and localforage in Onyx
Deleted SignOutManager lib - the updated Onyx.clear logic is synchronized across tabs and makes it unnecessary

Fixed Issues

$ #3867

Tests

~~#### For all platforms~~

1. Go offline
2. Open a conversation
3. Select and send an attachment
4. Terminated the app
5. Resume connectivity
6. Open the app again
7. Go to the conversation and see the file appearing after it's uploaded
- depending on how big is the file or how fast you do the last step - the file might already be uploaded

Edit: The change needed for this was reverted
It will be added and persisting attachments would work after we resolve: #6556

Regression check for Web

Multiple Tabs

Open multiple tabs
Open the same conversation on multiple tabs
Write a draft message (without sending it)
Observe the message is update across different tabs

Multiple Tabs (Signing out)

Again on multiple tabs
Log out from one tab
You should be logged out on all tabs
You should be able to log back in without problems (from the tab you logged out from or from another tab)

QA Steps

Tested On

Screenshots

Web

No regressions on persisting plain messages

New.Expensify.-.Google.Chrome.2021-12-15.01-57-51.mp4

Using multiple tabs/windows

2021-11-03_22-05-01.mp4

Mobile Web

Screen.Recording.2021-12-15.at.2.19.10.mov

Desktop

Screen.Recording.2021-12-15.at.2.12.51.mov

iOS

Screen.Recording.2021-12-15.at.2.15.09.mov

Android

Android.Emulator.-.Pixel_2_API_29_5554.2021-12-15.02-29-11.mp4

LocalForage fails to save the value because functions cannot be serialized

Remove the `!isAttachment` rule as now we can persist Files

The lock file was update after expo-file-system was uninstalled

Synchronize tab functionality is moved to Onyx

# Conflicts: # ios/Podfile.lock # package.json # src/components/AttachmentPicker/index.native.js # src/components/OnyxProvider.js # src/libs/listenToStorageEvents/index.js

kidroca

Mention again that registerStorageEventListener is no longer needed as it's covered internally in Onyx

kidroca · 2021-10-19T18:10:47Z

src/setup/index.js

-        registerStorageEventListener: (onStorageEvent) => {
-            listenToStorageEvents(onStorageEvent);
-        },


Synchronizing storage between tabs is handled internally by Onyx as it depends on the underlying storage implementation.
See: https://github.com/kidroca/react-native-onyx/blob/b6ad24440fe0b07b314373c1a4cfdf7914febc3f/lib/storage/index.js#L10-L15

# Conflicts: # android/app/src/main/java/com/expensify/chat/generated/BasePackageList.java # ios/Podfile.lock # package-lock.json # package.json # src/setup/index.js

Onyx no longer needs this Removing `prepareFile` usage

The document picker makes the attachment available until the application is terminated To persist Document Attachments and be able to use them on the next app launch we need to move them to the cache dir This way we can persist a reference for the attachment in Onyx and support offline files The `ImagePicker` library is doing this by default

There seems to be some problem and nothing happens when image is selected or when an image is shot

This fixes the issue with the image picker not working

kidroca

Updated the Document/Image picker libraries in an attempt to solve an issue I had
It turned it it wasn't caused by these libraries, but I suppose now it's a good time to update them - QA will use the Attachment picker to test the changes made here anyway

kidroca · 2021-11-02T15:56:50Z

src/components/AttachmentPicker/index.native.js

        name: fileData.fileName || fileData.name || 'chat_attachment',
        type: fileData.type,
-        uri: fileData.uri,
+        uri: fileData.fileCopyUri || fileData.uri,


The document picker makes the attachment available until the application is terminated
To persist Document Attachments and be able to use them on the next app launch we need to move them to the cache dir
This way we can persist a reference for the attachment in Onyx and support offline files

The ImagePicker library is already making a copy in cache by default

kidroca · 2021-11-02T18:22:59Z

src/components/AttachmentPicker/index.native.js

-      * @param {ImagePickerResponse|DocumentPickerResponse} attachment
+      * @param {Array<ImagePickerResponse|DocumentPickerResponse>} attachments
      */
-    pickAttachment(attachment) {
-        if (attachment) {
-            if (attachment.width === -1 || attachment.height === -1) {
-                this.showImageCorruptionAlert();
-                return;
-            }
-            const result = getDataForUpload(attachment);
-            this.completeAttachmentSelection(result);
+    pickAttachment(attachments = []) {
+        if (attachments.length === 0) {
+            return;
        }
+
+        const fileData = _.first(attachments);
+
+        if (fileData.width === -1 || fileData.height === -1) {
+            this.showImageCorruptionAlert();
+            return;
+        }
+
+        const result = getDataForUpload(fileData);
+        this.completeAttachmentSelection(result);


The updated document picker responds with an array. The image picker would also respond with array that's why this was changed to accept an array value

# Conflicts: # package-lock.json # package.json

The Pod file already has this version of Flipper and pipelines are failing due to the mismatch

kidroca · 2022-01-06T19:04:01Z

Hey @tgolen sorry just pushed an update to fix the pipeline

kidroca · 2022-01-12T12:25:02Z

I think we forgot to remove the [HOLD] label here
It was added to prevent merging this prematurely but it's in a ready state now
File handling would happen in 2 steps

Current PR transitions us to IndexedDb and the new Onyx implementation with storage providers
Another PR would enable file persistens after Network: Persisted Requests Queue - clear persisted requests after the response #6556 is merged

cc @thienlnam

thienlnam

Sorry for the late review, just a couple questions

thienlnam · 2022-01-11T19:53:57Z

src/ONYXKEYS.js

-    // Notifies all tabs that they should sign out and clear storage.
-    SHOULD_SIGN_OUT: 'shouldSignOut',
-


Is this just not used? Not seeing it in any changes from this PR

This was used here: src/libs/SignoutManager.js
But now the file is deleted - no longer necessary

What the SignoutManager did was to synchronize the logout across multiple tabs
But now with localForage this is no longer needed
We're touching on this here: Expensify/react-native-onyx#102 (comment)

thienlnam · 2022-01-12T22:32:21Z

src/components/AttachmentPicker/index.native.js

  */
 const documentPickerOptions = {
    type: [RNDocumentPicker.types.allFiles],
+    copyTo: 'cachesDirectory',


Does android have a cachesDirectory? I only see it for IOS

Documentation says:

[iOS and Android only] copyTo:"cachesDirectory" | "documentDirectory"
https://www.npmjs.com/package/react-native-document-picker#ios-and-android-only-copytocachesdirectory--documentdirectory

I take it the option copyTo and the value cachesDirectory is available in both

kidroca

Thanks @thienlnam, I hope these answer your questions well

kidroca · 2022-01-14T19:38:42Z

src/ONYXKEYS.js

-    // Notifies all tabs that they should sign out and clear storage.
-    SHOULD_SIGN_OUT: 'shouldSignOut',
-


This was used here: src/libs/SignoutManager.js
But now the file is deleted - no longer necessary

What the SignoutManager did was to synchronize the logout across multiple tabs
But now with localForage this is no longer needed
We're touching on this here: Expensify/react-native-onyx#102 (comment)

kidroca · 2022-01-14T19:44:19Z

src/components/AttachmentPicker/index.native.js

  */
 const documentPickerOptions = {
    type: [RNDocumentPicker.types.allFiles],
+    copyTo: 'cachesDirectory',


Documentation says:

[iOS and Android only] copyTo:"cachesDirectory" | "documentDirectory"
https://www.npmjs.com/package/react-native-document-picker#ios-and-android-only-copytocachesdirectory--documentdirectory

I take it the option copyTo and the value cachesDirectory is available in both

thienlnam

Thanks for those comments, this looks all good on my end 👍 Thanks for those comments, this looks all good on my end 👍

kidroca · 2022-01-14T20:14:12Z

🤦‍♂️ It seems this closed another PR due to the added description:

Edit: The change needed for this was reverted
It will be added and persisting attachments would work after we resolve: #6556

OSBotify · 2022-01-14T20:20:53Z

🚀 Deployed to staging by @thienlnam in version: 1.1.29-7 🚀

platform	result
🤖 android 🤖	success ✅
🖥 desktop 🖥	success ✅
🍎 iOS 🍎	failure ❌
🕸 web 🕸	success ✅

mountiny · 2022-01-16T22:50:40Z

There is a deploy blocker causing that PDF attachment shows an infinite spinner on Android (only as far as the QA team found).

This PR seems like the only one in the latest deployment, which is related. Would @kidroca be able to look into this and confirm if it is related?

If yes, would we revert this PR or would you be able to identify a fix to CP it soon?

Thank you for your help! 🙇

kidroca · 2022-01-17T07:35:47Z

I'll investigate

kidroca · 2022-01-17T10:45:33Z

@mountiny
Correct this PR introduced the regression, but I've found the problem and submitted a small PR: #7277

kidroca · 2022-01-17T10:47:45Z

BTW the current PR applies a migration: https://github.com/Expensify/App/pull/4908/files#diff-fc59a9d775dba3004d8ed5a4dec154bbd2487455e7c720afed29bc8003a5228f

Reverting the PR won't revert the migration and the result would be users getting logged out when they open the app

IMO we should not revert, but apply #7277

mountiny · 2022-01-17T12:34:12Z

@kidroca Thank you very much for looking into this. We should indeed be careful with migrations like this. It might not be very common to run a migration, but maybe it might be worth making PRs like this clearly state it.

For example, the PR could have a [Migration] prefix so then every engineer will immediately know it might not be easy to revert the PR. What do you think, @tgolen and @kidroca?

tgolen · 2022-01-18T16:31:28Z

I like the idea of adding that tag try avoid reverting a PR with a migration. We could use a label too (which would probably be better).

…

On Mon, Jan 17, 2022 at 5:34 AM Vit Horacek ***@***.***> wrote: @kidroca <https://github.com/kidroca> Thank you very much for looking into this. We should indeed be careful with migrations like this. It might not be very common to run a migration, but maybe it might be worth making PRs like this clearly state it. For example, the PR could have a [Migration] prefix so then every engineer will immediately know it might not be easy to revert the PR. What do you think, @tgolen <https://github.com/tgolen> and @kidroca <https://github.com/kidroca>? — Reply to this email directly, view it on GitHub <#4908 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJMAB2HIPSYIXVETSE6XSLUWQEFDANCNFSM5DB5KQ2A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

OSBotify · 2022-01-18T22:02:56Z

🚀 Deployed to production by @roryabraham in version: 1.1.30-3 🚀

platform	result
🤖 android 🤖	success ✅
🖥 desktop 🖥	success ✅
🍎 iOS 🍎	success ✅
🕸 web 🕸	success ✅

kidroca added 9 commits August 30, 2021 16:28

Package updates add localforage and updated Onyx

9c287f3

Omit function values from the persisted request

ee4f659

LocalForage fails to save the value because functions cannot be serialized

Persist all addAction requests

f37b502

Remove the `!isAttachment` rule as now we can persist Files

Add Expo File System to be used for mobile persistence

5fd5dab

Use Onyx.prepareFile in addAction

b677931

Fix: AttachmentPicker Capture size information correctly

4a484ec

Update onyx hash, update lock file

91d8f40

The lock file was update after expo-file-system was uninstalled

Run pod install and add expo-file-system

a2ac108

Remove storage event listener and logic

e474acd

Synchronize tab functionality is moved to Onyx

kidroca mentioned this pull request Aug 30, 2021

[Discussion] Add the ability to handle images in Onyx #3867

Closed

kidroca force-pushed the kidroca/file-handling-poc branch from 9e64fec to 90e0925 Compare August 30, 2021 15:11

Update onyx hash, update lock file

3ea51ec

kidroca force-pushed the kidroca/file-handling-poc branch from 90e0925 to 3ea51ec Compare August 30, 2021 15:46

kidroca closed this Aug 30, 2021

kidroca deleted the kidroca/file-handling-poc branch August 30, 2021 19:29

kidroca restored the kidroca/file-handling-poc branch August 30, 2021 19:29

kidroca reopened this Aug 30, 2021

Merge branch 'main' into kidroca/file-handling-poc

e9c5d50

# Conflicts: # ios/Podfile.lock # package.json # src/components/AttachmentPicker/index.native.js # src/components/OnyxProvider.js # src/libs/listenToStorageEvents/index.js

kidroca commented Oct 19, 2021

View reviewed changes

kidroca added 10 commits November 1, 2021 22:53

Update Onyx hash with the latest file storage POC

4073718

Merge branch 'main' into kidroca/file-handling-poc

5971da5

# Conflicts: # android/app/src/main/java/com/expensify/chat/generated/BasePackageList.java # ios/Podfile.lock # package-lock.json # package.json # src/setup/index.js

Remove expo-file-system

044278d

Onyx no longer needs this Removing `prepareFile` usage

Report.js remove Onyx.prepareFile call

1da8e87

Bump react-native-flipper version

8d0f6cd

Update Pods after package changes

d33702c

Update Onyx hash after file handling changes

7805eb0

Update image picker and document picker libraries

bbfdc5c

There seems to be some problem and nothing happens when image is selected or when an image is shot

Revert reanimated to the previous version

2a02fb1

This fixes the issue with the image picker not working

kidroca commented Nov 2, 2021

View reviewed changes

kidroca requested a review from tgolen January 6, 2022 14:50

Merge branch 'main' into kidroca/file-handling-poc

555d9ae

# Conflicts: # package-lock.json # package.json

kidroca force-pushed the kidroca/file-handling-poc branch from a2459be to 555d9ae Compare January 6, 2022 15:07

tgolen previously approved these changes Jan 6, 2022

View reviewed changes

Add flipper 0.117.0 to match the pod speck

3266600

The Pod file already has this version of Flipper and pipelines are failing due to the mismatch

kidroca dismissed tgolen’s stale review via 3266600 January 6, 2022 19:03

tgolen approved these changes Jan 7, 2022

View reviewed changes

kidroca changed the title ~~[HOLD] Onyx File handling - preparation~~ Onyx File handling - preparation Jan 12, 2022

thienlnam reviewed Jan 12, 2022

View reviewed changes

kidroca commented Jan 14, 2022

View reviewed changes

thienlnam approved these changes Jan 14, 2022

View reviewed changes

thienlnam merged commit 537101f into Expensify:main Jan 14, 2022

OSBotify mentioned this pull request Jan 14, 2022

Deploy Checklist: New Expensify 2022-01-14 #7235

Closed

66 tasks

kidroca mentioned this pull request Jan 14, 2022

Network: Persisted Requests Queue - clear persisted requests after the response #6556

Merged

5 tasks

kidroca deleted the kidroca/file-handling-poc branch January 14, 2022 20:22

mountiny mentioned this pull request Jan 16, 2022

Android - Chat - Infinite loading spinner appears when adding PDF attachment #7267

Closed

kidroca mentioned this pull request Jan 17, 2022

[CP Staging] Fix document upload on Android #7277

Merged

9 tasks

neil-marcellini mentioned this pull request Apr 22, 2022

Update README to explain that Onyx data is stored in IndexedDB on Web #8755

Closed

parasharrajat mentioned this pull request Aug 10, 2022

[HOLD #10894][Image][Performance][$2000] Feature Request - Attachments should be cached after first load. It currently load from internet every time - Reported by: @parasharrajat #6527

Closed

		// Notifies all tabs that they should sign out and clear storage.
		SHOULD_SIGN_OUT: 'shouldSignOut',

Conversation

kidroca commented Aug 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Details

Fixed Issues

Tests

Regression check for Web

Multiple Tabs

Multiple Tabs (Signing out)

QA Steps

Tested On

Screenshots

Web

No regressions on persisting plain messages

Using multiple tabs/windows

Mobile Web

Desktop

iOS

Android

Uh oh!

kidroca left a comment

Choose a reason for hiding this comment

Uh oh!

kidroca Oct 19, 2021

Choose a reason for hiding this comment

Uh oh!

kidroca left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kidroca Nov 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kidroca Nov 2, 2021

Choose a reason for hiding this comment

Uh oh!

kidroca commented Jan 6, 2022

Uh oh!

kidroca commented Jan 12, 2022

Uh oh!

thienlnam left a comment

Choose a reason for hiding this comment

Uh oh!

thienlnam Jan 11, 2022

Choose a reason for hiding this comment

Uh oh!

kidroca Jan 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thienlnam Jan 12, 2022

Choose a reason for hiding this comment

Uh oh!

kidroca Jan 14, 2022

Choose a reason for hiding this comment

Uh oh!

kidroca left a comment

Choose a reason for hiding this comment

Uh oh!

kidroca Jan 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kidroca Jan 14, 2022

Choose a reason for hiding this comment

Uh oh!

thienlnam left a comment

Choose a reason for hiding this comment

Uh oh!

kidroca commented Jan 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

OSBotify commented Jan 14, 2022

Uh oh!

mountiny commented Jan 16, 2022

Uh oh!

kidroca commented Jan 17, 2022

Uh oh!

kidroca commented Jan 17, 2022

kidroca commented Aug 30, 2021 •

edited

Loading

kidroca left a comment •

edited

Loading

kidroca Nov 2, 2021 •

edited

Loading

kidroca Jan 14, 2022 •

edited

Loading

kidroca Jan 14, 2022 •

edited

Loading

kidroca commented Jan 14, 2022 •

edited

Loading