Allow priority requests to go to the front of the sequential queue #23669
Conversation
|
@thesahindia Please copy/paste the Reviewer Checklist from here into a new comment on this PR and complete it. If you have the K2 extension, you can simply click: [this button] |
tgolen
requested review from
a team and
danieldoglas
and removed request for
thesahindia
melvin-bot
bot
requested review from
thesahindia
and removed request for
a team
|
@thesahindia @ One of you needs to copy/paste the Reviewer Checklist from here into a new comment on this PR and complete it. If you have the K2 extension, you can simply click: [this button] |
danieldoglas
left a comment
danieldoglas
left a comment
There was a problem hiding this comment.
PR looks great, but there's one change we should do for clarity.
I tested and everything worked as we expected for this PR. let's just call the correct method and we should be good!
src/libs/actions/User.js
Outdated
| App.reconnectApp(onyxUpdatesLastUpdateID, pushJSONLastUpdateID, true); | ||
| } |
There was a problem hiding this comment.
@tgolen I think you missed this, but you created a GetMissingOnyxMessages method in the API to use exactly on this situation instead of using ReconnectApp. I think we should use it here for clarity.
There was a problem hiding this comment.
OK, I will switch to using this.
src/libs/actions/User.js
Outdated
| console.debug('[OnyxUpdates] The lastUpdateID the client received was', onyxUpdatesLastUpdateID); | ||
|
|
There was a problem hiding this comment.
| console.debug('[OnyxUpdates] The lastUpdateID the client received was', onyxUpdatesLastUpdateID); | |
| console.debug('[OnyxUpdates] The lastUpdateID the client had was', onyxUpdatesLastUpdateID); | |
Just for clarity, I was confused if this was the one I received or the one we had when just looking at the logs
|
OK, I had to do a pretty major rewrite in order to DRY the code between Pusher responses and HTTPS responses. The problem was that the middleware for I got around the problem by using Onyx keys and a callback function to listen for when the updateIDs sent from the server have changed. |
danieldoglas
left a comment
danieldoglas
left a comment
There was a problem hiding this comment.
Looking great, just a few NAB comments and one thing we need to fix!
src/libs/Pusher/pusher.js
Outdated
| data = _.isObject(eventData) ? eventData : JSON.parse(eventData); | ||
| } catch (err) { | ||
| Log.alert('[Pusher] Unable to parse JSON response from Pusher', {error: err, eventData}); | ||
| Log.alert('[Pusher] Unable to parse JSON response from Pusher 1', {error: err, eventData}); |
There was a problem hiding this comment.
NAB: is this intentional?
There was a problem hiding this comment.
Sort of, yeah, I can do it better though. The same log happens from two different places and it wasn't possible to tell from which spot I was seeing an error.
src/libs/Pusher/pusher.js
Outdated
| eventCallback(JSON.parse(chunkedEvent.chunks.join(''))); | ||
| } catch (err) { | ||
| Log.alert('[Pusher] Unable to parse chunked JSON response from Pusher', { | ||
| Log.alert('[Pusher] Unable to parse chunked JSON response from Pusher 2', { |
There was a problem hiding this comment.
NAB: is this intentional?
src/libs/actions/App.js
Outdated
| // getMissingOnyxUpdates will fetch updates starting from the last update this client got and going to the last update the server sent. | ||
| if (lastUpdateIDAppliedToClient && previousUpdateIDFromServer && lastUpdateIDAppliedToClient < previousUpdateIDFromServer) { | ||
| console.debug('[OnyxUpdates] Gap detected in update IDs so fetching incremental updates'); | ||
| Log.info('Gap detected in update IDs from Pusher so fetching incremental updates', true, { |
There was a problem hiding this comment.
| Log.info('Gap detected in update IDs from Pusher so fetching incremental updates', true, { | |
| Log.info('Gap detected in update IDs from server so fetching incremental updates', true, { |
NAB: Just suggest we change this from server since this will be triggered from both Pusher and HTTPs requests
| // For most requests we can immediately update Onyx. For write requests we queue the updates and apply them after the sequential queue has flushed to prevent a replay effect in | ||
| // the UI. See https://github.com/Expensify/App/issues/12775 for more info. | ||
| const updateHandler = request.data.apiRequestType === CONST.API_REQUEST_TYPE.WRITE ? QueuedOnyxUpdates.queueOnyxUpdates : Onyx.update; | ||
| const updateHandler = request.data.apiRequestType === CONST.API_REQUEST_TYPE.WRITE ? QueuedOnyxUpdates.queueOnyxUpdates : updateOnyx; |
There was a problem hiding this comment.
I think we should change both requests here, not only the usage of Onyx.Update.
I say that because write requests are the ones that actually generate an onyx response for all clients. If you call OpenInitialSettings, which is a read, we don't send data through onyx to it since it's just for the local client, it won't change anything. So that's not written in onyxUpdates.
But if you call OpenReport, which is a write and returns lastUpdateID and previousUpdateID (since we update the lastRead property of the report), we're not saving updating those properties locally
So let's say we're sending messages to someone, then we go from report A to report B and receive a new message, that would trigger GetOnyxUpdates, which is not ideal.
There was a problem hiding this comment.
Thanks for pointing that out. Makes sense!
danieldoglas
left a comment
danieldoglas
left a comment
There was a problem hiding this comment.
Added a few more considerations after testing
src/libs/actions/App.js
Outdated
| // Update the local values to be the same as the values stored in Onyx | ||
| onyxUpdates.lastUpdateIDFromServer = lastUpdateIDFromServer || 0; | ||
| onyxUpdates.previousUpdateIDFromServer = previousUpdateIDFromServer || 0; |
There was a problem hiding this comment.
Why? We are not using this local variables for anything
src/libs/actions/App.js
Outdated
| const onyxUpdates = { | ||
| lastUpdateIDFromServer: 0, | ||
| previousUpdateIDFromServer: 0, | ||
| }; |
There was a problem hiding this comment.
This object is set and updated, but we never read from it. we can remove it since we use the values directly from Onyx in the function below
There was a problem hiding this comment.
I rewrote this so many times yesterday I got tunnelvision. Thanks for your fresh eyes on this! I've cleaned it up to remove this.
src/libs/actions/App.js
Outdated
| Onyx.merge(ONYXKEYS.ONYX_UPDATES_LAST_UPDATE_ID_APPLIED_TO_CLIENT, lastUpdateIDFromServer || 0); | ||
| }, |
There was a problem hiding this comment.
From your PR I was able to see something I wasn't seeing before: We also send GetOnyxUpdate responses through Pusher, so we're updating the current client and sending those updates to other clients. I think that's fine since if we're sending it to all clients, the changes the clients are updated are bigger.
What that does is:
- Client A and B are using account 1
- Client A has OnyxUpdatesLastUpdateIDAppliedToClient with 51
- Client B as OnyxUpdatesLastUpdateIDAppliedToClient with 47
- Account 2 sends a message to Account 1 which will be updateID 55
- Client B for some reason doesn't receive it
- Client A receives, and calls
GetOnyxUpdateswithfrom: 51andto: 55 - Client B, that has it's connection working now, receives the
GetOnyxUpdatesfrom Pusher, sees that the previousUpdateID in that update is bigger than 47, and calls GetOnyxUpdates with valuesfrom: 47andto: 51, and gets up to date.
This works OK!
Now we have an edge case that happened when I was testing.
- Client A and B are using account 1
- Client B as OnyxUpdatesLastUpdateIDAppliedToClient with 47
- Account 2 sends a message to Account 1 which will be updateID 55
- Client A receives it instantly and now has OnyxUpdatesLastUpdateIDAppliedToClient with 55. Then it does some actions, and that number rises to 60
- Client B for some reason doesn't receive those
- Client B connects and receives update 60. It notices it didn't have 47-59, and does a
GetOnyxUpdateswith those - Client A receives those updates since we're testing. It won't apply those, since the ID is less than the one it currently has BUT it will save
OnyxUpdatesLastUpdateIDAppliedToClientwith lastUpdateID 58 that was sent in the pusher update body. - So next time it tries to do anything, it will call
GetMissingOnyxUpdatesagain
I think we solve that by adding this to the if statement above.
There was a problem hiding this comment.
OK, to make sure I understand your proposed change clearly, you are suggesting to only update OnyxUpdatesLastUpdateIDAppliedToClient if the lastUpdateIDFromServer is larger. Right? That way the value will never decrease, only increase.
|
@thesahindia can you please review this? |
danieldoglas
requested review from
rushatgabhane
and removed request for
thesahindia
Reviewer Checklist
Screenshots/VideosWebScreen.Recording.2023-08-11.at.15.12.20.movScreen.Recording.2023-08-11.at.15.09.32.movMobile Web - Chrome
Mobile Web - SafariScreen.Recording.2023-08-11.at.16.36.27.movDesktopScreen.Recording.2023-08-11.at.16.36.27.moviOS9f60bd04-d856-4e20-8e5c-7e945be19055.mp4Android
|
|
|
||
| // If the previous update from the server does not match the last update the client got, then the client is missing some updates. | ||
| // getMissingOnyxUpdates will fetch updates starting from the last update this client got and going to the last update the server sent. | ||
| if (lastUpdateIDAppliedToClient && previousUpdateIDFromServer && lastUpdateIDAppliedToClient < previousUpdateIDFromServer) { |
There was a problem hiding this comment.
if expensify goes to mars and beyond, we mayy have more than 2^63 - 1 server updates for users by then.
and then they'll stop getting any updates.
There was a problem hiding this comment.
If that's the case, I'll happily come back here and update the logic 😄
src/ONYXKEYS.js
Outdated
| // The access token to be used with the Mapbox library | ||
| MAPBOX_ACCESS_TOKEN: 'mapboxAccessToken', | ||
| // Information about the onyx updates IDs that were received from the server | ||
| ONYX_UPDATES: 'onyxUpdates', |
There was a problem hiding this comment.
NAB: These are server updates only. I would have named them ONYX_SERVER_UPDATES
There was a problem hiding this comment.
Updated branch and renamed this to ONYX_UPDATES_FROM_SERVER following your suggestion (I think the name matches the other key better).
|
@cristipaval Please copy/paste the Reviewer Checklist from here into a new comment on this PR and complete it. If you have the K2 extension, you can simply click: [this button] |
|
🎯 @rushatgabhane, thanks for reviewing and testing this PR! 🎉 An E/App issue has been created to issue payment here: #24418. |
|
✋ This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release. |
|
🚀 Deployed to staging by https://github.com/danieldoglas in version: 1.3.54-0 🚀
|
1 similar comment
|
🚀 Deployed to staging by https://github.com/danieldoglas in version: 1.3.54-0 🚀
|
|
@tgolen are we ok to check it off? As its internal |
|
Yep, you can check it off. We'll be manually testing it this week |
|
🚀 Deployed to production by https://github.com/yuwenmemon in version: 1.3.54-13 🚀
|
Details
When the app reconnects to the network and needs to get incremental OnyxUpdates from the server, that request needs to be prioritized at the front of the sequential queue so that it is processed immediately and the pending updates are applied to the most recent data from the server.
Fixed Issues
$ https://github.com/Expensify/Expensify/issues/299547
Tests
Note: This can only be tested on Web by internal engineers for now
[OnyxUpdates]For C+ tester
Do a general test on the app and confirm that everything still works as intended.
Offline tests
None
QA Steps
Same as tests
PR Author Checklist
### Fixed Issuessection aboveTestssectionOffline stepssectionQA stepssectiontoggleReportand notonIconClick)myBool && <MyComponent />.src/languages/*files and using the translation methodWaiting for Copylabel for a copy review on the original GH to get the correct copy.STYLE.md) were followedAvatar, I verified the components usingAvatarare working as expected)/** comment above it */thisproperly so there are no scoping issues (i.e. foronClick={this.submit}the methodthis.submitshould be bound tothisin the constructor)thisare necessary to be bound (i.e. avoidthis.submit = this.submit.bind(this);ifthis.submitis never passed to a component event handler likeonClick)StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG))Avataris modified, I verified thatAvataris working as expected in all cases)ScrollViewcomponent to make it scrollable when more elements are added to the page.mainbranch was merged into this PR after a review, I tested again and verified the outcome was still expected according to theTeststeps.Screenshots/Videos
Web
Mobile Web - Chrome
Note: this is only testable on web
Mobile Web - Safari
Note: this is only testable on web
Desktop
Note: this is only testable on web
iOS
Note: this is only testable on web
Android
Note: this is only testable on web