Setting a lot of data in Onyx is slow

[iwiznia]

Coming from #21444

Threads with context:
https://expensify.slack.com/archives/C03TQ48KC/p1687893191020179
https://expensify.slack.com/archives/C036S0BVAHH/p1687812903398059

Problem

Setting a lot of data in Onyx is super slow.

This started/finished is calls to onyx. You can see it takes 20 seconds to set the key 19 and 1 minute 8 seconds the second one.

19:58:45.761 web.development.js:1304 started 19 () => mergeCollection(key, value)
19:58:45.762 web.development.js:1304 started 20 () => mergeCollection(key, value)
19:59:06.247 web.development.js:1305 finished 19 () => mergeCollection(key, value)
19:59:53.190 web.development.js:1305 finished 20 () => mergeCollection(key, value)

I have a file with the data that's failing, but I think it might contain sensitive information but I can say it's 30MB uncompressed. The 20th call above is the biggest as one might expect.

In order to test this in dev (for internal engineers only) is to change the OpenApp code to (data.json is the file with all the data):

    } elseif ($command === 'OpenApp') {
        $response = json_decode(file_get_contents('data.json'), true);
//        $response['onyxData'] = AppInit::openApp($authToken, $_REQUEST['policyIDList'] ?? '');

I assume it can also be reproduced by joining 1000s of policies. This user's data:

• 3216 policies
• 7k personal details (x 2)
• 9k reports

Solution

Either make onyx be faster or recognize we need to do pagination of policies and reports

Upwork Automation - Do Not Edit

• Upwork Job URL: https://www.upwork.com/jobs/~01a864f2f0848526ae
• Upwork Job ID: 1676300792032952320
• Last Price Increase: 2023-07-04

[melvin-bot]

Triggered auto assignment to @michaelhaxhiu (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

[melvin-bot]

Bug0 Triage Checklist (Main S/O)

• This "bug" occurs on a supported platform (ensure Platforms in OP are ✅)
• This bug is not a duplicate report (check E/App issues and #expensify-bugs)
  • If it is, comment with a link to the original report, close the issue and add any novel details to the original issue instead
• This bug is reproducible using the reproduction steps in the OP. S/O
  • If the reproduction steps are clear and you're unable to reproduce the bug, check with the reporter and QA first, then close the issue.
  • If the reproduction steps aren't clear and you determine the correct steps, please update the OP.
• This issue is filled out as thoroughly and clearly as possible
  • Pay special attention to the title, results, platforms where the bug occurs, and if the bug happens on staging/production.
• I have reviewed and subscribed to the linked Slack conversation to ensure Slack/Github stay in sync

[iwiznia]

Updated the data above, just realized that we have 2x the number of personal details that we need to due to the migration to accountIDs, so if we remove that, this will get a bit better, but not nearly enough since most of the time is on the reports.

[melvin-bot]

@michaelhaxhiu Eep! 4 days overdue now. Issues have feelings too...

[michaelhaxhiu]

@iwiznia what should i do with this one? Does it go to Engineering?

[iwiznia]

For the pagination stuff, yes.
For the making onyx faster it can go to contributors, in fact I thought there were some initiatives that could help with it (like sqlite backend for onyx), but not sure what state they are in.

[michaelhaxhiu]

Sorry I just mean what did you hope to see for this particular GH that you made? Do you want it to go to external or internal for now?