fix wasteful backup search in inbox on Gmail API by Kharchevskyi · Pull Request #355 · FlowCrypt/flowcrypt-ios

Kharchevskyi · 2021-06-17T20:31:40Z

This PR fix wasteful backup search in inbox on Gmail API

close #309

Tests:

Start working on GmailService tests
Added mocks implementations for GoogleUserServiceType and GmailBackupSearchQueryProviderType

To be filled by reviewers

I have reviewed that this PR... (tick whichever items you personally focused on during this review):

addresses the issue it closes (if any)
code is readable and understandable
is accompanied with tests, or tests are not needed
is free of vulnerabilities
is documented clearly and usefully, or doesn't need documentation

tomholub

Code looks good, will test functionality manually now

tomholub

This still does not address the issue. When I try it on my personal account, it shows:

"ℹ️[App Start][GlobalRouter] proceed for session nil"
"ℹ️[Keychain][KeyChainService] LocalStorage.secureKeychainPrefix generating new: wFjxOWGVu77zZW4o"
"ℹ️[Keychain][KeyChainService] generateAndSaveStorageEncryptionKey"
"ℹ️[Js][Core] JsContext took 0.532 ms to start"
"ℹ️[App Start][GlobalRouter] Sign in with gmailLogin(<FlowCrypt.SignInViewController: 0x108059c00>)"
"ℹ️[App Start][GlobalRouter] proceed for session Optional(FlowCrypt.SessionType.google(...)"
"ℹ️[App Start][GoogleUserService] Renew session for google user"
"ℹ️ [Setup] searching for backups in inbox"
"🏷[GmailService] will begin searching for backups"
"🏷[GmailService] searching done, found 24 backup messages"
"🏷[GmailService] processing backup \'Your FlowCrypt Backup\' with 1 attachments"
"🏷[GmailService] processing backup \'Fwd: Your FlowCrypt Backup\' with 1 attachments"
"🏷[GmailService] processing backup \'Fwd: Your FlowCrypt Backup\' with 1 attachments"
"🏷[GmailService] processing backup \'Re: Problems with my Flowcrypt account\' with 1 attachments"
"🏷[GmailService] processing backup \'Your FlowCrypt Backup\' with 1 attachments"
"🏷[GmailService] processing backup \'FlowCrypt Feedback from ...@gmail.com\' with 1 attachments"
"🏷[GmailService] processing backup \'Re: Your FlowCrypt Backup\' with 2 attachments"
"🏷[GmailService] processing backup \'Fwd: Your FlowCrypt Backup\' with 1 attachments"
"🏷[GmailService] processing backup \'Please Provide Assistance\' with 1 attachments"
"🏷[GmailService] processing backup \'Re: Your FlowCrypt Backup\' with 2 attachments"
"🏷[GmailService] processing backup \'FlowCrypt Feedback from ...@gmail.com\' with 1 attachments"
"🏷[GmailService] processing backup \'Fwd: Your FlowCrypt Backup\' with 1 attachments"
"🏷[GmailService] processing backup \'Re: Passphrase lost\' with 1 attachments"
"🏷[GmailService] processing backup \'Re: FlowCrypt Feedback from ...@gmail.com\' with 3 attachments"
"🏷[GmailService] processing backup \'Pass phrase\' with 1 attachments"
"🏷[GmailService] processing backup \'Re: Problems with my Flowcrypt account\' with 1 attachments"
"🏷[GmailService] processing backup \'Re: Your FlowCrypt Backup\' with 1 attachments"
"🏷[GmailService] processing backup \'Re: Passphrase lost\' with 1 attachments"
"🏷[GmailService] processing backup \'Re: Your FlowCrypt Backup\' with 2 attachments"
"🏷[GmailService] processing backup \'Fwd: Your FlowCrypt Backup\' with 1 attachments"
"🏷[GmailService] processing backup \'FlowCrypt Feedback from ...@gmail.com\' with 1 attachments"
"🏷[GmailService] processing backup \'Re: Your FlowCrypt Backup\' with 2 attachments"
"🏷[GmailService] processing backup \'FlowCrypt Feedback from ...@gmail.com\' with 1 attachments"
"🏷[GmailService] processing backup \'Re: Passphrase lost\' with 1 attachments"
"🏷[GmailService] downloading 30 attachments with possible backups in them"
"🏷[GmailService] downloaded 30 attachments that contain 2171kB of data"
"❗️[Js][Core] ------ js err -------\nCore parseKeys:\nTypeError: coreHost.log is not a function. (In \'coreHost.log(String(x))\', \'coreHost.log\' is undefined)\n_log\n\n\nasyncFunctionResume@[native code]\n[native code]\npromiseReactionJobWithoutPromise@[native code]\n------- end js err -----"

tomholub · 2021-06-21T14:25:52Z

Problems:

it's supposed to search for the text only in subject, but it actually searches the text anywhere in the email
it's supposed to search for backups from:<my primary email>, to:<my primary email> but these filters are missing, so it's picking up on emails that I've sent to others, or that others sent to me, which is not valid

Also it should NOT be searching for items that are in trash. I think in:anywhere may be also searching in Trash wrongly? (unsure)

tomholub · 2021-06-21T14:31:09Z

Btw you don't have to construct the search query manually. If you can supply a textual query, then use endpoint called gmailBackupSearch - just add a Core.gmailBackupSearch method. Input is acctEmail where you put the account email address, and back you get the constructed search query that you will know is correct.

https://github.com/FlowCrypt/flowcrypt-mobile-core/blob/b7d8d10bdd2b5c17e17ec03505801e8bf7e89073/source/mobile-interface/endpoints.ts#L224

tomholub · 2021-06-21T14:32:37Z

If you do prefer to construct this query in Swift anyway for some reason, then here is an exact test case to copy, the test case you wrote is too vague:

https://github.com/FlowCrypt/flowcrypt-mobile-core/blob/b7d8d10bdd2b5c17e17ec03505801e8bf7e89073/source/test.ts#L280

Kharchevskyi · 2021-06-22T16:31:44Z

@tomholub can you please check with query generated with Core

tomholub

This looks like it will work - will test

...rypt/Functionality/Mail Provider/SearchMessage Provider/GmailSearchExpressionGenerator.swift

FlowCrypt/Core/Core.swift

…ure/issue-309-backups

tomholub

Code looks good. Will test

tomholub · 2021-06-23T21:33:10Z

This worked for the search, but after entering pass phrase, it sent me back to splash screen. Log:

"ℹ️[Keychain][KeyChainService] LocalStorage.secureKeychainPrefix generating new: TyvDJ6VPktNBrGQO"
"ℹ️[Keychain][KeyChainService] generateAndSaveStorageEncryptionKey"
"ℹ️[App Start][GlobalRouter] proceed for session nil"
"ℹ️[Js][Core] JsContext took 0.288 ms to start"
"ℹ️[App Start][GlobalRouter] Sign in with gmailLogin(<FlowCrypt.SignInViewController: 0x15802b400>)"
"ℹ️[App Start][GlobalRouter] proceed for session Optional(Google tom@flowcrypt.com Tom J. H.)"
"ℹ️[App Start][GoogleUserService] Renew session for google user"
"ℹ️[Setup][SetupInitialViewController] Changed to new state searching"
"ℹ️[Setup][SetupInitialViewController] Searching for backups in inbox"
"🏷[GmailService] will begin searching for backups"
"🏷[GmailService] searching done, found 2 backup messages"
"🏷[GmailService] processing backup \'Your FlowCrypt Backup\' with 1 attachments"
"🏷[GmailService] processing backup \'Your FlowCrypt Backup\' with 1 attachments"
"🏷[GmailService] downloading 2 attachments with possible backups in them"
"🏷[GmailService] downloaded 2 attachments that contain 7kB of data"
"ℹ️[Setup][SetupInitialViewController] Finish searching for backups in inbox"
"ℹ️[Setup][SetupInitialViewController] 2 key backups found in inbox"
"ℹ️[Core][KeyMethods] Filtered. decryptedKey = nil"
"ℹ️[PassPhraseStorage] Save to storage 0C9C2E6A4D273C6F"
"ℹ️[App Start][GlobalRouter] proceed for session nil"

Kharchevskyi · 2021-06-24T15:35:58Z

I have added some logs. Can you please try to reproduce this issue once again?

tomholub · 2021-06-24T20:01:10Z

"ℹ️[Keychain][KeyChainService] LocalStorage.secureKeychainPrefix generating new: 6D1zqclvuM3iBVUz"
"ℹ️[Keychain][KeyChainService] generateAndSaveStorageEncryptionKey"
"ℹ️[App Start][GlobalRouter] proceed for session nil"
"ℹ️[AppStart] Initialize application with session nil"
"ℹ️[AppStart] Setup Core"
"ℹ️[Js][Core] JsContext took 0.271 ms to start"
"ℹ️[AppStart] Setup Migration"
"ℹ️[AppStart] Setup Session"
"ℹ️[AppStart] User is not logged in -> signIn"
"ℹ️[App Start][GlobalRouter] Sign in with gmailLogin(<FlowCrypt.SignInViewController: 0x105069400>)"
"ℹ️[App Start][GlobalRouter] proceed for session Optional(Google ... Tom J. H.)"
"ℹ️[AppStart] Initialize application with session Optional(Google ... Tom J. H.)"
"ℹ️[AppStart] Setup Core"
"ℹ️[AppStart] Setup Migration"
"ℹ️[AppStart] Setup Session"
"ℹ️[App Start][GoogleUserService] Renew session for google user"
"ℹ️ UserId = current user id"
"ℹ️[AppStart] User with session Google ... Tom J. H. -> setupFlow"
"ℹ️[Setup][SetupInitialViewController] Changed to new state searching"
"ℹ️[Setup][SetupInitialViewController] Searching for backups in inbox"
"🏷[GmailService] will begin searching for backups"
"🏷[GmailService] searching done, found 2 backup messages"
"🏷[GmailService] processing backup 'Your FlowCrypt Backup' with 1 attachments"
"🏷[GmailService] processing backup 'Your FlowCrypt Backup' with 1 attachments"
"🏷[GmailService] downloading 2 attachments with possible backups in them"
"🏷[GmailService] downloaded 2 attachments that contain 7kB of data"
"ℹ️[Setup][SetupInitialViewController] Finish searching for backups in inbox"
"ℹ️[Setup][SetupInitialViewController] 2 key backups found in inbox"
"ℹ️[Setup][SetupBackupsViewController] Start recoverAccount with 2"
"ℹ️[Core][KeyMethods] Filtered. decryptedKey = nil for key 84AE4C8A1923A36A"
"ℹ️[Setup][SetupBackupsViewController] matchingKeyBackups = 1"
"ℹ️[PassPhraseService] Save to storage 0C9C2E6A4D273C6F"
"ℹ️[App Start][GlobalRouter] proceed for session nil"
"ℹ️[AppStart] Initialize application with session nil"
"ℹ️[AppStart] Setup Core"
"ℹ️[AppStart] Setup Migration"
"ℹ️[AppStart] Setup Session"
"ℹ️[AppStart] User is not logged in -> signIn"

Kharchevskyi · 2021-06-24T20:27:18Z

thanks for reporting, will investigate this

tomholub · 2021-06-29T13:27:44Z

@Kharchevskyi I'll take a look. Please edit your comment per the above bot. Eg you can say cryptup.tester account.

tomholub · 2021-07-01T18:44:21Z

That's not what I meant. The comment was asking you to redact email address from your earlier comment here #355 (comment)

tomholub · 2021-07-01T18:44:58Z

This is a public repo, we don't want email addresses mentioned publicly here.

tomholub · 2021-07-01T18:52:24Z

my own account:

"ℹ️[UserAccountService] Clean up sessions"
"ℹ️[Keychain][KeyChainService] LocalStorage.secureKeychainPrefix generating new: ueAHpE1Bu1opbPk9"
"ℹ️[Keychain][KeyChainService] generateAndSaveStorageEncryptionKey"
"ℹ️[App Start][GlobalRouter] proceed for session nil"
"ℹ️[AppStart] Initialize application with session nil"
"ℹ️[AppStart] Setup Core"
"ℹ️[Js][Core] JsContext took 0.265 ms to start"
"ℹ️[AppStart] Setup Migration"
"ℹ️[AppStart] Setup Session"
"ℹ️[AppStart] User is not logged in -> signIn"
"ℹ️[App Start][GlobalRouter] Sign in with gmailLogin(<FlowCrypt.SignInViewController: 0x15c065200>)"
"ℹ️[App Start][GlobalRouter] proceed for session Optional(Google ... Tom J. H.)"
"ℹ️[AppStart] Initialize application with session Optional(Google ... Tom J. H.)"
"ℹ️[AppStart] Setup Core"
"ℹ️[AppStart] Setup Migration"
"ℹ️[AppStart] Setup Session"
"ℹ️[App Start][GoogleUserService] Renew session for google user"
"ℹ️ UserId = current user id"
"ℹ️[AppStart] User with session Google ... Tom J. H. -> setupFlow"
"ℹ️[Setup][SetupInitialViewController] Changed to new state searching"
"ℹ️[Setup][SetupInitialViewController] Searching for backups in inbox"
"🏷[GmailService] will begin searching for backups"
"🏷[GmailService] searching done, found 2 backup messages"
"🏷[GmailService] processing backup \'Your FlowCrypt Backup\' with 1 attachments"
"🏷[GmailService] processing backup \'Your FlowCrypt Backup\' with 1 attachments"
"🏷[GmailService] downloading 2 attachments with possible backups in them"
"🏷[GmailService] downloaded 2 attachments that contain 7kB of data"
"ℹ️[Setup][SetupInitialViewController] Finish searching for backups in inbox"
"ℹ️[Setup][SetupInitialViewController] 2 key backups found in inbox"
"ℹ️[Setup][SetupBackupsViewController] Start recoverAccount with 2"
"ℹ️[Core][KeyMethods] Filtered. decryptedKey = nil for key 84AE4C8A1923A36A"
"ℹ️[Setup][SetupBackupsViewController] matchingKeyBackups = 1"
"ℹ️[PassPhraseService] Save to storage 0C9C2E6A4D273C6F"
"ℹ️[KeyDataStorage] Add keys for [\"Human at FlowCrypt <...>\"]"
"ℹ️[UserAccountService] Clean up sessions"
"ℹ️[UserAccountService] User session to clean up ..."
"ℹ️[UserAccountService] Log out user with ..."
"ℹ️[App Start][GlobalRouter] proceed for session nil"
"ℹ️[AppStart] Initialize application with session nil"
"ℹ️[AppStart] Setup Core"
"ℹ️[AppStart] Setup Migration"
"ℹ️[AppStart] Setup Session"
"ℹ️[AppStart] User is not logged in -> signIn"

tomholub · 2021-07-01T18:54:23Z

compatibility account - worked normally:


"ℹ️[App Start][GlobalRouter] Sign in with gmailLogin(<FlowCrypt.SignInViewController: 0x15c0dfc00>)"
"ℹ️[App Start][GlobalRouter] proceed for session Optional(Google ... FlowCrypt Compatibility)"
"ℹ️[AppStart] Initialize application with session Optional(Google ... FlowCrypt Compatibility)"
"ℹ️[AppStart] Setup Core"
"ℹ️[AppStart] Setup Migration"
"ℹ️[AppStart] Setup Session"
"ℹ️[App Start][GoogleUserService] Renew session for google user"
"ℹ️ UserId = current user id"
"ℹ️[AppStart] User with session Google ... FlowCrypt Compatibility -> setupFlow"
"ℹ️[Setup][SetupInitialViewController] Changed to new state searching"
"ℹ️[Setup][SetupInitialViewController] Searching for backups in inbox"
"🏷[GmailService] will begin searching for backups"
"🏷[GmailService] searching done, found 3 backup messages"
"🏷[GmailService] processing backup \'Your FlowCrypt Backup\' with 1 attachments"
"🏷[GmailService] processing backup \'Your FlowCrypt Backup\' with 1 attachments"
"🏷[GmailService] processing backup \'Your FlowCrypt Backup\' with 1 attachments"
"🏷[GmailService] downloading 3 attachments with possible backups in them"
"🏷[GmailService] downloaded 3 attachments that contain 20kB of data"
"ℹ️[Setup][SetupInitialViewController] Finish searching for backups in inbox"
"ℹ️[Setup][SetupInitialViewController] 3 key backups found in inbox"
"ℹ️[Setup][SetupBackupsViewController] Start recoverAccount with 3"
"ℹ️[Core][KeyMethods] Filtered. decryptedKey = nil for key 7FDE685548AEA788"
"ℹ️[Core][KeyMethods] Filtered. decryptedKey = nil for key ADAC279C95093207"
"ℹ️[Setup][SetupBackupsViewController] matchingKeyBackups = 1"
"ℹ️[PassPhraseService] Save to storage ADAC279C95093207"
"ℹ️[KeyDataStorage] Add keys for [\"FlowCrypt Compatibility <...>\"]"
"ℹ️[UserAccountService] Clean up sessions"
"ℹ️[App Start][GlobalRouter] proceed for session nil"
"ℹ️[AppStart] Initialize application with session nil"
"ℹ️[AppStart] Setup Core"
"ℹ️[AppStart] Setup Migration"
"ℹ️[AppStart] Setup Session"
"ℹ️[App Start][GoogleUserService] Renew session for google user"
"ℹ️[AppStart] Setup finished -> mainFlow"
"ℹ️[GmailService] Skip category label with CATEGORY_FORUMS"
"ℹ️[GmailService] Skip category label with CATEGORY_UPDATES"
"ℹ️[GmailService] Skip category label with CATEGORY_PERSONAL"
"ℹ️[GmailService] Skip category label with CATEGORY_PROMOTIONS"
"ℹ️[GmailService] Skip category label with CATEGORY_SOCIAL"

Kharchevskyi · 2021-07-01T19:24:34Z

my own account:

probably issue is here "ℹ️[UserAccountService] Clean up sessions". Thank's for the logs

FlowCrypt.xcodeproj/xcshareddata/xcschemes/FlowCrypt.xcscheme

tomholub · 2021-07-05T11:43:08Z

Sent to your email

tomholub · 2021-07-05T16:26:48Z

Copying from email:

You may completely ignore the email addresses listed inside of the PGP keys. They don't have to match the account they are set up for.

tomholub

Code looks ok - I'm not at my mac until wednesday to test it. I'll merge it and create another issue in case it misbehaves.

tomholub · 2021-07-05T19:25:11Z

However tests are failing - can you take a look?

tomholub · 2021-07-07T21:36:48Z

This fixed the issue, thank you

tomholub · 2021-07-07T21:50:24Z

@Kharchevskyi the CI still cannot build it, I've updated from master

tomholub · 2021-07-08T08:51:53Z

also #381 was observed on this PR

Kharchevskyi · 2021-07-08T09:35:43Z

Last commits should also fix #381

fix wasteful backup search in inbox on Gmail API

d346006

Kharchevskyi requested a review from tomholub June 17, 2021 20:31

tomholub approved these changes Jun 21, 2021

View reviewed changes

tomholub requested changes Jun 21, 2021

View reviewed changes

tomholub marked this pull request as draft June 21, 2021 14:33

Kharchevskyi added 2 commits June 22, 2021 17:17

Merge origin

295c13e

Use Core to generate backup search query

e7e57dd

Kharchevskyi marked this pull request as ready for review June 22, 2021 16:31

tomholub reviewed Jun 22, 2021

View reviewed changes

...rypt/Functionality/Mail Provider/SearchMessage Provider/GmailSearchExpressionGenerator.swift Outdated Show resolved Hide resolved

FlowCrypt/Core/Core.swift Show resolved Hide resolved

Kharchevskyi added 4 commits June 23, 2021 22:54

Update Search backup functionality

ef8a552

Start working on GmailService tests

32f49f4

Merge branch 'master' of github.com:FlowCrypt/flowcrypt-ios into feat…

37db91a

…ure/issue-309-backups

Add testSearchBackupsWhenErrorInQuery

1f4d82b

tomholub previously approved these changes Jun 23, 2021

View reviewed changes

tomholub marked this pull request as draft June 23, 2021 21:34

Add logs to AppStart

1771be2

Kharchevskyi dismissed tomholub’s stale review via 1771be2 June 24, 2021 15:31

Merge origin

2ddc4b0

Fix KeyDetails longid

f7971ca

Kharchevskyi marked this pull request as ready for review June 24, 2021 17:03

Kharchevskyi marked this pull request as draft June 24, 2021 17:08

Change back AppStartup to be a struct

1a5af10

tomholub mentioned this pull request Jul 1, 2021

back icon on recovery screen doesn't work #374

Closed

Add debug logs

98b2ca5

tomholub reviewed Jul 5, 2021

View reviewed changes

FlowCrypt.xcodeproj/xcshareddata/xcschemes/FlowCrypt.xcscheme Outdated Show resolved Hide resolved

Kharchevskyi added 3 commits July 5, 2021 20:42

Add UserObject to KeyInfo object

58724ab

Remove buildForTesting in FlowCrypt.scheme

08c9b66

Update doesAnyKeyExist

61271d8

Kharchevskyi marked this pull request as ready for review July 5, 2021 17:48

tomholub reviewed Jul 5, 2021

View reviewed changes

Merge branch 'master' into feature/issue-309-backups

25ea223

tomholub enabled auto-merge (squash) July 7, 2021 21:37

Kharchevskyi added 2 commits July 8, 2021 12:19

Rearange order of deleting objects during logout

2976d6a

Fix mock for KeyInfo

c4f8208

Kharchevskyi mentioned this pull request Jul 8, 2021

update setup screen labels #325

Closed

tomholub approved these changes Jul 8, 2021

View reviewed changes

tomholub merged commit 1b46e17 into master Jul 8, 2021

tomholub deleted the feature/issue-309-backups branch July 8, 2021 10:39

Kharchevskyi mentioned this pull request Jul 8, 2021

log out crashes the app #381

Closed

Conversation

Kharchevskyi commented Jun 17, 2021 • edited by tomholub Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

close #309

To be filled by reviewers

Uh oh!

tomholub left a comment

Choose a reason for hiding this comment

Uh oh!

tomholub left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tomholub commented Jun 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tomholub commented Jun 21, 2021

Uh oh!

tomholub commented Jun 21, 2021

Uh oh!

Kharchevskyi commented Jun 22, 2021

Uh oh!

tomholub left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tomholub left a comment

Choose a reason for hiding this comment

Uh oh!

tomholub commented Jun 23, 2021

Uh oh!

Kharchevskyi commented Jun 24, 2021

Uh oh!

tomholub commented Jun 24, 2021

Uh oh!

Kharchevskyi commented Jun 24, 2021

Uh oh!

tomholub commented Jun 29, 2021

Uh oh!

tomholub commented Jul 1, 2021

Uh oh!

tomholub commented Jul 1, 2021

Uh oh!

tomholub commented Jul 1, 2021

Uh oh!

tomholub commented Jul 1, 2021

Uh oh!

Kharchevskyi commented Jul 1, 2021

Uh oh!

Uh oh!

tomholub commented Jul 5, 2021

Uh oh!

tomholub commented Jul 5, 2021

Uh oh!

tomholub left a comment

Choose a reason for hiding this comment

Uh oh!

tomholub commented Jul 5, 2021

Uh oh!

tomholub commented Jul 7, 2021

Uh oh!

tomholub commented Jul 7, 2021

Uh oh!

tomholub commented Jul 8, 2021

Uh oh!

Kharchevskyi commented Jul 8, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Kharchevskyi commented Jun 17, 2021 •

edited by tomholub

Loading

tomholub left a comment •

edited

Loading

tomholub commented Jun 21, 2021 •

edited

Loading