Skip to content
This repository was archived by the owner on Jun 14, 2024. It is now read-only.

Refresh delete and append should update "deleted" and "appended" files in metadata #191

Merged
imback82 merged 34 commits into from
Oct 10, 2020
Merged

Refresh delete and append should update "deleted" and "appended" files in metadata #191

imback82 merged 34 commits into from
Oct 10, 2020

Conversation

@apoorvedave1
Copy link
Contributor

@apoorvedave1 apoorvedave1 commented Oct 8, 2020 •
edited
Loading

What is the context for this pull request?

Refresh Append and Refresh Delete actions should update "appendedFiles" and "deletedFiles" in index metadata. This should happen as below:
RefreshDeleteAction

  1. it should now pick all the deletedFiles from the index metadata as well as look for additional recently deleted files.
  2. Once delete operation completes, clear deletedFiles from index metadata.
  3. Update appendedFiles in metadata by adding recently appended files to the appendedFiles list.

RefreshAppendAction

  1. It should now pick all the appendedFiles from index metadata as well as look for additional recently appended files.
  2. Once append operation completes, clear appendedFiles from index metadata.
  3. Update deletedFiles in metadata by adding recently deleted files to the deletedFiles list.

What changes were proposed in this pull request?

bugfixes for #179, #183

Does this PR introduce any user-facing change?

How was this patch tested?

added unit tests

@apoorvedave1 apoorvedave1 marked this pull request as draft October 8, 2020 05:00
apoorvedave1 and others added 24 commits October 7, 2020 22:03
@apoorvedave1
initial commit part 2
ad2e9df
@apoorvedave1
review comments
4d27fe2
@apoorvedave1
Merge branch 'errorhandling' of github.com:apoorvedave1/hyperspace-1 …
58aea9d 
…into errorhandling
@apoorvedave1
Merge branch 'errorhandling' of github.com:apoorvedave1/hyperspace-1 …
1c4ae1c 
…into bugfixes
@apoorvedave1
doc
87ff7b7
@apoorvedave1
Merge branch 'errorhandling' of github.com:apoorvedave1/hyperspace-1 …
300703b 
…into bugfixes
@apoorvedave1
build failure fix
e31f171
@apoorvedave1
add unit test for "appended"files update
47c6730
@apoorvedave1
test fix
7b4a6a8
@apoorvedave1
review comments
f8766c0
@apoorvedave1
Merge branch 'errorhandling' of github.com:apoorvedave1/hyperspace-1 …
5a171f9 
…into bugfixes

# Conflicts:
#	src/test/scala/com/microsoft/hyperspace/TestUtils.scala
@apoorvedave1
set logger at session initialization in test cases
58f1339
@apoorvedave1
Merge branch 'errorhandling' of github.com:apoorvedave1/hyperspace-1 …
4dc0ab6 
…into bugfixes
move mock logger setting to beginning of all SparkInvolvedSuite tests
2c55d86
@apoorvedave1 @imback82
refactoring
138e96a 
Co-authored-by: Terry Kim <yuminkim@gmail.com>
@apoorvedave1 @imback82
explicit assert in test
767513f 
Co-authored-by: Terry Kim <yuminkim@gmail.com>
review comets
7338de5
Merge branch 'errorhandling' of github.com:apoorvedave1/hyperspace-1 …
7f94a1d 
…into errorhandling
test bug fixes
29ca9b3
@apoorvedave1
Merge branch 'errorhandling' of github.com:apoorvedave1/hyperspace-1 …
5db1125 
…into bugfixes
whitespace fix
9d26ddc
Merge branch 'errorhandling' of github.com:apoorvedave1/hyperspace-1 …
c7816b6 
…into bugfixes
add refresh-append test for updating delted files
b43b29e
Merge remote-tracking branch 'upstream/master' into bugfixes
7ebda9a 
# Conflicts:
#	src/main/scala/com/microsoft/hyperspace/actions/RefreshDeleteAction.scala
#	src/test/scala/com/microsoft/hyperspace/index/RefreshIndexTests.scala
@apoorvedave1 apoorvedave1 changed the title [WIP] Bugfixes: refresh delete and append should update "deleted" and "appended" files in metadata Bugfixes: refresh delete and append should update "deleted" and "appended" files in metadata Oct 9, 2020
@apoorvedave1 apoorvedave1 requested a review from sezruby October 9, 2020 02:56
@apoorvedave1 apoorvedave1 self-assigned this Oct 9, 2020
@apoorvedave1 apoorvedave1 added the bug Something isn't working label Oct 9, 2020
@apoorvedave1 apoorvedave1 added this to the 0.4.0 milestone Oct 9, 2020
@apoorvedave1 apoorvedave1 marked this pull request as ready for review October 9, 2020 02:57
sezruby
sezruby reviewed Oct 9, 2020
View reviewed changes
@apoorvedave1 apoorvedave1 requested a review from sezruby October 9, 2020 17:06
@apoorvedave1 apoorvedave1 mentioned this pull request Oct 9, 2020
Merged
@apoorvedave1 apoorvedave1 mentioned this pull request Oct 9, 2020
Closed
@imback82 imback82 changed the title Bugfixes: refresh delete and append should update "deleted" and "appended" files in metadata Refresh delete and append should update "deleted" and "appended" files in metadata Oct 9, 2020
imback82
imback82 reviewed Oct 9, 2020
View reviewed changes
Copy link
Contributor

@imback82 imback82 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

generally, looking ok to me

This was referenced Oct 9, 2020
Merged
Closed
@apoorvedave1 apoorvedave1 requested a review from imback82 October 10, 2020 02:34
imback82
imback82 approved these changes Oct 10, 2020
View reviewed changes
Copy link
Contributor

@imback82 imback82 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @apoorvedave1!

I will wait on merging this until @sezruby approves since she had few pending comments.

@sezruby
Copy link
Collaborator

sezruby commented Oct 10, 2020

I commented in other PRs - let's resolve the comments with them.

Thanks @apoorvedave1 for this work!

@imback82 imback82 merged commit db3e5fb into microsoft:master Oct 10, 2020
@apoorvedave1 apoorvedave1 deleted the bugfixes branch October 10, 2020 04:32
@apoorvedave1 apoorvedave1 mentioned this pull request Oct 10, 2020
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@pirz pirz Awaiting requested review from pirz

@rapoth rapoth Awaiting requested review from rapoth

@sezruby sezruby Awaiting requested review from sezruby

1 more reviewer

@imback82 imback82 imback82 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@apoorvedave1 apoorvedave1

Labels

bug Something isn't working

Projects

None yet

Milestone

September 2020

Development

Successfully merging this pull request may close these issues.

RefreshDelete should update appended files list in metadata

3 participants

@apoorvedave1 @sezruby @imback82