Skip to content
This repository was archived by the owner on Jun 14, 2024. It is now read-only.

Conversation

@apoorvedave1
Copy link
Contributor

@apoorvedave1 apoorvedave1 commented Oct 9, 2020

Fixes #182

Note For Reviewers: This is a low priority non-blocker issue. It's very rare that users will replace data files with exact data files with same name and modified content. But still, for correctness, it can be picked as time allows

What is the context for this pull request?

What changes were proposed in this pull request?

Please refer #182

Does this PR introduce any user-facing change?

How was this patch tested?

apoorvedave1 and others added 30 commits October 7, 2020 20:39
…into bugfixes

# Conflicts:
#	src/test/scala/com/microsoft/hyperspace/TestUtils.scala
Co-authored-by: Terry Kim <yuminkim@gmail.com>
Co-authored-by: Terry Kim <yuminkim@gmail.com>
# Conflicts:
#	src/main/scala/com/microsoft/hyperspace/actions/RefreshDeleteAction.scala
#	src/test/scala/com/microsoft/hyperspace/index/RefreshIndexTests.scala
@apoorvedave1 apoorvedave1 self-assigned this Oct 9, 2020
@apoorvedave1 apoorvedave1 changed the title [WIP] Bug 182 Appended files should look at modified time and file size for comparison Oct 9, 2020
val relation = previousIndexLogEntry.relations.head
val originalFiles = relation.data.properties.content.fileInfos

val currentFiles = df.queryExecution.optimizedPlan
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you extract this as protected lazy val and use it for both appendedFiles and deletedFiles?

@sezruby
Copy link
Collaborator

sezruby commented Oct 15, 2020

Closed by #207

@sezruby sezruby closed this Oct 15, 2020
@apoorvedave1 apoorvedave1 deleted the bug_182 branch March 16, 2021 22:40
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve RefreshIncremental logic to take last modified time of files and file size into account.

2 participants