[SPARK-20971][SS] purge metadata log in FileStreamSource by CodingCat · Pull Request #18410 · apache/spark

CodingCat · 2017-06-23T22:31:38Z

What changes were proposed in this pull request?

Currently, there is no cleanup mechanism for FileStreamSource's metadata log so that the data is growing infinitely

This PR purges the log which is out of the retaining windowing

How was this patch tested?

existing tests

SparkQA · 2017-06-24T00:18:54Z

Test build #78549 has finished for PR 18410 at commit b37ec11.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

CodingCat · 2017-06-24T03:15:26Z

Jenkins, test it please

CodingCat · 2017-06-24T03:43:48Z

Jenkins, retest this please

SparkQA · 2017-06-24T05:19:38Z

Test build #78558 has finished for PR 18410 at commit b37ec11.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

CodingCat · 2017-06-24T06:24:53Z

retest this please

SparkQA · 2017-06-24T06:27:36Z

Test build #78560 has started for PR 18410 at commit b37ec11.

CodingCat · 2017-06-24T21:59:03Z

retest this please

SparkQA · 2017-06-24T23:53:11Z

Test build #78570 has finished for PR 18410 at commit b37ec11.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

CodingCat · 2017-06-24T23:58:26Z

@zsxwing would you mind taking a look at this PR...what does this pip packaging tests mean? it's a flaky test?

HyukjinKwon · 2017-06-25T15:11:56Z

(I believe the discussion here might be helpful for pip packaging failure - https://github.com/apache/spark/pull/15821#issuecomment-310894657)

CodingCat · 2017-06-26T14:44:41Z

@HyukjinKwon thanks for the pointer, is it fixed now?

CodingCat · 2017-06-27T15:07:26Z

retest this please

SparkQA · 2017-06-27T17:02:17Z

Test build #78704 has finished for PR 18410 at commit b37ec11.

This patch fails PySpark pip packaging tests.
This patch merges cleanly.
This patch adds no public classes.

CodingCat · 2017-06-29T16:14:29Z

retest this please

SparkQA · 2017-06-29T18:28:41Z

Test build #78919 has finished for PR 18410 at commit b37ec11.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-10-28T05:36:30Z

cc @zsxwing Could you check whether we should continue or close this PR?

HeartSaVioR · 2018-07-30T07:46:52Z

~~Looks like this PR is not needed, since CompactibleFileStreamLog also takes care of metadata log.~~

https://github.com/apache/spark/commits/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/CompactibleFileStreamLog.scala

Looks like deleting log for outdated batch is happening when "spark.sql.streaming.fileSource.log.deletion" is true and also default value is true, so I'm curious whether this patch is needed to remove outdated logs explicitly, or it is OK to keep it as it is.

SparkQA · 2018-10-22T07:05:03Z

Test build #97736 has finished for PR 18410 at commit b37ec11.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-10-22T07:05:05Z

Test build #97707 has finished for PR 18410 at commit b37ec11.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-10-22T14:29:49Z

Test build #97806 has finished for PR 18410 at commit b37ec11.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

mikedias · 2019-02-20T06:35:02Z

-    // No-op for now; FileStreamSource currently garbage-collects files based on timestamp
-    // and the value of the maxFileAge parameter.
+    if (currentLogOffset > minBatchesToRetain) {
+      metadataLog.purge(currentLogOffset - minBatchesToRetain)


What would be the behavior of this when spark.sql.streaming.fileSource.log.deletion=false? Looks like HDFSMetadataLog.purge will always delete the files.

HDFSMetadataLog is not aware of such configuration.

Btw, I've put my observation on CompactibleFileStreamLog.purge() in comment on SPARK-20971.

https://issues.apache.org/jira/browse/SPARK-20971?focusedCommentId=16772632&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16772632

Let me quote it here:

Btw, calling purge breaks CompactibleFileStreamLog since CompactibleFileStreamLog expects non-compacted batches to be exist, but purge just removes all of metadata files matching criteria. The safest way seems to be just disallowing purge for CompactibleFileStreamLog, otherwise we have to concern about the intention of calling purge, like I was curious of rationalization of this issue like above.

So I've got a feeling that this may bring unexpected behavior and should be avoided.

HeartSaVioR · 2019-02-20T13:22:11Z

I played a bit with my idea (filter out entities which were in committed batch when compacting), and realized it cannot solve the issue which filtered out files being re-read (not sure I'm just missing here).

HeartSaVioR@00534b8

Once the files are filtered out in metadata they could be included in source of new batch (even if SeenFilesMap could help a bit, it's not persisted to storage so the issue remains same when query is rerun).

I think the best way to do this safely would be incorporating this to #22952 - when files are successfully archived or deleted, we're safe to filter out them in metadata log as well. I'll address it to #22952.

HeartSaVioR · 2019-02-21T00:02:21Z

FYI: I've addressed removing obsolete file entries from compacted metadata logs. Please refer #22952 as well as 5dcece0

github-actions · 2020-01-16T00:08:30Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

CodingCat added 3 commits June 23, 2017 15:10

improve the doc for "spark.memory.offHeap.size"

9c21aca

fix

c82e382

purge log in metadatalog of filesourcestream

b37ec11

CodingCat changed the title ~~[SS][SPARK-20971] purge metadata log in FileStreamSource~~ [SPARK-20971][SS] purge metadata log in FileStreamSource Jun 23, 2017

CodingCat mentioned this pull request Aug 24, 2017

[build] explicitly install JDK8 apache/mxnet#7574

Merged

mikedias reviewed Feb 20, 2019

View reviewed changes

dongjoon-hyun added the STRUCTURED STREAMING label Jun 14, 2019

github-actions Bot added the Stale label Jan 16, 2020

github-actions Bot closed this Jan 17, 2020

Conversation

CodingCat commented Jun 23, 2017

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Jun 24, 2017

Uh oh!

CodingCat commented Jun 24, 2017

Uh oh!

CodingCat commented Jun 24, 2017

Uh oh!

SparkQA commented Jun 24, 2017

Uh oh!

CodingCat commented Jun 24, 2017

Uh oh!

SparkQA commented Jun 24, 2017

Uh oh!

CodingCat commented Jun 24, 2017

Uh oh!

SparkQA commented Jun 24, 2017

Uh oh!

CodingCat commented Jun 24, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon commented Jun 25, 2017

Uh oh!

CodingCat commented Jun 26, 2017

Uh oh!

CodingCat commented Jun 27, 2017

Uh oh!

SparkQA commented Jun 27, 2017

Uh oh!

CodingCat commented Jun 29, 2017

Uh oh!

SparkQA commented Jun 29, 2017

Uh oh!

gatorsmile commented Oct 28, 2017

Uh oh!

HeartSaVioR commented Jul 30, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Oct 22, 2018

Uh oh!

SparkQA commented Oct 22, 2018

Uh oh!

SparkQA commented Oct 22, 2018

Uh oh!

mikedias Feb 20, 2019

Choose a reason for hiding this comment

Uh oh!

HeartSaVioR Feb 20, 2019

Choose a reason for hiding this comment

Uh oh!

HeartSaVioR commented Feb 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HeartSaVioR commented Feb 21, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jan 16, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

CodingCat commented Jun 24, 2017 •

edited

Loading

HeartSaVioR commented Jul 30, 2018 •

edited

Loading

HeartSaVioR commented Feb 20, 2019 •

edited

Loading

HeartSaVioR commented Feb 21, 2019 •

edited

Loading