Skip to content

[SPARK-22258][SQL] Writing empty dataset fails with ORC format #19477

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dongjoon-hyun wants to merge 1 commit into from
Closed

[SPARK-22258][SQL] Writing empty dataset fails with ORC format #19477

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion ...re/src/main/scala/org/apache/spark/sql/execution/datasources/BasicWriteStatsTracker.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@

package org.apache.spark.sql.execution.datasources

import java.io.IOException

import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.Path

Expand Down Expand Up @@ -57,7 +59,14 @@ class BasicWriteTaskStatsTracker(hadoopConf: Configuration)
private def getFileSize(filePath: String): Long = {
val path = new Path(filePath)
val fs = path.getFileSystem(hadoopConf)
fs.getFileStatus(path).getLen()
// getFileStatus may throw IOException like FileNotFoundException. A file format like ORC
// may not create a file at all in case of writing empty dataset. Writing task should not fail
// due to this StatsTracker.
try {
fs.getFileStatus(path).getLen()
} catch {
case _: IOException => 0
}
}


Expand Down
8 changes: 8 additions & 0 deletions sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2050,4 +2050,12 @@ class SQLQuerySuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
}
}
}

Seq("orc", "parquet", "csv", "json", "text").foreach { format =>
Copy link
Member

@viirya viirya Oct 12, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems this test case is worth merging into. cc @steveloughran Shall we include this test into #18979?

Copy link
Member Author

@dongjoon-hyun dongjoon-hyun Oct 12, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. Please, @steveloughran . :)

test(s"Writing empty datasets should not fail - $format") {
withTempDir { dir =>
Seq("str").toDS.limit(0).write.format(format).save(dir.getCanonicalPath + "/tmp")
}
}
}
}