Skip to content

SPARK-8064, build against Hive 1.2.1 #7191

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
steveloughran wants to merge 68 commits into from
Closed

SPARK-8064, build against Hive 1.2.1 #7191

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
4890b9d
SPARK-8064, build against Hive 1.2.1
steveloughran Jul 2, 2015
f61a675
SPARK-8064 thrift server switched to Hive 1.2.1, though it doesn't co…
steveloughran Jul 15, 2015
6c310b4
SPARK-8064 subclass Hive ServerOptionsProcessor to make it public again
steveloughran Jul 16, 2015
f7aa9cb
SPARK-8064 everything compiles with some commenting and moving of cla…
steveloughran Jul 16, 2015
cb4f142
SPARK-8054 cut pentaho dependency from calcite
steveloughran Jul 17, 2015
00e50d6
SPARK-8064 stop excluding hive shims from dependency (commented out ,…
steveloughran Jul 22, 2015
27e8370
SPARK-8064 fix some style & IDE warnings
steveloughran Jul 22, 2015
c80979d
SPARK-8064: SparkSQLCLIDriver drops remote mode support. CLISuite Tes…
steveloughran Jul 22, 2015
3ebc279
SPARK-8064 move strings used to check for http/bin thrift services up…
steveloughran Jul 22, 2015
75733fc
SPARK-8064 change thrift binary startup message to "Starting ThriftBi…
steveloughran Jul 22, 2015
1d59100
SPARK-8064 (unsuccessful) attempt to get rid of pentaho as a transiti…
steveloughran Jul 23, 2015
2531099
SPARK-8064 successful attempt to get rid of pentaho as a transitive d…
steveloughran Jul 23, 2015
463a670
SPARK-8064 run-tests.py adds a hadoop-2.6 profile, and changes info m…
steveloughran Jul 23, 2015
dec12cb
SPARK-8064: when a CLI suite test fails include the full output text …
steveloughran Jul 23, 2015
8cb09c4
SPARK-8064: test resilience/assertion improvements. Independent of th…
steveloughran Jul 23, 2015
1126e5a
SPARK-8064: name of unrecognized file format wasn't appearing in erro…
steveloughran Jul 23, 2015
d0360f6
SPARK-8064: delicate merge in of the branch vanzin/hive-1.1
steveloughran Jul 24, 2015
3d64523
SPARK-8064 improve diagns on uknown token; fix scalastyle failure
steveloughran Jul 24, 2015
26eef1c
SPARK-8064: HIVE-9039 renamed TOK_UNION => TOK_UNIONALL while adding …
steveloughran Jul 24, 2015
d14d5ea
SPARK-8064: DATE is now a predicate; you can't use it as a field in s…
steveloughran Jul 24, 2015
13abaf1
SPARK-8064 Hive compatibilty tests sin sync with explain/show output …
steveloughran Jul 24, 2015
0b0f738
SPARK-8064: thrift server startup to fail fast on any exception in th…
steveloughran Jul 24, 2015
c829b8f
SPARK-8064: reinstate yarn-rm-server dependencies to hive-exec to ens…
steveloughran Jul 24, 2015
bf3a249
SPARK-8064: more resubmit than fix; tighten startup timeout to 60s. S…
steveloughran Jul 27, 2015
1ab9bc4
SPARK-8064 TestHive to use sered2.thrift.test.Complex
steveloughran Jul 27, 2015
2bc29a4
SPARK-8064 ParquetSuites to escape `date` field name
steveloughran Jul 27, 2015
41d6479
SPARK-8064 wrap tests with withTable() calls to avoid table-exists ex…
steveloughran Jul 27, 2015
bca55e5
SPARK-8064 missed one of the `date` escapes
steveloughran Jul 27, 2015
3ed872f
SPARK-8064 rename field double to dbl
steveloughran Jul 27, 2015
335357f
SPARK-8064 fail fast on thrive process spawning tests on exit codes a…
steveloughran Jul 27, 2015
97194dc
SPARK-8064 add extra logging to the YarnClusterSuite classpath test. …
steveloughran Jul 28, 2015
642b63a
SPARK-8064 reinstate something cut briefly during rebasing
steveloughran Jul 28, 2015
5a9ce6b
SPARK-8064 add explicit reason for kv split failure, rather than arra…
steveloughran Jul 28, 2015
8b1ef38
SPARK-8064: on failures executing spark-submit in HiveSparkSubmitSuit…
steveloughran Jul 28, 2015
f2bff01
SPARK-8064 better takeup of asynchronously caught error text
steveloughran Jul 29, 2015
aa43dc6
SPARK-8064 more robust teardown on JavaMetastoreDatasourcesSuite
steveloughran Jul 29, 2015
e6121e5
SPARK-8064 address review comments
steveloughran Jul 29, 2015
6684c60
SPARK-8064 ignore RTE raised in blocking process.exitValue() call
steveloughran Jul 29, 2015
051cc21
SPARK-8064 switch to an unshaded version of hive-exec-core, which mus…
steveloughran Jul 29, 2015
d3c1e4a
SPARK-8064 purge UnionType
steveloughran Jul 29, 2015
dc73ece
SPARK-8064 revert to master's determinstic pushdown strategy
steveloughran Jul 29, 2015
fd3aa5d
SPARK-8064 version of hive to d/l from ivy is 1.2.1
steveloughran Jul 29, 2015
fce73b6
SPARK-8064 poms rely implicitly on the version of kryo chill provides
steveloughran Jul 30, 2015
0b12d5f
HIVE-8064 use subset of hive complex type whose types deserialize
steveloughran Jul 30, 2015
54d9b06
SPARK-8064 fix compilation regression surfacing from rebase
steveloughran Jul 30, 2015
23eca7e
HIVE-8064 handle raw and escaped property tokens
steveloughran Jul 30, 2015
d029b92
SPARK-8064 rely on unescaping to have already taken place, so go stra…
steveloughran Jul 30, 2015
17b0341
SPARK-8064 IDE-hinted cleanups of Complex.java to reduce compiler war…
steveloughran Jul 30, 2015
314eb3c
SPARK-8064 deprecation warning noise in one of the tests
steveloughran Jul 30, 2015
4c8be8d
WIP: Partial fix for Thrift server and CLI tests
liancheng Jul 29, 2015
c188048
SPARK-8064 manage the Hive depencencies to that
steveloughran Jul 30, 2015
fa5ae7b
HIVE-8064 fix up hive-thriftserver dependencies and cut back on evict…
steveloughran Jul 31, 2015
312c0d4
SPARK-8064 maven/ivy dependency purge; calcite declaration needed
steveloughran Jul 31, 2015
832c164
SPARK-8064 try to supress compiler warnings on Complex.java pasted-th…
steveloughran Jul 31, 2015
7404f34
Add spark-hive staging repo
pwendell Jul 31, 2015
a775a75
SPARK-8064 cherry-pick-incomplete
steveloughran Jul 31, 2015
da310dc
Fixes for Hive tests.
marmbrus Jul 31, 2015
6901fa9
SPARK-8064 explicit protobuf import
steveloughran Jul 31, 2015
cc44020
SPARK-8064 remove hadoop.version from runtest.py, as profile will fix…
steveloughran Jul 31, 2015
2c74697
SPARK-8064 switch to the protobuf shaded hive-exec jar with tests to …
steveloughran Aug 1, 2015
376c003
SPARK-8064 purge duplicate protobuf declaration
steveloughran Aug 1, 2015
7a6c727
SPARK-8064 switch to second staging repo of the spark-hive artifacts.…
steveloughran Aug 1, 2015
fdf759b
SPARK-8064 classpath dependency suite to be in sync with shading in f…
steveloughran Aug 1, 2015
0bbe475
SPARK-8064 scalastyle rejects the standard Hadoop ASF license header...
steveloughran Aug 1, 2015
dcbb391
Adds com.twitter:parquet-hadoop-bundle:1.6.0 for Hive Parquet SerDe
liancheng Aug 2, 2015
6a92bb0
Overrides HiveConf time vars
liancheng Aug 2, 2015
ef4af62
Merge commit '6a92bb09f46a04d6cd8c41bdba3ecb727ebb9030' into stevel/f…
steveloughran Aug 2, 2015
7556d85
Updates .q files and corresponding golden files
liancheng Aug 3, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 0 additions & 20 deletions core/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,30 +46,10 @@
<dependency>
<groupId>com.twitter</groupId>
<artifactId>chill_${scala.binary.version}</artifactId>
<exclusions>
<exclusion>
<groupId>org.ow2.asm</groupId>
<artifactId>asm</artifactId>
</exclusion>
<exclusion>
<groupId>org.ow2.asm</groupId>
<artifactId>asm-commons</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.twitter</groupId>
<artifactId>chill-java</artifactId>
<exclusions>
<exclusion>
<groupId>org.ow2.asm</groupId>
<artifactId>asm</artifactId>
</exclusion>
<exclusion>
<groupId>org.ow2.asm</groupId>
<artifactId>asm-commons</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
Expand Down
7 changes: 4 additions & 3 deletions dev/run-tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -273,6 +273,7 @@ def get_hadoop_profiles(hadoop_version):
"hadoop2.0": ["-Phadoop-1", "-Dhadoop.version=2.0.0-mr1-cdh4.1.1"],
"hadoop2.2": ["-Pyarn", "-Phadoop-2.2"],
"hadoop2.3": ["-Pyarn", "-Phadoop-2.3", "-Dhadoop.version=2.3.0"],
"hadoop2.6": ["-Pyarn", "-Phadoop-2.6"],
}

if hadoop_version in sbt_maven_hadoop_profiles:
Expand All @@ -289,7 +290,7 @@ def build_spark_maven(hadoop_version):
mvn_goals = ["clean", "package", "-DskipTests"]
profiles_and_goals = build_profiles + mvn_goals

print("[info] Building Spark (w/Hive 0.13.1) using Maven with these arguments: ",
print("[info] Building Spark (w/Hive 1.2.1) using Maven with these arguments: ",
" ".join(profiles_and_goals))

exec_maven(profiles_and_goals)
Expand All @@ -305,14 +306,14 @@ def build_spark_sbt(hadoop_version):
"streaming-kinesis-asl-assembly/assembly"]
profiles_and_goals = build_profiles + sbt_goals

print("[info] Building Spark (w/Hive 0.13.1) using SBT with these arguments: ",
print("[info] Building Spark (w/Hive 1.2.1) using SBT with these arguments: ",
" ".join(profiles_and_goals))

exec_sbt(profiles_and_goals)


def build_apache_spark(build_tool, hadoop_version):
"""Will build Spark against Hive v0.13.1 given the passed in build tool (either `sbt` or
"""Will build Spark against Hive v1.2.1 given the passed in build tool (either `sbt` or
`maven`). Defaults to using `sbt`."""

set_title_and_block("Building Spark", "BLOCK_BUILD")
Expand Down
Loading