[SPARK-1517] Refactor release scripts to facilitate nightly publishing #7411

pwendell · 2015-07-15T03:41:03Z

This update contains some code changes to the release scripts that allow easier nightly publishing. I've been using these new scripts on Jenkins for cutting and publishing nightly snapshots for the last month or so, and it has been going well. I'd like to get them merged back upstream so this can be maintained by the community.

The main changes are:

Separates the release tagging from various build possibilities for an already tagged release (release-tag.sh and release-build.sh).
Allow for injecting credentials through the environment, including GPG keys. This is then paired with secure key injection in Jenkins.
Support for copying build results to a remote directory, and also "rotating" results, e.g. the ability to keep the last N copies of binary or doc builds.

I'm happy if anyone wants to take a look at this - it's not user facing but an internal utility used for generating releases.

SparkQA · 2015-07-15T03:46:11Z

Test build #37307 has finished for PR 7411 at commit d28c240.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class Exchange(newPartitioning: Partitioning, child: SparkPlan) extends UnaryNode

JoshRosen · 2015-08-01T17:59:17Z

Aw, looks like this conflicted with some recent Maven changes.

pwendell · 2015-08-01T19:06:17Z

Yeah I need to update this - but it's off the critical path for the release
so it's on hold.

On Sat, Aug 1, 2015 at 10:59 AM, Josh Rosen notifications@github.com
wrote:

Aw, looks like this conflicted with some recent Maven changes.

—
Reply to this email directly or view it on GitHub
#7411 (comment).

This update contains some code changes to the release scripts that allow easier nightly publishing. I've been using these new scripts on Jenkins for cutting and publishing nightly snapshots for the last month or so, and it has been going well. I'd like to get them merged back upstream so this can be maintained by the community. The main changes are: 1. Separates the release tagging from various build possibilities for an already tagged release (`release-tag.sh` and `release-build.sh`). 2. Allow for injecting credentials through the environment, including GPG keys. This is then paired with secure key injection in Jenkins. 3. Support for copying build results to a remote directory, and also "rotating" results, e.g. the ability to keep the last N copies of binary or doc builds. I'm happy if anyone wants to take a look at this - it's not user facing but an internal utility used for generating releases.

pwendell · 2015-08-10T21:00:40Z

Jenkins, test this please.

ryan-williams · 2015-08-10T21:04:17Z

dev/create-release/release-build.sh

you can drop these -DzincPort=$ZINC_PORTs as of #7944 right?

Yeah true, thanks for pointing that out.

On Mon, Aug 10, 2015 at 2:05 PM, Ryan Williams notifications@github.com
wrote:

In dev/create-release/release-build.sh
#7411 (comment):

-H "Content-Type:application/xml" -v \

$NEXUS_ROOT/profiles/$NEXUS_PROFILE/start)

staged_repo_id=$(echo $out | sed -e "s/.(orgapachespark-[0-9]{4})./\1/")

echo "Created Nexus staging repository: $staged_repo_id"

tmp_repo=$(mktemp -d spark-repo-XXXXX)

Generate random point for Zinc

export ZINC_PORT=$(python -S -c "import random; print random.randrange(3030,4030)")

build/mvn -DzincPort=$ZINC_PORT -Dmaven.repo.local=$tmp_repo -DskipTests $PUBLISH_PROFILES \

-Phive-thriftserver clean install

./dev/change-scala-version.sh 2.11

build/mvn -DzincPort=$ZINC_PORT -Dmaven.repo.local=$tmp_repo -Dscala-2.11 \

you can drop these -DzincPort=$ZINC_PORTs as of #7944
#7944 right?

—
Reply to this email directly or view it on GitHub
https://github.com/apache/spark/pull/7411/files#r36684398.

Actually since this is still used to build older versions, I think we'll
need to keep it for now, but we can eventually drop it.

On Mon, Aug 10, 2015 at 2:07 PM, Patrick Wendell patrick@databricks.com
wrote:

Yeah true, thanks for pointing that out.

On Mon, Aug 10, 2015 at 2:05 PM, Ryan Williams notifications@github.com
wrote:

In dev/create-release/release-build.sh
#7411 (comment):

-H "Content-Type:application/xml" -v \

$NEXUS_ROOT/profiles/$NEXUS_PROFILE/start)

staged_repo_id=$(echo $out | sed -e "s/.(orgapachespark-[0-9]{4})./\1/")

echo "Created Nexus staging repository: $staged_repo_id"

tmp_repo=$(mktemp -d spark-repo-XXXXX)

Generate random point for Zinc

export ZINC_PORT=$(python -S -c "import random; print random.randrange(3030,4030)")

build/mvn -DzincPort=$ZINC_PORT -Dmaven.repo.local=$tmp_repo -DskipTests $PUBLISH_PROFILES \

-Phive-thriftserver clean install

./dev/change-scala-version.sh 2.11

build/mvn -DzincPort=$ZINC_PORT -Dmaven.repo.local=$tmp_repo -Dscala-2.11 \

you can drop these -DzincPort=$ZINC_PORTs as of #7944
#7944 right?

—
Reply to this email directly or view it on GitHub
https://github.com/apache/spark/pull/7411/files#r36684398.

SparkQA · 2015-08-10T21:24:59Z

Test build #40321 has finished for PR 7411 at commit 233ce85.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-08-10T23:41:00Z

Test build #40327 has finished for PR 7411 at commit 74f9beb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

pwendell · 2015-08-11T05:16:46Z

Jenkins, retest this please.

SparkQA · 2015-08-11T07:30:35Z

Test build #40397 has finished for PR 7411 at commit 74f9beb.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

pwendell · 2015-08-11T16:07:52Z

Jenkins, retest this please.

pwendell · 2015-08-11T16:24:12Z

Jenkins, test this please.

pwendell · 2015-08-11T17:16:15Z

test

SparkQA · 2015-08-11T18:51:47Z

Test build #40463 has finished for PR 7411 at commit 74f9beb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

JoshRosen · 2015-08-11T19:13:39Z

Jenkins, retest this please.

SparkQA · 2015-08-11T21:48:59Z

Test build #40500 has finished for PR 7411 at commit 74f9beb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

pwendell · 2015-08-12T04:15:58Z

Okay will merge this - I've been keeping things in a separate repo and it's much better to have it in the upstream in case others want to modify it.

This update contains some code changes to the release scripts that allow easier nightly publishing. I've been using these new scripts on Jenkins for cutting and publishing nightly snapshots for the last month or so, and it has been going well. I'd like to get them merged back upstream so this can be maintained by the community. The main changes are: 1. Separates the release tagging from various build possibilities for an already tagged release (`release-tag.sh` and `release-build.sh`). 2. Allow for injecting credentials through the environment, including GPG keys. This is then paired with secure key injection in Jenkins. 3. Support for copying build results to a remote directory, and also "rotating" results, e.g. the ability to keep the last N copies of binary or doc builds. I'm happy if anyone wants to take a look at this - it's not user facing but an internal utility used for generating releases. Author: Patrick Wendell <patrick@databricks.com> Closes #7411 from pwendell/release-script-updates and squashes the following commits: 74f9beb [Patrick Wendell] Moving maven build command to a variable 233ce85 [Patrick Wendell] [SPARK-1517] Refactor release scripts to facilitate nightly publishing (cherry picked from commit 3ef0f32) Signed-off-by: Patrick Wendell <patrick@databricks.com>

This update contains some code changes to the release scripts that allow easier nightly publishing. I've been using these new scripts on Jenkins for cutting and publishing nightly snapshots for the last month or so, and it has been going well. I'd like to get them merged back upstream so this can be maintained by the community. The main changes are: 1. Separates the release tagging from various build possibilities for an already tagged release (`release-tag.sh` and `release-build.sh`). 2. Allow for injecting credentials through the environment, including GPG keys. This is then paired with secure key injection in Jenkins. 3. Support for copying build results to a remote directory, and also "rotating" results, e.g. the ability to keep the last N copies of binary or doc builds. I'm happy if anyone wants to take a look at this - it's not user facing but an internal utility used for generating releases. Author: Patrick Wendell <patrick@databricks.com> Closes apache#7411 from pwendell/release-script-updates and squashes the following commits: 74f9beb [Patrick Wendell] Moving maven build command to a variable 233ce85 [Patrick Wendell] [SPARK-1517] Refactor release scripts to facilitate nightly publishing

pwendell force-pushed the release-script-updates branch from d28c240 to 233ce85 Compare August 10, 2015 21:00

ryan-williams reviewed Aug 10, 2015
View reviewed changes

Moving maven build command to a variable

74f9beb

asfgit closed this in 3ef0f32 Aug 12, 2015

[SPARK-1517] Refactor release scripts to facilitate nightly publishing #7411

[SPARK-1517] Refactor release scripts to facilitate nightly publishing #7411

Uh oh!

Conversation

pwendell commented Jul 15, 2015

Uh oh!

SparkQA commented Jul 15, 2015

Uh oh!

JoshRosen commented Aug 1, 2015

Uh oh!

pwendell commented Aug 1, 2015

Uh oh!

pwendell commented Aug 10, 2015

Uh oh!

ryan-williams Aug 10, 2015

Choose a reason for hiding this comment

Uh oh!

pwendell Aug 10, 2015

Choose a reason for hiding this comment

Generate random point for Zinc

Uh oh!

pwendell Aug 10, 2015

Choose a reason for hiding this comment

Generate random point for Zinc

Uh oh!

SparkQA commented Aug 10, 2015

Uh oh!

SparkQA commented Aug 10, 2015

Uh oh!

pwendell commented Aug 11, 2015

Uh oh!

SparkQA commented Aug 11, 2015

Uh oh!

pwendell commented Aug 11, 2015

Uh oh!

pwendell commented Aug 11, 2015

Uh oh!

pwendell commented Aug 11, 2015 via email

Uh oh!

SparkQA commented Aug 11, 2015

Uh oh!

JoshRosen commented Aug 11, 2015

Uh oh!

SparkQA commented Aug 11, 2015

Uh oh!

pwendell commented Aug 12, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants