Skip to content

[SPARK-11891] Model export/import for RFormula and RFormulaModel #9884

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
yinxusen wants to merge 10 commits into from
Closed

[SPARK-11891] Model export/import for RFormula and RFormulaModel #9884

yinxusen wants to merge 10 commits into from

Conversation

@yinxusen
Copy link
Contributor

@yinxusen yinxusen commented Nov 21, 2015

@yinxusen
Copy link
Contributor Author

yinxusen commented Nov 21, 2015

I will update CrossValidator after it merging into master.

@SparkQA
Copy link

SparkQA commented Nov 21, 2015

Test build #46476 has finished for PR 9884 at commit aa2d9aa.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 21, 2015

Test build #46479 has finished for PR 9884 at commit a4b72a0.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * class RFormula(override val uid: String)\n * class VectorAttributeRewriterWriter(instance: VectorAttributeRewriter) extends MLWriter\n

@SparkQA
Copy link

SparkQA commented Nov 24, 2015

Test build #46605 has finished for PR 9884 at commit 7b9abf6.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * class RFormula(override val uid: String)\n * class VectorAttributeRewriterWriter(instance: VectorAttributeRewriter) extends MLWriter\n

@cafreeman
Copy link

cafreeman commented Dec 10, 2015

Just for clarification, will this approach allow for saving out a model object and then using it in a different SparkContext than the one in which it was created? I see that all the stages are written out and preserved, but I've seen SparkR throw a fit about trying to use things across contexts before.

@yinxusen
Copy link
Contributor Author

yinxusen commented Dec 11, 2015

@cafreeman I believe the saved pipeline can be loaded in another spark context.

@SparkQA
Copy link

SparkQA commented Jan 7, 2016

Test build #48939 has finished for PR 9884 at commit b18cea0.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@yinxusen
Copy link
Contributor Author

yinxusen commented Jan 8, 2016

ok to test

@yinxusen
Copy link
Contributor Author

yinxusen commented Jan 8, 2016

cc @jkbradley I updated the cross validator. Could you make a pass of it? Thx

@SparkQA
Copy link

SparkQA commented Jan 8, 2016

Test build #48998 has finished for PR 9884 at commit 3c8faab.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@yinxusen yinxusen mentioned this pull request Feb 15, 2016
Closed
asfgit pushed a commit that referenced this pull request Mar 4, 2016
@yinxusen @mengxr
[SPARK-13036][SPARK-13318][SPARK-13319] Add save/load for feature.py
83302c3 
Add save/load for feature.py. Meanwhile, add save/load for `ElementwiseProduct` in Scala side and fix a bug of missing `setDefault` in `VectorSlicer` and `StopWordsRemover`.

In this PR I ignore the `RFormula` and `RFormulaModel` because its Scala implementation is pending in #9884. I'll add them in this PR if #9884 gets merged first. Or add a follow-up JIRA for `RFormula`.

Author: Xusen Yin <yinxusen@gmail.com>

Closes #11203 from yinxusen/SPARK-13036.
@jkbradley
Copy link
Member

jkbradley commented Mar 16, 2016

I'll take a look now

@SparkQA
Copy link

SparkQA commented Mar 16, 2016

Test build #2644 has finished for PR 9884 at commit 3c8faab.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

jkbradley
jkbradley reviewed Mar 16, 2016
View reviewed changes
mllib/src/main/scala/org/apache/spark/ml/feature/RFormula.scala Outdated
resolvedFormula: ResolvedRFormula,
pipelineModel: PipelineModel)
extends Model[RFormulaModel] with RFormulaBase {
val resolvedFormula: ResolvedRFormula,
Copy link
Member

@jkbradley jkbradley Mar 16, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These vals should be private[feature]

@jkbradley
Copy link
Member

jkbradley commented Mar 16, 2016

I only saw a few issues. Thanks!

yinxusen
yinxusen reviewed Mar 17, 2016
View reviewed changes
mllib/src/main/scala/org/apache/spark/ml/feature/RFormula.scala
pipelineModel: PipelineModel)
extends Model[RFormulaModel] with RFormulaBase {
private[ml] val resolvedFormula: ResolvedRFormula,
private[ml] val pipelineModel: PipelineModel)
Copy link
Contributor Author

@yinxusen yinxusen Mar 17, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer to leave the private scope of ml, other than feature, because CrossValidator accesses the pipelineModel.

@SparkQA
Copy link

SparkQA commented Mar 17, 2016

Test build #53390 has finished for PR 9884 at commit 739eb87.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@jkbradley
Copy link
Member

jkbradley commented Mar 17, 2016

LGTM
Merging with master
Thanks @yinxusen !

@asfgit asfgit closed this in edf8b87 Mar 17, 2016
@yinxusen yinxusen mentioned this pull request Mar 18, 2016
Closed
roygao94 pushed a commit to roygao94/spark that referenced this pull request Mar 22, 2016
@yinxusen @roygao94
[SPARK-13036][SPARK-13318][SPARK-13319] Add save/load for feature.py
d554336 
Add save/load for feature.py. Meanwhile, add save/load for `ElementwiseProduct` in Scala side and fix a bug of missing `setDefault` in `VectorSlicer` and `StopWordsRemover`.

In this PR I ignore the `RFormula` and `RFormulaModel` because its Scala implementation is pending in apache#9884. I'll add them in this PR if apache#9884 gets merged first. Or add a follow-up JIRA for `RFormula`.

Author: Xusen Yin <yinxusen@gmail.com>

Closes apache#11203 from yinxusen/SPARK-13036.
roygao94 pushed a commit to roygao94/spark that referenced this pull request Mar 22, 2016
@yinxusen @roygao94
[SPARK-11891] Model export/import for RFormula and RFormulaModel
c92760f 
https://issues.apache.org/jira/browse/SPARK-11891

Author: Xusen Yin <yinxusen@gmail.com>

Closes apache#9884 from yinxusen/SPARK-11891.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@yinxusen @SparkQA @cafreeman @jkbradley