sklearn runinference regression example #22088

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

tvalentyn merged 36 commits into apache:master from ryanthompson591:sklearn_regression_example

Aug 17, 2022

Contributor

ryanthompson591 commented Jun 28, 2022

This is an example of a regression model using an sklearn pipeline.

This illustrates how to separate examples based on what features are available in the example and then using an sklearn pipeline made for that data.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.


          Add a sample of a regression for japanese housing data

5be797a

asf-ci commented Jun 28, 2022

Can one of the admins verify this patch?

4 similar comments

asf-ci commented Jun 28, 2022

Can one of the admins verify this patch?

asf-ci commented Jun 28, 2022

Can one of the admins verify this patch?

asf-ci commented Jun 28, 2022

Can one of the admins verify this patch?

asf-ci commented Jun 28, 2022

Can one of the admins verify this patch?

github-actions bot added examples python labels

Contributor Author

ryanthompson591 commented Jun 28, 2022

R: @AnandInguva

Contributor

github-actions bot commented Jun 28, 2022

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control

codecov bot commented Jun 28, 2022 •

edited

Loading

Codecov Report

Merging #22088 (12c24f1) into master (48513ad) will decrease coverage by 0.14%.
The diff coverage is 34.48%.

@@            Coverage Diff             @@
##           master   #22088      +/-   ##
==========================================
- Coverage   74.18%   74.03%   -0.15%     
==========================================
  Files         706      706              
  Lines       93229    93428     +199     
==========================================
+ Hits        69159    69168       +9     
- Misses      22802    22994     +192     
+ Partials     1268     1266       -2

Flag	Coverage Δ
python	`83.55% <34.48%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...s/inference/sklearn_japanese_housing_regression.py	`34.48% <34.48%> (ø)`
.../python/apache_beam/testing/test_stream_service.py	`88.09% <0.00%> (-4.77%)`	⬇️
sdks/go/pkg/beam/io/filesystem/memfs/memory.go	`92.00% <0.00%> (-4.16%)`	⬇️
.../apache_beam/runners/interactive/dataproc/types.py	`93.10% <0.00%> (-3.45%)`	⬇️
...che_beam/runners/interactive/interactive_runner.py	`90.06% <0.00%> (-1.33%)`	⬇️
...ks/go/pkg/beam/runners/dataflow/dataflowlib/job.go	`21.55% <0.00%> (-1.13%)`	⬇️
sdks/python/apache_beam/internal/metrics/metric.py	`93.00% <0.00%> (-1.00%)`	⬇️
sdks/go/pkg/beam/core/runtime/exec/input.go	`50.96% <0.00%> (-1.00%)`	⬇️
...hon/apache_beam/runners/direct/test_stream_impl.py	`93.28% <0.00%> (-0.75%)`	⬇️
sdks/python/apache_beam/io/fileio.py	`95.98% <0.00%> (-0.28%)`	⬇️
... and 34 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

AnandInguva reviewed

View reviewed changes

sdks/python/apache_beam/examples/inference/sklearn_japanese_housing_regression.py Outdated Show resolved Hide resolved

Contributor Author

ryanthompson591 commented Jun 30, 2022

AnandInguva reviewed

View reviewed changes

sdks/python/apache_beam/examples/inference/sklearn_japanese_housing_regression.py Outdated Show resolved Hide resolved

sdks/python/apache_beam/examples/inference/sklearn_japanese_housing_regression.py Outdated Show resolved Hide resolved

AnandInguva reviewed

View reviewed changes

sdks/python/apache_beam/examples/inference/sklearn_japanese_housing_regression.py Show resolved Hide resolved

sdks/python/apache_beam/examples/inference/sklearn_japanese_housing_regression.py Outdated Show resolved Hide resolved

sdks/python/apache_beam/examples/inference/sklearn_japanese_housing_regression.py Outdated Show resolved Hide resolved

ryanthompson591 and others added 8 commits

June 30, 2022 13:55


          Update sdks/python/apache_beam/examples/inference/sklearn_japanese_ho…

f331db5

…using_regression.py

Co-authored-by: Anand Inguva <34158215+AnandInguva@users.noreply.github.com>


          added link to data

359128e


          Merge branch 'sklearn_regression_example' of https://github.com/ryant…

49c5f64

…hompson591/beam into sklearn_regression_example


          Update sdks/python/apache_beam/examples/inference/sklearn_japanese_ho…

16ece4d

…using_regression.py

Co-authored-by: Anand Inguva <34158215+AnandInguva@users.noreply.github.com>


          Update sdks/python/apache_beam/examples/inference/sklearn_japanese_ho…

a9b6d56

…using_regression.py

Co-authored-by: Anand Inguva <34158215+AnandInguva@users.noreply.github.com>


          used DoFn instead of function

aee8bf1


          Update with anands changes

a656c40


          Merge branch 'master' into sklearn_regression_example

0ee6178

tvalentyn reviewed

View reviewed changes

sdks/python/apache_beam/examples/inference/sklearn_japanese_housing_regression.py Show resolved Hide resolved

ryanthompson591 added 2 commits

July 12, 2022 13:51


          Updated readme file

b2c178f


          Added more detail to readme.

1f0a889

ryanthompson591 requested a review from tvalentyn

July 12, 2022 19:18

AnandInguva reviewed

View reviewed changes

sdks/python/apache_beam/examples/inference/sklearn_japanese_housing_regression.py Show resolved Hide resolved

sdks/python/apache_beam/examples/inference/sklearn_japanese_housing_regression.py Show resolved Hide resolved

sdks/python/apache_beam/examples/inference/sklearn_japanese_housing_regression.py Show resolved Hide resolved

ryanthompson591 added 2 commits

July 12, 2022 16:11


          updated input file description

147771f


          added integration test for japanese housing regression

48e1d63

ryanthompson591 and others added 6 commits

August 2, 2022 10:38


          merged with master


          Merge branch 'sklearn_regression_example' of https://github.com/ryant…

0a8de32

…hompson591/beam into sklearn_regression_example


          Update sdks/python/apache_beam/examples/inference/sklearn_japanese_ho…

00751b6

…using_regression.py

Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com>


          added yapf free section

aa37d68


          fixed split section, fixed comment

bf7ae66


          Merge branch 'sklearn_regression_example' of https://github.com/ryant…

f5229e0

…hompson591/beam into sklearn_regression_example

Contributor Author

ryanthompson591 commented Aug 2, 2022

Run Python 3.7 PostCommit


          allow more of a difference in assertion

7e9e41d

Contributor Author

ryanthompson591 commented Aug 3, 2022

Run Python 3.7 PostCommit

Contributor

tvalentyn commented Aug 3, 2022

Let me know when this is ready to review/merge.

Contributor Author

ryanthompson591 commented Aug 4, 2022

Run Python 3.9 PostCommit


          added comments, fixed a bug that might choose bad models

e103969

Contributor Author

ryanthompson591 commented Aug 5, 2022

Run Python 3.9 PostCommit

Contributor Author

ryanthompson591 commented Aug 5, 2022

Run Python 3.7 PostCommit

Contributor Author

ryanthompson591 commented Aug 5, 2022

Run Python 3.9 PostCommit


          changed example

5288bf2

Contributor Author

ryanthompson591 commented Aug 8, 2022

@tvalentyn This looks ready to merge, the failing test is not the new test I added but rather the issue described here: #22605

Contributor Author

ryanthompson591 commented Aug 8, 2022

Run Python 3.9 PostCommit

tvalentyn approved these changes

View reviewed changes

sdks/python/apache_beam/examples/inference/sklearn_japanese_housing_regression.py Outdated Show resolved Hide resolved

sdks/python/apache_beam/ml/inference/sklearn_inference_test.py Outdated

    
              # pylint: disable=wrong-import-order, wrong-import-position, ungrouped-imports, unused-import

              try:

                from apache_beam.io.gcp.gcsfilesystem import GCSFileSystem

Contributor

tvalentyn Aug 9, 2022

why is this necessary? I am not seeing any usage of GCSFileSystem

Contributor Author

ryanthompson591 Aug 9, 2022

This might have been preemptive. Unless we are able to save files, I think these tests could fail. However, it doesn't seem relevant to this PR and we should defer fixing it until it becomes an issue.

ryanthompson591 and others added 3 commits

August 9, 2022 09:57


          Update sdks/python/apache_beam/examples/inference/sklearn_japanese_ho…

bd41c7e

…using_regression.py

Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com>


          removed not needed check

00fa945


          Merge branch 'sklearn_regression_example' of https://github.com/ryant…

12c24f1

…hompson591/beam into sklearn_regression_example

ryanthompson591 commented

View reviewed changes

Contributor Author

ryanthompson591 left a comment

@tvalentyn PTAL

sdks/python/apache_beam/ml/inference/sklearn_inference_test.py Outdated

    
              # pylint: disable=wrong-import-order, wrong-import-position, ungrouped-imports, unused-import

              try:

                from apache_beam.io.gcp.gcsfilesystem import GCSFileSystem

Contributor Author

ryanthompson591 Aug 9, 2022

This might have been preemptive. Unless we are able to save files, I think these tests could fail. However, it doesn't seem relevant to this PR and we should defer fixing it until it becomes an issue.

Contributor Author

ryanthompson591 commented Aug 9, 2022

Run Python 3.9 PostCommit

1 similar comment

Contributor

tvalentyn commented Aug 16, 2022

Run Python 3.9 PostCommit

tvalentyn merged commit 6df4939 into apache:master

This was referenced Aug 23, 2022

Unskip RunInference IT tests once the licensing on the datasets and models is complete. #21859

Closed

unskip sklearn IT test #22825

Merged

MarcoRob pushed a commit to MarcoRob/beam that referenced this pull request


          sklearn runinference regression example (apache#22088)

3048bbe

Co-authored-by: Anand Inguva <34158215+AnandInguva@users.noreply.github.com>
Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

examples python