Skip to content

Syncing with upstream master#10

Merged
Edilmo merged 425 commits intomasterfrom
edpalenc/upstream-sync-20200321
Apr 1, 2020
Merged

Syncing with upstream master#10
Edilmo merged 425 commits intomasterfrom
edpalenc/upstream-sync-20200321

Conversation

@Edilmo
Copy link

@Edilmo Edilmo commented Mar 22, 2020

Why are these changes needed?

Several important changes has been produced in upstream since the last sync and also several CI task were pending. In this PR we are addressing both, pending sync with upstream and pending CI tasks.

This PR includes a lot of changes because of the big amount of PRs produced in upstream since the last sync.
In order to help with the review, here you can find a list of the commits made by us in this PR. The rest of the commits are coming from upstream. Also keep in mind that some commits are equal or almost equal to the commits in the following PR against releases/0.6.6, so if you review those, you don't need to review the commits with the same name here.

  1. Updating Azure Pipelines with travis updates: first commit after the first sync. Here we are also including support to all the pending CI jobs. Before we had the most importan 5 jobs. Now we have 17 jobs that are defined in the CI of the upstream project (travi and jenking ones). There still pending some stress tests that they have which runs 1 per week or per month (we have to confirm because they are not public, they are ran in EC2 private cluster).
  2. Updating Azure Pipelines with more travis updates: after finishing above commit, more changes went through in upstream. This commit is required to adapt our CI to those changes.
  3. Updating CI with new cpp worker tests: they added a new CI job for the new C++ API for Ray. This was after previos commits.
  4. Setting code owners: customizing code ownership for PR approvals.
  5. Fixing the version number generation: the generation of the custom build number per PR had a unnecesary 0 included.
  6. Making main pipeline also our release pipeline: now we are using the same pipeline that runs in each PR, as our package release version generation from master or any of the protected branches under releases/*. The packages generated from this pipeline runs will have a version number without the "dev" tag.
  7. Updating Azure Pipelines with travis updates: after finishing above commit, more changes went through in upstream. This commit is required to adapt our CI to those changes.
  8. Fixing wheels test: the final test of the wheels for linux has a bug. For some unknown issue, the agent where pipelines run, is not allowing to install manylinux1 packages. This PR is trying to fix that partially with some possible fixes but the issue is still present. Currently we have the same hack that upstream ray with a TODO.

After this PR, our pipeline will be a multi stage pipeline that looks like this:
Multi-stage pipeline

Checks

ericl and others added 30 commits February 26, 2020 22:54
* Move log_once from rllib to tune.

* Move log_once from rllib to tune.

* LINT.

* Move to ray.util.debug.
* Fix MAP_SHARED check in sys/mman.h

* Fix missing :platform_shims dependency for ray_util

* dlmalloc patch for Arrow
ray-project#7273)" (ray-project#7361)

This partially reverts commit 357232d.

The addition of python/__init__.py broke the build on Windows. However, this is difficult to notice because Bazel doesn't seem to notice this dependency. You first have to go to a commit that fails on this issue, and then try to re-build this commit, so that Bazel actually performs a rebuild.

A useful command-line for triggering the exact build i:

bazel build --compile_one_dependency //:python/ray/_raylet.pyx
* fix

* test_hist

* remove unnecessary value check

* pbt

* queue

* skip_for_now

* Apply suggestions from code review
…project#7314)

* WIP.

* WIP.

* WIP.

* WIP.

* WIP.

* Fix

* WIP.

* Add TD3 quick Pendulum regresison.

* Cleanup.

* Fix.

* LINT.

* Fix.

* Sort quick_learning test cases, add TD3.

* Sort quick_learning test cases, add TD3.

* Revert test_checkpoint_restore.py (debugging) changes.

* Fix old soft_q settings in documentation and test configs.

* More doc fixes.

* Fix test case.

* Fix test case.

* Lower test load.

* WIP.
* Switch to mirrors.bazel.build where possible

* Switch from .zip to .tar.gz for smaller downloads (it's also the default download on UNIX)

* Use direct GitHub URLs in Bazel files for clarity

* Don't pass patches to local_repository

* Remove github_repository()

* Switch to GitHub actions/checkout@v2 which is faster

* Use faster extraction method for LLVm on Windows

* Move LLVM_VERSION_WINDOWS to the shell script since it's not a CI-specific value

* Change GITHUB_TOKEN to GITHUB

* Don't show timestamps for GitHub Actions

* Factor out some options from GitHub Actions

* Tell Bazel to stay on the same volume in GitHun Actions

* Display progress output when downloading toolchains

Co-authored-by: GitHub Web Flow <noreply@github.com>
* Add Exploration API documentation.

* Add Exploration API documentation.

* Add Exploration API documentation.

* Update exporation docs.
* Fix

* Fix

* Fix complie

* Fix lint

* Fix linting

* Fix testDeleteObject

* Fix linting

* Update src/ray/raylet/worker_pool.cc

Co-Authored-By: Hao Chen <chenh1024@gmail.com>

* Update src/ray/raylet/worker_pool.cc

Co-Authored-By: Hao Chen <chenh1024@gmail.com>

* Update src/ray/raylet/worker_pool.h

Co-Authored-By: Hao Chen <chenh1024@gmail.com>

* Update src/ray/raylet/worker_pool.cc

Co-Authored-By: Hao Chen <chenh1024@gmail.com>

* Address comments.

* FIx linting

Co-authored-by: Hao Chen <chenh1024@gmail.com>
@Edilmo Edilmo force-pushed the edpalenc/upstream-sync-20200321 branch from a36dfa6 to 8a611a6 Compare March 30, 2020 23:57
@Edilmo Edilmo force-pushed the edpalenc/upstream-sync-20200321 branch 4 times, most recently from cb8b072 to e9033bb Compare March 31, 2020 08:29
@Edilmo Edilmo force-pushed the edpalenc/upstream-sync-20200321 branch from e9033bb to b19415a Compare March 31, 2020 09:19
@Edilmo Edilmo merged commit c60dddb into master Apr 1, 2020
@Edilmo Edilmo deleted the edpalenc/upstream-sync-20200321 branch April 1, 2020 05:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.