Start building pre-releases with cythonized scheduler

[charlesbluca]

This PR modifies the Distributed pre-release recipe to build Cython variants of Distributed using the steps outlined by @jakirkham in #4442 (comment).

Choosing which Distributed variant to install should be doable by specifying the distributed-proc build string during an install / env creation command, though with track_features="cythonized-scheduler" specified for Cython builds Conda should always prioritize non-Cython builds by default.

• Closes #xxxx
• Tests added / passed
• Passes pre-commit run --all-files

[jrbourbeau]

Thanks @charlesbluca. There was some talk during the last monthly community meeting of dropping the cythonized version of the scheduler (also xref #5685). My guess is we should wait for a decision on that before adding extra builds here

[jakirkham]

James as part of that discussion we had decided that if some work was done here to setup nightlies and refactor out the Cythonized parts, we could keep it. This is one of those steps. However I think it is important for us to have the opportunity to progress here otherwise none of the actionables requested can complete. IOW it becomes a Catch 22

[jrbourbeau]

Ah, my mistake -- I thought we were just looking for progress on the refactor part. If nightlies were also requested, then let's carry on with the changes here 👍

[jakirkham]

Right this was one of the issues Florian had raised in the issue above and was discussed in the meeting. So this is our attempt to address that.

Agree there is still refactoring work to do as well.

[github-actions]

Unit Test Results

       12 files  +       4         12 suites  +4   7h 15m 20s ⏱️ + 2h 39m 10s
  2 621 tests +     20    2 540 ✔️ +     31    81 💤  -     9  0 ❌  - 2 
15 650 runs  +5 270  14 762 ✔️ +5 055  888 💤 +217  0 ❌  - 2 

Results for commit 369088b. ± Comparison against base commit 60d82c2.

♻️ This comment has been updated with latest results.

[jakirkham]

Thanks Charles! 😄

Had a few suggestions below. Generally this seems to be going in a good direction.

[charlesbluca]

Thanks for the thorough review @jakirkham 😄 a few responses:

[jakirkham]

Added some suggestions above to make this noarch. We should be able to drop the conda convert lines with this

[jakirkham]

Sorry for the 2nd review batch. Thinking this will resolve the CI issue

[jakirkham]

This is looking pretty good. Thanks for all of the work here Charles 😄

Had a couple suggestions related to how selectors and Jinja work. IIUC this should also fix the CI issue

[jakirkham]

Jinja unfortunately doesn't affect selectors. So we would need to perform this comparison in selectors

[charlesbluca]

Ah thanks for the clarification here - good to know for the future 🙂

[jakirkham]

Ofc. Showed another way we could do this in this review ( #5831 (review) ). Though ok with either approach 🙂

[jakirkham]

Not that we need to do this, but this is another way we could handle the Jinja/selector Cython portion

[charlesbluca]

Thanks for all the help here @jakirkham 😄 some questions w.r.t. shifting towards noarch, as I notice we are still building py38 / py39 packages:

[jakirkham]

The different versions are still needed for Cython (though not pure Python). That said, maybe there is a way to simplify things if we use a skip. Would also want to check what Python versions are built against by default with Conda-Build

[jakirkham]

Looks like it is doing the right thing now

Details

INFO :: The inputs making up the hashes for the built packages are as follows:
{
  "distributed-2022.02.1a220222-py38_g293ef607_47_cython": {
    "recipe": {
      "c_compiler": "gcc",
      "cython_enabled": "True",
      "target_platform": "linux-64"
    }
  },
  "distributed-2022.02.1a220222-py38_g293ef607_47_python": {
    "recipe": {
      "cython_enabled": "False"
    }
  },
  "distributed-2022.02.1a220222-py39_g293ef607_47_cython": {
    "recipe": {
      "c_compiler": "gcc",
      "cython_enabled": "True",
      "target_platform": "linux-64"
    }
  },
  "distributed-2022.02.1a220222-py39_g293ef607_47_python": {
    "recipe": {
      "cython_enabled": "False"
    }
  },
  "distributed-impl-2022.02.1a220222-cython": {
    "recipe": {}
  },
  "distributed-impl-2022.02.1a220222-python": {
    "recipe": {}
  }
}

[jakirkham]

LGTM. Thanks Charles! 😄

[jakirkham]

@jrbourbeau @jsignell, do either of you have more thoughts here? 🙂

[jsignell]

No thoughts from me. I'm happy if you are happy :)

[jrbourbeau]

Yep, same here, I'm happy if you're happy with the changes here.

(This is a non-blocking comment) I noticed the nightly CI build here takes ~10 minutes and is run on all pull requests and pushes to main. That seems like a significant amount of CI time to me. I'm wondering how much value there is in building that often vs. only on pushes to main or even a single nightly cron job. dask/dask and dask/distributed have been quite active recently and I've noticed longer wait times for CI builds to start. If we switched to only building on pushes to main or a nightly cron job, would folks using these builds miss out on much?

[jakirkham]

There is probably still some value in building these on PRs that change the Conda packages in particular (though maybe this can be skipped for other PRs)

Otherwise building on main should be fine

[jrbourbeau]

PRs that change the Conda packages in particular (though maybe this can be skipped for other PRs)

Ah, that sounds like a nice balance. It looks like GitHub has support for only triggering builds when certain files are modified https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#onpushpull_requestpull_request_targetpathspaths-ignore. Again, I don't mean for this to block this PR. We can make further updates in future PRs

[charlesbluca]

Yeah was about to suggest the same method - I'm okay with pushing a quick commit doing this targetting the following files:

• the conda recipes
• setup.py
• the workflow file controlling the pre-release builds

[jakirkham]

@jrbourbeau how do the changes above look? 🙂

(thanks Charles for adding those! 🙏)

[jrbourbeau]

Nice, LGTM -- thanks @charlesbluca

[jakirkham]

Thanks Charles for the PR and Julia & James for the reviews! 😄

[jakirkham]

Tested the file filtering logic in PR ( #5826 ) and it works as expected (Conda nightly build skipped)! 🎉

Edit: Also file filters work when modifying the recipes as in PR ( #5857 ) 🚀