From 7133a100d47b858d7915a0bff6e620203eaea988 Mon Sep 17 00:00:00 2001 From: iameskild Date: Thu, 16 Jun 2022 14:33:17 -0700 Subject: [PATCH 1/2] 0.4.0 release, clean up --- .gitignore | 1 + README.md | 19 +++++++- docs/source/conf.py | 2 +- docs/source/dev-guide.md | 85 +++++++++++++++++++++++++++++++++++ docs/source/index.md | 1 + docs/source/user-guide.md | 2 +- kbatch-proxy/requirements.txt | 9 ---- kbatch-proxy/setup.cfg | 10 ++++- kbatch/kbatch/__init__.py | 2 +- kbatch/setup.cfg | 8 +++- 10 files changed, 124 insertions(+), 15 deletions(-) create mode 100644 docs/source/dev-guide.md delete mode 100644 kbatch-proxy/requirements.txt diff --git a/.gitignore b/.gitignore index a8a8fac..880fc37 100644 --- a/.gitignore +++ b/.gitignore @@ -6,3 +6,4 @@ htmlcov dist/ docs/build .venv +.DS_store \ No newline at end of file diff --git a/README.md b/README.md index 057f0d9..5c4cc77 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ This repository contains two packages: [`kbatch`](./kbatch) is for users, [`kbat `kbatch` can be installed from source using pip: ``` -$ pip install "git+https://github.com/kbatch-dev/kbatch/#egg=kbatch&subdirectory=kbatch" +$ pip install kbatch ``` ## Usage @@ -101,6 +101,23 @@ Show the detail on a given job $ kbatch job show " ``` +### Submit cronjob + +Similar to jobs, cronjobs require `name`, `command` and container `image`. However cronjobs also require a `schedule` which follows the [cron schedule syntax](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#cron-schedule-syntax). + + +```{code-block} console +$ kbatch cronjob submit \ + --name=list-files \ + --image=alpine \ + --command='["ls", "-lh"]' \ + --schedule='0 22 * * 1-5' +``` + +This job will now run at 22:00 on every day-of-week from Monday through Friday **indefinitely**. + +> NOTE: Given that cronjobs run on a schedule **indefinitely**, the only way to stop them is to manually delete the cronjob. See the [user guide docs](https://kbatch.readthedocs.io/en/latest/user-guide.html#submit-a-cronjob) for more information. + ### Show pod logs Note that this is the **pod** id, not the **job** id. diff --git a/docs/source/conf.py b/docs/source/conf.py index 864ffe2..a04ecea 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -22,7 +22,7 @@ author = "Tom Augspurger" # The full version, including alpha/beta/rc tags -release = "0.3.2" +release = "0.4.0" # -- General configuration --------------------------------------------------- diff --git a/docs/source/dev-guide.md b/docs/source/dev-guide.md new file mode 100644 index 0000000..4571fac --- /dev/null +++ b/docs/source/dev-guide.md @@ -0,0 +1,85 @@ +# Dev Guide + +## Development setup + +Clone the `kbatch` repo onto your local and create a virtual environment (`conda`, `venv`, etc.) with Python 3.9 installed. From this directory, you can pip install the packages needed for development. + +``` +$ pip install -e ./kbatch[all] +... +$ pip install -e ./kbatch-proxy[all] +``` + +> NOTE: The `-e` signifies that this is an "editable install", read more [here](https://pip.pypa.io/en/stable/topics/local-project-installs/#editable-installs). + + +The first pip install command above should also installed `pre-commit` used to format and lint the code. Run the following command to start using `pre-commit`. + +``` +$ pre-commit install --install-hooks +``` + + +## `kbatch-proxy` development setup + +We don't have a fully working docker-ized setup, since we (i.e. Tom) don't know how to do Kubernetes within docker. So the current setup relies on + +1. k3d for Kubernetes +2. JupyterHub as a regular Python process +3. kbatch-proxy as a regular Python process + +### Create a cluster + +``` +$ k3d cluster create ksubmit +``` + +### Create a Hub + +make sure to `npm install configurable-http-proxy`. + +``` +$ cd hub +$ jupyterhub +``` + +### Start kbatch-proxy + +``` +KBATCH_PREFIX=/services/kbatch \ + KBATCH_PROFILE_FILE=tests/profile_template.yaml \ + JUPYTERHUB_API_TOKEN=super-secret \ + JUPYTERHUB_API_URL=http://127.0.0.1:8000/hub/api \ + JUPYTERHUB_HOST=http://127.0.0.1:8000 \ + uvicorn kbatch_proxy.main:app --reload --port=8050 +``` + +You'll might want to log in and create a token at http://localhost:8000/hub/token. The `kbatch configure` with that token. + +## Release process + +These are the current steps that it takes to cut a new release of `kbatch`. + +### Release new version of `kbatch`/`kbatch-proxy`: +1. Update the version in the following files: + - [`kbatch/setup.cfg`](https://github.com/kbatch-dev/kbatch/blob/main/kbatch/setup.cfg#L7) + - [`kbatch/kbatch/__init__.py`](https://github.com/kbatch-dev/kbatch/blob/main/kbatch/kbatch/__init__.py#L17) + - [`kbatch-proxy/setup.cfg`](https://github.com/kbatch-dev/kbatch/blob/main/kbatch-proxy/setup.cfg#L7) + - [`docs/source/conf.py`](https://github.com/kbatch-dev/kbatch/blob/main/docs/source/conf.py#L25) +3. Update any relevant docs and merge these changes into the codebase. +2. [Draft a release on the `kbatch` GitHub repo.](https://github.com/kbatch-dev/kbatch/releases). + - This triggers the [`publish-image.yaml`](https://github.com/kbatch-dev/kbatch/blob/main/.github/workflows/publish-image.yaml) workflow which: + - publishes a container image for `kbatch-proxy` to the [GitHub container registry](https://github.com/kbatch-dev/kbatch/pkgs/container/kbatch-proxy). + - publish `kbatch` and `kbatch-proxy` python libraries to PyPI. + + +### Release new version of `helm-chart`: +1. Once the container image and libraries have been successfully published, update the version of the `helm-chart` to match this new version of `kbatch`: + - [`kbatch/Chart.yaml`](https://github.com/kbatch-dev/helm-chart/blob/main/kbatch/Chart.yaml#L6) +2. Update any relevant docs and merge these changes into the codebase. +3. [Draft a release on the `helm-chart` GitHub repo.](https://github.com/kbatch-dev/helm-chart/releases) + - This triggers the [`publish-chart.yaml`](https://github.com/kbatch-dev/helm-chart/blob/main/.github/workflows/publish-charts.yml) workflow which: + - publishes a Helm chart to the [GitHub Pages site](https://kbatch-dev.github.io/helm-chart/) for this repo. + +> NOTE: there are known limitations to the pre-release process outlined [here](https://github.com/kbatch-dev/kbatch/issues/42). + diff --git a/docs/source/index.md b/docs/source/index.md index 795b988..4076014 100644 --- a/docs/source/index.md +++ b/docs/source/index.md @@ -59,4 +59,5 @@ For more on how to use `kbatch`, see the [User guide](./user-guide.md) user-guide examples/index.md kbatch-proxy +dev-guide ``` diff --git a/docs/source/user-guide.md b/docs/source/user-guide.md index a8ecf1a..cab8fb3 100644 --- a/docs/source/user-guide.md +++ b/docs/source/user-guide.md @@ -59,7 +59,7 @@ If you'd like your job to run on a repeating schedule, you can leverage CronJobs $ kbatch cronjob submit \ --name=list-files \ --image=alpine \ - --command='["ls", "-lh"]' + --command='["ls", "-lh"]' \ --schedule='0 22 * * 1-5' ``` diff --git a/kbatch-proxy/requirements.txt b/kbatch-proxy/requirements.txt deleted file mode 100644 index 62c748a..0000000 --- a/kbatch-proxy/requirements.txt +++ /dev/null @@ -1,9 +0,0 @@ -fastapi -httpx -kubernetes -uvicorn[standard] -gunicorn==20.1.0 -jupyterhub -escapism -rich -pydantic[dotenv] \ No newline at end of file diff --git a/kbatch-proxy/setup.cfg b/kbatch-proxy/setup.cfg index 7f42591..0f3a1b4 100644 --- a/kbatch-proxy/setup.cfg +++ b/kbatch-proxy/setup.cfg @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [metadata] name = kbatch-proxy -version = 0.3.2 +version = 0.4.0 description = Proxy batch job requests to kubernetes. long_description = file: README.md long_description_content_type = text/markdown @@ -23,10 +23,18 @@ install_requires = pydantic[dotenv] [options.extras_require] +all = + %(test)s + %(dev)s test = pytest pytest-mock +dev = + uvicorn[standard] + gunicorn==20.1.0 + rich + [options.entry_points] console_scripts = kbatch = kbatch.cli:cli \ No newline at end of file diff --git a/kbatch/kbatch/__init__.py b/kbatch/kbatch/__init__.py index 0129d4b..3f444bb 100644 --- a/kbatch/kbatch/__init__.py +++ b/kbatch/kbatch/__init__.py @@ -14,7 +14,7 @@ from ._backend import make_job, make_cronjob -__version__ = "0.3.2" +__version__ = "0.4.0" __all__ = [ "__version__", diff --git a/kbatch/setup.cfg b/kbatch/setup.cfg index 9f031e6..7973834 100644 --- a/kbatch/setup.cfg +++ b/kbatch/setup.cfg @@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta" [metadata] name = kbatch -version = 0.3.2 +version = 0.4.0 description = Submit batch jobs to Kubernetes. long_description = file: README.md long_description_content_type = text/markdown @@ -22,9 +22,15 @@ install_requires = kubernetes [options.extras_require] +all = + %(docs)s + %(test)s test = pytest respx + mypy + types-PyYAML + pre-commit docs = sphinx From 728e8ec2fe3d453e58019031deb813532556533b Mon Sep 17 00:00:00 2001 From: iameskild Date: Thu, 16 Jun 2022 14:44:46 -0700 Subject: [PATCH 2/2] Add more dev-guide docs --- docs/source/dev-guide.md | 34 +++++++++++++++++++++++++++++++--- 1 file changed, 31 insertions(+), 3 deletions(-) diff --git a/docs/source/dev-guide.md b/docs/source/dev-guide.md index 4571fac..98e7120 100644 --- a/docs/source/dev-guide.md +++ b/docs/source/dev-guide.md @@ -2,7 +2,7 @@ ## Development setup -Clone the `kbatch` repo onto your local and create a virtual environment (`conda`, `venv`, etc.) with Python 3.9 installed. From this directory, you can pip install the packages needed for development. +Clone the `kbatch` repo onto your local machine and create a virtual environment (`conda`, `venv`, etc.) with Python 3.9 installed. From this directory, you can pip install the packages needed for development ``` $ pip install -e ./kbatch[all] @@ -12,13 +12,41 @@ $ pip install -e ./kbatch-proxy[all] > NOTE: The `-e` signifies that this is an "editable install", read more [here](https://pip.pypa.io/en/stable/topics/local-project-installs/#editable-installs). - -The first pip install command above should also installed `pre-commit` used to format and lint the code. Run the following command to start using `pre-commit`. +The first pip install command above should've also installed `pre-commit` which is used to format and lint the code. Run the following command to start using `pre-commit` ``` $ pre-commit install --install-hooks ``` +### Run `pytest` + +To validate your changes against the test suite, run + +``` +$ pytest -v ./kbatch +... +$ pytest -v ./kbatch-proxy +``` + +### Run `mypy` + +To perform `mypy` type checking, run + +``` +$ mypy ./kbatch +... +$ mypy ./kbatch-proxy +``` + +### To build the docs locally + +To build the docs locally, run + +``` +$ cd ./docs +$ make html +``` + ## `kbatch-proxy` development setup