diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e37a9a2c..d019553c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -6,11 +6,28 @@ SPDX-License-Identifier: CC-BY-SA-4.0 -# HERMES development workflow +# Contribution Guidelines -## Preamble +## Feedback + +This is an open repository, and we are very happy to receive contributions to the HERMES workflow from the community, +for example as feedback, bug reports, feature requests, etc. + +We see our project as part of a global and interdisciplinary effort to improve the state of the art in +research software engineering, maintenance and scholarly communications around research software. We therefore +appreciate any feedback you may have on the HERMES project itself and any of its outputs. + +Either [create an issue](https://github.com/hermes-hmc/workflow/issues/new/choose) in our project repository or +[send us an email](mailto:team@software-metadata.pub?subject=HERMES%20WOrkflow%20Reachout). + +## HERMES development workflow + +The following describes the workflow for contributions. + +### Preamble > **Branching is cheap!** > @@ -18,92 +35,107 @@ SPDX-FileContributor: Stephan Druskat > > Aim for quick turnaround times! -## Branching +### Branching + +We loosely follow a mixture of [GitHubFlow](https://docs.github.com/en/get-started/quickstart/github-flow) and [GitFlow](https://nvie.com/posts/a-successful-git-branching-model/) with the following branches. + +#### `main` + +- This is the stable branch. +- Merges into `main` come only from `develop` or a hotfix branch (i.e., when something needs to be fixed in "production"). + +#### `develop` + +- This is the unstable development branch. +- Before you attempt to break something or add experimental changes, `git tag` the current state. + +#### `feature/` -We loosely follow a mixture of [GitHubFlow](https://docs.github.com/en/get-started/quickstart/github-flow) and [GitFlow](https://nvie.com/posts/a-successful-git-branching-model/): +- Naming convention: include an issue id if one exists, e.g., `feature/62-improve-broken-thing` or `feature/42-add-new-thing`. +- Branch from last tag on `develop`. -`main` - - Stable branch - - Merges come only from `develop` or a hotfix branch (i.e., when something needs to be fixed in "production") +#### `hotfix/` -`develop` - - Unstable development branch - - Last "stable" is tagged: tag before attempting to break something +- Naming convention: include an issue id if one exists, e.g., `hotfix/62-fix-broken-thing-in-release`. +- Branch from `main`. -`feature/` (including issue id when exists: `feature/62-improve-broken-thing`/`feature/42-add-new-thing`) - - Branch from last tag on `develop` +### Pull requests (PRs) -`hotfix/` (including issue id when exists: `hotfix/62-fix-broken-thing-in-release`) - - Branch from `main` +Project members may create pull requests from branches in the main repository, while external contributors need to follow +a [forking pattern](https://docs.github.com/en/get-started/quickstart/fork-a-repo). In both cases, please follow these rules: -### Pull requests -- As soon as you have made 1 commit in a feature branch, put up a *draft* pull request -- Keep pull requests small :skull: -- :warning: No pre-emptive reviews on PR drafts, **unless** the PR author @-mentions with this specific request -- When you think you're done, mark PR ready for review, see below (Merge Process) +- As soon as you have made 1 commit in a feature branch, put up a *draft* pull request. +- Keep pull requests small. +- ⚠️ Do not review *draft* pull requests, unless the PR author @-mentions you with this specific request. +- When you think you're done, mark the PR ready for review to start the [merge process](#merging-changes-into-develop). -## Merge Process (into `develop`) +### Merging changes into `develop` -- Create PR from `feature/...` against `develop` (PR author) -- Describe work in initial comment (PR author) +- *Create draft PR:* The **contributor** [creates a draft pull request (PR)](#pull-requests-prs) from `feature/...` against `develop` (and becomes the **PR author**). +- *Describe changes:* The **PR author** describes the changes in the PR in the initial comment: - Reference any related issues (use, e.g., `Fixes #n` or `- Related: #n`) - - What does the new code do? - - Optional: What should reviewers look at specifically - - Information on how to review: - - E.g. + - Describe what new code does + - Optional: Describe what reviewers should look at specifically + - Include information on how to review: + - E.g.: ```bash - pip install ./ - pytest test/ + poetry install + poetry run pytest test/ ``` -- Request review (PR author) - - Eligible reviewers: - - Python code: @led02, @sdruskat, @jkelling - - Documentation: +- *Request review:* The **PR author** requests one or more reviews. + - Eligible reviewers are: + - For Python code: @led02, @sdruskat, @jkelling + - For Documentation: - Workflow: @all - Project: @all -- Review (at least 1 reviewer) - - Follow instructions in PR +- *Review:* At least 1 **reviewer** reviews the changes: + - Follow the instructions in the PR - Review thoroughly beyond instructions - - Comments / change suggestions inline in files - - Submit review: - - **CASE 1:** Non-blocking change requests (typos, documentation wording, etc.) -> Document and Accept - - **CASE 2:** Blocking change requests (something doesn't work, bad quality code, docs not understandable): - - Ideally, fix things yourself in the branch -> Request Changes (or do them on your own) - - **CASE 3:** "Just a comment"s, pointers to potential future changes -> Document and Accept - - **CASE 4:** :warning: If you want a second pair of eyes on the PR, use "Comment" to finish review, request another reviewer and @-mention in comment. - - Optional: If you find something blocking after initial review, add review with "Request changes" outcome -- Act on review (PR author) + - Add comments or change suggestions inline in the respective file using GitHub's * changed* tab + - Submit the review with the correct review outcome: + - **CASE 1:** Non-blocking change requests (typos, documentation wording, etc.) -> *Document and Accept* + - **CASE 2:** Blocking change requests (something doesn't work, bad quality code, docs are not understandable): + - Ideally, fix things yourself in the branch -> *Request Changes* (or do them on your own) + - **CASE 3:** "Just a comment"s, pointers to potential future changes -> *Document and Accept* + - **CASE 4:** ⚠️ If you want a second pair of eyes on the PR, use *Comment* to finish the review, then request + another reviewer and @-mention them in a comment on the PR. + - Optional: If you find something blocking after your initial review, add another review with *Request changes* outcome. +- *Act on review:* The **PR author** acts on the review - React to comments - - Fix issues + - Fix issues (including non-blocking issues) - Discuss options -- Then: - - CASE 1: PR author to fix non-blocking issues and merge - - CASE 2: PR author to re-request review from original reviewer(s) - - CASE 3: PR author to react to comments and merge - - CASE 4: (PR author's reaction depends on outcome of second review) -- Re-review: - - See Review above -- Any maintainer - - Close PR if PR is not suitable for merge, and no further changes to improve it come from the PR author, - after having communicated sensible requests with a deadline for further work in the PR comments. - - Merge PR and delete remote branch if at least half of the invited reviewers have approved the PR, and no changes have been requested after review. - This implements lazy consensus to avoid bottlenecks, where a PR has been approved by some reviewers but cannot be closed due to missing reviews. - -## Release/stabilization process - -- Create release branch `release/v` from `develop` -- Check if everything looks good - - Audit source (using linters and stuff) - - Ensure test coverage of at least 72.3% - - Check if documentation aligns with code (also run tutorial to check completeness) - - Check if metadata is correct -- Put up PR from release branch against `main` -- Request review (workflow as above) -- Merge into `main` -- Tag `main` HEAD as `v` -- Push `main` -- Push tag -- Merge `main` into `develop` -- Delete release branch -- :bulb: If something goes wrong in the release branch, you can always delete, fix things in a feature branch, merge into `develop` following workflow above, and start anew + - Then: + - **CASE 1:** The **PR author** merges the PR. + - **CASE 2:** The **PR author** re-requests a review from the original reviewer(s). + - **CASE 3:** The **PR author** reacts to any comments. If all comments are resolved, the **PR author** merges the PR. + - **CASE 4:** The correct next step depends on the outcome of the second review. +- *Re-review:* + - *See Review* above +- **Any maintainer** can: + - Close a PR if the PR is not suitable for merging, and no further changes to improve it come from the PR author. + ⚠️ Only do this after after having communicated sensible requests with a deadline for further work in the PR comments. + - Merge a PR and delete the remote branch if at least half of the invited reviewers have approved the PR, and no changes + have been requested after review. This implements lazy consensus to avoid bottlenecks, where a PR has been + approved by some reviewers but cannot be closed due to missing reviews. + +### Stabilizing the codebase and making releases + +⚠️ The following steps can only be taken by maintainers. + +1. Create a release branch `release/v` from `develop`. +1. Check if everything looks good: + 1. Audit the source code (using linters and other tooling). + 1. Ensure test coverage is at least 65%, and that all tests pass. + 1. Check if the documentation aligns with the code (also run tutorial to check completeness). + 1. Check if the metadata is correct in all relevant places. +1. Put up a PR from the release branch against `main`. +1. Request a review (using the same workflow as above). +1. Merge the PR into `main`. +1. Tag `main`'s `HEAD` as `v`. +1. Push `main`. +1. Push tag. +1. Merge `main` into `develop`. +1. Delete the release branch. +1. 💡 If something goes wrong in the release branch, you can always delete it, fix things in a feature branch, merge + into `develop` following the workflow above, and start anew. diff --git a/LICENSE.md b/LICENSE.md index 7d695d24..a2c44025 100644 --- a/LICENSE.md +++ b/LICENSE.md @@ -14,7 +14,7 @@ The work contained in this repository is licensed under different licenses: - Source code is licensed under [`Apache-2.0`](LICENSES/Apache-2.0.txt). - Documentation (including images) is licensed under [`CC BY-SA 4.0`](LICENSES/CC-BY-SA-4.0.txt). - The HERMES visual ("logo") is licensed under [`CC BY-ND 4.0`](LICENSES/CC-BY-ND-4.0.txt). -- Other files are licensed under [CC0-1.0](LICENSES/CC0-1.0.txt). +- Other files are licensed under [`CC0-1.0`](LICENSES/CC0-1.0.txt). Please see the individual files for more accurate information. Please also let us know in a new issue if you find content that seems to be unlicensed. diff --git a/README.md b/README.md index a0ba921f..b20d9200 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@ @@ -7,35 +7,33 @@ SPDX-License-Identifier: CC-BY-SA-4.0 +![HERMES Key Visual](docs/source/_static/img/header.png) + # hermes -Implementation of the HERMES workflow. +Implementation of the HERMES workflow. For more extensive documentation, see the [HERMES workflow documentation](https://docs.software-metadata.pub/en/latest). -For more information about the HERMES project, see the [HERMES project website](https://software-metadata.pub). +(For more information about the HERMES [HMC](https://helmholtz-metadata.de) *project*, see the [HERMES project website](https://software-metadata.pub).) -## Structure +[![Project Status: WIP – Initial development is in progress, but there has not yet been a stable, usable release suitable for the public.](https://www.repostatus.org/badges/latest/wip.svg)](https://www.repostatus.org/#wip) +[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=hermes-hmc_workflow&metric=alert_status)](https://sonarcloud.io/summary/new_code?id=hermes-hmc_workflow) -- Python sources are in the `src` folder -- pytest tests are in the `test` folder -- [Architectural Design Records (ADR)](https://adr.github.io/) are in `docs/adr` -- API documentation is generated into `docs/source/api` -- All other documentation lives in `docs/source/*` -This project uses +## Installation and Usage -- a development branch (`develop`) to merge developments into, this is the default branch -- actual development is done on "feature" branches (this includes non-feature work such as bug fixing) -- a `main` branch which only includes releases +`hermes`' primary use case is to [use it in a continuous integration environment](https://docs.software-metadata.pub/en/latest/tutorials/001-prepare-your-project.html). -## Set up for development +In case you still want to install on your machine, you can (for example) use `pip`: -1. Clone this repository -2. If you want to use [`poetry`](https://python-poetry.org), run `poetry shell` and `poetry install` - ([Learn how to install `poetry`](https://python-poetry.org/docs/#installation)) - we require using `poetry>=1.2`. +```commandline +pip install git+https://github.com/hermes-hmc/workflow.git +``` -## Usage +- Note: you must have Python 3.10 or newer installed. +- Note: we plan to release stable versions to PyPI (and potentially Github Packages) in the future. The `hermes` application provides the entry point for the HERMES workflow. After installation, you can run it from your command line environment: @@ -45,39 +43,29 @@ hermes --help hermes harvest ``` -You can also call the `hermes` package as Python module: +You can also call the `hermes` package as a Python module: ```shell python -m hermes --help -python -m hermes +python -m hermes harvest ``` -## Testing - -Tests are implemented using [pytest](https://pytest.org). -You can generate coverage report using the `pytest-cov` plugin. -Both tools are specified as development dependencies in the `pyproject.toml`. +## Contributions, Extension and Development -To run tests with an extensive HTML report, run: - -```shell -poetry run pytest test --cov=hermes --cov-branch --cov-report=html --cov-report=term -``` +We welcome external contributions! Please follow our [contribution guidelines](CONTRIBUTING.md). -## Building documentation +HERMES was designed with extensibility in mind. Our [development guide](https://docs.software-metadata.pub/en/latest/dev/start.html) +contains in-depth information on how to get ready and start coding. -This project comes with extensive documentation that can be built using [Sphinx](https://www.sphinx-doc.org/en/master/). -This also includes automatic API documentation. -To build the documentation in your *poetry* environment, run the following commands from the project root: +## Acknowledgements -```shell -poetry install --with docs -poetry run task docs-build -``` +This project (ZT-I-PF-3-006) was funded by the *Initiative and Networking Fund* +of the [Helmholtz Association](https://www.helmholtz.de/en/about-us/structure-and-governance/initiating-and-networking) +in the framework of the [Helmholtz Metadata Collaboration](https://helmholtz-metadaten.de)'s +[2020 project call](https://helmholtz-metadaten.de/en/projects/hmc-projects-2020). -Or use [`sphinx-autobuild`](https://) to enable a self-updating preview service: +## License and Citation -```shell -poetry install --with docs -poetry run task docs-live -``` +Please see [`LICENSE.md`](LICENSE.md) for legal information. +We provide a [`CITATION.cff`](CITATION.cff) containing all metadata for citation, which is also easy to +use via the widget on the right-hand side. diff --git a/docs/source/LICENSES b/docs/source/LICENSES new file mode 120000 index 00000000..06ad40db --- /dev/null +++ b/docs/source/LICENSES @@ -0,0 +1 @@ +../../LICENSES \ No newline at end of file diff --git a/docs/source/_static/img/header.png b/docs/source/_static/img/header.png new file mode 100644 index 00000000..3f849aab Binary files /dev/null and b/docs/source/_static/img/header.png differ diff --git a/docs/source/_static/img/header.png.license b/docs/source/_static/img/header.png.license new file mode 100644 index 00000000..b0dc9b0c --- /dev/null +++ b/docs/source/_static/img/header.png.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2022 German Aerospace Center (DLR), Forschungszentrum Jülich + +SPDX-License-Identifier: CC-BY-ND-4.0 diff --git a/docs/source/_static/img/opengraph-workflow.png.license b/docs/source/_static/img/opengraph-workflow.png.license index f0a3c6c2..b0dc9b0c 100644 --- a/docs/source/_static/img/opengraph-workflow.png.license +++ b/docs/source/_static/img/opengraph-workflow.png.license @@ -1,3 +1,3 @@ SPDX-FileCopyrightText: 2022 German Aerospace Center (DLR), Forschungszentrum Jülich -SPDX-License-Identifier: CC-BY-SA-4.0 +SPDX-License-Identifier: CC-BY-ND-4.0 diff --git a/docs/source/_static/img/workflow-overview.svg b/docs/source/_static/img/workflow-overview.svg new file mode 100644 index 00000000..2f3c70b3 --- /dev/null +++ b/docs/source/_static/img/workflow-overview.svg @@ -0,0 +1,1643 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/source/_static/img/workflow-overview.svg.license b/docs/source/_static/img/workflow-overview.svg.license new file mode 100644 index 00000000..fd20a404 --- /dev/null +++ b/docs/source/_static/img/workflow-overview.svg.license @@ -0,0 +1,3 @@ +SPDX-FileCopyrightText: 2022 German Aerospace Center (DLR), Forschungszentrum Jülich, Helmholtz-Zentrum Dresden-Rossendorf + +SPDX-License-Identifier: CC-BY-SA-4.0 diff --git a/docs/source/conf.py b/docs/source/conf.py index 7f63ec70..1c7a0103 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -76,6 +76,7 @@ 'tasklist', 'deflist', ] +myst_heading_anchors = 4 # Sphinx API docs configuration, see https://sphinx-autoapi.readthedocs.io/en/latest/reference/config.html autoapi_type = "python" diff --git a/docs/source/dev/contribute.md b/docs/source/dev/contribute.md new file mode 100644 index 00000000..1b763d4a --- /dev/null +++ b/docs/source/dev/contribute.md @@ -0,0 +1,12 @@ + + + + +```{include} ../../../CONTRIBUTING.md +``` diff --git a/docs/source/dev/start.md b/docs/source/dev/start.md new file mode 100644 index 00000000..a88cb248 --- /dev/null +++ b/docs/source/dev/start.md @@ -0,0 +1,108 @@ + + + + +# Tutorial: Get started with development + +Follow this tutorial to set up a local copy of the code for development of the workflow itself. +This is not necessary if you only want to use HERMES or want to develop an extension plugin loaded at runtime. + +## Prepare your environment + +First, install Python 3.10 (or later). + +Additionally, you need to [install `poetry >= 1.2.0`](https://python-poetry.org/docs/#installation), either globally, or +within an environment of your choice. As a project, we chose `poetry` to manage our dependencies, builds, and deposits +as a state of the art solution within the Python ecosystem. + +## Get the source code + +Next, you need to obtain a version of the HERMES source code. + +You can either download it as a zipped package or clone the whole Git repository. +You can clone the repository and enter the project directory as follows: + +```shell +git clone https://gitlab.com/hermes-hmc/workflow.git +cd workflow +``` + +## Learn how our repo is structured + +- All Python sources are in the `src` folder +- `pytest` tests are in the `test` folder +- [Architectural Decision Records (ADR)](https://adr.github.io/) are in `docs/adr` +- API documentation is automatically generated into `docs/source/api` +- All other Sphinx-based documentation lives in `docs/source/*` + +This project uses + +- a development branch (`develop`) to merge developments into, this is the default branch +- actual development is done on "feature" branches (this includes non-feature work such as bug fixing), see also our + {doc}`contribute`. +- a `main` branch which only includes releases + +## Install HERMES and dependencies + +`poetry` comes with its own environment management. To create a development environment and install dependencies, run +```shell +# Create an environment dedicated to hermes development +poetry shell +# Install dependencies +poetry install +``` + +### Which dependencies do we use? + +Building a CLI application, we deliberately chose the [Click framework](https://click.palletsprojects.com) to implement +the different workflow parts as commands verbs. + +To learn what other packages `hermes` depends on, have a look at the project configuration file `pyproject.toml` +(in the root of the repository), or use `poetry show --only main`. + +To show dependencies that are only required for active development, or for building documentation run +`poetry show --only dev` and `poetry show --only docs` respectively. + + +## Verify installation works + +That's it, you should now have a working development copy of HERMES in your environment. +You can confirm this by running `hermes --help` to show available commands and options. + +## Verify tests can be run + +Tests are implemented using [pytest](https://pytest.org). + +To run all tests, execute `pytest test/` within the activated `poetry` environment. + +To create an extensive test coverage report in HTML, execute: + +```shell +poetry run pytest test --cov=hermes --cov-branch --cov-report=html --cov-report=term +``` + +## Optional: Verify docs build + +This project comes with extensive documentation that can be built using [Sphinx](https://www.sphinx-doc.org/en/master/). +This also includes automatic API documentation. To build the documentation in your *poetry* environment, run the +following commands from the project root: + +```shell +poetry install --with docs +poetry run task docs-build +``` + +Or use [`sphinx-autobuild`](https://github.com/executablebooks/sphinx-autobuild) to enable a self-updating preview service: + +```shell +poetry install --with docs +poetry run task docs-live +``` + diff --git a/docs/source/index.md b/docs/source/index.md index 5509e203..b0c9efe8 100644 --- a/docs/source/index.md +++ b/docs/source/index.md @@ -1,44 +1,50 @@ + + +![](_static/img/header.png) + # Overview ```{warning} This is a work in progress. Expect disruptive changes. ``` -## README +Research software must be formally published to satisfy FAIR principles and unlock academic credit. Publication +repositories enable this and provide PIDs for software versions, but only through tedious, mostly manual process. -```{include} ../../README.md -``` +The HERMES workflow enables automated publication of rich research software metadata and artifacts to publication +repositories using open source tooling. + +We follow a *push based* model and run in continuous integration (CI) infrastructures integrated in common code platforms +such as GitHub or GitLab to avoid going out of service and overcome limitations of pull-based web services. -## Developer Documentation +Rich descriptive metadata is the key element to useful software publications. We harvest existing metadata from source +code repos and connected platforms, then process, collate and present them for curation, thus preparing software for +automatic submission to publication repositories. + +![](_static/img/workflow-overview.svg) + +## Documentation ```{toctree} :maxdepth: 1 :caption: Developers +dev/contribute +Tutorial: Get started w/ development dev/data_model adr/index -``` - -## API Documentation - -```{toctree} -:maxdepth: 2 -:caption: API api/index ``` -## Indices and tables - -* [](genindex) -* [](modindex) -* [](search) - - ```{toctree} :hidden: :caption: Related @@ -50,11 +56,14 @@ Concept Paper This is an open repository to collect feedback on the HERMES workflow. -We see our project as part of a global and inter-disciplinary effort to improve the state of the art in research software engineering, maintenance and scholarly communications around research software. We therefore appreciate any feedback you may have on the HERMES project itself and any of its outputs. +We see our project as part of a global and inter-disciplinary effort to improve the state of the art in +research software engineering, maintenance and scholarly communications around research software. We therefore +appreciate any feedback you may have on the HERMES project itself and any of its outputs. **How to give feedback** -Either [create an issue](https://github.com/hermes-hmc/workflow/issues/new/choose) in our project repository or [send us an email](mailto:team@software-metadata.pub?subject=HERMES%20WOrkflow%20Reachout). +Either [create an issue](https://github.com/hermes-hmc/workflow/issues/new/choose) in our project repository or +[send us an email](mailto:team@software-metadata.pub?subject=HERMES%20WOrkflow%20Reachout). ## Acknowledgements @@ -63,8 +72,11 @@ of the [Helmholtz Association](https://www.helmholtz.de/en/about-us/structure-an in the framework of the [Helmholtz Metadata Collaboration](https://helmholtz-metadaten.de)'s [2020 project call](https://helmholtz-metadaten.de/en/projects/hmc-projects-2020). -## License +```{include} ../../LICENSE.md +``` -This project documentation is licensed under a [Creative Commons CC0 1.0 Universal (CC0 1.0) Public Domain Dedication](https://creativecommons.org/publicdomain/zero/1.0/legalcode). +## Indices and tables -Our logo / key visual / icon is licensed under [Creative Commons Attribution-NoDerivatives 4.0 International (CC BY-ND 4.0) ](https://creativecommons.org/licenses/by-nd/4.0) +* [](genindex) +* [](modindex) +* [](search) \ No newline at end of file