diff --git a/dev-support/HOW_TO_YETUS_LOCAL.md b/dev-support/HOW_TO_YETUS_LOCAL.md new file mode 100644 index 000000000000..0183424efdc4 --- /dev/null +++ b/dev-support/HOW_TO_YETUS_LOCAL.md @@ -0,0 +1,166 @@ +# How To Use Apache Yetus with Apache HBase + +This document describes how to get up and running with [Apache Yetus][yetus], +as pertains to the development of Apache HBase. Specifically this covers the use +of `test-patch`, of which HBase developers make use for tasks related to code +quality. These are normally run via automation in the foundation's +[Build][builds] infrastructure. They can also be run locally, which is the +subject of this document. + +The Yetus project provides its own documentation of `test-patch` in +[Basic PreCommit][yetus-basic-precommit]. By comparison, this document is +intended to be highly abbreviated, hands-on, and focused on the HBase use-case. +See that document for more complete explanations and further details. + +## Installation + +In order to run Yetus, you'll need to first install Yetus and its dependencies. +This is somewhat simplified when used in Docker mode. Yetus can be retrieved +from a [distribution artifact][yetus-downloads]. Homebrew/Linuxbrew users can +install from the tap, the process for which is also described on the downloads +page. + +# Usage Basics + +Apache Yetus is comprised of a number of different components. The focus of our +interest is `test-patch`. `test-patch` is a modular system. Many modules depend +on some external tool to provide the underlying functionality. For example, the +`compile` check delegates to a number of provider modules, for example `maven` +or `gradle` for JVM projects. In order to use these modules, those tools must +be installed. Yetus calls these modules "plugins". + +To see a list of all plugins available to `test-patch`, use + +```shell script +$ test-patch --list-plugins +``` + +To see a list of all the options available in both the core modules as well as +all the plugins, use + +```shell script +$ test-patch --plugins=all --help +``` + +An invocation of `test-patch` requires use of one or more plugins. Often times, +when the full suite of checks are run, specify the meta-plugin "all". Otherwise, +a limited selection of plugins can be selected using the `--plugins` argument. + +## The Workspace, The Patch + +`test-patch` operates within a "workspace," a checkout from a source control +repository. It has a number of options pertaining to this workspace, such as +the path to the workspace (`--basedir`) and whether it will permit the presence +of uncommitted changes therein (`--dirty-workspace`). + +Onto this workspace, it can optionally apply a change, a.k.a., the "patch" in +"test patch." The patch can come from a number of sources, including a patch +file, a JIRA ID, a Pull Request, &c. Or, explicitly inform Yetus that no patch +file is provided, and the repository should be checked as is, by passing +`--empty-patch`. + +## Personalities + +`test-patch` is extremely extensible. Even the functionality of its core +modules can be extended or overridden. It allows for this type of +"personalization" by way of "personality" overrides. Yetus ships a number of +these personalities; a pre-packaged personality can be selected via the +`--project` parameter. There is a provided HBase personality in Yetus, however +the HBase project maintains its own within the HBase source repository. Specify +the path to the personality file using `--personality`. The HBase repository +places this file under `dev-support/hbase-personality.sh`. + +## Docker mode + +Running Yetus in Docker mode simplifies the concerns of dependencies because +the provided `Dockerfile` handles their installation automatically. However, +for MacOS users, there are a number of known issues with running Docker on OSX, +so it may be preferable to instead run outside of the container. + +To run in Docker, of course you must install `docker` or some container runtime +equivalent. [Docker Desktop][docker-desktop] is an option for Mac and Windows +users. + +When running `test-patch` with HBase in docker mode, two parameters must be +provided. `--docker` activates the `docker` module, enabling the feature. +Additionally, `--dockerfile` points Yetus at HBase's provided `Dockerfile`, +instead of using the one provided out-of-the-box by Yetus. The HBase repository +places this file under `dev-support/docker/Dockerfile`. + +## JVM and JAVA_HOME + +HBase supports compilation and test execution on a number of different JVM +versions. To facilitate this, the `Dockerfile` installs multiple JVMs. An +invocation of `test-patch` requires additional parameters in order to specify +the target JVM. How you do this is slightly different based on whether you run +with or without Docker mode. + +### Setting `JAVA_HOME` Outside of Docker Mode + +Simply specify `JAVA_HOME` in the environment in which `test-patch` is launched. + +### Setting `JAVA_HOME` Inside of Docker Mode + +The docker image JDKs are available under `/usr/lib/jvm`. Symlinks are provided +for each supported major version, i.e., `java-11`. Use the `test-patch` argument +`--java-home`. + +# Example: Run SpotBugs + +One of the checks supported by Yetus is the static analysis tool +[SpotBugs][spotbugs]. Let's put together all the above and run SpotBugs with +`test-patch`. + +The SpotBugs check depends on a SpotBugs installation. This is provided by the +docker file. For running without docker, you'll need to download and unpack the +SpotBugs binary distribution. + +Running the SpotBugs check also depends on [Apache Maven][maven], the build +system used by HBase. In order for the check to function, both the `maven` +and `spotbugs` plugins must be specified. If this dependency is omitted, Yetus +will error with a message similar to + +``` +ERROR: you can't specify maven as the buildtool if you don't enable the plugin. +``` + +To run just the SpotBugs check, we must explicitly specify `spotbugs` and its +dependency using `--plugins=maven,spotbugs`. + +## Without Docker + +Putting it all together, without using docker: + +```shell script +$ cd /path/to/hbase +$ JAVA_HOME=/path/to/jdk-8 test-patch.sh \ + --plugins=maven,spotbugs \ + --spotbugs-home=/path/to/spotbugs/ + --dirty-workspace \ + --empty-patch \ + --personality=./dev-support/hbase-personality.sh +``` + +## With Docker + +Putting it all together, using docker: + +```shell script +$ cd /path/to/hbase +$ test-patch.sh \ + --plugins=maven,spotbugs \ + --dirty-workspace \ + --empty-patch \ + --personality=./dev-support/hbase-personality.sh \ + --docker \ + --dockerfile=./dev-support/docker/Dockerfile \ + --java-home=/usr/lib/jvm/java-8 +``` + +[builds]: https://builds.apache.org +[docker-desktop]: https://www.docker.com/products/docker-desktop +[maven]: https://maven.apache.org +[spotbugs]: https://spotbugs.github.io +[yetus]: https://yetus.apache.org +[yetus-basic-precommit]: https://yetus.apache.org/documentation/0.11.1/precommit-basic/ +[yetus-downloads]: https://yetus.apache.org/downloads/