Adds the ability to detect ion-java performance regression in GitHub workflow #3
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,7 @@ | ||
| # This workflow will build a Java project with Maven | ||
| # For more information see: https://help.github.com/actions/language-and-framework-guides/building-and-testing-java-with-maven | ||
|
|
||
| name: Ion Java performance regression detector | ||
|
|
||
| on: | ||
| push: | ||
|
|
@@ -10,8 +10,8 @@ on: | |
| branches: [ master ] | ||
|
|
||
| jobs: | ||
| detect-regression: | ||
| name: Detect Regression | ||
|
|
||
| runs-on: ubuntu-latest | ||
|
|
||
|
|
@@ -21,52 +21,107 @@ jobs: | |
| with: | ||
| java-version: 1.8 | ||
|
|
||
| - name: Checkout ion-java from the new commit. | ||
| uses: actions/checkout@v2 | ||
| with: | ||
| fetch-depth: 2 | ||
| submodules: recursive | ||
| path: ion-java | ||
|
|
||
| - name: Build ion-java from the new commit | ||
| run: cd ion-java && mvn clean install | ||
|
|
||
| - name: Checkout ion-java-benchmark-cli | ||
| uses: actions/checkout@v2 | ||
| with: | ||
| repository: amzn/ion-java-benchmark-cli | ||
| ref: master | ||
| path: ion-java-benchmark-cli | ||
|
|
||
| - name: Build ion-java-benchmark-cli | ||
| run: cd ion-java-benchmark-cli && mvn clean install | ||
|
|
||
| - name: Check the version of ion-java. | ||
| run: java -jar ion-java-benchmark-cli/target/ion-java-benchmark-cli-0.0.1-SNAPSHOT-jar-with-dependencies.jar --version | ||
|
|
||
| - name: Generate test Ion Data | ||
| run: | | ||
| mkdir -p testData | ||
| java -jar ion-java-benchmark-cli/target/ion-java-benchmark-cli-0.0.1-SNAPSHOT-jar-with-dependencies.jar generate -S 50000 --input-ion-schema ion-java-benchmark-cli/tst/com/amazon/ion/benchmark/testStruct.isl testData/testStruct.10n | ||
| java -jar ion-java-benchmark-cli/target/ion-java-benchmark-cli-0.0.1-SNAPSHOT-jar-with-dependencies.jar generate -S 50000 --input-ion-schema ion-java-benchmark-cli/tst/com/amazon/ion/benchmark/testList.isl testData/testList.10n | ||
| java -jar ion-java-benchmark-cli/target/ion-java-benchmark-cli-0.0.1-SNAPSHOT-jar-with-dependencies.jar generate -S 50000 --input-ion-schema ion-java-benchmark-cli/tst/com/amazon/ion/benchmark/testNestedStruct.isl testData/testNestedStruct.10n | ||
|
|
||
| - name: Upload test Ion Data to artifacts | ||
| uses: actions/upload-artifact@v2 | ||
| with: | ||
| name: test Ion Data | ||
| path: testData | ||
|
|
||
| - name: Benchmark ion-java from the new commit | ||
| run: | | ||
| mkdir -p benchmarkResults | ||
| cd ion-java-benchmark-cli && java -jar target/ion-java-benchmark-cli-0.0.1-SNAPSHOT-jar-with-dependencies.jar run-suite --test-ion-data /home/runner/work/ion-java/ion-java/testData --benchmark-options-combinations tst/com/amazon/ion/benchmark/optionsCombinations.ion /home/runner/work/ion-java/ion-java/benchmarkResults | ||
|
|
||
| - name: Upload benchmark results to artifacts | ||
| uses: actions/upload-artifact@v2 | ||
| with: | ||
| name: Benchmark result | ||
| path: benchmarkResults | ||
|
|
||
| - name: Clean maven dependencies repository | ||
| run : rm -r /home/runner/.m2 | ||
|
Collaborator
There was a problem hiding this comment. This comment is about line72, We only cleaned up the maven dependencies above but not ion-java right? So it might not necessarily checkout the ion-java again. Do you think we could just
Owner
Author
There was a problem hiding this comment.
Yes, that's a good idea, changes will be updated in the following commit. |
||
|
|
||
| - name: Build ion-java from the previous commit | ||
| run: cd ion-java && git checkout HEAD^ && mvn clean install | ||
|
|
||
| - name: Build ion-java-benchmark-cli | ||
| run: cd ion-java-benchmark-cli && mvn clean install | ||
|
|
||
| - name: Check the version of ion-java | ||
| run: java -jar ion-java-benchmark-cli/target/ion-java-benchmark-cli-0.0.1-SNAPSHOT-jar-with-dependencies.jar --version | ||
|
|
||
| - name: Create directories for test data and benchmark results | ||
| run: | | ||
| mkdir -p benchmarkResults | ||
| mkdir -p testData | ||
|
|
||
| - name: Download test Ion Data from artifacts | ||
| uses: actions/download-artifact@v2 | ||
| with: | ||
| name: test Ion Data | ||
| path: testData | ||
|
|
||
| - name: Download benchmark results of ion-java from the new commit from artifacts | ||
| uses: actions/download-artifact@v2 | ||
| with: | ||
| name: Benchmark result | ||
| path: benchmarkResults | ||
|
|
||
| - name: Benchmark ion-java from the previous commit and add the generated benchmark results to the existing directories | ||
| run: cd ion-java-benchmark-cli && java -jar target/ion-java-benchmark-cli-0.0.1-SNAPSHOT-jar-with-dependencies.jar run-suite --test-ion-data /home/runner/work/ion-java/ion-java/testData --benchmark-options-combinations tst/com/amazon/ion/benchmark/optionsCombinations.ion /home/runner/work/ion-java/ion-java/benchmarkResults | ||
|
|
||
| - name: Upload new benchmark results directory to artifacts | ||
| uses: actions/upload-artifact@v2 | ||
| with: | ||
| name: Benchmark result | ||
| path: benchmarkResults | ||
|
|
||
| - name: Detect performance regression | ||
| id: regression_result | ||
| run: | | ||
| result=true | ||
| cd benchmarkResults && for FILE in *; do message=$(java -jar /home/runner/work/ion-java/ion-java/ion-java-benchmark-cli/target/ion-java-benchmark-cli-0.0.1-SNAPSHOT-jar-with-dependencies.jar compare --benchmark-result-previous $FILE/previous.ion --benchmark-result-new $FILE/new.ion $FILE/report.ion | tee /dev/stderr) && if [ "$message" != "" ]; then result=false; fi; done | ||
| echo "::set-output name=regression-result::$result" | ||
|
Collaborator
There was a problem hiding this comment. Since we used a for loop to iterate all files, is it possible to know which file(s) fail the workflow?
Owner
Author
There was a problem hiding this comment.
Yes, the output message will contain these information, i.e. Including the test data and parameters we use which can help users to map with the benchmark results directory. |
||
| echo $result | ||
|
|
||
| - name: Upload comparison reports to the benchmark results directory | ||
| uses: actions/upload-artifact@v2 | ||
| with: | ||
| name: Benchmark result | ||
| path: benchmarkResults | ||
|
|
||
| - name: Fail the workflow if regression happened | ||
| env: | ||
| regression_detect: ${{steps.regression_result.outputs.regression-result}} | ||
| if: ${{ env.regression_detect == 'false' }} | ||
| run: exit 1 | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Quick questions since there is a
-poption here. Is it possible that the directory (e.g., testData) has already existed so we used-pto avoid failures here?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, because it's the start of the workflow and no existing directory, even there is the "-p" option can help to avoid failure.