Binary license management system by jihoonson · Pull Request #7998 · apache/druid

jihoonson · 2019-06-29T23:45:35Z

Motivation

We are currently maintaining LICENSE.BINARY file manually. The way we do now is

Generating dependency reports using maven and npm
Compare the licenses in LICENSE.BINARY against generated reports
Fix LICENSE.BINARY if something is wrong

This is a huge burden for the release manager since we have tons of binary dependency. We should automate this check.

Description

This PR adds a binary license management system. All binary licenses are maintained in license.yaml file. Here is a snippet of the file.

name: modified portions of the Guava library
license_category: source
license_name: Apache License version 2.0
module: java-core
copyright: The Guava Authors (https://github.com/google/guava)
source_paths:
  - Closer class: core/src/main/java/org/apache/druid/java/util/common/io/Closer.java
  - Splitter.splitToList() method: core/src/main/java/org/apache/druid/java/util/common/parsers/DelimitedParser.java
  - DirectExecutorService class: core/src/main/java/org/apache/druid/java/util/common/concurrent/DirectExecutorService.java

---

name: Checker Qual
license_category: binary
module: java-core
license_name: MIT License
version: 2.5.7
copyright: the Checker Framework developers
license_file_path: licenses/bin/checker-qual.MIT
libraries:
  - org.checkerframework: checker-qual

Note that skip_dependency_report_check is set to true for some licenses. For FindBugs JSR305, it's set to true since maven dependency plugin reports a wrong license (LGPL). Note that FindBugs is licensed under LGPL but JSR305 is licensed under BSD-3 license. This script also skips dependency report check for "Java Concurrency In Practice" Book Annotations since it points to a book and doesn't have a valid library version.

docs/_bin/generate-license.py reads this file, checks the registered licenses with maven dependency report, and generates the contents of LICENSE.BINARY file. You can run this program as below:

$ docs/_bin/generate-license-dependency-reports.py /path/to/druid/source/code /path/to/root/of/license-reports/
$ docs/_bin/generate-license.py licenses/APACHE2 license.yaml /path/to/root/of/license-reports/ LICENSE.BINARY

The generated LICENSE.BINARY file is a little bit different from the current one, mostly because of the difficulty of auto generation.

~~I will adjust distribution/pom.xml to generate LICENSE.BINARY when building binary distribution in a follow-up PR. I will also set up Travis to run license check automatically.~~
I have adjusted distribution/pom.xml to generate LICENSE.BINARY when building binary distribution. This is currently being checked automatically in Travis.

clintropolis · 2019-06-30T01:04:37Z

Heh, travis rat check failing for license.yaml https://travis-ci.org/apache/incubator-druid/jobs/552284036#L644

clintropolis

very nice.

For additional future improvement stuff, I think we should also probably add something to contributing guide docs and to the PR template about maintaining license.yaml, mentioning that if you add a new dependency or change a version of a dependency to update it accordingly. There is probably room for automating this part as well to some extent.

clintropolis · 2019-06-30T01:31:07Z

+from html.parser import HTMLParser
+
+
+apache_license_v2 = "\n\


This is a pretty big string, should we just keep this in an external file somewhere (or do we already have it somewhere maybe) and consider just loading it from that?

Extracted as licenses/APACHE2.

clintropolis · 2019-06-30T01:31:28Z

+
+
+class DependencyReportParser(HTMLParser):
+    # TODO: Change to comments


What does this mean?

Ah this was resolved. Removed now.

clintropolis · 2019-06-30T01:33:17Z

+                print("      * {}:{}".format(group_id, artifact_id))
+
+
+def check_licenses(license_yaml, dependencie_reports_root):


dependencie_reports_root -> dependencies_reports_root

Thanks, fixed.

clintropolis · 2019-06-30T01:34:39Z

+
+    print_error("Found {} reported licenses".format(len(reported_dep_to_licenses)))
+
+    # Compare licenses in registry and those in dependency reports


could you add a few additional comments about what is going on here to break up all these loops? makes it kind of rough to follow along with what is going on here

Added more comments here and other places. Please let me know if it's still unclear.

clintropolis · 2019-06-30T01:42:21Z

The generated LICENSE.BINARY file is a little bit different from the current one, mostly because of the difficulty of auto generation.

To clarify, are the contents of the generated LICENSE.BINARY currently the same, just not the same format?

jihoonson · 2019-06-30T04:09:41Z

To clarify, are the contents of the generated LICENSE.BINARY currently the same, just not the same format?

Yes, the contents should be same. Only the format is slightly different.

Fokko · 2019-07-01T06:58:30Z

Great work @jihoonson. Thanks for picking this up.

I fully agree with you; if it isn't checked by the CI, then it is hard to maintain. Maybe it is a good idea to add the lines on how to run the checker to the CONTRIBUTING.md file for further reference.

One final note on the findbugs dependency. I'm looking for replacing this one with spotbugs: https://mvnrepository.com/artifact/com.github.spotbugs/spotbugs-annotations/3.1.12
But it looks like this one has the same issue with the wrong license. They are working on removing the JSR305 component: spotbugs/spotbugs#421 but this will be tricky as we might expect.

clintropolis

👍 from me after you get CI fixed

jihoonson · 2019-07-02T01:04:30Z

@Fokko thanks for taking a look!

Maybe it is a good idea to add the lines on how to run the checker to the CONTRIBUTING.md file for further reference.

It sounds good. I think it would be nicer if we add that to our PullRequest template as well.

One final note on the findbugs dependency. I'm looking for replacing this one with spotbugs: https://mvnrepository.com/artifact/com.github.spotbugs/spotbugs-annotations/3.1.12
But it looks like this one has the same issue with the wrong license. They are working on removing the JSR305 component: spotbugs/spotbugs#421 but this will be tricky as we might expect.

What do you mean by "wrong license"? Spotbugs is also licensed under LGPL, so we shouldn't include it in our binary distribution unless the part you want to use it is particularly licensed under a compatible license.

jihoonson · 2019-07-06T06:01:20Z

Ok, now Travis looks stable.

jihoonson · 2019-07-08T20:52:11Z

Hmm, looks like Travis is currently getting stuck at generate-dependency-reports phase. Looking into this.

jihoonson added 3 commits June 29, 2019 16:10

Binary license management system

37cddcc

add missing file

3f91d61

add comment

ade4847

jihoonson added the Apache Items related to being a part of the ASF label Jun 29, 2019

clintropolis reviewed Jun 30, 2019

View reviewed changes

Address comments

49869aa

jihoonson added the WIP label Jun 30, 2019

jihoonson added 3 commits June 30, 2019 08:29

print missing licenses

9f56964

print druid module name

ec40876

Add missing licenses and update versions

e831d37

fix library versions and add missing ones. also fix pom.xml

d984ec9

clintropolis approved these changes Jul 1, 2019

View reviewed changes

jihoonson added 2 commits July 1, 2019 14:32

testing multi thread

bc98daa

Parallel report generation

9109329

fix build error

22a39f5

jihoonson mentioned this pull request Jul 2, 2019

Web console: update dependencies #8007

Merged

jihoonson added 2 commits July 1, 2019 20:38

install pyyaml and use old api

050af7c

install python3

ff22126

jihoonson added 19 commits July 5, 2019 12:29

debug

1e984a5

remove debug code

363909d

test build script

f814ee5

travis debug

a6c482a

still debug

9dbf207

add missing python lib

b8c7521

debug

8bfe079

debug

ad5a68a

fix travis

98f090f

fix travis

f36b08e

debug travis

47d8d38

flush print

8652ecb

print something more to keep travis alive

42d9df8

adjust print

0e324bc

single threaded

3e8eadd

single threaded

cf78bf2

debug

bda2d90

debug

dd524c4

remove debug

1a755cd

jihoonson removed the WIP label Jul 6, 2019

jihoonson added 2 commits July 7, 2019 00:04

remove deprecated-2017Q4 from travis conf

a6c4c8e

remove comments and duplicate sudo

a94b160

clintropolis approved these changes Jul 8, 2019

View reviewed changes

jihoonson merged commit 12f1267 into apache:master Jul 8, 2019

clintropolis mentioned this pull request Jul 9, 2019

add config to optionally disable all compression in intermediate segment persists while ingestion #7919

Merged

gianm mentioned this pull request Jul 9, 2019

SupervisorManager: Add authorization checks to bulk endpoints. #8044

Merged

clintropolis added this to the 0.16.0 milestone Aug 8, 2019

clintropolis mentioned this pull request Aug 22, 2019

Zookeeper version is updated. #8363

Merged

7 tasks



		class DependencyReportParser(HTMLParser):
		# TODO: Change to comments

		print(" * {}:{}".format(group_id, artifact_id))


		def check_licenses(license_yaml, dependencie_reports_root):


		print_error("Found {} reported licenses".format(len(reported_dep_to_licenses)))

		# Compare licenses in registry and those in dependency reports

Conversation

jihoonson commented Jun 29, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Description

Uh oh!

clintropolis commented Jun 30, 2019

Uh oh!

clintropolis left a comment

Choose a reason for hiding this comment

Uh oh!

clintropolis Jun 30, 2019

Choose a reason for hiding this comment

Uh oh!

jihoonson Jun 30, 2019

Choose a reason for hiding this comment

Uh oh!

clintropolis Jun 30, 2019

Choose a reason for hiding this comment

Uh oh!

jihoonson Jun 30, 2019

Choose a reason for hiding this comment

Uh oh!

clintropolis Jun 30, 2019

Choose a reason for hiding this comment

Uh oh!

jihoonson Jun 30, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clintropolis Jun 30, 2019

Choose a reason for hiding this comment

Uh oh!

jihoonson Jun 30, 2019

Choose a reason for hiding this comment

Uh oh!

clintropolis commented Jun 30, 2019

Uh oh!

jihoonson commented Jun 30, 2019

Uh oh!

Fokko commented Jul 1, 2019

Uh oh!

clintropolis left a comment

Choose a reason for hiding this comment

Uh oh!

jihoonson commented Jul 2, 2019

Uh oh!

jihoonson commented Jul 6, 2019

Uh oh!

jihoonson commented Jul 8, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jihoonson commented Jun 29, 2019 •

edited

Loading

jihoonson Jun 30, 2019 •

edited

Loading