WIP custom ant task to produce html report from Rat report and NetBeans clusters by ebarboni · Pull Request #70 · apache/netbeans

ebarboni · 2017-10-06T15:10:06Z

Hi,
this is a more deep integration with Netbeans of the rat report.

It filter the report using cluster.properties and check for local folder.

The filtering pattern is now a variable to get same result for txt and html report to keep the current txt report intact.

Regards

matthiasblaesing · 2017-10-06T16:26:05Z

Could you please have a look at the diff of build.xml? It looks as if it got a complete make over.

matthiasblaesing · 2017-10-06T16:27:09Z

And it should be noted, that merging this, will cause merge conflicts for anyone adding exclusions to the rat report.

ebarboni · 2017-10-06T22:16:17Z

build.xml revised. I do a full file xml formating within Netbeans, that was not a good idea. I do a partial formatting and add the exclusion. If merged, the ant task may be updated separatly later. Build.xml should not be impacted any more.

JaroslavTulach · 2017-10-07T01:03:11Z

nbbuild/antsrc/org/netbeans/nbbuild/reporting/ModuleInfo.java

+ * @author skygo
+ */
+class ModuleInfo {
+// approuved resource list


I guess this is a typo: "approved".

JaroslavTulach · 2017-10-07T01:04:40Z

nbbuild/antsrc/org/netbeans/nbbuild/reporting/RatHtmlReportTask.java

+                ModuleInfo get = unbymodule.get(mm);
+                for (String mmu : get.getUnapprouved()) {
+                    bw.write("<li class=\"unapprouved\">");
+                    bw.write("U:<a href=\"https://github.com/apache/incubator-netbeans/tree/master/" + mmu + "\">" + mmu + "</a>");


Hardcoding repository URL is probably not good idea.

Either make the URL an attribute of the task or (preferrably) obtain it from the .git/config file. There already are places where the build uses hg command to obtain the commit id (that should be changed to git one day), so precedent of using the repo information from builds exists.

nice addition thanks. will look at it

JaroslavTulach · 2017-10-07T01:13:33Z

nbbuild/build.xml

        </rat:report>
+        <taskdef name="rattohtml" classname="org.netbeans.nbbuild.reporting.RatHtmlReportTask" classpath="${build.ant.classes.dir}"/>
+        <loadproperties srcFile="cluster.properties" />
+        <rattohtml source="build/rat-report.xml" target="build/rat-report.html"/>


I am not sure whether HTML is that much better format than txt. Probably the direct hyperlinks to the Git repo are the biggest improvement, right?

In any case I was looking for a way to track progress of our rat related efforts. Instead of HTML the task should generate JUnit XML report files. If the output used JUnitReportWritter just like the other classes in the nbbuild/antsrc, we could set the continuous job to show us how many violations we still have and track the progress.

Sorry but the last commit with refactoring kill the converstation flow.

Maybe hyperlink is for now the only improvement for having HTML report. By cluster information can be done in txt file.

Helping tooling continuous job is a big point. But we should also think on people who will review and vote for release.
for example http://commons.apache.org/proper/commons-bcel/ has a project information section including a rat report.

JaroslavTulach · 2017-10-07T01:15:24Z

nbbuild/antsrc/org/netbeans/nbbuild/reporting/RatHtmlReportTask.java

+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.netbeans.nbbuild.reporting;


Does this really deserve a separate package? There is a lot of reporting tasks already present in org.netbeans.nbbuild... Moving into org.netbeans.nbbuild and merging ModuleInfo to be static inner class would, in my opinion, be more consistent.

This sounds good for me

JaroslavTulach · 2017-10-09T16:19:44Z

nbbuild/antsrc/org/netbeans/nbbuild/RatHtmlReportTask.java

+                bw.write("<h3>Module::" + mm + "</h3>");
+                bw.write("<ul>");
+                ModuleInfo get = unbymodule.get(mm);
+                for (String mmu : get.getUnapprouved()) {


approuved typos are everywhere...

ebarboni · 2017-10-10T16:33:49Z

I had to check on personal jenkins to see what I get from junit report. I hope the mutliple commit and rebase were not notified.

The report are by clusters in a rat folder in case we want to filter them later.

nbbuild/build/rat/**
by putting url in the report jenkins build hyperlink which make it equivalent of a html report that is good enough and make pure html maybe useless for development point of view.

ebarboni · 2017-10-11T16:32:26Z

@JaroslavTulach I think the WIP is in now in a stable state.
Unit report
are done for unaprouverd license
external library with no maven pattern

rebasing breaks in place review sorry for that :/
regards

jtulach

Once integrated we have to use this in our Jenkins jobs. I can administered them...

jtulach · 2017-10-11T18:18:45Z

nbbuild/antsrc/org/netbeans/nbbuild/RatReportTask.java

+    }
+
+    @Override
+    public void execute() throws BuildException {


Super.execute is probably useless.

jtulach · 2017-10-11T18:19:49Z

nbbuild/antsrc/org/netbeans/nbbuild/RatReportTask.java

+            commandAndArgs.add("config");
+            commandAndArgs.add("--get");
+            commandAndArgs.add("remote.origin.url");
+            Process p = new ProcessBuilder(commandAndArgs).directory(root).start();


R is never closed

jtulach · 2017-10-11T18:21:47Z

nbbuild/antsrc/org/netbeans/nbbuild/RatReportTask.java

+                    // XXx if amoduleInfo is null folder is not present source base incomplete 
+                }
+
+            }


ebarboni · 2017-10-11T20:12:48Z

@jtulach rebased again but rat task have more than only fileset from a PR by @matthiasblaesing

get git information from git process add scan for external

ebarboni · 2017-10-12T16:18:38Z

@jtulach or @JaroslavTulach , rebasing of build.xml is annoying.
I you prefer I can do a PR only for the ant task will let you put the configuration in build.xml later on ?
Let me know.

JaroslavTulach · 2017-10-12T17:20:27Z

#1 - I don't understand why rebase at all.
#2 - I've already approved your changes. Feel free to merge.

ebarboni · 2017-10-12T21:10:33Z

@JaroslavTulach
#1 I want to merge the exclude from license review otherwise it's conflicting
#2 I'm not commiter, I can only propose PR. (Commiter on other Apache Archiva project)

JaroslavTulach · 2017-10-13T15:02:09Z

OK, I try to merge.

JaroslavTulach · 2017-10-13T15:19:58Z

Integrated as 162850d - builds modified to pick the generated .xml files up.

JaroslavTulach · 2017-10-13T18:07:31Z

The build is failing: https://builds.apache.org/job/incubator-netbeans-linux/114/console
I am not sure why right now.

ebarboni · 2017-10-13T18:54:17Z

Strange,
I wipe and recreate a build base on apache/incubator-netbeans, on a private jenkins (it works). A difference I notice is that encoding on the apache H21,h24 node are iso while my jenkins os or docker image are UTF-8

ebarboni · 2017-10-13T22:38:23Z

parsing xml with file seems to be dangerous for encoding. #144 use InputSource. Seems to fix the issue on my new non UTF-8 image.

matthiasblaesing · 2017-10-14T09:02:25Z

I think the problem is in rat - with an output file this is what happens:

the Report Task creates an FileWriter that writes to the target file and wraps a PrintWriter to it (Report Task, line 196)
this PrintWriter is passed to the core Report report method, where an XmlWriter is created (Report Task, line 268, Report Class, line 411)
The XML prolog is written in org.apache.rat.report.xml.writer.impl.base.XmlWriter#startDocument XmlWriter

            if (reportFile == null) {
                out = new PrintWriter(
                          new OutputStreamWriter(
                              new LogOutputStream(this, Project.MSG_INFO)
                              )
                          );
            } else {
                out = new PrintWriter(new FileWriter(reportFile));
            }
            createReport(o

The prolog written in step 3 would be wrong in many cases as it does not write an encoding info. This means the encoding of the resulting document must be UTF-8. The FileWriter created in step 1 uses the default charset and if that is not UTF-8 a broken stream will result.

The Report Task would need to be changed from (line 196):

out = new PrintWriter(new FileWriter(reportFile));

to

out = new PrintWriter(new OutputStreamWriter(new FileOutputStream(reportFile), Charset.forName("UTF-8")));

JaroslavTulach reviewed Oct 7, 2017

View reviewed changes

JaroslavTulach reviewed Oct 9, 2017

View reviewed changes

jtulach approved these changes Oct 11, 2017

View reviewed changes

Produce junit report from Apache Rat and clusters.properties

d1d083e

get git information from git process add scan for external

ebarboni mentioned this pull request Oct 13, 2017

Use InputSource to parse xml avoid encoding issue #144

Closed

ebarboni closed this Oct 16, 2017

ebarboni deleted the ratui branch October 16, 2017 08:43

MidnightStallion mentioned this pull request Jul 17, 2024

Attempting to Add a Dependency Locks up NetBeans #7588

Open

Conversation

ebarboni commented Oct 6, 2017

Uh oh!

matthiasblaesing commented Oct 6, 2017

Uh oh!

matthiasblaesing commented Oct 6, 2017

Uh oh!

ebarboni commented Oct 6, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JaroslavTulach Oct 7, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JaroslavTulach Oct 7, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ebarboni commented Oct 10, 2017

Uh oh!

ebarboni commented Oct 11, 2017

Uh oh!

jtulach left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ebarboni commented Oct 11, 2017

Uh oh!

ebarboni commented Oct 12, 2017

Uh oh!

JaroslavTulach commented Oct 12, 2017

Uh oh!

ebarboni commented Oct 12, 2017

Uh oh!

JaroslavTulach commented Oct 13, 2017

Uh oh!

JaroslavTulach commented Oct 13, 2017

Uh oh!

JaroslavTulach commented Oct 13, 2017

Uh oh!

ebarboni commented Oct 13, 2017

Uh oh!

ebarboni commented Oct 13, 2017

Uh oh!

matthiasblaesing commented Oct 14, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

JaroslavTulach Oct 7, 2017 •

edited

Loading

JaroslavTulach Oct 7, 2017 •

edited

Loading