Logging: Verbose debug logging in server log happening on some machines post 4.10.1 release

[kcondon]

Several developer machines and at least one test box are showing verbose debug logging in the server log, for example when loading the homepage or where dataset cards are rendered. It appears in other places too but we have not enumerated them. They seem to be lower-level HTTP messages and message within message.

Testing indicates they may have started appearing with this merge:
#5484 5483-netbeans-code-coverage

Here are some examples:

[2019-02-13T12:14:58.355-0500] [glassfish 4.1] [INFO] [] [] [tid: _ThreadID=83 _ThreadName=Thread-8] [timeMillis: 1550078098355] [levelValue: 800] [[
12:14:58.355 [http-listener-1(51)] DEBUG org.apache.http.wire - http-outgoing-0 << "Fralin_Greek_Vases_20151120-kantharos.stl[0xff]"?[\n]"]]

[2019-02-13T12:14:58.354-0500] [glassfish 4.1] [INFO] [] [] [tid: _ThreadID=83 _ThreadName=Thread-8] [timeMillis: 1550078098354] [levelValue: 800] [[
12:14:58.354 [http-listener-1(51)] DEBUG o.a.h.impl.execchain.MainClientExec - Connection can be kept alive indefinitely]]

[2019-02-13T12:14:58.222-0500] [glassfish 4.1] [INFO] [] [] [tid: _ThreadID=83 _ThreadName=Thread-8] [timeMillis: 1550078098222] [levelValue: 800] [[
12:14:58.222 [http-listener-1(51)] DEBUG o.a.h.i.c.DefaultManagedHttpClientConnection - http-outgoing-0: set socket timeout to 600000]]

@qqmyers Any thoughts on this?

[kcondon]

It looks like Apache httpClient logging is turned on in some way?
https://hc.apache.org/
https://hc.apache.org/httpcomponents-client-ga/logging.html
https://stackoverflow.com/questions/4915414/disable-httpclient-logging

[matthew-a-dunlap]

I'm snooping through our closed PRs now to see what may be indicative of this issue.

[qqmyers]

@kcondon - I see this on one of our 3 machines at QDR and not the other two, even running the same code. I've looked through our config and haven't been able to find out why its occurring. I recall trying to look at wire logging during development but didn't get it to work - so it's possible I added and forgot to remove some code or a jar via maven that didn't work for me but is popping up on some machines(some load order aspect?)

I think a key clue is that the apache debug statements are getting wrapped in a glassfish info-level log entry. I don't know what that means except that the log isn't directly allowing debug level entries from apache...

[matthew-a-dunlap]

Things I've found:

• Reverting JaCoCo & SolrJ to their previous versions did not fix anything
  • Both machines are using Solr 7.3.0 alongside the SolrJ 7.3.1 library
• My linux vm is not spitting out either of these log messages, yet my OSX box is.
• Building and deploying the 4.10.1 release made the messages go away
• Reverting each pom change manually back to 4.10.1 release did not make the message go away
  • I think I missed the main duracloud change though

[pdurbin]

While brainstorming for solutions during standup we touched on the idea of simply reducing the logging. As I was saying, there is precedence for this. As part of #643 we ultimately added the following to http://guides.dataverse.org/en/4.10.1/installation/shibboleth.html#sslengine-warning-workaround

Obviously, any other fix would be better than this.

[matthew-a-dunlap]

I'm pretty sure I've identified a factor in this issue, it being related to #4706 OAI-ORE and BagIT development. I removed the the pom dependencies to duracloud and deleted the DuraCloudSubmitToArchiveCommand.java for the application to compile. Doing this the log messages disappeared when loading the search page.

I'm not really sure what is actually going on though, maybe duracloud is pulling in some library that maven is choosing that has these log messages on?

[matthew-a-dunlap]

I would like to know what is actually being logged, even if we quiet it. Is our network traffic being piped into this library?