[MRESOLVER-308] HTTP transport showdown

[jira-importer]

Tamas Cservenak opened MRESOLVER-308 and commented

For HTTP protocol resolver currently provides following transports:

• transport-wagon that uses Maven 2.x Wagon, that among other protocols supports HTTP/1.1 as well
• transport-http that uses directly Apache HttpClient 4.x supporting HTTP/1.1 but provides enhancements in form of "checksum strategy" (almost all of remote repositories emit those in headers, sparing one HTTP round-trip)

As we saw, is very easy to outperform these as:

• Maven Central supports HTTP/1.1 but also HTTP/2
• HTTP/3 is on the way as well

An experiment involving Jetty Client far outperformed both of existing transports, most probably due HTTP/2 support.

So, clients we should consider:

• Jetty Client
• OkHTTP
• Java 11 HttpClient

Point is, to invest into something that (ideally) transparently supports HTTP/1.1 and HTTP/2, and more ideal would be if even HTTP/3 would be transparently supported (Jetty 12 works on that). We could then simply compare these implementations, count in pros and cons, and decide where we want to go,

---

Issue Links:

• MRESOLVER-302 Introduce onSession close
  ("is blocked by")
• MRESOLVER-384 Support HTTP/2 in maven-resolver-transport-http
  ("is duplicated by")

Remote Links:

• GitHub Pull Request #231
  

Backported to: 2.0.0-alpha-2

[jira-importer]

Tamas Cservenak commented

Test: building maven-resolver directly against central (supports h1 and h2) with empty local repo

Cmd: `mvn clean install -Dmaven.repo.local=local -Drat.skip -DskipTests` (plus transport swaps)

Java: 17.0.5 Temurin

Maven: 3.9.0-SNAPSHOT

wagon  h1      01:22  slowest
native h1+h+r  00:51  good but only H1
jetty  h2+h+r  00:52  good, but huge dep trail, Jetty 10.0.13
okhttp h2+h+r  00:55  slowest H2, brings in kotlin (relatively small dep trail: 3 libs) OkHttp 4.10.0
java11 h2+h+r  00:49  clear winner (if we assume users run Maven on latest Java)

Legend:
h1 = HTTP/1.1
h2 = HTTP/2
h = extracts checksums from response headers
r = resumable download (can continue download if download was interrupted)

Note: okhttp and jetty client implementations are suboptimal, while java11 is not fully complete.
Planned to remove DAV support, today there is no need for it, but who uses DAV servers to deploy, should use "native". 

Tested sources https://github.com/cstamas/maven-resolver/tree/http2-transports

Note: Jetty surprised me as last time I measured something completely different (see here jetty/jetty.project#8974 (comment)) but back then it was Jetty 10.0.12 and today I use 10.0.13 and also this was against Central while that issues is about Google Central mirror, etc... but totally different numbers, And at the end, jetty transport code got changed quite a bit (and is sub-optimal).

[jira-importer]

Tamas Cservenak commented

Plan: create -multi release- JAR for maven-resolver-transport-java11, that is:

• on Java8 is doermant
• on Java11+ becomes default (instead of "native")

It works without multi release jar, as it raises module release to Java11, and on Java8 SISU will (silently) skip components from this JAR, so basically this happens: if on Java8, components are not discovered, nothing changes, while on Java11 components are discovered and takes over.

Update: This is DONE on branch: just build it (skip tests for now, some red herrings still not ironed out), and you will end up with java11 transport JAR that can be thrown into lib or lib/ext (does not matter). And from that moment on, if Maven used on Java8, everything is as today, but if on Java11+ the new transport becomes default.

[jira-importer]

Tamas Cservenak commented

Today progress, where modern clients and HTTP/2 does show difference (java11, jetty rewrote, okhttp pending):

Test buiding resolver against central

wagon  01:22
native 00:52
jetty  00:52
okhttp -- (did not rewrite since last time, unfair to measure)
java11 00:51

 

[jira-importer]

Tamas Cservenak commented

Third round, the transports can be considered feature complete (while the project has now quite a lot of redundancy, like the test HTTP server, this would yield a cleanup):

Java: 17.0.5 Temurin
Maven 3.9.0-SNAPSHOT

Cases:
A) building maven-resolver against Google Central Mirror (EU)
B) building maven-resolver against Maven Central

        A      B
wagon   00:49  01:18
native  00:45  00:50
jetty   00:45  00:51
okhttp  00:53  00:51
java11  00:41  00:50

Conclusion: Regarding existing transports, wagon is "battle tested", but is slowest, while native is rock solid and performant, but both are (and can be only) HTTP/1.1. The HTTP/2 is definitely better at least in C10k area (one TCP socket used, which may be interesting on CI server for example).

OkHttp is faster then wagon, but by a margin, plus it pulls in kotlin, that is I think undesired, so I'd eliminate it.

The two remaining contenders, Jetty and Java11 HttpClient are both interesting. Con for Jetty is relatively big dependency trail, while for Java11 HttpCiient is requirement of Java11 (but it is currently circumvented cleanly, so this transport can exist even in Maven 3.9.0 but remains dormant if Java8 used, while activates when Java11+ used).

[jira-importer]

Peter lynch commented

hey Tamas Cservenak - I have no good reason for asking other than curious ... Apache HTTP client 5.* was not in showdown because? It claims to support HTTP/2 - but feels like you know something I don't.

Happy new year

[jira-importer]

Tamas Cservenak commented

JDK and Jetty transports are present.