mvn javadoc:aggregate -P docFX takes 90 minutes

[suztomo]

@lqiu96 , @eaball35 , and I found that for certain Java repository, mvn javadoc:aggregate -P docFX (exact command is https://github.com/googleapis/java-aiplatform/blob/348c76afec3c2ee79861351d861c0dff33f194e0/.kokoro/release/publish_javadoc11.sh#L40) takes around 60 - 90 minutes to finish.

We're investigating the slowness. But if some of you know the cause of the slowness, that would be much appreciated.

Environment details

• Programming language: Java
• OS: Linux
• Language runtime version: Java 11
• Package version: 1.5.0 (but we also observed the same slowness in 1.6.0)

Steps to reproduce

Check Kokoro job "prod:cloud-devrel/client-libraries/java/java-aiplatform/release/publish_javadoc11".

OR

1. Checkout https://github.com/googleapis/java-aiplatform
2. Run mvn -X javadoc:aggregate -P docFX. You see your program gets stuck at

[INFO] Configuration changed, re-generating javadoc.
[DEBUG] /usr/local/google/home/suztomo/.sdkman/candidates/java/11.0.12-open/bin/javadoc @options @argfile

(waiting for 60-90 minutes, then it succeeds)

Observation

ClassBuilder.buildFilesForInnerClasses is taking the most of the time.

Other observations:

• The heap usage was just fine; it's well below the maximum heap size configured.
• top command showed the javadoc command hit 100% of one of the cores.
• strace command didn't show any system calls being made while waiting. (I attached to the process and waited for 1 minutes).
• lsof showed nothing suspicious; only ~50 files opened by the javadoc process.

Comparison to similar jobs in other language

• Java (cloud-devrel/client-libraries/java/java-aiplatform/release/publish_javadoc11) takes 96 minutes.
• Python (cloud-devrel/client-libraries/python/googleapis/python-aiplatform/docs/docs) takes 14 minutes.
• NodeJS (cloud-devrel/client-libraries/nodejs/release/googleapis/nodejs-ai-platform/docs) takes 19 minutes.

Steps

• Try to figure out how to attach debugger to the javadoc process. Adding -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 resulted in Exit code: 1 - javadoc: error - invalid flag: -agentlib:jdwp=transport=dt_socket.
  • mvn javadoc:aggregate-no-fork does fork javadoc command. The "no-fork" means no compilation.
• Try running the main class in https://github.com/googleapis/java-docfx-doclet/blob/main/third_party/docfx-doclet-143274/src/main/java/com/microsoft/doclet/DocletRunner.java#L13 against java-aiplatform repository, to reproduce the problem.
  • Memo: https://stackoverflow.com/questions/51384097/how-can-i-debug-a-doclet-in-java-10

[suztomo]

Or does anyone know how to get logging (or any other insights about the operations)?

[eaball35]

Great deep dive, thanks @suztomo & @lqiu96.

Interesting to see it's taking the other languages a long time as well. I was surprised to see how much content this particular client actually has (https://cloud.google.com/java/docs/reference/google-cloud-aiplatform/latest/overview). This is definitely one that has a lot of interfaces/classes/enums and Java consistently has more content than the other languages in general.

Looking at the publish_javadoc job (legacy docs standard javadoc run w/o doclet) as a control it already takes around 11 minutes just to run javadoc. The fact that it does finish successfully at the end of the cloud rad job is interesting. It seems possible there are some tasks in the doclet that are compounding to all that extra time.

Chat more in sync about it tomorrow.

[suztomo]

the other languages a long time as well

Java's 90 minutes is off compared to NodeJS and Python's. Looking forward to the meeting.

[suztomo]

How to run DocletRunner with #127

[alicejli]

This was fixed with #140