SOLR-7632 TikaServer as pluggable backend to existing extraction handler

[janhoy]

https://issues.apache.org/jira/browse/SOLR-7632

This work builds on the one in #3361 but instead of making a new module, we add it as a capability to the existing extraction handler through specifying extraction.backend=tikaserver.

This first required refactoring extraction handler to detach it from the Tika-v1 API. There is a new interface ExtractionBackend that takes generic ExtractionRequest object in and returns an ExtractionResult bean, and a new LocalTikaExtractionBackend implementation that encapsulates all Tikav1 api handling. This implementation can be deprecated, and in Solr 10, the tikaserver one can be made default.

All existing tests pass, and most of the existing extraction tests now also pass when running the tikaserver backend (running in TestContainers). Unfortunately docker is not available in Crave, so a new GH workflow is made to run only the extraction tests.

TODO's:

• Lots of code from GenAI, which needs review and rewrite / simplification.
• There may be debug print and TODOs left here and there
• The metadata back-compat map is AI generated and by no means complete or even correct 🤣
• Lack of JavaDoc everywhere
• Harden exception handling, retrying, timeout values etc
• Should probably use Jetty HTTP client instead of JDK one.
• Optimize throughput with async methods? The request thread is blocked on Tika response
• Explicit HTTPS / self-signed cert support?
• Complete the RefGuide docs

[epugh]

Exciting!

[janhoy]

Status:

• Parses docs using TikaServer
• Can switch between xml (html) and text format of the content field
• Randomized the choice of backend for the main test class
• ExtractOnly not fully implemented for tikaserver, some tests fail

TBD:

• The whole xpath / SAX parsing of XML response is missing
• We use JDK HTTP client, could perhaps use Jetty client. See other POC for example, including making timeouts configurable
• Must make sure that tikaserver.url is only configurable on requesthandler config in solrconfig, not as a request parameter (security)
• RefGuide docs, especially how to start TikaServer etc
• Remove the DummyExtractionBackend

Anyone, please feel free to hack away on this if it looks exciting, committing directly to the PR branch.

Question: Would it bring value to isolate the refactoring in one PR and then another one to add the tikaserver impl?

[epugh]

Any luck with security manager?? I had many difficulties

[janhoy]

Yea, that’s annoying. Perhaps we could disable JSM for this test or for tests in the entire module?

[iamsanjay]

I had the similar experience as I was upgrading kafka. And then I stopped.

[epugh]

When I first saw DummyExtractionBackend my first thought was that it should be in the test class hierarchy. However, would there be value in keeping it? If you wanted to test your set up in Solr (and not worry about the Tika side), could it be useful for that? "I send a doc and I get something back"....

[janhoy]

Did a series of final improvements, based on review feedback from Claude Code, added a new test, hardened input parameter validation etc.

Planning to merge this to main Wednesday morning CET, hopefully incorporating feedback from @epugh taking it for a spin.

If any of you have a review pending, let me know so I can incorporate that as well.

[janhoy]

Any luck with security manager?? I had many difficulties

Skipping JSM for extraction tests was the easy way out, and given that JSM is going away as soon as you upgrade to JRE 24+, I'm not too worried about this shortcut.

[janhoy]

To make an even cleaner separation, let's not import any org.apache.tika classes in the base classes that will survive the removal of local backend.

[epugh]

To make an even cleaner separation, let's not import any org.apache.tika classes in the base classes that will survive the removal of local backend.

Very astute comment! I am excited about adding tika-pipes as a replacement for local in the future...

[janhoy]

Here are more imports depending on Tika. These classes can also be copied into our project, but it starts to get deep, with a total of 18 java source files if we want to bring it all in.

Seems a bit much for ONE usage in TikaServerExtractionBackend, so I'll leave it in there and defer to later to find / write a replacement for the BodyContentHandler.