Better URI namespaced lookups behavior with single files

[gianm]

We had a lookup like this:

{
   "namespace" : "channel_lookup",
   "pollPeriod" : "P1D",
   "namespaceParseSpec" : {
      "keyFieldName" : "key",
      "valueFieldName" : "value",
      "format" : "customJson"
   },
   "uri" : "file:/opt/data/druid/channel-lookup.json",
   "type" : "uri"
}

In this situation I'd have expected it to match exactly the file /opt/data/druid/channel-lookup.json, but it actually matched a different file named "init" in the same directory. Reading through the code it looks like the file name of a uri is intentionally ignored in favor of searching the parent directory for things matching the versionRegex.

IMO this is pretty surprising behavior (even though the docs do suggest that this can happen).

Could we improve it in one (or all) of these ways?

• don't do the regex-matching-in-parent-dir thing when a specific file is provided and versionRegex = null
• when doing the regex-matching-in-parent-dir thing, only match files if they match the versionRegex and have the provided uri as a prefix (instead of only matching the versionRegex)
• include an example in the docs of how to use a specific file for a lookup
• describe the regex-matching-in-dir process front and center under "URI namespace update" rather than only under the section for versionRegex.

[drcrallen]

after #1576 goes in I'll update the extension to be able to use the dynamic configuration stuff. I'll see if I can get this fixed at the same time. I agree making it smarter and clearer about the uri is the right way to go.

My suggestion is having 3 fields:

uri - An absolute URI to grab data from

XOR

uriPrefix - A prefix for doing pattern matching.
nodeRegex - An optional regex for matching the node part of the uri path. something like "channel-lookup[\-0-9]*\.json" in your example would match any of the following: channel-lookup.json, channel-lookup-0000.json but NOT include channel-lookup.json.backup

[vogievetsky]

from an API perspective I think @drcrallen 's suggestion is the way to go.

[drcrallen]

@vogievetsky I amended the comment just a bit at the end to clarify regex behavior, just FYI