Skip to content

Exclude json-smart, not needed given how we are using json-path.#37

Merged
xvrl merged 1 commit intometamx:masterfrom
gianm:exclude-minidev-asm
Jan 4, 2016
Merged

Exclude json-smart, not needed given how we are using json-path.#37
xvrl merged 1 commit intometamx:masterfrom
gianm:exclude-minidev-asm

Conversation

@gianm
Copy link
Copy Markdown
Contributor

@gianm gianm commented Jan 4, 2016

@gianm
Copy link
Copy Markdown
Contributor Author

gianm commented Jan 4, 2016

See also #34

Comment thread pom.xml Outdated
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we exclude json-smart instead?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so, since it's linked against by a static "DEFAULTS" in one of the jsonpath configuration classes.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gianm I think if we upgrade to JSONPath 2.1.0, we can exclude json-smart completely:

json-path/JsonPath#128

This PR is present in 2.1.0 and it skips the creation of the DefaultsImpl object if both a JsonProvider and MappingProvider are given to the builder before build() is called

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, that's an even better solution!

@himanshug
Copy link
Copy Markdown
Contributor

i m not sure if this is enough because no-one in their extensions can use asm version higher than 1.0.2 because net.minidev.asm needs to be included in the final deployments even after this PR for someone to be able to use json based input row parser

@gianm
Copy link
Copy Markdown
Contributor Author

gianm commented Jan 4, 2016

@himanshug does it really? The json path tests still pass with this exclusion, I think it's only used by the json-smart impl for jsonpath but we aren't using that (it's the default impl but we don't use it). So you should be able to include whatever asm you want, or none at all.

@guobingkun
Copy link
Copy Markdown

@gianm We tried excluding the net.minidev.asm dependency. But then got ClassNotFoundException in a realtime indexing task.

2015-12-30T17:36:44,617 ERROR [task-runner-0] io.druid.indexing.common.task.RealtimeIndexTask - Exception aborted realtime processing[kafka_test]: {class=io.druid.indexing.common.task.RealtimeIndexTask, exceptionType=class java.lang.IllegalArgumentException, exceptionMessage=Instantiation of [simple type, class io.druid.data.input.impl.StringInputRowParser] value failed: net.minidev.asm.FieldFilter}
java.lang.IllegalArgumentException: Instantiation of [simple type, class io.druid.data.input.impl.StringInputRowParser] value failed: net.minidev.asm.FieldFilter
    at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2774) ~[jackson-databind-2.4.6.jar:2.4.6]
    at com.fasterxml.jackson.databind.ObjectMapper.convertValue(ObjectMapper.java:2700) ~[jackson-databind-2.4.6.jar:2.4.6]
    at io.druid.segment.indexing.DataSchema.getParser(DataSchema.java:101) ~[druid-server-0.9.0-1451494361-8b7f16c-1.jar:0.9.0-1451494361-8b7f16c-1]
    at io.druid.indexing.common.task.RealtimeIndexTask.run(RealtimeIndexTask.java:321) [druid-indexing-service-0.9.0-1451494361-8b7f16c-1.jar:0.9.0-1451494361-8b7f16c-1]
    at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:288) [druid-indexing-service-0.9.0-1451494361-8b7f16c-1.jar:0.9.0-1451494361-8b7f16c-1]
    at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:268) [druid-indexing-service-0.9.0-1451494361-8b7f16c-1.jar:0.9.0-1451494361-8b7f16c-1]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_60]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_60]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_60]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Instantiation of [simple type, class io.druid.data.input.impl.StringInputRowParser] value failed: net.minidev.asm.FieldFilter
    at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.wrapException(StdValueInstantiator.java:405) ~[jackson-databind-2.4.6.jar:2.4.6]
    at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:234) ~[jackson-databind-2.4.6.jar:2.4.6]
    at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:167) ~[jackson-databind-2.4.6.jar:2.4.6]
    at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:398) ~[jackson-databind-2.4.6.jar:2.4.6]
    at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6]
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6]
    at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:156) ~[jackson-databind-2.4.6.jar:2.4.6]
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:126) ~[jackson-databind-2.4.6.jar:2.4.6]
    at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113) ~[jackson-databind-2.4.6.jar:2.4.6]
    at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84) ~[jackson-databind-2.4.6.jar:2.4.6]
    at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:132) ~[jackson-databind-2.4.6.jar:2.4.6]
    at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:41) ~[jackson-databind-2.4.6.jar:2.4.6]
    at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2769) ~[jackson-databind-2.4.6.jar:2.4.6]
    ... 9 more
Caused by: java.lang.ClassNotFoundException: net.minidev.asm.FieldFilter
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_60]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_60]
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) ~[?:1.8.0_60]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_60]
    at net.minidev.json.reader.JsonWriter.<clinit>(JsonWriter.java:153) ~[json-smart-2.1.1.jar:?]
    at net.minidev.json.JSONValue.<clinit>(JSONValue.java:537) ~[json-smart-2.1.1.jar:?]
    at com.jayway.jsonpath.spi.json.JsonSmartJsonProvider.<init>(JsonSmartJsonProvider.java:39) ~[json-path-2.0.0.jar:2.0.0]
    at com.jayway.jsonpath.internal.DefaultsImpl.jsonProvider(DefaultsImpl.java:21) ~[json-path-2.0.0.jar:2.0.0]
    at com.jayway.jsonpath.Configuration.defaultConfiguration(Configuration.java:179) ~[json-path-2.0.0.jar:2.0.0]
    at com.metamx.common.parsers.JSONPathParser.<init>(JSONPathParser.java:64) ~[java-util-0.27.5.jar:?]
    at io.druid.data.input.impl.JSONParseSpec.makeParser(JSONParseSpec.java:72) ~[druid-api-0.3.15.jar:0.3.15]
    at io.druid.data.input.impl.StringInputRowParser.<init>(StringInputRowParser.java:56) ~[druid-api-0.3.15.jar:0.3.15]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_60]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_60]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_60]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422) ~[?:1.8.0_60]
    at com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call(AnnotatedConstructor.java:125) ~[jackson-databind-2.4.6.jar:2.4.6]
    at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:230) ~[jackson-databind-2.4.6.jar:2.4.6]
    at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:167) ~[jackson-databind-2.4.6.jar:2.4.6]
    at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:398) ~[jackson-databind-2.4.6.jar:2.4.6]
    at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1064) ~[jackson-databind-2.4.6.jar:2.4.6]
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:264) ~[jackson-databind-2.4.6.jar:2.4.6]
    at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:156) ~[jackson-databind-2.4.6.jar:2.4.6]
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:126) ~[jackson-databind-2.4.6.jar:2.4.6]
    at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113) ~[jackson-databind-2.4.6.jar:2.4.6]
    at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:84) ~[jackson-databind-2.4.6.jar:2.4.6]
    at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:132) ~[jackson-databind-2.4.6.jar:2.4.6]
    at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:41) ~[jackson-databind-2.4.6.jar:2.4.6]
    at com.fasterxml.jackson.databind.ObjectMapper._convert(ObjectMapper.java:2769) ~[jackson-databind-2.4.6.jar:2.4.6]
    ... 9 more

@gianm
Copy link
Copy Markdown
Contributor Author

gianm commented Jan 4, 2016

@guobingkun it should work if you also have the change in this PR (avoid calling Configuration.defaultConfiguration())

@gianm
Copy link
Copy Markdown
Contributor Author

gianm commented Jan 4, 2016

Another option is to avoid instantiating json-smart stuff altogether and then json-smart can be excluded. Unfortunately that involves tweaking a static field so it's probably better to do that in Druid than in java-util. The code would be:

    Configuration.setDefaults(
        new Configuration.Defaults()
        {
          @Override
          public JsonProvider jsonProvider()
          {
            return new JacksonJsonProvider();
          }

          @Override
          public Set<Option> options()
          {
            return EnumSet.noneOf(Option.class);
          }

          @Override
          public MappingProvider mappingProvider()
          {
            return new JacksonMappingProvider();
          }
        }
    );

@gianm
Copy link
Copy Markdown
Contributor Author

gianm commented Jan 4, 2016

we could probably add that to druid-api, doubt anyone would complain there.

@himanshug
Copy link
Copy Markdown
Contributor

Configuration class you mentioned is from com.jayway.json-smart , and since the dependency for same is introduced in java-util here, i believe, the workaround can be added to java-util itself.
in the ideal world, i wish a better alternative for json-smart exists which can be used instead.

@gianm
Copy link
Copy Markdown
Contributor Author

gianm commented Jan 4, 2016

@himanshug that Configuration class is from json-path, not json-smart.

It seems weird to me to have a utils library making changes to static fields in other libraries, so it made more sense to me to put a workaround in druid or druid-api.

we could also use a different json flattening library if you are aware of a better one. json-path was chosen somewhat arbitrarily (decent performance and features but not much stronger reason that than)

@himanshug
Copy link
Copy Markdown
Contributor

@gianm my bad, yeah its in json-path but that is also introduced in java-util , where are you planning to put the change in druid-api or druid otherwise?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bad for java8, please remove

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, IntelliJ does that automatically when I hit the formatter. Any idea how to turn that off?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

preferences->editor->code style->java->javaDoc->generate <p/> on empty lines

that's the one I know about

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok I'll give that a shot

@gianm gianm force-pushed the exclude-minidev-asm branch 3 times, most recently from 7a0c20b to 4389ca0 Compare January 4, 2016 21:07
@gianm
Copy link
Copy Markdown
Contributor Author

gianm commented Jan 4, 2016

@cheddar @himanshug @drcrallen @jon-wei updated with json-path 2.1.0 and excluded json-smart.

@gianm gianm changed the title Exclude net.minidev:asm, not needed given how we are using jsonpath. Exclude json-smart, not needed given how we are using json-path. Jan 4, 2016
@gianm gianm force-pushed the exclude-minidev-asm branch from 4389ca0 to f72ae3a Compare January 4, 2016 21:08
@drcrallen
Copy link
Copy Markdown
Contributor

👍

xvrl added a commit that referenced this pull request Jan 4, 2016
Exclude json-smart, not needed given how we are using json-path.
@xvrl xvrl merged commit ae5bd34 into metamx:master Jan 4, 2016
@himanshug
Copy link
Copy Markdown
Contributor

@gianm thanks for fixing this. @xvrl @drcrallen can one of you pls do a release?

@xvrl
Copy link
Copy Markdown
Contributor

xvrl commented Jan 4, 2016

@himanshug done, 0.27.6 should be up on central in about 20min

@gianm gianm deleted the exclude-minidev-asm branch January 5, 2016 16:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants