Auto Compaction does not work when coordinator is running on 0.17.0 and MiddleManager is running on 0.16.0

[maytasm]

Auto Compaction does not work when coordinator is running on 0.17.0 and MiddleManager is running on 0.16.0

Affected Version

coordinator is running on 0.17.0 and MiddleManager is running on 0.16.0

Description

Auto Compaction does not work when coordinator is running on 0.17.0 and MiddleManager is running on 0.16.0.
This is due to incompatibility when deserializing the json ingestion spec that is sent to MiddleManager from coordinator. This will automatically fix itself (since auto compaction is auto run) after coordinator upgraded to 0.17.0

2020-01-18T02:02:34,669 ERROR [TaskMonitorCache-0] org.apache.curator.framework.recipes.cache.PathChildrenCache - com.fasterxml.jackson.databind.exc.InvalidTypeIdException: Could not resolve type id 'index' as a subtype of org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexTuningConfig: known type ids = [index_parallel, realtime] (for POJO property 'tuningConfig') at [Source: (byte[])"{"type":"compact","id":"compact_auto-compact-test-new-4_2020-01-18T02:02:34.633Z","resource":{"availabilityGroup":"compact_auto-compact-test-new-4_2020-01-18T02:02:34.633Z","requiredCapacity":1},"dataSource":"auto-compact-test-new-4","interval":null,"segments":[{"dataSource":"auto-compact-test-new-4","interval":"2015-09-12T00:00:00.000Z/2015-09-13T00:00:00.000Z","version":"2020-01-18T01:33:43.101Z","loadSpec":{"type":"local","path":"/tmp/rolling-upgrade-test/var/druid/segments/auto-compact-test-"[truncated 30732 bytes]; line: 1, column: 30439] (through reference chain: org.apache.druid.indexing.common.task.CompactionTask["tuningConfig"]) at com.fasterxml.jackson.databind.exc.InvalidTypeIdException.from(InvalidTypeIdException.java:43) ~[jackson-databind-2.10.1.jar:2.10.1] at com.fasterxml.jackson.databind.DeserializationContext.invalidTypeIdException(DeserializationContext.java:1758) ~[jackson-databind-2.10.1.jar:2.10.1] at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownTypeId(DeserializationContext.java:1265) ~[jackson-databind-2.10.1.jar:2.10.1] at com.fasterxml.jackson.databind.jsontype.impl.TypeDeserializerBase._handleUnknownTypeId(TypeDeserializerBase.java:290) ~[jackson-databind-2.10.1.jar:2.10.1] at com.fasterxml.jackson.databind.jsontype.impl.TypeDeserializerBase._findDeserializer(TypeDeserializerBase.java:162) ~[jackson-databind-2.10.1.jar:2.10.1] at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113) ~[jackson-databind-2.10.1.jar:2.10.1] at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:97) ~[jackson-databind-2.10.1.jar:2.10.1] at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithType(BeanDeserializerBase.java:1178) ~[jackson-databind-2.10.1.jar:2.10.1] at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:527) ~[jackson-databind-2.10.1.jar:2.10.1] at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:528) ~[jackson-databind-2.10.1.jar:2.10.1] at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:417) ~[jackson-databind-2.10.1.jar:2.10.1] at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1287) ~[jackson-databind-2.10.1.jar:2.10.1] at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:326) ~[jackson-databind-2.10.1.jar:2.10.1] at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:194) ~[jackson-databind-2.10.1.jar:2.10.1] at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:161) ~[jackson-databind-2.10.1.jar:2.10.1] at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:130) ~[jackson-databind-2.10.1.jar:2.10.1] at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:97) ~[jackson-databind-2.10.1.jar:2.10.1] at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:254) ~[jackson-databind-2.10.1.jar:2.10.1] at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:68) ~[jackson-databind-2.10.1.jar:2.10.1] at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4202) ~[jackson-databind-2.10.1.jar:2.10.1] at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3266) ~[jackson-databind-2.10.1.jar:2.10.1] at org.apache.druid.indexing.worker.WorkerTaskMonitor$1.childEvent(WorkerTaskMonitor.java:165) ~[druid-indexing-service-0.17.0-SNAPSHOT.jar:0.17.0-SNAPSHOT] at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:538) [curator-recipes-4.1.0.jar:4.1.0] at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:532) [curator-recipes-4.1.0.jar:4.1.0] at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93) [curator-framework-4.1.0.jar:4.1.0] at org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435) [curator-client-4.1.0.jar:?] at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85) [curator-framework-4.1.0.jar:4.1.0] at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:530) [curator-recipes-4.1.0.jar:4.1.0] at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [curator-recipes-4.1.0.jar:4.1.0] at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:808) [curator-recipes-4.1.0.jar:4.1.0] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_232] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_232] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_232] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_232] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]

[clintropolis]

I wonder if it would be reasonable to make druid.indexer.runner.minWorkerVersion set by default to the release version, and also have middle manager/indexers also set the worker version, it could solve the problem?

I think it would still need to be user configurable to be backwards compatible with autoscaling usage, but might be a safer default than allowing the overlord to run tasks on any version of druid ever.

[clintropolis]

I wonder if it would be reasonable to make druid.indexer.runner.minWorkerVersion set by default to the release version, and also have middle manager/indexers also set the worker version, it could solve the problem?

Well, i guess it potentially interferes with task scheduling for non autoscaling based rolling updates, since you need to provision new workers sooner rather than later to prevent waiting tasks from piling up, if the strategy is waiting for existing workers to drain after a disable before provisioning a new one.