Flink backend configuration errors | Ed Easy Installer

[Marc70008]

Hi, I am following the Ed Easy Installer script to provision Ed platform
While running the Ed easy installer bash script, Job managers for the modules like question-set-republish, asset-enrichment, post-publish-processor, qrcode-image-generator, content-publish, are failing due to flink backend configuration errors and neo4j service unavailable exception. Because of these the task-managers of the respective modules are in crashloopbackoff state as they are trying to connect to job manager pods. Any help here?
Flink backend configuration error stack trace:

2023-06-12 12:07:34,259 ERROR org.apache.flink.core.fs.FileSystem                          [] - Failed to load a file system via services
java.util.ServiceConfigurationError: org.apache.flink.core.fs.FileSystemFactory: org.apache.flink.fs.s3presto.S3FileSystemFactory Unable to get public no-arg constructor
        at java.util.ServiceLoader.fail(Unknown Source) ~[?:?]
        at java.util.ServiceLoader.getConstructor(Unknown Source) ~[?:?]
        at java.util.ServiceLoader$LazyClassPathLookupIterator.hasNextService(Unknown Source) ~[?:?]
        at java.util.ServiceLoader$LazyClassPathLookupIterator.hasNext(Unknown Source) ~[?:?]
        at java.util.ServiceLoader$2.hasNext(Unknown Source) ~[?:?]
        at java.util.ServiceLoader$3.hasNext(Unknown Source) ~[?:?]
        at org.apache.flink.core.plugin.PluginLoader$ContextClassLoaderSettingIterator.hasNext(PluginLoader.java:112) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
        at org.apache.flink.shaded.guava18.com.google.common.collect.Iterators$5.hasNext(Iterators.java:547) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
        at org.apache.flink.shaded.guava18.com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:43) ~[flink-dist_2.12-1.13.5.jar:1.13.5]
        at org.apache.flink.core.fs.FileSystem.addAllFactoriesToList(FileSystem.java:1075) [flink-dist_2.12-1.13.5.jar:1.13.5]
        at org.apache.flink.core.fs.FileSystem.loadFileSystemFactories(FileSystem.java:1058) [flink-dist_2.12-1.13.5.jar:1.13.5]
        at org.apache.flink.core.fs.FileSystem.initialize(FileSystem.java:338) [flink-dist_2.12-1.13.5.jar:1.13.5]
        at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.configureFileSystems(ClusterEntrypoint.java:225) [flink-dist_2.12-1.13.5.jar:1.13.5]
        at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:182) [flink-dist_2.12-1.13.5.jar:1.13.5]
        at org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:600) [flink-dist_2.12-1.13.5.jar:1.13.5]
        at org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:85) [flink-dist_2.12-1.13.5.jar:1.13.5]
Caused by: java.lang.LinkageError: loader constraint violation for class org.apache.flink.fs.s3presto.S3FileSystemFactory: when selecting overriding method 'org.apache.hadoop.fs.FileSystem org.apache.flink.fs.s3presto.S3FileSystemFactory.createHadoopFileSystem()' the class loader org.apache.flink.core.plugin.PluginLoader$PluginClassLoader @3546d80f of the selected method's type org.apache.flink.fs.s3presto.S3FileSystemFactory, and the class loader 'app' for its super type org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory have different Class objects for the type org.apache.hadoop.fs.FileSystem used in the signature (org.apache.flink.fs.s3presto.S3FileSystemFactory is in unnamed module of loader org.apache.flink.core.plugin.PluginLoader$PluginClassLoader @3546d80f, parent loader 'platform'; org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory is in unnamed module of loader 'app')
        at java.lang.Class.getDeclaredConstructors0(Native Method) ~[?:?]
        at java.lang.Class.privateGetDeclaredConstructors(Unknown Source) ~[?:?]
        at java.lang.Class.getConstructor0(Unknown Source) ~[?:?]
        at java.lang.Class.getConstructor(Unknown Source) ~[?:?]
        at java.util.ServiceLoader$1.run(Unknown Source) ~[?:?]
        at java.util.ServiceLoader$1.run(Unknown Source) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        ... 15 more

Neo4j error stack trace:
2023-06-12 12:07:41,070 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - asset-enrichment-router -> (image-enrichment-process, video-enrichment-process -> Sink: Unnamed) (1/1) (ed9913e1ac8b65e829af4f58976e3854) switched from INITIALIZING to FAILED on 10.244.1.70:6122-dee158 @ 10.244.1.70 (dataPort=39809). org.neo4j.driver.v1.exceptions.ServiceUnavailableException: Unable to connect to neo4j.dev.svc.cluster.local:7687, ensure the database is running and that there is a working network connection to it. at org.neo4j.driver.internal.util.Futures.blockingGet(Futures.java:123) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.DriverFactory.verifyConnectivity(DriverFactory.java:349) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.DriverFactory.newInstance(DriverFactory.java:95) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.v1.GraphDatabase.driver(GraphDatabase.java:141) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.v1.GraphDatabase.driver(GraphDatabase.java:75) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.v1.GraphDatabase.driver(GraphDatabase.java:87) ~[asset-enrichment-1.0.0.jar:?] at org.sunbird.job.util.Neo4JUtil.<init>(Neo4JUtil.scala:17) ~[asset-enrichment-1.0.0.jar:?] at org.sunbird.job.assetenricment.functions.ImageEnrichmentFunction.open(ImageEnrichmentFunction.scala:29) ~[asset-enrichment-1.0.0.jar:?] at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:34) ~[flink-dist_2.12-1.13.5.jar:1.13.5] at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102) ~[flink-dist_2.12-1.13.5.jar:1.13.5] at org.apache.flink.streaming.api.operators.ProcessOperator.open(ProcessOperator.java:56) ~[flink-dist_2.12-1.13.5.jar:1.13.5] at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:442) ~[flink-dist_2.12-1.13.5.jar:1.13.5] at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:585) ~[flink-dist_2.12-1.13.5.jar:1.13.5] at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55) ~[flink-dist_2.12-1.13.5.jar:1.13.5] at org.apache.flink.streaming.runtime.tasks.StreamTask.executeRestore(StreamTask.java:565) ~[flink-dist_2.12-1.13.5.jar:1.13.5] at org.apache.flink.streaming.runtime.tasks.StreamTask.runWithCleanUpOnFail(StreamTask.java:650) ~[flink-dist_2.12-1.13.5.jar:1.13.5] at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:540) ~[flink-dist_2.12-1.13.5.jar:1.13.5] at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:759) ~[flink-dist_2.12-1.13.5.jar:1.13.5] at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566) ~[flink-dist_2.12-1.13.5.jar:1.13.5] at java.lang.Thread.run(Unknown Source) ~[?:?] Suppressed: org.neo4j.driver.internal.util.ErrorUtil$InternalExceptionCause at org.neo4j.driver.internal.async.ChannelConnectedListener.databaseUnavailableError(ChannelConnectedListener.java:76) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.async.ChannelConnectedListener.operationComplete(ChannelConnectedListener.java:70) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.async.ChannelConnectedListener.operationComplete(ChannelConnectedListener.java:37) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:485) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:162) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.async.ChannelConnectorImpl.installChannelConnectedListeners(ChannelConnectorImpl.java:101) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.async.ChannelConnectorImpl.connect(ChannelConnectorImpl.java:84) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.async.pool.NettyChannelPool.connectChannel(NettyChannelPool.java:66) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.channel.pool.SimpleChannelPool.acquireHealthyFromPoolOrNew(SimpleChannelPool.java:179) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.channel.pool.SimpleChannelPool.acquire(SimpleChannelPool.java:164) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.channel.pool.FixedChannelPool.acquire0(FixedChannelPool.java:266) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.channel.pool.FixedChannelPool.access$300(FixedChannelPool.java:38) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.channel.pool.FixedChannelPool$3.run(FixedChannelPool.java:240) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[asset-enrichment-1.0.0.jar:?] at java.lang.Thread.run(Unknown Source) ~[?:?] Caused by: java.net.UnknownHostException: neo4j.dev.svc.cluster.local at java.net.InetAddress$CachedAddresses.get(Unknown Source) ~[?:?] at java.net.InetAddress.getAllByName0(Unknown Source) ~[?:?] at java.net.InetAddress.getAllByName(Unknown Source) ~[?:?] at java.net.InetAddress.getAllByName(Unknown Source) ~[?:?] at java.net.InetAddress.getByName(Unknown Source) ~[?:?] at org.neo4j.driver.internal.shaded.io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:146) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.util.internal.SocketUtils$8.run(SocketUtils.java:143) ~[asset-enrichment-1.0.0.jar:?] at java.security.AccessController.doPrivileged(Native Method) ~[?:?] at org.neo4j.driver.internal.shaded.io.netty.util.internal.SocketUtils.addressByName(SocketUtils.java:143) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.resolver.DefaultNameResolver.doResolve(DefaultNameResolver.java:43) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:63) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.resolver.SimpleNameResolver.resolve(SimpleNameResolver.java:55) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:57) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.resolver.InetSocketAddressResolver.doResolve(InetSocketAddressResolver.java:32) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.resolver.AbstractAddressResolver.resolve(AbstractAddressResolver.java:108) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.bootstrap.Bootstrap.doResolveAndConnect0(Bootstrap.java:208) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:170) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:145) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.async.ChannelConnectorImpl.connect(ChannelConnectorImpl.java:78) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.async.pool.NettyChannelPool.connectChannel(NettyChannelPool.java:66) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.channel.pool.SimpleChannelPool.acquireHealthyFromPoolOrNew(SimpleChannelPool.java:179) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.channel.pool.SimpleChannelPool.acquire(SimpleChannelPool.java:164) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.channel.pool.FixedChannelPool.acquire0(FixedChannelPool.java:266) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.channel.pool.FixedChannelPool.access$300(FixedChannelPool.java:38) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.channel.pool.FixedChannelPool$3.run(FixedChannelPool.java:240) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) ~[asset-enrichment-1.0.0.jar:?] at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[asset-enrichment-1.0.0.jar:?] ... 1 more

[Krishnaj20]

@gandham-santhosh - Can you check and respond to this query please?

[tsprasath]

@Marc70008 Could you please check neo4j pod is up and running?

[Marc70008]

@tsprasath , sorry for the delayed response, i dont see the neo4j pod, nor in events
I have gone through the charts.csv files for respective BBs but couldnt find helm repo for neo4j
Am I missing something here?

[tsprasath]

@Marc70008 it already present https://github.com/project-sunbird/sunbird-devops/blob/oneclickinstaller/oneclickinstaller/knowlg-charts.csv#L1

[Marc70008]

Not sure how it got missed, recloned the repo and ran, working as expected. Thanks.

[mattkerr7r]

no problem here

[Marc70008]

neo4j is not installed,so the error. The helm repo is in knowlg-charts.csv.