HDDS-7869. Log configuration on component startup. #4271

SaketaChalamchala · 2023-02-13T17:18:08Z

What changes were proposed in this pull request?

Added a conf field in STARTUP_MSG for all Ozone roles with configs relevant to the role.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-7869

How was this patch tested?

Manual test

OM : 
/************************************************************
2023-02-12 20:11:34 STARTUP_MSG: Starting OzoneManager
...
2023-02-12 20:11:34 STARTUP_MSG:   java = 11.0.17
2023-02-12 20:11:34 STARTUP_MSG:   conf = {hadoop.hdds.db.rocksdb.logging.level=INFO, ozone.om.enable.ofs.shared.tmp.dir=false, ozone.om.ratis.log.appender.queue.num-elements=1024, hadoop.hdds.db.rocksdb.logging.enabled=false, ozone.om.snapshot.provider.socket.timeout=5000s, ozone.scm.info.wait.duration=10m, ozone.om.http.auth.type=kerberos, ozone.om.multitenancy.enabled=true, ozone.om.https-bind-host=0.0.0.0, ozone.om.ratis.server.request.timeout=3s, ozone.network.flexible.fqdn.resolution.enabled=false, ozone.om.snapshot.compaction.dag.max.time.allowed=30d, ozone.path.deleting.limit.per.task=10000, ozone.om.enable.filesystem.paths=false, ozone.recon.om.snapshot.task.flush.param=false, ozone.om.admin.protocol.max.retries=20, ozone.om.leader.election.minimum.timeout.duration=5s, ozone.om.ratis.log.appender.queue.byte-limit=32MB, ozone.network.jvm.address.cache.enabled=true, ozone.om.volume.listall.allowed=true, ozone.recon.om.socket.timeout=5s, ozone.om.keyname.character.check.enabled=false, ozone.om.open.key.cleanup.service.timeout=300s, ozone.om.ratis.server.retry.cache.timeout=600000ms, ozone.om.snapshot.provider.request.timeout=5000ms, ozone.directory.deleting.service.interval=1m, ozone.om.save.metrics.interval=5m, ozone.om.grpc.maximum.response.length=134217728, ozone.recon.om.snapshot.task.initial.delay=20s, ozone.om.https-address=0.0.0.0:9875, ozone.om.ratis.port=9872, ozone.key.deleting.limit.per.task=20000, ozone.om.ha.raft.server.retrycache.expirytime=300s, ozone.snapshot.filtering.service.interval=1m, ozone.scm.ca.list.retry.interval=10s, ozone.om.container.location.cache.size=100000, ozone.om.handler.count.key=100, ozone.om.client.trash.core.pool.size=5, hdds.db.profile=DISK, ozone.metastore.rocksdb.cf.write.buffer.size=128MB, ozone.fs.listing.page.size.max=5000, ozone.key.preallocation.max.blocks=64, ozone.om.http-bind-host=0.0.0.0, hdds.scm.http.auth.type=kerberos, ozone.om.fs.snapshot.max.limit=1000, ozone.om.snapshot.provider.connection.timeout=5000s, ozone.metadata.dirs=/data/metadata, ozone.recon.om.connection.timeout=5s, ozone.om.key.path.lock.enabled=false, ozone.metastore.rocksdb.statistics=OFF, ozone.om.ratis.log.purge.gap=1000000, ozone.om.group.rights=ALL, ozone.om.ratis.segment.size=4MB, ozone.om.multitenancy.ranger.sync.timeout=10s, ozone.om.open.key.expire.threshold=7d, ozone.om.snapshot.cache.max.size=10, ozone.om.user.max.volume=1024, ozone.om.client.rpc.timeout=15m, ozone.snapshot.filtering.limit.per.task=2, ozone.sst.filtering.service.timeout=300000ms, ozone.om.ratis.segment.preallocated.size=4MB, ozone.om.ratis.server.failure.timeout.duration=120s, ozone.om.unflushed.transaction.max.count=10000, ozone.om.user.rights=ALL, ozone.recon.om.snapshot.task.interval.delay=1m, ozone.om.container.location.cache.ttl=360m, ozone.om.upgrade.finalization.ratis.based.timeout=30s, ozone.om.http-address=om:9874, ozone.om.admin.protocol.wait.between.retries=1000, ozone.om.address=om, ozone.om.ratis.minimum.timeout=5s, ozone.om.snapshot.compaction.dag.prune.daemon.run.interval=3600s, ozone.om.open.key.cleanup.limit.per.task=1000, ozone.om.ratis.enable=true, ozone.service.shutdown.timeout=60s, hadoop.hdds.db.rocksdb.writeoption.sync=false, ozone.recon.om.connection.request.timeout=5000, ozone.om.multitenancy.ranger.sync.interval=30s, ozone.om.ratis.rpc.type=GRPC, ozone.om.ratis.server.leaderelection.pre-vote=false, ozone.om.http.enabled=true, ozone.om.open.key.cleanup.service.interval=24h, ozone.om.init.default.layout.version=-1}

SCM : 
2023-02-12 20:11:34 /************************************************************
2023-02-12 20:11:34 STARTUP_MSG: Starting StorageContainerManager
...
2023-02-12 20:11:34 STARTUP_MSG:   java = 11.0.17
2023-02-12 20:11:34 STARTUP_MSG:   conf = {ozone.scm.sequence.id.batch.size=1000, hdds.scm.replication.event.timeout=10s, ozone.scm.ha.ratis.snapshot.threshold=1000, hdds.scm.safemode.atleast.one.node.reported.pipeline.pct=0.90, ozone.scm.ha.ratis.server.leaderelection.pre-vote=false, hdds.scm.safemode.enabled=true, ozone.scm.client.port=9860, hdds.key.len=2048, ozone.network.jvm.address.cache.enabled=true, ozone.scm.pipeline.creation.interval=30s, hdds.datanode.storage.utilization.critical.threshold=0.95, hdds.scm.safemode.min.datanode=3, ozone.scm.client.bind.host=0.0.0.0, hdds.scm.block.deletion.per-interval.max=100000, hdds.scm.kerberos.keytab.file=/etc/security/keytabs/scm.keytab, ozone.scm.block.client.port=9863, ozone.scm.ha.ratis.segment.preallocated.size=4MB, hdds.scm.http.auth.kerberos.principal=HTTP/scm@EXAMPLE.COM, ozone.scm.block.deletion.max.retry=4096, ozone.scm.ratis.pipeline.limit=0, ozone.block.deleting.container.limit.per.interval=10, hdds.scm.http.auth.kerberos.keytab=/etc/security/keytabs/scm.keytab, ozone.block.deleting.service.interval=1m, ozone.scm.ca.list.retry.interval=10s, ozone.scm.ha.ratis.leader.election.timeout=5s, hdds.scm.safemode.pipeline.creation=true, ozone.metastore.rocksdb.cf.write.buffer.size=128MB, ozone.chunk.read.buffer.default.size=64KB, ozone.scm.ha.ratis.log.appender.queue.byte-limit=32MB, hdds.scm.kerberos.principal=scm/scm@EXAMPLE.COM, hdds.scmclient.failover.retry.interval=2s, hdds.scm.replication.maintenance.remaining.redundancy=1, ozone.scm.container.layout=FILE_PER_BLOCK, ozone.block.deleting.limit.per.task=1000, hdds.scm.replication.container.inflight.replication.limit=0, hdds.scmclient.max.retry.timeout=30s, hdds.datanode.block.deleting.service.interval=60s, ozone.scm.expired.container.replica.op.scrub.interval=5m, ozone.scm.block.client.bind.host=0.0.0.0, hdds.datanode.recovering.container.scrubbing.service.interval=1m, ozone.scm.block.client.address=scm, ozone.scm.ha.grpc.deadline.interval=30m, hdds.scm.init.default.layout.version=-1, hdds.scm.replication.command.deadline.factor=0.9, ozone.scm.pipeline.destroy.timeout=66s, ozone.recon.scm.snapshot.enabled=true, ozone.scm.pipeline.per.metadata.disk=2, hdds.datanode.storage.utilization.warning.threshold=0.75, hdds.key.dir.name=keys, ozone.scm.pipeline.owner.container.count=1, ozone.scm.ha.ratis.server.retry.cache.timeout=60s, ozone.scm.block.size=256MB, ozone.scm.pipeline.allocated.timeout=5m, hdds.scm.replication.thread.interval=5s, ozone.scm.ha.ratis.log.purge.enabled=false, ozone.scm.datanode.admin.monitor.interval=30s, ozone.scm.info.wait.duration=10m, hdds.scm.replication.maintenance.replica.minimum=2, hdds.scm.unknown-container.action=WARN, hdds.scm.replication.over.replicated.interval=30s, hdds.scmclient.failover.max.retry=15, ozone.network.flexible.fqdn.resolution.enabled=false, ozone.scm.ha.ratis.request.timeout=30s, hdds.scm.safemode.threshold.pct=0.99, ozone.scm.ratis.port=9894, hdds.scm.safemode.healthy.pipeline.pct=0.10, ozone.scm.ha.ratis.leader.ready.wait.timeout=60s, hdds.scm.wait.time.after.safemode.exit=5m, ozone.recon.scm.container.threshold=100, ozone.scm.pipeline.scrub.interval=5m, ozone.recon.scm.connection.request.timeout=5s, ozone.block.deleting.service.timeout=300000ms, ozone.block.deleting.service.workers=10, hdds.scm.block.deleting.service.interval=60s, ozone.scm.skip.bootstrap.validation=false, ozone.scm.ha.ratis.leader.ready.check.interval=2s, ozone.scm.datanode.disallow.same.peers=false, ozone.scm.update.client.crl.check.interval=600s, ozone.scm.pipeline.creation.auto.factor.one=true, hdds.scm.safemode.pipeline-availability.check=true, ozone.scm.event.ContainerReport.thread.pool.size=10, hdds.scm.replication.container.inflight.deletion.limit=0, ozone.metadata.dirs=/data/metadata, ozone.scm.ha.ratis.log.appender.queue.num-elements=1024, ozone.scm.ha.ratis.segment.size=4MB, ozone.scm.ha.ratis.rpc.type=GRPC, ozone.metastore.rocksdb.statistics=OFF, hdds.scm.replication.push=true, ozone.scm.datanode.pipeline.limit=1, ozone.scm.grpc.port=9895, hdds.scm.replication.under.replicated.interval=30s, ozone.recon.scm.connection.timeout=5s, hdds.scm.pipeline.choose.policy.impl=org.apache.hadoop.hdds.scm.pipeline.choose.algorithms.RandomPipelineChoosePolicy, ozone.scm.ha.ratis.server.failure.timeout.duration=120s, hdds.datanode.block.deleting.limit.per.interval=5000, ozone.service.shutdown.timeout=60s, ozone.scm.client.address=scm, ozone.scm.chunk.size=4MB, ozone.scm.ha.ratis.log.purge.gap=1000000, ozone.scm.pipeline.leader-choose.policy=
2023-02-12 20:11:34       org.apache.hadoop.hdds.scm.pipeline.leader.choose.algorithms.MinLeaderCountChoosePolicy
2023-02-12 20:11:34     , hdds.scmclient.rpc.timeout=15m}
2023-02-12 20:11:34 ************************************************************/

Recon: 
/************************************************************
2023-02-12 20:11:34 STARTUP_MSG: Starting ReconServer
...
2023-02-12 20:11:34 STARTUP_MSG:   java = 11.0.17
2023-02-12 20:11:34 STARTUP_MSG:   conf = {ozone.recon.http.auth.kerberos.principal=HTTP/recon@EXAMPLE.COM, ozone.recon.task.missingcontainer.interval=300s, ozone.recon.sql.db.conn.idle.max.age=3600s, ozone.recon.http.enabled=true, ozone.recon.task.pipelinesync.interval=300s, ozone.recon.http-address=0.0.0.0:9888, ozone.recon.nssummary.flush.db.max.threshold=150000, ozone.recon.om.connection.timeout=5s, recon.om.delta.update.loop.limit=10, ozone.recon.administrators=testuser2/scm@EXAMPLE.COM, recon.om.delta.update.limit=2000, ozone.recon.om.snapshot.task.flush.param=false, ozone.recon.sql.db.driver=org.apache.derby.jdbc.EmbeddedDriver, ozone.recon.security.client.datanode.container.protocol.acl=*, ozone.recon.scm.connection.timeout=5s, hdds.recon.heartbeat.interval=60s, ozone.recon.scm.container.threshold=100, ozone.recon.db.dir=/data/metadata/recon, ozone.recon.om.socket.timeout=5s, ozone.recon.sql.db.conn.timeout=30000ms, ozone.recon.https-bind-host=0.0.0.0, ozone.recon.om.snapshot.task.interval.delay=1m, ozone.recon.task.thread.count=1, ozone.recon.http.auth.type=kerberos, ozone.recon.sql.db.conn.idle.test.period=60s, ozone.recon.kerberos.principal=recon/recon@EXAMPLE.COM, ozone.recon.sql.db.conn.idle.test=SELECT 1, ozone.recon.sql.db.auto.commit=true, ozone.recon.scm.snapshot.enabled=true, ozone.recon.http-bind-host=0.0.0.0, ozone.recon.sql.db.jdbc.url=jdbc:derby:${ozone.recon.db.dir}/ozone_recon_derby.db, ozone.recon.scm.connection.request.timeout=5s, ozone.recon.om.snapshot.task.initial.delay=20s, ozone.service.shutdown.timeout=60s, ozone.recon.sql.db.conn.max.active=5, ozone.recon.scm.snapshot.task.initial.delay=1m, ozone.recon.scm.snapshot.task.interval.delay=24h, ozone.recon.om.connection.request.timeout=5000, ozone.recon.http.auth.kerberos.keytab=/etc/security/keytabs/recon.keytab, ozone.recon.task.safemode.wait.threshold=300s, ozone.recon.sql.db.jooq.dialect=DERBY, ozone.recon.sql.db.conn.max.age=1800s, ozone.recon.kerberos.keytab.file=/etc/security/keytabs/recon.keytab, ozone.recon.https-address=0.0.0.0:9889}

DN: 
/************************************************************
2023-02-12 20:11:34 STARTUP_MSG: Starting HddsDatanodeService
...
2023-02-12 20:11:34 STARTUP_MSG:   java = 11.0.17
2023-02-12 20:11:34 STARTUP_MSG:   conf = {hdds.datanode.metadata.rocksdb.cache.size=64MB, ozone.scm.datanode.ratis.volume.free-space.min=1GB, hdds.datanode.rocksdb.auto-compaction-small-sst-file-num-threshold=512, hdds.ratis.raft.server.delete.ratis.log.directory=true, ozone.network.jvm.address.cache.enabled=true, hdds.datanode.chunk.data.validation.check=false, hdds.datanode.rocksdb.log.level=INFO, hdds.datanode.container.delete.threads.max=2, hdds.ratis.raft.server.watch.timeout=180s, hdds.datanode.container.schema.v3.enabled=true, hdds.datanode.block.delete.threads.max=5, hdds.ratis.raft.server.notification.no-leader.timeout=300s, hdds.ratis.raft.server.datastream.client.pool.size=10, hdds.ratis.raft.server.datastream.request.threads=20, ozone.fs.datastream.enabled=false, hdds.datanode.wait.on.all.followers=false, hdds.ratis.raft.server.rpc.slowness.timeout=300s, ozone.scm.ca.list.retry.interval=10s, hdds.datanode.replication.streams.limit=10, hdds.datanode.rocksdb.auto-compaction-small-sst-file=true, hdds.datanode.failed.data.volumes.tolerated=-1, hdds.datanode.ratis.server.request.timeout=2m, hdds.datanode.rocksdb.max-open-files=1024, hdds.datanode.rocksdb.auto-compaction-small-sst-file-size-threshold=1MB, hdds.datanode.read.chunk.threads.per.volume=10, hdds.datanode.rocksdb.delete-obsolete-files-period=1h, hdds.datanode.container.schema.v3.key.separator=|, hdds.command.status.report.interval=30s, hdds.ratis.raft.server.leaderelection.pre-vote=false, hdds.datanode.block.delete.queue.limit=1440, hdds.ratis.raft.server.rpc.request.timeout=60s, hdds.container.replication.compression=NO_COMPRESSION, hdds.container.action.max.limit=20, hdds.datanode.periodic.disk.check.interval.minutes=60, hdds.datanode.rocksdb.log.max-file-num=64, hdds.datanode.df.refresh.period=5m, hdds.pipeline.action.max.limit=20, hdds.datanode.failed.db.volumes.tolerated=-1, hdds.datanode.rocksdb.log.max-file-size=32MB, hdds.datanode.disk.check.timeout=10m, hdds.container.close.threshold=0.9f, hdds.datanode.disk.check.min.gap=15m, ozone.service.shutdown.timeout=60s, hdds.ratis.raft.server.write.element-limit=1024, hdds.datanode.replication.port=9886, hdds.container.checksum.verification.enabled=true, hdds.datanode.failed.metadata.volumes.tolerated=-1, hdds.datanode.du.refresh.period=1h, hdds.datanode.http.auth.type=kerberos}

S3G: 
/************************************************************
2023-02-12 17:03:21 STARTUP_MSG: Starting Gateway
...
2023-02-12 17:03:21 STARTUP_MSG:   java = 11.0.17
2023-02-12 17:03:21 STARTUP_MSG:   conf = {hadoop.http.idle_timeout.ms=60000, ozone.s3g.volume.name=s3v, ozone.s3g.http-address=0.0.0.0:9878, ozone.om.grpc.maximum.response.length=134217728, ozone.s3g.http.enabled=true, ozone.service.shutdown.timeout=60s, ozone.s3g.client.buffer.size=4KB, ozone.s3g.http.auth.kerberos.principal=HTTP/s3g@EXAMPLE.COM, ozone.s3g.kerberos.principal=s3g/s3g@EXAMPLE.COM, ozone.s3g.kerberos.keytab.file=/etc/security/keytabs/s3g.keytab, ozone.s3g.http.auth.kerberos.keytab=/etc/security/keytabs/s3g.keytab, ozone.s3g.http-bind-host=0.0.0.0}

Ozone CSI:
sh-4.2$ ozone csi -conf=/etc/hadoop/x --verbose
...
2023-02-13 01:05:19,080 [main] INFO csi.CsiServer: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting CsiServer
STARTUP_MSG:   host = b453337cacf2/172.18.0.8
STARTUP_MSG:   args = [-conf=/etc/hadoop/x, --verbose]
STARTUP_MSG:   version = 1.4.0-SNAPSHOT
STARTUP_MSG:   classpath = /etc/hadoop...
STARTUP_MSG:   build = git@github.com:SaketaChalamchala/ozone.git/e2a52323ad6a6c0675717e89dd3e9f472e52c228 ; compiled by 'schalamchala' on 2023-02-13T02:12Z
STARTUP_MSG:   java = 11.0.17
STARTUP_MSG:   conf = {ozone.csi.s3g.address=http://localhost:9878, ozone.csi.mount.command=goofys --endpoint %s %s %s, ozone.csi.socket=/var/lib/csi.sock, ozone.csi.default-volume-size=1000000000}

kerneltime · 2023-02-13T17:31:47Z

cc @sumitagrawl @dombizita @duongkame can you'll take a look as well?

neils-dev · 2023-02-13T18:40:48Z

thanks @SaketaChalamchala - Looks to be some CI failures in the integration tests.

sumitagrawl

@SaketaChalamchala Thanks for working on this, one query is there related to completeness and correctness of configs getting logged.

...rver-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManagerStarter.java

adoroszlai · 2023-02-14T12:53:16Z

hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/StringUtils.java


  public static void startupShutdownMessage(VersionInfo versionInfo,
-      Class<?> clazz, String[] args, Logger log) {
+      Class<?> clazz, String[] args, Logger log, String conf) {


I think it would be better to pass a Map here instead of a fully formatted string. This would allow filtering some sensitive properties if needed, as well as sorting the config properties alphabetically. (Currently order is random, since Properties is just a Hashtable.)

Passing the configuration object as it is and I added a processForLogging utility function to redact sensitive properties and sort them before logging it.

errose28 · 2023-02-15T00:45:00Z

I don't think we should filter by tags or prefixes when printing the config. I think we should just print the contents of whatever config object the component has.

I think this would show only options that the user has explicitly set, which may be different than the defaults. I think this set of configs would be more useful for debugging.
We don't have to worry about missing config values in the log message due to improper tagging now or for future values.
It would show an accurate representation of the config the component has, even if it does not read some of the values.
- This removes some assumptions about what components are supposed to read vs. what they could potentially read, which could hinder debugging.
- Since this would usually be inspected for debugging purposes, users can filter the configs they are looking for anyways.

Also +1 to Attila's suggestion of sorting the configs alphabetically.

…d sort conf object for logging.

SaketaChalamchala · 2023-02-16T23:15:03Z

Thanks @errose28 @adoroszlai and @sumitagrawl for the review.

I don't think we should filter by tags or prefixes when printing the config. I think we should just print the contents of whatever config object the component has.

I'm printing all the properties in the config object used by each service which includes core-site, ozone-site and all the defaults. Below are examples. It seems like a lot. Let me know if it's readable.

Full List of configs from OM : 
/************************************************************
STARTUP_MSG: Starting OzoneManager
...
STARTUP_MSG:   java = 11.0.17
STARTUP_MSG:   conf = {adl.feature.ownerandgroup.enableupn=false, adl.http.timeout=-1, datanode.https.port=50475, dfs.balancer.address=0.0.0.0:0, dfs.balancer.block-move.timeout=0, dfs.balancer.dispatcherThreads=200, dfs.balancer.getBlocks.min-block-size=10485760, dfs.balancer.getBlocks.size=2147483648, dfs.balancer.keytab.enabled=false, dfs.balancer.max-iteration-time=1200000, dfs.balancer.max-no-move-interval=60000, dfs.balancer.max-size-to-move=10737418240, dfs.balancer.movedWinWidth=5400000, dfs.balancer.moverThreads=1000, dfs.balancer.service.interval=5m, dfs.balancer.service.retries.on.exception=5, dfs.batched.ls.limit=100, dfs.block.access.key.update.interval=600, dfs.block.access.token.enable=false, dfs.block.access.token.lifetime=600, dfs.block.access.token.protobuf.enable=false, dfs.block.invalidate.limit=1000, dfs.block.misreplication.processing.limit=10000, dfs.block.placement.ec.classname=org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyRackFaultTolerant, dfs.block.replicator.classname=org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault, dfs.block.scanner.skip.recent.accessed=false, dfs.block.scanner.volume.bytes.per.second=1048576, dfs.block.scanner.volume.join.timeout.ms=5000, dfs.blockreport.incremental.intervalMsec=0, dfs.blockreport.initialDelay=0, dfs.blockreport.intervalMsec=21600000, dfs.blockreport.split.threshold=1000000, dfs.blocksize=134217728, dfs.bytes-per-checksum=512, dfs.cachereport.intervalMsec=10000, dfs.checksum.combine.mode=MD5MD5CRC, dfs.checksum.ec.socket-timeout=3000, dfs.checksum.type=CRC32C, dfs.client-write-packet-size=65536, dfs.client.block.reader.remote.buffer.size=512, dfs.client.block.write.locateFollowingBlock.initial.delay.ms=400, dfs.client.block.write.locateFollowingBlock.max.delay.ms=60000, dfs.client.block.write.locateFollowingBlock.retries=5, dfs.client.block.write.replace-datanode-on-failure.best-effort=false, dfs.client.block.write.replace-datanode-on-failure.enable=true, dfs.client.block.write.replace-datanode-on-failure.min-replication=0, dfs.client.block.write.replace-datanode-on-failure.policy=DEFAULT, dfs.client.block.write.retries=3, dfs.client.cached.conn.retry=3, dfs.client.context=default, dfs.client.datanode-restart.timeout=30, dfs.client.deadnode.detection.enabled=false, dfs.client.deadnode.detection.idle.sleep.ms=10000, dfs.client.deadnode.detection.probe.connection.timeout.ms=20000, dfs.client.deadnode.detection.probe.deadnode.interval.ms=60000, dfs.client.deadnode.detection.probe.deadnode.threads=10, dfs.client.deadnode.detection.probe.suspectnode.interval.ms=300, dfs.client.deadnode.detection.probe.suspectnode.threads=10, dfs.client.deadnode.detection.rpc.threads=20, dfs.client.domain.socket.data.traffic=false, dfs.client.failover.connection.retries=0, dfs.client.failover.connection.retries.on.timeouts=0, dfs.client.failover.max.attempts=15, dfs.client.failover.random.order=false, dfs.client.failover.resolve-needed=false, dfs.client.failover.resolver.impl=org.apache.hadoop.net.DNSDomainNameResolver, dfs.client.failover.resolver.useFQDN=true, dfs.client.failover.sleep.base.millis=500, dfs.client.failover.sleep.max.millis=15000, dfs.client.hedged.read.threadpool.size=0, dfs.client.hedged.read.threshold.millis=500, dfs.client.https.keystore.resource=ssl-client.xml, dfs.client.https.need-auth=false, dfs.client.ignore.namenode.default.kms.uri=false, dfs.client.key.provider.cache.expiry=864000000, dfs.client.max.block.acquire.failures=3, dfs.client.mmap.cache.size=256, dfs.client.mmap.cache.timeout.ms=3600000, dfs.client.mmap.enabled=true, dfs.client.mmap.retry.timeout.ms=300000, dfs.client.read.short.circuit.replica.stale.threshold.ms=1800000, dfs.client.read.shortcircuit=false, dfs.client.read.shortcircuit.buffer.size=1048576, dfs.client.read.shortcircuit.skip.checksum=false, dfs.client.read.shortcircuit.streams.cache.expiry.ms=300000, dfs.client.read.shortcircuit.streams.cache.size=256, dfs.client.read.striped.threadpool.size=18, dfs.client.read.uri.cache.enabled=false, dfs.client.read.use.cache.priority=false, dfs.client.refresh.read-block-locations.ms=0, dfs.client.retry.interval-ms.get-last-block-length=4000, dfs.client.retry.max.attempts=10, dfs.client.retry.policy.enabled=false, dfs.client.retry.policy.spec=10000,6,60000,10, dfs.client.retry.times.get-last-block-length=3, dfs.client.retry.window.base=3000, dfs.client.server-defaults.validity.period.ms=3600000, dfs.client.short.circuit.num=1, dfs.client.short.circuit.replica.stale.threshold.ms=1800000, dfs.client.slow.io.warning.threshold.ms=30000, dfs.client.socket-timeout=60000, dfs.client.socket.send.buffer.size=0, dfs.client.socketcache.capacity=16, dfs.client.socketcache.expiryMsec=3000, dfs.client.test.drop.namenode.response.number=0, dfs.client.use.datanode.hostname=false, dfs.client.use.legacy.blockreader.local=false, dfs.client.write.byte-array-manager.count-limit=2048, dfs.client.write.byte-array-manager.count-reset-time-period-ms=10000, dfs.client.write.byte-array-manager.count-threshold=128, dfs.client.write.byte-array-manager.enabled=false, dfs.client.write.exclude.nodes.cache.expiry.interval.millis=600000, dfs.client.write.max-packets-in-flight=80, dfs.container.chunk.write.sync=false, dfs.container.ipc=9859, dfs.container.ipc.random.port=false, dfs.container.ratis.admin.port=9857, dfs.container.ratis.datastream.enabled=true, dfs.container.ratis.datastream.port=9855, dfs.container.ratis.datastream.random.port=false, dfs.container.ratis.enabled=false, dfs.container.ratis.ipc=9858, dfs.container.ratis.ipc.random.port=false, dfs.container.ratis.leader.pending.bytes.limit=1GB, dfs.container.ratis.log.appender.queue.byte-limit=32MB, dfs.container.ratis.log.appender.queue.num-elements=1, dfs.container.ratis.log.purge.gap=1000000, dfs.container.ratis.log.queue.byte-limit=4GB, dfs.container.ratis.log.queue.num-elements=1024, dfs.container.ratis.num.container.op.executors=10, dfs.container.ratis.num.write.chunk.threads.per.volume=10, dfs.container.ratis.replication.level=MAJORITY, dfs.container.ratis.rpc.type=GRPC, dfs.container.ratis.segment.preallocated.size=16KB, dfs.container.ratis.segment.size=1MB, dfs.container.ratis.server.port=9856, dfs.container.ratis.statemachine.max.pending.apply-transactions=10000, dfs.container.ratis.statemachinedata.sync.retries=-1, dfs.container.ratis.statemachinedata.sync.timeout=10s, dfs.content-summary.limit=5000, dfs.content-summary.sleep-microsec=500, dfs.data.transfer.client.tcpnodelay=true, dfs.data.transfer.protection=authentication, dfs.data.transfer.server.tcpnodelay=true, dfs.datanode.address=0.0.0.0:1019, dfs.datanode.available-space-volume-choosing-policy.balanced-space-preference-fraction=0.75f, dfs.datanode.available-space-volume-choosing-policy.balanced-space-threshold=10737418240, dfs.datanode.balance.bandwidthPerSec=100m, dfs.datanode.balance.max.concurrent.moves=100, dfs.datanode.block-pinning.enabled=false, dfs.datanode.block.id.layout.upgrade.threads=6, dfs.datanode.bp-ready.timeout=20, dfs.datanode.cache.revocation.polling.ms=500, dfs.datanode.cache.revocation.timeout.ms=900000, dfs.datanode.cached-dfsused.check.interval.ms=600000, dfs.datanode.data.dir=file://${hadoop.tmp.dir}/dfs/data, dfs.datanode.data.dir.perm=700, dfs.datanode.data.transfer.bandwidthPerSec=0, dfs.datanode.data.write.bandwidthPerSec=0, dfs.datanode.directoryscan.interval=21600, dfs.datanode.directoryscan.threads=1, dfs.datanode.directoryscan.throttle.limit.ms.per.sec=1000, dfs.datanode.disk.check.min.gap=15m, dfs.datanode.disk.check.timeout=10m, dfs.datanode.dns.interface=default, dfs.datanode.dns.nameserver=default, dfs.datanode.drop.cache.behind.reads=false, dfs.datanode.drop.cache.behind.writes=false, dfs.datanode.du.reserved=0, dfs.datanode.du.reserved.calculator=org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ReservedSpaceCalculator$ReservedSpaceCalculatorAbsolute, dfs.datanode.du.reserved.pct=0, dfs.datanode.ec.reconstruction.stripedread.buffer.size=65536, dfs.datanode.ec.reconstruction.stripedread.timeout.millis=5000, dfs.datanode.ec.reconstruction.threads=8, dfs.datanode.ec.reconstruction.validation=false, dfs.datanode.ec.reconstruction.xmits.weight=0.5, dfs.datanode.failed.volumes.tolerated=0, dfs.datanode.fileio.profiling.sampling.percentage=0, dfs.datanode.fixed.volume.size=false, dfs.datanode.fsdatasetasyncdisk.max.threads.per.volume=4, dfs.datanode.fsdatasetcache.max.threads.per.volume=4, dfs.datanode.handler.count=10, dfs.datanode.http.address=0.0.0.0:1012, dfs.datanode.http.internal-proxy.port=0, dfs.datanode.https.address=0.0.0.0:9865, dfs.datanode.httpserver.filter.handlers=org.apache.hadoop.hdfs.server.datanode.web.RestCsrfPreventionFilterHandler, dfs.datanode.ipc.address=0.0.0.0:9867, dfs.datanode.kerberos.keytab.file=/etc/security/keytabs/dn.keytab, dfs.datanode.kerberos.principal=dn/dn@EXAMPLE.COM, dfs.datanode.lazywriter.interval.sec=60, dfs.datanode.lock-reporting-threshold-ms=300, dfs.datanode.lock.fair=true, dfs.datanode.lock.read.write.enabled=true, dfs.datanode.max.locked.memory=0, dfs.datanode.max.transfer.threads=4096, dfs.datanode.metrics.logger.period.seconds=600, dfs.datanode.min.outlier.detection.disks=5, dfs.datanode.network.counts.cache.max.size=2147483647, dfs.datanode.oob.timeout-ms=1500,0,0,0, dfs.datanode.outliers.report.interval=30m, dfs.datanode.peer.metrics.min.outlier.detection.samples=1000, dfs.datanode.peer.stats.enabled=false, dfs.datanode.pmem.cache.recovery=true, dfs.datanode.processcommands.threshold=2s, dfs.datanode.readahead.bytes=4194304, dfs.datanode.replica.cache.expiry.time=5m, dfs.datanode.restart.replica.expiration=50, dfs.datanode.scan.period.hours=504, dfs.datanode.shared.file.descriptor.paths=/dev/shm,/tmp, dfs.datanode.slow.io.warning.threshold.ms=300, dfs.datanode.slowdisk.low.threshold.ms=20, dfs.datanode.socket.reuse.keepalive=4000, dfs.datanode.socket.write.timeout=480000, dfs.datanode.sync.behind.writes=false, dfs.datanode.sync.behind.writes.in.background=false, dfs.datanode.transfer.socket.recv.buffer.size=0, dfs.datanode.transfer.socket.send.buffer.size=0, dfs.datanode.transferTo.allowed=true, dfs.datanode.use.datanode.hostname=false, dfs.default.chunk.view.size=32768, dfs.disk.balancer.block.tolerance.percent=10, dfs.disk.balancer.enabled=true, dfs.disk.balancer.max.disk.errors=5, dfs.disk.balancer.max.disk.throughputInMBperSec=10, dfs.disk.balancer.plan.threshold.percent=10, dfs.disk.balancer.plan.valid.interval=1d, dfs.domain.socket.disable.interval.seconds=600, dfs.edit.log.transfer.bandwidthPerSec=0, dfs.edit.log.transfer.timeout=30000, dfs.encrypt.data.overwrite.downstream.derived.qop=false, dfs.encrypt.data.transfer=false, dfs.encrypt.data.transfer.cipher.key.bitlength=128, dfs.ha.allow.stale.reads=false, dfs.ha.automatic-failover.enabled=false, dfs.ha.fencing.ssh.connect-timeout=30000, dfs.ha.log-roll.period=120, dfs.ha.nn.not-become-active-in-safemode=false, dfs.ha.standby.checkpoints=true, dfs.ha.tail-edits.in-progress=false, dfs.ha.tail-edits.namenode-retries=3, dfs.ha.tail-edits.period=60, dfs.ha.tail-edits.period.backoff-max=0, dfs.ha.tail-edits.rolledits.timeout=60, dfs.ha.zkfc.nn.http.timeout.ms=20000, dfs.ha.zkfc.port=8019, dfs.heartbeat.interval=3, dfs.http.client.failover.max.attempts=15, dfs.http.client.failover.sleep.base.millis=500, dfs.http.client.failover.sleep.max.millis=15000, dfs.http.client.retry.max.attempts=10, dfs.http.client.retry.policy.enabled=false, dfs.http.client.retry.policy.spec=10000,6,60000,10, dfs.http.policy=HTTP_ONLY, dfs.https.server.keystore.resource=ssl-server.xml, dfs.image.compress=false, dfs.image.compression.codec=org.apache.hadoop.io.compress.DefaultCodec, dfs.image.parallel.inode.threshold=1000000, dfs.image.parallel.load=false, dfs.image.parallel.target.sections=12, dfs.image.parallel.threads=4, dfs.image.transfer-bootstrap-standby.bandwidthPerSec=0, dfs.image.transfer.bandwidthPerSec=52428800, dfs.image.transfer.chunksize=65536, dfs.image.transfer.timeout=60000, dfs.journalnode.edit-cache-size.bytes=1048576, dfs.journalnode.edits.dir=/tmp/hadoop/dfs/journalnode/, dfs.journalnode.edits.dir.perm=700, dfs.journalnode.enable.sync=true, dfs.journalnode.http-address=0.0.0.0:8480, dfs.journalnode.https-address=0.0.0.0:8481, dfs.journalnode.rpc-address=0.0.0.0:8485, dfs.journalnode.sync.interval=120000, dfs.lock.suppress.warning.interval=10s, dfs.ls.limit=1000, dfs.mover.address=0.0.0.0:0, dfs.mover.keytab.enabled=false, dfs.mover.max-no-move-interval=60000, dfs.mover.movedWinWidth=5400000, dfs.mover.moverThreads=1000, dfs.mover.retry.max.attempts=10, dfs.namenode.accesstime.precision=3600000, dfs.namenode.acls.enabled=true, dfs.namenode.audit.log.async=false, dfs.namenode.audit.log.async.blocking=true, dfs.namenode.audit.log.async.buffer.size=128, dfs.namenode.audit.log.token.tracking.id=false, dfs.namenode.audit.loggers=default, dfs.namenode.available-space-block-placement-policy.balance-local-node=false, dfs.namenode.available-space-block-placement-policy.balanced-space-preference-fraction=0.6, dfs.namenode.available-space-block-placement-policy.balanced-space-tolerance=5, dfs.namenode.available-space-rack-fault-tolerant-block-placement-policy.balanced-space-preference-fraction=0.6, dfs.namenode.available-space-rack-fault-tolerant-block-placement-policy.balanced-space-tolerance=5, dfs.namenode.avoid.read.slow.datanode=false, dfs.namenode.avoid.read.stale.datanode=false, dfs.namenode.avoid.write.stale.datanode=false, dfs.namenode.backup.address=0.0.0.0:50100, dfs.namenode.backup.http-address=0.0.0.0:50105, dfs.namenode.block-placement-policy.default.prefer-local-node=true, dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled=false, dfs.namenode.block.deletion.increment=1000, dfs.namenode.blockreport.max.lock.hold.time=4, dfs.namenode.blockreport.queue.size=1024, dfs.namenode.blocks.per.postponedblocks.rescan=10000, dfs.namenode.caching.enabled=true, dfs.namenode.checkpoint.check.period=60, dfs.namenode.checkpoint.check.quiet-multiplier=1.5, dfs.namenode.checkpoint.dir=file://${hadoop.tmp.dir}/dfs/namesecondary, dfs.namenode.checkpoint.edits.dir=${dfs.namenode.checkpoint.dir}, dfs.namenode.checkpoint.max-retries=3, dfs.namenode.checkpoint.period=3600, dfs.namenode.checkpoint.txns=1000000, dfs.namenode.corrupt.block.delete.immediately.enabled=true, dfs.namenode.datanode.registration.ip-hostname-check=true, dfs.namenode.decommission.backoff.monitor.pending.blocks.per.lock=1000, dfs.namenode.decommission.backoff.monitor.pending.limit=10000, dfs.namenode.decommission.blocks.per.interval=500000, dfs.namenode.decommission.interval=30, dfs.namenode.decommission.max.concurrent.tracked.nodes=100, dfs.namenode.decommission.monitor.class=org.apache.hadoop.hdfs.server.blockmanagement.DatanodeAdminDefaultMonitor, dfs.namenode.delegation.key.update-interval=86400000, dfs.namenode.delegation.token.always-use=false, dfs.namenode.delegation.token.max-lifetime=604800000, dfs.namenode.delegation.token.renew-interval=86400000, dfs.namenode.ec.policies.max.cellsize=4194304, dfs.namenode.ec.system.default.policy=RS-6-3-1024k, dfs.namenode.ec.userdefined.policy.allowed=true, dfs.namenode.edekcacheloader.initial.delay.ms=3000, dfs.namenode.edekcacheloader.interval.ms=1000, dfs.namenode.edit.log.autoroll.check.interval.ms=300000, dfs.namenode.edit.log.autoroll.multiplier.threshold=0.5, dfs.namenode.edits.asynclogging=true, dfs.namenode.edits.dir=${dfs.namenode.name.dir}, dfs.namenode.edits.dir.minimum=1, dfs.namenode.edits.journal-plugin.qjournal=org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager, dfs.namenode.edits.noeditlogchannelflush=false, dfs.namenode.enable.log.stale.datanode=false, dfs.namenode.enable.retrycache=true, dfs.namenode.file.close.num-committed-allowed=0, dfs.namenode.fs-limits.max-blocks-per-file=10000, dfs.namenode.fs-limits.max-component-length=255, dfs.namenode.fs-limits.max-directory-items=1048576, dfs.namenode.fs-limits.max-xattr-size=16384, dfs.namenode.fs-limits.max-xattrs-per-inode=32, dfs.namenode.fs-limits.min-block-size=1048576, dfs.namenode.fslock.fair=true, dfs.namenode.full.block.report.lease.length.ms=300000, dfs.namenode.gc.time.monitor.enable=true, dfs.namenode.gc.time.monitor.observation.window.ms=1m, dfs.namenode.gc.time.monitor.sleep.interval.ms=5s, dfs.namenode.get-blocks.max-qps=20, dfs.namenode.handler.count=10, dfs.namenode.heartbeat.recheck-interval=300000, dfs.namenode.hosts.provider.classname=org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager, dfs.namenode.http-address=0.0.0.0:9870, dfs.namenode.https-address=0.0.0.0:9871, dfs.namenode.inotify.max.events.per.rpc=1000, dfs.namenode.invalidate.work.pct.per.iteration=0.32f, dfs.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, dfs.namenode.kerberos.principal.pattern=*, dfs.namenode.lazypersist.file.scrub.interval.sec=300, dfs.namenode.lease-hard-limit-sec=1200, dfs.namenode.lease-recheck-interval-ms=2000, dfs.namenode.lifeline.handler.ratio=0.10, dfs.namenode.list.cache.directives.num.responses=100, dfs.namenode.list.cache.pools.num.responses=100, dfs.namenode.list.encryption.zones.num.responses=100, dfs.namenode.list.openfiles.num.responses=1000, dfs.namenode.list.reencryption.status.num.responses=100, dfs.namenode.lock.detailed-metrics.enabled=false, dfs.namenode.maintenance.replication.min=1, dfs.namenode.max-corrupt-file-blocks-returned=100, dfs.namenode.max-lock-hold-to-release-lease-ms=25, dfs.namenode.max-num-blocks-to-log=1000, dfs.namenode.max.extra.edits.segments.retained=10000, dfs.namenode.max.full.block.report.leases=6, dfs.namenode.max.objects=0, dfs.namenode.max.op.size=52428800, dfs.namenode.max.slowpeer.collect.nodes=5, dfs.namenode.metrics.logger.period.seconds=600, dfs.namenode.missing.checkpoint.periods.before.shutdown=3, dfs.namenode.name.cache.threshold=10, dfs.namenode.name.dir=file://${hadoop.tmp.dir}/dfs/name, dfs.namenode.name.dir.restore=false, dfs.namenode.num.checkpoints.retained=2, dfs.namenode.num.extra.edits.retained=1000000, dfs.namenode.path.based.cache.block.map.allocation.percent=0.25, dfs.namenode.path.based.cache.refresh.interval.ms=30000, dfs.namenode.path.based.cache.retry.interval.ms=30000, dfs.namenode.posix.acl.inheritance.enabled=true, dfs.namenode.provided.enabled=false, dfs.namenode.quota.init-threads=12, dfs.namenode.read-lock-reporting-threshold-ms=5000, dfs.namenode.read.considerLoad=false, dfs.namenode.read.considerStorageType=false, dfs.namenode.reconstruction.pending.timeout-sec=300, dfs.namenode.redundancy.considerLoad=true, dfs.namenode.redundancy.considerLoad.factor=2.0, dfs.namenode.redundancy.considerLoadByStorageType=false, dfs.namenode.redundancy.interval.seconds=3, dfs.namenode.redundancy.queue.restart.iterations=2400, dfs.namenode.reencrypt.batch.size=1000, dfs.namenode.reencrypt.edek.threads=10, dfs.namenode.reencrypt.sleep.interval=1m, dfs.namenode.reencrypt.throttle.limit.handler.ratio=1.0, dfs.namenode.reencrypt.throttle.limit.updater.ratio=1.0, dfs.namenode.reject-unresolved-dn-topology-mapping=false, dfs.namenode.replication.max-streams=2, dfs.namenode.replication.max-streams-hard-limit=4, dfs.namenode.replication.min=1, dfs.namenode.replication.work.multiplier.per.iteration=2, dfs.namenode.resource.check.interval=5000, dfs.namenode.resource.checked.volumes.minimum=1, dfs.namenode.resource.du.reserved=104857600, dfs.namenode.retrycache.expirytime.millis=600000, dfs.namenode.retrycache.heap.percent=0.03f, dfs.namenode.safemode.extension=30000, dfs.namenode.safemode.min.datanodes=0, dfs.namenode.safemode.threshold-pct=0.999f, dfs.namenode.secondary.http-address=0.0.0.0:9868, dfs.namenode.secondary.https-address=0.0.0.0:9869, dfs.namenode.send.qop.enabled=false, dfs.namenode.service.handler.count=10, dfs.namenode.slowpeer.collect.interval=30m, dfs.namenode.snapshot.capture.openfiles=false, dfs.namenode.snapshot.max.limit=65536, dfs.namenode.snapshot.skip.capture.accesstime-only-change=false, dfs.namenode.snapshot.skiplist.interval=10, dfs.namenode.snapshot.skiplist.max.levels=0, dfs.namenode.snapshotdiff.allow.snap-root-descendant=true, dfs.namenode.snapshotdiff.listing.limit=1000, dfs.namenode.stale.datanode.interval=30000, dfs.namenode.stale.datanode.minimum.interval=3, dfs.namenode.startup.delay.block.deletion.sec=0, dfs.namenode.state.context.enabled=false, dfs.namenode.storage.dir.perm=700, dfs.namenode.support.allow.format=true, dfs.namenode.top.enabled=true, dfs.namenode.top.num.users=10, dfs.namenode.top.window.num.buckets=10, dfs.namenode.top.windows.minutes=1,5,25, dfs.namenode.upgrade.domain.factor=${dfs.replication}, dfs.namenode.write-lock-reporting-threshold-ms=5000, dfs.namenode.write.stale.datanode.ratio=0.5f, dfs.namenode.xattrs.enabled=true, dfs.net.topology.impl=org.apache.hadoop.hdfs.net.DFSNetworkTopology, dfs.permissions.ContentSummary.subAccess=false, dfs.permissions.allow.owner.set.quota=false, dfs.permissions.enabled=true, dfs.permissions.superusergroup=supergroup, dfs.pipeline.ecn=false, dfs.protected.subdirectories.enable=false, dfs.provided.acls.import.enabled=false, dfs.provided.aliasmap.class=org.apache.hadoop.hdfs.server.common.blockaliasmap.impl.TextFileRegionAliasMap, dfs.provided.aliasmap.inmemory.batch-size=500, dfs.provided.aliasmap.inmemory.enabled=false, dfs.provided.aliasmap.inmemory.leveldb.dir=/tmp, dfs.provided.aliasmap.inmemory.server.log=false, dfs.provided.aliasmap.load.retries=0, dfs.provided.aliasmap.text.delimiter=,, dfs.provided.storage.id=DS-PROVIDED, dfs.qjm.operations.timeout=60s, dfs.qjournal.accept-recovery.timeout.ms=120000, dfs.qjournal.finalize-segment.timeout.ms=120000, dfs.qjournal.get-journal-state.timeout.ms=120000, dfs.qjournal.http.open.timeout.ms=60000, dfs.qjournal.http.read.timeout.ms=60000, dfs.qjournal.new-epoch.timeout.ms=120000, dfs.qjournal.parallel-read.num-threads=5, dfs.qjournal.prepare-recovery.timeout.ms=120000, dfs.qjournal.queued-edits.limit.mb=10, dfs.qjournal.select-input-streams.timeout.ms=20000, dfs.qjournal.start-segment.timeout.ms=20000, dfs.qjournal.write-txns.timeout.ms=20000, dfs.quota.by.storage.type.enabled=true, dfs.ratis.leader.election.minimum.timeout.duration=5s, dfs.ratis.server.retry-cache.timeout.duration=600000ms, dfs.ratis.snapshot.threshold=10000, dfs.reformat.disabled=false, dfs.replication=3, dfs.replication.max=512, dfs.secondary.namenode.kerberos.internal.spnego.principal=${dfs.web.authentication.kerberos.principal}, dfs.short.circuit.shared.memory.watcher.interrupt.check.ms=60000, dfs.storage.policy.enabled=true, dfs.storage.policy.permissions.superuser-only=false, dfs.storage.policy.satisfier.address=0.0.0.0:0, dfs.storage.policy.satisfier.datanode.cache.refresh.interval.ms=300000, dfs.storage.policy.satisfier.max.outstanding.paths=10000, dfs.storage.policy.satisfier.mode=none, dfs.storage.policy.satisfier.queue.limit=1000, dfs.storage.policy.satisfier.recheck.timeout.millis=60000, dfs.storage.policy.satisfier.retry.max.attempts=3, dfs.storage.policy.satisfier.self.retry.timeout.millis=300000, dfs.storage.policy.satisfier.work.multiplier.per.iteration=1, dfs.stream-buffer-size=4096, dfs.use.dfs.network.topology=true, dfs.user.home.dir.prefix=/user, dfs.web.authentication.kerberos.keytab=/etc/security/keytabs/HTTP.keytab, dfs.web.authentication.kerberos.principal=HTTP/ozone@EXAMPLE.COM, dfs.webhdfs.acl.provider.permission.pattern=^(default:)?(user|group|mask|other):[[A-Za-z_][A-Za-z0-9._-]]*:([rwx-]{3})?(,(default:)?(user|group|mask|other):[[A-Za-z_][A-Za-z0-9._-]]*:([rwx-]{3})?)*$, dfs.webhdfs.netty.high.watermark=65535, dfs.webhdfs.netty.low.watermark=32768, dfs.webhdfs.oauth2.enabled=false, dfs.webhdfs.rest-csrf.browser-useragents-regex=^Mozilla.*,^Opera.*, dfs.webhdfs.rest-csrf.custom-header=X-XSRF-HEADER, dfs.webhdfs.rest-csrf.enabled=false, dfs.webhdfs.rest-csrf.methods-to-ignore=GET,OPTIONS,HEAD,TRACE, dfs.webhdfs.socket.connect-timeout=60s, dfs.webhdfs.socket.read-timeout=60s, dfs.webhdfs.ugi.expire.after.access=600000, dfs.webhdfs.use.ipc.callq=true, dfs.webhdfs.user.provider.user.pattern=^[A-Za-z_][A-Za-z0-9._-]*[$]?$, dfs.xframe.enabled=true, dfs.xframe.value=SAMEORIGIN, file.blocksize=67108864, file.bytes-per-checksum=512, file.client-write-packet-size=65536, file.replication=1, file.stream-buffer-size=4096, fs.AbstractFileSystem.abfs.impl=org.apache.hadoop.fs.azurebfs.Abfs, fs.AbstractFileSystem.abfss.impl=org.apache.hadoop.fs.azurebfs.Abfss, fs.AbstractFileSystem.adl.impl=org.apache.hadoop.fs.adl.Adl, fs.AbstractFileSystem.file.impl=org.apache.hadoop.fs.local.LocalFs, fs.AbstractFileSystem.ftp.impl=org.apache.hadoop.fs.ftp.FtpFs, fs.AbstractFileSystem.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS, fs.AbstractFileSystem.har.impl=org.apache.hadoop.fs.HarFs, fs.AbstractFileSystem.hdfs.impl=org.apache.hadoop.fs.Hdfs, fs.AbstractFileSystem.s3a.impl=org.apache.hadoop.fs.s3a.S3A, fs.AbstractFileSystem.swebhdfs.impl=org.apache.hadoop.fs.SWebHdfs, fs.AbstractFileSystem.viewfs.impl=org.apache.hadoop.fs.viewfs.ViewFs, fs.AbstractFileSystem.wasb.impl=org.apache.hadoop.fs.azure.Wasb, fs.AbstractFileSystem.wasbs.impl=org.apache.hadoop.fs.azure.Wasbs, fs.AbstractFileSystem.webhdfs.impl=org.apache.hadoop.fs.WebHdfs, fs.abfs.impl=org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem, fs.abfss.impl=org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem, fs.adl.impl=org.apache.hadoop.fs.adl.AdlFileSystem, fs.adl.oauth2.access.token.provider.type=<redacted>, fs.automatic.close=true, fs.azure.authorization=false, fs.azure.authorization.caching.enable=true, fs.azure.buffer.dir=${hadoop.tmp.dir}/abfs, fs.azure.local.sas.key.mode=false, fs.azure.sas.expiry.period=90d, fs.azure.saskey.usecontainersaskeyforallaccess=true, fs.azure.secure.mode=false, fs.azure.user.agent.prefix=unknown, fs.client.resolve.remote.symlinks=true, fs.client.resolve.topology.enabled=false, fs.defaultFS=ofs://om, fs.df.interval=60000, fs.du.interval=600000, fs.ftp.data.connection.mode=ACTIVE_LOCAL_DATA_CONNECTION_MODE, fs.ftp.host=0.0.0.0, fs.ftp.host.port=21, fs.ftp.impl=org.apache.hadoop.fs.ftp.FTPFileSystem, fs.ftp.timeout=0, fs.ftp.transfer.mode=BLOCK_TRANSFER_MODE, fs.getspaceused.jitterMillis=60000, fs.har.impl.disable.cache=true, fs.permissions.umask-mode=022, fs.s3a.accesspoint.required=false, fs.s3a.assumed.role.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider, fs.s3a.assumed.role.session.duration=30m, fs.s3a.attempts.maximum=20, fs.s3a.aws.credentials.provider=
    org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider,
    org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider,
    com.amazonaws.auth.EnvironmentVariableCredentialsProvider,
    org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
  , fs.s3a.block.size=32M, fs.s3a.buffer.dir=${hadoop.tmp.dir}/s3a, fs.s3a.change.detection.mode=server, fs.s3a.change.detection.source=etag, fs.s3a.change.detection.version.required=true, fs.s3a.committer.abort.pending.uploads=true, fs.s3a.committer.magic.enabled=true, fs.s3a.committer.name=file, fs.s3a.committer.staging.conflict-mode=append, fs.s3a.committer.staging.tmp.path=tmp/staging, fs.s3a.committer.staging.unique-filenames=true, fs.s3a.committer.threads=8, fs.s3a.connection.establish.timeout=5000, fs.s3a.connection.maximum=96, fs.s3a.connection.request.timeout=0, fs.s3a.connection.ssl.enabled=true, fs.s3a.connection.timeout=200000, fs.s3a.downgrade.syncable.exceptions=true, fs.s3a.etag.checksum.enabled=false, fs.s3a.executor.capacity=16, fs.s3a.fast.upload.active.blocks=4, fs.s3a.fast.upload.buffer=disk, fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem, fs.s3a.list.version=2, fs.s3a.max.total.tasks=32, fs.s3a.metadatastore.authoritative=false, fs.s3a.metadatastore.fail.on.write.error=true, fs.s3a.metadatastore.impl=org.apache.hadoop.fs.s3a.s3guard.NullMetadataStore, fs.s3a.metadatastore.metadata.ttl=15m, fs.s3a.multiobjectdelete.enable=true, fs.s3a.multipart.purge=false, fs.s3a.multipart.purge.age=86400, fs.s3a.multipart.size=64M, fs.s3a.multipart.threshold=128M, fs.s3a.paging.maximum=5000, fs.s3a.path.style.access=false, fs.s3a.readahead.range=64K, fs.s3a.retry.interval=500ms, fs.s3a.retry.limit=7, fs.s3a.retry.throttle.interval=100ms, fs.s3a.retry.throttle.limit=20, fs.s3a.s3guard.cli.prune.age=86400000, fs.s3a.s3guard.consistency.retry.interval=2s, fs.s3a.s3guard.consistency.retry.limit=7, fs.s3a.s3guard.ddb.background.sleep=25ms, fs.s3a.s3guard.ddb.max.retries=9, fs.s3a.s3guard.ddb.table.capacity.read=0, fs.s3a.s3guard.ddb.table.capacity.write=0, fs.s3a.s3guard.ddb.table.create=false, fs.s3a.s3guard.ddb.table.sse.enabled=false, fs.s3a.s3guard.ddb.throttle.retry.interval=100ms, fs.s3a.select.enabled=true, fs.s3a.select.errors.include.sql=false, fs.s3a.select.input.compression=none, fs.s3a.select.input.csv.comment.marker=#, fs.s3a.select.input.csv.field.delimiter=,, fs.s3a.select.input.csv.header=none, fs.s3a.select.input.csv.quote.character=", fs.s3a.select.input.csv.quote.escape.character=\\, fs.s3a.select.input.csv.record.delimiter=\n, fs.s3a.select.output.csv.field.delimiter=,, fs.s3a.select.output.csv.quote.character=", fs.s3a.select.output.csv.quote.escape.character=\\, fs.s3a.select.output.csv.quote.fields=always, fs.s3a.select.output.csv.record.delimiter=\n, fs.s3a.socket.recv.buffer=8192, fs.s3a.socket.send.buffer=8192, fs.s3a.ssl.channel.mode=default_jsse, fs.s3a.threads.keepalivetime=60, fs.s3a.threads.max=64, fs.swift.impl=org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem, fs.trash.checkpoint.interval=0, fs.trash.classname=org.apache.hadoop.ozone.om.TrashPolicyOzone, fs.trash.interval=1, fs.viewfs.overload.scheme.target.abfs.impl=org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem, fs.viewfs.overload.scheme.target.abfss.impl=org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem, fs.viewfs.overload.scheme.target.file.impl=org.apache.hadoop.fs.LocalFileSystem, fs.viewfs.overload.scheme.target.ftp.impl=org.apache.hadoop.fs.ftp.FTPFileSystem, fs.viewfs.overload.scheme.target.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS, fs.viewfs.overload.scheme.target.hdfs.impl=org.apache.hadoop.hdfs.DistributedFileSystem, fs.viewfs.overload.scheme.target.http.impl=org.apache.hadoop.fs.http.HttpFileSystem, fs.viewfs.overload.scheme.target.https.impl=org.apache.hadoop.fs.http.HttpsFileSystem, fs.viewfs.overload.scheme.target.o3fs.impl=org.apache.hadoop.fs.ozone.OzoneFileSystem, fs.viewfs.overload.scheme.target.ofs.impl=org.apache.hadoop.fs.ozone.RootedOzoneFileSystem, fs.viewfs.overload.scheme.target.oss.impl=org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem, fs.viewfs.overload.scheme.target.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem, fs.viewfs.overload.scheme.target.swebhdfs.impl=org.apache.hadoop.hdfs.web.SWebHdfsFileSystem, fs.viewfs.overload.scheme.target.swift.impl=org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem, fs.viewfs.overload.scheme.target.wasb.impl=org.apache.hadoop.fs.azure.NativeAzureFileSystem, fs.viewfs.overload.scheme.target.webhdfs.impl=org.apache.hadoop.hdfs.web.WebHdfsFileSystem, fs.viewfs.rename.strategy=SAME_MOUNTPOINT, fs.wasb.impl=org.apache.hadoop.fs.azure.NativeAzureFileSystem, fs.wasbs.impl=org.apache.hadoop.fs.azure.NativeAzureFileSystem$Secure, ftp.blocksize=67108864, ftp.bytes-per-checksum=512, ftp.client-write-packet-size=65536, ftp.replication=3, ftp.stream-buffer-size=4096, ha.failover-controller.active-standby-elector.zk.op.retries=3, ha.failover-controller.cli-check.rpc-timeout.ms=20000, ha.failover-controller.graceful-fence.connection.retries=1, ha.failover-controller.graceful-fence.rpc-timeout.ms=5000, ha.failover-controller.new-active.rpc-timeout.ms=60000, ha.health-monitor.check-interval.ms=1000, ha.health-monitor.connect-retry-interval.ms=1000, ha.health-monitor.rpc-timeout.ms=45000, ha.health-monitor.rpc.connect.max.retries=1, ha.health-monitor.sleep-after-disconnect.ms=1000, ha.zookeeper.acl=world:anyone:rwcda, ha.zookeeper.parent-znode=/hadoop-ha, ha.zookeeper.session-timeout.ms=10000, hadoop.caller.context.enabled=false, hadoop.caller.context.max.size=128, hadoop.caller.context.signature.max.size=40, hadoop.common.configuration.version=3.0.0, hadoop.domainname.resolver.impl=org.apache.hadoop.net.DNSDomainNameResolver, hadoop.fuse.connection.timeout=300, hadoop.fuse.timer.period=5, hadoop.hdds.db.rocksdb.logging.enabled=false, hadoop.hdds.db.rocksdb.logging.level=INFO, hadoop.hdds.db.rocksdb.writeoption.sync=false, hadoop.hdfs.configuration.version=1, hadoop.http.authentication.kerberos.keytab=/etc/security/keytabs/HTTP.keytab, hadoop.http.authentication.kerberos.principal=HTTP/ozone@EXAMPLE.COM, hadoop.http.authentication.signature.secret.file=/etc/security/http_secret, hadoop.http.authentication.simple.anonymous.allowed=false, hadoop.http.authentication.token.validity=36000, hadoop.http.authentication.type=kerberos, hadoop.http.cross-origin.allowed-headers=X-Requested-With,Content-Type,Accept,Origin, hadoop.http.cross-origin.allowed-methods=GET,POST,HEAD, hadoop.http.cross-origin.allowed-origins=*, hadoop.http.cross-origin.enabled=false, hadoop.http.cross-origin.max-age=1800, hadoop.http.filter.initializers=org.apache.hadoop.http.lib.StaticUserWebFilter, hadoop.http.idle_timeout.ms=60000, hadoop.http.logs.enabled=true, hadoop.http.sni.host.check.enabled=false, hadoop.http.staticuser.user=dr.who, hadoop.jetty.logs.serve.aliases=true, hadoop.kerberos.keytab.login.autorenewal.enabled=false, hadoop.kerberos.kinit.command=kinit, hadoop.kerberos.min.seconds.before.relogin=60, hadoop.metrics.jvm.use-thread-mxbean=false, hadoop.prometheus.endpoint.enabled=false, hadoop.registry.jaas.context=Client, hadoop.registry.secure=false, hadoop.registry.system.acls=sasl:yarn@, sasl:mapred@, sasl:hdfs@, hadoop.registry.zk.connection.timeout.ms=15000, hadoop.registry.zk.quorum=localhost:2181, hadoop.registry.zk.retry.ceiling.ms=60000, hadoop.registry.zk.retry.interval.ms=1000, hadoop.registry.zk.retry.times=5, hadoop.registry.zk.root=/registry, hadoop.registry.zk.session.timeout.ms=60000, hadoop.rpc.protection=authentication, hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.StandardSocketFactory, hadoop.security.auth_to_local=DEFAULT, hadoop.security.auth_to_local.mechanism=hadoop, hadoop.security.authentication=kerberos, hadoop.security.authorization=true, hadoop.security.credential.clear-text-fallback=true, hadoop.security.crypto.buffer.size=8192, hadoop.security.crypto.cipher.suite=AES/CTR/NoPadding, hadoop.security.crypto.codec.classes.aes.ctr.nopadding=org.apache.hadoop.crypto.OpensslAesCtrCryptoCodec, org.apache.hadoop.crypto.JceAesCtrCryptoCodec, hadoop.security.dns.log-slow-lookups.enabled=false, hadoop.security.dns.log-slow-lookups.threshold.ms=1000, hadoop.security.group.mapping=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback, hadoop.security.group.mapping.ldap.connection.timeout.ms=60000, hadoop.security.group.mapping.ldap.conversion.rule=none, hadoop.security.group.mapping.ldap.directory.search.timeout=10000, hadoop.security.group.mapping.ldap.num.attempts=3, hadoop.security.group.mapping.ldap.num.attempts.before.failover=3, hadoop.security.group.mapping.ldap.posix.attr.gid.name=gidNumber, hadoop.security.group.mapping.ldap.posix.attr.uid.name=uidNumber, hadoop.security.group.mapping.ldap.read.timeout.ms=60000, hadoop.security.group.mapping.ldap.search.attr.group.name=cn, hadoop.security.group.mapping.ldap.search.attr.member=member, hadoop.security.group.mapping.ldap.search.filter.group=(objectClass=group), hadoop.security.group.mapping.ldap.search.filter.user=(&(objectClass=user)(sAMAccountName={0})), hadoop.security.group.mapping.ldap.search.group.hierarchy.levels=0, hadoop.security.group.mapping.ldap.ssl=false, hadoop.security.group.mapping.providers.combined=true, hadoop.security.groups.cache.background.reload=false, hadoop.security.groups.cache.background.reload.threads=3, hadoop.security.groups.cache.secs=300, hadoop.security.groups.cache.warn.after.ms=5000, hadoop.security.groups.negative-cache.secs=30, hadoop.security.groups.shell.command.timeout=0s, hadoop.security.instrumentation.requires.admin=false, hadoop.security.java.secure.random.algorithm=SHA1PRNG, hadoop.security.key.default.bitlength=128, hadoop.security.key.default.cipher=AES/CTR/NoPadding, hadoop.security.key.provider.path=kms://http@kms:9600/kms, hadoop.security.kms.client.authentication.retry-count=1, hadoop.security.kms.client.encrypted.key.cache.expiry=43200000, hadoop.security.kms.client.encrypted.key.cache.low-watermark=0.3f, hadoop.security.kms.client.encrypted.key.cache.num.refill.threads=2, hadoop.security.kms.client.encrypted.key.cache.size=500, hadoop.security.kms.client.failover.sleep.base.millis=100, hadoop.security.kms.client.failover.sleep.max.millis=2000, hadoop.security.kms.client.timeout=60, hadoop.security.random.device.file.path=/dev/urandom, hadoop.security.secure.random.impl=org.apache.hadoop.crypto.random.OpensslSecureRandom, hadoop.security.sensitive-config-keys=<redacted>, hadoop.security.token.service.use_ip=true, hadoop.security.uid.cache.secs=14400, hadoop.service.shutdown.timeout=30s, hadoop.shell.missing.defaultFs.warning=false, hadoop.shell.safely.delete.limit.num.files=100, hadoop.ssl.client.conf=ssl-client.xml, hadoop.ssl.enabled.protocols=TLSv1.2, hadoop.ssl.hostname.verifier=DEFAULT, hadoop.ssl.keystores.factory.class=org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory, hadoop.ssl.require.client.cert=false, hadoop.ssl.server.conf=ssl-server.xml, hadoop.system.tags=YARN,HDFS,NAMENODE,DATANODE,REQUIRED,SECURITY,KERBEROS,PERFORMANCE,CLIENT
      ,SERVER,DEBUG,DEPRECATED,COMMON,OPTIONAL, hadoop.tags.custom=OZONE,MANAGEMENT,SECURITY,PERFORMANCE,DEBUG,CLIENT,SERVER,OM,SCM,
      CRITICAL,RATIS,CONTAINER,REQUIRED,REST,STORAGE,PIPELINE,STANDALONE,S3GATEWAY,RECON, hadoop.tags.system=YARN,HDFS,NAMENODE,DATANODE,REQUIRED,SECURITY,KERBEROS,PERFORMANCE,CLIENT
      ,SERVER,DEBUG,DEPRECATED,COMMON,OPTIONAL, hadoop.tmp.dir=/tmp/hadoop-${user.name}, hadoop.user.group.static.mapping.overrides=dr.who=;, hadoop.util.hash.type=murmur, hadoop.workaround.non.threadsafe.getpwuid=true, hadoop.zk.acl=world:anyone:rwcda, hadoop.zk.num-retries=1000, hadoop.zk.retry-interval-ms=1000, hadoop.zk.timeout-ms=10000, hdds.block.token.enabled=true, hdds.block.token.expiry.time=1d, hdds.command.status.report.interval=30s, hdds.container.action.max.limit=20, hdds.container.checksum.verification.enabled=true, hdds.container.close.threshold=0.9f, hdds.container.replication.compression=NO_COMPRESSION, hdds.container.report.interval=60s, hdds.container.token.enabled=true, hdds.crl.status.report.interval=60000ms, hdds.datanode.df.refresh.period=5m, hdds.datanode.dir=/data/hdds, hdds.datanode.du.refresh.period=1h, hdds.datanode.http-address=0.0.0.0:9882, hdds.datanode.http-bind-host=0.0.0.0, hdds.datanode.http.auth.kerberos.keytab=/etc/security/keytabs/dn.keytab, hdds.datanode.http.auth.kerberos.principal=HTTP/dn@EXAMPLE.COM, hdds.datanode.http.auth.type=kerberos, hdds.datanode.http.enabled=true, hdds.datanode.https-address=0.0.0.0:9883, hdds.datanode.https-bind-host=0.0.0.0, hdds.datanode.metadata.rocksdb.cache.size=64MB, hdds.datanode.ratis.server.request.timeout=2m, hdds.datanode.storage.utilization.critical.threshold=0.95, hdds.datanode.storage.utilization.warning.threshold=0.75, hdds.db.profile=DISK, hdds.grpc.tls.enabled=true, hdds.grpc.tls.provider=OPENSSL, hdds.heartbeat.interval=30s, hdds.key.dir.name=keys, hdds.key.len=2048, hdds.node.report.interval=60000ms, hdds.pipeline.action.max.limit=20, hdds.pipeline.report.interval=60000ms, hdds.priv.key.file.name=private.pem, hdds.profiler.endpoint.enabled=false, hdds.prometheus.endpoint.enabled=true, hdds.public.key.file.name=public.pem, hdds.ratis.client.exponential.backoff.base.sleep=4s, hdds.ratis.client.exponential.backoff.max.sleep=40s, hdds.ratis.client.multilinear.random.retry.policy=5s, 5, 10s, 5, 15s, 5, 20s, 5, 25s, 5, 60s, 10, hdds.ratis.client.request.watch.timeout=3m, hdds.ratis.client.request.write.timeout=5m, hdds.ratis.client.retry.policy=org.apache.hadoop.hdds.ratis.retrypolicy.RequestTypeDependentRetryPolicyCreator, hdds.ratis.client.retrylimited.max.retries=180, hdds.ratis.client.retrylimited.retry.interval=1s, hdds.ratis.raft.client.async.outstanding-requests.max=32, hdds.ratis.raft.client.rpc.request.timeout=60s, hdds.ratis.raft.client.rpc.watch.request.timeout=180s, hdds.ratis.raft.grpc.flow.control.window=5MB, hdds.ratis.raft.grpc.message.size.max=32MB, hdds.ratis.raft.server.datastream.client.pool.size=10, hdds.ratis.raft.server.datastream.request.threads=20, hdds.ratis.raft.server.delete.ratis.log.directory=true, hdds.ratis.raft.server.leaderelection.pre-vote=false, hdds.ratis.raft.server.notification.no-leader.timeout=300s, hdds.ratis.raft.server.rpc.request.timeout=60s, hdds.ratis.raft.server.rpc.slowness.timeout=300s, hdds.ratis.raft.server.watch.timeout=180s, hdds.ratis.raft.server.write.element-limit=1024, hdds.recon.heartbeat.interval=60s, hdds.rest.http-address=0.0.0.0:9880, hdds.rest.netty.high.watermark=65535, hdds.rest.netty.low.watermark=32768, hdds.rest.rest-csrf.enabled=false, hdds.scm.block.deleting.service.interval=60s, hdds.scm.block.deletion.per-interval.max=100000, hdds.scm.http.auth.kerberos.keytab=/etc/security/keytabs/scm.keytab, hdds.scm.http.auth.kerberos.principal=HTTP/scm@EXAMPLE.COM, hdds.scm.http.auth.type=kerberos, hdds.scm.init.default.layout.version=-1, hdds.scm.kerberos.keytab.file=/etc/security/keytabs/scm.keytab, hdds.scm.kerberos.principal=scm/scm@EXAMPLE.COM, hdds.scm.pipeline.choose.policy.impl=org.apache.hadoop.hdds.scm.pipeline.choose.algorithms.RandomPipelineChoosePolicy, hdds.scm.replication.event.timeout=10s, hdds.scm.replication.push=true, hdds.scm.replication.thread.interval=5s, hdds.scm.safemode.atleast.one.node.reported.pipeline.pct=0.90, hdds.scm.safemode.enabled=true, hdds.scm.safemode.healthy.pipeline.pct=0.10, hdds.scm.safemode.min.datanode=1, hdds.scm.safemode.pipeline-availability.check=true, hdds.scm.safemode.pipeline.creation=true, hdds.scm.safemode.threshold.pct=0.99, hdds.scm.unknown-container.action=WARN, hdds.scm.wait.time.after.safemode.exit=5m, hdds.scmclient.failover.max.retry=15, hdds.scmclient.failover.retry.interval=2s, hdds.scmclient.max.retry.timeout=30s, hdds.scmclient.rpc.timeout=15m, hdds.security.client.datanode.container.protocol.acl=*, hdds.security.client.scm.block.protocol.acl=*, hdds.security.client.scm.certificate.protocol.acl=*, hdds.security.client.scm.container.protocol.acl=*, hdds.security.ssl.keystore.reload.interval=60s, hdds.security.ssl.truststore.reload.interval=60s, hdds.tracing.enabled=false, hdds.x509.default.duration=P365D, hdds.x509.dir.name=certs, hdds.x509.file.name=certificate.crt, hdds.x509.max.duration=P1865D, hdds.x509.renew.grace.duration=P28D, hdds.x509.signature.algorithm=SHA256withRSA, httpfs.buffer.size=4096, io.bytes.per.checksum=512, io.compression.codec.bzip2.library=system-native, io.erasurecode.codec.rs-legacy.rawcoders=rs-legacy_java, io.erasurecode.codec.rs.rawcoders=rs_native,rs_java, io.erasurecode.codec.xor.rawcoders=xor_native,xor_java, io.file.buffer.size=4096, io.map.index.interval=128, io.map.index.skip=0, io.mapfile.bloom.error.rate=0.005, io.mapfile.bloom.size=1048576, io.seqfile.compress.blocksize=1000000, io.seqfile.local.dir=${hadoop.tmp.dir}/io/local, io.serializations=org.apache.hadoop.io.serializer.WritableSerialization, org.apache.hadoop.io.serializer.avro.AvroSpecificSerialization, org.apache.hadoop.io.serializer.avro.AvroReflectSerialization, io.skip.checksum.errors=false, ipc.[port_number].backoff.enable=false, ipc.[port_number].callqueue.impl=java.util.concurrent.LinkedBlockingQueue, ipc.[port_number].cost-provider.impl=org.apache.hadoop.ipc.DefaultCostProvider, ipc.[port_number].decay-scheduler.backoff.responsetime.enable=false, ipc.[port_number].decay-scheduler.backoff.responsetime.thresholds=10s,20s,30s,40s, ipc.[port_number].decay-scheduler.decay-factor=0.5, ipc.[port_number].decay-scheduler.metrics.top.user.count=10, ipc.[port_number].decay-scheduler.period-ms=5000, ipc.[port_number].decay-scheduler.thresholds=13,25,50, ipc.[port_number].faircallqueue.multiplexer.weights=8,4,2,1, ipc.[port_number].identity-provider.impl=org.apache.hadoop.ipc.UserIdentityProvider, ipc.[port_number].scheduler.impl=org.apache.hadoop.ipc.DefaultRpcScheduler, ipc.[port_number].scheduler.priority.levels=4, ipc.[port_number].weighted-cost.handler=1, ipc.[port_number].weighted-cost.lockexclusive=100, ipc.[port_number].weighted-cost.lockfree=1, ipc.[port_number].weighted-cost.lockshared=10, ipc.[port_number].weighted-cost.response=1, ipc.client.bind.wildcard.addr=false, ipc.client.connect.max.retries=10, ipc.client.connect.max.retries.on.timeouts=45, ipc.client.connect.retry.interval=1000, ipc.client.connect.timeout=20000, ipc.client.connection.maxidletime=10000, ipc.client.fallback-to-simple-auth-allowed=false, ipc.client.idlethreshold=4000, ipc.client.kill.max=10, ipc.client.low-latency=false, ipc.client.ping=true, ipc.client.rpc-timeout.ms=0, ipc.client.tcpnodelay=true, ipc.maximum.data.length=134217728, ipc.maximum.response.length=134217728, ipc.ping.interval=60000, ipc.server.listen.queue.size=256, ipc.server.log.slow.rpc=false, ipc.server.max.connections=0, ipc.server.purge.interval=15, ipc.server.reuseaddr=true, net.topology.impl=org.apache.hadoop.net.NetworkTopology, net.topology.node.switch.mapping.impl=org.apache.hadoop.net.ScriptBasedMapping, net.topology.script.number.args=100, nfs.allow.insecure.ports=true, nfs.dump.dir=/tmp/.hdfs-nfs, nfs.exports.allowed.hosts=* rw, nfs.mountd.port=4242, nfs.rtmax=1048576, nfs.server.port=2049, nfs.wtmax=1048576, ozone.UnsafeByteOperations.enabled=true, ozone.acl.authorizer.class=org.apache.hadoop.ozone.security.acl.OzoneNativeAuthorizer, ozone.acl.enabled=true, ozone.administrators=testuser/scm@EXAMPLE.COM,testuser/s3g@EXAMPLE.COM,recon/recon@EXAMPLE.COM, ozone.block.deleting.container.limit.per.interval=10, ozone.block.deleting.limit.per.task=1000, ozone.block.deleting.service.interval=1m, ozone.block.deleting.service.timeout=300000ms, ozone.block.deleting.service.workers=10, ozone.chunk.read.buffer.default.size=64KB, ozone.client.bucket.replication.config.refresh.time.ms=30000, ozone.client.bytes.per.checksum=1MB, ozone.client.checksum.combine.mode=COMPOSITE_CRC, ozone.client.checksum.type=CRC32, ozone.client.connection.timeout=5000ms, ozone.client.datastream.buffer.flush.size=16MB, ozone.client.datastream.min.packet.size=1MB, ozone.client.datastream.pipeline.mode=true, ozone.client.datastream.window.size=64MB, ozone.client.ec.grpc.retries.enabled=true, ozone.client.ec.grpc.retries.max=3, ozone.client.ec.reconstruct.stripe.read.pool.limit=30, ozone.client.ec.stripe.queue.size=2, ozone.client.exclude.nodes.expiry.time=600000, ozone.client.failover.max.attempts=500, ozone.client.fs.default.bucket.layout=FILE_SYSTEM_OPTIMIZED, ozone.client.key.latest.version.location=true, ozone.client.key.provider.cache.expiry=10d, ozone.client.list.cache=1000, ozone.client.list.trash.keys.max=1000, ozone.client.max.ec.stripe.write.retries=10, ozone.client.max.retries=5, ozone.client.read.timeout=30s, ozone.client.retry.interval=0, ozone.client.socket.timeout=5000ms, ozone.client.stream.buffer.flush.delay=true, ozone.client.stream.buffer.flush.size=16MB, ozone.client.stream.buffer.increment=0B, ozone.client.stream.buffer.max.size=32MB, ozone.client.stream.buffer.size=4MB, ozone.client.verify.checksum=true, ozone.client.wait.between.retries.millis=2000, ozone.container.cache.lock.stripes=1024, ozone.container.cache.size=1024, ozone.default.bucket.layout=LEGACY, ozone.directory.deleting.service.interval=1m, ozone.freon.http-address=0.0.0.0:9884, ozone.freon.http-bind-host=0.0.0.0, ozone.freon.http.auth.kerberos.keytab=/etc/security/keytabs/HTTP.keytab, ozone.freon.http.auth.kerberos.principal=HTTP/_HOST@REALM, ozone.freon.http.auth.type=simple, ozone.freon.http.enabled=true, ozone.freon.https-address=0.0.0.0:9885, ozone.freon.https-bind-host=0.0.0.0, ozone.fs.datastream.enabled=false, ozone.fs.iterate.batch-size=100, ozone.fs.listing.page.size=1024, ozone.fs.listing.page.size.max=5000, ozone.handler.type=distributed, ozone.http.filter.initializers=org.apache.hadoop.security.AuthenticationFilterInitializer, ozone.http.policy=HTTP_ONLY, ozone.https.client.keystore.resource=ssl-client.xml, ozone.https.client.need-auth=false, ozone.https.server.keystore.resource=ssl-server.xml, ozone.key.deleting.limit.per.task=20000, ozone.key.preallocation.max.blocks=64, ozone.manager.db.checkpoint.transfer.bandwidthPerSec=0, ozone.manager.delegation.remover.scan.interval=3600000, ozone.manager.delegation.token.max-lifetime=7d, ozone.manager.delegation.token.renew-interval=1d, ozone.metadata.dirs=/data/metadata, ozone.metastore.rocksdb.cf.write.buffer.size=128MB, ozone.metastore.rocksdb.statistics=OFF, ozone.network.flexible.fqdn.resolution.enabled=false, ozone.network.jvm.address.cache.enabled=true, ozone.network.topology.aware.read=false, ozone.om.address=om, ozone.om.admin.protocol.max.retries=20, ozone.om.admin.protocol.wait.between.retries=1000, ozone.om.client.rpc.timeout=15m, ozone.om.client.trash.core.pool.size=5, ozone.om.container.location.cache.size=100000, ozone.om.container.location.cache.ttl=360m, ozone.om.enable.filesystem.paths=false, ozone.om.enable.ofs.shared.tmp.dir=false, ozone.om.fs.snapshot.max.limit=1000, ozone.om.group.rights=ALL, ozone.om.grpc.maximum.response.length=134217728, ozone.om.grpc.port=8981, ozone.om.ha.raft.server.retrycache.expirytime=300s, ozone.om.handler.count.key=100, ozone.om.http-address=om:9874, ozone.om.http-bind-host=0.0.0.0, ozone.om.http.auth.kerberos.keytab=/etc/security/keytabs/om.keytab, ozone.om.http.auth.kerberos.principal=HTTP/om@EXAMPLE.COM, ozone.om.http.auth.type=kerberos, ozone.om.http.enabled=true, ozone.om.https-address=0.0.0.0:9875, ozone.om.https-bind-host=0.0.0.0, ozone.om.init.default.layout.version=-1, ozone.om.kerberos.keytab.file=/etc/security/keytabs/om.keytab, ozone.om.kerberos.principal=om/om@EXAMPLE.COM, ozone.om.key.path.lock.enabled=false, ozone.om.keyname.character.check.enabled=false, ozone.om.leader.election.minimum.timeout.duration=5s, ozone.om.lock.fair=false, ozone.om.multitenancy.enabled=true, ozone.om.multitenancy.ranger.sync.interval=30s, ozone.om.multitenancy.ranger.sync.timeout=10s, ozone.om.open.key.cleanup.limit.per.task=1000, ozone.om.open.key.cleanup.service.interval=24h, ozone.om.open.key.cleanup.service.timeout=300s, ozone.om.open.key.expire.threshold=7d, ozone.om.ranger.https-address=https://ranger:6182, ozone.om.ranger.https.admin.api.passwd=Passwd1, ozone.om.ranger.https.admin.api.user=admin, ozone.om.ranger.service=cm_ozone, ozone.om.ratis.enable=true, ozone.om.ratis.log.appender.queue.byte-limit=32MB, ozone.om.ratis.log.appender.queue.num-elements=1024, ozone.om.ratis.log.purge.gap=1000000, ozone.om.ratis.minimum.timeout=5s, ozone.om.ratis.port=9872, ozone.om.ratis.rpc.type=GRPC, ozone.om.ratis.segment.preallocated.size=4MB, ozone.om.ratis.segment.size=4MB, ozone.om.ratis.server.failure.timeout.duration=120s, ozone.om.ratis.server.leaderelection.pre-vote=false, ozone.om.ratis.server.request.timeout=3s, ozone.om.ratis.server.retry.cache.timeout=600000ms, ozone.om.s3.grpc.server_enabled=true, ozone.om.save.metrics.interval=5m, ozone.om.security.admin.protocol.acl=*, ozone.om.security.client.protocol.acl=*, ozone.om.snapshot.cache.max.size=10, ozone.om.snapshot.compaction.dag.max.time.allowed=30d, ozone.om.snapshot.compaction.dag.prune.daemon.run.interval=3600s, ozone.om.snapshot.provider.connection.timeout=5000s, ozone.om.snapshot.provider.request.timeout=5000ms, ozone.om.snapshot.provider.socket.timeout=5000s, ozone.om.tenant.dev.skip.ranger=true, ozone.om.unflushed.transaction.max.count=10000, ozone.om.upgrade.finalization.ratis.based.timeout=30s, ozone.om.user.max.volume=1024, ozone.om.user.rights=ALL, ozone.om.volume.listall.allowed=true, ozone.path.deleting.limit.per.task=10000, ozone.recon.address=recon:9891, ozone.recon.administrators=testuser2/scm@EXAMPLE.COM, ozone.recon.db.dir=/data/metadata/recon, ozone.recon.http-address=0.0.0.0:9888, ozone.recon.http-bind-host=0.0.0.0, ozone.recon.http.auth.kerberos.keytab=/etc/security/keytabs/recon.keytab, ozone.recon.http.auth.kerberos.principal=HTTP/recon@EXAMPLE.COM, ozone.recon.http.auth.type=kerberos, ozone.recon.http.enabled=true, ozone.recon.https-address=0.0.0.0:9889, ozone.recon.https-bind-host=0.0.0.0, ozone.recon.kerberos.keytab.file=/etc/security/keytabs/recon.keytab, ozone.recon.kerberos.principal=recon/recon@EXAMPLE.COM, ozone.recon.nssummary.flush.db.max.threshold=150000, ozone.recon.om.connection.request.timeout=5000, ozone.recon.om.connection.timeout=5s, ozone.recon.om.snapshot.task.flush.param=false, ozone.recon.om.snapshot.task.initial.delay=20s, ozone.recon.om.snapshot.task.interval.delay=1m, ozone.recon.om.socket.timeout=5s, ozone.recon.scm.connection.request.timeout=5s, ozone.recon.scm.connection.timeout=5s, ozone.recon.scm.container.threshold=100, ozone.recon.scm.snapshot.enabled=true, ozone.recon.scm.snapshot.task.initial.delay=1m, ozone.recon.scm.snapshot.task.interval.delay=24h, ozone.recon.security.client.datanode.container.protocol.acl=*, ozone.recon.task.thread.count=1, ozone.replication=3, ozone.replication.allowed-configs=^((STANDALONE|RATIS)/(ONE|THREE))|(EC/(3-2|6-3|10-4))$, ozone.rest.client.http.connection.max=100, ozone.rest.client.http.connection.per-route.max=20, ozone.s3g.client.buffer.size=4KB, ozone.s3g.http-address=0.0.0.0:9878, ozone.s3g.http-bind-host=0.0.0.0, ozone.s3g.http.auth.kerberos.keytab=/etc/security/keytabs/s3g.keytab, ozone.s3g.http.auth.kerberos.principal=HTTP/s3g@EXAMPLE.COM, ozone.s3g.http.auth.type=kerberos, ozone.s3g.http.enabled=true, ozone.s3g.kerberos.keytab.file=/etc/security/keytabs/s3g.keytab, ozone.s3g.kerberos.principal=s3g/s3g@EXAMPLE.COM, ozone.s3g.volume.name=s3v, ozone.scm.block.client.address=scm, ozone.scm.block.client.bind.host=0.0.0.0, ozone.scm.block.client.port=9863, ozone.scm.block.deletion.max.retry=4096, ozone.scm.block.size=256MB, ozone.scm.ca.list.retry.interval=10s, ozone.scm.chunk.size=4MB, ozone.scm.client.address=scm, ozone.scm.client.bind.host=0.0.0.0, ozone.scm.client.port=9860, ozone.scm.container.layout=FILE_PER_BLOCK, ozone.scm.container.placement.ec.impl=
      org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackScatter
    , ozone.scm.container.placement.impl=
      org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRandom
    , ozone.scm.container.size=1GB, ozone.scm.datanode.admin.monitor.interval=30s, ozone.scm.datanode.disallow.same.peers=false, ozone.scm.datanode.id.dir=/data, ozone.scm.datanode.pipeline.limit=1, ozone.scm.datanode.port=9861, ozone.scm.datanode.ratis.volume.free-space.min=1GB, ozone.scm.dead.node.interval=45s, ozone.scm.event.ContainerReport.thread.pool.size=10, ozone.scm.expired.container.replica.op.scrub.interval=5m, ozone.scm.grpc.port=9895, ozone.scm.ha.grpc.deadline.interval=30m, ozone.scm.ha.ratis.leader.election.timeout=5s, ozone.scm.ha.ratis.leader.ready.check.interval=2s, ozone.scm.ha.ratis.leader.ready.wait.timeout=60s, ozone.scm.ha.ratis.log.appender.queue.byte-limit=32MB, ozone.scm.ha.ratis.log.appender.queue.num-elements=1024, ozone.scm.ha.ratis.log.purge.enabled=false, ozone.scm.ha.ratis.log.purge.gap=1000000, ozone.scm.ha.ratis.request.timeout=30s, ozone.scm.ha.ratis.rpc.type=GRPC, ozone.scm.ha.ratis.segment.preallocated.size=4MB, ozone.scm.ha.ratis.segment.size=4MB, ozone.scm.ha.ratis.server.failure.timeout.duration=120s, ozone.scm.ha.ratis.server.leaderelection.pre-vote=false, ozone.scm.ha.ratis.server.retry.cache.timeout=60s, ozone.scm.ha.ratis.snapshot.threshold=1000, ozone.scm.handler.count.key=100, ozone.scm.heartbeat.log.warn.interval.count=10, ozone.scm.heartbeat.rpc-retry-count=15, ozone.scm.heartbeat.rpc-retry-interval=1s, ozone.scm.heartbeat.rpc-timeout=5s, ozone.scm.heartbeat.thread.interval=3s, ozone.scm.http-address=scm:9876, ozone.scm.http-bind-host=0.0.0.0, ozone.scm.http.enabled=true, ozone.scm.https-address=0.0.0.0:9877, ozone.scm.https-bind-host=0.0.0.0, ozone.scm.info.wait.duration=10m, ozone.scm.keyvalue.container.deletion-choosing.policy=
      org.apache.hadoop.ozone.container.common.impl.TopNOrderedContainerDeletionChoosingPolicy
    , ozone.scm.names=scm, ozone.scm.network.topology.schema.file=network-topology-default.xml, ozone.scm.pipeline.allocated.timeout=5m, ozone.scm.pipeline.creation.auto.factor.one=true, ozone.scm.pipeline.creation.interval=30s, ozone.scm.pipeline.destroy.timeout=66s, ozone.scm.pipeline.leader-choose.policy=
      org.apache.hadoop.hdds.scm.pipeline.leader.choose.algorithms.MinLeaderCountChoosePolicy
    , ozone.scm.pipeline.owner.container.count=1, ozone.scm.pipeline.per.metadata.disk=2, ozone.scm.pipeline.scrub.interval=5m, ozone.scm.ratis.pipeline.limit=0, ozone.scm.ratis.port=9894, ozone.scm.security.handler.count.key=2, ozone.scm.security.service.bind.host=0.0.0.0, ozone.scm.security.service.port=9961, ozone.scm.sequence.id.batch.size=1000, ozone.scm.skip.bootstrap.validation=false, ozone.scm.stale.node.interval=30s, ozone.scm.update.client.crl.check.interval=600s, ozone.scm.update.service.port=9893, ozone.security.enabled=true, ozone.security.http.kerberos.enabled=true, ozone.server.default.replication=3, ozone.server.default.replication.type=RATIS, ozone.service.shutdown.timeout=60s, ozone.snapshot.filtering.limit.per.task=2, ozone.snapshot.filtering.service.interval=1m, ozone.sst.filtering.service.timeout=300000ms, ozone.tags.system=OZONE,MANAGEMENT,SECURITY,PERFORMANCE,DEBUG,CLIENT,SERVER,OM,SCM,
      CRITICAL,RATIS,CONTAINER,REQUIRED,REST,STORAGE,PIPELINE,STANDALONE,S3GATEWAY,TOKEN,TLS,RECON, ozone.trace.enabled=false, recon.om.delta.update.limit=2000, recon.om.delta.update.loop.limit=10, rpc.metrics.percentiles.intervals=60,300, rpc.metrics.quantile.enable=true, rpc.metrics.timeunit=MILLISECONDS, scm.container.client.idle.threshold=10s, scm.container.client.max.size=256, seq.io.sort.factor=100, seq.io.sort.mb=100, tfile.fs.input.buffer.size=262144, tfile.fs.output.buffer.size=262144, tfile.io.chunk.size=1048576}

SaketaChalamchala · 2023-02-16T23:26:05Z

Similar list from other services

SCM : 
/************************************************************
STARTUP_MSG: Starting StorageContainerManager
...
STARTUP_MSG:   java = 11.0.17
STARTUP_MSG:   conf = {adl.feature.ownerandgroup.enableupn=false, adl.http.timeout=-1, datanode.https.port=50475, dfs.balancer.address=0.0.0.0:0, dfs.balancer.block-move.timeout=0, dfs.balancer.dispatcherThreads=200, dfs.balancer.getBlocks.min-block-size=10485760, dfs.balancer.getBlocks.size=2147483648, dfs.balancer.keytab.enabled=false, dfs.balancer.max-iteration-time=1200000, dfs.balancer.max-no-move-interval=60000, dfs.balancer.max-size-to-move=10737418240, dfs.balancer.movedWinWidth=5400000, dfs.balancer.moverThreads=1000, dfs.balancer.service.interval=5m, dfs.balancer.service.retries.on.exception=5, dfs.batched.ls.limit=100, dfs.block.access.key.update.interval=600, dfs.block.access.token.enable=false, dfs.block.access.token.lifetime=600, ..., dfs.datanode.balance.bandwidthPerSec=100m, dfs.datanode.balance.max.concurrent.moves=100, dfs.datanode.block-pinning.enabled=false, dfs.datanode.block.id.layout.upgrade.threads=6, dfs.datanode.bp-ready.timeout=20, dfs.datanode.cache.revocation.polling.ms=500, dfs.datanode.cache.revocation.timeout.ms=900000, dfs.datanode.cached-dfsused.check.interval.ms=600000, dfs.datanode.data.dir=file://${hadoop.tmp.dir}/dfs/data, dfs.datanode.data.dir.perm=700, dfs.datanode.data.transfer.bandwidthPerSec=0, dfs.datanode.data.write.bandwidthPerSec=0, dfs.datanode.directoryscan.interval=21600, dfs.datanode.directoryscan.threads=1, dfs.datanode.directoryscan.throttle.limit.ms.per.sec=1000, dfs.datanode.disk.check.min.gap=15m, ... dfs.use.dfs.network.topology=true, dfs.user.home.dir.prefix=/user, dfs.web.authentication.kerberos.keytab=/etc/security/keytabs/HTTP.keytab, dfs.web.authentication.kerberos.principal=HTTP/ozone@EXAMPLE.COM, dfs.webhdfs.acl.provider.permission.pattern=^(default:)?(user|group|mask|other):[[A-Za-z_][A-Za-z0-9._-]]*:([rwx-]{3})?(,(default:)?(user|group|mask|other):[[A-Za-z_][A-Za-z0-9._-]]*:([rwx-]{3})?)*$, dfs.webhdfs.netty.high.watermark=65535, dfs.webhdfs.netty.low.watermark=32768, dfs.webhdfs.oauth2.enabled=false, dfs.webhdfs.rest-csrf.browser-useragents-regex=^Mozilla.*,^Opera.*, dfs.webhdfs.rest-csrf.custom-header=X-XSRF-HEADER, dfs.webhdfs.rest-csrf.enabled=false, dfs.webhdfs.rest-csrf.methods-to-ignore=GET,OPTIONS,HEAD,TRACE, dfs.webhdfs.socket.connect-timeout=60s, dfs.webhdfs.socket.read-timeout=60s, dfs.webhdfs.ugi.expire.after.access=600000, dfs.webhdfs.use.ipc.callq=true, dfs.webhdfs.user.provider.user.pattern=^[A-Za-z_][A-Za-z0-9._-]*[$]?$, dfs.xframe.enabled=true, dfs.xframe.value=SAMEORIGIN, file.blocksize=67108864, file.bytes-per-checksum=512, file.client-write-packet-size=65536, file.replication=1, file.stream-buffer-size=4096, fs.AbstractFileSystem.abfs.impl=org.apache.hadoop.fs.azurebfs.Abfs, fs.AbstractFileSystem.abfss.impl=org.apache.hadoop.fs.azurebfs.Abfss, fs.AbstractFileSystem.adl.impl=org.apache.hadoop.fs.adl.Adl, fs.AbstractFileSystem.file.impl=org.apache.hadoop.fs.local.LocalFs, fs.AbstractFileSystem.ftp.impl=org.apache.hadoop.fs.ftp.FtpFs, fs.AbstractFileSystem.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS, fs.AbstractFileSystem.har.impl=org.apache.hadoop.fs.HarFs, fs.AbstractFileSystem.hdfs.impl=org.apache.hadoop.fs.Hdfs, fs.AbstractFileSystem.s3a.impl=org.apache.hadoop.fs.s3a.S3A, fs.AbstractFileSystem.swebhdfs.impl=org.apache.hadoop.fs.SWebHdfs, fs.AbstractFileSystem.viewfs.impl=org.apache.hadoop.fs.viewfs.ViewFs, fs.AbstractFileSystem.wasb.impl=org.apache.hadoop.fs.azure.Wasb, fs.AbstractFileSystem.wasbs.impl=org.apache.hadoop.fs.azure.Wasbs, fs.AbstractFileSystem.webhdfs.impl=org.apache.hadoop.fs.WebHdfs, fs.abfs.impl=org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem, fs.abfss.impl=org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem, fs.adl.impl=org.apache.hadoop.fs.adl.AdlFileSystem, fs.adl.oauth2.access.token.provider.type=<redacted>, ...
    , ozone.scm.names=scm, ozone.scm.network.topology.schema.file=network-topology-default.xml, ozone.scm.pipeline.allocated.timeout=5m, ozone.scm.pipeline.creation.auto.factor.one=true, ozone.scm.pipeline.creation.interval=30s, ozone.scm.pipeline.destroy.timeout=66s, ozone.scm.pipeline.leader-choose.policy=
      org.apache.hadoop.hdds.scm.pipeline.leader.choose.algorithms.MinLeaderCountChoosePolicy
    , ozone.scm.pipeline.owner.container.count=1, ozone.scm.pipeline.per.metadata.disk=2, ozone.scm.pipeline.scrub.interval=5m, ozone.scm.ratis.pipeline.limit=0, ozone.scm.ratis.port=9894, ozone.scm.security.handler.count.key=2, ozone.scm.security.service.bind.host=0.0.0.0, ozone.scm.security.service.port=9961, ozone.scm.sequence.id.batch.size=1000, ozone.scm.skip.bootstrap.validation=false, ozone.scm.stale.node.interval=30s, ozone.scm.update.client.crl.check.interval=600s, ozone.scm.update.service.port=9893, ozone.security.enabled=true, ozone.security.http.kerberos.enabled=true, ozone.server.default.replication=3, ozone.server.default.replication.type=RATIS, ozone.service.shutdown.timeout=60s, ozone.snapshot.filtering.limit.per.task=2, ozone.snapshot.filtering.service.interval=1m, ozone.sst.filtering.service.timeout=300000ms, ozone.tags.system=OZONE,MANAGEMENT,SECURITY,PERFORMANCE,DEBUG,CLIENT,SERVER,OM,SCM,
      CRITICAL,RATIS,CONTAINER,REQUIRED,REST,STORAGE,PIPELINE,STANDALONE,S3GATEWAY,TOKEN,TLS,RECON, ozone.trace.enabled=false, recon.om.delta.update.limit=2000, recon.om.delta.update.loop.limit=10, rpc.metrics.percentiles.intervals=60,300, rpc.metrics.quantile.enable=true, rpc.metrics.timeunit=MILLISECONDS, scm.container.client.idle.threshold=10s, scm.container.client.max.size=256, seq.io.sort.factor=100, seq.io.sort.mb=100, tfile.fs.input.buffer.size=262144, tfile.fs.output.buffer.size=262144, tfile.io.chunk.size=1048576}l=300s, ozone.recon.task.safemode.wait.threshold=300s, ozone.recon.task.thread.count=1, ozone.service.shutdown.timeout=60s, recon.om.delta.update.limit=2000, recon.om.delta.update.loop.limit=10}

DN: 
/************************************************************
STARTUP_MSG: Starting HddsDatanodeService
...
STARTUP_MSG:   java = 11.0.17
STARTUP_MSG:   conf = {adl.feature.ownerandgroup.enableupn=false, adl.http.timeout=-1, datanode.https.port=50475, dfs.balancer.address=0.0.0.0:0, dfs.balancer.block-move.timeout=0, dfs.balancer.dispatcherThreads=200, dfs.balancer.getBlocks.min-block-size=10485760, dfs.balancer.getBlocks.size=2147483648, dfs.balancer.keytab.enabled=false, dfs.balancer.max-iteration-time=1200000, dfs.balancer.max-no-move-interval=60000, dfs.balancer.max-size-to-move=10737418240, dfs.balancer.movedWinWidth=5400000, dfs.balancer.moverThreads=1000, dfs.balancer.service.interval=5m, ..., fs.adl.oauth2.access.token.provider.type=<redacted>, fs.automatic.close=true, fs.azure.authorization=false, fs.azure.authorization.caching.enable=true, fs.azure.buffer.dir=${hadoop.tmp.dir}/abfs, fs.azure.local.sas.key.mode=false, fs.azure.sas.expiry.period=90d, fs.azure.saskey.usecontainersaskeyforallaccess=true, fs.azure.secure.mode=false, fs.azure.user.agent.prefix=unknown, fs.client.resolve.remote.symlinks=true, fs.client.resolve.topology.enabled=false, fs.defaultFS=ofs://om, fs.df.interval=60000, fs.du.interval=600000, fs.ftp.data.connection.mode=ACTIVE_LOCAL_DATA_CONNECTION_MODE, fs.ftp.host=0.0.0.0, fs.ftp.host.port=21, fs.ftp.impl=org.apache.hadoop.fs.ftp.FTPFileSystem, fs.ftp.timeout=0, fs.ftp.transfer.mode=BLOCK_TRANSFER_MODE, fs.getspaceused.jitterMillis=60000, fs.har.impl.disable.cache=true, fs.permissions.umask-mode=022, fs.s3a.accesspoint.required=false, ..., ozone.scm.heartbeat.log.warn.interval.count=10, ozone.scm.heartbeat.rpc-retry-count=15, ozone.scm.heartbeat.rpc-retry-interval=1s, ozone.scm.heartbeat.rpc-timeout=5s, ozone.scm.heartbeat.thread.interval=3s, ozone.scm.http-address=scm:9876, ozone.scm.http-bind-host=0.0.0.0, ozone.scm.http.enabled=true, ozone.scm.https-address=0.0.0.0:9877, ozone.scm.https-bind-host=0.0.0.0, ozone.scm.info.wait.duration=10m, ozone.scm.keyvalue.container.deletion-choosing.policy=
      org.apache.hadoop.ozone.container.common.impl.TopNOrderedContainerDeletionChoosingPolicy
    , ozone.scm.names=scm, ozone.scm.network.topology.schema.file=network-topology-default.xml, ozone.scm.pipeline.allocated.timeout=5m, ozone.scm.pipeline.creation.auto.factor.one=true, ozone.scm.pipeline.creation.interval=30s, ozone.scm.pipeline.destroy.timeout=66s, ozone.scm.pipeline.leader-choose.policy=
      org.apache.hadoop.hdds.scm.pipeline.leader.choose.algorithms.MinLeaderCountChoosePolicy
    , ozone.scm.pipeline.owner.container.count=1, ozone.scm.pipeline.per.metadata.disk=2, ozone.scm.pipeline.scrub.interval=5m, ozone.scm.ratis.pipeline.limit=0, ozone.scm.ratis.port=9894, ozone.scm.security.handler.count.key=2, ozone.scm.security.service.bind.host=0.0.0.0, ozone.scm.security.service.port=9961, ozone.scm.sequence.id.batch.size=1000, ozone.scm.skip.bootstrap.validation=false, ozone.scm.stale.node.interval=30s, ozone.scm.update.client.crl.check.interval=600s, ozone.scm.update.service.port=9893, ozone.security.enabled=true, ozone.security.http.kerberos.enabled=true, ozone.server.default.replication=3, ozone.server.default.replication.type=RATIS, ozone.service.shutdown.timeout=60s, ozone.snapshot.filtering.limit.per.task=2, ozone.snapshot.filtering.service.interval=1m, ozone.sst.filtering.service.timeout=300000ms, ozone.tags.system=OZONE,MANAGEMENT,SECURITY,PERFORMANCE,DEBUG,CLIENT,SERVER,OM,SCM,
      CRITICAL,RATIS,CONTAINER,REQUIRED,REST,STORAGE,PIPELINE,STANDALONE,S3GATEWAY,TOKEN,TLS,RECON, ozone.trace.enabled=false, recon.om.delta.update.limit=2000, recon.om.delta.update.loop.limit=10, rpc.metrics.percentiles.intervals=60,300, rpc.metrics.quantile.enable=true, rpc.metrics.timeunit=MILLISECONDS, scm.container.client.idle.threshold=10s, scm.container.client.max.size=256, seq.io.sort.factor=100, seq.io.sort.mb=100, tfile.fs.input.buffer.size=262144, tfile.fs.output.buffer.size=262144, tfile.io.chunk.size=1048576}

S3G: 
/************************************************************
...
STARTUP_MSG:   java = 11.0.17
STARTUP_MSG:   conf = {adl.feature.ownerandgroup.enableupn=false, adl.http.timeout=-1, bind.address=0.0.0.0, datanode.https.port=50475, dfs.balancer.address=0.0.0.0:0, dfs.balancer.block-move.timeout=0, dfs.balancer.dispatcherThreads=200, dfs.balancer.getBlocks.min-block-size=10485760, dfs.balancer.getBlocks.size=2147483648, dfs.balancer.keytab.enabled=false, dfs.balancer.max-iteration-time=1200000, dfs.balancer.max-no-move-interval=60000, dfs.balancer.max-size-to-move=10737418240, dfs.balancer.movedWinWidth=5400000, dfs.balancer.moverThreads=1000, dfs.balancer.service.interval=5m, dfs.balancer.service.retries.on.exception=5, dfs.batched.ls.limit=100, dfs.block.access.key.update.interval=600, dfs.block.access.token.enable=false, dfs.block.access.token.lifetime=600, ..., fs.adl.oauth2.access.token.provider.type=<redacted>, fs.automatic.close=true, fs.azure.authorization=false, fs.azure.authorization.caching.enable=true, fs.azure.buffer.dir=${hadoop.tmp.dir}/abfs, fs.azure.local.sas.key.mode=false, fs.azure.sas.expiry.period=90d, fs.azure.saskey.usecontainersaskeyforallaccess=true, fs.azure.secure.mode=false, fs.azure.user.agent.prefix=unknown, fs.client.resolve.remote.symlinks=true, fs.client.resolve.topology.enabled=false, fs.defaultFS=ofs://om, fs.df.interval=60000, fs.du.interval=600000, fs.ftp.data.connection.mode=ACTIVE_LOCAL_DATA_CONNECTION_MODE, fs.ftp.host=0.0.0.0, fs.ftp.host.port=21, fs.ftp.impl=org.apache.hadoop.fs.ftp.FTPFileSystem, fs.ftp.timeout=0, fs.ftp.transfer.mode=BLOCK_TRANSFER_MODE, fs.getspaceused.jitterMillis=60000, fs.har.impl.disable.cache=true, fs.permissions.umask-mode=022, fs.s3a.accesspoint.required=false, fs.s3a.assumed.role.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider, fs.s3a.assumed.role.session.duration=30m, fs.s3a.attempts.maximum=20, fs.s3a.aws.credentials.provider=
    org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider,
    org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider,
    com.amazonaws.auth.EnvironmentVariableCredentialsProvider,
    org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
  , fs.s3a.block.size=32M, fs.s3a.buffer.dir=${hadoop.tmp.dir}/s3a, fs.s3a.change.detection.mode=server, fs.s3a.change.detection.source=etag, fs.s3a.change.detection.version.required=true, fs.s3a.committer.abort.pending.uploads=true, fs.s3a.committer.magic.enabled=true, fs.s3a.committer.name=file, fs.s3a.committer.staging.conflict-mode=append, fs.s3a.committer.staging.tmp.path=tmp/staging, fs.s3a.committer.staging.unique-filenames=true, fs.s3a.committer.threads=8, fs.s3a.connection.establish.timeout=5000, fs.s3a.connection.maximum=96, fs.s3a.connection.request.timeout=0, ..., ozone.scm.ratis.pipeline.limit=0, ozone.scm.ratis.port=9894, ozone.scm.security.handler.count.key=2, ozone.scm.security.service.bind.host=0.0.0.0, ozone.scm.security.service.port=9961, ozone.scm.sequence.id.batch.size=1000, ozone.scm.skip.bootstrap.validation=false, ozone.scm.stale.node.interval=30s, ozone.scm.update.client.crl.check.interval=600s, ozone.scm.update.service.port=9893, ozone.security.enabled=true, ozone.security.http.kerberos.enabled=true, ozone.server.default.replication=3, ozone.server.default.replication.type=RATIS, ozone.service.shutdown.timeout=60s, ozone.snapshot.filtering.limit.per.task=2, ozone.snapshot.filtering.service.interval=1m, ozone.sst.filtering.service.timeout=300000ms, ozone.tags.system=OZONE,MANAGEMENT,SECURITY,PERFORMANCE,DEBUG,CLIENT,SERVER,OM,SCM,
      CRITICAL,RATIS,CONTAINER,REQUIRED,REST,STORAGE,PIPELINE,STANDALONE,S3GATEWAY,TOKEN,TLS,RECON, ozone.trace.enabled=false, recon.om.delta.update.limit=2000, recon.om.delta.update.loop.limit=10, rpc.metrics.percentiles.intervals=60,300, rpc.metrics.quantile.enable=true, rpc.metrics.timeunit=MILLISECONDS, scm.container.client.idle.threshold=10s, scm.container.client.max.size=256, seq.io.sort.factor=100, seq.io.sort.mb=100, tfile.fs.input.buffer.size=262144, tfile.fs.output.buffer.size=262144, tfile.io.chunk.size=1048576}

Ozone CSI:
sh-4.2$ ozone csi -conf=/etc/hadoop/x --verbose
...
/************************************************************
STARTUP_MSG: Starting CsiServer
...
STARTUP_MSG:   java = 11.0.17
STARTUP_MSG:   conf = {adl.feature.ownerandgroup.enableupn=false, adl.http.timeout=-1, datanode.https.port=50475, dfs.balancer.address=0.0.0.0:0, dfs.balancer.block-move.timeout=0, dfs.balancer.dispatcherThreads=200, dfs.balancer.getBlocks.min-block-size=10485760, dfs.balancer.getBlocks.size=2147483648, dfs.balancer.keytab.enabled=false, dfs.balancer.max-iteration-time=1200000, dfs.balancer.max-no-move-interval=60000, dfs.balancer.max-size-to-move=10737418240, dfs.balancer.movedWinWidth=5400000, dfs.balancer.moverThreads=1000, dfs.balancer.service.interval=5m, ..., fs.adl.oauth2.access.token.provider.type=<redacted>, fs.automatic.close=true, fs.azure.authorization=false, fs.azure.authorization.caching.enable=true, fs.azure.buffer.dir=${hadoop.tmp.dir}/abfs, fs.azure.local.sas.key.mode=false, fs.azure.sas.expiry.period=90d, fs.azure.saskey.usecontainersaskeyforallaccess=true, fs.azure.secure.mode=false, fs.azure.user.agent.prefix=unknown, fs.client.resolve.remote.symlinks=true, fs.client.resolve.topology.enabled=false, fs.defaultFS=ofs://om, fs.df.interval=60000, fs.du.interval=600000, fs.ftp.data.connection.mode=ACTIVE_LOCAL_DATA_CONNECTION_MODE, fs.ftp.host=0.0.0.0, fs.ftp.host.port=21, fs.ftp.impl=org.apache.hadoop.fs.ftp.FTPFileSystem, fs.ftp.timeout=0, fs.ftp.transfer.mode=BLOCK_TRANSFER_MODE, fs.getspaceused.jitterMillis=60000, fs.har.impl.disable.cache=true, fs.permissions.umask-mode=022, fs.s3a.accesspoint.required=false, fs.s3a.assumed.role.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider, fs.s3a.assumed.role.session.duration=30m, fs.s3a.attempts.maximum=20, fs.s3a.aws.credentials.provider=
    org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider,
    org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider,
    com.amazonaws.auth.EnvironmentVariableCredentialsProvider,
    org.apache.hadoop.fs.s3a.auth.IAMInstanceCredentialsProvider
  , fs.s3a.block.size=32M, fs.s3a.buffer.dir=${hadoop.tmp.dir}/s3a, fs.s3a.change.detection.mode=server, fs.s3a.change.detection.source=etag, fs.s3a.change.detection.version.required=true, ..., ozone.scm.security.service.port=9961, ozone.scm.sequence.id.batch.size=1000, ozone.scm.skip.bootstrap.validation=false, ozone.scm.stale.node.interval=30s, ozone.security.enabled=true, ozone.security.http.kerberos.enabled=true, ozone.server.default.replication=3, ozone.server.default.replication.type=RATIS, ozone.service.shutdown.timeout=60s, ozone.snapshot.filtering.limit.per.task=2, ozone.snapshot.filtering.service.interval=1m, ozone.sst.filtering.service.timeout=300000ms, ozone.tags.system=OZONE,MANAGEMENT,SECURITY,PERFORMANCE,DEBUG,CLIENT,SERVER,OM,SCM,
      CRITICAL,RATIS,CONTAINER,REQUIRED,REST,STORAGE,PIPELINE,STANDALONE,S3GATEWAY,TOKEN,TLS,RECON, ozone.trace.enabled=false, recon.om.delta.update.limit=2000, recon.om.delta.update.loop.limit=10, rpc.metrics.percentiles.intervals=60,300, rpc.metrics.quantile.enable=true, rpc.metrics.timeunit=MILLISECONDS, scm.container.client.idle.threshold=10s, scm.container.client.max.size=256, seq.io.sort.factor=100, seq.io.sort.mb=100, tfile.fs.input.buffer.size=262144, tfile.fs.output.buffer.size=262144, tfile.io.chunk.size=1048576}

…d whitespace errors.

SaketaChalamchala · 2023-02-28T02:02:09Z

Updated the patch to include Ozone configs only using regex.

## Full list of configs printed in all role logs : 
STARTUP_MSG:   conf = {dfs.container.chunk.write.sync=false, dfs.container.ipc=9859, dfs.container.ipc.random.port=false, dfs.container.ratis.admin.port=9857, dfs.container.ratis.datastream.enabled=true, dfs.container.ratis.datastream.port=9855, dfs.container.ratis.datastream.random.port=false, dfs.container.ratis.enabled=false, dfs.container.ratis.ipc=9858, dfs.container.ratis.ipc.random.port=false, dfs.container.ratis.leader.pending.bytes.limit=1GB, dfs.container.ratis.log.appender.queue.byte-limit=32MB, dfs.container.ratis.log.appender.queue.num-elements=1, dfs.container.ratis.log.purge.gap=1000000, dfs.container.ratis.log.queue.byte-limit=4GB, dfs.container.ratis.log.queue.num-elements=1024, dfs.container.ratis.num.container.op.executors=10, dfs.container.ratis.num.write.chunk.threads.per.volume=10, dfs.container.ratis.replication.level=MAJORITY, dfs.container.ratis.rpc.type=GRPC, dfs.container.ratis.segment.preallocated.size=16KB, dfs.container.ratis.segment.size=1MB, dfs.container.ratis.server.port=9856, dfs.container.ratis.statemachine.max.pending.apply-transactions=10000, dfs.container.ratis.statemachinedata.sync.retries=-1, dfs.container.ratis.statemachinedata.sync.timeout=10s, dfs.ratis.leader.election.minimum.timeout.duration=5s, dfs.ratis.server.retry-cache.timeout.duration=600000ms, dfs.ratis.snapshot.threshold=10000, fs.trash.classname=org.apache.hadoop.ozone.om.TrashPolicyOzone, hadoop.hdds.db.rocksdb.logging.enabled=false, hadoop.hdds.db.rocksdb.logging.level=INFO, hadoop.hdds.db.rocksdb.writeoption.sync=false, hadoop.http.idle_timeout.ms=60000, hdds.block.token.enabled=true, hdds.block.token.expiry.time=1d, hdds.command.status.report.interval=30s, hdds.container.action.max.limit=20, hdds.container.balancer.balancing.iteration.interval=70m, hdds.container.balancer.datanodes.involved.max.percentage.per.iteration=20, hdds.container.balancer.iterations=10, hdds.container.balancer.move.networkTopology.enable=false, hdds.container.balancer.move.timeout=30m, hdds.container.balancer.size.entering.target.max=26GB, hdds.container.balancer.size.leaving.source.max=26GB, hdds.container.balancer.size.moved.max.per.iteration=500GB, hdds.container.balancer.trigger.du.before.move.enable=false, hdds.container.balancer.utilization.threshold=10, hdds.container.checksum.verification.enabled=true, hdds.container.close.threshold=0.9f, hdds.container.replication.compression=NO_COMPRESSION, hdds.container.report.interval=60s, hdds.container.scrub.data.scan.interval=7d, hdds.container.scrub.enabled=false, hdds.container.scrub.metadata.scan.interval=3h, hdds.container.scrub.on.demand.volume.bytes.per.second=5242880, hdds.container.scrub.volume.bytes.per.second=5242880, hdds.container.token.enabled=true, hdds.crl.status.report.interval=60000ms, hdds.datanode.block.delete.queue.limit=1440, hdds.datanode.block.delete.threads.max=5, hdds.datanode.block.deleting.limit.per.interval=5000, hdds.datanode.block.deleting.service.interval=60s, hdds.datanode.chunk.data.validation.check=false, hdds.datanode.container.delete.threads.max=2, hdds.datanode.container.schema.v3.enabled=true, hdds.datanode.container.schema.v3.key.separator=|, hdds.datanode.df.refresh.period=5m, hdds.datanode.dir=/data/hdds, hdds.datanode.disk.check.min.gap=15m, hdds.datanode.disk.check.timeout=10m, hdds.datanode.du.refresh.period=1h, hdds.datanode.failed.data.volumes.tolerated=-1, hdds.datanode.failed.db.volumes.tolerated=-1, hdds.datanode.failed.metadata.volumes.tolerated=-1, hdds.datanode.http-address=0.0.0.0:9882, hdds.datanode.http-bind-host=0.0.0.0, hdds.datanode.http.auth.kerberos.keytab=/etc/security/keytabs/dn.keytab, hdds.datanode.http.auth.kerberos.principal=HTTP/dn@EXAMPLE.COM, hdds.datanode.http.auth.type=kerberos, hdds.datanode.http.enabled=true, hdds.datanode.https-address=0.0.0.0:9883, hdds.datanode.https-bind-host=0.0.0.0, hdds.datanode.metadata.rocksdb.cache.size=64MB, hdds.datanode.periodic.disk.check.interval.minutes=60, hdds.datanode.ratis.server.request.timeout=2m, hdds.datanode.read.chunk.threads.per.volume=10, hdds.datanode.recovering.container.scrubbing.service.interval=1m, hdds.datanode.replication.port=9886, hdds.datanode.replication.streams.limit=10, hdds.datanode.rocksdb.auto-compaction-small-sst-file=true, hdds.datanode.rocksdb.auto-compaction-small-sst-file-num-threshold=512, hdds.datanode.rocksdb.auto-compaction-small-sst-file-size-threshold=1MB, hdds.datanode.rocksdb.delete-obsolete-files-period=1h, hdds.datanode.rocksdb.log.level=INFO, hdds.datanode.rocksdb.log.max-file-num=64, hdds.datanode.rocksdb.log.max-file-size=32MB, hdds.datanode.rocksdb.max-open-files=1024, hdds.datanode.storage.utilization.critical.threshold=0.95, hdds.datanode.storage.utilization.warning.threshold=0.75, hdds.datanode.wait.on.all.followers=false, hdds.db.profile=DISK, hdds.grpc.tls.enabled=true, hdds.grpc.tls.provider=OPENSSL, hdds.heartbeat.interval=30s, hdds.key.dir.name=keys, hdds.key.len=2048, hdds.node.report.interval=60000ms, hdds.pipeline.action.max.limit=20, hdds.pipeline.report.interval=60000ms, hdds.priv.key.file.name=private.pem, hdds.profiler.endpoint.enabled=false, hdds.prometheus.endpoint.enabled=true, hdds.public.key.file.name=public.pem, hdds.ratis.client.exponential.backoff.base.sleep=4s, hdds.ratis.client.exponential.backoff.max.sleep=40s, hdds.ratis.client.multilinear.random.retry.policy=5s, 5, 10s, 5, 15s, 5, 20s, 5, 25s, 5, 60s, 10, hdds.ratis.client.request.watch.timeout=3m, hdds.ratis.client.request.write.timeout=5m, hdds.ratis.client.retry.policy=org.apache.hadoop.hdds.ratis.retrypolicy.RequestTypeDependentRetryPolicyCreator, hdds.ratis.client.retrylimited.max.retries=180, hdds.ratis.client.retrylimited.retry.interval=1s, hdds.ratis.raft.client.async.outstanding-requests.max=32, hdds.ratis.raft.client.rpc.request.timeout=60s, hdds.ratis.raft.client.rpc.watch.request.timeout=180s, hdds.ratis.raft.grpc.flow.control.window=5MB, hdds.ratis.raft.grpc.message.size.max=32MB, hdds.ratis.raft.server.datastream.client.pool.size=10, hdds.ratis.raft.server.datastream.request.threads=20, hdds.ratis.raft.server.delete.ratis.log.directory=true, hdds.ratis.raft.server.leaderelection.pre-vote=false, hdds.ratis.raft.server.notification.no-leader.timeout=300s, hdds.ratis.raft.server.rpc.request.timeout=60s, hdds.ratis.raft.server.rpc.slowness.timeout=300s, hdds.ratis.raft.server.watch.timeout=180s, hdds.ratis.raft.server.write.element-limit=1024, hdds.ratis.server.num.snapshots.retained=5, hdds.recon.heartbeat.interval=60s, hdds.scm.block.deleting.service.interval=60s, hdds.scm.block.deletion.per-interval.max=100000, hdds.scm.http.auth.kerberos.keytab=/etc/security/keytabs/scm.keytab, hdds.scm.http.auth.kerberos.principal=HTTP/scm@EXAMPLE.COM, hdds.scm.http.auth.type=kerberos, hdds.scm.init.default.layout.version=-1, hdds.scm.kerberos.keytab.file=/etc/security/keytabs/scm.keytab, hdds.scm.kerberos.principal=scm/scm@EXAMPLE.COM, hdds.scm.pipeline.choose.policy.impl=org.apache.hadoop.hdds.scm.pipeline.choose.algorithms.RandomPipelineChoosePolicy, hdds.scm.replication.command.deadline.factor=0.9, hdds.scm.replication.container.inflight.deletion.limit=0, hdds.scm.replication.container.inflight.replication.limit=0, hdds.scm.replication.event.timeout=10s, hdds.scm.replication.maintenance.remaining.redundancy=1, hdds.scm.replication.maintenance.replica.minimum=2, hdds.scm.replication.over.replicated.interval=30s, hdds.scm.replication.push=true, hdds.scm.replication.thread.interval=5s, hdds.scm.replication.under.replicated.interval=30s, hdds.scm.safemode.atleast.one.node.reported.pipeline.pct=0.90, hdds.scm.safemode.enabled=true, hdds.scm.safemode.healthy.pipeline.pct=0.10, hdds.scm.safemode.min.datanode=3, hdds.scm.safemode.pipeline-availability.check=true, hdds.scm.safemode.pipeline.creation=true, hdds.scm.safemode.threshold.pct=0.99, hdds.scm.unknown-container.action=WARN, hdds.scm.wait.time.after.safemode.exit=5m, hdds.scmclient.failover.max.retry=15, hdds.scmclient.failover.retry.interval=2s, hdds.scmclient.max.retry.timeout=30s, hdds.scmclient.rpc.timeout=15m, hdds.security.client.datanode.container.protocol.acl=*, hdds.security.client.scm.block.protocol.acl=*, hdds.security.client.scm.certificate.protocol.acl=*, hdds.security.client.scm.container.protocol.acl=*, hdds.security.ssl.keystore.reload.interval=60s, hdds.security.ssl.truststore.reload.interval=60s, hdds.tracing.enabled=false, hdds.x509.default.duration=P365D, hdds.x509.dir.name=certs, hdds.x509.file.name=certificate.crt, hdds.x509.max.duration=P1865D, hdds.x509.renew.grace.duration=P28D, hdds.x509.signature.algorithm=SHA256withRSA, ozone.UnsafeByteOperations.enabled=true, ozone.acl.authorizer.class=org.apache.hadoop.ozone.security.acl.OzoneNativeAuthorizer, ozone.acl.enabled=true, ozone.administrators=testuser/scm@EXAMPLE.COM,testuser/s3g@EXAMPLE.COM,recon/recon@EXAMPLE.COM, ozone.block.deleting.container.limit.per.interval=10, ozone.block.deleting.limit.per.task=1000, ozone.block.deleting.service.interval=1m, ozone.block.deleting.service.timeout=300000ms, ozone.block.deleting.service.workers=10, ozone.chunk.read.buffer.default.size=64KB, ozone.client.bucket.replication.config.refresh.time.ms=30000, ozone.client.bytes.per.checksum=1MB, ozone.client.checksum.combine.mode=COMPOSITE_CRC, ozone.client.checksum.type=CRC32, ozone.client.connection.timeout=5000ms, ozone.client.datastream.buffer.flush.size=16MB, ozone.client.datastream.min.packet.size=1MB, ozone.client.datastream.pipeline.mode=true, ozone.client.datastream.window.size=64MB, ozone.client.ec.grpc.retries.enabled=true, ozone.client.ec.grpc.retries.max=3, ozone.client.ec.reconstruct.stripe.read.pool.limit=30, ozone.client.ec.stripe.queue.size=2, ozone.client.exclude.nodes.expiry.time=600000, ozone.client.fs.default.bucket.layout=FILE_SYSTEM_OPTIMIZED, ozone.client.key.latest.version.location=true, ozone.client.key.provider.cache.expiry=10d, ozone.client.list.cache=1000, ozone.client.list.trash.keys.max=1000, ozone.client.max.ec.stripe.write.retries=10, ozone.client.max.retries=5, ozone.client.read.timeout=30s, ozone.client.retry.interval=0, ozone.client.socket.timeout=5000ms, ozone.client.stream.buffer.flush.delay=true, ozone.client.stream.buffer.flush.size=16MB, ozone.client.stream.buffer.increment=0B, ozone.client.stream.buffer.max.size=32MB, ozone.client.stream.buffer.size=4MB, ozone.client.verify.checksum=true, ozone.container.cache.lock.stripes=1024, ozone.container.cache.size=1024, ozone.default.bucket.layout=LEGACY, ozone.directory.deleting.service.interval=1m, ozone.freon.http-address=0.0.0.0:9884, ozone.freon.http-bind-host=0.0.0.0, ozone.freon.http.auth.kerberos.keytab=/etc/security/keytabs/HTTP.keytab, ozone.freon.http.auth.kerberos.principal=HTTP/_HOST@REALM, ozone.freon.http.auth.type=simple, ozone.freon.http.enabled=true, ozone.freon.https-address=0.0.0.0:9885, ozone.freon.https-bind-host=0.0.0.0, ozone.fs.datastream.enabled=false, ozone.fs.iterate.batch-size=100, ozone.fs.listing.page.size=1024, ozone.fs.listing.page.size.max=5000, ozone.http.filter.initializers=org.apache.hadoop.security.AuthenticationFilterInitializer, ozone.http.policy=HTTP_ONLY, ozone.https.client.keystore.resource=ssl-client.xml, ozone.https.client.need-auth=false, ozone.https.server.keystore.resource=ssl-server.xml, ozone.key.deleting.limit.per.task=20000, ozone.key.preallocation.max.blocks=64, ozone.manager.db.checkpoint.transfer.bandwidthPerSec=0, ozone.metadata.dirs=/data/metadata, ozone.metastore.rocksdb.cf.write.buffer.size=128MB, ozone.metastore.rocksdb.statistics=OFF, ozone.network.flexible.fqdn.resolution.enabled=false, ozone.network.jvm.address.cache.enabled=true, ozone.network.topology.aware.read=false, ozone.om.address=om, ozone.om.admin.protocol.max.retries=20, ozone.om.admin.protocol.wait.between.retries=1000, ozone.om.container.location.cache.size=100000, ozone.om.container.location.cache.ttl=360m, ozone.om.enable.filesystem.paths=false, ozone.om.enable.ofs.shared.tmp.dir=false, ozone.om.fs.snapshot.max.limit=1000, ozone.om.grpc.maximum.response.length=134217728, ozone.om.handler.count.key=100, ozone.om.http-address=om:9874, ozone.om.http-bind-host=0.0.0.0, ozone.om.http.auth.kerberos.keytab=/etc/security/keytabs/om.keytab, ozone.om.http.auth.kerberos.principal=HTTP/om@EXAMPLE.COM, ozone.om.http.auth.type=kerberos, ozone.om.http.enabled=true, ozone.om.https-address=0.0.0.0:9875, ozone.om.https-bind-host=0.0.0.0, ozone.om.kerberos.keytab.file=/etc/security/keytabs/om.keytab, ozone.om.kerberos.principal=om/om@EXAMPLE.COM, ozone.om.key.path.lock.enabled=false, ozone.om.keyname.character.check.enabled=false, ozone.om.leader.election.minimum.timeout.duration=5s, ozone.om.multitenancy.enabled=true, ozone.om.multitenancy.ranger.sync.interval=30s, ozone.om.multitenancy.ranger.sync.timeout=10s, ozone.om.open.key.cleanup.limit.per.task=1000, ozone.om.open.key.cleanup.service.interval=24h, ozone.om.open.key.cleanup.service.timeout=300s, ozone.om.open.key.expire.threshold=7d, ozone.om.ratis.enable=true, ozone.om.ratis.log.appender.queue.byte-limit=32MB, ozone.om.ratis.log.appender.queue.num-elements=1024, ozone.om.ratis.log.purge.gap=1000000, ozone.om.ratis.minimum.timeout=5s, ozone.om.ratis.port=9872, ozone.om.ratis.rpc.type=GRPC, ozone.om.ratis.segment.preallocated.size=4MB, ozone.om.ratis.segment.size=4MB, ozone.om.ratis.server.failure.timeout.duration=120s, ozone.om.ratis.server.leaderelection.pre-vote=false, ozone.om.ratis.server.request.timeout=3s, ozone.om.ratis.server.retry.cache.timeout=600000ms, ozone.om.save.metrics.interval=5m, ozone.om.security.admin.protocol.acl=*, ozone.om.security.client.protocol.acl=*, ozone.om.snapshot.cache.max.size=10, ozone.om.snapshot.compaction.dag.max.time.allowed=30d, ozone.om.snapshot.compaction.dag.prune.daemon.run.interval=3600s, ozone.om.snapshot.provider.connection.timeout=5000s, ozone.om.snapshot.provider.request.timeout=5000ms, ozone.om.snapshot.provider.socket.timeout=5000s, ozone.om.unflushed.transaction.max.count=10000, ozone.om.user.max.volume=1024, ozone.om.volume.listall.allowed=true, ozone.path.deleting.limit.per.task=10000, ozone.recon.administrators=testuser2/scm@EXAMPLE.COM, ozone.recon.db.dir=/data/metadata/recon, ozone.recon.http-address=0.0.0.0:9888, ozone.recon.http-bind-host=0.0.0.0, ozone.recon.http.auth.kerberos.keytab=/etc/security/keytabs/recon.keytab, ozone.recon.http.auth.kerberos.principal=HTTP/recon@EXAMPLE.COM, ozone.recon.http.auth.type=kerberos, ozone.recon.http.enabled=true, ozone.recon.https-address=0.0.0.0:9889, ozone.recon.https-bind-host=0.0.0.0, ozone.recon.kerberos.keytab.file=/etc/security/keytabs/recon.keytab, ozone.recon.kerberos.principal=recon/recon@EXAMPLE.COM, ozone.recon.nssummary.flush.db.max.threshold=150000, ozone.recon.om.connection.request.timeout=5000, ozone.recon.om.connection.timeout=5s, ozone.recon.om.snapshot.task.flush.param=false, ozone.recon.om.snapshot.task.initial.delay=20s, ozone.recon.om.snapshot.task.interval.delay=1m, ozone.recon.om.socket.timeout=5s, ozone.recon.scm.connection.request.timeout=5s, ozone.recon.scm.connection.timeout=5s, ozone.recon.scm.container.threshold=100, ozone.recon.scm.snapshot.enabled=true, ozone.recon.scm.snapshot.task.initial.delay=1m, ozone.recon.scm.snapshot.task.interval.delay=24h, ozone.recon.security.client.datanode.container.protocol.acl=*, ozone.recon.task.thread.count=1, ozone.replication.allowed-configs=^((STANDALONE|RATIS)/(ONE|THREE))|(EC/(3-2|6-3|10-4))$, ozone.rest.client.http.connection.max=100, ozone.rest.client.http.connection.per-route.max=20, ozone.s3g.client.buffer.size=4KB, ozone.s3g.http-address=0.0.0.0:9878, ozone.s3g.http-bind-host=0.0.0.0, ozone.s3g.http.auth.kerberos.keytab=/etc/security/keytabs/s3g.keytab, ozone.s3g.http.auth.kerberos.principal=HTTP/s3g@EXAMPLE.COM, ozone.s3g.http.auth.type=kerberos, ozone.s3g.http.enabled=true, ozone.s3g.kerberos.keytab.file=/etc/security/keytabs/s3g.keytab, ozone.s3g.kerberos.principal=s3g/s3g@EXAMPLE.COM, ozone.s3g.volume.name=s3v, ozone.scm.block.client.address=scm, ozone.scm.block.client.bind.host=0.0.0.0, ozone.scm.block.client.port=9863, ozone.scm.block.deletion.max.retry=4096, ozone.scm.block.size=256MB, ozone.scm.ca.list.retry.interval=10s, ozone.scm.chunk.size=4MB, ozone.scm.client.address=scm, ozone.scm.client.bind.host=0.0.0.0, ozone.scm.client.port=9860, ozone.scm.container.layout=FILE_PER_BLOCK2023-02-27T20:45:24.777288085Z , ozone.scm.container.placement.ec.impl=
2023-02-27 12:45:24       org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackScatter
2023-02-27 12:45:24     , ozone.scm.container.placement.impl=
2023-02-27 12:45:24       org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRandom
2023-02-27 12:45:24     , ozone.scm.container.size=1GB, ozone.scm.datanode.admin.monitor.interval=30s, ozone.scm.datanode.disallow.same.peers=false, ozone.scm.datanode.id.dir=/data, ozone.scm.datanode.pipeline.limit=1, ozone.scm.datanode.port=9861, ozone.scm.datanode.ratis.volume.free-space.min=1GB, ozone.scm.dead.node.interval=45s, ozone.scm.ec.pipeline.minimum=5, ozone.scm.event.ContainerReport.thread.pool.size=10, ozone.scm.expired.container.replica.op.scrub.interval=5m, ozone.scm.grpc.port=9895, ozone.scm.ha.grpc.deadline.interval=30m, ozone.scm.ha.ratis.leader.election.timeout=5s, ozone.scm.ha.ratis.leader.ready.check.interval=2s, ozone.scm.ha.ratis.leader.ready.wait.timeout=60s, ozone.scm.ha.ratis.log.appender.queue.byte-limit=32MB, ozone.scm.ha.ratis.log.appender.queue.num-elements=1024, ozone.scm.ha.ratis.log.purge.enabled=false, ozone.scm.ha.ratis.log.purge.gap=1000000, ozone.scm.ha.ratis.request.timeout=30s, ozone.scm.ha.ratis.rpc.type=GRPC, ozone.scm.ha.ratis.segment.preallocated.size=4MB, ozone.scm.ha.ratis.segment.size=4MB, ozone.scm.ha.ratis.server.failure.timeout.duration=120s, ozone.scm.ha.ratis.server.leaderelection.pre-vote=false, ozone.scm.ha.ratis.server.retry.cache.timeout=60s, ozone.scm.ha.ratis.snapshot.threshold=1000, ozone.scm.handler.count.key=100, ozone.scm.heartbeat.log.warn.interval.count=10, ozone.scm.heartbeat.rpc-retry-count=15, ozone.scm.heartbeat.rpc-retry-interval=1s, ozone.scm.heartbeat.rpc-timeout=5s, ozone.scm.heartbeat.thread.interval=3s, ozone.scm.http-address=scm:9876, ozone.scm.http-bind-host=0.0.0.0, ozone.scm.http.enabled=true, ozone.scm.https-address=0.0.0.0:9877, ozone.scm.https-bind-host=0.0.0.0, ozone.scm.info.wait.duration=10m, ozone.scm.keyvalue.container.deletion-choosing.policy=
2023-02-27 12:45:24       org.apache.hadoop.ozone.container.common.impl.TopNOrderedContainerDeletionChoosingPolicy
2023-02-27 12:45:24     , ozone.scm.names=scm, ozone.scm.network.topology.schema.file=network-topology-default.xml, ozone.scm.pipeline.allocated.timeout=5m, ozone.scm.pipeline.creation.auto.factor.one=true, ozone.scm.pipeline.creation.interval=30s, ozone.scm.pipeline.destroy.timeout=66s, ozone.scm.pipeline.leader-choose.policy=
2023-02-27 12:45:24       org.apache.hadoop.hdds.scm.pipeline.leader.choose.algorithms.MinLeaderCountChoosePolicy
2023-02-27 12:45:24     , ozone.scm.pipeline.owner.container.count=1, ozone.scm.pipeline.per.metadata.disk=2, ozone.scm.pipeline.scrub.interval=5m, ozone.scm.ratis.pipeline.limit=0, ozone.scm.ratis.port=9894, ozone.scm.security.handler.count.key=2, ozone.scm.security.service.bind.host=0.0.0.0, ozone.scm.security.service.port=9961, ozone.scm.sequence.id.batch.size=1000, ozone.scm.skip.bootstrap.validation=false, ozone.scm.stale.node.interval=30s, ozone.scm.update.client.crl.check.interval=600s, ozone.scm.update.service.port=9893, ozone.security.enabled=true, ozone.security.http.kerberos.enabled=true, ozone.server.default.replication=3, ozone.server.default.replication.type=RATIS, ozone.service.shutdown.timeout=60s, ozone.snapshot.filtering.limit.per.task=2, ozone.snapshot.filtering.service.interval=1m, ozone.sst.filtering.service.timeout=300000ms, ozone.trace.enabled=false, recon.om.delta.update.limit=2000, recon.om.delta.update.loop.limit=10, scm.container.client.idle.threshold=10s, scm.container.client.max.size=256}

hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java

adoroszlai

Thanks @SaketaChalamchala for updating the patch.

adoroszlai · 2023-02-28T07:30:30Z

One minor question:

## Full list of configs printed in all role logs : 
STARTUP_MSG:   conf = {... ozone.scm.container.placement.ec.impl=
2023-02-27 12:45:24       org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackScatter
2023-02-27 12:45:24     , ozone.scm.container.placement.impl=
2023-02-27 12:45:24       org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRandom
2023-02-27 12:45:24     , ... ozone.scm.keyvalue.container.deletion-choosing.policy=
2023-02-27 12:45:24       org.apache.hadoop.ozone.container.common.impl.TopNOrderedContainerDeletionChoosingPolicy
2023-02-27 12:45:24     , ... ozone.scm.pipeline.leader-choose.policy=
2023-02-27 12:45:24       org.apache.hadoop.hdds.scm.pipeline.leader.choose.algorithms.MinLeaderCountChoosePolicy
2023-02-27 12:45:24     , ...}

These newlines are the result of the whitespace embedded in some config properties' default values, e.g.:

ozone/hadoop-hdds/common/src/main/resources/ozone-default.xml

Lines 842 to 845 in e90e2dd

    
           <name>ozone.scm.keyvalue.container.deletion-choosing.policy</name> 
        
           <value> 
        
             org.apache.hadoop.ozone.container.common.impl.TopNOrderedContainerDeletionChoosingPolicy 
        
           </value>

It would be nice to ensure conf = {...} is printed in a single log line. Should we remove such whitespace from the specific properties in ozone-default.xml, or trim values programmatically in HddsUtils#processForLogging?

hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java

SaketaChalamchala · 2023-02-28T17:31:11Z

It would be nice to ensure conf = {...} is printed in a single log line. Should we remove such whitespace from the specific properties in ozone-default.xml, or trim values programmatically in HddsUtils#processForLogging?

It might be better to do trim the strings in HddsUtils#processForLogging. As Ethan mentioned before, we wouldn't have to worry about any new configs with whitespaces in the added in the future.

errose28

Thanks for the updates @SaketaChalamchala LGTM.

pertel-chen · 2024-12-26T16:54:52Z

hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/conf/OzoneConfiguration.java

this ozoneregex not contains "dfs.datanode.use.datanode.hostname",so "dfs.datanode.use.datanode.hostname" init only once and not modifiy

These dfs configs are supposed to be deprecated, but it looks like Ozone is still using some of them instead of migrating them to its own config keys. We migrated most of them, but missed the ones in DFSConfigKeysLegacy. I filed HDDS-12026 to fix this. Would you like to work on it?

HDDS-7869. Log configuration on component startup.

58d7db2

kerneltime self-requested a review February 13, 2023 17:31

sumitagrawl reviewed Feb 14, 2023

View reviewed changes

...rver-scm/src/main/java/org/apache/hadoop/hdds/scm/server/StorageContainerManagerStarter.java Outdated Show resolved Hide resolved

adoroszlai reviewed Feb 14, 2023

View reviewed changes

HDDS-7869. Printing all configs. Added utility function to process an…

286734e

…d sort conf object for logging.

saketa added 4 commits February 16, 2023 15:30

HDDS-7869. Fixed checkstyle. Removed unused import.

d362e5d

HDDS-7869. Fixed variable name and whitespace checkstyle errors.

ab57e03

HDDS-7869. Fixed checkstyle errors. Removed redundant import and fixe…

96167b6

…d whitespace errors.

HDDS-7869. Filtered Ozone configs only.

e2b0e6c

kerneltime approved these changes Feb 28, 2023

View reviewed changes

adoroszlai reviewed Feb 28, 2023

View reviewed changes

hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java Outdated Show resolved Hide resolved

Fix findbugs

316c812

adoroszlai approved these changes Feb 28, 2023

View reviewed changes

adoroszlai reviewed Feb 28, 2023

View reviewed changes

hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/HddsUtils.java Show resolved Hide resolved

adoroszlai mentioned this pull request Feb 28, 2023

HDDS-8046. Eliminate embedded newlines from config properties #4328

Merged

Trim config values

f5accde

adoroszlai requested a review from errose28 February 28, 2023 21:43

errose28 approved these changes Mar 1, 2023

View reviewed changes

errose28 merged commit 84f1523 into apache:master Mar 1, 2023

pertel-chen reviewed Dec 26, 2024

View reviewed changes

HDDS-7869. Log configuration on component startup. #4271

HDDS-7869. Log configuration on component startup. #4271

Uh oh!

Conversation

SaketaChalamchala commented Feb 13, 2023

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

kerneltime commented Feb 13, 2023

Uh oh!

neils-dev commented Feb 13, 2023

Uh oh!

sumitagrawl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adoroszlai Feb 14, 2023

Choose a reason for hiding this comment

Uh oh!

SaketaChalamchala Feb 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

errose28 commented Feb 15, 2023

Uh oh!

SaketaChalamchala commented Feb 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SaketaChalamchala commented Feb 16, 2023

Uh oh!

SaketaChalamchala commented Feb 28, 2023

Uh oh!

Uh oh!

adoroszlai left a comment

Choose a reason for hiding this comment

Uh oh!

adoroszlai commented Feb 28, 2023

Uh oh!

Uh oh!

SaketaChalamchala commented Feb 28, 2023

Uh oh!

errose28 left a comment

Choose a reason for hiding this comment

Uh oh!

pertel-chen Dec 26, 2024

Choose a reason for hiding this comment

Uh oh!

errose28 Jan 6, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

SaketaChalamchala Feb 16, 2023 •

edited

Loading

SaketaChalamchala commented Feb 16, 2023 •

edited

Loading