Skip to content

HDDS-2646. Start acceptance tests only if at least one THREE pipeline is available #282

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
anuengineer merged 6 commits into from
Dec 4, 2019
Merged

HDDS-2646. Start acceptance tests only if at least one THREE pipeline is available #282

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
30effc2
HDDS-2646. Start acceptance tests only if at least one THREE pipeline…
elek Nov 28, 2019
41733fc
retrigger build
elek Nov 28, 2019
758c7d6
Merge remote-tracking branch 'origin/master' into HDDS-2646
elek Nov 29, 2019
bd99167
remove configs where we don't need them
elek Dec 3, 2019
1300a0a
switch to use ENV based variable with default
elek Dec 3, 2019
0312cdf
Merge remote-tracking branch 'origin/master' into HDDS-2646
elek Dec 3, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/ozone-mr/common-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ OZONE-SITE.XML_ozone.scm.block.client.address=scm
OZONE-SITE.XML_ozone.metadata.dirs=/data/metadata
OZONE-SITE.XML_ozone.scm.client.address=scm
OZONE-SITE.XML_ozone.replication=3
OZONE-SITE.XML_hdds.scm.safemode.min.datanode=3
Copy link
Contributor

@ChenSammi ChenSammi Nov 29, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not very faimilar with docker-compose. Where do we tell docker-compose to start three datanodes with all these configurations?

Copy link
Member Author

@elek elek Dec 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test.sh calls start_docker_env from the testlib.sh which calls docker-compose scale datanode=3.

Unfortunately there is no easy way to define the expected number of the containers in the docker-compose.yaml. ( There is a deploy / replicas but it's available only for docker swarm and not for docker-compose)

Copy link
Contributor

@ChenSammi ChenSammi Dec 5, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. Thanks for the explanation.


OZONE-SITE.XML_hdds.datanode.dir=/data/hdds
HDFS-SITE.XML_dfs.datanode.address=0.0.0.0:1019
Expand Down
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/ozone-mr/hadoop27/docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ services:
- ../common-config
environment:
ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: "${OZONE_SAFEMODE_MIN_DATANODES:-1}"
command: ["/opt/hadoop/bin/ozone","scm"]
rm:
image: ${HADOOP_IMAGE}:${HADOOP_VERSION}
Expand Down
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/ozone-mr/hadoop31/docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ services:
- ../common-config
environment:
ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: "${OZONE_SAFEMODE_MIN_DATANODES:-1}"
command: ["/opt/hadoop/bin/ozone","scm"]
rm:
image: ${HADOOP_IMAGE}:${HADOOP_VERSION}
Expand Down
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/ozone-mr/hadoop32/docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ services:
- ../common-config
environment:
ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: "${OZONE_SAFEMODE_MIN_DATANODES:-1}"
command: ["/opt/hadoop/bin/ozone","scm"]
rm:
image: ${HADOOP_IMAGE}:${HADOOP_VERSION}
Expand Down
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/ozone-om-ha-s3/docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ services:
- ./docker-config
environment:
ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: "${OZONE_SAFEMODE_MIN_DATANODES:-1}"
command: ["/opt/hadoop/bin/ozone","scm"]
s3g:
image: apache/ozone-runner:${OZONE_RUNNER_VERSION}
Expand Down
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/ozone-recon/docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ services:
- ./docker-config
environment:
ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: "${OZONE_SAFEMODE_MIN_DATANODES:-1}"
command: ["/opt/hadoop/bin/ozone","scm"]
recon:
image: apache/ozone-runner:${OZONE_RUNNER_VERSION}
Expand Down
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/ozone-topology/docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,7 @@ services:
- ./docker-config
environment:
ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: "${OZONE_SAFEMODE_MIN_DATANODES:-1}"
command: ["/opt/hadoop/bin/ozone","scm"]
networks:
net:
Expand Down
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/ozone-topology/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ OZONE-SITE.XML_ozone.scm.container.placement.impl=org.apache.hadoop.hdds.scm.con
OZONE-SITE.XML_net.topology.node.switch.mapping.impl=org.apache.hadoop.net.TableMapping
OZONE-SITE.XML_net.topology.table.file.name=/opt/hadoop/compose/ozone-topology/network-config
OZONE-SITE.XML_dfs.network.topology.aware.read.enable=true

HDFS-SITE.XML_rpc.metrics.quantile.enable=true
HDFS-SITE.XML_rpc.metrics.percentiles.intervals=60,300
ASYNC_PROFILER_HOME=/opt/profiler
Expand Down
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/ozone/docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,4 +50,5 @@ services:
- ./docker-config
environment:
ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: "${OZONE_SAFEMODE_MIN_DATANODES:-1}"
command: ["/opt/hadoop/bin/ozone","scm"]
2 changes: 2 additions & 0 deletions hadoop-ozone/dist/src/main/compose/ozoneblockade/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@ OZONE-SITE.XML_hdds.scm.wait.time.after.safemode.exit=30s
OZONE-SITE.XML_hdds.scm.replication.thread.interval=6s
OZONE-SITE.XML_hdds.scm.replication.event.timeout=10s
OZONE-SITE.XML_dfs.ratis.server.failure.duration=35s
OZONE-SITE.XML_hdds.scm.safemode.min.datanode=3

HDFS-SITE.XML_rpc.metrics.quantile.enable=true
HDFS-SITE.XML_rpc.metrics.percentiles.intervals=60,300

Expand Down
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/ozoneperf/docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ services:
- ./docker-config
environment:
ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: "${OZONE_SAFEMODE_MIN_DATANODES:-1}"
command: ["ozone","scm"]
jaeger:
image: jaegertracing/all-in-one:latest
Expand Down
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/ozones3-haproxy/docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ services:
- ./docker-config
environment:
ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: "${OZONE_SAFEMODE_MIN_DATANODES:-1}"
command: ["ozone","scm"]
s3g1:
image: apache/ozone-runner:${OZONE_RUNNER_VERSION}
Expand Down
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/ozones3/docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ services:
- ./docker-config
environment:
ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: "${OZONE_SAFEMODE_MIN_DATANODES:-1}"
command: ["ozone","scm"]
s3g:
image: apache/ozone-runner:${OZONE_RUNNER_VERSION}
Expand Down
3 changes: 2 additions & 1 deletion hadoop-ozone/dist/src/main/compose/ozonescripts/docker-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,9 @@ OZONE-SITE.XML_ozone.scm.client.address=scm
OZONE-SITE.XML_ozone.replication=1
OZONE-SITE.XML_hdds.datanode.dir=/data/hdds
OZONE-SITE.XML_hdds.datanode.plugins=org.apache.hadoop.ozone.web.OzoneHddsDatanodeService

HDFS-SITE.XML_dfs.namenode.rpc-address=namenode:9000
HDFS-SITE.XML_dfs.namenode.name.dir=/data/namenode
HDFS-SITE.XML_rpc.metrics.quantile.enable=true
HDFS-SITE.XML_rpc.metrics.percentiles.intervals=60,300
HDFS-SITE.XML_dfs.datanode.plugins=org.apache.hadoop.ozone.HddsDatanodeService
HDFS-SITE.XML_dfs.datanode.plugins=org.apache.hadoop.ozone.HddsDatanodeService
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/ozonesecure-mr/docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@ services:
- docker-config
environment:
ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: "${OZONE_SAFEMODE_MIN_DATANODES:-1}"
command: ["/opt/hadoop/bin/ozone","scm"]
rm:
image: apache/hadoop:${HADOOP_VERSION}
Expand Down
1 change: 1 addition & 0 deletions hadoop-ozone/dist/src/main/compose/ozonesecure/docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,4 +88,5 @@ services:
- docker-config
environment:
ENSURE_SCM_INITIALIZED: /data/metadata/scm/current/VERSION
OZONE-SITE.XML_hdds.scm.safemode.min.datanode: "${OZONE_SAFEMODE_MIN_DATANODES:-1}"
command: ["/opt/hadoop/bin/ozone","scm"]
51 changes: 1 addition & 50 deletions hadoop-ozone/dist/src/main/compose/testlib.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,54 +31,6 @@ create_results_dir() {
chmod ogu+w "$RESULT_DIR"
}

## @description print the number of datanodes up
## @param the docker-compose file
count_datanodes() {
local compose_file=$1

local jmx_url='http://scm:9876/jmx?qry=Hadoop:service=SCMNodeManager,name=SCMNodeManagerInfo'
if [[ "${SECURITY_ENABLED}" == 'true' ]]; then
docker-compose -f "${compose_file}" exec -T scm bash -c "kinit -k HTTP/scm@EXAMPLE.COM -t /etc/security/keytabs/HTTP.keytab && curl --negotiate -u : -s '${jmx_url}'"
else
docker-compose -f "${compose_file}" exec -T scm curl -s "${jmx_url}"
fi \
| jq -r '.beans[0].NodeCount[] | select(.key=="HEALTHY") | .value' || true
}

## @description wait until datanodes are up (or 30 seconds)
## @param the docker-compose file
## @param number of datanodes to wait for (default: 3)
wait_for_datanodes(){
local compose_file=$1
local -i datanode_count=${2:-3}

#Reset the timer
SECONDS=0

#Don't give it up until 30 seconds
while [[ $SECONDS -lt 90 ]]; do

#This line checks the number of HEALTHY datanodes registered in scm over the
# jmx HTTP servlet
datanodes=$(count_datanodes "${compose_file}")
if [[ "$datanodes" ]]; then
if [[ ${datanodes} -ge ${datanode_count} ]]; then

#It's up and running. Let's return from the function.
echo "$datanodes datanodes are up and registered to the scm"
return
else

#Print it only if a number. Could be not a number if scm is not yet started
echo "$datanodes datanode is up and healthy (until now)"
fi
fi

sleep 2
done
echo "WARNING! Datanodes are not started successfully. Please check the docker-compose files"
return 1
}

## @description wait until safemode exit (or 30 seconds)
## @param the docker-compose file
Expand Down Expand Up @@ -120,10 +72,9 @@ start_docker_env(){
local -i datanode_count=${1:-3}

create_results_dir

export OZONE_SAFEMODE_MIN_DATANODE="${datanode_count}"
docker-compose -f "$COMPOSE_FILE" --no-ansi down
docker-compose -f "$COMPOSE_FILE" --no-ansi up -d --scale datanode="${datanode_count}" \
&& wait_for_datanodes "$COMPOSE_FILE" "${datanode_count}" \
&& wait_for_safemode_exit "$COMPOSE_FILE" \
&& sleep 10

Expand Down