Skip to content

Parameterize nightly script and remove user dir dependency #8047

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hickeng merged 1 commit into from
Jun 15, 2018
Merged

Parameterize nightly script and remove user dir dependency #8047

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions tests/manual-test-cases/Group5-Functional-Tests/5-11-Multiple-Cluster.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Multiple Cluster Setup
[Timeout] 110 minutes
Run Keyword And Ignore Error Nimbus Cleanup Single VM '*5-11-multiple-cluster*' ${false}
Log To Console \nStarting testbed deploy...
${out}= Deploy Nimbus Testbed %{NIMBUS_USER} %{NIMBUS_PASSWORD} --noSupportBundles --plugin testng --vcvaBuild ${VC_VERSION} --esxBuild ${ESX_VERSION} --testbedName vic-multiple-cluster --testbedSpecRubyFile /dbc/pa-dbc1111/mhagen/nimbus-testbeds/testbeds/vic-multiple-cluster.rb --runName 5-11-multiple-cluster
${out}= Deploy Nimbus Testbed %{NIMBUS_USER} %{NIMBUS_PASSWORD} spec=vic-multiple-cluster.rb args=--noSupportBundles --plugin testng --vcvaBuild ${VC_VERSION} --esxBuild ${ESX_VERSION} --testbedName vic-multiple-cluster --runName 5-11-multiple-cluster
Log ${out}

Open Connection %{NIMBUS_GW}
Expand Down Expand Up @@ -56,7 +56,6 @@ Multiple Cluster Setup
*** Test Cases ***
Test
Log To Console \nStarting test...
Custom Testbed Keepalive /dbc/pa-dbc1111/mhagen

Install VIC Appliance To Test Server certs=${false} vol=default
Run Regression Tests
2 changes: 1 addition & 1 deletion tests/manual-test-cases/Group5-Functional-Tests/5-15-NFS-Datastore.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ NFS Datastore Setup
[Timeout] 110 minutes
Run Keyword And Ignore Error Nimbus Cleanup Single VM '*5-15-nfs-datastore*' ${false}
Log To Console \nStarting testbed deploy...
${out}= Deploy Nimbus Testbed %{NIMBUS_USER} %{NIMBUS_PASSWORD} --noSupportBundles --plugin testng --vcvaBuild ${VC_VERSION} --esxBuild ${ESX_VERSION} --testbedName vic-simple-cluster --testbedSpecRubyFile /dbc/pa-dbc1111/mhagen/nimbus-testbeds/testbeds/vic-simple-cluster.rb --runName 5-15-nfs-datastore
${out}= Deploy Nimbus Testbed %{NIMBUS_USER} %{NIMBUS_PASSWORD} spec=vic-simple-cluster.rb args=--noSupportBundles --plugin testng --vcvaBuild ${VC_VERSION} --esxBuild ${ESX_VERSION} --testbedName vic-simple-cluster --runName 5-15-nfs-datastore
Log ${out}

Open Connection %{NIMBUS_GW}
Expand Down
2 changes: 1 addition & 1 deletion tests/manual-test-cases/Group5-Functional-Tests/5-22-NFS-Volume.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ Setup ESX And NFS Suite
Run Keyword And Ignore Error Nimbus Cleanup ${list} ${false}
Log To Console \nStarting test...

${nfs} ${nfs_ro} ${esx1} ${nfs_ip} ${nfs_ro_ip} ${esx1_ip}= Deploy Simple NFS Testbed %{NIMBUS_USER} %{NIMBUS_PASSWORD} additional-args=--testbedSpecRubyFile /dbc/w3-dbc302/rashok/vic-nfs.rb --esxBuild ${ESX_VERSION} --esxPxeDir ${ESX_VERSION} --plugin testng
${nfs} ${nfs_ro} ${esx1} ${nfs_ip} ${nfs_ro_ip} ${esx1_ip}= Deploy Simple NFS Testbed %{NIMBUS_USER} %{NIMBUS_PASSWORD} spec=vic-nfs.rb args=--esxBuild ${ESX_VERSION} --esxPxeDir ${ESX_VERSION} --plugin testng

Set Suite Variable @{list} ${esx1} ${nfs} ${nfs_ro}
Set Suite Variable ${ESX1} ${esx1}
Expand Down
1 change: 0 additions & 1 deletion tests/manual-test-cases/Group5-Functional-Tests/5-26-Static-IP-Address.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,6 @@ Setup VC With Static IP
*** Test Cases ***
Test
Log To Console \nStarting test...
Custom Testbed Keepalive /dbc/pa-dbc1111/mhagen

Install VIC Appliance To Test Server additional-args=--public-network-ip &{static}[ip]/&{static}[netmask] --public-network-gateway &{static}[gateway] --dns-server 10.170.16.48
Run Regression Tests
3 changes: 1 addition & 2 deletions tests/manual-test-cases/Group5-Functional-Tests/5-27-Selenium-Grid.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Selenium Grid Test Setup
Log To Console Starting testbed deployment...
Run Keyword And Ignore Error Nimbus Cleanup ${list} ${false}
${name}= Evaluate 'vic-5-27-' + str(random.randint(1000,9999)) modules=random
${out}= Deploy Nimbus Testbed %{NIMBUS_USER} %{NIMBUS_PASSWORD} --noSupportBundles --plugin testng --vcvaBuild ${VC_VERSION} --esxBuild ${ESX_VERSION} --testbedName vic-iscsi-cluster --testbedSpecRubyFile /dbc/pa-dbc1111/mhagen/nimbus-testbeds/testbeds/vic-iscsi-cluster.rb --runName ${name}
${out}= Deploy Nimbus Testbed %{NIMBUS_USER} %{NIMBUS_PASSWORD} spec=vic-iscsi-cluster.rb args=--noSupportBundles --plugin testng --vcvaBuild ${VC_VERSION} --esxBuild ${ESX_VERSION} --testbedName vic-iscsi-cluster --runName ${name}
Log ${out}

Open Connection %{NIMBUS_GW}
Expand Down Expand Up @@ -71,7 +71,6 @@ Wait Until Selenium Node Is Ready
*** Test Cases ***
Test
Log To Console Starting Selenium Grid test...
Custom Testbed Keepalive /dbc/pa-dbc1111/mhagen

Install VIC Appliance To Test Server

Expand Down
2 changes: 0 additions & 2 deletions tests/manual-test-cases/Group5-Functional-Tests/5-6-1-VSAN-Simple.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,8 +68,6 @@ Check VSAN DOMs In Datastore
Simple VSAN
Wait Until Keyword Succeeds 10x 30s Check VSAN DOMs In Datastore %{TEST_DATASTORE}

Custom Testbed Keepalive /dbc/pa-dbc1111/mhagen

Install VIC Appliance To Test Server
Run Regression Tests
Cleanup VIC Appliance On Test Server
Expand Down
2 changes: 0 additions & 2 deletions tests/manual-test-cases/Group5-Functional-Tests/5-6-2-VSAN-Complex.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,8 +71,6 @@ Check VSAN DOMs In Datastore
Complex VSAN
Wait Until Keyword Succeeds 10x 30s Check VSAN DOMs In Datastore %{TEST_DATASTORE}

Custom Testbed Keepalive /dbc/pa-dbc1111/mhagen

Install VIC Appliance To Test Server
Run Regression Tests
Cleanup VIC Appliance On Test Server
Expand Down
140 changes: 79 additions & 61 deletions tests/nightly/jenkins-nightly-run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,84 +16,102 @@
ESX_60_VERSION="ob-5251623"
Copy link
Contributor Author

@hickeng hickeng Jun 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add script usage doc.

VC_60_VERSION="ob-5112509"

ESX_65_VERSION="ob-7867845"
VC_65_VERSION="ob-7867539"

ESX_67_VERSION="ob-8169922"
VC_67_VERSION="ob-8217866"

DEFAULT_LOG_UPLOAD_DEST="vic-ci-logs"
DEFAULT_VCH_BUILD="*"
DEFAULT_TESTCASES="tests/manual-test-cases/Group5-Functional-Tests tests/manual-test-cases/Group13-vMotion tests/manual-test-cases/Group21-Registries tests/manual-test-cases/Group23-Future-Tests"


export RUN_AS_OPS_USER=0

if [[ $1 != "6.0" && $1 != "6.5" && $1 != "6.7" ]]; then
echo "Please specify a target cluster. One of: 6.0, 6.5, 6.7"
echo "Please specify a target version. One of: 6.0, 6.5, 6.7"
exit 1
fi

# process the CLI arguments
target="$1"
echo "Target cluster: "$target

input=$(gsutil ls -l gs://vic-engine-builds/vic_* | grep -v TOTAL | sort -k2 -r | head -n1 | xargs | cut -d ' ' -f 3 | cut -d '/' -f 4)
buildNumber=${input:4}

n=0
until [ $n -ge 5 ]
do
echo "Retry.. $n"
echo "Downloading gcp file $input"
wget https://storage.googleapis.com/vic-engine-builds/$input
if [ -f "$input" ]
then
echo "File found.."
break
else
echo "File NOT found"
fi
n=$[$n+1]
sleep 15
done

n=0
until [ $n -ge 5 ]
do
mkdir bin
echo "Extracting .tar.gz"
tar xvzf $input -C bin/ --strip 1
if [ -f "bin/vic-machine-linux" ]
then
echo "tar extraction complete.."
canContinue="Yes"
break
else
echo "tar extraction failed"
canContinue="No"
rm -rf bin
fi
n=$[$n+1]
sleep 15
done

if [[ $canContinue = "No" ]]; then
echo "Target version: ${target}"
shift
# Take the remaining CLI arguments as a test case list
testcases="${*:-$DEFAULT_TESTCASES}"

# TODO: the version downloaded by this logic is not coupled with the tests that will be run against it. This should be altered to pull a version that matches the commit SHA of the tests
# we will be running or similar mechanism.
VCH_BUILD=${VCH_BUILD:-${DEFAULT_VCH_BUILD}}
input=$(gsutil ls -l gs://vic-engine-builds/vic_${VCH_BUILD} | grep -v TOTAL | sort -k2 -r | head -n1 | xargs | cut -d ' ' -f 3 | cut -d '/' -f 4)
VCH_BUILD=${input:4}

# Enforce short SHA
GIT_COMMIT=${GIT_COMMIT:0:7}

case "$target" in
"6.0")
excludes="--exclude nsx"
ESX_BUILD=${ESX_BUILD:-$ESX_60_VERSION}
VC_BUILD=${VC_BUILD:-$VC_60_VERSION}
;;
"6.5")
ESX_BUILD=${ESX_BUILD:-$ESX_65_VERSION}
VC_BUILD=${VC_BUILD:-$VC_65_VERSION}
;;
"6.7")
excludes="--exclude nsx --exclude hetero"
ESX_BUILD=${ESX_BUILD:-$ESX_67_VERSION}
VC_BUILD=${VC_BUILD:-$VC_67_VERSION}
;;
esac

LOG_UPLOAD_DEST="${LOG_UPLOAD_DEST:-${DEFAULT_LOG_UPLOAD_DEST}}"

n=0 && rm -f "${input}"
until [ $n -ge 5 -o -f "${input}" ]; do
echo "Retry.. $n"
echo "Downloading gcp file ${input}"
wget -nv https://storage.googleapis.com/vic-engine-builds/${input}

((n++))
sleep 15
done

echo "Extracting .tar.gz"
mkdir bin && tar xvzf ${input} -C bin/ --strip 1

if [ ! -f "bin/vic-machine-linux" ]; then
echo "Tarball extraction failed..quitting the run"
break
rm -rf bin
exit
else
VCH_COMMIT=$(bin/vic-machine-linux version | awk -F '-' '{print $NF}')
echo "Tarball extraction passed, Running nightlies test.."
fi

if [[ $target == "6.0" ]]; then
echo "Executing nightly tests on vSphere 6.0"
pabot --processes 4 --removekeywords TAG:secret --exclude nsx --variable ESX_VERSION:$ESX_60_VERSION --variable VC_VERSION:$VC_60_VERSION -d 60/$i tests/manual-test-cases/Group5-Functional-Tests tests/manual-test-cases/Group13-vMotion tests/manual-test-cases/Group21-Registries tests/manual-test-cases/Group23-Future-Tests
cat 60/pabot_results/*/stdout.txt | grep '::' | grep -E 'PASS|FAIL' > console.log
elif [[ $target == "6.5" ]]; then
echo "Executing nightly tests on vSphere 6.5"
pabot --processes 4 --removekeywords TAG:secret -d 65/$i tests/manual-test-cases/Group5-Functional-Tests tests/manual-test-cases/Group13-vMotion tests/manual-test-cases/Group21-Registries tests/manual-test-cases/Group23-Future-Tests
cat 65/pabot_results/*/stdout.txt | grep '::' | grep -E 'PASS|FAIL' > console.log
elif [[ $target == "6.7" ]]; then
echo "Executing nightly tests on vSphere 6.7"
pabot --processes 4 --removekeywords TAG:secret --exclude nsx --exclude hetero --variable ESX_VERSION:$ESX_67_VERSION --variable VC_VERSION:$VC_67_VERSION -d 67/$i tests/manual-test-cases/Group5-Functional-Tests tests/manual-test-cases/Group13-vMotion tests/manual-test-cases/Group21-Registries tests/manual-test-cases/Group23-Future-Tests
cat 67/pabot_results/*/stdout.txt | grep '::' | grep -E 'PASS|FAIL' > console.log

pabot --processes 4 --removekeywords TAG:secret ${excludes} --variable ESX_VERSION:${ESX_BUILD} --variable VC_VERSION:${VC_BUILD} -d ${target} "${testcases}"
cat ${target}/pabot_results/*/stdout.txt | grep '::' | grep -E 'PASS|FAIL' > console.log

# See if any VMs leaked
# TODO: should be a warning until clean, then changed to a failure if any leak
echo "There should not be any VMs listed here"
echo "======================================="
timeout 60s sshpass -p ${NIMBUS_PASSWORD} ssh -o StrictHostKeyChecking\=no ${NIMBUS_USER}@${NIMBUS_GW} nimbus-ctl list
echo "======================================="
echo "If VMs are listed we should investigate why they are leaking"

# archive the logs
logarchive="logs_vch-${VCH_BUILD}-${VCH_COMMIT}_test-${BUILD_ID}-${GIT_COMMIT}_${BUILD_TIMESTAMP}.zip"
/usr/bin/zip -9 -r "${logarchive}" "${target}" *.zip *.log *.debug *.tgz
if [ $? -eq 0 ]; then
tests/nightly/upload-logs.sh ${logarchive} ${LOG_UPLOAD_DEST}
fi

# Pretty up the email results
sed -i -e 's/^/<br>/g' console.log
sed -i -e 's|PASS|<font color="green">PASS</font>|g' console.log
sed -i -e 's|FAIL|<font color="red">FAIL</font>|g' console.log

# See if any VMs leaked
timeout 60s sshpass -p $NIMBUS_PASSWORD ssh -o StrictHostKeyChecking\=no $NIMBUS_USER@$NIMBUS_GW nimbus-ctl list

tests/nightly/upload-logs.sh $target_$BUILD_TIMESTAMP
53 changes: 27 additions & 26 deletions tests/nightly/upload-logs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,28 +14,23 @@
# limitations under the License.
#

echo "Upload logs"
source=${1:?A source file must be specified}
# The target destination in cloud storage. If the destination ends with a / it will be treated as a directory.
# If not it _may_ be treated as a directory or a file depending on current remote objects. see gsutil cp doc.
dest=${2:?A bucket must be specified, eg vic-ci-logs, or vic-ci-logs/user/branch/}

set -x
gsutil version -l
set +x

outfile="vic_nightly_logs_"$1".zip"
echo $outfile
if [ ! -r "${source}" ]; then
echo "Specified source file does not exist or cannot be read: ${source}"
exit 1
fi

if [ -d "60" ]; then
/usr/bin/zip -9 -r $outfile 60 *.zip *.log *.debug *.tgz
elif [ -d "65" ]; then
/usr/bin/zip -9 -r $outfile 65 *.zip *.log *.debug *.tgz
elif [ -d "67" ]; then
/usr/bin/zip -9 -r $outfile 67 *.zip *.log *.debug *.tgz
else
echo "No output directories to upload!"
exit 1
if [ ${dest:0:1} == "/" ]; then
echo "Destination must start with a bucket name and no leading /"
exit 1
fi

# GC credentials
keyfile=~/vic-ci-logs.key
keyfile=~/${dest%%/*}.key
botofile=~/.boto
if [ ! -f $keyfile ]; then
echo -en $GS_PRIVATE_KEY > $keyfile
Expand All @@ -50,16 +45,22 @@ if [ ! -f $botofile ]; then
echo "default_project_id = $GS_PROJECT_ID" >> $botofile
fi

if [ -f "$outfile" ]; then
gsutil cp $outfile gs://vic-ci-logs
echo "----------------------------------------------"
echo "----------------------------------------------"
echo "Uploading logs to ${dest}"
echo "----------------------------------------------"
if gsutil cp "${source}" "gs://${dest}"; then
url="https://console.cloud.google.com/m/cloudstorage/b/${dest}/o/${source}?authuser=1"
echo "$url" > log-download.url
echo "Download test logs here:"
echo "https://console.cloud.google.com/m/cloudstorage/b/vic-ci-logs/o/$outfile?authuser=1"
echo "----------------------------------------------"
echo "$url"
else
echo "No log output file to upload"
echo "Log upload faled. Dumping gsutil version logic"
set -x
gsutil version -l
set +x
fi
echo "----------------------------------------------"

rm -f $keyfile


if [ -f "$keyfile" ]; then
rm -f $keyfile
fi
Loading