Skip to content

USHIFT-1312 Test: Rebooting healthy system should result in backing up the data #1866

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
openshift-merge-robot merged 2 commits into from
Jun 19, 2023
Merged

USHIFT-1312 Test: Rebooting healthy system should result in backing up the data #1866

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7a71506
rebooting healthy system should result in backing up the data
pmtk Jun 1, 2023
a7d4fdf
add JSON -> DotDict helper
pmtk Jun 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion test/resources/YAML.py → test/resources/DataFormats.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,21 @@
"""Expose YAML parser to robot tests.
"""Provides helpers for common data formats
"""
import json
import yaml
from robot.utils import DotDict


def json_parse(data):
"""Parse input string as JSON and return DotDict instance.

If the input data is empty, return an empty DotDict.
"""
if not data:
return DotDict()
parsed = json.loads(data)
return DotDict(parsed)


def yaml_parse(data):
"""Parse input string as YAML and return DotDict instance.

Expand Down
2 changes: 1 addition & 1 deletion test/resources/microshift-config.resource
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Library String
Library OperatingSystem

Library SSHLibrary
Library ../resources/YAML.py
Library DataFormats.py


*** Keywords ***
Expand Down
12 changes: 12 additions & 0 deletions test/resources/microshift-host.resource
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,15 @@ Login MicroShift Host
Logout MicroShift Host
[Documentation] Close the open ssh connection to the MicroShift host
SSHLibrary.Close Connection

Reboot MicroShift Host
Copy link
Copy Markdown
Contributor

@dhellmann dhellmann Jun 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be more robust if we looked for a different boot id after the reboot, to avoid any race condition with issuing the reboot command and sshd stopping. It doesn't seem like the most likely race, so let's make that change in another PR.

[Documentation] Reboot the MicroShift host
...
... Expects that SSH connection to MicroShift host is active.

SSHLibrary.Start Command reboot sudo=True
Sleep 30s
Wait Until Keyword Succeeds 5m 15s Login MicroShift Host

${rc}= SSHLibrary.Execute Command true return_stdout=False return_rc=True
Should Be Equal As Integers 0 ${rc}
21 changes: 17 additions & 4 deletions test/resources/microshift-process.resource
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@ Library String
Library OperatingSystem

Library SSHLibrary
Resource ../resources/oc.resource
Resource ../resources/systemd.resource
Resource ../resources/microshift-host.resource
Library ../resources/YAML.py
Resource oc.resource
Resource systemd.resource
Resource microshift-host.resource
Library DataFormats.py


*** Keywords ***
Expand Down Expand Up @@ -47,6 +47,19 @@ Wait For MicroShift
# greenboot check script to let us use it even when not on a
# greenboot host?

MicroShift Service Is Active
[Documentation] Check if microshift.service is active
... (process announced readiness)
${state}= Get Systemd Setting microshift.service ActiveState
Should Be Equal As Strings ${state} active

Wait For MicroShift Service
Comment thread
dhellmann marked this conversation as resolved.
Outdated
[Documentation] Wait for MicroShift service to be active
Wait Until Keyword Succeeds
... 50x
... 10s
... MicroShift Service Is Active

Restart MicroShift
[Documentation] Restart the MicroShift service
# Use separate stop and start steps to avoid a race condition with
Expand Down
2 changes: 1 addition & 1 deletion test/resources/oc.resource
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Documentation Keywords for using the oc command line.
... 'oc' is in the $PATH.

Resource common.resource
Library YAML.py
Library DataFormats.py


*** Keywords ***
Expand Down
63 changes: 63 additions & 0 deletions test/resources/ostree.resource
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
*** Settings ***
Documentation Keywords for OSTree-based systems

Library SSHLibrary
Library DataFormats.py


*** Variables ***
${BACKUP_STORAGE} /var/lib/microshift-backups
${HEALTH_FILE} /var/lib/microshift-backups/health.json


*** Keywords ***
Get Booted Deployment Status
[Documentation] Get rpm-ostree status of booted deployment as a json

${ostree_status} ${rc}= Execute Command
... rpm-ostree status --booted --json
... return_rc=True
Should Be Equal As Integers 0 ${rc}
${status}= Json Parse ${ostree_status}
RETURN ${status}

Get Booted Deployment ID
[Documentation] Get ID of currently booted deployment

${status}= Get Booted Deployment Status
RETURN ${status.deployments[0].id}

Get Booted Deployment Backup Prefix Path
[Documentation] Get backup path prefix for current deployment
...
... Prefix path is BACKUP_STORAGE/deployment-id.
... Globbing directories starting with the prefix will yield
... list of backups for the deployment.

${current_deployment_id}= Get Booted Deployment ID
Should Not Be Empty ${current_deployment_id}
Should Not Be Empty ${BACKUP_STORAGE}
${backup_path}= Catenate SEPARATOR=/ ${BACKUP_STORAGE} ${current_deployment_id}
RETURN ${backup_path}

Get System Health
[Documentation] Get system health information from health.json file

${health} ${stderr} ${rc}= Execute Command
... jq -r '.health' ${BACKUP_STORAGE}/health.json
Copy link
Copy Markdown
Contributor

@jogeo jogeo Jun 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'Evaluate json.loads' was used above in keyword "Get Booted Deployment Status". Should we be consistent in the pattern we use to ingest and parse json?

Copy link
Copy Markdown
Contributor

@jogeo jogeo Jun 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a side note, 'jq' is used by scripts such as microshift_set_healthy.sh, so it's not a bad thing that we are also using it in our test scripts. My initial comment was just a reaction to seeing json parsed in the same PR using two different approaches.

Copy link
Copy Markdown
Member Author

@pmtk pmtk Jun 12, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good point. jq here felt simpler, otherwise I'd have either cat and use json.loads or... I don't know what, I don't think SSHLibrary.Get File supports privilege escalation and health.json is only readable by root.
Also Get File scp's file, I'd rather use some kind of slurp which I haven't seen in that library.

Maybe we could create some helper keywords to slurping and modifying files as root and have flow consistent

Copy link
Copy Markdown
Contributor

@dhellmann dhellmann Jun 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the idea of keywords for grabbing files only readable by root. Save Default MicroShift Config uses cat to grab the contents of the config file, and it could use the same reusable keyword for that step. Then something like Upload String To File could be modified to take arguments for the owner and mode of the output file.

... sudo=True return_rc=True return_stderr=True
Should Be Equal As Integers 0 ${rc}
RETURN ${health}

Does Backup For Booted Deployment Exist
[Documentation] Checks if any backup for current deployment exists
... by finding paths starting with "backup prefix path"

${deploy_id}= Get Booted Deployment ID
${output} ${rc}= Execute Command
... find ${BACKUP_STORAGE} -name '${deploy_id}*' | wc -l
... sudo=True return_rc=True
Should Be Equal As Integers 0 ${rc}

IF "${output}" == "0" RETURN ${False}
RETURN ${True}
78 changes: 78 additions & 0 deletions test/suites-ostree/upgrade-ostree.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
*** Settings ***
Copy link
Copy Markdown
Member Author

@pmtk pmtk Jun 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file is in separate dir so e2e CI job doesn't try to run it against non-ostree system.
When we have e2e for ostree systems we can think if we want to move it back to suites/ and use tags (or other method to filter tests)

Documentation Tests related to MicroShift upgradeability on OSTree-based systems

Resource ../resources/common.resource
Resource ../resources/ostree.resource
Resource ../resources/systemd.resource
Resource ../resources/microshift-process.resource
Library Collections

Suite Setup Setup
Suite Teardown Teardown

Test Tags ostree


*** Variables ***
${USHIFT_HOST} ${EMPTY}
${USHIFT_USER} ${EMPTY}


*** Test Cases ***
Rebooting Healthy System Should Result In Data Backup
[Documentation] Check if rebooting healthy system will result in backing up of MicroShift data

Wait Until Greenboot Health Check Exited
System Should Be Healthy
Remove Existing Backup For Current Deployment

Reboot MicroShift Host
Wait For MicroShift Service

Backup For Booted Deployment Should Exist


*** Keywords ***
Setup
[Documentation] Test suite setup
Check Required Env Variables
Login MicroShift Host

Teardown
[Documentation] Test suite teardown
Logout MicroShift Host

Wait Until Greenboot Health Check Exited
[Documentation] Wait until greenboot healthchecks are done

Wait Until Keyword Succeeds 5m 15s
... Greenboot Health Check Exited

Greenboot Health Check Exited
[Documentation] Checks if greenboot-healthcheck finished running

${value}= Get Systemd Setting greenboot-healthcheck.service SubState
Should Be Equal As Strings ${value} exited

Remove Existing Backup For Current Deployment
[Documentation] Remove backups for current deployment

${path}= Get Booted Deployment Backup Prefix Path
Should Not Be Empty ${path}

Copy link
Copy Markdown
Contributor

@jogeo jogeo Jun 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we test path here with 'Should Not Be Empty'? I see that 'Get Booted Deployment Backup Prefix Path' has a check for empty values. However, if the code is changed to use a different key word at some point to get path we would not want to accidentally call 'rm -rf *'

${rm_output} ${rm_stderr} ${rc}= Execute Command
... rm -rf ${path}*
... sudo=True return_stderr=True return_rc=True
Log ${rm_stderr}
Log ${rm_output}
Should Be Equal As Integers 0 ${rc}

Backup For Booted Deployment Should Exist
[Documentation] Asserts that backup for currently booted deployment exists
${exists}= Does Backup For Booted Deployment Exist
Should Be True ${exists}

System Should Be Healthy
[Documentation] Asserts that persisted health information is "healthy"
${health}= Get System Health
Should Be Equal As Strings healthy ${health}
2 changes: 1 addition & 1 deletion test/suites/show-config.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Resource ../resources/common.resource
Resource ../resources/microshift-config.resource
Resource ../resources/microshift-host.resource
Resource ../resources/microshift-process.resource
Library ../resources/YAML.py
Library ../resources/DataFormats.py

Suite Setup Setup
Suite Teardown Teardown
Expand Down
2 changes: 1 addition & 1 deletion test/suites/version.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Resource ../resources/common.resource
Resource ../resources/oc.resource
Resource ../resources/microshift-process.resource
Library Collections
Library ../resources/YAML.py
Library ../resources/DataFormats.py

Suite Setup Setup
Suite Teardown Teardown
Expand Down