5-25- OPS-User-Grant Nightly Test Failure: Failure to install VCH

[sgairo]

2/1 vic engine nightly test failure on 6.0:
Suites.Group5-Functional-Tests.5-25-OPS-User-Grant.zip

time="2018-02-01T22:34:30Z" level=debug msg="VCH DNS servers: []" 
time="2018-02-01T22:34:30Z" level=debug msg="Unable to locate existing server certificate in cert path" 
time="2018-02-01T22:34:30Z" level=info msg="Generating CA certificate/key pair - private key in VCH-0-2630/ca-key.pem" 
time="2018-02-01T22:34:30Z" level=info msg="Generating server certificate/key pair - private key in VCH-0-2630/server-key.pem" 
time="2018-02-01T22:34:31Z" level=info msg="Generating client certificate/key pair - private key in VCH-0-2630/key.pem" 
time="2018-02-01T22:34:31Z" level=info msg="Generated browser friendly PFX client certificate - certificate in VCH-0-2630/cert.pfx" 
time="2018-02-01T22:34:31Z" level=debug msg="--target=https://10.160.194.207/datacenter1" 
    [ Message content over the limit has been removed. ]
time="2018-02-01T22:34:34Z" level=info msg="DRS check OK on:" 
time="2018-02-01T22:34:34Z" level=info msg="  \"/datacenter1/host/cls1\"" 
Feb  1 2018 22:34:34.200Z DEBUG URL: https://harbor.ci.drone.local/v2/
Feb  1 2018 22:34:34.333Z DEBUG URL: http://harbor.ci.drone.local/v2/
time="2018-02-01T22:34:34Z" level=warning msg="Unable to confirm insecure registry harbor.ci.drone.local is a valid registry at this time." 
time="2018-02-01T22:34:34Z" level=info msg="Insecure registries = harbor.ci.drone.local" 
time="2018-02-01T22:34:34Z" level=warning msg="Only one host can access all of the image/container/volume datastores. This may be a point of contention/performance degradation and HA/DRS may not work as intended." 
time="2018-02-01T22:34:34Z" level=error msg=-------------------- 
time="2018-02-01T22:34:34Z" level=error msg="Failed to validate operations credentials: ServerFaultCode: The operation is not supported on the object." 
time="2018-02-01T22:34:34Z" level=error msg="Create cannot continue: configuration validation failed" 
time="2018-02-01T22:34:34Z" level=error msg=-------------------- 
time="2018-02-01T22:34:34Z" level=error msg="vic-machine-linux create failed: validation of configuration failed\n" ' does not contain 'Installer completed successfully'

[sgairo]

Seen in 2/2 and 2/3 and 2/4 nightly test 6.0 as well.

[sgairo]

Seen in 2/5 nightly test for 6.0

[lcastellano]

This issue is intermittent. I have run the test myself in a loop and had not been able to reproduce it. However, the logs point to a failure of VC/PSC to correctly respond to a "user" search in the PSC user database. In order to narrow down the cause, we need VC logs and possibly the PCS logs as well. The code in this area has been modified recently and the timing has changed. In particular this method used to be called after the VCH had been created. In the new code base it is called earlier (during validation) and that could imply that it is called when the PSC is not fully initialized. PSC initialization has progressed further enough to create the user but possibly not further enough to allow vSphere to place a search call onto the PSC groups (the call that makes the test fail). I will take a look at the CI runs before and after this change was submitted to confirm this hypothesis.

[sgairo]

@lcastellano When you ran it yourself was it on 6.0 or 6.5? It has failed every single 6.0 run so far in the nightlies.

[lcastellano]

It looks like the method VIC is using to check that the ops-user is not part of the Administrator group is not supported in 6.0. The API is there in the mob browser but it always returns not supported. This test was added to the Nightly only recently (beginning of Feb) and it never worked on 6.0, while it always works on 6.5.

[sgairo]

I believe we saw this issue in the 5-3 ELM test last night as well:
Suites.Group5-Functional-Tests.5-3-Enhanced-Linked-Mode.zip

[AngieCris]

Seen again on nightly 02/14/18, and also 02/16/18, 02/17/18, 02/18/18, 02/19/18
VC version: 6.0
5-25-OPS-Grant

vic-machine.log:

Feb 15 2018 05:10:01.897Z WARN  op=10943.1: Unable to confirm insecure registry harbor.ci.drone.local is a valid registry at this time.
Feb 15 2018 05:10:01.897Z INFO  op=10943.1: Insecure registries = harbor.ci.drone.local
Feb 15 2018 05:10:01.897Z DEBUG [ END ] op=10943.1 [vic/lib/install/validate.(*Validator).registries:583] [265.966606ms] 
Feb 15 2018 05:10:01.897Z DEBUG [BEGIN] op=10943.1 [vic/lib/install/validate.(*Validator).compatibility:736]
Feb 15 2018 05:10:02.016Z DEBUG [BEGIN] op=10943.1 [vic/lib/install/validate.(*Validator).checkDatastoresAreWriteable:791]
Feb 15 2018 05:10:02.285Z WARN  op=10943.1: Only one host can access all of the image/container/volume datastores. This may be a point of contention/performance degradation and HA/DRS may not work as intended.
Feb 15 2018 05:10:02.286Z DEBUG [ END ] op=10943.1 [vic/lib/install/validate.(*Validator).checkDatastoresAreWriteable:791] [269.828863ms] 
Feb 15 2018 05:10:02.286Z DEBUG [ END ] op=10943.1 [vic/lib/install/validate.(*Validator).compatibility:736] [388.466911ms] 
Feb 15 2018 05:10:02.286Z DEBUG [BEGIN] op=10943.1 [vic/lib/install/validate.(*Validator).syslog:903]
Feb 15 2018 05:10:02.286Z DEBUG [ END ] op=10943.1 [vic/lib/install/validate.(*Validator).syslog:903] [14.337µs] 
Feb 15 2018 05:10:02.286Z DEBUG [BEGIN] op=10943.1 [vic/lib/install/validate.(*Validator).ListIssues:273]
Feb 15 2018 05:10:02.286Z ERROR op=10943.1: --------------------
Feb 15 2018 05:10:02.286Z ERROR op=10943.1: Failed to validate operations credentials: ServerFaultCode: The operation is not supported on the object.
Feb 15 2018 05:10:02.286Z DEBUG [ END ] op=10943.1 [vic/lib/install/validate.(*Validator).ListIssues:273] [112.186µs] 
Feb 15 2018 05:10:02.286Z DEBUG [ END ] op=10943.1 [vic/lib/install/validate.(*Validator).Validate:299] [2.388685302s] 
Feb 15 2018 05:10:02.286Z ERROR op=10943.1: Create cannot continue: configuration validation failed
Feb 15 2018 05:10:02.319Z ERROR op=10943.1: --------------------
Feb 15 2018 05:10:02.319Z ERROR op=10943.1: vic-machine-linux create failed: validation of configuration failed

Complete vic-machine.log:
vic-machine.log
vic-machine_02:16.log
(search for VCH-0-7691)
nightly test output:
Suites.Group5-Functional-Tests.5-25-OPS-User-Grant.zip
Suites.Group5-Functional-Tests.5-25-OPS-User-Grant_02:16.zip