Bug: Velero pod not running after sending Velero CR to OADP operator

[Frank51]

Contact Details

yunqian.chen@dell.com

Describe bug

A OADP-Operator 0.3.0 was deployed on the openshift cluster. And then a Velero CR was sent to OADP-Operator via 'oc apply -f ', but velero pod is not running on the namespace.

I also tried sending this json file via Openshift UI to oadp-operator, but it shows that the 'spec.backupStorageLocations' is 'Required value'.

Here is the Velero CR json file I applied.

{
"kind": "Velero",
"apiVersion": "konveyor.openshift.io/v1alpha1",
"metadata": {
"name": "oadp-velero",
"namespace": "velero-ppdm",
"creationTimestamp": null
},
"spec": {
"custom_velero_plugins": [{
"image": "velero-dd-pluin-uri",
"name": "powerprotect-velero-dd"
}],
"default_velero_plugins": ["openshift"],
"enable_restic": false,
"olm_managed": false,
"velero_feature_flags": "EnableLocalMode"
}
}

What happened?

The velero pod should running on the namespace. But the no velero pod showing on the namespace.

OADP Version

0.3.x (Beta)

OpenShift Version

4.6

Velero pod logs

Velero pod is not running.

Restic pod logs

No response

Operator pod logs

I1006 21:38:33.529251       1 request.go:665] Waited for 1.029013433s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/autoscaling.openshift.io/v1beta1?timeout=32s
2021-10-06T21:38:35.343Z	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": ":8080"}
2021-10-06T21:38:35.345Z	INFO	setup	starting manager
I1006 21:38:35.346433       1 leaderelection.go:248] attempting to acquire leader lease velero-ppdm/8b4defce.openshift.io...
2021-10-06T21:38:35.347Z	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
I1006 21:38:51.546526       1 leaderelection.go:258] successfully acquired lease velero-ppdm/8b4defce.openshift.io
2021-10-06T21:38:51.546Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"ConfigMap","namespace":"velero-ppdm","name":"8b4defce.openshift.io","uid":"b4db8fa0-ad24-4fd3-8767-47c4d6e5b673","apiVersion":"v1","resourceVersion":"83482205"}, "reason": "LeaderElection", "message": "oadp-operator-5f4f487447-kwwzl_562ab938-0160-4d16-9074-c0249a3b6e44 became leader"}
2021-10-06T21:38:51.547Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"Lease","namespace":"velero-ppdm","name":"8b4defce.openshift.io","uid":"b3d57fde-ee3c-42cb-9bad-d27c73865506","apiVersion":"coordination.k8s.io/v1","resourceVersion":"83482206"}, "reason": "LeaderElection", "message": "oadp-operator-5f4f487447-kwwzl_562ab938-0160-4d16-9074-c0249a3b6e44 became leader"}
2021-10-06T21:38:51.547Z	INFO	controller-runtime.manager.controller.velero	Starting EventSource	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "source": "kind source: /, Kind="}
2021-10-06T21:38:51.547Z	INFO	controller-runtime.manager.controller.velero	Starting EventSource	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "source": "kind source: /, Kind="}
2021-10-06T21:38:51.547Z	INFO	controller-runtime.manager.controller.velero	Starting EventSource	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "source": "kind source: /, Kind="}
2021-10-06T21:38:51.547Z	INFO	controller-runtime.manager.controller.velero	Starting EventSource	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "source": "kind source: /, Kind="}
2021-10-06T21:38:51.547Z	INFO	controller-runtime.manager.controller.velero	Starting EventSource	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "source": "kind source: /, Kind="}
2021-10-06T21:38:51.547Z	INFO	controller-runtime.manager.controller.velero	Starting EventSource	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "source": "kind source: /, Kind="}
2021-10-06T21:38:51.547Z	INFO	controller-runtime.manager.controller.velero	Starting EventSource	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "source": "kind source: /, Kind="}
2021-10-06T21:38:51.547Z	INFO	controller-runtime.manager.controller.velero	Starting EventSource	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "source": "kind source: /, Kind="}
2021-10-06T21:38:51.547Z	INFO	controller-runtime.manager.controller.velero	Starting EventSource	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "source": "kind source: /, Kind="}
2021-10-06T21:38:51.548Z	INFO	controller-runtime.manager.controller.velero	Starting EventSource	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "source": "kind source: /, Kind="}
2021-10-06T21:38:51.548Z	INFO	controller-runtime.manager.controller.velero	Starting Controller	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero"}
2021-10-06T21:38:59.105Z	INFO	controller-runtime.manager.controller.velero	Starting workers	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "worker count": 1}
2021-10-06T22:11:44.353Z	INFO	controller-runtime.manager.controller.velero	aws backupstoragelocation is configured but velero plugin for aws is not present	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:11:44.354Z	INFO	controller-runtime.manager.controller.velero	error validating aws provider secret:  velero-ppdm/cloud-credentials	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:11:44.355Z	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"ConfigMap","namespace":"velero-ppdm","name":"restic-restore-action-config","uid":"8a2a9f93-fe22-4113-8533-7f875af96ba6","apiVersion":"v1","resourceVersion":"83497039"}, "reason": "ReconcileResticRestoreHelperConfigReconciled", "message": "performed created on restic restore Helper config map velero-ppdm/restic-restore-action-config"}
2021-10-06T22:11:44.374Z	ERROR	controller-runtime.manager.controller.velero	Reconciler error	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm", "error": "Secret \"cloud-credentials\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:214
2021-10-06T22:11:44.380Z	INFO	controller-runtime.manager.controller.velero	aws backupstoragelocation is configured but velero plugin for aws is not present	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:11:44.381Z	INFO	controller-runtime.manager.controller.velero	error validating aws provider secret:  velero-ppdm/cloud-credentials	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:11:44.399Z	ERROR	controller-runtime.manager.controller.velero	Reconciler error	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm", "error": "Secret \"cloud-credentials\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:214
2021-10-06T22:11:44.411Z	INFO	controller-runtime.manager.controller.velero	aws backupstoragelocation is configured but velero plugin for aws is not present	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:11:44.411Z	INFO	controller-runtime.manager.controller.velero	error validating aws provider secret:  velero-ppdm/cloud-credentials	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:11:44.427Z	ERROR	controller-runtime.manager.controller.velero	Reconciler error	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm", "error": "Secret \"cloud-credentials\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:214
2021-10-06T22:11:44.448Z	INFO	controller-runtime.manager.controller.velero	aws backupstoragelocation is configured but velero plugin for aws is not present	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:11:44.448Z	INFO	controller-runtime.manager.controller.velero	error validating aws provider secret:  velero-ppdm/cloud-credentials	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:11:44.469Z	ERROR	controller-runtime.manager.controller.velero	Reconciler error	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm", "error": "Secret \"cloud-credentials\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:214
2021-10-06T22:11:44.510Z	INFO	controller-runtime.manager.controller.velero	aws backupstoragelocation is configured but velero plugin for aws is not present	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:11:44.510Z	INFO	controller-runtime.manager.controller.velero	error validating aws provider secret:  velero-ppdm/cloud-credentials	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:11:44.529Z	ERROR	controller-runtime.manager.controller.velero	Reconciler error	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm", "error": "Secret \"cloud-credentials\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:214
2021-10-06T22:11:44.610Z	INFO	controller-runtime.manager.controller.velero	aws backupstoragelocation is configured but velero plugin for aws is not present	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:11:44.610Z	INFO	controller-runtime.manager.controller.velero	error validating aws provider secret:  velero-ppdm/cloud-credentials	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:11:44.629Z	ERROR	controller-runtime.manager.controller.velero	Reconciler error	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm", "error": "Secret \"cloud-credentials\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:214
2021-10-06T22:11:44.790Z	INFO	controller-runtime.manager.controller.velero	aws backupstoragelocation is configured but velero plugin for aws is not present	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:11:44.790Z	INFO	controller-runtime.manager.controller.velero	error validating aws provider secret:  velero-ppdm/cloud-credentials	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:11:44.806Z	ERROR	controller-runtime.manager.controller.velero	Reconciler error	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm", "error": "Secret \"cloud-credentials\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:214
2021-10-06T22:11:45.127Z	INFO	controller-runtime.manager.controller.velero	aws backupstoragelocation is configured but velero plugin for aws is not present	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:11:45.127Z	INFO	controller-runtime.manager.controller.velero	error validating aws provider secret:  velero-ppdm/cloud-credentials	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:11:45.142Z	ERROR	controller-runtime.manager.controller.velero	Reconciler error	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm", "error": "Secret \"cloud-credentials\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:214
2021-10-06T22:11:45.783Z	INFO	controller-runtime.manager.controller.velero	aws backupstoragelocation is configured but velero plugin for aws is not present	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:11:45.783Z	INFO	controller-runtime.manager.controller.velero	error validating aws provider secret:  velero-ppdm/cloud-credentials	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:11:45.802Z	ERROR	controller-runtime.manager.controller.velero	Reconciler error	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm", "error": "Secret \"cloud-credentials\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:214
2021-10-06T22:11:47.083Z	INFO	controller-runtime.manager.controller.velero	aws backupstoragelocation is configured but velero plugin for aws is not present	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:11:47.083Z	INFO	controller-runtime.manager.controller.velero	error validating aws provider secret:  velero-ppdm/cloud-credentials	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:11:47.111Z	ERROR	controller-runtime.manager.controller.velero	Reconciler error	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm", "error": "Secret \"cloud-credentials\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:214
2021-10-06T22:11:49.673Z	INFO	controller-runtime.manager.controller.velero	aws backupstoragelocation is configured but velero plugin for aws is not present	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:11:49.673Z	INFO	controller-runtime.manager.controller.velero	error validating aws provider secret:  velero-ppdm/cloud-credentials	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:11:49.696Z	ERROR	controller-runtime.manager.controller.velero	Reconciler error	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm", "error": "Secret \"cloud-credentials\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:214
2021-10-06T22:11:54.818Z	INFO	controller-runtime.manager.controller.velero	aws backupstoragelocation is configured but velero plugin for aws is not present	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:11:54.818Z	INFO	controller-runtime.manager.controller.velero	error validating aws provider secret:  velero-ppdm/cloud-credentials	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:11:54.911Z	ERROR	controller-runtime.manager.controller.velero	Reconciler error	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm", "error": "Secret \"cloud-credentials\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:214
2021-10-06T22:12:05.152Z	INFO	controller-runtime.manager.controller.velero	aws backupstoragelocation is configured but velero plugin for aws is not present	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:12:05.153Z	INFO	controller-runtime.manager.controller.velero	error validating aws provider secret:  velero-ppdm/cloud-credentials	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:12:05.164Z	ERROR	controller-runtime.manager.controller.velero	Reconciler error	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm", "error": "Secret \"cloud-credentials\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:214
2021-10-06T22:12:25.645Z	INFO	controller-runtime.manager.controller.velero	aws backupstoragelocation is configured but velero plugin for aws is not present	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:12:25.645Z	INFO	controller-runtime.manager.controller.velero	error validating aws provider secret:  velero-ppdm/cloud-credentials	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:12:25.659Z	ERROR	controller-runtime.manager.controller.velero	Reconciler error	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm", "error": "Secret \"cloud-credentials\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:214
2021-10-06T22:13:06.621Z	INFO	controller-runtime.manager.controller.velero	aws backupstoragelocation is configured but velero plugin for aws is not present	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:13:06.621Z	INFO	controller-runtime.manager.controller.velero	error validating aws provider secret:  velero-ppdm/cloud-credentials	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:13:06.637Z	ERROR	controller-runtime.manager.controller.velero	Reconciler error	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm", "error": "Secret \"cloud-credentials\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:214
2021-10-06T22:14:28.558Z	INFO	controller-runtime.manager.controller.velero	aws backupstoragelocation is configured but velero plugin for aws is not present	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:14:28.559Z	INFO	controller-runtime.manager.controller.velero	error validating aws provider secret:  velero-ppdm/cloud-credentials	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:14:28.581Z	ERROR	controller-runtime.manager.controller.velero	Reconciler error	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm", "error": "Secret \"cloud-credentials\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:214
2021-10-06T22:17:12.423Z	INFO	controller-runtime.manager.controller.velero	aws backupstoragelocation is configured but velero plugin for aws is not present	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:17:12.423Z	INFO	controller-runtime.manager.controller.velero	error validating aws provider secret:  velero-ppdm/cloud-credentials	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:17:12.443Z	ERROR	controller-runtime.manager.controller.velero	Reconciler error	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm", "error": "Secret \"cloud-credentials\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:214
2021-10-06T22:22:40.124Z	INFO	controller-runtime.manager.controller.velero	aws backupstoragelocation is configured but velero plugin for aws is not present	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:22:40.125Z	INFO	controller-runtime.manager.controller.velero	error validating aws provider secret:  velero-ppdm/cloud-credentials	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:22:40.138Z	ERROR	controller-runtime.manager.controller.velero	Reconciler error	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm", "error": "Secret \"cloud-credentials\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:214
2021-10-06T22:33:35.499Z	INFO	controller-runtime.manager.controller.velero	aws backupstoragelocation is configured but velero plugin for aws is not present	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:33:35.500Z	INFO	controller-runtime.manager.controller.velero	error validating aws provider secret:  velero-ppdm/cloud-credentials	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}
2021-10-06T22:33:35.514Z	ERROR	controller-runtime.manager.controller.velero	Reconciler error	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm", "error": "Secret \"cloud-credentials\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:214
2021-10-06T22:44:37.736Z	ERROR	controller-runtime.manager.controller.velero	Reconciler error	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm", "error": "no valid provider configured for one of the backupstoragelocations"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:214
2021-10-06T22:50:15.532Z	ERROR	controller-runtime.manager.controller.velero	Reconciler error	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm", "error": "no valid provider configured for one of the backupstoragelocations"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:214
2021-10-06T23:06:55.550Z	ERROR	controller-runtime.manager.controller.velero	Reconciler error	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm", "error": "no valid provider configured for one of the backupstoragelocations"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:214
2021-10-06T23:23:35.572Z	ERROR	controller-runtime.manager.controller.velero	Reconciler error	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm", "error": "no valid provider configured for one of the backupstoragelocations"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:214
2021-10-06T23:40:15.587Z	ERROR	controller-runtime.manager.controller.velero	Reconciler error	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm", "error": "no valid provider configured for one of the backupstoragelocations"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:214
2021-10-06T23:56:55.610Z	ERROR	controller-runtime.manager.controller.velero	Reconciler error	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm", "error": "no valid provider configured for one of the backupstoragelocations"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:214
2021-10-06T23:58:06.217Z	ERROR	controller-runtime.manager.controller.velero	unable to fetch velero CR	{"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm", "velero": "velero-ppdm/oadp-velero", "error": "Velero.oadp.openshift.io \"oadp-velero\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:298
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.5/pkg/internal/controller/controller.go:214

New issue

• This issue is new

[kaovilai]

This is not a bug. There is an intentional change in the CR definition.
Please review https://github.com/openshift/oadp-operator/blob/master/docs/upgrade.md
Closing this issue. Let us know if you need further assistance.

[kaovilai]

You may also find https://github.com/openshift/oadp-operator/blob/master/docs/API_ref.md and
https://pkg.go.dev/github.com/openshift/oadp-operator@v0.3.0/api/v1alpha1#VeleroSpec helpful.

[kaovilai]

The main issue I see here is Velero CR json containing old definitions. Ensure you review the api references above.

Other than that..
Looking at the operator pod logs there are 2 repeating issues. I have outlined the solution to them below if you encounter them again.

2021-10-06T22:11:44.610Z INFO controller-runtime.manager.controller.velero aws backupstoragelocation is configured but velero plugin for aws is not present {"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}

We now require you to specify defaultVeleroPlugins for your storage provider plugins such as aws if those are defined in the BackupStorageLocation

2021-10-06T22:11:44.610Z INFO controller-runtime.manager.controller.velero error validating aws provider secret: velero-ppdm/cloud-credentials {"reconciler group": "oadp.openshift.io", "reconciler kind": "Velero", "name": "oadp-velero", "namespace": "velero-ppdm"}

Provider secrets cloud credentials should be defined prior to Velero CR creation inside the same namespace as the operator. See how to create cloud credentials secret

[Frank51]

Hi, does the "backupStorageLocations" right now is a required field in Velero CR?

[kaovilai]

Hi, does the "backupStorageLocations" right now is a required field in Velero CR?

See Velero Docs - Velero Backup Storage Locations

Velero must have at least one BackupStorageLocation.

[kaovilai]

@Frank51 track your customize install usecase in #379