Unable to create wildcard certificate

[GhataEmbectaTest]

I am using cert-manager with my AKS cluster to generate Lets Encrypt certificates for my database using this webhook. Facing issue with certificate generation with error: secret not found in secret

Steps followed following the link: https://cert-manager.io/docs/configuration/acme/dns01/

1. Installed cert-manager using command: helm install cert-manager jetstack/cert-manager --version v1.9.1 --set installCRDs=true -n cert-manager –debug
2. Installed godaddy-webhook using the github repo: https://github.com/snowdrop/godaddy-webhook
   Command used: helm upgrade -i godaddy-webhook godaddy-webhook/godaddy-webhook --set groupName=acme.mydomain.com --set image.tag=v1.24.6 --set image.pullPolicy=Always --set pod.securePort=8443 --namespace cert-manager –debug
3. Created secret containing godaddy api key and secret. Deployed it in namespace : cert-manager.

apiVersion: v1
kind: Secret
metadata:
  name: godaddy-api-key
type: Opaque
stringData:
   token: <key>:<value>

4. Created a clusterIssuer apiVersion: cert-manager.io/v1

kind: ClusterIssuer                             
metadata:                               
  name: cmmx-cluster-issuer  
spec:                            
  acme:                                 
    server: https://acme-v02.api.letsencrypt.org/directory
    email: <email>
    privateKeySecretRef:                                                                   
      name: clusterissuer-test-key
    solvers:
    - selector:
        dnsZones:
          - "mydomain.com"
          - "*.mydomain.com"
      dns01:
        webhook:
          config:
            apiKeySecretRef:
              name: godaddy-api-key
              key: token
            ttl: 900
          groupName: acme.mydomain.com
          solverName: godaddy

Deployment is successful.
5. Created a certificate
apiVersion: cert-manager.io/v1

kind: Certificate
metadata:
  name: couchbasecertificate
spec:
  dnsNames:
    - "cmmx.mydomain.com"
  secretName: couchbase-server-tls
  commonName: "cmmx.mydomain.com"
  issuerRef:
    name: cmmx-cluster-issuer
    kind: ClusterIssuer

Issuing certificate as Secret does not exist”
Also attached cert-manager pod logs for more details.
cert-manager/challenges/Present "msg"="presenting DNS01 challenge for domain" "dnsName"="cmmx.mydomain.com" "domain"="cmmx.mydomain.com" "resource_kind"="Challenge" "resource_name"="couchbasecertificate-th2jp-71168170-2435339236" "resource_namespace"="cert-manager" "resource_version"="v1" "type"="DNS-01" 1 controller.go:166] cert-manager/challenges "msg"="re-queuing item due to error processing" "error"="secret not found in secret \"godaddy-api-key/cert-manager\"" "key"="cert-manager/couchbasecertificate-th2jp-71168170-2435339236"

dnsZones needs only mydomain.com. Remove the *.mydomain.com bit.

That's the selectors.

Now to request the wildcard certificate:

You can rename wildcard-yourdomain-com-tls to couchbase-server-tls and wildcard-yourdomain-com to couchbasecertificate, cmmx-cluster-issuer to wildcard-cmmx-cluster-issuer etc
You can add more names under the dnsNames here.
And make sure ingress-nginx namespace matches yours. I'm using ingress-nginx controller.

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-yourdomain-com
  namespace: ingress-nginx
spec:
  secretName: wildcard-yourdomain-com-tls
  issuerRef:
    name: cmmx-cluster-issuer
    kind: ClusterIssuer
  commonName: '*.yourdomain.com'
  dnsNames:
  - '*.yourdomain.com'
EOF

[cmoulliard]

I am using cert-manager with my AKS cluster to generate Lets Encrypt certificates for my database using this webhook. Facing issue with certificate generation with error: secret not found in secret

Steps followed following the link: https://cert-manager.io/docs/configuration/acme/dns01/

1. Installed cert-manager using command: helm install cert-manager jetstack/cert-manager --version v1.9.1 --set installCRDs=true -n cert-manager –debug
2. Installed godaddy-webhook using the github repo: https://github.com/snowdrop/godaddy-webhook
   Command used: helm upgrade -i godaddy-webhook godaddy-webhook/godaddy-webhook --set groupName=acme.mydomain.com --set image.tag=v1.24.6 --set image.pullPolicy=Always --set pod.securePort=8443 --namespace cert-manager –debug
3. Created secret containing godaddy api key and secret. Deployed it in namespace : cert-manager.

apiVersion: v1
kind: Secret
metadata:
  name: godaddy-api-key
type: Opaque
stringData:
   token: <key>:<value>

4. Created a clusterIssuer apiVersion: cert-manager.io/v1

kind: ClusterIssuer                             
metadata:                               
  name: cmmx-cluster-issuer  
spec:                            
  acme:                                 
    server: https://acme-v02.api.letsencrypt.org/directory
    email: <email>
    privateKeySecretRef:                                                                   
      name: clusterissuer-test-key
    solvers:
    - selector:
        dnsZones:
          - "mydomain.com"
          - "*.mydomain.com"
      dns01:
        webhook:
          config:
            apiKeySecretRef:
              name: godaddy-api-key
              key: token
            ttl: 900
          groupName: acme.mydomain.com
          solverName: godaddy

Deployment is successful. 5. Created a certificate apiVersion: cert-manager.io/v1

kind: Certificate
metadata:
  name: couchbasecertificate
spec:
  dnsNames:
    - "cmmx.mydomain.com"
  secretName: couchbase-server-tls
  commonName: "cmmx.mydomain.com"
  issuerRef:
    name: cmmx-cluster-issuer
    kind: ClusterIssuer

Issuing certificate as Secret does not exist” Also attached cert-manager pod logs for more details. cert-manager/challenges/Present "msg"="presenting DNS01 challenge for domain" "dnsName"="cmmx.mydomain.com" "domain"="cmmx.mydomain.com" "resource_kind"="Challenge" "resource_name"="couchbasecertificate-th2jp-71168170-2435339236" "resource_namespace"="cert-manager" "resource_version"="v1" "type"="DNS-01" 1 controller.go:166] cert-manager/challenges "msg"="re-queuing item due to error processing" "error"="secret not found in secret \"godaddy-api-key/cert-manager\"" "key"="cert-manager/couchbasecertificate-th2jp-71168170-2435339236"

Can you check the content of the challenge to see if there an error HTTP 401 = authentication issue please ? @GhataEmbectaTest

[ahmadzana]

I'm facing the same issue, the created challenge is in a pending state and produces this log.
Warning PresentError 2m9s (x9 over 23m) cert-manager-challenges Error presenting challenge: the server is currently unable to handle the request (post godaddy.acme.mycompany.com)

Does anyone know whats the problem?

[cmoulliard]

Does anyone know whats the problem?

Can you check the content of the challenge and paste here please (see: https://cert-manager.io/docs/troubleshooting/acme/#3-troubleshooting-challenges) ?

[ahmadzana]

Dear @cmoulliard, Sure i will provide full details.
The clusterIssuer

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ahmed.zana@rigt.com
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
    - selector:
        dnsNames:
        - '*.rigt.online'
      dns01:
        webhook:
          config:
            apiKeySecretRef:
              name: godaddy-api-key-prod
              key: key
              secret: secret
            production: true
            ttl: 600
          groupName: acme.rigt.online
          solverName: godaddy

Certificate

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-rigt-online
  namespace: default
spec:
  secretName: wildcard-yourdomain-com-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  commonName: '*.rigt.online'
  dnsNames:
  - '*.rigt.online'

Full content of the challenge

Name:         wildcard-rigt-online-1-177342043-3278775856
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  acme.cert-manager.io/v1
Kind:         Challenge
Metadata:
  Creation Timestamp:  2023-10-30T08:59:50Z
  Finalizers:
    finalizer.acme.cert-manager.io
  Generation:  1
  Owner References:
    API Version:           acme.cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Order
    Name:                  wildcard-rigt-online-1-177342043
    UID:                   1b611778-1bdc-476a-9326-750f482e6553
  Resource Version:        3575405
  UID:                     79db4b46-6c6c-45de-abf9-8f8fab39e335
Spec:
  Authorization URL:  https://acme-v02.api.letsencrypt.org/acme/authz-v3/278591822466
  Dns Name:           rigt.online
  Issuer Ref:
    Kind:  ClusterIssuer
    Name:  letsencrypt-prod
  Key:     xgJC0_InXJoGr8Vw80cs0gwMa41rnDEDl4d-CXCR0dc
  Solver:
    dns01:
      Webhook:
        Config:
          API Key Secret Ref:
            Key:       key
            Name:      godaddy-api-key-prod
            Secret:    secret
          Production:  true
          Ttl:         600
        Group Name:    acme.rigt.online
        Solver Name:   godaddy
    Selector:
      Dns Names:
        *.rigt.online
  Token:     wYknK7WtDe-ThJwCVmqGj9uGBj8I2c2QiCVZFT2cYmQ
  Type:      DNS-01
  URL:       https://acme-v02.api.letsencrypt.org/acme/chall-v3/278591822466/UH9rWw
  Wildcard:  true
Status:
  Presented:   false
  Processing:  true
  Reason:      the server is currently unable to handle the request (post godaddy.acme.rigt.online)
  State:       pending
Events:
  Type     Reason        Age              From                     Message
  ----     ------        ----             ----                     -------
  Normal   Started       8s               cert-manager-challenges  Challenge scheduled for processing
  Warning  PresentError  3s (x3 over 8s)  cert-manager-challenges  Error presenting challenge: the server is currently unable to handle the request (post godaddy.acme.rigt.online)

[cmoulliard]

Dear @cmoulliard, Sure i will provide full details.

Can you format your YAML files please using fenced code block https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks#fenced-code-blocks ?

[ahmadzana]

Dear @cmoulliard, Sure i will provide full details.

Can you format your YAML files please using fenced code block https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks#fenced-code-blocks ?

Sorry about that, I've edited my comment.

[cmoulliard]

I think that you should go trough the list of the resources created by the cert manager to see if another error has been reported as events (= kubectl describe ...) or status (= kubectl get ... -oyaml) :
certificate -> certificaterequest -> order -> challenge

[ahmadzana]

The certificate resource indicates that some secret is missing but I don't know what dose it means.

Name:         wildcard-rigt-online
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2023-10-30T08:59:48Z
  Generation:          1
  Resource Version:    3575373
  UID:                 4d7c84f1-4f42-4bc0-b8d8-3a35569d7ae7
Spec:
  Common Name:  *.rigt.online
  Dns Names:
    *.rigt.online
  Issuer Ref:
    Kind:       ClusterIssuer
    Name:       letsencrypt-prod
  Secret Name:  wildcard-yourdomain-com-tls
Status:
  Conditions:
    Last Transition Time:        2023-10-30T08:59:48Z
    Message:                     Issuing certificate as Secret does not exist
    Observed Generation:         1
    Reason:                      DoesNotExist
    Status:                      False
    Type:                        Ready
    Last Transition Time:        2023-10-30T08:59:48Z
    Message:                     Issuing certificate as Secret does not exist
    Observed Generation:         1
    Reason:                      DoesNotExist
    Status:                      True
    Type:                        Issuing
  Next Private Key Secret Name:  wildcard-rigt-online-ttlb5
Events:                          <none>

Order resource

Name:         wildcard-rigt-online-1-177342043
Namespace:    default
Labels:       <none>
Annotations:  cert-manager.io/certificate-name: wildcard-rigt-online
              cert-manager.io/certificate-revision: 1
              cert-manager.io/private-key-secret-name: wildcard-rigt-online-ttlb5
API Version:  acme.cert-manager.io/v1
Kind:         Order
Metadata:
  Creation Timestamp:  2023-10-30T08:59:48Z
  Generation:          1
  Owner References:
    API Version:           cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  CertificateRequest
    Name:                  wildcard-rigt-online-1
    UID:                   ad2a21f0-7a88-4d62-ae35-1e6874121b8f
  Resource Version:        3575396
  UID:                     1b611778-1bdc-476a-9326-750f482e6553
Spec:
  Common Name:  *.rigt.online
  Dns Names:
    *.rigt.online
  Issuer Ref:
    Kind:   ClusterIssuer
    Name:   letsencrypt-prod
  Request:  LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ2xUQ0NBWDBDQVFBd0dERVdNQlFHQTFVRUF3d05LaTV5YVdkMExtOXViR2x1WlRDQ0FTSXdEUVlKS29aSQpodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU1CbTRqWm1pdExieWFxSHBGektWY2VNR0I2NU9ZQXhrZVhCCk9rdndHSkpBY3VPSlhFZHJXZXp1Y1ZlTmQ1eFVlMEYvdlY0b200WW9QK3FaUklpSU43TnhmOWliS1JJeGR6VGwKeitOWEtZSnpXbTh0dHY1elAweEN3bGs4RzlwUVM1ZU92RG1YMUgxdHV6RFBUV3YzbDdPVnpSU2pIa2NXRjAvNwplVEFHdFM0MWpKSFR1VnUvZkFHRlR1K1RuVVkvcnZtVDB4aDk0VlpjK2FpWm1jUFM0ZW9wNUtNYVRhUkdVR1NyCnBDWnJNK2pKenc2bmtKMFRobjNYUW52aEplRFZ6MktDdzRKSGptbnpPL0FQMUtnR2ppSll0N1VUellpU3pPY2wKNlh3YmxGclgzaG5xaUpqenJIMFMydHNtQkZDQW93WVVOLytzVnBjOVNHUW0wUXFQUU0wQ0F3RUFBYUE0TURZRwpDU3FHU0liM0RRRUpEakVwTUNjd0dBWURWUjBSQkJFd0Q0SU5LaTV5YVdkMExtOXViR2x1WlRBTEJnTlZIUThFCkJBTUNCYUF3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUtTVXZVcldQTTRkZjZtSklpSzJvM1NhQXFwVlk4eDQKd3hSdTFKMzNrdmRjYythb2FCbXBMd05uUy91NDkrWWZzR1NSWlhCbjVIa2poUDFLak9FM000c2FqTjhLV25sQwpJMEl4ZXZaR0ZWSit4SkVHVnRCS25FVVJEajVwSElSM0VRRGZkdEJSdDRieDFlQmUzNTE2TjJkUEp3eXV3aXpRCnNscG1maDJMOEt1a2hINUJibzBYUEVnaVpjd0lCdGdJbndCRXNnSVNxRS9GNHpkQTlNdEQ3ZEJWdDZUbDBGTU8KcXU0Y0NIOHJJTlBCMXoxdjA1eXJCcjFwWlh1WFNrYWNvV3dKV0JSUVNweVRhMWtIYmxkdmRSQnZ0Wjl5RzN2NgpDT3owMWJIZ3BiZS9PMUo3MUhvc3pscStHUUdlcm0wamZMdis2a09RdjQ1M1dpZ0xobTRCOGg4PQotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K
Status:
  Authorizations:
    Challenges:
      Token:        wYknK7WtDe-ThJwCVmqGj9uGBj8I2c2QiCVZFT2cYmQ
      Type:         dns-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/278591822466/UH9rWw
    Identifier:     rigt.online
    Initial State:  pending
    URL:            https://acme-v02.api.letsencrypt.org/acme/authz-v3/278591822466
    Wildcard:       true
  Finalize URL:     https://acme-v02.api.letsencrypt.org/acme/finalize/1385199376/218649226056
  State:            pending
  URL:              https://acme-v02.api.letsencrypt.org/acme/order/1385199376/218649226056
Events:             <none>

Also I'm running my kubeadm cluster under a strict network firewall. dose that have anything to do with that ? do I need to open any ports?
Also, I'm very thankful for your responses.

[cmoulliard]

The certificate resource indicates that some secret is missing but I don't know what dose it means.

This message happens (if I dont say something wrong) when no secret containing the TLS cert + key has been yet generated by Letsencrypt from DNS challenge response (= that it got from godaddy) and what finally triggers the creation of the CertificateRequest CR

[cmoulliard]

dose that have anything to do with that ? do I need to open any ports?

Maybe if HTTP traffic is blocked between your VM running the kube cluster and godaddy. Can you check on godaddy web site if a TXT record containing the ACME challenge for your domain name (see discussion https://stackoverflow.com/questions/63346728/issuing-certificate-as-secret-does-not-exist) exists ?

[ahmadzana]

I've tested the entire process across various environments, including a kind cluster and DigitalOcean Managed Kubernetes. I've also experimented with different domains, and each scenario has resulted in the same error message:

Warning  PresentError  32s (x2 over 32s)  cert-manager-challenges  Error presenting challenge: the server could not find the requested resource (post godaddy.acme.smsmtrivia.com)

It's evident that this issue is rooted in the webhook plugin, and I believe it's crucial to open a new issue to address and resolve this problem.

[cmoulliard]

Can you check on godaddy web site if a TXT record containing the ACME challenge for your domain name (see discussion https://stackoverflow.com/questions/63346728/issuing-certificate-as-secret-does-not-exist) exists ?

Can you check the following points please:

• Is there an ACME TXT record created on godaddy for the domain name tested ?
• Which log messages are reported by the wehbook. Here is the code used to call godaddy where you can see what we log: https://github.com/snowdrop/godaddy-webhook/blob/main/main.go#L340
• Can you issue curl commands using godaddy API from the machine running your kube cluster (https://github.com/snowdrop/godaddy-webhook/blob/main/main.go#L225) ?

[cmoulliard]

FYI: I use cert manager v1.12 + this godaddy webhook on a VPN (= behind a firewall) and my certificates are well rotated. Here is the last order issued some hours ago

[snowdrop@snowdrop-k8s ~]$ k get order/halkyon-io-sjps7-4051198276 -n halkyon-site
NAME                          STATE   AGE
halkyon-io-sjps7-4051198276   valid   17h

[snowdrop@snowdrop-k8s ~]$ k get order/halkyon-io-sjps7-4051198276 -n halkyon-site -oyaml
apiVersion: acme.cert-manager.io/v1
kind: Order
metadata:
  annotations:
    cert-manager.io/certificate-name: halkyon-io
    cert-manager.io/certificate-revision: "28"
    cert-manager.io/private-key-secret-name: halkyon-io-hpn5n
  creationTimestamp: "2023-10-30T14:15:59Z"
  generation: 1
  labels:
    app: ca-cert
  name: halkyon-io-sjps7-4051198276
  namespace: halkyon-site
  ownerReferences:
  - apiVersion: cert-manager.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: CertificateRequest
    name: halkyon-io-sjps7
    uid: 64851d67-dd98-479a-b0af-a09a5a06f449
  resourceVersion: "12666589"
  uid: 8752fe02-6c2d-49cc-aeea-ecb6e7e557ed
spec:
  dnsNames:
  - halkyon.io
  - www.halkyon.io
  issuerRef:
    kind: Issuer
    name: letsencrypt-prod-halkyon-io
  request: LS0tLS1CRUdJTiBD...QVRFIFJFUVVFU1QtLS0tLQo=
status:
  authorizations:
  - challenges:
    - token: zYEpvjAQ...Fvyht2iA
      type: dns-01
      url: https://acme-v02.api.letsencrypt.org/acme/chall-v3/275290227926/ZcQjtg
    identifier: halkyon.io
    initialState: valid
    url: https://acme-v02.api.letsencrypt.org/acme/authz-v3/275290227926
    wildcard: false
  - challenges:
    - token: CGT...EhK7w
      type: dns-01
      url: https://acme-v02.api.letsencrypt.org/acme/chall-v3/275290227936/mq7Vkw
    identifier: www.halkyon.io
    initialState: valid
    url: https://acme-v02.api.letsencrypt.org/acme/authz-v3/275290227936
    wildcard: false
  certificate: LS0tLS....