challenge is failed: urn:ietf:params:acme:error:dns: DNS problem: NXDOMAIN

[davidt-gh]

I was adding my first ingress which was x.domain.com, and the webhook went well.
few minutes later I've added x1.domain.com, and got error in the challenge says:

Accepting challenge authorization failed: acme: authorization error for x1.domain.com: 400 urn:ietf:params:acme:error:dns: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.x1.domain.com - check that a DNS record exists for this domain

The TXT record seem to be created by godaddy webhook, and deleted after 750ms, once it deleted the error (found in the challenge events) the error showed up.
Can we extend the deletion time? anyone familiar with this issue?

Edit: Anyone in here now how to extend the delay? I want to try this by the answer I got: https://community.letsencrypt.org/t/challenge-failed-and-got-error-dns-problem-nxdomain-looking-up-txt/225975/4?u=david-gh

@cmoulliard maybe you can help?

[cmoulliard]

Edit: Anyone in here now how to extend the delay? I want to try this by the answer I got: https://community.letsencrypt.org/t/challenge-failed-and-got-error-dns-problem-nxdomain-looking-up-txt/225975/4?u=david-gh

If I understand well the issue, this plugin should include an additional parameter in order to add an "extra sleep after adding the TXT record.". I will check the code to see if this doable. You could maybe also propose a PR ;-)

[davidt-gh]

Edit: Anyone in here now how to extend the delay? I want to try this by the answer I got: https://community.letsencrypt.org/t/challenge-failed-and-got-error-dns-problem-nxdomain-looking-up-txt/225975/4?u=david-gh

If I understand well the issue, this plugin should include an additional parameter in order to add an "extra sleep after adding the TXT record.". I will check the code to see if this doable. You could maybe also propose a PR ;-)

If you can point me to the relevant place in code, I'll do it. But for now I saw:

godaddy-webhook/main_test.go

Line 20 in ebac646

timeOut, _ := time.ParseDuration("3m")

But this is for tests. Hope you can point me and I'll work on it.

[cmoulliard]

This is where within the code we call GoDaddy to check if the TXT record exists: https://github.com/snowdrop/godaddy-webhook/blob/main/main.go#L226

[jonsbun]

This has become an annoying problem for the last half of the year. However, I am unsure if this is directly related to godaddy-webhook (v0.5.0) or cert-manager (v1.16.2).

godaddy-webhook logs:

I0116 06:55:27.179888       1 trace.go:236] Trace[1937146269]: "Create" accept:application/json, */*,audit-id:f8d5a73e-1e17-4d8b-8f5a-a41f320ffb38,client:10.10.160.153,api-group:acme.domain.com,api-version:v1alpha1,name:,subresource:,namespace:,protocol:HTTP/2.0,resource:godaddy,scope:resource,url:/apis/acme.domain.com/v1alpha1/godaddy,user-agent:cert-manager-challenges/v1.16.2 (linux/amd64) cert-manager/33df0f22ab5753b942ce2deb36d7e452bc78e49d,verb:POST (16-Jan-2025 06:55:26.475) (total time: 704ms):
Trace[1937146269]: ---"Write to database call succeeded" len:418 704ms (06:55:27.179)
Trace[1937146269]: [704.337595ms] [704.337595ms] END
INFO[0888] ### CleanUp should delete the relevant TXT record for the challengeKey: t41SUo5mG1IyFGznLGC5E6ozlY6Vv9SY68mlLiWWyy0
INFO[0888] ### URL request issued to check if the TXT DNS record is present: /v1/domains/domain.com/records/TXT/_acme-challenge.xyz.abc
INFO[0888] ### TXT Record collected from godaddy: main.DNSRecord{Type:"TXT", Name:"_acme-challenge.xyz.abc", Data:"t41SUo5mG1IyFGznLGC5E6ozlY6Vv9SY68mlLiWWyy0", Priority:0, TTL:600}
INFO[0888] ### TXT Record found : main.DNSRecord{Type:"TXT", Name:"_acme-challenge.xyz.abc", Data:"t41SUo5mG1IyFGznLGC5E6ozlY6Vv9SY68mlLiWWyy0", Priority:0, TTL:600}, for challengeKey: t41SUo5mG1IyFGznLGC5E6ozlY6Vv9SY68mlLiWWyy0
INFO[0888] ### Deleting entry=_acme-challenge.xyz.abc, domain=domain.com
INFO[0888] ### URL request issued to delete the DNS record: /v1/domains/domain.com/records/TXT/_acme-challenge.xyz.abc
INFO[0888] ### TXT Record deleted using Godaddy REST API

cert-manager logs:

E0116 06:56:46.532252       1 sync.go:403] "error waiting for authorization" err="acme: authorization error for xyz.abc.domain.com: 403 urn:ietf:params:acme:error:unauthorized: No TXT record found at _acme-challenge.xyz.abc.domain.com" logger="cert-manager.controller.acceptChallenge" resource_name="domain-wildcard-9-3197869102-3955686264" resource_namespace="traefik" resource_kind="Challenge" resource_version="v1" dnsName="xyz.abc.domain.com" type="DNS-01"
I0116 06:56:46.700794       1 conditions.go:192] Found status change for Certificate "domain-wildcard" condition "Issuing": "True" -> "False"; setting lastTransitionTime to 2025-01-16 06:56:46.700785188 +0000 UTC m=+592.061570259
I0116 06:56:46.713334       1 trigger_controller.go:202] "Backing off from issuance due to previously failed issuance(s). Issuance will next be attempted at 2025-01-16 07:56:46.000000741 +0000 UTC m=+4191.360785788" logger="cert-manager.controller" key="traefik/domain-wildcard"

It seems like the godaddy-webhook deletes the TXT record one second before the cert-manager checks it.

[davidt-gh]

@cmoulliard Can you take a look on @jonsbun ?
It's really give us hard time to make a new ingress using godaddy.

[cmoulliard]

I don't really know when the function cleanUp( is called or triggered. To figure out what it happens we will have to contact core engineers working on cert-manager.

[queil]

We've been facing the same issue in the last few days. Adding the record manually after it has been deleted is a temporary workaround but this bug is hard to ignore. As @jonsbun mentioned in #48 (comment) it seems like there is a race condition between the cert-manager and the webhook? I don't know the internals of either so just guessing.

[cmoulliard]

Here is how cert manager is calling the Webhook cleanup
From Adam Talbot

There are two cases I can see in the cert-manager codebase
When a challenge has reached it "final" state, IE the challenge either passed or has entered a failure state where ACME will no longer continue (invalid, expired, errored)
https://github.com/cert-manager/cert-manager/blob/5a7dba91a23dc85c5f4a5586171726abb785c584/pkg/controller/acmechallenges/sync.go#L127

Or if the challenge K8s object has been deleted, the "handleFinalizer" method will call CleanUp before removing the finalizer on the Challenge object
https://github.com/cert-manager/cert-manager/blob/5a7dba91a23dc85c5f4a5586171726abb785c584/pkg/controller/acmechallenges/sync.go#L271

[alexstojda]

Hey all, i'm also starting to see this now.

I did the workaround suggested here and it seems to have solved the problem for me, will continue to monitor and see if the solution sticks.

[davidt-gh]

I've just migrate my domain from GoDaddy to Route53. This integration isn't ready for production env's

[cmoulliard]

I propose that we add a new helm value to be able to contigure the DNS01 arguments as you did. Can you create a PR ? @skajake