Add .test to be classified as an "internal-only hostname"

[jaydrogers]

What this PR does

• This adds .test to be classified as an "internal" TLD

Background

The Caddy docs specifically reference the following TLDs as "internal":

• .localhost
• .local
• .internal
• .home.arpa

Why this change is proposed

1. .test is a domain that is reserved specifically for testing DNS and would never be resolved by a root TLD server (causing ACME challenges to never resolve anyways)
2. Many developers use .test for local development

How this improves the product if merged

It can improve developer experience allowing myapp.test to be set as a hostname without having Caddy trying to run an ACME challenge for trusted SSL.

Example Caddyfile

myapp.test {
  # My Caddy configuration
}

The expected result after this change would trigger the "internal" SSL generation instead of trying to obtain one through Let's Encrypt, etc.

⚠️ Warning: I'm not a Go developer

I was able to form this PR by following the structure of this previous commit: 6668587, which was the solution to this issue caddyserver/caddy#5800

Please feel free to scrutinize or improve this PR as needed. I just put this together to make it easy as possible for the maintainers as long as my syntax is true 🤠

Related docs change

If this change is accepted, I also prepared a PR on the docs site to document this change:

• docs: Add .test as an internal-only hostname website#491

References

Please see official documentation below to explain more on .test.

• IETF RFC 2606 - TLDs for testing
• IETF RFC 6761 - Domain Name Reservation Considerations for "test."

Thanks for your support and maintenance of this project 🙏

[mholt]

Thanks for the change. This is probably fine, but my one holdup is that there's nothing that I can find that says that .test is primarily used for local testing. The RFC says it's for "testing of current or new DNS related code" which is not really what we're doing here. There could in fact be a corporate CA set up to issue certs for .test domains.

It is probably a safe assumption most of the time that it's local-only though. I dunno.

The second RFC also says:

2. Application software SHOULD NOT recognize test names as special,
   and SHOULD use test names as they would other domain names.

Wondering if anyone has any thoughts on this. I'm torn.

[jaydrogers]

Thanks for the additional information. Just wanted to share some industry practices (not saying this makes it more correct or not).

Laravel Herd

Laravel PHP has a development tool called "Laravel Herd" that uses the .test pattern: https://herd.laravel.com/docs/macos/getting-started/sites

Local Development with Docker

Another modern use case is using Docker in local development. When you spin up multiple containers with Docker Compose, you do not want to use .localhost to reference other containers because the origin container will always reference itself.

From the selection we have from the RFC, .test makes the most sense based off of what is reserved:

  ".test" is recommended for use in testing of current or new DNS
  related code.

  ".example" is recommended for use in documentation or as examples.

  ".invalid" is intended for use in online construction of domain
  names that are sure to be invalid and which it is obvious at a
  glance are invalid.

  The ".localhost" TLD has traditionally been statically defined in
  host DNS implementations as having an A record pointing to the
  loop back IP address and is reserved for such use.  Any other use
  would conflict with widely deployed code which assumes this use.

Just wanted to add further detail on the motivation behind the proposed change 😃

Thanks again! Would love to hear further discussion if anyone has any additional thoughts.

[mholt]

That's good info -- is the motivation for this change simply because doing tls internal in a site block (or internal_certs at the top for all sites) is too tedious?

[mholt]

Okay, so the real issue is you want to transition from dev to production environments seamlessly by changing only an env var. (This PR is a good example of the XY problem!)

I think this approach would be more of a band-aid and will only serve a few use cases, whereas there are typically many things that need adjustment from dev to production.

Something that probably deserves more of my attention and some careful thought.

[jaydrogers]

Ha! Thanks for sharing the XY problem. 😃

To your comment:

I think this approach would be more of a band-aid and will only serve a few use cases, whereas there are typically many things that need adjustment from dev to production.

My use case really isn’t in scope of this discussion though, I was just sharing on what motivated me.

This PR is all about 👉 What hostnames qualify for publicly-trusted certificates?

My proposal is to add .test because from my understanding of Caddy (obviously I’m the novice here), I could not see a use case where someone needed an automated and trusted SSL certificate on an external server with .test. To my understanding, it would not even work because ACME validations would fail because .test would be an internal configuration.

Since I saw a more common use case of .test being in local development, I created the PR to have it act like .localhost.

Food for thought while you think this through.

I appreciate the conversation. Have a good weekend ✌️

[timelordx]

Where I ran into a challenge

The problem was that if I want to take the image from Development to Production, all I should need to do is change the environment variable:

Environment ${SERVER_NAME} Value
Development example.dev.test
Production example.com

My expected result

I was hoping the user could set ${SERVER_NAME} to whatever the domain was and it would:

1. Self-sign a certificate in development (because of .test)
2. Automatically generate a trusted certificate in production (because of .com)

Why this wont work

In order to use .test and .com across the same app, I would have to create two blocks:

Development (SERVER_NAME=example.dev.test)

{$SERVER_NAME} {
+       internal
	# My config
}

Production (SERVER_NAME=example.com)

Then I would need a separate block for production and remove internal, because I want automatic via ZeroSSL or Let's Encrypt.

{$SERVER_NAME} {
-       internal
	# My config
}

This can easily be addressed via another environment variable, for example, ${TLS_DIRECTIVE}, with a common stanza block for both environments:

{$SERVER_NAME} {
	{$TLS_DIRECTIVE}
	# My config
}

Development

SERVER_NAME=example.dev.test
TLS_DIRECTIVE='tls internal'

Production

SERVER_NAME=example.com
TLS_DIRECTIVE=''

[jaydrogers]

Thanks for chiming in with your offer to help @timelordx 👍 I greatly appreciate your efforts to assist the open source community. I did discover this method when I was engineering a solution (before I opened up this PR).

I just want to reiterate: My use case is outside the scope of discussion for this PR. I guess I shared too much information how I got here which made it sound like an "XY Problem".

In best efforts to keep this conversation on topic, I hid my own comment as "off topic". I really want to make sure we're keeping this an "X Problem" discussion 😃

The problem this PR solves

• Certmagic assumes .test domains can receive publicly-trusted certificates, when it's not possible

References

Section 6.2 in RFC 6761 states how .test domains should be handled, specifically defining it for use in "private networks":

The domain "test.", and any names falling within ".test.", are
special in the following ways:

1. Users are free to use these test names as they would any other
   domain names. However, since there is no central authority
   responsible for use of test names, users SHOULD be aware that
   these names are likely to yield different results on different
   networks.

2. Application software SHOULD NOT recognize test names as special,
   and SHOULD use test names as they would other domain names.

3. Name resolution APIs and libraries SHOULD NOT recognize test
   names as special and SHOULD NOT treat them differently. Name
   resolution APIs SHOULD send queries for test names to their
   configured caching DNS server(s).

4. Caching DNS servers SHOULD recognize test names as special and
   SHOULD NOT, by default, attempt to look up NS records for them,
   or otherwise query authoritative DNS servers in an attempt to
   resolve test names. Instead, caching DNS servers SHOULD, by
   default, generate immediate negative responses for all such
   queries. This is to avoid unnecessary load on the root name
   servers and other name servers. Caching DNS servers SHOULD offer
   a configuration option (disabled by default) to enable upstream
   resolving of test names, for use in networks where test names are
   known to be handled by an authoritative DNS server in said
   private network.

5. Authoritative DNS servers SHOULD recognize test names as special
   and SHOULD, by default, generate immediate negative responses for
   all such queries, unless explicitly configured by the
   administrator to give positive answers for test names.

6. DNS server operators SHOULD, if they are using test names,
   configure their authoritative DNS servers to act as authoritative
   for test names.

7. DNS Registries/Registrars MUST NOT grant requests to register
   test names in the normal way to any person or entity. Test names
   are reserved for use in private networks and fall outside the set
   of names available for allocation by registries/registrars.
   Attempting to allocate a test name as if it were a normal DNS
   domain name will probably not work as desired, for reasons 4, 5,
   and 6 above.

You can see that I highlighted 6.2.5, where it specifically calls for authoritative DNS servers should return immediate negative responses by default (unless they configure it for their internal network).

It becomes more clear in 6.2.7 that test names are specifically reserved for private networks.

Even though 6.2.2 calls for applications "SHOULD NOT recognize test names as special", that is within the scope of DNS resolution (which we're not touching in this PR). Certmagic is working within the scope of certificate generation (not changing how DNS resolves).

Conclusion

Because .test names are specifically named as private in IETF RFC 6761, it is not possible for a publicly-trusted certificate to be issued through an ACME challenge, therefore .test domains should be treated as "internal/private".

With all this discussion in mind and going through these notes again, I still see a valid proposal to merge this PR. Hope this helps keep clarity to my original intentions of opening the PR 🤓

Thanks again fellas 👍

[mholt]

@jaydrogers One other thing:

My proposal is to add .test because from my understanding of Caddy (obviously I’m the novice here), I could not see a use case where someone needed an automated and trusted SSL certificate on an external server with .test. To my understanding, it would not even work because ACME validations would fail because .test would be an internal configuration.

It's quite common to run your own ACME servers for internal infrastructure -- Caddy even has this feature built-in, so you can run foo.test domains in your company, for example, and serve them with certificates.

Even though 6.2.2 calls for applications "SHOULD NOT recognize test names as special", that is within the scope of DNS resolution (which we're not touching in this PR).

I don't know about that. The RFC is about the domain names in particular, and "applications" is usually used distinctly from "DNS servers" or "DNS resolvers", which are services for the applications to use.

Sorry to be so stubborn on this one but it still doesn't quite sit right with me to assume that .test will strictly be localhost.

[jaydrogers]

It's quite common to run your own ACME servers for internal infrastructure -- Caddy even has this feature built-in, so you can run foo.test domains in your company, for example, and serve them with certificates.

Oh boy, today I learned 😃

Then in this case, I would NOT merge this PR.

When I originally opened this issue, my critical assumption was ACME challenges would always be external -- which is obviously not the case.

Sorry to waste your time on this, but I greatly appreciate the conversation!

Sorry to be so stubborn on this one but it still doesn't quite sit right with me to assume that .test will strictly be localhost.

No apologies needed whatsoever! Thanks for not blindly merging PRs that create more headache 😃

[mholt]

Thanks for understanding. And thanks for the discussion!