Consider having Route not become `Ready` until DNS resolves

[evankanderson]

Expected Behavior

When a Route is Ready, I can reach it at some DNS name. (From places where I can resolve the DNS name*).

* if my cluster name is srv.cluster.local, I'd expect names to only resolve within the cluster.

Actual Behavior

Route will report that it is Ready even if the reported domainname doesn't exist.

Steps to Reproduce the Problem

1. Deploy a cluster.
2. Create a knative Service.
3. Run a job which runs curl against the Service's domainname.

Additional Info

We should have the Route controller check that the DNS name resolves (possibly check that it's pointed at the Istio ingress?), and report the status in a new Condition type. I suggest the name DnsResolved.

We should also switch config-domain.yaml to point to srv.cluster.local so that DNS works (within the cluster) out of the box.

[evankanderson]

/assign @evankanderson
/assign @mattmoor
/assign @tcnghia
/assign @scothis

PTAL and see if this makes sense.

[scothis]

Setting the default dns host to svc.cluster.local is a bit awkward since that value is hard coded into the Route's VirtualService for in cluster access. Assuming Istio doesn't fall over because of the duplicate value.

[tcnghia]

For svc.cluster.local domains, currently named routes aren't supported because we can't add a domain name of the form <target-name>.<route>.<ns>.cluster.local. I think CoreDNS may allow, but not Kube-DNS.

We could probably try something like <target-name>-dot-<route>.<ns>.cluster.local, but that will run into character limit issues. Another alternative is to replace named route with routes that uses the Revision name to sidestep this issue entirely. Since named route can only have only one target, this is functionally equivalent.

Or if we could limit the Route to only resolve cluster-internal names, and adds an [optional] Dispatch component to configure mappings from external domain names to cluster internal names, we could perform this check for Dispatch and not Route.

WDYT?

[tcnghia]

Also, I am not aware of a time-based trigger for a reconcilation, doing this probing in Route controller may be difficult.

WDYT about spinning up short-lived probers after a Route change? I proposed one in #1582 that can help with asserting effectiveness of changes in VirtualService, but we could generalize it to probe for a wider range of conditions.

[evankanderson]

With respect to svc.cluster.local -- you're right that subdomains/subnames won't work. At this point, the number of (DNS, SSL, Istio) items that end up adding more work to support subdomains does lean me toward simply using multiple Routes to get different Service-level names, rather than trying to add a new type of DNS to kube-dns.

Controllers will perform a periodic resync; I think we have the resync period configured to be 5 minutes right now, though:

https://github.com/knative/serving/blob/master/cmd/controller/main.go#L105
https://github.com/kubernetes/client-go/blob/master/informers/factory.go#L101

The DNS lookup could also be done in-memory, though I'd be worried about clogging our reconciliation threads.