Recursive resolver not setting `ad` bit -- unbound does.

[pinheadmz]

There's going to be a lot of logs in this post. In summary, running hsd with --rs-no-unbound fails to verify the subdomain sebastian.rasor whereas the unbound resolver does verify it:

Invalid RRSIGs.

Queries

--rs-no-unbound

--> dig @127.0.0.1 -p 5350 sebastian.rasor +dnssec

; <<>> DiG 9.16.10 <<>> @127.0.0.1 -p 5350 sebastian.rasor +dnssec
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45053
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
; COOKIE: 1743cb239b9132f9 (echoed)
;; QUESTION SECTION:
;sebastian.rasor.               IN      A

;; ANSWER SECTION:
sebastian.rasor.        300     IN      A       45.77.111.169
sebastian.rasor.        300     IN      RRSIG   A 8 2 300 20210311024641 20210209024641 3381 sebastian.rasor. Ipwqf6t2UuDuR2ygVidSiWCXvK6XB/D9AzWv0AFIzGEvd7iQaItOa1tm uXUK9DWtMwClZfBbxl10c3hMhdFZgHLWkXRCLiWgOcPlKyOekMFf+Dh8 zElNS6UEji5NV7vP2FCYbzAhweEsIJE28v9ofhHOZSVGsaRpQDeXnAif 5IEeFzA666QUE61xkShkTFOcrjwhdlq074lXpIOP2PyUBCK8ESwdn/kp 8ZFj5VCDgyHf9nTsfmSIpKYaL/s1jts6mRZhMCFlXGtRnE+ocj9LsGWa Ds6y8TKVDf3ZYPls17xgiBsTybL5PYkJgZzRrrc78F7avr7oa0mhxtZn qYAjvQ==

;; SIG0 PSEUDOSECTION:
.                       0       ANY     SIG     0 253 0 0 20210209101345 20210208221345 15350 . +zVVjI0QbVXeRCY/ojd0tx+Ekkof4o+OCWLDHfD4h1MWjVC3It5+ItJL ZN1NeGA/iW7RjcCUVlCOfjX1UKESzQ==

;; Query time: 69 msec
;; SERVER: 127.0.0.1#5350(127.0.0.1)
;; WHEN: Mon Feb 08 23:13:45 EST 2021
;; MSG SIZE  rcvd: 454

with unbound

--> dig @127.0.0.1 -p 5350 sebastian.rasor +dnssec

; <<>> DiG 9.16.10 <<>> @127.0.0.1 -p 5350 sebastian.rasor +dnssec
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 182
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
; COOKIE: d46ce3a95f53a17f (echoed)
;; QUESTION SECTION:
;sebastian.rasor.               IN      A

;; ANSWER SECTION:
sebastian.rasor.        300     IN      A       45.77.111.169
sebastian.rasor.        300     IN      RRSIG   A 8 2 300 20210311024641 20210209024641 3381 sebastian.rasor. Ipwqf6t2UuDuR2ygVidSiWCXvK6XB/D9AzWv0AFIzGEvd7iQaItOa1tm uXUK9DWtMwClZfBbxl10c3hMhdFZgHLWkXRCLiWgOcPlKyOekMFf+Dh8 zElNS6UEji5NV7vP2FCYbzAhweEsIJE28v9ofhHOZSVGsaRpQDeXnAif 5IEeFzA666QUE61xkShkTFOcrjwhdlq074lXpIOP2PyUBCK8ESwdn/kp 8ZFj5VCDgyHf9nTsfmSIpKYaL/s1jts6mRZhMCFlXGtRnE+ocj9LsGWa Ds6y8TKVDf3ZYPls17xgiBsTybL5PYkJgZzRrrc78F7avr7oa0mhxtZn qYAjvQ==

;; SIG0 PSEUDOSECTION:
.                       0       ANY     SIG     0 253 0 0 20210209101400 20210208221400 15350 . DIfBb2KgnM0zFbZG2EtY3e/w73Cj7ktlqC4+iwZYDdBibg0FqJ7BR3V0 6v/On4WDWl7Z/4DdoYtkuKc6omKt9A==

;; Query time: 95 msec
;; SERVER: 127.0.0.1#5350(127.0.0.1)
;; WHEN: Mon Feb 08 23:14:00 EST 2021
;; MSG SIZE  rcvd: 454

Logs --rs-no-unbound

[D:2021-02-09T04:13:45Z] (rs) Querying sebastian.rasor. (A).
[D:2021-02-09T04:13:45Z] (rs) Switching authority: hints.local.
[D:2021-02-09T04:13:45Z] (rs) Switching zone: [.]
[D:2021-02-09T04:13:45Z] (rs) Querying server: 127.0.0.1 (32414) (tcp=false)
[S:2021-02-09T04:13:45Z] (ns) DNS Request:
[S:2021-02-09T04:13:45Z] (ns) ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32414
[S:2021-02-09T04:13:45Z] (ns) ;; flags: ; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
[S:2021-02-09T04:13:45Z] (ns) 
[S:2021-02-09T04:13:45Z] (ns) ;; OPT PSEUDOSECTION:
[S:2021-02-09T04:13:45Z] (ns) ; EDNS: version: 0, flags: do; udp: 4096
[S:2021-02-09T04:13:45Z] (ns) ;; QUESTION SECTION:
[S:2021-02-09T04:13:45Z] (ns) ;sebastian.rasor. IN A
[S:2021-02-09T04:13:45Z] (ns) 
[S:2021-02-09T04:13:45Z] (ns) ;; MSG SIZE  rcvd: 44
[S:2021-02-09T04:13:45Z] (ns) DNS Response:
[S:2021-02-09T04:13:45Z] (ns) ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32414
[S:2021-02-09T04:13:45Z] (ns) ;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 3, ADDITIONAL: 2
[S:2021-02-09T04:13:45Z] (ns) 
[S:2021-02-09T04:13:45Z] (ns) ;; OPT PSEUDOSECTION:
[S:2021-02-09T04:13:45Z] (ns) ; EDNS: version: 0, flags: do; udp: 4096
[S:2021-02-09T04:13:45Z] (ns) ;; QUESTION SECTION:
[S:2021-02-09T04:13:45Z] (ns) ;sebastian.rasor. IN A
[S:2021-02-09T04:13:45Z] (ns) 
[S:2021-02-09T04:13:45Z] (ns) ;; AUTHORITY SECTION:
[S:2021-02-09T04:13:45Z] (ns) rasor. 21600 IN NS ns1.rasor.
[S:2021-02-09T04:13:45Z] (ns) rasor. 21600 IN DS 28654 8 2 8C5D881B7650C12FDB10BB9A6FAE3BF51DF47961F68D003EE17AA8E7 9F22BF83  ; alg = RSASHA256 ; hash = SHA256
[S:2021-02-09T04:13:45Z] (ns) rasor. 10800 IN RRSIG DS 13 1 21600 20220209041345 20210208041345 60944 . UmHpuZ/4RyvRziwNBHzDgXBSeHWWMDEIYRLp9emtiH1dsdwVd3iAmABk LCExWf6qm9Fsj9vimhhrNW20m79zTA==  ; alg = ECDSAP256SHA256
[S:2021-02-09T04:13:45Z] (ns) 
[S:2021-02-09T04:13:45Z] (ns) ;; ADDITIONAL SECTION:
[S:2021-02-09T04:13:45Z] (ns) ns1.rasor. 21600 IN A 45.77.111.169
[S:2021-02-09T04:13:45Z] (ns) 
[S:2021-02-09T04:13:45Z] (ns) ;; MSG SIZE  rcvd: 250
[D:2021-02-09T04:13:45Z] (rs) Verifying zone change to [.]
[D:2021-02-09T04:13:45Z] (rs) Checking signatures...
[D:2021-02-09T04:13:45Z] (rs) Querying server: 127.0.0.1 (55484) (tcp=false)
[S:2021-02-09T04:13:45Z] (ns) DNS Request:
[S:2021-02-09T04:13:45Z] (ns) ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55484
[S:2021-02-09T04:13:45Z] (ns) ;; flags: ; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
[S:2021-02-09T04:13:45Z] (ns) 
[S:2021-02-09T04:13:45Z] (ns) ;; OPT PSEUDOSECTION:
[S:2021-02-09T04:13:45Z] (ns) ; EDNS: version: 0, flags: do; udp: 4096
[S:2021-02-09T04:13:45Z] (ns) ;; QUESTION SECTION:
[S:2021-02-09T04:13:45Z] (ns) ;. IN DNSKEY
[S:2021-02-09T04:13:45Z] (ns) 
[S:2021-02-09T04:13:45Z] (ns) ;; MSG SIZE  rcvd: 28
[S:2021-02-09T04:13:45Z] (ns) DNS Response:
[S:2021-02-09T04:13:45Z] (ns) ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55484
[S:2021-02-09T04:13:45Z] (ns) ;; flags: qr aa; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
[S:2021-02-09T04:13:45Z] (ns) 
[S:2021-02-09T04:13:45Z] (ns) ;; OPT PSEUDOSECTION:
[S:2021-02-09T04:13:45Z] (ns) ; EDNS: version: 0, flags: do; udp: 4096
[S:2021-02-09T04:13:45Z] (ns) ;; QUESTION SECTION:
[S:2021-02-09T04:13:45Z] (ns) ;. IN DNSKEY
[S:2021-02-09T04:13:45Z] (ns) 
[S:2021-02-09T04:13:45Z] (ns) ;; ANSWER SECTION:
[S:2021-02-09T04:13:45Z] (ns) . 10800 IN DNSKEY 257 3 13 T9cURJ2M/Mz9q6UsZNY+Ospyvj+Uv+tgrrWkLtPQwgU/Xu5Yk0l02Sn5 ua2xAQfEYIzRO6v5iA+BejMeEwNP4Q==  ; KSK ; alg = ECDSAP256SHA256 ; key id = 35215
[S:2021-02-09T04:13:45Z] (ns) . 10800 IN DNSKEY 256 3 13 I5nPs6clFa1gnwn9IpVDGdJLfEONzgD1NcfuEwEIVuIoHdZGgvVblsLN bRO+spW3nQYHg92svhy1HOjTiFBIsQ==  ; ZSK ; alg = ECDSAP256SHA256 ; key id = 60944
[S:2021-02-09T04:13:45Z] (ns) . 10800 IN RRSIG DNSKEY 13 0 10800 20220209041345 20210208041345 35215 . z5XC4Q/6ryOkAcHyGHQ/O5B0sqBuTs73hgHGVh2JqHhomIx6YtrvkPsD czvZYft74nN6PWRC59bDQg9UOwxDJA==  ; alg = ECDSAP256SHA256
[S:2021-02-09T04:13:45Z] (ns) 
[S:2021-02-09T04:13:45Z] (ns) ;; MSG SIZE  rcvd: 280
[D:2021-02-09T04:13:45Z] (rs) Validated DNSSEC signatures.
[D:2021-02-09T04:13:45Z] (rs) Switching authority: ns1.rasor.
[D:2021-02-09T04:13:45Z] (rs) Switching zone: [.->rasor.]
[D:2021-02-09T04:13:45Z] (rs) Querying server: 45.77.111.169 (26868) (tcp=false)
[D:2021-02-09T04:13:45Z] (rs) Verifying zone change to [rasor.]
[D:2021-02-09T04:13:45Z] (rs) Checking signatures...
[D:2021-02-09T04:13:45Z] (rs) Querying server: 45.77.111.169 (18276) (tcp=false)
[D:2021-02-09T04:13:45Z] (rs) Invalid RRSIGs.
[D:2021-02-09T04:13:45Z] (rs) Trust chain broken due to lack of child verification.
[D:2021-02-09T04:13:45Z] (rs) Traversed zones: ., rasor. for sebastian.rasor. (A).
[D:2021-02-09T04:13:45Z] (rs) Finishing resolving sebastian.rasor. (A) (hops=1).

[rithvikvibhu]

Digging into this with another (live) example: e.xp

It looks like BNS is trying to compare an RRSIG for e.xp A with DNSKEYs from the xp zone. It doesn't recognize that xp and e.xp are different zones.

Why I think this affects only some subdomains:

• bns seems to switch zones when it gets authority instead of an answer
• but both the parent and child zones can be on the same nameserver
• So when bns asks the parent (xp) for a child record (e.xp A) and gets a good answer, it assumes that domain (e.xp) belongs to the same parent (xp) and tries to match it with DNSKEYs (29411) it knows for the parent zone (xp). - this is the Invalid RRSIGs. error, it's verifying with wrong keys
• It should instead know (somehow) that the domain (e.xp) is not in the parent zone, switch to the child zone (e.xp), verify the zone change, fetch keys (46984), and only then verify signatures for the record.

The DNS zone from https://sitecheck.htools.work/check/e.xp :

[rithvikvibhu]

Checking out other validators like delv which do validate them correctly, they go bottom-up - start from the domain and work their way up to the root. So I'm not sure how this will work in bns, which starts from the root and resolves to the domain.

[buffrr]

It should instead know (somehow) that the domain (e.xp) is not in the parent zone, switch to the child zone (e.xp), verify the zone change, fetch keys (46984), and only then verify signatures for the record.

bns is just following a simple algorithm to recurse not handling many of the edge cases the fancy recursors deal with. It should keep track of zone cuts more carefully:

zone cuts = [.]

dig @root e.xp A

xp. NS ns1.hshub.
xp. NS ns2.hshub.
xp. DS .......

There's a delegation xp. must be a zone.

zone cuts = [ ., xp. ]

dig @ns1.hshub. e.xp A
e.xp.                   20      IN      A       74.91.123.138
e.xp.                   20      IN      RRSIG   A 13 2 20 20230330000000 20230309000000 46984 e.xp. r7HzCiu1rqD+AqSZMhyfdOAMoEISQusDFyR9PVxFZ8ozwBj3X2MvlW400RuxajVpKe5DDpu8k8U9GpoNB0ghig==

We got an answer for e.xp. We only know of two zones but we're not sure if e.xp is its own zone or part of xp. zone yet. One way to find out is do an SOA lookup:

dig @ns1.hshub. e.xp SOA
e.xp.                   20      IN      SOA     ns1.varo. ops.varo.domains. 1 10800 3600 604800 3600

We got a positive answer an SOA with the owner name being e.xp. looks like e.xp is a zone!

zone cuts = [., xp., e.xp.]

That means we need to find a trust path to this zone and then verify the RRSIGs.

But also there's another way to tell without doing the SOA lookup notice the signer name for the A RRSIG is e.xp which means it's signed by the e.xp. zone dnskeys!! At this point you should be pretty confident that a zone cut exists at e.xp and just find a trust path to it.

[rithvikvibhu]

Didn't know about the RRSIG having the zone name, that's a clever way to identify them. Will try this method, thanks @buffrr