DNS Groups. #669

philbrookes · 2025-12-09T08:50:58Z

This PR introduces DNS Groups functionality to enable active-passive failover for DNS records across multiple clusters. Groups allow operators to control which cluster's DNS records are published based on a configurable "active groups" list, enabling scenarios like:

Multi-cluster disaster recovery with controlled failover
Geographic failover between regions
Blue-green deployments at the DNS level
Staged rollouts with DNS-based traffic shifting

Changes

API Changes:

Added ActiveGroups field to DNSRecordStatus to track the current list of active groups
Added IsActive() method to DNSRecord type (interface method for group-aware behavior)
New condition type ConditionTypeActive with reasons:
- ConditionReasonInActiveGroup - Record's group is active and will be published
- ConditionReasonNotInActiveGroup - Record's group is inactive and won't be published

Core Implementation (internal/controller/dnsrecord_groups.go - new file):

TXTResolver interface for querying active groups from DNS
DefaultTXTResolver implementation with support for custom nameservers
GroupAdapter wraps DNSRecordAccessor to provide group-aware IsActive() behavior
getActiveGroups() queries the special TXT record kuadrant-active-groups. to determine which groups are active
unpublishInactiveGroups() cleans up DNS records from previously active groups during failover

Controller Changes (internal/controller/dnsrecord_controller.go):

Integrated group adapter into reconciliation flow
Inactive group records exit early with 15s requeue time
Active group records unpublish inactive group DNS records after successful publish
Authoritative records are exempt from all of these changes

Test Coverage:

Added integration tests
Tests cover:
- Multiple groups with active/inactive switching
- Ungrouped records (always active)
- Inactive group cleanup
- Status condition updates
- Multi-cluster coordination

How It Works

Group Assignment

Each DNS operator instance is started with a group identifier:
--group=us-east
or
GROUP=us-east

Records managed by that operator inherit the group assignment in their status.

Active Groups Declaration

The active groups list is stored as a TXT record in DNS:
kuadrant-active-groups.example.com TXT "groups=us-east&&us-west;version=1"

Reconciliation Flow

Before publishing DNS records, each controller:

Queries the active groups TXT record
Compares its group against the active groups list
If inactive: Updates status condition and requeues (15s)
If active: Publishes its records AND cleans up records from inactive groups
Ungrouped Records

Records without a group assignment (group="") are always active and published alongside whichever groups are currently active. They will never process unpublishing of records.

Example Scenario:

Setup:

Cluster A (group="us-east") has DNSRecord → 1.2.3.4
Cluster B (group="us-west") has DNSRecord → 5.6.7.8
Cluster C (ungrouped) has DNSRecord → 9.9.9.9

Active groups = ["us-east"]:
Published: 1.2.3.4, 9.9.9.9

Switch active groups to ["us-west"]:
Published: 5.6.7.8, 9.9.9.9
(Cluster B unpublishes stale 1.2.3.4)

Manual Verification Instructions

Prerequisites

DNS domain managed by a provider (AWS Route53, GCP, Azure, or CoreDNS)
Two clusters with DNS operator installed
DNS provider credentials configured

Option 1: Verification with AWS Route53

Setup:

make local-setup with 2 clusters, and deploy true. Then edit the deployments to set the group runtime argument (e.g. us-east and us-west).
Create test DNSRecords:

In cluster-1

  cat <<EOF | kubectl apply -f -
  apiVersion: kuadrant.io/v1alpha1
  kind: DNSRecord
  metadata:
    name: test-record
  spec:
    rootHost: test-groups.example.com
    endpoints:
    - dnsName: test-groups.example.com
      recordType: A
      targets:
      - 1.2.3.4
  EOF

In cluster-2

  cat <<EOF | kubectl apply -f -
  apiVersion: kuadrant.io/v1alpha1
  kind: DNSRecord
  metadata:
    name: test-record
  spec:
    rootHost: test-groups.example.com
    endpoints:
    - dnsName: test-groups.example.com
      recordType: A
      targets:
      - 5.6.7.8
  EOF

Set initial active groups (us-east):
Using kuadrant-dns-cli, set the active group to us-east
Verify initial state:
Query DNS (should return 1.2.3.4)
dig test-groups.example.com +short

Check Route53 for published records
aws route53 list-resource-record-sets --hosted-zone-id
--query "ResourceRecordSets[?Name=='test-groups.example.com.']"

Test failover to us-west:

Use dns cli to update the TXT record in Route53 to us-west

Wait 15-30 seconds for reconciliation, then verify:

Query DNS (should now return 5.6.7.8)
dig test-groups.example.com +short

Verification Checklist

DNSRecord in active group shows Active: True condition
DNSRecord in inactive group shows Active: False condition
DNS queries return only targets from active groups and ungrouped clusters
Switching active groups triggers cleanup of old records from inactive groups
Ungrouped records are always published

Related Issues: #620

...etup/dnsrecords/delegating/coredns/loadbalanced/dnsrecord-loadbalanced-coredns-cluster1.yaml

coredns/examples/Corefile

philbrookes · 2025-12-15T15:35:59Z

config/coredns/Corefile

+    log
+
+    rewrite name regex kuadrant-active-groups\.(.*)k.example\.com kuadrant-active-groups-coredns.pb.hcpapps.net
+    forward kuadrant-active-groups-coredns.pb.hcpapps.net /etc/resolv.conf


this will need to be updated to allow this custom host to be passed in at set up time, somehow, that's for another ticket: #670

Boomatang

I have a number of questions. I know it is still a draft and subject to change. For the reason I didn't look at any test changes.

api/v1alpha1/dnsrecord_types.go

types/group.go

api/v1alpha1/conditions.go

internal/controller/base_dnsrecord_reconciler.go

internal/controller/dnsrecord_groups.go

internal/controller/remote_dnsrecord_controller.go

internal/external-dns/registry/txt.go

Boomatang

Still haven't looked at the test. Seen a few very small things. But I started to question the use of group field on the remote reconciles.

I am going to start setting this up locally and play around with it.

internal/controller/dnsrecord_groups.go

Boomatang · 2025-12-18T13:42:16Z

internal/controller/dnsrecord_groups.go

+	activeGroups := r.getActiveGroups(ctx, c, dnsRecord)
+
+	// only process unpublish when there are active groups and we are reconciling a record from an active group
+	if len(activeGroups) == 0 || !dnsRecord.IsActive() {


I think this now can be simplified.

Suggested change

if len(activeGroups) == 0 || !dnsRecord.IsActive() {

if !dnsRecord.IsActive() {

internal/controller/dnsrecord_groups.go

internal/controller/remote_dnsrecord_controller.go

Signed-off-by: Phil Brookes <pbrookes@redhat.com> rh-pre-commit.version: 2.3.2 rh-pre-commit.check-secrets: ENABLED

maksymvavilov · 2025-12-31T07:09:38Z

internal/controller/base_dnsrecord_reconciler.go

 )

+const (
+	activeGroupsTXTRecordName = "kuadrant-active-groups"


Can you replace this with this constant as well, please?

maksymvavilov · 2025-12-31T07:23:34Z

internal/controller/dnsrecord_accessor.go

 	SetStatusConditions(hadChanges bool)
 	SetStatusCondition(conditionType string, status metav1.ConditionStatus, reason, message string)
+	ClearStatusCondition(conditionType string)
+	GetStatusCondition(conditionType string) *metav1.Condition


We seem to never use the GetStatusCondition. We have meta.FindStatusCondition() instead. Is this an issue?

internal/controller/dnsrecord_controller.go

maksymvavilov · 2025-12-31T08:01:25Z

internal/controller/dnsrecord_groups.go

+			}
+
+		}
+	}


This will work with all TXT records in the zone. If it happens to hold a random TXT record with groups=cat in it, this might cause a problem. I don't see that happening, to be honest, but it is a theoretical possibility. If we can't rely on the name of the TXT record here, why don't we take the route ExternalDNS took with the heritage prefix?
I would not expect it to be done in this PR tho - as it is a "theoretical edge case that will, probably, never happen"

As an alternative, we could poke the provider (like we do below) instead of doing a lookup. But I'd imagine there was a good reason we went an extra mile to do a proper lookup.

maksymvavilov · 2025-12-31T08:03:42Z

internal/controller/dnsrecord_groups.go

+//
+// Returns a list of nameserver addresses (e.g., ["10.96.0.10:53", "10.96.0.11"])
+// Returns an empty list if no NAMESERVERS field is found or the secret doesn't exist.
+func (r *BaseDNSRecordReconciler) getNameserversFromProvider(ctx context.Context, c client.Client, dnsRecord DNSRecordAccessor) ([]string, error) {


These are the baseReconciler methods. Why aren't they in the base_dnsrecord_reconciler.go?

maksymvavilov · 2025-12-31T08:47:20Z

internal/controller/remote_dnsrecord_controller.go

 	dnsRecord.SetStatusCondition(string(v1alpha1.ConditionTypeReady), metav1.ConditionTrue, string(v1alpha1.ConditionReasonProviderSuccess), "Provider ensured the dns record")
 	dnsRecord.SetStatusObservedGeneration(dnsRecord.GetDNSRecord().GetGeneration())
+
+	logger.Info("unpublish section start")


This feels like a debug call that was left behind

maksymvavilov · 2025-12-31T08:55:15Z

internal/controller/suite_test.go

 			Scheme:          mgr.GetScheme(),
 			ProviderFactory: providerFactory,
 			DelegationRole:  delegationRole,
+			TXTResolver:     &MockTXTResolver{},


This only works because it is a pointer, and we manipulate this resolver as the test case goes? Manipulate == change the LookupTXT result by adding/removing DNSRecords from the resolver.
This is a curiosity question

maksymvavilov · 2025-12-31T09:05:38Z

internal/external-dns/registry/txt.go

+	Host            string
+	Groups          map[types.Group]*RegistryGroup
+	UngroupedOwners map[string]*RegistryOwner
+}


I found it strange earlier: the OwnerID of RegistryOwner, GroupID of RegistryGroup and Host of RegistryHost seem to never be read (except for one of them once in a test case) as they are repeating information we get from the keys of the respective maps.
I remember the reasoning behind this - convenience and flexibility. But can I ask for a mention somewhere around that the key in the Owners is the same as the OwnerID of the RegistryOwner (for all of the "duplicates")

maksymvavilov · 2025-12-31T09:16:56Z

internal/external-dns/registry/txt.go

 	txtFormatVersion         = "1"
 	externalDNSMapperVersion = ""
+
+	GroupLabelKey = "group"


We have this key in the types/group.go

maksymvavilov · 2025-12-31T09:23:32Z

internal/controller/dnsrecord_groups.go

+		return activeGroups
+	}
+	// Parse the TXT record to extract active groups
+	// Expected format: "groups=group1&&group2&&group3;version=1;other=value"


I expected most parts of this format to reside in constants and be reused in places we want to reuse them.

philbrookes force-pushed the gh-620 branch 8 times, most recently from 5f9673d to 29c24eb Compare December 11, 2025 15:55

philbrookes commented Dec 11, 2025

View reviewed changes

...etup/dnsrecords/delegating/coredns/loadbalanced/dnsrecord-loadbalanced-coredns-cluster1.yaml Outdated Show resolved Hide resolved

philbrookes force-pushed the gh-620 branch from 29c24eb to ad9cde1 Compare December 11, 2025 15:59

philbrookes commented Dec 11, 2025

View reviewed changes

coredns/examples/Corefile Outdated Show resolved Hide resolved

philbrookes force-pushed the gh-620 branch 2 times, most recently from eaf200f to 6795ee1 Compare December 15, 2025 15:09

philbrookes commented Dec 15, 2025

View reviewed changes

philbrookes force-pushed the gh-620 branch 2 times, most recently from 098c135 to e9364ce Compare December 15, 2025 15:38

Boomatang reviewed Dec 16, 2025

View reviewed changes

philbrookes force-pushed the gh-620 branch 3 times, most recently from b43bd9a to 9e54ed6 Compare December 18, 2025 11:38

Boomatang reviewed Dec 18, 2025

View reviewed changes

philbrookes force-pushed the gh-620 branch 4 times, most recently from 5caf25d to a26babc Compare December 29, 2025 12:13

DNS Groups

8f3eb41

Signed-off-by: Phil Brookes <pbrookes@redhat.com> rh-pre-commit.version: 2.3.2 rh-pre-commit.check-secrets: ENABLED

philbrookes force-pushed the gh-620 branch from a26babc to 8f3eb41 Compare December 29, 2025 12:52

philbrookes marked this pull request as ready for review December 29, 2025 14:51

philbrookes changed the title ~~[WIP] Gh 620~~ DNS Groups. Dec 29, 2025

maksymvavilov reviewed Dec 31, 2025

View reviewed changes

	if len(activeGroups) == 0 \|\| !dnsRecord.IsActive() {
	if !dnsRecord.IsActive() {

+              			}
+              		}
+              	}

DNS Groups. #669

Are you sure you want to change the base?

DNS Groups. #669

Uh oh!

Conversation

philbrookes commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

API Changes:

Core Implementation (internal/controller/dnsrecord_groups.go - new file):

Controller Changes (internal/controller/dnsrecord_controller.go):

Test Coverage:

How It Works

Example Scenario:

Manual Verification Instructions

Setup:

In cluster-1

In cluster-2

Verification Checklist

Related Issues: #620

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Boomatang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Boomatang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

philbrookes commented Dec 9, 2025 •

edited

Loading