operator keep crashing Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference

[linux-wizard]

operator version: altinity-clickhouse-operator-0.25.4 and altinity-clickhouse-operator-0.25.5
Issue: It seems that updating resources labels outside of the operator may causes issues and elad to the operator in crash loopback.
Recovering from this state is very complicated

altinity-clickhouse-operator I1204 19:44:40.689400       1 util.go:50] StatefulSet.Labels ARE DIFFERENT:
altinity-clickhouse-operator added:
altinity-clickhouse-operator none
altinity-clickhouse-operator modified:
altinity-clickhouse-operator Diff start -------------------------
altinity-clickhouse-operator modified .labels items num: 1
altinity-clickhouse-operator diff item [0]:'["clickhouse-keeper.altinity.com/object-version"]' = '"def6c294836ddef5341f2caea808cdd6e99b3b0b"'
altinity-clickhouse-operator Diff end -------------------------
altinity-clickhouse-operator
altinity-clickhouse-operator removed:
altinity-clickhouse-operator Diff start -------------------------
altinity-clickhouse-operator removed .labels items num: 6
altinity-clickhouse-operator diff item [0]:'["neozo.cloud/category"]' = '"inventory"'
altinity-clickhouse-operator diff item [1]:'["dip2/inventory.0"]' = '"dip2-uat-gold-ao-analytics"'
altinity-clickhouse-operator diff item [2]:'["dip2/inventory.1"]' = '"dip2-uat-gold-ao-international-bi"'
altinity-clickhouse-operator diff item [3]:'["dip2/inventory"]' = '"shared-inventories-dip2-uat"'
altinity-clickhouse-operator diff item [4]:'["dip2/db.engine"]' = '"clickhouse"'
altinity-clickhouse-operator diff item [5]:'["dip2/tenant"]' = '"dip2-uat"'
altinity-clickhouse-operator Diff end -------------------------
altinity-clickhouse-operator I1204 19:44:40.689425       1 util.go:59] Host:0-1[0/1]:shared-inventories-dip2-uat/shared-inventories-dip2-uat:StatefulSet.Annotations ARE EQUAL
altinity-clickhouse-operator I1204 19:44:40.689490       1 statefulset-reconciler.go:460] doDeleteStatefulSet():Host:0-1[0/1]:shared-inventories-dip2-uat/shared-inventories-dip2-uat:shared-inventories-dip2-uat/chk-shared-inventories-dip2-uat-r3-
altinity-clickhouse-operator I1204 19:44:40.735578       1 poller.go:80] Poll():shared-inventories-dip2-uat/0-0:OK shared-inventories-dip2-uat/0-0
altinity-clickhouse-operator I1204 19:44:40.748901       1 poller.go:80] Poll():shared-inventories-dip2-uat/0-0:OK shared-inventories-dip2-uat/0-0
altinity-clickhouse-operator I1204 19:44:40.789994       1 panic.go:783] worker-reconciler-chk.go:343:reconcileHostStatefulSet():end:Host:0-1[0/1]:shared-inventories-dip2-uat/shared-inventories-dip2-uat:reconcile StatefulSet end
altinity-clickhouse-operator I1204 19:44:40.790012       1 panic.go:783] worker-reconciler-chk.go:466:reconcileShardsAndHosts():end:reconcileShardsAndHosts end
altinity-clickhouse-operator I1204 19:44:40.790028       1 panic.go:783] worker-reconciler-chk.go:51:reconcileCR():end:unknown
altinity-clickhouse-operator 2025-12-04T19:44:40Z    INFO    Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference    {"controller": "clickhousekeeperinstallation", "controllerGroup": "clickhouse-keeper
altinity-clickhouse-operator panic: runtime error: invalid memory address or nil pointer dereference [recovered, repanicked]
altinity-clickhouse-operator [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1813a7e]
altinity-clickhouse-operator

Full Stack Trace:

altinity-clickhouse-operator goroutine 209 [running]:
altinity-clickhouse-operator sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
altinity-clickhouse-operator     /clickhouse-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:115 +0x1d4
altinity-clickhouse-operator panic({0x1c3a0e0?, 0x322f390?})
altinity-clickhouse-operator     /usr/local/go/src/runtime/panic.go:783 +0x132
altinity-clickhouse-operator github.com/altinity/clickhouse-operator/pkg/controller/common/poller/domain.(*HostObjectsPoller).WaitHostStatefulSetReady.func1({0x22352d0?, 0x32896a0?}, 0xc00557ea08)
altinity-clickhouse-operator     /clickhouse-operator/pkg/controller/common/poller/domain/poller-host-objects.go:98 +0x3e
altinity-clickhouse-operator github.com/altinity/clickhouse-operator/pkg/controller/common/poller/domain.(*HostObjectPoller[...]).Poll.func2({0x1efbc80?, 0xc00557ea08?})
altinity-clickhouse-operator     /clickhouse-operator/pkg/controller/common/poller/domain/poller-host-object.go:62 +0x3b
altinity-clickhouse-operator github.com/altinity/clickhouse-operator/pkg/controller/common/poller.(*Functions).CallIsDone(...)
altinity-clickhouse-operator     /clickhouse-operator/pkg/controller/common/poller/poller-functions.go:46
altinity-clickhouse-operator github.com/altinity/clickhouse-operator/pkg/controller/common/poller.(*poller).Poll(0xc0078a4c78)
altinity-clickhouse-operator     /clickhouse-operator/pkg/controller/common/poller/poller.go:78 +0x423
altinity-clickhouse-operator github.com/altinity/clickhouse-operator/pkg/controller/common/poller/domain.(*HostObjectPoller[...]).Poll(0x221c340, {0x22352d0, 0x32896a0}, 0xc0017f0000, 0xc005577638, {0x0, 0x0, 0x0})
altinity-clickhouse-operator     /clickhouse-operator/pkg/controller/common/poller/domain/poller-host-object.go:72 +0x405
altinity-clickhouse-operator github.com/altinity/clickhouse-operator/pkg/controller/common/poller/domain.(*HostObjectsPoller).WaitHostStatefulSetReady(0xc0025cc760, {0x22352d0, 0x32896a0}, 0xc0017f0000)
altinity-clickhouse-operator     /clickhouse-operator/pkg/controller/common/poller/domain/poller-host-objects.go:91 +0x1b2
altinity-clickhouse-operator github.com/altinity/clickhouse-operator/pkg/controller/common/statefulset.(*Reconciler).doDeleteStatefulSet(0xc0025990e0, {0x22352d0, 0x32896a0}, 0xc0017f0000)
altinity-clickhouse-operator     /clickhouse-operator/pkg/controller/common/statefulset/statefulset-reconciler.go:484 +0x8c9
altinity-clickhouse-operator github.com/altinity/clickhouse-operator/pkg/controller/common/statefulset.(*Reconciler).recreateStatefulSet(0xc0025990e0, {0x22352d0, 0x32896a0}, 0xc0017f0000, 0x1, 0x0)
altinity-clickhouse-operator     /clickhouse-operator/pkg/controller/common/statefulset/statefulset-reconciler.go:197 +0x73f
altinity-clickhouse-operator github.com/altinity/clickhouse-operator/pkg/controller/common/statefulset.(*Reconciler).updateStatefulSet(0xc0025990e0, {0x22352d0, 0x32896a0}, 0xc0017f0000, 0x1, 0x0)
altinity-clickhouse-operator     /clickhouse-operator/pkg/controller/common/statefulset/statefulset-reconciler.go:288 +0x1806
altinity-clickhouse-operator github.com/altinity/clickhouse-operator/pkg/controller/common/statefulset.(*Reconciler).ReconcileStatefulSet(0xc0025990e0, {0x22352d0, 0x32896a0}, 0xc0017f0000, 0x1, 0x0)
altinity-clickhouse-operator     /clickhouse-operator/pkg/controller/common/statefulset/statefulset-reconciler.go:173 +0xed7
altinity-clickhouse-operator github.com/altinity/clickhouse-operator/pkg/controller/chk.(*worker).reconcileHostStatefulSet(0xc00099a2d0, {0x22352d0, 0x32896a0}, 0xc0017f0000, 0x0)
altinity-clickhouse-operator     /clickhouse-operator/pkg/controller/chk/worker-reconciler-chk.go:364 +0xe4d
altinity-clickhouse-operator github.com/altinity/clickhouse-operator/pkg/controller/chk.(*worker).reconcileHostMain(0xc00099a2d0, {0x22352d0, 0x32896a0}, 0xc0017f0000)
altinity-clickhouse-operator     /clickhouse-operator/pkg/controller/chk/worker-reconciler-chk.go:673 +0xcac
altinity-clickhouse-operator github.com/altinity/clickhouse-operator/pkg/controller/chk.(*worker).reconcileHost(0xc00099a2d0, {0x22352d0, 0x32896a0}, 0xc0017f0000)
altinity-clickhouse-operator     /clickhouse-operator/pkg/controller/chk/worker-reconciler-chk.go:579 +0x833
altinity-clickhouse-operator github.com/altinity/clickhouse-operator/pkg/controller/chk.(*worker).reconcileShardWithHosts.func1(0x41d3c5?)
altinity-clickhouse-operator     /clickhouse-operator/pkg/controller/chk/worker-reconciler-chk.go:535 +0x25
altinity-clickhouse-operator github.com/altinity/clickhouse-operator/pkg/apis/clickhouse-keeper.altinity.com/v1.(*ChkShard).WalkHostsAbortOnError(0xc00036b980, 0xc00768be60)
altinity-clickhouse-operator     /clickhouse-operator/pkg/apis/clickhouse-keeper.altinity.com/v1/type_shard.go:124 +0x72
altinity-clickhouse-operator github.com/altinity/clickhouse-operator/pkg/controller/chk.(*worker).reconcileShardWithHosts(0xc00099a2d0, {0x22352d0, 0x32896a0}, {0x2249378, 0xc00036b980})
altinity-clickhouse-operator     /clickhouse-operator/pkg/controller/chk/worker-reconciler-chk.go:534 +0xa2
altinity-clickhouse-operator github.com/altinity/clickhouse-operator/pkg/controller/chk.(*worker).reconcileShardsAndHosts(0xc00099a2d0, {0x22352d0, 0x32896a0}, {0xc0076ba340, 0x1, 0x1})
altinity-clickhouse-operator     /clickhouse-operator/pkg/controller/chk/worker-reconciler-chk.go:481 +0x5fa
altinity-clickhouse-operator github.com/altinity/clickhouse-operator/pkg/apis/clickhouse-keeper.altinity.com/v1.(*ClickHouseKeeperInstallation).WalkTillError(0xc001c71500, {0x22352d0, 0x32896a0}, 0x0?, 0xc0078a73e8, 0xc0078a73d8, 0xc0078a73c8)
altinity-clickhouse-operator     /clickhouse-operator/pkg/apis/clickhouse-keeper.altinity.com/v1/type_chk.go:671 +0x15f
altinity-clickhouse-operator github.com/altinity/clickhouse-operator/pkg/controller/chk.(*worker).reconcile(0xc00099a2d0, {0x22352d0, 0x32896a0}, 0xc001c71500)
altinity-clickhouse-operator     /clickhouse-operator/pkg/controller/chk/worker-reconciler-chk.go:180 +0xbe5
altinity-clickhouse-operator github.com/altinity/clickhouse-operator/pkg/controller/chk.(*worker).reconcileCR(0xc00099a2d0, {0x22352d0, 0x32896a0}, 0x0?, 0xc000ae8000)
altinity-clickhouse-operator     /clickhouse-operator/pkg/controller/chk/worker-reconciler-chk.go:69 +0x8b1
altinity-clickhouse-operator github.com/altinity/clickhouse-operator/pkg/controller/chk.(*Controller).Reconcile(0xc0001a1900, {0x22351f0, 0xc00059f140}, {{{0xc000c8dc40?, 0x1de12a0?}, {0xc000c8dfe0?, 0xc000cb11f0?}}})
altinity-clickhouse-operator     /clickhouse-operator/pkg/controller/chk/controller.go:80 +0x3c5
altinity-clickhouse-operator sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x22351f0?, {0x22351f0?, 0xc00059f140?}, {{{0xc000c8dc40?, 0x1b15b60?}, {0xc000c8dfe0?, 0x0?}}})
altinity-clickhouse-operator     /clickhouse-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:118 +0xa5
altinity-clickhouse-operator sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00033bc20, {0x2235228, 0xc0002014f0}, {0x1d4b820, 0xc000d7c0a0})
altinity-clickhouse-operator     /clickhouse-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:314 +0x31c
altinity-clickhouse-operator sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00033bc20, {0x2235228, 0xc0002014f0})
altinity-clickhouse-operator     /clickhouse-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:265 +0x197
altinity-clickhouse-operator sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
altinity-clickhouse-operator     /clickhouse-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:226 +0x73
altinity-clickhouse-operator created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 125
altinity-clickhouse-operator     /clickhouse-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:222 +0x509
Stream closed EOF for clickhouse-operator/clickhouse-operator-5769f56c47-mbj5w (altinity-clickhouse-operator)

[veerendra2]

Hi, we get also similar errors and crashloops while updating the labels.

In our case, we use custom helm chart to manage ClickhouseInstallations, we are seing crashloops after bumping helm chart version, like below

...
removed:
none
I1208 09:41:56.202185       1 util.go:50] StatefulSet.Labels ARE DIFFERENT:
added:
none
modified:
Diff start -------------------------
modified .labels items num: 2
diff item [0]:'["helm.sh/chart"]' = '"clickhouse-0.3.0"'
diff item [1]:'["clickhouse-keeper.altinity.com/object-version"]' = '"b5658a164db6c216292d0e9c7691379ba534023b"'
Diff end -------------------------

removed:
none
I1208 09:41:56.202197       1 util.go:59] Host:0-0[0/0]:otel/clickhouse-otel:StatefulSet.Annotations ARE EQUAL
I1208 09:41:56.202259       1 statefulset-reconciler.go:460] doDeleteStatefulSet():Host:0-0[0/0]:otel/clickhouse-otel:otel/chk-clickhouse-otel-otel-0-0
I1208 09:41:56.208493       1 statefulset-reconciler.go:103] unknown:Have StatefulSet available, try to perform label-based comparison for sts: analytics/chi-clickhouse-alpha-alpha-1-0
I1208 09:41:56.208517       1 object-status.go:47] GetObjectStatusFromMetas():unknown:cur and new objects are equal based on object version label. Update of the object is not required. Object: analytics/chi-clickhouse-alpha-alpha-1-0
I1208 09:41:56.228953       1 panic.go:783] worker-reconciler-chk.go:342:reconcileHostStatefulSet():end:Host:0-0[0/0]:otel/clickhouse-otel:reconcile StatefulSet end
I1208 09:41:56.229882       1 worker-service.go:39] reconcileService():unknown:Service found: otel/keeper-clickhouse-otel. Will try to update
I1208 09:41:56.243202       1 worker-service.go:173] updateService():unknown:Update Service success: otel/keeper-clickhouse-otel
I1208 09:41:56.243246       1 worker-service.go:60] reconcileService():unknown:Service reconcile successful: otel/keeper-clickhouse-otel
I1208 09:41:56.243272       1 panic.go:783] worker-reconciler-chk.go:465:reconcileShardsAndHosts():end:reconcileShardsAndHosts end
I1208 09:41:56.243295       1 panic.go:783] worker-reconciler-chk.go:50:reconcileCR():end:unknown
2025-12-08T09:41:56Z    INFO    Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference    {"controller": "clickhousekeeperinstallation", "controllerGroup": "clickhouse-keeper.altinity.com", "controllerKind": "ClickHouseKeeperInstallation", "ClickHouseKeeperInstallation": {"name":"clickhouse-otel","namespace":"otel"}, "namespace": "otel", "name": "clickhouse-otel", "reconcileID": "a4e476e4-a40f-4f77-a000-11254b873620"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered, repanicked]
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x181599e]

goroutine 233 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
    /clickhouse-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:115 +0x1d4
panic({0x1c3d7c0?, 0x3236390?})
    /usr/local/go/src/runtime/panic.go:783 +0x132
github.com/altinity/clickhouse-operator/pkg/controller/common/poller/domain.(*HostObjectsPoller).WaitHostStatefulSetReady.func1({0x2239c70?, 0x32906a0?}, 0xc00a8def08)
    /clickhouse-operator/pkg/controller/common/poller/domain/poller-host-objects.go:98 +0x3e
github.com/altinity/clickhouse-operator/pkg/controller/common/poller/domain.(*HostObjectPoller[...]).Poll.func2({0x1effb40?, 0xc00a8def08?})
    /clickhouse-operator/pkg/controller/common/poller/domain/poller-host-object.go:62 +0x3b
github.com/altinity/clickhouse-operator/pkg/controller/common/poller.(*Functions).CallIsDone(...)
    /clickhouse-operator/pkg/controller/common/poller/poller-functions.go:46
github.com/altinity/clickhouse-operator/pkg/controller/common/poller.(*poller).Poll(0xc00a58ec60)
    /clickhouse-operator/pkg/controller/common/poller/poller.go:78 +0x423
github.com/altinity/clickhouse-operator/pkg/controller/common/poller/domain.(*HostObjectPoller[...]).Poll(0x2220ce0, {0x2239c70, 0x32906a0}, 0xc0037a3d40, 0xc007071e00, {0x0, 0x0, 0x0})
    /clickhouse-operator/pkg/controller/common/poller/domain/poller-host-object.go:72 +0x405
github.com/altinity/clickhouse-operator/pkg/controller/common/poller/domain.(*HostObjectsPoller).WaitHostStatefulSetReady(0xc004013560, {0x2239c70, 0x32906a0}, 0xc0037a3d40)
    /clickhouse-operator/pkg/controller/common/poller/domain/poller-host-objects.go:91 +0x1b2
github.com/altinity/clickhouse-operator/pkg/controller/common/statefulset.(*Reconciler).doDeleteStatefulSet(0xc00403a900, {0x2239c70, 0x32906a0}, 0xc0037a3d40)
    /clickhouse-operator/pkg/controller/common/statefulset/statefulset-reconciler.go:484 +0x8c9
github.com/altinity/clickhouse-operator/pkg/controller/common/statefulset.(*Reconciler).recreateStatefulSet(0xc00403a900, {0x2239c70, 0x32906a0}, 0xc0037a3d40, 0x1, 0x0)
    /clickhouse-operator/pkg/controller/common/statefulset/statefulset-reconciler.go:197 +0x73f
github.com/altinity/clickhouse-operator/pkg/controller/common/statefulset.(*Reconciler).updateStatefulSet(0xc00403a900, {0x2239c70, 0x32906a0}, 0xc0037a3d40, 0x1, 0x0)
    /clickhouse-operator/pkg/controller/common/statefulset/statefulset-reconciler.go:288 +0x1806
github.com/altinity/clickhouse-operator/pkg/controller/common/statefulset.(*Reconciler).ReconcileStatefulSet(0xc00403a900, {0x2239c70, 0x32906a0}, 0xc0037a3d40, 0x1, 0x0)
    /clickhouse-operator/pkg/controller/common/statefulset/statefulset-reconciler.go:173 +0xed7
github.com/altinity/clickhouse-operator/pkg/controller/chk.(*worker).reconcileHostStatefulSet(0xc003055b30, {0x2239c70, 0x32906a0}, 0xc0037a3d40, 0x0)
    /clickhouse-operator/pkg/controller/chk/worker-reconciler-chk.go:363 +0xe4d
github.com/altinity/clickhouse-operator/pkg/controller/chk.(*worker).reconcileHostMain(0xc003055b30, {0x2239c70, 0x32906a0}, 0xc0037a3d40)
    /clickhouse-operator/pkg/controller/chk/worker-reconciler-chk.go:672 +0xcac
github.com/altinity/clickhouse-operator/pkg/controller/chk.(*worker).reconcileHost(0xc003055b30, {0x2239c70, 0x32906a0}, 0xc0037a3d40)
    /clickhouse-operator/pkg/controller/chk/worker-reconciler-chk.go:578 +0x833
github.com/altinity/clickhouse-operator/pkg/controller/chk.(*worker).reconcileShardWithHosts.func1(0x41d3c5?)
    /clickhouse-operator/pkg/controller/chk/worker-reconciler-chk.go:534 +0x25
github.com/altinity/clickhouse-operator/pkg/apis/clickhouse-keeper.altinity.com/v1.(*ChkShard).WalkHostsAbortOnError(0xc001156180, 0xc009d57da0)
    /clickhouse-operator/pkg/apis/clickhouse-keeper.altinity.com/v1/type_shard.go:124 +0x72
github.com/altinity/clickhouse-operator/pkg/controller/chk.(*worker).reconcileShardWithHosts(0xc003055b30, {0x2239c70, 0x32906a0}, {0x224dd58, 0xc001156180})
    /clickhouse-operator/pkg/controller/chk/worker-reconciler-chk.go:533 +0xa2
github.com/altinity/clickhouse-operator/pkg/controller/chk.(*worker).reconcileShardsAndHosts(0xc003055b30, {0x2239c70, 0x32906a0}, {0xc001106c08, 0x1, 0x1})
    /clickhouse-operator/pkg/controller/chk/worker-reconciler-chk.go:480 +0x5fa
github.com/altinity/clickhouse-operator/pkg/apis/clickhouse-keeper.altinity.com/v1.(*ClickHouseKeeperInstallation).WalkTillError(0xc002124c00, {0x2239c70, 0x32906a0}, 0x0?, 0xc00a5913d0, 0xc00a5913c0, 0xc00a5913b0)
    /clickhouse-operator/pkg/apis/clickhouse-keeper.altinity.com/v1/type_chk.go:671 +0x15f
github.com/altinity/clickhouse-operator/pkg/controller/chk.(*worker).reconcile(0xc003055b30, {0x2239c70, 0x32906a0}, 0xc002124c00)
    /clickhouse-operator/pkg/controller/chk/worker-reconciler-chk.go:179 +0xbe5
github.com/altinity/clickhouse-operator/pkg/controller/chk.(*worker).reconcileCR(0xc003055b30, {0x2239c70, 0x32906a0}, 0x0?, 0xc00135ac00)
    /clickhouse-operator/pkg/controller/chk/worker-reconciler-chk.go:68 +0x845
github.com/altinity/clickhouse-operator/pkg/controller/chk.(*Controller).Reconcile(0xc00092fe00, {0x2239b90, 0xc00348da40}, {{{0xc000d075a0?, 0x1de4fc0?}, {0xc000d07670?, 0xc00236fb40?}}})
    /clickhouse-operator/pkg/controller/chk/controller.go:80 +0x3c5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x2239b90?, {0x2239b90?, 0xc00348da40?}, {{{0xc000d075a0?, 0x1b18e60?}, {0xc000d07670?, 0x0?}}})
    /clickhouse-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:118 +0xa5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000b1e8c0, {0x2239bc8, 0xc000527ae0}, {0x1d4f020, 0xc001315020})
    /clickhouse-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:314 +0x31c
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000b1e8c0, {0x2239bc8, 0xc000527ae0})
    /clickhouse-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:265 +0x197
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
    /clickhouse-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:226 +0x73
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 249
    /clickhouse-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:222 +0x509
stream closed: EOF for clickhouse-operator/clickhouse-operator-5f7c64595f-k7kd2 (clickhouse-operator)

[alex-zaitsev]

@linux-wizard , please check if the problem is present in 0.25.6. We have fixed a couple of possible panics in there