[Bug] temporal worker stuck in CrashLoopBackoff on Fresh Deployment #809
Description
What are you really trying to do?
I was trying to deploy a temporal server in my single-node Kubernetes cluster with helm chart.
Describe the bug
temporaltest-worker pod is consistently in a loop of Running -> Error -> CrashLoopBackOff every 2 minutes.
Minimal Reproduction
helm install \
--repo https://go.temporal.io/helm-charts \
--set server.replicaCount=1 \
--set cassandra.config.cluster_size=1 \
--set elasticsearch.replicas=1 \
--set prometheus.enabled=false \
--set grafana.enabled=false \
temporaltest temporal \
--timeout 15m
Environment/Versions
- OS and processor: Ubuntu 24.04.3 LTS, Intel i7-11850H
- Temporal Version: Unmodified. Based on what helm chart provides
- I am using Kubernetes
Additional Context
Log from temporal-worker
Defaulted container "temporal-worker" out of: temporal-worker, check-cassandra (init), check-cassandra-temporal-schema (init), check-elasticsearch-index (init)
TEMPORAL_ADDRESS is not set, setting it to 192.168.94.173:7233
2025/12/17 08:48:56 Loading config; env=docker,zone=,configDir=config
2025/12/17 08:48:56 Loading config files=[config/docker.yaml]
{"level":"info","ts":"2025-12-17T08:48:56.797Z","msg":"Build info.","git-time":"2025-10-29T23:52:28.000Z","git-revision":"b9a3ead06354c120324a92895b65576a8467b5e6","git-modified":false,"go-arch":"amd64","go-os":"linux","go-version":"go1.25.0","cgo-enabled":false,"server-version":"1.29.1","debug-mode":false,"logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/cmd/server/main.go:163"}
{"level":"info","ts":"2025-12-17T08:48:56.797Z","msg":"Updated dynamic config","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/dynamicconfig/file_based_client.go:193"}
{"level":"warn","ts":"2025-12-17T08:48:56.797Z","msg":"Not using any authorizer and flag `--allow-no-auth` not detected. Future versions will require using the flag `--allow-no-auth` if you do not want to set an authorizer.","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/cmd/server/main.go:193"}
{"level":"info","ts":"2025-12-17T08:48:56.843Z","msg":"Use rpc address 127.0.0.1:7233 for cluster active.","component":"metadata-initializer","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/temporal/cluster_metadata_loader.go:68"}
{"level":"info","ts":"2025-12-17T08:48:56.844Z","msg":"Service is not requested, skipping initialization.","service":"history","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:472"}
{"level":"info","ts":"2025-12-17T08:48:56.844Z","msg":"Service is not requested, skipping initialization.","service":"matching","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:492"}
{"level":"info","ts":"2025-12-17T08:48:56.844Z","msg":"Service is not requested, skipping initialization.","service":"frontend","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:521"}
{"level":"info","ts":"2025-12-17T08:48:56.844Z","msg":"Service is not requested, skipping initialization.","service":"internal-frontend","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:521"}
{"level":"info","ts":"2025-12-17T08:48:56.865Z","msg":"Quota changed","service":"worker","component":"persistence","scope":"host","current-quota":null,"new-quota":500,"logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/quotas/calculator/logged_calculator.go:100"}
{"level":"info","ts":"2025-12-17T08:48:56.869Z","msg":"Initialized service resolver for persistence rate limiting","service":"worker","service":"worker","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/service/fx.go:68"}
{"level":"info","ts":"2025-12-17T08:48:56.869Z","msg":"historyClient: ownership caching disabled","service":"worker","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/client/history/client.go:66"}
{"level":"info","ts":"2025-12-17T08:48:56.875Z","msg":"creating new visibility manager","visibility_plugin_name":"elasticsearch","visibility_index_name":"temporal_visibility_v1_dev","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/persistence/visibility/factory.go:131"}
{"level":"info","ts":"2025-12-17T08:48:56.876Z","msg":"Starting server for services","value":{"worker":{}},"logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/temporal/server_impl.go:84"}
{"level":"info","ts":"2025-12-17T08:48:56.876Z","msg":"PProf listen on ","host":"localhost","port":7936,"logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/pprof/pprof.go:58"}
{"level":"info","ts":"2025-12-17T08:48:56.920Z","msg":"RuntimeMetricsReporter started","service":"worker","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/metrics/runtime.go:121"}
{"level":"info","ts":"2025-12-17T08:48:56.920Z","msg":"worker starting","service":"worker","component":"worker","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/service/worker/service.go:214"}
{"level":"info","ts":"2025-12-17T08:48:56.924Z","msg":"Membership heartbeat upserted successfully","address":"192.168.94.173","port":6939,"hostId":"39377839-db25-11f0-8d87-6a27f3a71d4b","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/membership/ringpop/monitor.go:316"}
{"level":"info","ts":"2025-12-17T08:48:56.929Z","msg":"bootstrap hosts fetched","bootstrap-hostports":"192.168.94.173:6939,192.168.94.172:6933,192.168.94.170:6934,192.168.94.171:6935","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/membership/ringpop/monitor.go:358"}
{"level":"info","ts":"2025-12-17T08:48:56.930Z","msg":"Current reachable members","component":"service-resolver","service":"frontend","addresses":["192.168.94.172:7233"],"logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/membership/ringpop/service_resolver.go:305"}
{"level":"info","ts":"2025-12-17T08:48:56.930Z","msg":"Current reachable members","component":"service-resolver","service":"history","addresses":["192.168.94.170:7234"],"logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/membership/ringpop/service_resolver.go:305"}
{"level":"info","ts":"2025-12-17T08:48:56.930Z","msg":"Current reachable members","component":"service-resolver","service":"matching","addresses":["192.168.94.171:7235"],"logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/membership/ringpop/service_resolver.go:305"}
{"level":"info","ts":"2025-12-17T08:48:56.930Z","msg":"Current reachable members","component":"service-resolver","service":"worker","addresses":["192.168.94.173:0"],"logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/membership/ringpop/service_resolver.go:305"}
{"level":"warn","ts":"2025-12-17T08:49:02.939Z","msg":"error creating sdk client","service":"worker","error":"failed reaching server: context deadline exceeded","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/sdk/factory.go:98"}
{"level":"warn","ts":"2025-12-17T08:49:09.110Z","msg":"error creating sdk client","service":"worker","error":"failed reaching server: context deadline exceeded","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/sdk/factory.go:98"}
{"level":"warn","ts":"2025-12-17T08:49:15.483Z","msg":"error creating sdk client","service":"worker","error":"failed reaching server: context deadline exceeded","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/sdk/factory.go:98"}
{"level":"warn","ts":"2025-12-17T08:49:22.230Z","msg":"error creating sdk client","service":"worker","error":"failed reaching server: context deadline exceeded","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/sdk/factory.go:98"}
{"level":"warn","ts":"2025-12-17T08:49:29.754Z","msg":"error creating sdk client","service":"worker","error":"failed reaching server: context deadline exceeded","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/sdk/factory.go:98"}
{"level":"warn","ts":"2025-12-17T08:49:38.702Z","msg":"error creating sdk client","service":"worker","error":"failed reaching server: context deadline exceeded","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/sdk/factory.go:98"}
{"level":"warn","ts":"2025-12-17T08:49:48.907Z","msg":"error creating sdk client","service":"worker","error":"failed reaching server: context deadline exceeded","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/common/sdk/factory.go:98"}
{"level":"error","ts":"2025-12-17T08:49:56.912Z","msg":"start failed","component":"fx","error":"context deadline exceeded","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:1151","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/log/zap_logger.go:151\ngo.temporal.io/server/temporal.(*fxLogAdapter).LogEvent\n\t/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:1151\ngo.uber.org/fx.(*App).Start.func1\n\t/home/runner/go/pkg/mod/go.uber.org/fx@v1.23.0/app.go:664\ngo.uber.org/fx.(*App).Start\n\t/home/runner/go/pkg/mod/go.uber.org/fx@v1.23.0/app.go:672\ngo.temporal.io/server/temporal.(*ServerImpl).startServices\n\t/home/runner/work/docker-builds/docker-builds/temporal/temporal/server_impl.go:134\ngo.temporal.io/server/temporal.(*ServerImpl).Start\n\t/home/runner/work/docker-builds/docker-builds/temporal/temporal/server_impl.go:100\ngo.uber.org/fx/internal/lifecycle.(*Lifecycle).runStartHook\n\t/home/runner/go/pkg/mod/go.uber.org/fx@v1.23.0/internal/lifecycle/lifecycle.go:256\ngo.uber.org/fx/internal/lifecycle.(*Lifecycle).Start\n\t/home/runner/go/pkg/mod/go.uber.org/fx@v1.23.0/internal/lifecycle/lifecycle.go:216\ngo.uber.org/fx.(*App).start-fm.(*App).start.func1\n\t/home/runner/go/pkg/mod/go.uber.org/fx@v1.23.0/app.go:704\ngo.uber.org/fx.(*App).withRollback\n\t/home/runner/go/pkg/mod/go.uber.org/fx@v1.23.0/app.go:686\ngo.uber.org/fx.(*App).start\n\t/home/runner/go/pkg/mod/go.uber.org/fx@v1.23.0/app.go:703\ngo.uber.org/fx.withTimeout.func1\n\t/home/runner/go/pkg/mod/go.uber.org/fx@v1.23.0/app.go:803"}
{"level":"error","ts":"2025-12-17T08:49:56.912Z","msg":"OnStart hook failed","component":"fx","callee":"go.temporal.io/server/temporal.(*ServerImpl).Start-fm()","caller":"go.temporal.io/server/temporal.ServerLifetimeHooks","error":"failed to start service worker: context deadline exceeded","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:1043","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/log/zap_logger.go:151\ngo.temporal.io/server/temporal.(*fxLogAdapter).LogEvent\n\t/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:1043\ngo.uber.org/fx.appLogger.LogEvent\n\t/home/runner/go/pkg/mod/go.uber.org/fx@v1.23.0/app.go:831\ngo.uber.org/fx/internal/lifecycle.(*Lifecycle).runStartHook.func1\n\t/home/runner/go/pkg/mod/go.uber.org/fx@v1.23.0/internal/lifecycle/lifecycle.go:247\ngo.uber.org/fx/internal/lifecycle.(*Lifecycle).runStartHook\n\t/home/runner/go/pkg/mod/go.uber.org/fx@v1.23.0/internal/lifecycle/lifecycle.go:257\ngo.uber.org/fx/internal/lifecycle.(*Lifecycle).Start\n\t/home/runner/go/pkg/mod/go.uber.org/fx@v1.23.0/internal/lifecycle/lifecycle.go:216\ngo.uber.org/fx.(*App).start-fm.(*App).start.func1\n\t/home/runner/go/pkg/mod/go.uber.org/fx@v1.23.0/app.go:704\ngo.uber.org/fx.(*App).withRollback\n\t/home/runner/go/pkg/mod/go.uber.org/fx@v1.23.0/app.go:686\ngo.uber.org/fx.(*App).start\n\t/home/runner/go/pkg/mod/go.uber.org/fx@v1.23.0/app.go:703\ngo.uber.org/fx.withTimeout.func1\n\t/home/runner/go/pkg/mod/go.uber.org/fx@v1.23.0/app.go:803"}
{"level":"error","ts":"2025-12-17T08:49:56.912Z","msg":"start failed, rolling back","component":"fx","error":"failed to start service worker: context deadline exceeded","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:1144","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/log/zap_logger.go:151\ngo.temporal.io/server/temporal.(*fxLogAdapter).LogEvent\n\t/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:1144\ngo.uber.org/fx.(*App).withRollback\n\t/home/runner/go/pkg/mod/go.uber.org/fx@v1.23.0/app.go:687\ngo.uber.org/fx.(*App).start\n\t/home/runner/go/pkg/mod/go.uber.org/fx@v1.23.0/app.go:703\ngo.uber.org/fx.withTimeout.func1\n\t/home/runner/go/pkg/mod/go.uber.org/fx@v1.23.0/app.go:803"}
{"level":"error","ts":"2025-12-17T08:49:56.912Z","msg":"start failed","component":"fx","error":"failed to start service worker: context deadline exceeded","logging-call-at":"/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:1151","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/runner/work/docker-builds/docker-builds/temporal/common/log/zap_logger.go:151\ngo.temporal.io/server/temporal.(*fxLogAdapter).LogEvent\n\t/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:1151\ngo.uber.org/fx.(*App).Start.func1\n\t/home/runner/go/pkg/mod/go.uber.org/fx@v1.23.0/app.go:664\ngo.uber.org/fx.(*App).Start\n\t/home/runner/go/pkg/mod/go.uber.org/fx@v1.23.0/app.go:672\ngo.temporal.io/server/temporal.(*ServerFx).Start\n\t/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:300\nmain.buildCLI.func4\n\t/home/runner/work/docker-builds/docker-builds/temporal/cmd/server/main.go:229\ngithub.com/urfave/cli/v2.(*Command).Run\n\t/home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.27.5/command.go:276\ngithub.com/urfave/cli/v2.(*Command).Run\n\t/home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.27.5/command.go:269\ngithub.com/urfave/cli/v2.(*App).RunContext\n\t/home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.27.5/app.go:333\ngithub.com/urfave/cli/v2.(*App).Run\n\t/home/runner/go/pkg/mod/github.com/urfave/cli/v2@v2.27.5/app.go:307\nmain.main\n\t/home/runner/work/docker-builds/docker-builds/temporal/cmd/server/main.go:30\nruntime.main\n\t/opt/hostedtoolcache/go/1.25.0/x64/src/runtime/proc.go:285"}
Unable to start server. Error: failed to start service worker: context deadline exceeded