Troubleshoot deployments

​

Cluster stuck in updating: pods missing a role label

​

Symptoms

• The Cluster resource stays in phase Updating after all pods are Running and ready.
• One or more pods have no value in the kubeblocks.io/role label while their peers show primary or secondary:

kubectl get pods -n <namespace> -l app.kubernetes.io/instance=<cluster-name> -L kubeblocks.io/role

NAME                     READY   STATUS    AGE   ROLE
my-db-shard-abc-0        3/3     Running   17m
my-db-shard-abc-1        3/3     Running   17m   secondary

​

Diagnosis

1. Confirm the database itself is healthy. For a sharded cluster, each shard -0 pod should report master and cluster_state:ok:
   kubectl exec -n <namespace> <pod> -c falkordb-cluster -- sh -c \
     'redis-cli -a "$REDIS_DEFAULT_PASSWORD" --no-auth-warning role | head -1; \
      redis-cli -a "$REDIS_DEFAULT_PASSWORD" --no-auth-warning cluster info | grep cluster_state'
   
   For standalone, replicated, or Sentinel topologies, use the falkordb container name instead of falkordb-cluster.
2. Check that the role probe ran on the affected pod. The kbagent sidecar logs the probed role:
   kubectl logs -n <namespace> <pod> -c kbagent --tail 50 | grep roleProbe
   
   A healthy probe logs "output": "primary" (or secondary).
3. Check the KubeBlocks controller for a failed label update:
   kubectl logs -n kb-system -l app.kubernetes.io/component=apps --tail 400 | grep -iE '<cluster-name>.*(role|conflict|modified)'
   
   The signature of this case is an event reconcile error such as:
   Operation cannot be fulfilled on pods "<pod>": the object has been modified;
   please apply your changes to the latest version and try again
   

​

Root cause

​

Fix

kubectl label pod -n <namespace> <pod> kubeblocks.io/role=primary --overwrite

kubectl get cluster -n <namespace> <cluster-name> -o jsonpath='{.status.phase}'

Only set the label to the role the database actually reports. Labeling a replica as primary misroutes client traffic sent through the read-write Service.