> ## Documentation Index
> Fetch the complete documentation index at: https://docs.enterprise.falkordb.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshoot deployments

> Diagnose and fix common issues with FalkorDB deployments managed by KubeBlocks.

## Cluster stuck in updating: pods missing a role label

### Symptoms

* The Cluster resource stays in phase `Updating` after all pods are `Running` and ready.
* One or more pods have no value in the `kubeblocks.io/role` label while their peers show `primary` or `secondary`:

```bash theme={null}
kubectl get pods -n <namespace> -l app.kubernetes.io/instance=<cluster-name> -L kubeblocks.io/role
```

```text theme={null}
NAME                     READY   STATUS    AGE   ROLE
my-db-shard-abc-0        3/3     Running   17m
my-db-shard-abc-1        3/3     Running   17m   secondary
```

### Diagnosis

1. Confirm the database itself is healthy. For a sharded cluster, each shard `-0` pod should report `master` and `cluster_state:ok`:

   ```bash theme={null}
   kubectl exec -n <namespace> <pod> -c falkordb-cluster -- sh -c \
     'redis-cli -a "$REDIS_DEFAULT_PASSWORD" --no-auth-warning role | head -1; \
      redis-cli -a "$REDIS_DEFAULT_PASSWORD" --no-auth-warning cluster info | grep cluster_state'
   ```

   For standalone, replicated, or Sentinel topologies, use the `falkordb` container name instead of `falkordb-cluster`.

2. Check that the role probe ran on the affected pod. The kbagent sidecar logs the probed role:

   ```bash theme={null}
   kubectl logs -n <namespace> <pod> -c kbagent --tail 50 | grep roleProbe
   ```

   A healthy probe logs `"output": "primary"` (or `secondary`).

3. Check the KubeBlocks controller for a failed label update:

   ```bash theme={null}
   kubectl logs -n kb-system -l app.kubernetes.io/component=apps --tail 400 | grep -iE '<cluster-name>.*(role|conflict|modified)'
   ```

   The signature of this case is an event reconcile error such as:

   ```text theme={null}
   Operation cannot be fulfilled on pods "<pod>": the object has been modified;
   please apply your changes to the latest version and try again
   ```

### Root cause

The kbagent role probe reports the role once and re-emits it only when the role changes. If the KubeBlocks event controller loses an optimistic-concurrency conflict while writing the `kubeblocks.io/role` pod label (common during post-provision churn, when several controllers update the same pod), the write is dropped and never retried. The database is healthy; only the label is missing, which keeps the Cluster in `Updating`.

### Fix

Apply the label the controller failed to write, matching the actual role reported by the database in the diagnosis step:

```bash theme={null}
kubectl label pod -n <namespace> <pod> kubeblocks.io/role=primary --overwrite
```

The Cluster transitions to `Running` as soon as all pods carry a role label:

```bash theme={null}
kubectl get cluster -n <namespace> <cluster-name> -o jsonpath='{.status.phase}'
```

Alternatively, deleting the affected pod also resolves it: the replacement pod is probed fresh and labeled on startup. Prefer re-labeling, since it avoids a failover and is instantaneous.

> Only set the label to the role the database actually reports. Labeling a replica as `primary` misroutes client traffic sent through the read-write Service.