Fix ScyllaDB startup

Written by haloboy777 on 2024-11-04

How to fix "Failed to start a Raft group"

I've been getting the following error when trying to start a ScyllaDB cluster on my local machine:

node2  | ERROR 2024-11-03 23:01:03,249 [shard 0:main] init - Startup failed: seastar::internal::backtraced<std::runtime_error> (Failed to start a Raft group 22e5a340-996b-11ef-a7df-a3a43d2a3037: seastar::internal::backtraced<std::runtime_error> (topology[0xe00004c506a0]: node=0xe00004b6f6c0  idx=2 host_id=329b6bdd-3b84-4525-914d-d244cdcec46c endpoint=172.18.0.4 dc=datacenter1 rack=rack1 state=normal shards=1 this_node=false: node endpoint already mapped to node=0xe00004c58240  idx=0 host_id=7e833809-c052-449d-bbc0-b0b51188145f endpoint=172.18.0.4 dc=datacenter1 rack=rack1 state=normal shards=1 this_node=true Backtrace: 0x5325257 0x53257e7 0x4a7d15b 0x4a7ce97 0x517e70b 0x372c547 0x37303fb 0x3733b13 0x36dc667 0x375cbdf 0x390618f 0x38072d3 0x51b6f7b 0x51b8427 0x51b91ff 0x51b889b 0x514f5f3 0x514ead7 0x121b927 0x121cf3f 0x121a3f3 /opt/scylladb/libreloc/libc.so.6+0x30a1b /opt/scylladb/libreloc/libc.so.6+0x30afb 0x1217faf

Looking at the error message, I realized the issue was due to a node endpoint already mapped to another node.

Specifically, the error message says:

Failed to start a Raft group

and

node endpoint already mapped to node=0xe00004c58240  idx=0 host_id=7e833809-c052-449d-bbc0-b0b51188145f endpoint=

To fix it, I removed the existing node mappings and restarted the ScyllaDB cluster.

Fixing ScyllaDB startup issue

Step 1: Stop the ScyllaDB cluster using the following command:
```
docker-compose down
```

Step 2: Update docker-compose.yml file to use static IP for the nodes, which were available to the nodes when the cluster was started for the first time. For me it was the following configuration:

services:
  node1:
    image: scylladb/scylla
    container_name: node1
    command: --overprovisioned 1 --smp 1 --reactor-backend=epoll
    networks:
      default:
        ipv4_address: 172.18.0.4
    volumes:
      - /<USER_HOME>/scylladb/data/node1:/var/lib/scylla  # Mount node1's data to /var/lib/scylla

  node2:
    image: scylladb/scylla
    container_name: node2
    command: --seeds=node1 --overprovisioned 1 --smp 1 --reactor-backend=epoll
    networks:
      default:
        ipv4_address: 172.18.0.3
    volumes:
      - /<USER_HOME>/scylladb/data/node2:/var/lib/scylla  # Mount node2's data to /var/lib/scylla

  node3:
    image: scylladb/scylla
    container_name: node3
    command: --seeds=node1 --overprovisioned 1 --smp 1 --reactor-backend=epoll
    networks:
      default:
        ipv4_address: 172.18.0.2
    volumes:
      - /<USER_HOME>/scylladb/data/node3:/var/lib/scylla  # Mount node3's data to /var/lib/scylla


networks:
  default:
    driver: bridge
    ipam:
    config:
      - subnet: 172.18.0.0/16
        gateway: 172.18.0.1

Step 3: Start the ScyllaDB cluster again using the following command:
```
docker-compose up -d
```

The ScyllaDB cluster started successfully after updating the node mappings and restarting the cluster, resolving the issue without any further errors.