Styjun

I'm trying to setup Cassandra across 2 datacenters on Azure VMs as docker containers. I was going by the guide located here: https://gokhanatil.com/2018/02/build-a-cassandra-cluster-on-docker.html. What I don't understand, however, is that the guide seems to spawn cas2 and cas3 nodes on the same VM, despite specifying a different datacenter. Wouldn't they physically still be at the same location despite claiming a different datacenter?

What I tried to do is to launch cas3 node on a different VM, while specifying public IP of the seed node in the first datacenter (cas1). I also whitelisted public IPs and Cassandra ports (7000, 7001, 7199, 9042, 9160, 9142) on both of these VMs in Azure UI so they could see each other.

cas1 and cas2 have been spawned on the same VM as separate containers, can see eachother and are both up, as the guide claims:

admin@cassandra-1:~$ sudo docker exec -ti cas1 nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.17.0.3 295.15 KiB 256 49.2% c79d0871-d4a3-4735-a0e0-80304009989d rack1
UN 172.17.0.2 263.4 KiB 256 50.8% 5990e65d-9a69-4fe1-8879-100382f38349 rack1

cas3 that I spawn on VM in different region, specifying CASSANDRA_SEEDS matching Azure's external IP of cassandra-1 VM keeps dying with the following error:

Exception (java.lang.RuntimeException) encountered during startup: Unable to gossip with any peers
java.lang.RuntimeException: Unable to gossip with any peers
 at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1435)
 at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:566)
 at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:823)
 at org.apache.cassandra.service.StorageService.initServer(StorageService.java:683)
 at org.apache.cassandra.service.StorageService.initServer(StorageService.java:632)
 at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:388)
 at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:620)
 at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:732)
ERROR [main] 2019-03-28 21:01:19,890 CassandraDaemon.java:749 - Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any peers
 at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1435) ~[apache-cassandra-3.11.4.jar:3.11.4]
 at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:566) ~[apache-cassandra-3.11.4.jar:3.11.4]
 at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:823) ~[apache-cassandra-3.11.4.jar:3.11.4]
 at org.apache.cassandra.service.StorageService.initServer(StorageService.java:683) ~[apache-cassandra-3.11.4.jar:3.11.4]
 at org.apache.cassandra.service.StorageService.initServer(StorageService.java:632) ~[apache-cassandra-3.11.4.jar:3.11.4]
 at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:388) [apache-cassandra-3.11.4.jar:3.11.4]
 at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:620) [apache-cassandra-3.11.4.jar:3.11.4]
 at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:732) [apache-cassandra-3.11.4.jar:3.11.4]
INFO [StorageServiceShutdownHook] 2019-03-28 21:01:19,914 HintsService.java:209 - Paused hints dispatch
WARN [StorageServiceShutdownHook] 2019-03-28 21:01:19,914 Gossiper.java:1559 - No local state, state is in silent shutdown, or node hasn't joined, not announcing shutdown
INFO [StorageServiceShutdownHook] 2019-03-28 21:01:19,914 MessagingService.java:981 - Waiting for messaging service to quiesce
INFO [ACCEPT-/172.17.0.2] 2019-03-28 21:01:19,915 MessagingService.java:1336 - MessagingService has terminated the accept() thread
INFO [StorageServiceShutdownHook] 2019-03-28 21:01:23,224 HintsService.java:209 - Paused hints dispatch

I used the following commands to create the 2 containers on the first VM:

docker run --name cas1 -p 9042:9042 -e CASSANDRA_CLUSTER_NAME=cluster1 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_DC=datacenter1 -d cassandra
docker run --name cas2 -e CASSANDRA_SEEDS="$(docker inspect --format=' .NetworkSettings.IPAddress ' cas1)" -e CASSANDRA_CLUSTER_NAME=cluster1 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_DC=datacenter1 -d cassandra

And the following command on the 2nd VM where the container keeps failing to connect:

docker run --name cas3 -e CASSANDRA_SEEDS="[cassandra-1-external-ip]" -e CASSANDRA_CLUSTER_NAME=cluster1 -e CASSANDRA_ENDPOINT_SNITCH=GossipingPropertyFileSnitch -e CASSANDRA_DC=datacenter2 -d cassandra

I did not specify either CASSANDRA_LISTEN_ADDRESS or CASSANDRA_BROADCAST_ADDRESS since the instructions claim that default will work for most cases (https://hub.docker.com/_/cassandra). Is that not the case when using multi-datacenter setup?

For your issue, if you just run the Cassandra on different VMs, not with Docker, then it's a case when using multi-datacenter on different nodes. But you run the Cassandra on Docker. Then it comes into a problem that containers communicate through multi-host.

Communicate through multi-host for Docker containers, it's a complex problem. You should create a Docker cluster, then containers can communicate through multi-host. For you, it means Cassandra can communicate through different data center on different VMs.

I suggest you can try to run the Cassandra on AKS cluster with multiple nodes.