how to migrate zookeeper
# https://gist.github.com/iameugenejo/d101d01e1002b95076d5b762d4b3405d
# migrating 3-node zookeeper cluster to a new 3-node cluster
# o1, o2, o3 = old zk nodes, assume o1 is the leader
# n1, n2, n3 = new zk nodes
# quorum: minimum number of healhty nodes
# 6-node cluster's quorum is 4
# 5-node cluster's quorum is 3
# 4-node cluster's quorum is 3
# 3-node cluster's quorum is 2
1. Make 3 new nodes to join the cluster one at a time, and let them complete replication:
- add n1 with (o1, o2, o3, n1) configured
- add n2 with (o1, o2, o3, n1, n2) configured
- update n1 with (o1, o2, o3, n1, n2) configuration, restart
- add n3 with (o1, o2, o3, n1, n2, n3) configured
- update n1 with (o1, o2, o3, n1, n2, n3) configuration, restart
- update n2 with (o1, o2, o3, n1, n2, n3) configuration, restart
- at this point, from o1, o2, o3's perspective, it's 3-node cluster with quorum count of 2 and 3 healhty nodes
- from n1, n2, n3's perspective, it's 6-node cluster with quorum count of 3 and 6 healhty nodes.
- from the clients' perspective, it's still connected to the old 3-node cluster
2. update the clients that use the zookeeper cluster with (n1, n2, n3).:
- from the clients' perspective, now the clients are connected to the 6-node cluster.
3. update o2, o3 with (o1, o2, o3, n1, n2, n3) and restart:
- from o1's perspective it's still a 3-node cluster
- from the rest of the nodes, it's a 6-node cluster
4. kill o1 to cause the leader re-election:
- 3-node cluster is now gone
- 6-node cluster still meets the quorum with 5 healhty nodes (5 >= 4).
- Note: somehow zookeeper gets stuck at this point when one of the new nodes become a leader (try echo "get /zookeeper/config" | zookeeper-shell localhost:2181")
- when it gets stuck, restart all new zookeeper nodes (n1, n2, 3)
5. update every nodes with (o2, o3, n1, n2, n3) configured, one at a time:
- the intermediate state with the 6-node cluster meets the quorum with 5 healhty nodes (5 >= 3)
- the final state with the 5-node cluster meets the quorum with 4 healhty nodes (5 >= 3)
6. kill o2 and update every nodes with (o3, n1, n2, n3) configured, one at a time:
- the intermediate state with the 5-node cluster meets the quorum with 4 healhty nodes (4 >= 3)
- the final state with the 4-node cluster meets the quorum with 4 healhty nodes (4 >= 3)
7. kill o3 and update every nodes with (n1, n2, n3) configured, one at a time:
- the intermediate state with the 4-node cluster meets the quorum with 3 healhty nodes (3 >= 2)
- the final state with the 3-node cluster meets the quorum with 3 healhty nodes (3 >= 2)