So excited to see this: Witness support on Metro for the PowerStore Platform! I recorded a quick video to show you why that matters.
Powerstore Metro
Dell has has metro support embedded into the Powerstore platform for some time now, but lacked support of a witness. I often get into discussions where people tend to think that not having a witness can cause split-brain scenarios. Actually it won’t, but there are some implications of not having a witness in a metro cluster.
Powerstore Metro has what we call a “preferred side” and a “non-preferred side”. By default one array is always the preferred one, but on a volume-based level you can actually change that as well, so I could have 50% of my volumes “preferred left” while the other 50% is “preferred right”.
Next I would have to stretch my workloads over the two Powerstore locations (which we will call DC-A and DC-B). In the video I will use a stretched vSphere cluster. Within the cluster I define two groups of hosts: Hosts that below to DC-A and hosts that belong to DC-B. Next I add two VM group with rules that will tell the VMs in those groups “Should run on DC-A” and “Should run on DC-B” respectively. Sidedness: CHECK 🙂
Life without a Witness service
The stretched cluster with the metro volumes underneath will work, most importantly and they won’t ever split-brain. Why? Well this is where “preferred A and B” comes in.
As soon as a Powerstore looses contact with its counterpart, it will take action. Since it is unable to do a proper root-cause analysis (for now we will just consider the two cases of 1) interlink failure and 2) Array failure), it will proceed with degrading all “non-preferred” volumes to read-only. Both sides will behave the same: Anything non-preferred goes read-only. No split brain.
But was this the proper action? Well, in case of an interlink failure, yes. As both Powerstores continue to live in their respective datacenters, both can do read/writes. To avoid split brain the non-preferred volumes should indeed go into read-only.
But what if one of the Powerstores failed? In this case it would have been safe for the “surviving” array to continue to serve I/O, even on the non-preferred volumes as the preferred side is now gone anyway. And THIS is the reason you need a better analysis of the situation. Enter Witness services!
Life with a Witness service
By adding a witness to Powerstore Metro, the metro config is better capable of determining what happened and how to take action.
Looking at the same two failure scenarios:
- Interlink failure: Both Powerstores won’t see each other, but they will see the Witness. They can now determine that both sides are still alive, and non-preferred volumes have to be degraded into a read-only state to avoid split brain.
- Array failure: The surviving array sees the remote array disappear. It now asks the witness “what happened to the remote Powerstore?”. The answer is “I do not see it either, I declare it down and you are the sole survivor”. The surviving Powerstore now safely resumes I/O, even for the non-preferred volumes
See the difference? Powerstore can now recover from “preferred side” failures! The stretched cluster on top can now restart any failed workloads, even if they used to run on the preferred side. They can be safely restarted on their “non preferred” side, as the volumes there remain R/W enabled… All thanks to the Witness.
Installing and configuring a Powerstore Witness service
It is very easy to get this installed. I just grabbed a Linux VM, downloaded the Witness RPM, installed the service. Next you generate a token and then you are ready to add the witness into Powerstore:
[root@localhost ~]# cd /opt/dell-witness-service/scripts
[root@localhost scripts]# ls
data_collect.sh generate_token.sh replace_certificate.sh system thumbprint.sh uninstall.sh
[root@localhost scripts]# ./generate_token.sh
The generated token is: NZ6Nw6Ps
[root@localhost scripts]#All done, lets configure the Witness in Powerstore:
Pretty soon (after checking the connectivity etc), you’ll see the witness added and ready to go:
