In the latest release of the Dell CSM module “replication” support was added for the PowerFlex storage platform. In this post I’ll be describing how that works, and of course there is a demo video added as well (check out the bottom of this post)!
Dell CSM
The Dell Container Storage Modules or CSM add functionality to Kubernetes in order to make Kubernetes more feature-rich and enterprise-ready. Today we will be focusing on Dell CSM Replication which in its latest release (version 1.6.0) adds support for PowerFlex, and of course I just had to put that to the test 🙂
Topology
The topology we use for today’s demo is a dual datacenter setup. In each datacenter we have a vanilla (1.25.6) Kubernetes cluster with the PowerFlex CSI driver installed. This allows the two clusters to get presistent storage from two PowerFlex instances also running in those datacenters. By adding replication to the configuration of the PowerFlex systems we can now also install the Dell CSM Replication sidecar on both sides to allow Kubernetes to “understand” storage replication:
Read more: Dell CSM adds replication for PowerFlex in latest release
With this out of the way, there is nothing stopping us from deploying a demo app on the left datacenter, have that replicated to the right datacenter, and ultimately we failover the application to commence functioning from the right datacenter.
Building out the demo app “Yelb”
I really like this tiny application called “Yelb”. The original can be found here, and I’ve modified it to use stateful sets. Just using a simple piece of yaml code allows us to deploy the Yelb application to the left datacenter:
[root@myhost yelb]# kubectl create -f yelb.yaml
service/redis-server created
service/yelb-db created
service/yelb-appserver created
service/yelb-ui created
deployment.apps/yelb-ui created
statefulset.apps/redis-server created
statefulset.apps/yelb-db created
deployment.apps/yelb-appserver created
persistentvolumeclaim/redis-1 created
persistentvolumeclaim/pg-sql-1 created
[root@myhost yelb]# kubectl get pods
NAME READY STATUS RESTARTS AGE
redis-server-0 1/1 Running 0 61s
yelb-appserver-85fd4c58b4-vqdff 1/1 Running 0 61s
yelb-db-0 1/1 Running 0 61s
yelb-ui-768d9b964c-qhp5n 1/1 Running 0 61s
[root@myhost yelb]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pg-sql-1 Bound k8s04-1915d0433d 8Gi RWO vxflexos-replication 65s
redis-1 Bound k8s04-5d2f064444 8Gi RWO vxflexos-replication 65s
[root@myhost yelb]#as you can see, this little app deploys a redis and SQL instance (statefulSets) and an appserver with a UI (deployments).
A storage-view on a replicated cloud-native app
As you can see, the yelb app consists of two deployments and two stateful sets who each have their own persistent volume. On the PowerFlex in the left datacenter you can now see the two persistent volumes:
So this is normal CSI driver behavior. Where it gets interesting is where the PowerFlex now also has a Replication Consistency Group or RCG created by CSM Replication and replication has automatically been commenced to the array in the other datacenter:
Finally on the array in the right datacenter we can see the volumes that are getting replicated from the left side:
What we can see in the target Kubernetes cluster
The target kubernetes cluster was made aware of the two replicated volumes by the Dell CSM Replication module automatically. When we look at any Persistent Volumes (PV’s) we see two have been created for us:
[root@myhost yelb]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS STORAGECLASS AGE
k8s04-1915d0433d 8Gi RWO Delete Available vxflexos-replication 17m
k8s04-5d2f064444 8Gi RWO Delete Available vxflexos-replication 17m
[root@myhost yelb]#If we want to be able to failover, we need a means of talking to the CSM replication module. We do this through an API, and use the repctl utility to make things easy. Here you can clearly see that repctl has a clear undertstanding where the replicated volumes came from originally, including their (remote)PVC names etc.
[root@myhost yelb]# repctl get pv --clusters k8s05
[2023-03-30 10:02:33] INFO listing persistent volumes
[2023-03-30 10:02:33] INFO k8s05:
[2023-03-30 10:02:33] INFO Cluster: k8s05
+---------------------------------+
| PersistentVolume |
+---------------------------------+
Name RG SC rPVC rPVCNamespace
k8s04-1915d0433d rg-aad57ac2-d1813243d3bb vxflexos-replication pg-sql-1 default
k8s04-5d2f064444 rg-aad57ac2-d1813243d3bb vxflexos-replication redis-1 default
[2023-03-30 10:02:33] INFO
[root@myhost yelb]#Failing over the application
In order to perform a failover of the application to the other datacenter, we need to do four things:
- Break replication between the two datacenters
- Reverse replication (if possible – in case of a disaster this might not be possible)
- (Re)create the PVCs on the right datacenter
- Run the application on the right datacenter
Doing all of this is really easy:
[root@myhost yelb]# repctl --rg rg-aad57ac2-d1813243d3bb failover --target k8s05
[2023-03-30 10:11:41] INFO RG (rg-aad57ac2-d1813243d3bb), successfully updated with action: failover
[root@myhost yelb]# repctl --rg rg-aad57ac2-d1813243d3bb reprotect --at k8s05
[2023-03-30 10:12:06] INFO RG (rg-aad57ac2-d1813243d3bb), successfully updated with action: reprotect
[root@myhost yelb]# repctl create pvc --rg rg-aad57ac2-d1813243d3bb --clusters k8s05 -t default
[2023-03-30 10:15:09] INFO
Cluster: k8s05
+---------------------------------+
| PersistentVolume |
+---------------------------------+
Name RG SC rPVC rPVCNamespace
k8s04-1915d0433d rg-aad57ac2-d1813243d3bb vxflexos-replication pg-sql-1 default
k8s04-5d2f064444 rg-aad57ac2-d1813243d3bb vxflexos-replication redis-1 default
[2023-03-30 10:15:09] INFO
[2023-03-30 10:15:09] INFO Proceed with creation of PVCs [y/n]:
y
[2023-03-30 10:15:14] INFO Creating persistent volume claims
[2023-03-30 10:15:14] INFO Dry-Run: false. Successfully created PVC with name: pg-sql-1 using PV: k8s04-1915d0433d in the namespace: default
[2023-03-30 10:15:14] INFO Dry-Run: false. Successfully created PVC with name: redis-1 using PV: k8s04-5d2f064444 in the namespace: default
[root@myhost yelb]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pg-sql-1 Bound k8s04-1915d0433d 8Gi RWO vxflexos-replication 3m33s
redis-1 Bound k8s04-5d2f064444 8Gi RWO vxflexos-replication 3m33s
[root@myhost yelb]#With three repctl commands we managed to tackle steps one through three! If we now look at the PowerFlex systems again, we can verify that indeed storage replication was reversed:
Now all that remains is to start the Yelb app on the right datacenter! Please note the errors below: This is actually an expected outcome; as we already have PVC available from the replication, it makes sense that the yaml code cannot recreate the storage. This is exactly what we want 🙂
[root@myhost yelb]# kubectl create -f yelb.yaml
service/redis-server created
service/yelb-db created
service/yelb-appserver created
service/yelb-ui created
deployment.apps/yelb-ui created
statefulset.apps/redis-server created
statefulset.apps/yelb-db created
deployment.apps/yelb-appserver created
Error from server (AlreadyExists): error when creating "yelb.yaml": persistentvolumeclaims "redis-1" already exists
Error from server (AlreadyExists): error when creating "yelb.yaml": persistentvolumeclaims "pg-sql-1" already exists
[root@myhost yelb]#Successfully failed over a cloud-native application to the other datacenter
using PowerFlex replication 🙂
If you want to watch the full demo video you can see it here: