1. Kubernetes

Dynamic provisioning with StorageClass

  1. Can create and use volume using StorageClass

  2. Can create a new StorageClass use new parameters and it will take effect on the volume created by the storage class.

  3. If the PV reclaim policy is delete, once PVC and PV are deleted, Longhorn volume should be deleted.

Static provisioning using Longhorn created PV/PVC

  1. PVC can be used by the new workload

  2. Delete the PVC will not result in PV deletion

  3. Delete the PV will not result in Longhorn volume deletion and data loss.

Test cases using kubectl

# Test Case Test Instructions Expected Results Automated ? / test name
1 Disable Volume Expansion for volumes dynamically provisioned by a new StorageClass



StorageClass example manifest		 1. Create a new StorageClass with allowVolumeExpansion: false
2. Create a PVC with using the new StorageClass		 * New Storage class should be created
* Volume should be dynamically provisioned, it’s PV/PVC should be Bound
* Volume expansion feature should be disabled for all Volumes using the new Storage Class
2 Number of volume replicas volumes dynamically provisioned by a new StorageClass 1. Create a new StorageClass, updatenumberOfReplicas parameter
2. Create a PVC with using the new StorageClass		 * New Storage class should be created
* Volume should be dynamically provisioned, it’s PV/PVC should be Bound
* Volume number of replicas should match numberOfReplicas specified in storagecalss parameter
3 StorageClass frombackup parameter Prerequisite:

* Longhorn setting is set for Backup Target and Backup Target Credential Secret
* Backup store should contain a previous volume backup
* volume data checksum should be know at the time of volume backups (checksum#1)

1. Create a StorageClass, and set frombackup parameter to volume backup URL in the backup store
2. Create a PVC using the new StorageClass, and it’s size should be the original volume size
3. Attach the new Volume to a node and check it’s data checksum		 * Volume should be dynamically provisioned, it’s PV/PVC should be Bound
* Restore from backup process should be triggered
* New volume should contain data restored from the backup, data checksum should match (checksum#1)		 test_storage_class_from_backup
4 StorageClass diskSelector parameter Prerequisite:

* Longhorn Disks should have tags

1. Create a new StorageClass, set diskSelector parameter
2. Create a PVC with using the new StorageClass		 * New Storage class should be created
* Volume should be dynamically provisioned, it’s PV/PVC should be Bound, volume replicas should only scheduled to Disks with tags that match diskSelector parameter tags
5 StorageClass nodeSelector parameter Prerequisite:

* Longhorn Node should have tags

1. Create a new StorageClass, set nodeSelector parameter
2. Create a PVC with using the new StorageClass		 * New Storage class should be created
* Volume should be dynamically provisioned, it’s PV/PVC should be Bound, volume replicas should only scheduled to Nodes with tags that match nodeSelector parameter tags
6 StorageClass recurringJobs parameter 1. Create a new StorageClass, set recurringJobs parameter
2. Create a PVC with using the new StorageClass
3. Create a pod that consumes the created PVC
4. Check Volume recurring jobs		 * New Storage class should be created
* Volume should be dynamically provisioned, it’s PV/PVC should be Bound and attached to the pod
* Volume should have recurring snapshots and backups matches ones specified in recurringJobs StorageClass parameter		 test_statefulset_recurring_backup
7 StorageClass with reclaimPolicy parameter set to Delete 1. Create a new StorageClass, set reclaimPolicy parameter to Delete
2. Create a PVC with using the new StorageClass
3. Delete the PVC		 * Volume should be dynamically provisioned, it’s PV/PVC should be Bound
* Deleting PVC would trigger Volume delete
8 StorageClass with reclaimPolicy parameter set to Retain 1. Create a new StorageClass, set reclaimPolicy parameter to Retain
2. Create a PVC with using the new StorageClass
3. Delete PVC
4. Delete PV		 * Volume should be dynamically provisioned, it’s PV/PVC should be Bound
* Deleting PVC and PV will not delete longhorn volume.		 test_kubernetes_status
9 Static provisioning using Default Longhorn Static StorageClass Name Setting 1. Update Default Longhorn Static StorageClass Name setting, set a new StorageClass Name, StorageClass doesn’t have to exist or be created.
2. Create a Volume
3. From Longhorn, Create a PV/PVC for the volume
4. Check created PV persistentVolumeReclaimPolicy: Retain
5. Create a pod consuming created PVC
6. Delete the pod
7. Delete PV
8. Delete PVC		 * Volume should be created
* Volume PV should be created using new StorageClass Name defined in Default Longhorn Static StorageClass Name setting
* PVC should be consumed by the pod, volume should be accessible in the pod, write/read operations should work normally.
* Deleting PV/PVC will not trigger volume delete.		 test_pvc_creation_with_default_sc_set

Additional Tests to be executed from Rancher

# Scenario Steps Expected Results
1 Storage Class: Create a Longhorn storage class Pre condition:

* Longhorn is deployed in the cluster

Steps:

1. Go to cluster → Storage → Storage Classes
2. Click on Add class
3. Select Provisioner Longhorn
4. Give in other required parameters including replica count.
5. Click on Save.
6. Verify test-1 storage class is created
7. Go to Cluster → Project (default) → Workloads
8. Deploy a workload
9. In the Volumes section → Add a New Volume Claim → Use a Storage Class to provision a new persistent volume → Select test-1 from Storage class dropdown.
10. Enter capacity and Name. Click on Define
11. Enter Mount Point.
12. Click on create workload
13. Verify workload is created successfully.
14. Volume gets attached to the pod in the workload
15. Navigate to Longhorn UI.
16. Verify user is able to view the volume attached to the workload in the UI
17. Navigate to volume details page of the volume and Verify the replica count mentioned in Step 4 is available		 * Longhorn storage class should be created
* Workload should be deployed with the volume mounted from the storage class
* Verify volume is available on the Longhorn UI.
* Verify the replica count is as mentioned during storage class creation.
2 Persistent Volume: Create a PV Pre condition:

* Longhorn is deployed in the cluster

Steps:

1. Create a Volume in Longhorn UI test-volume
2. Go to cluster → Storage → Persistent Volumes
3. Click on Add PV
4. Select Volume Plugin Longhorn
5. Give in other required parameters including replica count.
6. Give in Volume Plugin - test-volume which an existing volume in longhorn
7. Click on Save.
8. Verify test-1 PV is created
9. Go to Cluster → Project (default) → Workloads
10. Deploy a workload
11. In the Volumes section → Add a New Volume Claim → Use an existing persistent volume → Select test-1 from PV dropdown.
12. Click on Define
13. Enter Mount Point.
14. Click on create workload
15. Verify workload is created successfully.
16. Volume gets attached to the pod in the workload
17. Navigate to Longhorn UI.
18. Verify user is able to view the volume attached to the workload in the UI
19. Navigate to volume details page of the volume and Verify the replica count mentioned in Step 4 is available		 * Longhorn PV should be created
* Workload should be deployed with the volume mounted from the PV
* Verify volume is available on the Longhorn UI.
* Verify the replica count is as mentioned during storage class creation.
3 Create Storage class in Rancher; From Longhorn create volumes from this storage class. Pre condition:

* Longhorn is deployed in the cluster

Steps:

1. Go to cluster → Storage → Storage Classes
2. Click on Add class
3. Select Provisioner Longhorn
4. Give in other required parameters including replica count.
5. Click on Save.
6. Verify test-1 storage class is created
7. Go to Longhorn UI
8. In the Settings page for “Default Longhorn Static StorageClass Name”, give in the value: “test-1”
9. Go to Volumes page, click on create volume.
10. Create a volume name : v1
11. Verify v1 is created
12. using kubectl -
13. kubectl get pv -o yaml
14. Verify “storageClassName:” —> test-1		 * Longhorn storage class should be created
* Value of Default Longhorn Static StorageClass Name should be changed in the settings page
* volume should be created in longhorn UI
* “storageClassName:” value should be test-1
4 Create Storage Class using backup URL 1. Create volume and PV/PVC/POD in Longhorn
2. Write test_data into pod
3. Create a snapshot and back it up. Get the backup URL
4. Create a new StorageClass longhorn-from-backup in rancher and set backup URL.
5. Use longhorn-from-backup to create a new PVC
6. Wait for the volume to be created and complete the restoration.
7. Create the pod using the PVC. Verify the data
5 Create Storage class - by using different values for the input list of parameters Pre condition:

* Longhorn is deployed in the cluster

Steps:

1. Go to cluster → Storage → Storage Classes
2. Click on Add class
3. Select Provisioner Longhorn
4. Give in other required parameters.
5. Click on Save.
6. Use this storage class to create a PVC and deploy in a workload.
7. Verify the parameters of the volume created.		 Volume parameters should match the storage class parameters.
6 StorageClass with reclaimPolicy parameter set to Delete - PVC from storage class Pre conditions:

* Create PVC from “Longhorn” storage class in rancher.
* It will have a dynamic PV bound

Steps:

1. ‘Delete PVC from Rancher
2. Verify PVC is deleted
3. Verify PV bound to this PVC is deleted - Rancher → Cluster → Storage → PV
4. Verify the volume(Dynamic PV) in Longhorn is deleted
7 Volume/PV/PVC created in Longhorn Pre conditions:

* Create volume, PV, PVC in longhorn

Steps:

1. ‘Delete PVC from Rancher
2. Verify PVC is deleted
3. PV will NOT. be deleted but be in “released” state in Rancher UI
4. Verify Volume does not get deleted
8 StorageClass with reclaimPolicy parameter set to Retain - PVC from storage class Pre conditions:

* Create PVC from “Longhorn” storage class in rancher.
* It will have a dynamic PV bound

Steps:

1. ‘Delete PVC from Rancher
2. Verify PVC is deleted
3. Verify PV bound to this PVC is NOT deleted - Rancher → Cluster → Storage → PV
4. Verify the volume(Dynamic PV) in Longhorn is NOT deleted
9 StorageClass with reclaimPolicy parameter set to Retain - Volume/PV/PVC created in Longhorn Pre conditions:

* Create volume, PV, PVC in longhorn

Steps:

1. ‘Delete PVC from Rancher
2. Verify PVC is deleted
3. PV will NOT. be deleted but be in “released” state in Rancher UI
4. Verify Volume does not get deleted
10 Power down node 1. Power down
2. Replica migrates
3. Power back on
4. Verify if the replicas in the node have been deleted		 * When a node is powered down, the replica is rebuilt on the 4th worker node.
* When the node is powered back on, and the replica on the powered down node is not available in Longhorn UI anymore, there is no data in /var/lib/longhorn/replicas folder in the powered on node.
11 Power down node with. Node tag/disk tag 1. Add a node tag/disk tag
2. Power down
3. Replica cannot migrate
4. Power back on
5. Replica should get rebuilt on this node		 * When a node is powered down, the replica is rebuilt on the 4th worker node.
* When the node is powered back on, and the replica on the powered down node is not available in Longhorn UI anymore, there is no data in /var/lib/longhorn/replicas folder in the powered on node.
* The new replica is rebuilt on a node which has a tag.
12 Drain a node 1. Drain use case — drain a worker node 
2. Check if the State of the node reflects in the Longhorn UI —> Node
3. Verify if replica is rebuilt on another node? 
4. Verify if the pod migrates
5. And the volume get migrated		 All the components should be successfully drained.
13 kubectl - force drain Using kubectl - force drain a node where the pod with the volume attached is available

Have snapshots before

Verify data after pod migrates		 Volume attaches on the new pod

2 of the 3 replicas are in “Stopped” state - Caused replica rebuild.
14 Cordon a node 1. Cordon state - cordon a worker node
15 Delete node where the pods/workload exists

workload type: deployment		 Verify the pods migrate to another node and verify the volume also re attaches on the pod on the other node * Create a workload with volume attached on n1
* write data to volume
* Delete node n1
* The workload gets reattached to another node n2.
* The volume gets attached after a minute
* The volume is accessible.
* data is accessible
16 Power down node where the pods/workload exists

workload type: deployment		 Verify the pods migrate to another node and verify the volume also re attaches on the pod on the other node host A → Pod a – Unknown – > Not able to unmount

host b → pod b - Creating – fails to attach here
17 Delete node where the pods/workload exists

workload type: stateful set		 * Create a workload, scale - 2 with volume attached on n1
* write data to volume
* Delete node n1, and n2		 * The workload gets reattached to another node n3 and n4.
* The volume gets attached after sometime
* The volume is accessible.
* data is accessible
18 Power down node where the pods/workload exists

workload type: stateful set		 * Create a workload, attach to a volume.
* Write some data
* Power down the node where the pod is running.		 * The pod should get recreated on another node.
* The volume should get reattached.
* The mount point should be accessible to read and write.
19 Delete worker node one by one * The volume should get reattach to healthy node.
20 Upgrade cluster - drain set - false 1. In Rancher - upgrade a cluster by changing the max-pods value
2. The cluster will go into “Updating” state
3. Verify upgrade completes successfully		 Upgrade should finish successfully
21 Upgrade cluster - drain set - true 1. In Rancher - upgrade a cluster by changing the max-pods value
2. The cluster will go into “Updating” state
3. Verify upgrade completes successfully		 Upgrade should finish successfully
[Edit]