Test upgrade for migrated Longhorn on Rancher

https://github.com/longhorn/longhorn/discussions/4198

Context: since few customers used our broken chart longhorn 100.2.1+up1.3.1 on Rancher (Now fixed) with the workaround. We would like to verify the future upgrade path for those customers.

Steps

  1. Set up a cluster of Kubernetes 1.20.
  2. Adding this repo to the apps section in new rancher UI
    1. repo: https://github.com/PhanLe1010/charts.git
    2. branch: release-v2.6-longhorn-1.3.1.
  3. Access old rancher UI by navigating to <your-rancher-url>/g.
  4. Install Longhorn 1.0.2.
  5. Create/attach some volumes. Create a few recurring snapshot/backup job that run every minutes.
  6. Upgrade Longhorn to v1.2.4.
  7. Migrate Longhorn to new chart in new rancher UI https://longhorn.io/kb/how-to-migrate-longhorn-chart-installed-in-old-rancher-ui-to-the-chart-in-new-rancher-ui/.
  8. Upgrade to longhorn 100.2.1+up1.3.1 in the UI.
  9. Verify the upgrade would be stuck at with error: 
    Patch CustomResourceDefinition "volumes.longhorn.io" in namespace
error updating the resource "volumes.longhorn.io":
         cannot patch "volumes.longhorn.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "volumes.longhorn.io" is invalid: spec.conversion.strategy: Invalid value: "Webhook": must be None if spec.preserveUnknownFields is true
Error: UPGRADE FAILED: cannot patch "engineimages.longhorn.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "engineimages.longhorn.io" is invalid: spec.conversion.strategy: Invalid value: "Webhook": must be None if spec.preserveUnknownFields is true && cannot patch "nodes.longhorn.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "nodes.longhorn.io" is invalid: spec.conversion.strategy: Invalid value: "Webhook": must be None if spec.preserveUnknownFields is true && cannot patch "volumes.longhorn.io" with kind CustomResourceDefinition: CustomResourceDefinition.apiextensions.k8s.io "volumes.longhorn.io" is invalid: spec.conversion.strategy: Invalid value: "Webhook": must be None if spec.preserveUnknownFields is true
  10. Apply the workaround
  11. Retry upgrade to longhorn 100.2.1+up1.3.1 in the UI.
  12. Verify the upgrade is ok.
  13. Verify that the user can successfully upgrade to future chart releases (e.g., Longhorn v1.3.2) from this version longhorn 100.2.1+up1.3.1.
[Edit]