System upgrade with compatible backing image manager image
- Deploy Longhorn. Then set
Concurrent Automatic Engine Upgrade Per Node Limit
to a positive value to enable volume engine auto upgrade.
- Create 2 backing images: a large one and a small one. Longhorn will start preparing the 1st file for both backing image immediately via launching backing image data source pods.
- Wait for the small backing image being ready in the 1st disk. Then create and attach volumes with the backing image.
- Wait for volumes attachment. Verify the backing image content then write random data in the volumes.
- Wait for the large backing image being ready in the 1st disk. Then create and attach one more volume with this large backing image.
- Before the large backing image is synced to other nodes and the volume becomes attached, upgrade the whole Longhorn system:
- A new engine image will be used.
- The default backing image manager image will be updated.
- The new longhorn manager is compatible with the old backing image manager.
- Wait for system upgrade complete. Then verify:
- All old backing image manager and the related pod will be cleaned up automatically after the current downloading is complete. And the existing backing image files won’t be removed.
- New default backing image manager will take over all backing image ownerships and show the info in the status map:
- All backing image files are ready on each nodes.
- All attached volumes still work fine without replica crash, and the content is correct in the volumes after the upgrade.
- The last volume get attached successfully without replica crash, and the content is correct.
- Verify volumes and backing images can be deleted.
System upgrade with incompatible backing image manager image
- Deploy Longhorn.
- Create a backing images. Wait for the backing image being ready in the 1st disk.
- Create and attach volumes with the backing image.
- Wait for volumes attachment. Verify the backing image content then write random data in the volumes.
- Upgrade the whole Longhorn system:
- The default backing image manager image will be updated.
- The new longhorn manager is not compatible with the old backing image manager.
- Wait for system upgrade complete. Then verify:
- All old incompatible backing image manager and the related pod will be cleaned up automatically.
- New default backing image manager will take over all backing image ownerships and show the info in the status map.
- All attached volumes still work fine without replica crash, and the content is correct in the volumes during/after the upgrade.
System upgrade with the same backing image manager image
- Deploy Longhorn.
- Create a backing images. Wait for the backing image being ready in the 1st disk.
- Create and attach volumes with the backing image. Wait for all disk files of the backing image being ready.
- Run
kubectl -n longhorn system get pod -w
in a separate session.
- Upgrade Longhorn manager but with the backing image manager image unchanged. (Actually we can mock this upgrade by removing all longhorn manager pods simultaneously.)
- After the longhorn manager pods restart, Verify there is no backing image data source pod launched for the backing image in the output of step4.
- Repeat step4 ~ step8 for 10 times.
Available test backing image URLs:
https://longhorn-backing-image.s3-us-west-1.amazonaws.com/parrot.qcow2
https://longhorn-backing-image.s3-us-west-1.amazonaws.com/parrot.raw
https://cloud-images.ubuntu.com/minimal/releases/focal/release-20200729/ubuntu-20.04-minimal-cloudimg-amd64.img
https://github.com/rancher/k3os/releases/download/v0.11.0/k3os-amd64.iso
The way to generate a longhorn-manager image with higher API version
- Download longhorn manager repo with command
git clone https://github.com/longhorn/longhorn-manager.git
.
- Increase the constant
CurrentBackingImageManagerAPIVersion
in longhorn-manager/engineapi/backing_image_manager.go
by 1.
- Run
make
to build a longhorn-manager image then push it to docker hub.
The way to generate a backing-image-manager image with higher API version
- Download backing image manager repo with command
git clone https://github.com/longhorn/backing-image-manager.git
.
- Increase the constants
BackingImageManagerAPIVersion
and BackingImageManagerAPIMinVersion
in backing-image-manager/pkg/meta/version.go
by 1.
- Run
make
to build a longhorn-manager image then push it to docker hub.