Module tests.test_recurring_job

Functions

def add_recurring_job_source_to_pvc(name, core_api)
def recurring_job_labels_test(client, labels, volume_name, size='16777216', backing_image='')
def remove_recurring_job_source_to_pvc(name, core_api)
def test_recurring_job(set_random_backupstore, client, volume_name)

Scenario : test recurring job (S3/NFS)

Given snapshot1 recurring job created and cron at 1 min and retain 2. backup1 recurring job created and cron at 2 min and retain 1. backup2 recurring job created and cron at 1 min and retain 2. And a volume created and attached.

When label volume with recurring job snapshot1. label volume with recurring job backup1. And wait until the 20th second since the beginning of an even minute. And write data to volume. wait for 2 minutes. And write data to volume. wait for 2 minutes. Then volume have 4 snapshots. (2 from snapshot1, 1 from backup1, 1 from volume-head)

When label volume with recurring job backup2 And write data to volume. wait for 2 minutes. And write data to volume. wait for 2 minutes. Then volume have 5 snapshots. (2 from snapshot1, 1 from backup1, 1 from backup2, 1 from volume-head)

When wait until backups complete. Then backup1 completed 1 backups. backup2 completed 2 backups.

def test_recurring_job_backup(set_random_backupstore, client, batch_v1_api)

Scenario: test recurring job backup (S3/NFS)

Given volume test-job-1 created, attached, and healthy. volume test-job-2 created, attached, and healthy.

When create a recurring job with default in groups. Then should have 1 cron job.

When write some data to volume test-job-1. write some data to volume test-job-2. And wait for backup1 cron job scheduled time. Then volume test-job-1 should have 1 backups. volume test-job-2 should have 1 backups.

When write some data to volume test-job-1. write some data to volume test-job-2. And wait for backup1 cron job scheduled time. Then volume test-job-1 should have 2 backups. volume test-job-2 should have 2 backups.

def test_recurring_job_default(client, batch_v1_api, volume_name)

Scenario: test recurring job set with default in groups

Given 1 volume created, attached, and healthy.

Setting recurring job in volume label should not remove the defaults.

When set snapshot recurring job in volume label. Then should contain default job-group in volume labels. should contain snapshot job in volume labels.

Should be able to remove the default label.

When delete recurring job-group default in volume label. Then volume should have snapshot job in job label. volume should not have default group in job label.

Remove all volume recurring job labels should bring in default

When delete all recurring jobs in volume label. Then volume should not have snapshot job in job label. volume should have default group in job label.

def test_recurring_job_delete(client, batch_v1_api, volume_name)

Scenario: test delete recurring job

Given 1 volume created, attached, and healthy.

When create snapshot1 recurring job with default, group-1 in groups. create snapshot2 recurring job with default in groups.. create snapshot3 recurring job with in groups. create <code>backup1</code> recurring job with `default, group-1` in groups. create <code>backup2</code> recurring job with <code>default</code> in groups. create <code>backup3</code> recurring job with in groups. Then default snapshot1 cron job should exist. default snapshot2 cron job should exist. snapshot3 cron job should exist. default backup1 cron job should exist. default backup2 cron job should exist. backup3 cron job should exist.

Delete snapshot2 recurring job should delete the cron job

When delete snapshot-2 recurring job. Then default snapshot1 cron job should exist. default snapshot2 cron job should not exist. snapshot3 cron job should exist. default backup1 cron job should exist. default backup2 cron job should exist. backup3 cron job should exist.

Delete multiple recurring jobs should reflect on the cron jobs.

When delete backup-1 recurring job. delete backup-2 recurring job. delete backup-3 recurring job. Then default snapshot1 cron job should exist. default snapshot2 cron job should not exist. snapshot3 cron job should exist. default backup1 cron job should not exist. default backup2 cron job should not exist. backup3 cron job should not exist.

def test_recurring_job_delete_should_remove_volume_label(client, batch_v1_api, volume_name)

Scenario: test delete recurring job should remove volume labels

Given 1 volume created. And create snapshot1 recurring job. create backup1 recurring job. And snapshot1 job applied to the volume. backup1 job applied to the volume. And snapshot1 job exist in volume recurring job label. backup1 job exist in volume recurring job label.

When delete snapshot1 recurring job. Then snapshot1 job not exist in volume recurring job label. backup1 job exist in volume recurring job label.

When delete backup1 recurring job. Then snapshot1 job not exist in volume recurring job label. backup1 job not exist in volume recurring job label.

Given create snapshot1 recurring job with group-1 in groups. And create snapshot2 recurring job with group-1 in groups. And create backup1 recurring job with group-1 in groups. And create backup2 recurring job with default in groups. // The default job-group automatically applies to the volumes // with no recurring job. We want to keep the test focused on the // behavior so removing the default job-group assignment first. And remove volume recurring job label default job-group. And group-1 job-group applied to the volume. And group-1 job-group exist in volume recurring job label. default job-group not exist in volume recurring job label.

When delete snapshot1 recurring job. Then group-1 job-group exist in volume recurring job label.

When delete snapshot2 recurring job. Then group-1 job-group exist in volume recurring job label.

When delete back1 recurring job. Then group-1 job-group not exist in volume recurring job label. And default job-group exist in volume recurring job label.

When delete back2 recurring job. Then should not remove default job-group in volume.

def test_recurring_job_detached_volume(client, batch_v1_api, volume_name)

Scenario: test recurring job while volume is detached

Given a volume created, and attached. And write some data to the volume. And detach the volume.

When create a recurring job running at 1 minute interval, and with default in groups, and with retain set to 2. And 1 cron job should be created. And wait for 2 minutes. And attach volume and wait until healthy

Then the volume should have 1 snapshot

When wait for 2 minute. Then then volume should have only 2 snapshots.

def test_recurring_job_duplicated(client)

Scenario: test create duplicated recurring jobs

Given recurring job created. When create same recurring job again. Then should fail.

def test_recurring_job_filesystem_trim(client, core_api, batch_v1_api, volume_name, csi_pv, pvc, pod_make, access_mode)

Scenario: test recurring job filesystem-trim

Given a workload (volume, pv, pvc, pod). And create 50mb file in volume. And actual size of the volume should increase. And delete the 50mb file in volume. And actual size of the volume should not decrease.

When create a recurring job with: - task: filesystem-trim - retain: 1 Then recurring job retain mutated to 0.

When assign the recurring job to volume. And wait for the cron job scheduled time. Then volume actual size should decrease 50mb.

def test_recurring_job_groups(set_random_backupstore, client, batch_v1_api)

Scenario: test recurring job groups (S3/NFS)

Given volume test-job-1 created, attached, and healthy. volume test-job-2 created, attached, and healthy. And create snapshot recurring job with group-1, group-2 in groups. set cron job to run every 2 minutes. set retain to 1. create backup recurring job with group-1 in groups. set cron job to run every 3 minutes. set retain to 1

When set group1 recurring job in volume test-job-1 label. set group2 recurring job in volume test-job-2 label. And write some data to volume test-job-1. write some data to volume test-job-2. And wait for 2 minutes. And write some data to volume test-job-1. write some data to volume test-job-2. And wait for 1 minute.

Then volume test-job-1 should have 3 snapshots after scheduled time. volume test-job-2 should have 2 snapshots after scheduled time. And volume test-job-1 should have 1 backup after scheduled time. volume test-job-2 should have 0 backup after scheduled time.

def test_recurring_job_in_storageclass(set_random_backupstore, client, core_api, storage_class, statefulset)

Test create volume with StorageClass contains recurring jobs

  1. Create a StorageClass with recurring jobs
  2. Create a StatefulSet with PVC template and StorageClass
  3. Verify the recurring jobs run correctly.
def test_recurring_job_in_volume_creation(client, volume_name)

Scenario: test create volume with recurring jobs

Given 2 recurring jobs created. And volume create and a attached.

When label recurring job to volume. And write data to volume. wait 2.5 minutes. And write data to volume. wait 2.5 minutes.

Then volume have 4 snapshots.

def test_recurring_job_kubernetes_status(set_random_backupstore, client, core_api, volume_name)

Scenario: test recurringJob properly backs up the KubernetesStatus (S3/NFS)

Given volume created and detached. And PV from volume created and verified.

When create backup recurring job to run every 2 minutes. And attach volume. And write some data to volume. And wait 5 minutes.

Then volume have 2 snapshots. volume have 1 backup. And backup have the Kubernetes Status labels.

def test_recurring_job_label_on_pvc(client, core_api, volume_name)

Scenario: test recurring job label on PVC

Given volume created. And PV created. and PVC created. and PVC source recurring job label exists. and Volume default recurring job label exists. and RecurringJobs created (recurring-test-1, recurring-test-2). and RecurringJob (recurring-test-2) is in group (recurring-test-2)

When add a recurring job label (recurring-test-1) on the PVC. And add a recurring job-group label (recurring-test-2) on the PVC. Then Volume has recurring job label (recurring-test-1). And Volume has recurring job-group label (recurring-test-2).

When remove PVC recurring job label (recurring-test-2). Then Volume has recurring job label (recurring-test-1).

When delete RecurringJobs (recurring-test-1). Then Volume has recurring job label (default). And PVC has no recurring job.

def test_recurring_job_labels(set_random_backupstore, client, random_labels, volume_name)

Scenario: test a recurring job with labels (S3/NFS)

Given a recurring job created, with default in groups, with random labels. And volume created and attached. And write data to volume.

When add another label to the recurring job. And write data to volume. And wait after scheduled time.

Then should have 2 snapshots. And backup should have correct labels.

def test_recurring_job_multiple_volumes(set_random_backupstore, client, batch_v1_api)

Scenario: test recurring job with multiple volumes

Given volume test-job-1 created, attached and healthy. And create backup1 recurring job with default in groups. create backup2 recurring job with ` in groups. And <code>default</code> group exist intest-job-1volume recurring job label. And <code>backup1</code> cron job exist. <code>backup2</code> cron job exist. And write data totest-job-1volume. And 2 snapshot exist intest-job-1volume. And 1 backup exist intest-job-1` volume.

When create and attach volume test-job-2. wait for volume test-job-2 to be healthy. And default group exist in test-job-2 volume recurring job label. And write data to test-job-1 volume. Then 2 snapshot exist in test-job-2 volume. 1 backup exist in test-job-2 volume.

When add backup2 in test-job-2 volume label. And default group exist in test-job-1 volume recurring job label. default group exist in test-job-2 volume recurring job label. backup2 group exist in test-job-2 volume recurring job label. And write data to test-job-1. write data to test-job-2. Then wait for schedule time. And 2 backup exist in test-job-2 volume. 1 backup exist in test-job-1 volume.

def test_recurring_job_restored_from_backup_target(set_random_backupstore, client, batch_v1_api)

Scenario: test recurring job backup (S3/NFS)

  1. Create a volume and attach it to a node or a workload.
  2. Create some recurring jobs (some are in groups)
  3. Label the volume with created recurring jobs (some are in groups)
  4. Create a backup or wait for a recurring job starting
  5. Wait for backup creation completed.

  6. Check if recurring jobs/groups information is stored in the backup volume configuration on the backup target

  7. Create a volume from the backup just created.

  8. Check the volume if it has labels of recurring jobs and groups.

  9. Delete recurring jobs that are already stored in the backup volume on the backup.

  10. Create a volume from the backup just created.
  11. Check if recurring jobs have been created.
  12. Check if restoring volume has labels of recurring jobs and groups.
def test_recurring_job_snapshot(client, batch_v1_api)

Scenario: test recurring job snapshot

Given volume test-job-1 created, attached, and healthy. volume test-job-2 created, attached, and healthy.

When create a recurring job with default in groups. Then should have 1 cron job. And volume test-job-1 should have volume-head 1 snapshot. volume test-job-2 should have volume-head 1 snapshot.

When write some data to volume test-job-1. write some data to volume test-job-2. And wait for cron job scheduled time. Then volume test-job-1 should have 2 snapshots after scheduled time. volume test-job-2 should have 2 snapshots after scheduled time.

When write some data to volume test-job-1. write some data to volume test-job-2. And wait for cron job scheduled time. Then volume test-job-1 should have 3 snapshots after scheduled time. volume test-job-2 should have 3 snapshots after scheduled time.

def test_recurring_job_snapshot_cleanup(set_random_backupstore, client, batch_v1_api, volume_name)

Scenario: test recurring job snapshot-cleanup

Given volume created, attached, and healthy. And 2 snapshot were created by system. And 1 snapshot were created by user. And volume has 4 snapshots. - 2 system-created - 1 user-created - 1 volume-head

When create a recurring job with: - task: system-snapshot-delete - retain: 1 Then recurring job retain mutated to 0.

When assign the recurring job to volume. And wait for the cron job scheduled time. Then volume should have 2 snapshots. - 0 system-created - 1 user-created - 1 volume-head

def test_recurring_job_snapshot_delete(set_random_backupstore, client, batch_v1_api, volume_name)

Scenario: test recurring job snapshot-delete

Given volume created, attached, and healthy. And 3 snapshots were created. And volume has 4 snapshots. - 3 user-created - 1 volume-head

When create a recurring job with: - task: snapshot-delete - retain: 1 And assign the recurring job to volume. And wait for the cron job scheduled time.

Then volume should have 2 snapshots. - 1 snapshot retained - 1 volume-head

When recurring job unassigned to volume. And create 5 snapshots. And volume has 7 snapshots. - 5 new snapshots - 1 old snapshot - 1 volume-head And update recurring job retain to 3. And assign the recurring job to volume. And wait for the cron job scheduled time.

Then volume should have 4 snapshots - 3 snapshots retained - 1 volume-head

def test_recurring_job_snapshot_delete_retain_0(set_random_backupstore, client, batch_v1_api, volume_name)

Scenario: test recurring job snapshot-delete with retain value 0

Given volume created, attached, and healthy. And 1 snapshots were created. And volume has 2 snapshots. - 1 user-created - 1 volume-head

When create a recurring job with: - task: snapshot-delete - retain: 0 And assign the recurring job to volume. And wait for the cron job scheduled time.

Then volume should have 1 snapshot. - 0 snapshot retained - 1 volume-head

def test_recurring_job_source_label(client, core_api, volume_name)

Scenario: test recurring job source label

Given volume created. And PV created. and PVC created. and Volume default recurring job label exists. and RecurringJob created (recurring-test).

When add source recurring job label on PVC. And add a recurring job label (recurring-test) on the PVC. And Volume has recurring job label (recurring-test).

When delete source recurring job label on PVC. And remove PVC recurring job label (recurring-test). Then wait some times for volume controller to resync (30 seconds) And Volume has recurring job label (recurring-test).

When add source recurring job label on PVC. Then Volume has recurring job label (default).

def test_recurring_job_volume_label_when_job_and_group_use_same_name(client, volume_name)

Scenario: test volume label recurring job when recurring job and job-group uses the same name.

Given volume-1 created. volume-2 created. volume-3 created. And create snapshot1 recurring job with snapshot-1 in groups. create snapshot2 recurring job with snapshot-1 in groups. And snapshot1 job-group applied to volume-1. snapshot1 job applied to volume-2. snapshot2 job applied to volume-3. And snapshot1 job-group exist in volume-1 recurring job label. snapshot1 job exist in volume-2 recurring job label. snapshot2 job exist in volume-3 recurring job label.

When delete snapshot1 recurring job. Then snapshot1 job-group exist in volume-1 recurring job label. snapshot1 job not exist in volume-2 recurring job label. snapshot2 job exist in volume-3 recurring job label.

When delete snapshot2 recurring job. Then snapshot1 job-group not exist in volume-1 recurring job label. snapshot2 job not exist in volume-3 recurring job label.

def test_recurring_jobs_allow_detached_volume(set_random_backupstore, client, core_api, apps_api, volume_name, make_deployment_with_pvc)

Scenario: test recurring jobs for detached volume with allow-recurring-job-while-volume-detached set to true

Context: In the current Longhorn implementation, users cannot do recurring backup when volumes are detached. This feature gives the users an option to do recurring backup even when volumes are detached. longhorn/longhorn#1509

Given allow-recurring-job-while-volume-detached set to true. And volume created and attached. And 50MB data written to volume. And volume detached.

When a recurring job created runs every minute. And wait for backup to complete.

Then volume have 1 backup in 2 minutes retry loop.

When delete the recurring job. And create a PV from volume. And create a PVC from volume. And create a deployment from PVC. And write 400MB data to the volume from the pod. And scale deployment replicas to 0. wait until the volume is detached. And create a recurring job runs every 2 minutes. And wait for backup to start. And scale deployment replicas to 1. Then volume's frontend is disabled. And pod cannot start.

When wait until backup complete. And delete the recurring job. Then pod can start in 10 minutes retry loop.

def test_recurring_jobs_maximum_retain(client, core_api, volume_name)

Scenario: test recurring jobs' maximum retain

Given set a recurring job retain to 101.

When create recurring job. Then should fail.

When set recurring job retain to 100. And create recurring job. Then recurring job created with retain equals to 100.

When update recurring job retain to 101. Then should fail.

def test_recurring_jobs_on_nodes_with_taints()

Test recurring jobs on nodes with taints

Context:

Test the prevention of creation of multiple pods due to recurring job's pod being rejected by Taint controller on nodes with taints

Steps:

  1. Set taint toleration for Longhorn components persistence=true:NoExecute
  2. Taint node-1 with persistence=true:NoExecute
  3. Create a volume, vol-1. Attach vol-1 to node-1 Write some data to vol-1
  4. Create a recurring backup job which: Has retain count 10 Runs every minute

  5. Wait for 3 minutes. Verify that the there is 1 backup created Verify that the total number of pod in longhorn-system namespace < 50 Verify that the number of pods of the cronjob is <= 2

  6. Taint all nodes with persistence=true:NoExecute

  7. Write some data to vol-1

  8. Wait for 3 minutes. Verify that the there are 2 backups created in total Verify that the total number of pod in longhorn-system namespace < 50 Verify that the number of pods of the cronjob is <= 2

  9. Remove persistence=true:NoExecute from all nodes and Longhorn setting Clean up backups, volumes

def test_recurring_jobs_when_volume_detached_unexpectedly(set_random_backupstore, client, core_api, apps_api, volume_name, make_deployment_with_pvc)

Scenario: test recurring jobs when volume detached unexpectedly

Context: If the volume is automatically attached by the recurring backup job, make sure that workload pod eventually is able to use the volume when volume is detached unexpectedly during the backup process.

Given allow-recurring-job-while-volume-detached set to true. And volume created and detached. And PV created from volume. And PVC created from volume. And deployment created from PVC. And 500MB data written to the volume. And deployment replica scaled to 0. And volume detached.

When create a backup recurring job runs every 2 minutes. And wait for backup to start. wait for backup progress > 50%. And kill the engine process of the volume. Then volume is attached and healthy.

When backup completed. Then volume is detached with frontendDisabled=false.

When deployment replica scaled to 1. Then the data exist in the deployment pod.

def wait_for_actual_size_change_mb(client, vol_name, old_size, retry_counts=60, wait_stablize=True)
def wait_for_recurring_backup_to_start(client, core_api, volume_name, expected_snapshot_count, minimum_progress=0)
def wait_until_begin_of_a_minute()
def wait_until_begin_of_an_even_minute()