Module tests.test_zone

Functions

def get_zone_replica_count(client, volume_name, zone_name, chk_running=False)
def k8s_node_zone_tags(client, core_api)
def test_replica_auto_balance_node_duplicates_in_multiple_zones(client, core_api, volume_name)

Scenario: replica auto-balance to nodes with duplicated replicas in the zone.

Given set replica-soft-anti-affinity to true. And set replica-zone-soft-anti-affinity to true. And set volume spec replicaAutoBalance to least-effort. And set node-1 to zone-1. set node-2 to zone-2. And disable scheduling for node-3. And create a volume with 3 replicas. And attach the volume to self-node. And zone-1 and zone-2 should contain 3 replica in total.

When set node-3 to the zone with duplicated replicas. And enable scheduling for node-3. Then count replicas running on each node. And 1 replica running on node-1 1 replica running on node-2 1 replica running on node-3. And count replicas running in each zone. And total of 3 replicas running in zone-1 and zone-2.

def test_replica_auto_balance_should_respect_node_selector(client, core_api, volume_name, pod)

Background:

Given Setting (replica-soft-anti-affinity) is (true). And Setting (replica-zone-soft-anti-affinity) is (true). And Node (node-1, node-2) has tag (tag-0). And Node (node-1) is in zone (lh-zone-1). Node (node-2) is in zone (lh-zone-2). Node (node-3) is in zone (should-not-schedule).

Scenario Outline: replica auto-balance should respect node-selector.

Issue: https://github.com/longhorn/longhorn/issues/5971

Given Volume created. And Volume replica number is (3). And Volume has node selector (tag-0). And Volume attached (node-1). And Replica is in zone (lh-zone-1, lh-zone-2).

When Setting (replica-auto-balance) is (least-effort).

Then Replica is in zone (lh-zone-1, lh-zone-2) (loop 10 sec).

def test_replica_auto_balance_when_disabled_disk_scheduling_in_zone(client, core_api, volume_name)

Scenario: replica auto-balance when disk scheduling is disabled on nodes in a zone.

Issue: https://github.com/longhorn/longhorn/issues/6508

Given replica-soft-anti-affinity setting is true. And node-1 is in zone-1. node-2 is in zone-2. node-3 is in zone-3. And disk scheduling is disabled on node-3. And create a volume with 3 replicas. And attach the volume to test pod node. And 3 replicas running in zone-1 and zone-2. 0 replicas running in zone-3.

When set replica-auto-balance to best-effort.

Then 3 replicas running in zone-1 and zone-2. 0 replicas running in zone-3. And replica count remains stable across zones and nodes.

def test_replica_auto_balance_when_no_storage_available_in_zone(client, core_api, volume_name)

Scenario: replica auto-balance when there is no storage available on nodes in a zone.

Issue: https://github.com/longhorn/longhorn/issues/6671

Given replica-soft-anti-affinity setting is true. And node-1 is in zone-1. node-2 is in zone-2. node-3 is in zone-3. And fill up the storage on node-3. And create a volume with 3 replicas. And attach the volume to test pod node. And 3 replicas running in zone-1 and zone-2. 0 replicas running in zone-3.

When set replica-auto-balance to best-effort.

Then 3 replicas running in zone-1 and zone-2. 0 replicas running in zone-3. And replica count remains stable across zones and nodes.

def test_replica_auto_balance_when_replica_on_unschedulable_node(client, core_api, volume_name, request)

Scenario: replica auto-balance when replica already running on an unschedulable node.

Issue: https://github.com/longhorn/longhorn/issues/4502

Given set replica-soft-anti-affinity to true. And set replica-zone-soft-anti-affinity to true. And set volume spec replicaAutoBalance to least-effort. And set node-1 to zone-1. set node-2 to zone-2. set node-3 to zone-3. And node-2 tagged AVAIL. node-3 tagged AVAIL. And create a volume with 2 replicas and nodeSelector AVAIL. And attach the volume to self-node. And 0 replicas running in zone-1. 1 replicas running in zone-2. 1 replicas running in zone-3.

When cordone node-2. Then replicas should remain balanced with, 0 replicas running in zone-1. 1 replicas running in zone-2. 1 replicas running in zone-3.

def test_replica_auto_balance_zone_best_effort(client, core_api, volume_name)

Scenario: replica auto-balance zones with best-effort.

Given set replica-soft-anti-affinity to true. And set replica-zone-soft-anti-affinity to true. And set volume spec replicaAutoBalance to best-effort. And set node-1 to zone-1. set node-2 to zone-2. set node-3 to zone-3. And disable scheduling for node-2. disable scheduling for node-3. And create a volume with 6 replicas. And attach the volume to self-node. And 6 replicas running in zone-1. 0 replicas running in zone-2. 0 replicas running in zone-3.

When enable scheduling for node-2. Then count replicas running on each node. And 3 replicas running in zone-1. 3 replicas running in zone-2. 0 replicas running in zone-3.

When enable scheduling for node-3. Then count replicas running on each node. And 2 replicas running in zone-1. 2 replicas running in zone-2. 2 replicas running in zone-3.

def test_replica_auto_balance_zone_best_effort_with_data_locality(client, core_api, volume_name, pod)

Background: Given set replica-soft-anti-affinity to true. And set replica-zone-soft-anti-affinity to true. And set default-data-locality to best-effort. And set replicaAutoBalance to best-effort. And set node-1 to zone-1. set node-2 to zone-1. set node-3 to zone-2. And create volume with 2 replicas. And create pv for volume. And create pvc for volume.

Scenario Outline: replica auto-balance zones with best-effort should not remove pod local replicas when data locality is enabled (best-effort).

Given create and wait pod on . And disable scheduling and evict node-3. And count replicas on each nodes. And 1 replica running on . 1 replica running on . 0 replica running on node-3.

When enable scheduling for node-3. Then count replicas on each nodes. And 1 replica running on . 0 replica running on . 1 replica running on node-3. And count replicas in each zones. And 1 replica running in zone-1. 1 replica running in zone-2. And loop 3 times with each wait 5 seconds and count replicas on each nodes. To ensure no addition scheduling is happening. 1 replica running on . 0 replica running on . 1 replica running on node-3.

And delete pod.

Examples

| pod-node | duplicate-node | | node-1 | node-2 | | node-2 | node-1 | | node-1 | node-2 |

def test_replica_auto_balance_zone_best_effort_with_uneven_node_in_zones(client, core_api, volume_name, pod)

Given set replica-soft-anti-affinity to true. And set replica-zone-soft-anti-affinity to true. And set replicaAutoBalance to best-effort. And set node-1 to zone-1. set node-2 to zone-1. set node-3 to zone-1. set node-4 to zone-2. set node-5 to zone-2. And disable scheduling for node-2. disable scheduling for node-3. disable scheduling for node-4. disable scheduling for node-5. And create volume with 4 replicas. And attach the volume to node-1.

Scenario: replica auto-balance zones with best-effort should balance replicas in zone.

Given 4 replica running on node-1. 0 replica running on node-2. 0 replica running on node-3. 0 replica running on node-4. 0 replica running on node-5.

When enable scheduling for node-4. Then count replicas on each zones. And 2 replica running on zode-1. 2 replica running on zode-2.

When enable scheduling for node-2. enable scheduling for node-3. Then count replicas on each nodes. And 1 replica running on node-1. 1 replica running on node-2. 1 replica running on node-3. 1 replica running on node-4. 0 replica running on node-5.

When enable scheduling for node-5. Then count replicas on each zones. And 2 replica running on zode-1. 2 replica running on zode-2.

def test_replica_auto_balance_zone_least_effort(client, core_api, volume_name)

Scenario: replica auto-balance zones with least-effort.

Given set replica-soft-anti-affinity to true. And set replica-zone-soft-anti-affinity to true. And set volume spec replicaAutoBalance to least-effort. And set node-1 to zone-1. set node-2 to zone-2. set node-3 to zone-3. And disable scheduling for node-2. disable scheduling for node-3. And create a volume with 6 replicas. And attach the volume to self-node. And 6 replicas running in zone-1. 0 replicas running in zone-2. 0 replicas running in zone-3.

When enable scheduling for node-2. Then count replicas running on each node. And zone-1 replica count != zone-2 replica count. zone-2 replica count != 0. zone-3 replica count == 0.

When enable scheduling for node-3. Then count replicas running on each node. And zone-1 replica count != zone-3 replica count. zone-2 replica count != 0. zone-3 replica count != 0.

def test_replica_zone_anti_affinity(client, core_api, volume_name, k8s_node_zone_tags)

Test replica scheduler with zone anti-affinity

  1. Set zone anti-affinity to hard.
  2. Label nodes 1 & 2 with same zone label "zone1". Label node 3 with zone label "zone2".
  3. Create a volume with 3 replicas.
  4. Wait for volume condition scheduled to be false.
  5. Label node 2 with zone label "zone3".
  6. Wait for volume condition scheduled to be success.
  7. Clear the volume.
  8. Set zone anti-affinity to soft.
  9. Change the zone labels on node 1 & 2 & 3 to "zone1".
  10. Create a volume.
  11. Wait for volume condition scheduled to be success.
  12. Clean up the replica count, the zone labels and the volume.
def test_zone_tags(client, core_api, volume_name, k8s_node_zone_tags)

Test anti affinity zone feature

  1. Add Kubernetes zone labels to the nodes
    1. Only two zones now: zone1 and zone2
  2. Create a volume with two replicas
  3. Verify zone1 and zone2 either has one replica.
  4. Remove a random replica and wait for volume to finish rebuild
  5. Verify zone1 and zone2 either has one replica.
  6. Repeat step 4-5 a few times.
  7. Update volume to 3 replicas, make sure they're scheduled on 3 nodes
  8. Remove a random replica and wait for volume to finish rebuild
  9. Make sure replicas are on different nodes
  10. Repeat step 8-9 a few times
def wait_longhorn_nodes_zone_not_empty(client)