Reserve VM capacity

When you finish, every node VM in your Azure cluster runs on capacity you reserved, so a create, scale-out, or node replacement does not fail for lack of capacity in your VM size. The later sections keep the reservation matched to the cluster as it grows, shrinks, and changes size.

Azure bills every reserved slot whether or not a node fills it, and Reserved VM Instance and savings plan discounts apply to those slots. A node beyond the reserved count still deploys, but without the capacity guarantee. Keep the reservation equal to what the cluster runs, and you pay for nothing idle and leave no node uncovered.

Before you start

• A cluster configuration from Deploy a production Azure cluster, or a running cluster you manage with one
• The Azure CLI signed in with az login, as an identity that holds the Azure permissions the provider needs
• vCPU quota in the region for every slot you plan to reserve, since a reservation counts against quota the way a running VM does

1. Work out how many slots each zone needs

The provider spreads nodes across availability_zones in turn. Give each zone node_count divided by the number of zones, rounded up.

node_count	availability_zones	Slots to reserve
6	["1", "2", "3"]	2 in each of zones 1, 2, and 3
5	["1", "2", "3"]	2 in each of zones 1, 2, and 3
6	["2"]	6 in zone 2
6	not set	6 with no zone

Reserve only for the node vm_type. The provisioner VM that builds the cluster never uses the group.

2. Create the group and its reservations

This example reserves the six-node, three-zone cluster from the production guide.

az capacity reservation group create -g <group-rg> -n <group-name> -l <region> --zones 1 2 3
az capacity reservation create -g <group-rg> -c <group-name> -n l16s-zone1 --sku Standard_L16s_v3 --zone 1 --capacity 2
az capacity reservation create -g <group-rg> -c <group-name> -n l16s-zone2 --sku Standard_L16s_v3 --zone 2 --capacity 2
az capacity reservation create -g <group-rg> -c <group-name> -n l16s-zone3 --sku Standard_L16s_v3 --zone 3 --capacity 2

Each command prints JSON with "provisioningState": "Succeeded". The group must sit in the cluster's region and list every zone the cluster uses, because Azure fixes a group's zones when you create it. For a cluster without availability_zones, drop --zones and --zone, and Azure places the reservation and every node in one zone it picks.

If a reservation fails with an allocation error, Azure has no spare capacity for that size in that zone today. Try again later, or ask Azure support for capacity. Reserve well ahead of the apply that needs the slots.

3. Add the group to the configuration

az capacity reservation group show -g <group-rg> -n <group-name> --query id -o tsv

The command prints an ID that ends in /providers/Microsoft.Compute/capacityReservationGroups/<group-name>. Add it to the qumulo_filesystem_azure block.

capacity_reservation_group_id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-capacity/providers/Microsoft.Compute/capacityReservationGroups/qumulo-prod" # <-- Replace with your group ID

terraform plan shows capacity_reservation_group_id set to your group ID. The identity that runs the apply needs Microsoft.Compute/capacityReservationGroups/deploy/action on the group, which Owner and Contributor both carry.

4. Apply and check every slot

terraform apply

After the apply, check each reservation. This command shows one, so run it once per zone.

az capacity reservation show -g <group-rg> -c <group-name> -n l16s-zone1 --instance-view \
    --query '{reserved: sku.capacity, nodes: length(instanceView.utilizationInfo.virtualMachinesAllocated || `[]`)}'

The output shows reserved and nodes for that zone. When they match, every node in the zone is covered and no slot sits idle. When reserved is higher, lower it to the node count with az capacity reservation update -g <group-rg> -c <group-name> -n l16s-zone1 --capacity <nodes>. When nodes is higher, raise reserved to match, or the extra nodes stay uncovered.

Before you scale out

Raise each zone's reservation by the number of nodes you are adding, divided by the number of zones, rounded up. Adding three nodes to the three-zone example means one more slot in each zone.

az capacity reservation update -g <group-rg> -c <group-name> -n l16s-zone1 --capacity 3

Repeat for each zone, then change node_count and apply. Afterwards, run the check from step 4 and lower any reservation with a spare slot.

Before you replace nodes

A change to vm_type, availability_zones, custom_image_id, marketplace_image, or node_replacement_when_changed replaces every node. The provider creates all the new nodes before it removes any old one, so for a while the cluster runs two full sets.

• New vm_type. Add a reservation for the new size in each zone, sized as in step 1, then apply. After the apply, delete the old size's reservations with az capacity reservation delete -g <group-rg> -c <group-name> -n <old-reservation> --yes.
• New availability_zones. A group cannot gain zones. Create a new group that lists every new zone, reserve as in steps 1 and 2, and change capacity_reservation_group_id to it in the same apply. Delete the old group afterwards, as After you destroy the cluster shows.
• Same size, new image or forced replacement. The old nodes hold their slots until the apply removes them. Double each reservation before you apply, then lower it back afterwards. Without the extra slots, the new nodes still deploy, but uncovered.

Warning

Without a reservation for the new vm_type in every zone the cluster uses, Azure rejects each new node and the apply fails.

After you scale in

The provider deletes the nodes it removes, and Azure frees their slots shortly after. Lower each zone's reservation to the step 1 count for the new node_count, then run the check from step 4.

Move existing nodes into the group

Setting capacity_reservation_group_id on a running cluster covers only nodes created after it. To cover the nodes it already has, associate them yourself.

For a cluster with availability_zones, Azure moves each node into the group in place, with no restart.

az vm list -g <cluster-rg> --query "[?tags.\"qumulo-deployment\"=='<deployment-unique-name>'].id" -o tsv |
    xargs -I{} az vm update --ids {} --capacity-reservation-group <group-id>

Replace <deployment-unique-name> with the cluster's deployment_unique_name output. Then run the check from step 4.

For a cluster without availability_zones, Azure requires each node to be deallocated before it joins, which takes that node offline. Replace the nodes instead. Set node_replacement_when_changed to a new value and apply, and every new node joins the group while the cluster stays in service. Reserve for the replacement first, as Before you replace nodes shows.

Stop using the group

Removing capacity_reservation_group_id stops new nodes from joining the group, but the existing nodes stay associated with it, and Azure keeps billing the reservations.

1. Remove the capacity_reservation_group_id line and apply. The plan shows 1 to change, and no node is touched.

2. Set each reservation to zero, so the nodes can leave without a restart.

   az capacity reservation update -g <group-rg> -c <group-name> -n l16s-zone1 --capacity 0
   

3. Detach each node the reservation lists.

   az capacity reservation show -g <group-rg> -c <group-name> -n l16s-zone1 --instance-view \
       --query "instanceView.utilizationInfo.virtualMachinesAllocated[].id" -o tsv |
       xargs -I{} az vm update --ids {} --capacity-reservation-group None
   

4. Repeat steps 2 and 3 for each zone, then delete the reservations and the group, as the next section shows.

After you destroy the cluster

terraform destroy deletes every node, and Azure frees their slots shortly after. Delete each reservation, then the group, or Azure keeps billing for the empty slots.

az capacity reservation delete -g <group-rg> -c <group-name> -n l16s-zone1 --yes
az capacity reservation group delete -g <group-rg> -n <group-name> --yes

Run the first command once per reservation. If Azure reports that VMs are still associated, wait a minute for the deletes to register and run it again.

Next

Troubleshoot an Azure deployment covers the reservation error an apply can print. The qumulo_filesystem_azure reference describes capacity_reservation_group_id, and Microsoft's capacity reservation overview covers supported VM sizes and limits.