Troubleshoot an AWS deployment

Find the symptom you see in the tables below, then apply the fix beside it. Each table covers one stage of a qumulo_filesystem_aws run.

Fix plan errors

Symptom	Fix
instance_type fails validation	Choose a type from AWS instance types.
An m6i or m7i type fails with an older cluster_version	Set cluster_version to 7.4.3 or later.
Requested capacity exceeds instance capacity	Lower soft_capacity_limit_tb, add nodes, or choose a larger type from AWS instance types.
Invalid subnet_ids count	Use 1 subnet for single-AZ or 3 or more for multi-AZ.
Invalid node_count for multi-AZ deployment	Use node_count = 3 or 5 and up, or deploy into a single subnet.
Warning Floating IPs are not configured in multi-AZ deployments	Use one subnet to keep floating IPs, or put a Network Load Balancer in front of the multi-AZ cluster.

Fix apply errors

Symptom	Fix
provisioner did not complete, with a last status of Missing S3 gateway!	Add an S3 gateway endpoint (com.amazonaws.<region>.s3, type Gateway) to the route table of every cluster subnet, then apply again.
The provisioner cannot download the installer	Give every cluster subnet a 0.0.0.0/0 route to a NAT gateway or internet gateway, then apply again.
each subnet must be in a distinct availability zone	Swap one of the two named subnets for a subnet in another zone.
The KMS key is not accessible	Use a key in the cluster's region, or a multi-region replica of it there.
iam:CreateRole or iam:AttachRolePolicy fails	Widen the permissions boundary to allow what the cluster roles need.
The cluster did not register with Nexus	Set the provider's nexus_api_token, give it the permissions the apply warning names, then apply again.
You need a GovCloud or ISO region	Set region to it. The provider maps us-gov-* to aws-us-gov, us-isob-* to aws-iso-b, and us-iso-* to aws-iso, so there is no partition attribute.

Fix destroy errors

Symptom	Fix
terraform destroy refuses to run	Set deletion_protection = false, run terraform apply, then run terraform destroy.
terraform destroy terminates the instances, then fails and leaves <random>-<deployment_name>-qps-<N> buckets	Grant tag:GetResources, then empty and delete those buckets by hand. Running terraform destroy again does not retry them.

Fix load balancer problems

Symptom	Fix
AWS rejects a target group name	Use hyphens, not underscores, in the qumulo_ports keys.
NFS locks misbehave through the load balancer	Mount clients with NFSv4 instead of NFSv3.

Get more detail

To see	Do this
The provider's full log	Run TF_LOG=DEBUG terraform apply.
Why the provisioner failed	Read the SSM parameter that the error names, which holds the provisioner's last status.
A node's or provisioner's rendered startup script	Run aws ec2 describe-instance-attribute --instance-id <instance-id> --attribute userData --query 'UserData.Value' --output text | base64 --decode | gunzip. EC2 stores the script gzip-compressed and base64-encoded.