Troubleshoot an AWS deployment

Find the symptom you see in the tables below, then apply the fix beside it. Each table covers one stage of a qumulo_filesystem_aws run.

Fix plan errors

Symptom	Fix
instance_type fails validation	Choose a type from AWS instance types.
An m6i or m7i type fails with an older cluster_version	Set cluster_version to 7.4.3 or later.
Requested capacity exceeds instance capacity	Lower soft_capacity_limit_tb, add nodes, or choose a larger type from AWS instance types.
Invalid subnet_ids count	Use 1 subnet for single-AZ or 3 or more for multi-AZ.
Invalid node_count for multi-AZ deployment	Use node_count = 3 or 5 and up, or deploy into a single subnet.
Warning Floating IPs are not configured in multi-AZ deployments	Use one subnet to keep floating IPs, or put a Network Load Balancer in front of the multi-AZ cluster.

Fix apply errors

Symptom	Fix
provisioner did not complete, with a last status of Missing S3 gateway!	Add an S3 gateway endpoint (com.amazonaws.<region>.s3, type Gateway) to the route table of every cluster subnet, then apply again.
provisioner did not complete, with a last status of MQ NOT reachable or Qumulo Nexus NOT reachable	Give every cluster subnet a 0.0.0.0/0 route to a NAT gateway or internet gateway, then apply again. For a subnet that is isolated on purpose, set air_gapped = true instead; see Deploy a cluster into an air-gapped subnet.
each subnet must be in a distinct availability zone	Swap one of the two named subnets for a subnet in another zone.
The KMS key is not accessible	Use a key in the cluster's region, or a multi-region replica of it there.
iam:CreateRole or iam:AttachRolePolicy fails	Widen the permissions boundary to allow what the cluster roles need.
The cluster did not register with Nexus	Set the provider's nexus_api_token, give it the permissions the apply warning names, then apply again.
You need a GovCloud or ISO region	Set region to it. The provider maps us-gov-* to aws-us-gov, us-isob-* to aws-iso-b, and us-iso-* to aws-iso, so there is no partition attribute.

Fix destroy errors

Symptom	Fix
terraform destroy refuses to run	Set deletion_protection = false, run terraform apply, then run terraform destroy.
terraform destroy terminates the instances, then fails and leaves <random>-<deployment_name>-qps-<N> buckets	Grant tag:GetResources, then run terraform destroy again. The cluster stays in state while any of its resources remain, so the second destroy finishes the cleanup.

Fix load balancer problems

Symptom	Fix
AWS rejects a target group name	Use hyphens, not underscores, in the qumulo_ports keys.
NFS locks misbehave through the load balancer	Mount clients with NFSv4 instead of NFSv3.

Get more detail

To see	Do this
The provider's full log	Run TF_LOG=DEBUG terraform apply.
Why the provisioner failed	Read the SSM parameter that the error names, which holds the provisioner's last status.
A node's or provisioner's rendered startup script	Run aws ec2 describe-instance-attribute --instance-id <instance-id> --attribute userData --query 'UserData.Value' --output text | base64 --decode | gunzip. EC2 stores the script gzip-compressed and base64-encoded.