Skip to content

AWS Custom AMI Example

This example demonstrates how to configure custom AMIs for Qumulo cluster nodes and the provisioner instance on AWS. It covers:

  • using your own hardened images
  • RHEL as an alternative to the default Ubuntu
  • cross-account AMI sharing
  • the regional scoping rules that apply to AMIs

Prerequisites

  • AWS account with permissions to create IAM roles, EC2 instances, S3 buckets, and security groups
  • Existing VPC and subnet
  • (Optional) Custom AMI(s) for cluster nodes and/or provisioner VM
  • Terraform >= 1.0

VPC Prerequisites

Your VPC must meet two requirements before deploying a cluster:

  • S3 gateway VPC endpoint: Attach an S3 gateway endpoint to the route table serving the subnet. The provider does not create this (cross-account VPCs may lack permission), and deployment will fail without it.
  • Outbound internet access: Cluster nodes and the provisioner instance must reach the public internet for image downloads and cluster registration. Use a NAT Gateway in a private subnet (recommended). An Internet Gateway in a public subnet also works but reduces network isolation.

Usage

  1. Provide variable values. Create a terraform.tfvars file, use TF_VAR_* env vars, or pass -var / -var-file flags. At minimum:
cluster_ami_id     = "ami-..."  # the AMI you want the cluster nodes to run
provisioner_ami_id = "ami-..."  # the AMI you want the provisioner VM to run (can match cluster_ami_id)
vpc_id             = "vpc-..."
subnet_ids         = ["subnet-..."]
admin_password     = "YourSecurePassword123!"
allow_cidrs        = ["10.0.0.0/8"]
  1. Initialize and deploy: 
    terraform init -upgrade
terraform plan
terraform apply

AMI Configuration Options

Default (No AMI Specified)

By default, Qumulo uses its curated AMI based on Ubuntu 24.04 LTS for both cluster nodes and the provisioner instance. Omit ami_id and provisioner_ami_id to get the default behavior.

Option 1: Custom Cluster AMI

Use your own AMI for cluster nodes (for example, a hardened base image that meets internal compliance requirements):

ami_id = "ami-0123456789abcdef0"

The AMI must be available in the same region as the cluster and must support cloud-init.

Option 2: Custom Provisioner AMI

Use a custom AMI for the provisioner instance independently of the cluster AMI. This is useful when your organization requires a specific base image for short-lived deployment automation:

provisioner_ami_id = "ami-0123456789abcdef0"

Because the provisioner is ephemeral, changing this value does NOT trigger a cluster replacement. The new AMI takes effect on the next terraform apply that launches a provisioner, for example when scaling node count, replacing cluster nodes, or applying any other change that requires the provisioner.

Option 3: Mixed (both custom)

Set both attributes to use custom AMIs for every instance the provider launches:

ami_id             = "ami-0123456789abcdef0"
provisioner_ami_id = "ami-0fedcba9876543210"

Option 4: RHEL Instead of Ubuntu

The provider supports RHEL 8, 9, and 10 as alternatives to the default Ubuntu AMI. Pass the AMI ID of any supported RHEL release:

# RHEL 8
ami_id = "ami-0abcdef1234567890"

# RHEL 9
ami_id = "ami-0bcdef12345678901"

# RHEL 10
ami_id = "ami-0cdef123456789012"

Use the AWS console or aws ec2 describe-images to find the current RHEL AMI ID for your target region and architecture.

Supported AMI Operating Systems

The provider accepts AMIs for these operating systems:

OS Version Notes
Ubuntu 24.04 LTS Default when ami_id is omitted
RHEL 8 Supported
RHEL 9 Supported
RHEL 10 Supported

Custom AMIs based on other distributions are not supported and will fail cluster bootstrap.

Cross-Account AMIs

You can launch a cluster using an AMI that lives in a different AWS account. This is common when a central platform team publishes hardened images for multiple workload accounts.

Requirements:

  • Launch permission: The AMI owner must grant the cluster's account launch permission on the AMI (aws ec2 modify-image-attribute --launch-permission Add=... or the equivalent console action).
  • KMS key access: If the AMI's backing snapshot is encrypted, the cluster account must be able to decrypt it. The KMS key policy in the AMI owner's account must allow the cluster account to use the key (kms:Decrypt, kms:DescribeKey, kms:CreateGrant). Without this, EC2 cannot launch instances from the AMI and deployment will fail at bootstrap.
  • AMI ownership: No change is required from the cluster account; reference the AMI by its ID as usual.

Cross-account AMIs are still subject to regional scoping, which is described in the next section.

Regional AMI Considerations

AMIs are region-scoped. An AMI ID only works in the region where it was registered. To deploy clusters in multiple regions, copy the AMI to each target region before deploying.

Use aws ec2 copy-image to copy an AMI across regions. The copy is an independent AMI with its own ID; update ami_id (and provisioner_ami_id if relevant) to match the target region's AMI. For encrypted AMIs, make sure a KMS key is available in the destination region and referenced by the copy.

See Copy an AMI in the AWS documentation for the full procedure and cross-region, cross-account variants.

Important Notes

  • Changing ami_id triggers a rolling node replacement via a two-phase quorum swap. The provider launches new instances with the new AMI, swaps them into the cluster while maintaining quorum, then terminates the old instances. Cluster data is preserved. Plan for a maintenance window as the operation takes time per node.
  • Changing provisioner_ami_id does NOT trigger cluster replacement (provisioner is ephemeral). The new AMI is used on the next deploy operation.
  • Custom AMIs must include cloud-init and be on an x86_64 architecture matching the supported EC2 instance families (for example, m6idn, m6i, i3en, i4i, i7i, i7ie).

Outputs

After deployment, you'll get:

  • cluster_name: Name of the Qumulo cluster
  • cluster_uuid: UUID of the Qumulo cluster
  • deployment_unique_name: Unique deployment identifier
  • endpoint_ips: Client-facing IPs. Floating IPs if configured, otherwise primary IPs.
  • primary_ips: Per-node primary IPs. Use these directly when no floating IPs are configured, or for per-node access.
  • endpoints: Connection endpoints for various protocols (web UI, API, NFS, SMB)

Full Configuration

# Example: AWS Qumulo Cluster with Custom AMIs
# Edit the values below directly, then run: terraform init -upgrade && terraform apply

terraform {
  required_version = ">= 1.0"
  required_providers {
    qumulo = {
      source  = "qumulo-terraform-registry.s3.us-east-1.amazonaws.com/qumulo/qumulo"
      version = "~> 1.0"
    }
  }
}

provider "qumulo" {
  aws {
    # Standard AWS credential chain applies (env vars, shared config, SSO,
    # instance profile). No credentials block is required here.
    # profile   = "my-profile"   # optional, overrides AWS_PROFILE
  }
}

variable "region" {
  description = "AWS region for deployment"
  type        = string
  default     = "us-east-1"
}

variable "vpc_id" {
  description = "VPC ID where the cluster is deployed"
  type        = string
}

variable "subnet_ids" {
  description = "Subnet IDs for cluster nodes (1 for single-AZ, 3+ for multi-AZ)"
  type        = list(string)
}

variable "admin_password" {
  description = "Cluster administrator password (8-128 chars)"
  type        = string
  sensitive   = true
  validation {
    condition     = length(var.admin_password) >= 8 && length(var.admin_password) <= 128
    error_message = "admin_password must be between 8 and 128 characters."
  }
}

variable "allow_cidrs" {
  description = "CIDR blocks allowed to access the cluster"
  type        = list(string)
  default     = ["0.0.0.0/0"]
}

variable "cluster_ami_id" {
  description = "AMI ID for cluster nodes. Replace with your own AMI (Ubuntu 24.04 or RHEL 8/9/10)."
  type        = string
  # Replace with a real AMI ID in your region before applying.
  default = "ami-0123456789abcdef0"
}

variable "provisioner_ami_id" {
  description = "AMI ID for the provisioner instance. Replace with your own AMI."
  type        = string
  # Replace with a real AMI ID in your region before applying.
  default = "ami-0fedcba9876543210"
}

resource "qumulo_filesystem_aws" "cluster" {
  provider = qumulo

  # =============================================================================
  # All attributes (alphabetical order).
  # Uncomment any attribute to use it. Required attributes are not commented.
  # =============================================================================

  # # Additional security group IDs to attach to cluster nodes.
  # additional_security_group_ids = ["sg-abc123"]

  # Administrator password for cluster access (8-128 characters).
  admin_password = var.admin_password

  # CIDR blocks allowed to access the cluster (required, at least one).
  allow_cidrs = var.allow_cidrs

  # AMI ID for cluster nodes. Supports Ubuntu 24.04 (default) and RHEL 8/9/10.
  # Changing this triggers a rolling replacement (quorum swap). Cluster data is preserved.
  # Alternative examples (commented):
  #   ami_id = "ami-0abcdef1234567890"   # RHEL 8
  #   ami_id = "ami-0bcdef12345678901"   # RHEL 9
  #   ami_id = "ami-0cdef123456789012"   # RHEL 10
  ami_id = var.cluster_ami_id

  # Cluster storage product type (immutable after creation).
  cluster_product_type = "HOT"

  # # Qumulo software version. Defaults to latest. Immutable after creation.
  # cluster_version = "7.5.0"

  # # EC2 key pair name for SSH access to cluster nodes.
  # ec2_key_pair = "my-keypair"

  # # Number of floating IPs. Requires networking_mode = "host_managed" AND single-AZ.
  # floating_ip_count = 3

  # EC2 instance type for cluster nodes.
  instance_type = "m6idn.2xlarge"

  # # KMS key ID for encrypting cluster data (immutable after creation).
  # # Required when using cross-account encrypted AMIs that reference a KMS key
  # # shared with this account.
  # kms_key_id = "arn:aws:kms:us-east-1:123456789012:key/abcd-ef01-..."

  # qfsd cluster name, shown in the Qumulo UI (2-15 chars, case preserved).
  cluster_name = "qumuloami"

  # Prefix for the AWS resources this cluster creates (2-15 lowercase chars).
  deployment_name = "qumuloami"

  # # Networking mode. Default "host_managed" preserves EC2 IPs across reboots.
  # networking_mode = "host_managed"

  # # Qumulo Nexus registration key for remote support.
  # nexus_registration_key = "your-nexus-key"

  # Number of nodes in the cluster.
  node_count = 3

  # # IAM permissions boundary ARN applied to cluster roles.
  # permissions_boundary_arn = "arn:aws:iam::123456789012:policy/boundary"

  # AMI ID for the provisioner instance. Ephemeral -- changes do NOT replace the cluster.
  provisioner_ami_id = var.provisioner_ami_id

  # # EC2 instance type for the provisioner VM (used during deploy operations).
  # provisioner_instance_type = "t3.medium"

  # AWS region for deployment.
  region = var.region

  # # Soft capacity limit in TB (50 to 50000).
  # soft_capacity_limit_tb = 1000

  # Subnet IDs for cluster nodes. Use 1 for single-AZ (this example), 3+ for multi-AZ.
  subnet_ids = var.subnet_ids

  # # Tags to apply to all AWS resources created for this cluster.
  # tags = {
  #   Environment = "Development"
  #   ManagedBy   = "Terraform"
  #   ImageType   = "Custom" # or "Default", "RHEL", "Mixed"
  # }

  # VPC ID. Derived from subnet_ids if omitted.
  vpc_id = var.vpc_id

  deletion_protection = true # recommended: guard the cluster's EC2 instances and S3 buckets

  timeouts {
    create = "90m"
    delete = "30m"
  }
}

output "cluster_name" {
  description = "Name of the Qumulo cluster"
  value       = qumulo_filesystem_aws.cluster.cluster_name
}

output "cluster_uuid" {
  description = "UUID of the Qumulo cluster"
  value       = qumulo_filesystem_aws.cluster.cluster_uuid
}

output "deployment_unique_name" {
  description = "Unique deployment identifier"
  value       = qumulo_filesystem_aws.cluster.deployment_unique_name
}

output "endpoint_ips" {
  description = "Client-facing IPs. Floating IPs if configured, otherwise primary IPs."
  value       = qumulo_filesystem_aws.cluster.endpoint_ips
}

output "primary_ips" {
  description = "Per-node primary IPs. Use these directly when no floating IPs are configured, or for per-node access."
  value       = qumulo_filesystem_aws.cluster.primary_ips
}

output "endpoints" {
  description = "Connection endpoints for various protocols"
  value = {
    web_ui = "https://${try(qumulo_filesystem_aws.cluster.endpoint_ips[0], "pending")}"
    api    = "https://${try(qumulo_filesystem_aws.cluster.endpoint_ips[0], "pending")}:8000"
    nfs    = "${try(qumulo_filesystem_aws.cluster.endpoint_ips[0], "pending")}:/"
    smb    = "\\\\${try(qumulo_filesystem_aws.cluster.endpoint_ips[0], "pending")}\\share"
  }
}