Cross-Account VPC Deployment on AWS

Deploy a Qumulo cluster into a VPC that lives in a different AWS account than the one running Terraform. The typical motivation is a centralized network team that owns shared VPCs and shares subnets with workload accounts via AWS Resource Access Manager (RAM).

When to use this

• A platform team owns VPCs and shares subnets to consumer accounts via RAM.
• Compliance or blast-radius requirements keep cluster compute resources isolated in their own AWS account.
• The Terraform credentials cannot create security groups in the VPC owner's account.

If your VPC and your Terraform run live in the same account, you do not need this guide. Use the standard AWS Basic or AWS Production examples instead.

Account layout

The provider deploys EC2 instances, ENIs, IAM roles, and S3 buckets in the consumer account (where Terraform runs) but launches those instances into a subnet that lives in the owner account. AWS allows ENIs in shared subnets, but it does not allow the consumer to create security groups in the owner's VPC. So the consumer account brings the cluster; the owner account brings the SGs.

┌─────────────────────────────────┐    ┌──────────────────────────────────┐
│ OWNER account                   │    │ CONSUMER account                 │
│                                 │    │                                  │
│  VPC                            │    │  Terraform runs here             │
│  ├── Subnet (shared via RAM) ───┼────┼──> EC2 cluster nodes (ENIs)      │
│  ├── S3 gateway endpoint        │    │      IAM roles                   │
│  ├── Cluster SG (BYO)           │    │      S3 persistent buckets       │
│  └── Provisioner SG (BYO)       │    │      Provisioner instance        │
└─────────────────────────────────┘    └──────────────────────────────────┘

Prerequisites

In the OWNER account:

• A VPC with at least one subnet.
• An S3 gateway VPC endpoint attached to the route table serving that subnet. The provider cannot create this from the consumer account.
• Outbound internet access for the subnet (NAT Gateway recommended; Internet Gateway works but reduces isolation).

In the CONSUMER account:

• Standard qumulo_filesystem_aws IAM permissions (see the import guide).
• No additional cross-account IAM is required for this pattern. The consumer launches instances in the shared subnet using its own credentials and references the owner's SGs by ID.

Step 1: Share the subnet

In the OWNER account:

aws ram create-resource-share \
  --name qumulo-shared-subnet \
  --resource-arns arn:aws:ec2:<region>:<owner-account-id>:subnet/<subnet-id> \
  --principals <consumer-account-id>

The consumer account will see the subnet via aws ec2 describe-subnets once the share is accepted. RAM auto-accepts within an Organization; outside an Organization the consumer must accept the invitation.

Step 2: Create security groups in the OWNER account

The provider's normal flow creates two security groups (cluster + provisioner) in the VPC. In a cross-account deployment that is not possible, so the OWNER must pre-create them with the same rule set. Set cluster_security_group_id and provisioner_security_group_id on the qumulo_filesystem_aws resource and the provider will not create or modify security groups.

Two constraints apply:

• Both attributes must be set together.
• allow_cidrs must be omitted; you own the ingress rules on your supplied security groups.

The Terraform module below produces an exact match for what the provider would create. Copy it, run it in the OWNER account targeting the shared VPC, and capture the two output IDs.

The allow_ipv6_cidrs variable and its _ipv6 resources are only needed when the cluster sets floating_ip_count_ipv6 to a non-zero value; leave allow_ipv6_cidrs empty (the default) for clusters with an IPv4 floating pool. When set (default it to the VPC's own IPv6 CIDR), every port ingress rule that exists for IPv4 gets a matching cidr_ipv6 rule on both the cluster and provisioner security groups, plus one additional egress rule per security group allowing ::/0.

variable "vpc_id"      { type = string }
variable "name_prefix" { type = string }
variable "allow_cidrs" {
  type    = list(string)
  default = ["10.0.0.0/8"]
}
# Only needed when the cluster sets floating_ip_count_ipv6 > 0; leave empty for
# clusters with an IPv4 floating pool. Defaults to the VPC's own IPv6 CIDR.
variable "allow_ipv6_cidrs" {
  type    = list(string)
  default = []
}

resource "aws_security_group" "cluster" {
  name        = "${var.name_prefix}-qumulo-cluster"
  description = "Qumulo cluster security group"
  vpc_id      = var.vpc_id
}

resource "aws_security_group" "provisioner" {
  name        = "${var.name_prefix}-qumulo-provisioner"
  description = "Qumulo provisioner security group"
  vpc_id      = var.vpc_id
}

locals {
  cluster_ports = [
    { port = 21,   proto = "tcp", desc = "FTP" },
    { port = 22,   proto = "tcp", desc = "SSH" },
    { port = 53,   proto = "tcp", desc = "DNS" },
    { port = 53,   proto = "udp", desc = "DNS" },
    { port = 111,  proto = "tcp", desc = "SUNRPC" },
    { port = 111,  proto = "udp", desc = "SUNRPC" },
    { port = 443,  proto = "tcp", desc = "HTTPS" },
    { port = 445,  proto = "tcp", desc = "SMB" },
    { port = 2049, proto = "tcp", desc = "NFS" },
    { port = 2049, proto = "udp", desc = "NFS" },
    { port = 3712, proto = "tcp", desc = "Replication" },
    { port = 3713, proto = "tcp", desc = "CDF" },
    { port = 5201, proto = "tcp", desc = "IPERF3" },
    { port = 5201, proto = "udp", desc = "IPERF3" },
    { port = 8000, proto = "tcp", desc = "REST API" },
    { port = 9000, proto = "tcp", desc = "S3" },
  ]
}

resource "aws_vpc_security_group_ingress_rule" "cluster_cidr" {
  for_each = { for pair in setproduct(range(length(local.cluster_ports)), var.allow_cidrs) :
               "${pair[0]}-${pair[1]}" => { rule = local.cluster_ports[pair[0]], cidr = pair[1] } }
  security_group_id = aws_security_group.cluster.id
  ip_protocol       = each.value.rule.proto
  from_port         = each.value.rule.port
  to_port           = each.value.rule.port
  cidr_ipv4         = each.value.cidr
  description       = each.value.rule.desc
}

resource "aws_vpc_security_group_ingress_rule" "cluster_cidr_ipv6" {
  for_each = { for pair in setproduct(range(length(local.cluster_ports)), var.allow_ipv6_cidrs) :
               "${pair[0]}-${pair[1]}" => { rule = local.cluster_ports[pair[0]], cidr = pair[1] } }
  security_group_id = aws_security_group.cluster.id
  ip_protocol       = each.value.rule.proto
  from_port         = each.value.rule.port
  to_port           = each.value.rule.port
  cidr_ipv6         = each.value.cidr
  description       = each.value.rule.desc
}

resource "aws_vpc_security_group_ingress_rule" "cluster_internode" {
  security_group_id            = aws_security_group.cluster.id
  ip_protocol                  = "-1"
  referenced_security_group_id = aws_security_group.cluster.id
  description                  = "Internode communication"
}

resource "aws_vpc_security_group_egress_rule" "cluster_egress" {
  security_group_id = aws_security_group.cluster.id
  ip_protocol       = "-1"
  cidr_ipv4         = "0.0.0.0/0"
}

resource "aws_vpc_security_group_egress_rule" "cluster_egress_ipv6" {
  count             = length(var.allow_ipv6_cidrs) > 0 ? 1 : 0
  security_group_id = aws_security_group.cluster.id
  ip_protocol       = "-1"
  cidr_ipv6         = "::/0"
  description       = "All outbound IPv6 traffic"
}

resource "aws_vpc_security_group_ingress_rule" "provisioner_ssh" {
  for_each          = toset(var.allow_cidrs)
  security_group_id = aws_security_group.provisioner.id
  ip_protocol       = "tcp"
  from_port         = 22
  to_port           = 22
  cidr_ipv4         = each.value
}

resource "aws_vpc_security_group_ingress_rule" "provisioner_ssh_ipv6" {
  for_each          = toset(var.allow_ipv6_cidrs)
  security_group_id = aws_security_group.provisioner.id
  ip_protocol       = "tcp"
  from_port         = 22
  to_port           = 22
  cidr_ipv6         = each.value
}

resource "aws_vpc_security_group_ingress_rule" "provisioner_https" {
  for_each          = toset(var.allow_cidrs)
  security_group_id = aws_security_group.provisioner.id
  ip_protocol       = "tcp"
  from_port         = 443
  to_port           = 443
  cidr_ipv4         = each.value
}

resource "aws_vpc_security_group_ingress_rule" "provisioner_https_ipv6" {
  for_each          = toset(var.allow_ipv6_cidrs)
  security_group_id = aws_security_group.provisioner.id
  ip_protocol       = "tcp"
  from_port         = 443
  to_port           = 443
  cidr_ipv6         = each.value
}

resource "aws_vpc_security_group_egress_rule" "provisioner_egress" {
  security_group_id = aws_security_group.provisioner.id
  ip_protocol       = "-1"
  cidr_ipv4         = "0.0.0.0/0"
}

resource "aws_vpc_security_group_egress_rule" "provisioner_egress_ipv6" {
  count             = length(var.allow_ipv6_cidrs) > 0 ? 1 : 0
  security_group_id = aws_security_group.provisioner.id
  ip_protocol       = "-1"
  cidr_ipv6         = "::/0"
  description       = "All outbound IPv6 traffic"
}

# Cross-SG rules: provisioner reaches cluster on 443 (qq download) and 8000 (REST).
resource "aws_vpc_security_group_ingress_rule" "cluster_from_provisioner_https" {
  security_group_id            = aws_security_group.cluster.id
  ip_protocol                  = "tcp"
  from_port                    = 443
  to_port                      = 443
  referenced_security_group_id = aws_security_group.provisioner.id
}

resource "aws_vpc_security_group_ingress_rule" "cluster_from_provisioner_rest" {
  security_group_id            = aws_security_group.cluster.id
  ip_protocol                  = "tcp"
  from_port                    = 8000
  to_port                      = 8000
  referenced_security_group_id = aws_security_group.provisioner.id
}

output "cluster_security_group_id"     { value = aws_security_group.cluster.id }
output "provisioner_security_group_id" { value = aws_security_group.provisioner.id }

Apply the module, then capture:

output "cluster_security_group_id"     # sg-XXXXXXXXXXXXXXXXX
output "provisioner_security_group_id" # sg-YYYYYYYYYYYYYYYYY

Step 3: Deploy the cluster from the CONSUMER account

provider "qumulo" {
  aws {}
}

resource "qumulo_filesystem_aws" "cluster" {
  cluster_name                  = "sharedvpc"
  deployment_name               = "sharedvpc"
  region                        = "<region>"
  node_count                    = 5
  instance_type                 = "m6idn.2xlarge"

  # Owner-account VPC and subnet
  vpc_id                        = "<owner-vpc-id>"
  subnet_ids                    = ["<owner-shared-subnet-id>"]

  # Owner-account security groups (Bring-your-own)
  cluster_security_group_id     = "<owner-cluster-sg-id>"
  provisioner_security_group_id = "<owner-provisioner-sg-id>"

  admin_password                = var.admin_password
  cluster_product_type          = "HOT"
  deletion_protection           = true # recommended: guard the cluster's EC2 instances and S3 buckets

  # NOTE: allow_cidrs is forbidden when cluster_security_group_id /
  # provisioner_security_group_id are set. The owner account controls
  # ingress rules directly on the security groups.

  timeouts {
    create = "90m"
    delete = "30m"
  }
}

Apply, then verify:

TF_LOG=DEBUG terraform apply

In the debug log you should see:

• No CreateSecurityGroup calls.
• Skipping cluster security group creation (Bring-your-own) and the matching provisioner message.
• Cluster reaches ready after ~10 minutes.

Step 4: Update ingress rules later

Because allow_cidrs is not set in this mode, the resource has no way to push ingress changes into the owner-account SGs. To change the allowed CIDRs:

1. In the OWNER account, edit the aws_vpc_security_group_ingress_rule resources in the SG module.
2. terraform apply in the OWNER account.

No change is needed in the CONSUMER account; the cluster picks up the new rules immediately because it references the SGs by ID.

Step 5: Tear down

In the CONSUMER account:

terraform destroy

Expect:

• No DeleteSecurityGroup calls in the debug log (the SGs belong to the OWNER).
• Cluster instances and the provisioner instance are terminated.
• S3 buckets and IAM roles created by the provider are deleted.

In the OWNER account, when you no longer need the SGs:

terraform destroy   # in the SG module's directory

Common failure modes

Symptom	Cause	Fix
validating Bring-your-own cluster security group: security group "sg-X" is in VPC "vpc-Y", expected "vpc-Z"	The supplied SG ID doesn't live in the VPC you set on vpc_id.	Check the OWNER account; the SG must be in the same VPC you reference on the qumulo_filesystem_aws resource.
Incomplete Bring-your-own security group configuration	Only one of cluster_security_group_id / provisioner_security_group_id is set.	Set both, or omit both.
allow_cidrs cannot be combined with Bring-your-own security groups	allow_cidrs is set alongside the BYO IDs.	Remove allow_cidrs. Configure ingress on the OWNER-account SG resources directly.
Cluster boot times out / can't reach itself	Missing internode rule or cross-SG rules on the cluster SG.	Re-apply the SG module from Step 2; the rule set is exact.
S3 access denied during cluster bring-up	Missing S3 gateway endpoint on the OWNER subnet's route table.	Add com.amazonaws.<region>.s3 (Gateway type) and associate it with every route table used by the shared subnet.

What this pattern does NOT cover

• Cross-account IAM role assumption. This guide assumes the consumer account has its own IAM execution role for the cluster. The provider does not assume a role into the OWNER account.
• Owner-account S3 buckets. Persistent storage buckets are created in the CONSUMER account.
• Owner-account KMS keys. If you set kms_key_id, the key must be accessible to the consumer account's principals. See the AWS Custom Images doc for cross-account KMS notes (the same encryption-key sharing applies here).