AWS Multi-AZ NLB

This page shows how to put an AWS Network Load Balancer (NLB) in front of a multi-AZ Qumulo cluster created with qumulo_filesystem_aws. It is the recommended pattern for clients that need a single, AZ-resilient endpoint.

When you need this

Multi-AZ AWS clusters expose endpoint_ips as the per-node primary IPs (one IP per node). On single-AZ clusters, floating_ip_count provides failover IPs through AWS secondary private IPs. Secondary private IPs are bound to a single subnet and cannot move across availability zones, so a multi-AZ deployment has no built-in failover IP. An NLB in front of the cluster gives clients a stable DNS name that fails over across AZs automatically.

• Multi-AZ cluster: an NLB is the standard way to expose a single endpoint. This page is written around that case.
• Single-AZ cluster: use floating_ip_count on qumulo_filesystem_aws instead. An NLB is also valid if you specifically want a stable DNS name, but it is optional.

Prerequisites

• A working qumulo_filesystem_aws cluster. See aws-production.md for the full multi-AZ cluster setup.
• The cluster's primary_ips output (one IP per node).
• VPC ID and the private subnet IDs to place the NLB in (typically the same subnets the cluster runs in).

Full example

The example below is one self-contained Terraform configuration: providers, a multi-AZ cluster, and a fully wired NLB with listeners and target groups for every Qumulo client and admin port. Copy it into a fresh main.tf, fill in variable values, and terraform init && terraform validate should pass.

terraform {
  required_providers {
    qumulo = {
      source  = "qumulo-terraform-registry.s3.us-east-1.amazonaws.com/qumulo/qumulo"
      version = "~> 1.0"
    }
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "qumulo" {
  aws {}
}

provider "aws" {
  region = var.region
}

variable "region" {
  description = "AWS region for deployment"
  type        = string
  default     = "us-east-1"
}

variable "vpc_id" {
  description = "VPC ID for the cluster and NLB"
  type        = string
}

variable "subnet_ids" {
  description = "Private subnet IDs for multi-AZ deployment (3+ subnets, one per AZ). The NLB and cluster share these subnets."
  type        = list(string)
}

variable "admin_password" {
  description = "Administrator password for cluster access"
  type        = string
  sensitive   = true
}

variable "allow_cidrs" {
  description = "CIDR blocks allowed to access the cluster"
  type        = list(string)
}

variable "ec2_key_pair" {
  description = "EC2 key pair name for SSH access"
  type        = string
}

variable "nexus_registration_key" {
  description = "Qumulo Nexus registration key"
  type        = string
  sensitive   = true
}

variable "node_count" {
  description = "Number of cluster nodes. Drives both the cluster size and the NLB target group attachment fan-out."
  type        = number
  default     = 6
}

# Minimal multi-AZ Qumulo cluster. See aws-production.md for the full set of
# cluster knobs (KMS, permissions boundary, custom AMIs, etc.).
resource "qumulo_filesystem_aws" "cluster" {
  cluster_name           = "qnlbdemo"
  deployment_name        = "qnlbdemo"
  region                 = var.region
  vpc_id                 = var.vpc_id
  subnet_ids             = var.subnet_ids
  node_count             = var.node_count
  instance_type          = "m6idn.4xlarge"
  cluster_product_type   = "HOT"
  soft_capacity_limit_tb = 5000
  ec2_key_pair           = var.ec2_key_pair
  admin_password         = var.admin_password
  nexus_registration_key = var.nexus_registration_key
  allow_cidrs            = var.allow_cidrs
  deletion_protection    = true # recommended: guard the cluster's EC2 instances and S3 buckets

  timeouts {
    create = "90m"
    delete = "30m"
  }
}

# -----------------------------------------------------------------------------
# NLB in front of the cluster.
# -----------------------------------------------------------------------------

locals {
  # One entry per Qumulo port the NLB should expose. Drop entries you don't need
  # (for example, replication-a / replication-b are only required if the NLB
  # also fronts replication traffic). Keys interpolate into AWS target group
  # names, so they use hyphens — AWS rejects underscores in LB resource names.
  qumulo_ports = {
    ssh           = { port = 22, protocol = "TCP", preserve_client_ip = true }
    http          = { port = 80, protocol = "TCP", preserve_client_ip = true }
    nfs-portmap   = { port = 111, protocol = "TCP_UDP", preserve_client_ip = true }
    https         = { port = 443, protocol = "TCP", preserve_client_ip = true }
    smb           = { port = 445, protocol = "TCP", preserve_client_ip = true }
    nfs           = { port = 2049, protocol = "TCP_UDP", preserve_client_ip = true }
    replication-a = { port = 3712, protocol = "TCP", preserve_client_ip = true }
    replication-b = { port = 3713, protocol = "TCP", preserve_client_ip = true }
    api           = { port = 8000, protocol = "TCP", preserve_client_ip = true }
    s3            = { port = 9000, protocol = "TCP", preserve_client_ip = true }
  }

  # Cross product of (port, node index) for target group attachments. The
  # enumeration is driven by var.node_count (known at plan time) rather than
  # iterating qumulo_filesystem_aws.cluster.primary_ips directly. primary_ips
  # is "(known after apply)" on first deploy, and Terraform rejects for_each
  # over unknown map keys, which would otherwise force a two-pass apply. The
  # node IP itself is referenced as a target_id value below, where unknown
  # values are allowed.
  qumulo_attachments = merge([
    for port_key, _ in local.qumulo_ports : {
      for node_idx in range(var.node_count) :
      "${port_key}-${node_idx}" => { port_key = port_key, node_idx = node_idx }
    }
  ]...)
}

resource "aws_lb" "qumulo" {
  name                             = "qumulo-nlb"
  internal                         = true
  load_balancer_type               = "network"
  subnets                          = var.subnet_ids
  enable_cross_zone_load_balancing = true
  ip_address_type                  = "ipv4"

  tags = {
    Name      = "qumulo-nlb"
    ManagedBy = "Terraform"
  }
}

resource "aws_lb_target_group" "qumulo" {
  for_each = local.qumulo_ports

  name               = "qumulo-${each.key}"
  port               = each.value.port
  protocol           = each.value.protocol
  target_type        = "ip"
  vpc_id             = var.vpc_id
  preserve_client_ip = each.value.preserve_client_ip

  # source_ip stickiness is valid for TCP, TLS, and UDP target groups, but
  # not for TCP_UDP. Skip it on the TCP_UDP groups (NFS portmap and NFS),
  # which would otherwise be rejected at apply time.
  dynamic "stickiness" {
    for_each = each.value.protocol == "TCP_UDP" ? [] : [1]
    content {
      enabled = true
      type    = "source_ip"
    }
  }
}

resource "aws_lb_target_group_attachment" "qumulo" {
  for_each = local.qumulo_attachments

  target_group_arn = aws_lb_target_group.qumulo[each.value.port_key].arn
  target_id        = qumulo_filesystem_aws.cluster.primary_ips[each.value.node_idx]
  port             = local.qumulo_ports[each.value.port_key].port
}

resource "aws_lb_listener" "qumulo" {
  for_each = local.qumulo_ports

  load_balancer_arn = aws_lb.qumulo.arn
  port              = each.value.port
  protocol          = each.value.protocol

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.qumulo[each.key].arn
  }
}

# -----------------------------------------------------------------------------
# Outputs
# -----------------------------------------------------------------------------

output "cluster_uuid" {
  description = "UUID of the Qumulo cluster"
  value       = qumulo_filesystem_aws.cluster.cluster_uuid
}

output "deployment_unique_name" {
  description = "Unique deployment identifier"
  value       = qumulo_filesystem_aws.cluster.deployment_unique_name
}

output "endpoint_ips" {
  description = "Client-facing IPs as the resource sees them. Multi-AZ AWS clusters return primary IPs here. Clients should use nlb_dns instead."
  value       = qumulo_filesystem_aws.cluster.endpoint_ips
}

output "primary_ips" {
  description = "Per-node primary IPs. Same set used by the NLB target group attachments above."
  value       = qumulo_filesystem_aws.cluster.primary_ips
}

output "nlb_dns" {
  description = "DNS name of the NLB. Use as the cluster endpoint for clients."
  value       = aws_lb.qumulo.dns_name
}

output "endpoints" {
  description = "Pre-formatted connection strings for client protocols."
  value = {
    web_ui = "https://${aws_lb.qumulo.dns_name}"
    api    = "https://${aws_lb.qumulo.dns_name}:8000"
    nfs    = "${aws_lb.qumulo.dns_name}:/<export>"
    smb    = "\\\\${aws_lb.qumulo.dns_name}\\<share>"
    s3     = "https://${aws_lb.qumulo.dns_name}:9000"
  }
}

Recommended defaults

The example sets four knobs that are not the AWS defaults. They are the recommended values for a multi-AZ failover deployment, with tradeoffs called out:

Setting	Value	Why	Tradeoff
enable_cross_zone_load_balancing on aws_lb	true	Without it, a client connection arriving on the LB ENI in one AZ only routes to nodes in that AZ. Lose the AZ and the client has nowhere to fail over. Cross-zone load balancing is the entire point of putting an NLB in front of a multi-AZ cluster.	AWS charges inter-AZ data transfer for cross-zone NLB traffic. Budget for it.
stickiness { enabled = true, type = "source_ip" } on each aws_lb_target_group	enabled on TCP groups; skipped on TCP_UDP groups (NFS portmap and NFS) because AWS only supports source_ip stickiness on TCP and TLS protocols	SMB multichannel and NFSv4 state benefit from a client returning to the same node across reconnects. NFS uses NFSv4 over TCP in practice; UDP traffic on the TCP_UDP listeners is best-effort and stateless anyway.	A NAT'd fleet sharing one source IP lands on a single node. Fine for correctness, suboptimal for throughput in narrow cases.
preserve_client_ip = true on each aws_lb_target_group	enabled everywhere	Qumulo audit logs and identity mapping rely on real client source IPs.	The cluster security group must allow the listener ports from the client CIDR, not the NLB ENI. See the security group note below.
internal = true on aws_lb	true	Qumulo clusters live in private subnets; clients reach the cluster VPC-internally.	If a public-facing endpoint is needed, set false. Consider WAF, security groups, and certificate management on top; that is outside this guide's scope.

Security group note

With preserve_client_ip = true (the recommended setting above), AWS forwards traffic to the cluster nodes with the client's source IP, not the NLB's ENI IPs. This means the security group on the cluster nodes must allow ingress on the listener ports from the client CIDRs, not from the NLB. The provider attaches a default security group to qumulo_filesystem_aws.cluster that honors allow_cidrs. If you need additional ports or sources, attach extra SGs via additional_security_group_ids on the cluster resource. See qumulo_filesystem_aws for the cluster-side options.

If you set preserve_client_ip = false instead, the cluster SG only needs to allow the NLB's ENI IPs, but you lose accurate client identity in audit logs and identity mapping. The example keeps it true everywhere for that reason.

Optional: friendly DNS name

The NLB ships with an AWS-generated DNS name (qumulo-nlb-xxxxxxxxxx.elb.<region>.amazonaws.com). Most customers want a friendlier name. Add a Route53 alias record:

data "aws_route53_zone" "internal" {
  name         = "internal.example.com"
  private_zone = true
}

resource "aws_route53_record" "qumulo" {
  zone_id = data.aws_route53_zone.internal.zone_id
  name    = "qumulo.internal.example.com"
  type    = "A"

  alias {
    name                   = aws_lb.qumulo.dns_name
    zone_id                = aws_lb.qumulo.zone_id
    evaluate_target_health = true
  }
}

This is optional. Skip it and clients can use the NLB's auto-generated DNS name directly via output.nlb_dns.

Health checks

The example does not declare a health_check {} block on the target groups. The NLB target group defaults work for Qumulo:

• TCP probe on the target port
• 30 second interval
• 10 second timeout
• 2/2 healthy and unhealthy thresholds

A node that fails the TCP probe is removed from rotation; a recovered node is re-added on the next successful probe.

If you want HTTPS-based health checks against the API port (8000), add health_check { protocol = "HTTPS", port = "8000", path = "/v1/version" } to the relevant target group. This is an extension, not the recommended path.

Outputs

• cluster_uuid: UUID of the Qumulo cluster
• deployment_unique_name: Unique deployment identifier
• endpoint_ips: Client-facing IPs as the resource sees them. Multi-AZ AWS clusters return primary IPs here. Clients should use nlb_dns instead.
• primary_ips: Per-node primary IPs. Same set used by the NLB target group attachments above.
• nlb_dns: DNS name of the NLB. Use as the cluster endpoint for clients.
• endpoints: Pre-formatted connection strings for client protocols (web UI, REST API, NFS, SMB, S3)

Two outputs matter for connecting clients: nlb_dns, which you can use directly for client mounts or alias via Route53 (above), and endpoints, whose NFS and SMB strings include placeholders (<export>, <share>) to replace with the actual export path or share name from your cluster.

Caveats

Gotchas worth knowing before you ship this to production:

• NFSv3 locking is unreliable through an NLB. The example exposes ports 111 and 2049 (TCP_UDP) for completeness, but NFSv3's lock manager (nlm) and status monitor (statd) use ephemeral RPC ports the LB does not see, and source-IP stickiness alone is not enough to keep a client's lock state on a single node across reconnects. Use NFSv4 if you need locks behind the LB. NFSv4 multiplexes everything over port 2049 and is well-behaved through an NLB.
• Cross-zone load balancing has an inter-AZ data transfer cost. This guide enables it (see the defaults table). Without it, the multi-AZ failover guarantees this page documents do not hold. Budget the inter-AZ traffic accordingly.
• Stickiness is source-IP-based. Acceptable for typical client fleets. Pathological for environments where many clients sit behind a single NAT IP: they all land on the same backend node.
• Replication ports (3712/3713) are only needed if the NLB fronts replication traffic. If your replication peers reach the cluster directly via node IPs, drop those entries from local.qumulo_ports and the matching listener and target groups disappear automatically.
• Single-AZ deployments don't need this. Use floating_ip_count on qumulo_filesystem_aws for in-cluster IP failover instead. An NLB in front of a single-AZ cluster is supported but is not a failover requirement.