Fleet Management Example

This example demonstrates enterprise-scale cluster fleet management using Qumulo's Cloud Data Fabric. A single Terraform configuration manages:

• Portal hub/spoke relationships between one hub and multiple regional spokes
• NFS exports across all clusters
• SMB shares across all clusters

Overview

The Cloud Data Fabric pattern distributes data from a central hub cluster to multiple spoke clusters across regions. Spokes cache data locally for low-latency access, and (with is_writable = true) can accept writes that sync back to the hub.

                    ┌─────────────────────┐
                    │   HUB CLUSTER       │
                    │   (central data)    │
                    │                     │
                    │  Portal Hub ────────┼───┐
                    │  NFS Export /data   │   │
                    │  SMB Share "data"   │   │
                    └─────────────────────┘   │
                              ▲               │
                              │               │  Bidirectional sync
           ┌──────────────────┼───────────────┼──────────────────┐
           │                  │               │                  │
           ▼                  ▼               ▼                  ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│  SPOKE: us-west │ │  SPOKE: us-east │ │  SPOKE: eu-west │ │  SPOKE: ap-east │
│  Portal Spoke   │ │  Portal Spoke   │ │  Portal Spoke   │ │  Portal Spoke   │
│  NFS Export     │ │  NFS Export     │ │  NFS Export     │ │  NFS Export     │
│  SMB Share      │ │  SMB Share      │ │  SMB Share      │ │  SMB Share      │
└─────────────────┘ └─────────────────┘ └─────────────────┘ └─────────────────┘

How It Works

Portal Handshake Flow

┌─────────────┐                              ┌─────────────┐
│   SPOKE     │                              │     HUB     │
│  CLUSTER    │                              │   CLUSTER   │
└──────┬──────┘                              └──────┬──────┘
       │                                            │
       │  1. qumulo_portal_spoke (PENDING)          │
       │  Creates portal relationship ──────────────│───┐
       │                                            │   │
       │                                            │   │
       │  2. qumulo_portal_hub (ACCEPTED)           │   │
       │ ◄──────────────────────────────────────────│───┘
       │  Accepts the relationship                  │
       │                                            │
       │  3. qumulo_portal_spoke_root               │
       │  Proposes directory pair ──────────────────│───┐
       │                                            │   │
       │  4. Update qumulo_portal_hub               │   │
       │ ◄──────────────────────────────────────────│───┘
       │  Authorizes root                           │
       │                                            │
       │  ═══════════════════════════════════════►  │
       │         Bidirectional data sync            │
       │  ◄═══════════════════════════════════════  │
       │                                            │

Terraform automates this handshake: qumulo_portal_spoke creates the relationship, qumulo_portal_hub accepts it, and qumulo_portal_spoke_root configures directory pairs.

Resource Dependencies

var.spoke_clusters
        │
        ▼
qumulo_portal_spoke.fleet[*]
        │  OUTPUT: spoke_cluster_uuid
        │
        ▼  (implicit dependency)
qumulo_portal_hub.fleet[*]
        │
        ▼  (depends_on)
qumulo_portal_spoke_root.fleet[*]

qumulo_nfs_export.hub + qumulo_nfs_export.spokes[*]
qumulo_smb_share.hub  + qumulo_smb_share.spokes[*]

Prerequisites

1. Multiple Qumulo clusters with REST API access (port 8000)
2. Network connectivity between hub and all spokes on port 3713 (bidirectional)
3. Admin credentials for all clusters (no Azure credentials required for cluster-only resources)

Network Requirements

                         ┌─────────────────┐
                         │   HUB CLUSTER   │
                         │   :3713 ◄───────┼─────┐
                         │   :8000 (API)   │     │
                         └────────┬────────┘     │
                                  │              │
            ┌─────────────────────┼──────────────┼─────────────────────┐
            │ :3713               │ :3713        │ :3713               │
            ▼                     ▼              ▼                     ▼
    ┌───────────────┐     ┌───────────────┐     ┌───────────────┐     ┌───────────────┐
    │   us-west     │     │   us-east     │     │   eu-west     │     │   ap-east     │
    │   :3713 ◄─────┼─────┼───► :3713 ◄───┼─────┼───► :3713 ◄───┼─────┼───► :3713     │
    │   :8000 (API) │     │   :8000 (API) │     │   :8000 (API) │     │   :8000 (API) │
    └───────────────┘     └───────────────┘     └───────────────┘     └───────────────┘

Usage

1. Create a terraform.tfvars file:

hub_cluster = {
  endpoint = "https://hub-cluster.example.com:8000"
  username = "admin"
  password = "hub-admin-password"
  address  = "hub-cluster.example.com"  # Hostname reachable from spokes
}

spoke_clusters = {
  us-west = {
    endpoint = "https://us-west-cluster.example.com:8000"
    username = "admin"
    password = "us-west-admin-password"
    address  = "us-west-cluster.example.com"
  }
  us-east = {
    endpoint = "https://us-east-cluster.example.com:8000"
    username = "admin"
    password = "us-east-admin-password"
    address  = "us-east-cluster.example.com"
  }
  eu-west = {
    endpoint = "https://eu-west-cluster.example.com:8000"
    username = "admin"
    password = "eu-west-admin-password"
    address  = "eu-west-cluster.example.com"
  }
}

# Filesystem path on the HUB where authoritative data lives (source of truth)
hub_root         = "/data"

# Filesystem path on each SPOKE where portal data appears (synced from hub)
spoke_cache_root = "/cache"

# NFS export path CLIENTS see (uniform across all clusters)
# On hub, exports hub_root; on spokes, exports spoke_cache_root
export_path      = "/data"

# SMB share name CLIENTS see (uniform across all clusters)
# On hub, shares hub_root; on spokes, shares spoke_cache_root
share_name       = "data"

1. Initialize and apply:

terraform init -upgrade
terraform plan
terraform apply

Variables

Name	Description	Type	Default
hub_cluster	Hub cluster connection details	object	-
spoke_clusters	Map of spoke clusters keyed by region	map(object)	-
hub_root	Filesystem path on the hub containing source data (distributed to spokes via portal)	string	/data
spoke_cache_root	Filesystem path on spokes where hub data appears via portal	string	/cache
export_path	NFS export path clients mount (same on all clusters, points to hub_root on hub, spoke_cache_root on spokes)	string	/data
share_name	SMB share name clients connect to (same on all clusters, serves hub_root on hub, spoke_cache_root on spokes)	string	data

Cluster Object Structure

{
  endpoint = string  # REST API URL (e.g., https://cluster:8000)
  username = string  # Admin username
  password = string  # Admin password
  address  = string  # Hostname/IP for portal sync (port 3713)
}

Outputs

Name	Description
portal_states	Portal state by spoke region
hub_states	Hub state by spoke region
nfs_export_ids	NFS export IDs by cluster
smb_share_ids	SMB share IDs by cluster
cluster_endpoints	Summary of all managed endpoints

Key Patterns

for_each with Maps

The spoke clusters are defined as a map, enabling for_each:

resource "qumulo_portal_spoke" "fleet" {
  for_each = var.spoke_clusters  # Iterate over the map

  connection_profile = each.key  # Use the spoke name as the profile name
  # ...
}

Separate Hub and Spoke Resources

NFS exports and SMB shares are created on both the hub and all spokes. Using separate resources keeps the code explicit and avoids conditional logic:

# Hub resource (single instance)
resource "qumulo_nfs_export" "hub" {
  connection_profile = "hub"
  fs_path            = var.hub_root        # Source data on hub
  export_path        = var.export_path     # Client-facing path
  # ...
}

# Spoke resources (one per spoke cluster)
resource "qumulo_nfs_export" "spokes" {
  for_each = var.spoke_clusters

  connection_profile = each.key
  fs_path            = var.spoke_cache_root  # Cached data on spokes
  export_path        = var.export_path       # Same client-facing path
  # ...
}

This pattern is clearer than using merge() with conditional fs_path logic because each resource explicitly shows its purpose.

Security Considerations

1. Credentials: Store sensitive values in environment variables or a secrets manager:

   export TF_VAR_hub_cluster='{"endpoint":"...","username":"admin","password":"secret","address":"..."}'
   

2. Export/Share Access: Consider adding host_restrictions to NFS exports and network_permission blocks to SMB shares to limit which clients can connect.

Adding a New Spoke

To add a new spoke cluster:

1. Add the cluster to spoke_clusters in your tfvars:

   spoke_clusters = {
     # existing spokes...
     ap-southeast = {
       endpoint = "https://ap-southeast-cluster:8000"
       username = "admin"
       password = "password"
       address  = "ap-southeast-cluster.example.com"
     }
   }
   

2. Run terraform apply. Terraform will:

3. Create a new spoke portal on the new cluster
4. Authorize it on the hub
5. Create NFS export and SMB share on the new cluster

Removing a Spoke

To remove a spoke:

1. Remove it from spoke_clusters in your tfvars
2. Run terraform apply. Terraform will destroy the portal and data access resources.

Troubleshooting

Spoke portal stuck in PENDING state

The hub authorization resource depends on the spoke's spoke_cluster_uuid. If authorization fails:

• Check network connectivity on port 3713
• Verify hub credentials
• Check hub cluster logs for authorization errors

Connection refused on port 3713

• Verify the address fields resolve correctly from each cluster's network

NFS/SMB resources fail to create

• Verify the filesystem paths (hub_root, spoke_cache_root) exist or that the cluster allows path creation
• Check cluster credentials have admin privileges

Full Configuration

# Fleet Management: Hub/Spoke Portal with Consistent Data Access
#
# This example demonstrates enterprise-scale cluster fleet management using
# Qumulo's Cloud Data Fabric. It configures:
#
#   - Portal hub/spoke relationships (one hub, multiple regional spokes)
#   - Root directory pairs for each spoke
#   - NFS exports across all clusters (hub and spokes)
#   - SMB shares across all clusters (hub and spokes)
#
# Architecture:
#
#                     ┌─────────────────────┐
#                     │   HUB CLUSTER       │
#                     │   (central data)    │
#                     │                     │
#                     │  Portal Hub ────────┼───┐
#                     │  NFS Export /data   │   │
#                     │  SMB Share "data"   │   │
#                     └─────────────────────┘   │
#                               ▲               │
#                               │               │  Bidirectional sync
#            ┌──────────────────┼───────────────┼──────────────────┐
#            │                  │               │                  │
#            ▼                  ▼               ▼                  ▼
#  ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
#  │  SPOKE: us-west │ │  SPOKE: us-east │ │  SPOKE: eu-west │ │  SPOKE: ap-east │
#  │  Portal Spoke   │ │  Portal Spoke   │ │  Portal Spoke   │ │  Portal Spoke   │
#  │  NFS Export     │ │  NFS Export     │ │  NFS Export     │ │  NFS Export     │
#  │  SMB Share      │ │  SMB Share      │ │  SMB Share      │ │  SMB Share      │
#  └─────────────────┘ └─────────────────┘ └─────────────────┘ └─────────────────┘
#
# Network requirements:
#   - Port 3713: Bidirectional between hub and all spokes (portal sync)
#   - Port 8000: Terraform host to all clusters (REST API)

terraform {
  required_version = ">= 1.0"
  required_providers {
    qumulo = {
      source  = "qumulo-terraform-registry.s3.us-east-1.amazonaws.com/qumulo/qumulo"
      version = "~> 1.0"
    }
  }
}

variable "hub_cluster" {
  description = "Hub cluster connection details"
  type = object({
    endpoint = string
    username = string
    password = string
    address  = string # Hostname/IP reachable from spokes for portal sync
  })
  sensitive = true
}

variable "spoke_clusters" {
  description = "Map of spoke clusters keyed by region/name (e.g., us-west, us-east)"
  type = map(object({
    endpoint = string
    username = string
    password = string
    address  = string # Hostname/IP reachable from hub for portal sync
  }))
  sensitive = true
}

variable "hub_root" {
  description = "Filesystem path on the hub cluster containing source data"
  type        = string
  default     = "/data"
}

variable "spoke_cache_root" {
  description = "Filesystem path on spoke clusters where portal data appears"
  type        = string
  default     = "/cache"
}

variable "export_path" {
  description = "NFS export path presented to clients (uniform across all clusters)"
  type        = string
  default     = "/data"
}

variable "share_name" {
  description = "SMB share name presented to clients (uniform across all clusters)"
  type        = string
  default     = "data"
}

# -----------------------------------------------------------------------------
# Provider Configuration with Connection Profiles
# -----------------------------------------------------------------------------

provider "qumulo" {
  connection_profiles = concat(
    [
      {
        name                 = "hub"
        endpoint             = var.hub_cluster.endpoint
        username             = var.hub_cluster.username
        password             = var.hub_cluster.password
        insecure_skip_verify = true
      }
    ],
    [
      for name, config in var.spoke_clusters : {
        name                 = name
        endpoint             = config.endpoint
        username             = config.username
        password             = config.password
        insecure_skip_verify = true
      }
    ]
  )
}

# -----------------------------------------------------------------------------
# Portal Spoke Resources
# -----------------------------------------------------------------------------
# Step 1: Create one spoke portal per hub/spoke cluster pair. Multiple
# qumulo_portal_spoke_root resources can share the same portal to replicate
# different directory trees.

resource "qumulo_portal_spoke" "fleet" {
  for_each = nonsensitive(var.spoke_clusters)

  connection_profile = each.key

  type        = "PORTAL_READ_WRITE" # Bidirectional: writes on spokes sync back to hub
  hub_address = var.hub_cluster.address
}

# -----------------------------------------------------------------------------
# Portal Hub Resources
# -----------------------------------------------------------------------------
# Step 2: Accept each spoke relationship on the hub cluster.

resource "qumulo_portal_hub" "fleet" {
  for_each = nonsensitive(var.spoke_clusters)

  connection_profile = "hub"

  spoke_cluster_uuid = qumulo_portal_spoke.fleet[each.key].spoke_cluster_uuid
  spoke_address      = each.value.address
}

# -----------------------------------------------------------------------------
# Portal Spoke Roots
# -----------------------------------------------------------------------------
# Step 3: Add root directory pairs for each spoke. You can add multiple
# qumulo_portal_spoke_root resources on the same portal to replicate different
# directory trees between the hub and spoke clusters.

resource "qumulo_portal_spoke_root" "fleet" {
  for_each = nonsensitive(var.spoke_clusters)

  connection_profile = each.key

  spoke_id        = qumulo_portal_spoke.fleet[each.key].id
  spoke_root_path = var.spoke_cache_root
  hub_root_path   = var.hub_root

  depends_on = [qumulo_portal_hub.fleet]
}

# -----------------------------------------------------------------------------
# NFS Exports
# -----------------------------------------------------------------------------

resource "qumulo_nfs_export" "hub" {
  connection_profile = "hub"

  export_path = var.export_path
  fs_path     = var.hub_root
  description = "Fleet data export (hub)"

  restriction {
    host_restrictions = ["*"]
    read_only         = false
  }
}

resource "qumulo_nfs_export" "spokes" {
  for_each = nonsensitive(var.spoke_clusters)

  connection_profile = each.key

  export_path = var.export_path
  fs_path     = var.spoke_cache_root
  description = "Fleet data export (${each.key})"

  restriction {
    host_restrictions = ["*"]
    read_only         = false
  }
}

# -----------------------------------------------------------------------------
# SMB Shares
# -----------------------------------------------------------------------------

resource "qumulo_smb_share" "hub" {
  connection_profile = "hub"

  share_name  = var.share_name
  fs_path     = var.hub_root
  description = "Fleet data share (hub)"
}

resource "qumulo_smb_share" "spokes" {
  for_each = nonsensitive(var.spoke_clusters)

  connection_profile = each.key

  share_name  = var.share_name
  fs_path     = var.spoke_cache_root
  description = "Fleet data share (${each.key})"
}

# -----------------------------------------------------------------------------
# Outputs
# -----------------------------------------------------------------------------

output "portal_states" {
  description = "Portal state by spoke region"
  value = {
    for k, v in qumulo_portal_spoke.fleet : k => {
      id     = v.id
      state  = v.state
      status = v.status
    }
  }
}

output "hub_states" {
  description = "Hub state by spoke region"
  value = {
    for k, v in qumulo_portal_hub.fleet : k => {
      id     = v.id
      state  = v.state
      status = v.status
    }
  }
}

output "nfs_export_ids" {
  description = "NFS export IDs by cluster"
  value = merge(
    { hub = qumulo_nfs_export.hub.id },
    { for k, v in qumulo_nfs_export.spokes : k => v.id }
  )
}

output "smb_share_ids" {
  description = "SMB share IDs by cluster"
  value = merge(
    { hub = qumulo_smb_share.hub.id },
    { for k, v in qumulo_smb_share.spokes : k => v.id }
  )
}

output "cluster_endpoints" {
  description = "Summary of all managed cluster endpoints"
  sensitive   = true
  value = {
    hub    = var.hub_cluster.endpoint
    spokes = { for k, v in var.spoke_clusters : k => v.endpoint }
  }
}