Cluster Replication Example

This example demonstrates how to set up data replication between Qumulo clusters and to S3-compatible object storage. Replication provides disaster recovery, data distribution, and archival capabilities.

Features

Four replication patterns are demonstrated:

1. Source Relationship - Initiates replication FROM this cluster to a target
2. Target Relationship - Authorizes replication TO this cluster from a source
3. Blackout Windows - Schedules replication around business hours
4. Object Storage - Replicates to/from S3-compatible storage (backup/archive)

Architecture

Cluster-to-Cluster Replication

┌─────────────────────────────────────────────────────────────────────────────┐
│                         REPLICATION FLOW                                     │
└─────────────────────────────────────────────────────────────────────────────┘

    SOURCE CLUSTER                                      TARGET CLUSTER
┌───────────────────────┐                         ┌───────────────────────┐
│                       │                         │                       │
│  /data ───────────────┼─── Port 3712 ──────────►│ /replica              │
│  (source_root_path)   │                         │ (target_root_path)    │
│                       │    1. Create source     │                       │
│  qumulo_              │       relationship      │  qumulo_              │
│  replication_source_  │                         │  replication_target_  │
│  relationship         │    2. Authorize on      │  relationship         │
│                       │       target            │                       │
│                       │                         │                       │
│  Terraform applies    │    3. Data flows        │  Terraform applies    │
│  via REST API :8000   │       continuously      │  via REST API :8000   │
│                       │                         │                       │
└───────────────────────┘                         └───────────────────────┘

           ▲                                                 ▲
           │                                                 │
           │                                                 │
     ┌─────┴─────┐                                     ┌─────┴─────┐
     │ Terraform │                                     │ Terraform │
     │  Host     │                                     │  (same)   │
     └───────────┘                                     └───────────┘

Object Storage Replication

┌─────────────────────────────────────────────────────────────────────────────┐
│                    OBJECT STORAGE REPLICATION                                │
└─────────────────────────────────────────────────────────────────────────────┘

    QUMULO CLUSTER                              S3-COMPATIBLE STORAGE
┌───────────────────────┐                    ┌───────────────────────┐
│                       │                    │                       │
│  /data/backup ────────┼─── COPY_TO_OBJECT ─►│  bucket/qumulo-backup │
│                       │    (archive)       │                       │
│                       │                    │                       │
│  /data/restored ◄─────┼── COPY_FROM_OBJECT─┼──bucket/qumulo-archive│
│                       │    (restore)       │                       │
│                       │                    │                       │
│  qumulo_              │    HTTPS :443      │  AWS S3, MinIO,       │
│  replication_object_  │                    │  Wasabi, etc.         │
│  relationship         │                    │                       │
│                       │                    │                       │
└───────────────────────┘                    └───────────────────────┘

Prerequisites

1. Qumulo clusters with REST API access (port 8000)
2. Network connectivity between clusters on port 3712 (bidirectional)
3. Admin credentials for all clusters
4. S3 credentials for object storage replication (optional)

Network Requirements

Port	Direction	Purpose
3712	Source -> Target	Replication data transfer
8000	Terraform -> Clusters	REST API management
443	Cluster -> S3	Object storage replication

Usage

1. Create a terraform.tfvars file:

source_cluster = {
  endpoint = "https://source-cluster.example.com:8000"
  username = "admin"
  password = "source-admin-password"
}

target_cluster = {
  endpoint = "https://target-cluster.example.com:8000"
  username = "admin"
  password = "target-admin-password"
  address  = "target-cluster.example.com"  # Reachable from source
}

s3_config = {
  endpoint          = "s3.amazonaws.com"
  bucket            = "my-qumulo-backup"
  region            = "us-west-2"
  access_key_id     = "AKIAIOSFODNN7EXAMPLE"
  secret_access_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}

1. Initialize and apply:

terraform init -upgrade
terraform plan
terraform apply

Variables

Name	Description	Type	Default
source_cluster	Source cluster connection details	object	-
target_cluster	Target cluster connection details (includes address)	object	-
s3_config	S3-compatible storage configuration	object	-

Cluster Object Structure

# Source cluster
{
  endpoint = string  # REST API URL (e.g., https://cluster:8000)
  username = string  # Admin username
  password = string  # Admin password
}

# Target cluster (includes address for replication)
{
  endpoint = string  # REST API URL
  username = string  # Admin username
  password = string  # Admin password
  address  = string  # Hostname/IP reachable from source (port 3712)
}

Outputs

Name	Description
source_relationship_id	ID of the source replication relationship
source_root_id	File ID of the source directory
target_relationship_state	State of the target relationship (ESTABLISHED when ready)
target_root_path	Path on target cluster where data is replicated
object_backup_relationship_id	ID of the object storage backup relationship
object_restore_relationship_id	ID of the object storage restore relationship

Replication Patterns

1. Basic Cluster-to-Cluster

# On source cluster: initiate replication
resource "qumulo_replication_source_relationship" "primary" {
  connection_profile = "source"

  target_address   = "target-cluster.example.com"
  target_root_path = "/replica"
  source_root_path = "/data"

  trigger_initial_replication = true
  replication_enabled         = true
}

# On target cluster: authorize the relationship
resource "qumulo_replication_target_relationship" "replica" {
  depends_on = [qumulo_replication_source_relationship.primary]

  connection_profile   = "target"
  id                   = qumulo_replication_source_relationship.primary.id
  allow_fs_path_create = true
}

2. Blackout Windows

Pause replication during business hours to preserve bandwidth:

resource "qumulo_replication_source_relationship" "with_blackout" {
  connection_profile = "source"

  target_address   = "target-cluster.example.com"
  target_root_path = "/replica-scheduled"
  source_root_path = "/data-scheduled"

  blackout_window_timezone = "America/Los_Angeles"

  blackout_window {
    start_hour   = 9
    start_minute = 0
    end_hour     = 17
    end_minute   = 0
    on_days      = ["MON", "TUE", "WED", "THU", "FRI"]
  }
}

3. Snapshot-Based Replication

Link replication to a snapshot policy for point-in-time consistency:

resource "qumulo_replication_source_relationship" "snapshot_based" {
  connection_profile = "source"

  target_address   = "target-cluster.example.com"
  target_root_path = "/replica-snapshots"
  source_root_path = "/data-snapshots"

  replication_mode = "REPLICATION_SNAPSHOT_POLICY"

  snapshot_policy {
    id                = 1  # Existing snapshot policy ID
    target_expiration = "7days"
  }
}

4. Object Storage (Backup/Archive)

# Backup TO S3
resource "qumulo_replication_object_relationship" "backup" {
  connection_profile = "source"

  direction            = "COPY_TO_OBJECT"
  local_directory_path = "/data/backup"

  object_store_address = "s3.amazonaws.com"
  bucket               = "my-backup-bucket"
  region               = "us-west-2"
  access_key_id        = var.s3_access_key
  secret_access_key    = var.s3_secret_key

  bucket_style  = "BUCKET_STYLE_PATH"
  object_folder = "qumulo-backup"
}

# Restore FROM S3
resource "qumulo_replication_object_relationship" "restore" {
  connection_profile = "source"

  direction            = "COPY_FROM_OBJECT"
  local_directory_path = "/data/restored"

  object_store_address = "s3.amazonaws.com"
  bucket               = "my-archive-bucket"
  region               = "us-west-2"
  access_key_id        = var.s3_access_key
  secret_access_key    = var.s3_secret_key

  object_folder = "qumulo-archive"
}

Security Considerations

1. Credentials: Store cluster and S3 credentials securely:

   export TF_VAR_source_cluster='{"endpoint":"...","username":"admin","password":"secret"}'
   export TF_VAR_s3_config='{"endpoint":"...","access_key_id":"...","secret_access_key":"..."}'
   

2. Encryption: Data in transit uses TLS. For object storage, ensure the bucket has server-side encryption enabled.

3. IAM Policies: For S3 replication, use IAM roles with minimal permissions:

   {
     "Version": "2012-10-17",
     "Statement": [{
       "Effect": "Allow",
       "Action": ["s3:GetObject", "s3:PutObject", "s3:ListBucket"],
       "Resource": ["arn:aws:s3:::my-bucket", "arn:aws:s3:::my-bucket/*"]
     }]
   }
   

Full Configuration

# Example: Cluster-to-Cluster and Object Storage Replication
#
# This example demonstrates how to set up replication between Qumulo clusters
# and to S3-compatible object storage using the replication resources.
#
# Three types of replication are shown:
# 1. Source relationship - defines replication FROM this cluster
# 2. Target relationship - authorizes replication TO this cluster
# 3. Object relationship - replicates to/from S3-compatible storage

terraform {
  required_providers {
    qumulo = {
      source  = "qumulo-terraform-registry.s3.us-east-1.amazonaws.com/qumulo/qumulo"
      version = "~> 1.0"
    }
  }
}

variable "source_cluster" {
  description = "Source cluster connection details"
  type = object({
    endpoint = string
    username = string
    password = string
  })
  sensitive = true
}

variable "target_cluster" {
  description = "Target cluster connection details"
  type = object({
    endpoint = string
    username = string
    password = string
    address  = string # IP/hostname reachable from source cluster
  })
  sensitive = true
}

variable "s3_config" {
  description = "S3-compatible storage configuration for object replication"
  type = object({
    endpoint          = string
    bucket            = string
    region            = string
    access_key_id     = string
    secret_access_key = string
  })
  sensitive = true
}

# Define connection profiles for both clusters
provider "qumulo" {
  connection_profiles = [
    {
      name                 = "source"
      endpoint             = var.source_cluster.endpoint
      username             = var.source_cluster.username
      password             = var.source_cluster.password
      insecure_skip_verify = true # Required for clusters with self-signed certificates
    },
    {
      name                 = "target"
      endpoint             = var.target_cluster.endpoint
      username             = var.target_cluster.username
      password             = var.target_cluster.password
      insecure_skip_verify = true # Required for clusters with self-signed certificates
    }
  ]
}

# =============================================================================
# Cluster-to-Cluster Replication
# =============================================================================

# Step 1: Create the source relationship on the source cluster
# This initiates replication to the target cluster
resource "qumulo_replication_source_relationship" "primary" {
  connection_profile = "source"

  # Required: Define what to replicate and where
  target_address   = var.target_cluster.address
  target_root_path = "/replica"
  source_root_path = "/data"

  # Optional: Trigger initial replication after authorization
  trigger_initial_replication = true

  # Optional: Configure replication behavior
  replication_enabled = true
  target_port         = 3712
}

# Step 2: Authorize the relationship on the target cluster
# IMPORTANT: Use depends_on to ensure source is created first
resource "qumulo_replication_target_relationship" "replica" {
  depends_on = [qumulo_replication_source_relationship.primary]

  connection_profile = "target"

  # The ID comes from the source relationship
  id = qumulo_replication_source_relationship.primary.id

  # Optional: Allow target directory creation if it doesn't exist
  allow_fs_path_create = true
}

# =============================================================================
# Cluster-to-Cluster Replication with Blackout Windows
# =============================================================================

# Replication with scheduled blackout windows (e.g., business hours)
resource "qumulo_replication_source_relationship" "with_blackout" {
  connection_profile = "source"

  target_address   = var.target_cluster.address
  target_root_path = "/replica-scheduled"
  source_root_path = "/data-scheduled"

  # Timezone for blackout windows
  blackout_window_timezone = "America/Los_Angeles"

  # Pause replication during business hours (9 AM - 5 PM weekdays)
  blackout_window {
    start_hour   = 9
    start_minute = 0
    end_hour     = 17
    end_minute   = 0
    on_days      = ["MON", "TUE", "WED", "THU", "FRI"]
  }
}

# =============================================================================
# Snapshot Policy Replication
# =============================================================================

# Replication linked to a snapshot policy
resource "qumulo_replication_source_relationship" "snapshot_based" {
  connection_profile = "source"

  target_address   = var.target_cluster.address
  target_root_path = "/replica-snapshots"
  source_root_path = "/data-snapshots"

  # Use snapshot policy mode
  replication_mode = "REPLICATION_SNAPSHOT_POLICY"

  # Link to an existing snapshot policy (policy ID must exist on cluster)
  snapshot_policy {
    id                = 1 # Replace with actual snapshot policy ID
    target_expiration = "7days"
  }
}

# =============================================================================
# Object Storage Replication (S3-Compatible)
# =============================================================================

# Copy data TO S3-compatible object storage (backup/archive)
resource "qumulo_replication_object_relationship" "backup_to_s3" {
  connection_profile = "source"

  # Direction: copy from cluster to object storage
  direction            = "COPY_TO_OBJECT"
  local_directory_path = "/data/backup"

  # S3 configuration
  object_store_address = var.s3_config.endpoint
  bucket               = var.s3_config.bucket
  region               = var.s3_config.region
  access_key_id        = var.s3_config.access_key_id
  secret_access_key    = var.s3_config.secret_access_key

  # Optional settings
  bucket_style  = "BUCKET_STYLE_PATH"
  object_folder = "qumulo-backup"
  port          = 443
}

# Copy data FROM S3-compatible object storage (restore)
resource "qumulo_replication_object_relationship" "restore_from_s3" {
  connection_profile = "source"

  # Direction: copy from object storage to cluster
  direction            = "COPY_FROM_OBJECT"
  local_directory_path = "/data/restored"

  # S3 configuration
  object_store_address = var.s3_config.endpoint
  bucket               = var.s3_config.bucket
  region               = var.s3_config.region
  access_key_id        = var.s3_config.access_key_id
  secret_access_key    = var.s3_config.secret_access_key

  bucket_style  = "BUCKET_STYLE_PATH"
  object_folder = "qumulo-archive"
}

# =============================================================================
# Outputs
# =============================================================================

output "source_relationship_id" {
  description = "ID of the source replication relationship"
  value       = qumulo_replication_source_relationship.primary.id
}

output "source_root_id" {
  description = "File ID of the source directory"
  value       = qumulo_replication_source_relationship.primary.source_root_id
}

output "target_relationship_state" {
  description = "State of the target relationship (ESTABLISHED when ready)"
  value       = qumulo_replication_target_relationship.replica.state
}

output "target_root_path" {
  description = "Path on target cluster where data is replicated"
  value       = qumulo_replication_target_relationship.replica.target_root_path
}

output "object_backup_relationship_id" {
  description = "ID of the object storage backup relationship"
  value       = qumulo_replication_object_relationship.backup_to_s3.id
}

output "object_restore_relationship_id" {
  description = "ID of the object storage restore relationship"
  value       = qumulo_replication_object_relationship.restore_from_s3.id
}