Customizing Cluster Boot¶

The qumulo_filesystem_aws, qumulo_filesystem_azure, and qumulo_filesystem_gcp resources let you customize the two shell scripts that boot a cluster:

the node boot script, which runs when a node instance boots and installs qumulo-core, and
the provisioner script, which runs on a short-lived orchestration VM and forms and configures the cluster.

Hooks (pre_run, post_run) splice your shell into the stock scripts at fixed anchor points; Qumulo's cluster-formation logic stays untouched. Typical uses: proxy/CA setup, bucket-policy fixes, monitoring agents, notifications.

Hooks live on two optional object attributes, node_hooks and provisioner_hooks, identical on all three resources:

resource "qumulo_filesystem_aws" "example" {
  # ... other required fields ...

  node_hooks = {
    pre_run = <<-EOT
      export https_proxy="http://proxy.corp.example.com:3128"
    EOT
  }

  provisioner_hooks = {
    post_run = file("${path.module}/hooks/notify.sh")
  }

  deletion_protection = true # recommended: guard the cluster's EC2 instances and S3 buckets

  timeouts {
    create = "90m"
    delete = "30m"
  }
}

Boot timeline¶

Hooks anchor at four fixed points. Both boot scripts are rendered during terraform apply, before any instance launches. Node instances and the provisioner VM then boot independently; the provisioner waits for the nodes partway through.

NODE INSTANCES (each node, at boot)
  |
  |  distro detection
  |  [node_hooks.pre_run]
  |  connectivity check
  |  package cache refresh, tools
  |  detect live cluster version
  |  download and install qumulo-core
  |  [node_hooks.post_run]
  v
node runs qfsd

PROVISIONER VM (one per operation)
  |
  |  error trap armed, deployment variables set
  |  [provisioner_hooks.pre_run]
  |  logging agent, base tooling, cloud CLI auth
  |  storage access validation
  |  wait for every node to answer on https://<node>:8000
  |  form or modify quorum
  |  configure cluster
  |  [provisioner_hooks.post_run]
  |  write completion signal
  v
provisioner self-terminates

Which anchor do I want?¶

Attribute	Field	Runs
`node_hooks`	`pre_run`	On each node, after distro detection, before the node script's first network operation (connectivity check, package-cache refresh, package download).
`node_hooks`	`post_run`	On each node, after qumulo-core is installed.
`provisioner_hooks`	`pre_run`	On the provisioner, after the error trap is armed and deployment variables are set, before the provisioner's first network operation (logging agent install, tooling installs, cloud CLI auth, storage-access validation).
`provisioner_hooks`	`post_run`	On the provisioner, after the cluster is formed and configured, right before the completion signal.

Both node_hooks.pre_run and provisioner_hooks.pre_run run before their script touches the network at all, so either is the right place to configure an HTTP proxy, a corporate CA, or an air-gapped package mirror. Because the hook is spliced into the same shell process, any exports or files it writes persist for the rest of the boot script.

The flip side: at pre_run time only the image's stock commands exist. The cloud CLI, jq, and the other tools the scripts install are not available yet, so anything CLI-dependent belongs in post_run (which runs after all tooling is installed and authenticated) or must install its own prerequisites.

Safety rules¶

Hooks run as root under set -xe

A failing command aborts the deploy. The scripts run with -e, so treat every hook line as load-bearing, or explicitly tolerate failure (|| true).
Never embed secrets. Hook text is stored in Terraform state, and -x tracing echoes every line into the boot logs (CloudWatch Logs on AWS, cloud-init output on Azure, Cloud Logging and the serial console on GCP). Failure details are also captured to SSM Parameter Store, Azure App Configuration, or the GCS status bucket. Get credentials from the instance's IAM role, managed identity, or service account, or from a cloud secret store.
Hooks must be idempotent. The provisioner script re-runs on scaling, node replacement, and other post-create operations; the node script runs on every node that boots (including nodes added later by scaling or replacement, and on GCP the startup script also re-runs when a node VM reboots).
Hook bodies are spliced verbatim. They are not templates and are not shell-escaped. A shell syntax error surfaces at boot, not at plan, so keep hooks in .sh files and run shellcheck on them.
A failure before the cloud CLI is ready has no way to report itself on AWS or Azure. provisioner_hooks.pre_run runs before the cloud CLI is installed and authenticated, so a failure there (including a failing pre_run hook itself) cannot write a status key, and terraform apply waits out its full timeout instead of failing fast. Consult the instance's boot log directly (CloudWatch Logs on AWS, cloud-init output on Azure) rather than waiting for the apply to fail. GCP is unaffected: it always reports success or failure through its status object.

Shell variables in scope¶

Hook bodies are spliced where these shell variables are already defined. Only the names below are a stable contract; anything else you spot in a rendered script may change without notice. Names follow each cloud's script, so they differ per cloud.

AWSAzureGCP

Provisioner script (provisioner_hooks):

Variable	Holds
`region`	AWS region
`deployment_name`	Deployment unique name (seed for cloud resource names and SSM paths)
`cluster_name`	qfsd cluster name
`cluster_version`	Qumulo Core version
`cluster_persistent_bucket_names`	Comma-separated S3 bucket names
`cluster_persistent_bucket_uris`	Comma-separated S3 bucket URIs
`cluster_persistent_storage_type`	`HOT` or `COLD`
`cluster_persistent_capacity_limit`	Soft capacity limit (TB)
`node_ips`, `existing_node_ips`, `final_node_ips`	Comma-separated node IPs: operation targets, stable existing nodes, nodes kept after the operation
`instance_ids`	Comma-separated EC2 instance IDs
`fault_domain_ids`	Comma-separated fault-domain IDs
`number_azs`	Availability-zone count
`float_ips`, `max_float_ips`	Floating IPs and the per-cluster maximum
`netmask`	Node netmask
`networking_mode`	`qumulo_managed` or `host_managed` networking mode
`s3_region`	Bucket region

Node script (node_hooks), in scope at pre_run:

Variable	Holds
`TEMPLATE_QUMULO_VERSION`	Target Qumulo Core version
`EXISTING_NODE_IPS`	Comma-separated existing cluster node IPs (empty on initial create)
`NETWORKING_MODE`	`qumulo_managed` or `host_managed` networking mode
`AWS_REGION`	AWS region
`DISTRO_ID`, `DISTRO_TYPE`, `PKG_FORMAT`, `PKG_INSTALL_CMD`	Distro-detection results

Provisioner script (provisioner_hooks):

Variable	Holds
`region`	Azure location
`deployment_name`	Deployment unique name
`cluster_name`	qfsd cluster name
`cluster_version`	Qumulo Core version
`cluster_product_type`	Qumulo product type
`cluster_object_storage_uri`	Object-storage URI list
`cluster_persistent_capacity_limit`	Soft capacity limit (TB)
`container_name`	Storage container name
`az_cloud_environment`	Azure cloud (`AzureCloud`, `AzureUSGovernment`, ...)
`managed_identity_client_id`	Managed-identity client ID
`node_ips`, `existing_node_ips`, `final_node_ips`	Comma-separated primary node IPs: operation targets, stable existing nodes, nodes kept after the operation
`instance_ids`	Comma-separated VM IDs
`num_azs`	Availability-zone count
`fault_domain_ids`	Comma-separated fault-domain IDs
`new_floating_ips`, `max_floating_ips`, `cluster_floating_ip_type`	Floating-IP settings
`expected_block_device_count`	Expected data-disk count
`keyvault_uri`	Key Vault URI
`subnet_id`, `subnet_cidr`, `network_security_group_id`	Network identifiers
`networking_mode`	`qumulo_managed` or `host_managed` networking mode

Node script (node_hooks), in scope at pre_run:

Variable	Holds
`TEMPLATE_QUMULO_VERSION`	Target Qumulo Core version
`EXISTING_NODE_IPS`	Comma-separated existing cluster node IPs (empty on initial create)
`NETWORKING_MODE`	`qumulo_managed` or `host_managed` networking mode
`DISTRO_ID`, `DISTRO_TYPE`, `PKG_FORMAT`, `PKG_INSTALL_CMD`	Distro-detection results

Provisioner script (provisioner_hooks):

Variable	Holds
`region`	GCP region
`deployment_name`	Deployment unique name
`cluster_name`	qfsd cluster name
`bucket_uris`	Comma-separated GCS bucket URIs
`capacity_limit_tb`	Soft capacity limit (TB)
`node_ips`, `existing_node_ips`, `final_node_ips`	Comma-separated node IPs: operation targets, stable existing nodes, nodes kept after the operation
`fault_domain_ids`	Comma-separated fault-domain IDs
`number_azs`	Zone count
`float_ips`	Floating IPs
`subnet_cidr`	Cluster subnet CIDR
`networking_mode`	`qumulo_managed` or `host_managed` networking mode

Node script (node_hooks), in scope at pre_run:

Variable	Holds
`TEMPLATE_QUMULO_VERSION`	Target Qumulo Core version
`EXISTING_NODE_IPS`	Comma-separated existing cluster node IPs (empty on initial create)
`NETWORKING_MODE`	`qumulo_managed` or `host_managed` networking mode
`DISTRO_ID`, `DISTRO_TYPE`, `PKG_FORMAT`, `PKG_INSTALL_CMD`	Distro-detection results

Two deliberate gaps:

The cluster admin password is set after pre_run and unset before post_run on every cloud, so it is never in scope for a hook.
The internal package download URLs are not resolved until after node pre_run; they depend on live-cluster version detection.

Examples¶

Every hook field is a plain string, so an inline heredoc (<<-EOT) and loading a script from a file with file(...) are interchangeable. Use heredocs for short snippets; keep longer scripts in their own .sh files so they get editor highlighting and shellcheck. The examples below show one of each.

Node hooks: route every node's package installs through the corporate proxy and trust the corporate CA before any download, then install a monitoring agent after qumulo-core is in place.

node_hooks = {
  # BEFORE the node script's first network call. Exports and written files
  # persist for the rest of boot, so cache refresh and core download use them.
  pre_run = <<-EOT
    export http_proxy="http://proxy.corp.example.com:3128"
    export https_proxy="$http_proxy"
    echo 'Acquire::http::Proxy "http://proxy.corp.example.com:3128";' > /etc/apt/apt.conf.d/95corp-proxy
    cp /opt/corp/ca/corp-root.crt /usr/local/share/ca-certificates/ && update-ca-certificates
  EOT

  # AFTER qumulo-core is installed.
  post_run = file("${path.module}/hooks/install-agent.sh")
}

Provisioner hooks: route the provisioner's egress through the corporate proxy before its first download (the reason pre_run exists), then notify an internal tracker once the cluster is up. Only distro-stock commands are available at pre_run; CLI-dependent work belongs in post_run.

provisioner_hooks = {
  # BEFORE the provisioner's first network operation. Everything after this,
  # including the cloud CLI install and Qumulo downloads, uses the proxy.
  pre_run = <<-EOT
    export https_proxy="http://proxy.corp.example.com:3128"
    export no_proxy="169.254.169.254,localhost,127.0.0.1"
  EOT

  # AFTER the cluster is formed and configured: report readiness to the CMDB.
  # Loaded from a file; the script can still reference the in-scope shell
  # variables (e.g. $deployment_name).
  post_run = file("${path.module}/hooks/provisioner-post-run.sh")
}

Where hooks/provisioner-post-run.sh contains:

#!/usr/bin/env bash
# provisioner_hooks.post_run - runs after the cluster is formed.
curl -fsS -X POST "https://cmdb.corp.example.com/api/clusters" \
  -H "Content-Type: application/json" \
  -d "{\"deployment\":\"${deployment_name}\",\"status\":\"ready\"}"

A complete runnable configuration is documented in the AWS user-data hooks example and lives in the provider repo under examples/aws-user-data-hooks/.

When hook changes take effect¶

Changing a hook is an in-place update: it never replaces the cluster and never re-runs boot on existing instances. The new value takes effect on the next event that runs the relevant script:

Provisioner hooks: the next operation that launches a provisioner VM (scaling, node replacement, storage updates, and similar).
Node hooks: the first boot of the next new node (added by scaling or replacement).