Scaling

As you start adding more resources to your configuration a few problems can arise:

• The Terraform state file becomes bigger and making changes with Terraform takes much longer.
• A single shared statefile is a risk. Making a change in a Development tenant could have implications to a Production tenant.
• No ability to run changes in parallel. Only one concurrent plan may run at any given time as the statefile is locked during the operation.

These problems are not unique to Network as Code and would occur when scaling any Terraform resources using a single state file. To address these problems users have to consider distributing state. For more information about Terraform state see: https://developer.hashicorp.com/terraform/language/state.

In Catalyst Center environments, this becomes particularly relevant when managing large deployments with multiple sites, fabric sites, device provisioning, IP pools, network profiles, wireless configurations, and templates. The Catalyst Center Terraform module supports managing different site configurations independently, allowing you to separate your infrastructure management by geographical locations, organizational boundaries, or functional areas.

When managing large Catalyst Center deployments across multiple sites, using a single Terraform state file can become problematic. As your deployment grows, several challenges arise:

• Performance: Large state files with hundreds of sites, devices, and configurations slow down Terraform operations significantly
• Risk Management: Changes to one site’s configuration could inadvertently affect other sites or global settings
• Parallel Operations: Only one Terraform operation can run at a time with a shared state, limiting deployment velocity
• Team Collaboration: Different teams managing different sites (e.g., regional teams) need isolation and independent deployment capabilities
• Blast Radius: Configuration errors or failed deployments affect the entire infrastructure rather than isolated components

Multistate configuration addresses these issues by separating the Terraform state while maintaining a shared data model for consistency across your Catalyst Center deployment.

The multistate approach separates your infrastructure into logical boundaries:

• COMMON: Manages global settings, templates, and parent sites
• Site-specific folders: Each site (Krakow, Warsaw, etc.) manages its own resources
• Shared data model: All instances use the same YAML configuration files

Here’s the recommended folder structure for a multistate Catalyst Center deployment:

The Catalyst Center Terraform module provides several flags to control what resources each instance manages:

• Type: Boolean (default: true)
• Purpose: Controls whether this instance manages global configuration items
• Rule: Only one Terraform instance should have this set to true
• Manages: Templates, global settings, global IP pools, L3 VNs

• Type: List of strings
• Purpose: Specifies which sites this instance should manage
• Format: Site hierarchy path (e.g., ["Global/Poland/Krakow"] and by default, it will also include all child sites)

• Type: Boolean (default: false)
• Purpose: When true, only manages the specified sites, not their children
• Use case: Prevents managing child sites when you only want to manage specific sites

Consider this site hierarchy:

Global
└── Poland
    ├── Krakow
    │   ├── Building_A
    │   └── Building_B
    └── Warsaw
        ├── Office_1
        └── Office_2

The COMMON folder manages global settings and the parent Poland site:

module "catalystcenter" {
  source  = "netascode/nac-catalystcenter/catalystcenter"
  version = "0.2.0"

  yaml_directories = ["../data/"]
  templates_directories = ["../data/templates/"]

  manage_global_settings = true
  managed_sites = ["Global/Poland"]
  manage_specific_sites_only = true
}

Key points for COMMON configuration:

• manage_global_settings = true: This instance manages global resources
• managed_sites = ["Global/Poland"]: Specify global parent site
• manage_specific_sites_only = true: Prevents managing Krakow and Warsaw (they’re managed separately)

module "catalystcenter" {
  source  = "netascode/nac-catalystcenter/catalystcenter"
  version = "0.2.0"

  yaml_directories = ["../data/"]
  templates_directories = ["../data/templates/"]

  manage_global_settings = false
  managed_sites = ["Global/Poland/Krakow"]
}

Key points for Krakow configuration:

• manage_global_settings = false: This instance doesn’t manage global resources
• managed_sites = ["Global/Poland/Krakow"]: Manages only the Krakow site
• manage_specific_sites_only defaults to false: Automatically manages child sites (Building_A, Building_B)

module "catalystcenter" {
  source  = "netascode/nac-catalystcenter/catalystcenter"
  version = "0.2.0"

  yaml_directories = ["../data/"]
  templates_directories = ["../data/templates/"]

  manage_global_settings = false
  managed_sites = ["Global/Poland/Warsaw"]
}

All Terraform instances share the same data model. Here’s an example of the site hierarchy definition:

---
catalyst_center:
  sites:
    areas:
      - name: Poland
        parent_name: Global
      - name: Krakow
        parent_name: Global/Poland
      - name: Warsaw
        parent_name: Global/Poland
      - name: Building_A
        parent_name: Global/Poland/Krakow
      - name: Building_B
        parent_name: Global/Poland/Krakow
      - name: Office_1
        parent_name: Global/Poland/Warsaw
      - name: Office_2
        parent_name: Global/Poland/Warsaw

cd COMMON
terraform init
terraform plan
terraform apply

This creates:

• Global settings and templates
• The Poland parent site
• L3 VNs and other global resources

# Deploy Krakow site
cd ../Krakow
terraform init
terraform plan
terraform apply

# Deploy Warsaw site (can be done in parallel)
cd ../Warsaw
terraform init
terraform plan
terraform apply

• Always deploy Global resources first (main.tf file in COMMON folder)
• Site-specific deployments can run in parallel after COMMON is complete

• Each folder maintains its own Terraform state
• Use remote state backends for team collaboration
• Consider state locking mechanisms

• Keep all YAML files in a shared data/ directory
• Use version control to ensure consistency across all instances
• Test changes in a development environment first

• Plan your site hierarchy before implementation
• Consider future expansion when designing the structure
• Document the hierarchy for team reference

Issue: Child sites are being managed by the wrong instance Solution: Check manage_specific_sites_only flag in parent site configurations

Issue: Global resources are being managed by multiple instances Solution: Ensure only one instance has manage_global_settings = true (typically in COMMON folder)

Issue: Sites not being created Solution: Verify the site hierarchy in your YAML files matches the managed_sites configuration

# Check what resources each instance will manage
terraform plan

# Verify site hierarchy
terraform show | grep -A 5 "catalystcenter_area"

# Check for state conflicts
terraform state list

For multi-region deployments, extend the pattern:

If you need to manage only specific child sites:

module "catalystcenter" {
  source = "..."

  manage_global_settings = false
  managed_sites = [
    "Global/Poland/Krakow/Building_A",
    "Global/Poland/Krakow/Building_B"
  ]
  manage_specific_sites_only = true
}