Skip to content
Network as Code

Bulk API Terraform Limitations

Terraform Update-in-Place Dependency Behavior with “use_bulk_api” flag in Cisco Catalyst Center

Section titled “Terraform Update-in-Place Dependency Behavior with “use_bulk_api” flag in Cisco Catalyst Center”

Applies to: Catalyst Center Provider >= 0.5.2, NAC Module >= 0.4.0 with use_bulk_api = true

This document describes Terraform’s update-in-place behavior when managing hierarchical site structures in Cisco Catalyst Center using the bulk API.

This behavior is most commonly observed when a user attempts to partially remove hierarchical site entries managed via bulk resources in a single Terraform apply

Overview

Section titled “Overview”

When use_bulk_api = true, the Cisco Catalyst Center module manages sites using bulk resources:

  • catalystcenter_areas.bulk_areas
  • catalystcenter_buildings.bulk_buildings
  • catalystcenter_floors.bulk_floors

Each resource contains a map of all items at that hierarchy level. Removing an item from the configuration triggers an update-in-place operation on the parent resource, not an explicit destroy.

The Core Issue

Section titled “The Core Issue”

Terraform treats update-in-place operations with the same dependency ordering as CREATE, not DESTROY.

Operation TypeExecution Order
CreateAreas → Buildings → Floors
Update in placeAreas → Buildings → Floors
DestroyFloors → Buildings → Areas (reversed)

When removing site hierarchy elements, Terraform sees “update the Areas list” and processes it before Buildings and Floors, even though Catalyst Center requires children to be deleted first.

Module Resource Definitions for catalystcenter_areas, catalystcenter_buildings and catalystcenter_floors

Section titled “Module Resource Definitions for catalystcenter_areas, catalystcenter_buildings and catalystcenter_floors”

The module defines explicit depends_on relationships that enforce creation order:

# Areas have no site dependencies (created first)
resource "catalystcenter_areas" "bulk_areas" {
...
...
  depends_on = [catalystcenter_discovery.discovery, ...]
}


# Buildings depend on Areas
resource "catalystcenter_buildings" "bulk_buildings" {
...
...


  depends_on = [catalystcenter_areas.bulk_areas, ...]  # <-- Buildings wait for Areas
}


# Floors depend on Buildings
resource "catalystcenter_floors" "bulk_floors" {
...
...


  depends_on = [catalystcenter_building.building, catalystcenter_buildings.bulk_buildings, ...]  # <-- Floors wait for Buildings
}

Key point: The depends_on relationships form a chain: Areas → Buildings → Floors. This ensures correct creation order, but during update-in-place operations, this same order is followed, which is against Catalyst Center’s deletion requirements.

Concrete Example: France Site Hierarchy

Section titled “Concrete Example: France Site Hierarchy”

Initial Configuration

Section titled “Initial Configuration”
sites.nac.yaml
catalyst_center:
  sites:
    areas:
      - name: Global
      - name: France
        parent_name: Global
      - name: Paris
        parent_name: Global/France
      - name: Poland
        parent_name: Global
      - name: Krakow
        parent_name: Global/Poland


    buildings:
      - name: Paris_Bld_A
        parent_name: Global/France/Paris
        latitude: 48.8575
        longitude: 2.3514
        country: France
      - name: Paris_Bld_B
        parent_name: Global/France/Paris
        latitude: 48.8575
        longitude: 2.3514
        country: France
      - name: KRK_Bld_A
        latitude: 50.0623225
        longitude: 19.937975
        country: Poland
        parent_name: Global/Poland/Krakow


    floors:
      - name: Floor_1
        floor_number: 1
        parent_name: Global/France/Paris/Paris_Bld_A
      - name: FLOOR_1
        floor_number: 1
        parent_name: Global/Poland/Krakow/KRK_Bld_A

This creates the following hierarchy in Catalyst Center:

Global
├── France                    (Area)
│   └── Paris                 (Area)
│       ├── Paris_Bld_A       (Building)
│       │   └── Floor_1       (Floor)
│       └── Paris_Bld_B       (Building)
│
└── Poland                    (Area)
    └── Krakow                (Area)
        └── KRK_Bld_A         (Building)
            └── FLOOR_1       (Floor)

Why Update-in-Place Occurs

Section titled “Why Update-in-Place Occurs”

When removing France from the configuration while Poland/Krakow remains, Terraform sees:

  • The catalystcenter_areas.bulk_areas resource still exists (Poland areas remain in the map)
  • The catalystcenter_buildings.bulk_buildings resource still exists (KRK_Bld_A remains in the map)
  • The catalystcenter_floors.bulk_floors resource still exists (FLOOR_1 remains in the map)

Because items remain in each map, Terraform classifies this as update-in-place, not destroy. The bulk resources continue to exist; only their contents change (France entries removed, Poland entries unchanged).

If you were removing all sites (emptying the maps entirely), Terraform would destroy the bulk resources, which would trigger proper DESTROY ordering (Floors → Buildings → Areas). But partial removal from a map is always an update-in-place operation.

Deletion Attempt: Single Apply (Fails)

Section titled “Deletion Attempt: Single Apply (Fails)”

Goal: Remove the entire France site hierarchy in one operation.

Action: Remove all France-related entries from sites.nac.yaml and run terraform apply.

Terraform Plan Output:

# module.catalyst_center.catalystcenter_areas.bulk_areas[0] will be updated in-place
~ resource "catalystcenter_areas" "bulk_areas" {
    ~ areas = {
        - "Global/France" = {
            - id                    = "4d359396-59b8-4931-ac80-274cf6b3db65" -> null
            - name                  = "France" -> null
            - parent_name_hierarchy = "Global" -> null
          }
        - "Global/France/Paris" = {
            - id                    = "c565b323-b3dc-4a67-ae95-fd34ce20f96d" -> null
            - name                  = "Paris" -> null
            - parent_name_hierarchy = "Global/France" -> null
          }
      }
  }


# module.catalyst_center.catalystcenter_buildings.bulk_buildings[0] will be updated in-place
~ resource "catalystcenter_buildings" "bulk_buildings" {
    ~ buildings = {
        - "Global/France/Paris/Paris_Bld_A" = {
            - id                    = "53855e96-25aa-436a-95a2-56a7ab8c71ab" -> null
            - name                  = "Paris_Bld_A" -> null
            - parent_name_hierarchy = "Global/France/Paris" -> null
          }
        - "Global/France/Paris/Paris_Bld_B" = {
            - id                    = "9b7550f4-f2ad-43bb-9834-3fad9e0e0b45" -> null
            - name                  = "Paris_Bld_B" -> null
            - parent_name_hierarchy = "Global/France/Paris" -> null
          }
      }
  }


Plan: 0 to add, 4 to change, 0 to destroy.

Result: Terraform attempts to update Areas first (following CREATE order), but Catalyst Center rejects the request:

Error: Client Error


  with module.catalyst_center.catalystcenter_areas.bulk_areas[0],
  on .terraform/modules/catalyst_center/cc_sites.tf line 12, in resource "catalystcenter_areas" "bulk_areas":
  12: resource "catalystcenter_areas" "bulk_areas" {


Failed to delete object (DELETE), got error: task failed:
"NCGR10012: Group cannot be deleted as there are child groups. Please delete them first."

Why it fails: Terraform processed the Areas update before Buildings and Floors because update-in-place follows CREATE ordering (Areas → Buildings → Floors), not DESTROY ordering (Floors → Buildings → Areas).

Solution: Multi-Step Deletion

Section titled “Solution: Multi-Step Deletion”

To align Terraform’s execution with Catalyst Center’s hierarchical constraints, deletion must be performed in multiple applies.

Step 1: Remove Floors

Section titled “Step 1: Remove Floors”

Edit sites.nac.yaml to remove only the France floor entries (Poland floors remain):

catalyst_center:
  sites:
    areas:
      - name: Global
      - name: France
        parent_name: Global
      - name: Paris
        parent_name: Global/France
      - name: Poland
        parent_name: Global
      - name: Krakow
        parent_name: Global/Poland


    buildings:
      - name: Paris_Bld_A
        parent_name: Global/France/Paris
        latitude: 48.8575
        longitude: 2.3514
        country: France
      - name: Paris_Bld_B
        parent_name: Global/France/Paris
        latitude: 48.8575
        longitude: 2.3514
        country: France
      - name: KRK_Bld_A
        latitude: 50.0623225
        longitude: 19.937975
        country: Poland
        parent_name: Global/Poland/Krakow


    floors:
      # France Floor_1 removed
      - name: FLOOR_1
        floor_number: 1
        parent_name: Global/Poland/Krakow/KRK_Bld_A

Run terraform apply. Only the Floors resource is updated (France floor removed), Areas and Buildings remain unchanged.

Step 2: Remove Buildings

Section titled “Step 2: Remove Buildings”

Edit sites.nac.yaml to remove France building entries (Poland buildings remain):

catalyst_center:
  sites:
    areas:
      - name: Global
      - name: France
        parent_name: Global
      - name: Paris
        parent_name: Global/France
      - name: Poland
        parent_name: Global
      - name: Krakow
        parent_name: Global/Poland


    buildings:
      # France buildings removed
      - name: KRK_Bld_A
        latitude: 50.0623225
        longitude: 19.937975
        country: Poland
        parent_name: Global/Poland/Krakow


    floors:
      - name: FLOOR_1
        floor_number: 1
        parent_name: Global/Poland/Krakow/KRK_Bld_A

Run terraform apply. France buildings are removed while Areas and Poland hierarchy remain.

Step 3: Remove Areas

Section titled “Step 3: Remove Areas”

Edit sites.nac.yaml to remove France area entries (Poland areas remain):

catalyst_center:
  sites:
    areas:
      - name: Global
      # France and Paris areas removed
      - name: Poland
        parent_name: Global
      - name: Krakow
        parent_name: Global/Poland


    buildings:
      - name: KRK_Bld_A
        latitude: 50.0623225
        longitude: 19.937975
        country: Poland
        parent_name: Global/Poland/Krakow


    floors:
      - name: FLOOR_1
        floor_number: 1
        parent_name: Global/Poland/Krakow/KRK_Bld_A

Run terraform apply. France areas are now safely removed. Poland hierarchy remains intact.

Why This Happens

Section titled “Why This Happens”

Terraform’s Perspective

Section titled “Terraform’s Perspective”

Terraform categorizes operations as:

  • Create: New resource
  • Update in place: Existing resource, modify attributes
  • Replace: Destroy + Create (ForceNew)
  • Destroy: Remove resource entirely

Removing items from a bulk resource’s map is an update-in-place, the resource continues to exist, just with fewer items. Terraform only reverses dependency order for explicit Destroy operations.

The Dependency Chain

Section titled “The Dependency Chain”
depends_on chain:
  catalystcenter_areas.bulk_areas
    └── catalystcenter_buildings.bulk_buildings
          └── catalystcenter_floors.bulk_floors

For CREATE: Areas → Buildings → Floors (correct) For UPDATE: Areas → Buildings → Floors (problematic for deletions) For DESTROY: Floors → Buildings → Areas (would be correct, but not triggered)

Device Deletion: Fabric Role Must Be Removed First

Section titled “Device Deletion: Fabric Role Must Be Removed First”

The same update-in-place dependency issue affects device deletion when using use_bulk_api = true. When a device with fabric roles needs to be deleted (state changed to INIT), the fabric role must be removed in a separate apply before the device state is changed.

Module Resource Definitions for provision_devices and fabric_devices

Section titled “Module Resource Definitions for provision_devices and fabric_devices”

The module defines explicit depends_on relationships that enforce creation order:

# provision_devices (created first)
resource "catalystcenter_provision_devices" "provision_devices"  {
...
...
  depends_on = [catalystcenter_assign_device_to_site.devices_to_site, ...]
}


# Fabric devices depend on Provision devices
resource "catalystcenter_fabric_devices" "fabric_devices"  {
...
...


  depends_on = [catalystcenter_device_role.role, catalystcenter_provision_devices.provision_devices, catalystcenter_provision_device.provision_device, ...]
  # <-- fabric_devices wait for provision_devices
}

Why This Happens

Section titled “Why This Happens”

When use_bulk_api = true, the following resources are managed as bulk collections:

  • catalystcenter_provision_devices — manages device provisioning state per site
  • catalystcenter_fabric_devices — manages fabric roles per fabric site
  • catalystcenter_fabric_port_assignments — manages port assignments per device
  • catalystcenter_device_role — manages device roles

Removing a device from the YAML configuration triggers update-in-place operations on these bulk resources rather than explicit destroys. Terraform processes these updates in CREATE order, meaning the device provisioning state removal (catalystcenter_provision_devices) will execute before the fabric role removal (catalystcenter_fabric_devices) completes.

Catalyst Center requires the fabric role to be removed from a device before the device can be deleted. If Terraform attempts to delete the device while the fabric role is still assigned, the operation fails.

Concrete Example: Removing an Edge Node Device

Section titled “Concrete Example: Removing an Edge Node Device”

Initial Configuration

Section titled “Initial Configuration”
devices.nac.yaml
catalyst_center:
  network_devices:
    - name: EDGE01.cisco.eu
      fqdn_name: EDGE01.cisco.eu
      device_ip: 198.18.130.1
      pid: C9KV-UADP-8P
      serial_number: CML12322UAD
      state: PROVISION
      device_role: ACCESS
      site: Global/Poland/Krakow/Bld A
      fabric_site: Global/Poland/Krakow
      fabric_roles:
        - EDGE_NODE


    - name: EDGE02.cisco.eu
      fqdn_name: EDGE02.cisco.eu
      device_ip: 198.18.130.2
      pid: C9KV-UADP-8P
      serial_number: CML12322UAD
      state: PROVISION
      device_role: ACCESS
      site: Global/Poland/Krakow/Bld A
      fabric_site: Global/Poland/Krakow
      fabric_roles:
        - EDGE_NODE

Deletion Attempt: Single Apply (Fails)

Section titled “Deletion Attempt: Single Apply (Fails)”

Goal: Remove EDGE01.cisco.eu by removing it from the configuration (or changing its state to INIT).

Action: Remove the EDGE01 device entry from the YAML and run terraform apply.

Terraform Plan Output:

Terraform used the selected providers to generate the following execution plan.
Resource actions are indicated with the following symbols:
  ~ update in-place
  - destroy


Terraform will perform the following actions:


  # module.catalyst_center.catalystcenter_device_role.role["EDGE01.cisco.eu"] will be destroyed
  # (because key ["EDGE01.cisco.eu"] is not in for_each map)
  - resource "catalystcenter_device_role" "role" {
      - device_id   = "7ef492ca-b008-479a-9de4-7e40438c7d10" -> null
      - id          = "7ef492ca-b008-479a-9de4-7e40438c7d10" -> null
      - role        = "ACCESS" -> null
      - role_source = "MANUAL" -> null
    }


  # module.catalyst_center.catalystcenter_fabric_devices.fabric_devices["Global/Poland/Krakow"]
  # will be updated in-place
  ~ resource "catalystcenter_fabric_devices" "fabric_devices" {
      ~ fabric_devices = [
          - {
              - device_roles     = [
                  - "EDGE_NODE",
                ] -> null
              - fabric_id        = "f86c78b2-4b63-46c3-a62f-059cd56919bf" -> null
              - id               = "83826e71-48a8-41f1-a55b-dfeafbe89d1f" -> null
              - network_device_id = "7ef492ca-b008-479a-9de4-7e40438c7d10" -> null
            },
          # (3 unchanged elements hidden)
        ]
        id = "f86c78b2-4b63-46c3-a62f-059cd56919bf"
        # (1 unchanged attribute hidden)
    }


  # module.catalyst_center.catalystcenter_fabric_port_assignments.port_assignments["EDGE01.cisco.eu"]
  # will be destroyed
  # (because key ["EDGE01.cisco.eu"] is not in for_each map)
  - resource "catalystcenter_fabric_port_assignments" "port_assignments" {
      - fabric_id         = "f86c78b2-4b63-46c3-a62f-059cd56919bf" -> null
      - id                = "7ef492ca-b008-479a-9de4-7e40438c7d10" -> null
      - network_device_id = "7ef492ca-b008-479a-9de4-7e40438c7d10" -> null
      - port_assignments  = [
          - {
              - authenticate_template_name = "No Authentication" -> null
              - connected_device_type      = "USER_DEVICE" -> null
              - data_vlan_name             = "CampusVN_VLAN" -> null
              - fabric_id                  = "f86c78b2-4b63-46c3-a62f-059cd56919bf" -> null
              - id                         = "0547a38c-f760-4b94-bbaf-a67ed63b9513" -> null
              - interface_name             = "GigabitEthernet1/0/2" -> null
              - network_device_id          = "7ef492ca-b008-479a-9de4-7e40438c7d10" -> null
            },
        ] -> null
    }


  # module.catalyst_center.catalystcenter_provision_devices.provision_devices["Global/Poland/Krakow/Bld A"]
  # will be updated in-place
  ~ resource "catalystcenter_provision_devices" "provision_devices" {
        id                = "9065de7e-57ee-4f8f-83b3-327fc8c0cacb"
      ~ provision_devices = [
          - {
              - id                = "83826e71-48a8-41f1-a55b-dfeafbe89d1f" -> null
              - network_device_id = "7ef492ca-b008-479a-9de4-7e40438c7d10" -> null
              - reprovision       = false -> null
              - site_id           = "9065de7e-57ee-4f8f-83b3-327fc8c0cacb" -> null
            },
          # (2 unchanged elements hidden)
        ]
        # (1 unchanged attribute hidden)
    }


Plan: 0 to add, 2 to change, 2 to destroy.

Notice the critical detail: both catalystcenter_fabric_devices (fabric role removal) and catalystcenter_provision_devices (device provisioning state removal) are both update-in-place operations. Because Terraform applies update-in-place changes in CREATE order, catalystcenter_provision_devices is processed first.

Terraform Apply Output:

module.catalyst_center.catalystcenter_fabric_port_assignments.port_assignments["EDGE01.cisco.eu"]: Destroying...
module.catalyst_center.catalystcenter_fabric_port_assignments.port_assignments["EDGE01.cisco.eu"]: Still destroying... [10s elapsed]
module.catalyst_center.catalystcenter_fabric_port_assignments.port_assignments["EDGE01.cisco.eu"]: Destruction complete after 13s
module.catalyst_center.catalystcenter_device_role.role["EDGE01.cisco.eu"]: Destroying...
module.catalyst_center.catalystcenter_device_role.role["EDGE01.cisco.eu"]: Destruction complete after 0s
module.catalyst_center.catalystcenter_provision_devices.provision_devices["Global/Poland/Krakow/Bld A"]: Modifying...
╷
│ Error: Client Error
│
│   with module.catalyst_center.catalystcenter_provision_devices.provision_devices["Global/Poland/Krakow/Bld A"],
│   on .terraform/modules/catalyst_center/cc_device_provision.tf line 243,
│   in resource "catalystcenter_provision_devices" "provision_devices":
│   243: resource "catalystcenter_provision_devices" "provision_devices" {
│
│ Failed to delete object (DELETE), got error: task '019cb2c6-c465-7375-b7f4-ddf102fed734' failed:
│ Network device deletion failed, NCIM10000: Error while deleting device 198.18.130.1.
╵

Why it fails: The apply log shows that Terraform began modifying catalystcenter_provision_devices (which triggers the device deletion in Catalyst Center) before catalystcenter_fabric_devices had a chance to remove the fabric role. Catalyst Center cannot delete a device that still has an active fabric role assigned to it.

The execution order was:

  1. catalystcenter_fabric_port_assignments — destroyed (explicit destroy, correct)
  2. catalystcenter_device_role — destroyed (explicit destroy, correct)
  3. catalystcenter_provision_devices — updated in-place (attempted device deletion) — failed
  4. catalystcenter_fabric_devices — updated in-place (fabric role removal) — never executed

Because both catalystcenter_provision_devices and catalystcenter_fabric_devices are update-in-place operations, Terraform does not enforce that the fabric role removal happens before the device state change.

Solution: Two-Step Device Deletion

Section titled “Solution: Two-Step Device Deletion”

To properly delete a device when using use_bulk_api = true, perform the deletion in two separate terraform apply runs.

Step 1: Remove the Fabric Role

Section titled “Step 1: Remove the Fabric Role”

Edit the device configuration to remove only the fabric_roles and fabric_site entries. Keep the device in PROVISION state:

devices.nac.yaml
catalyst_center:
  network_devices:
    - name: EDGE01.cisco.eu
      fqdn_name: EDGE01.cisco.eu
      device_ip: 198.18.130.1
      pid: C9KV-UADP-8P
      serial_number: CML12322UAD
      state: PROVISION
      device_role: ACCESS
      site: Global/Poland/Krakow/Bld A
      # fabric_site and fabric_roles removed


    - name: EDGE02.cisco.eu
      fqdn_name: EDGE02.cisco.eu
      device_ip: 198.18.130.2
      pid: C9KV-UADP-8P
      serial_number: CML12322UAD
      state: PROVISION
      device_role: ACCESS
      site: Global/Poland/Krakow/Bld A
      fabric_site: Global/Poland/Krakow
      fabric_roles:
        - EDGE_NODE

Run terraform apply. This removes the fabric role and port assignments for EDGE01 while keeping the device in provisioned state. Catalyst Center will remove the device from the fabric.

Step 2: Change Device State to INIT

Section titled “Step 2: Change Device State to INIT”

Now that the fabric role has been removed, change the device state to INIT to trigger deletion:

devices.nac.yaml
catalyst_center:
  network_devices:
    - name: EDGE01.cisco.eu
      fqdn_name: EDGE01.cisco.eu
      device_ip: 198.18.130.1
      pid: C9KV-UADP-8P
      serial_number: CML12322UAD
      state: INIT  # Changed from PROVISION to INIT
      device_role: ACCESS
      site: Global/Poland/Krakow/Bld A


    - name: EDGE02.cisco.eu
      fqdn_name: EDGE02.cisco.eu
      device_ip: 198.18.130.2
      pid: C9KV-UADP-8P
      serial_number: CML12322UAD
      state: PROVISION
      device_role: ACCESS
      site: Global/Poland/Krakow/Bld A
      fabric_site: Global/Poland/Krakow
      fabric_roles:
        - EDGE_NODE

Run terraform apply. The device is now safely deleted from Catalyst Center since no fabric role is attached.

Key Takeaways

Section titled “Key Takeaways”
  1. Update-in-place follows CREATE ordering, not DESTROY ordering
  2. Bulk resources (maps of items) treat partial item removal as an update-in-place
  3. Catalyst Center enforces hierarchy: children must be deleted before parents
  4. Multi-step deletion is required when partially removing entire site hierarchies
  5. Fabric roles must be removed before device deletion: when using use_bulk_api = true, remove the fabric role in a separate apply before changing the device state to INIT
  6. This behavior applies to Provider >= 0.5.2, Module >= 0.4.0 with use_bulk_api = true

One-Sentence Rule

Section titled “One-Sentence Rule”

Update-in-place follows CREATE ordering. Only explicit DESTROY reverses dependencies. Delete children before parents, and remove fabric roles before deleting devices using multiple applies.

This behavior may also occur for other hierarchical bulk-managed resources when the bulk_site_provisioning flag is enabled, as deletions in bulk collections are always handled as update-in-place operations.