String[Regex: ^[a-f0-9]{1}\.[a-f0-9]{1}\.[a-f0-9]{1}$] or String[Regex: ^[a-f0-9]{4}\.[a-f0-9]{4}\.[a-f0-9]{4}$] or String[Regex: ^[a-f0-9]{2}:[a-f0-9]{2}:[a-f0-9]{2}:[a-f0-9]{2}:[a-f0-9]{2}:[a-f0-9]{2}$] or String[Regex: ^[a-f0-9]{2}-[a-f0-9]{2}-[a-f0-9]{2}-[a-f0-9]{2}-[a-f0-9]{2}-[a-f0-9]{2}$]
In this example, we have a fabric named nac-msd. In this MSD fabric, all fabrics use the same anycast gateway mac: DE:AD:BE:EF:FE:ED.
The loopback ID used for the Anycast IP configured on each Border Gateway is: 100 and the subnet used to allocate an IP is: 10.10.0.0/24.
This configuration uses external ISN devices as the Route Server. In this configuration there are four (4) IP addresses:
100.64.0.1
100.64.0.2
100.64.0.3
100.64.0.4
Each IP will be associated to an Autonomous System (ASN), which can be different for each Route Server. In this example, all Route Servers will use the ASN 65000.100.
When underlay_autoconfig is true, you need to provide a subnet for the DCI point-to-point (p2p) connection(s). Here we used the subnet 10.10.1.0/24. Each p2p will used CIDR /30.
The value in bgp_asn should between double quotes.
Example:
global:
bgp_asn: "6512"
global:
bgp_asn: "65000.1"
global.nac.yaml
---
vxlan:
fabric:
name: nac-msd
type: MSD
multisite:
child_fabrics:
- name: nac-ndfc1
- name: nac-ndfc2
- name: nac-ndfc3
- name: nac-isn
anycast_gateway_mac: de:ad:be:ef:fe:ed
bgw_ip_tag: 54321
vtep_loopback_id: 100
ipv4_vtep_loopback_range: 10.10.0.0/24
overlay_dci:
underlay_autoconfig: true
deployment_method: 'Centralized_To_Route_Server'
ipv4_dci_subnet_range: 10.10.1.0/24
ipv4_dci_subnet_mask: 30
route_server:
peers:
- ip_address: 100.64.0.1
bgp_asn: "65000.100"# Use quotes
- ip_address: 100.64.0.2
bgp_asn: "65000.100"# Use quotes
- ip_address: 100.64.0.3
bgp_asn: "65000.100"# Use quotes
- ip_address: 100.64.0.4
bgp_asn: "65000.100"# Use quotes
redistribute_direct: false
ip_tag: 54321
enable_bgp_bfd: false
enable_bgp_log_neighbor_change: true
enable_bgp_send_community: true
enable_ebgp_password: false
enable_trm: false
delay_restore: 300
Starting with release 0.4.3, we are introducing a new key, vxlan.multisite.child_fabrics.bgw_anycast_vip_ipv4, which allows you to configure a custom Virtual IP (VIP) address or anycast IP for the Border Gateway. This key is utilized only when vxlan.underlay.general.manual_underlay_allocation is set to true for at least one child fabric.
auth_proto selects the SNMPv3 authentication/privacy protocol that ND uses when
discovering ISN switches for the first time. The default is MD5. The value applies to
every switch in the ISN fabric; it cannot be set per switch.
Value
Authentication
Privacy
MD5
MD5
—
SHA
SHA-1
—
MD5_DES
MD5
DES
MD5_AES
MD5
AES-128
SHA_DES
SHA-1
DES
SHA_AES
SHA-1
AES-128
Support for auth_proto in ISN fabrics is confirmed by code inspection.
It has not been validated live in this release cycle.
NX-OS prerequisite: The SNMPv3 user must already exist on each switch with
authentication and privacy algorithms that match the selected auth_proto. Create the local
user once, then apply the single snmp-server user line for your chosen value:
username <USERNAME> password <PASSWORD> role network-admin
! Apply ONE of the following, matching the auth_proto value
! MD5 (default) — authentication only
snmp-server user <USERNAME> network-admin auth md5 <PASSWORD>
! SHA — authentication only
snmp-server user <USERNAME> network-admin auth sha <PASSWORD>
! MD5_DES
snmp-server user <USERNAME> network-admin auth md5 <PASSWORD> priv des <PASSWORD>
! MD5_AES
snmp-server user <USERNAME> network-admin auth md5 <PASSWORD> priv aes-128 <PASSWORD>
! SHA_DES
snmp-server user <USERNAME> network-admin auth sha <PASSWORD> priv des <PASSWORD>
! SHA_AES
snmp-server user <USERNAME> network-admin auth sha <PASSWORD> priv aes-128 <PASSWORD>
Verify the result with show snmp user <USERNAME>. The Auth and Priv columns must match
the table above.
The authentication and privacy passphrases must be identical. NaC sends a single
password to ND, which uses it for both. Mismatched passphrases are the most common
cause of notManageable and SNMPv3 timeouts during discovery.
Environment variables: ND uses NDFC_SW_USERNAME and NDFC_SW_PASSWORD to reach the
switch during discovery. These values must match the <USERNAME> and <PASSWORD> used when
configuring the SNMPv3 user on the switch.
After a multisite VRF is created, name (the data model key used to identify the VRF) and vrf_id (L3 VNI / VNI) cannot be updated. If either value needs to change, delete and recreate the VRF.
This does not prevent updates to supported switch-facing values such as VLAN ID, vrf_vlan_name, vrf_description, VRF interface description (vrf_intf_desc), route maps, or child fabric settings.
After a multisite network is created, name (the data model key used to identify the network) and net_id (VXLAN L2VNID / VNI) cannot be updated. If either value needs to change, delete and recreate the network.
This does not prevent updates to supported switch-facing values such as VLAN ID, vlan_name, VLAN/SVI interface description (int_desc), gateway settings, attachments, or child fabric settings.
Example 1 - Layer2 network:
networks.nac.yaml
vxlan:
multisite:
overlay:
networks:
# simple example
- name: NetworkDMZ
is_l2_only: true
net_id: 33100
vlan_id: 3100
attach_group: dmz
network_attach_groups:
- name: dmz
switches:
- hostname: dc1-leaf1
ports:
- Ethernet1/23
- Ethernet1/24
- hostname: dc1-leaf2
ports:
- Ethernet1/23
- Ethernet1/24
Example 2 - Layer3 network:
This section defines a Layer 3 network with IPv4 and IPv6 gateway addresses that spans multiple child fabrics in a Multi-Site deployment. The Network_53001 configuration establishes a distributed SVI (Switch Virtual Interface) across both nac-ndfc1 and nac-ndfc2 fabrics, associated with NaC-VRF1. Each child fabric is configured with the same multicast group address for BUM (Broadcast, Unknown unicast, Multicast) traffic handling, and the l3gw_on_border: true parameter enables Layer 3 gateway functionality on the border leaf switches. This creates a stretched Layer 3 network where the gateway IP addresses (10.0.1.1/24 for IPv4 and 2001:0db8:0001:0000:0000:0000:0000:0001/64 for IPv6) are active on border devices in both data centers, providing localized gateway services and optimal traffic flow for hosts connected to this network segment.
Managing Multi-Cluster Fabrics Groups (MCFG) follow the same steps of managing an MSD Fabric. Additional keys and value must be defined in the data model and on the inventory file. In this example we have a fabric named nac-mcfg. This fabric is similar as the MSD example, the main difference is that nac-ndfc1 and nac-ndfc2 child fabrics are managed by nd-cluster-1 while nac-ndfc3 and nac-isn are managed by nd-cluster-2.
global.nac.yaml
---
vxlan:
fabric:
name: nac-mcfg
type: MCFG
multisite:
child_fabrics:
- name: nac-ndfc1
cluster: nd-cluster-1
- name: nac-ndfc2
cluster: nd-cluster-1
- name: nac-ndfc3
cluster: nd-cluster-2
- name: nac-isn
cluster: nd-cluster-2
anycast_gateway_mac: de:ad:be:ef:fe:ed
MCFG uses a feature called One Manage. This features requires remote authentication, which means that the inventory file must be updated to reflect the multiple clusters.
String[Regex: ^[a-f0-9]{1}\.[a-f0-9]{1}\.[a-f0-9]{1}$] or String[Regex: ^[a-f0-9]{4}\.[a-f0-9]{4}\.[a-f0-9]{4}$] or String[Regex: ^[a-f0-9]{2}:[a-f0-9]{2}:[a-f0-9]{2}:[a-f0-9]{2}:[a-f0-9]{2}:[a-f0-9]{2}$] or String[Regex: ^[a-f0-9]{2}-[a-f0-9]{2}-[a-f0-9]{2}-[a-f0-9]{2}-[a-f0-9]{2}-[a-f0-9]{2}$]
In this example, we have a fabric named nac-msd. In this MSD fabric, all fabrics use the same anycast gateway mac: DE:AD:BE:EF:FE:ED.
The loopback ID used for the Anycast IP configured on each Border Gateway is: 100 and the subnet used to allocate an IP is: 10.10.0.0/24.
This configuration uses external ISN devices as the Route Server. In this configuration there are four (4) IP addresses:
100.64.0.1
100.64.0.2
100.64.0.3
100.64.0.4
Each IP will be associated to an Autonomous System (ASN), which can be different for each Route Server. In this example, all Route Servers will use the ASN 65000.100.
When underlay_autoconfig is true, you need to provide a subnet for the DCI point-to-point (p2p) connection(s). Here we used the subnet 10.10.1.0/24. Each p2p will used CIDR /30.
The value in bgp_asn should between double quotes.
Example:
global:
bgp_asn: "6512"
global:
bgp_asn: "65000.1"
global.nac.yaml
---
vxlan:
fabric:
name: nac-msd
type: MSD
multisite:
child_fabrics:
- name: nac-ndfc1
- name: nac-ndfc2
- name: nac-ndfc3
- name: nac-isn
anycast_gateway_mac: de:ad:be:ef:fe:ed
bgw_ip_tag: 54321
vtep_loopback_id: 100
ipv4_vtep_loopback_range: 10.10.0.0/24
overlay_dci:
underlay_autoconfig: true
deployment_method: 'Centralized_To_Route_Server'
ipv4_dci_subnet_range: 10.10.1.0/24
ipv4_dci_subnet_mask: 30
route_server:
peers:
- ip_address: 100.64.0.1
bgp_asn: "65000.100"# Use quotes
- ip_address: 100.64.0.2
bgp_asn: "65000.100"# Use quotes
- ip_address: 100.64.0.3
bgp_asn: "65000.100"# Use quotes
- ip_address: 100.64.0.4
bgp_asn: "65000.100"# Use quotes
redistribute_direct: false
ip_tag: 54321
enable_bgp_bfd: false
enable_bgp_log_neighbor_change: true
enable_bgp_send_community: true
enable_ebgp_password: false
enable_trm: false
delay_restore: 300
Starting with release 0.4.3, we are introducing a new key, vxlan.multisite.child_fabrics.bgw_anycast_vip_ipv4, which allows you to configure a custom Virtual IP (VIP) address or anycast IP for the Border Gateway. This key is utilized only when vxlan.underlay.general.manual_underlay_allocation is set to true for at least one child fabric.
After a multisite VRF is created, name (the data model key used to identify the VRF) and vrf_id (L3 VNI / VNI) cannot be updated. If either value needs to change, delete and recreate the VRF.
This does not prevent updates to supported switch-facing values such as VLAN ID, vrf_vlan_name, vrf_description, VRF interface description (vrf_intf_desc), route maps, or child fabric settings.
After a multisite network is created, name (the data model key used to identify the network) and net_id (VXLAN L2VNID / VNI) cannot be updated. If either value needs to change, delete and recreate the network.
This does not prevent updates to supported switch-facing values such as VLAN ID, vlan_name, VLAN/SVI interface description (int_desc), gateway settings, attachments, or child fabric settings.
Example 1 - Layer2 network:
networks.nac.yaml
vxlan:
multisite:
overlay:
networks:
# simple example
- name: NetworkDMZ
is_l2_only: true
net_id: 33100
vlan_id: 3100
attach_group: dmz
network_attach_groups:
- name: dmz
switches:
- hostname: dc1-leaf1
ports:
- Ethernet1/23
- Ethernet1/24
- hostname: dc1-leaf2
ports:
- Ethernet1/23
- Ethernet1/24
Example 2 - Layer3 network:
This section defines a Layer 3 network with IPv4 and IPv6 gateway addresses that spans multiple child fabrics in a Multi-Site deployment. The Network_53001 configuration establishes a distributed SVI (Switch Virtual Interface) across both nac-ndfc1 and nac-ndfc2 fabrics, associated with NaC-VRF1. Each child fabric is configured with the same multicast group address for BUM (Broadcast, Unknown unicast, Multicast) traffic handling, and the l3gw_on_border: true parameter enables Layer 3 gateway functionality on the border leaf switches. This creates a stretched Layer 3 network where the gateway IP addresses (10.0.1.1/24 for IPv4 and 2001:0db8:0001:0000:0000:0000:0000:0001/64 for IPv6) are active on border devices in both data centers, providing localized gateway services and optimal traffic flow for hosts connected to this network segment.
Managing Multi-Cluster Fabrics Groups (MCFG) follow the same steps of managing an MSD Fabric. Additional keys and value must be defined in the data model and on the inventory file. In this example we have a fabric named nac-mcfg. This fabric is similar as the MSD example, the main difference is that nac-ndfc1 and nac-ndfc2 child fabrics are managed by nd-cluster-1 while nac-ndfc3 and nac-isn are managed by nd-cluster-2.
global.nac.yaml
---
vxlan:
fabric:
name: nac-mcfg
type: MCFG
multisite:
child_fabrics:
- name: nac-ndfc1
cluster: nd-cluster-1
- name: nac-ndfc2
cluster: nd-cluster-1
- name: nac-ndfc3
cluster: nd-cluster-2
- name: nac-isn
cluster: nd-cluster-2
anycast_gateway_mac: de:ad:be:ef:fe:ed
MCFG uses a feature called One Manage. This features requires remote authentication, which means that the inventory file must be updated to reflect the multiple clusters.
String[Regex: ^[a-f0-9]{1}\.[a-f0-9]{1}\.[a-f0-9]{1}$] or String[Regex: ^[a-f0-9]{4}\.[a-f0-9]{4}\.[a-f0-9]{4}$] or String[Regex: ^[a-f0-9]{2}:[a-f0-9]{2}:[a-f0-9]{2}:[a-f0-9]{2}:[a-f0-9]{2}:[a-f0-9]{2}$] or String[Regex: ^[a-f0-9]{2}-[a-f0-9]{2}-[a-f0-9]{2}-[a-f0-9]{2}-[a-f0-9]{2}-[a-f0-9]{2}$]
In this example, we have a fabric named nac-msd. In this MSD fabric, all fabrics use the same anycast gateway mac: DE:AD:BE:EF:FE:ED.
The loopback ID used for the Anycast IP configured on each Border Gateway is: 100 and the subnet used to allocate an IP is: 10.10.0.0/24.
This configuration uses external ISN devices as the Route Server. In this configuration there are four (4) IP addresses:
100.64.0.1
100.64.0.2
100.64.0.3
100.64.0.4
Each IP will be associated to an Autonomous System (ASN), which can be different for each Route Server. In this example, all Route Servers will use the ASN 65000.100.
When underlay_autoconfig is true, you need to provide a subnet for the DCI point-to-point (p2p) connection(s). Here we used the subnet 10.10.1.0/24. Each p2p will used CIDR /30.
The value in bgp_asn should between double quotes.
Example:
global:
bgp_asn: "6512"
global:
bgp_asn: "65000.1"
global.nac.yaml
---
vxlan:
fabric:
name: nac-msd
type: MSD
multisite:
child_fabrics:
- name: nac-ndfc1
- name: nac-ndfc2
- name: nac-ndfc3
- name: nac-isn
anycast_gateway_mac: de:ad:be:ef:fe:ed
bgw_ip_tag: 54321
vtep_loopback_id: 100
ipv4_vtep_loopback_range: 10.10.0.0/24
overlay_dci:
underlay_autoconfig: true
deployment_method: 'Centralized_To_Route_Server'
ipv4_dci_subnet_range: 10.10.1.0/24
ipv4_dci_subnet_mask: 30
route_server:
peers:
- ip_address: 100.64.0.1
bgp_asn: "65000.100"# Use quotes
- ip_address: 100.64.0.2
bgp_asn: "65000.100"# Use quotes
- ip_address: 100.64.0.3
bgp_asn: "65000.100"# Use quotes
- ip_address: 100.64.0.4
bgp_asn: "65000.100"# Use quotes
redistribute_direct: false
ip_tag: 54321
enable_bgp_bfd: false
enable_bgp_log_neighbor_change: true
enable_bgp_send_community: true
enable_ebgp_password: false
enable_trm: false
delay_restore: 300
Starting with release 0.4.3, we are introducing a new key, vxlan.multisite.child_fabrics.bgw_anycast_vip_ipv4, which allows you to configure a custom Virtual IP (VIP) address or anycast IP for the Border Gateway. This key is utilized only when vxlan.underlay.general.manual_underlay_allocation is set to true for at least one child fabric.
This section defines a Layer 3 network with IPv4 and IPv6 gateway addresses that spans multiple child fabrics in a Multi-Site deployment. The Network_53001 configuration establishes a distributed SVI (Switch Virtual Interface) across both nac-ndfc1 and nac-ndfc2 fabrics, associated with NaC-VRF1. Each child fabric is configured with the same multicast group address for BUM (Broadcast, Unknown unicast, Multicast) traffic handling, and the l3gw_on_border: true parameter enables Layer 3 gateway functionality on the border leaf switches. This creates a stretched Layer 3 network where the gateway IP addresses (10.0.1.1/24 for IPv4 and 2001:0db8:0001:0000:0000:0000:0000:0001/64 for IPv6) are active on border devices in both data centers, providing localized gateway services and optimal traffic flow for hosts connected to this network segment.
Managing Multi-Cluster Fabrics Groups (MCFG) follow the same steps of managing an MSD Fabric. Additional keys and value must be defined in the data model and on the inventory file. In this example we have a fabric named nac-mcfg. This fabric is similar as the MSD example, the main difference is that nac-ndfc1 and nac-ndfc2 child fabrics are managed by nd-cluster-1 while nac-ndfc3 and nac-isn are managed by nd-cluster-2.
global.nac.yaml
---
vxlan:
fabric:
name: nac-mcfg
type: MCFG
multisite:
child_fabrics:
- name: nac-ndfc1
cluster: nd-cluster-1
- name: nac-ndfc2
cluster: nd-cluster-1
- name: nac-ndfc3
cluster: nd-cluster-2
- name: nac-isn
cluster: nd-cluster-2
anycast_gateway_mac: de:ad:be:ef:fe:ed
MCFG uses a feature called One Manage. This features requires remote authentication, which means that the inventory file must be updated to reflect the multiple clusters.
