Skip to content
Network as Code

Testing

The ACI fabric uses a policy model to combine data into a health score. Health scores can be aggregated for a variety of areas such as for the system, infrastructure, tenants, applications, or services. Every configured object is also retrievable via the APIC REST API, which can be used to compare this against the intended configuration. This makes verifying that changes were succesful very simple. This is a much more efficient way than parsing the output of SSH commands. You might want to make this an automated step to validate your changes have been succesful. To do that you can make use of nac-test tool. nac-test is a CLI tool that renders and executes Robot Framework tests using Jinja templating. Robot Framework is a generic test automation framework for acceptance testing. Combining Robot’s language agnostic syntax with the flexibility of Jinja templating allows dynamically rendering a set of test suites from the desired infrastructure state expressed in YAML syntax.

All data from the YAML files (--data option) will first be combined into a single data structure which is then provided as input to the templating process. Each template in the --templates path will then be rendered and written to the --output path. If the --templates path has subfolders, the folder structure will be retained when rendering the templates.

Python 3.10+ is required to install nac-test. nac-test can be installed with pip:

Terminal window
pip install nac-test

Example output of running nac-test from the cli:

Terminal window
> nac-test --data ./data --output ./tests/results/aci --templates ./tests/   23s
Robot Framework remote server at 127.0.0.1:8270 started.
Storing .pabotsuitenames file
2023-09-08 12:26:36.921185 [PID:73054] [0] [ID:2] EXECUTING Aci.Templates.Ntp
2023-09-08 12:26:36.922841 [PID:73055] [1] [ID:1] EXECUTING Aci.Templates.Nodes
2023-09-08 12:26:36.923017 [PID:73057] [2] [ID:0] EXECUTING Aci.Templates.Bgp Rr
2023-09-08 12:26:39.902193 [PID:73054] [0] [ID:2] PASSED Aci.Templates.Ntp in 2.9 seconds
2023-09-08 12:26:39.916829 [PID:73057] [2] [ID:0] PASSED Aci.Templates.Bgp Rr in 2.9 seconds
2023-09-08 12:26:40.310553 [PID:73055] [1] [ID:1] PASSED Aci.Templates.Nodes in 3.3 seconds
22 tests, 22 passed, 0 failed, 0 skipped.
===================================================
Output:  /path/tests/results/aci/output.xml
XUnit:   /path/tests/results/aci/xunit.xml
Log:     /path/tests/results/aci/log.html
Report:  /path/tests/results/aci/report.html
Stopping PabotLib process
Robot Framework remote server at 127.0.0.1:8270 stopped.
PabotLib process stopped
Total testing: 9.10 seconds
Elapsed time:  3.99 seconds

The example template folder used for testing can be found here. This repository contains multiple .robot files that serve as an example to start writing your own tests. Examples are being provided for retrieving health-scores, configuration and functional tests (for example to verify that the NTP configuration resulted into nodes being synced with the specified NTP server.)

Consider the following fabric registration test:

*** Settings ***
Documentation   Verify Fabric Nodes
Suite Setup     Login APIC
Default Tags    apic   day1   config   node_policies
Resource        ./apic_common.resource


*** Test Cases ***
# Verify node fabric registration
{% for node in apic.node_policies.nodes | default([]) %}


{% if node.role != 'apic' %}
Verify fabric registration for Node-{{ node.id }}
    ${r}=   GET On Session   apic   /api/mo/uni/controller/nodeidentpol/nodep-{{ node.serial_number }}.json
    Should Be Equal Value Json String   ${r.json()}   $..fabricNodeIdentP.attributes.nodeId   {{ node.id }}
    Should Be Equal Value Json String   ${r.json()}   $..fabricNodeIdentP.attributes.podId   {{ node.pod }}


{% endif %}
{% endfor %}

With the following example node_policies.nac.yaml:

node_policies.nac.yaml
---
apic:
  node_policies:
    inb_endpoint_group: inb
    oob_endpoint_group: default


    nodes:
      - id: 101
        pod: 1
        role: leaf
        serial_number: FDOAAAA9JB
        name: leaf-101
        oob_address: 10.61.124.141/24
        oob_gateway: 10.61.124.1
        update_group: MG1
        fabric_policy_group: all-leafs
        access_policy_group: all-leafs


      - id: 102
        pod: 1
        role: leaf
        serial_number: FDAAAAA9V8
        name: leaf-102
        oob_address: 10.61.124.152/24
        oob_gateway: 10.61.124.1
        update_group: MG2
        fabric_policy_group: all-leafs
        access_policy_group: all-leafs

Running nac-test will start by merging all inventory files together, so that all nodes within the input folder become accessible in apic.node_policies.nodes. The robot test will then loop the tests for each node with the for loop contained in {% for node in apic.node_policies.nodes | default([]) %}. Because this test is only applicable to nodes that are not APIC, these are filtered out using {% if node.role != 'apic' %}.

Robot makes use of a construct called keywords. Think of a keyword as a single step. Just as a test is conceptually made up of many steps, a robot test is made up of many keywords. Several keywords may be leveraged by different tests and have been made available in apic.common.resource so that the logic to handle authentication against APIC is defined only once. This logic can then be leveraged by each test. The Suite Setup calls the Login APIC keyword which creates an authenticated session against APIC.

Verify fabric registration for Node-{{ node.id }} will run for each instance of node.id found in the inventory. Leveraging the authenticated session against APIC, the GET On Session keyword can be used to GET a particular object and its attributes. For this particular node test, objects are created for each node with their unique serial number. This information is retrieved from node.serial_number. The reply from APIC is then stored in variable ${r} which can be used to evaluate the configuration. The Should Be Equal Value Json String custom keyword is used to compare attributes found in the payload of the reply against the provided inventory values. In this case, the nodeId and podId are being compared against the values of node.id and node.pod. The tests will only succeed if these values match.

Running the test will result in a report.html in the specified output folder. The results of each test can be observed in detail:

The same logic can be used to validate other types of configuration. Robot has an extensive library of pre-defined keywords, such as the ability to SSH into a device, ping and much more.

If you are not quite sure which objects to leverage for testing, you can save configuration from the APIC UI to understand which objects are of interest. Navigating to https://your-apic-url/model-doc will provide documentation for each object and their attributes. Alternatively this is also available publicly, at https://developer.cisco.com/site/apic-mim-ref-api/.

This method of inventory driven testing can also be used for brownfield environments. This can be useful for any existing configuration that is not deployed using Terraform with the Network-as-Code module for ACI. As long as the inventory follows the structure provided by the data-model, it can be used as input for testing.

Gitlab provides a useful, native integration with these tests. The XUnit.xml output can be used to view the results directly in Gitlab. See the following url for more documentation.