Overview

Applying a data model doesn’t guarantee the resulting infrastructure actually behaves as intended — configuration can drift, dependencies can fail silently, or a change elsewhere in the fabric can have unexpected side effects. nac-test closes that gap by continuously verifying live devices and controllers against the same YAML data model used to deploy them, instead of trusting a deployment on faith.

A CLI tool to render and execute Robot Framework and PyATS tests using Jinja templating. Combining Robot’s language agnostic syntax with the flexibility of Jinja templating allows dynamically rendering a set of test suites from the desired infrastructure state expressed in YAML syntax. PyATS support (experimental) adds Python-based operational tests for network infrastructure validation. Both test types can be executed together (default) or independently using development flags. Robot Framework and PyATS results are unified into a combined HTML dashboard and merged xUnit output, giving a single pass/fail view across both test engines.

$ nac-test --help

 Usage: nac-test [OPTIONS]

 A CLI tool to render and execute Robot Framework and PyATS tests using Jinja
 templating.

 Additional Robot Framework options can be passed at the end of the command to
 further control test execution (e.g., --variable, --listener, --loglevel).
 These are appended to the pabot invocation. Pabot-specific options and test
 files/directories are not supported and will result in an error.

╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ *  --data                  -d      PATH                                 Path to data YAML files. [env var: NAC_TEST_DATA] [required]                     │
│ *  --templates             -t      DIRECTORY                            Path to test templates. [env var: NAC_TEST_TEMPLATES] [required]                │
│ *  --output                -o      DIRECTORY                            Path to output directory. [env var: NAC_TEST_OUTPUT] [required]                 │
│    --filters               -f      DIRECTORY                            Path to Jinja filters. [env var: NAC_TEST_FILTERS]                               │
│    --tests                         DIRECTORY                            Path to Jinja tests. [env var: NAC_TEST_TESTS]                                  │
│    --include               -i      TEXT                                 Selects the test cases by tag (include). [env var: NAC_TEST_INCLUDE]            │
│    --exclude               -e      TEXT                                 Selects the test cases by tag (exclude). [env var: NAC_TEST_EXCLUDE]            │
│    --render-only                                                        Only render tests without executing them. [env var: NAC_TEST_RENDER_ONLY]      │
│    --dry-run                                                            Dry run mode: validates test structure without execution. [env var:            │
│                                                                         NAC_TEST_DRY_RUN]                                                                │
│    --processes                     INTEGER                              Number of parallel processes for test execution (pabot --processes option),    │
│                                                                         default is max(2, cpu count). [env var: NAC_TEST_PROCESSES]                      │
│    --pyats                                                              [DEV ONLY] Run only PyATS tests (skips Robot Framework). [env var:               │
│                                                                         NAC_TEST_PYATS]                                                                  │
│    --robot                                                              [DEV ONLY] Run only Robot Framework tests (skips PyATS). [env var:               │
│                                                                         NAC_TEST_ROBOT]                                                                  │
│    --max-parallel-devices          INTEGER                              Maximum number of devices to test in parallel for SSH/D2D tests. [env var:      │
│                                                                         NAC_TEST_MAX_PARALLEL_DEVICES]                                                   │
│    --minimal-reports                                                    Only include detailed command outputs for failed/errored tests in HTML reports  │
│                                                                         (80-95% artifact size reduction). [env var: NAC_TEST_MINIMAL_REPORTS]            │
│    --testbed                       FILE                                 Path to custom PyATS testbed YAML. [env var: NAC_TEST_TESTBED]                  │
│    --loglevel               -l     [DEBUG|INFO|WARNING|ERROR|CRITICAL]  Log level. Default: WARNING (or DEBUG if --verbose is set). [env var:            │
│                                                                         NAC_TEST_LOGLEVEL]                                                               │
│    --version                                                            Display version number.                                                         │
│    --diagnostic                                                         Wrap execution with diagnostic collection (Linux/macOS only).                    │
│    --verbose                                                            Enable verbose mode: enables verbose output for nac-test, Robot and PyATS        │
│                                                                         execution. [env var: NAC_TEST_VERBOSE]                                           │
│    --help                                                               Show this message and exit.                                                     │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

All data from the YAML files (--data option) will first be combined into a single data structure which is then provided as input to the templating process. Each template in the --templates path will then be rendered and written to the --output path. If the --templates path has subfolders, the folder structure will be retained when rendering the templates.

After all templates have been rendered Pabot will execute all test suites in parallel and create a test report in the --output path. The --skiponfailure non-critical argument will be used by default, meaning all failed tests with a non-critical tag will show up as “skipped” instead of “failed” in the final test report. Robot artifacts (e.g. log.html, report.html) are created under a robot_results/ subdirectory, with backward-compatible links kept at the root of the output directory.

Working with NaC Variables (IOS XE, NX-OS, IOS XR, Meraki)

Several Network as Code architectures (IOS XE as Code, NX-OS as Code, IOS XR as Code, and Meraki as Code) support a templating and variable system that allows configuration reuse across devices. This includes ${VARIABLE} substitution, model templates (type: model), CLI templates (type: cli), and file templates (type: file). These features are resolved by the Terraform module during terraform apply - not by nac-test.

nac-test merges all YAML files passed via --data into a single data structure using a deep merge, but it does not resolve these variable references or expand templates. If your project uses variables (defined under global.variables, device_groups[].variables, or devices[].variables), you must pass the Terraform-generated merged model file to --data rather than the raw data directory:

# Project WITHOUT variables - point directly at data/
nac-test --data ./data --templates ./tests/templates --output ./tests/results

# Project WITH variables - use the Terraform-generated model
nac-test --data ./model.yaml --templates ./tests/templates --output ./tests/results

Generating the resolved model file

The only way to produce the fully resolved model is through Terraform. Neither nac-validate --output nor nac-test perform variable resolution or template expansion - they only do a plain YAML merge. You must configure the Terraform module to write the resolved model to disk by setting write_model_file in your main.tf. The following example shows this for IOS XE as Code (NX-OS, IOS XR, and Meraki modules use the same attributes with their respective module sources):

module "iosxe" {
  source           = "git::https://github.com/netascode/terraform-iosxe-nac-iosxe.git?ref=v0.1.0"
  yaml_directories = ["data/"]
  write_model_file = "model.yaml"
}

After running terraform apply, the module writes model.yaml - the fully resolved per-device configuration with all variables substituted, templates rendered, and the global/group/device hierarchy flattened. This is the file you pass to nac-test.

Note

This limitation applies to the architectures listed above (IOS XE, NX-OS, IOS XR, and Meraki) which share a common templating system. Other Network as Code architectures (ACI, Catalyst SD-WAN, Catalyst Center, etc.) do not use this variable/template syntax and can pass their data/ directory to nac-test directly.