Usage

This page covers the options and behavior shared by both test engines.

Running Tests

A test run needs the data model (--data), the test directory (--templates) and an output directory (--output):

nac-test --data ./data --templates ./tests/templates --output ./tests/results

The --templates directory can contain both kinds of tests: Robot Framework templates (.robot files) and PyATS test files (.py files). By default, nac-test runs the PyATS tests first, followed by the Robot Framework tests. During test development, --pyats or --robot restrict a run to one engine for faster iteration.

Input Data

All YAML files passed via --data (files or directories) are combined into a single data structure using a deep merge. This merged data model is the input for both engines: Robot templates are rendered from it, and PyATS tests access it as self.data_model.

Values are loaded as written — there is no ${VARIABLE} substitution or other templating of the data files. The YAML tags below can be used to read values from the environment or from encrypted strings.

Reading Environment Variables

The !env YAML tag can be used to read values from environment variables.

root:
  name: !env VAR_NAME

Ansible Vault Support

Values in YAML files can be encrypted using Ansible Vault. This requires Ansible (ansible-vault command) to be installed and the following two environment variables to be defined:

export ANSIBLE_VAULT_ID=dev
export ANSIBLE_VAULT_PASSWORD=Password123

ANSIBLE_VAULT_ID is optional, and if not defined will be omitted.

Working with NaC Variables (IOS XE, NX-OS, IOS XR, Meraki)

Several Network as Code architectures (IOS XE as Code, NX-OS as Code, IOS XR as Code, and Meraki as Code) support a templating and variable system that allows configuration reuse across devices. This includes ${VARIABLE} substitution, model templates (type: model), CLI templates (type: cli), and file templates (type: file). These features are resolved by the Terraform module during terraform apply - not by nac-test.

nac-test merges all YAML files passed via --data into a single data structure using a deep merge, but it does not resolve these variable references or expand templates. If your project uses variables (defined under global.variables, device_groups[].variables, or devices[].variables), you must pass the Terraform-generated merged model file to --data rather than the raw data directory:

# Project WITHOUT variables - point directly at data/
nac-test --data ./data --templates ./tests/templates --output ./tests/results

# Project WITH variables - use the Terraform-generated model
nac-test --data ./model.yaml --templates ./tests/templates --output ./tests/results

Generating the resolved model file

The only way to produce the fully resolved model is through Terraform. Neither nac-validate --output nor nac-test perform variable resolution or template expansion - they only do a plain YAML merge. You must configure the Terraform module to write the resolved model to disk by setting write_model_file in your main.tf. The following example shows this for IOS XE as Code (NX-OS, IOS XR, and Meraki modules use the same attributes with their respective module sources):

module "iosxe" {
  source           = "git::https://github.com/netascode/terraform-iosxe-nac-iosxe.git?ref=v0.1.0"
  yaml_directories = ["data/"]
  write_model_file = "model.yaml"
}

After running terraform apply, the module writes model.yaml - the fully resolved per-device configuration with all variables substituted, templates rendered, and the global/group/device hierarchy flattened. This is the file you pass to nac-test.

Note

This limitation applies to the architectures listed above (IOS XE, NX-OS, IOS XR, and Meraki) which share a common templating system. Other Network as Code architectures (ACI, Catalyst SD-WAN, Catalyst Center, etc.) do not use this variable/template syntax and can pass their data/ directory to nac-test directly.

Select Test Cases By Tag

The --include (-i) and --exclude (-e) options select tests by tag for both engines. Robot Framework tests use Robot’s standard test case tags; PyATS tests are tagged with a groups class attribute:

class VerifyBgpNeighbors(SDWANManagerTestBase):
    groups = ["bgp", "routing"]  # Matches --include bgp or --include routing

Both engines use Robot Framework’s tag patterns, so matching works the same way:

• Simple tags: bgp, health, ospf
• Wildcards: bgp*, ?est
• Boolean operators: bgpANDrouting, bgpORospf, bgpNOTnrfu
• Matching is case-insensitive and ignores underscores
• To include several tags, repeat the option (-i bgp -i ospf) or use OR. A comma-separated value such as --include bgp,ospf is a single pattern and matches neither tag
• Once an --include pattern is set, untagged tests are no longer selected
• A test matching an --exclude pattern never runs

# Run only tests tagged with 'bgp'
nac-test -d data/ -t templates/ -o output/ --include bgp

# Run tests tagged 'bgp' or 'ospf', but not those tagged 'nrfu'
nac-test -d data/ -t templates/ -o output/ -i bgp -i ospf --exclude nrfu

Results and Reports

All results are written to the --output directory:

output
├── combined_summary.html    # Summary of all Robot and PyATS results
├── xunit.xml                # Merged xUnit results of both engines (for CI/CD)
├── robot_results/           # Rendered Robot suites and Robot reports
│   ├── log.html, report.html, output.xml, xunit.xml
│   └── summary_report.html
├── log.html, report.html, output.xml -> robot_results/...   # Links for backward compatibility
└── pyats_results/           # PyATS reports
    ├── api/                 # Controller (API) tests
    └── d2d/                 # Device (SSH) tests

Start with combined_summary.html, which links to the detailed reports of each engine. The exit code of nac-test reflects the test results, with zero meaning all tests passed.

--minimal-reports only includes detailed command outputs for failed or errored tests in the HTML reports, which considerably reduces the size of the artifacts.

Log Level

The log level is set with -l/--loglevel (DEBUG, INFO, WARNING, ERROR, CRITICAL; default WARNING). --verbose enables verbose output for nac-test, Robot and PyATS execution.

The -v/--verbosity option is deprecated as of nac-test 2.0 in favor of -l/--loglevel; it is still accepted (hidden from --help) but prints a deprecation warning.