Overview
Applying a data model doesn’t guarantee the resulting infrastructure actually behaves as intended — configuration can drift, dependencies can fail silently, or a change elsewhere in the fabric can have unexpected side effects. nac-test closes that gap by continuously verifying live devices and controllers against the same YAML data model used to deploy them, instead of trusting a deployment on faith.
A CLI tool to render and execute Robot Framework and PyATS tests using Jinja templating. Combining Robot’s language agnostic syntax with the flexibility of Jinja templating allows dynamically rendering a set of test suites from the desired infrastructure state expressed in YAML syntax. PyATS support (experimental) adds Python-based operational tests for network infrastructure validation. Both test types can be executed together (default) or independently using development flags. Robot Framework and PyATS results are unified into a combined HTML dashboard and merged xUnit output, giving a single pass/fail view across both test engines.
$ nac-test --help
Usage: nac-test [OPTIONS]
A CLI tool to render and execute Robot Framework and PyATS tests using Jinja templating.
Additional Robot Framework options can be passed at the end of the command to further control test execution (e.g., --variable, --listener, --loglevel). These are appended to the pabot invocation. Pabot-specific options and test files/directories are not supported and will result in an error.
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮│ * --data -d PATH Path to data YAML files. [env var: NAC_TEST_DATA] [required] ││ * --templates -t DIRECTORY Path to test templates. [env var: NAC_TEST_TEMPLATES] [required] ││ * --output -o DIRECTORY Path to output directory. [env var: NAC_TEST_OUTPUT] [required] ││ --filters -f DIRECTORY Path to Jinja filters. [env var: NAC_TEST_FILTERS] ││ --tests DIRECTORY Path to Jinja tests. [env var: NAC_TEST_TESTS] ││ --include -i TEXT Selects the test cases by tag (include). [env var: NAC_TEST_INCLUDE] ││ --exclude -e TEXT Selects the test cases by tag (exclude). [env var: NAC_TEST_EXCLUDE] ││ --render-only Only render tests without executing them. [env var: NAC_TEST_RENDER_ONLY] ││ --dry-run Dry run mode: validates test structure without execution. [env var: ││ NAC_TEST_DRY_RUN] ││ --processes INTEGER Number of parallel processes for test execution (pabot --processes option), ││ default is max(2, cpu count). [env var: NAC_TEST_PROCESSES] ││ --pyats [DEV ONLY] Run only PyATS tests (skips Robot Framework). [env var: ││ NAC_TEST_PYATS] ││ --robot [DEV ONLY] Run only Robot Framework tests (skips PyATS). [env var: ││ NAC_TEST_ROBOT] ││ --max-parallel-devices INTEGER Maximum number of devices to test in parallel for SSH/D2D tests. [env var: ││ NAC_TEST_MAX_PARALLEL_DEVICES] ││ --minimal-reports Only include detailed command outputs for failed/errored tests in HTML reports ││ (80-95% artifact size reduction). [env var: NAC_TEST_MINIMAL_REPORTS] ││ --testbed FILE Path to custom PyATS testbed YAML. [env var: NAC_TEST_TESTBED] ││ --loglevel -l [DEBUG|INFO|WARNING|ERROR|CRITICAL] Log level. Default: WARNING (or DEBUG if --verbose is set). [env var: ││ NAC_TEST_LOGLEVEL] ││ --version Display version number. ││ --diagnostic Wrap execution with diagnostic collection (Linux/macOS only). ││ --verbose Enable verbose mode: enables verbose output for nac-test, Robot and PyATS ││ execution. [env var: NAC_TEST_VERBOSE] ││ --help Show this message and exit. │╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯All data from the YAML files (--data option) will first be combined into a single data structure which is then provided as input to the templating process. Each template in the --templates path will then be rendered and written to the --output path. If the --templates path has subfolders, the folder structure will be retained when rendering the templates.
After all templates have been rendered Pabot will execute all test suites in parallel and create a test report in the --output path. The --skiponfailure non-critical argument will be used by default, meaning all failed tests with a non-critical tag will show up as “skipped” instead of “failed” in the final test report. Robot artifacts (e.g. log.html, report.html) are created under a robot_results/ subdirectory, with backward-compatible links kept at the root of the output directory.