Usage

Log Level

The -v/--verbosity option is deprecated as of nac-test 2.0 in favor of -l/--loglevel; it is still accepted (hidden from --help) but prints a deprecation warning.

Ansible Vault Support

Values in YAML files can be encrypted using Ansible Vault. This requires Ansible (ansible-vault command) to be installed and the following two environment variables to be defined:

export ANSIBLE_VAULT_ID=dev
export ANSIBLE_VAULT_PASSWORD=Password123

ANSIBLE_VAULT_ID is optional, and if not defined will be omitted.

Additional Tags

Reading Environment Variables

The !env YAML tag can be used to read values from environment variables.

root:
  name: !env VAR_NAME

Example

data.yaml located in ./data folder:

---
root:
  children:
    - name: ABC
      param: value
    - name: DEF
      param: value

test1.robot located in ./templates folder:

*** Settings ***
Documentation   Test1

*** Test Cases ***
{% for child in root.children | default([]) %}

Test {{ child.name }}
    Should Be Equal   {{ child.param }}   value
{% endfor %}

After running nac-test with the following parameters:

nac-test --data ./data --templates ./templates --output ./tests

The following rendered Robot test suite can be found in the ./tests folder:

*** Settings ***
Documentation   Test1

*** Test Cases ***

Test ABC
    Should Be Equal   value   value

Test DEF
    Should Be Equal   value   value

As well as the test results and reports:

$ tree -L 2 tests
tests
├── combined_summary.html
├── robot_results/
│   ├── log.html
│   ├── output.xml
│   ├── report.html
│   └── xunit.xml
├── log.html -> robot_results/log.html
├── output.xml -> robot_results/output.xml
├── report.html -> robot_results/report.html
├── xunit.xml -> robot_results/xunit.xml
├── pabot_results/
└── test1.robot

Note: as of nac-test 2.0, Robot artifacts are created under robot_results/, with the root-level log.html, output.xml, report.html, and xunit.xml kept as links for backward compatibility. A combined_summary.html report is also generated, linking Robot and (if used) PyATS result summaries.

Custom Jinja Filters

Custom Jinja filters can be used by providing a set of Python classes where each filter is implemented as a separate Filter class in a .py file located in the --filters path. The class must have a single attribute named name, the filter name, and a classmethod() named filter which has one or more arguments. A sample filter can be found below.

class Filter:
    name = "filter1"

    @classmethod
    def filter(cls, data):
        return str(data) + "_filtered"

Custom Jinja Tests

Custom Jinja tests can be used by providing a set of Python classes where each test is implemented as a separate Test class in a .py file located in the --tests path. The class must have a single attribute named name, the test name, and a classmethod() named test which has one or more arguments. A sample test can be found below.

class Test:
    name = "test1"

    @classmethod
    def test(cls, data1, data2):
        return data1 == data2

Rendering Directives

Special rendering directives exist to render a single test suite per (YAML) list item. The directive can be added to the Robot template as a Jinja comment following this syntax:

{# iterate_list <YAML_PATH_TO_LIST> <LIST_ITEM_ID> <JINJA_VARIABLE_NAME> #}

After running nac-test with the data from the previous example and the following template:

{# iterate_list root.children name child_name #}
*** Settings ***
Documentation   Test1

*** Test Cases ***
{% for child in root.children | default([]) %}
{% if child.name == child_name %}

Test {{ child.name }}
    Should Be Equal   {{ child.param }}   value
{% endif %}
{% endfor %}

The following test suites will be rendered:

$ tree -L 2 tests
tests
├── ABC
│   └── test1.robot
└── DEF
    └── test1.robot

A similar directive exists to put the test suites in a common folder though with a unique filename.

{# iterate_list_folder <YAML_PATH_TO_LIST> <LIST_ITEM_ID> <JINJA_VARIABLE_NAME> #}

The following test suites will be rendered:

$ tree -L 2 tests
tests
└── test1
    ├── ABC.robot
    └── DEF.robot

An additional directive exists to render a single test suite per (YAML) list item in chunks, which is useful for handling large datasets by splitting them across multiple template files. This is a variant of iterate_list that would still create separate folders.

Note: This directive is experimental and may change in future versions. It is not subject to semantic versioning guarantees.

{# iterate_list_chunked <YAML_PATH_TO_LIST> <LIST_ITEM_ID> <JINJA_VARIABLE_NAME> <OBJECT_PATH> <CHUNK_SIZE> #}

All objects under the OBJECT_PATH will be counted and if their number is greater than the specified chunk size, the list will be split into multiple test suites with suffix _2, _3, etc.

Consider the following example:

---
root:
  children:
    - name: ABC
      param: value
      nested_children:
        - name: Child1
          param: value
        - name: Child2
          param: value
        - name: Child3
          param: value
    - name: DEF
      param: value
      nested_children:
        - name: Child1
          param: value

After running nac-test with this data from the previous and the following template:

{# iterate_list_chunked root.children name child_name nested_children 2 #}
*** Settings ***
Documentation   Test1

*** Test Cases ***
{% for child in root.children | default([]) %}
{% if child.name == child_name %}

Test {{ child.name }}
    Should Be Equal   {{ child.param }}   value

{% for nested_child in child.nested_children | default([]) %}

Test {{ child.name }} Child {{ nested_child.name }}
    Should Be Equal   {{ nested_child.param }}   value
{% endfor %}

{% endif %}
{% endfor %}

Objects from the nested_children path will be counted and if their number is greater than the specified chunk size (2), the list will be split into multiple test suites with suffix _002, _003, etc. The following test suites will be rendered:

$ tree -L 2 tests
tests
├── ABC
│   ├── test1_001.robot
│   └── test1_002.robot
└── DEF
    └── test1_001.robot

Select Test Cases By Tag

It is possible to include and exclude test cases by tag names with the --include and --exclude CLI options. These options are directly passed to the Pabot/Robot executor and are documented here.

Test Case Parallelization

By default, nac-test (via pabot) executes test suites (i.e., each robot file) in parallel. The number of parallel processes can be controlled via the --processes option.

However, suite-level parallelization may be inefficient for test suites containing multiple long-running test cases (e.g., >10 seconds each). If your test cases are independent and can run concurrently, you can enable test-level parallelization by adding the following metadata to the suite’s settings:

*** Settings ***
Metadata        Test Concurrency     True

Note: This approach benefits only long-running tests. For short tests, the scheduling overhead and log collection may offset any performance gains.

Tip: The Test Concurrency metadata is case-insensitive (test concurrency, TEST CONCURRENCY, etc.).

Implementation: nac-test checks the rendered robot files for the Metadata setting and instruct pabot to run each test within the respective suite in parallel (using pabot’s --testlevelsplit --ordering ordering.txt arguments). You can inspect the ordering.txt file in the output directory.

This behaviour can be disabled by setting the environment variable NAC_TEST_DISABLE_TESTLEVELSPLIT=true (as of nac-test 2.0; previously named NO_TESTLEVELSPLIT).