Semantic Validation

A data model can be valid against its schema and still be wrong: an interface can reference a node that doesn’t exist, two objects can use the same VLAN, or a name can violate an organization’s naming convention. Such checks depend on the relationships between values, possibly across the whole data model, and cannot be expressed in a schema. Semantic validation covers them with rules written in Python.

Rules are passed with -r/--rules (repeatable) and default to the .rules/ directory in the current directory.

How Rules Are Executed

Every .py file directly in the rules directories is loaded as a rule. Subdirectories are not searched.
Semantic validation only runs if syntactic validation passed.
All YAML files are merged into a single data structure (see Input Data), which is passed to every rule.
All rules are executed, and every rule returning violations is reported. If any rule returns a violation, nac-validate exits with code 1.

The data passed to rules is the merged content of the YAML files, as written. Default values of the data model are not applied, so optional keys may be missing or set to null. Variables and templates of Network as Code architectures that support them are not resolved either.

Writing a Rule

Each rule is a Python file containing a class named Rule, which subclasses RuleBase and implements a match classmethod. match receives the merged data and returns a list of violations — an empty list if the data is valid.

The following rule, saved as .rules/101_interface_node_reference.py, verifies that interfaces only reference nodes that are defined in the data model:

from nac_validate import RuleBase, Violation


class Rule(RuleBase):
    id = "101"
    description = "Verify interfaces reference existing nodes"
    severity = "HIGH"
    title = "Interface references unknown node"
    explanation = "Every interface must belong to a node defined under fabric.nodes."
    recommendation = "Add the node to fabric.nodes or fix the node_id of the interface."
    affected_items_label = "Affected Interfaces"

    @classmethod
    def match(cls, data):
        fabric = data.get("fabric") or {}
        node_ids = {node["id"] for node in fabric.get("nodes") or []}
        violations = []
        for interface in fabric.get("interfaces") or []:
            if interface["node_id"] not in node_ids:
                violations.append(
                    Violation(
                        message=f"Node {interface['node_id']} of interface {interface['name']} is not defined",
                        path=f"fabric.interfaces[node_id={interface['node_id']},name={interface['name']}].node_id",
                        details={"node_id": interface["node_id"]},
                    )
                )
        return violations

Validating data with an interface on an undefined node 103 reports:

[RULE 101] Verify interfaces reference existing nodes

================================================================================
Interface references unknown node
================================================================================

Found 1 violation(s).

Affected Interfaces:
  • Node 103 of interface eth1/3 is not defined
    Path: fabric.interfaces[node_id=103,name=eth1/3].node_id

--------------------------------------------------------------------------------
WHY THIS MATTERS:
--------------------------------------------------------------------------------

Every interface must belong to a node defined under fabric.nodes.

────────────────────────────────────────────────────────────────────────────────
RECOMMENDED FIX:
────────────────────────────────────────────────────────────────────────────────

Add the node to fabric.nodes or fix the node_id of the interface.

================================================================================

────────────────────────────────────────────────────────────────────────────────
Validation Summary
────────────────────────────────────────────────────────────────────────────────
  ✓ Syntax validation: PASSED (2 files)
  ✗ Semantic validation: FAILED (1 violation)

    • 1 HIGH severity violation (rules: 101)
────────────────────────────────────────────────────────────────────────────────

Rule Attributes

Attribute	Required	Default	Description
`id`	Yes	—	Unique rule ID, a number such as `"101"` or `101`. Other IDs log a warning.
`description`	Yes	—	Short description, shown in the output and in `--list-artifacts`
`severity`	No	`"HIGH"`	`HIGH`, `MEDIUM` or `LOW`. Used to group violations in the summary and to color the output.
`title`	No	`""`	Header of the violation output
`explanation`	No	`""`	Why the violation matters
`recommendation`	No	`""`	How to fix it, possibly with a YAML example
`affected_items_label`	No	`"Affected Items"`	Label of the list of violations
`references`	No	`[]`	Links to further documentation

The severity is informational: a violation of a rule with any severity fails the validation. If title, explanation or recommendation is set, the detailed output above is shown with the sections that are set. Otherwise, and with --compact, the violations are listed without these sections:

[RULE 101] Verify interfaces reference existing nodes

────────────────────────────────────────────────────────────────────────────────
  • fabric.interfaces[node_id=103,name=eth1/3].node_id - Node 103 of interface eth1/3 is not defined
────────────────────────────────────────────────────────────────────────────────

Rule IDs must be unique across all rules directories. A common convention is to group them by the kind of check, for example 1xx for unique keys, 2xx for references and 3xx for feature-specific checks.

The `match` Method

match accepts one or two arguments:

@classmethod
def match(cls, data):
    ...

@classmethod
def match(cls, data, schema):
    ...

data is the merged data model as plain Python dictionaries and lists. schema is the loaded Yamale schema (yamale.schema.Schema), or None if no schema is used. It allows writing generic rules that derive what to check from the schema instead of hardcoding paths.

Violations

match returns a list of Violation objects:

Field	Required	Description
`message`	Yes	Human-readable description of the problem
`path`	No	Location in the data model, e.g. `fabric.interfaces[node_id=103,name=eth1/3].node_id`
`details`	No	Dictionary of machine-readable data, included in the JSON output

Paths are free text, but following the format used for schema errors — dot-separated keys and list items identified by [key=value] — makes violations easy to locate.

Instead of Violation objects, match can also return a list of strings, which are shown as the messages. This is the format of rules written for earlier versions of nac-validate, which also didn’t require RuleBase:

class Rule:
    id = "102"
    description = "Verify node names"
    severity = "LOW"

    @classmethod
    def match(cls, data):
        results = []
        for node in (data.get("fabric") or {}).get("nodes") or []:
            if not node["name"].startswith(("leaf-", "spine-")):
                results.append(f"fabric.nodes[id={node['id']}].name - {node['name']}")
        return results

Writing Robust Rules

Handle missing data. Rules run against any valid data model, in which most keys are optional. An exception raised in a rule stops the validation with exit code 3 and a message that doesn’t name the rule, such as ERROR - Unexpected error: 'interfaces'. Use .get() and treat null like a missing key:

fabric = data.get("fabric") or {}
for node in fabric.get("nodes") or []:
    ...

Fields that the schema marks as required, such as a primary key, can be accessed directly.

Rely on the schema. Rules only run on data that passed the schema, so they don’t need to check types or formats again. Leave structural checks, such as unique keys within a list, to the schema where possible (see Primary Keys).

Consider defaults. As default values are not applied, a missing key means “use the default”. If a rule depends on a value that has a default, use the same default in the rule.

Report every violation. Return one violation per affected object rather than stopping at the first one, so all problems can be fixed in a single iteration.

Keep each file self-contained. Every .py file in a rules directory must contain a Rule class, so helper modules cannot be placed next to the rules. Share code between rules by installing it as a Python package instead.

Testing Rules

As match is a classmethod working on plain Python data, rules can be tested with pytest without running nac-validate:

import importlib.util
from pathlib import Path


def load_rule(path):
    spec = importlib.util.spec_from_file_location("rule", path)
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)
    return module.Rule


Rule = load_rule(Path(".rules/101_interface_node_reference.py"))


def test_undefined_node():
    data = {
        "fabric": {
            "nodes": [{"id": 101, "name": "leaf-101", "role": "leaf"}],
            "interfaces": [{"node_id": 103, "name": "eth1/3"}],
        }
    }
    violations = Rule.match(data)
    assert [v.path for v in violations] == ["fabric.interfaces[node_id=103,name=eth1/3].node_id"]


def test_missing_data():
    assert Rule.match({}) == []

nac-validate --list-artifacts lists all rules found in the rules directories, with their ID, description and severity, without executing them.