Automation Pipelines

Automation pipelines are a set of automated processes that allow you to build, test, and deploy your code in a consistent and repeatable manner. They are essential for ensuring that your code is always in a deployable state and that any changes made to the code are thoroughly tested before being deployed to production.

We hear this in the context of software development, which isn’t what you are normally used to discussing with network operations. Yet many of these concepts are applicable to network operations as well. In the context of Network as Code with SD-WAN, we utilize Terraform and automated testing to deploy configuration changes to SD-WAN Manager. The important element is that network changes are integrated into this software pipeline such that changes are integrated with validation and other software practices.

An automation pipeline is a series of automated steps that take your code from development to production. It typically includes the following stages:

1. Source Control: The code is stored in a version control system (VCS) such as Git. This allows for tracking changes, collaboration, and rollback if necessary.
2. Validate: The code is validated for syntax, semantics, and compliance before any deployment attempts.
3. Plan: The system calculates what changes need to be made to reach the desired state.
4. Deploy: The validated changes are applied to the target environment, such as SD-WAN Manager.
5. Test: Automated tests are run to ensure that the deployed configuration matches the intended state.

We can make the connection between these high level software concepts and network operations by looking at the following table:

When you observe this in the context of software development, developers commit code into branches that are processed via software linting and other tools to ensure the code matches expected standards. Once peer review approves the code this is then moved to control branches like develop for further testing and integration. Finally code is then merged, compiled and tested to established standards.

The advantage of this for SD-WAN configuration changes is the ability to integrate what we have mentioned before: Semantic validation, pre and post testing, and automated deployment. Network engineers simply codify intended state and the software pipeline is executed transparently to ensure all the required steps are completed to push the configuration correctly to the SD-WAN fabric.

For Network as Code with SD-WAN, we combine the different stages of execution using Terraform to create different stages based on Infrastructure as Code principles.

The SD-WAN as Code pipeline consists of five main stages:

stages:
  - validate
  - plan
  - deploy
  - test
  - notify

Each stage has specific responsibilities and dependencies, ensuring a robust and reliable deployment process.

The validation stage is the first and most critical step in our pipeline, serving as a gatekeeper to ensure that only valid configurations proceed to deployment. This stage performs comprehensive checks on the SD-WAN data model before any changes are applied to the network infrastructure.

What happens during validation:

1. Terraform Format Check: Ensures all Terraform files follow proper formatting standards:

   terraform fmt -check

2. Data Model Validation: Uses nac-validate to check data model syntax and semantics:

   nac-validate ./data/

3. Schema Compliance: The data model is validated against the predefined SD-WAN schema to ensure all required fields are present and data types are correct. This includes checking for:

   • Mandatory fields like chassis IDs and device models
   • Proper IP address formats
   • Valid device template references
   • Correct feature template associations

4. Semantic Validation: Beyond basic schema compliance, the validation includes semantic checks that ensure the configuration makes logical sense from an SD-WAN perspective:

   • Device template and feature template compatibility
   • Policy object references exist
   • Site configurations match template requirements
   • Variable consistency across templates

validate:
  stage: validate
  script:
    - set -o pipefail && terraform fmt -check |& tee fmt_output.txt
    - set -o pipefail && nac-validate ./data/ |& tee validate_output.txt
  artifacts:
    paths:
      - fmt_output.txt
      - validate_output.txt
  cache: []
  rules:
    - if: $CI_COMMIT_TAG == null

Branch Strategy Integration: The validation stage runs on all branches except tags, providing early feedback to network engineers. When a network operator commits changes to any branch, the pipeline automatically runs validation-only jobs, allowing for rapid iteration and error correction.

Benefits of Early Validation:

• Fast Feedback Loop: Engineers receive immediate feedback on their changes without waiting for full deployment cycles
• Reduced Risk: Invalid configurations are caught before they can impact the SD-WAN fabric
• Collaborative Development: Multiple engineers can work on different aspects of the SD-WAN configuration with confidence
• Cost Efficiency: Prevents costly deployment failures by catching errors early

The planning stage uses Terraform’s planning capabilities to calculate exactly what changes need to be made to the SD-WAN fabric to reach the desired state. This stage provides a preview of changes before any actual deployment occurs.

Planning Process:

1. Terraform Initialization: The pipeline initializes Terraform, downloading required providers and modules:

   terraform init -input=false

2. Plan Generation: Creates a detailed execution plan:

   terraform plan -out=plan.tfplan -input=false

3. Plan Analysis: Converts the plan to multiple formats for analysis:

   terraform show -no-color plan.tfplan > plan.txt
   terraform show -json plan.tfplan | jq > plan.json

4. Change Summary: Generates a summary of planned changes:

   terraform show -json plan.tfplan | jq '([.resource_changes[]?.change.actions?]|flatten)|{"create":(map(select(.=="create"))|length),"update":(map(select(.=="update"))|length),"delete":(map(select(.=="delete"))|length)}' > plan_gitlab.json

plan:
  stage: plan
  resource_group: sdwan
  script:
    - terraform init -input=false
    - terraform plan -out=plan.tfplan -input=false
    - terraform show -no-color plan.tfplan > plan.txt
    - terraform show -json plan.tfplan | jq > plan.json
    - terraform show -json plan.tfplan | jq '([.resource_changes[]?.change.actions?]|flatten)|{"create":(map(select(.=="create"))|length),"update":(map(select(.=="update"))|length),"delete":(map(select(.=="delete"))|length)}' > plan_gitlab.json
    - python3 .ci/gitlab-comment.py
  artifacts:
    paths:
      - plan.json
      - plan.txt
      - plan.tfplan
      - plan_gitlab.json
    reports:
      terraform: plan_gitlab.json
  dependencies: []
  needs:
    - validate
  only:
    - merge_requests
    - master

Planning Benefits:

• Change Visibility: Engineers can review exactly what will change before deployment
• Risk Assessment: Plans help identify potentially disruptive changes
• Approval Workflows: Plans can be reviewed in merge requests before deployment
• Consistency: The same plan is used for deployment, ensuring no drift between planning and execution
• Resource Groups: The sdwan resource group ensures only one plan/deploy can run at a time

The deployment stage transforms validated configuration data into actual SD-WAN state through Terraform’s declarative deployment model. This stage represents the critical transition from intent (as expressed in the data model) to reality (as configured in SD-WAN Manager).

Deployment Process:

1. Terraform Initialization: Re-initializes Terraform to ensure consistency
2. Plan Application: Applies the exact plan generated in the previous stage:

   terraform apply -input=false -auto-approve plan.tfplan

deploy:
  stage: deploy
  resource_group: sdwan
  script:
    - terraform init -input=false
    - terraform apply -input=false -auto-approve plan.tfplan
  dependencies:
    - plan
  needs:
    - plan
  only:
    - master

Safety Mechanisms:

• Branch Protection: Deploy stage only runs on the master branch
• Dependency Validation: Explicit dependencies on successful validation and planning stages
• Resource Groups: Prevents concurrent deployments that could cause conflicts
• Plan Consistency: Uses the exact plan from the previous stage, ensuring predictable results
• State Management: Terraform maintains state consistency across deployments

Deployment Benefits:

• Predictable Changes: Uses pre-approved plans for deployment
• Atomic Operations: Changes are applied in the correct order with proper dependencies
• State Tracking: Terraform state provides accurate tracking of managed resources
• Rollback Capability: Previous configurations can be restored if needed

The testing stage provides critical verification that the deployed configuration matches the intended state and that the SD-WAN fabric is functioning as expected. This stage includes both integration testing and idempotency verification.

Validates that the deployed configuration matches the data model:

test-integration:
  stage: test
  script:
    - set -o pipefail && iac-test -d ./data -d ./defaults.yaml -t ./tests/templates -f ./tests/filters -o ./tests/results/sdwan |& tee test_output.txt
  artifacts:
    when: always
    paths:
      - tests/results/sdwan/*.html
      - tests/results/sdwan/xunit.xml
      - test_output.txt
    reports:
      junit: tests/results/sdwan/xunit.xml
  dependencies:
    - deploy
  needs:
    - deploy
  only:
    - master

Verifies that the system is in a stable state with no configuration drift:

test-idempotency:
  stage: test
  resource_group: sdwan
  script:
    - terraform init -input=false
    - terraform plan -input=false -detailed-exitcode
  dependencies:
    - deploy
  needs:
    - deploy
  only:
    - master

What Testing Validates:

1. Configuration Accuracy: Verifies that the configuration deployed to SD-WAN Manager exactly matches what was defined in the data model
2. Template Integrity: Confirms device templates and feature templates are properly created and associated
3. Policy Application: Validates that centralized and localized policies are correctly applied
4. Site Configuration: Ensures devices are properly attached to templates with correct variables
5. System Stability: Idempotency tests confirm no unexpected changes are detected

Testing Benefits:

• Confidence in Changes: Engineers can be confident that deployed changes work as intended
• Rapid Issue Detection: Problems are identified immediately after deployment
• Compliance Documentation: Test reports serve as evidence for change documentation and audit processes
• Drift Detection: Idempotency tests identify configuration drift from manual changes

The notification stage provides stakeholders with real-time updates about deployment status and results through Webex Teams integration.

success:
  stage: notify
  script:
    - python3 .ci/webex-notification-gitlab.py -s
  when: on_success
  artifacts:
    when: always
    paths:
      - tests/results/sdwan/*.html
      - tests/results/sdwan/xunit.xml
      - plan.txt
      - fmt_output.txt
      - validate_output.txt
      - test_output.txt
  cache: []

failure:
  stage: notify
  script:
    - python3 .ci/webex-notification-gitlab.py -f
  when: on_failure
  artifacts:
    when: always
    paths:
      - tests/results/sdwan/*.html
      - tests/results/sdwan/xunit.xml
      - plan.txt
      - fmt_output.txt
      - validate_output.txt
      - test_output.txt
  cache: []

Notification Features:

• Webex Teams Integration: Sends rich notifications to designated Webex spaces
• Status Awareness: Different messages for success and failure scenarios
• Artifact Collection: All pipeline outputs are collected and made available
• Stakeholder Communication: Keeps teams informed without manual monitoring

The SD-WAN pipeline uses environment variables to securely manage credentials and configuration:

variables:
  ## Disable SSL verification for git operations (lab environments only)
  GIT_SSL_NO_VERIFY: "true"

  ## SD-WAN Manager Connection
  SDWAN_USERNAME:
    description: "Cisco SDWAN Username"
  SDWAN_PASSWORD:
    description: "Cisco SDWAN Password"
  SDWAN_URL:
    description: "Cisco SDWAN URL"

  ## GitLab Integration
  GITLAB_TOKEN:
    description: "User Access Token for GitLab API operations"
  GITLAB_API_URL:
    description: "GitLab API v4 root URL"
    value: "${CI_API_V4_URL}"

  ## Terraform State Management
  TF_HTTP_ADDRESS:
    description: "GitLab HTTP Address to store the TF state file"
    value: "${GITLAB_API_URL}/projects/${CI_PROJECT_ID}/terraform/state/tfstate"
  TF_HTTP_USERNAME:
    description: "GitLab Username for state operations"
  TF_HTTP_PASSWORD:
    description: "GitLab Access Token for state operations"

  ## Webex Notifications
  WEBEX_ROOM_ID:
    description: "Cisco Webex Room ID for notifications"
  WEBEX_TOKEN:
    description: "Cisco Webex Bot Token"

The pipeline uses GitLab as the Terraform backend for state management:

cache:
  key: terraform_modules_and_lock
  paths:
    - .terraform
    - .terraform.lock.hcl
    - defaults.yaml

Benefits of GitLab State Backend:

• Centralized State: All team members access the same state
• State Locking: Prevents concurrent modifications
• Version History: GitLab maintains state version history
• Access Control: Leverages GitLab’s existing security model

1. Create Feature Branch: Developers create branches for specific changes

   git checkout -b conf-change

2. Make Changes: Modify data model files as needed

3. Push Branch: Trigger validation and planning stages

   git add .
   git commit -m "Update TLOC preferences"
   git push origin conf-change

4. Create Merge Request: Request review and approval

5. Merge to Master: Trigger full deployment pipeline

This strategy provides:

• Safe Development: Feature branches only run validation
• Change Preview: Merge requests show planned changes
• Protected Deployment: Only master branch deploys to production

Risk Reduction: Every change is validated multiple times - first against schema and rules, then through Terraform planning, and finally through post-deployment testing. This multi-layered approach dramatically reduces the risk of network outages or misconfigurations.

Consistency: All network changes follow the same standardized process, eliminating the variability that comes from manual processes or individual engineer preferences.

Auditability: Every change is tracked in version control, with complete visibility into who made what changes, when they were made, and what testing was performed.

Scalability: As SD-WAN complexity grows, the pipeline approach scales much better than manual processes, allowing teams to manage larger and more complex network infrastructures.

• Credential Management: Use GitLab CI/CD variables with masking enabled
• State Security: Leverage GitLab’s access controls for state management
• Network Security: Use secure connections to SD-WAN Manager (HTTPS)

• Resource Groups: Prevent concurrent pipeline runs that could cause conflicts
• Artifact Preservation: Save all pipeline outputs for troubleshooting
• Comprehensive Testing: Include both configuration and idempotency tests

• Branch Protection: Require reviews before merging to master
• Clear Notifications: Use Webex integration to keep teams informed
• Documentation: Maintain clear commit messages and change descriptions

• Caching: Cache Terraform modules and state between runs
• Parallel Execution: Run independent tests in parallel where possible
• Selective Execution: Use rules to control when stages execute

Automation pipelines represent a fundamental shift in how SD-WAN changes are managed, bringing the reliability and consistency of software development practices to network operations. By implementing the multi-stage pipeline approach (validate, plan, deploy, test, notify) with Network as Code for SD-WAN, organizations can significantly reduce the risk of network outages while improving the speed and consistency of network changes.

The combination of data model validation, Terraform-based deployment, and post-deployment testing creates a robust framework that ensures SD-WAN configurations are both syntactically correct and operationally sound. This foundation prepares network engineering teams to embrace Infrastructure as Code practices while maintaining the reliability and security that network operations demand.

The pipeline approach provides:

• Predictable Change Management: Every change follows the same validated process
• Risk Mitigation: Multiple validation layers catch errors before they impact production
• Operational Visibility: Complete audit trail of all changes and their impacts
• Scalable Operations: Handle complex SD-WAN deployments with confidence
• Team Collaboration: Enable multiple engineers to work on SD-WAN configurations safely

In the next section, we will implement a working automation pipeline for Network as Code with SD-WAN, putting these concepts into practice with hands-on configuration and deployment.