Merging YAML

When dealing with large configurations, it becomes increasingly important to keep the configuration organized and modular. In order to achieve this, the configuration can be divided into multiple YAML files and folders. There is a specific logic applied to merge the content of multiple YAML files into a single configuration.

In general, the merging process follows these rules:

• The merging process is recursive, meaning that it will traverse through all nested dictionaries and lists.
• The merging process will deep merge content of nested dictionaries. This means that if the same key is present in multiple files, the values will be combined into a single dictionary.
• The merging process will overwrite the values of the same keys if the value is a primitive type (string, number, boolean). Therefore it should be avoided to use the same key in different files in such cases.
• The merging process will append the values of the same keys if the value is a list. This means that if the same key is present in multiple files, the values will be combined into a single list.
• If it is a list of dictionaries, the merging process will combine (deep merge) items if the values of all shared primitive keys match. Only keys with primitive values (string, number, boolean) will be considered for matching. Items will merge even if both sides have additional unique primitive keys.
• If any single file contains duplicate dictionary items in a list (items where all shared primitive values match), merging is disabled for that entire list and items are concatenated instead.

Merging Dictionaries

Merging Flat Dictionaries

The merging process will deep merge two dictionaries with the same name, combining the keys of both dictionaries.

File 1

dict:
  key1: value1

File 2

dict:
  key2: value2

Result

dict:
  key1: value1
  key2: value2

Merging Nested Dictionaries

The merging process will also deep merge two nested dictionaries, combining the keys of both dictionaries.

File 1

dict:
  name: a
  nested_dict:
    key1: value1
    key2: value2

File 2

dict:
  name: a
  nested_dict:
    key3: value3

Result

dict:
  name: a
  nested_dict:
    key1: value1
    key2: value2
    key3: value3

Merging Lists

Merging Lists of Primitive Values

The merging process will combine two lists of primitive values (strings, numbers, booleans) into a single list. The order of the values will be preserved, and duplicate values will be retained.

File 1

list:
  - value1
  - value1
  - value2

File 2

list:
  - value1

Result

list:
  - value1
  - value1
  - value2
  - value1

Merging Lists of Dictionaries

The merging process will combine two lists of dictionaries with the same name into a single list. The merging process will deep merge dictionary items if the values of all shared primitive keys match. Only keys with primitive values (string, number, boolean) are considered for matching. Items will merge even if both sides have additional unique primitive keys.

File 1

list:
  - name: a
    key1: value1
    dict:
      key1: value1

File 2

list:
  - name: a
    dict:
      key2: value2

Result

list:
    - name: a
      key1: value1
      dict:
        key1: value1
        key2: value2

Lists of dictionaries (list) will be merged by the value of all matching keys, while only considering primitive values. In this example the only matching attribute with primitive values is name and only one item has an extra key (key1) with a primitive value, so the two list items will be deep merged into a single item.

File 1

list:
  - name: a
    key1: value1
    dict:
      key1: value1

File 2

list:
  - name: a
    key2: value2
    dict:
      key2: value2

Result

list:
    - name: a
      key1: value1
      key2: value2
      dict:
        key1: value1
        key2: value2

In this example the only shared attribute with primitive values is name and the values match, so the two list items will be deep merged into a single item even though both sides have additional unique primitive keys (key1 and key2).

Duplicate-Aware Concatenation

If any single file contains duplicate dictionary items in a list (items where all shared primitive values match), merging is disabled for that entire list. Instead, items from all files are concatenated.

File 1

list:
  - name: a
    key1: value1
  - name: a
    key2: value2

File 2

list:
  - name: a
    key3: value3

Result

list:
    - name: a
      key1: value1
    - name: a
      key2: value2
    - name: a
      key3: value3

In this example, File 1 contains two items with name: a (all shared primitive values match), so merging is disabled for the entire list. All items are concatenated instead of merged.