Processing JSON in Ansible

I started using Chezmoi to manage my dotfiles a few years ago but I haven't really used it to its full potential. One of the reasons is because I couldn't understand how to use Go Templates at the time I started using Chezmoi. The other reason is that using Chezmoi to try and manage dotfiles outside $HOME isn't recommended. Instead, Tom Payne, the developer of Chezmoi, recommends using something like Ansible for the task of managing files in /etc. For me, this was a good excuse to learn Ansible because I'm also looking for a better job and learning how to use a supposedly declarative configuration management tool should, hopefully, be helpful.

Chezmoi

Using JQ and JMESPath

Before I started learning how to write playbooks, roles, and collections in Ansible, I was aware (to an extent) how to use jq to parse and filter JSON data. I realized I shouldn't use jq to process JSON data in Ansible because Ansible can use JMESPath, a query language for JSON written in Python. As part of writing a playbook, one of the tasks involved parsing a JSON string output from the lsblk command which also lists the PARTUUID and finding out the name of the partition with a specific PARTUUID, like the EFI partition, which should have the following PARTUUID:

c12a7328-f81f-11d2-ba4b-00a0c93ec93b

jaq

JMESPath

Let's assume that a variable called "lspart" has the JSON string output of lsblk with PARTUUID of a partition. We want to assign the name of that partition to a variable called "efi_part" if the PARTUUID of that partition is equal to the GPT UUID of an EFI partition. Here's a sample JSON file that we want to process.

the JSON output of lsblk with PARTUUID

When using jq (or jaq), the following expression works

.blockdevices[].children[] | select(.parttype == "c12a7328-f81f-11d2-ba4b-00a0c93ec93b").name

We can use "community.general.json_query" filter, which uses JMESPath, to get the same result.

# the value of the variable gpt.luks_uuid is equal to
# the GPT UUID of an EFI partition
- name: get the name of the EFI partition
  ansible.builtin.set_fact:
    efipart: >-
      {{ (lspart.stdout | from_json) |
      community.general.json_query('
      blockdevices[].children[?parttype ==
      '{{ gpt.luks_uuid }}'].name[]') }}

Using Ansible Builtin Filters

Although the JMESPath expression mentioned before before did work, I couldn't help but feel that parsing JSON in Ansible shouldn't be reliant on a module that's not in the ansible.builtin collection namespace. I'm not sure if it's Google Search getting worse or if I was lazy but I couldn't find any obvious answers to do what I wanted. The Ansible documentation didn't really help either because it also points to examples of JSON data being filtered using JMESPath. I asked for help on the #ansible IRC channel and someone (mackerman, if I recall correctly) pointed me to "from_json", "map", "selectattr", and "flatten" to do what I wanted.

We can write a mini playbook of sorts to process the "lspart.json" file linked above.

---
- name: showcase processing JSON
  hosts: localhost
  gather_facts: false

  tasks:
    - name: get the list of partitions
      ansible.builtin.command:
        cmd: >
          lsblk -J /dev/{{ disk.devices.root }}
          -o +PARTTYPE,UUID
      register: lspart
      changed_when: false
      vars:
        disk:
          devices:
            root: "nvme0n1"

    - name: get output
      ansible.builtin.debug:
        msg: >-
          {{ lspart.stdout | from_json }}
      vars:
        gpt:
          efi_uuid: "c12a7328-f81f-11d2-ba4b-00a0c93ec93b"

The second task that uses the "ansible.builtin.debug" module can be changed incremently and the "ansible-playbook" command can be used to view our progress as we create an expression to get the value we need. For starters, we convert the JSON string data from "lspart.stdout" to structured data that Ansible can work with. Ansible has the ability to use object identifiers, with some limitations, like jq does. We can append ".blockdevices" to the paranthesized expression mentioned above to get the array assigned to the "blockdevice" key. The expression now looks like:

{{ (lspart.stdout | from_json).blockdevices }}

It seems that Ansible doesn't have the ".[]" filter that jq and JMESPath have. We do, however, have the "map" filter from Jinja2, which can transform JSON based on the value of an attribute or a condition. We can select the "children" array inside the array we have right now.

{{ (lspart.stdout | from_json).blockdevices |
map(attribute='children') }}

This leaves us with a list within a list. We can reduce nested lists using the "ansible.builtin.flatten" filter.

{{ (lspart.stdout | from_json).blockdevices |
map(attribute='children') | ansible.builtin.flatten }}

We can now filter the objects inside the list to get the object(s) which have the value of "parttype" equal to a specific value using the "selectattr" filter from Jinja2.

{{ (lspart.stdout | from_json).blockdevices |
map(attribute='children') | ansible.builtin.flatten |
selectattr('parttype', 'eq', gpt.efi_uuid) }}

This gives us the object in which the PARTUUID we're looking for is located. Since the result is still inside a list, we can wrap the entire expression using parantheses, select the first and the only element at the index 0 and then the "name" of the partition within that element.

{{ ((lspart.stdout | from_json).blockdevices |
map(attribute='children') | ansible.builtin.flatten |
selectattr('parttype', 'eq', gpt.efi_uuid)).0.name }}

Of course, this assumes that the output of the "selectattr" filter will be a single object. If we do have two EFI partitions on a system, this expression will omit the names of any additional objects. If we do want the names of any additional EFI partitions on the system, we can use "map" to filter the objects with the key-value pairs of the key "name".

The final task playbook like as follows:

---
- name: showcase processing JSON
  hosts: localhost
  gather_facts: no

  tasks:
    - name: get the list of partitions
      ansible.builtin.command:
        cmd: >
          lsblk -J /dev/{{ disk.devices.root }}
          -o +PARTTYPE,UUID
      register: lspart
      changed_when: false
      vars:
        disk:
          devices:
            root: nvme0n1

    - name: get output
      ansible.builtin.debug:
        msg: >-
          {{ ((lspart.stdout | from_json).blockdevices |
          map(attribute='children') | ansible.builtin.flatten |
          selectattr('parttype', 'eq', gpt.efi_uuid)).0.name }}
      vars:
        gpt:
	  efi_uuid: "c12a7328-f81f-11d2-ba4b-00a0c93ec93b"

Although this might seem a bit more verbose than the JMESPath solution, we're no longer dependent on a community module and installation of a python package that isn't a dependency of Ansible itself.

---

Updated: 2023-08-07

Created: 2023-08-06

Home

-- Response ended

-- Page fetched on Thu May 9 23:10:15 2024