AZiT

The boards: flash and configure

Six Raspberry Pi CM5 boards with the same base, configured by Ansible.

Why Provisioning, on the tour

On this page 9 sections

Five of the six boards become the Kubernetes cluster; the sixth runs the Wi-Fi controller. They all start the same way: Raspberry Pi OS Lite, a fixed address from the firewall, and one Ansible run that turns them into what they need to be.

What you need#

The Super6c carrier board in its box, with one CM5 fitted and the boxes of the other five CM5 boards in front
The carrier board, and the six CM5 boards still in their boxes.
A Raspberry Pi Compute Module 5 seated in a slot of the Super6c, without its heatsink
One CM5 in its slot. The table printed on the board marks its memory and eMMC size.
BoardAddressRole
k8s-cp1, k8s-cp2, k8s-cp3192.168.5.10, .20, .30control planes
k8s-worker1, k8s-worker2192.168.5.40, .50workers; worker2 has the NVMe
unifi192.168.5.60UniFi OS Server, for the access point

1. Tools on the laptop#

~/Brewfile laptopDownload
# Everything the admin laptop needs, in one file: brew bundle --file Brewfile
# Basics
brew "git"
brew "gh"
# Kubernetes
brew "kubernetes-cli"
brew "helm"
tap "fluxcd/tap"
brew "fluxcd/tap/flux"
brew "kubectx"
brew "k9s"
# Automation and secrets
brew "ansible"
brew "sops"
brew "age"
# Utilities
brew "jq"
brew "yq"
brew "openssl@3"
laptop
brew bundle --file ~/Brewfile
gh auth login                                          # GitHub, for the repository and Flux
ls ~/.ssh/id_ed25519.pub || ssh-keygen -t ed25519      # a key for the boards, if you have none

2. Flash each board#

The CM5 boards boot from their own eMMC, which is written from the laptop over USB, one board at a time. rpiboot starts a small helper on the board that makes its eMMC appear on the laptop as an ordinary disk. Build it once:

laptop
brew install libusb pkg-config
cd ~ && git clone --depth 1 https://github.com/raspberrypi/usbboot && cd usbboot && make

Use this fresh build, from its own folder. An older rpiboot installed elsewhere may send a helper too old for the CM5, and then no disk appears.

Super6c: the nRPIBOOT jumper fitted on the pin header next to a board's heatsink
The nRPIBOOT jumper (in the red box), on the top pair of the board's pin header: nRPIBOOT to GND.

Then, for each board:

  1. With the Super6c off, fit the nRPIBOOT jumper for the board's slot, and connect the USB port to the laptop with a cable that carries data, not only power, directly, without a hub.
  2. Start rpiboot, then power the Super6c on:
laptop
cd ~/usbboot && sudo ./rpiboot -d mass-storage-gadget64

Check

laptop
diskutil list external

rpiboot ends with Second stage boot server done, and within a minute a disk of about 31 GB appears. If macOS offers to initialize it, choose Ignore. If rpiboot keeps saying Waiting for BCM…, the laptop does not see the board: try another cable first.

In Raspberry Pi Imager, choose the device Raspberry Pi 5 (the CM5 uses the same image), Raspberry Pi OS Lite (64-bit), the eMMC as the disk, and edit the settings before writing:

SettingValue
Hostnamethe board's name from the table, e.g. k8s-cp1
User@ADMIN_USER@, with a long random password. Keep it in your password manager: you log in with your key, but sudo asks for the password, and so does Ansible (-K).
SSHon, public-key authentication only, with your ~/.ssh/id_ed25519.pub
Wi-Finone: the base role switches Wi-Fi and Bluetooth off
Localeyour time zone and keyboard

On the board with the NVMe drive, Imager lists two disks: the eMMC (about 31 GB) and the NVMe drive. Choose the eMMC.

Raspberry Pi Imager: storage step
Storage: the eMMC, about 31 GB. On the board with the NVMe drive, that drive is offered too: do not pick it.
Raspberry Pi Imager: hostname step
Hostname: the board's name from the table.
Raspberry Pi Imager: localisation step
Localisation: your time zone and keyboard.
Raspberry Pi Imager: user step
User: your admin user, and a long random password.
Raspberry Pi Imager: wi-fi step
Wi-Fi: leave it empty and go on.
Raspberry Pi Imager: remote access step
Remote access: SSH with public-key authentication only.
Raspberry Pi Imager: raspberry pi connect step
Raspberry Pi Connect: off.
Raspberry Pi Imager: summary step
Summary: Raspberry Pi 5, Raspberry Pi OS Lite (64-bit), the eMMC.

When Imager is done: power off, remove the jumper, and move on to the next board. With the jumper still on, the board starts in USB mode again instead of from its eMMC.

3. Give them fixed addresses#

The servers network has no pool for unknown devices. For the first boot, give it a temporary one, read the boards' hardware addresses from the leases, then make them fixed. (Reflashing a board does not change its hardware address: if dhcpd.conf already has the boards, they come up on their fixed addresses straight away, and this step can be skipped.)

In your copy of dhcpd.conf, add this line to the SERVERS subnet, then install it on the firewall as in chapter 4 of the Firewall guide:

from /etc/dhcpd.conf
	range 192.168.5.150 192.168.5.199;      # temporary: remove once the boards have fixed addresses

Power the Super6c on, wait two minutes, and list the leases:

Check

firewall
doas grep -E '^lease|hardware ethernet|client-hostname' /var/db/dhcpd.leases | paste - - - | sort -u

Six leases, each with a hostname you set in the Imager and its hardware address.

The Super6c running with its lid off: a green light next to each of the six heatsinks
All six boards up: each slot has its own green lights.

Put each address in place of its @MAC@ in dhcpd.conf, remove the temporary range line, restart dhcpd, and restart the boards: they come back on their fixed addresses, which unbound already knows by name.

4. SSH from the laptop#

laptop
cat >> ~/.ssh/config <<'EOF'
Host k8s-* unifi
  User @ADMIN_USER@
  IdentityFile ~/.ssh/id_ed25519
  AddKeysToAgent yes
EOF
for h in k8s-cp1 k8s-cp2 k8s-cp3 k8s-worker1 k8s-worker2 unifi; do ssh $h hostname; done

Check

Six hostnames, after you accept each host key once.

5. Prepare the NVMe drive#

Once, on the worker with the drive: one partition, ext4, and its UUID for Ansible. This erases the drive. First check that it is the drive you think, and that nothing has it mounted:

Check

k8s-worker2
lsblk -o NAME,SIZE,TYPE,MOUNTPOINTS /dev/nvme0n1

The drive's size, and no mount point.

-F lets mkfs overwrite a file system already on the drive without stopping to ask.

k8s-worker2
sudo parted -s /dev/nvme0n1 mklabel gpt mkpart k8s-data ext4 0% 100%
sudo mkfs.ext4 -F -L k8s-data /dev/nvme0n1p1
sudo blkid -s UUID -o value /dev/nvme0n1p1
ansible/host_vars/k8s-worker2.yml repositoryDownload
nvme_uuid: @NVME_UUID@
  • Put the UUID from blkid here.

6. The Ansible project#

The repository holds everything for the boards and the cluster. Start it now, as a private repository on GitHub; Flux uses it in chapter 3.

laptop
mkdir -p ~/homelab && cd ~/homelab && git init -b main
ansible/ansible.cfg repositoryDownload
[defaults]
inventory = inventory.ini
remote_user = @ADMIN_USER@
private_key_file = ~/.ssh/id_ed25519
host_key_checking = True
interpreter_python = auto_silent
callback_result_format = yaml
[ssh_connection]
pipelining = True
ansible/inventory.ini repositoryDownload
[control_plane]
k8s-cp1
k8s-cp2
k8s-cp3
[workers]
k8s-worker1
k8s-worker2
[k8s:children]
control_plane
workers
[unifi_host]
unifi
[nvme_storage]
k8s-worker2
  • Names, not addresses: unbound on the firewall resolves them.
ansible/site.yml repositoryDownload
- name: Base configuration for all nodes
  hosts: all
  become: true
  roles:
    - base
- name: Kubernetes prerequisites
  hosts: k8s
  become: true
  roles:
    - k8s_prereqs
    - k8s_packages
- name: NVMe storage
  hosts: nvme_storage
  become: true
  roles:
    - nvme_storage
- name: UniFi OS Server host
  hosts: unifi_host
  become: true
  roles:
    - unifi_host

The roles

base, on every board: all updates and automatic security updates, no Avahi or Bluetooth, the locale, SSH with keys only, time from the firewall, and Wi-Fi and Bluetooth switched off in firmware.

ansible/roles/base/defaults/main.yml repositoryDownload
base_locale: en_GB.UTF-8
ansible/roles/base/tasks/main.yml repositoryDownload
Show the whole file, 88 lines
- name: Upgrade all packages
  ansible.builtin.apt:
    update_cache: true
    cache_valid_time: 3600
    upgrade: dist
- name: Install base packages
  ansible.builtin.apt:
    name:
      - unattended-upgrades
      - locales
    state: present
- name: Remove services we don't need
  ansible.builtin.apt:
    name:
      - avahi-daemon
      - bluez
    state: absent
    purge: true
- name: Turn on automatic security updates
  ansible.builtin.copy:
    dest: /etc/apt/apt.conf.d/20auto-upgrades
    content: |
      APT::Periodic::Update-Package-Lists "1";
      APT::Periodic::Unattended-Upgrade "1";
    mode: "0644"
- name: Generate the system locale
  ansible.builtin.lineinfile:
    path: /etc/locale.gen
    regexp: "^#?\\s*{{ base_locale }} UTF-8"
    line: "{{ base_locale }} UTF-8"
  notify: Regenerate locales
- name: Set the system locale
  ansible.builtin.copy:
    dest: /etc/locale.conf
    content: "LANG={{ base_locale }}\n"
    mode: "0644"
- name: Stop accepting locale variables from SSH clients
  ansible.builtin.lineinfile:
    path: /etc/ssh/sshd_config
    regexp: '^AcceptEnv'
    state: absent
  notify: Restart ssh
- name: SSH hardening
  ansible.builtin.copy:
    dest: /etc/ssh/sshd_config.d/10-hardening.conf
    content: |
      PermitRootLogin no
      AllowUsers @ADMIN_USER@
      AuthenticationMethods publickey
      PasswordAuthentication no
      KbdInteractiveAuthentication no
      X11Forwarding no
      MaxAuthTries 3
      LoginGraceTime 30
    mode: "0644"
  notify: Restart ssh
- name: Create the timesyncd drop-in directory
  ansible.builtin.file:
    path: /etc/systemd/timesyncd.conf.d
    state: directory
    mode: "0755"
- name: Take time from the firewall
  ansible.builtin.copy:
    dest: /etc/systemd/timesyncd.conf.d/firewall.conf
    content: |
      [Time]
      NTP=192.168.5.1
    mode: "0644"
  notify: Restart timesyncd
- name: Disable Wi-Fi and Bluetooth in firmware
  ansible.builtin.blockinfile:
    path: /boot/firmware/config.txt
    marker: "# {mark} ANSIBLE base role"
    block: |
      [all]
      dtoverlay=disable-wifi
      dtoverlay=disable-bt
  notify: Reboot
ansible/roles/base/handlers/main.yml repositoryDownload
- name: Regenerate locales
  ansible.builtin.command: locale-gen
- name: Check the SSH configuration
  ansible.builtin.command: sshd -t
  listen: Restart ssh
- name: Restart the SSH service
  ansible.builtin.service:
    name: ssh
    state: restarted
  listen: Restart ssh
- name: Restart timesyncd
  ansible.builtin.service:
    name: systemd-timesyncd
    state: restarted
- name: Reboot
  ansible.builtin.reboot:
    reboot_timeout: 600

k8s_prereqs, on the cluster's five: no swap, memory cgroups on, the kernel modules and settings Kubernetes networking needs, and the kernel headers, which Falco builds its driver against in chapter 8.

ansible/roles/k8s_prereqs/tasks/main.yml repositoryDownload
Show the whole file, 45 lines
- name: Remove swap and zram packages
  ansible.builtin.apt:
    name:
      - dphys-swapfile
      - rpi-swap
      - systemd-zram-generator
    state: absent
    purge: true
  notify: Reboot
- name: Turn off any active swap now
  ansible.builtin.command: swapoff -a
  when: ansible_facts['swaptotal_mb'] > 0
- name: Enable memory cgroups on the kernel command line
  ansible.builtin.lineinfile:
    path: /boot/firmware/cmdline.txt
    backrefs: true
    regexp: '^((?!.*cgroup_enable=memory).*)$'
    line: '\1 cgroup_enable=memory cgroup_memory=1'
  notify: Reboot
- name: Load kernel modules at boot
  ansible.builtin.copy:
    dest: /etc/modules-load.d/kubernetes.conf
    content: |
      overlay
      br_netfilter
    mode: "0644"
  notify: Reboot
- name: Kernel settings for Kubernetes networking
  ansible.builtin.copy:
    dest: /etc/sysctl.d/99-kubernetes.conf
    content: |
      net.bridge.bridge-nf-call-iptables = 1
      net.bridge.bridge-nf-call-ip6tables = 1
      net.ipv4.ip_forward = 1
    mode: "0644"
  notify: Reboot
- name: Install kernel headers meta-package (Falco builds its kmod driver against them)
  ansible.builtin.apt:
    name: linux-headers-rpi-2712
    state: present
ansible/roles/k8s_prereqs/handlers/main.yml repositoryDownload
- name: Reboot
  ansible.builtin.reboot:
    reboot_timeout: 600

k8s_packages: containerd with systemd cgroups, and kubeadm, kubelet and kubectl from the Kubernetes repository, held at their version.

ansible/roles/k8s_packages/defaults/main.yml repositoryDownload
k8s_minor_version: "1.37"
ansible/roles/k8s_packages/tasks/main.yml repositoryDownload
Show the whole file, 70 lines
- name: Install containerd
  ansible.builtin.apt:
    name: containerd
    state: present
    update_cache: true
- name: Read containerd's default configuration
  ansible.builtin.command: containerd config default
  register: containerd_default
  changed_when: false
  check_mode: false
  failed_when: containerd_default.rc != 0 and not ansible_check_mode
- name: Write containerd configuration with systemd cgroups
  ansible.builtin.copy:
    dest: /etc/containerd/config.toml
    content: "{{ containerd_default.stdout | replace('SystemdCgroup = false', 'SystemdCgroup = true') }}\n"
    mode: "0644"
  when: containerd_default.stdout | length > 0
  notify: Restart containerd
- name: Point crictl at containerd
  ansible.builtin.copy:
    dest: /etc/crictl.yaml
    content: |
      runtime-endpoint: unix:///run/containerd/containerd.sock
      image-endpoint: unix:///run/containerd/containerd.sock
    mode: "0644"
- name: Create the apt keyrings directory
  ansible.builtin.file:
    path: /etc/apt/keyrings
    state: directory
    mode: "0755"
- name: Add the Kubernetes signing key
  ansible.builtin.get_url:
    url: "https://pkgs.k8s.io/core:/stable:/v{{ k8s_minor_version }}/deb/Release.key"
    dest: /etc/apt/keyrings/kubernetes.asc
    mode: "0644"
- name: Add the Kubernetes package repository
  ansible.builtin.copy:
    dest: /etc/apt/sources.list.d/kubernetes.list
    content: "deb [signed-by=/etc/apt/keyrings/kubernetes.asc] https://pkgs.k8s.io/core:/stable:/v{{ k8s_minor_version }}/deb/ /\n"
    mode: "0644"
- name: Install kubeadm, kubelet and kubectl
  ansible.builtin.apt:
    name:
      - kubeadm
      - kubelet
      - kubectl
      - cri-tools
    state: present
    update_cache: true
- name: Hold the Kubernetes packages at their version
  ansible.builtin.dpkg_selections:
    name: "{{ item }}"
    selection: hold
  loop:
    - kubeadm
    - kubelet
    - kubectl
- name: Enable the kubelet
  ansible.builtin.service:
    name: kubelet
    enabled: true
ansible/roles/k8s_packages/handlers/main.yml repositoryDownload
- name: Restart containerd
  ansible.builtin.service:
    name: containerd
    state: restarted

nvme_storage, on the worker with the drive: mounted by UUID, a directory for OpenSearch, and the kernel setting OpenSearch needs.

ansible/roles/nvme_storage/defaults/main.yml repositoryDownload
nvme_mount_path: /mnt/k8s-data
nvme_dirs:
  - name: opensearch
    owner: "1000"
    group: "1000"
    mode: "2770"  # kubelet applies fsGroup 1000 (group rw, setgid) on every OpenSearch start
ansible/roles/nvme_storage/tasks/main.yml repositoryDownload
- name: Mount the NVMe by its UUID
  ansible.posix.mount:
    path: "{{ nvme_mount_path }}"
    src: "UUID={{ nvme_uuid }}"
    fstype: ext4
    opts: defaults,noatime,nofail
    state: mounted
- name: Create the data directories
  ansible.builtin.file:
    path: "{{ nvme_mount_path }}/{{ item.name }}"
    state: directory
    owner: "{{ item.owner }}"
    group: "{{ item.group }}"
    mode: "{{ item.mode }}"
  loop: "{{ nvme_dirs }}"
- name: Kernel setting OpenSearch needs
  ansible.builtin.copy:
    dest: /etc/sysctl.d/99-opensearch.conf
    content: "vm.max_map_count = 262144\n"
    mode: "0644"
  notify: Apply sysctl settings
ansible/roles/nvme_storage/handlers/main.yml repositoryDownload
- name: Apply sysctl settings
  ansible.builtin.command: sysctl --system

unifi_host, on the sixth board: Podman, which UniFi OS Server runs on, and your user in its group once it exists. Until chapter 1 of the Access point guide installs the server, that last task shows as skipping: that is expected.

ansible/roles/unifi_host/tasks/main.yml repositoryDownload
- name: Install Podman for UniFi OS Server
  ansible.builtin.apt:
    name:
      - podman
    state: present
    update_cache: true
- name: Check whether UniFi OS Server has created its group
  ansible.builtin.getent:
    database: group
    key: uosserver
    fail_key: false
- name: Let the admin user run uosserver commands
  ansible.builtin.user:
    name: @ADMIN_USER@
    groups: uosserver
    append: true
  when: ansible_facts['getent_group']['uosserver'] is not none

7. Run it#

laptop
ansible-galaxy collection install ansible.posix
cd ~/homelab/ansible
ansible all -m ping
ansible-playbook site.yml --check --diff -K --limit k8s      # what it would change
ansible-playbook site.yml -K --limit k8s

-K asks once for the password you set in the Imager, for sudo. --limit k8s runs it on the cluster boards only; the UniFi board gets its turn in chapter 1 of the Access point guide. In the dry run, the Kubernetes packages fail with No package matching 'kubeadm': their repository is only pretend-added in check mode, just as containerd is only pretend-installed, so its configuration is skipped. That is expected; the real run does both for real. The boards reboot when the firmware and kernel settings change; Ansible waits for them.

Check

laptop
ansible k8s -b -K -m shell -a "kubeadm version -o short; systemctl is-active containerd; swapon --show | wc -l"
ssh k8s-worker2 'findmnt /mnt/k8s-data; cat /proc/sys/vm/max_map_count'

Every cluster board shows the same kubeadm version, active and 0 (no swap). On worker2, the drive is mounted and the setting is 262144.

8. Move your admin user off uid 1000#

The Imager makes your admin user uid 1000. So do many container images: OpenSearch runs as 1000 and owns its data on the NVMe drive. Sharing the number makes your login user the owner of the SIEM's data, and of its processes, on that board: anyone holding your SSH key could change or delete logs, or kill OpenSearch, without sudo. The Imager cannot set another uid, so this playbook moves the user to 2000, before anything else runs as 1000.

ansible/move-admin-uid.yml repositoryDownload
Show the whole file, 134 lines
# Moves the admin user off uid/gid 1000, which many container images use
# (OpenSearch among them). Sharing it would let the admin user read, change
# or delete the SIEM's data, and signal its processes, without sudo.
#
# Run it once, after the first Ansible run and BEFORE Kubernetes: usermod refuses
# while any process runs as the uid, and container processes count. The playbook
# never kills anything but the admin's own login sessions; if something else runs
# as the uid, it stops and lists it.
#
#   ansible-playbook move-admin-uid.yml -K --limit k8s-worker1    # one board first
#   ansible-playbook move-admin-uid.yml -K --limit k8s            # then the rest
#
# usermod refuses to change a user that is logged in, and Ansible logs in as
# that user. So a temporary user does the move, and is removed at the end.
# sshd allows extra AllowUsers lines to add up, so a drop-in lets it in for the move.
- name: 1. A temporary admin, with the same SSH key
  hosts: all
  become: true
  vars:
    admin_user: "{{ lookup('config', 'DEFAULT_REMOTE_USER') }}"
  tasks:
    - name: Temporary user
      ansible.builtin.user:
        name: uidmove
        shell: /bin/bash
    - name: Its .ssh folder
      ansible.builtin.file:
        path: /home/uidmove/.ssh
        state: directory
        owner: uidmove
        group: uidmove
        mode: "0700"
    - name: Its SSH key is the admin's
      ansible.builtin.copy:
        src: "/home/{{ admin_user }}/.ssh/authorized_keys"
        dest: /home/uidmove/.ssh/authorized_keys
        remote_src: true
        owner: uidmove
        group: uidmove
        mode: "0600"
    - name: Sudo without a password, for this user only, for the next few minutes
      ansible.builtin.copy:
        dest: /etc/sudoers.d/uidmove
        content: "uidmove ALL=(ALL) NOPASSWD: ALL\n"
        mode: "0440"
        validate: visudo -cf %s
    - name: Let it log in over SSH (the base role allows only the admin)
      ansible.builtin.copy:
        dest: /etc/ssh/sshd_config.d/05-uidmove.conf
        content: "AllowUsers uidmove\n"
        mode: "0644"
    - name: Check and reload sshd (open sessions stay up)
      ansible.builtin.shell: sshd -t && systemctl reload ssh
      changed_when: true
    - name: Close Ansible's kept-open connection as the admin
      ansible.builtin.meta: reset_connection
- name: 2. Move the admin user, logged in as the temporary one
  hosts: all
  become: true
  remote_user: uidmove
  vars:
    admin_user: "{{ lookup('config', 'DEFAULT_REMOTE_USER') }}"
    new_id: 2000
  tasks:
    - name: End the admin's login sessions (nothing else is killed)
      ansible.builtin.command: loginctl terminate-user {{ admin_user }}
      register: term
      changed_when: term.rc == 0
      failed_when: false
    - name: Wait until no process runs as the admin
      ansible.builtin.command: pgrep -u {{ admin_user }}
      register: left
      until: left.rc == 1
      retries: 15
      delay: 2
      changed_when: false
      failed_when: false
    - name: Stop if something other than a login still runs as the admin's uid
      ansible.builtin.shell: ps -o pid,user,args -u {{ admin_user }}
      register: who
      changed_when: false
      failed_when: true
      when: left.rc != 1
    - name: New gid
      ansible.builtin.group:
        name: "{{ admin_user }}"
        gid: "{{ new_id }}"
    - name: New uid (usermod re-owns the home folder)
      ansible.builtin.user:
        name: "{{ admin_user }}"
        uid: "{{ new_id }}"
        group: "{{ admin_user }}"
    - name: Files in the home folder still in the old group
      ansible.builtin.command: find /home/{{ admin_user }} -xdev -gid 1000 -exec chgrp -h {{ admin_user }} {} +
      changed_when: false
    - name: Close Ansible's kept-open connection as the temporary user
      ansible.builtin.meta: reset_connection
- name: 3. Back as the admin, remove the temporary user
  hosts: all
  become: true
  tasks:
    - name: No more passwordless sudo, and no SSH for the temporary user
      ansible.builtin.file:
        path: "{{ item }}"
        state: absent
      loop:
        - /etc/sudoers.d/uidmove
        - /etc/ssh/sshd_config.d/05-uidmove.conf
    - name: Check and reload sshd
      ansible.builtin.shell: sshd -t && systemctl reload ssh
      changed_when: true
    - name: Wait until no process runs as the temporary user
      ansible.builtin.command: pgrep -u uidmove
      register: left
      until: left.rc == 1
      retries: 15
      delay: 2
      changed_when: false
      failed_when: left.rc != 1
    - name: Remove the temporary user and its home
      ansible.builtin.user:
        name: uidmove
        state: absent
        remove: true
    - name: Show the result
      ansible.builtin.command: id
      become: false
      register: who
      changed_when: false
    - ansible.builtin.debug:
        msg: "{{ who.stdout }}"
  • usermod refuses to change a user who is logged in, and Ansible logs in as that user. So a temporary user does the move, and is removed at the end.
  • The base role allows SSH logins for your admin user only; AllowUsers lines add up, so a drop-in lets the temporary user in for the move.
  • Ansible keeps its SSH connection open for a minute; reset_connection closes it, so no process of the user being changed is left.
  • It kills nothing but login sessions. If something else runs as the uid, it stops and lists it: run it before Kubernetes.
laptop
cd ~/homelab/ansible
ansible-playbook move-admin-uid.yml -K --limit k8s-worker1     # one board first
ansible-playbook move-admin-uid.yml -K --limit k8s

Check

laptop
ansible k8s -m command -a id
ssh k8s-worker2 'ls -ldn /mnt/k8s-data/opensearch'

Every board: uid=2000 and gid=2000, still in sudo. On worker2, the folder is still owned by 1000 1000, now a number with no user behind it.

laptop
cd ~/homelab && git add ansible && git commit -m "Ansible: base, Kubernetes and storage roles"
gh repo create homelab --private --source=. --push