Falco is highlighted; its page is below.

Falco

The firewall sees what crosses the network; Falco sees what happens inside the boards. It watches every system call on all five cluster nodes, matches them against rules, and sends what it finds to OpenSearch.

Runs as: A DaemonSet on all five boards, control planes included
Driver: The kernel module, built on each board with DKMS
Events: Falcosidekick to OpenSearch, one index per day, TLS verified against the home CA
Account: Its own write-only user, for falco-* indices and nothing else
Alert: Priority Error and above, with a saved search

What it watches for#

A shell in a container, a sensitive file read, a program run that was not in the image. Falco ranks every rule from Debug to Emergency: the common ones, Notice and Warning, are kept and searchable, and only Error and above raise an alert.

One exception is configured: kube-vip announces the API's address with ARP, which needs a packet socket. The exception matches its namespace, image and binary together, so any other packet socket still raises an event.

Why the driver is built on the boards#

Raspberry Pi kernels are built without BTF, so Falco's modern eBPF driver cannot run, and the loader image that normally compiles the kernel module has compilers too old for this kernel. So each board builds the module itself, with DKMS and its own compiler, and rebuilds it whenever a new kernel is installed. Falco's pod only loads what is already there.

Tested on purpose#

A throwaway pod opened a shell, read /etc/shadow, and ran a copy of a program from its writable layer. Falco reported all three, and the last one, which it ranks Critical, raised the alert.

In the configuration#

apps/falco/falco.yaml (Helm values, excerpt)

driver:
  kind: kmod        # no BTF on rpi-2712 kernels, so modern eBPF is not possible
  loader:
    enabled: false  # the module is built on each node with DKMS
falcosidekick:
  enabled: true
  config:
    existingSecret: falco-opensearch
    tlsclient:
      cacertfile: /etc/homelab-ca/ca.crt
    elasticsearch:
      hostport: https://opensearch-cluster-master.logging.svc:9200
      index: falco
      suffix: daily
      flattenfields: true
      checkcert: true

flattenfields turns Falco's dotted field names into plain ones, which OpenSearch would otherwise read as conflicting nested objects.