AZiT

Falco: what happens inside the boards

Runtime detection on every node, its events in the SIEM, and an alert for the serious ones.

Why Falco, on the tour

On this page 6 sections

The firewall sees what crosses the network; Falco sees what happens inside the boards. It watches every system call on every node through a kernel module, matches them against rules (a shell in a container, a sensitive file read, a program that was not in the image being run), and sends what it finds to OpenSearch through Falcosidekick. An alert fires for the serious ones.

Two facts about these boards shape the setup. Raspberry Pi kernels are built without BTF type information, so Falco's modern eBPF driver cannot run: it has to be the kernel module. And they are built with GCC 14, while the driver loader image that normally compiles the module inside Falco's pod has GCC 11 and 12, which cannot build for this kernel. So each board builds the module itself, with DKMS and its own compiler, and Falco's pod only uses it.

1. The kernel module, built on each board#

The kernel headers were installed in chapter 1. A new role builds Falco's driver against them: it takes only the driver's source from Falco's release, checked against a pinned checksum, and hands it to DKMS. DKMS also rebuilds it whenever apt installs a new kernel, with the board's own compiler, so a kernel update does not break Falco.

ansible/roles/falco_driver/defaults/main.yml repositoryDownload
---
# Must match the Falco release deployed by Flux (apps/falco). The driver version is the
# directory name under usr/src/ in the Falco tarball. Bump all three together.
falco_version: "0.45.1"
falco_driver_version: "11.0.1+driver"
falco_tarball_sha256: "fb2dbb07e1bca990a06432239c2294638b01be40ce2bf5c7a013418cdc1036b5"
falco_tarball_url: "https://download.falco.org/packages/bin/aarch64/falco-{{ falco_version }}-aarch64.tar.gz"
falco_tarball_path: "/var/cache/falco-{{ falco_version }}-aarch64.tar.gz"
  • The three values belong together and to the Falco release in the chart below. The driver version is the directory name under usr/src/ in Falco's tarball: tar -tzf falco-<version>-aarch64.tar.gz | grep dkms.conf.
  • Upgrading Falco is two changes: this file first, then the chart's version.
ansible/roles/falco_driver/tasks/main.yml repositoryDownload
Show the whole file, 46 lines
---
- name: Install DKMS and the compiler (no recommends, they would pull generic Debian headers)
  ansible.builtin.apt:
    name: [dkms, gcc, make]
    state: present
    install_recommends: false
- name: Download the Falco tarball (only its driver source is used)
  ansible.builtin.get_url:
    url: "{{ falco_tarball_url }}"
    dest: "{{ falco_tarball_path }}"
    checksum: "sha256:{{ falco_tarball_sha256 }}"
    mode: "0644"
- name: Extract the Falco driver source into /usr/src
  ansible.builtin.unarchive:
    src: "{{ falco_tarball_path }}"
    remote_src: true
    dest: /usr/src
    include: ["falco-{{ falco_version }}-aarch64/usr/src/falco-{{ falco_driver_version }}"]
    extra_opts: ["--strip-components=3"]
    creates: "/usr/src/falco-{{ falco_driver_version }}/dkms.conf"
- name: Check the Falco driver's DKMS status for the running kernel
  ansible.builtin.command: dkms status -m falco -v {{ falco_driver_version }} -k {{ ansible_facts['kernel'] }}
  register: falco_dkms
  changed_when: false
- name: Register the Falco driver source with DKMS
  ansible.builtin.command: dkms add -m falco -v {{ falco_driver_version }}
  when: falco_dkms.stdout == ""
- name: Build and install the Falco driver for the running kernel
  ansible.builtin.command: dkms install -m falco -v {{ falco_driver_version }} -k {{ ansible_facts['kernel'] }}
  when: "'installed' not in falco_dkms.stdout"
- name: Load the Falco module at boot
  ansible.builtin.copy:
    dest: /etc/modules-load.d/falco.conf
    content: "falco\n"
    mode: "0644"
- name: Load the Falco module now
  ansible.builtin.command: modprobe falco
  args:
    creates: /sys/module/falco
  • No recommended packages: Debian's dkms recommends a generic kernel headers package, which is not the Raspberry Pi kernel's.
  • Every task is safe to run again. The build runs only when dkms status says the module is not installed for the running kernel; modprobe only when it is not loaded.
ansible/site.yml repositoryDownload
- name: Base configuration for all nodes
  hosts: all
  become: true
  roles:
    - base
- name: Kubernetes prerequisites
  hosts: k8s
  become: true
  roles:
    - k8s_prereqs
    - k8s_packages
    - falco_driver
- name: NVMe storage
  hosts: nvme_storage
  become: true
  roles:
    - nvme_storage
- name: UniFi OS Server host
  hosts: unifi_host
  become: true
  roles:
    - unifi_host
laptop
cd ~/homelab/ansible
ansible-playbook site.yml --limit k8s -K -f 1

-f 1 works on one board at a time, so only one board compiles at once and the three control planes, which share etcd, are never all busy together. The build takes a few minutes per board.

Check

laptop
ansible k8s -m shell -a 'echo "$(lsmod | grep -c "^falco ") $(ls /dev/falco* | wc -l)/$(nproc)"'

1 4/4 on every board: the module is loaded, with one device per CPU. A second run of the playbook changes nothing.

Why not the driver loader with a writable mount: the loader's build writes into the headers' output tree, and with that tree mounted from the host it writes paths that only exist inside the pod. The host's own headers are then broken for every later build. If it ever happened, dpkg -V linux-headers-$(uname -r) lists the damaged files, and apt-get install --reinstall of that package repairs them.

2. Falco's account in OpenSearch#

The falco user and its role were made in chapter 5, with the others: it may create and write falco-* indices, and nothing else. Its secret, apps/falco/falco-opensearch.sops.yaml, carries the two environment variables Falcosidekick reads its login from.

Check

laptop
cd ~/homelab && . scripts/os.sh
os "$URL/_plugins/_security/api/rolesmapping/falco_writer" | grep -o '"users":\[[^]]*\]'
sops -d apps/falco/falco-opensearch.sops.yaml | yq '.data | keys'

"users":["falco"], and the keys ELASTICSEARCH_PASSWORD and ELASTICSEARCH_USERNAME.

If OpenSearch was set up before this account existed, the role mapping is not found. Run the script again (it keeps every existing password), commit and push, then load the new users into OpenSearch. OpenSearch reads its security files into an index only on first start; after that, securityadmin.sh loads them. The files are mounted with subPath, so the pod has to restart first to see the new ones:

laptop
cd ~/homelab && sh scripts/opensearch-secrets.sh
git add apps scripts && git commit -m "OpenSearch: write-only account for Falco" && git push
flux reconcile kustomization flux-system --with-source
kubectl -n logging delete pod opensearch-cluster-master-0
sleep 10; kubectl -n logging wait --for=condition=Ready pod/opensearch-cluster-master-0 --timeout=10m
kubectl -n logging exec opensearch-cluster-master-0 -c opensearch -- sh -c '
  A="-h opensearch-cluster-master.logging.svc -icl -cacert config/certs/ca.crt -cert config/admin-certs/tls.crt -key config/admin-certs/tls.key"
  for t in internalusers:internal_users roles:roles rolesmapping:roles_mapping; do
    plugins/opensearch-security/tools/securityadmin.sh $A -t ${t%%:*} -f config/opensearch-security/${t#*:}.yml
  done' 2>&1 | grep -E 'SUCC|FAIL|ERR'

One type per call, with -t: -cd with the whole directory would also reset the security plugin's other settings to the image's defaults.

3. How Falco's events are stored#

A template of their own, not the firewall's: no GeoIP pipeline, one shard and no replicas, and the fields you filter on as keywords. The retention policy is the same as the firewall's.

docs/opensearch/template-falco.json repositoryDownload
{
  "index_patterns": ["falco-*"],
  "priority": 100,
  "template": {
    "settings": {
      "index": {
        "number_of_shards": 1,
        "number_of_replicas": 0,
        "codec": "best_compression",
        "refresh_interval": "5s"
      }
    },
    "mappings": {
      "properties": {
        "time":     { "type": "date" },
        "rule":     { "type": "keyword" },
        "priority": { "type": "keyword" },
        "source":   { "type": "keyword" },
        "hostname": { "type": "keyword" },
        "tags":     { "type": "keyword" },
        "output":   { "type": "text" }
      }
    }
  }
}
  • Everything under output_fields (process, container, pod, file) is mapped when it first appears, as text with a keyword subfield.
  • No replicas: on a single node a replica can never be placed, and the index would stay yellow.
docs/opensearch/ism-retention.json repositoryDownload
{
  "policy": {
    "description": "Logs: compact after 1 day, delete after 180 days",
    "default_state": "hot",
    "states": [
      { "name": "hot", "actions": [],
        "transitions": [ { "state_name": "warm", "conditions": { "min_index_age": "1d" } } ] },
      { "name": "warm",
        "actions": [ { "read_only": {} }, { "force_merge": { "max_num_segments": 1 } } ],
        "transitions": [ { "state_name": "delete", "conditions": { "min_index_age": "180d" } } ] },
      { "name": "delete", "actions": [ { "delete": {} } ], "transitions": [] }
    ],
    "ism_template": [ { "index_patterns": ["fw-*", "unifi-*", "falco-*"], "priority": 100 } ]
  }
}
laptop
cd ~/homelab/docs/opensearch && . ../../scripts/os.sh
os -X PUT "$URL/_index_template/falco" -d @- < template-falco.json; echo
S=$(os "$URL/_plugins/_ism/policies/logs-retention" | jq -r '"if_seq_no=\(._seq_no)&if_primary_term=\(._primary_term)"')
os -X PUT "$URL/_plugins/_ism/policies/logs-retention?$S" -d @- < ism-retention.json | jq -c '.policy.policy.ism_template[0].index_patterns'

Changing a policy needs the sequence number and primary term of the version you read: if it changed in between, OpenSearch refuses rather than overwriting.

Check

laptop
os -X POST "$URL/_index_template/_simulate_index/falco-check?filter_path=template.settings.index.number_of_replicas,overlapping"; echo

"number_of_replicas":"0" and "overlapping":[]: a Falco index would get no replicas and no other template. Nothing is created by this check.

4. Falco and Falcosidekick#

apps/falco/namespace.yaml repositoryDownload
apiVersion: v1
kind: Namespace
metadata:
  name: falco
  labels:
    # Falco with the kmod driver must run privileged (it loads a kernel module).
    pod-security.kubernetes.io/enforce: privileged
  • Falco runs privileged: it reads every system call on the node. The label makes that an explicit, visible exception if you ever set a stricter cluster-wide default.
apps/falco/falco.yaml repositoryDownload
Show the whole file, 71 lines
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: falcosecurity
  namespace: falco
spec:
  interval: 24h
  url: https://falcosecurity.github.io/charts
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: falco
  namespace: falco
spec:
  interval: 30m
  install:
    remediation:
      retries: 3
  upgrade:
    remediation:
      retries: 3
  chart:
    spec:
      chart: falco
      version: "9.2.1"
      sourceRef:
        kind: HelmRepository
        name: falcosecurity
  values:
    driver:
      kind: kmod        # no BTF on rpi-2712 kernels, so modern eBPF is not possible
      loader:
        # The module is built on each node by Ansible (role falco_driver) with DKMS.
        # The loader image's GCC 11/12 cannot build for this GCC 14 kernel.
        enabled: false
    falcosidekick:
      enabled: true
      config:
        existingSecret: falco-opensearch      # ELASTICSEARCH_USERNAME / ELASTICSEARCH_PASSWORD
        tlsclient:
          cacertfile: /etc/homelab-ca/ca.crt  # verify OpenSearch's certificate against the homelab CA
        elasticsearch:
          hostport: https://opensearch-cluster-master.logging.svc:9200
          index: falco
          suffix: daily
          flattenfields: true                 # Falco field names are dotted; flattening avoids mapping conflicts
          checkcert: true
      extraVolumes:
        - name: homelab-ca
          configMap:
            name: homelab-ca
      extraVolumeMounts:
        - name: homelab-ca
          mountPath: /etc/homelab-ca
          readOnly: true
    customRules:
      homelab-exceptions.yaml: |-
        # kube-vip announces the API VIP with gratuitous ARP, which needs a packet socket (AF_PACKET).
        # Scoped to namespace + image + binary, so any other packet socket still alerts.
        - rule: Packet socket created in container
          exceptions:
            - name: kube_vip_arp
              fields: [k8s.ns.name, container.image.repository, proc.exepath]
              comps: [=, =, =]
              values:
                - [kube-system, ghcr.io/kube-vip/kube-vip, /kube-vip]
          override:
            exceptions: append
  • driver.kind: kmod, with the loader switched off: the module is already loaded by DKMS.
  • Falcosidekick is part of the chart. Falco sends it every event over HTTP, and it writes them to OpenSearch in batches, to one index per day. Its login comes from the secret; the chart's own secret leaves it empty, and the later source wins.
  • flattenfields: Falco's field names have dots (container.image and container.image.repository). OpenSearch would read those as nested objects that conflict, and reject the event. Flattened, they become container_image_repository.
  • The CA is mounted from a ConfigMap and added to Falcosidekick's trust, so OpenSearch's certificate and name are checked.
  • One exception: kube-vip announces the API's address with ARP, which needs a packet socket, and Falco reports that every few seconds. The exception matches namespace, image and binary together, so any other packet socket still raises an event. Check the image name against yours.
  • Retries: a failed install or upgrade is tried again, after rolling back to the last good release, instead of waiting for someone to notice.
apps/falco/kustomization.yaml repositoryDownload
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
  - falco-opensearch.sops.yaml
  - falco.yaml
configMapGenerator:
  - name: homelab-ca
    namespace: falco
    files:
      - ca.crt=homelab-ca.crt
generatorOptions:
  disableNameSuffixHash: true
apps/kustomization.yaml repositoryDownload
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - logging
  - falco
  - external-certs
laptop
cd ~/homelab && cp apps/logging/homelab-ca.crt apps/falco/homelab-ca.crt
git add apps && git commit -m "Falco with the kernel module from the host, Falcosidekick to OpenSearch" && git push
flux reconcile kustomization flux-system --with-source
kubectl -n falco wait --for=condition=Ready helmrelease/falco --timeout=10m
kubectl -n falco rollout status ds/falco
kubectl -n falco get pods -o wide

The control planes are tainted, but the chart tolerates that: Falco runs on all five boards.

Check

laptop
kubectl -n falco get pods --no-headers | awk '{print $2, $3}' | sort | uniq -c
kubectl -n falco logs ds/falco -c falco | grep "Opening 'syscall'"

5 2/2 Running for Falco and 2 1/1 Running for Falcosidekick, and Opening 'syscall' source with Kernel module.

A copy of the CA certificate is needed here because kustomize only reads files in its own directory: renewing the CA means updating both copies. The step from Falco to Falcosidekick is plain HTTP inside the cluster, and Falcosidekick holds events in memory for up to a second before sending them: it is a forwarder, not a queue.

5. An alert, and a search to see what happened#

Falco ranks every rule from Debug to Emergency. Reads of sensitive files and shells in containers are Notice and Warning: worth keeping and searching, but too common to alert on. The monitor fires for Error and above, the same way as the two in chapter 7.

docs/opensearch/monitors/falco-error.json repositoryDownload
Show the whole file, 67 lines
{
  "type": "monitor",
  "monitor_type": "query_level_monitor",
  "name": "Falco: event of priority Error or higher",
  "enabled": true,
  "schedule": {
    "period": {
      "interval": 5,
      "unit": "MINUTES"
    }
  },
  "inputs": [
    {
      "search": {
        "indices": [
          "falco-*"
        ],
        "query": {
          "size": 10,
          "_source": [
            "@timestamp",
            "message",
            "source.hostname"
          ],
          "query": {
            "bool": {
              "filter": [
                {
                  "terms": {
                    "priority": [
                      "Emergency",
                      "Alert",
                      "Critical",
                      "Error"
                    ]
                  }
                },
                {
                  "range": {
                    "@timestamp": {
                      "gte": "{{period_end}}||-5m",
                      "lte": "{{period_end}}",
                      "format": "epoch_millis"
                    }
                  }
                }
              ]
            }
          }
        }
      }
    }
  ],
  "triggers": [
    {
      "name": "Falco reported an event of priority Error or higher",
      "severity": "1",
      "condition": {
        "script": {
          "source": "ctx.results[0].hits.total.value > 0",
          "lang": "painless"
        }
      },
      "actions": []
    }
  ]
}
laptop
cd ~/homelab/docs/opensearch/monitors && . ../../../scripts/os.sh
os -X POST "$URL/_plugins/_alerting/monitors" -d @- < falco-error.json | grep -oE '"_id":"[^"]+"|"error"'
docs/opensearch/saved-objects-falco.ndjson repositoryDownload
{"attributes":{"displayName":"Falco","fields":"[{\"count\":0,\"name\":\"@timestamp\",\"type\":\"date\",\"esTypes\":[\"date\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"_id\",\"type\":\"string\",\"esTypes\":[\"_id\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"count\":0,\"name\":\"_index\",\"type\":\"string\",\"esTypes\":[\"_index\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"count\":0,\"name\":\"_score\",\"type\":\"number\",\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"_source\",\"type\":\"_source\",\"esTypes\":[\"_source\"],\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"_type\",\"type\":\"string\",\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":1,\"name\":\"hostname\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":1,\"name\":\"output\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"output_fields.container_id\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"output_fields.container_id.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"output_fields.container_id\"}}},{\"count\":0,\"name\":\"output_fields.container_image_repository\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"output_fields.container_image_repository.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"output_fields.container_image_repository\"}}},{\"count\":0,\"name\":\"output_fields.container_image_tag\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"output_fields.container_image_tag.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"output_fields.container_image_tag\"}}},{\"count\":0,\"name\":\"output_fields.container_name\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"output_fields.container_name.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"output_fields.container_name\"}}},{\"count\":0,\"name\":\"output_fields.container_start_ts\",\"type\":\"number\",\"esTypes\":[\"long\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"output_fields.evt_arg_flags\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"output_fields.evt_arg_flags.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"output_fields.evt_arg_flags\"}}},{\"count\":0,\"name\":\"output_fields.evt_args\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"output_fields.evt_args.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"output_fields.evt_args\"}}},{\"count\":0,\"name\":\"output_fields.evt_time\",\"type\":\"number\",\"esTypes\":[\"long\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"output_fields.evt_type\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"output_fields.evt_type.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"output_fields.evt_type\"}}},{\"count\":0,\"name\":\"output_fields.fd_l4proto\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"output_fields.fd_l4proto.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"output_fields.fd_l4proto\"}}},{\"count\":0,\"name\":\"output_fields.fd_name\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"output_fields.fd_name.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"output_fields.fd_name\"}}},{\"count\":0,\"name\":\"output_fields.fd_type\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"output_fields.fd_type.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"output_fields.fd_type\"}}},{\"count\":1,\"name\":\"output_fields.k8s_ns_name\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"output_fields.k8s_ns_name.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"output_fields.k8s_ns_name\"}}},{\"count\":1,\"name\":\"output_fields.k8s_pod_name\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"output_fields.k8s_pod_name.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"output_fields.k8s_pod_name\"}}},{\"count\":0,\"name\":\"output_fields.proc_aname[2]\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"output_fields.proc_aname[2].keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"output_fields.proc_aname[2]\"}}},{\"count\":0,\"name\":\"output_fields.proc_aname[3]\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"output_fields.proc_aname[3].keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"output_fields.proc_aname[3]\"}}},{\"count\":0,\"name\":\"output_fields.proc_cmdline\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"output_fields.proc_cmdline.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"output_fields.proc_cmdline\"}}},{\"count\":0,\"name\":\"output_fields.proc_cwd\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"output_fields.proc_cwd.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"output_fields.proc_cwd\"}}},{\"count\":0,\"name\":\"output_fields.proc_exe\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"output_fields.proc_exe.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"output_fields.proc_exe\"}}},{\"count\":0,\"name\":\"output_fields.proc_exe_ino_ctime\",\"type\":\"number\",\"esTypes\":[\"long\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"output_fields.proc_exe_ino_ctime_duration_proc_start\",\"type\":\"number\",\"esTypes\":[\"long\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"output_fields.proc_exe_ino_mtime\",\"type\":\"number\",\"esTypes\":[\"long\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"output_fields.proc_exepath\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"output_fields.proc_exepath.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"output_fields.proc_exepath\"}}},{\"count\":0,\"name\":\"output_fields.proc_name\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"output_fields.proc_name.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"output_fields.proc_name\"}}},{\"count\":0,\"name\":\"output_fields.proc_pname\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"output_fields.proc_pname.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"output_fields.proc_pname\"}}},{\"count\":0,\"name\":\"output_fields.proc_sname\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"output_fields.proc_sname.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"output_fields.proc_sname\"}}},{\"count\":0,\"name\":\"output_fields.proc_tty\",\"type\":\"number\",\"esTypes\":[\"long\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"output_fields.user_loginuid\",\"type\":\"number\",\"esTypes\":[\"long\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"output_fields.user_name\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"output_fields.user_name.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"output_fields.user_name\"}}},{\"count\":0,\"name\":\"output_fields.user_uid\",\"type\":\"number\",\"esTypes\":[\"long\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":1,\"name\":\"priority\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":1,\"name\":\"rule\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"source\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"tags\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"time\",\"type\":\"date\",\"esTypes\":[\"date\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"count\":0,\"name\":\"uuid\",\"type\":\"string\",\"esTypes\":[\"text\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"count\":0,\"name\":\"uuid.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"subType\":{\"multi\":{\"parent\":\"uuid\"}}}]","timeFieldName":"@timestamp","title":"falco-*"},"id":"139b6930-d071-11f1-a1cf-2fad06feaf1a","migrationVersion":{"index-pattern":"7.6.0"},"references":[],"type":"index-pattern","updated_at":"2026-10-25T12:40:53.929Z","version":"Wzc4LDNd"}
{"attributes":{"columns":["rule","priority","hostname","output_fields.k8s_ns_name","output_fields.k8s_pod_name","output"],"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"priority:(Emergency or Alert or Critical or Error)\",\"language\":\"kuery\"},\"highlightAll\":true,\"version\":true,\"aggs\":{\"2\":{\"date_histogram\":{\"field\":\"@timestamp\",\"fixed_interval\":\"30s\",\"time_zone\":\"Europe/Stockholm\",\"min_doc_count\":1}}},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"sort":[],"title":"Falco: priority Error or higher","version":1},"id":"57fdaac0-d071-11f1-a1cf-2fad06feaf1a","migrationVersion":{"search":"7.9.3"},"references":[{"id":"139b6930-d071-11f1-a1cf-2fad06feaf1a","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"search","updated_at":"2026-10-25T12:41:05.131Z","version":"Wzc5LDNd"}
{"exportedCount":2,"missingRefCount":0,"missingReferences":[]}
  • An index pattern for falco-* with @timestamp as its time, and a saved search with the monitor's filter and the columns that say what happened where: rule, priority, node, namespace, pod, and Falco's own sentence.

Import it as in chapter 7: Dashboards Management, Saved objects, Import, with Automatically overwrite conflicts.

Making the index pattern by hand instead: the field already ends in *, so type falco-. Typing falco-* gives a pattern called falco-**, which works but cannot be renamed.

6. Test it#

A throwaway pod, and three things an intruder in a container might do: open a shell, read the password hashes, and run a program that was not in the image. Monitors run first five minutes after they are created: test after that.

laptop
kubectl run falco-test --image=busybox:1.37 --restart=Never -- sleep 900
kubectl wait --for=condition=Ready pod/falco-test
kubectl exec -it falco-test -- sh
in the pod
cat /etc/shadow > /dev/null
cp /bin/busybox /tmp/busybox && /tmp/busybox echo dropped
exit

The copy lands in the container's writable layer, on top of the image's read-only ones. A program run from there was not in the image: that is what Falco's rule looks for. BusyBox picks its tool from the name it is called by, so the copy keeps the name busybox.

Check

laptop
. scripts/os.sh
os -X POST "$URL/falco-*/_search?size=0&filter_path=aggregations.*.buckets" \
  -d '{"query":{"term":{"output_fields.k8s_pod_name.keyword":"falco-test"}},"aggs":{"rule":{"terms":{"field":"rule"}}}}' \
  | jq -c '.aggregations.rule.buckets | map({(.key): .doc_count}) | add'
kubectl delete pod falco-test

Three rules, one event each: Terminal shell in container (Notice), Read sensitive file untrusted (Warning) and Drop and execute new binary in container (Critical). Within five minutes, the Critical one raises an alert under Alerting, and the saved search shows it. The other two are in the index, but not alerts.

On an idle cluster there may be no Falco events at all, so the falco- index for the day only appears with the first one, as with the firewall's.