OpenSearch and Dashboards
The SIEM's storage and search, with its own users, certificates and retention.
Why SIEM, on the tour
On this page 5 sections
OpenSearch stores and searches the logs, on the NVMe drive, with TLS everywhere and its own users. OpenSearch Dashboards is the web interface. Both are Helm charts, applied by Flux.
1. Storage on the NVMe drive#
A local volume, tied to the worker with the drive, and claimed by OpenSearch alone.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-nvme
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: opensearch-data-worker2
spec:
capacity:
storage: 800Gi
accessModes: ["ReadWriteOnce"]
persistentVolumeReclaimPolicy: Retain
storageClassName: local-nvme
claimRef:
namespace: logging
name: opensearch-cluster-master-opensearch-cluster-master-0
local:
path: /mnt/k8s-data/opensearch
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values: ["k8s-worker2"]apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- metallb-pool.yaml
- homelab-ca.sops.yaml
- cluster-issuer.yaml
- gateway.yaml
- storage.yaml2. The logging namespace and certificates#
apiVersion: v1
kind: Namespace
metadata:
name: loggingShow the whole file, 47 lines
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: opensearch-node
namespace: logging
spec:
secretName: opensearch-node-tls
commonName: opensearch-cluster-master.logging.svc
dnsNames:
- opensearch-cluster-master.logging.svc
- opensearch-cluster-master.logging.svc.cluster.local
- opensearch-cluster-master-headless.logging.svc
- opensearch-cluster-master-0.opensearch-cluster-master-headless.logging.svc
privateKey:
algorithm: RSA
size: 2048
encoding: PKCS8
rotationPolicy: Always
usages:
- server auth
- client auth
- digital signature
- key encipherment
issuerRef:
kind: ClusterIssuer
name: homelab-ca
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: opensearch-admin
namespace: logging
spec:
secretName: opensearch-admin-tls
commonName: admin
privateKey:
algorithm: RSA
size: 2048
encoding: PKCS8
rotationPolicy: Always
usages:
- client auth
- digital signature
- key encipherment
issuerRef:
kind: ClusterIssuer
name: homelab-ca- One certificate for the node, one for the admin client. The admin certificate is how you talk to OpenSearch's API without a password.
3. Users and passwords#
Four accounts: admin for you, kibanaserver for Dashboards, and two that may only write: vector for the firewall's logs, and falco for Falco's events in chapter 8. This script makes the passwords, hashes them for OpenSearch, and writes the secrets encrypted with SOPS. No password touches the disk. It is safe to run again: it keeps every password and hash it finds, makes only the missing ones, and rewrites a file only when its content changes.
Show the whole file, 180 lines
#!/bin/sh
# Creates or extends OpenSearch's users and passwords, encrypted with SOPS straight away:
# no password and no plaintext file ever lands on disk.
# apps/logging/opensearch-security.sops.yaml users, roles and role mappings
# apps/logging/opensearch-dashboards-account.sops.yaml the account Dashboards logs in with
# apps/logging/vector-opensearch.sops.yaml the write-only account Vector uses
# apps/falco/falco-opensearch.sops.yaml the write-only account Falcosidekick uses
# Safe to run again: stored passwords and hashes are kept, only missing ones are generated,
# and a file is rewritten only when its decrypted content changes.
# It prints the admin password only when it creates one: put it in your password manager.
set -eu
cd "$(dirname "$0")/.."
SEC=apps/logging/opensearch-security.sops.yaml
DASH=apps/logging/opensearch-dashboards-account.sops.yaml
VEC=apps/logging/vector-opensearch.sops.yaml
FALCO=apps/falco/falco-opensearch.sops.yaml
# 1. What already exists. A failed decryption stops the script (set -e): it must never
# fall through to "missing" and silently generate new passwords.
decrypt() { [ -f "$1" ] || return 0; sops -d "$1"; }
SEC_PLAIN=$(decrypt "$SEC")
DASH_PLAIN=$(decrypt "$DASH")
VEC_PLAIN=$(decrypt "$VEC")
FALCO_PLAIN=$(decrypt "$FALCO")
# field PLAINTEXT KEY: one decoded value from a decrypted Secret, empty if absent
field() { printf '%s\n' "$1" | yq ".data.\"$2\" // \"\"" | base64 --decode; }
USERS_PLAIN=$(field "$SEC_PLAIN" internal_users.yml)
# old_hash USER: that user's stored bcrypt hash, empty if absent
old_hash() { printf '%s\n' "$USERS_PLAIN" | yq ".\"$1\".hash // \"\""; }
hash() { htpasswd -bnBC 12 "" "$1" | tr -d ':\n'; }
# 2. Passwords and hashes: keep what is stored, generate what is missing.
# A newly generated password always gets a new hash, so the two never disagree.
NEW_ADMIN_PW=""
ADMIN_HASH=$(old_hash admin)
if [ -z "$ADMIN_HASH" ]; then
NEW_ADMIN_PW=$(openssl rand -base64 24 | tr -d '/+=')
ADMIN_HASH=$(hash "$NEW_ADMIN_PW")
fi
DASH_PW=$(field "$DASH_PLAIN" password)
if [ -z "$DASH_PW" ]; then DASH_PW=$(openssl rand -hex 24); DASH_HASH=$(hash "$DASH_PW")
else DASH_HASH=$(old_hash kibanaserver); [ -n "$DASH_HASH" ] || DASH_HASH=$(hash "$DASH_PW"); fi
VECTOR_PW=$(field "$VEC_PLAIN" password)
if [ -z "$VECTOR_PW" ]; then VECTOR_PW=$(openssl rand -hex 24); VECTOR_HASH=$(hash "$VECTOR_PW")
else VECTOR_HASH=$(old_hash vector); [ -n "$VECTOR_HASH" ] || VECTOR_HASH=$(hash "$VECTOR_PW"); fi
FALCO_PW=$(field "$FALCO_PLAIN" ELASTICSEARCH_PASSWORD)
if [ -z "$FALCO_PW" ]; then FALCO_PW=$(openssl rand -hex 24); FALCO_HASH=$(hash "$FALCO_PW")
else FALCO_HASH=$(old_hash falco); [ -n "$FALCO_HASH" ] || FALCO_HASH=$(hash "$FALCO_PW"); fi
COOKIE=$(field "$DASH_PLAIN" cookie)
[ -n "$COOKIE" ] || COOKIE=$(openssl rand -hex 32)
# 3. OpenSearch's security configuration
INTERNAL_USERS=$(cat <<YAML
_meta:
type: "internalusers"
config_version: 2
admin:
hash: "$ADMIN_HASH"
reserved: true
backend_roles:
- "admin"
description: "Human administrator"
kibanaserver:
hash: "$DASH_HASH"
reserved: true
description: "OpenSearch Dashboards server"
vector:
hash: "$VECTOR_HASH"
reserved: false
description: "Vector log shipper, write only"
falco:
hash: "$FALCO_HASH"
reserved: false
description: "Falcosidekick, write only"
YAML
)
ROLES=$(cat <<'YAML'
_meta:
type: "roles"
config_version: 2
vector_writer:
reserved: false
cluster_permissions:
- "cluster_monitor"
- "indices:data/write/bulk"
index_permissions:
- index_patterns:
- "fw-*"
- "unifi-*"
allowed_actions:
- "create_index"
- "write"
- "indices:admin/mapping/put"
falco_writer:
reserved: false
cluster_permissions:
- "cluster_monitor"
- "indices:data/write/bulk"
index_permissions:
- index_patterns:
- "falco-*"
allowed_actions:
- "create_index"
- "write"
- "indices:admin/mapping/put"
YAML
)
ROLES_MAPPING=$(cat <<'YAML'
_meta:
type: "rolesmapping"
config_version: 2
all_access:
reserved: false
backend_roles:
- "admin"
kibana_server:
reserved: true
users:
- "kibanaserver"
own_index:
reserved: false
users:
- "*"
vector_writer:
reserved: false
users:
- "vector"
falco_writer:
reserved: false
users:
- "falco"
YAML
)
# 4. Into SOPS, but only where something changed. SOPS encrypts with a fresh data key
# every time, so rewriting an unchanged secret would still change the file in Git.
norm() { yq -o=json -I=0 'sort_keys(..)'; }
put() {
if [ -f "$1" ] && [ "$(sops -d "$1" | norm)" = "$(printf '%s\n' "$2" | norm)" ]; then
echo "unchanged $1"; return 0
fi
ENC=$(printf '%s\n' "$2" | sops --encrypt --filename-override "$1" \
--input-type yaml --output-type yaml /dev/stdin)
mkdir -p "$(dirname "$1")"
printf '%s\n' "$ENC" > "$1"
echo "written $1"
}
put "$SEC" "$(kubectl create secret generic opensearch-security -n logging \
--from-literal=internal_users.yml="$INTERNAL_USERS" \
--from-literal=roles.yml="$ROLES" \
--from-literal=roles_mapping.yml="$ROLES_MAPPING" \
--dry-run=client -o yaml)"
put "$DASH" "$(kubectl create secret generic opensearch-dashboards-account -n logging \
--from-literal=username=kibanaserver --from-literal=password="$DASH_PW" \
--from-literal=cookie="$COOKIE" \
--dry-run=client -o yaml)"
put "$VEC" "$(kubectl create secret generic vector-opensearch -n logging \
--from-literal=username=vector --from-literal=password="$VECTOR_PW" \
--dry-run=client -o yaml)"
put "$FALCO" "$(kubectl create secret generic falco-opensearch -n falco \
--from-literal=ELASTICSEARCH_USERNAME=falco --from-literal=ELASTICSEARCH_PASSWORD="$FALCO_PW" \
--dry-run=client -o yaml)"
# 5. The one password you need, only when it was just created
if [ -n "$NEW_ADMIN_PW" ]; then
echo "New OpenSearch admin password: $NEW_ADMIN_PW"
echo "Put it in your password manager now."
ficd ~/homelab && sh scripts/opensearch-secrets.sh
cp infrastructure/configs/homelab-ca.crt apps/logging/homelab-ca.crtCheck
for f in apps/logging/*.sops.yaml; do echo "$f: $(grep -E '^ {4}[a-z_.]+: ENC\[' $f | grep -vc '^ *mac:') encrypted"; done
awk '/^data:/ {d=1; next} /^[^ ]/ {d=0} d && !/ENC\[/' apps/logging/*.sops.yaml | wc -lopensearch-dashboards-account 3, opensearch-security 3, vector-opensearch 2, and 0: every value is encrypted, none is left as plain base64. The script also wrote apps/falco/falco-opensearch.sops.yaml, which waits there until chapter 8. The admin password was printed once: it should be in your password manager now.
4. OpenSearch and Dashboards#
Show the whole file, 82 lines
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
name: opensearch
namespace: logging
spec:
interval: 24h
url: https://opensearch-project.github.io/helm-charts/
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: opensearch
namespace: logging
spec:
interval: 30m
chart:
spec:
chart: opensearch
version: "3.9.0"
sourceRef:
kind: HelmRepository
name: opensearch
values:
singleNode: true
replicas: 1
opensearchJavaOpts: "-Xms3g -Xmx3g"
resources:
requests:
cpu: "1"
memory: 5Gi
limits:
memory: 6Gi
persistence:
enabled: true
storageClass: local-nvme
accessModes: ["ReadWriteOnce"]
size: 800Gi
sysctlInit:
enabled: false
# No demo configuration: our own certificates and users
extraEnvs:
- name: DISABLE_INSTALL_DEMO_CONFIG
value: "true"
secretMounts:
- name: node-tls
secretName: opensearch-node-tls
path: /usr/share/opensearch/config/certs
- name: admin-tls
secretName: opensearch-admin-tls
path: /usr/share/opensearch/config/admin-certs
securityConfig:
enabled: true
internalUsersSecret: opensearch-security
rolesSecret: opensearch-security
rolesMappingSecret: opensearch-security
config:
opensearch.yml: |
cluster.name: opensearch-cluster
network.host: 0.0.0.0
plugins.security.ssl.transport.pemcert_filepath: certs/tls.crt
plugins.security.ssl.transport.pemkey_filepath: certs/tls.key
plugins.security.ssl.transport.pemtrustedcas_filepath: certs/ca.crt
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: certs/tls.crt
plugins.security.ssl.http.pemkey_filepath: certs/tls.key
plugins.security.ssl.http.pemtrustedcas_filepath: certs/ca.crt
plugins.security.allow_default_init_securityindex: true
# One node: a replica can never be placed. Back to 1 if a second data node is added.
cluster.default_number_of_replicas: 0
plugins.index_state_management.history.number_of_replicas: 0
plugins.security.authcz.admin_dn:
- "CN=admin"
plugins.security.nodes_dn:
- "CN=opensearch-cluster-master.logging.svc"
plugins.security.audit.type: internal_opensearch
plugins.security.restapi.roles_enabled:
- "all_access"
- "security_rest_api_access"- One node, 3 GB of heap on an 8 GB board.
DISABLE_INSTALL_DEMO_CONFIG: no demo certificates or demo users, only your own.
Show the whole file, 60 lines
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
name: opensearch-dashboards
namespace: logging
spec:
interval: 30m
dependsOn:
- name: opensearch
chart:
spec:
chart: opensearch-dashboards
version: "3.9.0"
sourceRef:
kind: HelmRepository
name: opensearch
values:
opensearchHosts: "https://opensearch-cluster-master.logging.svc:9200"
opensearchAccount:
secret: opensearch-dashboards-account
resources:
requests:
cpu: 250m
memory: 512Mi
limits:
cpu: "1"
memory: 1Gi
extraVolumes:
- name: homelab-ca
configMap:
name: homelab-ca
extraVolumeMounts:
- name: homelab-ca
mountPath: /usr/share/opensearch-dashboards/config/homelab-ca
readOnly: true
config:
opensearch_dashboards.yml: |
server.host: "0.0.0.0"
opensearch.hosts: ["https://opensearch-cluster-master.logging.svc:9200"]
opensearch.ssl.verificationMode: full
opensearch.ssl.certificateAuthorities: ["/usr/share/opensearch-dashboards/config/homelab-ca/ca.crt"]
opensearch.requestHeadersAllowlist: ["securitytenant", "Authorization"]
opensearch_security.multitenancy.enabled: false
opensearch_security.cookie.secure: true
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: opensearch-dashboards
namespace: logging
spec:
parentRefs:
- name: homelab
namespace: traefik
hostnames:
- dashboards.apps.internal
rules:
- backendRefs:
- name: opensearch-dashboards
port: 5601- Dashboards verifies OpenSearch's certificate against your CA, and is reached at
https://dashboards.apps.internalthrough the gateway.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- opensearch-certs.yaml
- opensearch-security.sops.yaml
- opensearch-dashboards-account.sops.yaml
- vector-opensearch.sops.yaml
- opensearch.yaml
- opensearch-dashboards.yaml
configMapGenerator:
- name: homelab-ca
namespace: logging
files:
- ca.crt=homelab-ca.crt
generatorOptions:
disableNameSuffixHash: trueapiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- logging
- external-certscd ~/homelab && git add infrastructure apps scripts && git commit -m "Logging: OpenSearch and Dashboards" && git push
flux reconcile kustomization flux-system --with-source
kubectl -n logging get pods -wOpenSearch is 1/1 Running after a minute or two, then Dashboards and Vector follow.
The rest of the guide talks to OpenSearch's API through a small helper:
# Talk to OpenSearch's REST API from your laptop, through kubectl.
# Load it into your shell with: . scripts/os.sh
# then, for example: os "$URL/_cluster/health?pretty"
#
# The request runs inside the OpenSearch pod and authenticates with the admin
# client certificate from cert-manager, so no password is typed or stored.
os() {
kubectl -n logging exec -i opensearch-cluster-master-0 -c opensearch -- \
curl -s --cacert config/certs/ca.crt \
--cert config/admin-certs/tls.crt --key config/admin-certs/tls.key \
-H 'Content-Type: application/json' "$@"
}
URL=https://opensearch-cluster-master.logging.svc:9200Check
. scripts/os.sh
os "$URL/_cluster/health?pretty" | grep status
os "$URL/_plugins/_security/authinfo" | grep user_name"status" : "green", and "user_name":"CN=admin": the name in the admin certificate, which OpenSearch treats as its super administrator.
Check
https://dashboards.apps.internal opens on the laptop, and you can log in as admin.
5. How logs are stored#
An index template for the firewall's indices: one shard, no replicas on a single node, strings as keywords, and the right types for addresses and ports. A retention policy compacts each day's index after a day and deletes it after 180.
Show the whole file, 128 lines
{
"index_patterns": [
"fw-*",
"unifi-*"
],
"priority": 100,
"template": {
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0,
"index.codec": "best_compression",
"refresh_interval": "5s",
"index.default_pipeline": "geoip-wan"
},
"mappings": {
"dynamic_templates": [
{
"strings_as_keyword": {
"match_mapping_type": "string",
"mapping": {
"type": "keyword",
"ignore_above": 1024
}
}
}
],
"properties": {
"@timestamp": {
"type": "date"
},
"message": {
"type": "text"
},
"source": {
"properties": {
"ip": {
"type": "ip"
},
"port": {
"type": "integer"
},
"geo": {
"properties": {
"location": {
"type": "geo_point"
}
}
},
"as": {
"properties": {
"number": {
"type": "long"
}
}
}
}
},
"destination": {
"properties": {
"ip": {
"type": "ip"
},
"port": {
"type": "integer"
}
}
},
"network": {
"properties": {
"transport": {
"type": "keyword"
},
"direction": {
"type": "keyword"
}
}
},
"event": {
"properties": {
"action": {
"type": "keyword"
},
"reason": {
"type": "keyword"
},
"dataset": {
"type": "keyword"
}
}
},
"observer": {
"properties": {
"ip": {
"type": "ip"
},
"hostname": {
"type": "keyword"
},
"ingress": {
"properties": {
"interface": {
"properties": {
"name": {
"type": "keyword"
},
"alias": {
"type": "keyword"
}
}
}
}
}
}
},
"rule": {
"properties": {
"id": {
"type": "keyword"
}
}
}
}
},
"aliases": {
"fw-all": {}
}
}
}{
"policy": {
"description": "Logs: compact after 1 day, delete after 180 days",
"default_state": "hot",
"states": [
{ "name": "hot", "actions": [],
"transitions": [ { "state_name": "warm", "conditions": { "min_index_age": "1d" } } ] },
{ "name": "warm",
"actions": [ { "read_only": {} }, { "force_merge": { "max_num_segments": 1 } } ],
"transitions": [ { "state_name": "delete", "conditions": { "min_index_age": "180d" } } ] },
{ "name": "delete", "actions": [ { "delete": {} } ], "transitions": [] }
],
"ism_template": [ { "index_patterns": ["fw-*", "unifi-*"], "priority": 100 } ]
}
}Traffic arriving from the internet gets a country and network owner from GeoIP, through an ingest pipeline. The template names the pipeline, so create it first:
{
"endpoint": "https://geoip.maps.opensearch.org/v1/geolite2-city/manifest.json",
"update_interval_in_days": 3
}{
"endpoint": "https://geoip.maps.opensearch.org/v1/geolite2-asn/manifest.json",
"update_interval_in_days": 3
}{
"description": "GeoIP and ASN for traffic arriving on WAN",
"processors": [
{ "ip2geo": {
"if": "ctx.observer?.ingress?.interface?.alias == 'WAN' && ctx.source?.ip != null",
"field": "source.ip",
"datasource": "geolite2-city",
"target_field": "source.geo",
"properties": ["country_iso_code", "country_name", "continent_name", "region_name", "city_name", "location"],
"ignore_missing": true,
"ignore_failure": true
} },
{ "ip2geo": {
"if": "ctx.observer?.ingress?.interface?.alias == 'WAN' && ctx.source?.ip != null",
"field": "source.ip",
"datasource": "geolite2-asn",
"target_field": "source.as",
"properties": ["asn", "organization_name"],
"ignore_missing": true,
"ignore_failure": true
} },
{ "rename": { "field": "source.as.asn", "target_field": "source.as.number", "ignore_missing": true } },
{ "rename": { "field": "source.as.organization_name", "target_field": "source.as.organization.name", "ignore_missing": true } }
]
}cd ~/homelab/docs/opensearch && . ../../scripts/os.sh
os -X PUT "$URL/_plugins/geospatial/ip2geo/datasource/geolite2-city" -d @- < geoip/datasource-city.json; echo
os -X PUT "$URL/_plugins/geospatial/ip2geo/datasource/geolite2-asn" -d @- < geoip/datasource-asn.json; echo
sleep 60
os -X PUT "$URL/_ingest/pipeline/geoip-wan" -d @- < geoip/pipeline-geoip-wan.json; echo
os -X PUT "$URL/_index_template/logs" -d @- < template-logs.json; echo
os -X PUT "$URL/_plugins/_ism/policies/logs-retention" -d @- < ism-retention.json; echoCheck
os "$URL/_plugins/geospatial/ip2geo/datasource/geolite2-city" | grep -o '"state":"[A-Z]*"'
os "$URL/_index_template/logs" | grep -o '"index_patterns":\[[^]]*\]'"state":"AVAILABLE", and the template covers fw-* and unifi-*. Each command above answered "acknowledged":true; the retention policy answers with the policy itself. The GeoIP download can stay CREATING for a few minutes.