AZiT

WireGuard and the kill switch

Every connection through the tunnel, and nothing at all when it is down.

Why VPN and kill switch, on the tour

On this page 8 sections

The WAN port moves into a routing domain of its own. The internal networks and the tunnel stay in the default one, whose only route to the internet is the tunnel. If the tunnel goes down, those networks simply have no path out, and pf blocks and logs anything that tries anyway.

1. A key, made on the firewall#

The private key is generated on the firewall and never leaves it. Only the public key goes to the VPN provider.

firewall
doas sh -c 'umask 077; openssl rand -base64 32 > /etc/wg0.key'
doas ifconfig wg0 create wgkey "$(doas cat /etc/wg0.key)"
doas ifconfig wg0 | grep wgpubkey
doas ifconfig wg0 destroy

Register the public key in your VPN account. In return you get a server (its public key and address), the address of your end of the tunnel, and the DNS server inside the tunnel. Those are the four @VPN_…@ placeholders.

If your provider does not accept a public key, but only offers to generate a configuration file with a private key in it, use that file instead, and skip the commands above. It works the same; the key has simply been seen by someone else. The file has all four values: Address is your tunnel address, DNS the provider's resolver, and under [Peer] are the server's PublicKey and Endpoint. Ignore its PostUp and PreDown lines: they are for Linux, and pf does that job here.

2. The tunnel interface#

/etc/hostname.wg0Download
description "VPN"
wgkey @WG_PRIVATE_KEY@
wgrtable 1
wgpeer @VPN_SERVER_PUBLIC_KEY@ wgendpoint @VPN_SERVER_ADDRESS@ 51820 wgaip 0.0.0.0/0 wgpka 25
inet @VPN_TUNNEL_ADDRESS@/32
!route -q add -inet default -link -iface wg0
  • wgrtable 1 is the trick: the tunnel's encrypted packets travel in the WAN's routing domain, while everything inside the tunnel belongs to the internal one.
  • The last line makes the tunnel the only default route in the internal routing domain.
  • The port, 51820, is the usual one; use your provider's if it differs.

Fill in the three @VPN_…@ values in the file on your laptop, then copy it over and put the key in place of @WG_PRIVATE_KEY@ on the firewall:

laptop
scp hostname.wg0 fw:/tmp/
firewall
doas install -o root -g wheel -m 0600 /tmp/hostname.wg0 /etc/hostname.wg0 && rm /tmp/hostname.wg0
doas sh -c 'sed -i "s|@WG_PRIVATE_KEY@|$(cat /etc/wg0.key)|" /etc/hostname.wg0'
doas rm /etc/wg0.key

With a key from the provider's file instead, fill it in on your laptop, where the file is, and delete the local copy once it is on the firewall:

laptop
KEY=$(awk '/^PrivateKey/ {print $3}' ~/Downloads/provider.conf)
sed "s|@WG_PRIVATE_KEY@|$KEY|" hostname.wg0 > /tmp/hostname.wg0
scp /tmp/hostname.wg0 fw:/tmp/ && rm /tmp/hostname.wg0
firewall
doas install -o root -g wheel -m 0600 /tmp/hostname.wg0 /etc/hostname.wg0 && rm /tmp/hostname.wg0

Check

firewall
doas grep -c '^wgkey ' /etc/hostname.wg0
doas grep -c @ /etc/hostname.wg0

1, then 0: the key is in, and no placeholder is left. Delete the provider's file from your laptop too.

3. Move the WAN into its own routing domain#

/etc/hostname.igc0Download
description "WAN"
rdomain 1
inet autoconf
  • One line more than before: rdomain 1.

4. The policy, with the kill switch#

Two changes from the previous chapter. Traffic is translated to the tunnel's address instead of the WAN's, and on the WAN port only three things may leave: the tunnel itself, DHCP, and nothing else, which is blocked and logged as block_wan_leak.

/etc/pf.confDownload
Show the whole file, 115 lines
# /etc/pf.conf - fw.internal
# Default deny. Blocked traffic is logged, and so is every new connection
# allowed by a pass rule marked "log". Every rule has a label; pflog2syslog
# adds it to each log line as label=<name>, which Vector turns into rule.name.
# --- Interfaces ----
wan = "igc0"
admin = "igc1"
apmgmt = "igc2"
ps5 = "igc3"
media = "igc4"
servers = "igc5"
trusted = "vlan10"
iot = "vlan20"
guest = "vlan30"
vpn = "wg0"
internal = "{ igc1 igc2 igc3 igc4 igc5 vlan10 vlan20 vlan30 }"
# --- Tables ----
table <admin_hosts> const { 192.168.1.10 192.168.10.10 }
table <k8s_nodes> const { 192.168.5.10 192.168.5.20 192.168.5.30 192.168.5.40 192.168.5.50 }
table <k8s_api> const { 192.168.5.100 }
table <k8s_services> const { 192.168.5.200/29 192.168.5.208/29 192.168.5.216/30 192.168.5.220 }
table <unifi_host> const { 192.168.5.60 }
table <unifi_ap> const { 192.168.2.2 }
table <home_assistant> const { 192.168.5.140 }
table <kodi> const { 192.168.4.10 }
table <public_dns> const { 9.9.9.9 149.112.112.112 }
table <vpn_server> const { @VPN_SERVER_ADDRESS@ }
table <internal_nets> const { 192.168.1.0/24 192.168.2.0/24 192.168.3.0/24 192.168.4.0/24 \
    192.168.5.0/24 192.168.10.0/24 192.168.20.0/24 192.168.30.0/24 }
table <martians> const { 0.0.0.0/8 10.0.0.0/8 100.64.0.0/10 127.0.0.0/8 169.254.0.0/16 \
    172.16.0.0/12 192.0.0.0/24 192.0.2.0/24 192.168.0.0/16 \
    198.18.0.0/15 198.51.100.0/24 203.0.113.0/24 224.0.0.0/3 }
# --- Options ----
set block-policy drop
set skip on lo
set loginterface $wan
# --- Normalisation and NAT ----
match in all scrub (no-df random-id)
match out on $vpn inet from <internal_nets> to any nat-to ($vpn)
match on $vpn scrub (max-mss 1380)
# --- Default: block and log everything, no IPv6 ----
block log all label "drop_all"
block quick inet6 all label "drop_ipv6"
# --- DHCP (before the spoofing checks; not logged, dhcpd logs it) ----
pass in quick on $wan inet proto udp from any port bootps to any port bootpc label "allow_wan_dhcp"
pass in quick on $internal inet proto udp from any port bootpc to any port bootps label "allow_dhcp"
# --- Spoofing ----
block in log quick from urpf-failed label "block_spoofed_source"
block in log quick on $wan from <martians> label "block_wan_martians"
# --- Kill switch, layer 2: on WAN only the tunnel and DHCP may leave ----
pass out quick on $wan inet proto udp to <vpn_server> port 51820 label "allow_wan_wireguard"
pass out quick on $wan inet proto udp from any port bootpc to any port bootps label "allow_wan_dhcp_out"
block out log quick on $wan label "block_wan_leak"
# --- Everything the firewall lets in may also leave (logged inbound) ----
pass out quick inet label "allow_out"
# --- Services on the firewall (not logged: very chatty) ----
pass in quick on $internal inet proto { tcp udp } to (self) port domain label "allow_dns_to_fw"
pass in quick on $internal inet proto udp to (self) port ntp label "allow_ntp_to_fw"
pass in quick on $internal inet proto icmp to (self) icmp-type echoreq label "allow_ping_to_fw"
# The PS5 and guests bypass the tunnel, so their DNS does too: a public resolver, straight out
pass in quick on { $ps5 $guest } inet proto { tcp udp } to <public_dns> port { domain 853 } label "allow_direct_dns"
block in log quick on $internal inet proto { tcp udp } to ! (self) port { domain 853 } label "block_external_dns"
# --- Admin access (logged) ----
pass in log quick on { $admin $trusted } inet proto tcp from <admin_hosts> to (self) port ssh label "allow_admin_ssh_fw"
pass in log quick on { $admin $trusted } inet proto tcp from <admin_hosts> to { <k8s_nodes> <unifi_host> <unifi_ap> <kodi> } port ssh label "allow_admin_ssh_hosts"
pass in log quick on { $admin $trusted } inet proto tcp from <admin_hosts> to <k8s_api> port 6443 label "allow_admin_k8s_api"
pass in log quick on { $admin $trusted } inet proto tcp from <admin_hosts> to <k8s_services> port https label "allow_admin_ingress"
pass in log quick on { $admin $trusted } inet proto tcp from <admin_hosts> to <unifi_host> port 11443 label "allow_admin_unifi_ui"
# WebRTC from the UniFi web UI: direct path from the admin laptop to the controller
pass in log quick on { $admin $trusted } inet proto { tcp udp } from <admin_hosts> to <unifi_host> label "allow_admin_unifi_webrtc"
pass in log quick on { $admin $trusted } inet proto icmp from <admin_hosts> to <internal_nets> icmp-type echoreq label "allow_admin_ping"
# --- Known noise: dropped without logging ----
# Broadcast and multicast discovery (NetBIOS, SSDP, ...)
block in quick on $internal inet to { 255.255.255.255 224.0.0.0/4 192.168.1.255 192.168.2.255 192.168.3.255 192.168.4.255 192.168.5.255 192.168.10.255 192.168.20.255 192.168.30.255 } label "drop_broadcast_multicast"
# Late TCP packets (FIN, ACK, RST+ACK) for connections pf no longer tracks.
# New connections (bare SYN) and tracked connections never reach this rule.
block in quick on $internal inet proto tcp from <internal_nets> flags A/A label "drop_late_tcp"
# WebRTC (UniFi UI) also tries the firewall's public address as a detour.
# Reject it immediately, so it gives up at once.
block return in quick on { $admin $trusted $servers } inet proto udp from { <admin_hosts> <unifi_host> } to ($wan) label "reject_webrtc_wan_detour"
# --- Nothing else may reach any of the firewall's own addresses ----
block in log quick on $internal to (self) label "block_to_fw"
# --- Between networks (logged) ----
pass in log quick on $trusted inet proto tcp from $trusted:network to <home_assistant> port 8123 label "allow_trusted_homeassistant"
pass in log quick on $trusted inet proto tcp from $trusted:network to <kodi> port { 18080 9090 } label "allow_trusted_kodi"
pass in log quick on $servers inet from <home_assistant> to $iot:network label "allow_homeassistant_iot"
pass in quick on $apmgmt inet proto tcp from <unifi_ap> to <unifi_host> port 8080 label "allow_ap_unifi_inform"
pass in log quick on $apmgmt inet proto udp from <unifi_ap> to <unifi_host> port 3478 label "allow_ap_unifi_stun"
pass in log quick on $apmgmt inet proto udp from <unifi_ap> to <unifi_host> port 5514 label "allow_ap_unifi_logs"
# The AC Lite checks its uplink by pinging 1.1.1.1 (not configurable in UniFi 10.x); not logged
pass in quick on $apmgmt inet proto icmp from <unifi_ap> to 1.1.1.1 icmp-type echoreq label "allow_ap_uplink_ping"
pass in log quick on $servers inet proto tcp from <unifi_host> to <unifi_ap> port ssh label "allow_unifi_ap_ssh"
# --- Internet access (logged; APMGMT gets none) ----
pass in log quick on $admin inet from $admin:network to ! <internal_nets> label "allow_admin_internet"
pass in log quick on $trusted inet from $trusted:network to ! <internal_nets> label "allow_trusted_internet"
pass in log quick on $ps5 inet from $ps5:network to ! <internal_nets> label "allow_ps5_internet"
pass in log quick on $media inet from $media:network to ! <internal_nets> label "allow_media_internet"
pass in log quick on $servers inet from $servers:network to ! <internal_nets> label "allow_servers_internet"
pass in log quick on $iot inet from $iot:network to ! <internal_nets> label "allow_iot_internet"
pass in log quick on $guest inet from $guest:network to ! <internal_nets> label "allow_guest_internet"
  • Fill in @VPN_SERVER_ADDRESS@ in the <vpn_server> table.
  • max-mss 1380 leaves room for WireGuard's own headers, so large packets do not get stuck.

5. DNS through the tunnel#

unbound now forwards every lookup to the provider's resolver, inside the tunnel. Many providers offer resolvers that also block ads, trackers or malware; pick the address for the blocking you want.

/var/unbound/etc/unbound.confDownload
Show the whole file, 110 lines
# /var/unbound/etc/unbound.conf - fw.internal
server:
	interface: 127.0.0.1
	interface: 192.168.1.1
	interface: 192.168.2.1
	interface: 192.168.3.1
	interface: 192.168.4.1
	interface: 192.168.5.1
	interface: 192.168.10.1
	interface: 192.168.20.1
	interface: 192.168.30.1
	do-ip6: no
	access-control: 0.0.0.0/0 refuse
	access-control: 127.0.0.0/8 allow
	access-control: 192.168.1.0/24 allow
	access-control: 192.168.2.0/24 allow
	access-control: 192.168.3.0/24 allow
	access-control: 192.168.4.0/24 allow
	access-control: 192.168.5.0/24 allow
	access-control: 192.168.10.0/24 allow
	access-control: 192.168.20.0/24 allow
	access-control: 192.168.30.0/24 allow
	# Hardening and privacy
	hide-identity: yes
	hide-version: yes
	harden-glue: yes
	harden-dnssec-stripped: yes
	qname-minimisation: yes
	aggressive-nsec: yes
	prefetch: yes
	# DNSSEC validation
	auto-trust-anchor-file: "/var/unbound/db/root.key"
	val-log-level: 1
	# DNS rebinding protection: public names may not resolve to internal addresses
	private-address: 10.0.0.0/8
	private-address: 100.64.0.0/10
	private-address: 169.254.0.0/16
	private-address: 172.16.0.0/12
	private-address: 192.168.0.0/16
	private-domain: "internal."
	# Tell Firefox not to switch to DNS over HTTPS on its own
	local-zone: "use-application-dns.net." always_nxdomain
	# iCloud Private Relay: switch it off for this network (Apple's documented method),
	# so Safari uses unbound and the VPN's DNS instead of Apple's DNS partners
	local-zone: "mask.icloud.com." always_nxdomain
	local-zone: "mask-h2.icloud.com." always_nxdomain
	# Local names
	local-zone: "internal." static
	local-data: "fw.internal.            IN A 192.168.1.1"
	local-data: "laptop.internal.          IN A 192.168.1.10"
	local-data: "unifi-ap.internal.      IN A 192.168.2.2"
	local-data: "ps5.internal.           IN A 192.168.3.10"
	local-data: "kodi.internal.          IN A 192.168.4.10"
	local-data: "k8s-cp1.internal.       IN A 192.168.5.10"
	local-data: "k8s-cp2.internal.       IN A 192.168.5.20"
	local-data: "k8s-cp3.internal.       IN A 192.168.5.30"
	local-data: "k8s-worker1.internal.   IN A 192.168.5.40"
	local-data: "k8s-worker2.internal.   IN A 192.168.5.50"
	local-data: "unifi.internal.         IN A 192.168.5.60"
	local-data: "k8s-api.internal.       IN A 192.168.5.100"
	local-data: "homeassistant.internal. IN A 192.168.5.140"
	local-data: "siem.internal.          IN A 192.168.5.200"
	local-data: "ingress.internal.       IN A 192.168.5.201"
	# Every *.apps.internal name goes to the ingress (Traefik on .201)
	local-zone: "apps.internal." redirect
	local-data: "apps.internal. IN A 192.168.5.201"
	local-data: "laptop-wifi.internal.     IN A 192.168.10.10"
	local-data: "phone.internal.        IN A 192.168.10.11"
	local-data: "watch.internal.   IN A 192.168.10.12"
	local-data: "vacuum.internal. IN A 192.168.20.10"
	# Reverse lookups (so logs and tools show names instead of IPs)
	local-data-ptr: "192.168.1.1 fw.internal."
	local-data-ptr: "192.168.1.10 laptop.internal."
	local-data-ptr: "192.168.2.2 unifi-ap.internal."
	local-data-ptr: "192.168.3.10 ps5.internal."
	local-data-ptr: "192.168.4.10 kodi.internal."
	local-data-ptr: "192.168.5.10 k8s-cp1.internal."
	local-data-ptr: "192.168.5.20 k8s-cp2.internal."
	local-data-ptr: "192.168.5.30 k8s-cp3.internal."
	local-data-ptr: "192.168.5.40 k8s-worker1.internal."
	local-data-ptr: "192.168.5.50 k8s-worker2.internal."
	local-data-ptr: "192.168.5.60 unifi.internal."
	local-data-ptr: "192.168.5.100 k8s-api.internal."
	local-data-ptr: "192.168.5.140 homeassistant.internal."
	local-data-ptr: "192.168.5.200 siem.internal."
	local-data-ptr: "192.168.5.201 ingress.internal."
	local-data-ptr: "192.168.10.10 laptop-wifi.internal."
	local-data-ptr: "192.168.10.11 phone.internal."
	local-data-ptr: "192.168.10.12 watch.internal."
	local-data-ptr: "192.168.20.10 vacuum.internal."
remote-control:
	control-enable: yes
	control-interface: /var/run/unbound.sock
# All lookups go to the VPN provider's DNS resolver inside the tunnel. If it offers
# ad and tracker blocking, use the address for the blocking you want.
# (Many providers intercept queries to other DNS servers inside the tunnel anyway.)
forward-zone:
	name: "."
	forward-addr: @VPN_DNS_ADDRESS@
  • Only the end of the file is new: the forward-zone.

6. Switch over#

Do this from the console, or from SSH on the admin network, which is not affected.

laptop
scp hostname.igc0 pf.conf unbound.conf fw:/tmp/
firewall
doas install -o root -g wheel -m 0640 /tmp/hostname.igc0 /etc/hostname.igc0
doas sh /etc/netstart igc0 wg0
doas sh -c 'sleep 300 && pfctl -d' &
doas pf-load
doas install -o root -g wheel -m 0644 /tmp/unbound.conf /var/unbound/etc/unbound.conf
doas unbound-checkconf && doas rcctl restart unbound

As in chapter 5: open a new SSH session from the laptop, and if it works, cancel the safety net with doas pkill -f 'sleep 300'.

Check

firewall
doas ifconfig wg0 | grep -E 'wgendpoint|handshake'
route -n show -inet | grep default
route -T 1 -n show -inet | grep default

A latest handshake a few seconds old. The internal routing table's default route is wg0; routing domain 1's default route is your provider's gateway on igc0.

Check

From the laptop, any "what is my IP" page shows the VPN's address, not yours, and your provider's own connection check (most have one, often a URL you can curl) says you are connected.

7. Know when the tunnel is down#

A small script runs every minute and logs an error when the last handshake is more than three minutes old. In chapter 7 of the Cluster and SIEM guide the SIEM alerts on it.

/usr/local/sbin/wg-checkDownload
#!/bin/sh
# Usage: wg-check [max_age_seconds]   (default 180; run from root's crontab every minute)
# Logs "tunnel wg0 DOWN" at daemon.err when the last WireGuard handshake is older
# than max_age, or has never happened. Forwarded to the SIEM, where an alert watches it.
max=${1:-180}
age=$(/sbin/ifconfig wg0 2>/dev/null | sed -n 's/.*last handshake: \([0-9][0-9]*\) seconds ago.*/\1/p')
if [ -z "$age" ] || [ "$age" -gt "$max" ]; then
	/usr/bin/logger -t wg-check -p daemon.err "tunnel wg0 DOWN (last handshake: ${age:-never} s ago)"
fi
/etc/crontab.root add to the endDownload
* * * * * /usr/local/sbin/wg-check
  • The line for root's crontab. The commands below add it without opening an editor.
laptop
scp wg-check fw:/tmp/
firewall
doas install -o root -g bin -m 0555 /tmp/wg-check /usr/local/sbin/wg-check
doas crontab -l > /tmp/crontab.root
echo '* * * * * /usr/local/sbin/wg-check' >> /tmp/crontab.root
doas crontab /tmp/crontab.root && rm /tmp/crontab.root
doas crontab -l | tail -1

The crontab goes through a file rather than a pipe: if doas stops to ask for a password in the middle of a pipe, the read can come back empty and replace root's crontab, which already holds the system's daily maintenance jobs. The last command must show the new line.

Check

firewall
doas wg-check 0
tail -n 1 /var/log/daemon

wg-check itself prints nothing; the log shows wg-check: tunnel wg0 DOWN (last handshake: … s ago). A threshold of zero forces the message, so the warning can be tested without breaking anything.

8. Prove the kill switch#

A kill switch is only worth something once it has been seen to hold. The write-up Testing a VPN kill switch properly goes through it in detail; the short version:

First make sure pf is actually running. A safety net left over from an earlier load switches it off after five minutes, and then the kill switch seems to fail:

Check

firewall
doas pfctl -s info | head -1
pgrep -fl 'sleep 300'

Status: Enabled, and no leftover sleep.

In one window, watch what leaves the WAN port, apart from the tunnel itself. (Without ether src, the screen fills with the internet's scanners knocking on your address.)

firewall
doas tcpdump -ni igc0 "ether src $(ifconfig igc0 | awk '/lladdr/ {print $2}') and not (udp and port 51820)"

In another, take the tunnel down, and try the internet from the laptop:

firewall
doas ifconfig wg0 down

Check

laptop
curl -m 5 https://www.openbsd.org
ping -c 2 9.9.9.9
dig +tries=1 +time=5 $(date +%s).openbsd.org

All three fail. The DNS test uses a name nobody has asked for: a name looked up before still answers, from unbound's cache, without anything leaving. In the tcpdump window, at most a few icmp: … udp port … unreachable replies to the VPN server: it is still sending to the tunnel's closed port. Nothing else.

Check

firewall
doas ping -V 1 -c 3 9.9.9.9
grep block_wan_leak /var/log/pf.log | tail -n 3

The ping is sent from the WAN's routing domain directly, outside the tunnel. It fails with sendmsg: Permission denied, and pf logs each one as block_wan_leak.

firewall
doas ifconfig wg0 up