Proxmox on bare metal, with nested virtual machines that work

What this builds

A bare metal machine running Proxmox VE as its own hypervisor, with a ZFS mirror across the NVMe drives, a bridge that hands guests real addresses, and nested virtualisation switched on and verified — meaning a guest on this machine can itself run guests.

Nesting is the reason to do this on metal rather than on a virtual instance. You cannot nest inside somebody else’s hypervisor with predictable results, and on a BM-E there is no hypervisor of ours in the way: you get IPMI, a power button and the board. What runs on it is your decision.

Before you start

• A BM-E: thirty-two Zen 4 cores, 256 GB of ECC, four NVMe drives, and eight IPv4 addresses. IPMI is reachable over a VPN, which support sets up when the machine is handed over.
• Proxmox VE 8 selected at order time, or a custom ISO uploaded through the panel. Both routes end at the same installer.
• Half an hour when nobody needs the machine, because step one reboots it several times.

1. Install

Connect to the console over the management VPN, mount the ISO and boot it. At the disk selection screen pick all four NVMe devices, choose zfs (RAID10), and open the advanced options:

• ashift at 12, which is the correct value for four-kilobyte sectors and cannot be changed afterwards without rebuilding the pool.
• compress at lz4. It costs a fraction of a core and typically returns a third of the space.
• hdsize reduced to leave the rest for guest storage, unless you want the root pool to own the lot.

Set the management address and gateway from the panel details, and use the machine’s fully qualified name as the hostname. Two reboots later you have a login prompt and a web interface on port 8006.

2. Repositories and packages

The enterprise repository will fail without a subscription and clutter every apt update until you deal with it:

rm -f /etc/apt/sources.list.d/pve-enterprise.sources
echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" > /etc/apt/sources.list.d/pve-community.list
apt update && apt full-upgrade -y
apt install -y ifupdown2 chrony zfsutils-linux
reboot

3. Memory for ZFS

The ARC will otherwise grow to half your RAM, and on a machine whose whole job is running guests that memory has a better use:

echo "options zfs zfs_arc_max=34359738368" > /etc/modprobe.d/zfs.conf
update-initramfs -u -k all

Thirty-two gigabytes of ARC on a 256 GB machine leaves plenty for caching without starving the guests. Reboot for it to apply, or set it live with echo into /sys/module/zfs/parameters/zfs_arc_max if you cannot.

4. Networking

Eight addresses is not many, so guests sit on a private bridge and only the ones that need a public address get one routed to them. Edit /etc/network/interfaces:

auto lo
iface lo inet loopback

auto enp1s0f0
iface enp1s0f0 inet static
  address <public address>/24
  gateway <gateway>

auto vmbr0
iface vmbr0 inet static
  address 10.20.0.1/24
  bridge-ports none
  bridge-stp off
  bridge-fd 0
  post-up   iptables -t nat -A POSTROUTING -s 10.20.0.0/24 -o enp1s0f0 -j MASQUERADE
  post-down iptables -t nat -D POSTROUTING -s 10.20.0.0/24 -o enp1s0f0 -j MASQUERADE

iface vmbr0 inet6 static
  address 2001:db8:1a2b:20::1/64

IPv6 needs no address translation, because the /64 routed to the machine is genuinely routed: assign addresses out of it to guests on the bridge and they reach the internet directly. That asymmetry between the two families is not elegant, but it is the honest shape of the address space we live in.

sysctl -w net.ipv4.ip_forward=1
echo "net.ipv4.ip_forward = 1" > /etc/sysctl.d/80-pve.conf
echo "net.ipv6.conf.all.forwarding = 1" >> /etc/sysctl.d/80-pve.conf
ifreload -a

5. Nested virtualisation

cat /sys/module/kvm_amd/parameters/nested

On current kernels this already prints 1. If it prints 0:

echo "options kvm-amd nested=1" > /etc/modprobe.d/kvm-amd.conf
update-initramfs -u && reboot

Nesting also requires the guest to see a CPU that admits to having the feature, which means the processor type must be host. Anything else presents a synthetic CPU model with the virtualisation extensions masked out, and the inner hypervisor refuses to start with an error that does not mention any of this.

6. A guest

cd /var/lib/vz/template/iso
wget https://cloud.debian.org/images/cloud/trixie/latest/debian-13-genericcloud-amd64.qcow2

qm create 100 --name inner --memory 16384 --cores 8 --cpu host \
  --net0 virtio,bridge=vmbr0 --scsihw virtio-scsi-single --ostype l26 \
  --agent enabled=1 --machine q35 --bios ovmf
qm set 100 --efidisk0 local-zfs:1
qm importdisk 100 debian-13-genericcloud-amd64.qcow2 local-zfs
qm set 100 --scsi0 local-zfs:vm-100-disk-1,discard=on,ssd=1
qm set 100 --boot order=scsi0
qm set 100 --ipconfig0 ip=10.20.0.100/24,gw=10.20.0.1
qm set 100 --sshkeys /root/.ssh/authorized_keys
qm set 100 --ciuser admin
qm start 100

discard=on with ssd=1 means deletions inside the guest are returned to the pool rather than accumulating as space you have lost track of.

7. Firewall

The web interface on port 8006 has no business being publicly reachable. Restrict it at the datacentre level to whatever address you administer from — the tunnel endpoint from the WireGuard guide is the obvious candidate:

cat > /etc/pve/firewall/cluster.fw <<EOF
[OPTIONS]
enable: 1
policy_in: DROP

[RULES]
IN SSH(ACCEPT) -source 10.7.0.0/24
IN ACCEPT -source 10.7.0.0/24 -dport 8006 -proto tcp
IN ACCEPT -p icmp
EOF
systemctl restart pve-firewall

Verify it

Host first:

pveversion -v | head -3
zpool status rpool
zpool list -o name,size,alloc,free,frag,cap
arc_summary | head -12

The pool reports ONLINE with two mirrored pairs, and the ARC target sits at the thirty-two gigabytes you set rather than at half the machine.

Now the actual point of this build. Log into the guest and check the processor it was given:

ssh [email protected]
lscpu | grep -E "Model name|Virtuali"
grep -c svm /proc/cpuinfo

The model name should be the real EPYC part, and the virtualisation line should report AMD-V. A zero from the third command means the processor type is not host, and everything below will fail.

Then run a machine inside the machine:

apt install -y qemu-system-x86 qemu-utils cpu-checker
kvm-ok
wget https://dl-cdn.alpinelinux.org/alpine/v3.22/releases/x86_64/alpine-virt-3.22.0-x86_64.iso
qemu-system-x86_64 -enable-kvm -m 2048 -cdrom alpine-virt-3.22.0-x86_64.iso -nographic

kvm-ok printing that acceleration can be used, followed by an Alpine boot that reaches a login prompt in a couple of seconds rather than crawling, is the verification. Without nesting, that same command either refuses outright or emulates in software at roughly one twentieth of the speed, which is unmistakable.

Last, confirm the interface is closed from outside:

curl -m5 -k https://<public address>:8006/ ; echo "exit $?"

A connection timeout is the correct answer.

Afterwards

Snapshots on ZFS are free until they diverge, so a nightly zfs-auto-snapshot schedule costs almost nothing and has saved more machines than any backup policy. Rebuilds on metal are same-day, but a rebuild returns you an empty machine, which is a different thing from returning your machine.