Get in when the network is down

What it is

An authenticated tunnel to the instance’s serial line and virtual display. It is included on every plan, it does not traverse your instance’s network stack, and it keeps working when sshd is dead, the firewall is wrong, the default route is missing or the root filesystem has gone read-only. Sessions are opened from the panel. Each one is a single-use grant, expires in minutes, and requires the second factor on your account.

Check the serial line before you need it

Most of our images are already correct. Confirm:

cat /proc/cmdline
systemctl status [email protected]

You want console=ttyS0,115200n8 on the command line and a running getty. On Debian, add console=tty0 console=ttyS0,115200n8 to GRUB_CMDLINE_LINUX in /etc/default/grub, then:

update-grub
systemctl enable --now [email protected]

AlmaLinux 10 uses boot loader specification entries, so edit them with grubby rather than by hand:

grubby --update-kernel=ALL --args="console=tty0 console=ttyS0,115200n8"
systemctl enable --now [email protected]

Getting a shell when there is no shell

From the console, reboot and interrupt the boot loader. GRUB shows for a couple of seconds; press e on the entry you want and edit the linux line.

• Services stopped, network down, root mounted: append systemd.unit=rescue.target, then Ctrl-X.
• Rescue itself failing: systemd.unit=emergency.target.
• No password and no key: init=/bin/bash.

mount -o remount,rw /
passwd root
mount -o remount,ro /

With init=/bin/bash there is no init to shut anything down, so remount read-only and power-cycle from the panel. Typing reboot at that prompt does nothing useful.

The firewall case

Locking yourself out with nftables is the single most common reason a console session gets opened. The way back:

nft list ruleset
nft flush ruleset
systemctl restart sshd

An empty ruleset means no filtering at all, so repair the file and load it again in the same sitting rather than leaving it until tomorrow. Next time, load a new ruleset behind a dead man’s switch:

systemd-run --on-active=120 nft flush ruleset
nft -f /etc/nftables.conf

The first command prints the transient unit it created. If you are still connected two minutes later, stop that timer and keep the rules. If you are not, it undoes them without you.

Reading a boot that failed

The serial console carries the whole boot, so a panic or a failed mount is something you read rather than something you infer from silence. Once you are back in, the previous boot is still in the journal:

journalctl -b -1 -p err

What it is not

It is not a rescue system with a root filesystem of its own. For that, boot the rescue image from the panel: a small live environment, your disks unmounted, and the tools to repair them.

It is also not somewhere to leave a session open. The grant lasts as long as the session does, so close it when the machine is fixed.