Ubuntu won't boot after a kernel update: fixing GRUB from the panel console

A familiar sequence: you run apt upgrade, a new kernel comes down with it, you reboot, and the server never answers again. No ping, SSH times out, the site is gone. When Ubuntu won't boot after a kernel update there is nothing left to fix over the network, because the machine never gets as far as networking. It is stuck on a screen you have probably never seen on a VPS, since nobody normally shows it to you: the GRUB boot menu. This walkthrough covers reaching the server console, catching that menu, booting the previous working kernel by hand, and making sure the next upgrade does not do the same thing.

Takeaways. SSH is not an option here. Use the VNC console (also called the web console) in your account panel: it attaches to the virtual machine's display and works with no network at all. Reboot the server from the panel and immediately start tapping Esc, or hold Shift on a BIOS machine, until the GRUB menu appears. Open "Advanced options for Ubuntu", pick the previous kernel from the list, and the system usually comes up. After that you can work over SSH again: df -h /boot (a full partition is the single most common cause), sudo dpkg --configure -a, sudo update-initramfs -u -k all, sudo update-grub. If you land on a grub rescue> prompt instead of a menu, the loader lost the path to its own directory: set it manually, boot once, then repair with grub-install. Finish by editing /etc/default/grub so the menu is actually displayed. On a machine with no monitor, a hidden menu saves three seconds and costs an hour of downtime.

Where boot actually breaks

Booting is a chain of four links. Knowing them pays off, because whatever is on the console screen tells you which link failed.

• Firmware (BIOS or UEFI, the code baked into the virtual machine's "motherboard") looks for a boot loader on disk and hands control to it.
• GRUB is that boot loader. It reads its config at /boot/grub/grub.cfg, draws the menu, and loads the kernel you pick.
• The Linux kernel lives in files like /boot/vmlinuz-6.8.0-84-generic. The number in the filename is the kernel version.
• initramfs is a small temporary filesystem the kernel unpacks into RAM so it can find and mount the real root disk. It sits next to the kernel as /boot/initrd.img-6.8.0-84-generic. Missing or corrupted, the kernel starts and immediately discovers it has no root filesystem.

After that systemd takes over, and that is a separate class of problem: once you see lines like Started ... and [ OK ], the boot loader did its job.

error: file '/boot/vmlinuz-...' not found

Kernel panic - not syncing: VFS: Unable to mount root fs

Step 1. The panel console, because SSH cannot help

SSH is a service inside an already running system. Until Ubuntu reaches the point of starting services, there is nothing to connect to, and retrying ssh root@... will never produce a different result. You need a different channel: the VNC console in your account panel. Technically it is a virtual monitor and keyboard plugged into the virtual machine, so it shows everything from the first firmware screen onward, GRUB menu included.

In my.weasel.cloud, open the page for the server and find the console or VNC section. The reboot control is usually in the same area, and you will need it in a moment. Open the console before rebooting: the window has to be live at the moment the machine starts, or you miss the few seconds in which the menu can be caught.

One detail that makes people give up early: a web console delivers keystrokes with a delay and occasionally drops them. Click inside the console window first so it holds keyboard focus, then type slowly and deliberately.

Step 2. Catching the GRUB menu when it refuses to show

Ubuntu server images hide the menu almost every time. With a single operating system installed, GRUB is configured to skip the choice instantly. To catch it, press reboot in the panel and immediately start tapping Esc about twice a second. That is the UEFI behaviour. On a classic BIOS machine you hold Shift instead, from the very first second. If you do not know which one you have, try each in turn over two reboots. Neither can damage anything.

If a menu appears with entries like Ubuntu and Advanced options for Ubuntu, go to step 3. If two or three attempts produce nothing, read the grub rescue section: most likely GRUB never gets as far as drawing a menu.

There is one case that looks exactly like a hung server. GRUB marks a failed boot with a recordfail flag and, on the next attempt, shows the menu with a very long wait instead of booting on its own. On a desktop that is reasonable behaviour. On a VPS it means the server appears dead for hours while it patiently waits for someone to press Enter. If the console shows a menu with no counter running, just select an entry by hand. Turning this off for good is covered below.

Step 3. Boot the previous kernel and see the real error

Ubuntu keeps the previous kernel installed precisely for this situation. Older versions live in a submenu: highlight Advanced options for Ubuntu, press Enter, and you get a list with two lines per version, a normal one and a (recovery mode) one.

Take the second normal entry from the top, which is the previous kernel, and press Enter. If the system comes up and SSH answers, the hard part is over and the rest is done over the network. Treat it as a one-time boot: the next reboot goes back to the broken kernel until you fix the config.

When you have no idea what is failing, look at the error first. Ubuntu hides kernel messages behind the quiet splash parameters. Highlight the entry, press e, and an editor for that entry opens. Find the line starting with linux, delete quiet splash from it, and press Ctrl+X (or F10) to boot with the change. The edit lives until reboot and never touches the disk.

Kernel messages will scroll past, and the last lines before the stop usually name the cause outright: root device not found, initramfs panic, filesystem error. Write those down, that is what you search for.

If every kernel in the list fails the same way, use the same editor to append systemd.unit=rescue.target to the linux line. The system comes up in a minimal single-user state and asks for the root password. On WeaselCloud you have it: the panel shows it on the server page, and the differences between root access methods are covered in the piece on root login on Ubuntu. On providers that ship root locked, this path is closed, which is one more reason to know your root password in advance.

When you get a grub rescue prompt

A grub rescue> prompt means something specific: the tiny first stage of GRUB started from disk, but it could not find the directory holding its modules and config (/boot/grub). It shows up after migrating or restoring a server from an image, after partition changes, and after an interrupted upgrade of the GRUB package itself. Only a handful of commands work at this prompt, but they are enough for one manual boot.

Start by listing what GRUB can see:

ls

You get output like (hd0) (hd0,gpt2) (hd0,gpt1). Now find which one holds the system. Check them one by one:

ls (hd0,gpt2)/

• directories named boot, etc, home: this is the root partition, the one you want;
• grub, vmlinuz-..., initrd.img-... at the top level: this is a separate /boot partition, so the path to the GRUB directory is shorter, without a leading /boot;
• unknown filesystem: wrong partition, try the next number.

Then tell GRUB where its home is and load the module for normal mode:

set prefix=(hd0,gpt2)/boot/grub

set root=(hd0,gpt2)

insmod normal

normal

• prefix is the directory GRUB loads its modules and grub.cfg from;
• root is the partition all other paths are resolved against;
• insmod normal loads the full-mode module and normal runs it. With the right values, the usual GRUB menu appears.

Boot any working kernel from that menu and make the repair permanent straight away, otherwise the next reboot drops you back at the same prompt. On a BIOS machine the loader is installed to the whole disk, not to a partition:

sudo grub-install /dev/vda

sudo update-grub

On UEFI the command differs and takes no disk argument, because the loader goes into the EFI system partition:

sudo grub-install --target=x86_64-efi --efi-directory=/boot/efi

sudo update-grub

To tell the two apart: if the directory /sys/firmware/efi exists, you are on UEFI. The check [ -d /sys/firmware/efi ] && echo UEFI || echo BIOS answers in one word.

Why the kernel update broke boot

You are back inside the system, so now find the cause. In the large majority of cases it is one of three things, and all three take a minute to check.

One: /boot ran out of space. If that directory sits on its own small partition, a couple of extra kernels are enough to fill it. The kernel package then installs, but its initramfs cannot be built, and the failure scrolls past in the apt output among a hundred other lines.

df -h /boot

ls -l /boot

Look at the Use% column. At or near 100 percent, you have your answer. Check the file listing at the same time: every version needs a matching pair, vmlinuz-version and initrd.img-version. A kernel with no initrd cannot boot, and that is exactly the unable to mount root fs panic.

Remove old kernels through the package manager, never with rm:

sudo apt autoremove --purge

This drops kernel packages nothing depends on any more and regenerates the GRUB config at the end. Deleting files from /boot by hand leaves the package database convinced the kernel is still installed, and the resulting menu is more wrong than before.

Two: the upgrade was interrupted halfway. The connection dropped during apt upgrade, someone pressed Ctrl+C, or the server rebooted on its own, and some packages are left half-configured. Two commands handle it:

sudo dpkg --configure -a

sudo apt --fix-broken install

The first finishes whatever was left unfinished, including building the initramfs and refreshing the GRUB menu. The second pulls in missing dependencies. Both are safe to run even if this was not the problem: on a healthy system they do nothing.

Three: the initramfs exists but is wrong. This follows manual experiments with modules or storage settings. Rebuild it for a specific version:

sudo update-initramfs -u -k 6.8.0-84-generic

Replace 6.8.0-84-generic with your own version from ls /boot. The -u flag means "update the existing image", -k selects the kernel. Use all in place of the version number to rebuild for every installed kernel.

After any of these, regenerate the menu:

sudo update-grub

The output should read Generating grub configuration file ..., then one Found linux image: and one Found initrd image: line per version, and done. A version missing from that list is missing from the disk, so the loader is not at fault. No Found initrd image lines at all means the initramfs were never built: go back to the disk space check.

Fixing /etc/default/grub: menu, timeout, and where output goes

The file /etc/default/grub holds the settings that update-grub compiles into the real /boot/grub/grub.cfg. Always edit the first one. Editing the second is pointless: it is overwritten on every kernel update, and your changes vanish at the worst possible moment.

sudo nano /etc/default/grub

A sensible set of values for a server that has a VNC console:

GRUB_TIMEOUT_STYLE=menu

GRUB_TIMEOUT=5

GRUB_RECORDFAIL_TIMEOUT=5

• GRUB_TIMEOUT_STYLE=menu shows the menu every time instead of hiding it. Five seconds of delay at boot is a fair trade against hammering Esc under pressure.
• GRUB_TIMEOUT=5 boots the default entry after five seconds. Zero means "never show it", and -1 means "wait forever", which is the last thing you want on a server.
• GRUB_RECORDFAIL_TIMEOUT=5 is what stops the machine from parking in the menu after a failed boot. Without it, one failure can leave the server waiting for a keypress until you open the console.

Now the trap that costs people an hour. You edit /etc/default/grub, run update-grub, and nothing changes. The reason is the neighbouring directory /etc/default/grub.d/: files in it are read after the main file, so they override it. Ubuntu cloud images usually ship one there with cloud-specific settings, and that is what sets the timeout to zero.

ls -l /etc/default/grub.d/

grep -r "GRUB_TIMEOUT" /etc/default/grub /etc/default/grub.d/

If some other file sets its own GRUB_TIMEOUT, do not edit it: a package update will bring it back. Create your own file with a name that sorts last:

sudo nano /etc/default/grub.d/99-console.cfg

Put the same three lines in it. The last assignment wins, so your values take effect. Only then run:

sudo update-grub

You can confirm the result without rebooting: grep -E "^set timeout|^\s+set timeout" /boot/grub/grub.cfg shows the values that actually made it into the menu.

The server keeps booting an old kernel

A separate symptom: the system boots fine, but uname -r reports a version you did not install. Usually this is a leftover from picking an entry by hand. When "remember my choice" is enabled, GRUB stores that selection and keeps using it.

sudo grub-editenv list

A saved_entry=... line in the output is the stuck choice. Clear it:

sudo grub-editenv /boot/grub/grubenv unset saved_entry

sudo update-grub

The server now boots the first menu entry again, which is the newest installed kernel. The reverse task, pinning a specific older kernel while you investigate the new one, is done with GRUB_DEFAULT in /etc/default/grub set to the exact entry name; list the names with sudo grep -E "menuentry |submenu " /boot/grub/grub.cfg | cut -d"'" -f2. A simpler and more reliable stopgap is sudo apt-mark hold linux-image-generic, which blocks new kernel installs. Do not leave it in place for long: you stop receiving kernel security fixes along with the new versions.

Making sure it does not happen again

Three habits that genuinely reduce the odds, in order of value.

• Reboot right after a kernel update, not "sometime later". Reboot a month later and you will no longer remember what changed, so you get to rediscover the link between the upgrade and the failure from scratch. Do it when you have half an hour to spare, not on a Friday evening.
• Keep two or three kernels and watch the free space. Keeping one is risky, since there is nothing to fall back to. Keeping five is pointless and, on a small /boot, is the failure itself. A periodic sudo apt autoremove --purge plus a monthly look at df -h /boot covers it.
• Take a disk snapshot before a major upgrade if the panel offers one. Rolling back to a snapshot takes minutes; diagnosing a broken boot over VNC takes hours.

One more thing about console output. Make sure GRUB_CMDLINE_LINUX_DEFAULT includes console=tty1, the ordinary display that the VNC console shows you. Cloud images often add console=ttyS0 for the serial port as well, and order matters: the last device listed becomes the primary one for system messages. If the panel console stays blank while the machine is clearly booting, the output almost certainly went to the serial port.

FAQ

How do I open the GRUB menu on a headless VPS?

Through the VNC or web console in the provider's panel. It is attached to the machine's virtual display and shows the screen from power-on, which SSH cannot do. Open the console, trigger a reboot, and immediately tap Esc (UEFI) or hold Shift (BIOS). The console window must be open and focused before the reboot starts, or the first keystrokes are lost.

What do I do at a grub rescue prompt?

It means GRUB started but could not find its /boot/grub module directory. Run ls, identify the right partition with ls (hd0,gpt2)/, then set set prefix=(hd0,gpt2)/boot/grub, set root=(hd0,gpt2), insmod normal, normal. That boots the system once. Make it permanent afterwards with sudo grub-install and sudo update-grub, or the next reboot returns to the same prompt.

Why does update-grub not see the new kernel?

Because the kernel files are not on disk, not because the loader skipped them. Check ls /boot: each version needs both a vmlinuz- and an initrd.img- file. The most common reason for a missing initrd is a full /boot partition that prevented the image from being built; the second is an interrupted upgrade, cleared with sudo dpkg --configure -a.

Can I just delete old kernels to free up /boot?

Remove them as packages, not as files: sudo apt autoremove --purge. Deleting vmlinuz and initrd.img with rm leaves the package database thinking the kernel is installed, and the GRUB menu keeps an entry that points at a missing file, which fails at boot with error: file not found. Always keep one working kernel besides the current one, or you have nothing to fall back to.

Do I have to reboot after every kernel update?

Yes, otherwise the new kernel just sits on disk unused: the running kernel is only replaced at boot. The file /var/run/reboot-required tells you a reboot is pending. Compare uname -r (what is running right now) with the newest version in ls /boot: if they differ, the reboot has not happened yet.

Takeaways

• When a server does not come back after an update, the VNC console in the panel is the only working channel. SSH cannot reach a system that has not booted.
• The GRUB menu is hidden on server images: catch it by tapping Esc on UEFI or holding Shift on BIOS right after you trigger the reboot.
• The previous kernel lives under "Advanced options for Ubuntu". That submenu is the supported rollback path, and the reason Ubuntu keeps old kernels around.
• Pressing e in the menu opens a one-time edit of an entry. Remove quiet splash to see the actual error; nothing is written to disk.
• grub rescue> is not a dead disk, it is a lost path to /boot/grub: set it with set prefix, then make it permanent with grub-install.
• Three usual causes after an upgrade: a full /boot, an interrupted upgrade (sudo dpkg --configure -a), and a missing initramfs (sudo update-initramfs -u -k all).
• Edit only /etc/default/grub and your own file in /etc/default/grub.d/. The file /boot/grub/grub.cfg is generated and gets overwritten.
• Files in /etc/default/grub.d/ are read after the main file and override it, so name yours to sort last.
• No change takes effect until you run sudo update-grub.
• A headless server should display its menu: GRUB_TIMEOUT_STYLE=menu, GRUB_TIMEOUT=5, and above all GRUB_RECORDFAIL_TIMEOUT=5, or one failed boot leaves the machine waiting for a keypress indefinitely.

Where to next

If the system boots but stays unreachable, the boot loader is no longer the problem: see how to restart networking on Ubuntu and Debian. The root password from the panel is useful beyond rescue mode too; the differences between su -, sudo -i and a direct login are covered in root login on Ubuntu. For the broader list of what usually breaks on a fresh server, see common Linux server problems.