Safely try new Linux kernel in Grub

How to safely try new Linux kernel remotely without loosing access to the server.

Install new kernel.

List kernels in Grub compatible format.

MENUID=$( grep -E "^\\s*(submenu) " /boot/grub/grub.cfg | grep -oP "gnuli.*(?=')" )
grep '(?<=menuentry_id_option ).gnuli\S+' /boot/grub/grub.cfg -Po | \
    sed "s@'@@g" |grep -vE 'recovery|simple|gnulinux-advanced' | \
    sed "s@^@GRUB_DEFAULT=\"${MENUID}>@; s@\$@\"@"

Sample output:

GRUB_DEFAULT="gnulinux-advanced-c6cd75ad-e041-45b4-ad2c-6fa10b5293ee>gnulinux-6.2.16-usb2-advanced-c6cd75ad-e041-45b4-ad2c-6fa10b5293ee"
GRUB_DEFAULT="gnulinux-advanced-c6cd75ad-e041-45b4-ad2c-6fa10b5293ee>gnulinux-6.2.0-36-generic-advanced-c6cd75ad-e041-45b4-ad2c-6fa10b5293ee"
GRUB_DEFAULT="gnulinux-advanced-c6cd75ad-e041-45b4-ad2c-6fa10b5293ee>gnulinux-6.2.0-26-generic-advanced-c6cd75ad-e041-45b4-ad2c-6fa10b5293ee"

So assume we want to try the new kernel 6.2.16-usb2 while old stable kernel is 6.2.0-36-generic.

• New Kernel ID: [line1]
• Old Kernel ID: [line2]

Edit /etc/default/grub and set GRUB_DEFAULT to the OLD kernel ID.

GRUB_DEFAULT="gnulinux-advanced-c6cd75ad-e041-45b4-ad2c-6fa10b5293ee>gnulinux-6.2.0-36-generic-advanced-c6cd75ad-e041-45b4-ad2c-6fa10b5293ee"

Run update-grub.

Mark New kernel to be booted up during the next reboot:

grub-reboot "gnulinux-advanced-c6cd75ad-e041-45b4-ad2c-6fa10b5293ee>gnulinux-6.2.16-usb2-advanced-c6cd75ad-e041-45b4-ad2c-6fa10b5293ee"

Reboot the server:

sync
reboot -f

Wait ~1 minute (depends..), then:

If the server was booted OK:

In /etc/default/grub, set GRUB_DEFAULT=0 (default kernel of higher version)

Run update-grub and sync and reboot

Make sure new kernel is booted up again (because of its higher version)

If the server was not booted up:

Reboot, old kernel will boot (because of GRUB_DEFAULT)

Remove new kernel with apt remove or dpkg -r

In /etc/default/grub, set GRUB_DEFAULT=0 and run update-grub