The problem originates because of installed updates, one after another during many months of usage, until a day … nothing special, you run another update once again and boom!!! The updates caused the /boot partition to be left with no space, and this caused the grub to be blocked when trying to mount the root fs. The solution is simple: just to remove the old kernel images and run a quick cleanup. Another cause would be disk space, which you can see below:

I mounted the volume onto a recover instance of mine and confirmed this using the following command:

$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 479M 12K 478M 1% /dev
tmpfs 97M 312K 97M 1% /run
/dev/nvme0n1p1 7.8G 815M 6.6G 11% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 483M 0 483M 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/nvme1n1p1 50G 50G 0 100% /mnt <—————–

In an attempt to free up space, I removed files in the following folder /mnt/var/cache/apt/archives/ which was taking up just over 5GiB of space:

$ sudo rm /mnt/var/cache/apt/archives/*

Once removed, we can confirm that there is enough space for the instance to boot now:

$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 479M 12K 478M 1% /dev
tmpfs 97M 312K 97M 1% /run
/dev/nvme0n1p1 7.8G 815M 6.6G 11% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 483M 0 483M 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/nvme1n1p1 50G 45G 3.0G 94% /mnt <—————–

 

If the above solution does not work, there is another way you could fix it as it might be that the files within the /boot/ folder are messed beyond repair:

To resolve the issue, I did the following:

-Detached volume of broken AMI 
-Attach it to a recovery instance 
-Mounted the secondary volume to /mnt 
-Removed the complete boot folder off of the secondary volume located under /mnt/boot/
-Once removed, copy the boot folder from the recovery instance to the secondary volume:

$ rsync -avzXA /boot/ /mnt/boot/

Use bind to mount the "sys", "dev", "proc" and "dev/pts" folders:

$ mount -o bind /dev /mnt/dev && mount -o bind /dev/pts /mnt/dev/pts && mount -o bind /proc /mnt/proc && mount -o bind /sys /mnt/sys

Once mounted, start "chroot":

$ chroot 

Once in chroot, install grub using the following:

$ grub-install --no-floppy --recheck --force /dev/nvme1n1p1

Installing for i386-pc platform.
grub-install: warning: File system `ext2' doesn't support embedding.
grub-install: warning: Embedding is not possible.  GRUB can only be installed in this setup by using blocklists.  However, blocklists are UNRELIABLE and their use is discouraged..
Installation finished. No error reported.

Once was successfully installed, update grub:

$ update-grub2

Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.13.0-161-generic
Found initrd image: /boot/initrd.img-3.13.0-161-generic
done

Lastly, rebooted the instance and detached the volume.

This should do the trick.

Leave a Reply

Your email address will not be published. Required fields are marked *