Hey guys,

I’m going to show you a quick way to reproduce a common kernel panic error we get with AWS instances which is  “Kernel panic – not syncing: VFS”.

Lets begin::::

Ssh into the instance:

[ec2-user@imraan ~]$ ssh -i germany.pem ec2-user@10.0.0.59

The authenticity of host ‘10.0.0.59 (10.0.0.59)’ can’t be established.

ECDSA key fingerprint is SHA256:LVWC1QuTyB+hLq4jbHbvMGhhBrkTG2PpNnI5KsNuTbE.

ECDSA key fingerprint is MD5:57:45:70:c7:f1:94:0a:aa:eb:5e:ee:5e:54:47:c7:64.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added ‘10.0.0.59’ (ECDSA) to the list of known hosts.

Last login: Tue Jul  9 07:44:16 2019 from 52.59.200.99

       __|  __|_  )

       _|  (     /   Amazon Linux 2 AMI

      ___|\___|___|

[ec2-user@ip-10-0-0-59 ~]$

Switch to sudo:

[ec2-user@ip-10-0-0-59 ~]$ sudo su

Switch to the /boot folder and list the files:

[root@ip-10-0-0-59 boot]# ls -l

total 118024

-rw-r–r–. 1 root root   117766 May  7 02:40 config-4.14.114-105.126.amzn2.x86_64

-rw-r–r–. 1 root root   119807 May 22 17:07 config-4.14.121-109.96.amzn2.x86_64

-rw-r–r–. 1 root root   119850 Jun 10 19:51 config-4.14.123-111.109.amzn2.x86_64

drwxr-xr-x. 3 root root       17 Jun 22  2018 efi

drwx——. 5 root root      114 Jun 19 06:52 grub2

-rw——-. 1 root root 31250593 May 24 09:56 initramfs-4.14.114-105.126.amzn2.x86_64.img

-rw——-. 1 root root 31210388 Jun 13 03:53 initramfs-4.14.121-109.96.amzn2.x86_64.img

-rw——-. 1 root root 31211474 Jun 19 06:52 initramfs-4.14.123-111.109.amzn2.x86_64.img

-rw-r–r–. 1 root root   646877 Jun 22  2018 initrd-plymouth.img

-rw-r–r–. 1 root root   232904 May  7 02:40 symvers-4.14.114-105.126.amzn2.x86_64.gz

-rw-r–r–. 1 root root   233611 May 22 17:08 symvers-4.14.121-109.96.amzn2.x86_64.gz

-rw-r–r–. 1 root root   233891 Jun 10 19:52 symvers-4.14.123-111.109.amzn2.x86_64.gz

-rw——-. 1 root root  2801678 May  7 02:40 System.map-4.14.114-105.126.amzn2.x86_64

-rw——-. 1 root root  2802558 May 22 17:07 System.map-4.14.121-109.96.amzn2.x86_64

-rw——-. 1 root root  2809597 Jun 10 19:51 System.map-4.14.123-111.109.amzn2.x86_64

-rwxr-xr-x. 1 root root  5665744 May  7 02:40 vmlinuz-4.14.114-105.126.amzn2.x86_64

-rwxr-xr-x. 1 root root  5673936 May 22 17:07 vmlinuz-4.14.121-109.96.amzn2.x86_64

-rwxr-xr-x. 1 root root  5690320 Jun 10 19:51 vmlinuz-4.14.123-111.109.amzn2.x86_64

The files we are going to remove are the initramfs files below:

-rw——-. 1 root root 31250593 May 24 09:56 initramfs-4.14.114-105.126.amzn2.x86_64.img

-rw——-. 1 root root 31210388 Jun 13 03:53 initramfs-4.14.121-109.96.amzn2.x86_64.img

-rw——-. 1 root root 31211474 Jun 19 06:52 initramfs-4.14.123-111.109.amzn2.x86_64.img

Lets create a folder to store the files for our test:

[root@ip-10-0-0-59 ~]# mkdir initramfs

Lets switch back to the boot folder and move the files to the new folder we created:

[root@ip-10-0-0-59 /]# cd /boot/

[root@ip-10-0-0-59 boot]# mv initramfs-4.14.114-105.126.amzn2.x86_64.img initramfs-4.14.121-109.96.amzn2.x86_64.img initramfs-4.14.123-111.109.amzn2.x86_64.img /root/initramfs/

Verify the files are gone:

[root@ip-10-0-0-59 boot]# ls -l

total 26544

-rw-r–r–. 1 root root  117766 May  7 02:40 config-4.14.114-105.126.amzn2.x86_64

-rw-r–r–. 1 root root  119807 May 22 17:07 config-4.14.121-109.96.amzn2.x86_64

-rw-r–r–. 1 root root  119850 Jun 10 19:51 config-4.14.123-111.109.amzn2.x86_64

drwxr-xr-x. 3 root root      17 Jun 22  2018 efi

drwx——. 5 root root     114 Jun 19 06:52 grub2

-rw-r–r–. 1 root root  646877 Jun 22  2018 initrd-plymouth.img

-rw-r–r–. 1 root root  232904 May  7 02:40 symvers-4.14.114-105.126.amzn2.x86_64.gz

-rw-r–r–. 1 root root  233611 May 22 17:08 symvers-4.14.121-109.96.amzn2.x86_64.gz

-rw-r–r–. 1 root root  233891 Jun 10 19:52 symvers-4.14.123-111.109.amzn2.x86_64.gz

-rw——-. 1 root root 2801678 May  7 02:40 System.map-4.14.114-105.126.amzn2.x86_64

-rw——-. 1 root root 2802558 May 22 17:07 System.map-4.14.121-109.96.amzn2.x86_64

-rw——-. 1 root root 2809597 Jun 10 19:51 System.map-4.14.123-111.109.amzn2.x86_64

-rwxr-xr-x. 1 root root 5665744 May  7 02:40 vmlinuz-4.14.114-105.126.amzn2.x86_64

-rwxr-xr-x. 1 root root 5673936 May 22 17:07 vmlinuz-4.14.121-109.96.amzn2.x86_64

-rwxr-xr-x. 1 root root 5690320 Jun 10 19:51 vmlinuz-4.14.123-111.109.amzn2.x86_64

Now reboot:

[root@ip-10-0-0-59 boot]# telinit 6

Connection to 10.0.0.59 closed by remote host.

Connection to 10.0.0.59 closed.

Now check the logs:

[    1.202482] No filesystem could mount root, tried:

[    1.202483]

[    1.208514] Kernel panic – not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

[    1.214971] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.14.123-111.109.amzn2.x86_64 #1

[    1.221402] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006

[    1.226005] Call Trace:

[    1.228488]  dump_stack+0x5c/0x82

[    1.231648]  panic+0xe4/0x252

[    1.234450]  mount_block_root+0x284/0x2b6

[    1.237876]  ? SyS_mknod+0x167/0x200

[    1.240956]  ? set_debug_rodata+0x11/0x11

[    1.244357]  prepare_namespace+0x135/0x16b

[    1.248893]  kernel_init_freeable+0x217/0x248

[    1.252625]  ? rest_init+0xb0/0xb0

[    1.255607]  kernel_init+0xa/0xfc

[    1.258563]  ret_from_fork+0x35/0x40

[    1.261989] Kernel Offset: disabled

As you can see, we reproduced a “Kernel panic – not syncing: VFS: Unable to mount root fs on unknown-block(0,0)”

Leave a Reply

Your email address will not be published. Required fields are marked *