Can't boot after Bullseye reinstal: /dev/mapper doesn't exist

[SqAtx]

Hi everyone,

I'm currently trying to reinstall Debian 11 (Bullseye) onto my laptop. The first install had worked perfectly a few months ago, but I'm currently stuck trying to boot after reinstalling.

Short description

I get the following error on boot:

ALERT! /dev/mapper/vg--1-lv--root does not exist. Dropping to a shell!

Indeed, if I do `ls /dev/mapper` in the shell I'm presented with, the folder doesn't exist.

But if I boot from the flash drive and execute a shell in the root partition:

Code: Select all

# ls /dev/mapper
control nvme0n1p8_crypt vg--1-lv-home vg--1-lv-root vg--1-lv-swap vg--1-lv-tmp vg--1-lv-var

How come /dev/mapper doesn't exist at boot time? Other devices in /dev seem to be there as normal.

Hopefully this is enough information for somebody to have an idea how to fix things. I've searched the forum and the web but couldn't find a previous occurrence of this particular problem.

Longer description

I originally got a laptop (Dell XPS-15) with a 1TB hard drive and 6 Windows partitions. I resized the big NTFS partition in order to create 750GB of free space. From this space I created a 1GB partition for /boot, and an encrypted volume with several logical volumes, for /home, /, /tmp, /var, and some swap storage.

So the partitioning looks like the following:

• #1 200MB EFI system partition, mounted to /boot/efi
• #2 140MB Microsoft recovery partition
• #3 to #6 NTFS partitions
• #7 1GB ext4 partition mounted to /boot
• #8 750GB encrypted volume

And the encrypted volume contains the following logical volumes:

• LV #1 540GB ext4 lv-home mounted to /home
• LV #2 150GB ext4 lv-root mounted to /root
• LV #3 16GB swap space
• LV #4 25GB ext4 lv-tmp mounted to /tmp
• LV #5 25GB ext4 lv-var mounted to /var

Partitions #1-#6 were here since I got the laptop and I haven't touched them. This setup worked on my first install.

When reinstalling, I booted from the same flash drive as the original install, used Rescue Mode and formatted all the LVs except /home. I also formatted partition #7, setting it to ext4 and mounting it to /boot as before. Note that in the installer, I was unable to set its bootable flag to "on". But I'm pretty sure that was also the case during the original install.

So far so good, but on boot I get

Code: Select all

[ <number> ] psmouse serio1: elantech: elantech_semd_cmd query 0x02 failed
[  1.598600] psmouse serio1: elantech: failed to query capabiities

Volume group "vg-1" not found
Cannot process volume group vg-1

<last 2 line srepeated many times>

Gave up waiting for suspend/resume device
  Volume group "vg-1" not found
  Cannot process volume group vg-1
Gave up waiting for root file system device.  Common problems:
- Boot args (cat /proc/cmdline)
  - Check rootdelay= (did the system wait long enough?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT!  /dev/mapper/vg--1-lv--root does not exist.  Dropping to a shell!

BusyBox v1.30.1 (Debian 1:1.30.1-6+b3) built-in shell (ash)
Enter 'help' for a list of built-in commands

(initramfs) cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-5.10.0-10-amd64 root=/dev/mapper/vg--1-lv--root ro quiet

The output of `cat /proc/cmdline` is a bit too verbose to be copied here. I don't really know how to parse it. What should I be looking for there?

From `ls /dev`, I can see that /dev/mapper doesn't exist, which explains the error message.

But then I executed a shell in the root logical volume using the following steps:

• Booting from flash drive
• Advanced options...
• Graphical rescue mode
• Select language, keyboard layout, network, hostname etc
• Enter passphrase for encrypted volume /dev/nvme0n1p8
• Select root system /dev/vg-1/lv-root
• Mount /boot partition: Yes
• Mount /boot/efi partition: Yes
• Execute a shell in /dev/vg-1/lv-root

and:

Code: Select all

# mount -a
# ls /dev/mapper
control nvme0n1p8_crypt vg--1-lv-home vg--1-lv-root vg--1-lv-swap vg--1-lv-tmp vg--1-lv-var

So /dev/mapper seems to exist and have the right content?

Things I've tried

I've tried reinstalling again.

From the graphical rescue mode, I also tried reinstalling GRUB into the boot partition (/dev/nvme0n1p7), to no avail.

In my searches, the closest thing I found to my problem was bug #968295, which was caused by a bad install of the cryptsetup package. That doesn't seem to be my problem. From a shell in the root partition:

Code: Select all

# mount -a
# dpkg -l cryptsetup lvm2
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-int/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Statu,Err; uppercase=bad)
||/ Name         Version      Architecture Description
++-=============-============-============-===========================
ii  cryptsetup   2:2.3.5-1    amd64        disk encryption support - startup scripts
ii  lvm2         2.03.11-2.1  amd64        Linux Logical Volume Manager

I'm running out of ideas and am considering deleting and recreating the encrypted volume. But since I don't understand what the problem is, this may not even fix it.

Thanks to anyone who has read the whole post, and I hope someone will be able to point me in the right direction!

[SqAtx]

(I'm going to post my partial answer here - I think it makes more sense than editing it inside the original post. Let me know if I'm wrong. If I make more progress before anyone else posts, then I'll just edit the current post and not add a new one)

One thing I just realized after reading more about LVM: the error happens before I'm asked to enter the passphrase to decrypt the LVM physical volume. So it makes sense that at that time, /dev/mapper is empty, since the system can't know that the physical volume contains.
And it makes sense that it contains the expected volumes when I access it from rescue mode, since rescue mode does ask for that passphrase.

I guess the question isn't so much "why is /dev/mapper empty?", but rather "why is the system looking for /dev/mapper/vg--1-lv--root before decrypting the volume?".

And the answer may be related to the previous logs:

Volume group "vg-1" not found
Cannot process volume group vg-1

So the /dev/mapper thing is likely a red herring. The system tried to process the volume group, couldn't, gave up, tried to perform the next step and obviously couldn't.

Now I need to figure out why vg-1 couldn't be found.

[p.H]

why is the system looking for /dev/mapper/vg--1-lv--root before decrypting the volume?

Because the initramfs is generic and does not know whether LVM is inside an encrypted volume or an encrypted volume is inside LVM. So it tries to activate LVM first, then unlock encrypted volumes, then activate LVM again. This way it even supports LVM inside encrypted volumes inside LVM, which is a useful feature. It does not support encrypted volumes inside LVM inside encrypted volumes, but dual layer encryption is not very useful IMO.

[p.H]

LV #2 150GB ext4 lv-root mounted to /root

I don't think so.

When reinstalling, I booted from the same flash drive as the original install, used Rescue Mode and formatted all the LVs except /home. I also formatted partition #7, setting it to ext4 and mounting it to /boot as before.

How in that process did you install a system ? Rescue mode is not intended for installing.

I also formatted partition #7, setting it to ext4 and mounting it to /boot as before. Note that in the installer, I was unable to set its bootable flag to "on"

Of course not. With GPT, the boot flag means an EFI partition. /boot is not an EFI partition.

If the initramfs contains cryptsetup, you can unlock the encrypted volume from the initramfs shell with

Code: Select all

cryptsetup luksOpen /dev/nvme0n1p8 nvme0n1p8_crypt

type the passphrase when prompted, then

Code: Select all

vgchange -ay

should show LVM. Press ctrl+d to resume the boot process.

dpkg -l cryptsetup lvm2

You also need cryptsetup-initramfs (should already be installed as a cryptsetup recommends) and /etc/crypttab must contain the proper line to unlock the encrypted partition. Then rebuild the initramfs to include them with

Code: Select all

update-initramfs -u

[SqAtx]

Thanks a lot for your comments! They were very helpful in understanding how the pieces fit together.

Yes I'd indeed convinced myself that I needed to use rescue mode, which obviously isn't the case. It seems that I've messed up my system by doing that, although I'm not sure exactly how.

Using cryptsetup from initramfs indeed worked, so I could complete the boot sequence. It wasn't the end of it, because my user no longer existed, but a good step in understanding what was going on.

In the end, my problem was solved by doing a proper reinstall. The biggest challenge was that the installer didn't decrypt the volume, so it could only see a generic 750GB partition. Fortunately, the steps to decrypt the volume before the partition step, are described very well at Install Debian on Existing encrypted LVM.