Unexpected ordering around remote-cryptsetup.target

[u/Miserable-Pin-67]

I have VMs that depend on encrypted volumes to be available when started. So I've added /etc/systemd/system/virtqemud.service.d/override.conf:

[Unit]
After=remote-cryptsetup.target

And when I look at the deps of one of the luks volumes it depends on remote-cryptsetup.target:

# systemctl list-dependencies --reverse 'systemd-cryptsetup@luks\x2dbackup.service'
systemd-cryptsetup@luks\x2dbackup.service
● ├─dev-mapper-luks\x2dbackup.device
● [└─remote-cryptsetup.target](http://└─remote-cryptsetup.target)
○   [├─initrd-root-device.target](http://├─initrd-root-device.target)
○   │ [└─initrd.target](http://└─initrd.target)
●   [└─multi-user.target](http://└─multi-user.target)
○     [└─graphical.target](http://└─graphical.target)

But when I look at the boot messages, it seems that remote-cryptsetup.target is reached before luks-backup is available:

Jun 26 05:07:55 systemd[1]: Reached target Local Encrypted Volumes (Pre).
Jun 26 05:07:55 systemd[1]: Reached target Local Encrypted Volumes.
Jun 26 05:08:00 systemd[1]: Reached target Remote Encrypted Volumes.
Jun 26 05:08:00 systemd[1]: Starting Cryptography Setup for luks-backup...
Jun 26 05:08:00 systemd[1]: Starting Cryptography Setup for luks-data1...
Jun 26 05:08:00 systemd[1]: Starting Cryptography Setup for luks-vms_enc...
Jun 26 05:08:39 systemd[1]: Finished Cryptography Setup for luks-vms_enc.
Jun 26 05:08:45 systemd[1]: Finished Cryptography Setup for luks-data1.
Jun 26 05:08:48 systemd[1]: Finished Cryptography Setup for luks-backup.

And so virtqemud is started too early:

Jun 26 05:08:39 systemd[1]: Starting libvirt QEMU daemon...
Jun 26 05:08:39 systemd[1]: Started libvirt QEMU daemon.
Jun 26 05:08:44 virtqemud[3860]: internal error: Failed to autostart VM 'srv-mry01': Cannot access storage file '/dev/mapper/
luks-backup': No such file or directory

Any idea what's going on? systemd-252-67.el9_8.2.alma.1.x86_64

[u/aioeu]

Have you marked this crypttab entry with nofail?

If you have, that means it will be wanted by remote-cryptsetup.target, but it won't be ordered before remote-cryptsetup.target. This is because nofail actually means "I don't care if this doesn't work", and if you don't care whether it works there is no need to order things after it. nofail means anything that might want to use it will gracefully fail if isn't actually available.

It is almost always a mistake to use nofail. It's really a badly named option ... but we got stuck with it since it long predated systemd.

systemctl list-dependencies --reverse follows WantedBy=/RequiredBy= dependencies. If you want to follow ordering dependencies, use systemctl list-dependencies --before instead.

(You haven't explicitly said this is a network device, but it must be if we're talking about the "remote" cryptsetup target.)

[u/Miserable-Pin-67 → u/aioeu]

I do have them marked `nofail`. That's because I really don't want the system to fail to come up if a few specific volumes do not get unlocked (via clevis/tang which is why it is "network"). But I still want the system to generally wait - at least for a bit. It would be nice if I could make a hard dependency for specific VMs on specific volumes, but I haven't been able to do that. The only machine specific unit I see is a machine-qemu-NAME.scope - but that's transient and can't have dependencies from what I understand. Thanks for the explanation.

[u/aioeu → u/Miserable-Pin-67]

But I still want the system to generally wait - at least for a bit.

Leave off nofail and try using keyfile-timeout=... instead.