mirror of
https://github.com/systemd/systemd
synced 2024-09-06 16:56:43 +00:00
![Frantisek Sumsal](/assets/img/avatar_default.png)
Since 56b2970 has proven to be a no-go for us, as it breaks existing links, let's embrace the trailing slash and use absolute links everywhere for our pages. This way we'll get around browser cleverly appending the relative link to the current location (since it ends with a slash), and given our docs/ layout is flat it's not much of a hassle either. Converted using this beauty: $ sed -ri 's/(\[.+\]\()([A-Z_]+\))/\1\/\2/g' *.md Resolves: #32088 (again) and #32310
195 lines
9.9 KiB
Markdown
195 lines
9.9 KiB
Markdown
---
|
|
title: Storage Daemons for the Root File System
|
|
category: Interfaces
|
|
layout: default
|
|
SPDX-License-Identifier: LGPL-2.1-or-later
|
|
---
|
|
|
|
# systemd and Storage Daemons for the Root File System
|
|
|
|
a.k.a. _Pax Cellae pro Radix Arbor_
|
|
|
|
(or something like that, my Latin is a bit rusty)
|
|
|
|
A number of complex storage technologies on Linux (e.g. RAID, volume
|
|
management, networked storage) require user space services to run while the
|
|
storage is active and mountable. This requirement becomes tricky as soon as the
|
|
root file system of the Linux operating system is stored on such storage
|
|
technology. Previously no clear path to make this work was available. This text
|
|
tries to clear up the resulting confusion, and what is now supported and what
|
|
is not.
|
|
|
|
## A Bit of Background
|
|
|
|
When complex storage technologies are used as backing for the root file system
|
|
this needs to be set up by the initrd, i.e. on Fedora by Dracut. In newer
|
|
systemd versions tear-down of the root file system backing is also done by the
|
|
initrd: after terminating all remaining running processes and unmounting all
|
|
file systems it can (which means excluding the root file system) systemd will
|
|
jump back into the initrd code allowing it to unmount the final file systems
|
|
(and its storage backing) that could not be unmounted as long as the OS was
|
|
still running from the main root file system. The job of the initrd is to
|
|
detach/unmount the root file system, i.e. inverting the exact commands it used
|
|
to set them up in the first place. This is not only cleaner, but also allows
|
|
for the first time arbitrary complex stacks of storage technology.
|
|
|
|
Previous attempts to handle root file system setups with complex storage as
|
|
backing usually tried to maintain the root storage with program code stored on
|
|
the root storage itself, thus creating a number of dependency loops. Safely
|
|
detaching such a root file system becomes messy, since the program code on the
|
|
storage needs to stay around longer than the storage, which is technically
|
|
contradicting.
|
|
|
|
## What's new?
|
|
|
|
As a result, we hereby clarify that we do not support storage technology setups
|
|
where the storage daemons are being run from the storage they maintain
|
|
themselves. In other words: a storage daemon backing the root file system cannot
|
|
be stored on the root file system itself.
|
|
|
|
What we do support instead is that these storage daemons are started from the
|
|
initrd, stay running all the time during normal operation and are terminated
|
|
only after we returned control back to the initrd and by the initrd. As such,
|
|
storage daemons involved with maintaining the root file system storage
|
|
conceptually are more like kernel threads than like normal system services:
|
|
from the perspective of the init system (i.e. systemd), these services have been
|
|
started before systemd was initialized and stay around until after systemd is
|
|
already gone. These daemons can only be updated by updating the initrd and
|
|
rebooting; a takeover from initrd-supplied services to replacements from the
|
|
root file system is not supported.
|
|
|
|
## What does this mean?
|
|
|
|
Near the end of system shutdown, systemd executes a small tool called
|
|
systemd-shutdown, replacing its own process. This tool (which runs as PID 1, as
|
|
it entirely replaces the systemd init process) then iterates through the
|
|
mounted file systems and running processes (as well as a couple of other
|
|
resources) and tries to unmount/read-only mount/detach/kill them. It continues
|
|
to do this in a tight loop as long as this results in any effect. From this
|
|
killing spree a couple of processes are automatically excluded: PID 1 itself of
|
|
course, as well as all kernel threads. After the killing/unmounting spree
|
|
control is passed back to the initrd, whose job is then to unmount/detach
|
|
whatever might be remaining.
|
|
|
|
The same killing spree logic (but not the unmount/detach/read-only logic) is
|
|
applied during the transition from the initrd to the main system (i.e. the
|
|
"`switch_root`" operation), so that no processes from the initrd survive to the
|
|
main system.
|
|
|
|
To implement the supported logic proposed above (i.e. where storage daemons
|
|
needed for the root file system which are started by the initrd stay around
|
|
during normal operation and are only killed after control is passed back to the
|
|
initrd), we need to exclude these daemons from the shutdown/switch_root killing
|
|
spree. To accomplish this, the following logic is available starting with
|
|
systemd 38:
|
|
|
|
Processes (run by the root user) whose first character of the zeroth command
|
|
line argument is `@` are excluded from the killing spree, much the same way as
|
|
kernel threads are excluded too. Thus, a daemon which wants to take advantage
|
|
of this logic needs to place the following at the top of its `main()` function:
|
|
|
|
```c
|
|
...
|
|
argv[0][0] = '@';
|
|
...
|
|
```
|
|
|
|
And that's already it. Note that this functionality is only to be used by
|
|
programs running from the initrd, and **not** for programs running from the
|
|
root file system itself. Programs which use this functionality and are running
|
|
from the root file system are considered buggy since they effectively prohibit
|
|
clean unmounting/detaching of the root file system and its backing storage.
|
|
|
|
_Again: if your code is being run from the root file system, then this logic
|
|
suggested above is **NOT** for you. Sorry. Talk to us, we can probably help you
|
|
to find a different solution to your problem._
|
|
|
|
The recommended way to distinguish between run-from-initrd and run-from-rootfs
|
|
for a daemon is to check for `/etc/initrd-release` (which exists on all modern
|
|
initrd implementations, see the [initrd Interface](/INITRD_INTERFACE) for
|
|
details) which when exists results in `argv[0][0]` being set to `@`, and
|
|
otherwise doesn't. Something like this:
|
|
|
|
```c
|
|
#include <unistd.h>
|
|
|
|
int main(int argc, char *argv[]) {
|
|
...
|
|
if (access("/etc/initrd-release", F_OK) >= 0)
|
|
argv[0][0] = '@';
|
|
...
|
|
}
|
|
```
|
|
|
|
Why `@`? Why `argv[0][0]`? First of all, a technique like this is not without
|
|
precedent: traditionally Unix login shells set `argv[0][0]` to `-` to clarify
|
|
they are login shells. This logic is also very easy to implement. We have been
|
|
looking for other ways to mark processes for exclusion from the killing spree,
|
|
but could not find any that was equally simple to implement and quick to read
|
|
when traversing through `/proc/`. Also, as a side effect replacing the first
|
|
character of `argv[0]` with `@` also visually invalidates the path normally
|
|
stored in `argv[0]` (which usually starts with `/`) thus helping the
|
|
administrator to understand that your daemon is actually not originating from
|
|
the actual root file system, but from a path in a completely different
|
|
namespace (i.e. the initrd namespace). Other than that we just think that `@`
|
|
is a cool character which looks pretty in the ps output... 😎
|
|
|
|
Note that your code should only modify `argv[0][0]` and leave the comm name
|
|
(i.e. `/proc/self/comm`) of your process untouched.
|
|
|
|
Since systemd v255, alternatively the `SurviveFinalKillSignal=yes` unit option
|
|
can be set, and provides the equivalent functionality to modifying `argv[0][0]`.
|
|
|
|
## To which technologies does this apply?
|
|
|
|
These recommendations apply to those storage daemons which need to stay around
|
|
until after the storage they maintain is unmounted. If your storage daemon is
|
|
fine with being shut down before its storage device is unmounted, you may ignore
|
|
the recommendations above.
|
|
|
|
This all applies to storage technology only, not to daemons with any other
|
|
(non-storage related) purposes.
|
|
|
|
## What else to keep in mind?
|
|
|
|
If your daemon implements the logic pointed out above, it should work nicely
|
|
from initrd environments. In many cases it might be necessary to additionally
|
|
support storage daemons to be started from within the actual OS, for example
|
|
when complex storage setups are used for auxiliary file systems, i.e. not the
|
|
root file system, or created by the administrator during runtime. Here are a
|
|
few additional notes for supporting these setups:
|
|
|
|
* If your storage daemon is run from the main OS (i.e. not the initrd) it will
|
|
also be terminated when the OS shuts down (i.e. before we pass control back
|
|
to the initrd). Your daemon needs to handle this properly.
|
|
|
|
* It is not acceptable to spawn off background processes transparently from
|
|
user commands or udev rules. Whenever a process is forked off on Unix it
|
|
inherits a multitude of process attributes (ranging from the obvious to the
|
|
not-so-obvious such as security contexts or audit trails) from its parent
|
|
process. It is practically impossible to fully detach a service from the
|
|
process context of the spawning process. In particular, systemd tracks which
|
|
processes belong to a service or login sessions very closely, and by spawning
|
|
off your storage daemon from udev or an administrator command you thus make
|
|
it part of its service/login. Effectively this means that whenever udev is
|
|
shut down, your storage daemon is killed too, resp. whenever the login
|
|
session goes away your storage might be terminated as well. (Also note that
|
|
recent udev versions will automatically kill all long running background
|
|
processes forked off udev rules now.) So, in summary: double-forking off
|
|
processes from user commands or udev rules is **NOT** OK!
|
|
|
|
* To automatically spawn storage daemons from udev rules or administrator
|
|
commands, the recommended technology is socket-based activation as
|
|
implemented by systemd. Transparently for your client code connecting to the
|
|
socket of your storage daemon will result in the storage to be started. For
|
|
that it is simply necessary to inform systemd about the socket you'd like it
|
|
to listen on behalf of your daemon and minimally modify the daemon to
|
|
receive the listening socket for its services from systemd instead of
|
|
creating it on its own. Such modifications can be minimal, and are easily
|
|
written in a way that does not negatively impact usability on non-systemd
|
|
systems. For more information on making use of socket activation in your
|
|
program consult this blog story: [Socket
|
|
Activation](https://0pointer.de/blog/projects/socket-activation.html)
|
|
|
|
* Consider having a look at the [initrd Interface of systemd](/INITRD_INTERFACE).
|