mirror of
https://github.com/systemd/systemd
synced 2024-09-06 16:56:43 +00:00
194 lines
9.8 KiB
Markdown
194 lines
9.8 KiB
Markdown
---
|
|
title: Storage Daemons for the Root File System
|
|
category: Interfaces
|
|
layout: default
|
|
---
|
|
|
|
# systemd and Storage Daemons for the Root File System
|
|
|
|
a.k.a. _Pax Cellae pro Radix Arbor_
|
|
|
|
(or something like that, my Latin is a bit rusty)
|
|
|
|
A number of complex storage technologies on Linux (e.g. RAID, volume
|
|
management, networked storage) require user space services to run while the
|
|
storage is active and mountable. This requirement becomes tricky as soon as the
|
|
root file system of the Linux operating system is stored on such storage
|
|
technology. Previously no clear path to make this work was available. This text
|
|
tries to clear up the resulting confusion, and what is now supported and what
|
|
is not.
|
|
|
|
## A Bit of Background
|
|
|
|
When complex storage technologies are used as backing for the root file system
|
|
this needs to be set up by the initial RAM file system (initrd), i.e. on Fedora
|
|
by Dracut. In newer systemd versions tear-down of the root file system backing
|
|
is also done by the initrd: after terminating all remaining running processes
|
|
and unmounting all file systems it can (which means excluding the root fs)
|
|
systemd will jump back into the initrd code allowing it to unmount the final
|
|
file systems (and its storage backing) that could not be unmounted as long as
|
|
the OS was still running from the main root file system. The initrd' job is to
|
|
detach/unmount the root fs, i.e. inverting the exact commands it used to set
|
|
them up in the first place. This is not only cleaner, but also allows for the
|
|
first time arbitrary complex stacks of storage technology.
|
|
|
|
Previous attempts to handle root file system setups with complex storage as
|
|
backing usually tried to maintain the root storage with program code stored on
|
|
the root storage itself, thus creating a number of dependency loops. Safely
|
|
detaching such a root file system becomes messy, since the program code on the
|
|
storage needs to stay around longer than the storage, which is technically
|
|
contradicting.
|
|
|
|
|
|
## What's new?
|
|
|
|
As a result, we hereby clarify that we do not support storage technology setups
|
|
where the storage daemons are being run from the storage it maintains
|
|
itself. In other words: a storage daemon backing the root file system cannot be
|
|
stored on the root file system itself.
|
|
|
|
What we do support instead is that these storage daemons are started from the
|
|
initrd, stay running all the time during normal operation and are terminated
|
|
only after we returned control back to the initrd and by the initrd. As such,
|
|
storage daemons involved with maintaining the root file system storage
|
|
conceptually are more like kernel threads than like normal system services:
|
|
from the perspective of the init system (i.e. systemd) these services have been
|
|
started before systemd got initialized and stay around until after systemd is
|
|
already gone. These daemons can only be updated by updating the initrd and
|
|
rebooting, a takeover from initrd-supplied services to replacements from the
|
|
root file system is not supported.
|
|
|
|
|
|
## What does this mean?
|
|
|
|
Near the end of system shutdown, systemd executes a small tool called
|
|
systemd-shutdown, replacing its own process. This tool (which runs as PID 1, as
|
|
it entirely replaces the systemd init process) then iterates through the
|
|
mounted file systems and running processes (as well as a couple of other
|
|
resources) and tries to unmount/read-only mount/detach/kill them. It continues
|
|
to do this in a tight loop as long as this results in any effect. From this
|
|
killing spree a couple of processes are automatically excluded: PID 1 itself of
|
|
course, as well as all kernel threads. After the killing/unmounting spree
|
|
control is passed back to the initrd, whose job is then to unmount/detach
|
|
whatever might be remaining.
|
|
|
|
The same killing spree logic (but not the unmount/detach/read-only logic) is
|
|
applied during the transition from the initrd to the main system (i.e. the
|
|
"`switch_root`" operation), so that no processes from the initrd survive to the
|
|
main system.
|
|
|
|
To implement the supported logic proposed above (i.e. where storage daemons
|
|
needed for the root fs which are started by the initrd stay around during
|
|
normal operation and are only killed after control is passed back to the
|
|
initrd) we need to exclude these daemons from the shutdown/switch_root killing
|
|
spree. To accomplish this the following logic is available starting with
|
|
systemd 38:
|
|
|
|
Processes (run by the root user) whose first character of the zeroth command
|
|
line argument is `@` are excluded from the killing spree, much the same way as
|
|
kernel threads are excluded too. Thus, a daemon which wants to take advantage
|
|
of this logic needs to place the following at the top of its `main()` function:
|
|
|
|
```c
|
|
...
|
|
argv[0][0] = '@';
|
|
...
|
|
```
|
|
|
|
And that's already it. Note that this functionality is only to be used by
|
|
programs running from the initrd, and **not** for programs running from the
|
|
root file system itself. Programs which use this functionality and are running
|
|
from the root file system are considered buggy since they effectively prohibit
|
|
clean unmounting/detaching of the root file system and its backing storage.
|
|
|
|
_Again: if your code is being run from the root file system, then this logic
|
|
suggested above is **NOT** for you. Sorry. Talk to us, we can probably help you
|
|
to find a different solution to your problem._
|
|
|
|
The recommended way to distinguish between run-from-initrd and run-from-rootfs
|
|
for a daemon is to check for `/etc/initrd-release` (which exists on all modern
|
|
initrd implementations, see the [initrd
|
|
Interface](https://systemd.io/INITRD_INTERFACE/) for details) which when exists
|
|
results in `argv[0][0]` being set to `@`, and otherwise doesn't. Something like
|
|
this:
|
|
|
|
```c
|
|
#include <unistd.h>
|
|
|
|
int main(int argc, char *argv[]) {
|
|
...
|
|
if (access("/etc/initrd-release", F_OK) >= 0)
|
|
argv[0][0] = '@';
|
|
...
|
|
}
|
|
```
|
|
|
|
Why `@`? Why `argv[0][0]`? First of all, a technique like this is not without
|
|
precedent: traditionally Unix login shells set `argv[0][0]` to `-` to clarify
|
|
they are login shells. This logic is also very easy to implement. We have been
|
|
looking for other ways to mark processes for exclusion from the killing spree,
|
|
but could not find any that was equally simple to implement and quick to read
|
|
when traversing through `/proc/`. Also, as a side effect replacing the first
|
|
character of `argv[0]` with `@` also visually invalidates the path normally
|
|
stored in `argv[0]` (which usually starts with `/`) thus helping the
|
|
administrator to understand that your daemon is actually not originating from
|
|
the actual root file system, but from a path in a completely different
|
|
namespace (i.e. the initrd namespace). Other than that we just think that `@`
|
|
is a cool character which looks pretty in the ps output... 😎
|
|
|
|
Note that your code should only modify `argv[0][0]` and leave the comm name
|
|
(i.e. `/proc/self/comm`) of your process untouched.
|
|
|
|
## To which technologies does this apply?
|
|
|
|
These recommendations apply to those storage daemons which need to stay around
|
|
until after the storage they maintain is unmounted. If your storage daemon is
|
|
fine with being shut down before its storage device is unmounted you may ignore
|
|
the recommendations above.
|
|
|
|
This all applies to storage technology only, not to daemons with any other
|
|
(non-storage related) purposes.
|
|
|
|
## What else to keep in mind?
|
|
|
|
If your daemon implements the logic pointed out above it should work nicely
|
|
from initrd environments. In many cases it might be necessary to additionally
|
|
support storage daemons to be started from within the actual OS, for example
|
|
when complex storage setups are used for auxiliary file systems, i.e. not the
|
|
root file system, or created by the administrator during runtime. Here are a
|
|
few additional notes for supporting these setups:
|
|
|
|
* If your storage daemon is run from the main OS (i.e. not the initrd) it will
|
|
also be terminated when the OS shuts down (i.e. before we pass control back
|
|
to the initrd). Your daemon needs to handle this properly.
|
|
|
|
* It is not acceptable to spawn off background processes transparently from
|
|
user commands or udev rules. Whenever a process is forked off on Unix it
|
|
inherits a multitude of process attributes (ranging from the obvious to the
|
|
not-so-obvious such as security contexts or audit trails) from its parent
|
|
process. It is practically impossible to fully detach a service from the
|
|
process context of the spawning process. In particular, systemd tracks which
|
|
processes belong to a service or login sessions very closely, and by spawning
|
|
off your storage daemon from udev or an administrator command you thus make
|
|
it part of its service/login. Effectively this means that whenever udev is
|
|
shut down, your storage daemon is killed too, resp. whenever the login
|
|
session goes away your storage might be terminated as well. (Also note that
|
|
recent udev versions will automatically kill all long running background
|
|
processes forked off udev rules now.) So, in summary: double-forking off
|
|
processes from user commands or udev rules is **NOT** OK!
|
|
|
|
* To automatically spawn storage daemons from udev rules or administrator
|
|
commands, the recommended technology is socket-based activation as
|
|
implemented by systemd. Transparently for your client code connecting to the
|
|
socket of your storage daemon will result in the storage to be started. For
|
|
that it is simply necessary to inform systemd about the socket you'd like it
|
|
to listen on on behalf of your daemon and minimally modify the daemon to
|
|
receive the listening socket for its services from systemd instead of
|
|
creating it on its own. Such modifications can be minimal, and are easily
|
|
written in a way that does not negatively impact usability on non-systemd
|
|
systems. For more information on making use of socket activation in your
|
|
program consult this blog story: [Socket
|
|
Activation](http://0pointer.de/blog/projects/socket-activation.html)
|
|
|
|
* Consider having a look at the [initrd Interface of systemd](https://systemd.io/INITRD_INTERFACE/).
|