Even with Storage=journal we would still attempt to open the final
dmesg.txt file which causes a lot of noise in the journal:
```
[ 5.764111] H testsuite-82.sh[658]: + systemctl start systemd-pstore
[ 5.806385] H systemd[1]: Starting modprobe@efi_pstore.service...
[ 5.808656] H systemd[1]: modprobe@efi_pstore.service: Deactivated successfully.
[ 5.808971] H systemd[1]: Finished modprobe@efi_pstore.service.
[ 5.818845] H kernel: audit: type=1130 audit(1682630623.637:114): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=modprobe@efi_pstore comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? termin>
[ 5.818865] H kernel: audit: type=1131 audit(1682630623.637:115): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=modprobe@efi_pstore comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? termin>
[ 5.816052] H systemd[1]: Starting systemd-pstore.service...
[ 5.840703] H systemd-pstore[806]: PStore dmesg-efi-168263062313014.
[ 5.841239] H systemd-pstore[806]: Failed to open file /var/lib/systemd/pstore/1682630623/014/dmesg.txt: Operation not permitted
[ 5.841428] H systemd-pstore[806]: PStore dmesg-efi-168263062312014.
[ 5.841575] H systemd-pstore[806]: Failed to open file /var/lib/systemd/pstore/1682630623/014/dmesg.txt: Operation not permitted
[ 5.841712] H systemd-pstore[806]: PStore dmesg-efi-168263062311014.
[ 5.841839] H systemd-pstore[806]: Failed to open file /var/lib/systemd/pstore/1682630623/014/dmesg.txt: Operation not permitted
[ 5.841989] H systemd-pstore[806]: PStore dmesg-efi-168263062310014.
[ 5.842141] H systemd-pstore[806]: Failed to open file /var/lib/systemd/pstore/1682630623/014/dmesg.txt: Operation not permitted
[ 5.842274] H systemd-pstore[806]: PStore dmesg-efi-168263062309014.
[ 5.842423] H systemd-pstore[806]: Failed to open file /var/lib/systemd/pstore/1682630623/014/dmesg.txt: Operation not permitted
[ 5.842589] H systemd-pstore[806]: PStore dmesg-efi-168263062308014.
[ 5.842722] H systemd-pstore[806]: Failed to open file /var/lib/systemd/pstore/1682630623/014/dmesg.txt: Operation not permitted
[ 5.842865] H systemd-pstore[806]: PStore dmesg-efi-168263062307014.
[ 5.843003] H systemd-pstore[806]: Failed to open file /var/lib/systemd/pstore/1682630623/014/dmesg.txt: Operation not permitted
[ 5.843153] H systemd-pstore[806]: PStore dmesg-efi-168263062306014.
[ 5.843280] H systemd-pstore[806]: Failed to open file /var/lib/systemd/pstore/1682630623/014/dmesg.txt: Operation not permitted
[ 5.843434] H systemd-pstore[806]: PStore dmesg-efi-168263062305014.
[ 5.843570] H systemd-pstore[806]: Failed to open file /var/lib/systemd/pstore/1682630623/014/dmesg.txt: Operation not permitted
[ 5.843702] H systemd-pstore[806]: PStore dmesg-efi-168263062304014.
[ 5.843831] H systemd-pstore[806]: Failed to open file /var/lib/systemd/pstore/1682630623/014/dmesg.txt: Operation not permitted
[ 5.843958] H systemd-pstore[806]: PStore dmesg-efi-168263062303014.
[ 5.844093] H systemd-pstore[806]: Failed to open file /var/lib/systemd/pstore/1682630623/014/dmesg.txt: Operation not permitted
[ 5.844250] H systemd-pstore[806]: PStore dmesg-efi-168263062302014.
[ 5.844412] H systemd-pstore[806]: Failed to open file /var/lib/systemd/pstore/1682630623/014/dmesg.txt: Operation not permitted
[ 5.844619] H systemd-pstore[806]: PStore dmesg-efi-168263062301014.
[ 5.844781] H systemd-pstore[806]: Failed to open file /var/lib/systemd/pstore/1682630623/014/dmesg.txt: Operation not permitted
[ 5.844956] H systemd-pstore[806]: PStore dmesg-efi-168263062300014.
[ 5.845168] H systemd-pstore[806]: Failed to open file /var/lib/systemd/pstore/1682630623/014/dmesg.txt: Operation not permitted
[ 5.851101] H systemd[1]: Finished systemd-pstore.service.
```
All daemons use a similar scheme to read their main config files and theirs
drop-ins. The main config files are always stored in /etc/systemd directory and
it's easy enough to construct the name of the drop-in directories based on the
name of the main config file.
Hence the new helper does that internally, which allows to reduce and simplify
the args passed previously to config_parse_many_nulstr().
Besides the overall code simplification it results:
16 files changed, 87 insertions(+), 159 deletions(-)
it allows to identify clearly the locations in the code where configuration
files are parsed.
-1 was used everywhere, but -EBADF or -EBADFD started being used in various
places. Let's make things consistent in the new style.
Note that there are two candidates:
EBADF 9 Bad file descriptor
EBADFD 77 File descriptor in bad state
Since we're initializating the fd, we're just assigning a value that means
"no fd yet", so it's just a bad file descriptor, and the first errno fits
better. If instead we had a valid file descriptor that became invalid because
of some operation or state change, the other errno would fit better.
In some places, initialization is dropped if unnecessary.
The approach to use '''…'''.split() instead of a list of strings was initially
used when converting from automake because it allowed identical blocks of lines
to be used for both, making the conversion easier.
But over the years we have been using normal lists more and more, especially
when there were just a few filenames listed. This converts the rest.
No functional change.
We recently started making more use of malloc_usable_size() and rely on
it (see the string_erase() story). Given that we don't really support
sytems where malloc_usable_size() cannot be trusted beyond statistics
anyway, let's go fully in and rework GREEDY_REALLOC() on top of it:
instead of passing around and maintaining the currenly allocated size
everywhere, let's just derive it automatically from
malloc_usable_size().
I am mostly after this for the simplicity this brings. It also brings
minor efficiency improvements I guess, but things become so much nicer
to look at if we can avoid these allocation size variables everywhere.
Note that the malloc_usable_size() man page says relying on it wasn't
"good programming practice", but I think it does this for reasons that
don't apply here: the greedy realloc logic specifically doesn't rely on
the returned extra size, beyond the fact that it is equal or larger than
what was requested.
(This commit was supposed to be a quick patch btw, but apparently we use
the greedy realloc stuff quite a bit across the codebase, so this ends
up touching *a*lot* of code.)
Wherever we read virtual files we better should use
read_full_virtual_file(), to make sure we get a consistent response
given how weird the kernel's handling with partial read on such file
systems is.
As suggested in https://github.com/systemd/systemd/pull/11484#issuecomment-775288617.
This does not touch anything exposed in src/systemd. Changing the defines there
would be a compatibility break.
Note that tests are broken after this commit. They will be fixed in the next one.
By default, systemd installs various sample configuration files
containing commented-out defaults. Systems seeking to minimize the
number of files in /etc may wish to install directories and
configuration files that have semantic effects, but not install not
commented-out sample configuration files.
Turn install-sysconfdir into a multi-valued option, with a "no-samples"
value to skip installing sample-only configuration files.
This is useful for development where overwriting files out side
the configured prefix will affect the host as well as stateless
systems such as NixOS that don't let packages install to /etc but handle
configuration on their own.
Alternative to https://github.com/systemd/systemd/pull/17501
tested with:
$ mkdir inst build && cd build
$ meson \
-Dcreate-log-dirs=false \
-Dsysvrcnd-path=$(realpath ../inst)/etc/rc.d \
-Dsysvinit-path=$(realpath ../inst)/etc/init.d \
-Drootprefix=$(realpath ../inst) \
-Dinstall-sysconfdir=false \
--prefix=$(realpath ../inst) ..
$ ninja install
This is a follow-up for 9f83091e3c.
Instead of reading the mtime off the configuration files after reading,
let's do so before reading, but with the fd we read the data from. This
is not only cleaner (as it allows us to save one stat()), but also has
the benefit that we'll detect changes that happen while we read the
files.
This also reworks unit file drop-ins to use the common code for
determining drop-in mtime, instead of reading system clock for that.
Semmle Security Reports report:
> The problem occurs on the way realloc is being used. When a size
> bigger than the chunk that wants to be reallocated is passed, realloc
> try to malloc a bigger size, however in the case that malloc fails
> (for example, by forcing a big allocation) realloc will return NULL.
>
> According to the man page:
> "The realloc() function returns a pointer to the newly allocated
> memory, which is suitably aligned for any built-in type and may be
> different from ptr, or NULL if the request fails. If size was
> equal to 0, either NULL or a pointer suitable to be passed to free()
> is returned. If realloc() fails, the original block is left
> untouched; it is not freed or moved."
>
> The problem occurs when the memory ptr passed to the first argument of
> realloc is the same as the one used for the result, for example in
> this case:
>
> dmesg = realloc(dmesg, dmesg_size + strlen(pe->dirent.d_name) +
> strlen(":\n") + pe->content_size + 1);
>
> f8bcb81955/files/src/pstore/pstore.c?sort=name&dir=A
> SC&mode=heatmap#L300
>
> If the malloc inside that realloc fails, then the original memory
> chunk will never be free but since realloc will return NULL, the
> pointer to that memory chunk will be lost and a memory leak will
> occur.
>
> In case you are curious, this is the query we used to find this problem:
> https://lgtm.com/query/8650323308193591473/
Let's use a more standard pattern: allocate memory using greedy_realloc, and
instead of freeing it when we wrote out a chunk, let's just move the cursor
back to the beginning and reuse the memory we allocated previously.
If we fail to allocate the memory for dmesg contents, don't write the dmesg
entry, but let's still process the files to move them out of pstore.
This makes it much easier to debug the program as a normal user, since we
don't need to set up fake input under /sys/fs/pstore/.
Also, let's make the debug output a bit nicer.
This patch introduces the systemd pstore service which will archive the
contents of the Linux persistent storage filesystem, pstore, to other storage,
thus preserving the existing information contained in the pstore, and clearing
pstore storage for future error events.
Linux provides a persistent storage file system, pstore[1], that can store
error records when the kernel dies (or reboots or powers-off). These records in
turn can be referenced to debug kernel problems (currently the kernel stuffs
the tail of the dmesg, which also contains a stack backtrace, into pstore).
The pstore file system supports a variety of backends that map onto persistent
storage, such as the ACPI ERST[2, Section 18.5 Error Serialization] and UEFI
variables[3 Appendix N Common Platform Error Record]. The pstore backends
typically offer a relatively small amount of persistent storage, e.g. 64KiB,
which can quickly fill up and thus prevent subsequent kernel crashes from
recording errors. Thus there is a need to monitor and extract the pstore
contents so that future kernel problems can also record information in the
pstore.
The pstore service is independent of the kdump service. In cloud environments
specifically, host and guest filesystems are on remote filesystems (eg. iSCSI
or NFS), thus kdump relies [implicitly and/or explicitly] upon proper operation
of networking software *and* hardware *and* infrastructure. Thus it may not be
possible to capture a kernel coredump to a file since writes over the network
may not be possible.
The pstore backend, on the other hand, is completely local and provides a path
to store error records which will survive a reboot and aid in post-mortem
debugging.
Usage Notes:
This tool moves files from /sys/fs/pstore into /var/lib/systemd/pstore.
To enable kernel recording of error records into pstore, one must either pass
crash_kexec_post_notifiers[4] to the kernel command line or enable via 'echo Y
> /sys/module/kernel/parameters/crash_kexec_post_notifiers'. This option
invokes the recording of errors into pstore *before* an attempt to kexec/kdump
on a kernel crash.
Optionally, to record reboots and shutdowns in the pstore, one can either pass
the printk.always_kmsg_dump[4] to the kernel command line or enable via 'echo Y >
/sys/module/printk/parameters/always_kmsg_dump'. This option enables code on the
shutdown path to record information via pstore.
This pstore service is a oneshot service. When run, the service invokes
systemd-pstore which is a tool that performs the following:
- reads the pstore.conf configuration file
- collects the lists of files in the pstore (eg. /sys/fs/pstore)
- for certain file types (eg. dmesg) a handler is invoked
- for all other files, the file is moved from pstore
- In the case of dmesg handler, final processing occurs as such:
- files processed in reverse lexigraphical order to faciliate
reconstruction of original dmesg
- the filename is examined to determine which dmesg it is a part
- the file is appended to the reconstructed dmesg
For example, the following pstore contents:
root@vm356:~# ls -al /sys/fs/pstore
total 0
drwxr-x--- 2 root root 0 May 9 09:50 .
drwxr-xr-x 7 root root 0 May 9 09:50 ..
-r--r--r-- 1 root root 1610 May 9 09:49 dmesg-efi-155741337601001
-r--r--r-- 1 root root 1778 May 9 09:49 dmesg-efi-155741337602001
-r--r--r-- 1 root root 1726 May 9 09:49 dmesg-efi-155741337603001
-r--r--r-- 1 root root 1746 May 9 09:49 dmesg-efi-155741337604001
-r--r--r-- 1 root root 1686 May 9 09:49 dmesg-efi-155741337605001
-r--r--r-- 1 root root 1690 May 9 09:49 dmesg-efi-155741337606001
-r--r--r-- 1 root root 1775 May 9 09:49 dmesg-efi-155741337607001
-r--r--r-- 1 root root 1811 May 9 09:49 dmesg-efi-155741337608001
-r--r--r-- 1 root root 1817 May 9 09:49 dmesg-efi-155741337609001
-r--r--r-- 1 root root 1795 May 9 09:49 dmesg-efi-155741337710001
-r--r--r-- 1 root root 1770 May 9 09:49 dmesg-efi-155741337711001
-r--r--r-- 1 root root 1796 May 9 09:49 dmesg-efi-155741337712001
-r--r--r-- 1 root root 1787 May 9 09:49 dmesg-efi-155741337713001
-r--r--r-- 1 root root 1808 May 9 09:49 dmesg-efi-155741337714001
-r--r--r-- 1 root root 1754 May 9 09:49 dmesg-efi-155741337715001
results in the following:
root@vm356:~# ls -al /var/lib/systemd/pstore/155741337/
total 92
drwxr-xr-x 2 root root 4096 May 9 09:50 .
drwxr-xr-x 4 root root 40 May 9 09:50 ..
-rw-r--r-- 1 root root 1610 May 9 09:50 dmesg-efi-155741337601001
-rw-r--r-- 1 root root 1778 May 9 09:50 dmesg-efi-155741337602001
-rw-r--r-- 1 root root 1726 May 9 09:50 dmesg-efi-155741337603001
-rw-r--r-- 1 root root 1746 May 9 09:50 dmesg-efi-155741337604001
-rw-r--r-- 1 root root 1686 May 9 09:50 dmesg-efi-155741337605001
-rw-r--r-- 1 root root 1690 May 9 09:50 dmesg-efi-155741337606001
-rw-r--r-- 1 root root 1775 May 9 09:50 dmesg-efi-155741337607001
-rw-r--r-- 1 root root 1811 May 9 09:50 dmesg-efi-155741337608001
-rw-r--r-- 1 root root 1817 May 9 09:50 dmesg-efi-155741337609001
-rw-r--r-- 1 root root 1795 May 9 09:50 dmesg-efi-155741337710001
-rw-r--r-- 1 root root 1770 May 9 09:50 dmesg-efi-155741337711001
-rw-r--r-- 1 root root 1796 May 9 09:50 dmesg-efi-155741337712001
-rw-r--r-- 1 root root 1787 May 9 09:50 dmesg-efi-155741337713001
-rw-r--r-- 1 root root 1808 May 9 09:50 dmesg-efi-155741337714001
-rw-r--r-- 1 root root 1754 May 9 09:50 dmesg-efi-155741337715001
-rw-r--r-- 1 root root 26754 May 9 09:50 dmesg.txt
where dmesg.txt is reconstructed from the group of related
dmesg-efi-155741337* files.
Configuration file:
The pstore.conf configuration file has four settings, described below.
- Storage : one of "none", "external", or "journal". With "none", this
tool leaves the contents of pstore untouched. With "external", the
contents of the pstore are moved into the /var/lib/systemd/pstore,
as well as logged into the journal. With "journal", the contents of
the pstore are recorded only in the systemd journal. The default is
"external".
- Unlink : is a boolean. When "true", the default, then files in the
pstore are removed once processed. When "false", processing of the
pstore occurs normally, but the pstore files remain.
References:
[1] "Persistent storage for a kernel's dying breath",
March 23, 2011.
https://lwn.net/Articles/434821/
[2] "Advanced Configuration and Power Interface Specification",
version 6.2, May 2017.
https://www.uefi.org/sites/default/files/resources/ACPI_6_2.pdf
[3] "Unified Extensible Firmware Interface Specification",
version 2.8, March 2019.
https://uefi.org/sites/default/files/resources/UEFI_Spec_2_8_final.pdf
[4] "The kernel’s command-line parameters",
https://static.lwn.net/kerneldoc/admin-guide/kernel-parameters.html
