system/systemd
system/systemd
1
0
Fork 0
mirror of https://github.com/systemd/systemd synced 2024-07-21 18:24:38 +00:00
Code Issues Actions 6 Packages Projects Releases Wiki Activity

docs/JOURNAL_FILE_FORMAT: format text

Browse source
This commit is contained in:
hulkoba 2024-03-05 11:45:58 +01:00
parent e73a7d03a9
commit a738c6d9b8
No known key found for this signature in database
GPG key ID: ACB6C4A3A4F2BE2F
1 changed files with 37 additions and 39 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

76
docs/JOURNAL_FILE_FORMAT.md
View file

@ -9,7 +9,8 @@ SPDX-License-Identifier: LGPL-2.1-or-later
_Note that this document describes the binary on-disk format of journals only.
For interfacing with web technologies there's the [Journal JSON Format](JOURNAL_EXPORT_FORMATS#journal-json-format).
For transfer of journal data across the network there's the [Journal Export Format](JOURNAL_EXPORT_FORMATS#journal-export-format)._
For transfer of journal data across the network there's the
[Journal Export Format](JOURNAL_EXPORT_FORMATS#journal-export-format)._
The systemd journal stores log data in a binary format with several features:
@ -20,65 +21,62 @@ The systemd journal stores log data in a binary format with several features:
* Support for in-line compression
* Support for in-line Forward Secure Sealing
This document explains the basic structure of the file format on disk. We are
making this available primarily to allow review and provide documentation. Note
that the actual implementation in the [systemd
codebase](https://github.com/systemd/systemd/blob/main/src/libsystemd/sd-journal/) is the
only ultimately authoritative description of the format, so if this document
and the code disagree, the code is right. That said we'll of course try hard to
keep this document up-to-date and accurate.
This document explains the basic structure of the file format on disk.
We are making this available primarily to allow review and provide documentation.
Note that the actual implementation in the
[systemd codebase](https://github.com/systemd/systemd/blob/main/src/libsystemd/sd-journal/)
is the only ultimately authoritative description of the format,
so if this document and the code disagree, the code is right.
That said we'll of course try hard to keep this document up-to-date and accurate.
Instead of implementing your own reader or writer for journal files we ask you
to use the [Journal's native C
API](https://www.freedesktop.org/software/systemd/man/sd-journal.html) to access
these files. It provides you with full access to the files, and will not
withhold any data. If you find a limitation, please ping us and we might add
some additional interfaces for you.
Instead of implementing your own reader or writer for journal files we ask you to use the
[Journal's native CAPI](https://www.freedesktop.org/software/systemd/man/sd-journal.html)
to access these files.
It provides you with full access to the files, and will not withhold any data.
If you find a limitation, please ping us and we might add some additional interfaces for you.
If you need access to the raw journal data in serialized stream form without C
API our recommendation is to make use of the [Journal Export
Format](https://systemd.io/JOURNAL_EXPORT_FORMATS#journal-export-format), which you can
get via `journalctl -o export` or via `systemd-journal-gatewayd`. The export
format is much simpler to parse, but complete and accurate. Due to its
stream-based nature it is not indexed.
If you need access to the raw journal data in serialized stream form without C API our recommendation is to make use of the
[Journal Export Format](JOURNAL_EXPORT_FORMATS#journal-export-format),
which you can get via `journalctl -o export` or via `systemd-journal-gatewayd`.
The export format is much simpler to parse, but complete and accurate.
Due to its stream-based nature it is not indexed.
_Or, to put this in other words: this low-level document is probably not what
you want to use as base of your project. You want our [C
API](https://www.freedesktop.org/software/systemd/man/sd-journal.html) instead!
_Or, to put this in other words: this low-level document is probably not what you want to use as base of your project.
You want our [C API](https://www.freedesktop.org/software/systemd/man/sd-journal.html) instead!
And if you really don't want the C API, then you want the
[Journal Export Format or Journal JSON Format](JOURNAL_EXPORT_FORMATS) instead!
This document is primarily for your entertainment and education.
Thank you!_
This document assumes you have a basic understanding of the journal concepts,
the properties of a journal entry and so on. If not, please go and read up,
then come back! This is a good opportunity to read about the [basic properties
of journal
entries](https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html),
in particular realize that they may include binary non-text data (though
usually don't), and the same field might have multiple values assigned within
the same entry.
This document assumes you have a basic understanding of the journal concepts, the properties of a journal entry and so on.
If not, please go and read up, then come back!
This is a good opportunity to read about the
[basic properties of journal entries](https://www.freedesktop.org/software/systemd/man/systemd.journal-fields.html),
in particular realize that they may include binary non-text data (though usually don't),
and the same field might have multiple values assigned within the same entry.
This document describes the current format of systemd 246. The documented
format is compatible with the format used in the first versions of the journal,
This document describes the current format of systemd 246.
The documented format is compatible with the format used in the first versions of the journal,
but received various compatible and incompatible additions since.
If you are wondering why the journal file format has been created in the first
place instead of adopting an existing database implementation, please have a
look [at this
thread](https://lists.freedesktop.org/archives/systemd-devel/2012-October/007054.html).
If you are wondering why the journal file format has been created in the first place instead of adopting an existing database implementation,
please have a look [at this thread](https://lists.freedesktop.org/archives/systemd-devel/2012-October/007054.html).
## Basics
* All offsets, sizes, time values, hashes (and most other numeric values) are 32-bit/64-bit unsigned integers in LE format.
* Offsets are always relative to the beginning of the file.
* The 64-bit hash function siphash24 is used for newer journal files. For older files [Jenkins lookup3](https://en.wikipedia.org/wiki/Jenkins_hash_function) is used, more specifically `jenkins_hashlittle2()` with the first 32-bit integer it returns as higher 32-bit part of the 64-bit value, and the second one uses as lower 32-bit part.
* The 64-bit hash function siphash24 is used for newer journal files.
For older files [Jenkins lookup3](https://en.wikipedia.org/wiki/Jenkins_hash_function) is used,
more specifically `jenkins_hashlittle2()` with the first 32-bit integer it returns as higher 32-bit part of the 64-bit value,
and the second one uses as lower 32-bit part.
* All structures are aligned to 64-bit boundaries and padded to multiples of 64-bit
* The format is designed to be read and written via memory mapping using multiple mapped windows.
* All time values are stored in usec since the respective epoch.
* Wall clock time values are relative to the Unix time epoch, i.e. January 1st, 1970. (`CLOCK_REALTIME`)
* Monotonic time values are always stored jointly with the kernel boot ID value (i.e. `/proc/sys/kernel/random/boot_id`) they belong to. They tend to be relative to the start of the boot, but aren't for containers. (`CLOCK_MONOTONIC`)
* Monotonic time values are always stored jointly with the kernel boot ID value (i.e. `/proc/sys/kernel/random/boot_id`) they belong to.
They tend to be relative to the start of the boot, but aren't for containers. (`CLOCK_MONOTONIC`)
* Randomized, unique 128-bit IDs are used in various locations. These are generally UUID v4 compatible, but this is not a requirement.
## General Rules