The PE header size calculation failed to take the PE magic and coff
header size into account, which will lead to header truncation if we are
writing only 5 sections.
CentOS 8 ships python 3.6 so let's try and stay compatible with that
since the only feature we're using that requires python 3.9 is the
streamlined type annotations which are trivial to convert back to
the older stuff to stay compatible with python 3.6.
The entries are sorted by speed. Some fields are left empty when there is no
clear value to use. The table is much longer now, but I think it's better to
document the allowed values, even if some are not terribly useful.
Fixes#26256.
At least shim will choke on an empty relocation section when loading the
binary. Note that the binary is still considered relocatable (just with
no base relocations to apply) as we do not set the
IMAGE_FILE_RELOCS_STRIPPED DLL characteristic.
Instead of /home/zbyszek/src/systemd-work/build/../src/xdg-autostart-generator/xdg-autostart-service.h:11,
print just src/xdg-autostart-generator/xdg-autostart-service.h:11.
This is a bit annoying that this requires so much verbosity, but the output
with the full names was too annoying.
This is a little helper I used when preparing the tests for auxv
parsing. Just looking at hexdump output is pretty hard. We could
enhance it to display some specific data types better.
This adds back sd-boot builds by using meson compile targets directly.
We can do this now, because userspace binaries use the special
dependency that allows us to easily separate flags, so that we don't
pass anything to EFI builds that shouldn't be passed.
Additionally, we pass a bunch of flags to hopefully disable/override any
distro provided flags that should not be used for EFI binaries.
Fixes: #12275
This drops all mentions of gnu-efi and its manual build machinery. A
future commit will bring bootloader builds back. A new bootloader meson
option is now used to control whether to build sd-boot and its userspace
tooling.
In 6abe882bae the renderer was made to
unconditionally append a newline to output. This works, but is ugly. A nicer
solution is to tell jinja2 to not strip the newline in the first place, via
keep_trailing_newline=True. It seems that the result is unchanged because all
our source files have exactly one trailing newline.
Also, enable lstrip_blocks=True. This would cause whitespace on the line before
an {%if block to be automatically stripped. It seems reasonable to enable that
if trim_blocks=True.
Overall, no change is expected, though I didn't test combinations of
configurations, so there might be a change in some cases. But now the rules of
rendering are more logical, e.g. we should be able to indent nested conditional
statements without getting unexpected whitespace in the output.
(The one case that is left unchanged is '< <(subcommand)'.)
This way, the style with no gap was already dominant. This way, the reader
immediately knows that ' < ' is a comparison operator and ' << ' is a shift.
In a few cases, replace custom EOF replacement by just EOF. There is no point
in using someting like "_EOL" unless "EOF" appears in the text.
Since the files with generated directives are now automatically
generated during build, they're now under the respective build directory
which the current oss-fuzz CI script didn't account for.
Follow-up to: #24958Resolves: #25859
We generate a "version string" that is reported by various tools. This patch
changes this version string to use the characters specified for the version
string in the Boot Loader Specification. We start using the special characters
we have in the spec for this exact purpose and thus fix version comparisons.
We also stop using '+' which is not part of the allowed charset and is used for
boot attempt counting and should not be part of the version string.
The version string is (among other places) used in sd-boot and the comparison
result is used by 'bootctl update' to decide whether to install a new binary.
Before, because 'nn-rc1' compares higher than 'nn', we would refuse to upgrade
pre-release versions.
The boot loader is the primary motivation. I'm not aware of programatic version
comparisons in other places, but it makes sense to use the same versions string
everywhere.
(This patch effectively only matters for non-distro builds, because distro
builds presumably use -Dversion-tag to set something meaningful. Ideally, those
version strings are compatible with our version strings, but this is outside of
our control.)
The lists of directives for fuzzer tests are maintained manually in the
repo. There is a tools/check-directives.sh script that runs during test
phase and reports stale directive lists.
Let's rework the script into a generator so that these directive files
are created on-the-flight and needn't be updated whenever a unit file
directives change. The scripts is rewritten in Python to get rid of gawk
dependency and each generated file is a separate meson target so that
incremental builds refresh what is just necessary (and parallelize
(negligible)).
Note: test/fuzz/fuzz-unit-file/directives-all.slice is kept since there
is not automated way to generate it (it is not covered by the check
script neither).
We're already using C.UTF-8 as the default locale for nspawn. Let's
make the same change for the default-locale option instead of deciding
what to use based on the locale used by the host system. Users can
still override the locale using the default-locale option if needed.
No need to involve a trivial shell script for this.
We could call the compiler directly, but test() expects arguments
to be passed separately and cc.cmd_array() can contain arguments
itself. Using env is easier than manually slicing the array because
meson has no builtins for that.
Bash will generate a very nice message for us:
/tmp/ff.sh: line 1: SOMEVAR: parameter null or not set
Let's save some keystrokes by not replacing this with our own inferior
messages.
GIT_VERSION is not available as a config.h variable, because it's rendered
into version.h during builds. Let's rework jinja2 rendering to also
parse version.h. No functional change, the new variable is so far unused.
I guess this will make partial rebuilds a bit slower, but it's useful
to be able to use the full version string.
This is very similar to (and directly based on) the test for --help. I think
it's nice to do this: the test is very quick, but it'll catch cases where we
forgot to hook up the option, or forgot to exit after printing --version, and
it'll also increase our test coverage a bit.
It seems that --invert-grep used to affect --author, but now it doesn't (with
git-2.35.1-1.fc36.x86_64), so effectively we would only show the one entry that
was supposed to be filtered out.
When we do mkdir, we should just use 0o777 and let the umask take care of the
rest. Specifying an explicit mode is inappropriate. And when touching the code,
let's replace black madness with normal python style.
It's like CIFuzz but unlike CIFuzz it's compatible with forks and
it should make it possible to run the fuzzers to make sure that
patches backported to them are backported correctly without introducing
new bugs and regressions.
It was copy-pasted directly from OSS-Fuzz where it makes sense to
kind of strip binaries to get nice backtraces but when the fuzzers
are built and run locally with gdb it would be nice to have a little
bit more than that.
It was initially discovered in elfutils where I put the same flags
and was surprised when I couldn't run the fuzzer comfortably step
by step, which led to the same change there: https://github.com/google/oss-fuzz/pull/7092
:-)
The scheme is very similar to libsystemd-shared.so: instead of building a
static library, we build a shared library from the same objects and link the
two users to it. Both systemd and systemd-analyze consist mostly of the fairly
big code in libcore, so we save a bit on the installation:
(-0g, no strip)
-rwxr-xr-x 5238864 Dec 14 12:52 /var/tmp/inst1/usr/lib/systemd/systemd
-rwxr-xr-x 5399600 Dec 14 12:52 /var/tmp/inst1/usr/bin/systemd-analyze
-rwxr-xr-x 244912 Dec 14 13:17 /var/tmp/inst2/usr/lib/systemd/systemd
-rwxr-xr-x 461224 Dec 14 13:17 /var/tmp/inst2/usr/bin/systemd-analyze
-rwxr-xr-x 5271568 Dec 14 13:17 /var/tmp/inst2/usr/lib/systemd/libsystemd-core-250.so
(-0g, strip)
-rwxr-xr-x 2522080 Dec 14 13:19 /var/tmp/inst1/usr/lib/systemd/systemd
-rwxr-xr-x 2604160 Dec 14 13:19 /var/tmp/inst1/usr/bin/systemd-analyze
-rwxr-xr-x 113304 Dec 14 13:19 /var/tmp/inst2/usr/lib/systemd/systemd
-rwxr-xr-x 207656 Dec 14 13:19 /var/tmp/inst2/usr/bin/systemd-analyze
-rwxr-xr-x 2648520 Dec 14 13:19 /var/tmp/inst2/usr/lib/systemd/libsystemd-core-250.so
So for systemd itself we grow a bit (2522080 → 2648520+113304=2761824), but
overall we save. The most is saved on all the test files that link to libcore,
if they are installed, because there's 15 of them:
$ du -s /var/tmp/inst?
220096 /var/tmp/inst1
122960 /var/tmp/inst2
I also considered making systemd-analyze a symlink to /usr/lib/systemd/systemd
and turning systemd into a multicall binary. We did something like this with
udevd and udevadm. But that solution doesn't fit well in this case.
systemd-analyze has a bunch of functionality that is not used in systemd,
so the systemd binary would need to grow quite a bit. And we're likely to
add new types of verification or introspection features in analyze, and this
baggage would only grow. In addition, there are the test binaries which also
benefit from this.
00db9a114e ("docs: generate table from header using a script") got the
descriptions for the partition types mixed up. After that change, the
spec claimed, for example, that the /usr partition should contain
"dm-verity integrity hash data for the matching root partition", and
that the /usr verity partition should be of type "Any native, optionally
in LUKS". This made the spec an extremely confusing read before I
figured out what must have happened!
I've gone through the table as it existed prior to 00db9a114e, and moved
the descriptions around in the script that generates the table until
they matched up with what they used to be. Then I regenerated the
table from the fixed script.
This adds a helper script:
$ python3 tools/list-discoverable-partitions.py <src/shared/gpt.h
<!-- generated with tools/list-discoverable-partitions.py -->
| Partition Type UUID | Name | Allowed File Systems | Explanation |
|---------------------|------|----------------------|-------------|
| _Root Partition (Alpha)_ | `6523f8ae-3eb1-4e2a-a05a-18b695ae656f` | [Root Partition] | [Root Partition more] |
| _Root Partition (ARC)_ | `d27f46ed-2919-4cb8-bd25-9531f3c16534` | ditto | ditto |
...
The output can be pasted into the markdown file. I think this works better than
trying to match the two lists by hand.
When using "capture : true" in custom_target()s the mode of the source
file is not preserved when the generated file is not installed and so
needs to be tweaked manually. Switch from output capture to creating the
target file and copy the permissions from the input file.
Signed-off-by: Christian Brauner <christian.brauner@ubuntu.com>
Imports are sorted in the usual fashion: stdlib first.
literal_eval() parses string/numbers/lists/sets/dicts, and nothing else, while
eval will execute any python code. Using literal_eval() is generally more
correct, because it avoids the risk of side effects from the parsed expression.
In this case, we generate the parsed strings ourselves, so it's very unlikely
to have anything unexpected in the expressions. But let's do the correct thing
anyway.
It makes it easier to process the license automatically like other files.
The text of the license in tools/chromiumos/LICENSE matches
https://spdx.org/licenses/BSD-3-Clause.html exactly.
Format output in a manner that can be copypasted as-is to NEWS.
That is, with 8 spaces indentation and wrapped at 80 columns.
Before:
$ tools/git-contrib.sh
Ben Stockett,
Carl Lei,
Frantisek Sumsal,
Gibeom Gwon,
Hugo Osvaldo Barrera,
James Hilliard,
Jan Palus,
Lennart Poettering,
Luca Boccassi,
Luca BRUNO,
Mike Gilbert,
nassir90,
nl6720,
Raul Tambre,
Yegor Alexeyev,
Yu Watanabe,
Zbigniew Jędrzejewski-Szmek,
After:
Contributions from: Ben Stockett, Carl Lei, Frantisek Sumsal,
Gibeom Gwon, Hugo Osvaldo Barrera, James Hilliard, Jan Palus,
Lennart Poettering, Luca Boccassi, Luca BRUNO, Mike Gilbert,
nassir90, nl6720, Raul Tambre, Yegor Alexeyev, Yu Watanabe,
Zbigniew Jędrzejewski-Szmek
Lines in the dumps are ordered by some pseudo-random hashmap entry order, which
makes it hard to diff two outputs. This sort the entries alphabetically, and
also sorts items within the entries, and supresses timestamps and other fields
which always vary.
We could sort the output inside of systemd itself, but it'd make things more
complex, and we probably don't need output to be sorted in most cases. It also
wouldn't be enough, because timestamps and such would still need to be ignored
to do a nice diff. So I think doing the sorting and suppression in a python
helper is a better approach.
m4 was hugely popular in the past, because autotools, automake, flex, bison and
many other things used it. But nowadays it much less popular, and might not even
be installed in the buildroot. (m4 is small, so it doesn't make a big difference.)
(FWIW, Fedora dropped make from the buildroot now,
https://fedoraproject.org/wiki/Changes/Remove_make_from_BuildRoot. I think it's
reasonable to assume that m4 will be dropped at some point too.)
The main reason to drop m4 is that the syntax is not very nice, and we should
minimize the number of different syntaxes that we use. We still have two
(configure_file() with @FOO@ and jinja2 templates with {{foo}} and the
pythonesque conditional expressions), but at least we don't need m4 (with
m4_dnl and `quotes').
m4 was nice in '85, but the syntax feels a bit dated. Since we use python for
meson, let's use a popular python templating engine to replace some m4 usage.
A little nicety is that typos are caught:
FAILED: sysusers.d/systemd-remote.conf
/usr/bin/meson --internal exe --capture sysusers.d/systemd-remote.conf -- /home/zbyszek/src/systemd/tools/meson-render-jinja2.py config.h ../sysusers.d/systemd-remote.conf.j2
Traceback (most recent call last):
File "/home/zbyszek/src/systemd/tools/meson-render-jinja2.py", line 28, in <module>
print(render(sys.argv[2], defines))
File "/home/zbyszek/src/systemd/tools/meson-render-jinja2.py", line 24, in render
return template.render(defines)
File "/usr/lib/python3.9/site-packages/jinja2/environment.py", line 1090, in render
self.environment.handle_exception()
File "/usr/lib/python3.9/site-packages/jinja2/environment.py", line 832, in handle_exception
reraise(*rewrite_traceback_stack(source=source))
File "/usr/lib/python3.9/site-packages/jinja2/_compat.py", line 28, in reraise
raise value.with_traceback(tb)
File "<template>", line 8, in top-level template code
jinja2.exceptions.UndefinedError: 'HAVE_MICROHTTP' is undefined
This checking mirrors what 349cc4a507 did for C defines.
This reverts commit a2031de849.
The patch itself seems OK, but it exposes a bug in lxml or libxml2-2.9.12 which
was just released. This is being resolved in
https://gitlab.gnome.org/GNOME/libxml2/-/issues/255, but it might be while. So
let's revert this for now to unbreak our CI.
Fixes#19601.
I occasionally do 'build/man/man systemd.directives' when working on man pages,
and it's annoying slow. By paralellizing the parsing of xml, we can make it a
bit faster.
This is still rather innefficient. Only the parsing part is serialized, xml is
still produced serially at the end, which is hard to avoid.
$ ninja -C build man/systemd.directives.xml
before:
8.20s user 0.21s system 99% cpu 8.460 total
8.33s user 0.18s system 98% cpu 8.619 total
8.72s user 0.19s system 98% cpu 9.019 total
after:
13.99s user 0.73s system 345% cpu 4.262 total
14.15s user 0.35s system 348% cpu 4.161 total
14.33s user 0.35s system 339% cpu 4.321 total
I.e. it uses almost twice as much cpu, but cuts the wallclock time down (on a
2-core/4-thread cpu) to about half too, which is an overall win if you're just
trying to render the man page.
The change from list and .append() to set and .add() is something that could
have been done before too, but it's noticable now. It cuts down on the
serialization/deserialization time (about .2s).
Add a build script to compile bpf source code. A program in restricted
C is compiled into an object file. Object file is converted to BPF
skeleton [0] header file.
If build with custom meson build rule, the target header will reside in
build/ directory (not in source tree), e.g the path for socket_bind:
`build/src/core/bpf/socket_bind/socket-bind.skel.h`
Script runs the phases:
* clang to generate *.o from restricted C
* llvm-strip to remove useless DWARF info
* bpf skeleton generation with bpftool
These phases are logged to stderr for debug purposes.
To include BTF debug information, -g option is passed to clang.
[0] https://lwn.net/Articles/806911/
When executed in test mode, "OUTDATED" is appropriate. But when executed
to actually update the text, after the tool executes, those pages are the
opposite, not outdated.
668b3a42fe allowed update-dbus-docs.py to start
running on Cent OS 8 (instead of skipping). But subprocess.check_output()'s
text argument didn't exist until Python 3.7 and C8 is still running
Python 3.6. Use universal_newlines instead for backwards compatibility.
