- assert that WITH_JANSSON and JANSSON_SONAME are defined consistently.
This check ensures that we can check at compile time that nm_json_vt()
will always fail (because JANSSON_SONAME) is undefined.
That is interesting, because this means you can do a compile time
for !WITH_JANSSON, and know if nm_json_vt() will *never* succeed.
With that, we could let the linker know when the code is unused
and remove all uses of nm_json_vt(), without using the traditional
conditional compilation with "#if WITH_JANSSON". But of course, we
currently don't do this micro optimization to remove defunct code.
- drop the "mode" helper variable and pass the flags directly to
dlopen().
jansson 2.7 was released October 2014. It's also in Ubuntu 16.06.
Other distros (like CentOS 7.5 and Debian Stretch/9) have both newer
versions.
Bump the requirement, simply because our CI does not use such old version
so it's not clear whether it works at all.
Our "nm-json-aux.h" redefines various things from <jansson.h> header.
Add a unit test that checks that what we redefine exactly matches what
libjansson would provide, so that they are compatible.
They serve a similar purpose.
Previously, nm-json-aux.h contained the virtual function table for accessing
the dynamically loaded libjansson. But there is no reason why our own
helper functions from nm-json.h cannot be there too.
nm-json.[hc] uses libjansson, but only loads it at runtime with dlopen. There
is no more run compile time dependency. Move it to shared, so that it can be
(theoretically) used by other components.
Also, drop the conditional compilation. Granted, if you don't build with
libjansson enabled, then the JANSSON_SONAME define is unset and the code
will fail to load at runtime (which is fine). However, we can still build
against our JSON wrappers. The code savings of conditional build are minimal
so drop it.
The output of nm_utils_format_variant_attributes() must be accepted by
nm_utils_parse_variant_attributes(), producing the initial attributes.
The latter has a special handling of some attributes, depending on the
input NMVariantAttributeSpec list. For example, if the
NMVariantAttributeSpec is a boolean with the 'no_value' flag, the
parser doesn't look for a value.
Pass the NMVariantAttributeSpec list to the format function so that it
can behave in the same way as the parse one.
Previously, we did not have a hard dependency on C99. Nowadays, we
actually build against C11, so we have <stdbool.h>. These definitions
are no longer necessary nor do we care about building against plain
C89.
We redefine _G_BOOLEAN_EXPR(), so let it use NM_BOOLEAN_EXPR().
Also, we use G_LIKELY() (and thus NM_BOOLEAN_EXPR()) inside nm_assert(),
and we use nm_assert() in some macros. To be able to nest nm_assert()
calls, we need to create unique variable names for NM_BOOLEAN_EXPR().
Having assertion macros that are disabled by default, is not
only useful for our glib code, but should also be available
for nm-std-aux. Move the macros.
- add unit test for nm_utils_parse_next_line()
- as line delimiter also accept "\r\n" and "\r" (beside "\n", "\0" and
EOF).
- fix returning lines with embedded "\0" characters. The line ends
on the first "\n" or "\0", whatever comes first. The code before
didn't ensure that with:
line_end = memchr (line_start, '\n', *inout_len);
if (!line_end)
line_end = memchr (line_start, '\0', *inout_len);
We want to parse "/proc/cmdline". That is space separated with support
for quoting and escaping. Our implementation becomes part of stable
behavior, and we should interpret the kernel command line the same way
as the system does. That means, our implementation should match
systemd's.
In the previous form, NM_STR_BUF_INIT() was a macro. That makes sense,
however it's not really possible to make that a macro without evaluating
the reservation length multiple times. That means,
NMStrBuf strbuf = NM_STR_BUF_INIT (nmtst_get_rand_uint32 () % 100, FALSE);
leads to a crash. That is unfortunate, so instead make it an inline
function that returns a NMStrBut struct. Usually, we avoid functions
that returns structs, but here we do it.
If g_vsnprintf() returns that it wants to write 5 characters, it
really needs space for 5+1 characters. If we have 5 characters
available, it would have written "0123\0", which leaves the buffer
broken.
Fixes: eda47170ed ('shared: add NMStrBuf util')
Previously, for simplicity, NMStrBuf did not support buffers without any
data allocated. However, supporting that has very little
overhead/complexity, so do it.
Now you can initialize buffers to have no data allocated, and when
appending data, it will automatically grow.
Iterating hash tables gives an undefined order. Often we want to have
a stable order, for example when printing the content of a hash or
when converting it to a "a{sv}" variant.
How to achieve that best? I think we should only iterate the hash once,
and not require additional lookups. nm_utils_named_values_from_strdict()
achieves that by returning the key and the value together. Also, often
we only need the list for a short time, so we can avoid heap allocating
the list, if it is short enough. This works by allowing the caller to
provide a pre-allocated buffer (usually on the stack) and only as fallback
allocate a new list.
If the value pointer is const, it is commonly inconvenient and requires
a cast. Requiring casts on a common base does not increase type safety,
but is annoying.
g_steal_pointer() is marked as GLIB_AVAILABLE_STATIC_INLINE_IN_2_44,
that means we get a deprecated warning. Avoid that. We anyway
re-implement the macro so that we can use it before 2.44 and so
that it always does the typeof() cast.
When parsing user input if is often convenient to allow stripping whitespace.
Especially with escaped strings, the user could still escape the whitespace,
if the space should be taken literally.
Add support for that to nm_utils_buf_utf8safe_unescape().
Note that this is not the same as calling g_strstrip() before/after
unescape. That is, because nm_utils_buf_utf8safe_unescape() correctly
preserves escaped whitespace. If you call g_strstrip() before/after
the unescape, you don't know whether the whitespace is escaped.
We want to use the function to unescape (compress) secrets. As such, we want
to be sure that no secrets are leaked in memory due to growing the buffer with
realloc. In fact, reallocation should never happen. Assert for that.
As reallocation cannot happen, we could directly fill a buffer with
API like nm_utils_strbuf_*(). But NMStrBuf has low overhead even in this
case.
We have nm_str_not_empty() which is the inverse of that. The purpose
of nm_str_not_empty() is to normalize a string to either return
%NULL or a non-empty string, like
const char *
get_name (Object *obj)
{
return nm_str_not_empty (obj->name);
}
Sometimes, we however want to check whether a string is not empty.
So, we previously had two choices:
1) use a temporary variable:
const char *tmp;
tmp = get_string ();
if (tmp && tmp[0])
...
The problem with this variant is that it's more verbose (by requiring a
temporary variable). Another downside is that there are multiple ways
how to check for an empty string (!tmp[0], tmp[0] == '\0', !strlen (tmp),
strlen (tmp) == 0), and sure enough they are all in use.
2) use !nm_str_not_empty(). But this double negation looks really odd
and confusing.
Add nm_str_is_empty() instead.
Macros preferably behave function-like, for example in that they evaluate
arguments exactly ones. Sometimes, we want to evaluate arguments
lazily, like in NM_IN_SET() or nm_g_set_error_take_lazy(). But it
is almost always undesirable to evaluate an argument more than once.
Fix NM_STR_HAS_PREFIX() for that.
Also, rename the local variable to not use the name "_str",
which may be a common name that the caller would like to use.
GPtrArray does not support NULL terminating the pointer array. That
makes it cumbersome to use it for tracking a strv array. Add a few
helper functions nm_strvarray_*() that help using a GArray instead.
When handling a GCancellable, you make decisions based on when the cancelled
property of a GCancellable changes. Correctly handling a cancellable becoming
uncancelled again is really complicated, nor is it clear what it even means:
should the flipping be treated as cancellation or not? Probably if the
cancelled property gets reset, you already start aborting and there is
no way back. So, you would want that a cancellation is always handled.
But it's hard to implement that correctly, and it's odd to claim
something was cancelled, if g_cancellable_is_cancelled() doesn't agree
(anymore).
Avoid such problems by preventing users to call g_cancellable_reset().
Add nm_utils_invoke_on_timeout() beside nm_utils_invoke_on_idle().
They are fundamentally similar, except one schedules an idle handler
and the other a timeout.
Also, use the current g_main_context_get_thread_default() as context
instead of the singleton instance. That is a change in behavior, but
the only caller of nm_utils_invoke_on_idle() is the daemon, which
doesn't use different main contexts. Anyway, to avoid anybody being
tripped up by this also change the order of arguments. It anyway
seems nicer to first pass the cancellable, and the callback and user
data as last arguments. It's more in line with glib's asynchronous
methods.
Also, in the unlikely case that the cancellable is already cancelled
from the start, always schedule an idle action to complete fast.
- in io_watch_have_data(), ensure that we handle incomplete lines
that don't yet have a newline by waiting for more data. That means,
if the current content of the in_buffer does not have a newline, we
wait longer.
- in io_watch_have_data(), implement (and ignore) certain commands
instead of failing the request.
- in io_watch_have_data(), no longer g_compress() the entire line.
"polkitagenthelper-pam.c" never backslash escapes the command, it
only escapes the arguments. Of course, there should be no difference
in practice, except that we don't want to handle escape sequences
in the commands.
- in io_watch_have_data(), compare SUCCESS/FAILURE literally.
"polkitagenthelper-pam.c" never appends any trailing garbage to these
commands, and we shouldn't handle that (although "polkitagentsession.c"
does).
- when io_watch_have_data() completes with success, we cannot destroy
AuthRequest right away. It probably still has data pending that we first
need to write to the polkit helper. Wait longer, and let io_watch_can_write()
complete the request.
- ensure we always answer the GDBusMethodInvocation. Otherwise, it gets
leaked.
- use NMStrBuf instead of GString.
We cannot just swallow EAGAIN and pretend that not bytes were read.
read() returning zero means end of file. The caller needs to distinguish
between end of file and EAGAIN.
NMStrBuf is not an opaque structure, so that we can allocate it on the
stack or embed it in a struct.
But most of the fields should not be touched outside of the
implementation.
Also, "len" and "allocated" fields may be accessed directly, but
they should not be modified.
Rename the fields to make that clearer.
We cannot actually mark the field as const, because then you could no
longer initialize a variable that contains a NMStrBuf with designated
initializers.
We also want to keep the "_allocated" alias, for the only places that
are allowed to mutate the field: inside "nm-str-buf.h". Add an alias
for that field, that is allowed to be read, provided that you don't
modify it!
The alternative would be a nm_str_buf_get_allocated() accessor, but
that seems unnecessarily verbose when you could just access the field.
Before, if a struct had a field of type NMStrBuf (which is sensible to do),
then you could not longer initialize the entire struct with
*ptr = (Type) { };
because NMStrBuf contained const fields.
The user should never set these fields directly and use nm_str_buf_*() to modify
them them. But no longer mark them as const, because that breaks valid
use cases.
The allocated buffes are not known to be written. It is unnecessary to
clear them.
If the user writes sensitive data to those locations, without using
the NMStrBuf API, then it is up to the user to bzero the memory
accordingly.
When we have a buffer that we want to grow exponentially with
nm_utils_get_next_realloc_size(), then there are certain buffer
sizes that are better suited.
For example, if you have an empty NMStrBuf (len == 0), and you
want to allocate roughly one kilobyte, then 1024 is a bad choice,
because nm_utils_get_next_realloc_size() will give you 2024 bytes.
NM_UTILS_GET_NEXT_REALLOC_SIZE_1000 might be better in this case.
NM_MORE_ASSERTS 0 means that more assertions are disabled.
NM_MORE_ASSERT_ONCE() should never be triggered when more
assertions are disabled altogether. It is thus not allowed
to called "if (NM_MORE_ASSERT_ONCE (0))", because that code
would always be enabled.
If you have a LIST with 7 elements, and you lookup a value that
is not in the (sorted) list and would lie before the first element,
the binary search will dig down to imin=0, imid=0, imax=0 and
strcmp will give positive cmp value (indicating that the searched
value is sorted before).
Then, we would do "imax = imid - 1;", which wrapped to G_MAXUINT,
and the following "if (G_UNLIKELY (imin > imax))" would not hit,
resulting in an out of bound access next.
The easy fix is to not used unsigned integers.
The binary search was adapted from nm_utils_array_find_binary_search()
and nm_utils_ptrarray_find_binary_search(), which already used signed
integers to avoid this problem.
Fixes: 17d9b852c8 ('shared: explicitly implement binary search in NM_UTILS_STRING_TABLE_LOOKUP_DEFINE*()')
Add flags to explicitly escape leading or trailing spaces. Note
that we were already escaping trailing spaces.
This will be used later when supporting backslash escapes for
option parameters for nmcli (vpn.data).
In the next commit, GString will be replaced by NMStrBuf. Then, we will
pre-allocate a string buffer with 16 bytes, and measure the performance
difference. To have it comparable, adjust the pre-allocation size also
with GString.
nm_utils_buf_utf8safe_unescape() is almost the same as g_strcompress(),
with the only difference is that if the string contains NUL escapes "\000",
it will be handled correctly.
In other words, g_strcompress() and nm_utils_str_utf8safe_unescape() can only
unescape values, that contain no NUL escapes. That's why we added our
own binary unescape function.
As we already have our g_strcompress() variant, use it. It just gives it more
testing and usage. Also, we have full control over it's behavior. For example,
g_strcompress() issues a g_warning() when encountering a trailing '\\'. I
think this makes it unsuitable to unescape untrusted data. Either the function
should fail, or just make the best of it. Currently, our implementation
does the latter.
Our own implementation of a string buffer like GString.
Advantages (in decreasing relevance):
- Since we are in control, we can easily let it nm_explicit_bzero()
the memory. The regular GString API cannot be used in such a case.
While nm_explicit_bzero() may or may not be of questionable benefit,
the problem is that if the underlying API counteracts the aim of
clearing memory, it gets impossible. As API like NMStrBuf supports
it, clearing memory is a easy as enable the right flag.
This would for example be useful for example when we read passwords
from a file or file descriptor (e.g. try_spawn_vpn_auth_helper()).
- We have API like
nmp_object_to_string (const NMPObject *obj,
NMPObjectToStringMode to_string_mode,
char *buf,
gsize buf_size);
which accept a fixed size output buffer. This has the problem of
how choosing the right sized buffer. With NMStrBuf such API could
be instead
nmp_object_to_string (const NMPObject *obj,
NMPObjectToStringMode to_string_mode,
NMStrBuf *buf);
which can automatically grow (using heap allocation). It would be
easy to extend NMStrBuf to use a fixed buffer or limiting the
maximum string length. The point is, that the to-string API wouldn't
have to change. Depending on the NMStrBuf passed in, you can fill
an unbounded heap allocated string, a heap allocated string up to
a fixed length, or a static string of fixed length. NMStrBuf currently
only implements the unbounded heap allocate string case, but it would
be simple to extend.
Note that we already have API like nm_utils_strbuf_*() to fill a buffer
of fixed size. GString is not useable for that (efficiently), hence
this API exists. NMStrBuf could be easily extended to replace this API
without usability or performance penalty. So, while this adds one new
API, it could replace other APIs.
- GString always requires a heap allocation for the container. In by far
most of the cases where we use GString, we use it to simply construct
a string dynamically. There is zero use for this overhead. If one
really needs a heap allocated buffer, NMStrBuf can easily embedded
in a malloc'ed memory and boxed that way.
- GString API supports inserting and removing range. We almost never
make use of that. We only require append-only, which is simple to
implement.
- GString needs to NUL terminate the buffer on every append. It
has unnecessary overhead for allowing a usage of where intermediate
buffer contents are valid strings too. That is not the case with
NMStrBuf: the API requires the user to call nm_str_buf_get_str() or
nm_str_buf_finalize(). In most cases, you would only access the string
once at the end, and not while constructing it.
- GString always grows the buffer size by doubling it. I don't think
that is optimal. I don't think there is one optimal approach for how
to grow the buffer, it depends on the usage patterns. However, trying
to make an optimal choice here makes a difference. QT also thinks so,
and I adopted their approach in nm_utils_get_next_realloc_size().
When growing a buffer by appending a previously unknown number
of elements, the often preferable strategy is growing it exponentially,
so that the amortized runtime and re-allocation costs scale linearly.
GString just always increases the buffer length to the next power of
two. That works.
I think there is value in trying to find an optimal next size. Because
while it doesn't matter in terms of asymptotic behavior, in practice
a better choice should make a difference. This is inspired by what QT
does ([1]), to take more care when growing the buffers:
- QString allocates 4 characters at a time until it reaches size 20.
- From 20 to 4084, it advances by doubling the size each time. More
precisely, it advances to the next power of two, minus 12. (Some memory
allocators perform worst when requested exact powers of two, because
they use a few bytes per block for book-keeping.)
- From 4084 on, it advances by blocks of 2048 characters (4096 bytes).
This makes sense because modern operating systems don't copy the entire
data when reallocating a buffer; the physical memory pages are simply
reordered, and only the data on the first and last pages actually needs
to be copied.
Note that a QT is talking about 12 characters, so we use 24 bytes
head room.
[1] https://doc.qt.io/qt-5/containers.html#growth-strategies
- add more code comments
- refactor the code flow in _get_hash_key_init() to follow a simpler
code path.
- use c_siphash_hash() instead of 3 separate steps.
- Drop "?: static_seed" from nm_hash_static(). It's not useful, because
the only _get_hash_key() for which _get_hash_key()^static_seed is zero
is ~static_seed. That means, only one value of all the static seeds
can result in zero here. At that point, we can just coerce that value
to 3679500967u directly.
Sometimes these function may set errno to unexpected values like EAGAIN.
This causes confusion. Avoid that by using our own wrappers that retry
in that case. For example, in rhbz#1797915 we have failures like:
errno = 0;
v = g_ascii_strtoll ("10", 0, &end);
if (errno != 0)
g_assert_not_reached ();
as g_ascii_strtoll() would return 10, but also set errno to EAGAIN.
Work around that by using wrapper functions that retry. This certainly
should be fixed in glib (or glibc), but the issues are severe enough to
warrant a workaround.
Note that our workarounds are very defensive. We only retry 2 times, if
we get an unexpected errno value. This is in the hope to recover from
a spurious EAGAIN. It won't recover from other errors.
https://bugzilla.redhat.com/show_bug.cgi?id=1797915
NMTST_SWAP() used memcpy() for copying the value, while NM_SWAP() uses
a temporary variable with typeof(). I think the latter is preferable.
Also, the macro is essentially doing the same thing.
NM_ASCII_SPACES contains the ASCII characters according to
g_ascii_isspace().
Add NM_ASCII_WHITESPACES which differs from NM_ASCII_SPACES by not
including "\f". In some cases, that character shall be excluded.
For example, this is what systemd uses as "WHITESPACE" define at
various places.
Also, reorder the spaces string so that plain space comes first. It is
expected that ' ' is much more frequently than newlines or tabs. While
the order here shouldn't matter, it seems preferably to order frequent
characters in front.
Move the assertion for valid LIST first. It only checks static data,
and regardless of the entry_cmd, it should be done first.
Fixes: f4d12f7b59 ('shared: add NM_UTILS_STRING_TABLE_LOOKUP_STRUCT_DEFINE() macro for lookup of structs')
Depending on the type, OVS interfaces also have a corresponding netdev
in kernel (e.g. type "internal" does, type "patch" does not).
Such a case is neither NMU_IFACE_OVS nor NMU_IFACE_KERNEL (alone). There should
be a special type to represent those cases.
Add NMU_IFACE_OVS_OR_KERNEL for that.
nm_utils_ifname_valid() is to validate "connection.interface-name"
property. But the exact validation depends on the connection type.
Add "NMU_IFACE_ANY" to validate the name to check whether it would be
valid for any connection type.
This is for completeness and for places where the caller might not know
the connection type.
"all" and "default" never works.
"bonding_masters" works if you unload the bonding module. Well,
that should not really be called working...
Reject these names.
Generally, it's dangerous to reject values that were accepted
previously. This will lead to NetworkManager being unable to load
a profile from disk, which was loadable previously.
On the other hand, kernel would not have treated this setting as
it was intended. So, I would argue that the such a setting was not
working (as intended) anyway.
We can only hope that users don't configure arbitrary interface names.
It generally isn't a good idea to do, so "breaking" such things is less
of a concern.
To really use multiple NM_GOBJECT_PROPERTIES_DEFINE_BASE*() defines in
the same source file, several fixes to the suffix handling are
necessary. This fixes commit f13c7e3bbd ('shared: extend
NM_GOBJECT_PROPERTIES_DEFINE*() macros to append suffix to defined
names') to really work.
Fixes: f13c7e3bbd ('shared: extend NM_GOBJECT_PROPERTIES_DEFINE*() macros to append suffix to defined names')
The interface name might come from the command line or from a filename
(like during nm_vpn_wireguard_import()). It's not clear that this
is valid UTF-8. Asserting against that requires the caller to ensure
that the encoding is valid. That is cumbersome, especially since we anyway
check. Just report a regular error.
quoting 'man ovs-vswitchd.conf.db':
"The name must be alphanumeric and must not contain forward or backward
slashes."
OVS actually accepts a wider range of chars (all printable UTF-8 chars),
NetworkManager restricts this to ASCII char as it's a safer option for
now since OVS is not well documented on this matter.
https://bugzilla.redhat.com/show_bug.cgi?id=1788432
Fixes: e7d72a14f6 ('libnm-core: use different ifname validation function for OVS bridges, ports and interfaces')
The only affected caller is nm_modem_manager_get_modems(), which
is used by NMDeviceBt for DUN connections. This is rather bad.
Fixes: 4154d9618c ('bluetooth: refactor BlueZ handling and let NMBluezManager cache ObjectManager data')
Fixes: e688e70b37 ('shared: add nm_utils_hash_values_to_array() helper')
$ git shortlog -n -s a3e75f3294 -- shared/nm-glib-aux/nm-jansson.h shared/nm-utils/nm-jansson.h
7 Thomas Haller
1 Lubomir Rintel
1 Beniamino Galvani
1 Francesco Giudici
All contributors agreed to the relicensing according to "RELICENSE.md".
$ git shortlog -n -s a3e75f3294 -- shared/nm-glib-aux/nm-glib.h shared/nm-utils/nm-glib.h shared/nm-glib.h include/nm-glib.h include/nm-glib-compat.h
35 Thomas Haller
12 Dan Winship
5 Dan Williams
3 Lubomir Rintel
2 Beniamino Galvani
1 Jan Kantert
1 Thomas Bechtold
The last two contributions by Thomas and Jan are no longer present in any
form, because they were for an older version of glib which is no longer
supported. All other contributors agree to the licences change according
to "RELICENSE.md" file.
For one, we by now require glib >= 2.34.0, so this is not used.
Also, I think G_DEFINE_QUARK() is rather ugly because it constructs
the defined function name (so you cannot grep for it). We should use
NM_CACHED_QUARK_FCN() instead.
Several macros are used to define function. They had a "_STATIC" variant,
to define the function as static.
I think those macros should not try to abstract entirely what they do.
They should not accept the function scope as argument (or have two
variants per scope). This also because it might make sense to add
additional __attribute__(()) to the function. That only works, if
the macro does not pretend to *not* define a plain function.
Instead, embrace what the function does and let the users place the
function scope as they see fit.
This also follows what is already done with
static NM_CACHED_QUARK_FCN ("autoconnect-root", autoconnect_root_quark)
When looking at nm_utils_array_find_binary_search(), we see that binary
search really isn't complicated. In nm_utils_array_find_binary_search()
it looks complicated, because that is our general purpose function which
accepts arbitrary lists, uses an opaque compare function, accepts a user
data argument, and returns the insertion position.
This is unnecessary for the narrow purpose in NM_UTILS_STRING_TABLE_LOOKUP_DEFINE*().
When we inline the binary search, it can be simplified, and the remaining
parts is simple enough to prefer this duplication of binary search over
using our general purpose function.
Also, this gives the compiler more chance for optimization. For
example, we can use "unsigned" as index type instead of gssize, because
we know (at compile time), that this type will always be large enough
for our LIST. Also, we can directly call strcmp().
The result is that the macro's implementation is also fast in the best
case (where the needle is found with only one strcmp()) and in the cases
where there is a small number of items to search.
It thus alleviates concerns that using the macro might be slower than
an optimized implementation.
The binary size of the defined function increases slightly (from 112
bytes to 192 bytes, on x86_64, GCC, -O2). But according to my tests it
is slightly and measurably faster.
The _NM_GET_PRIVATE_PTR() macro is possibly used from other macros. And
"_self" is a pretty good name to use. Don't let the lower layer macro
use this name.
