It is common to have some data indexed by a name.
If you want to sort a list of such data, you would
have to re-implement your own compare function each time.
Instead, add NMUtilsNamedEntry which as first field has
the name. So, you can create your own struct:
struct my_data {
const char *name;
... other fields
}
and compare them with with nm_utils_named_entry_cmp().
For convenience, add another struct NMUtilsNamedValue, which
has only one data field, a pointer.
This allows the compiler to inline the siphash24*() functions
for nm_hash_ptr() and nm_hash_str() (even without LTO).
This of course only applies to nm_hash_ptr() and nm_hash_str(),
which are implemented in "nm-hash-utils.c" itself. All other
nm_hash_*() functions are inline functions in "nm-hash-utils.h",
and thus these functions can be inlined instead. That is, in
other cases, the nm_hash_*() function instead can be inlined.
For nm_hash_ptr() and nm_hash_str() instead we want to inline the
siphash24*() functions.
So, no longer compile "siphash24.c" directly. Instead, only
build "nm-hash-utils.c" which internally #include "siphash24.c".
Next we will use siphash24() instead of the glib version g_direct_hash() or
g_str_hash(). Hence, the "nm-utils/nm-hash-utils.h" header becomes very
fundamental and will be needed basically everywhere.
Instead of requiring the users to include them, let it be included via
"nm-default.h" header.
siphash24() mixes the bits much better then our naive xor.
Don't bypass siphash24(). We supposedly use it for the
better hashing properties, so use it also for pointers.
When using siphash24(), the hash value depends on the hashed input
and the key from _get_hash_key(). If the input is static, so is also
the result of siphash24(), albeit the bits are scrabbled more.
Add a nm_hash_static() to get such a static key, but without actually
doing siphash24(). The static key is also xored with a static_seed.
For that, also mangle the first byte of the hash key using siphash24()
itself. That is, because nm_hash_static() only uses the first guint of the
random key. Hence, we want that this first guint has all the entropy
of the entire key. We use siphash24() itself, to mangle all bits
of the 16 byte key into the first guint.
The nm_close() wrapper should behave exactly the same as calling
close() directly. This is well known, documented behavior.
The only addition on top of that, should be the nm_assert() to catch
double-closing.
Prevously, when passing a negative file descriptor, we wouldn't properly
set errno. Also, the call would not show up in strace, which it should
(at least, if libc's close actually makes the syscall).
I would argue, that passing a negative file descriptor is a bug already
and we should never do that. Maybe we should even assert non-negative
fds. I don't do that now, because I am not sufficiently confident.
Anyway, the change should have not practical effect, because we
shouldn't actually pass negative fds already.
There is still a fallback detection in "shared/nm-utils/nm-macros-internal.h",
so that VPN-plugins and applet don't need to bother about adding these
configure checks.
The _NM_GET_PRIVATE() macro already preserved and propagated
the constness of @self to the resulting private pointer.
_NM_GET_PRIVATE_PTR() didn't do that. Extend the macro,
to make that possible.
- nm-ovsdb.c uses json_load_callback(), which is jansson v2.4.
Hence, it cannot build the OVS plugin in our Travis-CI, which is
still on Ubuntu Precise. Disable building the plugin in travis and
add a compiler warning when building against an older version.
- since jansson v2.3, there is json_object_key_to_iter() to implement
the for-each macros. Use it in json_object_foreach_safe() when
available.
We need to pass more alias-types. Instead of having numbered
versions, use variadic number of macro arguments.
Also, fix build failure with old compiler:
In file included from src/nm-ip6-config.c:24:
./src/nm-ip6-config.h:44:29: error: controlling expression type 'typeof (ipconf_iter->current->obj)' (aka 'const void *const') not compatible with any generic association type
*out_address = has_next ? NMP_OBJECT_CAST_IP6_ADDRESS (ipconf_iter->current->obj) : NULL;
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Fixes: b1810d7a68
_NM_GET_PRIVATE() used typeof() to propagate constness of the @self
pointer. However, that means, it could only be used with a self pointer
of the exact type. That means, you explicitly had to cast from (GObject *)
or from (void *).
The requirement is cumbersome, and often led us to either create @self
pointer we didn't need:
NMDeviceVlan *self = NM_DEVICE_VLAN (device);
NMDeviceVlanPrivate *priv = NM_DEVICE_VLAN_GET_PRIVATE (self);
or casting:
NMDeviceVlanPrivate *priv = NM_DEVICE_VLAN_GET_PRIVATE ((NMDevice *) device);
In both cases we forcefully cast the source variable, loosing help from
the compiler to detect a bug.
For "nm-linux-platform.c", instead we commonly have a pointer of type
NMPlatform. Hence, we always forcefully cast the type via _NM_GET_PRIVATE_VOID().
Rework the macro to use _Generic(). If compiler supports _Generic(), then we
will get all compile time checks as desired. If the compiler doesn't support
_Generic(), it will still work. You don't get the compile-time checking of course,
but you'd notice that something is wrong once you build with a suitable
compiler.
shared/nm-utils/nm-hash-utils.c:110:3: error: right shift count >= width of type [-Werror]
h = h ^ ((guint) (((uintptr_t) ptr) >> 32)) ^ ((guint) ((uintptr_t) ptr));
^
Even if the branch is not reached on 32-bit architectures, the
compiler still emits a warning for the 32-bit right shift.
Fixes: ee76b0979f
nm_close() is like close(), but throws an assertion if the input fd is
>=0 and invalid. Passing an invalid (i.e. already closed) fd to
close() is a programming error with potentially catastrophic effects,
as another thread may reuse the closed fd.
We often want to cascade hashing, meaning, to combine the
outcome of various hash functions in a larger hash.
Instead of having each hash function return a guint hash value,
accept a hash state argument. This saves the overhead of initializing
and completing the intermediate hash states.
It also avoids loosing entropy when we reduce the larger hash state
into the intermediate guint hash value.
By using a macro, we don't cast all the types to guint. Instead,
we use their native types directly. Hence, we don't need
nm_hash_update_uint64() nor nm_hash_update_ptr().
Also, for types smaller then guint like char, we save hashing
the all zero bytes.
siphash24() is wildly used by projects nowadays.
It's certainly slower then our djb hashing that we used before.
But quite likely it's fast enough for us, given how wildly it is
used. I think it would be hard to profile NetworkManager to show
that the performance of hash tables is the issue, be it with
djb or siphash24.
Certainly with siphash24() it's much harder to exploit the hashing
algorithm to cause worst case hash operations (provided that the
seed is kept private). Does this better resistance against a denial
of service matter for us? Probably not, but let's better be safe then
sorry.
Note that systemd's implementation uses a different seed for each hash
table (at least, after the hash table grows to a certain size).
We don't do that and use only one global seed.
The privious NM_HASH_* macros directly operated on a guint value
and were thus close to the actual implementation.
Replace them by adding a NMHashState struct and accessors to
update the hash state. This hides the implementation better
and would allow us to carry more state. For example, we could
switch to siphash24() transparently.
For now, we still do a form basically djb2 hashing, albeit with
differing start seed.
Also add nm_hash_str() and nm_str_hash():
- nm_hash_str() is our own string hashing implementation
- nm_str_hash() is our own string implementation, but with a
GHashFunc signature, suitable to pass it to g_hash_table_new().
Also, it has this name in order to remind you of g_str_hash(),
which it is replacing.
"nm-utils/nm-shared-utils.h" shall contain utility function without other
dependencies. It is intended to be used by other projects as-is.
nm_utils_random_bytes() requires getrandom() and a HAVE_GETRANDOM configure
check. That makes it more cumbersome to re-use "nm-shared-utils.h", in
cases where you don't care about nm_utils_random_bytes().
Split nm_utils_random_bytes() out to a separate file.
Same for hash utils, which depend on nm_utils_random_bytes(). Also, hash
utils will eventually be extended to use siphash24.
This makes hashing non-deterministic with the aim to
make it harder to exploit hash collisions.
Non-deterministic also means that for unit testing
we will get different values on each run. But since we
shall never assign any meaning to these hash values
nor rely on them being stable between restarts (or
upgrades), that doesn't hurt.
Introduce a NM_HASH_INIT() function. It makes the places
where we initialize a hash with a certain seed visually clear.
Also, move them from "shared/nm-utils/nm-shared-utils.h" to
"shared/nm-utils/nm-macros-internal.h". We might want to
have NM_HASH_INIT() non-inline (hence, define it in the
source file).
Add a new function nm_utils_random_bytes().
This function now preferably uses getrandom() syscall if it is
available.
As fallback, it always tries to fill the buffer from /dev/urandom.
If it cannot, as last fallback it uses GRand, which cannot fail.
Hence, the function always sets some (pseudo) random bytes.
It also returns FALSE if the obtained bytes are possibly not good
randomness.
Fixes the following:
shared/nm-utils/nm-shared-utils.c:136: Warning: NetworkManager: GTK-Doc comment block end token "*/" should not be preceded by comment text:
* Returns: the input buffer with the quoted string. */
We already have nm_strquote_a(). That is useful, but uses alloca(), hence it
is ill suited to be called from a macro, inside a loop, or from a function
that should be inlined.
Instead, add nm_strquote() that has the same purpose but writes to a provided
string buffer.
nm_dedup_multi_obj_ref() is a trivial function, that only uses the field
which is already declared in the same header file. Move it to the header
so that it can be inlined (without LTO).
A replacement for g_strsplit_set(). While g_strsplit_set()
does (n+1) malloc and n slice allocations, this needs
roughtly (O(log(n))) mallocs.
Another difference from g_strsplit_set() is that this function
treats multiple delimiters as one (and thus never returns empty
words). While I can see that sometimes you may want to keep empty
words (like parsing a CSV file and preserve empty cells), we usually
use this function for splitting user input. In such case, we want
to treat multiple delimiters as one.
Kernel does not allow to add IPv6 routes with "src", as long as the
corresponding address is still tentative (related bug rh#1457196).
The workaround for this is cumbersome. First, when we fail to add such a
route with "pref_src", we guess that it happend due to this issue. In
that case, nm_ip6_config_commit() returns the list of routes that could
not be added for the moment (but hopefully can be added later).
We track this list in NMDevice, and keep trying to merge the routes
back into ip6_config. In order to not try indefinitely, keep track of a
timestamp when we tried to add this route for the first time.
Another uglyness is that pending tentative routes don't explicitly block
activation. In practice they may do, because for these routes we also have
an IPv6 address that is still doing DAD, so the IP configuration is
still pending due to that.
https://bugzilla.redhat.com/show_bug.cgi?id=1452684
- nm_clear_g_object() is like g_clear_object() but:
- it returns a boolean value, indicating that something was cleared.
- it includes an nm_assert() to check that the pointer is still
valid.
- it uses typeof() instead of blindly casting the argument.
- nm_g_object_ref_set() combines nm_clear_g_object() and resetting
the pointer to a new object, including taking a reference.
- also returns a boolean, indicating whether something changed.
- it gets the order of operations right: first it increses the
ref-count, before unrefing the old object.
- like nm_clear_g_object() and nm_clear_g_free() it first sets
the destination to NULL, instead of leaving a dangling pointer
for the duraction of the unref/free call.
- fix nm_clear_g_free() not to use a possibly dangling pointer.
Striclty speaking, that is undefined behavior.
And relax the type for nm_auto_unref_gtypeclass macro. Like
g_type_class_unref() itself, you usually don't use it with a GTypeClass
base class, but some subtype like GObjectClass.
For RFC1918 private IPv4addresses, guess a better prefix length for
addresses and routes.
nmtui is an interactive program. It makes sense to be a bit smarter
about what the user probably meant.
It would be nice if nmtui would update the entry field immediately when
the cursor leaves the field, to show the guessed prefix length. However,
that is not easily possible, so lets to that another time.
For IPv6 addresses, default to /64 instead of /128.
https://bugzilla.redhat.com/show_bug.cgi?id=1474295
We rely on clearing the dirty flag. For example in nm_ip4_config_replace(),
we first mark all entries dirty, then force-append the ones we keep,
and finally remove the ones that are still dirty.
Since _nm_ip_config_add_obj() short-cuts nm_dedup_multi_index_add_full(),
it must clear the dirty flag on its own.
Compiler wouldn't recognize that the @route/@address argument is always
initialized. The right workaround seems to let the next() functions always
set the value.
In file included from src/nm-ip6-config.c:24:0:
src/nm-ip6-config.c: In function ‘nm_ip6_config_create_setting’:
src/nm-ip6-config.c:734:62: error: the address of ‘address’ will always evaluate as ‘true’ [-Werror=address]
nm_ip_config_iter_ip6_address_for_each (&ipconf_iter, self, &address) {
^
src/nm-ip6-config.h:60:17: note: in definition of macro ‘nm_ip_config_iter_ip6_address_for_each’
for (({ if (address) { *(((const NMPlatformIP6Address **) address)) = NULL; } }), nm_ip_config_iter_ip6_address_init ((iter), (self)); \
^
Fixes: 6e9aa9402a
Previously, we would add exclusive routes via netlink message flags
NLM_F_CREATE | NLM_F_REPLACE for RTM_NEWROUTE. Similar to `ip route replace`.
Using that form of RTM_NEWROUTE message, we could only add a certain
route with a certain network/plen,metric triple once. That was already
hugely inconvenient, because
- when configuring routes, multiple (managed) interfaces may get
conflicting routes (multihoming). Only one of the routes can be actually
configured using `ip route replace`, so we need to track routes that are
currently shadowed.
- when configuring routes, we might replace externally configured
routes on unmanaged interfaces. We should not interfere with such
routes.
That was worked around by having NMRouteManager (and NMDefaultRouteManager).
NMRouteManager would keep a list of the routes which NetworkManager would like
to configure, even if momentarily being unable to do so due to conflicting routes.
This worked mostly well but was complicated. It involved bumping metrics to
avoid conflicts for device routes, as we might require them for gateway routes.
Drop that now. Instead, use the corresponding of `ip route append` to configure
routes. This allows NetworkManager to confiure (almost) all routes that we care.
Especially, it can configure all routes on a managed interface, without
replacing/interfering with routes on other interfaces. Hence, NMRouteManager
becomes obsolete.
It practice it is a bit more complicated because:
- when adding an IPv4 address, kernel will automatically create a device route
for the subnet. We should avoid that by using the IFA_F_NOPREFIXROUTE flag for
IPv4 addresses (still to-do). But as kernel may not support that flag for IPv4
addresses yet (and we don't require such a kernel yet), we still need functionality
similar to nm_route_manager_ip4_route_register_device_route_purge_list().
This functionality is now handled via nm_platform_ip4_dev_route_blacklist_set().
- trying to configure an IPv6 route with a source address will be rejected
by kernel as long as the address is tentative (see related bug rh#1457196).
Preferably, NMDevice would keep the list of routes which should be configured,
while kernel would have the list of what actually is configured. There is a
feed-back loop where both affect each other (for example, when externally deleting
a route, NMDevice must forget about it too). Previously, NMRouteManager would have
the task of remembering all routes which we currently want to configure, but cannot
due to conflicting routes.
We get rid of that, because now we configure non-exclusive routes. We however still
will need to remember IPv6 routes with a source address, that currently cannot be
configured yet. Hence, we will need to keep track of routes that
currently cannot be configured, but later may be.
That is still not done yet, as NMRouteManager didn't handle this
correctly either.
For completeness of the API. remove_obj() is basically a shortcut
of nm_dedup_multi_index_lookup_obj() combined with
nm_dedup_multi_index_remove_entry(). As such, it is useful to return
the actually deleted object. Note that the lookup needle @obj is not
necessarily the same instance as the one that will be removed, it's
only an instance that compares equal according to the index's equality
operator.
The return value shall indicate whether the add-call changed anything.
Reordering shall count as a change too.
On the other hand, clearing the dirty flag of the entry does not count
as a change.
Until now, NetworkManager's platform cache for routes used the quadruple
network/plen,metric,ifindex for equaliy. That is not kernel's
understanding of how routes behave. For example, with `ip route append`
you can add two IPv4 routes that only differ by their gateway. To
the previous form of platform cache, these two routes would wrongly
look identical, as the cache could not contain both routes. This also
easily leads to cache-inconsistencies.
Now that we have NM_PLATFORM_IP_ROUTE_CMP_TYPE_ID, fix the route's
compare operator to match kernel's.
Well, not entirely. Kernel understands more properties for routes then
NetworkManager. Some of these properties may also be part of the ID according
to kernel. To NetworkManager such routes would still look identical as
they only differ in a property that is not understood. This can still
cause cache-inconsistencies. The only fix here is to add support for
all these properties in NetworkManager as well. However, it's less serious,
because with this commit we support several of the more important properties.
See also the related bug rh#1337855 for kernel.
Another difficulty is that `ip route replace` and `ip route change`
changes an existing route. The replaced route has the same
NM_PLATFORM_IP_ROUTE_CMP_TYPE_WEAK_ID, but differ in the actual
NM_PLATFORM_IP_ROUTE_CMP_TYPE_ID:
# ip -d -4 route show dev v
# ip monitor route &
# ip route add 192.168.5.0/24 dev v
192.168.5.0/24 dev v scope link
# ip route change 192.168.5.0/24 dev v scope 10
192.168.5.0/24 dev v scope 10
# ip -d -4 route show dev v
unicast 192.168.5.0/24 proto boot scope 10
Note that we only got one RTM_NEWROUTE message, although from NMPCache's
point of view, a new route (with a particular ID) was added and another
route (with a different ID) was deleted. The cumbersome workaround is,
to keep an ordered list of the routes, and figure out which route was
replaced in response to an RTM_NEWROUTE. In absence of bugs, this should
work fine. However, as we only rely on events, we might wrongly
introduce a cache-inconsistancy as well. See the related bug rh#1337860.
Also drop nm_platform_ip4_route_get() and the like. The ID of routes
is complex, so it makes little sense to look up a route directly.
Default g_log() logs to stdout for INFO level and higher, but logs to stderr
for DEBUG/TRACE. That is annoying, because especially when redirecting the streams,
the messages get mixed up. Install a log handler and just print to stdout for
the tests.
Reasons:
- it adds an O(1) lookup index for accessing NMIPxConfig's addresses.
Hence, operations like merge/intersect have now runtime O(n) instead
of O(n^2).
Arguably, we expect low numbers of addresses in general. For low
numbers, the O(n^2) doesn't matter and quite likely in those cases
the previous implementation was just fine -- maybe even faster.
But the simple case works fine either way. It's important to scale
well in the exceptional case.
- the tracked objects can be shared between the various NMPI4Config,
NMIP6Config instances with NMPlatform and everybody else.
- the NMPObject can be treated generically, meaning it enables code to
handle both IPv4 and IPv6, or addresses and routes. See for example
_nm_ip_config_add_obj().
- I want core to evolve to somewhere where we don't keep copies of
NMPlatformIP4Address, et al. instances. Instead they shall all be
shared. I hope this will reduce memory consumption (although tracking a
reference consumes some memory too). Also, it shortcuts nmp_object_equal()
when comparing the same object. Calling nmp_object_equal() on the
identical objects would be a common case after the hash function
pre-evaluates equality.
Add a stable, recursive merge sort for CList.
This could be improved by doing an iterative implementation.
The recursive implementation's stack depth is not an issue,
as it is bound by O(ln(n)). But an iterative implementation
would safe the overhead of O(n*log(n)) function calls and be
potentially faster.
And get rid of the unused obj_full_equality_allows_different_class.
It's hard to grasp how to implement different object types that can compare
despite having different klasses. The idea was, that stack allocated
objects (used as lookup needles), are some small lightweight objects,
that still compare equal to the full instance. But it's unused. Drop it.
by moving the core functionality to "nm-dedup-multi.c".
As the ref-counting mechanism now is part of "nm-dedup-multi.c",
this works better and is reusable outside of platform.
Implement the reference counting of NMPObject as part of
NMDedupMultiObj and get rid of NMDedupMultiBox.
With this change, the NMPObject is aware in which NMDedupMultiIndex
instance it is tracked.
- this saves an additional GSlice allocation for the NMDedupMultiBox.
- it is immediately known, whether an NMPObject is tracked by a
certain NMDedupMultiIndex or not. This saves an additional hash
lookup.
- previously, when all idx-types cease to reference an NMDedupMultiObj
instance, it was removed. Now, a tracked objects stays in the
NMDedupMultiIndex until it's last reference is deleted. This possibly
extends the lifetime of the object and we may reuse it better.
- it is no longer possible to add one object to more then one
NMDedupMultiIndex instance. As we anyway want to have only one
instance to deduplicate the objects, this is fine.
- the ref-counting implementation is now part of NMDedupMultiObj.
Previously, NMDedupMultiIndex could also track objects that were
not ref-counted. Hoever, the object anyway *must* implement the
NMDedupMultiObj API, so this flexibility is unneeded and was not
used.
- a downside is, that NMPObject grows by one pointer size, even if
it isn't tracked in the NMDedupMultiIndex. But we really want to
put all objects into the index for sharing and deduplication. So
this downside should be acceptable. Still, code like
nmp_object_stackinit*() needs to handle a larger object.
Add the NMDedupMultiIndex cache. It basically tracks
objects as doubly linked list. With the addition that
each object and the list head is indexed by a hash table.
Also, it supports tracking multiple distinct lists,
all indexed by the idx-type instance.
It also deduplicates the tracked objects and shares them.
- the objects that can be put into the cache must be immutable
and ref-counted. That is, the cache will deduplicate them
and share the reference. Also, as these objects are immutable
and ref-counted, it is safe that users outside the cache
own them too (as long as they keep them immutable and manage
their reference properly).
The deduplication uses obj_id_hash_func() and obj_id_equal_func().
These functions must cover *every* aspect of the objects when
comparing equality. For example nm_platform_ip4_route_cmp()
would be a function that qualifies as obj_id_equal_func().
The cache creates references to the objects as needed and
gives them back. This happens via obj_get_ref() and
obj_put_ref(). Note that obj_get_ref() is free to create
a new object, for example to convert a stack-allocated object
to a (ref-counted) heap allocated one.
The deduplication process creates NMDedupIndexBox instances
which are the ref-counted entity. In principle, the objects
themself don't need to be ref-counted as that is handled by
the boxing instance.
- The cache doesn't only do deduplication. It is a multi-index,
meaning, callers add objects using a index handle NMDedupMultiIdxType.
The NMDedupMultiIdxType instance is the access handle to lookup
the list and objects inside the cache. Note that the idx-type
instance may partition the objects in distinct lists.
For all operations there are cross-references and hash table lookups.
Hence, every operation of this data structure is O(1) and the memory
overhead for an index tracking an object is constant.
The cache preserves ordering (due to linked list) and exposes the list
as public API. This allows users to iterate the list without any
additional copying of elements.
Platform has it's own, simple implementation of object types:
NMPObject. Extract a base type and move it to "shared/nm-utils/nm-obj.h"
so it can be reused.
The base type is trival, but it allows us to implement other objects
which are compatible with NMPObjects. Currently there is no API for generic
NMObjBaseInst type, so compatible in this case only means, that they
can be used in the same context (see example below).
The only thing that you can do with a NMObjBaseInst is check it's
NMObjBaseClass.
Incidentally, NMObjBaseInst is also made compatible to GTypeInstance.
It means, an NMObjBaseInst is not necessarily a valid GTypeInstance (like NMPObject
is not), but it could be implemented as such.
For example, you could do:
if (NMP_CLASS_IS_VALID ((NMPClass *) obj->klass)) {
/* is an NMPObject */
} else if (G_TYPE_CHECK_INSTANCE_TYPE (obj, NM_TYPE_SOMETHING)) {
/* it a NMSometing GType */
} else {
/* something else? */
}
The reason why NMPObject is not implemented as proper GTypeInstance is
because it would require us to register a GType (like
g_type_register_fundamental). However, then the NMPClass struct can
no longer be const and immutable memory. But we could.
NMObjBaseInst may or may not be a GTypeInstance. In a sense, it's
a base type of GTypeInstance and all our objects should be based
on it (optionally, they we may make them valid GTypes too).
