Turns out Coccinelle can handle compound literals just fine, the parsing
errors were caused by incorrectly parsed macros in code before the
literals, so let's just provide simplified versions for such macros.
The parsing error in `Type *foo[ELEMENTSOF(bar)] = {};` is actually
harmless; it occurs only when creating an array of pointers for a type
that's in an external header and it occurs only on the first parser's
pass, subsequent passes resolve the type correctly.
Also, unset ENABLE_DEBUG_HASHMAP, so Coccinelle doesn't expand the
hashmap debug macros.
As for the remaining FIXMEs, I opened a couple of issues in the
Coccinelle upstream to see if they can be fixed there (or at least
properly analyzed).
Turns out I _really_ underestimated the impact of
--include-headers-for-types, as it significantly reduces both runtime
and storage penalties. For example, on my machine the runtime of
uncached run goes down from ~15 minutes to ~2 minutes, and similarly the
total storage needed by the cache goes from ~15 GiB down to ~3 GiB.
Drop the original macro file, since it's not needed anymore thanks to
resolving includes properly, but introduce a similar file -
parsing_hacks.h - that helps Coccinelle in some specific corner cases.
This eliminates most of the outstanding parsing errors in source files.
The remaining ones are limitations of the parsing engine (see the FIXMEs
in pasing_hacks.h) and need further investigation.
Since Coccinelle is originally a kernel tool, it doesn't search the
system include path by default for header files. Without this we're
missing a lot of types provides by stdlib (and other libraries we make
use of).
Turns out that the original way we did things was quite broken, as it
skipped a _lot_ of code. This was because we just threw everything into
one pile and tried to spatch it, but this made Coccinelle sad, like when
man page examples redefined some of our macros, causing typedef
conflicts.
For example, with a minimal reproducer that defines a cleanup macro in
two source files, Coccinelle has no issues when spatch-ing each one
separately:
$ spatch --verbose-parsing --sp-file zz-drop-braces.cocci main.c
init_defs_builtins: /usr/lib64/coccinelle/standard.h
HANDLING: main.c
SPECIAL NAMES: adding _cleanup_ as a attribute with arguments
SPECIAL NAMES: adding _cleanup_free_ as a attribute
$ spatch --verbose-parsing --sp-file zz-drop-braces.cocci
logcontrol-example.c
init_defs_builtins: /usr/lib64/coccinelle/standard.h
HANDLING: logcontrol-example.c
SPECIAL NAMES: adding _cleanup_ as a attribute with arguments
But when you try to spatch both of them at once, Coccinelle starts
complaining and skipping the "bad" code:
$ spatch --verbose-parsing --sp-file zz-drop-braces.cocci main.c logcontrol-example.c
init_defs_builtins: /usr/lib64/coccinelle/standard.h
HANDLING: main.c logcontrol-example.c
SPECIAL NAMES: adding _cleanup_ as a attribute with arguments
SPECIAL NAMES: adding _cleanup_free_ as a attribute
remapping: _cleanup_ to an ident in macro name
ERROR-RECOV: found sync end of #define, line 44
parsing pass2: try again
ERROR-RECOV: found sync end of #define, line 44
parse error
= File "logcontrol-example.c", line 44, column 21, charpos = 1719
around = '__attribute__',
whole content = #define _cleanup_(f) __attribute__((cleanup(f)))
badcount: 2
bad: #include <systemd/sd-journal.h>
bad:
BAD:!!!!! #define _cleanup_(f) __attribute__((cleanup(f)))
This was, unfortunately, hidden as it is visible only with
--verbose-parsing (or --parse-error-msg).
Another issue was how we handled includes. The original way of throwing
them into the pile of source files doesn't really work, leading up to
similar issues as above. The better way is to let Coccinelle properly
resolve all includes by telling it where to find our own include files
(basically the same thing we already do during compilation).
After fixing all this, Coccinelle now has a chance to process much more
of our code (there are still some issues in more complex macros, but
that requires further investigation). However, there's a huge downside
from all of this - doing a _proper_ code analysis is surprisingly time
and resource heavy; meaning that processing just one Coccinelle rule now
takes 15 - 30 minutes.
To make this slightly less painful, Coccinelle supports caching the
generated ASTs, which actually helps a lot - it gets the runtime of one
rule from 15 - 30 minutes down to ~1 minute. It, of course, has its own
downside - the cache is _really_ big (ATTOW the cache takes ~15 GiB).
However, even with the aggressive AST caching you're still looking at
~1 hour for one full Coccinelle run, which is a bit annoying, but I
guess that's the price of doing things _properly_ (but I'll definitely
look into ways of further optimizing this).
* flag-set.cocci: perform the transformation only if the second
argument is a constant
* sd-journal/lookup3.c: skip the cocci completely for this file, since
it's not "ours"
* strjoina.cocci: skip the transformation on the "test_strjoina" test,
since it intentionally tests the "incorrect" expression we're trying to
transform (the same thing was already done in strjoin.cocci)
Coccinelle can't do this automagically and requires we supply it
respective header files. Unfortunately, the option for this
(--macro-file=) can be used only once, so let's create our own
macro file by collecting macros needed for the semantic parser
to be happy.
My former dumb me didn't read the documentation properly, so with the
introduction of custom isomorphisms I caused two issues:
1) Masked all standard isomorphisms defined by Coccinelle
2) Replace the original issue with a completely new one
There's no point in running these transformation for certain files,
mainly anything from src/boot/efi and src/shared/linux, as this code
doesn't have access to our internal utility functions
Coccinelle needs a custom isomorphism file with rules (isomorphisms) how
to correctly rewrite conditions with explicit NULL checks (i.e.
if (ptr == NULL)) to their shorter form (i.e. if (!ptr)). Coccinelle
already contains such isomorphisms in its default .iso file, however,
they're in the opposite direction, which results in useless output from
coccinelle/equals-null.cocci.
With this fix, `spatch` should no longer report patches like:
@@ -628,8 +628,9 @@ static int path_deserialize_item(Unit *u
f = path_result_from_string(value);
if (f < 0)
log_unit_debug(u, "Failed to parse result value: %s", value);
- else if (f != PATH_SUCCESS)
- p->result = f;
+ else {if (f != PATH_SUCCESS)
+ p->result = f;
+ }
} else
log_unit_debug(u, "Unknown serialization key: %s", key);
We had all kinds of indentation: 2 sp, 3 sp, 4 sp, 8 sp, and mixed.
4 sp was the most common, in particular the majority of scripts under test/
used that. Let's standarize on 4 sp, because many commandlines are long and
there's a lot of nesting, and with 8sp indentation less stuff fits. 4 sp
also seems to be the default indentation, so this will make it less likely
that people will mess up if they don't load the editor config. (I think people
often use vi, and vi has no support to load project-wide configuration
automatically. We distribute a .vimrc file, but it is not loaded by default,
and even the instructions in it seem to discourage its use for security
reasons.)
Also remove the few vim config lines that were left. We should either have them
on all files, or none.
Also remove some strange stuff like '#!/bin/env bash', yikes.
spatch is single-threaded, i.e. slow. On my machine it allocates 5 GB of memory
and starts swapping, which makes it even slower. Using parallel makes the whole
thing pleasantly fast.
Also, allow run-cocinnelle.sh to be started from any directory.
Unfortunately set -x does not work nicely anymore, because the list is
too verbose. Replace it by an echo line.
Let's include the command line to use to get the requested output. This
makes it easy to copy/paste the command line out, and add "--in-place"
to actually apply the changes "run-coccinelle.sh" outputs.
It doesn't work, spits out only rubbish and was already excluded of
run-coccinelle.sh. It's a pitty it doesn't work, but let's drop this
dead piece of code for now.
