It turns out that certain types of early boot bugs can result in reboot
loops, even within a guest OS running under qemu/KVM. This commit
therefore upgrades the kvm-test-1-run-qemu.sh script's hang-detection
heuristics to detect such situations and to terminate the run when
they occur.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The kvm-test-1-run-qemu.sh script logs the torture-test start time and
also when it starts getting impatient for the test to finish. However, it
does not timestamp these log messages, which can make debugging needlessly
challenging. This commit therefore adds timestamps to these messages.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
There was a time long ago when the "test" command's documentation
claimed that the "-a" and "-o" arguments did something useful.
But this documentation now suggests letting the shell execute
these boolean operators, so this commit applies that suggestion to
kvm-test-1-run-qemu.sh.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit causes kvm-test-1-run-batch.sh to use the new
kvm-assign-cpus.sh and kvm-get-cpus-script.sh scripts to create a
TORTURE_AFFINITY environment variable containing either an empty string
(for no affinity) or a list of CPUs to pin the scenario's vCPUs to.
The additional change to kvm-test-1-run.sh places the per-scenario
number-of-CPUs information where it can easily be found.
If there is some reason why affinity cannot be supplied, this commit
prints and logs the reason via changes to kvm-again.sh.
Finally, this commit updates the kvm-remote.sh script to copy the
qemu-affinity output files back to the host system.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
There is "qemu-affinity", "qemu-cmd", "qemu-retval", but also "qemu_pid".
This is hard to remember, not so good for bash tab completion, and just
plain inconsistent. This commit therefore renames the "qemu_pid" file to
"qemu-pid". A couple of the scripts must deal with old runs, and thus
must handle both "qemu_pid" and "qemu-pid", but new runs will produce
"qemu-pid".
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The jitter.sh script has some entertaining awk code to generate a
hex mask from a randomly selected CPU number, which is handed to the
"taskset" command. Except that this command has a "-c" parameter to
take a comma/dash-separated list of CPU numbers. This commit therefore
saves a few lines of awk by switching to a single-number CPU list.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit causes the kvm-test-1-run-qemu.sh script to check the
TORTURE_AFFINITY environment variable and to add "taskset" commands to
the qemu-cmd file. The first "taskset" command is applied only if the
TORTURE_AFFINITY environment variable is a non-empty string, and this
command pins the current scenario's guest OS to the specified CPUs.
The second "taskset" command reports the guest OS's affinity in a new
"qemu-affinity" file.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Currently, kvm-test-1-run-qemu.sh applies redirection to each and every
line of each qemu-cmd script. Only the first line (the only one that
is not a bash comment) needs to be redirected. Although redirecting
the comments is currently harmless, just adding to the comment, it is
an accident waiting to happen. This commit therefore adjusts the "sed"
command to redirect only the qemu-system* command itself.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit causes kvm.sh to use the new kvm-assign-cpus.sh and
kvm-get-cpus-script.sh scripts to create a TORTURE_AFFINITY environment
variable containing either an empty string (for no affinity) or a list
of CPUs to pin the scenario's vCPUs to. A later commit will make
use of this information to actually pin the vCPUs.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit is a first step towards pinning guest-OS vCPUs so as
to force latency differences, especially on multi-socket systems.
The kvm.sh script puts its batch-creation awk script into a temporary
file so that later commits can add the awk code needed to dole out CPUs
so as to maximize latency differences. This awk code will be used by
multiple scripts.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The last line of kvm-test-1-run-qemu.sh invokes parse-console.sh, but
kvm-test-1-run-qemu.sh is unaware of the PATH containing this script
and does not have the job title handy. This commit therefore moves
the invocation of parse-console.sh to kvm-test-1-run.sh, which has
PATH and title at hand. This commit does not add an invocation of
parse-console.sh to kvm-test-1-run-batch.sh because this latter script
is run in the background, and the information will be gathered at the
end of the full run.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Currently, kvm-recheck.sh attempts to create a kcsan.sum file even for
build-only runs. This results in false-positive bash errors due to
there being no console.log files in that case. This commit therefore
makes kvm-recheck.sh skip creating the kcsan.sum file for build-only runs.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The kvm-remote.sh script places the datestamped directory containing
all the build artifacts in the destination systems' /tmp directories,
where they accumulate runtime artifacts such as console.log. This works,
but some systems have a habit of removing files in /tmp that have not
been recently accessed. This commit therefore runs a simple script that
periodically accesses all files in the datestamped directory.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The qemu-cmd file can contain comments that are not relevant to the
operation of kvm-recheck-lock.sh. This commit therefore strips these
comments before looking for timing information.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The qemu-cmd file can contain comments that are not relevant to the
operation of kvm-recheck-scf.sh. This commit therefore strips these
comments before looking for timing information.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Currently, each -kcsan run in a torture.sh group of runs has its own
kcsan.sum summary. This works, but there is usually a lot of duplication
between the runs. This commit therefore also creates an overall kcsan.sum
file for the entire torture.sh run, if there was at least one -kcsan run.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The kcsan-collapse.sh script assumes that it is being run over the output
of a single kvm.sh run, which is less than helpful for torture.sh runs.
This commit therefore changes the kcsan-collapse.sh script's "ls" pattern
with a "find" command to enable a KCSAN summary across all the -kcsan
runs in a full torture.sh run.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Currently, torture.sh accepts --doall on the one hand and --do-none
on the other, which is a bit inconsistent. This commit therefore adds
--do-all and --donone so that a fully consistent test may be used.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit adds three short tests of the clocksource-watchdog capability
to the torture.sh script, all to avoid otherwise-inevitable bitrot.
While in the area, fix an obsolete comment.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Currently, if a torture scenario requires more CPUs than are present
on the build system, kvm.sh and friends limit the CPUs available to
that scenario. This makes total sense when the build system and the
system running the scenarios are one and the same, but not so much when
remote systems might well have more CPUs.
This commit therefore introduces a --remote flag to kvm.sh that suppresses
this CPU-limiting behavior, and causes kvm-remote.sh to use this flag.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
In a long-duration kvm-remote.sh run, almost all of the remote accesses will
be simple file-existence checks. These are thus the most likely to be caught
out by network failures, which do happen from time to time.
This commit therefore takes a first step towards tolerating temporary
network outages by making the file-existence checks repeat in the face of
such an outage. They also print a message every minute during a outage,
allowing the user to take appropriate action.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Some of the code invoked directly and indirectly from kvm.sh parses
the output of commands. This parsing assumes English, which can cause
failures if the user has set some other language. In a few cases,
there are language-independent commands available, but this is not
always the case. Therefore, as an alternative to polyglot parsing,
this commit sets the LANG environment variable to en_US.UTF-8.
Reported-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Grepping for "CPU" on lscpu output isn't always successful, depending
on the local language setting. As a result, the build can be aborted
early with:
"make: the '-j' option requires a positive integer argument"
This commit therefore uses the human-language-independent approach
available via the getconf command, both in kvm-build.sh and in
kvm-remote.sh.
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Currently, kvm-find-errors.sh assumes that if "--buildonly" appears in
the log file, then the run did builds but ran no kernels. This breaks
with kvm-remote.sh, which uses kvm.sh to do a build, then kvm-again.sh
to run the kernels built on remote systems. This commit therefore adds
a check for a kvm-remote.sh run.
While in the area, this commit checks for "--build-only" as well as
"--build-only".
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Given remote rcutorture runs, it is quite possible that the build system
will have fewer CPUs than the system(s) running the actual test scenarios.
In such cases, using the number of CPUs on the test systems can overload
the build system, slowing down the build or, worse, OOMing the build
system. This commit therefore uses the build system's CPU count to set
N in "make -jN", and by tradition sets "N" to double the CPU count.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit reduces duplicate code by making kvm.sh use the new
kvm-end-run-stats.sh script rather than taking its historical approach
of open-coding it.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit abstractst the end-of-run summary from kvm-again.sh, and,
while in the area, brings its format into line with that of kvm.sh.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The kvm-again.sh script relies on shell comments added to the qemu-cmd
file, but this means that code extracting values from the QEMU command in
this file must grep out those commment. Which kvm-recheck-rcu.sh failed
to do, which destroyed its grace-period-per-second calculation. This
commit therefore adds the needed "grep -v '^#'" to kvm-recheck-rcu.sh.
Fixes: 315957cad4 ("torture: Prepare for splitting qemu execution from kvm-test-1-run.sh")
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit adds a kvm-remote.sh script that prepares a tarball that
is then downloaded to the remote system(s) and executed. The user is
responsible for having set up the remote systems to run qemu, but all the
kernel builds are done on the system running the kvm-remote.sh script.
The user is also responsible for setting up the remote systems so that
ssh can be run non-interactively, given that ssh is used to poll the
remote systems in order to detect completion of each batch.
See the script's header comment for usage information.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit saves a few lines of code by making kvm-again.sh use the
"scenarios" file rather than the "batches" file, both of which are
generated by kvm.sh.
This results in a break point because new versions of kvm-again.sh cannot
handle "res" directories produced by old versions of kvm.sh, which lack
the "scenarios" file. In the unlikely event that this becomes a problem,
a trivial script suffices to convert the "batches" file to a "scenarios"
file, and this script may be easily extracted from kvm.sh.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit adds "--dryrun scenarios" to kvm.sh, which prints something
like this:
1. TREE03
2. TREE07
3. SRCU-P SRCU-N
4. TREE01 TRACE01
5. TREE02 TRACE02
6. TREE04 RUDE01 TASKS01
7. TREE05 TASKS03 SRCU-T SRCU-U
8. TASKS02 TINY01 TINY02 TREE09
This format is more convenient for scripts that run batches of scenarios.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Although "eval" was removed from torture.sh, that commit failed to
update the KCSAN instance of $* to "$@". This results in failures when
(for example) --bootargs is given more than one argument. This commit
therefore makes this change.
There is one remaining instance of $* in torture.sh, but this
is used only in the "echo" command, where quoting doesn't matter
so much.
Fixes: 197220d4a3 ("torture: Remove use of "eval" in torture.sh")
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Some versions of grep are happy to interpret a nonsensically placed "-"
within a "[]" pattern as a dash, while others give an error message.
This commit therefore places the "-" at the end of the expression where
it was supposed to be in the first place.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Currently, kvm-again.sh updates the duration in the "seconds=" comment
in the qemu-cmd file, but kvm-transform.sh updates the duration in the
actual qemu command arguments. This is an accident waiting to happen.
This commit therefore consolidates these updates into kvm-transform.sh.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The kvm-again.sh script does not copy over the vmlinux files due to
their large size. This means that a gdb run must use the vmlinux file
from the original "res" directory. This commit therefore finds that
directory and prints it out so that the user can copy and pasted the
gdb command just as for the initial run.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Because the TORTURE_TRUST_MAKE environment variable is not recorded,
kvm-again.sh runs can result in the parse-build.sh script emitting
false-positive "BUG: TREE03 no build" messages. These messages are
intended to complain about any lack of compiler invocations when the
--trust-make flag is not given to kvm.sh. However, when this flag is
given to kvm.sh (and thus when TORTURE_TRUST_MAKE=y), lack of compiler
invocations is expected behavior when rebuilding from identical source
code.
This commit therefore makes kvm-test-1-run.sh record the value of the
TORTURE_TRUST_MAKE environment variable as an additional comment in the
qemu-cmd file, and also makes kvm-again.sh reconstitute that variable
from that comment.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
When rerunning an old run using kvm-again.sh, the jitter commands
will re-use the original "res" directory. This works, but is clearly
an accident waiting to happen. And this accident will happen with
remote runs, where the original directory lives on some other system.
This commit therefore updates the qemu-cmd commands to use the new res
directory created for this specific run.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit adds a --duration argument to kvm-again.sh to allow the user
to override the --duration specified for the original kvm.sh run.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit adds a kvm-again.sh script that, given the results directory
of a torture-test run, re-runs that test. This means that the kernels
need not be rebuilt, but it also is a step towards running torture tests
on remote systems.
This commit also adds a kvm-test-1-run-batch.sh script that runs one
batch out of the torture test. The idea is to copy a results directory
tree to remote systems, then use kvm-test-1-run-batch.sh to run batches
on these systems.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit creates a "batches" file in the res/$ds directory, where $ds
is the datestamp. This file contains the batches and the number of CPUs,
for example:
1 TREE03 16
1 SRCU-P 8
2 TREE07 16
2 TREE01 8
3 TREE02 8
3 TREE04 8
3 TREE05 8
4 SRCU-N 4
4 TRACE01 4
4 TRACE02 4
4 RUDE01 2
4 RUDE01.2 2
4 TASKS01 2
4 TASKS03 2
4 SRCU-t 1
4 SRCU-u 1
4 TASKS02 1
4 TINY01 1
5 TINY02 1
5 TREE09 1
The first column is the batch number, the second the scenario number
(possibly suffixed by a repetition number, as in "RUDE01.2"), and the
third is the number of CPUs required by that scenario. The last line
shows the number of CPUs expected by this batch file, which allows
the run to be re-batched if a different number of CPUs is available.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Although it might be unlikely that someone would name a scenario
"TORTURE_SUITE", they are within their rights to do so. This script
therefore renames the "TORTURE_SUITE" file in the top-level date-stamped
directory within "res" to "torture_suite" to avoid this name collision.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit enforces the defacto restriction on scenario names, which is
that they contain neither "/", ".", nor lowercase alphabetic characters.
This restriction avoids collisions between scenario names and the torture
scripting's files and directories.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
The cpus2use.sh script complains if the mpstat command is not available,
and instead uses all available CPUs. Unfortunately, this complaint
goes to stdout, where it confuses invokers who expect a single number.
This commit removes this error message in order to avoid this confusion.
The tendency of late has been to give rcutorture a full system, so this
should not cause issues.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit records the process IDs of the kvm-test-1-run.sh and
kvm-test-1-run-qemu.sh scripts to ease monitoring of remotely running
instances of these scripts.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Distributed runs of rcutorture will need to start and stop jittering on
the remote hosts, which means that the commands must be communicated to
those hosts. The commit therefore causes kvm.sh to place these commands
in new TORTURE_JITTER_START and TORTURE_JITTER_STOP environment variables
to communicate them to the scripts that will set this up. In addition,
this commit causes kvm-test-1-run.sh to append these commands to each
generated qemu-cmd file, which allows any remotely executing script to
extract the needed commands from this file.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Currently, kvm-test-1-run.sh both builds and runs an rcutorture kernel,
which is inconvenient when it is necessary to re-run an old run or to
carry out a run on a remote system. This commit therefore extracts the
portion of kvm-test-1-run.sh that invoke qemu to actually run rcutorture
and places it in kvm-test-1-run-qemu.sh.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
When re-running old rcutorture builds, if the original run involved
gdb, the re-run also needs to do so. This commit therefore records the
TORTURE_KCONFIG_GDB_ARG environment variable into the qemu-cmd file so
that the re-run can access it.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
This commit creates jitterstart.sh and jitterstop.sh scripts that handle
the starting and stopping of the jitter.sh scripts. These must be sourced
using the bash "." command to allow the generated script to wait on the
backgrounded jitter.sh scripts.
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
