Developer support to emulate unsatisfied prerequisites in tests to
ensure that the remainer of the tests still succeeds when tests
with prerequisites are skipped.
* ab/fail-prereqs-in-test:
tests: add a special setup where prerequisites fail
A prerequiste check in the test suite to see if a working jgit is
available was made more robust.
* tz/test-lib-check-working-jgit:
test-lib: try harder to ensure a working jgit
The JGIT prereq uses `type jgit` to determine whether jgit is present.
While this is usually sufficient, it won't help if the jgit found is
badly broken. This wastes time running tests which fail due to no fault
of our own.
Use `jgit --version` instead, to guard against cases where jgit is
present on the system, but will fail to run, e.g. because of some JRE
issue, or missing Java dependencies. Checking that it gets far enough
to process the '--version' argument isn't perfect, but seems to be good
enough in practice. It's also consistent with how we detect some other
dependencies, see e.g. the CURL and UNZIP prerequisites.
Signed-off-by: Todd Zullinger <tmz@pobox.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
As discussed in [1] there's a regression in the "pu" branch now
because a new test implicitly assumed that a previous test guarded by
a prerequisite had been run. Add a "GIT_TEST_FAIL_PREREQS" special
test setup where we'll skip (nearly) all tests guarded by
prerequisites, allowing us to easily emulate those platform where we
don't run these tests.
As noted in the documentation I'm adding I'm whitelisting the SYMLINKS
prerequisite for now. A lot of tests started failing if we lied about
not supporting symlinks. It's also unlikely that we'll have a failing
test due to a hard dependency on symlinks without that being the
obvious cause, so for now it's not worth the effort to make it work.
1. https://public-inbox.org/git/nycvar.QRO.7.76.6.1905131531000.44@tvgsbejvaqbjf.bet/
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The GETTEXT_POISON test option has been quite broken ever since it
was made runtime-tunable, which has been fixed.
* jc/gettext-test-fix:
gettext tests: export the restored GIT_TEST_GETTEXT_POISON
Test framework update to more robustly clean up leftover files and
processes after tests are done.
* sg/test-atexit:
t9811-git-p4-label-import: fix pipeline negation
git p4 test: disable '-x' tracing in the p4d watchdog loop
git p4 test: simplify timeout handling
git p4 test: clean up the p4d cleanup functions
git p4 test: use 'test_atexit' to kill p4d and the watchdog process
t0301-credential-cache: use 'test_atexit' to stop the credentials helper
tests: use 'test_atexit' to stop httpd
git-daemon: use 'test_atexit` to stop 'git-daemon'
test-lib: introduce 'test_atexit'
t/lib-git-daemon: make sure to kill the 'git-daemon' process
test-lib: fix interrupt handling with 'dash' and '--verbose-log -x'
6cdccfce ("i18n: make GETTEXT_POISON a runtime option", 2018-11-08)
made the gettext-poison test a runtime option (which was a good
move) and adjusted the test framework so that Git commands we run as
part of the framework, as opposed to the ones that are part of the
test proper, are not affected by the setting. The original value
for the GIT_TEST_GETTEXT_POISON environment variable is saved away
in another variable and gets unset, and then later the saved value
is restored to the environment variable.
But the code forgot to export the variable again, which is necessary
to restore the "export" bit that was lost when the variable was unset.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Git's command-line parsers support uniquely abbreviated options, e.g.
`git init --ba` would automatically expand `--ba` to `--bare`.
This is a very convenient feature in every day life for Git users, in
particular when tab completion is not available.
However, it is not a good idea to rely on that in Git's test suite, as
something that is a unique abbreviation of a command line option today
might no longer be a unique abbreviation tomorrow.
For example, if a future contribution added a new mode
`git init --babyproofing` and a previously-introduced test case used the
fact that `git init --ba` expanded to `git init --bare`, that future
contribution would now have to touch seemingly unrelated tests just to
keep the test suite from failing.
So let's disallow abbreviated options in the test suite by default.
Note: for ease of implementation, this patch really only touches the
`parse-options` machinery: more and more hand-rolled option parsers are
converted to use that internal API, and more and more scripts are
converted to built-ins (naturally using the parse-options API, too), so
in practice this catches most issues, and is definitely the biggest bang
for the buck.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Add GIT_TR2_* to the whitelist of environment variables that we don't
clear when running the test suite.
This allows us to use the test suite to produce trace2 test data,
which is handy to e.g. write consumers that collate the trace data
itself.
One caveat here is that we produce trace output for not *just* the
tests, but also e.g. from this line in test-lib.sh:
# It appears that people try to run tests without building...
"${GIT_TEST_INSTALLED:-$GIT_BUILD_DIR}/git$X" >/dev/null
[...]
I consider this not just OK but a feature. Let's log *all* the git
commands we're going to execute, not just those within
test_expect_*().
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When running Apache, 'git daemon', or p4d, we want to kill them at the
end of the test script, otherwise a leftover daemon process will keep
its port open indefinitely, and thus will interfere with subsequent
executions of the same test script.
So far, we stop these daemon processes "manually", i.e.:
- by registering functions or commands in the trap on EXIT to stop
the daemon while preserving the last seen exit code before the
trap (to deal with a failure when run with '--immediate' or with
interrupts by ctrl-C),
- and by invoking these functions/commands last thing before
'test_done' (and sometimes restoring the test framework's default
trap on EXIT, to prevent the daemons from being killed twice).
On one hand, we do this inconsistently, e.g. 'git p4' tests invoke
different functions in the trap on EXIT and in the last test before
'test_done', and they neither restore the test framework's default trap
on EXIT nor preserve the last seen exit code. On the other hand, this
is error prone, because, as shown in a previous patch in this series,
any output from the cleanup commands in the trap on EXIT can prevent a
proper cleanup when a test script run with '--verbose-log' and certain
shells, notably 'dash', is interrupted.
Let's introduce 'test_atexit', which is loosely modeled after
'test_when_finished', but has a broader scope: rather than running the
commands after the current test case, run them when the test script
finishes, and also run them when the test is interrupted, or exits
early in case of a failure while the '--immediate' option is in
effect.
When running the cleanup commands at the end of a successful test,
then they will be run in 'test_done' before it removes the trash
directory, i.e. the cleanup commands will still be able to access any
pidfiles or socket files in there. When running the cleanup commands
after an interrupt or failure with '--immediate', then they will be
run in the trap on EXIT. In both cases they will be run in
'test_eval_', i.e. both standard error and output of all cleanup
commands will go where they should according to the '-v' or
'--verbose-log' options, and thus won't cause any troubles when
interrupting a test script run with '--verbose-log'.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When a test script run with 'dash' and '--verbose-log -x' is
interrupted by ctrl-C, SIGTERM, or closing the terminal window, then
most of the time the registered EXIT trap actions are not executed.
This is an annoying issue with tests involving daemons, because they
should run cleanup commands to kill those daemon processes in the trap
on EXIT, but since these cleanup commands are not executed, the
daemons are left alive and keep their port open, thus interfering with
subsequent execution of the same test script.
The cause of this issue is the subtle combination of several factors
(bear with me, or skip over the indented part):
- Even when the test script is interrupted, the cleanup commands are
not run in the trap on INT, TERM, or HUP, but in the trap on EXIT
after the trap on the signals invokes 'exit' [1].
- According to POSIX [2]:
"The environment in which the shell executes a trap on EXIT
shall be identical to the environment immediately after the last
command executed before the trap on EXIT was taken."
Pertinent to the issue at hand is that all open file descriptors
and the state of '-x' tracing should be preserved. All shells
I've tried [3] preserve '-x'. Unfortunately, however:
- 'dash' doesn't conform to this when it comes to open file
descriptors: even when standard output and/or error are
redirected somewhere when 'exit' is invoked, anything written
to them in the trap on EXIT goes to the script's original
stdout and stderr [4].
We can't dismiss this with a simple "it doesn't conform to
POSIX, so we don't care", because 'dash' is the default
/bin/sh in some of the more popular Linux distros.
- As far as I can tell, POSIX doesn't explicitly say anything
about the environment of trap actions for various signals.
In practice it seems that most shells behave sensibly and
preserve both open file descriptors and the state of '-x'
tracing for the traps on INT, TERM, and HUP, including even
'dash'. The exceptions are 'mksh' and 'lksh': they do
preserve '-x', but not the open file descriptors.
- When a test script run with '-x' tracing enabled is interrupted,
then it's very likely that the signal arrives mid-test, i.e.:
- while '-x' tracing is enabled, and, consequently, our trap
actions on INT, TERM, HUP, and EXIT will produce trace output
as well.
- while standard output and error are redirected to a log file,
to the test script's original standard output and error, or to
/dev/null, depending on whether the test script was run with
'--verbose-log', '-v', or neither. According to the above, we
can't rely on these redirections still be in effect when
running the traps on INT, TERM, HUP, and/or EXIT.
- When a test script is run with '--verbose-log', then the test
script is re-executed with its standard output and error piped
into 'tee', in order to send the "regular" non-verbose test's
output both to the terminal and to the log file. When the test is
interrupted, then the signal interrupts the downstream 'tee' as
well.
Putting these together, when a test script run with 'dash' and
'--verbose-log -x' is interrupted, then 'dash' tries to write the
trace output from the EXIT trap to the script's original standard
error, but it very likely can't, because the 'tee' downstream of the
pipe is interrupted as well. This causes the shell running the test
script to die because of SIGPIPE, without running any of the commands
in the EXIT trap.
Disable '-x' tracing in the trap on INT, TERM, and HUP to avoid this
issue, as it disables tracing in the chained trap on EXIT as well.
Wrap it in a '{ ... } 2>/dev/null' block, so the trace of the command
disabling the tracing doesn't go to standard error either [5].
Note that it's not only '-x' tracing that can be problematic, but any
shell builtin, e.g. 'echo', that writes to standard output or error in
the trap on EXIT, while a test running with 'dash' and '--verbose-log'
(even without '-x') is interrupted. As far as I can tell, this is not
an issue at the moment:
- The cleanup commands to stop the credential-helper, Apache, or
'p4d' don't use any such shell builtins.
- stop_git_daemon() does use 'say' and 'error', both wrappers around
'echo', but it redirects 'say' to fd 3, i.e. to the log file, and
while 'error' does write to standard output, it comes only after
the daemon was killed.
- The non-builtin commands that actually stop the daemons ('kill',
'apache2 -k stop', 'git credential-cache exit') are silent, so they
won't get SIGPIPE before finishing their job.
[1] The trap on EXIT must run cleanup commands, because we want to
stop any daemons when a test script run with '--immediate' fails
and exits early with error. By chaining up the trap on signals to
the trap on EXIT we can deal with cleanup commands a bit simpler,
because the tests involving daemons only have to set a single
trap.
[2] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#trap
[3] The shells I tried: dash, Bash, ksh, ksh93, mksh, lksh, yash,
BusyBox sh, FreeBSD /bin/sh, NetBSD /bin/sh.
[4] $ cat trap-output.sh
#!/bin/sh
trap "echo output; echo error >&2" EXIT
{ exit; } >OUT 2>ERR
$ dash ./trap-output.sh
output
error
$ wc -c OUT ERR
0 OUT
0 ERR
On a related note, 'ksh', 'ksh93', and BusyBox sh don't conform to
the specs in this respect, either.
[5] This '{ set +x; } 2>/dev/null' trick won't help those shells that
show trace output for any redirections and don't preserve open
file descriptors for the trap on INT, TERM and HUP. The only such
shells I'm aware of are 'mksh' and 'lksh'.
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The --stress option currently accepts an argument, but it is confusing
to at least this user that the argument does not define the maximal
number of stress iterations, but instead the number of jobs to run in
parallel per stress iteration.
Let's introduce a separate option for that, whose name makes it more
obvious what it is about, and let --stress=<N> error out with a helpful
suggestion about the two options tha could possibly have been meant.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
It does not make much sense that running a test with
--stress-limit=<N> seemingly ignores that option because it does not
stress test at all.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Use a '!' character to start a non-matching pattern bracket
expression, as specified by POSIX in Shell Command Language section
2.13.1 Patterns Matching a Single Character [1].
I used '^' instead in three places in the previous three commits, to
verify that the arguments of the '--stress=' and '--stress-limit='
options and the values of various '*_PORT' environment variables are
valid numbers. With certain shells, at least with dash (upstream and
in Ubuntu 14.04) and mksh, this led to various undesired behaviors:
# error message in case of a valid number
$ ~/src/dash/src/dash ./t3903-stash.sh --stress=8
error: --stress=<N> requires the number of jobs to run
# not the expected error message
$ ~/src/dash/src/dash ./t3903-stash.sh --stress=foo
./t3903-stash.sh: 238: test: Illegal number: foo
# no error message at all?!
$ mksh ./t3903-stash.sh --stress=foo
$ echo $?
0
Some other shells, e.g. Bash (even in posix mode), ksh, dash in Ubuntu
16.04 or later, are apparently happy to accept '^' just as well.
[1] http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_13
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
During a review of a patch, we noticed that we use our own imitation
of 'yes' with the limit of 99 lines. It is very tempting to lift this
arbitrary limit, but the limit is there for a reason.
Add an in-code comment to prevent future developers from wasting
their time.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Let's suppose that a test somehow becomes flaky between 'master' and
'pu', and tends to fail within the first 50 repetitions when run with
'--stress'. In such a case we could use 'git bisect' to find the
culprit: if the test script fails with '--stress', then the commit is
definitely bad, but if it survives, say, 300 repetitions, then we could
consider it good with reasonable confidence.
Unfortunately, all this could only be done manually, because
'--stress' would run the test script repeatedly for all eternity on a
good commit, and it would exit with success even when it found a
failure on a bad commit.
So let's make '--stress' usable with 'git bisect run':
- Make it exit with failure if a failure is found.
- Add the '--stress-limit=<N>' option to repeat the test script
at most N times in each of the parallel jobs, and exit with
success when the limit is reached.
And then we could simply run something like:
$ git bisect start origin/pu master
$ git bisect run sh -c 'make && cd t &&
./t1234-foo.sh --stress --stress-limit=300'
Sure, as a brand new feature it won't be any useful right now, but in
a release or three most cooking topics will already contain this, so
we could automatically bisect at least newly introduced flakiness.
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Prepare to run test suite on Azure Pipeline.
* js/vsts-ci: (22 commits)
test-date: drop unused parameter to getnanos()
ci: parallelize testing on Windows
ci: speed up Windows phase
tests: optionally skip bin-wrappers/
t0061: workaround issues with --with-dashes and RUNTIME_PREFIX
tests: add t/helper/ to the PATH with --with-dashes
mingw: try to work around issues with the test cleanup
tests: include detailed trace logs with --write-junit-xml upon failure
tests: avoid calling Perl just to determine file sizes
README: add a build badge (status of the Azure Pipelines build)
mingw: be more generous when wrapping up the setitimer() emulation
ci: use git-sdk-64-minimal build artifact
ci: add a Windows job to the Azure Pipelines definition
Add a build definition for Azure DevOps
ci/lib.sh: add support for Azure Pipelines
tests: optionally write results as JUnit-style .xml
test-date: add a subcommand to measure times in shell scripts
ci: use a junction on Windows instead of a symlink
ci: inherit --jobs via MAKEFLAGS in run-build-and-tests
ci/lib.sh: encapsulate Travis-specific things
...
This speeds up the tests by a bit on Windows, where running Unix shell
scripts (and spawning processes) is not exactly a cheap operation.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
We really need to be able to find the test helpers... Really. This
change was forgotten when we moved the test helpers into t/helper/
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
It seems that every once in a while in the Git for Windows SDK, there
are some transient file locking issues preventing the test clean up to
delete the trash directory. Let's be gentle and try again five seconds
later, and only error out if it still fails the second time.
This change helps Windows, and does not hurt any other platform
(normally, it is highly unlikely that said deletion fails, and if it
does, normally it will fail again even 5 seconds later).
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The JUnit XML format lends itself to be presented in a powerful UI,
where you can drill down to the information you are interested in very
quickly.
For test failures, this usually means that you want to see the detailed
trace of the failing tests.
With Travis CI, we passed the `--verbose-log` option to get those
traces. However, that seems excessive, as we do not need/use the logs in
almost all of those cases: only when a test fails do we have a way to
include the trace.
So let's do something different when using Azure DevOps: let's run all
the tests with `--quiet` first, and only if a failure is encountered,
try to trace the commands as they are executed.
Of course, we cannot turn on `--verbose-log` after the fact. So let's
just re-run the test with all the same options, adding `--verbose-log`.
And then munging the output file into the JUnit XML on the fly.
Note: there is an off chance that re-running the test in verbose mode
"fixes" the failures (and this does happen from time to time!). That is
a possibility we should be able to live with. Ideally, we would label
this as "Passed upon rerun", and Azure Pipelines even know about that
outcome, but it is not available when using the JUnit XML format for
now:
https://github.com/Microsoft/azure-pipelines-agent/blob/master/src/Agent.Worker/TestResults/JunitResultReader.cs
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This will come in handy when publishing the results of Git's test suite
during an automated Azure DevOps run.
Note: we need to make extra sure that invalid UTF-8 encoding is turned
into valid UTF-8 (using the Replacement Character, \uFFFD) because
t9902's trace contains such invalid byte sequences, and the task in the
Azure Pipeline that uploads the test results would refuse to do anything
if it was asked to parse an .xml file with invalid UTF-8 in it.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In 8abfdf44c8 (tests: explicitly use `git.exe` on Windows,
2018-11-14), we made sure to use the `.exe` file extension when
using an absolute path to `git.exe`, to avoid getting confused with a
file or directory in the same place that lacks said file extension.
For the same reason, we need to handle test-tool.exe the same way.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Unfortunately, we have a few flaky tests, whose failures tend to be
hard to reproduce. We've found that the best we can do to reproduce
such a failure is to run the test script repeatedly while the machine
is under load, and wait in the hope that the load creates enough
variance in the timing of the test's commands that a failure is
evenually triggered. I have a command to do that, and I noticed that
two other contributors have rolled their own scripts to do the same,
all choosing slightly different approaches.
To help reproduce failures in flaky tests, introduce the '--stress'
option to run a test script repeatedly in multiple parallel jobs until
one of them fails, thereby using the test script itself to increase
the load on the machine.
The number of parallel jobs is determined by, in order of precedence:
the number specified as '--stress=<N>', or the value of the
GIT_TEST_STRESS_LOAD environment variable, or twice the number of
available processors (as reported by the 'getconf' utility), or 8.
Make '--stress' imply '--verbose -x --immediate' to get the most
information about rare failures; there is really no point in spending
all the extra effort to reproduce such a failure, and then not know
which command failed and why.
To prevent the several parallel invocations of the same test from
interfering with each other:
- Include the parallel job's number in the name of the trash
directory and the various output files under 't/test-results/' as
a '.stress-<Nr>' suffix.
- Add the parallel job's number to the port number specified by the
user or to the test number, so even tests involving daemons
listening on a TCP socket can be stressed.
- Redirect each parallel test run's verbose output to
't/test-results/$TEST_NAME.stress-<nr>.out', because dumping the
output of several parallel running tests to the terminal would
create a big ugly mess.
For convenience, print the output of the failed test job at the end,
and rename its trash directory to end with the '.stress-failed'
suffix, so it's easy to find in a predictable path (OTOH, all absolute
paths recorded in the trash directory become invalid; we'll see
whether this causes any issues in practice). If, in an unlikely case,
more than one jobs were to fail nearly at the same time, then print
the output of all failed jobs, and rename the trash directory of only
the last one (i.e. with the highest job number), as it is the trash
directory of the test whose output will be at the bottom of the user's
terminal.
Based on Jeff King's 'stress' script.
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
A later patch in this series will need to know the path to the trash
directory early in 'test-lib.sh', but $TRASH_DIRECTORY is set much
later.
Set $TRASH_DIRECTORY earlier, where the other test-specific path
variables are set.
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
There are two places where we strip off any leading path components
and the '.sh' suffix from the test script's pathname, and there are
four places where we construct the name of the 't/test-results'
directory or the name of various test-specific files in there. The
last patch in this series will add even more.
Factor these out into helper variables to avoid repeating ourselves.
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
'test-lib.sh' looks for the presence of certain options like '--tee'
and '--verbose-log', so it can execute the test script again to save
its standard output and error. It looks for '--valgrind' as well, to
set up some Valgrind-specific stuff. These all happen before the
actual option parsing loop, and the conditions looking for these
options look a bit odd, too. They are not completely correct, either,
because in a bogus invocation like './t1234-foo.sh -r --tee' they
recognize '--tee', although it should be handled as the required
argument of the '-r' option. This patch series will add two more
options to look out for early, and, in addition, will have to extract
these options' stuck arguments (i.e. '--opt=arg') as well.
So let's move the option parsing loop and the couple of related
conditions following it earlier in 'test-lib.sh', before the place
where the test script is executed again for '--tee' and its friends.
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
'test-lib.sh' looks for the presence of certain options like '--tee'
and '--verbose-log', so it can execute the test script again to save
its standard output and error, and to do so it needs the original
command line options the test was invoked with.
The next patch is about to move the option parsing loop earlier in
'test-lib.sh', but it is implemented using 'shift' in a while loop,
effecively destroying "$@" by the end of the option parsing. Not
good.
As a preparatory step, turn that option parsing loop into a 'for opt
in "$@"' loop to preserve "$@" intact while iterating over the
options, and taking extra care to handle the '-r' option's required
argument (or the lack thereof).
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
One of our test scripts, 't1510-repo-setup.sh' [1], still can't be
reliably run with '-x' tracing enabled, unless it's executed with a
Bash version supporting BASH_XTRACEFD (since v4.1). We have a lengthy
condition to check the version of the shell running the test script,
and disable tracing if it's not executed with a suitable Bash version
[2].
Move this check out from the option parsing loop, so other options can
imply '-x' by setting 'trace=t', without missing this Bash version
check.
[1] 5827506928 (t1510-repo-setup: mark as untraceable with '-x',
2018-02-24)
[2] 5fc98e79fc (t: add means to disable '-x' tracing for individual
test scripts, 2018-02-24)
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Right now if a test script receives SIGTERM or SIGHUP (e.g., because a
test was hanging and the user 'kill'-ed it or simply closed the
terminal window the test was running in), the shell exits immediately.
This can be annoying if the test script did any global setup, like
starting apache or git-daemon, as it will not have an opportunity to
clean up after itself. A subsequent run of the test won't be able to
start its own daemon, and will either fail or skip the tests.
Instead, let's trap SIGTERM and SIGHUP as well to make sure we do a
clean shutdown, and just chain it to a normal exit (which will trigger
any cleanup).
This patch follows suit of da706545f7 (t: translate SIGINT to an exit,
2015-03-13), and even stole its commit message as well.
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
One of our test scripts, 't1510-repo-setup.sh' [1], still can't be
reliably run with '-x' tracing enabled, unless it's executed with a
Bash version supporting BASH_XTRACEFD (since v4.1). We have a lengthy
condition to check the version of the shell running the test script,
and disable tracing if it's not executed with a suitable Bash version
[2].
This condition uses non-portable shell array accesses to easily get
Bash's major and minor version number. This didn't seem to be
problematic, because the simple commands expanding those array
accesses are only executed when the test script is actually run with
Bash. When run with Dash, the only shell I have at hand that doesn't
support shell arrays, there are no issues, as it apparently skips
right over the non-executed simple commands without noticing the
non-supported constructs.
Alas, it has been reported that NetBSD's /bin/sh does complain about
them:
./test-lib.sh: 327: Syntax error: Bad substitution
where line 327 contains the first ${BASH_VERSINFO[0]} array access.
To my understanding both shells are right and conform to POSIX,
because the standard allows both behaviors by stating the following
under '2.8.1 Consequences of Shell Errors' [3]:
"An expansion error is one that occurs when the shell expansions
define in wordexp are carried out (for example, "${x!y}", because
'!' is not a valid operator); an implementation may treat these as
syntax errors if it is able to detect them during tokenization,
rather than during expansion."
Avoid this issue with NetBSD's /bin/sh (and potentially with other,
less common shells) by hiding the shell array syntax behind 'eval'
that is only executed with Bash.
[1] 5827506928 (t1510-repo-setup: mark as untraceable with '-x',
2018-02-24)
[2] 5fc98e79fc (t: add means to disable '-x' tracing for individual
test scripts, 2018-02-24)
[3] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_08_01
Reported-by: Max Kirillov <max@max630.net>
Helped-by: Johannes Sixt <j6t@kdbg.org>
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Some of the functions in our test library check that they were invoked
properly with conditions like this:
test "$#" = 2 ||
error "bug in the test script: not 2 parameters to test-expect-success"
If this particular condition is triggered, then 'error' will abort the
whole test script with a bold red error message [1] right away.
However, under certain circumstances the test script will be aborted
completely silently, namely if:
- a similar condition in a test helper function like
'test_line_count' is triggered,
- which is invoked from the test script's "main" shell [2],
- and the test script is run manually (i.e. './t1234-foo.sh' as
opposed to 'make t1234-foo.sh' or 'make test') [3]
- and without the '--verbose' option,
because the error message is printed from within 'test_eval_', where
standard output is redirected either to /dev/null or to a log file.
The only indication that something is wrong is that not all tests in
the script are executed and at the end of the test script's output
there is no "# passed all N tests" message, which are subtle and can
easily go unnoticed, as I had to experience myself.
Send these "bug in the test script" error messages directly to the
test scripts standard error and thus to the terminal, so those bugs
will be much harder to overlook. Instead of updating all ~20 such
'error' calls with a redirection, let's add a BUG() function to
'test-lib.sh', wrapping an 'error' call with the proper redirection
and also including the common prefix of those error messages, and
convert all those call sites [4] to use this new BUG() function
instead.
[1] That particular error message from 'test_expect_success' is
printed in color only when running with or without '--verbose';
with '--tee' or '--verbose-log' the error is printed without
color, but it is printed to the terminal nonetheless.
[2] If such a condition is triggered in a subshell of a test, then
'error' won't be able to abort the whole test script, but only the
subshell, which in turn causes the test to fail in the usual way,
indicating loudly and clearly that something is wrong.
[3] Well, 'error' aborts the test script the same way when run
manually or by 'make' or 'prove', but both 'make' and 'prove' pay
attention to the test script's exit status, and even a silently
aborted test script would then trigger those tools' usual
noticable error messages.
[4] Strictly speaking, not all those 'error' calls need that
redirection to send their output to the terminal, see e.g.
'test_expect_success' in the opening example, but I think it's
better to be consistent.
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Update the "test installed Git" mode of our test suite to work better.
* js/test-git-installed:
tests: explicitly use `git.exe` on Windows
tests: do not require Git to be built when testing an installed Git
t/lib-gettext: test installed git-sh-i18n if GIT_TEST_INSTALLED is set
tests: respect GIT_TEST_INSTALLED when initializing repositories
tests: fix GIT_TEST_INSTALLED's PATH to include t/helper/
Our testing framework uses a special i18n "poisoned localization"
feature to find messages that ought to stay constant but are
incorrectly marked to be translated. This feature has been made
into a runtime option (it used to be a compile-time option).
* ab/dynamic-gettext-poison:
Makefile: ease dynamic-gettext-poison transition
i18n: make GETTEXT_POISON a runtime option
On Windows, when we refer to `/an/absolute/path/to/git`, it magically
resolves `git.exe` at that location. Except if something of the name
`git` exists next to that `git.exe`. So if we call `$BUILD_DIR/git`, it
will find `$BUILD_DIR/git.exe` *only* if there is not, say, a directory
called `$BUILD_DIR/git`.
Such a directory, however, exists in Git for Windows when building with
Visual Studio (our Visual Studio project generator defaults to putting
the build files into a directory whose name is the base name of the
corresponding `.exe`).
In the bin-wrappers/* scripts, we already take pains to use `git.exe`
rather than `git`, as this could pick up the wrong thing on Windows
(i.e. if there exists a `git` file or directory in the build directory).
Now we do the same in the tests' start-up code.
This also helps when testing an installed Git, as there might be even
more likely some stray file or directory in the way.
Note: the only way we can record whether the `.exe` suffix is by writing
it to the `GIT-BUILD-OPTIONS` file and sourcing it at the beginning of
`t/test-lib.sh`. This is not a requirement introduced by this patch, but
we move the call to be able to use the `$X` variable that holds the file
extension, if any.
Note also: the many, many calls to `git this` and `git that` are
unaffected, as the regular PATH search will find the `.exe` files on
Windows (and not be confused by a directory of the name `git` that is
in one of the directories listed in the `PATH` variable), while
`/path/to/git` would not, per se, know that it is looking for an
executable and happily prefer such a directory.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
We really only need the test helpers to be built in the worktree in that
case, but that is not what we test for.
On the other hand it is a perfect opportunity to verify that
`GIT_TEST_INSTALLED` points to a working Git.
So let's test the appropriate Git executable. While at it, also adjust
the error message in the `GIT_TEST_INSTALLED` case.
This patch is best viewed with `-w --patience`.
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
We really need to be able to find the test helpers... Really. This
change was forgotten when we moved the test helpers into t/helper/
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In preparation to the day when we can deprecate and remove the
"rebase -p", make sure we can skip and later remove tests for
it.
* js/rebase-p-tests:
tests: optionally skip `git rebase -p` tests
t3418: decouple test cases from a previous `rebase -p` test case
t3404: decouple some test cases from outcomes of previous test cases
Change the GETTEXT_POISON compile-time + runtime GIT_GETTEXT_POISON
test parameter to only be a GIT_TEST_GETTEXT_POISON=<non-empty?>
runtime parameter, to be consistent with other parameters documented
in "Running tests with special setups" in t/README.
When I added GETTEXT_POISON in bb946bba76 ("i18n: add GETTEXT_POISON
to simulate unfriendly translator", 2011-02-22) I was concerned with
ensuring that the _() function would get constant folded if NO_GETTEXT
was defined, and likewise that GETTEXT_POISON would be compiled out
unless it was defined.
But as the benchmark in my [1] shows doing a one-off runtime
getenv("GIT_TEST_[...]") is trivial, and since GETTEXT_POISON was
originally added the GIT_TEST_* env variables have become the common
idiom for turning on special test setups.
So change GETTEXT_POISON to work the same way. Now the
GETTEXT_POISON=YesPlease compile-time option is gone, and running the
tests with GIT_TEST_GETTEXT_POISON=[YesPlease|] can be toggled on/off
without recompiling.
This allows for conditionally amending tests to test with/without
poison, similar to what 859fdc0c3c ("commit-graph: define
GIT_TEST_COMMIT_GRAPH", 2018-08-29) did for GIT_TEST_COMMIT_GRAPH. Do
some of that, now we e.g. always run the t0205-gettext-poison.sh test.
I did enough there to remove the GETTEXT_POISON prerequisite, but its
inverse C_LOCALE_OUTPUT is still around, and surely some tests using
it can be converted to e.g. always set GIT_TEST_GETTEXT_POISON=.
Notes on the implementation:
* We still compile a dedicated GETTEXT_POISON build in Travis
CI. Perhaps this should be revisited and integrated into the
"linux-gcc" build, see ae59a4e44f ("travis: run tests with
GIT_TEST_SPLIT_INDEX", 2018-01-07) for prior art in that area. Then
again maybe not, see [2].
* We now skip a test in t0000-basic.sh under
GIT_TEST_GETTEXT_POISON=YesPlease that wasn't skipped before. This
test relies on C locale output, but due to an edge case in how the
previous implementation of GETTEXT_POISON worked (reading it from
GIT-BUILD-OPTIONS) wasn't enabling poison correctly. Now it does,
and needs to be skipped.
* The getenv() function is not reentrant, so out of paranoia about
code of the form:
printf(_("%s"), getenv("some-env"));
call use_gettext_poison() in our early setup in git_setup_gettext()
so we populate the "poison_requested" variable in a codepath that's
won't suffer from that race condition.
* We error out in the Makefile if you're still saying
GETTEXT_POISON=YesPlease to prompt users to change their
invocation.
* We should not print out poisoned messages during the test
initialization itself to keep it more readable, so the test library
hides the variable if set in $GIT_TEST_GETTEXT_POISON_ORIG during
setup. See [3].
See also [4] for more on the motivation behind this patch, and the
history of the GETTEXT_POISON facility.
1. https://public-inbox.org/git/871s8gd32p.fsf@evledraar.gmail.com/
2. https://public-inbox.org/git/20181102163725.GY30222@szeder.dev/
3. https://public-inbox.org/git/20181022202241.18629-2-szeder.dev@gmail.com/
4. https://public-inbox.org/git/878t2pd6yu.fsf@evledraar.gmail.com/
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Our test scripts can now take the '-V' option as a synonym for the
'--verbose-log' option.
* sg/test-verbose-log:
test-lib: introduce the '-V' short option for '--verbose-log'
The `--preserve-merges` mode of the `rebase` command is slated to be
deprecated soon, as the more powerful `--rebase-merges` mode is
available now, and the latter was designed with the express intent to
address the shortcomings of `--preserve-merges`' design (e.g. the
inability to reorder commits in an interactive rebase).
As such, we will eventually even remove the `--preserve-merges` support,
and along with it, its tests.
In preparation for this, and also to allow the Windows phase of our
automated tests to save some well-needed time when running the test
suite, this commit introduces a new prerequisite REBASE_P, which can be
forced to being unmet by setting the environment variable
`GIT_TEST_SKIP_REBASE_P` to any non-empty string.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
'--verbose-log' is one of the most useful and thus most frequently
used test options, but due to its length it's a pain to type on the
command line.
Let's introduce the corresponding short option '-V' to save some
keystrokes.
Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Some environment variables that control the runtime options of Git
used during tests are getting renamed for consistency.
* bp/rename-test-env-var:
t0000: do not get self-test disrupted by environment warnings
preload-index: update GIT_FORCE_PRELOAD_TEST support
read-cache: update TEST_GIT_INDEX_VERSION support
fsmonitor: update GIT_TEST_FSMONITOR support
preload-index: use git_env_bool() not getenv() for customization
t/README: correct spelling of "uncommon"
The test framework test-lib.sh itself would want to give warnings
and hints, e.g. when it sees a deprecated environment variable is in
use that we want to encourage users to migrate to another variable.
The self-test of test framework done in t0000 however do not expect
to see these warnings and hints, so depending on the settings of
environment variables, a running test may or may not produce these
messages to the standard error output, breaking the expectations of
self-test test framework does on itself. Here is what we see:
$ TEST_GIT_INDEX_VERSION=4 sh t0000-basic.sh -i -v
...
'err' is not empty, it contains:
warning: TEST_GIT_INDEX_VERSION is now GIT_TEST_INDEX_VERSION
hint: set GIT_TEST_INDEX_VERSION too during the transition period
not ok 5 - pretend we have a fully passing test suite
The following quick attempt to work it around does not work, because
some tests in t0000 do want to see expected errors from the test
framework itself.
t/t0000-basic.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/t/t0000-basic.sh b/t/t0000-basic.sh
index 850f651e4e..88c6ed4696 100755
--- a/t/t0000-basic.sh
+++ b/t/t0000-basic.sh
@@ -88,7 +88,7 @@ _run_sub_test_lib_test_common () {
'
# Point to the t/test-lib.sh, which isn't in ../ as usual
- . "\$TEST_DIRECTORY"/test-lib.sh
+ . "\$TEST_DIRECTORY"/test-lib.sh >/dev/null 2>&1
EOF
cat >>"$name.sh" &&
chmod +x "$name.sh" &&
There are a few possible ways to work this around:
* We could strip the warning: and hint: unconditionally from the
error output before the error messages are checked in the
self-test (helper functions check_sub_test_lib_test_err and
check_sub_test_lib_test); the problem with this approach is that
it will make it impossible to write self-tests to ensure that
right warnings and hints are given.
* We could force a sane environment settings before the test helper
_run_sub_test_lib_test_common dot-sources test-lib.sh; the
problem with this approach is that _run_sub_test_lib_test_common
now needs to be aware of what pairs of environment variables are
checked in test-lib.sh using check_var_migration helper.
The final patch I came up with is probably the solution that is
least bad. Set a variable to tell test-lib.sh that we are running
a self-test, so that various pieces in test-lib.sh can react to keep
the output stable.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
