mirror of
https://github.com/git/git
synced 2024-08-27 11:39:22 +00:00
test-lib: allow selecting tests by substring/glob with --run
Many of our test scripts have several "setup" tests. It's a lot easier to say ./t0050-filesystem.sh --run=setup,9 in order to run all the setup tests as well as test #9, than it is to track down what all the setup tests are and enter all their numbers in the list. Also, I often find myself wanting to run just one or a couple tests from the test file, but I don't know the numbering of any of the tests -- to get it I either have to first run the whole test file (or start counting by hand or figure out some other clever but non-obvious tricks). It's really convenient to be able to just look at the test description(s) and then run ./t6416-recursive-corner-cases.sh --run=symlink or ./t6402-merge-rename.sh --run='setup,unnecessary update' Add such an ability to test selection which relies on merely matching against the test description. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
This commit is contained in:
parent
d4a392452e
commit
f21ac368f1
46
t/README
46
t/README
|
@ -258,16 +258,21 @@ For an individual test suite --run could be used to specify that
|
|||
only some tests should be run or that some tests should be
|
||||
excluded from a run.
|
||||
|
||||
The argument for --run is a list of individual test numbers or
|
||||
ranges with an optional negation prefix that define what tests in
|
||||
a test suite to include in the run. A range is two numbers
|
||||
separated with a dash and matches a range of tests with both ends
|
||||
been included. You may omit the first or the second number to
|
||||
mean "from the first test" or "up to the very last test"
|
||||
respectively.
|
||||
The argument for --run, <test-selector>, is a list of description
|
||||
substrings or globs or individual test numbers or ranges with an
|
||||
optional negation prefix (of '!') that define what tests in a test
|
||||
suite to include (or exclude, if negated) in the run. A range is two
|
||||
numbers separated with a dash and matches a range of tests with both
|
||||
ends been included. You may omit the first or the second number to
|
||||
mean "from the first test" or "up to the very last test" respectively.
|
||||
|
||||
Optional prefix of '!' means that the test or a range of tests
|
||||
should be excluded from the run.
|
||||
The argument to --run is split on commas into separate strings,
|
||||
numbers, and ranges, and picks all tests that match any of the
|
||||
individual selection criteria. If the substring of the description
|
||||
text that you want to match includes a comma, use the glob character
|
||||
'?' instead. For example --run='rebase,merge?cherry-pick' would match
|
||||
on all tests that match either the glob *rebase* or the glob
|
||||
*merge?cherry-pick*.
|
||||
|
||||
If --run starts with an unprefixed number or range the initial
|
||||
set of tests to run is empty. If the first item starts with '!'
|
||||
|
@ -275,9 +280,6 @@ all the tests are added to the initial set. After initial set is
|
|||
determined every test number or range is added or excluded from
|
||||
the set one by one, from left to right.
|
||||
|
||||
Individual numbers or ranges could be separated either by a space
|
||||
or a comma.
|
||||
|
||||
For example, to run only tests up to a specific test (21), one
|
||||
could do this:
|
||||
|
||||
|
@ -290,7 +292,7 @@ or this:
|
|||
Common case is to run several setup tests (1, 2, 3) and then a
|
||||
specific test (21) that relies on that setup:
|
||||
|
||||
$ sh ./t9200-git-cvsexport-commit.sh --run='1 2 3 21'
|
||||
$ sh ./t9200-git-cvsexport-commit.sh --run='1,2,3,21'
|
||||
|
||||
or:
|
||||
|
||||
|
@ -298,17 +300,17 @@ or:
|
|||
|
||||
or:
|
||||
|
||||
$ sh ./t9200-git-cvsexport-commit.sh --run='-3 21'
|
||||
$ sh ./t9200-git-cvsexport-commit.sh --run='-3,21'
|
||||
|
||||
As noted above, the test set is built by going through the items
|
||||
from left to right, so this:
|
||||
|
||||
$ sh ./t9200-git-cvsexport-commit.sh --run='1-4 !3'
|
||||
$ sh ./t9200-git-cvsexport-commit.sh --run='1-4,!3'
|
||||
|
||||
will run tests 1, 2, and 4. Items that come later have higher
|
||||
precedence. It means that this:
|
||||
|
||||
$ sh ./t9200-git-cvsexport-commit.sh --run='!3 1-4'
|
||||
$ sh ./t9200-git-cvsexport-commit.sh --run='!3,1-4'
|
||||
|
||||
would just run tests from 1 to 4, including 3.
|
||||
|
||||
|
@ -317,6 +319,18 @@ test in the test suite except from 7 up to 11:
|
|||
|
||||
$ sh ./t9200-git-cvsexport-commit.sh --run='!7-11'
|
||||
|
||||
Sometimes there may be multiple tests with e.g. "setup" in their name
|
||||
that are needed and rather than figuring out the number for all of them
|
||||
we can just use "setup" as a substring/glob to match against the test
|
||||
description:
|
||||
|
||||
$ sh ./t0050-filesystem.sh --run=setup,9-11
|
||||
|
||||
or one could select both the setup tests and the rename ones (assuming all
|
||||
relevant tests had those words in their descriptions):
|
||||
|
||||
$ sh ./t0050-filesystem.sh --run=setup,rename
|
||||
|
||||
Some tests in a test suite rely on the previous tests performing
|
||||
certain actions, specifically some tests are designated as
|
||||
"setup" test, so you cannot _arbitrarily_ disable one test and
|
||||
|
|
|
@ -430,7 +430,7 @@ test_expect_success 'GIT_SKIP_TESTS does not skip unmatched suite' "
|
|||
|
||||
test_expect_success '--run basic' "
|
||||
run_sub_test_lib_test run-basic \
|
||||
'--run basic' --run='1 3 5' <<-\\EOF &&
|
||||
'--run basic' --run='1,3,5' <<-\\EOF &&
|
||||
for i in 1 2 3 4 5 6
|
||||
do
|
||||
test_expect_success \"passing test #\$i\" 'true'
|
||||
|
@ -472,7 +472,7 @@ test_expect_success '--run with a range' "
|
|||
|
||||
test_expect_success '--run with two ranges' "
|
||||
run_sub_test_lib_test run-two-ranges \
|
||||
'--run with two ranges' --run='1-2 5-6' <<-\\EOF &&
|
||||
'--run with two ranges' --run='1-2,5-6' <<-\\EOF &&
|
||||
for i in 1 2 3 4 5 6
|
||||
do
|
||||
test_expect_success \"passing test #\$i\" 'true'
|
||||
|
@ -556,7 +556,7 @@ test_expect_success '--run with basic negation' "
|
|||
|
||||
test_expect_success '--run with two negations' "
|
||||
run_sub_test_lib_test run-two-neg \
|
||||
'--run with two negations' --run='"'!3 !6'"' <<-\\EOF &&
|
||||
'--run with two negations' --run='"'!3,!6'"' <<-\\EOF &&
|
||||
for i in 1 2 3 4 5 6
|
||||
do
|
||||
test_expect_success \"passing test #\$i\" 'true'
|
||||
|
@ -577,7 +577,7 @@ test_expect_success '--run with two negations' "
|
|||
|
||||
test_expect_success '--run a range and negation' "
|
||||
run_sub_test_lib_test run-range-and-neg \
|
||||
'--run a range and negation' --run='"'-4 !2'"' <<-\\EOF &&
|
||||
'--run a range and negation' --run='"'-4,!2'"' <<-\\EOF &&
|
||||
for i in 1 2 3 4 5 6
|
||||
do
|
||||
test_expect_success \"passing test #\$i\" 'true'
|
||||
|
@ -620,7 +620,7 @@ test_expect_success '--run range negation' "
|
|||
test_expect_success '--run include, exclude and include' "
|
||||
run_sub_test_lib_test run-inc-neg-inc \
|
||||
'--run include, exclude and include' \
|
||||
--run='"'1-5 !1-3 2'"' <<-\\EOF &&
|
||||
--run='"'1-5,!1-3,2'"' <<-\\EOF &&
|
||||
for i in 1 2 3 4 5 6
|
||||
do
|
||||
test_expect_success \"passing test #\$i\" 'true'
|
||||
|
@ -664,7 +664,7 @@ test_expect_success '--run include, exclude and include, comma separated' "
|
|||
test_expect_success '--run exclude and include' "
|
||||
run_sub_test_lib_test run-neg-inc \
|
||||
'--run exclude and include' \
|
||||
--run='"'!3- 5'"' <<-\\EOF &&
|
||||
--run='"'!3-,5'"' <<-\\EOF &&
|
||||
for i in 1 2 3 4 5 6
|
||||
do
|
||||
test_expect_success \"passing test #\$i\" 'true'
|
||||
|
@ -705,7 +705,31 @@ test_expect_success '--run empty selectors' "
|
|||
EOF
|
||||
"
|
||||
|
||||
test_expect_success '--run invalid range start' "
|
||||
test_expect_success '--run substring selector' "
|
||||
run_sub_test_lib_test run-substring-selector \
|
||||
'--run empty selectors' \
|
||||
--run='relevant' <<-\\EOF &&
|
||||
test_expect_success \"relevant test\" 'true'
|
||||
for i in 1 2 3 4 5 6
|
||||
do
|
||||
test_expect_success \"other test #\$i\" 'true'
|
||||
done
|
||||
test_done
|
||||
EOF
|
||||
check_sub_test_lib_test run-substring-selector <<-\\EOF
|
||||
> ok 1 - relevant test
|
||||
> ok 2 # skip other test #1 (--run)
|
||||
> ok 3 # skip other test #2 (--run)
|
||||
> ok 4 # skip other test #3 (--run)
|
||||
> ok 5 # skip other test #4 (--run)
|
||||
> ok 6 # skip other test #5 (--run)
|
||||
> ok 7 # skip other test #6 (--run)
|
||||
> # passed all 7 test(s)
|
||||
> 1..7
|
||||
EOF
|
||||
"
|
||||
|
||||
test_expect_success '--run keyword selection' "
|
||||
run_sub_test_lib_test_err run-inv-range-start \
|
||||
'--run invalid range start' \
|
||||
--run='a-5' <<-\\EOF &&
|
||||
|
@ -735,21 +759,6 @@ test_expect_success '--run invalid range end' "
|
|||
EOF_ERR
|
||||
"
|
||||
|
||||
test_expect_success '--run invalid selector' "
|
||||
run_sub_test_lib_test_err run-inv-selector \
|
||||
'--run invalid selector' \
|
||||
--run='1?' <<-\\EOF &&
|
||||
test_expect_success \"passing test #1\" 'true'
|
||||
test_done
|
||||
EOF
|
||||
check_sub_test_lib_test_err run-inv-selector \
|
||||
<<-\\EOF_OUT 3<<-\\EOF_ERR
|
||||
> FATAL: Unexpected exit with code 1
|
||||
EOF_OUT
|
||||
> error: --run: invalid non-numeric in test selector: '1?'
|
||||
EOF_ERR
|
||||
"
|
||||
|
||||
|
||||
test_set_prereq HAVEIT
|
||||
haveit=no
|
||||
|
|
|
@ -769,15 +769,17 @@ match_pattern_list () {
|
|||
}
|
||||
|
||||
match_test_selector_list () {
|
||||
operation="$1"
|
||||
shift
|
||||
title="$1"
|
||||
shift
|
||||
arg="$1"
|
||||
shift
|
||||
test -z "$1" && return 0
|
||||
|
||||
# Both commas and whitespace are accepted as separators.
|
||||
# Commas are accepted as separators.
|
||||
OLDIFS=$IFS
|
||||
IFS=' ,'
|
||||
IFS=','
|
||||
set -- $1
|
||||
IFS=$OLDIFS
|
||||
|
||||
|
@ -805,13 +807,13 @@ match_test_selector_list () {
|
|||
*-*)
|
||||
if expr "z${selector%%-*}" : "z[0-9]*[^0-9]" >/dev/null
|
||||
then
|
||||
echo "error: $title: invalid non-numeric in range" \
|
||||
echo "error: $operation: invalid non-numeric in range" \
|
||||
"start: '$orig_selector'" >&2
|
||||
exit 1
|
||||
fi
|
||||
if expr "z${selector#*-}" : "z[0-9]*[^0-9]" >/dev/null
|
||||
then
|
||||
echo "error: $title: invalid non-numeric in range" \
|
||||
echo "error: $operation: invalid non-numeric in range" \
|
||||
"end: '$orig_selector'" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
@ -819,9 +821,11 @@ match_test_selector_list () {
|
|||
*)
|
||||
if expr "z$selector" : "z[0-9]*[^0-9]" >/dev/null
|
||||
then
|
||||
echo "error: $title: invalid non-numeric in test" \
|
||||
"selector: '$orig_selector'" >&2
|
||||
exit 1
|
||||
case "$title" in *${selector}*)
|
||||
include=$positive
|
||||
;;
|
||||
esac
|
||||
continue
|
||||
fi
|
||||
esac
|
||||
|
||||
|
@ -1031,7 +1035,7 @@ test_skip () {
|
|||
skipped_reason="GIT_SKIP_TESTS"
|
||||
fi
|
||||
if test -z "$to_skip" && test -n "$run_list" &&
|
||||
! match_test_selector_list '--run' $test_count "$run_list"
|
||||
! match_test_selector_list '--run' "$1" $test_count "$run_list"
|
||||
then
|
||||
to_skip=t
|
||||
skipped_reason="--run"
|
||||
|
|
Loading…
Reference in a new issue