From 9bb81452ffce9481f7613868ab9c8be998529bd6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Tue, 7 May 2019 12:54:29 +0200 Subject: [PATCH 1/6] perf README: correct docs for 3c8f12c96c regression MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Since 3c8f12c96c ("test-lib: reorder and include GIT-BUILD-OPTIONS a lot earlier", 2012-06-24) the suggested advice of overriding GIT_BUILD_DIR has not worked. We've printed a hard error like this given e.g. GIT_BUILD_DIR=/home/avar/g/git: /bin-wrappers/git is not executable; using GIT_EXEC_PATH error: You haven't built things yet, have you? Let's just suggest that the user run other gits via the "run" script. That'll do the right thing for setting the path to the other git, and running the "aggregate.perl" scripts afterwards will work. As an aside, if setting GIT_BUILD_DIR had still worked, then the MODERN_GIT feature/fix added in 1a0962dee5 ("t/perf: fix regression in testing older versions of git", 2016-06-22) would have broke. Signed-off-by: Ævar Arnfjörð Bjarmason --- t/perf/README | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/perf/README b/t/perf/README index be12090c38..c7b70e2d28 100644 --- a/t/perf/README +++ b/t/perf/README @@ -45,7 +45,7 @@ call the aggregation script to summarize the results: $ ./p0001-rev-list.sh [...] - $ GIT_BUILD_DIR=/path/to/other/git ./p0001-rev-list.sh + $ ./run /path/to/other/git -- ./p0001-rev-list.sh [...] $ ./aggregate.perl . /path/to/other/git ./p0001-rev-list.sh From c43b7e6089629b05d5af6995e99d5149ae5c3621 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Tue, 7 May 2019 12:54:30 +0200 Subject: [PATCH 2/6] perf aggregate: remove GIT_TEST_INSTALLED from --codespeed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Remove the setting of the "environment" from the --codespeed output. I don't think this is useful, and it helps with a later refactoring where we GIT_TEST_INSTALLED stop munging/reading GIT_TEST_INSTALLED in the perf tests in so many places. This was added in 05eb1c37ed ("perf/aggregate: implement codespeed JSON output", 2018-01-05), but since the "run" scripts uses "GIT_TEST_INSTALLED" internally this was only ever useful for one-off runs of a single revision as all the "environment" values would be ones for whatever directory the "run" script ran last. Let's instead fall back on the "uname -r" case, which is the sort of thing the environment should be set to, not something that duplicates other parts of the codpseed output. For setting the "environment" to something custom the perf.repoName variable can be used. See 19cf57a92e ("perf/run: read GIT_PERF_REPO_NAME from perf.repoName", 2018-01-05). Signed-off-by: Ævar Arnfjörð Bjarmason --- t/perf/aggregate.perl | 3 --- 1 file changed, 3 deletions(-) diff --git a/t/perf/aggregate.perl b/t/perf/aggregate.perl index 494907a892..f6518339dc 100755 --- a/t/perf/aggregate.perl +++ b/t/perf/aggregate.perl @@ -312,9 +312,6 @@ sub print_codespeed_results { $environment = $reponame; } elsif (exists $ENV{GIT_PERF_REPO_NAME} and $ENV{GIT_PERF_REPO_NAME} ne "") { $environment = $ENV{GIT_PERF_REPO_NAME}; - } elsif (exists $ENV{GIT_TEST_INSTALLED} and $ENV{GIT_TEST_INSTALLED} ne "") { - $environment = $ENV{GIT_TEST_INSTALLED}; - $environment =~ s|/bin-wrappers$||; } else { $environment = `uname -r`; chomp $environment; From 90e38154eeb57f05ac099c42f4b369c8dad9ad2c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Tue, 7 May 2019 12:54:31 +0200 Subject: [PATCH 3/6] perf-lib.sh: make "./run " use the correct gits MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fix a really bad regression in 0baf78e7bc ("perf-lib.sh: rely on test-lib.sh for --tee handling", 2019-03-15). Since that change all runs of different of git have used the git found in the user's $PATH, e.g. /usr/bin/git instead of the we just built and wanted to performance test. The problem starts with GIT_TEST_INSTALLED not working like our non-perf tests with the "run" script. I.e. you can't run performance tests against a given installed git. Instead we expect to use it ourselves to point GIT_TEST_INSTALLED to the we just built. However, we had been relying on '$(cd "$GIT_TEST_INSTALLED" && pwd)' to resolve that relative $GIT_TEST_INSTALLED to an absolute path *before* test-lib.sh was loaded, in cases where it was e.g. "build//bin-wrappers" and we wanted "build/...". This change post-dates another proposed solution by a few days[1], I didn't notice that version when I initially wrote this. I'm doing the most minimal thing to solve the regression here, a follow-up change will move this result prefix selection logic entirely into the "run" script. This makes e.g. these cases all work: ./run . $PWD/../../ origin/master origin/next HEAD -- As well as just a plain one-off: ./run And, since we're passing down the new GIT_PERF_DIR_MYDIR_REL we make sure the bug relating to aggregate.perl not finding our files as described in 0baf78e7bc doesn't happen again. What *doesn't* work is setting GIT_TEST_INSTALLED to a relative path, this will subtly fail in test-lib.sh. This has always been the case even before 0baf78e7bc, and as documented in t/README the GIT_TEST_INSTALLED variable should be set to an absolute path (needs to be set "to the bindir", which is always absolute), and the "perf" framework expects to munge it itself. Perhaps that should be dealt with in the future to allow manually setting GIT_TEST_INSTALLED, but as a preceding commit showed the user can just use the "run" script, which'll also pick the right output directory for the test results as expected by aggregate.perl. 1. https://public-inbox.org/git/20190502222409.GA15631@sigill.intra.peff.net/ Signed-off-by: Ævar Arnfjörð Bjarmason --- t/perf/perf-lib.sh | 4 ++++ t/perf/run | 8 ++++++-- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh index 169f92eae3..b15ee1d262 100644 --- a/t/perf/perf-lib.sh +++ b/t/perf/perf-lib.sh @@ -32,6 +32,10 @@ TEST_NO_MALLOC_CHECK=t if test -z "$GIT_TEST_INSTALLED"; then perf_results_prefix= else + if test -n "$GIT_PERF_DIR_MYDIR_REL" + then + GIT_TEST_INSTALLED=$GIT_PERF_DIR_MYDIR_REL + fi perf_results_prefix=$(printf "%s" "${GIT_TEST_INSTALLED%/bin-wrappers}" | tr -c "[a-zA-Z0-9]" "[_*]")"." GIT_TEST_INSTALLED=$ABSOLUTE_GIT_TEST_INSTALLED fi diff --git a/t/perf/run b/t/perf/run index 9aaa733c77..0a7c8744ab 100755 --- a/t/perf/run +++ b/t/perf/run @@ -91,10 +91,14 @@ run_dirs_helper () { if test "$mydir" = .; then unset GIT_TEST_INSTALLED else - GIT_TEST_INSTALLED="$mydir/bin-wrappers" + GIT_PERF_DIR_MYDIR_REL=$mydir + GIT_PERF_DIR_MYDIR_ABS=$(cd $mydir && pwd) + export GIT_PERF_DIR_MYDIR_REL GIT_PERF_DIR_MYDIR_ABS + + GIT_TEST_INSTALLED="$GIT_PERF_DIR_MYDIR_ABS/bin-wrappers" # Older versions of git lacked bin-wrappers; fallback to the # files in the root. - test -d "$GIT_TEST_INSTALLED" || GIT_TEST_INSTALLED=$mydir + test -d "$GIT_TEST_INSTALLED" || GIT_TEST_INSTALLED=$GIT_PERF_DIR_MYDIR_ABS export GIT_TEST_INSTALLED fi run_one_dir "$@" From df0f5021951bd0ae5e7db0d89fd7e5c141c334be Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Tue, 7 May 2019 12:54:32 +0200 Subject: [PATCH 4/6] perf-lib.sh: remove GIT_TEST_INSTALLED from perf-lib.sh MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follow-up my preceding change which fixed the immediate "./run " regression in 0baf78e7bc ("perf-lib.sh: rely on test-lib.sh for --tee handling", 2019-03-15) and entirely get rid of GIT_TEST_INSTALLED from perf-lib.sh (and aggregate.perl). As noted in that change the dance we're doing with GIT_TEST_INSTALLED perf-lib.sh isn't necessary, but there I was doing the most minimal set of changes to quickly fix a regression. But it's much simpler to never deal with the "GIT_TEST_INSTALLED" we were setting in perf-lib.sh at all. Instead the run_dirs_helper() sets the previously inferred $PERF_RESULTS_PREFIX directly. Setting this at the callsite that's already best positioned to exhaustively know about all the different cases we need to handle where PERF_RESULTS_PREFIX isn't what we want already (the empty string) makes the most sense. In one-off cases like: ./run ./p0000-perf-lib-sanity.sh ./p0000-perf-lib-sanity.sh We'll just do the right thing because PERF_RESULTS_PREFIX will be empty, and test-lib.sh takes care of finding where our git is. Any refactoring of this code needs to change both the shell code and the Perl code in aggregate.perl, because when running e.g.: ./run ../../ -- The "../../" path to a relative bindir needs to be munged to a filename containing the results, and critically aggregate.perl does not get passed the path to those aggregations, just "../..". Let's fix cases where aggregate.perl would print e.g. ".." in its report output for this, and "git" for "/home/avar/g/git", i.e. it would always pick the last element. Now'll always print the full path instead. This also makes the code sturdier, e.g. you can feed "../.." to "./run" and then an absolute path to the aggregate.perl script, as long as the absolute path and "../.." resolved to the same directory printing the aggregation will work. Also simplify the "[_*]" on the RHS of "tr -c", we're trimming everything to "_", so we don't need that. Signed-off-by: Ævar Arnfjörð Bjarmason --- t/perf/aggregate.perl | 10 ++++++---- t/perf/perf-lib.sh | 15 +-------------- t/perf/run | 43 ++++++++++++++++++++++++++++++------------- 3 files changed, 37 insertions(+), 31 deletions(-) diff --git a/t/perf/aggregate.perl b/t/perf/aggregate.perl index f6518339dc..c8f4a78903 100755 --- a/t/perf/aggregate.perl +++ b/t/perf/aggregate.perl @@ -6,6 +6,7 @@ use JSON; use Getopt::Long; use Git; +use Cwd qw(realpath); sub get_times { my $name = shift; @@ -103,13 +104,14 @@ sub format_size { if (! -d $arg) { my $rev = Git::command_oneline(qw(rev-parse --verify), $arg); $dir = "build/".$rev; + } elsif ($arg eq '.') { + $dir = '.'; } else { - $arg =~ s{/*$}{}; - $dir = $arg; - $dirabbrevs{$dir} = $dir; + $dir = realpath($arg); + $dirnames{$dir} = $dir; } push @dirs, $dir; - $dirnames{$dir} = $arg; + $dirnames{$dir} ||= $arg; my $prefix = $dir; $prefix =~ tr/^a-zA-Z0-9/_/c; $prefixes{$dir} = $prefix . '.'; diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh index b15ee1d262..9cdccba222 100644 --- a/t/perf/perf-lib.sh +++ b/t/perf/perf-lib.sh @@ -21,25 +21,12 @@ # because it will change our working directory. TEST_DIRECTORY=$(pwd)/.. TEST_OUTPUT_DIRECTORY=$(pwd) -ABSOLUTE_GIT_TEST_INSTALLED=$( - test -n "$GIT_TEST_INSTALLED" && cd "$GIT_TEST_INSTALLED" && pwd) TEST_NO_CREATE_REPO=t TEST_NO_MALLOC_CHECK=t . ../test-lib.sh -if test -z "$GIT_TEST_INSTALLED"; then - perf_results_prefix= -else - if test -n "$GIT_PERF_DIR_MYDIR_REL" - then - GIT_TEST_INSTALLED=$GIT_PERF_DIR_MYDIR_REL - fi - perf_results_prefix=$(printf "%s" "${GIT_TEST_INSTALLED%/bin-wrappers}" | tr -c "[a-zA-Z0-9]" "[_*]")"." - GIT_TEST_INSTALLED=$ABSOLUTE_GIT_TEST_INSTALLED -fi - # Variables from test-lib that are normally internal to the tests; we # need to export them for test_perf subshells export TEST_DIRECTORY TRASH_DIRECTORY GIT_BUILD_DIR GIT_TEST_CMP @@ -183,7 +170,7 @@ test_wrapper_ () { base=$(basename "$0" .sh) echo "$test_count" >>"$perf_results_dir"/$base.subtests echo "$1" >"$perf_results_dir"/$base.$test_count.descr - base="$perf_results_dir"/"$perf_results_prefix$(basename "$0" .sh)"."$test_count" + base="$perf_results_dir"/"$PERF_RESULTS_PREFIX$(basename "$0" .sh)"."$test_count" "$test_wrapper_func_" "$@" fi diff --git a/t/perf/run b/t/perf/run index 0a7c8744ab..85b7bd31d5 100755 --- a/t/perf/run +++ b/t/perf/run @@ -70,6 +70,22 @@ build_git_rev () { ) || die "failed to build revision '$mydir'" } +set_git_test_installed () { + mydir=$1 + + mydir_abs=$(cd $mydir && pwd) + mydir_abs_wrappers="$mydir_abs_wrappers/bin-wrappers" + if test -d "$mydir_abs_wrappers" + then + GIT_TEST_INSTALLED=$mydir_abs_wrappers + else + # Older versions of git lacked bin-wrappers; + # fallback to the files in the root. + GIT_TEST_INSTALLED=$mydir_abs + fi + export GIT_TEST_INSTALLED +} + run_dirs_helper () { mydir=${1%/} shift @@ -79,7 +95,16 @@ run_dirs_helper () { if test $# -gt 0 -a "$1" = --; then shift fi - if [ ! -d "$mydir" ]; then + + PERF_RESULTS_PREFIX= + if test "$mydir" = "." + then + unset GIT_TEST_INSTALLED + elif test -d "$mydir" + then + PERF_RESULTS_PREFIX=$(cd $mydir && printf "%s" "$(pwd)" | tr -c "[a-zA-Z0-9]" "_"). + set_git_test_installed "$mydir" + else rev=$(git rev-parse --verify "$mydir" 2>/dev/null) || die "'$mydir' is neither a directory nor a valid revision" if [ ! -d build/$rev ]; then @@ -87,20 +112,12 @@ run_dirs_helper () { fi build_git_rev $rev "$mydir" mydir=build/$rev - fi - if test "$mydir" = .; then - unset GIT_TEST_INSTALLED - else - GIT_PERF_DIR_MYDIR_REL=$mydir - GIT_PERF_DIR_MYDIR_ABS=$(cd $mydir && pwd) - export GIT_PERF_DIR_MYDIR_REL GIT_PERF_DIR_MYDIR_ABS - GIT_TEST_INSTALLED="$GIT_PERF_DIR_MYDIR_ABS/bin-wrappers" - # Older versions of git lacked bin-wrappers; fallback to the - # files in the root. - test -d "$GIT_TEST_INSTALLED" || GIT_TEST_INSTALLED=$GIT_PERF_DIR_MYDIR_ABS - export GIT_TEST_INSTALLED + PERF_RESULTS_PREFIX=build_$rev. + set_git_test_installed "$mydir" fi + export PERF_RESULTS_PREFIX + run_one_dir "$@" } From fab80ee79ddf59a5d00812005bef0fa3acf5b6bf Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Tue, 7 May 2019 12:54:33 +0200 Subject: [PATCH 5/6] perf tests: add "bindir" prefix to git tree test results MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Change the output file names in test-results/ to be "test-results/bindir_" rather than just "test-results/". This is for consistency with the "build_" directories we have for built revisions, i.e. "test-results/build_". There's no user-visible functional changes here, it just makes it easier to see at a glance what "test-results" files are of what "type" as they're all explicitly grouped together now, and to grep this code to find both the run_dirs_helper() implementation and its corresponding aggregate.perl code. Note that we already guarantee that the rest of the PERF_RESULTS_PREFIX is an absolute path, and since it'll start with e.g. "/" which we munge to "_" we'll up with a readable string like "bindir_home_avar[...]". Signed-off-by: Ævar Arnfjörð Bjarmason --- t/perf/aggregate.perl | 4 +++- t/perf/run | 2 +- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/t/perf/aggregate.perl b/t/perf/aggregate.perl index c8f4a78903..b951747e08 100755 --- a/t/perf/aggregate.perl +++ b/t/perf/aggregate.perl @@ -100,6 +100,7 @@ sub format_size { while (scalar @ARGV) { my $arg = $ARGV[0]; my $dir; + my $prefix = ''; last if -f $arg or $arg eq "--"; if (! -d $arg) { my $rev = Git::command_oneline(qw(rev-parse --verify), $arg); @@ -109,10 +110,11 @@ sub format_size { } else { $dir = realpath($arg); $dirnames{$dir} = $dir; + $prefix .= 'bindir'; } push @dirs, $dir; $dirnames{$dir} ||= $arg; - my $prefix = $dir; + $prefix .= $dir; $prefix =~ tr/^a-zA-Z0-9/_/c; $prefixes{$dir} = $prefix . '.'; shift @ARGV; diff --git a/t/perf/run b/t/perf/run index 85b7bd31d5..cd3882b117 100755 --- a/t/perf/run +++ b/t/perf/run @@ -102,7 +102,7 @@ run_dirs_helper () { unset GIT_TEST_INSTALLED elif test -d "$mydir" then - PERF_RESULTS_PREFIX=$(cd $mydir && printf "%s" "$(pwd)" | tr -c "[a-zA-Z0-9]" "_"). + PERF_RESULTS_PREFIX=bindir$(cd $mydir && printf "%s" "$(pwd)" | tr -c "[a-zA-Z0-9]" "_"). set_git_test_installed "$mydir" else rev=$(git rev-parse --verify "$mydir" 2>/dev/null) || From 82b7eb231d146678317fd4cf63a646c4233976e0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Date: Tue, 7 May 2019 12:54:34 +0200 Subject: [PATCH 6/6] perf-lib.sh: forbid the use of GIT_TEST_INSTALLED MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit As noted in preceding commits setting GIT_TEST_INSTALLED has never been supported or documented, and as noted in an earlier t/perf/README change to the extent that it's been documented nobody's notices that the example hasn't worked since 3c8f12c96c ("test-lib: reorder and include GIT-BUILD-OPTIONS a lot earlier", 2012-06-24). We could directly support GIT_TEST_INSTALLED for invocations without the "run" script, such as: GIT_TEST_INSTALLED=../../ ./p0000-perf-lib-sanity.sh GIT_TEST_INSTALLED=/home/avar/g/git ./p0000-perf-lib-sanity.sh But while not having this "error" will "work", it won't write the the resulting "test-results/*" files to the right place, and thus a subsequent call to aggregate.perl won't work as expected. Let's just tell the user that they need to use the "run" script, which'll correctly deal with this and set the right PERF_RESULTS_PREFIX. If someone's in desperate need of bypassing "run" for whatever reason they can trivially do so by setting "PERF_SET_GIT_TEST_INSTALLED", but not we won't have people who expect GIT_TEST_INSTALLED to just work wondering why their aggregation doesn't work, even though they're running the right "git". Signed-off-by: Ævar Arnfjörð Bjarmason --- t/perf/perf-lib.sh | 11 +++++++++++ t/perf/run | 2 ++ 2 files changed, 13 insertions(+) diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh index 9cdccba222..b58a43ea43 100644 --- a/t/perf/perf-lib.sh +++ b/t/perf/perf-lib.sh @@ -27,6 +27,17 @@ TEST_NO_MALLOC_CHECK=t . ../test-lib.sh +if test -n "$GIT_TEST_INSTALLED" -a -z "$PERF_SET_GIT_TEST_INSTALLED" +then + error "Do not use GIT_TEST_INSTALLED with the perf tests. + +Instead use: + + ./run -- + +See t/perf/README for details." +fi + # Variables from test-lib that are normally internal to the tests; we # need to export them for test_perf subshells export TEST_DIRECTORY TRASH_DIRECTORY GIT_BUILD_DIR GIT_TEST_CMP diff --git a/t/perf/run b/t/perf/run index cd3882b117..c7b86104e1 100755 --- a/t/perf/run +++ b/t/perf/run @@ -84,6 +84,8 @@ set_git_test_installed () { GIT_TEST_INSTALLED=$mydir_abs fi export GIT_TEST_INSTALLED + PERF_SET_GIT_TEST_INSTALLED=true + export PERF_SET_GIT_TEST_INSTALLED } run_dirs_helper () {