git/t/perf
Patrick Steinhardt 6754159767 refs: implement reference transaction hook
The low-level reference transactions used to update references are
currently completely opaque to the user. While certainly desirable in
most usecases, there are some which might want to hook into the
transaction to observe all queued reference updates as well as observing
the abortion or commit of a prepared transaction.

One such usecase would be to have a set of replicas of a given Git
repository, where we perform Git operations on all of the repositories
at once and expect the outcome to be the same in all of them. While
there exist hooks already for a certain subset of Git commands that
could be used to implement a voting mechanism for this, many others
currently don't have any mechanism for this.

The above scenario is the motivation for the new "reference-transaction"
hook that reaches directly into Git's reference transaction mechanism.
The hook receives as parameter the current state the transaction was
moved to ("prepared", "committed" or "aborted") and gets via its
standard input all queued reference updates. While the exit code gets
ignored in the "committed" and "aborted" states, a non-zero exit code in
the "prepared" state will cause the transaction to be aborted
prematurely.

Given the usecase described above, a voting mechanism can now be
implemented via this hook: as soon as it gets called, it will take all
of stdin and use it to cast a vote to a central service. When all
replicas of the repository agree, the hook will exit with zero,
otherwise it will abort the transaction by returning non-zero. The most
important upside is that this will catch _all_ commands writing
references at once, allowing to implement strong consistency for
reference updates via a single mechanism.

In order to test the impact on the case where we don't have any
"reference-transaction" hook installed in the repository, this commit
introduce two new performance tests for git-update-refs(1). Run against
an empty repository, it produces the following results:

  Test                         origin/master     HEAD
  --------------------------------------------------------------------
  1400.2: update-ref           2.70(2.10+0.71)   2.71(2.10+0.73) +0.4%
  1400.3: update-ref --stdin   0.21(0.09+0.11)   0.21(0.07+0.14) +0.0%

The performance test p1400.2 creates, updates and deletes a branch a
thousand times, thus averaging runtime of git-update-refs over 3000
invocations. p1400.3 instead calls `git-update-refs --stdin` three times
and queues a thousand creations, updates and deletes respectively.

As expected, p1400.3 consistently shows no noticeable impact, as for
each batch of updates there's a single call to access(3P) for the
negative hook lookup. On the other hand, for p1400.2, one can see an
impact caused by this patchset. But doing five runs of the performance
tests where each one was run with GIT_PERF_REPEAT_COUNT=10, the overhead
ranged from -1.5% to +1.1%. These inconsistent performance numbers can
be explained by the overhead of spawning 3000 processes. This shows that
the overhead of assembling the hook path and executing access(3P) once
to check if it's there is mostly outweighed by the operating system's
overhead.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-06-19 10:46:13 -07:00
..
repos p0006-read-tree-checkout: perf test to time read-tree 2017-04-19 20:33:01 -07:00
.gitignore
aggregate.perl Merge branch 'jk/perf-wo-git-dot-pm' 2019-12-10 13:11:44 -08:00
bisect_regression Fix spelling errors in messages shown to users 2019-11-10 16:00:54 +09:00
bisect_run_script perf/bisect_run_script: disable codespeed 2018-05-06 13:04:54 +09:00
lib-pack.sh t/perf/lib-pack: use fast-import checkpoint to create packs 2017-11-21 11:07:28 +09:00
Makefile
min_time.perl
p0000-perf-lib-sanity.sh t/perf: correctly align non-ASCII descriptions in output 2017-04-23 21:33:15 -07:00
p0001-rev-list.sh revision: use a prio_queue to hold rewritten parents 2019-04-04 18:21:54 +09:00
p0002-read-cache.sh t/helper: merge test-read-cache into test-tool 2018-03-27 08:45:47 -07:00
p0003-delta-base-cache.sh t/perf: add basic perf tests for delta base cache 2016-08-23 15:26:16 -07:00
p0004-lazy-init-name-hash.sh t/helper: merge test-lazy-init-name-hash into test-tool 2018-03-27 08:45:47 -07:00
p0005-status.sh string-list: use ALLOC_GROW macro when reallocing string_list 2017-04-15 02:04:41 -07:00
p0006-read-tree-checkout.sh p0006-read-tree-checkout: perf test to time read-tree 2017-04-19 20:33:01 -07:00
p0007-write-cache.sh t/helper: merge test-write-cache into test-tool 2018-03-27 08:45:47 -07:00
p0071-sort.sh t/helper: merge test-string-list into test-tool 2018-03-27 08:45:47 -07:00
p0100-globbing.sh perf: add test showing exponential growth in path globbing 2017-05-12 10:07:43 +09:00
p1400-update-ref.sh refs: implement reference transaction hook 2020-06-19 10:46:13 -07:00
p1450-fsck.sh fsck: add a performance test 2018-09-12 15:17:46 -07:00
p1451-fsck-skip-list.sh fsck: add a performance test for skipList 2018-09-12 15:17:46 -07:00
p3400-rebase.sh p3400: replace calls to git checkout -b' by git checkout -B' 2018-11-12 16:40:55 +09:00
p3404-rebase-interactive.sh perf: run "rebase -i" under perf 2016-05-13 11:07:12 -07:00
p4000-diff-algorithms.sh
p4001-diff-no-index.sh
p4205-log-pretty-formats.sh p4205: add perf test script for pretty log formats 2017-06-24 11:05:02 -07:00
p4211-line-log.sh sha1_file: use strbuf_add() instead of strbuf_addf() 2017-12-04 10:38:55 -08:00
p4220-log-grep-engines.sh perf: add a comparison test of log --grep regex engines 2017-05-26 12:52:37 +09:00
p4221-log-grep-engines-fixed.sh perf: add a comparison test of log --grep regex engines with -F 2017-05-26 12:52:37 +09:00
p5302-pack-index.sh p5302: create the repo in each index-pack test 2019-04-23 09:56:44 +09:00
p5303-many-packs.sh Merge branch 'cs/store-packfiles-in-hashmap' 2019-12-16 13:08:32 -08:00
p5304-prune.sh prune: use bitmaps for reachability traversal 2019-02-14 15:25:33 -08:00
p5310-pack-bitmaps.sh pack-bitmap: pass object filter to fill-in traversal 2020-05-04 21:57:58 -07:00
p5311-pack-bitmaps-fetch.sh pack-objects: default to writing bitmap hash-cache 2019-03-18 14:11:15 +09:00
p5550-fetch-tags.sh p5550: factor out nonsense-pack creation 2017-11-21 11:07:12 +09:00
p5551-fetch-rescan.sh p5551: add a script to test fetch pack-dir rescans 2017-11-21 11:08:20 +09:00
p5600-partial-clone.sh t/perf: add perf script for partial clones 2019-05-05 14:03:57 +09:00
p5601-clone-reference.sh t/perf: rename duplicate-numbered test script 2019-08-12 09:05:13 -07:00
p7000-filter-branch.sh p7000: add test for filter-branch with --prune-empty 2017-03-03 12:43:37 -08:00
p7300-clean.sh resolve_gitlink_ref: ignore non-repository paths 2016-01-25 11:42:13 -08:00
p7519-fsmonitor.sh t/helper: merge test-drop-caches into test-tool 2018-03-27 08:45:47 -07:00
p7810-grep.sh
p7820-grep-engines.sh perf: amend the grep tests to test grep.threads 2018-01-04 10:24:48 -08:00
p7821-grep-engines-fixed.sh perf: amend the grep tests to test grep.threads 2018-01-04 10:24:48 -08:00
p9300-fast-import-export.sh fast-import: replace custom hash with hashmap.c 2020-04-06 13:41:24 -07:00
perf-lib.sh perf-lib: use a single filename for all measurement types 2019-11-27 10:48:25 +09:00
README perf README: correct docs for 3c8f12c96c regression 2019-05-08 11:00:28 +09:00
run perf-lib.sh: forbid the use of GIT_TEST_INSTALLED 2019-05-08 11:00:28 +09:00

Git performance tests
=====================

This directory holds performance testing scripts for git tools.  The
first part of this document describes the various ways in which you
can run them.

When fixing the tools or adding enhancements, you are strongly
encouraged to add tests in this directory to cover what you are
trying to fix or enhance.  The later part of this short document
describes how your test scripts should be organized.


Running Tests
-------------

The easiest way to run tests is to say "make".  This runs all
the tests on the current git repository.

    === Running 2 tests in this tree ===
    [...]
    Test                                     this tree
    ---------------------------------------------------------
    0001.1: rev-list --all                   0.54(0.51+0.02)
    0001.2: rev-list --all --objects         6.14(5.99+0.11)
    7810.1: grep worktree, cheap regex       0.16(0.16+0.35)
    7810.2: grep worktree, expensive regex   7.90(29.75+0.37)
    7810.3: grep --cached, cheap regex       3.07(3.02+0.25)
    7810.4: grep --cached, expensive regex   9.39(30.57+0.24)

You can compare multiple repositories and even git revisions with the
'run' script:

    $ ./run . origin/next /path/to/git-tree p0001-rev-list.sh

where . stands for the current git tree.  The full invocation is

    ./run [<revision|directory>...] [--] [<test-script>...]

A '.' argument is implied if you do not pass any other
revisions/directories.

You can also manually test this or another git build tree, and then
call the aggregation script to summarize the results:

    $ ./p0001-rev-list.sh
    [...]
    $ ./run /path/to/other/git -- ./p0001-rev-list.sh
    [...]
    $ ./aggregate.perl . /path/to/other/git ./p0001-rev-list.sh

aggregate.perl has the same invocation as 'run', it just does not run
anything beforehand.

You can set the following variables (also in your config.mak):

    GIT_PERF_REPEAT_COUNT
	Number of times a test should be repeated for best-of-N
	measurements.  Defaults to 3.

    GIT_PERF_MAKE_OPTS
	Options to use when automatically building a git tree for
	performance testing. E.g., -j6 would be useful. Passed
	directly to make as "make $GIT_PERF_MAKE_OPTS".

    GIT_PERF_MAKE_COMMAND
	An arbitrary command that'll be run in place of the make
	command, if set the GIT_PERF_MAKE_OPTS variable is
	ignored. Useful in cases where source tree changes might
	require issuing a different make command to different
	revisions.

	This can be (ab)used to monkeypatch or otherwise change the
	tree about to be built. Note that the build directory can be
	re-used for subsequent runs so the make command might get
	executed multiple times on the same tree, but don't count on
	any of that, that's an implementation detail that might change
	in the future.

    GIT_PERF_REPO
    GIT_PERF_LARGE_REPO
	Repositories to copy for the performance tests.  The normal
	repo should be at least git.git size.  The large repo should
	probably be about linux.git size for optimal results.
	Both default to the git.git you are running from.

You can also pass the options taken by ordinary git tests; the most
useful one is:

--root=<directory>::
	Create "trash" directories used to store all temporary data during
	testing under <directory>, instead of the t/ directory.
	Using this option with a RAM-based filesystem (such as tmpfs)
	can massively speed up the test suite.


Naming Tests
------------

The performance test files are named as:

	pNNNN-commandname-details.sh

where N is a decimal digit.  The same conventions for choosing NNNN as
for normal tests apply.


Writing Tests
-------------

The perf script starts much like a normal test script, except it
sources perf-lib.sh:

	#!/bin/sh
	#
	# Copyright (c) 2005 Junio C Hamano
	#

	test_description='xxx performance test'
	. ./perf-lib.sh

After that you will want to use some of the following:

	test_perf_fresh_repo    # sets up an empty repository
	test_perf_default_repo  # sets up a "normal" repository
	test_perf_large_repo    # sets up a "large" repository

	test_perf_default_repo sub  # ditto, in a subdir "sub"

        test_checkout_worktree  # if you need the worktree too

At least one of the first two is required!

You can use test_expect_success as usual. In both test_expect_success
and in test_perf, running "git" points to the version that is being
perf-tested. The $MODERN_GIT variable points to the git wrapper for the
currently checked-out version (i.e., the one that matches the t/perf
scripts you are running).  This is useful if your setup uses commands
that only work with newer versions of git than what you might want to
test (but obviously your new commands must still create a state that can
be used by the older version of git you are testing).

For actual performance tests, use

	test_perf 'descriptive string' '
		command1 &&
		command2
	'

test_perf spawns a subshell, for lack of better options.  This means
that

* you _must_ export all variables that you need in the subshell

* you _must_ flag all variables that you want to persist from the
  subshell with 'test_export':

	test_perf 'descriptive string' '
		foo=$(git rev-parse HEAD) &&
		test_export foo
	'

  The so-exported variables are automatically marked for export in the
  shell executing the perf test.  For your convenience, test_export is
  the same as export in the main shell.

  This feature relies on a bit of magic using 'set' and 'source'.
  While we have tried to make sure that it can cope with embedded
  whitespace and other special characters, it will not work with
  multi-line data.

Rather than tracking the performance by run-time as `test_perf` does, you
may also track output size by using `test_size`. The stdout of the
function should be a single numeric value, which will be captured and
shown in the aggregated output. For example:

	test_perf 'time foo' '
		./foo >foo.out
	'

	test_size 'output size'
		wc -c <foo.out
	'

might produce output like:

	Test                origin           HEAD
	-------------------------------------------------------------
	1234.1 time foo     0.37(0.79+0.02)  0.26(0.51+0.02) -29.7%
	1234.2 output size             4.3M             3.6M -14.7%

The item being measured (and its units) is up to the test; the context
and the test title should make it clear to the user whether bigger or
smaller numbers are better. Unlike test_perf, the test code will only be
run once, since output sizes tend to be more deterministic than timings.