development/git
development/git
1
0
Fork 0
mirror of https://github.com/git/git synced 2024-08-27 11:39:22 +00:00
Code Issues Packages Projects Releases Wiki Activity
git/t/t6113-rev-list-bitmap-filters.sh

161 lines
5.6 KiB
Bash
Raw Normal View History

rev-list: fallback to non-bitmap traversal when filtering The "--use-bitmap-index" option is usually aspirational: if we have bitmaps and the request can be fulfilled more quickly using them we'll do so, but otherwise fall back to a non-bitmap traversal. The exception is object filtering, which explicitly dies if the two options are combined. Let's convert this to the usual fallback behavior. This is a minor convenience for now (since the caller can easily know that --filter and --use-bitmap-index don't combine), but will become much more useful as we start to support _some_ filters with bitmaps, but not others. The test infrastructure here is bigger than necessary for checking this one small feature. But it will serve as the basis for more filtering bitmap tests in future patches. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-02-13 02:17:30 +00:00
#!/bin/sh
test_description='rev-list combining bitmaps and filters'
t6113: mark as leak-free This test does not leak since a96015a517 (pack-bitmap: plug leak in find_objects(), 2023-12-14) when the annotation TEST_PASSES_SANITIZE_LEAK=true was also added. Unfortunately it was added after test-lib.sh is sourced, which makes GIT_TEST_PASSING_SANITIZE_LEAK=check error: $ make SANITIZE=leak GIT_TEST_PASSING_SANITIZE_LEAK=check test T=t6113-rev-list-bitmap-filters.sh ... make[2]: Entering directory '/tmp/git/git/t' *** t6113-rev-list-bitmap-filters.sh *** in GIT_TEST_PASSING_SANITIZE_LEAK=check mode, setting --invert-exit-code for TEST_PASSES_SANITIZE_LEAK != true ok 1 - set up bitmapped repo ok 2 - filters fallback to non-bitmap traversal ok 3 - blob:none filter ok 4 - blob:none filter with specified blob ok 5 - blob:limit filter ok 6 - blob:limit filter with specified blob ok 7 - tree:0 filter ok 8 - tree:0 filter with specified blob, tree ok 9 - tree:1 filter ok 10 - object:type filter ok 11 - object:type filter with --filter-provided-objects ok 12 - combine filter ok 13 - combine filter with --filter-provided-objects ok 14 - bitmap traversal with --unpacked # passed all 14 test(s) 1..14 # faking up non-zero exit with --invert-exit-code make[2]: *** [Makefile:68: t6113-rev-list-bitmap-filters.sh] Error 1 make[2]: Leaving directory '/tmp/git/git/t' make[1]: *** [Makefile:55: test] Error 2 make[1]: Leaving directory '/tmp/git/git/t' make: *** [Makefile:3212: test] Error 2 Let's move the annotation before sourcing test-lib.sh, to make GIT_TEST_PASSING_SANITIZE_LEAK=check happy. Signed-off-by: Rubén Justo <rjusto@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2024-01-29 21:08:34 +00:00
TEST_PASSES_SANITIZE_LEAK=true
rev-list: fallback to non-bitmap traversal when filtering The "--use-bitmap-index" option is usually aspirational: if we have bitmaps and the request can be fulfilled more quickly using them we'll do so, but otherwise fall back to a non-bitmap traversal. The exception is object filtering, which explicitly dies if the two options are combined. Let's convert this to the usual fallback behavior. This is a minor convenience for now (since the caller can easily know that --filter and --use-bitmap-index don't combine), but will become much more useful as we start to support _some_ filters with bitmaps, but not others. The test infrastructure here is bigger than necessary for checking this one small feature. But it will serve as the basis for more filtering bitmap tests in future patches. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-02-13 02:17:30 +00:00
. ./test-lib.sh
test-lib-functions: move function to lib-bitmap.sh Move a function added to test-lib-functions.sh in ea047a8eb4f (t5310: factor out bitmap traversal comparison, 2020-02-14) into a new lib-bitmap.sh. The test-lib-functions.sh file should be for functions that are widely used across the test suite, if something's only used by a few tests it makes more sense to have it in a lib-*.sh file. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-02-09 21:41:54 +00:00
. "$TEST_DIRECTORY"/lib-bitmap.sh
rev-list: fallback to non-bitmap traversal when filtering The "--use-bitmap-index" option is usually aspirational: if we have bitmaps and the request can be fulfilled more quickly using them we'll do so, but otherwise fall back to a non-bitmap traversal. The exception is object filtering, which explicitly dies if the two options are combined. Let's convert this to the usual fallback behavior. This is a minor convenience for now (since the caller can easily know that --filter and --use-bitmap-index don't combine), but will become much more useful as we start to support _some_ filters with bitmaps, but not others. The test infrastructure here is bigger than necessary for checking this one small feature. But it will serve as the basis for more filtering bitmap tests in future patches. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-02-13 02:17:30 +00:00
pack-bitmap: plug leak in find_objects() The `find_objects()` function creates an object_list for any tips of the reachability query which do not have corresponding bitmaps. The object_list is not used outside of `find_objects()`, but we never free it with `object_list_free()`, resulting in a leak. Let's plug that leak by calling `object_list_free()`, which results in t6113 becoming leak-free. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-12-14 22:23:45 +00:00
rev-list: fallback to non-bitmap traversal when filtering The "--use-bitmap-index" option is usually aspirational: if we have bitmaps and the request can be fulfilled more quickly using them we'll do so, but otherwise fall back to a non-bitmap traversal. The exception is object filtering, which explicitly dies if the two options are combined. Let's convert this to the usual fallback behavior. This is a minor convenience for now (since the caller can easily know that --filter and --use-bitmap-index don't combine), but will become much more useful as we start to support _some_ filters with bitmaps, but not others. The test infrastructure here is bigger than necessary for checking this one small feature. But it will serve as the basis for more filtering bitmap tests in future patches. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-02-13 02:17:30 +00:00
test_expect_success 'set up bitmapped repo' '
# one commit will have bitmaps, the other will not
test_commit one &&
pack-bitmap: implement BLOB_LIMIT filtering Just as the previous commit implemented BLOB_NONE, we can support BLOB_LIMIT filters by looking at the sizes of any blobs in the result and unsetting their bits as appropriate. This is slightly more expensive than BLOB_NONE, but still produces a noticeable speedup (these results are on git.git): Test HEAD~2 HEAD ------------------------------------------------------------------------------------ 5310.9: rev-list count with blob:none 1.80(1.77+0.02) 0.22(0.20+0.02) -87.8% 5310.10: rev-list count with blob:limit=1k 1.99(1.96+0.03) 0.29(0.25+0.03) -85.4% The implementation is similar to the BLOB_NONE one, with the exception that we have to go object-by-object while walking the blob-type bitmap (since we can't mask out the matches, but must look up the size individually for each blob). The trick with using ctz64() is taken from show_objects_for_type(), which likewise needs to find individual bits (but wants to quickly skip over big chunks without blobs). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-02-14 18:22:39 +00:00
test_commit much-larger-blob-one &&
rev-list: fallback to non-bitmap traversal when filtering The "--use-bitmap-index" option is usually aspirational: if we have bitmaps and the request can be fulfilled more quickly using them we'll do so, but otherwise fall back to a non-bitmap traversal. The exception is object filtering, which explicitly dies if the two options are combined. Let's convert this to the usual fallback behavior. This is a minor convenience for now (since the caller can easily know that --filter and --use-bitmap-index don't combine), but will become much more useful as we start to support _some_ filters with bitmaps, but not others. The test infrastructure here is bigger than necessary for checking this one small feature. But it will serve as the basis for more filtering bitmap tests in future patches. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-02-13 02:17:30 +00:00
git repack -adb &&
pack-bitmap: implement BLOB_LIMIT filtering Just as the previous commit implemented BLOB_NONE, we can support BLOB_LIMIT filters by looking at the sizes of any blobs in the result and unsetting their bits as appropriate. This is slightly more expensive than BLOB_NONE, but still produces a noticeable speedup (these results are on git.git): Test HEAD~2 HEAD ------------------------------------------------------------------------------------ 5310.9: rev-list count with blob:none 1.80(1.77+0.02) 0.22(0.20+0.02) -87.8% 5310.10: rev-list count with blob:limit=1k 1.99(1.96+0.03) 0.29(0.25+0.03) -85.4% The implementation is similar to the BLOB_NONE one, with the exception that we have to go object-by-object while walking the blob-type bitmap (since we can't mask out the matches, but must look up the size individually for each blob). The trick with using ctz64() is taken from show_objects_for_type(), which likewise needs to find individual bits (but wants to quickly skip over big chunks without blobs). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-02-14 18:22:39 +00:00
test_commit two &&
pack-bitmap: implement object type filter The preceding commit has added a new object filter for git-rev-list(1) which allows to filter objects by type. Implement the equivalent filter for packfile bitmaps so that we can answer these queries fast. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-19 11:46:58 +00:00
test_commit much-larger-blob-two &&
git tag tag
rev-list: fallback to non-bitmap traversal when filtering The "--use-bitmap-index" option is usually aspirational: if we have bitmaps and the request can be fulfilled more quickly using them we'll do so, but otherwise fall back to a non-bitmap traversal. The exception is object filtering, which explicitly dies if the two options are combined. Let's convert this to the usual fallback behavior. This is a minor convenience for now (since the caller can easily know that --filter and --use-bitmap-index don't combine), but will become much more useful as we start to support _some_ filters with bitmaps, but not others. The test infrastructure here is bigger than necessary for checking this one small feature. But it will serve as the basis for more filtering bitmap tests in future patches. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-02-13 02:17:30 +00:00
'
test_expect_success 'filters fallback to non-bitmap traversal' '
# use a path-based filter, since they are inherently incompatible with
# bitmaps (i.e., this test will never get confused by later code to
# combine the features)
filter=$(echo "!one" | git hash-object -w --stdin) &&
git rev-list --objects --filter=sparse:oid=$filter HEAD >expect &&
git rev-list --use-bitmap-index \
--objects --filter=sparse:oid=$filter HEAD >actual &&
test_cmp expect actual
'
pack-bitmap: implement BLOB_NONE filtering We can easily support BLOB_NONE filters with bitmaps. Since we know the types of all of the objects, we just need to clear the result bits of any blobs. Note two subtleties in the implementation (which I also called out in comments): - we have to include any blobs that were specifically asked for (and not reached through graph traversal) to match the non-bitmap version - we have to handle in-pack and "ext_index" objects separately. Arguably prepare_bitmap_walk() could be adding these ext_index objects to the type bitmaps. But it doesn't for now, so let's match the rest of the bitmap code here (it probably wouldn't be an efficiency improvement to do so since the cost of extending those bitmaps is about the same as our loop here, but it might make the code a bit simpler). Here are perf results for the new test on git.git: Test HEAD^ HEAD -------------------------------------------------------------------------------- 5310.9: rev-list count with blob:none 1.67(1.62+0.05) 0.22(0.21+0.02) -86.8% Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-02-14 18:22:36 +00:00
test_expect_success 'blob:none filter' '
git rev-list --objects --filter=blob:none HEAD >expect &&
git rev-list --use-bitmap-index \
--objects --filter=blob:none HEAD >actual &&
test_bitmap_traversal expect actual
'
test_expect_success 'blob:none filter with specified blob' '
git rev-list --objects --filter=blob:none HEAD HEAD:two.t >expect &&
git rev-list --use-bitmap-index \
--objects --filter=blob:none HEAD HEAD:two.t >actual &&
test_bitmap_traversal expect actual
'
pack-bitmap: implement BLOB_LIMIT filtering Just as the previous commit implemented BLOB_NONE, we can support BLOB_LIMIT filters by looking at the sizes of any blobs in the result and unsetting their bits as appropriate. This is slightly more expensive than BLOB_NONE, but still produces a noticeable speedup (these results are on git.git): Test HEAD~2 HEAD ------------------------------------------------------------------------------------ 5310.9: rev-list count with blob:none 1.80(1.77+0.02) 0.22(0.20+0.02) -87.8% 5310.10: rev-list count with blob:limit=1k 1.99(1.96+0.03) 0.29(0.25+0.03) -85.4% The implementation is similar to the BLOB_NONE one, with the exception that we have to go object-by-object while walking the blob-type bitmap (since we can't mask out the matches, but must look up the size individually for each blob). The trick with using ctz64() is taken from show_objects_for_type(), which likewise needs to find individual bits (but wants to quickly skip over big chunks without blobs). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-02-14 18:22:39 +00:00
test_expect_success 'blob:limit filter' '
git rev-list --objects --filter=blob:limit=5 HEAD >expect &&
git rev-list --use-bitmap-index \
--objects --filter=blob:limit=5 HEAD >actual &&
test_bitmap_traversal expect actual
'
test_expect_success 'blob:limit filter with specified blob' '
git rev-list --objects --filter=blob:limit=5 \
HEAD HEAD:much-larger-blob-two.t >expect &&
git rev-list --use-bitmap-index \
--objects --filter=blob:limit=5 \
HEAD HEAD:much-larger-blob-two.t >actual &&
test_bitmap_traversal expect actual
'
pack-bitmap.c: support 'tree:0' filtering In the previous patch, we made it easy to define other filters that exclude all objects of a certain type. Use that in order to implement bitmap-level filtering for the '--filter=tree:<n>' filter when 'n' is equal to 0. The general case is not helped by bitmaps, since for values of 'n > 0', the object filtering machinery requires a full-blown tree traversal in order to determine the depth of a given tree. Caching this is non-obvious, too, since the same tree object can have a different depth depending on the context (e.g., a tree was moved up in the directory hierarchy between two commits). But, the 'n = 0' case can be helped, and this patch does so. Running p5310.11 in this tree and on master with the kernel, we can see that this case is helped substantially: Test master this tree -------------------------------------------------------------------------------- 5310.11: rev-list count with tree:0 10.68(10.39+0.27) 0.06(0.04+0.01) -99.4% Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-05-04 23:12:35 +00:00
test_expect_success 'tree:0 filter' '
git rev-list --objects --filter=tree:0 HEAD >expect &&
git rev-list --use-bitmap-index \
--objects --filter=tree:0 HEAD >actual &&
test_bitmap_traversal expect actual
'
test_expect_success 'tree:0 filter with specified blob, tree' '
git rev-list --objects --filter=tree:0 HEAD HEAD:two.t >expect &&
git rev-list --use-bitmap-index \
--objects --filter=tree:0 HEAD HEAD:two.t >actual &&
test_bitmap_traversal expect actual
'
test_expect_success 'tree:1 filter' '
git rev-list --objects --filter=tree:1 HEAD >expect &&
git rev-list --use-bitmap-index \
--objects --filter=tree:1 HEAD >actual &&
test_cmp expect actual
'
pack-bitmap: implement object type filter The preceding commit has added a new object filter for git-rev-list(1) which allows to filter objects by type. Implement the equivalent filter for packfile bitmaps so that we can answer these queries fast. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-19 11:46:58 +00:00
test_expect_success 'object:type filter' '
git rev-list --objects --filter=object:type=tag tag >expect &&
git rev-list --use-bitmap-index \
--objects --filter=object:type=tag tag >actual &&
test_cmp expect actual &&
git rev-list --objects --filter=object:type=commit tag >expect &&
git rev-list --use-bitmap-index \
--objects --filter=object:type=commit tag >actual &&
test_bitmap_traversal expect actual &&
git rev-list --objects --filter=object:type=tree tag >expect &&
git rev-list --use-bitmap-index \
--objects --filter=object:type=tree tag >actual &&
test_bitmap_traversal expect actual &&
git rev-list --objects --filter=object:type=blob tag >expect &&
git rev-list --use-bitmap-index \
--objects --filter=object:type=blob tag >actual &&
test_bitmap_traversal expect actual
'
rev-list: allow filtering of provided items When providing an object filter, it is currently impossible to also filter provided items. E.g. when executing `git rev-list HEAD` , the commit this reference points to will be treated as user-provided and is thus excluded from the filtering mechanism. This makes it harder than necessary to properly use the new `--filter=object:type` filter given that even if the user wants to only see blobs, he'll still see commits of provided references. Improve this by introducing a new `--filter-provided-objects` option to the git-rev-parse(1) command. If given, then all user-provided references will be subject to filtering. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-19 11:47:06 +00:00
test_expect_success 'object:type filter with --filter-provided-objects' '
git rev-list --objects --filter-provided-objects --filter=object:type=tag tag >expect &&
git rev-list --use-bitmap-index \
--objects --filter-provided-objects --filter=object:type=tag tag >actual &&
test_cmp expect actual &&
git rev-list --objects --filter-provided-objects --filter=object:type=commit tag >expect &&
git rev-list --use-bitmap-index \
--objects --filter-provided-objects --filter=object:type=commit tag >actual &&
test_bitmap_traversal expect actual &&
git rev-list --objects --filter-provided-objects --filter=object:type=tree tag >expect &&
git rev-list --use-bitmap-index \
--objects --filter-provided-objects --filter=object:type=tree tag >actual &&
test_bitmap_traversal expect actual &&
git rev-list --objects --filter-provided-objects --filter=object:type=blob tag >expect &&
git rev-list --use-bitmap-index \
--objects --filter-provided-objects --filter=object:type=blob tag >actual &&
test_bitmap_traversal expect actual
'
pack-bitmap: implement combined filter When the user has multiple objects filters specified, then this is internally represented by having a "combined" filter. These combined filters aren't yet supported by bitmap indices and can thus not be accelerated. Fix this by implementing support for these combined filters. The implementation is quite trivial: when there's a combined filter, we simply recurse into `filter_bitmap()` for all of the sub-filters. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-19 11:47:02 +00:00
test_expect_success 'combine filter' '
git rev-list --objects --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&
git rev-list --use-bitmap-index \
--objects --filter=blob:limit=1000 --filter=object:type=blob tag >actual &&
test_bitmap_traversal expect actual
'
rev-list: allow filtering of provided items When providing an object filter, it is currently impossible to also filter provided items. E.g. when executing `git rev-list HEAD` , the commit this reference points to will be treated as user-provided and is thus excluded from the filtering mechanism. This makes it harder than necessary to properly use the new `--filter=object:type` filter given that even if the user wants to only see blobs, he'll still see commits of provided references. Improve this by introducing a new `--filter-provided-objects` option to the git-rev-parse(1) command. If given, then all user-provided references will be subject to filtering. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-04-19 11:47:06 +00:00
test_expect_success 'combine filter with --filter-provided-objects' '
git rev-list --objects --filter-provided-objects --filter=blob:limit=1000 --filter=object:type=blob tag >expect &&
git rev-list --use-bitmap-index \
--objects --filter-provided-objects --filter=blob:limit=1000 --filter=object:type=blob tag >actual &&
test_bitmap_traversal expect actual &&
git cat-file --batch-check="%(objecttype) %(objectsize)" <actual >objects &&
while read objecttype objectsize
do
test "$objecttype" = blob || return 1
test "$objectsize" -le 1000 || return 1
done <objects
'
pack-bitmap: drop --unpacked non-commit objects from results When performing revision queries with `--objects` and `--use-bitmap-index`, the output may incorrectly contain objects which are packed, even when the `--unpacked` option is given. This affects traversals, but also other querying operations, like `--count`, `--disk-usage`, etc. Like in the previous commit, the fix is to exclude those objects from the result set before they are shown to the user (or, in this case, before the bitmap containing the result of the traversal is enumerated and its objects listed). This is performed by a new function in pack-bitmap.c, called `filter_packed_objects_from_bitmap()`. Note that we do not have to inspect individual bits in the result bitmap, since we know that the first N (where N is the number of objects in the bitmap's pack/MIDX) bits correspond to objects which packed by definition. In other words, for an object to have a bitmap position (not in the extended index), it must appear in either the bitmap's pack or one of the packs in its MIDX. This presents an appealing optimization to us, which is that we can simply memset() the corresponding number of `eword_t`'s to zero, provided that we handle any objects which spill into the next word (but don't occupy all 64 bits of the word itself). We only have to handle objects in the bitmap's extended index. These objects may (or may not) appear in one or more pack(s). Since these objects are known to not appear in either the bitmap's MIDX or pack, they may be stored as loose, appear in other pack(s), or both. Before returning a bitmap containing the result of the traversal back to the caller, drop any bits from the extended index which appear in one or more packs. This implements the correct behavior for rev-list operations which use the bitmap index to compute their result. Co-authored-by: Jeff King <peff@peff.net> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-11-06 22:56:33 +00:00
test_expect_success 'bitmap traversal with --unpacked' '
git repack -adb &&
test_commit unpacked &&
git rev-list --objects --no-object-names unpacked^.. >expect.raw &&
sort expect.raw >expect &&
git rev-list --use-bitmap-index --objects --all --unpacked >actual.raw &&
sort actual.raw >actual &&
test_cmp expect actual
'
rev-list: fallback to non-bitmap traversal when filtering The "--use-bitmap-index" option is usually aspirational: if we have bitmaps and the request can be fulfilled more quickly using them we'll do so, but otherwise fall back to a non-bitmap traversal. The exception is object filtering, which explicitly dies if the two options are combined. Let's convert this to the usual fallback behavior. This is a minor convenience for now (since the caller can easily know that --filter and --use-bitmap-index don't combine), but will become much more useful as we start to support _some_ filters with bitmaps, but not others. The test infrastructure here is bigger than necessary for checking this one small feature. But it will serve as the basis for more filtering bitmap tests in future patches. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-02-13 02:17:30 +00:00
test_done
Reference in a new issue Copy permalink