development/git
development/git
1
0
Fork 0
mirror of https://github.com/git/git synced 2024-07-05 00:58:49 +00:00
Code Issues Projects Releases Wiki Activity
64a6151da7
git/t/t7700-repack.sh

421 lines
12 KiB
Bash
Raw Normal View History

t7700: demonstrate mishandling of objects in packs with a .keep file Objects residing in pack files that have an associated .keep file are not supposed to be repacked into new pack files, but they are. Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-11-12 17:59:02 +00:00
#!/bin/sh
test_description='git repack works correctly'
. ./test-lib.sh
builtin/repack.c: support writing a MIDX while repacking Teach `git repack` a new `--write-midx` option for callers that wish to persist a multi-pack index in their repository while repacking. There are two existing alternatives to this new flag, but they don't cover our particular use-case. These alternatives are: - Call 'git multi-pack-index write' after running 'git repack', or - Set 'GIT_TEST_MULTI_PACK_INDEX=1' in your environment when running 'git repack'. The former works, but introduces a gap in bitmap coverage between repacking and writing a new MIDX (since the repack may have deleted a pack included in the existing MIDX, invalidating it altogether). Setting the 'GIT_TEST_' environment variable is obviously unsupported. In fact, even if it were supported officially, it still wouldn't work, because it generates the MIDX *after* redundant packs have been dropped, leading to the same issue as above. Introduce a new option which eliminates this race by teaching `git repack` to generate the MIDX at the critical point: after the new packs have been written and moved into place, but before the redundant packs have been removed. This option is compatible with `git repack`'s '--bitmap' option (it changes the interpretation to be: "write a bitmap corresponding to the MIDX after one has been generated"). There is a little bit of additional noise in the patch below to avoid repeating ourselves when selecting which packs to delete. Instead of a single loop as before (where we iterate over 'existing_packs', decide if a pack is worth deleting, and if so, delete it), we have two loops (the first where we decide which ones are worth deleting, and the second where we actually do the deleting). This makes it so we have a single check we can make consistently when (1) telling the MIDX which packs we want to exclude, and (2) actually unlinking the redundant packs. There is also a tiny change to short-circuit the body of write_midx_included_packs() when no packs remain in the case of an empty repository. The MIDX code does not handle this, so avoid trying to generate a MIDX covering zero packs in the first place. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-29 01:55:18 +00:00
. "${TEST_DIRECTORY}/lib-bitmap.sh"
. "${TEST_DIRECTORY}/lib-midx.sh"
repack: make '--quiet' disable progress While testing some ideas in 'git repack', I ran it with '--quiet' and discovered that some progress output was still shown. Specifically, the output for writing the multi-pack-index showed the progress. The 'show_progress' variable in cmd_repack() is initialized with isatty(2) and is not modified at all by the '--quiet' flag. The '--quiet' flag modifies the po_args.quiet option which is translated into a '--quiet' flag for the 'git pack-objects' child process. However, 'show_progress' is used to directly send progress information to the multi-pack-index writing logic which does not use a child process. The fix here is to modify 'show_progress' to be false if po_opts.quiet is true, and isatty(2) otherwise. This new expectation simplifies a later condition that checks both. Update the documentation to make it clear that '-q' will disable all progress in addition to ensuring the 'git pack-objects' child process will receive the flag. Use 'test_terminal' to check that this works to get around the isatty(2) check. Helped-by: Jeff King <peff@peff.net> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-12-20 14:48:11 +00:00
. "${TEST_DIRECTORY}/lib-terminal.sh"
t7700: demonstrate mishandling of objects in packs with a .keep file Objects residing in pack files that have an associated .keep file are not supposed to be repacked into new pack files, but they are. Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-11-12 17:59:02 +00:00
t7700: consolidate code into test_no_missing_in_packs() The code to test that objects were not missing from the packfile was duplicated many times. Extract the duplicated code into test_no_missing_in_packs() and use that instead. Refactor the resulting extraction so that if any git commands fail, their return codes are not silently lost. Instead of verifying each file of `alt_objects/pack/*.idx` individually in a for-loop, batch them together into one verification step. The original testing construct was O(n^2): it used a grep in a loop to test whether any objects were missing in the packfile. Rewrite this to extract the hash using sed or cut, sort the files, then use `comm -23` so that finding missing lines from the original file is done more efficiently. While we're at it, add a space to `commit_and_pack ()` for style. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:09 +00:00
commit_and_pack () {
t7700: drop redirections to /dev/null Since output is silenced when running without `-v` and debugging output is useful with `-v`, remove redirections to /dev/null as it is not useful. In one case where the output of stdout is consumed, redirect the output of test_commit to stderr. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-27 19:53:45 +00:00
test_commit "$@" 1>&2 &&
t7700: make references to SHA-1 generic Make the test more hash-agnostic by renaming variables from "sha1" to some variation of "oid" or "packid". Also, replace the regex, `[0-9a-f]\{40\}` with `$OID_REGEX`. A better name for "incrpackid" (incremental pack-id) might have been just "packid". However, later in the test suite, we have other uses of "packid". Although the scopes of these variables don't conflict, a future developer may think that commit_and_pack() and test_has_duplicate_object() are semantically related somehow since they share the same variable name. Give them distinct names so that it's clear these uses are unrelated. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:24 +00:00
incrpackid=$(git pack-objects --all --unpacked --incremental .git/objects/pack/pack </dev/null) &&
echo pack-${incrpackid}.pack
repack: add --keep-pack option We allow to keep existing packs by having companion .keep files. This is helpful when a pack is permanently kept. In the next patch, git-gc just wants to keep a pack temporarily, for one pack-objects run. git-gc can use --keep-pack for this use case. A note about why the pack_keep field cannot be reused and pack_keep_in_core has to be added. This is about the case when --keep-pack is specified together with either --keep-unreachable or --unpack-unreachable, but --honor-pack-keep is NOT specified. In this case, we want to exclude objects from the packs specified on command line, not from ones with .keep files. If only one bit flag is used, we have to clear pack_keep on pack files with the .keep file. But we can't make any assumption about unreachable objects in .keep packs. If "pack_keep" field is false for .keep packs, we could potentially pull lots of unreachable objects into the new pack, or unpack them loose. The safer approach is ignore all packs with either .keep file or --keep-pack. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-04-15 15:36:13 +00:00
}
t7700: consolidate code into test_no_missing_in_packs() The code to test that objects were not missing from the packfile was duplicated many times. Extract the duplicated code into test_no_missing_in_packs() and use that instead. Refactor the resulting extraction so that if any git commands fail, their return codes are not silently lost. Instead of verifying each file of `alt_objects/pack/*.idx` individually in a for-loop, batch them together into one verification step. The original testing construct was O(n^2): it used a grep in a loop to test whether any objects were missing in the packfile. Rewrite this to extract the hash using sed or cut, sort the files, then use `comm -23` so that finding missing lines from the original file is done more efficiently. While we're at it, add a space to `commit_and_pack ()` for style. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:09 +00:00
test_no_missing_in_packs () {
myidx=$(ls -1 .git/objects/pack/*.idx) &&
test_path_is_file "$myidx" &&
git verify-pack -v alt_objects/pack/*.idx >orig.raw &&
t7700: make references to SHA-1 generic Make the test more hash-agnostic by renaming variables from "sha1" to some variation of "oid" or "packid". Also, replace the regex, `[0-9a-f]\{40\}` with `$OID_REGEX`. A better name for "incrpackid" (incremental pack-id) might have been just "packid". However, later in the test suite, we have other uses of "packid". Although the scopes of these variables don't conflict, a future developer may think that commit_and_pack() and test_has_duplicate_object() are semantically related somehow since they share the same variable name. Give them distinct names so that it's clear these uses are unrelated. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:24 +00:00
sed -n -e "s/^\($OID_REGEX\).*/\1/p" orig.raw | sort >orig &&
t7700: consolidate code into test_no_missing_in_packs() The code to test that objects were not missing from the packfile was duplicated many times. Extract the duplicated code into test_no_missing_in_packs() and use that instead. Refactor the resulting extraction so that if any git commands fail, their return codes are not silently lost. Instead of verifying each file of `alt_objects/pack/*.idx` individually in a for-loop, batch them together into one verification step. The original testing construct was O(n^2): it used a grep in a loop to test whether any objects were missing in the packfile. Rewrite this to extract the hash using sed or cut, sort the files, then use `comm -23` so that finding missing lines from the original file is done more efficiently. While we're at it, add a space to `commit_and_pack ()` for style. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:09 +00:00
git verify-pack -v $myidx >dest.raw &&
cut -d" " -f1 dest.raw | sort >dest &&
comm -23 orig dest >missing &&
test_must_be_empty missing
}
t7700: make references to SHA-1 generic Make the test more hash-agnostic by renaming variables from "sha1" to some variation of "oid" or "packid". Also, replace the regex, `[0-9a-f]\{40\}` with `$OID_REGEX`. A better name for "incrpackid" (incremental pack-id) might have been just "packid". However, later in the test suite, we have other uses of "packid". Although the scopes of these variables don't conflict, a future developer may think that commit_and_pack() and test_has_duplicate_object() are semantically related somehow since they share the same variable name. Give them distinct names so that it's clear these uses are unrelated. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:24 +00:00
# we expect $packid and $oid to be defined
t7700: consolidate code into test_has_duplicate_object() The code to test that objects were not duplicated from the packfile was duplicated many times. Extract the duplicated code into test_has_duplicate_object() and use that instead. Refactor the resulting extraction so that if the git command fails, the return code is not silently lost. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:14 +00:00
test_has_duplicate_object () {
want_duplicate_object="$1"
found_duplicate_object=false
for p in .git/objects/pack/*.idx
do
idx=$(basename $p)
t7700: make references to SHA-1 generic Make the test more hash-agnostic by renaming variables from "sha1" to some variation of "oid" or "packid". Also, replace the regex, `[0-9a-f]\{40\}` with `$OID_REGEX`. A better name for "incrpackid" (incremental pack-id) might have been just "packid". However, later in the test suite, we have other uses of "packid". Although the scopes of these variables don't conflict, a future developer may think that commit_and_pack() and test_has_duplicate_object() are semantically related somehow since they share the same variable name. Give them distinct names so that it's clear these uses are unrelated. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:24 +00:00
test "pack-$packid.idx" = "$idx" && continue
t7700: consolidate code into test_has_duplicate_object() The code to test that objects were not duplicated from the packfile was duplicated many times. Extract the duplicated code into test_has_duplicate_object() and use that instead. Refactor the resulting extraction so that if the git command fails, the return code is not silently lost. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:14 +00:00
git verify-pack -v $p >packlist || return $?
t7700: make references to SHA-1 generic Make the test more hash-agnostic by renaming variables from "sha1" to some variation of "oid" or "packid". Also, replace the regex, `[0-9a-f]\{40\}` with `$OID_REGEX`. A better name for "incrpackid" (incremental pack-id) might have been just "packid". However, later in the test suite, we have other uses of "packid". Although the scopes of these variables don't conflict, a future developer may think that commit_and_pack() and test_has_duplicate_object() are semantically related somehow since they share the same variable name. Give them distinct names so that it's clear these uses are unrelated. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:24 +00:00
if grep "^$oid" packlist
t7700: consolidate code into test_has_duplicate_object() The code to test that objects were not duplicated from the packfile was duplicated many times. Extract the duplicated code into test_has_duplicate_object() and use that instead. Refactor the resulting extraction so that if the git command fails, the return code is not silently lost. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:14 +00:00
then
found_duplicate_object=true
echo "DUPLICATE OBJECT FOUND"
break
fi
done &&
test "$want_duplicate_object" = "$found_duplicate_object"
}
repack: don't repack local objects in packs with .keep file If the user created a .keep file for a local pack, then it can be inferred that the user does not want those objects repacked. This fixes the repack bug tested by t7700. Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-11-12 17:59:05 +00:00
test_expect_success 'objects in packs marked .keep are not repacked' '
t7700: remove spaces after redirect operators For shell scripts, the usual convention is for there to be no space after redirection operators, (e.g. `>file`, not `> file`). Remove these spaces wherever they appear. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-27 19:53:47 +00:00
echo content1 >file1 &&
echo content2 >file2 &&
t7700: demonstrate mishandling of objects in packs with a .keep file Objects residing in pack files that have an associated .keep file are not supposed to be repacked into new pack files, but they are. Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-11-12 17:59:02 +00:00
git add . &&
more war on "sleep" in tests Two more tests that sleep only to waste tick can be converted to use test_tick and take expiry parameters relative to $test_tick. The basic idea is to replace "sleep 1" with "test_tick" to cause the "time" to pass. These tests are interested in expiring things with "now" as the timestamp, soo use a timestamp relative to $test_tick to give them more stability and reproducibility. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-04-14 22:09:57 +00:00
test_tick &&
t7700: demonstrate mishandling of objects in packs with a .keep file Objects residing in pack files that have an associated .keep file are not supposed to be repacked into new pack files, but they are. Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-11-12 17:59:02 +00:00
git commit -m initial_commit &&
# Create two packs
# The first pack will contain all of the objects except one
t7700: stop losing return codes of git commands In a pipe, only the return code of the last command is used. Thus, all other commands will have their return codes masked. Rewrite pipes so that there are no git commands upstream so that we will know if a command fails. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:30 +00:00
git rev-list --objects --all >objs &&
grep -v file2 objs | git pack-objects pack &&
t7700: demonstrate mishandling of objects in packs with a .keep file Objects residing in pack files that have an associated .keep file are not supposed to be repacked into new pack files, but they are. Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-11-12 17:59:02 +00:00
# The second pack will contain the excluded object
t7700: stop losing return codes of git commands In a pipe, only the return code of the last command is used. Thus, all other commands will have their return codes masked. Rewrite pipes so that there are no git commands upstream so that we will know if a command fails. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:30 +00:00
packid=$(grep file2 objs | git pack-objects pack) &&
t7700: make references to SHA-1 generic Make the test more hash-agnostic by renaming variables from "sha1" to some variation of "oid" or "packid". Also, replace the regex, `[0-9a-f]\{40\}` with `$OID_REGEX`. A better name for "incrpackid" (incremental pack-id) might have been just "packid". However, later in the test suite, we have other uses of "packid". Although the scopes of these variables don't conflict, a future developer may think that commit_and_pack() and test_has_duplicate_object() are semantically related somehow since they share the same variable name. Give them distinct names so that it's clear these uses are unrelated. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:24 +00:00
>pack-$packid.keep &&
t7700: stop losing return codes of git commands In a pipe, only the return code of the last command is used. Thus, all other commands will have their return codes masked. Rewrite pipes so that there are no git commands upstream so that we will know if a command fails. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:30 +00:00
git verify-pack -v pack-$packid.idx >packlist &&
oid=$(head -n 1 packlist | sed -e "s/^\($OID_REGEX\).*/\1/") &&
t7700: demonstrate mishandling of objects in packs with a .keep file Objects residing in pack files that have an associated .keep file are not supposed to be repacked into new pack files, but they are. Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-11-12 17:59:02 +00:00
mv pack-* .git/objects/pack/ &&
t7700: drop explicit --no-pack-kept-objects from .keep test We want to make sure that the default behavior of git-repack, without any options, continues to treat .keep files as it always has. Adding an explicit --no-pack-kept-objects, as ee34a2b did, is a much less interesting test, and prevented us from noticing the bug fixed by 64d3dc9 (repack: do not accidentally pack kept objects by default, 2014-06-10). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-06-11 06:32:45 +00:00
git repack -A -d -l &&
t7700: demonstrate mishandling of objects in packs with a .keep file Objects residing in pack files that have an associated .keep file are not supposed to be repacked into new pack files, but they are. Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-11-12 17:59:02 +00:00
git prune-packed &&
t7700: consolidate code into test_has_duplicate_object() The code to test that objects were not duplicated from the packfile was duplicated many times. Extract the duplicated code into test_has_duplicate_object() and use that instead. Refactor the resulting extraction so that if the git command fails, the return code is not silently lost. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:14 +00:00
test_has_duplicate_object false
t7700: demonstrate mishandling of objects in packs with a .keep file Objects residing in pack files that have an associated .keep file are not supposed to be repacked into new pack files, but they are. Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-11-12 17:59:02 +00:00
'
repack: respect pack.writebitmaps The config option to turn on bitmaps is read all the way down in the plumbing of pack-objects. This makes it hard for other options in the porcelain of repack to make decisions based on the bitmap setting. For example, repack.packKeptObjects tries to kick in by default only when bitmaps are turned on. But it can't do so reliably because it doesn't yet know whether we are using bitmaps. This patch teaches repack to respect pack.writebitmaps. It means we pass a redundant command-line flag to pack-objects, but that's OK; it shouldn't affect the outcome. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-06-10 20:09:23 +00:00
test_expect_success 'writing bitmaps via command-line can duplicate .keep objects' '
t7700: make references to SHA-1 generic Make the test more hash-agnostic by renaming variables from "sha1" to some variation of "oid" or "packid". Also, replace the regex, `[0-9a-f]\{40\}` with `$OID_REGEX`. A better name for "incrpackid" (incremental pack-id) might have been just "packid". However, later in the test suite, we have other uses of "packid". Although the scopes of these variables don't conflict, a future developer may think that commit_and_pack() and test_has_duplicate_object() are semantically related somehow since they share the same variable name. Give them distinct names so that it's clear these uses are unrelated. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:24 +00:00
# build on $oid, $packid, and .keep state from previous
t7700: update to work with MIDX bitmap test knob A number of these tests are focused only on pack-based bitmaps and need to be updated to disable 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' where necessary. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-08-31 20:52:41 +00:00
GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 git repack -Adbl &&
t7700: consolidate code into test_has_duplicate_object() The code to test that objects were not duplicated from the packfile was duplicated many times. Extract the duplicated code into test_has_duplicate_object() and use that instead. Refactor the resulting extraction so that if the git command fails, the return code is not silently lost. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:14 +00:00
test_has_duplicate_object true
repack: add `repack.packKeptObjects` config var The git-repack command always passes `--honor-pack-keep` to pack-objects. This has traditionally been a good thing, as we do not want to duplicate those objects in a new pack, and we are not going to delete the old pack. However, when bitmaps are in use, it is important for a full repack to include all reachable objects, even if they may be duplicated in a .keep pack. Otherwise, we cannot generate the bitmaps, as the on-disk format requires the set of objects in the pack to be fully closed. Even if the repository does not generally have .keep files, a simultaneous push could cause a race condition in which a .keep file exists at the moment of a repack. The repack may try to include those objects in one of two situations: 1. The pushed .keep pack contains objects that were already in the repository (e.g., blobs due to a revert of an old commit). 2. Receive-pack updates the refs, making the objects reachable, but before it removes the .keep file, the repack runs. In either case, we may prefer to duplicate some objects in the new, full pack, and let the next repack (after the .keep file is cleaned up) take care of removing them. This patch introduces both a command-line and config option to disable the `--honor-pack-keep` option. By default, it is triggered when pack.writeBitmaps (or `--write-bitmap-index` is turned on), but specifying it explicitly can override the behavior (e.g., in cases where you prefer .keep files to bitmaps, but only when they are present). Note that this option just disables the pack-objects behavior. We still leave packs with a .keep in place, as we do not necessarily know that we have duplicated all of their objects. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-03-03 20:04:20 +00:00
'
repack: respect pack.writebitmaps The config option to turn on bitmaps is read all the way down in the plumbing of pack-objects. This makes it hard for other options in the porcelain of repack to make decisions based on the bitmap setting. For example, repack.packKeptObjects tries to kick in by default only when bitmaps are turned on. But it can't do so reliably because it doesn't yet know whether we are using bitmaps. This patch teaches repack to respect pack.writebitmaps. It means we pass a redundant command-line flag to pack-objects, but that's OK; it shouldn't affect the outcome. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-06-10 20:09:23 +00:00
test_expect_success 'writing bitmaps via config can duplicate .keep objects' '
t7700: make references to SHA-1 generic Make the test more hash-agnostic by renaming variables from "sha1" to some variation of "oid" or "packid". Also, replace the regex, `[0-9a-f]\{40\}` with `$OID_REGEX`. A better name for "incrpackid" (incremental pack-id) might have been just "packid". However, later in the test suite, we have other uses of "packid". Although the scopes of these variables don't conflict, a future developer may think that commit_and_pack() and test_has_duplicate_object() are semantically related somehow since they share the same variable name. Give them distinct names so that it's clear these uses are unrelated. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:24 +00:00
# build on $oid, $packid, and .keep state from previous
t7700: update to work with MIDX bitmap test knob A number of these tests are focused only on pack-based bitmaps and need to be updated to disable 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' where necessary. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-08-31 20:52:41 +00:00
GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
git -c repack.writebitmaps=true repack -Adl &&
t7700: consolidate code into test_has_duplicate_object() The code to test that objects were not duplicated from the packfile was duplicated many times. Extract the duplicated code into test_has_duplicate_object() and use that instead. Refactor the resulting extraction so that if the git command fails, the return code is not silently lost. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:14 +00:00
test_has_duplicate_object true
repack: add `repack.packKeptObjects` config var The git-repack command always passes `--honor-pack-keep` to pack-objects. This has traditionally been a good thing, as we do not want to duplicate those objects in a new pack, and we are not going to delete the old pack. However, when bitmaps are in use, it is important for a full repack to include all reachable objects, even if they may be duplicated in a .keep pack. Otherwise, we cannot generate the bitmaps, as the on-disk format requires the set of objects in the pack to be fully closed. Even if the repository does not generally have .keep files, a simultaneous push could cause a race condition in which a .keep file exists at the moment of a repack. The repack may try to include those objects in one of two situations: 1. The pushed .keep pack contains objects that were already in the repository (e.g., blobs due to a revert of an old commit). 2. Receive-pack updates the refs, making the objects reachable, but before it removes the .keep file, the repack runs. In either case, we may prefer to duplicate some objects in the new, full pack, and let the next repack (after the .keep file is cleaned up) take care of removing them. This patch introduces both a command-line and config option to disable the `--honor-pack-keep` option. By default, it is triggered when pack.writeBitmaps (or `--write-bitmap-index` is turned on), but specifying it explicitly can override the behavior (e.g., in cases where you prefer .keep files to bitmaps, but only when they are present). Note that this option just disables the pack-objects behavior. We still leave packs with a .keep in place, as we do not necessarily know that we have duplicated all of their objects. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-03-03 20:04:20 +00:00
'
pack-objects: extend --local to mean ignore non-local loose objects too With this patch, --local means pack only local objects that are not already packed. Additionally, this fixes t7700 testing whether loose objects in an alternate object database are repacked. Signed-off-by: Brandon Casey <drafnel@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-11-10 05:59:58 +00:00
test_expect_success 'loose objects in alternate ODB are not repacked' '
t7700: demonstrate mishandling of loose objects in an alternate ODB Loose objects residing in an alternate object database should not be packed when the -l option to repack is used. Signed-off-by: Brandon Casey <drafnel@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-11-10 05:59:56 +00:00
mkdir alt_objects &&
t7700: remove spaces after redirect operators For shell scripts, the usual convention is for there to be no space after redirection operators, (e.g. `>file`, not `> file`). Remove these spaces wherever they appear. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-27 19:53:47 +00:00
echo $(pwd)/alt_objects >.git/objects/info/alternates &&
echo content3 >file3 &&
t7700: make references to SHA-1 generic Make the test more hash-agnostic by renaming variables from "sha1" to some variation of "oid" or "packid". Also, replace the regex, `[0-9a-f]\{40\}` with `$OID_REGEX`. A better name for "incrpackid" (incremental pack-id) might have been just "packid". However, later in the test suite, we have other uses of "packid". Although the scopes of these variables don't conflict, a future developer may think that commit_and_pack() and test_has_duplicate_object() are semantically related somehow since they share the same variable name. Give them distinct names so that it's clear these uses are unrelated. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:24 +00:00
oid=$(GIT_OBJECT_DIRECTORY=alt_objects git hash-object -w file3) &&
t7700: demonstrate mishandling of loose objects in an alternate ODB Loose objects residing in an alternate object database should not be packed when the -l option to repack is used. Signed-off-by: Brandon Casey <drafnel@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-11-10 05:59:56 +00:00
git add file3 &&
more war on "sleep" in tests Two more tests that sleep only to waste tick can be converted to use test_tick and take expiry parameters relative to $test_tick. The basic idea is to replace "sleep 1" with "test_tick" to cause the "time" to pass. These tests are interested in expiring things with "now" as the timestamp, soo use a timestamp relative to $test_tick to give them more stability and reproducibility. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-04-14 22:09:57 +00:00
test_tick &&
t7700: demonstrate mishandling of loose objects in an alternate ODB Loose objects residing in an alternate object database should not be packed when the -l option to repack is used. Signed-off-by: Brandon Casey <drafnel@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-11-10 05:59:56 +00:00
git commit -m commit_file3 &&
git repack -a -d -l &&
git prune-packed &&
t7700: consolidate code into test_has_duplicate_object() The code to test that objects were not duplicated from the packfile was duplicated many times. Extract the duplicated code into test_has_duplicate_object() and use that instead. Refactor the resulting extraction so that if the git command fails, the return code is not silently lost. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:14 +00:00
test_has_duplicate_object false
t7700: demonstrate mishandling of loose objects in an alternate ODB Loose objects residing in an alternate object database should not be packed when the -l option to repack is used. Signed-off-by: Brandon Casey <drafnel@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-11-10 05:59:56 +00:00
'
t7700: test that 'repack -a' packs alternate packed objects Previously, when 'repack -a' was called and there were no packs in the local repository without a .keep file, the repack would fall back to calling pack-objects with '--unpacked --incremental'. This resulted in the created pack file, if any, to be missing the packed objects in the alternate object store. Test that this specific case has been fixed. Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-11-13 00:50:26 +00:00
test_expect_success 'packed obs in alt ODB are repacked even when local repo is packless' '
tests: add missing &&, batch 2 Same rules as before: this patch only adds " &&" to the end of some lines in the test suite. Intended to be applied on top of or squashed with the last batch if they look okay. Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-10-31 07:30:58 +00:00
mkdir alt_objects/pack &&
t7700: test that 'repack -a' packs alternate packed objects Previously, when 'repack -a' was called and there were no packs in the local repository without a .keep file, the repack would fall back to calling pack-objects with '--unpacked --incremental'. This resulted in the created pack file, if any, to be missing the packed objects in the alternate object store. Test that this specific case has been fixed. Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-11-13 00:50:26 +00:00
mv .git/objects/pack/* alt_objects/pack &&
git repack -a &&
t7700: consolidate code into test_no_missing_in_packs() The code to test that objects were not missing from the packfile was duplicated many times. Extract the duplicated code into test_no_missing_in_packs() and use that instead. Refactor the resulting extraction so that if any git commands fail, their return codes are not silently lost. Instead of verifying each file of `alt_objects/pack/*.idx` individually in a for-loop, batch them together into one verification step. The original testing construct was O(n^2): it used a grep in a loop to test whether any objects were missing in the packfile. Rewrite this to extract the hash using sed or cut, sort the files, then use `comm -23` so that finding missing lines from the original file is done more efficiently. While we're at it, add a space to `commit_and_pack ()` for style. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:09 +00:00
test_no_missing_in_packs
t7700: test that 'repack -a' packs alternate packed objects Previously, when 'repack -a' was called and there were no packs in the local repository without a .keep file, the repack would fall back to calling pack-objects with '--unpacked --incremental'. This resulted in the created pack file, if any, to be missing the packed objects in the alternate object store. Test that this specific case has been fixed. Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-11-13 00:50:26 +00:00
'
t7700-repack: repack -a now works properly, expect success from test Since the recent rework of the object listing mechanism of pack-objects/rev-list, git-repack now properly packs objects from alternate repositories even when the local repository contains packs. Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-04-24 23:18:53 +00:00
test_expect_success 'packed obs in alt ODB are repacked when local repo has packs' '
t7700: demonstrate misbehavior of 'repack -a' when local packs exist The ability to "...fatten [the] local repository by packing everything that is needed by the local ref into a single new pack, including things that are borrowed from alternates"[1] is supposed to be provided by the '-a' or '-A' options to repack when '-l' is not used, but there is a flaw. For each pack in the local repository without a .keep file, repack supplies a --unpacked=<pack> argument to pack-objects. The --unpacked option to pack-objects, with or without an argument, causes pack-objects to ignore any object which is packed in a pack not mentioned in an argument to --unpacked=. So, if there are local packs, and 'repack -a' is called, then any objects which reside in packs accessible through alternates will _not_ be packed. If there are no local packs, then no --unpacked argument will be supplied, and repack will behave as expected. [1] http://mid.gmane.org/7v8wrwidi3.fsf@gitster.siamese.dyndns.org Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-01-09 22:14:39 +00:00
rm -f .git/objects/pack/* &&
t7700: remove spaces after redirect operators For shell scripts, the usual convention is for there to be no space after redirection operators, (e.g. `>file`, not `> file`). Remove these spaces wherever they appear. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-27 19:53:47 +00:00
echo new_content >>file1 &&
t7700: demonstrate misbehavior of 'repack -a' when local packs exist The ability to "...fatten [the] local repository by packing everything that is needed by the local ref into a single new pack, including things that are borrowed from alternates"[1] is supposed to be provided by the '-a' or '-A' options to repack when '-l' is not used, but there is a flaw. For each pack in the local repository without a .keep file, repack supplies a --unpacked=<pack> argument to pack-objects. The --unpacked option to pack-objects, with or without an argument, causes pack-objects to ignore any object which is packed in a pack not mentioned in an argument to --unpacked=. So, if there are local packs, and 'repack -a' is called, then any objects which reside in packs accessible through alternates will _not_ be packed. If there are no local packs, then no --unpacked argument will be supplied, and repack will behave as expected. [1] http://mid.gmane.org/7v8wrwidi3.fsf@gitster.siamese.dyndns.org Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-01-09 22:14:39 +00:00
git add file1 &&
more war on "sleep" in tests Two more tests that sleep only to waste tick can be converted to use test_tick and take expiry parameters relative to $test_tick. The basic idea is to replace "sleep 1" with "test_tick" to cause the "time" to pass. These tests are interested in expiring things with "now" as the timestamp, soo use a timestamp relative to $test_tick to give them more stability and reproducibility. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-04-14 22:09:57 +00:00
test_tick &&
t7700: demonstrate misbehavior of 'repack -a' when local packs exist The ability to "...fatten [the] local repository by packing everything that is needed by the local ref into a single new pack, including things that are borrowed from alternates"[1] is supposed to be provided by the '-a' or '-A' options to repack when '-l' is not used, but there is a flaw. For each pack in the local repository without a .keep file, repack supplies a --unpacked=<pack> argument to pack-objects. The --unpacked option to pack-objects, with or without an argument, causes pack-objects to ignore any object which is packed in a pack not mentioned in an argument to --unpacked=. So, if there are local packs, and 'repack -a' is called, then any objects which reside in packs accessible through alternates will _not_ be packed. If there are no local packs, then no --unpacked argument will be supplied, and repack will behave as expected. [1] http://mid.gmane.org/7v8wrwidi3.fsf@gitster.siamese.dyndns.org Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-01-09 22:14:39 +00:00
git commit -m more_content &&
git repack &&
git repack -a -d &&
t7700: consolidate code into test_no_missing_in_packs() The code to test that objects were not missing from the packfile was duplicated many times. Extract the duplicated code into test_no_missing_in_packs() and use that instead. Refactor the resulting extraction so that if any git commands fail, their return codes are not silently lost. Instead of verifying each file of `alt_objects/pack/*.idx` individually in a for-loop, batch them together into one verification step. The original testing construct was O(n^2): it used a grep in a loop to test whether any objects were missing in the packfile. Rewrite this to extract the hash using sed or cut, sort the files, then use `comm -23` so that finding missing lines from the original file is done more efficiently. While we're at it, add a space to `commit_and_pack ()` for style. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:09 +00:00
test_no_missing_in_packs
t7700: demonstrate misbehavior of 'repack -a' when local packs exist The ability to "...fatten [the] local repository by packing everything that is needed by the local ref into a single new pack, including things that are borrowed from alternates"[1] is supposed to be provided by the '-a' or '-A' options to repack when '-l' is not used, but there is a flaw. For each pack in the local repository without a .keep file, repack supplies a --unpacked=<pack> argument to pack-objects. The --unpacked option to pack-objects, with or without an argument, causes pack-objects to ignore any object which is packed in a pack not mentioned in an argument to --unpacked=. So, if there are local packs, and 'repack -a' is called, then any objects which reside in packs accessible through alternates will _not_ be packed. If there are no local packs, then no --unpacked argument will be supplied, and repack will behave as expected. [1] http://mid.gmane.org/7v8wrwidi3.fsf@gitster.siamese.dyndns.org Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-01-09 22:14:39 +00:00
'
git-repack.sh: don't use --kept-pack-only option to pack-objects The --kept-pack-only option to pack-objects treats all kept packs as equal. This results in objects that reside in an alternate pack that has a .keep file, not being packed into a newly created pack when the user specifies the -a option to repack. Since the user may not have any control over the alternate database, git should not refrain from repacking those objects even though they are in a pack with a .keep file. This fixes the 'packed obs in alternate ODB kept pack are repacked' test in t7700. Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-03-20 03:47:51 +00:00
test_expect_success 'packed obs in alternate ODB kept pack are repacked' '
t7700-repack: add two new tests demonstrating repacking flaws 1) The new --kept-pack-only mechansim of rev-list/pack-objects has replaced --unpacked=. This new mechansim does not operate solely on "local" packs now. The result is that objects residing in an alternate pack which has a .keep file will not be repacked with repack -a. This flaw is only apparent when a commit object is the one residing in an alternate kept pack. 2) The 'repack unpacked objects' and 'loosen unpacked objects' mechanisms of pack-objects, i.e. --keep-unreachable and --unpack-unreachable, now do not operate solely on local packs. The --keep-unreachable option no longer has any callers, but --unpack-unreachable is used when repack is called with '-A -d' and the local repo has existing packs. In this case, objects residing in alternate, not-kept packs will be loosened, and then immediately deleted by repack's call to prune-packed. The test must manually call pack-objects to avoid the call to prune-packed that is made by repack when -d is used. Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-03-20 03:47:50 +00:00
# swap the .keep so the commit object is in the pack with .keep
for p in alt_objects/pack/*.pack
do
tests: add missing &&, batch 2 Same rules as before: this patch only adds " &&" to the end of some lines in the test suite. Intended to be applied on top of or squashed with the last batch if they look okay. Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-10-31 07:30:58 +00:00
base_name=$(basename $p .pack) &&
t7700: s/test -f/test_path_is_file/ Since we have debugging-friendly alternatives to `test -f`, replace instances of `test -f` with `test_path_is_file` so that if a command ever fails, we get better debugging information. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-27 19:53:52 +00:00
if test_path_is_file alt_objects/pack/$base_name.keep
t7700-repack: add two new tests demonstrating repacking flaws 1) The new --kept-pack-only mechansim of rev-list/pack-objects has replaced --unpacked=. This new mechansim does not operate solely on "local" packs now. The result is that objects residing in an alternate pack which has a .keep file will not be repacked with repack -a. This flaw is only apparent when a commit object is the one residing in an alternate kept pack. 2) The 'repack unpacked objects' and 'loosen unpacked objects' mechanisms of pack-objects, i.e. --keep-unreachable and --unpack-unreachable, now do not operate solely on local packs. The --keep-unreachable option no longer has any callers, but --unpack-unreachable is used when repack is called with '-A -d' and the local repo has existing packs. In this case, objects residing in alternate, not-kept packs will be loosened, and then immediately deleted by repack's call to prune-packed. The test must manually call pack-objects to avoid the call to prune-packed that is made by repack when -d is used. Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-03-20 03:47:50 +00:00
then
rm alt_objects/pack/$base_name.keep
else
touch alt_objects/pack/$base_name.keep
t6000-t9999: detect and signal failure within loop Failures within `for` and `while` loops can go unnoticed if not detected and signaled manually since the loop itself does not abort when a contained command fails, nor will a failure necessarily be detected when the loop finishes since the loop returns the exit code of the last command it ran on the final iteration, which may not be the command which failed. Therefore, detect and signal failures manually within loops using the idiom `|| return 1` (or `|| exit 1` within subshells). Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Reviewed-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-12-09 05:11:15 +00:00
fi || return 1
tests: add missing &&, batch 2 Same rules as before: this patch only adds " &&" to the end of some lines in the test suite. Intended to be applied on top of or squashed with the last batch if they look okay. Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-10-31 07:30:58 +00:00
done &&
t7700-repack: add two new tests demonstrating repacking flaws 1) The new --kept-pack-only mechansim of rev-list/pack-objects has replaced --unpacked=. This new mechansim does not operate solely on "local" packs now. The result is that objects residing in an alternate pack which has a .keep file will not be repacked with repack -a. This flaw is only apparent when a commit object is the one residing in an alternate kept pack. 2) The 'repack unpacked objects' and 'loosen unpacked objects' mechanisms of pack-objects, i.e. --keep-unreachable and --unpack-unreachable, now do not operate solely on local packs. The --keep-unreachable option no longer has any callers, but --unpack-unreachable is used when repack is called with '-A -d' and the local repo has existing packs. In this case, objects residing in alternate, not-kept packs will be loosened, and then immediately deleted by repack's call to prune-packed. The test must manually call pack-objects to avoid the call to prune-packed that is made by repack when -d is used. Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-03-20 03:47:50 +00:00
git repack -a -d &&
t7700: consolidate code into test_no_missing_in_packs() The code to test that objects were not missing from the packfile was duplicated many times. Extract the duplicated code into test_no_missing_in_packs() and use that instead. Refactor the resulting extraction so that if any git commands fail, their return codes are not silently lost. Instead of verifying each file of `alt_objects/pack/*.idx` individually in a for-loop, batch them together into one verification step. The original testing construct was O(n^2): it used a grep in a loop to test whether any objects were missing in the packfile. Rewrite this to extract the hash using sed or cut, sort the files, then use `comm -23` so that finding missing lines from the original file is done more efficiently. While we're at it, add a space to `commit_and_pack ()` for style. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:09 +00:00
test_no_missing_in_packs
t7700-repack: add two new tests demonstrating repacking flaws 1) The new --kept-pack-only mechansim of rev-list/pack-objects has replaced --unpacked=. This new mechansim does not operate solely on "local" packs now. The result is that objects residing in an alternate pack which has a .keep file will not be repacked with repack -a. This flaw is only apparent when a commit object is the one residing in an alternate kept pack. 2) The 'repack unpacked objects' and 'loosen unpacked objects' mechanisms of pack-objects, i.e. --keep-unreachable and --unpack-unreachable, now do not operate solely on local packs. The --keep-unreachable option no longer has any callers, but --unpack-unreachable is used when repack is called with '-A -d' and the local repo has existing packs. In this case, objects residing in alternate, not-kept packs will be loosened, and then immediately deleted by repack's call to prune-packed. The test must manually call pack-objects to avoid the call to prune-packed that is made by repack when -d is used. Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-03-20 03:47:50 +00:00
'
pack-objects: only repack or loosen objects residing in "local" packs These two features were invented for use by repack when repack will delete the local packs that have been made redundant. The packs accessible through alternates are not deleted by repack, so the objects contained in them are still accessible after the local packs are deleted. They do not need to be repacked into the new pack or loosened. For the case of loosening they would immediately be deleted by the subsequent prune-packed that is called by repack anyway. This fixes the test 'packed unreachable obs in alternate ODB are not loosened' in t7700. Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-03-20 03:47:52 +00:00
test_expect_success 'packed unreachable obs in alternate ODB are not loosened' '
t7700-repack: add two new tests demonstrating repacking flaws 1) The new --kept-pack-only mechansim of rev-list/pack-objects has replaced --unpacked=. This new mechansim does not operate solely on "local" packs now. The result is that objects residing in an alternate pack which has a .keep file will not be repacked with repack -a. This flaw is only apparent when a commit object is the one residing in an alternate kept pack. 2) The 'repack unpacked objects' and 'loosen unpacked objects' mechanisms of pack-objects, i.e. --keep-unreachable and --unpack-unreachable, now do not operate solely on local packs. The --keep-unreachable option no longer has any callers, but --unpack-unreachable is used when repack is called with '-A -d' and the local repo has existing packs. In this case, objects residing in alternate, not-kept packs will be loosened, and then immediately deleted by repack's call to prune-packed. The test must manually call pack-objects to avoid the call to prune-packed that is made by repack when -d is used. Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-03-20 03:47:50 +00:00
rm -f alt_objects/pack/*.keep &&
mv .git/objects/pack/* alt_objects/pack/ &&
t7700: make references to SHA-1 generic Make the test more hash-agnostic by renaming variables from "sha1" to some variation of "oid" or "packid". Also, replace the regex, `[0-9a-f]\{40\}` with `$OID_REGEX`. A better name for "incrpackid" (incremental pack-id) might have been just "packid". However, later in the test suite, we have other uses of "packid". Although the scopes of these variables don't conflict, a future developer may think that commit_and_pack() and test_has_duplicate_object() are semantically related somehow since they share the same variable name. Give them distinct names so that it's clear these uses are unrelated. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:24 +00:00
coid=$(git rev-parse HEAD^{commit}) &&
t7700-repack: add two new tests demonstrating repacking flaws 1) The new --kept-pack-only mechansim of rev-list/pack-objects has replaced --unpacked=. This new mechansim does not operate solely on "local" packs now. The result is that objects residing in an alternate pack which has a .keep file will not be repacked with repack -a. This flaw is only apparent when a commit object is the one residing in an alternate kept pack. 2) The 'repack unpacked objects' and 'loosen unpacked objects' mechanisms of pack-objects, i.e. --keep-unreachable and --unpack-unreachable, now do not operate solely on local packs. The --keep-unreachable option no longer has any callers, but --unpack-unreachable is used when repack is called with '-A -d' and the local repo has existing packs. In this case, objects residing in alternate, not-kept packs will be loosened, and then immediately deleted by repack's call to prune-packed. The test must manually call pack-objects to avoid the call to prune-packed that is made by repack when -d is used. Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-03-20 03:47:50 +00:00
git reset --hard HEAD^ &&
more war on "sleep" in tests Two more tests that sleep only to waste tick can be converted to use test_tick and take expiry parameters relative to $test_tick. The basic idea is to replace "sleep 1" with "test_tick" to cause the "time" to pass. These tests are interested in expiring things with "now" as the timestamp, soo use a timestamp relative to $test_tick to give them more stability and reproducibility. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-04-14 22:09:57 +00:00
test_tick &&
git reflog expire --expire=$test_tick --expire-unreachable=$test_tick --all &&
t7700-repack: add two new tests demonstrating repacking flaws 1) The new --kept-pack-only mechansim of rev-list/pack-objects has replaced --unpacked=. This new mechansim does not operate solely on "local" packs now. The result is that objects residing in an alternate pack which has a .keep file will not be repacked with repack -a. This flaw is only apparent when a commit object is the one residing in an alternate kept pack. 2) The 'repack unpacked objects' and 'loosen unpacked objects' mechanisms of pack-objects, i.e. --keep-unreachable and --unpack-unreachable, now do not operate solely on local packs. The --keep-unreachable option no longer has any callers, but --unpack-unreachable is used when repack is called with '-A -d' and the local repo has existing packs. In this case, objects residing in alternate, not-kept packs will be loosened, and then immediately deleted by repack's call to prune-packed. The test must manually call pack-objects to avoid the call to prune-packed that is made by repack when -d is used. Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-03-20 03:47:50 +00:00
# The pack-objects call on the next line is equivalent to
# git repack -A -d without the call to prune-packed
git pack-objects --honor-pack-keep --non-empty --all --reflog \
--unpack-unreachable </dev/null pack &&
rm -f .git/objects/pack/* &&
mv pack-* .git/objects/pack/ &&
t7700: stop losing return codes of git commands In a pipe, only the return code of the last command is used. Thus, all other commands will have their return codes masked. Rewrite pipes so that there are no git commands upstream so that we will know if a command fails. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:30 +00:00
git verify-pack -v -- .git/objects/pack/*.idx >packlist &&
! grep "^$coid " packlist &&
t7700: remove spaces after redirect operators For shell scripts, the usual convention is for there to be no space after redirection operators, (e.g. `>file`, not `> file`). Remove these spaces wherever they appear. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-27 19:53:47 +00:00
echo >.git/objects/info/alternates &&
t7700: make references to SHA-1 generic Make the test more hash-agnostic by renaming variables from "sha1" to some variation of "oid" or "packid". Also, replace the regex, `[0-9a-f]\{40\}` with `$OID_REGEX`. A better name for "incrpackid" (incremental pack-id) might have been just "packid". However, later in the test suite, we have other uses of "packid". Although the scopes of these variables don't conflict, a future developer may think that commit_and_pack() and test_has_duplicate_object() are semantically related somehow since they share the same variable name. Give them distinct names so that it's clear these uses are unrelated. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:24 +00:00
test_must_fail git show $coid
t7700-repack: add two new tests demonstrating repacking flaws 1) The new --kept-pack-only mechansim of rev-list/pack-objects has replaced --unpacked=. This new mechansim does not operate solely on "local" packs now. The result is that objects residing in an alternate pack which has a .keep file will not be repacked with repack -a. This flaw is only apparent when a commit object is the one residing in an alternate kept pack. 2) The 'repack unpacked objects' and 'loosen unpacked objects' mechanisms of pack-objects, i.e. --keep-unreachable and --unpack-unreachable, now do not operate solely on local packs. The --keep-unreachable option no longer has any callers, but --unpack-unreachable is used when repack is called with '-A -d' and the local repo has existing packs. In this case, objects residing in alternate, not-kept packs will be loosened, and then immediately deleted by repack's call to prune-packed. The test must manually call pack-objects to avoid the call to prune-packed that is made by repack when -d is used. Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-03-20 03:47:50 +00:00
'
pack-objects: don't loosen objects available in alternate or kept packs If pack-objects is called with the --unpack-unreachable option then it will unpack (i.e. loosen) all unreferenced objects from local not-kept packs, including those that also exist in packs residing in an alternate object database or a locally kept pack. The only user of this option is git-repack. In this case, repack will follow the call to pack-objects with a call to prune-packed, which will delete these newly loosened objects, making the act of loosening a waste of time. The unnecessary loosening can be avoided by checking whether an object exists in a non-local pack or a locally kept pack before loosening it. This fixes the 'local packed unreachable obs that exist in alternate ODB are not loosened' test in t7700. Signed-off-by: Brandon Casey <drafnel@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-03-21 22:26:11 +00:00
test_expect_success 'local packed unreachable obs that exist in alternate ODB are not loosened' '
t7700: remove spaces after redirect operators For shell scripts, the usual convention is for there to be no space after redirection operators, (e.g. `>file`, not `> file`). Remove these spaces wherever they appear. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-27 19:53:47 +00:00
echo $(pwd)/alt_objects >.git/objects/info/alternates &&
t7700: make references to SHA-1 generic Make the test more hash-agnostic by renaming variables from "sha1" to some variation of "oid" or "packid". Also, replace the regex, `[0-9a-f]\{40\}` with `$OID_REGEX`. A better name for "incrpackid" (incremental pack-id) might have been just "packid". However, later in the test suite, we have other uses of "packid". Although the scopes of these variables don't conflict, a future developer may think that commit_and_pack() and test_has_duplicate_object() are semantically related somehow since they share the same variable name. Give them distinct names so that it's clear these uses are unrelated. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:24 +00:00
echo "$coid" | git pack-objects --non-empty --all --reflog pack &&
t7700: demonstrate repack flaw which may loosen objects unnecessarily If an unreferenced object exists in both a local pack and in either a pack residing in an alternate object database or a local kept pack, then the pack-objects call made by repack will loosen that object only to have it immediately pruned by repack's call to prune-packed. Signed-off-by: Brandon Casey <drafnel@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-03-21 22:25:30 +00:00
rm -f .git/objects/pack/* &&
mv pack-* .git/objects/pack/ &&
# The pack-objects call on the next line is equivalent to
# git repack -A -d without the call to prune-packed
git pack-objects --honor-pack-keep --non-empty --all --reflog \
--unpack-unreachable </dev/null pack &&
rm -f .git/objects/pack/* &&
mv pack-* .git/objects/pack/ &&
t7700: stop losing return codes of git commands In a pipe, only the return code of the last command is used. Thus, all other commands will have their return codes masked. Rewrite pipes so that there are no git commands upstream so that we will know if a command fails. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:30 +00:00
git verify-pack -v -- .git/objects/pack/*.idx >packlist &&
! grep "^$coid " &&
t7700: remove spaces after redirect operators For shell scripts, the usual convention is for there to be no space after redirection operators, (e.g. `>file`, not `> file`). Remove these spaces wherever they appear. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-27 19:53:47 +00:00
echo >.git/objects/info/alternates &&
t7700: make references to SHA-1 generic Make the test more hash-agnostic by renaming variables from "sha1" to some variation of "oid" or "packid". Also, replace the regex, `[0-9a-f]\{40\}` with `$OID_REGEX`. A better name for "incrpackid" (incremental pack-id) might have been just "packid". However, later in the test suite, we have other uses of "packid". Although the scopes of these variables don't conflict, a future developer may think that commit_and_pack() and test_has_duplicate_object() are semantically related somehow since they share the same variable name. Give them distinct names so that it's clear these uses are unrelated. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-12-04 22:03:24 +00:00
test_must_fail git show $coid
t7700: demonstrate repack flaw which may loosen objects unnecessarily If an unreferenced object exists in both a local pack and in either a pack residing in an alternate object database or a local kept pack, then the pack-objects call made by repack will loosen that object only to have it immediately pruned by repack's call to prune-packed. Signed-off-by: Brandon Casey <drafnel@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-03-21 22:25:30 +00:00
'
git repack: keep commits hidden by a graft When you have grafts that pretend that a given commit has different parents than the ones recorded in the commit object, it is dangerous to let 'git repack' remove those hidden parents, as you can easily remove the graft and end up with a broken repository. So let's play it safe and keep those parent objects and everything that is reachable by them, in addition to the grafted parents. As this behavior can only be triggered by git pack-objects, and as that command handles duplicate parents gracefully, we do not bother to cull duplicated parents that may result by using both true and grafted parents. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-07-23 15:33:49 +00:00
test_expect_success 'objects made unreachable by grafts only are kept' '
Add a test showing that 'git repack' throws away grafted-away parents Signed-off-by: Björn Steinbrink <B.Steinbrink@gmx.de> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-07-23 15:33:45 +00:00
test_tick &&
git commit --allow-empty -m "commit 4" &&
H0=$(git rev-parse HEAD) &&
H1=$(git rev-parse HEAD^) &&
H2=$(git rev-parse HEAD^^) &&
t7700: remove spaces after redirect operators For shell scripts, the usual convention is for there to be no space after redirection operators, (e.g. `>file`, not `> file`). Remove these spaces wherever they appear. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-27 19:53:47 +00:00
echo "$H0 $H2" >.git/info/grafts &&
more war on "sleep" in tests Two more tests that sleep only to waste tick can be converted to use test_tick and take expiry parameters relative to $test_tick. The basic idea is to replace "sleep 1" with "test_tick" to cause the "time" to pass. These tests are interested in expiring things with "now" as the timestamp, soo use a timestamp relative to $test_tick to give them more stability and reproducibility. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-04-14 22:09:57 +00:00
git reflog expire --expire=$test_tick --expire-unreachable=$test_tick --all &&
Add a test showing that 'git repack' throws away grafted-away parents Signed-off-by: Björn Steinbrink <B.Steinbrink@gmx.de> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-07-23 15:33:45 +00:00
git repack -a -d &&
git cat-file -t $H1
t7700: have closing quote of a test at the beginning of line The closing quote of a test body by convention is always at the start of line. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-04-15 15:36:12 +00:00
'
Add a test showing that 'git repack' throws away grafted-away parents Signed-off-by: Björn Steinbrink <B.Steinbrink@gmx.de> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-07-23 15:33:45 +00:00
repack: add --keep-pack option We allow to keep existing packs by having companion .keep files. This is helpful when a pack is permanently kept. In the next patch, git-gc just wants to keep a pack temporarily, for one pack-objects run. git-gc can use --keep-pack for this use case. A note about why the pack_keep field cannot be reused and pack_keep_in_core has to be added. This is about the case when --keep-pack is specified together with either --keep-unreachable or --unpack-unreachable, but --honor-pack-keep is NOT specified. In this case, we want to exclude objects from the packs specified on command line, not from ones with .keep files. If only one bit flag is used, we have to clear pack_keep on pack files with the .keep file. But we can't make any assumption about unreachable objects in .keep packs. If "pack_keep" field is false for .keep packs, we could potentially pull lots of unreachable objects into the new pack, or unpack them loose. The safer approach is ignore all packs with either .keep file or --keep-pack. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-04-15 15:36:13 +00:00
test_expect_success 'repack --keep-pack' '
test_create_repo keep-pack &&
(
cd keep-pack &&
P1=$(commit_and_pack 1) &&
P2=$(commit_and_pack 2) &&
P3=$(commit_and_pack 3) &&
P4=$(commit_and_pack 4) &&
ls .git/objects/pack/*.pack >old-counts &&
test_line_count = 4 old-counts &&
git repack -a -d --keep-pack $P1 --keep-pack $P4 &&
ls .git/objects/pack/*.pack >new-counts &&
grep -q $P1 new-counts &&
grep -q $P4 new-counts &&
test_line_count = 3 new-counts &&
git fsck
)
'
repack: enable bitmaps by default on bare repos A typical use case for bare repos is for serving clones and fetches to clients. Enable bitmaps by default on bare repos to make it easier for admins to host git repos in a performant way. Signed-off-by: Eric Wong <e@80x24.org> Helped-by: Jeff King <peff@peff.net> Acked-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-03-14 09:12:54 +00:00
test_expect_success 'bitmaps are created by default in bare repos' '
git clone --bare .git bare.git &&
t7700: update to work with MIDX bitmap test knob A number of these tests are focused only on pack-based bitmaps and need to be updated to disable 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' where necessary. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-08-31 20:52:41 +00:00
rm -f bare.git/objects/pack/*.bitmap &&
GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
git -C bare.git repack -ad &&
repack: enable bitmaps by default on bare repos A typical use case for bare repos is for serving clones and fetches to clients. Enable bitmaps by default on bare repos to make it easier for admins to host git repos in a performant way. Signed-off-by: Eric Wong <e@80x24.org> Helped-by: Jeff King <peff@peff.net> Acked-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-03-14 09:12:54 +00:00
bitmap=$(ls bare.git/objects/pack/*.bitmap) &&
test_path_is_file "$bitmap"
'
test_expect_success 'incremental repack does not complain' '
git -C bare.git repack -q 2>repack.err &&
test_must_be_empty repack.err
'
t7700: demonstrate mishandling of objects in packs with a .keep file Objects residing in pack files that have an associated .keep file are not supposed to be repacked into new pack files, but they are. Signed-off-by: Brandon Casey <casey@nrlssc.navy.mil> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-11-12 17:59:02 +00:00
repack: enable bitmaps by default on bare repos A typical use case for bare repos is for serving clones and fetches to clients. Enable bitmaps by default on bare repos to make it easier for admins to host git repos in a performant way. Signed-off-by: Eric Wong <e@80x24.org> Helped-by: Jeff King <peff@peff.net> Acked-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-03-14 09:12:54 +00:00
test_expect_success 'bitmaps can be disabled on bare repos' '
t7700: update to work with MIDX bitmap test knob A number of these tests are focused only on pack-based bitmaps and need to be updated to disable 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' where necessary. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-08-31 20:52:41 +00:00
GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
git -c repack.writeBitmaps=false -C bare.git repack -ad &&
t7700: drop redirections to /dev/null Since output is silenced when running without `-v` and debugging output is useful with `-v`, remove redirections to /dev/null as it is not useful. In one case where the output of stdout is consumed, redirect the output of test_commit to stderr. Signed-off-by: Denton Liu <liu.denton@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-11-27 19:53:45 +00:00
bitmap=$(ls bare.git/objects/pack/*.bitmap || :) &&
repack: enable bitmaps by default on bare repos A typical use case for bare repos is for serving clones and fetches to clients. Enable bitmaps by default on bare repos to make it easier for admins to host git repos in a performant way. Signed-off-by: Eric Wong <e@80x24.org> Helped-by: Jeff King <peff@peff.net> Acked-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-03-14 09:12:54 +00:00
test -z "$bitmap"
'
repack: disable bitmaps-by-default if .keep files exist Bitmaps aren't useful with multiple packs, and users with .keep files ended up with redundant packs when bitmaps got enabled by default in bare repos. So detect when .keep files exist and stop enabling bitmaps by default in that case. Wasteful (but otherwise harmless) race conditions with .keep files documented by Jeff King still apply and there's a chance we'd still end up with redundant data on the FS: https://public-inbox.org/git/20190623224244.GB1100@sigill.intra.peff.net/ v2: avoid subshell in test case, be multi-index aware Fixes: 36eba0323d3288a8 ("repack: enable bitmaps by default on bare repos") Signed-off-by: Eric Wong <e@80x24.org> Helped-by: Jeff King <peff@peff.net> Reported-by: Janos Farkas <chexum@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-06-29 19:13:59 +00:00
test_expect_success 'no bitmaps created if .keep files present' '
pack=$(ls bare.git/objects/pack/*.pack) &&
test_path_is_file "$pack" &&
keep=${pack%.pack}.keep &&
t7700: clean up .keep file in bitmap-writing test After our test snippet finishes, the .keep file is left in place, making it hard to do further tests of the auto-bitmap-writing code (since it suppresses the feature completely). Let's clean it up. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-07-31 05:37:36 +00:00
test_when_finished "rm -f \"\$keep\"" &&
repack: disable bitmaps-by-default if .keep files exist Bitmaps aren't useful with multiple packs, and users with .keep files ended up with redundant packs when bitmaps got enabled by default in bare repos. So detect when .keep files exist and stop enabling bitmaps by default in that case. Wasteful (but otherwise harmless) race conditions with .keep files documented by Jeff King still apply and there's a chance we'd still end up with redundant data on the FS: https://public-inbox.org/git/20190623224244.GB1100@sigill.intra.peff.net/ v2: avoid subshell in test case, be multi-index aware Fixes: 36eba0323d3288a8 ("repack: enable bitmaps by default on bare repos") Signed-off-by: Eric Wong <e@80x24.org> Helped-by: Jeff King <peff@peff.net> Reported-by: Janos Farkas <chexum@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-06-29 19:13:59 +00:00
>"$keep" &&
t7700: update to work with MIDX bitmap test knob A number of these tests are focused only on pack-based bitmaps and need to be updated to disable 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' where necessary. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-08-31 20:52:41 +00:00
GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
git -C bare.git repack -ad 2>stderr &&
repack: simplify handling of auto-bitmaps and .keep files Commit 7328482253 (repack: disable bitmaps-by-default if .keep files exist, 2019-06-29) taught repack to prefer disabling bitmaps to duplicating objects (unless bitmaps were asked for explicitly). But there's an easier way to do this: if we keep passing the --honor-pack-keep flag to pack-objects when auto-enabling bitmaps, then pack-objects already makes the same decision (it will disable bitmaps rather than duplicate). Better still, pack-objects can actually decide to do so based not just on the presence of a .keep file, but on whether that .keep file actually impacts the new pack we're making (so if we're racing with a push or fetch, for example, their temporary .keep file will not block us from generating bitmaps if they haven't yet updated their refs). And because repack uses the --write-bitmap-index-quiet flag, we don't have to worry about pack-objects generating confusing warnings when it does see a .keep file. We can confirm this by tweaking the .keep test to check repack's stderr. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-07-31 05:40:56 +00:00
test_must_be_empty stderr &&
repack: disable bitmaps-by-default if .keep files exist Bitmaps aren't useful with multiple packs, and users with .keep files ended up with redundant packs when bitmaps got enabled by default in bare repos. So detect when .keep files exist and stop enabling bitmaps by default in that case. Wasteful (but otherwise harmless) race conditions with .keep files documented by Jeff King still apply and there's a chance we'd still end up with redundant data on the FS: https://public-inbox.org/git/20190623224244.GB1100@sigill.intra.peff.net/ v2: avoid subshell in test case, be multi-index aware Fixes: 36eba0323d3288a8 ("repack: enable bitmaps by default on bare repos") Signed-off-by: Eric Wong <e@80x24.org> Helped-by: Jeff King <peff@peff.net> Reported-by: Janos Farkas <chexum@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-06-29 19:13:59 +00:00
find bare.git/objects/pack/ -type f -name "*.bitmap" >actual &&
test_must_be_empty actual
'
repack: silence warnings when auto-enabled bitmaps cannot be built Depending on various config options, a full repack may not be able to build a reachability bitmap index (e.g., if pack.packSizeLimit forces us to write multiple packs). In these cases pack-objects may write a warning to stderr. Since 36eba0323d (repack: enable bitmaps by default on bare repos, 2019-03-14), we may generate these warnings even when the user did not explicitly ask for bitmaps. This has two downsides: - it can be confusing, if they don't know what bitmaps are - a daemonized auto-gc will write this to its log file, and the presence of the warning may suppress further auto-gc (until gc.logExpiry has elapsed) Let's have repack communicate to pack-objects that the choice to turn on bitmaps was not made explicitly by the user, which in turn allows pack-objects to suppress these warnings. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-07-31 05:39:27 +00:00
test_expect_success 'auto-bitmaps do not complain if unavailable' '
test_config -C bare.git pack.packSizeLimit 1M &&
blob=$(test-tool genrandom big $((1024*1024)) |
git -C bare.git hash-object -w --stdin) &&
git -C bare.git update-ref refs/tags/big $blob &&
t7700: update to work with MIDX bitmap test knob A number of these tests are focused only on pack-based bitmaps and need to be updated to disable 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' where necessary. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-08-31 20:52:41 +00:00
GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
git -C bare.git repack -ad 2>stderr &&
repack: silence warnings when auto-enabled bitmaps cannot be built Depending on various config options, a full repack may not be able to build a reachability bitmap index (e.g., if pack.packSizeLimit forces us to write multiple packs). In these cases pack-objects may write a warning to stderr. Since 36eba0323d (repack: enable bitmaps by default on bare repos, 2019-03-14), we may generate these warnings even when the user did not explicitly ask for bitmaps. This has two downsides: - it can be confusing, if they don't know what bitmaps are - a daemonized auto-gc will write this to its log file, and the presence of the warning may suppress further auto-gc (until gc.logExpiry has elapsed) Let's have repack communicate to pack-objects that the choice to turn on bitmaps was not made explicitly by the user, which in turn allows pack-objects to suppress these warnings. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-07-31 05:39:27 +00:00
test_must_be_empty stderr &&
find bare.git/objects/pack -type f -name "*.bitmap" >actual &&
test_must_be_empty actual
'
builtin/repack.c: support writing a MIDX while repacking Teach `git repack` a new `--write-midx` option for callers that wish to persist a multi-pack index in their repository while repacking. There are two existing alternatives to this new flag, but they don't cover our particular use-case. These alternatives are: - Call 'git multi-pack-index write' after running 'git repack', or - Set 'GIT_TEST_MULTI_PACK_INDEX=1' in your environment when running 'git repack'. The former works, but introduces a gap in bitmap coverage between repacking and writing a new MIDX (since the repack may have deleted a pack included in the existing MIDX, invalidating it altogether). Setting the 'GIT_TEST_' environment variable is obviously unsupported. In fact, even if it were supported officially, it still wouldn't work, because it generates the MIDX *after* redundant packs have been dropped, leading to the same issue as above. Introduce a new option which eliminates this race by teaching `git repack` to generate the MIDX at the critical point: after the new packs have been written and moved into place, but before the redundant packs have been removed. This option is compatible with `git repack`'s '--bitmap' option (it changes the interpretation to be: "write a bitmap corresponding to the MIDX after one has been generated"). There is a little bit of additional noise in the patch below to avoid repeating ourselves when selecting which packs to delete. Instead of a single loop as before (where we iterate over 'existing_packs', decide if a pack is worth deleting, and if so, delete it), we have two loops (the first where we decide which ones are worth deleting, and the second where we actually do the deleting). This makes it so we have a single check we can make consistently when (1) telling the MIDX which packs we want to exclude, and (2) actually unlinking the redundant packs. There is also a tiny change to short-circuit the body of write_midx_included_packs() when no packs remain in the case of an empty repository. The MIDX code does not handle this, so avoid trying to generate a MIDX covering zero packs in the first place. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-09-29 01:55:18 +00:00
objdir=.git/objects
midx=$objdir/pack/multi-pack-index
test_expect_success 'setup for --write-midx tests' '
git init midx &&
(
cd midx &&
git config core.multiPackIndex true &&
test_commit base
)
'
test_expect_success '--write-midx unchanged' '
(
cd midx &&
GIT_TEST_MULTI_PACK_INDEX=0 git repack &&
test_path_is_missing $midx &&
test_path_is_missing $midx-*.bitmap &&
GIT_TEST_MULTI_PACK_INDEX=0 git repack --write-midx &&
test_path_is_file $midx &&
test_path_is_missing $midx-*.bitmap &&
test_midx_consistent $objdir
)
'
test_expect_success '--write-midx with a new pack' '
(
cd midx &&
test_commit loose &&
GIT_TEST_MULTI_PACK_INDEX=0 git repack --write-midx &&
test_path_is_file $midx &&
test_path_is_missing $midx-*.bitmap &&
test_midx_consistent $objdir
)
'
test_expect_success '--write-midx with -b' '
(
cd midx &&
GIT_TEST_MULTI_PACK_INDEX=0 git repack -mb &&
test_path_is_file $midx &&
test_path_is_file $midx-*.bitmap &&
test_midx_consistent $objdir
)
'
test_expect_success '--write-midx with -d' '
(
cd midx &&
test_commit repack &&
GIT_TEST_MULTI_PACK_INDEX=0 git repack -Ad --write-midx &&
test_path_is_file $midx &&
test_path_is_missing $midx-*.bitmap &&
test_midx_consistent $objdir
)
'
test_expect_success 'cleans up MIDX when appropriate' '
(
cd midx &&
test_commit repack-2 &&
GIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb --write-midx &&
checksum=$(midx_checksum $objdir) &&
test_path_is_file $midx &&
test_path_is_file $midx-$checksum.bitmap &&
test_path_is_file $midx-$checksum.rev &&
test_commit repack-3 &&
GIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb --write-midx &&
test_path_is_file $midx &&
test_path_is_missing $midx-$checksum.bitmap &&
test_path_is_missing $midx-$checksum.rev &&
test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
test_path_is_file $midx-$(midx_checksum $objdir).rev &&
test_commit repack-4 &&
GIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb &&
find $objdir/pack -type f -name "multi-pack-index*" >files &&
test_must_be_empty files
)
'
builtin/repack.c: pass `--refs-snapshot` when writing bitmaps To prevent the race described in an earlier patch, generate and pass a reference snapshot to the multi-pack bitmap code, if we are writing one from `git repack`. This patch is mostly limited to creating a temporary file, and then calling for_each_ref(). Except we try to minimize duplicates, since doing so can drastically reduce the size in network-of-forks style repositories. In the kernel's fork network (the repository containing all objects from the kernel and all its forks), deduplicating the references drops the snapshot size from 934 MB to just 12 MB. But since we're handling duplicates in this way, we have to make sure that we preferred references (those listed in pack.preferBitmapTips) before non-preferred ones (to avoid recording an object which is pointed at by a preferred tip as non-preferred). We accomplish this by doing separate passes over the references: first visiting each prefix in pack.preferBitmapTips, and then over the rest of the references. Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-01 22:38:10 +00:00
test_expect_success '--write-midx with preferred bitmap tips' '
git init midx-preferred-tips &&
test_when_finished "rm -fr midx-preferred-tips" &&
(
cd midx-preferred-tips &&
test_commit_bulk --message="%s" 103 &&
git log --format="%H" >commits.raw &&
sort <commits.raw >commits &&
git log --format="create refs/tags/%s/%s %H" HEAD >refs &&
git update-ref --stdin <refs &&
git repack --write-midx --write-bitmap-index &&
test_path_is_file $midx &&
test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
test-tool bitmap list-commits | sort >bitmaps &&
comm -13 bitmaps commits >before &&
test_line_count = 1 before &&
rm -fr $midx-$(midx_checksum $objdir).bitmap &&
rm -fr $midx-$(midx_checksum $objdir).rev &&
rm -fr $midx &&
# instead of constructing the snapshot ourselves (c.f., the test
# "write a bitmap with --refs-snapshot (preferred tips)" in
# t5326), mark the missing commit as preferred by adding it to
# the pack.preferBitmapTips configuration.
git for-each-ref --format="%(refname:rstrip=1)" \
--points-at="$(cat before)" >missing &&
git config pack.preferBitmapTips "$(cat missing)" &&
git repack --write-midx --write-bitmap-index &&
test-tool bitmap list-commits | sort >bitmaps &&
comm -13 bitmaps commits >after &&
! test_cmp before after
)
'
repack: respect kept objects with '--write-midx -b' Historically, we needed a single packfile in order to have reachability bitmaps. This introduced logic that when 'git repack' had a '-b' option that we should stop sending the '--honor-pack-keep' option to the 'git pack-objects' child process, ensuring that we create a packfile containing all reachable objects. In the world of multi-pack-index bitmaps, we no longer need to repack all objects into a single pack to have valid bitmaps. Thus, we should continue sending the '--honor-pack-keep' flag to 'git pack-objects'. The fix is very simple: only disable the flag when writing bitmaps but also _not_ writing the multi-pack-index. This opens the door to new repacking strategies that might want to keep some historical set of objects in a stable pack-file while only repacking more recent objects. To test, create a new 'test_subcommand_inexact' helper that is more flexible than 'test_subcommand'. This allows us to look for the --honor-pack-keep flag without over-indexing on the exact set of arguments. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-12-20 14:48:10 +00:00
test_expect_success '--write-midx -b packs non-kept objects' '
GIT_TRACE2_EVENT="$(pwd)/trace.txt" \
git repack --write-midx -a -b &&
test_subcommand_inexact git pack-objects --honor-pack-keep <trace.txt
'
repack: make '--quiet' disable progress While testing some ideas in 'git repack', I ran it with '--quiet' and discovered that some progress output was still shown. Specifically, the output for writing the multi-pack-index showed the progress. The 'show_progress' variable in cmd_repack() is initialized with isatty(2) and is not modified at all by the '--quiet' flag. The '--quiet' flag modifies the po_args.quiet option which is translated into a '--quiet' flag for the 'git pack-objects' child process. However, 'show_progress' is used to directly send progress information to the multi-pack-index writing logic which does not use a child process. The fix here is to modify 'show_progress' to be false if po_opts.quiet is true, and isatty(2) otherwise. This new expectation simplifies a later condition that checks both. Update the documentation to make it clear that '-q' will disable all progress in addition to ensuring the 'git pack-objects' child process will receive the flag. Use 'test_terminal' to check that this works to get around the isatty(2) check. Helped-by: Jeff King <peff@peff.net> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-12-20 14:48:11 +00:00
test_expect_success TTY '--quiet disables progress' '
test_terminal env GIT_PROGRESS_DELAY=0 \
git -C midx repack -ad --quiet --write-midx 2>stderr &&
test_must_be_empty stderr
'
repack: refactor to avoid double-negation of update-server-info By default, git-repack(1) runs `update_server_info()` to generate info required for the dumb HTTP protocol. This can be disabled via the `-n` flag, which then sets the `no_update_server_info` flag. Further down the code this leads to some double-negation logic, which is about to become more confusing as we're about to add a new config which allows the user to permanently disable generation of the info. Refactor the code to avoid the double-negation and add some tests which verify that the flag continues to work as expected. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-03-14 07:42:46 +00:00
test_expect_success 'setup for update-server-info' '
git init update-server-info &&
test_commit -C update-server-info message
'
test_server_info_present () {
test_path_is_file update-server-info/.git/objects/info/packs &&
test_path_is_file update-server-info/.git/info/refs
}
test_server_info_missing () {
test_path_is_missing update-server-info/.git/objects/info/packs &&
test_path_is_missing update-server-info/.git/info/refs
}
test_server_info_cleanup () {
rm -f update-server-info/.git/objects/info/packs update-server-info/.git/info/refs &&
test_server_info_missing
}
test_expect_success 'updates server info by default' '
test_server_info_cleanup &&
git -C update-server-info repack &&
test_server_info_present
'
test_expect_success '-n skips updating server info' '
test_server_info_cleanup &&
git -C update-server-info repack -n &&
test_server_info_missing
'
repack: enable bitmaps by default on bare repos A typical use case for bare repos is for serving clones and fetches to clients. Enable bitmaps by default on bare repos to make it easier for admins to host git repos in a performant way. Signed-off-by: Eric Wong <e@80x24.org> Helped-by: Jeff King <peff@peff.net> Acked-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-03-14 09:12:54 +00:00
test_done
Reference in New Issue Copy Permalink