git/t/t7703-repack-geometric.sh
Taylor Blau 197443e80a repack: don't remove .keep packs with --pack-kept-objects
`git repack` supports a `--pack-kept-objects` flag which more or less
translates to whether or not we pass `--honor-pack-keep` down to `git
pack-objects` when assembling a new pack.

This behavior has existed since ee34a2bead (repack: add
`repack.packKeptObjects` config var, 2014-03-03). In that commit, the
documentation was extended to say:

    [...] Note that we still do not delete `.keep` packs after
    `pack-objects` finishes.

Unfortunately, this is not the case when `--pack-kept-objects` is
combined with a `--geometric` repack. When doing a geometric repack, we
include `.keep` packs when enumerating available packs only when
`pack_kept_objects` is set.

So this all works fine when `--no-pack-kept-objects` (or similar) is
given. Kept packs are excluded from the geometric roll-up, so when we go
to delete redundant packs (with `-d`), no `.keep` packs appear "below
the split" in our geometric progression.

But when `--pack-kept-objects` is given, things can go awry. Namely,
when a kept pack is included in the list of packs tracked by the
`pack_geometry` struct *and* part of the pack roll-up, we will delete
the `.keep` pack when we shouldn't.

Note that this *doesn't* result in object corruption, since the `.keep`
pack's objects are still present in the new pack. But the `.keep` pack
itself is removed, which violates our promise from back in ee34a2bead.

But there's more. Because `repack` computes the geometric roll-up
independently from selecting which packs belong in a MIDX (with
`--write-midx`), this can lead to odd behavior. Consider when a `.keep`
pack appears below the geometric split (ie., its objects will be part of
the new pack we generate).

We'll write a MIDX containing the new pack along with the existing
`.keep` pack. But because the `.keep` pack appears below the geometric
split line, we'll (incorrectly) try to remove it. While this doesn't
corrupt the repository, it does cause us to remove the MIDX we just
wrote, since removing that pack would invalidate the new MIDX.

Funny enough, this behavior became far less noticeable after e4d0c11c04
(repack: respect kept objects with '--write-midx -b', 2021-12-20), which
made `pack_kept_objects` be enabled by default only when we were writing
a non-MIDX bitmap.

But e4d0c11c04 didn't resolve this bug, it just made it harder to notice
unless callers explicitly passed `--pack-kept-objects`.

The solution is to avoid trying to remove `.keep` packs during
`--geometric` repacks, even when they appear below the geometric split
line, which is the approach this patch implements.

Co-authored-by: Victoria Dye <vdye@github.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-10-17 21:29:23 -07:00

284 lines
7.6 KiB
Bash
Executable file

#!/bin/sh
test_description='git repack --geometric works correctly'
. ./test-lib.sh
GIT_TEST_MULTI_PACK_INDEX=0
objdir=.git/objects
packdir=$objdir/pack
midx=$objdir/pack/multi-pack-index
test_expect_success '--geometric with no packs' '
git init geometric &&
test_when_finished "rm -fr geometric" &&
(
cd geometric &&
git repack --write-midx --geometric 2 >out &&
test_i18ngrep "Nothing new to pack" out
)
'
test_expect_success '--geometric with one pack' '
git init geometric &&
test_when_finished "rm -fr geometric" &&
(
cd geometric &&
test_commit "base" &&
git repack -d &&
git repack --geometric 2 >out &&
test_i18ngrep "Nothing new to pack" out
)
'
test_expect_success '--geometric with an intact progression' '
git init geometric &&
test_when_finished "rm -fr geometric" &&
(
cd geometric &&
# These packs already form a geometric progression.
test_commit_bulk --start=1 1 && # 3 objects
test_commit_bulk --start=2 2 && # 6 objects
test_commit_bulk --start=4 4 && # 12 objects
find $objdir/pack -name "*.pack" | sort >expect &&
git repack --geometric 2 -d &&
find $objdir/pack -name "*.pack" | sort >actual &&
test_cmp expect actual
)
'
test_expect_success '--geometric with loose objects' '
git init geometric &&
test_when_finished "rm -fr geometric" &&
(
cd geometric &&
# These packs already form a geometric progression.
test_commit_bulk --start=1 1 && # 3 objects
test_commit_bulk --start=2 2 && # 6 objects
# The loose objects are packed together, breaking the
# progression.
test_commit loose && # 3 objects
find $objdir/pack -name "*.pack" | sort >before &&
git repack --geometric 2 -d &&
find $objdir/pack -name "*.pack" | sort >after &&
comm -13 before after >new &&
comm -23 before after >removed &&
test_line_count = 1 new &&
test_must_be_empty removed &&
git repack --geometric 2 -d &&
find $objdir/pack -name "*.pack" | sort >after &&
# The progression (3, 3, 6) is combined into one new pack.
test_line_count = 1 after
)
'
test_expect_success '--geometric with small-pack rollup' '
git init geometric &&
test_when_finished "rm -fr geometric" &&
(
cd geometric &&
test_commit_bulk --start=1 1 && # 3 objects
test_commit_bulk --start=2 1 && # 3 objects
find $objdir/pack -name "*.pack" | sort >small &&
test_commit_bulk --start=3 4 && # 12 objects
test_commit_bulk --start=7 8 && # 24 objects
find $objdir/pack -name "*.pack" | sort >before &&
git repack --geometric 2 -d &&
# Three packs in total; two of the existing large ones, and one
# new one.
find $objdir/pack -name "*.pack" | sort >after &&
test_line_count = 3 after &&
comm -3 small before | tr -d "\t" >large &&
grep -qFf large after
)
'
test_expect_success '--geometric with small- and large-pack rollup' '
git init geometric &&
test_when_finished "rm -fr geometric" &&
(
cd geometric &&
# size(small1) + size(small2) > size(medium) / 2
test_commit_bulk --start=1 1 && # 3 objects
test_commit_bulk --start=2 1 && # 3 objects
test_commit_bulk --start=2 3 && # 7 objects
test_commit_bulk --start=6 9 && # 27 objects &&
find $objdir/pack -name "*.pack" | sort >before &&
git repack --geometric 2 -d &&
find $objdir/pack -name "*.pack" | sort >after &&
comm -12 before after >untouched &&
# Two packs in total; the largest pack from before running "git
# repack", and one new one.
test_line_count = 1 untouched &&
test_line_count = 2 after
)
'
test_expect_success '--geometric ignores kept packs' '
git init geometric &&
test_when_finished "rm -fr geometric" &&
(
cd geometric &&
test_commit kept && # 3 objects
test_commit pack && # 3 objects
KEPT=$(git pack-objects --revs $objdir/pack/pack <<-EOF
refs/tags/kept
EOF
) &&
PACK=$(git pack-objects --revs $objdir/pack/pack <<-EOF
refs/tags/pack
^refs/tags/kept
EOF
) &&
# neither pack contains more than twice the number of objects in
# the other, so they should be combined. but, marking one as
# .kept on disk will "freeze" it, so the pack structure should
# remain unchanged.
touch $objdir/pack/pack-$KEPT.keep &&
find $objdir/pack -name "*.pack" | sort >before &&
git repack --geometric 2 -d &&
find $objdir/pack -name "*.pack" | sort >after &&
# both packs should still exist
test_path_is_file $objdir/pack/pack-$KEPT.pack &&
test_path_is_file $objdir/pack/pack-$PACK.pack &&
# and no new packs should be created
test_cmp before after &&
# Passing --pack-kept-objects causes packs with a .keep file to
# be repacked, too.
git repack --geometric 2 -d --pack-kept-objects &&
# After repacking, two packs remain: one new one (containing the
# objects in both the .keep and non-kept pack), and the .keep
# pack (since `--pack-kept-objects -d` does not actually delete
# the kept pack).
find $objdir/pack -name "*.pack" >after &&
test_line_count = 2 after
)
'
test_expect_success '--geometric ignores --keep-pack packs' '
git init geometric &&
test_when_finished "rm -fr geometric" &&
(
cd geometric &&
# Create two equal-sized packs
test_commit kept && # 3 objects
git repack -d &&
test_commit pack && # 3 objects
git repack -d &&
find $objdir/pack -type f -name "*.pack" | sort >packs.before &&
git repack --geometric 2 -dm \
--keep-pack="$(basename "$(head -n 1 packs.before)")" >out &&
find $objdir/pack -type f -name "*.pack" | sort >packs.after &&
# Packs should not have changed (only one non-kept pack, no
# loose objects), but $midx should now exist.
grep "Nothing new to pack" out &&
test_path_is_file $midx &&
test_cmp packs.before packs.after &&
git fsck
)
'
test_expect_success '--geometric chooses largest MIDX preferred pack' '
git init geometric &&
test_when_finished "rm -fr geometric" &&
(
cd geometric &&
# These packs already form a geometric progression.
test_commit_bulk --start=1 1 && # 3 objects
test_commit_bulk --start=2 2 && # 6 objects
ls $objdir/pack/pack-*.idx >before &&
test_commit_bulk --start=4 4 && # 12 objects
ls $objdir/pack/pack-*.idx >after &&
git repack --geometric 2 -dbm &&
comm -3 before after | xargs -n 1 basename >expect &&
test-tool read-midx --preferred-pack $objdir >actual &&
test_cmp expect actual
)
'
test_expect_success '--geometric with pack.packSizeLimit' '
git init pack-rewrite &&
test_when_finished "rm -fr pack-rewrite" &&
(
cd pack-rewrite &&
test-tool genrandom foo 1048576 >foo &&
test-tool genrandom bar 1048576 >bar &&
git add foo bar &&
test_tick &&
git commit -m base &&
git rev-parse HEAD:foo HEAD:bar >p1.objects &&
git rev-parse HEAD HEAD^{tree} >p2.objects &&
# These two packs each contain two objects, so the following
# `--geometric` repack will try to combine them.
p1="$(git pack-objects $packdir/pack <p1.objects)" &&
p2="$(git pack-objects $packdir/pack <p2.objects)" &&
# Remove any loose objects in packs, since we do not want extra
# copies around (which would mask over potential object
# corruption issues).
git prune-packed &&
# Both p1 and p2 will be rolled up, but pack-objects will write
# three packs:
#
# - one containing object "foo",
# - another containing object "bar",
# - a final pack containing the commit and tree objects
# (identical to p2 above)
git repack --geometric 2 -d --max-pack-size=1048576 &&
# Ensure `repack` can detect that the third pack it wrote
# (containing just the tree and commit objects) was identical to
# one that was below the geometric split, so that we can save it
# from deletion.
#
# If `repack` fails to do that, we will incorrectly delete p2,
# causing object corruption.
git fsck
)
'
test_done