2013-12-05 13:02:40 +00:00
|
|
|
#!/bin/sh
|
|
|
|
|
|
|
|
|
|
test_description='fetch/clone from a shallow clone'
|
|
|
|
|
|
2020-11-18 23:44:33 +00:00
|
|
|
GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
|
tests: mark tests relying on the current default for `init.defaultBranch`
In addition to the manual adjustment to let the `linux-gcc` CI job run
the test suite with `master` and then with `main`, this patch makes sure
that GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME is set in all test scripts
that currently rely on the initial branch name being `master by default.
To determine which test scripts to mark up, the first step was to
force-set the default branch name to `master` in
- all test scripts that contain the keyword `master`,
- t4211, which expects `t/t4211/history.export` with a hard-coded ref to
initialize the default branch,
- t5560 because it sources `t/t556x_common` which uses `master`,
- t8002 and t8012 because both source `t/annotate-tests.sh` which also
uses `master`)
This trick was performed by this command:
$ sed -i '/^ *\. \.\/\(test-lib\|lib-\(bash\|cvs\|git-svn\)\|gitweb-lib\)\.sh$/i\
GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\
export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\
' $(git grep -l master t/t[0-9]*.sh) \
t/t4211*.sh t/t5560*.sh t/t8002*.sh t/t8012*.sh
After that, careful, manual inspection revealed that some of the test
scripts containing the needle `master` do not actually rely on a
specific default branch name: either they mention `master` only in a
comment, or they initialize that branch specificially, or they do not
actually refer to the current default branch. Therefore, the
aforementioned modification was undone in those test scripts thusly:
$ git checkout HEAD -- \
t/t0027-auto-crlf.sh t/t0060-path-utils.sh \
t/t1011-read-tree-sparse-checkout.sh \
t/t1305-config-include.sh t/t1309-early-config.sh \
t/t1402-check-ref-format.sh t/t1450-fsck.sh \
t/t2024-checkout-dwim.sh \
t/t2106-update-index-assume-unchanged.sh \
t/t3040-subprojects-basic.sh t/t3301-notes.sh \
t/t3308-notes-merge.sh t/t3423-rebase-reword.sh \
t/t3436-rebase-more-options.sh \
t/t4015-diff-whitespace.sh t/t4257-am-interactive.sh \
t/t5323-pack-redundant.sh t/t5401-update-hooks.sh \
t/t5511-refspec.sh t/t5526-fetch-submodules.sh \
t/t5529-push-errors.sh t/t5530-upload-pack-error.sh \
t/t5548-push-porcelain.sh \
t/t5552-skipping-fetch-negotiator.sh \
t/t5572-pull-submodule.sh t/t5608-clone-2gb.sh \
t/t5614-clone-submodules-shallow.sh \
t/t7508-status.sh t/t7606-merge-custom.sh \
t/t9302-fast-import-unpack-limit.sh
We excluded one set of test scripts in these commands, though: the range
of `git p4` tests. The reason? `git p4` stores the (foreign) remote
branch in the branch called `p4/master`, which is obviously not the
default branch. Manual analysis revealed that only five of these tests
actually require a specific default branch name to pass; They were
modified thusly:
$ sed -i '/^ *\. \.\/lib-git-p4\.sh$/i\
GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master\
export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\
' t/t980[0167]*.sh t/t9811*.sh
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-11-18 23:44:19 +00:00
|
|
|
export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
|
|
|
|
|
|
2013-12-05 13:02:40 +00:00
|
|
|
. ./test-lib.sh
|
|
|
|
|
|
|
|
|
|
commit() {
|
|
|
|
|
echo "$1" >tracked &&
|
|
|
|
|
git add tracked &&
|
|
|
|
|
git commit -m "$1"
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
test_expect_success 'setup' '
|
|
|
|
|
commit 1 &&
|
|
|
|
|
commit 2 &&
|
|
|
|
|
commit 3 &&
|
|
|
|
|
commit 4 &&
|
2019-12-21 19:49:32 +00:00
|
|
|
git config --global transfer.fsckObjects true &&
|
2020-04-24 17:11:39 +00:00
|
|
|
test_oid_cache <<-\EOF
|
2020-03-04 15:53:12 +00:00
|
|
|
perl sha1:s/0034shallow %s/0036unshallow %s/
|
|
|
|
|
perl sha256:s/004cshallow %s/004eunshallow %s/
|
2019-12-21 19:49:32 +00:00
|
|
|
EOF
|
2013-12-05 13:02:40 +00:00
|
|
|
'
|
|
|
|
|
|
|
|
|
|
test_expect_success 'setup shallow clone' '
|
|
|
|
|
git clone --no-local --depth=2 .git shallow &&
|
|
|
|
|
git --git-dir=shallow/.git log --format=%s >actual &&
|
2020-04-24 17:11:39 +00:00
|
|
|
test_write_lines 4 3 >expect &&
|
2013-12-05 13:02:40 +00:00
|
|
|
test_cmp expect actual
|
|
|
|
|
'
|
|
|
|
|
|
|
|
|
|
test_expect_success 'clone from shallow clone' '
|
|
|
|
|
git clone --no-local shallow shallow2 &&
|
|
|
|
|
(
|
|
|
|
|
cd shallow2 &&
|
|
|
|
|
git fsck &&
|
|
|
|
|
git log --format=%s >actual &&
|
2020-04-24 17:11:39 +00:00
|
|
|
test_write_lines 4 3 >expect &&
|
2013-12-05 13:02:40 +00:00
|
|
|
test_cmp expect actual
|
|
|
|
|
)
|
|
|
|
|
'
|
|
|
|
|
|
|
|
|
|
test_expect_success 'fetch from shallow clone' '
|
|
|
|
|
(
|
|
|
|
|
cd shallow &&
|
|
|
|
|
commit 5
|
|
|
|
|
) &&
|
|
|
|
|
(
|
|
|
|
|
cd shallow2 &&
|
|
|
|
|
git fetch &&
|
|
|
|
|
git fsck &&
|
2020-11-18 23:44:33 +00:00
|
|
|
git log --format=%s origin/main >actual &&
|
2020-04-24 17:11:39 +00:00
|
|
|
test_write_lines 5 4 3 >expect &&
|
2013-12-05 13:02:40 +00:00
|
|
|
test_cmp expect actual
|
|
|
|
|
)
|
|
|
|
|
'
|
|
|
|
|
|
|
|
|
|
test_expect_success 'fetch --depth from shallow clone' '
|
|
|
|
|
(
|
|
|
|
|
cd shallow &&
|
|
|
|
|
commit 6
|
|
|
|
|
) &&
|
|
|
|
|
(
|
|
|
|
|
cd shallow2 &&
|
|
|
|
|
git fetch --depth=2 &&
|
|
|
|
|
git fsck &&
|
2020-11-18 23:44:33 +00:00
|
|
|
git log --format=%s origin/main >actual &&
|
2020-04-24 17:11:39 +00:00
|
|
|
test_write_lines 6 5 >expect &&
|
2013-12-05 13:02:40 +00:00
|
|
|
test_cmp expect actual
|
|
|
|
|
)
|
|
|
|
|
'
|
|
|
|
|
|
2013-12-05 13:02:41 +00:00
|
|
|
test_expect_success 'fetch --unshallow from shallow clone' '
|
|
|
|
|
(
|
|
|
|
|
cd shallow2 &&
|
|
|
|
|
git fetch --unshallow &&
|
|
|
|
|
git fsck &&
|
2020-11-18 23:44:33 +00:00
|
|
|
git log --format=%s origin/main >actual &&
|
2020-04-24 17:11:39 +00:00
|
|
|
test_write_lines 6 5 4 3 >expect &&
|
2013-12-05 13:02:41 +00:00
|
|
|
test_cmp expect actual
|
|
|
|
|
)
|
|
|
|
|
'
|
|
|
|
|
|
commit.c: don't persist substituted parents when unshallowing
Since 37b9dcabfc (shallow.c: use '{commit,rollback}_shallow_file',
2020-04-22), Git knows how to reset stat-validity checks for the
$GIT_DIR/shallow file, allowing it to change between a shallow and
non-shallow state in the same process (e.g., in the case of 'git fetch
--unshallow').
However, when $GIT_DIR/shallow changes, Git does not alter or remove any
grafts (nor substituted parents) in memory.
This comes up in a "git fetch --unshallow" with fetch.writeCommitGraph
set to true. Ordinarily in a shallow repository (and before 37b9dcabfc,
even in this case), commit_graph_compatible() would return false,
indicating that the repository should not be used to write a
commit-graphs (since commit-graph files cannot represent a shallow
history). But since 37b9dcabfc, in an --unshallow operation that check
succeeds.
Thus even though the repository isn't shallow any longer (that is, we
have all of the objects), the in-core representation of those objects
still has munged parents at the shallow boundaries. When the
commit-graph write proceeds, we use the incorrect parentage, producing
wrong results.
There are two ways for a user to work around this: either (1) set
'fetch.writeCommitGraph' to 'false', or (2) drop the commit-graph after
unshallowing.
One way to fix this would be to reset the parsed object pool entirely
(flushing the cache and thus preventing subsequent reads from modifying
their parents) after unshallowing. That would produce a problem when
callers have a now-stale reference to the old pool, and so this patch
implements a different approach. Instead, attach a new bit to the pool,
'substituted_parent', which indicates if the repository *ever* stored a
commit which had its parents modified (i.e., the shallow boundary
prior to unshallowing).
This bit needs to be sticky because all reads subsequent to modifying a
commit's parents are unreliable when unshallowing. Modify the check in
'commit_graph_compatible' to take this bit into account, and correctly
avoid generating commit-graphs in this case, thus solving the bug.
Helped-by: Derrick Stolee <dstolee@microsoft.com>
Helped-by: Jonathan Nieder <jrnieder@gmail.com>
Reported-by: Jay Conrod <jayconrod@google.com>
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-07-08 21:10:53 +00:00
|
|
|
test_expect_success 'fetch --unshallow from a full clone' '
|
|
|
|
|
git clone --no-local --depth=2 .git shallow3 &&
|
|
|
|
|
(
|
|
|
|
|
cd shallow3 &&
|
|
|
|
|
git log --format=%s >actual &&
|
|
|
|
|
test_write_lines 4 3 >expect &&
|
|
|
|
|
test_cmp expect actual &&
|
|
|
|
|
git -c fetch.writeCommitGraph fetch --unshallow &&
|
2020-11-18 23:44:33 +00:00
|
|
|
git log origin/main --format=%s >actual &&
|
commit.c: don't persist substituted parents when unshallowing
Since 37b9dcabfc (shallow.c: use '{commit,rollback}_shallow_file',
2020-04-22), Git knows how to reset stat-validity checks for the
$GIT_DIR/shallow file, allowing it to change between a shallow and
non-shallow state in the same process (e.g., in the case of 'git fetch
--unshallow').
However, when $GIT_DIR/shallow changes, Git does not alter or remove any
grafts (nor substituted parents) in memory.
This comes up in a "git fetch --unshallow" with fetch.writeCommitGraph
set to true. Ordinarily in a shallow repository (and before 37b9dcabfc,
even in this case), commit_graph_compatible() would return false,
indicating that the repository should not be used to write a
commit-graphs (since commit-graph files cannot represent a shallow
history). But since 37b9dcabfc, in an --unshallow operation that check
succeeds.
Thus even though the repository isn't shallow any longer (that is, we
have all of the objects), the in-core representation of those objects
still has munged parents at the shallow boundaries. When the
commit-graph write proceeds, we use the incorrect parentage, producing
wrong results.
There are two ways for a user to work around this: either (1) set
'fetch.writeCommitGraph' to 'false', or (2) drop the commit-graph after
unshallowing.
One way to fix this would be to reset the parsed object pool entirely
(flushing the cache and thus preventing subsequent reads from modifying
their parents) after unshallowing. That would produce a problem when
callers have a now-stale reference to the old pool, and so this patch
implements a different approach. Instead, attach a new bit to the pool,
'substituted_parent', which indicates if the repository *ever* stored a
commit which had its parents modified (i.e., the shallow boundary
prior to unshallowing).
This bit needs to be sticky because all reads subsequent to modifying a
commit's parents are unreliable when unshallowing. Modify the check in
'commit_graph_compatible' to take this bit into account, and correctly
avoid generating commit-graphs in this case, thus solving the bug.
Helped-by: Derrick Stolee <dstolee@microsoft.com>
Helped-by: Jonathan Nieder <jrnieder@gmail.com>
Reported-by: Jay Conrod <jayconrod@google.com>
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-07-08 21:10:53 +00:00
|
|
|
test_write_lines 4 3 2 1 >expect &&
|
|
|
|
|
test_cmp expect actual
|
|
|
|
|
)
|
|
|
|
|
'
|
|
|
|
|
|
2013-12-05 13:02:40 +00:00
|
|
|
test_expect_success 'fetch something upstream has but hidden by clients shallow boundaries' '
|
|
|
|
|
# the blob "1" is available in .git but hidden by the
|
|
|
|
|
# shallow2/.git/shallow and it should be resent
|
2016-01-04 09:10:45 +00:00
|
|
|
! git --git-dir=shallow2/.git cat-file blob $(echo 1|git hash-object --stdin) >/dev/null &&
|
2013-12-05 13:02:40 +00:00
|
|
|
echo 1 >1.t &&
|
|
|
|
|
git add 1.t &&
|
|
|
|
|
git commit -m add-1-back &&
|
|
|
|
|
(
|
|
|
|
|
cd shallow2 &&
|
2020-11-18 23:44:33 +00:00
|
|
|
git fetch ../.git +refs/heads/main:refs/remotes/top/main &&
|
2013-12-05 13:02:40 +00:00
|
|
|
git fsck &&
|
2020-11-18 23:44:33 +00:00
|
|
|
git log --format=%s top/main >actual &&
|
2020-04-24 17:11:39 +00:00
|
|
|
test_write_lines add-1-back 4 3 >expect &&
|
2013-12-05 13:02:40 +00:00
|
|
|
test_cmp expect actual
|
|
|
|
|
) &&
|
2016-01-04 09:10:45 +00:00
|
|
|
git --git-dir=shallow2/.git cat-file blob $(echo 1|git hash-object --stdin) >/dev/null
|
2013-12-05 13:02:40 +00:00
|
|
|
'
|
|
|
|
|
|
|
|
|
|
test_expect_success 'fetch that requires changes in .git/shallow is filtered' '
|
|
|
|
|
(
|
|
|
|
|
cd shallow &&
|
|
|
|
|
git checkout --orphan no-shallow &&
|
|
|
|
|
commit no-shallow
|
|
|
|
|
) &&
|
|
|
|
|
git init notshallow &&
|
|
|
|
|
(
|
|
|
|
|
cd notshallow &&
|
2020-03-22 21:14:22 +00:00
|
|
|
git fetch ../shallow/.git refs/heads/*:refs/remotes/shallow/* &&
|
2013-12-05 13:02:40 +00:00
|
|
|
git for-each-ref --format="%(refname)" >actual.refs &&
|
2020-04-24 17:11:39 +00:00
|
|
|
echo refs/remotes/shallow/no-shallow >expect.refs &&
|
2013-12-05 13:02:40 +00:00
|
|
|
test_cmp expect.refs actual.refs &&
|
|
|
|
|
git log --format=%s shallow/no-shallow >actual &&
|
2020-04-24 17:11:39 +00:00
|
|
|
echo no-shallow >expect &&
|
2013-12-05 13:02:40 +00:00
|
|
|
test_cmp expect actual
|
|
|
|
|
)
|
|
|
|
|
'
|
|
|
|
|
|
2013-12-05 13:02:42 +00:00
|
|
|
test_expect_success 'fetch --update-shallow' '
|
|
|
|
|
(
|
|
|
|
|
cd shallow &&
|
2020-11-18 23:44:33 +00:00
|
|
|
git checkout main &&
|
2013-12-05 13:02:42 +00:00
|
|
|
commit 7 &&
|
|
|
|
|
git tag -m foo heavy-tag HEAD^ &&
|
|
|
|
|
git tag light-tag HEAD^:tracked
|
|
|
|
|
) &&
|
|
|
|
|
(
|
|
|
|
|
cd notshallow &&
|
|
|
|
|
git fetch --update-shallow ../shallow/.git refs/heads/*:refs/remotes/shallow/* &&
|
|
|
|
|
git fsck &&
|
|
|
|
|
git for-each-ref --sort=refname --format="%(refname)" >actual.refs &&
|
2020-04-24 17:11:39 +00:00
|
|
|
cat <<-\EOF >expect.refs &&
|
2020-11-18 23:44:33 +00:00
|
|
|
refs/remotes/shallow/main
|
2020-04-24 17:11:39 +00:00
|
|
|
refs/remotes/shallow/no-shallow
|
|
|
|
|
refs/tags/heavy-tag
|
|
|
|
|
refs/tags/light-tag
|
|
|
|
|
EOF
|
2013-12-05 13:02:42 +00:00
|
|
|
test_cmp expect.refs actual.refs &&
|
2020-11-18 23:44:33 +00:00
|
|
|
git log --format=%s shallow/main >actual &&
|
2020-04-24 17:11:39 +00:00
|
|
|
test_write_lines 7 6 5 4 3 >expect &&
|
2013-12-05 13:02:42 +00:00
|
|
|
test_cmp expect actual
|
|
|
|
|
)
|
|
|
|
|
'
|
|
|
|
|
|
shallow.c: use '{commit,rollback}_shallow_file'
In bd0b42aed3 (fetch-pack: do not take shallow lock unnecessarily,
2019-01-10), the author noted that 'is_repository_shallow' produces
visible side-effect(s) by setting 'is_shallow' and 'shallow_stat'.
This is a problem for e.g., fetching with '--update-shallow' in a
shallow repository with 'fetch.writeCommitGraph' enabled, since the
update to '.git/shallow' will cause Git to think that the repository
isn't shallow when it is, thereby circumventing the commit-graph
compatibility check.
This causes problems in shallow repositories with at least shallow refs
that have at least one ancestor (since the client won't have those
objects, and therefore can't take the reachability closure over commits
when writing a commit-graph).
Address this by introducing thin wrappers over 'commit_lock_file' and
'rollback_lock_file' for use specifically when the lock is held over
'.git/shallow'. These wrappers (appropriately called
'commit_shallow_file' and 'rollback_shallow_file') call into their
respective functions in 'lockfile.h', but additionally reset validity
checks used by the shallow machinery.
Replace each instance of 'commit_lock_file' and 'rollback_lock_file'
with 'commit_shallow_file' and 'rollback_shallow_file' when the lock
being held is over the '.git/shallow' file.
As a result, 'prune_shallow' can now only be called once (since
'check_shallow_file_for_update' will die after calling
'reset_repository_shallow'). But, this is OK since we only call
'prune_shallow' at most once per process.
Helped-by: Jonathan Tan <jonathantanmy@google.com>
Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
Reviewed-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-04-23 00:25:45 +00:00
|
|
|
test_expect_success 'fetch --update-shallow (with fetch.writeCommitGraph)' '
|
|
|
|
|
(
|
|
|
|
|
cd shallow &&
|
2020-11-18 23:44:33 +00:00
|
|
|
git checkout main &&
|
shallow.c: use '{commit,rollback}_shallow_file'
In bd0b42aed3 (fetch-pack: do not take shallow lock unnecessarily,
2019-01-10), the author noted that 'is_repository_shallow' produces
visible side-effect(s) by setting 'is_shallow' and 'shallow_stat'.
This is a problem for e.g., fetching with '--update-shallow' in a
shallow repository with 'fetch.writeCommitGraph' enabled, since the
update to '.git/shallow' will cause Git to think that the repository
isn't shallow when it is, thereby circumventing the commit-graph
compatibility check.
This causes problems in shallow repositories with at least shallow refs
that have at least one ancestor (since the client won't have those
objects, and therefore can't take the reachability closure over commits
when writing a commit-graph).
Address this by introducing thin wrappers over 'commit_lock_file' and
'rollback_lock_file' for use specifically when the lock is held over
'.git/shallow'. These wrappers (appropriately called
'commit_shallow_file' and 'rollback_shallow_file') call into their
respective functions in 'lockfile.h', but additionally reset validity
checks used by the shallow machinery.
Replace each instance of 'commit_lock_file' and 'rollback_lock_file'
with 'commit_shallow_file' and 'rollback_shallow_file' when the lock
being held is over the '.git/shallow' file.
As a result, 'prune_shallow' can now only be called once (since
'check_shallow_file_for_update' will die after calling
'reset_repository_shallow'). But, this is OK since we only call
'prune_shallow' at most once per process.
Helped-by: Jonathan Tan <jonathantanmy@google.com>
Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
Reviewed-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-04-23 00:25:45 +00:00
|
|
|
commit 8 &&
|
|
|
|
|
git tag -m foo heavy-tag-for-graph HEAD^ &&
|
|
|
|
|
git tag light-tag-for-graph HEAD^:tracked
|
|
|
|
|
) &&
|
|
|
|
|
test_config -C notshallow fetch.writeCommitGraph true &&
|
|
|
|
|
(
|
|
|
|
|
cd notshallow &&
|
|
|
|
|
git fetch --update-shallow ../shallow/.git refs/heads/*:refs/remotes/shallow/* &&
|
|
|
|
|
git fsck &&
|
|
|
|
|
git for-each-ref --sort=refname --format="%(refname)" >actual.refs &&
|
|
|
|
|
cat <<-EOF >expect.refs &&
|
2020-11-18 23:44:33 +00:00
|
|
|
refs/remotes/shallow/main
|
shallow.c: use '{commit,rollback}_shallow_file'
In bd0b42aed3 (fetch-pack: do not take shallow lock unnecessarily,
2019-01-10), the author noted that 'is_repository_shallow' produces
visible side-effect(s) by setting 'is_shallow' and 'shallow_stat'.
This is a problem for e.g., fetching with '--update-shallow' in a
shallow repository with 'fetch.writeCommitGraph' enabled, since the
update to '.git/shallow' will cause Git to think that the repository
isn't shallow when it is, thereby circumventing the commit-graph
compatibility check.
This causes problems in shallow repositories with at least shallow refs
that have at least one ancestor (since the client won't have those
objects, and therefore can't take the reachability closure over commits
when writing a commit-graph).
Address this by introducing thin wrappers over 'commit_lock_file' and
'rollback_lock_file' for use specifically when the lock is held over
'.git/shallow'. These wrappers (appropriately called
'commit_shallow_file' and 'rollback_shallow_file') call into their
respective functions in 'lockfile.h', but additionally reset validity
checks used by the shallow machinery.
Replace each instance of 'commit_lock_file' and 'rollback_lock_file'
with 'commit_shallow_file' and 'rollback_shallow_file' when the lock
being held is over the '.git/shallow' file.
As a result, 'prune_shallow' can now only be called once (since
'check_shallow_file_for_update' will die after calling
'reset_repository_shallow'). But, this is OK since we only call
'prune_shallow' at most once per process.
Helped-by: Jonathan Tan <jonathantanmy@google.com>
Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
Reviewed-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-04-23 00:25:45 +00:00
|
|
|
refs/remotes/shallow/no-shallow
|
|
|
|
|
refs/tags/heavy-tag
|
|
|
|
|
refs/tags/heavy-tag-for-graph
|
|
|
|
|
refs/tags/light-tag
|
|
|
|
|
refs/tags/light-tag-for-graph
|
|
|
|
|
EOF
|
2013-12-05 13:02:42 +00:00
|
|
|
test_cmp expect.refs actual.refs &&
|
2020-11-18 23:44:33 +00:00
|
|
|
git log --format=%s shallow/main >actual &&
|
shallow.c: use '{commit,rollback}_shallow_file'
In bd0b42aed3 (fetch-pack: do not take shallow lock unnecessarily,
2019-01-10), the author noted that 'is_repository_shallow' produces
visible side-effect(s) by setting 'is_shallow' and 'shallow_stat'.
This is a problem for e.g., fetching with '--update-shallow' in a
shallow repository with 'fetch.writeCommitGraph' enabled, since the
update to '.git/shallow' will cause Git to think that the repository
isn't shallow when it is, thereby circumventing the commit-graph
compatibility check.
This causes problems in shallow repositories with at least shallow refs
that have at least one ancestor (since the client won't have those
objects, and therefore can't take the reachability closure over commits
when writing a commit-graph).
Address this by introducing thin wrappers over 'commit_lock_file' and
'rollback_lock_file' for use specifically when the lock is held over
'.git/shallow'. These wrappers (appropriately called
'commit_shallow_file' and 'rollback_shallow_file') call into their
respective functions in 'lockfile.h', but additionally reset validity
checks used by the shallow machinery.
Replace each instance of 'commit_lock_file' and 'rollback_lock_file'
with 'commit_shallow_file' and 'rollback_shallow_file' when the lock
being held is over the '.git/shallow' file.
As a result, 'prune_shallow' can now only be called once (since
'check_shallow_file_for_update' will die after calling
'reset_repository_shallow'). But, this is OK since we only call
'prune_shallow' at most once per process.
Helped-by: Jonathan Tan <jonathantanmy@google.com>
Helped-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
Reviewed-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-04-23 00:25:45 +00:00
|
|
|
test_write_lines 8 7 6 5 4 3 >expect &&
|
2013-12-05 13:02:42 +00:00
|
|
|
test_cmp expect actual
|
|
|
|
|
)
|
|
|
|
|
'
|
|
|
|
|
|
2014-03-11 12:59:46 +00:00
|
|
|
test_expect_success POSIXPERM,SANITY 'shallow fetch from a read-only repo' '
|
|
|
|
|
cp -R .git read-only.git &&
|
|
|
|
|
test_when_finished "find read-only.git -type d -print | xargs chmod +w" &&
|
2018-06-15 18:13:39 +00:00
|
|
|
find read-only.git -print | xargs chmod -w &&
|
2014-03-11 12:59:46 +00:00
|
|
|
git clone --no-local --depth=2 read-only.git from-read-only &&
|
|
|
|
|
git --git-dir=from-read-only/.git log --format=%s >actual &&
|
2020-04-24 17:11:39 +00:00
|
|
|
test_write_lines add-1-back 4 >expect &&
|
2014-03-11 12:59:46 +00:00
|
|
|
test_cmp expect actual
|
|
|
|
|
'
|
|
|
|
|
|
repack -ad: prune the list of shallow commits
`git repack` can drop unreachable commits without further warning,
making the corresponding entries in `.git/shallow` invalid, which causes
serious problems when deepening the branches.
One scenario where unreachable commits are dropped by `git repack` is
when a `git fetch --prune` (or even a `git fetch` when a ref was
force-pushed in the meantime) can make a commit unreachable that was
reachable before.
Therefore it is not safe to assume that a `git repack -adlf` will keep
unreachable commits alone (under the assumption that they had not been
packed in the first place, which is an assumption at least some of Git's
code seems to make).
This is particularly important to keep in mind when looking at the
`.git/shallow` file: if any commits listed in that file become
unreachable, it is not a problem, but if they go missing, it *is* a
problem. One symptom of this problem is that a deepening fetch may now
fail with
fatal: error in object: unshallow <commit-hash>
To avoid this problem, let's prune the shallow list in `git repack` when
the `-d` option is passed, unless `-A` is passed, too (which would force
the now-unreachable objects to be turned into loose objects instead of
being deleted). Additionally, we also need to take `--keep-reachable`
and `--unpack-unreachable=<date>` into account.
Note: an alternative solution discussed during the review of this patch
was to teach `git fetch` to simply ignore entries in .git/shallow if the
corresponding commits do not exist locally. A quick test, however,
revealed that the .git/shallow file is written during a shallow *clone*,
in which case the commits do not exist, either, but the "shallow" line
*does* need to be sent. Therefore, this approach would be a lot more
finicky than the approach presented by the this patch.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-10-24 15:56:13 +00:00
|
|
|
test_expect_success '.git/shallow is edited by repack' '
|
2018-10-24 15:56:10 +00:00
|
|
|
git init shallow-server &&
|
|
|
|
|
test_commit -C shallow-server A &&
|
|
|
|
|
test_commit -C shallow-server B &&
|
|
|
|
|
git -C shallow-server checkout -b branch &&
|
|
|
|
|
test_commit -C shallow-server C &&
|
|
|
|
|
test_commit -C shallow-server E &&
|
|
|
|
|
test_commit -C shallow-server D &&
|
|
|
|
|
d="$(git -C shallow-server rev-parse --verify D^0)" &&
|
2020-11-18 23:44:33 +00:00
|
|
|
git -C shallow-server checkout main &&
|
2018-10-24 15:56:10 +00:00
|
|
|
|
|
|
|
|
git clone --depth=1 --no-tags --no-single-branch \
|
|
|
|
|
"file://$PWD/shallow-server" shallow-client &&
|
|
|
|
|
|
|
|
|
|
: now remove the branch and fetch with prune &&
|
|
|
|
|
git -C shallow-server branch -D branch &&
|
|
|
|
|
git -C shallow-client fetch --prune --depth=1 \
|
|
|
|
|
origin "+refs/heads/*:refs/remotes/origin/*" &&
|
|
|
|
|
git -C shallow-client repack -adfl &&
|
|
|
|
|
test_must_fail git -C shallow-client rev-parse --verify $d^0 &&
|
|
|
|
|
! grep $d shallow-client/.git/shallow &&
|
|
|
|
|
|
|
|
|
|
git -C shallow-server branch branch-orig $d &&
|
|
|
|
|
git -C shallow-client fetch --prune --depth=2 \
|
|
|
|
|
origin "+refs/heads/*:refs/remotes/origin/*"
|
|
|
|
|
'
|
|
|
|
|
|
fetch-pack: write shallow, then check connectivity
When fetching, connectivity is checked after the shallow file is
updated. There are 2 issues with this: (1) the connectivity check is
only performed up to ancestors of existing refs (which is not thorough
enough if we were deepening an existing ref in the first place), and (2)
there is no rollback of the shallow file if the connectivity check
fails.
To solve (1), update the connectivity check to check the ancestry chain
completely in the case of a deepening fetch by refraining from passing
"--not --all" when invoking rev-list in connected.c.
To solve (2), have fetch_pack() perform its own connectivity check
before updating the shallow file. To support existing use cases in which
"git fetch-pack" is used to download objects without much regard as to
the connectivity of the resulting objects with respect to the existing
repository, the connectivity check is only done if necessary (that is,
the fetch is not a clone, and the fetch involves shallow/deepen
functionality). "git fetch" still performs its own connectivity check,
preserving correctness but sometimes performing redundant work. This
redundancy is mitigated by the fact that fetch_pack() reports if it has
performed a connectivity check itself, and if the transport supports
connect or stateless-connect, it will bubble up that report so that "git
fetch" knows not to perform the connectivity check in such a case.
This was noticed when a user tried to deepen an existing repository by
fetching with --no-shallow from a server that did not send all necessary
objects - the connectivity check as run by "git fetch" succeeded, but a
subsequent "git fsck" failed.
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-07-02 22:08:43 +00:00
|
|
|
. "$TEST_DIRECTORY"/lib-httpd.sh
|
|
|
|
|
start_httpd
|
|
|
|
|
|
|
|
|
|
REPO="$HTTPD_DOCUMENT_ROOT_PATH/repo"
|
|
|
|
|
|
|
|
|
|
test_expect_success 'shallow fetches check connectivity before writing shallow file' '
|
|
|
|
|
rm -rf "$REPO" client &&
|
|
|
|
|
|
|
|
|
|
git init "$REPO" &&
|
|
|
|
|
test_commit -C "$REPO" one &&
|
|
|
|
|
test_commit -C "$REPO" two &&
|
|
|
|
|
test_commit -C "$REPO" three &&
|
|
|
|
|
|
|
|
|
|
git init client &&
|
|
|
|
|
|
|
|
|
|
# Use protocol v2 to ensure that shallow information is sent exactly
|
|
|
|
|
# once by the server, since we are planning to manipulate it.
|
|
|
|
|
git -C "$REPO" config protocol.version 2 &&
|
|
|
|
|
git -C client config protocol.version 2 &&
|
|
|
|
|
|
2020-11-18 23:44:33 +00:00
|
|
|
git -C client fetch --depth=2 "$HTTPD_URL/one_time_perl/repo" main:a_branch &&
|
fetch-pack: write shallow, then check connectivity
When fetching, connectivity is checked after the shallow file is
updated. There are 2 issues with this: (1) the connectivity check is
only performed up to ancestors of existing refs (which is not thorough
enough if we were deepening an existing ref in the first place), and (2)
there is no rollback of the shallow file if the connectivity check
fails.
To solve (1), update the connectivity check to check the ancestry chain
completely in the case of a deepening fetch by refraining from passing
"--not --all" when invoking rev-list in connected.c.
To solve (2), have fetch_pack() perform its own connectivity check
before updating the shallow file. To support existing use cases in which
"git fetch-pack" is used to download objects without much regard as to
the connectivity of the resulting objects with respect to the existing
repository, the connectivity check is only done if necessary (that is,
the fetch is not a clone, and the fetch involves shallow/deepen
functionality). "git fetch" still performs its own connectivity check,
preserving correctness but sometimes performing redundant work. This
redundancy is mitigated by the fact that fetch_pack() reports if it has
performed a connectivity check itself, and if the transport supports
connect or stateless-connect, it will bubble up that report so that "git
fetch" knows not to perform the connectivity check in such a case.
This was noticed when a user tried to deepen an existing repository by
fetching with --no-shallow from a server that did not send all necessary
objects - the connectivity check as run by "git fetch" succeeded, but a
subsequent "git fsck" failed.
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-07-02 22:08:43 +00:00
|
|
|
|
|
|
|
|
# Craft a situation in which the server sends back an unshallow request
|
|
|
|
|
# with an empty packfile. This is done by refetching with a shorter
|
|
|
|
|
# depth (to ensure that the packfile is empty), and overwriting the
|
|
|
|
|
# shallow line in the response with the unshallow line we want.
|
2020-03-04 15:53:12 +00:00
|
|
|
printf "$(test_oid perl)" \
|
fetch-pack: write shallow, then check connectivity
When fetching, connectivity is checked after the shallow file is
updated. There are 2 issues with this: (1) the connectivity check is
only performed up to ancestors of existing refs (which is not thorough
enough if we were deepening an existing ref in the first place), and (2)
there is no rollback of the shallow file if the connectivity check
fails.
To solve (1), update the connectivity check to check the ancestry chain
completely in the case of a deepening fetch by refraining from passing
"--not --all" when invoking rev-list in connected.c.
To solve (2), have fetch_pack() perform its own connectivity check
before updating the shallow file. To support existing use cases in which
"git fetch-pack" is used to download objects without much regard as to
the connectivity of the resulting objects with respect to the existing
repository, the connectivity check is only done if necessary (that is,
the fetch is not a clone, and the fetch involves shallow/deepen
functionality). "git fetch" still performs its own connectivity check,
preserving correctness but sometimes performing redundant work. This
redundancy is mitigated by the fact that fetch_pack() reports if it has
performed a connectivity check itself, and if the transport supports
connect or stateless-connect, it will bubble up that report so that "git
fetch" knows not to perform the connectivity check in such a case.
This was noticed when a user tried to deepen an existing repository by
fetching with --no-shallow from a server that did not send all necessary
objects - the connectivity check as run by "git fetch" succeeded, but a
subsequent "git fsck" failed.
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-07-02 22:08:43 +00:00
|
|
|
"$(git -C "$REPO" rev-parse HEAD)" \
|
|
|
|
|
"$(git -C "$REPO" rev-parse HEAD^)" \
|
t/lib-httpd: avoid using macOS' sed
Among other differences relative to GNU sed, macOS' sed always ends its
output with a trailing newline, even if the input did not have such a
trailing newline.
Surprisingly, this makes three httpd-based tests fail on macOS: t5616,
t5702 and t5703. ("Surprisingly" because those tests have been around
for some time, but apparently nobody runs them on macOS with a working
Apache2 setup.)
The reason is that we use `sed` in those tests to filter the response of
the web server. Apart from the fact that we use GNU constructs (such as
using a space after the `c` command instead of a backslash and a
newline), we have another problem: macOS' sed LF-only newlines while
webservers are supposed to use CR/LF ones.
Even worse, t5616 uses `sed` to replace a binary part of the response
with a new binary part (kind of hoping that the replaced binary part
does not contain a 0x0a byte which would be interpreted as a newline).
To that end, it calls on Perl to read the binary pack file and
hex-encode it, then calls on `sed` to prefix every hex digit pair with a
`\x` in order to construct the text that the `c` statement of the `sed`
invocation is supposed to insert. So we call Perl and sed to construct a
sed statement. The final nail in the coffin is that macOS' sed does not
even interpret those `\x<hex>` constructs.
Let's just replace all of that by Perl snippets. With Perl, at least, we
do not have to deal with GNU vs macOS semantics, we do not have to worry
about unwanted trailing newlines, and we do not have to spawn commands
to construct arguments for other commands to be spawned (i.e. we can
avoid a whole lot of shell scripting complexity).
The upshot is that this fixes t5616, t5702 and t5703 on macOS with
Apache2.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-02-27 13:23:11 +00:00
|
|
|
>"$HTTPD_ROOT_PATH/one-time-perl" &&
|
2019-01-16 19:28:15 +00:00
|
|
|
test_must_fail env GIT_TEST_SIDEBAND_ALL=0 git -C client \
|
t/lib-httpd: avoid using macOS' sed
Among other differences relative to GNU sed, macOS' sed always ends its
output with a trailing newline, even if the input did not have such a
trailing newline.
Surprisingly, this makes three httpd-based tests fail on macOS: t5616,
t5702 and t5703. ("Surprisingly" because those tests have been around
for some time, but apparently nobody runs them on macOS with a working
Apache2 setup.)
The reason is that we use `sed` in those tests to filter the response of
the web server. Apart from the fact that we use GNU constructs (such as
using a space after the `c` command instead of a backslash and a
newline), we have another problem: macOS' sed LF-only newlines while
webservers are supposed to use CR/LF ones.
Even worse, t5616 uses `sed` to replace a binary part of the response
with a new binary part (kind of hoping that the replaced binary part
does not contain a 0x0a byte which would be interpreted as a newline).
To that end, it calls on Perl to read the binary pack file and
hex-encode it, then calls on `sed` to prefix every hex digit pair with a
`\x` in order to construct the text that the `c` statement of the `sed`
invocation is supposed to insert. So we call Perl and sed to construct a
sed statement. The final nail in the coffin is that macOS' sed does not
even interpret those `\x<hex>` constructs.
Let's just replace all of that by Perl snippets. With Perl, at least, we
do not have to deal with GNU vs macOS semantics, we do not have to worry
about unwanted trailing newlines, and we do not have to spawn commands
to construct arguments for other commands to be spawned (i.e. we can
avoid a whole lot of shell scripting complexity).
The upshot is that this fixes t5616, t5702 and t5703 on macOS with
Apache2.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-02-27 13:23:11 +00:00
|
|
|
fetch --depth=1 "$HTTPD_URL/one_time_perl/repo" \
|
2020-11-18 23:44:33 +00:00
|
|
|
main:a_branch &&
|
fetch-pack: write shallow, then check connectivity
When fetching, connectivity is checked after the shallow file is
updated. There are 2 issues with this: (1) the connectivity check is
only performed up to ancestors of existing refs (which is not thorough
enough if we were deepening an existing ref in the first place), and (2)
there is no rollback of the shallow file if the connectivity check
fails.
To solve (1), update the connectivity check to check the ancestry chain
completely in the case of a deepening fetch by refraining from passing
"--not --all" when invoking rev-list in connected.c.
To solve (2), have fetch_pack() perform its own connectivity check
before updating the shallow file. To support existing use cases in which
"git fetch-pack" is used to download objects without much regard as to
the connectivity of the resulting objects with respect to the existing
repository, the connectivity check is only done if necessary (that is,
the fetch is not a clone, and the fetch involves shallow/deepen
functionality). "git fetch" still performs its own connectivity check,
preserving correctness but sometimes performing redundant work. This
redundancy is mitigated by the fact that fetch_pack() reports if it has
performed a connectivity check itself, and if the transport supports
connect or stateless-connect, it will bubble up that report so that "git
fetch" knows not to perform the connectivity check in such a case.
This was noticed when a user tried to deepen an existing repository by
fetching with --no-shallow from a server that did not send all necessary
objects - the connectivity check as run by "git fetch" succeeded, but a
subsequent "git fsck" failed.
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-07-02 22:08:43 +00:00
|
|
|
|
t/lib-httpd: avoid using macOS' sed
Among other differences relative to GNU sed, macOS' sed always ends its
output with a trailing newline, even if the input did not have such a
trailing newline.
Surprisingly, this makes three httpd-based tests fail on macOS: t5616,
t5702 and t5703. ("Surprisingly" because those tests have been around
for some time, but apparently nobody runs them on macOS with a working
Apache2 setup.)
The reason is that we use `sed` in those tests to filter the response of
the web server. Apart from the fact that we use GNU constructs (such as
using a space after the `c` command instead of a backslash and a
newline), we have another problem: macOS' sed LF-only newlines while
webservers are supposed to use CR/LF ones.
Even worse, t5616 uses `sed` to replace a binary part of the response
with a new binary part (kind of hoping that the replaced binary part
does not contain a 0x0a byte which would be interpreted as a newline).
To that end, it calls on Perl to read the binary pack file and
hex-encode it, then calls on `sed` to prefix every hex digit pair with a
`\x` in order to construct the text that the `c` statement of the `sed`
invocation is supposed to insert. So we call Perl and sed to construct a
sed statement. The final nail in the coffin is that macOS' sed does not
even interpret those `\x<hex>` constructs.
Let's just replace all of that by Perl snippets. With Perl, at least, we
do not have to deal with GNU vs macOS semantics, we do not have to worry
about unwanted trailing newlines, and we do not have to spawn commands
to construct arguments for other commands to be spawned (i.e. we can
avoid a whole lot of shell scripting complexity).
The upshot is that this fixes t5616, t5702 and t5703 on macOS with
Apache2.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-02-27 13:23:11 +00:00
|
|
|
# Ensure that the one-time-perl script was used.
|
|
|
|
|
! test -e "$HTTPD_ROOT_PATH/one-time-perl" &&
|
fetch-pack: write shallow, then check connectivity
When fetching, connectivity is checked after the shallow file is
updated. There are 2 issues with this: (1) the connectivity check is
only performed up to ancestors of existing refs (which is not thorough
enough if we were deepening an existing ref in the first place), and (2)
there is no rollback of the shallow file if the connectivity check
fails.
To solve (1), update the connectivity check to check the ancestry chain
completely in the case of a deepening fetch by refraining from passing
"--not --all" when invoking rev-list in connected.c.
To solve (2), have fetch_pack() perform its own connectivity check
before updating the shallow file. To support existing use cases in which
"git fetch-pack" is used to download objects without much regard as to
the connectivity of the resulting objects with respect to the existing
repository, the connectivity check is only done if necessary (that is,
the fetch is not a clone, and the fetch involves shallow/deepen
functionality). "git fetch" still performs its own connectivity check,
preserving correctness but sometimes performing redundant work. This
redundancy is mitigated by the fact that fetch_pack() reports if it has
performed a connectivity check itself, and if the transport supports
connect or stateless-connect, it will bubble up that report so that "git
fetch" knows not to perform the connectivity check in such a case.
This was noticed when a user tried to deepen an existing repository by
fetching with --no-shallow from a server that did not send all necessary
objects - the connectivity check as run by "git fetch" succeeded, but a
subsequent "git fsck" failed.
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-07-02 22:08:43 +00:00
|
|
|
|
|
|
|
|
# Ensure that the resulting repo is consistent, despite our failure to
|
|
|
|
|
# fetch.
|
|
|
|
|
git -C client fsck
|
|
|
|
|
'
|
|
|
|
|
|
2019-08-01 15:53:09 +00:00
|
|
|
# DO NOT add non-httpd-specific tests here, because the last part of this
|
|
|
|
|
# test script is only executed when httpd is available and enabled.
|
|
|
|
|
|
2013-12-05 13:02:40 +00:00
|
|
|
test_done
|