git/t/t5510-fetch.sh

1254 lines
37 KiB
Bash
Raw Normal View History

#!/bin/sh
# Copyright (c) 2006, Junio C Hamano.
test_description='Per branch config variables affects "git fetch".
'
. ./test-lib.sh
. "$TEST_DIRECTORY"/lib-bundle.sh
D=$(pwd)
test_expect_success setup '
echo >file original &&
git add file &&
git commit -a -m original &&
git branch -M main
'
test_expect_success "clone and setup child repos" '
git clone . one &&
(
cd one &&
echo >file updated by one &&
git commit -a -m "updated by one"
) &&
git clone . two &&
(
cd two &&
git config branch.main.remote one &&
git config remote.one.url ../one/.git/ &&
git config remote.one.fetch refs/heads/main:refs/heads/one
) &&
git clone . three &&
(
cd three &&
git config branch.main.remote two &&
git config branch.main.merge refs/heads/one &&
mkdir -p .git/remotes &&
cat >.git/remotes/two <<-\EOF
URL: ../two/.git/
Pull: refs/heads/main:refs/heads/two
Pull: refs/heads/one:refs/heads/one
EOF
) &&
clean up error conventions of remote.c:match_explicit match_explicit is called for each push refspec to try to fully resolve the source and destination sides of the refspec. Currently, we look at each refspec and report errors on both the source and the dest side before aborting. It makes sense to report errors for each refspec, since an error in one is independent of an error in the other. However, reporting errors on the 'dst' side of a refspec if there has been an error on the 'src' side does not necessarily make sense, since the interpretation of the 'dst' side depends on the 'src' side (for example, when creating a new unqualified remote ref, we use the same type as the src ref). This patch lets match_explicit return early when the src side of the refspec is bogus. We still look at all of the refspecs before aborting the push, though. At the same time, we clean up the call signature, which previously took an extra "errs" flag. This was pointless, as we didn't act on that flag, but rather just passed it back to the caller. Instead, we now use the more traditional "return -1" to signal an error, and the caller aggregates the error count. This change fixes two bugs, as well: - the early return avoids a segfault when passing a NULL matched_src to guess_ref() - the check for multiple sources pointing to a single dest aborted if the "err" flag was set. Presumably the intent was not to bother with the check if we had no matched_src. However, since the err flag was passed in from the caller, we might abort the check just because a previous refspec had a problem, which doesn't make sense. In practice, this didn't matter, since due to the error flag we end up aborting the push anyway. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-06-16 16:15:02 +00:00
git clone . bundle &&
git clone . seven
'
test_expect_success "fetch test" '
cd "$D" &&
echo >file updated by origin &&
git commit -a -m "updated by origin" &&
cd two &&
git fetch &&
git rev-parse --verify refs/heads/one &&
mine=$(git rev-parse refs/heads/one) &&
his=$(cd ../one && git rev-parse refs/heads/main) &&
test "z$mine" = "z$his"
'
test_expect_success "fetch test for-merge" '
cd "$D" &&
cd three &&
git fetch &&
git rev-parse --verify refs/heads/two &&
git rev-parse --verify refs/heads/one &&
main_in_two=$(cd ../two && git rev-parse main) &&
one_in_two=$(cd ../two && git rev-parse one) &&
{
echo "$one_in_two " &&
echo "$main_in_two not-for-merge"
} >expected &&
cut -f -2 .git/FETCH_HEAD >actual &&
test_cmp expected actual'
test_expect_success 'fetch --prune on its own works as expected' '
cd "$D" &&
git clone . prune &&
cd prune &&
git update-ref refs/remotes/origin/extrabranch main &&
git fetch --prune origin &&
test_must_fail git rev-parse origin/extrabranch
'
test_expect_success 'fetch --prune with a branch name keeps branches' '
cd "$D" &&
git clone . prune-branch &&
cd prune-branch &&
git update-ref refs/remotes/origin/extrabranch main &&
git fetch --prune origin main &&
git rev-parse origin/extrabranch
'
test_expect_success 'fetch --prune with a namespace keeps other namespaces' '
cd "$D" &&
git clone . prune-namespace &&
cd prune-namespace &&
git fetch --prune origin refs/heads/a/*:refs/remotes/origin/a/* &&
git rev-parse origin/main
'
test_expect_success 'fetch --prune handles overlapping refspecs' '
cd "$D" &&
git update-ref refs/pull/42/head main &&
git clone . prune-overlapping &&
cd prune-overlapping &&
git config --add remote.origin.fetch refs/pull/*/head:refs/remotes/origin/pr/* &&
git fetch --prune origin &&
git rev-parse origin/main &&
git rev-parse origin/pr/42 &&
git config --unset-all remote.origin.fetch &&
git config remote.origin.fetch refs/pull/*/head:refs/remotes/origin/pr/* &&
git config --add remote.origin.fetch refs/heads/*:refs/remotes/origin/* &&
git fetch --prune origin &&
git rev-parse origin/main &&
git rev-parse origin/pr/42
'
fetch --prune: prune only based on explicit refspecs The old behavior of "fetch --prune" was to prune whatever was being fetched. In particular, "fetch --prune --tags" caused tags not only to be fetched, but also to be pruned. This is inappropriate because there is only one tags namespace that is shared among the local repository and all remotes. Therefore, if the user defines a local tag and then runs "git fetch --prune --tags", then the local tag is deleted. Moreover, "--prune" and "--tags" can also be configured via fetch.prune / remote.<name>.prune and remote.<name>.tagopt, making it even less obvious that an invocation of "git fetch" could result in tag lossage. Since the command "git remote update" invokes "git fetch", it had the same problem. The command "git remote prune", on the other hand, disregarded the setting of remote.<name>.tagopt, and so its behavior was inconsistent with that of the other commands. So the old behavior made it too easy to lose tags. To fix this problem, change "fetch --prune" to prune references based only on refspecs specified explicitly by the user, either on the command line or via remote.<name>.fetch. Thus, tags are no longer made subject to pruning by the --tags option or the remote.<name>.tagopt setting. However, tags *are* still subject to pruning if they are fetched as part of a refspec, and that is good. For example: * On the command line, git fetch --prune 'refs/tags/*:refs/tags/*' causes tags, and only tags, to be fetched and pruned, and is therefore a simple way for the user to get the equivalent of the old behavior of "--prune --tag". * For a remote that was configured with the "--mirror" option, the configuration is set to include [remote "name"] fetch = +refs/*:refs/* , which causes tags to be subject to pruning along with all other references. This is the behavior that will typically be desired for a mirror. Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-10-30 05:33:00 +00:00
test_expect_success 'fetch --prune --tags prunes branches but not tags' '
cd "$D" &&
git clone . prune-tags &&
cd prune-tags &&
git tag sometag main &&
# Create what looks like a remote-tracking branch from an earlier
# fetch that has since been deleted from the remote:
git update-ref refs/remotes/origin/fake-remote main &&
git fetch --prune --tags origin &&
git rev-parse origin/main &&
test_must_fail git rev-parse origin/fake-remote &&
fetch --prune: prune only based on explicit refspecs The old behavior of "fetch --prune" was to prune whatever was being fetched. In particular, "fetch --prune --tags" caused tags not only to be fetched, but also to be pruned. This is inappropriate because there is only one tags namespace that is shared among the local repository and all remotes. Therefore, if the user defines a local tag and then runs "git fetch --prune --tags", then the local tag is deleted. Moreover, "--prune" and "--tags" can also be configured via fetch.prune / remote.<name>.prune and remote.<name>.tagopt, making it even less obvious that an invocation of "git fetch" could result in tag lossage. Since the command "git remote update" invokes "git fetch", it had the same problem. The command "git remote prune", on the other hand, disregarded the setting of remote.<name>.tagopt, and so its behavior was inconsistent with that of the other commands. So the old behavior made it too easy to lose tags. To fix this problem, change "fetch --prune" to prune references based only on refspecs specified explicitly by the user, either on the command line or via remote.<name>.fetch. Thus, tags are no longer made subject to pruning by the --tags option or the remote.<name>.tagopt setting. However, tags *are* still subject to pruning if they are fetched as part of a refspec, and that is good. For example: * On the command line, git fetch --prune 'refs/tags/*:refs/tags/*' causes tags, and only tags, to be fetched and pruned, and is therefore a simple way for the user to get the equivalent of the old behavior of "--prune --tag". * For a remote that was configured with the "--mirror" option, the configuration is set to include [remote "name"] fetch = +refs/*:refs/* , which causes tags to be subject to pruning along with all other references. This is the behavior that will typically be desired for a mirror. Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-10-30 05:33:00 +00:00
git rev-parse sometag
'
fetch --prune: prune only based on explicit refspecs The old behavior of "fetch --prune" was to prune whatever was being fetched. In particular, "fetch --prune --tags" caused tags not only to be fetched, but also to be pruned. This is inappropriate because there is only one tags namespace that is shared among the local repository and all remotes. Therefore, if the user defines a local tag and then runs "git fetch --prune --tags", then the local tag is deleted. Moreover, "--prune" and "--tags" can also be configured via fetch.prune / remote.<name>.prune and remote.<name>.tagopt, making it even less obvious that an invocation of "git fetch" could result in tag lossage. Since the command "git remote update" invokes "git fetch", it had the same problem. The command "git remote prune", on the other hand, disregarded the setting of remote.<name>.tagopt, and so its behavior was inconsistent with that of the other commands. So the old behavior made it too easy to lose tags. To fix this problem, change "fetch --prune" to prune references based only on refspecs specified explicitly by the user, either on the command line or via remote.<name>.fetch. Thus, tags are no longer made subject to pruning by the --tags option or the remote.<name>.tagopt setting. However, tags *are* still subject to pruning if they are fetched as part of a refspec, and that is good. For example: * On the command line, git fetch --prune 'refs/tags/*:refs/tags/*' causes tags, and only tags, to be fetched and pruned, and is therefore a simple way for the user to get the equivalent of the old behavior of "--prune --tag". * For a remote that was configured with the "--mirror" option, the configuration is set to include [remote "name"] fetch = +refs/*:refs/* , which causes tags to be subject to pruning along with all other references. This is the behavior that will typically be desired for a mirror. Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-10-30 05:33:00 +00:00
test_expect_success 'fetch --prune --tags with branch does not prune other things' '
cd "$D" &&
git clone . prune-tags-branch &&
cd prune-tags-branch &&
git tag sometag main &&
git update-ref refs/remotes/origin/extrabranch main &&
git fetch --prune --tags origin main &&
git rev-parse origin/extrabranch &&
fetch --prune: prune only based on explicit refspecs The old behavior of "fetch --prune" was to prune whatever was being fetched. In particular, "fetch --prune --tags" caused tags not only to be fetched, but also to be pruned. This is inappropriate because there is only one tags namespace that is shared among the local repository and all remotes. Therefore, if the user defines a local tag and then runs "git fetch --prune --tags", then the local tag is deleted. Moreover, "--prune" and "--tags" can also be configured via fetch.prune / remote.<name>.prune and remote.<name>.tagopt, making it even less obvious that an invocation of "git fetch" could result in tag lossage. Since the command "git remote update" invokes "git fetch", it had the same problem. The command "git remote prune", on the other hand, disregarded the setting of remote.<name>.tagopt, and so its behavior was inconsistent with that of the other commands. So the old behavior made it too easy to lose tags. To fix this problem, change "fetch --prune" to prune references based only on refspecs specified explicitly by the user, either on the command line or via remote.<name>.fetch. Thus, tags are no longer made subject to pruning by the --tags option or the remote.<name>.tagopt setting. However, tags *are* still subject to pruning if they are fetched as part of a refspec, and that is good. For example: * On the command line, git fetch --prune 'refs/tags/*:refs/tags/*' causes tags, and only tags, to be fetched and pruned, and is therefore a simple way for the user to get the equivalent of the old behavior of "--prune --tag". * For a remote that was configured with the "--mirror" option, the configuration is set to include [remote "name"] fetch = +refs/*:refs/* , which causes tags to be subject to pruning along with all other references. This is the behavior that will typically be desired for a mirror. Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-10-30 05:33:00 +00:00
git rev-parse sometag
'
test_expect_success 'fetch --prune --tags with refspec prunes based on refspec' '
cd "$D" &&
git clone . prune-tags-refspec &&
cd prune-tags-refspec &&
git tag sometag main &&
git update-ref refs/remotes/origin/foo/otherbranch main &&
git update-ref refs/remotes/origin/extrabranch main &&
git fetch --prune --tags origin refs/heads/foo/*:refs/remotes/origin/foo/* &&
test_must_fail git rev-parse refs/remotes/origin/foo/otherbranch &&
git rev-parse origin/extrabranch &&
fetch --prune: prune only based on explicit refspecs The old behavior of "fetch --prune" was to prune whatever was being fetched. In particular, "fetch --prune --tags" caused tags not only to be fetched, but also to be pruned. This is inappropriate because there is only one tags namespace that is shared among the local repository and all remotes. Therefore, if the user defines a local tag and then runs "git fetch --prune --tags", then the local tag is deleted. Moreover, "--prune" and "--tags" can also be configured via fetch.prune / remote.<name>.prune and remote.<name>.tagopt, making it even less obvious that an invocation of "git fetch" could result in tag lossage. Since the command "git remote update" invokes "git fetch", it had the same problem. The command "git remote prune", on the other hand, disregarded the setting of remote.<name>.tagopt, and so its behavior was inconsistent with that of the other commands. So the old behavior made it too easy to lose tags. To fix this problem, change "fetch --prune" to prune references based only on refspecs specified explicitly by the user, either on the command line or via remote.<name>.fetch. Thus, tags are no longer made subject to pruning by the --tags option or the remote.<name>.tagopt setting. However, tags *are* still subject to pruning if they are fetched as part of a refspec, and that is good. For example: * On the command line, git fetch --prune 'refs/tags/*:refs/tags/*' causes tags, and only tags, to be fetched and pruned, and is therefore a simple way for the user to get the equivalent of the old behavior of "--prune --tag". * For a remote that was configured with the "--mirror" option, the configuration is set to include [remote "name"] fetch = +refs/*:refs/* , which causes tags to be subject to pruning along with all other references. This is the behavior that will typically be desired for a mirror. Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-10-30 05:33:00 +00:00
git rev-parse sometag
'
test_expect_success REFFILES 'fetch --prune fails to delete branches' '
cd "$D" &&
git clone . prune-fail &&
cd prune-fail &&
git update-ref refs/remotes/origin/extrabranch main &&
: this will prevent --prune from locking packed-refs for deleting refs, but adding loose refs still succeeds &&
>.git/packed-refs.new &&
test_must_fail git fetch --prune origin
'
fetch: implement support for atomic reference updates When executing a fetch, then git will currently allocate one reference transaction per reference update and directly commit it. This means that fetches are non-atomic: even if some of the reference updates fail, others may still succeed and modify local references. This is fine in many scenarios, but this strategy has its downsides. - The view of remote references may be inconsistent and may show a bastardized state of the remote repository. - Batching together updates may improve performance in certain scenarios. While the impact probably isn't as pronounced with loose references, the upcoming reftable backend may benefit as it needs to write less files in case the update is batched. - The reference-update hook is currently being executed twice per updated reference. While this doesn't matter when there is no such hook, we have seen severe performance regressions when doing a git-fetch(1) with reference-transaction hook when the remote repository has hundreds of thousands of references. Similar to `git push --atomic`, this commit thus introduces atomic fetches. Instead of allocating one reference transaction per updated reference, it causes us to only allocate a single transaction and commit it as soon as all updates were received. If locking of any reference fails, then we abort the complete transaction and don't update any reference, which gives us an all-or-nothing fetch. Note that this may not completely fix the first of above downsides, as the consistent view also depends on the server-side. If the server doesn't have a consistent view of its own references during the reference negotiation phase, then the client would get the same inconsistent view the server has. This is a separate problem though and, if it actually exists, can be fixed at a later point. This commit also changes the way we write FETCH_HEAD in case `--atomic` is passed. Instead of writing changes as we go, we need to accumulate all changes first and only commit them at the end when we know that all reference updates succeeded. Ideally, we'd just do so via a temporary file so that we don't need to carry all updates in-memory. This isn't trivially doable though considering the `--append` mode, where we do not truncate the file but simply append to it. And given that we support concurrent processes appending to FETCH_HEAD at the same time without any loss of data, seeding the temporary file with current contents of FETCH_HEAD initially and then doing a rename wouldn't work either. So this commit implements the simple strategy of buffering all changes and appending them to the file on commit. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-01-12 12:27:52 +00:00
test_expect_success 'fetch --atomic works with a single branch' '
test_when_finished "rm -rf \"$D\"/atomic" &&
cd "$D" &&
git clone . atomic &&
git branch atomic-branch &&
oid=$(git rev-parse atomic-branch) &&
echo "$oid" >expected &&
git -C atomic fetch --atomic origin &&
git -C atomic rev-parse origin/atomic-branch >actual &&
test_cmp expected actual &&
test $oid = "$(git -C atomic rev-parse --verify FETCH_HEAD)"
'
test_expect_success 'fetch --atomic works with multiple branches' '
test_when_finished "rm -rf \"$D\"/atomic" &&
cd "$D" &&
git clone . atomic &&
git branch atomic-branch-1 &&
git branch atomic-branch-2 &&
git branch atomic-branch-3 &&
git rev-parse refs/heads/atomic-branch-1 refs/heads/atomic-branch-2 refs/heads/atomic-branch-3 >actual &&
git -C atomic fetch --atomic origin &&
git -C atomic rev-parse refs/remotes/origin/atomic-branch-1 refs/remotes/origin/atomic-branch-2 refs/remotes/origin/atomic-branch-3 >expected &&
test_cmp expected actual
'
test_expect_success 'fetch --atomic works with mixed branches and tags' '
test_when_finished "rm -rf \"$D\"/atomic" &&
cd "$D" &&
git clone . atomic &&
git branch atomic-mixed-branch &&
git tag atomic-mixed-tag &&
git rev-parse refs/heads/atomic-mixed-branch refs/tags/atomic-mixed-tag >actual &&
git -C atomic fetch --tags --atomic origin &&
git -C atomic rev-parse refs/remotes/origin/atomic-mixed-branch refs/tags/atomic-mixed-tag >expected &&
test_cmp expected actual
'
test_expect_success 'fetch --atomic prunes references' '
test_when_finished "rm -rf \"$D\"/atomic" &&
cd "$D" &&
git branch atomic-prune-delete &&
git clone . atomic &&
git branch --delete atomic-prune-delete &&
git branch atomic-prune-create &&
git rev-parse refs/heads/atomic-prune-create >actual &&
git -C atomic fetch --prune --atomic origin &&
test_must_fail git -C atomic rev-parse refs/remotes/origin/atomic-prune-delete &&
git -C atomic rev-parse refs/remotes/origin/atomic-prune-create >expected &&
test_cmp expected actual
'
test_expect_success 'fetch --atomic aborts with non-fast-forward update' '
test_when_finished "rm -rf \"$D\"/atomic" &&
cd "$D" &&
git branch atomic-non-ff &&
git clone . atomic &&
git rev-parse HEAD >actual &&
git branch atomic-new-branch &&
parent_commit=$(git rev-parse atomic-non-ff~) &&
git update-ref refs/heads/atomic-non-ff $parent_commit &&
test_must_fail git -C atomic fetch --atomic origin refs/heads/*:refs/remotes/origin/* &&
test_must_fail git -C atomic rev-parse refs/remotes/origin/atomic-new-branch &&
git -C atomic rev-parse refs/remotes/origin/atomic-non-ff >expected &&
test_cmp expected actual &&
test_must_be_empty atomic/.git/FETCH_HEAD
'
test_expect_success 'fetch --atomic executes a single reference transaction only' '
test_when_finished "rm -rf \"$D\"/atomic" &&
cd "$D" &&
git clone . atomic &&
git branch atomic-hooks-1 &&
git branch atomic-hooks-2 &&
head_oid=$(git rev-parse HEAD) &&
cat >expected <<-EOF &&
prepared
$ZERO_OID $head_oid refs/remotes/origin/atomic-hooks-1
$ZERO_OID $head_oid refs/remotes/origin/atomic-hooks-2
committed
$ZERO_OID $head_oid refs/remotes/origin/atomic-hooks-1
$ZERO_OID $head_oid refs/remotes/origin/atomic-hooks-2
EOF
rm -f atomic/actual &&
test_hook -C atomic reference-transaction <<-\EOF &&
fetch: implement support for atomic reference updates When executing a fetch, then git will currently allocate one reference transaction per reference update and directly commit it. This means that fetches are non-atomic: even if some of the reference updates fail, others may still succeed and modify local references. This is fine in many scenarios, but this strategy has its downsides. - The view of remote references may be inconsistent and may show a bastardized state of the remote repository. - Batching together updates may improve performance in certain scenarios. While the impact probably isn't as pronounced with loose references, the upcoming reftable backend may benefit as it needs to write less files in case the update is batched. - The reference-update hook is currently being executed twice per updated reference. While this doesn't matter when there is no such hook, we have seen severe performance regressions when doing a git-fetch(1) with reference-transaction hook when the remote repository has hundreds of thousands of references. Similar to `git push --atomic`, this commit thus introduces atomic fetches. Instead of allocating one reference transaction per updated reference, it causes us to only allocate a single transaction and commit it as soon as all updates were received. If locking of any reference fails, then we abort the complete transaction and don't update any reference, which gives us an all-or-nothing fetch. Note that this may not completely fix the first of above downsides, as the consistent view also depends on the server-side. If the server doesn't have a consistent view of its own references during the reference negotiation phase, then the client would get the same inconsistent view the server has. This is a separate problem though and, if it actually exists, can be fixed at a later point. This commit also changes the way we write FETCH_HEAD in case `--atomic` is passed. Instead of writing changes as we go, we need to accumulate all changes first and only commit them at the end when we know that all reference updates succeeded. Ideally, we'd just do so via a temporary file so that we don't need to carry all updates in-memory. This isn't trivially doable though considering the `--append` mode, where we do not truncate the file but simply append to it. And given that we support concurrent processes appending to FETCH_HEAD at the same time without any loss of data, seeding the temporary file with current contents of FETCH_HEAD initially and then doing a rename wouldn't work either. So this commit implements the simple strategy of buffering all changes and appending them to the file on commit. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-01-12 12:27:52 +00:00
( echo "$*" && cat ) >>actual
EOF
git -C atomic fetch --atomic origin &&
test_cmp expected atomic/actual
'
test_expect_success 'fetch --atomic aborts all reference updates if hook aborts' '
test_when_finished "rm -rf \"$D\"/atomic" &&
cd "$D" &&
git clone . atomic &&
git branch atomic-hooks-abort-1 &&
git branch atomic-hooks-abort-2 &&
git branch atomic-hooks-abort-3 &&
git tag atomic-hooks-abort &&
head_oid=$(git rev-parse HEAD) &&
cat >expected <<-EOF &&
prepared
$ZERO_OID $head_oid refs/remotes/origin/atomic-hooks-abort-1
$ZERO_OID $head_oid refs/remotes/origin/atomic-hooks-abort-2
$ZERO_OID $head_oid refs/remotes/origin/atomic-hooks-abort-3
$ZERO_OID $head_oid refs/tags/atomic-hooks-abort
aborted
$ZERO_OID $head_oid refs/remotes/origin/atomic-hooks-abort-1
$ZERO_OID $head_oid refs/remotes/origin/atomic-hooks-abort-2
$ZERO_OID $head_oid refs/remotes/origin/atomic-hooks-abort-3
$ZERO_OID $head_oid refs/tags/atomic-hooks-abort
EOF
rm -f atomic/actual &&
test_hook -C atomic/.git reference-transaction <<-\EOF &&
fetch: implement support for atomic reference updates When executing a fetch, then git will currently allocate one reference transaction per reference update and directly commit it. This means that fetches are non-atomic: even if some of the reference updates fail, others may still succeed and modify local references. This is fine in many scenarios, but this strategy has its downsides. - The view of remote references may be inconsistent and may show a bastardized state of the remote repository. - Batching together updates may improve performance in certain scenarios. While the impact probably isn't as pronounced with loose references, the upcoming reftable backend may benefit as it needs to write less files in case the update is batched. - The reference-update hook is currently being executed twice per updated reference. While this doesn't matter when there is no such hook, we have seen severe performance regressions when doing a git-fetch(1) with reference-transaction hook when the remote repository has hundreds of thousands of references. Similar to `git push --atomic`, this commit thus introduces atomic fetches. Instead of allocating one reference transaction per updated reference, it causes us to only allocate a single transaction and commit it as soon as all updates were received. If locking of any reference fails, then we abort the complete transaction and don't update any reference, which gives us an all-or-nothing fetch. Note that this may not completely fix the first of above downsides, as the consistent view also depends on the server-side. If the server doesn't have a consistent view of its own references during the reference negotiation phase, then the client would get the same inconsistent view the server has. This is a separate problem though and, if it actually exists, can be fixed at a later point. This commit also changes the way we write FETCH_HEAD in case `--atomic` is passed. Instead of writing changes as we go, we need to accumulate all changes first and only commit them at the end when we know that all reference updates succeeded. Ideally, we'd just do so via a temporary file so that we don't need to carry all updates in-memory. This isn't trivially doable though considering the `--append` mode, where we do not truncate the file but simply append to it. And given that we support concurrent processes appending to FETCH_HEAD at the same time without any loss of data, seeding the temporary file with current contents of FETCH_HEAD initially and then doing a rename wouldn't work either. So this commit implements the simple strategy of buffering all changes and appending them to the file on commit. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-01-12 12:27:52 +00:00
( echo "$*" && cat ) >>actual
exit 1
EOF
git -C atomic for-each-ref >expected-refs &&
test_must_fail git -C atomic fetch --tags --atomic origin &&
git -C atomic for-each-ref >actual-refs &&
test_cmp expected-refs actual-refs &&
test_must_be_empty atomic/.git/FETCH_HEAD
'
test_expect_success 'fetch --atomic --append appends to FETCH_HEAD' '
test_when_finished "rm -rf \"$D\"/atomic" &&
cd "$D" &&
git clone . atomic &&
oid=$(git rev-parse HEAD) &&
git branch atomic-fetch-head-1 &&
git -C atomic fetch --atomic origin atomic-fetch-head-1 &&
test_line_count = 1 atomic/.git/FETCH_HEAD &&
git branch atomic-fetch-head-2 &&
git -C atomic fetch --atomic --append origin atomic-fetch-head-2 &&
test_line_count = 2 atomic/.git/FETCH_HEAD &&
cp atomic/.git/FETCH_HEAD expected &&
test_hook -C atomic reference-transaction <<-\EOF &&
fetch: implement support for atomic reference updates When executing a fetch, then git will currently allocate one reference transaction per reference update and directly commit it. This means that fetches are non-atomic: even if some of the reference updates fail, others may still succeed and modify local references. This is fine in many scenarios, but this strategy has its downsides. - The view of remote references may be inconsistent and may show a bastardized state of the remote repository. - Batching together updates may improve performance in certain scenarios. While the impact probably isn't as pronounced with loose references, the upcoming reftable backend may benefit as it needs to write less files in case the update is batched. - The reference-update hook is currently being executed twice per updated reference. While this doesn't matter when there is no such hook, we have seen severe performance regressions when doing a git-fetch(1) with reference-transaction hook when the remote repository has hundreds of thousands of references. Similar to `git push --atomic`, this commit thus introduces atomic fetches. Instead of allocating one reference transaction per updated reference, it causes us to only allocate a single transaction and commit it as soon as all updates were received. If locking of any reference fails, then we abort the complete transaction and don't update any reference, which gives us an all-or-nothing fetch. Note that this may not completely fix the first of above downsides, as the consistent view also depends on the server-side. If the server doesn't have a consistent view of its own references during the reference negotiation phase, then the client would get the same inconsistent view the server has. This is a separate problem though and, if it actually exists, can be fixed at a later point. This commit also changes the way we write FETCH_HEAD in case `--atomic` is passed. Instead of writing changes as we go, we need to accumulate all changes first and only commit them at the end when we know that all reference updates succeeded. Ideally, we'd just do so via a temporary file so that we don't need to carry all updates in-memory. This isn't trivially doable though considering the `--append` mode, where we do not truncate the file but simply append to it. And given that we support concurrent processes appending to FETCH_HEAD at the same time without any loss of data, seeding the temporary file with current contents of FETCH_HEAD initially and then doing a rename wouldn't work either. So this commit implements the simple strategy of buffering all changes and appending them to the file on commit. Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-01-12 12:27:52 +00:00
exit 1
EOF
git branch atomic-fetch-head-3 &&
test_must_fail git -C atomic fetch --atomic --append origin atomic-fetch-head-3 &&
test_cmp expected atomic/.git/FETCH_HEAD
'
fetch: document and test --refmap="" To prevent long blocking time during a 'git fetch' call, a user may want to set up a schedule for background 'git fetch' processes. However, these runs will update the refs/remotes branches due to the default refspec set in the config when Git adds a remote. Hence the user will not notice when remote refs are updated during their foreground fetches. In fact, they may _want_ those refs to stay put so they can work with the refs from their last foreground fetch call. This can be accomplished by overriding the configured refspec using '--refmap=' along with a custom refspec: git fetch --refmap='' <remote> +refs/heads/*:refs/hidden/<remote>/* to populate a custom ref space and download a pack of the new reachable objects. This kind of call allows a few things to happen: 1. We download a new pack if refs have updated. 2. Since the refs/hidden branches exist, GC will not remove the newly-downloaded data. 3. With fetch.writeCommitGraph enabled, the refs/hidden refs are used to update the commit-graph file. To avoid the refs/hidden directory from filling without bound, the --prune option can be included. When providing a refspec like this, the --prune option does not delete remote refs and instead only deletes refs in the target refspace. Update the documentation to clarify how '--refmap=""' works and create tests to guarantee this behavior remains in the future. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-01-21 01:38:12 +00:00
test_expect_success '--refmap="" ignores configured refspec' '
cd "$TRASH_DIRECTORY" &&
git clone "$D" remote-refs &&
git -C remote-refs rev-parse remotes/origin/main >old &&
git -C remote-refs update-ref refs/remotes/origin/main main~1 &&
git -C remote-refs rev-parse remotes/origin/main >new &&
fetch: document and test --refmap="" To prevent long blocking time during a 'git fetch' call, a user may want to set up a schedule for background 'git fetch' processes. However, these runs will update the refs/remotes branches due to the default refspec set in the config when Git adds a remote. Hence the user will not notice when remote refs are updated during their foreground fetches. In fact, they may _want_ those refs to stay put so they can work with the refs from their last foreground fetch call. This can be accomplished by overriding the configured refspec using '--refmap=' along with a custom refspec: git fetch --refmap='' <remote> +refs/heads/*:refs/hidden/<remote>/* to populate a custom ref space and download a pack of the new reachable objects. This kind of call allows a few things to happen: 1. We download a new pack if refs have updated. 2. Since the refs/hidden branches exist, GC will not remove the newly-downloaded data. 3. With fetch.writeCommitGraph enabled, the refs/hidden refs are used to update the commit-graph file. To avoid the refs/hidden directory from filling without bound, the --prune option can be included. When providing a refspec like this, the --prune option does not delete remote refs and instead only deletes refs in the target refspace. Update the documentation to clarify how '--refmap=""' works and create tests to guarantee this behavior remains in the future. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-01-21 01:38:12 +00:00
git -C remote-refs fetch --refmap= origin "+refs/heads/*:refs/hidden/origin/*" &&
git -C remote-refs rev-parse remotes/origin/main >actual &&
fetch: document and test --refmap="" To prevent long blocking time during a 'git fetch' call, a user may want to set up a schedule for background 'git fetch' processes. However, these runs will update the refs/remotes branches due to the default refspec set in the config when Git adds a remote. Hence the user will not notice when remote refs are updated during their foreground fetches. In fact, they may _want_ those refs to stay put so they can work with the refs from their last foreground fetch call. This can be accomplished by overriding the configured refspec using '--refmap=' along with a custom refspec: git fetch --refmap='' <remote> +refs/heads/*:refs/hidden/<remote>/* to populate a custom ref space and download a pack of the new reachable objects. This kind of call allows a few things to happen: 1. We download a new pack if refs have updated. 2. Since the refs/hidden branches exist, GC will not remove the newly-downloaded data. 3. With fetch.writeCommitGraph enabled, the refs/hidden refs are used to update the commit-graph file. To avoid the refs/hidden directory from filling without bound, the --prune option can be included. When providing a refspec like this, the --prune option does not delete remote refs and instead only deletes refs in the target refspace. Update the documentation to clarify how '--refmap=""' works and create tests to guarantee this behavior remains in the future. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-01-21 01:38:12 +00:00
test_cmp new actual &&
git -C remote-refs fetch origin &&
git -C remote-refs rev-parse remotes/origin/main >actual &&
fetch: document and test --refmap="" To prevent long blocking time during a 'git fetch' call, a user may want to set up a schedule for background 'git fetch' processes. However, these runs will update the refs/remotes branches due to the default refspec set in the config when Git adds a remote. Hence the user will not notice when remote refs are updated during their foreground fetches. In fact, they may _want_ those refs to stay put so they can work with the refs from their last foreground fetch call. This can be accomplished by overriding the configured refspec using '--refmap=' along with a custom refspec: git fetch --refmap='' <remote> +refs/heads/*:refs/hidden/<remote>/* to populate a custom ref space and download a pack of the new reachable objects. This kind of call allows a few things to happen: 1. We download a new pack if refs have updated. 2. Since the refs/hidden branches exist, GC will not remove the newly-downloaded data. 3. With fetch.writeCommitGraph enabled, the refs/hidden refs are used to update the commit-graph file. To avoid the refs/hidden directory from filling without bound, the --prune option can be included. When providing a refspec like this, the --prune option does not delete remote refs and instead only deletes refs in the target refspace. Update the documentation to clarify how '--refmap=""' works and create tests to guarantee this behavior remains in the future. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-01-21 01:38:12 +00:00
test_cmp old actual
'
test_expect_success '--refmap="" and --prune' '
git -C remote-refs update-ref refs/remotes/origin/foo/otherbranch main &&
git -C remote-refs update-ref refs/hidden/foo/otherbranch main &&
fetch: document and test --refmap="" To prevent long blocking time during a 'git fetch' call, a user may want to set up a schedule for background 'git fetch' processes. However, these runs will update the refs/remotes branches due to the default refspec set in the config when Git adds a remote. Hence the user will not notice when remote refs are updated during their foreground fetches. In fact, they may _want_ those refs to stay put so they can work with the refs from their last foreground fetch call. This can be accomplished by overriding the configured refspec using '--refmap=' along with a custom refspec: git fetch --refmap='' <remote> +refs/heads/*:refs/hidden/<remote>/* to populate a custom ref space and download a pack of the new reachable objects. This kind of call allows a few things to happen: 1. We download a new pack if refs have updated. 2. Since the refs/hidden branches exist, GC will not remove the newly-downloaded data. 3. With fetch.writeCommitGraph enabled, the refs/hidden refs are used to update the commit-graph file. To avoid the refs/hidden directory from filling without bound, the --prune option can be included. When providing a refspec like this, the --prune option does not delete remote refs and instead only deletes refs in the target refspace. Update the documentation to clarify how '--refmap=""' works and create tests to guarantee this behavior remains in the future. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-01-21 01:38:12 +00:00
git -C remote-refs fetch --prune --refmap="" origin +refs/heads/*:refs/hidden/* &&
git -C remote-refs rev-parse remotes/origin/foo/otherbranch &&
test_must_fail git -C remote-refs rev-parse refs/hidden/foo/otherbranch &&
git -C remote-refs fetch --prune origin &&
test_must_fail git -C remote-refs rev-parse remotes/origin/foo/otherbranch
'
test_expect_success 'fetch tags when there is no tags' '
cd "$D" &&
mkdir notags &&
cd notags &&
git init &&
git fetch -t ..
'
test_expect_success 'fetch following tags' '
cd "$D" &&
git tag -a -m "annotated" anno HEAD &&
git tag light HEAD &&
mkdir four &&
cd four &&
git init &&
git fetch .. :track &&
git show-ref --verify refs/tags/anno &&
git show-ref --verify refs/tags/light
'
test_expect_success 'fetch uses remote ref names to describe new refs' '
cd "$D" &&
git init descriptive &&
(
cd descriptive &&
git config remote.o.url .. &&
git config remote.o.fetch "refs/heads/*:refs/crazyheads/*" &&
git config --add remote.o.fetch "refs/others/*:refs/heads/*" &&
git fetch o
) &&
git tag -a -m "Descriptive tag" descriptive-tag &&
git branch descriptive-branch &&
git checkout descriptive-branch &&
echo "Nuts" >crazy &&
git add crazy &&
git commit -a -m "descriptive commit" &&
git update-ref refs/others/crazy HEAD &&
(
cd descriptive &&
git fetch o 2>actual &&
test_i18ngrep "new branch.* -> refs/crazyheads/descriptive-branch$" actual &&
test_i18ngrep "new tag.* -> descriptive-tag$" actual &&
test_i18ngrep "new ref.* -> crazy$" actual
) &&
git checkout main
'
Sane use of test_expect_failure Originally, test_expect_failure was designed to be the opposite of test_expect_success, but this was a bad decision. Most tests run a series of commands that leads to the single command that needs to be tested, like this: test_expect_{success,failure} 'test title' ' setup1 && setup2 && setup3 && what is to be tested ' And expecting a failure exit from the whole sequence misses the point of writing tests. Your setup$N that are supposed to succeed may have failed without even reaching what you are trying to test. The only valid use of test_expect_failure is to check a trivial single command that is expected to fail, which is a minority in tests of Porcelain-ish commands. This large-ish patch rewrites all uses of test_expect_failure to use test_expect_success and rewrites the condition of what is tested, like this: test_expect_success 'test title' ' setup1 && setup2 && setup3 && ! this command should fail ' test_expect_failure is redefined to serve as a reminder that that test *should* succeed but due to a known breakage in git it currently does not pass. So if git-foo command should create a file 'bar' but you discovered a bug that it doesn't, you can write a test like this: test_expect_failure 'git-foo should create bar' ' rm -f bar && git foo && test -f bar ' This construct acts similar to test_expect_success, but instead of reporting "ok/FAIL" like test_expect_success does, the outcome is reported as "FIXED/still broken". Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-02-01 09:50:53 +00:00
test_expect_success 'fetch must not resolve short tag name' '
2007-11-11 14:01:48 +00:00
cd "$D" &&
mkdir five &&
cd five &&
git init &&
test_must_fail git fetch .. anno:five
2007-11-11 14:01:48 +00:00
'
test_expect_success 'fetch can now resolve short remote name' '
2007-11-11 14:01:48 +00:00
cd "$D" &&
git update-ref refs/remotes/six/HEAD HEAD &&
2007-11-11 14:01:48 +00:00
mkdir six &&
cd six &&
git init &&
git fetch .. six:six
2007-11-11 14:01:48 +00:00
'
test_expect_success 'create bundle 1' '
cd "$D" &&
echo >file updated again by origin &&
git commit -a -m "tip" &&
git bundle create --version=3 bundle1 main^..main
'
test_expect_success 'header of bundle looks right' '
cat >expect <<-EOF &&
# v3 git bundle
@object-format=$(test_oid algo)
-OID updated by origin
OID refs/heads/main
EOF
sed -e "s/$OID_REGEX/OID/g" -e "5q" "$D"/bundle1 >actual &&
test_cmp expect actual
'
test_expect_success 'create bundle 2' '
cd "$D" &&
git bundle create bundle2 main~2..main
'
Sane use of test_expect_failure Originally, test_expect_failure was designed to be the opposite of test_expect_success, but this was a bad decision. Most tests run a series of commands that leads to the single command that needs to be tested, like this: test_expect_{success,failure} 'test title' ' setup1 && setup2 && setup3 && what is to be tested ' And expecting a failure exit from the whole sequence misses the point of writing tests. Your setup$N that are supposed to succeed may have failed without even reaching what you are trying to test. The only valid use of test_expect_failure is to check a trivial single command that is expected to fail, which is a minority in tests of Porcelain-ish commands. This large-ish patch rewrites all uses of test_expect_failure to use test_expect_success and rewrites the condition of what is tested, like this: test_expect_success 'test title' ' setup1 && setup2 && setup3 && ! this command should fail ' test_expect_failure is redefined to serve as a reminder that that test *should* succeed but due to a known breakage in git it currently does not pass. So if git-foo command should create a file 'bar' but you discovered a bug that it doesn't, you can write a test like this: test_expect_failure 'git-foo should create bar' ' rm -f bar && git foo && test -f bar ' This construct acts similar to test_expect_success, but instead of reporting "ok/FAIL" like test_expect_success does, the outcome is reported as "FIXED/still broken". Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-02-01 09:50:53 +00:00
test_expect_success 'unbundle 1' '
cd "$D/bundle" &&
git checkout -b some-branch &&
test_must_fail git fetch "$D/bundle1" main:main
'
test_expect_success 'bundle 1 has only 3 files ' '
cd "$D" &&
test_bundle_object_count bundle1 3
'
test_expect_success 'unbundle 2' '
cd "$D/bundle" &&
git fetch ../bundle2 main:main &&
test "tip" = "$(git log -1 --pretty=oneline main | cut -d" " -f2)"
'
test_expect_success 'bundle does not prerequisite objects' '
cd "$D" &&
touch file2 &&
git add file2 &&
git commit -m add.file2 file2 &&
git bundle create bundle3 -1 HEAD &&
test_bundle_object_count bundle3 3
'
test_expect_success 'bundle should be able to create a full history' '
cd "$D" &&
git tag -a -m "1.0" v1.0 main &&
git bundle create bundle4 v1.0
'
test_expect_success 'fetch with a non-applying branch.<name>.merge' '
git config branch.main.remote yeti &&
git config branch.main.merge refs/heads/bigfoot &&
git config remote.blub.url one &&
git config remote.blub.fetch "refs/heads/*:refs/remotes/one/*" &&
git fetch blub
'
# URL supplied to fetch does not match the url of the configured branch's remote
test_expect_success 'fetch from GIT URL with a non-applying branch.<name>.merge [1]' '
one_head=$(cd one && git rev-parse HEAD) &&
this_head=$(git rev-parse HEAD) &&
git update-ref -d FETCH_HEAD &&
git fetch one &&
test $one_head = "$(git rev-parse --verify FETCH_HEAD)" &&
test $this_head = "$(git rev-parse --verify HEAD)"
'
# URL supplied to fetch matches the url of the configured branch's remote and
# the merge spec matches the branch the remote HEAD points to
test_expect_success 'fetch from GIT URL with a non-applying branch.<name>.merge [2]' '
one_ref=$(cd one && git symbolic-ref HEAD) &&
git config branch.main.remote blub &&
git config branch.main.merge "$one_ref" &&
git update-ref -d FETCH_HEAD &&
git fetch one &&
test $one_head = "$(git rev-parse --verify FETCH_HEAD)" &&
test $this_head = "$(git rev-parse --verify HEAD)"
'
# URL supplied to fetch matches the url of the configured branch's remote, but
# the merge spec does not match the branch the remote HEAD points to
test_expect_success 'fetch from GIT URL with a non-applying branch.<name>.merge [3]' '
git config branch.main.merge "${one_ref}_not" &&
git update-ref -d FETCH_HEAD &&
git fetch one &&
test $one_head = "$(git rev-parse --verify FETCH_HEAD)" &&
test $this_head = "$(git rev-parse --verify HEAD)"
'
# the strange name is: a\!'b
test_expect_success 'quoting of a strangely named repo' '
test_must_fail git fetch "a\\!'\''b" > result 2>&1 &&
grep "fatal: '\''a\\\\!'\''b'\''" result
'
test_expect_success 'bundle should record HEAD correctly' '
cd "$D" &&
git bundle create bundle5 HEAD main &&
git bundle list-heads bundle5 >actual &&
for h in HEAD refs/heads/main
do
echo "$(git rev-parse --verify $h) $h" || return 1
done >expect &&
test_cmp expect actual
'
test_expect_success 'mark initial state of origin/main' '
(
cd three &&
git tag base-origin-main refs/remotes/origin/main
)
'
test_expect_success 'explicit fetch should update tracking' '
cd "$D" &&
git branch -f side &&
(
cd three &&
git update-ref refs/remotes/origin/main base-origin-main &&
o=$(git rev-parse --verify refs/remotes/origin/main) &&
git fetch origin main &&
n=$(git rev-parse --verify refs/remotes/origin/main) &&
test "$o" != "$n" &&
test_must_fail git rev-parse --verify refs/remotes/origin/side
)
'
test_expect_success 'explicit pull should update tracking' '
cd "$D" &&
git branch -f side &&
(
cd three &&
git update-ref refs/remotes/origin/main base-origin-main &&
o=$(git rev-parse --verify refs/remotes/origin/main) &&
git pull origin main &&
n=$(git rev-parse --verify refs/remotes/origin/main) &&
test "$o" != "$n" &&
test_must_fail git rev-parse --verify refs/remotes/origin/side
)
'
test_expect_success 'explicit --refmap is allowed only with command-line refspec' '
cd "$D" &&
(
cd three &&
test_must_fail git fetch --refmap="*:refs/remotes/none/*"
)
'
test_expect_success 'explicit --refmap option overrides remote.*.fetch' '
cd "$D" &&
git branch -f side &&
(
cd three &&
git update-ref refs/remotes/origin/main base-origin-main &&
o=$(git rev-parse --verify refs/remotes/origin/main) &&
git fetch --refmap="refs/heads/*:refs/remotes/other/*" origin main &&
n=$(git rev-parse --verify refs/remotes/origin/main) &&
test "$o" = "$n" &&
test_must_fail git rev-parse --verify refs/remotes/origin/side &&
git rev-parse --verify refs/remotes/other/main
)
'
test_expect_success 'explicitly empty --refmap option disables remote.*.fetch' '
cd "$D" &&
git branch -f side &&
(
cd three &&
git update-ref refs/remotes/origin/main base-origin-main &&
o=$(git rev-parse --verify refs/remotes/origin/main) &&
git fetch --refmap="" origin main &&
n=$(git rev-parse --verify refs/remotes/origin/main) &&
test "$o" = "$n" &&
test_must_fail git rev-parse --verify refs/remotes/origin/side
)
'
test_expect_success 'configured fetch updates tracking' '
cd "$D" &&
git branch -f side &&
(
cd three &&
git update-ref refs/remotes/origin/main base-origin-main &&
o=$(git rev-parse --verify refs/remotes/origin/main) &&
git fetch origin &&
n=$(git rev-parse --verify refs/remotes/origin/main) &&
test "$o" != "$n" &&
git rev-parse --verify refs/remotes/origin/side
)
'
test_expect_success 'non-matching refspecs do not confuse tracking update' '
cd "$D" &&
git update-ref refs/odd/location HEAD &&
(
cd three &&
git update-ref refs/remotes/origin/main base-origin-main &&
git config --add remote.origin.fetch \
refs/odd/location:refs/remotes/origin/odd &&
o=$(git rev-parse --verify refs/remotes/origin/main) &&
git fetch origin main &&
n=$(git rev-parse --verify refs/remotes/origin/main) &&
test "$o" != "$n" &&
test_must_fail git rev-parse --verify refs/remotes/origin/odd
)
'
clean up error conventions of remote.c:match_explicit match_explicit is called for each push refspec to try to fully resolve the source and destination sides of the refspec. Currently, we look at each refspec and report errors on both the source and the dest side before aborting. It makes sense to report errors for each refspec, since an error in one is independent of an error in the other. However, reporting errors on the 'dst' side of a refspec if there has been an error on the 'src' side does not necessarily make sense, since the interpretation of the 'dst' side depends on the 'src' side (for example, when creating a new unqualified remote ref, we use the same type as the src ref). This patch lets match_explicit return early when the src side of the refspec is bogus. We still look at all of the refspecs before aborting the push, though. At the same time, we clean up the call signature, which previously took an extra "errs" flag. This was pointless, as we didn't act on that flag, but rather just passed it back to the caller. Instead, we now use the more traditional "return -1" to signal an error, and the caller aggregates the error count. This change fixes two bugs, as well: - the early return avoids a segfault when passing a NULL matched_src to guess_ref() - the check for multiple sources pointing to a single dest aborted if the "err" flag was set. Presumably the intent was not to bother with the check if we had no matched_src. However, since the err flag was passed in from the caller, we might abort the check just because a previous refspec had a problem, which doesn't make sense. In practice, this didn't matter, since due to the error flag we end up aborting the push anyway. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-06-16 16:15:02 +00:00
test_expect_success 'pushing nonexistent branch by mistake should not segv' '
cd "$D" &&
test_must_fail git push seven no:no
'
test_expect_success 'auto tag following fetches minimum' '
cd "$D" &&
git clone .git follow &&
git checkout HEAD^0 &&
(
for i in 1 2 3 4 5 6 7
do
echo $i >>file &&
git commit -m $i -a &&
git tag -a -m $i excess-$i || exit 1
done
) &&
git checkout main &&
(
cd follow &&
git fetch
)
'
test_expect_success 'refuse to fetch into the current branch' '
test_must_fail git fetch . side:main
'
test_expect_success 'fetch into the current branch with --update-head-ok' '
git fetch --update-head-ok . side:main
'
test_expect_success 'fetch --dry-run does not touch FETCH_HEAD, but still prints what would be written' '
rm -f .git/FETCH_HEAD err &&
git fetch --dry-run . 2>err &&
! test -f .git/FETCH_HEAD &&
grep FETCH_HEAD err
'
test_expect_success '--no-write-fetch-head does not touch FETCH_HEAD, and does not print what would be written' '
rm -f .git/FETCH_HEAD err &&
git fetch --no-write-fetch-head . 2>err &&
! test -f .git/FETCH_HEAD &&
! grep FETCH_HEAD err
'
test_expect_success '--write-fetch-head gets defeated by --dry-run' '
rm -f .git/FETCH_HEAD &&
git fetch --dry-run --write-fetch-head . &&
! test -f .git/FETCH_HEAD
'
test_expect_success "should be able to fetch with duplicate refspecs" '
mkdir dups &&
(
cd dups &&
git init &&
git config branch.main.remote three &&
git config remote.three.url ../three/.git &&
git config remote.three.fetch +refs/heads/*:refs/remotes/origin/* &&
git config --add remote.three.fetch +refs/heads/*:refs/remotes/origin/* &&
git fetch three
)
'
remote: make refspec follow the same disambiguation rule as local refs When matching a non-wildcard LHS of a refspec against a list of refs, find_ref_by_name_abbrev() returns the first ref that matches using any DWIM rules used by refname_match() in refs.c, even if a better match occurs later in the list of refs. This causes unexpected behavior when (for example) fetching using the refspec "refs/heads/s:<something>" from a remote with both "refs/heads/refs/heads/s" and "refs/heads/s"; even if the former was inadvertently created, one would still expect the latter to be fetched. Similarly, when both a tag T and a branch T exist, fetching T should favor the tag, just like how local refname disambiguation rule works. But because the code walks over ls-remote output from the remote, which happens to be sorted in alphabetical order and has refs/heads/T before refs/tags/T, a request to fetch T is (mis)interpreted as fetching refs/heads/T. Update refname_match(), all of whose current callers care only if it returns non-zero (i.e. matches) to see if an abbreviated name can mean the full name being tested, so that it returns a positive integer whose magnitude can be used to tell the precedence, and fix the find_ref_by_name_abbrev() function not to stop at the first match but find the match with the highest precedence. This is based on an earlier work, which special cased only the exact matches, by Jonathan Tan. Helped-by: Jonathan Tan <jonathantanmy@google.com> Helped-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-08-01 16:22:37 +00:00
test_expect_success 'LHS of refspec follows ref disambiguation rules' '
mkdir lhs-ambiguous &&
(
cd lhs-ambiguous &&
git init server &&
test_commit -C server unwanted &&
test_commit -C server wanted &&
git init client &&
# Check a name coming after "refs" alphabetically ...
git -C server update-ref refs/heads/s wanted &&
git -C server update-ref refs/heads/refs/heads/s unwanted &&
git -C client fetch ../server +refs/heads/s:refs/heads/checkthis &&
git -C server rev-parse wanted >expect &&
git -C client rev-parse checkthis >actual &&
test_cmp expect actual &&
# ... and one before.
git -C server update-ref refs/heads/q wanted &&
git -C server update-ref refs/heads/refs/heads/q unwanted &&
git -C client fetch ../server +refs/heads/q:refs/heads/checkthis &&
git -C server rev-parse wanted >expect &&
git -C client rev-parse checkthis >actual &&
test_cmp expect actual &&
# Tags are preferred over branches like refs/{heads,tags}/*
git -C server update-ref refs/tags/t wanted &&
git -C server update-ref refs/heads/t unwanted &&
git -C client fetch ../server +t:refs/heads/checkthis &&
git -C server rev-parse wanted >expect &&
git -C client rev-parse checkthis >actual
)
'
test_expect_success 'fetch.writeCommitGraph' '
git clone three write &&
(
cd three &&
test_commit new
) &&
(
cd write &&
git -c fetch.writeCommitGraph fetch origin &&
test_path_is_file .git/objects/info/commit-graphs/commit-graph-chain
)
'
commit-graph: fix writing first commit-graph during fetch The previous commit includes a failing test for an issue around fetch.writeCommitGraph and fetching in a repo with a submodule. Here, we fix that bug and set the test to "test_expect_success". The problem arises with this set of commands when the remote repo at <url> has a submodule. Note that --recurse-submodules is not needed to demonstrate the bug. $ git clone <url> test $ cd test $ git -c fetch.writeCommitGraph=true fetch origin Computing commit graph generation numbers: 100% (12/12), done. BUG: commit-graph.c:886: missing parent <hash1> for commit <hash2> Aborted (core dumped) As an initial fix, I converted the code in builtin/fetch.c that calls write_commit_graph_reachable() to instead launch a "git commit-graph write --reachable --split" process. That code worked, but is not how we want the feature to work long-term. That test did demonstrate that the issue must be something to do with internal state of the 'git fetch' process. The write_commit_graph() method in commit-graph.c ensures the commits we plan to write are "closed under reachability" using close_reachable(). This method walks from the input commits, and uses the UNINTERESTING flag to mark which commits have already been visited. This allows the walk to take O(N) time, where N is the number of commits, instead of O(P) time, where P is the number of paths. (The number of paths can be exponential in the number of commits.) However, the UNINTERESTING flag is used in lots of places in the codebase. This flag usually means some barrier to stop a commit walk, such as in revision-walking to compare histories. It is not often cleared after the walk completes because the starting points of those walks do not have the UNINTERESTING flag, and clear_commit_marks() would stop immediately. This is happening during a 'git fetch' call with a remote. The fetch negotiation is comparing the remote refs with the local refs and marking some commits as UNINTERESTING. I tested running clear_commit_marks_many() to clear the UNINTERESTING flag inside close_reachable(), but the tips did not have the flag, so that did nothing. It turns out that the calculate_changed_submodule_paths() method is at fault. Thanks, Peff, for pointing out this detail! More specifically, for each submodule, the collect_changed_submodules() runs a revision walk to essentially do file-history on the list of submodules. That revision walk marks commits UNININTERESTING if they are simplified away by not changing the submodule. Instead, I finally arrived on the conclusion that I should use a flag that is not used in any other part of the code. In commit-reach.c, a number of flags were defined for commit walk algorithms. The REACHABLE flag seemed like it made the most sense, and it seems it was not actually used in the file. The REACHABLE flag was used in early versions of commit-reach.c, but was removed by 4fbcca4 (commit-reach: make can_all_from_reach... linear, 2018-07-20). Add the REACHABLE flag to commit-graph.c and use it instead of UNINTERESTING in close_reachable(). This fixes the bug in manual testing. Reported-by: Johannes Schindelin <johannes.schindelin@gmx.de> Helped-by: Jeff King <peff@peff.net> Helped-by: Szeder Gábor <szeder.dev@gmail.com> Signed-off-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-24 13:40:42 +00:00
test_expect_success 'fetch.writeCommitGraph with submodules' '
git clone dups super &&
(
cd super &&
git submodule add "file://$TRASH_DIRECTORY/three" &&
git commit -m "add submodule"
) &&
git clone "super" super-clone &&
(
cd super-clone &&
rm -rf .git/objects/info &&
git -c fetch.writeCommitGraph=true fetch origin &&
test_path_is_file .git/objects/info/commit-graphs/commit-graph-chain
)
'
# configured prune tests
set_config_tristate () {
# var=$1 val=$2
case "$2" in
unset)
test_unconfig "$1"
;;
*)
git config "$1" "$2"
key=$(echo $1 | sed -e 's/^remote\.origin/fetch/')
git_fetch_c="$git_fetch_c -c $key=$2"
;;
esac
}
test_configured_prune () {
test_configured_prune_type "$@" "name"
test_configured_prune_type "$@" "link"
}
test_configured_prune_type () {
fetch_prune=$1
remote_origin_prune=$2
fetch_prune_tags=$3
remote_origin_prune_tags=$4
expected_branch=$5
expected_tag=$6
cmdline=$7
mode=$8
if test -z "$cmdline_setup"
then
test_expect_success 'setup cmdline_setup variable for subsequent test' '
remote_url="file://$(git -C one config remote.origin.url)" &&
remote_fetch="$(git -C one config remote.origin.fetch)" &&
cmdline_setup="\"$remote_url\" \"$remote_fetch\""
'
fi
if test "$mode" = 'link'
then
new_cmdline=""
if test "$cmdline" = ""
then
new_cmdline=$cmdline_setup
else
new_cmdline=$(printf "%s" "$cmdline" | perl -pe 's[origin(?!/)]["'"$remote_url"'"]g')
fi
fetch: add a --prune-tags option and fetch.pruneTags config Add a --prune-tags option to git-fetch, along with fetch.pruneTags config option and a -P shorthand (-p is --prune). This allows for doing any of: git fetch -p -P git fetch --prune --prune-tags git fetch -p -P origin git fetch --prune --prune-tags origin Or simply: git config fetch.prune true && git config fetch.pruneTags true && git fetch Instead of the much more verbose: git fetch --prune origin 'refs/tags/*:refs/tags/*' '+refs/heads/*:refs/remotes/origin/*' Before this feature it was painful to support the use-case of pulling from a repo which is having both its branches *and* tags deleted regularly, and have our local references to reflect upstream. At work we create deployment tags in the repo for each rollout, and there's *lots* of those, so they're archived within weeks for performance reasons. Without this change it's hard to centrally configure such repos in /etc/gitconfig (on servers that are only used for working with them). You need to set fetch.prune=true globally, and then for each repo: git -C {} config --replace-all remote.origin.fetch "refs/tags/*:refs/tags/*" "^\+*refs/tags/\*:refs/tags/\*$" Now I can simply set fetch.pruneTags=true in /etc/gitconfig as well, and users running "git pull" will automatically get the pruning semantics I want. Even though "git remote" has corresponding "prune" and "update --prune" subcommands I'm intentionally not adding a corresponding prune-tags or "update --prune --prune-tags" mode to that command. It's advertised (as noted in my recent "git remote doc: correct dangerous lies about what prune does") as only modifying remote tracking references, whereas any --prune-tags option is always going to modify what from the user's perspective is a local copy of the tag, since there's no such thing as a remote tracking tag. Ideally add_prune_tags_to_fetch_refspec() would be something that would use ALLOC_GROW() to grow the 'fetch` member of the 'remote' struct. Instead I'm realloc-ing remote->fetch and adding the tag_refspec to the end. The reason is that parse_{fetch,push}_refspec which allocate the refspec (ultimately remote->fetch) struct are called many places that don't have access to a 'remote' struct. It would be hard to change all their callsites to be amenable to carry around the bookkeeping variables required for dynamic allocation. All the other callers of the API first incrementally construct the string version of the refspec in remote->fetch_refspec via add_fetch_refspec(), before finally calling parse_fetch_refspec() via some variation of remote_get(). It's less of a pain to deal with the one special case that needs to modify already constructed refspecs than to chase down and change all the other callsites. The API I'm adding is intentionally not generalized because if we add more of these we'd probably want to re-visit how this is done. See my "Re: [BUG] git remote prune removes local tags, depending on fetch config" (87po6ahx87.fsf@evledraar.gmail.com; https://public-inbox.org/git/87po6ahx87.fsf@evledraar.gmail.com/) for more background info. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-02-09 20:32:15 +00:00
if test "$fetch_prune_tags" = 'true' ||
test "$remote_origin_prune_tags" = 'true'
then
if ! printf '%s' "$cmdline\n" | grep -q refs/remotes/origin/
then
new_cmdline="$new_cmdline refs/tags/*:refs/tags/*"
fi
fi
cmdline="$new_cmdline"
fi
test_expect_success "$mode prune fetch.prune=$1 remote.origin.prune=$2 fetch.pruneTags=$3 remote.origin.pruneTags=$4${7:+ $7}; branch:$5 tag:$6" '
# make sure a newbranch is there in . and also in one
git branch -f newbranch &&
git tag -f newtag &&
(
cd one &&
test_unconfig fetch.prune &&
test_unconfig fetch.pruneTags &&
test_unconfig remote.origin.prune &&
test_unconfig remote.origin.pruneTags &&
git fetch '"$cmdline_setup"' &&
git rev-parse --verify refs/remotes/origin/newbranch &&
git rev-parse --verify refs/tags/newtag
) &&
# now remove them
git branch -d newbranch &&
git tag -d newtag &&
# then test
(
cd one &&
git_fetch_c="" &&
set_config_tristate fetch.prune $fetch_prune &&
set_config_tristate fetch.pruneTags $fetch_prune_tags &&
set_config_tristate remote.origin.prune $remote_origin_prune &&
set_config_tristate remote.origin.pruneTags $remote_origin_prune_tags &&
if test "$mode" != "link"
then
git_fetch_c=""
fi &&
git$git_fetch_c fetch '"$cmdline"' &&
case "$expected_branch" in
pruned)
test_must_fail git rev-parse --verify refs/remotes/origin/newbranch
;;
kept)
git rev-parse --verify refs/remotes/origin/newbranch
;;
esac &&
case "$expected_tag" in
pruned)
test_must_fail git rev-parse --verify refs/tags/newtag
;;
kept)
git rev-parse --verify refs/tags/newtag
;;
esac
)
'
}
# $1 config: fetch.prune
# $2 config: remote.<name>.prune
# $3 config: fetch.pruneTags
# $4 config: remote.<name>.pruneTags
# $5 expect: branch to be pruned?
# $6 expect: tag to be pruned?
# $7 git-fetch $cmdline:
#
# $1 $2 $3 $4 $5 $6 $7
test_configured_prune unset unset unset unset kept kept ""
test_configured_prune unset unset unset unset kept kept "--no-prune"
test_configured_prune unset unset unset unset pruned kept "--prune"
test_configured_prune unset unset unset unset kept pruned \
"--prune origin refs/tags/*:refs/tags/*"
test_configured_prune unset unset unset unset pruned pruned \
"--prune origin refs/tags/*:refs/tags/* +refs/heads/*:refs/remotes/origin/*"
test_configured_prune false unset unset unset kept kept ""
test_configured_prune false unset unset unset kept kept "--no-prune"
test_configured_prune false unset unset unset pruned kept "--prune"
test_configured_prune true unset unset unset pruned kept ""
test_configured_prune true unset unset unset pruned kept "--prune"
test_configured_prune true unset unset unset kept kept "--no-prune"
test_configured_prune unset false unset unset kept kept ""
test_configured_prune unset false unset unset kept kept "--no-prune"
test_configured_prune unset false unset unset pruned kept "--prune"
test_configured_prune false false unset unset kept kept ""
test_configured_prune false false unset unset kept kept "--no-prune"
test_configured_prune false false unset unset pruned kept "--prune"
test_configured_prune false false unset unset kept pruned \
"--prune origin refs/tags/*:refs/tags/*"
test_configured_prune false false unset unset pruned pruned \
"--prune origin refs/tags/*:refs/tags/* +refs/heads/*:refs/remotes/origin/*"
test_configured_prune true false unset unset kept kept ""
test_configured_prune true false unset unset pruned kept "--prune"
test_configured_prune true false unset unset kept kept "--no-prune"
test_configured_prune unset true unset unset pruned kept ""
test_configured_prune unset true unset unset kept kept "--no-prune"
test_configured_prune unset true unset unset pruned kept "--prune"
test_configured_prune false true unset unset pruned kept ""
test_configured_prune false true unset unset kept kept "--no-prune"
test_configured_prune false true unset unset pruned kept "--prune"
test_configured_prune true true unset unset pruned kept ""
test_configured_prune true true unset unset pruned kept "--prune"
test_configured_prune true true unset unset kept kept "--no-prune"
test_configured_prune true true unset unset kept pruned \
"--prune origin refs/tags/*:refs/tags/*"
test_configured_prune true true unset unset pruned pruned \
"--prune origin refs/tags/*:refs/tags/* +refs/heads/*:refs/remotes/origin/*"
fetch: add a --prune-tags option and fetch.pruneTags config Add a --prune-tags option to git-fetch, along with fetch.pruneTags config option and a -P shorthand (-p is --prune). This allows for doing any of: git fetch -p -P git fetch --prune --prune-tags git fetch -p -P origin git fetch --prune --prune-tags origin Or simply: git config fetch.prune true && git config fetch.pruneTags true && git fetch Instead of the much more verbose: git fetch --prune origin 'refs/tags/*:refs/tags/*' '+refs/heads/*:refs/remotes/origin/*' Before this feature it was painful to support the use-case of pulling from a repo which is having both its branches *and* tags deleted regularly, and have our local references to reflect upstream. At work we create deployment tags in the repo for each rollout, and there's *lots* of those, so they're archived within weeks for performance reasons. Without this change it's hard to centrally configure such repos in /etc/gitconfig (on servers that are only used for working with them). You need to set fetch.prune=true globally, and then for each repo: git -C {} config --replace-all remote.origin.fetch "refs/tags/*:refs/tags/*" "^\+*refs/tags/\*:refs/tags/\*$" Now I can simply set fetch.pruneTags=true in /etc/gitconfig as well, and users running "git pull" will automatically get the pruning semantics I want. Even though "git remote" has corresponding "prune" and "update --prune" subcommands I'm intentionally not adding a corresponding prune-tags or "update --prune --prune-tags" mode to that command. It's advertised (as noted in my recent "git remote doc: correct dangerous lies about what prune does") as only modifying remote tracking references, whereas any --prune-tags option is always going to modify what from the user's perspective is a local copy of the tag, since there's no such thing as a remote tracking tag. Ideally add_prune_tags_to_fetch_refspec() would be something that would use ALLOC_GROW() to grow the 'fetch` member of the 'remote' struct. Instead I'm realloc-ing remote->fetch and adding the tag_refspec to the end. The reason is that parse_{fetch,push}_refspec which allocate the refspec (ultimately remote->fetch) struct are called many places that don't have access to a 'remote' struct. It would be hard to change all their callsites to be amenable to carry around the bookkeeping variables required for dynamic allocation. All the other callers of the API first incrementally construct the string version of the refspec in remote->fetch_refspec via add_fetch_refspec(), before finally calling parse_fetch_refspec() via some variation of remote_get(). It's less of a pain to deal with the one special case that needs to modify already constructed refspecs than to chase down and change all the other callsites. The API I'm adding is intentionally not generalized because if we add more of these we'd probably want to re-visit how this is done. See my "Re: [BUG] git remote prune removes local tags, depending on fetch config" (87po6ahx87.fsf@evledraar.gmail.com; https://public-inbox.org/git/87po6ahx87.fsf@evledraar.gmail.com/) for more background info. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-02-09 20:32:15 +00:00
# --prune-tags on its own does nothing, needs --prune as well, same
# for fetch.pruneTags without fetch.prune
fetch: add a --prune-tags option and fetch.pruneTags config Add a --prune-tags option to git-fetch, along with fetch.pruneTags config option and a -P shorthand (-p is --prune). This allows for doing any of: git fetch -p -P git fetch --prune --prune-tags git fetch -p -P origin git fetch --prune --prune-tags origin Or simply: git config fetch.prune true && git config fetch.pruneTags true && git fetch Instead of the much more verbose: git fetch --prune origin 'refs/tags/*:refs/tags/*' '+refs/heads/*:refs/remotes/origin/*' Before this feature it was painful to support the use-case of pulling from a repo which is having both its branches *and* tags deleted regularly, and have our local references to reflect upstream. At work we create deployment tags in the repo for each rollout, and there's *lots* of those, so they're archived within weeks for performance reasons. Without this change it's hard to centrally configure such repos in /etc/gitconfig (on servers that are only used for working with them). You need to set fetch.prune=true globally, and then for each repo: git -C {} config --replace-all remote.origin.fetch "refs/tags/*:refs/tags/*" "^\+*refs/tags/\*:refs/tags/\*$" Now I can simply set fetch.pruneTags=true in /etc/gitconfig as well, and users running "git pull" will automatically get the pruning semantics I want. Even though "git remote" has corresponding "prune" and "update --prune" subcommands I'm intentionally not adding a corresponding prune-tags or "update --prune --prune-tags" mode to that command. It's advertised (as noted in my recent "git remote doc: correct dangerous lies about what prune does") as only modifying remote tracking references, whereas any --prune-tags option is always going to modify what from the user's perspective is a local copy of the tag, since there's no such thing as a remote tracking tag. Ideally add_prune_tags_to_fetch_refspec() would be something that would use ALLOC_GROW() to grow the 'fetch` member of the 'remote' struct. Instead I'm realloc-ing remote->fetch and adding the tag_refspec to the end. The reason is that parse_{fetch,push}_refspec which allocate the refspec (ultimately remote->fetch) struct are called many places that don't have access to a 'remote' struct. It would be hard to change all their callsites to be amenable to carry around the bookkeeping variables required for dynamic allocation. All the other callers of the API first incrementally construct the string version of the refspec in remote->fetch_refspec via add_fetch_refspec(), before finally calling parse_fetch_refspec() via some variation of remote_get(). It's less of a pain to deal with the one special case that needs to modify already constructed refspecs than to chase down and change all the other callsites. The API I'm adding is intentionally not generalized because if we add more of these we'd probably want to re-visit how this is done. See my "Re: [BUG] git remote prune removes local tags, depending on fetch config" (87po6ahx87.fsf@evledraar.gmail.com; https://public-inbox.org/git/87po6ahx87.fsf@evledraar.gmail.com/) for more background info. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-02-09 20:32:15 +00:00
test_configured_prune unset unset unset unset kept kept "--prune-tags"
test_configured_prune unset unset true unset kept kept ""
test_configured_prune unset unset unset true kept kept ""
# These will prune the tags
test_configured_prune unset unset unset unset pruned pruned "--prune --prune-tags"
test_configured_prune true unset true unset pruned pruned ""
test_configured_prune unset true unset true pruned pruned ""
# remote.<name>.pruneTags overrides fetch.pruneTags, just like
# remote.<name>.prune overrides fetch.prune if set.
test_configured_prune true unset true unset pruned pruned ""
test_configured_prune false true false true pruned pruned ""
test_configured_prune true false true false kept kept ""
# When --prune-tags is supplied it's ignored if an explicit refspec is
# given, same for the configuration options.
test_configured_prune unset unset unset unset pruned kept \
"--prune --prune-tags origin +refs/heads/*:refs/remotes/origin/*"
test_configured_prune unset unset true unset pruned kept \
"--prune origin +refs/heads/*:refs/remotes/origin/*"
test_configured_prune unset unset unset true pruned kept \
"--prune origin +refs/heads/*:refs/remotes/origin/*"
# Pruning that also takes place if a file:// url replaces a named
# remote. However, because there's no implicit
fetch: add a --prune-tags option and fetch.pruneTags config Add a --prune-tags option to git-fetch, along with fetch.pruneTags config option and a -P shorthand (-p is --prune). This allows for doing any of: git fetch -p -P git fetch --prune --prune-tags git fetch -p -P origin git fetch --prune --prune-tags origin Or simply: git config fetch.prune true && git config fetch.pruneTags true && git fetch Instead of the much more verbose: git fetch --prune origin 'refs/tags/*:refs/tags/*' '+refs/heads/*:refs/remotes/origin/*' Before this feature it was painful to support the use-case of pulling from a repo which is having both its branches *and* tags deleted regularly, and have our local references to reflect upstream. At work we create deployment tags in the repo for each rollout, and there's *lots* of those, so they're archived within weeks for performance reasons. Without this change it's hard to centrally configure such repos in /etc/gitconfig (on servers that are only used for working with them). You need to set fetch.prune=true globally, and then for each repo: git -C {} config --replace-all remote.origin.fetch "refs/tags/*:refs/tags/*" "^\+*refs/tags/\*:refs/tags/\*$" Now I can simply set fetch.pruneTags=true in /etc/gitconfig as well, and users running "git pull" will automatically get the pruning semantics I want. Even though "git remote" has corresponding "prune" and "update --prune" subcommands I'm intentionally not adding a corresponding prune-tags or "update --prune --prune-tags" mode to that command. It's advertised (as noted in my recent "git remote doc: correct dangerous lies about what prune does") as only modifying remote tracking references, whereas any --prune-tags option is always going to modify what from the user's perspective is a local copy of the tag, since there's no such thing as a remote tracking tag. Ideally add_prune_tags_to_fetch_refspec() would be something that would use ALLOC_GROW() to grow the 'fetch` member of the 'remote' struct. Instead I'm realloc-ing remote->fetch and adding the tag_refspec to the end. The reason is that parse_{fetch,push}_refspec which allocate the refspec (ultimately remote->fetch) struct are called many places that don't have access to a 'remote' struct. It would be hard to change all their callsites to be amenable to carry around the bookkeeping variables required for dynamic allocation. All the other callers of the API first incrementally construct the string version of the refspec in remote->fetch_refspec via add_fetch_refspec(), before finally calling parse_fetch_refspec() via some variation of remote_get(). It's less of a pain to deal with the one special case that needs to modify already constructed refspecs than to chase down and change all the other callsites. The API I'm adding is intentionally not generalized because if we add more of these we'd probably want to re-visit how this is done. See my "Re: [BUG] git remote prune removes local tags, depending on fetch config" (87po6ahx87.fsf@evledraar.gmail.com; https://public-inbox.org/git/87po6ahx87.fsf@evledraar.gmail.com/) for more background info. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-02-09 20:32:15 +00:00
# +refs/heads/*:refs/remotes/origin/* refspec and supplying it on the
# command-line negates --prune-tags, the branches will not be pruned.
test_configured_prune_type unset unset unset unset kept kept "origin --prune-tags" "name"
test_configured_prune_type unset unset unset unset kept kept "origin --prune-tags" "link"
test_configured_prune_type unset unset unset unset pruned pruned "origin --prune --prune-tags" "name"
test_configured_prune_type unset unset unset unset kept pruned "origin --prune --prune-tags" "link"
fetch: add a --prune-tags option and fetch.pruneTags config Add a --prune-tags option to git-fetch, along with fetch.pruneTags config option and a -P shorthand (-p is --prune). This allows for doing any of: git fetch -p -P git fetch --prune --prune-tags git fetch -p -P origin git fetch --prune --prune-tags origin Or simply: git config fetch.prune true && git config fetch.pruneTags true && git fetch Instead of the much more verbose: git fetch --prune origin 'refs/tags/*:refs/tags/*' '+refs/heads/*:refs/remotes/origin/*' Before this feature it was painful to support the use-case of pulling from a repo which is having both its branches *and* tags deleted regularly, and have our local references to reflect upstream. At work we create deployment tags in the repo for each rollout, and there's *lots* of those, so they're archived within weeks for performance reasons. Without this change it's hard to centrally configure such repos in /etc/gitconfig (on servers that are only used for working with them). You need to set fetch.prune=true globally, and then for each repo: git -C {} config --replace-all remote.origin.fetch "refs/tags/*:refs/tags/*" "^\+*refs/tags/\*:refs/tags/\*$" Now I can simply set fetch.pruneTags=true in /etc/gitconfig as well, and users running "git pull" will automatically get the pruning semantics I want. Even though "git remote" has corresponding "prune" and "update --prune" subcommands I'm intentionally not adding a corresponding prune-tags or "update --prune --prune-tags" mode to that command. It's advertised (as noted in my recent "git remote doc: correct dangerous lies about what prune does") as only modifying remote tracking references, whereas any --prune-tags option is always going to modify what from the user's perspective is a local copy of the tag, since there's no such thing as a remote tracking tag. Ideally add_prune_tags_to_fetch_refspec() would be something that would use ALLOC_GROW() to grow the 'fetch` member of the 'remote' struct. Instead I'm realloc-ing remote->fetch and adding the tag_refspec to the end. The reason is that parse_{fetch,push}_refspec which allocate the refspec (ultimately remote->fetch) struct are called many places that don't have access to a 'remote' struct. It would be hard to change all their callsites to be amenable to carry around the bookkeeping variables required for dynamic allocation. All the other callers of the API first incrementally construct the string version of the refspec in remote->fetch_refspec via add_fetch_refspec(), before finally calling parse_fetch_refspec() via some variation of remote_get(). It's less of a pain to deal with the one special case that needs to modify already constructed refspecs than to chase down and change all the other callsites. The API I'm adding is intentionally not generalized because if we add more of these we'd probably want to re-visit how this is done. See my "Re: [BUG] git remote prune removes local tags, depending on fetch config" (87po6ahx87.fsf@evledraar.gmail.com; https://public-inbox.org/git/87po6ahx87.fsf@evledraar.gmail.com/) for more background info. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-02-09 20:32:15 +00:00
test_configured_prune_type unset unset unset unset pruned pruned "--prune --prune-tags origin" "name"
test_configured_prune_type unset unset unset unset kept pruned "--prune --prune-tags origin" "link"
fetch: add a --prune-tags option and fetch.pruneTags config Add a --prune-tags option to git-fetch, along with fetch.pruneTags config option and a -P shorthand (-p is --prune). This allows for doing any of: git fetch -p -P git fetch --prune --prune-tags git fetch -p -P origin git fetch --prune --prune-tags origin Or simply: git config fetch.prune true && git config fetch.pruneTags true && git fetch Instead of the much more verbose: git fetch --prune origin 'refs/tags/*:refs/tags/*' '+refs/heads/*:refs/remotes/origin/*' Before this feature it was painful to support the use-case of pulling from a repo which is having both its branches *and* tags deleted regularly, and have our local references to reflect upstream. At work we create deployment tags in the repo for each rollout, and there's *lots* of those, so they're archived within weeks for performance reasons. Without this change it's hard to centrally configure such repos in /etc/gitconfig (on servers that are only used for working with them). You need to set fetch.prune=true globally, and then for each repo: git -C {} config --replace-all remote.origin.fetch "refs/tags/*:refs/tags/*" "^\+*refs/tags/\*:refs/tags/\*$" Now I can simply set fetch.pruneTags=true in /etc/gitconfig as well, and users running "git pull" will automatically get the pruning semantics I want. Even though "git remote" has corresponding "prune" and "update --prune" subcommands I'm intentionally not adding a corresponding prune-tags or "update --prune --prune-tags" mode to that command. It's advertised (as noted in my recent "git remote doc: correct dangerous lies about what prune does") as only modifying remote tracking references, whereas any --prune-tags option is always going to modify what from the user's perspective is a local copy of the tag, since there's no such thing as a remote tracking tag. Ideally add_prune_tags_to_fetch_refspec() would be something that would use ALLOC_GROW() to grow the 'fetch` member of the 'remote' struct. Instead I'm realloc-ing remote->fetch and adding the tag_refspec to the end. The reason is that parse_{fetch,push}_refspec which allocate the refspec (ultimately remote->fetch) struct are called many places that don't have access to a 'remote' struct. It would be hard to change all their callsites to be amenable to carry around the bookkeeping variables required for dynamic allocation. All the other callers of the API first incrementally construct the string version of the refspec in remote->fetch_refspec via add_fetch_refspec(), before finally calling parse_fetch_refspec() via some variation of remote_get(). It's less of a pain to deal with the one special case that needs to modify already constructed refspecs than to chase down and change all the other callsites. The API I'm adding is intentionally not generalized because if we add more of these we'd probably want to re-visit how this is done. See my "Re: [BUG] git remote prune removes local tags, depending on fetch config" (87po6ahx87.fsf@evledraar.gmail.com; https://public-inbox.org/git/87po6ahx87.fsf@evledraar.gmail.com/) for more background info. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-02-09 20:32:15 +00:00
test_configured_prune_type unset unset true unset pruned pruned "--prune origin" "name"
test_configured_prune_type unset unset true unset kept pruned "--prune origin" "link"
test_configured_prune_type unset unset unset true pruned pruned "--prune origin" "name"
test_configured_prune_type unset unset unset true kept pruned "--prune origin" "link"
test_configured_prune_type true unset true unset pruned pruned "origin" "name"
test_configured_prune_type true unset true unset kept pruned "origin" "link"
test_configured_prune_type unset true true unset pruned pruned "origin" "name"
test_configured_prune_type unset true true unset kept pruned "origin" "link"
test_configured_prune_type unset true unset true pruned pruned "origin" "name"
test_configured_prune_type unset true unset true kept pruned "origin" "link"
# When all remote.origin.fetch settings are deleted a --prune
# --prune-tags still implicitly supplies refs/tags/*:refs/tags/* so
# tags, but not tracking branches, will be deleted.
fetch: add a --prune-tags option and fetch.pruneTags config Add a --prune-tags option to git-fetch, along with fetch.pruneTags config option and a -P shorthand (-p is --prune). This allows for doing any of: git fetch -p -P git fetch --prune --prune-tags git fetch -p -P origin git fetch --prune --prune-tags origin Or simply: git config fetch.prune true && git config fetch.pruneTags true && git fetch Instead of the much more verbose: git fetch --prune origin 'refs/tags/*:refs/tags/*' '+refs/heads/*:refs/remotes/origin/*' Before this feature it was painful to support the use-case of pulling from a repo which is having both its branches *and* tags deleted regularly, and have our local references to reflect upstream. At work we create deployment tags in the repo for each rollout, and there's *lots* of those, so they're archived within weeks for performance reasons. Without this change it's hard to centrally configure such repos in /etc/gitconfig (on servers that are only used for working with them). You need to set fetch.prune=true globally, and then for each repo: git -C {} config --replace-all remote.origin.fetch "refs/tags/*:refs/tags/*" "^\+*refs/tags/\*:refs/tags/\*$" Now I can simply set fetch.pruneTags=true in /etc/gitconfig as well, and users running "git pull" will automatically get the pruning semantics I want. Even though "git remote" has corresponding "prune" and "update --prune" subcommands I'm intentionally not adding a corresponding prune-tags or "update --prune --prune-tags" mode to that command. It's advertised (as noted in my recent "git remote doc: correct dangerous lies about what prune does") as only modifying remote tracking references, whereas any --prune-tags option is always going to modify what from the user's perspective is a local copy of the tag, since there's no such thing as a remote tracking tag. Ideally add_prune_tags_to_fetch_refspec() would be something that would use ALLOC_GROW() to grow the 'fetch` member of the 'remote' struct. Instead I'm realloc-ing remote->fetch and adding the tag_refspec to the end. The reason is that parse_{fetch,push}_refspec which allocate the refspec (ultimately remote->fetch) struct are called many places that don't have access to a 'remote' struct. It would be hard to change all their callsites to be amenable to carry around the bookkeeping variables required for dynamic allocation. All the other callers of the API first incrementally construct the string version of the refspec in remote->fetch_refspec via add_fetch_refspec(), before finally calling parse_fetch_refspec() via some variation of remote_get(). It's less of a pain to deal with the one special case that needs to modify already constructed refspecs than to chase down and change all the other callsites. The API I'm adding is intentionally not generalized because if we add more of these we'd probably want to re-visit how this is done. See my "Re: [BUG] git remote prune removes local tags, depending on fetch config" (87po6ahx87.fsf@evledraar.gmail.com; https://public-inbox.org/git/87po6ahx87.fsf@evledraar.gmail.com/) for more background info. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-02-09 20:32:15 +00:00
test_expect_success 'remove remote.origin.fetch "one"' '
(
cd one &&
git config --unset-all remote.origin.fetch
)
'
test_configured_prune_type unset unset unset unset kept pruned "origin --prune --prune-tags" "name"
test_configured_prune_type unset unset unset unset kept pruned "origin --prune --prune-tags" "link"
test_expect_success 'all boundary commits are excluded' '
test_commit base &&
test_commit oneside &&
git checkout HEAD^ &&
test_commit otherside &&
git checkout main &&
test_tick &&
git merge otherside &&
ad=$(git log --no-walk --format=%ad HEAD) &&
git bundle create twoside-boundary.bdl main --since="$ad" &&
test_bundle_object_count --thin twoside-boundary.bdl 3
'
test_expect_success 'fetch --prune prints the remotes url' '
git branch goodbye &&
git clone . only-prunes &&
git branch -D goodbye &&
(
cd only-prunes &&
git fetch --prune origin 2>&1 | head -n1 >../actual
) &&
echo "From ${D}/." >expect &&
test_cmp expect actual
'
test_expect_success 'branchname D/F conflict resolved by --prune' '
git branch dir/file &&
git clone . prune-df-conflict &&
git branch -D dir/file &&
git branch dir &&
(
cd prune-df-conflict &&
git fetch --prune &&
git rev-parse origin/dir >../actual
) &&
git rev-parse dir >expect &&
test_cmp expect actual
'
fetch-pack: do not filter out one-level refs Currently fetching a one-level ref like "refs/foo" does not work consistently. The outer "git fetch" program filters the list of refs, checking each against check_refname_format. Then it feeds the result to do_fetch_pack to actually negotiate the haves/wants and get the pack. The fetch-pack code does its own filter, and it behaves differently. The fetch-pack filter looks for refs in "refs/", and then feeds everything _after_ the slash (i.e., just "foo") into check_refname_format. But check_refname_format is not designed to look at a partial refname. It complains that the ref has only one component, thinking it is at the root (i.e., alongside "HEAD"), when in reality we just fed it a partial refname. As a result, we omit a ref like "refs/foo" from the pack request, even though "git fetch" then tries to store the resulting ref. If we happen to get the object anyway (e.g., because the ref is contained in another ref we are fetching), then the fetch succeeds. But if it is a unique object, we fail when trying to update "refs/foo". We can fix this by just passing the whole refname into check_refname_format; we know the part we were omitting is "refs/", which is acceptable in a refname. This at least makes the checks consistent with each other. This problem happens most commonly with "refs/stash", which is the only one-level ref in wide use. However, our test does not use "refs/stash", as we may later want to restrict it specifically (not because it is one-level, but because of the semantics of stashes). We may also want to do away with the multiple levels of filtering (which can cause problems when they are out of sync), or even forbid one-level refs entirely. However, those decisions can come later; this fixes the most immediate problem, which is the mismatch between the two. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-01-15 10:46:13 +00:00
test_expect_success 'fetching a one-level ref works' '
test_commit extra &&
git reset --hard HEAD^ &&
git update-ref refs/foo extra &&
git init one-level &&
(
cd one-level &&
git fetch .. HEAD refs/foo
)
'
test_expect_success 'fetching with auto-gc does not lock up' '
write_script askyesno <<-\EOF &&
echo "$*" &&
false
EOF
git clone "file://$D" auto-gc &&
test_commit test2 &&
(
cd auto-gc &&
git config fetch.unpackLimit 1 &&
git config gc.autoPackLimit 1 &&
git config gc.autoDetach false &&
GIT_ASK_YESNO="$D/askyesno" git fetch --verbose >fetch.out 2>&1 &&
test_i18ngrep "Auto packing the repository" fetch.out &&
! grep "Should I try again" fetch.out
)
'
test_expect_success 'fetch aligned output' '
git clone . full-output &&
test_commit looooooooooooong-tag &&
(
cd full-output &&
git -c fetch.output=full fetch origin >actual 2>&1 &&
grep -e "->" actual | cut -c 22- >../actual
) &&
cat >expect <<-\EOF &&
main -> origin/main
looooooooooooong-tag -> looooooooooooong-tag
EOF
test_cmp expect actual
'
test_expect_success 'fetch compact output' '
git clone . compact &&
test_commit extraaa &&
(
cd compact &&
git -c fetch.output=compact fetch origin >actual 2>&1 &&
grep -e "->" actual | cut -c 22- >../actual
) &&
cat >expect <<-\EOF &&
main -> origin/*
extraaa -> *
EOF
test_cmp expect actual
'
test_expect_success '--no-show-forced-updates' '
mkdir forced-updates &&
(
cd forced-updates &&
git init &&
test_commit 1 &&
test_commit 2
) &&
git clone forced-updates forced-update-clone &&
git clone forced-updates no-forced-update-clone &&
git -C forced-updates reset --hard HEAD~1 &&
(
cd forced-update-clone &&
git fetch --show-forced-updates origin 2>output &&
test_i18ngrep "(forced update)" output
) &&
(
cd no-forced-update-clone &&
git fetch --no-show-forced-updates origin 2>output &&
test_i18ngrep ! "(forced update)" output
)
'
setup_negotiation_tip () {
SERVER="$1"
URL="$2"
USE_PROTOCOL_V2="$3"
rm -rf "$SERVER" client trace &&
git init -b main "$SERVER" &&
test_commit -C "$SERVER" alpha_1 &&
test_commit -C "$SERVER" alpha_2 &&
git -C "$SERVER" checkout --orphan beta &&
test_commit -C "$SERVER" beta_1 &&
test_commit -C "$SERVER" beta_2 &&
git clone "$URL" client &&
if test "$USE_PROTOCOL_V2" -eq 1
then
git -C "$SERVER" config protocol.version 2 &&
git -C client config protocol.version 2
fi &&
test_commit -C "$SERVER" beta_s &&
git -C "$SERVER" checkout main &&
test_commit -C "$SERVER" alpha_s &&
git -C "$SERVER" tag -d alpha_1 alpha_2 beta_1 beta_2
}
check_negotiation_tip () {
# Ensure that {alpha,beta}_1 are sent as "have", but not {alpha_beta}_2
ALPHA_1=$(git -C client rev-parse alpha_1) &&
grep "fetch> have $ALPHA_1" trace &&
BETA_1=$(git -C client rev-parse beta_1) &&
grep "fetch> have $BETA_1" trace &&
ALPHA_2=$(git -C client rev-parse alpha_2) &&
! grep "fetch> have $ALPHA_2" trace &&
BETA_2=$(git -C client rev-parse beta_2) &&
! grep "fetch> have $BETA_2" trace
}
test_expect_success '--negotiation-tip limits "have" lines sent' '
setup_negotiation_tip server server 0 &&
GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \
--negotiation-tip=alpha_1 --negotiation-tip=beta_1 \
origin alpha_s beta_s &&
check_negotiation_tip
'
test_expect_success '--negotiation-tip understands globs' '
setup_negotiation_tip server server 0 &&
GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \
--negotiation-tip=*_1 \
origin alpha_s beta_s &&
check_negotiation_tip
'
test_expect_success '--negotiation-tip understands abbreviated SHA-1' '
setup_negotiation_tip server server 0 &&
GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \
--negotiation-tip=$(git -C client rev-parse --short alpha_1) \
--negotiation-tip=$(git -C client rev-parse --short beta_1) \
origin alpha_s beta_s &&
check_negotiation_tip
'
test_expect_success '--negotiation-tip rejects missing OIDs' '
setup_negotiation_tip server server 0 &&
test_must_fail git -C client fetch \
--negotiation-tip=alpha_1 \
--negotiation-tip=$(test_oid zero) \
origin alpha_s beta_s 2>err &&
cat >fatal-expect <<-EOF &&
fatal: the object $(test_oid zero) does not exist
EOF
grep fatal: err >fatal-actual &&
test_cmp fatal-expect fatal-actual
'
. "$TEST_DIRECTORY"/lib-httpd.sh
start_httpd
test_expect_success '--negotiation-tip limits "have" lines sent with HTTP protocol v2' '
setup_negotiation_tip "$HTTPD_DOCUMENT_ROOT_PATH/server" \
"$HTTPD_URL/smart/server" 1 &&
GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \
--negotiation-tip=alpha_1 --negotiation-tip=beta_1 \
origin alpha_s beta_s &&
check_negotiation_tip
'
# DO NOT add non-httpd-specific tests here, because the last part of this
# test script is only executed when httpd is available and enabled.
test_done