git/transport-helper.c

1648 lines
42 KiB
C
Raw Normal View History

#include "git-compat-util.h"
#include "transport.h"
#include "quote.h"
#include "run-command.h"
#include "commit.h"
#include "environment.h"
#include "gettext.h"
#include "hex.h"
#include "object-name.h"
#include "repository.h"
#include "remote.h"
#include "string-list.h"
#include "thread-utils.h"
disconnect from remote helpers more gently When git spawns a remote helper program (like git-remote-http), the last thing we do before closing the pipe to the child process is to send a blank line, telling the helper that we are done issuing commands. However, the helper may already have exited, in which case the parent git process will receive SIGPIPE and die. In particular, this can happen with the remote-curl helper when it encounters errors during a push. The helper reports individual errors for each ref back to git-push, and then exits with a non-zero exit code. Depending on the exact timing of the write, the parent process may or may not receive SIGPIPE. This causes intermittent test failure in t5541.8, and is a side effect of 5238cbf (remote-curl: Fix push status report when all branches fail). Before that commit, remote-curl would not send the final blank line to indicate that the list of status lines was complete; it would just exit, closing the pipe. The parent git-push would notice the closed pipe while reading the status report and exit immediately itself, propagating the failing exit code. But post-5238cbf, remote-curl completes the status list before exiting, git-push actually runs to completion, and then it tries to cleanly disconnect the helper, leading to the SIGPIPE race above. This patch drops all error-checking when sending the final "we are about to hang up" blank line to helpers. There is nothing useful for the parent process to do about errors at that point anyway, and certainly failing to send our "we are done with commands" line to a helper that has already exited is not a problem. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-02-23 10:04:34 +00:00
#include "sigchain.h"
#include "strvec.h"
#include "refs.h"
#include "refspec.h"
#include "transport-internal.h"
#include "protocol.h"
transport-helper: re-examine object dir after fetching This patch fixes a bug where fetch over http (or any helper) using the v0 protocol may sometimes fail to auto-follow tags. The bug comes from 61c7711cfe (sha1-file: use loose object cache for quick existence check, 2018-11-12). But to explain why (and why this is the right fix), let's take a step back. After fetching a pack, the object database has changed, but we may still hold in-memory caches that are now out of date. Traditionally this was just the packed_git list, but 61c7711cfe started using a loose-object cache, as well. Usually these caches are invalidated automatically. When an expected object cannot be found, the low-level object lookup routines call reprepare_packed_git(), which re-scans the set of packs (and thanks to some preparatory patches ahead of 61c7711cfe, throws away the loose object cache). But not all calls do this! In some cases we expect that the object might not exist, and pass OBJECT_INFO_QUICK to tell the low-level routines not to bother re-scanning. And the tag auto-following code is one such caller, since we are asking about oids that the other side has (but we might not have locally). To deal with this, we explicitly call reprepare_packed_git() ourselves after fetching a pack; this goes all the way back to 48ec3e5c07 (Incorporate fetched packs in future object traversal, 2008-06-15). But that only helps if we call fetch_pack() in the main fetch process. When we're using a transport helper, it happens in a separate sub-process, and the parent process is left with old values. So this is only a problem with protocols which require a separate helper process (like http). This patch fixes it by teaching the parent process in the transport helper relationship to make that same reprepare call after the helper finishes fetching. You might be left with some lingering questions, like: 1. Why only the v0 protocol, and not v2? It's because in v2 the child helper doesn't actually run fetch_pack(); it merely establishes a tunnel over which the main process can talk to the remote side (so the fetch_pack() and reprepare happen in the main process). 2. Wouldn't we have the same bug even before the 61c7711cfe added the loose object cache? For example, when we store the fetch as a pack locally, wouldn't our packed_git list still be out of date? If we store a pack, everything works because other parts of the fetch process happen to trigger a call to reprepare_packed_git(). In particular, before storing whatever ref was originally requested, we'll make sure we have the pointed-to object, and that call happens without the QUICK flag. So in that case we'll see that we don't know about it, reprepare, and then repeat our lookup. And now we _do_ know about the pack, and further calls with QUICK will find its contents. Whereas when we unpack the result into loose objects, we never get that same invalidation trigger. We didn't have packs before, and we don't after. But when we do the loose object lookup, we find the object. There's no way to realize that we didn't have the object before the pack, and that having it now means things have changed (in theory we could do a superfluous cache lookup to see that it was missing from the old cache; but depending on the tags the other side showed us, we might not even have filled in that part of the cache earlier). 3. Why does the included test use "--depth 1"? This is important because without it, we happen to invalidate the cache as a side effect of other parts of the fetch process. What happens in a non-shallow fetch is something like this: 1. we call find_non_local_tags() once before actually getting the pack, to see if there are any tags we can fill in from what we already have. This fills in the cache (which is obviously missing objects we're about to fetch). 2. before fetching the actual pack, fetch_and_consume_refs() calls check_exist_and_connected(), to see if we even need to fetch a pack at all. This doesn't use QUICK (though arguably it could, as it's purely an optimization). And since it sees there are objects we are indeed missing, that triggers a reprepare_packed_git() call, which throws out the loose object cache. 3. after fetching, now we call find_non_local_tags() again. And since step (2) invalidated our loose object cache, we find the new objects and create the tags. So everything works, but mostly due to luck. Whereas in a fetch with --depth, we skip step 2 entirely, and thus the out-of-date cache is still in place for step 3, giving us the wrong answer. So the test works with a small "--depth 1" fetch, which makes sure that we don't store the pack from the other side, and that we don't trigger the accidental cache invalidation. And of course it forces the use of v0 along with using the http protocol. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2024-01-24 01:00:56 +00:00
#include "packfile.h"
static int debug;
struct helper_data {
const char *name;
struct child_process *helper;
FILE *out;
unsigned fetch : 1,
import : 1,
bidi_import : 1,
export : 1,
option : 1,
push : 1,
connect : 1,
stateless_connect : 1,
signed_tags : 1,
check_connectivity : 1,
no_disconnect_req : 1,
no_private_update : 1,
object_format : 1;
/*
* As an optimization, the transport code may invoke fetch before
* get_refs_list. If this happens, and if the transport helper doesn't
* support connect or stateless_connect, we need to invoke
* get_refs_list ourselves if we haven't already done so. Keep track of
* whether we have invoked get_refs_list.
*/
unsigned get_refs_list_called : 1;
char *export_marks;
char *import_marks;
/* These go from remote name (as in "list") to private name */
struct refspec rs;
/* Transport options for fetch-pack/send-pack (should one of
* those be invoked).
*/
struct git_transport_options transport_options;
};
static void sendline(struct helper_data *helper, struct strbuf *buffer)
{
if (debug)
fprintf(stderr, "Debug: Remote helper: -> %s", buffer->buf);
avoid "write_in_full(fd, buf, len) != len" pattern The return value of write_in_full() is either "-1", or the requested number of bytes[1]. If we make a partial write before seeing an error, we still return -1, not a partial value. This goes back to f6aa66cb95 (write_in_full: really write in full or return error on disk full., 2007-01-11). So checking anything except "was the return value negative" is pointless. And there are a couple of reasons not to do so: 1. It can do a funny signed/unsigned comparison. If your "len" is signed (e.g., a size_t) then the compiler will promote the "-1" to its unsigned variant. This works out for "!= len" (unless you really were trying to write the maximum size_t bytes), but is a bug if you check "< len" (an example of which was fixed recently in config.c). We should avoid promoting the mental model that you need to check the length at all, so that new sites are not tempted to copy us. 2. Checking for a negative value is shorter to type, especially when the length is an expression. 3. Linus says so. In d34cf19b89 (Clean up write_in_full() users, 2007-01-11), right after the write_in_full() semantics were changed, he wrote: I really wish every "write_in_full()" user would just check against "<0" now, but this fixes the nasty and stupid ones. Appeals to authority aside, this makes it clear that writing it this way does not have an intentional benefit. It's a historical curiosity that we never bothered to clean up (and which was undoubtedly cargo-culted into new sites). So let's convert these obviously-correct cases (this includes write_str_in_full(), which is just a wrapper for write_in_full()). [1] A careful reader may notice there is one way that write_in_full() can return a different value. If we ask write() to write N bytes and get a return value that is _larger_ than N, we could return a larger total. But besides the fact that this would imply a totally broken version of write(), it would already invoke undefined behavior. Our internal remaining counter is an unsigned size_t, which means that subtracting too many byte will wrap it around to a very large number. So we'll instantly begin reading off the end of the buffer, trying to write gigabytes (or petabytes) of data. Signed-off-by: Jeff King <peff@peff.net> Reviewed-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-09-13 17:16:03 +00:00
if (write_in_full(helper->helper->in, buffer->buf, buffer->len) < 0)
die_errno(_("full write to remote helper failed"));
}
static int recvline_fh(FILE *helper, struct strbuf *buffer)
{
strbuf_reset(buffer);
if (debug)
fprintf(stderr, "Debug: Remote helper: Waiting...\n");
if (strbuf_getline(buffer, helper) == EOF) {
if (debug)
fprintf(stderr, "Debug: Remote helper quit.\n");
return 1;
}
if (debug)
fprintf(stderr, "Debug: Remote helper: <- %s\n", buffer->buf);
return 0;
}
static int recvline(struct helper_data *helper, struct strbuf *buffer)
{
return recvline_fh(helper->out, buffer);
}
static void write_constant(int fd, const char *str)
{
if (debug)
fprintf(stderr, "Debug: Remote helper: -> %s", str);
avoid "write_in_full(fd, buf, len) != len" pattern The return value of write_in_full() is either "-1", or the requested number of bytes[1]. If we make a partial write before seeing an error, we still return -1, not a partial value. This goes back to f6aa66cb95 (write_in_full: really write in full or return error on disk full., 2007-01-11). So checking anything except "was the return value negative" is pointless. And there are a couple of reasons not to do so: 1. It can do a funny signed/unsigned comparison. If your "len" is signed (e.g., a size_t) then the compiler will promote the "-1" to its unsigned variant. This works out for "!= len" (unless you really were trying to write the maximum size_t bytes), but is a bug if you check "< len" (an example of which was fixed recently in config.c). We should avoid promoting the mental model that you need to check the length at all, so that new sites are not tempted to copy us. 2. Checking for a negative value is shorter to type, especially when the length is an expression. 3. Linus says so. In d34cf19b89 (Clean up write_in_full() users, 2007-01-11), right after the write_in_full() semantics were changed, he wrote: I really wish every "write_in_full()" user would just check against "<0" now, but this fixes the nasty and stupid ones. Appeals to authority aside, this makes it clear that writing it this way does not have an intentional benefit. It's a historical curiosity that we never bothered to clean up (and which was undoubtedly cargo-culted into new sites). So let's convert these obviously-correct cases (this includes write_str_in_full(), which is just a wrapper for write_in_full()). [1] A careful reader may notice there is one way that write_in_full() can return a different value. If we ask write() to write N bytes and get a return value that is _larger_ than N, we could return a larger total. But besides the fact that this would imply a totally broken version of write(), it would already invoke undefined behavior. Our internal remaining counter is an unsigned size_t, which means that subtracting too many byte will wrap it around to a very large number. So we'll instantly begin reading off the end of the buffer, trying to write gigabytes (or petabytes) of data. Signed-off-by: Jeff King <peff@peff.net> Reviewed-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-09-13 17:16:03 +00:00
if (write_in_full(fd, str, strlen(str)) < 0)
die_errno(_("full write to remote helper failed"));
}
Fix sparse warnings Fix warnings from 'make check'. - These files don't include 'builtin.h' causing sparse to complain that cmd_* isn't declared: builtin/clone.c:364, builtin/fetch-pack.c:797, builtin/fmt-merge-msg.c:34, builtin/hash-object.c:78, builtin/merge-index.c:69, builtin/merge-recursive.c:22 builtin/merge-tree.c:341, builtin/mktag.c:156, builtin/notes.c:426 builtin/notes.c:822, builtin/pack-redundant.c:596, builtin/pack-refs.c:10, builtin/patch-id.c:60, builtin/patch-id.c:149, builtin/remote.c:1512, builtin/remote-ext.c:240, builtin/remote-fd.c:53, builtin/reset.c:236, builtin/send-pack.c:384, builtin/unpack-file.c:25, builtin/var.c:75 - These files have symbols which should be marked static since they're only file scope: submodule.c:12, diff.c:631, replace_object.c:92, submodule.c:13, submodule.c:14, trace.c:78, transport.c:195, transport-helper.c:79, unpack-trees.c:19, url.c:3, url.c:18, url.c:104, url.c:117, url.c:123, url.c:129, url.c:136, thread-utils.c:21, thread-utils.c:48 - These files redeclare symbols to be different types: builtin/index-pack.c:210, parse-options.c:564, parse-options.c:571, usage.c:49, usage.c:58, usage.c:63, usage.c:72 - These files use a literal integer 0 when they really should use a NULL pointer: daemon.c:663, fast-import.c:2942, imap-send.c:1072, notes-merge.c:362 While we're in the area, clean up some unused #includes in builtin files (mostly exec_cmd.h). Signed-off-by: Stephen Boyd <bebarino@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-03-22 07:51:05 +00:00
static const char *remove_ext_force(const char *url)
{
if (url) {
const char *colon = strchr(url, ':');
if (colon && colon[1] == ':')
return colon + 2;
}
return url;
}
static void do_take_over(struct transport *transport)
{
struct helper_data *data;
data = (struct helper_data *)transport->data;
transport_take_over(transport, data->helper);
fclose(data->out);
free(data);
}
static void standard_options(struct transport *t);
static struct child_process *get_helper(struct transport *transport)
{
struct helper_data *data = transport->data;
struct strbuf buf = STRBUF_INIT;
struct child_process *helper;
int duped;
int code;
if (data->helper)
return data->helper;
helper = xmalloc(sizeof(*helper));
child_process_init(helper);
helper->in = -1;
helper->out = -1;
helper->err = 0;
strvec_pushf(&helper->args, "remote-%s", data->name);
strvec_push(&helper->args, transport->remote->name);
strvec_push(&helper->args, remove_ext_force(transport->url));
helper->git_cmd = 1;
helper->silent_exec_failure = 1;
remote helpers: avoid blind fall-back to ".git" when setting GIT_DIR To push from or fetch to the current repository, remote helpers need to know what repository that is. Accordingly, Git sets the GIT_DIR environment variable to the path to the current repository when invoking remote helpers. There is a special case it does not handle: "git ls-remote" and "git archive --remote" can be run to inspect a remote repository without being run from any local repository. GIT_DIR is not useful in this scenario: - if we are not in a repository, we don't need to set GIT_DIR to override an existing GIT_DIR value from the environment. If GIT_DIR is present then we would be in a repository if it were valid and would have called die() if it weren't. - not setting GIT_DIR may cause a helper to do the usual discovery walk to find the repository. But we know we're not in one, or we would have found it ourselves. So in the worst case it may expend a little extra effort to try to find a repository and fail (for example, remote-curl would do this to try to find repository-level configuration). So leave GIT_DIR unset in this case. This makes GIT_DIR easier to understand for remote helper authors and makes transport code less of a special case for repository discovery. Noticed using b1ef400e (setup_git_env: avoid blind fall-back to ".git", 2016-10-20) from 'next': $ cd /tmp $ git ls-remote https://kernel.googlesource.com/pub/scm/git/git fatal: BUG: setup_git_env called without repository Helped-by: Jeff King <peff@peff.net> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-02-14 20:36:19 +00:00
if (have_git_dir())
strvec_pushf(&helper->env, "%s=%s",
GIT_DIR_ENVIRONMENT, get_git_dir());
helper->trace2_child_class = helper->args.v[0]; /* "remote-<name>" */
code = start_command(helper);
if (code < 0 && errno == ENOENT)
die(_("unable to find remote helper for '%s'"), data->name);
else if (code != 0)
exit(code);
data->helper = helper;
data->no_disconnect_req = 0;
refspec_init(&data->rs, REFSPEC_FETCH);
/*
* Open the output as FILE* so strbuf_getline_*() family of
* functions can be used.
* Do this with duped fd because fclose() will close the fd,
* and stuff like taking over will require the fd to remain.
*/
duped = dup(helper->out);
if (duped < 0)
die_errno(_("can't dup helper output fd"));
data->out = xfdopen(duped, "r");
write_constant(helper->in, "capabilities\n");
while (1) {
const char *capname, *arg;
int mandatory = 0;
if (recvline(data, &buf))
exit(128);
if (!*buf.buf)
break;
if (*buf.buf == '*') {
capname = buf.buf + 1;
mandatory = 1;
} else
capname = buf.buf;
if (debug)
fprintf(stderr, "Debug: Got cap %s\n", capname);
if (!strcmp(capname, "fetch"))
data->fetch = 1;
else if (!strcmp(capname, "option"))
data->option = 1;
else if (!strcmp(capname, "push"))
data->push = 1;
else if (!strcmp(capname, "import"))
data->import = 1;
else if (!strcmp(capname, "bidi-import"))
data->bidi_import = 1;
else if (!strcmp(capname, "export"))
data->export = 1;
else if (!strcmp(capname, "check-connectivity"))
data->check_connectivity = 1;
else if (skip_prefix(capname, "refspec ", &arg)) {
refspec_append(&data->rs, arg);
} else if (!strcmp(capname, "connect")) {
data->connect = 1;
} else if (!strcmp(capname, "stateless-connect")) {
data->stateless_connect = 1;
} else if (!strcmp(capname, "signed-tags")) {
data->signed_tags = 1;
} else if (skip_prefix(capname, "export-marks ", &arg)) {
data->export_marks = xstrdup(arg);
} else if (skip_prefix(capname, "import-marks ", &arg)) {
data->import_marks = xstrdup(arg);
} else if (starts_with(capname, "no-private-update")) {
data->no_private_update = 1;
} else if (starts_with(capname, "object-format")) {
data->object_format = 1;
} else if (mandatory) {
die(_("unknown mandatory capability %s; this remote "
"helper probably needs newer version of Git"),
capname);
}
}
if (!data->rs.nr && (data->import || data->bidi_import || data->export)) {
warning(_("this remote helper should implement refspec capability"));
}
strbuf_release(&buf);
if (debug)
fprintf(stderr, "Debug: Capabilities complete.\n");
standard_options(transport);
return data->helper;
}
static int disconnect_helper(struct transport *transport)
{
struct helper_data *data = transport->data;
int res = 0;
if (data->helper) {
if (debug)
fprintf(stderr, "Debug: Disconnecting.\n");
if (!data->no_disconnect_req) {
disconnect from remote helpers more gently When git spawns a remote helper program (like git-remote-http), the last thing we do before closing the pipe to the child process is to send a blank line, telling the helper that we are done issuing commands. However, the helper may already have exited, in which case the parent git process will receive SIGPIPE and die. In particular, this can happen with the remote-curl helper when it encounters errors during a push. The helper reports individual errors for each ref back to git-push, and then exits with a non-zero exit code. Depending on the exact timing of the write, the parent process may or may not receive SIGPIPE. This causes intermittent test failure in t5541.8, and is a side effect of 5238cbf (remote-curl: Fix push status report when all branches fail). Before that commit, remote-curl would not send the final blank line to indicate that the list of status lines was complete; it would just exit, closing the pipe. The parent git-push would notice the closed pipe while reading the status report and exit immediately itself, propagating the failing exit code. But post-5238cbf, remote-curl completes the status list before exiting, git-push actually runs to completion, and then it tries to cleanly disconnect the helper, leading to the SIGPIPE race above. This patch drops all error-checking when sending the final "we are about to hang up" blank line to helpers. There is nothing useful for the parent process to do about errors at that point anyway, and certainly failing to send our "we are done with commands" line to a helper that has already exited is not a problem. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-02-23 10:04:34 +00:00
/*
* Ignore write errors; there's nothing we can do,
* since we're about to close the pipe anyway. And the
* most likely error is EPIPE due to the helper dying
* to report an error itself.
*/
sigchain_push(SIGPIPE, SIG_IGN);
xwrite(data->helper->in, "\n", 1);
sigchain_pop(SIGPIPE);
}
close(data->helper->in);
close(data->helper->out);
fclose(data->out);
res = finish_command(data->helper);
FREE_AND_NULL(data->helper);
}
return res;
}
static const char *unsupported_options[] = {
TRANS_OPT_UPLOADPACK,
TRANS_OPT_RECEIVEPACK,
TRANS_OPT_THIN,
TRANS_OPT_KEEP
};
static const char *boolean_options[] = {
TRANS_OPT_THIN,
TRANS_OPT_KEEP,
signed push: teach smart-HTTP to pass "git push --signed" around The "--signed" option received by "git push" is first passed to the transport layer, which the native transport directly uses to notice that a push certificate needs to be sent. When the transport-helper is involved, however, the option needs to be told to the helper with set_helper_option(), and the helper needs to take necessary action. For the smart-HTTP helper, the "necessary action" involves spawning the "git send-pack" subprocess with the "--signed" option. Once the above all gets wired in, the smart-HTTP transport now can use the push certificate mechanism to authenticate its pushes. Add a test that is modeled after tests for the native transport in t5534-push-signed.sh to t5541-http-push-smart.sh. Update the test Apache configuration to pass GNUPGHOME environment variable through. As PassEnv would trigger warnings for an environment variable that is not set, export it from test-lib.sh set to a harmless value when GnuPG is not being used in the tests. Note that the added test is deliberately loose and does not check the nonce in this step. This is because the stateless RPC mode is inevitably flaky and a nonce that comes back in the actual push processing is one issued by a different process; if the two interactions with the server crossed a second boundary, the nonces will not match and such a check will fail. A later patch in the series will work around this shortcoming. Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-09-15 21:59:00 +00:00
TRANS_OPT_FOLLOWTAGS,
fetch, upload-pack: --deepen=N extends shallow boundary by N commits In git-fetch, --depth argument is always relative with the latest remote refs. This makes it a bit difficult to cover this use case, where the user wants to make the shallow history, say 3 levels deeper. It would work if remote refs have not moved yet, but nobody can guarantee that, especially when that use case is performed a couple months after the last clone or "git fetch --depth". Also, modifying shallow boundary using --depth does not work well with clones created by --since or --not. This patch fixes that. A new argument --deepen=<N> will add <N> more (*) parent commits to the current history regardless of where remote refs are. Have/Want negotiation is still respected. So if remote refs move, the server will send two chunks: one between "have" and "want" and another to extend shallow history. In theory, the client could send no "want"s in order to get the second chunk only. But the protocol does not allow that. Either you send no want lines, which means ls-remote; or you have to send at least one want line that carries deep-relative to the server.. The main work was done by Dongcan Jiang. I fixed it up here and there. And of course all the bugs belong to me. (*) We could even support --deepen=<N> where <N> is negative. In that case we can cut some history from the shallow clone. This operation (and --depth=<shorter depth>) does not require interaction with remote side (and more complicated to implement as a result). Helped-by: Duy Nguyen <pclouds@gmail.com> Helped-by: Eric Sunshine <sunshine@sunshineco.com> Helped-by: Junio C Hamano <gitster@pobox.com> Signed-off-by: Dongcan Jiang <dongcan.jiang@gmail.com> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-06-12 10:54:09 +00:00
TRANS_OPT_DEEPEN_RELATIVE
};
static int strbuf_set_helper_option(struct helper_data *data,
struct strbuf *buf)
{
int ret;
sendline(data, buf);
if (recvline(data, buf))
exit(128);
if (!strcmp(buf->buf, "ok"))
ret = 0;
else if (starts_with(buf->buf, "error"))
ret = -1;
else if (!strcmp(buf->buf, "unsupported"))
ret = 1;
else {
warning(_("%s unexpectedly said: '%s'"), data->name, buf->buf);
ret = 1;
}
return ret;
}
static int string_list_set_helper_option(struct helper_data *data,
const char *name,
struct string_list *list)
{
struct strbuf buf = STRBUF_INIT;
int i, ret = 0;
for (i = 0; i < list->nr; i++) {
strbuf_addf(&buf, "option %s ", name);
quote_c_style(list->items[i].string, &buf, NULL, 0);
strbuf_addch(&buf, '\n');
if ((ret = strbuf_set_helper_option(data, &buf)))
break;
strbuf_reset(&buf);
}
strbuf_release(&buf);
return ret;
}
static int set_helper_option(struct transport *transport,
const char *name, const char *value)
{
struct helper_data *data = transport->data;
struct strbuf buf = STRBUF_INIT;
int i, ret, is_bool = 0;
get_helper(transport);
if (!data->option)
return 1;
if (!strcmp(name, "deepen-not"))
return string_list_set_helper_option(data, name,
(struct string_list *)value);
for (i = 0; i < ARRAY_SIZE(unsupported_options); i++) {
if (!strcmp(name, unsupported_options[i]))
return 1;
}
for (i = 0; i < ARRAY_SIZE(boolean_options); i++) {
if (!strcmp(name, boolean_options[i])) {
is_bool = 1;
break;
}
}
strbuf_addf(&buf, "option %s ", name);
if (is_bool)
strbuf_addstr(&buf, value ? "true" : "false");
else
quote_c_style(value, &buf, NULL, 0);
strbuf_addch(&buf, '\n');
ret = strbuf_set_helper_option(data, &buf);
strbuf_release(&buf);
return ret;
}
static void standard_options(struct transport *t)
{
char buf[16];
int v = t->verbose;
set_helper_option(t, "progress", t->progress ? "true" : "false");
xsnprintf(buf, sizeof(buf), "%d", v + 1);
set_helper_option(t, "verbosity", buf);
switch (t->family) {
case TRANSPORT_FAMILY_ALL:
/*
* this is already the default,
* do not break old remote helpers by setting "all" here
*/
break;
case TRANSPORT_FAMILY_IPV4:
set_helper_option(t, "family", "ipv4");
break;
case TRANSPORT_FAMILY_IPV6:
set_helper_option(t, "family", "ipv6");
break;
}
}
static int release_helper(struct transport *transport)
{
int res = 0;
struct helper_data *data = transport->data;
refspec_clear(&data->rs);
res = disconnect_helper(transport);
free(transport->data);
return res;
}
static int fetch_with_fetch(struct transport *transport,
int nr_heads, struct ref **to_fetch)
{
struct helper_data *data = transport->data;
int i;
struct strbuf buf = STRBUF_INIT;
for (i = 0; i < nr_heads; i++) {
const struct ref *posn = to_fetch[i];
if (posn->status & REF_STATUS_UPTODATE)
continue;
strbuf_addf(&buf, "fetch %s %s\n",
oid_to_hex(&posn->old_oid),
transport-helper: do not request symbolic refs to remote helpers A typical remote helper will return a `list` of refs containing a symbolic ref HEAD, pointing to, e.g. refs/heads/master. In the case of a clone, all the refs are being requested through `fetch` or `import`, including the symbolic ref. While this works properly, in some cases of a fetch, like `git fetch url` or `git fetch origin HEAD`, or any fetch command involving a symbolic ref without also fetching the corresponding ref it points to, the fetch command fails with: fatal: bad object 0000000000000000000000000000000000000000 error: <remote> did not send all necessary objects (in the case the remote helper returned '?' values to the `list` command). This is because there is only one ref given to fetch(), and it's not further resolved to something at the end of fetch_with_import(). While this can be somehow handled in the remote helper itself, by adding a refspec for the symbolic ref, and storing an explicit ref in a private namespace, and then handling the `import` for that symbolic ref specifically, very few existing remote helpers are actually doing that. So, instead of requesting the exact list of wanted refs to remote helpers, treat symbolic refs differently and request the ref they point to instead. Then, resolve the symbolic refs values based on the pointed ref. This assumes there is no more than one level of indirection (a symbolic ref doesn't point to another symbolic ref). Signed-off-by: Mike Hommey <mh@glandium.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-01-19 01:35:07 +00:00
posn->symref ? posn->symref : posn->name);
}
strbuf_addch(&buf, '\n');
sendline(data, &buf);
while (1) {
const char *name;
if (recvline(data, &buf))
exit(128);
if (skip_prefix(buf.buf, "lock ", &name)) {
if (transport->pack_lockfiles.nr)
warning(_("%s also locked %s"), data->name, name);
else
string_list_append(&transport->pack_lockfiles,
name);
}
else if (data->check_connectivity &&
data->transport_options.check_self_contained_and_connected &&
!strcmp(buf.buf, "connectivity-ok"))
data->transport_options.self_contained_and_connected = 1;
else if (!buf.len)
break;
else
warning(_("%s unexpectedly said: '%s'"), data->name, buf.buf);
}
strbuf_release(&buf);
transport-helper: re-examine object dir after fetching This patch fixes a bug where fetch over http (or any helper) using the v0 protocol may sometimes fail to auto-follow tags. The bug comes from 61c7711cfe (sha1-file: use loose object cache for quick existence check, 2018-11-12). But to explain why (and why this is the right fix), let's take a step back. After fetching a pack, the object database has changed, but we may still hold in-memory caches that are now out of date. Traditionally this was just the packed_git list, but 61c7711cfe started using a loose-object cache, as well. Usually these caches are invalidated automatically. When an expected object cannot be found, the low-level object lookup routines call reprepare_packed_git(), which re-scans the set of packs (and thanks to some preparatory patches ahead of 61c7711cfe, throws away the loose object cache). But not all calls do this! In some cases we expect that the object might not exist, and pass OBJECT_INFO_QUICK to tell the low-level routines not to bother re-scanning. And the tag auto-following code is one such caller, since we are asking about oids that the other side has (but we might not have locally). To deal with this, we explicitly call reprepare_packed_git() ourselves after fetching a pack; this goes all the way back to 48ec3e5c07 (Incorporate fetched packs in future object traversal, 2008-06-15). But that only helps if we call fetch_pack() in the main fetch process. When we're using a transport helper, it happens in a separate sub-process, and the parent process is left with old values. So this is only a problem with protocols which require a separate helper process (like http). This patch fixes it by teaching the parent process in the transport helper relationship to make that same reprepare call after the helper finishes fetching. You might be left with some lingering questions, like: 1. Why only the v0 protocol, and not v2? It's because in v2 the child helper doesn't actually run fetch_pack(); it merely establishes a tunnel over which the main process can talk to the remote side (so the fetch_pack() and reprepare happen in the main process). 2. Wouldn't we have the same bug even before the 61c7711cfe added the loose object cache? For example, when we store the fetch as a pack locally, wouldn't our packed_git list still be out of date? If we store a pack, everything works because other parts of the fetch process happen to trigger a call to reprepare_packed_git(). In particular, before storing whatever ref was originally requested, we'll make sure we have the pointed-to object, and that call happens without the QUICK flag. So in that case we'll see that we don't know about it, reprepare, and then repeat our lookup. And now we _do_ know about the pack, and further calls with QUICK will find its contents. Whereas when we unpack the result into loose objects, we never get that same invalidation trigger. We didn't have packs before, and we don't after. But when we do the loose object lookup, we find the object. There's no way to realize that we didn't have the object before the pack, and that having it now means things have changed (in theory we could do a superfluous cache lookup to see that it was missing from the old cache; but depending on the tags the other side showed us, we might not even have filled in that part of the cache earlier). 3. Why does the included test use "--depth 1"? This is important because without it, we happen to invalidate the cache as a side effect of other parts of the fetch process. What happens in a non-shallow fetch is something like this: 1. we call find_non_local_tags() once before actually getting the pack, to see if there are any tags we can fill in from what we already have. This fills in the cache (which is obviously missing objects we're about to fetch). 2. before fetching the actual pack, fetch_and_consume_refs() calls check_exist_and_connected(), to see if we even need to fetch a pack at all. This doesn't use QUICK (though arguably it could, as it's purely an optimization). And since it sees there are objects we are indeed missing, that triggers a reprepare_packed_git() call, which throws out the loose object cache. 3. after fetching, now we call find_non_local_tags() again. And since step (2) invalidated our loose object cache, we find the new objects and create the tags. So everything works, but mostly due to luck. Whereas in a fetch with --depth, we skip step 2 entirely, and thus the out-of-date cache is still in place for step 3, giving us the wrong answer. So the test works with a small "--depth 1" fetch, which makes sure that we don't store the pack from the other side, and that we don't trigger the accidental cache invalidation. And of course it forces the use of v0 along with using the http protocol. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2024-01-24 01:00:56 +00:00
reprepare_packed_git(the_repository);
return 0;
}
static int get_importer(struct transport *transport, struct child_process *fastimport)
{
struct child_process *helper = get_helper(transport);
struct helper_data *data = transport->data;
int cat_blob_fd, code;
child_process_init(fastimport);
fastimport->in = xdup(helper->out);
strvec_push(&fastimport->args, "fast-import");
strvec_push(&fastimport->args, "--allow-unsafe-features");
strvec_push(&fastimport->args, debug ? "--stats" : "--quiet");
if (data->bidi_import) {
cat_blob_fd = xdup(helper->in);
strvec_pushf(&fastimport->args, "--cat-blob-fd=%d", cat_blob_fd);
}
fastimport->git_cmd = 1;
code = start_command(fastimport);
return code;
}
static int get_exporter(struct transport *transport,
struct child_process *fastexport,
struct string_list *revlist_args)
{
struct helper_data *data = transport->data;
struct child_process *helper = get_helper(transport);
int i;
child_process_init(fastexport);
/* we need to duplicate helper->in because we want to use it after
* fastexport is done with it. */
fastexport->out = dup(helper->in);
strvec_push(&fastexport->args, "fast-export");
strvec_push(&fastexport->args, "--use-done-feature");
strvec_push(&fastexport->args, data->signed_tags ?
"--signed-tags=verbatim" : "--signed-tags=warn-strip");
if (data->export_marks)
strvec_pushf(&fastexport->args, "--export-marks=%s.tmp", data->export_marks);
if (data->import_marks)
strvec_pushf(&fastexport->args, "--import-marks=%s", data->import_marks);
for (i = 0; i < revlist_args->nr; i++)
strvec_push(&fastexport->args, revlist_args->items[i].string);
fastexport->git_cmd = 1;
return start_command(fastexport);
}
static int fetch_with_import(struct transport *transport,
int nr_heads, struct ref **to_fetch)
{
struct child_process fastimport;
struct helper_data *data = transport->data;
int i;
struct ref *posn;
struct strbuf buf = STRBUF_INIT;
get_helper(transport);
if (get_importer(transport, &fastimport))
die(_("couldn't run fast-import"));
for (i = 0; i < nr_heads; i++) {
posn = to_fetch[i];
if (posn->status & REF_STATUS_UPTODATE)
continue;
transport-helper: do not request symbolic refs to remote helpers A typical remote helper will return a `list` of refs containing a symbolic ref HEAD, pointing to, e.g. refs/heads/master. In the case of a clone, all the refs are being requested through `fetch` or `import`, including the symbolic ref. While this works properly, in some cases of a fetch, like `git fetch url` or `git fetch origin HEAD`, or any fetch command involving a symbolic ref without also fetching the corresponding ref it points to, the fetch command fails with: fatal: bad object 0000000000000000000000000000000000000000 error: <remote> did not send all necessary objects (in the case the remote helper returned '?' values to the `list` command). This is because there is only one ref given to fetch(), and it's not further resolved to something at the end of fetch_with_import(). While this can be somehow handled in the remote helper itself, by adding a refspec for the symbolic ref, and storing an explicit ref in a private namespace, and then handling the `import` for that symbolic ref specifically, very few existing remote helpers are actually doing that. So, instead of requesting the exact list of wanted refs to remote helpers, treat symbolic refs differently and request the ref they point to instead. Then, resolve the symbolic refs values based on the pointed ref. This assumes there is no more than one level of indirection (a symbolic ref doesn't point to another symbolic ref). Signed-off-by: Mike Hommey <mh@glandium.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-01-19 01:35:07 +00:00
strbuf_addf(&buf, "import %s\n",
posn->symref ? posn->symref : posn->name);
sendline(data, &buf);
strbuf_reset(&buf);
}
write_constant(data->helper->in, "\n");
/*
* remote-helpers that advertise the bidi-import capability are required to
* buffer the complete batch of import commands until this newline before
* sending data to fast-import.
* These helpers read back data from fast-import on their stdin, which could
* be mixed with import commands, otherwise.
*/
if (finish_command(&fastimport))
die(_("error while running fast-import"));
/*
* The fast-import stream of a remote helper that advertises
* the "refspec" capability writes to the refs named after the
* right hand side of the first refspec matching each ref we
* were fetching.
*
* (If no "refspec" capability was specified, for historical
* reasons we default to the equivalent of *:*.)
*
* Store the result in to_fetch[i].old_sha1. Callers such
* as "git fetch" can use the value to write feedback to the
* terminal, populate FETCH_HEAD, and determine what new value
* should be written to peer_ref if the update is a
* fast-forward or this is a forced update.
*/
for (i = 0; i < nr_heads; i++) {
transport-helper: do not request symbolic refs to remote helpers A typical remote helper will return a `list` of refs containing a symbolic ref HEAD, pointing to, e.g. refs/heads/master. In the case of a clone, all the refs are being requested through `fetch` or `import`, including the symbolic ref. While this works properly, in some cases of a fetch, like `git fetch url` or `git fetch origin HEAD`, or any fetch command involving a symbolic ref without also fetching the corresponding ref it points to, the fetch command fails with: fatal: bad object 0000000000000000000000000000000000000000 error: <remote> did not send all necessary objects (in the case the remote helper returned '?' values to the `list` command). This is because there is only one ref given to fetch(), and it's not further resolved to something at the end of fetch_with_import(). While this can be somehow handled in the remote helper itself, by adding a refspec for the symbolic ref, and storing an explicit ref in a private namespace, and then handling the `import` for that symbolic ref specifically, very few existing remote helpers are actually doing that. So, instead of requesting the exact list of wanted refs to remote helpers, treat symbolic refs differently and request the ref they point to instead. Then, resolve the symbolic refs values based on the pointed ref. This assumes there is no more than one level of indirection (a symbolic ref doesn't point to another symbolic ref). Signed-off-by: Mike Hommey <mh@glandium.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-01-19 01:35:07 +00:00
char *private, *name;
posn = to_fetch[i];
if (posn->status & REF_STATUS_UPTODATE)
continue;
transport-helper: do not request symbolic refs to remote helpers A typical remote helper will return a `list` of refs containing a symbolic ref HEAD, pointing to, e.g. refs/heads/master. In the case of a clone, all the refs are being requested through `fetch` or `import`, including the symbolic ref. While this works properly, in some cases of a fetch, like `git fetch url` or `git fetch origin HEAD`, or any fetch command involving a symbolic ref without also fetching the corresponding ref it points to, the fetch command fails with: fatal: bad object 0000000000000000000000000000000000000000 error: <remote> did not send all necessary objects (in the case the remote helper returned '?' values to the `list` command). This is because there is only one ref given to fetch(), and it's not further resolved to something at the end of fetch_with_import(). While this can be somehow handled in the remote helper itself, by adding a refspec for the symbolic ref, and storing an explicit ref in a private namespace, and then handling the `import` for that symbolic ref specifically, very few existing remote helpers are actually doing that. So, instead of requesting the exact list of wanted refs to remote helpers, treat symbolic refs differently and request the ref they point to instead. Then, resolve the symbolic refs values based on the pointed ref. This assumes there is no more than one level of indirection (a symbolic ref doesn't point to another symbolic ref). Signed-off-by: Mike Hommey <mh@glandium.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-01-19 01:35:07 +00:00
name = posn->symref ? posn->symref : posn->name;
if (data->rs.nr)
private = apply_refspecs(&data->rs, name);
else
transport-helper: do not request symbolic refs to remote helpers A typical remote helper will return a `list` of refs containing a symbolic ref HEAD, pointing to, e.g. refs/heads/master. In the case of a clone, all the refs are being requested through `fetch` or `import`, including the symbolic ref. While this works properly, in some cases of a fetch, like `git fetch url` or `git fetch origin HEAD`, or any fetch command involving a symbolic ref without also fetching the corresponding ref it points to, the fetch command fails with: fatal: bad object 0000000000000000000000000000000000000000 error: <remote> did not send all necessary objects (in the case the remote helper returned '?' values to the `list` command). This is because there is only one ref given to fetch(), and it's not further resolved to something at the end of fetch_with_import(). While this can be somehow handled in the remote helper itself, by adding a refspec for the symbolic ref, and storing an explicit ref in a private namespace, and then handling the `import` for that symbolic ref specifically, very few existing remote helpers are actually doing that. So, instead of requesting the exact list of wanted refs to remote helpers, treat symbolic refs differently and request the ref they point to instead. Then, resolve the symbolic refs values based on the pointed ref. This assumes there is no more than one level of indirection (a symbolic ref doesn't point to another symbolic ref). Signed-off-by: Mike Hommey <mh@glandium.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-01-19 01:35:07 +00:00
private = xstrdup(name);
if (private) {
if (read_ref(private, &posn->old_oid) < 0)
die(_("could not read ref %s"), private);
free(private);
}
}
strbuf_release(&buf);
return 0;
}
static int run_connect(struct transport *transport, struct strbuf *cmdbuf)
{
struct helper_data *data = transport->data;
int ret = 0;
int duped;
FILE *input;
struct child_process *helper;
helper = get_helper(transport);
/*
* Yes, dup the pipe another time, as we need unbuffered version
* of input pipe as FILE*. fclose() closes the underlying fd and
* stream buffering only can be changed before first I/O operation
* on it.
*/
duped = dup(helper->out);
if (duped < 0)
die_errno(_("can't dup helper output fd"));
input = xfdopen(duped, "r");
setvbuf(input, NULL, _IONBF, 0);
sendline(data, cmdbuf);
if (recvline_fh(input, cmdbuf))
exit(128);
if (!strcmp(cmdbuf->buf, "")) {
data->no_disconnect_req = 1;
if (debug)
fprintf(stderr, "Debug: Smart transport connection "
"ready.\n");
ret = 1;
} else if (!strcmp(cmdbuf->buf, "fallback")) {
if (debug)
fprintf(stderr, "Debug: Falling back to dumb "
"transport.\n");
} else {
die(_("unknown response to connect: %s"),
cmdbuf->buf);
}
fclose(input);
return ret;
}
static int process_connect_service(struct transport *transport,
const char *name, const char *exec)
{
struct helper_data *data = transport->data;
struct strbuf cmdbuf = STRBUF_INIT;
int ret = 0;
/*
* Handle --upload-pack and friends. This is fire and forget...
* just warn if it fails.
*/
if (strcmp(name, exec)) {
int r = set_helper_option(transport, "servpath", exec);
if (r > 0)
warning(_("setting remote service path not supported by protocol"));
else if (r < 0)
warning(_("invalid remote service path"));
}
if (data->connect) {
strbuf_addf(&cmdbuf, "connect %s\n", name);
ret = run_connect(transport, &cmdbuf);
} else if (data->stateless_connect &&
(get_protocol_version_config() == protocol_v2) &&
(!strcmp("git-upload-pack", name) ||
!strcmp("git-upload-archive", name))) {
strbuf_addf(&cmdbuf, "stateless-connect %s\n", name);
ret = run_connect(transport, &cmdbuf);
if (ret)
transport->stateless_rpc = 1;
}
strbuf_release(&cmdbuf);
return ret;
}
static int process_connect(struct transport *transport,
int for_push)
{
struct helper_data *data = transport->data;
const char *name;
const char *exec;
int ret;
name = for_push ? "git-receive-pack" : "git-upload-pack";
if (for_push)
exec = data->transport_options.receivepack;
else
exec = data->transport_options.uploadpack;
ret = process_connect_service(transport, name, exec);
if (ret)
do_take_over(transport);
return ret;
}
static int connect_helper(struct transport *transport, const char *name,
const char *exec, int fd[2])
{
struct helper_data *data = transport->data;
/* Get_helper so connect is inited. */
get_helper(transport);
if (!process_connect_service(transport, name, exec))
die(_("can't connect to subservice %s"), name);
fd[0] = data->helper->out;
fd[1] = data->helper->in;
do_take_over(transport);
return 0;
}
static struct ref *get_refs_list_using_list(struct transport *transport,
int for_push);
static int fetch_refs(struct transport *transport,
int nr_heads, struct ref **to_fetch)
{
struct helper_data *data = transport->data;
int i, count;
get_helper(transport);
if (process_connect(transport, 0))
return transport->vtable->fetch_refs(transport, nr_heads, to_fetch);
fetch: teach independent negotiation (no packfile) Currently, the packfile negotiation step within a Git fetch cannot be done independent of sending the packfile, even though there is at least one application wherein this is useful. Therefore, make it possible for this negotiation step to be done independently. A subsequent commit will use this for one such application - push negotiation. This feature is for protocol v2 only. (An implementation for protocol v0 would require a separate implementation in the fetch, transport, and transport helper code.) In the protocol, the main hindrance towards independent negotiation is that the server can unilaterally decide to send the packfile. This is solved by a "wait-for-done" argument: the server will then wait for the client to say "done". In practice, the client will never say it; instead it will cease requests once it is satisfied. In the client, the main change lies in the transport and transport helper code. fetch_refs_via_pack() performs everything needed - protocol version and capability checks, and the negotiation itself. There are 2 code paths that do not go through fetch_refs_via_pack() that needed to be individually excluded: the bundle transport (excluded through requiring smart_options, which the bundle transport doesn't support) and transport helpers that do not support takeover. If or when we support independent negotiation for protocol v0, we will need to modify these 2 code paths to support it. But for now, report failure if independent negotiation is requested in these cases. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-04 21:16:01 +00:00
/*
* If we reach here, then the server, the client, and/or the transport
* helper does not support protocol v2. --negotiate-only requires
* protocol v2.
*/
if (data->transport_options.acked_commits) {
warning(_("--negotiate-only requires protocol v2"));
return -1;
}
if (!data->get_refs_list_called)
get_refs_list_using_list(transport, 0);
count = 0;
for (i = 0; i < nr_heads; i++)
if (!(to_fetch[i]->status & REF_STATUS_UPTODATE))
count++;
if (!count)
return 0;
if (data->check_connectivity &&
data->transport_options.check_self_contained_and_connected)
set_helper_option(transport, "check-connectivity", "true");
if (transport->cloning)
set_helper_option(transport, "cloning", "true");
if (data->transport_options.update_shallow)
set_helper_option(transport, "update-shallow", "true");
if (data->transport_options.refetch)
set_helper_option(transport, "refetch", "true");
if (data->transport_options.filter_options.choice) {
const char *spec = expand_list_objects_filter_spec(
&data->transport_options.filter_options);
set_helper_option(transport, "filter", spec);
}
if (data->transport_options.negotiation_tips)
warning("Ignoring --negotiation-tip because the protocol does not support it.");
if (data->fetch)
return fetch_with_fetch(transport, nr_heads, to_fetch);
if (data->import)
return fetch_with_import(transport, nr_heads, to_fetch);
return -1;
}
2020-08-27 15:45:46 +00:00
struct push_update_ref_state {
struct ref *hint;
struct ref_push_report *report;
int new_report;
};
static int push_update_ref_status(struct strbuf *buf,
2020-08-27 15:45:46 +00:00
struct push_update_ref_state *state,
struct ref *remote_refs)
{
char *refname, *msg;
int status, forced = 0;
2020-08-27 15:45:46 +00:00
if (starts_with(buf->buf, "option ")) {
struct object_id old_oid, new_oid;
const char *key, *val;
char *p;
if (!state->hint || !(state->report || state->new_report))
die(_("'option' without a matching 'ok/error' directive"));
if (state->new_report) {
if (!state->hint->report) {
CALLOC_ARRAY(state->hint->report, 1);
2020-08-27 15:45:46 +00:00
state->report = state->hint->report;
} else {
state->report = state->hint->report;
while (state->report->next)
state->report = state->report->next;
CALLOC_ARRAY(state->report->next, 1);
2020-08-27 15:45:46 +00:00
state->report = state->report->next;
}
state->new_report = 0;
}
key = buf->buf + 7;
p = strchr(key, ' ');
if (p)
*p++ = '\0';
val = p;
if (!strcmp(key, "refname"))
state->report->ref_name = xstrdup_or_null(val);
else if (!strcmp(key, "old-oid") && val &&
!parse_oid_hex(val, &old_oid, &val))
state->report->old_oid = oiddup(&old_oid);
else if (!strcmp(key, "new-oid") && val &&
!parse_oid_hex(val, &new_oid, &val))
state->report->new_oid = oiddup(&new_oid);
else if (!strcmp(key, "forced-update"))
state->report->forced_update = 1;
/* Not update remote namespace again. */
return 1;
}
state->report = NULL;
state->new_report = 0;
if (starts_with(buf->buf, "ok ")) {
status = REF_STATUS_OK;
refname = buf->buf + 3;
} else if (starts_with(buf->buf, "error ")) {
status = REF_STATUS_REMOTE_REJECT;
refname = buf->buf + 6;
} else
die(_("expected ok/error, helper said '%s'"), buf->buf);
msg = strchr(refname, ' ');
if (msg) {
struct strbuf msg_buf = STRBUF_INIT;
const char *end;
*msg++ = '\0';
if (!unquote_c_style(&msg_buf, msg, &end))
msg = strbuf_detach(&msg_buf, NULL);
else
msg = xstrdup(msg);
strbuf_release(&msg_buf);
if (!strcmp(msg, "no match")) {
status = REF_STATUS_NONE;
FREE_AND_NULL(msg);
}
else if (!strcmp(msg, "up to date")) {
status = REF_STATUS_UPTODATE;
FREE_AND_NULL(msg);
}
else if (!strcmp(msg, "non-fast forward")) {
status = REF_STATUS_REJECT_NONFASTFORWARD;
FREE_AND_NULL(msg);
}
else if (!strcmp(msg, "already exists")) {
status = REF_STATUS_REJECT_ALREADY_EXISTS;
FREE_AND_NULL(msg);
}
push: introduce REJECT_FETCH_FIRST and REJECT_NEEDS_FORCE When we push to update an existing ref, if: * the object at the tip of the remote is not a commit; or * the object we are pushing is not a commit, it won't be correct to suggest to fetch, integrate and push again, as the old and new objects will not "merge". We should explain that the push must be forced when there is a non-committish object is involved in such a case. If we do not have the current object at the tip of the remote, we do not even know that object, when fetched, is something that can be merged. In such a case, suggesting to pull first just like non-fast-forward case may not be technically correct, but in practice, most such failures are seen when you try to push your work to a branch without knowing that somebody else already pushed to update the same branch since you forked, so "pull first" would work as a suggestion most of the time. And if the object at the tip is not a commit, "pull first" will fail, without making any permanent damage. As a side effect, it also makes the error message the user will get during the next "push" attempt easier to understand, now the user is aware that a non-commit object is involved. In these cases, the current code already rejects such a push on the client end, but we used the same error and advice messages as the ones used when rejecting a non-fast-forward push, i.e. pull from there and integrate before pushing again. Introduce new rejection reasons and reword the messages appropriately. [jc: with help by Peff on message details] Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-01-23 21:55:30 +00:00
else if (!strcmp(msg, "fetch first")) {
status = REF_STATUS_REJECT_FETCH_FIRST;
FREE_AND_NULL(msg);
push: introduce REJECT_FETCH_FIRST and REJECT_NEEDS_FORCE When we push to update an existing ref, if: * the object at the tip of the remote is not a commit; or * the object we are pushing is not a commit, it won't be correct to suggest to fetch, integrate and push again, as the old and new objects will not "merge". We should explain that the push must be forced when there is a non-committish object is involved in such a case. If we do not have the current object at the tip of the remote, we do not even know that object, when fetched, is something that can be merged. In such a case, suggesting to pull first just like non-fast-forward case may not be technically correct, but in practice, most such failures are seen when you try to push your work to a branch without knowing that somebody else already pushed to update the same branch since you forked, so "pull first" would work as a suggestion most of the time. And if the object at the tip is not a commit, "pull first" will fail, without making any permanent damage. As a side effect, it also makes the error message the user will get during the next "push" attempt easier to understand, now the user is aware that a non-commit object is involved. In these cases, the current code already rejects such a push on the client end, but we used the same error and advice messages as the ones used when rejecting a non-fast-forward push, i.e. pull from there and integrate before pushing again. Introduce new rejection reasons and reword the messages appropriately. [jc: with help by Peff on message details] Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-01-23 21:55:30 +00:00
}
else if (!strcmp(msg, "needs force")) {
status = REF_STATUS_REJECT_NEEDS_FORCE;
FREE_AND_NULL(msg);
push: introduce REJECT_FETCH_FIRST and REJECT_NEEDS_FORCE When we push to update an existing ref, if: * the object at the tip of the remote is not a commit; or * the object we are pushing is not a commit, it won't be correct to suggest to fetch, integrate and push again, as the old and new objects will not "merge". We should explain that the push must be forced when there is a non-committish object is involved in such a case. If we do not have the current object at the tip of the remote, we do not even know that object, when fetched, is something that can be merged. In such a case, suggesting to pull first just like non-fast-forward case may not be technically correct, but in practice, most such failures are seen when you try to push your work to a branch without knowing that somebody else already pushed to update the same branch since you forked, so "pull first" would work as a suggestion most of the time. And if the object at the tip is not a commit, "pull first" will fail, without making any permanent damage. As a side effect, it also makes the error message the user will get during the next "push" attempt easier to understand, now the user is aware that a non-commit object is involved. In these cases, the current code already rejects such a push on the client end, but we used the same error and advice messages as the ones used when rejecting a non-fast-forward push, i.e. pull from there and integrate before pushing again. Introduce new rejection reasons and reword the messages appropriately. [jc: with help by Peff on message details] Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-01-23 21:55:30 +00:00
}
else if (!strcmp(msg, "stale info")) {
status = REF_STATUS_REJECT_STALE;
FREE_AND_NULL(msg);
}
else if (!strcmp(msg, "remote ref updated since checkout")) {
status = REF_STATUS_REJECT_REMOTE_UPDATED;
FREE_AND_NULL(msg);
}
else if (!strcmp(msg, "forced update")) {
forced = 1;
FREE_AND_NULL(msg);
}
else if (!strcmp(msg, "expecting report")) {
status = REF_STATUS_EXPECTING_REPORT;
FREE_AND_NULL(msg);
}
}
2020-08-27 15:45:46 +00:00
if (state->hint)
state->hint = find_ref_by_name(state->hint, refname);
if (!state->hint)
state->hint = find_ref_by_name(remote_refs, refname);
if (!state->hint) {
warning(_("helper reported unexpected status of %s"), refname);
return 1;
}
2020-08-27 15:45:46 +00:00
if (state->hint->status != REF_STATUS_NONE) {
/*
* Earlier, the ref was marked not to be pushed, so ignore the ref
* status reported by the remote helper if the latter is 'no match'.
*/
if (status == REF_STATUS_NONE)
return 1;
}
2020-08-27 15:45:46 +00:00
if (status == REF_STATUS_OK)
state->new_report = 1;
state->hint->status = status;
state->hint->forced_update |= forced;
state->hint->remote_status = msg;
return !(status == REF_STATUS_OK);
}
static int push_update_refs_status(struct helper_data *data,
struct ref *remote_refs,
int flags)
{
2020-08-27 15:45:46 +00:00
struct ref *ref;
struct ref_push_report *report;
struct strbuf buf = STRBUF_INIT;
2020-08-27 15:45:46 +00:00
struct push_update_ref_state state = { remote_refs, NULL, 0 };
for (;;) {
if (recvline(data, &buf)) {
2020-08-27 15:45:46 +00:00
strbuf_release(&buf);
return 1;
}
if (!buf.len)
break;
2020-08-27 15:45:46 +00:00
push_update_ref_status(&buf, &state, remote_refs);
}
strbuf_release(&buf);
2020-08-27 15:45:46 +00:00
if (flags & TRANSPORT_PUSH_DRY_RUN || !data->rs.nr || data->no_private_update)
return 0;
2020-08-27 15:45:46 +00:00
/* propagate back the update to the remote namespace */
for (ref = remote_refs; ref; ref = ref->next) {
char *private;
2020-08-27 15:45:46 +00:00
if (ref->status != REF_STATUS_OK)
continue;
2020-08-27 15:45:46 +00:00
if (!ref->report) {
private = apply_refspecs(&data->rs, ref->name);
if (!private)
continue;
update_ref("update by helper", private, &(ref->new_oid),
NULL, 0, 0);
free(private);
} else {
for (report = ref->report; report; report = report->next) {
private = apply_refspecs(&data->rs,
report->ref_name
? report->ref_name
: ref->name);
if (!private)
continue;
update_ref("update by helper", private,
report->new_oid
? report->new_oid
: &(ref->new_oid),
NULL, 0, 0);
free(private);
}
}
}
2020-08-27 15:45:46 +00:00
return 0;
}
static void set_common_push_options(struct transport *transport,
const char *name, int flags)
{
if (flags & TRANSPORT_PUSH_DRY_RUN) {
if (set_helper_option(transport, "dry-run", "true") != 0)
die(_("helper %s does not support dry-run"), name);
} else if (flags & TRANSPORT_PUSH_CERT_ALWAYS) {
if (set_helper_option(transport, TRANS_OPT_PUSH_CERT, "true") != 0)
die(_("helper %s does not support --signed"), name);
} else if (flags & TRANSPORT_PUSH_CERT_IF_ASKED) {
if (set_helper_option(transport, TRANS_OPT_PUSH_CERT, "if-asked") != 0)
die(_("helper %s does not support --signed=if-asked"), name);
}
remote-curl: pass on atomic capability to remote side When pushing more than one reference with the --atomic option, the server is supposed to perform a single atomic transaction to update the references, leaving them either all to succeed or all to fail. This works fine when pushing locally or over SSH, but when pushing over HTTP, we fail to pass the atomic capability to the remote side. In fact, we have not reported this capability to any remote helpers during the life of the feature. Now normally, things happen to work nevertheless, since we actually check for most types of failures, such as non-fast-forward updates, on the client side, and just abort the entire attempt. However, if the server side reports a problem, such as the inability to lock a ref, the transaction isn't atomic, because we haven't passed the appropriate capability over and the remote side has no way of knowing that we wanted atomic behavior. Fix this by passing the option from the transport code through to remote helpers, and from the HTTP remote helper down to send-pack. With this change, we can detect if the server side rejects the push and report back appropriately. Note the difference in the messages: the remote side reports "atomic transaction failed", while our own checking rejects pushes with the message "atomic push failed". Document the atomic option in the remote helper documentation, so other implementers can implement it if they like. Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-16 23:45:34 +00:00
if (flags & TRANSPORT_PUSH_ATOMIC)
if (set_helper_option(transport, TRANS_OPT_ATOMIC, "true") != 0)
die(_("helper %s does not support --atomic"), name);
if (flags & TRANSPORT_PUSH_FORCE_IF_INCLUDES)
if (set_helper_option(transport, TRANS_OPT_FORCE_IF_INCLUDES, "true") != 0)
die(_("helper %s does not support --%s"),
name, TRANS_OPT_FORCE_IF_INCLUDES);
if (flags & TRANSPORT_PUSH_OPTIONS) {
struct string_list_item *item;
for_each_string_list_item(item, transport->push_options)
if (set_helper_option(transport, "push-option", item->string) != 0)
die(_("helper %s does not support 'push-option'"), name);
}
}
static int push_refs_with_push(struct transport *transport,
struct ref *remote_refs, int flags)
{
int force_all = flags & TRANSPORT_PUSH_FORCE;
int mirror = flags & TRANSPORT_PUSH_MIRROR;
int atomic = flags & TRANSPORT_PUSH_ATOMIC;
struct helper_data *data = transport->data;
struct strbuf buf = STRBUF_INIT;
struct ref *ref;
struct string_list cas_options = STRING_LIST_INIT_DUP;
struct string_list_item *cas_option;
get_helper(transport);
if (!data->push)
return 1;
for (ref = remote_refs; ref; ref = ref->next) {
if (!ref->peer_ref && !mirror)
continue;
/* Check for statuses set by set_ref_status_for_push() */
switch (ref->status) {
case REF_STATUS_REJECT_NONFASTFORWARD:
case REF_STATUS_REJECT_STALE:
case REF_STATUS_REJECT_ALREADY_EXISTS:
case REF_STATUS_REJECT_REMOTE_UPDATED:
if (atomic) {
reject_atomic_push(remote_refs, mirror);
string_list_clear(&cas_options, 0);
return 0;
} else
continue;
case REF_STATUS_UPTODATE:
continue;
default:
; /* do nothing */
}
if (force_all)
ref->force = 1;
strbuf_addstr(&buf, "push ");
if (!ref->deletion) {
if (ref->force)
strbuf_addch(&buf, '+');
if (ref->peer_ref)
strbuf_addstr(&buf, ref->peer_ref->name);
else
strbuf_addstr(&buf, oid_to_hex(&ref->new_oid));
}
strbuf_addch(&buf, ':');
strbuf_addstr(&buf, ref->name);
strbuf_addch(&buf, '\n');
/*
* The "--force-with-lease" options without explicit
* values to expect have already been expanded into
* the ref->old_oid_expect[] field; we can ignore
* transport->smart_options->cas altogether and instead
* can enumerate them from the refs.
*/
if (ref->expect_old_sha1) {
struct strbuf cas = STRBUF_INIT;
strbuf_addf(&cas, "%s:%s",
ref->name, oid_to_hex(&ref->old_oid_expect));
string_list_append_nodup(&cas_options,
strbuf_detach(&cas, NULL));
}
}
if (buf.len == 0) {
string_list_clear(&cas_options, 0);
return 0;
}
for_each_string_list_item(cas_option, &cas_options)
set_helper_option(transport, "cas", cas_option->string);
set_common_push_options(transport, data->name, flags);
strbuf_addch(&buf, '\n');
sendline(data, &buf);
strbuf_release(&buf);
string_list_clear(&cas_options, 0);
return push_update_refs_status(data, remote_refs, flags);
}
static int push_refs_with_export(struct transport *transport,
struct ref *remote_refs, int flags)
{
struct ref *ref;
struct child_process *helper, exporter;
struct helper_data *data = transport->data;
struct string_list revlist_args = STRING_LIST_INIT_DUP;
struct strbuf buf = STRBUF_INIT;
if (!data->rs.nr)
die(_("remote-helper doesn't support push; refspec needed"));
set_common_push_options(transport, data->name, flags);
if (flags & TRANSPORT_PUSH_FORCE) {
if (set_helper_option(transport, "force", "true") != 0)
warning(_("helper %s does not support '--force'"), data->name);
}
helper = get_helper(transport);
write_constant(helper->in, "export\n");
for (ref = remote_refs; ref; ref = ref->next) {
char *private;
struct object_id oid;
private = apply_refspecs(&data->rs, ref->name);
if (private && !repo_get_oid(the_repository, private, &oid)) {
strbuf_addf(&buf, "^%s", private);
string_list_append_nodup(&revlist_args,
strbuf_detach(&buf, NULL));
oidcpy(&ref->old_oid, &oid);
}
free(private);
if (ref->peer_ref) {
if (strcmp(ref->name, ref->peer_ref->name)) {
if (!ref->deletion) {
const char *name;
int flag;
/* Follow symbolic refs (mainly for HEAD). */
name = resolve_ref_unsafe(ref->peer_ref->name,
RESOLVE_REF_READING,
&oid, &flag);
if (!name || !(flag & REF_ISSYMREF))
name = ref->peer_ref->name;
strbuf_addf(&buf, "%s:%s", name, ref->name);
} else
strbuf_addf(&buf, ":%s", ref->name);
string_list_append(&revlist_args, "--refspec");
string_list_append(&revlist_args, buf.buf);
strbuf_release(&buf);
}
if (!ref->deletion)
string_list_append(&revlist_args, ref->peer_ref->name);
}
}
if (get_exporter(transport, &exporter, &revlist_args))
die(_("couldn't run fast-export"));
string_list_clear(&revlist_args, 1);
if (finish_command(&exporter))
die(_("error while running fast-export"));
if (push_update_refs_status(data, remote_refs, flags))
return 1;
if (data->export_marks) {
strbuf_addf(&buf, "%s.tmp", data->export_marks);
rename(buf.buf, data->export_marks);
strbuf_release(&buf);
}
return 0;
}
static int push_refs(struct transport *transport,
struct ref *remote_refs, int flags)
{
struct helper_data *data = transport->data;
if (process_connect(transport, 1))
return transport->vtable->push_refs(transport, remote_refs, flags);
if (!remote_refs) {
fprintf(stderr,
_("No refs in common and none specified; doing nothing.\n"
"Perhaps you should specify a branch.\n"));
return 0;
}
if (data->push)
return push_refs_with_push(transport, remote_refs, flags);
if (data->export)
return push_refs_with_export(transport, remote_refs, flags);
return -1;
}
static int has_attribute(const char *attrs, const char *attr)
{
int len;
if (!attrs)
return 0;
len = strlen(attr);
for (;;) {
const char *space = strchrnul(attrs, ' ');
if (len == space - attrs && !strncmp(attrs, attr, len))
return 1;
if (!*space)
return 0;
attrs = space + 1;
}
}
static struct ref *get_refs_list(struct transport *transport, int for_push,
struct transport_ls_refs_options *transport_options)
{
get_helper(transport);
if (process_connect(transport, for_push))
return transport->vtable->get_refs_list(transport, for_push,
transport_options);
return get_refs_list_using_list(transport, for_push);
}
static struct ref *get_refs_list_using_list(struct transport *transport,
int for_push)
{
struct helper_data *data = transport->data;
struct child_process *helper;
struct ref *ret = NULL;
struct ref **tail = &ret;
struct ref *posn;
struct strbuf buf = STRBUF_INIT;
data->get_refs_list_called = 1;
helper = get_helper(transport);
transport-helper: send "true" value for object-format option The documentation in gitremote-helpers.txt claims that after a helper has advertised the "object-format" capability, Git may then send "option object-format true" to indicate that it would like to hear which object format the helper is using when it returns refs. However, the code implementing this has always written just "option object-format", without the extra "true" value. Nobody noticed in practice or in the tests because the only two helpers we ship are: - remote-curl, which quietly converts missing values into "true". This goes all the way back to ef08ef9ea0 (remote-helpers: Support custom transport options, 2009-10-30), despite the fact that I don't think any other option has ever made use of it. - remote-testgit in t5801 does insist on having a "true" value. But since it sends the ":object-format" response regardless of whether it thinks the caller asked for it (technically breaking protocol), everything just works, albeit with an extra shell error: .../git/t/t5801/git-remote-testgit: 150: test: =: unexpected operator printed to stderr, which you can see running t5801 with --verbose. (The problem is that $val is the empty string, and since we don't double-quote it in "test $val = true", we invoke "test = true" instead). When the documentation and code do not match, it is often good to fix the documentation rather than break compatibility. And in this case, we have had the mis-match since 8b85ee4f47 (transport-helper: implement object-format extensions, 2020-05-25). However, the sha256 feature was listed as experimental until 8e42eb0e9a (doc: sha256 is no longer experimental, 2023-07-31). It's possible there are some third party helpers that tried to follow the documentation, and are broken. Changing the code will fix them. It's also possible that there are ones that follow the code and will be broken if we change it. I suspect neither is the case given that no helper authors have brought this up as an issue (I only noticed it because I was running t5801 in verbose mode for other reasons and wondered about the weird shell error). That, coupled with the relative new-ness of sha256, makes me think nobody has really worked on helpers for it yet, which gives us an opportunity to correct the code before too much time passes. And doing so has some value: it brings "object-format" in line with the syntax of other options, making the protocol more consistent. It also lets us use set_helper_option(), which has better error reporting. Note that we don't really need to allow any other values like "false" here. The point is for Git to tell the helper that it understands ":object-format" lines coming back as part of the ref listing. There's no point in future versions saying "no, I don't understand that". To make sure everything works as expected, we can improve the remote-testgit helper from t5801 to send the ":object-format" line only if the other side correctly asked for it (which modern Git will always do). With that test change and without the matching code fix here, t5801 will fail when run with GIT_TEST_DEFAULT_HASH=sha256. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2024-03-20 09:41:03 +00:00
if (data->object_format)
set_helper_option(transport, "object-format", "true");
if (data->push && for_push)
write_constant(helper->in, "list for-push\n");
else
write_constant(helper->in, "list\n");
while (1) {
char *eov, *eon;
if (recvline(data, &buf))
exit(128);
if (!*buf.buf)
break;
else if (buf.buf[0] == ':') {
const char *value;
if (skip_prefix(buf.buf, ":object-format ", &value)) {
int algo = hash_algo_by_name(value);
if (algo == GIT_HASH_UNKNOWN)
die(_("unsupported object format '%s'"),
value);
transport->hash_algo = &hash_algos[algo];
}
continue;
}
eov = strchr(buf.buf, ' ');
if (!eov)
die(_("malformed response in ref list: %s"), buf.buf);
eon = strchr(eov + 1, ' ');
*eov = '\0';
if (eon)
*eon = '\0';
*tail = alloc_ref(eov + 1);
if (buf.buf[0] == '@')
(*tail)->symref = xstrdup(buf.buf + 1);
else if (buf.buf[0] != '?')
get_oid_hex_algop(buf.buf, &(*tail)->old_oid, transport->hash_algo);
if (eon) {
if (has_attribute(eon + 1, "unchanged")) {
(*tail)->status |= REF_STATUS_UPTODATE;
if (read_ref((*tail)->name, &(*tail)->old_oid) < 0)
die(_("could not read ref %s"),
(*tail)->name);
}
}
tail = &((*tail)->next);
}
if (debug)
fprintf(stderr, "Debug: Read ref listing.\n");
strbuf_release(&buf);
for (posn = ret; posn; posn = posn->next)
resolve_remote_symref(posn, ret);
return ret;
}
static int get_bundle_uri(struct transport *transport)
{
get_helper(transport);
if (process_connect(transport, 0))
return transport->vtable->get_bundle_uri(transport);
return -1;
}
static struct transport_vtable vtable = {
.set_option = set_helper_option,
.get_refs_list = get_refs_list,
.get_bundle_uri = get_bundle_uri,
.fetch_refs = fetch_refs,
.push_refs = push_refs,
.connect = connect_helper,
.disconnect = release_helper
};
int transport_helper_init(struct transport *transport, const char *name)
{
struct helper_data *data = xcalloc(1, sizeof(*data));
data->name = name;
transport: add a protocol-whitelist environment variable If we are cloning an untrusted remote repository into a sandbox, we may also want to fetch remote submodules in order to get the complete view as intended by the other side. However, that opens us up to attacks where a malicious user gets us to clone something they would not otherwise have access to (this is not necessarily a problem by itself, but we may then act on the cloned contents in a way that exposes them to the attacker). Ideally such a setup would sandbox git entirely away from high-value items, but this is not always practical or easy to set up (e.g., OS network controls may block multiple protocols, and we would want to enable some but not others). We can help this case by providing a way to restrict particular protocols. We use a whitelist in the environment. This is more annoying to set up than a blacklist, but defaults to safety if the set of protocols git supports grows). If no whitelist is specified, we continue to default to allowing all protocols (this is an "unsafe" default, but since the minority of users will want this sandboxing effect, it is the only sensible one). A note on the tests: ideally these would all be in a single test file, but the git-daemon and httpd test infrastructure is an all-or-nothing proposition rather than a test-by-test prerequisite. By putting them all together, we would be unable to test the file-local code on machines without apache. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-09-16 17:12:52 +00:00
transport_check_allowed(name);
if (getenv("GIT_TRANSPORT_HELPER_DEBUG"))
debug = 1;
list-objects-filter: add and use initializers In 7e2619d8ff (list_objects_filter_options: plug leak of filter_spec strings, 2022-09-08), we noted that the filter_spec string_list was inconsistent in how it handled memory ownership of strings stored in the list. The fix there was a bit of a band-aid to set the "strdup_strings" variable right before adding anything. That works OK, and it lets the users of the API continue to zero-initialize the struct. But it makes the code a bit hard to follow and accident-prone, as any other spots appending the filter_spec need to think about whether to set the strdup_strings value, too (there's one such spot in partial_clone_get_default_filter_spec(), which is probably a possible memory leak). So let's do that full cleanup now. We'll introduce a LIST_OBJECTS_FILTER_INIT macro and matching function, and use them as appropriate (though it is for the "_options" struct, this matches the corresponding list_objects_filter_release() function). This is harder than it seems! Many other structs, like git_transport_data, embed the filter struct. So they need to initialize it themselves even if the rest of the enclosing struct is OK with zero-initialization. I found all of the relevant spots by grepping manually for declarations of list_objects_filter_options. And then doing so recursively for structs which embed it, and ones which embed those, and so on. I'm pretty sure I got everything, but there's no change that would alert the compiler if any topics in flight added new declarations. To catch this case, we now double-check in the parsing function that things were initialized as expected and BUG() if appropriate. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-09-11 05:03:07 +00:00
list_objects_filter_init(&data->transport_options.filter_options);
transport->data = data;
transport->vtable = &vtable;
transport->smart_options = &(data->transport_options);
return 0;
}
/*
* Linux pipes can buffer 65536 bytes at once (and most platforms can
* buffer less), so attempt reads and writes with up to that size.
*/
#define BUFFERSIZE 65536
/* This should be enough to hold debugging message. */
#define PBUFFERSIZE 8192
/* Print bidirectional transfer loop debug message. */
__attribute__((format (printf, 1, 2)))
static void transfer_debug(const char *fmt, ...)
{
/*
* NEEDSWORK: This function is sometimes used from multiple threads, and
* we end up using debug_enabled racily. That "should not matter" since
* we always write the same value, but it's still wrong. This function
* is listed in .tsan-suppressions for the time being.
*/
va_list args;
char msgbuf[PBUFFERSIZE];
static int debug_enabled = -1;
if (debug_enabled < 0)
debug_enabled = getenv("GIT_TRANSLOOP_DEBUG") ? 1 : 0;
if (!debug_enabled)
return;
va_start(args, fmt);
vsnprintf(msgbuf, PBUFFERSIZE, fmt, args);
va_end(args);
fprintf(stderr, "Transfer loop debugging: %s\n", msgbuf);
}
/* Stream state: More data may be coming in this direction. */
#define SSTATE_TRANSFERRING 0
/*
* Stream state: No more data coming in this direction, flushing rest of
* data.
*/
#define SSTATE_FLUSHING 1
/* Stream state: Transfer in this direction finished. */
#define SSTATE_FINISHED 2
#define STATE_NEEDS_READING(state) ((state) <= SSTATE_TRANSFERRING)
#define STATE_NEEDS_WRITING(state) ((state) <= SSTATE_FLUSHING)
#define STATE_NEEDS_CLOSING(state) ((state) == SSTATE_FLUSHING)
/* Unidirectional transfer. */
struct unidirectional_transfer {
/* Source */
int src;
/* Destination */
int dest;
/* Is source socket? */
int src_is_sock;
/* Is destination socket? */
int dest_is_sock;
/* Transfer state (TRANSFERRING/FLUSHING/FINISHED) */
int state;
/* Buffer. */
char buf[BUFFERSIZE];
/* Buffer used. */
size_t bufuse;
/* Name of source. */
const char *src_name;
/* Name of destination. */
const char *dest_name;
};
/* Closes the target (for writing) if transfer has finished. */
static void udt_close_if_finished(struct unidirectional_transfer *t)
{
if (STATE_NEEDS_CLOSING(t->state) && !t->bufuse) {
t->state = SSTATE_FINISHED;
if (t->dest_is_sock)
shutdown(t->dest, SHUT_WR);
else
close(t->dest);
transfer_debug("Closed %s.", t->dest_name);
}
}
/*
* Tries to read data from source into buffer. If buffer is full,
* no data is read. Returns 0 on success, -1 on error.
*/
static int udt_do_read(struct unidirectional_transfer *t)
{
ssize_t bytes;
if (t->bufuse == BUFFERSIZE)
return 0; /* No space for more. */
transfer_debug("%s is readable", t->src_name);
bytes = xread(t->src, t->buf + t->bufuse, BUFFERSIZE - t->bufuse);
if (bytes < 0) {
error_errno(_("read(%s) failed"), t->src_name);
return -1;
} else if (bytes == 0) {
transfer_debug("%s EOF (with %i bytes in buffer)",
t->src_name, (int)t->bufuse);
t->state = SSTATE_FLUSHING;
} else if (bytes > 0) {
t->bufuse += bytes;
transfer_debug("Read %i bytes from %s (buffer now at %i)",
(int)bytes, t->src_name, (int)t->bufuse);
}
return 0;
}
/* Tries to write data from buffer into destination. If buffer is empty,
* no data is written. Returns 0 on success, -1 on error.
*/
static int udt_do_write(struct unidirectional_transfer *t)
{
ssize_t bytes;
if (t->bufuse == 0)
return 0; /* Nothing to write. */
transfer_debug("%s is writable", t->dest_name);
bytes = xwrite(t->dest, t->buf, t->bufuse);
if (bytes < 0) {
error_errno(_("write(%s) failed"), t->dest_name);
return -1;
} else if (bytes > 0) {
t->bufuse -= bytes;
if (t->bufuse)
memmove(t->buf, t->buf + bytes, t->bufuse);
transfer_debug("Wrote %i bytes to %s (buffer now at %i)",
(int)bytes, t->dest_name, (int)t->bufuse);
}
return 0;
}
/* State of bidirectional transfer loop. */
struct bidirectional_transfer_state {
/* Direction from program to git. */
struct unidirectional_transfer ptg;
/* Direction from git to program. */
struct unidirectional_transfer gtp;
};
static void *udt_copy_task_routine(void *udt)
{
struct unidirectional_transfer *t = (struct unidirectional_transfer *)udt;
while (t->state != SSTATE_FINISHED) {
if (STATE_NEEDS_READING(t->state))
if (udt_do_read(t))
return NULL;
if (STATE_NEEDS_WRITING(t->state))
if (udt_do_write(t))
return NULL;
if (STATE_NEEDS_CLOSING(t->state))
udt_close_if_finished(t);
}
return udt; /* Just some non-NULL value. */
}
#ifndef NO_PTHREADS
/*
* Join thread, with appropriate errors on failure. Name is name for the
* thread (for error messages). Returns 0 on success, 1 on failure.
*/
static int tloop_join(pthread_t thread, const char *name)
{
int err;
void *tret;
err = pthread_join(thread, &tret);
if (!tret) {
error(_("%s thread failed"), name);
return 1;
}
if (err) {
error(_("%s thread failed to join: %s"), name, strerror(err));
return 1;
}
return 0;
}
/*
* Spawn the transfer tasks and then wait for them. Returns 0 on success,
* -1 on failure.
*/
static int tloop_spawnwait_tasks(struct bidirectional_transfer_state *s)
{
pthread_t gtp_thread;
pthread_t ptg_thread;
int err;
int ret = 0;
err = pthread_create(&gtp_thread, NULL, udt_copy_task_routine,
&s->gtp);
if (err)
die(_("can't start thread for copying data: %s"), strerror(err));
err = pthread_create(&ptg_thread, NULL, udt_copy_task_routine,
&s->ptg);
if (err)
die(_("can't start thread for copying data: %s"), strerror(err));
ret |= tloop_join(gtp_thread, "Git to program copy");
ret |= tloop_join(ptg_thread, "Program to git copy");
return ret;
}
#else
/* Close the source and target (for writing) for transfer. */
static void udt_kill_transfer(struct unidirectional_transfer *t)
{
t->state = SSTATE_FINISHED;
/*
* Socket read end left open isn't a disaster if nobody
* attempts to read from it (mingw compat headers do not
* have SHUT_RD)...
*
* We can't fully close the socket since otherwise gtp
* task would first close the socket it sends data to
* while closing the ptg file descriptors.
*/
if (!t->src_is_sock)
close(t->src);
if (t->dest_is_sock)
shutdown(t->dest, SHUT_WR);
else
close(t->dest);
}
/*
* Join process, with appropriate errors on failure. Name is name for the
* process (for error messages). Returns 0 on success, 1 on failure.
*/
static int tloop_join(pid_t pid, const char *name)
{
int tret;
if (waitpid(pid, &tret, 0) < 0) {
error_errno(_("%s process failed to wait"), name);
return 1;
}
if (!WIFEXITED(tret) || WEXITSTATUS(tret)) {
error(_("%s process failed"), name);
return 1;
}
return 0;
}
/*
* Spawn the transfer tasks and then wait for them. Returns 0 on success,
* -1 on failure.
*/
static int tloop_spawnwait_tasks(struct bidirectional_transfer_state *s)
{
pid_t pid1, pid2;
int ret = 0;
/* Fork thread #1: git to program. */
pid1 = fork();
if (pid1 < 0)
die_errno(_("can't start thread for copying data"));
else if (pid1 == 0) {
udt_kill_transfer(&s->ptg);
exit(udt_copy_task_routine(&s->gtp) ? 0 : 1);
}
/* Fork thread #2: program to git. */
pid2 = fork();
if (pid2 < 0)
die_errno(_("can't start thread for copying data"));
else if (pid2 == 0) {
udt_kill_transfer(&s->gtp);
exit(udt_copy_task_routine(&s->ptg) ? 0 : 1);
}
/*
* Close both streams in parent as to not interfere with
* end of file detection and wait for both tasks to finish.
*/
udt_kill_transfer(&s->gtp);
udt_kill_transfer(&s->ptg);
ret |= tloop_join(pid1, "Git to program copy");
ret |= tloop_join(pid2, "Program to git copy");
return ret;
}
#endif
/*
* Copies data from stdin to output and from input to stdout simultaneously.
* Additionally filtering through given filter. If filter is NULL, uses
* identity filter.
*/
int bidirectional_transfer_loop(int input, int output)
{
struct bidirectional_transfer_state state;
/* Fill the state fields. */
state.ptg.src = input;
state.ptg.dest = 1;
state.ptg.src_is_sock = (input == output);
state.ptg.dest_is_sock = 0;
state.ptg.state = SSTATE_TRANSFERRING;
state.ptg.bufuse = 0;
state.ptg.src_name = "remote input";
state.ptg.dest_name = "stdout";
state.gtp.src = 0;
state.gtp.dest = output;
state.gtp.src_is_sock = 0;
state.gtp.dest_is_sock = (input == output);
state.gtp.state = SSTATE_TRANSFERRING;
state.gtp.bufuse = 0;
state.gtp.src_name = "stdin";
state.gtp.dest_name = "remote output";
return tloop_spawnwait_tasks(&state);
}
void reject_atomic_push(struct ref *remote_refs, int mirror_mode)
{
struct ref *ref;
/* Mark other refs as failed */
for (ref = remote_refs; ref; ref = ref->next) {
if (!ref->peer_ref && !mirror_mode)
continue;
switch (ref->status) {
case REF_STATUS_NONE:
case REF_STATUS_OK:
case REF_STATUS_EXPECTING_REPORT:
ref->status = REF_STATUS_ATOMIC_PUSH_FAILED;
continue;
default:
break; /* do nothing */
}
}
return;
}