git/serve.c

350 lines
8.3 KiB
C
Raw Normal View History

#include "git-compat-util.h"
#include "repository.h"
#include "config.h"
#include "hash-ll.h"
#include "pkt-line.h"
#include "version.h"
#include "ls-refs.h"
#include "protocol-caps.h"
#include "serve.h"
#include "upload-pack.h"
protocol v2: add server-side "bundle-uri" skeleton Add a skeleton server-side implementation of a new "bundle-uri" command to protocol v2. This will allow conforming clients to optionally seed their initial clones or incremental fetches from URLs containing "*.bundle" files created with "git bundle create". This change only performs the basic boilerplate of advertising a new protocol v2 capability. The new 'bundle-uri' capability allows a client to request a list of bundles. Right now, the server only returns a flush packet, which corresponds to an empty advertisement. The bundle.* config namespace describes which key-value pairs will be communicated across this interface in future updates. The critical bit right now is that the new boolean uploadPack.adverstiseBundleURIs config value signals whether or not this capability should be advertised at all. An earlier version of this patch [1] used a different transfer format than the "key=value" pairs in the current implementation. The change was made to unify the protocol v2 command with the bundle lists provided by independent bundle servers. Further, the standard allows for the server to advertise a URI that contains a bundle list. This allows users automatically discovering bundle providers that are loosely associated with the origin server, but without the origin server knowing exactly which bundles are currently available. [1] https://lore.kernel.org/git/RFC-patch-v2-01.13-2fc87ce092b-20220311T155841Z-avarab@gmail.com/ The very-deep headings needed to be modified to stop at level 4 due to documentation build issues. These were not recognized in earlier builds since the file was previously in the Documentation/technical/ directory and was built in a different way. With its current location, the heavily-nested details were causing build issues and they are now replaced with a bulletted list of details. Co-authored-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-12-22 15:14:07 +00:00
#include "bundle-uri.h"
#include "trace2.h"
static int advertise_sid = -1;
upload-pack: disallow object-info capability by default We added an "object-info" capability to the v2 upload-pack protocol in a2ba162cda (object-info: support for retrieving object info, 2021-04-20). In the almost 3 years since, we have not added any client-side support, and it does not appear to exist in other implementations either (JGit understands the verb on the server side, but not on the client side). Since this largely unused code is accessible over the network by default, it increases the attack surface of upload-pack. I don't know of any particularly severe problem, but one issue is that because of the request/response nature of the v2 protocol, it will happily read an unbounded number of packets, adding each one to a string list (without regard to whether they are objects we know about, duplicates, etc). This may be something we want to improve in the long run, but in the short term it makes sense to disable the feature entirely. We'll add a config option as an escape hatch for anybody who wants to develop the feature further. A more gentle option would be to add the config option to let people disable it manually, but leave it enabled by default. But given that there's no client side support, that seems like the wrong balance with security. Disabling by default will slow adoption a bit once client-side support does become available (there were some patches[1] in 2022, but nothing got merged and there's been nothing since). But clients have to deal with older servers that do not understand the option anyway (and the capability system handles that), so it will just be a matter of servers flipping their config at that point (and hopefully once any unbounded allocations have been addressed). [jk: this is a patch that GitHub has been running for several years, but rebased forward and with a new commit message for upstream] [1] https://lore.kernel.org/git/20220208231911.725273-1-calvinwan@google.com/ Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2024-02-28 22:38:58 +00:00
static int advertise_object_info = -1;
static int client_hash_algo = GIT_HASH_SHA1;
static int always_advertise(struct repository *r UNUSED,
struct strbuf *value UNUSED)
{
return 1;
}
static int agent_advertise(struct repository *r UNUSED,
struct strbuf *value)
{
if (value)
strbuf_addstr(value, git_user_agent_sanitized());
return 1;
}
static int object_format_advertise(struct repository *r,
struct strbuf *value)
{
if (value)
strbuf_addstr(value, r->hash_algo->name);
return 1;
}
static void object_format_receive(struct repository *r UNUSED,
const char *algo_name)
{
if (!algo_name)
die("object-format capability requires an argument");
client_hash_algo = hash_algo_by_name(algo_name);
if (client_hash_algo == GIT_HASH_UNKNOWN)
die("unknown object format '%s'", algo_name);
}
static int session_id_advertise(struct repository *r, struct strbuf *value)
{
if (advertise_sid == -1 &&
repo_config_get_bool(r, "transfer.advertisesid", &advertise_sid))
advertise_sid = 0;
if (!advertise_sid)
return 0;
if (value)
strbuf_addstr(value, trace2_session_id());
return 1;
}
static void session_id_receive(struct repository *r UNUSED,
const char *client_sid)
{
if (!client_sid)
client_sid = "";
trace2_data_string("transfer", NULL, "client-sid", client_sid);
}
upload-pack: disallow object-info capability by default We added an "object-info" capability to the v2 upload-pack protocol in a2ba162cda (object-info: support for retrieving object info, 2021-04-20). In the almost 3 years since, we have not added any client-side support, and it does not appear to exist in other implementations either (JGit understands the verb on the server side, but not on the client side). Since this largely unused code is accessible over the network by default, it increases the attack surface of upload-pack. I don't know of any particularly severe problem, but one issue is that because of the request/response nature of the v2 protocol, it will happily read an unbounded number of packets, adding each one to a string list (without regard to whether they are objects we know about, duplicates, etc). This may be something we want to improve in the long run, but in the short term it makes sense to disable the feature entirely. We'll add a config option as an escape hatch for anybody who wants to develop the feature further. A more gentle option would be to add the config option to let people disable it manually, but leave it enabled by default. But given that there's no client side support, that seems like the wrong balance with security. Disabling by default will slow adoption a bit once client-side support does become available (there were some patches[1] in 2022, but nothing got merged and there's been nothing since). But clients have to deal with older servers that do not understand the option anyway (and the capability system handles that), so it will just be a matter of servers flipping their config at that point (and hopefully once any unbounded allocations have been addressed). [jk: this is a patch that GitHub has been running for several years, but rebased forward and with a new commit message for upstream] [1] https://lore.kernel.org/git/20220208231911.725273-1-calvinwan@google.com/ Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2024-02-28 22:38:58 +00:00
static int object_info_advertise(struct repository *r, struct strbuf *value UNUSED)
{
if (advertise_object_info == -1 &&
repo_config_get_bool(r, "transfer.advertiseobjectinfo",
&advertise_object_info)) {
/* disabled by default */
advertise_object_info = 0;
}
return advertise_object_info;
}
struct protocol_capability {
/*
* The name of the capability. The server uses this name when
* advertising this capability, and the client uses this name to
* specify this capability.
*/
const char *name;
/*
* Function queried to see if a capability should be advertised.
* Optionally a value can be specified by adding it to 'value'.
* If a value is added to 'value', the server will advertise this
* capability as "<name>=<value>" instead of "<name>".
*/
int (*advertise)(struct repository *r, struct strbuf *value);
/*
* Function called when a client requests the capability as a command.
* Will be provided a struct packet_reader 'request' which it should
* use to read the command specific part of the request. Every command
* MUST read until a flush packet is seen before sending a response.
*
* This field should be NULL for capabilities which are not commands.
*/
int (*command)(struct repository *r, struct packet_reader *request);
/*
* Function called when a client requests the capability as a
* non-command. This may be NULL if the capability does nothing.
*
* For a capability of the form "foo=bar", the value string points to
* the content after the "=" (i.e., "bar"). For simple capabilities
* (just "foo"), it is NULL.
*/
void (*receive)(struct repository *r, const char *value);
};
static struct protocol_capability capabilities[] = {
{
.name = "agent",
.advertise = agent_advertise,
},
{
.name = "ls-refs",
.advertise = ls_refs_advertise,
.command = ls_refs,
},
{
.name = "fetch",
.advertise = upload_pack_advertise,
.command = upload_pack_v2,
},
{
.name = "server-option",
.advertise = always_advertise,
},
{
.name = "object-format",
.advertise = object_format_advertise,
.receive = object_format_receive,
},
{
.name = "session-id",
.advertise = session_id_advertise,
.receive = session_id_receive,
},
{
.name = "object-info",
upload-pack: disallow object-info capability by default We added an "object-info" capability to the v2 upload-pack protocol in a2ba162cda (object-info: support for retrieving object info, 2021-04-20). In the almost 3 years since, we have not added any client-side support, and it does not appear to exist in other implementations either (JGit understands the verb on the server side, but not on the client side). Since this largely unused code is accessible over the network by default, it increases the attack surface of upload-pack. I don't know of any particularly severe problem, but one issue is that because of the request/response nature of the v2 protocol, it will happily read an unbounded number of packets, adding each one to a string list (without regard to whether they are objects we know about, duplicates, etc). This may be something we want to improve in the long run, but in the short term it makes sense to disable the feature entirely. We'll add a config option as an escape hatch for anybody who wants to develop the feature further. A more gentle option would be to add the config option to let people disable it manually, but leave it enabled by default. But given that there's no client side support, that seems like the wrong balance with security. Disabling by default will slow adoption a bit once client-side support does become available (there were some patches[1] in 2022, but nothing got merged and there's been nothing since). But clients have to deal with older servers that do not understand the option anyway (and the capability system handles that), so it will just be a matter of servers flipping their config at that point (and hopefully once any unbounded allocations have been addressed). [jk: this is a patch that GitHub has been running for several years, but rebased forward and with a new commit message for upstream] [1] https://lore.kernel.org/git/20220208231911.725273-1-calvinwan@google.com/ Signed-off-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2024-02-28 22:38:58 +00:00
.advertise = object_info_advertise,
.command = cap_object_info,
},
protocol v2: add server-side "bundle-uri" skeleton Add a skeleton server-side implementation of a new "bundle-uri" command to protocol v2. This will allow conforming clients to optionally seed their initial clones or incremental fetches from URLs containing "*.bundle" files created with "git bundle create". This change only performs the basic boilerplate of advertising a new protocol v2 capability. The new 'bundle-uri' capability allows a client to request a list of bundles. Right now, the server only returns a flush packet, which corresponds to an empty advertisement. The bundle.* config namespace describes which key-value pairs will be communicated across this interface in future updates. The critical bit right now is that the new boolean uploadPack.adverstiseBundleURIs config value signals whether or not this capability should be advertised at all. An earlier version of this patch [1] used a different transfer format than the "key=value" pairs in the current implementation. The change was made to unify the protocol v2 command with the bundle lists provided by independent bundle servers. Further, the standard allows for the server to advertise a URI that contains a bundle list. This allows users automatically discovering bundle providers that are loosely associated with the origin server, but without the origin server knowing exactly which bundles are currently available. [1] https://lore.kernel.org/git/RFC-patch-v2-01.13-2fc87ce092b-20220311T155841Z-avarab@gmail.com/ The very-deep headings needed to be modified to stop at level 4 due to documentation build issues. These were not recognized in earlier builds since the file was previously in the Documentation/technical/ directory and was built in a different way. With its current location, the heavily-nested details were causing build issues and they are now replaced with a bulletted list of details. Co-authored-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Derrick Stolee <derrickstolee@github.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-12-22 15:14:07 +00:00
{
.name = "bundle-uri",
.advertise = bundle_uri_advertise,
.command = bundle_uri_command,
},
};
serve.[ch]: remove "serve_options", split up --advertise-refs code The "advertise capabilities" mode of serve.c added in ed10cb952d3 (serve: introduce git-serve, 2018-03-15) is only used by the http-backend.c to call {upload,receive}-pack with the --advertise-refs parameter. See 42526b478e3 (Add stateless RPC options to upload-pack, receive-pack, 2009-10-30). Let's just make cmd_upload_pack() take the two (v2) or three (v2) parameters the the v2/v1 servicing functions need directly, and pass those in via the function signature. The logic of whether daemon mode is implied by the timeout belongs in the v1 function (only used there). Once we split up the "advertise v2 refs" from "serve v2 request" it becomes clear that v2 never cared about those in combination. The only time it mattered was for v1 to emit its ref advertisement, in that case we wanted to emit the smart-http-only "no-done" capability. Since we only do that in the --advertise-refs codepath let's just have it set "do_done" itself in v1's upload_pack() just before send_ref(), at that point --advertise-refs and --stateless-rpc in combination are redundant (the only user is get_info_refs() in http-backend.c), so we can just pass in --advertise-refs only. Since we need to touch all the serve() and advertise_capabilities() codepaths let's rename them to less clever and obvious names, it's been suggested numerous times, the latest of which is [1]'s suggestion for protocol_v2_serve_loop(). Let's go with that. 1. https://lore.kernel.org/git/CAFQ2z_NyGb8rju5CKzmo6KhZXD0Dp21u-BbyCb2aNxLEoSPRJw@mail.gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-08-05 01:25:42 +00:00
void protocol_v2_advertise_capabilities(void)
{
struct strbuf capability = STRBUF_INIT;
struct strbuf value = STRBUF_INIT;
int i;
/* serve by default supports v2 */
packet_write_fmt(1, "version 2\n");
for (i = 0; i < ARRAY_SIZE(capabilities); i++) {
struct protocol_capability *c = &capabilities[i];
if (c->advertise(the_repository, &value)) {
strbuf_addstr(&capability, c->name);
if (value.len) {
strbuf_addch(&capability, '=');
strbuf_addbuf(&capability, &value);
}
strbuf_addch(&capability, '\n');
packet_write(1, capability.buf, capability.len);
}
strbuf_reset(&capability);
strbuf_reset(&value);
}
packet_flush(1);
strbuf_release(&capability);
strbuf_release(&value);
}
static struct protocol_capability *get_capability(const char *key, const char **value)
{
int i;
if (!key)
return NULL;
for (i = 0; i < ARRAY_SIZE(capabilities); i++) {
struct protocol_capability *c = &capabilities[i];
const char *out;
if (!skip_prefix(key, c->name, &out))
continue;
if (!*out) {
*value = NULL;
return c;
}
if (*out++ == '=') {
*value = out;
return c;
}
}
return NULL;
}
static int receive_client_capability(const char *key)
{
const char *value;
const struct protocol_capability *c = get_capability(key, &value);
if (!c || c->command || !c->advertise(the_repository, NULL))
return 0;
if (c->receive)
c->receive(the_repository, value);
return 1;
}
static int parse_command(const char *key, struct protocol_capability **command)
{
const char *out;
if (skip_prefix(key, "command=", &out)) {
const char *value;
struct protocol_capability *cmd = get_capability(out, &value);
if (*command)
die("command '%s' requested after already requesting command '%s'",
out, (*command)->name);
if (!cmd || !cmd->advertise(the_repository, NULL) || !cmd->command || value)
die("invalid command '%s'", out);
*command = cmd;
return 1;
}
return 0;
}
enum request_state {
PROCESS_REQUEST_KEYS,
PROCESS_REQUEST_DONE,
};
static int process_request(void)
{
enum request_state state = PROCESS_REQUEST_KEYS;
struct packet_reader reader;
int seen_capability_or_command = 0;
struct protocol_capability *command = NULL;
packet_reader_init(&reader, 0, NULL, 0,
PACKET_READ_CHOMP_NEWLINE |
PACKET_READ_GENTLE_ON_EOF |
PACKET_READ_DIE_ON_ERR_PACKET);
/*
* Check to see if the client closed their end before sending another
* request. If so we can terminate the connection.
*/
if (packet_reader_peek(&reader) == PACKET_READ_EOF)
return 1;
reader.options &= ~PACKET_READ_GENTLE_ON_EOF;
while (state != PROCESS_REQUEST_DONE) {
switch (packet_reader_peek(&reader)) {
case PACKET_READ_EOF:
BUG("Should have already died when seeing EOF");
case PACKET_READ_NORMAL:
if (parse_command(reader.line, &command) ||
receive_client_capability(reader.line))
seen_capability_or_command = 1;
else
die("unknown capability '%s'", reader.line);
/* Consume the peeked line */
packet_reader_read(&reader);
break;
case PACKET_READ_FLUSH:
/*
* If no command and no keys were given then the client
* wanted to terminate the connection.
*/
if (!seen_capability_or_command)
return 1;
/*
* The flush packet isn't consume here like it is in
* the other parts of this switch statement. This is
* so that the command can read the flush packet and
* see the end of the request in the same way it would
* if command specific arguments were provided after a
* delim packet.
*/
state = PROCESS_REQUEST_DONE;
break;
case PACKET_READ_DELIM:
/* Consume the peeked line */
packet_reader_read(&reader);
state = PROCESS_REQUEST_DONE;
break;
case PACKET_READ_RESPONSE_END:
BUG("unexpected response end packet");
}
}
if (!command)
die("no command requested");
if (client_hash_algo != hash_algo_by_ptr(the_repository->hash_algo))
die("mismatched object format: server %s; client %s\n",
the_repository->hash_algo->name,
hash_algos[client_hash_algo].name);
command->command(the_repository, &reader);
return 0;
}
serve.[ch]: remove "serve_options", split up --advertise-refs code The "advertise capabilities" mode of serve.c added in ed10cb952d3 (serve: introduce git-serve, 2018-03-15) is only used by the http-backend.c to call {upload,receive}-pack with the --advertise-refs parameter. See 42526b478e3 (Add stateless RPC options to upload-pack, receive-pack, 2009-10-30). Let's just make cmd_upload_pack() take the two (v2) or three (v2) parameters the the v2/v1 servicing functions need directly, and pass those in via the function signature. The logic of whether daemon mode is implied by the timeout belongs in the v1 function (only used there). Once we split up the "advertise v2 refs" from "serve v2 request" it becomes clear that v2 never cared about those in combination. The only time it mattered was for v1 to emit its ref advertisement, in that case we wanted to emit the smart-http-only "no-done" capability. Since we only do that in the --advertise-refs codepath let's just have it set "do_done" itself in v1's upload_pack() just before send_ref(), at that point --advertise-refs and --stateless-rpc in combination are redundant (the only user is get_info_refs() in http-backend.c), so we can just pass in --advertise-refs only. Since we need to touch all the serve() and advertise_capabilities() codepaths let's rename them to less clever and obvious names, it's been suggested numerous times, the latest of which is [1]'s suggestion for protocol_v2_serve_loop(). Let's go with that. 1. https://lore.kernel.org/git/CAFQ2z_NyGb8rju5CKzmo6KhZXD0Dp21u-BbyCb2aNxLEoSPRJw@mail.gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-08-05 01:25:42 +00:00
void protocol_v2_serve_loop(int stateless_rpc)
{
serve.[ch]: remove "serve_options", split up --advertise-refs code The "advertise capabilities" mode of serve.c added in ed10cb952d3 (serve: introduce git-serve, 2018-03-15) is only used by the http-backend.c to call {upload,receive}-pack with the --advertise-refs parameter. See 42526b478e3 (Add stateless RPC options to upload-pack, receive-pack, 2009-10-30). Let's just make cmd_upload_pack() take the two (v2) or three (v2) parameters the the v2/v1 servicing functions need directly, and pass those in via the function signature. The logic of whether daemon mode is implied by the timeout belongs in the v1 function (only used there). Once we split up the "advertise v2 refs" from "serve v2 request" it becomes clear that v2 never cared about those in combination. The only time it mattered was for v1 to emit its ref advertisement, in that case we wanted to emit the smart-http-only "no-done" capability. Since we only do that in the --advertise-refs codepath let's just have it set "do_done" itself in v1's upload_pack() just before send_ref(), at that point --advertise-refs and --stateless-rpc in combination are redundant (the only user is get_info_refs() in http-backend.c), so we can just pass in --advertise-refs only. Since we need to touch all the serve() and advertise_capabilities() codepaths let's rename them to less clever and obvious names, it's been suggested numerous times, the latest of which is [1]'s suggestion for protocol_v2_serve_loop(). Let's go with that. 1. https://lore.kernel.org/git/CAFQ2z_NyGb8rju5CKzmo6KhZXD0Dp21u-BbyCb2aNxLEoSPRJw@mail.gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-08-05 01:25:42 +00:00
if (!stateless_rpc)
protocol_v2_advertise_capabilities();
/*
* If stateless-rpc was requested then exit after
* a single request/response exchange
*/
serve.[ch]: remove "serve_options", split up --advertise-refs code The "advertise capabilities" mode of serve.c added in ed10cb952d3 (serve: introduce git-serve, 2018-03-15) is only used by the http-backend.c to call {upload,receive}-pack with the --advertise-refs parameter. See 42526b478e3 (Add stateless RPC options to upload-pack, receive-pack, 2009-10-30). Let's just make cmd_upload_pack() take the two (v2) or three (v2) parameters the the v2/v1 servicing functions need directly, and pass those in via the function signature. The logic of whether daemon mode is implied by the timeout belongs in the v1 function (only used there). Once we split up the "advertise v2 refs" from "serve v2 request" it becomes clear that v2 never cared about those in combination. The only time it mattered was for v1 to emit its ref advertisement, in that case we wanted to emit the smart-http-only "no-done" capability. Since we only do that in the --advertise-refs codepath let's just have it set "do_done" itself in v1's upload_pack() just before send_ref(), at that point --advertise-refs and --stateless-rpc in combination are redundant (the only user is get_info_refs() in http-backend.c), so we can just pass in --advertise-refs only. Since we need to touch all the serve() and advertise_capabilities() codepaths let's rename them to less clever and obvious names, it's been suggested numerous times, the latest of which is [1]'s suggestion for protocol_v2_serve_loop(). Let's go with that. 1. https://lore.kernel.org/git/CAFQ2z_NyGb8rju5CKzmo6KhZXD0Dp21u-BbyCb2aNxLEoSPRJw@mail.gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-08-05 01:25:42 +00:00
if (stateless_rpc) {
process_request();
} else {
for (;;)
if (process_request())
break;
}
}