2005-11-18 19:02:58 +00:00
|
|
|
#ifndef HTTP_H
|
|
|
|
#define HTTP_H
|
|
|
|
|
2023-04-11 07:41:51 +00:00
|
|
|
struct packed_git;
|
|
|
|
|
|
|
|
#include "git-zlib.h"
|
2005-11-18 19:02:58 +00:00
|
|
|
|
|
|
|
#include <curl/curl.h>
|
|
|
|
#include <curl/easy.h>
|
|
|
|
|
2007-12-09 19:30:59 +00:00
|
|
|
#include "strbuf.h"
|
2008-02-27 20:35:50 +00:00
|
|
|
#include "remote.h"
|
2007-12-09 19:30:59 +00:00
|
|
|
|
2005-11-18 19:02:58 +00:00
|
|
|
#define DEFAULT_MAX_REQUESTS 5
|
|
|
|
|
2011-03-16 07:08:34 +00:00
|
|
|
struct slot_results {
|
2006-01-31 19:06:55 +00:00
|
|
|
CURLcode curl_result;
|
|
|
|
long http_code;
|
2013-10-31 06:35:31 +00:00
|
|
|
long auth_avail;
|
http: use credential API to handle proxy authentication
Currently, the only way to pass proxy credentials to curl is by including them
in the proxy URL. Usually, this means they will end up on disk unencrypted, one
way or another (by inclusion in ~/.gitconfig, shell profile or history). Since
proxy authentication often uses a domain user, credentials can be security
sensitive; therefore, a safer way of passing credentials is desirable.
If the configured proxy contains a username but not a password, query the
credential API for one. Also, make sure we approve/reject proxy credentials
properly.
For consistency reasons, add parsing of http_proxy/https_proxy/all_proxy
environment variables, which would otherwise be evaluated as a fallback by curl.
Without this, we would have different semantics for git configuration and
environment variables.
Helped-by: Junio C Hamano <gitster@pobox.com>
Helped-by: Eric Sunshine <sunshine@sunshineco.com>
Helped-by: Elia Pinto <gitter.spiros@gmail.com>
Signed-off-by: Knut Franke <k.franke@science-computing.de>
Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-01-26 13:02:48 +00:00
|
|
|
long http_connectcode;
|
2006-01-31 19:06:55 +00:00
|
|
|
};
|
|
|
|
|
2011-03-16 07:08:34 +00:00
|
|
|
struct active_request_slot {
|
2005-11-18 19:02:58 +00:00
|
|
|
CURL *curl;
|
|
|
|
int in_use;
|
|
|
|
CURLcode curl_result;
|
|
|
|
long http_code;
|
2006-03-11 04:18:01 +00:00
|
|
|
int *finished;
|
2006-01-31 19:06:55 +00:00
|
|
|
struct slot_results *results;
|
2005-11-18 19:02:58 +00:00
|
|
|
void *callback_data;
|
|
|
|
void (*callback_func)(void *data);
|
|
|
|
struct active_request_slot *next;
|
|
|
|
};
|
|
|
|
|
2011-03-16 07:08:34 +00:00
|
|
|
struct buffer {
|
2007-12-09 19:30:59 +00:00
|
|
|
struct strbuf buf;
|
|
|
|
size_t posn;
|
2005-11-18 19:02:58 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
/* Curl request read/write callbacks */
|
2019-04-29 08:28:14 +00:00
|
|
|
size_t fread_buffer(char *ptr, size_t eltsize, size_t nmemb, void *strbuf);
|
|
|
|
size_t fwrite_buffer(char *ptr, size_t eltsize, size_t nmemb, void *strbuf);
|
|
|
|
size_t fwrite_null(char *ptr, size_t eltsize, size_t nmemb, void *strbuf);
|
http: prefer CURLOPT_SEEKFUNCTION to CURLOPT_IOCTLFUNCTION
The IOCTLFUNCTION option has been deprecated, and generates a compiler
warning in recent versions of curl. We can switch to using SEEKFUNCTION
instead. It was added in 2008 via curl 7.18.0; our INSTALL file already
indicates we require at least curl 7.19.4.
But there's one catch: curl says we should use CURL_SEEKFUNC_{OK,FAIL},
and those didn't arrive until 7.19.5. One workaround would be to use a
bare 0/1 here (or define our own macros). But let's just bump the
minimum required version to 7.19.5. That version is only a minor version
bump from our existing requirement, and is only a 2 month time bump for
versions that are almost 13 years old. So it's not likely that anybody
cares about the distinction.
Switching means we have to rewrite the ioctl functions into seek
functions. In some ways they are simpler (seeking is the only
operation), but in some ways more complex (the ioctl allowed only a full
rewind, but now we can seek to arbitrary offsets).
Curl will only ever use SEEK_SET (per their documentation), so I didn't
bother implementing anything else, since it would naturally be
completely untested. This seems unlikely to change, but I added an
assertion just in case.
Likewise, I doubt curl will ever try to seek outside of the buffer sizes
we've told it, but I erred on the defensive side here, rather than do an
out-of-bounds read.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-17 03:04:44 +00:00
|
|
|
int seek_buffer(void *clientp, curl_off_t offset, int origin);
|
2005-11-18 19:02:58 +00:00
|
|
|
|
|
|
|
/* Slot lifecycle functions */
|
2019-04-29 08:28:14 +00:00
|
|
|
struct active_request_slot *get_active_slot(void);
|
|
|
|
int start_active_slot(struct active_request_slot *slot);
|
|
|
|
void run_active_slot(struct active_request_slot *slot);
|
|
|
|
void finish_all_active_slots(void);
|
2005-11-18 19:02:58 +00:00
|
|
|
|
http: never use curl_easy_perform
We currently don't reuse http connections when fetching via
the smart-http protocol. This is bad because the TCP
handshake introduces latency, and especially because SSL
connection setup may be non-trivial.
We can fix it by consistently using curl's "multi"
interface. The reason is rather complicated:
Our http code has two ways of being used: queuing many
"slots" to be fetched in parallel, or fetching a single
request in a blocking manner. The parallel code is built on
curl's "multi" interface. Most of the single-request code
uses http_request, which is built on top of the parallel
code (we just feed it one slot, and wait until it finishes).
However, one could also accomplish the single-request scheme
by avoiding curl's multi interface entirely and just using
curl_easy_perform. This is simpler, and is used by post_rpc
in the smart-http protocol.
It does work to use the same curl handle in both contexts,
as long as it is not at the same time. However, internally
curl may not share all of the cached resources between both
contexts. In particular, a connection formed using the
"multi" code will go into a reuse pool connected to the
"multi" object. Further requests using the "easy" interface
will not be able to reuse that connection.
The smart http protocol does ref discovery via http_request,
which uses the "multi" interface, and then follows up with
the "easy" interface for its rpc calls. As a result, we make
two HTTP connections rather than reusing a single one.
We could teach the ref discovery to use the "easy"
interface. But it is only once we have done this discovery
that we know whether the protocol will be smart or dumb. If
it is dumb, then our further requests, which want to fetch
objects in parallel, will not be able to reuse the same
connection.
Instead, this patch switches post_rpc to build on the
parallel interface, which means that we use it consistently
everywhere. It's a little more complicated to use, but since
we have the infrastructure already, it doesn't add any code;
we can just factor out the relevant bits from http_request.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-02-18 10:34:20 +00:00
|
|
|
/*
|
|
|
|
* This will run one slot to completion in a blocking manner, similar to how
|
|
|
|
* curl_easy_perform would work (but we don't want to use that, because
|
|
|
|
* we do not want to intermingle calls to curl_multi and curl_easy).
|
|
|
|
*
|
|
|
|
*/
|
|
|
|
int run_one_slot(struct active_request_slot *slot,
|
|
|
|
struct slot_results *results);
|
|
|
|
|
2019-04-29 08:28:14 +00:00
|
|
|
void fill_active_slots(void);
|
|
|
|
void add_fill_function(void *data, int (*fill)(void *));
|
|
|
|
void step_active_slots(void);
|
2005-11-18 19:02:58 +00:00
|
|
|
|
2019-04-29 08:28:14 +00:00
|
|
|
void http_init(struct remote *remote, const char *url,
|
2019-04-29 08:28:23 +00:00
|
|
|
int proactive_auth);
|
2019-04-29 08:28:14 +00:00
|
|
|
void http_cleanup(void);
|
|
|
|
struct curl_slist *http_copy_default_headers(void);
|
2005-11-18 19:02:58 +00:00
|
|
|
|
2016-02-03 04:09:14 +00:00
|
|
|
extern long int git_curl_ipresolve;
|
2005-11-18 19:02:58 +00:00
|
|
|
extern int active_requests;
|
2009-06-06 08:43:41 +00:00
|
|
|
extern int http_is_verbose;
|
2017-04-11 18:13:57 +00:00
|
|
|
extern ssize_t http_post_buffer;
|
http: hoist credential request out of handle_curl_result
When we are handling a curl response code in http_request or
in the remote-curl RPC code, we use the handle_curl_result
helper to translate curl's response into an easy-to-use
code. When we see an HTTP 401, we do one of two things:
1. If we already had a filled-in credential, we mark it as
rejected, and then return HTTP_NOAUTH to indicate to
the caller that we failed.
2. If we didn't, then we ask for a new credential and tell
the caller HTTP_REAUTH to indicate that they may want
to try again.
Rejecting in the first case makes sense; it is the natural
result of the request we just made. However, prompting for
more credentials in the second step does not always make
sense. We do not know for sure that the caller is going to
make a second request, and nor are we sure that it will be
to the same URL. Logically, the prompt belongs not to the
request we just finished, but to the request we are (maybe)
about to make.
In practice, it is very hard to trigger any bad behavior.
Currently, if we make a second request, it will always be to
the same URL (even in the face of redirects, because curl
handles the redirects internally). And we almost always
retry on HTTP_REAUTH these days. The one exception is if we
are streaming a large RPC request to the server (e.g., a
pushed packfile), in which case we cannot restart. It's
extremely unlikely to see a 401 response at this stage,
though, as we would typically have seen it when we sent a
probe request, before streaming the data.
This patch drops the automatic prompt out of case 2, and
instead requires the caller to do it. This is a few extra
lines of code, and the bug it fixes is unlikely to come up
in practice. But it is conceptually cleaner, and paves the
way for better handling of credentials across redirects.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
2013-09-28 08:31:45 +00:00
|
|
|
extern struct credential http_auth;
|
2005-11-18 19:02:58 +00:00
|
|
|
|
|
|
|
extern char curl_errorstr[CURL_ERROR_SIZE];
|
|
|
|
|
http: make redirects more obvious
We instruct curl to always follow HTTP redirects. This is
convenient, but it creates opportunities for malicious
servers to create confusing situations. For instance,
imagine Alice is a git user with access to a private
repository on Bob's server. Mallory runs her own server and
wants to access objects from Bob's repository.
Mallory may try a few tricks that involve asking Alice to
clone from her, build on top, and then push the result:
1. Mallory may simply redirect all fetch requests to Bob's
server. Git will transparently follow those redirects
and fetch Bob's history, which Alice may believe she
got from Mallory. The subsequent push seems like it is
just feeding Mallory back her own objects, but is
actually leaking Bob's objects. There is nothing in
git's output to indicate that Bob's repository was
involved at all.
The downside (for Mallory) of this attack is that Alice
will have received Bob's entire repository, and is
likely to notice that when building on top of it.
2. If Mallory happens to know the sha1 of some object X in
Bob's repository, she can instead build her own history
that references that object. She then runs a dumb http
server, and Alice's client will fetch each object
individually. When it asks for X, Mallory redirects her
to Bob's server. The end result is that Alice obtains
objects from Bob, but they may be buried deep in
history. Alice is less likely to notice.
Both of these attacks are fairly hard to pull off. There's a
social component in getting Mallory to convince Alice to
work with her. Alice may be prompted for credentials in
accessing Bob's repository (but not always, if she is using
a credential helper that caches). Attack (1) requires a
certain amount of obliviousness on Alice's part while making
a new commit. Attack (2) requires that Mallory knows a sha1
in Bob's repository, that Bob's server supports dumb http,
and that the object in question is loose on Bob's server.
But we can probably make things a bit more obvious without
any loss of functionality. This patch does two things to
that end.
First, when we encounter a whole-repo redirect during the
initial ref discovery, we now inform the user on stderr,
making attack (1) much more obvious.
Second, the decision to follow redirects is now
configurable. The truly paranoid can set the new
http.followRedirects to false to avoid any redirection
entirely. But for a more practical default, we will disallow
redirects only after the initial ref discovery. This is
enough to thwart attacks similar to (2), while still
allowing the common use of redirects at the repository
level. Since c93c92f30 (http: update base URLs when we see
redirects, 2013-09-28) we re-root all further requests from
the redirect destination, which should generally mean that
no further redirection is necessary.
As an escape hatch, in case there really is a server that
needs to redirect individual requests, the user can set
http.followRedirects to "true" (and this can be done on a
per-server basis via http.*.followRedirects config).
Reported-by: Jann Horn <jannh@google.com>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-12-06 18:24:41 +00:00
|
|
|
enum http_follow_config {
|
|
|
|
HTTP_FOLLOW_NONE,
|
|
|
|
HTTP_FOLLOW_ALWAYS,
|
|
|
|
HTTP_FOLLOW_INITIAL
|
|
|
|
};
|
|
|
|
extern enum http_follow_config http_follow_config;
|
|
|
|
|
2007-12-10 21:36:09 +00:00
|
|
|
static inline int missing__target(int code, int result)
|
|
|
|
{
|
|
|
|
return /* file:// URL -- do we ever use one??? */
|
|
|
|
(result == CURLE_FILE_COULDNT_READ_FILE) ||
|
|
|
|
/* http:// and https:// URL */
|
|
|
|
(code == 404 && result == CURLE_HTTP_RETURNED_ERROR) ||
|
|
|
|
/* ftp:// URL */
|
|
|
|
(code == 550 && result == CURLE_FTP_COULDNT_RETR_FILE)
|
|
|
|
;
|
|
|
|
}
|
|
|
|
|
|
|
|
#define missing_target(a) missing__target((a)->http_code, (a)->curl_result)
|
|
|
|
|
2019-03-24 12:08:38 +00:00
|
|
|
/*
|
|
|
|
* Normalize curl results to handle CURL_FAILONERROR (or lack thereof). Failing
|
|
|
|
* http codes have their "result" converted to CURLE_HTTP_RETURNED_ERROR, and
|
|
|
|
* an appropriate string placed in the errorstr buffer (pass curl_errorstr if
|
|
|
|
* you don't have a custom buffer).
|
|
|
|
*/
|
|
|
|
void normalize_curl_result(CURLcode *result, long http_code, char *errorstr,
|
|
|
|
size_t errorlen);
|
|
|
|
|
http*: add helper methods for fetching objects (loose)
The code handling the fetching of loose objects in http-push.c and
http-walker.c have been refactored into new methods and a new struct
(object_http_request) in http.c. They are not meant to be invoked
elsewhere.
The new methods in http.c are
- new_http_object_request
- process_http_object_request
- finish_http_object_request
- abort_http_object_request
- release_http_object_request
and the new struct is http_object_request.
RANGER_HEADER_SIZE and no_pragma_header is no longer made available
outside of http.c, since after the above changes, there are no other
instances of usage outside of http.c.
Remove members of the transfer_request struct in http-push.c and
http-walker.c, including filename, real_sha1 and zret, as they are used
no longer used.
Move the methods append_remote_object_url() and get_remote_object_url()
from http-push.c to http.c. Additionally, get_remote_object_url() is no
longer defined only when USE_CURL_MULTI is defined, since
non-USE_CURL_MULTI code in http.c uses it (namely, in
new_http_object_request()).
Refactor code from http-push.c::start_fetch_loose() and
http-walker.c::start_object_fetch_request() that deals with the details
of coming up with the filename to store the retrieved object, resuming
a previously aborted request, and making a new curl request, into a new
function, new_http_object_request().
Refactor code from http-walker.c::process_object_request() into the
function, process_http_object_request().
Refactor code from http-push.c::finish_request() and
http-walker.c::finish_object_request() into a new function,
finish_http_object_request(). It returns the result of the
move_temp_to_file() invocation.
Add a function, release_http_object_request(), which cleans up object
request data. http-push.c and http-walker.c invoke this function
separately; http-push.c::release_request() and
http-walker.c::release_object_request() do not invoke this function.
Add a function, abort_http_object_request(), which unlink()s the object
file and invokes release_http_object_request(). Update
http-walker.c::abort_object_request() to use this.
Signed-off-by: Tay Ray Chuan <rctay89@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-06-06 08:44:02 +00:00
|
|
|
/* Helpers for modifying and creating URLs */
|
2019-04-29 08:28:14 +00:00
|
|
|
void append_remote_object_url(struct strbuf *buf, const char *url,
|
2019-04-29 08:28:23 +00:00
|
|
|
const char *hex,
|
|
|
|
int only_two_digit_prefix);
|
2019-04-29 08:28:14 +00:00
|
|
|
char *get_remote_object_url(const char *url, const char *hex,
|
2019-04-29 08:28:23 +00:00
|
|
|
int only_two_digit_prefix);
|
http*: add helper methods for fetching objects (loose)
The code handling the fetching of loose objects in http-push.c and
http-walker.c have been refactored into new methods and a new struct
(object_http_request) in http.c. They are not meant to be invoked
elsewhere.
The new methods in http.c are
- new_http_object_request
- process_http_object_request
- finish_http_object_request
- abort_http_object_request
- release_http_object_request
and the new struct is http_object_request.
RANGER_HEADER_SIZE and no_pragma_header is no longer made available
outside of http.c, since after the above changes, there are no other
instances of usage outside of http.c.
Remove members of the transfer_request struct in http-push.c and
http-walker.c, including filename, real_sha1 and zret, as they are used
no longer used.
Move the methods append_remote_object_url() and get_remote_object_url()
from http-push.c to http.c. Additionally, get_remote_object_url() is no
longer defined only when USE_CURL_MULTI is defined, since
non-USE_CURL_MULTI code in http.c uses it (namely, in
new_http_object_request()).
Refactor code from http-push.c::start_fetch_loose() and
http-walker.c::start_object_fetch_request() that deals with the details
of coming up with the filename to store the retrieved object, resuming
a previously aborted request, and making a new curl request, into a new
function, new_http_object_request().
Refactor code from http-walker.c::process_object_request() into the
function, process_http_object_request().
Refactor code from http-push.c::finish_request() and
http-walker.c::finish_object_request() into a new function,
finish_http_object_request(). It returns the result of the
move_temp_to_file() invocation.
Add a function, release_http_object_request(), which cleans up object
request data. http-push.c and http-walker.c invoke this function
separately; http-push.c::release_request() and
http-walker.c::release_object_request() do not invoke this function.
Add a function, abort_http_object_request(), which unlink()s the object
file and invokes release_http_object_request(). Update
http-walker.c::abort_object_request() to use this.
Signed-off-by: Tay Ray Chuan <rctay89@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-06-06 08:44:02 +00:00
|
|
|
|
2013-09-28 08:31:23 +00:00
|
|
|
/* Options for http_get_*() */
|
|
|
|
struct http_get_options {
|
|
|
|
unsigned no_cache:1,
|
http: make redirects more obvious
We instruct curl to always follow HTTP redirects. This is
convenient, but it creates opportunities for malicious
servers to create confusing situations. For instance,
imagine Alice is a git user with access to a private
repository on Bob's server. Mallory runs her own server and
wants to access objects from Bob's repository.
Mallory may try a few tricks that involve asking Alice to
clone from her, build on top, and then push the result:
1. Mallory may simply redirect all fetch requests to Bob's
server. Git will transparently follow those redirects
and fetch Bob's history, which Alice may believe she
got from Mallory. The subsequent push seems like it is
just feeding Mallory back her own objects, but is
actually leaking Bob's objects. There is nothing in
git's output to indicate that Bob's repository was
involved at all.
The downside (for Mallory) of this attack is that Alice
will have received Bob's entire repository, and is
likely to notice that when building on top of it.
2. If Mallory happens to know the sha1 of some object X in
Bob's repository, she can instead build her own history
that references that object. She then runs a dumb http
server, and Alice's client will fetch each object
individually. When it asks for X, Mallory redirects her
to Bob's server. The end result is that Alice obtains
objects from Bob, but they may be buried deep in
history. Alice is less likely to notice.
Both of these attacks are fairly hard to pull off. There's a
social component in getting Mallory to convince Alice to
work with her. Alice may be prompted for credentials in
accessing Bob's repository (but not always, if she is using
a credential helper that caches). Attack (1) requires a
certain amount of obliviousness on Alice's part while making
a new commit. Attack (2) requires that Mallory knows a sha1
in Bob's repository, that Bob's server supports dumb http,
and that the object in question is loose on Bob's server.
But we can probably make things a bit more obvious without
any loss of functionality. This patch does two things to
that end.
First, when we encounter a whole-repo redirect during the
initial ref discovery, we now inform the user on stderr,
making attack (1) much more obvious.
Second, the decision to follow redirects is now
configurable. The truly paranoid can set the new
http.followRedirects to false to avoid any redirection
entirely. But for a more practical default, we will disallow
redirects only after the initial ref discovery. This is
enough to thwart attacks similar to (2), while still
allowing the common use of redirects at the repository
level. Since c93c92f30 (http: update base URLs when we see
redirects, 2013-09-28) we re-root all further requests from
the redirect destination, which should generally mean that
no further redirection is necessary.
As an escape hatch, in case there really is a server that
needs to redirect individual requests, the user can set
http.followRedirects to "true" (and this can be done on a
per-server basis via http.*.followRedirects config).
Reported-by: Jann Horn <jannh@google.com>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-12-06 18:24:41 +00:00
|
|
|
initial_request:1;
|
2009-06-06 08:43:53 +00:00
|
|
|
|
2013-09-28 08:31:23 +00:00
|
|
|
/* If non-NULL, returns the content-type of the response. */
|
|
|
|
struct strbuf *content_type;
|
2013-09-28 08:32:02 +00:00
|
|
|
|
2014-05-22 09:30:05 +00:00
|
|
|
/*
|
|
|
|
* If non-NULL, and content_type above is non-NULL, returns
|
|
|
|
* the charset parameter from the content-type. If none is
|
|
|
|
* present, returns an empty string.
|
|
|
|
*/
|
|
|
|
struct strbuf *charset;
|
|
|
|
|
2013-09-28 08:32:02 +00:00
|
|
|
/*
|
|
|
|
* If non-NULL, returns the URL we ended up at, including any
|
|
|
|
* redirects we followed.
|
|
|
|
*/
|
|
|
|
struct strbuf *effective_url;
|
http: update base URLs when we see redirects
If a caller asks the http_get_* functions to go to a
particular URL and we end up elsewhere due to a redirect,
the effective_url field can tell us where we went.
It would be nice to remember this redirect and short-cut
further requests for two reasons:
1. It's more efficient. Otherwise we spend an extra http
round-trip to the server for each subsequent request,
just to get redirected.
2. If we end up with an http 401 and are going to ask for
credentials, it is to feed them to the redirect target.
If the redirect is an http->https upgrade, this means
our credentials may be provided on the http leg, just
to end up redirected to https. And if the redirect
crosses server boundaries, then curl will drop the
credentials entirely as it follows the redirect.
However, it, it is not enough to simply record the effective
URL we saw and use that for subsequent requests. We were
originally fed a "base" url like:
http://example.com/foo.git
and we want to figure out what the new base is, even though
the URLs we see may be:
original: http://example.com/foo.git/info/refs
effective: http://example.com/bar.git/info/refs
Subsequent requests will not be for "info/refs", but for
other paths relative to the base. We must ask the caller to
pass in the original base, and we must pass the redirected
base back to the caller (so that it can generate more URLs
from it). Furthermore, we need to feed the new base to the
credential code, so that requests to credential helpers (or
to the user) match the URL we will be requesting.
This patch teaches http_request_reauth to do this munging.
Since it is the caller who cares about making more URLs, it
seems at first glance that callers could simply check
effective_url themselves and handle it. However, since we
need to update the credential struct before the second
re-auth request, we have to do it inside http_request_reauth.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
2013-09-28 08:34:05 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If both base_url and effective_url are non-NULL, the base URL will
|
|
|
|
* be munged to reflect any redirections going from the requested url
|
|
|
|
* to effective_url. See the definition of update_url_from_redirect
|
|
|
|
* for details.
|
|
|
|
*/
|
|
|
|
struct strbuf *base_url;
|
2018-03-15 17:31:38 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If not NULL, contains additional HTTP headers to be sent with the
|
|
|
|
* request. The strings in the list must not be freed until after the
|
|
|
|
* request has completed.
|
|
|
|
*/
|
|
|
|
struct string_list *extra_headers;
|
2013-09-28 08:31:23 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
/* Return values for http_get_*() */
|
2009-06-06 08:43:53 +00:00
|
|
|
#define HTTP_OK 0
|
|
|
|
#define HTTP_MISSING_TARGET 1
|
|
|
|
#define HTTP_ERROR 2
|
|
|
|
#define HTTP_START_FAILED 3
|
2010-04-01 22:14:35 +00:00
|
|
|
#define HTTP_REAUTH 4
|
|
|
|
#define HTTP_NOAUTH 5
|
2021-09-24 10:08:20 +00:00
|
|
|
#define HTTP_NOMATCHPUBLICKEY 6
|
2009-06-06 08:43:53 +00:00
|
|
|
|
|
|
|
/*
|
2012-03-28 08:41:54 +00:00
|
|
|
* Requests a URL and stores the result in a strbuf.
|
2009-06-06 08:43:53 +00:00
|
|
|
*
|
|
|
|
* If the result pointer is NULL, a HTTP HEAD request is made instead of GET.
|
|
|
|
*/
|
2013-09-28 08:31:23 +00:00
|
|
|
int http_get_strbuf(const char *url, struct strbuf *result, struct http_get_options *options);
|
2009-06-06 08:43:53 +00:00
|
|
|
|
2022-05-16 20:11:02 +00:00
|
|
|
/*
|
|
|
|
* Downloads a URL and stores the result in the given file.
|
|
|
|
*
|
|
|
|
* If a previous interrupted download is detected (i.e. a previous temporary
|
|
|
|
* file is still around) the download is resumed.
|
|
|
|
*/
|
|
|
|
int http_get_file(const char *url, const char *filename,
|
|
|
|
struct http_get_options *options);
|
|
|
|
|
2019-04-29 08:28:14 +00:00
|
|
|
int http_fetch_ref(const char *base, struct ref *ref);
|
2007-12-10 23:08:25 +00:00
|
|
|
|
2009-06-06 08:43:59 +00:00
|
|
|
/* Helpers for fetching packs */
|
2019-04-29 08:28:14 +00:00
|
|
|
int http_get_info_packs(const char *base_url,
|
2019-04-29 08:28:23 +00:00
|
|
|
struct packed_git **packs_head);
|
2009-06-06 08:43:59 +00:00
|
|
|
|
2022-07-11 05:58:54 +00:00
|
|
|
/* Helper for getting Accept-Language header */
|
|
|
|
const char *http_get_accept_language_header(void);
|
|
|
|
|
2011-03-16 07:08:34 +00:00
|
|
|
struct http_pack_request {
|
2009-06-06 08:44:01 +00:00
|
|
|
char *url;
|
2020-06-10 20:57:18 +00:00
|
|
|
|
|
|
|
/*
|
2021-02-22 19:20:06 +00:00
|
|
|
* index-pack command to run. Must be terminated by NULL.
|
|
|
|
*
|
|
|
|
* If NULL, defaults to {"index-pack", "--stdin", NULL}.
|
2020-06-10 20:57:18 +00:00
|
|
|
*/
|
2021-02-22 19:20:06 +00:00
|
|
|
const char **index_pack_args;
|
|
|
|
unsigned preserve_index_pack_stdout : 1;
|
2020-06-10 20:57:18 +00:00
|
|
|
|
2009-06-06 08:44:01 +00:00
|
|
|
FILE *packfile;
|
2018-05-19 01:56:37 +00:00
|
|
|
struct strbuf tmpfile;
|
2009-06-06 08:44:01 +00:00
|
|
|
struct active_request_slot *slot;
|
2024-04-17 00:02:27 +00:00
|
|
|
struct curl_slist *headers;
|
2009-06-06 08:44:01 +00:00
|
|
|
};
|
|
|
|
|
2019-04-29 08:28:14 +00:00
|
|
|
struct http_pack_request *new_http_pack_request(
|
http: refactor finish_http_pack_request()
finish_http_pack_request() does multiple tasks, including some
housekeeping on a struct packed_git - (1) closing its index, (2)
removing it from a list, and (3) installing it. These concerns are
independent of fetching a pack through HTTP: they are there only because
(1) the calling code opens the pack's index before deciding to fetch it,
(2) the calling code maintains a list of packfiles that can be fetched,
and (3) the calling code fetches it in order to make use of its objects
in the same process.
In preparation for a subsequent commit, which adds a feature that does
not need any of this housekeeping, remove (1), (2), and (3) from
finish_http_pack_request(). (2) and (3) are now done by a helper
function, and (1) is the responsibility of the caller (in this patch,
done closer to the point where the pack index is opened).
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-06-10 20:57:16 +00:00
|
|
|
const unsigned char *packed_git_hash, const char *base_url);
|
2020-06-10 20:57:18 +00:00
|
|
|
struct http_pack_request *new_direct_http_pack_request(
|
|
|
|
const unsigned char *packed_git_hash, char *url);
|
2019-04-29 08:28:14 +00:00
|
|
|
int finish_http_pack_request(struct http_pack_request *preq);
|
|
|
|
void release_http_pack_request(struct http_pack_request *preq);
|
2009-06-06 08:44:01 +00:00
|
|
|
|
http: refactor finish_http_pack_request()
finish_http_pack_request() does multiple tasks, including some
housekeeping on a struct packed_git - (1) closing its index, (2)
removing it from a list, and (3) installing it. These concerns are
independent of fetching a pack through HTTP: they are there only because
(1) the calling code opens the pack's index before deciding to fetch it,
(2) the calling code maintains a list of packfiles that can be fetched,
and (3) the calling code fetches it in order to make use of its objects
in the same process.
In preparation for a subsequent commit, which adds a feature that does
not need any of this housekeeping, remove (1), (2), and (3) from
finish_http_pack_request(). (2) and (3) are now done by a helper
function, and (1) is the responsibility of the caller (in this patch,
done closer to the point where the pack index is opened).
Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2020-06-10 20:57:16 +00:00
|
|
|
/*
|
|
|
|
* Remove p from the given list, and invoke install_packed_git() on it.
|
|
|
|
*
|
|
|
|
* This is a convenience function for users that have obtained a list of packs
|
|
|
|
* from http_get_info_packs() and have chosen a specific pack to fetch.
|
|
|
|
*/
|
|
|
|
void http_install_packfile(struct packed_git *p,
|
|
|
|
struct packed_git **list_to_remove_from);
|
|
|
|
|
http*: add helper methods for fetching objects (loose)
The code handling the fetching of loose objects in http-push.c and
http-walker.c have been refactored into new methods and a new struct
(object_http_request) in http.c. They are not meant to be invoked
elsewhere.
The new methods in http.c are
- new_http_object_request
- process_http_object_request
- finish_http_object_request
- abort_http_object_request
- release_http_object_request
and the new struct is http_object_request.
RANGER_HEADER_SIZE and no_pragma_header is no longer made available
outside of http.c, since after the above changes, there are no other
instances of usage outside of http.c.
Remove members of the transfer_request struct in http-push.c and
http-walker.c, including filename, real_sha1 and zret, as they are used
no longer used.
Move the methods append_remote_object_url() and get_remote_object_url()
from http-push.c to http.c. Additionally, get_remote_object_url() is no
longer defined only when USE_CURL_MULTI is defined, since
non-USE_CURL_MULTI code in http.c uses it (namely, in
new_http_object_request()).
Refactor code from http-push.c::start_fetch_loose() and
http-walker.c::start_object_fetch_request() that deals with the details
of coming up with the filename to store the retrieved object, resuming
a previously aborted request, and making a new curl request, into a new
function, new_http_object_request().
Refactor code from http-walker.c::process_object_request() into the
function, process_http_object_request().
Refactor code from http-push.c::finish_request() and
http-walker.c::finish_object_request() into a new function,
finish_http_object_request(). It returns the result of the
move_temp_to_file() invocation.
Add a function, release_http_object_request(), which cleans up object
request data. http-push.c and http-walker.c invoke this function
separately; http-push.c::release_request() and
http-walker.c::release_object_request() do not invoke this function.
Add a function, abort_http_object_request(), which unlink()s the object
file and invokes release_http_object_request(). Update
http-walker.c::abort_object_request() to use this.
Signed-off-by: Tay Ray Chuan <rctay89@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-06-06 08:44:02 +00:00
|
|
|
/* Helpers for fetching object */
|
2011-03-16 07:08:34 +00:00
|
|
|
struct http_object_request {
|
http*: add helper methods for fetching objects (loose)
The code handling the fetching of loose objects in http-push.c and
http-walker.c have been refactored into new methods and a new struct
(object_http_request) in http.c. They are not meant to be invoked
elsewhere.
The new methods in http.c are
- new_http_object_request
- process_http_object_request
- finish_http_object_request
- abort_http_object_request
- release_http_object_request
and the new struct is http_object_request.
RANGER_HEADER_SIZE and no_pragma_header is no longer made available
outside of http.c, since after the above changes, there are no other
instances of usage outside of http.c.
Remove members of the transfer_request struct in http-push.c and
http-walker.c, including filename, real_sha1 and zret, as they are used
no longer used.
Move the methods append_remote_object_url() and get_remote_object_url()
from http-push.c to http.c. Additionally, get_remote_object_url() is no
longer defined only when USE_CURL_MULTI is defined, since
non-USE_CURL_MULTI code in http.c uses it (namely, in
new_http_object_request()).
Refactor code from http-push.c::start_fetch_loose() and
http-walker.c::start_object_fetch_request() that deals with the details
of coming up with the filename to store the retrieved object, resuming
a previously aborted request, and making a new curl request, into a new
function, new_http_object_request().
Refactor code from http-walker.c::process_object_request() into the
function, process_http_object_request().
Refactor code from http-push.c::finish_request() and
http-walker.c::finish_object_request() into a new function,
finish_http_object_request(). It returns the result of the
move_temp_to_file() invocation.
Add a function, release_http_object_request(), which cleans up object
request data. http-push.c and http-walker.c invoke this function
separately; http-push.c::release_request() and
http-walker.c::release_object_request() do not invoke this function.
Add a function, abort_http_object_request(), which unlink()s the object
file and invokes release_http_object_request(). Update
http-walker.c::abort_object_request() to use this.
Signed-off-by: Tay Ray Chuan <rctay89@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-06-06 08:44:02 +00:00
|
|
|
char *url;
|
2018-05-19 01:56:37 +00:00
|
|
|
struct strbuf tmpfile;
|
http*: add helper methods for fetching objects (loose)
The code handling the fetching of loose objects in http-push.c and
http-walker.c have been refactored into new methods and a new struct
(object_http_request) in http.c. They are not meant to be invoked
elsewhere.
The new methods in http.c are
- new_http_object_request
- process_http_object_request
- finish_http_object_request
- abort_http_object_request
- release_http_object_request
and the new struct is http_object_request.
RANGER_HEADER_SIZE and no_pragma_header is no longer made available
outside of http.c, since after the above changes, there are no other
instances of usage outside of http.c.
Remove members of the transfer_request struct in http-push.c and
http-walker.c, including filename, real_sha1 and zret, as they are used
no longer used.
Move the methods append_remote_object_url() and get_remote_object_url()
from http-push.c to http.c. Additionally, get_remote_object_url() is no
longer defined only when USE_CURL_MULTI is defined, since
non-USE_CURL_MULTI code in http.c uses it (namely, in
new_http_object_request()).
Refactor code from http-push.c::start_fetch_loose() and
http-walker.c::start_object_fetch_request() that deals with the details
of coming up with the filename to store the retrieved object, resuming
a previously aborted request, and making a new curl request, into a new
function, new_http_object_request().
Refactor code from http-walker.c::process_object_request() into the
function, process_http_object_request().
Refactor code from http-push.c::finish_request() and
http-walker.c::finish_object_request() into a new function,
finish_http_object_request(). It returns the result of the
move_temp_to_file() invocation.
Add a function, release_http_object_request(), which cleans up object
request data. http-push.c and http-walker.c invoke this function
separately; http-push.c::release_request() and
http-walker.c::release_object_request() do not invoke this function.
Add a function, abort_http_object_request(), which unlink()s the object
file and invokes release_http_object_request(). Update
http-walker.c::abort_object_request() to use this.
Signed-off-by: Tay Ray Chuan <rctay89@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-06-06 08:44:02 +00:00
|
|
|
int localfile;
|
|
|
|
CURLcode curl_result;
|
|
|
|
char errorstr[CURL_ERROR_SIZE];
|
|
|
|
long http_code;
|
2019-01-07 08:34:40 +00:00
|
|
|
struct object_id oid;
|
|
|
|
struct object_id real_oid;
|
2019-02-19 00:05:14 +00:00
|
|
|
git_hash_ctx c;
|
2011-06-10 18:52:15 +00:00
|
|
|
git_zstream stream;
|
http*: add helper methods for fetching objects (loose)
The code handling the fetching of loose objects in http-push.c and
http-walker.c have been refactored into new methods and a new struct
(object_http_request) in http.c. They are not meant to be invoked
elsewhere.
The new methods in http.c are
- new_http_object_request
- process_http_object_request
- finish_http_object_request
- abort_http_object_request
- release_http_object_request
and the new struct is http_object_request.
RANGER_HEADER_SIZE and no_pragma_header is no longer made available
outside of http.c, since after the above changes, there are no other
instances of usage outside of http.c.
Remove members of the transfer_request struct in http-push.c and
http-walker.c, including filename, real_sha1 and zret, as they are used
no longer used.
Move the methods append_remote_object_url() and get_remote_object_url()
from http-push.c to http.c. Additionally, get_remote_object_url() is no
longer defined only when USE_CURL_MULTI is defined, since
non-USE_CURL_MULTI code in http.c uses it (namely, in
new_http_object_request()).
Refactor code from http-push.c::start_fetch_loose() and
http-walker.c::start_object_fetch_request() that deals with the details
of coming up with the filename to store the retrieved object, resuming
a previously aborted request, and making a new curl request, into a new
function, new_http_object_request().
Refactor code from http-walker.c::process_object_request() into the
function, process_http_object_request().
Refactor code from http-push.c::finish_request() and
http-walker.c::finish_object_request() into a new function,
finish_http_object_request(). It returns the result of the
move_temp_to_file() invocation.
Add a function, release_http_object_request(), which cleans up object
request data. http-push.c and http-walker.c invoke this function
separately; http-push.c::release_request() and
http-walker.c::release_object_request() do not invoke this function.
Add a function, abort_http_object_request(), which unlink()s the object
file and invokes release_http_object_request(). Update
http-walker.c::abort_object_request() to use this.
Signed-off-by: Tay Ray Chuan <rctay89@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-06-06 08:44:02 +00:00
|
|
|
int zret;
|
|
|
|
int rename;
|
|
|
|
struct active_request_slot *slot;
|
2024-04-17 00:02:27 +00:00
|
|
|
struct curl_slist *headers;
|
http*: add helper methods for fetching objects (loose)
The code handling the fetching of loose objects in http-push.c and
http-walker.c have been refactored into new methods and a new struct
(object_http_request) in http.c. They are not meant to be invoked
elsewhere.
The new methods in http.c are
- new_http_object_request
- process_http_object_request
- finish_http_object_request
- abort_http_object_request
- release_http_object_request
and the new struct is http_object_request.
RANGER_HEADER_SIZE and no_pragma_header is no longer made available
outside of http.c, since after the above changes, there are no other
instances of usage outside of http.c.
Remove members of the transfer_request struct in http-push.c and
http-walker.c, including filename, real_sha1 and zret, as they are used
no longer used.
Move the methods append_remote_object_url() and get_remote_object_url()
from http-push.c to http.c. Additionally, get_remote_object_url() is no
longer defined only when USE_CURL_MULTI is defined, since
non-USE_CURL_MULTI code in http.c uses it (namely, in
new_http_object_request()).
Refactor code from http-push.c::start_fetch_loose() and
http-walker.c::start_object_fetch_request() that deals with the details
of coming up with the filename to store the retrieved object, resuming
a previously aborted request, and making a new curl request, into a new
function, new_http_object_request().
Refactor code from http-walker.c::process_object_request() into the
function, process_http_object_request().
Refactor code from http-push.c::finish_request() and
http-walker.c::finish_object_request() into a new function,
finish_http_object_request(). It returns the result of the
move_temp_to_file() invocation.
Add a function, release_http_object_request(), which cleans up object
request data. http-push.c and http-walker.c invoke this function
separately; http-push.c::release_request() and
http-walker.c::release_object_request() do not invoke this function.
Add a function, abort_http_object_request(), which unlink()s the object
file and invokes release_http_object_request(). Update
http-walker.c::abort_object_request() to use this.
Signed-off-by: Tay Ray Chuan <rctay89@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-06-06 08:44:02 +00:00
|
|
|
};
|
|
|
|
|
2019-04-29 08:28:14 +00:00
|
|
|
struct http_object_request *new_http_object_request(
|
2019-01-07 08:34:40 +00:00
|
|
|
const char *base_url, const struct object_id *oid);
|
2019-04-29 08:28:14 +00:00
|
|
|
void process_http_object_request(struct http_object_request *freq);
|
|
|
|
int finish_http_object_request(struct http_object_request *freq);
|
|
|
|
void abort_http_object_request(struct http_object_request *freq);
|
|
|
|
void release_http_object_request(struct http_object_request *freq);
|
http*: add helper methods for fetching objects (loose)
The code handling the fetching of loose objects in http-push.c and
http-walker.c have been refactored into new methods and a new struct
(object_http_request) in http.c. They are not meant to be invoked
elsewhere.
The new methods in http.c are
- new_http_object_request
- process_http_object_request
- finish_http_object_request
- abort_http_object_request
- release_http_object_request
and the new struct is http_object_request.
RANGER_HEADER_SIZE and no_pragma_header is no longer made available
outside of http.c, since after the above changes, there are no other
instances of usage outside of http.c.
Remove members of the transfer_request struct in http-push.c and
http-walker.c, including filename, real_sha1 and zret, as they are used
no longer used.
Move the methods append_remote_object_url() and get_remote_object_url()
from http-push.c to http.c. Additionally, get_remote_object_url() is no
longer defined only when USE_CURL_MULTI is defined, since
non-USE_CURL_MULTI code in http.c uses it (namely, in
new_http_object_request()).
Refactor code from http-push.c::start_fetch_loose() and
http-walker.c::start_object_fetch_request() that deals with the details
of coming up with the filename to store the retrieved object, resuming
a previously aborted request, and making a new curl request, into a new
function, new_http_object_request().
Refactor code from http-walker.c::process_object_request() into the
function, process_http_object_request().
Refactor code from http-push.c::finish_request() and
http-walker.c::finish_object_request() into a new function,
finish_http_object_request(). It returns the result of the
move_temp_to_file() invocation.
Add a function, release_http_object_request(), which cleans up object
request data. http-push.c and http-walker.c invoke this function
separately; http-push.c::release_request() and
http-walker.c::release_object_request() do not invoke this function.
Add a function, abort_http_object_request(), which unlink()s the object
file and invokes release_http_object_request(). Update
http-walker.c::abort_object_request() to use this.
Signed-off-by: Tay Ray Chuan <rctay89@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-06-06 08:44:02 +00:00
|
|
|
|
2020-05-11 17:43:10 +00:00
|
|
|
/*
|
|
|
|
* Instead of using environment variables to determine if curl tracing happens,
|
|
|
|
* behave as if GIT_TRACE_CURL=1 and GIT_TRACE_CURL_NO_DATA=1 is set. Call this
|
|
|
|
* before calling setup_curl_trace().
|
|
|
|
*/
|
|
|
|
void http_trace_curl_no_data(void);
|
|
|
|
|
2016-05-23 13:44:02 +00:00
|
|
|
/* setup routine for curl_easy_setopt CURLOPT_DEBUGFUNCTION */
|
|
|
|
void setup_curl_trace(CURL *handle);
|
2005-11-18 19:02:58 +00:00
|
|
|
#endif /* HTTP_H */
|