2005-04-18 20:04:43 +00:00
|
|
|
/*
|
|
|
|
* GIT - The information manager from hell
|
|
|
|
*
|
|
|
|
* Copyright (C) Linus Torvalds, 2005
|
|
|
|
*
|
|
|
|
* This handles basic git sha1 object files - packing, unpacking,
|
|
|
|
* creation etc.
|
|
|
|
*/
|
|
|
|
#include "cache.h"
|
2017-06-14 18:07:36 +00:00
|
|
|
#include "config.h"
|
2012-11-05 08:41:22 +00:00
|
|
|
#include "string-list.h"
|
2014-10-01 10:28:42 +00:00
|
|
|
#include "lockfile.h"
|
2005-06-27 10:35:33 +00:00
|
|
|
#include "delta.h"
|
2005-06-28 21:21:02 +00:00
|
|
|
#include "pack.h"
|
2006-04-02 12:44:09 +00:00
|
|
|
#include "blob.h"
|
|
|
|
#include "commit.h"
|
2011-05-08 08:47:35 +00:00
|
|
|
#include "run-command.h"
|
2006-04-02 12:44:09 +00:00
|
|
|
#include "tag.h"
|
|
|
|
#include "tree.h"
|
2011-02-05 10:52:21 +00:00
|
|
|
#include "tree-walk.h"
|
2007-04-10 04:20:29 +00:00
|
|
|
#include "refs.h"
|
2008-02-28 05:25:19 +00:00
|
|
|
#include "pack-revindex.h"
|
sha1-lookup: more memory efficient search in sorted list of SHA-1
Currently, when looking for a packed object from the pack idx, a
simple binary search is used.
A conventional binary search loop looks like this:
unsigned lo, hi;
do {
unsigned mi = (lo + hi) / 2;
int cmp = "entry pointed at by mi" minus "target";
if (!cmp)
return mi; "mi is the wanted one"
if (cmp > 0)
hi = mi; "mi is larger than target"
else
lo = mi+1; "mi is smaller than target"
} while (lo < hi);
"did not find what we wanted"
The invariants are:
- When entering the loop, 'lo' points at a slot that is never
above the target (it could be at the target), 'hi' points at
a slot that is guaranteed to be above the target (it can
never be at the target).
- We find a point 'mi' between 'lo' and 'hi' ('mi' could be
the same as 'lo', but never can be as high as 'hi'), and
check if 'mi' hits the target. There are three cases:
- if it is a hit, we have found what we are looking for;
- if it is strictly higher than the target, we set it to
'hi', and repeat the search.
- if it is strictly lower than the target, we update 'lo'
to one slot after it, because we allow 'lo' to be at the
target and 'mi' is known to be below the target.
If the loop exits, there is no matching entry.
When choosing 'mi', we do not have to take the "middle" but
anywhere in between 'lo' and 'hi', as long as lo <= mi < hi is
satisfied. When we somehow know that the distance between the
target and 'lo' is much shorter than the target and 'hi', we
could pick 'mi' that is much closer to 'lo' than (hi+lo)/2,
which a conventional binary search would pick.
This patch takes advantage of the fact that the SHA-1 is a good
hash function, and as long as there are enough entries in the
table, we can expect uniform distribution. An entry that begins
with for example "deadbeef..." is much likely to appear much
later than in the midway of a reasonably populated table. In
fact, it can be expected to be near 87% (222/256) from the top
of the table.
This is a work-in-progress and has switches to allow easier
experiments and debugging. Exporting GIT_USE_LOOKUP environment
variable enables this code.
On my admittedly memory starved machine, with a partial KDE
repository (3.0G pack with 95M idx):
$ GIT_USE_LOOKUP=t git log -800 --stat HEAD >/dev/null
3.93user 0.16system 0:04.09elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+55588minor)pagefaults 0swaps
Without the patch, the numbers are:
$ git log -800 --stat HEAD >/dev/null
4.00user 0.15system 0:04.17elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+60258minor)pagefaults 0swaps
In the same repository:
$ GIT_USE_LOOKUP=t git log -2000 HEAD >/dev/null
0.12user 0.00system 0:00.12elapsed 97%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+4241minor)pagefaults 0swaps
Without the patch, the numbers are:
$ git log -2000 HEAD >/dev/null
0.05user 0.01system 0:00.07elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+8506minor)pagefaults 0swaps
There isn't much time difference, but the number of minor faults
seems to show that we are touching much smaller number of pages,
which is expected.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-12-29 10:05:47 +00:00
|
|
|
#include "sha1-lookup.h"
|
2011-10-28 21:48:40 +00:00
|
|
|
#include "bulk-checkin.h"
|
2018-03-23 17:20:57 +00:00
|
|
|
#include "repository.h"
|
2018-04-12 00:21:06 +00:00
|
|
|
#include "replace-object.h"
|
2012-03-07 10:54:18 +00:00
|
|
|
#include "streaming.h"
|
2013-02-15 12:07:10 +00:00
|
|
|
#include "dir.h"
|
2016-08-22 21:59:42 +00:00
|
|
|
#include "list.h"
|
2016-09-13 17:54:42 +00:00
|
|
|
#include "mergesort.h"
|
alternates: accept double-quoted paths
We read lists of alternates from objects/info/alternates
files (delimited by newline), as well as from the
GIT_ALTERNATE_OBJECT_DIRECTORIES environment variable
(delimited by colon or semi-colon, depending on the
platform).
There's no mechanism for quoting the delimiters, so it's
impossible to specify an alternate path that contains a
colon in the environment, or one that contains a newline in
a file. We've lived with that restriction for ages because
both alternates and filenames with colons are relatively
rare, and it's only a problem when the two meet. But since
722ff7f87 (receive-pack: quarantine objects until
pre-receive accepts, 2016-10-03), which builds on the
alternates system, every push causes the receiver to set
GIT_ALTERNATE_OBJECT_DIRECTORIES internally.
It would be convenient to have some way to quote the
delimiter so that we can represent arbitrary paths.
The simplest thing would be an escape character before a
quoted delimiter (e.g., "\:" as a literal colon). But that
creates a backwards compatibility problem: any path which
uses that escape character is now broken, and we've just
shifted the problem. We could choose an unlikely escape
character (e.g., something from the non-printable ASCII
range), but that's awkward to use.
Instead, let's treat names as unquoted unless they begin
with a double-quote, in which case they are interpreted via
our usual C-stylke quoting rules. This also breaks
backwards-compatibility, but in a smaller way: it only
matters if your file has a double-quote as the very _first_
character in the path (whereas an escape character is a
problem anywhere in the path). It's also consistent with
many other parts of git, which accept either a bare pathname
or a double-quoted one, and the sender can choose to quote
or not as required.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-12-12 19:52:22 +00:00
|
|
|
#include "quote.h"
|
2017-08-18 22:20:16 +00:00
|
|
|
#include "packfile.h"
|
2017-12-08 15:27:14 +00:00
|
|
|
#include "fetch-object.h"
|
2018-03-23 17:20:55 +00:00
|
|
|
#include "object-store.h"
|
2007-01-10 04:07:11 +00:00
|
|
|
|
2018-03-12 02:27:55 +00:00
|
|
|
/* The maximum size for an object header. */
|
|
|
|
#define MAX_HEADER_LEN 32
|
|
|
|
|
2018-05-02 00:26:07 +00:00
|
|
|
|
|
|
|
#define EMPTY_TREE_SHA1_BIN_LITERAL \
|
|
|
|
"\x4b\x82\x5d\xc6\x42\xcb\x6e\xb9\xa0\x60" \
|
|
|
|
"\xe5\x4b\xf8\xd6\x92\x88\xfb\xee\x49\x04"
|
|
|
|
|
|
|
|
#define EMPTY_BLOB_SHA1_BIN_LITERAL \
|
|
|
|
"\xe6\x9d\xe2\x9b\xb2\xd1\xd6\x43\x4b\x8b" \
|
|
|
|
"\x29\xae\x77\x5a\xd8\xc2\xe4\x8c\x53\x91"
|
|
|
|
|
2017-08-18 01:59:35 +00:00
|
|
|
const unsigned char null_sha1[GIT_MAX_RAWSZ];
|
2015-12-06 22:16:35 +00:00
|
|
|
const struct object_id null_oid;
|
2018-05-02 00:26:07 +00:00
|
|
|
static const struct object_id empty_tree_oid = {
|
2016-08-31 23:27:18 +00:00
|
|
|
EMPTY_TREE_SHA1_BIN_LITERAL
|
|
|
|
};
|
2018-05-02 00:26:07 +00:00
|
|
|
static const struct object_id empty_blob_oid = {
|
2016-08-31 23:27:18 +00:00
|
|
|
EMPTY_BLOB_SHA1_BIN_LITERAL
|
|
|
|
};
|
2005-09-30 21:02:47 +00:00
|
|
|
|
2018-02-01 02:18:38 +00:00
|
|
|
static void git_hash_sha1_init(git_hash_ctx *ctx)
|
2017-11-12 21:28:52 +00:00
|
|
|
{
|
2018-02-01 02:18:38 +00:00
|
|
|
git_SHA1_Init(&ctx->sha1);
|
2017-11-12 21:28:52 +00:00
|
|
|
}
|
|
|
|
|
2018-02-01 02:18:38 +00:00
|
|
|
static void git_hash_sha1_update(git_hash_ctx *ctx, const void *data, size_t len)
|
2017-11-12 21:28:52 +00:00
|
|
|
{
|
2018-02-01 02:18:38 +00:00
|
|
|
git_SHA1_Update(&ctx->sha1, data, len);
|
2017-11-12 21:28:52 +00:00
|
|
|
}
|
|
|
|
|
2018-02-01 02:18:38 +00:00
|
|
|
static void git_hash_sha1_final(unsigned char *hash, git_hash_ctx *ctx)
|
2017-11-12 21:28:52 +00:00
|
|
|
{
|
2018-02-01 02:18:38 +00:00
|
|
|
git_SHA1_Final(hash, &ctx->sha1);
|
2017-11-12 21:28:52 +00:00
|
|
|
}
|
|
|
|
|
2018-02-01 02:18:38 +00:00
|
|
|
static void git_hash_unknown_init(git_hash_ctx *ctx)
|
2017-11-12 21:28:52 +00:00
|
|
|
{
|
2018-07-21 07:49:19 +00:00
|
|
|
BUG("trying to init unknown hash");
|
2017-11-12 21:28:52 +00:00
|
|
|
}
|
|
|
|
|
2018-02-01 02:18:38 +00:00
|
|
|
static void git_hash_unknown_update(git_hash_ctx *ctx, const void *data, size_t len)
|
2017-11-12 21:28:52 +00:00
|
|
|
{
|
2018-07-21 07:49:19 +00:00
|
|
|
BUG("trying to update unknown hash");
|
2017-11-12 21:28:52 +00:00
|
|
|
}
|
|
|
|
|
2018-02-01 02:18:38 +00:00
|
|
|
static void git_hash_unknown_final(unsigned char *hash, git_hash_ctx *ctx)
|
2017-11-12 21:28:52 +00:00
|
|
|
{
|
2018-07-21 07:49:19 +00:00
|
|
|
BUG("trying to finalize unknown hash");
|
2017-11-12 21:28:52 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {
|
|
|
|
{
|
|
|
|
NULL,
|
|
|
|
0x00000000,
|
|
|
|
0,
|
|
|
|
0,
|
|
|
|
git_hash_unknown_init,
|
|
|
|
git_hash_unknown_update,
|
|
|
|
git_hash_unknown_final,
|
|
|
|
NULL,
|
|
|
|
NULL,
|
|
|
|
},
|
|
|
|
{
|
|
|
|
"sha-1",
|
|
|
|
/* "sha1", big-endian */
|
|
|
|
0x73686131,
|
|
|
|
GIT_SHA1_RAWSZ,
|
|
|
|
GIT_SHA1_HEXSZ,
|
|
|
|
git_hash_sha1_init,
|
|
|
|
git_hash_sha1_update,
|
|
|
|
git_hash_sha1_final,
|
|
|
|
&empty_tree_oid,
|
|
|
|
&empty_blob_oid,
|
|
|
|
},
|
|
|
|
};
|
|
|
|
|
2018-05-02 00:25:54 +00:00
|
|
|
const char *empty_tree_oid_hex(void)
|
|
|
|
{
|
|
|
|
static char buf[GIT_MAX_HEXSZ + 1];
|
|
|
|
return oid_to_hex_r(buf, the_hash_algo->empty_tree);
|
|
|
|
}
|
|
|
|
|
|
|
|
const char *empty_blob_oid_hex(void)
|
|
|
|
{
|
|
|
|
static char buf[GIT_MAX_HEXSZ + 1];
|
|
|
|
return oid_to_hex_r(buf, the_hash_algo->empty_blob);
|
|
|
|
}
|
|
|
|
|
2011-02-05 14:03:01 +00:00
|
|
|
/*
|
|
|
|
* This is meant to hold a *small* number of objects that you would
|
|
|
|
* want read_sha1_file() to be able to return, but yet you do not want
|
|
|
|
* to write them into the object store (e.g. a browse-only
|
|
|
|
* application).
|
|
|
|
*/
|
|
|
|
static struct cached_object {
|
2018-05-02 00:26:03 +00:00
|
|
|
struct object_id oid;
|
2011-02-05 14:03:01 +00:00
|
|
|
enum object_type type;
|
|
|
|
void *buf;
|
|
|
|
unsigned long size;
|
|
|
|
} *cached_objects;
|
|
|
|
static int cached_object_nr, cached_object_alloc;
|
|
|
|
|
|
|
|
static struct cached_object empty_tree = {
|
2018-05-02 00:26:03 +00:00
|
|
|
{ EMPTY_TREE_SHA1_BIN_LITERAL },
|
2011-02-05 14:03:01 +00:00
|
|
|
OBJ_TREE,
|
|
|
|
"",
|
|
|
|
0
|
|
|
|
};
|
|
|
|
|
2018-05-02 00:26:03 +00:00
|
|
|
static struct cached_object *find_cached_object(const struct object_id *oid)
|
2011-02-05 14:03:01 +00:00
|
|
|
{
|
|
|
|
int i;
|
|
|
|
struct cached_object *co = cached_objects;
|
|
|
|
|
|
|
|
for (i = 0; i < cached_object_nr; i++, co++) {
|
convert "oidcmp() == 0" to oideq()
Using the more restrictive oideq() should, in the long run,
give the compiler more opportunities to optimize these
callsites. For now, this conversion should be a complete
noop with respect to the generated code.
The result is also perhaps a little more readable, as it
avoids the "zero is equal" idiom. Since it's so prevalent in
C, I think seasoned programmers tend not to even notice it
anymore, but it can sometimes make for awkward double
negations (e.g., we can drop a few !!oidcmp() instances
here).
This patch was generated almost entirely by the included
coccinelle patch. This mechanical conversion should be
completely safe, because we check explicitly for cases where
oidcmp() is compared to 0, which is what oideq() is doing
under the hood. Note that we don't have to catch "!oidcmp()"
separately; coccinelle's standard isomorphisms make sure the
two are treated equivalently.
I say "almost" because I did hand-edit the coccinelle output
to fix up a few style violations (it mostly keeps the
original formatting, but sometimes unwraps long lines).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-08-28 21:22:40 +00:00
|
|
|
if (oideq(&co->oid, oid))
|
2011-02-05 14:03:01 +00:00
|
|
|
return co;
|
|
|
|
}
|
convert "oidcmp() == 0" to oideq()
Using the more restrictive oideq() should, in the long run,
give the compiler more opportunities to optimize these
callsites. For now, this conversion should be a complete
noop with respect to the generated code.
The result is also perhaps a little more readable, as it
avoids the "zero is equal" idiom. Since it's so prevalent in
C, I think seasoned programmers tend not to even notice it
anymore, but it can sometimes make for awkward double
negations (e.g., we can drop a few !!oidcmp() instances
here).
This patch was generated almost entirely by the included
coccinelle patch. This mechanical conversion should be
completely safe, because we check explicitly for cases where
oidcmp() is compared to 0, which is what oideq() is doing
under the hood. Note that we don't have to catch "!oidcmp()"
separately; coccinelle's standard isomorphisms make sure the
two are treated equivalently.
I say "almost" because I did hand-edit the coccinelle output
to fix up a few style violations (it mostly keeps the
original formatting, but sometimes unwraps long lines).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-08-28 21:22:40 +00:00
|
|
|
if (oideq(oid, the_hash_algo->empty_tree))
|
2011-02-05 14:03:01 +00:00
|
|
|
return &empty_tree;
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
2017-11-16 16:38:28 +00:00
|
|
|
|
2018-01-13 22:49:31 +00:00
|
|
|
static int get_conv_flags(unsigned flags)
|
2017-11-16 16:38:28 +00:00
|
|
|
{
|
|
|
|
if (flags & HASH_RENORMALIZE)
|
2018-01-13 22:49:31 +00:00
|
|
|
return CONV_EOL_RENORMALIZE;
|
2017-11-16 16:38:28 +00:00
|
|
|
else if (flags & HASH_WRITE_OBJECT)
|
2018-04-15 18:16:07 +00:00
|
|
|
return global_conv_flags_eol | CONV_WRITE_OBJECT;
|
2017-11-16 16:38:28 +00:00
|
|
|
else
|
2018-01-13 22:49:31 +00:00
|
|
|
return 0;
|
2017-11-16 16:38:28 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
2011-03-11 00:02:50 +00:00
|
|
|
int mkdir_in_gitdir(const char *path)
|
|
|
|
{
|
|
|
|
if (mkdir(path, 0777)) {
|
|
|
|
int saved_errno = errno;
|
|
|
|
struct stat st;
|
|
|
|
struct strbuf sb = STRBUF_INIT;
|
|
|
|
|
|
|
|
if (errno != EEXIST)
|
|
|
|
return -1;
|
|
|
|
/*
|
|
|
|
* Are we looking at a path in a symlinked worktree
|
|
|
|
* whose original repository does not yet have it?
|
|
|
|
* e.g. .git/rr-cache pointing at its original
|
|
|
|
* repository in which the user hasn't performed any
|
|
|
|
* conflict resolution yet?
|
|
|
|
*/
|
|
|
|
if (lstat(path, &st) || !S_ISLNK(st.st_mode) ||
|
|
|
|
strbuf_readlink(&sb, path, st.st_size) ||
|
|
|
|
!is_absolute_path(sb.buf) ||
|
|
|
|
mkdir(sb.buf, 0777)) {
|
|
|
|
strbuf_release(&sb);
|
|
|
|
errno = saved_errno;
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
strbuf_release(&sb);
|
|
|
|
}
|
|
|
|
return adjust_shared_perm(path);
|
|
|
|
}
|
|
|
|
|
2014-01-06 13:45:25 +00:00
|
|
|
enum scld_error safe_create_leading_directories(char *path)
|
2005-07-06 08:11:52 +00:00
|
|
|
{
|
2014-01-06 13:45:22 +00:00
|
|
|
char *next_component = path + offset_1st_component(path);
|
2014-01-06 13:45:25 +00:00
|
|
|
enum scld_error ret = SCLD_OK;
|
2006-02-10 01:56:13 +00:00
|
|
|
|
2014-01-06 13:45:25 +00:00
|
|
|
while (ret == SCLD_OK && next_component) {
|
2014-01-06 13:45:20 +00:00
|
|
|
struct stat st;
|
2014-01-18 23:40:44 +00:00
|
|
|
char *slash = next_component, slash_character;
|
2014-01-06 13:45:20 +00:00
|
|
|
|
2014-01-18 23:40:44 +00:00
|
|
|
while (*slash && !is_dir_sep(*slash))
|
|
|
|
slash++;
|
|
|
|
|
|
|
|
if (!*slash)
|
2005-07-06 08:11:52 +00:00
|
|
|
break;
|
2014-01-06 13:45:23 +00:00
|
|
|
|
2014-01-06 13:45:22 +00:00
|
|
|
next_component = slash + 1;
|
2014-01-18 23:40:44 +00:00
|
|
|
while (is_dir_sep(*next_component))
|
2014-01-06 13:45:23 +00:00
|
|
|
next_component++;
|
2014-01-06 13:45:22 +00:00
|
|
|
if (!*next_component)
|
2008-09-02 21:10:15 +00:00
|
|
|
break;
|
2014-01-06 13:45:21 +00:00
|
|
|
|
2014-01-18 23:40:44 +00:00
|
|
|
slash_character = *slash;
|
2014-01-06 13:45:21 +00:00
|
|
|
*slash = '\0';
|
2006-02-10 01:56:13 +00:00
|
|
|
if (!stat(path, &st)) {
|
|
|
|
/* path exists */
|
2017-01-06 16:22:25 +00:00
|
|
|
if (!S_ISDIR(st.st_mode)) {
|
|
|
|
errno = ENOTDIR;
|
2014-01-06 13:45:25 +00:00
|
|
|
ret = SCLD_EXISTS;
|
2017-01-06 16:22:25 +00:00
|
|
|
}
|
2014-01-06 13:45:19 +00:00
|
|
|
} else if (mkdir(path, 0777)) {
|
2013-03-17 14:09:27 +00:00
|
|
|
if (errno == EEXIST &&
|
2014-01-06 13:45:24 +00:00
|
|
|
!stat(path, &st) && S_ISDIR(st.st_mode))
|
2013-03-17 14:09:27 +00:00
|
|
|
; /* somebody created it since we checked */
|
2014-01-06 13:45:27 +00:00
|
|
|
else if (errno == ENOENT)
|
|
|
|
/*
|
|
|
|
* Either mkdir() failed because
|
|
|
|
* somebody just pruned the containing
|
|
|
|
* directory, or stat() failed because
|
|
|
|
* the file that was in our way was
|
|
|
|
* just removed. Either way, inform
|
|
|
|
* the caller that it might be worth
|
|
|
|
* trying again:
|
|
|
|
*/
|
|
|
|
ret = SCLD_VANISHED;
|
2014-01-06 13:45:24 +00:00
|
|
|
else
|
2014-01-06 13:45:25 +00:00
|
|
|
ret = SCLD_FAILED;
|
2014-01-06 13:45:19 +00:00
|
|
|
} else if (adjust_shared_perm(path)) {
|
2014-01-06 13:45:25 +00:00
|
|
|
ret = SCLD_PERMS;
|
2005-12-22 22:13:56 +00:00
|
|
|
}
|
2014-01-18 23:40:44 +00:00
|
|
|
*slash = slash_character;
|
2005-07-06 08:11:52 +00:00
|
|
|
}
|
2014-01-06 13:45:24 +00:00
|
|
|
return ret;
|
2005-07-06 08:11:52 +00:00
|
|
|
}
|
2005-07-05 18:31:32 +00:00
|
|
|
|
2014-01-06 13:45:25 +00:00
|
|
|
enum scld_error safe_create_leading_directories_const(const char *path)
|
2008-06-25 05:41:34 +00:00
|
|
|
{
|
2017-01-06 16:22:24 +00:00
|
|
|
int save_errno;
|
2008-06-25 05:41:34 +00:00
|
|
|
/* path points to cache entries, so xstrdup before messing with it */
|
|
|
|
char *buf = xstrdup(path);
|
2014-01-06 13:45:25 +00:00
|
|
|
enum scld_error result = safe_create_leading_directories(buf);
|
2017-01-06 16:22:24 +00:00
|
|
|
|
|
|
|
save_errno = errno;
|
2008-06-25 05:41:34 +00:00
|
|
|
free(buf);
|
2017-01-06 16:22:24 +00:00
|
|
|
errno = save_errno;
|
2008-06-25 05:41:34 +00:00
|
|
|
return result;
|
|
|
|
}
|
|
|
|
|
2017-01-06 16:22:26 +00:00
|
|
|
int raceproof_create_file(const char *path, create_file_fn fn, void *cb)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* The number of times we will try to remove empty directories
|
|
|
|
* in the way of path. This is only 1 because if another
|
|
|
|
* process is racily creating directories that conflict with
|
|
|
|
* us, we don't want to fight against them.
|
|
|
|
*/
|
|
|
|
int remove_directories_remaining = 1;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The number of times that we will try to create the
|
|
|
|
* directories containing path. We are willing to attempt this
|
|
|
|
* more than once, because another process could be trying to
|
|
|
|
* clean up empty directories at the same time as we are
|
|
|
|
* trying to create them.
|
|
|
|
*/
|
|
|
|
int create_directories_remaining = 3;
|
|
|
|
|
|
|
|
/* A scratch copy of path, filled lazily if we need it: */
|
|
|
|
struct strbuf path_copy = STRBUF_INIT;
|
|
|
|
|
|
|
|
int ret, save_errno;
|
|
|
|
|
|
|
|
/* Sanity check: */
|
|
|
|
assert(*path);
|
|
|
|
|
|
|
|
retry_fn:
|
|
|
|
ret = fn(path, cb);
|
|
|
|
save_errno = errno;
|
|
|
|
if (!ret)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
if (errno == EISDIR && remove_directories_remaining-- > 0) {
|
|
|
|
/*
|
|
|
|
* A directory is in the way. Maybe it is empty; try
|
|
|
|
* to remove it:
|
|
|
|
*/
|
|
|
|
if (!path_copy.len)
|
|
|
|
strbuf_addstr(&path_copy, path);
|
|
|
|
|
|
|
|
if (!remove_dir_recursively(&path_copy, REMOVE_DIR_EMPTY_ONLY))
|
|
|
|
goto retry_fn;
|
|
|
|
} else if (errno == ENOENT && create_directories_remaining-- > 0) {
|
|
|
|
/*
|
|
|
|
* Maybe the containing directory didn't exist, or
|
|
|
|
* maybe it was just deleted by a process that is
|
|
|
|
* racing with us to clean up empty directories. Try
|
|
|
|
* to create it:
|
|
|
|
*/
|
|
|
|
enum scld_error scld_result;
|
|
|
|
|
|
|
|
if (!path_copy.len)
|
|
|
|
strbuf_addstr(&path_copy, path);
|
|
|
|
|
|
|
|
do {
|
|
|
|
scld_result = safe_create_leading_directories(path_copy.buf);
|
|
|
|
if (scld_result == SCLD_OK)
|
|
|
|
goto retry_fn;
|
|
|
|
} while (scld_result == SCLD_VANISHED && create_directories_remaining-- > 0);
|
|
|
|
}
|
|
|
|
|
|
|
|
out:
|
|
|
|
strbuf_release(&path_copy);
|
|
|
|
errno = save_errno;
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
fill_sha1_file: write into a strbuf
It's currently the responsibility of the caller to give
fill_sha1_file() enough bytes to write into, leading them to
manually compute the required lengths. Instead, let's just
write into a strbuf so that it's impossible to get this
wrong.
The alt_odb caller already has a strbuf, so this makes
things strictly simpler. The other caller, sha1_file_name(),
uses a static PATH_MAX buffer and dies when it would
overflow. We can convert this to a static strbuf, which
means our allocation cost is amortized (and as a bonus, we
no longer have to worry about PATH_MAX being too short for
normal use).
This does introduce some small overhead in fill_sha1_file(),
as each strbuf_addchar() will check whether it needs to
grow. However, between the optimization in fec501d
(strbuf_addch: avoid calling strbuf_grow, 2015-04-16) and
the fact that this is not generally called in a tight loop
(after all, the next step is typically to access the file!)
this probably doesn't matter. And even if it did, the right
place to micro-optimize is inside fill_sha1_file(), by
calling a single strbuf_grow() there.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:36:09 +00:00
|
|
|
static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)
|
2005-05-07 07:38:04 +00:00
|
|
|
{
|
|
|
|
int i;
|
2018-07-16 01:28:07 +00:00
|
|
|
for (i = 0; i < the_hash_algo->rawsz; i++) {
|
2005-05-07 07:38:04 +00:00
|
|
|
static char hex[] = "0123456789abcdef";
|
|
|
|
unsigned int val = sha1[i];
|
fill_sha1_file: write into a strbuf
It's currently the responsibility of the caller to give
fill_sha1_file() enough bytes to write into, leading them to
manually compute the required lengths. Instead, let's just
write into a strbuf so that it's impossible to get this
wrong.
The alt_odb caller already has a strbuf, so this makes
things strictly simpler. The other caller, sha1_file_name(),
uses a static PATH_MAX buffer and dies when it would
overflow. We can convert this to a static strbuf, which
means our allocation cost is amortized (and as a bonus, we
no longer have to worry about PATH_MAX being too short for
normal use).
This does introduce some small overhead in fill_sha1_file(),
as each strbuf_addchar() will check whether it needs to
grow. However, between the optimization in fec501d
(strbuf_addch: avoid calling strbuf_grow, 2015-04-16) and
the fact that this is not generally called in a tight loop
(after all, the next step is typically to access the file!)
this probably doesn't matter. And even if it did, the right
place to micro-optimize is inside fill_sha1_file(), by
calling a single strbuf_grow() there.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:36:09 +00:00
|
|
|
strbuf_addch(buf, hex[val >> 4]);
|
|
|
|
strbuf_addch(buf, hex[val & 0xf]);
|
2016-10-03 20:35:55 +00:00
|
|
|
if (!i)
|
fill_sha1_file: write into a strbuf
It's currently the responsibility of the caller to give
fill_sha1_file() enough bytes to write into, leading them to
manually compute the required lengths. Instead, let's just
write into a strbuf so that it's impossible to get this
wrong.
The alt_odb caller already has a strbuf, so this makes
things strictly simpler. The other caller, sha1_file_name(),
uses a static PATH_MAX buffer and dies when it would
overflow. We can convert this to a static strbuf, which
means our allocation cost is amortized (and as a bonus, we
no longer have to worry about PATH_MAX being too short for
normal use).
This does introduce some small overhead in fill_sha1_file(),
as each strbuf_addchar() will check whether it needs to
grow. However, between the optimization in fec501d
(strbuf_addch: avoid calling strbuf_grow, 2015-04-16) and
the fact that this is not generally called in a tight loop
(after all, the next step is typically to access the file!)
this probably doesn't matter. And even if it did, the right
place to micro-optimize is inside fill_sha1_file(), by
calling a single strbuf_grow() there.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:36:09 +00:00
|
|
|
strbuf_addch(buf, '/');
|
2005-05-07 07:38:04 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2018-03-23 17:21:16 +00:00
|
|
|
void sha1_file_name(struct repository *r, struct strbuf *buf, const unsigned char *sha1)
|
2005-04-18 20:04:43 +00:00
|
|
|
{
|
2018-03-23 17:21:16 +00:00
|
|
|
strbuf_addstr(buf, r->objects->objectdir);
|
2018-01-18 10:08:54 +00:00
|
|
|
strbuf_addch(buf, '/');
|
2018-01-17 17:54:54 +00:00
|
|
|
fill_sha1_path(buf, sha1);
|
2005-04-18 20:04:43 +00:00
|
|
|
}
|
|
|
|
|
2016-10-03 20:36:04 +00:00
|
|
|
struct strbuf *alt_scratch_buf(struct alternate_object_database *alt)
|
|
|
|
{
|
|
|
|
strbuf_setlen(&alt->scratch, alt->base_len);
|
|
|
|
return &alt->scratch;
|
|
|
|
}
|
|
|
|
|
2016-10-03 20:35:43 +00:00
|
|
|
static const char *alt_sha1_path(struct alternate_object_database *alt,
|
|
|
|
const unsigned char *sha1)
|
|
|
|
{
|
2016-10-03 20:36:04 +00:00
|
|
|
struct strbuf *buf = alt_scratch_buf(alt);
|
fill_sha1_file: write into a strbuf
It's currently the responsibility of the caller to give
fill_sha1_file() enough bytes to write into, leading them to
manually compute the required lengths. Instead, let's just
write into a strbuf so that it's impossible to get this
wrong.
The alt_odb caller already has a strbuf, so this makes
things strictly simpler. The other caller, sha1_file_name(),
uses a static PATH_MAX buffer and dies when it would
overflow. We can convert this to a static strbuf, which
means our allocation cost is amortized (and as a bonus, we
no longer have to worry about PATH_MAX being too short for
normal use).
This does introduce some small overhead in fill_sha1_file(),
as each strbuf_addchar() will check whether it needs to
grow. However, between the optimization in fec501d
(strbuf_addch: avoid calling strbuf_grow, 2015-04-16) and
the fact that this is not generally called in a tight loop
(after all, the next step is typically to access the file!)
this probably doesn't matter. And even if it did, the right
place to micro-optimize is inside fill_sha1_file(), by
calling a single strbuf_grow() there.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:36:09 +00:00
|
|
|
fill_sha1_path(buf, sha1);
|
2016-10-03 20:36:04 +00:00
|
|
|
return buf->buf;
|
2005-04-18 20:04:43 +00:00
|
|
|
}
|
|
|
|
|
link_alt_odb_entry: refactor string handling
The string handling in link_alt_odb_entry() is mostly an
artifact of the original version, which took the path as a
ptr/len combo, and did not have a NUL-terminated string
until we created one in the alternate_object_database
struct. But since 5bdf0a8 (sha1_file: normalize alt_odb
path before comparing and storing, 2011-09-07), the first
thing we do is put the path into a strbuf, which gives us
some easy opportunities for cleanup.
In particular:
- we call strlen(pathbuf.buf), which is silly; we can look
at pathbuf.len.
- even though we have a strbuf, we don't maintain its
"len" field when chomping extra slashes from the
end, and instead keep a separate "pfxlen" variable. We
can fix this and then drop "pfxlen" entirely.
- we don't check whether the path is usable until after we
allocate the new struct, making extra cleanup work for
ourselves. Since we have a NUL-terminated string, we can
bump the "is it usable" checks higher in the function.
While we're at it, we can move that logic to its own
helper, which makes the flow of link_alt_odb_entry()
easier to follow.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:34:48 +00:00
|
|
|
/*
|
|
|
|
* Return non-zero iff the path is usable as an alternate object database.
|
|
|
|
*/
|
2018-03-23 17:21:03 +00:00
|
|
|
static int alt_odb_usable(struct raw_object_store *o,
|
|
|
|
struct strbuf *path,
|
|
|
|
const char *normalized_objdir)
|
link_alt_odb_entry: refactor string handling
The string handling in link_alt_odb_entry() is mostly an
artifact of the original version, which took the path as a
ptr/len combo, and did not have a NUL-terminated string
until we created one in the alternate_object_database
struct. But since 5bdf0a8 (sha1_file: normalize alt_odb
path before comparing and storing, 2011-09-07), the first
thing we do is put the path into a strbuf, which gives us
some easy opportunities for cleanup.
In particular:
- we call strlen(pathbuf.buf), which is silly; we can look
at pathbuf.len.
- even though we have a strbuf, we don't maintain its
"len" field when chomping extra slashes from the
end, and instead keep a separate "pfxlen" variable. We
can fix this and then drop "pfxlen" entirely.
- we don't check whether the path is usable until after we
allocate the new struct, making extra cleanup work for
ourselves. Since we have a NUL-terminated string, we can
bump the "is it usable" checks higher in the function.
While we're at it, we can move that logic to its own
helper, which makes the flow of link_alt_odb_entry()
easier to follow.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:34:48 +00:00
|
|
|
{
|
|
|
|
struct alternate_object_database *alt;
|
|
|
|
|
|
|
|
/* Detect cases where alternate disappeared */
|
|
|
|
if (!is_directory(path->buf)) {
|
2018-07-21 07:49:39 +00:00
|
|
|
error(_("object directory %s does not exist; "
|
|
|
|
"check .git/objects/info/alternates"),
|
link_alt_odb_entry: refactor string handling
The string handling in link_alt_odb_entry() is mostly an
artifact of the original version, which took the path as a
ptr/len combo, and did not have a NUL-terminated string
until we created one in the alternate_object_database
struct. But since 5bdf0a8 (sha1_file: normalize alt_odb
path before comparing and storing, 2011-09-07), the first
thing we do is put the path into a strbuf, which gives us
some easy opportunities for cleanup.
In particular:
- we call strlen(pathbuf.buf), which is silly; we can look
at pathbuf.len.
- even though we have a strbuf, we don't maintain its
"len" field when chomping extra slashes from the
end, and instead keep a separate "pfxlen" variable. We
can fix this and then drop "pfxlen" entirely.
- we don't check whether the path is usable until after we
allocate the new struct, making extra cleanup work for
ourselves. Since we have a NUL-terminated string, we can
bump the "is it usable" checks higher in the function.
While we're at it, we can move that logic to its own
helper, which makes the flow of link_alt_odb_entry()
easier to follow.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:34:48 +00:00
|
|
|
path->buf);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Prevent the common mistake of listing the same
|
|
|
|
* thing twice, or object directory itself.
|
|
|
|
*/
|
2018-03-23 17:21:03 +00:00
|
|
|
for (alt = o->alt_odb_list; alt; alt = alt->next) {
|
2016-10-03 20:36:26 +00:00
|
|
|
if (!fspathcmp(path->buf, alt->path))
|
link_alt_odb_entry: refactor string handling
The string handling in link_alt_odb_entry() is mostly an
artifact of the original version, which took the path as a
ptr/len combo, and did not have a NUL-terminated string
until we created one in the alternate_object_database
struct. But since 5bdf0a8 (sha1_file: normalize alt_odb
path before comparing and storing, 2011-09-07), the first
thing we do is put the path into a strbuf, which gives us
some easy opportunities for cleanup.
In particular:
- we call strlen(pathbuf.buf), which is silly; we can look
at pathbuf.len.
- even though we have a strbuf, we don't maintain its
"len" field when chomping extra slashes from the
end, and instead keep a separate "pfxlen" variable. We
can fix this and then drop "pfxlen" entirely.
- we don't check whether the path is usable until after we
allocate the new struct, making extra cleanup work for
ourselves. Since we have a NUL-terminated string, we can
bump the "is it usable" checks higher in the function.
While we're at it, we can move that logic to its own
helper, which makes the flow of link_alt_odb_entry()
easier to follow.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:34:48 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
if (!fspathcmp(path->buf, normalized_objdir))
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
2005-05-08 20:51:13 +00:00
|
|
|
/*
|
|
|
|
* Prepare alternate object database registry.
|
2005-08-15 00:25:57 +00:00
|
|
|
*
|
|
|
|
* The variable alt_odb_list points at the list of struct
|
|
|
|
* alternate_object_database. The elements on this list come from
|
|
|
|
* non-empty elements from colon separated ALTERNATE_DB_ENVIRONMENT
|
|
|
|
* environment variable, and $GIT_OBJECT_DIRECTORY/info/alternates,
|
2005-12-05 06:48:43 +00:00
|
|
|
* whose contents is similar to that environment variable but can be
|
|
|
|
* LF separated. Its base points at a statically allocated buffer that
|
2005-08-15 00:25:57 +00:00
|
|
|
* contains "/the/directory/corresponding/to/.git/objects/...", while
|
|
|
|
* its name points just after the slash at the end of ".git/objects/"
|
|
|
|
* in the example above, and has enough space to hold 40-byte hex
|
|
|
|
* SHA1, an extra slash for the first level indirection, and the
|
|
|
|
* terminating NUL.
|
2005-05-08 20:51:13 +00:00
|
|
|
*/
|
2018-03-23 17:21:08 +00:00
|
|
|
static void read_info_alternates(struct repository *r,
|
|
|
|
const char *relative_base,
|
|
|
|
int depth);
|
|
|
|
static int link_alt_odb_entry(struct repository *r, const char *entry,
|
2018-03-23 17:21:04 +00:00
|
|
|
const char *relative_base, int depth, const char *normalized_objdir)
|
2005-05-07 07:38:04 +00:00
|
|
|
{
|
2006-05-07 18:19:21 +00:00
|
|
|
struct alternate_object_database *ent;
|
2011-09-07 10:37:47 +00:00
|
|
|
struct strbuf pathbuf = STRBUF_INIT;
|
2005-08-15 00:25:57 +00:00
|
|
|
|
2007-11-13 20:05:00 +00:00
|
|
|
if (!is_absolute_path(entry) && relative_base) {
|
2016-12-12 18:16:55 +00:00
|
|
|
strbuf_realpath(&pathbuf, relative_base, 1);
|
2011-09-07 10:37:47 +00:00
|
|
|
strbuf_addch(&pathbuf, '/');
|
2006-05-07 18:19:21 +00:00
|
|
|
}
|
2012-11-05 08:41:22 +00:00
|
|
|
strbuf_addstr(&pathbuf, entry);
|
2006-05-07 18:19:21 +00:00
|
|
|
|
alternates: re-allow relative paths from environment
Commit 670c359da (link_alt_odb_entry: handle normalize_path
errors, 2016-10-03) regressed the handling of relative paths
in the GIT_ALTERNATE_OBJECT_DIRECTORIES variable. It's not
entirely clear this was ever meant to work, but it _has_
worked for several years, so this commit restores the
original behavior.
When we get a path in GIT_ALTERNATE_OBJECT_DIRECTORIES, we
add it the path to the list of alternate object directories
as if it were found in objects/info/alternates, but with one
difference: we do not provide the link_alt_odb_entry()
function with a base for relative paths. That function
doesn't turn it into an absolute path, and we end up feeding
the relative path to the strbuf_normalize_path() function.
Most relative paths break out of the top-level directory
(e.g., "../foo.git/objects"), and thus normalizing fails.
Prior to 670c359da, we simply ignored the error, and due to
the way normalize_path_copy() was implemented it happened to
return the original path in this case. We then accessed the
alternate objects using this relative path.
By storing the relative path in the alt_odb list, the path
is relative to wherever we happen to be at the time we do an
object lookup. That means we look from $GIT_DIR in a bare
repository, and from the top of the worktree in a non-bare
repository.
If this were being designed from scratch, it would make
sense to pick a stable location (probably $GIT_DIR, or even
the object directory) and use that as the relative base,
turning the result into an absolute path. However, given
the history, at this point the minimal fix is to match the
pre-670c359da behavior.
We can do this simply by ignoring the error when we have no
relative base and using the original value (which we now
reliably have, thanks to strbuf_normalize_path()).
That still leaves us with a relative path that foils our
duplicate detection, and may act strangely if we ever
chdir() later in the process. We could solve that by storing
an absolute path based on getcwd(). That may be a good
future direction; for now we'll do just the minimum to fix
the regression.
The new t5615 script demonstrates the fix in its final three
tests. Since we didn't have any tests of the alternates
environment variable at all, it also adds some tests of
absolute paths.
Reported-by: Bryan Turner <bturner@atlassian.com>
Signed-off-by: Jeff King <peff@peff.net>
2016-11-08 04:50:17 +00:00
|
|
|
if (strbuf_normalize_path(&pathbuf) < 0 && relative_base) {
|
2018-07-21 07:49:39 +00:00
|
|
|
error(_("unable to normalize alternate object path: %s"),
|
link_alt_odb_entry: handle normalize_path errors
When we add a new alternate to the list, we try to normalize
out any redundant "..", etc. However, we do not look at the
return value of normalize_path_copy(), and will happily
continue with a path that could not be normalized. Worse,
the normalizing process is done in-place, so we are left
with whatever half-finished working state the normalizing
function was in.
Fortunately, this cannot cause us to read past the end of
our buffer, as that working state will always leave the
NUL from the original path in place. And we do tend to
notice problems when we check is_directory() on the path.
But you can see the nonsense that we feed to is_directory
with an entry like:
this/../../is/../../way/../../too/../../deep/../../to/../../resolve
in your objects/info/alternates, which yields:
error: object directory
/to/e/deep/too/way//ects/this/../../is/../../way/../../too/../../deep/../../to/../../resolve
does not exist; check .git/objects/info/alternates.
We can easily fix this just by checking the return value.
But that makes it hard to generate a good error message,
since we're normalizing in-place and our input value has
been overwritten by cruft.
Instead, let's provide a strbuf helper that does an in-place
normalize, but restores the original contents on error. This
uses a second buffer under the hood, which is slightly less
efficient, but this is not a performance-critical code path.
The strbuf helper can also properly set the "len" parameter
of the strbuf before returning. Just doing:
normalize_path_copy(buf.buf, buf.buf);
will shorten the string, but leave buf.len at the original
length. That may be confusing to later code which uses the
strbuf.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:34:17 +00:00
|
|
|
pathbuf.buf);
|
|
|
|
strbuf_release(&pathbuf);
|
|
|
|
return -1;
|
|
|
|
}
|
2011-09-07 10:37:47 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* The trailing slash after the directory name is given by
|
|
|
|
* this function at the end. Remove duplicates.
|
|
|
|
*/
|
link_alt_odb_entry: refactor string handling
The string handling in link_alt_odb_entry() is mostly an
artifact of the original version, which took the path as a
ptr/len combo, and did not have a NUL-terminated string
until we created one in the alternate_object_database
struct. But since 5bdf0a8 (sha1_file: normalize alt_odb
path before comparing and storing, 2011-09-07), the first
thing we do is put the path into a strbuf, which gives us
some easy opportunities for cleanup.
In particular:
- we call strlen(pathbuf.buf), which is silly; we can look
at pathbuf.len.
- even though we have a strbuf, we don't maintain its
"len" field when chomping extra slashes from the
end, and instead keep a separate "pfxlen" variable. We
can fix this and then drop "pfxlen" entirely.
- we don't check whether the path is usable until after we
allocate the new struct, making extra cleanup work for
ourselves. Since we have a NUL-terminated string, we can
bump the "is it usable" checks higher in the function.
While we're at it, we can move that logic to its own
helper, which makes the flow of link_alt_odb_entry()
easier to follow.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:34:48 +00:00
|
|
|
while (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')
|
|
|
|
strbuf_setlen(&pathbuf, pathbuf.len - 1);
|
2011-09-07 10:37:47 +00:00
|
|
|
|
2018-03-23 17:21:08 +00:00
|
|
|
if (!alt_odb_usable(r->objects, &pathbuf, normalized_objdir)) {
|
link_alt_odb_entry: refactor string handling
The string handling in link_alt_odb_entry() is mostly an
artifact of the original version, which took the path as a
ptr/len combo, and did not have a NUL-terminated string
until we created one in the alternate_object_database
struct. But since 5bdf0a8 (sha1_file: normalize alt_odb
path before comparing and storing, 2011-09-07), the first
thing we do is put the path into a strbuf, which gives us
some easy opportunities for cleanup.
In particular:
- we call strlen(pathbuf.buf), which is silly; we can look
at pathbuf.len.
- even though we have a strbuf, we don't maintain its
"len" field when chomping extra slashes from the
end, and instead keep a separate "pfxlen" variable. We
can fix this and then drop "pfxlen" entirely.
- we don't check whether the path is usable until after we
allocate the new struct, making extra cleanup work for
ourselves. Since we have a NUL-terminated string, we can
bump the "is it usable" checks higher in the function.
While we're at it, we can move that logic to its own
helper, which makes the flow of link_alt_odb_entry()
easier to follow.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:34:48 +00:00
|
|
|
strbuf_release(&pathbuf);
|
2006-05-07 18:19:21 +00:00
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
2016-10-03 20:35:31 +00:00
|
|
|
ent = alloc_alt_odb(pathbuf.buf);
|
2006-05-07 18:19:21 +00:00
|
|
|
|
|
|
|
/* add the alternate entry */
|
2018-03-23 17:21:08 +00:00
|
|
|
*r->objects->alt_odb_tail = ent;
|
|
|
|
r->objects->alt_odb_tail = &(ent->next);
|
2006-05-07 18:19:21 +00:00
|
|
|
ent->next = NULL;
|
|
|
|
|
|
|
|
/* recursively add alternates */
|
2018-03-23 17:21:08 +00:00
|
|
|
read_info_alternates(r, pathbuf.buf, depth + 1);
|
2006-05-07 18:19:21 +00:00
|
|
|
|
link_alt_odb_entry: refactor string handling
The string handling in link_alt_odb_entry() is mostly an
artifact of the original version, which took the path as a
ptr/len combo, and did not have a NUL-terminated string
until we created one in the alternate_object_database
struct. But since 5bdf0a8 (sha1_file: normalize alt_odb
path before comparing and storing, 2011-09-07), the first
thing we do is put the path into a strbuf, which gives us
some easy opportunities for cleanup.
In particular:
- we call strlen(pathbuf.buf), which is silly; we can look
at pathbuf.len.
- even though we have a strbuf, we don't maintain its
"len" field when chomping extra slashes from the
end, and instead keep a separate "pfxlen" variable. We
can fix this and then drop "pfxlen" entirely.
- we don't check whether the path is usable until after we
allocate the new struct, making extra cleanup work for
ourselves. Since we have a NUL-terminated string, we can
bump the "is it usable" checks higher in the function.
While we're at it, we can move that logic to its own
helper, which makes the flow of link_alt_odb_entry()
easier to follow.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:34:48 +00:00
|
|
|
strbuf_release(&pathbuf);
|
2006-05-07 18:19:21 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
alternates: accept double-quoted paths
We read lists of alternates from objects/info/alternates
files (delimited by newline), as well as from the
GIT_ALTERNATE_OBJECT_DIRECTORIES environment variable
(delimited by colon or semi-colon, depending on the
platform).
There's no mechanism for quoting the delimiters, so it's
impossible to specify an alternate path that contains a
colon in the environment, or one that contains a newline in
a file. We've lived with that restriction for ages because
both alternates and filenames with colons are relatively
rare, and it's only a problem when the two meet. But since
722ff7f87 (receive-pack: quarantine objects until
pre-receive accepts, 2016-10-03), which builds on the
alternates system, every push causes the receiver to set
GIT_ALTERNATE_OBJECT_DIRECTORIES internally.
It would be convenient to have some way to quote the
delimiter so that we can represent arbitrary paths.
The simplest thing would be an escape character before a
quoted delimiter (e.g., "\:" as a literal colon). But that
creates a backwards compatibility problem: any path which
uses that escape character is now broken, and we've just
shifted the problem. We could choose an unlikely escape
character (e.g., something from the non-printable ASCII
range), but that's awkward to use.
Instead, let's treat names as unquoted unless they begin
with a double-quote, in which case they are interpreted via
our usual C-stylke quoting rules. This also breaks
backwards-compatibility, but in a smaller way: it only
matters if your file has a double-quote as the very _first_
character in the path (whereas an escape character is a
problem anywhere in the path). It's also consistent with
many other parts of git, which accept either a bare pathname
or a double-quoted one, and the sender can choose to quote
or not as required.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-12-12 19:52:22 +00:00
|
|
|
static const char *parse_alt_odb_entry(const char *string,
|
|
|
|
int sep,
|
|
|
|
struct strbuf *out)
|
|
|
|
{
|
|
|
|
const char *end;
|
|
|
|
|
|
|
|
strbuf_reset(out);
|
|
|
|
|
|
|
|
if (*string == '#') {
|
|
|
|
/* comment; consume up to next separator */
|
|
|
|
end = strchrnul(string, sep);
|
|
|
|
} else if (*string == '"' && !unquote_c_style(out, string, &end)) {
|
|
|
|
/*
|
|
|
|
* quoted path; unquote_c_style has copied the
|
|
|
|
* data for us and set "end". Broken quoting (e.g.,
|
|
|
|
* an entry that doesn't end with a quote) falls
|
|
|
|
* back to the unquoted case below.
|
|
|
|
*/
|
|
|
|
} else {
|
|
|
|
/* normal, unquoted path */
|
|
|
|
end = strchrnul(string, sep);
|
|
|
|
strbuf_add(out, string, end - string);
|
|
|
|
}
|
|
|
|
|
|
|
|
if (*end)
|
|
|
|
end++;
|
|
|
|
return end;
|
|
|
|
}
|
|
|
|
|
2018-03-23 17:21:08 +00:00
|
|
|
static void link_alt_odb_entries(struct repository *r, const char *alt,
|
|
|
|
int sep, const char *relative_base, int depth)
|
2006-05-07 18:19:21 +00:00
|
|
|
{
|
2014-07-15 11:29:45 +00:00
|
|
|
struct strbuf objdirbuf = STRBUF_INIT;
|
alternates: accept double-quoted paths
We read lists of alternates from objects/info/alternates
files (delimited by newline), as well as from the
GIT_ALTERNATE_OBJECT_DIRECTORIES environment variable
(delimited by colon or semi-colon, depending on the
platform).
There's no mechanism for quoting the delimiters, so it's
impossible to specify an alternate path that contains a
colon in the environment, or one that contains a newline in
a file. We've lived with that restriction for ages because
both alternates and filenames with colons are relatively
rare, and it's only a problem when the two meet. But since
722ff7f87 (receive-pack: quarantine objects until
pre-receive accepts, 2016-10-03), which builds on the
alternates system, every push causes the receiver to set
GIT_ALTERNATE_OBJECT_DIRECTORIES internally.
It would be convenient to have some way to quote the
delimiter so that we can represent arbitrary paths.
The simplest thing would be an escape character before a
quoted delimiter (e.g., "\:" as a literal colon). But that
creates a backwards compatibility problem: any path which
uses that escape character is now broken, and we've just
shifted the problem. We could choose an unlikely escape
character (e.g., something from the non-printable ASCII
range), but that's awkward to use.
Instead, let's treat names as unquoted unless they begin
with a double-quote, in which case they are interpreted via
our usual C-stylke quoting rules. This also breaks
backwards-compatibility, but in a smaller way: it only
matters if your file has a double-quote as the very _first_
character in the path (whereas an escape character is a
problem anywhere in the path). It's also consistent with
many other parts of git, which accept either a bare pathname
or a double-quoted one, and the sender can choose to quote
or not as required.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-12-12 19:52:22 +00:00
|
|
|
struct strbuf entry = STRBUF_INIT;
|
2006-05-07 18:19:21 +00:00
|
|
|
|
2017-11-12 10:27:39 +00:00
|
|
|
if (!alt || !*alt)
|
|
|
|
return;
|
|
|
|
|
2006-05-07 18:19:21 +00:00
|
|
|
if (depth > 5) {
|
2018-07-21 07:49:39 +00:00
|
|
|
error(_("%s: ignoring alternate object stores, nesting too deep"),
|
2006-05-07 18:19:21 +00:00
|
|
|
relative_base);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2018-03-23 17:21:08 +00:00
|
|
|
strbuf_add_absolute_path(&objdirbuf, r->objects->objectdir);
|
link_alt_odb_entry: handle normalize_path errors
When we add a new alternate to the list, we try to normalize
out any redundant "..", etc. However, we do not look at the
return value of normalize_path_copy(), and will happily
continue with a path that could not be normalized. Worse,
the normalizing process is done in-place, so we are left
with whatever half-finished working state the normalizing
function was in.
Fortunately, this cannot cause us to read past the end of
our buffer, as that working state will always leave the
NUL from the original path in place. And we do tend to
notice problems when we check is_directory() on the path.
But you can see the nonsense that we feed to is_directory
with an entry like:
this/../../is/../../way/../../too/../../deep/../../to/../../resolve
in your objects/info/alternates, which yields:
error: object directory
/to/e/deep/too/way//ects/this/../../is/../../way/../../too/../../deep/../../to/../../resolve
does not exist; check .git/objects/info/alternates.
We can easily fix this just by checking the return value.
But that makes it hard to generate a good error message,
since we're normalizing in-place and our input value has
been overwritten by cruft.
Instead, let's provide a strbuf helper that does an in-place
normalize, but restores the original contents on error. This
uses a second buffer under the hood, which is slightly less
efficient, but this is not a performance-critical code path.
The strbuf helper can also properly set the "len" parameter
of the strbuf before returning. Just doing:
normalize_path_copy(buf.buf, buf.buf);
will shorten the string, but leave buf.len at the original
length. That may be confusing to later code which uses the
strbuf.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:34:17 +00:00
|
|
|
if (strbuf_normalize_path(&objdirbuf) < 0)
|
2018-07-21 07:49:39 +00:00
|
|
|
die(_("unable to normalize object directory: %s"),
|
link_alt_odb_entry: handle normalize_path errors
When we add a new alternate to the list, we try to normalize
out any redundant "..", etc. However, we do not look at the
return value of normalize_path_copy(), and will happily
continue with a path that could not be normalized. Worse,
the normalizing process is done in-place, so we are left
with whatever half-finished working state the normalizing
function was in.
Fortunately, this cannot cause us to read past the end of
our buffer, as that working state will always leave the
NUL from the original path in place. And we do tend to
notice problems when we check is_directory() on the path.
But you can see the nonsense that we feed to is_directory
with an entry like:
this/../../is/../../way/../../too/../../deep/../../to/../../resolve
in your objects/info/alternates, which yields:
error: object directory
/to/e/deep/too/way//ects/this/../../is/../../way/../../too/../../deep/../../to/../../resolve
does not exist; check .git/objects/info/alternates.
We can easily fix this just by checking the return value.
But that makes it hard to generate a good error message,
since we're normalizing in-place and our input value has
been overwritten by cruft.
Instead, let's provide a strbuf helper that does an in-place
normalize, but restores the original contents on error. This
uses a second buffer under the hood, which is slightly less
efficient, but this is not a performance-critical code path.
The strbuf helper can also properly set the "len" parameter
of the strbuf before returning. Just doing:
normalize_path_copy(buf.buf, buf.buf);
will shorten the string, but leave buf.len at the original
length. That may be confusing to later code which uses the
strbuf.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:34:17 +00:00
|
|
|
objdirbuf.buf);
|
2014-07-15 11:29:45 +00:00
|
|
|
|
alternates: accept double-quoted paths
We read lists of alternates from objects/info/alternates
files (delimited by newline), as well as from the
GIT_ALTERNATE_OBJECT_DIRECTORIES environment variable
(delimited by colon or semi-colon, depending on the
platform).
There's no mechanism for quoting the delimiters, so it's
impossible to specify an alternate path that contains a
colon in the environment, or one that contains a newline in
a file. We've lived with that restriction for ages because
both alternates and filenames with colons are relatively
rare, and it's only a problem when the two meet. But since
722ff7f87 (receive-pack: quarantine objects until
pre-receive accepts, 2016-10-03), which builds on the
alternates system, every push causes the receiver to set
GIT_ALTERNATE_OBJECT_DIRECTORIES internally.
It would be convenient to have some way to quote the
delimiter so that we can represent arbitrary paths.
The simplest thing would be an escape character before a
quoted delimiter (e.g., "\:" as a literal colon). But that
creates a backwards compatibility problem: any path which
uses that escape character is now broken, and we've just
shifted the problem. We could choose an unlikely escape
character (e.g., something from the non-printable ASCII
range), but that's awkward to use.
Instead, let's treat names as unquoted unless they begin
with a double-quote, in which case they are interpreted via
our usual C-stylke quoting rules. This also breaks
backwards-compatibility, but in a smaller way: it only
matters if your file has a double-quote as the very _first_
character in the path (whereas an escape character is a
problem anywhere in the path). It's also consistent with
many other parts of git, which accept either a bare pathname
or a double-quoted one, and the sender can choose to quote
or not as required.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-12-12 19:52:22 +00:00
|
|
|
while (*alt) {
|
|
|
|
alt = parse_alt_odb_entry(alt, sep, &entry);
|
|
|
|
if (!entry.len)
|
2005-08-17 01:22:05 +00:00
|
|
|
continue;
|
2018-03-23 17:21:08 +00:00
|
|
|
link_alt_odb_entry(r, entry.buf,
|
2018-03-23 17:21:04 +00:00
|
|
|
relative_base, depth, objdirbuf.buf);
|
2005-08-17 01:22:05 +00:00
|
|
|
}
|
alternates: accept double-quoted paths
We read lists of alternates from objects/info/alternates
files (delimited by newline), as well as from the
GIT_ALTERNATE_OBJECT_DIRECTORIES environment variable
(delimited by colon or semi-colon, depending on the
platform).
There's no mechanism for quoting the delimiters, so it's
impossible to specify an alternate path that contains a
colon in the environment, or one that contains a newline in
a file. We've lived with that restriction for ages because
both alternates and filenames with colons are relatively
rare, and it's only a problem when the two meet. But since
722ff7f87 (receive-pack: quarantine objects until
pre-receive accepts, 2016-10-03), which builds on the
alternates system, every push causes the receiver to set
GIT_ALTERNATE_OBJECT_DIRECTORIES internally.
It would be convenient to have some way to quote the
delimiter so that we can represent arbitrary paths.
The simplest thing would be an escape character before a
quoted delimiter (e.g., "\:" as a literal colon). But that
creates a backwards compatibility problem: any path which
uses that escape character is now broken, and we've just
shifted the problem. We could choose an unlikely escape
character (e.g., something from the non-printable ASCII
range), but that's awkward to use.
Instead, let's treat names as unquoted unless they begin
with a double-quote, in which case they are interpreted via
our usual C-stylke quoting rules. This also breaks
backwards-compatibility, but in a smaller way: it only
matters if your file has a double-quote as the very _first_
character in the path (whereas an escape character is a
problem anywhere in the path). It's also consistent with
many other parts of git, which accept either a bare pathname
or a double-quoted one, and the sender can choose to quote
or not as required.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-12-12 19:52:22 +00:00
|
|
|
strbuf_release(&entry);
|
2014-07-15 11:29:45 +00:00
|
|
|
strbuf_release(&objdirbuf);
|
2005-08-15 00:25:57 +00:00
|
|
|
}
|
|
|
|
|
2018-03-23 17:21:08 +00:00
|
|
|
static void read_info_alternates(struct repository *r,
|
|
|
|
const char *relative_base,
|
|
|
|
int depth)
|
2005-08-15 00:25:57 +00:00
|
|
|
{
|
2015-08-19 18:12:45 +00:00
|
|
|
char *path;
|
read_info_alternates: read contents into strbuf
This patch fixes a regression in v2.11.1 where we might read
past the end of an mmap'd buffer. It was introduced in
cf3c635210.
The link_alt_odb_entries() function has always taken a
ptr/len pair as input. Until cf3c635210 (alternates: accept
double-quoted paths, 2016-12-12), we made a copy of those
bytes in a string. But after that commit, we switched to
parsing the input left-to-right, and we ignore "len"
totally, instead reading until we hit a NUL.
This has mostly gone unnoticed for a few reasons:
1. All but one caller passes a NUL-terminated string, with
"len" pointing to the NUL.
2. The remaining caller, read_info_alternates(), passes in
an mmap'd file. Unless the file is an exact multiple of
the page size, it will generally be followed by NUL
padding to the end of the page, which just works.
The easiest way to demonstrate the problem is to build with:
make SANITIZE=address NO_MMAP=Nope test
Any test which involves $GIT_DIR/info/alternates will fail,
as the mmap emulation (correctly) does not add an extra NUL,
and ASAN complains about reading past the end of the buffer.
One solution would be to teach link_alt_odb_entries() to
respect "len". But it's actually a bit tricky, since we
depend on unquote_c_style() under the hood, and it has no
ptr/len variant.
We could also just make a NUL-terminated copy of the input
bytes and operate on that. But since all but one caller
already is passing a string, instead let's just fix that
caller to provide NUL-terminated input in the first place,
by swapping out mmap for strbuf_read_file().
There's no advantage to using mmap on the alternates file.
It's not expected to be large (and anyway, we're copying its
contents into an in-memory linked list). Nor is using
git_open() buying us anything here, since we don't keep the
descriptor open for a long period of time.
Let's also drop the "len" parameter entirely from
link_alt_odb_entries(), since it's completely ignored. That
will avoid any new callers re-introducing a similar bug.
Reported-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-09-19 19:41:07 +00:00
|
|
|
struct strbuf buf = STRBUF_INIT;
|
2005-08-15 00:25:57 +00:00
|
|
|
|
2015-08-19 18:12:45 +00:00
|
|
|
path = xstrfmt("%s/info/alternates", relative_base);
|
read_info_alternates: read contents into strbuf
This patch fixes a regression in v2.11.1 where we might read
past the end of an mmap'd buffer. It was introduced in
cf3c635210.
The link_alt_odb_entries() function has always taken a
ptr/len pair as input. Until cf3c635210 (alternates: accept
double-quoted paths, 2016-12-12), we made a copy of those
bytes in a string. But after that commit, we switched to
parsing the input left-to-right, and we ignore "len"
totally, instead reading until we hit a NUL.
This has mostly gone unnoticed for a few reasons:
1. All but one caller passes a NUL-terminated string, with
"len" pointing to the NUL.
2. The remaining caller, read_info_alternates(), passes in
an mmap'd file. Unless the file is an exact multiple of
the page size, it will generally be followed by NUL
padding to the end of the page, which just works.
The easiest way to demonstrate the problem is to build with:
make SANITIZE=address NO_MMAP=Nope test
Any test which involves $GIT_DIR/info/alternates will fail,
as the mmap emulation (correctly) does not add an extra NUL,
and ASAN complains about reading past the end of the buffer.
One solution would be to teach link_alt_odb_entries() to
respect "len". But it's actually a bit tricky, since we
depend on unquote_c_style() under the hood, and it has no
ptr/len variant.
We could also just make a NUL-terminated copy of the input
bytes and operate on that. But since all but one caller
already is passing a string, instead let's just fix that
caller to provide NUL-terminated input in the first place,
by swapping out mmap for strbuf_read_file().
There's no advantage to using mmap on the alternates file.
It's not expected to be large (and anyway, we're copying its
contents into an in-memory linked list). Nor is using
git_open() buying us anything here, since we don't keep the
descriptor open for a long period of time.
Let's also drop the "len" parameter entirely from
link_alt_odb_entries(), since it's completely ignored. That
will avoid any new callers re-introducing a similar bug.
Reported-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-09-19 19:41:07 +00:00
|
|
|
if (strbuf_read_file(&buf, path, 1024) < 0) {
|
2017-09-19 19:41:10 +00:00
|
|
|
warn_on_fopen_errors(path);
|
read_info_alternates: read contents into strbuf
This patch fixes a regression in v2.11.1 where we might read
past the end of an mmap'd buffer. It was introduced in
cf3c635210.
The link_alt_odb_entries() function has always taken a
ptr/len pair as input. Until cf3c635210 (alternates: accept
double-quoted paths, 2016-12-12), we made a copy of those
bytes in a string. But after that commit, we switched to
parsing the input left-to-right, and we ignore "len"
totally, instead reading until we hit a NUL.
This has mostly gone unnoticed for a few reasons:
1. All but one caller passes a NUL-terminated string, with
"len" pointing to the NUL.
2. The remaining caller, read_info_alternates(), passes in
an mmap'd file. Unless the file is an exact multiple of
the page size, it will generally be followed by NUL
padding to the end of the page, which just works.
The easiest way to demonstrate the problem is to build with:
make SANITIZE=address NO_MMAP=Nope test
Any test which involves $GIT_DIR/info/alternates will fail,
as the mmap emulation (correctly) does not add an extra NUL,
and ASAN complains about reading past the end of the buffer.
One solution would be to teach link_alt_odb_entries() to
respect "len". But it's actually a bit tricky, since we
depend on unquote_c_style() under the hood, and it has no
ptr/len variant.
We could also just make a NUL-terminated copy of the input
bytes and operate on that. But since all but one caller
already is passing a string, instead let's just fix that
caller to provide NUL-terminated input in the first place,
by swapping out mmap for strbuf_read_file().
There's no advantage to using mmap on the alternates file.
It's not expected to be large (and anyway, we're copying its
contents into an in-memory linked list). Nor is using
git_open() buying us anything here, since we don't keep the
descriptor open for a long period of time.
Let's also drop the "len" parameter entirely from
link_alt_odb_entries(), since it's completely ignored. That
will avoid any new callers re-introducing a similar bug.
Reported-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-09-19 19:41:07 +00:00
|
|
|
free(path);
|
2005-06-28 21:56:57 +00:00
|
|
|
return;
|
2005-05-07 07:38:04 +00:00
|
|
|
}
|
2005-08-15 00:25:57 +00:00
|
|
|
|
2018-03-23 17:21:08 +00:00
|
|
|
link_alt_odb_entries(r, buf.buf, '\n', relative_base, depth);
|
read_info_alternates: read contents into strbuf
This patch fixes a regression in v2.11.1 where we might read
past the end of an mmap'd buffer. It was introduced in
cf3c635210.
The link_alt_odb_entries() function has always taken a
ptr/len pair as input. Until cf3c635210 (alternates: accept
double-quoted paths, 2016-12-12), we made a copy of those
bytes in a string. But after that commit, we switched to
parsing the input left-to-right, and we ignore "len"
totally, instead reading until we hit a NUL.
This has mostly gone unnoticed for a few reasons:
1. All but one caller passes a NUL-terminated string, with
"len" pointing to the NUL.
2. The remaining caller, read_info_alternates(), passes in
an mmap'd file. Unless the file is an exact multiple of
the page size, it will generally be followed by NUL
padding to the end of the page, which just works.
The easiest way to demonstrate the problem is to build with:
make SANITIZE=address NO_MMAP=Nope test
Any test which involves $GIT_DIR/info/alternates will fail,
as the mmap emulation (correctly) does not add an extra NUL,
and ASAN complains about reading past the end of the buffer.
One solution would be to teach link_alt_odb_entries() to
respect "len". But it's actually a bit tricky, since we
depend on unquote_c_style() under the hood, and it has no
ptr/len variant.
We could also just make a NUL-terminated copy of the input
bytes and operate on that. But since all but one caller
already is passing a string, instead let's just fix that
caller to provide NUL-terminated input in the first place,
by swapping out mmap for strbuf_read_file().
There's no advantage to using mmap on the alternates file.
It's not expected to be large (and anyway, we're copying its
contents into an in-memory linked list). Nor is using
git_open() buying us anything here, since we don't keep the
descriptor open for a long period of time.
Let's also drop the "len" parameter entirely from
link_alt_odb_entries(), since it's completely ignored. That
will avoid any new callers re-introducing a similar bug.
Reported-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-09-19 19:41:07 +00:00
|
|
|
strbuf_release(&buf);
|
|
|
|
free(path);
|
2005-05-07 07:38:04 +00:00
|
|
|
}
|
|
|
|
|
2016-10-03 20:35:31 +00:00
|
|
|
struct alternate_object_database *alloc_alt_odb(const char *dir)
|
|
|
|
{
|
|
|
|
struct alternate_object_database *ent;
|
|
|
|
|
2016-10-03 20:35:51 +00:00
|
|
|
FLEX_ALLOC_STR(ent, path, dir);
|
2016-10-03 20:36:04 +00:00
|
|
|
strbuf_init(&ent->scratch, 0);
|
|
|
|
strbuf_addf(&ent->scratch, "%s/", dir);
|
|
|
|
ent->base_len = ent->scratch.len;
|
2016-10-03 20:35:31 +00:00
|
|
|
|
|
|
|
return ent;
|
|
|
|
}
|
|
|
|
|
2008-04-17 23:32:30 +00:00
|
|
|
void add_to_alternates_file(const char *reference)
|
|
|
|
{
|
2017-10-05 20:32:03 +00:00
|
|
|
struct lock_file lock = LOCK_INIT;
|
add_to_alternates_file: don't add duplicate entries
The add_to_alternates_file function blindly uses
hold_lock_file_for_append to copy the existing contents, and
then adds the new line to it. This has two minor problems:
1. We might add duplicate entries, which are ugly and
inefficient.
2. We do not check that the file ends with a newline, in
which case we would bogusly append to the final line.
This is quite unlikely in practice, though, as we call
this function only from git-clone, so presumably we are
the only writers of the file (and we always add a
newline).
Instead of using hold_lock_file_for_append, let's copy the
file line by line, which ensures all records are properly
terminated. If we see an extra line, we can simply abort the
update (there is no point in even copying the rest, as we
know that it would be identical to the original).
As a bonus, we also get rid of some calls to the
static-buffer mkpath and git_path functions.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-08-10 09:34:46 +00:00
|
|
|
char *alts = git_pathdup("objects/info/alternates");
|
|
|
|
FILE *in, *out;
|
2017-10-05 20:32:03 +00:00
|
|
|
int found = 0;
|
add_to_alternates_file: don't add duplicate entries
The add_to_alternates_file function blindly uses
hold_lock_file_for_append to copy the existing contents, and
then adds the new line to it. This has two minor problems:
1. We might add duplicate entries, which are ugly and
inefficient.
2. We do not check that the file ends with a newline, in
which case we would bogusly append to the final line.
This is quite unlikely in practice, though, as we call
this function only from git-clone, so presumably we are
the only writers of the file (and we always add a
newline).
Instead of using hold_lock_file_for_append, let's copy the
file line by line, which ensures all records are properly
terminated. If we see an extra line, we can simply abort the
update (there is no point in even copying the rest, as we
know that it would be identical to the original).
As a bonus, we also get rid of some calls to the
static-buffer mkpath and git_path functions.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-08-10 09:34:46 +00:00
|
|
|
|
2017-10-05 20:32:03 +00:00
|
|
|
hold_lock_file_for_update(&lock, alts, LOCK_DIE_ON_ERROR);
|
|
|
|
out = fdopen_lock_file(&lock, "w");
|
add_to_alternates_file: don't add duplicate entries
The add_to_alternates_file function blindly uses
hold_lock_file_for_append to copy the existing contents, and
then adds the new line to it. This has two minor problems:
1. We might add duplicate entries, which are ugly and
inefficient.
2. We do not check that the file ends with a newline, in
which case we would bogusly append to the final line.
This is quite unlikely in practice, though, as we call
this function only from git-clone, so presumably we are
the only writers of the file (and we always add a
newline).
Instead of using hold_lock_file_for_append, let's copy the
file line by line, which ensures all records are properly
terminated. If we see an extra line, we can simply abort the
update (there is no point in even copying the rest, as we
know that it would be identical to the original).
As a bonus, we also get rid of some calls to the
static-buffer mkpath and git_path functions.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-08-10 09:34:46 +00:00
|
|
|
if (!out)
|
2018-07-21 07:49:39 +00:00
|
|
|
die_errno(_("unable to fdopen alternates lockfile"));
|
add_to_alternates_file: don't add duplicate entries
The add_to_alternates_file function blindly uses
hold_lock_file_for_append to copy the existing contents, and
then adds the new line to it. This has two minor problems:
1. We might add duplicate entries, which are ugly and
inefficient.
2. We do not check that the file ends with a newline, in
which case we would bogusly append to the final line.
This is quite unlikely in practice, though, as we call
this function only from git-clone, so presumably we are
the only writers of the file (and we always add a
newline).
Instead of using hold_lock_file_for_append, let's copy the
file line by line, which ensures all records are properly
terminated. If we see an extra line, we can simply abort the
update (there is no point in even copying the rest, as we
know that it would be identical to the original).
As a bonus, we also get rid of some calls to the
static-buffer mkpath and git_path functions.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-08-10 09:34:46 +00:00
|
|
|
|
|
|
|
in = fopen(alts, "r");
|
|
|
|
if (in) {
|
|
|
|
struct strbuf line = STRBUF_INIT;
|
|
|
|
|
2015-10-28 20:29:24 +00:00
|
|
|
while (strbuf_getline(&line, in) != EOF) {
|
add_to_alternates_file: don't add duplicate entries
The add_to_alternates_file function blindly uses
hold_lock_file_for_append to copy the existing contents, and
then adds the new line to it. This has two minor problems:
1. We might add duplicate entries, which are ugly and
inefficient.
2. We do not check that the file ends with a newline, in
which case we would bogusly append to the final line.
This is quite unlikely in practice, though, as we call
this function only from git-clone, so presumably we are
the only writers of the file (and we always add a
newline).
Instead of using hold_lock_file_for_append, let's copy the
file line by line, which ensures all records are properly
terminated. If we see an extra line, we can simply abort the
update (there is no point in even copying the rest, as we
know that it would be identical to the original).
As a bonus, we also get rid of some calls to the
static-buffer mkpath and git_path functions.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-08-10 09:34:46 +00:00
|
|
|
if (!strcmp(reference, line.buf)) {
|
|
|
|
found = 1;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
fprintf_or_die(out, "%s\n", line.buf);
|
|
|
|
}
|
|
|
|
|
|
|
|
strbuf_release(&line);
|
|
|
|
fclose(in);
|
|
|
|
}
|
|
|
|
else if (errno != ENOENT)
|
2018-07-21 07:49:39 +00:00
|
|
|
die_errno(_("unable to read alternates file"));
|
add_to_alternates_file: don't add duplicate entries
The add_to_alternates_file function blindly uses
hold_lock_file_for_append to copy the existing contents, and
then adds the new line to it. This has two minor problems:
1. We might add duplicate entries, which are ugly and
inefficient.
2. We do not check that the file ends with a newline, in
which case we would bogusly append to the final line.
This is quite unlikely in practice, though, as we call
this function only from git-clone, so presumably we are
the only writers of the file (and we always add a
newline).
Instead of using hold_lock_file_for_append, let's copy the
file line by line, which ensures all records are properly
terminated. If we see an extra line, we can simply abort the
update (there is no point in even copying the rest, as we
know that it would be identical to the original).
As a bonus, we also get rid of some calls to the
static-buffer mkpath and git_path functions.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-08-10 09:34:46 +00:00
|
|
|
|
2017-10-05 20:32:03 +00:00
|
|
|
if (found) {
|
|
|
|
rollback_lock_file(&lock);
|
|
|
|
} else {
|
add_to_alternates_file: don't add duplicate entries
The add_to_alternates_file function blindly uses
hold_lock_file_for_append to copy the existing contents, and
then adds the new line to it. This has two minor problems:
1. We might add duplicate entries, which are ugly and
inefficient.
2. We do not check that the file ends with a newline, in
which case we would bogusly append to the final line.
This is quite unlikely in practice, though, as we call
this function only from git-clone, so presumably we are
the only writers of the file (and we always add a
newline).
Instead of using hold_lock_file_for_append, let's copy the
file line by line, which ensures all records are properly
terminated. If we see an extra line, we can simply abort the
update (there is no point in even copying the rest, as we
know that it would be identical to the original).
As a bonus, we also get rid of some calls to the
static-buffer mkpath and git_path functions.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-08-10 09:34:46 +00:00
|
|
|
fprintf_or_die(out, "%s\n", reference);
|
2017-10-05 20:32:03 +00:00
|
|
|
if (commit_lock_file(&lock))
|
2018-07-21 07:49:39 +00:00
|
|
|
die_errno(_("unable to move new alternates file into place"));
|
2018-03-23 17:20:57 +00:00
|
|
|
if (the_repository->objects->alt_odb_tail)
|
2018-03-23 17:21:06 +00:00
|
|
|
link_alt_odb_entries(the_repository, reference,
|
|
|
|
'\n', NULL, 0);
|
add_to_alternates_file: don't add duplicate entries
The add_to_alternates_file function blindly uses
hold_lock_file_for_append to copy the existing contents, and
then adds the new line to it. This has two minor problems:
1. We might add duplicate entries, which are ugly and
inefficient.
2. We do not check that the file ends with a newline, in
which case we would bogusly append to the final line.
This is quite unlikely in practice, though, as we call
this function only from git-clone, so presumably we are
the only writers of the file (and we always add a
newline).
Instead of using hold_lock_file_for_append, let's copy the
file line by line, which ensures all records are properly
terminated. If we see an extra line, we can simply abort the
update (there is no point in even copying the rest, as we
know that it would be identical to the original).
As a bonus, we also get rid of some calls to the
static-buffer mkpath and git_path functions.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-08-10 09:34:46 +00:00
|
|
|
}
|
|
|
|
free(alts);
|
2008-04-17 23:32:30 +00:00
|
|
|
}
|
|
|
|
|
2016-10-03 20:35:03 +00:00
|
|
|
void add_to_alternates_memory(const char *reference)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* Make sure alternates are initialized, or else our entry may be
|
|
|
|
* overwritten when they are.
|
|
|
|
*/
|
2018-03-23 17:21:07 +00:00
|
|
|
prepare_alt_odb(the_repository);
|
2016-10-03 20:35:03 +00:00
|
|
|
|
2018-03-23 17:21:06 +00:00
|
|
|
link_alt_odb_entries(the_repository, reference,
|
|
|
|
'\n', NULL, 0);
|
2016-10-03 20:35:03 +00:00
|
|
|
}
|
|
|
|
|
2016-08-15 21:53:24 +00:00
|
|
|
/*
|
|
|
|
* Compute the exact path an alternate is at and returns it. In case of
|
|
|
|
* error NULL is returned and the human readable error is added to `err`
|
2018-06-03 14:32:50 +00:00
|
|
|
* `path` may be relative and should point to $GIT_DIR.
|
2016-08-15 21:53:24 +00:00
|
|
|
* `err` must not be null.
|
|
|
|
*/
|
|
|
|
char *compute_alternate_path(const char *path, struct strbuf *err)
|
|
|
|
{
|
|
|
|
char *ref_git = NULL;
|
|
|
|
const char *repo, *ref_git_s;
|
|
|
|
int seen_error = 0;
|
|
|
|
|
|
|
|
ref_git_s = real_path_if_valid(path);
|
|
|
|
if (!ref_git_s) {
|
|
|
|
seen_error = 1;
|
|
|
|
strbuf_addf(err, _("path '%s' does not exist"), path);
|
|
|
|
goto out;
|
|
|
|
} else
|
|
|
|
/*
|
|
|
|
* Beware: read_gitfile(), real_path() and mkpath()
|
|
|
|
* return static buffer
|
|
|
|
*/
|
|
|
|
ref_git = xstrdup(ref_git_s);
|
|
|
|
|
|
|
|
repo = read_gitfile(ref_git);
|
|
|
|
if (!repo)
|
|
|
|
repo = read_gitfile(mkpath("%s/.git", ref_git));
|
|
|
|
if (repo) {
|
|
|
|
free(ref_git);
|
|
|
|
ref_git = xstrdup(repo);
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!repo && is_directory(mkpath("%s/.git/objects", ref_git))) {
|
|
|
|
char *ref_git_git = mkpathdup("%s/.git", ref_git);
|
|
|
|
free(ref_git);
|
|
|
|
ref_git = ref_git_git;
|
|
|
|
} else if (!is_directory(mkpath("%s/objects", ref_git))) {
|
|
|
|
struct strbuf sb = STRBUF_INIT;
|
|
|
|
seen_error = 1;
|
|
|
|
if (get_common_dir(&sb, ref_git)) {
|
|
|
|
strbuf_addf(err,
|
|
|
|
_("reference repository '%s' as a linked "
|
|
|
|
"checkout is not supported yet."),
|
|
|
|
path);
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
strbuf_addf(err, _("reference repository '%s' is not a "
|
|
|
|
"local repository."), path);
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!access(mkpath("%s/shallow", ref_git), F_OK)) {
|
|
|
|
strbuf_addf(err, _("reference repository '%s' is shallow"),
|
|
|
|
path);
|
|
|
|
seen_error = 1;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!access(mkpath("%s/info/grafts", ref_git), F_OK)) {
|
|
|
|
strbuf_addf(err,
|
|
|
|
_("reference repository '%s' is grafted"),
|
|
|
|
path);
|
|
|
|
seen_error = 1;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
out:
|
|
|
|
if (seen_error) {
|
2017-06-15 23:15:46 +00:00
|
|
|
FREE_AND_NULL(ref_git);
|
2016-08-15 21:53:24 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
return ref_git;
|
|
|
|
}
|
|
|
|
|
2014-10-15 22:33:13 +00:00
|
|
|
int foreach_alt_odb(alt_odb_fn fn, void *cb)
|
push: receiver end advertises refs from alternate repositories
Earlier, when pushing into a repository that borrows from alternate object
stores, we followed the longstanding design decision not to trust refs in
the alternate repository that houses the object store we are borrowing
from. If your public repository is borrowing from Linus's public
repository, you pushed into it long time ago, and now when you try to push
your updated history that is in sync with more recent history from Linus,
you will end up sending not just your own development, but also the
changes you acquired through Linus's tree, even though the objects needed
for the latter already exists at the receiving end. This is because the
receiving end does not advertise that the objects only reachable from the
borrowed repository (i.e. Linus's) are already available there.
This solves the issue by making the receiving end advertise refs from
borrowed repositories. They are not sent with their true names but with a
phoney name ".have" to make sure that the old senders will safely ignore
them (otherwise, the old senders will misbehave, trying to push matching
refs, and mirror push that deletes refs that only exist at the receiving
end).
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-09-09 08:27:10 +00:00
|
|
|
{
|
|
|
|
struct alternate_object_database *ent;
|
2014-10-15 22:33:13 +00:00
|
|
|
int r = 0;
|
push: receiver end advertises refs from alternate repositories
Earlier, when pushing into a repository that borrows from alternate object
stores, we followed the longstanding design decision not to trust refs in
the alternate repository that houses the object store we are borrowing
from. If your public repository is borrowing from Linus's public
repository, you pushed into it long time ago, and now when you try to push
your updated history that is in sync with more recent history from Linus,
you will end up sending not just your own development, but also the
changes you acquired through Linus's tree, even though the objects needed
for the latter already exists at the receiving end. This is because the
receiving end does not advertise that the objects only reachable from the
borrowed repository (i.e. Linus's) are already available there.
This solves the issue by making the receiving end advertise refs from
borrowed repositories. They are not sent with their true names but with a
phoney name ".have" to make sure that the old senders will safely ignore
them (otherwise, the old senders will misbehave, trying to push matching
refs, and mirror push that deletes refs that only exist at the receiving
end).
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-09-09 08:27:10 +00:00
|
|
|
|
2018-03-23 17:21:07 +00:00
|
|
|
prepare_alt_odb(the_repository);
|
2018-03-23 17:20:57 +00:00
|
|
|
for (ent = the_repository->objects->alt_odb_list; ent; ent = ent->next) {
|
2014-10-15 22:33:13 +00:00
|
|
|
r = fn(ent, cb);
|
|
|
|
if (r)
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
return r;
|
push: receiver end advertises refs from alternate repositories
Earlier, when pushing into a repository that borrows from alternate object
stores, we followed the longstanding design decision not to trust refs in
the alternate repository that houses the object store we are borrowing
from. If your public repository is borrowing from Linus's public
repository, you pushed into it long time ago, and now when you try to push
your updated history that is in sync with more recent history from Linus,
you will end up sending not just your own development, but also the
changes you acquired through Linus's tree, even though the objects needed
for the latter already exists at the receiving end. This is because the
receiving end does not advertise that the objects only reachable from the
borrowed repository (i.e. Linus's) are already available there.
This solves the issue by making the receiving end advertise refs from
borrowed repositories. They are not sent with their true names but with a
phoney name ".have" to make sure that the old senders will safely ignore
them (otherwise, the old senders will misbehave, trying to push matching
refs, and mirror push that deletes refs that only exist at the receiving
end).
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-09-09 08:27:10 +00:00
|
|
|
}
|
|
|
|
|
2018-03-23 17:21:09 +00:00
|
|
|
void prepare_alt_odb(struct repository *r)
|
2006-05-07 18:19:21 +00:00
|
|
|
{
|
2018-03-23 17:21:09 +00:00
|
|
|
if (r->objects->alt_odb_tail)
|
2007-05-26 05:24:40 +00:00
|
|
|
return;
|
|
|
|
|
2018-03-23 17:21:09 +00:00
|
|
|
r->objects->alt_odb_tail = &r->objects->alt_odb_list;
|
|
|
|
link_alt_odb_entries(r, r->objects->alternate_db, PATH_SEP, NULL, 0);
|
2006-05-07 18:19:21 +00:00
|
|
|
|
2018-03-23 17:21:09 +00:00
|
|
|
read_info_alternates(r, r->objects->objectdir, 0);
|
2006-05-07 18:19:21 +00:00
|
|
|
}
|
|
|
|
|
check_and_freshen_file: fix reversed success-check
When we want to write out a loose object file, we have
always first made sure we don't already have the object
somewhere. Since 33d4221 (write_sha1_file: freshen existing
objects, 2014-10-15), we also update the timestamp on the
file, so that a simultaneous prune knows somebody is
likely to reference it soon.
If our utime() call fails, we treat this the same as not
having the object in the first place; the safe thing to do
is write out another copy. However, the loose-object check
accidentally inverts the utime() check; it returns failure
_only_ when the utime() call actually succeeded. Thus it was
failing to protect us there, and in the normal case where
utime() succeeds, it caused us to pointlessly write out and
link the object.
This passed our freshening tests, because writing out the
new object is certainly _one_ way of updating its utime. So
the normal case was inefficient, but not wrong.
While we're here, let's also drop a comment in front of the
check_and_freshen functions, making a note of their return
type (since it is not our usual "0 for success, -1 for
error").
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-07-08 20:33:52 +00:00
|
|
|
/* Returns 1 if we have successfully freshened the file, 0 otherwise. */
|
2014-10-15 22:42:22 +00:00
|
|
|
static int freshen_file(const char *fn)
|
2005-05-07 07:38:04 +00:00
|
|
|
{
|
2014-10-15 22:42:22 +00:00
|
|
|
struct utimbuf t;
|
|
|
|
t.actime = t.modtime = time(NULL);
|
|
|
|
return !utime(fn, &t);
|
2008-11-10 05:59:57 +00:00
|
|
|
}
|
2005-05-07 07:38:04 +00:00
|
|
|
|
check_and_freshen_file: fix reversed success-check
When we want to write out a loose object file, we have
always first made sure we don't already have the object
somewhere. Since 33d4221 (write_sha1_file: freshen existing
objects, 2014-10-15), we also update the timestamp on the
file, so that a simultaneous prune knows somebody is
likely to reference it soon.
If our utime() call fails, we treat this the same as not
having the object in the first place; the safe thing to do
is write out another copy. However, the loose-object check
accidentally inverts the utime() check; it returns failure
_only_ when the utime() call actually succeeded. Thus it was
failing to protect us there, and in the normal case where
utime() succeeds, it caused us to pointlessly write out and
link the object.
This passed our freshening tests, because writing out the
new object is certainly _one_ way of updating its utime. So
the normal case was inefficient, but not wrong.
While we're here, let's also drop a comment in front of the
check_and_freshen functions, making a note of their return
type (since it is not our usual "0 for success, -1 for
error").
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-07-08 20:33:52 +00:00
|
|
|
/*
|
|
|
|
* All of the check_and_freshen functions return 1 if the file exists and was
|
|
|
|
* freshened (if freshening was requested), 0 otherwise. If they return
|
|
|
|
* 0, you should not assume that it is safe to skip a write of the object (it
|
|
|
|
* either does not exist on disk, or has a stale mtime and may be subject to
|
|
|
|
* pruning).
|
|
|
|
*/
|
2017-02-27 18:00:11 +00:00
|
|
|
int check_and_freshen_file(const char *fn, int freshen)
|
2014-10-15 22:42:22 +00:00
|
|
|
{
|
|
|
|
if (access(fn, F_OK))
|
|
|
|
return 0;
|
check_and_freshen_file: fix reversed success-check
When we want to write out a loose object file, we have
always first made sure we don't already have the object
somewhere. Since 33d4221 (write_sha1_file: freshen existing
objects, 2014-10-15), we also update the timestamp on the
file, so that a simultaneous prune knows somebody is
likely to reference it soon.
If our utime() call fails, we treat this the same as not
having the object in the first place; the safe thing to do
is write out another copy. However, the loose-object check
accidentally inverts the utime() check; it returns failure
_only_ when the utime() call actually succeeded. Thus it was
failing to protect us there, and in the normal case where
utime() succeeds, it caused us to pointlessly write out and
link the object.
This passed our freshening tests, because writing out the
new object is certainly _one_ way of updating its utime. So
the normal case was inefficient, but not wrong.
While we're here, let's also drop a comment in front of the
check_and_freshen functions, making a note of their return
type (since it is not our usual "0 for success, -1 for
error").
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-07-08 20:33:52 +00:00
|
|
|
if (freshen && !freshen_file(fn))
|
2014-10-15 22:42:22 +00:00
|
|
|
return 0;
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
2018-05-02 00:25:34 +00:00
|
|
|
static int check_and_freshen_local(const struct object_id *oid, int freshen)
|
2014-10-15 22:42:22 +00:00
|
|
|
{
|
2018-01-17 17:54:54 +00:00
|
|
|
static struct strbuf buf = STRBUF_INIT;
|
|
|
|
|
|
|
|
strbuf_reset(&buf);
|
2018-05-02 00:25:34 +00:00
|
|
|
sha1_file_name(the_repository, &buf, oid->hash);
|
2018-01-17 17:54:54 +00:00
|
|
|
|
|
|
|
return check_and_freshen_file(buf.buf, freshen);
|
2014-10-15 22:42:22 +00:00
|
|
|
}
|
|
|
|
|
2018-05-02 00:25:34 +00:00
|
|
|
static int check_and_freshen_nonlocal(const struct object_id *oid, int freshen)
|
2008-11-10 05:59:57 +00:00
|
|
|
{
|
|
|
|
struct alternate_object_database *alt;
|
2018-03-23 17:21:07 +00:00
|
|
|
prepare_alt_odb(the_repository);
|
2018-03-23 17:20:57 +00:00
|
|
|
for (alt = the_repository->objects->alt_odb_list; alt; alt = alt->next) {
|
2018-05-02 00:25:34 +00:00
|
|
|
const char *path = alt_sha1_path(alt, oid->hash);
|
2016-10-03 20:35:43 +00:00
|
|
|
if (check_and_freshen_file(path, freshen))
|
2008-06-14 18:43:01 +00:00
|
|
|
return 1;
|
2005-05-07 07:38:04 +00:00
|
|
|
}
|
2008-06-14 18:43:01 +00:00
|
|
|
return 0;
|
2005-05-07 07:38:04 +00:00
|
|
|
}
|
|
|
|
|
2018-05-02 00:25:34 +00:00
|
|
|
static int check_and_freshen(const struct object_id *oid, int freshen)
|
2014-10-15 22:42:22 +00:00
|
|
|
{
|
2018-05-02 00:25:34 +00:00
|
|
|
return check_and_freshen_local(oid, freshen) ||
|
|
|
|
check_and_freshen_nonlocal(oid, freshen);
|
2014-10-15 22:42:22 +00:00
|
|
|
}
|
|
|
|
|
2018-05-02 00:25:34 +00:00
|
|
|
int has_loose_object_nonlocal(const struct object_id *oid)
|
2014-10-15 22:42:22 +00:00
|
|
|
{
|
2018-05-02 00:25:34 +00:00
|
|
|
return check_and_freshen_nonlocal(oid, 0);
|
2014-10-15 22:42:22 +00:00
|
|
|
}
|
|
|
|
|
2018-05-02 00:25:34 +00:00
|
|
|
static int has_loose_object(const struct object_id *oid)
|
2008-11-10 05:59:57 +00:00
|
|
|
{
|
2018-05-02 00:25:34 +00:00
|
|
|
return check_and_freshen(oid, 0);
|
2008-11-10 05:59:57 +00:00
|
|
|
}
|
|
|
|
|
2014-08-26 15:23:23 +00:00
|
|
|
static void mmap_limit_check(size_t length)
|
|
|
|
{
|
|
|
|
static size_t limit = 0;
|
|
|
|
if (!limit) {
|
|
|
|
limit = git_env_ulong("GIT_MMAP_LIMIT", 0);
|
|
|
|
if (!limit)
|
|
|
|
limit = SIZE_MAX;
|
|
|
|
}
|
|
|
|
if (length > limit)
|
2018-07-21 07:49:39 +00:00
|
|
|
die(_("attempting to mmap %"PRIuMAX" over limit %"PRIuMAX),
|
2014-08-26 15:23:23 +00:00
|
|
|
(uintmax_t)length, (uintmax_t)limit);
|
|
|
|
}
|
|
|
|
|
2015-05-28 07:56:15 +00:00
|
|
|
void *xmmap_gently(void *start, size_t length,
|
|
|
|
int prot, int flags, int fd, off_t offset)
|
2010-11-06 11:44:11 +00:00
|
|
|
{
|
2014-08-26 15:23:23 +00:00
|
|
|
void *ret;
|
|
|
|
|
|
|
|
mmap_limit_check(length);
|
|
|
|
ret = mmap(start, length, prot, flags, fd, offset);
|
2010-11-06 11:44:11 +00:00
|
|
|
if (ret == MAP_FAILED) {
|
|
|
|
if (!length)
|
|
|
|
return NULL;
|
2013-07-31 19:51:37 +00:00
|
|
|
release_pack_memory(length);
|
2010-11-06 11:44:11 +00:00
|
|
|
ret = mmap(start, length, prot, flags, fd, offset);
|
|
|
|
}
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2015-05-28 07:56:15 +00:00
|
|
|
void *xmmap(void *start, size_t length,
|
|
|
|
int prot, int flags, int fd, off_t offset)
|
|
|
|
{
|
|
|
|
void *ret = xmmap_gently(start, length, prot, flags, fd, offset);
|
|
|
|
if (ret == MAP_FAILED)
|
2018-07-21 07:49:39 +00:00
|
|
|
die_errno(_("mmap failed"));
|
2015-05-28 07:56:15 +00:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
sha1_file: introduce close_one_pack() to close packs on fd pressure
When the number of open packs exceeds pack_max_fds, unuse_one_window()
is called repeatedly to attempt to release the least-recently-used
pack windows, which, as a side-effect, will also close a pack file
after closing its last open window. If a pack file has been opened,
but no windows have been allocated into it, it will never be selected
by unuse_one_window() and hence its file descriptor will not be
closed. When this happens, git may exceed the number of file
descriptors permitted by the system.
This latter situation can occur in show-ref or receive-pack during ref
advertisement. During ref advertisement, receive-pack will iterate
over every ref in the repository and advertise it to the client after
ensuring that the ref exists in the local repository. If the ref is
located inside a pack, then the pack is opened to ensure that it
exists, but since the object is not actually read from the pack, no
mmap windows are allocated. When the number of open packs exceeds
pack_max_fds, unuse_one_window() will not be able to find any windows to
free and will not be able to close any packs. Once the per-process
file descriptor limit is exceeded, receive-pack will produce a warning,
not an error, for each pack it cannot open, and will then most likely
fail with an error to spawn rev-list or index-pack like:
error: cannot create standard input pipe for rev-list: Too many open files
error: Could not run 'git rev-list'
This may also occur during upload-pack when refs are packed (in the
packed-refs file) and the number of packs that must be opened to
verify that these packed refs exist exceeds the file descriptor
limit. If the refs are loose, then upload-pack will read each ref
from the object database (if the object is in a pack, allocating one
or more mmap windows for it) in order to peel tags and advertise the
underlying object. But when the refs are packed and peeled,
upload-pack will use the peeled sha1 in the packed-refs file and
will not need to read from the pack files, so no mmap windows will
be allocated and just like with receive-pack, unuse_one_window()
will never select these opened packs to close.
When we have file descriptor pressure, we just need to find an open
pack to close. We can leave the existing mmap windows open. If
additional windows need to be mapped into the pack file, it will be
reopened when necessary. If the pack file has been rewritten in the
mean time, open_packed_git_1() should notice when it compares the file
size or the pack's sha1 checksum to what was previously read from the
pack index, and reject it.
Let's introduce a new function close_one_pack() designed specifically
for this purpose to search for and close the least-recently-used pack,
where LRU is defined as (in order of preference):
* pack with oldest mtime and no allocated mmap windows
* pack with the least-recently-used windows, i.e. the pack
with the oldest most-recently-used window, where none of
the windows are in use
* pack with the least-recently-used windows
Signed-off-by: Brandon Casey <drafnel@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-08-02 05:36:33 +00:00
|
|
|
/*
|
2012-03-07 10:54:18 +00:00
|
|
|
* With an in-core object data in "map", rehash it to make sure the
|
|
|
|
* object name actually matches "sha1" to detect object corruption.
|
|
|
|
* With "map" == NULL, try reading the object named with "sha1" using
|
|
|
|
* the streaming interface and rehash it to do the same.
|
sha1_file: introduce close_one_pack() to close packs on fd pressure
When the number of open packs exceeds pack_max_fds, unuse_one_window()
is called repeatedly to attempt to release the least-recently-used
pack windows, which, as a side-effect, will also close a pack file
after closing its last open window. If a pack file has been opened,
but no windows have been allocated into it, it will never be selected
by unuse_one_window() and hence its file descriptor will not be
closed. When this happens, git may exceed the number of file
descriptors permitted by the system.
This latter situation can occur in show-ref or receive-pack during ref
advertisement. During ref advertisement, receive-pack will iterate
over every ref in the repository and advertise it to the client after
ensuring that the ref exists in the local repository. If the ref is
located inside a pack, then the pack is opened to ensure that it
exists, but since the object is not actually read from the pack, no
mmap windows are allocated. When the number of open packs exceeds
pack_max_fds, unuse_one_window() will not be able to find any windows to
free and will not be able to close any packs. Once the per-process
file descriptor limit is exceeded, receive-pack will produce a warning,
not an error, for each pack it cannot open, and will then most likely
fail with an error to spawn rev-list or index-pack like:
error: cannot create standard input pipe for rev-list: Too many open files
error: Could not run 'git rev-list'
This may also occur during upload-pack when refs are packed (in the
packed-refs file) and the number of packs that must be opened to
verify that these packed refs exist exceeds the file descriptor
limit. If the refs are loose, then upload-pack will read each ref
from the object database (if the object is in a pack, allocating one
or more mmap windows for it) in order to peel tags and advertise the
underlying object. But when the refs are packed and peeled,
upload-pack will use the peeled sha1 in the packed-refs file and
will not need to read from the pack files, so no mmap windows will
be allocated and just like with receive-pack, unuse_one_window()
will never select these opened packs to close.
When we have file descriptor pressure, we just need to find an open
pack to close. We can leave the existing mmap windows open. If
additional windows need to be mapped into the pack file, it will be
reopened when necessary. If the pack file has been rewritten in the
mean time, open_packed_git_1() should notice when it compares the file
size or the pack's sha1 checksum to what was previously read from the
pack index, and reject it.
Let's introduce a new function close_one_pack() designed specifically
for this purpose to search for and close the least-recently-used pack,
where LRU is defined as (in order of preference):
* pack with oldest mtime and no allocated mmap windows
* pack with the least-recently-used windows, i.e. the pack
with the oldest most-recently-used window, where none of
the windows are in use
* pack with the least-recently-used windows
Signed-off-by: Brandon Casey <drafnel@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-08-02 05:36:33 +00:00
|
|
|
*/
|
2018-03-12 02:27:39 +00:00
|
|
|
int check_object_signature(const struct object_id *oid, void *map,
|
|
|
|
unsigned long size, const char *type)
|
sha1_file: introduce close_one_pack() to close packs on fd pressure
When the number of open packs exceeds pack_max_fds, unuse_one_window()
is called repeatedly to attempt to release the least-recently-used
pack windows, which, as a side-effect, will also close a pack file
after closing its last open window. If a pack file has been opened,
but no windows have been allocated into it, it will never be selected
by unuse_one_window() and hence its file descriptor will not be
closed. When this happens, git may exceed the number of file
descriptors permitted by the system.
This latter situation can occur in show-ref or receive-pack during ref
advertisement. During ref advertisement, receive-pack will iterate
over every ref in the repository and advertise it to the client after
ensuring that the ref exists in the local repository. If the ref is
located inside a pack, then the pack is opened to ensure that it
exists, but since the object is not actually read from the pack, no
mmap windows are allocated. When the number of open packs exceeds
pack_max_fds, unuse_one_window() will not be able to find any windows to
free and will not be able to close any packs. Once the per-process
file descriptor limit is exceeded, receive-pack will produce a warning,
not an error, for each pack it cannot open, and will then most likely
fail with an error to spawn rev-list or index-pack like:
error: cannot create standard input pipe for rev-list: Too many open files
error: Could not run 'git rev-list'
This may also occur during upload-pack when refs are packed (in the
packed-refs file) and the number of packs that must be opened to
verify that these packed refs exist exceeds the file descriptor
limit. If the refs are loose, then upload-pack will read each ref
from the object database (if the object is in a pack, allocating one
or more mmap windows for it) in order to peel tags and advertise the
underlying object. But when the refs are packed and peeled,
upload-pack will use the peeled sha1 in the packed-refs file and
will not need to read from the pack files, so no mmap windows will
be allocated and just like with receive-pack, unuse_one_window()
will never select these opened packs to close.
When we have file descriptor pressure, we just need to find an open
pack to close. We can leave the existing mmap windows open. If
additional windows need to be mapped into the pack file, it will be
reopened when necessary. If the pack file has been rewritten in the
mean time, open_packed_git_1() should notice when it compares the file
size or the pack's sha1 checksum to what was previously read from the
pack index, and reject it.
Let's introduce a new function close_one_pack() designed specifically
for this purpose to search for and close the least-recently-used pack,
where LRU is defined as (in order of preference):
* pack with oldest mtime and no allocated mmap windows
* pack with the least-recently-used windows, i.e. the pack
with the oldest most-recently-used window, where none of
the windows are in use
* pack with the least-recently-used windows
Signed-off-by: Brandon Casey <drafnel@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-08-02 05:36:33 +00:00
|
|
|
{
|
2018-01-28 00:13:13 +00:00
|
|
|
struct object_id real_oid;
|
2012-03-07 10:54:18 +00:00
|
|
|
enum object_type obj_type;
|
|
|
|
struct git_istream *st;
|
2018-02-01 02:18:41 +00:00
|
|
|
git_hash_ctx c;
|
2018-03-12 02:27:55 +00:00
|
|
|
char hdr[MAX_HEADER_LEN];
|
2012-03-07 10:54:18 +00:00
|
|
|
int hdrlen;
|
sha1_file: introduce close_one_pack() to close packs on fd pressure
When the number of open packs exceeds pack_max_fds, unuse_one_window()
is called repeatedly to attempt to release the least-recently-used
pack windows, which, as a side-effect, will also close a pack file
after closing its last open window. If a pack file has been opened,
but no windows have been allocated into it, it will never be selected
by unuse_one_window() and hence its file descriptor will not be
closed. When this happens, git may exceed the number of file
descriptors permitted by the system.
This latter situation can occur in show-ref or receive-pack during ref
advertisement. During ref advertisement, receive-pack will iterate
over every ref in the repository and advertise it to the client after
ensuring that the ref exists in the local repository. If the ref is
located inside a pack, then the pack is opened to ensure that it
exists, but since the object is not actually read from the pack, no
mmap windows are allocated. When the number of open packs exceeds
pack_max_fds, unuse_one_window() will not be able to find any windows to
free and will not be able to close any packs. Once the per-process
file descriptor limit is exceeded, receive-pack will produce a warning,
not an error, for each pack it cannot open, and will then most likely
fail with an error to spawn rev-list or index-pack like:
error: cannot create standard input pipe for rev-list: Too many open files
error: Could not run 'git rev-list'
This may also occur during upload-pack when refs are packed (in the
packed-refs file) and the number of packs that must be opened to
verify that these packed refs exist exceeds the file descriptor
limit. If the refs are loose, then upload-pack will read each ref
from the object database (if the object is in a pack, allocating one
or more mmap windows for it) in order to peel tags and advertise the
underlying object. But when the refs are packed and peeled,
upload-pack will use the peeled sha1 in the packed-refs file and
will not need to read from the pack files, so no mmap windows will
be allocated and just like with receive-pack, unuse_one_window()
will never select these opened packs to close.
When we have file descriptor pressure, we just need to find an open
pack to close. We can leave the existing mmap windows open. If
additional windows need to be mapped into the pack file, it will be
reopened when necessary. If the pack file has been rewritten in the
mean time, open_packed_git_1() should notice when it compares the file
size or the pack's sha1 checksum to what was previously read from the
pack index, and reject it.
Let's introduce a new function close_one_pack() designed specifically
for this purpose to search for and close the least-recently-used pack,
where LRU is defined as (in order of preference):
* pack with oldest mtime and no allocated mmap windows
* pack with the least-recently-used windows, i.e. the pack
with the oldest most-recently-used window, where none of
the windows are in use
* pack with the least-recently-used windows
Signed-off-by: Brandon Casey <drafnel@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-08-02 05:36:33 +00:00
|
|
|
|
2012-03-07 10:54:18 +00:00
|
|
|
if (map) {
|
2018-01-28 00:13:13 +00:00
|
|
|
hash_object_file(map, size, type, &real_oid);
|
2018-08-28 21:22:48 +00:00
|
|
|
return !oideq(oid, &real_oid) ? -1 : 0;
|
sha1_file: introduce close_one_pack() to close packs on fd pressure
When the number of open packs exceeds pack_max_fds, unuse_one_window()
is called repeatedly to attempt to release the least-recently-used
pack windows, which, as a side-effect, will also close a pack file
after closing its last open window. If a pack file has been opened,
but no windows have been allocated into it, it will never be selected
by unuse_one_window() and hence its file descriptor will not be
closed. When this happens, git may exceed the number of file
descriptors permitted by the system.
This latter situation can occur in show-ref or receive-pack during ref
advertisement. During ref advertisement, receive-pack will iterate
over every ref in the repository and advertise it to the client after
ensuring that the ref exists in the local repository. If the ref is
located inside a pack, then the pack is opened to ensure that it
exists, but since the object is not actually read from the pack, no
mmap windows are allocated. When the number of open packs exceeds
pack_max_fds, unuse_one_window() will not be able to find any windows to
free and will not be able to close any packs. Once the per-process
file descriptor limit is exceeded, receive-pack will produce a warning,
not an error, for each pack it cannot open, and will then most likely
fail with an error to spawn rev-list or index-pack like:
error: cannot create standard input pipe for rev-list: Too many open files
error: Could not run 'git rev-list'
This may also occur during upload-pack when refs are packed (in the
packed-refs file) and the number of packs that must be opened to
verify that these packed refs exist exceeds the file descriptor
limit. If the refs are loose, then upload-pack will read each ref
from the object database (if the object is in a pack, allocating one
or more mmap windows for it) in order to peel tags and advertise the
underlying object. But when the refs are packed and peeled,
upload-pack will use the peeled sha1 in the packed-refs file and
will not need to read from the pack files, so no mmap windows will
be allocated and just like with receive-pack, unuse_one_window()
will never select these opened packs to close.
When we have file descriptor pressure, we just need to find an open
pack to close. We can leave the existing mmap windows open. If
additional windows need to be mapped into the pack file, it will be
reopened when necessary. If the pack file has been rewritten in the
mean time, open_packed_git_1() should notice when it compares the file
size or the pack's sha1 checksum to what was previously read from the
pack index, and reject it.
Let's introduce a new function close_one_pack() designed specifically
for this purpose to search for and close the least-recently-used pack,
where LRU is defined as (in order of preference):
* pack with oldest mtime and no allocated mmap windows
* pack with the least-recently-used windows, i.e. the pack
with the oldest most-recently-used window, where none of
the windows are in use
* pack with the least-recently-used windows
Signed-off-by: Brandon Casey <drafnel@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-08-02 05:36:33 +00:00
|
|
|
}
|
|
|
|
|
2018-03-12 02:27:40 +00:00
|
|
|
st = open_istream(oid, &obj_type, &size, NULL);
|
2012-03-07 10:54:18 +00:00
|
|
|
if (!st)
|
|
|
|
return -1;
|
sha1_file: introduce close_one_pack() to close packs on fd pressure
When the number of open packs exceeds pack_max_fds, unuse_one_window()
is called repeatedly to attempt to release the least-recently-used
pack windows, which, as a side-effect, will also close a pack file
after closing its last open window. If a pack file has been opened,
but no windows have been allocated into it, it will never be selected
by unuse_one_window() and hence its file descriptor will not be
closed. When this happens, git may exceed the number of file
descriptors permitted by the system.
This latter situation can occur in show-ref or receive-pack during ref
advertisement. During ref advertisement, receive-pack will iterate
over every ref in the repository and advertise it to the client after
ensuring that the ref exists in the local repository. If the ref is
located inside a pack, then the pack is opened to ensure that it
exists, but since the object is not actually read from the pack, no
mmap windows are allocated. When the number of open packs exceeds
pack_max_fds, unuse_one_window() will not be able to find any windows to
free and will not be able to close any packs. Once the per-process
file descriptor limit is exceeded, receive-pack will produce a warning,
not an error, for each pack it cannot open, and will then most likely
fail with an error to spawn rev-list or index-pack like:
error: cannot create standard input pipe for rev-list: Too many open files
error: Could not run 'git rev-list'
This may also occur during upload-pack when refs are packed (in the
packed-refs file) and the number of packs that must be opened to
verify that these packed refs exist exceeds the file descriptor
limit. If the refs are loose, then upload-pack will read each ref
from the object database (if the object is in a pack, allocating one
or more mmap windows for it) in order to peel tags and advertise the
underlying object. But when the refs are packed and peeled,
upload-pack will use the peeled sha1 in the packed-refs file and
will not need to read from the pack files, so no mmap windows will
be allocated and just like with receive-pack, unuse_one_window()
will never select these opened packs to close.
When we have file descriptor pressure, we just need to find an open
pack to close. We can leave the existing mmap windows open. If
additional windows need to be mapped into the pack file, it will be
reopened when necessary. If the pack file has been rewritten in the
mean time, open_packed_git_1() should notice when it compares the file
size or the pack's sha1 checksum to what was previously read from the
pack index, and reject it.
Let's introduce a new function close_one_pack() designed specifically
for this purpose to search for and close the least-recently-used pack,
where LRU is defined as (in order of preference):
* pack with oldest mtime and no allocated mmap windows
* pack with the least-recently-used windows, i.e. the pack
with the oldest most-recently-used window, where none of
the windows are in use
* pack with the least-recently-used windows
Signed-off-by: Brandon Casey <drafnel@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-08-02 05:36:33 +00:00
|
|
|
|
2012-03-07 10:54:18 +00:00
|
|
|
/* Generate the header */
|
2018-02-14 18:59:24 +00:00
|
|
|
hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %lu", type_name(obj_type), size) + 1;
|
sha1_file: introduce close_one_pack() to close packs on fd pressure
When the number of open packs exceeds pack_max_fds, unuse_one_window()
is called repeatedly to attempt to release the least-recently-used
pack windows, which, as a side-effect, will also close a pack file
after closing its last open window. If a pack file has been opened,
but no windows have been allocated into it, it will never be selected
by unuse_one_window() and hence its file descriptor will not be
closed. When this happens, git may exceed the number of file
descriptors permitted by the system.
This latter situation can occur in show-ref or receive-pack during ref
advertisement. During ref advertisement, receive-pack will iterate
over every ref in the repository and advertise it to the client after
ensuring that the ref exists in the local repository. If the ref is
located inside a pack, then the pack is opened to ensure that it
exists, but since the object is not actually read from the pack, no
mmap windows are allocated. When the number of open packs exceeds
pack_max_fds, unuse_one_window() will not be able to find any windows to
free and will not be able to close any packs. Once the per-process
file descriptor limit is exceeded, receive-pack will produce a warning,
not an error, for each pack it cannot open, and will then most likely
fail with an error to spawn rev-list or index-pack like:
error: cannot create standard input pipe for rev-list: Too many open files
error: Could not run 'git rev-list'
This may also occur during upload-pack when refs are packed (in the
packed-refs file) and the number of packs that must be opened to
verify that these packed refs exist exceeds the file descriptor
limit. If the refs are loose, then upload-pack will read each ref
from the object database (if the object is in a pack, allocating one
or more mmap windows for it) in order to peel tags and advertise the
underlying object. But when the refs are packed and peeled,
upload-pack will use the peeled sha1 in the packed-refs file and
will not need to read from the pack files, so no mmap windows will
be allocated and just like with receive-pack, unuse_one_window()
will never select these opened packs to close.
When we have file descriptor pressure, we just need to find an open
pack to close. We can leave the existing mmap windows open. If
additional windows need to be mapped into the pack file, it will be
reopened when necessary. If the pack file has been rewritten in the
mean time, open_packed_git_1() should notice when it compares the file
size or the pack's sha1 checksum to what was previously read from the
pack index, and reject it.
Let's introduce a new function close_one_pack() designed specifically
for this purpose to search for and close the least-recently-used pack,
where LRU is defined as (in order of preference):
* pack with oldest mtime and no allocated mmap windows
* pack with the least-recently-used windows, i.e. the pack
with the oldest most-recently-used window, where none of
the windows are in use
* pack with the least-recently-used windows
Signed-off-by: Brandon Casey <drafnel@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-08-02 05:36:33 +00:00
|
|
|
|
2012-03-07 10:54:18 +00:00
|
|
|
/* Sha1.. */
|
2018-02-01 02:18:41 +00:00
|
|
|
the_hash_algo->init_fn(&c);
|
|
|
|
the_hash_algo->update_fn(&c, hdr, hdrlen);
|
2012-03-07 10:54:18 +00:00
|
|
|
for (;;) {
|
|
|
|
char buf[1024 * 16];
|
|
|
|
ssize_t readlen = read_istream(st, buf, sizeof(buf));
|
2005-06-27 10:35:33 +00:00
|
|
|
|
2013-03-25 20:17:17 +00:00
|
|
|
if (readlen < 0) {
|
|
|
|
close_istream(st);
|
|
|
|
return -1;
|
|
|
|
}
|
2012-03-07 10:54:18 +00:00
|
|
|
if (!readlen)
|
|
|
|
break;
|
2018-02-01 02:18:41 +00:00
|
|
|
the_hash_algo->update_fn(&c, buf, readlen);
|
2010-04-19 14:23:06 +00:00
|
|
|
}
|
2018-02-01 02:18:41 +00:00
|
|
|
the_hash_algo->final_fn(real_oid.hash, &c);
|
2012-03-07 10:54:18 +00:00
|
|
|
close_istream(st);
|
2018-08-28 21:22:48 +00:00
|
|
|
return !oideq(oid, &real_oid) ? -1 : 0;
|
2010-04-19 14:23:06 +00:00
|
|
|
}
|
|
|
|
|
2016-10-28 13:23:07 +00:00
|
|
|
int git_open_cloexec(const char *name, int flags)
|
2012-08-24 09:52:22 +00:00
|
|
|
{
|
2016-10-31 17:41:41 +00:00
|
|
|
int fd;
|
|
|
|
static int o_cloexec = O_CLOEXEC;
|
2012-08-24 09:52:22 +00:00
|
|
|
|
2016-10-31 17:41:41 +00:00
|
|
|
fd = open(name, flags | o_cloexec);
|
|
|
|
if ((o_cloexec & O_CLOEXEC) && fd < 0 && errno == EINVAL) {
|
2016-10-24 18:02:59 +00:00
|
|
|
/* Try again w/o O_CLOEXEC: the kernel might not support it */
|
2016-10-31 17:41:41 +00:00
|
|
|
o_cloexec &= ~O_CLOEXEC;
|
|
|
|
fd = open(name, flags | o_cloexec);
|
2013-12-18 22:59:12 +00:00
|
|
|
}
|
|
|
|
|
2017-07-15 18:55:40 +00:00
|
|
|
#if defined(F_GETFD) && defined(F_SETFD) && defined(FD_CLOEXEC)
|
2013-12-18 22:59:12 +00:00
|
|
|
{
|
2016-10-31 17:41:41 +00:00
|
|
|
static int fd_cloexec = FD_CLOEXEC;
|
2012-08-24 09:52:22 +00:00
|
|
|
|
2016-10-31 17:41:41 +00:00
|
|
|
if (!o_cloexec && 0 <= fd && fd_cloexec) {
|
|
|
|
/* Opened w/o O_CLOEXEC? try with fcntl(2) to add it */
|
2017-07-15 18:55:40 +00:00
|
|
|
int flags = fcntl(fd, F_GETFD);
|
|
|
|
if (fcntl(fd, F_SETFD, flags | fd_cloexec))
|
2016-10-31 17:41:41 +00:00
|
|
|
fd_cloexec = 0;
|
2016-10-24 18:02:59 +00:00
|
|
|
}
|
2008-06-14 18:32:37 +00:00
|
|
|
}
|
2012-08-24 09:52:22 +00:00
|
|
|
#endif
|
2016-10-28 13:23:07 +00:00
|
|
|
return fd;
|
2012-08-24 09:52:22 +00:00
|
|
|
}
|
|
|
|
|
2007-02-02 08:00:03 +00:00
|
|
|
/*
|
2017-01-13 17:54:39 +00:00
|
|
|
* Find "sha1" as a loose object in the local repository or in an alternate.
|
|
|
|
* Returns 0 on success, negative on failure.
|
|
|
|
*
|
|
|
|
* The "path" out-parameter will give the path of the object we found (if any).
|
|
|
|
* Note that it may point to static storage and is only valid until another
|
|
|
|
* call to sha1_file_name(), etc.
|
2007-02-02 08:00:03 +00:00
|
|
|
*/
|
2018-03-23 17:21:17 +00:00
|
|
|
static int stat_sha1_file(struct repository *r, const unsigned char *sha1,
|
|
|
|
struct stat *st, const char **path)
|
2005-06-27 10:35:33 +00:00
|
|
|
{
|
sha1_loose_object_info: make type lookup optional
Until recently, the only items to request from
sha1_object_info_extended were type and size. This meant
that we always had to open a loose object file to determine
one or the other. But with the addition of the disk_size
query, it's possible that we can fulfill the query without
even opening the object file at all. However, since the
function interface always returns the type, we have no way
of knowing whether the caller cares about it or not.
This patch only modified sha1_loose_object_info to make type
lookup optional using an out-parameter, similar to the way
the size is handled (and the return value is "0" or "-1" for
success or error, respectively).
There should be no functional change yet, though, as
sha1_object_info_extended, the only caller, will always ask
for a type.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-07-12 06:30:48 +00:00
|
|
|
struct alternate_object_database *alt;
|
2018-01-17 17:54:54 +00:00
|
|
|
static struct strbuf buf = STRBUF_INIT;
|
|
|
|
|
|
|
|
strbuf_reset(&buf);
|
2018-03-23 17:21:17 +00:00
|
|
|
sha1_file_name(r, &buf, sha1);
|
2018-01-17 17:54:54 +00:00
|
|
|
*path = buf.buf;
|
2007-05-26 05:24:19 +00:00
|
|
|
|
2017-01-13 17:54:39 +00:00
|
|
|
if (!lstat(*path, st))
|
sha1_loose_object_info: make type lookup optional
Until recently, the only items to request from
sha1_object_info_extended were type and size. This meant
that we always had to open a loose object file to determine
one or the other. But with the addition of the disk_size
query, it's possible that we can fulfill the query without
even opening the object file at all. However, since the
function interface always returns the type, we have no way
of knowing whether the caller cares about it or not.
This patch only modified sha1_loose_object_info to make type
lookup optional using an out-parameter, similar to the way
the size is handled (and the return value is "0" or "-1" for
success or error, respectively).
There should be no functional change yet, though, as
sha1_object_info_extended, the only caller, will always ask
for a type.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-07-12 06:30:48 +00:00
|
|
|
return 0;
|
2011-02-28 20:52:39 +00:00
|
|
|
|
2018-03-23 17:21:17 +00:00
|
|
|
prepare_alt_odb(r);
|
sha1_loose_object_info: make type lookup optional
Until recently, the only items to request from
sha1_object_info_extended were type and size. This meant
that we always had to open a loose object file to determine
one or the other. But with the addition of the disk_size
query, it's possible that we can fulfill the query without
even opening the object file at all. However, since the
function interface always returns the type, we have no way
of knowing whether the caller cares about it or not.
This patch only modified sha1_loose_object_info to make type
lookup optional using an out-parameter, similar to the way
the size is handled (and the return value is "0" or "-1" for
success or error, respectively).
There should be no functional change yet, though, as
sha1_object_info_extended, the only caller, will always ask
for a type.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-07-12 06:30:48 +00:00
|
|
|
errno = ENOENT;
|
2018-03-23 17:21:17 +00:00
|
|
|
for (alt = r->objects->alt_odb_list; alt; alt = alt->next) {
|
2017-01-13 17:54:39 +00:00
|
|
|
*path = alt_sha1_path(alt, sha1);
|
|
|
|
if (!lstat(*path, st))
|
sha1_loose_object_info: make type lookup optional
Until recently, the only items to request from
sha1_object_info_extended were type and size. This meant
that we always had to open a loose object file to determine
one or the other. But with the addition of the disk_size
query, it's possible that we can fulfill the query without
even opening the object file at all. However, since the
function interface always returns the type, we have no way
of knowing whether the caller cares about it or not.
This patch only modified sha1_loose_object_info to make type
lookup optional using an out-parameter, similar to the way
the size is handled (and the return value is "0" or "-1" for
success or error, respectively).
There should be no functional change yet, though, as
sha1_object_info_extended, the only caller, will always ask
for a type.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-07-12 06:30:48 +00:00
|
|
|
return 0;
|
2011-02-28 20:52:39 +00:00
|
|
|
}
|
|
|
|
|
2007-02-02 08:00:03 +00:00
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
2017-01-13 17:54:39 +00:00
|
|
|
/*
|
|
|
|
* Like stat_sha1_file(), but actually open the object and return the
|
|
|
|
* descriptor. See the caveats on the "path" parameter above.
|
|
|
|
*/
|
2018-03-23 17:21:18 +00:00
|
|
|
static int open_sha1_file(struct repository *r,
|
|
|
|
const unsigned char *sha1, const char **path)
|
2006-12-23 07:34:28 +00:00
|
|
|
{
|
2008-06-14 18:32:37 +00:00
|
|
|
int fd;
|
|
|
|
struct alternate_object_database *alt;
|
2014-05-15 08:54:06 +00:00
|
|
|
int most_interesting_errno;
|
2018-01-17 17:54:54 +00:00
|
|
|
static struct strbuf buf = STRBUF_INIT;
|
|
|
|
|
|
|
|
strbuf_reset(&buf);
|
2018-03-23 17:21:18 +00:00
|
|
|
sha1_file_name(r, &buf, sha1);
|
2018-01-17 17:54:54 +00:00
|
|
|
*path = buf.buf;
|
2006-12-23 07:34:28 +00:00
|
|
|
|
2017-01-13 17:54:39 +00:00
|
|
|
fd = git_open(*path);
|
2008-06-14 18:32:37 +00:00
|
|
|
if (fd >= 0)
|
|
|
|
return fd;
|
2014-05-15 08:54:06 +00:00
|
|
|
most_interesting_errno = errno;
|
2006-12-23 07:34:08 +00:00
|
|
|
|
2018-03-23 17:21:18 +00:00
|
|
|
prepare_alt_odb(r);
|
|
|
|
for (alt = r->objects->alt_odb_list; alt; alt = alt->next) {
|
2017-01-13 17:54:39 +00:00
|
|
|
*path = alt_sha1_path(alt, sha1);
|
|
|
|
fd = git_open(*path);
|
2008-06-14 18:32:37 +00:00
|
|
|
if (fd >= 0)
|
|
|
|
return fd;
|
2014-05-15 08:54:06 +00:00
|
|
|
if (most_interesting_errno == ENOENT)
|
|
|
|
most_interesting_errno = errno;
|
2006-12-23 07:34:08 +00:00
|
|
|
}
|
2014-05-15 08:54:06 +00:00
|
|
|
errno = most_interesting_errno;
|
2008-06-14 18:32:37 +00:00
|
|
|
return -1;
|
2005-06-27 10:35:33 +00:00
|
|
|
}
|
|
|
|
|
2017-01-13 17:58:16 +00:00
|
|
|
/*
|
|
|
|
* Map the loose object at "path" if it is not NULL, or the path found by
|
|
|
|
* searching for a loose object named "sha1".
|
|
|
|
*/
|
2018-03-23 17:21:19 +00:00
|
|
|
static void *map_sha1_file_1(struct repository *r, const char *path,
|
|
|
|
const unsigned char *sha1, unsigned long *size)
|
2008-06-24 22:58:06 +00:00
|
|
|
{
|
2005-04-18 20:04:43 +00:00
|
|
|
void *map;
|
2005-04-23 18:09:32 +00:00
|
|
|
int fd;
|
2013-03-27 20:03:41 +00:00
|
|
|
|
2017-01-13 17:58:16 +00:00
|
|
|
if (path)
|
|
|
|
fd = git_open(path);
|
delta_base_cache: use hashmap.h
The fundamental data structure of the delta base cache is a
hash table mapping pairs of "(packfile, offset)" into
structs containing the actual object data. The hash table
implementation dates back to e5e0161 (Implement a simple
delta_base cache, 2007-03-17), and uses a fixed-size table.
The current size is a hard-coded 256 entries.
Because we need to be able to remove objects from the hash
table, entry lookup does not do any kind of probing to
handle collisions. Colliding items simply replace whatever
is in their slot. As a result, we have fewer usable slots
than even the 256 we allocate. At half full, each new item
has a 50% chance of displacing another one. Or another way
to think about it: every item has a 1/256 chance of being
ejected due to hash collision, without regard to our LRU
strategy.
So it would be interesting to see the effect of increasing
the cache size on the runtime for some common operations. As
with the previous patch, we'll measure "git log --raw" for
tree-only operations, and "git log -Sfoo --raw" for
operations that touch trees and blobs. All times are
wall-clock best-of-3, done against fully packed repos with
--depth=50, and the default core.deltaBaseCacheLimit of
96MB.
Here are timings for various values of MAX_DELTA_CACHE
against git.git (the asterisk marks the minimum time for
each operation):
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m02.227s 0m12.821s
512 0m02.143s 0m10.602s
1024 0m02.127s 0m08.642s
2048 0m02.148s 0m07.123s
4096 0m02.194s 0m06.448s*
8192 0m02.239s 0m06.504s
16384 0m02.144s* 0m06.502s
32768 0m02.202s 0m06.622s
65536 0m02.230s 0m06.677s
The log-raw case isn't changed much at all here (probably
because our trees just aren't that big in the first place,
or possibly because we have so _few_ trees in git.git that
the 256-entry cache is enough). But once we start putting
blobs in the cache, too, we see a big improvement (almost
50%). The curve levels off around 4096, which means that we
can hold about that many entries before hitting the 96MB
memory limit (or possibly that the workload is small enough
that there is simply no more work to be optimized out by
caching more).
(As a side note, I initially timed my existing git.git pack,
which was a base of --aggressive combined with some pulls on
top. So it had quite a few deeper delta chains. The
256-cache case was more like 15s, and it still dropped to
~6.5s in the same way).
Here are the timings for linux.git:
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m41.661s 5m12.410s
512 0m39.547s 5m07.920s
1024 0m37.054s 4m54.666s
2048 0m35.871s 4m41.194s*
4096 0m34.646s 4m51.648s
8192 0m33.881s 4m55.342s
16384 0m35.190s 5m00.122s
32768 0m35.060s 4m58.851s
65536 0m33.311s* 4m51.420s
As we grow we see a nice 20% speedup in the tree traversal,
and more modest 10% in the log-S. This is probably an
indication that we are bound less by the number of entries,
and more by the memory limit (more on that below). What is
interesting is that the numbers bounce around a bit;
increasing the number of entries isn't always a strict
improvement.
Partially this is due to noise in the measurement. But it
may also be an indication that our LRU ejection scheme is
not optimal. The smaller cache sizes introduce some
randomness into the ejection (due to collisions), which may
sometimes work in our favor (and sometimes not!).
So what is the optimal setting of MAX_DELTA_CACHE? The
"bouncing" in the linux.git log-S numbers notwithstanding,
it mostly seems like bigger is better. And even if we were
to try to find a "sweet spot", these are just two
repositories, that are not necessarily representative. The
shape of history, the size of trees and blobs, the memory
limit configuration, etc, all will affect the outcome.
Rather than trying to find the "right" number, another
strategy is to just switch to a hash table that can actually
store collisions: namely our hashmap.h implementation.
Here are numbers for that compared to the "best" we saw from
adjusting MAX_DELTA_CACHE:
| log-raw | log-S
| best hashmap | best hashmap
| --------- --------- | --------- ---------
git | 0m02.144s 0m02.144s | 0m06.448s 0m06.688s
linux | 0m33.311s 0m33.092s | 4m41.194s 4m57.172s
We can see the results are similar in most cases, which is
what we'd expect. We're not ejecting due to collisions at
all, so this is purely representing the LRU. So really, we'd
expect this to model most closely the larger values of the
static MAX_DELTA_CACHE limit. And that does seem to be
what's happening, including the "bounce" in the linux log-S
case.
So while the value for that case _isn't_ as good as the
optimal one measured above (which was 2048 entries), given
the bouncing I'm hesitant to suggest that 2048 is any kind
of optimum (not even for linux.git, let alone as a general
rule). The generic hashmap has the appeal that it drops the
number of tweakable numbers by one, which means we can focus
on tuning other elements, like the LRU strategy or the
core.deltaBaseCacheLimit setting.
And indeed, if we bump the cache limit to 1G (which is
probably silly for general use, but maybe something people
with big workstations would want to do), the linux.git log-S
time drops to 3m32s. That's something you really _can't_ do
easily with the static hash table, because the number of
entries needs to grow in proportion to the memory limit (so
2048 is almost certainly not going to be the right value
there).
This patch takes that direction, and drops the static hash
table entirely in favor of using the hashmap.h API.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-22 22:00:07 +00:00
|
|
|
else
|
2018-03-23 17:21:19 +00:00
|
|
|
fd = open_sha1_file(r, sha1, &path);
|
2008-06-14 18:32:37 +00:00
|
|
|
map = NULL;
|
|
|
|
if (fd >= 0) {
|
|
|
|
struct stat st;
|
2005-06-27 10:35:33 +00:00
|
|
|
|
2008-06-14 18:32:37 +00:00
|
|
|
if (!fstat(fd, &st)) {
|
|
|
|
*size = xsize_t(st.st_size);
|
2012-02-06 16:24:52 +00:00
|
|
|
if (!*size) {
|
|
|
|
/* mmap() is forbidden on empty files */
|
2018-07-21 07:49:39 +00:00
|
|
|
error(_("object file %s is empty"), path);
|
2012-02-06 16:24:52 +00:00
|
|
|
return NULL;
|
|
|
|
}
|
2008-06-14 18:32:37 +00:00
|
|
|
map = xmmap(NULL, *size, PROT_READ, MAP_PRIVATE, fd, 0);
|
2005-04-23 18:09:32 +00:00
|
|
|
}
|
2008-06-14 18:32:37 +00:00
|
|
|
close(fd);
|
2007-04-09 05:06:35 +00:00
|
|
|
}
|
2005-04-18 20:04:43 +00:00
|
|
|
return map;
|
2007-04-09 05:06:35 +00:00
|
|
|
}
|
|
|
|
|
2018-03-23 17:21:20 +00:00
|
|
|
void *map_sha1_file(struct repository *r,
|
|
|
|
const unsigned char *sha1, unsigned long *size)
|
2017-02-21 23:47:34 +00:00
|
|
|
{
|
2018-03-23 17:21:20 +00:00
|
|
|
return map_sha1_file_1(r, NULL, sha1, size);
|
2017-02-21 23:47:34 +00:00
|
|
|
}
|
|
|
|
|
unpack_sha1_header(): detect malformed object header
When opening a loose object file, we often do this sequence:
- prepare a short buffer for the object header (on stack)
- call unpack_sha1_header() and have early part of the object data
inflated, enough to fill the buffer
- parse that data in the short buffer, assuming that the first part
of the object is <typename> SP <length> NUL
Because the parsing function parse_sha1_header_extended() is not
given the number of bytes inflated into the header buffer, it you
craft a file whose early part inflates a garbage sequence without SP
or NUL, and replace a loose object with it, it will end up reading
past the end of the inflated data.
To correct this, do the following four things:
- rename unpack_sha1_header() to unpack_sha1_short_header() and
have unpack_sha1_header_to_strbuf() keep calling that as its
helper function. This will detect and report zlib errors, but is
not aware of the format of a loose object (as before).
- introduce unpack_sha1_header() that calls the same helper
function, and when zlib reports it inflated OK into the buffer,
check if the inflated data has NUL. This would ensure that
parsing function will terminate within the buffer that holds the
inflated header.
- update unpack_sha1_header_to_strbuf() to check if the resulting
buffer has NUL for the same effect.
- update parse_sha1_header_extended() to make sure that its loop to
find the SP that terminates the <typename> stops at NUL.
Essentially, this makes unpack_*() functions that are asked to
unpack a loose object header to be a bit more strict and detect an
input that cannot possibly be a valid object header, even before the
parsing function kicks in.
Reported-by: Gustavo Grieco <gustavo.grieco@imag.fr>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-09-26 04:29:04 +00:00
|
|
|
static int unpack_sha1_short_header(git_zstream *stream,
|
|
|
|
unsigned char *map, unsigned long mapsize,
|
|
|
|
void *buffer, unsigned long bufsiz)
|
2016-02-25 14:22:52 +00:00
|
|
|
{
|
2005-06-02 00:54:59 +00:00
|
|
|
/* Get the data stream */
|
|
|
|
memset(stream, 0, sizeof(*stream));
|
|
|
|
stream->next_in = map;
|
|
|
|
stream->avail_in = mapsize;
|
|
|
|
stream->next_out = buffer;
|
2006-07-11 19:48:08 +00:00
|
|
|
stream->avail_out = bufsiz;
|
2016-02-25 14:22:52 +00:00
|
|
|
|
2009-01-08 03:54:47 +00:00
|
|
|
git_inflate_init(stream);
|
2011-06-08 18:29:01 +00:00
|
|
|
return git_inflate(stream, 0);
|
2005-06-28 21:56:57 +00:00
|
|
|
}
|
|
|
|
|
unpack_sha1_header(): detect malformed object header
When opening a loose object file, we often do this sequence:
- prepare a short buffer for the object header (on stack)
- call unpack_sha1_header() and have early part of the object data
inflated, enough to fill the buffer
- parse that data in the short buffer, assuming that the first part
of the object is <typename> SP <length> NUL
Because the parsing function parse_sha1_header_extended() is not
given the number of bytes inflated into the header buffer, it you
craft a file whose early part inflates a garbage sequence without SP
or NUL, and replace a loose object with it, it will end up reading
past the end of the inflated data.
To correct this, do the following four things:
- rename unpack_sha1_header() to unpack_sha1_short_header() and
have unpack_sha1_header_to_strbuf() keep calling that as its
helper function. This will detect and report zlib errors, but is
not aware of the format of a loose object (as before).
- introduce unpack_sha1_header() that calls the same helper
function, and when zlib reports it inflated OK into the buffer,
check if the inflated data has NUL. This would ensure that
parsing function will terminate within the buffer that holds the
inflated header.
- update unpack_sha1_header_to_strbuf() to check if the resulting
buffer has NUL for the same effect.
- update parse_sha1_header_extended() to make sure that its loop to
find the SP that terminates the <typename> stops at NUL.
Essentially, this makes unpack_*() functions that are asked to
unpack a loose object header to be a bit more strict and detect an
input that cannot possibly be a valid object header, even before the
parsing function kicks in.
Reported-by: Gustavo Grieco <gustavo.grieco@imag.fr>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-09-26 04:29:04 +00:00
|
|
|
int unpack_sha1_header(git_zstream *stream,
|
|
|
|
unsigned char *map, unsigned long mapsize,
|
|
|
|
void *buffer, unsigned long bufsiz)
|
2005-06-27 10:35:33 +00:00
|
|
|
{
|
unpack_sha1_header(): detect malformed object header
When opening a loose object file, we often do this sequence:
- prepare a short buffer for the object header (on stack)
- call unpack_sha1_header() and have early part of the object data
inflated, enough to fill the buffer
- parse that data in the short buffer, assuming that the first part
of the object is <typename> SP <length> NUL
Because the parsing function parse_sha1_header_extended() is not
given the number of bytes inflated into the header buffer, it you
craft a file whose early part inflates a garbage sequence without SP
or NUL, and replace a loose object with it, it will end up reading
past the end of the inflated data.
To correct this, do the following four things:
- rename unpack_sha1_header() to unpack_sha1_short_header() and
have unpack_sha1_header_to_strbuf() keep calling that as its
helper function. This will detect and report zlib errors, but is
not aware of the format of a loose object (as before).
- introduce unpack_sha1_header() that calls the same helper
function, and when zlib reports it inflated OK into the buffer,
check if the inflated data has NUL. This would ensure that
parsing function will terminate within the buffer that holds the
inflated header.
- update unpack_sha1_header_to_strbuf() to check if the resulting
buffer has NUL for the same effect.
- update parse_sha1_header_extended() to make sure that its loop to
find the SP that terminates the <typename> stops at NUL.
Essentially, this makes unpack_*() functions that are asked to
unpack a loose object header to be a bit more strict and detect an
input that cannot possibly be a valid object header, even before the
parsing function kicks in.
Reported-by: Gustavo Grieco <gustavo.grieco@imag.fr>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-09-26 04:29:04 +00:00
|
|
|
int status = unpack_sha1_short_header(stream, map, mapsize,
|
|
|
|
buffer, bufsiz);
|
2007-03-16 20:42:50 +00:00
|
|
|
|
unpack_sha1_header(): detect malformed object header
When opening a loose object file, we often do this sequence:
- prepare a short buffer for the object header (on stack)
- call unpack_sha1_header() and have early part of the object data
inflated, enough to fill the buffer
- parse that data in the short buffer, assuming that the first part
of the object is <typename> SP <length> NUL
Because the parsing function parse_sha1_header_extended() is not
given the number of bytes inflated into the header buffer, it you
craft a file whose early part inflates a garbage sequence without SP
or NUL, and replace a loose object with it, it will end up reading
past the end of the inflated data.
To correct this, do the following four things:
- rename unpack_sha1_header() to unpack_sha1_short_header() and
have unpack_sha1_header_to_strbuf() keep calling that as its
helper function. This will detect and report zlib errors, but is
not aware of the format of a loose object (as before).
- introduce unpack_sha1_header() that calls the same helper
function, and when zlib reports it inflated OK into the buffer,
check if the inflated data has NUL. This would ensure that
parsing function will terminate within the buffer that holds the
inflated header.
- update unpack_sha1_header_to_strbuf() to check if the resulting
buffer has NUL for the same effect.
- update parse_sha1_header_extended() to make sure that its loop to
find the SP that terminates the <typename> stops at NUL.
Essentially, this makes unpack_*() functions that are asked to
unpack a loose object header to be a bit more strict and detect an
input that cannot possibly be a valid object header, even before the
parsing function kicks in.
Reported-by: Gustavo Grieco <gustavo.grieco@imag.fr>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-09-26 04:29:04 +00:00
|
|
|
if (status < Z_OK)
|
|
|
|
return status;
|
sha1-lookup: more memory efficient search in sorted list of SHA-1
Currently, when looking for a packed object from the pack idx, a
simple binary search is used.
A conventional binary search loop looks like this:
unsigned lo, hi;
do {
unsigned mi = (lo + hi) / 2;
int cmp = "entry pointed at by mi" minus "target";
if (!cmp)
return mi; "mi is the wanted one"
if (cmp > 0)
hi = mi; "mi is larger than target"
else
lo = mi+1; "mi is smaller than target"
} while (lo < hi);
"did not find what we wanted"
The invariants are:
- When entering the loop, 'lo' points at a slot that is never
above the target (it could be at the target), 'hi' points at
a slot that is guaranteed to be above the target (it can
never be at the target).
- We find a point 'mi' between 'lo' and 'hi' ('mi' could be
the same as 'lo', but never can be as high as 'hi'), and
check if 'mi' hits the target. There are three cases:
- if it is a hit, we have found what we are looking for;
- if it is strictly higher than the target, we set it to
'hi', and repeat the search.
- if it is strictly lower than the target, we update 'lo'
to one slot after it, because we allow 'lo' to be at the
target and 'mi' is known to be below the target.
If the loop exits, there is no matching entry.
When choosing 'mi', we do not have to take the "middle" but
anywhere in between 'lo' and 'hi', as long as lo <= mi < hi is
satisfied. When we somehow know that the distance between the
target and 'lo' is much shorter than the target and 'hi', we
could pick 'mi' that is much closer to 'lo' than (hi+lo)/2,
which a conventional binary search would pick.
This patch takes advantage of the fact that the SHA-1 is a good
hash function, and as long as there are enough entries in the
table, we can expect uniform distribution. An entry that begins
with for example "deadbeef..." is much likely to appear much
later than in the midway of a reasonably populated table. In
fact, it can be expected to be near 87% (222/256) from the top
of the table.
This is a work-in-progress and has switches to allow easier
experiments and debugging. Exporting GIT_USE_LOOKUP environment
variable enables this code.
On my admittedly memory starved machine, with a partial KDE
repository (3.0G pack with 95M idx):
$ GIT_USE_LOOKUP=t git log -800 --stat HEAD >/dev/null
3.93user 0.16system 0:04.09elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+55588minor)pagefaults 0swaps
Without the patch, the numbers are:
$ git log -800 --stat HEAD >/dev/null
4.00user 0.15system 0:04.17elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+60258minor)pagefaults 0swaps
In the same repository:
$ GIT_USE_LOOKUP=t git log -2000 HEAD >/dev/null
0.12user 0.00system 0:00.12elapsed 97%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+4241minor)pagefaults 0swaps
Without the patch, the numbers are:
$ git log -2000 HEAD >/dev/null
0.05user 0.01system 0:00.07elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+8506minor)pagefaults 0swaps
There isn't much time difference, but the number of minor faults
seems to show that we are touching much smaller number of pages,
which is expected.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-12-29 10:05:47 +00:00
|
|
|
|
unpack_sha1_header(): detect malformed object header
When opening a loose object file, we often do this sequence:
- prepare a short buffer for the object header (on stack)
- call unpack_sha1_header() and have early part of the object data
inflated, enough to fill the buffer
- parse that data in the short buffer, assuming that the first part
of the object is <typename> SP <length> NUL
Because the parsing function parse_sha1_header_extended() is not
given the number of bytes inflated into the header buffer, it you
craft a file whose early part inflates a garbage sequence without SP
or NUL, and replace a loose object with it, it will end up reading
past the end of the inflated data.
To correct this, do the following four things:
- rename unpack_sha1_header() to unpack_sha1_short_header() and
have unpack_sha1_header_to_strbuf() keep calling that as its
helper function. This will detect and report zlib errors, but is
not aware of the format of a loose object (as before).
- introduce unpack_sha1_header() that calls the same helper
function, and when zlib reports it inflated OK into the buffer,
check if the inflated data has NUL. This would ensure that
parsing function will terminate within the buffer that holds the
inflated header.
- update unpack_sha1_header_to_strbuf() to check if the resulting
buffer has NUL for the same effect.
- update parse_sha1_header_extended() to make sure that its loop to
find the SP that terminates the <typename> stops at NUL.
Essentially, this makes unpack_*() functions that are asked to
unpack a loose object header to be a bit more strict and detect an
input that cannot possibly be a valid object header, even before the
parsing function kicks in.
Reported-by: Gustavo Grieco <gustavo.grieco@imag.fr>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-09-26 04:29:04 +00:00
|
|
|
/* Make sure we have the terminating NUL */
|
|
|
|
if (!memchr(buffer, '\0', stream->next_out - (unsigned char *)buffer))
|
|
|
|
return -1;
|
2005-06-27 10:35:33 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2015-05-03 14:29:59 +00:00
|
|
|
static int unpack_sha1_header_to_strbuf(git_zstream *stream, unsigned char *map,
|
|
|
|
unsigned long mapsize, void *buffer,
|
|
|
|
unsigned long bufsiz, struct strbuf *header)
|
2011-03-02 18:01:54 +00:00
|
|
|
{
|
2015-05-03 14:29:59 +00:00
|
|
|
int status;
|
|
|
|
|
unpack_sha1_header(): detect malformed object header
When opening a loose object file, we often do this sequence:
- prepare a short buffer for the object header (on stack)
- call unpack_sha1_header() and have early part of the object data
inflated, enough to fill the buffer
- parse that data in the short buffer, assuming that the first part
of the object is <typename> SP <length> NUL
Because the parsing function parse_sha1_header_extended() is not
given the number of bytes inflated into the header buffer, it you
craft a file whose early part inflates a garbage sequence without SP
or NUL, and replace a loose object with it, it will end up reading
past the end of the inflated data.
To correct this, do the following four things:
- rename unpack_sha1_header() to unpack_sha1_short_header() and
have unpack_sha1_header_to_strbuf() keep calling that as its
helper function. This will detect and report zlib errors, but is
not aware of the format of a loose object (as before).
- introduce unpack_sha1_header() that calls the same helper
function, and when zlib reports it inflated OK into the buffer,
check if the inflated data has NUL. This would ensure that
parsing function will terminate within the buffer that holds the
inflated header.
- update unpack_sha1_header_to_strbuf() to check if the resulting
buffer has NUL for the same effect.
- update parse_sha1_header_extended() to make sure that its loop to
find the SP that terminates the <typename> stops at NUL.
Essentially, this makes unpack_*() functions that are asked to
unpack a loose object header to be a bit more strict and detect an
input that cannot possibly be a valid object header, even before the
parsing function kicks in.
Reported-by: Gustavo Grieco <gustavo.grieco@imag.fr>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-09-26 04:29:04 +00:00
|
|
|
status = unpack_sha1_short_header(stream, map, mapsize, buffer, bufsiz);
|
|
|
|
if (status < Z_OK)
|
|
|
|
return -1;
|
2011-03-02 18:01:54 +00:00
|
|
|
|
2015-05-03 14:29:59 +00:00
|
|
|
/*
|
|
|
|
* Check if entire header is unpacked in the first iteration.
|
2011-03-02 18:01:54 +00:00
|
|
|
*/
|
2015-05-03 14:29:59 +00:00
|
|
|
if (memchr(buffer, '\0', stream->next_out - (unsigned char *)buffer))
|
|
|
|
return 0;
|
2011-03-02 18:01:54 +00:00
|
|
|
|
2015-05-03 14:29:59 +00:00
|
|
|
/*
|
|
|
|
* buffer[0..bufsiz] was not large enough. Copy the partial
|
|
|
|
* result out to header, and then append the result of further
|
|
|
|
* reading the stream.
|
|
|
|
*/
|
|
|
|
strbuf_add(header, buffer, stream->next_out - (unsigned char *)buffer);
|
|
|
|
stream->next_out = buffer;
|
|
|
|
stream->avail_out = bufsiz;
|
2011-03-02 18:01:54 +00:00
|
|
|
|
2015-05-03 14:29:59 +00:00
|
|
|
do {
|
|
|
|
status = git_inflate(stream, 0);
|
|
|
|
strbuf_add(header, buffer, stream->next_out - (unsigned char *)buffer);
|
|
|
|
if (memchr(buffer, '\0', stream->next_out - (unsigned char *)buffer))
|
|
|
|
return 0;
|
|
|
|
stream->next_out = buffer;
|
|
|
|
stream->avail_out = bufsiz;
|
|
|
|
} while (status != Z_STREAM_END);
|
|
|
|
return -1;
|
2011-03-02 18:01:54 +00:00
|
|
|
}
|
|
|
|
|
2011-06-10 18:52:15 +00:00
|
|
|
static void *unpack_sha1_rest(git_zstream *stream, void *buffer, unsigned long size, const unsigned char *sha1)
|
2012-02-01 13:48:54 +00:00
|
|
|
{
|
2005-06-02 14:57:25 +00:00
|
|
|
int bytes = strlen(buffer) + 1;
|
2010-01-26 18:24:14 +00:00
|
|
|
unsigned char *buf = xmallocz(size);
|
2006-07-11 19:48:08 +00:00
|
|
|
unsigned long n;
|
2007-03-05 08:21:37 +00:00
|
|
|
int status = Z_OK;
|
2012-02-01 13:48:54 +00:00
|
|
|
|
2006-07-11 19:48:08 +00:00
|
|
|
n = stream->total_out - bytes;
|
|
|
|
if (n > size)
|
|
|
|
n = size;
|
|
|
|
memcpy(buf, (char *) buffer + bytes, n);
|
|
|
|
bytes = n;
|
2007-03-20 05:49:53 +00:00
|
|
|
if (bytes <= size) {
|
|
|
|
/*
|
|
|
|
* The above condition must be (bytes <= size), not
|
|
|
|
* (bytes < size). In other words, even though we
|
2011-05-15 19:16:03 +00:00
|
|
|
* expect no more output and set avail_out to zero,
|
2007-03-20 05:49:53 +00:00
|
|
|
* the input zlib stream may have bytes that express
|
|
|
|
* "this concludes the stream", and we *do* want to
|
|
|
|
* eat that input.
|
|
|
|
*
|
|
|
|
* Otherwise we would not be able to test that we
|
|
|
|
* consumed all the input to reach the expected size;
|
|
|
|
* we also want to check that zlib tells us that all
|
|
|
|
* went well with status == Z_STREAM_END at the end.
|
|
|
|
*/
|
2005-06-02 14:57:25 +00:00
|
|
|
stream->next_out = buf + bytes;
|
|
|
|
stream->avail_out = size - bytes;
|
2007-03-05 08:21:37 +00:00
|
|
|
while (status == Z_OK)
|
2009-01-08 03:54:47 +00:00
|
|
|
status = git_inflate(stream, Z_FINISH);
|
2005-06-02 14:57:25 +00:00
|
|
|
}
|
2007-03-20 05:49:53 +00:00
|
|
|
if (status == Z_STREAM_END && !stream->avail_in) {
|
2009-01-08 03:54:47 +00:00
|
|
|
git_inflate_end(stream);
|
2007-03-05 08:21:37 +00:00
|
|
|
return buf;
|
2012-02-01 13:48:54 +00:00
|
|
|
}
|
|
|
|
|
2007-03-05 08:21:37 +00:00
|
|
|
if (status < 0)
|
2018-07-21 07:49:39 +00:00
|
|
|
error(_("corrupt loose object '%s'"), sha1_to_hex(sha1));
|
2007-03-05 08:21:37 +00:00
|
|
|
else if (stream->avail_in)
|
2018-07-21 07:49:39 +00:00
|
|
|
error(_("garbage at end of loose object '%s'"),
|
2007-03-05 08:21:37 +00:00
|
|
|
sha1_to_hex(sha1));
|
|
|
|
free(buf);
|
|
|
|
return NULL;
|
2012-02-01 13:48:54 +00:00
|
|
|
}
|
|
|
|
|
2014-02-21 16:32:06 +00:00
|
|
|
/*
|
2005-06-02 14:57:25 +00:00
|
|
|
* We used to just use "sscanf()", but that's actually way
|
|
|
|
* too permissive for what we want to check. So do an anal
|
|
|
|
* object header parse by hand.
|
2014-02-21 16:32:06 +00:00
|
|
|
*/
|
2015-05-03 14:29:59 +00:00
|
|
|
static int parse_sha1_header_extended(const char *hdr, struct object_info *oi,
|
|
|
|
unsigned int flags)
|
2005-06-27 10:35:33 +00:00
|
|
|
{
|
2015-05-03 14:29:59 +00:00
|
|
|
const char *type_buf = hdr;
|
2005-06-02 14:57:25 +00:00
|
|
|
unsigned long size;
|
2015-05-03 14:29:59 +00:00
|
|
|
int type, type_len = 0;
|
2006-09-21 04:05:37 +00:00
|
|
|
|
2005-06-02 14:57:25 +00:00
|
|
|
/*
|
2015-05-03 14:29:59 +00:00
|
|
|
* The type can be of any size but is followed by
|
2007-02-26 19:55:59 +00:00
|
|
|
* a space.
|
2005-06-02 14:57:25 +00:00
|
|
|
*/
|
|
|
|
for (;;) {
|
|
|
|
char c = *hdr++;
|
unpack_sha1_header(): detect malformed object header
When opening a loose object file, we often do this sequence:
- prepare a short buffer for the object header (on stack)
- call unpack_sha1_header() and have early part of the object data
inflated, enough to fill the buffer
- parse that data in the short buffer, assuming that the first part
of the object is <typename> SP <length> NUL
Because the parsing function parse_sha1_header_extended() is not
given the number of bytes inflated into the header buffer, it you
craft a file whose early part inflates a garbage sequence without SP
or NUL, and replace a loose object with it, it will end up reading
past the end of the inflated data.
To correct this, do the following four things:
- rename unpack_sha1_header() to unpack_sha1_short_header() and
have unpack_sha1_header_to_strbuf() keep calling that as its
helper function. This will detect and report zlib errors, but is
not aware of the format of a loose object (as before).
- introduce unpack_sha1_header() that calls the same helper
function, and when zlib reports it inflated OK into the buffer,
check if the inflated data has NUL. This would ensure that
parsing function will terminate within the buffer that holds the
inflated header.
- update unpack_sha1_header_to_strbuf() to check if the resulting
buffer has NUL for the same effect.
- update parse_sha1_header_extended() to make sure that its loop to
find the SP that terminates the <typename> stops at NUL.
Essentially, this makes unpack_*() functions that are asked to
unpack a loose object header to be a bit more strict and detect an
input that cannot possibly be a valid object header, even before the
parsing function kicks in.
Reported-by: Gustavo Grieco <gustavo.grieco@imag.fr>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-09-26 04:29:04 +00:00
|
|
|
if (!c)
|
|
|
|
return -1;
|
2005-06-02 14:57:25 +00:00
|
|
|
if (c == ' ')
|
|
|
|
break;
|
2015-05-03 14:29:59 +00:00
|
|
|
type_len++;
|
2005-06-02 14:57:25 +00:00
|
|
|
}
|
2005-06-27 10:35:33 +00:00
|
|
|
|
2015-05-03 14:29:59 +00:00
|
|
|
type = type_from_string_gently(type_buf, type_len, 1);
|
2018-02-14 18:59:23 +00:00
|
|
|
if (oi->type_name)
|
|
|
|
strbuf_add(oi->type_name, type_buf, type_len);
|
2015-05-03 14:29:59 +00:00
|
|
|
/*
|
|
|
|
* Set type to 0 if its an unknown object and
|
2016-08-09 08:53:38 +00:00
|
|
|
* we're obtaining the type using '--allow-unknown-type'
|
2015-05-03 14:29:59 +00:00
|
|
|
* option.
|
|
|
|
*/
|
2017-06-22 00:40:18 +00:00
|
|
|
if ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE) && (type < 0))
|
2015-05-03 14:29:59 +00:00
|
|
|
type = 0;
|
|
|
|
else if (type < 0)
|
2018-07-21 07:49:39 +00:00
|
|
|
die(_("invalid object type"));
|
2015-05-03 14:29:59 +00:00
|
|
|
if (oi->typep)
|
|
|
|
*oi->typep = type;
|
2005-06-02 14:57:25 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* The length must follow immediately, and be in canonical
|
|
|
|
* decimal format (ie "010" is not valid).
|
|
|
|
*/
|
|
|
|
size = *hdr++ - '0';
|
|
|
|
if (size > 9)
|
|
|
|
return -1;
|
|
|
|
if (size) {
|
|
|
|
for (;;) {
|
|
|
|
unsigned long c = *hdr - '0';
|
|
|
|
if (c > 9)
|
|
|
|
break;
|
|
|
|
hdr++;
|
|
|
|
size = size * 10 + c;
|
2014-02-21 16:32:04 +00:00
|
|
|
}
|
2012-02-01 13:48:55 +00:00
|
|
|
}
|
2015-05-03 14:29:59 +00:00
|
|
|
|
|
|
|
if (oi->sizep)
|
|
|
|
*oi->sizep = size;
|
2005-06-02 14:57:25 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* The length must be followed by a zero byte
|
|
|
|
*/
|
2015-05-03 14:29:59 +00:00
|
|
|
return *hdr ? -1 : type;
|
2005-06-27 10:35:33 +00:00
|
|
|
}
|
|
|
|
|
2015-05-03 14:29:59 +00:00
|
|
|
int parse_sha1_header(const char *hdr, unsigned long *sizep)
|
2005-08-01 00:53:44 +00:00
|
|
|
{
|
provide an initializer for "struct object_info"
An all-zero initializer is fine for this struct, but because
the first element is a pointer, call sites need to know to
use "NULL" instead of "0". Otherwise some static checkers
like "sparse" will complain; see d099b71 (Fix some sparse
warnings, 2013-07-18) for example. So let's provide an
initializer to make this easier to get right.
But let's also comment that memset() to zero is explicitly
OK[1]. One of the callers embeds object_info in another
struct which is initialized via memset (expand_data in
builtin/cat-file.c). Since our subset of C doesn't allow
assignment from a compound literal, handling this in any
other way is awkward, so we'd like to keep the ability to
initialize by memset(). By documenting this property, it
should make anybody who wants to change the initializer
think twice before doing so.
There's one other caller of interest. In parse_sha1_header(),
we did not initialize the struct fully in the first place.
This turned out not to be a bug because the sub-function it
calls does not look at any other fields except the ones we
did initialize. But that assumption might not hold in the
future, so it's a dangerous construct. This patch switches
it to initializing the whole struct, which protects us
against unexpected reads of the other fields.
[1] Obviously using memset() to initialize a pointer
violates the C standard, but we long ago decided that it
was an acceptable tradeoff in the real world.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-11 09:24:35 +00:00
|
|
|
struct object_info oi = OBJECT_INFO_INIT;
|
2007-02-26 19:55:59 +00:00
|
|
|
|
2015-05-03 14:29:59 +00:00
|
|
|
oi.sizep = sizep;
|
2017-06-22 00:40:19 +00:00
|
|
|
return parse_sha1_header_extended(hdr, &oi, 0);
|
2005-08-01 00:53:44 +00:00
|
|
|
}
|
|
|
|
|
2018-03-23 17:21:21 +00:00
|
|
|
static int sha1_loose_object_info(struct repository *r,
|
|
|
|
const unsigned char *sha1,
|
|
|
|
struct object_info *oi, int flags)
|
2005-06-02 22:20:54 +00:00
|
|
|
{
|
2015-05-03 14:29:59 +00:00
|
|
|
int status = 0;
|
|
|
|
unsigned long mapsize;
|
2005-06-02 22:20:54 +00:00
|
|
|
void *map;
|
2011-06-10 18:52:15 +00:00
|
|
|
git_zstream stream;
|
2018-03-12 02:27:55 +00:00
|
|
|
char hdr[MAX_HEADER_LEN];
|
2015-05-03 14:29:59 +00:00
|
|
|
struct strbuf hdrbuf = STRBUF_INIT;
|
2017-06-22 00:40:21 +00:00
|
|
|
unsigned long size_scratch;
|
2005-06-02 22:20:54 +00:00
|
|
|
|
2013-12-21 14:24:20 +00:00
|
|
|
if (oi->delta_base_sha1)
|
|
|
|
hashclr(oi->delta_base_sha1);
|
|
|
|
|
sha1_loose_object_info: make type lookup optional
Until recently, the only items to request from
sha1_object_info_extended were type and size. This meant
that we always had to open a loose object file to determine
one or the other. But with the addition of the disk_size
query, it's possible that we can fulfill the query without
even opening the object file at all. However, since the
function interface always returns the type, we have no way
of knowing whether the caller cares about it or not.
This patch only modified sha1_loose_object_info to make type
lookup optional using an out-parameter, similar to the way
the size is handled (and the return value is "0" or "-1" for
success or error, respectively).
There should be no functional change yet, though, as
sha1_object_info_extended, the only caller, will always ask
for a type.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-07-12 06:30:48 +00:00
|
|
|
/*
|
|
|
|
* If we don't care about type or size, then we don't
|
2013-11-06 18:00:57 +00:00
|
|
|
* need to look inside the object at all. Note that we
|
|
|
|
* do not optimize out the stat call, even if the
|
|
|
|
* caller doesn't care about the disk-size, since our
|
|
|
|
* return value implicitly indicates whether the
|
|
|
|
* object even exists.
|
sha1_loose_object_info: make type lookup optional
Until recently, the only items to request from
sha1_object_info_extended were type and size. This meant
that we always had to open a loose object file to determine
one or the other. But with the addition of the disk_size
query, it's possible that we can fulfill the query without
even opening the object file at all. However, since the
function interface always returns the type, we have no way
of knowing whether the caller cares about it or not.
This patch only modified sha1_loose_object_info to make type
lookup optional using an out-parameter, similar to the way
the size is handled (and the return value is "0" or "-1" for
success or error, respectively).
There should be no functional change yet, though, as
sha1_object_info_extended, the only caller, will always ask
for a type.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-07-12 06:30:48 +00:00
|
|
|
*/
|
2018-02-14 18:59:23 +00:00
|
|
|
if (!oi->typep && !oi->type_name && !oi->sizep && !oi->contentp) {
|
2017-01-13 17:54:39 +00:00
|
|
|
const char *path;
|
2013-11-06 18:00:57 +00:00
|
|
|
struct stat st;
|
2018-03-23 17:21:21 +00:00
|
|
|
if (stat_sha1_file(r, sha1, &st, &path) < 0)
|
2013-11-06 18:00:57 +00:00
|
|
|
return -1;
|
|
|
|
if (oi->disk_sizep)
|
2013-07-12 06:37:53 +00:00
|
|
|
*oi->disk_sizep = st.st_size;
|
sha1_loose_object_info: make type lookup optional
Until recently, the only items to request from
sha1_object_info_extended were type and size. This meant
that we always had to open a loose object file to determine
one or the other. But with the addition of the disk_size
query, it's possible that we can fulfill the query without
even opening the object file at all. However, since the
function interface always returns the type, we have no way
of knowing whether the caller cares about it or not.
This patch only modified sha1_loose_object_info to make type
lookup optional using an out-parameter, similar to the way
the size is handled (and the return value is "0" or "-1" for
success or error, respectively).
There should be no functional change yet, though, as
sha1_object_info_extended, the only caller, will always ask
for a type.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-07-12 06:30:48 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2018-03-23 17:21:21 +00:00
|
|
|
map = map_sha1_file(r, sha1, &mapsize);
|
2006-11-27 23:18:55 +00:00
|
|
|
if (!map)
|
2013-05-30 20:00:22 +00:00
|
|
|
return -1;
|
2017-06-22 00:40:21 +00:00
|
|
|
|
|
|
|
if (!oi->sizep)
|
|
|
|
oi->sizep = &size_scratch;
|
|
|
|
|
2013-07-12 06:37:53 +00:00
|
|
|
if (oi->disk_sizep)
|
|
|
|
*oi->disk_sizep = mapsize;
|
2017-06-22 00:40:18 +00:00
|
|
|
if ((flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE)) {
|
2015-05-03 14:29:59 +00:00
|
|
|
if (unpack_sha1_header_to_strbuf(&stream, map, mapsize, hdr, sizeof(hdr), &hdrbuf) < 0)
|
2018-07-21 07:49:39 +00:00
|
|
|
status = error(_("unable to unpack %s header with --allow-unknown-type"),
|
2015-05-03 14:29:59 +00:00
|
|
|
sha1_to_hex(sha1));
|
|
|
|
} else if (unpack_sha1_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0)
|
2018-07-21 07:49:39 +00:00
|
|
|
status = error(_("unable to unpack %s header"),
|
2005-06-27 10:34:06 +00:00
|
|
|
sha1_to_hex(sha1));
|
2015-05-03 14:29:59 +00:00
|
|
|
if (status < 0)
|
|
|
|
; /* Do nothing */
|
|
|
|
else if (hdrbuf.len) {
|
|
|
|
if ((status = parse_sha1_header_extended(hdrbuf.buf, oi, flags)) < 0)
|
2018-07-21 07:49:39 +00:00
|
|
|
status = error(_("unable to parse %s header with --allow-unknown-type"),
|
2015-05-03 14:29:59 +00:00
|
|
|
sha1_to_hex(sha1));
|
|
|
|
} else if ((status = parse_sha1_header_extended(hdr, oi, flags)) < 0)
|
2018-07-21 07:49:39 +00:00
|
|
|
status = error(_("unable to parse %s header"), sha1_to_hex(sha1));
|
2017-06-22 00:40:21 +00:00
|
|
|
|
sha1_loose_object_info: handle errors from unpack_sha1_rest
When a caller of sha1_object_info_extended() sets the
"contentp" field in object_info, we call unpack_sha1_rest()
but do not check whether it signaled an error.
This causes two problems:
1. We pass back NULL to the caller via the contentp field,
but the function returns "0" for success. A caller
might reasonably expect after a successful return that
it can access contentp without a NULL check and
segfault.
As it happens, this is impossible to trigger in the
current code. There is exactly one caller which uses
contentp, read_object(). And the only thing it does
after a successful call is to return the content
pointer to its caller, using NULL as a sentinel for
errors. So in effect it converts the success code from
sha1_object_info_extended() back into an error!
But this is still worth addressing avoid problems for
future users of "contentp".
2. Callers of unpack_sha1_rest() are expected to close the
zlib stream themselves on error. Which means that we're
leaking the stream.
The problem in (1) comes from from c84a1f3ed4 (sha1_file:
refactor read_object, 2017-06-21), which added the contentp
field. Before that, we called unpack_sha1_rest() via
unpack_sha1_file(), which directly used the NULL to signal
an error.
But note that the leak in (2) is actually older than that.
The original unpack_sha1_file() directly returned the result
of unpack_sha1_rest() to its caller, when it should have
been closing the zlib stream itself on error.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-10-05 05:59:52 +00:00
|
|
|
if (status >= 0 && oi->contentp) {
|
2017-06-22 00:40:21 +00:00
|
|
|
*oi->contentp = unpack_sha1_rest(&stream, hdr,
|
|
|
|
*oi->sizep, sha1);
|
sha1_loose_object_info: handle errors from unpack_sha1_rest
When a caller of sha1_object_info_extended() sets the
"contentp" field in object_info, we call unpack_sha1_rest()
but do not check whether it signaled an error.
This causes two problems:
1. We pass back NULL to the caller via the contentp field,
but the function returns "0" for success. A caller
might reasonably expect after a successful return that
it can access contentp without a NULL check and
segfault.
As it happens, this is impossible to trigger in the
current code. There is exactly one caller which uses
contentp, read_object(). And the only thing it does
after a successful call is to return the content
pointer to its caller, using NULL as a sentinel for
errors. So in effect it converts the success code from
sha1_object_info_extended() back into an error!
But this is still worth addressing avoid problems for
future users of "contentp".
2. Callers of unpack_sha1_rest() are expected to close the
zlib stream themselves on error. Which means that we're
leaking the stream.
The problem in (1) comes from from c84a1f3ed4 (sha1_file:
refactor read_object, 2017-06-21), which added the contentp
field. Before that, we called unpack_sha1_rest() via
unpack_sha1_file(), which directly used the NULL to signal
an error.
But note that the leak in (2) is actually older than that.
The original unpack_sha1_file() directly returned the result
of unpack_sha1_rest() to its caller, when it should have
been closing the zlib stream itself on error.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-10-05 05:59:52 +00:00
|
|
|
if (!*oi->contentp) {
|
|
|
|
git_inflate_end(&stream);
|
|
|
|
status = -1;
|
|
|
|
}
|
|
|
|
} else
|
2017-06-22 00:40:21 +00:00
|
|
|
git_inflate_end(&stream);
|
|
|
|
|
2005-06-02 22:20:54 +00:00
|
|
|
munmap(map, mapsize);
|
2015-05-03 14:29:59 +00:00
|
|
|
if (status && oi->typep)
|
2013-07-12 06:37:53 +00:00
|
|
|
*oi->typep = status;
|
2017-06-22 00:40:21 +00:00
|
|
|
if (oi->sizep == &size_scratch)
|
|
|
|
oi->sizep = NULL;
|
2015-05-03 14:29:59 +00:00
|
|
|
strbuf_release(&hdrbuf);
|
2017-08-11 20:36:14 +00:00
|
|
|
oi->whence = OI_LOOSE;
|
sha1_loose_object_info: return error for corrupted objects
When sha1_loose_object_info() finds that a loose object file
cannot be stat(2)ed or mmap(2)ed, it returns -1 to signal an
error to the caller. However, if it found that the loose
object file is corrupt and the object data cannot be used
from it, it stuffs OBJ_BAD into "type" field of the
object_info, but returns zero (i.e., success), which can
confuse callers.
This is due to 052fe5eac (sha1_loose_object_info: make type
lookup optional, 2013-07-12), which switched the return to a
strict success/error, rather than returning the type (but
botched the return).
Callers of regular sha1_object_info() don't notice the
difference, as that function returns the type (which is
OBJ_BAD in this case). However, direct callers of
sha1_object_info_extended() see the function return success,
but without setting any meaningful values in the object_info
struct, leading them to access potentially uninitialized
memory.
The easiest way to see the bug is via "cat-file -s", which
will happily ignore the corruption and report whatever
value happened to be in the "size" variable.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-04-01 08:05:21 +00:00
|
|
|
return (status < 0) ? status : 0;
|
2005-06-02 22:20:54 +00:00
|
|
|
}
|
|
|
|
|
2017-12-08 15:27:14 +00:00
|
|
|
int fetch_if_missing = 1;
|
|
|
|
|
2018-04-25 18:21:06 +00:00
|
|
|
int oid_object_info_extended(struct repository *r, const struct object_id *oid,
|
|
|
|
struct object_info *oi, unsigned flags)
|
2006-11-27 23:18:55 +00:00
|
|
|
{
|
2017-06-22 00:40:23 +00:00
|
|
|
static struct object_info blank_oi = OBJECT_INFO_INIT;
|
2006-11-27 23:18:55 +00:00
|
|
|
struct pack_entry e;
|
2013-07-12 06:34:57 +00:00
|
|
|
int rtype;
|
2018-03-12 02:27:54 +00:00
|
|
|
const struct object_id *real = oid;
|
2017-12-08 15:27:14 +00:00
|
|
|
int already_retried = 0;
|
2006-11-27 23:18:55 +00:00
|
|
|
|
2018-03-12 02:27:54 +00:00
|
|
|
if (flags & OBJECT_INFO_LOOKUP_REPLACE)
|
2018-04-25 18:21:06 +00:00
|
|
|
real = lookup_replace_object(r, oid);
|
2006-11-27 23:18:55 +00:00
|
|
|
|
2018-03-12 02:27:54 +00:00
|
|
|
if (is_null_oid(real))
|
sha1_file: fast-path null sha1 as a missing object
In theory nobody should ever ask the low-level object code
for a null sha1. It's used as a sentinel for "no such
object" in lots of places, so leaking through to this level
is a sign that the higher-level code is not being careful
about its error-checking. In practice, though, quite a few
code paths seem to rely on the null sha1 lookup failing as a
way to quietly propagate non-existence (e.g., by feeding it
to lookup_commit_reference_gently(), which then returns
NULL).
When this happens, we do two inefficient things:
1. We actually search for the null sha1 in packs and in
the loose object directory.
2. When we fail to find it, we re-scan the pack directory
in case a simultaneous repack happened to move it from
loose to packed. This can be very expensive if you have
a large number of packs.
Only the second one actually causes noticeable performance
problems, so we could treat them independently. But for the
sake of simplicity (both of code and of reasoning about it),
it makes sense to just declare that the null sha1 cannot be
a real on-disk object, and looking it up will always return
"no such object".
There's no real loss of functionality to do so Its use as a
sentinel value means that anybody who is unlucky enough to
hit the 2^-160th chance of generating an object with that
sha1 is already going to find the object largely unusable.
In an ideal world, we'd simply fix all of the callers to
notice the null sha1 and avoid passing it to us. But a
simple experiment to catch this with a BUG() shows that
there are a large number of code paths that do so.
So in the meantime, let's fix the performance problem by
taking a fast exit from the object lookup when we see a null
sha1. p5551 shows off the improvement (when a fetched ref is
new, the "old" sha1 is 0{40}, which ends up being passed for
fast-forward checks, the status table abbreviations, etc):
Test HEAD^ HEAD
--------------------------------------------------------
5551.4: fetch 5.51(5.03+0.48) 0.17(0.10+0.06) -96.9%
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-11-21 23:17:39 +00:00
|
|
|
return -1;
|
|
|
|
|
2017-06-22 00:40:23 +00:00
|
|
|
if (!oi)
|
|
|
|
oi = &blank_oi;
|
|
|
|
|
2017-06-22 00:40:22 +00:00
|
|
|
if (!(flags & OBJECT_INFO_SKIP_CACHED)) {
|
2018-05-02 00:26:03 +00:00
|
|
|
struct cached_object *co = find_cached_object(real);
|
2017-06-22 00:40:22 +00:00
|
|
|
if (co) {
|
|
|
|
if (oi->typep)
|
|
|
|
*(oi->typep) = co->type;
|
|
|
|
if (oi->sizep)
|
|
|
|
*(oi->sizep) = co->size;
|
|
|
|
if (oi->disk_sizep)
|
|
|
|
*(oi->disk_sizep) = 0;
|
|
|
|
if (oi->delta_base_sha1)
|
|
|
|
hashclr(oi->delta_base_sha1);
|
2018-02-14 18:59:23 +00:00
|
|
|
if (oi->type_name)
|
2018-02-14 18:59:24 +00:00
|
|
|
strbuf_addstr(oi->type_name, type_name(co->type));
|
2017-06-22 00:40:22 +00:00
|
|
|
if (oi->contentp)
|
|
|
|
*oi->contentp = xmemdupz(co->buf, co->size);
|
|
|
|
oi->whence = OI_CACHED;
|
|
|
|
return 0;
|
|
|
|
}
|
2011-02-05 14:03:02 +00:00
|
|
|
}
|
|
|
|
|
2017-12-08 15:27:14 +00:00
|
|
|
while (1) {
|
2018-05-30 05:04:10 +00:00
|
|
|
if (find_pack_entry(r, real, &e))
|
2017-12-08 15:27:14 +00:00
|
|
|
break;
|
|
|
|
|
2018-03-14 06:32:42 +00:00
|
|
|
if (flags & OBJECT_INFO_IGNORE_LOOSE)
|
|
|
|
return -1;
|
|
|
|
|
2008-08-05 20:08:41 +00:00
|
|
|
/* Most likely it's a loose object. */
|
2018-04-25 18:21:06 +00:00
|
|
|
if (!sha1_loose_object_info(r, real->hash, oi, flags))
|
2013-07-12 06:34:57 +00:00
|
|
|
return 0;
|
2008-08-05 20:08:41 +00:00
|
|
|
|
|
|
|
/* Not a loose object; someone else may have just packed it. */
|
2018-03-13 15:30:29 +00:00
|
|
|
if (!(flags & OBJECT_INFO_QUICK)) {
|
2018-04-25 18:21:06 +00:00
|
|
|
reprepare_packed_git(r);
|
2018-05-30 05:04:10 +00:00
|
|
|
if (find_pack_entry(r, real, &e))
|
2018-03-13 15:30:29 +00:00
|
|
|
break;
|
|
|
|
}
|
2017-12-08 15:27:14 +00:00
|
|
|
|
|
|
|
/* Check if it is a missing object */
|
|
|
|
if (fetch_if_missing && repository_format_partial_clone &&
|
2018-04-25 18:21:06 +00:00
|
|
|
!already_retried && r == the_repository) {
|
2017-12-08 15:27:14 +00:00
|
|
|
/*
|
2018-04-25 18:21:06 +00:00
|
|
|
* TODO Investigate having fetch_object() return
|
2017-12-08 15:27:14 +00:00
|
|
|
* TODO error/success and stopping the music here.
|
2018-04-25 18:21:06 +00:00
|
|
|
* TODO Pass a repository struct through fetch_object,
|
|
|
|
* such that arbitrary repositories work.
|
2017-12-08 15:27:14 +00:00
|
|
|
*/
|
2018-09-12 15:47:37 +00:00
|
|
|
fetch_objects(repository_format_partial_clone, real, 1);
|
2017-12-08 15:27:14 +00:00
|
|
|
already_retried = 1;
|
|
|
|
continue;
|
2017-06-22 00:40:22 +00:00
|
|
|
}
|
2017-12-08 15:27:14 +00:00
|
|
|
|
|
|
|
return -1;
|
2006-11-27 23:18:55 +00:00
|
|
|
}
|
2008-10-29 23:02:47 +00:00
|
|
|
|
2017-06-22 00:40:23 +00:00
|
|
|
if (oi == &blank_oi)
|
|
|
|
/*
|
|
|
|
* We know that the caller doesn't actually need the
|
|
|
|
* information below, so return early.
|
|
|
|
*/
|
|
|
|
return 0;
|
2018-04-25 18:21:06 +00:00
|
|
|
rtype = packed_object_info(r, e.p, e.offset, oi);
|
2013-07-12 06:32:25 +00:00
|
|
|
if (rtype < 0) {
|
2018-03-12 02:27:54 +00:00
|
|
|
mark_bad_packed_object(e.p, real->hash);
|
2018-04-25 18:21:06 +00:00
|
|
|
return oid_object_info_extended(r, real, oi, 0);
|
2017-08-11 20:36:14 +00:00
|
|
|
} else if (oi->whence == OI_PACKED) {
|
2011-05-12 22:51:38 +00:00
|
|
|
oi->u.packed.offset = e.offset;
|
|
|
|
oi->u.packed.pack = e.p;
|
|
|
|
oi->u.packed.is_delta = (rtype == OBJ_REF_DELTA ||
|
|
|
|
rtype == OBJ_OFS_DELTA);
|
2008-10-29 23:02:47 +00:00
|
|
|
}
|
|
|
|
|
2013-07-12 06:34:57 +00:00
|
|
|
return 0;
|
2006-11-27 23:18:55 +00:00
|
|
|
}
|
|
|
|
|
2013-10-26 22:34:30 +00:00
|
|
|
/* returns enum object_type or negative */
|
2018-04-25 18:21:06 +00:00
|
|
|
int oid_object_info(struct repository *r,
|
|
|
|
const struct object_id *oid,
|
|
|
|
unsigned long *sizep)
|
2011-05-12 22:51:38 +00:00
|
|
|
{
|
2013-07-12 06:34:57 +00:00
|
|
|
enum object_type type;
|
provide an initializer for "struct object_info"
An all-zero initializer is fine for this struct, but because
the first element is a pointer, call sites need to know to
use "NULL" instead of "0". Otherwise some static checkers
like "sparse" will complain; see d099b71 (Fix some sparse
warnings, 2013-07-18) for example. So let's provide an
initializer to make this easier to get right.
But let's also comment that memset() to zero is explicitly
OK[1]. One of the callers embeds object_info in another
struct which is initialized via memset (expand_data in
builtin/cat-file.c). Since our subset of C doesn't allow
assignment from a compound literal, handling this in any
other way is awkward, so we'd like to keep the ability to
initialize by memset(). By documenting this property, it
should make anybody who wants to change the initializer
think twice before doing so.
There's one other caller of interest. In parse_sha1_header(),
we did not initialize the struct fully in the first place.
This turned out not to be a bug because the sub-function it
calls does not look at any other fields except the ones we
did initialize. But that assumption might not hold in the
future, so it's a dangerous construct. This patch switches
it to initializing the whole struct, which protects us
against unexpected reads of the other fields.
[1] Obviously using memset() to initialize a pointer
violates the C standard, but we long ago decided that it
was an acceptable tradeoff in the real world.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-11 09:24:35 +00:00
|
|
|
struct object_info oi = OBJECT_INFO_INIT;
|
2011-05-12 22:51:38 +00:00
|
|
|
|
2013-07-12 06:34:57 +00:00
|
|
|
oi.typep = &type;
|
2011-05-12 22:51:38 +00:00
|
|
|
oi.sizep = sizep;
|
2018-04-25 18:21:06 +00:00
|
|
|
if (oid_object_info_extended(r, oid, &oi,
|
|
|
|
OBJECT_INFO_LOOKUP_REPLACE) < 0)
|
2013-07-12 06:34:57 +00:00
|
|
|
return -1;
|
|
|
|
return type;
|
2011-05-12 22:51:38 +00:00
|
|
|
}
|
|
|
|
|
2017-08-18 22:20:30 +00:00
|
|
|
static void *read_object(const unsigned char *sha1, enum object_type *type,
|
|
|
|
unsigned long *size)
|
|
|
|
{
|
2018-03-12 02:27:45 +00:00
|
|
|
struct object_id oid;
|
2017-08-18 22:20:30 +00:00
|
|
|
struct object_info oi = OBJECT_INFO_INIT;
|
|
|
|
void *content;
|
|
|
|
oi.typep = type;
|
|
|
|
oi.sizep = size;
|
|
|
|
oi.contentp = &content;
|
|
|
|
|
2018-03-12 02:27:45 +00:00
|
|
|
hashcpy(oid.hash, sha1);
|
|
|
|
|
2018-04-25 18:20:58 +00:00
|
|
|
if (oid_object_info_extended(the_repository, &oid, &oi, 0) < 0)
|
2017-08-18 22:20:30 +00:00
|
|
|
return NULL;
|
|
|
|
return content;
|
|
|
|
}
|
|
|
|
|
2018-01-28 00:13:11 +00:00
|
|
|
int pretend_object_file(void *buf, unsigned long len, enum object_type type,
|
|
|
|
struct object_id *oid)
|
2007-02-05 05:42:38 +00:00
|
|
|
{
|
|
|
|
struct cached_object *co;
|
|
|
|
|
2018-03-06 22:54:07 +00:00
|
|
|
hash_object_file(buf, len, type_name(type), oid);
|
2018-05-02 00:26:03 +00:00
|
|
|
if (has_sha1_file(oid->hash) || find_cached_object(oid))
|
2007-02-05 05:42:38 +00:00
|
|
|
return 0;
|
2014-03-03 22:32:02 +00:00
|
|
|
ALLOC_GROW(cached_objects, cached_object_nr + 1, cached_object_alloc);
|
2007-02-05 05:42:38 +00:00
|
|
|
co = &cached_objects[cached_object_nr++];
|
|
|
|
co->size = len;
|
2007-02-26 19:55:59 +00:00
|
|
|
co->type = type;
|
2007-02-16 01:02:06 +00:00
|
|
|
co->buf = xmalloc(len);
|
|
|
|
memcpy(co->buf, buf, len);
|
2018-05-02 00:26:03 +00:00
|
|
|
oidcpy(&co->oid, oid);
|
2007-02-05 05:42:38 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2010-10-28 18:13:06 +00:00
|
|
|
/*
|
|
|
|
* This function dies on corrupt objects; the callers who want to
|
|
|
|
* deal with them should arrange to call read_object() and give error
|
|
|
|
* messages themselves.
|
|
|
|
*/
|
sha1_file: convert read_sha1_file to struct object_id
Convert read_sha1_file to take a pointer to struct object_id and rename
it read_object_file. Do the same for read_sha1_file_extended.
Convert one use in grep.c to use the new function without any other code
change, since the pointer being passed is a void pointer that is already
initialized with a pointer to struct object_id. Update the declaration
and definitions of the modified functions, and apply the following
semantic patch to convert the remaining callers:
@@
expression E1, E2, E3;
@@
- read_sha1_file(E1.hash, E2, E3)
+ read_object_file(&E1, E2, E3)
@@
expression E1, E2, E3;
@@
- read_sha1_file(E1->hash, E2, E3)
+ read_object_file(E1, E2, E3)
@@
expression E1, E2, E3, E4;
@@
- read_sha1_file_extended(E1.hash, E2, E3, E4)
+ read_object_file_extended(&E1, E2, E3, E4)
@@
expression E1, E2, E3, E4;
@@
- read_sha1_file_extended(E1->hash, E2, E3, E4)
+ read_object_file_extended(E1, E2, E3, E4)
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-03-12 02:27:53 +00:00
|
|
|
void *read_object_file_extended(const struct object_id *oid,
|
|
|
|
enum object_type *type,
|
|
|
|
unsigned long *size,
|
|
|
|
int lookup_replace)
|
2008-07-15 01:46:48 +00:00
|
|
|
{
|
2010-10-28 18:13:06 +00:00
|
|
|
void *data;
|
2010-10-28 18:13:06 +00:00
|
|
|
const struct packed_git *p;
|
2017-01-13 17:54:39 +00:00
|
|
|
const char *path;
|
|
|
|
struct stat st;
|
2018-04-12 00:21:13 +00:00
|
|
|
const struct object_id *repl = lookup_replace ?
|
|
|
|
lookup_replace_object(the_repository, oid) : oid;
|
2010-10-28 18:13:06 +00:00
|
|
|
|
2010-10-28 18:13:06 +00:00
|
|
|
errno = 0;
|
2018-03-12 02:27:54 +00:00
|
|
|
data = read_object(repl->hash, type, size);
|
2011-05-15 19:54:52 +00:00
|
|
|
if (data)
|
2010-10-28 18:13:06 +00:00
|
|
|
return data;
|
2009-01-23 09:06:53 +00:00
|
|
|
|
2011-01-20 20:12:20 +00:00
|
|
|
if (errno && errno != ENOENT)
|
2018-07-21 07:49:39 +00:00
|
|
|
die_errno(_("failed to read object %s"), oid_to_hex(oid));
|
2010-10-28 18:13:06 +00:00
|
|
|
|
2009-01-23 09:06:53 +00:00
|
|
|
/* die if we replaced an object with one that does not exist */
|
2018-03-12 02:27:54 +00:00
|
|
|
if (repl != oid)
|
2018-07-21 07:49:39 +00:00
|
|
|
die(_("replacement %s not found for %s"),
|
2018-03-12 02:27:54 +00:00
|
|
|
oid_to_hex(repl), oid_to_hex(oid));
|
2009-01-23 09:06:53 +00:00
|
|
|
|
2018-04-11 04:09:55 +00:00
|
|
|
if (!stat_sha1_file(the_repository, repl->hash, &st, &path))
|
2018-07-21 07:49:39 +00:00
|
|
|
die(_("loose object %s (stored in %s) is corrupt"),
|
2018-03-12 02:27:54 +00:00
|
|
|
oid_to_hex(repl), path);
|
2009-01-23 09:06:53 +00:00
|
|
|
|
2018-03-12 02:27:54 +00:00
|
|
|
if ((p = has_packed_and_bad(repl->hash)) != NULL)
|
2018-07-21 07:49:39 +00:00
|
|
|
die(_("packed object %s (stored in %s) is corrupt"),
|
2018-03-12 02:27:54 +00:00
|
|
|
oid_to_hex(repl), p->pack_name);
|
2009-01-23 09:07:01 +00:00
|
|
|
|
2010-10-28 18:13:06 +00:00
|
|
|
return NULL;
|
2008-07-15 01:46:48 +00:00
|
|
|
}
|
|
|
|
|
2018-03-12 02:27:52 +00:00
|
|
|
void *read_object_with_reference(const struct object_id *oid,
|
2007-02-26 19:55:59 +00:00
|
|
|
const char *required_type_name,
|
2005-04-28 23:42:27 +00:00
|
|
|
unsigned long *size,
|
2018-03-12 02:27:52 +00:00
|
|
|
struct object_id *actual_oid_return)
|
2005-04-21 01:06:49 +00:00
|
|
|
{
|
2007-02-26 19:55:59 +00:00
|
|
|
enum object_type type, required_type;
|
2005-04-21 01:06:49 +00:00
|
|
|
void *buffer;
|
|
|
|
unsigned long isize;
|
2018-03-12 02:27:52 +00:00
|
|
|
struct object_id actual_oid;
|
2005-04-21 01:06:49 +00:00
|
|
|
|
2007-02-26 19:55:59 +00:00
|
|
|
required_type = type_from_string(required_type_name);
|
2018-03-12 02:27:52 +00:00
|
|
|
oidcpy(&actual_oid, oid);
|
2005-04-28 23:42:27 +00:00
|
|
|
while (1) {
|
|
|
|
int ref_length = -1;
|
|
|
|
const char *ref_type = NULL;
|
2005-04-21 01:06:49 +00:00
|
|
|
|
sha1_file: convert read_sha1_file to struct object_id
Convert read_sha1_file to take a pointer to struct object_id and rename
it read_object_file. Do the same for read_sha1_file_extended.
Convert one use in grep.c to use the new function without any other code
change, since the pointer being passed is a void pointer that is already
initialized with a pointer to struct object_id. Update the declaration
and definitions of the modified functions, and apply the following
semantic patch to convert the remaining callers:
@@
expression E1, E2, E3;
@@
- read_sha1_file(E1.hash, E2, E3)
+ read_object_file(&E1, E2, E3)
@@
expression E1, E2, E3;
@@
- read_sha1_file(E1->hash, E2, E3)
+ read_object_file(E1, E2, E3)
@@
expression E1, E2, E3, E4;
@@
- read_sha1_file_extended(E1.hash, E2, E3, E4)
+ read_object_file_extended(&E1, E2, E3, E4)
@@
expression E1, E2, E3, E4;
@@
- read_sha1_file_extended(E1->hash, E2, E3, E4)
+ read_object_file_extended(E1, E2, E3, E4)
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-03-12 02:27:53 +00:00
|
|
|
buffer = read_object_file(&actual_oid, &type, &isize);
|
2005-04-28 23:42:27 +00:00
|
|
|
if (!buffer)
|
|
|
|
return NULL;
|
2007-02-26 19:55:59 +00:00
|
|
|
if (type == required_type) {
|
2005-04-28 23:42:27 +00:00
|
|
|
*size = isize;
|
2018-03-12 02:27:52 +00:00
|
|
|
if (actual_oid_return)
|
|
|
|
oidcpy(actual_oid_return, &actual_oid);
|
2005-04-28 23:42:27 +00:00
|
|
|
return buffer;
|
|
|
|
}
|
|
|
|
/* Handle references */
|
2007-02-26 19:55:59 +00:00
|
|
|
else if (type == OBJ_COMMIT)
|
2005-04-28 23:42:27 +00:00
|
|
|
ref_type = "tree ";
|
2007-02-26 19:55:59 +00:00
|
|
|
else if (type == OBJ_TAG)
|
2005-04-28 23:42:27 +00:00
|
|
|
ref_type = "object ";
|
|
|
|
else {
|
|
|
|
free(buffer);
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
ref_length = strlen(ref_type);
|
2005-04-21 01:06:49 +00:00
|
|
|
|
2018-07-16 01:28:07 +00:00
|
|
|
if (ref_length + the_hash_algo->hexsz > isize ||
|
2008-02-18 20:47:52 +00:00
|
|
|
memcmp(buffer, ref_type, ref_length) ||
|
2018-03-12 02:27:52 +00:00
|
|
|
get_oid_hex((char *) buffer + ref_length, &actual_oid)) {
|
2005-04-28 23:42:27 +00:00
|
|
|
free(buffer);
|
|
|
|
return NULL;
|
|
|
|
}
|
2005-08-08 18:44:43 +00:00
|
|
|
free(buffer);
|
2005-04-28 23:42:27 +00:00
|
|
|
/* Now we have the ID of the referred-to object in
|
2018-03-12 02:27:52 +00:00
|
|
|
* actual_oid. Check again. */
|
2005-04-21 01:06:49 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2018-01-28 00:13:19 +00:00
|
|
|
static void write_object_file_prepare(const void *buf, unsigned long len,
|
|
|
|
const char *type, struct object_id *oid,
|
|
|
|
char *hdr, int *hdrlen)
|
2005-06-28 02:03:13 +00:00
|
|
|
{
|
2018-02-01 02:18:41 +00:00
|
|
|
git_hash_ctx c;
|
2005-06-28 02:03:13 +00:00
|
|
|
|
|
|
|
/* Generate the header */
|
2015-09-24 21:06:42 +00:00
|
|
|
*hdrlen = xsnprintf(hdr, *hdrlen, "%s %lu", type, len)+1;
|
2005-06-28 02:03:13 +00:00
|
|
|
|
|
|
|
/* Sha1.. */
|
2018-02-01 02:18:41 +00:00
|
|
|
the_hash_algo->init_fn(&c);
|
|
|
|
the_hash_algo->update_fn(&c, hdr, *hdrlen);
|
|
|
|
the_hash_algo->update_fn(&c, buf, len);
|
2018-02-15 22:55:47 +00:00
|
|
|
the_hash_algo->final_fn(oid->hash, &c);
|
2005-06-28 02:03:13 +00:00
|
|
|
}
|
|
|
|
|
Create object subdirectories on demand
This makes it possible to have a "sparse" git object subdirectory
structure, something that has become much more attractive now that people
use pack-files all the time.
As a result of pack-files, a git object directory doesn't necessarily have
any individual objects lying around, and in that case it's just wasting
space to keep the empty first-level object directories around: on many
filesystems the 256 empty directories will be aboue 1MB of diskspace.
Even more importantly, after you re-pack a project that _used_ to be
unpacked, you could be left with huge directories that no longer contain
anything, but that waste space and take time to look through.
With this change, "git prune-packed" can just do an rmdir() on the
directories, and they'll get removed if empty, and re-created on demand.
This patch also tries to fix up "write_sha1_from_fd()" to use the new
common infrastructure for creating the object files, closing a hole where
we might otherwise leave half-written objects in the object database.
[jc: I unoptimized the part that really removes the fan-out directories
to ease transition. init-db still wastes 1MB of diskspace to hold 256
empty fan-outs, and prune-packed rmdir()'s the grown but empty directories,
but runs mkdir() immediately after that -- reducing the saving from 150KB
to 146KB. These parts will be re-introduced when everybody has the
on-demand capability.]
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-10-08 22:54:01 +00:00
|
|
|
/*
|
2009-03-25 23:19:36 +00:00
|
|
|
* Move the just written object into its final resting place.
|
Create object subdirectories on demand
This makes it possible to have a "sparse" git object subdirectory
structure, something that has become much more attractive now that people
use pack-files all the time.
As a result of pack-files, a git object directory doesn't necessarily have
any individual objects lying around, and in that case it's just wasting
space to keep the empty first-level object directories around: on many
filesystems the 256 empty directories will be aboue 1MB of diskspace.
Even more importantly, after you re-pack a project that _used_ to be
unpacked, you could be left with huge directories that no longer contain
anything, but that waste space and take time to look through.
With this change, "git prune-packed" can just do an rmdir() on the
directories, and they'll get removed if empty, and re-created on demand.
This patch also tries to fix up "write_sha1_from_fd()" to use the new
common infrastructure for creating the object files, closing a hole where
we might otherwise leave half-written objects in the object database.
[jc: I unoptimized the part that really removes the fan-out directories
to ease transition. init-db still wastes 1MB of diskspace to hold 256
empty fan-outs, and prune-packed rmdir()'s the grown but empty directories,
but runs mkdir() immediately after that -- reducing the saving from 150KB
to 146KB. These parts will be re-introduced when everybody has the
on-demand capability.]
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-10-08 22:54:01 +00:00
|
|
|
*/
|
2015-08-07 21:40:24 +00:00
|
|
|
int finalize_object_file(const char *tmpfile, const char *filename)
|
Create object subdirectories on demand
This makes it possible to have a "sparse" git object subdirectory
structure, something that has become much more attractive now that people
use pack-files all the time.
As a result of pack-files, a git object directory doesn't necessarily have
any individual objects lying around, and in that case it's just wasting
space to keep the empty first-level object directories around: on many
filesystems the 256 empty directories will be aboue 1MB of diskspace.
Even more importantly, after you re-pack a project that _used_ to be
unpacked, you could be left with huge directories that no longer contain
anything, but that waste space and take time to look through.
With this change, "git prune-packed" can just do an rmdir() on the
directories, and they'll get removed if empty, and re-created on demand.
This patch also tries to fix up "write_sha1_from_fd()" to use the new
common infrastructure for creating the object files, closing a hole where
we might otherwise leave half-written objects in the object database.
[jc: I unoptimized the part that really removes the fan-out directories
to ease transition. init-db still wastes 1MB of diskspace to hold 256
empty fan-outs, and prune-packed rmdir()'s the grown but empty directories,
but runs mkdir() immediately after that -- reducing the saving from 150KB
to 146KB. These parts will be re-introduced when everybody has the
on-demand capability.]
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-10-08 22:54:01 +00:00
|
|
|
{
|
2008-09-18 22:24:46 +00:00
|
|
|
int ret = 0;
|
2009-03-25 23:19:36 +00:00
|
|
|
|
2009-04-27 22:32:25 +00:00
|
|
|
if (object_creation_mode == OBJECT_CREATION_USES_RENAMES)
|
2009-04-25 09:57:14 +00:00
|
|
|
goto try_rename;
|
|
|
|
else if (link(tmpfile, filename))
|
2008-09-18 22:24:46 +00:00
|
|
|
ret = errno;
|
2005-10-26 17:27:36 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Coda hack - coda doesn't like cross-directory links,
|
|
|
|
* so we fall back to a rename, which will mean that it
|
|
|
|
* won't be able to check collisions, but that's not a
|
|
|
|
* big deal.
|
|
|
|
*
|
|
|
|
* The same holds for FAT formatted media.
|
|
|
|
*
|
2009-03-28 06:14:39 +00:00
|
|
|
* When this succeeds, we just return. We have nothing
|
2005-10-26 17:27:36 +00:00
|
|
|
* left to unlink.
|
|
|
|
*/
|
|
|
|
if (ret && ret != EEXIST) {
|
2009-04-25 09:57:14 +00:00
|
|
|
try_rename:
|
2005-10-26 17:27:36 +00:00
|
|
|
if (!rename(tmpfile, filename))
|
2009-03-28 06:14:39 +00:00
|
|
|
goto out;
|
2005-10-25 23:41:20 +00:00
|
|
|
ret = errno;
|
Create object subdirectories on demand
This makes it possible to have a "sparse" git object subdirectory
structure, something that has become much more attractive now that people
use pack-files all the time.
As a result of pack-files, a git object directory doesn't necessarily have
any individual objects lying around, and in that case it's just wasting
space to keep the empty first-level object directories around: on many
filesystems the 256 empty directories will be aboue 1MB of diskspace.
Even more importantly, after you re-pack a project that _used_ to be
unpacked, you could be left with huge directories that no longer contain
anything, but that waste space and take time to look through.
With this change, "git prune-packed" can just do an rmdir() on the
directories, and they'll get removed if empty, and re-created on demand.
This patch also tries to fix up "write_sha1_from_fd()" to use the new
common infrastructure for creating the object files, closing a hole where
we might otherwise leave half-written objects in the object database.
[jc: I unoptimized the part that really removes the fan-out directories
to ease transition. init-db still wastes 1MB of diskspace to hold 256
empty fan-outs, and prune-packed rmdir()'s the grown but empty directories,
but runs mkdir() immediately after that -- reducing the saving from 150KB
to 146KB. These parts will be re-introduced when everybody has the
on-demand capability.]
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-10-08 22:54:01 +00:00
|
|
|
}
|
2009-04-29 21:22:56 +00:00
|
|
|
unlink_or_warn(tmpfile);
|
Create object subdirectories on demand
This makes it possible to have a "sparse" git object subdirectory
structure, something that has become much more attractive now that people
use pack-files all the time.
As a result of pack-files, a git object directory doesn't necessarily have
any individual objects lying around, and in that case it's just wasting
space to keep the empty first-level object directories around: on many
filesystems the 256 empty directories will be aboue 1MB of diskspace.
Even more importantly, after you re-pack a project that _used_ to be
unpacked, you could be left with huge directories that no longer contain
anything, but that waste space and take time to look through.
With this change, "git prune-packed" can just do an rmdir() on the
directories, and they'll get removed if empty, and re-created on demand.
This patch also tries to fix up "write_sha1_from_fd()" to use the new
common infrastructure for creating the object files, closing a hole where
we might otherwise leave half-written objects in the object database.
[jc: I unoptimized the part that really removes the fan-out directories
to ease transition. init-db still wastes 1MB of diskspace to hold 256
empty fan-outs, and prune-packed rmdir()'s the grown but empty directories,
but runs mkdir() immediately after that -- reducing the saving from 150KB
to 146KB. These parts will be re-introduced when everybody has the
on-demand capability.]
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-10-08 22:54:01 +00:00
|
|
|
if (ret) {
|
|
|
|
if (ret != EEXIST) {
|
2018-07-21 07:49:39 +00:00
|
|
|
return error_errno(_("unable to write sha1 filename %s"), filename);
|
Create object subdirectories on demand
This makes it possible to have a "sparse" git object subdirectory
structure, something that has become much more attractive now that people
use pack-files all the time.
As a result of pack-files, a git object directory doesn't necessarily have
any individual objects lying around, and in that case it's just wasting
space to keep the empty first-level object directories around: on many
filesystems the 256 empty directories will be aboue 1MB of diskspace.
Even more importantly, after you re-pack a project that _used_ to be
unpacked, you could be left with huge directories that no longer contain
anything, but that waste space and take time to look through.
With this change, "git prune-packed" can just do an rmdir() on the
directories, and they'll get removed if empty, and re-created on demand.
This patch also tries to fix up "write_sha1_from_fd()" to use the new
common infrastructure for creating the object files, closing a hole where
we might otherwise leave half-written objects in the object database.
[jc: I unoptimized the part that really removes the fan-out directories
to ease transition. init-db still wastes 1MB of diskspace to hold 256
empty fan-outs, and prune-packed rmdir()'s the grown but empty directories,
but runs mkdir() immediately after that -- reducing the saving from 150KB
to 146KB. These parts will be re-introduced when everybody has the
on-demand capability.]
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-10-08 22:54:01 +00:00
|
|
|
}
|
|
|
|
/* FIXME!!! Collision check here ? */
|
|
|
|
}
|
|
|
|
|
2009-03-28 06:14:39 +00:00
|
|
|
out:
|
2010-02-22 22:32:16 +00:00
|
|
|
if (adjust_shared_perm(filename))
|
2018-07-21 07:49:39 +00:00
|
|
|
return error(_("unable to set permission to '%s'"), filename);
|
Create object subdirectories on demand
This makes it possible to have a "sparse" git object subdirectory
structure, something that has become much more attractive now that people
use pack-files all the time.
As a result of pack-files, a git object directory doesn't necessarily have
any individual objects lying around, and in that case it's just wasting
space to keep the empty first-level object directories around: on many
filesystems the 256 empty directories will be aboue 1MB of diskspace.
Even more importantly, after you re-pack a project that _used_ to be
unpacked, you could be left with huge directories that no longer contain
anything, but that waste space and take time to look through.
With this change, "git prune-packed" can just do an rmdir() on the
directories, and they'll get removed if empty, and re-created on demand.
This patch also tries to fix up "write_sha1_from_fd()" to use the new
common infrastructure for creating the object files, closing a hole where
we might otherwise leave half-written objects in the object database.
[jc: I unoptimized the part that really removes the fan-out directories
to ease transition. init-db still wastes 1MB of diskspace to hold 256
empty fan-outs, and prune-packed rmdir()'s the grown but empty directories,
but runs mkdir() immediately after that -- reducing the saving from 150KB
to 146KB. These parts will be re-introduced when everybody has the
on-demand capability.]
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-10-08 22:54:01 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2006-05-24 15:30:54 +00:00
|
|
|
static int write_buffer(int fd, const void *buf, size_t len)
|
|
|
|
{
|
2007-01-12 04:23:00 +00:00
|
|
|
if (write_in_full(fd, buf, len) < 0)
|
2018-07-21 07:49:39 +00:00
|
|
|
return error_errno(_("file write error"));
|
2006-05-24 15:30:54 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2018-01-28 00:13:13 +00:00
|
|
|
int hash_object_file(const void *buf, unsigned long len, const char *type,
|
|
|
|
struct object_id *oid)
|
2006-10-14 10:45:36 +00:00
|
|
|
{
|
2018-03-12 02:27:55 +00:00
|
|
|
char hdr[MAX_HEADER_LEN];
|
2015-09-24 21:06:42 +00:00
|
|
|
int hdrlen = sizeof(hdr);
|
2018-01-28 00:13:19 +00:00
|
|
|
write_object_file_prepare(buf, len, type, oid, hdr, &hdrlen);
|
2006-10-14 10:45:36 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2008-06-11 01:47:18 +00:00
|
|
|
/* Finalize a file on disk, and close it. */
|
|
|
|
static void close_sha1_file(int fd)
|
|
|
|
{
|
2008-06-18 22:18:44 +00:00
|
|
|
if (fsync_object_files)
|
|
|
|
fsync_or_die(fd, "sha1 file");
|
2008-06-11 01:47:18 +00:00
|
|
|
if (close(fd) != 0)
|
2018-07-21 07:49:39 +00:00
|
|
|
die_errno(_("error when closing sha1 file"));
|
2008-06-11 01:47:18 +00:00
|
|
|
}
|
|
|
|
|
2008-06-14 17:50:12 +00:00
|
|
|
/* Size of directory component, including the ending '/' */
|
|
|
|
static inline int directory_size(const char *filename)
|
|
|
|
{
|
|
|
|
const char *s = strrchr(filename, '/');
|
|
|
|
if (!s)
|
|
|
|
return 0;
|
|
|
|
return s - filename + 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This creates a temporary file in the same directory as the final
|
|
|
|
* 'filename'
|
|
|
|
*
|
|
|
|
* We want to avoid cross-directory filename renames, because those
|
|
|
|
* can have problems on various filesystems (FAT, NFS, Coda).
|
|
|
|
*/
|
2015-09-24 21:07:49 +00:00
|
|
|
static int create_tmpfile(struct strbuf *tmp, const char *filename)
|
2008-06-14 17:50:12 +00:00
|
|
|
{
|
|
|
|
int fd, dirlen = directory_size(filename);
|
|
|
|
|
2015-09-24 21:07:49 +00:00
|
|
|
strbuf_reset(tmp);
|
|
|
|
strbuf_add(tmp, filename, dirlen);
|
|
|
|
strbuf_addstr(tmp, "tmp_obj_XXXXXX");
|
|
|
|
fd = git_mkstemp_mode(tmp->buf, 0444);
|
sha1_file: avoid bogus "file exists" error message
This avoids the following misleading error message:
error: unable to create temporary sha1 filename ./objects/15: File exists
mkstemp can fail for many reasons, one of which, ENOENT, can occur if
the directory for the temp file doesn't exist. create_tmpfile tried to
handle this case by always trying to mkdir the directory, even if it
already existed. This caused errno to be clobbered, so one cannot tell
why mkstemp really failed, and it truncated the buffer to just the
directory name, resulting in the strange error message shown above.
Note that in both occasions that I've seen this failure, it has not been
due to a missing directory, or bad permissions, but some other, unknown
mkstemp failure mode that did not occur when I ran git again. This code
could perhaps be made more robust by retrying mkstemp, in case it was a
transient failure.
Signed-off-by: Joey Hess <joey@kitenet.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-11-20 18:56:28 +00:00
|
|
|
if (fd < 0 && dirlen && errno == ENOENT) {
|
2015-09-24 21:07:49 +00:00
|
|
|
/*
|
|
|
|
* Make sure the directory exists; note that the contents
|
|
|
|
* of the buffer are undefined after mkstemp returns an
|
|
|
|
* error, so we have to rewrite the whole buffer from
|
|
|
|
* scratch.
|
|
|
|
*/
|
|
|
|
strbuf_reset(tmp);
|
|
|
|
strbuf_add(tmp, filename, dirlen - 1);
|
|
|
|
if (mkdir(tmp->buf, 0777) && errno != EEXIST)
|
sha1_file.c:create_tmpfile(): Fix race when creating loose object dirs
There are cases (e.g. when running concurrent fetches in a repo) where
multiple Git processes concurrently attempt to create loose objects
within the same objects/XX/ dir. The creation of the loose object files
is (AFAICS) safe from races, but the creation of the objects/XX/ dir in
which the loose objects reside is unsafe, for example:
Two concurrent fetches - A and B. As part of its fetch, A needs to store
12aaaaa as a loose object. B, on the other hand, needs to store 12bbbbb
as a loose object. The objects/12 directory does not already exist.
Concurrently, both A and B determine that they need to create the
objects/12 directory (because their first call to git_mkstemp_mode()
within create_tmpfile() fails witn ENOENT). One of them - let's say A -
executes the following mkdir() call before the other. This first call
returns success, and A moves on. When B gets around to calling mkdir(),
it fails with EEXIST, because A won the race. The mkdir() error causes B
to return -1 from create_tmpfile(), which propagates all the way,
resulting in the fetch failing with:
error: unable to create temporary file: File exists
fatal: failed to write object
fatal: unpack-objects failed
Although it's hard to add a testcase reproducing this issue, it's easy
to provoke if we insert a sleep after the
if (mkdir(buffer, 0777) || adjust_shared_perm(buffer))
return -1;
block, and then run two concurrent "git fetch"es against the same repo.
The fix is to simply handle mkdir() failing with EEXIST as a success.
If EEXIST is somehow returned for the wrong reasons (because the relevant
objects/XX is not a directory, or is otherwise unsuitable for object
storage), the following call to adjust_shared_perm(), or ultimately the
retried call to git_mkstemp_mode() will fail, and we end up returning
error from create_tmpfile() in any case.
Note that there are still cases where two users with unsuitable umasks
in a shared repo can end up in two races where one user first wins the
mkdir() race to create an objects/XX/ directory, and then the other user
wins the adjust_shared_perms() race to chmod() that directory, but fails
because it is (transiently, until the first users completes its chmod())
unwriteable to the other user. However, (an equivalent of) this race also
exists before this patch, and is made no worse by this patch.
Signed-off-by: Johan Herland <johan@herland.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-10-27 11:35:43 +00:00
|
|
|
return -1;
|
2015-09-24 21:07:49 +00:00
|
|
|
if (adjust_shared_perm(tmp->buf))
|
2008-06-14 17:50:12 +00:00
|
|
|
return -1;
|
|
|
|
|
|
|
|
/* Try again */
|
2015-09-24 21:07:49 +00:00
|
|
|
strbuf_addstr(tmp, "/tmp_obj_XXXXXX");
|
|
|
|
fd = git_mkstemp_mode(tmp->buf, 0444);
|
2008-06-14 17:50:12 +00:00
|
|
|
}
|
|
|
|
return fd;
|
|
|
|
}
|
|
|
|
|
2018-01-28 00:13:21 +00:00
|
|
|
static int write_loose_object(const struct object_id *oid, char *hdr,
|
|
|
|
int hdrlen, const void *buf, unsigned long len,
|
|
|
|
time_t mtime)
|
2005-04-18 20:04:43 +00:00
|
|
|
{
|
2009-01-29 05:56:34 +00:00
|
|
|
int fd, ret;
|
2010-02-21 04:27:31 +00:00
|
|
|
unsigned char compressed[4096];
|
2011-06-10 18:52:15 +00:00
|
|
|
git_zstream stream;
|
2018-02-01 02:18:41 +00:00
|
|
|
git_hash_ctx c;
|
2018-01-28 00:13:21 +00:00
|
|
|
struct object_id parano_oid;
|
2015-09-24 21:07:49 +00:00
|
|
|
static struct strbuf tmp_file = STRBUF_INIT;
|
2018-01-17 17:54:54 +00:00
|
|
|
static struct strbuf filename = STRBUF_INIT;
|
|
|
|
|
|
|
|
strbuf_reset(&filename);
|
2018-04-11 04:09:55 +00:00
|
|
|
sha1_file_name(the_repository, &filename, oid->hash);
|
2005-04-25 17:19:53 +00:00
|
|
|
|
2018-01-17 17:54:54 +00:00
|
|
|
fd = create_tmpfile(&tmp_file, filename.buf);
|
2005-05-03 18:46:16 +00:00
|
|
|
if (fd < 0) {
|
2008-11-14 07:19:34 +00:00
|
|
|
if (errno == EACCES)
|
2018-07-21 07:49:39 +00:00
|
|
|
return error(_("insufficient permission for adding an object to repository database %s"), get_object_directory());
|
2006-11-09 12:52:05 +00:00
|
|
|
else
|
2018-07-21 07:49:39 +00:00
|
|
|
return error_errno(_("unable to create temporary file"));
|
2005-05-03 18:46:16 +00:00
|
|
|
}
|
|
|
|
|
2005-04-18 20:04:43 +00:00
|
|
|
/* Set it up */
|
2011-06-10 17:55:10 +00:00
|
|
|
git_deflate_init(&stream, zlib_compression_level);
|
2005-04-18 20:04:43 +00:00
|
|
|
stream.next_out = compressed;
|
2010-02-21 04:27:31 +00:00
|
|
|
stream.avail_out = sizeof(compressed);
|
2018-02-01 02:18:41 +00:00
|
|
|
the_hash_algo->init_fn(&c);
|
2005-04-25 17:19:53 +00:00
|
|
|
|
|
|
|
/* First header.. */
|
2007-02-26 19:55:55 +00:00
|
|
|
stream.next_in = (unsigned char *)hdr;
|
2005-04-25 17:19:53 +00:00
|
|
|
stream.avail_in = hdrlen;
|
2011-06-10 17:55:10 +00:00
|
|
|
while (git_deflate(&stream, 0) == Z_OK)
|
|
|
|
; /* nothing */
|
2018-02-01 02:18:41 +00:00
|
|
|
the_hash_algo->update_fn(&c, hdr, hdrlen);
|
2005-04-25 17:19:53 +00:00
|
|
|
|
|
|
|
/* Then the data itself.. */
|
2010-04-02 00:03:18 +00:00
|
|
|
stream.next_in = (void *)buf;
|
2005-04-25 17:19:53 +00:00
|
|
|
stream.avail_in = len;
|
2010-02-21 04:27:31 +00:00
|
|
|
do {
|
2010-02-21 20:48:06 +00:00
|
|
|
unsigned char *in0 = stream.next_in;
|
2011-06-10 17:55:10 +00:00
|
|
|
ret = git_deflate(&stream, Z_FINISH);
|
2018-02-01 02:18:41 +00:00
|
|
|
the_hash_algo->update_fn(&c, in0, stream.next_in - in0);
|
2010-02-21 04:27:31 +00:00
|
|
|
if (write_buffer(fd, compressed, stream.next_out - compressed) < 0)
|
2018-07-21 07:49:39 +00:00
|
|
|
die(_("unable to write sha1 file"));
|
2010-02-21 04:27:31 +00:00
|
|
|
stream.next_out = compressed;
|
|
|
|
stream.avail_out = sizeof(compressed);
|
|
|
|
} while (ret == Z_OK);
|
|
|
|
|
Be more careful about zlib return values
When creating a new object, we use "deflate(stream, Z_FINISH)" in a loop
until it no longer returns Z_OK, and then we do "deflateEnd()" to finish
up business.
That should all work, but the fact is, it's not how you're _supposed_ to
use the zlib return values properly:
- deflate() should never return Z_OK in the first place, except if we
need to increase the output buffer size (which we're not doing, and
should never need to do, since we pre-allocated a buffer that is
supposed to be able to hold the output in full). So the "while()" loop
was incorrect: Z_OK doesn't actually mean "ok, continue", it means "ok,
allocate more memory for me and continue"!
- if we got an error return, we would consider it to be end-of-stream,
but it could be some internal zlib error. In short, we should check
for Z_STREAM_END explicitly, since that's the only valid return value
anyway for the Z_FINISH case.
- we never checked deflateEnd() return codes at all.
Now, admittedly, none of these issues should ever happen, unless there is
some internal bug in zlib. So this patch should make zero difference, but
it seems to be the right thing to do.
We should probablybe anal and check the return value of "deflateInit()"
too!
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-03-20 18:38:34 +00:00
|
|
|
if (ret != Z_STREAM_END)
|
2018-07-21 07:49:39 +00:00
|
|
|
die(_("unable to deflate new object %s (%d)"), oid_to_hex(oid),
|
2018-01-28 00:13:21 +00:00
|
|
|
ret);
|
2011-06-10 17:55:10 +00:00
|
|
|
ret = git_deflate_end_gently(&stream);
|
Be more careful about zlib return values
When creating a new object, we use "deflate(stream, Z_FINISH)" in a loop
until it no longer returns Z_OK, and then we do "deflateEnd()" to finish
up business.
That should all work, but the fact is, it's not how you're _supposed_ to
use the zlib return values properly:
- deflate() should never return Z_OK in the first place, except if we
need to increase the output buffer size (which we're not doing, and
should never need to do, since we pre-allocated a buffer that is
supposed to be able to hold the output in full). So the "while()" loop
was incorrect: Z_OK doesn't actually mean "ok, continue", it means "ok,
allocate more memory for me and continue"!
- if we got an error return, we would consider it to be end-of-stream,
but it could be some internal zlib error. In short, we should check
for Z_STREAM_END explicitly, since that's the only valid return value
anyway for the Z_FINISH case.
- we never checked deflateEnd() return codes at all.
Now, admittedly, none of these issues should ever happen, unless there is
some internal bug in zlib. So this patch should make zero difference, but
it seems to be the right thing to do.
We should probablybe anal and check the return value of "deflateInit()"
too!
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-03-20 18:38:34 +00:00
|
|
|
if (ret != Z_OK)
|
2018-07-21 07:49:39 +00:00
|
|
|
die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid),
|
2018-01-28 00:13:21 +00:00
|
|
|
ret);
|
2018-02-01 02:18:41 +00:00
|
|
|
the_hash_algo->final_fn(parano_oid.hash, &c);
|
2018-08-28 21:22:48 +00:00
|
|
|
if (!oideq(oid, ¶no_oid))
|
2018-07-21 07:49:39 +00:00
|
|
|
die(_("confused by unstable object source data for %s"),
|
2018-01-28 00:13:21 +00:00
|
|
|
oid_to_hex(oid));
|
Be more careful about zlib return values
When creating a new object, we use "deflate(stream, Z_FINISH)" in a loop
until it no longer returns Z_OK, and then we do "deflateEnd()" to finish
up business.
That should all work, but the fact is, it's not how you're _supposed_ to
use the zlib return values properly:
- deflate() should never return Z_OK in the first place, except if we
need to increase the output buffer size (which we're not doing, and
should never need to do, since we pre-allocated a buffer that is
supposed to be able to hold the output in full). So the "while()" loop
was incorrect: Z_OK doesn't actually mean "ok, continue", it means "ok,
allocate more memory for me and continue"!
- if we got an error return, we would consider it to be end-of-stream,
but it could be some internal zlib error. In short, we should check
for Z_STREAM_END explicitly, since that's the only valid return value
anyway for the Z_FINISH case.
- we never checked deflateEnd() return codes at all.
Now, admittedly, none of these issues should ever happen, unless there is
some internal bug in zlib. So this patch should make zero difference, but
it seems to be the right thing to do.
We should probablybe anal and check the return value of "deflateInit()"
too!
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-03-20 18:38:34 +00:00
|
|
|
|
2008-06-11 01:47:18 +00:00
|
|
|
close_sha1_file(fd);
|
2005-04-18 20:04:43 +00:00
|
|
|
|
2008-05-14 05:32:48 +00:00
|
|
|
if (mtime) {
|
|
|
|
struct utimbuf utb;
|
|
|
|
utb.actime = mtime;
|
|
|
|
utb.modtime = mtime;
|
2015-09-24 21:07:49 +00:00
|
|
|
if (utime(tmp_file.buf, &utb) < 0)
|
2018-07-21 07:49:39 +00:00
|
|
|
warning_errno(_("failed utime() on %s"), tmp_file.buf);
|
2008-05-14 05:32:48 +00:00
|
|
|
}
|
|
|
|
|
2018-01-17 17:54:54 +00:00
|
|
|
return finalize_object_file(tmp_file.buf, filename.buf);
|
2005-04-18 20:04:43 +00:00
|
|
|
}
|
2005-04-24 01:47:23 +00:00
|
|
|
|
2018-05-02 00:25:34 +00:00
|
|
|
static int freshen_loose_object(const struct object_id *oid)
|
2014-10-15 22:42:22 +00:00
|
|
|
{
|
2018-05-02 00:25:34 +00:00
|
|
|
return check_and_freshen(oid, 1);
|
2014-10-15 22:42:22 +00:00
|
|
|
}
|
|
|
|
|
2018-05-02 00:25:34 +00:00
|
|
|
static int freshen_packed_object(const struct object_id *oid)
|
2014-10-15 22:42:22 +00:00
|
|
|
{
|
|
|
|
struct pack_entry e;
|
2018-05-02 00:25:35 +00:00
|
|
|
if (!find_pack_entry(the_repository, oid, &e))
|
2015-04-20 19:55:00 +00:00
|
|
|
return 0;
|
|
|
|
if (e.p->freshened)
|
|
|
|
return 1;
|
|
|
|
if (!freshen_file(e.p->pack_name))
|
|
|
|
return 0;
|
|
|
|
e.p->freshened = 1;
|
|
|
|
return 1;
|
2014-10-15 22:42:22 +00:00
|
|
|
}
|
|
|
|
|
2018-01-28 00:13:19 +00:00
|
|
|
int write_object_file(const void *buf, unsigned long len, const char *type,
|
|
|
|
struct object_id *oid)
|
2008-05-14 05:32:48 +00:00
|
|
|
{
|
2018-03-12 02:27:55 +00:00
|
|
|
char hdr[MAX_HEADER_LEN];
|
2015-09-24 21:06:42 +00:00
|
|
|
int hdrlen = sizeof(hdr);
|
2008-05-14 05:32:48 +00:00
|
|
|
|
|
|
|
/* Normally if we have it in the pack then we do not bother writing
|
|
|
|
* it out into .git/objects/??/?{38} file.
|
|
|
|
*/
|
2018-01-28 00:13:19 +00:00
|
|
|
write_object_file_prepare(buf, len, type, oid, hdr, &hdrlen);
|
2018-05-02 00:25:34 +00:00
|
|
|
if (freshen_packed_object(oid) || freshen_loose_object(oid))
|
2008-05-14 05:32:48 +00:00
|
|
|
return 0;
|
2018-01-28 00:13:21 +00:00
|
|
|
return write_loose_object(oid, hdr, hdrlen, buf, len, 0);
|
2008-05-14 05:32:48 +00:00
|
|
|
}
|
|
|
|
|
2018-01-28 00:13:22 +00:00
|
|
|
int hash_object_file_literally(const void *buf, unsigned long len,
|
|
|
|
const char *type, struct object_id *oid,
|
|
|
|
unsigned flags)
|
2015-05-04 07:25:15 +00:00
|
|
|
{
|
|
|
|
char *header;
|
|
|
|
int hdrlen, status = 0;
|
|
|
|
|
|
|
|
/* type string, SP, %lu of the length plus NUL must fit this */
|
2018-03-12 02:27:55 +00:00
|
|
|
hdrlen = strlen(type) + MAX_HEADER_LEN;
|
2015-09-24 21:06:42 +00:00
|
|
|
header = xmalloc(hdrlen);
|
2018-01-28 00:13:19 +00:00
|
|
|
write_object_file_prepare(buf, len, type, oid, header, &hdrlen);
|
2015-05-04 07:25:15 +00:00
|
|
|
|
|
|
|
if (!(flags & HASH_WRITE_OBJECT))
|
|
|
|
goto cleanup;
|
2018-05-02 00:25:34 +00:00
|
|
|
if (freshen_packed_object(oid) || freshen_loose_object(oid))
|
2015-05-04 07:25:15 +00:00
|
|
|
goto cleanup;
|
2018-01-28 00:13:21 +00:00
|
|
|
status = write_loose_object(oid, header, hdrlen, buf, len, 0);
|
2015-05-04 07:25:15 +00:00
|
|
|
|
|
|
|
cleanup:
|
|
|
|
free(header);
|
|
|
|
return status;
|
|
|
|
}
|
|
|
|
|
2018-01-28 00:13:20 +00:00
|
|
|
int force_object_loose(const struct object_id *oid, time_t mtime)
|
2008-05-14 05:32:48 +00:00
|
|
|
{
|
|
|
|
void *buf;
|
|
|
|
unsigned long len;
|
|
|
|
enum object_type type;
|
2018-03-12 02:27:55 +00:00
|
|
|
char hdr[MAX_HEADER_LEN];
|
2008-05-14 05:32:48 +00:00
|
|
|
int hdrlen;
|
2008-10-18 00:37:31 +00:00
|
|
|
int ret;
|
2008-05-14 05:32:48 +00:00
|
|
|
|
2018-05-02 00:25:34 +00:00
|
|
|
if (has_loose_object(oid))
|
2008-05-14 05:32:48 +00:00
|
|
|
return 0;
|
2018-01-28 00:13:20 +00:00
|
|
|
buf = read_object(oid->hash, &type, &len);
|
2008-05-14 05:32:48 +00:00
|
|
|
if (!buf)
|
2018-07-21 07:49:39 +00:00
|
|
|
return error(_("cannot read sha1_file for %s"), oid_to_hex(oid));
|
2018-02-14 18:59:24 +00:00
|
|
|
hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %lu", type_name(type), len) + 1;
|
2018-01-28 00:13:21 +00:00
|
|
|
ret = write_loose_object(oid, hdr, hdrlen, buf, len, mtime);
|
2008-10-18 00:37:31 +00:00
|
|
|
free(buf);
|
|
|
|
|
|
|
|
return ret;
|
2008-05-14 05:32:48 +00:00
|
|
|
}
|
|
|
|
|
2015-06-09 17:24:37 +00:00
|
|
|
int has_sha1_file_with_flags(const unsigned char *sha1, int flags)
|
2005-04-24 01:47:23 +00:00
|
|
|
{
|
2018-03-12 02:27:45 +00:00
|
|
|
struct object_id oid;
|
2017-04-11 22:47:13 +00:00
|
|
|
if (!startup_info->have_repository)
|
|
|
|
return 0;
|
2018-03-12 02:27:45 +00:00
|
|
|
hashcpy(oid.hash, sha1);
|
2018-04-25 18:20:58 +00:00
|
|
|
return oid_object_info_extended(the_repository, &oid, NULL,
|
sha1_file: convert sha1_object_info* to object_id
Convert sha1_object_info and sha1_object_info_extended to take pointers
to struct object_id and rename them to use "oid" instead of "sha1" in
their names. Update the declaration and definition and apply the
following semantic patch, plus the standard object_id transforms:
@@
expression E1, E2;
@@
- sha1_object_info(E1.hash, E2)
+ oid_object_info(&E1, E2)
@@
expression E1, E2;
@@
- sha1_object_info(E1->hash, E2)
+ oid_object_info(E1, E2)
@@
expression E1, E2, E3;
@@
- sha1_object_info_extended(E1.hash, E2, E3)
+ oid_object_info_extended(&E1, E2, E3)
@@
expression E1, E2, E3;
@@
- sha1_object_info_extended(E1->hash, E2, E3)
+ oid_object_info_extended(E1, E2, E3)
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-03-12 02:27:46 +00:00
|
|
|
flags | OBJECT_INFO_SKIP_CACHED) >= 0;
|
2015-11-10 02:22:19 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
int has_object_file(const struct object_id *oid)
|
|
|
|
{
|
|
|
|
return has_sha1_file(oid->hash);
|
2005-04-24 01:47:23 +00:00
|
|
|
}
|
2005-05-02 06:45:49 +00:00
|
|
|
|
fetch: use "quick" has_sha1_file for tag following
When we auto-follow tags in a fetch, we look at all of the
tags advertised by the remote and fetch ones where we don't
already have the tag, but we do have the object it peels to.
This involves a lot of calls to has_sha1_file(), some of
which we can reasonably expect to fail. Since 45e8a74
(has_sha1_file: re-check pack directory before giving up,
2013-08-30), this may cause many calls to
reprepare_packed_git(), which is potentially expensive.
This has gone unnoticed for several years because it
requires a fairly unique setup to matter:
1. You need to have a lot of packs on the client side to
make reprepare_packed_git() expensive (the most
expensive part is finding duplicates in an unsorted
list, which is currently quadratic).
2. You need a large number of tag refs on the server side
that are candidates for auto-following (i.e., that the
client doesn't have). Each one triggers a re-read of
the pack directory.
3. Under normal circumstances, the client would
auto-follow those tags and after one large fetch, (2)
would no longer be true. But if those tags point to
history which is disconnected from what the client
otherwise fetches, then it will never auto-follow, and
those candidates will impact it on every fetch.
So when all three are true, each fetch pays an extra
O(nr_tags * nr_packs^2) cost, mostly in string comparisons
on the pack names. This was exacerbated by 47bf4b0
(prepare_packed_git_one: refactor duplicate-pack check,
2014-06-30) which uses a slightly more expensive string
check, under the assumption that the duplicate check doesn't
happen very often (and it shouldn't; the real problem here
is how often we are calling reprepare_packed_git()).
This patch teaches fetch to use HAS_SHA1_QUICK to sacrifice
accuracy for speed, in cases where we might be racy with a
simultaneous repack. This is similar to the fix in 0eeb077
(index-pack: avoid excessive re-reading of pack directory,
2015-06-09). As with that case, it's OK for has_sha1_file()
occasionally say "no I don't have it" when we do, because
the worst case is not a corruption, but simply that we may
fail to auto-follow a tag that points to it.
Here are results from the included perf script, which sets
up a situation similar to the one described above:
Test HEAD^ HEAD
----------------------------------------------------------
5550.4: fetch 11.21(10.42+0.78) 0.08(0.04+0.02) -99.3%
Reported-by: Vegard Nossum <vegard.nossum@oracle.com>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-13 16:53:44 +00:00
|
|
|
int has_object_file_with_flags(const struct object_id *oid, int flags)
|
|
|
|
{
|
|
|
|
return has_sha1_file_with_flags(oid->hash, flags);
|
|
|
|
}
|
|
|
|
|
2011-02-05 10:52:21 +00:00
|
|
|
static void check_tree(const void *buf, size_t size)
|
|
|
|
{
|
|
|
|
struct tree_desc desc;
|
|
|
|
struct name_entry entry;
|
|
|
|
|
|
|
|
init_tree_desc(&desc, buf, size);
|
|
|
|
while (tree_entry(&desc, &entry))
|
|
|
|
/* do nothing
|
|
|
|
* tree_entry() will die() on malformed entries */
|
|
|
|
;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void check_commit(const void *buf, size_t size)
|
|
|
|
{
|
|
|
|
struct commit c;
|
|
|
|
memset(&c, 0, sizeof(c));
|
2018-06-29 01:22:00 +00:00
|
|
|
if (parse_commit_buffer(the_repository, &c, buf, size, 0))
|
2018-07-21 07:49:39 +00:00
|
|
|
die(_("corrupt commit"));
|
2011-02-05 10:52:21 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
static void check_tag(const void *buf, size_t size)
|
|
|
|
{
|
|
|
|
struct tag t;
|
|
|
|
memset(&t, 0, sizeof(t));
|
2018-06-29 01:22:04 +00:00
|
|
|
if (parse_tag_buffer(the_repository, &t, buf, size))
|
2018-07-21 07:49:39 +00:00
|
|
|
die(_("corrupt tag"));
|
2011-02-05 10:52:21 +00:00
|
|
|
}
|
|
|
|
|
2018-09-21 15:57:31 +00:00
|
|
|
static int index_mem(struct index_state *istate,
|
|
|
|
struct object_id *oid, void *buf, size_t size,
|
2011-05-08 08:47:33 +00:00
|
|
|
enum object_type type,
|
|
|
|
const char *path, unsigned flags)
|
2006-05-23 18:19:04 +00:00
|
|
|
{
|
Lazy man's auto-CRLF
It currently does NOT know about file attributes, so it does its
conversion purely based on content. Maybe that is more in the "git
philosophy" anyway, since content is king, but I think we should try to do
the file attributes to turn it off on demand.
Anyway, BY DEFAULT it is off regardless, because it requires a
[core]
AutoCRLF = true
in your config file to be enabled. We could make that the default for
Windows, of course, the same way we do some other things (filemode etc).
But you can actually enable it on UNIX, and it will cause:
- "git update-index" will write blobs without CRLF
- "git diff" will diff working tree files without CRLF
- "git checkout" will write files to the working tree _with_ CRLF
and things work fine.
Funnily, it actually shows an odd file in git itself:
git clone -n git test-crlf
cd test-crlf
git config core.autocrlf true
git checkout
git diff
shows a diff for "Documentation/docbook-xsl.css". Why? Because we have
actually checked in that file *with* CRLF! So when "core.autocrlf" is
true, we'll always generate a *different* hash for it in the index,
because the index hash will be for the content _without_ CRLF.
Is this complete? I dunno. It seems to work for me. It doesn't use the
filename at all right now, and that's probably a deficiency (we could
certainly make the "is_binary()" heuristics also take standard filename
heuristics into account).
I don't pass in the filename at all for the "index_fd()" case
(git-update-index), so that would need to be passed around, but this
actually works fine.
NOTE NOTE NOTE! The "is_binary()" heuristics are totally made-up by yours
truly. I will not guarantee that they work at all reasonable. Caveat
emptor. But it _is_ simple, and it _is_ safe, since it's all off by
default.
The patch is pretty simple - the biggest part is the new "convert.c" file,
but even that is really just basic stuff that anybody can write in
"Teaching C 101" as a final project for their first class in programming.
Not to say that it's bug-free, of course - but at least we're not talking
about rocket surgery here.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-13 19:07:23 +00:00
|
|
|
int ret, re_allocated = 0;
|
2011-05-08 08:47:33 +00:00
|
|
|
int write_object = flags & HASH_WRITE_OBJECT;
|
2005-05-02 06:45:49 +00:00
|
|
|
|
2005-07-08 23:51:55 +00:00
|
|
|
if (!type)
|
2007-02-28 19:45:56 +00:00
|
|
|
type = OBJ_BLOB;
|
Lazy man's auto-CRLF
It currently does NOT know about file attributes, so it does its
conversion purely based on content. Maybe that is more in the "git
philosophy" anyway, since content is king, but I think we should try to do
the file attributes to turn it off on demand.
Anyway, BY DEFAULT it is off regardless, because it requires a
[core]
AutoCRLF = true
in your config file to be enabled. We could make that the default for
Windows, of course, the same way we do some other things (filemode etc).
But you can actually enable it on UNIX, and it will cause:
- "git update-index" will write blobs without CRLF
- "git diff" will diff working tree files without CRLF
- "git checkout" will write files to the working tree _with_ CRLF
and things work fine.
Funnily, it actually shows an odd file in git itself:
git clone -n git test-crlf
cd test-crlf
git config core.autocrlf true
git checkout
git diff
shows a diff for "Documentation/docbook-xsl.css". Why? Because we have
actually checked in that file *with* CRLF! So when "core.autocrlf" is
true, we'll always generate a *different* hash for it in the index,
because the index hash will be for the content _without_ CRLF.
Is this complete? I dunno. It seems to work for me. It doesn't use the
filename at all right now, and that's probably a deficiency (we could
certainly make the "is_binary()" heuristics also take standard filename
heuristics into account).
I don't pass in the filename at all for the "index_fd()" case
(git-update-index), so that would need to be passed around, but this
actually works fine.
NOTE NOTE NOTE! The "is_binary()" heuristics are totally made-up by yours
truly. I will not guarantee that they work at all reasonable. Caveat
emptor. But it _is_ simple, and it _is_ safe, since it's all off by
default.
The patch is pretty simple - the biggest part is the new "convert.c" file,
but even that is really just basic stuff that anybody can write in
"Teaching C 101" as a final project for their first class in programming.
Not to say that it's bug-free, of course - but at least we're not talking
about rocket surgery here.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-13 19:07:23 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Convert blobs to git internal format
|
|
|
|
*/
|
2008-08-03 04:39:16 +00:00
|
|
|
if ((type == OBJ_BLOB) && path) {
|
2008-10-09 19:12:12 +00:00
|
|
|
struct strbuf nbuf = STRBUF_INIT;
|
2018-09-21 15:57:31 +00:00
|
|
|
if (convert_to_git(istate, path, buf, size, &nbuf,
|
2018-01-13 22:49:31 +00:00
|
|
|
get_conv_flags(flags))) {
|
2007-09-27 10:58:23 +00:00
|
|
|
buf = strbuf_detach(&nbuf, &size);
|
Lazy man's auto-CRLF
It currently does NOT know about file attributes, so it does its
conversion purely based on content. Maybe that is more in the "git
philosophy" anyway, since content is king, but I think we should try to do
the file attributes to turn it off on demand.
Anyway, BY DEFAULT it is off regardless, because it requires a
[core]
AutoCRLF = true
in your config file to be enabled. We could make that the default for
Windows, of course, the same way we do some other things (filemode etc).
But you can actually enable it on UNIX, and it will cause:
- "git update-index" will write blobs without CRLF
- "git diff" will diff working tree files without CRLF
- "git checkout" will write files to the working tree _with_ CRLF
and things work fine.
Funnily, it actually shows an odd file in git itself:
git clone -n git test-crlf
cd test-crlf
git config core.autocrlf true
git checkout
git diff
shows a diff for "Documentation/docbook-xsl.css". Why? Because we have
actually checked in that file *with* CRLF! So when "core.autocrlf" is
true, we'll always generate a *different* hash for it in the index,
because the index hash will be for the content _without_ CRLF.
Is this complete? I dunno. It seems to work for me. It doesn't use the
filename at all right now, and that's probably a deficiency (we could
certainly make the "is_binary()" heuristics also take standard filename
heuristics into account).
I don't pass in the filename at all for the "index_fd()" case
(git-update-index), so that would need to be passed around, but this
actually works fine.
NOTE NOTE NOTE! The "is_binary()" heuristics are totally made-up by yours
truly. I will not guarantee that they work at all reasonable. Caveat
emptor. But it _is_ simple, and it _is_ safe, since it's all off by
default.
The patch is pretty simple - the biggest part is the new "convert.c" file,
but even that is really just basic stuff that anybody can write in
"Teaching C 101" as a final project for their first class in programming.
Not to say that it's bug-free, of course - but at least we're not talking
about rocket surgery here.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-13 19:07:23 +00:00
|
|
|
re_allocated = 1;
|
|
|
|
}
|
|
|
|
}
|
2011-05-08 08:47:33 +00:00
|
|
|
if (flags & HASH_FORMAT_CHECK) {
|
2011-02-05 10:52:21 +00:00
|
|
|
if (type == OBJ_TREE)
|
|
|
|
check_tree(buf, size);
|
|
|
|
if (type == OBJ_COMMIT)
|
|
|
|
check_commit(buf, size);
|
|
|
|
if (type == OBJ_TAG)
|
|
|
|
check_tag(buf, size);
|
|
|
|
}
|
Lazy man's auto-CRLF
It currently does NOT know about file attributes, so it does its
conversion purely based on content. Maybe that is more in the "git
philosophy" anyway, since content is king, but I think we should try to do
the file attributes to turn it off on demand.
Anyway, BY DEFAULT it is off regardless, because it requires a
[core]
AutoCRLF = true
in your config file to be enabled. We could make that the default for
Windows, of course, the same way we do some other things (filemode etc).
But you can actually enable it on UNIX, and it will cause:
- "git update-index" will write blobs without CRLF
- "git diff" will diff working tree files without CRLF
- "git checkout" will write files to the working tree _with_ CRLF
and things work fine.
Funnily, it actually shows an odd file in git itself:
git clone -n git test-crlf
cd test-crlf
git config core.autocrlf true
git checkout
git diff
shows a diff for "Documentation/docbook-xsl.css". Why? Because we have
actually checked in that file *with* CRLF! So when "core.autocrlf" is
true, we'll always generate a *different* hash for it in the index,
because the index hash will be for the content _without_ CRLF.
Is this complete? I dunno. It seems to work for me. It doesn't use the
filename at all right now, and that's probably a deficiency (we could
certainly make the "is_binary()" heuristics also take standard filename
heuristics into account).
I don't pass in the filename at all for the "index_fd()" case
(git-update-index), so that would need to be passed around, but this
actually works fine.
NOTE NOTE NOTE! The "is_binary()" heuristics are totally made-up by yours
truly. I will not guarantee that they work at all reasonable. Caveat
emptor. But it _is_ simple, and it _is_ safe, since it's all off by
default.
The patch is pretty simple - the biggest part is the new "convert.c" file,
but even that is really just basic stuff that anybody can write in
"Teaching C 101" as a final project for their first class in programming.
Not to say that it's bug-free, of course - but at least we're not talking
about rocket surgery here.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-13 19:07:23 +00:00
|
|
|
|
2005-07-08 23:51:55 +00:00
|
|
|
if (write_object)
|
2018-03-06 22:54:07 +00:00
|
|
|
ret = write_object_file(buf, size, type_name(type), oid);
|
2006-10-14 10:45:36 +00:00
|
|
|
else
|
2018-03-06 22:54:07 +00:00
|
|
|
ret = hash_object_file(buf, size, type_name(type), oid);
|
2008-08-03 04:39:16 +00:00
|
|
|
if (re_allocated)
|
Lazy man's auto-CRLF
It currently does NOT know about file attributes, so it does its
conversion purely based on content. Maybe that is more in the "git
philosophy" anyway, since content is king, but I think we should try to do
the file attributes to turn it off on demand.
Anyway, BY DEFAULT it is off regardless, because it requires a
[core]
AutoCRLF = true
in your config file to be enabled. We could make that the default for
Windows, of course, the same way we do some other things (filemode etc).
But you can actually enable it on UNIX, and it will cause:
- "git update-index" will write blobs without CRLF
- "git diff" will diff working tree files without CRLF
- "git checkout" will write files to the working tree _with_ CRLF
and things work fine.
Funnily, it actually shows an odd file in git itself:
git clone -n git test-crlf
cd test-crlf
git config core.autocrlf true
git checkout
git diff
shows a diff for "Documentation/docbook-xsl.css". Why? Because we have
actually checked in that file *with* CRLF! So when "core.autocrlf" is
true, we'll always generate a *different* hash for it in the index,
because the index hash will be for the content _without_ CRLF.
Is this complete? I dunno. It seems to work for me. It doesn't use the
filename at all right now, and that's probably a deficiency (we could
certainly make the "is_binary()" heuristics also take standard filename
heuristics into account).
I don't pass in the filename at all for the "index_fd()" case
(git-update-index), so that would need to be passed around, but this
actually works fine.
NOTE NOTE NOTE! The "is_binary()" heuristics are totally made-up by yours
truly. I will not guarantee that they work at all reasonable. Caveat
emptor. But it _is_ simple, and it _is_ safe, since it's all off by
default.
The patch is pretty simple - the biggest part is the new "convert.c" file,
but even that is really just basic stuff that anybody can write in
"Teaching C 101" as a final project for their first class in programming.
Not to say that it's bug-free, of course - but at least we're not talking
about rocket surgery here.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-13 19:07:23 +00:00
|
|
|
free(buf);
|
2008-08-03 04:39:16 +00:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2018-09-21 15:57:31 +00:00
|
|
|
static int index_stream_convert_blob(struct index_state *istate,
|
|
|
|
struct object_id *oid,
|
|
|
|
int fd,
|
|
|
|
const char *path,
|
|
|
|
unsigned flags)
|
2014-08-26 15:23:25 +00:00
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
const int write_object = flags & HASH_WRITE_OBJECT;
|
|
|
|
struct strbuf sbuf = STRBUF_INIT;
|
|
|
|
|
|
|
|
assert(path);
|
2018-09-21 15:57:31 +00:00
|
|
|
assert(would_convert_to_git_filter_fd(istate, path));
|
2014-08-26 15:23:25 +00:00
|
|
|
|
2018-09-21 15:57:31 +00:00
|
|
|
convert_to_git_filter_fd(istate, path, fd, &sbuf,
|
2018-01-13 22:49:31 +00:00
|
|
|
get_conv_flags(flags));
|
2014-08-26 15:23:25 +00:00
|
|
|
|
|
|
|
if (write_object)
|
2018-03-06 22:54:07 +00:00
|
|
|
ret = write_object_file(sbuf.buf, sbuf.len, type_name(OBJ_BLOB),
|
2018-01-28 00:13:19 +00:00
|
|
|
oid);
|
2014-08-26 15:23:25 +00:00
|
|
|
else
|
2018-03-06 22:54:07 +00:00
|
|
|
ret = hash_object_file(sbuf.buf, sbuf.len, type_name(OBJ_BLOB),
|
2018-01-28 00:13:13 +00:00
|
|
|
oid);
|
2014-08-26 15:23:25 +00:00
|
|
|
strbuf_release(&sbuf);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2018-09-21 15:57:31 +00:00
|
|
|
static int index_pipe(struct index_state *istate, struct object_id *oid,
|
|
|
|
int fd, enum object_type type,
|
2011-05-08 08:47:34 +00:00
|
|
|
const char *path, unsigned flags)
|
|
|
|
{
|
|
|
|
struct strbuf sbuf = STRBUF_INIT;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
if (strbuf_read(&sbuf, fd, 4096) >= 0)
|
2018-09-21 15:57:31 +00:00
|
|
|
ret = index_mem(istate, oid, sbuf.buf, sbuf.len, type, path, flags);
|
2011-05-08 08:47:34 +00:00
|
|
|
else
|
|
|
|
ret = -1;
|
|
|
|
strbuf_release(&sbuf);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2010-02-21 06:32:19 +00:00
|
|
|
#define SMALL_FILE_SIZE (32*1024)
|
|
|
|
|
2018-09-21 15:57:31 +00:00
|
|
|
static int index_core(struct index_state *istate,
|
|
|
|
struct object_id *oid, int fd, size_t size,
|
2011-05-08 08:47:34 +00:00
|
|
|
enum object_type type, const char *path,
|
|
|
|
unsigned flags)
|
2008-08-03 04:39:16 +00:00
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
|
2011-05-08 08:47:34 +00:00
|
|
|
if (!size) {
|
2018-09-21 15:57:31 +00:00
|
|
|
ret = index_mem(istate, oid, "", size, type, path, flags);
|
2010-02-21 06:32:19 +00:00
|
|
|
} else if (size <= SMALL_FILE_SIZE) {
|
|
|
|
char *buf = xmalloc(size);
|
2017-09-27 06:01:07 +00:00
|
|
|
ssize_t read_result = read_in_full(fd, buf, size);
|
|
|
|
if (read_result < 0)
|
2018-07-21 07:49:39 +00:00
|
|
|
ret = error_errno(_("read error while indexing %s"),
|
2017-09-27 06:01:07 +00:00
|
|
|
path ? path : "<unknown>");
|
|
|
|
else if (read_result != size)
|
2018-07-21 07:49:39 +00:00
|
|
|
ret = error(_("short read while indexing %s"),
|
2017-09-27 06:01:07 +00:00
|
|
|
path ? path : "<unknown>");
|
2010-02-21 06:32:19 +00:00
|
|
|
else
|
2018-09-21 15:57:31 +00:00
|
|
|
ret = index_mem(istate, oid, buf, size, type, path, flags);
|
2010-02-21 06:32:19 +00:00
|
|
|
free(buf);
|
2010-05-10 21:38:17 +00:00
|
|
|
} else {
|
2008-08-03 04:39:16 +00:00
|
|
|
void *buf = xmmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0);
|
2018-09-21 15:57:31 +00:00
|
|
|
ret = index_mem(istate, oid, buf, size, type, path, flags);
|
2005-05-03 18:46:16 +00:00
|
|
|
munmap(buf, size);
|
2010-05-10 21:38:17 +00:00
|
|
|
}
|
2011-05-08 08:47:34 +00:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2011-05-08 08:47:35 +00:00
|
|
|
/*
|
2011-10-28 21:48:40 +00:00
|
|
|
* This creates one packfile per large blob unless bulk-checkin
|
|
|
|
* machinery is "plugged".
|
2011-05-08 08:47:35 +00:00
|
|
|
*
|
|
|
|
* This also bypasses the usual "convert-to-git" dance, and that is on
|
|
|
|
* purpose. We could write a streaming version of the converting
|
|
|
|
* functions and insert that before feeding the data to fast-import
|
do not stream large files to pack when filters are in use
Because git's object format requires us to specify the
number of bytes in the object in its header, we must know
the size before streaming a blob into the object database.
This is not a problem when adding a regular file, as we can
get the size from stat(). However, when filters are in use
(such as autocrlf, or the ident, filter, or eol
gitattributes), we have no idea what the ultimate size will
be.
The current code just punts on the whole issue and ignores
filter configuration entirely for files larger than
core.bigfilethreshold. This can generate confusing results
if you use filters for large binary files, as the filter
will suddenly stop working as the file goes over a certain
size. Rather than try to handle unknown input sizes with
streaming, this patch just turns off the streaming
optimization when filters are in use.
This has a slight performance regression in a very specific
case: if you have autocrlf on, but no gitattributes, a large
binary file will avoid the streaming code path because we
don't know beforehand whether it will need conversion or
not. But if you are handling large binary files, you should
be marking them as such via attributes (or at least not
using autocrlf, and instead marking your text files as
such). And the flip side is that if you have a large
_non_-binary file, there is a correctness improvement;
before we did not apply the conversion at all.
The first half of the new t1051 script covers these failures
on input. The second half tests the matching output code
paths. These already work correctly, and do not need any
adjustment.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-02-24 22:10:17 +00:00
|
|
|
* (or equivalent in-core API described above). However, that is
|
|
|
|
* somewhat complicated, as we do not know the size of the filter
|
|
|
|
* result, which we need to know beforehand when writing a git object.
|
|
|
|
* Since the primary motivation for trying to stream from the working
|
|
|
|
* tree file and to avoid mmaping it in core is to deal with large
|
|
|
|
* binary blobs, they generally do not want to get any conversion, and
|
|
|
|
* callers should avoid this code path when filters are requested.
|
2011-05-08 08:47:35 +00:00
|
|
|
*/
|
2017-08-20 20:09:31 +00:00
|
|
|
static int index_stream(struct object_id *oid, int fd, size_t size,
|
2011-05-08 08:47:35 +00:00
|
|
|
enum object_type type, const char *path,
|
|
|
|
unsigned flags)
|
|
|
|
{
|
2018-03-12 02:27:21 +00:00
|
|
|
return index_bulk_checkin(oid, fd, size, type, path, flags);
|
2011-05-08 08:47:35 +00:00
|
|
|
}
|
|
|
|
|
2018-09-21 15:57:31 +00:00
|
|
|
int index_fd(struct index_state *istate, struct object_id *oid,
|
|
|
|
int fd, struct stat *st,
|
2011-05-08 08:47:34 +00:00
|
|
|
enum object_type type, const char *path, unsigned flags)
|
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
|
2014-09-21 10:03:26 +00:00
|
|
|
/*
|
|
|
|
* Call xsize_t() only when needed to avoid potentially unnecessary
|
|
|
|
* die() for large files.
|
|
|
|
*/
|
2018-09-21 15:57:31 +00:00
|
|
|
if (type == OBJ_BLOB && path && would_convert_to_git_filter_fd(istate, path))
|
|
|
|
ret = index_stream_convert_blob(istate, oid, fd, path, flags);
|
2014-08-26 15:23:25 +00:00
|
|
|
else if (!S_ISREG(st->st_mode))
|
2018-09-21 15:57:31 +00:00
|
|
|
ret = index_pipe(istate, oid, fd, type, path, flags);
|
2014-09-21 10:03:26 +00:00
|
|
|
else if (st->st_size <= big_file_threshold || type != OBJ_BLOB ||
|
2018-09-21 15:57:31 +00:00
|
|
|
(path && would_convert_to_git(istate, path)))
|
|
|
|
ret = index_core(istate, oid, fd, xsize_t(st->st_size),
|
|
|
|
type, path, flags);
|
2011-05-08 08:47:35 +00:00
|
|
|
else
|
2017-08-20 20:09:31 +00:00
|
|
|
ret = index_stream(oid, fd, xsize_t(st->st_size), type, path,
|
2014-09-21 10:03:26 +00:00
|
|
|
flags);
|
2008-08-03 04:39:16 +00:00
|
|
|
close(fd);
|
2005-05-03 18:46:16 +00:00
|
|
|
return ret;
|
2005-05-02 06:45:49 +00:00
|
|
|
}
|
2005-10-07 10:42:00 +00:00
|
|
|
|
2018-09-21 15:57:31 +00:00
|
|
|
int index_path(struct index_state *istate, struct object_id *oid,
|
|
|
|
const char *path, struct stat *st, unsigned flags)
|
2005-10-07 10:42:00 +00:00
|
|
|
{
|
|
|
|
int fd;
|
2008-12-17 17:51:53 +00:00
|
|
|
struct strbuf sb = STRBUF_INIT;
|
2017-08-30 18:00:29 +00:00
|
|
|
int rc = 0;
|
2005-10-07 10:42:00 +00:00
|
|
|
|
|
|
|
switch (st->st_mode & S_IFMT) {
|
|
|
|
case S_IFREG:
|
|
|
|
fd = open(path, O_RDONLY);
|
|
|
|
if (fd < 0)
|
2016-05-08 09:47:56 +00:00
|
|
|
return error_errno("open(\"%s\")", path);
|
2018-09-21 15:57:31 +00:00
|
|
|
if (index_fd(istate, oid, fd, st, OBJ_BLOB, path, flags) < 0)
|
2018-07-21 07:49:39 +00:00
|
|
|
return error(_("%s: failed to insert into database"),
|
2005-10-07 10:42:00 +00:00
|
|
|
path);
|
|
|
|
break;
|
|
|
|
case S_IFLNK:
|
2016-05-08 09:47:56 +00:00
|
|
|
if (strbuf_readlink(&sb, path, st->st_size))
|
|
|
|
return error_errno("readlink(\"%s\")", path);
|
2011-05-08 08:47:33 +00:00
|
|
|
if (!(flags & HASH_WRITE_OBJECT))
|
2018-01-28 00:13:13 +00:00
|
|
|
hash_object_file(sb.buf, sb.len, blob_type, oid);
|
2018-01-28 00:13:19 +00:00
|
|
|
else if (write_object_file(sb.buf, sb.len, blob_type, oid))
|
2018-07-21 07:49:39 +00:00
|
|
|
rc = error(_("%s: failed to insert into database"), path);
|
2008-12-17 17:51:53 +00:00
|
|
|
strbuf_release(&sb);
|
2005-10-07 10:42:00 +00:00
|
|
|
break;
|
2007-04-10 04:20:29 +00:00
|
|
|
case S_IFDIR:
|
refs: convert resolve_gitlink_ref to struct object_id
Convert the declaration and definition of resolve_gitlink_ref to use
struct object_id and apply the following semantic patch:
@@
expression E1, E2, E3;
@@
- resolve_gitlink_ref(E1, E2, E3.hash)
+ resolve_gitlink_ref(E1, E2, &E3)
@@
expression E1, E2, E3;
@@
- resolve_gitlink_ref(E1, E2, E3->hash)
+ resolve_gitlink_ref(E1, E2, E3)
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-10-15 22:07:07 +00:00
|
|
|
return resolve_gitlink_ref(path, "HEAD", oid);
|
2005-10-07 10:42:00 +00:00
|
|
|
default:
|
2018-07-21 07:49:39 +00:00
|
|
|
return error(_("%s: unsupported file type"), path);
|
2005-10-07 10:42:00 +00:00
|
|
|
}
|
2017-08-30 18:00:29 +00:00
|
|
|
return rc;
|
2005-10-07 10:42:00 +00:00
|
|
|
}
|
2007-01-23 05:55:18 +00:00
|
|
|
|
|
|
|
int read_pack_header(int fd, struct pack_header *header)
|
|
|
|
{
|
2017-09-13 18:47:22 +00:00
|
|
|
if (read_in_full(fd, header, sizeof(*header)) != sizeof(*header))
|
2008-05-03 13:27:26 +00:00
|
|
|
/* "eof before pack header was fully read" */
|
|
|
|
return PH_ERROR_EOF;
|
|
|
|
|
2007-01-23 05:55:18 +00:00
|
|
|
if (header->hdr_signature != htonl(PACK_SIGNATURE))
|
|
|
|
/* "protocol error (pack signature mismatch detected)" */
|
|
|
|
return PH_ERROR_PACK_SIGNATURE;
|
|
|
|
if (!pack_version_ok(header->hdr_version))
|
|
|
|
/* "protocol error (pack version unsupported)" */
|
|
|
|
return PH_ERROR_PROTOCOL;
|
|
|
|
return 0;
|
|
|
|
}
|
make commit_tree a library function
Until now, this has been part of the commit-tree builtin.
However, it is already used by other builtins (like commit,
merge, and notes), and it would be useful to access it from
library code.
The check_valid helper has to come along, too, but is given
a more library-ish name of "assert_sha1_type".
Otherwise, the code is unchanged. There are still a few
rough edges for a library function, like printing the utf8
warning to stderr, but we can address those if and when they
come up as inappropriate.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-04-02 00:05:23 +00:00
|
|
|
|
2018-03-12 02:27:42 +00:00
|
|
|
void assert_oid_type(const struct object_id *oid, enum object_type expect)
|
make commit_tree a library function
Until now, this has been part of the commit-tree builtin.
However, it is already used by other builtins (like commit,
merge, and notes), and it would be useful to access it from
library code.
The check_valid helper has to come along, too, but is given
a more library-ish name of "assert_sha1_type".
Otherwise, the code is unchanged. There are still a few
rough edges for a library function, like printing the utf8
warning to stderr, but we can address those if and when they
come up as inappropriate.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-04-02 00:05:23 +00:00
|
|
|
{
|
2018-04-25 18:20:59 +00:00
|
|
|
enum object_type type = oid_object_info(the_repository, oid, NULL);
|
make commit_tree a library function
Until now, this has been part of the commit-tree builtin.
However, it is already used by other builtins (like commit,
merge, and notes), and it would be useful to access it from
library code.
The check_valid helper has to come along, too, but is given
a more library-ish name of "assert_sha1_type".
Otherwise, the code is unchanged. There are still a few
rough edges for a library function, like printing the utf8
warning to stderr, but we can address those if and when they
come up as inappropriate.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-04-02 00:05:23 +00:00
|
|
|
if (type < 0)
|
2018-07-21 07:49:39 +00:00
|
|
|
die(_("%s is not a valid object"), oid_to_hex(oid));
|
make commit_tree a library function
Until now, this has been part of the commit-tree builtin.
However, it is already used by other builtins (like commit,
merge, and notes), and it would be useful to access it from
library code.
The check_valid helper has to come along, too, but is given
a more library-ish name of "assert_sha1_type".
Otherwise, the code is unchanged. There are still a few
rough edges for a library function, like printing the utf8
warning to stderr, but we can address those if and when they
come up as inappropriate.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-04-02 00:05:23 +00:00
|
|
|
if (type != expect)
|
2018-07-21 07:49:39 +00:00
|
|
|
die(_("%s is not a valid '%s' object"), oid_to_hex(oid),
|
2018-02-14 18:59:24 +00:00
|
|
|
type_name(expect));
|
make commit_tree a library function
Until now, this has been part of the commit-tree builtin.
However, it is already used by other builtins (like commit,
merge, and notes), and it would be useful to access it from
library code.
The check_valid helper has to come along, too, but is given
a more library-ish name of "assert_sha1_type".
Otherwise, the code is unchanged. There are still a few
rough edges for a library function, like printing the utf8
warning to stderr, but we can address those if and when they
come up as inappropriate.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-04-02 00:05:23 +00:00
|
|
|
}
|
2014-10-15 22:38:55 +00:00
|
|
|
|
2017-06-24 14:09:39 +00:00
|
|
|
int for_each_file_in_obj_subdir(unsigned int subdir_nr,
|
2017-06-22 18:19:48 +00:00
|
|
|
struct strbuf *path,
|
|
|
|
each_loose_object_fn obj_cb,
|
|
|
|
each_loose_cruft_fn cruft_cb,
|
|
|
|
each_loose_subdir_fn subdir_cb,
|
|
|
|
void *data)
|
2014-10-15 22:38:55 +00:00
|
|
|
{
|
2017-06-24 12:12:30 +00:00
|
|
|
size_t origlen, baselen;
|
|
|
|
DIR *dir;
|
2014-10-15 22:38:55 +00:00
|
|
|
struct dirent *de;
|
|
|
|
int r = 0;
|
2017-10-31 13:50:06 +00:00
|
|
|
struct object_id oid;
|
2014-10-15 22:38:55 +00:00
|
|
|
|
2017-06-24 14:09:39 +00:00
|
|
|
if (subdir_nr > 0xff)
|
|
|
|
BUG("invalid loose object subdirectory: %x", subdir_nr);
|
|
|
|
|
2017-06-24 12:12:30 +00:00
|
|
|
origlen = path->len;
|
|
|
|
strbuf_complete(path, '/');
|
|
|
|
strbuf_addf(path, "%02x", subdir_nr);
|
|
|
|
|
|
|
|
dir = opendir(path->buf);
|
2014-10-15 22:38:55 +00:00
|
|
|
if (!dir) {
|
2017-06-24 12:12:30 +00:00
|
|
|
if (errno != ENOENT)
|
2018-07-21 07:49:39 +00:00
|
|
|
r = error_errno(_("unable to open %s"), path->buf);
|
2017-06-24 12:12:30 +00:00
|
|
|
strbuf_setlen(path, origlen);
|
|
|
|
return r;
|
2014-10-15 22:38:55 +00:00
|
|
|
}
|
|
|
|
|
2017-10-31 13:50:06 +00:00
|
|
|
oid.hash[0] = subdir_nr;
|
2017-12-04 14:06:03 +00:00
|
|
|
strbuf_addch(path, '/');
|
|
|
|
baselen = path->len;
|
2017-10-31 13:50:06 +00:00
|
|
|
|
2014-10-15 22:38:55 +00:00
|
|
|
while ((de = readdir(dir))) {
|
2017-12-04 14:06:03 +00:00
|
|
|
size_t namelen;
|
2014-10-15 22:38:55 +00:00
|
|
|
if (is_dot_or_dotdot(de->d_name))
|
|
|
|
continue;
|
|
|
|
|
2017-12-04 14:06:03 +00:00
|
|
|
namelen = strlen(de->d_name);
|
2014-10-15 22:38:55 +00:00
|
|
|
strbuf_setlen(path, baselen);
|
2017-12-04 14:06:03 +00:00
|
|
|
strbuf_add(path, de->d_name, namelen);
|
2018-07-16 01:28:07 +00:00
|
|
|
if (namelen == the_hash_algo->hexsz - 2 &&
|
2017-10-31 13:50:06 +00:00
|
|
|
!hex_to_bytes(oid.hash + 1, de->d_name,
|
2018-07-16 01:28:07 +00:00
|
|
|
the_hash_algo->rawsz - 1)) {
|
2017-10-31 13:50:06 +00:00
|
|
|
if (obj_cb) {
|
|
|
|
r = obj_cb(&oid, path->buf, data);
|
|
|
|
if (r)
|
|
|
|
break;
|
2014-10-15 22:38:55 +00:00
|
|
|
}
|
2017-10-31 13:50:06 +00:00
|
|
|
continue;
|
2014-10-15 22:38:55 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
if (cruft_cb) {
|
|
|
|
r = cruft_cb(de->d_name, path->buf, data);
|
|
|
|
if (r)
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
2015-08-12 17:43:01 +00:00
|
|
|
closedir(dir);
|
2014-10-15 22:38:55 +00:00
|
|
|
|
2017-12-04 14:06:03 +00:00
|
|
|
strbuf_setlen(path, baselen - 1);
|
2014-10-15 22:38:55 +00:00
|
|
|
if (!r && subdir_cb)
|
|
|
|
r = subdir_cb(subdir_nr, path->buf, data);
|
|
|
|
|
2017-06-24 12:12:30 +00:00
|
|
|
strbuf_setlen(path, origlen);
|
|
|
|
|
2014-10-15 22:38:55 +00:00
|
|
|
return r;
|
|
|
|
}
|
|
|
|
|
2015-02-09 01:13:22 +00:00
|
|
|
int for_each_loose_file_in_objdir_buf(struct strbuf *path,
|
2014-10-15 22:38:55 +00:00
|
|
|
each_loose_object_fn obj_cb,
|
|
|
|
each_loose_cruft_fn cruft_cb,
|
|
|
|
each_loose_subdir_fn subdir_cb,
|
|
|
|
void *data)
|
|
|
|
{
|
|
|
|
int r = 0;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = 0; i < 256; i++) {
|
2015-02-09 01:13:22 +00:00
|
|
|
r = for_each_file_in_obj_subdir(i, path, obj_cb, cruft_cb,
|
2014-10-15 22:38:55 +00:00
|
|
|
subdir_cb, data);
|
|
|
|
if (r)
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
2015-02-09 01:13:22 +00:00
|
|
|
return r;
|
|
|
|
}
|
|
|
|
|
|
|
|
int for_each_loose_file_in_objdir(const char *path,
|
|
|
|
each_loose_object_fn obj_cb,
|
|
|
|
each_loose_cruft_fn cruft_cb,
|
|
|
|
each_loose_subdir_fn subdir_cb,
|
|
|
|
void *data)
|
|
|
|
{
|
|
|
|
struct strbuf buf = STRBUF_INIT;
|
|
|
|
int r;
|
|
|
|
|
|
|
|
strbuf_addstr(&buf, path);
|
|
|
|
r = for_each_loose_file_in_objdir_buf(&buf, obj_cb, cruft_cb,
|
|
|
|
subdir_cb, data);
|
2014-10-15 22:38:55 +00:00
|
|
|
strbuf_release(&buf);
|
2015-02-09 01:13:22 +00:00
|
|
|
|
2014-10-15 22:38:55 +00:00
|
|
|
return r;
|
|
|
|
}
|
2014-10-15 22:41:21 +00:00
|
|
|
|
|
|
|
struct loose_alt_odb_data {
|
|
|
|
each_loose_object_fn *cb;
|
|
|
|
void *data;
|
|
|
|
};
|
|
|
|
|
|
|
|
static int loose_from_alt_odb(struct alternate_object_database *alt,
|
|
|
|
void *vdata)
|
|
|
|
{
|
|
|
|
struct loose_alt_odb_data *data = vdata;
|
2015-02-09 01:15:39 +00:00
|
|
|
struct strbuf buf = STRBUF_INIT;
|
|
|
|
int r;
|
|
|
|
|
2016-10-03 20:35:51 +00:00
|
|
|
strbuf_addstr(&buf, alt->path);
|
2015-02-09 01:15:39 +00:00
|
|
|
r = for_each_loose_file_in_objdir_buf(&buf,
|
|
|
|
data->cb, NULL, NULL,
|
|
|
|
data->data);
|
|
|
|
strbuf_release(&buf);
|
|
|
|
return r;
|
2014-10-15 22:41:21 +00:00
|
|
|
}
|
|
|
|
|
2018-08-10 23:09:44 +00:00
|
|
|
int for_each_loose_object(each_loose_object_fn cb, void *data,
|
|
|
|
enum for_each_object_flags flags)
|
2014-10-15 22:41:21 +00:00
|
|
|
{
|
|
|
|
struct loose_alt_odb_data alt;
|
|
|
|
int r;
|
|
|
|
|
|
|
|
r = for_each_loose_file_in_objdir(get_object_directory(),
|
|
|
|
cb, NULL, NULL, data);
|
|
|
|
if (r)
|
|
|
|
return r;
|
|
|
|
|
reachable: only mark local objects as recent
When pruning and repacking a repository that has an
alternate object store configured, we may traverse a large
number of objects in the alternate. This serves no purpose,
and may be expensive to do. A longer explanation is below.
Commits d3038d2 and abcb865 taught prune and pack-objects
(respectively) to treat "recent" objects as tips for
reachability, so that we keep whole chunks of history. They
built on the object traversal in 660c889 (sha1_file: add
for_each iterators for loose and packed objects,
2014-10-15), which covers both local and alternate objects.
In both cases, covering alternate objects is unnecessary, as
both commands can only drop objects from the local
repository. In the case of prune, we traverse only the local
object directory. And in the case of repacking, while we may
or may not include local objects in our pack, we will never
reach into the alternate with "repack -d". The "-l" option
is only a question of whether we are migrating objects from
the alternate into our repository, or leaving them
untouched.
It is possible that we may drop an object that is depended
upon by another object in the alternate. For example,
imagine two repositories, A and B, with A pointing to B as
an alternate. Now imagine a commit that is in B which
references a tree that is only in A. Traversing from recent
objects in B might prevent A from dropping that tree. But
this case isn't worth covering. Repo B should take
responsibility for its own objects. It would never have had
the commit in the first place if it did not also have the
tree, and assuming it is using the same "keep recent chunks
of history" scheme, then it would itself keep the tree, as
well.
So checking the alternate objects is not worth doing, and
come with a significant performance impact. In both cases,
we skip any recent objects that have already been marked
SEEN (i.e., that we know are already reachable for prune, or
included in the pack for a repack). So there is a slight
waste of time in opening the alternate packs at all, only to
notice that we have already considered each object. But much
worse, the alternate repository may have a large number of
objects that are not reachable from the local repository at
all, and we end up adding them to the traversal.
We can fix this by considering only local unseen objects.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-03-27 11:32:41 +00:00
|
|
|
if (flags & FOR_EACH_OBJECT_LOCAL_ONLY)
|
|
|
|
return 0;
|
|
|
|
|
2014-10-15 22:41:21 +00:00
|
|
|
alt.cb = cb;
|
|
|
|
alt.data = data;
|
|
|
|
return foreach_alt_odb(loose_from_alt_odb, &alt);
|
|
|
|
}
|
|
|
|
|
2017-01-13 17:58:16 +00:00
|
|
|
static int check_stream_sha1(git_zstream *stream,
|
|
|
|
const char *hdr,
|
|
|
|
unsigned long size,
|
|
|
|
const char *path,
|
|
|
|
const unsigned char *expected_sha1)
|
|
|
|
{
|
2018-02-01 02:18:41 +00:00
|
|
|
git_hash_ctx c;
|
2017-03-26 16:01:25 +00:00
|
|
|
unsigned char real_sha1[GIT_MAX_RAWSZ];
|
2017-01-13 17:58:16 +00:00
|
|
|
unsigned char buf[4096];
|
|
|
|
unsigned long total_read;
|
|
|
|
int status = Z_OK;
|
|
|
|
|
2018-02-01 02:18:41 +00:00
|
|
|
the_hash_algo->init_fn(&c);
|
|
|
|
the_hash_algo->update_fn(&c, hdr, stream->total_out);
|
2017-01-13 17:58:16 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* We already read some bytes into hdr, but the ones up to the NUL
|
|
|
|
* do not count against the object's content size.
|
|
|
|
*/
|
|
|
|
total_read = stream->total_out - strlen(hdr) - 1;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This size comparison must be "<=" to read the final zlib packets;
|
|
|
|
* see the comment in unpack_sha1_rest for details.
|
|
|
|
*/
|
|
|
|
while (total_read <= size &&
|
|
|
|
(status == Z_OK || status == Z_BUF_ERROR)) {
|
|
|
|
stream->next_out = buf;
|
|
|
|
stream->avail_out = sizeof(buf);
|
|
|
|
if (size - total_read < stream->avail_out)
|
|
|
|
stream->avail_out = size - total_read;
|
|
|
|
status = git_inflate(stream, Z_FINISH);
|
2018-02-01 02:18:41 +00:00
|
|
|
the_hash_algo->update_fn(&c, buf, stream->next_out - buf);
|
2017-01-13 17:58:16 +00:00
|
|
|
total_read += stream->next_out - buf;
|
|
|
|
}
|
|
|
|
git_inflate_end(stream);
|
|
|
|
|
|
|
|
if (status != Z_STREAM_END) {
|
2018-07-21 07:49:39 +00:00
|
|
|
error(_("corrupt loose object '%s'"), sha1_to_hex(expected_sha1));
|
2017-01-13 17:58:16 +00:00
|
|
|
return -1;
|
|
|
|
}
|
2017-01-13 18:00:25 +00:00
|
|
|
if (stream->avail_in) {
|
2018-07-21 07:49:39 +00:00
|
|
|
error(_("garbage at end of loose object '%s'"),
|
2017-01-13 18:00:25 +00:00
|
|
|
sha1_to_hex(expected_sha1));
|
|
|
|
return -1;
|
|
|
|
}
|
2017-01-13 17:58:16 +00:00
|
|
|
|
2018-02-01 02:18:41 +00:00
|
|
|
the_hash_algo->final_fn(real_sha1, &c);
|
2018-08-28 21:22:52 +00:00
|
|
|
if (!hasheq(expected_sha1, real_sha1)) {
|
2018-07-21 07:49:39 +00:00
|
|
|
error(_("sha1 mismatch for %s (expected %s)"), path,
|
2017-01-13 17:58:16 +00:00
|
|
|
sha1_to_hex(expected_sha1));
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
int read_loose_object(const char *path,
|
2018-03-12 02:27:38 +00:00
|
|
|
const struct object_id *expected_oid,
|
2017-01-13 17:58:16 +00:00
|
|
|
enum object_type *type,
|
|
|
|
unsigned long *size,
|
|
|
|
void **contents)
|
|
|
|
{
|
|
|
|
int ret = -1;
|
|
|
|
void *map = NULL;
|
|
|
|
unsigned long mapsize;
|
|
|
|
git_zstream stream;
|
2018-03-12 02:27:55 +00:00
|
|
|
char hdr[MAX_HEADER_LEN];
|
2017-01-13 17:58:16 +00:00
|
|
|
|
|
|
|
*contents = NULL;
|
|
|
|
|
2018-03-23 17:21:13 +00:00
|
|
|
map = map_sha1_file_1(the_repository, path, NULL, &mapsize);
|
2017-01-13 17:58:16 +00:00
|
|
|
if (!map) {
|
2018-07-21 07:49:39 +00:00
|
|
|
error_errno(_("unable to mmap %s"), path);
|
2017-01-13 17:58:16 +00:00
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (unpack_sha1_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0) {
|
2018-07-21 07:49:39 +00:00
|
|
|
error(_("unable to unpack header of %s"), path);
|
2017-01-13 17:58:16 +00:00
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
*type = parse_sha1_header(hdr, size);
|
|
|
|
if (*type < 0) {
|
2018-07-21 07:49:39 +00:00
|
|
|
error(_("unable to parse header of %s"), path);
|
2017-01-13 17:58:16 +00:00
|
|
|
git_inflate_end(&stream);
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
fsck: actually fsck blob data
Because fscking a blob has always been a noop, we didn't
bother passing around the blob data. In preparation for
content-level checks, let's fix up a few things:
1. The fsck_object() function just returns success for any
blob. Let's a noop fsck_blob(), which we can fill in
with actual logic later.
2. The fsck_loose() function in builtin/fsck.c
just threw away blob content after loading it. Let's
hold onto it until after we've called fsck_object().
The easiest way to do this is to just drop the
parse_loose_object() helper entirely. Incidentally,
this also fixes a memory leak: if we successfully
loaded the object data but did not parse it, we would
have left the function without freeing it.
3. When fsck_loose() loads the object data, it
does so with a custom read_loose_object() helper. This
function streams any blobs, regardless of size, under
the assumption that we're only checking the sha1.
Instead, let's actually load blobs smaller than
big_file_threshold, as the normal object-reading
code-paths would do. This lets us fsck small files, and
a NULL return is an indication that the blob was so big
that it needed to be streamed, and we can pass that
information along to fsck_blob().
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 19:44:51 +00:00
|
|
|
if (*type == OBJ_BLOB && *size > big_file_threshold) {
|
2018-03-12 02:27:38 +00:00
|
|
|
if (check_stream_sha1(&stream, hdr, *size, path, expected_oid->hash) < 0)
|
2017-01-13 17:58:16 +00:00
|
|
|
goto out;
|
|
|
|
} else {
|
2018-03-12 02:27:38 +00:00
|
|
|
*contents = unpack_sha1_rest(&stream, hdr, *size, expected_oid->hash);
|
2017-01-13 17:58:16 +00:00
|
|
|
if (!*contents) {
|
2018-07-21 07:49:39 +00:00
|
|
|
error(_("unable to unpack contents of %s"), path);
|
2017-01-13 17:58:16 +00:00
|
|
|
git_inflate_end(&stream);
|
|
|
|
goto out;
|
|
|
|
}
|
2018-03-12 02:27:39 +00:00
|
|
|
if (check_object_signature(expected_oid, *contents,
|
2018-02-14 18:59:24 +00:00
|
|
|
*size, type_name(*type))) {
|
2018-07-21 07:49:39 +00:00
|
|
|
error(_("sha1 mismatch for %s (expected %s)"), path,
|
2018-03-12 02:27:38 +00:00
|
|
|
oid_to_hex(expected_oid));
|
2017-01-13 17:58:16 +00:00
|
|
|
free(*contents);
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
ret = 0; /* everything checks out */
|
|
|
|
|
|
|
|
out:
|
|
|
|
if (map)
|
|
|
|
munmap(map, mapsize);
|
|
|
|
return ret;
|
|
|
|
}
|