2005-04-18 20:04:43 +00:00
|
|
|
/*
|
|
|
|
* GIT - The information manager from hell
|
|
|
|
*
|
|
|
|
* Copyright (C) Linus Torvalds, 2005
|
|
|
|
*
|
|
|
|
* This handles basic git sha1 object files - packing, unpacking,
|
|
|
|
* creation etc.
|
|
|
|
*/
|
|
|
|
#include "cache.h"
|
2012-11-05 08:41:22 +00:00
|
|
|
#include "string-list.h"
|
2014-10-01 10:28:42 +00:00
|
|
|
#include "lockfile.h"
|
2005-06-27 10:35:33 +00:00
|
|
|
#include "delta.h"
|
2005-06-28 21:21:02 +00:00
|
|
|
#include "pack.h"
|
2006-04-02 12:44:09 +00:00
|
|
|
#include "blob.h"
|
|
|
|
#include "commit.h"
|
2011-05-08 08:47:35 +00:00
|
|
|
#include "run-command.h"
|
2006-04-02 12:44:09 +00:00
|
|
|
#include "tag.h"
|
|
|
|
#include "tree.h"
|
2011-02-05 10:52:21 +00:00
|
|
|
#include "tree-walk.h"
|
2007-04-10 04:20:29 +00:00
|
|
|
#include "refs.h"
|
2008-02-28 05:25:19 +00:00
|
|
|
#include "pack-revindex.h"
|
sha1-lookup: more memory efficient search in sorted list of SHA-1
Currently, when looking for a packed object from the pack idx, a
simple binary search is used.
A conventional binary search loop looks like this:
unsigned lo, hi;
do {
unsigned mi = (lo + hi) / 2;
int cmp = "entry pointed at by mi" minus "target";
if (!cmp)
return mi; "mi is the wanted one"
if (cmp > 0)
hi = mi; "mi is larger than target"
else
lo = mi+1; "mi is smaller than target"
} while (lo < hi);
"did not find what we wanted"
The invariants are:
- When entering the loop, 'lo' points at a slot that is never
above the target (it could be at the target), 'hi' points at
a slot that is guaranteed to be above the target (it can
never be at the target).
- We find a point 'mi' between 'lo' and 'hi' ('mi' could be
the same as 'lo', but never can be as high as 'hi'), and
check if 'mi' hits the target. There are three cases:
- if it is a hit, we have found what we are looking for;
- if it is strictly higher than the target, we set it to
'hi', and repeat the search.
- if it is strictly lower than the target, we update 'lo'
to one slot after it, because we allow 'lo' to be at the
target and 'mi' is known to be below the target.
If the loop exits, there is no matching entry.
When choosing 'mi', we do not have to take the "middle" but
anywhere in between 'lo' and 'hi', as long as lo <= mi < hi is
satisfied. When we somehow know that the distance between the
target and 'lo' is much shorter than the target and 'hi', we
could pick 'mi' that is much closer to 'lo' than (hi+lo)/2,
which a conventional binary search would pick.
This patch takes advantage of the fact that the SHA-1 is a good
hash function, and as long as there are enough entries in the
table, we can expect uniform distribution. An entry that begins
with for example "deadbeef..." is much likely to appear much
later than in the midway of a reasonably populated table. In
fact, it can be expected to be near 87% (222/256) from the top
of the table.
This is a work-in-progress and has switches to allow easier
experiments and debugging. Exporting GIT_USE_LOOKUP environment
variable enables this code.
On my admittedly memory starved machine, with a partial KDE
repository (3.0G pack with 95M idx):
$ GIT_USE_LOOKUP=t git log -800 --stat HEAD >/dev/null
3.93user 0.16system 0:04.09elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+55588minor)pagefaults 0swaps
Without the patch, the numbers are:
$ git log -800 --stat HEAD >/dev/null
4.00user 0.15system 0:04.17elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+60258minor)pagefaults 0swaps
In the same repository:
$ GIT_USE_LOOKUP=t git log -2000 HEAD >/dev/null
0.12user 0.00system 0:00.12elapsed 97%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+4241minor)pagefaults 0swaps
Without the patch, the numbers are:
$ git log -2000 HEAD >/dev/null
0.05user 0.01system 0:00.07elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+8506minor)pagefaults 0swaps
There isn't much time difference, but the number of minor faults
seems to show that we are touching much smaller number of pages,
which is expected.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-12-29 10:05:47 +00:00
|
|
|
#include "sha1-lookup.h"
|
2011-10-28 21:48:40 +00:00
|
|
|
#include "bulk-checkin.h"
|
2012-03-07 10:54:18 +00:00
|
|
|
#include "streaming.h"
|
2013-02-15 12:07:10 +00:00
|
|
|
#include "dir.h"
|
find_pack_entry: replace last_found_pack with MRU cache
Each pack has an index for looking up entries in O(log n)
time, but if we have multiple packs, we have to scan through
them linearly. This can produce a measurable overhead for
some operations.
We dealt with this long ago in f7c22cc (always start looking
up objects in the last used pack first, 2007-05-30), which
keeps what is essentially a 1-element most-recently-used
cache. In theory, we should be able to do better by keeping
a similar but longer cache, that is the same length as the
pack-list itself.
Since we now have a convenient generic MRU structure, we can
plug it in and measure. Here are the numbers for running
p5303 against linux.git:
Test HEAD^ HEAD
------------------------------------------------------------------------
5303.3: rev-list (1) 31.56(31.28+0.27) 31.30(31.08+0.20) -0.8%
5303.4: repack (1) 40.62(39.35+2.36) 40.60(39.27+2.44) -0.0%
5303.6: rev-list (50) 31.31(31.06+0.23) 31.23(31.00+0.22) -0.3%
5303.7: repack (50) 58.65(69.12+1.94) 58.27(68.64+2.05) -0.6%
5303.9: rev-list (1000) 38.74(38.40+0.33) 31.87(31.62+0.24) -17.7%
5303.10: repack (1000) 367.20(441.80+4.62) 342.00(414.04+3.72) -6.9%
The main numbers of interest here are the rev-list ones
(since that is exercising the normal object lookup code
path). The single-pack case shouldn't improve at all; the
260ms speedup there is just part of the run-to-run noise
(but it's important to note that we didn't make anything
worse with the overhead of maintaining our cache). In the
50-pack case, we see similar results. There may be a slight
improvement, but it's mostly within the noise.
The 1000-pack case does show a big improvement, though. That
carries over to the repack case, as well. Even though we
haven't touched its pack-search loop yet, it does still do a
lot of normal object lookups (e.g., for the internal
revision walk), and so improves.
As a point of reference, I also ran the 1000-pack test
against a version of HEAD^ with the last_found_pack
optimization disabled. It takes ~60s, so that gives an
indication of how much even the single-element cache is
helping.
For comparison, here's a smaller repository, git.git:
Test HEAD^ HEAD
---------------------------------------------------------------------
5303.3: rev-list (1) 1.56(1.54+0.01) 1.54(1.51+0.02) -1.3%
5303.4: repack (1) 1.84(1.80+0.10) 1.82(1.80+0.09) -1.1%
5303.6: rev-list (50) 1.58(1.55+0.02) 1.59(1.57+0.01) +0.6%
5303.7: repack (50) 2.50(3.18+0.04) 2.50(3.14+0.04) +0.0%
5303.9: rev-list (1000) 2.76(2.71+0.04) 2.24(2.21+0.02) -18.8%
5303.10: repack (1000) 13.21(19.56+0.25) 11.66(18.01+0.21) -11.7%
You can see that the percentage improvement is similar.
That's because the lookup we are optimizing is roughly
O(nr_objects * nr_packs). Since the number of packs is
constant in both tests, we'd expect the improvement to be
linear in the number of objects. But the whole process is
also linear in the number of objects, so the improvement
is a constant factor.
The exact improvement does also depend on the contents of
the packs. In p5303, the extra packs all have 5 first-parent
commits in them, which is a reasonable simulation of a
pushed-to repository. But it also means that only 250
first-parent commits are in those packs (compared to almost
50,000 total in linux.git), and the rest are in the huge
"base" pack. So once we start looking at history in taht big
pack, that's where we'll find most everything, and even the
1-element cache gets close to 100% cache hits. You could
almost certainly show better numbers with a more
pathological case (e.g., distributing the objects more
evenly across the packs). But that's simply not that
realistic a scenario, so it makes more sense to focus on
these numbers.
The implementation itself is a straightforward application
of the MRU code. We provide an MRU-ordered list of packs
that shadows the packed_git list. This is easy to do because
we only create and revise the pack list in one place. The
"reprepare" code path actually drops the whole MRU and
replaces it for simplicity. It would be more efficient to
just add new entries, but there's not much point in
optimizing here; repreparing happens rarely, and only after
doing a lot of other expensive work. The key things to keep
optimized are traversal (which is just a normal linked list,
albeit with one extra level of indirection over the regular
packed_git list), and marking (which is a constant number of
pointer assignments, though slightly more than the old
last_found_pack was; it doesn't seem to create a measurable
slowdown, though).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-07-29 04:09:46 +00:00
|
|
|
#include "mru.h"
|
2016-08-22 21:59:42 +00:00
|
|
|
#include "list.h"
|
2016-09-13 17:54:42 +00:00
|
|
|
#include "mergesort.h"
|
alternates: accept double-quoted paths
We read lists of alternates from objects/info/alternates
files (delimited by newline), as well as from the
GIT_ALTERNATE_OBJECT_DIRECTORIES environment variable
(delimited by colon or semi-colon, depending on the
platform).
There's no mechanism for quoting the delimiters, so it's
impossible to specify an alternate path that contains a
colon in the environment, or one that contains a newline in
a file. We've lived with that restriction for ages because
both alternates and filenames with colons are relatively
rare, and it's only a problem when the two meet. But since
722ff7f87 (receive-pack: quarantine objects until
pre-receive accepts, 2016-10-03), which builds on the
alternates system, every push causes the receiver to set
GIT_ALTERNATE_OBJECT_DIRECTORIES internally.
It would be convenient to have some way to quote the
delimiter so that we can represent arbitrary paths.
The simplest thing would be an escape character before a
quoted delimiter (e.g., "\:" as a literal colon). But that
creates a backwards compatibility problem: any path which
uses that escape character is now broken, and we've just
shifted the problem. We could choose an unlikely escape
character (e.g., something from the non-printable ASCII
range), but that's awkward to use.
Instead, let's treat names as unquoted unless they begin
with a double-quote, in which case they are interpreted via
our usual C-stylke quoting rules. This also breaks
backwards-compatibility, but in a smaller way: it only
matters if your file has a double-quote as the very _first_
character in the path (whereas an escape character is a
problem anywhere in the path). It's also consistent with
many other parts of git, which accept either a bare pathname
or a double-quoted one, and the sender can choose to quote
or not as required.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-12-12 19:52:22 +00:00
|
|
|
#include "quote.h"
|
2005-04-18 20:04:43 +00:00
|
|
|
|
2011-03-16 05:15:31 +00:00
|
|
|
#define SZ_FMT PRIuMAX
|
|
|
|
static inline uintmax_t sz_fmt(size_t s) { return s; }
|
2007-01-10 04:07:11 +00:00
|
|
|
|
2006-08-15 17:23:48 +00:00
|
|
|
const unsigned char null_sha1[20];
|
2015-12-06 22:16:35 +00:00
|
|
|
const struct object_id null_oid;
|
2016-08-31 23:27:18 +00:00
|
|
|
const struct object_id empty_tree_oid = {
|
|
|
|
EMPTY_TREE_SHA1_BIN_LITERAL
|
|
|
|
};
|
|
|
|
const struct object_id empty_blob_oid = {
|
|
|
|
EMPTY_BLOB_SHA1_BIN_LITERAL
|
|
|
|
};
|
2005-09-30 21:02:47 +00:00
|
|
|
|
2011-02-05 14:03:01 +00:00
|
|
|
/*
|
|
|
|
* This is meant to hold a *small* number of objects that you would
|
|
|
|
* want read_sha1_file() to be able to return, but yet you do not want
|
|
|
|
* to write them into the object store (e.g. a browse-only
|
|
|
|
* application).
|
|
|
|
*/
|
|
|
|
static struct cached_object {
|
|
|
|
unsigned char sha1[20];
|
|
|
|
enum object_type type;
|
|
|
|
void *buf;
|
|
|
|
unsigned long size;
|
|
|
|
} *cached_objects;
|
|
|
|
static int cached_object_nr, cached_object_alloc;
|
|
|
|
|
|
|
|
static struct cached_object empty_tree = {
|
2011-02-07 08:17:27 +00:00
|
|
|
EMPTY_TREE_SHA1_BIN_LITERAL,
|
2011-02-05 14:03:01 +00:00
|
|
|
OBJ_TREE,
|
|
|
|
"",
|
|
|
|
0
|
|
|
|
};
|
|
|
|
|
|
|
|
static struct cached_object *find_cached_object(const unsigned char *sha1)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
struct cached_object *co = cached_objects;
|
|
|
|
|
|
|
|
for (i = 0; i < cached_object_nr; i++, co++) {
|
|
|
|
if (!hashcmp(co->sha1, sha1))
|
|
|
|
return co;
|
|
|
|
}
|
|
|
|
if (!hashcmp(sha1, empty_tree.sha1))
|
|
|
|
return &empty_tree;
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
2011-03-11 00:02:50 +00:00
|
|
|
int mkdir_in_gitdir(const char *path)
|
|
|
|
{
|
|
|
|
if (mkdir(path, 0777)) {
|
|
|
|
int saved_errno = errno;
|
|
|
|
struct stat st;
|
|
|
|
struct strbuf sb = STRBUF_INIT;
|
|
|
|
|
|
|
|
if (errno != EEXIST)
|
|
|
|
return -1;
|
|
|
|
/*
|
|
|
|
* Are we looking at a path in a symlinked worktree
|
|
|
|
* whose original repository does not yet have it?
|
|
|
|
* e.g. .git/rr-cache pointing at its original
|
|
|
|
* repository in which the user hasn't performed any
|
|
|
|
* conflict resolution yet?
|
|
|
|
*/
|
|
|
|
if (lstat(path, &st) || !S_ISLNK(st.st_mode) ||
|
|
|
|
strbuf_readlink(&sb, path, st.st_size) ||
|
|
|
|
!is_absolute_path(sb.buf) ||
|
|
|
|
mkdir(sb.buf, 0777)) {
|
|
|
|
strbuf_release(&sb);
|
|
|
|
errno = saved_errno;
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
strbuf_release(&sb);
|
|
|
|
}
|
|
|
|
return adjust_shared_perm(path);
|
|
|
|
}
|
|
|
|
|
2014-01-06 13:45:25 +00:00
|
|
|
enum scld_error safe_create_leading_directories(char *path)
|
2005-07-06 08:11:52 +00:00
|
|
|
{
|
2014-01-06 13:45:22 +00:00
|
|
|
char *next_component = path + offset_1st_component(path);
|
2014-01-06 13:45:25 +00:00
|
|
|
enum scld_error ret = SCLD_OK;
|
2006-02-10 01:56:13 +00:00
|
|
|
|
2014-01-06 13:45:25 +00:00
|
|
|
while (ret == SCLD_OK && next_component) {
|
2014-01-06 13:45:20 +00:00
|
|
|
struct stat st;
|
2014-01-18 23:40:44 +00:00
|
|
|
char *slash = next_component, slash_character;
|
2014-01-06 13:45:20 +00:00
|
|
|
|
2014-01-18 23:40:44 +00:00
|
|
|
while (*slash && !is_dir_sep(*slash))
|
|
|
|
slash++;
|
|
|
|
|
|
|
|
if (!*slash)
|
2005-07-06 08:11:52 +00:00
|
|
|
break;
|
2014-01-06 13:45:23 +00:00
|
|
|
|
2014-01-06 13:45:22 +00:00
|
|
|
next_component = slash + 1;
|
2014-01-18 23:40:44 +00:00
|
|
|
while (is_dir_sep(*next_component))
|
2014-01-06 13:45:23 +00:00
|
|
|
next_component++;
|
2014-01-06 13:45:22 +00:00
|
|
|
if (!*next_component)
|
2008-09-02 21:10:15 +00:00
|
|
|
break;
|
2014-01-06 13:45:21 +00:00
|
|
|
|
2014-01-18 23:40:44 +00:00
|
|
|
slash_character = *slash;
|
2014-01-06 13:45:21 +00:00
|
|
|
*slash = '\0';
|
2006-02-10 01:56:13 +00:00
|
|
|
if (!stat(path, &st)) {
|
|
|
|
/* path exists */
|
2017-01-06 16:22:25 +00:00
|
|
|
if (!S_ISDIR(st.st_mode)) {
|
|
|
|
errno = ENOTDIR;
|
2014-01-06 13:45:25 +00:00
|
|
|
ret = SCLD_EXISTS;
|
2017-01-06 16:22:25 +00:00
|
|
|
}
|
2014-01-06 13:45:19 +00:00
|
|
|
} else if (mkdir(path, 0777)) {
|
2013-03-17 14:09:27 +00:00
|
|
|
if (errno == EEXIST &&
|
2014-01-06 13:45:24 +00:00
|
|
|
!stat(path, &st) && S_ISDIR(st.st_mode))
|
2013-03-17 14:09:27 +00:00
|
|
|
; /* somebody created it since we checked */
|
2014-01-06 13:45:27 +00:00
|
|
|
else if (errno == ENOENT)
|
|
|
|
/*
|
|
|
|
* Either mkdir() failed because
|
|
|
|
* somebody just pruned the containing
|
|
|
|
* directory, or stat() failed because
|
|
|
|
* the file that was in our way was
|
|
|
|
* just removed. Either way, inform
|
|
|
|
* the caller that it might be worth
|
|
|
|
* trying again:
|
|
|
|
*/
|
|
|
|
ret = SCLD_VANISHED;
|
2014-01-06 13:45:24 +00:00
|
|
|
else
|
2014-01-06 13:45:25 +00:00
|
|
|
ret = SCLD_FAILED;
|
2014-01-06 13:45:19 +00:00
|
|
|
} else if (adjust_shared_perm(path)) {
|
2014-01-06 13:45:25 +00:00
|
|
|
ret = SCLD_PERMS;
|
2005-12-22 22:13:56 +00:00
|
|
|
}
|
2014-01-18 23:40:44 +00:00
|
|
|
*slash = slash_character;
|
2005-07-06 08:11:52 +00:00
|
|
|
}
|
2014-01-06 13:45:24 +00:00
|
|
|
return ret;
|
2005-07-06 08:11:52 +00:00
|
|
|
}
|
2005-07-05 18:31:32 +00:00
|
|
|
|
2014-01-06 13:45:25 +00:00
|
|
|
enum scld_error safe_create_leading_directories_const(const char *path)
|
2008-06-25 05:41:34 +00:00
|
|
|
{
|
2017-01-06 16:22:24 +00:00
|
|
|
int save_errno;
|
2008-06-25 05:41:34 +00:00
|
|
|
/* path points to cache entries, so xstrdup before messing with it */
|
|
|
|
char *buf = xstrdup(path);
|
2014-01-06 13:45:25 +00:00
|
|
|
enum scld_error result = safe_create_leading_directories(buf);
|
2017-01-06 16:22:24 +00:00
|
|
|
|
|
|
|
save_errno = errno;
|
2008-06-25 05:41:34 +00:00
|
|
|
free(buf);
|
2017-01-06 16:22:24 +00:00
|
|
|
errno = save_errno;
|
2008-06-25 05:41:34 +00:00
|
|
|
return result;
|
|
|
|
}
|
|
|
|
|
2017-01-06 16:22:26 +00:00
|
|
|
int raceproof_create_file(const char *path, create_file_fn fn, void *cb)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* The number of times we will try to remove empty directories
|
|
|
|
* in the way of path. This is only 1 because if another
|
|
|
|
* process is racily creating directories that conflict with
|
|
|
|
* us, we don't want to fight against them.
|
|
|
|
*/
|
|
|
|
int remove_directories_remaining = 1;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* The number of times that we will try to create the
|
|
|
|
* directories containing path. We are willing to attempt this
|
|
|
|
* more than once, because another process could be trying to
|
|
|
|
* clean up empty directories at the same time as we are
|
|
|
|
* trying to create them.
|
|
|
|
*/
|
|
|
|
int create_directories_remaining = 3;
|
|
|
|
|
|
|
|
/* A scratch copy of path, filled lazily if we need it: */
|
|
|
|
struct strbuf path_copy = STRBUF_INIT;
|
|
|
|
|
|
|
|
int ret, save_errno;
|
|
|
|
|
|
|
|
/* Sanity check: */
|
|
|
|
assert(*path);
|
|
|
|
|
|
|
|
retry_fn:
|
|
|
|
ret = fn(path, cb);
|
|
|
|
save_errno = errno;
|
|
|
|
if (!ret)
|
|
|
|
goto out;
|
|
|
|
|
|
|
|
if (errno == EISDIR && remove_directories_remaining-- > 0) {
|
|
|
|
/*
|
|
|
|
* A directory is in the way. Maybe it is empty; try
|
|
|
|
* to remove it:
|
|
|
|
*/
|
|
|
|
if (!path_copy.len)
|
|
|
|
strbuf_addstr(&path_copy, path);
|
|
|
|
|
|
|
|
if (!remove_dir_recursively(&path_copy, REMOVE_DIR_EMPTY_ONLY))
|
|
|
|
goto retry_fn;
|
|
|
|
} else if (errno == ENOENT && create_directories_remaining-- > 0) {
|
|
|
|
/*
|
|
|
|
* Maybe the containing directory didn't exist, or
|
|
|
|
* maybe it was just deleted by a process that is
|
|
|
|
* racing with us to clean up empty directories. Try
|
|
|
|
* to create it:
|
|
|
|
*/
|
|
|
|
enum scld_error scld_result;
|
|
|
|
|
|
|
|
if (!path_copy.len)
|
|
|
|
strbuf_addstr(&path_copy, path);
|
|
|
|
|
|
|
|
do {
|
|
|
|
scld_result = safe_create_leading_directories(path_copy.buf);
|
|
|
|
if (scld_result == SCLD_OK)
|
|
|
|
goto retry_fn;
|
|
|
|
} while (scld_result == SCLD_VANISHED && create_directories_remaining-- > 0);
|
|
|
|
}
|
|
|
|
|
|
|
|
out:
|
|
|
|
strbuf_release(&path_copy);
|
|
|
|
errno = save_errno;
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
fill_sha1_file: write into a strbuf
It's currently the responsibility of the caller to give
fill_sha1_file() enough bytes to write into, leading them to
manually compute the required lengths. Instead, let's just
write into a strbuf so that it's impossible to get this
wrong.
The alt_odb caller already has a strbuf, so this makes
things strictly simpler. The other caller, sha1_file_name(),
uses a static PATH_MAX buffer and dies when it would
overflow. We can convert this to a static strbuf, which
means our allocation cost is amortized (and as a bonus, we
no longer have to worry about PATH_MAX being too short for
normal use).
This does introduce some small overhead in fill_sha1_file(),
as each strbuf_addchar() will check whether it needs to
grow. However, between the optimization in fec501d
(strbuf_addch: avoid calling strbuf_grow, 2015-04-16) and
the fact that this is not generally called in a tight loop
(after all, the next step is typically to access the file!)
this probably doesn't matter. And even if it did, the right
place to micro-optimize is inside fill_sha1_file(), by
calling a single strbuf_grow() there.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:36:09 +00:00
|
|
|
static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)
|
2005-05-07 07:38:04 +00:00
|
|
|
{
|
|
|
|
int i;
|
|
|
|
for (i = 0; i < 20; i++) {
|
|
|
|
static char hex[] = "0123456789abcdef";
|
|
|
|
unsigned int val = sha1[i];
|
fill_sha1_file: write into a strbuf
It's currently the responsibility of the caller to give
fill_sha1_file() enough bytes to write into, leading them to
manually compute the required lengths. Instead, let's just
write into a strbuf so that it's impossible to get this
wrong.
The alt_odb caller already has a strbuf, so this makes
things strictly simpler. The other caller, sha1_file_name(),
uses a static PATH_MAX buffer and dies when it would
overflow. We can convert this to a static strbuf, which
means our allocation cost is amortized (and as a bonus, we
no longer have to worry about PATH_MAX being too short for
normal use).
This does introduce some small overhead in fill_sha1_file(),
as each strbuf_addchar() will check whether it needs to
grow. However, between the optimization in fec501d
(strbuf_addch: avoid calling strbuf_grow, 2015-04-16) and
the fact that this is not generally called in a tight loop
(after all, the next step is typically to access the file!)
this probably doesn't matter. And even if it did, the right
place to micro-optimize is inside fill_sha1_file(), by
calling a single strbuf_grow() there.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:36:09 +00:00
|
|
|
strbuf_addch(buf, hex[val >> 4]);
|
|
|
|
strbuf_addch(buf, hex[val & 0xf]);
|
2016-10-03 20:35:55 +00:00
|
|
|
if (!i)
|
fill_sha1_file: write into a strbuf
It's currently the responsibility of the caller to give
fill_sha1_file() enough bytes to write into, leading them to
manually compute the required lengths. Instead, let's just
write into a strbuf so that it's impossible to get this
wrong.
The alt_odb caller already has a strbuf, so this makes
things strictly simpler. The other caller, sha1_file_name(),
uses a static PATH_MAX buffer and dies when it would
overflow. We can convert this to a static strbuf, which
means our allocation cost is amortized (and as a bonus, we
no longer have to worry about PATH_MAX being too short for
normal use).
This does introduce some small overhead in fill_sha1_file(),
as each strbuf_addchar() will check whether it needs to
grow. However, between the optimization in fec501d
(strbuf_addch: avoid calling strbuf_grow, 2015-04-16) and
the fact that this is not generally called in a tight loop
(after all, the next step is typically to access the file!)
this probably doesn't matter. And even if it did, the right
place to micro-optimize is inside fill_sha1_file(), by
calling a single strbuf_grow() there.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:36:09 +00:00
|
|
|
strbuf_addch(buf, '/');
|
2005-05-07 07:38:04 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2014-02-21 16:32:05 +00:00
|
|
|
const char *sha1_file_name(const unsigned char *sha1)
|
2005-04-18 20:04:43 +00:00
|
|
|
{
|
fill_sha1_file: write into a strbuf
It's currently the responsibility of the caller to give
fill_sha1_file() enough bytes to write into, leading them to
manually compute the required lengths. Instead, let's just
write into a strbuf so that it's impossible to get this
wrong.
The alt_odb caller already has a strbuf, so this makes
things strictly simpler. The other caller, sha1_file_name(),
uses a static PATH_MAX buffer and dies when it would
overflow. We can convert this to a static strbuf, which
means our allocation cost is amortized (and as a bonus, we
no longer have to worry about PATH_MAX being too short for
normal use).
This does introduce some small overhead in fill_sha1_file(),
as each strbuf_addchar() will check whether it needs to
grow. However, between the optimization in fec501d
(strbuf_addch: avoid calling strbuf_grow, 2015-04-16) and
the fact that this is not generally called in a tight loop
(after all, the next step is typically to access the file!)
this probably doesn't matter. And even if it did, the right
place to micro-optimize is inside fill_sha1_file(), by
calling a single strbuf_grow() there.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:36:09 +00:00
|
|
|
static struct strbuf buf = STRBUF_INIT;
|
2005-04-18 20:04:43 +00:00
|
|
|
|
fill_sha1_file: write into a strbuf
It's currently the responsibility of the caller to give
fill_sha1_file() enough bytes to write into, leading them to
manually compute the required lengths. Instead, let's just
write into a strbuf so that it's impossible to get this
wrong.
The alt_odb caller already has a strbuf, so this makes
things strictly simpler. The other caller, sha1_file_name(),
uses a static PATH_MAX buffer and dies when it would
overflow. We can convert this to a static strbuf, which
means our allocation cost is amortized (and as a bonus, we
no longer have to worry about PATH_MAX being too short for
normal use).
This does introduce some small overhead in fill_sha1_file(),
as each strbuf_addchar() will check whether it needs to
grow. However, between the optimization in fec501d
(strbuf_addch: avoid calling strbuf_grow, 2015-04-16) and
the fact that this is not generally called in a tight loop
(after all, the next step is typically to access the file!)
this probably doesn't matter. And even if it did, the right
place to micro-optimize is inside fill_sha1_file(), by
calling a single strbuf_grow() there.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:36:09 +00:00
|
|
|
strbuf_reset(&buf);
|
|
|
|
strbuf_addf(&buf, "%s/", get_object_directory());
|
remove over-eager caching in sha1_file_name
This function takes a sha1 and produces a loose object
filename. It caches the location of the object directory so
that it can fill the sha1 information directly without
allocating a new buffer (and in its original incarnation,
without calling getenv(), though these days we cache that
with the code in environment.c).
This cached base directory can become stale, however, if in
a single process git changes the location of the object
directory (e.g., by running setup_work_tree, which will
chdir to the new worktree).
In most cases this isn't a problem, because we tend to set
up the git repository location and do any chdir()s before
actually looking up any objects, so the first lookup will
cache the correct location. In the case of reset --hard,
however, we do something like:
1. look up the commit object
2. notice we are doing --hard, run setup_work_tree
3. look up the tree object to reset
Step (3) fails because our cache object directory value is
bogus.
This patch simply removes the caching. We use a static
buffer instead of allocating one each time (the original
version treated the malloc'd buffer as a static, so there is
no change in calling semantics).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-05-22 06:59:42 +00:00
|
|
|
|
fill_sha1_file: write into a strbuf
It's currently the responsibility of the caller to give
fill_sha1_file() enough bytes to write into, leading them to
manually compute the required lengths. Instead, let's just
write into a strbuf so that it's impossible to get this
wrong.
The alt_odb caller already has a strbuf, so this makes
things strictly simpler. The other caller, sha1_file_name(),
uses a static PATH_MAX buffer and dies when it would
overflow. We can convert this to a static strbuf, which
means our allocation cost is amortized (and as a bonus, we
no longer have to worry about PATH_MAX being too short for
normal use).
This does introduce some small overhead in fill_sha1_file(),
as each strbuf_addchar() will check whether it needs to
grow. However, between the optimization in fec501d
(strbuf_addch: avoid calling strbuf_grow, 2015-04-16) and
the fact that this is not generally called in a tight loop
(after all, the next step is typically to access the file!)
this probably doesn't matter. And even if it did, the right
place to micro-optimize is inside fill_sha1_file(), by
calling a single strbuf_grow() there.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:36:09 +00:00
|
|
|
fill_sha1_path(&buf, sha1);
|
|
|
|
return buf.buf;
|
2005-04-18 20:04:43 +00:00
|
|
|
}
|
|
|
|
|
2016-10-03 20:36:04 +00:00
|
|
|
struct strbuf *alt_scratch_buf(struct alternate_object_database *alt)
|
|
|
|
{
|
|
|
|
strbuf_setlen(&alt->scratch, alt->base_len);
|
|
|
|
return &alt->scratch;
|
|
|
|
}
|
|
|
|
|
2016-10-03 20:35:43 +00:00
|
|
|
static const char *alt_sha1_path(struct alternate_object_database *alt,
|
|
|
|
const unsigned char *sha1)
|
|
|
|
{
|
2016-10-03 20:36:04 +00:00
|
|
|
struct strbuf *buf = alt_scratch_buf(alt);
|
fill_sha1_file: write into a strbuf
It's currently the responsibility of the caller to give
fill_sha1_file() enough bytes to write into, leading them to
manually compute the required lengths. Instead, let's just
write into a strbuf so that it's impossible to get this
wrong.
The alt_odb caller already has a strbuf, so this makes
things strictly simpler. The other caller, sha1_file_name(),
uses a static PATH_MAX buffer and dies when it would
overflow. We can convert this to a static strbuf, which
means our allocation cost is amortized (and as a bonus, we
no longer have to worry about PATH_MAX being too short for
normal use).
This does introduce some small overhead in fill_sha1_file(),
as each strbuf_addchar() will check whether it needs to
grow. However, between the optimization in fec501d
(strbuf_addch: avoid calling strbuf_grow, 2015-04-16) and
the fact that this is not generally called in a tight loop
(after all, the next step is typically to access the file!)
this probably doesn't matter. And even if it did, the right
place to micro-optimize is inside fill_sha1_file(), by
calling a single strbuf_grow() there.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:36:09 +00:00
|
|
|
fill_sha1_path(buf, sha1);
|
2016-10-03 20:36:04 +00:00
|
|
|
return buf->buf;
|
2005-04-18 20:04:43 +00:00
|
|
|
}
|
|
|
|
|
2017-03-16 14:27:06 +00:00
|
|
|
char *odb_pack_name(struct strbuf *buf,
|
|
|
|
const unsigned char *sha1,
|
|
|
|
const char *ext)
|
2005-08-01 00:53:44 +00:00
|
|
|
{
|
2015-09-24 21:07:34 +00:00
|
|
|
strbuf_reset(buf);
|
|
|
|
strbuf_addf(buf, "%s/pack/pack-%s.%s", get_object_directory(),
|
2017-03-16 14:27:06 +00:00
|
|
|
sha1_to_hex(sha1), ext);
|
2015-09-24 21:07:34 +00:00
|
|
|
return buf->buf;
|
2005-08-01 00:53:44 +00:00
|
|
|
}
|
|
|
|
|
2008-05-23 22:43:55 +00:00
|
|
|
char *sha1_pack_name(const unsigned char *sha1)
|
2005-08-01 00:53:44 +00:00
|
|
|
{
|
2015-09-24 21:07:34 +00:00
|
|
|
static struct strbuf buf = STRBUF_INIT;
|
2017-03-16 14:27:06 +00:00
|
|
|
return odb_pack_name(&buf, sha1, "pack");
|
2008-05-23 22:43:55 +00:00
|
|
|
}
|
2005-08-01 00:53:44 +00:00
|
|
|
|
2008-05-23 22:43:55 +00:00
|
|
|
char *sha1_pack_index_name(const unsigned char *sha1)
|
|
|
|
{
|
2015-09-24 21:07:34 +00:00
|
|
|
static struct strbuf buf = STRBUF_INIT;
|
2017-03-16 14:27:06 +00:00
|
|
|
return odb_pack_name(&buf, sha1, "idx");
|
2005-08-01 00:53:44 +00:00
|
|
|
}
|
|
|
|
|
2005-08-15 00:25:57 +00:00
|
|
|
struct alternate_object_database *alt_odb_list;
|
|
|
|
static struct alternate_object_database **alt_odb_tail;
|
2005-05-07 07:38:04 +00:00
|
|
|
|
link_alt_odb_entry: refactor string handling
The string handling in link_alt_odb_entry() is mostly an
artifact of the original version, which took the path as a
ptr/len combo, and did not have a NUL-terminated string
until we created one in the alternate_object_database
struct. But since 5bdf0a8 (sha1_file: normalize alt_odb
path before comparing and storing, 2011-09-07), the first
thing we do is put the path into a strbuf, which gives us
some easy opportunities for cleanup.
In particular:
- we call strlen(pathbuf.buf), which is silly; we can look
at pathbuf.len.
- even though we have a strbuf, we don't maintain its
"len" field when chomping extra slashes from the
end, and instead keep a separate "pfxlen" variable. We
can fix this and then drop "pfxlen" entirely.
- we don't check whether the path is usable until after we
allocate the new struct, making extra cleanup work for
ourselves. Since we have a NUL-terminated string, we can
bump the "is it usable" checks higher in the function.
While we're at it, we can move that logic to its own
helper, which makes the flow of link_alt_odb_entry()
easier to follow.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:34:48 +00:00
|
|
|
/*
|
|
|
|
* Return non-zero iff the path is usable as an alternate object database.
|
|
|
|
*/
|
|
|
|
static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)
|
|
|
|
{
|
|
|
|
struct alternate_object_database *alt;
|
|
|
|
|
|
|
|
/* Detect cases where alternate disappeared */
|
|
|
|
if (!is_directory(path->buf)) {
|
|
|
|
error("object directory %s does not exist; "
|
|
|
|
"check .git/objects/info/alternates.",
|
|
|
|
path->buf);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Prevent the common mistake of listing the same
|
|
|
|
* thing twice, or object directory itself.
|
|
|
|
*/
|
|
|
|
for (alt = alt_odb_list; alt; alt = alt->next) {
|
2016-10-03 20:36:26 +00:00
|
|
|
if (!fspathcmp(path->buf, alt->path))
|
link_alt_odb_entry: refactor string handling
The string handling in link_alt_odb_entry() is mostly an
artifact of the original version, which took the path as a
ptr/len combo, and did not have a NUL-terminated string
until we created one in the alternate_object_database
struct. But since 5bdf0a8 (sha1_file: normalize alt_odb
path before comparing and storing, 2011-09-07), the first
thing we do is put the path into a strbuf, which gives us
some easy opportunities for cleanup.
In particular:
- we call strlen(pathbuf.buf), which is silly; we can look
at pathbuf.len.
- even though we have a strbuf, we don't maintain its
"len" field when chomping extra slashes from the
end, and instead keep a separate "pfxlen" variable. We
can fix this and then drop "pfxlen" entirely.
- we don't check whether the path is usable until after we
allocate the new struct, making extra cleanup work for
ourselves. Since we have a NUL-terminated string, we can
bump the "is it usable" checks higher in the function.
While we're at it, we can move that logic to its own
helper, which makes the flow of link_alt_odb_entry()
easier to follow.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:34:48 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
if (!fspathcmp(path->buf, normalized_objdir))
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
2005-05-08 20:51:13 +00:00
|
|
|
/*
|
|
|
|
* Prepare alternate object database registry.
|
2005-08-15 00:25:57 +00:00
|
|
|
*
|
|
|
|
* The variable alt_odb_list points at the list of struct
|
|
|
|
* alternate_object_database. The elements on this list come from
|
|
|
|
* non-empty elements from colon separated ALTERNATE_DB_ENVIRONMENT
|
|
|
|
* environment variable, and $GIT_OBJECT_DIRECTORY/info/alternates,
|
2005-12-05 06:48:43 +00:00
|
|
|
* whose contents is similar to that environment variable but can be
|
|
|
|
* LF separated. Its base points at a statically allocated buffer that
|
2005-08-15 00:25:57 +00:00
|
|
|
* contains "/the/directory/corresponding/to/.git/objects/...", while
|
|
|
|
* its name points just after the slash at the end of ".git/objects/"
|
|
|
|
* in the example above, and has enough space to hold 40-byte hex
|
|
|
|
* SHA1, an extra slash for the first level indirection, and the
|
|
|
|
* terminating NUL.
|
2005-05-08 20:51:13 +00:00
|
|
|
*/
|
2014-07-15 11:29:45 +00:00
|
|
|
static int link_alt_odb_entry(const char *entry, const char *relative_base,
|
|
|
|
int depth, const char *normalized_objdir)
|
2005-05-07 07:38:04 +00:00
|
|
|
{
|
2006-05-07 18:19:21 +00:00
|
|
|
struct alternate_object_database *ent;
|
2011-09-07 10:37:47 +00:00
|
|
|
struct strbuf pathbuf = STRBUF_INIT;
|
2005-08-15 00:25:57 +00:00
|
|
|
|
2007-11-13 20:05:00 +00:00
|
|
|
if (!is_absolute_path(entry) && relative_base) {
|
2016-12-12 18:16:55 +00:00
|
|
|
strbuf_realpath(&pathbuf, relative_base, 1);
|
2011-09-07 10:37:47 +00:00
|
|
|
strbuf_addch(&pathbuf, '/');
|
2006-05-07 18:19:21 +00:00
|
|
|
}
|
2012-11-05 08:41:22 +00:00
|
|
|
strbuf_addstr(&pathbuf, entry);
|
2006-05-07 18:19:21 +00:00
|
|
|
|
alternates: re-allow relative paths from environment
Commit 670c359da (link_alt_odb_entry: handle normalize_path
errors, 2016-10-03) regressed the handling of relative paths
in the GIT_ALTERNATE_OBJECT_DIRECTORIES variable. It's not
entirely clear this was ever meant to work, but it _has_
worked for several years, so this commit restores the
original behavior.
When we get a path in GIT_ALTERNATE_OBJECT_DIRECTORIES, we
add it the path to the list of alternate object directories
as if it were found in objects/info/alternates, but with one
difference: we do not provide the link_alt_odb_entry()
function with a base for relative paths. That function
doesn't turn it into an absolute path, and we end up feeding
the relative path to the strbuf_normalize_path() function.
Most relative paths break out of the top-level directory
(e.g., "../foo.git/objects"), and thus normalizing fails.
Prior to 670c359da, we simply ignored the error, and due to
the way normalize_path_copy() was implemented it happened to
return the original path in this case. We then accessed the
alternate objects using this relative path.
By storing the relative path in the alt_odb list, the path
is relative to wherever we happen to be at the time we do an
object lookup. That means we look from $GIT_DIR in a bare
repository, and from the top of the worktree in a non-bare
repository.
If this were being designed from scratch, it would make
sense to pick a stable location (probably $GIT_DIR, or even
the object directory) and use that as the relative base,
turning the result into an absolute path. However, given
the history, at this point the minimal fix is to match the
pre-670c359da behavior.
We can do this simply by ignoring the error when we have no
relative base and using the original value (which we now
reliably have, thanks to strbuf_normalize_path()).
That still leaves us with a relative path that foils our
duplicate detection, and may act strangely if we ever
chdir() later in the process. We could solve that by storing
an absolute path based on getcwd(). That may be a good
future direction; for now we'll do just the minimum to fix
the regression.
The new t5615 script demonstrates the fix in its final three
tests. Since we didn't have any tests of the alternates
environment variable at all, it also adds some tests of
absolute paths.
Reported-by: Bryan Turner <bturner@atlassian.com>
Signed-off-by: Jeff King <peff@peff.net>
2016-11-08 04:50:17 +00:00
|
|
|
if (strbuf_normalize_path(&pathbuf) < 0 && relative_base) {
|
link_alt_odb_entry: handle normalize_path errors
When we add a new alternate to the list, we try to normalize
out any redundant "..", etc. However, we do not look at the
return value of normalize_path_copy(), and will happily
continue with a path that could not be normalized. Worse,
the normalizing process is done in-place, so we are left
with whatever half-finished working state the normalizing
function was in.
Fortunately, this cannot cause us to read past the end of
our buffer, as that working state will always leave the
NUL from the original path in place. And we do tend to
notice problems when we check is_directory() on the path.
But you can see the nonsense that we feed to is_directory
with an entry like:
this/../../is/../../way/../../too/../../deep/../../to/../../resolve
in your objects/info/alternates, which yields:
error: object directory
/to/e/deep/too/way//ects/this/../../is/../../way/../../too/../../deep/../../to/../../resolve
does not exist; check .git/objects/info/alternates.
We can easily fix this just by checking the return value.
But that makes it hard to generate a good error message,
since we're normalizing in-place and our input value has
been overwritten by cruft.
Instead, let's provide a strbuf helper that does an in-place
normalize, but restores the original contents on error. This
uses a second buffer under the hood, which is slightly less
efficient, but this is not a performance-critical code path.
The strbuf helper can also properly set the "len" parameter
of the strbuf before returning. Just doing:
normalize_path_copy(buf.buf, buf.buf);
will shorten the string, but leave buf.len at the original
length. That may be confusing to later code which uses the
strbuf.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:34:17 +00:00
|
|
|
error("unable to normalize alternate object path: %s",
|
|
|
|
pathbuf.buf);
|
|
|
|
strbuf_release(&pathbuf);
|
|
|
|
return -1;
|
|
|
|
}
|
2011-09-07 10:37:47 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* The trailing slash after the directory name is given by
|
|
|
|
* this function at the end. Remove duplicates.
|
|
|
|
*/
|
link_alt_odb_entry: refactor string handling
The string handling in link_alt_odb_entry() is mostly an
artifact of the original version, which took the path as a
ptr/len combo, and did not have a NUL-terminated string
until we created one in the alternate_object_database
struct. But since 5bdf0a8 (sha1_file: normalize alt_odb
path before comparing and storing, 2011-09-07), the first
thing we do is put the path into a strbuf, which gives us
some easy opportunities for cleanup.
In particular:
- we call strlen(pathbuf.buf), which is silly; we can look
at pathbuf.len.
- even though we have a strbuf, we don't maintain its
"len" field when chomping extra slashes from the
end, and instead keep a separate "pfxlen" variable. We
can fix this and then drop "pfxlen" entirely.
- we don't check whether the path is usable until after we
allocate the new struct, making extra cleanup work for
ourselves. Since we have a NUL-terminated string, we can
bump the "is it usable" checks higher in the function.
While we're at it, we can move that logic to its own
helper, which makes the flow of link_alt_odb_entry()
easier to follow.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:34:48 +00:00
|
|
|
while (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')
|
|
|
|
strbuf_setlen(&pathbuf, pathbuf.len - 1);
|
2011-09-07 10:37:47 +00:00
|
|
|
|
link_alt_odb_entry: refactor string handling
The string handling in link_alt_odb_entry() is mostly an
artifact of the original version, which took the path as a
ptr/len combo, and did not have a NUL-terminated string
until we created one in the alternate_object_database
struct. But since 5bdf0a8 (sha1_file: normalize alt_odb
path before comparing and storing, 2011-09-07), the first
thing we do is put the path into a strbuf, which gives us
some easy opportunities for cleanup.
In particular:
- we call strlen(pathbuf.buf), which is silly; we can look
at pathbuf.len.
- even though we have a strbuf, we don't maintain its
"len" field when chomping extra slashes from the
end, and instead keep a separate "pfxlen" variable. We
can fix this and then drop "pfxlen" entirely.
- we don't check whether the path is usable until after we
allocate the new struct, making extra cleanup work for
ourselves. Since we have a NUL-terminated string, we can
bump the "is it usable" checks higher in the function.
While we're at it, we can move that logic to its own
helper, which makes the flow of link_alt_odb_entry()
easier to follow.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:34:48 +00:00
|
|
|
if (!alt_odb_usable(&pathbuf, normalized_objdir)) {
|
|
|
|
strbuf_release(&pathbuf);
|
2006-05-07 18:19:21 +00:00
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
2016-10-03 20:35:31 +00:00
|
|
|
ent = alloc_alt_odb(pathbuf.buf);
|
2006-05-07 18:19:21 +00:00
|
|
|
|
|
|
|
/* add the alternate entry */
|
|
|
|
*alt_odb_tail = ent;
|
|
|
|
alt_odb_tail = &(ent->next);
|
|
|
|
ent->next = NULL;
|
|
|
|
|
|
|
|
/* recursively add alternates */
|
link_alt_odb_entry: refactor string handling
The string handling in link_alt_odb_entry() is mostly an
artifact of the original version, which took the path as a
ptr/len combo, and did not have a NUL-terminated string
until we created one in the alternate_object_database
struct. But since 5bdf0a8 (sha1_file: normalize alt_odb
path before comparing and storing, 2011-09-07), the first
thing we do is put the path into a strbuf, which gives us
some easy opportunities for cleanup.
In particular:
- we call strlen(pathbuf.buf), which is silly; we can look
at pathbuf.len.
- even though we have a strbuf, we don't maintain its
"len" field when chomping extra slashes from the
end, and instead keep a separate "pfxlen" variable. We
can fix this and then drop "pfxlen" entirely.
- we don't check whether the path is usable until after we
allocate the new struct, making extra cleanup work for
ourselves. Since we have a NUL-terminated string, we can
bump the "is it usable" checks higher in the function.
While we're at it, we can move that logic to its own
helper, which makes the flow of link_alt_odb_entry()
easier to follow.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:34:48 +00:00
|
|
|
read_info_alternates(pathbuf.buf, depth + 1);
|
2006-05-07 18:19:21 +00:00
|
|
|
|
link_alt_odb_entry: refactor string handling
The string handling in link_alt_odb_entry() is mostly an
artifact of the original version, which took the path as a
ptr/len combo, and did not have a NUL-terminated string
until we created one in the alternate_object_database
struct. But since 5bdf0a8 (sha1_file: normalize alt_odb
path before comparing and storing, 2011-09-07), the first
thing we do is put the path into a strbuf, which gives us
some easy opportunities for cleanup.
In particular:
- we call strlen(pathbuf.buf), which is silly; we can look
at pathbuf.len.
- even though we have a strbuf, we don't maintain its
"len" field when chomping extra slashes from the
end, and instead keep a separate "pfxlen" variable. We
can fix this and then drop "pfxlen" entirely.
- we don't check whether the path is usable until after we
allocate the new struct, making extra cleanup work for
ourselves. Since we have a NUL-terminated string, we can
bump the "is it usable" checks higher in the function.
While we're at it, we can move that logic to its own
helper, which makes the flow of link_alt_odb_entry()
easier to follow.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:34:48 +00:00
|
|
|
strbuf_release(&pathbuf);
|
2006-05-07 18:19:21 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
alternates: accept double-quoted paths
We read lists of alternates from objects/info/alternates
files (delimited by newline), as well as from the
GIT_ALTERNATE_OBJECT_DIRECTORIES environment variable
(delimited by colon or semi-colon, depending on the
platform).
There's no mechanism for quoting the delimiters, so it's
impossible to specify an alternate path that contains a
colon in the environment, or one that contains a newline in
a file. We've lived with that restriction for ages because
both alternates and filenames with colons are relatively
rare, and it's only a problem when the two meet. But since
722ff7f87 (receive-pack: quarantine objects until
pre-receive accepts, 2016-10-03), which builds on the
alternates system, every push causes the receiver to set
GIT_ALTERNATE_OBJECT_DIRECTORIES internally.
It would be convenient to have some way to quote the
delimiter so that we can represent arbitrary paths.
The simplest thing would be an escape character before a
quoted delimiter (e.g., "\:" as a literal colon). But that
creates a backwards compatibility problem: any path which
uses that escape character is now broken, and we've just
shifted the problem. We could choose an unlikely escape
character (e.g., something from the non-printable ASCII
range), but that's awkward to use.
Instead, let's treat names as unquoted unless they begin
with a double-quote, in which case they are interpreted via
our usual C-stylke quoting rules. This also breaks
backwards-compatibility, but in a smaller way: it only
matters if your file has a double-quote as the very _first_
character in the path (whereas an escape character is a
problem anywhere in the path). It's also consistent with
many other parts of git, which accept either a bare pathname
or a double-quoted one, and the sender can choose to quote
or not as required.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-12-12 19:52:22 +00:00
|
|
|
static const char *parse_alt_odb_entry(const char *string,
|
|
|
|
int sep,
|
|
|
|
struct strbuf *out)
|
|
|
|
{
|
|
|
|
const char *end;
|
|
|
|
|
|
|
|
strbuf_reset(out);
|
|
|
|
|
|
|
|
if (*string == '#') {
|
|
|
|
/* comment; consume up to next separator */
|
|
|
|
end = strchrnul(string, sep);
|
|
|
|
} else if (*string == '"' && !unquote_c_style(out, string, &end)) {
|
|
|
|
/*
|
|
|
|
* quoted path; unquote_c_style has copied the
|
|
|
|
* data for us and set "end". Broken quoting (e.g.,
|
|
|
|
* an entry that doesn't end with a quote) falls
|
|
|
|
* back to the unquoted case below.
|
|
|
|
*/
|
|
|
|
} else {
|
|
|
|
/* normal, unquoted path */
|
|
|
|
end = strchrnul(string, sep);
|
|
|
|
strbuf_add(out, string, end - string);
|
|
|
|
}
|
|
|
|
|
|
|
|
if (*end)
|
|
|
|
end++;
|
|
|
|
return end;
|
|
|
|
}
|
|
|
|
|
2012-11-05 08:41:23 +00:00
|
|
|
static void link_alt_odb_entries(const char *alt, int len, int sep,
|
2006-05-07 18:19:21 +00:00
|
|
|
const char *relative_base, int depth)
|
|
|
|
{
|
2014-07-15 11:29:45 +00:00
|
|
|
struct strbuf objdirbuf = STRBUF_INIT;
|
alternates: accept double-quoted paths
We read lists of alternates from objects/info/alternates
files (delimited by newline), as well as from the
GIT_ALTERNATE_OBJECT_DIRECTORIES environment variable
(delimited by colon or semi-colon, depending on the
platform).
There's no mechanism for quoting the delimiters, so it's
impossible to specify an alternate path that contains a
colon in the environment, or one that contains a newline in
a file. We've lived with that restriction for ages because
both alternates and filenames with colons are relatively
rare, and it's only a problem when the two meet. But since
722ff7f87 (receive-pack: quarantine objects until
pre-receive accepts, 2016-10-03), which builds on the
alternates system, every push causes the receiver to set
GIT_ALTERNATE_OBJECT_DIRECTORIES internally.
It would be convenient to have some way to quote the
delimiter so that we can represent arbitrary paths.
The simplest thing would be an escape character before a
quoted delimiter (e.g., "\:" as a literal colon). But that
creates a backwards compatibility problem: any path which
uses that escape character is now broken, and we've just
shifted the problem. We could choose an unlikely escape
character (e.g., something from the non-printable ASCII
range), but that's awkward to use.
Instead, let's treat names as unquoted unless they begin
with a double-quote, in which case they are interpreted via
our usual C-stylke quoting rules. This also breaks
backwards-compatibility, but in a smaller way: it only
matters if your file has a double-quote as the very _first_
character in the path (whereas an escape character is a
problem anywhere in the path). It's also consistent with
many other parts of git, which accept either a bare pathname
or a double-quoted one, and the sender can choose to quote
or not as required.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-12-12 19:52:22 +00:00
|
|
|
struct strbuf entry = STRBUF_INIT;
|
2006-05-07 18:19:21 +00:00
|
|
|
|
|
|
|
if (depth > 5) {
|
|
|
|
error("%s: ignoring alternate object stores, nesting too deep.",
|
|
|
|
relative_base);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2014-09-02 20:27:45 +00:00
|
|
|
strbuf_add_absolute_path(&objdirbuf, get_object_directory());
|
link_alt_odb_entry: handle normalize_path errors
When we add a new alternate to the list, we try to normalize
out any redundant "..", etc. However, we do not look at the
return value of normalize_path_copy(), and will happily
continue with a path that could not be normalized. Worse,
the normalizing process is done in-place, so we are left
with whatever half-finished working state the normalizing
function was in.
Fortunately, this cannot cause us to read past the end of
our buffer, as that working state will always leave the
NUL from the original path in place. And we do tend to
notice problems when we check is_directory() on the path.
But you can see the nonsense that we feed to is_directory
with an entry like:
this/../../is/../../way/../../too/../../deep/../../to/../../resolve
in your objects/info/alternates, which yields:
error: object directory
/to/e/deep/too/way//ects/this/../../is/../../way/../../too/../../deep/../../to/../../resolve
does not exist; check .git/objects/info/alternates.
We can easily fix this just by checking the return value.
But that makes it hard to generate a good error message,
since we're normalizing in-place and our input value has
been overwritten by cruft.
Instead, let's provide a strbuf helper that does an in-place
normalize, but restores the original contents on error. This
uses a second buffer under the hood, which is slightly less
efficient, but this is not a performance-critical code path.
The strbuf helper can also properly set the "len" parameter
of the strbuf before returning. Just doing:
normalize_path_copy(buf.buf, buf.buf);
will shorten the string, but leave buf.len at the original
length. That may be confusing to later code which uses the
strbuf.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-03 20:34:17 +00:00
|
|
|
if (strbuf_normalize_path(&objdirbuf) < 0)
|
|
|
|
die("unable to normalize object directory: %s",
|
|
|
|
objdirbuf.buf);
|
2014-07-15 11:29:45 +00:00
|
|
|
|
alternates: accept double-quoted paths
We read lists of alternates from objects/info/alternates
files (delimited by newline), as well as from the
GIT_ALTERNATE_OBJECT_DIRECTORIES environment variable
(delimited by colon or semi-colon, depending on the
platform).
There's no mechanism for quoting the delimiters, so it's
impossible to specify an alternate path that contains a
colon in the environment, or one that contains a newline in
a file. We've lived with that restriction for ages because
both alternates and filenames with colons are relatively
rare, and it's only a problem when the two meet. But since
722ff7f87 (receive-pack: quarantine objects until
pre-receive accepts, 2016-10-03), which builds on the
alternates system, every push causes the receiver to set
GIT_ALTERNATE_OBJECT_DIRECTORIES internally.
It would be convenient to have some way to quote the
delimiter so that we can represent arbitrary paths.
The simplest thing would be an escape character before a
quoted delimiter (e.g., "\:" as a literal colon). But that
creates a backwards compatibility problem: any path which
uses that escape character is now broken, and we've just
shifted the problem. We could choose an unlikely escape
character (e.g., something from the non-printable ASCII
range), but that's awkward to use.
Instead, let's treat names as unquoted unless they begin
with a double-quote, in which case they are interpreted via
our usual C-stylke quoting rules. This also breaks
backwards-compatibility, but in a smaller way: it only
matters if your file has a double-quote as the very _first_
character in the path (whereas an escape character is a
problem anywhere in the path). It's also consistent with
many other parts of git, which accept either a bare pathname
or a double-quoted one, and the sender can choose to quote
or not as required.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-12-12 19:52:22 +00:00
|
|
|
while (*alt) {
|
|
|
|
alt = parse_alt_odb_entry(alt, sep, &entry);
|
|
|
|
if (!entry.len)
|
2005-08-17 01:22:05 +00:00
|
|
|
continue;
|
alternates: accept double-quoted paths
We read lists of alternates from objects/info/alternates
files (delimited by newline), as well as from the
GIT_ALTERNATE_OBJECT_DIRECTORIES environment variable
(delimited by colon or semi-colon, depending on the
platform).
There's no mechanism for quoting the delimiters, so it's
impossible to specify an alternate path that contains a
colon in the environment, or one that contains a newline in
a file. We've lived with that restriction for ages because
both alternates and filenames with colons are relatively
rare, and it's only a problem when the two meet. But since
722ff7f87 (receive-pack: quarantine objects until
pre-receive accepts, 2016-10-03), which builds on the
alternates system, every push causes the receiver to set
GIT_ALTERNATE_OBJECT_DIRECTORIES internally.
It would be convenient to have some way to quote the
delimiter so that we can represent arbitrary paths.
The simplest thing would be an escape character before a
quoted delimiter (e.g., "\:" as a literal colon). But that
creates a backwards compatibility problem: any path which
uses that escape character is now broken, and we've just
shifted the problem. We could choose an unlikely escape
character (e.g., something from the non-printable ASCII
range), but that's awkward to use.
Instead, let's treat names as unquoted unless they begin
with a double-quote, in which case they are interpreted via
our usual C-stylke quoting rules. This also breaks
backwards-compatibility, but in a smaller way: it only
matters if your file has a double-quote as the very _first_
character in the path (whereas an escape character is a
problem anywhere in the path). It's also consistent with
many other parts of git, which accept either a bare pathname
or a double-quoted one, and the sender can choose to quote
or not as required.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-12-12 19:52:22 +00:00
|
|
|
link_alt_odb_entry(entry.buf, relative_base, depth, objdirbuf.buf);
|
2005-08-17 01:22:05 +00:00
|
|
|
}
|
alternates: accept double-quoted paths
We read lists of alternates from objects/info/alternates
files (delimited by newline), as well as from the
GIT_ALTERNATE_OBJECT_DIRECTORIES environment variable
(delimited by colon or semi-colon, depending on the
platform).
There's no mechanism for quoting the delimiters, so it's
impossible to specify an alternate path that contains a
colon in the environment, or one that contains a newline in
a file. We've lived with that restriction for ages because
both alternates and filenames with colons are relatively
rare, and it's only a problem when the two meet. But since
722ff7f87 (receive-pack: quarantine objects until
pre-receive accepts, 2016-10-03), which builds on the
alternates system, every push causes the receiver to set
GIT_ALTERNATE_OBJECT_DIRECTORIES internally.
It would be convenient to have some way to quote the
delimiter so that we can represent arbitrary paths.
The simplest thing would be an escape character before a
quoted delimiter (e.g., "\:" as a literal colon). But that
creates a backwards compatibility problem: any path which
uses that escape character is now broken, and we've just
shifted the problem. We could choose an unlikely escape
character (e.g., something from the non-printable ASCII
range), but that's awkward to use.
Instead, let's treat names as unquoted unless they begin
with a double-quote, in which case they are interpreted via
our usual C-stylke quoting rules. This also breaks
backwards-compatibility, but in a smaller way: it only
matters if your file has a double-quote as the very _first_
character in the path (whereas an escape character is a
problem anywhere in the path). It's also consistent with
many other parts of git, which accept either a bare pathname
or a double-quoted one, and the sender can choose to quote
or not as required.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-12-12 19:52:22 +00:00
|
|
|
strbuf_release(&entry);
|
2014-07-15 11:29:45 +00:00
|
|
|
strbuf_release(&objdirbuf);
|
2005-08-15 00:25:57 +00:00
|
|
|
}
|
|
|
|
|
2012-05-14 16:24:45 +00:00
|
|
|
void read_info_alternates(const char * relative_base, int depth)
|
2005-08-15 00:25:57 +00:00
|
|
|
{
|
2005-08-17 01:22:05 +00:00
|
|
|
char *map;
|
2007-03-07 01:44:37 +00:00
|
|
|
size_t mapsz;
|
2005-08-15 00:25:57 +00:00
|
|
|
struct stat st;
|
2015-08-19 18:12:45 +00:00
|
|
|
char *path;
|
2006-05-07 18:19:21 +00:00
|
|
|
int fd;
|
2005-08-15 00:25:57 +00:00
|
|
|
|
2015-08-19 18:12:45 +00:00
|
|
|
path = xstrfmt("%s/info/alternates", relative_base);
|
2016-10-24 18:02:59 +00:00
|
|
|
fd = git_open(path);
|
2015-08-19 18:12:45 +00:00
|
|
|
free(path);
|
2005-08-15 00:25:57 +00:00
|
|
|
if (fd < 0)
|
|
|
|
return;
|
|
|
|
if (fstat(fd, &st) || (st.st_size == 0)) {
|
|
|
|
close(fd);
|
2005-06-28 21:56:57 +00:00
|
|
|
return;
|
2005-05-07 07:38:04 +00:00
|
|
|
}
|
2007-03-07 01:44:37 +00:00
|
|
|
mapsz = xsize_t(st.st_size);
|
|
|
|
map = xmmap(NULL, mapsz, PROT_READ, MAP_PRIVATE, fd, 0);
|
2005-08-15 00:25:57 +00:00
|
|
|
close(fd);
|
|
|
|
|
2012-11-05 08:41:23 +00:00
|
|
|
link_alt_odb_entries(map, mapsz, '\n', relative_base, depth);
|
2006-05-07 18:19:21 +00:00
|
|
|
|
2007-03-07 01:44:37 +00:00
|
|
|
munmap(map, mapsz);
|
2005-05-07 07:38:04 +00:00
|
|
|
}
|
|
|
|
|
2016-10-03 20:35:31 +00:00
|
|
|
struct alternate_object_database *alloc_alt_odb(const char *dir)
|
|
|
|
{
|
|
|
|
struct alternate_object_database *ent;
|
|
|
|
|
2016-10-03 20:35:51 +00:00
|
|
|
FLEX_ALLOC_STR(ent, path, dir);
|
2016-10-03 20:36:04 +00:00
|
|
|
strbuf_init(&ent->scratch, 0);
|
|
|
|
strbuf_addf(&ent->scratch, "%s/", dir);
|
|
|
|
ent->base_len = ent->scratch.len;
|
2016-10-03 20:35:31 +00:00
|
|
|
|
|
|
|
return ent;
|
|
|
|
}
|
|
|
|
|
2008-04-17 23:32:30 +00:00
|
|
|
void add_to_alternates_file(const char *reference)
|
|
|
|
{
|
|
|
|
struct lock_file *lock = xcalloc(1, sizeof(struct lock_file));
|
add_to_alternates_file: don't add duplicate entries
The add_to_alternates_file function blindly uses
hold_lock_file_for_append to copy the existing contents, and
then adds the new line to it. This has two minor problems:
1. We might add duplicate entries, which are ugly and
inefficient.
2. We do not check that the file ends with a newline, in
which case we would bogusly append to the final line.
This is quite unlikely in practice, though, as we call
this function only from git-clone, so presumably we are
the only writers of the file (and we always add a
newline).
Instead of using hold_lock_file_for_append, let's copy the
file line by line, which ensures all records are properly
terminated. If we see an extra line, we can simply abort the
update (there is no point in even copying the rest, as we
know that it would be identical to the original).
As a bonus, we also get rid of some calls to the
static-buffer mkpath and git_path functions.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-08-10 09:34:46 +00:00
|
|
|
char *alts = git_pathdup("objects/info/alternates");
|
|
|
|
FILE *in, *out;
|
|
|
|
|
|
|
|
hold_lock_file_for_update(lock, alts, LOCK_DIE_ON_ERROR);
|
|
|
|
out = fdopen_lock_file(lock, "w");
|
|
|
|
if (!out)
|
|
|
|
die_errno("unable to fdopen alternates lockfile");
|
|
|
|
|
|
|
|
in = fopen(alts, "r");
|
|
|
|
if (in) {
|
|
|
|
struct strbuf line = STRBUF_INIT;
|
|
|
|
int found = 0;
|
|
|
|
|
2015-10-28 20:29:24 +00:00
|
|
|
while (strbuf_getline(&line, in) != EOF) {
|
add_to_alternates_file: don't add duplicate entries
The add_to_alternates_file function blindly uses
hold_lock_file_for_append to copy the existing contents, and
then adds the new line to it. This has two minor problems:
1. We might add duplicate entries, which are ugly and
inefficient.
2. We do not check that the file ends with a newline, in
which case we would bogusly append to the final line.
This is quite unlikely in practice, though, as we call
this function only from git-clone, so presumably we are
the only writers of the file (and we always add a
newline).
Instead of using hold_lock_file_for_append, let's copy the
file line by line, which ensures all records are properly
terminated. If we see an extra line, we can simply abort the
update (there is no point in even copying the rest, as we
know that it would be identical to the original).
As a bonus, we also get rid of some calls to the
static-buffer mkpath and git_path functions.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-08-10 09:34:46 +00:00
|
|
|
if (!strcmp(reference, line.buf)) {
|
|
|
|
found = 1;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
fprintf_or_die(out, "%s\n", line.buf);
|
|
|
|
}
|
|
|
|
|
|
|
|
strbuf_release(&line);
|
|
|
|
fclose(in);
|
|
|
|
|
|
|
|
if (found) {
|
|
|
|
rollback_lock_file(lock);
|
|
|
|
lock = NULL;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
else if (errno != ENOENT)
|
|
|
|
die_errno("unable to read alternates file");
|
|
|
|
|
|
|
|
if (lock) {
|
|
|
|
fprintf_or_die(out, "%s\n", reference);
|
|
|
|
if (commit_lock_file(lock))
|
|
|
|
die_errno("unable to move new alternates file into place");
|
|
|
|
if (alt_odb_tail)
|
|
|
|
link_alt_odb_entries(reference, strlen(reference), '\n', NULL, 0);
|
|
|
|
}
|
|
|
|
free(alts);
|
2008-04-17 23:32:30 +00:00
|
|
|
}
|
|
|
|
|
2016-10-03 20:35:03 +00:00
|
|
|
void add_to_alternates_memory(const char *reference)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* Make sure alternates are initialized, or else our entry may be
|
|
|
|
* overwritten when they are.
|
|
|
|
*/
|
|
|
|
prepare_alt_odb();
|
|
|
|
|
|
|
|
link_alt_odb_entries(reference, strlen(reference), '\n', NULL, 0);
|
|
|
|
}
|
|
|
|
|
2016-08-15 21:53:24 +00:00
|
|
|
/*
|
|
|
|
* Compute the exact path an alternate is at and returns it. In case of
|
|
|
|
* error NULL is returned and the human readable error is added to `err`
|
|
|
|
* `path` may be relative and should point to $GITDIR.
|
|
|
|
* `err` must not be null.
|
|
|
|
*/
|
|
|
|
char *compute_alternate_path(const char *path, struct strbuf *err)
|
|
|
|
{
|
|
|
|
char *ref_git = NULL;
|
|
|
|
const char *repo, *ref_git_s;
|
|
|
|
int seen_error = 0;
|
|
|
|
|
|
|
|
ref_git_s = real_path_if_valid(path);
|
|
|
|
if (!ref_git_s) {
|
|
|
|
seen_error = 1;
|
|
|
|
strbuf_addf(err, _("path '%s' does not exist"), path);
|
|
|
|
goto out;
|
|
|
|
} else
|
|
|
|
/*
|
|
|
|
* Beware: read_gitfile(), real_path() and mkpath()
|
|
|
|
* return static buffer
|
|
|
|
*/
|
|
|
|
ref_git = xstrdup(ref_git_s);
|
|
|
|
|
|
|
|
repo = read_gitfile(ref_git);
|
|
|
|
if (!repo)
|
|
|
|
repo = read_gitfile(mkpath("%s/.git", ref_git));
|
|
|
|
if (repo) {
|
|
|
|
free(ref_git);
|
|
|
|
ref_git = xstrdup(repo);
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!repo && is_directory(mkpath("%s/.git/objects", ref_git))) {
|
|
|
|
char *ref_git_git = mkpathdup("%s/.git", ref_git);
|
|
|
|
free(ref_git);
|
|
|
|
ref_git = ref_git_git;
|
|
|
|
} else if (!is_directory(mkpath("%s/objects", ref_git))) {
|
|
|
|
struct strbuf sb = STRBUF_INIT;
|
|
|
|
seen_error = 1;
|
|
|
|
if (get_common_dir(&sb, ref_git)) {
|
|
|
|
strbuf_addf(err,
|
|
|
|
_("reference repository '%s' as a linked "
|
|
|
|
"checkout is not supported yet."),
|
|
|
|
path);
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
strbuf_addf(err, _("reference repository '%s' is not a "
|
|
|
|
"local repository."), path);
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!access(mkpath("%s/shallow", ref_git), F_OK)) {
|
|
|
|
strbuf_addf(err, _("reference repository '%s' is shallow"),
|
|
|
|
path);
|
|
|
|
seen_error = 1;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!access(mkpath("%s/info/grafts", ref_git), F_OK)) {
|
|
|
|
strbuf_addf(err,
|
|
|
|
_("reference repository '%s' is grafted"),
|
|
|
|
path);
|
|
|
|
seen_error = 1;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
out:
|
|
|
|
if (seen_error) {
|
|
|
|
free(ref_git);
|
|
|
|
ref_git = NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
return ref_git;
|
|
|
|
}
|
|
|
|
|
2014-10-15 22:33:13 +00:00
|
|
|
int foreach_alt_odb(alt_odb_fn fn, void *cb)
|
push: receiver end advertises refs from alternate repositories
Earlier, when pushing into a repository that borrows from alternate object
stores, we followed the longstanding design decision not to trust refs in
the alternate repository that houses the object store we are borrowing
from. If your public repository is borrowing from Linus's public
repository, you pushed into it long time ago, and now when you try to push
your updated history that is in sync with more recent history from Linus,
you will end up sending not just your own development, but also the
changes you acquired through Linus's tree, even though the objects needed
for the latter already exists at the receiving end. This is because the
receiving end does not advertise that the objects only reachable from the
borrowed repository (i.e. Linus's) are already available there.
This solves the issue by making the receiving end advertise refs from
borrowed repositories. They are not sent with their true names but with a
phoney name ".have" to make sure that the old senders will safely ignore
them (otherwise, the old senders will misbehave, trying to push matching
refs, and mirror push that deletes refs that only exist at the receiving
end).
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-09-09 08:27:10 +00:00
|
|
|
{
|
|
|
|
struct alternate_object_database *ent;
|
2014-10-15 22:33:13 +00:00
|
|
|
int r = 0;
|
push: receiver end advertises refs from alternate repositories
Earlier, when pushing into a repository that borrows from alternate object
stores, we followed the longstanding design decision not to trust refs in
the alternate repository that houses the object store we are borrowing
from. If your public repository is borrowing from Linus's public
repository, you pushed into it long time ago, and now when you try to push
your updated history that is in sync with more recent history from Linus,
you will end up sending not just your own development, but also the
changes you acquired through Linus's tree, even though the objects needed
for the latter already exists at the receiving end. This is because the
receiving end does not advertise that the objects only reachable from the
borrowed repository (i.e. Linus's) are already available there.
This solves the issue by making the receiving end advertise refs from
borrowed repositories. They are not sent with their true names but with a
phoney name ".have" to make sure that the old senders will safely ignore
them (otherwise, the old senders will misbehave, trying to push matching
refs, and mirror push that deletes refs that only exist at the receiving
end).
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-09-09 08:27:10 +00:00
|
|
|
|
|
|
|
prepare_alt_odb();
|
2014-10-15 22:33:13 +00:00
|
|
|
for (ent = alt_odb_list; ent; ent = ent->next) {
|
|
|
|
r = fn(ent, cb);
|
|
|
|
if (r)
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
return r;
|
push: receiver end advertises refs from alternate repositories
Earlier, when pushing into a repository that borrows from alternate object
stores, we followed the longstanding design decision not to trust refs in
the alternate repository that houses the object store we are borrowing
from. If your public repository is borrowing from Linus's public
repository, you pushed into it long time ago, and now when you try to push
your updated history that is in sync with more recent history from Linus,
you will end up sending not just your own development, but also the
changes you acquired through Linus's tree, even though the objects needed
for the latter already exists at the receiving end. This is because the
receiving end does not advertise that the objects only reachable from the
borrowed repository (i.e. Linus's) are already available there.
This solves the issue by making the receiving end advertise refs from
borrowed repositories. They are not sent with their true names but with a
phoney name ".have" to make sure that the old senders will safely ignore
them (otherwise, the old senders will misbehave, trying to push matching
refs, and mirror push that deletes refs that only exist at the receiving
end).
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-09-09 08:27:10 +00:00
|
|
|
}
|
|
|
|
|
2006-05-07 18:19:21 +00:00
|
|
|
void prepare_alt_odb(void)
|
|
|
|
{
|
2006-06-28 09:04:39 +00:00
|
|
|
const char *alt;
|
2006-05-07 18:19:21 +00:00
|
|
|
|
2007-05-26 05:24:40 +00:00
|
|
|
if (alt_odb_tail)
|
|
|
|
return;
|
|
|
|
|
2006-05-07 18:19:21 +00:00
|
|
|
alt = getenv(ALTERNATE_DB_ENVIRONMENT);
|
|
|
|
if (!alt) alt = "";
|
|
|
|
|
|
|
|
alt_odb_tail = &alt_odb_list;
|
2012-11-05 08:41:23 +00:00
|
|
|
link_alt_odb_entries(alt, strlen(alt), PATH_SEP, NULL, 0);
|
2006-05-07 18:19:21 +00:00
|
|
|
|
|
|
|
read_info_alternates(get_object_directory(), 0);
|
|
|
|
}
|
|
|
|
|
check_and_freshen_file: fix reversed success-check
When we want to write out a loose object file, we have
always first made sure we don't already have the object
somewhere. Since 33d4221 (write_sha1_file: freshen existing
objects, 2014-10-15), we also update the timestamp on the
file, so that a simultaneous prune knows somebody is
likely to reference it soon.
If our utime() call fails, we treat this the same as not
having the object in the first place; the safe thing to do
is write out another copy. However, the loose-object check
accidentally inverts the utime() check; it returns failure
_only_ when the utime() call actually succeeded. Thus it was
failing to protect us there, and in the normal case where
utime() succeeds, it caused us to pointlessly write out and
link the object.
This passed our freshening tests, because writing out the
new object is certainly _one_ way of updating its utime. So
the normal case was inefficient, but not wrong.
While we're here, let's also drop a comment in front of the
check_and_freshen functions, making a note of their return
type (since it is not our usual "0 for success, -1 for
error").
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-07-08 20:33:52 +00:00
|
|
|
/* Returns 1 if we have successfully freshened the file, 0 otherwise. */
|
2014-10-15 22:42:22 +00:00
|
|
|
static int freshen_file(const char *fn)
|
2005-05-07 07:38:04 +00:00
|
|
|
{
|
2014-10-15 22:42:22 +00:00
|
|
|
struct utimbuf t;
|
|
|
|
t.actime = t.modtime = time(NULL);
|
|
|
|
return !utime(fn, &t);
|
2008-11-10 05:59:57 +00:00
|
|
|
}
|
2005-05-07 07:38:04 +00:00
|
|
|
|
check_and_freshen_file: fix reversed success-check
When we want to write out a loose object file, we have
always first made sure we don't already have the object
somewhere. Since 33d4221 (write_sha1_file: freshen existing
objects, 2014-10-15), we also update the timestamp on the
file, so that a simultaneous prune knows somebody is
likely to reference it soon.
If our utime() call fails, we treat this the same as not
having the object in the first place; the safe thing to do
is write out another copy. However, the loose-object check
accidentally inverts the utime() check; it returns failure
_only_ when the utime() call actually succeeded. Thus it was
failing to protect us there, and in the normal case where
utime() succeeds, it caused us to pointlessly write out and
link the object.
This passed our freshening tests, because writing out the
new object is certainly _one_ way of updating its utime. So
the normal case was inefficient, but not wrong.
While we're here, let's also drop a comment in front of the
check_and_freshen functions, making a note of their return
type (since it is not our usual "0 for success, -1 for
error").
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-07-08 20:33:52 +00:00
|
|
|
/*
|
|
|
|
* All of the check_and_freshen functions return 1 if the file exists and was
|
|
|
|
* freshened (if freshening was requested), 0 otherwise. If they return
|
|
|
|
* 0, you should not assume that it is safe to skip a write of the object (it
|
|
|
|
* either does not exist on disk, or has a stale mtime and may be subject to
|
|
|
|
* pruning).
|
|
|
|
*/
|
2017-02-27 18:00:11 +00:00
|
|
|
int check_and_freshen_file(const char *fn, int freshen)
|
2014-10-15 22:42:22 +00:00
|
|
|
{
|
|
|
|
if (access(fn, F_OK))
|
|
|
|
return 0;
|
check_and_freshen_file: fix reversed success-check
When we want to write out a loose object file, we have
always first made sure we don't already have the object
somewhere. Since 33d4221 (write_sha1_file: freshen existing
objects, 2014-10-15), we also update the timestamp on the
file, so that a simultaneous prune knows somebody is
likely to reference it soon.
If our utime() call fails, we treat this the same as not
having the object in the first place; the safe thing to do
is write out another copy. However, the loose-object check
accidentally inverts the utime() check; it returns failure
_only_ when the utime() call actually succeeded. Thus it was
failing to protect us there, and in the normal case where
utime() succeeds, it caused us to pointlessly write out and
link the object.
This passed our freshening tests, because writing out the
new object is certainly _one_ way of updating its utime. So
the normal case was inefficient, but not wrong.
While we're here, let's also drop a comment in front of the
check_and_freshen functions, making a note of their return
type (since it is not our usual "0 for success, -1 for
error").
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-07-08 20:33:52 +00:00
|
|
|
if (freshen && !freshen_file(fn))
|
2014-10-15 22:42:22 +00:00
|
|
|
return 0;
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int check_and_freshen_local(const unsigned char *sha1, int freshen)
|
|
|
|
{
|
|
|
|
return check_and_freshen_file(sha1_file_name(sha1), freshen);
|
|
|
|
}
|
|
|
|
|
|
|
|
static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)
|
2008-11-10 05:59:57 +00:00
|
|
|
{
|
|
|
|
struct alternate_object_database *alt;
|
2005-06-28 21:56:57 +00:00
|
|
|
prepare_alt_odb();
|
2005-08-15 00:25:57 +00:00
|
|
|
for (alt = alt_odb_list; alt; alt = alt->next) {
|
2016-10-03 20:35:43 +00:00
|
|
|
const char *path = alt_sha1_path(alt, sha1);
|
|
|
|
if (check_and_freshen_file(path, freshen))
|
2008-06-14 18:43:01 +00:00
|
|
|
return 1;
|
2005-05-07 07:38:04 +00:00
|
|
|
}
|
2008-06-14 18:43:01 +00:00
|
|
|
return 0;
|
2005-05-07 07:38:04 +00:00
|
|
|
}
|
|
|
|
|
2014-10-15 22:42:22 +00:00
|
|
|
static int check_and_freshen(const unsigned char *sha1, int freshen)
|
|
|
|
{
|
|
|
|
return check_and_freshen_local(sha1, freshen) ||
|
|
|
|
check_and_freshen_nonlocal(sha1, freshen);
|
|
|
|
}
|
|
|
|
|
|
|
|
int has_loose_object_nonlocal(const unsigned char *sha1)
|
|
|
|
{
|
|
|
|
return check_and_freshen_nonlocal(sha1, 0);
|
|
|
|
}
|
|
|
|
|
2008-11-10 05:59:57 +00:00
|
|
|
static int has_loose_object(const unsigned char *sha1)
|
|
|
|
{
|
2014-10-15 22:42:22 +00:00
|
|
|
return check_and_freshen(sha1, 0);
|
2008-11-10 05:59:57 +00:00
|
|
|
}
|
|
|
|
|
2006-12-23 07:34:28 +00:00
|
|
|
static unsigned int pack_used_ctr;
|
2006-12-23 07:34:47 +00:00
|
|
|
static unsigned int pack_mmap_calls;
|
|
|
|
static unsigned int peak_pack_open_windows;
|
|
|
|
static unsigned int pack_open_windows;
|
2011-02-28 20:52:39 +00:00
|
|
|
static unsigned int pack_open_fds;
|
|
|
|
static unsigned int pack_max_fds;
|
2006-12-23 07:34:47 +00:00
|
|
|
static size_t peak_pack_mapped;
|
2006-12-23 07:34:28 +00:00
|
|
|
static size_t pack_mapped;
|
2005-06-28 21:56:57 +00:00
|
|
|
struct packed_git *packed_git;
|
2005-06-27 10:35:33 +00:00
|
|
|
|
find_pack_entry: replace last_found_pack with MRU cache
Each pack has an index for looking up entries in O(log n)
time, but if we have multiple packs, we have to scan through
them linearly. This can produce a measurable overhead for
some operations.
We dealt with this long ago in f7c22cc (always start looking
up objects in the last used pack first, 2007-05-30), which
keeps what is essentially a 1-element most-recently-used
cache. In theory, we should be able to do better by keeping
a similar but longer cache, that is the same length as the
pack-list itself.
Since we now have a convenient generic MRU structure, we can
plug it in and measure. Here are the numbers for running
p5303 against linux.git:
Test HEAD^ HEAD
------------------------------------------------------------------------
5303.3: rev-list (1) 31.56(31.28+0.27) 31.30(31.08+0.20) -0.8%
5303.4: repack (1) 40.62(39.35+2.36) 40.60(39.27+2.44) -0.0%
5303.6: rev-list (50) 31.31(31.06+0.23) 31.23(31.00+0.22) -0.3%
5303.7: repack (50) 58.65(69.12+1.94) 58.27(68.64+2.05) -0.6%
5303.9: rev-list (1000) 38.74(38.40+0.33) 31.87(31.62+0.24) -17.7%
5303.10: repack (1000) 367.20(441.80+4.62) 342.00(414.04+3.72) -6.9%
The main numbers of interest here are the rev-list ones
(since that is exercising the normal object lookup code
path). The single-pack case shouldn't improve at all; the
260ms speedup there is just part of the run-to-run noise
(but it's important to note that we didn't make anything
worse with the overhead of maintaining our cache). In the
50-pack case, we see similar results. There may be a slight
improvement, but it's mostly within the noise.
The 1000-pack case does show a big improvement, though. That
carries over to the repack case, as well. Even though we
haven't touched its pack-search loop yet, it does still do a
lot of normal object lookups (e.g., for the internal
revision walk), and so improves.
As a point of reference, I also ran the 1000-pack test
against a version of HEAD^ with the last_found_pack
optimization disabled. It takes ~60s, so that gives an
indication of how much even the single-element cache is
helping.
For comparison, here's a smaller repository, git.git:
Test HEAD^ HEAD
---------------------------------------------------------------------
5303.3: rev-list (1) 1.56(1.54+0.01) 1.54(1.51+0.02) -1.3%
5303.4: repack (1) 1.84(1.80+0.10) 1.82(1.80+0.09) -1.1%
5303.6: rev-list (50) 1.58(1.55+0.02) 1.59(1.57+0.01) +0.6%
5303.7: repack (50) 2.50(3.18+0.04) 2.50(3.14+0.04) +0.0%
5303.9: rev-list (1000) 2.76(2.71+0.04) 2.24(2.21+0.02) -18.8%
5303.10: repack (1000) 13.21(19.56+0.25) 11.66(18.01+0.21) -11.7%
You can see that the percentage improvement is similar.
That's because the lookup we are optimizing is roughly
O(nr_objects * nr_packs). Since the number of packs is
constant in both tests, we'd expect the improvement to be
linear in the number of objects. But the whole process is
also linear in the number of objects, so the improvement
is a constant factor.
The exact improvement does also depend on the contents of
the packs. In p5303, the extra packs all have 5 first-parent
commits in them, which is a reasonable simulation of a
pushed-to repository. But it also means that only 250
first-parent commits are in those packs (compared to almost
50,000 total in linux.git), and the rest are in the huge
"base" pack. So once we start looking at history in taht big
pack, that's where we'll find most everything, and even the
1-element cache gets close to 100% cache hits. You could
almost certainly show better numbers with a more
pathological case (e.g., distributing the objects more
evenly across the packs). But that's simply not that
realistic a scenario, so it makes more sense to focus on
these numbers.
The implementation itself is a straightforward application
of the MRU code. We provide an MRU-ordered list of packs
that shadows the packed_git list. This is easy to do because
we only create and revise the pack list in one place. The
"reprepare" code path actually drops the whole MRU and
replaces it for simplicity. It would be more efficient to
just add new entries, but there's not much point in
optimizing here; repreparing happens rarely, and only after
doing a lot of other expensive work. The key things to keep
optimized are traversal (which is just a normal linked list,
albeit with one extra level of indirection over the regular
packed_git list), and marking (which is a constant number of
pointer assignments, though slightly more than the old
last_found_pack was; it doesn't seem to create a measurable
slowdown, though).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-07-29 04:09:46 +00:00
|
|
|
static struct mru packed_git_mru_storage;
|
|
|
|
struct mru *packed_git_mru = &packed_git_mru_storage;
|
|
|
|
|
2007-06-13 08:22:51 +00:00
|
|
|
void pack_report(void)
|
2006-12-23 07:34:47 +00:00
|
|
|
{
|
|
|
|
fprintf(stderr,
|
2007-01-10 04:07:11 +00:00
|
|
|
"pack_report: getpagesize() = %10" SZ_FMT "\n"
|
|
|
|
"pack_report: core.packedGitWindowSize = %10" SZ_FMT "\n"
|
|
|
|
"pack_report: core.packedGitLimit = %10" SZ_FMT "\n",
|
2007-11-21 20:27:19 +00:00
|
|
|
sz_fmt(getpagesize()),
|
|
|
|
sz_fmt(packed_git_window_size),
|
|
|
|
sz_fmt(packed_git_limit));
|
2006-12-23 07:34:47 +00:00
|
|
|
fprintf(stderr,
|
|
|
|
"pack_report: pack_used_ctr = %10u\n"
|
|
|
|
"pack_report: pack_mmap_calls = %10u\n"
|
|
|
|
"pack_report: pack_open_windows = %10u / %10u\n"
|
2007-01-10 04:07:11 +00:00
|
|
|
"pack_report: pack_mapped = "
|
|
|
|
"%10" SZ_FMT " / %10" SZ_FMT "\n",
|
2006-12-23 07:34:47 +00:00
|
|
|
pack_used_ctr,
|
|
|
|
pack_mmap_calls,
|
|
|
|
pack_open_windows, peak_pack_open_windows,
|
2007-11-21 20:27:19 +00:00
|
|
|
sz_fmt(pack_mapped), sz_fmt(peak_pack_mapped));
|
2006-12-23 07:34:47 +00:00
|
|
|
}
|
|
|
|
|
2014-02-21 16:32:06 +00:00
|
|
|
/*
|
|
|
|
* Open and mmap the index file at path, perform a couple of
|
|
|
|
* consistency checks, then record its information to p. Return 0 on
|
|
|
|
* success.
|
|
|
|
*/
|
|
|
|
static int check_packed_git_idx(const char *path, struct packed_git *p)
|
2005-06-27 10:35:33 +00:00
|
|
|
{
|
|
|
|
void *idx_map;
|
2007-03-16 20:42:50 +00:00
|
|
|
struct pack_idx_header *hdr;
|
2007-03-07 01:44:37 +00:00
|
|
|
size_t idx_size;
|
2007-04-09 05:06:35 +00:00
|
|
|
uint32_t version, nr, i, *index;
|
2016-10-24 18:02:59 +00:00
|
|
|
int fd = git_open(path);
|
2005-06-27 10:35:33 +00:00
|
|
|
struct stat st;
|
2007-03-16 20:42:50 +00:00
|
|
|
|
2005-06-27 10:35:33 +00:00
|
|
|
if (fd < 0)
|
|
|
|
return -1;
|
|
|
|
if (fstat(fd, &st)) {
|
|
|
|
close(fd);
|
|
|
|
return -1;
|
|
|
|
}
|
2007-03-07 01:44:37 +00:00
|
|
|
idx_size = xsize_t(st.st_size);
|
2007-03-07 01:44:11 +00:00
|
|
|
if (idx_size < 4 * 256 + 20 + 20) {
|
|
|
|
close(fd);
|
|
|
|
return error("index file %s is too small", path);
|
|
|
|
}
|
2006-12-24 05:47:23 +00:00
|
|
|
idx_map = xmmap(NULL, idx_size, PROT_READ, MAP_PRIVATE, fd, 0);
|
2005-06-27 10:35:33 +00:00
|
|
|
close(fd);
|
|
|
|
|
2007-03-16 20:42:50 +00:00
|
|
|
hdr = idx_map;
|
|
|
|
if (hdr->idx_signature == htonl(PACK_IDX_SIGNATURE)) {
|
2007-04-09 05:06:35 +00:00
|
|
|
version = ntohl(hdr->idx_version);
|
|
|
|
if (version < 2 || version > 2) {
|
|
|
|
munmap(idx_map, idx_size);
|
2008-07-03 15:52:09 +00:00
|
|
|
return error("index file %s is version %"PRIu32
|
2007-04-09 05:06:35 +00:00
|
|
|
" and is not supported by this binary"
|
|
|
|
" (try upgrading GIT to a newer version)",
|
|
|
|
path, version);
|
|
|
|
}
|
|
|
|
} else
|
|
|
|
version = 1;
|
2007-01-18 01:43:57 +00:00
|
|
|
|
2005-06-27 10:35:33 +00:00
|
|
|
nr = 0;
|
2007-03-16 20:42:50 +00:00
|
|
|
index = idx_map;
|
2007-04-09 05:06:35 +00:00
|
|
|
if (version > 1)
|
|
|
|
index += 2; /* skip index header */
|
2005-06-27 10:35:33 +00:00
|
|
|
for (i = 0; i < 256; i++) {
|
2007-03-07 01:44:19 +00:00
|
|
|
uint32_t n = ntohl(index[i]);
|
2007-03-07 01:44:11 +00:00
|
|
|
if (n < nr) {
|
|
|
|
munmap(idx_map, idx_size);
|
2007-01-18 01:43:57 +00:00
|
|
|
return error("non-monotonic index %s", path);
|
2007-03-07 01:44:11 +00:00
|
|
|
}
|
2005-06-27 10:35:33 +00:00
|
|
|
nr = n;
|
|
|
|
}
|
|
|
|
|
2007-04-09 05:06:35 +00:00
|
|
|
if (version == 1) {
|
|
|
|
/*
|
|
|
|
* Total size:
|
|
|
|
* - 256 index entries 4 bytes each
|
|
|
|
* - 24-byte entries * nr (20-byte sha1 + 4-byte offset)
|
|
|
|
* - 20-byte SHA1 of the packfile
|
|
|
|
* - 20-byte SHA1 file checksum
|
|
|
|
*/
|
|
|
|
if (idx_size != 4*256 + nr * 24 + 20 + 20) {
|
|
|
|
munmap(idx_map, idx_size);
|
2007-08-14 19:42:37 +00:00
|
|
|
return error("wrong index v1 file size in %s", path);
|
2007-04-09 05:06:35 +00:00
|
|
|
}
|
|
|
|
} else if (version == 2) {
|
|
|
|
/*
|
|
|
|
* Minimum size:
|
|
|
|
* - 8 bytes of header
|
|
|
|
* - 256 index entries 4 bytes each
|
|
|
|
* - 20-byte sha1 entry * nr
|
|
|
|
* - 4-byte crc entry * nr
|
|
|
|
* - 4-byte offset entry * nr
|
|
|
|
* - 20-byte SHA1 of the packfile
|
|
|
|
* - 20-byte SHA1 file checksum
|
|
|
|
* And after the 4-byte offset table might be a
|
|
|
|
* variable sized table containing 8-byte entries
|
|
|
|
* for offsets larger than 2^31.
|
|
|
|
*/
|
|
|
|
unsigned long min_size = 8 + 4*256 + nr*(20 + 4 + 4) + 20 + 20;
|
2007-06-26 21:34:02 +00:00
|
|
|
unsigned long max_size = min_size;
|
|
|
|
if (nr)
|
|
|
|
max_size += (nr - 1)*8;
|
|
|
|
if (idx_size < min_size || idx_size > max_size) {
|
2007-04-09 05:06:35 +00:00
|
|
|
munmap(idx_map, idx_size);
|
2007-08-14 19:42:37 +00:00
|
|
|
return error("wrong index v2 file size in %s", path);
|
2007-04-09 05:06:35 +00:00
|
|
|
}
|
2007-10-29 18:53:55 +00:00
|
|
|
if (idx_size != min_size &&
|
|
|
|
/*
|
|
|
|
* make sure we can deal with large pack offsets.
|
|
|
|
* 31-bit signed offset won't be enough, neither
|
|
|
|
* 32-bit unsigned one will be.
|
|
|
|
*/
|
|
|
|
(sizeof(off_t) <= 4)) {
|
|
|
|
munmap(idx_map, idx_size);
|
|
|
|
return error("pack too large for current definition of off_t in %s", path);
|
2007-04-09 05:06:35 +00:00
|
|
|
}
|
2007-03-07 01:44:11 +00:00
|
|
|
}
|
2005-06-27 10:35:33 +00:00
|
|
|
|
2007-04-09 05:06:35 +00:00
|
|
|
p->index_version = version;
|
2007-03-16 20:42:50 +00:00
|
|
|
p->index_data = idx_map;
|
|
|
|
p->index_size = idx_size;
|
2007-04-09 05:06:28 +00:00
|
|
|
p->num_objects = nr;
|
2005-06-27 10:35:33 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2007-05-30 06:13:42 +00:00
|
|
|
int open_pack_index(struct packed_git *p)
|
2007-05-26 05:24:19 +00:00
|
|
|
{
|
|
|
|
char *idx_name;
|
2015-09-24 21:07:09 +00:00
|
|
|
size_t len;
|
2007-05-26 05:24:19 +00:00
|
|
|
int ret;
|
|
|
|
|
|
|
|
if (p->index_data)
|
|
|
|
return 0;
|
|
|
|
|
2015-09-24 21:07:09 +00:00
|
|
|
if (!strip_suffix(p->pack_name, ".pack", &len))
|
|
|
|
die("BUG: pack_name does not end in .pack");
|
|
|
|
idx_name = xstrfmt("%.*s.idx", (int)len, p->pack_name);
|
2007-05-26 05:24:19 +00:00
|
|
|
ret = check_packed_git_idx(idx_name, p);
|
|
|
|
free(idx_name);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2006-12-23 07:34:44 +00:00
|
|
|
static void scan_windows(struct packed_git *p,
|
|
|
|
struct packed_git **lru_p,
|
|
|
|
struct pack_window **lru_w,
|
|
|
|
struct pack_window **lru_l)
|
2005-06-27 10:35:33 +00:00
|
|
|
{
|
2006-12-23 07:34:44 +00:00
|
|
|
struct pack_window *w, *w_l;
|
|
|
|
|
|
|
|
for (w_l = NULL, w = p->windows; w; w = w->next) {
|
|
|
|
if (!w->inuse_cnt) {
|
|
|
|
if (!*lru_w || w->last_used < (*lru_w)->last_used) {
|
|
|
|
*lru_p = p;
|
|
|
|
*lru_w = w;
|
|
|
|
*lru_l = w_l;
|
2006-12-23 07:34:23 +00:00
|
|
|
}
|
|
|
|
}
|
2006-12-23 07:34:44 +00:00
|
|
|
w_l = w;
|
2005-06-29 09:51:27 +00:00
|
|
|
}
|
2006-12-23 07:34:44 +00:00
|
|
|
}
|
|
|
|
|
2013-07-31 19:51:37 +00:00
|
|
|
static int unuse_one_window(struct packed_git *current)
|
2006-12-23 07:34:44 +00:00
|
|
|
{
|
|
|
|
struct packed_git *p, *lru_p = NULL;
|
|
|
|
struct pack_window *lru_w = NULL, *lru_l = NULL;
|
|
|
|
|
|
|
|
if (current)
|
|
|
|
scan_windows(current, &lru_p, &lru_w, &lru_l);
|
|
|
|
for (p = packed_git; p; p = p->next)
|
|
|
|
scan_windows(p, &lru_p, &lru_w, &lru_l);
|
2006-12-23 07:34:23 +00:00
|
|
|
if (lru_p) {
|
|
|
|
munmap(lru_w->base, lru_w->len);
|
|
|
|
pack_mapped -= lru_w->len;
|
|
|
|
if (lru_l)
|
|
|
|
lru_l->next = lru_w->next;
|
2013-07-31 19:51:37 +00:00
|
|
|
else
|
2006-12-23 07:34:23 +00:00
|
|
|
lru_p->windows = lru_w->next;
|
|
|
|
free(lru_w);
|
2006-12-23 07:34:47 +00:00
|
|
|
pack_open_windows--;
|
2006-12-23 07:34:23 +00:00
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
return 0;
|
2005-06-29 09:51:27 +00:00
|
|
|
}
|
|
|
|
|
2013-07-31 19:51:37 +00:00
|
|
|
void release_pack_memory(size_t need)
|
2006-12-24 05:47:19 +00:00
|
|
|
{
|
|
|
|
size_t cur = pack_mapped;
|
2013-07-31 19:51:37 +00:00
|
|
|
while (need >= (cur - pack_mapped) && unuse_one_window(NULL))
|
2006-12-24 05:47:19 +00:00
|
|
|
; /* nothing */
|
|
|
|
}
|
|
|
|
|
2014-08-26 15:23:23 +00:00
|
|
|
static void mmap_limit_check(size_t length)
|
|
|
|
{
|
|
|
|
static size_t limit = 0;
|
|
|
|
if (!limit) {
|
|
|
|
limit = git_env_ulong("GIT_MMAP_LIMIT", 0);
|
|
|
|
if (!limit)
|
|
|
|
limit = SIZE_MAX;
|
|
|
|
}
|
|
|
|
if (length > limit)
|
|
|
|
die("attempting to mmap %"PRIuMAX" over limit %"PRIuMAX,
|
|
|
|
(uintmax_t)length, (uintmax_t)limit);
|
|
|
|
}
|
|
|
|
|
2015-05-28 07:56:15 +00:00
|
|
|
void *xmmap_gently(void *start, size_t length,
|
|
|
|
int prot, int flags, int fd, off_t offset)
|
2010-11-06 11:44:11 +00:00
|
|
|
{
|
2014-08-26 15:23:23 +00:00
|
|
|
void *ret;
|
|
|
|
|
|
|
|
mmap_limit_check(length);
|
|
|
|
ret = mmap(start, length, prot, flags, fd, offset);
|
2010-11-06 11:44:11 +00:00
|
|
|
if (ret == MAP_FAILED) {
|
|
|
|
if (!length)
|
|
|
|
return NULL;
|
2013-07-31 19:51:37 +00:00
|
|
|
release_pack_memory(length);
|
2010-11-06 11:44:11 +00:00
|
|
|
ret = mmap(start, length, prot, flags, fd, offset);
|
|
|
|
}
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2015-05-28 07:56:15 +00:00
|
|
|
void *xmmap(void *start, size_t length,
|
|
|
|
int prot, int flags, int fd, off_t offset)
|
|
|
|
{
|
|
|
|
void *ret = xmmap_gently(start, length, prot, flags, fd, offset);
|
|
|
|
if (ret == MAP_FAILED)
|
2015-05-27 20:30:29 +00:00
|
|
|
die_errno("mmap failed");
|
2015-05-28 07:56:15 +00:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2008-01-18 03:57:00 +00:00
|
|
|
void close_pack_windows(struct packed_git *p)
|
|
|
|
{
|
|
|
|
while (p->windows) {
|
|
|
|
struct pack_window *w = p->windows;
|
|
|
|
|
|
|
|
if (w->inuse_cnt)
|
|
|
|
die("pack '%s' still has open windows to it",
|
|
|
|
p->pack_name);
|
|
|
|
munmap(w->base, w->len);
|
|
|
|
pack_mapped -= w->len;
|
|
|
|
pack_open_windows--;
|
|
|
|
p->windows = w->next;
|
|
|
|
free(w);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2015-10-05 20:30:24 +00:00
|
|
|
static int close_pack_fd(struct packed_git *p)
|
|
|
|
{
|
|
|
|
if (p->pack_fd < 0)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
close(p->pack_fd);
|
|
|
|
pack_open_fds--;
|
|
|
|
p->pack_fd = -1;
|
|
|
|
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
2015-10-06 13:18:34 +00:00
|
|
|
static void close_pack(struct packed_git *p)
|
|
|
|
{
|
|
|
|
close_pack_windows(p);
|
|
|
|
close_pack_fd(p);
|
|
|
|
close_pack_index(p);
|
|
|
|
}
|
|
|
|
|
|
|
|
void close_all_packs(void)
|
|
|
|
{
|
|
|
|
struct packed_git *p;
|
|
|
|
|
|
|
|
for (p = packed_git; p; p = p->next)
|
|
|
|
if (p->do_not_close)
|
2016-07-26 16:05:50 +00:00
|
|
|
die("BUG: want to close pack marked 'do-not-close'");
|
2015-10-06 13:18:34 +00:00
|
|
|
else
|
|
|
|
close_pack(p);
|
|
|
|
}
|
|
|
|
|
|
|
|
|
sha1_file: introduce close_one_pack() to close packs on fd pressure
When the number of open packs exceeds pack_max_fds, unuse_one_window()
is called repeatedly to attempt to release the least-recently-used
pack windows, which, as a side-effect, will also close a pack file
after closing its last open window. If a pack file has been opened,
but no windows have been allocated into it, it will never be selected
by unuse_one_window() and hence its file descriptor will not be
closed. When this happens, git may exceed the number of file
descriptors permitted by the system.
This latter situation can occur in show-ref or receive-pack during ref
advertisement. During ref advertisement, receive-pack will iterate
over every ref in the repository and advertise it to the client after
ensuring that the ref exists in the local repository. If the ref is
located inside a pack, then the pack is opened to ensure that it
exists, but since the object is not actually read from the pack, no
mmap windows are allocated. When the number of open packs exceeds
pack_max_fds, unuse_one_window() will not be able to find any windows to
free and will not be able to close any packs. Once the per-process
file descriptor limit is exceeded, receive-pack will produce a warning,
not an error, for each pack it cannot open, and will then most likely
fail with an error to spawn rev-list or index-pack like:
error: cannot create standard input pipe for rev-list: Too many open files
error: Could not run 'git rev-list'
This may also occur during upload-pack when refs are packed (in the
packed-refs file) and the number of packs that must be opened to
verify that these packed refs exist exceeds the file descriptor
limit. If the refs are loose, then upload-pack will read each ref
from the object database (if the object is in a pack, allocating one
or more mmap windows for it) in order to peel tags and advertise the
underlying object. But when the refs are packed and peeled,
upload-pack will use the peeled sha1 in the packed-refs file and
will not need to read from the pack files, so no mmap windows will
be allocated and just like with receive-pack, unuse_one_window()
will never select these opened packs to close.
When we have file descriptor pressure, we just need to find an open
pack to close. We can leave the existing mmap windows open. If
additional windows need to be mapped into the pack file, it will be
reopened when necessary. If the pack file has been rewritten in the
mean time, open_packed_git_1() should notice when it compares the file
size or the pack's sha1 checksum to what was previously read from the
pack index, and reject it.
Let's introduce a new function close_one_pack() designed specifically
for this purpose to search for and close the least-recently-used pack,
where LRU is defined as (in order of preference):
* pack with oldest mtime and no allocated mmap windows
* pack with the least-recently-used windows, i.e. the pack
with the oldest most-recently-used window, where none of
the windows are in use
* pack with the least-recently-used windows
Signed-off-by: Brandon Casey <drafnel@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-08-02 05:36:33 +00:00
|
|
|
/*
|
|
|
|
* The LRU pack is the one with the oldest MRU window, preferring packs
|
|
|
|
* with no used windows, or the oldest mtime if it has no windows allocated.
|
|
|
|
*/
|
|
|
|
static void find_lru_pack(struct packed_git *p, struct packed_git **lru_p, struct pack_window **mru_w, int *accept_windows_inuse)
|
|
|
|
{
|
|
|
|
struct pack_window *w, *this_mru_w;
|
|
|
|
int has_windows_inuse = 0;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Reject this pack if it has windows and the previously selected
|
|
|
|
* one does not. If this pack does not have windows, reject
|
|
|
|
* it if the pack file is newer than the previously selected one.
|
|
|
|
*/
|
|
|
|
if (*lru_p && !*mru_w && (p->windows || p->mtime > (*lru_p)->mtime))
|
|
|
|
return;
|
|
|
|
|
|
|
|
for (w = this_mru_w = p->windows; w; w = w->next) {
|
|
|
|
/*
|
|
|
|
* Reject this pack if any of its windows are in use,
|
|
|
|
* but the previously selected pack did not have any
|
|
|
|
* inuse windows. Otherwise, record that this pack
|
|
|
|
* has windows in use.
|
|
|
|
*/
|
|
|
|
if (w->inuse_cnt) {
|
|
|
|
if (*accept_windows_inuse)
|
|
|
|
has_windows_inuse = 1;
|
|
|
|
else
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (w->last_used > this_mru_w->last_used)
|
|
|
|
this_mru_w = w;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Reject this pack if it has windows that have been
|
|
|
|
* used more recently than the previously selected pack.
|
|
|
|
* If the previously selected pack had windows inuse and
|
|
|
|
* we have not encountered a window in this pack that is
|
|
|
|
* inuse, skip this check since we prefer a pack with no
|
|
|
|
* inuse windows to one that has inuse windows.
|
|
|
|
*/
|
|
|
|
if (*mru_w && *accept_windows_inuse == has_windows_inuse &&
|
|
|
|
this_mru_w->last_used > (*mru_w)->last_used)
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Select this pack.
|
|
|
|
*/
|
|
|
|
*mru_w = this_mru_w;
|
|
|
|
*lru_p = p;
|
|
|
|
*accept_windows_inuse = has_windows_inuse;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int close_one_pack(void)
|
|
|
|
{
|
|
|
|
struct packed_git *p, *lru_p = NULL;
|
|
|
|
struct pack_window *mru_w = NULL;
|
|
|
|
int accept_windows_inuse = 1;
|
|
|
|
|
|
|
|
for (p = packed_git; p; p = p->next) {
|
|
|
|
if (p->pack_fd == -1)
|
|
|
|
continue;
|
|
|
|
find_lru_pack(p, &lru_p, &mru_w, &accept_windows_inuse);
|
|
|
|
}
|
|
|
|
|
2015-10-05 20:30:24 +00:00
|
|
|
if (lru_p)
|
|
|
|
return close_pack_fd(lru_p);
|
sha1_file: introduce close_one_pack() to close packs on fd pressure
When the number of open packs exceeds pack_max_fds, unuse_one_window()
is called repeatedly to attempt to release the least-recently-used
pack windows, which, as a side-effect, will also close a pack file
after closing its last open window. If a pack file has been opened,
but no windows have been allocated into it, it will never be selected
by unuse_one_window() and hence its file descriptor will not be
closed. When this happens, git may exceed the number of file
descriptors permitted by the system.
This latter situation can occur in show-ref or receive-pack during ref
advertisement. During ref advertisement, receive-pack will iterate
over every ref in the repository and advertise it to the client after
ensuring that the ref exists in the local repository. If the ref is
located inside a pack, then the pack is opened to ensure that it
exists, but since the object is not actually read from the pack, no
mmap windows are allocated. When the number of open packs exceeds
pack_max_fds, unuse_one_window() will not be able to find any windows to
free and will not be able to close any packs. Once the per-process
file descriptor limit is exceeded, receive-pack will produce a warning,
not an error, for each pack it cannot open, and will then most likely
fail with an error to spawn rev-list or index-pack like:
error: cannot create standard input pipe for rev-list: Too many open files
error: Could not run 'git rev-list'
This may also occur during upload-pack when refs are packed (in the
packed-refs file) and the number of packs that must be opened to
verify that these packed refs exist exceeds the file descriptor
limit. If the refs are loose, then upload-pack will read each ref
from the object database (if the object is in a pack, allocating one
or more mmap windows for it) in order to peel tags and advertise the
underlying object. But when the refs are packed and peeled,
upload-pack will use the peeled sha1 in the packed-refs file and
will not need to read from the pack files, so no mmap windows will
be allocated and just like with receive-pack, unuse_one_window()
will never select these opened packs to close.
When we have file descriptor pressure, we just need to find an open
pack to close. We can leave the existing mmap windows open. If
additional windows need to be mapped into the pack file, it will be
reopened when necessary. If the pack file has been rewritten in the
mean time, open_packed_git_1() should notice when it compares the file
size or the pack's sha1 checksum to what was previously read from the
pack index, and reject it.
Let's introduce a new function close_one_pack() designed specifically
for this purpose to search for and close the least-recently-used pack,
where LRU is defined as (in order of preference):
* pack with oldest mtime and no allocated mmap windows
* pack with the least-recently-used windows, i.e. the pack
with the oldest most-recently-used window, where none of
the windows are in use
* pack with the least-recently-used windows
Signed-off-by: Brandon Casey <drafnel@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-08-02 05:36:33 +00:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2006-12-23 07:34:08 +00:00
|
|
|
void unuse_pack(struct pack_window **w_cursor)
|
2005-06-29 09:51:27 +00:00
|
|
|
{
|
2006-12-23 07:34:08 +00:00
|
|
|
struct pack_window *w = *w_cursor;
|
|
|
|
if (w) {
|
|
|
|
w->inuse_cnt--;
|
|
|
|
*w_cursor = NULL;
|
|
|
|
}
|
2005-06-27 10:35:33 +00:00
|
|
|
}
|
|
|
|
|
2010-04-19 14:23:06 +00:00
|
|
|
void close_pack_index(struct packed_git *p)
|
|
|
|
{
|
|
|
|
if (p->index_data) {
|
|
|
|
munmap((void *)p->index_data, p->index_size);
|
|
|
|
p->index_data = NULL;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2012-08-24 09:52:22 +00:00
|
|
|
static unsigned int get_max_fd_limit(void)
|
|
|
|
{
|
|
|
|
#ifdef RLIMIT_NOFILE
|
2013-12-18 22:59:12 +00:00
|
|
|
{
|
|
|
|
struct rlimit lim;
|
2012-08-24 09:52:22 +00:00
|
|
|
|
2013-12-18 22:59:12 +00:00
|
|
|
if (!getrlimit(RLIMIT_NOFILE, &lim))
|
|
|
|
return lim.rlim_cur;
|
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
|
|
|
#ifdef _SC_OPEN_MAX
|
|
|
|
{
|
|
|
|
long open_max = sysconf(_SC_OPEN_MAX);
|
|
|
|
if (0 < open_max)
|
|
|
|
return open_max;
|
|
|
|
/*
|
|
|
|
* Otherwise, we got -1 for one of the two
|
|
|
|
* reasons:
|
|
|
|
*
|
|
|
|
* (1) sysconf() did not understand _SC_OPEN_MAX
|
|
|
|
* and signaled an error with -1; or
|
|
|
|
* (2) sysconf() said there is no limit.
|
|
|
|
*
|
|
|
|
* We _could_ clear errno before calling sysconf() to
|
|
|
|
* tell these two cases apart and return a huge number
|
|
|
|
* in the latter case to let the caller cap it to a
|
|
|
|
* value that is not so selfish, but letting the
|
|
|
|
* fallback OPEN_MAX codepath take care of these cases
|
|
|
|
* is a lot simpler.
|
|
|
|
*/
|
|
|
|
}
|
|
|
|
#endif
|
2012-08-24 09:52:22 +00:00
|
|
|
|
2013-12-18 22:59:12 +00:00
|
|
|
#ifdef OPEN_MAX
|
2012-08-24 09:52:22 +00:00
|
|
|
return OPEN_MAX;
|
|
|
|
#else
|
|
|
|
return 1; /* see the caller ;-) */
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
2007-02-02 08:00:03 +00:00
|
|
|
/*
|
|
|
|
* Do not call this directly as this leaks p->pack_fd on error return;
|
|
|
|
* call open_packed_git() instead.
|
|
|
|
*/
|
|
|
|
static int open_packed_git_1(struct packed_git *p)
|
2005-06-27 10:35:33 +00:00
|
|
|
{
|
2006-12-23 07:34:01 +00:00
|
|
|
struct stat st;
|
|
|
|
struct pack_header hdr;
|
|
|
|
unsigned char sha1[20];
|
|
|
|
unsigned char *idx_sha1;
|
2006-12-29 08:30:01 +00:00
|
|
|
long fd_flag;
|
2006-12-23 07:34:01 +00:00
|
|
|
|
2007-05-26 05:24:19 +00:00
|
|
|
if (!p->index_data && open_pack_index(p))
|
|
|
|
return error("packfile %s index unavailable", p->pack_name);
|
|
|
|
|
2011-02-28 20:52:39 +00:00
|
|
|
if (!pack_max_fds) {
|
2012-08-24 09:52:22 +00:00
|
|
|
unsigned int max_fds = get_max_fd_limit();
|
2011-02-28 20:52:39 +00:00
|
|
|
|
|
|
|
/* Save 3 for stdin/stdout/stderr, 22 for work */
|
|
|
|
if (25 < max_fds)
|
|
|
|
pack_max_fds = max_fds - 25;
|
|
|
|
else
|
|
|
|
pack_max_fds = 1;
|
|
|
|
}
|
|
|
|
|
sha1_file: introduce close_one_pack() to close packs on fd pressure
When the number of open packs exceeds pack_max_fds, unuse_one_window()
is called repeatedly to attempt to release the least-recently-used
pack windows, which, as a side-effect, will also close a pack file
after closing its last open window. If a pack file has been opened,
but no windows have been allocated into it, it will never be selected
by unuse_one_window() and hence its file descriptor will not be
closed. When this happens, git may exceed the number of file
descriptors permitted by the system.
This latter situation can occur in show-ref or receive-pack during ref
advertisement. During ref advertisement, receive-pack will iterate
over every ref in the repository and advertise it to the client after
ensuring that the ref exists in the local repository. If the ref is
located inside a pack, then the pack is opened to ensure that it
exists, but since the object is not actually read from the pack, no
mmap windows are allocated. When the number of open packs exceeds
pack_max_fds, unuse_one_window() will not be able to find any windows to
free and will not be able to close any packs. Once the per-process
file descriptor limit is exceeded, receive-pack will produce a warning,
not an error, for each pack it cannot open, and will then most likely
fail with an error to spawn rev-list or index-pack like:
error: cannot create standard input pipe for rev-list: Too many open files
error: Could not run 'git rev-list'
This may also occur during upload-pack when refs are packed (in the
packed-refs file) and the number of packs that must be opened to
verify that these packed refs exist exceeds the file descriptor
limit. If the refs are loose, then upload-pack will read each ref
from the object database (if the object is in a pack, allocating one
or more mmap windows for it) in order to peel tags and advertise the
underlying object. But when the refs are packed and peeled,
upload-pack will use the peeled sha1 in the packed-refs file and
will not need to read from the pack files, so no mmap windows will
be allocated and just like with receive-pack, unuse_one_window()
will never select these opened packs to close.
When we have file descriptor pressure, we just need to find an open
pack to close. We can leave the existing mmap windows open. If
additional windows need to be mapped into the pack file, it will be
reopened when necessary. If the pack file has been rewritten in the
mean time, open_packed_git_1() should notice when it compares the file
size or the pack's sha1 checksum to what was previously read from the
pack index, and reject it.
Let's introduce a new function close_one_pack() designed specifically
for this purpose to search for and close the least-recently-used pack,
where LRU is defined as (in order of preference):
* pack with oldest mtime and no allocated mmap windows
* pack with the least-recently-used windows, i.e. the pack
with the oldest most-recently-used window, where none of
the windows are in use
* pack with the least-recently-used windows
Signed-off-by: Brandon Casey <drafnel@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-08-02 05:36:33 +00:00
|
|
|
while (pack_max_fds <= pack_open_fds && close_one_pack())
|
2011-02-28 20:52:39 +00:00
|
|
|
; /* nothing */
|
|
|
|
|
2016-10-24 18:02:59 +00:00
|
|
|
p->pack_fd = git_open(p->pack_name);
|
2006-12-23 07:34:01 +00:00
|
|
|
if (p->pack_fd < 0 || fstat(p->pack_fd, &st))
|
2007-02-01 20:52:33 +00:00
|
|
|
return -1;
|
2011-02-28 20:52:39 +00:00
|
|
|
pack_open_fds++;
|
2006-12-23 07:34:01 +00:00
|
|
|
|
|
|
|
/* If we created the struct before we had the pack we lack size. */
|
2005-08-01 00:53:44 +00:00
|
|
|
if (!p->pack_size) {
|
|
|
|
if (!S_ISREG(st.st_mode))
|
2007-02-01 20:52:33 +00:00
|
|
|
return error("packfile %s not a regular file", p->pack_name);
|
2005-08-01 00:53:44 +00:00
|
|
|
p->pack_size = st.st_size;
|
2006-12-23 07:34:01 +00:00
|
|
|
} else if (p->pack_size != st.st_size)
|
2007-02-01 20:52:33 +00:00
|
|
|
return error("packfile %s size changed", p->pack_name);
|
2006-12-23 07:34:01 +00:00
|
|
|
|
2006-12-29 08:30:01 +00:00
|
|
|
/* We leave these file descriptors open with sliding mmap;
|
|
|
|
* there is no point keeping them open across exec(), though.
|
|
|
|
*/
|
|
|
|
fd_flag = fcntl(p->pack_fd, F_GETFD, 0);
|
|
|
|
if (fd_flag < 0)
|
2007-02-01 20:52:33 +00:00
|
|
|
return error("cannot determine file descriptor flags");
|
2006-12-29 08:30:01 +00:00
|
|
|
fd_flag |= FD_CLOEXEC;
|
|
|
|
if (fcntl(p->pack_fd, F_SETFD, fd_flag) == -1)
|
2007-02-01 20:52:33 +00:00
|
|
|
return error("cannot set FD_CLOEXEC");
|
2006-12-29 08:30:01 +00:00
|
|
|
|
2006-12-23 07:34:01 +00:00
|
|
|
/* Verify we recognize this pack file format. */
|
2007-01-14 06:01:49 +00:00
|
|
|
if (read_in_full(p->pack_fd, &hdr, sizeof(hdr)) != sizeof(hdr))
|
2007-02-01 20:52:33 +00:00
|
|
|
return error("file %s is far too short to be a packfile", p->pack_name);
|
2006-12-23 07:34:01 +00:00
|
|
|
if (hdr.hdr_signature != htonl(PACK_SIGNATURE))
|
2007-02-01 20:52:33 +00:00
|
|
|
return error("file %s is not a GIT packfile", p->pack_name);
|
2006-12-23 07:34:01 +00:00
|
|
|
if (!pack_version_ok(hdr.hdr_version))
|
2008-07-03 15:52:09 +00:00
|
|
|
return error("packfile %s is version %"PRIu32" and not"
|
|
|
|
" supported (try upgrading GIT to a newer version)",
|
2006-12-23 07:34:01 +00:00
|
|
|
p->pack_name, ntohl(hdr.hdr_version));
|
|
|
|
|
|
|
|
/* Verify the pack matches its index. */
|
2007-04-09 05:06:28 +00:00
|
|
|
if (p->num_objects != ntohl(hdr.hdr_entries))
|
2008-07-03 15:52:09 +00:00
|
|
|
return error("packfile %s claims to have %"PRIu32" objects"
|
|
|
|
" while index indicates %"PRIu32" objects",
|
2007-04-09 05:06:28 +00:00
|
|
|
p->pack_name, ntohl(hdr.hdr_entries),
|
|
|
|
p->num_objects);
|
2006-12-23 07:34:01 +00:00
|
|
|
if (lseek(p->pack_fd, p->pack_size - sizeof(sha1), SEEK_SET) == -1)
|
2007-02-01 20:52:33 +00:00
|
|
|
return error("end of packfile %s is unavailable", p->pack_name);
|
2007-01-14 06:01:49 +00:00
|
|
|
if (read_in_full(p->pack_fd, sha1, sizeof(sha1)) != sizeof(sha1))
|
2007-02-01 20:52:33 +00:00
|
|
|
return error("packfile %s signature is unavailable", p->pack_name);
|
2007-03-16 20:42:50 +00:00
|
|
|
idx_sha1 = ((unsigned char *)p->index_data) + p->index_size - 40;
|
2006-12-23 07:34:01 +00:00
|
|
|
if (hashcmp(sha1, idx_sha1))
|
2007-02-01 20:52:33 +00:00
|
|
|
return error("packfile %s does not match index", p->pack_name);
|
|
|
|
return 0;
|
2006-12-23 07:34:01 +00:00
|
|
|
}
|
|
|
|
|
2007-02-02 08:00:03 +00:00
|
|
|
static int open_packed_git(struct packed_git *p)
|
|
|
|
{
|
|
|
|
if (!open_packed_git_1(p))
|
|
|
|
return 0;
|
2015-10-05 20:30:24 +00:00
|
|
|
close_pack_fd(p);
|
2007-02-02 08:00:03 +00:00
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
2007-03-07 01:44:30 +00:00
|
|
|
static int in_window(struct pack_window *win, off_t offset)
|
2006-12-23 07:34:28 +00:00
|
|
|
{
|
|
|
|
/* We must promise at least 20 bytes (one hash) after the
|
|
|
|
* offset is available from this window, otherwise the offset
|
|
|
|
* is not actually in this window and a different window (which
|
|
|
|
* has that one hash excess) must be used. This is to support
|
|
|
|
* the object header and delta base parsing routines below.
|
|
|
|
*/
|
|
|
|
off_t win_off = win->offset;
|
|
|
|
return win_off <= offset
|
|
|
|
&& (offset + 20) <= (win_off + win->len);
|
|
|
|
}
|
|
|
|
|
2009-05-01 09:06:36 +00:00
|
|
|
unsigned char *use_pack(struct packed_git *p,
|
2006-12-23 07:34:08 +00:00
|
|
|
struct pack_window **w_cursor,
|
2007-03-07 01:44:30 +00:00
|
|
|
off_t offset,
|
2011-06-10 18:52:15 +00:00
|
|
|
unsigned long *left)
|
2006-12-23 07:34:01 +00:00
|
|
|
{
|
2006-12-23 07:34:28 +00:00
|
|
|
struct pack_window *win = *w_cursor;
|
2006-12-23 07:34:08 +00:00
|
|
|
|
2009-02-24 21:59:05 +00:00
|
|
|
/* Since packfiles end in a hash of their content and it's
|
2006-12-23 07:34:28 +00:00
|
|
|
* pointless to ask for an offset into the middle of that
|
|
|
|
* hash, and the in_window function above wouldn't match
|
|
|
|
* don't allow an offset too close to the end of the file.
|
|
|
|
*/
|
2011-03-02 18:01:54 +00:00
|
|
|
if (!p->pack_size && p->pack_fd == -1 && open_packed_git(p))
|
|
|
|
die("packfile %s cannot be accessed", p->pack_name);
|
2006-12-23 07:34:28 +00:00
|
|
|
if (offset > (p->pack_size - 20))
|
|
|
|
die("offset beyond end of packfile (truncated pack?)");
|
2016-02-25 14:23:26 +00:00
|
|
|
if (offset < 0)
|
2016-02-27 07:49:33 +00:00
|
|
|
die(_("offset before end of packfile (broken .idx?)"));
|
2006-12-23 07:34:28 +00:00
|
|
|
|
|
|
|
if (!win || !in_window(win, offset)) {
|
|
|
|
if (win)
|
|
|
|
win->inuse_cnt--;
|
|
|
|
for (win = p->windows; win; win = win->next) {
|
|
|
|
if (in_window(win, offset))
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
if (!win) {
|
2007-02-14 17:11:40 +00:00
|
|
|
size_t window_align = packed_git_window_size / 2;
|
2007-03-07 01:44:37 +00:00
|
|
|
off_t len;
|
2011-03-02 18:01:54 +00:00
|
|
|
|
|
|
|
if (p->pack_fd == -1 && open_packed_git(p))
|
|
|
|
die("packfile %s cannot be accessed", p->pack_name);
|
|
|
|
|
2006-12-23 07:34:28 +00:00
|
|
|
win = xcalloc(1, sizeof(*win));
|
2007-02-14 17:11:40 +00:00
|
|
|
win->offset = (offset / window_align) * window_align;
|
2007-03-07 01:44:37 +00:00
|
|
|
len = p->pack_size - win->offset;
|
|
|
|
if (len > packed_git_window_size)
|
|
|
|
len = packed_git_window_size;
|
|
|
|
win->len = (size_t)len;
|
2006-12-23 07:34:28 +00:00
|
|
|
pack_mapped += win->len;
|
2006-12-23 07:34:44 +00:00
|
|
|
while (packed_git_limit < pack_mapped
|
2013-07-31 19:51:37 +00:00
|
|
|
&& unuse_one_window(p))
|
2006-12-23 07:34:28 +00:00
|
|
|
; /* nothing */
|
2006-12-24 05:47:23 +00:00
|
|
|
win->base = xmmap(NULL, win->len,
|
2006-12-23 07:34:28 +00:00
|
|
|
PROT_READ, MAP_PRIVATE,
|
|
|
|
p->pack_fd, win->offset);
|
|
|
|
if (win->base == MAP_FAILED)
|
2016-05-08 09:47:56 +00:00
|
|
|
die_errno("packfile %s cannot be mapped",
|
|
|
|
p->pack_name);
|
2011-03-02 18:01:54 +00:00
|
|
|
if (!win->offset && win->len == p->pack_size
|
2015-10-05 20:30:24 +00:00
|
|
|
&& !p->do_not_close)
|
|
|
|
close_pack_fd(p);
|
2006-12-23 07:34:47 +00:00
|
|
|
pack_mmap_calls++;
|
|
|
|
pack_open_windows++;
|
|
|
|
if (pack_mapped > peak_pack_mapped)
|
|
|
|
peak_pack_mapped = pack_mapped;
|
|
|
|
if (pack_open_windows > peak_pack_open_windows)
|
|
|
|
peak_pack_open_windows = pack_open_windows;
|
2006-12-23 07:34:28 +00:00
|
|
|
win->next = p->windows;
|
|
|
|
p->windows = win;
|
|
|
|
}
|
2005-06-27 10:35:33 +00:00
|
|
|
}
|
2006-12-23 07:34:08 +00:00
|
|
|
if (win != *w_cursor) {
|
|
|
|
win->last_used = pack_used_ctr++;
|
|
|
|
win->inuse_cnt++;
|
|
|
|
*w_cursor = win;
|
|
|
|
}
|
2006-12-23 07:34:28 +00:00
|
|
|
offset -= win->offset;
|
2006-12-23 07:34:08 +00:00
|
|
|
if (left)
|
2007-03-07 01:44:37 +00:00
|
|
|
*left = win->len - xsize_t(offset);
|
2006-12-23 07:34:08 +00:00
|
|
|
return win->base + offset;
|
2005-06-27 10:35:33 +00:00
|
|
|
}
|
|
|
|
|
2008-06-24 22:58:06 +00:00
|
|
|
static struct packed_git *alloc_packed_git(int extra)
|
|
|
|
{
|
2016-02-22 22:44:35 +00:00
|
|
|
struct packed_git *p = xmalloc(st_add(sizeof(*p), extra));
|
2008-06-24 22:58:06 +00:00
|
|
|
memset(p, 0, sizeof(*p));
|
|
|
|
p->pack_fd = -1;
|
|
|
|
return p;
|
|
|
|
}
|
|
|
|
|
2010-11-06 19:00:38 +00:00
|
|
|
static void try_to_free_pack_memory(size_t size)
|
|
|
|
{
|
2013-07-31 19:51:37 +00:00
|
|
|
release_pack_memory(size);
|
2010-11-06 19:00:38 +00:00
|
|
|
}
|
|
|
|
|
2015-09-24 21:06:55 +00:00
|
|
|
struct packed_git *add_packed_git(const char *path, size_t path_len, int local)
|
2005-06-27 10:35:33 +00:00
|
|
|
{
|
2010-11-06 19:00:38 +00:00
|
|
|
static int have_set_try_to_free_routine;
|
2005-06-27 10:35:33 +00:00
|
|
|
struct stat st;
|
2015-09-24 21:06:55 +00:00
|
|
|
size_t alloc;
|
|
|
|
struct packed_git *p;
|
2005-06-27 10:35:33 +00:00
|
|
|
|
2010-11-06 19:00:38 +00:00
|
|
|
if (!have_set_try_to_free_routine) {
|
|
|
|
have_set_try_to_free_routine = 1;
|
|
|
|
set_try_to_free_routine(try_to_free_pack_memory);
|
|
|
|
}
|
|
|
|
|
2007-03-16 20:42:50 +00:00
|
|
|
/*
|
|
|
|
* Make sure a corresponding .pack file exists and that
|
|
|
|
* the index looks sane.
|
|
|
|
*/
|
2015-09-24 21:06:55 +00:00
|
|
|
if (!strip_suffix_mem(path, &path_len, ".idx"))
|
2005-06-27 10:35:33 +00:00
|
|
|
return NULL;
|
2015-09-24 21:06:55 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* ".pack" is long enough to hold any suffix we're adding (and
|
|
|
|
* the use xsnprintf double-checks that)
|
|
|
|
*/
|
2016-02-22 22:44:35 +00:00
|
|
|
alloc = st_add3(path_len, strlen(".pack"), 1);
|
2015-09-24 21:06:55 +00:00
|
|
|
p = alloc_packed_git(alloc);
|
2007-03-16 20:42:50 +00:00
|
|
|
memcpy(p->pack_name, path, path_len);
|
2008-11-12 17:59:03 +00:00
|
|
|
|
2015-09-24 21:06:55 +00:00
|
|
|
xsnprintf(p->pack_name + path_len, alloc - path_len, ".keep");
|
2008-11-12 17:59:03 +00:00
|
|
|
if (!access(p->pack_name, F_OK))
|
|
|
|
p->pack_keep = 1;
|
|
|
|
|
2015-09-24 21:06:55 +00:00
|
|
|
xsnprintf(p->pack_name + path_len, alloc - path_len, ".pack");
|
2007-05-26 05:24:19 +00:00
|
|
|
if (stat(p->pack_name, &st) || !S_ISREG(st.st_mode)) {
|
2007-03-16 20:42:50 +00:00
|
|
|
free(p);
|
2005-06-27 10:35:33 +00:00
|
|
|
return NULL;
|
|
|
|
}
|
2007-03-16 20:42:50 +00:00
|
|
|
|
2005-06-27 10:35:33 +00:00
|
|
|
/* ok, it looks sane as far as we can check without
|
|
|
|
* actually mapping the pack file.
|
|
|
|
*/
|
|
|
|
p->pack_size = st.st_size;
|
2005-10-13 22:38:28 +00:00
|
|
|
p->pack_local = local;
|
2007-03-09 11:52:12 +00:00
|
|
|
p->mtime = st.st_mtime;
|
2007-03-16 20:42:50 +00:00
|
|
|
if (path_len < 40 || get_sha1_hex(path + path_len - 40, p->sha1))
|
|
|
|
hashclr(p->sha1);
|
2005-06-27 10:35:33 +00:00
|
|
|
return p;
|
|
|
|
}
|
|
|
|
|
2010-04-19 14:23:08 +00:00
|
|
|
struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path)
|
2005-08-16 04:10:03 +00:00
|
|
|
{
|
2007-03-16 20:42:50 +00:00
|
|
|
const char *path = sha1_pack_name(sha1);
|
2016-02-22 22:44:35 +00:00
|
|
|
size_t alloc = st_add(strlen(path), 1);
|
avoid sprintf and strcpy with flex arrays
When we are allocating a struct with a FLEX_ARRAY member, we
generally compute the size of the array and then sprintf or
strcpy into it. Normally we could improve a dynamic allocation
like this by using xstrfmt, but it doesn't work here; we
have to account for the size of the rest of the struct.
But we can improve things a bit by storing the length that
we use for the allocation, and then feeding it to xsnprintf
or memcpy, which makes it more obvious that we are not
writing more than the allocated number of bytes.
It would be nice if we had some kind of helper for
allocating generic flex arrays, but it doesn't work that
well:
- the call signature is a little bit unwieldy:
d = flex_struct(sizeof(*d), offsetof(d, path), fmt, ...);
You need offsetof here instead of just writing to the
end of the base size, because we don't know how the
struct is packed (partially this is because FLEX_ARRAY
might not be zero, though we can account for that; but
the size of the struct may actually be rounded up for
alignment, and we can't know that).
- some sites do clever things, like over-allocating because
they know they will write larger things into the buffer
later (e.g., struct packed_git here).
So we're better off to just write out each allocation (or
add type-specific helpers, though many of these are one-off
allocations anyway).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-09-24 21:08:12 +00:00
|
|
|
struct packed_git *p = alloc_packed_git(alloc);
|
2005-08-01 00:53:44 +00:00
|
|
|
|
avoid sprintf and strcpy with flex arrays
When we are allocating a struct with a FLEX_ARRAY member, we
generally compute the size of the array and then sprintf or
strcpy into it. Normally we could improve a dynamic allocation
like this by using xstrfmt, but it doesn't work here; we
have to account for the size of the rest of the struct.
But we can improve things a bit by storing the length that
we use for the allocation, and then feeding it to xsnprintf
or memcpy, which makes it more obvious that we are not
writing more than the allocated number of bytes.
It would be nice if we had some kind of helper for
allocating generic flex arrays, but it doesn't work that
well:
- the call signature is a little bit unwieldy:
d = flex_struct(sizeof(*d), offsetof(d, path), fmt, ...);
You need offsetof here instead of just writing to the
end of the base size, because we don't know how the
struct is packed (partially this is because FLEX_ARRAY
might not be zero, though we can account for that; but
the size of the struct may actually be rounded up for
alignment, and we can't know that).
- some sites do clever things, like over-allocating because
they know they will write larger things into the buffer
later (e.g., struct packed_git here).
So we're better off to just write out each allocation (or
add type-specific helpers, though many of these are one-off
allocations anyway).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-09-24 21:08:12 +00:00
|
|
|
memcpy(p->pack_name, path, alloc); /* includes NUL */
|
2008-06-24 22:58:06 +00:00
|
|
|
hashcpy(p->sha1, sha1);
|
2007-03-16 20:42:50 +00:00
|
|
|
if (check_packed_git_idx(idx_path, p)) {
|
|
|
|
free(p);
|
2005-08-01 00:53:44 +00:00
|
|
|
return NULL;
|
2007-03-16 20:42:50 +00:00
|
|
|
}
|
2005-08-01 00:53:44 +00:00
|
|
|
|
|
|
|
return p;
|
|
|
|
}
|
|
|
|
|
|
|
|
void install_packed_git(struct packed_git *pack)
|
|
|
|
{
|
2011-02-28 20:52:39 +00:00
|
|
|
if (pack->pack_fd != -1)
|
|
|
|
pack_open_fds++;
|
|
|
|
|
2005-08-01 00:53:44 +00:00
|
|
|
pack->next = packed_git;
|
|
|
|
packed_git = pack;
|
|
|
|
}
|
|
|
|
|
2015-08-13 18:02:52 +00:00
|
|
|
void (*report_garbage)(unsigned seen_bits, const char *path);
|
2013-02-15 12:07:10 +00:00
|
|
|
|
|
|
|
static void report_helper(const struct string_list *list,
|
|
|
|
int seen_bits, int first, int last)
|
|
|
|
{
|
2015-08-13 18:02:52 +00:00
|
|
|
if (seen_bits == (PACKDIR_FILE_PACK|PACKDIR_FILE_IDX))
|
2013-02-15 12:07:10 +00:00
|
|
|
return;
|
2015-08-13 18:02:52 +00:00
|
|
|
|
2013-02-15 12:07:10 +00:00
|
|
|
for (; first < last; first++)
|
2015-08-13 18:02:52 +00:00
|
|
|
report_garbage(seen_bits, list->items[first].string);
|
2013-02-15 12:07:10 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
static void report_pack_garbage(struct string_list *list)
|
|
|
|
{
|
|
|
|
int i, baselen = -1, first = 0, seen_bits = 0;
|
|
|
|
|
|
|
|
if (!report_garbage)
|
|
|
|
return;
|
|
|
|
|
2014-11-25 08:02:35 +00:00
|
|
|
string_list_sort(list);
|
2013-02-15 12:07:10 +00:00
|
|
|
|
|
|
|
for (i = 0; i < list->nr; i++) {
|
|
|
|
const char *path = list->items[i].string;
|
|
|
|
if (baselen != -1 &&
|
|
|
|
strncmp(path, list->items[first].string, baselen)) {
|
|
|
|
report_helper(list, seen_bits, first, i);
|
|
|
|
baselen = -1;
|
|
|
|
seen_bits = 0;
|
|
|
|
}
|
|
|
|
if (baselen == -1) {
|
|
|
|
const char *dot = strrchr(path, '.');
|
|
|
|
if (!dot) {
|
2015-08-13 18:02:52 +00:00
|
|
|
report_garbage(PACKDIR_FILE_GARBAGE, path);
|
2013-02-15 12:07:10 +00:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
baselen = dot - path + 1;
|
|
|
|
first = i;
|
|
|
|
}
|
|
|
|
if (!strcmp(path + baselen, "pack"))
|
|
|
|
seen_bits |= 1;
|
|
|
|
else if (!strcmp(path + baselen, "idx"))
|
|
|
|
seen_bits |= 2;
|
|
|
|
}
|
|
|
|
report_helper(list, seen_bits, first, list->nr);
|
|
|
|
}
|
|
|
|
|
2005-10-13 22:38:28 +00:00
|
|
|
static void prepare_packed_git_one(char *objdir, int local)
|
2005-06-27 10:35:33 +00:00
|
|
|
{
|
2014-06-30 16:55:52 +00:00
|
|
|
struct strbuf path = STRBUF_INIT;
|
|
|
|
size_t dirnamelen;
|
2005-06-27 10:35:33 +00:00
|
|
|
DIR *dir;
|
|
|
|
struct dirent *de;
|
2013-02-15 12:07:10 +00:00
|
|
|
struct string_list garbage = STRING_LIST_INIT_DUP;
|
2005-06-27 10:35:33 +00:00
|
|
|
|
2014-06-30 16:55:52 +00:00
|
|
|
strbuf_addstr(&path, objdir);
|
|
|
|
strbuf_addstr(&path, "/pack");
|
|
|
|
dir = opendir(path.buf);
|
2006-02-18 00:14:52 +00:00
|
|
|
if (!dir) {
|
2006-02-22 19:16:38 +00:00
|
|
|
if (errno != ENOENT)
|
2016-05-08 09:47:56 +00:00
|
|
|
error_errno("unable to open object pack directory: %s",
|
|
|
|
path.buf);
|
2014-06-30 16:55:52 +00:00
|
|
|
strbuf_release(&path);
|
2005-06-27 10:35:33 +00:00
|
|
|
return;
|
2006-02-18 00:14:52 +00:00
|
|
|
}
|
2014-06-30 16:55:52 +00:00
|
|
|
strbuf_addch(&path, '/');
|
|
|
|
dirnamelen = path.len;
|
2005-06-27 10:35:33 +00:00
|
|
|
while ((de = readdir(dir)) != NULL) {
|
|
|
|
struct packed_git *p;
|
prepare_packed_git_one: refactor duplicate-pack check
When we are reloading the list of packs, we check whether a
particular pack has been loaded. This is slightly tricky,
because we load packs based on the presence of their ".idx"
files, but record the name of the matching ".pack" file.
Therefore we want to compare their bases.
The existing code stripped off ".idx" from a file we found,
then compared that whole base length to strings containing
the ".pack" version. This meant we could end up comparing
bytes past what the ".pack" string contained, if the ".idx"
file name was much longer.
In practice, it worked OK because memcmp would end up seeing
a difference in the two strings and would return early
before hitting the full length. However, memcmp may
sometimes read extra bytes past a difference (e.g., because
it is comparing 64-bit words), or is even free to compare in
reverse order.
Furthermore, our memcmp made no guarantees that we matched
the whole pack name, up to ".pack". So "foo.idx" would match
"foo-bar.pack", which is wrong (but does not typically
happen, because our pack names have a fixed size).
We can fix both issues, avoid magic numbers, and document
that we expect to compare against a string with ".pack" by
using strip_suffix.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-06-30 17:04:03 +00:00
|
|
|
size_t base_len;
|
2013-02-15 12:07:10 +00:00
|
|
|
|
|
|
|
if (is_dot_or_dotdot(de->d_name))
|
2007-07-03 10:40:20 +00:00
|
|
|
continue;
|
|
|
|
|
2014-06-30 16:55:52 +00:00
|
|
|
strbuf_setlen(&path, dirnamelen);
|
|
|
|
strbuf_addstr(&path, de->d_name);
|
2013-02-13 09:13:17 +00:00
|
|
|
|
prepare_packed_git_one: refactor duplicate-pack check
When we are reloading the list of packs, we check whether a
particular pack has been loaded. This is slightly tricky,
because we load packs based on the presence of their ".idx"
files, but record the name of the matching ".pack" file.
Therefore we want to compare their bases.
The existing code stripped off ".idx" from a file we found,
then compared that whole base length to strings containing
the ".pack" version. This meant we could end up comparing
bytes past what the ".pack" string contained, if the ".idx"
file name was much longer.
In practice, it worked OK because memcmp would end up seeing
a difference in the two strings and would return early
before hitting the full length. However, memcmp may
sometimes read extra bytes past a difference (e.g., because
it is comparing 64-bit words), or is even free to compare in
reverse order.
Furthermore, our memcmp made no guarantees that we matched
the whole pack name, up to ".pack". So "foo.idx" would match
"foo-bar.pack", which is wrong (but does not typically
happen, because our pack names have a fixed size).
We can fix both issues, avoid magic numbers, and document
that we expect to compare against a string with ".pack" by
using strip_suffix.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-06-30 17:04:03 +00:00
|
|
|
base_len = path.len;
|
|
|
|
if (strip_suffix_mem(path.buf, &base_len, ".idx")) {
|
2013-02-13 09:13:17 +00:00
|
|
|
/* Don't reopen a pack we already have. */
|
|
|
|
for (p = packed_git; p; p = p->next) {
|
prepare_packed_git_one: refactor duplicate-pack check
When we are reloading the list of packs, we check whether a
particular pack has been loaded. This is slightly tricky,
because we load packs based on the presence of their ".idx"
files, but record the name of the matching ".pack" file.
Therefore we want to compare their bases.
The existing code stripped off ".idx" from a file we found,
then compared that whole base length to strings containing
the ".pack" version. This meant we could end up comparing
bytes past what the ".pack" string contained, if the ".idx"
file name was much longer.
In practice, it worked OK because memcmp would end up seeing
a difference in the two strings and would return early
before hitting the full length. However, memcmp may
sometimes read extra bytes past a difference (e.g., because
it is comparing 64-bit words), or is even free to compare in
reverse order.
Furthermore, our memcmp made no guarantees that we matched
the whole pack name, up to ".pack". So "foo.idx" would match
"foo-bar.pack", which is wrong (but does not typically
happen, because our pack names have a fixed size).
We can fix both issues, avoid magic numbers, and document
that we expect to compare against a string with ".pack" by
using strip_suffix.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-06-30 17:04:03 +00:00
|
|
|
size_t len;
|
|
|
|
if (strip_suffix(p->pack_name, ".pack", &len) &&
|
|
|
|
len == base_len &&
|
|
|
|
!memcmp(p->pack_name, path.buf, len))
|
2013-02-13 09:13:17 +00:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
if (p == NULL &&
|
|
|
|
/*
|
|
|
|
* See if it really is a valid .idx file with
|
|
|
|
* corresponding .pack file that we can map.
|
|
|
|
*/
|
2014-06-30 16:55:52 +00:00
|
|
|
(p = add_packed_git(path.buf, path.len, local)) != NULL)
|
2013-02-13 09:13:17 +00:00
|
|
|
install_packed_git(p);
|
2006-06-02 16:49:32 +00:00
|
|
|
}
|
2013-02-15 12:07:10 +00:00
|
|
|
|
|
|
|
if (!report_garbage)
|
|
|
|
continue;
|
|
|
|
|
2014-06-30 16:58:25 +00:00
|
|
|
if (ends_with(de->d_name, ".idx") ||
|
|
|
|
ends_with(de->d_name, ".pack") ||
|
|
|
|
ends_with(de->d_name, ".bitmap") ||
|
|
|
|
ends_with(de->d_name, ".keep"))
|
2014-06-30 16:55:52 +00:00
|
|
|
string_list_append(&garbage, path.buf);
|
2013-02-15 12:07:10 +00:00
|
|
|
else
|
2015-08-13 18:02:52 +00:00
|
|
|
report_garbage(PACKDIR_FILE_GARBAGE, path.buf);
|
2005-06-27 10:35:33 +00:00
|
|
|
}
|
2005-07-06 06:52:17 +00:00
|
|
|
closedir(dir);
|
2013-02-15 12:07:10 +00:00
|
|
|
report_pack_garbage(&garbage);
|
|
|
|
string_list_clear(&garbage, 0);
|
2014-06-30 16:55:52 +00:00
|
|
|
strbuf_release(&path);
|
2005-06-27 10:35:33 +00:00
|
|
|
}
|
|
|
|
|
2016-10-03 23:47:28 +00:00
|
|
|
static int approximate_object_count_valid;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Give a fast, rough count of the number of objects in the repository. This
|
|
|
|
* ignores loose objects completely. If you have a lot of them, then either
|
|
|
|
* you should repack because your performance will be awful, or they are
|
|
|
|
* all unreachable objects about to be pruned, in which case they're not really
|
|
|
|
* interesting as a measure of repo size in the first place.
|
|
|
|
*/
|
|
|
|
unsigned long approximate_object_count(void)
|
|
|
|
{
|
|
|
|
static unsigned long count;
|
|
|
|
if (!approximate_object_count_valid) {
|
|
|
|
struct packed_git *p;
|
|
|
|
|
|
|
|
prepare_packed_git();
|
|
|
|
count = 0;
|
|
|
|
for (p = packed_git; p; p = p->next) {
|
|
|
|
if (open_pack_index(p))
|
|
|
|
continue;
|
|
|
|
count += p->num_objects;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return count;
|
|
|
|
}
|
|
|
|
|
2016-09-13 17:54:42 +00:00
|
|
|
static void *get_next_packed_git(const void *p)
|
|
|
|
{
|
|
|
|
return ((const struct packed_git *)p)->next;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void set_next_packed_git(void *p, void *next)
|
|
|
|
{
|
|
|
|
((struct packed_git *)p)->next = next;
|
|
|
|
}
|
|
|
|
|
2007-03-09 11:52:12 +00:00
|
|
|
static int sort_pack(const void *a_, const void *b_)
|
|
|
|
{
|
2016-09-13 17:54:42 +00:00
|
|
|
const struct packed_git *a = a_;
|
|
|
|
const struct packed_git *b = b_;
|
2007-03-09 11:52:12 +00:00
|
|
|
int st;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Local packs tend to contain objects specific to our
|
|
|
|
* variant of the project than remote ones. In addition,
|
|
|
|
* remote ones could be on a network mounted filesystem.
|
|
|
|
* Favor local ones for these reasons.
|
|
|
|
*/
|
|
|
|
st = a->pack_local - b->pack_local;
|
|
|
|
if (st)
|
|
|
|
return -st;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Younger packs tend to contain more recent objects,
|
|
|
|
* and more recent objects tend to get accessed more
|
|
|
|
* often.
|
|
|
|
*/
|
|
|
|
if (a->mtime < b->mtime)
|
|
|
|
return 1;
|
|
|
|
else if (a->mtime == b->mtime)
|
|
|
|
return 0;
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void rearrange_packed_git(void)
|
|
|
|
{
|
2016-09-13 17:54:42 +00:00
|
|
|
packed_git = llist_mergesort(packed_git, get_next_packed_git,
|
|
|
|
set_next_packed_git, sort_pack);
|
2007-03-09 11:52:12 +00:00
|
|
|
}
|
|
|
|
|
find_pack_entry: replace last_found_pack with MRU cache
Each pack has an index for looking up entries in O(log n)
time, but if we have multiple packs, we have to scan through
them linearly. This can produce a measurable overhead for
some operations.
We dealt with this long ago in f7c22cc (always start looking
up objects in the last used pack first, 2007-05-30), which
keeps what is essentially a 1-element most-recently-used
cache. In theory, we should be able to do better by keeping
a similar but longer cache, that is the same length as the
pack-list itself.
Since we now have a convenient generic MRU structure, we can
plug it in and measure. Here are the numbers for running
p5303 against linux.git:
Test HEAD^ HEAD
------------------------------------------------------------------------
5303.3: rev-list (1) 31.56(31.28+0.27) 31.30(31.08+0.20) -0.8%
5303.4: repack (1) 40.62(39.35+2.36) 40.60(39.27+2.44) -0.0%
5303.6: rev-list (50) 31.31(31.06+0.23) 31.23(31.00+0.22) -0.3%
5303.7: repack (50) 58.65(69.12+1.94) 58.27(68.64+2.05) -0.6%
5303.9: rev-list (1000) 38.74(38.40+0.33) 31.87(31.62+0.24) -17.7%
5303.10: repack (1000) 367.20(441.80+4.62) 342.00(414.04+3.72) -6.9%
The main numbers of interest here are the rev-list ones
(since that is exercising the normal object lookup code
path). The single-pack case shouldn't improve at all; the
260ms speedup there is just part of the run-to-run noise
(but it's important to note that we didn't make anything
worse with the overhead of maintaining our cache). In the
50-pack case, we see similar results. There may be a slight
improvement, but it's mostly within the noise.
The 1000-pack case does show a big improvement, though. That
carries over to the repack case, as well. Even though we
haven't touched its pack-search loop yet, it does still do a
lot of normal object lookups (e.g., for the internal
revision walk), and so improves.
As a point of reference, I also ran the 1000-pack test
against a version of HEAD^ with the last_found_pack
optimization disabled. It takes ~60s, so that gives an
indication of how much even the single-element cache is
helping.
For comparison, here's a smaller repository, git.git:
Test HEAD^ HEAD
---------------------------------------------------------------------
5303.3: rev-list (1) 1.56(1.54+0.01) 1.54(1.51+0.02) -1.3%
5303.4: repack (1) 1.84(1.80+0.10) 1.82(1.80+0.09) -1.1%
5303.6: rev-list (50) 1.58(1.55+0.02) 1.59(1.57+0.01) +0.6%
5303.7: repack (50) 2.50(3.18+0.04) 2.50(3.14+0.04) +0.0%
5303.9: rev-list (1000) 2.76(2.71+0.04) 2.24(2.21+0.02) -18.8%
5303.10: repack (1000) 13.21(19.56+0.25) 11.66(18.01+0.21) -11.7%
You can see that the percentage improvement is similar.
That's because the lookup we are optimizing is roughly
O(nr_objects * nr_packs). Since the number of packs is
constant in both tests, we'd expect the improvement to be
linear in the number of objects. But the whole process is
also linear in the number of objects, so the improvement
is a constant factor.
The exact improvement does also depend on the contents of
the packs. In p5303, the extra packs all have 5 first-parent
commits in them, which is a reasonable simulation of a
pushed-to repository. But it also means that only 250
first-parent commits are in those packs (compared to almost
50,000 total in linux.git), and the rest are in the huge
"base" pack. So once we start looking at history in taht big
pack, that's where we'll find most everything, and even the
1-element cache gets close to 100% cache hits. You could
almost certainly show better numbers with a more
pathological case (e.g., distributing the objects more
evenly across the packs). But that's simply not that
realistic a scenario, so it makes more sense to focus on
these numbers.
The implementation itself is a straightforward application
of the MRU code. We provide an MRU-ordered list of packs
that shadows the packed_git list. This is easy to do because
we only create and revise the pack list in one place. The
"reprepare" code path actually drops the whole MRU and
replaces it for simplicity. It would be more efficient to
just add new entries, but there's not much point in
optimizing here; repreparing happens rarely, and only after
doing a lot of other expensive work. The key things to keep
optimized are traversal (which is just a normal linked list,
albeit with one extra level of indirection over the regular
packed_git list), and marking (which is a constant number of
pointer assignments, though slightly more than the old
last_found_pack was; it doesn't seem to create a measurable
slowdown, though).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-07-29 04:09:46 +00:00
|
|
|
static void prepare_packed_git_mru(void)
|
|
|
|
{
|
|
|
|
struct packed_git *p;
|
|
|
|
|
|
|
|
mru_clear(packed_git_mru);
|
|
|
|
for (p = packed_git; p; p = p->next)
|
|
|
|
mru_append(packed_git_mru, p);
|
|
|
|
}
|
|
|
|
|
2006-06-02 15:32:23 +00:00
|
|
|
static int prepare_packed_git_run_once = 0;
|
2005-06-28 21:56:57 +00:00
|
|
|
void prepare_packed_git(void)
|
2005-06-27 10:35:33 +00:00
|
|
|
{
|
2005-08-15 00:25:57 +00:00
|
|
|
struct alternate_object_database *alt;
|
2005-06-27 10:35:33 +00:00
|
|
|
|
2006-06-02 15:32:23 +00:00
|
|
|
if (prepare_packed_git_run_once)
|
2005-06-27 10:35:33 +00:00
|
|
|
return;
|
2005-10-13 22:38:28 +00:00
|
|
|
prepare_packed_git_one(get_object_directory(), 1);
|
2005-06-28 21:56:57 +00:00
|
|
|
prepare_alt_odb();
|
2016-10-03 20:35:51 +00:00
|
|
|
for (alt = alt_odb_list; alt; alt = alt->next)
|
|
|
|
prepare_packed_git_one(alt->path, 0);
|
2007-03-09 11:52:12 +00:00
|
|
|
rearrange_packed_git();
|
find_pack_entry: replace last_found_pack with MRU cache
Each pack has an index for looking up entries in O(log n)
time, but if we have multiple packs, we have to scan through
them linearly. This can produce a measurable overhead for
some operations.
We dealt with this long ago in f7c22cc (always start looking
up objects in the last used pack first, 2007-05-30), which
keeps what is essentially a 1-element most-recently-used
cache. In theory, we should be able to do better by keeping
a similar but longer cache, that is the same length as the
pack-list itself.
Since we now have a convenient generic MRU structure, we can
plug it in and measure. Here are the numbers for running
p5303 against linux.git:
Test HEAD^ HEAD
------------------------------------------------------------------------
5303.3: rev-list (1) 31.56(31.28+0.27) 31.30(31.08+0.20) -0.8%
5303.4: repack (1) 40.62(39.35+2.36) 40.60(39.27+2.44) -0.0%
5303.6: rev-list (50) 31.31(31.06+0.23) 31.23(31.00+0.22) -0.3%
5303.7: repack (50) 58.65(69.12+1.94) 58.27(68.64+2.05) -0.6%
5303.9: rev-list (1000) 38.74(38.40+0.33) 31.87(31.62+0.24) -17.7%
5303.10: repack (1000) 367.20(441.80+4.62) 342.00(414.04+3.72) -6.9%
The main numbers of interest here are the rev-list ones
(since that is exercising the normal object lookup code
path). The single-pack case shouldn't improve at all; the
260ms speedup there is just part of the run-to-run noise
(but it's important to note that we didn't make anything
worse with the overhead of maintaining our cache). In the
50-pack case, we see similar results. There may be a slight
improvement, but it's mostly within the noise.
The 1000-pack case does show a big improvement, though. That
carries over to the repack case, as well. Even though we
haven't touched its pack-search loop yet, it does still do a
lot of normal object lookups (e.g., for the internal
revision walk), and so improves.
As a point of reference, I also ran the 1000-pack test
against a version of HEAD^ with the last_found_pack
optimization disabled. It takes ~60s, so that gives an
indication of how much even the single-element cache is
helping.
For comparison, here's a smaller repository, git.git:
Test HEAD^ HEAD
---------------------------------------------------------------------
5303.3: rev-list (1) 1.56(1.54+0.01) 1.54(1.51+0.02) -1.3%
5303.4: repack (1) 1.84(1.80+0.10) 1.82(1.80+0.09) -1.1%
5303.6: rev-list (50) 1.58(1.55+0.02) 1.59(1.57+0.01) +0.6%
5303.7: repack (50) 2.50(3.18+0.04) 2.50(3.14+0.04) +0.0%
5303.9: rev-list (1000) 2.76(2.71+0.04) 2.24(2.21+0.02) -18.8%
5303.10: repack (1000) 13.21(19.56+0.25) 11.66(18.01+0.21) -11.7%
You can see that the percentage improvement is similar.
That's because the lookup we are optimizing is roughly
O(nr_objects * nr_packs). Since the number of packs is
constant in both tests, we'd expect the improvement to be
linear in the number of objects. But the whole process is
also linear in the number of objects, so the improvement
is a constant factor.
The exact improvement does also depend on the contents of
the packs. In p5303, the extra packs all have 5 first-parent
commits in them, which is a reasonable simulation of a
pushed-to repository. But it also means that only 250
first-parent commits are in those packs (compared to almost
50,000 total in linux.git), and the rest are in the huge
"base" pack. So once we start looking at history in taht big
pack, that's where we'll find most everything, and even the
1-element cache gets close to 100% cache hits. You could
almost certainly show better numbers with a more
pathological case (e.g., distributing the objects more
evenly across the packs). But that's simply not that
realistic a scenario, so it makes more sense to focus on
these numbers.
The implementation itself is a straightforward application
of the MRU code. We provide an MRU-ordered list of packs
that shadows the packed_git list. This is easy to do because
we only create and revise the pack list in one place. The
"reprepare" code path actually drops the whole MRU and
replaces it for simplicity. It would be more efficient to
just add new entries, but there's not much point in
optimizing here; repreparing happens rarely, and only after
doing a lot of other expensive work. The key things to keep
optimized are traversal (which is just a normal linked list,
albeit with one extra level of indirection over the regular
packed_git list), and marking (which is a constant number of
pointer assignments, though slightly more than the old
last_found_pack was; it doesn't seem to create a measurable
slowdown, though).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-07-29 04:09:46 +00:00
|
|
|
prepare_packed_git_mru();
|
2006-06-02 15:32:23 +00:00
|
|
|
prepare_packed_git_run_once = 1;
|
|
|
|
}
|
|
|
|
|
2006-11-01 22:06:21 +00:00
|
|
|
void reprepare_packed_git(void)
|
2006-06-02 15:32:23 +00:00
|
|
|
{
|
2016-10-03 23:47:28 +00:00
|
|
|
approximate_object_count_valid = 0;
|
2006-06-02 15:32:23 +00:00
|
|
|
prepare_packed_git_run_once = 0;
|
|
|
|
prepare_packed_git();
|
2005-06-27 10:35:33 +00:00
|
|
|
}
|
|
|
|
|
2008-06-24 01:23:39 +00:00
|
|
|
static void mark_bad_packed_object(struct packed_git *p,
|
|
|
|
const unsigned char *sha1)
|
|
|
|
{
|
|
|
|
unsigned i;
|
|
|
|
for (i = 0; i < p->num_bad_objects; i++)
|
2016-02-22 22:44:35 +00:00
|
|
|
if (!hashcmp(sha1, p->bad_object_sha1 + GIT_SHA1_RAWSZ * i))
|
2008-06-24 01:23:39 +00:00
|
|
|
return;
|
2016-02-22 22:44:35 +00:00
|
|
|
p->bad_object_sha1 = xrealloc(p->bad_object_sha1,
|
2017-03-26 16:01:25 +00:00
|
|
|
st_mult(GIT_MAX_RAWSZ,
|
2016-02-22 22:44:35 +00:00
|
|
|
st_add(p->num_bad_objects, 1)));
|
|
|
|
hashcpy(p->bad_object_sha1 + GIT_SHA1_RAWSZ * p->num_bad_objects, sha1);
|
2008-06-24 01:23:39 +00:00
|
|
|
p->num_bad_objects++;
|
|
|
|
}
|
|
|
|
|
2010-10-28 18:13:06 +00:00
|
|
|
static const struct packed_git *has_packed_and_bad(const unsigned char *sha1)
|
2008-07-15 01:46:48 +00:00
|
|
|
{
|
|
|
|
struct packed_git *p;
|
|
|
|
unsigned i;
|
|
|
|
|
|
|
|
for (p = packed_git; p; p = p->next)
|
|
|
|
for (i = 0; i < p->num_bad_objects; i++)
|
|
|
|
if (!hashcmp(sha1, p->bad_object_sha1 + 20 * i))
|
2010-10-28 18:13:06 +00:00
|
|
|
return p;
|
|
|
|
return NULL;
|
2008-07-15 01:46:48 +00:00
|
|
|
}
|
|
|
|
|
2012-03-07 10:54:18 +00:00
|
|
|
/*
|
|
|
|
* With an in-core object data in "map", rehash it to make sure the
|
|
|
|
* object name actually matches "sha1" to detect object corruption.
|
|
|
|
* With "map" == NULL, try reading the object named with "sha1" using
|
|
|
|
* the streaming interface and rehash it to do the same.
|
|
|
|
*/
|
|
|
|
int check_sha1_signature(const unsigned char *sha1, void *map,
|
|
|
|
unsigned long size, const char *type)
|
2005-04-18 20:04:43 +00:00
|
|
|
{
|
|
|
|
unsigned char real_sha1[20];
|
2012-03-07 10:54:18 +00:00
|
|
|
enum object_type obj_type;
|
|
|
|
struct git_istream *st;
|
|
|
|
git_SHA_CTX c;
|
|
|
|
char hdr[32];
|
|
|
|
int hdrlen;
|
|
|
|
|
|
|
|
if (map) {
|
|
|
|
hash_sha1_file(map, size, type, real_sha1);
|
|
|
|
return hashcmp(sha1, real_sha1) ? -1 : 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
st = open_istream(sha1, &obj_type, &size, NULL);
|
|
|
|
if (!st)
|
|
|
|
return -1;
|
|
|
|
|
|
|
|
/* Generate the header */
|
2015-09-24 21:06:42 +00:00
|
|
|
hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %lu", typename(obj_type), size) + 1;
|
2012-03-07 10:54:18 +00:00
|
|
|
|
|
|
|
/* Sha1.. */
|
|
|
|
git_SHA1_Init(&c);
|
|
|
|
git_SHA1_Update(&c, hdr, hdrlen);
|
|
|
|
for (;;) {
|
|
|
|
char buf[1024 * 16];
|
|
|
|
ssize_t readlen = read_istream(st, buf, sizeof(buf));
|
|
|
|
|
2013-03-25 20:17:17 +00:00
|
|
|
if (readlen < 0) {
|
|
|
|
close_istream(st);
|
|
|
|
return -1;
|
|
|
|
}
|
2012-03-07 10:54:18 +00:00
|
|
|
if (!readlen)
|
|
|
|
break;
|
|
|
|
git_SHA1_Update(&c, buf, readlen);
|
|
|
|
}
|
|
|
|
git_SHA1_Final(real_sha1, &c);
|
|
|
|
close_istream(st);
|
2006-08-17 18:54:57 +00:00
|
|
|
return hashcmp(sha1, real_sha1) ? -1 : 0;
|
2005-04-18 20:04:43 +00:00
|
|
|
}
|
|
|
|
|
2016-10-28 13:23:07 +00:00
|
|
|
int git_open_cloexec(const char *name, int flags)
|
2008-06-14 18:32:37 +00:00
|
|
|
{
|
2016-10-31 17:41:41 +00:00
|
|
|
int fd;
|
|
|
|
static int o_cloexec = O_CLOEXEC;
|
2010-11-01 22:54:21 +00:00
|
|
|
|
2016-10-31 17:41:41 +00:00
|
|
|
fd = open(name, flags | o_cloexec);
|
|
|
|
if ((o_cloexec & O_CLOEXEC) && fd < 0 && errno == EINVAL) {
|
2016-10-24 18:02:59 +00:00
|
|
|
/* Try again w/o O_CLOEXEC: the kernel might not support it */
|
2016-10-31 17:41:41 +00:00
|
|
|
o_cloexec &= ~O_CLOEXEC;
|
|
|
|
fd = open(name, flags | o_cloexec);
|
2016-10-28 13:23:07 +00:00
|
|
|
}
|
2010-11-01 22:54:21 +00:00
|
|
|
|
2016-10-31 17:41:41 +00:00
|
|
|
#if defined(F_GETFL) && defined(F_SETFL) && defined(FD_CLOEXEC)
|
|
|
|
{
|
|
|
|
static int fd_cloexec = FD_CLOEXEC;
|
|
|
|
|
|
|
|
if (!o_cloexec && 0 <= fd && fd_cloexec) {
|
|
|
|
/* Opened w/o O_CLOEXEC? try with fcntl(2) to add it */
|
|
|
|
int flags = fcntl(fd, F_GETFL);
|
|
|
|
if (fcntl(fd, F_SETFL, flags | fd_cloexec))
|
|
|
|
fd_cloexec = 0;
|
2016-10-24 18:02:59 +00:00
|
|
|
}
|
2008-06-14 18:32:37 +00:00
|
|
|
}
|
2016-10-31 17:41:41 +00:00
|
|
|
#endif
|
2016-10-28 13:23:07 +00:00
|
|
|
return fd;
|
2008-06-14 18:32:37 +00:00
|
|
|
}
|
|
|
|
|
2017-01-13 17:54:39 +00:00
|
|
|
/*
|
|
|
|
* Find "sha1" as a loose object in the local repository or in an alternate.
|
|
|
|
* Returns 0 on success, negative on failure.
|
|
|
|
*
|
|
|
|
* The "path" out-parameter will give the path of the object we found (if any).
|
|
|
|
* Note that it may point to static storage and is only valid until another
|
|
|
|
* call to sha1_file_name(), etc.
|
|
|
|
*/
|
|
|
|
static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
|
|
|
|
const char **path)
|
sha1_loose_object_info: make type lookup optional
Until recently, the only items to request from
sha1_object_info_extended were type and size. This meant
that we always had to open a loose object file to determine
one or the other. But with the addition of the disk_size
query, it's possible that we can fulfill the query without
even opening the object file at all. However, since the
function interface always returns the type, we have no way
of knowing whether the caller cares about it or not.
This patch only modified sha1_loose_object_info to make type
lookup optional using an out-parameter, similar to the way
the size is handled (and the return value is "0" or "-1" for
success or error, respectively).
There should be no functional change yet, though, as
sha1_object_info_extended, the only caller, will always ask
for a type.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-07-12 06:30:48 +00:00
|
|
|
{
|
|
|
|
struct alternate_object_database *alt;
|
|
|
|
|
2017-01-13 17:54:39 +00:00
|
|
|
*path = sha1_file_name(sha1);
|
|
|
|
if (!lstat(*path, st))
|
sha1_loose_object_info: make type lookup optional
Until recently, the only items to request from
sha1_object_info_extended were type and size. This meant
that we always had to open a loose object file to determine
one or the other. But with the addition of the disk_size
query, it's possible that we can fulfill the query without
even opening the object file at all. However, since the
function interface always returns the type, we have no way
of knowing whether the caller cares about it or not.
This patch only modified sha1_loose_object_info to make type
lookup optional using an out-parameter, similar to the way
the size is handled (and the return value is "0" or "-1" for
success or error, respectively).
There should be no functional change yet, though, as
sha1_object_info_extended, the only caller, will always ask
for a type.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-07-12 06:30:48 +00:00
|
|
|
return 0;
|
|
|
|
|
|
|
|
prepare_alt_odb();
|
|
|
|
errno = ENOENT;
|
|
|
|
for (alt = alt_odb_list; alt; alt = alt->next) {
|
2017-01-13 17:54:39 +00:00
|
|
|
*path = alt_sha1_path(alt, sha1);
|
|
|
|
if (!lstat(*path, st))
|
sha1_loose_object_info: make type lookup optional
Until recently, the only items to request from
sha1_object_info_extended were type and size. This meant
that we always had to open a loose object file to determine
one or the other. But with the addition of the disk_size
query, it's possible that we can fulfill the query without
even opening the object file at all. However, since the
function interface always returns the type, we have no way
of knowing whether the caller cares about it or not.
This patch only modified sha1_loose_object_info to make type
lookup optional using an out-parameter, similar to the way
the size is handled (and the return value is "0" or "-1" for
success or error, respectively).
There should be no functional change yet, though, as
sha1_object_info_extended, the only caller, will always ask
for a type.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-07-12 06:30:48 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
2017-01-13 17:54:39 +00:00
|
|
|
/*
|
|
|
|
* Like stat_sha1_file(), but actually open the object and return the
|
|
|
|
* descriptor. See the caveats on the "path" parameter above.
|
|
|
|
*/
|
|
|
|
static int open_sha1_file(const unsigned char *sha1, const char **path)
|
2008-06-14 18:32:37 +00:00
|
|
|
{
|
|
|
|
int fd;
|
|
|
|
struct alternate_object_database *alt;
|
2014-05-15 08:54:06 +00:00
|
|
|
int most_interesting_errno;
|
2008-06-14 18:32:37 +00:00
|
|
|
|
2017-01-13 17:54:39 +00:00
|
|
|
*path = sha1_file_name(sha1);
|
|
|
|
fd = git_open(*path);
|
2008-06-14 18:32:37 +00:00
|
|
|
if (fd >= 0)
|
|
|
|
return fd;
|
2014-05-15 08:54:06 +00:00
|
|
|
most_interesting_errno = errno;
|
2008-06-14 18:32:37 +00:00
|
|
|
|
|
|
|
prepare_alt_odb();
|
|
|
|
for (alt = alt_odb_list; alt; alt = alt->next) {
|
2017-01-13 17:54:39 +00:00
|
|
|
*path = alt_sha1_path(alt, sha1);
|
|
|
|
fd = git_open(*path);
|
2008-06-14 18:32:37 +00:00
|
|
|
if (fd >= 0)
|
|
|
|
return fd;
|
2014-05-15 08:54:06 +00:00
|
|
|
if (most_interesting_errno == ENOENT)
|
|
|
|
most_interesting_errno = errno;
|
2008-06-14 18:32:37 +00:00
|
|
|
}
|
2014-05-15 08:54:06 +00:00
|
|
|
errno = most_interesting_errno;
|
2008-06-14 18:32:37 +00:00
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
2017-01-13 17:58:16 +00:00
|
|
|
/*
|
|
|
|
* Map the loose object at "path" if it is not NULL, or the path found by
|
|
|
|
* searching for a loose object named "sha1".
|
|
|
|
*/
|
|
|
|
static void *map_sha1_file_1(const char *path,
|
|
|
|
const unsigned char *sha1,
|
|
|
|
unsigned long *size)
|
2005-04-18 20:04:43 +00:00
|
|
|
{
|
|
|
|
void *map;
|
2005-04-23 18:09:32 +00:00
|
|
|
int fd;
|
2005-05-07 07:38:04 +00:00
|
|
|
|
2017-01-13 17:58:16 +00:00
|
|
|
if (path)
|
|
|
|
fd = git_open(path);
|
|
|
|
else
|
|
|
|
fd = open_sha1_file(sha1, &path);
|
2008-06-14 18:32:37 +00:00
|
|
|
map = NULL;
|
|
|
|
if (fd >= 0) {
|
|
|
|
struct stat st;
|
2005-04-18 20:04:43 +00:00
|
|
|
|
2008-06-14 18:32:37 +00:00
|
|
|
if (!fstat(fd, &st)) {
|
|
|
|
*size = xsize_t(st.st_size);
|
2012-02-06 16:24:52 +00:00
|
|
|
if (!*size) {
|
|
|
|
/* mmap() is forbidden on empty files */
|
2017-01-13 17:54:39 +00:00
|
|
|
error("object file %s is empty", path);
|
2012-02-06 16:24:52 +00:00
|
|
|
return NULL;
|
|
|
|
}
|
2008-06-14 18:32:37 +00:00
|
|
|
map = xmmap(NULL, *size, PROT_READ, MAP_PRIVATE, fd, 0);
|
2005-04-23 18:09:32 +00:00
|
|
|
}
|
2008-06-14 18:32:37 +00:00
|
|
|
close(fd);
|
2005-04-18 20:04:43 +00:00
|
|
|
}
|
|
|
|
return map;
|
|
|
|
}
|
|
|
|
|
2017-01-13 17:58:16 +00:00
|
|
|
void *map_sha1_file(const unsigned char *sha1, unsigned long *size)
|
|
|
|
{
|
|
|
|
return map_sha1_file_1(NULL, sha1, size);
|
|
|
|
}
|
|
|
|
|
2008-10-29 23:02:46 +00:00
|
|
|
unsigned long unpack_object_header_buffer(const unsigned char *buf,
|
|
|
|
unsigned long len, enum object_type *type, unsigned long *sizep)
|
2005-06-02 00:54:59 +00:00
|
|
|
{
|
2006-09-01 22:17:01 +00:00
|
|
|
unsigned shift;
|
Fix big left-shifts of unsigned char
Shifting 'unsigned char' or 'unsigned short' left can result in sign
extension errors, since the C integer promotion rules means that the
unsigned char/short will get implicitly promoted to a signed 'int' due to
the shift (or due to other operations).
This normally doesn't matter, but if you shift things up sufficiently, it
will now set the sign bit in 'int', and a subsequent cast to a bigger type
(eg 'long' or 'unsigned long') will now sign-extend the value despite the
original expression being unsigned.
One example of this would be something like
unsigned long size;
unsigned char c;
size += c << 24;
where despite all the variables being unsigned, 'c << 24' ends up being a
signed entity, and will get sign-extended when then doing the addition in
an 'unsigned long' type.
Since git uses 'unsigned char' pointers extensively, we actually have this
bug in a couple of places.
I may have missed some, but this is the result of looking at
git grep '[^0-9 ][ ]*<<[ ][a-z]' -- '*.c' '*.h'
git grep '<<[ ]*24'
which catches at least the common byte cases (shifting variables by a
variable amount, and shifting by 24 bits).
I also grepped for just 'unsigned char' variables in general, and
converted the ones that most obviously ended up getting implicitly cast
immediately anyway (eg hash_name(), encode_85()).
In addition to just avoiding 'unsigned char', this patch also tries to use
a common idiom for the delta header size thing. We had three different
variations on it: "& 0x7fUL" in one place (getting the sign extension
right), and "& ~0x80" and "& 0x7f" in two other places (not getting it
right). Apart from making them all just avoid using "unsigned char" at
all, I also unified them to then use a simple "& 0x7f".
I considered making a sparse extension which warns about doing implicit
casts from unsigned types to signed types, but it gets rather complex very
quickly, so this is just a hack.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-06-18 00:22:27 +00:00
|
|
|
unsigned long size, c;
|
2006-09-01 22:17:01 +00:00
|
|
|
unsigned long used = 0;
|
|
|
|
|
|
|
|
c = buf[used++];
|
|
|
|
*type = (c >> 4) & 7;
|
|
|
|
size = c & 15;
|
|
|
|
shift = 4;
|
|
|
|
while (c & 0x80) {
|
2009-07-22 21:34:34 +00:00
|
|
|
if (len <= used || bitsizeof(long) <= shift) {
|
2008-10-29 23:02:46 +00:00
|
|
|
error("bad object header");
|
2011-10-27 18:42:57 +00:00
|
|
|
size = used = 0;
|
|
|
|
break;
|
2008-10-29 23:02:46 +00:00
|
|
|
}
|
2006-09-01 22:17:01 +00:00
|
|
|
c = buf[used++];
|
|
|
|
size += (c & 0x7f) << shift;
|
|
|
|
shift += 7;
|
|
|
|
}
|
|
|
|
*sizep = size;
|
|
|
|
return used;
|
|
|
|
}
|
|
|
|
|
unpack_sha1_header(): detect malformed object header
When opening a loose object file, we often do this sequence:
- prepare a short buffer for the object header (on stack)
- call unpack_sha1_header() and have early part of the object data
inflated, enough to fill the buffer
- parse that data in the short buffer, assuming that the first part
of the object is <typename> SP <length> NUL
Because the parsing function parse_sha1_header_extended() is not
given the number of bytes inflated into the header buffer, it you
craft a file whose early part inflates a garbage sequence without SP
or NUL, and replace a loose object with it, it will end up reading
past the end of the inflated data.
To correct this, do the following four things:
- rename unpack_sha1_header() to unpack_sha1_short_header() and
have unpack_sha1_header_to_strbuf() keep calling that as its
helper function. This will detect and report zlib errors, but is
not aware of the format of a loose object (as before).
- introduce unpack_sha1_header() that calls the same helper
function, and when zlib reports it inflated OK into the buffer,
check if the inflated data has NUL. This would ensure that
parsing function will terminate within the buffer that holds the
inflated header.
- update unpack_sha1_header_to_strbuf() to check if the resulting
buffer has NUL for the same effect.
- update parse_sha1_header_extended() to make sure that its loop to
find the SP that terminates the <typename> stops at NUL.
Essentially, this makes unpack_*() functions that are asked to
unpack a loose object header to be a bit more strict and detect an
input that cannot possibly be a valid object header, even before the
parsing function kicks in.
Reported-by: Gustavo Grieco <gustavo.grieco@imag.fr>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-09-26 04:29:04 +00:00
|
|
|
static int unpack_sha1_short_header(git_zstream *stream,
|
|
|
|
unsigned char *map, unsigned long mapsize,
|
|
|
|
void *buffer, unsigned long bufsiz)
|
2006-09-01 22:17:01 +00:00
|
|
|
{
|
2005-06-02 00:54:59 +00:00
|
|
|
/* Get the data stream */
|
|
|
|
memset(stream, 0, sizeof(*stream));
|
|
|
|
stream->next_in = map;
|
|
|
|
stream->avail_in = mapsize;
|
|
|
|
stream->next_out = buffer;
|
2006-07-11 19:48:08 +00:00
|
|
|
stream->avail_out = bufsiz;
|
|
|
|
|
2009-01-08 03:54:47 +00:00
|
|
|
git_inflate_init(stream);
|
2011-06-08 18:29:01 +00:00
|
|
|
return git_inflate(stream, 0);
|
2005-06-02 00:54:59 +00:00
|
|
|
}
|
|
|
|
|
unpack_sha1_header(): detect malformed object header
When opening a loose object file, we often do this sequence:
- prepare a short buffer for the object header (on stack)
- call unpack_sha1_header() and have early part of the object data
inflated, enough to fill the buffer
- parse that data in the short buffer, assuming that the first part
of the object is <typename> SP <length> NUL
Because the parsing function parse_sha1_header_extended() is not
given the number of bytes inflated into the header buffer, it you
craft a file whose early part inflates a garbage sequence without SP
or NUL, and replace a loose object with it, it will end up reading
past the end of the inflated data.
To correct this, do the following four things:
- rename unpack_sha1_header() to unpack_sha1_short_header() and
have unpack_sha1_header_to_strbuf() keep calling that as its
helper function. This will detect and report zlib errors, but is
not aware of the format of a loose object (as before).
- introduce unpack_sha1_header() that calls the same helper
function, and when zlib reports it inflated OK into the buffer,
check if the inflated data has NUL. This would ensure that
parsing function will terminate within the buffer that holds the
inflated header.
- update unpack_sha1_header_to_strbuf() to check if the resulting
buffer has NUL for the same effect.
- update parse_sha1_header_extended() to make sure that its loop to
find the SP that terminates the <typename> stops at NUL.
Essentially, this makes unpack_*() functions that are asked to
unpack a loose object header to be a bit more strict and detect an
input that cannot possibly be a valid object header, even before the
parsing function kicks in.
Reported-by: Gustavo Grieco <gustavo.grieco@imag.fr>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-09-26 04:29:04 +00:00
|
|
|
int unpack_sha1_header(git_zstream *stream,
|
|
|
|
unsigned char *map, unsigned long mapsize,
|
|
|
|
void *buffer, unsigned long bufsiz)
|
|
|
|
{
|
|
|
|
int status = unpack_sha1_short_header(stream, map, mapsize,
|
|
|
|
buffer, bufsiz);
|
|
|
|
|
|
|
|
if (status < Z_OK)
|
|
|
|
return status;
|
|
|
|
|
|
|
|
/* Make sure we have the terminating NUL */
|
|
|
|
if (!memchr(buffer, '\0', stream->next_out - (unsigned char *)buffer))
|
|
|
|
return -1;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2015-05-03 14:29:59 +00:00
|
|
|
static int unpack_sha1_header_to_strbuf(git_zstream *stream, unsigned char *map,
|
|
|
|
unsigned long mapsize, void *buffer,
|
|
|
|
unsigned long bufsiz, struct strbuf *header)
|
|
|
|
{
|
|
|
|
int status;
|
|
|
|
|
unpack_sha1_header(): detect malformed object header
When opening a loose object file, we often do this sequence:
- prepare a short buffer for the object header (on stack)
- call unpack_sha1_header() and have early part of the object data
inflated, enough to fill the buffer
- parse that data in the short buffer, assuming that the first part
of the object is <typename> SP <length> NUL
Because the parsing function parse_sha1_header_extended() is not
given the number of bytes inflated into the header buffer, it you
craft a file whose early part inflates a garbage sequence without SP
or NUL, and replace a loose object with it, it will end up reading
past the end of the inflated data.
To correct this, do the following four things:
- rename unpack_sha1_header() to unpack_sha1_short_header() and
have unpack_sha1_header_to_strbuf() keep calling that as its
helper function. This will detect and report zlib errors, but is
not aware of the format of a loose object (as before).
- introduce unpack_sha1_header() that calls the same helper
function, and when zlib reports it inflated OK into the buffer,
check if the inflated data has NUL. This would ensure that
parsing function will terminate within the buffer that holds the
inflated header.
- update unpack_sha1_header_to_strbuf() to check if the resulting
buffer has NUL for the same effect.
- update parse_sha1_header_extended() to make sure that its loop to
find the SP that terminates the <typename> stops at NUL.
Essentially, this makes unpack_*() functions that are asked to
unpack a loose object header to be a bit more strict and detect an
input that cannot possibly be a valid object header, even before the
parsing function kicks in.
Reported-by: Gustavo Grieco <gustavo.grieco@imag.fr>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-09-26 04:29:04 +00:00
|
|
|
status = unpack_sha1_short_header(stream, map, mapsize, buffer, bufsiz);
|
|
|
|
if (status < Z_OK)
|
|
|
|
return -1;
|
2015-05-03 14:29:59 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Check if entire header is unpacked in the first iteration.
|
|
|
|
*/
|
|
|
|
if (memchr(buffer, '\0', stream->next_out - (unsigned char *)buffer))
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* buffer[0..bufsiz] was not large enough. Copy the partial
|
|
|
|
* result out to header, and then append the result of further
|
|
|
|
* reading the stream.
|
|
|
|
*/
|
|
|
|
strbuf_add(header, buffer, stream->next_out - (unsigned char *)buffer);
|
|
|
|
stream->next_out = buffer;
|
|
|
|
stream->avail_out = bufsiz;
|
|
|
|
|
|
|
|
do {
|
|
|
|
status = git_inflate(stream, 0);
|
|
|
|
strbuf_add(header, buffer, stream->next_out - (unsigned char *)buffer);
|
|
|
|
if (memchr(buffer, '\0', stream->next_out - (unsigned char *)buffer))
|
|
|
|
return 0;
|
|
|
|
stream->next_out = buffer;
|
|
|
|
stream->avail_out = bufsiz;
|
|
|
|
} while (status != Z_STREAM_END);
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
2011-06-10 18:52:15 +00:00
|
|
|
static void *unpack_sha1_rest(git_zstream *stream, void *buffer, unsigned long size, const unsigned char *sha1)
|
2005-06-02 14:57:25 +00:00
|
|
|
{
|
|
|
|
int bytes = strlen(buffer) + 1;
|
2010-01-26 18:24:14 +00:00
|
|
|
unsigned char *buf = xmallocz(size);
|
2006-07-11 19:48:08 +00:00
|
|
|
unsigned long n;
|
2007-03-05 08:21:37 +00:00
|
|
|
int status = Z_OK;
|
2005-06-02 14:57:25 +00:00
|
|
|
|
2006-07-11 19:48:08 +00:00
|
|
|
n = stream->total_out - bytes;
|
|
|
|
if (n > size)
|
|
|
|
n = size;
|
|
|
|
memcpy(buf, (char *) buffer + bytes, n);
|
|
|
|
bytes = n;
|
2007-03-20 05:49:53 +00:00
|
|
|
if (bytes <= size) {
|
|
|
|
/*
|
|
|
|
* The above condition must be (bytes <= size), not
|
|
|
|
* (bytes < size). In other words, even though we
|
2011-05-15 19:16:03 +00:00
|
|
|
* expect no more output and set avail_out to zero,
|
2007-03-20 05:49:53 +00:00
|
|
|
* the input zlib stream may have bytes that express
|
|
|
|
* "this concludes the stream", and we *do* want to
|
|
|
|
* eat that input.
|
|
|
|
*
|
|
|
|
* Otherwise we would not be able to test that we
|
|
|
|
* consumed all the input to reach the expected size;
|
|
|
|
* we also want to check that zlib tells us that all
|
|
|
|
* went well with status == Z_STREAM_END at the end.
|
|
|
|
*/
|
2005-06-02 14:57:25 +00:00
|
|
|
stream->next_out = buf + bytes;
|
|
|
|
stream->avail_out = size - bytes;
|
2007-03-05 08:21:37 +00:00
|
|
|
while (status == Z_OK)
|
2009-01-08 03:54:47 +00:00
|
|
|
status = git_inflate(stream, Z_FINISH);
|
2005-06-02 14:57:25 +00:00
|
|
|
}
|
2007-03-20 05:49:53 +00:00
|
|
|
if (status == Z_STREAM_END && !stream->avail_in) {
|
2009-01-08 03:54:47 +00:00
|
|
|
git_inflate_end(stream);
|
2007-03-05 08:21:37 +00:00
|
|
|
return buf;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (status < 0)
|
|
|
|
error("corrupt loose object '%s'", sha1_to_hex(sha1));
|
|
|
|
else if (stream->avail_in)
|
|
|
|
error("garbage at end of loose object '%s'",
|
|
|
|
sha1_to_hex(sha1));
|
|
|
|
free(buf);
|
|
|
|
return NULL;
|
2005-06-02 14:57:25 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We used to just use "sscanf()", but that's actually way
|
|
|
|
* too permissive for what we want to check. So do an anal
|
|
|
|
* object header parse by hand.
|
|
|
|
*/
|
2015-05-03 14:29:59 +00:00
|
|
|
static int parse_sha1_header_extended(const char *hdr, struct object_info *oi,
|
|
|
|
unsigned int flags)
|
2005-06-02 14:57:25 +00:00
|
|
|
{
|
2015-05-03 14:29:59 +00:00
|
|
|
const char *type_buf = hdr;
|
2005-06-02 14:57:25 +00:00
|
|
|
unsigned long size;
|
2015-05-03 14:29:59 +00:00
|
|
|
int type, type_len = 0;
|
2005-06-02 14:57:25 +00:00
|
|
|
|
|
|
|
/*
|
2015-05-03 14:29:59 +00:00
|
|
|
* The type can be of any size but is followed by
|
2007-02-26 19:55:59 +00:00
|
|
|
* a space.
|
2005-06-02 14:57:25 +00:00
|
|
|
*/
|
|
|
|
for (;;) {
|
|
|
|
char c = *hdr++;
|
unpack_sha1_header(): detect malformed object header
When opening a loose object file, we often do this sequence:
- prepare a short buffer for the object header (on stack)
- call unpack_sha1_header() and have early part of the object data
inflated, enough to fill the buffer
- parse that data in the short buffer, assuming that the first part
of the object is <typename> SP <length> NUL
Because the parsing function parse_sha1_header_extended() is not
given the number of bytes inflated into the header buffer, it you
craft a file whose early part inflates a garbage sequence without SP
or NUL, and replace a loose object with it, it will end up reading
past the end of the inflated data.
To correct this, do the following four things:
- rename unpack_sha1_header() to unpack_sha1_short_header() and
have unpack_sha1_header_to_strbuf() keep calling that as its
helper function. This will detect and report zlib errors, but is
not aware of the format of a loose object (as before).
- introduce unpack_sha1_header() that calls the same helper
function, and when zlib reports it inflated OK into the buffer,
check if the inflated data has NUL. This would ensure that
parsing function will terminate within the buffer that holds the
inflated header.
- update unpack_sha1_header_to_strbuf() to check if the resulting
buffer has NUL for the same effect.
- update parse_sha1_header_extended() to make sure that its loop to
find the SP that terminates the <typename> stops at NUL.
Essentially, this makes unpack_*() functions that are asked to
unpack a loose object header to be a bit more strict and detect an
input that cannot possibly be a valid object header, even before the
parsing function kicks in.
Reported-by: Gustavo Grieco <gustavo.grieco@imag.fr>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-09-26 04:29:04 +00:00
|
|
|
if (!c)
|
|
|
|
return -1;
|
2005-06-02 14:57:25 +00:00
|
|
|
if (c == ' ')
|
|
|
|
break;
|
2015-05-03 14:29:59 +00:00
|
|
|
type_len++;
|
2005-06-02 14:57:25 +00:00
|
|
|
}
|
2015-05-03 14:29:59 +00:00
|
|
|
|
|
|
|
type = type_from_string_gently(type_buf, type_len, 1);
|
|
|
|
if (oi->typename)
|
|
|
|
strbuf_add(oi->typename, type_buf, type_len);
|
|
|
|
/*
|
|
|
|
* Set type to 0 if its an unknown object and
|
2016-08-09 08:53:38 +00:00
|
|
|
* we're obtaining the type using '--allow-unknown-type'
|
2015-05-03 14:29:59 +00:00
|
|
|
* option.
|
|
|
|
*/
|
|
|
|
if ((flags & LOOKUP_UNKNOWN_OBJECT) && (type < 0))
|
|
|
|
type = 0;
|
|
|
|
else if (type < 0)
|
|
|
|
die("invalid object type");
|
|
|
|
if (oi->typep)
|
|
|
|
*oi->typep = type;
|
2005-06-02 14:57:25 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* The length must follow immediately, and be in canonical
|
|
|
|
* decimal format (ie "010" is not valid).
|
|
|
|
*/
|
|
|
|
size = *hdr++ - '0';
|
|
|
|
if (size > 9)
|
|
|
|
return -1;
|
|
|
|
if (size) {
|
|
|
|
for (;;) {
|
|
|
|
unsigned long c = *hdr - '0';
|
|
|
|
if (c > 9)
|
|
|
|
break;
|
|
|
|
hdr++;
|
|
|
|
size = size * 10 + c;
|
|
|
|
}
|
|
|
|
}
|
2015-05-03 14:29:59 +00:00
|
|
|
|
|
|
|
if (oi->sizep)
|
|
|
|
*oi->sizep = size;
|
2005-06-02 14:57:25 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* The length must be followed by a zero byte
|
|
|
|
*/
|
2015-05-03 14:29:59 +00:00
|
|
|
return *hdr ? -1 : type;
|
|
|
|
}
|
|
|
|
|
|
|
|
int parse_sha1_header(const char *hdr, unsigned long *sizep)
|
|
|
|
{
|
provide an initializer for "struct object_info"
An all-zero initializer is fine for this struct, but because
the first element is a pointer, call sites need to know to
use "NULL" instead of "0". Otherwise some static checkers
like "sparse" will complain; see d099b71 (Fix some sparse
warnings, 2013-07-18) for example. So let's provide an
initializer to make this easier to get right.
But let's also comment that memset() to zero is explicitly
OK[1]. One of the callers embeds object_info in another
struct which is initialized via memset (expand_data in
builtin/cat-file.c). Since our subset of C doesn't allow
assignment from a compound literal, handling this in any
other way is awkward, so we'd like to keep the ability to
initialize by memset(). By documenting this property, it
should make anybody who wants to change the initializer
think twice before doing so.
There's one other caller of interest. In parse_sha1_header(),
we did not initialize the struct fully in the first place.
This turned out not to be a bug because the sub-function it
calls does not look at any other fields except the ones we
did initialize. But that assumption might not hold in the
future, so it's a dangerous construct. This patch switches
it to initializing the whole struct, which protects us
against unexpected reads of the other fields.
[1] Obviously using memset() to initialize a pointer
violates the C standard, but we long ago decided that it
was an acceptable tradeoff in the real world.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-11 09:24:35 +00:00
|
|
|
struct object_info oi = OBJECT_INFO_INIT;
|
2015-05-03 14:29:59 +00:00
|
|
|
|
|
|
|
oi.sizep = sizep;
|
|
|
|
return parse_sha1_header_extended(hdr, &oi, LOOKUP_REPLACE_OBJECT);
|
2005-06-02 14:57:25 +00:00
|
|
|
}
|
|
|
|
|
2007-03-05 08:21:37 +00:00
|
|
|
static void *unpack_sha1_file(void *map, unsigned long mapsize, enum object_type *type, unsigned long *size, const unsigned char *sha1)
|
2005-04-18 20:04:43 +00:00
|
|
|
{
|
2005-06-02 14:57:25 +00:00
|
|
|
int ret;
|
2011-06-10 18:52:15 +00:00
|
|
|
git_zstream stream;
|
2005-06-02 14:57:25 +00:00
|
|
|
char hdr[8192];
|
2005-04-18 20:04:43 +00:00
|
|
|
|
2005-06-02 14:57:25 +00:00
|
|
|
ret = unpack_sha1_header(&stream, map, mapsize, hdr, sizeof(hdr));
|
2007-02-26 19:55:59 +00:00
|
|
|
if (ret < Z_OK || (*type = parse_sha1_header(hdr, size)) < 0)
|
2005-04-18 20:04:43 +00:00
|
|
|
return NULL;
|
|
|
|
|
2007-03-05 08:21:37 +00:00
|
|
|
return unpack_sha1_rest(&stream, hdr, *size, sha1);
|
2005-04-18 20:04:43 +00:00
|
|
|
}
|
|
|
|
|
2007-04-16 16:31:56 +00:00
|
|
|
unsigned long get_size_from_delta(struct packed_git *p,
|
|
|
|
struct pack_window **w_curs,
|
|
|
|
off_t curpos)
|
|
|
|
{
|
|
|
|
const unsigned char *data;
|
|
|
|
unsigned char delta_head[20], *in;
|
2011-06-10 18:52:15 +00:00
|
|
|
git_zstream stream;
|
2007-04-16 16:31:56 +00:00
|
|
|
int st;
|
|
|
|
|
|
|
|
memset(&stream, 0, sizeof(stream));
|
|
|
|
stream.next_out = delta_head;
|
|
|
|
stream.avail_out = sizeof(delta_head);
|
|
|
|
|
2009-01-08 03:54:47 +00:00
|
|
|
git_inflate_init(&stream);
|
2007-04-16 16:31:56 +00:00
|
|
|
do {
|
|
|
|
in = use_pack(p, w_curs, curpos, &stream.avail_in);
|
|
|
|
stream.next_in = in;
|
2009-01-08 03:54:47 +00:00
|
|
|
st = git_inflate(&stream, Z_FINISH);
|
2007-04-16 16:31:56 +00:00
|
|
|
curpos += stream.next_in - in;
|
|
|
|
} while ((st == Z_OK || st == Z_BUF_ERROR) &&
|
|
|
|
stream.total_out < sizeof(delta_head));
|
2009-01-08 03:54:47 +00:00
|
|
|
git_inflate_end(&stream);
|
2008-10-29 23:02:47 +00:00
|
|
|
if ((st != Z_STREAM_END) && stream.total_out != sizeof(delta_head)) {
|
|
|
|
error("delta data unpack-initial failed");
|
|
|
|
return 0;
|
|
|
|
}
|
2007-04-16 16:31:56 +00:00
|
|
|
|
|
|
|
/* Examine the initial part of the delta to figure out
|
|
|
|
* the result size.
|
|
|
|
*/
|
|
|
|
data = delta_head;
|
|
|
|
|
|
|
|
/* ignore base size */
|
|
|
|
get_delta_hdr_size(&data, delta_head+sizeof(delta_head));
|
|
|
|
|
|
|
|
/* Read the result size */
|
|
|
|
return get_delta_hdr_size(&data, delta_head+sizeof(delta_head));
|
|
|
|
}
|
|
|
|
|
2007-03-07 01:44:30 +00:00
|
|
|
static off_t get_delta_base(struct packed_git *p,
|
2006-12-23 07:34:08 +00:00
|
|
|
struct pack_window **w_curs,
|
2007-03-07 01:44:30 +00:00
|
|
|
off_t *curpos,
|
2007-02-26 19:55:59 +00:00
|
|
|
enum object_type type,
|
2007-03-07 01:44:30 +00:00
|
|
|
off_t delta_obj_offset)
|
2006-09-21 04:06:49 +00:00
|
|
|
{
|
2007-02-26 19:55:56 +00:00
|
|
|
unsigned char *base_info = use_pack(p, w_curs, *curpos, NULL);
|
2007-03-07 01:44:30 +00:00
|
|
|
off_t base_offset;
|
2006-09-21 04:06:49 +00:00
|
|
|
|
2006-12-23 07:34:18 +00:00
|
|
|
/* use_pack() assured us we have [base_info, base_info + 20)
|
|
|
|
* as a range that we can look at without walking off the
|
|
|
|
* end of the mapped window. Its actually the hash size
|
|
|
|
* that is assured. An OFS_DELTA longer than the hash size
|
|
|
|
* is stupid, as then a REF_DELTA would be smaller to store.
|
|
|
|
*/
|
2007-02-26 19:55:59 +00:00
|
|
|
if (type == OBJ_OFS_DELTA) {
|
2006-09-21 04:06:49 +00:00
|
|
|
unsigned used = 0;
|
|
|
|
unsigned char c = base_info[used++];
|
|
|
|
base_offset = c & 127;
|
|
|
|
while (c & 128) {
|
|
|
|
base_offset += 1;
|
2007-04-09 05:06:29 +00:00
|
|
|
if (!base_offset || MSB(base_offset, 7))
|
2008-06-24 01:23:39 +00:00
|
|
|
return 0; /* overflow */
|
2006-09-21 04:06:49 +00:00
|
|
|
c = base_info[used++];
|
|
|
|
base_offset = (base_offset << 7) + (c & 127);
|
|
|
|
}
|
|
|
|
base_offset = delta_obj_offset - base_offset;
|
2008-10-29 23:02:45 +00:00
|
|
|
if (base_offset <= 0 || base_offset >= delta_obj_offset)
|
2008-06-24 01:23:39 +00:00
|
|
|
return 0; /* out of bound */
|
2007-02-26 19:55:56 +00:00
|
|
|
*curpos += used;
|
2007-02-26 19:55:59 +00:00
|
|
|
} else if (type == OBJ_REF_DELTA) {
|
2006-09-21 04:06:49 +00:00
|
|
|
/* The base entry _must_ be in the same pack */
|
|
|
|
base_offset = find_pack_entry_one(base_info, p);
|
2007-02-26 19:55:56 +00:00
|
|
|
*curpos += 20;
|
2006-09-21 04:06:49 +00:00
|
|
|
} else
|
|
|
|
die("I am totally screwed");
|
2007-02-26 19:55:56 +00:00
|
|
|
return base_offset;
|
2006-09-21 04:06:49 +00:00
|
|
|
}
|
|
|
|
|
2013-12-21 14:24:20 +00:00
|
|
|
/*
|
|
|
|
* Like get_delta_base above, but we return the sha1 instead of the pack
|
|
|
|
* offset. This means it is cheaper for REF deltas (we do not have to do
|
|
|
|
* the final object lookup), but more expensive for OFS deltas (we
|
|
|
|
* have to load the revidx to convert the offset back into a sha1).
|
|
|
|
*/
|
|
|
|
static const unsigned char *get_delta_base_sha1(struct packed_git *p,
|
|
|
|
struct pack_window **w_curs,
|
|
|
|
off_t curpos,
|
|
|
|
enum object_type type,
|
|
|
|
off_t delta_obj_offset)
|
|
|
|
{
|
|
|
|
if (type == OBJ_REF_DELTA) {
|
|
|
|
unsigned char *base = use_pack(p, w_curs, curpos, NULL);
|
|
|
|
return base;
|
|
|
|
} else if (type == OBJ_OFS_DELTA) {
|
|
|
|
struct revindex_entry *revidx;
|
|
|
|
off_t base_offset = get_delta_base(p, w_curs, &curpos,
|
|
|
|
type, delta_obj_offset);
|
|
|
|
|
|
|
|
if (!base_offset)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
revidx = find_pack_revindex(p, base_offset);
|
|
|
|
if (!revidx)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
return nth_packed_object_sha1(p, revidx->nr);
|
|
|
|
} else
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
2011-05-13 22:33:33 +00:00
|
|
|
int unpack_object_header(struct packed_git *p,
|
|
|
|
struct pack_window **w_curs,
|
|
|
|
off_t *curpos,
|
|
|
|
unsigned long *sizep)
|
2005-06-28 21:21:02 +00:00
|
|
|
{
|
2006-12-23 07:34:08 +00:00
|
|
|
unsigned char *base;
|
2011-06-10 18:52:15 +00:00
|
|
|
unsigned long left;
|
2006-09-01 22:17:01 +00:00
|
|
|
unsigned long used;
|
2007-02-26 19:55:56 +00:00
|
|
|
enum object_type type;
|
2005-06-28 21:21:02 +00:00
|
|
|
|
2006-12-23 07:34:18 +00:00
|
|
|
/* use_pack() assures us we have [base, base + 20) available
|
2011-04-13 15:39:40 +00:00
|
|
|
* as a range that we can look at. (Its actually the hash
|
2007-02-04 04:49:16 +00:00
|
|
|
* size that is assured.) With our object header encoding
|
2006-12-23 07:34:18 +00:00
|
|
|
* the maximum deflated object size is 2^137, which is just
|
|
|
|
* insane, so we know won't exceed what we have been given.
|
|
|
|
*/
|
2007-02-26 19:55:56 +00:00
|
|
|
base = use_pack(p, w_curs, *curpos, &left);
|
2008-10-29 23:02:46 +00:00
|
|
|
used = unpack_object_header_buffer(base, left, &type, sizep);
|
|
|
|
if (!used) {
|
|
|
|
type = OBJ_BAD;
|
|
|
|
} else
|
|
|
|
*curpos += used;
|
2006-09-01 22:17:01 +00:00
|
|
|
|
2007-02-26 19:55:56 +00:00
|
|
|
return type;
|
2005-06-28 21:21:02 +00:00
|
|
|
}
|
|
|
|
|
sha1_file: remove recursion in packed_object_info
packed_object_info() and packed_delta_info() were mutually recursive.
The former would handle ordinary types and defer deltas to the latter;
the latter would use the former to resolve the delta base.
This arrangement, however, leads to trouble with threaded index-pack
and long delta chains on platforms where thread stacks are small, as
happened on OS X (512kB thread stacks by default) with the chromium
repo.
The task of the two functions is not all that hard to describe without
any recursion, however. It proceeds in three steps:
- determine the representation type and size, based on the outermost
object (delta or not)
- follow through the delta chain, if any
- determine the object type from what is found at the end of the delta
chain
The only complication stems from the error recovery. If parsing fails
at any step, we want to mark that object (within the pack) as bad and
try getting the corresponding SHA1 from elsewhere. If that also
fails, we want to repeat this process back up the delta chain until we
find a reasonable solution or conclude that there is no way to
reconstruct the object. (This is conveniently checked by t5303.)
To achieve that within the pack, we keep track of the entire delta
chain in a stack. When things go sour, we process that stack from the
top, marking entries as bad and attempting to re-resolve by sha1. To
avoid excessive malloc(), the stack starts out with a small
stack-allocated array. The choice of 64 is based on the default of
pack.depth, which is 50, in the hope that it covers "most" delta
chains without any need for malloc().
It's much harder to make the actual re-resolving by sha1 nonrecursive,
so we skip that. If you can't afford *that* recursion, your
corruption problems are more serious than your stack size problems.
Reported-by: Stefan Zager <szager@google.com>
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-03-25 18:07:39 +00:00
|
|
|
static int retry_bad_packed_offset(struct packed_git *p, off_t obj_offset)
|
|
|
|
{
|
|
|
|
int type;
|
|
|
|
struct revindex_entry *revidx;
|
|
|
|
const unsigned char *sha1;
|
|
|
|
revidx = find_pack_revindex(p, obj_offset);
|
|
|
|
if (!revidx)
|
|
|
|
return OBJ_BAD;
|
|
|
|
sha1 = nth_packed_object_sha1(p, revidx->nr);
|
|
|
|
mark_bad_packed_object(p, sha1);
|
|
|
|
type = sha1_object_info(sha1, NULL);
|
|
|
|
if (type <= OBJ_NONE)
|
|
|
|
return OBJ_BAD;
|
|
|
|
return type;
|
|
|
|
}
|
|
|
|
|
|
|
|
#define POI_STACK_PREALLOC 64
|
|
|
|
|
2013-07-12 06:31:57 +00:00
|
|
|
static enum object_type packed_to_object_type(struct packed_git *p,
|
|
|
|
off_t obj_offset,
|
|
|
|
enum object_type type,
|
|
|
|
struct pack_window **w_curs,
|
|
|
|
off_t curpos)
|
2005-06-27 10:35:33 +00:00
|
|
|
{
|
sha1_file: remove recursion in packed_object_info
packed_object_info() and packed_delta_info() were mutually recursive.
The former would handle ordinary types and defer deltas to the latter;
the latter would use the former to resolve the delta base.
This arrangement, however, leads to trouble with threaded index-pack
and long delta chains on platforms where thread stacks are small, as
happened on OS X (512kB thread stacks by default) with the chromium
repo.
The task of the two functions is not all that hard to describe without
any recursion, however. It proceeds in three steps:
- determine the representation type and size, based on the outermost
object (delta or not)
- follow through the delta chain, if any
- determine the object type from what is found at the end of the delta
chain
The only complication stems from the error recovery. If parsing fails
at any step, we want to mark that object (within the pack) as bad and
try getting the corresponding SHA1 from elsewhere. If that also
fails, we want to repeat this process back up the delta chain until we
find a reasonable solution or conclude that there is no way to
reconstruct the object. (This is conveniently checked by t5303.)
To achieve that within the pack, we keep track of the entire delta
chain in a stack. When things go sour, we process that stack from the
top, marking entries as bad and attempting to re-resolve by sha1. To
avoid excessive malloc(), the stack starts out with a small
stack-allocated array. The choice of 64 is based on the default of
pack.depth, which is 50, in the hope that it covers "most" delta
chains without any need for malloc().
It's much harder to make the actual re-resolving by sha1 nonrecursive,
so we skip that. If you can't afford *that* recursion, your
corruption problems are more serious than your stack size problems.
Reported-by: Stefan Zager <szager@google.com>
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-03-25 18:07:39 +00:00
|
|
|
off_t small_poi_stack[POI_STACK_PREALLOC];
|
|
|
|
off_t *poi_stack = small_poi_stack;
|
|
|
|
int poi_stack_nr = 0, poi_stack_alloc = POI_STACK_PREALLOC;
|
2005-06-28 06:58:08 +00:00
|
|
|
|
sha1_file: remove recursion in packed_object_info
packed_object_info() and packed_delta_info() were mutually recursive.
The former would handle ordinary types and defer deltas to the latter;
the latter would use the former to resolve the delta base.
This arrangement, however, leads to trouble with threaded index-pack
and long delta chains on platforms where thread stacks are small, as
happened on OS X (512kB thread stacks by default) with the chromium
repo.
The task of the two functions is not all that hard to describe without
any recursion, however. It proceeds in three steps:
- determine the representation type and size, based on the outermost
object (delta or not)
- follow through the delta chain, if any
- determine the object type from what is found at the end of the delta
chain
The only complication stems from the error recovery. If parsing fails
at any step, we want to mark that object (within the pack) as bad and
try getting the corresponding SHA1 from elsewhere. If that also
fails, we want to repeat this process back up the delta chain until we
find a reasonable solution or conclude that there is no way to
reconstruct the object. (This is conveniently checked by t5303.)
To achieve that within the pack, we keep track of the entire delta
chain in a stack. When things go sour, we process that stack from the
top, marking entries as bad and attempting to re-resolve by sha1. To
avoid excessive malloc(), the stack starts out with a small
stack-allocated array. The choice of 64 is based on the default of
pack.depth, which is 50, in the hope that it covers "most" delta
chains without any need for malloc().
It's much harder to make the actual re-resolving by sha1 nonrecursive,
so we skip that. If you can't afford *that* recursion, your
corruption problems are more serious than your stack size problems.
Reported-by: Stefan Zager <szager@google.com>
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-03-25 18:07:39 +00:00
|
|
|
while (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {
|
|
|
|
off_t base_offset;
|
2013-07-12 06:31:57 +00:00
|
|
|
unsigned long size;
|
sha1_file: remove recursion in packed_object_info
packed_object_info() and packed_delta_info() were mutually recursive.
The former would handle ordinary types and defer deltas to the latter;
the latter would use the former to resolve the delta base.
This arrangement, however, leads to trouble with threaded index-pack
and long delta chains on platforms where thread stacks are small, as
happened on OS X (512kB thread stacks by default) with the chromium
repo.
The task of the two functions is not all that hard to describe without
any recursion, however. It proceeds in three steps:
- determine the representation type and size, based on the outermost
object (delta or not)
- follow through the delta chain, if any
- determine the object type from what is found at the end of the delta
chain
The only complication stems from the error recovery. If parsing fails
at any step, we want to mark that object (within the pack) as bad and
try getting the corresponding SHA1 from elsewhere. If that also
fails, we want to repeat this process back up the delta chain until we
find a reasonable solution or conclude that there is no way to
reconstruct the object. (This is conveniently checked by t5303.)
To achieve that within the pack, we keep track of the entire delta
chain in a stack. When things go sour, we process that stack from the
top, marking entries as bad and attempting to re-resolve by sha1. To
avoid excessive malloc(), the stack starts out with a small
stack-allocated array. The choice of 64 is based on the default of
pack.depth, which is 50, in the hope that it covers "most" delta
chains without any need for malloc().
It's much harder to make the actual re-resolving by sha1 nonrecursive,
so we skip that. If you can't afford *that* recursion, your
corruption problems are more serious than your stack size problems.
Reported-by: Stefan Zager <szager@google.com>
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-03-25 18:07:39 +00:00
|
|
|
/* Push the object we're going to leave behind */
|
|
|
|
if (poi_stack_nr >= poi_stack_alloc && poi_stack == small_poi_stack) {
|
|
|
|
poi_stack_alloc = alloc_nr(poi_stack_nr);
|
2016-02-22 22:44:25 +00:00
|
|
|
ALLOC_ARRAY(poi_stack, poi_stack_alloc);
|
sha1_file: remove recursion in packed_object_info
packed_object_info() and packed_delta_info() were mutually recursive.
The former would handle ordinary types and defer deltas to the latter;
the latter would use the former to resolve the delta base.
This arrangement, however, leads to trouble with threaded index-pack
and long delta chains on platforms where thread stacks are small, as
happened on OS X (512kB thread stacks by default) with the chromium
repo.
The task of the two functions is not all that hard to describe without
any recursion, however. It proceeds in three steps:
- determine the representation type and size, based on the outermost
object (delta or not)
- follow through the delta chain, if any
- determine the object type from what is found at the end of the delta
chain
The only complication stems from the error recovery. If parsing fails
at any step, we want to mark that object (within the pack) as bad and
try getting the corresponding SHA1 from elsewhere. If that also
fails, we want to repeat this process back up the delta chain until we
find a reasonable solution or conclude that there is no way to
reconstruct the object. (This is conveniently checked by t5303.)
To achieve that within the pack, we keep track of the entire delta
chain in a stack. When things go sour, we process that stack from the
top, marking entries as bad and attempting to re-resolve by sha1. To
avoid excessive malloc(), the stack starts out with a small
stack-allocated array. The choice of 64 is based on the default of
pack.depth, which is 50, in the hope that it covers "most" delta
chains without any need for malloc().
It's much harder to make the actual re-resolving by sha1 nonrecursive,
so we skip that. If you can't afford *that* recursion, your
corruption problems are more serious than your stack size problems.
Reported-by: Stefan Zager <szager@google.com>
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-03-25 18:07:39 +00:00
|
|
|
memcpy(poi_stack, small_poi_stack, sizeof(off_t)*poi_stack_nr);
|
|
|
|
} else {
|
|
|
|
ALLOC_GROW(poi_stack, poi_stack_nr+1, poi_stack_alloc);
|
|
|
|
}
|
|
|
|
poi_stack[poi_stack_nr++] = obj_offset;
|
|
|
|
/* If parsing the base offset fails, just unwind */
|
2013-07-12 06:31:57 +00:00
|
|
|
base_offset = get_delta_base(p, w_curs, &curpos, type, obj_offset);
|
sha1_file: remove recursion in packed_object_info
packed_object_info() and packed_delta_info() were mutually recursive.
The former would handle ordinary types and defer deltas to the latter;
the latter would use the former to resolve the delta base.
This arrangement, however, leads to trouble with threaded index-pack
and long delta chains on platforms where thread stacks are small, as
happened on OS X (512kB thread stacks by default) with the chromium
repo.
The task of the two functions is not all that hard to describe without
any recursion, however. It proceeds in three steps:
- determine the representation type and size, based on the outermost
object (delta or not)
- follow through the delta chain, if any
- determine the object type from what is found at the end of the delta
chain
The only complication stems from the error recovery. If parsing fails
at any step, we want to mark that object (within the pack) as bad and
try getting the corresponding SHA1 from elsewhere. If that also
fails, we want to repeat this process back up the delta chain until we
find a reasonable solution or conclude that there is no way to
reconstruct the object. (This is conveniently checked by t5303.)
To achieve that within the pack, we keep track of the entire delta
chain in a stack. When things go sour, we process that stack from the
top, marking entries as bad and attempting to re-resolve by sha1. To
avoid excessive malloc(), the stack starts out with a small
stack-allocated array. The choice of 64 is based on the default of
pack.depth, which is 50, in the hope that it covers "most" delta
chains without any need for malloc().
It's much harder to make the actual re-resolving by sha1 nonrecursive,
so we skip that. If you can't afford *that* recursion, your
corruption problems are more serious than your stack size problems.
Reported-by: Stefan Zager <szager@google.com>
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-03-25 18:07:39 +00:00
|
|
|
if (!base_offset)
|
|
|
|
goto unwind;
|
|
|
|
curpos = obj_offset = base_offset;
|
2013-07-12 06:31:57 +00:00
|
|
|
type = unpack_object_header(p, w_curs, &curpos, &size);
|
sha1_file: remove recursion in packed_object_info
packed_object_info() and packed_delta_info() were mutually recursive.
The former would handle ordinary types and defer deltas to the latter;
the latter would use the former to resolve the delta base.
This arrangement, however, leads to trouble with threaded index-pack
and long delta chains on platforms where thread stacks are small, as
happened on OS X (512kB thread stacks by default) with the chromium
repo.
The task of the two functions is not all that hard to describe without
any recursion, however. It proceeds in three steps:
- determine the representation type and size, based on the outermost
object (delta or not)
- follow through the delta chain, if any
- determine the object type from what is found at the end of the delta
chain
The only complication stems from the error recovery. If parsing fails
at any step, we want to mark that object (within the pack) as bad and
try getting the corresponding SHA1 from elsewhere. If that also
fails, we want to repeat this process back up the delta chain until we
find a reasonable solution or conclude that there is no way to
reconstruct the object. (This is conveniently checked by t5303.)
To achieve that within the pack, we keep track of the entire delta
chain in a stack. When things go sour, we process that stack from the
top, marking entries as bad and attempting to re-resolve by sha1. To
avoid excessive malloc(), the stack starts out with a small
stack-allocated array. The choice of 64 is based on the default of
pack.depth, which is 50, in the hope that it covers "most" delta
chains without any need for malloc().
It's much harder to make the actual re-resolving by sha1 nonrecursive,
so we skip that. If you can't afford *that* recursion, your
corruption problems are more serious than your stack size problems.
Reported-by: Stefan Zager <szager@google.com>
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-03-25 18:07:39 +00:00
|
|
|
if (type <= OBJ_NONE) {
|
|
|
|
/* If getting the base itself fails, we first
|
|
|
|
* retry the base, otherwise unwind */
|
|
|
|
type = retry_bad_packed_offset(p, base_offset);
|
|
|
|
if (type > OBJ_NONE)
|
|
|
|
goto out;
|
|
|
|
goto unwind;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2007-02-26 19:55:59 +00:00
|
|
|
switch (type) {
|
sha1_file: remove recursion in packed_object_info
packed_object_info() and packed_delta_info() were mutually recursive.
The former would handle ordinary types and defer deltas to the latter;
the latter would use the former to resolve the delta base.
This arrangement, however, leads to trouble with threaded index-pack
and long delta chains on platforms where thread stacks are small, as
happened on OS X (512kB thread stacks by default) with the chromium
repo.
The task of the two functions is not all that hard to describe without
any recursion, however. It proceeds in three steps:
- determine the representation type and size, based on the outermost
object (delta or not)
- follow through the delta chain, if any
- determine the object type from what is found at the end of the delta
chain
The only complication stems from the error recovery. If parsing fails
at any step, we want to mark that object (within the pack) as bad and
try getting the corresponding SHA1 from elsewhere. If that also
fails, we want to repeat this process back up the delta chain until we
find a reasonable solution or conclude that there is no way to
reconstruct the object. (This is conveniently checked by t5303.)
To achieve that within the pack, we keep track of the entire delta
chain in a stack. When things go sour, we process that stack from the
top, marking entries as bad and attempting to re-resolve by sha1. To
avoid excessive malloc(), the stack starts out with a small
stack-allocated array. The choice of 64 is based on the default of
pack.depth, which is 50, in the hope that it covers "most" delta
chains without any need for malloc().
It's much harder to make the actual re-resolving by sha1 nonrecursive,
so we skip that. If you can't afford *that* recursion, your
corruption problems are more serious than your stack size problems.
Reported-by: Stefan Zager <szager@google.com>
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-03-25 18:07:39 +00:00
|
|
|
case OBJ_BAD:
|
2005-06-28 21:21:02 +00:00
|
|
|
case OBJ_COMMIT:
|
|
|
|
case OBJ_TREE:
|
|
|
|
case OBJ_BLOB:
|
|
|
|
case OBJ_TAG:
|
2005-06-28 16:58:23 +00:00
|
|
|
break;
|
2005-06-27 10:35:33 +00:00
|
|
|
default:
|
2008-10-29 23:02:47 +00:00
|
|
|
error("unknown object type %i at offset %"PRIuMAX" in %s",
|
|
|
|
type, (uintmax_t)obj_offset, p->pack_name);
|
|
|
|
type = OBJ_BAD;
|
2005-06-27 10:35:33 +00:00
|
|
|
}
|
sha1_file: remove recursion in packed_object_info
packed_object_info() and packed_delta_info() were mutually recursive.
The former would handle ordinary types and defer deltas to the latter;
the latter would use the former to resolve the delta base.
This arrangement, however, leads to trouble with threaded index-pack
and long delta chains on platforms where thread stacks are small, as
happened on OS X (512kB thread stacks by default) with the chromium
repo.
The task of the two functions is not all that hard to describe without
any recursion, however. It proceeds in three steps:
- determine the representation type and size, based on the outermost
object (delta or not)
- follow through the delta chain, if any
- determine the object type from what is found at the end of the delta
chain
The only complication stems from the error recovery. If parsing fails
at any step, we want to mark that object (within the pack) as bad and
try getting the corresponding SHA1 from elsewhere. If that also
fails, we want to repeat this process back up the delta chain until we
find a reasonable solution or conclude that there is no way to
reconstruct the object. (This is conveniently checked by t5303.)
To achieve that within the pack, we keep track of the entire delta
chain in a stack. When things go sour, we process that stack from the
top, marking entries as bad and attempting to re-resolve by sha1. To
avoid excessive malloc(), the stack starts out with a small
stack-allocated array. The choice of 64 is based on the default of
pack.depth, which is 50, in the hope that it covers "most" delta
chains without any need for malloc().
It's much harder to make the actual re-resolving by sha1 nonrecursive,
so we skip that. If you can't afford *that* recursion, your
corruption problems are more serious than your stack size problems.
Reported-by: Stefan Zager <szager@google.com>
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-03-25 18:07:39 +00:00
|
|
|
|
|
|
|
out:
|
|
|
|
if (poi_stack != small_poi_stack)
|
|
|
|
free(poi_stack);
|
2007-02-26 19:55:59 +00:00
|
|
|
return type;
|
sha1_file: remove recursion in packed_object_info
packed_object_info() and packed_delta_info() were mutually recursive.
The former would handle ordinary types and defer deltas to the latter;
the latter would use the former to resolve the delta base.
This arrangement, however, leads to trouble with threaded index-pack
and long delta chains on platforms where thread stacks are small, as
happened on OS X (512kB thread stacks by default) with the chromium
repo.
The task of the two functions is not all that hard to describe without
any recursion, however. It proceeds in three steps:
- determine the representation type and size, based on the outermost
object (delta or not)
- follow through the delta chain, if any
- determine the object type from what is found at the end of the delta
chain
The only complication stems from the error recovery. If parsing fails
at any step, we want to mark that object (within the pack) as bad and
try getting the corresponding SHA1 from elsewhere. If that also
fails, we want to repeat this process back up the delta chain until we
find a reasonable solution or conclude that there is no way to
reconstruct the object. (This is conveniently checked by t5303.)
To achieve that within the pack, we keep track of the entire delta
chain in a stack. When things go sour, we process that stack from the
top, marking entries as bad and attempting to re-resolve by sha1. To
avoid excessive malloc(), the stack starts out with a small
stack-allocated array. The choice of 64 is based on the default of
pack.depth, which is 50, in the hope that it covers "most" delta
chains without any need for malloc().
It's much harder to make the actual re-resolving by sha1 nonrecursive,
so we skip that. If you can't afford *that* recursion, your
corruption problems are more serious than your stack size problems.
Reported-by: Stefan Zager <szager@google.com>
Signed-off-by: Thomas Rast <trast@student.ethz.ch>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-03-25 18:07:39 +00:00
|
|
|
|
|
|
|
unwind:
|
|
|
|
while (poi_stack_nr) {
|
|
|
|
obj_offset = poi_stack[--poi_stack_nr];
|
|
|
|
type = retry_bad_packed_offset(p, obj_offset);
|
|
|
|
if (type > OBJ_NONE)
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
type = OBJ_BAD;
|
|
|
|
goto out;
|
2005-06-27 10:35:33 +00:00
|
|
|
}
|
|
|
|
|
2016-08-11 09:25:32 +00:00
|
|
|
int packed_object_info(struct packed_git *p, off_t obj_offset,
|
|
|
|
struct object_info *oi)
|
2013-07-12 06:31:57 +00:00
|
|
|
{
|
|
|
|
struct pack_window *w_curs = NULL;
|
|
|
|
unsigned long size;
|
|
|
|
off_t curpos = obj_offset;
|
|
|
|
enum object_type type;
|
|
|
|
|
2013-07-12 06:32:25 +00:00
|
|
|
/*
|
|
|
|
* We always get the representation type, but only convert it to
|
|
|
|
* a "real" type later if the caller is interested.
|
|
|
|
*/
|
2013-07-12 06:31:57 +00:00
|
|
|
type = unpack_object_header(p, &w_curs, &curpos, &size);
|
|
|
|
|
2013-07-12 06:37:53 +00:00
|
|
|
if (oi->sizep) {
|
2013-07-12 06:31:57 +00:00
|
|
|
if (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {
|
|
|
|
off_t tmp_pos = curpos;
|
|
|
|
off_t base_offset = get_delta_base(p, &w_curs, &tmp_pos,
|
|
|
|
type, obj_offset);
|
|
|
|
if (!base_offset) {
|
|
|
|
type = OBJ_BAD;
|
|
|
|
goto out;
|
|
|
|
}
|
2013-07-12 06:37:53 +00:00
|
|
|
*oi->sizep = get_size_from_delta(p, &w_curs, tmp_pos);
|
|
|
|
if (*oi->sizep == 0) {
|
2013-07-12 06:31:57 +00:00
|
|
|
type = OBJ_BAD;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
} else {
|
2013-07-12 06:37:53 +00:00
|
|
|
*oi->sizep = size;
|
2013-07-12 06:31:57 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2013-07-12 06:37:53 +00:00
|
|
|
if (oi->disk_sizep) {
|
2013-07-12 06:31:57 +00:00
|
|
|
struct revindex_entry *revidx = find_pack_revindex(p, obj_offset);
|
2013-07-12 06:37:53 +00:00
|
|
|
*oi->disk_sizep = revidx[1].offset - obj_offset;
|
2013-07-12 06:31:57 +00:00
|
|
|
}
|
|
|
|
|
2013-07-12 06:37:53 +00:00
|
|
|
if (oi->typep) {
|
|
|
|
*oi->typep = packed_to_object_type(p, obj_offset, type, &w_curs, curpos);
|
|
|
|
if (*oi->typep < 0) {
|
2013-07-12 06:32:25 +00:00
|
|
|
type = OBJ_BAD;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
}
|
2013-07-12 06:31:57 +00:00
|
|
|
|
2013-12-21 14:24:20 +00:00
|
|
|
if (oi->delta_base_sha1) {
|
|
|
|
if (type == OBJ_OFS_DELTA || type == OBJ_REF_DELTA) {
|
|
|
|
const unsigned char *base;
|
|
|
|
|
|
|
|
base = get_delta_base_sha1(p, &w_curs, curpos,
|
|
|
|
type, obj_offset);
|
|
|
|
if (!base) {
|
|
|
|
type = OBJ_BAD;
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
hashcpy(oi->delta_base_sha1, base);
|
|
|
|
} else
|
|
|
|
hashclr(oi->delta_base_sha1);
|
|
|
|
}
|
|
|
|
|
2013-07-12 06:31:57 +00:00
|
|
|
out:
|
|
|
|
unuse_pack(&w_curs);
|
|
|
|
return type;
|
|
|
|
}
|
|
|
|
|
2006-08-26 08:12:27 +00:00
|
|
|
static void *unpack_compressed_entry(struct packed_git *p,
|
2006-12-23 07:34:08 +00:00
|
|
|
struct pack_window **w_curs,
|
2007-03-07 01:44:30 +00:00
|
|
|
off_t curpos,
|
2006-08-26 08:12:27 +00:00
|
|
|
unsigned long size)
|
2006-08-26 08:10:43 +00:00
|
|
|
{
|
|
|
|
int st;
|
2011-06-10 18:52:15 +00:00
|
|
|
git_zstream stream;
|
2006-12-23 07:34:13 +00:00
|
|
|
unsigned char *buffer, *in;
|
2006-08-26 08:10:43 +00:00
|
|
|
|
2014-08-16 03:08:03 +00:00
|
|
|
buffer = xmallocz_gently(size);
|
|
|
|
if (!buffer)
|
|
|
|
return NULL;
|
2006-08-26 08:10:43 +00:00
|
|
|
memset(&stream, 0, sizeof(stream));
|
|
|
|
stream.next_out = buffer;
|
Fix incorrect error check while reading deflated pack data
The loop in get_size_from_delta() feeds a deflated delta data from the
pack stream _until_ we get inflated result of 20 bytes[*] or we reach the
end of stream.
Side note. This magic number 20 does not have anything to do with the
size of the hash we use, but comes from 1a3b55c (reduce delta head
inflated size, 2006-10-18).
The loop reads like this:
do {
in = use_pack();
stream.next_in = in;
st = git_inflate(&stream, Z_FINISH);
curpos += stream.next_in - in;
} while ((st == Z_OK || st == Z_BUF_ERROR) &&
stream.total_out < sizeof(delta_head));
This git_inflate() can return:
- Z_STREAM_END, if use_pack() fed it enough input and the delta itself
was smaller than 20 bytes;
- Z_OK, when some progress has been made;
- Z_BUF_ERROR, if no progress is possible, because we either ran out of
input (due to corrupt pack), or we ran out of output before we saw the
end of the stream.
The fix b3118bd (sha1_file: Fix infinite loop when pack is corrupted,
2009-10-14) attempted was against a corruption that appears to be a valid
stream that produces a result larger than the output buffer, but we are
not even trying to read the stream to the end in this loop. If avail_out
becomes zero, total_out will be the same as sizeof(delta_head) so the loop
will terminate without the "fix". There is no fix from b3118bd needed for
this loop, in other words.
The loop in unpack_compressed_entry() is quite a different story. It
feeds a deflated stream (either delta or base) and allows the stream to
produce output up to what we expect but no more.
do {
in = use_pack();
stream.next_in = in;
st = git_inflate(&stream, Z_FINISH);
curpos += stream.next_in - in;
} while (st == Z_OK || st == Z_BUF_ERROR)
This _does_ risk falling into an endless interation, as we can exhaust
avail_out if the length we expect is smaller than what the stream wants to
produce (due to pack corruption). In such a case, avail_out will become
zero and inflate() will return Z_BUF_ERROR, while avail_in may (or may
not) be zero.
But this is not a right fix:
do {
in = use_pack();
stream.next_in = in;
st = git_inflate(&stream, Z_FINISH);
+ if (st == Z_BUF_ERROR && (stream.avail_in || !stream.avail_out)
+ break; /* wants more input??? */
curpos += stream.next_in - in;
} while (st == Z_OK || st == Z_BUF_ERROR)
as Z_BUF_ERROR from inflate() may be telling us that avail_in has also run
out before reading the end of stream marker. In such a case, both avail_in
and avail_out would be zero, and the loop should iterate to allow the end
of stream marker to be seen by inflate from the input stream.
The right fix for this loop is likely to be to increment the initial
avail_out by one (we allocate one extra byte to terminate it with NUL
anyway, so there is no risk to overrun the buffer), and break out if we
see that avail_out has become zero, in order to detect that the stream
wants to produce more than what we expect. After the loop, we have a
check that exactly tests this condition:
if ((st != Z_STREAM_END) || stream.total_out != size) {
free(buffer);
return NULL;
}
So here is a patch (without my previous botched attempts) to fix this
issue. The first hunk reverts the corresponding hunk from b3118bd, and
the second hunk is the same fix proposed earlier.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-10-22 06:06:14 +00:00
|
|
|
stream.avail_out = size + 1;
|
2006-08-26 08:10:43 +00:00
|
|
|
|
2009-01-08 03:54:47 +00:00
|
|
|
git_inflate_init(&stream);
|
2006-12-23 07:34:13 +00:00
|
|
|
do {
|
2007-02-26 19:55:56 +00:00
|
|
|
in = use_pack(p, w_curs, curpos, &stream.avail_in);
|
2006-12-23 07:34:13 +00:00
|
|
|
stream.next_in = in;
|
2009-01-08 03:54:47 +00:00
|
|
|
st = git_inflate(&stream, Z_FINISH);
|
Fix incorrect error check while reading deflated pack data
The loop in get_size_from_delta() feeds a deflated delta data from the
pack stream _until_ we get inflated result of 20 bytes[*] or we reach the
end of stream.
Side note. This magic number 20 does not have anything to do with the
size of the hash we use, but comes from 1a3b55c (reduce delta head
inflated size, 2006-10-18).
The loop reads like this:
do {
in = use_pack();
stream.next_in = in;
st = git_inflate(&stream, Z_FINISH);
curpos += stream.next_in - in;
} while ((st == Z_OK || st == Z_BUF_ERROR) &&
stream.total_out < sizeof(delta_head));
This git_inflate() can return:
- Z_STREAM_END, if use_pack() fed it enough input and the delta itself
was smaller than 20 bytes;
- Z_OK, when some progress has been made;
- Z_BUF_ERROR, if no progress is possible, because we either ran out of
input (due to corrupt pack), or we ran out of output before we saw the
end of the stream.
The fix b3118bd (sha1_file: Fix infinite loop when pack is corrupted,
2009-10-14) attempted was against a corruption that appears to be a valid
stream that produces a result larger than the output buffer, but we are
not even trying to read the stream to the end in this loop. If avail_out
becomes zero, total_out will be the same as sizeof(delta_head) so the loop
will terminate without the "fix". There is no fix from b3118bd needed for
this loop, in other words.
The loop in unpack_compressed_entry() is quite a different story. It
feeds a deflated stream (either delta or base) and allows the stream to
produce output up to what we expect but no more.
do {
in = use_pack();
stream.next_in = in;
st = git_inflate(&stream, Z_FINISH);
curpos += stream.next_in - in;
} while (st == Z_OK || st == Z_BUF_ERROR)
This _does_ risk falling into an endless interation, as we can exhaust
avail_out if the length we expect is smaller than what the stream wants to
produce (due to pack corruption). In such a case, avail_out will become
zero and inflate() will return Z_BUF_ERROR, while avail_in may (or may
not) be zero.
But this is not a right fix:
do {
in = use_pack();
stream.next_in = in;
st = git_inflate(&stream, Z_FINISH);
+ if (st == Z_BUF_ERROR && (stream.avail_in || !stream.avail_out)
+ break; /* wants more input??? */
curpos += stream.next_in - in;
} while (st == Z_OK || st == Z_BUF_ERROR)
as Z_BUF_ERROR from inflate() may be telling us that avail_in has also run
out before reading the end of stream marker. In such a case, both avail_in
and avail_out would be zero, and the loop should iterate to allow the end
of stream marker to be seen by inflate from the input stream.
The right fix for this loop is likely to be to increment the initial
avail_out by one (we allocate one extra byte to terminate it with NUL
anyway, so there is no risk to overrun the buffer), and break out if we
see that avail_out has become zero, in order to detect that the stream
wants to produce more than what we expect. After the loop, we have a
check that exactly tests this condition:
if ((st != Z_STREAM_END) || stream.total_out != size) {
free(buffer);
return NULL;
}
So here is a patch (without my previous botched attempts) to fix this
issue. The first hunk reverts the corresponding hunk from b3118bd, and
the second hunk is the same fix proposed earlier.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2009-10-22 06:06:14 +00:00
|
|
|
if (!stream.avail_out)
|
|
|
|
break; /* the payload is larger than it should be */
|
2007-02-26 19:55:56 +00:00
|
|
|
curpos += stream.next_in - in;
|
2006-12-23 07:34:13 +00:00
|
|
|
} while (st == Z_OK || st == Z_BUF_ERROR);
|
2009-01-08 03:54:47 +00:00
|
|
|
git_inflate_end(&stream);
|
2006-08-26 08:10:43 +00:00
|
|
|
if ((st != Z_STREAM_END) || stream.total_out != size) {
|
|
|
|
free(buffer);
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
return buffer;
|
|
|
|
}
|
|
|
|
|
delta_base_cache: use hashmap.h
The fundamental data structure of the delta base cache is a
hash table mapping pairs of "(packfile, offset)" into
structs containing the actual object data. The hash table
implementation dates back to e5e0161 (Implement a simple
delta_base cache, 2007-03-17), and uses a fixed-size table.
The current size is a hard-coded 256 entries.
Because we need to be able to remove objects from the hash
table, entry lookup does not do any kind of probing to
handle collisions. Colliding items simply replace whatever
is in their slot. As a result, we have fewer usable slots
than even the 256 we allocate. At half full, each new item
has a 50% chance of displacing another one. Or another way
to think about it: every item has a 1/256 chance of being
ejected due to hash collision, without regard to our LRU
strategy.
So it would be interesting to see the effect of increasing
the cache size on the runtime for some common operations. As
with the previous patch, we'll measure "git log --raw" for
tree-only operations, and "git log -Sfoo --raw" for
operations that touch trees and blobs. All times are
wall-clock best-of-3, done against fully packed repos with
--depth=50, and the default core.deltaBaseCacheLimit of
96MB.
Here are timings for various values of MAX_DELTA_CACHE
against git.git (the asterisk marks the minimum time for
each operation):
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m02.227s 0m12.821s
512 0m02.143s 0m10.602s
1024 0m02.127s 0m08.642s
2048 0m02.148s 0m07.123s
4096 0m02.194s 0m06.448s*
8192 0m02.239s 0m06.504s
16384 0m02.144s* 0m06.502s
32768 0m02.202s 0m06.622s
65536 0m02.230s 0m06.677s
The log-raw case isn't changed much at all here (probably
because our trees just aren't that big in the first place,
or possibly because we have so _few_ trees in git.git that
the 256-entry cache is enough). But once we start putting
blobs in the cache, too, we see a big improvement (almost
50%). The curve levels off around 4096, which means that we
can hold about that many entries before hitting the 96MB
memory limit (or possibly that the workload is small enough
that there is simply no more work to be optimized out by
caching more).
(As a side note, I initially timed my existing git.git pack,
which was a base of --aggressive combined with some pulls on
top. So it had quite a few deeper delta chains. The
256-cache case was more like 15s, and it still dropped to
~6.5s in the same way).
Here are the timings for linux.git:
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m41.661s 5m12.410s
512 0m39.547s 5m07.920s
1024 0m37.054s 4m54.666s
2048 0m35.871s 4m41.194s*
4096 0m34.646s 4m51.648s
8192 0m33.881s 4m55.342s
16384 0m35.190s 5m00.122s
32768 0m35.060s 4m58.851s
65536 0m33.311s* 4m51.420s
As we grow we see a nice 20% speedup in the tree traversal,
and more modest 10% in the log-S. This is probably an
indication that we are bound less by the number of entries,
and more by the memory limit (more on that below). What is
interesting is that the numbers bounce around a bit;
increasing the number of entries isn't always a strict
improvement.
Partially this is due to noise in the measurement. But it
may also be an indication that our LRU ejection scheme is
not optimal. The smaller cache sizes introduce some
randomness into the ejection (due to collisions), which may
sometimes work in our favor (and sometimes not!).
So what is the optimal setting of MAX_DELTA_CACHE? The
"bouncing" in the linux.git log-S numbers notwithstanding,
it mostly seems like bigger is better. And even if we were
to try to find a "sweet spot", these are just two
repositories, that are not necessarily representative. The
shape of history, the size of trees and blobs, the memory
limit configuration, etc, all will affect the outcome.
Rather than trying to find the "right" number, another
strategy is to just switch to a hash table that can actually
store collisions: namely our hashmap.h implementation.
Here are numbers for that compared to the "best" we saw from
adjusting MAX_DELTA_CACHE:
| log-raw | log-S
| best hashmap | best hashmap
| --------- --------- | --------- ---------
git | 0m02.144s 0m02.144s | 0m06.448s 0m06.688s
linux | 0m33.311s 0m33.092s | 4m41.194s 4m57.172s
We can see the results are similar in most cases, which is
what we'd expect. We're not ejecting due to collisions at
all, so this is purely representing the LRU. So really, we'd
expect this to model most closely the larger values of the
static MAX_DELTA_CACHE limit. And that does seem to be
what's happening, including the "bounce" in the linux log-S
case.
So while the value for that case _isn't_ as good as the
optimal one measured above (which was 2048 entries), given
the bouncing I'm hesitant to suggest that 2048 is any kind
of optimum (not even for linux.git, let alone as a general
rule). The generic hashmap has the appeal that it drops the
number of tweakable numbers by one, which means we can focus
on tuning other elements, like the LRU strategy or the
core.deltaBaseCacheLimit setting.
And indeed, if we bump the cache limit to 1G (which is
probably silly for general use, but maybe something people
with big workstations would want to do), the linux.git log-S
time drops to 3m32s. That's something you really _can't_ do
easily with the static hash table, because the number of
entries needs to grow in proportion to the memory limit (so
2048 is almost certainly not going to be the right value
there).
This patch takes that direction, and drops the static hash
table entirely in favor of using the hashmap.h API.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-22 22:00:07 +00:00
|
|
|
static struct hashmap delta_base_cache;
|
2007-03-19 05:14:37 +00:00
|
|
|
static size_t delta_base_cached;
|
2007-03-19 20:31:04 +00:00
|
|
|
|
2016-08-22 21:59:42 +00:00
|
|
|
static LIST_HEAD(delta_base_cache_lru);
|
2007-03-19 20:31:04 +00:00
|
|
|
|
delta_base_cache: use hashmap.h
The fundamental data structure of the delta base cache is a
hash table mapping pairs of "(packfile, offset)" into
structs containing the actual object data. The hash table
implementation dates back to e5e0161 (Implement a simple
delta_base cache, 2007-03-17), and uses a fixed-size table.
The current size is a hard-coded 256 entries.
Because we need to be able to remove objects from the hash
table, entry lookup does not do any kind of probing to
handle collisions. Colliding items simply replace whatever
is in their slot. As a result, we have fewer usable slots
than even the 256 we allocate. At half full, each new item
has a 50% chance of displacing another one. Or another way
to think about it: every item has a 1/256 chance of being
ejected due to hash collision, without regard to our LRU
strategy.
So it would be interesting to see the effect of increasing
the cache size on the runtime for some common operations. As
with the previous patch, we'll measure "git log --raw" for
tree-only operations, and "git log -Sfoo --raw" for
operations that touch trees and blobs. All times are
wall-clock best-of-3, done against fully packed repos with
--depth=50, and the default core.deltaBaseCacheLimit of
96MB.
Here are timings for various values of MAX_DELTA_CACHE
against git.git (the asterisk marks the minimum time for
each operation):
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m02.227s 0m12.821s
512 0m02.143s 0m10.602s
1024 0m02.127s 0m08.642s
2048 0m02.148s 0m07.123s
4096 0m02.194s 0m06.448s*
8192 0m02.239s 0m06.504s
16384 0m02.144s* 0m06.502s
32768 0m02.202s 0m06.622s
65536 0m02.230s 0m06.677s
The log-raw case isn't changed much at all here (probably
because our trees just aren't that big in the first place,
or possibly because we have so _few_ trees in git.git that
the 256-entry cache is enough). But once we start putting
blobs in the cache, too, we see a big improvement (almost
50%). The curve levels off around 4096, which means that we
can hold about that many entries before hitting the 96MB
memory limit (or possibly that the workload is small enough
that there is simply no more work to be optimized out by
caching more).
(As a side note, I initially timed my existing git.git pack,
which was a base of --aggressive combined with some pulls on
top. So it had quite a few deeper delta chains. The
256-cache case was more like 15s, and it still dropped to
~6.5s in the same way).
Here are the timings for linux.git:
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m41.661s 5m12.410s
512 0m39.547s 5m07.920s
1024 0m37.054s 4m54.666s
2048 0m35.871s 4m41.194s*
4096 0m34.646s 4m51.648s
8192 0m33.881s 4m55.342s
16384 0m35.190s 5m00.122s
32768 0m35.060s 4m58.851s
65536 0m33.311s* 4m51.420s
As we grow we see a nice 20% speedup in the tree traversal,
and more modest 10% in the log-S. This is probably an
indication that we are bound less by the number of entries,
and more by the memory limit (more on that below). What is
interesting is that the numbers bounce around a bit;
increasing the number of entries isn't always a strict
improvement.
Partially this is due to noise in the measurement. But it
may also be an indication that our LRU ejection scheme is
not optimal. The smaller cache sizes introduce some
randomness into the ejection (due to collisions), which may
sometimes work in our favor (and sometimes not!).
So what is the optimal setting of MAX_DELTA_CACHE? The
"bouncing" in the linux.git log-S numbers notwithstanding,
it mostly seems like bigger is better. And even if we were
to try to find a "sweet spot", these are just two
repositories, that are not necessarily representative. The
shape of history, the size of trees and blobs, the memory
limit configuration, etc, all will affect the outcome.
Rather than trying to find the "right" number, another
strategy is to just switch to a hash table that can actually
store collisions: namely our hashmap.h implementation.
Here are numbers for that compared to the "best" we saw from
adjusting MAX_DELTA_CACHE:
| log-raw | log-S
| best hashmap | best hashmap
| --------- --------- | --------- ---------
git | 0m02.144s 0m02.144s | 0m06.448s 0m06.688s
linux | 0m33.311s 0m33.092s | 4m41.194s 4m57.172s
We can see the results are similar in most cases, which is
what we'd expect. We're not ejecting due to collisions at
all, so this is purely representing the LRU. So really, we'd
expect this to model most closely the larger values of the
static MAX_DELTA_CACHE limit. And that does seem to be
what's happening, including the "bounce" in the linux log-S
case.
So while the value for that case _isn't_ as good as the
optimal one measured above (which was 2048 entries), given
the bouncing I'm hesitant to suggest that 2048 is any kind
of optimum (not even for linux.git, let alone as a general
rule). The generic hashmap has the appeal that it drops the
number of tweakable numbers by one, which means we can focus
on tuning other elements, like the LRU strategy or the
core.deltaBaseCacheLimit setting.
And indeed, if we bump the cache limit to 1G (which is
probably silly for general use, but maybe something people
with big workstations would want to do), the linux.git log-S
time drops to 3m32s. That's something you really _can't_ do
easily with the static hash table, because the number of
entries needs to grow in proportion to the memory limit (so
2048 is almost certainly not going to be the right value
there).
This patch takes that direction, and drops the static hash
table entirely in favor of using the hashmap.h API.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-22 22:00:07 +00:00
|
|
|
struct delta_base_cache_key {
|
2007-03-17 19:44:06 +00:00
|
|
|
struct packed_git *p;
|
|
|
|
off_t base_offset;
|
delta_base_cache: use hashmap.h
The fundamental data structure of the delta base cache is a
hash table mapping pairs of "(packfile, offset)" into
structs containing the actual object data. The hash table
implementation dates back to e5e0161 (Implement a simple
delta_base cache, 2007-03-17), and uses a fixed-size table.
The current size is a hard-coded 256 entries.
Because we need to be able to remove objects from the hash
table, entry lookup does not do any kind of probing to
handle collisions. Colliding items simply replace whatever
is in their slot. As a result, we have fewer usable slots
than even the 256 we allocate. At half full, each new item
has a 50% chance of displacing another one. Or another way
to think about it: every item has a 1/256 chance of being
ejected due to hash collision, without regard to our LRU
strategy.
So it would be interesting to see the effect of increasing
the cache size on the runtime for some common operations. As
with the previous patch, we'll measure "git log --raw" for
tree-only operations, and "git log -Sfoo --raw" for
operations that touch trees and blobs. All times are
wall-clock best-of-3, done against fully packed repos with
--depth=50, and the default core.deltaBaseCacheLimit of
96MB.
Here are timings for various values of MAX_DELTA_CACHE
against git.git (the asterisk marks the minimum time for
each operation):
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m02.227s 0m12.821s
512 0m02.143s 0m10.602s
1024 0m02.127s 0m08.642s
2048 0m02.148s 0m07.123s
4096 0m02.194s 0m06.448s*
8192 0m02.239s 0m06.504s
16384 0m02.144s* 0m06.502s
32768 0m02.202s 0m06.622s
65536 0m02.230s 0m06.677s
The log-raw case isn't changed much at all here (probably
because our trees just aren't that big in the first place,
or possibly because we have so _few_ trees in git.git that
the 256-entry cache is enough). But once we start putting
blobs in the cache, too, we see a big improvement (almost
50%). The curve levels off around 4096, which means that we
can hold about that many entries before hitting the 96MB
memory limit (or possibly that the workload is small enough
that there is simply no more work to be optimized out by
caching more).
(As a side note, I initially timed my existing git.git pack,
which was a base of --aggressive combined with some pulls on
top. So it had quite a few deeper delta chains. The
256-cache case was more like 15s, and it still dropped to
~6.5s in the same way).
Here are the timings for linux.git:
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m41.661s 5m12.410s
512 0m39.547s 5m07.920s
1024 0m37.054s 4m54.666s
2048 0m35.871s 4m41.194s*
4096 0m34.646s 4m51.648s
8192 0m33.881s 4m55.342s
16384 0m35.190s 5m00.122s
32768 0m35.060s 4m58.851s
65536 0m33.311s* 4m51.420s
As we grow we see a nice 20% speedup in the tree traversal,
and more modest 10% in the log-S. This is probably an
indication that we are bound less by the number of entries,
and more by the memory limit (more on that below). What is
interesting is that the numbers bounce around a bit;
increasing the number of entries isn't always a strict
improvement.
Partially this is due to noise in the measurement. But it
may also be an indication that our LRU ejection scheme is
not optimal. The smaller cache sizes introduce some
randomness into the ejection (due to collisions), which may
sometimes work in our favor (and sometimes not!).
So what is the optimal setting of MAX_DELTA_CACHE? The
"bouncing" in the linux.git log-S numbers notwithstanding,
it mostly seems like bigger is better. And even if we were
to try to find a "sweet spot", these are just two
repositories, that are not necessarily representative. The
shape of history, the size of trees and blobs, the memory
limit configuration, etc, all will affect the outcome.
Rather than trying to find the "right" number, another
strategy is to just switch to a hash table that can actually
store collisions: namely our hashmap.h implementation.
Here are numbers for that compared to the "best" we saw from
adjusting MAX_DELTA_CACHE:
| log-raw | log-S
| best hashmap | best hashmap
| --------- --------- | --------- ---------
git | 0m02.144s 0m02.144s | 0m06.448s 0m06.688s
linux | 0m33.311s 0m33.092s | 4m41.194s 4m57.172s
We can see the results are similar in most cases, which is
what we'd expect. We're not ejecting due to collisions at
all, so this is purely representing the LRU. So really, we'd
expect this to model most closely the larger values of the
static MAX_DELTA_CACHE limit. And that does seem to be
what's happening, including the "bounce" in the linux log-S
case.
So while the value for that case _isn't_ as good as the
optimal one measured above (which was 2048 entries), given
the bouncing I'm hesitant to suggest that 2048 is any kind
of optimum (not even for linux.git, let alone as a general
rule). The generic hashmap has the appeal that it drops the
number of tweakable numbers by one, which means we can focus
on tuning other elements, like the LRU strategy or the
core.deltaBaseCacheLimit setting.
And indeed, if we bump the cache limit to 1G (which is
probably silly for general use, but maybe something people
with big workstations would want to do), the linux.git log-S
time drops to 3m32s. That's something you really _can't_ do
easily with the static hash table, because the number of
entries needs to grow in proportion to the memory limit (so
2048 is almost certainly not going to be the right value
there).
This patch takes that direction, and drops the static hash
table entirely in favor of using the hashmap.h API.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-22 22:00:07 +00:00
|
|
|
};
|
|
|
|
|
|
|
|
struct delta_base_cache_entry {
|
|
|
|
struct hashmap hash;
|
|
|
|
struct delta_base_cache_key key;
|
|
|
|
struct list_head lru;
|
|
|
|
void *data;
|
2007-03-17 19:44:06 +00:00
|
|
|
unsigned long size;
|
|
|
|
enum object_type type;
|
delta_base_cache: use hashmap.h
The fundamental data structure of the delta base cache is a
hash table mapping pairs of "(packfile, offset)" into
structs containing the actual object data. The hash table
implementation dates back to e5e0161 (Implement a simple
delta_base cache, 2007-03-17), and uses a fixed-size table.
The current size is a hard-coded 256 entries.
Because we need to be able to remove objects from the hash
table, entry lookup does not do any kind of probing to
handle collisions. Colliding items simply replace whatever
is in their slot. As a result, we have fewer usable slots
than even the 256 we allocate. At half full, each new item
has a 50% chance of displacing another one. Or another way
to think about it: every item has a 1/256 chance of being
ejected due to hash collision, without regard to our LRU
strategy.
So it would be interesting to see the effect of increasing
the cache size on the runtime for some common operations. As
with the previous patch, we'll measure "git log --raw" for
tree-only operations, and "git log -Sfoo --raw" for
operations that touch trees and blobs. All times are
wall-clock best-of-3, done against fully packed repos with
--depth=50, and the default core.deltaBaseCacheLimit of
96MB.
Here are timings for various values of MAX_DELTA_CACHE
against git.git (the asterisk marks the minimum time for
each operation):
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m02.227s 0m12.821s
512 0m02.143s 0m10.602s
1024 0m02.127s 0m08.642s
2048 0m02.148s 0m07.123s
4096 0m02.194s 0m06.448s*
8192 0m02.239s 0m06.504s
16384 0m02.144s* 0m06.502s
32768 0m02.202s 0m06.622s
65536 0m02.230s 0m06.677s
The log-raw case isn't changed much at all here (probably
because our trees just aren't that big in the first place,
or possibly because we have so _few_ trees in git.git that
the 256-entry cache is enough). But once we start putting
blobs in the cache, too, we see a big improvement (almost
50%). The curve levels off around 4096, which means that we
can hold about that many entries before hitting the 96MB
memory limit (or possibly that the workload is small enough
that there is simply no more work to be optimized out by
caching more).
(As a side note, I initially timed my existing git.git pack,
which was a base of --aggressive combined with some pulls on
top. So it had quite a few deeper delta chains. The
256-cache case was more like 15s, and it still dropped to
~6.5s in the same way).
Here are the timings for linux.git:
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m41.661s 5m12.410s
512 0m39.547s 5m07.920s
1024 0m37.054s 4m54.666s
2048 0m35.871s 4m41.194s*
4096 0m34.646s 4m51.648s
8192 0m33.881s 4m55.342s
16384 0m35.190s 5m00.122s
32768 0m35.060s 4m58.851s
65536 0m33.311s* 4m51.420s
As we grow we see a nice 20% speedup in the tree traversal,
and more modest 10% in the log-S. This is probably an
indication that we are bound less by the number of entries,
and more by the memory limit (more on that below). What is
interesting is that the numbers bounce around a bit;
increasing the number of entries isn't always a strict
improvement.
Partially this is due to noise in the measurement. But it
may also be an indication that our LRU ejection scheme is
not optimal. The smaller cache sizes introduce some
randomness into the ejection (due to collisions), which may
sometimes work in our favor (and sometimes not!).
So what is the optimal setting of MAX_DELTA_CACHE? The
"bouncing" in the linux.git log-S numbers notwithstanding,
it mostly seems like bigger is better. And even if we were
to try to find a "sweet spot", these are just two
repositories, that are not necessarily representative. The
shape of history, the size of trees and blobs, the memory
limit configuration, etc, all will affect the outcome.
Rather than trying to find the "right" number, another
strategy is to just switch to a hash table that can actually
store collisions: namely our hashmap.h implementation.
Here are numbers for that compared to the "best" we saw from
adjusting MAX_DELTA_CACHE:
| log-raw | log-S
| best hashmap | best hashmap
| --------- --------- | --------- ---------
git | 0m02.144s 0m02.144s | 0m06.448s 0m06.688s
linux | 0m33.311s 0m33.092s | 4m41.194s 4m57.172s
We can see the results are similar in most cases, which is
what we'd expect. We're not ejecting due to collisions at
all, so this is purely representing the LRU. So really, we'd
expect this to model most closely the larger values of the
static MAX_DELTA_CACHE limit. And that does seem to be
what's happening, including the "bounce" in the linux log-S
case.
So while the value for that case _isn't_ as good as the
optimal one measured above (which was 2048 entries), given
the bouncing I'm hesitant to suggest that 2048 is any kind
of optimum (not even for linux.git, let alone as a general
rule). The generic hashmap has the appeal that it drops the
number of tweakable numbers by one, which means we can focus
on tuning other elements, like the LRU strategy or the
core.deltaBaseCacheLimit setting.
And indeed, if we bump the cache limit to 1G (which is
probably silly for general use, but maybe something people
with big workstations would want to do), the linux.git log-S
time drops to 3m32s. That's something you really _can't_ do
easily with the static hash table, because the number of
entries needs to grow in proportion to the memory limit (so
2048 is almost certainly not going to be the right value
there).
This patch takes that direction, and drops the static hash
table entirely in favor of using the hashmap.h API.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-22 22:00:07 +00:00
|
|
|
};
|
2007-03-17 19:44:06 +00:00
|
|
|
|
delta_base_cache: use hashmap.h
The fundamental data structure of the delta base cache is a
hash table mapping pairs of "(packfile, offset)" into
structs containing the actual object data. The hash table
implementation dates back to e5e0161 (Implement a simple
delta_base cache, 2007-03-17), and uses a fixed-size table.
The current size is a hard-coded 256 entries.
Because we need to be able to remove objects from the hash
table, entry lookup does not do any kind of probing to
handle collisions. Colliding items simply replace whatever
is in their slot. As a result, we have fewer usable slots
than even the 256 we allocate. At half full, each new item
has a 50% chance of displacing another one. Or another way
to think about it: every item has a 1/256 chance of being
ejected due to hash collision, without regard to our LRU
strategy.
So it would be interesting to see the effect of increasing
the cache size on the runtime for some common operations. As
with the previous patch, we'll measure "git log --raw" for
tree-only operations, and "git log -Sfoo --raw" for
operations that touch trees and blobs. All times are
wall-clock best-of-3, done against fully packed repos with
--depth=50, and the default core.deltaBaseCacheLimit of
96MB.
Here are timings for various values of MAX_DELTA_CACHE
against git.git (the asterisk marks the minimum time for
each operation):
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m02.227s 0m12.821s
512 0m02.143s 0m10.602s
1024 0m02.127s 0m08.642s
2048 0m02.148s 0m07.123s
4096 0m02.194s 0m06.448s*
8192 0m02.239s 0m06.504s
16384 0m02.144s* 0m06.502s
32768 0m02.202s 0m06.622s
65536 0m02.230s 0m06.677s
The log-raw case isn't changed much at all here (probably
because our trees just aren't that big in the first place,
or possibly because we have so _few_ trees in git.git that
the 256-entry cache is enough). But once we start putting
blobs in the cache, too, we see a big improvement (almost
50%). The curve levels off around 4096, which means that we
can hold about that many entries before hitting the 96MB
memory limit (or possibly that the workload is small enough
that there is simply no more work to be optimized out by
caching more).
(As a side note, I initially timed my existing git.git pack,
which was a base of --aggressive combined with some pulls on
top. So it had quite a few deeper delta chains. The
256-cache case was more like 15s, and it still dropped to
~6.5s in the same way).
Here are the timings for linux.git:
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m41.661s 5m12.410s
512 0m39.547s 5m07.920s
1024 0m37.054s 4m54.666s
2048 0m35.871s 4m41.194s*
4096 0m34.646s 4m51.648s
8192 0m33.881s 4m55.342s
16384 0m35.190s 5m00.122s
32768 0m35.060s 4m58.851s
65536 0m33.311s* 4m51.420s
As we grow we see a nice 20% speedup in the tree traversal,
and more modest 10% in the log-S. This is probably an
indication that we are bound less by the number of entries,
and more by the memory limit (more on that below). What is
interesting is that the numbers bounce around a bit;
increasing the number of entries isn't always a strict
improvement.
Partially this is due to noise in the measurement. But it
may also be an indication that our LRU ejection scheme is
not optimal. The smaller cache sizes introduce some
randomness into the ejection (due to collisions), which may
sometimes work in our favor (and sometimes not!).
So what is the optimal setting of MAX_DELTA_CACHE? The
"bouncing" in the linux.git log-S numbers notwithstanding,
it mostly seems like bigger is better. And even if we were
to try to find a "sweet spot", these are just two
repositories, that are not necessarily representative. The
shape of history, the size of trees and blobs, the memory
limit configuration, etc, all will affect the outcome.
Rather than trying to find the "right" number, another
strategy is to just switch to a hash table that can actually
store collisions: namely our hashmap.h implementation.
Here are numbers for that compared to the "best" we saw from
adjusting MAX_DELTA_CACHE:
| log-raw | log-S
| best hashmap | best hashmap
| --------- --------- | --------- ---------
git | 0m02.144s 0m02.144s | 0m06.448s 0m06.688s
linux | 0m33.311s 0m33.092s | 4m41.194s 4m57.172s
We can see the results are similar in most cases, which is
what we'd expect. We're not ejecting due to collisions at
all, so this is purely representing the LRU. So really, we'd
expect this to model most closely the larger values of the
static MAX_DELTA_CACHE limit. And that does seem to be
what's happening, including the "bounce" in the linux log-S
case.
So while the value for that case _isn't_ as good as the
optimal one measured above (which was 2048 entries), given
the bouncing I'm hesitant to suggest that 2048 is any kind
of optimum (not even for linux.git, let alone as a general
rule). The generic hashmap has the appeal that it drops the
number of tweakable numbers by one, which means we can focus
on tuning other elements, like the LRU strategy or the
core.deltaBaseCacheLimit setting.
And indeed, if we bump the cache limit to 1G (which is
probably silly for general use, but maybe something people
with big workstations would want to do), the linux.git log-S
time drops to 3m32s. That's something you really _can't_ do
easily with the static hash table, because the number of
entries needs to grow in proportion to the memory limit (so
2048 is almost certainly not going to be the right value
there).
This patch takes that direction, and drops the static hash
table entirely in favor of using the hashmap.h API.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-22 22:00:07 +00:00
|
|
|
static unsigned int pack_entry_hash(struct packed_git *p, off_t base_offset)
|
2007-03-17 19:44:06 +00:00
|
|
|
{
|
delta_base_cache: use hashmap.h
The fundamental data structure of the delta base cache is a
hash table mapping pairs of "(packfile, offset)" into
structs containing the actual object data. The hash table
implementation dates back to e5e0161 (Implement a simple
delta_base cache, 2007-03-17), and uses a fixed-size table.
The current size is a hard-coded 256 entries.
Because we need to be able to remove objects from the hash
table, entry lookup does not do any kind of probing to
handle collisions. Colliding items simply replace whatever
is in their slot. As a result, we have fewer usable slots
than even the 256 we allocate. At half full, each new item
has a 50% chance of displacing another one. Or another way
to think about it: every item has a 1/256 chance of being
ejected due to hash collision, without regard to our LRU
strategy.
So it would be interesting to see the effect of increasing
the cache size on the runtime for some common operations. As
with the previous patch, we'll measure "git log --raw" for
tree-only operations, and "git log -Sfoo --raw" for
operations that touch trees and blobs. All times are
wall-clock best-of-3, done against fully packed repos with
--depth=50, and the default core.deltaBaseCacheLimit of
96MB.
Here are timings for various values of MAX_DELTA_CACHE
against git.git (the asterisk marks the minimum time for
each operation):
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m02.227s 0m12.821s
512 0m02.143s 0m10.602s
1024 0m02.127s 0m08.642s
2048 0m02.148s 0m07.123s
4096 0m02.194s 0m06.448s*
8192 0m02.239s 0m06.504s
16384 0m02.144s* 0m06.502s
32768 0m02.202s 0m06.622s
65536 0m02.230s 0m06.677s
The log-raw case isn't changed much at all here (probably
because our trees just aren't that big in the first place,
or possibly because we have so _few_ trees in git.git that
the 256-entry cache is enough). But once we start putting
blobs in the cache, too, we see a big improvement (almost
50%). The curve levels off around 4096, which means that we
can hold about that many entries before hitting the 96MB
memory limit (or possibly that the workload is small enough
that there is simply no more work to be optimized out by
caching more).
(As a side note, I initially timed my existing git.git pack,
which was a base of --aggressive combined with some pulls on
top. So it had quite a few deeper delta chains. The
256-cache case was more like 15s, and it still dropped to
~6.5s in the same way).
Here are the timings for linux.git:
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m41.661s 5m12.410s
512 0m39.547s 5m07.920s
1024 0m37.054s 4m54.666s
2048 0m35.871s 4m41.194s*
4096 0m34.646s 4m51.648s
8192 0m33.881s 4m55.342s
16384 0m35.190s 5m00.122s
32768 0m35.060s 4m58.851s
65536 0m33.311s* 4m51.420s
As we grow we see a nice 20% speedup in the tree traversal,
and more modest 10% in the log-S. This is probably an
indication that we are bound less by the number of entries,
and more by the memory limit (more on that below). What is
interesting is that the numbers bounce around a bit;
increasing the number of entries isn't always a strict
improvement.
Partially this is due to noise in the measurement. But it
may also be an indication that our LRU ejection scheme is
not optimal. The smaller cache sizes introduce some
randomness into the ejection (due to collisions), which may
sometimes work in our favor (and sometimes not!).
So what is the optimal setting of MAX_DELTA_CACHE? The
"bouncing" in the linux.git log-S numbers notwithstanding,
it mostly seems like bigger is better. And even if we were
to try to find a "sweet spot", these are just two
repositories, that are not necessarily representative. The
shape of history, the size of trees and blobs, the memory
limit configuration, etc, all will affect the outcome.
Rather than trying to find the "right" number, another
strategy is to just switch to a hash table that can actually
store collisions: namely our hashmap.h implementation.
Here are numbers for that compared to the "best" we saw from
adjusting MAX_DELTA_CACHE:
| log-raw | log-S
| best hashmap | best hashmap
| --------- --------- | --------- ---------
git | 0m02.144s 0m02.144s | 0m06.448s 0m06.688s
linux | 0m33.311s 0m33.092s | 4m41.194s 4m57.172s
We can see the results are similar in most cases, which is
what we'd expect. We're not ejecting due to collisions at
all, so this is purely representing the LRU. So really, we'd
expect this to model most closely the larger values of the
static MAX_DELTA_CACHE limit. And that does seem to be
what's happening, including the "bounce" in the linux log-S
case.
So while the value for that case _isn't_ as good as the
optimal one measured above (which was 2048 entries), given
the bouncing I'm hesitant to suggest that 2048 is any kind
of optimum (not even for linux.git, let alone as a general
rule). The generic hashmap has the appeal that it drops the
number of tweakable numbers by one, which means we can focus
on tuning other elements, like the LRU strategy or the
core.deltaBaseCacheLimit setting.
And indeed, if we bump the cache limit to 1G (which is
probably silly for general use, but maybe something people
with big workstations would want to do), the linux.git log-S
time drops to 3m32s. That's something you really _can't_ do
easily with the static hash table, because the number of
entries needs to grow in proportion to the memory limit (so
2048 is almost certainly not going to be the right value
there).
This patch takes that direction, and drops the static hash
table entirely in favor of using the hashmap.h API.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-22 22:00:07 +00:00
|
|
|
unsigned int hash;
|
2007-03-17 19:44:06 +00:00
|
|
|
|
delta_base_cache: use hashmap.h
The fundamental data structure of the delta base cache is a
hash table mapping pairs of "(packfile, offset)" into
structs containing the actual object data. The hash table
implementation dates back to e5e0161 (Implement a simple
delta_base cache, 2007-03-17), and uses a fixed-size table.
The current size is a hard-coded 256 entries.
Because we need to be able to remove objects from the hash
table, entry lookup does not do any kind of probing to
handle collisions. Colliding items simply replace whatever
is in their slot. As a result, we have fewer usable slots
than even the 256 we allocate. At half full, each new item
has a 50% chance of displacing another one. Or another way
to think about it: every item has a 1/256 chance of being
ejected due to hash collision, without regard to our LRU
strategy.
So it would be interesting to see the effect of increasing
the cache size on the runtime for some common operations. As
with the previous patch, we'll measure "git log --raw" for
tree-only operations, and "git log -Sfoo --raw" for
operations that touch trees and blobs. All times are
wall-clock best-of-3, done against fully packed repos with
--depth=50, and the default core.deltaBaseCacheLimit of
96MB.
Here are timings for various values of MAX_DELTA_CACHE
against git.git (the asterisk marks the minimum time for
each operation):
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m02.227s 0m12.821s
512 0m02.143s 0m10.602s
1024 0m02.127s 0m08.642s
2048 0m02.148s 0m07.123s
4096 0m02.194s 0m06.448s*
8192 0m02.239s 0m06.504s
16384 0m02.144s* 0m06.502s
32768 0m02.202s 0m06.622s
65536 0m02.230s 0m06.677s
The log-raw case isn't changed much at all here (probably
because our trees just aren't that big in the first place,
or possibly because we have so _few_ trees in git.git that
the 256-entry cache is enough). But once we start putting
blobs in the cache, too, we see a big improvement (almost
50%). The curve levels off around 4096, which means that we
can hold about that many entries before hitting the 96MB
memory limit (or possibly that the workload is small enough
that there is simply no more work to be optimized out by
caching more).
(As a side note, I initially timed my existing git.git pack,
which was a base of --aggressive combined with some pulls on
top. So it had quite a few deeper delta chains. The
256-cache case was more like 15s, and it still dropped to
~6.5s in the same way).
Here are the timings for linux.git:
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m41.661s 5m12.410s
512 0m39.547s 5m07.920s
1024 0m37.054s 4m54.666s
2048 0m35.871s 4m41.194s*
4096 0m34.646s 4m51.648s
8192 0m33.881s 4m55.342s
16384 0m35.190s 5m00.122s
32768 0m35.060s 4m58.851s
65536 0m33.311s* 4m51.420s
As we grow we see a nice 20% speedup in the tree traversal,
and more modest 10% in the log-S. This is probably an
indication that we are bound less by the number of entries,
and more by the memory limit (more on that below). What is
interesting is that the numbers bounce around a bit;
increasing the number of entries isn't always a strict
improvement.
Partially this is due to noise in the measurement. But it
may also be an indication that our LRU ejection scheme is
not optimal. The smaller cache sizes introduce some
randomness into the ejection (due to collisions), which may
sometimes work in our favor (and sometimes not!).
So what is the optimal setting of MAX_DELTA_CACHE? The
"bouncing" in the linux.git log-S numbers notwithstanding,
it mostly seems like bigger is better. And even if we were
to try to find a "sweet spot", these are just two
repositories, that are not necessarily representative. The
shape of history, the size of trees and blobs, the memory
limit configuration, etc, all will affect the outcome.
Rather than trying to find the "right" number, another
strategy is to just switch to a hash table that can actually
store collisions: namely our hashmap.h implementation.
Here are numbers for that compared to the "best" we saw from
adjusting MAX_DELTA_CACHE:
| log-raw | log-S
| best hashmap | best hashmap
| --------- --------- | --------- ---------
git | 0m02.144s 0m02.144s | 0m06.448s 0m06.688s
linux | 0m33.311s 0m33.092s | 4m41.194s 4m57.172s
We can see the results are similar in most cases, which is
what we'd expect. We're not ejecting due to collisions at
all, so this is purely representing the LRU. So really, we'd
expect this to model most closely the larger values of the
static MAX_DELTA_CACHE limit. And that does seem to be
what's happening, including the "bounce" in the linux log-S
case.
So while the value for that case _isn't_ as good as the
optimal one measured above (which was 2048 entries), given
the bouncing I'm hesitant to suggest that 2048 is any kind
of optimum (not even for linux.git, let alone as a general
rule). The generic hashmap has the appeal that it drops the
number of tweakable numbers by one, which means we can focus
on tuning other elements, like the LRU strategy or the
core.deltaBaseCacheLimit setting.
And indeed, if we bump the cache limit to 1G (which is
probably silly for general use, but maybe something people
with big workstations would want to do), the linux.git log-S
time drops to 3m32s. That's something you really _can't_ do
easily with the static hash table, because the number of
entries needs to grow in proportion to the memory limit (so
2048 is almost certainly not going to be the right value
there).
This patch takes that direction, and drops the static hash
table entirely in favor of using the hashmap.h API.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-22 22:00:07 +00:00
|
|
|
hash = (unsigned int)(intptr_t)p + (unsigned int)base_offset;
|
2007-03-17 19:44:06 +00:00
|
|
|
hash += (hash >> 8) + (hash >> 16);
|
delta_base_cache: use hashmap.h
The fundamental data structure of the delta base cache is a
hash table mapping pairs of "(packfile, offset)" into
structs containing the actual object data. The hash table
implementation dates back to e5e0161 (Implement a simple
delta_base cache, 2007-03-17), and uses a fixed-size table.
The current size is a hard-coded 256 entries.
Because we need to be able to remove objects from the hash
table, entry lookup does not do any kind of probing to
handle collisions. Colliding items simply replace whatever
is in their slot. As a result, we have fewer usable slots
than even the 256 we allocate. At half full, each new item
has a 50% chance of displacing another one. Or another way
to think about it: every item has a 1/256 chance of being
ejected due to hash collision, without regard to our LRU
strategy.
So it would be interesting to see the effect of increasing
the cache size on the runtime for some common operations. As
with the previous patch, we'll measure "git log --raw" for
tree-only operations, and "git log -Sfoo --raw" for
operations that touch trees and blobs. All times are
wall-clock best-of-3, done against fully packed repos with
--depth=50, and the default core.deltaBaseCacheLimit of
96MB.
Here are timings for various values of MAX_DELTA_CACHE
against git.git (the asterisk marks the minimum time for
each operation):
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m02.227s 0m12.821s
512 0m02.143s 0m10.602s
1024 0m02.127s 0m08.642s
2048 0m02.148s 0m07.123s
4096 0m02.194s 0m06.448s*
8192 0m02.239s 0m06.504s
16384 0m02.144s* 0m06.502s
32768 0m02.202s 0m06.622s
65536 0m02.230s 0m06.677s
The log-raw case isn't changed much at all here (probably
because our trees just aren't that big in the first place,
or possibly because we have so _few_ trees in git.git that
the 256-entry cache is enough). But once we start putting
blobs in the cache, too, we see a big improvement (almost
50%). The curve levels off around 4096, which means that we
can hold about that many entries before hitting the 96MB
memory limit (or possibly that the workload is small enough
that there is simply no more work to be optimized out by
caching more).
(As a side note, I initially timed my existing git.git pack,
which was a base of --aggressive combined with some pulls on
top. So it had quite a few deeper delta chains. The
256-cache case was more like 15s, and it still dropped to
~6.5s in the same way).
Here are the timings for linux.git:
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m41.661s 5m12.410s
512 0m39.547s 5m07.920s
1024 0m37.054s 4m54.666s
2048 0m35.871s 4m41.194s*
4096 0m34.646s 4m51.648s
8192 0m33.881s 4m55.342s
16384 0m35.190s 5m00.122s
32768 0m35.060s 4m58.851s
65536 0m33.311s* 4m51.420s
As we grow we see a nice 20% speedup in the tree traversal,
and more modest 10% in the log-S. This is probably an
indication that we are bound less by the number of entries,
and more by the memory limit (more on that below). What is
interesting is that the numbers bounce around a bit;
increasing the number of entries isn't always a strict
improvement.
Partially this is due to noise in the measurement. But it
may also be an indication that our LRU ejection scheme is
not optimal. The smaller cache sizes introduce some
randomness into the ejection (due to collisions), which may
sometimes work in our favor (and sometimes not!).
So what is the optimal setting of MAX_DELTA_CACHE? The
"bouncing" in the linux.git log-S numbers notwithstanding,
it mostly seems like bigger is better. And even if we were
to try to find a "sweet spot", these are just two
repositories, that are not necessarily representative. The
shape of history, the size of trees and blobs, the memory
limit configuration, etc, all will affect the outcome.
Rather than trying to find the "right" number, another
strategy is to just switch to a hash table that can actually
store collisions: namely our hashmap.h implementation.
Here are numbers for that compared to the "best" we saw from
adjusting MAX_DELTA_CACHE:
| log-raw | log-S
| best hashmap | best hashmap
| --------- --------- | --------- ---------
git | 0m02.144s 0m02.144s | 0m06.448s 0m06.688s
linux | 0m33.311s 0m33.092s | 4m41.194s 4m57.172s
We can see the results are similar in most cases, which is
what we'd expect. We're not ejecting due to collisions at
all, so this is purely representing the LRU. So really, we'd
expect this to model most closely the larger values of the
static MAX_DELTA_CACHE limit. And that does seem to be
what's happening, including the "bounce" in the linux log-S
case.
So while the value for that case _isn't_ as good as the
optimal one measured above (which was 2048 entries), given
the bouncing I'm hesitant to suggest that 2048 is any kind
of optimum (not even for linux.git, let alone as a general
rule). The generic hashmap has the appeal that it drops the
number of tweakable numbers by one, which means we can focus
on tuning other elements, like the LRU strategy or the
core.deltaBaseCacheLimit setting.
And indeed, if we bump the cache limit to 1G (which is
probably silly for general use, but maybe something people
with big workstations would want to do), the linux.git log-S
time drops to 3m32s. That's something you really _can't_ do
easily with the static hash table, because the number of
entries needs to grow in proportion to the memory limit (so
2048 is almost certainly not going to be the right value
there).
This patch takes that direction, and drops the static hash
table entirely in favor of using the hashmap.h API.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-22 22:00:07 +00:00
|
|
|
return hash;
|
2007-03-17 19:44:06 +00:00
|
|
|
}
|
|
|
|
|
2013-03-27 20:03:41 +00:00
|
|
|
static struct delta_base_cache_entry *
|
|
|
|
get_delta_base_cache_entry(struct packed_git *p, off_t base_offset)
|
2011-05-13 20:20:43 +00:00
|
|
|
{
|
delta_base_cache: use hashmap.h
The fundamental data structure of the delta base cache is a
hash table mapping pairs of "(packfile, offset)" into
structs containing the actual object data. The hash table
implementation dates back to e5e0161 (Implement a simple
delta_base cache, 2007-03-17), and uses a fixed-size table.
The current size is a hard-coded 256 entries.
Because we need to be able to remove objects from the hash
table, entry lookup does not do any kind of probing to
handle collisions. Colliding items simply replace whatever
is in their slot. As a result, we have fewer usable slots
than even the 256 we allocate. At half full, each new item
has a 50% chance of displacing another one. Or another way
to think about it: every item has a 1/256 chance of being
ejected due to hash collision, without regard to our LRU
strategy.
So it would be interesting to see the effect of increasing
the cache size on the runtime for some common operations. As
with the previous patch, we'll measure "git log --raw" for
tree-only operations, and "git log -Sfoo --raw" for
operations that touch trees and blobs. All times are
wall-clock best-of-3, done against fully packed repos with
--depth=50, and the default core.deltaBaseCacheLimit of
96MB.
Here are timings for various values of MAX_DELTA_CACHE
against git.git (the asterisk marks the minimum time for
each operation):
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m02.227s 0m12.821s
512 0m02.143s 0m10.602s
1024 0m02.127s 0m08.642s
2048 0m02.148s 0m07.123s
4096 0m02.194s 0m06.448s*
8192 0m02.239s 0m06.504s
16384 0m02.144s* 0m06.502s
32768 0m02.202s 0m06.622s
65536 0m02.230s 0m06.677s
The log-raw case isn't changed much at all here (probably
because our trees just aren't that big in the first place,
or possibly because we have so _few_ trees in git.git that
the 256-entry cache is enough). But once we start putting
blobs in the cache, too, we see a big improvement (almost
50%). The curve levels off around 4096, which means that we
can hold about that many entries before hitting the 96MB
memory limit (or possibly that the workload is small enough
that there is simply no more work to be optimized out by
caching more).
(As a side note, I initially timed my existing git.git pack,
which was a base of --aggressive combined with some pulls on
top. So it had quite a few deeper delta chains. The
256-cache case was more like 15s, and it still dropped to
~6.5s in the same way).
Here are the timings for linux.git:
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m41.661s 5m12.410s
512 0m39.547s 5m07.920s
1024 0m37.054s 4m54.666s
2048 0m35.871s 4m41.194s*
4096 0m34.646s 4m51.648s
8192 0m33.881s 4m55.342s
16384 0m35.190s 5m00.122s
32768 0m35.060s 4m58.851s
65536 0m33.311s* 4m51.420s
As we grow we see a nice 20% speedup in the tree traversal,
and more modest 10% in the log-S. This is probably an
indication that we are bound less by the number of entries,
and more by the memory limit (more on that below). What is
interesting is that the numbers bounce around a bit;
increasing the number of entries isn't always a strict
improvement.
Partially this is due to noise in the measurement. But it
may also be an indication that our LRU ejection scheme is
not optimal. The smaller cache sizes introduce some
randomness into the ejection (due to collisions), which may
sometimes work in our favor (and sometimes not!).
So what is the optimal setting of MAX_DELTA_CACHE? The
"bouncing" in the linux.git log-S numbers notwithstanding,
it mostly seems like bigger is better. And even if we were
to try to find a "sweet spot", these are just two
repositories, that are not necessarily representative. The
shape of history, the size of trees and blobs, the memory
limit configuration, etc, all will affect the outcome.
Rather than trying to find the "right" number, another
strategy is to just switch to a hash table that can actually
store collisions: namely our hashmap.h implementation.
Here are numbers for that compared to the "best" we saw from
adjusting MAX_DELTA_CACHE:
| log-raw | log-S
| best hashmap | best hashmap
| --------- --------- | --------- ---------
git | 0m02.144s 0m02.144s | 0m06.448s 0m06.688s
linux | 0m33.311s 0m33.092s | 4m41.194s 4m57.172s
We can see the results are similar in most cases, which is
what we'd expect. We're not ejecting due to collisions at
all, so this is purely representing the LRU. So really, we'd
expect this to model most closely the larger values of the
static MAX_DELTA_CACHE limit. And that does seem to be
what's happening, including the "bounce" in the linux log-S
case.
So while the value for that case _isn't_ as good as the
optimal one measured above (which was 2048 entries), given
the bouncing I'm hesitant to suggest that 2048 is any kind
of optimum (not even for linux.git, let alone as a general
rule). The generic hashmap has the appeal that it drops the
number of tweakable numbers by one, which means we can focus
on tuning other elements, like the LRU strategy or the
core.deltaBaseCacheLimit setting.
And indeed, if we bump the cache limit to 1G (which is
probably silly for general use, but maybe something people
with big workstations would want to do), the linux.git log-S
time drops to 3m32s. That's something you really _can't_ do
easily with the static hash table, because the number of
entries needs to grow in proportion to the memory limit (so
2048 is almost certainly not going to be the right value
there).
This patch takes that direction, and drops the static hash
table entirely in favor of using the hashmap.h API.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-22 22:00:07 +00:00
|
|
|
struct hashmap_entry entry;
|
|
|
|
struct delta_base_cache_key key;
|
|
|
|
|
|
|
|
if (!delta_base_cache.cmpfn)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
hashmap_entry_init(&entry, pack_entry_hash(p, base_offset));
|
|
|
|
key.p = p;
|
|
|
|
key.base_offset = base_offset;
|
|
|
|
return hashmap_get(&delta_base_cache, &entry, &key);
|
|
|
|
}
|
|
|
|
|
|
|
|
static int delta_base_cache_key_eq(const struct delta_base_cache_key *a,
|
|
|
|
const struct delta_base_cache_key *b)
|
|
|
|
{
|
|
|
|
return a->p == b->p && a->base_offset == b->base_offset;
|
2013-03-27 20:03:41 +00:00
|
|
|
}
|
|
|
|
|
delta_base_cache: use hashmap.h
The fundamental data structure of the delta base cache is a
hash table mapping pairs of "(packfile, offset)" into
structs containing the actual object data. The hash table
implementation dates back to e5e0161 (Implement a simple
delta_base cache, 2007-03-17), and uses a fixed-size table.
The current size is a hard-coded 256 entries.
Because we need to be able to remove objects from the hash
table, entry lookup does not do any kind of probing to
handle collisions. Colliding items simply replace whatever
is in their slot. As a result, we have fewer usable slots
than even the 256 we allocate. At half full, each new item
has a 50% chance of displacing another one. Or another way
to think about it: every item has a 1/256 chance of being
ejected due to hash collision, without regard to our LRU
strategy.
So it would be interesting to see the effect of increasing
the cache size on the runtime for some common operations. As
with the previous patch, we'll measure "git log --raw" for
tree-only operations, and "git log -Sfoo --raw" for
operations that touch trees and blobs. All times are
wall-clock best-of-3, done against fully packed repos with
--depth=50, and the default core.deltaBaseCacheLimit of
96MB.
Here are timings for various values of MAX_DELTA_CACHE
against git.git (the asterisk marks the minimum time for
each operation):
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m02.227s 0m12.821s
512 0m02.143s 0m10.602s
1024 0m02.127s 0m08.642s
2048 0m02.148s 0m07.123s
4096 0m02.194s 0m06.448s*
8192 0m02.239s 0m06.504s
16384 0m02.144s* 0m06.502s
32768 0m02.202s 0m06.622s
65536 0m02.230s 0m06.677s
The log-raw case isn't changed much at all here (probably
because our trees just aren't that big in the first place,
or possibly because we have so _few_ trees in git.git that
the 256-entry cache is enough). But once we start putting
blobs in the cache, too, we see a big improvement (almost
50%). The curve levels off around 4096, which means that we
can hold about that many entries before hitting the 96MB
memory limit (or possibly that the workload is small enough
that there is simply no more work to be optimized out by
caching more).
(As a side note, I initially timed my existing git.git pack,
which was a base of --aggressive combined with some pulls on
top. So it had quite a few deeper delta chains. The
256-cache case was more like 15s, and it still dropped to
~6.5s in the same way).
Here are the timings for linux.git:
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m41.661s 5m12.410s
512 0m39.547s 5m07.920s
1024 0m37.054s 4m54.666s
2048 0m35.871s 4m41.194s*
4096 0m34.646s 4m51.648s
8192 0m33.881s 4m55.342s
16384 0m35.190s 5m00.122s
32768 0m35.060s 4m58.851s
65536 0m33.311s* 4m51.420s
As we grow we see a nice 20% speedup in the tree traversal,
and more modest 10% in the log-S. This is probably an
indication that we are bound less by the number of entries,
and more by the memory limit (more on that below). What is
interesting is that the numbers bounce around a bit;
increasing the number of entries isn't always a strict
improvement.
Partially this is due to noise in the measurement. But it
may also be an indication that our LRU ejection scheme is
not optimal. The smaller cache sizes introduce some
randomness into the ejection (due to collisions), which may
sometimes work in our favor (and sometimes not!).
So what is the optimal setting of MAX_DELTA_CACHE? The
"bouncing" in the linux.git log-S numbers notwithstanding,
it mostly seems like bigger is better. And even if we were
to try to find a "sweet spot", these are just two
repositories, that are not necessarily representative. The
shape of history, the size of trees and blobs, the memory
limit configuration, etc, all will affect the outcome.
Rather than trying to find the "right" number, another
strategy is to just switch to a hash table that can actually
store collisions: namely our hashmap.h implementation.
Here are numbers for that compared to the "best" we saw from
adjusting MAX_DELTA_CACHE:
| log-raw | log-S
| best hashmap | best hashmap
| --------- --------- | --------- ---------
git | 0m02.144s 0m02.144s | 0m06.448s 0m06.688s
linux | 0m33.311s 0m33.092s | 4m41.194s 4m57.172s
We can see the results are similar in most cases, which is
what we'd expect. We're not ejecting due to collisions at
all, so this is purely representing the LRU. So really, we'd
expect this to model most closely the larger values of the
static MAX_DELTA_CACHE limit. And that does seem to be
what's happening, including the "bounce" in the linux log-S
case.
So while the value for that case _isn't_ as good as the
optimal one measured above (which was 2048 entries), given
the bouncing I'm hesitant to suggest that 2048 is any kind
of optimum (not even for linux.git, let alone as a general
rule). The generic hashmap has the appeal that it drops the
number of tweakable numbers by one, which means we can focus
on tuning other elements, like the LRU strategy or the
core.deltaBaseCacheLimit setting.
And indeed, if we bump the cache limit to 1G (which is
probably silly for general use, but maybe something people
with big workstations would want to do), the linux.git log-S
time drops to 3m32s. That's something you really _can't_ do
easily with the static hash table, because the number of
entries needs to grow in proportion to the memory limit (so
2048 is almost certainly not going to be the right value
there).
This patch takes that direction, and drops the static hash
table entirely in favor of using the hashmap.h API.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-22 22:00:07 +00:00
|
|
|
static int delta_base_cache_hash_cmp(const void *va, const void *vb,
|
|
|
|
const void *vkey)
|
2013-03-27 20:03:41 +00:00
|
|
|
{
|
delta_base_cache: use hashmap.h
The fundamental data structure of the delta base cache is a
hash table mapping pairs of "(packfile, offset)" into
structs containing the actual object data. The hash table
implementation dates back to e5e0161 (Implement a simple
delta_base cache, 2007-03-17), and uses a fixed-size table.
The current size is a hard-coded 256 entries.
Because we need to be able to remove objects from the hash
table, entry lookup does not do any kind of probing to
handle collisions. Colliding items simply replace whatever
is in their slot. As a result, we have fewer usable slots
than even the 256 we allocate. At half full, each new item
has a 50% chance of displacing another one. Or another way
to think about it: every item has a 1/256 chance of being
ejected due to hash collision, without regard to our LRU
strategy.
So it would be interesting to see the effect of increasing
the cache size on the runtime for some common operations. As
with the previous patch, we'll measure "git log --raw" for
tree-only operations, and "git log -Sfoo --raw" for
operations that touch trees and blobs. All times are
wall-clock best-of-3, done against fully packed repos with
--depth=50, and the default core.deltaBaseCacheLimit of
96MB.
Here are timings for various values of MAX_DELTA_CACHE
against git.git (the asterisk marks the minimum time for
each operation):
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m02.227s 0m12.821s
512 0m02.143s 0m10.602s
1024 0m02.127s 0m08.642s
2048 0m02.148s 0m07.123s
4096 0m02.194s 0m06.448s*
8192 0m02.239s 0m06.504s
16384 0m02.144s* 0m06.502s
32768 0m02.202s 0m06.622s
65536 0m02.230s 0m06.677s
The log-raw case isn't changed much at all here (probably
because our trees just aren't that big in the first place,
or possibly because we have so _few_ trees in git.git that
the 256-entry cache is enough). But once we start putting
blobs in the cache, too, we see a big improvement (almost
50%). The curve levels off around 4096, which means that we
can hold about that many entries before hitting the 96MB
memory limit (or possibly that the workload is small enough
that there is simply no more work to be optimized out by
caching more).
(As a side note, I initially timed my existing git.git pack,
which was a base of --aggressive combined with some pulls on
top. So it had quite a few deeper delta chains. The
256-cache case was more like 15s, and it still dropped to
~6.5s in the same way).
Here are the timings for linux.git:
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m41.661s 5m12.410s
512 0m39.547s 5m07.920s
1024 0m37.054s 4m54.666s
2048 0m35.871s 4m41.194s*
4096 0m34.646s 4m51.648s
8192 0m33.881s 4m55.342s
16384 0m35.190s 5m00.122s
32768 0m35.060s 4m58.851s
65536 0m33.311s* 4m51.420s
As we grow we see a nice 20% speedup in the tree traversal,
and more modest 10% in the log-S. This is probably an
indication that we are bound less by the number of entries,
and more by the memory limit (more on that below). What is
interesting is that the numbers bounce around a bit;
increasing the number of entries isn't always a strict
improvement.
Partially this is due to noise in the measurement. But it
may also be an indication that our LRU ejection scheme is
not optimal. The smaller cache sizes introduce some
randomness into the ejection (due to collisions), which may
sometimes work in our favor (and sometimes not!).
So what is the optimal setting of MAX_DELTA_CACHE? The
"bouncing" in the linux.git log-S numbers notwithstanding,
it mostly seems like bigger is better. And even if we were
to try to find a "sweet spot", these are just two
repositories, that are not necessarily representative. The
shape of history, the size of trees and blobs, the memory
limit configuration, etc, all will affect the outcome.
Rather than trying to find the "right" number, another
strategy is to just switch to a hash table that can actually
store collisions: namely our hashmap.h implementation.
Here are numbers for that compared to the "best" we saw from
adjusting MAX_DELTA_CACHE:
| log-raw | log-S
| best hashmap | best hashmap
| --------- --------- | --------- ---------
git | 0m02.144s 0m02.144s | 0m06.448s 0m06.688s
linux | 0m33.311s 0m33.092s | 4m41.194s 4m57.172s
We can see the results are similar in most cases, which is
what we'd expect. We're not ejecting due to collisions at
all, so this is purely representing the LRU. So really, we'd
expect this to model most closely the larger values of the
static MAX_DELTA_CACHE limit. And that does seem to be
what's happening, including the "bounce" in the linux log-S
case.
So while the value for that case _isn't_ as good as the
optimal one measured above (which was 2048 entries), given
the bouncing I'm hesitant to suggest that 2048 is any kind
of optimum (not even for linux.git, let alone as a general
rule). The generic hashmap has the appeal that it drops the
number of tweakable numbers by one, which means we can focus
on tuning other elements, like the LRU strategy or the
core.deltaBaseCacheLimit setting.
And indeed, if we bump the cache limit to 1G (which is
probably silly for general use, but maybe something people
with big workstations would want to do), the linux.git log-S
time drops to 3m32s. That's something you really _can't_ do
easily with the static hash table, because the number of
entries needs to grow in proportion to the memory limit (so
2048 is almost certainly not going to be the right value
there).
This patch takes that direction, and drops the static hash
table entirely in favor of using the hashmap.h API.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-22 22:00:07 +00:00
|
|
|
const struct delta_base_cache_entry *a = va, *b = vb;
|
|
|
|
const struct delta_base_cache_key *key = vkey;
|
|
|
|
if (key)
|
|
|
|
return !delta_base_cache_key_eq(&a->key, key);
|
|
|
|
else
|
|
|
|
return !delta_base_cache_key_eq(&a->key, &b->key);
|
2011-05-13 20:20:43 +00:00
|
|
|
}
|
|
|
|
|
2013-03-27 20:03:41 +00:00
|
|
|
static int in_delta_base_cache(struct packed_git *p, off_t base_offset)
|
|
|
|
{
|
delta_base_cache: use hashmap.h
The fundamental data structure of the delta base cache is a
hash table mapping pairs of "(packfile, offset)" into
structs containing the actual object data. The hash table
implementation dates back to e5e0161 (Implement a simple
delta_base cache, 2007-03-17), and uses a fixed-size table.
The current size is a hard-coded 256 entries.
Because we need to be able to remove objects from the hash
table, entry lookup does not do any kind of probing to
handle collisions. Colliding items simply replace whatever
is in their slot. As a result, we have fewer usable slots
than even the 256 we allocate. At half full, each new item
has a 50% chance of displacing another one. Or another way
to think about it: every item has a 1/256 chance of being
ejected due to hash collision, without regard to our LRU
strategy.
So it would be interesting to see the effect of increasing
the cache size on the runtime for some common operations. As
with the previous patch, we'll measure "git log --raw" for
tree-only operations, and "git log -Sfoo --raw" for
operations that touch trees and blobs. All times are
wall-clock best-of-3, done against fully packed repos with
--depth=50, and the default core.deltaBaseCacheLimit of
96MB.
Here are timings for various values of MAX_DELTA_CACHE
against git.git (the asterisk marks the minimum time for
each operation):
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m02.227s 0m12.821s
512 0m02.143s 0m10.602s
1024 0m02.127s 0m08.642s
2048 0m02.148s 0m07.123s
4096 0m02.194s 0m06.448s*
8192 0m02.239s 0m06.504s
16384 0m02.144s* 0m06.502s
32768 0m02.202s 0m06.622s
65536 0m02.230s 0m06.677s
The log-raw case isn't changed much at all here (probably
because our trees just aren't that big in the first place,
or possibly because we have so _few_ trees in git.git that
the 256-entry cache is enough). But once we start putting
blobs in the cache, too, we see a big improvement (almost
50%). The curve levels off around 4096, which means that we
can hold about that many entries before hitting the 96MB
memory limit (or possibly that the workload is small enough
that there is simply no more work to be optimized out by
caching more).
(As a side note, I initially timed my existing git.git pack,
which was a base of --aggressive combined with some pulls on
top. So it had quite a few deeper delta chains. The
256-cache case was more like 15s, and it still dropped to
~6.5s in the same way).
Here are the timings for linux.git:
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m41.661s 5m12.410s
512 0m39.547s 5m07.920s
1024 0m37.054s 4m54.666s
2048 0m35.871s 4m41.194s*
4096 0m34.646s 4m51.648s
8192 0m33.881s 4m55.342s
16384 0m35.190s 5m00.122s
32768 0m35.060s 4m58.851s
65536 0m33.311s* 4m51.420s
As we grow we see a nice 20% speedup in the tree traversal,
and more modest 10% in the log-S. This is probably an
indication that we are bound less by the number of entries,
and more by the memory limit (more on that below). What is
interesting is that the numbers bounce around a bit;
increasing the number of entries isn't always a strict
improvement.
Partially this is due to noise in the measurement. But it
may also be an indication that our LRU ejection scheme is
not optimal. The smaller cache sizes introduce some
randomness into the ejection (due to collisions), which may
sometimes work in our favor (and sometimes not!).
So what is the optimal setting of MAX_DELTA_CACHE? The
"bouncing" in the linux.git log-S numbers notwithstanding,
it mostly seems like bigger is better. And even if we were
to try to find a "sweet spot", these are just two
repositories, that are not necessarily representative. The
shape of history, the size of trees and blobs, the memory
limit configuration, etc, all will affect the outcome.
Rather than trying to find the "right" number, another
strategy is to just switch to a hash table that can actually
store collisions: namely our hashmap.h implementation.
Here are numbers for that compared to the "best" we saw from
adjusting MAX_DELTA_CACHE:
| log-raw | log-S
| best hashmap | best hashmap
| --------- --------- | --------- ---------
git | 0m02.144s 0m02.144s | 0m06.448s 0m06.688s
linux | 0m33.311s 0m33.092s | 4m41.194s 4m57.172s
We can see the results are similar in most cases, which is
what we'd expect. We're not ejecting due to collisions at
all, so this is purely representing the LRU. So really, we'd
expect this to model most closely the larger values of the
static MAX_DELTA_CACHE limit. And that does seem to be
what's happening, including the "bounce" in the linux log-S
case.
So while the value for that case _isn't_ as good as the
optimal one measured above (which was 2048 entries), given
the bouncing I'm hesitant to suggest that 2048 is any kind
of optimum (not even for linux.git, let alone as a general
rule). The generic hashmap has the appeal that it drops the
number of tweakable numbers by one, which means we can focus
on tuning other elements, like the LRU strategy or the
core.deltaBaseCacheLimit setting.
And indeed, if we bump the cache limit to 1G (which is
probably silly for general use, but maybe something people
with big workstations would want to do), the linux.git log-S
time drops to 3m32s. That's something you really _can't_ do
easily with the static hash table, because the number of
entries needs to grow in proportion to the memory limit (so
2048 is almost certainly not going to be the right value
there).
This patch takes that direction, and drops the static hash
table entirely in favor of using the hashmap.h API.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-22 22:00:07 +00:00
|
|
|
return !!get_delta_base_cache_entry(p, base_offset);
|
2013-03-27 20:03:41 +00:00
|
|
|
}
|
|
|
|
|
clear_delta_base_cache_entry: use a more descriptive name
The delta base cache entries are stored in a fixed-length
hash table. So the way to remove an entry is to "clear" the
slot in the table, and that is what this function does.
However, the name is a leaky abstraction. If we were to
change the hash table implementation, it would no longer be
about "clearing". We should name it after _what_ it does,
not _how_ it does it. I.e., something like "remove" instead
of "clear".
But that does not tell the whole story, either. The subtle
thing about this function is that it removes the entry, but
does not free the entry data. So a more descriptive name is
"detach"; we give ownership of the data buffer to the
caller, and remove any other resources.
This patch uses the name detach_delta_base_cache_entry().
We could further model this after functions like
strbuf_detach(), which pass back all of the detached
information. However, since there are so many bits of
information in the struct (the data, the size, the type),
and so few callers (only one), it's not worth that
awkwardness. The name change and a comment can make the
intent clear.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-22 21:57:53 +00:00
|
|
|
/*
|
|
|
|
* Remove the entry from the cache, but do _not_ free the associated
|
|
|
|
* entry data. The caller takes ownership of the "data" buffer, and
|
|
|
|
* should copy out any fields it wants before detaching.
|
|
|
|
*/
|
|
|
|
static void detach_delta_base_cache_entry(struct delta_base_cache_entry *ent)
|
2013-03-27 20:03:41 +00:00
|
|
|
{
|
delta_base_cache: use hashmap.h
The fundamental data structure of the delta base cache is a
hash table mapping pairs of "(packfile, offset)" into
structs containing the actual object data. The hash table
implementation dates back to e5e0161 (Implement a simple
delta_base cache, 2007-03-17), and uses a fixed-size table.
The current size is a hard-coded 256 entries.
Because we need to be able to remove objects from the hash
table, entry lookup does not do any kind of probing to
handle collisions. Colliding items simply replace whatever
is in their slot. As a result, we have fewer usable slots
than even the 256 we allocate. At half full, each new item
has a 50% chance of displacing another one. Or another way
to think about it: every item has a 1/256 chance of being
ejected due to hash collision, without regard to our LRU
strategy.
So it would be interesting to see the effect of increasing
the cache size on the runtime for some common operations. As
with the previous patch, we'll measure "git log --raw" for
tree-only operations, and "git log -Sfoo --raw" for
operations that touch trees and blobs. All times are
wall-clock best-of-3, done against fully packed repos with
--depth=50, and the default core.deltaBaseCacheLimit of
96MB.
Here are timings for various values of MAX_DELTA_CACHE
against git.git (the asterisk marks the minimum time for
each operation):
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m02.227s 0m12.821s
512 0m02.143s 0m10.602s
1024 0m02.127s 0m08.642s
2048 0m02.148s 0m07.123s
4096 0m02.194s 0m06.448s*
8192 0m02.239s 0m06.504s
16384 0m02.144s* 0m06.502s
32768 0m02.202s 0m06.622s
65536 0m02.230s 0m06.677s
The log-raw case isn't changed much at all here (probably
because our trees just aren't that big in the first place,
or possibly because we have so _few_ trees in git.git that
the 256-entry cache is enough). But once we start putting
blobs in the cache, too, we see a big improvement (almost
50%). The curve levels off around 4096, which means that we
can hold about that many entries before hitting the 96MB
memory limit (or possibly that the workload is small enough
that there is simply no more work to be optimized out by
caching more).
(As a side note, I initially timed my existing git.git pack,
which was a base of --aggressive combined with some pulls on
top. So it had quite a few deeper delta chains. The
256-cache case was more like 15s, and it still dropped to
~6.5s in the same way).
Here are the timings for linux.git:
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m41.661s 5m12.410s
512 0m39.547s 5m07.920s
1024 0m37.054s 4m54.666s
2048 0m35.871s 4m41.194s*
4096 0m34.646s 4m51.648s
8192 0m33.881s 4m55.342s
16384 0m35.190s 5m00.122s
32768 0m35.060s 4m58.851s
65536 0m33.311s* 4m51.420s
As we grow we see a nice 20% speedup in the tree traversal,
and more modest 10% in the log-S. This is probably an
indication that we are bound less by the number of entries,
and more by the memory limit (more on that below). What is
interesting is that the numbers bounce around a bit;
increasing the number of entries isn't always a strict
improvement.
Partially this is due to noise in the measurement. But it
may also be an indication that our LRU ejection scheme is
not optimal. The smaller cache sizes introduce some
randomness into the ejection (due to collisions), which may
sometimes work in our favor (and sometimes not!).
So what is the optimal setting of MAX_DELTA_CACHE? The
"bouncing" in the linux.git log-S numbers notwithstanding,
it mostly seems like bigger is better. And even if we were
to try to find a "sweet spot", these are just two
repositories, that are not necessarily representative. The
shape of history, the size of trees and blobs, the memory
limit configuration, etc, all will affect the outcome.
Rather than trying to find the "right" number, another
strategy is to just switch to a hash table that can actually
store collisions: namely our hashmap.h implementation.
Here are numbers for that compared to the "best" we saw from
adjusting MAX_DELTA_CACHE:
| log-raw | log-S
| best hashmap | best hashmap
| --------- --------- | --------- ---------
git | 0m02.144s 0m02.144s | 0m06.448s 0m06.688s
linux | 0m33.311s 0m33.092s | 4m41.194s 4m57.172s
We can see the results are similar in most cases, which is
what we'd expect. We're not ejecting due to collisions at
all, so this is purely representing the LRU. So really, we'd
expect this to model most closely the larger values of the
static MAX_DELTA_CACHE limit. And that does seem to be
what's happening, including the "bounce" in the linux log-S
case.
So while the value for that case _isn't_ as good as the
optimal one measured above (which was 2048 entries), given
the bouncing I'm hesitant to suggest that 2048 is any kind
of optimum (not even for linux.git, let alone as a general
rule). The generic hashmap has the appeal that it drops the
number of tweakable numbers by one, which means we can focus
on tuning other elements, like the LRU strategy or the
core.deltaBaseCacheLimit setting.
And indeed, if we bump the cache limit to 1G (which is
probably silly for general use, but maybe something people
with big workstations would want to do), the linux.git log-S
time drops to 3m32s. That's something you really _can't_ do
easily with the static hash table, because the number of
entries needs to grow in proportion to the memory limit (so
2048 is almost certainly not going to be the right value
there).
This patch takes that direction, and drops the static hash
table entirely in favor of using the hashmap.h API.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-22 22:00:07 +00:00
|
|
|
hashmap_remove(&delta_base_cache, ent, &ent->key);
|
2016-08-22 21:59:42 +00:00
|
|
|
list_del(&ent->lru);
|
2013-03-27 20:03:41 +00:00
|
|
|
delta_base_cached -= ent->size;
|
delta_base_cache: use hashmap.h
The fundamental data structure of the delta base cache is a
hash table mapping pairs of "(packfile, offset)" into
structs containing the actual object data. The hash table
implementation dates back to e5e0161 (Implement a simple
delta_base cache, 2007-03-17), and uses a fixed-size table.
The current size is a hard-coded 256 entries.
Because we need to be able to remove objects from the hash
table, entry lookup does not do any kind of probing to
handle collisions. Colliding items simply replace whatever
is in their slot. As a result, we have fewer usable slots
than even the 256 we allocate. At half full, each new item
has a 50% chance of displacing another one. Or another way
to think about it: every item has a 1/256 chance of being
ejected due to hash collision, without regard to our LRU
strategy.
So it would be interesting to see the effect of increasing
the cache size on the runtime for some common operations. As
with the previous patch, we'll measure "git log --raw" for
tree-only operations, and "git log -Sfoo --raw" for
operations that touch trees and blobs. All times are
wall-clock best-of-3, done against fully packed repos with
--depth=50, and the default core.deltaBaseCacheLimit of
96MB.
Here are timings for various values of MAX_DELTA_CACHE
against git.git (the asterisk marks the minimum time for
each operation):
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m02.227s 0m12.821s
512 0m02.143s 0m10.602s
1024 0m02.127s 0m08.642s
2048 0m02.148s 0m07.123s
4096 0m02.194s 0m06.448s*
8192 0m02.239s 0m06.504s
16384 0m02.144s* 0m06.502s
32768 0m02.202s 0m06.622s
65536 0m02.230s 0m06.677s
The log-raw case isn't changed much at all here (probably
because our trees just aren't that big in the first place,
or possibly because we have so _few_ trees in git.git that
the 256-entry cache is enough). But once we start putting
blobs in the cache, too, we see a big improvement (almost
50%). The curve levels off around 4096, which means that we
can hold about that many entries before hitting the 96MB
memory limit (or possibly that the workload is small enough
that there is simply no more work to be optimized out by
caching more).
(As a side note, I initially timed my existing git.git pack,
which was a base of --aggressive combined with some pulls on
top. So it had quite a few deeper delta chains. The
256-cache case was more like 15s, and it still dropped to
~6.5s in the same way).
Here are the timings for linux.git:
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m41.661s 5m12.410s
512 0m39.547s 5m07.920s
1024 0m37.054s 4m54.666s
2048 0m35.871s 4m41.194s*
4096 0m34.646s 4m51.648s
8192 0m33.881s 4m55.342s
16384 0m35.190s 5m00.122s
32768 0m35.060s 4m58.851s
65536 0m33.311s* 4m51.420s
As we grow we see a nice 20% speedup in the tree traversal,
and more modest 10% in the log-S. This is probably an
indication that we are bound less by the number of entries,
and more by the memory limit (more on that below). What is
interesting is that the numbers bounce around a bit;
increasing the number of entries isn't always a strict
improvement.
Partially this is due to noise in the measurement. But it
may also be an indication that our LRU ejection scheme is
not optimal. The smaller cache sizes introduce some
randomness into the ejection (due to collisions), which may
sometimes work in our favor (and sometimes not!).
So what is the optimal setting of MAX_DELTA_CACHE? The
"bouncing" in the linux.git log-S numbers notwithstanding,
it mostly seems like bigger is better. And even if we were
to try to find a "sweet spot", these are just two
repositories, that are not necessarily representative. The
shape of history, the size of trees and blobs, the memory
limit configuration, etc, all will affect the outcome.
Rather than trying to find the "right" number, another
strategy is to just switch to a hash table that can actually
store collisions: namely our hashmap.h implementation.
Here are numbers for that compared to the "best" we saw from
adjusting MAX_DELTA_CACHE:
| log-raw | log-S
| best hashmap | best hashmap
| --------- --------- | --------- ---------
git | 0m02.144s 0m02.144s | 0m06.448s 0m06.688s
linux | 0m33.311s 0m33.092s | 4m41.194s 4m57.172s
We can see the results are similar in most cases, which is
what we'd expect. We're not ejecting due to collisions at
all, so this is purely representing the LRU. So really, we'd
expect this to model most closely the larger values of the
static MAX_DELTA_CACHE limit. And that does seem to be
what's happening, including the "bounce" in the linux log-S
case.
So while the value for that case _isn't_ as good as the
optimal one measured above (which was 2048 entries), given
the bouncing I'm hesitant to suggest that 2048 is any kind
of optimum (not even for linux.git, let alone as a general
rule). The generic hashmap has the appeal that it drops the
number of tweakable numbers by one, which means we can focus
on tuning other elements, like the LRU strategy or the
core.deltaBaseCacheLimit setting.
And indeed, if we bump the cache limit to 1G (which is
probably silly for general use, but maybe something people
with big workstations would want to do), the linux.git log-S
time drops to 3m32s. That's something you really _can't_ do
easily with the static hash table, because the number of
entries needs to grow in proportion to the memory limit (so
2048 is almost certainly not going to be the right value
there).
This patch takes that direction, and drops the static hash
table entirely in favor of using the hashmap.h API.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-22 22:00:07 +00:00
|
|
|
free(ent);
|
2013-03-27 20:03:41 +00:00
|
|
|
}
|
|
|
|
|
2007-03-17 19:42:15 +00:00
|
|
|
static void *cache_or_unpack_entry(struct packed_git *p, off_t base_offset,
|
2016-08-22 21:57:45 +00:00
|
|
|
unsigned long *base_size, enum object_type *type)
|
2007-03-17 19:42:15 +00:00
|
|
|
{
|
2013-03-27 20:03:41 +00:00
|
|
|
struct delta_base_cache_entry *ent;
|
2007-03-17 19:44:06 +00:00
|
|
|
|
2013-03-27 20:03:41 +00:00
|
|
|
ent = get_delta_base_cache_entry(p, base_offset);
|
delta_base_cache: use hashmap.h
The fundamental data structure of the delta base cache is a
hash table mapping pairs of "(packfile, offset)" into
structs containing the actual object data. The hash table
implementation dates back to e5e0161 (Implement a simple
delta_base cache, 2007-03-17), and uses a fixed-size table.
The current size is a hard-coded 256 entries.
Because we need to be able to remove objects from the hash
table, entry lookup does not do any kind of probing to
handle collisions. Colliding items simply replace whatever
is in their slot. As a result, we have fewer usable slots
than even the 256 we allocate. At half full, each new item
has a 50% chance of displacing another one. Or another way
to think about it: every item has a 1/256 chance of being
ejected due to hash collision, without regard to our LRU
strategy.
So it would be interesting to see the effect of increasing
the cache size on the runtime for some common operations. As
with the previous patch, we'll measure "git log --raw" for
tree-only operations, and "git log -Sfoo --raw" for
operations that touch trees and blobs. All times are
wall-clock best-of-3, done against fully packed repos with
--depth=50, and the default core.deltaBaseCacheLimit of
96MB.
Here are timings for various values of MAX_DELTA_CACHE
against git.git (the asterisk marks the minimum time for
each operation):
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m02.227s 0m12.821s
512 0m02.143s 0m10.602s
1024 0m02.127s 0m08.642s
2048 0m02.148s 0m07.123s
4096 0m02.194s 0m06.448s*
8192 0m02.239s 0m06.504s
16384 0m02.144s* 0m06.502s
32768 0m02.202s 0m06.622s
65536 0m02.230s 0m06.677s
The log-raw case isn't changed much at all here (probably
because our trees just aren't that big in the first place,
or possibly because we have so _few_ trees in git.git that
the 256-entry cache is enough). But once we start putting
blobs in the cache, too, we see a big improvement (almost
50%). The curve levels off around 4096, which means that we
can hold about that many entries before hitting the 96MB
memory limit (or possibly that the workload is small enough
that there is simply no more work to be optimized out by
caching more).
(As a side note, I initially timed my existing git.git pack,
which was a base of --aggressive combined with some pulls on
top. So it had quite a few deeper delta chains. The
256-cache case was more like 15s, and it still dropped to
~6.5s in the same way).
Here are the timings for linux.git:
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m41.661s 5m12.410s
512 0m39.547s 5m07.920s
1024 0m37.054s 4m54.666s
2048 0m35.871s 4m41.194s*
4096 0m34.646s 4m51.648s
8192 0m33.881s 4m55.342s
16384 0m35.190s 5m00.122s
32768 0m35.060s 4m58.851s
65536 0m33.311s* 4m51.420s
As we grow we see a nice 20% speedup in the tree traversal,
and more modest 10% in the log-S. This is probably an
indication that we are bound less by the number of entries,
and more by the memory limit (more on that below). What is
interesting is that the numbers bounce around a bit;
increasing the number of entries isn't always a strict
improvement.
Partially this is due to noise in the measurement. But it
may also be an indication that our LRU ejection scheme is
not optimal. The smaller cache sizes introduce some
randomness into the ejection (due to collisions), which may
sometimes work in our favor (and sometimes not!).
So what is the optimal setting of MAX_DELTA_CACHE? The
"bouncing" in the linux.git log-S numbers notwithstanding,
it mostly seems like bigger is better. And even if we were
to try to find a "sweet spot", these are just two
repositories, that are not necessarily representative. The
shape of history, the size of trees and blobs, the memory
limit configuration, etc, all will affect the outcome.
Rather than trying to find the "right" number, another
strategy is to just switch to a hash table that can actually
store collisions: namely our hashmap.h implementation.
Here are numbers for that compared to the "best" we saw from
adjusting MAX_DELTA_CACHE:
| log-raw | log-S
| best hashmap | best hashmap
| --------- --------- | --------- ---------
git | 0m02.144s 0m02.144s | 0m06.448s 0m06.688s
linux | 0m33.311s 0m33.092s | 4m41.194s 4m57.172s
We can see the results are similar in most cases, which is
what we'd expect. We're not ejecting due to collisions at
all, so this is purely representing the LRU. So really, we'd
expect this to model most closely the larger values of the
static MAX_DELTA_CACHE limit. And that does seem to be
what's happening, including the "bounce" in the linux log-S
case.
So while the value for that case _isn't_ as good as the
optimal one measured above (which was 2048 entries), given
the bouncing I'm hesitant to suggest that 2048 is any kind
of optimum (not even for linux.git, let alone as a general
rule). The generic hashmap has the appeal that it drops the
number of tweakable numbers by one, which means we can focus
on tuning other elements, like the LRU strategy or the
core.deltaBaseCacheLimit setting.
And indeed, if we bump the cache limit to 1G (which is
probably silly for general use, but maybe something people
with big workstations would want to do), the linux.git log-S
time drops to 3m32s. That's something you really _can't_ do
easily with the static hash table, because the number of
entries needs to grow in proportion to the memory limit (so
2048 is almost certainly not going to be the right value
there).
This patch takes that direction, and drops the static hash
table entirely in favor of using the hashmap.h API.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-22 22:00:07 +00:00
|
|
|
if (!ent)
|
2008-10-09 00:11:24 +00:00
|
|
|
return unpack_entry(p, base_offset, type, base_size);
|
2007-03-17 19:44:06 +00:00
|
|
|
|
|
|
|
*type = ent->type;
|
|
|
|
*base_size = ent->size;
|
2016-08-22 21:57:45 +00:00
|
|
|
return xmemdupz(ent->data, ent->size);
|
2007-03-17 19:42:15 +00:00
|
|
|
}
|
|
|
|
|
2007-03-19 05:14:37 +00:00
|
|
|
static inline void release_delta_base_cache(struct delta_base_cache_entry *ent)
|
|
|
|
{
|
delta_base_cache: use hashmap.h
The fundamental data structure of the delta base cache is a
hash table mapping pairs of "(packfile, offset)" into
structs containing the actual object data. The hash table
implementation dates back to e5e0161 (Implement a simple
delta_base cache, 2007-03-17), and uses a fixed-size table.
The current size is a hard-coded 256 entries.
Because we need to be able to remove objects from the hash
table, entry lookup does not do any kind of probing to
handle collisions. Colliding items simply replace whatever
is in their slot. As a result, we have fewer usable slots
than even the 256 we allocate. At half full, each new item
has a 50% chance of displacing another one. Or another way
to think about it: every item has a 1/256 chance of being
ejected due to hash collision, without regard to our LRU
strategy.
So it would be interesting to see the effect of increasing
the cache size on the runtime for some common operations. As
with the previous patch, we'll measure "git log --raw" for
tree-only operations, and "git log -Sfoo --raw" for
operations that touch trees and blobs. All times are
wall-clock best-of-3, done against fully packed repos with
--depth=50, and the default core.deltaBaseCacheLimit of
96MB.
Here are timings for various values of MAX_DELTA_CACHE
against git.git (the asterisk marks the minimum time for
each operation):
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m02.227s 0m12.821s
512 0m02.143s 0m10.602s
1024 0m02.127s 0m08.642s
2048 0m02.148s 0m07.123s
4096 0m02.194s 0m06.448s*
8192 0m02.239s 0m06.504s
16384 0m02.144s* 0m06.502s
32768 0m02.202s 0m06.622s
65536 0m02.230s 0m06.677s
The log-raw case isn't changed much at all here (probably
because our trees just aren't that big in the first place,
or possibly because we have so _few_ trees in git.git that
the 256-entry cache is enough). But once we start putting
blobs in the cache, too, we see a big improvement (almost
50%). The curve levels off around 4096, which means that we
can hold about that many entries before hitting the 96MB
memory limit (or possibly that the workload is small enough
that there is simply no more work to be optimized out by
caching more).
(As a side note, I initially timed my existing git.git pack,
which was a base of --aggressive combined with some pulls on
top. So it had quite a few deeper delta chains. The
256-cache case was more like 15s, and it still dropped to
~6.5s in the same way).
Here are the timings for linux.git:
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m41.661s 5m12.410s
512 0m39.547s 5m07.920s
1024 0m37.054s 4m54.666s
2048 0m35.871s 4m41.194s*
4096 0m34.646s 4m51.648s
8192 0m33.881s 4m55.342s
16384 0m35.190s 5m00.122s
32768 0m35.060s 4m58.851s
65536 0m33.311s* 4m51.420s
As we grow we see a nice 20% speedup in the tree traversal,
and more modest 10% in the log-S. This is probably an
indication that we are bound less by the number of entries,
and more by the memory limit (more on that below). What is
interesting is that the numbers bounce around a bit;
increasing the number of entries isn't always a strict
improvement.
Partially this is due to noise in the measurement. But it
may also be an indication that our LRU ejection scheme is
not optimal. The smaller cache sizes introduce some
randomness into the ejection (due to collisions), which may
sometimes work in our favor (and sometimes not!).
So what is the optimal setting of MAX_DELTA_CACHE? The
"bouncing" in the linux.git log-S numbers notwithstanding,
it mostly seems like bigger is better. And even if we were
to try to find a "sweet spot", these are just two
repositories, that are not necessarily representative. The
shape of history, the size of trees and blobs, the memory
limit configuration, etc, all will affect the outcome.
Rather than trying to find the "right" number, another
strategy is to just switch to a hash table that can actually
store collisions: namely our hashmap.h implementation.
Here are numbers for that compared to the "best" we saw from
adjusting MAX_DELTA_CACHE:
| log-raw | log-S
| best hashmap | best hashmap
| --------- --------- | --------- ---------
git | 0m02.144s 0m02.144s | 0m06.448s 0m06.688s
linux | 0m33.311s 0m33.092s | 4m41.194s 4m57.172s
We can see the results are similar in most cases, which is
what we'd expect. We're not ejecting due to collisions at
all, so this is purely representing the LRU. So really, we'd
expect this to model most closely the larger values of the
static MAX_DELTA_CACHE limit. And that does seem to be
what's happening, including the "bounce" in the linux log-S
case.
So while the value for that case _isn't_ as good as the
optimal one measured above (which was 2048 entries), given
the bouncing I'm hesitant to suggest that 2048 is any kind
of optimum (not even for linux.git, let alone as a general
rule). The generic hashmap has the appeal that it drops the
number of tweakable numbers by one, which means we can focus
on tuning other elements, like the LRU strategy or the
core.deltaBaseCacheLimit setting.
And indeed, if we bump the cache limit to 1G (which is
probably silly for general use, but maybe something people
with big workstations would want to do), the linux.git log-S
time drops to 3m32s. That's something you really _can't_ do
easily with the static hash table, because the number of
entries needs to grow in proportion to the memory limit (so
2048 is almost certainly not going to be the right value
there).
This patch takes that direction, and drops the static hash
table entirely in favor of using the hashmap.h API.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-22 22:00:07 +00:00
|
|
|
free(ent->data);
|
|
|
|
detach_delta_base_cache_entry(ent);
|
2007-03-19 05:14:37 +00:00
|
|
|
}
|
|
|
|
|
2009-02-10 21:36:12 +00:00
|
|
|
void clear_delta_base_cache(void)
|
|
|
|
{
|
clear_delta_base_cache(): don't modify hashmap while iterating
On Thu, Jan 19, 2017 at 03:03:46PM +0100, Ulrich Spörlein wrote:
> > I suspect the patch below may fix things for you. It works around it by
> > walking over the lru list (either is fine, as they both contain all
> > entries, and since we're clearing everything, we don't care about the
> > order).
>
> Confirmed. With the patch applied, I can import the whole 55G in one go
> without any crashes or aborts. Thanks much!
Thanks. Here it is rolled up with a commit message.
-- >8 --
Subject: clear_delta_base_cache(): don't modify hashmap while iterating
Removing entries while iterating causes fast-import to
access an already-freed `struct packed_git`, leading to
various confusing errors.
What happens is that clear_delta_base_cache() drops the
whole contents of the cache by iterating over the hashmap,
calling release_delta_base_cache() on each entry. That
function removes the item from the hashmap. The hashmap code
may then shrink the table, but the hashmap_iter struct
retains an offset from the old table.
As a result, the next call to hashmap_iter_next() may claim
that the iteration is done, even though some items haven't
been visited.
The only caller of clear_delta_base_cache() is fast-import,
which wants to clear the cache because it is discarding the
packed_git struct for its temporary pack. So by failing to
remove all of the entries, we still have references to the
freed packed_git.
To make things even more confusing, this doesn't seem to
trigger with the test suite, because it depends on
complexities like the size of the hash table, which entries
got cleared, whether we try to access them before they're
evicted from the cache, etc.
So I've been able to identify the problem with large
imports like freebsd's svn import, or a fast-export of
linux.git. But nothing that would be reasonable to run as
part of the normal test suite.
We can fix this easily by iterating over the lru linked list
instead of the hashmap. They both contain the same entries,
and we can use the "safe" variant of the list iterator,
which exists for exactly this case.
Let's also add a warning to the hashmap API documentation to
reduce the chances of getting bit by this again.
Reported-by: Ulrich Spörlein <uqs@freebsd.org>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-01-19 16:33:50 +00:00
|
|
|
struct list_head *lru, *tmp;
|
|
|
|
list_for_each_safe(lru, tmp, &delta_base_cache_lru) {
|
|
|
|
struct delta_base_cache_entry *entry =
|
|
|
|
list_entry(lru, struct delta_base_cache_entry, lru);
|
delta_base_cache: use hashmap.h
The fundamental data structure of the delta base cache is a
hash table mapping pairs of "(packfile, offset)" into
structs containing the actual object data. The hash table
implementation dates back to e5e0161 (Implement a simple
delta_base cache, 2007-03-17), and uses a fixed-size table.
The current size is a hard-coded 256 entries.
Because we need to be able to remove objects from the hash
table, entry lookup does not do any kind of probing to
handle collisions. Colliding items simply replace whatever
is in their slot. As a result, we have fewer usable slots
than even the 256 we allocate. At half full, each new item
has a 50% chance of displacing another one. Or another way
to think about it: every item has a 1/256 chance of being
ejected due to hash collision, without regard to our LRU
strategy.
So it would be interesting to see the effect of increasing
the cache size on the runtime for some common operations. As
with the previous patch, we'll measure "git log --raw" for
tree-only operations, and "git log -Sfoo --raw" for
operations that touch trees and blobs. All times are
wall-clock best-of-3, done against fully packed repos with
--depth=50, and the default core.deltaBaseCacheLimit of
96MB.
Here are timings for various values of MAX_DELTA_CACHE
against git.git (the asterisk marks the minimum time for
each operation):
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m02.227s 0m12.821s
512 0m02.143s 0m10.602s
1024 0m02.127s 0m08.642s
2048 0m02.148s 0m07.123s
4096 0m02.194s 0m06.448s*
8192 0m02.239s 0m06.504s
16384 0m02.144s* 0m06.502s
32768 0m02.202s 0m06.622s
65536 0m02.230s 0m06.677s
The log-raw case isn't changed much at all here (probably
because our trees just aren't that big in the first place,
or possibly because we have so _few_ trees in git.git that
the 256-entry cache is enough). But once we start putting
blobs in the cache, too, we see a big improvement (almost
50%). The curve levels off around 4096, which means that we
can hold about that many entries before hitting the 96MB
memory limit (or possibly that the workload is small enough
that there is simply no more work to be optimized out by
caching more).
(As a side note, I initially timed my existing git.git pack,
which was a base of --aggressive combined with some pulls on
top. So it had quite a few deeper delta chains. The
256-cache case was more like 15s, and it still dropped to
~6.5s in the same way).
Here are the timings for linux.git:
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m41.661s 5m12.410s
512 0m39.547s 5m07.920s
1024 0m37.054s 4m54.666s
2048 0m35.871s 4m41.194s*
4096 0m34.646s 4m51.648s
8192 0m33.881s 4m55.342s
16384 0m35.190s 5m00.122s
32768 0m35.060s 4m58.851s
65536 0m33.311s* 4m51.420s
As we grow we see a nice 20% speedup in the tree traversal,
and more modest 10% in the log-S. This is probably an
indication that we are bound less by the number of entries,
and more by the memory limit (more on that below). What is
interesting is that the numbers bounce around a bit;
increasing the number of entries isn't always a strict
improvement.
Partially this is due to noise in the measurement. But it
may also be an indication that our LRU ejection scheme is
not optimal. The smaller cache sizes introduce some
randomness into the ejection (due to collisions), which may
sometimes work in our favor (and sometimes not!).
So what is the optimal setting of MAX_DELTA_CACHE? The
"bouncing" in the linux.git log-S numbers notwithstanding,
it mostly seems like bigger is better. And even if we were
to try to find a "sweet spot", these are just two
repositories, that are not necessarily representative. The
shape of history, the size of trees and blobs, the memory
limit configuration, etc, all will affect the outcome.
Rather than trying to find the "right" number, another
strategy is to just switch to a hash table that can actually
store collisions: namely our hashmap.h implementation.
Here are numbers for that compared to the "best" we saw from
adjusting MAX_DELTA_CACHE:
| log-raw | log-S
| best hashmap | best hashmap
| --------- --------- | --------- ---------
git | 0m02.144s 0m02.144s | 0m06.448s 0m06.688s
linux | 0m33.311s 0m33.092s | 4m41.194s 4m57.172s
We can see the results are similar in most cases, which is
what we'd expect. We're not ejecting due to collisions at
all, so this is purely representing the LRU. So really, we'd
expect this to model most closely the larger values of the
static MAX_DELTA_CACHE limit. And that does seem to be
what's happening, including the "bounce" in the linux log-S
case.
So while the value for that case _isn't_ as good as the
optimal one measured above (which was 2048 entries), given
the bouncing I'm hesitant to suggest that 2048 is any kind
of optimum (not even for linux.git, let alone as a general
rule). The generic hashmap has the appeal that it drops the
number of tweakable numbers by one, which means we can focus
on tuning other elements, like the LRU strategy or the
core.deltaBaseCacheLimit setting.
And indeed, if we bump the cache limit to 1G (which is
probably silly for general use, but maybe something people
with big workstations would want to do), the linux.git log-S
time drops to 3m32s. That's something you really _can't_ do
easily with the static hash table, because the number of
entries needs to grow in proportion to the memory limit (so
2048 is almost certainly not going to be the right value
there).
This patch takes that direction, and drops the static hash
table entirely in favor of using the hashmap.h API.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-22 22:00:07 +00:00
|
|
|
release_delta_base_cache(entry);
|
|
|
|
}
|
2009-02-10 21:36:12 +00:00
|
|
|
}
|
|
|
|
|
2007-03-17 19:42:15 +00:00
|
|
|
static void add_delta_base_cache(struct packed_git *p, off_t base_offset,
|
|
|
|
void *base, unsigned long base_size, enum object_type type)
|
|
|
|
{
|
delta_base_cache: use hashmap.h
The fundamental data structure of the delta base cache is a
hash table mapping pairs of "(packfile, offset)" into
structs containing the actual object data. The hash table
implementation dates back to e5e0161 (Implement a simple
delta_base cache, 2007-03-17), and uses a fixed-size table.
The current size is a hard-coded 256 entries.
Because we need to be able to remove objects from the hash
table, entry lookup does not do any kind of probing to
handle collisions. Colliding items simply replace whatever
is in their slot. As a result, we have fewer usable slots
than even the 256 we allocate. At half full, each new item
has a 50% chance of displacing another one. Or another way
to think about it: every item has a 1/256 chance of being
ejected due to hash collision, without regard to our LRU
strategy.
So it would be interesting to see the effect of increasing
the cache size on the runtime for some common operations. As
with the previous patch, we'll measure "git log --raw" for
tree-only operations, and "git log -Sfoo --raw" for
operations that touch trees and blobs. All times are
wall-clock best-of-3, done against fully packed repos with
--depth=50, and the default core.deltaBaseCacheLimit of
96MB.
Here are timings for various values of MAX_DELTA_CACHE
against git.git (the asterisk marks the minimum time for
each operation):
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m02.227s 0m12.821s
512 0m02.143s 0m10.602s
1024 0m02.127s 0m08.642s
2048 0m02.148s 0m07.123s
4096 0m02.194s 0m06.448s*
8192 0m02.239s 0m06.504s
16384 0m02.144s* 0m06.502s
32768 0m02.202s 0m06.622s
65536 0m02.230s 0m06.677s
The log-raw case isn't changed much at all here (probably
because our trees just aren't that big in the first place,
or possibly because we have so _few_ trees in git.git that
the 256-entry cache is enough). But once we start putting
blobs in the cache, too, we see a big improvement (almost
50%). The curve levels off around 4096, which means that we
can hold about that many entries before hitting the 96MB
memory limit (or possibly that the workload is small enough
that there is simply no more work to be optimized out by
caching more).
(As a side note, I initially timed my existing git.git pack,
which was a base of --aggressive combined with some pulls on
top. So it had quite a few deeper delta chains. The
256-cache case was more like 15s, and it still dropped to
~6.5s in the same way).
Here are the timings for linux.git:
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m41.661s 5m12.410s
512 0m39.547s 5m07.920s
1024 0m37.054s 4m54.666s
2048 0m35.871s 4m41.194s*
4096 0m34.646s 4m51.648s
8192 0m33.881s 4m55.342s
16384 0m35.190s 5m00.122s
32768 0m35.060s 4m58.851s
65536 0m33.311s* 4m51.420s
As we grow we see a nice 20% speedup in the tree traversal,
and more modest 10% in the log-S. This is probably an
indication that we are bound less by the number of entries,
and more by the memory limit (more on that below). What is
interesting is that the numbers bounce around a bit;
increasing the number of entries isn't always a strict
improvement.
Partially this is due to noise in the measurement. But it
may also be an indication that our LRU ejection scheme is
not optimal. The smaller cache sizes introduce some
randomness into the ejection (due to collisions), which may
sometimes work in our favor (and sometimes not!).
So what is the optimal setting of MAX_DELTA_CACHE? The
"bouncing" in the linux.git log-S numbers notwithstanding,
it mostly seems like bigger is better. And even if we were
to try to find a "sweet spot", these are just two
repositories, that are not necessarily representative. The
shape of history, the size of trees and blobs, the memory
limit configuration, etc, all will affect the outcome.
Rather than trying to find the "right" number, another
strategy is to just switch to a hash table that can actually
store collisions: namely our hashmap.h implementation.
Here are numbers for that compared to the "best" we saw from
adjusting MAX_DELTA_CACHE:
| log-raw | log-S
| best hashmap | best hashmap
| --------- --------- | --------- ---------
git | 0m02.144s 0m02.144s | 0m06.448s 0m06.688s
linux | 0m33.311s 0m33.092s | 4m41.194s 4m57.172s
We can see the results are similar in most cases, which is
what we'd expect. We're not ejecting due to collisions at
all, so this is purely representing the LRU. So really, we'd
expect this to model most closely the larger values of the
static MAX_DELTA_CACHE limit. And that does seem to be
what's happening, including the "bounce" in the linux log-S
case.
So while the value for that case _isn't_ as good as the
optimal one measured above (which was 2048 entries), given
the bouncing I'm hesitant to suggest that 2048 is any kind
of optimum (not even for linux.git, let alone as a general
rule). The generic hashmap has the appeal that it drops the
number of tweakable numbers by one, which means we can focus
on tuning other elements, like the LRU strategy or the
core.deltaBaseCacheLimit setting.
And indeed, if we bump the cache limit to 1G (which is
probably silly for general use, but maybe something people
with big workstations would want to do), the linux.git log-S
time drops to 3m32s. That's something you really _can't_ do
easily with the static hash table, because the number of
entries needs to grow in proportion to the memory limit (so
2048 is almost certainly not going to be the right value
there).
This patch takes that direction, and drops the static hash
table entirely in favor of using the hashmap.h API.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-22 22:00:07 +00:00
|
|
|
struct delta_base_cache_entry *ent = xmalloc(sizeof(*ent));
|
2016-09-12 16:46:17 +00:00
|
|
|
struct list_head *lru, *tmp;
|
2007-03-17 19:44:06 +00:00
|
|
|
|
2007-03-19 05:14:37 +00:00
|
|
|
delta_base_cached += base_size;
|
2007-03-19 20:31:04 +00:00
|
|
|
|
2016-09-12 16:46:17 +00:00
|
|
|
list_for_each_safe(lru, tmp, &delta_base_cache_lru) {
|
2016-08-22 21:59:42 +00:00
|
|
|
struct delta_base_cache_entry *f =
|
|
|
|
list_entry(lru, struct delta_base_cache_entry, lru);
|
|
|
|
if (delta_base_cached <= delta_base_cache_limit)
|
|
|
|
break;
|
2007-03-19 20:31:04 +00:00
|
|
|
release_delta_base_cache(f);
|
|
|
|
}
|
2007-03-19 05:14:37 +00:00
|
|
|
|
delta_base_cache: use hashmap.h
The fundamental data structure of the delta base cache is a
hash table mapping pairs of "(packfile, offset)" into
structs containing the actual object data. The hash table
implementation dates back to e5e0161 (Implement a simple
delta_base cache, 2007-03-17), and uses a fixed-size table.
The current size is a hard-coded 256 entries.
Because we need to be able to remove objects from the hash
table, entry lookup does not do any kind of probing to
handle collisions. Colliding items simply replace whatever
is in their slot. As a result, we have fewer usable slots
than even the 256 we allocate. At half full, each new item
has a 50% chance of displacing another one. Or another way
to think about it: every item has a 1/256 chance of being
ejected due to hash collision, without regard to our LRU
strategy.
So it would be interesting to see the effect of increasing
the cache size on the runtime for some common operations. As
with the previous patch, we'll measure "git log --raw" for
tree-only operations, and "git log -Sfoo --raw" for
operations that touch trees and blobs. All times are
wall-clock best-of-3, done against fully packed repos with
--depth=50, and the default core.deltaBaseCacheLimit of
96MB.
Here are timings for various values of MAX_DELTA_CACHE
against git.git (the asterisk marks the minimum time for
each operation):
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m02.227s 0m12.821s
512 0m02.143s 0m10.602s
1024 0m02.127s 0m08.642s
2048 0m02.148s 0m07.123s
4096 0m02.194s 0m06.448s*
8192 0m02.239s 0m06.504s
16384 0m02.144s* 0m06.502s
32768 0m02.202s 0m06.622s
65536 0m02.230s 0m06.677s
The log-raw case isn't changed much at all here (probably
because our trees just aren't that big in the first place,
or possibly because we have so _few_ trees in git.git that
the 256-entry cache is enough). But once we start putting
blobs in the cache, too, we see a big improvement (almost
50%). The curve levels off around 4096, which means that we
can hold about that many entries before hitting the 96MB
memory limit (or possibly that the workload is small enough
that there is simply no more work to be optimized out by
caching more).
(As a side note, I initially timed my existing git.git pack,
which was a base of --aggressive combined with some pulls on
top. So it had quite a few deeper delta chains. The
256-cache case was more like 15s, and it still dropped to
~6.5s in the same way).
Here are the timings for linux.git:
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m41.661s 5m12.410s
512 0m39.547s 5m07.920s
1024 0m37.054s 4m54.666s
2048 0m35.871s 4m41.194s*
4096 0m34.646s 4m51.648s
8192 0m33.881s 4m55.342s
16384 0m35.190s 5m00.122s
32768 0m35.060s 4m58.851s
65536 0m33.311s* 4m51.420s
As we grow we see a nice 20% speedup in the tree traversal,
and more modest 10% in the log-S. This is probably an
indication that we are bound less by the number of entries,
and more by the memory limit (more on that below). What is
interesting is that the numbers bounce around a bit;
increasing the number of entries isn't always a strict
improvement.
Partially this is due to noise in the measurement. But it
may also be an indication that our LRU ejection scheme is
not optimal. The smaller cache sizes introduce some
randomness into the ejection (due to collisions), which may
sometimes work in our favor (and sometimes not!).
So what is the optimal setting of MAX_DELTA_CACHE? The
"bouncing" in the linux.git log-S numbers notwithstanding,
it mostly seems like bigger is better. And even if we were
to try to find a "sweet spot", these are just two
repositories, that are not necessarily representative. The
shape of history, the size of trees and blobs, the memory
limit configuration, etc, all will affect the outcome.
Rather than trying to find the "right" number, another
strategy is to just switch to a hash table that can actually
store collisions: namely our hashmap.h implementation.
Here are numbers for that compared to the "best" we saw from
adjusting MAX_DELTA_CACHE:
| log-raw | log-S
| best hashmap | best hashmap
| --------- --------- | --------- ---------
git | 0m02.144s 0m02.144s | 0m06.448s 0m06.688s
linux | 0m33.311s 0m33.092s | 4m41.194s 4m57.172s
We can see the results are similar in most cases, which is
what we'd expect. We're not ejecting due to collisions at
all, so this is purely representing the LRU. So really, we'd
expect this to model most closely the larger values of the
static MAX_DELTA_CACHE limit. And that does seem to be
what's happening, including the "bounce" in the linux log-S
case.
So while the value for that case _isn't_ as good as the
optimal one measured above (which was 2048 entries), given
the bouncing I'm hesitant to suggest that 2048 is any kind
of optimum (not even for linux.git, let alone as a general
rule). The generic hashmap has the appeal that it drops the
number of tweakable numbers by one, which means we can focus
on tuning other elements, like the LRU strategy or the
core.deltaBaseCacheLimit setting.
And indeed, if we bump the cache limit to 1G (which is
probably silly for general use, but maybe something people
with big workstations would want to do), the linux.git log-S
time drops to 3m32s. That's something you really _can't_ do
easily with the static hash table, because the number of
entries needs to grow in proportion to the memory limit (so
2048 is almost certainly not going to be the right value
there).
This patch takes that direction, and drops the static hash
table entirely in favor of using the hashmap.h API.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-22 22:00:07 +00:00
|
|
|
ent->key.p = p;
|
|
|
|
ent->key.base_offset = base_offset;
|
2007-03-17 19:44:06 +00:00
|
|
|
ent->type = type;
|
|
|
|
ent->data = base;
|
|
|
|
ent->size = base_size;
|
2016-08-22 21:59:42 +00:00
|
|
|
list_add_tail(&ent->lru, &delta_base_cache_lru);
|
delta_base_cache: use hashmap.h
The fundamental data structure of the delta base cache is a
hash table mapping pairs of "(packfile, offset)" into
structs containing the actual object data. The hash table
implementation dates back to e5e0161 (Implement a simple
delta_base cache, 2007-03-17), and uses a fixed-size table.
The current size is a hard-coded 256 entries.
Because we need to be able to remove objects from the hash
table, entry lookup does not do any kind of probing to
handle collisions. Colliding items simply replace whatever
is in their slot. As a result, we have fewer usable slots
than even the 256 we allocate. At half full, each new item
has a 50% chance of displacing another one. Or another way
to think about it: every item has a 1/256 chance of being
ejected due to hash collision, without regard to our LRU
strategy.
So it would be interesting to see the effect of increasing
the cache size on the runtime for some common operations. As
with the previous patch, we'll measure "git log --raw" for
tree-only operations, and "git log -Sfoo --raw" for
operations that touch trees and blobs. All times are
wall-clock best-of-3, done against fully packed repos with
--depth=50, and the default core.deltaBaseCacheLimit of
96MB.
Here are timings for various values of MAX_DELTA_CACHE
against git.git (the asterisk marks the minimum time for
each operation):
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m02.227s 0m12.821s
512 0m02.143s 0m10.602s
1024 0m02.127s 0m08.642s
2048 0m02.148s 0m07.123s
4096 0m02.194s 0m06.448s*
8192 0m02.239s 0m06.504s
16384 0m02.144s* 0m06.502s
32768 0m02.202s 0m06.622s
65536 0m02.230s 0m06.677s
The log-raw case isn't changed much at all here (probably
because our trees just aren't that big in the first place,
or possibly because we have so _few_ trees in git.git that
the 256-entry cache is enough). But once we start putting
blobs in the cache, too, we see a big improvement (almost
50%). The curve levels off around 4096, which means that we
can hold about that many entries before hitting the 96MB
memory limit (or possibly that the workload is small enough
that there is simply no more work to be optimized out by
caching more).
(As a side note, I initially timed my existing git.git pack,
which was a base of --aggressive combined with some pulls on
top. So it had quite a few deeper delta chains. The
256-cache case was more like 15s, and it still dropped to
~6.5s in the same way).
Here are the timings for linux.git:
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m41.661s 5m12.410s
512 0m39.547s 5m07.920s
1024 0m37.054s 4m54.666s
2048 0m35.871s 4m41.194s*
4096 0m34.646s 4m51.648s
8192 0m33.881s 4m55.342s
16384 0m35.190s 5m00.122s
32768 0m35.060s 4m58.851s
65536 0m33.311s* 4m51.420s
As we grow we see a nice 20% speedup in the tree traversal,
and more modest 10% in the log-S. This is probably an
indication that we are bound less by the number of entries,
and more by the memory limit (more on that below). What is
interesting is that the numbers bounce around a bit;
increasing the number of entries isn't always a strict
improvement.
Partially this is due to noise in the measurement. But it
may also be an indication that our LRU ejection scheme is
not optimal. The smaller cache sizes introduce some
randomness into the ejection (due to collisions), which may
sometimes work in our favor (and sometimes not!).
So what is the optimal setting of MAX_DELTA_CACHE? The
"bouncing" in the linux.git log-S numbers notwithstanding,
it mostly seems like bigger is better. And even if we were
to try to find a "sweet spot", these are just two
repositories, that are not necessarily representative. The
shape of history, the size of trees and blobs, the memory
limit configuration, etc, all will affect the outcome.
Rather than trying to find the "right" number, another
strategy is to just switch to a hash table that can actually
store collisions: namely our hashmap.h implementation.
Here are numbers for that compared to the "best" we saw from
adjusting MAX_DELTA_CACHE:
| log-raw | log-S
| best hashmap | best hashmap
| --------- --------- | --------- ---------
git | 0m02.144s 0m02.144s | 0m06.448s 0m06.688s
linux | 0m33.311s 0m33.092s | 4m41.194s 4m57.172s
We can see the results are similar in most cases, which is
what we'd expect. We're not ejecting due to collisions at
all, so this is purely representing the LRU. So really, we'd
expect this to model most closely the larger values of the
static MAX_DELTA_CACHE limit. And that does seem to be
what's happening, including the "bounce" in the linux log-S
case.
So while the value for that case _isn't_ as good as the
optimal one measured above (which was 2048 entries), given
the bouncing I'm hesitant to suggest that 2048 is any kind
of optimum (not even for linux.git, let alone as a general
rule). The generic hashmap has the appeal that it drops the
number of tweakable numbers by one, which means we can focus
on tuning other elements, like the LRU strategy or the
core.deltaBaseCacheLimit setting.
And indeed, if we bump the cache limit to 1G (which is
probably silly for general use, but maybe something people
with big workstations would want to do), the linux.git log-S
time drops to 3m32s. That's something you really _can't_ do
easily with the static hash table, because the number of
entries needs to grow in proportion to the memory limit (so
2048 is almost certainly not going to be the right value
there).
This patch takes that direction, and drops the static hash
table entirely in favor of using the hashmap.h API.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-22 22:00:07 +00:00
|
|
|
|
|
|
|
if (!delta_base_cache.cmpfn)
|
|
|
|
hashmap_init(&delta_base_cache, delta_base_cache_hash_cmp, 0);
|
|
|
|
hashmap_entry_init(ent, pack_entry_hash(p, base_offset));
|
|
|
|
hashmap_add(&delta_base_cache, ent);
|
2007-03-17 19:42:15 +00:00
|
|
|
}
|
|
|
|
|
2009-01-12 17:42:24 +00:00
|
|
|
static void *read_object(const unsigned char *sha1, enum object_type *type,
|
|
|
|
unsigned long *size);
|
|
|
|
|
2011-07-07 02:08:55 +00:00
|
|
|
static void write_pack_access_log(struct packed_git *p, off_t obj_offset)
|
|
|
|
{
|
2014-07-12 00:01:38 +00:00
|
|
|
static struct trace_key pack_access = TRACE_KEY_INIT(PACK_ACCESS);
|
|
|
|
trace_printf_key(&pack_access, "%s %"PRIuMAX"\n",
|
|
|
|
p->pack_name, (uintmax_t)obj_offset);
|
2011-07-07 02:08:55 +00:00
|
|
|
}
|
|
|
|
|
close another possibility for propagating pack corruption
Abstract
--------
With index v2 we have a per object CRC to allow quick and safe reuse of
pack data when repacking. This, however, doesn't currently prevent a
stealth corruption from being propagated into a new pack when _not_
reusing pack data as demonstrated by the modification to t5302 included
here.
The Context
-----------
The Git database is all checksummed with SHA1 hashes. Any kind of
corruption can be confirmed by verifying this per object hash against
corresponding data. However this can be costly to perform systematically
and therefore this check is often not performed at run time when
accessing the object database.
First, the loose object format is entirely compressed with zlib which
already provide a CRC verification of its own when inflating data. Any
disk corruption would be caught already in this case.
Then, packed objects are also compressed with zlib but only for their
actual payload. The object headers and delta base references are not
deflated for obvious performance reasons, however this leave them
vulnerable to potentially undetected disk corruptions. Object types
are often validated against the expected type when they're requested,
and deflated size must always match the size recorded in the object header,
so those cases are pretty much covered as well.
Where corruptions could go unnoticed is in the delta base reference.
Of course, in the OBJ_REF_DELTA case, the odds for a SHA1 reference to
get corrupted so it actually matches the SHA1 of another object with the
same size (the delta header stores the expected size of the base object
to apply against) are virtually zero. In the OBJ_OFS_DELTA case, the
reference is a pack offset which would have to match the start boundary
of a different base object but still with the same size, and although this
is relatively much more "probable" than in the OBJ_REF_DELTA case, the
probability is also about zero in absolute terms. Still, the possibility
exists as demonstrated in t5302 and is certainly greater than a SHA1
collision, especially in the OBJ_OFS_DELTA case which is now the default
when repacking.
Again, repacking by reusing existing pack data is OK since the per object
CRC provided by index v2 guards against any such corruptions. What t5302
failed to test is a full repack in such case.
The Solution
------------
As unlikely as this kind of stealth corruption can be in practice, it
certainly isn't acceptable to propagate it into a freshly created pack.
But, because this is so unlikely, we don't want to pay the run time cost
associated with extra validation checks all the time either. Furthermore,
consequences of such corruption in anything but repacking should be rather
visible, and even if it could be quite unpleasant, it still has far less
severe consequences than actively creating bad packs.
So the best compromize is to check packed object CRC when unpacking
objects, and only during the compression/writing phase of a repack, and
only when not streaming the result. The cost of this is minimal (less
than 1% CPU time), and visible only with a full repack.
Someone with a stats background could provide an objective evaluation of
this, but I suspect that it's bad RAM that has more potential for data
corruptions at this point, even in those cases where this extra check
is not performed. Still, it is best to prevent a known hole for
corruption when recreating object data into a new pack.
What about the streamed pack case? Well, any client receiving a pack
must always consider that pack as untrusty and perform full validation
anyway, hence no such stealth corruption could be propagated to remote
repositoryes already. It is therefore worthless doing local validation
in that case.
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-10-31 15:31:08 +00:00
|
|
|
int do_check_packed_object_crc;
|
|
|
|
|
2013-03-27 20:03:42 +00:00
|
|
|
#define UNPACK_ENTRY_STACK_PREALLOC 64
|
|
|
|
struct unpack_entry_stack_ent {
|
|
|
|
off_t obj_offset;
|
|
|
|
off_t curpos;
|
|
|
|
unsigned long size;
|
|
|
|
};
|
|
|
|
|
2007-03-07 01:44:30 +00:00
|
|
|
void *unpack_entry(struct packed_git *p, off_t obj_offset,
|
2013-03-27 20:03:42 +00:00
|
|
|
enum object_type *final_type, unsigned long *final_size)
|
2005-06-27 10:35:33 +00:00
|
|
|
{
|
2006-12-23 07:34:08 +00:00
|
|
|
struct pack_window *w_curs = NULL;
|
2007-03-07 01:44:30 +00:00
|
|
|
off_t curpos = obj_offset;
|
2013-03-27 20:03:42 +00:00
|
|
|
void *data = NULL;
|
|
|
|
unsigned long size;
|
|
|
|
enum object_type type;
|
|
|
|
struct unpack_entry_stack_ent small_delta_stack[UNPACK_ENTRY_STACK_PREALLOC];
|
|
|
|
struct unpack_entry_stack_ent *delta_stack = small_delta_stack;
|
|
|
|
int delta_stack_nr = 0, delta_stack_alloc = UNPACK_ENTRY_STACK_PREALLOC;
|
|
|
|
int base_from_cache = 0;
|
2005-06-27 10:35:33 +00:00
|
|
|
|
2014-07-12 00:01:38 +00:00
|
|
|
write_pack_access_log(p, obj_offset);
|
2011-07-07 02:08:55 +00:00
|
|
|
|
2013-03-27 20:03:42 +00:00
|
|
|
/* PHASE 1: drill down to the innermost base object */
|
|
|
|
for (;;) {
|
|
|
|
off_t base_offset;
|
|
|
|
int i;
|
|
|
|
struct delta_base_cache_entry *ent;
|
|
|
|
|
2013-09-13 11:03:00 +00:00
|
|
|
ent = get_delta_base_cache_entry(p, curpos);
|
delta_base_cache: use hashmap.h
The fundamental data structure of the delta base cache is a
hash table mapping pairs of "(packfile, offset)" into
structs containing the actual object data. The hash table
implementation dates back to e5e0161 (Implement a simple
delta_base cache, 2007-03-17), and uses a fixed-size table.
The current size is a hard-coded 256 entries.
Because we need to be able to remove objects from the hash
table, entry lookup does not do any kind of probing to
handle collisions. Colliding items simply replace whatever
is in their slot. As a result, we have fewer usable slots
than even the 256 we allocate. At half full, each new item
has a 50% chance of displacing another one. Or another way
to think about it: every item has a 1/256 chance of being
ejected due to hash collision, without regard to our LRU
strategy.
So it would be interesting to see the effect of increasing
the cache size on the runtime for some common operations. As
with the previous patch, we'll measure "git log --raw" for
tree-only operations, and "git log -Sfoo --raw" for
operations that touch trees and blobs. All times are
wall-clock best-of-3, done against fully packed repos with
--depth=50, and the default core.deltaBaseCacheLimit of
96MB.
Here are timings for various values of MAX_DELTA_CACHE
against git.git (the asterisk marks the minimum time for
each operation):
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m02.227s 0m12.821s
512 0m02.143s 0m10.602s
1024 0m02.127s 0m08.642s
2048 0m02.148s 0m07.123s
4096 0m02.194s 0m06.448s*
8192 0m02.239s 0m06.504s
16384 0m02.144s* 0m06.502s
32768 0m02.202s 0m06.622s
65536 0m02.230s 0m06.677s
The log-raw case isn't changed much at all here (probably
because our trees just aren't that big in the first place,
or possibly because we have so _few_ trees in git.git that
the 256-entry cache is enough). But once we start putting
blobs in the cache, too, we see a big improvement (almost
50%). The curve levels off around 4096, which means that we
can hold about that many entries before hitting the 96MB
memory limit (or possibly that the workload is small enough
that there is simply no more work to be optimized out by
caching more).
(As a side note, I initially timed my existing git.git pack,
which was a base of --aggressive combined with some pulls on
top. So it had quite a few deeper delta chains. The
256-cache case was more like 15s, and it still dropped to
~6.5s in the same way).
Here are the timings for linux.git:
MAX_DELTA_CACHE log-raw log-S
--------------- --------- ---------
256 0m41.661s 5m12.410s
512 0m39.547s 5m07.920s
1024 0m37.054s 4m54.666s
2048 0m35.871s 4m41.194s*
4096 0m34.646s 4m51.648s
8192 0m33.881s 4m55.342s
16384 0m35.190s 5m00.122s
32768 0m35.060s 4m58.851s
65536 0m33.311s* 4m51.420s
As we grow we see a nice 20% speedup in the tree traversal,
and more modest 10% in the log-S. This is probably an
indication that we are bound less by the number of entries,
and more by the memory limit (more on that below). What is
interesting is that the numbers bounce around a bit;
increasing the number of entries isn't always a strict
improvement.
Partially this is due to noise in the measurement. But it
may also be an indication that our LRU ejection scheme is
not optimal. The smaller cache sizes introduce some
randomness into the ejection (due to collisions), which may
sometimes work in our favor (and sometimes not!).
So what is the optimal setting of MAX_DELTA_CACHE? The
"bouncing" in the linux.git log-S numbers notwithstanding,
it mostly seems like bigger is better. And even if we were
to try to find a "sweet spot", these are just two
repositories, that are not necessarily representative. The
shape of history, the size of trees and blobs, the memory
limit configuration, etc, all will affect the outcome.
Rather than trying to find the "right" number, another
strategy is to just switch to a hash table that can actually
store collisions: namely our hashmap.h implementation.
Here are numbers for that compared to the "best" we saw from
adjusting MAX_DELTA_CACHE:
| log-raw | log-S
| best hashmap | best hashmap
| --------- --------- | --------- ---------
git | 0m02.144s 0m02.144s | 0m06.448s 0m06.688s
linux | 0m33.311s 0m33.092s | 4m41.194s 4m57.172s
We can see the results are similar in most cases, which is
what we'd expect. We're not ejecting due to collisions at
all, so this is purely representing the LRU. So really, we'd
expect this to model most closely the larger values of the
static MAX_DELTA_CACHE limit. And that does seem to be
what's happening, including the "bounce" in the linux log-S
case.
So while the value for that case _isn't_ as good as the
optimal one measured above (which was 2048 entries), given
the bouncing I'm hesitant to suggest that 2048 is any kind
of optimum (not even for linux.git, let alone as a general
rule). The generic hashmap has the appeal that it drops the
number of tweakable numbers by one, which means we can focus
on tuning other elements, like the LRU strategy or the
core.deltaBaseCacheLimit setting.
And indeed, if we bump the cache limit to 1G (which is
probably silly for general use, but maybe something people
with big workstations would want to do), the linux.git log-S
time drops to 3m32s. That's something you really _can't_ do
easily with the static hash table, because the number of
entries needs to grow in proportion to the memory limit (so
2048 is almost certainly not going to be the right value
there).
This patch takes that direction, and drops the static hash
table entirely in favor of using the hashmap.h API.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-22 22:00:07 +00:00
|
|
|
if (ent) {
|
2013-09-13 11:03:00 +00:00
|
|
|
type = ent->type;
|
|
|
|
data = ent->data;
|
|
|
|
size = ent->size;
|
clear_delta_base_cache_entry: use a more descriptive name
The delta base cache entries are stored in a fixed-length
hash table. So the way to remove an entry is to "clear" the
slot in the table, and that is what this function does.
However, the name is a leaky abstraction. If we were to
change the hash table implementation, it would no longer be
about "clearing". We should name it after _what_ it does,
not _how_ it does it. I.e., something like "remove" instead
of "clear".
But that does not tell the whole story, either. The subtle
thing about this function is that it removes the entry, but
does not free the entry data. So a more descriptive name is
"detach"; we give ownership of the data buffer to the
caller, and remove any other resources.
This patch uses the name detach_delta_base_cache_entry().
We could further model this after functions like
strbuf_detach(), which pass back all of the detached
information. However, since there are so many bits of
information in the struct (the data, the size, the type),
and so few callers (only one), it's not worth that
awkwardness. The name change and a comment can make the
intent clear.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-22 21:57:53 +00:00
|
|
|
detach_delta_base_cache_entry(ent);
|
2013-09-13 11:03:00 +00:00
|
|
|
base_from_cache = 1;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
2013-03-27 20:03:42 +00:00
|
|
|
if (do_check_packed_object_crc && p->index_version > 1) {
|
|
|
|
struct revindex_entry *revidx = find_pack_revindex(p, obj_offset);
|
2016-07-05 17:05:54 +00:00
|
|
|
off_t len = revidx[1].offset - obj_offset;
|
2013-03-27 20:03:42 +00:00
|
|
|
if (check_pack_crc(p, &w_curs, obj_offset, len, revidx->nr)) {
|
|
|
|
const unsigned char *sha1 =
|
|
|
|
nth_packed_object_sha1(p, revidx->nr);
|
|
|
|
error("bad packed object CRC for %s",
|
|
|
|
sha1_to_hex(sha1));
|
|
|
|
mark_bad_packed_object(p, sha1);
|
|
|
|
unuse_pack(&w_curs);
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
type = unpack_object_header(p, &w_curs, &curpos, &size);
|
|
|
|
if (type != OBJ_OFS_DELTA && type != OBJ_REF_DELTA)
|
|
|
|
break;
|
|
|
|
|
|
|
|
base_offset = get_delta_base(p, &w_curs, &curpos, type, obj_offset);
|
|
|
|
if (!base_offset) {
|
|
|
|
error("failed to validate delta base reference "
|
|
|
|
"at offset %"PRIuMAX" from %s",
|
|
|
|
(uintmax_t)curpos, p->pack_name);
|
|
|
|
/* bail to phase 2, in hopes of recovery */
|
|
|
|
data = NULL;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* push object, proceed to base */
|
|
|
|
if (delta_stack_nr >= delta_stack_alloc
|
|
|
|
&& delta_stack == small_delta_stack) {
|
|
|
|
delta_stack_alloc = alloc_nr(delta_stack_nr);
|
2016-02-22 22:44:25 +00:00
|
|
|
ALLOC_ARRAY(delta_stack, delta_stack_alloc);
|
2013-03-27 20:03:42 +00:00
|
|
|
memcpy(delta_stack, small_delta_stack,
|
|
|
|
sizeof(*delta_stack)*delta_stack_nr);
|
|
|
|
} else {
|
|
|
|
ALLOC_GROW(delta_stack, delta_stack_nr+1, delta_stack_alloc);
|
|
|
|
}
|
|
|
|
i = delta_stack_nr++;
|
|
|
|
delta_stack[i].obj_offset = obj_offset;
|
|
|
|
delta_stack[i].curpos = curpos;
|
|
|
|
delta_stack[i].size = size;
|
|
|
|
|
|
|
|
curpos = obj_offset = base_offset;
|
close another possibility for propagating pack corruption
Abstract
--------
With index v2 we have a per object CRC to allow quick and safe reuse of
pack data when repacking. This, however, doesn't currently prevent a
stealth corruption from being propagated into a new pack when _not_
reusing pack data as demonstrated by the modification to t5302 included
here.
The Context
-----------
The Git database is all checksummed with SHA1 hashes. Any kind of
corruption can be confirmed by verifying this per object hash against
corresponding data. However this can be costly to perform systematically
and therefore this check is often not performed at run time when
accessing the object database.
First, the loose object format is entirely compressed with zlib which
already provide a CRC verification of its own when inflating data. Any
disk corruption would be caught already in this case.
Then, packed objects are also compressed with zlib but only for their
actual payload. The object headers and delta base references are not
deflated for obvious performance reasons, however this leave them
vulnerable to potentially undetected disk corruptions. Object types
are often validated against the expected type when they're requested,
and deflated size must always match the size recorded in the object header,
so those cases are pretty much covered as well.
Where corruptions could go unnoticed is in the delta base reference.
Of course, in the OBJ_REF_DELTA case, the odds for a SHA1 reference to
get corrupted so it actually matches the SHA1 of another object with the
same size (the delta header stores the expected size of the base object
to apply against) are virtually zero. In the OBJ_OFS_DELTA case, the
reference is a pack offset which would have to match the start boundary
of a different base object but still with the same size, and although this
is relatively much more "probable" than in the OBJ_REF_DELTA case, the
probability is also about zero in absolute terms. Still, the possibility
exists as demonstrated in t5302 and is certainly greater than a SHA1
collision, especially in the OBJ_OFS_DELTA case which is now the default
when repacking.
Again, repacking by reusing existing pack data is OK since the per object
CRC provided by index v2 guards against any such corruptions. What t5302
failed to test is a full repack in such case.
The Solution
------------
As unlikely as this kind of stealth corruption can be in practice, it
certainly isn't acceptable to propagate it into a freshly created pack.
But, because this is so unlikely, we don't want to pay the run time cost
associated with extra validation checks all the time either. Furthermore,
consequences of such corruption in anything but repacking should be rather
visible, and even if it could be quite unpleasant, it still has far less
severe consequences than actively creating bad packs.
So the best compromize is to check packed object CRC when unpacking
objects, and only during the compression/writing phase of a repack, and
only when not streaming the result. The cost of this is minimal (less
than 1% CPU time), and visible only with a full repack.
Someone with a stats background could provide an objective evaluation of
this, but I suspect that it's bad RAM that has more potential for data
corruptions at this point, even in those cases where this extra check
is not performed. Still, it is best to prevent a known hole for
corruption when recreating object data into a new pack.
What about the streamed pack case? Well, any client receiving a pack
must always consider that pack as untrusty and perform full validation
anyway, hence no such stealth corruption could be propagated to remote
repositoryes already. It is therefore worthless doing local validation
in that case.
Signed-off-by: Nicolas Pitre <nico@cam.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-10-31 15:31:08 +00:00
|
|
|
}
|
|
|
|
|
2013-03-27 20:03:42 +00:00
|
|
|
/* PHASE 2: handle the base */
|
|
|
|
switch (type) {
|
2006-09-21 04:06:49 +00:00
|
|
|
case OBJ_OFS_DELTA:
|
|
|
|
case OBJ_REF_DELTA:
|
2013-03-27 20:03:42 +00:00
|
|
|
if (data)
|
2016-07-26 16:05:50 +00:00
|
|
|
die("BUG: unpack_entry: left loop at a valid delta");
|
2006-12-23 07:33:25 +00:00
|
|
|
break;
|
2005-06-28 21:21:02 +00:00
|
|
|
case OBJ_COMMIT:
|
|
|
|
case OBJ_TREE:
|
|
|
|
case OBJ_BLOB:
|
|
|
|
case OBJ_TAG:
|
2013-03-27 20:03:42 +00:00
|
|
|
if (!base_from_cache)
|
|
|
|
data = unpack_compressed_entry(p, &w_curs, curpos, size);
|
2006-12-23 07:33:25 +00:00
|
|
|
break;
|
2005-06-27 10:35:33 +00:00
|
|
|
default:
|
2008-06-24 01:23:39 +00:00
|
|
|
data = NULL;
|
|
|
|
error("unknown object type %i at offset %"PRIuMAX" in %s",
|
2013-03-27 20:03:42 +00:00
|
|
|
type, (uintmax_t)obj_offset, p->pack_name);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* PHASE 3: apply deltas in order */
|
|
|
|
|
|
|
|
/* invariants:
|
|
|
|
* 'data' holds the base data, or NULL if there was corruption
|
|
|
|
*/
|
|
|
|
while (delta_stack_nr) {
|
|
|
|
void *delta_data;
|
|
|
|
void *base = data;
|
2017-02-25 10:02:28 +00:00
|
|
|
void *external_base = NULL;
|
2013-03-27 20:03:42 +00:00
|
|
|
unsigned long delta_size, base_size = size;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
data = NULL;
|
|
|
|
|
|
|
|
if (base)
|
|
|
|
add_delta_base_cache(p, obj_offset, base, base_size, type);
|
|
|
|
|
|
|
|
if (!base) {
|
|
|
|
/*
|
|
|
|
* We're probably in deep shit, but let's try to fetch
|
|
|
|
* the required base anyway from another pack or loose.
|
|
|
|
* This is costly but should happen only in the presence
|
|
|
|
* of a corrupted pack, and is better than failing outright.
|
|
|
|
*/
|
|
|
|
struct revindex_entry *revidx;
|
|
|
|
const unsigned char *base_sha1;
|
|
|
|
revidx = find_pack_revindex(p, obj_offset);
|
|
|
|
if (revidx) {
|
|
|
|
base_sha1 = nth_packed_object_sha1(p, revidx->nr);
|
|
|
|
error("failed to read delta base object %s"
|
|
|
|
" at offset %"PRIuMAX" from %s",
|
|
|
|
sha1_to_hex(base_sha1), (uintmax_t)obj_offset,
|
|
|
|
p->pack_name);
|
|
|
|
mark_bad_packed_object(p, base_sha1);
|
|
|
|
base = read_object(base_sha1, &type, &base_size);
|
2017-02-25 10:02:28 +00:00
|
|
|
external_base = base;
|
2013-03-27 20:03:42 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
i = --delta_stack_nr;
|
|
|
|
obj_offset = delta_stack[i].obj_offset;
|
|
|
|
curpos = delta_stack[i].curpos;
|
|
|
|
delta_size = delta_stack[i].size;
|
|
|
|
|
|
|
|
if (!base)
|
|
|
|
continue;
|
|
|
|
|
|
|
|
delta_data = unpack_compressed_entry(p, &w_curs, curpos, delta_size);
|
|
|
|
|
|
|
|
if (!delta_data) {
|
|
|
|
error("failed to unpack compressed delta "
|
|
|
|
"at offset %"PRIuMAX" from %s",
|
|
|
|
(uintmax_t)curpos, p->pack_name);
|
|
|
|
data = NULL;
|
2017-02-25 10:02:28 +00:00
|
|
|
free(external_base);
|
2013-03-27 20:03:42 +00:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
data = patch_delta(base, base_size,
|
|
|
|
delta_data, delta_size,
|
|
|
|
&size);
|
unpack_entry: do not die when we fail to apply a delta
When we try to load an object from disk and fail, our
general strategy is to see if we can get it from somewhere
else (e.g., a loose object). That lets users fix corruption
problems by copying known-good versions of objects into the
object database.
We already handle the case where we were not able to read
the delta from disk. However, when we find that the delta we
read does not apply, we simply die. This case is harder to
trigger, as corruption in the delta data itself would
trigger a crc error from zlib. However, a corruption that
pointed us at the wrong delta base might cause it.
We can do the same "fail and try to find the object
elsewhere" trick instead of dying. This not only gives us a
chance to recover, but also puts us on code paths that will
alert the user to the problem (with the current message,
they do not even know which sha1 caused the problem).
Note that unlike some other pack corruptions, we do not
recover automatically from this case when doing a repack.
There is nothing apparently wrong with the delta, as it
points to a valid, accessible object, and we realize the
error only when the resulting size does not match up. And in
theory, one could even have a case where the corrupted size
is the same, and the problem would only be noticed by
recomputing the sha1.
We can get around this by recomputing the deltas with
--no-reuse-delta, which our test does (and this is probably
good advice for anyone recovering from pack corruption).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-06-14 21:53:34 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* We could not apply the delta; warn the user, but keep going.
|
|
|
|
* Our failure will be noticed either in the next iteration of
|
|
|
|
* the loop, or if this is the final delta, in the caller when
|
|
|
|
* we return NULL. Those code paths will take care of making
|
|
|
|
* a more explicit warning and retrying with another copy of
|
|
|
|
* the object.
|
|
|
|
*/
|
2013-03-27 20:03:42 +00:00
|
|
|
if (!data)
|
unpack_entry: do not die when we fail to apply a delta
When we try to load an object from disk and fail, our
general strategy is to see if we can get it from somewhere
else (e.g., a loose object). That lets users fix corruption
problems by copying known-good versions of objects into the
object database.
We already handle the case where we were not able to read
the delta from disk. However, when we find that the delta we
read does not apply, we simply die. This case is harder to
trigger, as corruption in the delta data itself would
trigger a crc error from zlib. However, a corruption that
pointed us at the wrong delta base might cause it.
We can do the same "fail and try to find the object
elsewhere" trick instead of dying. This not only gives us a
chance to recover, but also puts us on code paths that will
alert the user to the problem (with the current message,
they do not even know which sha1 caused the problem).
Note that unlike some other pack corruptions, we do not
recover automatically from this case when doing a repack.
There is nothing apparently wrong with the delta, as it
points to a valid, accessible object, and we realize the
error only when the resulting size does not match up. And in
theory, one could even have a case where the corrupted size
is the same, and the problem would only be noticed by
recomputing the sha1.
We can get around this by recomputing the deltas with
--no-reuse-delta, which our test does (and this is probably
good advice for anyone recovering from pack corruption).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-06-14 21:53:34 +00:00
|
|
|
error("failed to apply delta");
|
2013-03-27 20:03:42 +00:00
|
|
|
|
2013-05-30 13:56:21 +00:00
|
|
|
free(delta_data);
|
2017-02-25 10:02:28 +00:00
|
|
|
free(external_base);
|
2005-06-27 10:35:33 +00:00
|
|
|
}
|
2013-03-27 20:03:42 +00:00
|
|
|
|
|
|
|
*final_type = type;
|
|
|
|
*final_size = size;
|
|
|
|
|
2006-12-23 07:34:08 +00:00
|
|
|
unuse_pack(&w_curs);
|
2014-02-20 23:47:47 +00:00
|
|
|
|
|
|
|
if (delta_stack != small_delta_stack)
|
|
|
|
free(delta_stack);
|
|
|
|
|
2007-02-26 19:55:59 +00:00
|
|
|
return data;
|
2005-06-27 10:35:33 +00:00
|
|
|
}
|
|
|
|
|
2007-05-26 05:24:19 +00:00
|
|
|
const unsigned char *nth_packed_object_sha1(struct packed_git *p,
|
2007-04-04 20:49:04 +00:00
|
|
|
uint32_t n)
|
2005-06-28 21:56:57 +00:00
|
|
|
{
|
2007-03-16 20:42:50 +00:00
|
|
|
const unsigned char *index = p->index_data;
|
2007-05-26 05:24:19 +00:00
|
|
|
if (!index) {
|
|
|
|
if (open_pack_index(p))
|
|
|
|
return NULL;
|
|
|
|
index = p->index_data;
|
|
|
|
}
|
2007-04-09 05:06:28 +00:00
|
|
|
if (n >= p->num_objects)
|
2007-04-04 20:49:04 +00:00
|
|
|
return NULL;
|
2007-04-09 05:06:35 +00:00
|
|
|
index += 4 * 256;
|
|
|
|
if (p->index_version == 1) {
|
|
|
|
return index + 24 * n + 4;
|
|
|
|
} else {
|
|
|
|
index += 8;
|
|
|
|
return index + 20 * n;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-02-21 23:47:34 +00:00
|
|
|
const struct object_id *nth_packed_object_oid(struct object_id *oid,
|
|
|
|
struct packed_git *p,
|
|
|
|
uint32_t n)
|
|
|
|
{
|
|
|
|
const unsigned char *hash = nth_packed_object_sha1(p, n);
|
|
|
|
if (!hash)
|
|
|
|
return NULL;
|
|
|
|
hashcpy(oid->hash, hash);
|
|
|
|
return oid;
|
|
|
|
}
|
|
|
|
|
2016-02-25 14:22:52 +00:00
|
|
|
void check_pack_index_ptr(const struct packed_git *p, const void *vptr)
|
|
|
|
{
|
|
|
|
const unsigned char *ptr = vptr;
|
|
|
|
const unsigned char *start = p->index_data;
|
|
|
|
const unsigned char *end = start + p->index_size;
|
|
|
|
if (ptr < start)
|
2016-02-27 07:49:33 +00:00
|
|
|
die(_("offset before start of pack index for %s (corrupt index?)"),
|
2016-02-25 14:22:52 +00:00
|
|
|
p->pack_name);
|
|
|
|
/* No need to check for underflow; .idx files must be at least 8 bytes */
|
|
|
|
if (ptr >= end - 8)
|
2016-02-27 07:49:33 +00:00
|
|
|
die(_("offset beyond end of pack index for %s (truncated index?)"),
|
2016-02-25 14:22:52 +00:00
|
|
|
p->pack_name);
|
|
|
|
}
|
|
|
|
|
2008-06-25 03:17:12 +00:00
|
|
|
off_t nth_packed_object_offset(const struct packed_git *p, uint32_t n)
|
2007-04-09 05:06:35 +00:00
|
|
|
{
|
|
|
|
const unsigned char *index = p->index_data;
|
|
|
|
index += 4 * 256;
|
|
|
|
if (p->index_version == 1) {
|
|
|
|
return ntohl(*((uint32_t *)(index + 24 * n)));
|
|
|
|
} else {
|
|
|
|
uint32_t off;
|
|
|
|
index += 8 + p->num_objects * (20 + 4);
|
|
|
|
off = ntohl(*((uint32_t *)(index + 4 * n)));
|
|
|
|
if (!(off & 0x80000000))
|
|
|
|
return off;
|
|
|
|
index += p->num_objects * 4 + (off & 0x7fffffff) * 8;
|
2016-02-25 14:22:52 +00:00
|
|
|
check_pack_index_ptr(p, index);
|
2007-04-09 05:06:35 +00:00
|
|
|
return (((uint64_t)ntohl(*((uint32_t *)(index + 0)))) << 32) |
|
|
|
|
ntohl(*((uint32_t *)(index + 4)));
|
|
|
|
}
|
2005-06-28 21:56:57 +00:00
|
|
|
}
|
|
|
|
|
2007-03-07 01:44:30 +00:00
|
|
|
off_t find_pack_entry_one(const unsigned char *sha1,
|
2006-09-21 04:05:37 +00:00
|
|
|
struct packed_git *p)
|
2005-06-27 10:35:33 +00:00
|
|
|
{
|
2007-03-16 20:42:50 +00:00
|
|
|
const uint32_t *level1_ofs = p->index_data;
|
|
|
|
const unsigned char *index = p->index_data;
|
sha1-lookup: more memory efficient search in sorted list of SHA-1
Currently, when looking for a packed object from the pack idx, a
simple binary search is used.
A conventional binary search loop looks like this:
unsigned lo, hi;
do {
unsigned mi = (lo + hi) / 2;
int cmp = "entry pointed at by mi" minus "target";
if (!cmp)
return mi; "mi is the wanted one"
if (cmp > 0)
hi = mi; "mi is larger than target"
else
lo = mi+1; "mi is smaller than target"
} while (lo < hi);
"did not find what we wanted"
The invariants are:
- When entering the loop, 'lo' points at a slot that is never
above the target (it could be at the target), 'hi' points at
a slot that is guaranteed to be above the target (it can
never be at the target).
- We find a point 'mi' between 'lo' and 'hi' ('mi' could be
the same as 'lo', but never can be as high as 'hi'), and
check if 'mi' hits the target. There are three cases:
- if it is a hit, we have found what we are looking for;
- if it is strictly higher than the target, we set it to
'hi', and repeat the search.
- if it is strictly lower than the target, we update 'lo'
to one slot after it, because we allow 'lo' to be at the
target and 'mi' is known to be below the target.
If the loop exits, there is no matching entry.
When choosing 'mi', we do not have to take the "middle" but
anywhere in between 'lo' and 'hi', as long as lo <= mi < hi is
satisfied. When we somehow know that the distance between the
target and 'lo' is much shorter than the target and 'hi', we
could pick 'mi' that is much closer to 'lo' than (hi+lo)/2,
which a conventional binary search would pick.
This patch takes advantage of the fact that the SHA-1 is a good
hash function, and as long as there are enough entries in the
table, we can expect uniform distribution. An entry that begins
with for example "deadbeef..." is much likely to appear much
later than in the midway of a reasonably populated table. In
fact, it can be expected to be near 87% (222/256) from the top
of the table.
This is a work-in-progress and has switches to allow easier
experiments and debugging. Exporting GIT_USE_LOOKUP environment
variable enables this code.
On my admittedly memory starved machine, with a partial KDE
repository (3.0G pack with 95M idx):
$ GIT_USE_LOOKUP=t git log -800 --stat HEAD >/dev/null
3.93user 0.16system 0:04.09elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+55588minor)pagefaults 0swaps
Without the patch, the numbers are:
$ git log -800 --stat HEAD >/dev/null
4.00user 0.15system 0:04.17elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+60258minor)pagefaults 0swaps
In the same repository:
$ GIT_USE_LOOKUP=t git log -2000 HEAD >/dev/null
0.12user 0.00system 0:00.12elapsed 97%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+4241minor)pagefaults 0swaps
Without the patch, the numbers are:
$ git log -2000 HEAD >/dev/null
0.05user 0.01system 0:00.07elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+8506minor)pagefaults 0swaps
There isn't much time difference, but the number of minor faults
seems to show that we are touching much smaller number of pages,
which is expected.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-12-29 10:05:47 +00:00
|
|
|
unsigned hi, lo, stride;
|
|
|
|
static int use_lookup = -1;
|
|
|
|
static int debug_lookup = -1;
|
|
|
|
|
|
|
|
if (debug_lookup < 0)
|
|
|
|
debug_lookup = !!getenv("GIT_DEBUG_LOOKUP");
|
2007-03-16 20:42:50 +00:00
|
|
|
|
2007-05-26 05:24:19 +00:00
|
|
|
if (!index) {
|
|
|
|
if (open_pack_index(p))
|
|
|
|
return 0;
|
|
|
|
level1_ofs = p->index_data;
|
|
|
|
index = p->index_data;
|
|
|
|
}
|
2007-04-09 05:06:35 +00:00
|
|
|
if (p->index_version > 1) {
|
|
|
|
level1_ofs += 2;
|
|
|
|
index += 8;
|
|
|
|
}
|
2007-03-16 20:42:50 +00:00
|
|
|
index += 4 * 256;
|
2007-04-09 05:06:35 +00:00
|
|
|
hi = ntohl(level1_ofs[*sha1]);
|
|
|
|
lo = ((*sha1 == 0x0) ? 0 : ntohl(level1_ofs[*sha1 - 1]));
|
sha1-lookup: more memory efficient search in sorted list of SHA-1
Currently, when looking for a packed object from the pack idx, a
simple binary search is used.
A conventional binary search loop looks like this:
unsigned lo, hi;
do {
unsigned mi = (lo + hi) / 2;
int cmp = "entry pointed at by mi" minus "target";
if (!cmp)
return mi; "mi is the wanted one"
if (cmp > 0)
hi = mi; "mi is larger than target"
else
lo = mi+1; "mi is smaller than target"
} while (lo < hi);
"did not find what we wanted"
The invariants are:
- When entering the loop, 'lo' points at a slot that is never
above the target (it could be at the target), 'hi' points at
a slot that is guaranteed to be above the target (it can
never be at the target).
- We find a point 'mi' between 'lo' and 'hi' ('mi' could be
the same as 'lo', but never can be as high as 'hi'), and
check if 'mi' hits the target. There are three cases:
- if it is a hit, we have found what we are looking for;
- if it is strictly higher than the target, we set it to
'hi', and repeat the search.
- if it is strictly lower than the target, we update 'lo'
to one slot after it, because we allow 'lo' to be at the
target and 'mi' is known to be below the target.
If the loop exits, there is no matching entry.
When choosing 'mi', we do not have to take the "middle" but
anywhere in between 'lo' and 'hi', as long as lo <= mi < hi is
satisfied. When we somehow know that the distance between the
target and 'lo' is much shorter than the target and 'hi', we
could pick 'mi' that is much closer to 'lo' than (hi+lo)/2,
which a conventional binary search would pick.
This patch takes advantage of the fact that the SHA-1 is a good
hash function, and as long as there are enough entries in the
table, we can expect uniform distribution. An entry that begins
with for example "deadbeef..." is much likely to appear much
later than in the midway of a reasonably populated table. In
fact, it can be expected to be near 87% (222/256) from the top
of the table.
This is a work-in-progress and has switches to allow easier
experiments and debugging. Exporting GIT_USE_LOOKUP environment
variable enables this code.
On my admittedly memory starved machine, with a partial KDE
repository (3.0G pack with 95M idx):
$ GIT_USE_LOOKUP=t git log -800 --stat HEAD >/dev/null
3.93user 0.16system 0:04.09elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+55588minor)pagefaults 0swaps
Without the patch, the numbers are:
$ git log -800 --stat HEAD >/dev/null
4.00user 0.15system 0:04.17elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+60258minor)pagefaults 0swaps
In the same repository:
$ GIT_USE_LOOKUP=t git log -2000 HEAD >/dev/null
0.12user 0.00system 0:00.12elapsed 97%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+4241minor)pagefaults 0swaps
Without the patch, the numbers are:
$ git log -2000 HEAD >/dev/null
0.05user 0.01system 0:00.07elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+8506minor)pagefaults 0swaps
There isn't much time difference, but the number of minor faults
seems to show that we are touching much smaller number of pages,
which is expected.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-12-29 10:05:47 +00:00
|
|
|
if (p->index_version > 1) {
|
|
|
|
stride = 20;
|
|
|
|
} else {
|
|
|
|
stride = 24;
|
|
|
|
index += 4;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (debug_lookup)
|
2008-07-03 15:52:09 +00:00
|
|
|
printf("%02x%02x%02x... lo %u hi %u nr %"PRIu32"\n",
|
sha1-lookup: more memory efficient search in sorted list of SHA-1
Currently, when looking for a packed object from the pack idx, a
simple binary search is used.
A conventional binary search loop looks like this:
unsigned lo, hi;
do {
unsigned mi = (lo + hi) / 2;
int cmp = "entry pointed at by mi" minus "target";
if (!cmp)
return mi; "mi is the wanted one"
if (cmp > 0)
hi = mi; "mi is larger than target"
else
lo = mi+1; "mi is smaller than target"
} while (lo < hi);
"did not find what we wanted"
The invariants are:
- When entering the loop, 'lo' points at a slot that is never
above the target (it could be at the target), 'hi' points at
a slot that is guaranteed to be above the target (it can
never be at the target).
- We find a point 'mi' between 'lo' and 'hi' ('mi' could be
the same as 'lo', but never can be as high as 'hi'), and
check if 'mi' hits the target. There are three cases:
- if it is a hit, we have found what we are looking for;
- if it is strictly higher than the target, we set it to
'hi', and repeat the search.
- if it is strictly lower than the target, we update 'lo'
to one slot after it, because we allow 'lo' to be at the
target and 'mi' is known to be below the target.
If the loop exits, there is no matching entry.
When choosing 'mi', we do not have to take the "middle" but
anywhere in between 'lo' and 'hi', as long as lo <= mi < hi is
satisfied. When we somehow know that the distance between the
target and 'lo' is much shorter than the target and 'hi', we
could pick 'mi' that is much closer to 'lo' than (hi+lo)/2,
which a conventional binary search would pick.
This patch takes advantage of the fact that the SHA-1 is a good
hash function, and as long as there are enough entries in the
table, we can expect uniform distribution. An entry that begins
with for example "deadbeef..." is much likely to appear much
later than in the midway of a reasonably populated table. In
fact, it can be expected to be near 87% (222/256) from the top
of the table.
This is a work-in-progress and has switches to allow easier
experiments and debugging. Exporting GIT_USE_LOOKUP environment
variable enables this code.
On my admittedly memory starved machine, with a partial KDE
repository (3.0G pack with 95M idx):
$ GIT_USE_LOOKUP=t git log -800 --stat HEAD >/dev/null
3.93user 0.16system 0:04.09elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+55588minor)pagefaults 0swaps
Without the patch, the numbers are:
$ git log -800 --stat HEAD >/dev/null
4.00user 0.15system 0:04.17elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+60258minor)pagefaults 0swaps
In the same repository:
$ GIT_USE_LOOKUP=t git log -2000 HEAD >/dev/null
0.12user 0.00system 0:00.12elapsed 97%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+4241minor)pagefaults 0swaps
Without the patch, the numbers are:
$ git log -2000 HEAD >/dev/null
0.05user 0.01system 0:00.07elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+8506minor)pagefaults 0swaps
There isn't much time difference, but the number of minor faults
seems to show that we are touching much smaller number of pages,
which is expected.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-12-29 10:05:47 +00:00
|
|
|
sha1[0], sha1[1], sha1[2], lo, hi, p->num_objects);
|
|
|
|
|
|
|
|
if (use_lookup < 0)
|
|
|
|
use_lookup = !!getenv("GIT_USE_LOOKUP");
|
|
|
|
if (use_lookup) {
|
|
|
|
int pos = sha1_entry_pos(index, stride, 0,
|
|
|
|
lo, hi, p->num_objects, sha1);
|
|
|
|
if (pos < 0)
|
|
|
|
return 0;
|
|
|
|
return nth_packed_object_offset(p, pos);
|
|
|
|
}
|
2005-06-27 10:35:33 +00:00
|
|
|
|
|
|
|
do {
|
2007-04-09 05:06:35 +00:00
|
|
|
unsigned mi = (lo + hi) / 2;
|
sha1-lookup: more memory efficient search in sorted list of SHA-1
Currently, when looking for a packed object from the pack idx, a
simple binary search is used.
A conventional binary search loop looks like this:
unsigned lo, hi;
do {
unsigned mi = (lo + hi) / 2;
int cmp = "entry pointed at by mi" minus "target";
if (!cmp)
return mi; "mi is the wanted one"
if (cmp > 0)
hi = mi; "mi is larger than target"
else
lo = mi+1; "mi is smaller than target"
} while (lo < hi);
"did not find what we wanted"
The invariants are:
- When entering the loop, 'lo' points at a slot that is never
above the target (it could be at the target), 'hi' points at
a slot that is guaranteed to be above the target (it can
never be at the target).
- We find a point 'mi' between 'lo' and 'hi' ('mi' could be
the same as 'lo', but never can be as high as 'hi'), and
check if 'mi' hits the target. There are three cases:
- if it is a hit, we have found what we are looking for;
- if it is strictly higher than the target, we set it to
'hi', and repeat the search.
- if it is strictly lower than the target, we update 'lo'
to one slot after it, because we allow 'lo' to be at the
target and 'mi' is known to be below the target.
If the loop exits, there is no matching entry.
When choosing 'mi', we do not have to take the "middle" but
anywhere in between 'lo' and 'hi', as long as lo <= mi < hi is
satisfied. When we somehow know that the distance between the
target and 'lo' is much shorter than the target and 'hi', we
could pick 'mi' that is much closer to 'lo' than (hi+lo)/2,
which a conventional binary search would pick.
This patch takes advantage of the fact that the SHA-1 is a good
hash function, and as long as there are enough entries in the
table, we can expect uniform distribution. An entry that begins
with for example "deadbeef..." is much likely to appear much
later than in the midway of a reasonably populated table. In
fact, it can be expected to be near 87% (222/256) from the top
of the table.
This is a work-in-progress and has switches to allow easier
experiments and debugging. Exporting GIT_USE_LOOKUP environment
variable enables this code.
On my admittedly memory starved machine, with a partial KDE
repository (3.0G pack with 95M idx):
$ GIT_USE_LOOKUP=t git log -800 --stat HEAD >/dev/null
3.93user 0.16system 0:04.09elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+55588minor)pagefaults 0swaps
Without the patch, the numbers are:
$ git log -800 --stat HEAD >/dev/null
4.00user 0.15system 0:04.17elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+60258minor)pagefaults 0swaps
In the same repository:
$ GIT_USE_LOOKUP=t git log -2000 HEAD >/dev/null
0.12user 0.00system 0:00.12elapsed 97%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+4241minor)pagefaults 0swaps
Without the patch, the numbers are:
$ git log -2000 HEAD >/dev/null
0.05user 0.01system 0:00.07elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+8506minor)pagefaults 0swaps
There isn't much time difference, but the number of minor faults
seems to show that we are touching much smaller number of pages,
which is expected.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2007-12-29 10:05:47 +00:00
|
|
|
int cmp = hashcmp(index + mi * stride, sha1);
|
|
|
|
|
|
|
|
if (debug_lookup)
|
|
|
|
printf("lo %u hi %u rg %u mi %u\n",
|
|
|
|
lo, hi, hi - lo, mi);
|
2006-09-21 04:05:37 +00:00
|
|
|
if (!cmp)
|
2007-04-09 05:06:35 +00:00
|
|
|
return nth_packed_object_offset(p, mi);
|
2005-06-27 10:35:33 +00:00
|
|
|
if (cmp > 0)
|
|
|
|
hi = mi;
|
|
|
|
else
|
|
|
|
lo = mi+1;
|
|
|
|
} while (lo < hi);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
pack-objects: protect against disappearing packs
It's possible that while pack-objects is running, a
simultaneously running prune process might delete a pack
that we are interested in. Because we load the pack indices
early on, we know that the pack contains our item, but by
the time we try to open and map it, it is gone.
Since c715f78, we already protect against this in the normal
object access code path, but pack-objects accesses the packs
at a lower level. In the normal access path, we call
find_pack_entry, which will call find_pack_entry_one on each
pack index, which does the actual lookup. If it gets a hit,
we will actually open and verify the validity of the
matching packfile (using c715f78's is_pack_valid). If we
can't open it, we'll issue a warning and pretend that we
didn't find it, causing us to go on to the next pack (or on
to loose objects).
Furthermore, we will cache the descriptor to the opened
packfile. Which means that later, when we actually try to
access the object, we are likely to still have that packfile
opened, and won't care if it has been unlinked from the
filesystem.
Notice the "likely" above. If there is another pack access
in the interim, and we run out of descriptors, we could
close the pack. And then a later attempt to access the
closed pack could fail (we'll try to re-open it, of course,
but it may have been deleted). In practice, this doesn't
happen because we tend to look up items and then access them
immediately.
Pack-objects does not follow this code path. Instead, it
accesses the packs at a much lower level, using
find_pack_entry_one directly. This means we skip the
is_pack_valid check, and may end up with the name of a
packfile, but no open descriptor.
We can add the same is_pack_valid check here. Unfortunately,
the access patterns of pack-objects are not quite as nice
for keeping lookup and object access together. We look up
each object as we find out about it, and the only later when
writing the packfile do we necessarily access it. Which
means that the opened packfile may be closed in the interim.
In practice, however, adding this check still has value, for
three reasons.
1. If you have a reasonable number of packs and/or a
reasonable file descriptor limit, you can keep all of
your packs open simultaneously. If this is the case,
then the race is impossible to trigger.
2. Even if you can't keep all packs open at once, you
may end up keeping the deleted one open (i.e., you may
get lucky).
3. The race window is shortened. You may notice early that
the pack is gone, and not try to access it. Triggering
the problem without this check means deleting the pack
any time after we read the list of index files, but
before we access the looked-up objects. Triggering it
with this check means deleting the pack means deleting
the pack after we do a lookup (and successfully access
the packfile), but before we access the object. Which
is a smaller window.
Acked-by: Nicolas Pitre <nico@fluxnic.net>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-10-14 18:03:48 +00:00
|
|
|
int is_pack_valid(struct packed_git *p)
|
2011-03-02 18:01:54 +00:00
|
|
|
{
|
|
|
|
/* An already open pack is known to be valid. */
|
|
|
|
if (p->pack_fd != -1)
|
|
|
|
return 1;
|
|
|
|
|
|
|
|
/* If the pack has one window completely covering the
|
|
|
|
* file size, the pack is known to be valid even if
|
|
|
|
* the descriptor is not currently open.
|
|
|
|
*/
|
|
|
|
if (p->windows) {
|
|
|
|
struct pack_window *w = p->windows;
|
|
|
|
|
|
|
|
if (!w->offset && w->len == p->pack_size)
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Force the pack to open to prove its valid. */
|
|
|
|
return !open_packed_git(p);
|
|
|
|
}
|
|
|
|
|
2012-02-01 13:48:54 +00:00
|
|
|
static int fill_pack_entry(const unsigned char *sha1,
|
|
|
|
struct pack_entry *e,
|
|
|
|
struct packed_git *p)
|
|
|
|
{
|
|
|
|
off_t offset;
|
|
|
|
|
|
|
|
if (p->num_bad_objects) {
|
|
|
|
unsigned i;
|
|
|
|
for (i = 0; i < p->num_bad_objects; i++)
|
|
|
|
if (!hashcmp(sha1, p->bad_object_sha1 + 20 * i))
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
offset = find_pack_entry_one(sha1, p);
|
|
|
|
if (!offset)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We are about to tell the caller where they can locate the
|
|
|
|
* requested object. We better make sure the packfile is
|
|
|
|
* still here and can be accessed before supplying that
|
|
|
|
* answer, as it may have been deleted since the index was
|
|
|
|
* loaded!
|
|
|
|
*/
|
sha1_file: squelch "packfile cannot be accessed" warnings
When we find an object in a packfile index, we make sure we
can still open the packfile itself (or that it is already
open), as it might have been deleted by a simultaneous
repack. If we can't access the packfile, we print a warning
for the user and tell the caller that we don't have the
object (we can then look in other packfiles, or find a loose
version, before giving up).
The warning we print to the user isn't really accomplishing
anything, and it is potentially confusing to users. In the
normal case, it is complete noise; we find the object
elsewhere, and the user does not have to care that we racily
saw a packfile index that became stale. It didn't affect the
operation at all.
A possibly more interesting case is when we later can't find
the object, and report failure to the user. In this case the
warning could be considered a clue toward that ultimate
failure. But it's not really a useful clue in practice. We
wouldn't even print it consistently (since we are racing
with another process, we might not even see the .idx file,
or we might win the race and open the packfile, completing
the operation).
This patch drops the warning entirely (not only from the
fill_pack_entry site, but also from an identical use in
pack-objects). If we did find the warning interesting in the
error case, we could stuff it away and reveal it to the user
when we later die() due to the broken object. But that
complexity just isn't worth it.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-03-31 00:47:38 +00:00
|
|
|
if (!is_pack_valid(p))
|
2012-02-01 13:48:54 +00:00
|
|
|
return 0;
|
|
|
|
e->offset = offset;
|
|
|
|
e->p = p;
|
|
|
|
hashcpy(e->sha1, sha1);
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
2014-02-21 16:32:06 +00:00
|
|
|
/*
|
|
|
|
* Iff a pack file contains the object named by sha1, return true and
|
|
|
|
* store its location to e.
|
|
|
|
*/
|
2009-03-20 03:47:54 +00:00
|
|
|
static int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)
|
2005-06-27 10:35:33 +00:00
|
|
|
{
|
find_pack_entry: replace last_found_pack with MRU cache
Each pack has an index for looking up entries in O(log n)
time, but if we have multiple packs, we have to scan through
them linearly. This can produce a measurable overhead for
some operations.
We dealt with this long ago in f7c22cc (always start looking
up objects in the last used pack first, 2007-05-30), which
keeps what is essentially a 1-element most-recently-used
cache. In theory, we should be able to do better by keeping
a similar but longer cache, that is the same length as the
pack-list itself.
Since we now have a convenient generic MRU structure, we can
plug it in and measure. Here are the numbers for running
p5303 against linux.git:
Test HEAD^ HEAD
------------------------------------------------------------------------
5303.3: rev-list (1) 31.56(31.28+0.27) 31.30(31.08+0.20) -0.8%
5303.4: repack (1) 40.62(39.35+2.36) 40.60(39.27+2.44) -0.0%
5303.6: rev-list (50) 31.31(31.06+0.23) 31.23(31.00+0.22) -0.3%
5303.7: repack (50) 58.65(69.12+1.94) 58.27(68.64+2.05) -0.6%
5303.9: rev-list (1000) 38.74(38.40+0.33) 31.87(31.62+0.24) -17.7%
5303.10: repack (1000) 367.20(441.80+4.62) 342.00(414.04+3.72) -6.9%
The main numbers of interest here are the rev-list ones
(since that is exercising the normal object lookup code
path). The single-pack case shouldn't improve at all; the
260ms speedup there is just part of the run-to-run noise
(but it's important to note that we didn't make anything
worse with the overhead of maintaining our cache). In the
50-pack case, we see similar results. There may be a slight
improvement, but it's mostly within the noise.
The 1000-pack case does show a big improvement, though. That
carries over to the repack case, as well. Even though we
haven't touched its pack-search loop yet, it does still do a
lot of normal object lookups (e.g., for the internal
revision walk), and so improves.
As a point of reference, I also ran the 1000-pack test
against a version of HEAD^ with the last_found_pack
optimization disabled. It takes ~60s, so that gives an
indication of how much even the single-element cache is
helping.
For comparison, here's a smaller repository, git.git:
Test HEAD^ HEAD
---------------------------------------------------------------------
5303.3: rev-list (1) 1.56(1.54+0.01) 1.54(1.51+0.02) -1.3%
5303.4: repack (1) 1.84(1.80+0.10) 1.82(1.80+0.09) -1.1%
5303.6: rev-list (50) 1.58(1.55+0.02) 1.59(1.57+0.01) +0.6%
5303.7: repack (50) 2.50(3.18+0.04) 2.50(3.14+0.04) +0.0%
5303.9: rev-list (1000) 2.76(2.71+0.04) 2.24(2.21+0.02) -18.8%
5303.10: repack (1000) 13.21(19.56+0.25) 11.66(18.01+0.21) -11.7%
You can see that the percentage improvement is similar.
That's because the lookup we are optimizing is roughly
O(nr_objects * nr_packs). Since the number of packs is
constant in both tests, we'd expect the improvement to be
linear in the number of objects. But the whole process is
also linear in the number of objects, so the improvement
is a constant factor.
The exact improvement does also depend on the contents of
the packs. In p5303, the extra packs all have 5 first-parent
commits in them, which is a reasonable simulation of a
pushed-to repository. But it also means that only 250
first-parent commits are in those packs (compared to almost
50,000 total in linux.git), and the rest are in the huge
"base" pack. So once we start looking at history in taht big
pack, that's where we'll find most everything, and even the
1-element cache gets close to 100% cache hits. You could
almost certainly show better numbers with a more
pathological case (e.g., distributing the objects more
evenly across the packs). But that's simply not that
realistic a scenario, so it makes more sense to focus on
these numbers.
The implementation itself is a straightforward application
of the MRU code. We provide an MRU-ordered list of packs
that shadows the packed_git list. This is easy to do because
we only create and revise the pack list in one place. The
"reprepare" code path actually drops the whole MRU and
replaces it for simplicity. It would be more efficient to
just add new entries, but there's not much point in
optimizing here; repreparing happens rarely, and only after
doing a lot of other expensive work. The key things to keep
optimized are traversal (which is just a normal linked list,
albeit with one extra level of indirection over the regular
packed_git list), and marking (which is a constant number of
pointer assignments, though slightly more than the old
last_found_pack was; it doesn't seem to create a measurable
slowdown, though).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-07-29 04:09:46 +00:00
|
|
|
struct mru_entry *p;
|
2006-09-21 04:05:37 +00:00
|
|
|
|
2005-06-27 10:35:33 +00:00
|
|
|
prepare_packed_git();
|
2007-05-31 02:48:13 +00:00
|
|
|
if (!packed_git)
|
|
|
|
return 0;
|
2005-06-27 10:35:33 +00:00
|
|
|
|
find_pack_entry: replace last_found_pack with MRU cache
Each pack has an index for looking up entries in O(log n)
time, but if we have multiple packs, we have to scan through
them linearly. This can produce a measurable overhead for
some operations.
We dealt with this long ago in f7c22cc (always start looking
up objects in the last used pack first, 2007-05-30), which
keeps what is essentially a 1-element most-recently-used
cache. In theory, we should be able to do better by keeping
a similar but longer cache, that is the same length as the
pack-list itself.
Since we now have a convenient generic MRU structure, we can
plug it in and measure. Here are the numbers for running
p5303 against linux.git:
Test HEAD^ HEAD
------------------------------------------------------------------------
5303.3: rev-list (1) 31.56(31.28+0.27) 31.30(31.08+0.20) -0.8%
5303.4: repack (1) 40.62(39.35+2.36) 40.60(39.27+2.44) -0.0%
5303.6: rev-list (50) 31.31(31.06+0.23) 31.23(31.00+0.22) -0.3%
5303.7: repack (50) 58.65(69.12+1.94) 58.27(68.64+2.05) -0.6%
5303.9: rev-list (1000) 38.74(38.40+0.33) 31.87(31.62+0.24) -17.7%
5303.10: repack (1000) 367.20(441.80+4.62) 342.00(414.04+3.72) -6.9%
The main numbers of interest here are the rev-list ones
(since that is exercising the normal object lookup code
path). The single-pack case shouldn't improve at all; the
260ms speedup there is just part of the run-to-run noise
(but it's important to note that we didn't make anything
worse with the overhead of maintaining our cache). In the
50-pack case, we see similar results. There may be a slight
improvement, but it's mostly within the noise.
The 1000-pack case does show a big improvement, though. That
carries over to the repack case, as well. Even though we
haven't touched its pack-search loop yet, it does still do a
lot of normal object lookups (e.g., for the internal
revision walk), and so improves.
As a point of reference, I also ran the 1000-pack test
against a version of HEAD^ with the last_found_pack
optimization disabled. It takes ~60s, so that gives an
indication of how much even the single-element cache is
helping.
For comparison, here's a smaller repository, git.git:
Test HEAD^ HEAD
---------------------------------------------------------------------
5303.3: rev-list (1) 1.56(1.54+0.01) 1.54(1.51+0.02) -1.3%
5303.4: repack (1) 1.84(1.80+0.10) 1.82(1.80+0.09) -1.1%
5303.6: rev-list (50) 1.58(1.55+0.02) 1.59(1.57+0.01) +0.6%
5303.7: repack (50) 2.50(3.18+0.04) 2.50(3.14+0.04) +0.0%
5303.9: rev-list (1000) 2.76(2.71+0.04) 2.24(2.21+0.02) -18.8%
5303.10: repack (1000) 13.21(19.56+0.25) 11.66(18.01+0.21) -11.7%
You can see that the percentage improvement is similar.
That's because the lookup we are optimizing is roughly
O(nr_objects * nr_packs). Since the number of packs is
constant in both tests, we'd expect the improvement to be
linear in the number of objects. But the whole process is
also linear in the number of objects, so the improvement
is a constant factor.
The exact improvement does also depend on the contents of
the packs. In p5303, the extra packs all have 5 first-parent
commits in them, which is a reasonable simulation of a
pushed-to repository. But it also means that only 250
first-parent commits are in those packs (compared to almost
50,000 total in linux.git), and the rest are in the huge
"base" pack. So once we start looking at history in taht big
pack, that's where we'll find most everything, and even the
1-element cache gets close to 100% cache hits. You could
almost certainly show better numbers with a more
pathological case (e.g., distributing the objects more
evenly across the packs). But that's simply not that
realistic a scenario, so it makes more sense to focus on
these numbers.
The implementation itself is a straightforward application
of the MRU code. We provide an MRU-ordered list of packs
that shadows the packed_git list. This is easy to do because
we only create and revise the pack list in one place. The
"reprepare" code path actually drops the whole MRU and
replaces it for simplicity. It would be more efficient to
just add new entries, but there's not much point in
optimizing here; repreparing happens rarely, and only after
doing a lot of other expensive work. The key things to keep
optimized are traversal (which is just a normal linked list,
albeit with one extra level of indirection over the regular
packed_git list), and marking (which is a constant number of
pointer assignments, though slightly more than the old
last_found_pack was; it doesn't seem to create a measurable
slowdown, though).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-07-29 04:09:46 +00:00
|
|
|
for (p = packed_git_mru->head; p; p = p->next) {
|
|
|
|
if (fill_pack_entry(sha1, e, p->item)) {
|
|
|
|
mru_mark(packed_git_mru, p);
|
2014-02-21 16:32:04 +00:00
|
|
|
return 1;
|
|
|
|
}
|
2012-02-01 13:48:55 +00:00
|
|
|
}
|
2005-06-27 10:35:33 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2007-06-07 07:04:01 +00:00
|
|
|
struct packed_git *find_sha1_pack(const unsigned char *sha1,
|
2005-08-01 00:53:44 +00:00
|
|
|
struct packed_git *packs)
|
|
|
|
{
|
|
|
|
struct packed_git *p;
|
|
|
|
|
|
|
|
for (p = packs; p; p = p->next) {
|
2006-09-21 04:05:37 +00:00
|
|
|
if (find_pack_entry_one(sha1, p))
|
2005-08-01 00:53:44 +00:00
|
|
|
return p;
|
|
|
|
}
|
|
|
|
return NULL;
|
2007-02-26 19:55:59 +00:00
|
|
|
|
2005-08-01 00:53:44 +00:00
|
|
|
}
|
|
|
|
|
sha1_loose_object_info: make type lookup optional
Until recently, the only items to request from
sha1_object_info_extended were type and size. This meant
that we always had to open a loose object file to determine
one or the other. But with the addition of the disk_size
query, it's possible that we can fulfill the query without
even opening the object file at all. However, since the
function interface always returns the type, we have no way
of knowing whether the caller cares about it or not.
This patch only modified sha1_loose_object_info to make type
lookup optional using an out-parameter, similar to the way
the size is handled (and the return value is "0" or "-1" for
success or error, respectively).
There should be no functional change yet, though, as
sha1_object_info_extended, the only caller, will always ask
for a type.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-07-12 06:30:48 +00:00
|
|
|
static int sha1_loose_object_info(const unsigned char *sha1,
|
2015-05-03 14:29:59 +00:00
|
|
|
struct object_info *oi,
|
|
|
|
int flags)
|
2005-06-02 22:20:54 +00:00
|
|
|
{
|
2015-05-03 14:29:59 +00:00
|
|
|
int status = 0;
|
|
|
|
unsigned long mapsize;
|
2005-06-02 22:20:54 +00:00
|
|
|
void *map;
|
2011-06-10 18:52:15 +00:00
|
|
|
git_zstream stream;
|
2007-02-26 19:55:55 +00:00
|
|
|
char hdr[32];
|
2015-05-03 14:29:59 +00:00
|
|
|
struct strbuf hdrbuf = STRBUF_INIT;
|
2005-06-02 22:20:54 +00:00
|
|
|
|
2013-12-21 14:24:20 +00:00
|
|
|
if (oi->delta_base_sha1)
|
|
|
|
hashclr(oi->delta_base_sha1);
|
|
|
|
|
sha1_loose_object_info: make type lookup optional
Until recently, the only items to request from
sha1_object_info_extended were type and size. This meant
that we always had to open a loose object file to determine
one or the other. But with the addition of the disk_size
query, it's possible that we can fulfill the query without
even opening the object file at all. However, since the
function interface always returns the type, we have no way
of knowing whether the caller cares about it or not.
This patch only modified sha1_loose_object_info to make type
lookup optional using an out-parameter, similar to the way
the size is handled (and the return value is "0" or "-1" for
success or error, respectively).
There should be no functional change yet, though, as
sha1_object_info_extended, the only caller, will always ask
for a type.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-07-12 06:30:48 +00:00
|
|
|
/*
|
|
|
|
* If we don't care about type or size, then we don't
|
2013-11-06 18:00:57 +00:00
|
|
|
* need to look inside the object at all. Note that we
|
|
|
|
* do not optimize out the stat call, even if the
|
|
|
|
* caller doesn't care about the disk-size, since our
|
|
|
|
* return value implicitly indicates whether the
|
|
|
|
* object even exists.
|
sha1_loose_object_info: make type lookup optional
Until recently, the only items to request from
sha1_object_info_extended were type and size. This meant
that we always had to open a loose object file to determine
one or the other. But with the addition of the disk_size
query, it's possible that we can fulfill the query without
even opening the object file at all. However, since the
function interface always returns the type, we have no way
of knowing whether the caller cares about it or not.
This patch only modified sha1_loose_object_info to make type
lookup optional using an out-parameter, similar to the way
the size is handled (and the return value is "0" or "-1" for
success or error, respectively).
There should be no functional change yet, though, as
sha1_object_info_extended, the only caller, will always ask
for a type.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-07-12 06:30:48 +00:00
|
|
|
*/
|
2015-05-03 14:29:59 +00:00
|
|
|
if (!oi->typep && !oi->typename && !oi->sizep) {
|
2017-01-13 17:54:39 +00:00
|
|
|
const char *path;
|
2013-11-06 18:00:57 +00:00
|
|
|
struct stat st;
|
2017-01-13 17:54:39 +00:00
|
|
|
if (stat_sha1_file(sha1, &st, &path) < 0)
|
2013-11-06 18:00:57 +00:00
|
|
|
return -1;
|
|
|
|
if (oi->disk_sizep)
|
2013-07-12 06:37:53 +00:00
|
|
|
*oi->disk_sizep = st.st_size;
|
sha1_loose_object_info: make type lookup optional
Until recently, the only items to request from
sha1_object_info_extended were type and size. This meant
that we always had to open a loose object file to determine
one or the other. But with the addition of the disk_size
query, it's possible that we can fulfill the query without
even opening the object file at all. However, since the
function interface always returns the type, we have no way
of knowing whether the caller cares about it or not.
This patch only modified sha1_loose_object_info to make type
lookup optional using an out-parameter, similar to the way
the size is handled (and the return value is "0" or "-1" for
success or error, respectively).
There should be no functional change yet, though, as
sha1_object_info_extended, the only caller, will always ask
for a type.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-07-12 06:30:48 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2006-07-17 22:04:47 +00:00
|
|
|
map = map_sha1_file(sha1, &mapsize);
|
2006-11-27 23:18:55 +00:00
|
|
|
if (!map)
|
2013-05-30 20:00:22 +00:00
|
|
|
return -1;
|
2013-07-12 06:37:53 +00:00
|
|
|
if (oi->disk_sizep)
|
|
|
|
*oi->disk_sizep = mapsize;
|
2015-05-03 14:29:59 +00:00
|
|
|
if ((flags & LOOKUP_UNKNOWN_OBJECT)) {
|
|
|
|
if (unpack_sha1_header_to_strbuf(&stream, map, mapsize, hdr, sizeof(hdr), &hdrbuf) < 0)
|
|
|
|
status = error("unable to unpack %s header with --allow-unknown-type",
|
|
|
|
sha1_to_hex(sha1));
|
|
|
|
} else if (unpack_sha1_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0)
|
2005-06-27 10:34:06 +00:00
|
|
|
status = error("unable to unpack %s header",
|
|
|
|
sha1_to_hex(sha1));
|
2015-05-03 14:29:59 +00:00
|
|
|
if (status < 0)
|
|
|
|
; /* Do nothing */
|
|
|
|
else if (hdrbuf.len) {
|
|
|
|
if ((status = parse_sha1_header_extended(hdrbuf.buf, oi, flags)) < 0)
|
|
|
|
status = error("unable to parse %s header with --allow-unknown-type",
|
|
|
|
sha1_to_hex(sha1));
|
|
|
|
} else if ((status = parse_sha1_header_extended(hdr, oi, flags)) < 0)
|
2005-06-27 10:34:06 +00:00
|
|
|
status = error("unable to parse %s header", sha1_to_hex(sha1));
|
2009-01-08 03:54:47 +00:00
|
|
|
git_inflate_end(&stream);
|
2005-06-02 22:20:54 +00:00
|
|
|
munmap(map, mapsize);
|
2015-05-03 14:29:59 +00:00
|
|
|
if (status && oi->typep)
|
2013-07-12 06:37:53 +00:00
|
|
|
*oi->typep = status;
|
2015-05-03 14:29:59 +00:00
|
|
|
strbuf_release(&hdrbuf);
|
sha1_loose_object_info: make type lookup optional
Until recently, the only items to request from
sha1_object_info_extended were type and size. This meant
that we always had to open a loose object file to determine
one or the other. But with the addition of the disk_size
query, it's possible that we can fulfill the query without
even opening the object file at all. However, since the
function interface always returns the type, we have no way
of knowing whether the caller cares about it or not.
This patch only modified sha1_loose_object_info to make type
lookup optional using an out-parameter, similar to the way
the size is handled (and the return value is "0" or "-1" for
success or error, respectively).
There should be no functional change yet, though, as
sha1_object_info_extended, the only caller, will always ask
for a type.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-07-12 06:30:48 +00:00
|
|
|
return 0;
|
2005-06-02 22:20:54 +00:00
|
|
|
}
|
|
|
|
|
2013-12-11 07:46:07 +00:00
|
|
|
int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi, unsigned flags)
|
2006-11-27 23:18:55 +00:00
|
|
|
{
|
2011-02-05 14:03:02 +00:00
|
|
|
struct cached_object *co;
|
2006-11-27 23:18:55 +00:00
|
|
|
struct pack_entry e;
|
2013-07-12 06:34:57 +00:00
|
|
|
int rtype;
|
2015-05-03 14:29:59 +00:00
|
|
|
enum object_type real_type;
|
2013-12-11 07:46:09 +00:00
|
|
|
const unsigned char *real = lookup_replace_object_extended(sha1, flags);
|
2006-11-27 23:18:55 +00:00
|
|
|
|
2013-12-11 07:46:09 +00:00
|
|
|
co = find_cached_object(real);
|
2011-02-05 14:03:02 +00:00
|
|
|
if (co) {
|
2013-07-12 06:34:57 +00:00
|
|
|
if (oi->typep)
|
|
|
|
*(oi->typep) = co->type;
|
2011-05-12 22:51:38 +00:00
|
|
|
if (oi->sizep)
|
|
|
|
*(oi->sizep) = co->size;
|
2013-07-07 10:04:00 +00:00
|
|
|
if (oi->disk_sizep)
|
|
|
|
*(oi->disk_sizep) = 0;
|
2013-12-21 14:24:20 +00:00
|
|
|
if (oi->delta_base_sha1)
|
|
|
|
hashclr(oi->delta_base_sha1);
|
2015-05-03 14:29:59 +00:00
|
|
|
if (oi->typename)
|
|
|
|
strbuf_addstr(oi->typename, typename(co->type));
|
2011-05-12 22:51:38 +00:00
|
|
|
oi->whence = OI_CACHED;
|
2013-07-12 06:34:57 +00:00
|
|
|
return 0;
|
2011-02-05 14:03:02 +00:00
|
|
|
}
|
|
|
|
|
2013-12-11 07:46:09 +00:00
|
|
|
if (!find_pack_entry(real, &e)) {
|
2008-08-05 20:08:41 +00:00
|
|
|
/* Most likely it's a loose object. */
|
2015-05-03 14:29:59 +00:00
|
|
|
if (!sha1_loose_object_info(real, oi, flags)) {
|
2011-05-12 22:51:38 +00:00
|
|
|
oi->whence = OI_LOOSE;
|
2013-07-12 06:34:57 +00:00
|
|
|
return 0;
|
2011-05-12 22:51:38 +00:00
|
|
|
}
|
2008-08-05 20:08:41 +00:00
|
|
|
|
|
|
|
/* Not a loose object; someone else may have just packed it. */
|
2006-11-27 23:18:55 +00:00
|
|
|
reprepare_packed_git();
|
2013-12-11 07:46:09 +00:00
|
|
|
if (!find_pack_entry(real, &e))
|
sha1_loose_object_info: make type lookup optional
Until recently, the only items to request from
sha1_object_info_extended were type and size. This meant
that we always had to open a loose object file to determine
one or the other. But with the addition of the disk_size
query, it's possible that we can fulfill the query without
even opening the object file at all. However, since the
function interface always returns the type, we have no way
of knowing whether the caller cares about it or not.
This patch only modified sha1_loose_object_info to make type
lookup optional using an out-parameter, similar to the way
the size is handled (and the return value is "0" or "-1" for
success or error, respectively).
There should be no functional change yet, though, as
sha1_object_info_extended, the only caller, will always ask
for a type.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-07-12 06:30:48 +00:00
|
|
|
return -1;
|
2006-11-27 23:18:55 +00:00
|
|
|
}
|
2008-10-29 23:02:47 +00:00
|
|
|
|
2015-05-03 14:29:59 +00:00
|
|
|
/*
|
|
|
|
* packed_object_info() does not follow the delta chain to
|
|
|
|
* find out the real type, unless it is given oi->typep.
|
|
|
|
*/
|
|
|
|
if (oi->typename && !oi->typep)
|
|
|
|
oi->typep = &real_type;
|
|
|
|
|
2013-07-12 06:37:53 +00:00
|
|
|
rtype = packed_object_info(e.p, e.offset, oi);
|
2013-07-12 06:32:25 +00:00
|
|
|
if (rtype < 0) {
|
2013-12-11 07:46:09 +00:00
|
|
|
mark_bad_packed_object(e.p, real);
|
2015-05-03 14:29:59 +00:00
|
|
|
if (oi->typep == &real_type)
|
|
|
|
oi->typep = NULL;
|
2013-12-11 07:46:09 +00:00
|
|
|
return sha1_object_info_extended(real, oi, 0);
|
2011-05-13 20:20:43 +00:00
|
|
|
} else if (in_delta_base_cache(e.p, e.offset)) {
|
|
|
|
oi->whence = OI_DBCACHED;
|
2011-05-12 22:51:38 +00:00
|
|
|
} else {
|
|
|
|
oi->whence = OI_PACKED;
|
|
|
|
oi->u.packed.offset = e.offset;
|
|
|
|
oi->u.packed.pack = e.p;
|
|
|
|
oi->u.packed.is_delta = (rtype == OBJ_REF_DELTA ||
|
|
|
|
rtype == OBJ_OFS_DELTA);
|
2008-10-29 23:02:47 +00:00
|
|
|
}
|
2015-05-03 14:29:59 +00:00
|
|
|
if (oi->typename)
|
|
|
|
strbuf_addstr(oi->typename, typename(*oi->typep));
|
|
|
|
if (oi->typep == &real_type)
|
|
|
|
oi->typep = NULL;
|
2008-10-29 23:02:47 +00:00
|
|
|
|
2013-07-12 06:34:57 +00:00
|
|
|
return 0;
|
2006-11-27 23:18:55 +00:00
|
|
|
}
|
|
|
|
|
2013-10-26 22:34:30 +00:00
|
|
|
/* returns enum object_type or negative */
|
2011-05-12 22:51:38 +00:00
|
|
|
int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)
|
|
|
|
{
|
2013-07-12 06:34:57 +00:00
|
|
|
enum object_type type;
|
provide an initializer for "struct object_info"
An all-zero initializer is fine for this struct, but because
the first element is a pointer, call sites need to know to
use "NULL" instead of "0". Otherwise some static checkers
like "sparse" will complain; see d099b71 (Fix some sparse
warnings, 2013-07-18) for example. So let's provide an
initializer to make this easier to get right.
But let's also comment that memset() to zero is explicitly
OK[1]. One of the callers embeds object_info in another
struct which is initialized via memset (expand_data in
builtin/cat-file.c). Since our subset of C doesn't allow
assignment from a compound literal, handling this in any
other way is awkward, so we'd like to keep the ability to
initialize by memset(). By documenting this property, it
should make anybody who wants to change the initializer
think twice before doing so.
There's one other caller of interest. In parse_sha1_header(),
we did not initialize the struct fully in the first place.
This turned out not to be a bug because the sub-function it
calls does not look at any other fields except the ones we
did initialize. But that assumption might not hold in the
future, so it's a dangerous construct. This patch switches
it to initializing the whole struct, which protects us
against unexpected reads of the other fields.
[1] Obviously using memset() to initialize a pointer
violates the C standard, but we long ago decided that it
was an acceptable tradeoff in the real world.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-08-11 09:24:35 +00:00
|
|
|
struct object_info oi = OBJECT_INFO_INIT;
|
2011-05-12 22:51:38 +00:00
|
|
|
|
2013-07-12 06:34:57 +00:00
|
|
|
oi.typep = &type;
|
2011-05-12 22:51:38 +00:00
|
|
|
oi.sizep = sizep;
|
2013-12-11 07:46:07 +00:00
|
|
|
if (sha1_object_info_extended(sha1, &oi, LOOKUP_REPLACE_OBJECT) < 0)
|
2013-07-12 06:34:57 +00:00
|
|
|
return -1;
|
|
|
|
return type;
|
2011-05-12 22:51:38 +00:00
|
|
|
}
|
|
|
|
|
2007-02-26 19:55:59 +00:00
|
|
|
static void *read_packed_sha1(const unsigned char *sha1,
|
|
|
|
enum object_type *type, unsigned long *size)
|
2005-06-27 10:35:33 +00:00
|
|
|
{
|
|
|
|
struct pack_entry e;
|
2008-06-24 01:23:39 +00:00
|
|
|
void *data;
|
2005-06-27 10:35:33 +00:00
|
|
|
|
2009-02-28 07:15:53 +00:00
|
|
|
if (!find_pack_entry(sha1, &e))
|
2005-06-27 10:35:33 +00:00
|
|
|
return NULL;
|
2016-08-22 21:57:45 +00:00
|
|
|
data = cache_or_unpack_entry(e.p, e.offset, size, type);
|
2008-06-24 01:23:39 +00:00
|
|
|
if (!data) {
|
|
|
|
/*
|
|
|
|
* We're probably in deep shit, but let's try to fetch
|
|
|
|
* the required object anyway from another pack or loose.
|
|
|
|
* This should happen only in the presence of a corrupted
|
|
|
|
* pack, and is better than failing outright.
|
|
|
|
*/
|
|
|
|
error("failed to read object %s at offset %"PRIuMAX" from %s",
|
|
|
|
sha1_to_hex(sha1), (uintmax_t)e.offset, e.p->pack_name);
|
|
|
|
mark_bad_packed_object(e.p, sha1);
|
2008-07-15 01:46:48 +00:00
|
|
|
data = read_object(sha1, type, size);
|
2008-06-24 01:23:39 +00:00
|
|
|
}
|
|
|
|
return data;
|
2005-06-27 10:35:33 +00:00
|
|
|
}
|
|
|
|
|
2007-02-26 19:55:59 +00:00
|
|
|
int pretend_sha1_file(void *buf, unsigned long len, enum object_type type,
|
|
|
|
unsigned char *sha1)
|
2007-02-05 05:42:38 +00:00
|
|
|
{
|
|
|
|
struct cached_object *co;
|
|
|
|
|
2007-02-26 19:55:59 +00:00
|
|
|
hash_sha1_file(buf, len, typename(type), sha1);
|
2007-02-05 05:42:38 +00:00
|
|
|
if (has_sha1_file(sha1) || find_cached_object(sha1))
|
|
|
|
return 0;
|
2014-03-03 22:32:02 +00:00
|
|
|
ALLOC_GROW(cached_objects, cached_object_nr + 1, cached_object_alloc);
|
2007-02-05 05:42:38 +00:00
|
|
|
co = &cached_objects[cached_object_nr++];
|
|
|
|
co->size = len;
|
2007-02-26 19:55:59 +00:00
|
|
|
co->type = type;
|
2007-02-16 01:02:06 +00:00
|
|
|
co->buf = xmalloc(len);
|
|
|
|
memcpy(co->buf, buf, len);
|
2007-02-05 05:42:38 +00:00
|
|
|
hashcpy(co->sha1, sha1);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2009-01-12 17:42:24 +00:00
|
|
|
static void *read_object(const unsigned char *sha1, enum object_type *type,
|
|
|
|
unsigned long *size)
|
2005-04-18 20:04:43 +00:00
|
|
|
{
|
|
|
|
unsigned long mapsize;
|
|
|
|
void *map, *buf;
|
2007-02-05 05:42:38 +00:00
|
|
|
struct cached_object *co;
|
|
|
|
|
|
|
|
co = find_cached_object(sha1);
|
|
|
|
if (co) {
|
2007-02-26 19:55:59 +00:00
|
|
|
*type = co->type;
|
2007-02-05 05:42:38 +00:00
|
|
|
*size = co->size;
|
2007-09-15 22:32:36 +00:00
|
|
|
return xmemdupz(co->buf, co->size);
|
2007-02-05 05:42:38 +00:00
|
|
|
}
|
2005-04-18 20:04:43 +00:00
|
|
|
|
2007-01-22 20:29:45 +00:00
|
|
|
buf = read_packed_sha1(sha1, type, size);
|
|
|
|
if (buf)
|
|
|
|
return buf;
|
2006-07-17 22:04:47 +00:00
|
|
|
map = map_sha1_file(sha1, &mapsize);
|
2005-04-18 20:04:43 +00:00
|
|
|
if (map) {
|
2007-03-05 08:21:37 +00:00
|
|
|
buf = unpack_sha1_file(map, mapsize, type, size, sha1);
|
2005-04-18 20:04:43 +00:00
|
|
|
munmap(map, mapsize);
|
|
|
|
return buf;
|
|
|
|
}
|
2006-06-02 15:32:23 +00:00
|
|
|
reprepare_packed_git();
|
2007-01-22 20:29:45 +00:00
|
|
|
return read_packed_sha1(sha1, type, size);
|
2005-04-18 20:04:43 +00:00
|
|
|
}
|
|
|
|
|
2010-10-28 18:13:06 +00:00
|
|
|
/*
|
|
|
|
* This function dies on corrupt objects; the callers who want to
|
|
|
|
* deal with them should arrange to call read_object() and give error
|
|
|
|
* messages themselves.
|
|
|
|
*/
|
2011-05-15 19:54:54 +00:00
|
|
|
void *read_sha1_file_extended(const unsigned char *sha1,
|
|
|
|
enum object_type *type,
|
|
|
|
unsigned long *size,
|
|
|
|
unsigned flag)
|
2008-07-15 01:46:48 +00:00
|
|
|
{
|
2010-10-28 18:13:06 +00:00
|
|
|
void *data;
|
2010-10-28 18:13:06 +00:00
|
|
|
const struct packed_git *p;
|
2017-01-13 17:54:39 +00:00
|
|
|
const char *path;
|
|
|
|
struct stat st;
|
2013-12-11 07:46:06 +00:00
|
|
|
const unsigned char *repl = lookup_replace_object_extended(sha1, flag);
|
2010-10-28 18:13:06 +00:00
|
|
|
|
2010-10-28 18:13:06 +00:00
|
|
|
errno = 0;
|
|
|
|
data = read_object(repl, type, size);
|
2011-05-15 19:54:52 +00:00
|
|
|
if (data)
|
2010-10-28 18:13:06 +00:00
|
|
|
return data;
|
2009-01-23 09:06:53 +00:00
|
|
|
|
2011-01-20 20:12:20 +00:00
|
|
|
if (errno && errno != ENOENT)
|
2010-10-28 18:13:06 +00:00
|
|
|
die_errno("failed to read object %s", sha1_to_hex(sha1));
|
|
|
|
|
2009-01-23 09:06:53 +00:00
|
|
|
/* die if we replaced an object with one that does not exist */
|
2010-10-28 18:13:06 +00:00
|
|
|
if (repl != sha1)
|
2009-01-23 09:06:53 +00:00
|
|
|
die("replacement %s not found for %s",
|
|
|
|
sha1_to_hex(repl), sha1_to_hex(sha1));
|
|
|
|
|
2017-01-13 17:54:39 +00:00
|
|
|
if (!stat_sha1_file(repl, &st, &path))
|
2010-10-28 18:13:06 +00:00
|
|
|
die("loose object %s (stored in %s) is corrupt",
|
|
|
|
sha1_to_hex(repl), path);
|
2009-01-23 09:06:53 +00:00
|
|
|
|
2010-10-28 18:13:06 +00:00
|
|
|
if ((p = has_packed_and_bad(repl)) != NULL)
|
|
|
|
die("packed object %s (stored in %s) is corrupt",
|
|
|
|
sha1_to_hex(repl), p->pack_name);
|
2009-01-23 09:07:01 +00:00
|
|
|
|
2010-10-28 18:13:06 +00:00
|
|
|
return NULL;
|
2008-07-15 01:46:48 +00:00
|
|
|
}
|
|
|
|
|
2005-04-28 23:42:27 +00:00
|
|
|
void *read_object_with_reference(const unsigned char *sha1,
|
2007-02-26 19:55:59 +00:00
|
|
|
const char *required_type_name,
|
2005-04-28 23:42:27 +00:00
|
|
|
unsigned long *size,
|
|
|
|
unsigned char *actual_sha1_return)
|
2005-04-21 01:06:49 +00:00
|
|
|
{
|
2007-02-26 19:55:59 +00:00
|
|
|
enum object_type type, required_type;
|
2005-04-21 01:06:49 +00:00
|
|
|
void *buffer;
|
|
|
|
unsigned long isize;
|
2005-04-28 23:42:27 +00:00
|
|
|
unsigned char actual_sha1[20];
|
2005-04-21 01:06:49 +00:00
|
|
|
|
2007-02-26 19:55:59 +00:00
|
|
|
required_type = type_from_string(required_type_name);
|
2006-08-23 06:49:00 +00:00
|
|
|
hashcpy(actual_sha1, sha1);
|
2005-04-28 23:42:27 +00:00
|
|
|
while (1) {
|
|
|
|
int ref_length = -1;
|
|
|
|
const char *ref_type = NULL;
|
2005-04-21 01:06:49 +00:00
|
|
|
|
2007-02-26 19:55:59 +00:00
|
|
|
buffer = read_sha1_file(actual_sha1, &type, &isize);
|
2005-04-28 23:42:27 +00:00
|
|
|
if (!buffer)
|
|
|
|
return NULL;
|
2007-02-26 19:55:59 +00:00
|
|
|
if (type == required_type) {
|
2005-04-28 23:42:27 +00:00
|
|
|
*size = isize;
|
|
|
|
if (actual_sha1_return)
|
2006-08-23 06:49:00 +00:00
|
|
|
hashcpy(actual_sha1_return, actual_sha1);
|
2005-04-28 23:42:27 +00:00
|
|
|
return buffer;
|
|
|
|
}
|
|
|
|
/* Handle references */
|
2007-02-26 19:55:59 +00:00
|
|
|
else if (type == OBJ_COMMIT)
|
2005-04-28 23:42:27 +00:00
|
|
|
ref_type = "tree ";
|
2007-02-26 19:55:59 +00:00
|
|
|
else if (type == OBJ_TAG)
|
2005-04-28 23:42:27 +00:00
|
|
|
ref_type = "object ";
|
|
|
|
else {
|
|
|
|
free(buffer);
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
ref_length = strlen(ref_type);
|
2005-04-21 01:06:49 +00:00
|
|
|
|
2008-02-18 20:47:52 +00:00
|
|
|
if (ref_length + 40 > isize ||
|
|
|
|
memcmp(buffer, ref_type, ref_length) ||
|
2006-06-18 15:18:09 +00:00
|
|
|
get_sha1_hex((char *) buffer + ref_length, actual_sha1)) {
|
2005-04-28 23:42:27 +00:00
|
|
|
free(buffer);
|
|
|
|
return NULL;
|
|
|
|
}
|
2005-08-08 18:44:43 +00:00
|
|
|
free(buffer);
|
2005-04-28 23:42:27 +00:00
|
|
|
/* Now we have the ID of the referred-to object in
|
|
|
|
* actual_sha1. Check again. */
|
2005-04-21 01:06:49 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2007-03-20 20:02:09 +00:00
|
|
|
static void write_sha1_file_prepare(const void *buf, unsigned long len,
|
2006-10-15 12:02:03 +00:00
|
|
|
const char *type, unsigned char *sha1,
|
2007-02-26 19:55:55 +00:00
|
|
|
char *hdr, int *hdrlen)
|
2005-06-28 02:03:13 +00:00
|
|
|
{
|
2008-10-01 18:05:20 +00:00
|
|
|
git_SHA_CTX c;
|
2005-06-28 02:03:13 +00:00
|
|
|
|
|
|
|
/* Generate the header */
|
2015-09-24 21:06:42 +00:00
|
|
|
*hdrlen = xsnprintf(hdr, *hdrlen, "%s %lu", type, len)+1;
|
2005-06-28 02:03:13 +00:00
|
|
|
|
|
|
|
/* Sha1.. */
|
2008-10-01 18:05:20 +00:00
|
|
|
git_SHA1_Init(&c);
|
|
|
|
git_SHA1_Update(&c, hdr, *hdrlen);
|
|
|
|
git_SHA1_Update(&c, buf, len);
|
|
|
|
git_SHA1_Final(sha1, &c);
|
2005-06-28 02:03:13 +00:00
|
|
|
}
|
|
|
|
|
Create object subdirectories on demand
This makes it possible to have a "sparse" git object subdirectory
structure, something that has become much more attractive now that people
use pack-files all the time.
As a result of pack-files, a git object directory doesn't necessarily have
any individual objects lying around, and in that case it's just wasting
space to keep the empty first-level object directories around: on many
filesystems the 256 empty directories will be aboue 1MB of diskspace.
Even more importantly, after you re-pack a project that _used_ to be
unpacked, you could be left with huge directories that no longer contain
anything, but that waste space and take time to look through.
With this change, "git prune-packed" can just do an rmdir() on the
directories, and they'll get removed if empty, and re-created on demand.
This patch also tries to fix up "write_sha1_from_fd()" to use the new
common infrastructure for creating the object files, closing a hole where
we might otherwise leave half-written objects in the object database.
[jc: I unoptimized the part that really removes the fan-out directories
to ease transition. init-db still wastes 1MB of diskspace to hold 256
empty fan-outs, and prune-packed rmdir()'s the grown but empty directories,
but runs mkdir() immediately after that -- reducing the saving from 150KB
to 146KB. These parts will be re-introduced when everybody has the
on-demand capability.]
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-10-08 22:54:01 +00:00
|
|
|
/*
|
2009-03-25 23:19:36 +00:00
|
|
|
* Move the just written object into its final resting place.
|
Create object subdirectories on demand
This makes it possible to have a "sparse" git object subdirectory
structure, something that has become much more attractive now that people
use pack-files all the time.
As a result of pack-files, a git object directory doesn't necessarily have
any individual objects lying around, and in that case it's just wasting
space to keep the empty first-level object directories around: on many
filesystems the 256 empty directories will be aboue 1MB of diskspace.
Even more importantly, after you re-pack a project that _used_ to be
unpacked, you could be left with huge directories that no longer contain
anything, but that waste space and take time to look through.
With this change, "git prune-packed" can just do an rmdir() on the
directories, and they'll get removed if empty, and re-created on demand.
This patch also tries to fix up "write_sha1_from_fd()" to use the new
common infrastructure for creating the object files, closing a hole where
we might otherwise leave half-written objects in the object database.
[jc: I unoptimized the part that really removes the fan-out directories
to ease transition. init-db still wastes 1MB of diskspace to hold 256
empty fan-outs, and prune-packed rmdir()'s the grown but empty directories,
but runs mkdir() immediately after that -- reducing the saving from 150KB
to 146KB. These parts will be re-introduced when everybody has the
on-demand capability.]
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-10-08 22:54:01 +00:00
|
|
|
*/
|
2015-08-07 21:40:24 +00:00
|
|
|
int finalize_object_file(const char *tmpfile, const char *filename)
|
Create object subdirectories on demand
This makes it possible to have a "sparse" git object subdirectory
structure, something that has become much more attractive now that people
use pack-files all the time.
As a result of pack-files, a git object directory doesn't necessarily have
any individual objects lying around, and in that case it's just wasting
space to keep the empty first-level object directories around: on many
filesystems the 256 empty directories will be aboue 1MB of diskspace.
Even more importantly, after you re-pack a project that _used_ to be
unpacked, you could be left with huge directories that no longer contain
anything, but that waste space and take time to look through.
With this change, "git prune-packed" can just do an rmdir() on the
directories, and they'll get removed if empty, and re-created on demand.
This patch also tries to fix up "write_sha1_from_fd()" to use the new
common infrastructure for creating the object files, closing a hole where
we might otherwise leave half-written objects in the object database.
[jc: I unoptimized the part that really removes the fan-out directories
to ease transition. init-db still wastes 1MB of diskspace to hold 256
empty fan-outs, and prune-packed rmdir()'s the grown but empty directories,
but runs mkdir() immediately after that -- reducing the saving from 150KB
to 146KB. These parts will be re-introduced when everybody has the
on-demand capability.]
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-10-08 22:54:01 +00:00
|
|
|
{
|
2008-09-18 22:24:46 +00:00
|
|
|
int ret = 0;
|
2009-03-25 23:19:36 +00:00
|
|
|
|
2009-04-27 22:32:25 +00:00
|
|
|
if (object_creation_mode == OBJECT_CREATION_USES_RENAMES)
|
2009-04-25 09:57:14 +00:00
|
|
|
goto try_rename;
|
|
|
|
else if (link(tmpfile, filename))
|
2008-09-18 22:24:46 +00:00
|
|
|
ret = errno;
|
2005-10-26 17:27:36 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Coda hack - coda doesn't like cross-directory links,
|
|
|
|
* so we fall back to a rename, which will mean that it
|
|
|
|
* won't be able to check collisions, but that's not a
|
|
|
|
* big deal.
|
|
|
|
*
|
|
|
|
* The same holds for FAT formatted media.
|
|
|
|
*
|
2009-03-28 06:14:39 +00:00
|
|
|
* When this succeeds, we just return. We have nothing
|
2005-10-26 17:27:36 +00:00
|
|
|
* left to unlink.
|
|
|
|
*/
|
|
|
|
if (ret && ret != EEXIST) {
|
2009-04-25 09:57:14 +00:00
|
|
|
try_rename:
|
2005-10-26 17:27:36 +00:00
|
|
|
if (!rename(tmpfile, filename))
|
2009-03-28 06:14:39 +00:00
|
|
|
goto out;
|
2005-10-25 23:41:20 +00:00
|
|
|
ret = errno;
|
Create object subdirectories on demand
This makes it possible to have a "sparse" git object subdirectory
structure, something that has become much more attractive now that people
use pack-files all the time.
As a result of pack-files, a git object directory doesn't necessarily have
any individual objects lying around, and in that case it's just wasting
space to keep the empty first-level object directories around: on many
filesystems the 256 empty directories will be aboue 1MB of diskspace.
Even more importantly, after you re-pack a project that _used_ to be
unpacked, you could be left with huge directories that no longer contain
anything, but that waste space and take time to look through.
With this change, "git prune-packed" can just do an rmdir() on the
directories, and they'll get removed if empty, and re-created on demand.
This patch also tries to fix up "write_sha1_from_fd()" to use the new
common infrastructure for creating the object files, closing a hole where
we might otherwise leave half-written objects in the object database.
[jc: I unoptimized the part that really removes the fan-out directories
to ease transition. init-db still wastes 1MB of diskspace to hold 256
empty fan-outs, and prune-packed rmdir()'s the grown but empty directories,
but runs mkdir() immediately after that -- reducing the saving from 150KB
to 146KB. These parts will be re-introduced when everybody has the
on-demand capability.]
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-10-08 22:54:01 +00:00
|
|
|
}
|
2009-04-29 21:22:56 +00:00
|
|
|
unlink_or_warn(tmpfile);
|
Create object subdirectories on demand
This makes it possible to have a "sparse" git object subdirectory
structure, something that has become much more attractive now that people
use pack-files all the time.
As a result of pack-files, a git object directory doesn't necessarily have
any individual objects lying around, and in that case it's just wasting
space to keep the empty first-level object directories around: on many
filesystems the 256 empty directories will be aboue 1MB of diskspace.
Even more importantly, after you re-pack a project that _used_ to be
unpacked, you could be left with huge directories that no longer contain
anything, but that waste space and take time to look through.
With this change, "git prune-packed" can just do an rmdir() on the
directories, and they'll get removed if empty, and re-created on demand.
This patch also tries to fix up "write_sha1_from_fd()" to use the new
common infrastructure for creating the object files, closing a hole where
we might otherwise leave half-written objects in the object database.
[jc: I unoptimized the part that really removes the fan-out directories
to ease transition. init-db still wastes 1MB of diskspace to hold 256
empty fan-outs, and prune-packed rmdir()'s the grown but empty directories,
but runs mkdir() immediately after that -- reducing the saving from 150KB
to 146KB. These parts will be re-introduced when everybody has the
on-demand capability.]
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-10-08 22:54:01 +00:00
|
|
|
if (ret) {
|
|
|
|
if (ret != EEXIST) {
|
2016-05-08 09:47:56 +00:00
|
|
|
return error_errno("unable to write sha1 filename %s", filename);
|
Create object subdirectories on demand
This makes it possible to have a "sparse" git object subdirectory
structure, something that has become much more attractive now that people
use pack-files all the time.
As a result of pack-files, a git object directory doesn't necessarily have
any individual objects lying around, and in that case it's just wasting
space to keep the empty first-level object directories around: on many
filesystems the 256 empty directories will be aboue 1MB of diskspace.
Even more importantly, after you re-pack a project that _used_ to be
unpacked, you could be left with huge directories that no longer contain
anything, but that waste space and take time to look through.
With this change, "git prune-packed" can just do an rmdir() on the
directories, and they'll get removed if empty, and re-created on demand.
This patch also tries to fix up "write_sha1_from_fd()" to use the new
common infrastructure for creating the object files, closing a hole where
we might otherwise leave half-written objects in the object database.
[jc: I unoptimized the part that really removes the fan-out directories
to ease transition. init-db still wastes 1MB of diskspace to hold 256
empty fan-outs, and prune-packed rmdir()'s the grown but empty directories,
but runs mkdir() immediately after that -- reducing the saving from 150KB
to 146KB. These parts will be re-introduced when everybody has the
on-demand capability.]
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-10-08 22:54:01 +00:00
|
|
|
}
|
|
|
|
/* FIXME!!! Collision check here ? */
|
|
|
|
}
|
|
|
|
|
2009-03-28 06:14:39 +00:00
|
|
|
out:
|
2010-02-22 22:32:16 +00:00
|
|
|
if (adjust_shared_perm(filename))
|
2009-03-25 23:19:36 +00:00
|
|
|
return error("unable to set permission to '%s'", filename);
|
Create object subdirectories on demand
This makes it possible to have a "sparse" git object subdirectory
structure, something that has become much more attractive now that people
use pack-files all the time.
As a result of pack-files, a git object directory doesn't necessarily have
any individual objects lying around, and in that case it's just wasting
space to keep the empty first-level object directories around: on many
filesystems the 256 empty directories will be aboue 1MB of diskspace.
Even more importantly, after you re-pack a project that _used_ to be
unpacked, you could be left with huge directories that no longer contain
anything, but that waste space and take time to look through.
With this change, "git prune-packed" can just do an rmdir() on the
directories, and they'll get removed if empty, and re-created on demand.
This patch also tries to fix up "write_sha1_from_fd()" to use the new
common infrastructure for creating the object files, closing a hole where
we might otherwise leave half-written objects in the object database.
[jc: I unoptimized the part that really removes the fan-out directories
to ease transition. init-db still wastes 1MB of diskspace to hold 256
empty fan-outs, and prune-packed rmdir()'s the grown but empty directories,
but runs mkdir() immediately after that -- reducing the saving from 150KB
to 146KB. These parts will be re-introduced when everybody has the
on-demand capability.]
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2005-10-08 22:54:01 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2006-05-24 15:30:54 +00:00
|
|
|
static int write_buffer(int fd, const void *buf, size_t len)
|
|
|
|
{
|
2007-01-12 04:23:00 +00:00
|
|
|
if (write_in_full(fd, buf, len) < 0)
|
2016-05-08 09:47:56 +00:00
|
|
|
return error_errno("file write error");
|
2006-05-24 15:30:54 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2007-03-20 20:02:09 +00:00
|
|
|
int hash_sha1_file(const void *buf, unsigned long len, const char *type,
|
2006-10-14 10:45:36 +00:00
|
|
|
unsigned char *sha1)
|
|
|
|
{
|
2007-02-26 19:55:55 +00:00
|
|
|
char hdr[32];
|
2015-09-24 21:06:42 +00:00
|
|
|
int hdrlen = sizeof(hdr);
|
2006-10-14 10:45:36 +00:00
|
|
|
write_sha1_file_prepare(buf, len, type, sha1, hdr, &hdrlen);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2008-06-11 01:47:18 +00:00
|
|
|
/* Finalize a file on disk, and close it. */
|
|
|
|
static void close_sha1_file(int fd)
|
|
|
|
{
|
2008-06-18 22:18:44 +00:00
|
|
|
if (fsync_object_files)
|
|
|
|
fsync_or_die(fd, "sha1 file");
|
2008-06-11 01:47:18 +00:00
|
|
|
if (close(fd) != 0)
|
2009-06-27 15:58:46 +00:00
|
|
|
die_errno("error when closing sha1 file");
|
2008-06-11 01:47:18 +00:00
|
|
|
}
|
|
|
|
|
2008-06-14 17:50:12 +00:00
|
|
|
/* Size of directory component, including the ending '/' */
|
|
|
|
static inline int directory_size(const char *filename)
|
|
|
|
{
|
|
|
|
const char *s = strrchr(filename, '/');
|
|
|
|
if (!s)
|
|
|
|
return 0;
|
|
|
|
return s - filename + 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This creates a temporary file in the same directory as the final
|
|
|
|
* 'filename'
|
|
|
|
*
|
|
|
|
* We want to avoid cross-directory filename renames, because those
|
|
|
|
* can have problems on various filesystems (FAT, NFS, Coda).
|
|
|
|
*/
|
2015-09-24 21:07:49 +00:00
|
|
|
static int create_tmpfile(struct strbuf *tmp, const char *filename)
|
2008-06-14 17:50:12 +00:00
|
|
|
{
|
|
|
|
int fd, dirlen = directory_size(filename);
|
|
|
|
|
2015-09-24 21:07:49 +00:00
|
|
|
strbuf_reset(tmp);
|
|
|
|
strbuf_add(tmp, filename, dirlen);
|
|
|
|
strbuf_addstr(tmp, "tmp_obj_XXXXXX");
|
|
|
|
fd = git_mkstemp_mode(tmp->buf, 0444);
|
sha1_file: avoid bogus "file exists" error message
This avoids the following misleading error message:
error: unable to create temporary sha1 filename ./objects/15: File exists
mkstemp can fail for many reasons, one of which, ENOENT, can occur if
the directory for the temp file doesn't exist. create_tmpfile tried to
handle this case by always trying to mkdir the directory, even if it
already existed. This caused errno to be clobbered, so one cannot tell
why mkstemp really failed, and it truncated the buffer to just the
directory name, resulting in the strange error message shown above.
Note that in both occasions that I've seen this failure, it has not been
due to a missing directory, or bad permissions, but some other, unknown
mkstemp failure mode that did not occur when I ran git again. This code
could perhaps be made more robust by retrying mkstemp, in case it was a
transient failure.
Signed-off-by: Joey Hess <joey@kitenet.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-11-20 18:56:28 +00:00
|
|
|
if (fd < 0 && dirlen && errno == ENOENT) {
|
2015-09-24 21:07:49 +00:00
|
|
|
/*
|
|
|
|
* Make sure the directory exists; note that the contents
|
|
|
|
* of the buffer are undefined after mkstemp returns an
|
|
|
|
* error, so we have to rewrite the whole buffer from
|
|
|
|
* scratch.
|
|
|
|
*/
|
|
|
|
strbuf_reset(tmp);
|
|
|
|
strbuf_add(tmp, filename, dirlen - 1);
|
|
|
|
if (mkdir(tmp->buf, 0777) && errno != EEXIST)
|
sha1_file.c:create_tmpfile(): Fix race when creating loose object dirs
There are cases (e.g. when running concurrent fetches in a repo) where
multiple Git processes concurrently attempt to create loose objects
within the same objects/XX/ dir. The creation of the loose object files
is (AFAICS) safe from races, but the creation of the objects/XX/ dir in
which the loose objects reside is unsafe, for example:
Two concurrent fetches - A and B. As part of its fetch, A needs to store
12aaaaa as a loose object. B, on the other hand, needs to store 12bbbbb
as a loose object. The objects/12 directory does not already exist.
Concurrently, both A and B determine that they need to create the
objects/12 directory (because their first call to git_mkstemp_mode()
within create_tmpfile() fails witn ENOENT). One of them - let's say A -
executes the following mkdir() call before the other. This first call
returns success, and A moves on. When B gets around to calling mkdir(),
it fails with EEXIST, because A won the race. The mkdir() error causes B
to return -1 from create_tmpfile(), which propagates all the way,
resulting in the fetch failing with:
error: unable to create temporary file: File exists
fatal: failed to write object
fatal: unpack-objects failed
Although it's hard to add a testcase reproducing this issue, it's easy
to provoke if we insert a sleep after the
if (mkdir(buffer, 0777) || adjust_shared_perm(buffer))
return -1;
block, and then run two concurrent "git fetch"es against the same repo.
The fix is to simply handle mkdir() failing with EEXIST as a success.
If EEXIST is somehow returned for the wrong reasons (because the relevant
objects/XX is not a directory, or is otherwise unsuitable for object
storage), the following call to adjust_shared_perm(), or ultimately the
retried call to git_mkstemp_mode() will fail, and we end up returning
error from create_tmpfile() in any case.
Note that there are still cases where two users with unsuitable umasks
in a shared repo can end up in two races where one user first wins the
mkdir() race to create an objects/XX/ directory, and then the other user
wins the adjust_shared_perms() race to chmod() that directory, but fails
because it is (transiently, until the first users completes its chmod())
unwriteable to the other user. However, (an equivalent of) this race also
exists before this patch, and is made no worse by this patch.
Signed-off-by: Johan Herland <johan@herland.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-10-27 11:35:43 +00:00
|
|
|
return -1;
|
2015-09-24 21:07:49 +00:00
|
|
|
if (adjust_shared_perm(tmp->buf))
|
2008-06-14 17:50:12 +00:00
|
|
|
return -1;
|
|
|
|
|
|
|
|
/* Try again */
|
2015-09-24 21:07:49 +00:00
|
|
|
strbuf_addstr(tmp, "/tmp_obj_XXXXXX");
|
|
|
|
fd = git_mkstemp_mode(tmp->buf, 0444);
|
2008-06-14 17:50:12 +00:00
|
|
|
}
|
|
|
|
return fd;
|
|
|
|
}
|
|
|
|
|
2008-05-14 05:32:48 +00:00
|
|
|
static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,
|
2010-04-02 00:03:18 +00:00
|
|
|
const void *buf, unsigned long len, time_t mtime)
|
2005-04-18 20:04:43 +00:00
|
|
|
{
|
2009-01-29 05:56:34 +00:00
|
|
|
int fd, ret;
|
2010-02-21 04:27:31 +00:00
|
|
|
unsigned char compressed[4096];
|
2011-06-10 18:52:15 +00:00
|
|
|
git_zstream stream;
|
2010-02-21 20:48:06 +00:00
|
|
|
git_SHA_CTX c;
|
|
|
|
unsigned char parano_sha1[20];
|
2015-09-24 21:07:49 +00:00
|
|
|
static struct strbuf tmp_file = STRBUF_INIT;
|
2014-02-21 16:32:05 +00:00
|
|
|
const char *filename = sha1_file_name(sha1);
|
2005-04-25 17:19:53 +00:00
|
|
|
|
2015-09-24 21:07:49 +00:00
|
|
|
fd = create_tmpfile(&tmp_file, filename);
|
2005-05-03 18:46:16 +00:00
|
|
|
if (fd < 0) {
|
2008-11-14 07:19:34 +00:00
|
|
|
if (errno == EACCES)
|
2012-04-30 00:28:45 +00:00
|
|
|
return error("insufficient permission for adding an object to repository database %s", get_object_directory());
|
2006-11-09 12:52:05 +00:00
|
|
|
else
|
2016-05-08 09:47:56 +00:00
|
|
|
return error_errno("unable to create temporary file");
|
2005-05-03 18:46:16 +00:00
|
|
|
}
|
|
|
|
|
2005-04-18 20:04:43 +00:00
|
|
|
/* Set it up */
|
2011-06-10 17:55:10 +00:00
|
|
|
git_deflate_init(&stream, zlib_compression_level);
|
2005-04-18 20:04:43 +00:00
|
|
|
stream.next_out = compressed;
|
2010-02-21 04:27:31 +00:00
|
|
|
stream.avail_out = sizeof(compressed);
|
2010-02-21 20:48:06 +00:00
|
|
|
git_SHA1_Init(&c);
|
2005-04-25 17:19:53 +00:00
|
|
|
|
|
|
|
/* First header.. */
|
2007-02-26 19:55:55 +00:00
|
|
|
stream.next_in = (unsigned char *)hdr;
|
2005-04-25 17:19:53 +00:00
|
|
|
stream.avail_in = hdrlen;
|
2011-06-10 17:55:10 +00:00
|
|
|
while (git_deflate(&stream, 0) == Z_OK)
|
|
|
|
; /* nothing */
|
2010-02-21 20:48:06 +00:00
|
|
|
git_SHA1_Update(&c, hdr, hdrlen);
|
2005-04-25 17:19:53 +00:00
|
|
|
|
|
|
|
/* Then the data itself.. */
|
2010-04-02 00:03:18 +00:00
|
|
|
stream.next_in = (void *)buf;
|
2005-04-25 17:19:53 +00:00
|
|
|
stream.avail_in = len;
|
2010-02-21 04:27:31 +00:00
|
|
|
do {
|
2010-02-21 20:48:06 +00:00
|
|
|
unsigned char *in0 = stream.next_in;
|
2011-06-10 17:55:10 +00:00
|
|
|
ret = git_deflate(&stream, Z_FINISH);
|
2010-02-21 20:48:06 +00:00
|
|
|
git_SHA1_Update(&c, in0, stream.next_in - in0);
|
2010-02-21 04:27:31 +00:00
|
|
|
if (write_buffer(fd, compressed, stream.next_out - compressed) < 0)
|
|
|
|
die("unable to write sha1 file");
|
|
|
|
stream.next_out = compressed;
|
|
|
|
stream.avail_out = sizeof(compressed);
|
|
|
|
} while (ret == Z_OK);
|
|
|
|
|
Be more careful about zlib return values
When creating a new object, we use "deflate(stream, Z_FINISH)" in a loop
until it no longer returns Z_OK, and then we do "deflateEnd()" to finish
up business.
That should all work, but the fact is, it's not how you're _supposed_ to
use the zlib return values properly:
- deflate() should never return Z_OK in the first place, except if we
need to increase the output buffer size (which we're not doing, and
should never need to do, since we pre-allocated a buffer that is
supposed to be able to hold the output in full). So the "while()" loop
was incorrect: Z_OK doesn't actually mean "ok, continue", it means "ok,
allocate more memory for me and continue"!
- if we got an error return, we would consider it to be end-of-stream,
but it could be some internal zlib error. In short, we should check
for Z_STREAM_END explicitly, since that's the only valid return value
anyway for the Z_FINISH case.
- we never checked deflateEnd() return codes at all.
Now, admittedly, none of these issues should ever happen, unless there is
some internal bug in zlib. So this patch should make zero difference, but
it seems to be the right thing to do.
We should probablybe anal and check the return value of "deflateInit()"
too!
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-03-20 18:38:34 +00:00
|
|
|
if (ret != Z_STREAM_END)
|
|
|
|
die("unable to deflate new object %s (%d)", sha1_to_hex(sha1), ret);
|
2011-06-10 17:55:10 +00:00
|
|
|
ret = git_deflate_end_gently(&stream);
|
Be more careful about zlib return values
When creating a new object, we use "deflate(stream, Z_FINISH)" in a loop
until it no longer returns Z_OK, and then we do "deflateEnd()" to finish
up business.
That should all work, but the fact is, it's not how you're _supposed_ to
use the zlib return values properly:
- deflate() should never return Z_OK in the first place, except if we
need to increase the output buffer size (which we're not doing, and
should never need to do, since we pre-allocated a buffer that is
supposed to be able to hold the output in full). So the "while()" loop
was incorrect: Z_OK doesn't actually mean "ok, continue", it means "ok,
allocate more memory for me and continue"!
- if we got an error return, we would consider it to be end-of-stream,
but it could be some internal zlib error. In short, we should check
for Z_STREAM_END explicitly, since that's the only valid return value
anyway for the Z_FINISH case.
- we never checked deflateEnd() return codes at all.
Now, admittedly, none of these issues should ever happen, unless there is
some internal bug in zlib. So this patch should make zero difference, but
it seems to be the right thing to do.
We should probablybe anal and check the return value of "deflateInit()"
too!
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-03-20 18:38:34 +00:00
|
|
|
if (ret != Z_OK)
|
|
|
|
die("deflateEnd on object %s failed (%d)", sha1_to_hex(sha1), ret);
|
2010-02-21 20:48:06 +00:00
|
|
|
git_SHA1_Final(parano_sha1, &c);
|
|
|
|
if (hashcmp(sha1, parano_sha1) != 0)
|
|
|
|
die("confused by unstable object source data for %s", sha1_to_hex(sha1));
|
Be more careful about zlib return values
When creating a new object, we use "deflate(stream, Z_FINISH)" in a loop
until it no longer returns Z_OK, and then we do "deflateEnd()" to finish
up business.
That should all work, but the fact is, it's not how you're _supposed_ to
use the zlib return values properly:
- deflate() should never return Z_OK in the first place, except if we
need to increase the output buffer size (which we're not doing, and
should never need to do, since we pre-allocated a buffer that is
supposed to be able to hold the output in full). So the "while()" loop
was incorrect: Z_OK doesn't actually mean "ok, continue", it means "ok,
allocate more memory for me and continue"!
- if we got an error return, we would consider it to be end-of-stream,
but it could be some internal zlib error. In short, we should check
for Z_STREAM_END explicitly, since that's the only valid return value
anyway for the Z_FINISH case.
- we never checked deflateEnd() return codes at all.
Now, admittedly, none of these issues should ever happen, unless there is
some internal bug in zlib. So this patch should make zero difference, but
it seems to be the right thing to do.
We should probablybe anal and check the return value of "deflateInit()"
too!
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-03-20 18:38:34 +00:00
|
|
|
|
2008-06-11 01:47:18 +00:00
|
|
|
close_sha1_file(fd);
|
2005-04-18 20:04:43 +00:00
|
|
|
|
2008-05-14 05:32:48 +00:00
|
|
|
if (mtime) {
|
|
|
|
struct utimbuf utb;
|
|
|
|
utb.actime = mtime;
|
|
|
|
utb.modtime = mtime;
|
2015-09-24 21:07:49 +00:00
|
|
|
if (utime(tmp_file.buf, &utb) < 0)
|
2016-05-08 09:47:56 +00:00
|
|
|
warning_errno("failed utime() on %s", tmp_file.buf);
|
2008-05-14 05:32:48 +00:00
|
|
|
}
|
|
|
|
|
2015-09-24 21:07:49 +00:00
|
|
|
return finalize_object_file(tmp_file.buf, filename);
|
2005-04-18 20:04:43 +00:00
|
|
|
}
|
2005-04-24 01:47:23 +00:00
|
|
|
|
2014-10-15 22:42:22 +00:00
|
|
|
static int freshen_loose_object(const unsigned char *sha1)
|
|
|
|
{
|
|
|
|
return check_and_freshen(sha1, 1);
|
|
|
|
}
|
|
|
|
|
|
|
|
static int freshen_packed_object(const unsigned char *sha1)
|
|
|
|
{
|
|
|
|
struct pack_entry e;
|
2015-04-20 19:55:00 +00:00
|
|
|
if (!find_pack_entry(sha1, &e))
|
|
|
|
return 0;
|
|
|
|
if (e.p->freshened)
|
|
|
|
return 1;
|
|
|
|
if (!freshen_file(e.p->pack_name))
|
|
|
|
return 0;
|
|
|
|
e.p->freshened = 1;
|
|
|
|
return 1;
|
2014-10-15 22:42:22 +00:00
|
|
|
}
|
|
|
|
|
2015-05-04 18:08:10 +00:00
|
|
|
int write_sha1_file(const void *buf, unsigned long len, const char *type, unsigned char *sha1)
|
2008-05-14 05:32:48 +00:00
|
|
|
{
|
|
|
|
char hdr[32];
|
2015-09-24 21:06:42 +00:00
|
|
|
int hdrlen = sizeof(hdr);
|
2008-05-14 05:32:48 +00:00
|
|
|
|
|
|
|
/* Normally if we have it in the pack then we do not bother writing
|
|
|
|
* it out into .git/objects/??/?{38} file.
|
|
|
|
*/
|
|
|
|
write_sha1_file_prepare(buf, len, type, sha1, hdr, &hdrlen);
|
2015-04-20 19:54:03 +00:00
|
|
|
if (freshen_packed_object(sha1) || freshen_loose_object(sha1))
|
2008-05-14 05:32:48 +00:00
|
|
|
return 0;
|
|
|
|
return write_loose_object(sha1, hdr, hdrlen, buf, len, 0);
|
|
|
|
}
|
|
|
|
|
2015-05-04 07:25:15 +00:00
|
|
|
int hash_sha1_file_literally(const void *buf, unsigned long len, const char *type,
|
|
|
|
unsigned char *sha1, unsigned flags)
|
|
|
|
{
|
|
|
|
char *header;
|
|
|
|
int hdrlen, status = 0;
|
|
|
|
|
|
|
|
/* type string, SP, %lu of the length plus NUL must fit this */
|
2015-09-24 21:06:42 +00:00
|
|
|
hdrlen = strlen(type) + 32;
|
|
|
|
header = xmalloc(hdrlen);
|
2015-05-04 07:25:15 +00:00
|
|
|
write_sha1_file_prepare(buf, len, type, sha1, header, &hdrlen);
|
|
|
|
|
|
|
|
if (!(flags & HASH_WRITE_OBJECT))
|
|
|
|
goto cleanup;
|
2015-05-11 21:23:59 +00:00
|
|
|
if (freshen_packed_object(sha1) || freshen_loose_object(sha1))
|
2015-05-04 07:25:15 +00:00
|
|
|
goto cleanup;
|
|
|
|
status = write_loose_object(sha1, header, hdrlen, buf, len, 0);
|
|
|
|
|
|
|
|
cleanup:
|
|
|
|
free(header);
|
|
|
|
return status;
|
|
|
|
}
|
|
|
|
|
2008-05-14 05:32:48 +00:00
|
|
|
int force_object_loose(const unsigned char *sha1, time_t mtime)
|
|
|
|
{
|
|
|
|
void *buf;
|
|
|
|
unsigned long len;
|
|
|
|
enum object_type type;
|
|
|
|
char hdr[32];
|
|
|
|
int hdrlen;
|
2008-10-18 00:37:31 +00:00
|
|
|
int ret;
|
2008-05-14 05:32:48 +00:00
|
|
|
|
2008-06-14 18:43:01 +00:00
|
|
|
if (has_loose_object(sha1))
|
2008-05-14 05:32:48 +00:00
|
|
|
return 0;
|
|
|
|
buf = read_packed_sha1(sha1, &type, &len);
|
|
|
|
if (!buf)
|
|
|
|
return error("cannot read sha1_file for %s", sha1_to_hex(sha1));
|
2015-09-24 21:06:42 +00:00
|
|
|
hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %lu", typename(type), len) + 1;
|
2008-10-18 00:37:31 +00:00
|
|
|
ret = write_loose_object(sha1, hdr, hdrlen, buf, len, mtime);
|
|
|
|
free(buf);
|
|
|
|
|
|
|
|
return ret;
|
2008-05-14 05:32:48 +00:00
|
|
|
}
|
|
|
|
|
2005-08-01 00:53:44 +00:00
|
|
|
int has_pack_index(const unsigned char *sha1)
|
|
|
|
{
|
|
|
|
struct stat st;
|
|
|
|
if (stat(sha1_pack_index_name(sha1), &st))
|
|
|
|
return 0;
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
2009-02-28 07:15:53 +00:00
|
|
|
int has_sha1_pack(const unsigned char *sha1)
|
|
|
|
{
|
|
|
|
struct pack_entry e;
|
|
|
|
return find_pack_entry(sha1, &e);
|
|
|
|
}
|
|
|
|
|
2015-06-09 17:24:37 +00:00
|
|
|
int has_sha1_file_with_flags(const unsigned char *sha1, int flags)
|
2005-04-24 01:47:23 +00:00
|
|
|
{
|
2005-06-27 10:35:33 +00:00
|
|
|
struct pack_entry e;
|
|
|
|
|
2009-02-28 07:15:53 +00:00
|
|
|
if (find_pack_entry(sha1, &e))
|
2005-06-27 10:35:33 +00:00
|
|
|
return 1;
|
has_sha1_file: re-check pack directory before giving up
When we read a sha1 file, we first look for a packed
version, then a loose version, and then re-check the pack
directory again before concluding that we cannot find it.
This lets us handle a process that is writing to the
repository simultaneously (e.g., receive-pack writing a new
pack followed by a ref update, or git-repack packing
existing loose objects into a new pack).
However, we do not do the same trick with has_sha1_file; we
only check the packed objects once, followed by loose
objects. This means that we might incorrectly report that we
do not have an object, even though we could find it if we
simply re-checked the pack directory.
By itself, this is usually not a big deal. The other process
is running simultaneously, so we may run has_sha1_file
before it writes, anyway. It is a race whether we see the
object or not. However, we may also see other things
the writing process has done (like updating refs); and in
that case, we must be able to also see the new objects.
For example, imagine we are doing a for_each_ref iteration,
and somebody simultaneously pushes. Receive-pack may write
the pack and update a ref after we have examined the
objects/pack directory, but before the iteration gets to the
updated ref. When we do finally see the updated ref,
for_each_ref will call has_sha1_file to check whether the
ref is broken. If has_sha1_file returns the wrong answer, we
erroneously will think that the ref is broken.
For a normal iteration without DO_FOR_EACH_INCLUDE_BROKEN,
this means that the caller does not see the ref at all
(neither the old nor the new value). So not only will we
fail to see the new value of the ref (which is acceptable,
since we are running simultaneously with the writer, and we
might well read the ref before the writer commits its
write), but we will not see the old value either. For
programs that act on reachability like pack-objects or
prune, this can cause data loss, as we may see the objects
referenced by the original ref value as dangling (and either
omit them from the pack, or delete them via prune).
There's no test included here, because the success case is
two processes running simultaneously forever. But you can
replicate the issue with:
# base.sh
# run this in one terminal; it creates and pushes
# repeatedly to a repository
git init parent &&
(cd parent &&
# create a base commit that will trigger us looking at
# the objects/pack directory before we hit the updated ref
echo content >file &&
git add file &&
git commit -m base &&
# set the unpack limit abnormally low, which
# lets us simulate full-size pushes using tiny ones
git config receive.unpackLimit 1
) &&
git clone parent child &&
cd child &&
n=0 &&
while true; do
echo $n >file && git add file && git commit -m $n &&
git push origin HEAD:refs/remotes/child/master &&
n=$(($n + 1))
done
# fsck.sh
# now run this simultaneously in another terminal; it
# repeatedly fscks, looking for us to consider the
# newly-pushed ref broken. We cannot use for-each-ref
# here, as it uses DO_FOR_EACH_INCLUDE_BROKEN, which
# skips the has_sha1_file check (and if it wants
# more information on the object, it will actually read
# the object, which does the proper two-step lookup)
cd parent &&
while true; do
broken=`git fsck 2>&1 | grep remotes/child`
if test -n "$broken"; then
echo $broken
exit 1
fi
done
Without this patch, the fsck loop fails within a few seconds
(and almost instantly if the test repository actually has a
large number of refs). With it, the two can run
indefinitely.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-08-30 19:14:13 +00:00
|
|
|
if (has_loose_object(sha1))
|
|
|
|
return 1;
|
2015-06-09 17:24:37 +00:00
|
|
|
if (flags & HAS_SHA1_QUICK)
|
|
|
|
return 0;
|
has_sha1_file: re-check pack directory before giving up
When we read a sha1 file, we first look for a packed
version, then a loose version, and then re-check the pack
directory again before concluding that we cannot find it.
This lets us handle a process that is writing to the
repository simultaneously (e.g., receive-pack writing a new
pack followed by a ref update, or git-repack packing
existing loose objects into a new pack).
However, we do not do the same trick with has_sha1_file; we
only check the packed objects once, followed by loose
objects. This means that we might incorrectly report that we
do not have an object, even though we could find it if we
simply re-checked the pack directory.
By itself, this is usually not a big deal. The other process
is running simultaneously, so we may run has_sha1_file
before it writes, anyway. It is a race whether we see the
object or not. However, we may also see other things
the writing process has done (like updating refs); and in
that case, we must be able to also see the new objects.
For example, imagine we are doing a for_each_ref iteration,
and somebody simultaneously pushes. Receive-pack may write
the pack and update a ref after we have examined the
objects/pack directory, but before the iteration gets to the
updated ref. When we do finally see the updated ref,
for_each_ref will call has_sha1_file to check whether the
ref is broken. If has_sha1_file returns the wrong answer, we
erroneously will think that the ref is broken.
For a normal iteration without DO_FOR_EACH_INCLUDE_BROKEN,
this means that the caller does not see the ref at all
(neither the old nor the new value). So not only will we
fail to see the new value of the ref (which is acceptable,
since we are running simultaneously with the writer, and we
might well read the ref before the writer commits its
write), but we will not see the old value either. For
programs that act on reachability like pack-objects or
prune, this can cause data loss, as we may see the objects
referenced by the original ref value as dangling (and either
omit them from the pack, or delete them via prune).
There's no test included here, because the success case is
two processes running simultaneously forever. But you can
replicate the issue with:
# base.sh
# run this in one terminal; it creates and pushes
# repeatedly to a repository
git init parent &&
(cd parent &&
# create a base commit that will trigger us looking at
# the objects/pack directory before we hit the updated ref
echo content >file &&
git add file &&
git commit -m base &&
# set the unpack limit abnormally low, which
# lets us simulate full-size pushes using tiny ones
git config receive.unpackLimit 1
) &&
git clone parent child &&
cd child &&
n=0 &&
while true; do
echo $n >file && git add file && git commit -m $n &&
git push origin HEAD:refs/remotes/child/master &&
n=$(($n + 1))
done
# fsck.sh
# now run this simultaneously in another terminal; it
# repeatedly fscks, looking for us to consider the
# newly-pushed ref broken. We cannot use for-each-ref
# here, as it uses DO_FOR_EACH_INCLUDE_BROKEN, which
# skips the has_sha1_file check (and if it wants
# more information on the object, it will actually read
# the object, which does the proper two-step lookup)
cd parent &&
while true; do
broken=`git fsck 2>&1 | grep remotes/child`
if test -n "$broken"; then
echo $broken
exit 1
fi
done
Without this patch, the fsck loop fails within a few seconds
(and almost instantly if the test repository actually has a
large number of refs). With it, the two can run
indefinitely.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-08-30 19:14:13 +00:00
|
|
|
reprepare_packed_git();
|
|
|
|
return find_pack_entry(sha1, &e);
|
2015-11-10 02:22:19 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
int has_object_file(const struct object_id *oid)
|
|
|
|
{
|
|
|
|
return has_sha1_file(oid->hash);
|
2005-04-24 01:47:23 +00:00
|
|
|
}
|
2005-05-02 06:45:49 +00:00
|
|
|
|
fetch: use "quick" has_sha1_file for tag following
When we auto-follow tags in a fetch, we look at all of the
tags advertised by the remote and fetch ones where we don't
already have the tag, but we do have the object it peels to.
This involves a lot of calls to has_sha1_file(), some of
which we can reasonably expect to fail. Since 45e8a74
(has_sha1_file: re-check pack directory before giving up,
2013-08-30), this may cause many calls to
reprepare_packed_git(), which is potentially expensive.
This has gone unnoticed for several years because it
requires a fairly unique setup to matter:
1. You need to have a lot of packs on the client side to
make reprepare_packed_git() expensive (the most
expensive part is finding duplicates in an unsorted
list, which is currently quadratic).
2. You need a large number of tag refs on the server side
that are candidates for auto-following (i.e., that the
client doesn't have). Each one triggers a re-read of
the pack directory.
3. Under normal circumstances, the client would
auto-follow those tags and after one large fetch, (2)
would no longer be true. But if those tags point to
history which is disconnected from what the client
otherwise fetches, then it will never auto-follow, and
those candidates will impact it on every fetch.
So when all three are true, each fetch pays an extra
O(nr_tags * nr_packs^2) cost, mostly in string comparisons
on the pack names. This was exacerbated by 47bf4b0
(prepare_packed_git_one: refactor duplicate-pack check,
2014-06-30) which uses a slightly more expensive string
check, under the assumption that the duplicate check doesn't
happen very often (and it shouldn't; the real problem here
is how often we are calling reprepare_packed_git()).
This patch teaches fetch to use HAS_SHA1_QUICK to sacrifice
accuracy for speed, in cases where we might be racy with a
simultaneous repack. This is similar to the fix in 0eeb077
(index-pack: avoid excessive re-reading of pack directory,
2015-06-09). As with that case, it's OK for has_sha1_file()
occasionally say "no I don't have it" when we do, because
the worst case is not a corruption, but simply that we may
fail to auto-follow a tag that points to it.
Here are results from the included perf script, which sets
up a situation similar to the one described above:
Test HEAD^ HEAD
----------------------------------------------------------
5550.4: fetch 11.21(10.42+0.78) 0.08(0.04+0.02) -99.3%
Reported-by: Vegard Nossum <vegard.nossum@oracle.com>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2016-10-13 16:53:44 +00:00
|
|
|
int has_object_file_with_flags(const struct object_id *oid, int flags)
|
|
|
|
{
|
|
|
|
return has_sha1_file_with_flags(oid->hash, flags);
|
|
|
|
}
|
|
|
|
|
2011-02-05 10:52:21 +00:00
|
|
|
static void check_tree(const void *buf, size_t size)
|
|
|
|
{
|
|
|
|
struct tree_desc desc;
|
|
|
|
struct name_entry entry;
|
|
|
|
|
|
|
|
init_tree_desc(&desc, buf, size);
|
|
|
|
while (tree_entry(&desc, &entry))
|
|
|
|
/* do nothing
|
|
|
|
* tree_entry() will die() on malformed entries */
|
|
|
|
;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void check_commit(const void *buf, size_t size)
|
|
|
|
{
|
|
|
|
struct commit c;
|
|
|
|
memset(&c, 0, sizeof(c));
|
|
|
|
if (parse_commit_buffer(&c, buf, size))
|
|
|
|
die("corrupt commit");
|
|
|
|
}
|
|
|
|
|
|
|
|
static void check_tag(const void *buf, size_t size)
|
|
|
|
{
|
|
|
|
struct tag t;
|
|
|
|
memset(&t, 0, sizeof(t));
|
|
|
|
if (parse_tag_buffer(&t, buf, size))
|
|
|
|
die("corrupt tag");
|
|
|
|
}
|
|
|
|
|
2008-08-03 04:39:16 +00:00
|
|
|
static int index_mem(unsigned char *sha1, void *buf, size_t size,
|
2011-05-08 08:47:33 +00:00
|
|
|
enum object_type type,
|
|
|
|
const char *path, unsigned flags)
|
2006-05-23 18:19:04 +00:00
|
|
|
{
|
Lazy man's auto-CRLF
It currently does NOT know about file attributes, so it does its
conversion purely based on content. Maybe that is more in the "git
philosophy" anyway, since content is king, but I think we should try to do
the file attributes to turn it off on demand.
Anyway, BY DEFAULT it is off regardless, because it requires a
[core]
AutoCRLF = true
in your config file to be enabled. We could make that the default for
Windows, of course, the same way we do some other things (filemode etc).
But you can actually enable it on UNIX, and it will cause:
- "git update-index" will write blobs without CRLF
- "git diff" will diff working tree files without CRLF
- "git checkout" will write files to the working tree _with_ CRLF
and things work fine.
Funnily, it actually shows an odd file in git itself:
git clone -n git test-crlf
cd test-crlf
git config core.autocrlf true
git checkout
git diff
shows a diff for "Documentation/docbook-xsl.css". Why? Because we have
actually checked in that file *with* CRLF! So when "core.autocrlf" is
true, we'll always generate a *different* hash for it in the index,
because the index hash will be for the content _without_ CRLF.
Is this complete? I dunno. It seems to work for me. It doesn't use the
filename at all right now, and that's probably a deficiency (we could
certainly make the "is_binary()" heuristics also take standard filename
heuristics into account).
I don't pass in the filename at all for the "index_fd()" case
(git-update-index), so that would need to be passed around, but this
actually works fine.
NOTE NOTE NOTE! The "is_binary()" heuristics are totally made-up by yours
truly. I will not guarantee that they work at all reasonable. Caveat
emptor. But it _is_ simple, and it _is_ safe, since it's all off by
default.
The patch is pretty simple - the biggest part is the new "convert.c" file,
but even that is really just basic stuff that anybody can write in
"Teaching C 101" as a final project for their first class in programming.
Not to say that it's bug-free, of course - but at least we're not talking
about rocket surgery here.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-13 19:07:23 +00:00
|
|
|
int ret, re_allocated = 0;
|
2011-05-08 08:47:33 +00:00
|
|
|
int write_object = flags & HASH_WRITE_OBJECT;
|
2005-05-02 06:45:49 +00:00
|
|
|
|
2005-07-08 23:51:55 +00:00
|
|
|
if (!type)
|
2007-02-28 19:45:56 +00:00
|
|
|
type = OBJ_BLOB;
|
Lazy man's auto-CRLF
It currently does NOT know about file attributes, so it does its
conversion purely based on content. Maybe that is more in the "git
philosophy" anyway, since content is king, but I think we should try to do
the file attributes to turn it off on demand.
Anyway, BY DEFAULT it is off regardless, because it requires a
[core]
AutoCRLF = true
in your config file to be enabled. We could make that the default for
Windows, of course, the same way we do some other things (filemode etc).
But you can actually enable it on UNIX, and it will cause:
- "git update-index" will write blobs without CRLF
- "git diff" will diff working tree files without CRLF
- "git checkout" will write files to the working tree _with_ CRLF
and things work fine.
Funnily, it actually shows an odd file in git itself:
git clone -n git test-crlf
cd test-crlf
git config core.autocrlf true
git checkout
git diff
shows a diff for "Documentation/docbook-xsl.css". Why? Because we have
actually checked in that file *with* CRLF! So when "core.autocrlf" is
true, we'll always generate a *different* hash for it in the index,
because the index hash will be for the content _without_ CRLF.
Is this complete? I dunno. It seems to work for me. It doesn't use the
filename at all right now, and that's probably a deficiency (we could
certainly make the "is_binary()" heuristics also take standard filename
heuristics into account).
I don't pass in the filename at all for the "index_fd()" case
(git-update-index), so that would need to be passed around, but this
actually works fine.
NOTE NOTE NOTE! The "is_binary()" heuristics are totally made-up by yours
truly. I will not guarantee that they work at all reasonable. Caveat
emptor. But it _is_ simple, and it _is_ safe, since it's all off by
default.
The patch is pretty simple - the biggest part is the new "convert.c" file,
but even that is really just basic stuff that anybody can write in
"Teaching C 101" as a final project for their first class in programming.
Not to say that it's bug-free, of course - but at least we're not talking
about rocket surgery here.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-13 19:07:23 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Convert blobs to git internal format
|
|
|
|
*/
|
2008-08-03 04:39:16 +00:00
|
|
|
if ((type == OBJ_BLOB) && path) {
|
2008-10-09 19:12:12 +00:00
|
|
|
struct strbuf nbuf = STRBUF_INIT;
|
safecrlf: Add mechanism to warn about irreversible crlf conversions
CRLF conversion bears a slight chance of corrupting data.
autocrlf=true will convert CRLF to LF during commit and LF to
CRLF during checkout. A file that contains a mixture of LF and
CRLF before the commit cannot be recreated by git. For text
files this is the right thing to do: it corrects line endings
such that we have only LF line endings in the repository.
But for binary files that are accidentally classified as text the
conversion can corrupt data.
If you recognize such corruption early you can easily fix it by
setting the conversion type explicitly in .gitattributes. Right
after committing you still have the original file in your work
tree and this file is not yet corrupted. You can explicitly tell
git that this file is binary and git will handle the file
appropriately.
Unfortunately, the desired effect of cleaning up text files with
mixed line endings and the undesired effect of corrupting binary
files cannot be distinguished. In both cases CRLFs are removed
in an irreversible way. For text files this is the right thing
to do because CRLFs are line endings, while for binary files
converting CRLFs corrupts data.
This patch adds a mechanism that can either warn the user about
an irreversible conversion or can even refuse to convert. The
mechanism is controlled by the variable core.safecrlf, with the
following values:
- false: disable safecrlf mechanism
- warn: warn about irreversible conversions
- true: refuse irreversible conversions
The default is to warn. Users are only affected by this default
if core.autocrlf is set. But the current default of git is to
leave core.autocrlf unset, so users will not see warnings unless
they deliberately chose to activate the autocrlf mechanism.
The safecrlf mechanism's details depend on the git command. The
general principles when safecrlf is active (not false) are:
- we warn/error out if files in the work tree can modified in an
irreversible way without giving the user a chance to backup the
original file.
- for read-only operations that do not modify files in the work tree
we do not not print annoying warnings.
There are exceptions. Even though...
- "git add" itself does not touch the files in the work tree, the
next checkout would, so the safety triggers;
- "git apply" to update a text file with a patch does touch the files
in the work tree, but the operation is about text files and CRLF
conversion is about fixing the line ending inconsistencies, so the
safety does not trigger;
- "git diff" itself does not touch the files in the work tree, it is
often run to inspect the changes you intend to next "git add". To
catch potential problems early, safety triggers.
The concept of a safety check was originally proposed in a similar
way by Linus Torvalds. Thanks to Dimitry Potapov for insisting
on getting the naked LF/autocrlf=true case right.
Signed-off-by: Steffen Prohaska <prohaska@zib.de>
2008-02-06 11:25:58 +00:00
|
|
|
if (convert_to_git(path, buf, size, &nbuf,
|
2011-11-15 16:59:39 +00:00
|
|
|
write_object ? safe_crlf : SAFE_CRLF_FALSE)) {
|
2007-09-27 10:58:23 +00:00
|
|
|
buf = strbuf_detach(&nbuf, &size);
|
Lazy man's auto-CRLF
It currently does NOT know about file attributes, so it does its
conversion purely based on content. Maybe that is more in the "git
philosophy" anyway, since content is king, but I think we should try to do
the file attributes to turn it off on demand.
Anyway, BY DEFAULT it is off regardless, because it requires a
[core]
AutoCRLF = true
in your config file to be enabled. We could make that the default for
Windows, of course, the same way we do some other things (filemode etc).
But you can actually enable it on UNIX, and it will cause:
- "git update-index" will write blobs without CRLF
- "git diff" will diff working tree files without CRLF
- "git checkout" will write files to the working tree _with_ CRLF
and things work fine.
Funnily, it actually shows an odd file in git itself:
git clone -n git test-crlf
cd test-crlf
git config core.autocrlf true
git checkout
git diff
shows a diff for "Documentation/docbook-xsl.css". Why? Because we have
actually checked in that file *with* CRLF! So when "core.autocrlf" is
true, we'll always generate a *different* hash for it in the index,
because the index hash will be for the content _without_ CRLF.
Is this complete? I dunno. It seems to work for me. It doesn't use the
filename at all right now, and that's probably a deficiency (we could
certainly make the "is_binary()" heuristics also take standard filename
heuristics into account).
I don't pass in the filename at all for the "index_fd()" case
(git-update-index), so that would need to be passed around, but this
actually works fine.
NOTE NOTE NOTE! The "is_binary()" heuristics are totally made-up by yours
truly. I will not guarantee that they work at all reasonable. Caveat
emptor. But it _is_ simple, and it _is_ safe, since it's all off by
default.
The patch is pretty simple - the biggest part is the new "convert.c" file,
but even that is really just basic stuff that anybody can write in
"Teaching C 101" as a final project for their first class in programming.
Not to say that it's bug-free, of course - but at least we're not talking
about rocket surgery here.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-13 19:07:23 +00:00
|
|
|
re_allocated = 1;
|
|
|
|
}
|
|
|
|
}
|
2011-05-08 08:47:33 +00:00
|
|
|
if (flags & HASH_FORMAT_CHECK) {
|
2011-02-05 10:52:21 +00:00
|
|
|
if (type == OBJ_TREE)
|
|
|
|
check_tree(buf, size);
|
|
|
|
if (type == OBJ_COMMIT)
|
|
|
|
check_commit(buf, size);
|
|
|
|
if (type == OBJ_TAG)
|
|
|
|
check_tag(buf, size);
|
|
|
|
}
|
Lazy man's auto-CRLF
It currently does NOT know about file attributes, so it does its
conversion purely based on content. Maybe that is more in the "git
philosophy" anyway, since content is king, but I think we should try to do
the file attributes to turn it off on demand.
Anyway, BY DEFAULT it is off regardless, because it requires a
[core]
AutoCRLF = true
in your config file to be enabled. We could make that the default for
Windows, of course, the same way we do some other things (filemode etc).
But you can actually enable it on UNIX, and it will cause:
- "git update-index" will write blobs without CRLF
- "git diff" will diff working tree files without CRLF
- "git checkout" will write files to the working tree _with_ CRLF
and things work fine.
Funnily, it actually shows an odd file in git itself:
git clone -n git test-crlf
cd test-crlf
git config core.autocrlf true
git checkout
git diff
shows a diff for "Documentation/docbook-xsl.css". Why? Because we have
actually checked in that file *with* CRLF! So when "core.autocrlf" is
true, we'll always generate a *different* hash for it in the index,
because the index hash will be for the content _without_ CRLF.
Is this complete? I dunno. It seems to work for me. It doesn't use the
filename at all right now, and that's probably a deficiency (we could
certainly make the "is_binary()" heuristics also take standard filename
heuristics into account).
I don't pass in the filename at all for the "index_fd()" case
(git-update-index), so that would need to be passed around, but this
actually works fine.
NOTE NOTE NOTE! The "is_binary()" heuristics are totally made-up by yours
truly. I will not guarantee that they work at all reasonable. Caveat
emptor. But it _is_ simple, and it _is_ safe, since it's all off by
default.
The patch is pretty simple - the biggest part is the new "convert.c" file,
but even that is really just basic stuff that anybody can write in
"Teaching C 101" as a final project for their first class in programming.
Not to say that it's bug-free, of course - but at least we're not talking
about rocket surgery here.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-13 19:07:23 +00:00
|
|
|
|
2005-07-08 23:51:55 +00:00
|
|
|
if (write_object)
|
2007-02-28 19:45:56 +00:00
|
|
|
ret = write_sha1_file(buf, size, typename(type), sha1);
|
2006-10-14 10:45:36 +00:00
|
|
|
else
|
2007-02-28 19:45:56 +00:00
|
|
|
ret = hash_sha1_file(buf, size, typename(type), sha1);
|
2008-08-03 04:39:16 +00:00
|
|
|
if (re_allocated)
|
Lazy man's auto-CRLF
It currently does NOT know about file attributes, so it does its
conversion purely based on content. Maybe that is more in the "git
philosophy" anyway, since content is king, but I think we should try to do
the file attributes to turn it off on demand.
Anyway, BY DEFAULT it is off regardless, because it requires a
[core]
AutoCRLF = true
in your config file to be enabled. We could make that the default for
Windows, of course, the same way we do some other things (filemode etc).
But you can actually enable it on UNIX, and it will cause:
- "git update-index" will write blobs without CRLF
- "git diff" will diff working tree files without CRLF
- "git checkout" will write files to the working tree _with_ CRLF
and things work fine.
Funnily, it actually shows an odd file in git itself:
git clone -n git test-crlf
cd test-crlf
git config core.autocrlf true
git checkout
git diff
shows a diff for "Documentation/docbook-xsl.css". Why? Because we have
actually checked in that file *with* CRLF! So when "core.autocrlf" is
true, we'll always generate a *different* hash for it in the index,
because the index hash will be for the content _without_ CRLF.
Is this complete? I dunno. It seems to work for me. It doesn't use the
filename at all right now, and that's probably a deficiency (we could
certainly make the "is_binary()" heuristics also take standard filename
heuristics into account).
I don't pass in the filename at all for the "index_fd()" case
(git-update-index), so that would need to be passed around, but this
actually works fine.
NOTE NOTE NOTE! The "is_binary()" heuristics are totally made-up by yours
truly. I will not guarantee that they work at all reasonable. Caveat
emptor. But it _is_ simple, and it _is_ safe, since it's all off by
default.
The patch is pretty simple - the biggest part is the new "convert.c" file,
but even that is really just basic stuff that anybody can write in
"Teaching C 101" as a final project for their first class in programming.
Not to say that it's bug-free, of course - but at least we're not talking
about rocket surgery here.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-02-13 19:07:23 +00:00
|
|
|
free(buf);
|
2008-08-03 04:39:16 +00:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2014-08-26 15:23:25 +00:00
|
|
|
static int index_stream_convert_blob(unsigned char *sha1, int fd,
|
|
|
|
const char *path, unsigned flags)
|
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
const int write_object = flags & HASH_WRITE_OBJECT;
|
|
|
|
struct strbuf sbuf = STRBUF_INIT;
|
|
|
|
|
|
|
|
assert(path);
|
|
|
|
assert(would_convert_to_git_filter_fd(path));
|
|
|
|
|
|
|
|
convert_to_git_filter_fd(path, fd, &sbuf,
|
|
|
|
write_object ? safe_crlf : SAFE_CRLF_FALSE);
|
|
|
|
|
|
|
|
if (write_object)
|
|
|
|
ret = write_sha1_file(sbuf.buf, sbuf.len, typename(OBJ_BLOB),
|
|
|
|
sha1);
|
|
|
|
else
|
|
|
|
ret = hash_sha1_file(sbuf.buf, sbuf.len, typename(OBJ_BLOB),
|
|
|
|
sha1);
|
|
|
|
strbuf_release(&sbuf);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2011-05-08 08:47:34 +00:00
|
|
|
static int index_pipe(unsigned char *sha1, int fd, enum object_type type,
|
|
|
|
const char *path, unsigned flags)
|
|
|
|
{
|
|
|
|
struct strbuf sbuf = STRBUF_INIT;
|
|
|
|
int ret;
|
|
|
|
|
|
|
|
if (strbuf_read(&sbuf, fd, 4096) >= 0)
|
|
|
|
ret = index_mem(sha1, sbuf.buf, sbuf.len, type, path, flags);
|
|
|
|
else
|
|
|
|
ret = -1;
|
|
|
|
strbuf_release(&sbuf);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2010-02-21 06:32:19 +00:00
|
|
|
#define SMALL_FILE_SIZE (32*1024)
|
|
|
|
|
2011-05-08 08:47:34 +00:00
|
|
|
static int index_core(unsigned char *sha1, int fd, size_t size,
|
|
|
|
enum object_type type, const char *path,
|
|
|
|
unsigned flags)
|
2008-08-03 04:39:16 +00:00
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
|
2011-05-08 08:47:34 +00:00
|
|
|
if (!size) {
|
sha1_file: pass empty buffer to index empty file
`git add` of an empty file with a filter pops complaints from
`copy_fd` about a bad file descriptor.
This traces back to these lines in sha1_file.c:index_core:
if (!size) {
ret = index_mem(sha1, NULL, size, type, path, flags);
The problem here is that content to be added to the index can be
supplied from an fd, or from a memory buffer, or from a pathname. This
call is supplying a NULL buffer pointer and a zero size.
Downstream logic takes the complete absence of a buffer to mean the
data is to be found elsewhere -- for instance, these, from convert.c:
if (params->src) {
write_err = (write_in_full(child_process.in, params->src, params->size) < 0);
} else {
write_err = copy_fd(params->fd, child_process.in);
}
~If there's a buffer, write from that, otherwise the data must be coming
from an open fd.~
Perfectly reasonable logic in a routine that's going to write from
either a buffer or an fd.
So change `index_core` to supply an empty buffer when indexing an empty
file.
There's a patch out there that instead changes the logic quoted above to
take a `-1` fd to mean "use the buffer", but it seems to me that the
distinction between a missing buffer and an empty one carries intrinsic
semantics, where the logic change is adapting the code to handle
incorrect arguments.
Signed-off-by: Jim Hill <gjthill@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-05-18 00:41:45 +00:00
|
|
|
ret = index_mem(sha1, "", size, type, path, flags);
|
2010-02-21 06:32:19 +00:00
|
|
|
} else if (size <= SMALL_FILE_SIZE) {
|
|
|
|
char *buf = xmalloc(size);
|
|
|
|
if (size == read_in_full(fd, buf, size))
|
2011-05-08 08:47:33 +00:00
|
|
|
ret = index_mem(sha1, buf, size, type, path, flags);
|
2010-02-21 06:32:19 +00:00
|
|
|
else
|
2016-05-08 09:47:56 +00:00
|
|
|
ret = error_errno("short read");
|
2010-02-21 06:32:19 +00:00
|
|
|
free(buf);
|
2010-05-10 21:38:17 +00:00
|
|
|
} else {
|
2008-08-03 04:39:16 +00:00
|
|
|
void *buf = xmmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0);
|
2011-05-08 08:47:33 +00:00
|
|
|
ret = index_mem(sha1, buf, size, type, path, flags);
|
2005-05-03 18:46:16 +00:00
|
|
|
munmap(buf, size);
|
2010-05-10 21:38:17 +00:00
|
|
|
}
|
2011-05-08 08:47:34 +00:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2011-05-08 08:47:35 +00:00
|
|
|
/*
|
2011-10-28 21:48:40 +00:00
|
|
|
* This creates one packfile per large blob unless bulk-checkin
|
|
|
|
* machinery is "plugged".
|
2011-05-08 08:47:35 +00:00
|
|
|
*
|
|
|
|
* This also bypasses the usual "convert-to-git" dance, and that is on
|
|
|
|
* purpose. We could write a streaming version of the converting
|
|
|
|
* functions and insert that before feeding the data to fast-import
|
do not stream large files to pack when filters are in use
Because git's object format requires us to specify the
number of bytes in the object in its header, we must know
the size before streaming a blob into the object database.
This is not a problem when adding a regular file, as we can
get the size from stat(). However, when filters are in use
(such as autocrlf, or the ident, filter, or eol
gitattributes), we have no idea what the ultimate size will
be.
The current code just punts on the whole issue and ignores
filter configuration entirely for files larger than
core.bigfilethreshold. This can generate confusing results
if you use filters for large binary files, as the filter
will suddenly stop working as the file goes over a certain
size. Rather than try to handle unknown input sizes with
streaming, this patch just turns off the streaming
optimization when filters are in use.
This has a slight performance regression in a very specific
case: if you have autocrlf on, but no gitattributes, a large
binary file will avoid the streaming code path because we
don't know beforehand whether it will need conversion or
not. But if you are handling large binary files, you should
be marking them as such via attributes (or at least not
using autocrlf, and instead marking your text files as
such). And the flip side is that if you have a large
_non_-binary file, there is a correctness improvement;
before we did not apply the conversion at all.
The first half of the new t1051 script covers these failures
on input. The second half tests the matching output code
paths. These already work correctly, and do not need any
adjustment.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-02-24 22:10:17 +00:00
|
|
|
* (or equivalent in-core API described above). However, that is
|
|
|
|
* somewhat complicated, as we do not know the size of the filter
|
|
|
|
* result, which we need to know beforehand when writing a git object.
|
|
|
|
* Since the primary motivation for trying to stream from the working
|
|
|
|
* tree file and to avoid mmaping it in core is to deal with large
|
|
|
|
* binary blobs, they generally do not want to get any conversion, and
|
|
|
|
* callers should avoid this code path when filters are requested.
|
2011-05-08 08:47:35 +00:00
|
|
|
*/
|
|
|
|
static int index_stream(unsigned char *sha1, int fd, size_t size,
|
|
|
|
enum object_type type, const char *path,
|
|
|
|
unsigned flags)
|
|
|
|
{
|
2011-10-28 21:48:40 +00:00
|
|
|
return index_bulk_checkin(sha1, fd, size, type, path, flags);
|
2011-05-08 08:47:35 +00:00
|
|
|
}
|
|
|
|
|
2011-05-08 08:47:34 +00:00
|
|
|
int index_fd(unsigned char *sha1, int fd, struct stat *st,
|
|
|
|
enum object_type type, const char *path, unsigned flags)
|
|
|
|
{
|
|
|
|
int ret;
|
|
|
|
|
2014-09-21 10:03:26 +00:00
|
|
|
/*
|
|
|
|
* Call xsize_t() only when needed to avoid potentially unnecessary
|
|
|
|
* die() for large files.
|
|
|
|
*/
|
2014-08-26 15:23:25 +00:00
|
|
|
if (type == OBJ_BLOB && path && would_convert_to_git_filter_fd(path))
|
|
|
|
ret = index_stream_convert_blob(sha1, fd, path, flags);
|
|
|
|
else if (!S_ISREG(st->st_mode))
|
2011-05-08 08:47:34 +00:00
|
|
|
ret = index_pipe(sha1, fd, type, path, flags);
|
2014-09-21 10:03:26 +00:00
|
|
|
else if (st->st_size <= big_file_threshold || type != OBJ_BLOB ||
|
2014-08-21 16:05:08 +00:00
|
|
|
(path && would_convert_to_git(path)))
|
2014-09-21 10:03:26 +00:00
|
|
|
ret = index_core(sha1, fd, xsize_t(st->st_size), type, path,
|
|
|
|
flags);
|
2011-05-08 08:47:35 +00:00
|
|
|
else
|
2014-09-21 10:03:26 +00:00
|
|
|
ret = index_stream(sha1, fd, xsize_t(st->st_size), type, path,
|
|
|
|
flags);
|
2008-08-03 04:39:16 +00:00
|
|
|
close(fd);
|
2005-05-03 18:46:16 +00:00
|
|
|
return ret;
|
2005-05-02 06:45:49 +00:00
|
|
|
}
|
2005-10-07 10:42:00 +00:00
|
|
|
|
2011-05-08 08:47:33 +00:00
|
|
|
int index_path(unsigned char *sha1, const char *path, struct stat *st, unsigned flags)
|
2005-10-07 10:42:00 +00:00
|
|
|
{
|
|
|
|
int fd;
|
2008-12-17 17:51:53 +00:00
|
|
|
struct strbuf sb = STRBUF_INIT;
|
2005-10-07 10:42:00 +00:00
|
|
|
|
|
|
|
switch (st->st_mode & S_IFMT) {
|
|
|
|
case S_IFREG:
|
|
|
|
fd = open(path, O_RDONLY);
|
|
|
|
if (fd < 0)
|
2016-05-08 09:47:56 +00:00
|
|
|
return error_errno("open(\"%s\")", path);
|
2011-05-08 08:47:33 +00:00
|
|
|
if (index_fd(sha1, fd, st, OBJ_BLOB, path, flags) < 0)
|
2005-10-07 10:42:00 +00:00
|
|
|
return error("%s: failed to insert into database",
|
|
|
|
path);
|
|
|
|
break;
|
|
|
|
case S_IFLNK:
|
2016-05-08 09:47:56 +00:00
|
|
|
if (strbuf_readlink(&sb, path, st->st_size))
|
|
|
|
return error_errno("readlink(\"%s\")", path);
|
2011-05-08 08:47:33 +00:00
|
|
|
if (!(flags & HASH_WRITE_OBJECT))
|
2008-12-17 17:51:53 +00:00
|
|
|
hash_sha1_file(sb.buf, sb.len, blob_type, sha1);
|
|
|
|
else if (write_sha1_file(sb.buf, sb.len, blob_type, sha1))
|
2005-10-07 10:42:00 +00:00
|
|
|
return error("%s: failed to insert into database",
|
|
|
|
path);
|
2008-12-17 17:51:53 +00:00
|
|
|
strbuf_release(&sb);
|
2005-10-07 10:42:00 +00:00
|
|
|
break;
|
2007-04-10 04:20:29 +00:00
|
|
|
case S_IFDIR:
|
|
|
|
return resolve_gitlink_ref(path, "HEAD", sha1);
|
2005-10-07 10:42:00 +00:00
|
|
|
default:
|
|
|
|
return error("%s: unsupported file type", path);
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
2007-01-23 05:55:18 +00:00
|
|
|
|
|
|
|
int read_pack_header(int fd, struct pack_header *header)
|
|
|
|
{
|
2008-05-03 13:27:26 +00:00
|
|
|
if (read_in_full(fd, header, sizeof(*header)) < sizeof(*header))
|
|
|
|
/* "eof before pack header was fully read" */
|
|
|
|
return PH_ERROR_EOF;
|
|
|
|
|
2007-01-23 05:55:18 +00:00
|
|
|
if (header->hdr_signature != htonl(PACK_SIGNATURE))
|
|
|
|
/* "protocol error (pack signature mismatch detected)" */
|
|
|
|
return PH_ERROR_PACK_SIGNATURE;
|
|
|
|
if (!pack_version_ok(header->hdr_version))
|
|
|
|
/* "protocol error (pack version unsupported)" */
|
|
|
|
return PH_ERROR_PROTOCOL;
|
|
|
|
return 0;
|
|
|
|
}
|
make commit_tree a library function
Until now, this has been part of the commit-tree builtin.
However, it is already used by other builtins (like commit,
merge, and notes), and it would be useful to access it from
library code.
The check_valid helper has to come along, too, but is given
a more library-ish name of "assert_sha1_type".
Otherwise, the code is unchanged. There are still a few
rough edges for a library function, like printing the utf8
warning to stderr, but we can address those if and when they
come up as inappropriate.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-04-02 00:05:23 +00:00
|
|
|
|
|
|
|
void assert_sha1_type(const unsigned char *sha1, enum object_type expect)
|
|
|
|
{
|
|
|
|
enum object_type type = sha1_object_info(sha1, NULL);
|
|
|
|
if (type < 0)
|
|
|
|
die("%s is not a valid object", sha1_to_hex(sha1));
|
|
|
|
if (type != expect)
|
|
|
|
die("%s is not a valid '%s' object", sha1_to_hex(sha1),
|
|
|
|
typename(expect));
|
|
|
|
}
|
2014-10-15 22:38:55 +00:00
|
|
|
|
|
|
|
static int for_each_file_in_obj_subdir(int subdir_nr,
|
|
|
|
struct strbuf *path,
|
|
|
|
each_loose_object_fn obj_cb,
|
|
|
|
each_loose_cruft_fn cruft_cb,
|
|
|
|
each_loose_subdir_fn subdir_cb,
|
|
|
|
void *data)
|
|
|
|
{
|
|
|
|
size_t baselen = path->len;
|
|
|
|
DIR *dir = opendir(path->buf);
|
|
|
|
struct dirent *de;
|
|
|
|
int r = 0;
|
|
|
|
|
|
|
|
if (!dir) {
|
|
|
|
if (errno == ENOENT)
|
|
|
|
return 0;
|
2016-05-08 09:47:56 +00:00
|
|
|
return error_errno("unable to open %s", path->buf);
|
2014-10-15 22:38:55 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
while ((de = readdir(dir))) {
|
|
|
|
if (is_dot_or_dotdot(de->d_name))
|
|
|
|
continue;
|
|
|
|
|
|
|
|
strbuf_setlen(path, baselen);
|
|
|
|
strbuf_addf(path, "/%s", de->d_name);
|
|
|
|
|
2017-02-21 23:47:35 +00:00
|
|
|
if (strlen(de->d_name) == GIT_SHA1_HEXSZ - 2) {
|
2017-03-26 16:01:24 +00:00
|
|
|
char hex[GIT_MAX_HEXSZ+1];
|
2017-02-21 23:47:35 +00:00
|
|
|
struct object_id oid;
|
2014-10-15 22:38:55 +00:00
|
|
|
|
|
|
|
snprintf(hex, sizeof(hex), "%02x%s",
|
|
|
|
subdir_nr, de->d_name);
|
2017-02-21 23:47:35 +00:00
|
|
|
if (!get_oid_hex(hex, &oid)) {
|
2014-10-15 22:38:55 +00:00
|
|
|
if (obj_cb) {
|
2017-02-21 23:47:35 +00:00
|
|
|
r = obj_cb(&oid, path->buf, data);
|
2014-10-15 22:38:55 +00:00
|
|
|
if (r)
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (cruft_cb) {
|
|
|
|
r = cruft_cb(de->d_name, path->buf, data);
|
|
|
|
if (r)
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
2015-08-12 17:43:01 +00:00
|
|
|
closedir(dir);
|
2014-10-15 22:38:55 +00:00
|
|
|
|
2015-08-12 17:43:01 +00:00
|
|
|
strbuf_setlen(path, baselen);
|
2014-10-15 22:38:55 +00:00
|
|
|
if (!r && subdir_cb)
|
|
|
|
r = subdir_cb(subdir_nr, path->buf, data);
|
|
|
|
|
|
|
|
return r;
|
|
|
|
}
|
|
|
|
|
2015-02-09 01:13:22 +00:00
|
|
|
int for_each_loose_file_in_objdir_buf(struct strbuf *path,
|
2014-10-15 22:38:55 +00:00
|
|
|
each_loose_object_fn obj_cb,
|
|
|
|
each_loose_cruft_fn cruft_cb,
|
|
|
|
each_loose_subdir_fn subdir_cb,
|
|
|
|
void *data)
|
|
|
|
{
|
2015-02-09 01:13:22 +00:00
|
|
|
size_t baselen = path->len;
|
2014-10-15 22:38:55 +00:00
|
|
|
int r = 0;
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = 0; i < 256; i++) {
|
2015-02-09 01:13:22 +00:00
|
|
|
strbuf_addf(path, "/%02x", i);
|
|
|
|
r = for_each_file_in_obj_subdir(i, path, obj_cb, cruft_cb,
|
2014-10-15 22:38:55 +00:00
|
|
|
subdir_cb, data);
|
2015-02-09 01:13:22 +00:00
|
|
|
strbuf_setlen(path, baselen);
|
2014-10-15 22:38:55 +00:00
|
|
|
if (r)
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
2015-02-09 01:13:22 +00:00
|
|
|
return r;
|
|
|
|
}
|
|
|
|
|
|
|
|
int for_each_loose_file_in_objdir(const char *path,
|
|
|
|
each_loose_object_fn obj_cb,
|
|
|
|
each_loose_cruft_fn cruft_cb,
|
|
|
|
each_loose_subdir_fn subdir_cb,
|
|
|
|
void *data)
|
|
|
|
{
|
|
|
|
struct strbuf buf = STRBUF_INIT;
|
|
|
|
int r;
|
|
|
|
|
|
|
|
strbuf_addstr(&buf, path);
|
|
|
|
r = for_each_loose_file_in_objdir_buf(&buf, obj_cb, cruft_cb,
|
|
|
|
subdir_cb, data);
|
2014-10-15 22:38:55 +00:00
|
|
|
strbuf_release(&buf);
|
2015-02-09 01:13:22 +00:00
|
|
|
|
2014-10-15 22:38:55 +00:00
|
|
|
return r;
|
|
|
|
}
|
2014-10-15 22:41:21 +00:00
|
|
|
|
|
|
|
struct loose_alt_odb_data {
|
|
|
|
each_loose_object_fn *cb;
|
|
|
|
void *data;
|
|
|
|
};
|
|
|
|
|
|
|
|
static int loose_from_alt_odb(struct alternate_object_database *alt,
|
|
|
|
void *vdata)
|
|
|
|
{
|
|
|
|
struct loose_alt_odb_data *data = vdata;
|
2015-02-09 01:15:39 +00:00
|
|
|
struct strbuf buf = STRBUF_INIT;
|
|
|
|
int r;
|
|
|
|
|
2016-10-03 20:35:51 +00:00
|
|
|
strbuf_addstr(&buf, alt->path);
|
2015-02-09 01:15:39 +00:00
|
|
|
r = for_each_loose_file_in_objdir_buf(&buf,
|
|
|
|
data->cb, NULL, NULL,
|
|
|
|
data->data);
|
|
|
|
strbuf_release(&buf);
|
|
|
|
return r;
|
2014-10-15 22:41:21 +00:00
|
|
|
}
|
|
|
|
|
reachable: only mark local objects as recent
When pruning and repacking a repository that has an
alternate object store configured, we may traverse a large
number of objects in the alternate. This serves no purpose,
and may be expensive to do. A longer explanation is below.
Commits d3038d2 and abcb865 taught prune and pack-objects
(respectively) to treat "recent" objects as tips for
reachability, so that we keep whole chunks of history. They
built on the object traversal in 660c889 (sha1_file: add
for_each iterators for loose and packed objects,
2014-10-15), which covers both local and alternate objects.
In both cases, covering alternate objects is unnecessary, as
both commands can only drop objects from the local
repository. In the case of prune, we traverse only the local
object directory. And in the case of repacking, while we may
or may not include local objects in our pack, we will never
reach into the alternate with "repack -d". The "-l" option
is only a question of whether we are migrating objects from
the alternate into our repository, or leaving them
untouched.
It is possible that we may drop an object that is depended
upon by another object in the alternate. For example,
imagine two repositories, A and B, with A pointing to B as
an alternate. Now imagine a commit that is in B which
references a tree that is only in A. Traversing from recent
objects in B might prevent A from dropping that tree. But
this case isn't worth covering. Repo B should take
responsibility for its own objects. It would never have had
the commit in the first place if it did not also have the
tree, and assuming it is using the same "keep recent chunks
of history" scheme, then it would itself keep the tree, as
well.
So checking the alternate objects is not worth doing, and
come with a significant performance impact. In both cases,
we skip any recent objects that have already been marked
SEEN (i.e., that we know are already reachable for prune, or
included in the pack for a repack). So there is a slight
waste of time in opening the alternate packs at all, only to
notice that we have already considered each object. But much
worse, the alternate repository may have a large number of
objects that are not reachable from the local repository at
all, and we end up adding them to the traversal.
We can fix this by considering only local unseen objects.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-03-27 11:32:41 +00:00
|
|
|
int for_each_loose_object(each_loose_object_fn cb, void *data, unsigned flags)
|
2014-10-15 22:41:21 +00:00
|
|
|
{
|
|
|
|
struct loose_alt_odb_data alt;
|
|
|
|
int r;
|
|
|
|
|
|
|
|
r = for_each_loose_file_in_objdir(get_object_directory(),
|
|
|
|
cb, NULL, NULL, data);
|
|
|
|
if (r)
|
|
|
|
return r;
|
|
|
|
|
reachable: only mark local objects as recent
When pruning and repacking a repository that has an
alternate object store configured, we may traverse a large
number of objects in the alternate. This serves no purpose,
and may be expensive to do. A longer explanation is below.
Commits d3038d2 and abcb865 taught prune and pack-objects
(respectively) to treat "recent" objects as tips for
reachability, so that we keep whole chunks of history. They
built on the object traversal in 660c889 (sha1_file: add
for_each iterators for loose and packed objects,
2014-10-15), which covers both local and alternate objects.
In both cases, covering alternate objects is unnecessary, as
both commands can only drop objects from the local
repository. In the case of prune, we traverse only the local
object directory. And in the case of repacking, while we may
or may not include local objects in our pack, we will never
reach into the alternate with "repack -d". The "-l" option
is only a question of whether we are migrating objects from
the alternate into our repository, or leaving them
untouched.
It is possible that we may drop an object that is depended
upon by another object in the alternate. For example,
imagine two repositories, A and B, with A pointing to B as
an alternate. Now imagine a commit that is in B which
references a tree that is only in A. Traversing from recent
objects in B might prevent A from dropping that tree. But
this case isn't worth covering. Repo B should take
responsibility for its own objects. It would never have had
the commit in the first place if it did not also have the
tree, and assuming it is using the same "keep recent chunks
of history" scheme, then it would itself keep the tree, as
well.
So checking the alternate objects is not worth doing, and
come with a significant performance impact. In both cases,
we skip any recent objects that have already been marked
SEEN (i.e., that we know are already reachable for prune, or
included in the pack for a repack). So there is a slight
waste of time in opening the alternate packs at all, only to
notice that we have already considered each object. But much
worse, the alternate repository may have a large number of
objects that are not reachable from the local repository at
all, and we end up adding them to the traversal.
We can fix this by considering only local unseen objects.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-03-27 11:32:41 +00:00
|
|
|
if (flags & FOR_EACH_OBJECT_LOCAL_ONLY)
|
|
|
|
return 0;
|
|
|
|
|
2014-10-15 22:41:21 +00:00
|
|
|
alt.cb = cb;
|
|
|
|
alt.data = data;
|
|
|
|
return foreach_alt_odb(loose_from_alt_odb, &alt);
|
|
|
|
}
|
|
|
|
|
|
|
|
static int for_each_object_in_pack(struct packed_git *p, each_packed_object_fn cb, void *data)
|
|
|
|
{
|
|
|
|
uint32_t i;
|
|
|
|
int r = 0;
|
|
|
|
|
|
|
|
for (i = 0; i < p->num_objects; i++) {
|
2017-02-21 23:47:34 +00:00
|
|
|
struct object_id oid;
|
2014-10-15 22:41:21 +00:00
|
|
|
|
2017-02-21 23:47:34 +00:00
|
|
|
if (!nth_packed_object_oid(&oid, p, i))
|
2014-10-15 22:41:21 +00:00
|
|
|
return error("unable to get sha1 of object %u in %s",
|
|
|
|
i, p->pack_name);
|
|
|
|
|
2017-02-21 23:47:35 +00:00
|
|
|
r = cb(&oid, p, i, data);
|
2014-10-15 22:41:21 +00:00
|
|
|
if (r)
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
return r;
|
|
|
|
}
|
|
|
|
|
reachable: only mark local objects as recent
When pruning and repacking a repository that has an
alternate object store configured, we may traverse a large
number of objects in the alternate. This serves no purpose,
and may be expensive to do. A longer explanation is below.
Commits d3038d2 and abcb865 taught prune and pack-objects
(respectively) to treat "recent" objects as tips for
reachability, so that we keep whole chunks of history. They
built on the object traversal in 660c889 (sha1_file: add
for_each iterators for loose and packed objects,
2014-10-15), which covers both local and alternate objects.
In both cases, covering alternate objects is unnecessary, as
both commands can only drop objects from the local
repository. In the case of prune, we traverse only the local
object directory. And in the case of repacking, while we may
or may not include local objects in our pack, we will never
reach into the alternate with "repack -d". The "-l" option
is only a question of whether we are migrating objects from
the alternate into our repository, or leaving them
untouched.
It is possible that we may drop an object that is depended
upon by another object in the alternate. For example,
imagine two repositories, A and B, with A pointing to B as
an alternate. Now imagine a commit that is in B which
references a tree that is only in A. Traversing from recent
objects in B might prevent A from dropping that tree. But
this case isn't worth covering. Repo B should take
responsibility for its own objects. It would never have had
the commit in the first place if it did not also have the
tree, and assuming it is using the same "keep recent chunks
of history" scheme, then it would itself keep the tree, as
well.
So checking the alternate objects is not worth doing, and
come with a significant performance impact. In both cases,
we skip any recent objects that have already been marked
SEEN (i.e., that we know are already reachable for prune, or
included in the pack for a repack). So there is a slight
waste of time in opening the alternate packs at all, only to
notice that we have already considered each object. But much
worse, the alternate repository may have a large number of
objects that are not reachable from the local repository at
all, and we end up adding them to the traversal.
We can fix this by considering only local unseen objects.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-03-27 11:32:41 +00:00
|
|
|
int for_each_packed_object(each_packed_object_fn cb, void *data, unsigned flags)
|
2014-10-15 22:41:21 +00:00
|
|
|
{
|
|
|
|
struct packed_git *p;
|
|
|
|
int r = 0;
|
2015-06-22 10:40:50 +00:00
|
|
|
int pack_errors = 0;
|
2014-10-15 22:41:21 +00:00
|
|
|
|
|
|
|
prepare_packed_git();
|
|
|
|
for (p = packed_git; p; p = p->next) {
|
reachable: only mark local objects as recent
When pruning and repacking a repository that has an
alternate object store configured, we may traverse a large
number of objects in the alternate. This serves no purpose,
and may be expensive to do. A longer explanation is below.
Commits d3038d2 and abcb865 taught prune and pack-objects
(respectively) to treat "recent" objects as tips for
reachability, so that we keep whole chunks of history. They
built on the object traversal in 660c889 (sha1_file: add
for_each iterators for loose and packed objects,
2014-10-15), which covers both local and alternate objects.
In both cases, covering alternate objects is unnecessary, as
both commands can only drop objects from the local
repository. In the case of prune, we traverse only the local
object directory. And in the case of repacking, while we may
or may not include local objects in our pack, we will never
reach into the alternate with "repack -d". The "-l" option
is only a question of whether we are migrating objects from
the alternate into our repository, or leaving them
untouched.
It is possible that we may drop an object that is depended
upon by another object in the alternate. For example,
imagine two repositories, A and B, with A pointing to B as
an alternate. Now imagine a commit that is in B which
references a tree that is only in A. Traversing from recent
objects in B might prevent A from dropping that tree. But
this case isn't worth covering. Repo B should take
responsibility for its own objects. It would never have had
the commit in the first place if it did not also have the
tree, and assuming it is using the same "keep recent chunks
of history" scheme, then it would itself keep the tree, as
well.
So checking the alternate objects is not worth doing, and
come with a significant performance impact. In both cases,
we skip any recent objects that have already been marked
SEEN (i.e., that we know are already reachable for prune, or
included in the pack for a repack). So there is a slight
waste of time in opening the alternate packs at all, only to
notice that we have already considered each object. But much
worse, the alternate repository may have a large number of
objects that are not reachable from the local repository at
all, and we end up adding them to the traversal.
We can fix this by considering only local unseen objects.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-03-27 11:32:41 +00:00
|
|
|
if ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)
|
|
|
|
continue;
|
2015-06-22 10:40:50 +00:00
|
|
|
if (open_pack_index(p)) {
|
|
|
|
pack_errors = 1;
|
|
|
|
continue;
|
|
|
|
}
|
2014-10-15 22:41:21 +00:00
|
|
|
r = for_each_object_in_pack(p, cb, data);
|
|
|
|
if (r)
|
|
|
|
break;
|
|
|
|
}
|
2015-06-22 10:40:50 +00:00
|
|
|
return r ? r : pack_errors;
|
2014-10-15 22:41:21 +00:00
|
|
|
}
|
2017-01-13 17:58:16 +00:00
|
|
|
|
|
|
|
static int check_stream_sha1(git_zstream *stream,
|
|
|
|
const char *hdr,
|
|
|
|
unsigned long size,
|
|
|
|
const char *path,
|
|
|
|
const unsigned char *expected_sha1)
|
|
|
|
{
|
|
|
|
git_SHA_CTX c;
|
2017-03-26 16:01:25 +00:00
|
|
|
unsigned char real_sha1[GIT_MAX_RAWSZ];
|
2017-01-13 17:58:16 +00:00
|
|
|
unsigned char buf[4096];
|
|
|
|
unsigned long total_read;
|
|
|
|
int status = Z_OK;
|
|
|
|
|
|
|
|
git_SHA1_Init(&c);
|
|
|
|
git_SHA1_Update(&c, hdr, stream->total_out);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We already read some bytes into hdr, but the ones up to the NUL
|
|
|
|
* do not count against the object's content size.
|
|
|
|
*/
|
|
|
|
total_read = stream->total_out - strlen(hdr) - 1;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This size comparison must be "<=" to read the final zlib packets;
|
|
|
|
* see the comment in unpack_sha1_rest for details.
|
|
|
|
*/
|
|
|
|
while (total_read <= size &&
|
|
|
|
(status == Z_OK || status == Z_BUF_ERROR)) {
|
|
|
|
stream->next_out = buf;
|
|
|
|
stream->avail_out = sizeof(buf);
|
|
|
|
if (size - total_read < stream->avail_out)
|
|
|
|
stream->avail_out = size - total_read;
|
|
|
|
status = git_inflate(stream, Z_FINISH);
|
|
|
|
git_SHA1_Update(&c, buf, stream->next_out - buf);
|
|
|
|
total_read += stream->next_out - buf;
|
|
|
|
}
|
|
|
|
git_inflate_end(stream);
|
|
|
|
|
|
|
|
if (status != Z_STREAM_END) {
|
|
|
|
error("corrupt loose object '%s'", sha1_to_hex(expected_sha1));
|
|
|
|
return -1;
|
|
|
|
}
|
2017-01-13 18:00:25 +00:00
|
|
|
if (stream->avail_in) {
|
|
|
|
error("garbage at end of loose object '%s'",
|
|
|
|
sha1_to_hex(expected_sha1));
|
|
|
|
return -1;
|
|
|
|
}
|
2017-01-13 17:58:16 +00:00
|
|
|
|
|
|
|
git_SHA1_Final(real_sha1, &c);
|
|
|
|
if (hashcmp(expected_sha1, real_sha1)) {
|
|
|
|
error("sha1 mismatch for %s (expected %s)", path,
|
|
|
|
sha1_to_hex(expected_sha1));
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
int read_loose_object(const char *path,
|
|
|
|
const unsigned char *expected_sha1,
|
|
|
|
enum object_type *type,
|
|
|
|
unsigned long *size,
|
|
|
|
void **contents)
|
|
|
|
{
|
|
|
|
int ret = -1;
|
|
|
|
int fd = -1;
|
|
|
|
void *map = NULL;
|
|
|
|
unsigned long mapsize;
|
|
|
|
git_zstream stream;
|
|
|
|
char hdr[32];
|
|
|
|
|
|
|
|
*contents = NULL;
|
|
|
|
|
|
|
|
map = map_sha1_file_1(path, NULL, &mapsize);
|
|
|
|
if (!map) {
|
|
|
|
error_errno("unable to mmap %s", path);
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (unpack_sha1_header(&stream, map, mapsize, hdr, sizeof(hdr)) < 0) {
|
|
|
|
error("unable to unpack header of %s", path);
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
*type = parse_sha1_header(hdr, size);
|
|
|
|
if (*type < 0) {
|
|
|
|
error("unable to parse header of %s", path);
|
|
|
|
git_inflate_end(&stream);
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (*type == OBJ_BLOB) {
|
|
|
|
if (check_stream_sha1(&stream, hdr, *size, path, expected_sha1) < 0)
|
|
|
|
goto out;
|
|
|
|
} else {
|
|
|
|
*contents = unpack_sha1_rest(&stream, hdr, *size, expected_sha1);
|
|
|
|
if (!*contents) {
|
|
|
|
error("unable to unpack contents of %s", path);
|
|
|
|
git_inflate_end(&stream);
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
if (check_sha1_signature(expected_sha1, *contents,
|
|
|
|
*size, typename(*type))) {
|
|
|
|
error("sha1 mismatch for %s (expected %s)", path,
|
|
|
|
sha1_to_hex(expected_sha1));
|
|
|
|
free(*contents);
|
|
|
|
goto out;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
ret = 0; /* everything checks out */
|
|
|
|
|
|
|
|
out:
|
|
|
|
if (map)
|
|
|
|
munmap(map, mapsize);
|
|
|
|
if (fd >= 0)
|
|
|
|
close(fd);
|
|
|
|
return ret;
|
|
|
|
}
|