mirror of
https://github.com/git/git
synced 2024-09-29 21:27:13 +00:00
Merge branch 'tb/incremental-midx-part-1'
Incremental updates of multi-pack index files. * tb/incremental-midx-part-1: midx: implement support for writing incremental MIDX chains t/t5313-pack-bounds-checks.sh: prepare for sub-directories t: retire 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' midx: implement verification support for incremental MIDXs midx: support reading incremental MIDX chains midx: teach `midx_fanout_add_midx_fanout()` about incremental MIDXs midx: teach `midx_preferred_pack()` about incremental MIDXs midx: teach `midx_contains_pack()` about incremental MIDXs midx: remove unused `midx_locate_pack()` midx: teach `fill_midx_entry()` about incremental MIDXs midx: teach `nth_midxed_offset()` about incremental MIDXs midx: teach `bsearch_midx()` about incremental MIDXs midx: introduce `bsearch_one_midx()` midx: teach `nth_bitmapped_pack()` about incremental MIDXs midx: teach `nth_midxed_object_oid()` about incremental MIDXs midx: teach `prepare_midx_pack()` about incremental MIDXs midx: teach `nth_midxed_pack_int_id()` about incremental MIDXs midx: add new fields for incremental MIDX chains Documentation: describe incremental MIDX format
This commit is contained in:
Junio C Hamano
commit
b9497848df
|
|
@ -64,6 +64,12 @@ The file given at `<path>` is expected to be readable, and can contain
|
|||
duplicates. (If a given OID is given more than once, it is marked as
|
||||
preferred if at least one instance of it begins with the special `+`
|
||||
marker).
|
||||
|
||||
--incremental::
|
||||
Write an incremental MIDX file containing only objects
|
||||
and packs not present in an existing MIDX layer.
|
||||
Migrates non-incremental MIDXs to incremental ones when
|
||||
necessary. Incompatible with `--bitmap`.
|
||||
--
|
||||
|
||||
verify::
|
||||
|
|
@ -74,6 +80,8 @@ expire::
|
|||
have no objects referenced by the MIDX (with the exception of
|
||||
`.keep` packs and cruft packs). Rewrite the MIDX file afterward
|
||||
to remove all references to these pack-files.
|
||||
+
|
||||
NOTE: this mode is incompatible with incremental MIDX files.
|
||||
|
||||
repack::
|
||||
Create a new pack-file containing objects in small pack-files
|
||||
|
|
@ -95,7 +103,8 @@ repack::
|
|||
+
|
||||
If `repack.packKeptObjects` is `false`, then any pack-files with an
|
||||
associated `.keep` file will not be selected for the batch to repack.
|
||||
|
||||
+
|
||||
NOTE: this mode is incompatible with incremental MIDX files.
|
||||
|
||||
EXAMPLES
|
||||
--------
|
||||
|
|
|
|||
|
|
@ -61,6 +61,109 @@ Design Details
|
|||
- The MIDX file format uses a chunk-based approach (similar to the
|
||||
commit-graph file) that allows optional data to be added.
|
||||
|
||||
Incremental multi-pack indexes
|
||||
------------------------------
|
||||
|
||||
As repositories grow in size, it becomes more expensive to write a
|
||||
multi-pack index (MIDX) that includes all packfiles. To accommodate
|
||||
this, the "incremental multi-pack indexes" feature allows for combining
|
||||
a "chain" of multi-pack indexes.
|
||||
|
||||
Each individual component of the chain need only contain a small number
|
||||
of packfiles. Appending to the chain does not invalidate earlier parts
|
||||
of the chain, so repositories can control how much time is spent
|
||||
updating the MIDX chain by determining the number of packs in each layer
|
||||
of the MIDX chain.
|
||||
|
||||
=== Design state
|
||||
|
||||
At present, the incremental multi-pack indexes feature is missing two
|
||||
important components:
|
||||
|
||||
- The ability to rewrite earlier portions of the MIDX chain (i.e., to
|
||||
"compact" some collection of adjacent MIDX layers into a single
|
||||
MIDX). At present the only supported way of shrinking a MIDX chain
|
||||
is to rewrite the entire chain from scratch without the `--split`
|
||||
flag.
|
||||
+
|
||||
There are no fundamental limitations that stand in the way of being able
|
||||
to implement this feature. It is omitted from the initial implementation
|
||||
in order to reduce the complexity, but will be added later.
|
||||
|
||||
- Support for reachability bitmaps. The classic single MIDX
|
||||
implementation does support reachability bitmaps (see the section
|
||||
titled "multi-pack-index reverse indexes" in
|
||||
linkgit:gitformat-pack[5] for more details).
|
||||
+
|
||||
As above, there are no fundamental limitations that stand in the way of
|
||||
extending the incremental MIDX format to support reachability bitmaps.
|
||||
The design below specifically takes this into account, and support for
|
||||
reachability bitmaps will be added in a future patch series. It is
|
||||
omitted from the current implementation for the same reason as above.
|
||||
+
|
||||
In brief, to support reachability bitmaps with the incremental MIDX
|
||||
feature, the concept of the pseudo-pack order is extended across each
|
||||
layer of the incremental MIDX chain to form a concatenated pseudo-pack
|
||||
order. This concatenation takes place in the same order as the chain
|
||||
itself (in other words, the concatenated pseudo-pack order for a chain
|
||||
`{$H1, $H2, $H3}` would be the pseudo-pack order for `$H1`, followed by
|
||||
the pseudo-pack order for `$H2`, followed by the pseudo-pack order for
|
||||
`$H3`).
|
||||
+
|
||||
The layout will then be extended so that each layer of the incremental
|
||||
MIDX chain can write a `*.bitmap`. The objects in each layer's bitmap
|
||||
are offset by the number of objects in the previous layers of the chain.
|
||||
|
||||
=== File layout
|
||||
|
||||
Instead of storing a single `multi-pack-index` file (with an optional
|
||||
`.rev` and `.bitmap` extension) in `$GIT_DIR/objects/pack`, incremental
|
||||
MIDXs are stored in the following layout:
|
||||
|
||||
----
|
||||
$GIT_DIR/objects/pack/multi-pack-index.d/
|
||||
$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-chain
|
||||
$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-$H1.midx
|
||||
$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-$H2.midx
|
||||
$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-$H3.midx
|
||||
----
|
||||
|
||||
The `multi-pack-index-chain` file contains a list of the incremental
|
||||
MIDX files in the chain, in order. The above example shows a chain whose
|
||||
`multi-pack-index-chain` file would contain the following lines:
|
||||
|
||||
----
|
||||
$H1
|
||||
$H2
|
||||
$H3
|
||||
----
|
||||
|
||||
The `multi-pack-index-$H1.midx` file contains the first layer of the
|
||||
multi-pack-index chain. The `multi-pack-index-$H2.midx` file contains
|
||||
the second layer of the chain, and so on.
|
||||
|
||||
When both an incremental- and non-incremental MIDX are present, the
|
||||
non-incremental MIDX is always read first.
|
||||
|
||||
=== Object positions for incremental MIDXs
|
||||
|
||||
In the original multi-pack-index design, we refer to objects via their
|
||||
lexicographic position (by object IDs) within the repository's singular
|
||||
multi-pack-index. In the incremental multi-pack-index design, we refer
|
||||
to objects via their index into a concatenated lexicographic ordering
|
||||
among each component in the MIDX chain.
|
||||
|
||||
If `objects_nr()` is a function that returns the number of objects in a
|
||||
given MIDX layer, then the index of an object at lexicographic position
|
||||
`i` within, say, $H3 is defined as:
|
||||
|
||||
----
|
||||
objects_nr($H2) + objects_nr($H1) + i
|
||||
----
|
||||
|
||||
(in the C implementation, this is often computed as `i +
|
||||
m->num_objects_in_base`).
|
||||
|
||||
Future Work
|
||||
-----------
|
||||
|
||||
|
|
|
|||
|
|
@ -129,6 +129,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv,
|
|||
MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX),
|
||||
OPT_BIT(0, "progress", &opts.flags,
|
||||
N_("force progress reporting"), MIDX_PROGRESS),
|
||||
OPT_BIT(0, "incremental", &opts.flags,
|
||||
N_("write a new incremental MIDX"), MIDX_WRITE_INCREMENTAL),
|
||||
OPT_BOOL(0, "stdin-packs", &opts.stdin_packs,
|
||||
N_("write multi-pack index containing only given indexes")),
|
||||
OPT_FILENAME(0, "refs-snapshot", &opts.refs_snapshot,
|
||||
|
|
|
|||
|
|
@ -1218,10 +1218,6 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
|
|||
if (!write_midx &&
|
||||
(!(pack_everything & ALL_INTO_ONE) || !is_bare_repository()))
|
||||
write_bitmaps = 0;
|
||||
} else if (write_bitmaps &&
|
||||
git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0) &&
|
||||
git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0)) {
|
||||
write_bitmaps = 0;
|
||||
}
|
||||
if (pack_kept_objects < 0)
|
||||
pack_kept_objects = write_bitmaps > 0 && !write_midx;
|
||||
|
|
@ -1521,8 +1517,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
|
|||
|
||||
if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) {
|
||||
unsigned flags = 0;
|
||||
if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0))
|
||||
flags |= MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX;
|
||||
if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL, 0))
|
||||
flags |= MIDX_WRITE_INCREMENTAL;
|
||||
write_midx_file(get_object_directory(), NULL, NULL, flags);
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -25,7 +25,7 @@ linux-TEST-vars)
|
|||
export GIT_TEST_COMMIT_GRAPH=1
|
||||
export GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1
|
||||
export GIT_TEST_MULTI_PACK_INDEX=1
|
||||
export GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=1
|
||||
export GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=1
|
||||
export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master
|
||||
export GIT_TEST_NO_WRITE_REV_INDEX=1
|
||||
export GIT_TEST_CHECKOUT_WORKERS=2
|
||||
|
|
|
|||
324
midx-write.c
324
midx-write.c
|
|
@ -17,6 +17,8 @@
|
|||
#include "refs.h"
|
||||
#include "revision.h"
|
||||
#include "list-objects.h"
|
||||
#include "path.h"
|
||||
#include "pack-revindex.h"
|
||||
|
||||
#define PACK_EXPIRED UINT_MAX
|
||||
#define BITMAP_POS_UNKNOWN (~((uint32_t)0))
|
||||
|
|
@ -25,7 +27,11 @@
|
|||
|
||||
extern int midx_checksum_valid(struct multi_pack_index *m);
|
||||
extern void clear_midx_files_ext(const char *object_dir, const char *ext,
|
||||
unsigned char *keep_hash);
|
||||
const char *keep_hash);
|
||||
extern void clear_incremental_midx_files_ext(const char *object_dir,
|
||||
const char *ext,
|
||||
const char **keep_hashes,
|
||||
uint32_t hashes_nr);
|
||||
extern int cmp_idx_or_pack_name(const char *idx_or_pack_name,
|
||||
const char *idx_name);
|
||||
|
||||
|
|
@ -86,6 +92,7 @@ struct write_midx_context {
|
|||
size_t nr;
|
||||
size_t alloc;
|
||||
struct multi_pack_index *m;
|
||||
struct multi_pack_index *base_midx;
|
||||
struct progress *progress;
|
||||
unsigned pack_paths_checked;
|
||||
|
||||
|
|
@ -99,6 +106,9 @@ struct write_midx_context {
|
|||
|
||||
int preferred_pack_idx;
|
||||
|
||||
int incremental;
|
||||
uint32_t num_multi_pack_indexes_before;
|
||||
|
||||
struct string_list *to_include;
|
||||
};
|
||||
|
||||
|
|
@ -122,6 +132,9 @@ static int should_include_pack(const struct write_midx_context *ctx,
|
|||
*/
|
||||
if (ctx->m && midx_contains_pack(ctx->m, file_name))
|
||||
return 0;
|
||||
else if (ctx->base_midx && midx_contains_pack(ctx->base_midx,
|
||||
file_name))
|
||||
return 0;
|
||||
else if (ctx->to_include &&
|
||||
!string_list_has_string(ctx->to_include, file_name))
|
||||
return 0;
|
||||
|
|
@ -196,7 +209,7 @@ static int nth_midxed_pack_midx_entry(struct multi_pack_index *m,
|
|||
struct pack_midx_entry *e,
|
||||
uint32_t pos)
|
||||
{
|
||||
if (pos >= m->num_objects)
|
||||
if (pos >= m->num_objects + m->num_objects_in_base)
|
||||
return 1;
|
||||
|
||||
nth_midxed_object_oid(&e->oid, m, pos);
|
||||
|
|
@ -247,12 +260,16 @@ static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout,
|
|||
uint32_t cur_fanout,
|
||||
int preferred_pack)
|
||||
{
|
||||
uint32_t start = 0, end;
|
||||
uint32_t start = m->num_objects_in_base, end;
|
||||
uint32_t cur_object;
|
||||
|
||||
if (m->base_midx)
|
||||
midx_fanout_add_midx_fanout(fanout, m->base_midx, cur_fanout,
|
||||
preferred_pack);
|
||||
|
||||
if (cur_fanout)
|
||||
start = ntohl(m->chunk_oid_fanout[cur_fanout - 1]);
|
||||
end = ntohl(m->chunk_oid_fanout[cur_fanout]);
|
||||
start += ntohl(m->chunk_oid_fanout[cur_fanout - 1]);
|
||||
end = m->num_objects_in_base + ntohl(m->chunk_oid_fanout[cur_fanout]);
|
||||
|
||||
for (cur_object = start; cur_object < end; cur_object++) {
|
||||
if ((preferred_pack > -1) &&
|
||||
|
|
@ -334,7 +351,7 @@ static void compute_sorted_entries(struct write_midx_context *ctx,
|
|||
for (cur_fanout = 0; cur_fanout < 256; cur_fanout++) {
|
||||
fanout.nr = 0;
|
||||
|
||||
if (ctx->m)
|
||||
if (ctx->m && !ctx->incremental)
|
||||
midx_fanout_add_midx_fanout(&fanout, ctx->m, cur_fanout,
|
||||
ctx->preferred_pack_idx);
|
||||
|
||||
|
|
@ -360,6 +377,10 @@ static void compute_sorted_entries(struct write_midx_context *ctx,
|
|||
if (cur_object && oideq(&fanout.entries[cur_object - 1].oid,
|
||||
&fanout.entries[cur_object].oid))
|
||||
continue;
|
||||
if (ctx->incremental && ctx->base_midx &&
|
||||
midx_has_oid(ctx->base_midx,
|
||||
&fanout.entries[cur_object].oid))
|
||||
continue;
|
||||
|
||||
ALLOC_GROW(ctx->entries, st_add(ctx->entries_nr, 1),
|
||||
alloc_objects);
|
||||
|
|
@ -543,10 +564,16 @@ static int write_midx_revindex(struct hashfile *f,
|
|||
void *data)
|
||||
{
|
||||
struct write_midx_context *ctx = data;
|
||||
uint32_t i;
|
||||
uint32_t i, nr_base;
|
||||
|
||||
if (ctx->incremental && ctx->base_midx)
|
||||
nr_base = ctx->base_midx->num_objects +
|
||||
ctx->base_midx->num_objects_in_base;
|
||||
else
|
||||
nr_base = 0;
|
||||
|
||||
for (i = 0; i < ctx->entries_nr; i++)
|
||||
hashwrite_be32(f, ctx->pack_order[i]);
|
||||
hashwrite_be32(f, ctx->pack_order[i] + nr_base);
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
|
@ -575,12 +602,18 @@ static int midx_pack_order_cmp(const void *va, const void *vb)
|
|||
static uint32_t *midx_pack_order(struct write_midx_context *ctx)
|
||||
{
|
||||
struct midx_pack_order_data *data;
|
||||
uint32_t *pack_order;
|
||||
uint32_t *pack_order, base_objects = 0;
|
||||
uint32_t i;
|
||||
|
||||
trace2_region_enter("midx", "midx_pack_order", the_repository);
|
||||
|
||||
if (ctx->incremental && ctx->base_midx)
|
||||
base_objects = ctx->base_midx->num_objects +
|
||||
ctx->base_midx->num_objects_in_base;
|
||||
|
||||
ALLOC_ARRAY(pack_order, ctx->entries_nr);
|
||||
ALLOC_ARRAY(data, ctx->entries_nr);
|
||||
|
||||
for (i = 0; i < ctx->entries_nr; i++) {
|
||||
struct pack_midx_entry *e = &ctx->entries[i];
|
||||
data[i].nr = i;
|
||||
|
|
@ -592,12 +625,11 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx)
|
|||
|
||||
QSORT(data, ctx->entries_nr, midx_pack_order_cmp);
|
||||
|
||||
ALLOC_ARRAY(pack_order, ctx->entries_nr);
|
||||
for (i = 0; i < ctx->entries_nr; i++) {
|
||||
struct pack_midx_entry *e = &ctx->entries[data[i].nr];
|
||||
struct pack_info *pack = &ctx->info[ctx->pack_perm[e->pack_int_id]];
|
||||
if (pack->bitmap_pos == BITMAP_POS_UNKNOWN)
|
||||
pack->bitmap_pos = i;
|
||||
pack->bitmap_pos = i + base_objects;
|
||||
pack->bitmap_nr++;
|
||||
pack_order[i] = data[i].nr;
|
||||
}
|
||||
|
|
@ -645,7 +677,8 @@ static void prepare_midx_packing_data(struct packing_data *pdata,
|
|||
prepare_packing_data(the_repository, pdata);
|
||||
|
||||
for (i = 0; i < ctx->entries_nr; i++) {
|
||||
struct pack_midx_entry *from = &ctx->entries[ctx->pack_order[i]];
|
||||
uint32_t pos = ctx->pack_order[i];
|
||||
struct pack_midx_entry *from = &ctx->entries[pos];
|
||||
struct object_entry *to = packlist_alloc(pdata, &from->oid);
|
||||
|
||||
oe_set_in_pack(pdata, to,
|
||||
|
|
@ -893,35 +926,128 @@ static struct multi_pack_index *lookup_multi_pack_index(struct repository *r,
|
|||
static int fill_packs_from_midx(struct write_midx_context *ctx,
|
||||
const char *preferred_pack_name, uint32_t flags)
|
||||
{
|
||||
uint32_t i;
|
||||
struct multi_pack_index *m;
|
||||
|
||||
for (i = 0; i < ctx->m->num_packs; i++) {
|
||||
ALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);
|
||||
for (m = ctx->m; m; m = m->base_midx) {
|
||||
uint32_t i;
|
||||
|
||||
for (i = 0; i < m->num_packs; i++) {
|
||||
ALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc);
|
||||
|
||||
if (flags & MIDX_WRITE_REV_INDEX || preferred_pack_name) {
|
||||
/*
|
||||
* If generating a reverse index, need to have
|
||||
* packed_git's loaded to compare their
|
||||
* mtimes and object count.
|
||||
*
|
||||
*
|
||||
* If a preferred pack is specified, need to
|
||||
* have packed_git's loaded to ensure the chosen
|
||||
* preferred pack has a non-zero object count.
|
||||
*/
|
||||
if (prepare_midx_pack(the_repository, ctx->m, i))
|
||||
return error(_("could not load pack"));
|
||||
if (flags & MIDX_WRITE_REV_INDEX ||
|
||||
preferred_pack_name) {
|
||||
if (prepare_midx_pack(the_repository, m,
|
||||
m->num_packs_in_base + i)) {
|
||||
error(_("could not load pack"));
|
||||
return 1;
|
||||
}
|
||||
|
||||
if (open_pack_index(ctx->m->packs[i]))
|
||||
die(_("could not open index for %s"),
|
||||
ctx->m->packs[i]->pack_name);
|
||||
if (open_pack_index(m->packs[i]))
|
||||
die(_("could not open index for %s"),
|
||||
m->packs[i]->pack_name);
|
||||
}
|
||||
|
||||
fill_pack_info(&ctx->info[ctx->nr++], m->packs[i],
|
||||
m->pack_names[i],
|
||||
m->num_packs_in_base + i);
|
||||
}
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
fill_pack_info(&ctx->info[ctx->nr++], ctx->m->packs[i],
|
||||
ctx->m->pack_names[i], i);
|
||||
static struct {
|
||||
const char *non_split;
|
||||
const char *split;
|
||||
} midx_exts[] = {
|
||||
{NULL, MIDX_EXT_MIDX},
|
||||
{MIDX_EXT_BITMAP, MIDX_EXT_BITMAP},
|
||||
{MIDX_EXT_REV, MIDX_EXT_REV},
|
||||
};
|
||||
|
||||
static int link_midx_to_chain(struct multi_pack_index *m)
|
||||
{
|
||||
struct strbuf from = STRBUF_INIT;
|
||||
struct strbuf to = STRBUF_INIT;
|
||||
int ret = 0;
|
||||
size_t i;
|
||||
|
||||
if (!m || m->has_chain) {
|
||||
/*
|
||||
* Either no MIDX previously existed, or it was already
|
||||
* part of a MIDX chain. In both cases, we have nothing
|
||||
* to link, so return early.
|
||||
*/
|
||||
goto done;
|
||||
}
|
||||
|
||||
return 0;
|
||||
for (i = 0; i < ARRAY_SIZE(midx_exts); i++) {
|
||||
const unsigned char *hash = get_midx_checksum(m);
|
||||
|
||||
get_midx_filename_ext(&from, m->object_dir, hash,
|
||||
midx_exts[i].non_split);
|
||||
get_split_midx_filename_ext(&to, m->object_dir, hash,
|
||||
midx_exts[i].split);
|
||||
|
||||
if (link(from.buf, to.buf) < 0 && errno != ENOENT) {
|
||||
ret = error_errno(_("unable to link '%s' to '%s'"),
|
||||
from.buf, to.buf);
|
||||
goto done;
|
||||
}
|
||||
|
||||
strbuf_reset(&from);
|
||||
strbuf_reset(&to);
|
||||
}
|
||||
|
||||
done:
|
||||
strbuf_release(&from);
|
||||
strbuf_release(&to);
|
||||
return ret;
|
||||
}
|
||||
|
||||
static void clear_midx_files(const char *object_dir,
|
||||
const char **hashes,
|
||||
uint32_t hashes_nr,
|
||||
unsigned incremental)
|
||||
{
|
||||
/*
|
||||
* if incremental:
|
||||
* - remove all non-incremental MIDX files
|
||||
* - remove any incremental MIDX files not in the current one
|
||||
*
|
||||
* if non-incremental:
|
||||
* - remove all incremental MIDX files
|
||||
* - remove any non-incremental MIDX files not matching the current
|
||||
* hash
|
||||
*/
|
||||
struct strbuf buf = STRBUF_INIT;
|
||||
const char *exts[] = { MIDX_EXT_BITMAP, MIDX_EXT_REV, MIDX_EXT_MIDX };
|
||||
uint32_t i, j;
|
||||
|
||||
for (i = 0; i < ARRAY_SIZE(exts); i++) {
|
||||
clear_incremental_midx_files_ext(object_dir, exts[i],
|
||||
hashes, hashes_nr);
|
||||
for (j = 0; j < hashes_nr; j++)
|
||||
clear_midx_files_ext(object_dir, exts[i], hashes[j]);
|
||||
}
|
||||
|
||||
if (incremental)
|
||||
get_midx_filename(&buf, object_dir);
|
||||
else
|
||||
get_midx_chain_filename(&buf, object_dir);
|
||||
|
||||
if (unlink(buf.buf) && errno != ENOENT)
|
||||
die_errno(_("failed to clear multi-pack-index at %s"), buf.buf);
|
||||
|
||||
strbuf_release(&buf);
|
||||
}
|
||||
|
||||
static int write_midx_internal(const char *object_dir,
|
||||
|
|
@ -936,42 +1062,66 @@ static int write_midx_internal(const char *object_dir,
|
|||
uint32_t i, start_pack;
|
||||
struct hashfile *f = NULL;
|
||||
struct lock_file lk;
|
||||
struct tempfile *incr;
|
||||
struct write_midx_context ctx = { 0 };
|
||||
int bitmapped_packs_concat_len = 0;
|
||||
int pack_name_concat_len = 0;
|
||||
int dropped_packs = 0;
|
||||
int result = 0;
|
||||
const char **keep_hashes = NULL;
|
||||
struct chunkfile *cf;
|
||||
|
||||
trace2_region_enter("midx", "write_midx_internal", the_repository);
|
||||
|
||||
get_midx_filename(&midx_name, object_dir);
|
||||
ctx.incremental = !!(flags & MIDX_WRITE_INCREMENTAL);
|
||||
if (ctx.incremental && (flags & MIDX_WRITE_BITMAP))
|
||||
die(_("cannot write incremental MIDX with bitmap"));
|
||||
|
||||
if (ctx.incremental)
|
||||
strbuf_addf(&midx_name,
|
||||
"%s/pack/multi-pack-index.d/tmp_midx_XXXXXX",
|
||||
object_dir);
|
||||
else
|
||||
get_midx_filename(&midx_name, object_dir);
|
||||
if (safe_create_leading_directories(midx_name.buf))
|
||||
die_errno(_("unable to create leading directories of %s"),
|
||||
midx_name.buf);
|
||||
|
||||
if (!packs_to_include) {
|
||||
/*
|
||||
* Only reference an existing MIDX when not filtering which
|
||||
* packs to include, since all packs and objects are copied
|
||||
* blindly from an existing MIDX if one is present.
|
||||
*/
|
||||
ctx.m = lookup_multi_pack_index(the_repository, object_dir);
|
||||
}
|
||||
if (!packs_to_include || ctx.incremental) {
|
||||
struct multi_pack_index *m = lookup_multi_pack_index(the_repository,
|
||||
object_dir);
|
||||
if (m && !midx_checksum_valid(m)) {
|
||||
warning(_("ignoring existing multi-pack-index; checksum mismatch"));
|
||||
m = NULL;
|
||||
}
|
||||
|
||||
if (ctx.m && !midx_checksum_valid(ctx.m)) {
|
||||
warning(_("ignoring existing multi-pack-index; checksum mismatch"));
|
||||
ctx.m = NULL;
|
||||
if (m) {
|
||||
/*
|
||||
* Only reference an existing MIDX when not filtering
|
||||
* which packs to include, since all packs and objects
|
||||
* are copied blindly from an existing MIDX if one is
|
||||
* present.
|
||||
*/
|
||||
if (ctx.incremental)
|
||||
ctx.base_midx = m;
|
||||
else if (!packs_to_include)
|
||||
ctx.m = m;
|
||||
}
|
||||
}
|
||||
|
||||
ctx.nr = 0;
|
||||
ctx.alloc = ctx.m ? ctx.m->num_packs : 16;
|
||||
ctx.alloc = ctx.m ? ctx.m->num_packs + ctx.m->num_packs_in_base : 16;
|
||||
ctx.info = NULL;
|
||||
ALLOC_ARRAY(ctx.info, ctx.alloc);
|
||||
|
||||
if (ctx.m && fill_packs_from_midx(&ctx, preferred_pack_name,
|
||||
flags) < 0) {
|
||||
result = 1;
|
||||
if (ctx.incremental) {
|
||||
struct multi_pack_index *m = ctx.base_midx;
|
||||
while (m) {
|
||||
ctx.num_multi_pack_indexes_before++;
|
||||
m = m->base_midx;
|
||||
}
|
||||
} else if (ctx.m && fill_packs_from_midx(&ctx, preferred_pack_name,
|
||||
flags) < 0) {
|
||||
goto cleanup;
|
||||
}
|
||||
|
||||
|
|
@ -988,7 +1138,8 @@ static int write_midx_internal(const char *object_dir,
|
|||
for_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx);
|
||||
stop_progress(&ctx.progress);
|
||||
|
||||
if ((ctx.m && ctx.nr == ctx.m->num_packs) &&
|
||||
if ((ctx.m && ctx.nr == ctx.m->num_packs + ctx.m->num_packs_in_base) &&
|
||||
!ctx.incremental &&
|
||||
!(packs_to_include || packs_to_drop)) {
|
||||
struct bitmap_index *bitmap_git;
|
||||
int bitmap_exists;
|
||||
|
|
@ -1004,12 +1155,14 @@ static int write_midx_internal(const char *object_dir,
|
|||
* corresponding bitmap (or one wasn't requested).
|
||||
*/
|
||||
if (!want_bitmap)
|
||||
clear_midx_files_ext(object_dir, ".bitmap",
|
||||
NULL);
|
||||
clear_midx_files_ext(object_dir, "bitmap", NULL);
|
||||
goto cleanup;
|
||||
}
|
||||
}
|
||||
|
||||
if (ctx.incremental && !ctx.nr)
|
||||
goto cleanup; /* nothing to do */
|
||||
|
||||
if (preferred_pack_name) {
|
||||
ctx.preferred_pack_idx = -1;
|
||||
|
||||
|
|
@ -1155,8 +1308,30 @@ static int write_midx_internal(const char *object_dir,
|
|||
pack_name_concat_len += MIDX_CHUNK_ALIGNMENT -
|
||||
(pack_name_concat_len % MIDX_CHUNK_ALIGNMENT);
|
||||
|
||||
hold_lock_file_for_update(&lk, midx_name.buf, LOCK_DIE_ON_ERROR);
|
||||
f = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));
|
||||
if (ctx.incremental) {
|
||||
struct strbuf lock_name = STRBUF_INIT;
|
||||
|
||||
get_midx_chain_filename(&lock_name, object_dir);
|
||||
hold_lock_file_for_update(&lk, lock_name.buf, LOCK_DIE_ON_ERROR);
|
||||
strbuf_release(&lock_name);
|
||||
|
||||
incr = mks_tempfile_m(midx_name.buf, 0444);
|
||||
if (!incr) {
|
||||
error(_("unable to create temporary MIDX layer"));
|
||||
return -1;
|
||||
}
|
||||
|
||||
if (adjust_shared_perm(get_tempfile_path(incr))) {
|
||||
error(_("unable to adjust shared permissions for '%s'"),
|
||||
get_tempfile_path(incr));
|
||||
return -1;
|
||||
}
|
||||
|
||||
f = hashfd(get_tempfile_fd(incr), get_tempfile_path(incr));
|
||||
} else {
|
||||
hold_lock_file_for_update(&lk, midx_name.buf, LOCK_DIE_ON_ERROR);
|
||||
f = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk));
|
||||
}
|
||||
|
||||
if (ctx.nr - dropped_packs == 0) {
|
||||
error(_("no pack files to index."));
|
||||
|
|
@ -1249,14 +1424,55 @@ static int write_midx_internal(const char *object_dir,
|
|||
* have been freed in the previous if block.
|
||||
*/
|
||||
|
||||
if (ctx.m)
|
||||
CALLOC_ARRAY(keep_hashes, ctx.num_multi_pack_indexes_before + 1);
|
||||
|
||||
if (ctx.incremental) {
|
||||
FILE *chainf = fdopen_lock_file(&lk, "w");
|
||||
struct strbuf final_midx_name = STRBUF_INIT;
|
||||
struct multi_pack_index *m = ctx.base_midx;
|
||||
|
||||
if (!chainf) {
|
||||
error_errno(_("unable to open multi-pack-index chain file"));
|
||||
return -1;
|
||||
}
|
||||
|
||||
if (link_midx_to_chain(ctx.base_midx) < 0)
|
||||
return -1;
|
||||
|
||||
get_split_midx_filename_ext(&final_midx_name, object_dir,
|
||||
midx_hash, MIDX_EXT_MIDX);
|
||||
|
||||
if (rename_tempfile(&incr, final_midx_name.buf) < 0) {
|
||||
error_errno(_("unable to rename new multi-pack-index layer"));
|
||||
return -1;
|
||||
}
|
||||
|
||||
keep_hashes[ctx.num_multi_pack_indexes_before] =
|
||||
xstrdup(hash_to_hex(midx_hash));
|
||||
|
||||
for (i = 0; i < ctx.num_multi_pack_indexes_before; i++) {
|
||||
uint32_t j = ctx.num_multi_pack_indexes_before - i - 1;
|
||||
|
||||
keep_hashes[j] = xstrdup(hash_to_hex(get_midx_checksum(m)));
|
||||
m = m->base_midx;
|
||||
}
|
||||
|
||||
for (i = 0; i < ctx.num_multi_pack_indexes_before + 1; i++)
|
||||
fprintf(get_lock_file_fp(&lk), "%s\n", keep_hashes[i]);
|
||||
} else {
|
||||
keep_hashes[ctx.num_multi_pack_indexes_before] =
|
||||
xstrdup(hash_to_hex(midx_hash));
|
||||
}
|
||||
|
||||
if (ctx.m || ctx.base_midx)
|
||||
close_object_store(the_repository->objects);
|
||||
|
||||
if (commit_lock_file(&lk) < 0)
|
||||
die_errno(_("could not write multi-pack-index"));
|
||||
|
||||
clear_midx_files_ext(object_dir, ".bitmap", midx_hash);
|
||||
clear_midx_files_ext(object_dir, ".rev", midx_hash);
|
||||
clear_midx_files(object_dir, keep_hashes,
|
||||
ctx.num_multi_pack_indexes_before + 1,
|
||||
ctx.incremental);
|
||||
|
||||
cleanup:
|
||||
for (i = 0; i < ctx.nr; i++) {
|
||||
|
|
@ -1271,6 +1487,11 @@ static int write_midx_internal(const char *object_dir,
|
|||
free(ctx.entries);
|
||||
free(ctx.pack_perm);
|
||||
free(ctx.pack_order);
|
||||
if (keep_hashes) {
|
||||
for (i = 0; i < ctx.num_multi_pack_indexes_before + 1; i++)
|
||||
free((char *)keep_hashes[i]);
|
||||
free(keep_hashes);
|
||||
}
|
||||
strbuf_release(&midx_name);
|
||||
|
||||
trace2_region_leave("midx", "write_midx_internal", the_repository);
|
||||
|
|
@ -1307,6 +1528,9 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla
|
|||
if (!m)
|
||||
return 0;
|
||||
|
||||
if (m->base_midx)
|
||||
die(_("cannot expire packs from an incremental multi-pack-index"));
|
||||
|
||||
CALLOC_ARRAY(count, m->num_packs);
|
||||
|
||||
if (flags & MIDX_PROGRESS)
|
||||
|
|
@ -1481,6 +1705,8 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size,
|
|||
|
||||
if (!m)
|
||||
return 0;
|
||||
if (m->base_midx)
|
||||
die(_("cannot repack an incremental multi-pack-index"));
|
||||
|
||||
CALLOC_ARRAY(include_pack, m->num_packs);
|
||||
|
||||
|
|
|
|||
403
midx.c
403
midx.c
|
|
@ -16,7 +16,10 @@
|
|||
|
||||
int midx_checksum_valid(struct multi_pack_index *m);
|
||||
void clear_midx_files_ext(const char *object_dir, const char *ext,
|
||||
unsigned char *keep_hash);
|
||||
const char *keep_hash);
|
||||
void clear_incremental_midx_files_ext(const char *object_dir, const char *ext,
|
||||
char **keep_hashes,
|
||||
uint32_t hashes_nr);
|
||||
int cmp_idx_or_pack_name(const char *idx_or_pack_name,
|
||||
const char *idx_name);
|
||||
|
||||
|
|
@ -91,7 +94,9 @@ static int midx_read_object_offsets(const unsigned char *chunk_start,
|
|||
|
||||
#define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + the_hash_algo->rawsz)
|
||||
|
||||
struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local)
|
||||
static struct multi_pack_index *load_multi_pack_index_one(const char *object_dir,
|
||||
const char *midx_name,
|
||||
int local)
|
||||
{
|
||||
struct multi_pack_index *m = NULL;
|
||||
int fd;
|
||||
|
|
@ -99,31 +104,26 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local
|
|||
size_t midx_size;
|
||||
void *midx_map = NULL;
|
||||
uint32_t hash_version;
|
||||
struct strbuf midx_name = STRBUF_INIT;
|
||||
uint32_t i;
|
||||
const char *cur_pack_name;
|
||||
struct chunkfile *cf = NULL;
|
||||
|
||||
get_midx_filename(&midx_name, object_dir);
|
||||
|
||||
fd = git_open(midx_name.buf);
|
||||
fd = git_open(midx_name);
|
||||
|
||||
if (fd < 0)
|
||||
goto cleanup_fail;
|
||||
if (fstat(fd, &st)) {
|
||||
error_errno(_("failed to read %s"), midx_name.buf);
|
||||
error_errno(_("failed to read %s"), midx_name);
|
||||
goto cleanup_fail;
|
||||
}
|
||||
|
||||
midx_size = xsize_t(st.st_size);
|
||||
|
||||
if (midx_size < MIDX_MIN_SIZE) {
|
||||
error(_("multi-pack-index file %s is too small"), midx_name.buf);
|
||||
error(_("multi-pack-index file %s is too small"), midx_name);
|
||||
goto cleanup_fail;
|
||||
}
|
||||
|
||||
strbuf_release(&midx_name);
|
||||
|
||||
midx_map = xmmap(NULL, midx_size, PROT_READ, MAP_PRIVATE, fd, 0);
|
||||
close(fd);
|
||||
|
||||
|
|
@ -213,7 +213,6 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local
|
|||
|
||||
cleanup_fail:
|
||||
free(m);
|
||||
strbuf_release(&midx_name);
|
||||
free_chunkfile(cf);
|
||||
if (midx_map)
|
||||
munmap(midx_map, midx_size);
|
||||
|
|
@ -222,6 +221,173 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local
|
|||
return NULL;
|
||||
}
|
||||
|
||||
void get_midx_chain_dirname(struct strbuf *buf, const char *object_dir)
|
||||
{
|
||||
strbuf_addf(buf, "%s/pack/multi-pack-index.d", object_dir);
|
||||
}
|
||||
|
||||
void get_midx_chain_filename(struct strbuf *buf, const char *object_dir)
|
||||
{
|
||||
get_midx_chain_dirname(buf, object_dir);
|
||||
strbuf_addstr(buf, "/multi-pack-index-chain");
|
||||
}
|
||||
|
||||
void get_split_midx_filename_ext(struct strbuf *buf, const char *object_dir,
|
||||
const unsigned char *hash, const char *ext)
|
||||
{
|
||||
get_midx_chain_dirname(buf, object_dir);
|
||||
strbuf_addf(buf, "/multi-pack-index-%s.%s", hash_to_hex(hash), ext);
|
||||
}
|
||||
|
||||
static int open_multi_pack_index_chain(const char *chain_file,
|
||||
int *fd, struct stat *st)
|
||||
{
|
||||
*fd = git_open(chain_file);
|
||||
if (*fd < 0)
|
||||
return 0;
|
||||
if (fstat(*fd, st)) {
|
||||
close(*fd);
|
||||
return 0;
|
||||
}
|
||||
if (st->st_size < the_hash_algo->hexsz) {
|
||||
close(*fd);
|
||||
if (!st->st_size) {
|
||||
/* treat empty files the same as missing */
|
||||
errno = ENOENT;
|
||||
} else {
|
||||
warning(_("multi-pack-index chain file too small"));
|
||||
errno = EINVAL;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
return 1;
|
||||
}
|
||||
|
||||
static int add_midx_to_chain(struct multi_pack_index *midx,
|
||||
struct multi_pack_index *midx_chain,
|
||||
struct object_id *oids,
|
||||
int n)
|
||||
{
|
||||
if (midx_chain) {
|
||||
if (unsigned_add_overflows(midx_chain->num_packs,
|
||||
midx_chain->num_packs_in_base)) {
|
||||
warning(_("pack count in base MIDX too high: %"PRIuMAX),
|
||||
(uintmax_t)midx_chain->num_packs_in_base);
|
||||
return 0;
|
||||
}
|
||||
if (unsigned_add_overflows(midx_chain->num_objects,
|
||||
midx_chain->num_objects_in_base)) {
|
||||
warning(_("object count in base MIDX too high: %"PRIuMAX),
|
||||
(uintmax_t)midx_chain->num_objects_in_base);
|
||||
return 0;
|
||||
}
|
||||
midx->num_packs_in_base = midx_chain->num_packs +
|
||||
midx_chain->num_packs_in_base;
|
||||
midx->num_objects_in_base = midx_chain->num_objects +
|
||||
midx_chain->num_objects_in_base;
|
||||
}
|
||||
|
||||
midx->base_midx = midx_chain;
|
||||
midx->has_chain = 1;
|
||||
|
||||
return 1;
|
||||
}
|
||||
|
||||
static struct multi_pack_index *load_midx_chain_fd_st(const char *object_dir,
|
||||
int local,
|
||||
int fd, struct stat *st,
|
||||
int *incomplete_chain)
|
||||
{
|
||||
struct multi_pack_index *midx_chain = NULL;
|
||||
struct strbuf buf = STRBUF_INIT;
|
||||
struct object_id *layers = NULL;
|
||||
int valid = 1;
|
||||
uint32_t i, count;
|
||||
FILE *fp = xfdopen(fd, "r");
|
||||
|
||||
count = st->st_size / (the_hash_algo->hexsz + 1);
|
||||
CALLOC_ARRAY(layers, count);
|
||||
|
||||
for (i = 0; i < count; i++) {
|
||||
struct multi_pack_index *m;
|
||||
|
||||
if (strbuf_getline_lf(&buf, fp) == EOF)
|
||||
break;
|
||||
|
||||
if (get_oid_hex(buf.buf, &layers[i])) {
|
||||
warning(_("invalid multi-pack-index chain: line '%s' "
|
||||
"not a hash"),
|
||||
buf.buf);
|
||||
valid = 0;
|
||||
break;
|
||||
}
|
||||
|
||||
valid = 0;
|
||||
|
||||
strbuf_reset(&buf);
|
||||
get_split_midx_filename_ext(&buf, object_dir, layers[i].hash,
|
||||
MIDX_EXT_MIDX);
|
||||
m = load_multi_pack_index_one(object_dir, buf.buf, local);
|
||||
|
||||
if (m) {
|
||||
if (add_midx_to_chain(m, midx_chain, layers, i)) {
|
||||
midx_chain = m;
|
||||
valid = 1;
|
||||
} else {
|
||||
close_midx(m);
|
||||
}
|
||||
}
|
||||
if (!valid) {
|
||||
warning(_("unable to find all multi-pack index files"));
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
free(layers);
|
||||
fclose(fp);
|
||||
strbuf_release(&buf);
|
||||
|
||||
*incomplete_chain = !valid;
|
||||
return midx_chain;
|
||||
}
|
||||
|
||||
static struct multi_pack_index *load_multi_pack_index_chain(const char *object_dir,
|
||||
int local)
|
||||
{
|
||||
struct strbuf chain_file = STRBUF_INIT;
|
||||
struct stat st;
|
||||
int fd;
|
||||
struct multi_pack_index *m = NULL;
|
||||
|
||||
get_midx_chain_filename(&chain_file, object_dir);
|
||||
if (open_multi_pack_index_chain(chain_file.buf, &fd, &st)) {
|
||||
int incomplete;
|
||||
/* ownership of fd is taken over by load function */
|
||||
m = load_midx_chain_fd_st(object_dir, local, fd, &st,
|
||||
&incomplete);
|
||||
}
|
||||
|
||||
strbuf_release(&chain_file);
|
||||
return m;
|
||||
}
|
||||
|
||||
struct multi_pack_index *load_multi_pack_index(const char *object_dir,
|
||||
int local)
|
||||
{
|
||||
struct strbuf midx_name = STRBUF_INIT;
|
||||
struct multi_pack_index *m;
|
||||
|
||||
get_midx_filename(&midx_name, object_dir);
|
||||
|
||||
m = load_multi_pack_index_one(object_dir, midx_name.buf, local);
|
||||
if (!m)
|
||||
m = load_multi_pack_index_chain(object_dir, local);
|
||||
|
||||
strbuf_release(&midx_name);
|
||||
|
||||
return m;
|
||||
}
|
||||
|
||||
void close_midx(struct multi_pack_index *m)
|
||||
{
|
||||
uint32_t i;
|
||||
|
|
@ -230,6 +396,7 @@ void close_midx(struct multi_pack_index *m)
|
|||
return;
|
||||
|
||||
close_midx(m->next);
|
||||
close_midx(m->base_midx);
|
||||
|
||||
munmap((unsigned char *)m->data, m->data_len);
|
||||
|
||||
|
|
@ -242,14 +409,49 @@ void close_midx(struct multi_pack_index *m)
|
|||
free(m);
|
||||
}
|
||||
|
||||
int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id)
|
||||
static uint32_t midx_for_object(struct multi_pack_index **_m, uint32_t pos)
|
||||
{
|
||||
struct multi_pack_index *m = *_m;
|
||||
while (m && pos < m->num_objects_in_base)
|
||||
m = m->base_midx;
|
||||
|
||||
if (!m)
|
||||
BUG("NULL multi-pack-index for object position: %"PRIu32, pos);
|
||||
|
||||
if (pos >= m->num_objects + m->num_objects_in_base)
|
||||
die(_("invalid MIDX object position, MIDX is likely corrupt"));
|
||||
|
||||
*_m = m;
|
||||
|
||||
return pos - m->num_objects_in_base;
|
||||
}
|
||||
|
||||
static uint32_t midx_for_pack(struct multi_pack_index **_m,
|
||||
uint32_t pack_int_id)
|
||||
{
|
||||
struct multi_pack_index *m = *_m;
|
||||
while (m && pack_int_id < m->num_packs_in_base)
|
||||
m = m->base_midx;
|
||||
|
||||
if (!m)
|
||||
BUG("NULL multi-pack-index for pack ID: %"PRIu32, pack_int_id);
|
||||
|
||||
if (pack_int_id >= m->num_packs + m->num_packs_in_base)
|
||||
die(_("bad pack-int-id: %u (%u total packs)"),
|
||||
pack_int_id, m->num_packs + m->num_packs_in_base);
|
||||
|
||||
*_m = m;
|
||||
|
||||
return pack_int_id - m->num_packs_in_base;
|
||||
}
|
||||
|
||||
int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,
|
||||
uint32_t pack_int_id)
|
||||
{
|
||||
struct strbuf pack_name = STRBUF_INIT;
|
||||
struct packed_git *p;
|
||||
|
||||
if (pack_int_id >= m->num_packs)
|
||||
die(_("bad pack-int-id: %u (%u total packs)"),
|
||||
pack_int_id, m->num_packs);
|
||||
pack_int_id = midx_for_pack(&m, pack_int_id);
|
||||
|
||||
if (m->packs[pack_int_id])
|
||||
return 0;
|
||||
|
|
@ -271,41 +473,71 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t
|
|||
return 0;
|
||||
}
|
||||
|
||||
struct packed_git *nth_midxed_pack(struct multi_pack_index *m,
|
||||
uint32_t pack_int_id)
|
||||
{
|
||||
uint32_t local_pack_int_id = midx_for_pack(&m, pack_int_id);
|
||||
return m->packs[local_pack_int_id];
|
||||
}
|
||||
|
||||
#define MIDX_CHUNK_BITMAPPED_PACKS_WIDTH (2 * sizeof(uint32_t))
|
||||
|
||||
int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,
|
||||
struct bitmapped_pack *bp, uint32_t pack_int_id)
|
||||
{
|
||||
uint32_t local_pack_int_id = midx_for_pack(&m, pack_int_id);
|
||||
|
||||
if (!m->chunk_bitmapped_packs)
|
||||
return error(_("MIDX does not contain the BTMP chunk"));
|
||||
|
||||
if (prepare_midx_pack(r, m, pack_int_id))
|
||||
return error(_("could not load bitmapped pack %"PRIu32), pack_int_id);
|
||||
|
||||
bp->p = m->packs[pack_int_id];
|
||||
bp->p = m->packs[local_pack_int_id];
|
||||
bp->bitmap_pos = get_be32((char *)m->chunk_bitmapped_packs +
|
||||
MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * pack_int_id);
|
||||
MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * local_pack_int_id);
|
||||
bp->bitmap_nr = get_be32((char *)m->chunk_bitmapped_packs +
|
||||
MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * pack_int_id +
|
||||
MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * local_pack_int_id +
|
||||
sizeof(uint32_t));
|
||||
bp->pack_int_id = pack_int_id;
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result)
|
||||
int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m,
|
||||
uint32_t *result)
|
||||
{
|
||||
return bsearch_hash(oid->hash, m->chunk_oid_fanout, m->chunk_oid_lookup,
|
||||
the_hash_algo->rawsz, result);
|
||||
int ret = bsearch_hash(oid->hash, m->chunk_oid_fanout,
|
||||
m->chunk_oid_lookup, the_hash_algo->rawsz,
|
||||
result);
|
||||
if (result)
|
||||
*result += m->num_objects_in_base;
|
||||
return ret;
|
||||
}
|
||||
|
||||
int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m,
|
||||
uint32_t *result)
|
||||
{
|
||||
for (; m; m = m->base_midx)
|
||||
if (bsearch_one_midx(oid, m, result))
|
||||
return 1;
|
||||
return 0;
|
||||
}
|
||||
|
||||
int midx_has_oid(struct multi_pack_index *m, const struct object_id *oid)
|
||||
{
|
||||
return bsearch_midx(oid, m, NULL);
|
||||
}
|
||||
|
||||
struct object_id *nth_midxed_object_oid(struct object_id *oid,
|
||||
struct multi_pack_index *m,
|
||||
uint32_t n)
|
||||
{
|
||||
if (n >= m->num_objects)
|
||||
if (n >= m->num_objects + m->num_objects_in_base)
|
||||
return NULL;
|
||||
|
||||
n = midx_for_object(&m, n);
|
||||
|
||||
oidread(oid, m->chunk_oid_lookup + st_mult(m->hash_len, n),
|
||||
the_repository->hash_algo);
|
||||
return oid;
|
||||
|
|
@ -316,6 +548,8 @@ off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)
|
|||
const unsigned char *offset_data;
|
||||
uint32_t offset32;
|
||||
|
||||
pos = midx_for_object(&m, pos);
|
||||
|
||||
offset_data = m->chunk_object_offsets + (off_t)pos * MIDX_CHUNK_OFFSET_WIDTH;
|
||||
offset32 = get_be32(offset_data + sizeof(uint32_t));
|
||||
|
||||
|
|
@ -334,8 +568,10 @@ off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos)
|
|||
|
||||
uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos)
|
||||
{
|
||||
return get_be32(m->chunk_object_offsets +
|
||||
(off_t)pos * MIDX_CHUNK_OFFSET_WIDTH);
|
||||
pos = midx_for_object(&m, pos);
|
||||
|
||||
return m->num_packs_in_base + get_be32(m->chunk_object_offsets +
|
||||
(off_t)pos * MIDX_CHUNK_OFFSET_WIDTH);
|
||||
}
|
||||
|
||||
int fill_midx_entry(struct repository *r,
|
||||
|
|
@ -350,14 +586,12 @@ int fill_midx_entry(struct repository *r,
|
|||
if (!bsearch_midx(oid, m, &pos))
|
||||
return 0;
|
||||
|
||||
if (pos >= m->num_objects)
|
||||
return 0;
|
||||
|
||||
midx_for_object(&m, pos);
|
||||
pack_int_id = nth_midxed_pack_int_id(m, pos);
|
||||
|
||||
if (prepare_midx_pack(r, m, pack_int_id))
|
||||
return 0;
|
||||
p = m->packs[pack_int_id];
|
||||
p = m->packs[pack_int_id - m->num_packs_in_base];
|
||||
|
||||
/*
|
||||
* We are about to tell the caller where they can locate the
|
||||
|
|
@ -411,8 +645,8 @@ int cmp_idx_or_pack_name(const char *idx_or_pack_name,
|
|||
return strcmp(idx_or_pack_name, idx_name);
|
||||
}
|
||||
|
||||
int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name,
|
||||
uint32_t *pos)
|
||||
static int midx_contains_pack_1(struct multi_pack_index *m,
|
||||
const char *idx_or_pack_name)
|
||||
{
|
||||
uint32_t first = 0, last = m->num_packs;
|
||||
|
||||
|
|
@ -423,11 +657,8 @@ int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name,
|
|||
|
||||
current = m->pack_names[mid];
|
||||
cmp = cmp_idx_or_pack_name(idx_or_pack_name, current);
|
||||
if (!cmp) {
|
||||
if (pos)
|
||||
*pos = mid;
|
||||
if (!cmp)
|
||||
return 1;
|
||||
}
|
||||
if (cmp > 0) {
|
||||
first = mid + 1;
|
||||
continue;
|
||||
|
|
@ -440,19 +671,25 @@ int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name,
|
|||
|
||||
int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name)
|
||||
{
|
||||
return midx_locate_pack(m, idx_or_pack_name, NULL);
|
||||
for (; m; m = m->base_midx)
|
||||
if (midx_contains_pack_1(m, idx_or_pack_name))
|
||||
return 1;
|
||||
return 0;
|
||||
}
|
||||
|
||||
int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id)
|
||||
{
|
||||
if (m->preferred_pack_idx == -1) {
|
||||
uint32_t midx_pos;
|
||||
if (load_midx_revindex(m) < 0) {
|
||||
m->preferred_pack_idx = -2;
|
||||
return -1;
|
||||
}
|
||||
|
||||
m->preferred_pack_idx =
|
||||
nth_midxed_pack_int_id(m, pack_pos_to_midx(m, 0));
|
||||
midx_pos = pack_pos_to_midx(m, m->num_objects_in_base);
|
||||
|
||||
m->preferred_pack_idx = nth_midxed_pack_int_id(m, midx_pos);
|
||||
|
||||
} else if (m->preferred_pack_idx == -2)
|
||||
return -1; /* no revindex */
|
||||
|
||||
|
|
@ -494,7 +731,8 @@ int midx_checksum_valid(struct multi_pack_index *m)
|
|||
}
|
||||
|
||||
struct clear_midx_data {
|
||||
char *keep;
|
||||
char **keep;
|
||||
uint32_t keep_nr;
|
||||
const char *ext;
|
||||
};
|
||||
|
||||
|
|
@ -502,32 +740,63 @@ static void clear_midx_file_ext(const char *full_path, size_t full_path_len UNUS
|
|||
const char *file_name, void *_data)
|
||||
{
|
||||
struct clear_midx_data *data = _data;
|
||||
uint32_t i;
|
||||
|
||||
if (!(starts_with(file_name, "multi-pack-index-") &&
|
||||
ends_with(file_name, data->ext)))
|
||||
return;
|
||||
if (data->keep && !strcmp(data->keep, file_name))
|
||||
return;
|
||||
|
||||
for (i = 0; i < data->keep_nr; i++) {
|
||||
if (!strcmp(data->keep[i], file_name))
|
||||
return;
|
||||
}
|
||||
if (unlink(full_path))
|
||||
die_errno(_("failed to remove %s"), full_path);
|
||||
}
|
||||
|
||||
void clear_midx_files_ext(const char *object_dir, const char *ext,
|
||||
unsigned char *keep_hash)
|
||||
const char *keep_hash)
|
||||
{
|
||||
struct clear_midx_data data;
|
||||
memset(&data, 0, sizeof(struct clear_midx_data));
|
||||
|
||||
if (keep_hash)
|
||||
data.keep = xstrfmt("multi-pack-index-%s%s",
|
||||
hash_to_hex(keep_hash), ext);
|
||||
if (keep_hash) {
|
||||
ALLOC_ARRAY(data.keep, 1);
|
||||
|
||||
data.keep[0] = xstrfmt("multi-pack-index-%s.%s", keep_hash, ext);
|
||||
data.keep_nr = 1;
|
||||
}
|
||||
data.ext = ext;
|
||||
|
||||
for_each_file_in_pack_dir(object_dir,
|
||||
clear_midx_file_ext,
|
||||
&data);
|
||||
|
||||
if (keep_hash)
|
||||
free(data.keep[0]);
|
||||
free(data.keep);
|
||||
}
|
||||
|
||||
void clear_incremental_midx_files_ext(const char *object_dir, const char *ext,
|
||||
char **keep_hashes,
|
||||
uint32_t hashes_nr)
|
||||
{
|
||||
struct clear_midx_data data;
|
||||
uint32_t i;
|
||||
|
||||
memset(&data, 0, sizeof(struct clear_midx_data));
|
||||
|
||||
ALLOC_ARRAY(data.keep, hashes_nr);
|
||||
for (i = 0; i < hashes_nr; i++)
|
||||
data.keep[i] = xstrfmt("multi-pack-index-%s.%s", keep_hashes[i],
|
||||
ext);
|
||||
data.keep_nr = hashes_nr;
|
||||
data.ext = ext;
|
||||
|
||||
for_each_file_in_pack_subdir(object_dir, "multi-pack-index.d",
|
||||
clear_midx_file_ext, &data);
|
||||
|
||||
for (i = 0; i < hashes_nr; i++)
|
||||
free(data.keep[i]);
|
||||
free(data.keep);
|
||||
}
|
||||
|
||||
|
|
@ -545,8 +814,8 @@ void clear_midx_file(struct repository *r)
|
|||
if (remove_path(midx.buf))
|
||||
die(_("failed to clear multi-pack-index at %s"), midx.buf);
|
||||
|
||||
clear_midx_files_ext(r->objects->odb->path, ".bitmap", NULL);
|
||||
clear_midx_files_ext(r->objects->odb->path, ".rev", NULL);
|
||||
clear_midx_files_ext(r->objects->odb->path, MIDX_EXT_BITMAP, NULL);
|
||||
clear_midx_files_ext(r->objects->odb->path, MIDX_EXT_REV, NULL);
|
||||
|
||||
strbuf_release(&midx);
|
||||
}
|
||||
|
|
@ -596,6 +865,7 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag
|
|||
uint32_t i;
|
||||
struct progress *progress = NULL;
|
||||
struct multi_pack_index *m = load_multi_pack_index(object_dir, 1);
|
||||
struct multi_pack_index *curr;
|
||||
verify_midx_error = 0;
|
||||
|
||||
if (!m) {
|
||||
|
|
@ -618,8 +888,8 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag
|
|||
|
||||
if (flags & MIDX_PROGRESS)
|
||||
progress = start_delayed_progress(_("Looking for referenced packfiles"),
|
||||
m->num_packs);
|
||||
for (i = 0; i < m->num_packs; i++) {
|
||||
m->num_packs + m->num_packs_in_base);
|
||||
for (i = 0; i < m->num_packs + m->num_packs_in_base; i++) {
|
||||
if (prepare_midx_pack(r, m, i))
|
||||
midx_report("failed to load pack in position %d", i);
|
||||
|
||||
|
|
@ -639,17 +909,20 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag
|
|||
if (flags & MIDX_PROGRESS)
|
||||
progress = start_sparse_progress(_("Verifying OID order in multi-pack-index"),
|
||||
m->num_objects - 1);
|
||||
for (i = 0; i < m->num_objects - 1; i++) {
|
||||
struct object_id oid1, oid2;
|
||||
|
||||
nth_midxed_object_oid(&oid1, m, i);
|
||||
nth_midxed_object_oid(&oid2, m, i + 1);
|
||||
for (curr = m; curr; curr = curr->base_midx) {
|
||||
for (i = 0; i < m->num_objects - 1; i++) {
|
||||
struct object_id oid1, oid2;
|
||||
|
||||
if (oidcmp(&oid1, &oid2) >= 0)
|
||||
midx_report(_("oid lookup out of order: oid[%d] = %s >= %s = oid[%d]"),
|
||||
i, oid_to_hex(&oid1), oid_to_hex(&oid2), i + 1);
|
||||
nth_midxed_object_oid(&oid1, m, m->num_objects_in_base + i);
|
||||
nth_midxed_object_oid(&oid2, m, m->num_objects_in_base + i + 1);
|
||||
|
||||
midx_display_sparse_progress(progress, i + 1);
|
||||
if (oidcmp(&oid1, &oid2) >= 0)
|
||||
midx_report(_("oid lookup out of order: oid[%d] = %s >= %s = oid[%d]"),
|
||||
i, oid_to_hex(&oid1), oid_to_hex(&oid2), i + 1);
|
||||
|
||||
midx_display_sparse_progress(progress, i + 1);
|
||||
}
|
||||
}
|
||||
stop_progress(&progress);
|
||||
|
||||
|
|
@ -659,8 +932,8 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag
|
|||
* each of the objects and only require 1 packfile to be open at a
|
||||
* time.
|
||||
*/
|
||||
ALLOC_ARRAY(pairs, m->num_objects);
|
||||
for (i = 0; i < m->num_objects; i++) {
|
||||
ALLOC_ARRAY(pairs, m->num_objects + m->num_objects_in_base);
|
||||
for (i = 0; i < m->num_objects + m->num_objects_in_base; i++) {
|
||||
pairs[i].pos = i;
|
||||
pairs[i].pack_int_id = nth_midxed_pack_int_id(m, i);
|
||||
}
|
||||
|
|
@ -674,16 +947,18 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag
|
|||
|
||||
if (flags & MIDX_PROGRESS)
|
||||
progress = start_sparse_progress(_("Verifying object offsets"), m->num_objects);
|
||||
for (i = 0; i < m->num_objects; i++) {
|
||||
for (i = 0; i < m->num_objects + m->num_objects_in_base; i++) {
|
||||
struct object_id oid;
|
||||
struct pack_entry e;
|
||||
off_t m_offset, p_offset;
|
||||
|
||||
if (i > 0 && pairs[i-1].pack_int_id != pairs[i].pack_int_id &&
|
||||
m->packs[pairs[i-1].pack_int_id])
|
||||
{
|
||||
close_pack_fd(m->packs[pairs[i-1].pack_int_id]);
|
||||
close_pack_index(m->packs[pairs[i-1].pack_int_id]);
|
||||
nth_midxed_pack(m, pairs[i-1].pack_int_id)) {
|
||||
uint32_t pack_int_id = pairs[i-1].pack_int_id;
|
||||
struct packed_git *p = nth_midxed_pack(m, pack_int_id);
|
||||
|
||||
close_pack_fd(p);
|
||||
close_pack_index(p);
|
||||
}
|
||||
|
||||
nth_midxed_object_oid(&oid, m, pairs[i].pos);
|
||||
|
|
|
|||
26
midx.h
26
midx.h
|
|
@ -24,12 +24,13 @@ struct bitmapped_pack;
|
|||
#define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* "OOFF" */
|
||||
#define MIDX_CHUNKID_LARGEOFFSETS 0x4c4f4646 /* "LOFF" */
|
||||
#define MIDX_CHUNKID_REVINDEX 0x52494458 /* "RIDX" */
|
||||
#define MIDX_CHUNKID_BASE 0x42415345 /* "BASE" */
|
||||
#define MIDX_CHUNK_OFFSET_WIDTH (2 * sizeof(uint32_t))
|
||||
#define MIDX_LARGE_OFFSET_NEEDED 0x80000000
|
||||
|
||||
#define GIT_TEST_MULTI_PACK_INDEX "GIT_TEST_MULTI_PACK_INDEX"
|
||||
#define GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP \
|
||||
"GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP"
|
||||
#define GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL \
|
||||
"GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL"
|
||||
|
||||
struct multi_pack_index {
|
||||
struct multi_pack_index *next;
|
||||
|
|
@ -50,6 +51,7 @@ struct multi_pack_index {
|
|||
int preferred_pack_idx;
|
||||
|
||||
int local;
|
||||
int has_chain;
|
||||
|
||||
const unsigned char *chunk_pack_names;
|
||||
size_t chunk_pack_names_len;
|
||||
|
|
@ -63,6 +65,10 @@ struct multi_pack_index {
|
|||
const unsigned char *chunk_revindex;
|
||||
size_t chunk_revindex_len;
|
||||
|
||||
struct multi_pack_index *base_midx;
|
||||
uint32_t num_objects_in_base;
|
||||
uint32_t num_packs_in_base;
|
||||
|
||||
const char **pack_names;
|
||||
struct packed_git **packs;
|
||||
char object_dir[FLEX_ARRAY];
|
||||
|
|
@ -73,20 +79,32 @@ struct multi_pack_index {
|
|||
#define MIDX_WRITE_BITMAP (1 << 2)
|
||||
#define MIDX_WRITE_BITMAP_HASH_CACHE (1 << 3)
|
||||
#define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4)
|
||||
#define MIDX_WRITE_INCREMENTAL (1 << 5)
|
||||
|
||||
#define MIDX_EXT_REV "rev"
|
||||
#define MIDX_EXT_BITMAP "bitmap"
|
||||
#define MIDX_EXT_MIDX "midx"
|
||||
|
||||
const unsigned char *get_midx_checksum(struct multi_pack_index *m);
|
||||
void get_midx_filename(struct strbuf *out, const char *object_dir);
|
||||
void get_midx_filename_ext(struct strbuf *out, const char *object_dir,
|
||||
const unsigned char *hash, const char *ext);
|
||||
void get_midx_chain_dirname(struct strbuf *buf, const char *object_dir);
|
||||
void get_midx_chain_filename(struct strbuf *buf, const char *object_dir);
|
||||
void get_split_midx_filename_ext(struct strbuf *buf, const char *object_dir,
|
||||
const unsigned char *hash, const char *ext);
|
||||
|
||||
struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local);
|
||||
int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id);
|
||||
struct packed_git *nth_midxed_pack(struct multi_pack_index *m,
|
||||
uint32_t pack_int_id);
|
||||
int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m,
|
||||
struct bitmapped_pack *bp, uint32_t pack_int_id);
|
||||
int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result);
|
||||
int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m,
|
||||
uint32_t *result);
|
||||
int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m,
|
||||
uint32_t *result);
|
||||
int midx_has_oid(struct multi_pack_index *m, const struct object_id *oid);
|
||||
off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos);
|
||||
uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos);
|
||||
struct object_id *nth_midxed_object_oid(struct object_id *oid,
|
||||
|
|
@ -95,8 +113,6 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid,
|
|||
int fill_midx_entry(struct repository *r, const struct object_id *oid, struct pack_entry *e, struct multi_pack_index *m);
|
||||
int midx_contains_pack(struct multi_pack_index *m,
|
||||
const char *idx_or_pack_name);
|
||||
int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name,
|
||||
uint32_t *pos);
|
||||
int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id);
|
||||
int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local);
|
||||
|
||||
|
|
|
|||
|
|
@ -135,28 +135,32 @@ static int match_hash(unsigned len, const unsigned char *a, const unsigned char
|
|||
static void unique_in_midx(struct multi_pack_index *m,
|
||||
struct disambiguate_state *ds)
|
||||
{
|
||||
uint32_t num, i, first = 0;
|
||||
const struct object_id *current = NULL;
|
||||
int len = ds->len > ds->repo->hash_algo->hexsz ?
|
||||
ds->repo->hash_algo->hexsz : ds->len;
|
||||
num = m->num_objects;
|
||||
for (; m; m = m->base_midx) {
|
||||
uint32_t num, i, first = 0;
|
||||
const struct object_id *current = NULL;
|
||||
int len = ds->len > ds->repo->hash_algo->hexsz ?
|
||||
ds->repo->hash_algo->hexsz : ds->len;
|
||||
|
||||
if (!num)
|
||||
return;
|
||||
if (!m->num_objects)
|
||||
continue;
|
||||
|
||||
bsearch_midx(&ds->bin_pfx, m, &first);
|
||||
num = m->num_objects + m->num_objects_in_base;
|
||||
|
||||
/*
|
||||
* At this point, "first" is the location of the lowest object
|
||||
* with an object name that could match "bin_pfx". See if we have
|
||||
* 0, 1 or more objects that actually match(es).
|
||||
*/
|
||||
for (i = first; i < num && !ds->ambiguous; i++) {
|
||||
struct object_id oid;
|
||||
current = nth_midxed_object_oid(&oid, m, i);
|
||||
if (!match_hash(len, ds->bin_pfx.hash, current->hash))
|
||||
break;
|
||||
update_candidates(ds, current);
|
||||
bsearch_one_midx(&ds->bin_pfx, m, &first);
|
||||
|
||||
/*
|
||||
* At this point, "first" is the location of the lowest
|
||||
* object with an object name that could match
|
||||
* "bin_pfx". See if we have 0, 1 or more objects that
|
||||
* actually match(es).
|
||||
*/
|
||||
for (i = first; i < num && !ds->ambiguous; i++) {
|
||||
struct object_id oid;
|
||||
current = nth_midxed_object_oid(&oid, m, i);
|
||||
if (!match_hash(len, ds->bin_pfx.hash, current->hash))
|
||||
break;
|
||||
update_candidates(ds, current);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -709,37 +713,40 @@ static int repo_extend_abbrev_len(struct repository *r UNUSED,
|
|||
static void find_abbrev_len_for_midx(struct multi_pack_index *m,
|
||||
struct min_abbrev_data *mad)
|
||||
{
|
||||
int match = 0;
|
||||
uint32_t num, first = 0;
|
||||
struct object_id oid;
|
||||
const struct object_id *mad_oid;
|
||||
for (; m; m = m->base_midx) {
|
||||
int match = 0;
|
||||
uint32_t num, first = 0;
|
||||
struct object_id oid;
|
||||
const struct object_id *mad_oid;
|
||||
|
||||
if (!m->num_objects)
|
||||
return;
|
||||
if (!m->num_objects)
|
||||
continue;
|
||||
|
||||
num = m->num_objects;
|
||||
mad_oid = mad->oid;
|
||||
match = bsearch_midx(mad_oid, m, &first);
|
||||
num = m->num_objects + m->num_objects_in_base;
|
||||
mad_oid = mad->oid;
|
||||
match = bsearch_one_midx(mad_oid, m, &first);
|
||||
|
||||
/*
|
||||
* first is now the position in the packfile where we would insert
|
||||
* mad->hash if it does not exist (or the position of mad->hash if
|
||||
* it does exist). Hence, we consider a maximum of two objects
|
||||
* nearby for the abbreviation length.
|
||||
*/
|
||||
mad->init_len = 0;
|
||||
if (!match) {
|
||||
if (nth_midxed_object_oid(&oid, m, first))
|
||||
extend_abbrev_len(&oid, mad);
|
||||
} else if (first < num - 1) {
|
||||
if (nth_midxed_object_oid(&oid, m, first + 1))
|
||||
extend_abbrev_len(&oid, mad);
|
||||
/*
|
||||
* first is now the position in the packfile where we
|
||||
* would insert mad->hash if it does not exist (or the
|
||||
* position of mad->hash if it does exist). Hence, we
|
||||
* consider a maximum of two objects nearby for the
|
||||
* abbreviation length.
|
||||
*/
|
||||
mad->init_len = 0;
|
||||
if (!match) {
|
||||
if (nth_midxed_object_oid(&oid, m, first))
|
||||
extend_abbrev_len(&oid, mad);
|
||||
} else if (first < num - 1) {
|
||||
if (nth_midxed_object_oid(&oid, m, first + 1))
|
||||
extend_abbrev_len(&oid, mad);
|
||||
}
|
||||
if (first > 0) {
|
||||
if (nth_midxed_object_oid(&oid, m, first - 1))
|
||||
extend_abbrev_len(&oid, mad);
|
||||
}
|
||||
mad->init_len = mad->cur_len;
|
||||
}
|
||||
if (first > 0) {
|
||||
if (nth_midxed_object_oid(&oid, m, first - 1))
|
||||
extend_abbrev_len(&oid, mad);
|
||||
}
|
||||
mad->init_len = mad->cur_len;
|
||||
}
|
||||
|
||||
static void find_abbrev_len_for_pack(struct packed_git *p,
|
||||
|
|
|
|||
21
packfile.c
21
packfile.c
|
|
@ -815,9 +815,10 @@ static void report_pack_garbage(struct string_list *list)
|
|||
report_helper(list, seen_bits, first, list->nr);
|
||||
}
|
||||
|
||||
void for_each_file_in_pack_dir(const char *objdir,
|
||||
each_file_in_pack_dir_fn fn,
|
||||
void *data)
|
||||
void for_each_file_in_pack_subdir(const char *objdir,
|
||||
const char *subdir,
|
||||
each_file_in_pack_dir_fn fn,
|
||||
void *data)
|
||||
{
|
||||
struct strbuf path = STRBUF_INIT;
|
||||
size_t dirnamelen;
|
||||
|
|
@ -826,6 +827,8 @@ void for_each_file_in_pack_dir(const char *objdir,
|
|||
|
||||
strbuf_addstr(&path, objdir);
|
||||
strbuf_addstr(&path, "/pack");
|
||||
if (subdir)
|
||||
strbuf_addf(&path, "/%s", subdir);
|
||||
dir = opendir(path.buf);
|
||||
if (!dir) {
|
||||
if (errno != ENOENT)
|
||||
|
|
@ -847,6 +850,13 @@ void for_each_file_in_pack_dir(const char *objdir,
|
|||
strbuf_release(&path);
|
||||
}
|
||||
|
||||
void for_each_file_in_pack_dir(const char *objdir,
|
||||
each_file_in_pack_dir_fn fn,
|
||||
void *data)
|
||||
{
|
||||
for_each_file_in_pack_subdir(objdir, NULL, fn, data);
|
||||
}
|
||||
|
||||
struct prepare_pack_data {
|
||||
struct repository *r;
|
||||
struct string_list *garbage;
|
||||
|
|
@ -880,7 +890,8 @@ static void prepare_pack(const char *full_name, size_t full_name_len,
|
|||
if (!report_garbage)
|
||||
return;
|
||||
|
||||
if (!strcmp(file_name, "multi-pack-index"))
|
||||
if (!strcmp(file_name, "multi-pack-index") ||
|
||||
!strcmp(file_name, "multi-pack-index.d"))
|
||||
return;
|
||||
if (starts_with(file_name, "multi-pack-index") &&
|
||||
(ends_with(file_name, ".bitmap") || ends_with(file_name, ".rev")))
|
||||
|
|
@ -1064,7 +1075,7 @@ struct packed_git *get_all_packs(struct repository *r)
|
|||
prepare_packed_git(r);
|
||||
for (m = r->objects->multi_pack_index; m; m = m->next) {
|
||||
uint32_t i;
|
||||
for (i = 0; i < m->num_packs; i++)
|
||||
for (i = 0; i < m->num_packs + m->num_packs_in_base; i++)
|
||||
prepare_midx_pack(r, m, i);
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -55,6 +55,10 @@ struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);
|
|||
|
||||
typedef void each_file_in_pack_dir_fn(const char *full_path, size_t full_path_len,
|
||||
const char *file_name, void *data);
|
||||
void for_each_file_in_pack_subdir(const char *objdir,
|
||||
const char *subdir,
|
||||
each_file_in_pack_dir_fn fn,
|
||||
void *data);
|
||||
void for_each_file_in_pack_dir(const char *objdir,
|
||||
each_file_in_pack_dir_fn fn,
|
||||
void *data);
|
||||
|
|
|
|||
6
t/README
6
t/README
|
|
@ -445,9 +445,9 @@ GIT_TEST_MULTI_PACK_INDEX=<boolean>, when true, forces the multi-pack-
|
|||
index to be written after every 'git repack' command, and overrides the
|
||||
'core.multiPackIndex' setting to true.
|
||||
|
||||
GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=<boolean>, when true, sets the
|
||||
'--bitmap' option on all invocations of 'git multi-pack-index write',
|
||||
and ignores pack-objects' '--write-bitmap-index'.
|
||||
GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=<boolean>, when true, sets
|
||||
the '--incremental' option on all invocations of 'git multi-pack-index
|
||||
write'.
|
||||
|
||||
GIT_TEST_SIDEBAND_ALL=<boolean>, when true, overrides the
|
||||
'uploadpack.allowSidebandAll' setting to true, and when false, forces
|
||||
|
|
|
|||
|
|
@ -9,8 +9,10 @@
|
|||
#include "packfile.h"
|
||||
#include "setup.h"
|
||||
#include "gettext.h"
|
||||
#include "pack-revindex.h"
|
||||
|
||||
static int read_midx_file(const char *object_dir, int show_objects)
|
||||
static int read_midx_file(const char *object_dir, const char *checksum,
|
||||
int show_objects)
|
||||
{
|
||||
uint32_t i;
|
||||
struct multi_pack_index *m;
|
||||
|
|
@ -21,6 +23,13 @@ static int read_midx_file(const char *object_dir, int show_objects)
|
|||
if (!m)
|
||||
return 1;
|
||||
|
||||
if (checksum) {
|
||||
while (m && strcmp(hash_to_hex(get_midx_checksum(m)), checksum))
|
||||
m = m->base_midx;
|
||||
if (!m)
|
||||
return 1;
|
||||
}
|
||||
|
||||
printf("header: %08x %d %d %d %d\n",
|
||||
m->signature,
|
||||
m->version,
|
||||
|
|
@ -54,7 +63,8 @@ static int read_midx_file(const char *object_dir, int show_objects)
|
|||
struct pack_entry e;
|
||||
|
||||
for (i = 0; i < m->num_objects; i++) {
|
||||
nth_midxed_object_oid(&oid, m, i);
|
||||
nth_midxed_object_oid(&oid, m,
|
||||
i + m->num_objects_in_base);
|
||||
fill_midx_entry(the_repository, &oid, &e, m);
|
||||
|
||||
printf("%s %"PRIu64"\t%s\n",
|
||||
|
|
@ -111,7 +121,7 @@ static int read_midx_bitmapped_packs(const char *object_dir)
|
|||
if (!midx)
|
||||
return 1;
|
||||
|
||||
for (i = 0; i < midx->num_packs; i++) {
|
||||
for (i = 0; i < midx->num_packs + midx->num_packs_in_base; i++) {
|
||||
if (nth_bitmapped_pack(the_repository, midx, &pack, i) < 0)
|
||||
return 1;
|
||||
|
||||
|
|
@ -127,16 +137,16 @@ static int read_midx_bitmapped_packs(const char *object_dir)
|
|||
|
||||
int cmd__read_midx(int argc, const char **argv)
|
||||
{
|
||||
if (!(argc == 2 || argc == 3))
|
||||
usage("read-midx [--show-objects|--checksum|--preferred-pack|--bitmap] <object-dir>");
|
||||
if (!(argc == 2 || argc == 3 || argc == 4))
|
||||
usage("read-midx [--show-objects|--checksum|--preferred-pack|--bitmap] <object-dir> <checksum>");
|
||||
|
||||
if (!strcmp(argv[1], "--show-objects"))
|
||||
return read_midx_file(argv[2], 1);
|
||||
return read_midx_file(argv[2], argv[3], 1);
|
||||
else if (!strcmp(argv[1], "--checksum"))
|
||||
return read_midx_checksum(argv[2]);
|
||||
else if (!strcmp(argv[1], "--preferred-pack"))
|
||||
return read_midx_preferred_pack(argv[2]);
|
||||
else if (!strcmp(argv[1], "--bitmap"))
|
||||
return read_midx_bitmapped_packs(argv[2]);
|
||||
return read_midx_file(argv[1], 0);
|
||||
return read_midx_file(argv[1], argv[2], 0);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,8 @@
|
|||
# Helpers for scripts testing bitmap functionality; see t5310 for
|
||||
# example usage.
|
||||
|
||||
. "$TEST_DIRECTORY"/lib-midx.sh
|
||||
|
||||
objdir=.git/objects
|
||||
midx=$objdir/pack/multi-pack-index
|
||||
|
||||
|
|
@ -264,10 +266,6 @@ have_delta () {
|
|||
test_cmp expect actual
|
||||
}
|
||||
|
||||
midx_checksum () {
|
||||
test-tool read-midx --checksum "$1"
|
||||
}
|
||||
|
||||
# midx_pack_source <obj>
|
||||
midx_pack_source () {
|
||||
test-tool read-midx --show-objects .git/objects | grep "^$1 " | cut -f2
|
||||
|
|
|
|||
|
|
@ -6,3 +6,31 @@ test_midx_consistent () {
|
|||
test_cmp expect actual &&
|
||||
git multi-pack-index --object-dir=$1 verify
|
||||
}
|
||||
|
||||
midx_checksum () {
|
||||
test-tool read-midx --checksum "$1"
|
||||
}
|
||||
|
||||
midx_git_two_modes () {
|
||||
git -c core.multiPackIndex=false $1 >expect &&
|
||||
git -c core.multiPackIndex=true $1 >actual &&
|
||||
if [ "$2" = "sorted" ]
|
||||
then
|
||||
sort <expect >expect.sorted &&
|
||||
mv expect.sorted expect &&
|
||||
sort <actual >actual.sorted &&
|
||||
mv actual.sorted actual
|
||||
fi &&
|
||||
test_cmp expect actual
|
||||
}
|
||||
|
||||
compare_results_with_midx () {
|
||||
MSG=$1
|
||||
test_expect_success "check normal git operations: $MSG" '
|
||||
midx_git_two_modes "rev-list --objects --all" &&
|
||||
midx_git_two_modes "log --raw" &&
|
||||
midx_git_two_modes "count-objects --verbose" &&
|
||||
midx_git_two_modes "cat-file --batch-all-objects --batch-check" &&
|
||||
midx_git_two_modes "cat-file --batch-all-objects --batch-check --unordered" sorted
|
||||
'
|
||||
}
|
||||
|
|
|
|||
|
|
@ -5,8 +5,6 @@ test_description='partial clone'
|
|||
. ./test-lib.sh
|
||||
. "$TEST_DIRECTORY"/lib-terminal.sh
|
||||
|
||||
# missing promisor objects cause repacks which write bitmaps to fail
|
||||
GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0
|
||||
# When enabled, some commands will write commit-graphs. This causes fsck
|
||||
# to fail when delete_object() is called because fsck will attempt to
|
||||
# verify the out-of-sync commit graph.
|
||||
|
|
|
|||
|
|
@ -5,10 +5,6 @@ test_description='exercise basic bitmap functionality'
|
|||
. ./test-lib.sh
|
||||
. "$TEST_DIRECTORY"/lib-bitmap.sh
|
||||
|
||||
# t5310 deals only with single-pack bitmaps, so don't write MIDX bitmaps in
|
||||
# their place.
|
||||
GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0
|
||||
|
||||
# Likewise, allow individual tests to control whether or not they use
|
||||
# the boundary-based traversal.
|
||||
sane_unset GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL
|
||||
|
|
|
|||
|
|
@ -7,11 +7,11 @@ TEST_PASSES_SANITIZE_LEAK=true
|
|||
|
||||
clear_base () {
|
||||
test_when_finished 'restore_base' &&
|
||||
rm -f $base
|
||||
rm -r -f $base
|
||||
}
|
||||
|
||||
restore_base () {
|
||||
cp base-backup/* .git/objects/pack/
|
||||
cp -r base-backup/* .git/objects/pack/
|
||||
}
|
||||
|
||||
do_pack () {
|
||||
|
|
@ -64,9 +64,9 @@ test_expect_success 'set up base packfile and variables' '
|
|||
git commit -m base &&
|
||||
git repack -ad &&
|
||||
base=$(echo .git/objects/pack/*) &&
|
||||
chmod +w $base &&
|
||||
chmod -R +w $base &&
|
||||
mkdir base-backup &&
|
||||
cp $base base-backup/ &&
|
||||
cp -r $base base-backup/ &&
|
||||
object=$(git rev-parse HEAD:file)
|
||||
'
|
||||
|
||||
|
|
|
|||
|
|
@ -3,8 +3,11 @@
|
|||
test_description='multi-pack-indexes'
|
||||
. ./test-lib.sh
|
||||
. "$TEST_DIRECTORY"/lib-chunk.sh
|
||||
. "$TEST_DIRECTORY"/lib-midx.sh
|
||||
|
||||
GIT_TEST_MULTI_PACK_INDEX=0
|
||||
GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0
|
||||
GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0
|
||||
objdir=.git/objects
|
||||
|
||||
HASH_LEN=$(test_oid rawsz)
|
||||
|
|
@ -107,30 +110,6 @@ test_expect_success 'write midx with one v1 pack' '
|
|||
midx_read_expect 1 18 4 $objdir
|
||||
'
|
||||
|
||||
midx_git_two_modes () {
|
||||
git -c core.multiPackIndex=false $1 >expect &&
|
||||
git -c core.multiPackIndex=true $1 >actual &&
|
||||
if [ "$2" = "sorted" ]
|
||||
then
|
||||
sort <expect >expect.sorted &&
|
||||
mv expect.sorted expect &&
|
||||
sort <actual >actual.sorted &&
|
||||
mv actual.sorted actual
|
||||
fi &&
|
||||
test_cmp expect actual
|
||||
}
|
||||
|
||||
compare_results_with_midx () {
|
||||
MSG=$1
|
||||
test_expect_success "check normal git operations: $MSG" '
|
||||
midx_git_two_modes "rev-list --objects --all" &&
|
||||
midx_git_two_modes "log --raw" &&
|
||||
midx_git_two_modes "count-objects --verbose" &&
|
||||
midx_git_two_modes "cat-file --batch-all-objects --batch-check" &&
|
||||
midx_git_two_modes "cat-file --batch-all-objects --batch-check --unordered" sorted
|
||||
'
|
||||
}
|
||||
|
||||
test_expect_success 'write midx with one v2 pack' '
|
||||
git pack-objects --index-version=2,0x40 $objdir/pack/test <obj-list &&
|
||||
git multi-pack-index --object-dir=$objdir write &&
|
||||
|
|
@ -600,8 +579,7 @@ test_expect_success 'repack preserves multi-pack-index when creating packs' '
|
|||
compare_results_with_midx "after repack"
|
||||
|
||||
test_expect_success 'multi-pack-index and pack-bitmap' '
|
||||
GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
|
||||
git -c repack.writeBitmaps=true repack -ad &&
|
||||
git -c repack.writeBitmaps=true repack -ad &&
|
||||
git multi-pack-index write &&
|
||||
git rev-list --test-bitmap HEAD
|
||||
'
|
||||
|
|
|
|||
|
|
@ -4,10 +4,10 @@ test_description='exercise basic multi-pack bitmap functionality'
|
|||
. ./test-lib.sh
|
||||
. "${TEST_DIRECTORY}/lib-bitmap.sh"
|
||||
|
||||
# We'll be writing our own midx and bitmaps, so avoid getting confused by the
|
||||
# We'll be writing our own MIDX, so avoid getting confused by the
|
||||
# automatic ones.
|
||||
GIT_TEST_MULTI_PACK_INDEX=0
|
||||
GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0
|
||||
GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0
|
||||
|
||||
# This test exercise multi-pack bitmap functionality where the object order is
|
||||
# stored and read from a special chunk within the MIDX, so use the default
|
||||
|
|
|
|||
|
|
@ -5,10 +5,10 @@ test_description='exercise basic multi-pack bitmap functionality (.rev files)'
|
|||
. ./test-lib.sh
|
||||
. "${TEST_DIRECTORY}/lib-bitmap.sh"
|
||||
|
||||
# We'll be writing our own midx and bitmaps, so avoid getting confused by the
|
||||
# automatic ones.
|
||||
# We'll be writing our own MIDX, so avoid getting confused by the automatic
|
||||
# ones.
|
||||
GIT_TEST_MULTI_PACK_INDEX=0
|
||||
GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0
|
||||
GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0
|
||||
|
||||
# Unlike t5326, this test exercise multi-pack bitmap functionality where the
|
||||
# object order is stored in a separate .rev file.
|
||||
|
|
|
|||
|
|
@ -6,6 +6,8 @@ TEST_PASSES_SANITIZE_LEAK=true
|
|||
. ./test-lib.sh
|
||||
. "$TEST_DIRECTORY"/lib-bitmap.sh
|
||||
|
||||
GIT_TEST_MULTI_PACK_INDEX=0
|
||||
GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0
|
||||
objdir=.git/objects
|
||||
packdir=$objdir/pack
|
||||
|
||||
|
|
|
|||
46
t/t5334-incremental-multi-pack-index.sh
Executable file
46
t/t5334-incremental-multi-pack-index.sh
Executable file
|
|
@ -0,0 +1,46 @@
|
|||
#!/bin/sh
|
||||
|
||||
test_description='incremental multi-pack-index'
|
||||
. ./test-lib.sh
|
||||
. "$TEST_DIRECTORY"/lib-midx.sh
|
||||
|
||||
GIT_TEST_MULTI_PACK_INDEX=0
|
||||
export GIT_TEST_MULTI_PACK_INDEX
|
||||
|
||||
objdir=.git/objects
|
||||
packdir=$objdir/pack
|
||||
midxdir=$packdir/multi-pack-index.d
|
||||
midx_chain=$midxdir/multi-pack-index-chain
|
||||
|
||||
test_expect_success 'convert non-incremental MIDX to incremental' '
|
||||
test_commit base &&
|
||||
git repack -ad &&
|
||||
git multi-pack-index write &&
|
||||
|
||||
test_path_is_file $packdir/multi-pack-index &&
|
||||
old_hash="$(midx_checksum $objdir)" &&
|
||||
|
||||
test_commit other &&
|
||||
git repack -d &&
|
||||
git multi-pack-index write --incremental &&
|
||||
|
||||
test_path_is_missing $packdir/multi-pack-index &&
|
||||
test_path_is_file $midx_chain &&
|
||||
test_line_count = 2 $midx_chain &&
|
||||
grep $old_hash $midx_chain
|
||||
'
|
||||
|
||||
compare_results_with_midx 'incremental MIDX'
|
||||
|
||||
test_expect_success 'convert incremental to non-incremental' '
|
||||
test_commit squash &&
|
||||
git repack -d &&
|
||||
git multi-pack-index write &&
|
||||
|
||||
test_path_is_file $packdir/multi-pack-index &&
|
||||
test_dir_is_empty $midxdir
|
||||
'
|
||||
|
||||
compare_results_with_midx 'non-incremental MIDX conversion'
|
||||
|
||||
test_done
|
||||
|
|
@ -7,6 +7,9 @@ test_description='git repack works correctly'
|
|||
. "${TEST_DIRECTORY}/lib-midx.sh"
|
||||
. "${TEST_DIRECTORY}/lib-terminal.sh"
|
||||
|
||||
GIT_TEST_MULTI_PACK_INDEX=0
|
||||
GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0
|
||||
|
||||
commit_and_pack () {
|
||||
test_commit "$@" 1>&2 &&
|
||||
incrpackid=$(git pack-objects --all --unpacked --incremental .git/objects/pack/pack </dev/null) &&
|
||||
|
|
@ -70,14 +73,13 @@ test_expect_success 'objects in packs marked .keep are not repacked' '
|
|||
|
||||
test_expect_success 'writing bitmaps via command-line can duplicate .keep objects' '
|
||||
# build on $oid, $packid, and .keep state from previous
|
||||
GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 git repack -Adbl &&
|
||||
git repack -Adbl &&
|
||||
test_has_duplicate_object true
|
||||
'
|
||||
|
||||
test_expect_success 'writing bitmaps via config can duplicate .keep objects' '
|
||||
# build on $oid, $packid, and .keep state from previous
|
||||
GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
|
||||
git -c repack.writebitmaps=true repack -Adl &&
|
||||
git -c repack.writebitmaps=true repack -Adl &&
|
||||
test_has_duplicate_object true
|
||||
'
|
||||
|
||||
|
|
@ -118,7 +120,7 @@ test_expect_success '--local disables writing bitmaps when connected to alternat
|
|||
(
|
||||
cd member &&
|
||||
test_commit "object" &&
|
||||
GIT_TEST_MULTI_PACK_INDEX=0 git repack -Adl --write-bitmap-index 2>err &&
|
||||
git repack -Adl --write-bitmap-index 2>err &&
|
||||
cat >expect <<-EOF &&
|
||||
warning: disabling bitmap writing, as some objects are not being packed
|
||||
EOF
|
||||
|
|
@ -284,8 +286,7 @@ test_expect_success 'repacking fails when missing .pack actually means missing o
|
|||
test_expect_success 'bitmaps are created by default in bare repos' '
|
||||
git clone --bare .git bare.git &&
|
||||
rm -f bare.git/objects/pack/*.bitmap &&
|
||||
GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
|
||||
git -C bare.git repack -ad &&
|
||||
git -C bare.git repack -ad &&
|
||||
bitmap=$(ls bare.git/objects/pack/*.bitmap) &&
|
||||
test_path_is_file "$bitmap"
|
||||
'
|
||||
|
|
@ -296,8 +297,7 @@ test_expect_success 'incremental repack does not complain' '
|
|||
'
|
||||
|
||||
test_expect_success 'bitmaps can be disabled on bare repos' '
|
||||
GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
|
||||
git -c repack.writeBitmaps=false -C bare.git repack -ad &&
|
||||
git -c repack.writeBitmaps=false -C bare.git repack -ad &&
|
||||
bitmap=$(ls bare.git/objects/pack/*.bitmap || :) &&
|
||||
test -z "$bitmap"
|
||||
'
|
||||
|
|
@ -308,8 +308,7 @@ test_expect_success 'no bitmaps created if .keep files present' '
|
|||
keep=${pack%.pack}.keep &&
|
||||
test_when_finished "rm -f \"\$keep\"" &&
|
||||
>"$keep" &&
|
||||
GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
|
||||
git -C bare.git repack -ad 2>stderr &&
|
||||
git -C bare.git repack -ad 2>stderr &&
|
||||
test_must_be_empty stderr &&
|
||||
find bare.git/objects/pack/ -type f -name "*.bitmap" >actual &&
|
||||
test_must_be_empty actual
|
||||
|
|
@ -320,8 +319,7 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' '
|
|||
blob=$(test-tool genrandom big $((1024*1024)) |
|
||||
git -C bare.git hash-object -w --stdin) &&
|
||||
git -C bare.git update-ref refs/tags/big $blob &&
|
||||
GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
|
||||
git -C bare.git repack -ad 2>stderr &&
|
||||
git -C bare.git repack -ad 2>stderr &&
|
||||
test_must_be_empty stderr &&
|
||||
find bare.git/objects/pack -type f -name "*.bitmap" >actual &&
|
||||
test_must_be_empty actual
|
||||
|
|
@ -342,9 +340,7 @@ test_expect_success 'repacking with a filter works' '
|
|||
'
|
||||
|
||||
test_expect_success '--filter fails with --write-bitmap-index' '
|
||||
test_must_fail \
|
||||
env GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \
|
||||
git -C bare.git repack -a -d --write-bitmap-index --filter=blob:none
|
||||
test_must_fail git -C bare.git repack -a -d --write-bitmap-index --filter=blob:none
|
||||
'
|
||||
|
||||
test_expect_success 'repacking with two filters works' '
|
||||
|
|
@ -540,11 +536,11 @@ test_expect_success 'setup for --write-midx tests' '
|
|||
test_expect_success '--write-midx unchanged' '
|
||||
(
|
||||
cd midx &&
|
||||
GIT_TEST_MULTI_PACK_INDEX=0 git repack &&
|
||||
git repack &&
|
||||
test_path_is_missing $midx &&
|
||||
test_path_is_missing $midx-*.bitmap &&
|
||||
|
||||
GIT_TEST_MULTI_PACK_INDEX=0 git repack --write-midx &&
|
||||
git repack --write-midx &&
|
||||
|
||||
test_path_is_file $midx &&
|
||||
test_path_is_missing $midx-*.bitmap &&
|
||||
|
|
@ -557,7 +553,7 @@ test_expect_success '--write-midx with a new pack' '
|
|||
cd midx &&
|
||||
test_commit loose &&
|
||||
|
||||
GIT_TEST_MULTI_PACK_INDEX=0 git repack --write-midx &&
|
||||
git repack --write-midx &&
|
||||
|
||||
test_path_is_file $midx &&
|
||||
test_path_is_missing $midx-*.bitmap &&
|
||||
|
|
@ -568,7 +564,7 @@ test_expect_success '--write-midx with a new pack' '
|
|||
test_expect_success '--write-midx with -b' '
|
||||
(
|
||||
cd midx &&
|
||||
GIT_TEST_MULTI_PACK_INDEX=0 git repack -mb &&
|
||||
git repack -mb &&
|
||||
|
||||
test_path_is_file $midx &&
|
||||
test_path_is_file $midx-*.bitmap &&
|
||||
|
|
@ -581,7 +577,7 @@ test_expect_success '--write-midx with -d' '
|
|||
cd midx &&
|
||||
test_commit repack &&
|
||||
|
||||
GIT_TEST_MULTI_PACK_INDEX=0 git repack -Ad --write-midx &&
|
||||
git repack -Ad --write-midx &&
|
||||
|
||||
test_path_is_file $midx &&
|
||||
test_path_is_missing $midx-*.bitmap &&
|
||||
|
|
@ -594,21 +590,21 @@ test_expect_success 'cleans up MIDX when appropriate' '
|
|||
cd midx &&
|
||||
|
||||
test_commit repack-2 &&
|
||||
GIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb --write-midx &&
|
||||
git repack -Adb --write-midx &&
|
||||
|
||||
checksum=$(midx_checksum $objdir) &&
|
||||
test_path_is_file $midx &&
|
||||
test_path_is_file $midx-$checksum.bitmap &&
|
||||
|
||||
test_commit repack-3 &&
|
||||
GIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb --write-midx &&
|
||||
git repack -Adb --write-midx &&
|
||||
|
||||
test_path_is_file $midx &&
|
||||
test_path_is_missing $midx-$checksum.bitmap &&
|
||||
test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
|
||||
|
||||
test_commit repack-4 &&
|
||||
GIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb &&
|
||||
git repack -Adb &&
|
||||
|
||||
find $objdir/pack -type f -name "multi-pack-index*" >files &&
|
||||
test_must_be_empty files
|
||||
|
|
@ -629,7 +625,6 @@ test_expect_success '--write-midx with preferred bitmap tips' '
|
|||
git log --format="create refs/tags/%s/%s %H" HEAD >refs &&
|
||||
git update-ref --stdin <refs &&
|
||||
|
||||
GIT_TEST_MULTI_PACK_INDEX=0 \
|
||||
git repack --write-midx --write-bitmap-index &&
|
||||
test_path_is_file $midx &&
|
||||
test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
|
||||
|
|
@ -721,13 +716,13 @@ test_expect_success '--write-midx removes stale pack-based bitmaps' '
|
|||
(
|
||||
cd repo &&
|
||||
test_commit base &&
|
||||
GIT_TEST_MULTI_PACK_INDEX=0 git repack -Ab &&
|
||||
git repack -Ab &&
|
||||
|
||||
pack_bitmap=$(ls $objdir/pack/pack-*.bitmap) &&
|
||||
test_path_is_file "$pack_bitmap" &&
|
||||
|
||||
test_commit tip &&
|
||||
GIT_TEST_MULTI_PACK_INDEX=0 git repack -bm &&
|
||||
git repack -bm &&
|
||||
|
||||
test_path_is_file $midx &&
|
||||
test_path_is_file $midx-$(midx_checksum $objdir).bitmap &&
|
||||
|
|
@ -750,7 +745,6 @@ test_expect_success '--write-midx with --pack-kept-objects' '
|
|||
keep="$objdir/pack/pack-$one.keep" &&
|
||||
touch "$keep" &&
|
||||
|
||||
GIT_TEST_MULTI_PACK_INDEX=0 \
|
||||
git repack --write-midx --write-bitmap-index --geometric=2 -d \
|
||||
--pack-kept-objects &&
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue