When c6458e60 (index-pack: kill union delta_base to save memory,
2015-04-18) attempted to reduce the memory footprint of index-pack,
one of the key thing it did was to keep track of ref-deltas and
ofs-deltas separately.
In fix_unresolved_deltas(), however it forgot that it now wants to
look only at ref deltas in one place. The code allocated an array
for nr_unresolved, which is sum of number of ref- and ofs-deltas
minus nr_resolved, which may be larger or smaller than the number
ref-deltas. Depending on nr_resolved, this was either under or over
allocating.
Also, the old code before this change had to use 'i' and 'n' because
some of the things we see in the (old) deltas[] array we scanned
with 'i' would not make it into the sorted_by_pos[] array in the old
world order, but now because you have only ref delta in a separate
ref_deltas[] array, they increment lock&step. We no longer need
separate variables. And most importantly, we shouldn't pass the
nr_unresolved parameter, as this number does not play a role in the
working of this helper function.
Helped-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Once we know the number of objects in the input pack, we allocate an
array of nr_objects of struct delta_entry. On x86-64, this struct is
32 bytes long. The union delta_base, which is part of struct
delta_entry, provides enough space to store either ofs-delta (8 bytes)
or ref-delta (20 bytes).
Because ofs-delta encoding is more efficient space-wise and more
performant at runtime than ref-delta encoding, Git packers try to use
ofs-delta whenever possible, and it is expected that objects encoded
as ref-delta are minority.
In the best clone case where no ref-delta object is present, we waste
(20-8) * nr_objects bytes because of this union. That's about 38MB out
of 100MB for deltas[] with 3.4M objects, or 38%. deltas[] would be
around 62MB without the waste.
This patch attempts to eliminate that. deltas[] array is split into
two: one for ofs-delta and one for ref-delta. Many functions are also
duplicated because of this split. With this patch, ofs_deltas[] array
takes 51MB. ref_deltas[] should remain unallocated in clone case (0
bytes). This array grows as we see ref-delta. We save about half in
this case, or 25% of total bookkeeping.
The saving is more than the calculation above because some padding in
the old delta_entry struct is removed. ofs_delta_entry is 16 bytes,
including the 4 bytes padding. That's 13MB for padding, but packing
the struct could break platforms that do not support unaligned
access. If someone on 32-bit is really low on memory and only deals
with packs smaller than 2G, using 32-bit off_t would eliminate the
padding and save 27MB on top.
A note about ofs_deltas allocation. We could use ref_deltas memory
allocation strategy for ofs_deltas. But that probably just adds more
overhead on top. ofs-deltas are generally the majority (1/2 to 2/3) in
any pack. Incremental realloc may lead to too many memcpy. And if we
preallocate, say 1/2 or 2/3 of nr_objects initially, the growth rate
of ALLOC_GROW() could make this array larger than nr_objects, wasting
more memory.
Brought-up-by: Matthew Sporleder <msporleder@gmail.com>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
For each object in the input pack, we need one struct object_entry. On
x86-64, this struct is 64 bytes long. Although:
- The 8 bytes for delta_depth and base_object_no are only useful when
show_stat is set. And it's never set unless someone is debugging.
- The three fields hdr_size, type and real_type take 4 bytes each
even though they never use more than 4 bits.
By moving delta_depth and base_object_no out of struct object_entry
and make the other 3 fields one byte long instead of 4, we shrink 25%
of this struct.
On a 3.4M object repo (*) that's about 53MB. The saving is less
impressive compared to index-pack memory use for basic bookkeeping (**),
about 16%.
(*) linux-2.6.git already has 4M objects as of v3.19-rc7 so this is
not an unrealistic number of objects that we have to deal with.
(**) 3.4M * (sizeof(object_entry) + sizeof(delta_entry)) = 311MB
Brought-up-by: Matthew Sporleder <msporleder@gmail.com>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
This message is about leaving orphaned commits behind, not about
behind an upstream branch. Try to make this clear.
Signed-off-by: Michael J Gruber <git@drmicha.warpmail.net>
Signed-off-by: Ralf Thielow <ralf.thielow@gmail.com>
Version numbers in asciidoc-generated content (such as man pages)
went missing as of da8a366 (Documentation: refactor common operations
into variables). Fix by putting the underscore back in the variable
name.
Signed-off-by: Sven van Haastregt <svenvh@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Commit 908a320363 introduced indentation
to here documents in t3301.sh. However in one place <<-EOF was missing
-, which broke this test when run with mksh-50d. This commit fixes it.
Signed-off-by: Kacper Kornet <draenog@pld-linux.org>
Acked-by: Johan Herland <johan@herland.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
An indentation error was found right after we started l10n round 2, and
commit d6589d1 (show-branch: fix indentation of usage string) and this
update would fix it.
Signed-off-by: Jiang Xin <worldhello.net@gmail.com>
When commit 695d95d refactored the color parsing, it missed
a "return 0" when parsing literal numbers 0-8 (which
represent basic ANSI colors), leading us to report these
colors as an error.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Updated translations for Git 2.3.0 l10n round 2, and fixed various
translations for command arguments.
Signed-off-by: Jiang Xin <worldhello.net@gmail.com>
It's a simple matter of opening the directory specified in the gitfile.
[ew: tweaked check to avoid open() on directories]
Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
ref_id should not match "refs/remotes/".
[ew: dropped initial hunk for GIT_SVN_ID at Ramkumar's request]
Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
For some unknown reason, the dd on my Windows box segfaults randomly,
but since recently, it does so much more often than it used to, which
makes running the test suite burdensome.
Use printf to write large files instead of dd. To emphasize that three
of the large blobs are exact copies, use cp to allocate them.
The new code makes the files a bit smaller, and they are not sparse
anymore, but the tests do not depend on these properties. We do not want
to use test-genrandom here (which is used to generate large files
elsewhere in t1050), so that the files can be compressed well (which
keeps the run-time short).
The files are now large text files, not binary files. But since they
are larger than core.bigfilethreshold they are diagnosed as binary
by Git. For this reason, the 'git diff' tests that check the output
for "Binary files differ" still pass.
Signed-off-by: Johannes Sixt <j6t@kdbg.org>
Reviewed-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Build update for older RHEL.
* rh/autoconf-rhel3:
configure.ac: check for HMAC_CTX_cleanup
configure.ac: check for clock_gettime and CLOCK_MONOTONIC
configure.ac: check 'tv_nsec' field in 'struct stat'
