When generating a new pack, notice if we have already needed
objects in existing packs. If an object is stored deltified,
and its base object is also what we are going to pack, then
reuse the existing deltified representation unconditionally,
bypassing all the expensive find_deltas() and try_deltas()
calls.
Also, notice if what we are going to write out exactly match
what is already in an existing pack (either deltified or just
compressed). In such a case, we can just copy it instead of
going through the usual uncompressing & recompressing cycle.
Without this patch, in linux-2.6 repository with about 1500
loose objects and a single mega pack:
$ git-rev-list --objects v2.6.16-rc3 >RL
$ wc -l RL
184141 RL
$ time git-pack-objects p <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects....................
a1fc7b3e537fcb9b3c46b7505df859f0a11e79d2
real 12m4.323s
user 11m2.560s
sys 0m55.950s
With this patch, the same input:
$ time ../git.junio/git-pack-objects q <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects.....................
a1fc7b3e537fcb9b3c46b7505df859f0a11e79d2
Total 184141, written 184141, reused 182441
real 1m2.608s
user 0m55.090s
sys 0m1.830s
Signed-off-by: Junio C Hamano <junkio@cox.net>
The real problem triggered an earlier fix was that an alternate
entry was pointing at a removed directory. Complaining on
object/pack directory that cannot be opendir-ed produces noise
in an ancient repository that does not have object/pack
directory and has never been packed.
Detect the real user error and report it. Also if opendir
failed for other reasons (e.g. no read permissions), report that
as well.
Spotted by Andrew Vasquez <andrew.vasquez@qlogic.com>.
Signed-off-by: Junio C Hamano <junkio@cox.net>
* jc/pack-reuse:
pack-objects: avoid delta chains that are too long.
git-repack: allow passing a couple of flags to pack-objects.
pack-objects: finishing touches.
pack-objects: reuse data from existing packs.
When generating a new pack, notice if we have already needed
objects in existing packs. If an object is stored deltified,
and its base object is also what we are going to pack, then
reuse the existing deltified representation unconditionally,
bypassing all the expensive find_deltas() and try_deltas()
calls.
Also, notice if what we are going to write out exactly match
what is already in an existing pack (either deltified or just
compressed). In such a case, we can just copy it instead of
going through the usual uncompressing & recompressing cycle.
Without this patch, in linux-2.6 repository with about 1500
loose objects and a single mega pack:
$ git-rev-list --objects v2.6.16-rc3 >RL
$ wc -l RL
184141 RL
$ time git-pack-objects p <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects....................
a1fc7b3e537fcb9b3c46b7505df859f0a11e79d2
real 12m4.323s
user 11m2.560s
sys 0m55.950s
With this patch, the same input:
$ time ../git.junio/git-pack-objects q <RL
Generating pack...
Done counting 184141 objects.
Packing 184141 objects.....................
a1fc7b3e537fcb9b3c46b7505df859f0a11e79d2
Total 184141, written 184141, reused 182441
real 1m2.608s
user 0m55.090s
sys 0m1.830s
Signed-off-by: Junio C Hamano <junkio@cox.net>
Use stat() to explicitly check for existence rather than
relying on the non-portable EEXIST error in sha1_file.c's
safe_create_leading_directories(). There certainly are
optimizations possible, but then the code becomes almost
the same as that in coreutil's lib/mkdir-p.c.
Other uses of EEXIST seem ok. Tested on Solaris 8, AIX 5.2L,
and a few Linux versions. AIX has some unrelated (I think)
failures right now; I haven't tried many recent gits there.
Anyone have an old Ultrix box to break everything? ;)
Also remove extraneous #includes. Everything's already in
git-compat-util.h, included through cache.h.
Signed-off-by: Jason Riedy <ejr@cs.berkeley.edu>
Signed-off-by: Junio C Hamano <junkio@cox.net>
This reverts c5ced64578 commit.
It turns out that doing this check every time we map the idx file
is quite expensive. A corrupt idx file is caught by git-fsck-objects,
so this check is not strictly necessary.
In one unscientific test, 0.99.9m spent 10 seconds usertime for
the same task 1.1.3 takes 37 seconds usertime. Reverting this gives
us the performance of 0.99.9 back.
If the config variable 'core.sharedrepository' is set, the directories
$GIT_DIR/objects/
$GIT_DIR/objects/??
$GIT_DIR/objects/pack
$GIT_DIR/refs
$GIT_DIR/refs/heads
$GIT_DIR/refs/heads/tags
are set group writable (and g+s, since the git group may be not the primary
group of all users).
Since all files are written as lock files first, and then moved to
their destination, they do not have to be group writable. Indeed, if
this leads to problems you found a bug.
Note that -- as in my first attempt -- the config variable is set in the
function which checks the repository format. If this were done in
git_default_config instead, a lot of programs would need to be modified
to call git_config(git_default_config) first.
[jc: git variables should be in environment.c unless there is a
compelling reason to do otherwise.]
Signed-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Although pack-check.c had routine to verify the checksum for the
pack index file itself, the core did not check it before using
it.
This is stolen from the patch to tighten packname requirements.
Signed-off-by: Junio C Hamano <junkio@cox.net>
(cherry picked from 797bd6f490 commit)
Although pack-check.c had routine to verify the checksum for the
pack index file itself, the core did not check it before using
it.
This is stolen from the patch to tighten packname requirements.
Signed-off-by: Junio C Hamano <junkio@cox.net>
sha1_to_hex() returns a pointer to a static buffer. Some of its users
modify that buffer by appending a newline character. Other users rely
on the fact that you can call
printf("%s", sha1_to_hex(sha1));
Just to be on the safe side, terminate the SHA1 in sha1_to_hex().
Signed-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: Junio C Hamano <junkio@cox.net>
fprintf and die sometimes have missing/excessive "\n" in their arguments,
correct the strings where I think it would be appropriate.
Signed-off-by: Alex Riesen <raa.lkml@gmail.com>
Signed-off-by: Junio C Hamano <junkio@cox.net>
add_packed_git() tries to get the pack SHA1 by parsing its name. It may
access uninitialized memory for packs with short names.
Signed-off-by: Pavel Roskin <proski@gnu.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
In order to support getting data into git with scripts, this adds a
--stdin option to git-hash-object, which will make it read from stdin.
Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
We somehow ended up registering packs in alternate object
directories as "dir/object//pack/pack-*", which confusd the
update-server-info code very badly. Also we did not attempt to
detect a mistake of listing the object directory itself as one
of the alternates. This does not lead to incorrect behaviour,
but is simply wasteful, so try to do so when we are trivially
able to.
Signed-off-by: Junio C Hamano <junkio@cox.net>
This patch adds the program git-pack-intersect. It is
used to find redundant packs in git repositories.
Signed-off-by: Lukas Sandström <lukass@etek.chalmers.se>
Signed-off-by: Junio C Hamano <junkio@cox.net>
This fixes a problem in safe_create_leading_directories() when the
argument starts with a '/' (i.e. the path is absolute).
Signed-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Recent FAT workaround caused compilation trouble on OpenBSD;
different platforms use different error codes when we try to
hardlink the temporary file to its final location. Existing
Coda hack also checks its own error code, but the thing is,
the case we care about is if link failed for a reason other
than that the final file has already existed (which would be
normal, or it could mean collision). So just check the error
code against EEXIST.
Signed-off-by: Junio C Hamano <junkio@cox.net>
FAT -- like Coda -- does not like cross-directory hard links. To be
precise, FAT does not like links at all. But links are not needed either.
So get rid of them.
Signed-off-by: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Signed-off-by: Junio C Hamano <junkio@cox.net>
If we want to re-pack just local packfiles, we need to know whether a
particular object is local or not.
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
This starts using the "user.name" and "user.email" config variables if
they exist as the default name and email when committing. This means
that you don't have to use the GIT_COMMITTER_EMAIL environment variable
to override your email - you can just edit the config file instead.
The patch looks bigger than it is because it makes the default name and
email information non-static and renames it appropriately. And it moves
the common git environment variables into a new library file, so that
you can link against libgit.a and get the git environment without having
to link in zlib and libcrypt.
In short, most of it is renaming and moving, the real change core is
just a few new lines in "git_default_config()" that copies the user
config values to the new base.
It also changes "git-var -l" to list the config variables.
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
The http commit walker cannot use the same temporary file
creation code because it needs to use predictable temporary
filename for partial fetch continuation purposes, but the code
to move the temporary file to the final location should be
usable from the ordinary object creation codepath.
Export move_temp_to_file from sha1_file.c and use it, while
losing the custom relink_or_rename function from http-fetch.c.
Also the temporary object file creation part needs to make sure
the leading path exists, in preparation of the really lazy
fan-out directory creation.
Signed-off-by: Junio C Hamano <junkio@cox.net>
This makes it possible to have a "sparse" git object subdirectory
structure, something that has become much more attractive now that people
use pack-files all the time.
As a result of pack-files, a git object directory doesn't necessarily have
any individual objects lying around, and in that case it's just wasting
space to keep the empty first-level object directories around: on many
filesystems the 256 empty directories will be aboue 1MB of diskspace.
Even more importantly, after you re-pack a project that _used_ to be
unpacked, you could be left with huge directories that no longer contain
anything, but that waste space and take time to look through.
With this change, "git prune-packed" can just do an rmdir() on the
directories, and they'll get removed if empty, and re-created on demand.
This patch also tries to fix up "write_sha1_from_fd()" to use the new
common infrastructure for creating the object files, closing a hole where
we might otherwise leave half-written objects in the object database.
[jc: I unoptimized the part that really removes the fan-out directories
to ease transition. init-db still wastes 1MB of diskspace to hold 256
empty fan-outs, and prune-packed rmdir()'s the grown but empty directories,
but runs mkdir() immediately after that -- reducing the saving from 150KB
to 146KB. These parts will be re-introduced when everybody has the
on-demand capability.]
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
This adds more cruft to diff --git header to record the blob SHA1 and
the mode the patch/diff is intended to be applied against, to help the
receiving end fall back on a three-way merge. The new header looks
like this:
diff --git a/apply.c b/apply.c
index 7be5041..8366082 100644
--- a/apply.c
+++ b/apply.c
@@ -14,6 +14,7 @@
// files that are being modified, but doesn't apply the patch
// --stat does just a diffstat, and doesn't actually apply
+// --show-index-info shows the old and new index info for...
...
Upon receiving such a patch, if the patch did not apply cleanly to the
target tree, the recipient can try to find the matching old objects in
her object database and create a temporary tree, apply the patch to
that temporary tree, and attempt a 3-way merge between the patched
temporary tree and the target tree using the original temporary tree
as the common ancestor.
The patch lifts the code to compute the hash for an on-filesystem
object from update-index.c and makes it available to the diff output
routine.
Signed-off-by: Junio C Hamano <junkio@cox.net>
The core part detected and died upon seeing a corrupted packfile, but
did not help the user by telling which packfile is corrupt and how.
Signed-off-by: Junio C Hamano <junkio@cox.net>
An entry in the alternates file can name a directory relative to
the object store it describes. A typical linux-2.6 maintainer
repository would have "../../../torvalds/linux-2.6.git/objects" there,
because the subsystem maintainer object store would live in
/pub/scm/linux/kernel/git/$u/$system.git/objects/
and the object store of Linus tree is in
/pub/scm/linux/kernel/git/torvalds/linux-2.6.git/objects/
This unfortunately is different from GIT_ALTERNATE_OBJECT_DIRECTORIES
which is relative to the cwd of the running process, but there is no
way to make it consistent with the behaviour of the environment
variable. The process typically is run in $system.git/ directory for
a naked repository, or one level up for a repository with a working
tree, so we just define it to be relative to the objects/ directory
to be different from either ;-).
Later, the dumb transport could be updated to read from info/alternates
and make requests for the repository the repository borrows from.
Signed-off-by: Junio C Hamano <junkio@cox.net>
We have deprecated the old environment variable names for quite a
while and now it's time to remove them. Gone are:
SHA1_FILE_DIRECTORIES AUTHOR_DATE AUTHOR_EMAIL AUTHOR_NAME
COMMIT_AUTHOR_EMAIL COMMIT_AUTHOR_NAME SHA1_FILE_DIRECTORY
Signed-off-by: Junio C Hamano <junkio@cox.net>
Hi. This patch contains the following possible cleanups:
* Make some needlessly global functions in local-pull.c static
* Change 'char *' to 'const char *' where appropriate
Signed-off-by: Peter Hagervall <hager@cs.umu.se>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Omitting the first branch in ?: is a GNU extension. Cute,
but not supported by other compilers. Replaced mostly
by explicit tests. Calls to getenv() simply are repeated
on non-GNU compilers.
Signed-off-by: Jason Riedy <ejr@cs.berkeley.edu>
Yes, using the same format for the file and the environment variable
was a big mistake. This uses LF as the path separator, and allows
lines that begin with '#' to be comments. ':' is no longer a separator
in objects/info/alternates file.
Signed-off-by: Junio C Hamano <junkio@cox.net>
Note that the pack file has to be in the usual location if it gets
installed later.
Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
It was a mistake to use GIT_ALTERNATE_OBJECT_DIRECTORIES
environment variable to specify what alternate object pools to
look for missing objects when working with an object database.
It is not a property of the process running the git commands,
but a property of the object database that is partial and needs
other object pools to complete the set of objects it lacks.
This patch allows you to have $GIT_OBJECT_DIRECTORY/info/alternates
whose contents is in exactly the same format as the environment
variable, to let an object database name alternate object pools
it depends on.
Signed-off-by: Junio C Hamano <junkio@cox.net>
This patch fixes the only warning reported by gcc 4.0.1 on Fedora Core 4
for x86_64:
sha1_file.c:1391: warning: pointer targets in assignment differ in
signedness
Signed-off-by: Pavel Roskin <proski@gnu.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
If the object to write was packed, both its uncompressed and compressed
data were leaked. If the object was not packed, its file was not unmapped.
[jc: I think it still leaks on the write error path of
write_sha1_to_fd(), but that should be fixable in a small separate
patch.]
Signed-off-by: Sergey Vlasov <vsu@altlinux.ru>
Signed-off-by: Junio C Hamano <junkio@cox.net>
When following a reference, read_object_with_reference() did not free the
intermediate object data.
Signed-off-by: Sergey Vlasov <vsu@altlinux.ru>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Everybody envies rev-parse, who is the only one that can grok
the extended sha1 format. Move the get_extended_sha1() out of
rev-parse, rename it to get_sha1() and make it available to
everybody else.
The one I posted earlier to the list had one bug where it did
not handle a name that ends with a digit correctly (it
incorrectly tried the "Nth parent" path). This commit fixes it.
Signed-off-by: Junio C Hamano <junkio@cox.net>
This causes ssh-pull to request objects in prefetch() and read then in
fetch(), such that it reduces the unpipelined round-trip time.
This also makes sha1_write_from_fd() support having a buffer of data
which it accidentally read from the fd after the object; this was
formerly not a problem, because it would always get a short read at
the end of an object, because the next object had not been
requested. This is no longer true.
Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
This adds support for reading an uninstalled index, and installing a
pack file that was added while the program was running, as well as
functions for determining where to put the file.
Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Introduce a new file $GIT_DIR/info/grafts (or $GIT_GRAFT_FILE)
which is a list of "fake commit parent records". Each line of
this file is a commit ID, followed by parent commit IDs, all
40-byte hex SHA1 separated by a single SP in between. The
records override the parent information we would normally read
from the commit objects, allowing both adding "fake" parents
(i.e. grafting), and pretending as if a commit is not a child of
some of its real parents (i.e. cauterizing).
Signed-off-by: Junio C Hamano <junkio@cox.net>
I have reviewed all occurrences of mmap() in git and fixed three types
of errors/defects:
1) The result is not checked.
2) The file descriptor is closed if mmap() succeeds, but not when it
fails.
3) Various casts applied to -1 are used instead of MAP_FAILED, which is
specifically defined to check mmap() return value.
[jc: This is a second round of Pavel's patch. He fixed up the problem
that close() potentially clobbering the errno from mmap, which
the first round had.]
Signed-off-by: Pavel Roskin <proski@gnu.org>
Signed-off-by: Junio C Hamano <junkio@cox.net>
This reverses the order of object lookup, to check pack index first and
then go to the filesystem to find .git/objects/??/ hierarchy.
When most of the objects are packed, this saves quite many stat() calls
and negative dcache entries; while the price this approach has to pay is
negligible, even when most of the objects are outside pack, because
checking pack index file is quite cheap.
Signed-off-by: Junio C Hamano <junkio@cox.net>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
