git/builtin
Jonathan Tan 8b4c0103a9 sha1_file: support lazily fetching missing objects
Teach sha1_file to fetch objects from the remote configured in
extensions.partialclone whenever an object is requested but missing.

The fetching of objects can be suppressed through a global variable.
This is used by fsck and index-pack.

However, by default, such fetching is not suppressed. This is meant as a
temporary measure to ensure that all Git commands work in such a
situation. Future patches will update some commands to either tolerate
missing objects (without fetching them) or be more efficient in fetching
them.

In order to determine the code changes in sha1_file.c necessary, I
investigated the following:
 (1) functions in sha1_file.c that take in a hash, without the user
     regarding how the object is stored (loose or packed)
 (2) functions in packfile.c (because I need to check callers that know
     about the loose/packed distinction and operate on both differently,
     and ensure that they can handle the concept of objects that are
     neither loose nor packed)

(1) is handled by the modification to sha1_object_info_extended().

For (2), I looked at for_each_packed_object and others.  For
for_each_packed_object, the callers either already work or are fixed in
this patch:
 - reachable - only to find recent objects
 - builtin/fsck - already knows about missing objects
 - builtin/cat-file - warning message added in this commit

Callers of the other functions do not need to be changed:
 - parse_pack_index
   - http - indirectly from http_get_info_packs
   - find_pack_entry_one
     - this searches a single pack that is provided as an argument; the
       caller already knows (through other means) that the sought object
       is in a specific pack
 - find_sha1_pack
   - fast-import - appears to be an optimization to not store a file if
     it is already in a pack
   - http-walker - to search through a struct alt_base
   - http-push - to search through remote packs
 - has_sha1_pack
   - builtin/fsck - already knows about promisor objects
   - builtin/count-objects - informational purposes only (check if loose
     object is also packed)
   - builtin/prune-packed - check if object to be pruned is packed (if
     not, don't prune it)
   - revision - used to exclude packed objects if requested by user
   - diff - just for optimization

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-12-08 09:52:42 -08:00
..
add.c Merge branch 'rj/add-chmod-error-message' into maint 2017-09-10 17:03:00 +09:00
am.c Merge branch 'pw/am-signoff' into maint 2017-09-10 17:02:51 +09:00
annotate.c
apply.c Convert read_mmblob to take struct object_id. 2016-09-07 12:59:42 -07:00
archive.c archive: read local configuration 2016-11-22 13:55:20 -08:00
bisect--helper.c
blame.c Merge branch 'po/read-graft-line' 2017-09-06 13:11:25 +09:00
branch.c Merge branch 'jk/ref-filter-colors-fix' 2017-10-18 10:19:08 +09:00
bundle.c bundle: use prefix_filename with bundle path 2017-03-21 11:18:41 -07:00
cat-file.c sha1_file: support lazily fetching missing objects 2017-12-08 09:52:42 -08:00
check-attr.c config: don't include config.h by default 2017-06-15 12:56:22 -07:00
check-ignore.c Merge branch 'bw/config-h' 2017-06-24 14:28:41 -07:00
check-mailmap.c config: don't include config.h by default 2017-06-15 12:56:22 -07:00
check-ref-format.c check-ref-format: release strbuf after use in check_ref_format_branch() 2017-09-07 08:49:26 +09:00
checkout-index.c config: don't include config.h by default 2017-06-15 12:56:22 -07:00
checkout.c Merge branch 'ma/builtin-unleak' 2017-10-07 16:27:55 +09:00
clean.c Merge branch 'jk/ref-filter-colors-fix' 2017-10-18 10:19:08 +09:00
clone.c Merge branch 'bw/clone-recursive-quiet' into maint 2017-09-10 17:02:49 +09:00
column.c config: don't include config.h by default 2017-06-15 12:56:22 -07:00
commit-tree.c Merge branch 'rk/commit-tree-make-F-verbatim' 2017-09-19 10:47:57 +09:00
commit.c Merge branch 'jk/no-optional-locks' 2017-10-03 15:42:49 +09:00
config.c add UNLEAK annotation for reducing leak false positives 2017-09-08 15:43:17 +09:00
count-objects.c pack: move open_pack_index(), parse_pack_index() 2017-08-23 15:12:06 -07:00
credential.c credential: handle invalid arguments earlier 2017-05-30 14:45:03 +09:00
describe.c Merge branch 'jk/describe-omit-some-refs' 2017-09-28 14:47:52 +09:00
diff-files.c submodule: remove gitmodules_config 2017-08-03 13:11:02 -07:00
diff-index.c Merge branch 'ma/builtin-unleak' 2017-10-07 16:27:55 +09:00
diff-tree.c submodule: remove gitmodules_config 2017-08-03 13:11:02 -07:00
diff.c Merge branch 'ma/builtin-unleak' 2017-10-07 16:27:55 +09:00
difftool.c sha1_file: convert index_fd to struct object_id 2017-08-20 21:52:08 -07:00
fast-export.c Merge branch 'ma/leakplugs' 2017-09-29 11:23:43 +09:00
fetch-pack.c sha1_file: support lazily fetching missing objects 2017-12-08 09:52:42 -08:00
fetch.c Merge branch 'jt/packmigrate' 2017-08-26 22:55:09 -07:00
fmt-merge-msg.c Merge branch 'dc/fmt-merge-msg-microcleanup' into maint 2017-08-23 14:33:52 -07:00
for-each-ref.c provide --color option for all ref-filter users 2017-10-04 11:35:29 +09:00
fsck.c sha1_file: support lazily fetching missing objects 2017-12-08 09:52:42 -08:00
gc.c Merge branch 'aw/gc-lockfile-fscanf-fix' 2017-09-25 15:24:09 +09:00
get-tar-commit-id.c distinguish error versus short read from read_in_full() 2017-09-27 15:45:24 +09:00
grep.c Merge branch 'jk/ref-filter-colors-fix' 2017-10-18 10:19:08 +09:00
hash-object.c sha1_file: convert hash_sha1_file_literally to struct object_id 2017-08-20 21:52:53 -07:00
help.c Merge branch 'ks/help-alias-label' 2017-09-25 15:24:07 +09:00
index-pack.c sha1_file: support lazily fetching missing objects 2017-12-08 09:52:42 -08:00
init-db.c add UNLEAK annotation for reducing leak false positives 2017-09-08 15:43:17 +09:00
interpret-trailers.c Merge branch 'jk/trailers-parse' 2017-08-26 22:55:04 -07:00
log.c refs: pass NULL to resolve_ref_unsafe() if hash is not needed 2017-09-24 10:18:21 +09:00
ls-files.c add UNLEAK annotation for reducing leak false positives 2017-09-08 15:43:17 +09:00
ls-remote.c wildmatch: remove unused wildopts parameter 2017-06-23 18:27:07 -07:00
ls-tree.c Merge branch 'bw/config-h' 2017-06-24 14:28:41 -07:00
mailinfo.c prefix_filename: return newly allocated string 2017-03-21 11:18:41 -07:00
mailsplit.c mailinfo & mailsplit: check for EOF while parsing 2017-05-08 12:18:19 +09:00
merge-base.c Merge branch 'bw/config-h' 2017-06-24 14:28:41 -07:00
merge-file.c config: don't include config.h by default 2017-06-15 12:56:22 -07:00
merge-index.c Convert GIT_SHA1_HEXSZ used for allocation to GIT_MAX_HEXSZ 2017-03-26 22:08:21 -07:00
merge-ours.c
merge-recursive.c i18n: merge-recursive: mark verbose message for translation 2016-09-15 13:17:32 -07:00
merge-tree.c tree-walk: convert fill_tree_descriptor() to object_id 2017-08-14 12:38:54 -07:00
merge.c Merge branch 'rs/strbuf-leakfix' 2017-09-19 10:47:57 +09:00
mktag.c
mktree.c mktree: plug memory leaks reported by Coverity 2017-05-08 12:18:19 +09:00
mv.c submodule: remove gitmodules_config 2017-08-03 13:11:02 -07:00
name-rev.c Merge branch 'ma/builtin-unleak' 2017-10-07 16:27:55 +09:00
notes.c notes: release strbuf after use in notes_copy_from_stdin() 2017-09-07 08:49:28 +09:00
pack-objects.c pack-objects: add list-objects filtering 2017-11-22 14:11:57 +09:00
pack-redundant.c pack: move pack name-related functions 2017-08-23 15:12:06 -07:00
pack-refs.c refs: delete pack_refs() in favor of refs_pack_refs() 2017-04-14 03:53:25 -07:00
patch-id.c config: don't include config.h by default 2017-06-15 12:56:22 -07:00
prune-packed.c Merge branch 'jt/packmigrate' 2017-08-26 22:55:09 -07:00
prune.c progress: simplify "delayed" progress API 2017-08-19 14:01:34 -07:00
pull.c pull: honor submodule.recurse config option 2017-09-07 09:52:18 +09:00
push.c Merge branch 'ma/parse-maybe-bool' 2017-08-22 10:29:03 -07:00
read-tree.c submodule: remove gitmodules_config 2017-08-03 13:11:02 -07:00
rebase--helper.c rebase -i: rearrange fixup/squash lines using the rebase--helper 2017-07-27 15:35:06 -07:00
receive-pack.c Merge branch 'rs/resolve-ref-optional-result' 2017-10-05 13:48:19 +09:00
reflog.c object_array: add and use object_array_pop() 2017-09-24 10:06:04 +09:00
remote-ext.c consistently use "fallthrough" comments in switches 2017-09-22 12:49:57 +09:00
remote-fd.c remote-{ext,fd}: print usage message on invalid arguments 2017-05-30 14:45:04 +09:00
remote.c refs: pass NULL to resolve_ref_unsafe() if hash is not needed 2017-09-24 10:18:21 +09:00
repack.c Merge branch 'bw/config-h' 2017-06-24 14:28:41 -07:00
replace.c sha1_file: convert index_fd to struct object_id 2017-08-20 21:52:08 -07:00
rerere.c avoid "write_in_full(fd, buf, len) != len" pattern 2017-09-14 15:17:59 +09:00
reset.c Merge branch 'jk/leak-checkers' 2017-09-19 10:47:55 +09:00
rev-list.c rev-list: support --no-filter argument 2017-12-05 09:44:37 -08:00
rev-parse.c Merge branch 'bc/rev-parse-parseopt-fix' 2017-10-03 15:42:47 +09:00
revert.c cherry-pick/revert: reject --rerere-autoupdate when continuing 2017-08-02 15:16:09 -07:00
rm.c submodule: remove gitmodules_config 2017-08-03 13:11:02 -07:00
send-pack.c Merge branch 'ma/parse-maybe-bool' 2017-08-22 10:29:03 -07:00
shortlog.c shortlog: skip format/parse roundtrip for internal traversal 2017-09-09 01:57:03 +09:00
show-branch.c Merge branch 'jk/ref-filter-colors-fix' 2017-10-18 10:19:08 +09:00
show-ref.c show-ref: remove a stale comment 2017-01-23 18:51:56 -08:00
stripspace.c config: don't include config.h by default 2017-06-15 12:56:22 -07:00
submodule--helper.c Merge branch 'rs/resolve-ref-optional-result' 2017-09-28 14:47:56 +09:00
symbolic-ref.c refs: pass NULL to resolve_ref_unsafe() if hash is not needed 2017-09-24 10:18:21 +09:00
tag.c Merge branch 'jk/ref-filter-colors-fix' 2017-10-18 10:19:08 +09:00
unpack-file.c Merge branch 'jk/write-in-full-fix' 2017-09-25 15:24:06 +09:00
unpack-objects.c cleanup: fix possible overflow errors in binary search 2017-10-10 08:57:24 +09:00
update-index.c Merge branch 'jk/leak-checkers' 2017-09-19 10:47:55 +09:00
update-ref.c builtin/update_ref: convert to struct object_id 2017-07-17 13:54:38 -07:00
update-server-info.c config: don't include config.h by default 2017-06-15 12:56:22 -07:00
upload-archive.c upload-archive: handle "-h" option early 2017-05-30 14:45:04 +09:00
var.c config: don't include config.h by default 2017-06-15 12:56:22 -07:00
verify-commit.c Merge branch 'bw/config-h' 2017-06-24 14:28:41 -07:00
verify-pack.c config: don't include config.h by default 2017-06-15 12:56:22 -07:00
verify-tag.c Merge branch 'jk/ref-filter-colors' 2017-08-11 13:26:58 -07:00
worktree.c worktree: check the result of read_in_full() 2017-09-27 15:46:05 +09:00
write-tree.c config: don't include config.h by default 2017-06-15 12:56:22 -07:00