Commit graph

63125 commits

Author SHA1 Message Date
Ævar Arnfjörð Bjarmason dccb32bf01 object-file.c: stop dying in parse_loose_header()
Make parse_loose_header() return error codes and data instead of
invoking die() by itself.

For now we'll move the relevant die() call to loose_object_info() and
read_loose_object() to keep this change smaller. In a subsequent
commit we'll make read_loose_object() return an error code instead of
dying. We should also address the "allow_unknown" case (should be
moved to builtin/cat-file.c), but for now I'll be leaving it.

For making parse_loose_header() not die() change its prototype to
accept a "struct object_info *" instead of the "unsigned long *sizep"
it accepted before. Its callers can now check the populated populated
"oi->typep".

Because of this we don't need to pass in the "unsigned int flags"
which we used for OBJECT_INFO_ALLOW_UNKNOWN_TYPE, we can instead do
that check in loose_object_info().

This also refactors some confusing control flow around the "status"
variable. In some cases we set it to the return value of "error()",
i.e. -1, and later checked if "status < 0" was true.

Since 93cff9a978 (sha1_loose_object_info: return error for corrupted
objects, 2017-04-01) the return value of loose_object_info() (then
named sha1_loose_object_info()) had been a "status" variable that be
any negative value, as we were expecting to return the "enum
object_type".

The only negative type happens to be OBJ_BAD, but the code still
assumed that more might be added. This was then used later in
e.g. c84a1f3ed4 (sha1_file: refactor read_object, 2017-06-21). Now
that parse_loose_header() will return 0 on success instead of the
type (which it'll stick into the "struct object_info") we don't need
to conflate these two cases in its callers.

Since parse_loose_header() doesn't need to return an arbitrary
"status" we only need to treat its "ret < 0" specially, but can
idiomatically overwrite it with our own error() return. This along
with having made unpack_loose_header() return an "enum
unpack_loose_header_result" in an earlier commit means that we can
move the previously nested if/else cases mostly into the "ULHR_OK"
branch of the "switch" statement.

We should be less silent if we reach that "status = -1" branch, which
happens if we've got trailing garbage in loose objects, see
f6371f9210 (sha1_file: add read_loose_object() function, 2017-01-13)
for a better way to handle it. For now let's punt on it, a subsequent
commit will address that edge case.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-01 15:06:00 -07:00
Ævar Arnfjörð Bjarmason 5848fb11ac object-file.c: return ULHR_TOO_LONG on "header too long"
Split up the return code for "header too long" from the generic
negative return value unpack_loose_header() returns, and report via
error() if we exceed MAX_HEADER_LEN.

As a test added earlier in this series in t1006-cat-file.sh shows
we'll correctly emit zlib errors from zlib.c already in this case, so
we have no need to carry those return codes further down the
stack. Let's instead just return ULHR_TOO_LONG saying we ran into the
MAX_HEADER_LEN limit, or other negative values for "unable to unpack
<OID> header".

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-01 15:06:00 -07:00
Ævar Arnfjörð Bjarmason 3b6a8db3b0 object-file.c: use "enum" return type for unpack_loose_header()
In a preceding commit we changed and documented unpack_loose_header()
from its previous behavior of returning any negative value or zero, to
only -1 or 0.

Let's add an "enum unpack_loose_header_result" type and use it for
these return values, and have the compiler assert that we're
exhaustively covering all of them.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-01 15:06:00 -07:00
Ævar Arnfjörð Bjarmason 01cab97679 object-file.c: simplify unpack_loose_short_header()
Combine the unpack_loose_short_header(),
unpack_loose_header_to_strbuf() and unpack_loose_header() functions
into one.

The unpack_loose_header_to_strbuf() function was added in
46f034483e (sha1_file: support reading from a loose object of unknown
type, 2015-05-03).

Its code was mostly copy/pasted between it and both of
unpack_loose_header() and unpack_loose_short_header(). We now have a
single unpack_loose_header() function which accepts an optional
"struct strbuf *" instead.

I think the remaining unpack_loose_header() function could be further
simplified, we're carrying some complexity just to be able to emit a
garbage type longer than MAX_HEADER_LEN, we could alternatively just
say "we found a garbage type <first 32 bytes>..." instead. But let's
leave the current behavior in place for now.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-01 15:06:00 -07:00
Ævar Arnfjörð Bjarmason ddb3474b66 object-file.c: make parse_loose_header_extended() public
Make the parse_loose_header_extended() function public and remove the
parse_loose_header() wrapper. The only direct user of it outside of
object-file.c itself was in streaming.c, that caller can simply pass
the required "struct object-info *" instead.

This change is being done in preparation for teaching
read_loose_object() to accept a flag to pass to
parse_loose_header(). It isn't strictly necessary for that change, we
could simply use parse_loose_header_extended() there, but will leave
the API in a better end state.

It would be a better end-state to have already moved the declaration
of these functions to object-store.h to avoid the forward declaration
of "struct object_info" in cache.h, but let's leave that cleanup for
some other time.

1. https://lore.kernel.org/git/patch-v6-09.22-5b9278e7bb4-20210907T104559Z-avarab@gmail.com/

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-01 15:06:00 -07:00
Ævar Arnfjörð Bjarmason bfff2c4833 object-file.c: return -1, not "status" from unpack_loose_header()
Return a -1 when git_inflate() fails instead of whatever Z_* status
we'd get from zlib.c. This makes no difference to any error we report,
but makes it more obvious that we don't care about the specific zlib
error codes here.

See d21f842690 (unpack_sha1_header(): detect malformed object header,
2016-09-25) for the commit that added the "return status" code. As far
as I can tell there was never a real reason (e.g. different reporting)
for carrying down the "status" as opposed to "-1".

At the time that d21f842690 was written there was a corresponding
"ret < Z_OK" check right after the unpack_sha1_header() call (the
"unpack_sha1_header()" function was later rename to our current
"unpack_loose_header()").

However, that check was removed in c84a1f3ed4 (sha1_file: refactor
read_object, 2017-06-21) without changing the corresponding return
code.

So let's do the minor cleanup of also changing this function to return
a -1.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-01 15:06:00 -07:00
Ævar Arnfjörð Bjarmason 74ad250a1c object-file.c: don't set "typep" when returning non-zero
When the loose_object_info() function returns an error stop faking up
the "oi->typep" to OBJ_BAD. Let the return value of the function
itself suffice. This code cleanup simplifies subsequent changes.

That we set this at all is a relic from the past. Before
052fe5eaca (sha1_loose_object_info: make type lookup optional,
2013-07-12) we would always return the type_from_string(type) via the
parse_sha1_header() function, or -1 (i.e. OBJ_BAD) if we couldn't
parse it.

Then in a combination of 46f034483e (sha1_file: support reading from
a loose object of unknown type, 2015-05-03) and
b3ea7dd32d (sha1_loose_object_info: handle errors from
unpack_sha1_rest, 2017-10-05) our API drifted even further towards
conflating the two again.

Having read the code paths involved carefully I think this is OK. We
are just about to return -1, and we have only one caller:
do_oid_object_info_extended(). That function will in turn go on to
return -1 when we return -1 here.

This might be introducing a subtle bug where a caller of
oid_object_info_extended() would inspect its "typep" and expect a
meaningful value if the function returned -1.

Such a problem would not occur for its simpler oid_object_info()
sister function. That one always returns the "enum object_type", which
in the case of -1 would be the OBJ_BAD.

Having read the code for all the callers of these functions I don't
believe any such bug is being introduced here, and in any case we'd
likely already have such a bug for the "sizep" member (although
blindly checking "typep" first would be a more common case).

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-01 15:06:00 -07:00
Ævar Arnfjörð Bjarmason dd45a56246 cat-file tests: test for current --allow-unknown-type behavior
Add more tests for the current --allow-unknown-type behavior. As noted
in [1] I don't think much of this makes sense, but let's test for it
as-is so we can see if the behavior changes in the future.

1. https://lore.kernel.org/git/87r1i4qf4h.fsf@evledraar.gmail.com/

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-01 15:06:00 -07:00
Ævar Arnfjörð Bjarmason 7e7d220d9d cat-file tests: add corrupt loose object test
Fix a blindspot in the tests for "cat-file" (and by proxy, the guts of
object-file.c) by testing that when we can't decode a loose object
with zlib we'll emit an error from zlib.c.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-01 15:05:59 -07:00
Ævar Arnfjörð Bjarmason 59b8283d55 cat-file tests: test for missing/bogus object with -t, -s and -p
When we look up a missing object with cat_one_file() what error we
print out currently depends on whether we'll error out early in
get_oid_with_context(), or if we'll get an error later from
oid_object_info_extended().

The --allow-unknown-type flag then changes whether we pass the
"OBJECT_INFO_ALLOW_UNKNOWN_TYPE" flag to get_oid_with_context() or
not.

The "-p" flag is yet another special-case in printing the same output
on the deadbeef OID as we'd emit on the deadbeef_short OID for the
"-s" and "-t" options, it also doesn't support the
"--allow-unknown-type" flag at all.

Let's test the combination of the two sets of [-t, -s, -p] and
[--{no-}allow-unknown-type] (the --no-allow-unknown-type is implicit
in not supplying it), as well as a [missing,bogus] object pair.

This extends tests added in 3e370f9faf (t1006: add tests for git
cat-file --allow-unknown-type, 2015-05-03).

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-01 15:05:59 -07:00
Ævar Arnfjörð Bjarmason 70e4a57762 cat-file tests: move bogus_* variable declarations earlier
Change the short/long bogus bogus object type variables into a form
where the two sets can be used concurrently. This'll be used by
subsequently added tests.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-01 15:05:59 -07:00
Ævar Arnfjörð Bjarmason a5ed333121 fsck tests: test for garbage appended to a loose object
There wasn't any output tests for this scenario, let's ensure that we
don't regress on it in the changes that come after this.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-01 15:05:59 -07:00
Ævar Arnfjörð Bjarmason 42cd635b21 fsck tests: test current hash/type mismatch behavior
If fsck we move an object around between .git/objects/?? directories
to simulate a hash mismatch "git fsck" will currently hard die() in
object-file.c. This behavior will be fixed in subsequent commits, but
let's test for it as-is for now.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-01 15:05:59 -07:00
Ævar Arnfjörð Bjarmason f7a0dba7a2 fsck tests: refactor one test to use a sub-repo
Refactor one of the fsck tests to use a throwaway repository. It's a
pervasive pattern in t1450-fsck.sh to spend a lot of effort on the
teardown of a tests so we're not leaving corrupt content for the next
test.

We can instead use the pattern of creating a named sub-repository,
then we don't have to worry about cleaning up after ourselves, nobody
will care what state the broken "hash-mismatch" repository is after
this test runs.

See [1] for related discussion on various "modern" test patterns that
can be used to avoid verbosity and increase reliability.

1. https://lore.kernel.org/git/87y27veeyj.fsf@evledraar.gmail.com/

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-01 15:05:59 -07:00
Ævar Arnfjörð Bjarmason 093fffdfbe fsck tests: add test for fsck-ing an unknown type
Fix a blindspot in the fsck tests by checking what we do when we
encounter an unknown "garbage" type produced with hash-object's
--literally option.

This behavior needs to be improved, which'll be done in subsequent
patches, but for now let's test for the current behavior.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-01 15:05:59 -07:00
Junio C Hamano 107691cb07 Merge branch 'ds/sparse-index-protections'
Fix access to uninitialized piece of memory, introduced during this
cycle.

* ds/sparse-index-protections:
  sparse-index: fix uninitialized jump
2021-05-21 05:50:38 +09:00
Junio C Hamano 2b8b1aa6ad Merge branch 'tz/c-locale-output-is-no-more'
Test update.

* tz/c-locale-output-is-no-more:
  t7500: remove non-existant C_LOCALE_OUTPUT prereq
2021-05-21 05:50:32 +09:00
Junio C Hamano c69f2f8c86 Merge branch 'cs/http-use-basic-after-failed-negotiate'
Regression fix for a change made during this cycle.

* cs/http-use-basic-after-failed-negotiate:
  Revert "remote-curl: fall back to basic auth if Negotiate fails"
  t5551: test http interaction with credential helpers
2021-05-21 05:49:41 +09:00
Junio C Hamano 88dd4282d9 A handful more topics before -rc1
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-20 08:55:00 +09:00
Junio C Hamano cb227d5cd6 Merge branch 'jk/test-chainlint-softer'
The "chainlint" feature in the test framework is a handy way to
catch common mistakes in writing new tests, but tends to get
expensive.  An knob to selectively disable it has been introduced
to help running tests that the developer has not modified.

* jk/test-chainlint-softer:
  t: avoid sed-based chain-linting in some expensive cases
2021-05-20 08:55:00 +09:00
Junio C Hamano 02112fcb70 Merge branch 'en/prompt-under-set-u'
The bash prompt script (in contrib/) did not work under "set -u".

* en/prompt-under-set-u:
  git-prompt: work under set -u
2021-05-20 08:55:00 +09:00
Junio C Hamano 36a255acd1 Merge branch 'zh/ref-filter-push-remote-fix'
The handling of "%(push)" formatting element of "for-each-ref" and
friends was broken when the same codepath started handling
"%(push:<what>)", which has been corrected.

* zh/ref-filter-push-remote-fix:
  ref-filter: fix read invalid union member bug
2021-05-20 08:55:00 +09:00
Junio C Hamano bdff0419da Merge branch 'ew/sha256-clone-remote-curl-fix'
"git clone" from SHA256 repository by Git built with SHA-1 as the
default hash algorithm over the dumb HTTP protocol did not
correctly set up the resulting repository, which has been corrected.

* ew/sha256-clone-remote-curl-fix:
  remote-curl: fix clone on sha256 repos
2021-05-20 08:54:59 +09:00
Junio C Hamano 33be431c0c Merge branch 'en/dir-traversal'
"git clean" and "git ls-files -i" had confusion around working on
or showing ignored paths inside an ignored directory, which has
been corrected.

* en/dir-traversal:
  dir: introduce readdir_skip_dot_and_dotdot() helper
  dir: update stale description of treat_directory()
  dir: traverse into untracked directories if they may have ignored subfiles
  dir: avoid unnecessary traversal into ignored directory
  t3001, t7300: add testcase showcasing missed directory traversal
  t7300: add testcase showing unnecessary traversal into ignored directory
  ls-files: error out on -i unless -o or -c are specified
  dir: report number of visited directories and paths with trace2
  dir: convert trace calls to trace2 equivalents
2021-05-20 08:54:59 +09:00
Junio C Hamano 2e2ed74be0 Merge branch 'ab/perl-makefile-cleanup'
Build procedure clean-up.

* ab/perl-makefile-cleanup:
  Makefile: make PERL_DEFINES recursively expanded
  perl: use mock i18n functions under NO_GETTEXT=Y
  Makefile: regenerate *.pm on NO_PERL_CPAN_FALLBACKS change
  Makefile: regenerate perl/build/* if GIT-PERL-DEFINES changes
  Makefile: don't re-define PERL_DEFINES
2021-05-20 08:54:58 +09:00
Jeff King ecf7b129fa Revert "remote-curl: fall back to basic auth if Negotiate fails"
This reverts commit 1b0d9545bb.

That commit does fix the situation it intended to (avoiding Negotiate
even when the credentials were provided in the URL), but it creates a
more serious regression: we now never hit the conditional for "we had a
username and password, tried them, but the server still gave us a 401".
That has two bad effects:

 1. we never call credential_reject(), and thus a bogus credential
    stored by a helper will live on forever

 2. we never return HTTP_NOAUTH, so the error message the user gets is
    "The requested URL returned error: 401", instead of "Authentication
    failed".

Doing this correctly seems non-trivial, as we don't know whether the
Negotiate auth was a problem. Since this is a regression in the upcoming
v2.23.0 release (for which we're in -rc0), let's revert for now and work
on a fix separately.

(Note that this isn't a pure revert; the previous commit added a test
showing the regression, so we can now flip it to expect_success).

Reported-by: Ben Humphreys <behumphreys@atlassian.com>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-19 10:09:58 +09:00
Jeff King b694f1e49e t5551: test http interaction with credential helpers
We test authentication with http, and we independently test that
credential helpers work, but we don't have any tests that cover the
two features working together. Let's add two:

  1. Make sure that a successful request asks the helper to save the
     credential. This works as expected.

  2. Make sure that a failed request asks the helper to forget the
     credential. This is marked as expect_failure, as it was recently
     regressed by 1b0d9545bb (remote-curl: fall back to basic auth if
     Negotiate fails, 2021-03-22). The symptom here is that the second
     request should prompt the user, but doesn't.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-19 10:09:57 +09:00
Derrick Stolee 4279cb1c6e sparse-index: fix uninitialized jump
While testing the sparse-index, I verified a test with --valgrind and it
complained about an uninitialized value being used in a jump in the
path_matches_pattern_list() method. The line was this one:

	if (*dtype == DT_UNKNOWN)

In the call stack, the culprit was the initialization of the dtype
variable in convert_to_sparse_rec().

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-18 06:29:17 +09:00
Todd Zullinger 58cf6056c9 t7500: remove non-existant C_LOCALE_OUTPUT prereq
The C_LOCALE_OUTPUT prerequisite was removed in b1e079807b (tests:
remove last uses of C_LOCALE_OUTPUT, 2021-02-11), where Ævar noted:

    I'm not leaving the prerequisite itself in place for in-flight changes
    as there currently are none that introduce new tests that rely on it,
    and because C_LOCALE_OUTPUT is currently a noop on the master branch
    we likely won't have any new submissions that use it.

One more use of C_LOCALE_OUTPUT did creep in with 3d1bda6b5b (t7500: add
tests for --fixup=[amend|reword] options, 2021-03-15).  This causes a
number of the tests to be skipped by default:

    ok 35 # SKIP --fixup=reword: incompatible with --all (missing C_LOCALE_OUTPUT)
    ok 36 # SKIP --fixup=reword: incompatible with --include (missing C_LOCALE_OUTPUT)
    ok 37 # SKIP --fixup=reword: incompatible with --only (missing C_LOCALE_OUTPUT)
    ok 38 # SKIP --fixup=reword: incompatible with --interactive (missing C_LOCALE_OUTPUT)
    ok 39 # SKIP --fixup=reword: incompatible with --patch (missing C_LOCALE_OUTPUT)

Remove the C_LOCALE_OUTPUT prerequisite from these tests so they are
not skipped.

Signed-off-by: Todd Zullinger <tmz@pobox.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-18 04:48:30 +09:00
Junio C Hamano bf949ade81 Git 2.32-rc0
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-16 21:05:24 +09:00
Junio C Hamano e004fd6b69 Merge branch 'ls/typofix'
* ls/typofix:
  pretty: fix a typo in the documentation for %(trailers)
2021-05-16 21:05:24 +09:00
Junio C Hamano a8a2491e62 Merge branch 'dl/stash-show-untracked-fixup'
The code to handle options recently added to "git stash show"
around untracked part of the stash segfaulted when these options
were used on a stash entry that does not record untracked part.

* dl/stash-show-untracked-fixup:
  stash show: fix segfault with --{include,only}-untracked
  t3905: correct test title
2021-05-16 21:05:24 +09:00
Junio C Hamano 16f91451fa Merge branch 'wc/packed-ref-removal-cleanup'
When "git update-ref -d" removes a ref that is packed, it left
empty directories under $GIT_DIR/refs/ for

* wc/packed-ref-removal-cleanup:
  refs: cleanup directories when deleting packed ref
2021-05-16 21:05:24 +09:00
Junio C Hamano 94294e92e1 Merge branch 'lh/maintenance-leakfix'
* lh/maintenance-leakfix:
  maintenance: fix two memory leaks
2021-05-16 21:05:24 +09:00
Junio C Hamano caf6840be0 Merge branch 'ma/typofixes'
A couple of trivial typofixes.

* ma/typofixes:
  pretty-formats.txt: add missing space
  git-repack.txt: remove spurious ")"
2021-05-16 21:05:24 +09:00
Junio C Hamano c7c7c460f8 Merge branch 'ah/merge-ort-i18n'
An i18n fix.

* ah/merge-ort-i18n:
  merge-ort: split "distinct types" message into two translatable messages
2021-05-16 21:05:23 +09:00
Junio C Hamano 483932a3d8 Merge branch 'dd/mailinfo-quoted-cr'
"git mailinfo" (hence "git am") learned the "--quoted-cr" option to
control how lines ending with CRLF wrapped in base64 or qp are
handled.

* dd/mailinfo-quoted-cr:
  am: learn to process quoted lines that ends with CRLF
  mailinfo: allow stripping quoted CR without warning
  mailinfo: allow squelching quoted CRLF warning
  mailinfo: warn if CRLF found in decoded base64/QP email
  mailinfo: stop parsing options manually
  mailinfo: load default metainfo_charset lazily
2021-05-16 21:05:23 +09:00
Junio C Hamano c8e34a7ac2 Merge branch 'ab/sparse-index-cleanup'
Code clean-up.

* ab/sparse-index-cleanup:
  sparse-index.c: remove set_index_sparse_config()
2021-05-16 21:05:23 +09:00
Junio C Hamano 502a67891c Merge branch 'ab/streaming-simplify'
Code clean-up.

* ab/streaming-simplify:
  streaming.c: move {open,close,read} from vtable to "struct git_istream"
  streaming.c: stop passing around "object_info *" to open()
  streaming.c: remove {open,close,read}_method_decl() macros
  streaming.c: remove enum/function/vtbl indirection
  streaming.c: avoid forward declarations
2021-05-16 21:05:23 +09:00
Junio C Hamano a737e1f1d2 Merge branch 'mt/parallel-checkout-part-3'
The final part of "parallel checkout".

* mt/parallel-checkout-part-3:
  ci: run test round with parallel-checkout enabled
  parallel-checkout: add tests related to .gitattributes
  t0028: extract encoding helpers to lib-encoding.sh
  parallel-checkout: add tests related to path collisions
  parallel-checkout: add tests for basic operations
  checkout-index: add parallel checkout support
  builtin/checkout.c: complete parallel checkout support
  make_transient_cache_entry(): optionally alloc from mem_pool
2021-05-16 21:05:23 +09:00
Junio C Hamano 644f4a2046 Merge branch 'jt/push-negotiation'
"git push" learns to discover common ancestor with the receiving
end over protocol v2.

* jt/push-negotiation:
  send-pack: support push negotiation
  fetch: teach independent negotiation (no packfile)
  fetch-pack: refactor command and capability write
  fetch-pack: refactor add_haves()
  fetch-pack: refactor process_acks()
2021-05-16 21:05:22 +09:00
Junio C Hamano 97eea85a0a The seventeenth batch
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-14 08:26:11 +09:00
Junio C Hamano 52371bf449 Merge branch 'mt/clean-clean'
Code clean-up.

* mt/clean-clean:
  clean: remove unnecessary variable
2021-05-14 08:26:11 +09:00
Junio C Hamano 47fa106617 Merge branch 'ow/no-dryrun-in-add-i'
"git add -i --dry-run" does not dry-run, which was surprising.  The
combination of options has taught to error out.

* ow/no-dryrun-in-add-i:
  add: die if both --dry-run and --interactive are given
2021-05-14 08:26:09 +09:00
Junio C Hamano e289f681ed Merge branch 'jk/p4-locate-branch-point-optim'
"git p4" learned to find branch points more efficiently.

* jk/p4-locate-branch-point-optim:
  git-p4: speed up search for branch parent
  git-p4: ensure complex branches are cloned correctly
2021-05-14 08:26:08 +09:00
Junio C Hamano eede71149e Merge branch 'ba/object-info'
Over-the-wire protocol learns a new request type to ask for object
sizes given a list of object names.

* ba/object-info:
  object-info: support for retrieving object info
2021-05-14 08:26:08 +09:00
Junio C Hamano daffa8961b Merge branch 'pw/patience-diff-clean-up'
Code clean-up.

* pw/patience-diff-clean-up:
  patience diff: remove unused variable
  patience diff: remove unnecessary string comparisons
2021-05-14 08:26:08 +09:00
Junio C Hamano 65c18913de Merge branch 'pw/word-diff-zero-width-matches'
The word-diff mode has been taught to work better with a word
regexp that can match an empty string.

* pw/word-diff-zero-width-matches:
  word diff: handle zero length matches
2021-05-14 08:26:06 +09:00
Jeff King 2d86a96220 t: avoid sed-based chain-linting in some expensive cases
Commit 878f988350 (t/test-lib: teach --chain-lint to detect broken
&&-chains in subshells, 2018-07-11) introduced additional chain-lint
tests which add an extra "sed" pipeline to each test we run. This has a
measurable impact on runtime. Here are timings with and without a new
environment variable (added by this patch) that lets you disable just
the additional sed-based chain-lint tests:

  Benchmark #1: GIT_TEST_CHAIN_LINT_HARDER=1 make test
    Time (mean ± σ):     64.202 s ±  1.030 s    [User: 622.469 s, System: 301.402 s]
    Range (min … max):   61.571 s … 65.662 s    10 runs

  Benchmark #2: GIT_TEST_CHAIN_LINT_HARDER=0 make test
    Time (mean ± σ):     57.591 s ±  0.333 s    [User: 529.368 s, System: 270.618 s]
    Range (min … max):   57.143 s … 58.309 s    10 runs

  Summary
    'GIT_TEST_CHAIN_LINT_HARDER=0 make test' ran
      1.11 ± 0.02 times faster than 'GIT_TEST_CHAIN_LINT_HARDER=1 make test'

Of course those extra lint checks are doing something useful, so paying
a few extra seconds (at least on Linux) isn't so bad (though note the
CPU time; we're bounded in our parallel run here by the slowest test, so
it really is ~120s of CPU improvement).

But we can observe that there are some test scripts where they produce a
much stronger effect, and provide less value. In t0027 and t3070 we run
a very large number of small tests, all driven by a series of
functions/loops which are filling in the test bodies. There we get much
less bang for our buck in terms of bug-finding versus CPU cost.

This patch introduces a mechanism for controlling when those extra
lint checks are run, at two levels:

  - a user can ask to disable or to force-enable the checks by setting
    GIT_TEST_CHAIN_LINT_HARDER

  - if the user hasn't specified a preference, individual scripts can
    disable the checks by setting GIT_TEST_CHAIN_LINT_HARDER_DEFAULT;
    scripts which don't set that get the current behavior of enabling
    them.

In addition, this patch flips the default for t0027 and t3070's
mass-generated sections to disable the extra checks. Here are the timing
results for t0027:

  Benchmark #1: GIT_TEST_CHAIN_LINT_HARDER=1 ./t0027-auto-crlf.sh
    Time (mean ± σ):     17.078 s ±  0.848 s    [User: 14.878 s, System: 7.075 s]
    Range (min … max):   15.952 s … 18.421 s    10 runs

  Benchmark #2: GIT_TEST_CHAIN_LINT_HARDER=0 ./t0027-auto-crlf.sh
    Time (mean ± σ):      9.063 s ±  0.759 s    [User: 7.890 s, System: 3.362 s]
    Range (min … max):    7.747 s … 10.619 s    10 runs

  Benchmark #3: ./t0027-auto-crlf.sh
    Time (mean ± σ):      9.186 s ±  0.881 s    [User: 7.957 s, System: 3.427 s]
    Range (min … max):    7.796 s … 10.498 s    10 runs

  Summary
    'GIT_TEST_CHAIN_LINT_HARDER=0 ./t0027-auto-crlf.sh' ran
      1.01 ± 0.13 times faster than './t0027-auto-crlf.sh'
      1.88 ± 0.18 times faster than 'GIT_TEST_CHAIN_LINT_HARDER=1 ./t0027-auto-crlf.sh'

We can see that disabling the checks for the whole script buys us an
almost 2x speedup. But the new default behavior, disabling them only for
the mass-generated part, gets us most of that speedup (but still leaves
the checks on for further manual tests people might write).

  As a side note, I'd caution about comparing runtimes and CPU seconds
  between this timing and the earlier "make test" one. In "make test",
  we're running a lot of scripts in parallel, so the CPU is throttling
  down (and thus a CPU second saved here would count for more during a
  parallel run; the same work takes more CPU seconds there).

We get similar results for t3070:

  Benchmark #1: GIT_TEST_CHAIN_LINT_HARDER=1 ./t3070-wildmatch.sh
    Time (mean ± σ):     20.054 s ±  3.967 s    [User: 16.003 s, System: 8.286 s]
    Range (min … max):   11.891 s … 23.671 s    10 runs

  Benchmark #2: GIT_TEST_CHAIN_LINT_HARDER=0 ./t3070-wildmatch.sh
    Time (mean ± σ):     12.399 s ±  2.256 s    [User: 7.542 s, System: 5.342 s]
    Range (min … max):    9.606 s … 15.727 s    10 runs

  Benchmark #3: ./t3070-wildmatch.sh
    Time (mean ± σ):     10.726 s ±  3.476 s    [User: 6.790 s, System: 4.365 s]
    Range (min … max):    5.444 s … 15.376 s    10 runs

  Summary
    './t3070-wildmatch.sh' ran
      1.16 ± 0.43 times faster than 'GIT_TEST_CHAIN_LINT_HARDER=0 ./t3070-wildmatch.sh'
      1.87 ± 0.71 times faster than 'GIT_TEST_CHAIN_LINT_HARDER=1 ./t3070-wildmatch.sh'

Again, we get almost a 2x speedup disabling these. In this case, there
are no tests not covered by the script's "default to disable" behavior,
so the second two benchmarks should be the same (and while they do
differ, you can see the variance is quite high but they're within one
standard deviation).

So it seems like for these two scripts, at least, disabling the extra
checks is a reasonable tradeoff. Sadly, the overall runtime of "make
test" on my system doesn't get much faster. But that's because we're
mostly limited by the cost of the single biggest test. Here are the
top-5 tests by wall-clock time from a parallel run, before my patch:

  57.9192368984222 t9001-send-email.sh
  45.6329638957977 t0027-auto-crlf.sh
  32.5278220176697 t3070-wildmatch.sh
  22.2701289653778 t7610-mergetool.sh
  20.8635759353638 t1701-racy-split-index.sh

And after:

  57.1476998329163 t9001-send-email.sh
  33.776211977005 t0027-auto-crlf.sh
  21.3116669654846 t7610-mergetool.sh
  20.7748689651489 t1701-racy-split-index.sh
  19.6957249641418 t7112-reset-submodule.sh

We dropped 12s from t0027, and t3070 dropped off our list entirely at
around 16s. In both cases we're bound by t9001, but its slowness is
due to the actual tests, so we'll have to deal with it in a different
way. But this reduces overall CPU, and means that dealing with t9001 (by
improving the speed of send-email or splitting it apart) will let us
reduce our overall runtime even on multi-core machines.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-13 15:50:44 +09:00
Elijah Newren 5c0cbdb107 git-prompt: work under set -u
Commit afda36dbf3 ("git-prompt: include sparsity state as well",
2020-06-21) added the use of some variables to control how to show
sparsity state in the git prompt, but implicitly assumed that undefined
variables would be treated as the empty string.  This breaks users who
run under 'set -u'; fix the code to be more explicit.

Signed-off-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-13 15:50:26 +09:00