* jk/http-backend-deadlock-2.2:
http-backend: spool ref negotiation requests to buffer
t5551: factor out tag creation
http-backend: fix die recursion with custom handler
When http-backend spawns "upload-pack" to do ref
negotiation, it streams the http request body to
upload-pack, who then streams the http response back to the
client as it reads. In theory, git can go full-duplex; the
client can consume our response while it is still sending
the request. In practice, however, HTTP is a half-duplex
protocol. Even if our client is ready to read and write
simultaneously, we may have other HTTP infrastructure in the
way, including the webserver that spawns our CGI, or any
intermediate proxies.
In at least one documented case[1], this leads to deadlock
when trying a fetch over http. What happens is basically:
1. Apache proxies the request to the CGI, http-backend.
2. http-backend gzip-inflates the data and sends
the result to upload-pack.
3. upload-pack acts on the data and generates output over
the pipe back to Apache. Apache isn't reading because
it's busy writing (step 1).
This works fine most of the time, because the upload-pack
output ends up in a system pipe buffer, and Apache reads
it as soon as it finishes writing. But if both the request
and the response exceed the system pipe buffer size, then we
deadlock (Apache blocks writing to http-backend,
http-backend blocks writing to upload-pack, and upload-pack
blocks writing to Apache).
We need to break the deadlock by spooling either the input
or the output. In this case, it's ideal to spool the input,
because Apache does not start reading either stdout _or_
stderr until we have consumed all of the input. So until we
do so, we cannot even get an error message out to the
client.
The solution is fairly straight-forward: we read the request
body into an in-memory buffer in http-backend, freeing up
Apache, and then feed the data ourselves to upload-pack. But
there are a few important things to note:
1. We limit the in-memory buffer to prevent an obvious
denial-of-service attack. This is a new hard limit on
requests, but it's unlikely to come into play. The
default value is 10MB, which covers even the ridiculous
100,000-ref negotation in the included test (that
actually caps out just over 5MB). But it's configurable
on the off chance that you don't mind spending some
extra memory to make even ridiculous requests work.
2. We must take care only to buffer when we have to. For
pushes, the incoming packfile may be of arbitrary
size, and we should connect the input directly to
receive-pack. There's no deadlock problem here, though,
because we do not produce any output until the whole
packfile has been read.
For upload-pack's initial ref advertisement, we
similarly do not need to buffer. Even though we may
generate a lot of output, there is no request body at
all (i.e., it is a GET, not a POST).
[1] http://article.gmane.org/gmane.comp.version-control.git/269020
Test-adapted-from: Dennis Kaarsemaker <dennis@kaarsemaker.net>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
One of our tests in t5551 creates a large number of tags,
and jumps through some hoops to do it efficiently. Let's
factor that out into a function so we can make other similar
tests.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When we die() in http-backend, we call a custom handler that
writes an HTTP 500 response to stdout, then reports the
error to stderr. Our routines for writing out the HTTP
response may themselves die, leading to us entering die()
again.
When it was originally written, that was OK; our custom
handler keeps a variable to notice this and does not
recurse. However, since cd163d4 (usage.c: detect recursion
in die routines and bail out immediately, 2012-11-14), the
main die() implementation detects recursion before we even
get to our custom handler, and bails without printing
anything useful.
We can handle this case by doing two things:
1. Installing a custom die_is_recursing handler that
allows us to enter up to one level of recursion. Only
the first call to our custom handler will try to write
out the error response. So if we die again, that is OK.
If we end up dying more than that, it is a sign that we
are in an infinite recursion.
2. Reporting the error to stderr before trying to write
out the HTTP response. In the current code, if we do
die() trying to write out the response, we'll exit
immediately from this second die(), and never get a
chance to output the original error (which is almost
certainly the more interesting one; the second die is
just going to be along the lines of "I tried to write
to stdout but it was closed").
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
We create 50,000 tags to check that we don't overflow the
command-line of fetch-pack. But by using run_with_cmdline_limit,
we can get the same effect with a much smaller number of
tags. This makes the test fast enough that we can drop the
EXPENSIVE prereq, which means people will actually run it.
It was not documented to do so, but this test was also the
only test of a clone-over-http that requires multiple POSTs
during the conversation. We can continue to test that by
dropping http.postbuffer to its minimum size, and checking
that we get two POSTs.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
We use this to test http pushing with a restricted
commandline. Other scripts (like t5551, which does http
fetching) will want to use it, too.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Apache removes GIT_TRACE from the environment before running
git-http-backend. This can make it hard to debug the server
side of an http session. Let's let it through.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
If you run a test script like:
GIT_TRACE=1 ./t0061-run-command.sh
you may get test failures, because some tests capture and
check the stderr output from git commands (and with
GIT_TRACE set to 1, the trace output will be included
there).
When we see GIT_TRACE set like this, we print a warning to
the user. However, we can do even better than that by just
pointing it to descriptor 4, which all tests leave connected
to the test script's stderr. That's likely what the user
intended (and any scripts that do want to see GIT_TRACE
output will set GIT_TRACE themselves).
Not only does this avoid false negatives in the tests, but
it means the user will actually see trace output for git
calls that redirect their stderr (whereas before, it was
sometimes confusingly buried in a file).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Right now if a test script receives SIGINT (e.g., because a
test was hanging and the user hit ^C), the shell exits
immediately. This can be annoying if the test script did any
global setup, like starting apache or git-daemon, as it will
not have an opportunity to clean up after itself. A
subsequent run of the test won't be able to start its own
daemon, and will either fail or skip the tests.
Instead, let's trap SIGINT to make sure we do a clean
shutdown, and just chain it to a normal exit (which will
trigger any cleanup).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Code cleanups.
* rs/simple-cleanups:
sha1_name: use strlcpy() to copy strings
pretty: use starts_with() to check for a prefix
for-each-ref: use skip_prefix() to avoid duplicate string comparison
connect: use strcmp() for string comparison
The configuration variable 'mailinfo.scissors' was hard to
discover in the documentation.
* mm/am-c-doc:
Documentation/git-am.txt: mention mailinfo.scissors config variable
Documentation/config.txt: document mailinfo.scissors
Correct a breakage to git-svn around v2.2 era that triggers
premature closing of FileHandle.
* ew/svn-maint-fixes:
Git::SVN::*: avoid premature FileHandle closure
git-svn: fix localtime=true on non-glibc environments
Even though we officially haven't dropped Perl 5.8 support, the
Getopt::Long package that came with it does not support "--no-"
prefix to negate a boolean option; manually add support to help
people with older Getopt::Long package.
* km/send-email-getopt-long-workarounds:
git-send-email.perl: support no- prefix with older GetOptions
"update-index --refresh" used to leak when an entry cannot be
refreshed for whatever reason.
* sb/plug-leak-in-make-cache-entry:
read-cache.c: free cache entry when refreshing fails
"git fast-import" used to crash when it could not close and
conclude the resulting packfile cleanly.
* jk/fast-import-die-nicely-fix:
fast-import: avoid running end_packfile recursively
In v2.2.0, we broke "git prune" that runs in a repository that
borrows from an alternate object store.
* jk/prune-mtime:
sha1_file: fix iterating loose alternate objects
for_each_loose_file_in_objdir: take an optional strbuf path
Certain older vintages of cURL give irregular output from
"curl-config --vernum", which confused our build system.
* tc/curl-vernum-output-broken-in-7.11:
Makefile: handle broken curl version number in version check
An earlier workaround to squelch unhelpful deprecation warnings
from the complier on Mac OSX unnecessarily set minimum required
version of the OS, which the user might want to raise (or lower)
for other reasons.
* es/squelch-openssl-warnings-on-macosx:
git-compat-util: do not step on MAC_OS_X_VERSION_MIN_REQUIRED
Longstanding configuration variable naming rules has been added to
the documentation.
* jc/conf-var-doc:
CodingGuidelines: describe naming rules for configuration variables
config.txt: mark deprecated variables more prominently
config.txt: clarify that add.ignore-errors is deprecated
The credential helper for Windows (in contrib/) used to mishandle
a user name with an at-sign in it.
* av/wincred-with-at-in-username-fix:
wincred: fix get credential if username has "@"
Older GnuPG implementations may not correctly import the keyring
material we prepare for the tests to use.
* ch/new-gpg-drops-rfc-1991:
t/lib-gpg: sanity-check that we can actually sign
t/lib-gpg: include separate public keys in keyring.gpg
Clarify in the documentation that "remote.<nick>.pushURL" and
"remote.<nick>.URL" are there to name the same repository accessed
via different transports, not two separate repositories.
* jc/remote-set-url-doc:
Documentation/git-remote.txt: stress that set-url is not for triangular
Reading configuration from a blob object, when it ends with a lone
CR, use to confuse the configuration parser.
* jk/config-no-ungetc-eof:
config_buf_ungetc: warn when pushing back a random character
config: do not ungetc EOF
We didn't format an integer that wouldn't fit in "int" but in
"uintmax_t" correctly.
* jk/decimal-width-for-uintmax:
decimal_width: avoid integer overflow
"git push --signed" gave an incorrectly worded error message when
the other side did not support the capability.
* jc/push-cert:
transport-helper: fix typo in error message when --signed is not supported
"git fetch" over a remote-helper that cannot respond to "list"
command could not fetch from a symbolic reference e.g. HEAD.
* mh/deref-symref-over-helper-transport:
transport-helper: do not request symbolic refs to remote helpers
The insn sheet "git rebase -i" creates did not fully honor
core.abbrev settings.
* ks/rebase-i-abbrev:
rebase -i: use full object name internally throughout the script
The tests that wanted to see that file becomes unreadable after
running "chmod a-r file", and the tests that wanted to make sure it
is not run as root, we used "can we write into the / directory?" as
a cheap substitute, but on some platforms that is not a good
heuristics. The tests and their prerequisites have been updated to
check what they really require.
* jk/sanity:
test-lib.sh: set prerequisite SANITY by testing what we really need
tests: correct misuses of POSIXPERM
t/lib-httpd: switch SANITY check for NOT_ROOT
Since b19138b (git-svn: Make it incrementally faster by minimizing temp
files, v1.6.0), git-svn has been using the Git.pm temp_acquire and
temp_release mechanism to avoid unnecessary temp file churn and provide
a speed boost.
However, that change introduced a call to temp_acquire inside the
Git::SVN::Fetcher::close_file function for an 'svn_hash' temp file.
Because an SVN::Pool is active at the time this function is called, if
the Git::temp_acquire function ends up actually creating a new
FileHandle for the temp file (which it will the first time it's called
with the name 'svn_hash') that FileHandle will end up in the SVN::Pool
and should that pool have SVN::Pool::clear called on it that FileHandle
will be closed out from under Git::temp_acquire.
Since the only call site to Git::temp_acquire with the name 'svn_hash'
is inside the close_file function, if an 'svn_hash' temp file is ever
created its FileHandle is guaranteed to be created in the active
SVN::Pool.
This has not been a problem in the past because the SVN::Pool was not
being cleared. However, since dfa72fdb (git-svn: reload RA every
log-window-size, v2.2.0) the pool has been getting cleared periodically
at which point the FileHandle for the 'svn_hash' temp file gets closed.
Any subsequent calls to Git::temp_acquire for 'svn_hash', however,
succeed without creating/opening a new temporary file since it still has
the now invalid FileHandle in its cache. Callers that then attempt to
use that FileHandle fail with an error.
We avoid this problem by making sure the 'svn_hash' temp file is created
in the same place the 'svn_delta_...' and 'git_blob_...' temp files are
(and then temp_release'd) so that it can be safely used inside the
close_file function without having its FileHandle end up in an SVN::Pool
that gets cleared.
Additionally the Git.pm cat_blob function creates a bidirectional pipe
FileHandle using the IPC::Open2::open2 function. If that handle is
created too late, it also gets caught up in the SVN::Pool and incorrectly
closed by the SVN::Pool::clear call. But this only seems to happen with
more recent versions of Perl and svn.
To avoid this problem we add an explicit call to _open_cat_blob_if_needed
before the first call to SVN::Pool->new_default to make sure the open2
handle does not end up in the SVN::Pool.
Signed-off-by: Kyle J. McKay <mackyle@gmail.com>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
git svn uses POSIX::strftime('%s', $sec, $min, ...) to make unix epoch time.
But lowercase %s formatting character is a GNU extention. This causes problem
in git svn fetch --localtime on non-glibc systems, such as msys or cygwin.
Using Time::Local::timelocal($sec, $min, ...) fixes it.
Signed-off-by: Ryuichi Kokubo <ryu1kkb@gmail.com>
Signed-off-by: Eric Wong <normalperson@yhbt.net>
Notes:
lowercase %s format character in strftime is a GNU extension and not widely supported.
POSIX::strftime affected by underlying crt's strftime because POSIX::strftime just calls crt's one.
Time::Local is good function to replace POSIX::strftime because it's a perl core module function.
Document about Time::Local.
http://perldoc.perl.org/Time/Local.html
These are specifications of strftime.
The GNU C Library Reference Manual.
http://www.gnu.org/software/libc/manual/html_node/Formatting-Calendar-Time.html
perl POSIX module's strftime document. It does not have '%s'.
http://perldoc.perl.org/POSIX.html
strftime document of Microsort Windows C Run-Time library.
https://msdn.microsoft.com/en-us/library/fe06s4ak.aspx
The Open Group's old specification does not have '%s' too.
http://pubs.opengroup.org/onlinepubs/007908799/xsh/strftime.html
On my environment, following problems happened.
- msys : git svn fetch does not progress at all with perl.exe consuming CPU.
- cygwin : git svn fetch progresses but time stamp information is dropped.
Every commits have unix epoch timestamp.
I would like to thank git developer and contibutors.
git helps me so much everyday.
Thank you.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
The interactive "show a list and let the user choose from it"
interface "add -i" used showed and prompted to the user even when
the candidate list was empty, against which the only "choice" the
user could have made was to choose nothing.
* ak/add-i-empty-candidates:
add -i: return from list_and_choose if there is no candidate
"git apply --whitespace=fix" used to under-allocate the memory
when the fix resulted in a longer text than the original patch.
* jc/apply-ws-fix-expands:
apply: count the size of postimage correctly
apply: make update_pre_post_images() sanity check the given postlen
apply.c: typofix
"git log --help" used to show rev-list options that are irrelevant
to the "log" command.
* jc/doc-log-rev-list-options:
Documentation: what does "git log --indexed-objects" even mean?
The error message from "git commit", when a non-existing author
name was given as value to the "--author=" parameter, has been
reworded to avoid misunderstanding.
* mg/commit-author-no-match-malformed-message:
commit: reword --author error message
A broken pack .idx file in the receiving repository prevented the
dumb http transport from fetching a good copy of it from the other
side.
* jk/dumb-http-idx-fetch-fix:
dumb-http: do not pass NULL path to parse_pack_index
The documentation incorrectly said that C(opy) and R(ename) are the
only ones that can be followed by the score number in the output in
the --raw format.
* jc/diff-format-doc:
diff-format doc: a score can follow M for rewrite
Code to read branch name from various files in .git/ directory
would have misbehaved if the code to write them left an empty file.
* jk/status-read-branch-name-fix:
read_and_strip_branch: fix typo'd address-of operator
The "git push" documentation made the "--repo=<there>" option
easily misunderstood.
* mg/push-repo-option-doc:
git-push.txt: document the behavior of --repo
