This allows to run external commands in parallel with ordered output
on stderr.
If we run external commands in parallel we cannot pipe the output directly
to the our stdout/err as it would mix up. So each process's output will
flow through a pipe, which we buffer. One subprocess can be directly
piped to out stdout/err for a low latency feedback to the user.
Example:
Let's assume we have 5 submodules A,B,C,D,E and each fetch takes a
different amount of time as the different submodules vary in size, then
the output of fetches in sequential order might look like this:
time -->
output: |---A---| |-B-| |-------C-------| |-D-| |-E-|
When we schedule these submodules into maximal two parallel processes,
a schedule and sample output over time may look like this:
process 1: |---A---| |-D-| |-E-|
process 2: |-B-| |-------C-------|
output: |---A---|B|---C-------|DE
So A will be perceived as it would run normally in the single child
version. As B has finished by the time A is done, we can dump its whole
progress buffer on stderr, such that it looks like it finished in no
time. Once that is done, C is determined to be the visible child and
its progress will be reported in real time.
So this way of output is really good for human consumption, as it only
changes the timing, not the actual output.
For machine consumption the output needs to be prepared in the tasks,
by either having a prefix per line or per block to indicate whose tasks
output is displayed, because the output order may not follow the
original sequential ordering:
|----A----| |--B--| |-C-|
will be scheduled to be all parallel:
process 1: |----A----|
process 2: |--B--|
process 3: |-C-|
output: |----A----|CB
This happens because C finished before B did, so it will be queued for
output before B.
To detect when a child has finished executing, we check interleaved
with other actions (such as checking the liveliness of children or
starting new processes) whether the stderr pipe still exists. Once a
child closed its stderr stream, we assume it is terminating very soon,
and use `finish_command()` from the single external process execution
interface to collect the exit status.
By maintaining the strong assumption of stderr being open until the
very end of a child process, we can avoid other hassle such as an
implementation using `waitpid(-1)`, which is not implemented in Windows.
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
POSIXPERM requires that a later call to stat(2) (hence "ls -l")
faithfully reproduces what an earlier chmod(2) did. Some
filesystems cannot satisify this.
SANITY requires that a file or a directory is indeed accessible (or
inaccessible) when its permission bits would say it ought to be
accessible (or inaccessible). Running tests as root would lose this
prerequisite for obvious reasons.
Fix a few tests that misuse POSIXPERM.
t0061-run-command.sh has two uses of POSIXPERM.
- One checks that an attempt to execute a file that is marked as
unexecutable results in a failure with EACCES; I do not think
having root-ness or any other capability that busts the
filesystem permission mode bits will make you run an unexecutable
file, so this should be left as-is. The test does not have
anything to do with SANITY.
- The other one expects 'git nitfol' runs the alias when an
alias.nitfol is defined and a directory on the PATH is marked as
unreadable and unsearchable. I _think_ the test tries to reject
the alternative expectation that we want to refuse to run the
alias because it would break "no alias may mask a command" rule
if a file 'git-nitfol' exists in the unreadable directory but we
cannot even determine if that is the case. Under !SANITY that
busts the permission bits, this test no longer checks that, so it
must be protected with SANITY.
t1509-root-worktree.sh expects to be run on a / that is writable by
the user and sees if Git behaves "sensibly" when /.git is the
repository to govern a worktree that is the whole filesystem, and
also if Git behaves "sensibly" when / itself is a bare repository
with refs, objects, and friends (I find the definition of "behaves
sensibly" under these conditions hard to fathom, but it is a
different matter).
The implementation of the test is very much problematic.
- It requires POSIXPERM, but it does not do chmod or checks modes
in any way.
- It runs "rm /*" and "rm -fr /refs /objects ..." in one of the
tests, and also does "cd / && git init --bare". If done on a
live system that takes advantages of the "feature" being tested,
these obviously will clobber the system. But there is no guard
against such a breakage.
- It uses "test $UID = 0" to see rootness, which now should be
spelled "! test_have_prereq NOT_ROOT"
Signed-off-by: Junio C Hamano <gitster@pobox.com>
When execvp reports EACCES, it can be one of two things:
1. We found a file to execute, but did not have
permissions to do so.
2. We did not have permissions to look in some directory
in the $PATH.
In the former case, we want to consider this a
permissions problem and report it to the user as such (since
getting this for something like "git foo" is likely a
configuration error).
In the latter case, there is a good chance that the
inaccessible directory does not contain anything of
interest. Reporting "permission denied" is confusing to the
user (and prevents our usual "did you mean...?" lookup). It
also prevents git from trying alias lookup, since we do so
only when an external command does not exist (not when it
exists but has an error).
This patch detects EACCES from execvp, checks whether we are
in case (2), and if so converts errno to ENOENT. This
behavior matches that of "bash" (but not of simpler shells
that use execvp more directly, like "dash").
Test stolen from Junio.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
In git versions starting at v1.7.5-rc0~29^2 until v1.7.5-rc3~2 (Revert
"run-command: prettify -D_FORTIFY_SOURCE workaround", 2011-04-18)
fixed it, the run_command facility would write a truncated error
message when the command is present but cannot be executed for some
other reason. For example, if I add a 'hello' command to git:
$ echo 'echo hello' >git-hello
$ chmod +x git-hello
$ PATH=.:$PATH git hello
hello
and make it non-executable, this is what I normally get:
$ chmod -x git-hello
$ git hello
fatal: cannot exec 'git-hello': Permission denied
But with the problematic versions, we get disturbing output:
$ PATH=.:$PATH git hello
fatal: $
Add some tests to make sure it doesn't happen again.
The hello-script used in these tests uses cat instead of echo because
on Windows the bash spawned by git converts LF to CRLF in text written
by echo while the bash running tests does not, causing the test to
fail if "echo" is used. Thanks to Hannes for noticing.
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Improved-by: Johannes Sixt <j6t@kdbg.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Previously, failures during execvp could be detected only by
finish_command. However, in some situations it is beneficial for the
parent process to know earlier that the child process will not run.
The idea to use a pipe to signal failures to the parent process and
the test case were lifted from patches by Ilari Liusvaara.
Signed-off-by: Johannes Sixt <j6t@kdbg.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
