Commit graph

19 commits

Author SHA1 Message Date
Junio C Hamano
070d276cc1 Merge branch 'jh/filter-empty-contents' into maint
The clean/smudge interface did not work well when filtering an
empty contents (failed and then passed the empty input through).
It can be argued that a filter that produces anything but empty for
an empty input is nonsense, but if the user wants to do strange
things, then why not?

* jh/filter-empty-contents:
  sha1_file: pass empty buffer to index empty file
2015-06-16 14:33:44 -07:00
Jim Hill
f6a1e1e288 sha1_file: pass empty buffer to index empty file
`git add` of an empty file with a filter pops complaints from
`copy_fd` about a bad file descriptor.

This traces back to these lines in sha1_file.c:index_core:

	if (!size) {
		ret = index_mem(sha1, NULL, size, type, path, flags);

The problem here is that content to be added to the index can be
supplied from an fd, or from a memory buffer, or from a pathname. This
call is supplying a NULL buffer pointer and a zero size.

Downstream logic takes the complete absence of a buffer to mean the
data is to be found elsewhere -- for instance, these, from convert.c:

	if (params->src) {
		write_err = (write_in_full(child_process.in, params->src, params->size) < 0);
	} else {
		write_err = copy_fd(params->fd, child_process.in);
	}

~If there's a buffer, write from that, otherwise the data must be coming
from an open fd.~

Perfectly reasonable logic in a routine that's going to write from
either a buffer or an fd.

So change `index_core` to supply an empty buffer when indexing an empty
file.

There's a patch out there that instead changes the logic quoted above to
take a `-1` fd to mean "use the buffer", but it seems to me that the
distinction between a missing buffer and an empty one carries intrinsic
semantics, where the logic change is adapting the code to handle
incorrect arguments.

Signed-off-by: Jim Hill <gjthill@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2015-05-18 10:15:20 -07:00
Steffen Prohaska
9035d75a2b convert: stream from fd to required clean filter to reduce used address space
The data is streamed to the filter process anyway.  Better avoid mapping
the file if possible.  This is especially useful if a clean filter
reduces the size, for example if it computes a sha1 for binary data,
like git media.  The file size that the previous implementation could
handle was limited by the available address space; large files for
example could not be handled with (32-bit) msysgit.  The new
implementation can filter files of any size as long as the filter output
is small enough.

The new code path is only taken if the filter is required.  The filter
consumes data directly from the fd.  If it fails, the original data is
not immediately available.  The condition can easily be handled as
a fatal error, which is expected for a required filter anyway.

If the filter was not required, the condition would need to be handled
in a different way, like seeking to 0 and reading the data.  But this
would require more restructuring of the code and is probably not worth
it.  The obvious approach of falling back to reading all data would not
help achieving the main purpose of this patch, which is to handle large
files with limited address space.  If reading all data is an option, we
can simply take the old code path right away and mmap the entire file.

The environment variable GIT_MMAP_LIMIT, which has been introduced in
a previous commit is used to test that the expected code path is taken.
A related test that exercises required filters is modified to verify
that the data actually has been modified on its way from the file system
to the object store.

Signed-off-by: Steffen Prohaska <prohaska@zib.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-08-28 10:25:15 -07:00
Junio C Hamano
6219bb22ba test: turn EXPENSIVE into a lazy prerequisite
Two test scripts (t0021 and t5551) had copy & paste code to set
EXPENSIVE prerequisite.  Use the test_lazy_prereq helper to define
them in the common t/test-lib.sh.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-06-09 14:18:55 -07:00
Steffen Prohaska
0b6806b9e4 xread, xwrite: limit size of IO to 8MB
Checking out 2GB or more through an external filter (see test) fails
on Mac OS X 10.8.4 (12E55) for a 64-bit executable with:

    error: read from external filter cat failed
    error: cannot feed the input to external filter cat
    error: cat died of signal 13
    error: external filter cat failed 141
    error: external filter cat failed

The reason is that read() immediately returns with EINVAL when asked
to read more than 2GB.  According to POSIX [1], if the value of
nbyte passed to read() is greater than SSIZE_MAX, the result is
implementation-defined.  The write function has the same restriction
[2].  Since OS X still supports running 32-bit executables, the
32-bit limit (SSIZE_MAX = INT_MAX = 2GB - 1) seems to be also
imposed on 64-bit executables under certain conditions.  For write,
the problem has been addressed earlier [6c642a].

Address the problem for read() and write() differently, by limiting
size of IO chunks unconditionally on all platforms in xread() and
xwrite().  Large chunks only cause problems, like causing latencies
when killing the process, even if OS X was not buggy.  Doing IO in
reasonably sized smaller chunks should have no negative impact on
performance.

The compat wrapper clipped_write() introduced earlier [6c642a] is
not needed anymore.  It will be reverted in a separate commit.  The
new test catches read and write problems.

Note that 'git add' exits with 0 even if it prints filtering errors
to stderr.  The test, therefore, checks stderr.  'git add' should
probably be changed (sometime in another commit) to exit with
nonzero if filtering fails.  The test could then be changed to use
test_must_fail.

Thanks to the following people for suggestions and testing:

    Johannes Sixt <j6t@kdbg.org>
    John Keeping <john@keeping.me.uk>
    Jonathan Nieder <jrnieder@gmail.com>
    Kyle J. McKay <mackyle@gmail.com>
    Linus Torvalds <torvalds@linux-foundation.org>
    Torsten Bögershausen <tboegi@web.de>

[1] http://pubs.opengroup.org/onlinepubs/009695399/functions/read.html
[2] http://pubs.opengroup.org/onlinepubs/009695399/functions/write.html

[6c642a] commit 6c642a8786
    compate/clipped-write.c: large write(2) fails on Mac OS X/XNU

Signed-off-by: Steffen Prohaska <prohaska@zib.de>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-08-20 11:10:59 -07:00
Jehan Bing
36daaaca00 Add a setting to require a filter to be successful
By default, a missing filter driver or a failure from the filter driver is
not an error, but merely makes the filter operation a no-op pass through.
This is useful to massage the content into a shape that is more convenient
for the platform, filesystem, and the user to use, and the content filter
mechanism is not used to turn something unusable into usable.

However, we could also use of the content filtering mechanism and store
the content that cannot be directly used in the repository (e.g. a UUID
that refers to the true content stored outside git, or an encrypted
content) and turn it into a usable form upon checkout (e.g. download the
external content, or decrypt the encrypted content).  For such a use case,
the content cannot be used when filter driver fails, and we need a way to
tell Git to abort the whole operation for such a failing or missing filter
driver.

Add a new "filter.<driver>.required" configuration variable to mark the
second use case.  When it is set, git will abort the operation when the
filter driver does not exist or exits with a non-zero status code.

Signed-off-by: Jehan Bing <jehan@orb.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2012-02-17 07:37:08 -08:00
Junio C Hamano
6b6cab3f9a t0021: test application of both crlf and ident
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-05-26 16:47:16 -07:00
René Scharfe
dd555d8bed t0021-conversion.sh: fix NoTerminatingSymbolAtEOF test
The last line of the test file "expanded-keywords" ended in a newline,
which is a valid terminator for ident.  Use printf instead of echo to omit
it and thus really test if a file that ends unexpectedly in the middle of
an ident tag is handled properly.

Also take the oppertunity to calculate the expected ID dynamically
instead of hardcoding it into the test script.  This should make future
changes easier.

Signed-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2011-05-26 16:47:15 -07:00
Junio C Hamano
4290f69067 t0021: avoid getting filter killed with SIGPIPE
The fake filter did not read from the standard input at all,
which caused the calling side to die with SIGPIPE, depending
on the timing.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-12-22 15:18:47 -08:00
Pete Wyckoff
a2b665de4b convert filter: supply path to external driver
Filtering to support keyword expansion may need the name of
the file being filtered.  In particular, to support p4 keywords
like

    $File: //depot/product/dir/script.sh $

the smudge filter needs to know the name of the file it is
smudging.

Allow "%f" in the custom filter command line specified in the
configuration.  This will be substituted by the filename
inside a single-quote pair to be passed to the shell.

Signed-off-by: Pete Wyckoff <pw@padd.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-12-22 10:19:32 -08:00
Henrik Grubbström
07814d9009 convert: Keep foreign $Id$ on checkout.
If there are foreign $Id$ keywords in the repository, they are most
likely there for a reason. Let's keep them on checkout (which is also
what the documentation indicates). Foreign $Id$ keywords are now
recognized by there being multiple space separated fields in $Id:xxxxx$.

Signed-off-by: Henrik Grubbström <grubba@grubba.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-04-10 21:45:01 -07:00
Henrik Grubbström
a9f3049f6c convert: Safer handling of $Id$ contraction.
The code to contract $Id:xxxxx$ strings could eat an arbitrary amount
of source text if the terminating $ was lost. It now refuses to
contract $Id:xxxxx$ strings spanning multiple lines.

Signed-off-by: Henrik Grubbström <grubba@grubba.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-04-10 21:45:00 -07:00
Johannes Sixt
fa7151a61e t0021: use $SHELL_PATH for the filter script
On Windows, we need the shbang line to correctly invoke shell scripts via
a POSIX shell, except when the script is invoked via 'sh -c' because sh (a
bash) does "the right thing".  But the clean and smudge filters will not
always be invoked via 'sh -c'; to futureproof, we should mark the the one
in t0021-conversion with #!$SHELL_PATH.

Signed-off-by: Johannes Sixt <j6t@kdbg.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2010-01-05 23:40:43 -08:00
Jeff King
7339eb0823 t0021: tr portability fix for Solaris
Solaris' /usr/bin/tr doesn't seem to like multiple character
ranges in brackets (it simply prints "Bad string").

Instead, let's just enumerate the transformation we want.

Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2008-03-11 21:14:59 -07:00
Johannes Sixt
a0ae35ae2d t0021-conversion.sh: Test that the clean filter really cleans content.
This test uses a rot13 filter, which is its own inverse. It tested only
that the content was the same as the original after both the 'clean' and
the 'smudge' filter were applied. This way it would not detect whether
any filter was run at all. Hence, here we add another test that checks
that the repository contained content that was processed by the 'clean'
filter.

Signed-off-by: Johannes Sixt <johannes.sixt@telecom.at>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
2007-10-21 01:30:42 -04:00
Andy Parkins
dfab71cb92 Add test case for $Id$ expanded in the repository
This test case would have caught the bug fixed by revision
c23290d5.

It puts various forms of $Id$ into a file in the repository,
without allowing git to collapse them to uniformity.  Then enables the
$Id$ expansion on checkout, and checks that what is checked out has
coped with the various forms.

Signed-off-by: Andy Parkins <andyparkins@gmail.com>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-05-28 23:54:35 -07:00
Andy Parkins
af9b54bb2c Use $Id$ as the ident attribute keyword rather than $ident$ to be consistent with other VCSs
$Id$ is present already in SVN and CVS; it would mean that people
converting their existing repositories won't have to make any changes to
the source files should they want to make use of the ident attribute.

Given that it's a feature that's meant to calm those very people, it
seems obtuse to make them edit every file just to make use of it.

I think that bzr uses $Id$; Mercurial has examples hooks for $Id$;
monotone has $Id$ on its wishlist.  I can't think of a good reason not
to stick with the de-facto standard and call ours $Id$ instead of
$ident$.

Signed-off-by: Andy Parkins <andyparkins@gmail.com>
Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-05-14 19:03:32 -07:00
Junio C Hamano
aa4ed402c9 Add 'filter' attribute and external filter driver definition.
The interface is similar to the custom low-level merge drivers.

First you configure your filter driver by defining 'filter.<name>.*'
variables in the configuration.

	filter.<name>.clean	filter command to run upon checkin
	filter.<name>.smudge	filter command to run upon checkout

Then you assign filter attribute to each path, whose name
matches the custom filter driver's name.

Example:

	(in .gitattributes)
	*.c	filter=indent

	(in config)
	[filter "indent"]
		clean = indent
		smudge = cat

Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-04-24 22:38:51 -07:00
Junio C Hamano
3fed15f568 Add 'ident' conversion.
The 'ident' attribute set to path squashes "$ident:<any bytes
except dollor sign>$" to "$ident$" upon checkin, and expands it
to "$ident: <blob SHA-1> $" upon checkout.

As we have two conversions that affect checkin/checkout paths,
clarify how they interact with each other.

Signed-off-by: Junio C Hamano <junkio@cox.net>
2007-04-24 22:38:51 -07:00