2005-05-10 21:32:30 +00:00
|
|
|
git-cat-file(1)
|
|
|
|
|
===============
|
|
|
|
|
|
|
|
|
|
NAME
|
|
|
|
|
----
|
2009-03-23 01:00:15 +00:00
|
|
|
git-cat-file - Provide content or type and size information for repository objects
|
2005-05-10 21:32:30 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
SYNOPSIS
|
|
|
|
|
--------
|
2008-06-03 11:06:12 +00:00
|
|
|
[verse]
|
2021-12-28 13:28:44 +00:00
|
|
|
'git cat-file' <type> <object>
|
|
|
|
|
'git cat-file' (-e | -p) <object>
|
2022-01-10 22:08:45 +00:00
|
|
|
'git cat-file' (-t | -s) [--allow-unknown-type] <object>
|
2022-04-07 19:08:59 +00:00
|
|
|
'git cat-file' (--batch | --batch-check | --batch-command) [--batch-all-objects]
|
2021-12-28 13:28:44 +00:00
|
|
|
[--buffer] [--follow-symlinks] [--unordered]
|
cat-file: add option '-Z' that delimits input and output with NUL
In db9d67f2e9 (builtin/cat-file.c: support NUL-delimited input with
`-z`, 2022-07-22), we have introduced a new mode to read the input via
NUL-delimited records instead of newline-delimited records. This allows
the user to query for revisions that have newlines in their path
component. While unusual, such queries are perfectly valid and thus it
is clear that we should be able to support them properly.
Unfortunately, the commit only changed the input to be NUL-delimited,
but didn't change the output at the same time. While this is fine for
queries that are processed successfully, it is less so for queries that
aren't. In the case of missing commits for example the result can become
entirely unparsable:
```
$ printf "7ce4f05bae8120d9fa258e854a8669f6ea9cb7b1 blob 10\n1234567890\n\n\commit000" |
git cat-file --batch -z
7ce4f05bae8120d9fa258e854a8669f6ea9cb7b1 blob 10
1234567890
commit missing
```
This is of course a crafted query that is intentionally gaming the
deficiency, but more benign queries that contain newlines would have
similar problems.
Ideally, we should have also changed the output to be NUL-delimited when
`-z` is specified to avoid this problem. As the input is NUL-delimited,
it is clear that the output in this case cannot ever contain NUL
characters by itself. Furthermore, Git does not allow NUL characters in
revisions anyway, further stressing the point that using NUL-delimited
output is safe. The only exception is of course the object data itself,
but as git-cat-file(1) prints the size of the object data clients should
read until that specified size has been consumed.
But even though `-z` has only been introduced a few releases ago in Git
v2.38.0, changing the output format retroactively to also NUL-delimit
output would be a backwards incompatible change. And while one could
make the argument that the output is inherently broken already, we need
to assume that there are existing users out there that use it just fine
given that revisions containing newlines are quite exotic.
Instead, introduce a new option `-Z` that switches to NUL-delimited
input and output. While this new option could arguably only switch the
output format to be NUL-delimited, the consequence would be that users
have to always specify both `-z` and `-Z` when the input may contain
newlines. On the other hand, if the user knows that there never will be
newlines in the input, they don't have to use either of those options.
There is thus no usecase that would warrant treating input and output
format separately, which is why we instead opt to "do the right thing"
and have `-Z` mean to NUL-terminate both formats.
The old `-z` option is marked as deprecated with a hint that its output
may become unparsable. It is thus hidden both from the synopsis as well
as the command's help output.
Co-authored-by: Toon Claes <toon@iotcl.com>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-06-06 05:19:45 +00:00
|
|
|
[--textconv | --filters] [-Z]
|
2022-01-10 22:08:45 +00:00
|
|
|
'git cat-file' (--textconv | --filters)
|
2021-12-28 13:28:44 +00:00
|
|
|
[<rev>:<path|tree-ish> | --path=<path|tree-ish> <rev>]
|
2005-05-10 21:32:30 +00:00
|
|
|
|
|
|
|
|
DESCRIPTION
|
|
|
|
|
-----------
|
2009-03-23 01:00:15 +00:00
|
|
|
In its first form, the command provides the content or the type of an object in
|
2016-06-28 11:40:10 +00:00
|
|
|
the repository. The type is required unless `-t` or `-p` is used to find the
|
2016-08-24 12:23:39 +00:00
|
|
|
object type, or `-s` is used to find the object size, or `--textconv` or
|
|
|
|
|
`--filters` is used (which imply type "blob").
|
2008-04-23 19:17:46 +00:00
|
|
|
|
2009-03-23 01:00:15 +00:00
|
|
|
In the second form, a list of objects (separated by linefeeds) is provided on
|
2016-09-09 10:10:54 +00:00
|
|
|
stdin, and the SHA-1, type, and size of each object is printed on stdout. The
|
|
|
|
|
output format can be overridden using the optional `<format>` argument. If
|
|
|
|
|
either `--textconv` or `--filters` was specified, the input is expected to
|
2018-11-23 11:16:51 +00:00
|
|
|
list the object names followed by the path name, separated by a single
|
|
|
|
|
whitespace, so that the appropriate drivers can be determined.
|
2005-05-10 21:32:30 +00:00
|
|
|
|
|
|
|
|
OPTIONS
|
|
|
|
|
-------
|
|
|
|
|
<object>::
|
2007-01-30 07:56:51 +00:00
|
|
|
The name of the object to show.
|
|
|
|
|
For a more complete list of ways to spell object names, see
|
2010-10-11 16:03:32 +00:00
|
|
|
the "SPECIFYING REVISIONS" section in linkgit:gitrevisions[7].
|
2005-05-10 21:32:30 +00:00
|
|
|
|
|
|
|
|
-t::
|
|
|
|
|
Instead of the content, show the object type identified by
|
2021-03-03 09:21:55 +00:00
|
|
|
`<object>`.
|
2005-05-10 21:32:30 +00:00
|
|
|
|
2005-06-28 06:59:18 +00:00
|
|
|
-s::
|
|
|
|
|
Instead of the content, show the object size identified by
|
2022-12-20 06:01:12 +00:00
|
|
|
`<object>`. If used with `--use-mailmap` option, will show
|
|
|
|
|
the size of updated object after replacing idents using the
|
|
|
|
|
mailmap mechanism.
|
2005-06-28 06:59:18 +00:00
|
|
|
|
2005-12-04 01:57:48 +00:00
|
|
|
-e::
|
2021-03-03 09:21:55 +00:00
|
|
|
Exit with zero status if `<object>` exists and is a valid
|
|
|
|
|
object. If `<object>` is of an invalid format exit with non-zero and
|
2018-01-10 12:55:53 +00:00
|
|
|
emits an error on stderr.
|
2005-12-04 01:57:48 +00:00
|
|
|
|
2006-05-25 01:22:32 +00:00
|
|
|
-p::
|
2021-03-03 09:21:55 +00:00
|
|
|
Pretty-print the contents of `<object>` based on its type.
|
2006-05-25 01:22:32 +00:00
|
|
|
|
2005-05-10 21:32:30 +00:00
|
|
|
<type>::
|
2021-03-03 09:21:55 +00:00
|
|
|
Typically this matches the real type of `<object>` but asking
|
2005-10-05 22:08:26 +00:00
|
|
|
for a type that can trivially be dereferenced from the given
|
2021-03-03 09:21:55 +00:00
|
|
|
`<object>` is also permitted. An example is to ask for a
|
|
|
|
|
"tree" with `<object>` being a commit object that contains it,
|
|
|
|
|
or to ask for a "blob" with `<object>` being a tag object that
|
2005-05-10 21:32:30 +00:00
|
|
|
points at it.
|
|
|
|
|
|
2022-07-18 19:51:02 +00:00
|
|
|
--[no-]mailmap::
|
|
|
|
|
--[no-]use-mailmap::
|
|
|
|
|
Use mailmap file to map author, committer and tagger names
|
|
|
|
|
and email addresses to canonical real names and email addresses.
|
|
|
|
|
See linkgit:git-shortlog[1].
|
|
|
|
|
|
2010-06-24 12:56:55 +00:00
|
|
|
--textconv::
|
|
|
|
|
Show the content as transformed by a textconv filter. In this case,
|
2021-03-03 09:21:55 +00:00
|
|
|
`<object>` has to be of the form `<tree-ish>:<path>`, or `:<path>` in
|
2016-08-24 12:23:36 +00:00
|
|
|
order to apply the filter to the content recorded in the index at
|
2021-03-03 09:21:55 +00:00
|
|
|
`<path>`.
|
2010-06-24 12:56:55 +00:00
|
|
|
|
2016-08-24 12:23:39 +00:00
|
|
|
--filters::
|
|
|
|
|
Show the content as converted by the filters configured in
|
2021-03-03 09:21:55 +00:00
|
|
|
the current working tree for the given `<path>` (i.e. smudge filters,
|
|
|
|
|
end-of-line conversion, etc). In this case, `<object>` has to be of
|
|
|
|
|
the form `<tree-ish>:<path>`, or `:<path>`.
|
2016-08-24 12:23:39 +00:00
|
|
|
|
2016-09-09 10:10:50 +00:00
|
|
|
--path=<path>::
|
2021-03-03 09:21:55 +00:00
|
|
|
For use with `--textconv` or `--filters`, to allow specifying an object
|
2016-09-09 10:10:50 +00:00
|
|
|
name and a path separately, e.g. when it is difficult to figure out
|
|
|
|
|
the revision from which the blob came.
|
|
|
|
|
|
2008-04-23 19:17:47 +00:00
|
|
|
--batch::
|
2013-07-10 11:45:47 +00:00
|
|
|
--batch=<format>::
|
|
|
|
|
Print object information and contents for each object provided
|
2022-12-20 06:01:13 +00:00
|
|
|
on stdin. May not be combined with any other options or arguments
|
|
|
|
|
except `--textconv`, `--filters`, or `--use-mailmap`.
|
git-cat-file.txt: fix list continuations rendering literally
With Asciidoctor, all of the '+' introduced in a797c0ea04 ("cat-file:
add mailmap support to --batch-check option", 2022-12-20) render
literally rather than functioning as list continuations. With asciidoc,
this renders just fine. It's not too surprising that there is room for
ambiguity and surprises here, since we have lists within lists.
Simply replacing all of these '+' with empty lines makes this render
fine using both tools. Except, in the third hunk, where after this inner
'*' list ends, we want to continue with more contents of the outer list
item (`--batch-command=<format>`). We can solve any ambiguity here and
make this clear to both tools by wrapping the inner list in an open
block (using "--").
For consistency, let's wrap all three of these inner lists from
a797c0ea04 in open blocks. This also future-proofs us a little -- if we
ever gain more contents after any of those first two lists, as we did
already in a797c0ea04 for the third list, we're prepared and should
render fine with both asciidoc and Asciidoctor from the start.
Signed-off-by: Martin Ågren <martin.agren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-18 08:27:49 +00:00
|
|
|
+
|
|
|
|
|
--
|
2022-12-20 06:01:13 +00:00
|
|
|
* When used with `--textconv` or `--filters`, the input lines
|
|
|
|
|
must specify the path, separated by whitespace. See the section
|
|
|
|
|
`BATCH OUTPUT` below for details.
|
git-cat-file.txt: fix list continuations rendering literally
With Asciidoctor, all of the '+' introduced in a797c0ea04 ("cat-file:
add mailmap support to --batch-check option", 2022-12-20) render
literally rather than functioning as list continuations. With asciidoc,
this renders just fine. It's not too surprising that there is room for
ambiguity and surprises here, since we have lists within lists.
Simply replacing all of these '+' with empty lines makes this render
fine using both tools. Except, in the third hunk, where after this inner
'*' list ends, we want to continue with more contents of the outer list
item (`--batch-command=<format>`). We can solve any ambiguity here and
make this clear to both tools by wrapping the inner list in an open
block (using "--").
For consistency, let's wrap all three of these inner lists from
a797c0ea04 in open blocks. This also future-proofs us a little -- if we
ever gain more contents after any of those first two lists, as we did
already in a797c0ea04 for the third list, we're prepared and should
render fine with both asciidoc and Asciidoctor from the start.
Signed-off-by: Martin Ågren <martin.agren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-18 08:27:49 +00:00
|
|
|
|
2022-12-20 06:01:13 +00:00
|
|
|
* When used with `--use-mailmap`, for commit and tag objects, the
|
|
|
|
|
contents part of the output shows the identities replaced using the
|
|
|
|
|
mailmap mechanism, while the information part of the output shows
|
|
|
|
|
the size of the object as if it actually recorded the replacement
|
|
|
|
|
identities.
|
git-cat-file.txt: fix list continuations rendering literally
With Asciidoctor, all of the '+' introduced in a797c0ea04 ("cat-file:
add mailmap support to --batch-check option", 2022-12-20) render
literally rather than functioning as list continuations. With asciidoc,
this renders just fine. It's not too surprising that there is room for
ambiguity and surprises here, since we have lists within lists.
Simply replacing all of these '+' with empty lines makes this render
fine using both tools. Except, in the third hunk, where after this inner
'*' list ends, we want to continue with more contents of the outer list
item (`--batch-command=<format>`). We can solve any ambiguity here and
make this clear to both tools by wrapping the inner list in an open
block (using "--").
For consistency, let's wrap all three of these inner lists from
a797c0ea04 in open blocks. This also future-proofs us a little -- if we
ever gain more contents after any of those first two lists, as we did
already in a797c0ea04 for the third list, we're prepared and should
render fine with both asciidoc and Asciidoctor from the start.
Signed-off-by: Martin Ågren <martin.agren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-18 08:27:49 +00:00
|
|
|
--
|
2008-04-23 19:17:47 +00:00
|
|
|
|
2008-04-23 19:17:46 +00:00
|
|
|
--batch-check::
|
2013-07-10 11:45:47 +00:00
|
|
|
--batch-check=<format>::
|
2022-12-20 06:01:13 +00:00
|
|
|
Print object information for each object provided on stdin. May not be
|
|
|
|
|
combined with any other options or arguments except `--textconv`, `--filters`
|
|
|
|
|
or `--use-mailmap`.
|
git-cat-file.txt: fix list continuations rendering literally
With Asciidoctor, all of the '+' introduced in a797c0ea04 ("cat-file:
add mailmap support to --batch-check option", 2022-12-20) render
literally rather than functioning as list continuations. With asciidoc,
this renders just fine. It's not too surprising that there is room for
ambiguity and surprises here, since we have lists within lists.
Simply replacing all of these '+' with empty lines makes this render
fine using both tools. Except, in the third hunk, where after this inner
'*' list ends, we want to continue with more contents of the outer list
item (`--batch-command=<format>`). We can solve any ambiguity here and
make this clear to both tools by wrapping the inner list in an open
block (using "--").
For consistency, let's wrap all three of these inner lists from
a797c0ea04 in open blocks. This also future-proofs us a little -- if we
ever gain more contents after any of those first two lists, as we did
already in a797c0ea04 for the third list, we're prepared and should
render fine with both asciidoc and Asciidoctor from the start.
Signed-off-by: Martin Ågren <martin.agren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-18 08:27:49 +00:00
|
|
|
+
|
|
|
|
|
--
|
2022-12-20 06:01:13 +00:00
|
|
|
* When used with `--textconv` or `--filters`, the input lines must
|
|
|
|
|
specify the path, separated by whitespace. See the section
|
|
|
|
|
`BATCH OUTPUT` below for details.
|
git-cat-file.txt: fix list continuations rendering literally
With Asciidoctor, all of the '+' introduced in a797c0ea04 ("cat-file:
add mailmap support to --batch-check option", 2022-12-20) render
literally rather than functioning as list continuations. With asciidoc,
this renders just fine. It's not too surprising that there is room for
ambiguity and surprises here, since we have lists within lists.
Simply replacing all of these '+' with empty lines makes this render
fine using both tools. Except, in the third hunk, where after this inner
'*' list ends, we want to continue with more contents of the outer list
item (`--batch-command=<format>`). We can solve any ambiguity here and
make this clear to both tools by wrapping the inner list in an open
block (using "--").
For consistency, let's wrap all three of these inner lists from
a797c0ea04 in open blocks. This also future-proofs us a little -- if we
ever gain more contents after any of those first two lists, as we did
already in a797c0ea04 for the third list, we're prepared and should
render fine with both asciidoc and Asciidoctor from the start.
Signed-off-by: Martin Ågren <martin.agren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-18 08:27:49 +00:00
|
|
|
|
2022-12-20 06:01:13 +00:00
|
|
|
* When used with `--use-mailmap`, for commit and tag objects, the
|
|
|
|
|
printed object information shows the size of the object as if the
|
|
|
|
|
identities recorded in it were replaced by the mailmap mechanism.
|
git-cat-file.txt: fix list continuations rendering literally
With Asciidoctor, all of the '+' introduced in a797c0ea04 ("cat-file:
add mailmap support to --batch-check option", 2022-12-20) render
literally rather than functioning as list continuations. With asciidoc,
this renders just fine. It's not too surprising that there is room for
ambiguity and surprises here, since we have lists within lists.
Simply replacing all of these '+' with empty lines makes this render
fine using both tools. Except, in the third hunk, where after this inner
'*' list ends, we want to continue with more contents of the outer list
item (`--batch-command=<format>`). We can solve any ambiguity here and
make this clear to both tools by wrapping the inner list in an open
block (using "--").
For consistency, let's wrap all three of these inner lists from
a797c0ea04 in open blocks. This also future-proofs us a little -- if we
ever gain more contents after any of those first two lists, as we did
already in a797c0ea04 for the third list, we're prepared and should
render fine with both asciidoc and Asciidoctor from the start.
Signed-off-by: Martin Ågren <martin.agren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-18 08:27:49 +00:00
|
|
|
--
|
2008-04-23 19:17:46 +00:00
|
|
|
|
cat-file: add --batch-command mode
Add a new flag --batch-command that accepts commands and arguments
from stdin, similar to git-update-ref --stdin.
At GitLab, we use a pair of long running cat-file processes when
accessing object content. One for iterating over object metadata with
--batch-check, and the other to grab object contents with --batch.
However, if we had --batch-command, we wouldn't need to keep both
processes around, and instead just have one --batch-command process
where we can flip between getting object info, and getting object
contents. Since we have a pair of cat-file processes per repository,
this means we can get rid of roughly half of long lived git cat-file
processes. Given there are many repositories being accessed at any given
time, this can lead to huge savings.
git cat-file --batch-command
will enter an interactive command mode whereby the user can enter in
commands and their arguments that get queued in memory:
<command1> [arg1] [arg2] LF
<command2> [arg1] [arg2] LF
When --buffer mode is used, commands will be queued in memory until a
flush command is issued that execute them:
flush LF
The reason for a flush command is that when a consumer process (A)
talks to a git cat-file process (B) and interactively writes to and
reads from it in --buffer mode, (A) needs to be able to control when
the buffer is flushed to stdout.
Currently, from (A)'s perspective, the only way is to either
1. kill (B)'s process
2. send an invalid object to stdin.
1. is not ideal from a performance perspective as it will require
spawning a new cat-file process each time, and 2. is hacky and not a
good long term solution.
With this mechanism of queueing up commands and letting (A) issue a
flush command, process (A) can control when the buffer is flushed and
can guarantee it will receive all of the output when in --buffer mode.
--batch-command also will not allow (B) to flush to stdout until a flush
is received.
This patch adds the basic structure for adding command which can be
extended in the future to add more commands. It also adds the following
two commands (on top of the flush command):
contents <object> LF
info <object> LF
The contents command takes an <object> argument and prints out the object
contents.
The info command takes an <object> argument and prints out the object
metadata.
These can be used in the following way with --buffer:
info <object> LF
contents <object> LF
contents <object> LF
info <object> LF
flush LF
info <object> LF
flush LF
When used without --buffer:
info <object> LF
contents <object> LF
contents <object> LF
info <object> LF
info <object> LF
Helped-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: John Cai <johncai86@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-02-18 18:23:14 +00:00
|
|
|
--batch-command::
|
|
|
|
|
--batch-command=<format>::
|
|
|
|
|
Enter a command mode that reads commands and arguments from stdin. May
|
2022-12-20 06:01:13 +00:00
|
|
|
only be combined with `--buffer`, `--textconv`, `--use-mailmap` or
|
|
|
|
|
`--filters`.
|
git-cat-file.txt: fix list continuations rendering literally
With Asciidoctor, all of the '+' introduced in a797c0ea04 ("cat-file:
add mailmap support to --batch-check option", 2022-12-20) render
literally rather than functioning as list continuations. With asciidoc,
this renders just fine. It's not too surprising that there is room for
ambiguity and surprises here, since we have lists within lists.
Simply replacing all of these '+' with empty lines makes this render
fine using both tools. Except, in the third hunk, where after this inner
'*' list ends, we want to continue with more contents of the outer list
item (`--batch-command=<format>`). We can solve any ambiguity here and
make this clear to both tools by wrapping the inner list in an open
block (using "--").
For consistency, let's wrap all three of these inner lists from
a797c0ea04 in open blocks. This also future-proofs us a little -- if we
ever gain more contents after any of those first two lists, as we did
already in a797c0ea04 for the third list, we're prepared and should
render fine with both asciidoc and Asciidoctor from the start.
Signed-off-by: Martin Ågren <martin.agren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-18 08:27:49 +00:00
|
|
|
+
|
|
|
|
|
--
|
2022-12-20 06:01:13 +00:00
|
|
|
* When used with `--textconv` or `--filters`, the input lines must
|
|
|
|
|
specify the path, separated by whitespace. See the section
|
|
|
|
|
`BATCH OUTPUT` below for details.
|
git-cat-file.txt: fix list continuations rendering literally
With Asciidoctor, all of the '+' introduced in a797c0ea04 ("cat-file:
add mailmap support to --batch-check option", 2022-12-20) render
literally rather than functioning as list continuations. With asciidoc,
this renders just fine. It's not too surprising that there is room for
ambiguity and surprises here, since we have lists within lists.
Simply replacing all of these '+' with empty lines makes this render
fine using both tools. Except, in the third hunk, where after this inner
'*' list ends, we want to continue with more contents of the outer list
item (`--batch-command=<format>`). We can solve any ambiguity here and
make this clear to both tools by wrapping the inner list in an open
block (using "--").
For consistency, let's wrap all three of these inner lists from
a797c0ea04 in open blocks. This also future-proofs us a little -- if we
ever gain more contents after any of those first two lists, as we did
already in a797c0ea04 for the third list, we're prepared and should
render fine with both asciidoc and Asciidoctor from the start.
Signed-off-by: Martin Ågren <martin.agren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-18 08:27:49 +00:00
|
|
|
|
2022-12-20 06:01:13 +00:00
|
|
|
* When used with `--use-mailmap`, for commit and tag objects, the
|
|
|
|
|
`contents` command shows the identities replaced using the
|
|
|
|
|
mailmap mechanism, while the `info` command shows the size
|
|
|
|
|
of the object as if it actually recorded the replacement
|
|
|
|
|
identities.
|
git-cat-file.txt: fix list continuations rendering literally
With Asciidoctor, all of the '+' introduced in a797c0ea04 ("cat-file:
add mailmap support to --batch-check option", 2022-12-20) render
literally rather than functioning as list continuations. With asciidoc,
this renders just fine. It's not too surprising that there is room for
ambiguity and surprises here, since we have lists within lists.
Simply replacing all of these '+' with empty lines makes this render
fine using both tools. Except, in the third hunk, where after this inner
'*' list ends, we want to continue with more contents of the outer list
item (`--batch-command=<format>`). We can solve any ambiguity here and
make this clear to both tools by wrapping the inner list in an open
block (using "--").
For consistency, let's wrap all three of these inner lists from
a797c0ea04 in open blocks. This also future-proofs us a little -- if we
ever gain more contents after any of those first two lists, as we did
already in a797c0ea04 for the third list, we're prepared and should
render fine with both asciidoc and Asciidoctor from the start.
Signed-off-by: Martin Ågren <martin.agren@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-18 08:27:49 +00:00
|
|
|
--
|
cat-file: add --batch-command mode
Add a new flag --batch-command that accepts commands and arguments
from stdin, similar to git-update-ref --stdin.
At GitLab, we use a pair of long running cat-file processes when
accessing object content. One for iterating over object metadata with
--batch-check, and the other to grab object contents with --batch.
However, if we had --batch-command, we wouldn't need to keep both
processes around, and instead just have one --batch-command process
where we can flip between getting object info, and getting object
contents. Since we have a pair of cat-file processes per repository,
this means we can get rid of roughly half of long lived git cat-file
processes. Given there are many repositories being accessed at any given
time, this can lead to huge savings.
git cat-file --batch-command
will enter an interactive command mode whereby the user can enter in
commands and their arguments that get queued in memory:
<command1> [arg1] [arg2] LF
<command2> [arg1] [arg2] LF
When --buffer mode is used, commands will be queued in memory until a
flush command is issued that execute them:
flush LF
The reason for a flush command is that when a consumer process (A)
talks to a git cat-file process (B) and interactively writes to and
reads from it in --buffer mode, (A) needs to be able to control when
the buffer is flushed to stdout.
Currently, from (A)'s perspective, the only way is to either
1. kill (B)'s process
2. send an invalid object to stdin.
1. is not ideal from a performance perspective as it will require
spawning a new cat-file process each time, and 2. is hacky and not a
good long term solution.
With this mechanism of queueing up commands and letting (A) issue a
flush command, process (A) can control when the buffer is flushed and
can guarantee it will receive all of the output when in --buffer mode.
--batch-command also will not allow (B) to flush to stdout until a flush
is received.
This patch adds the basic structure for adding command which can be
extended in the future to add more commands. It also adds the following
two commands (on top of the flush command):
contents <object> LF
info <object> LF
The contents command takes an <object> argument and prints out the object
contents.
The info command takes an <object> argument and prints out the object
metadata.
These can be used in the following way with --buffer:
info <object> LF
contents <object> LF
contents <object> LF
info <object> LF
flush LF
info <object> LF
flush LF
When used without --buffer:
info <object> LF
contents <object> LF
contents <object> LF
info <object> LF
info <object> LF
Helped-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: John Cai <johncai86@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-02-18 18:23:14 +00:00
|
|
|
+
|
|
|
|
|
`--batch-command` recognizes the following commands:
|
|
|
|
|
+
|
|
|
|
|
--
|
|
|
|
|
contents <object>::
|
|
|
|
|
Print object contents for object reference `<object>`. This corresponds to
|
|
|
|
|
the output of `--batch`.
|
|
|
|
|
|
|
|
|
|
info <object>::
|
|
|
|
|
Print object info for object reference `<object>`. This corresponds to the
|
|
|
|
|
output of `--batch-check`.
|
|
|
|
|
|
|
|
|
|
flush::
|
|
|
|
|
Used with `--buffer` to execute all preceding commands that were issued
|
|
|
|
|
since the beginning or since the last flush was issued. When `--buffer`
|
|
|
|
|
is used, no output will come until a `flush` is issued. When `--buffer`
|
|
|
|
|
is not used, commands are flushed each time without issuing `flush`.
|
|
|
|
|
--
|
|
|
|
|
+
|
|
|
|
|
|
2015-06-22 10:45:59 +00:00
|
|
|
--batch-all-objects::
|
|
|
|
|
Instead of reading a list of objects on stdin, perform the
|
|
|
|
|
requested batch operation on all objects in the repository and
|
|
|
|
|
any alternate object stores (not just reachable objects).
|
2021-10-05 20:31:08 +00:00
|
|
|
Requires `--batch` or `--batch-check` be specified. By default,
|
|
|
|
|
the objects are visited in order sorted by their hashes; see
|
cat-file: disable refs/replace with --batch-all-objects
When we're enumerating all objects in the object database, it doesn't
make sense to respect refs/replace. The point of this option is to
enumerate all of the objects in the database at a low level. By
definition we'd already show the replacement object's contents (under
its real oid), and showing those contents under another oid is almost
certainly working against what the user is trying to do.
Note that you could make the same argument for something like:
git show-index <foo.idx |
awk '{print $2}' |
git cat-file --batch
but there we can't know in cat-file exactly what the user intended,
because we don't know the source of the input. They could be trying to
do low-level debugging, or they could be doing something more high-level
(e.g., imagine a porcelain built around cat-file for its object
accesses). So in those cases, we'll have to rely on the user specifying
"git --no-replace-objects" to tell us what to do.
One _could_ make an argument that "cat-file --batch" is sufficiently
low-level plumbing that it should not respect replace-objects at all
(and the caller should do any replacement if they want it). But we have
been doing so for some time. The history is a little tangled:
- looking back as far as v1.6.6, we would not respect replace refs for
--batch-check, but would for --batch (because the former used
sha1_object_info(), and the replace mechanism only affected actual
object reads)
- this discrepancy was made even weirder by 98e2092b50 (cat-file:
teach --batch to stream blob objects, 2013-07-10), where we always
output the header using the --batch-check code, and then printed the
object separately. This could lead to "cat-file --batch" dying (when
it notices the size or type changed for a non-blob object) or even
producing bogus output (in streaming mode, we didn't notice that we
wrote the wrong number of bytes).
- that persisted until 1f7117ef7a (sha1_file: perform object
replacement in sha1_object_info_extended(), 2013-12-11), which then
respected replace refs for both forms.
So it has worked reliably this way for over 7 years, and we should make
sure it continues to do so. That could also be an argument that
--batch-all-objects should not change behavior (which this patch is
doing), but I really consider the current behavior to be an unintended
bug. It's a side effect of how the code is implemented (feeding the oids
back into oid_object_info() rather than looking at what we found while
reading the loose and packed object storage).
The implementation is straight-forward: we just disable the global
read_replace_refs flag when we're in --batch-all-objects mode. It would
perhaps be a little cleaner to change the flag we pass to
oid_object_info_extended(), but that's not enough. We also read objects
via read_object_file() and stream_blob_to_fd(). The former could switch
to its _extended() form, but the streaming code has no mechanism for
disabling replace refs. Setting the global flag works, and as a bonus,
it's impossible to have any "oops, we're sometimes replacing the object
and sometimes not" bugs in the output (like the ones caused by
98e2092b50 above).
The tests here cover the regular-input and --batch-all-objects cases,
for both --batch-check and --batch. There is a test in t6050 that covers
the regular-input case with --batch already, but this new one goes much
further in actually verifying the output (plus covering --batch-check
explicitly). This is perhaps a little overkill and the tests would be
simpler just covering --batch-check, but I wanted to make sure we're
checking that --batch output is consistent between the header and the
content. The global-flag technique used here makes that easy to get
right, but this is future-proofing us against regressions.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-10-05 20:36:07 +00:00
|
|
|
also `--unordered` below. Objects are presented as-is, without
|
|
|
|
|
respecting the "replace" mechanism of linkgit:git-replace[1].
|
2015-06-22 10:45:59 +00:00
|
|
|
|
2015-06-22 10:45:17 +00:00
|
|
|
--buffer::
|
|
|
|
|
Normally batch output is flushed after each object is output, so
|
|
|
|
|
that a process can interactively read and write from
|
|
|
|
|
`cat-file`. With this option, the output uses normal stdio
|
|
|
|
|
buffering; this is much more efficient when invoking
|
cat-file: add --batch-command mode
Add a new flag --batch-command that accepts commands and arguments
from stdin, similar to git-update-ref --stdin.
At GitLab, we use a pair of long running cat-file processes when
accessing object content. One for iterating over object metadata with
--batch-check, and the other to grab object contents with --batch.
However, if we had --batch-command, we wouldn't need to keep both
processes around, and instead just have one --batch-command process
where we can flip between getting object info, and getting object
contents. Since we have a pair of cat-file processes per repository,
this means we can get rid of roughly half of long lived git cat-file
processes. Given there are many repositories being accessed at any given
time, this can lead to huge savings.
git cat-file --batch-command
will enter an interactive command mode whereby the user can enter in
commands and their arguments that get queued in memory:
<command1> [arg1] [arg2] LF
<command2> [arg1] [arg2] LF
When --buffer mode is used, commands will be queued in memory until a
flush command is issued that execute them:
flush LF
The reason for a flush command is that when a consumer process (A)
talks to a git cat-file process (B) and interactively writes to and
reads from it in --buffer mode, (A) needs to be able to control when
the buffer is flushed to stdout.
Currently, from (A)'s perspective, the only way is to either
1. kill (B)'s process
2. send an invalid object to stdin.
1. is not ideal from a performance perspective as it will require
spawning a new cat-file process each time, and 2. is hacky and not a
good long term solution.
With this mechanism of queueing up commands and letting (A) issue a
flush command, process (A) can control when the buffer is flushed and
can guarantee it will receive all of the output when in --buffer mode.
--batch-command also will not allow (B) to flush to stdout until a flush
is received.
This patch adds the basic structure for adding command which can be
extended in the future to add more commands. It also adds the following
two commands (on top of the flush command):
contents <object> LF
info <object> LF
The contents command takes an <object> argument and prints out the object
contents.
The info command takes an <object> argument and prints out the object
metadata.
These can be used in the following way with --buffer:
info <object> LF
contents <object> LF
contents <object> LF
info <object> LF
flush LF
info <object> LF
flush LF
When used without --buffer:
info <object> LF
contents <object> LF
contents <object> LF
info <object> LF
info <object> LF
Helped-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: John Cai <johncai86@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-02-18 18:23:14 +00:00
|
|
|
`--batch-check` or `--batch-command` on a large number of objects.
|
2015-06-22 10:45:17 +00:00
|
|
|
|
cat-file: support "unordered" output for --batch-all-objects
If you're going to access the contents of every object in a
packfile, it's generally much more efficient to do so in
pack order, rather than in hash order. That increases the
locality of access within the packfile, which in turn is
friendlier to the delta base cache, since the packfile puts
related deltas next to each other. By contrast, hash order
is effectively random, since the sha1 has no discernible
relationship to the content.
This patch introduces an "--unordered" option to cat-file
which iterates over packs in pack-order under the hood. You
can see the results when dumping all of the file content:
$ time ./git cat-file --batch-all-objects --buffer --batch | wc -c
6883195596
real 0m44.491s
user 0m42.902s
sys 0m5.230s
$ time ./git cat-file --unordered \
--batch-all-objects --buffer --batch | wc -c
6883195596
real 0m6.075s
user 0m4.774s
sys 0m3.548s
Same output, different order, way faster. The same speed-up
applies even if you end up accessing the object content in a
different process, like:
git cat-file --batch-all-objects --buffer --batch-check |
grep blob |
git cat-file --batch='%(objectname) %(rest)' |
wc -c
Adding "--unordered" to the first command drops the runtime
in git.git from 24s to 3.5s.
Side note: there are actually further speedups available
for doing it all in-process now. Since we are outputting
the object content during the actual pack iteration, we
know where to find the object and could skip the extra
lookup done by oid_object_info(). This patch stops short
of that optimization since the underlying API isn't ready
for us to make those sorts of direct requests.
So if --unordered is so much better, why not make it the
default? Two reasons:
1. We've promised in the documentation that --batch-all-objects
outputs in hash order. Since cat-file is plumbing,
people may be relying on that default, and we can't
change it.
2. It's actually _slower_ for some cases. We have to
compute the pack revindex to walk in pack order. And
our de-duplication step uses an oidset, rather than a
sort-and-dedup, which can end up being more expensive.
If we're just accessing the type and size of each
object, for example, like:
git cat-file --batch-all-objects --buffer --batch-check
my best-of-five warm cache timings go from 900ms to
1100ms using --unordered. Though it's possible in a
cold-cache or under memory pressure that we could do
better, since we'd have better locality within the
packfile.
And one final question: why is it "--unordered" and not
"--pack-order"? The answer is again two-fold:
1. "pack order" isn't a well-defined thing across the
whole set of objects. We're hitting loose objects, as
well as objects in multiple packs, and the only
ordering we're promising is _within_ a single pack. The
rest is apparently random.
2. The point here is optimization. So we don't want to
promise any particular ordering, but only to say that
we will choose an ordering which is likely to be
efficient for accessing the object content. That leaves
the door open for further changes in the future without
having to add another compatibility option.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-08-10 23:24:57 +00:00
|
|
|
--unordered::
|
|
|
|
|
When `--batch-all-objects` is in use, visit objects in an
|
|
|
|
|
order which may be more efficient for accessing the object
|
|
|
|
|
contents than hash order. The exact details of the order are
|
|
|
|
|
unspecified, but if you do not require a specific order, this
|
|
|
|
|
should generally result in faster output, especially with
|
|
|
|
|
`--batch`. Note that `cat-file` will still show each object
|
|
|
|
|
only once, even if it is stored multiple times in the
|
|
|
|
|
repository.
|
|
|
|
|
|
2015-05-03 14:30:01 +00:00
|
|
|
--allow-unknown-type::
|
2021-03-03 09:21:55 +00:00
|
|
|
Allow `-s` or `-t` to query broken/corrupt objects of unknown type.
|
2015-05-03 14:30:01 +00:00
|
|
|
|
2015-05-20 17:03:40 +00:00
|
|
|
--follow-symlinks::
|
2021-03-03 09:21:55 +00:00
|
|
|
With `--batch` or `--batch-check`, follow symlinks inside the
|
2015-05-20 17:03:40 +00:00
|
|
|
repository when requesting objects with extended SHA-1
|
|
|
|
|
expressions of the form tree-ish:path-in-tree. Instead of
|
|
|
|
|
providing output about the link itself, provide output about
|
|
|
|
|
the linked-to object. If a symlink points outside the
|
2021-03-03 09:21:55 +00:00
|
|
|
tree-ish (e.g. a link to `/foo` or a root-level link to `../foo`),
|
2015-05-20 17:03:40 +00:00
|
|
|
the portion of the link which is outside the tree will be
|
|
|
|
|
printed.
|
|
|
|
|
+
|
|
|
|
|
This option does not (currently) work correctly when an object in the
|
|
|
|
|
index is specified (e.g. `:link` instead of `HEAD:link`) rather than
|
|
|
|
|
one in the tree.
|
|
|
|
|
+
|
|
|
|
|
This option cannot (currently) be used unless `--batch` or
|
|
|
|
|
`--batch-check` is used.
|
|
|
|
|
+
|
|
|
|
|
For example, consider a git repository containing:
|
|
|
|
|
+
|
|
|
|
|
--
|
|
|
|
|
f: a file containing "hello\n"
|
|
|
|
|
link: a symlink to f
|
|
|
|
|
dir/link: a symlink to ../f
|
|
|
|
|
plink: a symlink to ../f
|
|
|
|
|
alink: a symlink to /etc/passwd
|
|
|
|
|
--
|
|
|
|
|
+
|
|
|
|
|
For a regular file `f`, `echo HEAD:f | git cat-file --batch` would print
|
|
|
|
|
+
|
|
|
|
|
--
|
|
|
|
|
ce013625030ba8dba906f756967f9e9ca394464a blob 6
|
|
|
|
|
--
|
|
|
|
|
+
|
|
|
|
|
And `echo HEAD:link | git cat-file --batch --follow-symlinks` would
|
|
|
|
|
print the same thing, as would `HEAD:dir/link`, as they both point at
|
|
|
|
|
`HEAD:f`.
|
|
|
|
|
+
|
|
|
|
|
Without `--follow-symlinks`, these would print data about the symlink
|
|
|
|
|
itself. In the case of `HEAD:link`, you would see
|
|
|
|
|
+
|
|
|
|
|
--
|
|
|
|
|
4d1ae35ba2c8ec712fa2a379db44ad639ca277bd blob 1
|
|
|
|
|
--
|
|
|
|
|
+
|
|
|
|
|
Both `plink` and `alink` point outside the tree, so they would
|
|
|
|
|
respectively print:
|
|
|
|
|
+
|
|
|
|
|
--
|
|
|
|
|
symlink 4
|
|
|
|
|
../f
|
|
|
|
|
|
|
|
|
|
symlink 11
|
|
|
|
|
/etc/passwd
|
|
|
|
|
--
|
|
|
|
|
|
cat-file: add option '-Z' that delimits input and output with NUL
In db9d67f2e9 (builtin/cat-file.c: support NUL-delimited input with
`-z`, 2022-07-22), we have introduced a new mode to read the input via
NUL-delimited records instead of newline-delimited records. This allows
the user to query for revisions that have newlines in their path
component. While unusual, such queries are perfectly valid and thus it
is clear that we should be able to support them properly.
Unfortunately, the commit only changed the input to be NUL-delimited,
but didn't change the output at the same time. While this is fine for
queries that are processed successfully, it is less so for queries that
aren't. In the case of missing commits for example the result can become
entirely unparsable:
```
$ printf "7ce4f05bae8120d9fa258e854a8669f6ea9cb7b1 blob 10\n1234567890\n\n\commit000" |
git cat-file --batch -z
7ce4f05bae8120d9fa258e854a8669f6ea9cb7b1 blob 10
1234567890
commit missing
```
This is of course a crafted query that is intentionally gaming the
deficiency, but more benign queries that contain newlines would have
similar problems.
Ideally, we should have also changed the output to be NUL-delimited when
`-z` is specified to avoid this problem. As the input is NUL-delimited,
it is clear that the output in this case cannot ever contain NUL
characters by itself. Furthermore, Git does not allow NUL characters in
revisions anyway, further stressing the point that using NUL-delimited
output is safe. The only exception is of course the object data itself,
but as git-cat-file(1) prints the size of the object data clients should
read until that specified size has been consumed.
But even though `-z` has only been introduced a few releases ago in Git
v2.38.0, changing the output format retroactively to also NUL-delimit
output would be a backwards incompatible change. And while one could
make the argument that the output is inherently broken already, we need
to assume that there are existing users out there that use it just fine
given that revisions containing newlines are quite exotic.
Instead, introduce a new option `-Z` that switches to NUL-delimited
input and output. While this new option could arguably only switch the
output format to be NUL-delimited, the consequence would be that users
have to always specify both `-z` and `-Z` when the input may contain
newlines. On the other hand, if the user knows that there never will be
newlines in the input, they don't have to use either of those options.
There is thus no usecase that would warrant treating input and output
format separately, which is why we instead opt to "do the right thing"
and have `-Z` mean to NUL-terminate both formats.
The old `-z` option is marked as deprecated with a hint that its output
may become unparsable. It is thus hidden both from the synopsis as well
as the command's help output.
Co-authored-by: Toon Claes <toon@iotcl.com>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-06-06 05:19:45 +00:00
|
|
|
-Z::
|
|
|
|
|
Only meaningful with `--batch`, `--batch-check`, or
|
|
|
|
|
`--batch-command`; input and output is NUL-delimited instead of
|
|
|
|
|
newline-delimited.
|
|
|
|
|
|
builtin/cat-file.c: support NUL-delimited input with `-z`
When callers are using `cat-file` via one of the stdin-driven `--batch`
modes, all input is newline-delimited. This presents a problem when
callers wish to ask about, e.g. tree-entries that have a newline
character present in their filename.
To support this niche scenario, introduce a new `-z` mode to the
`--batch`, `--batch-check`, and `--batch-command` suite of options that
instructs `cat-file` to treat its input as NUL-delimited, allowing the
individual commands themselves to have newlines present.
The refactoring here is slightly unfortunate, since we turn loops like:
while (strbuf_getline(&buf, stdin) != EOF)
into:
while (1) {
int ret;
if (opt->nul_terminated)
ret = strbuf_getline_nul(&input, stdin);
else
ret = strbuf_getline(&input, stdin);
if (ret == EOF)
break;
}
It's tempting to think that we could use `strbuf_getwholeline()` and
specify either `\n` or `\0` as the terminating character. But for input
on platforms that include a CR character preceeding the LF, this
wouldn't quite be the same, since `strbuf_getline(...)` will trim any
trailing CR, while `strbuf_getwholeline(&buf, stdin, '\n')` will not.
In the future, we could clean this up further by introducing a variant
of `strbuf_getwholeline()` that addresses the aforementioned gap, but
that approach felt too heavy-handed for this pair of uses.
Some tests are added in t1006 to ensure that `cat-file` produces the
same output in `--batch`, `--batch-check`, and `--batch-command` modes
with and without the new `-z` option.
Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-07-22 23:29:05 +00:00
|
|
|
-z::
|
|
|
|
|
Only meaningful with `--batch`, `--batch-check`, or
|
|
|
|
|
`--batch-command`; input is NUL-delimited instead of
|
cat-file: add option '-Z' that delimits input and output with NUL
In db9d67f2e9 (builtin/cat-file.c: support NUL-delimited input with
`-z`, 2022-07-22), we have introduced a new mode to read the input via
NUL-delimited records instead of newline-delimited records. This allows
the user to query for revisions that have newlines in their path
component. While unusual, such queries are perfectly valid and thus it
is clear that we should be able to support them properly.
Unfortunately, the commit only changed the input to be NUL-delimited,
but didn't change the output at the same time. While this is fine for
queries that are processed successfully, it is less so for queries that
aren't. In the case of missing commits for example the result can become
entirely unparsable:
```
$ printf "7ce4f05bae8120d9fa258e854a8669f6ea9cb7b1 blob 10\n1234567890\n\n\commit000" |
git cat-file --batch -z
7ce4f05bae8120d9fa258e854a8669f6ea9cb7b1 blob 10
1234567890
commit missing
```
This is of course a crafted query that is intentionally gaming the
deficiency, but more benign queries that contain newlines would have
similar problems.
Ideally, we should have also changed the output to be NUL-delimited when
`-z` is specified to avoid this problem. As the input is NUL-delimited,
it is clear that the output in this case cannot ever contain NUL
characters by itself. Furthermore, Git does not allow NUL characters in
revisions anyway, further stressing the point that using NUL-delimited
output is safe. The only exception is of course the object data itself,
but as git-cat-file(1) prints the size of the object data clients should
read until that specified size has been consumed.
But even though `-z` has only been introduced a few releases ago in Git
v2.38.0, changing the output format retroactively to also NUL-delimit
output would be a backwards incompatible change. And while one could
make the argument that the output is inherently broken already, we need
to assume that there are existing users out there that use it just fine
given that revisions containing newlines are quite exotic.
Instead, introduce a new option `-Z` that switches to NUL-delimited
input and output. While this new option could arguably only switch the
output format to be NUL-delimited, the consequence would be that users
have to always specify both `-z` and `-Z` when the input may contain
newlines. On the other hand, if the user knows that there never will be
newlines in the input, they don't have to use either of those options.
There is thus no usecase that would warrant treating input and output
format separately, which is why we instead opt to "do the right thing"
and have `-Z` mean to NUL-terminate both formats.
The old `-z` option is marked as deprecated with a hint that its output
may become unparsable. It is thus hidden both from the synopsis as well
as the command's help output.
Co-authored-by: Toon Claes <toon@iotcl.com>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-06-06 05:19:45 +00:00
|
|
|
newline-delimited. This option is deprecated in favor of
|
|
|
|
|
`-Z` as the output can otherwise be ambiguous.
|
builtin/cat-file.c: support NUL-delimited input with `-z`
When callers are using `cat-file` via one of the stdin-driven `--batch`
modes, all input is newline-delimited. This presents a problem when
callers wish to ask about, e.g. tree-entries that have a newline
character present in their filename.
To support this niche scenario, introduce a new `-z` mode to the
`--batch`, `--batch-check`, and `--batch-command` suite of options that
instructs `cat-file` to treat its input as NUL-delimited, allowing the
individual commands themselves to have newlines present.
The refactoring here is slightly unfortunate, since we turn loops like:
while (strbuf_getline(&buf, stdin) != EOF)
into:
while (1) {
int ret;
if (opt->nul_terminated)
ret = strbuf_getline_nul(&input, stdin);
else
ret = strbuf_getline(&input, stdin);
if (ret == EOF)
break;
}
It's tempting to think that we could use `strbuf_getwholeline()` and
specify either `\n` or `\0` as the terminating character. But for input
on platforms that include a CR character preceeding the LF, this
wouldn't quite be the same, since `strbuf_getline(...)` will trim any
trailing CR, while `strbuf_getwholeline(&buf, stdin, '\n')` will not.
In the future, we could clean this up further by introducing a variant
of `strbuf_getwholeline()` that addresses the aforementioned gap, but
that approach felt too heavy-handed for this pair of uses.
Some tests are added in t1006 to ensure that `cat-file` produces the
same output in `--batch`, `--batch-check`, and `--batch-command` modes
with and without the new `-z` option.
Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-07-22 23:29:05 +00:00
|
|
|
|
2015-05-20 17:03:40 +00:00
|
|
|
|
2005-05-10 21:32:30 +00:00
|
|
|
OUTPUT
|
|
|
|
|
------
|
2021-03-03 09:21:55 +00:00
|
|
|
If `-t` is specified, one of the `<type>`.
|
2005-12-04 01:57:48 +00:00
|
|
|
|
2021-03-03 09:21:55 +00:00
|
|
|
If `-s` is specified, the size of the `<object>` in bytes.
|
2005-12-04 01:57:48 +00:00
|
|
|
|
2021-03-03 09:21:55 +00:00
|
|
|
If `-e` is specified, no output, unless the `<object>` is malformed.
|
2005-05-10 21:32:30 +00:00
|
|
|
|
2021-03-03 09:21:55 +00:00
|
|
|
If `-p` is specified, the contents of `<object>` are pretty-printed.
|
2006-05-25 01:22:32 +00:00
|
|
|
|
2021-03-03 09:21:55 +00:00
|
|
|
If `<type>` is specified, the raw (though uncompressed) contents of the `<object>`
|
2008-04-23 19:17:46 +00:00
|
|
|
will be returned.
|
|
|
|
|
|
2013-07-10 11:45:47 +00:00
|
|
|
BATCH OUTPUT
|
|
|
|
|
------------
|
|
|
|
|
|
|
|
|
|
If `--batch` or `--batch-check` is given, `cat-file` will read objects
|
cat-file: only split on whitespace when %(rest) is used
Commit c334b87b (cat-file: split --batch input lines on whitespace,
2013-07-11) taught `cat-file --batch-check` to split input lines on
the first whitespace, and stash everything after the first token
into the %(rest) output format element. It claimed:
Object names cannot contain spaces, so any input with
spaces would have resulted in a "missing" line.
But that is not correct. Refs, object sha1s, and various peeling
suffixes cannot contain spaces, but some object names can. In
particular:
1. Tree paths like "[<tree>]:path with whitespace"
2. Reflog specifications like "@{2 days ago}"
3. Commit searches like "rev^{/grep me}" or ":/grep me"
To remain backwards compatible, we cannot split on whitespace by
default, hence we will ship 1.8.4 with the commit reverted.
Resurrect its attempt but in a weaker form; only do the splitting
when "%(rest)" is used in the output format. Since that element did
not exist at all before c334b87, old scripts cannot be affected.
The existence of object names with spaces does mean that you
cannot reliably do:
echo ":path with space and other data" |
git cat-file --batch-check="%(objectname) %(rest)"
as it would split the path and feed only ":path" to get_sha1. But
that command is nonsensical. If you wanted to see "and other data"
in "%(rest)", git cannot possibly know where the filename ends and
the "rest" begins.
It might be more robust to have something like "-z" to separate the
input elements. But this patch is still a reasonable step before
having that. It makes the easy cases easy; people who do not care
about %(rest) do not have to consider it, and the %(rest) code
handles the spaces and newlines of "rev-list --objects" correctly.
Hard cases remain hard but possible (if you might get whitespace in
your input, you do not get to use %(rest) and must split and join
the output yourself using more flexible tools). And most
importantly, it does not preclude us from having different splitting
rules later if a "-z" (or similar) option is added. So we can make
the hard cases easier later, if we choose to.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-08-02 11:59:07 +00:00
|
|
|
from stdin, one per line, and print information about them. By default,
|
|
|
|
|
the whole line is considered as an object, as if it were fed to
|
|
|
|
|
linkgit:git-rev-parse[1].
|
2013-07-10 11:45:47 +00:00
|
|
|
|
cat-file: add --batch-command mode
Add a new flag --batch-command that accepts commands and arguments
from stdin, similar to git-update-ref --stdin.
At GitLab, we use a pair of long running cat-file processes when
accessing object content. One for iterating over object metadata with
--batch-check, and the other to grab object contents with --batch.
However, if we had --batch-command, we wouldn't need to keep both
processes around, and instead just have one --batch-command process
where we can flip between getting object info, and getting object
contents. Since we have a pair of cat-file processes per repository,
this means we can get rid of roughly half of long lived git cat-file
processes. Given there are many repositories being accessed at any given
time, this can lead to huge savings.
git cat-file --batch-command
will enter an interactive command mode whereby the user can enter in
commands and their arguments that get queued in memory:
<command1> [arg1] [arg2] LF
<command2> [arg1] [arg2] LF
When --buffer mode is used, commands will be queued in memory until a
flush command is issued that execute them:
flush LF
The reason for a flush command is that when a consumer process (A)
talks to a git cat-file process (B) and interactively writes to and
reads from it in --buffer mode, (A) needs to be able to control when
the buffer is flushed to stdout.
Currently, from (A)'s perspective, the only way is to either
1. kill (B)'s process
2. send an invalid object to stdin.
1. is not ideal from a performance perspective as it will require
spawning a new cat-file process each time, and 2. is hacky and not a
good long term solution.
With this mechanism of queueing up commands and letting (A) issue a
flush command, process (A) can control when the buffer is flushed and
can guarantee it will receive all of the output when in --buffer mode.
--batch-command also will not allow (B) to flush to stdout until a flush
is received.
This patch adds the basic structure for adding command which can be
extended in the future to add more commands. It also adds the following
two commands (on top of the flush command):
contents <object> LF
info <object> LF
The contents command takes an <object> argument and prints out the object
contents.
The info command takes an <object> argument and prints out the object
metadata.
These can be used in the following way with --buffer:
info <object> LF
contents <object> LF
contents <object> LF
info <object> LF
flush LF
info <object> LF
flush LF
When used without --buffer:
info <object> LF
contents <object> LF
contents <object> LF
info <object> LF
info <object> LF
Helped-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: John Cai <johncai86@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-02-18 18:23:14 +00:00
|
|
|
When `--batch-command` is given, `cat-file` will read commands from stdin,
|
|
|
|
|
one per line, and print information based on the command given. With
|
|
|
|
|
`--batch-command`, the `info` command followed by an object will print
|
|
|
|
|
information about the object the same way `--batch-check` would, and the
|
|
|
|
|
`contents` command followed by an object prints contents in the same way
|
|
|
|
|
`--batch` would.
|
|
|
|
|
|
2013-07-10 11:45:47 +00:00
|
|
|
You can specify the information shown for each object by using a custom
|
|
|
|
|
`<format>`. The `<format>` is copied literally to stdout for each
|
|
|
|
|
object, with placeholders of the form `%(atom)` expanded, followed by a
|
|
|
|
|
newline. The available atoms are:
|
|
|
|
|
|
|
|
|
|
`objectname`::
|
2021-03-03 09:21:56 +00:00
|
|
|
The full hex representation of the object name.
|
2013-07-10 11:45:47 +00:00
|
|
|
|
|
|
|
|
`objecttype`::
|
2017-09-12 22:58:39 +00:00
|
|
|
The type of the object (the same as `cat-file -t` reports).
|
2013-07-10 11:45:47 +00:00
|
|
|
|
|
|
|
|
`objectsize`::
|
|
|
|
|
The size, in bytes, of the object (the same as `cat-file -s`
|
|
|
|
|
reports).
|
|
|
|
|
|
2013-07-10 11:46:25 +00:00
|
|
|
`objectsize:disk`::
|
|
|
|
|
The size, in bytes, that the object takes up on disk. See the
|
|
|
|
|
note about on-disk sizes in the `CAVEATS` section below.
|
|
|
|
|
|
2013-12-21 14:25:22 +00:00
|
|
|
`deltabase`::
|
|
|
|
|
If the object is stored as a delta on-disk, this expands to the
|
2021-03-03 09:21:56 +00:00
|
|
|
full hex representation of the delta base object name.
|
|
|
|
|
Otherwise, expands to the null OID (all zeroes). See `CAVEATS`
|
|
|
|
|
below.
|
2013-12-21 14:25:22 +00:00
|
|
|
|
cat-file: only split on whitespace when %(rest) is used
Commit c334b87b (cat-file: split --batch input lines on whitespace,
2013-07-11) taught `cat-file --batch-check` to split input lines on
the first whitespace, and stash everything after the first token
into the %(rest) output format element. It claimed:
Object names cannot contain spaces, so any input with
spaces would have resulted in a "missing" line.
But that is not correct. Refs, object sha1s, and various peeling
suffixes cannot contain spaces, but some object names can. In
particular:
1. Tree paths like "[<tree>]:path with whitespace"
2. Reflog specifications like "@{2 days ago}"
3. Commit searches like "rev^{/grep me}" or ":/grep me"
To remain backwards compatible, we cannot split on whitespace by
default, hence we will ship 1.8.4 with the commit reverted.
Resurrect its attempt but in a weaker form; only do the splitting
when "%(rest)" is used in the output format. Since that element did
not exist at all before c334b87, old scripts cannot be affected.
The existence of object names with spaces does mean that you
cannot reliably do:
echo ":path with space and other data" |
git cat-file --batch-check="%(objectname) %(rest)"
as it would split the path and feed only ":path" to get_sha1. But
that command is nonsensical. If you wanted to see "and other data"
in "%(rest)", git cannot possibly know where the filename ends and
the "rest" begins.
It might be more robust to have something like "-z" to separate the
input elements. But this patch is still a reasonable step before
having that. It makes the easy cases easy; people who do not care
about %(rest) do not have to consider it, and the %(rest) code
handles the spaces and newlines of "rev-list --objects" correctly.
Hard cases remain hard but possible (if you might get whitespace in
your input, you do not get to use %(rest) and must split and join
the output yourself using more flexible tools). And most
importantly, it does not preclude us from having different splitting
rules later if a "-z" (or similar) option is added. So we can make
the hard cases easier later, if we choose to.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2013-08-02 11:59:07 +00:00
|
|
|
`rest`::
|
|
|
|
|
If this atom is used in the output string, input lines are split
|
|
|
|
|
at the first whitespace boundary. All characters before that
|
|
|
|
|
whitespace are considered to be the object name; characters
|
|
|
|
|
after that first run of whitespace (i.e., the "rest" of the
|
|
|
|
|
line) are output in place of the `%(rest)` atom.
|
|
|
|
|
|
2013-07-10 11:45:47 +00:00
|
|
|
If no format is specified, the default format is `%(objectname)
|
|
|
|
|
%(objecttype) %(objectsize)`.
|
|
|
|
|
|
cat-file: add --batch-command mode
Add a new flag --batch-command that accepts commands and arguments
from stdin, similar to git-update-ref --stdin.
At GitLab, we use a pair of long running cat-file processes when
accessing object content. One for iterating over object metadata with
--batch-check, and the other to grab object contents with --batch.
However, if we had --batch-command, we wouldn't need to keep both
processes around, and instead just have one --batch-command process
where we can flip between getting object info, and getting object
contents. Since we have a pair of cat-file processes per repository,
this means we can get rid of roughly half of long lived git cat-file
processes. Given there are many repositories being accessed at any given
time, this can lead to huge savings.
git cat-file --batch-command
will enter an interactive command mode whereby the user can enter in
commands and their arguments that get queued in memory:
<command1> [arg1] [arg2] LF
<command2> [arg1] [arg2] LF
When --buffer mode is used, commands will be queued in memory until a
flush command is issued that execute them:
flush LF
The reason for a flush command is that when a consumer process (A)
talks to a git cat-file process (B) and interactively writes to and
reads from it in --buffer mode, (A) needs to be able to control when
the buffer is flushed to stdout.
Currently, from (A)'s perspective, the only way is to either
1. kill (B)'s process
2. send an invalid object to stdin.
1. is not ideal from a performance perspective as it will require
spawning a new cat-file process each time, and 2. is hacky and not a
good long term solution.
With this mechanism of queueing up commands and letting (A) issue a
flush command, process (A) can control when the buffer is flushed and
can guarantee it will receive all of the output when in --buffer mode.
--batch-command also will not allow (B) to flush to stdout until a flush
is received.
This patch adds the basic structure for adding command which can be
extended in the future to add more commands. It also adds the following
two commands (on top of the flush command):
contents <object> LF
info <object> LF
The contents command takes an <object> argument and prints out the object
contents.
The info command takes an <object> argument and prints out the object
metadata.
These can be used in the following way with --buffer:
info <object> LF
contents <object> LF
contents <object> LF
info <object> LF
flush LF
info <object> LF
flush LF
When used without --buffer:
info <object> LF
contents <object> LF
contents <object> LF
info <object> LF
info <object> LF
Helped-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: John Cai <johncai86@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-02-18 18:23:14 +00:00
|
|
|
If `--batch` is specified, or if `--batch-command` is used with the `contents`
|
|
|
|
|
command, the object information is followed by the object contents (consisting
|
|
|
|
|
of `%(objectsize)` bytes), followed by a newline.
|
2013-07-10 11:45:47 +00:00
|
|
|
|
|
|
|
|
For example, `--batch` without a custom format would produce:
|
2008-04-23 19:17:47 +00:00
|
|
|
|
|
|
|
|
------------
|
2021-03-03 09:21:56 +00:00
|
|
|
<oid> SP <type> SP <size> LF
|
2008-04-23 19:17:47 +00:00
|
|
|
<contents> LF
|
|
|
|
|
------------
|
|
|
|
|
|
2013-07-10 11:45:47 +00:00
|
|
|
Whereas `--batch-check='%(objectname) %(objecttype)'` would produce:
|
2008-04-23 19:17:46 +00:00
|
|
|
|
|
|
|
|
------------
|
2021-03-03 09:21:56 +00:00
|
|
|
<oid> SP <type> LF
|
2008-04-23 19:17:46 +00:00
|
|
|
------------
|
|
|
|
|
|
2013-07-10 11:45:47 +00:00
|
|
|
If a name is specified on stdin that cannot be resolved to an object in
|
|
|
|
|
the repository, then `cat-file` will ignore any custom format and print:
|
2005-05-10 21:32:30 +00:00
|
|
|
|
2008-04-23 19:17:46 +00:00
|
|
|
------------
|
|
|
|
|
<object> SP missing LF
|
|
|
|
|
------------
|
2005-05-10 21:32:30 +00:00
|
|
|
|
2019-01-18 04:19:43 +00:00
|
|
|
If a name is specified that might refer to more than one object (an ambiguous short sha), then `cat-file` will ignore any custom format and print:
|
|
|
|
|
|
|
|
|
|
------------
|
|
|
|
|
<object> SP ambiguous LF
|
|
|
|
|
------------
|
|
|
|
|
|
2021-03-03 09:21:55 +00:00
|
|
|
If `--follow-symlinks` is used, and a symlink in the repository points
|
2015-05-20 17:03:40 +00:00
|
|
|
outside the repository, then `cat-file` will ignore any custom format
|
|
|
|
|
and print:
|
|
|
|
|
|
|
|
|
|
------------
|
|
|
|
|
symlink SP <size> LF
|
|
|
|
|
<symlink> LF
|
|
|
|
|
------------
|
|
|
|
|
|
2021-03-03 09:21:55 +00:00
|
|
|
The symlink will either be absolute (beginning with a `/`), or relative
|
|
|
|
|
to the tree root. For instance, if dir/link points to `../../foo`, then
|
|
|
|
|
`<symlink>` will be `../foo`. `<size>` is the size of the symlink in bytes.
|
2015-05-20 17:03:40 +00:00
|
|
|
|
2021-03-03 09:21:55 +00:00
|
|
|
If `--follow-symlinks` is used, the following error messages will be
|
2015-05-20 17:03:40 +00:00
|
|
|
displayed:
|
|
|
|
|
|
|
|
|
|
------------
|
|
|
|
|
<object> SP missing LF
|
|
|
|
|
------------
|
|
|
|
|
is printed when the initial symlink requested does not exist.
|
|
|
|
|
|
|
|
|
|
------------
|
|
|
|
|
dangling SP <size> LF
|
|
|
|
|
<object> LF
|
|
|
|
|
------------
|
|
|
|
|
is printed when the initial symlink exists, but something that
|
|
|
|
|
it (transitive-of) points to does not.
|
|
|
|
|
|
|
|
|
|
------------
|
|
|
|
|
loop SP <size> LF
|
|
|
|
|
<object> LF
|
|
|
|
|
------------
|
|
|
|
|
is printed for symlink loops (or any symlinks that
|
|
|
|
|
require more than 40 link resolutions to resolve).
|
|
|
|
|
|
|
|
|
|
------------
|
|
|
|
|
notdir SP <size> LF
|
|
|
|
|
<object> LF
|
|
|
|
|
------------
|
|
|
|
|
is printed when, during symlink resolution, a file is used as a
|
|
|
|
|
directory name.
|
2013-07-10 11:46:25 +00:00
|
|
|
|
cat-file: add option '-Z' that delimits input and output with NUL
In db9d67f2e9 (builtin/cat-file.c: support NUL-delimited input with
`-z`, 2022-07-22), we have introduced a new mode to read the input via
NUL-delimited records instead of newline-delimited records. This allows
the user to query for revisions that have newlines in their path
component. While unusual, such queries are perfectly valid and thus it
is clear that we should be able to support them properly.
Unfortunately, the commit only changed the input to be NUL-delimited,
but didn't change the output at the same time. While this is fine for
queries that are processed successfully, it is less so for queries that
aren't. In the case of missing commits for example the result can become
entirely unparsable:
```
$ printf "7ce4f05bae8120d9fa258e854a8669f6ea9cb7b1 blob 10\n1234567890\n\n\commit000" |
git cat-file --batch -z
7ce4f05bae8120d9fa258e854a8669f6ea9cb7b1 blob 10
1234567890
commit missing
```
This is of course a crafted query that is intentionally gaming the
deficiency, but more benign queries that contain newlines would have
similar problems.
Ideally, we should have also changed the output to be NUL-delimited when
`-z` is specified to avoid this problem. As the input is NUL-delimited,
it is clear that the output in this case cannot ever contain NUL
characters by itself. Furthermore, Git does not allow NUL characters in
revisions anyway, further stressing the point that using NUL-delimited
output is safe. The only exception is of course the object data itself,
but as git-cat-file(1) prints the size of the object data clients should
read until that specified size has been consumed.
But even though `-z` has only been introduced a few releases ago in Git
v2.38.0, changing the output format retroactively to also NUL-delimit
output would be a backwards incompatible change. And while one could
make the argument that the output is inherently broken already, we need
to assume that there are existing users out there that use it just fine
given that revisions containing newlines are quite exotic.
Instead, introduce a new option `-Z` that switches to NUL-delimited
input and output. While this new option could arguably only switch the
output format to be NUL-delimited, the consequence would be that users
have to always specify both `-z` and `-Z` when the input may contain
newlines. On the other hand, if the user knows that there never will be
newlines in the input, they don't have to use either of those options.
There is thus no usecase that would warrant treating input and output
format separately, which is why we instead opt to "do the right thing"
and have `-Z` mean to NUL-terminate both formats.
The old `-z` option is marked as deprecated with a hint that its output
may become unparsable. It is thus hidden both from the synopsis as well
as the command's help output.
Co-authored-by: Toon Claes <toon@iotcl.com>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-06-06 05:19:45 +00:00
|
|
|
Alternatively, when `-Z` is passed, the line feeds in any of the above examples
|
|
|
|
|
are replaced with NUL terminators. This ensures that output will be parsable if
|
|
|
|
|
the output itself would contain a linefeed and is thus recommended for
|
|
|
|
|
scripting purposes.
|
|
|
|
|
|
2013-07-10 11:46:25 +00:00
|
|
|
CAVEATS
|
|
|
|
|
-------
|
|
|
|
|
|
|
|
|
|
Note that the sizes of objects on disk are reported accurately, but care
|
|
|
|
|
should be taken in drawing conclusions about which refs or objects are
|
|
|
|
|
responsible for disk usage. The size of a packed non-delta object may be
|
|
|
|
|
much larger than the size of objects which delta against it, but the
|
|
|
|
|
choice of which object is the base and which is the delta is arbitrary
|
2013-12-21 14:25:22 +00:00
|
|
|
and is subject to change during a repack.
|
2013-07-10 11:46:25 +00:00
|
|
|
|
2013-12-21 14:25:22 +00:00
|
|
|
Note also that multiple copies of an object may be present in the object
|
|
|
|
|
database; in this case, it is undefined which copy's size or delta base
|
|
|
|
|
will be reported.
|
2013-07-10 11:46:25 +00:00
|
|
|
|
2005-05-10 21:32:30 +00:00
|
|
|
GIT
|
|
|
|
|
---
|
2008-06-06 07:07:32 +00:00
|
|
|
Part of the linkgit:git[1] suite
|