2023-04-11 07:41:56 +00:00
|
|
|
#include "git-compat-util.h"
|
2023-04-22 20:17:26 +00:00
|
|
|
#include "date.h"
|
2023-05-16 06:34:06 +00:00
|
|
|
#include "dir.h"
|
2023-02-24 00:09:27 +00:00
|
|
|
#include "hex.h"
|
2023-05-16 06:34:06 +00:00
|
|
|
#include "object-store-ll.h"
|
2023-05-16 06:33:59 +00:00
|
|
|
#include "path.h"
|
2018-06-29 01:21:51 +00:00
|
|
|
#include "repository.h"
|
2008-02-25 21:46:04 +00:00
|
|
|
#include "object.h"
|
2022-12-01 14:46:09 +00:00
|
|
|
#include "attr.h"
|
2008-02-25 21:46:04 +00:00
|
|
|
#include "blob.h"
|
|
|
|
#include "tree.h"
|
|
|
|
#include "tree-walk.h"
|
|
|
|
#include "commit.h"
|
|
|
|
#include "tag.h"
|
|
|
|
#include "fsck.h"
|
2014-09-11 14:26:38 +00:00
|
|
|
#include "refs.h"
|
2020-04-19 03:52:34 +00:00
|
|
|
#include "url.h"
|
fsck: complain about HFS+ ".git" aliases in trees
Now that the index can block pathnames that case-fold to
".git" on HFS+, it would be helpful for fsck to notice such
problematic paths. This lets servers which use
receive.fsckObjects block them before the damage spreads.
Note that the fsck check is always on, even for systems
without core.protectHFS set. This is technically more
restrictive than we need to be, as a set of users on ext4
could happily use these odd filenames without caring about
HFS+.
However, on balance, it's helpful for all servers to block
these (because the paths can be used for mischief, and
servers which bother to fsck would want to stop the spread
whether they are on HFS+ themselves or not), and hardly
anybody will be affected (because the blocked names are
variants of .git with invisible Unicode code-points mixed
in, meaning mischief is almost certainly what the tree
author had in mind).
Ideally these would be controlled by a separate
"fsck.protectHFS" flag. However, it would be much nicer to
be able to enable/disable _any_ fsck flag individually, and
any scheme we choose should match such a system. Given the
likelihood of anybody using such a path in practice, it is
not unreasonable to wait until such a system materializes.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-12-15 23:21:57 +00:00
|
|
|
#include "utf8.h"
|
2016-07-17 10:59:49 +00:00
|
|
|
#include "decorate.h"
|
fsck: detect gitmodules files
In preparation for performing fsck checks on .gitmodules
files, this commit plumbs in the actual detection of the
files. Note that unlike most other fsck checks, this cannot
be a property of a single object: we must know that the
object is found at a ".gitmodules" path at the root tree of
a commit.
Since the fsck code only sees one object at a time, we have
to mark the related objects to fit the puzzle together. When
we see a commit we mark its tree as a root tree, and when
we see a root tree with a .gitmodules file, we mark the
corresponding blob to be checked.
In an ideal world, we'd check the objects in topological
order: commits followed by trees followed by blobs. In that
case we can avoid ever loading an object twice, since all
markings would be complete by the time we get to the marked
objects. And indeed, if we are checking a single packfile,
this is the order in which Git will generally write the
objects. But we can't count on that:
1. git-fsck may show us the objects in arbitrary order
(loose objects are fed in sha1 order, but we may also
have multiple packs, and we process each pack fully in
sequence).
2. The type ordering is just what git-pack-objects happens
to write now. The pack format does not require a
specific order, and it's possible that future versions
of Git (or a custom version trying to fool official
Git's fsck checks!) may order it differently.
3. We may not even be fscking all of the relevant objects
at once. Consider pushing with transfer.fsckObjects,
where one push adds a blob at path "foo", and then a
second push adds the same blob at path ".gitmodules".
The blob is not part of the second push at all, but we
need to mark and check it.
So in the general case, we need to make up to three passes
over the objects: once to make sure we've seen all commits,
then once to cover any trees we might have missed, and then
a final pass to cover any .gitmodules blobs we found in the
second pass.
We can simplify things a bit by loosening the requirement
that we find .gitmodules only at root trees. Technically
a file like "subdir/.gitmodules" is not parsed by Git, but
it's not unreasonable for us to declare that Git is aware of
all ".gitmodules" files and make them eligible for checking.
That lets us drop the root-tree requirement, which
eliminates one pass entirely. And it makes our worst case
much better: instead of potentially queueing every root tree
to be re-examined, the worst case is that we queue each
unique .gitmodules blob for a second look.
This patch just adds the boilerplate to find .gitmodules
files. The actual content checks will come in a subsequent
commit.
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 21:20:08 +00:00
|
|
|
#include "oidset.h"
|
2018-05-14 16:22:48 +00:00
|
|
|
#include "packfile.h"
|
2018-05-02 21:25:27 +00:00
|
|
|
#include "submodule-config.h"
|
|
|
|
#include "config.h"
|
2020-03-11 22:48:24 +00:00
|
|
|
#include "credential.h"
|
2018-05-26 13:55:24 +00:00
|
|
|
#include "help.h"
|
fsck: detect gitmodules files
In preparation for performing fsck checks on .gitmodules
files, this commit plumbs in the actual detection of the
files. Note that unlike most other fsck checks, this cannot
be a property of a single object: we must know that the
object is found at a ".gitmodules" path at the root tree of
a commit.
Since the fsck code only sees one object at a time, we have
to mark the related objects to fit the puzzle together. When
we see a commit we mark its tree as a root tree, and when
we see a root tree with a .gitmodules file, we mark the
corresponding blob to be checked.
In an ideal world, we'd check the objects in topological
order: commits followed by trees followed by blobs. In that
case we can avoid ever loading an object twice, since all
markings would be complete by the time we get to the marked
objects. And indeed, if we are checking a single packfile,
this is the order in which Git will generally write the
objects. But we can't count on that:
1. git-fsck may show us the objects in arbitrary order
(loose objects are fed in sha1 order, but we may also
have multiple packs, and we process each pack fully in
sequence).
2. The type ordering is just what git-pack-objects happens
to write now. The pack format does not require a
specific order, and it's possible that future versions
of Git (or a custom version trying to fool official
Git's fsck checks!) may order it differently.
3. We may not even be fscking all of the relevant objects
at once. Consider pushing with transfer.fsckObjects,
where one push adds a blob at path "foo", and then a
second push adds the same blob at path ".gitmodules".
The blob is not part of the second push at all, but we
need to mark and check it.
So in the general case, we need to make up to three passes
over the objects: once to make sure we've seen all commits,
then once to cover any trees we might have missed, and then
a final pass to cover any .gitmodules blobs we found in the
second pass.
We can simplify things a bit by loosening the requirement
that we find .gitmodules only at root trees. Technically
a file like "subdir/.gitmodules" is not parsed by Git, but
it's not unreasonable for us to declare that Git is aware of
all ".gitmodules" files and make them eligible for checking.
That lets us drop the root-tree requirement, which
eliminates one pass entirely. And it makes our worst case
much better: instead of potentially queueing every root tree
to be re-examined, the worst case is that we queue each
unique .gitmodules blob for a second look.
This patch just adds the boilerplate to find .gitmodules
files. The actual content checks will come in a subsequent
commit.
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 21:20:08 +00:00
|
|
|
|
2015-06-22 15:25:14 +00:00
|
|
|
#define STR(x) #x
|
2018-05-26 13:55:25 +00:00
|
|
|
#define MSG_ID(id, msg_type) { STR(id), NULL, NULL, FSCK_##msg_type },
|
2015-06-22 15:25:09 +00:00
|
|
|
static struct {
|
2015-06-22 15:25:14 +00:00
|
|
|
const char *id_string;
|
|
|
|
const char *downcased;
|
2018-05-26 13:55:25 +00:00
|
|
|
const char *camelcased;
|
fsck.h: move FSCK_{FATAL,INFO,ERROR,WARN,IGNORE} into an enum
Move the FSCK_{FATAL,INFO,ERROR,WARN,IGNORE} defines into a new
fsck_msg_type enum.
These defines were originally introduced in:
- ba002f3b28a (builtin-fsck: move common object checking code to
fsck.c, 2008-02-25)
- f50c4407305 (fsck: disallow demoting grave fsck errors to warnings,
2015-06-22)
- efaba7cc77f (fsck: optionally ignore specific fsck issues
completely, 2015-06-22)
- f27d05b1704 (fsck: allow upgrading fsck warnings to errors,
2015-06-22)
The reason these were defined in two different places is because we
use FSCK_{IGNORE,INFO,FATAL} only in fsck.c, but FSCK_{ERROR,WARN} are
used by external callbacks.
Untangling that would take some more work, since we expose the new
"enum fsck_msg_type" to both. Similar to "enum object_type" it's not
worth structuring the API in such a way that only those who need
FSCK_{ERROR,WARN} pass around a different type.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-03-28 13:15:40 +00:00
|
|
|
enum fsck_msg_type msg_type;
|
2015-06-22 15:25:09 +00:00
|
|
|
} msg_id_info[FSCK_MSG_MAX + 1] = {
|
2021-03-28 13:15:44 +00:00
|
|
|
FOREACH_FSCK_MSG_ID(MSG_ID)
|
2018-05-26 13:55:25 +00:00
|
|
|
{ NULL, NULL, NULL, -1 }
|
2015-06-22 15:25:09 +00:00
|
|
|
};
|
|
|
|
#undef MSG_ID
|
2021-03-28 13:15:43 +00:00
|
|
|
#undef STR
|
2015-06-22 15:25:09 +00:00
|
|
|
|
2018-05-26 13:55:23 +00:00
|
|
|
static void prepare_msg_ids(void)
|
2015-06-22 15:25:14 +00:00
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
2018-05-26 13:55:23 +00:00
|
|
|
if (msg_id_info[0].downcased)
|
|
|
|
return;
|
|
|
|
|
|
|
|
/* convert id_string to lower case, without underscores. */
|
|
|
|
for (i = 0; i < FSCK_MSG_MAX; i++) {
|
|
|
|
const char *p = msg_id_info[i].id_string;
|
|
|
|
int len = strlen(p);
|
|
|
|
char *q = xmalloc(len);
|
|
|
|
|
|
|
|
msg_id_info[i].downcased = q;
|
|
|
|
while (*p)
|
|
|
|
if (*p == '_')
|
|
|
|
p++;
|
|
|
|
else
|
|
|
|
*(q)++ = tolower(*(p)++);
|
|
|
|
*q = '\0';
|
2018-05-26 13:55:25 +00:00
|
|
|
|
|
|
|
p = msg_id_info[i].id_string;
|
|
|
|
q = xmalloc(len);
|
|
|
|
msg_id_info[i].camelcased = q;
|
|
|
|
while (*p) {
|
|
|
|
if (*p == '_') {
|
|
|
|
p++;
|
|
|
|
if (*p)
|
|
|
|
*q++ = *p++;
|
|
|
|
} else {
|
|
|
|
*q++ = tolower(*p++);
|
|
|
|
}
|
2015-06-22 15:25:14 +00:00
|
|
|
}
|
2018-05-26 13:55:25 +00:00
|
|
|
*q = '\0';
|
2015-06-22 15:25:14 +00:00
|
|
|
}
|
2018-05-26 13:55:23 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
static int parse_msg_id(const char *text)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
prepare_msg_ids();
|
2015-06-22 15:25:14 +00:00
|
|
|
|
|
|
|
for (i = 0; i < FSCK_MSG_MAX; i++)
|
|
|
|
if (!strcmp(text, msg_id_info[i].downcased))
|
|
|
|
return i;
|
|
|
|
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
2018-05-26 13:55:24 +00:00
|
|
|
void list_config_fsck_msg_ids(struct string_list *list, const char *prefix)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
prepare_msg_ids();
|
|
|
|
|
|
|
|
for (i = 0; i < FSCK_MSG_MAX; i++)
|
2018-05-26 13:55:25 +00:00
|
|
|
list_config_item(list, prefix, msg_id_info[i].camelcased);
|
2018-05-26 13:55:24 +00:00
|
|
|
}
|
|
|
|
|
fsck.h: move FSCK_{FATAL,INFO,ERROR,WARN,IGNORE} into an enum
Move the FSCK_{FATAL,INFO,ERROR,WARN,IGNORE} defines into a new
fsck_msg_type enum.
These defines were originally introduced in:
- ba002f3b28a (builtin-fsck: move common object checking code to
fsck.c, 2008-02-25)
- f50c4407305 (fsck: disallow demoting grave fsck errors to warnings,
2015-06-22)
- efaba7cc77f (fsck: optionally ignore specific fsck issues
completely, 2015-06-22)
- f27d05b1704 (fsck: allow upgrading fsck warnings to errors,
2015-06-22)
The reason these were defined in two different places is because we
use FSCK_{IGNORE,INFO,FATAL} only in fsck.c, but FSCK_{ERROR,WARN} are
used by external callbacks.
Untangling that would take some more work, since we expose the new
"enum fsck_msg_type" to both. Similar to "enum object_type" it's not
worth structuring the API in such a way that only those who need
FSCK_{ERROR,WARN} pass around a different type.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-03-28 13:15:40 +00:00
|
|
|
static enum fsck_msg_type fsck_msg_type(enum fsck_msg_id msg_id,
|
2015-06-22 15:25:09 +00:00
|
|
|
struct fsck_options *options)
|
|
|
|
{
|
2015-06-22 15:25:25 +00:00
|
|
|
assert(msg_id >= 0 && msg_id < FSCK_MSG_MAX);
|
|
|
|
|
2021-03-28 13:15:39 +00:00
|
|
|
if (!options->msg_type) {
|
fsck.h: move FSCK_{FATAL,INFO,ERROR,WARN,IGNORE} into an enum
Move the FSCK_{FATAL,INFO,ERROR,WARN,IGNORE} defines into a new
fsck_msg_type enum.
These defines were originally introduced in:
- ba002f3b28a (builtin-fsck: move common object checking code to
fsck.c, 2008-02-25)
- f50c4407305 (fsck: disallow demoting grave fsck errors to warnings,
2015-06-22)
- efaba7cc77f (fsck: optionally ignore specific fsck issues
completely, 2015-06-22)
- f27d05b1704 (fsck: allow upgrading fsck warnings to errors,
2015-06-22)
The reason these were defined in two different places is because we
use FSCK_{IGNORE,INFO,FATAL} only in fsck.c, but FSCK_{ERROR,WARN} are
used by external callbacks.
Untangling that would take some more work, since we expose the new
"enum fsck_msg_type" to both. Similar to "enum object_type" it's not
worth structuring the API in such a way that only those who need
FSCK_{ERROR,WARN} pass around a different type.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-03-28 13:15:40 +00:00
|
|
|
enum fsck_msg_type msg_type = msg_id_info[msg_id].msg_type;
|
2021-03-28 13:15:39 +00:00
|
|
|
|
2015-06-22 15:25:25 +00:00
|
|
|
if (options->strict && msg_type == FSCK_WARN)
|
|
|
|
msg_type = FSCK_ERROR;
|
2021-03-28 13:15:39 +00:00
|
|
|
return msg_type;
|
2015-06-22 15:25:25 +00:00
|
|
|
}
|
2015-06-22 15:25:09 +00:00
|
|
|
|
2021-03-28 13:15:39 +00:00
|
|
|
return options->msg_type[msg_id];
|
2015-06-22 15:25:09 +00:00
|
|
|
}
|
|
|
|
|
fsck.h: move FSCK_{FATAL,INFO,ERROR,WARN,IGNORE} into an enum
Move the FSCK_{FATAL,INFO,ERROR,WARN,IGNORE} defines into a new
fsck_msg_type enum.
These defines were originally introduced in:
- ba002f3b28a (builtin-fsck: move common object checking code to
fsck.c, 2008-02-25)
- f50c4407305 (fsck: disallow demoting grave fsck errors to warnings,
2015-06-22)
- efaba7cc77f (fsck: optionally ignore specific fsck issues
completely, 2015-06-22)
- f27d05b1704 (fsck: allow upgrading fsck warnings to errors,
2015-06-22)
The reason these were defined in two different places is because we
use FSCK_{IGNORE,INFO,FATAL} only in fsck.c, but FSCK_{ERROR,WARN} are
used by external callbacks.
Untangling that would take some more work, since we expose the new
"enum fsck_msg_type" to both. Similar to "enum object_type" it's not
worth structuring the API in such a way that only those who need
FSCK_{ERROR,WARN} pass around a different type.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-03-28 13:15:40 +00:00
|
|
|
static enum fsck_msg_type parse_msg_type(const char *str)
|
2015-06-22 15:25:25 +00:00
|
|
|
{
|
|
|
|
if (!strcmp(str, "error"))
|
|
|
|
return FSCK_ERROR;
|
|
|
|
else if (!strcmp(str, "warn"))
|
|
|
|
return FSCK_WARN;
|
2015-06-22 15:26:48 +00:00
|
|
|
else if (!strcmp(str, "ignore"))
|
|
|
|
return FSCK_IGNORE;
|
2015-06-22 15:25:25 +00:00
|
|
|
else
|
|
|
|
die("Unknown fsck message type: '%s'", str);
|
|
|
|
}
|
|
|
|
|
2015-06-22 15:25:31 +00:00
|
|
|
int is_valid_msg_type(const char *msg_id, const char *msg_type)
|
|
|
|
{
|
|
|
|
if (parse_msg_id(msg_id) < 0)
|
|
|
|
return 0;
|
|
|
|
parse_msg_type(msg_type);
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
2021-03-28 13:15:47 +00:00
|
|
|
void fsck_set_msg_type_from_ids(struct fsck_options *options,
|
|
|
|
enum fsck_msg_id msg_id,
|
|
|
|
enum fsck_msg_type msg_type)
|
|
|
|
{
|
|
|
|
if (!options->msg_type) {
|
|
|
|
int i;
|
|
|
|
enum fsck_msg_type *severity;
|
|
|
|
ALLOC_ARRAY(severity, FSCK_MSG_MAX);
|
|
|
|
for (i = 0; i < FSCK_MSG_MAX; i++)
|
|
|
|
severity[i] = fsck_msg_type(i, options);
|
|
|
|
options->msg_type = severity;
|
|
|
|
}
|
|
|
|
|
|
|
|
options->msg_type[msg_id] = msg_type;
|
|
|
|
}
|
|
|
|
|
2015-06-22 15:25:25 +00:00
|
|
|
void fsck_set_msg_type(struct fsck_options *options,
|
2021-03-28 13:15:36 +00:00
|
|
|
const char *msg_id_str, const char *msg_type_str)
|
2015-06-22 15:25:25 +00:00
|
|
|
{
|
fsck.h: move FSCK_{FATAL,INFO,ERROR,WARN,IGNORE} into an enum
Move the FSCK_{FATAL,INFO,ERROR,WARN,IGNORE} defines into a new
fsck_msg_type enum.
These defines were originally introduced in:
- ba002f3b28a (builtin-fsck: move common object checking code to
fsck.c, 2008-02-25)
- f50c4407305 (fsck: disallow demoting grave fsck errors to warnings,
2015-06-22)
- efaba7cc77f (fsck: optionally ignore specific fsck issues
completely, 2015-06-22)
- f27d05b1704 (fsck: allow upgrading fsck warnings to errors,
2015-06-22)
The reason these were defined in two different places is because we
use FSCK_{IGNORE,INFO,FATAL} only in fsck.c, but FSCK_{ERROR,WARN} are
used by external callbacks.
Untangling that would take some more work, since we expose the new
"enum fsck_msg_type" to both. Similar to "enum object_type" it's not
worth structuring the API in such a way that only those who need
FSCK_{ERROR,WARN} pass around a different type.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-03-28 13:15:40 +00:00
|
|
|
int msg_id = parse_msg_id(msg_id_str);
|
2021-03-28 13:15:42 +00:00
|
|
|
enum fsck_msg_type msg_type = parse_msg_type(msg_type_str);
|
2015-06-22 15:25:25 +00:00
|
|
|
|
2021-03-28 13:15:36 +00:00
|
|
|
if (msg_id < 0)
|
|
|
|
die("Unhandled message id: %s", msg_id_str);
|
2015-06-22 15:25:25 +00:00
|
|
|
|
2021-03-28 13:15:36 +00:00
|
|
|
if (msg_type != FSCK_ERROR && msg_id_info[msg_id].msg_type == FSCK_FATAL)
|
|
|
|
die("Cannot demote %s to %s", msg_id_str, msg_type_str);
|
2015-06-22 15:26:42 +00:00
|
|
|
|
2021-03-28 13:15:47 +00:00
|
|
|
fsck_set_msg_type_from_ids(options, msg_id, msg_type);
|
2015-06-22 15:25:25 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
void fsck_set_msg_types(struct fsck_options *options, const char *values)
|
|
|
|
{
|
|
|
|
char *buf = xstrdup(values), *to_free = buf;
|
|
|
|
int done = 0;
|
|
|
|
|
|
|
|
while (!done) {
|
|
|
|
int len = strcspn(buf, " ,|"), equal;
|
|
|
|
|
|
|
|
done = !buf[len];
|
|
|
|
if (!len) {
|
|
|
|
buf++;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
buf[len] = '\0';
|
|
|
|
|
|
|
|
for (equal = 0;
|
|
|
|
equal < len && buf[equal] != '=' && buf[equal] != ':';
|
|
|
|
equal++)
|
|
|
|
buf[equal] = tolower(buf[equal]);
|
|
|
|
buf[equal] = '\0';
|
|
|
|
|
2015-06-22 15:27:18 +00:00
|
|
|
if (!strcmp(buf, "skiplist")) {
|
|
|
|
if (equal == len)
|
|
|
|
die("skiplist requires a path");
|
2019-05-15 21:44:56 +00:00
|
|
|
oidset_parse_file(&options->skiplist, buf + equal + 1);
|
2015-06-22 15:27:18 +00:00
|
|
|
buf += len + 1;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
2015-06-22 15:25:25 +00:00
|
|
|
if (equal == len)
|
|
|
|
die("Missing '=': '%s'", buf);
|
2008-02-25 21:46:04 +00:00
|
|
|
|
2015-06-22 15:25:25 +00:00
|
|
|
fsck_set_msg_type(options, buf, buf + equal + 1);
|
|
|
|
buf += len + 1;
|
|
|
|
}
|
|
|
|
free(to_free);
|
|
|
|
}
|
|
|
|
|
2019-10-18 04:58:51 +00:00
|
|
|
static int object_on_skiplist(struct fsck_options *opts,
|
|
|
|
const struct object_id *oid)
|
fsck: check skiplist for object in fsck_blob()
Since commit ed8b10f631 ("fsck: check .gitmodules content", 2018-05-02),
fsck will issue an error message for '.gitmodules' content that cannot
be parsed correctly. This is the case, even when the corresponding blob
object has been included on the skiplist. For example, using the cgit
repository, we see the following:
$ git fsck
Checking object directories: 100% (256/256), done.
error: bad config line 5 in blob .gitmodules
error in blob 51dd1eff1edc663674df9ab85d2786a40f7ae3a5: gitmodulesParse: could not parse gitmodules blob
Checking objects: 100% (6626/6626), done.
$
$ git config fsck.skiplist '.git/skip'
$ echo 51dd1eff1edc663674df9ab85d2786a40f7ae3a5 >.git/skip
$
$ git fsck
Checking object directories: 100% (256/256), done.
error: bad config line 5 in blob .gitmodules
Checking objects: 100% (6626/6626), done.
$
Note that the error message issued by the config parser is still
present, despite adding the object-id of the blob to the skiplist.
One solution would be to provide a means of suppressing the messages
issued by the config parser. However, given that (logically) we are
asking fsck to ignore this object, a simpler approach is to just not
call the config parser if the object is to be skipped. Add a check to
the 'fsck_blob()' processing function, to determine if the object is
on the skiplist and, if so, exit the function early.
Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-06-27 18:39:53 +00:00
|
|
|
{
|
2019-10-18 04:58:51 +00:00
|
|
|
return opts && oid && oidset_contains(&opts->skiplist, oid);
|
fsck: check skiplist for object in fsck_blob()
Since commit ed8b10f631 ("fsck: check .gitmodules content", 2018-05-02),
fsck will issue an error message for '.gitmodules' content that cannot
be parsed correctly. This is the case, even when the corresponding blob
object has been included on the skiplist. For example, using the cgit
repository, we see the following:
$ git fsck
Checking object directories: 100% (256/256), done.
error: bad config line 5 in blob .gitmodules
error in blob 51dd1eff1edc663674df9ab85d2786a40f7ae3a5: gitmodulesParse: could not parse gitmodules blob
Checking objects: 100% (6626/6626), done.
$
$ git config fsck.skiplist '.git/skip'
$ echo 51dd1eff1edc663674df9ab85d2786a40f7ae3a5 >.git/skip
$
$ git fsck
Checking object directories: 100% (256/256), done.
error: bad config line 5 in blob .gitmodules
Checking objects: 100% (6626/6626), done.
$
Note that the error message issued by the config parser is still
present, despite adding the object-id of the blob to the skiplist.
One solution would be to provide a means of suppressing the messages
issued by the config parser. However, given that (logically) we are
asking fsck to ignore this object, a simpler approach is to just not
call the config parser if the object is to be skipped. Add a check to
the 'fsck_blob()' processing function, to determine if the object is
on the skiplist and, if so, exit the function early.
Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-06-27 18:39:53 +00:00
|
|
|
}
|
|
|
|
|
2019-10-18 04:59:15 +00:00
|
|
|
__attribute__((format (printf, 5, 6)))
|
|
|
|
static int report(struct fsck_options *options,
|
|
|
|
const struct object_id *oid, enum object_type object_type,
|
2021-03-28 13:15:38 +00:00
|
|
|
enum fsck_msg_id msg_id, const char *fmt, ...)
|
2015-06-22 15:25:09 +00:00
|
|
|
{
|
|
|
|
va_list ap;
|
|
|
|
struct strbuf sb = STRBUF_INIT;
|
fsck.h: move FSCK_{FATAL,INFO,ERROR,WARN,IGNORE} into an enum
Move the FSCK_{FATAL,INFO,ERROR,WARN,IGNORE} defines into a new
fsck_msg_type enum.
These defines were originally introduced in:
- ba002f3b28a (builtin-fsck: move common object checking code to
fsck.c, 2008-02-25)
- f50c4407305 (fsck: disallow demoting grave fsck errors to warnings,
2015-06-22)
- efaba7cc77f (fsck: optionally ignore specific fsck issues
completely, 2015-06-22)
- f27d05b1704 (fsck: allow upgrading fsck warnings to errors,
2015-06-22)
The reason these were defined in two different places is because we
use FSCK_{IGNORE,INFO,FATAL} only in fsck.c, but FSCK_{ERROR,WARN} are
used by external callbacks.
Untangling that would take some more work, since we expose the new
"enum fsck_msg_type" to both. Similar to "enum object_type" it's not
worth structuring the API in such a way that only those who need
FSCK_{ERROR,WARN} pass around a different type.
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-03-28 13:15:40 +00:00
|
|
|
enum fsck_msg_type msg_type = fsck_msg_type(msg_id, options);
|
|
|
|
int result;
|
2015-06-22 15:25:09 +00:00
|
|
|
|
2015-06-22 15:26:48 +00:00
|
|
|
if (msg_type == FSCK_IGNORE)
|
|
|
|
return 0;
|
|
|
|
|
2019-10-18 04:59:15 +00:00
|
|
|
if (object_on_skiplist(options, oid))
|
2015-06-22 15:27:18 +00:00
|
|
|
return 0;
|
|
|
|
|
2015-06-22 15:26:42 +00:00
|
|
|
if (msg_type == FSCK_FATAL)
|
|
|
|
msg_type = FSCK_ERROR;
|
2015-06-22 15:26:54 +00:00
|
|
|
else if (msg_type == FSCK_INFO)
|
|
|
|
msg_type = FSCK_WARN;
|
2015-06-22 15:26:42 +00:00
|
|
|
|
2021-03-28 13:15:37 +00:00
|
|
|
prepare_msg_ids();
|
2021-03-28 13:15:38 +00:00
|
|
|
strbuf_addf(&sb, "%s: ", msg_id_info[msg_id].camelcased);
|
2015-06-22 15:25:52 +00:00
|
|
|
|
2015-06-22 15:25:09 +00:00
|
|
|
va_start(ap, fmt);
|
|
|
|
strbuf_vaddf(&sb, fmt, ap);
|
2019-10-18 04:59:15 +00:00
|
|
|
result = options->error_func(options, oid, object_type,
|
2021-03-28 13:15:46 +00:00
|
|
|
msg_type, msg_id, sb.buf);
|
2015-06-22 15:25:09 +00:00
|
|
|
strbuf_release(&sb);
|
|
|
|
va_end(ap);
|
|
|
|
|
|
|
|
return result;
|
|
|
|
}
|
|
|
|
|
fsck: unify object-name code
Commit 90cf590f53 (fsck: optionally show more helpful info for broken
links, 2016-07-17) added a system for decorating objects with names. The
code is split across builtin/fsck.c (which gives the initial names) and
fsck.c (which adds to the names as it traverses the object graph). This
leads to some duplication, where both sites have near-identical
describe_object() functions (the difference being that the one in
builtin/fsck.c uses a circular array of buffers to allow multiple calls
in a single printf).
Let's provide a unified object_name API for fsck. That lets us drop the
duplication, as well as making the interface boundaries more clear
(which will let us refactor the implementation more in a future patch).
We'll leave describe_object() in builtin/fsck.c as a thin wrapper around
the new API, as it relies on a static global to make its many callers a
bit shorter.
We'll also convert the bare add_decoration() calls in builtin/fsck.c to
put_object_name(). This fixes two minor bugs:
1. We leak many small strings. add_decoration() has a last-one-wins
approach: it updates the decoration to the new string and returns
the old one. But we ignore the return value, leaking the old
string. This is quite common to trigger, since we look at reflogs:
the tip of any ref will be described both by looking at the actual
ref, as well as the latest reflog entry. So we'd always end up
leaking one of those strings.
2. The last-one-wins approach gives us lousy names. For instance, we
first look at all of the refs, and then all of the reflogs. So
rather than seeing "refs/heads/master", we're likely to overwrite
it with "HEAD@{12345678}". We're generally better off using the
first name we find.
And indeed, the test in t1450 expects this ugly HEAD@{} name. After
this patch, we've switched to using fsck_put_object_name()'s
first-one-wins semantics, and we output the more human-friendly
"refs/tags/julius" (and the test is updated accordingly).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-18 04:56:13 +00:00
|
|
|
void fsck_enable_object_names(struct fsck_options *options)
|
|
|
|
{
|
|
|
|
if (!options->object_names)
|
2019-10-18 04:57:37 +00:00
|
|
|
options->object_names = kh_init_oid_map();
|
fsck: unify object-name code
Commit 90cf590f53 (fsck: optionally show more helpful info for broken
links, 2016-07-17) added a system for decorating objects with names. The
code is split across builtin/fsck.c (which gives the initial names) and
fsck.c (which adds to the names as it traverses the object graph). This
leads to some duplication, where both sites have near-identical
describe_object() functions (the difference being that the one in
builtin/fsck.c uses a circular array of buffers to allow multiple calls
in a single printf).
Let's provide a unified object_name API for fsck. That lets us drop the
duplication, as well as making the interface boundaries more clear
(which will let us refactor the implementation more in a future patch).
We'll leave describe_object() in builtin/fsck.c as a thin wrapper around
the new API, as it relies on a static global to make its many callers a
bit shorter.
We'll also convert the bare add_decoration() calls in builtin/fsck.c to
put_object_name(). This fixes two minor bugs:
1. We leak many small strings. add_decoration() has a last-one-wins
approach: it updates the decoration to the new string and returns
the old one. But we ignore the return value, leaking the old
string. This is quite common to trigger, since we look at reflogs:
the tip of any ref will be described both by looking at the actual
ref, as well as the latest reflog entry. So we'd always end up
leaking one of those strings.
2. The last-one-wins approach gives us lousy names. For instance, we
first look at all of the refs, and then all of the reflogs. So
rather than seeing "refs/heads/master", we're likely to overwrite
it with "HEAD@{12345678}". We're generally better off using the
first name we find.
And indeed, the test in t1450 expects this ugly HEAD@{} name. After
this patch, we've switched to using fsck_put_object_name()'s
first-one-wins semantics, and we output the more human-friendly
"refs/tags/julius" (and the test is updated accordingly).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-18 04:56:13 +00:00
|
|
|
}
|
|
|
|
|
2019-10-18 04:57:37 +00:00
|
|
|
const char *fsck_get_object_name(struct fsck_options *options,
|
|
|
|
const struct object_id *oid)
|
2016-07-17 10:59:49 +00:00
|
|
|
{
|
2019-10-18 04:57:37 +00:00
|
|
|
khiter_t pos;
|
2016-07-17 10:59:49 +00:00
|
|
|
if (!options->object_names)
|
|
|
|
return NULL;
|
2019-10-18 04:57:37 +00:00
|
|
|
pos = kh_get_oid_map(options->object_names, *oid);
|
|
|
|
if (pos >= kh_end(options->object_names))
|
|
|
|
return NULL;
|
|
|
|
return kh_value(options->object_names, pos);
|
2016-07-17 10:59:49 +00:00
|
|
|
}
|
|
|
|
|
2019-10-18 04:57:37 +00:00
|
|
|
void fsck_put_object_name(struct fsck_options *options,
|
|
|
|
const struct object_id *oid,
|
fsck: unify object-name code
Commit 90cf590f53 (fsck: optionally show more helpful info for broken
links, 2016-07-17) added a system for decorating objects with names. The
code is split across builtin/fsck.c (which gives the initial names) and
fsck.c (which adds to the names as it traverses the object graph). This
leads to some duplication, where both sites have near-identical
describe_object() functions (the difference being that the one in
builtin/fsck.c uses a circular array of buffers to allow multiple calls
in a single printf).
Let's provide a unified object_name API for fsck. That lets us drop the
duplication, as well as making the interface boundaries more clear
(which will let us refactor the implementation more in a future patch).
We'll leave describe_object() in builtin/fsck.c as a thin wrapper around
the new API, as it relies on a static global to make its many callers a
bit shorter.
We'll also convert the bare add_decoration() calls in builtin/fsck.c to
put_object_name(). This fixes two minor bugs:
1. We leak many small strings. add_decoration() has a last-one-wins
approach: it updates the decoration to the new string and returns
the old one. But we ignore the return value, leaking the old
string. This is quite common to trigger, since we look at reflogs:
the tip of any ref will be described both by looking at the actual
ref, as well as the latest reflog entry. So we'd always end up
leaking one of those strings.
2. The last-one-wins approach gives us lousy names. For instance, we
first look at all of the refs, and then all of the reflogs. So
rather than seeing "refs/heads/master", we're likely to overwrite
it with "HEAD@{12345678}". We're generally better off using the
first name we find.
And indeed, the test in t1450 expects this ugly HEAD@{} name. After
this patch, we've switched to using fsck_put_object_name()'s
first-one-wins semantics, and we output the more human-friendly
"refs/tags/julius" (and the test is updated accordingly).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-18 04:56:13 +00:00
|
|
|
const char *fmt, ...)
|
2016-07-17 10:59:49 +00:00
|
|
|
{
|
|
|
|
va_list ap;
|
|
|
|
struct strbuf buf = STRBUF_INIT;
|
2019-10-18 04:57:37 +00:00
|
|
|
khiter_t pos;
|
|
|
|
int hashret;
|
2016-07-17 10:59:49 +00:00
|
|
|
|
|
|
|
if (!options->object_names)
|
|
|
|
return;
|
2019-10-18 04:57:37 +00:00
|
|
|
|
|
|
|
pos = kh_put_oid_map(options->object_names, *oid, &hashret);
|
|
|
|
if (!hashret)
|
2016-07-17 10:59:49 +00:00
|
|
|
return;
|
|
|
|
va_start(ap, fmt);
|
|
|
|
strbuf_vaddf(&buf, fmt, ap);
|
2019-10-18 04:57:37 +00:00
|
|
|
kh_value(options->object_names, pos) = strbuf_detach(&buf, NULL);
|
2016-07-17 10:59:49 +00:00
|
|
|
va_end(ap);
|
|
|
|
}
|
|
|
|
|
fsck: unify object-name code
Commit 90cf590f53 (fsck: optionally show more helpful info for broken
links, 2016-07-17) added a system for decorating objects with names. The
code is split across builtin/fsck.c (which gives the initial names) and
fsck.c (which adds to the names as it traverses the object graph). This
leads to some duplication, where both sites have near-identical
describe_object() functions (the difference being that the one in
builtin/fsck.c uses a circular array of buffers to allow multiple calls
in a single printf).
Let's provide a unified object_name API for fsck. That lets us drop the
duplication, as well as making the interface boundaries more clear
(which will let us refactor the implementation more in a future patch).
We'll leave describe_object() in builtin/fsck.c as a thin wrapper around
the new API, as it relies on a static global to make its many callers a
bit shorter.
We'll also convert the bare add_decoration() calls in builtin/fsck.c to
put_object_name(). This fixes two minor bugs:
1. We leak many small strings. add_decoration() has a last-one-wins
approach: it updates the decoration to the new string and returns
the old one. But we ignore the return value, leaking the old
string. This is quite common to trigger, since we look at reflogs:
the tip of any ref will be described both by looking at the actual
ref, as well as the latest reflog entry. So we'd always end up
leaking one of those strings.
2. The last-one-wins approach gives us lousy names. For instance, we
first look at all of the refs, and then all of the reflogs. So
rather than seeing "refs/heads/master", we're likely to overwrite
it with "HEAD@{12345678}". We're generally better off using the
first name we find.
And indeed, the test in t1450 expects this ugly HEAD@{} name. After
this patch, we've switched to using fsck_put_object_name()'s
first-one-wins semantics, and we output the more human-friendly
"refs/tags/julius" (and the test is updated accordingly).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-18 04:56:13 +00:00
|
|
|
const char *fsck_describe_object(struct fsck_options *options,
|
2019-10-18 04:57:37 +00:00
|
|
|
const struct object_id *oid)
|
2016-07-17 11:00:02 +00:00
|
|
|
{
|
fsck: unify object-name code
Commit 90cf590f53 (fsck: optionally show more helpful info for broken
links, 2016-07-17) added a system for decorating objects with names. The
code is split across builtin/fsck.c (which gives the initial names) and
fsck.c (which adds to the names as it traverses the object graph). This
leads to some duplication, where both sites have near-identical
describe_object() functions (the difference being that the one in
builtin/fsck.c uses a circular array of buffers to allow multiple calls
in a single printf).
Let's provide a unified object_name API for fsck. That lets us drop the
duplication, as well as making the interface boundaries more clear
(which will let us refactor the implementation more in a future patch).
We'll leave describe_object() in builtin/fsck.c as a thin wrapper around
the new API, as it relies on a static global to make its many callers a
bit shorter.
We'll also convert the bare add_decoration() calls in builtin/fsck.c to
put_object_name(). This fixes two minor bugs:
1. We leak many small strings. add_decoration() has a last-one-wins
approach: it updates the decoration to the new string and returns
the old one. But we ignore the return value, leaking the old
string. This is quite common to trigger, since we look at reflogs:
the tip of any ref will be described both by looking at the actual
ref, as well as the latest reflog entry. So we'd always end up
leaking one of those strings.
2. The last-one-wins approach gives us lousy names. For instance, we
first look at all of the refs, and then all of the reflogs. So
rather than seeing "refs/heads/master", we're likely to overwrite
it with "HEAD@{12345678}". We're generally better off using the
first name we find.
And indeed, the test in t1450 expects this ugly HEAD@{} name. After
this patch, we've switched to using fsck_put_object_name()'s
first-one-wins semantics, and we output the more human-friendly
"refs/tags/julius" (and the test is updated accordingly).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-18 04:56:13 +00:00
|
|
|
static struct strbuf bufs[] = {
|
|
|
|
STRBUF_INIT, STRBUF_INIT, STRBUF_INIT, STRBUF_INIT
|
|
|
|
};
|
|
|
|
static int b = 0;
|
|
|
|
struct strbuf *buf;
|
2019-10-18 04:57:37 +00:00
|
|
|
const char *name = fsck_get_object_name(options, oid);
|
fsck: unify object-name code
Commit 90cf590f53 (fsck: optionally show more helpful info for broken
links, 2016-07-17) added a system for decorating objects with names. The
code is split across builtin/fsck.c (which gives the initial names) and
fsck.c (which adds to the names as it traverses the object graph). This
leads to some duplication, where both sites have near-identical
describe_object() functions (the difference being that the one in
builtin/fsck.c uses a circular array of buffers to allow multiple calls
in a single printf).
Let's provide a unified object_name API for fsck. That lets us drop the
duplication, as well as making the interface boundaries more clear
(which will let us refactor the implementation more in a future patch).
We'll leave describe_object() in builtin/fsck.c as a thin wrapper around
the new API, as it relies on a static global to make its many callers a
bit shorter.
We'll also convert the bare add_decoration() calls in builtin/fsck.c to
put_object_name(). This fixes two minor bugs:
1. We leak many small strings. add_decoration() has a last-one-wins
approach: it updates the decoration to the new string and returns
the old one. But we ignore the return value, leaking the old
string. This is quite common to trigger, since we look at reflogs:
the tip of any ref will be described both by looking at the actual
ref, as well as the latest reflog entry. So we'd always end up
leaking one of those strings.
2. The last-one-wins approach gives us lousy names. For instance, we
first look at all of the refs, and then all of the reflogs. So
rather than seeing "refs/heads/master", we're likely to overwrite
it with "HEAD@{12345678}". We're generally better off using the
first name we find.
And indeed, the test in t1450 expects this ugly HEAD@{} name. After
this patch, we've switched to using fsck_put_object_name()'s
first-one-wins semantics, and we output the more human-friendly
"refs/tags/julius" (and the test is updated accordingly).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-18 04:56:13 +00:00
|
|
|
|
|
|
|
buf = bufs + b;
|
|
|
|
b = (b + 1) % ARRAY_SIZE(bufs);
|
|
|
|
strbuf_reset(buf);
|
2019-10-18 04:57:37 +00:00
|
|
|
strbuf_addstr(buf, oid_to_hex(oid));
|
fsck: unify object-name code
Commit 90cf590f53 (fsck: optionally show more helpful info for broken
links, 2016-07-17) added a system for decorating objects with names. The
code is split across builtin/fsck.c (which gives the initial names) and
fsck.c (which adds to the names as it traverses the object graph). This
leads to some duplication, where both sites have near-identical
describe_object() functions (the difference being that the one in
builtin/fsck.c uses a circular array of buffers to allow multiple calls
in a single printf).
Let's provide a unified object_name API for fsck. That lets us drop the
duplication, as well as making the interface boundaries more clear
(which will let us refactor the implementation more in a future patch).
We'll leave describe_object() in builtin/fsck.c as a thin wrapper around
the new API, as it relies on a static global to make its many callers a
bit shorter.
We'll also convert the bare add_decoration() calls in builtin/fsck.c to
put_object_name(). This fixes two minor bugs:
1. We leak many small strings. add_decoration() has a last-one-wins
approach: it updates the decoration to the new string and returns
the old one. But we ignore the return value, leaking the old
string. This is quite common to trigger, since we look at reflogs:
the tip of any ref will be described both by looking at the actual
ref, as well as the latest reflog entry. So we'd always end up
leaking one of those strings.
2. The last-one-wins approach gives us lousy names. For instance, we
first look at all of the refs, and then all of the reflogs. So
rather than seeing "refs/heads/master", we're likely to overwrite
it with "HEAD@{12345678}". We're generally better off using the
first name we find.
And indeed, the test in t1450 expects this ugly HEAD@{} name. After
this patch, we've switched to using fsck_put_object_name()'s
first-one-wins semantics, and we output the more human-friendly
"refs/tags/julius" (and the test is updated accordingly).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-18 04:56:13 +00:00
|
|
|
if (name)
|
|
|
|
strbuf_addf(buf, " (%s)", name);
|
2016-07-17 11:00:02 +00:00
|
|
|
|
fsck: unify object-name code
Commit 90cf590f53 (fsck: optionally show more helpful info for broken
links, 2016-07-17) added a system for decorating objects with names. The
code is split across builtin/fsck.c (which gives the initial names) and
fsck.c (which adds to the names as it traverses the object graph). This
leads to some duplication, where both sites have near-identical
describe_object() functions (the difference being that the one in
builtin/fsck.c uses a circular array of buffers to allow multiple calls
in a single printf).
Let's provide a unified object_name API for fsck. That lets us drop the
duplication, as well as making the interface boundaries more clear
(which will let us refactor the implementation more in a future patch).
We'll leave describe_object() in builtin/fsck.c as a thin wrapper around
the new API, as it relies on a static global to make its many callers a
bit shorter.
We'll also convert the bare add_decoration() calls in builtin/fsck.c to
put_object_name(). This fixes two minor bugs:
1. We leak many small strings. add_decoration() has a last-one-wins
approach: it updates the decoration to the new string and returns
the old one. But we ignore the return value, leaking the old
string. This is quite common to trigger, since we look at reflogs:
the tip of any ref will be described both by looking at the actual
ref, as well as the latest reflog entry. So we'd always end up
leaking one of those strings.
2. The last-one-wins approach gives us lousy names. For instance, we
first look at all of the refs, and then all of the reflogs. So
rather than seeing "refs/heads/master", we're likely to overwrite
it with "HEAD@{12345678}". We're generally better off using the
first name we find.
And indeed, the test in t1450 expects this ugly HEAD@{} name. After
this patch, we've switched to using fsck_put_object_name()'s
first-one-wins semantics, and we output the more human-friendly
"refs/tags/julius" (and the test is updated accordingly).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-18 04:56:13 +00:00
|
|
|
return buf->buf;
|
2016-07-17 11:00:02 +00:00
|
|
|
}
|
|
|
|
|
2015-06-22 15:25:00 +00:00
|
|
|
static int fsck_walk_tree(struct tree *tree, void *data, struct fsck_options *options)
|
2008-02-25 21:46:04 +00:00
|
|
|
{
|
|
|
|
struct tree_desc desc;
|
|
|
|
struct name_entry entry;
|
|
|
|
int res = 0;
|
2016-07-17 10:59:49 +00:00
|
|
|
const char *name;
|
2008-02-25 21:46:04 +00:00
|
|
|
|
|
|
|
if (parse_tree(tree))
|
|
|
|
return -1;
|
|
|
|
|
2019-10-18 04:57:37 +00:00
|
|
|
name = fsck_get_object_name(options, &tree->object.oid);
|
tree-walk: add a mechanism for getting non-canonicalized modes
When using init_tree_desc() and tree_entry() to iterate over a tree, we
always canonicalize the modes coming out of the tree. This is a good
thing to prevent bugs or oddities in normal code paths, but it's
counter-productive for tools like fsck that want to see the exact
contents.
We can address this by adding an option to avoid the extra
canonicalization. A few notes on the implementation:
- I've attached the new option to the tree_desc struct itself. The
actual code change is in decode_tree_entry(), which is in turn
called by the public update_tree_entry(), tree_entry(), and
init_tree_desc() functions, plus their "gently" counterparts.
By letting it ride along in the struct, we can avoid changing the
signature of those functions, which are called many times. Plus it's
conceptually simpler: you really want a particular iteration of a
tree to be "raw" or not, rather than individual calls.
- We still have to set the new option somewhere. The struct is
initialized by init_tree_desc(). I added the new flags field only to
the "gently" version. That avoids disturbing the much more numerous
non-gentle callers, and it makes sense that anybody being careful
about looking at raw modes would also be careful about bogus trees
(i.e., the caller will be something like fsck in the first place).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2022-08-10 21:01:17 +00:00
|
|
|
if (init_tree_desc_gently(&desc, tree->buffer, tree->size, 0))
|
2016-09-27 20:59:51 +00:00
|
|
|
return -1;
|
|
|
|
while (tree_entry_gently(&desc, &entry)) {
|
2016-07-17 10:59:49 +00:00
|
|
|
struct object *obj;
|
2008-02-25 21:46:04 +00:00
|
|
|
int result;
|
|
|
|
|
|
|
|
if (S_ISGITLINK(entry.mode))
|
|
|
|
continue;
|
2016-07-17 10:59:49 +00:00
|
|
|
|
|
|
|
if (S_ISDIR(entry.mode)) {
|
2019-01-15 00:39:44 +00:00
|
|
|
obj = (struct object *)lookup_tree(the_repository, &entry.oid);
|
2017-10-05 19:41:26 +00:00
|
|
|
if (name && obj)
|
2019-10-18 04:57:37 +00:00
|
|
|
fsck_put_object_name(options, &entry.oid, "%s%s/",
|
fsck: unify object-name code
Commit 90cf590f53 (fsck: optionally show more helpful info for broken
links, 2016-07-17) added a system for decorating objects with names. The
code is split across builtin/fsck.c (which gives the initial names) and
fsck.c (which adds to the names as it traverses the object graph). This
leads to some duplication, where both sites have near-identical
describe_object() functions (the difference being that the one in
builtin/fsck.c uses a circular array of buffers to allow multiple calls
in a single printf).
Let's provide a unified object_name API for fsck. That lets us drop the
duplication, as well as making the interface boundaries more clear
(which will let us refactor the implementation more in a future patch).
We'll leave describe_object() in builtin/fsck.c as a thin wrapper around
the new API, as it relies on a static global to make its many callers a
bit shorter.
We'll also convert the bare add_decoration() calls in builtin/fsck.c to
put_object_name(). This fixes two minor bugs:
1. We leak many small strings. add_decoration() has a last-one-wins
approach: it updates the decoration to the new string and returns
the old one. But we ignore the return value, leaking the old
string. This is quite common to trigger, since we look at reflogs:
the tip of any ref will be described both by looking at the actual
ref, as well as the latest reflog entry. So we'd always end up
leaking one of those strings.
2. The last-one-wins approach gives us lousy names. For instance, we
first look at all of the refs, and then all of the reflogs. So
rather than seeing "refs/heads/master", we're likely to overwrite
it with "HEAD@{12345678}". We're generally better off using the
first name we find.
And indeed, the test in t1450 expects this ugly HEAD@{} name. After
this patch, we've switched to using fsck_put_object_name()'s
first-one-wins semantics, and we output the more human-friendly
"refs/tags/julius" (and the test is updated accordingly).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-18 04:56:13 +00:00
|
|
|
name, entry.path);
|
2016-07-17 10:59:49 +00:00
|
|
|
result = options->walk(obj, OBJ_TREE, data, options);
|
|
|
|
}
|
|
|
|
else if (S_ISREG(entry.mode) || S_ISLNK(entry.mode)) {
|
2019-01-15 00:39:44 +00:00
|
|
|
obj = (struct object *)lookup_blob(the_repository, &entry.oid);
|
2017-10-05 19:41:26 +00:00
|
|
|
if (name && obj)
|
2019-10-18 04:57:37 +00:00
|
|
|
fsck_put_object_name(options, &entry.oid, "%s%s",
|
fsck: unify object-name code
Commit 90cf590f53 (fsck: optionally show more helpful info for broken
links, 2016-07-17) added a system for decorating objects with names. The
code is split across builtin/fsck.c (which gives the initial names) and
fsck.c (which adds to the names as it traverses the object graph). This
leads to some duplication, where both sites have near-identical
describe_object() functions (the difference being that the one in
builtin/fsck.c uses a circular array of buffers to allow multiple calls
in a single printf).
Let's provide a unified object_name API for fsck. That lets us drop the
duplication, as well as making the interface boundaries more clear
(which will let us refactor the implementation more in a future patch).
We'll leave describe_object() in builtin/fsck.c as a thin wrapper around
the new API, as it relies on a static global to make its many callers a
bit shorter.
We'll also convert the bare add_decoration() calls in builtin/fsck.c to
put_object_name(). This fixes two minor bugs:
1. We leak many small strings. add_decoration() has a last-one-wins
approach: it updates the decoration to the new string and returns
the old one. But we ignore the return value, leaking the old
string. This is quite common to trigger, since we look at reflogs:
the tip of any ref will be described both by looking at the actual
ref, as well as the latest reflog entry. So we'd always end up
leaking one of those strings.
2. The last-one-wins approach gives us lousy names. For instance, we
first look at all of the refs, and then all of the reflogs. So
rather than seeing "refs/heads/master", we're likely to overwrite
it with "HEAD@{12345678}". We're generally better off using the
first name we find.
And indeed, the test in t1450 expects this ugly HEAD@{} name. After
this patch, we've switched to using fsck_put_object_name()'s
first-one-wins semantics, and we output the more human-friendly
"refs/tags/julius" (and the test is updated accordingly).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-18 04:56:13 +00:00
|
|
|
name, entry.path);
|
2016-07-17 10:59:49 +00:00
|
|
|
result = options->walk(obj, OBJ_BLOB, data, options);
|
|
|
|
}
|
2008-02-25 21:46:04 +00:00
|
|
|
else {
|
2012-04-30 00:28:45 +00:00
|
|
|
result = error("in tree %s: entry %s has bad mode %.6o",
|
2019-10-18 04:57:37 +00:00
|
|
|
fsck_describe_object(options, &tree->object.oid),
|
fsck: unify object-name code
Commit 90cf590f53 (fsck: optionally show more helpful info for broken
links, 2016-07-17) added a system for decorating objects with names. The
code is split across builtin/fsck.c (which gives the initial names) and
fsck.c (which adds to the names as it traverses the object graph). This
leads to some duplication, where both sites have near-identical
describe_object() functions (the difference being that the one in
builtin/fsck.c uses a circular array of buffers to allow multiple calls
in a single printf).
Let's provide a unified object_name API for fsck. That lets us drop the
duplication, as well as making the interface boundaries more clear
(which will let us refactor the implementation more in a future patch).
We'll leave describe_object() in builtin/fsck.c as a thin wrapper around
the new API, as it relies on a static global to make its many callers a
bit shorter.
We'll also convert the bare add_decoration() calls in builtin/fsck.c to
put_object_name(). This fixes two minor bugs:
1. We leak many small strings. add_decoration() has a last-one-wins
approach: it updates the decoration to the new string and returns
the old one. But we ignore the return value, leaking the old
string. This is quite common to trigger, since we look at reflogs:
the tip of any ref will be described both by looking at the actual
ref, as well as the latest reflog entry. So we'd always end up
leaking one of those strings.
2. The last-one-wins approach gives us lousy names. For instance, we
first look at all of the refs, and then all of the reflogs. So
rather than seeing "refs/heads/master", we're likely to overwrite
it with "HEAD@{12345678}". We're generally better off using the
first name we find.
And indeed, the test in t1450 expects this ugly HEAD@{} name. After
this patch, we've switched to using fsck_put_object_name()'s
first-one-wins semantics, and we output the more human-friendly
"refs/tags/julius" (and the test is updated accordingly).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-18 04:56:13 +00:00
|
|
|
entry.path, entry.mode);
|
2008-02-25 21:46:04 +00:00
|
|
|
}
|
|
|
|
if (result < 0)
|
|
|
|
return result;
|
|
|
|
if (!res)
|
|
|
|
res = result;
|
|
|
|
}
|
|
|
|
return res;
|
|
|
|
}
|
|
|
|
|
2015-06-22 15:25:00 +00:00
|
|
|
static int fsck_walk_commit(struct commit *commit, void *data, struct fsck_options *options)
|
2008-02-25 21:46:04 +00:00
|
|
|
{
|
2016-07-17 10:59:49 +00:00
|
|
|
int counter = 0, generation = 0, name_prefix_len = 0;
|
2008-02-25 21:46:04 +00:00
|
|
|
struct commit_list *parents;
|
|
|
|
int res;
|
|
|
|
int result;
|
2016-07-17 10:59:49 +00:00
|
|
|
const char *name;
|
2008-02-25 21:46:04 +00:00
|
|
|
|
2023-03-28 13:58:48 +00:00
|
|
|
if (repo_parse_commit(the_repository, commit))
|
2008-02-25 21:46:04 +00:00
|
|
|
return -1;
|
|
|
|
|
2019-10-18 04:57:37 +00:00
|
|
|
name = fsck_get_object_name(options, &commit->object.oid);
|
2016-07-17 10:59:49 +00:00
|
|
|
if (name)
|
2019-10-18 04:57:37 +00:00
|
|
|
fsck_put_object_name(options, get_commit_tree_oid(commit),
|
fsck: unify object-name code
Commit 90cf590f53 (fsck: optionally show more helpful info for broken
links, 2016-07-17) added a system for decorating objects with names. The
code is split across builtin/fsck.c (which gives the initial names) and
fsck.c (which adds to the names as it traverses the object graph). This
leads to some duplication, where both sites have near-identical
describe_object() functions (the difference being that the one in
builtin/fsck.c uses a circular array of buffers to allow multiple calls
in a single printf).
Let's provide a unified object_name API for fsck. That lets us drop the
duplication, as well as making the interface boundaries more clear
(which will let us refactor the implementation more in a future patch).
We'll leave describe_object() in builtin/fsck.c as a thin wrapper around
the new API, as it relies on a static global to make its many callers a
bit shorter.
We'll also convert the bare add_decoration() calls in builtin/fsck.c to
put_object_name(). This fixes two minor bugs:
1. We leak many small strings. add_decoration() has a last-one-wins
approach: it updates the decoration to the new string and returns
the old one. But we ignore the return value, leaking the old
string. This is quite common to trigger, since we look at reflogs:
the tip of any ref will be described both by looking at the actual
ref, as well as the latest reflog entry. So we'd always end up
leaking one of those strings.
2. The last-one-wins approach gives us lousy names. For instance, we
first look at all of the refs, and then all of the reflogs. So
rather than seeing "refs/heads/master", we're likely to overwrite
it with "HEAD@{12345678}". We're generally better off using the
first name we find.
And indeed, the test in t1450 expects this ugly HEAD@{} name. After
this patch, we've switched to using fsck_put_object_name()'s
first-one-wins semantics, and we output the more human-friendly
"refs/tags/julius" (and the test is updated accordingly).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-18 04:56:13 +00:00
|
|
|
"%s:", name);
|
2016-07-17 10:59:49 +00:00
|
|
|
|
2023-03-28 13:58:48 +00:00
|
|
|
result = options->walk((struct object *) repo_get_commit_tree(the_repository, commit),
|
2018-04-06 19:09:38 +00:00
|
|
|
OBJ_TREE, data, options);
|
2008-02-25 21:46:04 +00:00
|
|
|
if (result < 0)
|
|
|
|
return result;
|
|
|
|
res = result;
|
|
|
|
|
|
|
|
parents = commit->parents;
|
2016-07-17 10:59:49 +00:00
|
|
|
if (name && parents) {
|
|
|
|
int len = strlen(name), power;
|
|
|
|
|
|
|
|
if (len && name[len - 1] == '^') {
|
|
|
|
generation = 1;
|
|
|
|
name_prefix_len = len - 1;
|
|
|
|
}
|
|
|
|
else { /* parse ~<generation> suffix */
|
|
|
|
for (generation = 0, power = 1;
|
|
|
|
len && isdigit(name[len - 1]);
|
|
|
|
power *= 10)
|
|
|
|
generation += power * (name[--len] - '0');
|
|
|
|
if (power > 1 && len && name[len - 1] == '~')
|
|
|
|
name_prefix_len = len - 1;
|
2021-02-10 18:01:30 +00:00
|
|
|
else {
|
|
|
|
/* Maybe a non-first parent, e.g. HEAD^2 */
|
|
|
|
generation = 0;
|
|
|
|
name_prefix_len = len;
|
|
|
|
}
|
2016-07-17 10:59:49 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2008-02-25 21:46:04 +00:00
|
|
|
while (parents) {
|
2016-07-17 10:59:49 +00:00
|
|
|
if (name) {
|
2019-10-18 04:57:37 +00:00
|
|
|
struct object_id *oid = &parents->item->object.oid;
|
2016-07-17 10:59:49 +00:00
|
|
|
|
2018-10-24 01:25:12 +00:00
|
|
|
if (counter++)
|
2019-10-18 04:57:37 +00:00
|
|
|
fsck_put_object_name(options, oid, "%s^%d",
|
fsck: unify object-name code
Commit 90cf590f53 (fsck: optionally show more helpful info for broken
links, 2016-07-17) added a system for decorating objects with names. The
code is split across builtin/fsck.c (which gives the initial names) and
fsck.c (which adds to the names as it traverses the object graph). This
leads to some duplication, where both sites have near-identical
describe_object() functions (the difference being that the one in
builtin/fsck.c uses a circular array of buffers to allow multiple calls
in a single printf).
Let's provide a unified object_name API for fsck. That lets us drop the
duplication, as well as making the interface boundaries more clear
(which will let us refactor the implementation more in a future patch).
We'll leave describe_object() in builtin/fsck.c as a thin wrapper around
the new API, as it relies on a static global to make its many callers a
bit shorter.
We'll also convert the bare add_decoration() calls in builtin/fsck.c to
put_object_name(). This fixes two minor bugs:
1. We leak many small strings. add_decoration() has a last-one-wins
approach: it updates the decoration to the new string and returns
the old one. But we ignore the return value, leaking the old
string. This is quite common to trigger, since we look at reflogs:
the tip of any ref will be described both by looking at the actual
ref, as well as the latest reflog entry. So we'd always end up
leaking one of those strings.
2. The last-one-wins approach gives us lousy names. For instance, we
first look at all of the refs, and then all of the reflogs. So
rather than seeing "refs/heads/master", we're likely to overwrite
it with "HEAD@{12345678}". We're generally better off using the
first name we find.
And indeed, the test in t1450 expects this ugly HEAD@{} name. After
this patch, we've switched to using fsck_put_object_name()'s
first-one-wins semantics, and we output the more human-friendly
"refs/tags/julius" (and the test is updated accordingly).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-18 04:56:13 +00:00
|
|
|
name, counter);
|
2016-07-17 10:59:49 +00:00
|
|
|
else if (generation > 0)
|
2019-10-18 04:57:37 +00:00
|
|
|
fsck_put_object_name(options, oid, "%.*s~%d",
|
fsck: unify object-name code
Commit 90cf590f53 (fsck: optionally show more helpful info for broken
links, 2016-07-17) added a system for decorating objects with names. The
code is split across builtin/fsck.c (which gives the initial names) and
fsck.c (which adds to the names as it traverses the object graph). This
leads to some duplication, where both sites have near-identical
describe_object() functions (the difference being that the one in
builtin/fsck.c uses a circular array of buffers to allow multiple calls
in a single printf).
Let's provide a unified object_name API for fsck. That lets us drop the
duplication, as well as making the interface boundaries more clear
(which will let us refactor the implementation more in a future patch).
We'll leave describe_object() in builtin/fsck.c as a thin wrapper around
the new API, as it relies on a static global to make its many callers a
bit shorter.
We'll also convert the bare add_decoration() calls in builtin/fsck.c to
put_object_name(). This fixes two minor bugs:
1. We leak many small strings. add_decoration() has a last-one-wins
approach: it updates the decoration to the new string and returns
the old one. But we ignore the return value, leaking the old
string. This is quite common to trigger, since we look at reflogs:
the tip of any ref will be described both by looking at the actual
ref, as well as the latest reflog entry. So we'd always end up
leaking one of those strings.
2. The last-one-wins approach gives us lousy names. For instance, we
first look at all of the refs, and then all of the reflogs. So
rather than seeing "refs/heads/master", we're likely to overwrite
it with "HEAD@{12345678}". We're generally better off using the
first name we find.
And indeed, the test in t1450 expects this ugly HEAD@{} name. After
this patch, we've switched to using fsck_put_object_name()'s
first-one-wins semantics, and we output the more human-friendly
"refs/tags/julius" (and the test is updated accordingly).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-18 04:56:13 +00:00
|
|
|
name_prefix_len, name,
|
|
|
|
generation + 1);
|
2016-07-17 10:59:49 +00:00
|
|
|
else
|
2019-10-18 04:57:37 +00:00
|
|
|
fsck_put_object_name(options, oid, "%s^", name);
|
2016-07-17 10:59:49 +00:00
|
|
|
}
|
2015-06-22 15:25:00 +00:00
|
|
|
result = options->walk((struct object *)parents->item, OBJ_COMMIT, data, options);
|
2008-02-25 21:46:04 +00:00
|
|
|
if (result < 0)
|
|
|
|
return result;
|
|
|
|
if (!res)
|
|
|
|
res = result;
|
|
|
|
parents = parents->next;
|
|
|
|
}
|
|
|
|
return res;
|
|
|
|
}
|
|
|
|
|
2015-06-22 15:25:00 +00:00
|
|
|
static int fsck_walk_tag(struct tag *tag, void *data, struct fsck_options *options)
|
2008-02-25 21:46:04 +00:00
|
|
|
{
|
2019-10-18 04:57:37 +00:00
|
|
|
const char *name = fsck_get_object_name(options, &tag->object.oid);
|
2016-07-17 10:59:49 +00:00
|
|
|
|
2008-02-25 21:46:04 +00:00
|
|
|
if (parse_tag(tag))
|
|
|
|
return -1;
|
2016-07-17 10:59:49 +00:00
|
|
|
if (name)
|
2019-10-18 04:57:37 +00:00
|
|
|
fsck_put_object_name(options, &tag->tagged->oid, "%s", name);
|
2015-06-22 15:25:00 +00:00
|
|
|
return options->walk(tag->tagged, OBJ_ANY, data, options);
|
2008-02-25 21:46:04 +00:00
|
|
|
}
|
|
|
|
|
2015-06-22 15:25:00 +00:00
|
|
|
int fsck_walk(struct object *obj, void *data, struct fsck_options *options)
|
2008-02-25 21:46:04 +00:00
|
|
|
{
|
|
|
|
if (!obj)
|
|
|
|
return -1;
|
fsck: lazily load types under --connectivity-only
The recent fixes to "fsck --connectivity-only" load all of
the objects with their correct types. This keeps the
connectivity-only code path close to the regular one, but it
also introduces some unnecessary inefficiency. While getting
the type of an object is cheap compared to actually opening
and parsing the object (as the non-connectivity-only case
would do), it's still not free.
For reachable non-blob objects, we end up having to parse
them later anyway (to see what they point to), making our
type lookup here redundant.
For unreachable objects, we might never hit them at all in
the reachability traversal, making the lookup completely
wasted. And in some cases, we might have quite a few
unreachable objects (e.g., when alternates are used for
shared object storage between repositories, it's normal for
there to be objects reachable from other repositories but
not the one running fsck).
The comment in mark_object_for_connectivity() claims two
benefits to getting the type up front:
1. We need to know the types during fsck_walk(). (And not
explicitly mentioned, but we also need them when
printing the types of broken or dangling commits).
We can address this by lazy-loading the types as
necessary. Most objects never need this lazy-load at
all, because they fall into one of these categories:
a. Reachable from our tips, and are coerced into the
correct type as we traverse (e.g., a parent link
will call lookup_commit(), which converts OBJ_NONE
to OBJ_COMMIT).
b. Unreachable, but not at the tip of a chunk of
unreachable history. We only mention the tips as
"dangling", so an unreachable commit which links
to hundreds of other objects needs only report the
type of the tip commit.
2. It serves as a cross-check that the coercion in (1a) is
correct (i.e., we'll complain about a parent link that
points to a blob). But we get most of this for free
already, because right after coercing, we'll parse any
non-blob objects. So we'd notice then if we expected a
commit and got a blob.
The one exception is when we expect a blob, in which
case we never actually read the object contents.
So this is a slight weakening, but given that the whole
point of --connectivity-only is to sacrifice some data
integrity checks for speed, this seems like an
acceptable tradeoff.
Here are before and after timings for an extreme case with
~5M reachable objects and another ~12M unreachable (it's the
torvalds/linux repository on GitHub, connected to shared
storage for all of the other kernel forks):
[before]
$ time git fsck --no-dangling --connectivity-only
real 3m4.323s
user 1m25.121s
sys 1m38.710s
[after]
$ time git fsck --no-dangling --connectivity-only
real 0m51.497s
user 0m49.575s
sys 0m1.776s
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-01-26 04:12:07 +00:00
|
|
|
|
|
|
|
if (obj->type == OBJ_NONE)
|
2018-06-29 01:21:51 +00:00
|
|
|
parse_object(the_repository, &obj->oid);
|
fsck: lazily load types under --connectivity-only
The recent fixes to "fsck --connectivity-only" load all of
the objects with their correct types. This keeps the
connectivity-only code path close to the regular one, but it
also introduces some unnecessary inefficiency. While getting
the type of an object is cheap compared to actually opening
and parsing the object (as the non-connectivity-only case
would do), it's still not free.
For reachable non-blob objects, we end up having to parse
them later anyway (to see what they point to), making our
type lookup here redundant.
For unreachable objects, we might never hit them at all in
the reachability traversal, making the lookup completely
wasted. And in some cases, we might have quite a few
unreachable objects (e.g., when alternates are used for
shared object storage between repositories, it's normal for
there to be objects reachable from other repositories but
not the one running fsck).
The comment in mark_object_for_connectivity() claims two
benefits to getting the type up front:
1. We need to know the types during fsck_walk(). (And not
explicitly mentioned, but we also need them when
printing the types of broken or dangling commits).
We can address this by lazy-loading the types as
necessary. Most objects never need this lazy-load at
all, because they fall into one of these categories:
a. Reachable from our tips, and are coerced into the
correct type as we traverse (e.g., a parent link
will call lookup_commit(), which converts OBJ_NONE
to OBJ_COMMIT).
b. Unreachable, but not at the tip of a chunk of
unreachable history. We only mention the tips as
"dangling", so an unreachable commit which links
to hundreds of other objects needs only report the
type of the tip commit.
2. It serves as a cross-check that the coercion in (1a) is
correct (i.e., we'll complain about a parent link that
points to a blob). But we get most of this for free
already, because right after coercing, we'll parse any
non-blob objects. So we'd notice then if we expected a
commit and got a blob.
The one exception is when we expect a blob, in which
case we never actually read the object contents.
So this is a slight weakening, but given that the whole
point of --connectivity-only is to sacrifice some data
integrity checks for speed, this seems like an
acceptable tradeoff.
Here are before and after timings for an extreme case with
~5M reachable objects and another ~12M unreachable (it's the
torvalds/linux repository on GitHub, connected to shared
storage for all of the other kernel forks):
[before]
$ time git fsck --no-dangling --connectivity-only
real 3m4.323s
user 1m25.121s
sys 1m38.710s
[after]
$ time git fsck --no-dangling --connectivity-only
real 0m51.497s
user 0m49.575s
sys 0m1.776s
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-01-26 04:12:07 +00:00
|
|
|
|
2008-02-25 21:46:04 +00:00
|
|
|
switch (obj->type) {
|
|
|
|
case OBJ_BLOB:
|
|
|
|
return 0;
|
|
|
|
case OBJ_TREE:
|
2015-06-22 15:25:00 +00:00
|
|
|
return fsck_walk_tree((struct tree *)obj, data, options);
|
2008-02-25 21:46:04 +00:00
|
|
|
case OBJ_COMMIT:
|
2015-06-22 15:25:00 +00:00
|
|
|
return fsck_walk_commit((struct commit *)obj, data, options);
|
2008-02-25 21:46:04 +00:00
|
|
|
case OBJ_TAG:
|
2015-06-22 15:25:00 +00:00
|
|
|
return fsck_walk_tag((struct tag *)obj, data, options);
|
2008-02-25 21:46:04 +00:00
|
|
|
default:
|
fsck: unify object-name code
Commit 90cf590f53 (fsck: optionally show more helpful info for broken
links, 2016-07-17) added a system for decorating objects with names. The
code is split across builtin/fsck.c (which gives the initial names) and
fsck.c (which adds to the names as it traverses the object graph). This
leads to some duplication, where both sites have near-identical
describe_object() functions (the difference being that the one in
builtin/fsck.c uses a circular array of buffers to allow multiple calls
in a single printf).
Let's provide a unified object_name API for fsck. That lets us drop the
duplication, as well as making the interface boundaries more clear
(which will let us refactor the implementation more in a future patch).
We'll leave describe_object() in builtin/fsck.c as a thin wrapper around
the new API, as it relies on a static global to make its many callers a
bit shorter.
We'll also convert the bare add_decoration() calls in builtin/fsck.c to
put_object_name(). This fixes two minor bugs:
1. We leak many small strings. add_decoration() has a last-one-wins
approach: it updates the decoration to the new string and returns
the old one. But we ignore the return value, leaking the old
string. This is quite common to trigger, since we look at reflogs:
the tip of any ref will be described both by looking at the actual
ref, as well as the latest reflog entry. So we'd always end up
leaking one of those strings.
2. The last-one-wins approach gives us lousy names. For instance, we
first look at all of the refs, and then all of the reflogs. So
rather than seeing "refs/heads/master", we're likely to overwrite
it with "HEAD@{12345678}". We're generally better off using the
first name we find.
And indeed, the test in t1450 expects this ugly HEAD@{} name. After
this patch, we've switched to using fsck_put_object_name()'s
first-one-wins semantics, and we output the more human-friendly
"refs/tags/julius" (and the test is updated accordingly).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-18 04:56:13 +00:00
|
|
|
error("Unknown object type for %s",
|
2019-10-18 04:57:37 +00:00
|
|
|
fsck_describe_object(options, &obj->oid));
|
2008-02-25 21:46:04 +00:00
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
}
|
2008-02-25 21:46:08 +00:00
|
|
|
|
2020-05-10 16:12:16 +00:00
|
|
|
struct name_stack {
|
|
|
|
const char **names;
|
|
|
|
size_t nr, alloc;
|
|
|
|
};
|
|
|
|
|
|
|
|
static void name_stack_push(struct name_stack *stack, const char *name)
|
|
|
|
{
|
|
|
|
ALLOC_GROW(stack->names, stack->nr + 1, stack->alloc);
|
|
|
|
stack->names[stack->nr++] = name;
|
|
|
|
}
|
|
|
|
|
|
|
|
static const char *name_stack_pop(struct name_stack *stack)
|
|
|
|
{
|
|
|
|
return stack->nr ? stack->names[--stack->nr] : NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
static void name_stack_clear(struct name_stack *stack)
|
|
|
|
{
|
|
|
|
FREE_AND_NULL(stack->names);
|
|
|
|
stack->nr = stack->alloc = 0;
|
|
|
|
}
|
|
|
|
|
2008-02-25 21:46:08 +00:00
|
|
|
/*
|
|
|
|
* The entries in a tree are ordered in the _path_ order,
|
|
|
|
* which means that a directory entry is ordered by adding
|
|
|
|
* a slash to the end of it.
|
|
|
|
*
|
|
|
|
* So a directory called "a" is ordered _after_ a file
|
|
|
|
* called "a.c", because "a/" sorts after "a.c".
|
|
|
|
*/
|
|
|
|
#define TREE_UNORDERED (-1)
|
|
|
|
#define TREE_HAS_DUPS (-2)
|
|
|
|
|
2020-05-10 16:12:16 +00:00
|
|
|
static int is_less_than_slash(unsigned char c)
|
|
|
|
{
|
|
|
|
return '\0' < c && c < '/';
|
|
|
|
}
|
|
|
|
|
|
|
|
static int verify_ordered(unsigned mode1, const char *name1,
|
|
|
|
unsigned mode2, const char *name2,
|
|
|
|
struct name_stack *candidates)
|
2008-02-25 21:46:08 +00:00
|
|
|
{
|
|
|
|
int len1 = strlen(name1);
|
|
|
|
int len2 = strlen(name2);
|
|
|
|
int len = len1 < len2 ? len1 : len2;
|
|
|
|
unsigned char c1, c2;
|
|
|
|
int cmp;
|
|
|
|
|
|
|
|
cmp = memcmp(name1, name2, len);
|
|
|
|
if (cmp < 0)
|
|
|
|
return 0;
|
|
|
|
if (cmp > 0)
|
|
|
|
return TREE_UNORDERED;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Ok, the first <len> characters are the same.
|
|
|
|
* Now we need to order the next one, but turn
|
|
|
|
* a '\0' into a '/' for a directory entry.
|
|
|
|
*/
|
|
|
|
c1 = name1[len];
|
|
|
|
c2 = name2[len];
|
|
|
|
if (!c1 && !c2)
|
|
|
|
/*
|
|
|
|
* git-write-tree used to write out a nonsense tree that has
|
|
|
|
* entries with the same name, one blob and one tree. Make
|
|
|
|
* sure we do not have duplicate entries.
|
|
|
|
*/
|
|
|
|
return TREE_HAS_DUPS;
|
|
|
|
if (!c1 && S_ISDIR(mode1))
|
|
|
|
c1 = '/';
|
|
|
|
if (!c2 && S_ISDIR(mode2))
|
|
|
|
c2 = '/';
|
2020-05-10 16:12:16 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* There can be non-consecutive duplicates due to the implicitly
|
2020-05-21 09:52:04 +00:00
|
|
|
* added slash, e.g.:
|
2020-05-10 16:12:16 +00:00
|
|
|
*
|
|
|
|
* foo
|
|
|
|
* foo.bar
|
|
|
|
* foo.bar.baz
|
|
|
|
* foo.bar/
|
|
|
|
* foo/
|
|
|
|
*
|
|
|
|
* Record non-directory candidates (like "foo" and "foo.bar" in
|
|
|
|
* the example) on a stack and check directory candidates (like
|
|
|
|
* foo/" and "foo.bar/") against that stack.
|
|
|
|
*/
|
|
|
|
if (!c1 && is_less_than_slash(c2)) {
|
|
|
|
name_stack_push(candidates, name1);
|
|
|
|
} else if (c2 == '/' && is_less_than_slash(c1)) {
|
|
|
|
for (;;) {
|
|
|
|
const char *p;
|
|
|
|
const char *f_name = name_stack_pop(candidates);
|
|
|
|
|
|
|
|
if (!f_name)
|
|
|
|
break;
|
|
|
|
if (!skip_prefix(name2, f_name, &p))
|
2020-05-21 09:52:54 +00:00
|
|
|
continue;
|
2020-05-10 16:12:16 +00:00
|
|
|
if (!*p)
|
|
|
|
return TREE_HAS_DUPS;
|
|
|
|
if (is_less_than_slash(*p)) {
|
|
|
|
name_stack_push(candidates, f_name);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2008-02-25 21:46:08 +00:00
|
|
|
return c1 < c2 ? 0 : TREE_UNORDERED;
|
|
|
|
}
|
|
|
|
|
2021-05-01 15:41:38 +00:00
|
|
|
static int fsck_tree(const struct object_id *tree_oid,
|
fsck: require an actual buffer for non-blobs
The fsck_object() function takes in a buffer, but also a "struct
object". The rules for using these vary between types:
- for a commit, we'll use the provided buffer; if it's NULL, we'll
fall back to get_commit_buffer(), which loads from either an
in-memory cache or from disk. If the latter fails, we'd die(), which
is non-ideal for fsck.
- for a tag, a NULL buffer will fall back to loading the object from
disk (and failure would lead to an fsck error)
- for a tree, we _never_ look at the provided buffer, and always use
tree->buffer
- for a blob, we usually don't look at the buffer at all, unless it
has been marked as a .gitmodule file. In that case we check the
buffer given to us, or assume a NULL buffer is a very large blob
(and complain about it)
This is much more complex than it needs to be. It turns out that nobody
ever feeds a NULL buffer that isn't a blob:
- git-fsck calls fsck_object() only from fsck_obj(). That in turn is
called by one of:
- fsck_obj_buffer(), which is a callback to verify_pack(), which
unpacks everything except large blobs into a buffer (see
pack-check.c, lines 131-141).
- fsck_loose(), which hits a BUG() on non-blobs with a NULL buffer
(builtin/fsck.c, lines 639-640)
And in either case, we'll have just called parse_object_buffer()
anyway, which would segfault on a NULL buffer for commits or tags
(not for trees, but it would install a NULL tree->buffer which would
later cause a segfault)
- git-index-pack asserts that the buffer is non-NULL unless the object
is a blob (see builtin/index-pack.c, line 832)
- git-unpack-objects always writes a non-NULL buffer into its
obj_buffer hash, which is then fed to fsck_object(). (There is
actually a funny thing here where it does not store blob buffers at
all, nor does it call fsck on them; it does check any needed blobs
via fsck_finish() though).
Let's make the rules simpler, which reduces the amount of code and gives
us more flexibility in refactoring the fsck code. The new rules are:
- only blobs are allowed to pass a NULL buffer
- we always use the provided buffer, never pulling information from
the object struct
We don't have to adjust any callers, because they were already adhering
to these. Note that we do drop a few fsck identifiers for missing tags,
but that was all dead code (because nobody passed a NULL tag buffer).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-18 04:54:12 +00:00
|
|
|
const char *buffer, unsigned long size,
|
|
|
|
struct fsck_options *options)
|
2008-02-25 21:46:08 +00:00
|
|
|
{
|
2016-09-27 20:59:51 +00:00
|
|
|
int retval = 0;
|
2012-07-28 15:06:29 +00:00
|
|
|
int has_null_sha1 = 0;
|
2008-02-25 21:46:08 +00:00
|
|
|
int has_full_path = 0;
|
|
|
|
int has_empty_name = 0;
|
2012-11-28 02:27:37 +00:00
|
|
|
int has_dot = 0;
|
|
|
|
int has_dotdot = 0;
|
2012-11-28 21:35:29 +00:00
|
|
|
int has_dotgit = 0;
|
2008-02-25 21:46:08 +00:00
|
|
|
int has_zero_pad = 0;
|
|
|
|
int has_bad_modes = 0;
|
|
|
|
int has_dup_entries = 0;
|
|
|
|
int not_properly_sorted = 0;
|
|
|
|
struct tree_desc desc;
|
|
|
|
unsigned o_mode;
|
|
|
|
const char *o_name;
|
2020-05-10 16:12:16 +00:00
|
|
|
struct name_stack df_dup_candidates = { NULL };
|
2008-02-25 21:46:08 +00:00
|
|
|
|
2022-08-10 21:02:45 +00:00
|
|
|
if (init_tree_desc_gently(&desc, buffer, size, TREE_DESC_RAW_MODES)) {
|
2021-05-01 15:41:43 +00:00
|
|
|
retval += report(options, tree_oid, OBJ_TREE,
|
|
|
|
FSCK_MSG_BAD_TREE,
|
|
|
|
"cannot be parsed as a tree");
|
2016-09-27 20:59:51 +00:00
|
|
|
return retval;
|
|
|
|
}
|
2008-02-25 21:46:08 +00:00
|
|
|
|
|
|
|
o_mode = 0;
|
|
|
|
o_name = NULL;
|
|
|
|
|
|
|
|
while (desc.size) {
|
2019-04-05 15:00:12 +00:00
|
|
|
unsigned short mode;
|
is_ntfs_dotgit(): only verify the leading segment
The config setting `core.protectNTFS` is specifically designed to work
not only on Windows, but anywhere, to allow for repositories hosted on,
say, Linux servers to be protected against NTFS-specific attack vectors.
As a consequence, `is_ntfs_dotgit()` manually splits backslash-separated
paths (but does not do the same for paths separated by forward slashes),
under the assumption that the backslash might not be a valid directory
separator on the _current_ Operating System.
However, the two callers, `verify_path()` and `fsck_tree()`, are
supposed to feed only individual path segments to the `is_ntfs_dotgit()`
function.
This causes a lot of duplicate scanning (and very inefficient scanning,
too, as the inner loop of `is_ntfs_dotgit()` was optimized for
readability rather than for speed.
Let's simplify the design of `is_ntfs_dotgit()` by putting the burden of
splitting the paths by backslashes as directory separators on the
callers of said function.
Consequently, the `verify_path()` function, which already splits the
path by directory separators, now treats backslashes as directory
separators _explicitly_ when `core.protectNTFS` is turned on, even on
platforms where the backslash is _not_ a directory separator.
Note that we have to repeat some code in `verify_path()`: if the
backslash is not a directory separator on the current Operating System,
we want to allow file names like `\`, but we _do_ want to disallow paths
that are clearly intended to cause harm when the repository is cloned on
Windows.
The `fsck_tree()` function (the other caller of `is_ntfs_dotgit()`) now
needs to look for backslashes in tree entries' names specifically when
`core.protectNTFS` is turned on. While it would be tempting to
completely disallow backslashes in that case (much like `fsck` reports
names containing forward slashes as "full paths"), this would be
overzealous: when `core.protectNTFS` is turned on in a non-Windows
setup, backslashes are perfectly valid characters in file names while we
_still_ want to disallow tree entries that are clearly designed to
exploit NTFS-specific behavior.
This simplification will make subsequent changes easier to implement,
such as turning `core.protectNTFS` on by default (not only on Windows)
or protecting against attack vectors involving NTFS Alternate Data
Streams.
Incidentally, this change allows for catching malicious repositories
that contain tree entries of the form `dir\.gitmodules` already on the
server side rather than only on the client side (and previously only on
Windows): in contrast to `is_ntfs_dotgit()`, the
`is_ntfs_dotgitmodules()` function already expects the caller to split
the paths by directory separators.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2019-09-23 06:58:11 +00:00
|
|
|
const char *name, *backslash;
|
2021-05-01 15:41:38 +00:00
|
|
|
const struct object_id *entry_oid;
|
2008-02-25 21:46:08 +00:00
|
|
|
|
2021-05-01 15:41:38 +00:00
|
|
|
entry_oid = tree_entry_extract(&desc, &name, &mode);
|
2008-02-25 21:46:08 +00:00
|
|
|
|
2021-05-01 15:41:38 +00:00
|
|
|
has_null_sha1 |= is_null_oid(entry_oid);
|
2014-03-19 23:02:04 +00:00
|
|
|
has_full_path |= !!strchr(name, '/');
|
|
|
|
has_empty_name |= !*name;
|
|
|
|
has_dot |= !strcmp(name, ".");
|
|
|
|
has_dotdot |= !strcmp(name, "..");
|
2018-05-13 16:35:37 +00:00
|
|
|
has_dotgit |= is_hfs_dotgit(name) || is_ntfs_dotgit(name);
|
2008-02-25 21:46:08 +00:00
|
|
|
has_zero_pad |= *(char *)desc.buffer == '0';
|
fsck: detect gitmodules files
In preparation for performing fsck checks on .gitmodules
files, this commit plumbs in the actual detection of the
files. Note that unlike most other fsck checks, this cannot
be a property of a single object: we must know that the
object is found at a ".gitmodules" path at the root tree of
a commit.
Since the fsck code only sees one object at a time, we have
to mark the related objects to fit the puzzle together. When
we see a commit we mark its tree as a root tree, and when
we see a root tree with a .gitmodules file, we mark the
corresponding blob to be checked.
In an ideal world, we'd check the objects in topological
order: commits followed by trees followed by blobs. In that
case we can avoid ever loading an object twice, since all
markings would be complete by the time we get to the marked
objects. And indeed, if we are checking a single packfile,
this is the order in which Git will generally write the
objects. But we can't count on that:
1. git-fsck may show us the objects in arbitrary order
(loose objects are fed in sha1 order, but we may also
have multiple packs, and we process each pack fully in
sequence).
2. The type ordering is just what git-pack-objects happens
to write now. The pack format does not require a
specific order, and it's possible that future versions
of Git (or a custom version trying to fool official
Git's fsck checks!) may order it differently.
3. We may not even be fscking all of the relevant objects
at once. Consider pushing with transfer.fsckObjects,
where one push adds a blob at path "foo", and then a
second push adds the same blob at path ".gitmodules".
The blob is not part of the second push at all, but we
need to mark and check it.
So in the general case, we need to make up to three passes
over the objects: once to make sure we've seen all commits,
then once to cover any trees we might have missed, and then
a final pass to cover any .gitmodules blobs we found in the
second pass.
We can simplify things a bit by loosening the requirement
that we find .gitmodules only at root trees. Technically
a file like "subdir/.gitmodules" is not parsed by Git, but
it's not unreasonable for us to declare that Git is aware of
all ".gitmodules" files and make them eligible for checking.
That lets us drop the root-tree requirement, which
eliminates one pass entirely. And it makes our worst case
much better: instead of potentially queueing every root tree
to be re-examined, the worst case is that we queue each
unique .gitmodules blob for a second look.
This patch just adds the boilerplate to find .gitmodules
files. The actual content checks will come in a subsequent
commit.
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 21:20:08 +00:00
|
|
|
|
2018-05-05 00:03:35 +00:00
|
|
|
if (is_hfs_dotgitmodules(name) || is_ntfs_dotgitmodules(name)) {
|
|
|
|
if (!S_ISLNK(mode))
|
2021-05-01 15:41:38 +00:00
|
|
|
oidset_insert(&options->gitmodules_found,
|
|
|
|
entry_oid);
|
2018-05-05 00:03:35 +00:00
|
|
|
else
|
2019-10-18 04:59:15 +00:00
|
|
|
retval += report(options,
|
2021-05-01 15:41:38 +00:00
|
|
|
tree_oid, OBJ_TREE,
|
2018-05-05 00:03:35 +00:00
|
|
|
FSCK_MSG_GITMODULES_SYMLINK,
|
|
|
|
".gitmodules is a symbolic link");
|
|
|
|
}
|
fsck: detect gitmodules files
In preparation for performing fsck checks on .gitmodules
files, this commit plumbs in the actual detection of the
files. Note that unlike most other fsck checks, this cannot
be a property of a single object: we must know that the
object is found at a ".gitmodules" path at the root tree of
a commit.
Since the fsck code only sees one object at a time, we have
to mark the related objects to fit the puzzle together. When
we see a commit we mark its tree as a root tree, and when
we see a root tree with a .gitmodules file, we mark the
corresponding blob to be checked.
In an ideal world, we'd check the objects in topological
order: commits followed by trees followed by blobs. In that
case we can avoid ever loading an object twice, since all
markings would be complete by the time we get to the marked
objects. And indeed, if we are checking a single packfile,
this is the order in which Git will generally write the
objects. But we can't count on that:
1. git-fsck may show us the objects in arbitrary order
(loose objects are fed in sha1 order, but we may also
have multiple packs, and we process each pack fully in
sequence).
2. The type ordering is just what git-pack-objects happens
to write now. The pack format does not require a
specific order, and it's possible that future versions
of Git (or a custom version trying to fool official
Git's fsck checks!) may order it differently.
3. We may not even be fscking all of the relevant objects
at once. Consider pushing with transfer.fsckObjects,
where one push adds a blob at path "foo", and then a
second push adds the same blob at path ".gitmodules".
The blob is not part of the second push at all, but we
need to mark and check it.
So in the general case, we need to make up to three passes
over the objects: once to make sure we've seen all commits,
then once to cover any trees we might have missed, and then
a final pass to cover any .gitmodules blobs we found in the
second pass.
We can simplify things a bit by loosening the requirement
that we find .gitmodules only at root trees. Technically
a file like "subdir/.gitmodules" is not parsed by Git, but
it's not unreasonable for us to declare that Git is aware of
all ".gitmodules" files and make them eligible for checking.
That lets us drop the root-tree requirement, which
eliminates one pass entirely. And it makes our worst case
much better: instead of potentially queueing every root tree
to be re-examined, the worst case is that we queue each
unique .gitmodules blob for a second look.
This patch just adds the boilerplate to find .gitmodules
files. The actual content checks will come in a subsequent
commit.
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 21:20:08 +00:00
|
|
|
|
2022-12-01 14:46:05 +00:00
|
|
|
if (is_hfs_dotgitattributes(name) || is_ntfs_dotgitattributes(name)) {
|
2022-12-01 14:46:09 +00:00
|
|
|
if (!S_ISLNK(mode))
|
|
|
|
oidset_insert(&options->gitattributes_found,
|
|
|
|
entry_oid);
|
|
|
|
else
|
2022-12-01 14:46:05 +00:00
|
|
|
retval += report(options, tree_oid, OBJ_TREE,
|
|
|
|
FSCK_MSG_GITATTRIBUTES_SYMLINK,
|
|
|
|
".gitattributes is a symlink");
|
|
|
|
}
|
|
|
|
|
fsck: warn about symlinked dotfiles we'll open with O_NOFOLLOW
In the commits merged in via 204333b015 (Merge branch
'jk/open-dotgitx-with-nofollow', 2021-03-22), we stopped following
symbolic links for .gitattributes, .gitignore, and .mailmap files.
Let's teach fsck to warn that these symlinks are not going to do
anything. Note that this is just a warning, and won't block the objects
via transfer.fsckObjects, since there are reported to be cases of this
in the wild (and even once fixed, they will continue to exist in the
commit history of those projects, but are not particularly dangerous).
Note that we won't add these to the existing gitmodules block in the
fsck code. The logic for gitmodules is a bit more complicated, as we
also check the content of non-symlink instances we find. But for these
new files, there is no content check; we're just looking at the name and
mode of the tree entry (and we can avoid even the complicated name
checks in the common case that the mode doesn't indicate a symlink).
We can reuse the test helper function we defined for .gitmodules, though
(it needs some slight adjustments for the fsck error code, and because
we don't block these symlinks via verify_path()).
Note that I didn't explicitly test the transfer.fsckObjects case here
(nor does the existing .gitmodules test that it blocks a push). The
translation of fsck severities to outcomes is covered in general in
t5504.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-05-03 20:43:25 +00:00
|
|
|
if (S_ISLNK(mode)) {
|
|
|
|
if (is_hfs_dotgitignore(name) ||
|
|
|
|
is_ntfs_dotgitignore(name))
|
|
|
|
retval += report(options, tree_oid, OBJ_TREE,
|
|
|
|
FSCK_MSG_GITIGNORE_SYMLINK,
|
|
|
|
".gitignore is a symlink");
|
|
|
|
if (is_hfs_dotmailmap(name) ||
|
|
|
|
is_ntfs_dotmailmap(name))
|
|
|
|
retval += report(options, tree_oid, OBJ_TREE,
|
|
|
|
FSCK_MSG_MAILMAP_SYMLINK,
|
|
|
|
".mailmap is a symlink");
|
|
|
|
}
|
|
|
|
|
is_ntfs_dotgit(): only verify the leading segment
The config setting `core.protectNTFS` is specifically designed to work
not only on Windows, but anywhere, to allow for repositories hosted on,
say, Linux servers to be protected against NTFS-specific attack vectors.
As a consequence, `is_ntfs_dotgit()` manually splits backslash-separated
paths (but does not do the same for paths separated by forward slashes),
under the assumption that the backslash might not be a valid directory
separator on the _current_ Operating System.
However, the two callers, `verify_path()` and `fsck_tree()`, are
supposed to feed only individual path segments to the `is_ntfs_dotgit()`
function.
This causes a lot of duplicate scanning (and very inefficient scanning,
too, as the inner loop of `is_ntfs_dotgit()` was optimized for
readability rather than for speed.
Let's simplify the design of `is_ntfs_dotgit()` by putting the burden of
splitting the paths by backslashes as directory separators on the
callers of said function.
Consequently, the `verify_path()` function, which already splits the
path by directory separators, now treats backslashes as directory
separators _explicitly_ when `core.protectNTFS` is turned on, even on
platforms where the backslash is _not_ a directory separator.
Note that we have to repeat some code in `verify_path()`: if the
backslash is not a directory separator on the current Operating System,
we want to allow file names like `\`, but we _do_ want to disallow paths
that are clearly intended to cause harm when the repository is cloned on
Windows.
The `fsck_tree()` function (the other caller of `is_ntfs_dotgit()`) now
needs to look for backslashes in tree entries' names specifically when
`core.protectNTFS` is turned on. While it would be tempting to
completely disallow backslashes in that case (much like `fsck` reports
names containing forward slashes as "full paths"), this would be
overzealous: when `core.protectNTFS` is turned on in a non-Windows
setup, backslashes are perfectly valid characters in file names while we
_still_ want to disallow tree entries that are clearly designed to
exploit NTFS-specific behavior.
This simplification will make subsequent changes easier to implement,
such as turning `core.protectNTFS` on by default (not only on Windows)
or protecting against attack vectors involving NTFS Alternate Data
Streams.
Incidentally, this change allows for catching malicious repositories
that contain tree entries of the form `dir\.gitmodules` already on the
server side rather than only on the client side (and previously only on
Windows): in contrast to `is_ntfs_dotgit()`, the
`is_ntfs_dotgitmodules()` function already expects the caller to split
the paths by directory separators.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2019-09-23 06:58:11 +00:00
|
|
|
if ((backslash = strchr(name, '\\'))) {
|
|
|
|
while (backslash) {
|
|
|
|
backslash++;
|
|
|
|
has_dotgit |= is_ntfs_dotgit(backslash);
|
2019-12-04 20:52:10 +00:00
|
|
|
if (is_ntfs_dotgitmodules(backslash)) {
|
|
|
|
if (!S_ISLNK(mode))
|
2021-05-01 15:41:38 +00:00
|
|
|
oidset_insert(&options->gitmodules_found,
|
|
|
|
entry_oid);
|
2019-12-04 20:52:10 +00:00
|
|
|
else
|
2021-05-01 15:41:38 +00:00
|
|
|
retval += report(options, tree_oid, OBJ_TREE,
|
2019-12-04 20:52:10 +00:00
|
|
|
FSCK_MSG_GITMODULES_SYMLINK,
|
|
|
|
".gitmodules is a symbolic link");
|
|
|
|
}
|
is_ntfs_dotgit(): only verify the leading segment
The config setting `core.protectNTFS` is specifically designed to work
not only on Windows, but anywhere, to allow for repositories hosted on,
say, Linux servers to be protected against NTFS-specific attack vectors.
As a consequence, `is_ntfs_dotgit()` manually splits backslash-separated
paths (but does not do the same for paths separated by forward slashes),
under the assumption that the backslash might not be a valid directory
separator on the _current_ Operating System.
However, the two callers, `verify_path()` and `fsck_tree()`, are
supposed to feed only individual path segments to the `is_ntfs_dotgit()`
function.
This causes a lot of duplicate scanning (and very inefficient scanning,
too, as the inner loop of `is_ntfs_dotgit()` was optimized for
readability rather than for speed.
Let's simplify the design of `is_ntfs_dotgit()` by putting the burden of
splitting the paths by backslashes as directory separators on the
callers of said function.
Consequently, the `verify_path()` function, which already splits the
path by directory separators, now treats backslashes as directory
separators _explicitly_ when `core.protectNTFS` is turned on, even on
platforms where the backslash is _not_ a directory separator.
Note that we have to repeat some code in `verify_path()`: if the
backslash is not a directory separator on the current Operating System,
we want to allow file names like `\`, but we _do_ want to disallow paths
that are clearly intended to cause harm when the repository is cloned on
Windows.
The `fsck_tree()` function (the other caller of `is_ntfs_dotgit()`) now
needs to look for backslashes in tree entries' names specifically when
`core.protectNTFS` is turned on. While it would be tempting to
completely disallow backslashes in that case (much like `fsck` reports
names containing forward slashes as "full paths"), this would be
overzealous: when `core.protectNTFS` is turned on in a non-Windows
setup, backslashes are perfectly valid characters in file names while we
_still_ want to disallow tree entries that are clearly designed to
exploit NTFS-specific behavior.
This simplification will make subsequent changes easier to implement,
such as turning `core.protectNTFS` on by default (not only on Windows)
or protecting against attack vectors involving NTFS Alternate Data
Streams.
Incidentally, this change allows for catching malicious repositories
that contain tree entries of the form `dir\.gitmodules` already on the
server side rather than only on the client side (and previously only on
Windows): in contrast to `is_ntfs_dotgit()`, the
`is_ntfs_dotgitmodules()` function already expects the caller to split
the paths by directory separators.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2019-09-23 06:58:11 +00:00
|
|
|
backslash = strchr(backslash, '\\');
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2016-09-27 20:59:51 +00:00
|
|
|
if (update_tree_entry_gently(&desc)) {
|
2021-05-01 15:41:43 +00:00
|
|
|
retval += report(options, tree_oid, OBJ_TREE,
|
|
|
|
FSCK_MSG_BAD_TREE,
|
|
|
|
"cannot be parsed as a tree");
|
2016-09-27 20:59:51 +00:00
|
|
|
break;
|
|
|
|
}
|
2008-02-25 21:46:08 +00:00
|
|
|
|
|
|
|
switch (mode) {
|
|
|
|
/*
|
|
|
|
* Standard modes..
|
|
|
|
*/
|
|
|
|
case S_IFREG | 0755:
|
|
|
|
case S_IFREG | 0644:
|
|
|
|
case S_IFLNK:
|
|
|
|
case S_IFDIR:
|
|
|
|
case S_IFGITLINK:
|
|
|
|
break;
|
|
|
|
/*
|
|
|
|
* This is nonstandard, but we had a few of these
|
|
|
|
* early on when we honored the full set of mode
|
|
|
|
* bits..
|
|
|
|
*/
|
|
|
|
case S_IFREG | 0664:
|
2015-06-22 15:25:00 +00:00
|
|
|
if (!options->strict)
|
2008-02-25 21:46:08 +00:00
|
|
|
break;
|
consistently use "fallthrough" comments in switches
Gcc 7 adds -Wimplicit-fallthrough, which can warn when a
switch case falls through to the next case. The general idea
is that the compiler can't tell if this was intentional or
not, so you should annotate any intentional fall-throughs as
such, leaving it to complain about any unannotated ones.
There's a GNU __attribute__ which can be used for
annotation, but of course we'd have to #ifdef it away on
non-gcc compilers. Gcc will also recognize
specially-formatted comments, which matches our current
practice. Let's extend that practice to all of the
unannotated sites (which I did look over and verify that
they were behaving as intended).
Ideally in each case we'd actually give some reasons in the
comment about why we're falling through, or what we're
falling through to. And gcc does support that with
-Wimplicit-fallthrough=2, which relaxes the comment pattern
matching to anything that contains "fallthrough" (or a
variety of spelling variants). However, this isn't the
default for -Wimplicit-fallthrough, nor for -Wextra. In the
name of simplicity, it's probably better for us to support
the default level, which requires "fallthrough" to be the
only thing in the comment (modulo some window dressing like
"else" and some punctuation; see the gcc manual for the
complete set of patterns).
This patch suppresses all warnings due to
-Wimplicit-fallthrough. We might eventually want to add that
to the DEVELOPER Makefile knob, but we should probably wait
until gcc 7 is more widely adopted (since earlier versions
will complain about the unknown warning type).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-09-21 06:25:41 +00:00
|
|
|
/* fallthrough */
|
2008-02-25 21:46:08 +00:00
|
|
|
default:
|
|
|
|
has_bad_modes = 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (o_name) {
|
2020-05-10 16:12:16 +00:00
|
|
|
switch (verify_ordered(o_mode, o_name, mode, name,
|
|
|
|
&df_dup_candidates)) {
|
2008-02-25 21:46:08 +00:00
|
|
|
case TREE_UNORDERED:
|
|
|
|
not_properly_sorted = 1;
|
|
|
|
break;
|
|
|
|
case TREE_HAS_DUPS:
|
|
|
|
has_dup_entries = 1;
|
|
|
|
break;
|
|
|
|
default:
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
o_mode = mode;
|
|
|
|
o_name = name;
|
|
|
|
}
|
|
|
|
|
2020-05-10 16:12:16 +00:00
|
|
|
name_stack_clear(&df_dup_candidates);
|
|
|
|
|
2012-07-28 15:06:29 +00:00
|
|
|
if (has_null_sha1)
|
2021-05-01 15:41:43 +00:00
|
|
|
retval += report(options, tree_oid, OBJ_TREE,
|
|
|
|
FSCK_MSG_NULL_SHA1,
|
|
|
|
"contains entries pointing to null sha1");
|
2008-02-25 21:46:08 +00:00
|
|
|
if (has_full_path)
|
2021-05-01 15:41:43 +00:00
|
|
|
retval += report(options, tree_oid, OBJ_TREE,
|
|
|
|
FSCK_MSG_FULL_PATHNAME,
|
|
|
|
"contains full pathnames");
|
2008-02-25 21:46:08 +00:00
|
|
|
if (has_empty_name)
|
2021-05-01 15:41:43 +00:00
|
|
|
retval += report(options, tree_oid, OBJ_TREE,
|
|
|
|
FSCK_MSG_EMPTY_NAME,
|
|
|
|
"contains empty pathname");
|
2012-11-28 02:27:37 +00:00
|
|
|
if (has_dot)
|
2021-05-01 15:41:43 +00:00
|
|
|
retval += report(options, tree_oid, OBJ_TREE,
|
|
|
|
FSCK_MSG_HAS_DOT,
|
|
|
|
"contains '.'");
|
2012-11-28 02:27:37 +00:00
|
|
|
if (has_dotdot)
|
2021-05-01 15:41:43 +00:00
|
|
|
retval += report(options, tree_oid, OBJ_TREE,
|
|
|
|
FSCK_MSG_HAS_DOTDOT,
|
|
|
|
"contains '..'");
|
2012-11-28 21:35:29 +00:00
|
|
|
if (has_dotgit)
|
2021-05-01 15:41:43 +00:00
|
|
|
retval += report(options, tree_oid, OBJ_TREE,
|
|
|
|
FSCK_MSG_HAS_DOTGIT,
|
|
|
|
"contains '.git'");
|
2008-02-25 21:46:08 +00:00
|
|
|
if (has_zero_pad)
|
2021-05-01 15:41:43 +00:00
|
|
|
retval += report(options, tree_oid, OBJ_TREE,
|
|
|
|
FSCK_MSG_ZERO_PADDED_FILEMODE,
|
|
|
|
"contains zero-padded file modes");
|
2008-02-25 21:46:08 +00:00
|
|
|
if (has_bad_modes)
|
2021-05-01 15:41:43 +00:00
|
|
|
retval += report(options, tree_oid, OBJ_TREE,
|
|
|
|
FSCK_MSG_BAD_FILEMODE,
|
|
|
|
"contains bad file modes");
|
2008-02-25 21:46:08 +00:00
|
|
|
if (has_dup_entries)
|
2021-05-01 15:41:43 +00:00
|
|
|
retval += report(options, tree_oid, OBJ_TREE,
|
|
|
|
FSCK_MSG_DUPLICATE_ENTRIES,
|
|
|
|
"contains duplicate file entries");
|
2008-02-25 21:46:08 +00:00
|
|
|
if (not_properly_sorted)
|
2021-05-01 15:41:43 +00:00
|
|
|
retval += report(options, tree_oid, OBJ_TREE,
|
|
|
|
FSCK_MSG_TREE_NOT_SORTED,
|
|
|
|
"not properly sorted");
|
2008-02-25 21:46:08 +00:00
|
|
|
return retval;
|
|
|
|
}
|
|
|
|
|
fsck: do not assume NUL-termination of buffers
The fsck code operates on an object buffer represented as a pointer/len
combination. However, the parsing of commits and tags is a little bit
loose; we mostly scan left-to-right through the buffer, without checking
whether we've gone past the length we were given.
This has traditionally been OK because the buffers we feed to fsck
always have an extra NUL after the end of the object content, which ends
any left-to-right scan. That has always been true for objects we read
from the odb, and we made it true for incoming index-pack/unpack-objects
checks in a1e920a0a7 (index-pack: terminate object buffers with NUL,
2014-12-08).
However, we recently added an exception: hash-object asks index_fd() to
do fsck checks. That _may_ have an extra NUL (if we read from a pipe
into a strbuf), but it might not (if we read the contents from the
file). Nor can we just teach it to always add a NUL. We may mmap the
on-disk file, which will not have any extra bytes (if it's a multiple of
the page size). Not to mention that this is a rather subtle assumption
for the fsck code to make.
Instead, let's make sure that the fsck parsers don't ever look past the
size of the buffer they've been given. This _almost_ works already,
thanks to earlier work in 4d0d89755e (Make sure fsck_commit_buffer()
does not run out of the buffer, 2014-09-11). The theory there is that we
check up front whether we have the end of header double-newline
separator. And then any left-to-right scanning we do is OK as long as it
stops when it hits that boundary.
However, we later softened that in 84d18c0bcf (fsck: it is OK for a tag
and a commit to lack the body, 2015-06-28), which allows the
double-newline header to be missing, but does require that the header
ends in a newline. That was OK back then, because of the NUL-termination
guarantees (including the one from a1e920a0a7 mentioned above).
Because 84d18c0bcf guarantees that any header line does end in a
newline, we are still OK with most of the left-to-right scanning. We
only need to take care after completing a line, to check that there is
another line (and we didn't run out of buffer).
Most of these checks are just need to check "buffer < buffer_end" (where
buffer is advanced as we parse) before scanning for the next header
line. But here are a few notes:
- we don't technically need to check for remaining buffer before
parsing the very first line ("tree" for a commit, or "object" for a
tag), because verify_headers() rejects a totally empty buffer. But
we'll do so in the name of consistency and defensiveness.
- there are some calls to strchr('\n'). These are actually OK by the
"the final header line must end in a newline" guarantee from
verify_headers(). They will always find that rather than run off the
end of the buffer. Curiously, they do check for a NULL return and
complain, but I believe that condition can never be reached.
However, I converted them to use memchr() with a proper size and
retained the NULL checks. Using memchr() is not much longer and
makes it more obvious what is going on. Likewise, retaining the NULL
checks serves as a defensive measure in case my analysis is wrong.
- commit 9a1a3a4d4c (mktag: allow omitting the header/body \n
separator, 2021-01-05), does check for the end-of-buffer condition,
but does so with "!*buffer", relying explicitly on the NUL
termination. We can accomplish the same thing with a pointer
comparison. I also folded it into the follow-on conditional that
checks the contents of the buffer, for consistency with the other
checks.
- fsck_ident() uses parse_timestamp(), which is based on strtoumax().
That function will happily skip past leading whitespace, including
newlines, which makes it a risk. We can fix this by scanning to the
first digit ourselves, and then using parse_timestamp() to do the
actual numeric conversion.
Note that as a side effect this fixes the fact that we missed
zero-padded timestamps like "<email> 0123" (whereas we would
complain about "<email> 0123"). I doubt anybody cares, but I
mention it here for completeness.
- fsck_tree() does not need any modifications. It relies on
decode_tree_entry() to do the actual parsing, and that function
checks both that there are enough bytes in the buffer to represent
an entry, and that there is a NUL at the appropriate spot (one
hash-length from the end; this may not be the NUL for the entry we
are parsing, but we know that in the worst case, everything from our
current position to that NUL is a filename, so we won't run out of
bytes).
In addition to fixing the code itself, we'd like to make sure our rather
subtle assumptions are not violated in the future. So this patch does
two more things:
- add comments around verify_headers() documenting the link between
what it checks and the memory safety of the callers. I don't expect
this code to be modified frequently, but this may help somebody from
accidentally breaking things.
- add a thorough set of tests covering truncations at various key
spots (e.g., for a "tree $oid" line, in the middle of the word
"tree", right after it, after the space, in the middle of the $oid,
and right at the end of the line. Most of these are fine already (it
is only truncating right at the end of the line that is currently
broken). And some of them are not even possible with the current
code (we parse "tree " as a unit, so truncating before the space is
equivalent). But I aimed here to consider the code a black box and
look for any truncations that would be a problem for a left-to-right
parser.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-19 23:13:29 +00:00
|
|
|
/*
|
|
|
|
* Confirm that the headers of a commit or tag object end in a reasonable way,
|
|
|
|
* either with the usual "\n\n" separator, or at least with a trailing newline
|
|
|
|
* on the final header line.
|
|
|
|
*
|
|
|
|
* This property is important for the memory safety of our callers. It allows
|
|
|
|
* them to scan the buffer linewise without constantly checking the remaining
|
|
|
|
* size as long as:
|
|
|
|
*
|
|
|
|
* - they check that there are bytes left in the buffer at the start of any
|
|
|
|
* line (i.e., that the last newline they saw was not the final one we
|
|
|
|
* found here)
|
|
|
|
*
|
|
|
|
* - any intra-line scanning they do will stop at a newline, which will worst
|
|
|
|
* case hit the newline we found here as the end-of-header. This makes it
|
|
|
|
* OK for them to use helpers like parse_oid_hex(), or even skip_prefix().
|
|
|
|
*/
|
2015-06-28 18:18:31 +00:00
|
|
|
static int verify_headers(const void *data, unsigned long size,
|
2019-10-18 05:00:50 +00:00
|
|
|
const struct object_id *oid, enum object_type type,
|
|
|
|
struct fsck_options *options)
|
2014-09-11 14:26:33 +00:00
|
|
|
{
|
|
|
|
const char *buffer = (const char *)data;
|
|
|
|
unsigned long i;
|
|
|
|
|
|
|
|
for (i = 0; i < size; i++) {
|
|
|
|
switch (buffer[i]) {
|
|
|
|
case '\0':
|
2019-10-18 05:00:50 +00:00
|
|
|
return report(options, oid, type,
|
2015-06-22 15:25:09 +00:00
|
|
|
FSCK_MSG_NUL_IN_HEADER,
|
|
|
|
"unterminated header: NUL at offset %ld", i);
|
2014-09-11 14:26:33 +00:00
|
|
|
case '\n':
|
|
|
|
if (i + 1 < size && buffer[i + 1] == '\n')
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2015-06-28 18:18:31 +00:00
|
|
|
/*
|
|
|
|
* We did not find double-LF that separates the header
|
|
|
|
* and the body. Not having a body is not a crime but
|
|
|
|
* we do want to see the terminating LF for the last header
|
|
|
|
* line.
|
|
|
|
*/
|
|
|
|
if (size && buffer[size - 1] == '\n')
|
|
|
|
return 0;
|
|
|
|
|
2019-10-18 05:00:50 +00:00
|
|
|
return report(options, oid, type,
|
2015-06-22 15:25:09 +00:00
|
|
|
FSCK_MSG_UNTERMINATED_HEADER, "unterminated header");
|
2014-09-11 14:26:33 +00:00
|
|
|
}
|
|
|
|
|
2019-10-18 05:00:04 +00:00
|
|
|
static int fsck_ident(const char **ident,
|
|
|
|
const struct object_id *oid, enum object_type type,
|
|
|
|
struct fsck_options *options)
|
2010-04-24 16:06:08 +00:00
|
|
|
{
|
2015-06-22 15:26:03 +00:00
|
|
|
const char *p = *ident;
|
fsck: report integer overflow in author timestamps
When we check commit objects, we complain if commit->date is
ULONG_MAX, which is an indication that we saw integer
overflow when parsing it. However, we do not do any check at
all for author lines, which also contain a timestamp.
Let's actually check the timestamps on each ident line
with strtoul. This catches both author and committer lines,
and we can get rid of the now-redundant commit->date check.
Note that like the existing check, we compare only against
ULONG_MAX. Now that we are calling strtoul at the site of
the check, we could be slightly more careful and also check
that errno is set to ERANGE. However, this will make further
refactoring in future patches a little harder, and it
doesn't really matter in practice.
For 32-bit systems, one would have to create a commit at the
exact wrong second in 2038. But by the time we get close to
that, all systems will hopefully have moved to 64-bit (and
if they haven't, they have a real problem one second later).
For 64-bit systems, by the time we get close to ULONG_MAX,
all systems will hopefully have been consumed in the fiery
wrath of our expanding Sun.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-02-24 07:39:04 +00:00
|
|
|
char *end;
|
|
|
|
|
2015-06-22 15:26:03 +00:00
|
|
|
*ident = strchrnul(*ident, '\n');
|
|
|
|
if (**ident == '\n')
|
|
|
|
(*ident)++;
|
|
|
|
|
|
|
|
if (*p == '<')
|
2019-10-18 05:00:04 +00:00
|
|
|
return report(options, oid, type, FSCK_MSG_MISSING_NAME_BEFORE_EMAIL, "invalid author/committer line - missing space before email");
|
2015-06-22 15:26:03 +00:00
|
|
|
p += strcspn(p, "<>\n");
|
|
|
|
if (*p == '>')
|
2019-10-18 05:00:04 +00:00
|
|
|
return report(options, oid, type, FSCK_MSG_BAD_NAME, "invalid author/committer line - bad name");
|
2015-06-22 15:26:03 +00:00
|
|
|
if (*p != '<')
|
2019-10-18 05:00:04 +00:00
|
|
|
return report(options, oid, type, FSCK_MSG_MISSING_EMAIL, "invalid author/committer line - missing email");
|
2015-06-22 15:26:03 +00:00
|
|
|
if (p[-1] != ' ')
|
2019-10-18 05:00:04 +00:00
|
|
|
return report(options, oid, type, FSCK_MSG_MISSING_SPACE_BEFORE_EMAIL, "invalid author/committer line - missing space before email");
|
2015-06-22 15:26:03 +00:00
|
|
|
p++;
|
|
|
|
p += strcspn(p, "<>\n");
|
|
|
|
if (*p != '>')
|
2019-10-18 05:00:04 +00:00
|
|
|
return report(options, oid, type, FSCK_MSG_BAD_EMAIL, "invalid author/committer line - bad email");
|
2015-06-22 15:26:03 +00:00
|
|
|
p++;
|
|
|
|
if (*p != ' ')
|
2019-10-18 05:00:04 +00:00
|
|
|
return report(options, oid, type, FSCK_MSG_MISSING_SPACE_BEFORE_DATE, "invalid author/committer line - missing space before date");
|
2015-06-22 15:26:03 +00:00
|
|
|
p++;
|
fsck: do not assume NUL-termination of buffers
The fsck code operates on an object buffer represented as a pointer/len
combination. However, the parsing of commits and tags is a little bit
loose; we mostly scan left-to-right through the buffer, without checking
whether we've gone past the length we were given.
This has traditionally been OK because the buffers we feed to fsck
always have an extra NUL after the end of the object content, which ends
any left-to-right scan. That has always been true for objects we read
from the odb, and we made it true for incoming index-pack/unpack-objects
checks in a1e920a0a7 (index-pack: terminate object buffers with NUL,
2014-12-08).
However, we recently added an exception: hash-object asks index_fd() to
do fsck checks. That _may_ have an extra NUL (if we read from a pipe
into a strbuf), but it might not (if we read the contents from the
file). Nor can we just teach it to always add a NUL. We may mmap the
on-disk file, which will not have any extra bytes (if it's a multiple of
the page size). Not to mention that this is a rather subtle assumption
for the fsck code to make.
Instead, let's make sure that the fsck parsers don't ever look past the
size of the buffer they've been given. This _almost_ works already,
thanks to earlier work in 4d0d89755e (Make sure fsck_commit_buffer()
does not run out of the buffer, 2014-09-11). The theory there is that we
check up front whether we have the end of header double-newline
separator. And then any left-to-right scanning we do is OK as long as it
stops when it hits that boundary.
However, we later softened that in 84d18c0bcf (fsck: it is OK for a tag
and a commit to lack the body, 2015-06-28), which allows the
double-newline header to be missing, but does require that the header
ends in a newline. That was OK back then, because of the NUL-termination
guarantees (including the one from a1e920a0a7 mentioned above).
Because 84d18c0bcf guarantees that any header line does end in a
newline, we are still OK with most of the left-to-right scanning. We
only need to take care after completing a line, to check that there is
another line (and we didn't run out of buffer).
Most of these checks are just need to check "buffer < buffer_end" (where
buffer is advanced as we parse) before scanning for the next header
line. But here are a few notes:
- we don't technically need to check for remaining buffer before
parsing the very first line ("tree" for a commit, or "object" for a
tag), because verify_headers() rejects a totally empty buffer. But
we'll do so in the name of consistency and defensiveness.
- there are some calls to strchr('\n'). These are actually OK by the
"the final header line must end in a newline" guarantee from
verify_headers(). They will always find that rather than run off the
end of the buffer. Curiously, they do check for a NULL return and
complain, but I believe that condition can never be reached.
However, I converted them to use memchr() with a proper size and
retained the NULL checks. Using memchr() is not much longer and
makes it more obvious what is going on. Likewise, retaining the NULL
checks serves as a defensive measure in case my analysis is wrong.
- commit 9a1a3a4d4c (mktag: allow omitting the header/body \n
separator, 2021-01-05), does check for the end-of-buffer condition,
but does so with "!*buffer", relying explicitly on the NUL
termination. We can accomplish the same thing with a pointer
comparison. I also folded it into the follow-on conditional that
checks the contents of the buffer, for consistency with the other
checks.
- fsck_ident() uses parse_timestamp(), which is based on strtoumax().
That function will happily skip past leading whitespace, including
newlines, which makes it a risk. We can fix this by scanning to the
first digit ourselves, and then using parse_timestamp() to do the
actual numeric conversion.
Note that as a side effect this fixes the fact that we missed
zero-padded timestamps like "<email> 0123" (whereas we would
complain about "<email> 0123"). I doubt anybody cares, but I
mention it here for completeness.
- fsck_tree() does not need any modifications. It relies on
decode_tree_entry() to do the actual parsing, and that function
checks both that there are enough bytes in the buffer to represent
an entry, and that there is a NUL at the appropriate spot (one
hash-length from the end; this may not be the NUL for the entry we
are parsing, but we know that in the worst case, everything from our
current position to that NUL is a filename, so we won't run out of
bytes).
In addition to fixing the code itself, we'd like to make sure our rather
subtle assumptions are not violated in the future. So this patch does
two more things:
- add comments around verify_headers() documenting the link between
what it checks and the memory safety of the callers. I don't expect
this code to be modified frequently, but this may help somebody from
accidentally breaking things.
- add a thorough set of tests covering truncations at various key
spots (e.g., for a "tree $oid" line, in the middle of the word
"tree", right after it, after the space, in the middle of the $oid,
and right at the end of the line. Most of these are fine already (it
is only truncating right at the end of the line that is currently
broken). And some of them are not even possible with the current
code (we parse "tree " as a unit, so truncating before the space is
equivalent). But I aimed here to consider the code a black box and
look for any truncations that would be a problem for a left-to-right
parser.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-19 23:13:29 +00:00
|
|
|
/*
|
|
|
|
* Our timestamp parser is based on the C strto*() functions, which
|
|
|
|
* will happily eat whitespace, including the newline that is supposed
|
|
|
|
* to prevent us walking past the end of the buffer. So do our own
|
|
|
|
* scan, skipping linear whitespace but not newlines, and then
|
|
|
|
* confirming we found a digit. We _could_ be even more strict here,
|
|
|
|
* as we really expect only a single space, but since we have
|
|
|
|
* traditionally allowed extra whitespace, we'll continue to do so.
|
|
|
|
*/
|
|
|
|
while (*p == ' ' || *p == '\t')
|
|
|
|
p++;
|
|
|
|
if (!isdigit(*p))
|
|
|
|
return report(options, oid, type, FSCK_MSG_BAD_DATE,
|
|
|
|
"invalid author/committer line - bad date");
|
2015-06-22 15:26:03 +00:00
|
|
|
if (*p == '0' && p[1] != ' ')
|
2019-10-18 05:00:04 +00:00
|
|
|
return report(options, oid, type, FSCK_MSG_ZERO_PADDED_DATE, "invalid author/committer line - zero-padded date");
|
2017-04-21 10:45:44 +00:00
|
|
|
if (date_overflows(parse_timestamp(p, &end, 10)))
|
2019-10-18 05:00:04 +00:00
|
|
|
return report(options, oid, type, FSCK_MSG_BAD_DATE_OVERFLOW, "invalid author/committer line - date causes integer overflow");
|
2015-06-22 15:26:03 +00:00
|
|
|
if ((end == p || *end != ' '))
|
2019-10-18 05:00:04 +00:00
|
|
|
return report(options, oid, type, FSCK_MSG_BAD_DATE, "invalid author/committer line - bad date");
|
2015-06-22 15:26:03 +00:00
|
|
|
p = end + 1;
|
|
|
|
if ((*p != '+' && *p != '-') ||
|
|
|
|
!isdigit(p[1]) ||
|
|
|
|
!isdigit(p[2]) ||
|
|
|
|
!isdigit(p[3]) ||
|
|
|
|
!isdigit(p[4]) ||
|
|
|
|
(p[5] != '\n'))
|
2019-10-18 05:00:04 +00:00
|
|
|
return report(options, oid, type, FSCK_MSG_BAD_TIMEZONE, "invalid author/committer line - bad time zone");
|
2015-06-22 15:26:03 +00:00
|
|
|
p += 6;
|
2010-04-24 16:06:08 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2019-10-18 05:01:48 +00:00
|
|
|
static int fsck_commit(const struct object_id *oid,
|
|
|
|
const char *buffer, unsigned long size,
|
|
|
|
struct fsck_options *options)
|
2008-02-25 21:46:08 +00:00
|
|
|
{
|
2019-10-18 05:00:59 +00:00
|
|
|
struct object_id tree_oid, parent_oid;
|
fsck: stop checking commit->parent counts
In 4516338243 (builtin-fsck: reports missing parent commits,
2008-02-25), we added code to check that fsck found the same number of
parents from parsing the commit itself as we see in the commit struct we
got from parse_commit_buffer(). Back then the rationale was that the
normal commit parser might skip some bad parents.
But earlier in this series, we started treating that reliably as a
parsing error, meaning that we'd complain about it before we even hit
the code in fsck.c.
Let's drop this code, which now makes fsck_commit_buffer() completely
independent of any parsed values in the commit struct (that's
conceptually cleaner, and also opens up more refactoring options).
Note that we can also drop the MISSING_PARENT and MISSING_GRAFT fsck
identifiers. This is no loss, as these would not trigger reliably
anyway. We'd hit them only when lookup_commit() failed, which occurs
only if we happen to have seen the object with another type already in
the same process. In most cases, we'd actually run into the problem
during the connectivity walk, not here.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-18 04:49:10 +00:00
|
|
|
unsigned author_count;
|
2010-04-24 16:06:08 +00:00
|
|
|
int err;
|
2016-04-14 17:58:22 +00:00
|
|
|
const char *buffer_begin = buffer;
|
fsck: do not assume NUL-termination of buffers
The fsck code operates on an object buffer represented as a pointer/len
combination. However, the parsing of commits and tags is a little bit
loose; we mostly scan left-to-right through the buffer, without checking
whether we've gone past the length we were given.
This has traditionally been OK because the buffers we feed to fsck
always have an extra NUL after the end of the object content, which ends
any left-to-right scan. That has always been true for objects we read
from the odb, and we made it true for incoming index-pack/unpack-objects
checks in a1e920a0a7 (index-pack: terminate object buffers with NUL,
2014-12-08).
However, we recently added an exception: hash-object asks index_fd() to
do fsck checks. That _may_ have an extra NUL (if we read from a pipe
into a strbuf), but it might not (if we read the contents from the
file). Nor can we just teach it to always add a NUL. We may mmap the
on-disk file, which will not have any extra bytes (if it's a multiple of
the page size). Not to mention that this is a rather subtle assumption
for the fsck code to make.
Instead, let's make sure that the fsck parsers don't ever look past the
size of the buffer they've been given. This _almost_ works already,
thanks to earlier work in 4d0d89755e (Make sure fsck_commit_buffer()
does not run out of the buffer, 2014-09-11). The theory there is that we
check up front whether we have the end of header double-newline
separator. And then any left-to-right scanning we do is OK as long as it
stops when it hits that boundary.
However, we later softened that in 84d18c0bcf (fsck: it is OK for a tag
and a commit to lack the body, 2015-06-28), which allows the
double-newline header to be missing, but does require that the header
ends in a newline. That was OK back then, because of the NUL-termination
guarantees (including the one from a1e920a0a7 mentioned above).
Because 84d18c0bcf guarantees that any header line does end in a
newline, we are still OK with most of the left-to-right scanning. We
only need to take care after completing a line, to check that there is
another line (and we didn't run out of buffer).
Most of these checks are just need to check "buffer < buffer_end" (where
buffer is advanced as we parse) before scanning for the next header
line. But here are a few notes:
- we don't technically need to check for remaining buffer before
parsing the very first line ("tree" for a commit, or "object" for a
tag), because verify_headers() rejects a totally empty buffer. But
we'll do so in the name of consistency and defensiveness.
- there are some calls to strchr('\n'). These are actually OK by the
"the final header line must end in a newline" guarantee from
verify_headers(). They will always find that rather than run off the
end of the buffer. Curiously, they do check for a NULL return and
complain, but I believe that condition can never be reached.
However, I converted them to use memchr() with a proper size and
retained the NULL checks. Using memchr() is not much longer and
makes it more obvious what is going on. Likewise, retaining the NULL
checks serves as a defensive measure in case my analysis is wrong.
- commit 9a1a3a4d4c (mktag: allow omitting the header/body \n
separator, 2021-01-05), does check for the end-of-buffer condition,
but does so with "!*buffer", relying explicitly on the NUL
termination. We can accomplish the same thing with a pointer
comparison. I also folded it into the follow-on conditional that
checks the contents of the buffer, for consistency with the other
checks.
- fsck_ident() uses parse_timestamp(), which is based on strtoumax().
That function will happily skip past leading whitespace, including
newlines, which makes it a risk. We can fix this by scanning to the
first digit ourselves, and then using parse_timestamp() to do the
actual numeric conversion.
Note that as a side effect this fixes the fact that we missed
zero-padded timestamps like "<email> 0123" (whereas we would
complain about "<email> 0123"). I doubt anybody cares, but I
mention it here for completeness.
- fsck_tree() does not need any modifications. It relies on
decode_tree_entry() to do the actual parsing, and that function
checks both that there are enough bytes in the buffer to represent
an entry, and that there is a NUL at the appropriate spot (one
hash-length from the end; this may not be the NUL for the entry we
are parsing, but we know that in the worst case, everything from our
current position to that NUL is a filename, so we won't run out of
bytes).
In addition to fixing the code itself, we'd like to make sure our rather
subtle assumptions are not violated in the future. So this patch does
two more things:
- add comments around verify_headers() documenting the link between
what it checks and the memory safety of the callers. I don't expect
this code to be modified frequently, but this may help somebody from
accidentally breaking things.
- add a thorough set of tests covering truncations at various key
spots (e.g., for a "tree $oid" line, in the middle of the word
"tree", right after it, after the space, in the middle of the $oid,
and right at the end of the line. Most of these are fine already (it
is only truncating right at the end of the line that is currently
broken). And some of them are not even possible with the current
code (we parse "tree " as a unit, so truncating before the space is
equivalent). But I aimed here to consider the code a black box and
look for any truncations that would be a problem for a left-to-right
parser.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-19 23:13:29 +00:00
|
|
|
const char *buffer_end = buffer + size;
|
2018-05-02 00:25:41 +00:00
|
|
|
const char *p;
|
2008-02-25 21:46:08 +00:00
|
|
|
|
fsck: do not assume NUL-termination of buffers
The fsck code operates on an object buffer represented as a pointer/len
combination. However, the parsing of commits and tags is a little bit
loose; we mostly scan left-to-right through the buffer, without checking
whether we've gone past the length we were given.
This has traditionally been OK because the buffers we feed to fsck
always have an extra NUL after the end of the object content, which ends
any left-to-right scan. That has always been true for objects we read
from the odb, and we made it true for incoming index-pack/unpack-objects
checks in a1e920a0a7 (index-pack: terminate object buffers with NUL,
2014-12-08).
However, we recently added an exception: hash-object asks index_fd() to
do fsck checks. That _may_ have an extra NUL (if we read from a pipe
into a strbuf), but it might not (if we read the contents from the
file). Nor can we just teach it to always add a NUL. We may mmap the
on-disk file, which will not have any extra bytes (if it's a multiple of
the page size). Not to mention that this is a rather subtle assumption
for the fsck code to make.
Instead, let's make sure that the fsck parsers don't ever look past the
size of the buffer they've been given. This _almost_ works already,
thanks to earlier work in 4d0d89755e (Make sure fsck_commit_buffer()
does not run out of the buffer, 2014-09-11). The theory there is that we
check up front whether we have the end of header double-newline
separator. And then any left-to-right scanning we do is OK as long as it
stops when it hits that boundary.
However, we later softened that in 84d18c0bcf (fsck: it is OK for a tag
and a commit to lack the body, 2015-06-28), which allows the
double-newline header to be missing, but does require that the header
ends in a newline. That was OK back then, because of the NUL-termination
guarantees (including the one from a1e920a0a7 mentioned above).
Because 84d18c0bcf guarantees that any header line does end in a
newline, we are still OK with most of the left-to-right scanning. We
only need to take care after completing a line, to check that there is
another line (and we didn't run out of buffer).
Most of these checks are just need to check "buffer < buffer_end" (where
buffer is advanced as we parse) before scanning for the next header
line. But here are a few notes:
- we don't technically need to check for remaining buffer before
parsing the very first line ("tree" for a commit, or "object" for a
tag), because verify_headers() rejects a totally empty buffer. But
we'll do so in the name of consistency and defensiveness.
- there are some calls to strchr('\n'). These are actually OK by the
"the final header line must end in a newline" guarantee from
verify_headers(). They will always find that rather than run off the
end of the buffer. Curiously, they do check for a NULL return and
complain, but I believe that condition can never be reached.
However, I converted them to use memchr() with a proper size and
retained the NULL checks. Using memchr() is not much longer and
makes it more obvious what is going on. Likewise, retaining the NULL
checks serves as a defensive measure in case my analysis is wrong.
- commit 9a1a3a4d4c (mktag: allow omitting the header/body \n
separator, 2021-01-05), does check for the end-of-buffer condition,
but does so with "!*buffer", relying explicitly on the NUL
termination. We can accomplish the same thing with a pointer
comparison. I also folded it into the follow-on conditional that
checks the contents of the buffer, for consistency with the other
checks.
- fsck_ident() uses parse_timestamp(), which is based on strtoumax().
That function will happily skip past leading whitespace, including
newlines, which makes it a risk. We can fix this by scanning to the
first digit ourselves, and then using parse_timestamp() to do the
actual numeric conversion.
Note that as a side effect this fixes the fact that we missed
zero-padded timestamps like "<email> 0123" (whereas we would
complain about "<email> 0123"). I doubt anybody cares, but I
mention it here for completeness.
- fsck_tree() does not need any modifications. It relies on
decode_tree_entry() to do the actual parsing, and that function
checks both that there are enough bytes in the buffer to represent
an entry, and that there is a NUL at the appropriate spot (one
hash-length from the end; this may not be the NUL for the entry we
are parsing, but we know that in the worst case, everything from our
current position to that NUL is a filename, so we won't run out of
bytes).
In addition to fixing the code itself, we'd like to make sure our rather
subtle assumptions are not violated in the future. So this patch does
two more things:
- add comments around verify_headers() documenting the link between
what it checks and the memory safety of the callers. I don't expect
this code to be modified frequently, but this may help somebody from
accidentally breaking things.
- add a thorough set of tests covering truncations at various key
spots (e.g., for a "tree $oid" line, in the middle of the word
"tree", right after it, after the space, in the middle of the $oid,
and right at the end of the line. Most of these are fine already (it
is only truncating right at the end of the line that is currently
broken). And some of them are not even possible with the current
code (we parse "tree " as a unit, so truncating before the space is
equivalent). But I aimed here to consider the code a black box and
look for any truncations that would be a problem for a left-to-right
parser.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-19 23:13:29 +00:00
|
|
|
/*
|
|
|
|
* We _must_ stop parsing immediately if this reports failure, as the
|
|
|
|
* memory safety of the rest of the function depends on it. See the
|
|
|
|
* comment above the definition of verify_headers() for more details.
|
|
|
|
*/
|
2019-10-18 05:01:48 +00:00
|
|
|
if (verify_headers(buffer, size, oid, OBJ_COMMIT, options))
|
2014-09-11 14:26:33 +00:00
|
|
|
return -1;
|
|
|
|
|
fsck: do not assume NUL-termination of buffers
The fsck code operates on an object buffer represented as a pointer/len
combination. However, the parsing of commits and tags is a little bit
loose; we mostly scan left-to-right through the buffer, without checking
whether we've gone past the length we were given.
This has traditionally been OK because the buffers we feed to fsck
always have an extra NUL after the end of the object content, which ends
any left-to-right scan. That has always been true for objects we read
from the odb, and we made it true for incoming index-pack/unpack-objects
checks in a1e920a0a7 (index-pack: terminate object buffers with NUL,
2014-12-08).
However, we recently added an exception: hash-object asks index_fd() to
do fsck checks. That _may_ have an extra NUL (if we read from a pipe
into a strbuf), but it might not (if we read the contents from the
file). Nor can we just teach it to always add a NUL. We may mmap the
on-disk file, which will not have any extra bytes (if it's a multiple of
the page size). Not to mention that this is a rather subtle assumption
for the fsck code to make.
Instead, let's make sure that the fsck parsers don't ever look past the
size of the buffer they've been given. This _almost_ works already,
thanks to earlier work in 4d0d89755e (Make sure fsck_commit_buffer()
does not run out of the buffer, 2014-09-11). The theory there is that we
check up front whether we have the end of header double-newline
separator. And then any left-to-right scanning we do is OK as long as it
stops when it hits that boundary.
However, we later softened that in 84d18c0bcf (fsck: it is OK for a tag
and a commit to lack the body, 2015-06-28), which allows the
double-newline header to be missing, but does require that the header
ends in a newline. That was OK back then, because of the NUL-termination
guarantees (including the one from a1e920a0a7 mentioned above).
Because 84d18c0bcf guarantees that any header line does end in a
newline, we are still OK with most of the left-to-right scanning. We
only need to take care after completing a line, to check that there is
another line (and we didn't run out of buffer).
Most of these checks are just need to check "buffer < buffer_end" (where
buffer is advanced as we parse) before scanning for the next header
line. But here are a few notes:
- we don't technically need to check for remaining buffer before
parsing the very first line ("tree" for a commit, or "object" for a
tag), because verify_headers() rejects a totally empty buffer. But
we'll do so in the name of consistency and defensiveness.
- there are some calls to strchr('\n'). These are actually OK by the
"the final header line must end in a newline" guarantee from
verify_headers(). They will always find that rather than run off the
end of the buffer. Curiously, they do check for a NULL return and
complain, but I believe that condition can never be reached.
However, I converted them to use memchr() with a proper size and
retained the NULL checks. Using memchr() is not much longer and
makes it more obvious what is going on. Likewise, retaining the NULL
checks serves as a defensive measure in case my analysis is wrong.
- commit 9a1a3a4d4c (mktag: allow omitting the header/body \n
separator, 2021-01-05), does check for the end-of-buffer condition,
but does so with "!*buffer", relying explicitly on the NUL
termination. We can accomplish the same thing with a pointer
comparison. I also folded it into the follow-on conditional that
checks the contents of the buffer, for consistency with the other
checks.
- fsck_ident() uses parse_timestamp(), which is based on strtoumax().
That function will happily skip past leading whitespace, including
newlines, which makes it a risk. We can fix this by scanning to the
first digit ourselves, and then using parse_timestamp() to do the
actual numeric conversion.
Note that as a side effect this fixes the fact that we missed
zero-padded timestamps like "<email> 0123" (whereas we would
complain about "<email> 0123"). I doubt anybody cares, but I
mention it here for completeness.
- fsck_tree() does not need any modifications. It relies on
decode_tree_entry() to do the actual parsing, and that function
checks both that there are enough bytes in the buffer to represent
an entry, and that there is a NUL at the appropriate spot (one
hash-length from the end; this may not be the NUL for the entry we
are parsing, but we know that in the worst case, everything from our
current position to that NUL is a filename, so we won't run out of
bytes).
In addition to fixing the code itself, we'd like to make sure our rather
subtle assumptions are not violated in the future. So this patch does
two more things:
- add comments around verify_headers() documenting the link between
what it checks and the memory safety of the callers. I don't expect
this code to be modified frequently, but this may help somebody from
accidentally breaking things.
- add a thorough set of tests covering truncations at various key
spots (e.g., for a "tree $oid" line, in the middle of the word
"tree", right after it, after the space, in the middle of the $oid,
and right at the end of the line. Most of these are fine already (it
is only truncating right at the end of the line that is currently
broken). And some of them are not even possible with the current
code (we parse "tree " as a unit, so truncating before the space is
equivalent). But I aimed here to consider the code a black box and
look for any truncations that would be a problem for a left-to-right
parser.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-19 23:13:29 +00:00
|
|
|
if (buffer >= buffer_end || !skip_prefix(buffer, "tree ", &buffer))
|
2019-10-18 05:01:48 +00:00
|
|
|
return report(options, oid, OBJ_COMMIT, FSCK_MSG_MISSING_TREE, "invalid format - expected 'tree' line");
|
2018-05-02 00:25:41 +00:00
|
|
|
if (parse_oid_hex(buffer, &tree_oid, &p) || *p != '\n') {
|
2019-10-18 05:01:48 +00:00
|
|
|
err = report(options, oid, OBJ_COMMIT, FSCK_MSG_BAD_TREE_SHA1, "invalid 'tree' line format - bad sha1");
|
2015-06-22 15:26:11 +00:00
|
|
|
if (err)
|
|
|
|
return err;
|
|
|
|
}
|
2018-05-02 00:25:41 +00:00
|
|
|
buffer = p + 1;
|
fsck: do not assume NUL-termination of buffers
The fsck code operates on an object buffer represented as a pointer/len
combination. However, the parsing of commits and tags is a little bit
loose; we mostly scan left-to-right through the buffer, without checking
whether we've gone past the length we were given.
This has traditionally been OK because the buffers we feed to fsck
always have an extra NUL after the end of the object content, which ends
any left-to-right scan. That has always been true for objects we read
from the odb, and we made it true for incoming index-pack/unpack-objects
checks in a1e920a0a7 (index-pack: terminate object buffers with NUL,
2014-12-08).
However, we recently added an exception: hash-object asks index_fd() to
do fsck checks. That _may_ have an extra NUL (if we read from a pipe
into a strbuf), but it might not (if we read the contents from the
file). Nor can we just teach it to always add a NUL. We may mmap the
on-disk file, which will not have any extra bytes (if it's a multiple of
the page size). Not to mention that this is a rather subtle assumption
for the fsck code to make.
Instead, let's make sure that the fsck parsers don't ever look past the
size of the buffer they've been given. This _almost_ works already,
thanks to earlier work in 4d0d89755e (Make sure fsck_commit_buffer()
does not run out of the buffer, 2014-09-11). The theory there is that we
check up front whether we have the end of header double-newline
separator. And then any left-to-right scanning we do is OK as long as it
stops when it hits that boundary.
However, we later softened that in 84d18c0bcf (fsck: it is OK for a tag
and a commit to lack the body, 2015-06-28), which allows the
double-newline header to be missing, but does require that the header
ends in a newline. That was OK back then, because of the NUL-termination
guarantees (including the one from a1e920a0a7 mentioned above).
Because 84d18c0bcf guarantees that any header line does end in a
newline, we are still OK with most of the left-to-right scanning. We
only need to take care after completing a line, to check that there is
another line (and we didn't run out of buffer).
Most of these checks are just need to check "buffer < buffer_end" (where
buffer is advanced as we parse) before scanning for the next header
line. But here are a few notes:
- we don't technically need to check for remaining buffer before
parsing the very first line ("tree" for a commit, or "object" for a
tag), because verify_headers() rejects a totally empty buffer. But
we'll do so in the name of consistency and defensiveness.
- there are some calls to strchr('\n'). These are actually OK by the
"the final header line must end in a newline" guarantee from
verify_headers(). They will always find that rather than run off the
end of the buffer. Curiously, they do check for a NULL return and
complain, but I believe that condition can never be reached.
However, I converted them to use memchr() with a proper size and
retained the NULL checks. Using memchr() is not much longer and
makes it more obvious what is going on. Likewise, retaining the NULL
checks serves as a defensive measure in case my analysis is wrong.
- commit 9a1a3a4d4c (mktag: allow omitting the header/body \n
separator, 2021-01-05), does check for the end-of-buffer condition,
but does so with "!*buffer", relying explicitly on the NUL
termination. We can accomplish the same thing with a pointer
comparison. I also folded it into the follow-on conditional that
checks the contents of the buffer, for consistency with the other
checks.
- fsck_ident() uses parse_timestamp(), which is based on strtoumax().
That function will happily skip past leading whitespace, including
newlines, which makes it a risk. We can fix this by scanning to the
first digit ourselves, and then using parse_timestamp() to do the
actual numeric conversion.
Note that as a side effect this fixes the fact that we missed
zero-padded timestamps like "<email> 0123" (whereas we would
complain about "<email> 0123"). I doubt anybody cares, but I
mention it here for completeness.
- fsck_tree() does not need any modifications. It relies on
decode_tree_entry() to do the actual parsing, and that function
checks both that there are enough bytes in the buffer to represent
an entry, and that there is a NUL at the appropriate spot (one
hash-length from the end; this may not be the NUL for the entry we
are parsing, but we know that in the worst case, everything from our
current position to that NUL is a filename, so we won't run out of
bytes).
In addition to fixing the code itself, we'd like to make sure our rather
subtle assumptions are not violated in the future. So this patch does
two more things:
- add comments around verify_headers() documenting the link between
what it checks and the memory safety of the callers. I don't expect
this code to be modified frequently, but this may help somebody from
accidentally breaking things.
- add a thorough set of tests covering truncations at various key
spots (e.g., for a "tree $oid" line, in the middle of the word
"tree", right after it, after the space, in the middle of the $oid,
and right at the end of the line. Most of these are fine already (it
is only truncating right at the end of the line that is currently
broken). And some of them are not even possible with the current
code (we parse "tree " as a unit, so truncating before the space is
equivalent). But I aimed here to consider the code a black box and
look for any truncations that would be a problem for a left-to-right
parser.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-19 23:13:29 +00:00
|
|
|
while (buffer < buffer_end && skip_prefix(buffer, "parent ", &buffer)) {
|
2019-10-18 05:00:59 +00:00
|
|
|
if (parse_oid_hex(buffer, &parent_oid, &p) || *p != '\n') {
|
2019-10-18 05:01:48 +00:00
|
|
|
err = report(options, oid, OBJ_COMMIT, FSCK_MSG_BAD_PARENT_SHA1, "invalid 'parent' line format - bad sha1");
|
2015-06-22 15:26:11 +00:00
|
|
|
if (err)
|
|
|
|
return err;
|
|
|
|
}
|
2018-05-02 00:25:41 +00:00
|
|
|
buffer = p + 1;
|
2008-02-25 21:46:08 +00:00
|
|
|
}
|
2015-06-22 15:26:23 +00:00
|
|
|
author_count = 0;
|
fsck: do not assume NUL-termination of buffers
The fsck code operates on an object buffer represented as a pointer/len
combination. However, the parsing of commits and tags is a little bit
loose; we mostly scan left-to-right through the buffer, without checking
whether we've gone past the length we were given.
This has traditionally been OK because the buffers we feed to fsck
always have an extra NUL after the end of the object content, which ends
any left-to-right scan. That has always been true for objects we read
from the odb, and we made it true for incoming index-pack/unpack-objects
checks in a1e920a0a7 (index-pack: terminate object buffers with NUL,
2014-12-08).
However, we recently added an exception: hash-object asks index_fd() to
do fsck checks. That _may_ have an extra NUL (if we read from a pipe
into a strbuf), but it might not (if we read the contents from the
file). Nor can we just teach it to always add a NUL. We may mmap the
on-disk file, which will not have any extra bytes (if it's a multiple of
the page size). Not to mention that this is a rather subtle assumption
for the fsck code to make.
Instead, let's make sure that the fsck parsers don't ever look past the
size of the buffer they've been given. This _almost_ works already,
thanks to earlier work in 4d0d89755e (Make sure fsck_commit_buffer()
does not run out of the buffer, 2014-09-11). The theory there is that we
check up front whether we have the end of header double-newline
separator. And then any left-to-right scanning we do is OK as long as it
stops when it hits that boundary.
However, we later softened that in 84d18c0bcf (fsck: it is OK for a tag
and a commit to lack the body, 2015-06-28), which allows the
double-newline header to be missing, but does require that the header
ends in a newline. That was OK back then, because of the NUL-termination
guarantees (including the one from a1e920a0a7 mentioned above).
Because 84d18c0bcf guarantees that any header line does end in a
newline, we are still OK with most of the left-to-right scanning. We
only need to take care after completing a line, to check that there is
another line (and we didn't run out of buffer).
Most of these checks are just need to check "buffer < buffer_end" (where
buffer is advanced as we parse) before scanning for the next header
line. But here are a few notes:
- we don't technically need to check for remaining buffer before
parsing the very first line ("tree" for a commit, or "object" for a
tag), because verify_headers() rejects a totally empty buffer. But
we'll do so in the name of consistency and defensiveness.
- there are some calls to strchr('\n'). These are actually OK by the
"the final header line must end in a newline" guarantee from
verify_headers(). They will always find that rather than run off the
end of the buffer. Curiously, they do check for a NULL return and
complain, but I believe that condition can never be reached.
However, I converted them to use memchr() with a proper size and
retained the NULL checks. Using memchr() is not much longer and
makes it more obvious what is going on. Likewise, retaining the NULL
checks serves as a defensive measure in case my analysis is wrong.
- commit 9a1a3a4d4c (mktag: allow omitting the header/body \n
separator, 2021-01-05), does check for the end-of-buffer condition,
but does so with "!*buffer", relying explicitly on the NUL
termination. We can accomplish the same thing with a pointer
comparison. I also folded it into the follow-on conditional that
checks the contents of the buffer, for consistency with the other
checks.
- fsck_ident() uses parse_timestamp(), which is based on strtoumax().
That function will happily skip past leading whitespace, including
newlines, which makes it a risk. We can fix this by scanning to the
first digit ourselves, and then using parse_timestamp() to do the
actual numeric conversion.
Note that as a side effect this fixes the fact that we missed
zero-padded timestamps like "<email> 0123" (whereas we would
complain about "<email> 0123"). I doubt anybody cares, but I
mention it here for completeness.
- fsck_tree() does not need any modifications. It relies on
decode_tree_entry() to do the actual parsing, and that function
checks both that there are enough bytes in the buffer to represent
an entry, and that there is a NUL at the appropriate spot (one
hash-length from the end; this may not be the NUL for the entry we
are parsing, but we know that in the worst case, everything from our
current position to that NUL is a filename, so we won't run out of
bytes).
In addition to fixing the code itself, we'd like to make sure our rather
subtle assumptions are not violated in the future. So this patch does
two more things:
- add comments around verify_headers() documenting the link between
what it checks and the memory safety of the callers. I don't expect
this code to be modified frequently, but this may help somebody from
accidentally breaking things.
- add a thorough set of tests covering truncations at various key
spots (e.g., for a "tree $oid" line, in the middle of the word
"tree", right after it, after the space, in the middle of the $oid,
and right at the end of the line. Most of these are fine already (it
is only truncating right at the end of the line that is currently
broken). And some of them are not even possible with the current
code (we parse "tree " as a unit, so truncating before the space is
equivalent). But I aimed here to consider the code a black box and
look for any truncations that would be a problem for a left-to-right
parser.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-19 23:13:29 +00:00
|
|
|
while (buffer < buffer_end && skip_prefix(buffer, "author ", &buffer)) {
|
2015-06-22 15:26:23 +00:00
|
|
|
author_count++;
|
2019-10-18 05:01:48 +00:00
|
|
|
err = fsck_ident(&buffer, oid, OBJ_COMMIT, options);
|
2015-06-22 15:26:23 +00:00
|
|
|
if (err)
|
|
|
|
return err;
|
2008-02-25 21:46:08 +00:00
|
|
|
}
|
2015-06-22 15:26:23 +00:00
|
|
|
if (author_count < 1)
|
2019-10-18 05:01:48 +00:00
|
|
|
err = report(options, oid, OBJ_COMMIT, FSCK_MSG_MISSING_AUTHOR, "invalid format - expected 'author' line");
|
2015-06-22 15:26:23 +00:00
|
|
|
else if (author_count > 1)
|
2019-10-18 05:01:48 +00:00
|
|
|
err = report(options, oid, OBJ_COMMIT, FSCK_MSG_MULTIPLE_AUTHORS, "invalid format - multiple 'author' lines");
|
2010-04-24 16:06:08 +00:00
|
|
|
if (err)
|
|
|
|
return err;
|
fsck: do not assume NUL-termination of buffers
The fsck code operates on an object buffer represented as a pointer/len
combination. However, the parsing of commits and tags is a little bit
loose; we mostly scan left-to-right through the buffer, without checking
whether we've gone past the length we were given.
This has traditionally been OK because the buffers we feed to fsck
always have an extra NUL after the end of the object content, which ends
any left-to-right scan. That has always been true for objects we read
from the odb, and we made it true for incoming index-pack/unpack-objects
checks in a1e920a0a7 (index-pack: terminate object buffers with NUL,
2014-12-08).
However, we recently added an exception: hash-object asks index_fd() to
do fsck checks. That _may_ have an extra NUL (if we read from a pipe
into a strbuf), but it might not (if we read the contents from the
file). Nor can we just teach it to always add a NUL. We may mmap the
on-disk file, which will not have any extra bytes (if it's a multiple of
the page size). Not to mention that this is a rather subtle assumption
for the fsck code to make.
Instead, let's make sure that the fsck parsers don't ever look past the
size of the buffer they've been given. This _almost_ works already,
thanks to earlier work in 4d0d89755e (Make sure fsck_commit_buffer()
does not run out of the buffer, 2014-09-11). The theory there is that we
check up front whether we have the end of header double-newline
separator. And then any left-to-right scanning we do is OK as long as it
stops when it hits that boundary.
However, we later softened that in 84d18c0bcf (fsck: it is OK for a tag
and a commit to lack the body, 2015-06-28), which allows the
double-newline header to be missing, but does require that the header
ends in a newline. That was OK back then, because of the NUL-termination
guarantees (including the one from a1e920a0a7 mentioned above).
Because 84d18c0bcf guarantees that any header line does end in a
newline, we are still OK with most of the left-to-right scanning. We
only need to take care after completing a line, to check that there is
another line (and we didn't run out of buffer).
Most of these checks are just need to check "buffer < buffer_end" (where
buffer is advanced as we parse) before scanning for the next header
line. But here are a few notes:
- we don't technically need to check for remaining buffer before
parsing the very first line ("tree" for a commit, or "object" for a
tag), because verify_headers() rejects a totally empty buffer. But
we'll do so in the name of consistency and defensiveness.
- there are some calls to strchr('\n'). These are actually OK by the
"the final header line must end in a newline" guarantee from
verify_headers(). They will always find that rather than run off the
end of the buffer. Curiously, they do check for a NULL return and
complain, but I believe that condition can never be reached.
However, I converted them to use memchr() with a proper size and
retained the NULL checks. Using memchr() is not much longer and
makes it more obvious what is going on. Likewise, retaining the NULL
checks serves as a defensive measure in case my analysis is wrong.
- commit 9a1a3a4d4c (mktag: allow omitting the header/body \n
separator, 2021-01-05), does check for the end-of-buffer condition,
but does so with "!*buffer", relying explicitly on the NUL
termination. We can accomplish the same thing with a pointer
comparison. I also folded it into the follow-on conditional that
checks the contents of the buffer, for consistency with the other
checks.
- fsck_ident() uses parse_timestamp(), which is based on strtoumax().
That function will happily skip past leading whitespace, including
newlines, which makes it a risk. We can fix this by scanning to the
first digit ourselves, and then using parse_timestamp() to do the
actual numeric conversion.
Note that as a side effect this fixes the fact that we missed
zero-padded timestamps like "<email> 0123" (whereas we would
complain about "<email> 0123"). I doubt anybody cares, but I
mention it here for completeness.
- fsck_tree() does not need any modifications. It relies on
decode_tree_entry() to do the actual parsing, and that function
checks both that there are enough bytes in the buffer to represent
an entry, and that there is a NUL at the appropriate spot (one
hash-length from the end; this may not be the NUL for the entry we
are parsing, but we know that in the worst case, everything from our
current position to that NUL is a filename, so we won't run out of
bytes).
In addition to fixing the code itself, we'd like to make sure our rather
subtle assumptions are not violated in the future. So this patch does
two more things:
- add comments around verify_headers() documenting the link between
what it checks and the memory safety of the callers. I don't expect
this code to be modified frequently, but this may help somebody from
accidentally breaking things.
- add a thorough set of tests covering truncations at various key
spots (e.g., for a "tree $oid" line, in the middle of the word
"tree", right after it, after the space, in the middle of the $oid,
and right at the end of the line. Most of these are fine already (it
is only truncating right at the end of the line that is currently
broken). And some of them are not even possible with the current
code (we parse "tree " as a unit, so truncating before the space is
equivalent). But I aimed here to consider the code a black box and
look for any truncations that would be a problem for a left-to-right
parser.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-19 23:13:29 +00:00
|
|
|
if (buffer >= buffer_end || !skip_prefix(buffer, "committer ", &buffer))
|
2019-10-18 05:01:48 +00:00
|
|
|
return report(options, oid, OBJ_COMMIT, FSCK_MSG_MISSING_COMMITTER, "invalid format - expected 'committer' line");
|
|
|
|
err = fsck_ident(&buffer, oid, OBJ_COMMIT, options);
|
2010-04-24 16:06:08 +00:00
|
|
|
if (err)
|
|
|
|
return err;
|
2016-04-14 17:58:22 +00:00
|
|
|
if (memchr(buffer_begin, '\0', size)) {
|
2019-10-18 05:01:48 +00:00
|
|
|
err = report(options, oid, OBJ_COMMIT, FSCK_MSG_NUL_IN_COMMIT,
|
2016-04-14 17:58:22 +00:00
|
|
|
"NUL byte in the commit object body");
|
|
|
|
if (err)
|
|
|
|
return err;
|
|
|
|
}
|
2008-02-25 21:46:08 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2019-10-18 05:01:26 +00:00
|
|
|
static int fsck_tag(const struct object_id *oid, const char *buffer,
|
fsck: stop checking tag->tagged
Way back in 92d4c85d24 (fsck-cache: fix SIGSEGV on bad tag object,
2005-05-03), we added an fsck check that the "tagged" field of a tag
struct isn't NULL. But that was mainly protecting the printing code for
"--tags", and that code wasn't moved along with the check as part of
ba002f3b28 (builtin-fsck: move common object checking code to fsck.c,
2008-02-25).
It could also serve to detect type mismatch problems (where a tag points
to object X as a commit, but really X is a blob), but it couldn't do so
reliably (we'd call lookup_commit(X), but it will only notice the
problem if we happen to have previously called lookup_blob(X) in the
same process). And as of a commit earlier in this series, we'd consider
that a parse error and complain about the object even before getting to
this point anyway.
So let's drop this "tag->tagged" check. It's not helping anything, and
getting rid of it makes the function conceptually cleaner, as it really
is just checking the buffer we feed it. In fact, we can get rid of our
one-line wrapper and just unify fsck_tag() and fsck_tag_buffer().
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-18 04:51:19 +00:00
|
|
|
unsigned long size, struct fsck_options *options)
|
2014-09-11 14:26:38 +00:00
|
|
|
{
|
2019-10-18 05:00:59 +00:00
|
|
|
struct object_id tagged_oid;
|
mktag: use fsck instead of custom verify_tag()
Change the validation logic in "mktag" to use fsck's fsck_tag()
instead of its own custom parser. Curiously the logic for both dates
back to the same commit[1]. Let's unify them so we're not maintaining
two sets functions to verify that a tag is OK.
The behavior of fsck_tag() and the old "mktag" code being removed here
is different in few aspects.
I think it makes sense to remove some of those checks, namely:
A. fsck only cares that the timezone matches [-+][0-9]{4}. The mktag
code disallowed values larger than 1400.
Yes there's currently no timezone with a greater offset[2], but
since we allow any number of non-offical timezones (e.g. +1234)
passing this through seems fine. Git also won't break in the
future if e.g. French Polynesia decides it needs to outdo the Line
Islands when it comes to timezone extravagance.
B. fsck allows missing author names such as "tagger <email>", mktag
wouldn't, but would allow e.g. "tagger [2 spaces] <email>" (but
not "tagger [1 space] <email>"). Now we allow all of these.
C. Like B, but "mktag" disallowed spaces in the <email> part, fsck
allows it.
In some ways fsck_tag() is stricter than "mktag" was, namely:
D. fsck disallows zero-padded dates, but mktag didn't care. So
e.g. the timestamp "0000000000 +0000" produces an error now. A
test in "t1006-cat-file.sh" relied on this, it's been changed to
use "hash-object" (without fsck) instead.
There was one check I deemed worth keeping by porting it over to
fsck_tag():
E. "mktag" did not allow any custom headers, and by extension (as an
empty commit is allowed) also forbade an extra stray trailing
newline after the headers it knew about.
Add a new check in the "ignore" category to fsck and use it. This
somewhat abuses the facility added in efaba7cc77f (fsck:
optionally ignore specific fsck issues completely, 2015-06-22).
This is somewhat of hack, but probably the least invasive change
we can make here. The fsck command will shuffle these categories
around, e.g. under --strict the "info" becomes a "warn" and "warn"
becomes "error". Existing users of fsck's (and others,
e.g. index-pack) --strict option rely on this.
So we need to put something into a category that'll be ignored by
all existing users of the API. Pretending that
fsck.extraHeaderEntry=error ("ignore" by default) was set serves
to do this for us.
1. ec4465adb38 (Add "tag" objects that can be used to sign other
objects., 2005-04-25)
2. https://en.wikipedia.org/wiki/List_of_UTC_time_offsets
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-01-05 19:42:46 +00:00
|
|
|
int tagged_type;
|
|
|
|
return fsck_tag_standalone(oid, buffer, size, options, &tagged_oid,
|
|
|
|
&tagged_type);
|
|
|
|
}
|
|
|
|
|
|
|
|
int fsck_tag_standalone(const struct object_id *oid, const char *buffer,
|
|
|
|
unsigned long size, struct fsck_options *options,
|
|
|
|
struct object_id *tagged_oid,
|
|
|
|
int *tagged_type)
|
|
|
|
{
|
2014-09-11 14:26:38 +00:00
|
|
|
int ret = 0;
|
fsck: require an actual buffer for non-blobs
The fsck_object() function takes in a buffer, but also a "struct
object". The rules for using these vary between types:
- for a commit, we'll use the provided buffer; if it's NULL, we'll
fall back to get_commit_buffer(), which loads from either an
in-memory cache or from disk. If the latter fails, we'd die(), which
is non-ideal for fsck.
- for a tag, a NULL buffer will fall back to loading the object from
disk (and failure would lead to an fsck error)
- for a tree, we _never_ look at the provided buffer, and always use
tree->buffer
- for a blob, we usually don't look at the buffer at all, unless it
has been marked as a .gitmodule file. In that case we check the
buffer given to us, or assume a NULL buffer is a very large blob
(and complain about it)
This is much more complex than it needs to be. It turns out that nobody
ever feeds a NULL buffer that isn't a blob:
- git-fsck calls fsck_object() only from fsck_obj(). That in turn is
called by one of:
- fsck_obj_buffer(), which is a callback to verify_pack(), which
unpacks everything except large blobs into a buffer (see
pack-check.c, lines 131-141).
- fsck_loose(), which hits a BUG() on non-blobs with a NULL buffer
(builtin/fsck.c, lines 639-640)
And in either case, we'll have just called parse_object_buffer()
anyway, which would segfault on a NULL buffer for commits or tags
(not for trees, but it would install a NULL tree->buffer which would
later cause a segfault)
- git-index-pack asserts that the buffer is non-NULL unless the object
is a blob (see builtin/index-pack.c, line 832)
- git-unpack-objects always writes a non-NULL buffer into its
obj_buffer hash, which is then fed to fsck_object(). (There is
actually a funny thing here where it does not store blob buffers at
all, nor does it call fsck on them; it does check any needed blobs
via fsck_finish() though).
Let's make the rules simpler, which reduces the amount of code and gives
us more flexibility in refactoring the fsck code. The new rules are:
- only blobs are allowed to pass a NULL buffer
- we always use the provided buffer, never pulling information from
the object struct
We don't have to adjust any callers, because they were already adhering
to these. Note that we do drop a few fsck identifiers for missing tags,
but that was all dead code (because nobody passed a NULL tag buffer).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2019-10-18 04:54:12 +00:00
|
|
|
char *eol;
|
2014-09-11 14:26:38 +00:00
|
|
|
struct strbuf sb = STRBUF_INIT;
|
fsck: do not assume NUL-termination of buffers
The fsck code operates on an object buffer represented as a pointer/len
combination. However, the parsing of commits and tags is a little bit
loose; we mostly scan left-to-right through the buffer, without checking
whether we've gone past the length we were given.
This has traditionally been OK because the buffers we feed to fsck
always have an extra NUL after the end of the object content, which ends
any left-to-right scan. That has always been true for objects we read
from the odb, and we made it true for incoming index-pack/unpack-objects
checks in a1e920a0a7 (index-pack: terminate object buffers with NUL,
2014-12-08).
However, we recently added an exception: hash-object asks index_fd() to
do fsck checks. That _may_ have an extra NUL (if we read from a pipe
into a strbuf), but it might not (if we read the contents from the
file). Nor can we just teach it to always add a NUL. We may mmap the
on-disk file, which will not have any extra bytes (if it's a multiple of
the page size). Not to mention that this is a rather subtle assumption
for the fsck code to make.
Instead, let's make sure that the fsck parsers don't ever look past the
size of the buffer they've been given. This _almost_ works already,
thanks to earlier work in 4d0d89755e (Make sure fsck_commit_buffer()
does not run out of the buffer, 2014-09-11). The theory there is that we
check up front whether we have the end of header double-newline
separator. And then any left-to-right scanning we do is OK as long as it
stops when it hits that boundary.
However, we later softened that in 84d18c0bcf (fsck: it is OK for a tag
and a commit to lack the body, 2015-06-28), which allows the
double-newline header to be missing, but does require that the header
ends in a newline. That was OK back then, because of the NUL-termination
guarantees (including the one from a1e920a0a7 mentioned above).
Because 84d18c0bcf guarantees that any header line does end in a
newline, we are still OK with most of the left-to-right scanning. We
only need to take care after completing a line, to check that there is
another line (and we didn't run out of buffer).
Most of these checks are just need to check "buffer < buffer_end" (where
buffer is advanced as we parse) before scanning for the next header
line. But here are a few notes:
- we don't technically need to check for remaining buffer before
parsing the very first line ("tree" for a commit, or "object" for a
tag), because verify_headers() rejects a totally empty buffer. But
we'll do so in the name of consistency and defensiveness.
- there are some calls to strchr('\n'). These are actually OK by the
"the final header line must end in a newline" guarantee from
verify_headers(). They will always find that rather than run off the
end of the buffer. Curiously, they do check for a NULL return and
complain, but I believe that condition can never be reached.
However, I converted them to use memchr() with a proper size and
retained the NULL checks. Using memchr() is not much longer and
makes it more obvious what is going on. Likewise, retaining the NULL
checks serves as a defensive measure in case my analysis is wrong.
- commit 9a1a3a4d4c (mktag: allow omitting the header/body \n
separator, 2021-01-05), does check for the end-of-buffer condition,
but does so with "!*buffer", relying explicitly on the NUL
termination. We can accomplish the same thing with a pointer
comparison. I also folded it into the follow-on conditional that
checks the contents of the buffer, for consistency with the other
checks.
- fsck_ident() uses parse_timestamp(), which is based on strtoumax().
That function will happily skip past leading whitespace, including
newlines, which makes it a risk. We can fix this by scanning to the
first digit ourselves, and then using parse_timestamp() to do the
actual numeric conversion.
Note that as a side effect this fixes the fact that we missed
zero-padded timestamps like "<email> 0123" (whereas we would
complain about "<email> 0123"). I doubt anybody cares, but I
mention it here for completeness.
- fsck_tree() does not need any modifications. It relies on
decode_tree_entry() to do the actual parsing, and that function
checks both that there are enough bytes in the buffer to represent
an entry, and that there is a NUL at the appropriate spot (one
hash-length from the end; this may not be the NUL for the entry we
are parsing, but we know that in the worst case, everything from our
current position to that NUL is a filename, so we won't run out of
bytes).
In addition to fixing the code itself, we'd like to make sure our rather
subtle assumptions are not violated in the future. So this patch does
two more things:
- add comments around verify_headers() documenting the link between
what it checks and the memory safety of the callers. I don't expect
this code to be modified frequently, but this may help somebody from
accidentally breaking things.
- add a thorough set of tests covering truncations at various key
spots (e.g., for a "tree $oid" line, in the middle of the word
"tree", right after it, after the space, in the middle of the $oid,
and right at the end of the line. Most of these are fine already (it
is only truncating right at the end of the line that is currently
broken). And some of them are not even possible with the current
code (we parse "tree " as a unit, so truncating before the space is
equivalent). But I aimed here to consider the code a black box and
look for any truncations that would be a problem for a left-to-right
parser.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-19 23:13:29 +00:00
|
|
|
const char *buffer_end = buffer + size;
|
2018-05-02 00:25:41 +00:00
|
|
|
const char *p;
|
2014-09-11 14:26:38 +00:00
|
|
|
|
fsck: do not assume NUL-termination of buffers
The fsck code operates on an object buffer represented as a pointer/len
combination. However, the parsing of commits and tags is a little bit
loose; we mostly scan left-to-right through the buffer, without checking
whether we've gone past the length we were given.
This has traditionally been OK because the buffers we feed to fsck
always have an extra NUL after the end of the object content, which ends
any left-to-right scan. That has always been true for objects we read
from the odb, and we made it true for incoming index-pack/unpack-objects
checks in a1e920a0a7 (index-pack: terminate object buffers with NUL,
2014-12-08).
However, we recently added an exception: hash-object asks index_fd() to
do fsck checks. That _may_ have an extra NUL (if we read from a pipe
into a strbuf), but it might not (if we read the contents from the
file). Nor can we just teach it to always add a NUL. We may mmap the
on-disk file, which will not have any extra bytes (if it's a multiple of
the page size). Not to mention that this is a rather subtle assumption
for the fsck code to make.
Instead, let's make sure that the fsck parsers don't ever look past the
size of the buffer they've been given. This _almost_ works already,
thanks to earlier work in 4d0d89755e (Make sure fsck_commit_buffer()
does not run out of the buffer, 2014-09-11). The theory there is that we
check up front whether we have the end of header double-newline
separator. And then any left-to-right scanning we do is OK as long as it
stops when it hits that boundary.
However, we later softened that in 84d18c0bcf (fsck: it is OK for a tag
and a commit to lack the body, 2015-06-28), which allows the
double-newline header to be missing, but does require that the header
ends in a newline. That was OK back then, because of the NUL-termination
guarantees (including the one from a1e920a0a7 mentioned above).
Because 84d18c0bcf guarantees that any header line does end in a
newline, we are still OK with most of the left-to-right scanning. We
only need to take care after completing a line, to check that there is
another line (and we didn't run out of buffer).
Most of these checks are just need to check "buffer < buffer_end" (where
buffer is advanced as we parse) before scanning for the next header
line. But here are a few notes:
- we don't technically need to check for remaining buffer before
parsing the very first line ("tree" for a commit, or "object" for a
tag), because verify_headers() rejects a totally empty buffer. But
we'll do so in the name of consistency and defensiveness.
- there are some calls to strchr('\n'). These are actually OK by the
"the final header line must end in a newline" guarantee from
verify_headers(). They will always find that rather than run off the
end of the buffer. Curiously, they do check for a NULL return and
complain, but I believe that condition can never be reached.
However, I converted them to use memchr() with a proper size and
retained the NULL checks. Using memchr() is not much longer and
makes it more obvious what is going on. Likewise, retaining the NULL
checks serves as a defensive measure in case my analysis is wrong.
- commit 9a1a3a4d4c (mktag: allow omitting the header/body \n
separator, 2021-01-05), does check for the end-of-buffer condition,
but does so with "!*buffer", relying explicitly on the NUL
termination. We can accomplish the same thing with a pointer
comparison. I also folded it into the follow-on conditional that
checks the contents of the buffer, for consistency with the other
checks.
- fsck_ident() uses parse_timestamp(), which is based on strtoumax().
That function will happily skip past leading whitespace, including
newlines, which makes it a risk. We can fix this by scanning to the
first digit ourselves, and then using parse_timestamp() to do the
actual numeric conversion.
Note that as a side effect this fixes the fact that we missed
zero-padded timestamps like "<email> 0123" (whereas we would
complain about "<email> 0123"). I doubt anybody cares, but I
mention it here for completeness.
- fsck_tree() does not need any modifications. It relies on
decode_tree_entry() to do the actual parsing, and that function
checks both that there are enough bytes in the buffer to represent
an entry, and that there is a NUL at the appropriate spot (one
hash-length from the end; this may not be the NUL for the entry we
are parsing, but we know that in the worst case, everything from our
current position to that NUL is a filename, so we won't run out of
bytes).
In addition to fixing the code itself, we'd like to make sure our rather
subtle assumptions are not violated in the future. So this patch does
two more things:
- add comments around verify_headers() documenting the link between
what it checks and the memory safety of the callers. I don't expect
this code to be modified frequently, but this may help somebody from
accidentally breaking things.
- add a thorough set of tests covering truncations at various key
spots (e.g., for a "tree $oid" line, in the middle of the word
"tree", right after it, after the space, in the middle of the $oid,
and right at the end of the line. Most of these are fine already (it
is only truncating right at the end of the line that is currently
broken). And some of them are not even possible with the current
code (we parse "tree " as a unit, so truncating before the space is
equivalent). But I aimed here to consider the code a black box and
look for any truncations that would be a problem for a left-to-right
parser.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-19 23:13:29 +00:00
|
|
|
/*
|
|
|
|
* We _must_ stop parsing immediately if this reports failure, as the
|
|
|
|
* memory safety of the rest of the function depends on it. See the
|
|
|
|
* comment above the definition of verify_headers() for more details.
|
|
|
|
*/
|
2019-10-18 05:01:26 +00:00
|
|
|
ret = verify_headers(buffer, size, oid, OBJ_TAG, options);
|
2015-11-19 16:25:31 +00:00
|
|
|
if (ret)
|
2014-09-11 14:26:38 +00:00
|
|
|
goto done;
|
|
|
|
|
fsck: do not assume NUL-termination of buffers
The fsck code operates on an object buffer represented as a pointer/len
combination. However, the parsing of commits and tags is a little bit
loose; we mostly scan left-to-right through the buffer, without checking
whether we've gone past the length we were given.
This has traditionally been OK because the buffers we feed to fsck
always have an extra NUL after the end of the object content, which ends
any left-to-right scan. That has always been true for objects we read
from the odb, and we made it true for incoming index-pack/unpack-objects
checks in a1e920a0a7 (index-pack: terminate object buffers with NUL,
2014-12-08).
However, we recently added an exception: hash-object asks index_fd() to
do fsck checks. That _may_ have an extra NUL (if we read from a pipe
into a strbuf), but it might not (if we read the contents from the
file). Nor can we just teach it to always add a NUL. We may mmap the
on-disk file, which will not have any extra bytes (if it's a multiple of
the page size). Not to mention that this is a rather subtle assumption
for the fsck code to make.
Instead, let's make sure that the fsck parsers don't ever look past the
size of the buffer they've been given. This _almost_ works already,
thanks to earlier work in 4d0d89755e (Make sure fsck_commit_buffer()
does not run out of the buffer, 2014-09-11). The theory there is that we
check up front whether we have the end of header double-newline
separator. And then any left-to-right scanning we do is OK as long as it
stops when it hits that boundary.
However, we later softened that in 84d18c0bcf (fsck: it is OK for a tag
and a commit to lack the body, 2015-06-28), which allows the
double-newline header to be missing, but does require that the header
ends in a newline. That was OK back then, because of the NUL-termination
guarantees (including the one from a1e920a0a7 mentioned above).
Because 84d18c0bcf guarantees that any header line does end in a
newline, we are still OK with most of the left-to-right scanning. We
only need to take care after completing a line, to check that there is
another line (and we didn't run out of buffer).
Most of these checks are just need to check "buffer < buffer_end" (where
buffer is advanced as we parse) before scanning for the next header
line. But here are a few notes:
- we don't technically need to check for remaining buffer before
parsing the very first line ("tree" for a commit, or "object" for a
tag), because verify_headers() rejects a totally empty buffer. But
we'll do so in the name of consistency and defensiveness.
- there are some calls to strchr('\n'). These are actually OK by the
"the final header line must end in a newline" guarantee from
verify_headers(). They will always find that rather than run off the
end of the buffer. Curiously, they do check for a NULL return and
complain, but I believe that condition can never be reached.
However, I converted them to use memchr() with a proper size and
retained the NULL checks. Using memchr() is not much longer and
makes it more obvious what is going on. Likewise, retaining the NULL
checks serves as a defensive measure in case my analysis is wrong.
- commit 9a1a3a4d4c (mktag: allow omitting the header/body \n
separator, 2021-01-05), does check for the end-of-buffer condition,
but does so with "!*buffer", relying explicitly on the NUL
termination. We can accomplish the same thing with a pointer
comparison. I also folded it into the follow-on conditional that
checks the contents of the buffer, for consistency with the other
checks.
- fsck_ident() uses parse_timestamp(), which is based on strtoumax().
That function will happily skip past leading whitespace, including
newlines, which makes it a risk. We can fix this by scanning to the
first digit ourselves, and then using parse_timestamp() to do the
actual numeric conversion.
Note that as a side effect this fixes the fact that we missed
zero-padded timestamps like "<email> 0123" (whereas we would
complain about "<email> 0123"). I doubt anybody cares, but I
mention it here for completeness.
- fsck_tree() does not need any modifications. It relies on
decode_tree_entry() to do the actual parsing, and that function
checks both that there are enough bytes in the buffer to represent
an entry, and that there is a NUL at the appropriate spot (one
hash-length from the end; this may not be the NUL for the entry we
are parsing, but we know that in the worst case, everything from our
current position to that NUL is a filename, so we won't run out of
bytes).
In addition to fixing the code itself, we'd like to make sure our rather
subtle assumptions are not violated in the future. So this patch does
two more things:
- add comments around verify_headers() documenting the link between
what it checks and the memory safety of the callers. I don't expect
this code to be modified frequently, but this may help somebody from
accidentally breaking things.
- add a thorough set of tests covering truncations at various key
spots (e.g., for a "tree $oid" line, in the middle of the word
"tree", right after it, after the space, in the middle of the $oid,
and right at the end of the line. Most of these are fine already (it
is only truncating right at the end of the line that is currently
broken). And some of them are not even possible with the current
code (we parse "tree " as a unit, so truncating before the space is
equivalent). But I aimed here to consider the code a black box and
look for any truncations that would be a problem for a left-to-right
parser.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-19 23:13:29 +00:00
|
|
|
if (buffer >= buffer_end || !skip_prefix(buffer, "object ", &buffer)) {
|
2019-10-18 05:01:26 +00:00
|
|
|
ret = report(options, oid, OBJ_TAG, FSCK_MSG_MISSING_OBJECT, "invalid format - expected 'object' line");
|
2014-09-11 14:26:38 +00:00
|
|
|
goto done;
|
|
|
|
}
|
mktag: use fsck instead of custom verify_tag()
Change the validation logic in "mktag" to use fsck's fsck_tag()
instead of its own custom parser. Curiously the logic for both dates
back to the same commit[1]. Let's unify them so we're not maintaining
two sets functions to verify that a tag is OK.
The behavior of fsck_tag() and the old "mktag" code being removed here
is different in few aspects.
I think it makes sense to remove some of those checks, namely:
A. fsck only cares that the timezone matches [-+][0-9]{4}. The mktag
code disallowed values larger than 1400.
Yes there's currently no timezone with a greater offset[2], but
since we allow any number of non-offical timezones (e.g. +1234)
passing this through seems fine. Git also won't break in the
future if e.g. French Polynesia decides it needs to outdo the Line
Islands when it comes to timezone extravagance.
B. fsck allows missing author names such as "tagger <email>", mktag
wouldn't, but would allow e.g. "tagger [2 spaces] <email>" (but
not "tagger [1 space] <email>"). Now we allow all of these.
C. Like B, but "mktag" disallowed spaces in the <email> part, fsck
allows it.
In some ways fsck_tag() is stricter than "mktag" was, namely:
D. fsck disallows zero-padded dates, but mktag didn't care. So
e.g. the timestamp "0000000000 +0000" produces an error now. A
test in "t1006-cat-file.sh" relied on this, it's been changed to
use "hash-object" (without fsck) instead.
There was one check I deemed worth keeping by porting it over to
fsck_tag():
E. "mktag" did not allow any custom headers, and by extension (as an
empty commit is allowed) also forbade an extra stray trailing
newline after the headers it knew about.
Add a new check in the "ignore" category to fsck and use it. This
somewhat abuses the facility added in efaba7cc77f (fsck:
optionally ignore specific fsck issues completely, 2015-06-22).
This is somewhat of hack, but probably the least invasive change
we can make here. The fsck command will shuffle these categories
around, e.g. under --strict the "info" becomes a "warn" and "warn"
becomes "error". Existing users of fsck's (and others,
e.g. index-pack) --strict option rely on this.
So we need to put something into a category that'll be ignored by
all existing users of the API. Pretending that
fsck.extraHeaderEntry=error ("ignore" by default) was set serves
to do this for us.
1. ec4465adb38 (Add "tag" objects that can be used to sign other
objects., 2005-04-25)
2. https://en.wikipedia.org/wiki/List_of_UTC_time_offsets
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-01-05 19:42:46 +00:00
|
|
|
if (parse_oid_hex(buffer, tagged_oid, &p) || *p != '\n') {
|
2019-10-18 05:01:26 +00:00
|
|
|
ret = report(options, oid, OBJ_TAG, FSCK_MSG_BAD_OBJECT_SHA1, "invalid 'object' line format - bad sha1");
|
2015-06-22 15:26:30 +00:00
|
|
|
if (ret)
|
|
|
|
goto done;
|
2014-09-11 14:26:38 +00:00
|
|
|
}
|
2018-05-02 00:25:41 +00:00
|
|
|
buffer = p + 1;
|
2014-09-11 14:26:38 +00:00
|
|
|
|
fsck: do not assume NUL-termination of buffers
The fsck code operates on an object buffer represented as a pointer/len
combination. However, the parsing of commits and tags is a little bit
loose; we mostly scan left-to-right through the buffer, without checking
whether we've gone past the length we were given.
This has traditionally been OK because the buffers we feed to fsck
always have an extra NUL after the end of the object content, which ends
any left-to-right scan. That has always been true for objects we read
from the odb, and we made it true for incoming index-pack/unpack-objects
checks in a1e920a0a7 (index-pack: terminate object buffers with NUL,
2014-12-08).
However, we recently added an exception: hash-object asks index_fd() to
do fsck checks. That _may_ have an extra NUL (if we read from a pipe
into a strbuf), but it might not (if we read the contents from the
file). Nor can we just teach it to always add a NUL. We may mmap the
on-disk file, which will not have any extra bytes (if it's a multiple of
the page size). Not to mention that this is a rather subtle assumption
for the fsck code to make.
Instead, let's make sure that the fsck parsers don't ever look past the
size of the buffer they've been given. This _almost_ works already,
thanks to earlier work in 4d0d89755e (Make sure fsck_commit_buffer()
does not run out of the buffer, 2014-09-11). The theory there is that we
check up front whether we have the end of header double-newline
separator. And then any left-to-right scanning we do is OK as long as it
stops when it hits that boundary.
However, we later softened that in 84d18c0bcf (fsck: it is OK for a tag
and a commit to lack the body, 2015-06-28), which allows the
double-newline header to be missing, but does require that the header
ends in a newline. That was OK back then, because of the NUL-termination
guarantees (including the one from a1e920a0a7 mentioned above).
Because 84d18c0bcf guarantees that any header line does end in a
newline, we are still OK with most of the left-to-right scanning. We
only need to take care after completing a line, to check that there is
another line (and we didn't run out of buffer).
Most of these checks are just need to check "buffer < buffer_end" (where
buffer is advanced as we parse) before scanning for the next header
line. But here are a few notes:
- we don't technically need to check for remaining buffer before
parsing the very first line ("tree" for a commit, or "object" for a
tag), because verify_headers() rejects a totally empty buffer. But
we'll do so in the name of consistency and defensiveness.
- there are some calls to strchr('\n'). These are actually OK by the
"the final header line must end in a newline" guarantee from
verify_headers(). They will always find that rather than run off the
end of the buffer. Curiously, they do check for a NULL return and
complain, but I believe that condition can never be reached.
However, I converted them to use memchr() with a proper size and
retained the NULL checks. Using memchr() is not much longer and
makes it more obvious what is going on. Likewise, retaining the NULL
checks serves as a defensive measure in case my analysis is wrong.
- commit 9a1a3a4d4c (mktag: allow omitting the header/body \n
separator, 2021-01-05), does check for the end-of-buffer condition,
but does so with "!*buffer", relying explicitly on the NUL
termination. We can accomplish the same thing with a pointer
comparison. I also folded it into the follow-on conditional that
checks the contents of the buffer, for consistency with the other
checks.
- fsck_ident() uses parse_timestamp(), which is based on strtoumax().
That function will happily skip past leading whitespace, including
newlines, which makes it a risk. We can fix this by scanning to the
first digit ourselves, and then using parse_timestamp() to do the
actual numeric conversion.
Note that as a side effect this fixes the fact that we missed
zero-padded timestamps like "<email> 0123" (whereas we would
complain about "<email> 0123"). I doubt anybody cares, but I
mention it here for completeness.
- fsck_tree() does not need any modifications. It relies on
decode_tree_entry() to do the actual parsing, and that function
checks both that there are enough bytes in the buffer to represent
an entry, and that there is a NUL at the appropriate spot (one
hash-length from the end; this may not be the NUL for the entry we
are parsing, but we know that in the worst case, everything from our
current position to that NUL is a filename, so we won't run out of
bytes).
In addition to fixing the code itself, we'd like to make sure our rather
subtle assumptions are not violated in the future. So this patch does
two more things:
- add comments around verify_headers() documenting the link between
what it checks and the memory safety of the callers. I don't expect
this code to be modified frequently, but this may help somebody from
accidentally breaking things.
- add a thorough set of tests covering truncations at various key
spots (e.g., for a "tree $oid" line, in the middle of the word
"tree", right after it, after the space, in the middle of the $oid,
and right at the end of the line. Most of these are fine already (it
is only truncating right at the end of the line that is currently
broken). And some of them are not even possible with the current
code (we parse "tree " as a unit, so truncating before the space is
equivalent). But I aimed here to consider the code a black box and
look for any truncations that would be a problem for a left-to-right
parser.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-19 23:13:29 +00:00
|
|
|
if (buffer >= buffer_end || !skip_prefix(buffer, "type ", &buffer)) {
|
2019-10-18 05:01:26 +00:00
|
|
|
ret = report(options, oid, OBJ_TAG, FSCK_MSG_MISSING_TYPE_ENTRY, "invalid format - expected 'type' line");
|
2014-09-11 14:26:38 +00:00
|
|
|
goto done;
|
|
|
|
}
|
fsck: do not assume NUL-termination of buffers
The fsck code operates on an object buffer represented as a pointer/len
combination. However, the parsing of commits and tags is a little bit
loose; we mostly scan left-to-right through the buffer, without checking
whether we've gone past the length we were given.
This has traditionally been OK because the buffers we feed to fsck
always have an extra NUL after the end of the object content, which ends
any left-to-right scan. That has always been true for objects we read
from the odb, and we made it true for incoming index-pack/unpack-objects
checks in a1e920a0a7 (index-pack: terminate object buffers with NUL,
2014-12-08).
However, we recently added an exception: hash-object asks index_fd() to
do fsck checks. That _may_ have an extra NUL (if we read from a pipe
into a strbuf), but it might not (if we read the contents from the
file). Nor can we just teach it to always add a NUL. We may mmap the
on-disk file, which will not have any extra bytes (if it's a multiple of
the page size). Not to mention that this is a rather subtle assumption
for the fsck code to make.
Instead, let's make sure that the fsck parsers don't ever look past the
size of the buffer they've been given. This _almost_ works already,
thanks to earlier work in 4d0d89755e (Make sure fsck_commit_buffer()
does not run out of the buffer, 2014-09-11). The theory there is that we
check up front whether we have the end of header double-newline
separator. And then any left-to-right scanning we do is OK as long as it
stops when it hits that boundary.
However, we later softened that in 84d18c0bcf (fsck: it is OK for a tag
and a commit to lack the body, 2015-06-28), which allows the
double-newline header to be missing, but does require that the header
ends in a newline. That was OK back then, because of the NUL-termination
guarantees (including the one from a1e920a0a7 mentioned above).
Because 84d18c0bcf guarantees that any header line does end in a
newline, we are still OK with most of the left-to-right scanning. We
only need to take care after completing a line, to check that there is
another line (and we didn't run out of buffer).
Most of these checks are just need to check "buffer < buffer_end" (where
buffer is advanced as we parse) before scanning for the next header
line. But here are a few notes:
- we don't technically need to check for remaining buffer before
parsing the very first line ("tree" for a commit, or "object" for a
tag), because verify_headers() rejects a totally empty buffer. But
we'll do so in the name of consistency and defensiveness.
- there are some calls to strchr('\n'). These are actually OK by the
"the final header line must end in a newline" guarantee from
verify_headers(). They will always find that rather than run off the
end of the buffer. Curiously, they do check for a NULL return and
complain, but I believe that condition can never be reached.
However, I converted them to use memchr() with a proper size and
retained the NULL checks. Using memchr() is not much longer and
makes it more obvious what is going on. Likewise, retaining the NULL
checks serves as a defensive measure in case my analysis is wrong.
- commit 9a1a3a4d4c (mktag: allow omitting the header/body \n
separator, 2021-01-05), does check for the end-of-buffer condition,
but does so with "!*buffer", relying explicitly on the NUL
termination. We can accomplish the same thing with a pointer
comparison. I also folded it into the follow-on conditional that
checks the contents of the buffer, for consistency with the other
checks.
- fsck_ident() uses parse_timestamp(), which is based on strtoumax().
That function will happily skip past leading whitespace, including
newlines, which makes it a risk. We can fix this by scanning to the
first digit ourselves, and then using parse_timestamp() to do the
actual numeric conversion.
Note that as a side effect this fixes the fact that we missed
zero-padded timestamps like "<email> 0123" (whereas we would
complain about "<email> 0123"). I doubt anybody cares, but I
mention it here for completeness.
- fsck_tree() does not need any modifications. It relies on
decode_tree_entry() to do the actual parsing, and that function
checks both that there are enough bytes in the buffer to represent
an entry, and that there is a NUL at the appropriate spot (one
hash-length from the end; this may not be the NUL for the entry we
are parsing, but we know that in the worst case, everything from our
current position to that NUL is a filename, so we won't run out of
bytes).
In addition to fixing the code itself, we'd like to make sure our rather
subtle assumptions are not violated in the future. So this patch does
two more things:
- add comments around verify_headers() documenting the link between
what it checks and the memory safety of the callers. I don't expect
this code to be modified frequently, but this may help somebody from
accidentally breaking things.
- add a thorough set of tests covering truncations at various key
spots (e.g., for a "tree $oid" line, in the middle of the word
"tree", right after it, after the space, in the middle of the $oid,
and right at the end of the line. Most of these are fine already (it
is only truncating right at the end of the line that is currently
broken). And some of them are not even possible with the current
code (we parse "tree " as a unit, so truncating before the space is
equivalent). But I aimed here to consider the code a black box and
look for any truncations that would be a problem for a left-to-right
parser.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-19 23:13:29 +00:00
|
|
|
eol = memchr(buffer, '\n', buffer_end - buffer);
|
2014-09-11 14:26:38 +00:00
|
|
|
if (!eol) {
|
2019-10-18 05:01:26 +00:00
|
|
|
ret = report(options, oid, OBJ_TAG, FSCK_MSG_MISSING_TYPE, "invalid format - unexpected end after 'type' line");
|
2014-09-11 14:26:38 +00:00
|
|
|
goto done;
|
|
|
|
}
|
mktag: use fsck instead of custom verify_tag()
Change the validation logic in "mktag" to use fsck's fsck_tag()
instead of its own custom parser. Curiously the logic for both dates
back to the same commit[1]. Let's unify them so we're not maintaining
two sets functions to verify that a tag is OK.
The behavior of fsck_tag() and the old "mktag" code being removed here
is different in few aspects.
I think it makes sense to remove some of those checks, namely:
A. fsck only cares that the timezone matches [-+][0-9]{4}. The mktag
code disallowed values larger than 1400.
Yes there's currently no timezone with a greater offset[2], but
since we allow any number of non-offical timezones (e.g. +1234)
passing this through seems fine. Git also won't break in the
future if e.g. French Polynesia decides it needs to outdo the Line
Islands when it comes to timezone extravagance.
B. fsck allows missing author names such as "tagger <email>", mktag
wouldn't, but would allow e.g. "tagger [2 spaces] <email>" (but
not "tagger [1 space] <email>"). Now we allow all of these.
C. Like B, but "mktag" disallowed spaces in the <email> part, fsck
allows it.
In some ways fsck_tag() is stricter than "mktag" was, namely:
D. fsck disallows zero-padded dates, but mktag didn't care. So
e.g. the timestamp "0000000000 +0000" produces an error now. A
test in "t1006-cat-file.sh" relied on this, it's been changed to
use "hash-object" (without fsck) instead.
There was one check I deemed worth keeping by porting it over to
fsck_tag():
E. "mktag" did not allow any custom headers, and by extension (as an
empty commit is allowed) also forbade an extra stray trailing
newline after the headers it knew about.
Add a new check in the "ignore" category to fsck and use it. This
somewhat abuses the facility added in efaba7cc77f (fsck:
optionally ignore specific fsck issues completely, 2015-06-22).
This is somewhat of hack, but probably the least invasive change
we can make here. The fsck command will shuffle these categories
around, e.g. under --strict the "info" becomes a "warn" and "warn"
becomes "error". Existing users of fsck's (and others,
e.g. index-pack) --strict option rely on this.
So we need to put something into a category that'll be ignored by
all existing users of the API. Pretending that
fsck.extraHeaderEntry=error ("ignore" by default) was set serves
to do this for us.
1. ec4465adb38 (Add "tag" objects that can be used to sign other
objects., 2005-04-25)
2. https://en.wikipedia.org/wiki/List_of_UTC_time_offsets
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-01-05 19:42:46 +00:00
|
|
|
*tagged_type = type_from_string_gently(buffer, eol - buffer, 1);
|
|
|
|
if (*tagged_type < 0)
|
2019-10-18 05:01:26 +00:00
|
|
|
ret = report(options, oid, OBJ_TAG, FSCK_MSG_BAD_TYPE, "invalid 'type' value");
|
2014-09-11 14:26:38 +00:00
|
|
|
if (ret)
|
|
|
|
goto done;
|
|
|
|
buffer = eol + 1;
|
|
|
|
|
fsck: do not assume NUL-termination of buffers
The fsck code operates on an object buffer represented as a pointer/len
combination. However, the parsing of commits and tags is a little bit
loose; we mostly scan left-to-right through the buffer, without checking
whether we've gone past the length we were given.
This has traditionally been OK because the buffers we feed to fsck
always have an extra NUL after the end of the object content, which ends
any left-to-right scan. That has always been true for objects we read
from the odb, and we made it true for incoming index-pack/unpack-objects
checks in a1e920a0a7 (index-pack: terminate object buffers with NUL,
2014-12-08).
However, we recently added an exception: hash-object asks index_fd() to
do fsck checks. That _may_ have an extra NUL (if we read from a pipe
into a strbuf), but it might not (if we read the contents from the
file). Nor can we just teach it to always add a NUL. We may mmap the
on-disk file, which will not have any extra bytes (if it's a multiple of
the page size). Not to mention that this is a rather subtle assumption
for the fsck code to make.
Instead, let's make sure that the fsck parsers don't ever look past the
size of the buffer they've been given. This _almost_ works already,
thanks to earlier work in 4d0d89755e (Make sure fsck_commit_buffer()
does not run out of the buffer, 2014-09-11). The theory there is that we
check up front whether we have the end of header double-newline
separator. And then any left-to-right scanning we do is OK as long as it
stops when it hits that boundary.
However, we later softened that in 84d18c0bcf (fsck: it is OK for a tag
and a commit to lack the body, 2015-06-28), which allows the
double-newline header to be missing, but does require that the header
ends in a newline. That was OK back then, because of the NUL-termination
guarantees (including the one from a1e920a0a7 mentioned above).
Because 84d18c0bcf guarantees that any header line does end in a
newline, we are still OK with most of the left-to-right scanning. We
only need to take care after completing a line, to check that there is
another line (and we didn't run out of buffer).
Most of these checks are just need to check "buffer < buffer_end" (where
buffer is advanced as we parse) before scanning for the next header
line. But here are a few notes:
- we don't technically need to check for remaining buffer before
parsing the very first line ("tree" for a commit, or "object" for a
tag), because verify_headers() rejects a totally empty buffer. But
we'll do so in the name of consistency and defensiveness.
- there are some calls to strchr('\n'). These are actually OK by the
"the final header line must end in a newline" guarantee from
verify_headers(). They will always find that rather than run off the
end of the buffer. Curiously, they do check for a NULL return and
complain, but I believe that condition can never be reached.
However, I converted them to use memchr() with a proper size and
retained the NULL checks. Using memchr() is not much longer and
makes it more obvious what is going on. Likewise, retaining the NULL
checks serves as a defensive measure in case my analysis is wrong.
- commit 9a1a3a4d4c (mktag: allow omitting the header/body \n
separator, 2021-01-05), does check for the end-of-buffer condition,
but does so with "!*buffer", relying explicitly on the NUL
termination. We can accomplish the same thing with a pointer
comparison. I also folded it into the follow-on conditional that
checks the contents of the buffer, for consistency with the other
checks.
- fsck_ident() uses parse_timestamp(), which is based on strtoumax().
That function will happily skip past leading whitespace, including
newlines, which makes it a risk. We can fix this by scanning to the
first digit ourselves, and then using parse_timestamp() to do the
actual numeric conversion.
Note that as a side effect this fixes the fact that we missed
zero-padded timestamps like "<email> 0123" (whereas we would
complain about "<email> 0123"). I doubt anybody cares, but I
mention it here for completeness.
- fsck_tree() does not need any modifications. It relies on
decode_tree_entry() to do the actual parsing, and that function
checks both that there are enough bytes in the buffer to represent
an entry, and that there is a NUL at the appropriate spot (one
hash-length from the end; this may not be the NUL for the entry we
are parsing, but we know that in the worst case, everything from our
current position to that NUL is a filename, so we won't run out of
bytes).
In addition to fixing the code itself, we'd like to make sure our rather
subtle assumptions are not violated in the future. So this patch does
two more things:
- add comments around verify_headers() documenting the link between
what it checks and the memory safety of the callers. I don't expect
this code to be modified frequently, but this may help somebody from
accidentally breaking things.
- add a thorough set of tests covering truncations at various key
spots (e.g., for a "tree $oid" line, in the middle of the word
"tree", right after it, after the space, in the middle of the $oid,
and right at the end of the line. Most of these are fine already (it
is only truncating right at the end of the line that is currently
broken). And some of them are not even possible with the current
code (we parse "tree " as a unit, so truncating before the space is
equivalent). But I aimed here to consider the code a black box and
look for any truncations that would be a problem for a left-to-right
parser.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-19 23:13:29 +00:00
|
|
|
if (buffer >= buffer_end || !skip_prefix(buffer, "tag ", &buffer)) {
|
2019-10-18 05:01:26 +00:00
|
|
|
ret = report(options, oid, OBJ_TAG, FSCK_MSG_MISSING_TAG_ENTRY, "invalid format - expected 'tag' line");
|
2014-09-11 14:26:38 +00:00
|
|
|
goto done;
|
|
|
|
}
|
fsck: do not assume NUL-termination of buffers
The fsck code operates on an object buffer represented as a pointer/len
combination. However, the parsing of commits and tags is a little bit
loose; we mostly scan left-to-right through the buffer, without checking
whether we've gone past the length we were given.
This has traditionally been OK because the buffers we feed to fsck
always have an extra NUL after the end of the object content, which ends
any left-to-right scan. That has always been true for objects we read
from the odb, and we made it true for incoming index-pack/unpack-objects
checks in a1e920a0a7 (index-pack: terminate object buffers with NUL,
2014-12-08).
However, we recently added an exception: hash-object asks index_fd() to
do fsck checks. That _may_ have an extra NUL (if we read from a pipe
into a strbuf), but it might not (if we read the contents from the
file). Nor can we just teach it to always add a NUL. We may mmap the
on-disk file, which will not have any extra bytes (if it's a multiple of
the page size). Not to mention that this is a rather subtle assumption
for the fsck code to make.
Instead, let's make sure that the fsck parsers don't ever look past the
size of the buffer they've been given. This _almost_ works already,
thanks to earlier work in 4d0d89755e (Make sure fsck_commit_buffer()
does not run out of the buffer, 2014-09-11). The theory there is that we
check up front whether we have the end of header double-newline
separator. And then any left-to-right scanning we do is OK as long as it
stops when it hits that boundary.
However, we later softened that in 84d18c0bcf (fsck: it is OK for a tag
and a commit to lack the body, 2015-06-28), which allows the
double-newline header to be missing, but does require that the header
ends in a newline. That was OK back then, because of the NUL-termination
guarantees (including the one from a1e920a0a7 mentioned above).
Because 84d18c0bcf guarantees that any header line does end in a
newline, we are still OK with most of the left-to-right scanning. We
only need to take care after completing a line, to check that there is
another line (and we didn't run out of buffer).
Most of these checks are just need to check "buffer < buffer_end" (where
buffer is advanced as we parse) before scanning for the next header
line. But here are a few notes:
- we don't technically need to check for remaining buffer before
parsing the very first line ("tree" for a commit, or "object" for a
tag), because verify_headers() rejects a totally empty buffer. But
we'll do so in the name of consistency and defensiveness.
- there are some calls to strchr('\n'). These are actually OK by the
"the final header line must end in a newline" guarantee from
verify_headers(). They will always find that rather than run off the
end of the buffer. Curiously, they do check for a NULL return and
complain, but I believe that condition can never be reached.
However, I converted them to use memchr() with a proper size and
retained the NULL checks. Using memchr() is not much longer and
makes it more obvious what is going on. Likewise, retaining the NULL
checks serves as a defensive measure in case my analysis is wrong.
- commit 9a1a3a4d4c (mktag: allow omitting the header/body \n
separator, 2021-01-05), does check for the end-of-buffer condition,
but does so with "!*buffer", relying explicitly on the NUL
termination. We can accomplish the same thing with a pointer
comparison. I also folded it into the follow-on conditional that
checks the contents of the buffer, for consistency with the other
checks.
- fsck_ident() uses parse_timestamp(), which is based on strtoumax().
That function will happily skip past leading whitespace, including
newlines, which makes it a risk. We can fix this by scanning to the
first digit ourselves, and then using parse_timestamp() to do the
actual numeric conversion.
Note that as a side effect this fixes the fact that we missed
zero-padded timestamps like "<email> 0123" (whereas we would
complain about "<email> 0123"). I doubt anybody cares, but I
mention it here for completeness.
- fsck_tree() does not need any modifications. It relies on
decode_tree_entry() to do the actual parsing, and that function
checks both that there are enough bytes in the buffer to represent
an entry, and that there is a NUL at the appropriate spot (one
hash-length from the end; this may not be the NUL for the entry we
are parsing, but we know that in the worst case, everything from our
current position to that NUL is a filename, so we won't run out of
bytes).
In addition to fixing the code itself, we'd like to make sure our rather
subtle assumptions are not violated in the future. So this patch does
two more things:
- add comments around verify_headers() documenting the link between
what it checks and the memory safety of the callers. I don't expect
this code to be modified frequently, but this may help somebody from
accidentally breaking things.
- add a thorough set of tests covering truncations at various key
spots (e.g., for a "tree $oid" line, in the middle of the word
"tree", right after it, after the space, in the middle of the $oid,
and right at the end of the line. Most of these are fine already (it
is only truncating right at the end of the line that is currently
broken). And some of them are not even possible with the current
code (we parse "tree " as a unit, so truncating before the space is
equivalent). But I aimed here to consider the code a black box and
look for any truncations that would be a problem for a left-to-right
parser.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-19 23:13:29 +00:00
|
|
|
eol = memchr(buffer, '\n', buffer_end - buffer);
|
2014-09-11 14:26:38 +00:00
|
|
|
if (!eol) {
|
2019-10-18 05:01:26 +00:00
|
|
|
ret = report(options, oid, OBJ_TAG, FSCK_MSG_MISSING_TAG, "invalid format - unexpected end after 'type' line");
|
2014-09-11 14:26:38 +00:00
|
|
|
goto done;
|
|
|
|
}
|
|
|
|
strbuf_addf(&sb, "refs/tags/%.*s", (int)(eol - buffer), buffer);
|
2015-06-22 15:26:54 +00:00
|
|
|
if (check_refname_format(sb.buf, 0)) {
|
2019-10-18 05:01:26 +00:00
|
|
|
ret = report(options, oid, OBJ_TAG,
|
2019-10-18 04:59:15 +00:00
|
|
|
FSCK_MSG_BAD_TAG_NAME,
|
|
|
|
"invalid 'tag' name: %.*s",
|
|
|
|
(int)(eol - buffer), buffer);
|
2015-06-22 15:26:54 +00:00
|
|
|
if (ret)
|
|
|
|
goto done;
|
|
|
|
}
|
2014-09-11 14:26:38 +00:00
|
|
|
buffer = eol + 1;
|
|
|
|
|
fsck: do not assume NUL-termination of buffers
The fsck code operates on an object buffer represented as a pointer/len
combination. However, the parsing of commits and tags is a little bit
loose; we mostly scan left-to-right through the buffer, without checking
whether we've gone past the length we were given.
This has traditionally been OK because the buffers we feed to fsck
always have an extra NUL after the end of the object content, which ends
any left-to-right scan. That has always been true for objects we read
from the odb, and we made it true for incoming index-pack/unpack-objects
checks in a1e920a0a7 (index-pack: terminate object buffers with NUL,
2014-12-08).
However, we recently added an exception: hash-object asks index_fd() to
do fsck checks. That _may_ have an extra NUL (if we read from a pipe
into a strbuf), but it might not (if we read the contents from the
file). Nor can we just teach it to always add a NUL. We may mmap the
on-disk file, which will not have any extra bytes (if it's a multiple of
the page size). Not to mention that this is a rather subtle assumption
for the fsck code to make.
Instead, let's make sure that the fsck parsers don't ever look past the
size of the buffer they've been given. This _almost_ works already,
thanks to earlier work in 4d0d89755e (Make sure fsck_commit_buffer()
does not run out of the buffer, 2014-09-11). The theory there is that we
check up front whether we have the end of header double-newline
separator. And then any left-to-right scanning we do is OK as long as it
stops when it hits that boundary.
However, we later softened that in 84d18c0bcf (fsck: it is OK for a tag
and a commit to lack the body, 2015-06-28), which allows the
double-newline header to be missing, but does require that the header
ends in a newline. That was OK back then, because of the NUL-termination
guarantees (including the one from a1e920a0a7 mentioned above).
Because 84d18c0bcf guarantees that any header line does end in a
newline, we are still OK with most of the left-to-right scanning. We
only need to take care after completing a line, to check that there is
another line (and we didn't run out of buffer).
Most of these checks are just need to check "buffer < buffer_end" (where
buffer is advanced as we parse) before scanning for the next header
line. But here are a few notes:
- we don't technically need to check for remaining buffer before
parsing the very first line ("tree" for a commit, or "object" for a
tag), because verify_headers() rejects a totally empty buffer. But
we'll do so in the name of consistency and defensiveness.
- there are some calls to strchr('\n'). These are actually OK by the
"the final header line must end in a newline" guarantee from
verify_headers(). They will always find that rather than run off the
end of the buffer. Curiously, they do check for a NULL return and
complain, but I believe that condition can never be reached.
However, I converted them to use memchr() with a proper size and
retained the NULL checks. Using memchr() is not much longer and
makes it more obvious what is going on. Likewise, retaining the NULL
checks serves as a defensive measure in case my analysis is wrong.
- commit 9a1a3a4d4c (mktag: allow omitting the header/body \n
separator, 2021-01-05), does check for the end-of-buffer condition,
but does so with "!*buffer", relying explicitly on the NUL
termination. We can accomplish the same thing with a pointer
comparison. I also folded it into the follow-on conditional that
checks the contents of the buffer, for consistency with the other
checks.
- fsck_ident() uses parse_timestamp(), which is based on strtoumax().
That function will happily skip past leading whitespace, including
newlines, which makes it a risk. We can fix this by scanning to the
first digit ourselves, and then using parse_timestamp() to do the
actual numeric conversion.
Note that as a side effect this fixes the fact that we missed
zero-padded timestamps like "<email> 0123" (whereas we would
complain about "<email> 0123"). I doubt anybody cares, but I
mention it here for completeness.
- fsck_tree() does not need any modifications. It relies on
decode_tree_entry() to do the actual parsing, and that function
checks both that there are enough bytes in the buffer to represent
an entry, and that there is a NUL at the appropriate spot (one
hash-length from the end; this may not be the NUL for the entry we
are parsing, but we know that in the worst case, everything from our
current position to that NUL is a filename, so we won't run out of
bytes).
In addition to fixing the code itself, we'd like to make sure our rather
subtle assumptions are not violated in the future. So this patch does
two more things:
- add comments around verify_headers() documenting the link between
what it checks and the memory safety of the callers. I don't expect
this code to be modified frequently, but this may help somebody from
accidentally breaking things.
- add a thorough set of tests covering truncations at various key
spots (e.g., for a "tree $oid" line, in the middle of the word
"tree", right after it, after the space, in the middle of the $oid,
and right at the end of the line. Most of these are fine already (it
is only truncating right at the end of the line that is currently
broken). And some of them are not even possible with the current
code (we parse "tree " as a unit, so truncating before the space is
equivalent). But I aimed here to consider the code a black box and
look for any truncations that would be a problem for a left-to-right
parser.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-19 23:13:29 +00:00
|
|
|
if (buffer >= buffer_end || !skip_prefix(buffer, "tagger ", &buffer)) {
|
2014-09-11 14:26:38 +00:00
|
|
|
/* early tags do not contain 'tagger' lines; warn only */
|
2019-10-18 05:01:26 +00:00
|
|
|
ret = report(options, oid, OBJ_TAG, FSCK_MSG_MISSING_TAGGER_ENTRY, "invalid format - expected 'tagger' line");
|
2015-06-22 15:26:54 +00:00
|
|
|
if (ret)
|
|
|
|
goto done;
|
|
|
|
}
|
2014-09-11 14:26:38 +00:00
|
|
|
else
|
2019-10-18 05:01:26 +00:00
|
|
|
ret = fsck_ident(&buffer, oid, OBJ_TAG, options);
|
2014-09-11 14:26:38 +00:00
|
|
|
|
fsck: do not assume NUL-termination of buffers
The fsck code operates on an object buffer represented as a pointer/len
combination. However, the parsing of commits and tags is a little bit
loose; we mostly scan left-to-right through the buffer, without checking
whether we've gone past the length we were given.
This has traditionally been OK because the buffers we feed to fsck
always have an extra NUL after the end of the object content, which ends
any left-to-right scan. That has always been true for objects we read
from the odb, and we made it true for incoming index-pack/unpack-objects
checks in a1e920a0a7 (index-pack: terminate object buffers with NUL,
2014-12-08).
However, we recently added an exception: hash-object asks index_fd() to
do fsck checks. That _may_ have an extra NUL (if we read from a pipe
into a strbuf), but it might not (if we read the contents from the
file). Nor can we just teach it to always add a NUL. We may mmap the
on-disk file, which will not have any extra bytes (if it's a multiple of
the page size). Not to mention that this is a rather subtle assumption
for the fsck code to make.
Instead, let's make sure that the fsck parsers don't ever look past the
size of the buffer they've been given. This _almost_ works already,
thanks to earlier work in 4d0d89755e (Make sure fsck_commit_buffer()
does not run out of the buffer, 2014-09-11). The theory there is that we
check up front whether we have the end of header double-newline
separator. And then any left-to-right scanning we do is OK as long as it
stops when it hits that boundary.
However, we later softened that in 84d18c0bcf (fsck: it is OK for a tag
and a commit to lack the body, 2015-06-28), which allows the
double-newline header to be missing, but does require that the header
ends in a newline. That was OK back then, because of the NUL-termination
guarantees (including the one from a1e920a0a7 mentioned above).
Because 84d18c0bcf guarantees that any header line does end in a
newline, we are still OK with most of the left-to-right scanning. We
only need to take care after completing a line, to check that there is
another line (and we didn't run out of buffer).
Most of these checks are just need to check "buffer < buffer_end" (where
buffer is advanced as we parse) before scanning for the next header
line. But here are a few notes:
- we don't technically need to check for remaining buffer before
parsing the very first line ("tree" for a commit, or "object" for a
tag), because verify_headers() rejects a totally empty buffer. But
we'll do so in the name of consistency and defensiveness.
- there are some calls to strchr('\n'). These are actually OK by the
"the final header line must end in a newline" guarantee from
verify_headers(). They will always find that rather than run off the
end of the buffer. Curiously, they do check for a NULL return and
complain, but I believe that condition can never be reached.
However, I converted them to use memchr() with a proper size and
retained the NULL checks. Using memchr() is not much longer and
makes it more obvious what is going on. Likewise, retaining the NULL
checks serves as a defensive measure in case my analysis is wrong.
- commit 9a1a3a4d4c (mktag: allow omitting the header/body \n
separator, 2021-01-05), does check for the end-of-buffer condition,
but does so with "!*buffer", relying explicitly on the NUL
termination. We can accomplish the same thing with a pointer
comparison. I also folded it into the follow-on conditional that
checks the contents of the buffer, for consistency with the other
checks.
- fsck_ident() uses parse_timestamp(), which is based on strtoumax().
That function will happily skip past leading whitespace, including
newlines, which makes it a risk. We can fix this by scanning to the
first digit ourselves, and then using parse_timestamp() to do the
actual numeric conversion.
Note that as a side effect this fixes the fact that we missed
zero-padded timestamps like "<email> 0123" (whereas we would
complain about "<email> 0123"). I doubt anybody cares, but I
mention it here for completeness.
- fsck_tree() does not need any modifications. It relies on
decode_tree_entry() to do the actual parsing, and that function
checks both that there are enough bytes in the buffer to represent
an entry, and that there is a NUL at the appropriate spot (one
hash-length from the end; this may not be the NUL for the entry we
are parsing, but we know that in the worst case, everything from our
current position to that NUL is a filename, so we won't run out of
bytes).
In addition to fixing the code itself, we'd like to make sure our rather
subtle assumptions are not violated in the future. So this patch does
two more things:
- add comments around verify_headers() documenting the link between
what it checks and the memory safety of the callers. I don't expect
this code to be modified frequently, but this may help somebody from
accidentally breaking things.
- add a thorough set of tests covering truncations at various key
spots (e.g., for a "tree $oid" line, in the middle of the word
"tree", right after it, after the space, in the middle of the $oid,
and right at the end of the line. Most of these are fine already (it
is only truncating right at the end of the line that is currently
broken). And some of them are not even possible with the current
code (we parse "tree " as a unit, so truncating before the space is
equivalent). But I aimed here to consider the code a black box and
look for any truncations that would be a problem for a left-to-right
parser.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-01-19 23:13:29 +00:00
|
|
|
if (buffer < buffer_end && !starts_with(buffer, "\n")) {
|
mktag: use fsck instead of custom verify_tag()
Change the validation logic in "mktag" to use fsck's fsck_tag()
instead of its own custom parser. Curiously the logic for both dates
back to the same commit[1]. Let's unify them so we're not maintaining
two sets functions to verify that a tag is OK.
The behavior of fsck_tag() and the old "mktag" code being removed here
is different in few aspects.
I think it makes sense to remove some of those checks, namely:
A. fsck only cares that the timezone matches [-+][0-9]{4}. The mktag
code disallowed values larger than 1400.
Yes there's currently no timezone with a greater offset[2], but
since we allow any number of non-offical timezones (e.g. +1234)
passing this through seems fine. Git also won't break in the
future if e.g. French Polynesia decides it needs to outdo the Line
Islands when it comes to timezone extravagance.
B. fsck allows missing author names such as "tagger <email>", mktag
wouldn't, but would allow e.g. "tagger [2 spaces] <email>" (but
not "tagger [1 space] <email>"). Now we allow all of these.
C. Like B, but "mktag" disallowed spaces in the <email> part, fsck
allows it.
In some ways fsck_tag() is stricter than "mktag" was, namely:
D. fsck disallows zero-padded dates, but mktag didn't care. So
e.g. the timestamp "0000000000 +0000" produces an error now. A
test in "t1006-cat-file.sh" relied on this, it's been changed to
use "hash-object" (without fsck) instead.
There was one check I deemed worth keeping by porting it over to
fsck_tag():
E. "mktag" did not allow any custom headers, and by extension (as an
empty commit is allowed) also forbade an extra stray trailing
newline after the headers it knew about.
Add a new check in the "ignore" category to fsck and use it. This
somewhat abuses the facility added in efaba7cc77f (fsck:
optionally ignore specific fsck issues completely, 2015-06-22).
This is somewhat of hack, but probably the least invasive change
we can make here. The fsck command will shuffle these categories
around, e.g. under --strict the "info" becomes a "warn" and "warn"
becomes "error". Existing users of fsck's (and others,
e.g. index-pack) --strict option rely on this.
So we need to put something into a category that'll be ignored by
all existing users of the API. Pretending that
fsck.extraHeaderEntry=error ("ignore" by default) was set serves
to do this for us.
1. ec4465adb38 (Add "tag" objects that can be used to sign other
objects., 2005-04-25)
2. https://en.wikipedia.org/wiki/List_of_UTC_time_offsets
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2021-01-05 19:42:46 +00:00
|
|
|
/*
|
|
|
|
* The verify_headers() check will allow
|
|
|
|
* e.g. "[...]tagger <tagger>\nsome
|
|
|
|
* garbage\n\nmessage" to pass, thinking "some
|
|
|
|
* garbage" could be a custom header. E.g. "mktag"
|
|
|
|
* doesn't want any unknown headers.
|
|
|
|
*/
|
|
|
|
ret = report(options, oid, OBJ_TAG, FSCK_MSG_EXTRA_HEADER_ENTRY, "invalid format - extra header(s) after 'tagger'");
|
|
|
|
if (ret)
|
|
|
|
goto done;
|
|
|
|
}
|
|
|
|
|
2014-09-11 14:26:38 +00:00
|
|
|
done:
|
|
|
|
strbuf_release(&sb);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2022-05-16 20:10:59 +00:00
|
|
|
static int starts_with_dot_slash(const char *const path)
|
2020-04-19 03:52:34 +00:00
|
|
|
{
|
2022-05-16 20:10:59 +00:00
|
|
|
return path_match_flags(path, PATH_MATCH_STARTS_WITH_DOT_SLASH |
|
|
|
|
PATH_MATCH_XPLATFORM);
|
2020-04-19 03:52:34 +00:00
|
|
|
}
|
|
|
|
|
2022-05-16 20:10:59 +00:00
|
|
|
static int starts_with_dot_dot_slash(const char *const path)
|
2020-04-19 03:52:34 +00:00
|
|
|
{
|
2022-05-16 20:10:59 +00:00
|
|
|
return path_match_flags(path, PATH_MATCH_STARTS_WITH_DOT_DOT_SLASH |
|
|
|
|
PATH_MATCH_XPLATFORM);
|
2020-04-19 03:52:34 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
static int submodule_url_is_relative(const char *url)
|
|
|
|
{
|
|
|
|
return starts_with_dot_slash(url) || starts_with_dot_dot_slash(url);
|
|
|
|
}
|
|
|
|
|
credential: treat URL without scheme as invalid
libcurl permits making requests without a URL scheme specified. In
this case, it guesses the URL from the hostname, so I can run
git ls-remote http::ftp.example.com/path/to/repo
and it would make an FTP request.
Any user intentionally using such a URL is likely to have made a typo.
Unfortunately, credential_from_url is not able to determine the host and
protocol in order to determine appropriate credentials to send, and
until "credential: refuse to operate when missing host or protocol",
this resulted in another host's credentials being leaked to the named
host.
Teach credential_from_url_gently to consider such a URL to be invalid
so that fsck can detect and block gitmodules files with such URLs,
allowing server operators to avoid serving them to downstream users
running older versions of Git.
This also means that when such URLs are passed on the command line, Git
will print a clearer error so affected users can switch to the simpler
URL that explicitly specifies the host and protocol they intend.
One subtlety: .gitmodules files can contain relative URLs, representing
a URL relative to the URL they were cloned from. The relative URL
resolver used for .gitmodules can follow ".." components out of the path
part and past the host part of a URL, meaning that such a relative URL
can be used to traverse from a https://foo.example.com/innocent
superproject to a https::attacker.example.com/exploit submodule.
Fortunately a leading ':' in the first path component after a series of
leading './' and '../' components is unlikely to show up in other
contexts, so we can catch this by detecting that pattern.
Reported-by: Jeff King <peff@peff.net>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Reviewed-by: Jeff King <peff@peff.net>
2020-04-19 03:54:13 +00:00
|
|
|
/*
|
|
|
|
* Count directory components that a relative submodule URL should chop
|
|
|
|
* from the remote_url it is to be resolved against.
|
|
|
|
*
|
|
|
|
* In other words, this counts "../" components at the start of a
|
|
|
|
* submodule URL.
|
|
|
|
*
|
|
|
|
* Returns the number of directory components to chop and writes a
|
|
|
|
* pointer to the next character of url after all leading "./" and
|
|
|
|
* "../" components to out.
|
|
|
|
*/
|
|
|
|
static int count_leading_dotdots(const char *url, const char **out)
|
|
|
|
{
|
|
|
|
int result = 0;
|
|
|
|
while (1) {
|
|
|
|
if (starts_with_dot_dot_slash(url)) {
|
|
|
|
result++;
|
|
|
|
url += strlen("../");
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
if (starts_with_dot_slash(url)) {
|
|
|
|
url += strlen("./");
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
*out = url;
|
|
|
|
return result;
|
|
|
|
}
|
|
|
|
}
|
2020-04-19 03:52:34 +00:00
|
|
|
/*
|
|
|
|
* Check whether a transport is implemented by git-remote-curl.
|
|
|
|
*
|
|
|
|
* If it is, returns 1 and writes the URL that would be passed to
|
|
|
|
* git-remote-curl to the "out" parameter.
|
|
|
|
*
|
|
|
|
* Otherwise, returns 0 and leaves "out" untouched.
|
|
|
|
*
|
|
|
|
* Examples:
|
|
|
|
* http::https://example.com/repo.git -> 1, https://example.com/repo.git
|
|
|
|
* https://example.com/repo.git -> 1, https://example.com/repo.git
|
|
|
|
* git://example.com/repo.git -> 0
|
|
|
|
*
|
|
|
|
* This is for use in checking for previously exploitable bugs that
|
|
|
|
* required a submodule URL to be passed to git-remote-curl.
|
|
|
|
*/
|
|
|
|
static int url_to_curl_url(const char *url, const char **out)
|
|
|
|
{
|
|
|
|
/*
|
|
|
|
* We don't need to check for case-aliases, "http.exe", and so
|
|
|
|
* on because in the default configuration, is_transport_allowed
|
|
|
|
* prevents URLs with those schemes from being cloned
|
|
|
|
* automatically.
|
|
|
|
*/
|
|
|
|
if (skip_prefix(url, "http::", out) ||
|
|
|
|
skip_prefix(url, "https::", out) ||
|
|
|
|
skip_prefix(url, "ftp::", out) ||
|
|
|
|
skip_prefix(url, "ftps::", out))
|
|
|
|
return 1;
|
|
|
|
if (starts_with(url, "http://") ||
|
|
|
|
starts_with(url, "https://") ||
|
|
|
|
starts_with(url, "ftp://") ||
|
|
|
|
starts_with(url, "ftps://")) {
|
|
|
|
*out = url;
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2020-03-11 22:48:24 +00:00
|
|
|
static int check_submodule_url(const char *url)
|
|
|
|
{
|
2020-04-19 03:52:34 +00:00
|
|
|
const char *curl_url;
|
2020-03-11 22:48:24 +00:00
|
|
|
|
|
|
|
if (looks_like_command_line_option(url))
|
|
|
|
return -1;
|
|
|
|
|
2021-01-07 09:44:17 +00:00
|
|
|
if (submodule_url_is_relative(url) || starts_with(url, "git://")) {
|
credential: treat URL without scheme as invalid
libcurl permits making requests without a URL scheme specified. In
this case, it guesses the URL from the hostname, so I can run
git ls-remote http::ftp.example.com/path/to/repo
and it would make an FTP request.
Any user intentionally using such a URL is likely to have made a typo.
Unfortunately, credential_from_url is not able to determine the host and
protocol in order to determine appropriate credentials to send, and
until "credential: refuse to operate when missing host or protocol",
this resulted in another host's credentials being leaked to the named
host.
Teach credential_from_url_gently to consider such a URL to be invalid
so that fsck can detect and block gitmodules files with such URLs,
allowing server operators to avoid serving them to downstream users
running older versions of Git.
This also means that when such URLs are passed on the command line, Git
will print a clearer error so affected users can switch to the simpler
URL that explicitly specifies the host and protocol they intend.
One subtlety: .gitmodules files can contain relative URLs, representing
a URL relative to the URL they were cloned from. The relative URL
resolver used for .gitmodules can follow ".." components out of the path
part and past the host part of a URL, meaning that such a relative URL
can be used to traverse from a https://foo.example.com/innocent
superproject to a https::attacker.example.com/exploit submodule.
Fortunately a leading ':' in the first path component after a series of
leading './' and '../' components is unlikely to show up in other
contexts, so we can catch this by detecting that pattern.
Reported-by: Jeff King <peff@peff.net>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Reviewed-by: Jeff King <peff@peff.net>
2020-04-19 03:54:13 +00:00
|
|
|
char *decoded;
|
|
|
|
const char *next;
|
|
|
|
int has_nl;
|
|
|
|
|
2020-04-19 03:52:34 +00:00
|
|
|
/*
|
|
|
|
* This could be appended to an http URL and url-decoded;
|
|
|
|
* check for malicious characters.
|
|
|
|
*/
|
credential: treat URL without scheme as invalid
libcurl permits making requests without a URL scheme specified. In
this case, it guesses the URL from the hostname, so I can run
git ls-remote http::ftp.example.com/path/to/repo
and it would make an FTP request.
Any user intentionally using such a URL is likely to have made a typo.
Unfortunately, credential_from_url is not able to determine the host and
protocol in order to determine appropriate credentials to send, and
until "credential: refuse to operate when missing host or protocol",
this resulted in another host's credentials being leaked to the named
host.
Teach credential_from_url_gently to consider such a URL to be invalid
so that fsck can detect and block gitmodules files with such URLs,
allowing server operators to avoid serving them to downstream users
running older versions of Git.
This also means that when such URLs are passed on the command line, Git
will print a clearer error so affected users can switch to the simpler
URL that explicitly specifies the host and protocol they intend.
One subtlety: .gitmodules files can contain relative URLs, representing
a URL relative to the URL they were cloned from. The relative URL
resolver used for .gitmodules can follow ".." components out of the path
part and past the host part of a URL, meaning that such a relative URL
can be used to traverse from a https://foo.example.com/innocent
superproject to a https::attacker.example.com/exploit submodule.
Fortunately a leading ':' in the first path component after a series of
leading './' and '../' components is unlikely to show up in other
contexts, so we can catch this by detecting that pattern.
Reported-by: Jeff King <peff@peff.net>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Reviewed-by: Jeff King <peff@peff.net>
2020-04-19 03:54:13 +00:00
|
|
|
decoded = url_decode(url);
|
|
|
|
has_nl = !!strchr(decoded, '\n');
|
|
|
|
|
2020-04-19 03:52:34 +00:00
|
|
|
free(decoded);
|
|
|
|
if (has_nl)
|
|
|
|
return -1;
|
credential: treat URL without scheme as invalid
libcurl permits making requests without a URL scheme specified. In
this case, it guesses the URL from the hostname, so I can run
git ls-remote http::ftp.example.com/path/to/repo
and it would make an FTP request.
Any user intentionally using such a URL is likely to have made a typo.
Unfortunately, credential_from_url is not able to determine the host and
protocol in order to determine appropriate credentials to send, and
until "credential: refuse to operate when missing host or protocol",
this resulted in another host's credentials being leaked to the named
host.
Teach credential_from_url_gently to consider such a URL to be invalid
so that fsck can detect and block gitmodules files with such URLs,
allowing server operators to avoid serving them to downstream users
running older versions of Git.
This also means that when such URLs are passed on the command line, Git
will print a clearer error so affected users can switch to the simpler
URL that explicitly specifies the host and protocol they intend.
One subtlety: .gitmodules files can contain relative URLs, representing
a URL relative to the URL they were cloned from. The relative URL
resolver used for .gitmodules can follow ".." components out of the path
part and past the host part of a URL, meaning that such a relative URL
can be used to traverse from a https://foo.example.com/innocent
superproject to a https::attacker.example.com/exploit submodule.
Fortunately a leading ':' in the first path component after a series of
leading './' and '../' components is unlikely to show up in other
contexts, so we can catch this by detecting that pattern.
Reported-by: Jeff King <peff@peff.net>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Reviewed-by: Jeff King <peff@peff.net>
2020-04-19 03:54:13 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* URLs which escape their root via "../" can overwrite
|
|
|
|
* the host field and previous components, resolving to
|
2020-04-19 03:57:22 +00:00
|
|
|
* URLs like https::example.com/submodule.git and
|
|
|
|
* https:///example.com/submodule.git that were
|
credential: treat URL without scheme as invalid
libcurl permits making requests without a URL scheme specified. In
this case, it guesses the URL from the hostname, so I can run
git ls-remote http::ftp.example.com/path/to/repo
and it would make an FTP request.
Any user intentionally using such a URL is likely to have made a typo.
Unfortunately, credential_from_url is not able to determine the host and
protocol in order to determine appropriate credentials to send, and
until "credential: refuse to operate when missing host or protocol",
this resulted in another host's credentials being leaked to the named
host.
Teach credential_from_url_gently to consider such a URL to be invalid
so that fsck can detect and block gitmodules files with such URLs,
allowing server operators to avoid serving them to downstream users
running older versions of Git.
This also means that when such URLs are passed on the command line, Git
will print a clearer error so affected users can switch to the simpler
URL that explicitly specifies the host and protocol they intend.
One subtlety: .gitmodules files can contain relative URLs, representing
a URL relative to the URL they were cloned from. The relative URL
resolver used for .gitmodules can follow ".." components out of the path
part and past the host part of a URL, meaning that such a relative URL
can be used to traverse from a https://foo.example.com/innocent
superproject to a https::attacker.example.com/exploit submodule.
Fortunately a leading ':' in the first path component after a series of
leading './' and '../' components is unlikely to show up in other
contexts, so we can catch this by detecting that pattern.
Reported-by: Jeff King <peff@peff.net>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Reviewed-by: Jeff King <peff@peff.net>
2020-04-19 03:54:13 +00:00
|
|
|
* susceptible to CVE-2020-11008.
|
|
|
|
*/
|
|
|
|
if (count_leading_dotdots(url, &next) > 0 &&
|
2020-04-19 03:57:22 +00:00
|
|
|
(*next == ':' || *next == '/'))
|
credential: treat URL without scheme as invalid
libcurl permits making requests without a URL scheme specified. In
this case, it guesses the URL from the hostname, so I can run
git ls-remote http::ftp.example.com/path/to/repo
and it would make an FTP request.
Any user intentionally using such a URL is likely to have made a typo.
Unfortunately, credential_from_url is not able to determine the host and
protocol in order to determine appropriate credentials to send, and
until "credential: refuse to operate when missing host or protocol",
this resulted in another host's credentials being leaked to the named
host.
Teach credential_from_url_gently to consider such a URL to be invalid
so that fsck can detect and block gitmodules files with such URLs,
allowing server operators to avoid serving them to downstream users
running older versions of Git.
This also means that when such URLs are passed on the command line, Git
will print a clearer error so affected users can switch to the simpler
URL that explicitly specifies the host and protocol they intend.
One subtlety: .gitmodules files can contain relative URLs, representing
a URL relative to the URL they were cloned from. The relative URL
resolver used for .gitmodules can follow ".." components out of the path
part and past the host part of a URL, meaning that such a relative URL
can be used to traverse from a https://foo.example.com/innocent
superproject to a https::attacker.example.com/exploit submodule.
Fortunately a leading ':' in the first path component after a series of
leading './' and '../' components is unlikely to show up in other
contexts, so we can catch this by detecting that pattern.
Reported-by: Jeff King <peff@peff.net>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Reviewed-by: Jeff King <peff@peff.net>
2020-04-19 03:54:13 +00:00
|
|
|
return -1;
|
2020-04-19 03:52:34 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
else if (url_to_curl_url(url, &curl_url)) {
|
|
|
|
struct credential c = CREDENTIAL_INIT;
|
2020-04-19 03:57:22 +00:00
|
|
|
int ret = 0;
|
|
|
|
if (credential_from_url_gently(&c, curl_url, 1) ||
|
|
|
|
!*c.host)
|
|
|
|
ret = -1;
|
2020-04-19 03:52:34 +00:00
|
|
|
credential_clear(&c);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
2020-03-11 22:48:24 +00:00
|
|
|
}
|
|
|
|
|
2018-05-02 21:25:27 +00:00
|
|
|
struct fsck_gitmodules_data {
|
2019-10-18 04:59:29 +00:00
|
|
|
const struct object_id *oid;
|
2018-05-02 21:25:27 +00:00
|
|
|
struct fsck_options *options;
|
|
|
|
int ret;
|
|
|
|
};
|
|
|
|
|
config: add ctx arg to config_fn_t
Add a new "const struct config_context *ctx" arg to config_fn_t to hold
additional information about the config iteration operation.
config_context has a "struct key_value_info kvi" member that holds
metadata about the config source being read (e.g. what kind of config
source it is, the filename, etc). In this series, we're only interested
in .kvi, so we could have just used "struct key_value_info" as an arg,
but config_context makes it possible to add/adjust members in the future
without changing the config_fn_t signature. We could also consider other
ways of organizing the args (e.g. moving the config name and value into
config_context or key_value_info), but in my experiments, the
incremental benefit doesn't justify the added complexity (e.g. a
config_fn_t will sometimes invoke another config_fn_t but with a
different config value).
In subsequent commits, the .kvi member will replace the global "struct
config_reader" in config.c, making config iteration a global-free
operation. It requires much more work for the machinery to provide
meaningful values of .kvi, so for now, merely change the signature and
call sites, pass NULL as a placeholder value, and don't rely on the arg
in any meaningful way.
Most of the changes are performed by
contrib/coccinelle/config_fn_ctx.pending.cocci, which, for every
config_fn_t:
- Modifies the signature to accept "const struct config_context *ctx"
- Passes "ctx" to any inner config_fn_t, if needed
- Adds UNUSED attributes to "ctx", if needed
Most config_fn_t instances are easily identified by seeing if they are
called by the various config functions. Most of the remaining ones are
manually named in the .cocci patch. Manual cleanups are still needed,
but the majority of it is trivial; it's either adjusting config_fn_t
that the .cocci patch didn't catch, or adding forward declarations of
"struct config_context ctx" to make the signatures make sense.
The non-trivial changes are in cases where we are invoking a config_fn_t
outside of config machinery, and we now need to decide what value of
"ctx" to pass. These cases are:
- trace2/tr2_cfg.c:tr2_cfg_set_fl()
This is indirectly called by git_config_set() so that the trace2
machinery can notice the new config values and update its settings
using the tr2 config parsing function, i.e. tr2_cfg_cb().
- builtin/checkout.c:checkout_main()
This calls git_xmerge_config() as a shorthand for parsing a CLI arg.
This might be worth refactoring away in the future, since
git_xmerge_config() can call git_default_config(), which can do much
more than just parsing.
Handle them by creating a KVI_INIT macro that initializes "struct
key_value_info" to a reasonable default, and use that to construct the
"ctx" arg.
Signed-off-by: Glen Choo <chooglen@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-06-28 19:26:22 +00:00
|
|
|
static int fsck_gitmodules_fn(const char *var, const char *value,
|
|
|
|
const struct config_context *ctx UNUSED,
|
|
|
|
void *vdata)
|
2018-05-02 21:25:27 +00:00
|
|
|
{
|
|
|
|
struct fsck_gitmodules_data *data = vdata;
|
|
|
|
const char *subsection, *key;
|
2020-04-10 19:44:28 +00:00
|
|
|
size_t subsection_len;
|
2018-05-02 21:25:27 +00:00
|
|
|
char *name;
|
|
|
|
|
|
|
|
if (parse_config_key(var, "submodule", &subsection, &subsection_len, &key) < 0 ||
|
|
|
|
!subsection)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
name = xmemdupz(subsection, subsection_len);
|
|
|
|
if (check_submodule_name(name) < 0)
|
2019-10-18 04:59:15 +00:00
|
|
|
data->ret |= report(data->options,
|
2019-10-18 04:59:29 +00:00
|
|
|
data->oid, OBJ_BLOB,
|
2018-05-02 21:25:27 +00:00
|
|
|
FSCK_MSG_GITMODULES_NAME,
|
|
|
|
"disallowed submodule name: %s",
|
|
|
|
name);
|
2018-09-24 08:37:17 +00:00
|
|
|
if (!strcmp(key, "url") && value &&
|
2020-03-11 22:48:24 +00:00
|
|
|
check_submodule_url(value) < 0)
|
2019-10-18 04:59:15 +00:00
|
|
|
data->ret |= report(data->options,
|
2019-10-18 04:59:29 +00:00
|
|
|
data->oid, OBJ_BLOB,
|
2018-09-24 08:37:17 +00:00
|
|
|
FSCK_MSG_GITMODULES_URL,
|
|
|
|
"disallowed submodule url: %s",
|
|
|
|
value);
|
fsck: detect submodule paths starting with dash
As with urls, submodule paths with dashes are ignored by
git, but may end up confusing older versions. Detecting them
via fsck lets us prevent modern versions of git from being a
vector to spread broken .gitmodules to older versions.
Compared to blocking leading-dash urls, though, this
detection may be less of a good idea:
1. While such paths provide confusing and broken results,
they don't seem to actually work as option injections
against anything except "cd". In particular, the
submodule code seems to canonicalize to an absolute
path before running "git clone" (so it passes
/your/clone/-sub).
2. It's more likely that we may one day make such names
actually work correctly. Even after we revert this fsck
check, it will continue to be a hassle until hosting
servers are all updated.
On the other hand, it's not entirely clear that the behavior
in older versions is safe. And if we do want to eventually
allow this, we may end up doing so with a special syntax
anyway (e.g., writing "./-sub" in the .gitmodules file, and
teaching the submodule code to canonicalize it when
comparing).
So on balance, this is probably a good protection.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-09-24 08:42:19 +00:00
|
|
|
if (!strcmp(key, "path") && value &&
|
|
|
|
looks_like_command_line_option(value))
|
2019-10-18 04:59:15 +00:00
|
|
|
data->ret |= report(data->options,
|
2019-10-18 04:59:29 +00:00
|
|
|
data->oid, OBJ_BLOB,
|
fsck: detect submodule paths starting with dash
As with urls, submodule paths with dashes are ignored by
git, but may end up confusing older versions. Detecting them
via fsck lets us prevent modern versions of git from being a
vector to spread broken .gitmodules to older versions.
Compared to blocking leading-dash urls, though, this
detection may be less of a good idea:
1. While such paths provide confusing and broken results,
they don't seem to actually work as option injections
against anything except "cd". In particular, the
submodule code seems to canonicalize to an absolute
path before running "git clone" (so it passes
/your/clone/-sub).
2. It's more likely that we may one day make such names
actually work correctly. Even after we revert this fsck
check, it will continue to be a hassle until hosting
servers are all updated.
On the other hand, it's not entirely clear that the behavior
in older versions is safe. And if we do want to eventually
allow this, we may end up doing so with a special syntax
anyway (e.g., writing "./-sub" in the .gitmodules file, and
teaching the submodule code to canonicalize it when
comparing).
So on balance, this is probably a good protection.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-09-24 08:42:19 +00:00
|
|
|
FSCK_MSG_GITMODULES_PATH,
|
|
|
|
"disallowed submodule path: %s",
|
|
|
|
value);
|
2019-12-05 09:30:43 +00:00
|
|
|
if (!strcmp(key, "update") && value &&
|
|
|
|
parse_submodule_update_type(value) == SM_UPDATE_COMMAND)
|
2019-12-10 06:17:55 +00:00
|
|
|
data->ret |= report(data->options, data->oid, OBJ_BLOB,
|
2019-12-05 09:30:43 +00:00
|
|
|
FSCK_MSG_GITMODULES_UPDATE,
|
|
|
|
"disallowed submodule update setting: %s",
|
|
|
|
value);
|
2018-05-02 21:25:27 +00:00
|
|
|
free(name);
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2019-10-18 04:59:29 +00:00
|
|
|
static int fsck_blob(const struct object_id *oid, const char *buf,
|
fsck: actually fsck blob data
Because fscking a blob has always been a noop, we didn't
bother passing around the blob data. In preparation for
content-level checks, let's fix up a few things:
1. The fsck_object() function just returns success for any
blob. Let's a noop fsck_blob(), which we can fill in
with actual logic later.
2. The fsck_loose() function in builtin/fsck.c
just threw away blob content after loading it. Let's
hold onto it until after we've called fsck_object().
The easiest way to do this is to just drop the
parse_loose_object() helper entirely. Incidentally,
this also fixes a memory leak: if we successfully
loaded the object data but did not parse it, we would
have left the function without freeing it.
3. When fsck_loose() loads the object data, it
does so with a custom read_loose_object() helper. This
function streams any blobs, regardless of size, under
the assumption that we're only checking the sha1.
Instead, let's actually load blobs smaller than
big_file_threshold, as the normal object-reading
code-paths would do. This lets us fsck small files, and
a NULL return is an indication that the blob was so big
that it needed to be streamed, and we can pass that
information along to fsck_blob().
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 19:44:51 +00:00
|
|
|
unsigned long size, struct fsck_options *options)
|
|
|
|
{
|
2022-12-01 14:45:57 +00:00
|
|
|
int ret = 0;
|
2018-05-02 21:25:27 +00:00
|
|
|
|
2019-10-18 04:59:29 +00:00
|
|
|
if (object_on_skiplist(options, oid))
|
fsck: check skiplist for object in fsck_blob()
Since commit ed8b10f631 ("fsck: check .gitmodules content", 2018-05-02),
fsck will issue an error message for '.gitmodules' content that cannot
be parsed correctly. This is the case, even when the corresponding blob
object has been included on the skiplist. For example, using the cgit
repository, we see the following:
$ git fsck
Checking object directories: 100% (256/256), done.
error: bad config line 5 in blob .gitmodules
error in blob 51dd1eff1edc663674df9ab85d2786a40f7ae3a5: gitmodulesParse: could not parse gitmodules blob
Checking objects: 100% (6626/6626), done.
$
$ git config fsck.skiplist '.git/skip'
$ echo 51dd1eff1edc663674df9ab85d2786a40f7ae3a5 >.git/skip
$
$ git fsck
Checking object directories: 100% (256/256), done.
error: bad config line 5 in blob .gitmodules
Checking objects: 100% (6626/6626), done.
$
Note that the error message issued by the config parser is still
present, despite adding the object-id of the blob to the skiplist.
One solution would be to provide a means of suppressing the messages
issued by the config parser. However, given that (logically) we are
asking fsck to ignore this object, a simpler approach is to just not
call the config parser if the object is to be skipped. Add a check to
the 'fsck_blob()' processing function, to determine if the object is
on the skiplist and, if so, exit the function early.
Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-06-27 18:39:53 +00:00
|
|
|
return 0;
|
|
|
|
|
2022-12-01 14:45:57 +00:00
|
|
|
if (oidset_contains(&options->gitmodules_found, oid)) {
|
|
|
|
struct config_options config_opts = { 0 };
|
|
|
|
struct fsck_gitmodules_data data;
|
|
|
|
|
|
|
|
oidset_insert(&options->gitmodules_done, oid);
|
|
|
|
|
|
|
|
if (!buf) {
|
|
|
|
/*
|
|
|
|
* A missing buffer here is a sign that the caller found the
|
|
|
|
* blob too gigantic to load into memory. Let's just consider
|
|
|
|
* that an error.
|
|
|
|
*/
|
|
|
|
return report(options, oid, OBJ_BLOB,
|
|
|
|
FSCK_MSG_GITMODULES_LARGE,
|
|
|
|
".gitmodules too large to parse");
|
|
|
|
}
|
|
|
|
|
|
|
|
data.oid = oid;
|
|
|
|
data.options = options;
|
|
|
|
data.ret = 0;
|
|
|
|
config_opts.error_action = CONFIG_ERROR_SILENT;
|
|
|
|
if (git_config_from_mem(fsck_gitmodules_fn, CONFIG_ORIGIN_BLOB,
|
2023-06-28 19:26:24 +00:00
|
|
|
".gitmodules", buf, size, &data,
|
|
|
|
CONFIG_SCOPE_UNKNOWN, &config_opts))
|
2022-12-01 14:45:57 +00:00
|
|
|
data.ret |= report(options, oid, OBJ_BLOB,
|
|
|
|
FSCK_MSG_GITMODULES_PARSE,
|
|
|
|
"could not parse gitmodules blob");
|
|
|
|
ret |= data.ret;
|
2018-05-02 21:25:27 +00:00
|
|
|
}
|
|
|
|
|
2022-12-01 14:46:09 +00:00
|
|
|
if (oidset_contains(&options->gitattributes_found, oid)) {
|
|
|
|
const char *ptr;
|
|
|
|
|
|
|
|
oidset_insert(&options->gitattributes_done, oid);
|
|
|
|
|
|
|
|
if (!buf || size > ATTR_MAX_FILE_SIZE) {
|
|
|
|
/*
|
|
|
|
* A missing buffer here is a sign that the caller found the
|
|
|
|
* blob too gigantic to load into memory. Let's just consider
|
|
|
|
* that an error.
|
|
|
|
*/
|
|
|
|
return report(options, oid, OBJ_BLOB,
|
|
|
|
FSCK_MSG_GITATTRIBUTES_LARGE,
|
|
|
|
".gitattributes too large to parse");
|
|
|
|
}
|
|
|
|
|
|
|
|
for (ptr = buf; *ptr; ) {
|
|
|
|
const char *eol = strchrnul(ptr, '\n');
|
|
|
|
if (eol - ptr >= ATTR_MAX_LINE_LENGTH) {
|
|
|
|
ret |= report(options, oid, OBJ_BLOB,
|
|
|
|
FSCK_MSG_GITATTRIBUTES_LINE_LENGTH,
|
|
|
|
".gitattributes has too long lines to parse");
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
ptr = *eol ? eol + 1 : eol;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2022-12-01 14:45:57 +00:00
|
|
|
return ret;
|
fsck: actually fsck blob data
Because fscking a blob has always been a noop, we didn't
bother passing around the blob data. In preparation for
content-level checks, let's fix up a few things:
1. The fsck_object() function just returns success for any
blob. Let's a noop fsck_blob(), which we can fill in
with actual logic later.
2. The fsck_loose() function in builtin/fsck.c
just threw away blob content after loading it. Let's
hold onto it until after we've called fsck_object().
The easiest way to do this is to just drop the
parse_loose_object() helper entirely. Incidentally,
this also fixes a memory leak: if we successfully
loaded the object data but did not parse it, we would
have left the function without freeing it.
3. When fsck_loose() loads the object data, it
does so with a custom read_loose_object() helper. This
function streams any blobs, regardless of size, under
the assumption that we're only checking the sha1.
Instead, let's actually load blobs smaller than
big_file_threshold, as the normal object-reading
code-paths would do. This lets us fsck small files, and
a NULL return is an indication that the blob was so big
that it needed to be streamed, and we can pass that
information along to fsck_blob().
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 19:44:51 +00:00
|
|
|
}
|
|
|
|
|
2014-09-10 13:52:51 +00:00
|
|
|
int fsck_object(struct object *obj, void *data, unsigned long size,
|
2015-06-22 15:25:00 +00:00
|
|
|
struct fsck_options *options)
|
2008-02-25 21:46:08 +00:00
|
|
|
{
|
|
|
|
if (!obj)
|
2019-10-18 04:59:15 +00:00
|
|
|
return report(options, NULL, OBJ_NONE, FSCK_MSG_BAD_OBJECT_SHA1, "no valid object to fsck");
|
2008-02-25 21:46:08 +00:00
|
|
|
|
2023-01-18 20:43:53 +00:00
|
|
|
return fsck_buffer(&obj->oid, obj->type, data, size, options);
|
|
|
|
}
|
|
|
|
|
|
|
|
int fsck_buffer(const struct object_id *oid, enum object_type type,
|
|
|
|
void *data, unsigned long size,
|
|
|
|
struct fsck_options *options)
|
|
|
|
{
|
|
|
|
if (type == OBJ_BLOB)
|
|
|
|
return fsck_blob(oid, data, size, options);
|
|
|
|
if (type == OBJ_TREE)
|
|
|
|
return fsck_tree(oid, data, size, options);
|
|
|
|
if (type == OBJ_COMMIT)
|
|
|
|
return fsck_commit(oid, data, size, options);
|
|
|
|
if (type == OBJ_TAG)
|
|
|
|
return fsck_tag(oid, data, size, options);
|
|
|
|
|
|
|
|
return report(options, oid, type,
|
2019-10-18 04:59:15 +00:00
|
|
|
FSCK_MSG_UNKNOWN_TYPE,
|
|
|
|
"unknown type '%d' (internal fsck error)",
|
2023-01-18 20:43:53 +00:00
|
|
|
type);
|
2008-02-25 21:46:08 +00:00
|
|
|
}
|
2008-02-25 21:46:09 +00:00
|
|
|
|
2016-07-17 10:59:57 +00:00
|
|
|
int fsck_error_function(struct fsck_options *o,
|
2019-10-18 04:58:40 +00:00
|
|
|
const struct object_id *oid,
|
2023-07-03 06:44:18 +00:00
|
|
|
enum object_type object_type UNUSED,
|
2021-03-28 13:15:46 +00:00
|
|
|
enum fsck_msg_type msg_type,
|
2023-07-03 06:44:18 +00:00
|
|
|
enum fsck_msg_id msg_id UNUSED,
|
2021-03-28 13:15:46 +00:00
|
|
|
const char *message)
|
2008-02-25 21:46:09 +00:00
|
|
|
{
|
2015-06-22 15:25:25 +00:00
|
|
|
if (msg_type == FSCK_WARN) {
|
2019-10-18 04:58:40 +00:00
|
|
|
warning("object %s: %s", fsck_describe_object(o, oid), message);
|
2015-06-22 15:25:25 +00:00
|
|
|
return 0;
|
|
|
|
}
|
2019-10-18 04:58:40 +00:00
|
|
|
error("object %s: %s", fsck_describe_object(o, oid), message);
|
2008-02-25 21:46:09 +00:00
|
|
|
return 1;
|
|
|
|
}
|
fsck: detect gitmodules files
In preparation for performing fsck checks on .gitmodules
files, this commit plumbs in the actual detection of the
files. Note that unlike most other fsck checks, this cannot
be a property of a single object: we must know that the
object is found at a ".gitmodules" path at the root tree of
a commit.
Since the fsck code only sees one object at a time, we have
to mark the related objects to fit the puzzle together. When
we see a commit we mark its tree as a root tree, and when
we see a root tree with a .gitmodules file, we mark the
corresponding blob to be checked.
In an ideal world, we'd check the objects in topological
order: commits followed by trees followed by blobs. In that
case we can avoid ever loading an object twice, since all
markings would be complete by the time we get to the marked
objects. And indeed, if we are checking a single packfile,
this is the order in which Git will generally write the
objects. But we can't count on that:
1. git-fsck may show us the objects in arbitrary order
(loose objects are fed in sha1 order, but we may also
have multiple packs, and we process each pack fully in
sequence).
2. The type ordering is just what git-pack-objects happens
to write now. The pack format does not require a
specific order, and it's possible that future versions
of Git (or a custom version trying to fool official
Git's fsck checks!) may order it differently.
3. We may not even be fscking all of the relevant objects
at once. Consider pushing with transfer.fsckObjects,
where one push adds a blob at path "foo", and then a
second push adds the same blob at path ".gitmodules".
The blob is not part of the second push at all, but we
need to mark and check it.
So in the general case, we need to make up to three passes
over the objects: once to make sure we've seen all commits,
then once to cover any trees we might have missed, and then
a final pass to cover any .gitmodules blobs we found in the
second pass.
We can simplify things a bit by loosening the requirement
that we find .gitmodules only at root trees. Technically
a file like "subdir/.gitmodules" is not parsed by Git, but
it's not unreasonable for us to declare that Git is aware of
all ".gitmodules" files and make them eligible for checking.
That lets us drop the root-tree requirement, which
eliminates one pass entirely. And it makes our worst case
much better: instead of potentially queueing every root tree
to be re-examined, the worst case is that we queue each
unique .gitmodules blob for a second look.
This patch just adds the boilerplate to find .gitmodules
files. The actual content checks will come in a subsequent
commit.
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 21:20:08 +00:00
|
|
|
|
2022-12-01 14:46:01 +00:00
|
|
|
static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done,
|
|
|
|
enum fsck_msg_id msg_missing, enum fsck_msg_id msg_type,
|
|
|
|
struct fsck_options *options, const char *blob_type)
|
fsck: detect gitmodules files
In preparation for performing fsck checks on .gitmodules
files, this commit plumbs in the actual detection of the
files. Note that unlike most other fsck checks, this cannot
be a property of a single object: we must know that the
object is found at a ".gitmodules" path at the root tree of
a commit.
Since the fsck code only sees one object at a time, we have
to mark the related objects to fit the puzzle together. When
we see a commit we mark its tree as a root tree, and when
we see a root tree with a .gitmodules file, we mark the
corresponding blob to be checked.
In an ideal world, we'd check the objects in topological
order: commits followed by trees followed by blobs. In that
case we can avoid ever loading an object twice, since all
markings would be complete by the time we get to the marked
objects. And indeed, if we are checking a single packfile,
this is the order in which Git will generally write the
objects. But we can't count on that:
1. git-fsck may show us the objects in arbitrary order
(loose objects are fed in sha1 order, but we may also
have multiple packs, and we process each pack fully in
sequence).
2. The type ordering is just what git-pack-objects happens
to write now. The pack format does not require a
specific order, and it's possible that future versions
of Git (or a custom version trying to fool official
Git's fsck checks!) may order it differently.
3. We may not even be fscking all of the relevant objects
at once. Consider pushing with transfer.fsckObjects,
where one push adds a blob at path "foo", and then a
second push adds the same blob at path ".gitmodules".
The blob is not part of the second push at all, but we
need to mark and check it.
So in the general case, we need to make up to three passes
over the objects: once to make sure we've seen all commits,
then once to cover any trees we might have missed, and then
a final pass to cover any .gitmodules blobs we found in the
second pass.
We can simplify things a bit by loosening the requirement
that we find .gitmodules only at root trees. Technically
a file like "subdir/.gitmodules" is not parsed by Git, but
it's not unreasonable for us to declare that Git is aware of
all ".gitmodules" files and make them eligible for checking.
That lets us drop the root-tree requirement, which
eliminates one pass entirely. And it makes our worst case
much better: instead of potentially queueing every root tree
to be re-examined, the worst case is that we queue each
unique .gitmodules blob for a second look.
This patch just adds the boilerplate to find .gitmodules
files. The actual content checks will come in a subsequent
commit.
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 21:20:08 +00:00
|
|
|
{
|
|
|
|
int ret = 0;
|
|
|
|
struct oidset_iter iter;
|
|
|
|
const struct object_id *oid;
|
|
|
|
|
2022-12-01 14:46:01 +00:00
|
|
|
oidset_iter_init(blobs_found, &iter);
|
fsck: detect gitmodules files
In preparation for performing fsck checks on .gitmodules
files, this commit plumbs in the actual detection of the
files. Note that unlike most other fsck checks, this cannot
be a property of a single object: we must know that the
object is found at a ".gitmodules" path at the root tree of
a commit.
Since the fsck code only sees one object at a time, we have
to mark the related objects to fit the puzzle together. When
we see a commit we mark its tree as a root tree, and when
we see a root tree with a .gitmodules file, we mark the
corresponding blob to be checked.
In an ideal world, we'd check the objects in topological
order: commits followed by trees followed by blobs. In that
case we can avoid ever loading an object twice, since all
markings would be complete by the time we get to the marked
objects. And indeed, if we are checking a single packfile,
this is the order in which Git will generally write the
objects. But we can't count on that:
1. git-fsck may show us the objects in arbitrary order
(loose objects are fed in sha1 order, but we may also
have multiple packs, and we process each pack fully in
sequence).
2. The type ordering is just what git-pack-objects happens
to write now. The pack format does not require a
specific order, and it's possible that future versions
of Git (or a custom version trying to fool official
Git's fsck checks!) may order it differently.
3. We may not even be fscking all of the relevant objects
at once. Consider pushing with transfer.fsckObjects,
where one push adds a blob at path "foo", and then a
second push adds the same blob at path ".gitmodules".
The blob is not part of the second push at all, but we
need to mark and check it.
So in the general case, we need to make up to three passes
over the objects: once to make sure we've seen all commits,
then once to cover any trees we might have missed, and then
a final pass to cover any .gitmodules blobs we found in the
second pass.
We can simplify things a bit by loosening the requirement
that we find .gitmodules only at root trees. Technically
a file like "subdir/.gitmodules" is not parsed by Git, but
it's not unreasonable for us to declare that Git is aware of
all ".gitmodules" files and make them eligible for checking.
That lets us drop the root-tree requirement, which
eliminates one pass entirely. And it makes our worst case
much better: instead of potentially queueing every root tree
to be re-examined, the worst case is that we queue each
unique .gitmodules blob for a second look.
This patch just adds the boilerplate to find .gitmodules
files. The actual content checks will come in a subsequent
commit.
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 21:20:08 +00:00
|
|
|
while ((oid = oidset_iter_next(&iter))) {
|
|
|
|
enum object_type type;
|
|
|
|
unsigned long size;
|
|
|
|
char *buf;
|
|
|
|
|
2022-12-01 14:46:01 +00:00
|
|
|
if (oidset_contains(blobs_done, oid))
|
fsck: detect gitmodules files
In preparation for performing fsck checks on .gitmodules
files, this commit plumbs in the actual detection of the
files. Note that unlike most other fsck checks, this cannot
be a property of a single object: we must know that the
object is found at a ".gitmodules" path at the root tree of
a commit.
Since the fsck code only sees one object at a time, we have
to mark the related objects to fit the puzzle together. When
we see a commit we mark its tree as a root tree, and when
we see a root tree with a .gitmodules file, we mark the
corresponding blob to be checked.
In an ideal world, we'd check the objects in topological
order: commits followed by trees followed by blobs. In that
case we can avoid ever loading an object twice, since all
markings would be complete by the time we get to the marked
objects. And indeed, if we are checking a single packfile,
this is the order in which Git will generally write the
objects. But we can't count on that:
1. git-fsck may show us the objects in arbitrary order
(loose objects are fed in sha1 order, but we may also
have multiple packs, and we process each pack fully in
sequence).
2. The type ordering is just what git-pack-objects happens
to write now. The pack format does not require a
specific order, and it's possible that future versions
of Git (or a custom version trying to fool official
Git's fsck checks!) may order it differently.
3. We may not even be fscking all of the relevant objects
at once. Consider pushing with transfer.fsckObjects,
where one push adds a blob at path "foo", and then a
second push adds the same blob at path ".gitmodules".
The blob is not part of the second push at all, but we
need to mark and check it.
So in the general case, we need to make up to three passes
over the objects: once to make sure we've seen all commits,
then once to cover any trees we might have missed, and then
a final pass to cover any .gitmodules blobs we found in the
second pass.
We can simplify things a bit by loosening the requirement
that we find .gitmodules only at root trees. Technically
a file like "subdir/.gitmodules" is not parsed by Git, but
it's not unreasonable for us to declare that Git is aware of
all ".gitmodules" files and make them eligible for checking.
That lets us drop the root-tree requirement, which
eliminates one pass entirely. And it makes our worst case
much better: instead of potentially queueing every root tree
to be re-examined, the worst case is that we queue each
unique .gitmodules blob for a second look.
This patch just adds the boilerplate to find .gitmodules
files. The actual content checks will come in a subsequent
commit.
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 21:20:08 +00:00
|
|
|
continue;
|
|
|
|
|
2023-03-28 13:58:50 +00:00
|
|
|
buf = repo_read_object_file(the_repository, oid, &type, &size);
|
fsck: detect gitmodules files
In preparation for performing fsck checks on .gitmodules
files, this commit plumbs in the actual detection of the
files. Note that unlike most other fsck checks, this cannot
be a property of a single object: we must know that the
object is found at a ".gitmodules" path at the root tree of
a commit.
Since the fsck code only sees one object at a time, we have
to mark the related objects to fit the puzzle together. When
we see a commit we mark its tree as a root tree, and when
we see a root tree with a .gitmodules file, we mark the
corresponding blob to be checked.
In an ideal world, we'd check the objects in topological
order: commits followed by trees followed by blobs. In that
case we can avoid ever loading an object twice, since all
markings would be complete by the time we get to the marked
objects. And indeed, if we are checking a single packfile,
this is the order in which Git will generally write the
objects. But we can't count on that:
1. git-fsck may show us the objects in arbitrary order
(loose objects are fed in sha1 order, but we may also
have multiple packs, and we process each pack fully in
sequence).
2. The type ordering is just what git-pack-objects happens
to write now. The pack format does not require a
specific order, and it's possible that future versions
of Git (or a custom version trying to fool official
Git's fsck checks!) may order it differently.
3. We may not even be fscking all of the relevant objects
at once. Consider pushing with transfer.fsckObjects,
where one push adds a blob at path "foo", and then a
second push adds the same blob at path ".gitmodules".
The blob is not part of the second push at all, but we
need to mark and check it.
So in the general case, we need to make up to three passes
over the objects: once to make sure we've seen all commits,
then once to cover any trees we might have missed, and then
a final pass to cover any .gitmodules blobs we found in the
second pass.
We can simplify things a bit by loosening the requirement
that we find .gitmodules only at root trees. Technically
a file like "subdir/.gitmodules" is not parsed by Git, but
it's not unreasonable for us to declare that Git is aware of
all ".gitmodules" files and make them eligible for checking.
That lets us drop the root-tree requirement, which
eliminates one pass entirely. And it makes our worst case
much better: instead of potentially queueing every root tree
to be re-examined, the worst case is that we queue each
unique .gitmodules blob for a second look.
This patch just adds the boilerplate to find .gitmodules
files. The actual content checks will come in a subsequent
commit.
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 21:20:08 +00:00
|
|
|
if (!buf) {
|
2019-10-18 04:59:54 +00:00
|
|
|
if (is_promisor_object(oid))
|
2018-05-14 16:22:48 +00:00
|
|
|
continue;
|
2019-10-18 04:59:15 +00:00
|
|
|
ret |= report(options,
|
2022-12-01 14:46:01 +00:00
|
|
|
oid, OBJ_BLOB, msg_missing,
|
|
|
|
"unable to read %s blob", blob_type);
|
fsck: detect gitmodules files
In preparation for performing fsck checks on .gitmodules
files, this commit plumbs in the actual detection of the
files. Note that unlike most other fsck checks, this cannot
be a property of a single object: we must know that the
object is found at a ".gitmodules" path at the root tree of
a commit.
Since the fsck code only sees one object at a time, we have
to mark the related objects to fit the puzzle together. When
we see a commit we mark its tree as a root tree, and when
we see a root tree with a .gitmodules file, we mark the
corresponding blob to be checked.
In an ideal world, we'd check the objects in topological
order: commits followed by trees followed by blobs. In that
case we can avoid ever loading an object twice, since all
markings would be complete by the time we get to the marked
objects. And indeed, if we are checking a single packfile,
this is the order in which Git will generally write the
objects. But we can't count on that:
1. git-fsck may show us the objects in arbitrary order
(loose objects are fed in sha1 order, but we may also
have multiple packs, and we process each pack fully in
sequence).
2. The type ordering is just what git-pack-objects happens
to write now. The pack format does not require a
specific order, and it's possible that future versions
of Git (or a custom version trying to fool official
Git's fsck checks!) may order it differently.
3. We may not even be fscking all of the relevant objects
at once. Consider pushing with transfer.fsckObjects,
where one push adds a blob at path "foo", and then a
second push adds the same blob at path ".gitmodules".
The blob is not part of the second push at all, but we
need to mark and check it.
So in the general case, we need to make up to three passes
over the objects: once to make sure we've seen all commits,
then once to cover any trees we might have missed, and then
a final pass to cover any .gitmodules blobs we found in the
second pass.
We can simplify things a bit by loosening the requirement
that we find .gitmodules only at root trees. Technically
a file like "subdir/.gitmodules" is not parsed by Git, but
it's not unreasonable for us to declare that Git is aware of
all ".gitmodules" files and make them eligible for checking.
That lets us drop the root-tree requirement, which
eliminates one pass entirely. And it makes our worst case
much better: instead of potentially queueing every root tree
to be re-examined, the worst case is that we queue each
unique .gitmodules blob for a second look.
This patch just adds the boilerplate to find .gitmodules
files. The actual content checks will come in a subsequent
commit.
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 21:20:08 +00:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (type == OBJ_BLOB)
|
2019-10-18 04:59:54 +00:00
|
|
|
ret |= fsck_blob(oid, buf, size, options);
|
fsck: detect gitmodules files
In preparation for performing fsck checks on .gitmodules
files, this commit plumbs in the actual detection of the
files. Note that unlike most other fsck checks, this cannot
be a property of a single object: we must know that the
object is found at a ".gitmodules" path at the root tree of
a commit.
Since the fsck code only sees one object at a time, we have
to mark the related objects to fit the puzzle together. When
we see a commit we mark its tree as a root tree, and when
we see a root tree with a .gitmodules file, we mark the
corresponding blob to be checked.
In an ideal world, we'd check the objects in topological
order: commits followed by trees followed by blobs. In that
case we can avoid ever loading an object twice, since all
markings would be complete by the time we get to the marked
objects. And indeed, if we are checking a single packfile,
this is the order in which Git will generally write the
objects. But we can't count on that:
1. git-fsck may show us the objects in arbitrary order
(loose objects are fed in sha1 order, but we may also
have multiple packs, and we process each pack fully in
sequence).
2. The type ordering is just what git-pack-objects happens
to write now. The pack format does not require a
specific order, and it's possible that future versions
of Git (or a custom version trying to fool official
Git's fsck checks!) may order it differently.
3. We may not even be fscking all of the relevant objects
at once. Consider pushing with transfer.fsckObjects,
where one push adds a blob at path "foo", and then a
second push adds the same blob at path ".gitmodules".
The blob is not part of the second push at all, but we
need to mark and check it.
So in the general case, we need to make up to three passes
over the objects: once to make sure we've seen all commits,
then once to cover any trees we might have missed, and then
a final pass to cover any .gitmodules blobs we found in the
second pass.
We can simplify things a bit by loosening the requirement
that we find .gitmodules only at root trees. Technically
a file like "subdir/.gitmodules" is not parsed by Git, but
it's not unreasonable for us to declare that Git is aware of
all ".gitmodules" files and make them eligible for checking.
That lets us drop the root-tree requirement, which
eliminates one pass entirely. And it makes our worst case
much better: instead of potentially queueing every root tree
to be re-examined, the worst case is that we queue each
unique .gitmodules blob for a second look.
This patch just adds the boilerplate to find .gitmodules
files. The actual content checks will come in a subsequent
commit.
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 21:20:08 +00:00
|
|
|
else
|
2022-12-01 14:46:01 +00:00
|
|
|
ret |= report(options, oid, type, msg_type,
|
|
|
|
"non-blob found at %s", blob_type);
|
fsck: detect gitmodules files
In preparation for performing fsck checks on .gitmodules
files, this commit plumbs in the actual detection of the
files. Note that unlike most other fsck checks, this cannot
be a property of a single object: we must know that the
object is found at a ".gitmodules" path at the root tree of
a commit.
Since the fsck code only sees one object at a time, we have
to mark the related objects to fit the puzzle together. When
we see a commit we mark its tree as a root tree, and when
we see a root tree with a .gitmodules file, we mark the
corresponding blob to be checked.
In an ideal world, we'd check the objects in topological
order: commits followed by trees followed by blobs. In that
case we can avoid ever loading an object twice, since all
markings would be complete by the time we get to the marked
objects. And indeed, if we are checking a single packfile,
this is the order in which Git will generally write the
objects. But we can't count on that:
1. git-fsck may show us the objects in arbitrary order
(loose objects are fed in sha1 order, but we may also
have multiple packs, and we process each pack fully in
sequence).
2. The type ordering is just what git-pack-objects happens
to write now. The pack format does not require a
specific order, and it's possible that future versions
of Git (or a custom version trying to fool official
Git's fsck checks!) may order it differently.
3. We may not even be fscking all of the relevant objects
at once. Consider pushing with transfer.fsckObjects,
where one push adds a blob at path "foo", and then a
second push adds the same blob at path ".gitmodules".
The blob is not part of the second push at all, but we
need to mark and check it.
So in the general case, we need to make up to three passes
over the objects: once to make sure we've seen all commits,
then once to cover any trees we might have missed, and then
a final pass to cover any .gitmodules blobs we found in the
second pass.
We can simplify things a bit by loosening the requirement
that we find .gitmodules only at root trees. Technically
a file like "subdir/.gitmodules" is not parsed by Git, but
it's not unreasonable for us to declare that Git is aware of
all ".gitmodules" files and make them eligible for checking.
That lets us drop the root-tree requirement, which
eliminates one pass entirely. And it makes our worst case
much better: instead of potentially queueing every root tree
to be re-examined, the worst case is that we queue each
unique .gitmodules blob for a second look.
This patch just adds the boilerplate to find .gitmodules
files. The actual content checks will come in a subsequent
commit.
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 21:20:08 +00:00
|
|
|
free(buf);
|
|
|
|
}
|
|
|
|
|
2022-12-01 14:46:01 +00:00
|
|
|
oidset_clear(blobs_found);
|
|
|
|
oidset_clear(blobs_done);
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
int fsck_finish(struct fsck_options *options)
|
|
|
|
{
|
|
|
|
int ret = 0;
|
|
|
|
|
|
|
|
ret |= fsck_blobs(&options->gitmodules_found, &options->gitmodules_done,
|
|
|
|
FSCK_MSG_GITMODULES_MISSING, FSCK_MSG_GITMODULES_BLOB,
|
|
|
|
options, ".gitmodules");
|
2022-12-01 14:46:09 +00:00
|
|
|
ret |= fsck_blobs(&options->gitattributes_found, &options->gitattributes_done,
|
|
|
|
FSCK_MSG_GITATTRIBUTES_MISSING, FSCK_MSG_GITATTRIBUTES_BLOB,
|
|
|
|
options, ".gitattributes");
|
fsck: detect gitmodules files
In preparation for performing fsck checks on .gitmodules
files, this commit plumbs in the actual detection of the
files. Note that unlike most other fsck checks, this cannot
be a property of a single object: we must know that the
object is found at a ".gitmodules" path at the root tree of
a commit.
Since the fsck code only sees one object at a time, we have
to mark the related objects to fit the puzzle together. When
we see a commit we mark its tree as a root tree, and when
we see a root tree with a .gitmodules file, we mark the
corresponding blob to be checked.
In an ideal world, we'd check the objects in topological
order: commits followed by trees followed by blobs. In that
case we can avoid ever loading an object twice, since all
markings would be complete by the time we get to the marked
objects. And indeed, if we are checking a single packfile,
this is the order in which Git will generally write the
objects. But we can't count on that:
1. git-fsck may show us the objects in arbitrary order
(loose objects are fed in sha1 order, but we may also
have multiple packs, and we process each pack fully in
sequence).
2. The type ordering is just what git-pack-objects happens
to write now. The pack format does not require a
specific order, and it's possible that future versions
of Git (or a custom version trying to fool official
Git's fsck checks!) may order it differently.
3. We may not even be fscking all of the relevant objects
at once. Consider pushing with transfer.fsckObjects,
where one push adds a blob at path "foo", and then a
second push adds the same blob at path ".gitmodules".
The blob is not part of the second push at all, but we
need to mark and check it.
So in the general case, we need to make up to three passes
over the objects: once to make sure we've seen all commits,
then once to cover any trees we might have missed, and then
a final pass to cover any .gitmodules blobs we found in the
second pass.
We can simplify things a bit by loosening the requirement
that we find .gitmodules only at root trees. Technically
a file like "subdir/.gitmodules" is not parsed by Git, but
it's not unreasonable for us to declare that Git is aware of
all ".gitmodules" files and make them eligible for checking.
That lets us drop the root-tree requirement, which
eliminates one pass entirely. And it makes our worst case
much better: instead of potentially queueing every root tree
to be re-examined, the worst case is that we queue each
unique .gitmodules blob for a second look.
This patch just adds the boilerplate to find .gitmodules
files. The actual content checks will come in a subsequent
commit.
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 21:20:08 +00:00
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
2021-01-05 19:42:47 +00:00
|
|
|
|
config: add ctx arg to config_fn_t
Add a new "const struct config_context *ctx" arg to config_fn_t to hold
additional information about the config iteration operation.
config_context has a "struct key_value_info kvi" member that holds
metadata about the config source being read (e.g. what kind of config
source it is, the filename, etc). In this series, we're only interested
in .kvi, so we could have just used "struct key_value_info" as an arg,
but config_context makes it possible to add/adjust members in the future
without changing the config_fn_t signature. We could also consider other
ways of organizing the args (e.g. moving the config name and value into
config_context or key_value_info), but in my experiments, the
incremental benefit doesn't justify the added complexity (e.g. a
config_fn_t will sometimes invoke another config_fn_t but with a
different config value).
In subsequent commits, the .kvi member will replace the global "struct
config_reader" in config.c, making config iteration a global-free
operation. It requires much more work for the machinery to provide
meaningful values of .kvi, so for now, merely change the signature and
call sites, pass NULL as a placeholder value, and don't rely on the arg
in any meaningful way.
Most of the changes are performed by
contrib/coccinelle/config_fn_ctx.pending.cocci, which, for every
config_fn_t:
- Modifies the signature to accept "const struct config_context *ctx"
- Passes "ctx" to any inner config_fn_t, if needed
- Adds UNUSED attributes to "ctx", if needed
Most config_fn_t instances are easily identified by seeing if they are
called by the various config functions. Most of the remaining ones are
manually named in the .cocci patch. Manual cleanups are still needed,
but the majority of it is trivial; it's either adjusting config_fn_t
that the .cocci patch didn't catch, or adding forward declarations of
"struct config_context ctx" to make the signatures make sense.
The non-trivial changes are in cases where we are invoking a config_fn_t
outside of config machinery, and we now need to decide what value of
"ctx" to pass. These cases are:
- trace2/tr2_cfg.c:tr2_cfg_set_fl()
This is indirectly called by git_config_set() so that the trace2
machinery can notice the new config values and update its settings
using the tr2 config parsing function, i.e. tr2_cfg_cb().
- builtin/checkout.c:checkout_main()
This calls git_xmerge_config() as a shorthand for parsing a CLI arg.
This might be worth refactoring away in the future, since
git_xmerge_config() can call git_default_config(), which can do much
more than just parsing.
Handle them by creating a KVI_INIT macro that initializes "struct
key_value_info" to a reasonable default, and use that to construct the
"ctx" arg.
Signed-off-by: Glen Choo <chooglen@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-06-28 19:26:22 +00:00
|
|
|
int git_fsck_config(const char *var, const char *value,
|
|
|
|
const struct config_context *ctx, void *cb)
|
2021-01-05 19:42:47 +00:00
|
|
|
{
|
2021-03-17 18:20:36 +00:00
|
|
|
struct fsck_options *options = cb;
|
2021-01-05 19:42:47 +00:00
|
|
|
if (strcmp(var, "fsck.skiplist") == 0) {
|
|
|
|
const char *path;
|
|
|
|
struct strbuf sb = STRBUF_INIT;
|
|
|
|
|
|
|
|
if (git_config_pathname(&path, var, value))
|
|
|
|
return 1;
|
|
|
|
strbuf_addf(&sb, "skiplist=%s", path);
|
|
|
|
free((char *)path);
|
|
|
|
fsck_set_msg_types(options, sb.buf);
|
|
|
|
strbuf_release(&sb);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (skip_prefix(var, "fsck.", &var)) {
|
|
|
|
fsck_set_msg_type(options, var, value);
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
config: add ctx arg to config_fn_t
Add a new "const struct config_context *ctx" arg to config_fn_t to hold
additional information about the config iteration operation.
config_context has a "struct key_value_info kvi" member that holds
metadata about the config source being read (e.g. what kind of config
source it is, the filename, etc). In this series, we're only interested
in .kvi, so we could have just used "struct key_value_info" as an arg,
but config_context makes it possible to add/adjust members in the future
without changing the config_fn_t signature. We could also consider other
ways of organizing the args (e.g. moving the config name and value into
config_context or key_value_info), but in my experiments, the
incremental benefit doesn't justify the added complexity (e.g. a
config_fn_t will sometimes invoke another config_fn_t but with a
different config value).
In subsequent commits, the .kvi member will replace the global "struct
config_reader" in config.c, making config iteration a global-free
operation. It requires much more work for the machinery to provide
meaningful values of .kvi, so for now, merely change the signature and
call sites, pass NULL as a placeholder value, and don't rely on the arg
in any meaningful way.
Most of the changes are performed by
contrib/coccinelle/config_fn_ctx.pending.cocci, which, for every
config_fn_t:
- Modifies the signature to accept "const struct config_context *ctx"
- Passes "ctx" to any inner config_fn_t, if needed
- Adds UNUSED attributes to "ctx", if needed
Most config_fn_t instances are easily identified by seeing if they are
called by the various config functions. Most of the remaining ones are
manually named in the .cocci patch. Manual cleanups are still needed,
but the majority of it is trivial; it's either adjusting config_fn_t
that the .cocci patch didn't catch, or adding forward declarations of
"struct config_context ctx" to make the signatures make sense.
The non-trivial changes are in cases where we are invoking a config_fn_t
outside of config machinery, and we now need to decide what value of
"ctx" to pass. These cases are:
- trace2/tr2_cfg.c:tr2_cfg_set_fl()
This is indirectly called by git_config_set() so that the trace2
machinery can notice the new config values and update its settings
using the tr2 config parsing function, i.e. tr2_cfg_cb().
- builtin/checkout.c:checkout_main()
This calls git_xmerge_config() as a shorthand for parsing a CLI arg.
This might be worth refactoring away in the future, since
git_xmerge_config() can call git_default_config(), which can do much
more than just parsing.
Handle them by creating a KVI_INIT macro that initializes "struct
key_value_info" to a reasonable default, and use that to construct the
"ctx" arg.
Signed-off-by: Glen Choo <chooglen@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2023-06-28 19:26:22 +00:00
|
|
|
return git_default_config(var, value, ctx, cb);
|
2021-01-05 19:42:47 +00:00
|
|
|
}
|
2021-03-28 13:15:51 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* Custom error callbacks that are used in more than one place.
|
|
|
|
*/
|
|
|
|
|
|
|
|
int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,
|
|
|
|
const struct object_id *oid,
|
|
|
|
enum object_type object_type,
|
|
|
|
enum fsck_msg_type msg_type,
|
|
|
|
enum fsck_msg_id msg_id,
|
|
|
|
const char *message)
|
|
|
|
{
|
|
|
|
if (msg_id == FSCK_MSG_GITMODULES_MISSING) {
|
|
|
|
puts(oid_to_hex(oid));
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
return fsck_error_function(o, oid, object_type, msg_type, msg_id, message);
|
|
|
|
}
|