2008-02-25 21:46:04 +00:00
|
|
|
#include "cache.h"
|
2018-05-15 23:42:15 +00:00
|
|
|
#include "object-store.h"
|
2018-06-29 01:21:51 +00:00
|
|
|
#include "repository.h"
|
2008-02-25 21:46:04 +00:00
|
|
|
#include "object.h"
|
|
|
|
#include "blob.h"
|
|
|
|
#include "tree.h"
|
|
|
|
#include "tree-walk.h"
|
|
|
|
#include "commit.h"
|
|
|
|
#include "tag.h"
|
|
|
|
#include "fsck.h"
|
2014-09-11 14:26:38 +00:00
|
|
|
#include "refs.h"
|
fsck: complain about HFS+ ".git" aliases in trees
Now that the index can block pathnames that case-fold to
".git" on HFS+, it would be helpful for fsck to notice such
problematic paths. This lets servers which use
receive.fsckObjects block them before the damage spreads.
Note that the fsck check is always on, even for systems
without core.protectHFS set. This is technically more
restrictive than we need to be, as a set of users on ext4
could happily use these odd filenames without caring about
HFS+.
However, on balance, it's helpful for all servers to block
these (because the paths can be used for mischief, and
servers which bother to fsck would want to stop the spread
whether they are on HFS+ themselves or not), and hardly
anybody will be affected (because the blocked names are
variants of .git with invisible Unicode code-points mixed
in, meaning mischief is almost certainly what the tree
author had in mind).
Ideally these would be controlled by a separate
"fsck.protectHFS" flag. However, it would be much nicer to
be able to enable/disable _any_ fsck flag individually, and
any scheme we choose should match such a system. Given the
likelihood of anybody using such a path in practice, it is
not unreasonable to wait until such a system materializes.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-12-15 23:21:57 +00:00
|
|
|
#include "utf8.h"
|
2015-06-22 15:27:18 +00:00
|
|
|
#include "sha1-array.h"
|
2016-07-17 10:59:49 +00:00
|
|
|
#include "decorate.h"
|
fsck: detect gitmodules files
In preparation for performing fsck checks on .gitmodules
files, this commit plumbs in the actual detection of the
files. Note that unlike most other fsck checks, this cannot
be a property of a single object: we must know that the
object is found at a ".gitmodules" path at the root tree of
a commit.
Since the fsck code only sees one object at a time, we have
to mark the related objects to fit the puzzle together. When
we see a commit we mark its tree as a root tree, and when
we see a root tree with a .gitmodules file, we mark the
corresponding blob to be checked.
In an ideal world, we'd check the objects in topological
order: commits followed by trees followed by blobs. In that
case we can avoid ever loading an object twice, since all
markings would be complete by the time we get to the marked
objects. And indeed, if we are checking a single packfile,
this is the order in which Git will generally write the
objects. But we can't count on that:
1. git-fsck may show us the objects in arbitrary order
(loose objects are fed in sha1 order, but we may also
have multiple packs, and we process each pack fully in
sequence).
2. The type ordering is just what git-pack-objects happens
to write now. The pack format does not require a
specific order, and it's possible that future versions
of Git (or a custom version trying to fool official
Git's fsck checks!) may order it differently.
3. We may not even be fscking all of the relevant objects
at once. Consider pushing with transfer.fsckObjects,
where one push adds a blob at path "foo", and then a
second push adds the same blob at path ".gitmodules".
The blob is not part of the second push at all, but we
need to mark and check it.
So in the general case, we need to make up to three passes
over the objects: once to make sure we've seen all commits,
then once to cover any trees we might have missed, and then
a final pass to cover any .gitmodules blobs we found in the
second pass.
We can simplify things a bit by loosening the requirement
that we find .gitmodules only at root trees. Technically
a file like "subdir/.gitmodules" is not parsed by Git, but
it's not unreasonable for us to declare that Git is aware of
all ".gitmodules" files and make them eligible for checking.
That lets us drop the root-tree requirement, which
eliminates one pass entirely. And it makes our worst case
much better: instead of potentially queueing every root tree
to be re-examined, the worst case is that we queue each
unique .gitmodules blob for a second look.
This patch just adds the boilerplate to find .gitmodules
files. The actual content checks will come in a subsequent
commit.
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 21:20:08 +00:00
|
|
|
#include "oidset.h"
|
2018-05-14 16:22:48 +00:00
|
|
|
#include "packfile.h"
|
2018-05-02 21:25:27 +00:00
|
|
|
#include "submodule-config.h"
|
|
|
|
#include "config.h"
|
2018-05-26 13:55:24 +00:00
|
|
|
#include "help.h"
|
fsck: detect gitmodules files
In preparation for performing fsck checks on .gitmodules
files, this commit plumbs in the actual detection of the
files. Note that unlike most other fsck checks, this cannot
be a property of a single object: we must know that the
object is found at a ".gitmodules" path at the root tree of
a commit.
Since the fsck code only sees one object at a time, we have
to mark the related objects to fit the puzzle together. When
we see a commit we mark its tree as a root tree, and when
we see a root tree with a .gitmodules file, we mark the
corresponding blob to be checked.
In an ideal world, we'd check the objects in topological
order: commits followed by trees followed by blobs. In that
case we can avoid ever loading an object twice, since all
markings would be complete by the time we get to the marked
objects. And indeed, if we are checking a single packfile,
this is the order in which Git will generally write the
objects. But we can't count on that:
1. git-fsck may show us the objects in arbitrary order
(loose objects are fed in sha1 order, but we may also
have multiple packs, and we process each pack fully in
sequence).
2. The type ordering is just what git-pack-objects happens
to write now. The pack format does not require a
specific order, and it's possible that future versions
of Git (or a custom version trying to fool official
Git's fsck checks!) may order it differently.
3. We may not even be fscking all of the relevant objects
at once. Consider pushing with transfer.fsckObjects,
where one push adds a blob at path "foo", and then a
second push adds the same blob at path ".gitmodules".
The blob is not part of the second push at all, but we
need to mark and check it.
So in the general case, we need to make up to three passes
over the objects: once to make sure we've seen all commits,
then once to cover any trees we might have missed, and then
a final pass to cover any .gitmodules blobs we found in the
second pass.
We can simplify things a bit by loosening the requirement
that we find .gitmodules only at root trees. Technically
a file like "subdir/.gitmodules" is not parsed by Git, but
it's not unreasonable for us to declare that Git is aware of
all ".gitmodules" files and make them eligible for checking.
That lets us drop the root-tree requirement, which
eliminates one pass entirely. And it makes our worst case
much better: instead of potentially queueing every root tree
to be re-examined, the worst case is that we queue each
unique .gitmodules blob for a second look.
This patch just adds the boilerplate to find .gitmodules
files. The actual content checks will come in a subsequent
commit.
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 21:20:08 +00:00
|
|
|
|
|
|
|
static struct oidset gitmodules_found = OIDSET_INIT;
|
|
|
|
static struct oidset gitmodules_done = OIDSET_INIT;
|
2008-02-25 21:46:04 +00:00
|
|
|
|
2015-06-22 15:26:42 +00:00
|
|
|
#define FSCK_FATAL -1
|
2015-06-22 15:26:54 +00:00
|
|
|
#define FSCK_INFO -2
|
2015-06-22 15:26:42 +00:00
|
|
|
|
2015-06-22 15:25:09 +00:00
|
|
|
#define FOREACH_MSG_ID(FUNC) \
|
2015-06-22 15:26:42 +00:00
|
|
|
/* fatal errors */ \
|
|
|
|
FUNC(NUL_IN_HEADER, FATAL) \
|
|
|
|
FUNC(UNTERMINATED_HEADER, FATAL) \
|
2015-06-22 15:25:09 +00:00
|
|
|
/* errors */ \
|
|
|
|
FUNC(BAD_DATE, ERROR) \
|
|
|
|
FUNC(BAD_DATE_OVERFLOW, ERROR) \
|
|
|
|
FUNC(BAD_EMAIL, ERROR) \
|
|
|
|
FUNC(BAD_NAME, ERROR) \
|
|
|
|
FUNC(BAD_OBJECT_SHA1, ERROR) \
|
|
|
|
FUNC(BAD_PARENT_SHA1, ERROR) \
|
|
|
|
FUNC(BAD_TAG_OBJECT, ERROR) \
|
|
|
|
FUNC(BAD_TIMEZONE, ERROR) \
|
|
|
|
FUNC(BAD_TREE, ERROR) \
|
|
|
|
FUNC(BAD_TREE_SHA1, ERROR) \
|
|
|
|
FUNC(BAD_TYPE, ERROR) \
|
|
|
|
FUNC(DUPLICATE_ENTRIES, ERROR) \
|
|
|
|
FUNC(MISSING_AUTHOR, ERROR) \
|
|
|
|
FUNC(MISSING_COMMITTER, ERROR) \
|
|
|
|
FUNC(MISSING_EMAIL, ERROR) \
|
|
|
|
FUNC(MISSING_GRAFT, ERROR) \
|
|
|
|
FUNC(MISSING_NAME_BEFORE_EMAIL, ERROR) \
|
|
|
|
FUNC(MISSING_OBJECT, ERROR) \
|
|
|
|
FUNC(MISSING_PARENT, ERROR) \
|
|
|
|
FUNC(MISSING_SPACE_BEFORE_DATE, ERROR) \
|
|
|
|
FUNC(MISSING_SPACE_BEFORE_EMAIL, ERROR) \
|
|
|
|
FUNC(MISSING_TAG, ERROR) \
|
|
|
|
FUNC(MISSING_TAG_ENTRY, ERROR) \
|
|
|
|
FUNC(MISSING_TAG_OBJECT, ERROR) \
|
|
|
|
FUNC(MISSING_TREE, ERROR) \
|
fsck: detect gitmodules files
In preparation for performing fsck checks on .gitmodules
files, this commit plumbs in the actual detection of the
files. Note that unlike most other fsck checks, this cannot
be a property of a single object: we must know that the
object is found at a ".gitmodules" path at the root tree of
a commit.
Since the fsck code only sees one object at a time, we have
to mark the related objects to fit the puzzle together. When
we see a commit we mark its tree as a root tree, and when
we see a root tree with a .gitmodules file, we mark the
corresponding blob to be checked.
In an ideal world, we'd check the objects in topological
order: commits followed by trees followed by blobs. In that
case we can avoid ever loading an object twice, since all
markings would be complete by the time we get to the marked
objects. And indeed, if we are checking a single packfile,
this is the order in which Git will generally write the
objects. But we can't count on that:
1. git-fsck may show us the objects in arbitrary order
(loose objects are fed in sha1 order, but we may also
have multiple packs, and we process each pack fully in
sequence).
2. The type ordering is just what git-pack-objects happens
to write now. The pack format does not require a
specific order, and it's possible that future versions
of Git (or a custom version trying to fool official
Git's fsck checks!) may order it differently.
3. We may not even be fscking all of the relevant objects
at once. Consider pushing with transfer.fsckObjects,
where one push adds a blob at path "foo", and then a
second push adds the same blob at path ".gitmodules".
The blob is not part of the second push at all, but we
need to mark and check it.
So in the general case, we need to make up to three passes
over the objects: once to make sure we've seen all commits,
then once to cover any trees we might have missed, and then
a final pass to cover any .gitmodules blobs we found in the
second pass.
We can simplify things a bit by loosening the requirement
that we find .gitmodules only at root trees. Technically
a file like "subdir/.gitmodules" is not parsed by Git, but
it's not unreasonable for us to declare that Git is aware of
all ".gitmodules" files and make them eligible for checking.
That lets us drop the root-tree requirement, which
eliminates one pass entirely. And it makes our worst case
much better: instead of potentially queueing every root tree
to be re-examined, the worst case is that we queue each
unique .gitmodules blob for a second look.
This patch just adds the boilerplate to find .gitmodules
files. The actual content checks will come in a subsequent
commit.
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 21:20:08 +00:00
|
|
|
FUNC(MISSING_TREE_OBJECT, ERROR) \
|
2015-06-22 15:25:09 +00:00
|
|
|
FUNC(MISSING_TYPE, ERROR) \
|
|
|
|
FUNC(MISSING_TYPE_ENTRY, ERROR) \
|
2015-06-22 15:26:23 +00:00
|
|
|
FUNC(MULTIPLE_AUTHORS, ERROR) \
|
2015-06-22 15:25:09 +00:00
|
|
|
FUNC(TAG_OBJECT_NOT_TAG, ERROR) \
|
|
|
|
FUNC(TREE_NOT_SORTED, ERROR) \
|
|
|
|
FUNC(UNKNOWN_TYPE, ERROR) \
|
|
|
|
FUNC(ZERO_PADDED_DATE, ERROR) \
|
fsck: detect gitmodules files
In preparation for performing fsck checks on .gitmodules
files, this commit plumbs in the actual detection of the
files. Note that unlike most other fsck checks, this cannot
be a property of a single object: we must know that the
object is found at a ".gitmodules" path at the root tree of
a commit.
Since the fsck code only sees one object at a time, we have
to mark the related objects to fit the puzzle together. When
we see a commit we mark its tree as a root tree, and when
we see a root tree with a .gitmodules file, we mark the
corresponding blob to be checked.
In an ideal world, we'd check the objects in topological
order: commits followed by trees followed by blobs. In that
case we can avoid ever loading an object twice, since all
markings would be complete by the time we get to the marked
objects. And indeed, if we are checking a single packfile,
this is the order in which Git will generally write the
objects. But we can't count on that:
1. git-fsck may show us the objects in arbitrary order
(loose objects are fed in sha1 order, but we may also
have multiple packs, and we process each pack fully in
sequence).
2. The type ordering is just what git-pack-objects happens
to write now. The pack format does not require a
specific order, and it's possible that future versions
of Git (or a custom version trying to fool official
Git's fsck checks!) may order it differently.
3. We may not even be fscking all of the relevant objects
at once. Consider pushing with transfer.fsckObjects,
where one push adds a blob at path "foo", and then a
second push adds the same blob at path ".gitmodules".
The blob is not part of the second push at all, but we
need to mark and check it.
So in the general case, we need to make up to three passes
over the objects: once to make sure we've seen all commits,
then once to cover any trees we might have missed, and then
a final pass to cover any .gitmodules blobs we found in the
second pass.
We can simplify things a bit by loosening the requirement
that we find .gitmodules only at root trees. Technically
a file like "subdir/.gitmodules" is not parsed by Git, but
it's not unreasonable for us to declare that Git is aware of
all ".gitmodules" files and make them eligible for checking.
That lets us drop the root-tree requirement, which
eliminates one pass entirely. And it makes our worst case
much better: instead of potentially queueing every root tree
to be re-examined, the worst case is that we queue each
unique .gitmodules blob for a second look.
This patch just adds the boilerplate to find .gitmodules
files. The actual content checks will come in a subsequent
commit.
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 21:20:08 +00:00
|
|
|
FUNC(GITMODULES_MISSING, ERROR) \
|
|
|
|
FUNC(GITMODULES_BLOB, ERROR) \
|
2018-07-13 19:39:53 +00:00
|
|
|
FUNC(GITMODULES_LARGE, ERROR) \
|
2018-05-02 21:25:27 +00:00
|
|
|
FUNC(GITMODULES_NAME, ERROR) \
|
2018-05-05 00:03:35 +00:00
|
|
|
FUNC(GITMODULES_SYMLINK, ERROR) \
|
2015-06-22 15:25:09 +00:00
|
|
|
/* warnings */ \
|
|
|
|
FUNC(BAD_FILEMODE, WARN) \
|
|
|
|
FUNC(EMPTY_NAME, WARN) \
|
|
|
|
FUNC(FULL_PATHNAME, WARN) \
|
|
|
|
FUNC(HAS_DOT, WARN) \
|
|
|
|
FUNC(HAS_DOTDOT, WARN) \
|
|
|
|
FUNC(HAS_DOTGIT, WARN) \
|
|
|
|
FUNC(NULL_SHA1, WARN) \
|
2015-06-22 15:26:54 +00:00
|
|
|
FUNC(ZERO_PADDED_FILEMODE, WARN) \
|
2016-04-14 17:58:22 +00:00
|
|
|
FUNC(NUL_IN_COMMIT, WARN) \
|
2015-06-22 15:26:54 +00:00
|
|
|
/* infos (reported as warnings, but ignored by default) */ \
|
fsck: downgrade gitmodulesParse default to "info"
We added an fsck check in ed8b10f631 (fsck: check
.gitmodules content, 2018-05-02) as a defense against the
vulnerability from 0383bbb901 (submodule-config: verify
submodule names as paths, 2018-04-30). With the idea that
up-to-date hosting sites could protect downstream unpatched
clients that fetch from them.
As part of that defense, we reject any ".gitmodules" entry
that is not syntactically valid. The theory is that if we
cannot even parse the file, we cannot accurately check it
for vulnerabilities. And anybody with a broken .gitmodules
file would eventually want to know anyway.
But there are a few reasons this is a bad tradeoff in
practice:
- for this particular vulnerability, the client has to be
able to parse the file. So you cannot sneak an attack
through using a broken file, assuming the config parsers
for the process running fsck and the eventual victim are
functionally equivalent.
- a broken .gitmodules file is not necessarily a problem.
Our fsck check detects .gitmodules in _any_ tree, not
just at the root. And the presence of a .gitmodules file
does not necessarily mean it will be used; you'd have to
also have gitlinks in the tree. The cgit repository, for
example, has a file named .gitmodules from a
pre-submodule attempt at sharing code, but does not
actually have any gitlinks.
- when the fsck check is used to reject a push, it's often
hard to work around. The pusher may not have full control
over the destination repository (e.g., if it's on a
hosting server, they may need to contact the hosting
site's support). And the broken .gitmodules may be too
far back in history for rewriting to be feasible (again,
this is an issue for cgit).
So we're being unnecessarily restrictive without actually
improving the security in a meaningful way. It would be more
convenient to downgrade this check to "info", which means
we'd still comment on it, but not reject a push. Site admins
can already do this via config, but we should ship sensible
defaults.
There are a few counterpoints to consider in favor of
keeping the check as an error:
- the first point above assumes that the config parsers for
the victim and the fsck process are equivalent. This is
pretty true now, but as time goes on will become less so.
Hosting sites are likely to upgrade their version of Git,
whereas vulnerable clients will be stagnant (if they did
upgrade, they'd cease to be vulnerable!). So in theory we
may see drift over time between what two config parsers
will accept.
In practice, this is probably OK. The config format is
pretty established at this point and shouldn't change a
lot. And the farther we get from the announcement of the
vulnerability, the less interesting this extra layer of
protection becomes. I.e., it was _most_ valuable on day
0, when everybody's client was still vulnerable and
hosting sites could protect people. But as time goes on
and people upgrade, the population of vulnerable clients
becomes smaller and smaller.
- In theory this could protect us from other
vulnerabilities in the future. E.g., .gitmodules are the
only way for a malicious repository to feed data to the
config parser, so this check could similarly protect
clients from a future (to-be-found) bug there.
But that's trading a hypothetical case for real-world
pain today. If we do find such a bug, the hosting site
would need to be updated to fix it, too. At which point
we could figure out whether it's possible to detect
_just_ the malicious case without hurting existing
broken-but-not-evil cases.
- Until recently, we hadn't made any restrictions on
.gitmodules content. So now in tightening that we're
hitting cases where certain things used to work, but
don't anymore. There's some moderate pain now. But as
time goes on, we'll see more (and more varied) cases that
will make tightening harder in the future. So there's
some argument for putting rules in place _now_, before
users grow more cases that violate them.
Again, this is trading pain now for hypothetical benefit
in the future. And if we try hard in the future to keep
our tightening to a minimum (i.e., rejecting true
maliciousness without hurting broken-but-not-evil repos),
then that reduces even the hypothetical benefit.
Considering both sets of arguments, it makes sense to loosen
this check for now.
Note that we have to tweak the test in t7415 since fsck will
no longer consider this a fatal error. But we still check
that it reports the warning, and that we don't get the
spurious error from the config code.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-07-13 19:39:58 +00:00
|
|
|
FUNC(GITMODULES_PARSE, INFO) \
|
2015-06-22 15:26:54 +00:00
|
|
|
FUNC(BAD_TAG_NAME, INFO) \
|
|
|
|
FUNC(MISSING_TAGGER_ENTRY, INFO)
|
2015-06-22 15:25:09 +00:00
|
|
|
|
|
|
|
#define MSG_ID(id, msg_type) FSCK_MSG_##id,
|
|
|
|
enum fsck_msg_id {
|
|
|
|
FOREACH_MSG_ID(MSG_ID)
|
|
|
|
FSCK_MSG_MAX
|
|
|
|
};
|
|
|
|
#undef MSG_ID
|
|
|
|
|
2015-06-22 15:25:14 +00:00
|
|
|
#define STR(x) #x
|
2018-05-26 13:55:25 +00:00
|
|
|
#define MSG_ID(id, msg_type) { STR(id), NULL, NULL, FSCK_##msg_type },
|
2015-06-22 15:25:09 +00:00
|
|
|
static struct {
|
2015-06-22 15:25:14 +00:00
|
|
|
const char *id_string;
|
|
|
|
const char *downcased;
|
2018-05-26 13:55:25 +00:00
|
|
|
const char *camelcased;
|
2015-06-22 15:25:09 +00:00
|
|
|
int msg_type;
|
|
|
|
} msg_id_info[FSCK_MSG_MAX + 1] = {
|
|
|
|
FOREACH_MSG_ID(MSG_ID)
|
2018-05-26 13:55:25 +00:00
|
|
|
{ NULL, NULL, NULL, -1 }
|
2015-06-22 15:25:09 +00:00
|
|
|
};
|
|
|
|
#undef MSG_ID
|
|
|
|
|
2018-05-26 13:55:23 +00:00
|
|
|
static void prepare_msg_ids(void)
|
2015-06-22 15:25:14 +00:00
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
2018-05-26 13:55:23 +00:00
|
|
|
if (msg_id_info[0].downcased)
|
|
|
|
return;
|
|
|
|
|
|
|
|
/* convert id_string to lower case, without underscores. */
|
|
|
|
for (i = 0; i < FSCK_MSG_MAX; i++) {
|
|
|
|
const char *p = msg_id_info[i].id_string;
|
|
|
|
int len = strlen(p);
|
|
|
|
char *q = xmalloc(len);
|
|
|
|
|
|
|
|
msg_id_info[i].downcased = q;
|
|
|
|
while (*p)
|
|
|
|
if (*p == '_')
|
|
|
|
p++;
|
|
|
|
else
|
|
|
|
*(q)++ = tolower(*(p)++);
|
|
|
|
*q = '\0';
|
2018-05-26 13:55:25 +00:00
|
|
|
|
|
|
|
p = msg_id_info[i].id_string;
|
|
|
|
q = xmalloc(len);
|
|
|
|
msg_id_info[i].camelcased = q;
|
|
|
|
while (*p) {
|
|
|
|
if (*p == '_') {
|
|
|
|
p++;
|
|
|
|
if (*p)
|
|
|
|
*q++ = *p++;
|
|
|
|
} else {
|
|
|
|
*q++ = tolower(*p++);
|
|
|
|
}
|
2015-06-22 15:25:14 +00:00
|
|
|
}
|
2018-05-26 13:55:25 +00:00
|
|
|
*q = '\0';
|
2015-06-22 15:25:14 +00:00
|
|
|
}
|
2018-05-26 13:55:23 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
static int parse_msg_id(const char *text)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
prepare_msg_ids();
|
2015-06-22 15:25:14 +00:00
|
|
|
|
|
|
|
for (i = 0; i < FSCK_MSG_MAX; i++)
|
|
|
|
if (!strcmp(text, msg_id_info[i].downcased))
|
|
|
|
return i;
|
|
|
|
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
2018-05-26 13:55:24 +00:00
|
|
|
void list_config_fsck_msg_ids(struct string_list *list, const char *prefix)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
prepare_msg_ids();
|
|
|
|
|
|
|
|
for (i = 0; i < FSCK_MSG_MAX; i++)
|
2018-05-26 13:55:25 +00:00
|
|
|
list_config_item(list, prefix, msg_id_info[i].camelcased);
|
2018-05-26 13:55:24 +00:00
|
|
|
}
|
|
|
|
|
2015-06-22 15:25:09 +00:00
|
|
|
static int fsck_msg_type(enum fsck_msg_id msg_id,
|
|
|
|
struct fsck_options *options)
|
|
|
|
{
|
|
|
|
int msg_type;
|
|
|
|
|
2015-06-22 15:25:25 +00:00
|
|
|
assert(msg_id >= 0 && msg_id < FSCK_MSG_MAX);
|
|
|
|
|
|
|
|
if (options->msg_type)
|
|
|
|
msg_type = options->msg_type[msg_id];
|
|
|
|
else {
|
|
|
|
msg_type = msg_id_info[msg_id].msg_type;
|
|
|
|
if (options->strict && msg_type == FSCK_WARN)
|
|
|
|
msg_type = FSCK_ERROR;
|
|
|
|
}
|
2015-06-22 15:25:09 +00:00
|
|
|
|
|
|
|
return msg_type;
|
|
|
|
}
|
|
|
|
|
2015-06-22 15:27:18 +00:00
|
|
|
static void init_skiplist(struct fsck_options *options, const char *path)
|
|
|
|
{
|
2017-03-31 01:40:00 +00:00
|
|
|
static struct oid_array skiplist = OID_ARRAY_INIT;
|
2015-06-22 15:27:18 +00:00
|
|
|
int sorted, fd;
|
2017-03-26 16:01:30 +00:00
|
|
|
char buffer[GIT_MAX_HEXSZ + 1];
|
|
|
|
struct object_id oid;
|
2015-06-22 15:27:18 +00:00
|
|
|
|
|
|
|
if (options->skiplist)
|
|
|
|
sorted = options->skiplist->sorted;
|
|
|
|
else {
|
|
|
|
sorted = 1;
|
|
|
|
options->skiplist = &skiplist;
|
|
|
|
}
|
|
|
|
|
|
|
|
fd = open(path, O_RDONLY);
|
|
|
|
if (fd < 0)
|
|
|
|
die("Could not open skip list: %s", path);
|
|
|
|
for (;;) {
|
2017-03-26 16:01:30 +00:00
|
|
|
const char *p;
|
2015-06-22 15:27:18 +00:00
|
|
|
int result = read_in_full(fd, buffer, sizeof(buffer));
|
|
|
|
if (result < 0)
|
|
|
|
die_errno("Could not read '%s'", path);
|
|
|
|
if (!result)
|
|
|
|
break;
|
2017-03-26 16:01:30 +00:00
|
|
|
if (parse_oid_hex(buffer, &oid, &p) || *p != '\n')
|
2015-06-22 15:27:18 +00:00
|
|
|
die("Invalid SHA-1: %s", buffer);
|
2017-03-31 01:40:00 +00:00
|
|
|
oid_array_append(&skiplist, &oid);
|
2015-06-22 15:27:18 +00:00
|
|
|
if (sorted && skiplist.nr > 1 &&
|
2017-03-26 16:01:37 +00:00
|
|
|
oidcmp(&skiplist.oid[skiplist.nr - 2],
|
|
|
|
&oid) > 0)
|
2015-06-22 15:27:18 +00:00
|
|
|
sorted = 0;
|
|
|
|
}
|
|
|
|
close(fd);
|
|
|
|
|
|
|
|
if (sorted)
|
|
|
|
skiplist.sorted = 1;
|
|
|
|
}
|
|
|
|
|
2015-06-22 15:25:25 +00:00
|
|
|
static int parse_msg_type(const char *str)
|
|
|
|
{
|
|
|
|
if (!strcmp(str, "error"))
|
|
|
|
return FSCK_ERROR;
|
|
|
|
else if (!strcmp(str, "warn"))
|
|
|
|
return FSCK_WARN;
|
2015-06-22 15:26:48 +00:00
|
|
|
else if (!strcmp(str, "ignore"))
|
|
|
|
return FSCK_IGNORE;
|
2015-06-22 15:25:25 +00:00
|
|
|
else
|
|
|
|
die("Unknown fsck message type: '%s'", str);
|
|
|
|
}
|
|
|
|
|
2015-06-22 15:25:31 +00:00
|
|
|
int is_valid_msg_type(const char *msg_id, const char *msg_type)
|
|
|
|
{
|
|
|
|
if (parse_msg_id(msg_id) < 0)
|
|
|
|
return 0;
|
|
|
|
parse_msg_type(msg_type);
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
2015-06-22 15:25:25 +00:00
|
|
|
void fsck_set_msg_type(struct fsck_options *options,
|
|
|
|
const char *msg_id, const char *msg_type)
|
|
|
|
{
|
|
|
|
int id = parse_msg_id(msg_id), type;
|
|
|
|
|
|
|
|
if (id < 0)
|
|
|
|
die("Unhandled message id: %s", msg_id);
|
|
|
|
type = parse_msg_type(msg_type);
|
|
|
|
|
2015-06-22 15:26:42 +00:00
|
|
|
if (type != FSCK_ERROR && msg_id_info[id].msg_type == FSCK_FATAL)
|
|
|
|
die("Cannot demote %s to %s", msg_id, msg_type);
|
|
|
|
|
2015-06-22 15:25:25 +00:00
|
|
|
if (!options->msg_type) {
|
|
|
|
int i;
|
2016-02-22 22:44:25 +00:00
|
|
|
int *msg_type;
|
|
|
|
ALLOC_ARRAY(msg_type, FSCK_MSG_MAX);
|
2015-06-22 15:25:25 +00:00
|
|
|
for (i = 0; i < FSCK_MSG_MAX; i++)
|
|
|
|
msg_type[i] = fsck_msg_type(i, options);
|
|
|
|
options->msg_type = msg_type;
|
|
|
|
}
|
|
|
|
|
|
|
|
options->msg_type[id] = type;
|
|
|
|
}
|
|
|
|
|
|
|
|
void fsck_set_msg_types(struct fsck_options *options, const char *values)
|
|
|
|
{
|
|
|
|
char *buf = xstrdup(values), *to_free = buf;
|
|
|
|
int done = 0;
|
|
|
|
|
|
|
|
while (!done) {
|
|
|
|
int len = strcspn(buf, " ,|"), equal;
|
|
|
|
|
|
|
|
done = !buf[len];
|
|
|
|
if (!len) {
|
|
|
|
buf++;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
buf[len] = '\0';
|
|
|
|
|
|
|
|
for (equal = 0;
|
|
|
|
equal < len && buf[equal] != '=' && buf[equal] != ':';
|
|
|
|
equal++)
|
|
|
|
buf[equal] = tolower(buf[equal]);
|
|
|
|
buf[equal] = '\0';
|
|
|
|
|
2015-06-22 15:27:18 +00:00
|
|
|
if (!strcmp(buf, "skiplist")) {
|
|
|
|
if (equal == len)
|
|
|
|
die("skiplist requires a path");
|
|
|
|
init_skiplist(options, buf + equal + 1);
|
|
|
|
buf += len + 1;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
2015-06-22 15:25:25 +00:00
|
|
|
if (equal == len)
|
|
|
|
die("Missing '=': '%s'", buf);
|
2008-02-25 21:46:04 +00:00
|
|
|
|
2015-06-22 15:25:25 +00:00
|
|
|
fsck_set_msg_type(options, buf, buf + equal + 1);
|
|
|
|
buf += len + 1;
|
|
|
|
}
|
|
|
|
free(to_free);
|
|
|
|
}
|
|
|
|
|
2015-06-22 15:25:52 +00:00
|
|
|
static void append_msg_id(struct strbuf *sb, const char *msg_id)
|
|
|
|
{
|
|
|
|
for (;;) {
|
|
|
|
char c = *(msg_id)++;
|
|
|
|
|
|
|
|
if (!c)
|
|
|
|
break;
|
|
|
|
if (c != '_')
|
|
|
|
strbuf_addch(sb, tolower(c));
|
|
|
|
else {
|
|
|
|
assert(*msg_id);
|
|
|
|
strbuf_addch(sb, *(msg_id)++);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
strbuf_addstr(sb, ": ");
|
|
|
|
}
|
|
|
|
|
fsck: check skiplist for object in fsck_blob()
Since commit ed8b10f631 ("fsck: check .gitmodules content", 2018-05-02),
fsck will issue an error message for '.gitmodules' content that cannot
be parsed correctly. This is the case, even when the corresponding blob
object has been included on the skiplist. For example, using the cgit
repository, we see the following:
$ git fsck
Checking object directories: 100% (256/256), done.
error: bad config line 5 in blob .gitmodules
error in blob 51dd1eff1edc663674df9ab85d2786a40f7ae3a5: gitmodulesParse: could not parse gitmodules blob
Checking objects: 100% (6626/6626), done.
$
$ git config fsck.skiplist '.git/skip'
$ echo 51dd1eff1edc663674df9ab85d2786a40f7ae3a5 >.git/skip
$
$ git fsck
Checking object directories: 100% (256/256), done.
error: bad config line 5 in blob .gitmodules
Checking objects: 100% (6626/6626), done.
$
Note that the error message issued by the config parser is still
present, despite adding the object-id of the blob to the skiplist.
One solution would be to provide a means of suppressing the messages
issued by the config parser. However, given that (logically) we are
asking fsck to ignore this object, a simpler approach is to just not
call the config parser if the object is to be skipped. Add a check to
the 'fsck_blob()' processing function, to determine if the object is
on the skiplist and, if so, exit the function early.
Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-06-27 18:39:53 +00:00
|
|
|
static int object_on_skiplist(struct fsck_options *opts, struct object *obj)
|
|
|
|
{
|
|
|
|
if (opts && opts->skiplist && obj)
|
|
|
|
return oid_array_lookup(opts->skiplist, &obj->oid) >= 0;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2015-06-22 15:25:09 +00:00
|
|
|
__attribute__((format (printf, 4, 5)))
|
|
|
|
static int report(struct fsck_options *options, struct object *object,
|
|
|
|
enum fsck_msg_id id, const char *fmt, ...)
|
|
|
|
{
|
|
|
|
va_list ap;
|
|
|
|
struct strbuf sb = STRBUF_INIT;
|
|
|
|
int msg_type = fsck_msg_type(id, options), result;
|
|
|
|
|
2015-06-22 15:26:48 +00:00
|
|
|
if (msg_type == FSCK_IGNORE)
|
|
|
|
return 0;
|
|
|
|
|
fsck: check skiplist for object in fsck_blob()
Since commit ed8b10f631 ("fsck: check .gitmodules content", 2018-05-02),
fsck will issue an error message for '.gitmodules' content that cannot
be parsed correctly. This is the case, even when the corresponding blob
object has been included on the skiplist. For example, using the cgit
repository, we see the following:
$ git fsck
Checking object directories: 100% (256/256), done.
error: bad config line 5 in blob .gitmodules
error in blob 51dd1eff1edc663674df9ab85d2786a40f7ae3a5: gitmodulesParse: could not parse gitmodules blob
Checking objects: 100% (6626/6626), done.
$
$ git config fsck.skiplist '.git/skip'
$ echo 51dd1eff1edc663674df9ab85d2786a40f7ae3a5 >.git/skip
$
$ git fsck
Checking object directories: 100% (256/256), done.
error: bad config line 5 in blob .gitmodules
Checking objects: 100% (6626/6626), done.
$
Note that the error message issued by the config parser is still
present, despite adding the object-id of the blob to the skiplist.
One solution would be to provide a means of suppressing the messages
issued by the config parser. However, given that (logically) we are
asking fsck to ignore this object, a simpler approach is to just not
call the config parser if the object is to be skipped. Add a check to
the 'fsck_blob()' processing function, to determine if the object is
on the skiplist and, if so, exit the function early.
Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-06-27 18:39:53 +00:00
|
|
|
if (object_on_skiplist(options, object))
|
2015-06-22 15:27:18 +00:00
|
|
|
return 0;
|
|
|
|
|
2015-06-22 15:26:42 +00:00
|
|
|
if (msg_type == FSCK_FATAL)
|
|
|
|
msg_type = FSCK_ERROR;
|
2015-06-22 15:26:54 +00:00
|
|
|
else if (msg_type == FSCK_INFO)
|
|
|
|
msg_type = FSCK_WARN;
|
2015-06-22 15:26:42 +00:00
|
|
|
|
2015-06-22 15:25:52 +00:00
|
|
|
append_msg_id(&sb, msg_id_info[id].id_string);
|
|
|
|
|
2015-06-22 15:25:09 +00:00
|
|
|
va_start(ap, fmt);
|
|
|
|
strbuf_vaddf(&sb, fmt, ap);
|
2016-07-17 10:59:57 +00:00
|
|
|
result = options->error_func(options, object, msg_type, sb.buf);
|
2015-06-22 15:25:09 +00:00
|
|
|
strbuf_release(&sb);
|
|
|
|
va_end(ap);
|
|
|
|
|
|
|
|
return result;
|
|
|
|
}
|
|
|
|
|
2016-07-17 10:59:49 +00:00
|
|
|
static char *get_object_name(struct fsck_options *options, struct object *obj)
|
|
|
|
{
|
|
|
|
if (!options->object_names)
|
|
|
|
return NULL;
|
|
|
|
return lookup_decoration(options->object_names, obj);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void put_object_name(struct fsck_options *options, struct object *obj,
|
|
|
|
const char *fmt, ...)
|
|
|
|
{
|
|
|
|
va_list ap;
|
|
|
|
struct strbuf buf = STRBUF_INIT;
|
|
|
|
char *existing;
|
|
|
|
|
|
|
|
if (!options->object_names)
|
|
|
|
return;
|
|
|
|
existing = lookup_decoration(options->object_names, obj);
|
|
|
|
if (existing)
|
|
|
|
return;
|
|
|
|
va_start(ap, fmt);
|
|
|
|
strbuf_vaddf(&buf, fmt, ap);
|
|
|
|
add_decoration(options->object_names, obj, strbuf_detach(&buf, NULL));
|
|
|
|
va_end(ap);
|
|
|
|
}
|
|
|
|
|
2016-07-17 11:00:02 +00:00
|
|
|
static const char *describe_object(struct fsck_options *o, struct object *obj)
|
|
|
|
{
|
|
|
|
static struct strbuf buf = STRBUF_INIT;
|
|
|
|
char *name;
|
|
|
|
|
|
|
|
strbuf_reset(&buf);
|
|
|
|
strbuf_addstr(&buf, oid_to_hex(&obj->oid));
|
|
|
|
if (o->object_names && (name = lookup_decoration(o->object_names, obj)))
|
|
|
|
strbuf_addf(&buf, " (%s)", name);
|
|
|
|
|
|
|
|
return buf.buf;
|
|
|
|
}
|
|
|
|
|
2015-06-22 15:25:00 +00:00
|
|
|
static int fsck_walk_tree(struct tree *tree, void *data, struct fsck_options *options)
|
2008-02-25 21:46:04 +00:00
|
|
|
{
|
|
|
|
struct tree_desc desc;
|
|
|
|
struct name_entry entry;
|
|
|
|
int res = 0;
|
2016-07-17 10:59:49 +00:00
|
|
|
const char *name;
|
2008-02-25 21:46:04 +00:00
|
|
|
|
|
|
|
if (parse_tree(tree))
|
|
|
|
return -1;
|
|
|
|
|
2016-07-17 10:59:49 +00:00
|
|
|
name = get_object_name(options, &tree->object);
|
2016-09-27 20:59:51 +00:00
|
|
|
if (init_tree_desc_gently(&desc, tree->buffer, tree->size))
|
|
|
|
return -1;
|
|
|
|
while (tree_entry_gently(&desc, &entry)) {
|
2016-07-17 10:59:49 +00:00
|
|
|
struct object *obj;
|
2008-02-25 21:46:04 +00:00
|
|
|
int result;
|
|
|
|
|
|
|
|
if (S_ISGITLINK(entry.mode))
|
|
|
|
continue;
|
2016-07-17 10:59:49 +00:00
|
|
|
|
|
|
|
if (S_ISDIR(entry.mode)) {
|
2018-06-29 01:21:56 +00:00
|
|
|
obj = (struct object *)lookup_tree(the_repository, entry.oid);
|
2017-10-05 19:41:26 +00:00
|
|
|
if (name && obj)
|
2016-07-17 10:59:49 +00:00
|
|
|
put_object_name(options, obj, "%s%s/", name,
|
|
|
|
entry.path);
|
|
|
|
result = options->walk(obj, OBJ_TREE, data, options);
|
|
|
|
}
|
|
|
|
else if (S_ISREG(entry.mode) || S_ISLNK(entry.mode)) {
|
2018-06-29 01:21:55 +00:00
|
|
|
obj = (struct object *)lookup_blob(the_repository, entry.oid);
|
2017-10-05 19:41:26 +00:00
|
|
|
if (name && obj)
|
2016-07-17 10:59:49 +00:00
|
|
|
put_object_name(options, obj, "%s%s", name,
|
|
|
|
entry.path);
|
|
|
|
result = options->walk(obj, OBJ_BLOB, data, options);
|
|
|
|
}
|
2008-02-25 21:46:04 +00:00
|
|
|
else {
|
2012-04-30 00:28:45 +00:00
|
|
|
result = error("in tree %s: entry %s has bad mode %.6o",
|
2016-07-17 11:00:02 +00:00
|
|
|
describe_object(options, &tree->object), entry.path, entry.mode);
|
2008-02-25 21:46:04 +00:00
|
|
|
}
|
|
|
|
if (result < 0)
|
|
|
|
return result;
|
|
|
|
if (!res)
|
|
|
|
res = result;
|
|
|
|
}
|
|
|
|
return res;
|
|
|
|
}
|
|
|
|
|
2015-06-22 15:25:00 +00:00
|
|
|
static int fsck_walk_commit(struct commit *commit, void *data, struct fsck_options *options)
|
2008-02-25 21:46:04 +00:00
|
|
|
{
|
2016-07-17 10:59:49 +00:00
|
|
|
int counter = 0, generation = 0, name_prefix_len = 0;
|
2008-02-25 21:46:04 +00:00
|
|
|
struct commit_list *parents;
|
|
|
|
int res;
|
|
|
|
int result;
|
2016-07-17 10:59:49 +00:00
|
|
|
const char *name;
|
2008-02-25 21:46:04 +00:00
|
|
|
|
|
|
|
if (parse_commit(commit))
|
|
|
|
return -1;
|
|
|
|
|
2016-07-17 10:59:49 +00:00
|
|
|
name = get_object_name(options, &commit->object);
|
|
|
|
if (name)
|
2018-04-06 19:09:38 +00:00
|
|
|
put_object_name(options, &get_commit_tree(commit)->object,
|
|
|
|
"%s:", name);
|
2016-07-17 10:59:49 +00:00
|
|
|
|
2018-04-06 19:09:38 +00:00
|
|
|
result = options->walk((struct object *)get_commit_tree(commit),
|
|
|
|
OBJ_TREE, data, options);
|
2008-02-25 21:46:04 +00:00
|
|
|
if (result < 0)
|
|
|
|
return result;
|
|
|
|
res = result;
|
|
|
|
|
|
|
|
parents = commit->parents;
|
2016-07-17 10:59:49 +00:00
|
|
|
if (name && parents) {
|
|
|
|
int len = strlen(name), power;
|
|
|
|
|
|
|
|
if (len && name[len - 1] == '^') {
|
|
|
|
generation = 1;
|
|
|
|
name_prefix_len = len - 1;
|
|
|
|
}
|
|
|
|
else { /* parse ~<generation> suffix */
|
|
|
|
for (generation = 0, power = 1;
|
|
|
|
len && isdigit(name[len - 1]);
|
|
|
|
power *= 10)
|
|
|
|
generation += power * (name[--len] - '0');
|
|
|
|
if (power > 1 && len && name[len - 1] == '~')
|
|
|
|
name_prefix_len = len - 1;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2008-02-25 21:46:04 +00:00
|
|
|
while (parents) {
|
2016-07-17 10:59:49 +00:00
|
|
|
if (name) {
|
|
|
|
struct object *obj = &parents->item->object;
|
|
|
|
|
|
|
|
if (++counter > 1)
|
|
|
|
put_object_name(options, obj, "%s^%d",
|
|
|
|
name, counter);
|
|
|
|
else if (generation > 0)
|
|
|
|
put_object_name(options, obj, "%.*s~%d",
|
|
|
|
name_prefix_len, name, generation + 1);
|
|
|
|
else
|
|
|
|
put_object_name(options, obj, "%s^", name);
|
|
|
|
}
|
2015-06-22 15:25:00 +00:00
|
|
|
result = options->walk((struct object *)parents->item, OBJ_COMMIT, data, options);
|
2008-02-25 21:46:04 +00:00
|
|
|
if (result < 0)
|
|
|
|
return result;
|
|
|
|
if (!res)
|
|
|
|
res = result;
|
|
|
|
parents = parents->next;
|
|
|
|
}
|
|
|
|
return res;
|
|
|
|
}
|
|
|
|
|
2015-06-22 15:25:00 +00:00
|
|
|
static int fsck_walk_tag(struct tag *tag, void *data, struct fsck_options *options)
|
2008-02-25 21:46:04 +00:00
|
|
|
{
|
2016-07-17 10:59:49 +00:00
|
|
|
char *name = get_object_name(options, &tag->object);
|
|
|
|
|
2008-02-25 21:46:04 +00:00
|
|
|
if (parse_tag(tag))
|
|
|
|
return -1;
|
2016-07-17 10:59:49 +00:00
|
|
|
if (name)
|
|
|
|
put_object_name(options, tag->tagged, "%s", name);
|
2015-06-22 15:25:00 +00:00
|
|
|
return options->walk(tag->tagged, OBJ_ANY, data, options);
|
2008-02-25 21:46:04 +00:00
|
|
|
}
|
|
|
|
|
2015-06-22 15:25:00 +00:00
|
|
|
int fsck_walk(struct object *obj, void *data, struct fsck_options *options)
|
2008-02-25 21:46:04 +00:00
|
|
|
{
|
|
|
|
if (!obj)
|
|
|
|
return -1;
|
fsck: lazily load types under --connectivity-only
The recent fixes to "fsck --connectivity-only" load all of
the objects with their correct types. This keeps the
connectivity-only code path close to the regular one, but it
also introduces some unnecessary inefficiency. While getting
the type of an object is cheap compared to actually opening
and parsing the object (as the non-connectivity-only case
would do), it's still not free.
For reachable non-blob objects, we end up having to parse
them later anyway (to see what they point to), making our
type lookup here redundant.
For unreachable objects, we might never hit them at all in
the reachability traversal, making the lookup completely
wasted. And in some cases, we might have quite a few
unreachable objects (e.g., when alternates are used for
shared object storage between repositories, it's normal for
there to be objects reachable from other repositories but
not the one running fsck).
The comment in mark_object_for_connectivity() claims two
benefits to getting the type up front:
1. We need to know the types during fsck_walk(). (And not
explicitly mentioned, but we also need them when
printing the types of broken or dangling commits).
We can address this by lazy-loading the types as
necessary. Most objects never need this lazy-load at
all, because they fall into one of these categories:
a. Reachable from our tips, and are coerced into the
correct type as we traverse (e.g., a parent link
will call lookup_commit(), which converts OBJ_NONE
to OBJ_COMMIT).
b. Unreachable, but not at the tip of a chunk of
unreachable history. We only mention the tips as
"dangling", so an unreachable commit which links
to hundreds of other objects needs only report the
type of the tip commit.
2. It serves as a cross-check that the coercion in (1a) is
correct (i.e., we'll complain about a parent link that
points to a blob). But we get most of this for free
already, because right after coercing, we'll parse any
non-blob objects. So we'd notice then if we expected a
commit and got a blob.
The one exception is when we expect a blob, in which
case we never actually read the object contents.
So this is a slight weakening, but given that the whole
point of --connectivity-only is to sacrifice some data
integrity checks for speed, this seems like an
acceptable tradeoff.
Here are before and after timings for an extreme case with
~5M reachable objects and another ~12M unreachable (it's the
torvalds/linux repository on GitHub, connected to shared
storage for all of the other kernel forks):
[before]
$ time git fsck --no-dangling --connectivity-only
real 3m4.323s
user 1m25.121s
sys 1m38.710s
[after]
$ time git fsck --no-dangling --connectivity-only
real 0m51.497s
user 0m49.575s
sys 0m1.776s
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-01-26 04:12:07 +00:00
|
|
|
|
|
|
|
if (obj->type == OBJ_NONE)
|
2018-06-29 01:21:51 +00:00
|
|
|
parse_object(the_repository, &obj->oid);
|
fsck: lazily load types under --connectivity-only
The recent fixes to "fsck --connectivity-only" load all of
the objects with their correct types. This keeps the
connectivity-only code path close to the regular one, but it
also introduces some unnecessary inefficiency. While getting
the type of an object is cheap compared to actually opening
and parsing the object (as the non-connectivity-only case
would do), it's still not free.
For reachable non-blob objects, we end up having to parse
them later anyway (to see what they point to), making our
type lookup here redundant.
For unreachable objects, we might never hit them at all in
the reachability traversal, making the lookup completely
wasted. And in some cases, we might have quite a few
unreachable objects (e.g., when alternates are used for
shared object storage between repositories, it's normal for
there to be objects reachable from other repositories but
not the one running fsck).
The comment in mark_object_for_connectivity() claims two
benefits to getting the type up front:
1. We need to know the types during fsck_walk(). (And not
explicitly mentioned, but we also need them when
printing the types of broken or dangling commits).
We can address this by lazy-loading the types as
necessary. Most objects never need this lazy-load at
all, because they fall into one of these categories:
a. Reachable from our tips, and are coerced into the
correct type as we traverse (e.g., a parent link
will call lookup_commit(), which converts OBJ_NONE
to OBJ_COMMIT).
b. Unreachable, but not at the tip of a chunk of
unreachable history. We only mention the tips as
"dangling", so an unreachable commit which links
to hundreds of other objects needs only report the
type of the tip commit.
2. It serves as a cross-check that the coercion in (1a) is
correct (i.e., we'll complain about a parent link that
points to a blob). But we get most of this for free
already, because right after coercing, we'll parse any
non-blob objects. So we'd notice then if we expected a
commit and got a blob.
The one exception is when we expect a blob, in which
case we never actually read the object contents.
So this is a slight weakening, but given that the whole
point of --connectivity-only is to sacrifice some data
integrity checks for speed, this seems like an
acceptable tradeoff.
Here are before and after timings for an extreme case with
~5M reachable objects and another ~12M unreachable (it's the
torvalds/linux repository on GitHub, connected to shared
storage for all of the other kernel forks):
[before]
$ time git fsck --no-dangling --connectivity-only
real 3m4.323s
user 1m25.121s
sys 1m38.710s
[after]
$ time git fsck --no-dangling --connectivity-only
real 0m51.497s
user 0m49.575s
sys 0m1.776s
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-01-26 04:12:07 +00:00
|
|
|
|
2008-02-25 21:46:04 +00:00
|
|
|
switch (obj->type) {
|
|
|
|
case OBJ_BLOB:
|
|
|
|
return 0;
|
|
|
|
case OBJ_TREE:
|
2015-06-22 15:25:00 +00:00
|
|
|
return fsck_walk_tree((struct tree *)obj, data, options);
|
2008-02-25 21:46:04 +00:00
|
|
|
case OBJ_COMMIT:
|
2015-06-22 15:25:00 +00:00
|
|
|
return fsck_walk_commit((struct commit *)obj, data, options);
|
2008-02-25 21:46:04 +00:00
|
|
|
case OBJ_TAG:
|
2015-06-22 15:25:00 +00:00
|
|
|
return fsck_walk_tag((struct tag *)obj, data, options);
|
2008-02-25 21:46:04 +00:00
|
|
|
default:
|
2016-07-17 11:00:02 +00:00
|
|
|
error("Unknown object type for %s", describe_object(options, obj));
|
2008-02-25 21:46:04 +00:00
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
}
|
2008-02-25 21:46:08 +00:00
|
|
|
|
|
|
|
/*
|
|
|
|
* The entries in a tree are ordered in the _path_ order,
|
|
|
|
* which means that a directory entry is ordered by adding
|
|
|
|
* a slash to the end of it.
|
|
|
|
*
|
|
|
|
* So a directory called "a" is ordered _after_ a file
|
|
|
|
* called "a.c", because "a/" sorts after "a.c".
|
|
|
|
*/
|
|
|
|
#define TREE_UNORDERED (-1)
|
|
|
|
#define TREE_HAS_DUPS (-2)
|
|
|
|
|
|
|
|
static int verify_ordered(unsigned mode1, const char *name1, unsigned mode2, const char *name2)
|
|
|
|
{
|
|
|
|
int len1 = strlen(name1);
|
|
|
|
int len2 = strlen(name2);
|
|
|
|
int len = len1 < len2 ? len1 : len2;
|
|
|
|
unsigned char c1, c2;
|
|
|
|
int cmp;
|
|
|
|
|
|
|
|
cmp = memcmp(name1, name2, len);
|
|
|
|
if (cmp < 0)
|
|
|
|
return 0;
|
|
|
|
if (cmp > 0)
|
|
|
|
return TREE_UNORDERED;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Ok, the first <len> characters are the same.
|
|
|
|
* Now we need to order the next one, but turn
|
|
|
|
* a '\0' into a '/' for a directory entry.
|
|
|
|
*/
|
|
|
|
c1 = name1[len];
|
|
|
|
c2 = name2[len];
|
|
|
|
if (!c1 && !c2)
|
|
|
|
/*
|
|
|
|
* git-write-tree used to write out a nonsense tree that has
|
|
|
|
* entries with the same name, one blob and one tree. Make
|
|
|
|
* sure we do not have duplicate entries.
|
|
|
|
*/
|
|
|
|
return TREE_HAS_DUPS;
|
|
|
|
if (!c1 && S_ISDIR(mode1))
|
|
|
|
c1 = '/';
|
|
|
|
if (!c2 && S_ISDIR(mode2))
|
|
|
|
c2 = '/';
|
|
|
|
return c1 < c2 ? 0 : TREE_UNORDERED;
|
|
|
|
}
|
|
|
|
|
2015-06-22 15:25:00 +00:00
|
|
|
static int fsck_tree(struct tree *item, struct fsck_options *options)
|
2008-02-25 21:46:08 +00:00
|
|
|
{
|
2016-09-27 20:59:51 +00:00
|
|
|
int retval = 0;
|
2012-07-28 15:06:29 +00:00
|
|
|
int has_null_sha1 = 0;
|
2008-02-25 21:46:08 +00:00
|
|
|
int has_full_path = 0;
|
|
|
|
int has_empty_name = 0;
|
2012-11-28 02:27:37 +00:00
|
|
|
int has_dot = 0;
|
|
|
|
int has_dotdot = 0;
|
2012-11-28 21:35:29 +00:00
|
|
|
int has_dotgit = 0;
|
2008-02-25 21:46:08 +00:00
|
|
|
int has_zero_pad = 0;
|
|
|
|
int has_bad_modes = 0;
|
|
|
|
int has_dup_entries = 0;
|
|
|
|
int not_properly_sorted = 0;
|
|
|
|
struct tree_desc desc;
|
|
|
|
unsigned o_mode;
|
|
|
|
const char *o_name;
|
|
|
|
|
2016-09-27 20:59:51 +00:00
|
|
|
if (init_tree_desc_gently(&desc, item->buffer, item->size)) {
|
|
|
|
retval += report(options, &item->object, FSCK_MSG_BAD_TREE, "cannot be parsed as a tree");
|
|
|
|
return retval;
|
|
|
|
}
|
2008-02-25 21:46:08 +00:00
|
|
|
|
|
|
|
o_mode = 0;
|
|
|
|
o_name = NULL;
|
|
|
|
|
|
|
|
while (desc.size) {
|
|
|
|
unsigned mode;
|
|
|
|
const char *name;
|
2016-04-17 23:10:40 +00:00
|
|
|
const struct object_id *oid;
|
2008-02-25 21:46:08 +00:00
|
|
|
|
2016-04-17 23:10:40 +00:00
|
|
|
oid = tree_entry_extract(&desc, &name, &mode);
|
2008-02-25 21:46:08 +00:00
|
|
|
|
2016-04-17 23:10:40 +00:00
|
|
|
has_null_sha1 |= is_null_oid(oid);
|
2014-03-19 23:02:04 +00:00
|
|
|
has_full_path |= !!strchr(name, '/');
|
|
|
|
has_empty_name |= !*name;
|
|
|
|
has_dot |= !strcmp(name, ".");
|
|
|
|
has_dotdot |= !strcmp(name, "..");
|
2018-05-13 16:35:37 +00:00
|
|
|
has_dotgit |= is_hfs_dotgit(name) || is_ntfs_dotgit(name);
|
2008-02-25 21:46:08 +00:00
|
|
|
has_zero_pad |= *(char *)desc.buffer == '0';
|
fsck: detect gitmodules files
In preparation for performing fsck checks on .gitmodules
files, this commit plumbs in the actual detection of the
files. Note that unlike most other fsck checks, this cannot
be a property of a single object: we must know that the
object is found at a ".gitmodules" path at the root tree of
a commit.
Since the fsck code only sees one object at a time, we have
to mark the related objects to fit the puzzle together. When
we see a commit we mark its tree as a root tree, and when
we see a root tree with a .gitmodules file, we mark the
corresponding blob to be checked.
In an ideal world, we'd check the objects in topological
order: commits followed by trees followed by blobs. In that
case we can avoid ever loading an object twice, since all
markings would be complete by the time we get to the marked
objects. And indeed, if we are checking a single packfile,
this is the order in which Git will generally write the
objects. But we can't count on that:
1. git-fsck may show us the objects in arbitrary order
(loose objects are fed in sha1 order, but we may also
have multiple packs, and we process each pack fully in
sequence).
2. The type ordering is just what git-pack-objects happens
to write now. The pack format does not require a
specific order, and it's possible that future versions
of Git (or a custom version trying to fool official
Git's fsck checks!) may order it differently.
3. We may not even be fscking all of the relevant objects
at once. Consider pushing with transfer.fsckObjects,
where one push adds a blob at path "foo", and then a
second push adds the same blob at path ".gitmodules".
The blob is not part of the second push at all, but we
need to mark and check it.
So in the general case, we need to make up to three passes
over the objects: once to make sure we've seen all commits,
then once to cover any trees we might have missed, and then
a final pass to cover any .gitmodules blobs we found in the
second pass.
We can simplify things a bit by loosening the requirement
that we find .gitmodules only at root trees. Technically
a file like "subdir/.gitmodules" is not parsed by Git, but
it's not unreasonable for us to declare that Git is aware of
all ".gitmodules" files and make them eligible for checking.
That lets us drop the root-tree requirement, which
eliminates one pass entirely. And it makes our worst case
much better: instead of potentially queueing every root tree
to be re-examined, the worst case is that we queue each
unique .gitmodules blob for a second look.
This patch just adds the boilerplate to find .gitmodules
files. The actual content checks will come in a subsequent
commit.
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 21:20:08 +00:00
|
|
|
|
2018-05-05 00:03:35 +00:00
|
|
|
if (is_hfs_dotgitmodules(name) || is_ntfs_dotgitmodules(name)) {
|
|
|
|
if (!S_ISLNK(mode))
|
|
|
|
oidset_insert(&gitmodules_found, oid);
|
|
|
|
else
|
|
|
|
retval += report(options, &item->object,
|
|
|
|
FSCK_MSG_GITMODULES_SYMLINK,
|
|
|
|
".gitmodules is a symbolic link");
|
|
|
|
}
|
fsck: detect gitmodules files
In preparation for performing fsck checks on .gitmodules
files, this commit plumbs in the actual detection of the
files. Note that unlike most other fsck checks, this cannot
be a property of a single object: we must know that the
object is found at a ".gitmodules" path at the root tree of
a commit.
Since the fsck code only sees one object at a time, we have
to mark the related objects to fit the puzzle together. When
we see a commit we mark its tree as a root tree, and when
we see a root tree with a .gitmodules file, we mark the
corresponding blob to be checked.
In an ideal world, we'd check the objects in topological
order: commits followed by trees followed by blobs. In that
case we can avoid ever loading an object twice, since all
markings would be complete by the time we get to the marked
objects. And indeed, if we are checking a single packfile,
this is the order in which Git will generally write the
objects. But we can't count on that:
1. git-fsck may show us the objects in arbitrary order
(loose objects are fed in sha1 order, but we may also
have multiple packs, and we process each pack fully in
sequence).
2. The type ordering is just what git-pack-objects happens
to write now. The pack format does not require a
specific order, and it's possible that future versions
of Git (or a custom version trying to fool official
Git's fsck checks!) may order it differently.
3. We may not even be fscking all of the relevant objects
at once. Consider pushing with transfer.fsckObjects,
where one push adds a blob at path "foo", and then a
second push adds the same blob at path ".gitmodules".
The blob is not part of the second push at all, but we
need to mark and check it.
So in the general case, we need to make up to three passes
over the objects: once to make sure we've seen all commits,
then once to cover any trees we might have missed, and then
a final pass to cover any .gitmodules blobs we found in the
second pass.
We can simplify things a bit by loosening the requirement
that we find .gitmodules only at root trees. Technically
a file like "subdir/.gitmodules" is not parsed by Git, but
it's not unreasonable for us to declare that Git is aware of
all ".gitmodules" files and make them eligible for checking.
That lets us drop the root-tree requirement, which
eliminates one pass entirely. And it makes our worst case
much better: instead of potentially queueing every root tree
to be re-examined, the worst case is that we queue each
unique .gitmodules blob for a second look.
This patch just adds the boilerplate to find .gitmodules
files. The actual content checks will come in a subsequent
commit.
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 21:20:08 +00:00
|
|
|
|
2016-09-27 20:59:51 +00:00
|
|
|
if (update_tree_entry_gently(&desc)) {
|
|
|
|
retval += report(options, &item->object, FSCK_MSG_BAD_TREE, "cannot be parsed as a tree");
|
|
|
|
break;
|
|
|
|
}
|
2008-02-25 21:46:08 +00:00
|
|
|
|
|
|
|
switch (mode) {
|
|
|
|
/*
|
|
|
|
* Standard modes..
|
|
|
|
*/
|
|
|
|
case S_IFREG | 0755:
|
|
|
|
case S_IFREG | 0644:
|
|
|
|
case S_IFLNK:
|
|
|
|
case S_IFDIR:
|
|
|
|
case S_IFGITLINK:
|
|
|
|
break;
|
|
|
|
/*
|
|
|
|
* This is nonstandard, but we had a few of these
|
|
|
|
* early on when we honored the full set of mode
|
|
|
|
* bits..
|
|
|
|
*/
|
|
|
|
case S_IFREG | 0664:
|
2015-06-22 15:25:00 +00:00
|
|
|
if (!options->strict)
|
2008-02-25 21:46:08 +00:00
|
|
|
break;
|
consistently use "fallthrough" comments in switches
Gcc 7 adds -Wimplicit-fallthrough, which can warn when a
switch case falls through to the next case. The general idea
is that the compiler can't tell if this was intentional or
not, so you should annotate any intentional fall-throughs as
such, leaving it to complain about any unannotated ones.
There's a GNU __attribute__ which can be used for
annotation, but of course we'd have to #ifdef it away on
non-gcc compilers. Gcc will also recognize
specially-formatted comments, which matches our current
practice. Let's extend that practice to all of the
unannotated sites (which I did look over and verify that
they were behaving as intended).
Ideally in each case we'd actually give some reasons in the
comment about why we're falling through, or what we're
falling through to. And gcc does support that with
-Wimplicit-fallthrough=2, which relaxes the comment pattern
matching to anything that contains "fallthrough" (or a
variety of spelling variants). However, this isn't the
default for -Wimplicit-fallthrough, nor for -Wextra. In the
name of simplicity, it's probably better for us to support
the default level, which requires "fallthrough" to be the
only thing in the comment (modulo some window dressing like
"else" and some punctuation; see the gcc manual for the
complete set of patterns).
This patch suppresses all warnings due to
-Wimplicit-fallthrough. We might eventually want to add that
to the DEVELOPER Makefile knob, but we should probably wait
until gcc 7 is more widely adopted (since earlier versions
will complain about the unknown warning type).
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2017-09-21 06:25:41 +00:00
|
|
|
/* fallthrough */
|
2008-02-25 21:46:08 +00:00
|
|
|
default:
|
|
|
|
has_bad_modes = 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (o_name) {
|
|
|
|
switch (verify_ordered(o_mode, o_name, mode, name)) {
|
|
|
|
case TREE_UNORDERED:
|
|
|
|
not_properly_sorted = 1;
|
|
|
|
break;
|
|
|
|
case TREE_HAS_DUPS:
|
|
|
|
has_dup_entries = 1;
|
|
|
|
break;
|
|
|
|
default:
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
o_mode = mode;
|
|
|
|
o_name = name;
|
|
|
|
}
|
|
|
|
|
2012-07-28 15:06:29 +00:00
|
|
|
if (has_null_sha1)
|
2015-06-22 15:25:09 +00:00
|
|
|
retval += report(options, &item->object, FSCK_MSG_NULL_SHA1, "contains entries pointing to null sha1");
|
2008-02-25 21:46:08 +00:00
|
|
|
if (has_full_path)
|
2015-06-22 15:25:09 +00:00
|
|
|
retval += report(options, &item->object, FSCK_MSG_FULL_PATHNAME, "contains full pathnames");
|
2008-02-25 21:46:08 +00:00
|
|
|
if (has_empty_name)
|
2015-06-22 15:25:09 +00:00
|
|
|
retval += report(options, &item->object, FSCK_MSG_EMPTY_NAME, "contains empty pathname");
|
2012-11-28 02:27:37 +00:00
|
|
|
if (has_dot)
|
2015-06-22 15:25:09 +00:00
|
|
|
retval += report(options, &item->object, FSCK_MSG_HAS_DOT, "contains '.'");
|
2012-11-28 02:27:37 +00:00
|
|
|
if (has_dotdot)
|
2015-06-22 15:25:09 +00:00
|
|
|
retval += report(options, &item->object, FSCK_MSG_HAS_DOTDOT, "contains '..'");
|
2012-11-28 21:35:29 +00:00
|
|
|
if (has_dotgit)
|
2015-06-22 15:25:09 +00:00
|
|
|
retval += report(options, &item->object, FSCK_MSG_HAS_DOTGIT, "contains '.git'");
|
2008-02-25 21:46:08 +00:00
|
|
|
if (has_zero_pad)
|
2015-06-22 15:25:09 +00:00
|
|
|
retval += report(options, &item->object, FSCK_MSG_ZERO_PADDED_FILEMODE, "contains zero-padded file modes");
|
2008-02-25 21:46:08 +00:00
|
|
|
if (has_bad_modes)
|
2015-06-22 15:25:09 +00:00
|
|
|
retval += report(options, &item->object, FSCK_MSG_BAD_FILEMODE, "contains bad file modes");
|
2008-02-25 21:46:08 +00:00
|
|
|
if (has_dup_entries)
|
2015-06-22 15:25:09 +00:00
|
|
|
retval += report(options, &item->object, FSCK_MSG_DUPLICATE_ENTRIES, "contains duplicate file entries");
|
2008-02-25 21:46:08 +00:00
|
|
|
if (not_properly_sorted)
|
2015-06-22 15:25:09 +00:00
|
|
|
retval += report(options, &item->object, FSCK_MSG_TREE_NOT_SORTED, "not properly sorted");
|
2008-02-25 21:46:08 +00:00
|
|
|
return retval;
|
|
|
|
}
|
|
|
|
|
2015-06-28 18:18:31 +00:00
|
|
|
static int verify_headers(const void *data, unsigned long size,
|
2015-08-03 18:01:18 +00:00
|
|
|
struct object *obj, struct fsck_options *options)
|
2014-09-11 14:26:33 +00:00
|
|
|
{
|
|
|
|
const char *buffer = (const char *)data;
|
|
|
|
unsigned long i;
|
|
|
|
|
|
|
|
for (i = 0; i < size; i++) {
|
|
|
|
switch (buffer[i]) {
|
|
|
|
case '\0':
|
2015-06-22 15:25:09 +00:00
|
|
|
return report(options, obj,
|
|
|
|
FSCK_MSG_NUL_IN_HEADER,
|
|
|
|
"unterminated header: NUL at offset %ld", i);
|
2014-09-11 14:26:33 +00:00
|
|
|
case '\n':
|
|
|
|
if (i + 1 < size && buffer[i + 1] == '\n')
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2015-06-28 18:18:31 +00:00
|
|
|
/*
|
|
|
|
* We did not find double-LF that separates the header
|
|
|
|
* and the body. Not having a body is not a crime but
|
|
|
|
* we do want to see the terminating LF for the last header
|
|
|
|
* line.
|
|
|
|
*/
|
|
|
|
if (size && buffer[size - 1] == '\n')
|
|
|
|
return 0;
|
|
|
|
|
2015-06-22 15:25:09 +00:00
|
|
|
return report(options, obj,
|
|
|
|
FSCK_MSG_UNTERMINATED_HEADER, "unterminated header");
|
2014-09-11 14:26:33 +00:00
|
|
|
}
|
|
|
|
|
2015-06-22 15:25:00 +00:00
|
|
|
static int fsck_ident(const char **ident, struct object *obj, struct fsck_options *options)
|
2010-04-24 16:06:08 +00:00
|
|
|
{
|
2015-06-22 15:26:03 +00:00
|
|
|
const char *p = *ident;
|
fsck: report integer overflow in author timestamps
When we check commit objects, we complain if commit->date is
ULONG_MAX, which is an indication that we saw integer
overflow when parsing it. However, we do not do any check at
all for author lines, which also contain a timestamp.
Let's actually check the timestamps on each ident line
with strtoul. This catches both author and committer lines,
and we can get rid of the now-redundant commit->date check.
Note that like the existing check, we compare only against
ULONG_MAX. Now that we are calling strtoul at the site of
the check, we could be slightly more careful and also check
that errno is set to ERANGE. However, this will make further
refactoring in future patches a little harder, and it
doesn't really matter in practice.
For 32-bit systems, one would have to create a commit at the
exact wrong second in 2038. But by the time we get close to
that, all systems will hopefully have moved to 64-bit (and
if they haven't, they have a real problem one second later).
For 64-bit systems, by the time we get close to ULONG_MAX,
all systems will hopefully have been consumed in the fiery
wrath of our expanding Sun.
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-02-24 07:39:04 +00:00
|
|
|
char *end;
|
|
|
|
|
2015-06-22 15:26:03 +00:00
|
|
|
*ident = strchrnul(*ident, '\n');
|
|
|
|
if (**ident == '\n')
|
|
|
|
(*ident)++;
|
|
|
|
|
|
|
|
if (*p == '<')
|
2015-06-22 15:25:09 +00:00
|
|
|
return report(options, obj, FSCK_MSG_MISSING_NAME_BEFORE_EMAIL, "invalid author/committer line - missing space before email");
|
2015-06-22 15:26:03 +00:00
|
|
|
p += strcspn(p, "<>\n");
|
|
|
|
if (*p == '>')
|
2015-06-22 15:25:09 +00:00
|
|
|
return report(options, obj, FSCK_MSG_BAD_NAME, "invalid author/committer line - bad name");
|
2015-06-22 15:26:03 +00:00
|
|
|
if (*p != '<')
|
2015-06-22 15:25:09 +00:00
|
|
|
return report(options, obj, FSCK_MSG_MISSING_EMAIL, "invalid author/committer line - missing email");
|
2015-06-22 15:26:03 +00:00
|
|
|
if (p[-1] != ' ')
|
2015-06-22 15:25:09 +00:00
|
|
|
return report(options, obj, FSCK_MSG_MISSING_SPACE_BEFORE_EMAIL, "invalid author/committer line - missing space before email");
|
2015-06-22 15:26:03 +00:00
|
|
|
p++;
|
|
|
|
p += strcspn(p, "<>\n");
|
|
|
|
if (*p != '>')
|
2015-06-22 15:25:09 +00:00
|
|
|
return report(options, obj, FSCK_MSG_BAD_EMAIL, "invalid author/committer line - bad email");
|
2015-06-22 15:26:03 +00:00
|
|
|
p++;
|
|
|
|
if (*p != ' ')
|
2015-06-22 15:25:09 +00:00
|
|
|
return report(options, obj, FSCK_MSG_MISSING_SPACE_BEFORE_DATE, "invalid author/committer line - missing space before date");
|
2015-06-22 15:26:03 +00:00
|
|
|
p++;
|
|
|
|
if (*p == '0' && p[1] != ' ')
|
2015-06-22 15:25:09 +00:00
|
|
|
return report(options, obj, FSCK_MSG_ZERO_PADDED_DATE, "invalid author/committer line - zero-padded date");
|
2017-04-21 10:45:44 +00:00
|
|
|
if (date_overflows(parse_timestamp(p, &end, 10)))
|
2015-06-22 15:25:09 +00:00
|
|
|
return report(options, obj, FSCK_MSG_BAD_DATE_OVERFLOW, "invalid author/committer line - date causes integer overflow");
|
2015-06-22 15:26:03 +00:00
|
|
|
if ((end == p || *end != ' '))
|
2015-06-22 15:25:09 +00:00
|
|
|
return report(options, obj, FSCK_MSG_BAD_DATE, "invalid author/committer line - bad date");
|
2015-06-22 15:26:03 +00:00
|
|
|
p = end + 1;
|
|
|
|
if ((*p != '+' && *p != '-') ||
|
|
|
|
!isdigit(p[1]) ||
|
|
|
|
!isdigit(p[2]) ||
|
|
|
|
!isdigit(p[3]) ||
|
|
|
|
!isdigit(p[4]) ||
|
|
|
|
(p[5] != '\n'))
|
2015-06-22 15:25:09 +00:00
|
|
|
return report(options, obj, FSCK_MSG_BAD_TIMEZONE, "invalid author/committer line - bad time zone");
|
2015-06-22 15:26:03 +00:00
|
|
|
p += 6;
|
2010-04-24 16:06:08 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2014-06-10 21:41:51 +00:00
|
|
|
static int fsck_commit_buffer(struct commit *commit, const char *buffer,
|
2015-06-22 15:25:00 +00:00
|
|
|
unsigned long size, struct fsck_options *options)
|
2008-02-25 21:46:08 +00:00
|
|
|
{
|
2018-05-02 00:25:41 +00:00
|
|
|
struct object_id tree_oid, oid;
|
2008-02-25 21:46:08 +00:00
|
|
|
struct commit_graft *graft;
|
2015-06-22 15:26:23 +00:00
|
|
|
unsigned parent_count, parent_line_count = 0, author_count;
|
2010-04-24 16:06:08 +00:00
|
|
|
int err;
|
2016-04-14 17:58:22 +00:00
|
|
|
const char *buffer_begin = buffer;
|
2018-05-02 00:25:41 +00:00
|
|
|
const char *p;
|
2008-02-25 21:46:08 +00:00
|
|
|
|
2015-08-03 18:01:18 +00:00
|
|
|
if (verify_headers(buffer, size, &commit->object, options))
|
2014-09-11 14:26:33 +00:00
|
|
|
return -1;
|
|
|
|
|
refactor skip_prefix to return a boolean
The skip_prefix() function returns a pointer to the content
past the prefix, or NULL if the prefix was not found. While
this is nice and simple, in practice it makes it hard to use
for two reasons:
1. When you want to conditionally skip or keep the string
as-is, you have to introduce a temporary variable.
For example:
tmp = skip_prefix(buf, "foo");
if (tmp)
buf = tmp;
2. It is verbose to check the outcome in a conditional, as
you need extra parentheses to silence compiler
warnings. For example:
if ((cp = skip_prefix(buf, "foo"))
/* do something with cp */
Both of these make it harder to use for long if-chains, and
we tend to use starts_with() instead. However, the first line
of "do something" is often to then skip forward in buf past
the prefix, either using a magic constant or with an extra
strlen(3) (which is generally computed at compile time, but
means we are repeating ourselves).
This patch refactors skip_prefix() to return a simple boolean,
and to provide the pointer value as an out-parameter. If the
prefix is not found, the out-parameter is untouched. This
lets you write:
if (skip_prefix(arg, "foo ", &arg))
do_foo(arg);
else if (skip_prefix(arg, "bar ", &arg))
do_bar(arg);
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-06-18 19:44:19 +00:00
|
|
|
if (!skip_prefix(buffer, "tree ", &buffer))
|
2015-06-22 15:25:09 +00:00
|
|
|
return report(options, &commit->object, FSCK_MSG_MISSING_TREE, "invalid format - expected 'tree' line");
|
2018-05-02 00:25:41 +00:00
|
|
|
if (parse_oid_hex(buffer, &tree_oid, &p) || *p != '\n') {
|
2015-06-22 15:26:11 +00:00
|
|
|
err = report(options, &commit->object, FSCK_MSG_BAD_TREE_SHA1, "invalid 'tree' line format - bad sha1");
|
|
|
|
if (err)
|
|
|
|
return err;
|
|
|
|
}
|
2018-05-02 00:25:41 +00:00
|
|
|
buffer = p + 1;
|
refactor skip_prefix to return a boolean
The skip_prefix() function returns a pointer to the content
past the prefix, or NULL if the prefix was not found. While
this is nice and simple, in practice it makes it hard to use
for two reasons:
1. When you want to conditionally skip or keep the string
as-is, you have to introduce a temporary variable.
For example:
tmp = skip_prefix(buf, "foo");
if (tmp)
buf = tmp;
2. It is verbose to check the outcome in a conditional, as
you need extra parentheses to silence compiler
warnings. For example:
if ((cp = skip_prefix(buf, "foo"))
/* do something with cp */
Both of these make it harder to use for long if-chains, and
we tend to use starts_with() instead. However, the first line
of "do something" is often to then skip forward in buf past
the prefix, either using a magic constant or with an extra
strlen(3) (which is generally computed at compile time, but
means we are repeating ourselves).
This patch refactors skip_prefix() to return a simple boolean,
and to provide the pointer value as an out-parameter. If the
prefix is not found, the out-parameter is untouched. This
lets you write:
if (skip_prefix(arg, "foo ", &arg))
do_foo(arg);
else if (skip_prefix(arg, "bar ", &arg))
do_bar(arg);
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-06-18 19:44:19 +00:00
|
|
|
while (skip_prefix(buffer, "parent ", &buffer)) {
|
2018-05-02 00:25:41 +00:00
|
|
|
if (parse_oid_hex(buffer, &oid, &p) || *p != '\n') {
|
2015-06-22 15:26:11 +00:00
|
|
|
err = report(options, &commit->object, FSCK_MSG_BAD_PARENT_SHA1, "invalid 'parent' line format - bad sha1");
|
|
|
|
if (err)
|
|
|
|
return err;
|
|
|
|
}
|
2018-05-02 00:25:41 +00:00
|
|
|
buffer = p + 1;
|
2014-07-10 09:48:26 +00:00
|
|
|
parent_line_count++;
|
2008-02-25 21:46:08 +00:00
|
|
|
}
|
2018-05-17 22:51:42 +00:00
|
|
|
graft = lookup_commit_graft(the_repository, &commit->object.oid);
|
2014-07-10 09:48:26 +00:00
|
|
|
parent_count = commit_list_count(commit->parents);
|
2008-02-25 21:46:08 +00:00
|
|
|
if (graft) {
|
2014-07-10 09:48:26 +00:00
|
|
|
if (graft->nr_parent == -1 && !parent_count)
|
2008-02-25 21:46:08 +00:00
|
|
|
; /* shallow commit */
|
2015-06-22 15:26:11 +00:00
|
|
|
else if (graft->nr_parent != parent_count) {
|
|
|
|
err = report(options, &commit->object, FSCK_MSG_MISSING_GRAFT, "graft objects missing");
|
|
|
|
if (err)
|
|
|
|
return err;
|
|
|
|
}
|
2008-02-25 21:46:08 +00:00
|
|
|
} else {
|
2015-06-22 15:26:11 +00:00
|
|
|
if (parent_count != parent_line_count) {
|
|
|
|
err = report(options, &commit->object, FSCK_MSG_MISSING_PARENT, "parent objects missing");
|
|
|
|
if (err)
|
|
|
|
return err;
|
|
|
|
}
|
2008-02-25 21:46:08 +00:00
|
|
|
}
|
2015-06-22 15:26:23 +00:00
|
|
|
author_count = 0;
|
|
|
|
while (skip_prefix(buffer, "author ", &buffer)) {
|
|
|
|
author_count++;
|
|
|
|
err = fsck_ident(&buffer, &commit->object, options);
|
|
|
|
if (err)
|
|
|
|
return err;
|
2008-02-25 21:46:08 +00:00
|
|
|
}
|
2015-06-22 15:26:23 +00:00
|
|
|
if (author_count < 1)
|
|
|
|
err = report(options, &commit->object, FSCK_MSG_MISSING_AUTHOR, "invalid format - expected 'author' line");
|
|
|
|
else if (author_count > 1)
|
|
|
|
err = report(options, &commit->object, FSCK_MSG_MULTIPLE_AUTHORS, "invalid format - multiple 'author' lines");
|
2010-04-24 16:06:08 +00:00
|
|
|
if (err)
|
|
|
|
return err;
|
refactor skip_prefix to return a boolean
The skip_prefix() function returns a pointer to the content
past the prefix, or NULL if the prefix was not found. While
this is nice and simple, in practice it makes it hard to use
for two reasons:
1. When you want to conditionally skip or keep the string
as-is, you have to introduce a temporary variable.
For example:
tmp = skip_prefix(buf, "foo");
if (tmp)
buf = tmp;
2. It is verbose to check the outcome in a conditional, as
you need extra parentheses to silence compiler
warnings. For example:
if ((cp = skip_prefix(buf, "foo"))
/* do something with cp */
Both of these make it harder to use for long if-chains, and
we tend to use starts_with() instead. However, the first line
of "do something" is often to then skip forward in buf past
the prefix, either using a magic constant or with an extra
strlen(3) (which is generally computed at compile time, but
means we are repeating ourselves).
This patch refactors skip_prefix() to return a simple boolean,
and to provide the pointer value as an out-parameter. If the
prefix is not found, the out-parameter is untouched. This
lets you write:
if (skip_prefix(arg, "foo ", &arg))
do_foo(arg);
else if (skip_prefix(arg, "bar ", &arg))
do_bar(arg);
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2014-06-18 19:44:19 +00:00
|
|
|
if (!skip_prefix(buffer, "committer ", &buffer))
|
2015-06-22 15:25:09 +00:00
|
|
|
return report(options, &commit->object, FSCK_MSG_MISSING_COMMITTER, "invalid format - expected 'committer' line");
|
2015-06-22 15:25:00 +00:00
|
|
|
err = fsck_ident(&buffer, &commit->object, options);
|
2010-04-24 16:06:08 +00:00
|
|
|
if (err)
|
|
|
|
return err;
|
2018-04-06 19:09:38 +00:00
|
|
|
if (!get_commit_tree(commit)) {
|
2018-05-02 00:25:41 +00:00
|
|
|
err = report(options, &commit->object, FSCK_MSG_BAD_TREE, "could not load commit's tree %s", oid_to_hex(&tree_oid));
|
2016-04-14 17:18:11 +00:00
|
|
|
if (err)
|
|
|
|
return err;
|
|
|
|
}
|
2016-04-14 17:58:22 +00:00
|
|
|
if (memchr(buffer_begin, '\0', size)) {
|
|
|
|
err = report(options, &commit->object, FSCK_MSG_NUL_IN_COMMIT,
|
|
|
|
"NUL byte in the commit object body");
|
|
|
|
if (err)
|
|
|
|
return err;
|
|
|
|
}
|
2008-02-25 21:46:08 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2014-09-10 13:52:51 +00:00
|
|
|
static int fsck_commit(struct commit *commit, const char *data,
|
2015-06-22 15:25:00 +00:00
|
|
|
unsigned long size, struct fsck_options *options)
|
2014-06-10 21:41:51 +00:00
|
|
|
{
|
2014-09-10 13:52:51 +00:00
|
|
|
const char *buffer = data ? data : get_commit_buffer(commit, &size);
|
2015-06-22 15:25:00 +00:00
|
|
|
int ret = fsck_commit_buffer(commit, buffer, size, options);
|
2014-09-10 13:52:51 +00:00
|
|
|
if (!data)
|
|
|
|
unuse_commit_buffer(commit, buffer);
|
2014-06-10 21:41:51 +00:00
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2014-09-11 14:26:38 +00:00
|
|
|
static int fsck_tag_buffer(struct tag *tag, const char *data,
|
2015-06-22 15:25:00 +00:00
|
|
|
unsigned long size, struct fsck_options *options)
|
2014-09-11 14:26:38 +00:00
|
|
|
{
|
2018-05-02 00:25:41 +00:00
|
|
|
struct object_id oid;
|
2014-09-11 14:26:38 +00:00
|
|
|
int ret = 0;
|
|
|
|
const char *buffer;
|
|
|
|
char *to_free = NULL, *eol;
|
|
|
|
struct strbuf sb = STRBUF_INIT;
|
2018-05-02 00:25:41 +00:00
|
|
|
const char *p;
|
2014-09-11 14:26:38 +00:00
|
|
|
|
|
|
|
if (data)
|
|
|
|
buffer = data;
|
|
|
|
else {
|
|
|
|
enum object_type type;
|
|
|
|
|
|
|
|
buffer = to_free =
|
sha1_file: convert read_sha1_file to struct object_id
Convert read_sha1_file to take a pointer to struct object_id and rename
it read_object_file. Do the same for read_sha1_file_extended.
Convert one use in grep.c to use the new function without any other code
change, since the pointer being passed is a void pointer that is already
initialized with a pointer to struct object_id. Update the declaration
and definitions of the modified functions, and apply the following
semantic patch to convert the remaining callers:
@@
expression E1, E2, E3;
@@
- read_sha1_file(E1.hash, E2, E3)
+ read_object_file(&E1, E2, E3)
@@
expression E1, E2, E3;
@@
- read_sha1_file(E1->hash, E2, E3)
+ read_object_file(E1, E2, E3)
@@
expression E1, E2, E3, E4;
@@
- read_sha1_file_extended(E1.hash, E2, E3, E4)
+ read_object_file_extended(&E1, E2, E3, E4)
@@
expression E1, E2, E3, E4;
@@
- read_sha1_file_extended(E1->hash, E2, E3, E4)
+ read_object_file_extended(E1, E2, E3, E4)
Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-03-12 02:27:53 +00:00
|
|
|
read_object_file(&tag->object.oid, &type, &size);
|
2014-09-11 14:26:38 +00:00
|
|
|
if (!buffer)
|
2015-06-22 15:25:09 +00:00
|
|
|
return report(options, &tag->object,
|
|
|
|
FSCK_MSG_MISSING_TAG_OBJECT,
|
2014-09-11 14:26:38 +00:00
|
|
|
"cannot read tag object");
|
|
|
|
|
|
|
|
if (type != OBJ_TAG) {
|
2015-06-22 15:25:09 +00:00
|
|
|
ret = report(options, &tag->object,
|
|
|
|
FSCK_MSG_TAG_OBJECT_NOT_TAG,
|
2014-09-11 14:26:38 +00:00
|
|
|
"expected tag got %s",
|
2018-02-14 18:59:24 +00:00
|
|
|
type_name(type));
|
2014-09-11 14:26:38 +00:00
|
|
|
goto done;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2015-11-19 16:25:31 +00:00
|
|
|
ret = verify_headers(buffer, size, &tag->object, options);
|
|
|
|
if (ret)
|
2014-09-11 14:26:38 +00:00
|
|
|
goto done;
|
|
|
|
|
|
|
|
if (!skip_prefix(buffer, "object ", &buffer)) {
|
2015-06-22 15:25:09 +00:00
|
|
|
ret = report(options, &tag->object, FSCK_MSG_MISSING_OBJECT, "invalid format - expected 'object' line");
|
2014-09-11 14:26:38 +00:00
|
|
|
goto done;
|
|
|
|
}
|
2018-05-02 00:25:41 +00:00
|
|
|
if (parse_oid_hex(buffer, &oid, &p) || *p != '\n') {
|
2015-06-22 15:25:09 +00:00
|
|
|
ret = report(options, &tag->object, FSCK_MSG_BAD_OBJECT_SHA1, "invalid 'object' line format - bad sha1");
|
2015-06-22 15:26:30 +00:00
|
|
|
if (ret)
|
|
|
|
goto done;
|
2014-09-11 14:26:38 +00:00
|
|
|
}
|
2018-05-02 00:25:41 +00:00
|
|
|
buffer = p + 1;
|
2014-09-11 14:26:38 +00:00
|
|
|
|
|
|
|
if (!skip_prefix(buffer, "type ", &buffer)) {
|
2015-06-22 15:25:09 +00:00
|
|
|
ret = report(options, &tag->object, FSCK_MSG_MISSING_TYPE_ENTRY, "invalid format - expected 'type' line");
|
2014-09-11 14:26:38 +00:00
|
|
|
goto done;
|
|
|
|
}
|
|
|
|
eol = strchr(buffer, '\n');
|
|
|
|
if (!eol) {
|
2015-06-22 15:25:09 +00:00
|
|
|
ret = report(options, &tag->object, FSCK_MSG_MISSING_TYPE, "invalid format - unexpected end after 'type' line");
|
2014-09-11 14:26:38 +00:00
|
|
|
goto done;
|
|
|
|
}
|
|
|
|
if (type_from_string_gently(buffer, eol - buffer, 1) < 0)
|
2015-06-22 15:25:09 +00:00
|
|
|
ret = report(options, &tag->object, FSCK_MSG_BAD_TYPE, "invalid 'type' value");
|
2014-09-11 14:26:38 +00:00
|
|
|
if (ret)
|
|
|
|
goto done;
|
|
|
|
buffer = eol + 1;
|
|
|
|
|
|
|
|
if (!skip_prefix(buffer, "tag ", &buffer)) {
|
2015-06-22 15:25:09 +00:00
|
|
|
ret = report(options, &tag->object, FSCK_MSG_MISSING_TAG_ENTRY, "invalid format - expected 'tag' line");
|
2014-09-11 14:26:38 +00:00
|
|
|
goto done;
|
|
|
|
}
|
|
|
|
eol = strchr(buffer, '\n');
|
|
|
|
if (!eol) {
|
2015-06-22 15:25:09 +00:00
|
|
|
ret = report(options, &tag->object, FSCK_MSG_MISSING_TAG, "invalid format - unexpected end after 'type' line");
|
2014-09-11 14:26:38 +00:00
|
|
|
goto done;
|
|
|
|
}
|
|
|
|
strbuf_addf(&sb, "refs/tags/%.*s", (int)(eol - buffer), buffer);
|
2015-06-22 15:26:54 +00:00
|
|
|
if (check_refname_format(sb.buf, 0)) {
|
|
|
|
ret = report(options, &tag->object, FSCK_MSG_BAD_TAG_NAME,
|
2015-06-22 15:25:09 +00:00
|
|
|
"invalid 'tag' name: %.*s",
|
2014-12-08 05:48:13 +00:00
|
|
|
(int)(eol - buffer), buffer);
|
2015-06-22 15:26:54 +00:00
|
|
|
if (ret)
|
|
|
|
goto done;
|
|
|
|
}
|
2014-09-11 14:26:38 +00:00
|
|
|
buffer = eol + 1;
|
|
|
|
|
2015-06-22 15:26:54 +00:00
|
|
|
if (!skip_prefix(buffer, "tagger ", &buffer)) {
|
2014-09-11 14:26:38 +00:00
|
|
|
/* early tags do not contain 'tagger' lines; warn only */
|
2015-06-22 15:26:54 +00:00
|
|
|
ret = report(options, &tag->object, FSCK_MSG_MISSING_TAGGER_ENTRY, "invalid format - expected 'tagger' line");
|
|
|
|
if (ret)
|
|
|
|
goto done;
|
|
|
|
}
|
2014-09-11 14:26:38 +00:00
|
|
|
else
|
2015-06-22 15:25:00 +00:00
|
|
|
ret = fsck_ident(&buffer, &tag->object, options);
|
2014-09-11 14:26:38 +00:00
|
|
|
|
|
|
|
done:
|
|
|
|
strbuf_release(&sb);
|
|
|
|
free(to_free);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
2014-09-10 13:52:51 +00:00
|
|
|
static int fsck_tag(struct tag *tag, const char *data,
|
2015-06-22 15:25:00 +00:00
|
|
|
unsigned long size, struct fsck_options *options)
|
2008-02-25 21:46:08 +00:00
|
|
|
{
|
|
|
|
struct object *tagged = tag->tagged;
|
|
|
|
|
|
|
|
if (!tagged)
|
2015-06-22 15:25:09 +00:00
|
|
|
return report(options, &tag->object, FSCK_MSG_BAD_TAG_OBJECT, "could not load tagged object");
|
2014-09-11 14:26:38 +00:00
|
|
|
|
2015-06-22 15:25:00 +00:00
|
|
|
return fsck_tag_buffer(tag, data, size, options);
|
2008-02-25 21:46:08 +00:00
|
|
|
}
|
|
|
|
|
2018-05-02 21:25:27 +00:00
|
|
|
struct fsck_gitmodules_data {
|
|
|
|
struct object *obj;
|
|
|
|
struct fsck_options *options;
|
|
|
|
int ret;
|
|
|
|
};
|
|
|
|
|
|
|
|
static int fsck_gitmodules_fn(const char *var, const char *value, void *vdata)
|
|
|
|
{
|
|
|
|
struct fsck_gitmodules_data *data = vdata;
|
|
|
|
const char *subsection, *key;
|
|
|
|
int subsection_len;
|
|
|
|
char *name;
|
|
|
|
|
|
|
|
if (parse_config_key(var, "submodule", &subsection, &subsection_len, &key) < 0 ||
|
|
|
|
!subsection)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
name = xmemdupz(subsection, subsection_len);
|
|
|
|
if (check_submodule_name(name) < 0)
|
|
|
|
data->ret |= report(data->options, data->obj,
|
|
|
|
FSCK_MSG_GITMODULES_NAME,
|
|
|
|
"disallowed submodule name: %s",
|
|
|
|
name);
|
|
|
|
free(name);
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
fsck: actually fsck blob data
Because fscking a blob has always been a noop, we didn't
bother passing around the blob data. In preparation for
content-level checks, let's fix up a few things:
1. The fsck_object() function just returns success for any
blob. Let's a noop fsck_blob(), which we can fill in
with actual logic later.
2. The fsck_loose() function in builtin/fsck.c
just threw away blob content after loading it. Let's
hold onto it until after we've called fsck_object().
The easiest way to do this is to just drop the
parse_loose_object() helper entirely. Incidentally,
this also fixes a memory leak: if we successfully
loaded the object data but did not parse it, we would
have left the function without freeing it.
3. When fsck_loose() loads the object data, it
does so with a custom read_loose_object() helper. This
function streams any blobs, regardless of size, under
the assumption that we're only checking the sha1.
Instead, let's actually load blobs smaller than
big_file_threshold, as the normal object-reading
code-paths would do. This lets us fsck small files, and
a NULL return is an indication that the blob was so big
that it needed to be streamed, and we can pass that
information along to fsck_blob().
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 19:44:51 +00:00
|
|
|
static int fsck_blob(struct blob *blob, const char *buf,
|
|
|
|
unsigned long size, struct fsck_options *options)
|
|
|
|
{
|
2018-05-02 21:25:27 +00:00
|
|
|
struct fsck_gitmodules_data data;
|
2018-06-28 22:06:04 +00:00
|
|
|
struct config_options config_opts = { 0 };
|
2018-05-02 21:25:27 +00:00
|
|
|
|
|
|
|
if (!oidset_contains(&gitmodules_found, &blob->object.oid))
|
|
|
|
return 0;
|
|
|
|
oidset_insert(&gitmodules_done, &blob->object.oid);
|
|
|
|
|
fsck: check skiplist for object in fsck_blob()
Since commit ed8b10f631 ("fsck: check .gitmodules content", 2018-05-02),
fsck will issue an error message for '.gitmodules' content that cannot
be parsed correctly. This is the case, even when the corresponding blob
object has been included on the skiplist. For example, using the cgit
repository, we see the following:
$ git fsck
Checking object directories: 100% (256/256), done.
error: bad config line 5 in blob .gitmodules
error in blob 51dd1eff1edc663674df9ab85d2786a40f7ae3a5: gitmodulesParse: could not parse gitmodules blob
Checking objects: 100% (6626/6626), done.
$
$ git config fsck.skiplist '.git/skip'
$ echo 51dd1eff1edc663674df9ab85d2786a40f7ae3a5 >.git/skip
$
$ git fsck
Checking object directories: 100% (256/256), done.
error: bad config line 5 in blob .gitmodules
Checking objects: 100% (6626/6626), done.
$
Note that the error message issued by the config parser is still
present, despite adding the object-id of the blob to the skiplist.
One solution would be to provide a means of suppressing the messages
issued by the config parser. However, given that (logically) we are
asking fsck to ignore this object, a simpler approach is to just not
call the config parser if the object is to be skipped. Add a check to
the 'fsck_blob()' processing function, to determine if the object is
on the skiplist and, if so, exit the function early.
Signed-off-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2018-06-27 18:39:53 +00:00
|
|
|
if (object_on_skiplist(options, &blob->object))
|
|
|
|
return 0;
|
|
|
|
|
2018-05-02 21:25:27 +00:00
|
|
|
if (!buf) {
|
|
|
|
/*
|
|
|
|
* A missing buffer here is a sign that the caller found the
|
|
|
|
* blob too gigantic to load into memory. Let's just consider
|
|
|
|
* that an error.
|
|
|
|
*/
|
|
|
|
return report(options, &blob->object,
|
2018-07-13 19:39:53 +00:00
|
|
|
FSCK_MSG_GITMODULES_LARGE,
|
2018-05-02 21:25:27 +00:00
|
|
|
".gitmodules too large to parse");
|
|
|
|
}
|
|
|
|
|
|
|
|
data.obj = &blob->object;
|
|
|
|
data.options = options;
|
|
|
|
data.ret = 0;
|
2018-06-28 22:06:04 +00:00
|
|
|
config_opts.error_action = CONFIG_ERROR_SILENT;
|
2018-05-02 21:25:27 +00:00
|
|
|
if (git_config_from_mem(fsck_gitmodules_fn, CONFIG_ORIGIN_BLOB,
|
2018-06-28 22:06:04 +00:00
|
|
|
".gitmodules", buf, size, &data, &config_opts))
|
2018-05-02 21:25:27 +00:00
|
|
|
data.ret |= report(options, &blob->object,
|
|
|
|
FSCK_MSG_GITMODULES_PARSE,
|
|
|
|
"could not parse gitmodules blob");
|
|
|
|
|
|
|
|
return data.ret;
|
fsck: actually fsck blob data
Because fscking a blob has always been a noop, we didn't
bother passing around the blob data. In preparation for
content-level checks, let's fix up a few things:
1. The fsck_object() function just returns success for any
blob. Let's a noop fsck_blob(), which we can fill in
with actual logic later.
2. The fsck_loose() function in builtin/fsck.c
just threw away blob content after loading it. Let's
hold onto it until after we've called fsck_object().
The easiest way to do this is to just drop the
parse_loose_object() helper entirely. Incidentally,
this also fixes a memory leak: if we successfully
loaded the object data but did not parse it, we would
have left the function without freeing it.
3. When fsck_loose() loads the object data, it
does so with a custom read_loose_object() helper. This
function streams any blobs, regardless of size, under
the assumption that we're only checking the sha1.
Instead, let's actually load blobs smaller than
big_file_threshold, as the normal object-reading
code-paths would do. This lets us fsck small files, and
a NULL return is an indication that the blob was so big
that it needed to be streamed, and we can pass that
information along to fsck_blob().
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 19:44:51 +00:00
|
|
|
}
|
|
|
|
|
2014-09-10 13:52:51 +00:00
|
|
|
int fsck_object(struct object *obj, void *data, unsigned long size,
|
2015-06-22 15:25:00 +00:00
|
|
|
struct fsck_options *options)
|
2008-02-25 21:46:08 +00:00
|
|
|
{
|
|
|
|
if (!obj)
|
2015-06-22 15:25:09 +00:00
|
|
|
return report(options, obj, FSCK_MSG_BAD_OBJECT_SHA1, "no valid object to fsck");
|
2008-02-25 21:46:08 +00:00
|
|
|
|
|
|
|
if (obj->type == OBJ_BLOB)
|
fsck: actually fsck blob data
Because fscking a blob has always been a noop, we didn't
bother passing around the blob data. In preparation for
content-level checks, let's fix up a few things:
1. The fsck_object() function just returns success for any
blob. Let's a noop fsck_blob(), which we can fill in
with actual logic later.
2. The fsck_loose() function in builtin/fsck.c
just threw away blob content after loading it. Let's
hold onto it until after we've called fsck_object().
The easiest way to do this is to just drop the
parse_loose_object() helper entirely. Incidentally,
this also fixes a memory leak: if we successfully
loaded the object data but did not parse it, we would
have left the function without freeing it.
3. When fsck_loose() loads the object data, it
does so with a custom read_loose_object() helper. This
function streams any blobs, regardless of size, under
the assumption that we're only checking the sha1.
Instead, let's actually load blobs smaller than
big_file_threshold, as the normal object-reading
code-paths would do. This lets us fsck small files, and
a NULL return is an indication that the blob was so big
that it needed to be streamed, and we can pass that
information along to fsck_blob().
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 19:44:51 +00:00
|
|
|
return fsck_blob((struct blob *)obj, data, size, options);
|
2008-02-25 21:46:08 +00:00
|
|
|
if (obj->type == OBJ_TREE)
|
2015-06-22 15:25:00 +00:00
|
|
|
return fsck_tree((struct tree *) obj, options);
|
2008-02-25 21:46:08 +00:00
|
|
|
if (obj->type == OBJ_COMMIT)
|
2014-09-10 13:52:51 +00:00
|
|
|
return fsck_commit((struct commit *) obj, (const char *) data,
|
2015-06-22 15:25:00 +00:00
|
|
|
size, options);
|
2008-02-25 21:46:08 +00:00
|
|
|
if (obj->type == OBJ_TAG)
|
2014-09-10 13:52:51 +00:00
|
|
|
return fsck_tag((struct tag *) obj, (const char *) data,
|
2015-06-22 15:25:00 +00:00
|
|
|
size, options);
|
2008-02-25 21:46:08 +00:00
|
|
|
|
2015-06-22 15:25:09 +00:00
|
|
|
return report(options, obj, FSCK_MSG_UNKNOWN_TYPE, "unknown type '%d' (internal fsck error)",
|
2008-02-25 21:46:08 +00:00
|
|
|
obj->type);
|
|
|
|
}
|
2008-02-25 21:46:09 +00:00
|
|
|
|
2016-07-17 10:59:57 +00:00
|
|
|
int fsck_error_function(struct fsck_options *o,
|
|
|
|
struct object *obj, int msg_type, const char *message)
|
2008-02-25 21:46:09 +00:00
|
|
|
{
|
2015-06-22 15:25:25 +00:00
|
|
|
if (msg_type == FSCK_WARN) {
|
2016-07-17 11:00:02 +00:00
|
|
|
warning("object %s: %s", describe_object(o, obj), message);
|
2015-06-22 15:25:25 +00:00
|
|
|
return 0;
|
|
|
|
}
|
2016-07-17 11:00:02 +00:00
|
|
|
error("object %s: %s", describe_object(o, obj), message);
|
2008-02-25 21:46:09 +00:00
|
|
|
return 1;
|
|
|
|
}
|
fsck: detect gitmodules files
In preparation for performing fsck checks on .gitmodules
files, this commit plumbs in the actual detection of the
files. Note that unlike most other fsck checks, this cannot
be a property of a single object: we must know that the
object is found at a ".gitmodules" path at the root tree of
a commit.
Since the fsck code only sees one object at a time, we have
to mark the related objects to fit the puzzle together. When
we see a commit we mark its tree as a root tree, and when
we see a root tree with a .gitmodules file, we mark the
corresponding blob to be checked.
In an ideal world, we'd check the objects in topological
order: commits followed by trees followed by blobs. In that
case we can avoid ever loading an object twice, since all
markings would be complete by the time we get to the marked
objects. And indeed, if we are checking a single packfile,
this is the order in which Git will generally write the
objects. But we can't count on that:
1. git-fsck may show us the objects in arbitrary order
(loose objects are fed in sha1 order, but we may also
have multiple packs, and we process each pack fully in
sequence).
2. The type ordering is just what git-pack-objects happens
to write now. The pack format does not require a
specific order, and it's possible that future versions
of Git (or a custom version trying to fool official
Git's fsck checks!) may order it differently.
3. We may not even be fscking all of the relevant objects
at once. Consider pushing with transfer.fsckObjects,
where one push adds a blob at path "foo", and then a
second push adds the same blob at path ".gitmodules".
The blob is not part of the second push at all, but we
need to mark and check it.
So in the general case, we need to make up to three passes
over the objects: once to make sure we've seen all commits,
then once to cover any trees we might have missed, and then
a final pass to cover any .gitmodules blobs we found in the
second pass.
We can simplify things a bit by loosening the requirement
that we find .gitmodules only at root trees. Technically
a file like "subdir/.gitmodules" is not parsed by Git, but
it's not unreasonable for us to declare that Git is aware of
all ".gitmodules" files and make them eligible for checking.
That lets us drop the root-tree requirement, which
eliminates one pass entirely. And it makes our worst case
much better: instead of potentially queueing every root tree
to be re-examined, the worst case is that we queue each
unique .gitmodules blob for a second look.
This patch just adds the boilerplate to find .gitmodules
files. The actual content checks will come in a subsequent
commit.
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 21:20:08 +00:00
|
|
|
|
|
|
|
int fsck_finish(struct fsck_options *options)
|
|
|
|
{
|
|
|
|
int ret = 0;
|
|
|
|
struct oidset_iter iter;
|
|
|
|
const struct object_id *oid;
|
|
|
|
|
|
|
|
oidset_iter_init(&gitmodules_found, &iter);
|
|
|
|
while ((oid = oidset_iter_next(&iter))) {
|
|
|
|
struct blob *blob;
|
|
|
|
enum object_type type;
|
|
|
|
unsigned long size;
|
|
|
|
char *buf;
|
|
|
|
|
|
|
|
if (oidset_contains(&gitmodules_done, oid))
|
|
|
|
continue;
|
|
|
|
|
2018-06-29 01:21:55 +00:00
|
|
|
blob = lookup_blob(the_repository, oid);
|
fsck: detect gitmodules files
In preparation for performing fsck checks on .gitmodules
files, this commit plumbs in the actual detection of the
files. Note that unlike most other fsck checks, this cannot
be a property of a single object: we must know that the
object is found at a ".gitmodules" path at the root tree of
a commit.
Since the fsck code only sees one object at a time, we have
to mark the related objects to fit the puzzle together. When
we see a commit we mark its tree as a root tree, and when
we see a root tree with a .gitmodules file, we mark the
corresponding blob to be checked.
In an ideal world, we'd check the objects in topological
order: commits followed by trees followed by blobs. In that
case we can avoid ever loading an object twice, since all
markings would be complete by the time we get to the marked
objects. And indeed, if we are checking a single packfile,
this is the order in which Git will generally write the
objects. But we can't count on that:
1. git-fsck may show us the objects in arbitrary order
(loose objects are fed in sha1 order, but we may also
have multiple packs, and we process each pack fully in
sequence).
2. The type ordering is just what git-pack-objects happens
to write now. The pack format does not require a
specific order, and it's possible that future versions
of Git (or a custom version trying to fool official
Git's fsck checks!) may order it differently.
3. We may not even be fscking all of the relevant objects
at once. Consider pushing with transfer.fsckObjects,
where one push adds a blob at path "foo", and then a
second push adds the same blob at path ".gitmodules".
The blob is not part of the second push at all, but we
need to mark and check it.
So in the general case, we need to make up to three passes
over the objects: once to make sure we've seen all commits,
then once to cover any trees we might have missed, and then
a final pass to cover any .gitmodules blobs we found in the
second pass.
We can simplify things a bit by loosening the requirement
that we find .gitmodules only at root trees. Technically
a file like "subdir/.gitmodules" is not parsed by Git, but
it's not unreasonable for us to declare that Git is aware of
all ".gitmodules" files and make them eligible for checking.
That lets us drop the root-tree requirement, which
eliminates one pass entirely. And it makes our worst case
much better: instead of potentially queueing every root tree
to be re-examined, the worst case is that we queue each
unique .gitmodules blob for a second look.
This patch just adds the boilerplate to find .gitmodules
files. The actual content checks will come in a subsequent
commit.
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 21:20:08 +00:00
|
|
|
if (!blob) {
|
2018-06-11 08:35:45 +00:00
|
|
|
struct object *obj = lookup_unknown_object(oid->hash);
|
|
|
|
ret |= report(options, obj,
|
fsck: detect gitmodules files
In preparation for performing fsck checks on .gitmodules
files, this commit plumbs in the actual detection of the
files. Note that unlike most other fsck checks, this cannot
be a property of a single object: we must know that the
object is found at a ".gitmodules" path at the root tree of
a commit.
Since the fsck code only sees one object at a time, we have
to mark the related objects to fit the puzzle together. When
we see a commit we mark its tree as a root tree, and when
we see a root tree with a .gitmodules file, we mark the
corresponding blob to be checked.
In an ideal world, we'd check the objects in topological
order: commits followed by trees followed by blobs. In that
case we can avoid ever loading an object twice, since all
markings would be complete by the time we get to the marked
objects. And indeed, if we are checking a single packfile,
this is the order in which Git will generally write the
objects. But we can't count on that:
1. git-fsck may show us the objects in arbitrary order
(loose objects are fed in sha1 order, but we may also
have multiple packs, and we process each pack fully in
sequence).
2. The type ordering is just what git-pack-objects happens
to write now. The pack format does not require a
specific order, and it's possible that future versions
of Git (or a custom version trying to fool official
Git's fsck checks!) may order it differently.
3. We may not even be fscking all of the relevant objects
at once. Consider pushing with transfer.fsckObjects,
where one push adds a blob at path "foo", and then a
second push adds the same blob at path ".gitmodules".
The blob is not part of the second push at all, but we
need to mark and check it.
So in the general case, we need to make up to three passes
over the objects: once to make sure we've seen all commits,
then once to cover any trees we might have missed, and then
a final pass to cover any .gitmodules blobs we found in the
second pass.
We can simplify things a bit by loosening the requirement
that we find .gitmodules only at root trees. Technically
a file like "subdir/.gitmodules" is not parsed by Git, but
it's not unreasonable for us to declare that Git is aware of
all ".gitmodules" files and make them eligible for checking.
That lets us drop the root-tree requirement, which
eliminates one pass entirely. And it makes our worst case
much better: instead of potentially queueing every root tree
to be re-examined, the worst case is that we queue each
unique .gitmodules blob for a second look.
This patch just adds the boilerplate to find .gitmodules
files. The actual content checks will come in a subsequent
commit.
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 21:20:08 +00:00
|
|
|
FSCK_MSG_GITMODULES_BLOB,
|
|
|
|
"non-blob found at .gitmodules");
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
2018-05-29 08:09:58 +00:00
|
|
|
buf = read_object_file(oid, &type, &size);
|
fsck: detect gitmodules files
In preparation for performing fsck checks on .gitmodules
files, this commit plumbs in the actual detection of the
files. Note that unlike most other fsck checks, this cannot
be a property of a single object: we must know that the
object is found at a ".gitmodules" path at the root tree of
a commit.
Since the fsck code only sees one object at a time, we have
to mark the related objects to fit the puzzle together. When
we see a commit we mark its tree as a root tree, and when
we see a root tree with a .gitmodules file, we mark the
corresponding blob to be checked.
In an ideal world, we'd check the objects in topological
order: commits followed by trees followed by blobs. In that
case we can avoid ever loading an object twice, since all
markings would be complete by the time we get to the marked
objects. And indeed, if we are checking a single packfile,
this is the order in which Git will generally write the
objects. But we can't count on that:
1. git-fsck may show us the objects in arbitrary order
(loose objects are fed in sha1 order, but we may also
have multiple packs, and we process each pack fully in
sequence).
2. The type ordering is just what git-pack-objects happens
to write now. The pack format does not require a
specific order, and it's possible that future versions
of Git (or a custom version trying to fool official
Git's fsck checks!) may order it differently.
3. We may not even be fscking all of the relevant objects
at once. Consider pushing with transfer.fsckObjects,
where one push adds a blob at path "foo", and then a
second push adds the same blob at path ".gitmodules".
The blob is not part of the second push at all, but we
need to mark and check it.
So in the general case, we need to make up to three passes
over the objects: once to make sure we've seen all commits,
then once to cover any trees we might have missed, and then
a final pass to cover any .gitmodules blobs we found in the
second pass.
We can simplify things a bit by loosening the requirement
that we find .gitmodules only at root trees. Technically
a file like "subdir/.gitmodules" is not parsed by Git, but
it's not unreasonable for us to declare that Git is aware of
all ".gitmodules" files and make them eligible for checking.
That lets us drop the root-tree requirement, which
eliminates one pass entirely. And it makes our worst case
much better: instead of potentially queueing every root tree
to be re-examined, the worst case is that we queue each
unique .gitmodules blob for a second look.
This patch just adds the boilerplate to find .gitmodules
files. The actual content checks will come in a subsequent
commit.
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 21:20:08 +00:00
|
|
|
if (!buf) {
|
2018-05-14 16:22:48 +00:00
|
|
|
if (is_promisor_object(&blob->object.oid))
|
|
|
|
continue;
|
fsck: detect gitmodules files
In preparation for performing fsck checks on .gitmodules
files, this commit plumbs in the actual detection of the
files. Note that unlike most other fsck checks, this cannot
be a property of a single object: we must know that the
object is found at a ".gitmodules" path at the root tree of
a commit.
Since the fsck code only sees one object at a time, we have
to mark the related objects to fit the puzzle together. When
we see a commit we mark its tree as a root tree, and when
we see a root tree with a .gitmodules file, we mark the
corresponding blob to be checked.
In an ideal world, we'd check the objects in topological
order: commits followed by trees followed by blobs. In that
case we can avoid ever loading an object twice, since all
markings would be complete by the time we get to the marked
objects. And indeed, if we are checking a single packfile,
this is the order in which Git will generally write the
objects. But we can't count on that:
1. git-fsck may show us the objects in arbitrary order
(loose objects are fed in sha1 order, but we may also
have multiple packs, and we process each pack fully in
sequence).
2. The type ordering is just what git-pack-objects happens
to write now. The pack format does not require a
specific order, and it's possible that future versions
of Git (or a custom version trying to fool official
Git's fsck checks!) may order it differently.
3. We may not even be fscking all of the relevant objects
at once. Consider pushing with transfer.fsckObjects,
where one push adds a blob at path "foo", and then a
second push adds the same blob at path ".gitmodules".
The blob is not part of the second push at all, but we
need to mark and check it.
So in the general case, we need to make up to three passes
over the objects: once to make sure we've seen all commits,
then once to cover any trees we might have missed, and then
a final pass to cover any .gitmodules blobs we found in the
second pass.
We can simplify things a bit by loosening the requirement
that we find .gitmodules only at root trees. Technically
a file like "subdir/.gitmodules" is not parsed by Git, but
it's not unreasonable for us to declare that Git is aware of
all ".gitmodules" files and make them eligible for checking.
That lets us drop the root-tree requirement, which
eliminates one pass entirely. And it makes our worst case
much better: instead of potentially queueing every root tree
to be re-examined, the worst case is that we queue each
unique .gitmodules blob for a second look.
This patch just adds the boilerplate to find .gitmodules
files. The actual content checks will come in a subsequent
commit.
Signed-off-by: Jeff King <peff@peff.net>
2018-05-02 21:20:08 +00:00
|
|
|
ret |= report(options, &blob->object,
|
|
|
|
FSCK_MSG_GITMODULES_MISSING,
|
|
|
|
"unable to read .gitmodules blob");
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (type == OBJ_BLOB)
|
|
|
|
ret |= fsck_blob(blob, buf, size, options);
|
|
|
|
else
|
|
|
|
ret |= report(options, &blob->object,
|
|
|
|
FSCK_MSG_GITMODULES_BLOB,
|
|
|
|
"non-blob found at .gitmodules");
|
|
|
|
free(buf);
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
oidset_clear(&gitmodules_found);
|
|
|
|
oidset_clear(&gitmodules_done);
|
|
|
|
return ret;
|
|
|
|
}
|