git/object-file.h
Taylor Blau b1b8dfde69 finalize_object_file(): implement collision check
We've had "FIXME!!! Collision check here ?" in finalize_object_file()
since aac1794132 (Improve sha1 object file writing., 2005-05-03). That
is, when we try to write a file with the same name, we assume the
on-disk contents are the same and blindly throw away the new copy.

One of the reasons we never implemented this is because the files it
moves are all named after the cryptographic hash of their contents
(either loose objects, or packs which have their hash in the name these
days). So we are unlikely to see such a collision by accident. And even
though there are weaknesses in sha1, we assume they are mitigated by our
use of sha1dc.

So while it's a theoretical concern now, it hasn't been a priority.
However, if we start using weaker hashes for pack checksums and names,
this will become a practical concern. So in preparation, let's actually
implement a byte-for-byte collision check.

The new check will cause the write of new differing content to be a
failure, rather than a silent noop, and we'll retain the temporary file
on disk. If there's no collision present, we'll clean up the temporary
file as usual after either rename()-ing or link()-ing it into place.

Note that this may cause some extra computation when the files are in
fact identical, but this should happen rarely.

Loose objects are exempt from this check, and the collision check may be
skipped by calling the _flags variant of this function with the
FOF_SKIP_COLLISION_CHECK bit set. This is done for a couple of reasons:

  - We don't treat the hash of the loose object file's contents as a
    checksum, since the same loose object can be stored using different
    bytes on disk (e.g., when adjusting core.compression, using a
    different version of zlib, etc.).

    This is fundamentally different from cases where
    finalize_object_file() is operating over a file which uses the hash
    value as a checksum of the contents. In other words, a pair of
    identical loose objects can be stored using different bytes on disk,
    and that should not be treated as a collision.

  - We already use the path of the loose object as its hash value /
    object name, so checking for collisions at the content level doesn't
    add anything.

    Adding a content-level collision check would have to happen at a
    higher level than in finalize_object_file(), since (avoiding race
    conditions) writing an object loose which already exists in the
    repository will prevent us from even reaching finalize_object_file()
    via the object freshening code.

    There is a collision check in index-pack via its `check_collision()`
    function, but there isn't an analogous function in unpack-objects,
    which just feeds the result to write_object_file().

    So skipping the collision check here does not change for better or
    worse the hardness of loose object writes.

As a small note related to the latter bullet point above, we must teach
the tmp-objdir routines to similarly skip the content-level collision
checks when calling migrate_one() on a loose object file, which we do by
setting the FOF_SKIP_COLLISION_CHECK bit when we are inside of a loose
object shard.

Co-authored-by: Jeff King <peff@peff.net>
Signed-off-by: Jeff King <peff@peff.net>
Helped-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
2024-09-27 11:27:47 -07:00

138 lines
4.8 KiB
C

#ifndef OBJECT_FILE_H
#define OBJECT_FILE_H
#include "git-zlib.h"
#include "object.h"
struct index_state;
/*
* Set this to 0 to prevent oid_object_info_extended() from fetching missing
* blobs. This has a difference only if extensions.partialClone is set.
*
* Its default value is 1.
*/
extern int fetch_if_missing;
#define HASH_WRITE_OBJECT 1
#define HASH_FORMAT_CHECK 2
#define HASH_RENORMALIZE 4
#define HASH_SILENT 8
int index_fd(struct index_state *istate, struct object_id *oid, int fd, struct stat *st, enum object_type type, const char *path, unsigned flags);
int index_path(struct index_state *istate, struct object_id *oid, const char *path, struct stat *st, unsigned flags);
/*
* Create the directory containing the named path, using care to be
* somewhat safe against races. Return one of the scld_error values to
* indicate success/failure. On error, set errno to describe the
* problem.
*
* SCLD_VANISHED indicates that one of the ancestor directories of the
* path existed at one point during the function call and then
* suddenly vanished, probably because another process pruned the
* directory while we were working. To be robust against this kind of
* race, callers might want to try invoking the function again when it
* returns SCLD_VANISHED.
*
* safe_create_leading_directories() temporarily changes path while it
* is working but restores it before returning.
* safe_create_leading_directories_const() doesn't modify path, even
* temporarily. Both these variants adjust the permissions of the
* created directories to honor core.sharedRepository, so they are best
* suited for files inside the git dir. For working tree files, use
* safe_create_leading_directories_no_share() instead, as it ignores
* the core.sharedRepository setting.
*/
enum scld_error {
SCLD_OK = 0,
SCLD_FAILED = -1,
SCLD_PERMS = -2,
SCLD_EXISTS = -3,
SCLD_VANISHED = -4
};
enum scld_error safe_create_leading_directories(char *path);
enum scld_error safe_create_leading_directories_const(const char *path);
enum scld_error safe_create_leading_directories_no_share(char *path);
int mkdir_in_gitdir(const char *path);
int git_open_cloexec(const char *name, int flags);
#define git_open(name) git_open_cloexec(name, O_RDONLY)
/**
* unpack_loose_header() initializes the data stream needed to unpack
* a loose object header.
*
* Returns:
*
* - ULHR_OK on success
* - ULHR_BAD on error
* - ULHR_TOO_LONG if the header was too long
*
* It will only parse up to MAX_HEADER_LEN bytes unless an optional
* "hdrbuf" argument is non-NULL. This is intended for use with
* OBJECT_INFO_ALLOW_UNKNOWN_TYPE to extract the bad type for (error)
* reporting. The full header will be extracted to "hdrbuf" for use
* with parse_loose_header(), ULHR_TOO_LONG will still be returned
* from this function to indicate that the header was too long.
*/
enum unpack_loose_header_result {
ULHR_OK,
ULHR_BAD,
ULHR_TOO_LONG,
};
enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,
unsigned char *map,
unsigned long mapsize,
void *buffer,
unsigned long bufsiz,
struct strbuf *hdrbuf);
/**
* parse_loose_header() parses the starting "<type> <len>\0" of an
* object. If it doesn't follow that format -1 is returned. To check
* the validity of the <type> populate the "typep" in the "struct
* object_info". It will be OBJ_BAD if the object type is unknown. The
* parsed <len> can be retrieved via "oi->sizep", and from there
* passed to unpack_loose_rest().
*/
struct object_info;
int parse_loose_header(const char *hdr, struct object_info *oi);
/**
* With in-core object data in "buf", rehash it to make sure the
* object name actually matches "oid" to detect object corruption.
*
* A negative value indicates an error, usually that the OID is not
* what we expected, but it might also indicate another error.
*/
int check_object_signature(struct repository *r, const struct object_id *oid,
void *map, unsigned long size,
enum object_type type);
/**
* A streaming version of check_object_signature().
* Try reading the object named with "oid" using
* the streaming interface and rehash it to do the same.
*/
int stream_object_signature(struct repository *r, const struct object_id *oid);
enum finalize_object_file_flags {
FOF_SKIP_COLLISION_CHECK = 1,
};
int finalize_object_file(const char *tmpfile, const char *filename);
int finalize_object_file_flags(const char *tmpfile, const char *filename,
enum finalize_object_file_flags flags);
/* Helper to check and "touch" a file */
int check_and_freshen_file(const char *fn, int freshen);
void *read_object_with_reference(struct repository *r,
const struct object_id *oid,
enum object_type required_type,
unsigned long *size,
struct object_id *oid_ret);
#endif /* OBJECT_FILE_H */