mirror of
https://github.com/git/git
synced 2024-08-27 11:39:22 +00:00
![Patrick Steinhardt](/assets/img/avatar_default.png)
When serving a push, git-receive-pack(1) needs to verify that the packfile sent by the client contains all objects that are required by the updated references. This connectivity check works by marking all preexisting references as uninteresting and using the new reference tips as starting point for a graph walk. Marking all preexisting references as uninteresting can be a problem when it comes to performance. Git forges tend to do internal bookkeeping to keep alive sets of objects for internal use or make them easy to find via certain references. These references are typically hidden away from the user so that they are neither advertised nor writeable. At GitLab, we have one particular repository that contains a total of 7 million references, of which 6.8 million are indeed internal references. With the current connectivity check we are forced to load all these references in order to mark them as uninteresting, and this alone takes around 15 seconds to compute. We can optimize this by only taking into account the set of visible refs when marking objects as uninteresting. This means that we may now walk more objects until we hit any object that is marked as uninteresting. But it is rather unlikely that clients send objects that make large parts of objects reachable that have previously only ever been hidden, whereas the common case is to push incremental changes that build on top of the visible object graph. This provides a huge boost to performance in the mentioned repository, where the vast majority of its refs hidden. Pushing a new commit into this repo with `transfer.hideRefs` set up to hide 6.8 million of 7 refs as it is configured in Gitaly leads to a 4.5-fold speedup: Benchmark 1: main Time (mean ± σ): 30.977 s ± 0.157 s [User: 30.226 s, System: 1.083 s] Range (min … max): 30.796 s … 31.071 s 3 runs Benchmark 2: pks-connectivity-check-hide-refs Time (mean ± σ): 6.799 s ± 0.063 s [User: 6.803 s, System: 0.354 s] Range (min … max): 6.729 s … 6.850 s 3 runs Summary 'pks-connectivity-check-hide-refs' ran 4.56 ± 0.05 times faster than 'main' As we mostly go through the same codepaths even in the case where there are no hidden refs at all compared to the code before there is no change in performance when no refs are hidden: Benchmark 1: main Time (mean ± σ): 48.188 s ± 0.432 s [User: 49.326 s, System: 5.009 s] Range (min … max): 47.706 s … 48.539 s 3 runs Benchmark 2: pks-connectivity-check-hide-refs Time (mean ± σ): 48.027 s ± 0.500 s [User: 48.934 s, System: 5.025 s] Range (min … max): 47.504 s … 48.500 s 3 runs Summary 'pks-connectivity-check-hide-refs' ran 1.00 ± 0.01 times faster than 'main' Signed-off-by: Patrick Steinhardt <ps@pks.im> Signed-off-by: Taylor Blau <me@ttaylorr.com>
73 lines
1.9 KiB
C
73 lines
1.9 KiB
C
#ifndef CONNECTED_H
|
|
#define CONNECTED_H
|
|
|
|
struct object_id;
|
|
struct transport;
|
|
|
|
/*
|
|
* Take callback data, and return next object name in the buffer.
|
|
* When called after returning the name for the last object, return -1
|
|
* to signal EOF, otherwise return 0.
|
|
*/
|
|
typedef const struct object_id *(*oid_iterate_fn)(void *);
|
|
|
|
/*
|
|
* Named-arguments struct for check_connected. All arguments are
|
|
* optional, and can be left to defaults as set by CHECK_CONNECTED_INIT.
|
|
*/
|
|
struct check_connected_options {
|
|
/* Avoid printing any errors to stderr. */
|
|
int quiet;
|
|
|
|
/* --shallow-file to pass to rev-list sub-process */
|
|
const char *shallow_file;
|
|
|
|
/* Transport whose objects we are checking, if available. */
|
|
struct transport *transport;
|
|
|
|
/*
|
|
* If non-zero, send error messages to this descriptor rather
|
|
* than stderr. The descriptor is closed before check_connected
|
|
* returns.
|
|
*/
|
|
int err_fd;
|
|
|
|
/* If non-zero, show progress as we traverse the objects. */
|
|
int progress;
|
|
|
|
/*
|
|
* Insert these variables into the environment of the child process.
|
|
*/
|
|
const char **env;
|
|
|
|
/*
|
|
* If non-zero, check the ancestry chain completely, not stopping at
|
|
* any existing ref. This is necessary when deepening existing refs
|
|
* during a fetch.
|
|
*/
|
|
unsigned is_deepening_fetch : 1;
|
|
|
|
/*
|
|
* If not NULL, use `--exclude-hidden=$section` to exclude all refs
|
|
* hidden via the `$section.hideRefs` config from the set of
|
|
* already-reachable refs.
|
|
*/
|
|
const char *exclude_hidden_refs_section;
|
|
};
|
|
|
|
#define CHECK_CONNECTED_INIT { 0 }
|
|
|
|
/*
|
|
* Make sure that all given objects and all objects reachable from them
|
|
* either exist in our object store or (if the repository is a partial
|
|
* clone) are promised to be available.
|
|
*
|
|
* Return 0 if Ok, non zero otherwise (i.e. some missing objects)
|
|
*
|
|
* If "opt" is NULL, behaves as if CHECK_CONNECTED_INIT was passed.
|
|
*/
|
|
int check_connected(oid_iterate_fn fn, void *cb_data,
|
|
struct check_connected_options *opt);
|
|
|
|
#endif /* CONNECTED_H */
|